llm-eval

Open repos and models their makers tagged llm-eval, most starred first. The same for everyone.

AI on my own machineRepo

promptfoo

Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.

  • 27k
  • TypeScript
  • MIT
  • 10 h ago
promptfooTake it
AI on my own machineRepo

opencompass

OpenCompass is an LLM evaluation platform, supporting a wide range of models from OpenAI, Anthropic, Gemini, Qwen, GLM, DeepSeek, etc, across 100+ datasets covering knowledge, reasoning, coding, science, language, long-context, and safety.

  • 7.6k
  • Python
  • Apache-2.0
  • 1 d ago
open-compassTake it
HomelabRepo

uptrain

UpTrain is an open-source unified platform to evaluate and improve Generative AI applications. We provide grades for 20+ preconfigured checks (covering language, code, embedding use-cases), perform root cause analysis on failure cases and give insights on how to resolve them.

  • 2.4k
  • Python
  • Apache-2.0
  • 2 y ago
uptrain-aiTake it
GitHubRepo

verifywise

Complete AI governance and LLM Evals platform with support for EU AI Act, ISO 42001, NIST AI RMF and 20+ more AI frameworks and regulations. Join our Discord channel: https://discord.com/invite/d3k3E4uEpR

  • 377
  • TypeScript
  • 12 h ago
GitHubRepo

dsh-design-qa

Design-fidelity QA for DeepSeek Harness: lend any text-only model an eye, then judge whether the implementation matches the mock. Ships the benchmark behind that judgement — four fixtures, 23 injected defects, and every raw model transcript. Retires itself when DeepSeek ships vision.

  • 44
  • JavaScript
  • MIT
  • 4 wk ago
sunxin-aiTake it
AI on my own machineRepo

evalcore

Snapshot testing for LLM apps and agents, built to run locally and block regressions in CI.

  • 19
  • Rust
  • Apache-2.0
  • 4 mo ago
Eval-coreTake it
AI on my own machineRepo

llm-comparison-backend

This is an opensource project allowing you to compare two LLM's head to head with a given prompt, this section will be regarding the backend of this project, allowing for llm api's to be incorporated and used in the front-end

  • 23
  • Python
  • Apache-2.0
  • 10 mo ago
SupahandsTake it
AI on my own machineRepo

ollama-multirun

Run a prompt against all, or some, of your models running on Ollama. Creates web pages with the output, performance statistics and model info. All in a single Bash shell script.

  • 16
  • Shell
  • MIT
  • 4 mo ago
attogramTake it
Search GitHub and Hugging Face for llm-eval

Get , open it once and sign in. Then come back here and press the button again.

llm-eval: open source repos and models on Take My Repo