Judgeval is the open-source core of Judgment Labs’ continuous-improvement stack for AI agents: tracing, LLM-as-judge evaluation, and online monitoring that help teams detect failures, triage root causes, and ship fixes validated against production data. The Python SDK is Apache-2.0 licensed, has earned over 1,000 stars on GitHub, and is under very active development, with commits landing in October 2026.
Judgeval instruments your application with OpenTelemetry-based tracing โ decorate a function with its observe decorator and it captures inputs, outputs, and LLM token usage automatically, staying compatible with your existing observability stack. On top of the traces sit agent judges: prompt-based scorers that evaluate behavior at scale and produce structured, labeled records of how your agents actually acted, which accumulate into a searchable history of agent behavior over time. Judges can score live production traffic server-side, or replay against historical traces, so you can verify a fix before it ships. Integrations cover popular agent frameworks such as LangChain, LangGraph, and LlamaIndex, and the SDK also supports reinforcement-learning workflows for agent improvement.
The project positions evaluation as a continuous loop rather than a one-off test: production traces feed judges, judges surface failure patterns, and the same harness replays those cases after a prompt or model change to confirm the fix and catch regressions. Environment data collected from real traffic can also feed reinforcement-learning workflows, turning observed agent behavior into a training signal for the next iteration. For teams iterating quickly on agents, that tight loop between observation, evaluation, and improvement is the product’s central promise.
The Judgeval SDK is free and open source under Apache-2.0. Judgment Labs also offers a hosted platform with managed monitoring and infrastructure on paid plans, so the open-source core follows a freemium model.
Judgeval is built for teams running LLM agents in production who need systematic evaluation and observability rather than spot-checking outputs by hand.