Skip to content

Make agent performance measurable — and improvable

Evaluate skills and agent capabilities with one declarative eval.yaml: analyze, generate cases, run, judge, trace in MLflow, then optimize; compare runs to keep only real gains. Same config on your laptop, Harbor containers, or EvalHub.

Get started eval.yaml reference

Python 3.11+ Apache-2.0 Claude plugin MLflow Harbor

Made at Red Hat

How the loop works

  • 1 · Analyze

    Point /eval-analyze at a skill or a prompt brief. The harness writes eval.yaml with judges, schema, and thresholds.

  • 2 · Dataset

    /eval-dataset fills cases from your schema — or bring your own cases/ tree with gold references.

  • 3 · Run & judge

    /eval-run executes on Claude Code (or another runner), scores with LLM + code judges, and emits a rich HTML report.

  • 4 · Review

    Optional /eval-review captures human feedback on scored cases before you change the skill or config.

  • 5 · Trace

    Optional /eval-mlflow syncs metrics, artifacts, and hierarchical GenAI traces for every case.

  • 6 · Optimize

    /eval-optimize proposes skill fixes from failures and re-runs so you keep only real gains.

  • ↻ · Compare

    Close the loop with evidence: /eval-compare lines runs up side by side, and /eval-anova tells you whether a model or config difference is statistically real before you keep it.

See the full pipeline guide

What you get

  • Skill or prompt mode

    Test a packaged skill (execution.skill) or agent capability directly (execution.prompt) — including agentic documentation checks.

    Execution model

  • LLM + code judges

    Built-in judges, inline Python checks, rubrics, pairwise A/B, and N-sample stability — all in one config. LLM judges reason before they rule, read graded material fenced as untrusted (prompt-injection mitigation), and calibrate on few-shot anchors harvested from your /eval-review labels.

    Judges & scoring

  • Is the difference real?

    /eval-compare puts runs side by side; /eval-anova sweeps a model × config matrix and answers statistically — per-term Wald tests with Holm/Benjamini–Hochberg correction, post-hoc level contrasts, per-judge screening, and a cost/quality Pareto frontier.

    Analysis of variance

  • One config, three backends

    Local subprocess, Harbor (Podman / OpenShift), or EvalHub — backend is a CLI flag, never baked into eval.yaml.

    Execution backends

  • Any agent runtime

    Claude Code out of the box; bring OpenCode or a custom CLI / Responses API runner when you need it.

    Runners

  • Any model provider

    Claude on Anthropic or Vertex by default; open-weights models through OpenRouter on the agent and judge roles with openrouter:/<author>/<slug>; OpenAI-compatible judges.

    Model providers

  • Reward API for RL

    Collapse judges into a [0, 1] reward for GRPO-style training via Harbor / NeMo Gym / SkyRL.

    Reward API

  • MLflow-native

    Experiments, datasets, hierarchical traces, and feedback sync — opt in with one mlflow: block.

    Tracing

Choose your path

Path Use it when Start
Claude Code plugin You want slash commands in an existing project claude plugin install agent-eval-harness@opendatahub-skills
Local clone You are hacking on the harness itself git clone https://github.com/opendatahub-io/agent-eval-harness
Harbor / OpenShift You need containerized, reproducible trials Running on Harbor 
claude plugin install agent-eval-harness@opendatahub-skills
/eval-setup
/eval-analyze --skill my-skill
/eval-dataset
/eval-run --model opus

Explore the docs

  • Get Started

    Install and run your first evaluation end to end.

    Get Started

  • Guides

    Task-oriented how-tos for every skill and backend.

    Guides

  • Concepts

    Execution model, judges, rewards, and tracing.

    Concepts

  • Cookbook

    Worked configs for common evaluation scenarios.

    Cookbook

  • Reference

    eval.yaml schema, CLI, config fields, and glossary.

    Reference