Skip to content

Guides

Task-oriented how-tos for the harness. Start with the pipeline overview, then dive into the skill that matches your step. Every guide uses the same eval.yaml — only the CLI flags change between local, containerized, and platform runs.

New here?

Read Skill mode vs prompt mode first — it decides which eval.yaml shape (and which guides) apply to you.

Orientation

  • The eval pipeline

    How setup, analyze, dataset, run, compare/anova, review, optimize, and mlflow fit together.

    Pipeline at a glance

  • Skill vs prompt mode

    Test a predefined skill (execution.skill) or an agent capability directly (execution.prompt).

    Choose a mode

The eval-* skills

The /eval-* slash commands drive the workflow in order. Each maps to a skill under skills/.

  • /eval-analyze

    Understand a skill or docs and generate an eval.yaml (+ eval.md).

    Generate a config

  • /eval-dataset

    Build test cases by skill authoring, synthetic generation, or from traces.

    Build a dataset

  • /eval-run

    Execute the suite, collect artifacts, score with judges, build the HTML report.

    Run an eval

  • /eval-compare

    Compare models or runs side-by-side into one self-contained HTML report — with the ANOVA, contrasts, and Pareto section when an anova.json is present.

    Compare models/runs

  • /eval-anova

    Sweep a matrix of models/configs and test whether score differences are statistically real: per-term tests with Holm/BH-corrected p-values, post-hoc level contrasts ("A vs B: estimate, adjusted p"), and opt-in per-judge screening.

    Analyze variance

  • /eval-review

    Present results, collect human feedback, and turn it into targeted changes — and into judge calibration: your labels become few-shot anchors via judges[].examples.

    Review results

  • /eval-optimize

    Run the automated refinement loop (composes with /eval-run) until judges pass.

    Optimize a skill

  • /eval-mlflow

    Sync datasets, log run results, and push/pull trace feedback.

    Log to MLflow

  • /eval-check

    Scan the whole harness for skill/command overlap and configuration issues.

    Health-check

Running headless & at scale

The same eval.yaml runs unchanged across execution backends — the backend is a --runner CLI flag, never a config key. The model provider is not a backend either: it is the id you write on a role (see Model providers).

  • Running headless

    Tool interception auto-answers AskUserQuestion and gates external services so skills run unattended.

    Headless execution

  • Harbor (containers)

    Run in containers via Podman (local) or Kubernetes/OpenShift.

    Run on Harbor

  • EvalHub

    Run the eval in-process inside an EvalHub Job pod.

    Run on EvalHub

  • Running on OpenRouter

    Put the agent under test or the judges on an open-weights model with an openrouter:/<author>/<slug> id — key, routing pins, enforcement, cost provenance.

    Run on OpenRouter

Continuous integration

Looking for field-by-field details?

These guides are how-tos. For the exhaustive config reference, see the eval.yaml schema; for the underlying ideas, see Concepts.