Cookbook

Worked, copy-pasteable recipes for common evaluation scenarios. Each one is a full eval.yaml plus the CLI commands to run it. Pick the card that matches what you're testing.

Same config, any backend

Every recipe is an ordinary eval.yaml. The execution backend (Local, Harbor, EvalHub) is always the --runner flag — never a config key — so any recipe runs unchanged across all three. See backends.

Recipes

• Evaluate a skill (case mode)

  ---

  When you have a skill and want one invocation per test case — the default flow.

• Evaluate a skill (batch mode)

  ---

  When your skill loops over all cases internally in a single batch.yaml invocation.

• Agentic documentation testing

  ---

  When you want to test whether agents can navigate and correctly use your docs.

• Cross-runner: OpenCode

  ---

  When the agent under test isn't Claude Code — drive it through the opaque cli runner.

• Reward shaping for RL

  ---

  When you need to collapse judges into a single [0, 1] scalar for GRPO-style training.

• Comparing runs with ANOVA

  ---

  When you're sweeping models or configs and want to know which score differences are statistically real — corrected p-values, level contrasts, per-judge screening.

• Write your own analysis recipe

  ---

  When you want /eval-analyze --prompt to bootstrap configs for a new domain.

• Write custom judges

  ---

  When builtin judges aren't enough — author check, module, and LLM judges: boolean-first prompts, one failure mode each, calibrated from human labels.

Choosing a recipe

flowchart TD
    A[What are you evaluating?] --> B{Testing a skill<br/>or a capability?}
    B -->|A predefined skill| C{One invocation<br/>per case?}
    B -->|Agent capability| D[Prompt mode]
    C -->|Yes| E[skill-case]
    C -->|No, loops internally| F[skill-batch]
    D --> G{Testing docs?}
    G -->|Yes| H[agentic-docs]
    G -->|No, custom domain| I[custom-analysis-recipe]
    A --> J{Non-Claude agent?}
    J -->|Yes| K[cross-runner-opencode]
    A --> L{Need a reward scalar?}
    L -->|Yes| M[reward-rl]
    A --> N{Scoring gaps?}
    N -->|Yes| O[custom-judges]
    A --> P{Comparing configs<br/>for significance?}
    P -->|Yes| Q[anova]
    A --> S{A judge disagrees<br/>with you?}
    S -->|Yes| T["eval-review → judges[].examples"]

The last branch isn't a recipe but a loop: label the disputed cases with /eval-review, triage the disagreement (underspecified skill prompt first, bad case second, miscalibrated judge last), and declare examples: on the judge so your labels become its few-shot calibration anchors on the next run.

Start from the walkthrough

New to the harness? Do your first eval end to end first, then come back here for the variant that fits your case. The full field reference lives in the eval.yaml schema.