The eval.yaml schema¶

A single eval.yaml in your project root drives everything. It is parsed into an EvalConfig and validated at load time. Every top-level key is optional and has a sensible default — a minimal config is just a name, what to execute, a dataset, and one judge.

Describe what, not where

eval.yaml describes what to evaluate. The execution backend (Local, Harbor, EvalHub) is always a CLI flag (--runner), never a config key — so the same file runs unchanged everywhere.

Every top-level key is optional. name, description, and title are the only non-deprecated top-level scalars. LLM marks natural-language fields interpreted by agents & judges, PY marks Python code or expressions, and [] marks a list.

Define · what to evaluate

dataset

Where cases live and what they contain

path schemaLLM workspace.files[]

generation

How /eval-dataset sources cases

strategy contextLLM seeds[]

inputs.tools[]

Tool interception for headless runs

matchLLM promptLLM prompt_fileLLM

outputs[]

Artifacts and tool calls to collect

path tool schemaLLM batch_pattern types

Execute · how it runs

execution

What runs (skill or prompt), per case or batch

mode skill promptLLM arguments timeout max_budget_usd parallelism env

runner

Agent runtime and knobs

type effort settings plugin_dirs[] env system_promptLLM command workspace_mode

models

Model per role (CLI flags override)

skill subagent judge hook providers

permissions

Tool allow / deny for headless runs

allow[] deny[]

hooks

Lifecycle shell commands

before_all[] before_each[] after_each[] before_scoring[] after_all[] before_report[]

skilldeprecated

Top-level skill: — use execution.skill

Score · how it's judged

judges[]

How each case is scored — five judge types

name descriptionLLM ifPY checkPY promptLLM prompt_fileLLM llm_rubricLLM builtin modulePY functionPY agent context[] model arguments samples examples score_range feedback_type

thresholds

Regression gates per judge

min_mean min_pass_rate min_win_rate max_error_rate

reward

Collapse judges into an RL scalar in [0, 1]

judge normalize formulaPY weights gate score_range raw[]

matrix

Factorial grid /eval-anova fans out and tests

factors replications analysis.correction analysis.per_judge

Observe · what's captured

mlflow

Experiment tracking — presence opts in

experiment tracking_uri tags

traces

Execution data captured for judges

stdout stderr events metrics

Top-level keys¶

Key	Purpose	Reference
`name`	Experiment / run name (defaults to the file stem)	(inline)
`description`	Human-readable description	(inline)
`title`	HTML report heading (default: `Agent Eval Report`; `--title` overrides)	(inline)
`extends`	Layer this file over a base config (a profile)	extends
`execution`	What to run and how cases are processed	execution
`runner`	Agent runtime + runtime-specific knobs	runner
`models`	Model per role: skill, subagent, judge, hook; `providers` registers how `provider:/` ids are served	models · models.providers
`permissions`	Tool allow/deny for headless runs	permissions
`mlflow`	Experiment tracking (opt-in)	mlflow
`dataset`	Where cases live and what they contain	dataset
`generation`	How `/eval-dataset` sources cases	generation
`inputs`	Tool interception handlers (`inputs.tools`)	inputs.tools
`outputs`	Artifacts / tool calls to collect	outputs
`traces`	Which execution data to capture	traces
`hooks`	Lifecycle shell hooks	hooks
`judges`	How each case is scored	judges
`thresholds`	Regression gates per judge	thresholds
`reward`	Collapse judges into an RL reward scalar	reward
`matrix`	Factorial design for `/eval-anova`: factors, replications, p-value correction	matrix
`skill`	Deprecated — use `execution.skill`	(see below)

skill: at the top level is deprecated

A top-level skill: still works but is auto-normalized into execution.skill with a deprecation warning. Always author new configs with execution.skill.

Two minimal configs¶

Which keys you set depends on whether you're testing a skill or a capability. See the execution model for the difference.

Skill modePrompt mode

name: my-skill-eval

execution:
  mode: case
  skill: my-skill
  arguments: "{prompt}"

dataset:
  path: eval/dataset/cases
  schema: "Each case has an input.yaml with a 'prompt' field."

judges:
  - name: completeness
    prompt: "Score 1-5 how completely the output covers the request (1 = most requirements missing, 3 = basics with gaps, 5 = complete).\n\nRequest:\n{{ inputs }}\n\nOutput:\n{{ outputs }}"
    score_range: [1, 5]   # declare the scale — omitting it warns at config load

name: docs-navigation-eval

execution:
  mode: case
  prompt: "{{ input.prompt }}"

runner:
  type: claude-code
  workspace_mode: repo   # navigate the real repository

dataset:
  path: eval/dataset/cases
  schema: "Each case has an input.yaml with a 'prompt' field."

judges:
  - name: used_docs
    builtin: consulted_docs

A fully annotated config¶

The repository's root eval.yaml is the canonical, heavily-commented reference — every block with inline comments and commented-out variants for all five judge types, tool interception, batch_pattern, and thresholds. It's the best single file to copy from.

eval.yaml (excerpt)

name: my-skill-eval
description: Evaluate the main skill pipeline
title: My Skill Eval Report   # HTML report heading (default: Agent Eval Report)

execution:
  mode: case              # per-case (default) or batch
  skill: my-skill-name    # skill to test (use `prompt:` for prompt mode)
  arguments: "{prompt}"   # resolved per case from input.yaml fields

runner:
  type: claude-code       # claude-code | cursor | codex | cli | responses-api
  # effort: high          # accepted values are runner-specific: claude-code
  #                       # low|medium|high|xhigh|max; codex low|medium|high|xhigh;
  #                       # cursor is model-specific

models:
  skill: claude-opus-4-6  # required (or pass --model); openrouter:/<author>/<slug> activates models.providers
  judge: claude-opus-4-6  # used by LLM and pairwise judges
  # providers:            # the openrouter block behind provider:/ ids — see models.providers

permissions:
  deny:
    - "mcp__*"            # block all MCP tools during eval

mlflow:
  experiment: my-skill-eval   # opt-in: omit the block to disable tracking

dataset:
  path: eval/dataset/cases
  schema: |
    Each case has input.yaml (a 'prompt' field) and reference.md (gold output).

outputs:
  - path: artifacts
    schema: "One markdown file per case, named NNN-slug.md."

traces:
  stdout: true
  stderr: true
  events: false
  metrics: true

judges:
  - name: has_content
    check: |
      content = outputs["main_content"]
      if len(content.strip()) < 100:
          return False, f"Output too short ({len(content.strip())} chars)"
      return True, f"Output has {len(content.strip())} chars"

  - name: completeness
    prompt: "Score 1-5 how completely the output covers the request (1 = most requirements missing, 3 = basics with gaps, 5 = complete).\n\nRequest:\n{{ inputs }}\n\nOutput:\n{{ outputs }}"
    score_range: [1, 5]     # declare the scale — omitting it warns at config load

thresholds:
  has_content: { min_pass_rate: 1.0 }
  completeness: { min_mean: 3.5 }

Conventions¶

Schema fields are natural language. dataset.schema and outputs[].schema are documentation for the LLM agents and judges — scripts operate on file paths, not a parsed spec. There are no hardcoded field names.
Load-time validation is strict. Mutually-exclusive keys (skill + prompt), invalid enums (execution.mode), and malformed reward formulas fail at load, not mid-run.
One loader. Every reader of an eval config goes through the same loader, so a profile that extends a base is merged identically for execution, scoring, reporting, Harbor bundling and validation.

Per-key reference¶

extends — layer a profile over a base config; merge policy and !replace
execution — mode, skill/prompt, arguments, timeout, budget, parallelism, env
runner — type, effort, permission_mode, settings, plugin_dirs, env, system_prompt, command, workspace_mode
models — skill, subagent, judge, hook roles and precedence
models.providers — the openrouter block behind provider:/ ids: keys, routing, enforcement
permissions — allow/deny patterns and the path-based compiler
mlflow — experiment, tracking_uri, tags
dataset — path, schema, workspace.files
generation — strategy, context, seeds
inputs.tools — tool interception handlers
outputs — path vs tool, schema, batch_pattern, types
traces — stdout, stderr, events, metrics
hooks — before/after all/each, before_scoring
judges — the five judge types and all fields, including examples
thresholds — min_mean, min_pass_rate, min_win_rate, max_error_rate
reward — single-judge and formula reward modes
matrix — factors, replications, analysis.correction, analysis.per_judge