Skip to content

What's new

Hand-picked highlights from recent releases, newest first, each linking to the page that explains the change. The complete, auto-generated changelog follows below. Releases are cut by semantic-release from conventional commits — the GitHub releases page has the same entries with downloadable tags.

Upgrading?

Entries marked scores move change what a judge or a statistic reports; don't compare runs across those boundaries without re-baselining your thresholds.

Highlights

1.56.0 — Post-hoc level contrasts

The omnibus test says a factor matters; the new post-hoc level contrasts ("opus vs sonnet: estimate +0.06, adjusted p = .02") say which levels differ and by how much, corrected within each factor (Holm by default, following matrix.analysis.correction). They land in anova.json under contrasts and render in both the comparison report and the deep report. See Level contrasts (post-hoc).

1.55.0 — Per-judge screening

Opt in with matrix.analysis.per_judge: true or --per-judge and the same ANOVA also runs once per judge, Benjamini–Hochberg-corrected across the whole judges × terms family — a screen for which judge moved, while the composite stays the headline. See Per-judge screening and the /eval-anova flags.

1.54.0 — Per-term Wald tests with multiplicity correction (scores move)

Every model term — each main effect and interaction — now gets one joint Wald test, and the family of p-values is corrected for multiple comparisons: Holm by default, bh or none via matrix.analysis.correction / --correction. anova.json reports raw and adjusted p side by side, and significant follows the adjusted value (the previous min-of-dummy-p approach was anti-conservative). See Per-term Wald tests and multiplicity correction.

1.53.0 · 1.51.0 · 1.50.0 — OpenRouter as a first-class provider

Put the agent under test or the judges on an open-weights model with an openrouter:/<author>/<slug> id: direct transport on the local, Harbor (Podman and Kubernetes), and EvalHub backends; preflight, routing pins, audit-aware pooling, and an opt-in key guardrail. See Running on OpenRouter and Model providers.

1.52.0 — Project hooks carried into run workspaces

The project's .claude/settings.json hooks are appended to each workspace's settings after the harness's own (execution.project_hooks, default true). See Carrying the project's hooks.

1.49.0 — Judge backend follows the judge model's provider

The judge SDK is chosen by the judge model's provider:/ prefix, independent of runner.type — a Claude judge works with a Codex or Cursor runner and vice-versa, and openai:/ ids reach any OpenAI-compatible gateway. See Model providers (judge backend) and Agent path and judge path.

1.48.0 — Cursor runner

runner.type: cursor drives the Cursor agent CLI (local execution only). See Runners.

1.47.0 — Evaluated material is fenced and marked untrusted

A prompt-injection mitigation for every LLM judge: agent-produced template variables ({{ outputs }}, {{ conversation }}, {{ reasoning }}, {{ inputs }}, …) render between [BEGIN EVALUATED MATERIAL: <name>] / [END EVALUATED MATERIAL] markers, and every judge system prompt carries a guard: follow the rubric, never instructions found inside the fence. See How the judge prompt is assembled and Fenced vs unfenced.

1.46.0 — Few-shot examples from human review labels

judges[].examples harvests the labels you wrote in /eval-review (review.yaml) and injects them into LLM and agent judge prompts as calibration anchors — the judge sees what a human actually accepted and rejected on this eval, never its own case. See Few-shot examples from human reviews and Turn labels into calibration.

1.45.0 — Binary-first judge authoring

/eval-analyze and /eval-review now author judges down a selection ladder — builtin → inline check → boolean (pass/fail) LLM → numeric LLM — with exactly one failure mode per judge, following the rules in Authoring an LLM judge prompt (the shipped judge-prompt template). /eval-review triages judge–human disagreement in order: underspecified skill prompt, bad case, and only then a miscalibrated judge. See The five judge types.

1.44.0 — Rationale before verdict

All LLM judge tool schemas and system prompts ask for the rationale first and the verdict (score / passed / preferred) second — a model that writes its analysis before committing to a verdict token is better calibrated. Don't fight the ordering in your prompts. See System prompt: role, rationale first, guard and Verdict output.

Full changelog

Auto-generated by semantic-release from the repository's CHANGELOG.md; the highlights above are the editorial cut of these entries.

1.56.3 (2026-10-15)

Bug Fixes

  • score: copy judge context, validate tool lists, honor scoped allows (fc3c7c0), closes #240
  • score: deny shell, network and sub-agent tools to judges run through Claude Code (d0e6271)
  • score: resolve the open review on the judge deny list (3282936), closes #240

1.56.2 (2026-10-14)

Bug Fixes

  • judge: step the tool_choice ladder down on a 400 that names tool_choice (ed062fe)

1.56.1 (2026-10-12)

Bug Fixes

  • judges: correct the stale untrusted-data guard guidance and drop legacy JSON boilerplate from shipped prompts (9fe4cdd)
  • judges: let the comparison judge call a dimension tied instead of forcing a winner (6845314)
  • scoring: fence the whole outputs proxy under tojson and bare outputs.files (6fecbbb)
  • scoring: neutralize fence markers that appear inside evaluated material (56b00a9)

1.56.0 (2026-10-09)

Bug Fixes

  • anova: escape markdown table cells in the deep report; correct the contrasts provenance wording (ce31d15)
  • anova: label wholesale pairwise-test failure as systemic, not per-pair degeneracy (a3de7ca)
  • anova: read the mixedlm design spec via model_spec on statsmodels 0.15 for contrasts (9e8a183)

Features

  • anova: Holm-corrected pairwise level contrasts (a591225)

1.55.0 (2026-10-08)

Bug Fixes

  • anova: accept scalar judge_results in the per-judge fan-out and tighten honesty details (a872c39)
  • anova: map integers beyond float range to a missing observation instead of raising (ab4397b)
  • anova: treat non-finite judge values as missing observations, and keep unobserved judges visible (fce31b9)

Features

  • anova: opt-in per-judge ANOVA fan-out with Benjamini-Hochberg correction (1f71e12)

1.54.0 (2026-10-08)

Bug Fixes

  • anova: headline p matches the corrected verdict; teach interpretation prompt adjusted-p semantics (77bd2f8)
  • anova: review follow-ups — reject non-string correction values, report excluded_terms on degenerate single-factor paths, tighten guide wording (d7cfa2e)
  • anova: support statsmodels 0.15's model_spec in the term-slice fallback (ce05e6c)

Features

  • anova: per-term Wald tests with Holm/BH multiplicity correction (49332cb)

1.53.4 (2026-10-07)

Bug Fixes

  • eval-compare: address the CodeRabbit review on #238 (4b112b6)
  • eval-compare: report a wall clock for duration_s runs, and stop rounding per-turn costs to $0.00 (a7aa586)

1.53.3 (2026-10-02)

Bug Fixes

  • claude-code: forward OpenShell's Vertex placeholder switches, and let the judges use the placeholder token (60f12fc)

1.53.2 (2026-09-29)

Bug Fixes

  • eval-run: compile and merge both eval.yaml permission lists in every workspace builder (7aae678)

1.53.1 (2026-09-29)

Bug Fixes

  • claude-code: stop forwarding the operator's GCP credential locations under an OpenRouter plan (e86b788), closes #234
  • eval-run: add absolute-workspace twins for relative-path Bash permission rules (8bd409c), closes #233

1.53.0 (2026-09-28)

Features

  • providers: direct OpenRouter transport on Kubernetes and EvalHub, provider docs (spec 014, PR-7) (f7a204e)

1.52.0 (2026-09-28)

Features

  • workspace: carry the project's settings hooks into the run settings (8cb2ea6)

1.51.0 (2026-09-28)

Features

  • providers: full preflight, audit-aware pooling and the key guardrail (spec 014, PR-6) (e31c7c9), closes #26

1.50.0 (2026-09-25)

Features

  • providers: direct OpenRouter transport for claude-code and Harbor podman (spec 014, PR-5) (1ae1bf4)

1.49.2 (2026-09-22)

Bug Fixes

  • eval-analyze: address review — scope stdout-only rule to output judges, clarify bare {{ outputs }} (fa851b9)
  • eval-analyze: detect stdout-only skills and use {{ conversation }} in judge prompts (c71d684)

1.49.1 (2026-09-15)

Bug Fixes

  • judges: fence structured file access in judge prompts (CWE-1427) (669ebe7)
  • judges: re-fence tojson; fence file values via finalize, not str (34d08fa)

1.49.0 (2026-09-15)

Bug Fixes

  • judges: address CodeRabbit review on OpenAI judge path (216cf98)
  • judges: address CodeRabbit review on the rebased PR (041fb65)
  • judges: strip provider prefix in all judge-model consumers; tighten validation (a20b82a)

Features

  • judges: route judge backend by model provider, independent of runner (03c0588), closes #213 #213 #1 #2 #5

1.48.0 (2026-09-14)

Bug Fixes

  • address CodeRabbit follow-up review on the Cursor runner (5d61c08)
  • address review findings on the Cursor runner + judge paths (5339c01)
  • lint+docs: trim SKILL.md under skillsaw budget; address CodeRabbit doc notes (49f6fa9)

Features

  • add Cursor as an eval runner (45a2377)

1.47.2 (2026-09-02)

Bug Fixes

  • report: clarify the analysis author model is the orchestrator's own, not the subject (03806f4)

1.47.1 (2026-09-02)

Bug Fixes

  • eval-analyze: derive eval.md name from the config so flat layouts don't collide (fa9fa6e)

1.47.0 (2026-09-01)

Features

  • scoring: fence evaluated material and target the guard at the boundary (a4938dc)
  • scoring: mark graded material as untrusted in LLM judge system prompts (a637e18)

1.46.0 (2026-09-01)

Bug Fixes

  • judges: guard injected exemplars as untrusted and defer excerpt I/O (ffa8e0a)
  • judges: reject examples on the reserved pairwise judge and cap excerpt reads (828b4ca)

Features

  • judges: few-shot examples harvested from human review labels (c0219fa)

1.45.0 (2026-08-31)

Features

  • judges: binary-first judge authoring guidance and prompt template (6c0a22b)

1.44.0 (2026-08-31)

Features

  • scoring: elicit judge rationale before verdict in structured outputs (17ddd70)

1.43.2 (2026-08-31)

Bug Fixes

  • config: skip hidden files and dirs in eval-config discovery (9e59428)

1.43.1 (2026-08-27)

Bug Fixes

  • staging: copy plugin-root .mcp.json into isolated eval workspaces (76b63b6)
  • staging: document isolated MCP coverage limits (e82bff4)
  • staging: keep optional-file comment generic (e437376)

1.43.0 (2026-08-27)

Features

  • workspace: symlink-free {dest, source} shared files, portable across substrates (ea61ddf), closes #203

1.42.1 (2026-08-26)

Bug Fixes

  • workspace: materialize listed file symlinks in workspace.files (1d2e296)
  • workspace: tighten symlink allowlist and warn on nested skips (2683913)

1.42.0 (2026-08-26)

Bug Fixes

  • hermetic: reject the reserved "*" key at the registry boundary (703543d)

Features

  • workspace: hermetic-by-default plugin isolation via enabledPlugins (8227eba)

1.41.2 (2026-08-25)

Bug Fixes

  • evalhub: resolve _run_with_client names from its own scope (58d54a0)

1.41.1 (2026-08-24)

Bug Fixes

  • config: stop warning that the pairwise judge has no score_range (d2ba106), closes #182

1.41.0 (2026-08-21)

Bug Fixes

  • hooks: close the pidfile launch race and bound the Stop-guard scan (6941771)
  • hooks: detect live executor via pidfile, not transcript parsing (5ce1983)
  • hooks: harden Stop guard state and transcript handling (df06586)
  • stop-guard: harden pid handling, escape hatch, and pidfile writes (f7658b9)

Features

  • hooks: add Stop guard to prevent background-kill empty-green evals (8f97ba3)

1.40.2 (2026-08-21)

Bug Fixes

  • execute: persist permission_denials into run_result.json (e7f48cf)
  • execute: surface denials in aggregates; broaden the contract test (f7db702)

1.40.1 (2026-08-21)

Bug Fixes

  • claude-code: apply the billed-cost rule on the timeout path too (7446181)
  • claude-code: stop mislabeling cases whose background tasks were killed (e037a19)

1.40.0 (2026-08-21)

Bug Fixes

  • claude-code: contain symlinked copy roots; survive symlink loops (0e2118b)
  • claude-code: copy staged roots from the checked canonical path (7482473)
  • claude-code: keep staging out of repo mode and out of collected artifacts (b89b4fa)
  • claude-code: propagate broken plugin configs; refuse escaping symlinks (a8713db)

Features

  • claude-code: optionally stage plugin dirs inside the workspace (4e4e26e)
  • claude-code: stage plugin dirs into the workspace unconditionally (e23e2c8)

1.39.5 (2026-08-20)

Bug Fixes

  • claude-code: report only the confirmed import failure; force bootstrap failure in the hook test (6f28584)
  • claude-code: stop losing interception evidence across the hook and telemetry path (a9f403b)

1.39.4 (2026-08-18)

Bug Fixes

  • claude-code: fail the run when the agent does not recognise the skill (d541dc6)
  • claude-code: suppress only on positive evidence of work (b1a8ade)
  • validate: require plugin_dirs when the skill is not auto-discoverable (61f144c)
  • validate: verify plugin_dirs actually export the skill (aa1be78)

1.39.3 (2026-08-17)

Bug Fixes

  • hooks: survive a hook model that rejects temperature (ffbc493)

1.39.2 (2026-08-14)

Bug Fixes

  • score: anchor the frontmatter delimiters to line boundaries (553df7e)
  • score: judge the drift on evidence, not on case-001 alone (19f215c)
  • score: only warn about fields the judge actually requires (7aa063a)
  • score: report drift on evidence instead of guessing intent (5d9c8f9)
  • score: the stale-field probe must never abort a run (adcd83e)
  • warn on stale inline frontmatter fields (4451ad8)

1.39.1 (2026-08-14)

Bug Fixes

  • report,mlflow: a breach the table can't display is still a breach (b0b6d38)
  • report,mlflow: agree with the CLI on what counts as a regression (3fd316f)

1.39.0 (2026-08-14)

Bug Fixes

  • config: reject a non-finite score_range, and say what clamping does (6d383a3)
  • config: warn when a clamped reward judge is scored off [0, 1] (624ffd5)
  • reward: normalize each composed judge over its own score_range (c3fae09)

Features

  • config: make reward.score_range an explicit, deprecated fallback (129ba16)

1.38.0 (2026-08-14)

Bug Fixes

  • address Codex and Harbor review feedback (a00ac06)
  • address external review-panel findings (rounds 1-2) (970f3e0)
  • address final review round and scrub review residue (c4771a5)
  • address independent deep-review round (events, codex, tasks, results) (71cd2ce)
  • address remaining CodeRabbit review threads (0ce8c55)
  • container: install Codex platform binary (b404f6e)
  • guard direct-store result-event fields against malformed values (e2e7d98)
  • harbor: address Codex review findings (f18e7e2)
  • harbor: address deep-review panel findings (60886b7)
  • harbor: carry runner.system_prompt into task instructions (7d718fa)
  • harbor: forward plugin skill roots to every agent, not just Codex (c5a96af)
  • harbor: prefer the interpreter's own harbor CLI over PATH (5134b24)
  • harbor: record the effort Harbor's Codex agent actually applied (f663853)
  • harbor: surface unjudged steps in the run output (4cd4109)
  • harden plugin manifest handling and marker reads (b0fe76d)
  • per-case console lines no longer print $0.00 for unknown cost (6b3bca6)
  • polish Codex event parsing and Harbor task-dir guard messages (cf1bc9e)
  • track explicit Codex document reads (62152de)

Features

  • codex: estimate local run cost via LiteLLM pricing (f63157d)
  • harbor: add Codex runner and data mounts (03271c6)

1.37.4 (2026-08-13)

Bug Fixes

  • config: validate judge score scales at load (1e9c4bd), closes #182
  • config: warn the judge that actually runs unbounded (0e58c2b)
  • eval-analyze: stop the validator rejecting score_range (b67c53f)
  • judges: close the gaps an adversarial pass found in this series (feb1db7)
  • judges: honor score_range in LLM judge prompt, schema, and scoring (baffec3), closes #182
  • judges: keep the sign when the scale goes below zero (66f587b)
  • judges: read integer-ness off the declared bounds (326c62f)
  • judges: reject a non-finite judge value (add4b12)
  • judges: round an agent verdict on the scale its contract states (b9996a4)
  • judges: tell the MLflow fallback judge its scale (61394b2)
  • judges: validate the score range without rewriting the value (0e8e6c8)
  • report: keep the fractional part of a declared score_range (3e3e52b), closes #182
  • report: pass judge_ranges through the reward overview (e3aaceb)
  • report: size the histogram axis before truncating to bins (4b6743c)
  • reward: an unscored trial is not a perfect trial (37f3a29)
  • score: keep max_error_rate working on a persisted summary (cd8b40d)
  • score: name the real cause of an unavailable metric, and gate coverage (3585140)

1.37.3 (2026-08-12)

Bug Fixes

  • anova: make the missing-[anova]-extra error reachable and actionable (018636e)
  • anova: point the install hint at .eval-venv, keep the missing module name (ac704db)
  • anova: shell-quote the generated install command (0360453)
  • venv: activate the venv in eval-anova/check/compare scripts (be4ad8e)

1.37.2 (2026-08-11)

Bug Fixes

  • deps: report eval.yaml discovery failures instead of silently degrading (bf26404)
  • deps: resolve venv deps from every eval.yaml, not just the first (52d8a28)

1.37.1 (2026-08-11)

Bug Fixes

  • eval-analyze: drop phantom events var; cover validate_eval; tighten test (c6358f5), closes #179
  • resolve bootstrap venv via realpath; sync validate_eval Jinja vars (1d3ac2b)
  • tests: replace vacuous bootstrap test with sys.path check (fa2227f)

1.37.0 (2026-08-10)

Bug Fixes

  • address CodeRabbit review feedback (2aef4d3)
  • address CodeRabbit round-2 feedback (7ebb34d)
  • address re-review feedback from astefanutti (82b4027)
  • eval-check: address CodeRabbit findings on the previous commit (2baa972)
  • eval-check: reduce reference-checker false positives (a30a89e)

Features

  • add cross-component reference validation to eval-check (d2899a2)
  • add inline check syntax validation (ast.parse) (520bcc1)

1.36.0 (2026-08-07)

Bug Fixes

  • report: reject path traversal + symlinks in dataset case/input resolution (2d378ad)

Features

  • report: rename default title to "Agent Eval Report" + make it configurable (9a63db3)
  • report: scannable, diff-safe per-case Input/Output sections (2b8c74d)

1.35.1 (2026-08-06)

Bug Fixes

  • report: band reward-overview cells by score_range, drop (0,1) guess (bfa6f15)
  • report: color per-case cells by score_range, not aggregate min_mean (d014d1a)
  • report: use each judge's score_range for per-case histograms (79d2681)
  • report: validate score_range from raw config; cap histogram bins (b515fdc)

1.35.0 (2026-08-06)

Bug Fixes

  • runner: validate permission_mode by type, not truthiness (fa917f5)

Features

  • runner: add runner.permission_mode for Claude Code (3d8a479)

1.34.0 (2026-08-05)

Bug Fixes

  • mlflow: harden harness-snapshot load and MLflow fetch (f12a568)
  • mlflow: keep mlflow.runName immutable on tag merge (14b77df)

Features

  • mlflow: load harness-snapshot from disk and MLflow (3ee0025)

1.33.0 (2026-08-05)

Bug Fixes

  • harbor: emit multi_step_reward_strategy as a root-level key (45156dc)
  • harbor: share TOML serializer + UTF-8 writes for multi-step tasks (39d1e73), closes #148
  • multi-step: address CodeRabbit review on #172 (4a08221)
  • multi-step: surface non-zero step exit codes; strengthen symlink test (8aa2233)

Features

  • config: multi-step execution.steps + per-step hooks + step-scoped judges (8e94fa3)
  • execute: sequential multi-step execution + per-step hooks (a428f86)
  • harbor: generate multi-step [[steps]] task packages (schema 1.4) (e8bd271)
  • score: per-step judge scoping (JudgeConfig.step) (6d2ce3f)

1.32.0 (2026-08-05)

Bug Fixes

  • agent-judge: address CodeRabbit review (caa1728)
  • agent-judge: address CodeRabbit round 2 (bba2959)
  • agent-judge: chain the --no-llm-judges registry-failure error (B904) (c779f5d)
  • agent-judge: copy context for write-capable judges; fix blocking guidance (6c5567a)
  • skill: trim eval-run SKILL.md under the skillsaw context budget (ca41ff3)

Features

  • judges: add first-class agent judge type (8e80378)
  • report: group agent judges with LLM judges in the HTML report (8debd27)

1.31.0 (2026-08-05)

Features

  • pass GCP_SA_ACCESS_TOKEN to AnthropicVertex when available (9bad5fb)

1.30.2 (2026-08-05)

Bug Fixes

  • harbor: mark task.description truncation with an ellipsis (2e0714f)
  • harbor: serialize all task.toml string fields, escape U+007F (d6ee63c), closes #148
  • harbor: write generated task files as UTF-8 (c720c35)

1.30.1 (2026-07-30)

Bug Fixes

  • website: remove the sticky sidebar-header drop shadow (dffc0c5)

1.30.0 (2026-07-30)

Bug Fixes

  • eval-anova: address CodeRabbit findings on kept code (fb0a68c)
  • eval-anova: clamp report heat colours and bar widths to range (c76e203), closes #report
  • eval-anova: derive report labels from data, not a hardcoded roster (e3bf683), closes #11 #18
  • eval-anova: exclude cases missing from any condition and report them (2adfa90)
  • eval-anova: HTML-escape user-controlled values in reports (XSS) (6fab8a0)
  • eval-anova: ignore single-level factors when selecting the ANOVA (b177428)
  • eval-anova: report actual sample size in the non-significant callout (906c73e), closes #10
  • eval-anova: stop misreporting fractional composites as failures (09a4515)
  • eval-anova: treat non-finite/negative F as a degenerate design (fd9feeb)

Features

  • eval-anova: add eval/anova-example — real maas tasks on the generic path (9902237), closes #17
  • eval-anova: analyze stats over a directory of standard eval-run runs (e12f100)
  • eval-anova: DoE/ANOVA matrix testing skill with harbor-maas-v1 benchmark (53fb1ce), closes #104
  • eval-anova: prefer Greenhouse-Geisser corrected p in rm-ANOVA (32ddc20)
  • eval-anova: restore tests_pass gate + match original context A/B; add Harbor docs (33fcb42)
  • eval-anova: runnable matrix orchestrator that fans out over eval-run (b9ebca5), closes #5 #14 #15 #17
  • eval-compare: render eval-anova statistics section when present (efa3306)
  • eval-run: --input-override to inject run-level values into case inputs (#17) (4474605)
  • eval-run: add {config_dir} placeholder to the cli runner (3e0a9f1)

1.29.0 (2026-07-30)

Features

  • eval-run: mirror execute console to console.log + no-redirect launch (9c759a2)
  • workspace: auto-set EVAL_RUN_HEADER from --run-id (fb1576a)

1.28.0 (2026-07-30)

Bug Fixes

  • address CodeRabbit review feedback (c379d0a)
  • address CodeRabbit review feedback (55a8fed)
  • eval-compare: address review — correctness, robustness, report accuracy (74f5d1e)
  • remove Best Quality badge, strengthen Best Value criteria (a6d26f8)
  • remove hardcoded judge/report names from eval-compare (e8a61b3)
  • simplify Best Value badge — best model considering quality and cost (b2eaeaf)
  • use double quotes in f-strings for Python 3.11 compat (e066dab)

Features

  • add eval-compare skill for cross-model comparison reports (4c5568e)
  • eval-compare: light/dark theme matching the per-run reports (2e95cdd)
  • eval-compare: make report fully generic + add LLM analysis sections (9ea4e47)

1.27.0 (2026-07-30)

Bug Fixes

  • stream-capture: exclude placeholder models from per-model accounting (2378499)

Features

  • harbor: per-model usage/cost breakdown (70f79aa)

1.26.0 (2026-07-30)

Bug Fixes

  • hooks: register before_report in the standalone hooks CLI --phase choices (edd37b7)
  • report-hooks: address CodeRabbit #165 review (ad956b4)

Features

  • hooks: add before_report lifecycle phase + gate reward section for judge-only evals (cbbe9ad)

1.25.0 (2026-07-30)

Bug Fixes

  • eval-analyze: address review feedback on assess_skills (6dab5d7)
  • eval-analyze: address review feedback on scoped tools, EXISTS detection, excerpt fences, and symlinks (5793cd3)
  • eval-analyze: clean up stale --assess guard and Modes block placement (9270432)
  • eval-analyze: correct EXISTS detection and tool/excerpt handling in assess (205e4ef)
  • eval-analyze: guard glob results against symlink path traversal (46b5452)
  • eval-analyze: make out-of-project skill skip actionable (e51698f)
  • eval-analyze: treat skill_body_excerpt as untrusted data (220b9a4)
  • eval-analyze: trim SKILL.md below 4500 token warn limit (1b9bd3c)
  • eval-analyze: use correct eval/ directory layout in _has_existing_eval (73c99d8)

Features

  • eval-analyze: add --assess flag to score skills for eval-worthiness (6aa8163), closes #144

1.24.0 (2026-07-30)

Bug Fixes

  • collect: route per-case subagent traces in batch mode (1150370)
  • eval-run: warn on silent trace-routing gaps (addresses CodeRabbit #163) (efb10d6)

Features

  • events: preserve extended-thinking (chain-of-thought) in parsed traces (187f386)
  • score: CoT-inclusive {{ reasoning }} judge var + warn on undefined template vars (70d676d)

1.23.0 (2026-07-28)

Bug Fixes

  • mlflow: attribute thinking to its own turn's step, not the next one (6f88fd1)
  • mlflow: clamp trajectory timestamps and backfill reasoning_content (59a76cf)
  • mlflow: drop unused imports and use _resolve_skill for Harbor trace names (a9add3b)
  • mlflow: isolate tool-only thinking steps and tighten import skips (08f0855)
  • mlflow: resolve CodeRabbit findings on trajectory-enriched traces (be73c30)

Features

  • mlflow: enrich Harbor traces with tool content and step detail (336799a)
  • mlflow: richer Harbor traces with user turns and thinking (6141895)

1.22.1 (2026-07-27)

Bug Fixes

  • cover interactive idle-SIGTERM in session lifecycle warning (00e8bb9)
  • require polling after backgrounded execute.py to prevent false-green CI (b4cf178), closes #155

1.22.0 (2026-07-10)

Bug Fixes

  • agent,mlflow: forward extra_env in run_skill shim; correct settings env key (acb3317)
  • config: resolve skill/prompt target consistently across substrates (cfc845b)
  • docs,examples: migrate example configs to execution.skill; fix broken README example (20a894c)
  • eval-analyze: eliminate validate_eval false positives (56551c5)
  • eval-analyze: move $ARGUMENTS to end, wire reorganize.py, fix bullets (c3edc5d)
  • eval-dataset: fix setext-heading rule + clarify provenance-independent steps (ab17539)
  • eval-run: honor deny rules, isolate in-repo failures, strict args (8c1637f)
  • eval: complete prompt-mode implementation with eval_name() and Jinja2 support (399ba9b)
  • eval: include subagent reads in consulted_docs judge by default (c86f4f9)
  • permissions: correct + share the path-based deny compiler across substrates (3d6b60a)
  • scoring: annotations text, doc coverage, regression + conversation (908ce4c)
  • skills: clear skillsaw lint warnings (c21eb94)

Features

  • evaluation: add prompt-based evaluation mode for direct agent capability testing (936897a)
  • judges: add tool_trace template variable and enhance consulted_docs (9374524)

1.21.0 (2026-07-08)

Bug Fixes

  • address @astefanutti review findings on PR #146 (2ba452e)
  • address CodeRabbit review findings on PR #146 (36c038c)
  • address second review cycle (a5dc960)
  • report: classify builtin judges by kind, not by category (8206c22)
  • report: thread reward_cfg so the report matches the trained reward (0ae3691)
  • score: normalize raw .jsonl events into the flat schema consumers expect (4a33038)
  • score: sharpen evidence extraction across runners and shell args (c1f948c)

Features

  • config: add per-judge score_range, close report/schema drift (0a2c3a1)
  • report: add per-case reward overview table and Harbor scoring improvements (725842c)

1.20.0 (2026-06-30)

Features

  • mlflow: break out cache tokens in trace token usage (4b9ac26)

1.19.0 (2026-06-29)

Bug Fixes

  • bootstrap: address CodeRabbit review on OS trust store (617f62f)

Features

  • bootstrap: verify TLS against OS trust store when no CA bundle set (df134af)

1.18.0 (2026-06-29)

Bug Fixes

  • mlflow: address CodeRabbit review on harbor traces (1c9e0b7)
  • mlflow: source harbor trace cost/tokens from the transcript (be78da3)

Features

  • mlflow: build per-step MLflow traces for harbor runs (815c615)

1.17.1 (2026-06-26)

Bug Fixes

  • harbor: upload files in chunks to avoid E2BIG on large dirs (aa395ed)

1.17.0 (2026-06-26)

Bug Fixes

  • harbor: address review on exec retry and infra-error surfacing (54db1e3)

Features

  • harbor: don't score a missing verifier reward as 0 (1b41578)
  • harbor: retry transient k8s exec establishment failures (1b121aa), closes hi#parallelism
  • harbor: surface trials that failed before producing any reward (e750281)

1.16.1 (2026-06-26)

Bug Fixes

  • harbor: address review on the venv-activation guard (526ba2c)
  • harbor: prevent duplicate pod creation from mid-run venv re-exec (9d0fca6)

1.16.0 (2026-06-25)

Features

  • harbor: add reward.judge single-judge mode (f4c2ec7)

1.15.0 (2026-06-25)

Features

  • harbor: add grpo_reward judge and configurable (633a8b3)
  • harbor: bound reward-formula sandbox; fix reward doc accuracy (0a252a7)
  • harbor: harden reward config — normalize single-judge, validate formulas (fcf6a9a)

1.14.2 (2026-06-24)

Bug Fixes

  • use explicit None checks so empty strings are not skipped (5e518e7)
  • validate run_id and baseline as single path segments before path construction (CWE-22) (017b66a)

1.14.1 (2026-06-19)

Bug Fixes

  • harbor: restore project ConfigMap into workspace directory tree (051ae16)

1.14.0 (2026-06-18)

Bug Fixes

  • address CodeRabbit review findings on Harbor integration (c02b05f)
  • address second-round CodeRabbit findings (7a87c55)
  • containerfile: Harbor compatibility fixes (31064d7)
  • harbor/k8s: fix tar upload permissions for non-root pods (eaf4981)
  • harbor/k8s: match claude.ai/install.sh in skip-pkg-installs regex (37cc074)
  • harbor: only copy declared output paths in verifier test.sh (3ac0b2f)
  • harbor: propagate SIGTERM/SIGINT to harbor run subprocess (d83e76b)
  • harbor: use wall-clock duration from Harbor job timestamps (544d54e)
  • report: show judge type for multi-step and non-config judges (965b2e1)
  • update test for simplified instruction.md and fix zero-cost bug (5e3694f)

Features

  • add Harbor integration for containerized eval execution (96c9fdd)
  • eval-dataset: add --harbor flag for task package generation (63f2c59)
  • eval-setup: add --harbor flag for Harbor dependency installation (abbe3f3)
  • harbor/k8s: stream agent logs to pod stdout (029e1a4)
  • harbor: add --env flag and default skip-pkg-installs for K8s (f79e0c1)
  • harbor: add multi-step trial parsing (d0605f5)
  • harbor: extract turns, duration, version from agent transcripts (5910342)
  • harbor: generate per-case batch.yaml for batch-mode evals (75c38c8)
  • harbor: merge judge engine results into multi-step trials (2a25ad9)
  • harbor: simplify instruction.md template (3920939)
  • score: support ANTHROPIC_AUTH_TOKEN and ANTHROPIC_BASE_URL for LLM judges (5b4ce15)

1.13.2 (2026-06-09)

Bug Fixes

  • trace_builder: return server trace ID from _log_trace to fix FK failures (db004f0), closes #95
  • warn when _log_trace returns no backend ID (e877d96)

1.13.1 (2026-06-09)

Bug Fixes

  • don't sample deterministic judges even with CLI --samples (cbef112)
  • let --samples 1 override per-judge config (0346683)
  • preserve numbered list continuity across blank lines in rationales (17baf24)

1.13.0 (2026-06-09)

Bug Fixes

  • add extra_env parameter to ResponsesAPIRunner.run_skill (4e0360e)
  • address CodeRabbit review feedback (bd90b5f)
  • address PR #88 review feedback from astefanutti (04e9cec), closes #70
  • guarantee after_each hooks run and flow hook env to CLI runner (122bab7)
  • move before_each inside try/finally so after_each runs on setup failure (b114c02)
  • return synthetic failed result instead of None on case errors (5f8887f)

Features

  • add execution lifecycle hooks to eval pipeline (5bc1e05)
  • hooks: implement hook outputs for passing state from hooks to runners and judges (bb21862)

1.12.0 (2026-06-08)

Features

  • tabbed rationale view for non-stable sampled judges (e23f9c6)

1.11.0 (2026-06-08)

Bug Fixes

  • address CodeRabbit review findings on sampling stability (4704c3f)
  • force structured output for score/bool LLM judges (e1bc1eb)
  • force structured output for the pairwise judge (ba1d521)
  • pass 1-5 scale bounds to score histogram in the report (9cf507f)
  • stop truncating score/bool judge rationales to 200 chars (c922532)

Features

  • add --repeat to pairwise for verdict-stability measurement (796b614)
  • annotate judge scores with sampling stability in the report (1ae938b)
  • per-judge samples config in eval.yaml (f478aaf)
  • render pairwise stability section in the report (19f1704)
  • sample LLM judges N times for score stability (judges --repeat) (6b3677d)
  • visualise judge sampling stability in the report (c6b955f)

1.10.0 (2026-06-05)

Bug Fixes

  • pairwise comparison sends full artifacts and captures judge reasoning (288bd3b)

Features

  • show pairwise verdict and reasoning per case in the report (f28b689)

1.9.1 (2026-06-05)

Bug Fixes

  • skip workspace.files in batch mode to prevent silent overwrites (#112) (bb7e8bd), closes #111

1.9.0 (2026-06-05)

Features

  • add dataset.workspace.files for provisioning case source files (#70) (f687a3c)

1.8.0 (2026-06-04)

Features

  • change runner.env from list to dict with $VAR resolution (#108) (b1abc0b)

1.7.2 (2026-06-04)

Bug Fixes

  • eval-run: include cache tokens in input token metric (#107) (4ef40fd)

1.7.1 (2026-06-04)

Bug Fixes

  • eval-run: validate baseline run-id exists in preflight check (#106) (2a8ac0d)

1.7.0 (2026-06-04)

Features

  • replace runner.env_strip with runner.env for additive env forwarding (#105) (b052fe1), closes #103

1.6.0 (2026-06-03)

Features

  • add /eval-check for harness-level context and skills scanning / checking (#74) (de0ca7c)

1.5.0 (2026-06-02)

Features

1.4.1 (2026-06-02)

Bug Fixes

  • eval-run: reflow soft-wrapped paragraphs in HTML report (#92) (7a6fd41)

1.4.0 (2026-05-29)

Features

  • eval-dataset: builtin judges, conditional coverage, run-aware expansion (#84) (9c3995d)

1.3.0 (2026-05-29)

Features

  • eval-optimize: add judge type awareness, targeted re-runs, smarter analysis (#83) (ce89bc5)

1.2.3 (2026-05-29)

Bug Fixes

  • eval-review: update for v1.2 judge types, exact case matching (#82) (2f7c0ea)

1.2.2 (2026-05-29)

Bug Fixes

  • eval-analyze: replace {{ stdout }} with {{ conversation }} in template (#81) (50ab3cb)

1.2.1 (2026-05-29)

Bug Fixes

  • eval-run: rename --no-judge, --case-filter, exact case matching (#80) (06a3d0c)

1.2.0 (2026-05-29)

Features

  • eval-analyze: update skills for builtin judges and add list_builtins script (#79) (c2aff38)

1.1.0 (2026-05-29)

Features

  • add reusable judges library with builtin registry (#66) (1e51b41)

1.0.0 (2026-05-27)

Bug Fixes

  • address CodeRabbit findings in ensure_deps.py (f17cd72)
  • address CodeRabbit findings on PR #25 (4d4064f)
  • address CodeRabbit review feedback on CLI runner PR (2d90738)
  • address CodeRabbit review feedback on EvalHub PR (b86f754)
  • address CodeRabbit review feedback on release pipeline (7938e61)
  • address CodeRabbit review findings (0719088)
  • address CodeRabbit review findings on report.py (0992b80)
  • address eval-analyze skill review findings (183e606)
  • address remaining CodeRabbit review items on CLI runner (94cbcb1)
  • apply Rui's EvalHub provider registration corrections (628a85e)
  • bootstrap pyyaml before parsing eval.yaml in ensure_deps (aec5acf)
  • bump plugin.json and marketplace.json versions during release (0b73d63)
  • ci: bump Node.js to 22 for semantic-release (0b409ff)
  • default model examples to claude-opus-4-6 for skill/judge, sonnet for hook (5fac006)
  • detect and surface permission denials during eval-run execution (ba9b9a0), closes #34
  • disable persist-credentials in tests.yml checkout (b5ab1ec)
  • handle multiple MLflow runs per eval_run_id in from_traces.py (b6cf4ef)
  • improve execution mode detection in eval-analyze (42b62a2)
  • improve report badge rendering for regression and markdown tables (2a55174)
  • initialize git repos in eval workspaces for settings discovery (801a255)
  • merge eval.yaml permissions.allow into workspace settings.json (8810451)
  • remove beads data, gitignore .beads/, fix plugin order (9fbc1c7)
  • remove unused RunnerConfig.plugins field (7177d73)
  • resolve merge conflict with main in report.py (1da05f8)
  • revert dev marketplace.json to local source reference (f7077b3)
  • score: restore stdout loading and add batch-mode fallbacks (1634d6d)
  • tighten permission-denial matcher and e2e assertion (b4d3852)
  • update remaining 4-7 model IDs to 4-6 in eval.yaml (3c8b1a8)
  • use GitHub source reference in dev marketplace.json (c61eb04)
  • use jq for JSON version bumps instead of sed (e2ee502)
  • validate thresholds is a mapping before iterating (a9e8aa5)

Features

  • add [EXTERNAL] convention for external-state fields in dataset schema (b44c268), closes #34
  • add opaque CLI runner for arbitrary agent commands (6879853)
  • add parallel case execution for eval-run (b920489)
  • add semantic-release pipeline for automated versioning (c05128e)
  • attach batch.yaml/input.yaml as MLflow run artifacts for from-traces (a20961f)
  • auto-install Python dependencies via SessionStart hook (403b442)
  • collect: generate events.json from batch-mode stdout (6a9db37)
  • container image, rfe-assess benchmark, and provider config (624125f)
  • EvalHub provider for agent skill evaluation (798756b)
  • use structured permission_denials from CLI result event (8b939d8)

Changelog