Palash DebnathandClaude Fable 5 1195b4e0dd test(evals): LLM-judge eval tier — non-gating semantic suites (Wave 0.3) (#355) ...

Ports Patter's eval harness (MIT, attribution headers) into tests/evals/
with the judge transport swapped to services/llm_backend.py — the judge
runs against whatever local Ollama/LM Studio/OpenAI-compat endpoint the
user configured, keeping local-first. Both Patter hardening details kept
verbatim: verdict recomputed locally from the score (hallucinated
'passed: true' at score 0.2 fails), and tolerant JSON parsing (fences
stripped, invalid JSON -> fail-with-reasoning). Per-case containment:
agent exceptions keep the partial transcript and still judge it; a judge
failure records score 0 instead of aborting the suite.

HARD RULE preserved: LLM judges never gate CI. The scheduled workflow
(weekly + dispatch) is continue-on-error with the JSON report as artifact;
run_evals.py exits 0 always and skips cleanly when the active LLM backend
is 'off'. Deterministic probe judges remain the only gates; the harness
unit tests (10, no LLM needed) do run in gating CI.

First suite: dub translation naturalness v1 (4 cases) driving the real
cinematic_refine_sync reflect+adapt chain. The telephony-specific
session/assertions layers were deliberately not ported. The
dictation-refinement suite lands with Wave 1.1/2.1.

Spec: docs/competitive-analysis.md Spec 9b / parity program Wave 0.3.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

2026-06-11 22:00:30 +05:30

..

suites

test(evals): LLM-judge eval tier — non-gating semantic suites (Wave 0.3) (#355)

2026-06-11 22:00:30 +05:30

__init__.py

test(evals): LLM-judge eval tier — non-gating semantic suites (Wave 0.3) (#355)

2026-06-11 22:00:30 +05:30

case.py

test(evals): LLM-judge eval tier — non-gating semantic suites (Wave 0.3) (#355)

2026-06-11 22:00:30 +05:30

judge.py

test(evals): LLM-judge eval tier — non-gating semantic suites (Wave 0.3) (#355)

2026-06-11 22:00:30 +05:30

run_evals.py

test(evals): LLM-judge eval tier — non-gating semantic suites (Wave 0.3) (#355)

2026-06-11 22:00:30 +05:30

runner.py

test(evals): LLM-judge eval tier — non-gating semantic suites (Wave 0.3) (#355)

2026-06-11 22:00:30 +05:30

test_evals_harness.py

test(evals): LLM-judge eval tier — non-gating semantic suites (Wave 0.3) (#355)

2026-06-11 22:00:30 +05:30