Files
utsav-oaiandutsav-oai e224a3314b fix(cli): hillclimb dependency investigation eval & speed (#2737)
* feat(cli): add independently reviewed dependency checks

* Compare frozen Python advisory versions as releases

* fix(cli): bind unnamed root manifests to observed services

* test(cli): evaluate dependency investigations across public fixtures

* test(sdk): bound the budget wait after the mock scan starts

* test(cli): exercise MCP evidence and missing caller access

* Bind graded citations to the approved use evidence

* test: restore dependency investigation quality controls

* fix(cli): restore dependency investigation and report compatibility

* test: grade malformed dependency baseline answers

* fix(cli): scope headless tool access to the investigator

* test(cli): lock down reviewer and external evidence tool access

* fix(cli): keep linked evidence tied to its originating call

* Reconcile application mappings and expose original evidence hashes

* test: verify selected catalog rows against original replies

* fix(cli): recover incomplete reviews and preserve selected evidence rows

* fix(cli): preserve wire evidence and verify finding citations

* Finalize from approved submissions and explain remaining uses

* fix(cli): restore coordinator recovery, independent review and report contracts

* Review withdrawals under the original dependency assessment ID

* test(cli): compare trace progress with completed search scopes

* Recheck dependency coverage after concurrent investigation changes

* Preserve exploratory scope explanations during dependency review

* feat(cli): track shared source work in dependency investigations

* Include shared source work in final coverage checks

* Require deciding evidence on the caller it establishes

* Preserve generic MCP search outcomes and scopes in dependency reports

* Share pinned source notes between dependency investigators

* Ship the source worklist and search audit modules

* refactor(cli): name the dependency coordinator for its ability to resume

* refactor(cli): simplify the tools for dependency investigations

* fix(cli): keep investigation tool arguments visible to Codex and reuse the evidence store

* fix(cli): send missing verdict citations back to the reviewer

* fix(cli): verify caller bindings and input flow across investigation steps

* fix(cli): require direct citations for asserted runtime facts on every use

* test(cli): verify failed catalog lookups without a timed source scan

---------

Co-authored-by: utsav-oai <348536218+utsav-oai@users.noreply.github.com>
2026-11-19 11:10:07 -08:00
..

Evaluations

Start with the deterministic triage checks or the harness tests linked below. These run locally without model calls. Model evaluations are opt-in and consume usage on the account you configure.

  • Dependency reachability grades local CVE and package investigations against controlled application builds and measured stack evidence.
  • Triage finding: Promptfoo input contracts, OSS calibration, and SastBench. This suite owns its private package and lockfile.
  • Secret discovery: checks whether the production core audit finds synthetic credentials in source and keeps them in its final findings, with deterministic grading and harness checks.
  • Standard time budget: paired comparison of production Standard scans with and without an advisory time allowance on synthetic vulnerabilities and safe controls, with source-evidence grading.
  • Workflow routing: checks whether repository-installed production skills route natural-language requests to the appropriate workflow, stopping before workflow execution.
  • The skill review fixtures cover pre-install decisions for individual skill files and bundles, including cross-file behavior.

CI runs the deterministic triage, secret-discovery, time-budget, and workflow-routing checks. The MCP suite covers Deep Scan iterations, recovery, and publication.

Build the evaluation tools

For the secret-discovery, Standard time-budget and workflow-routing suites, install the SDK and MCP dependencies, then run these commands from the repository root. The eval type-check includes the triage sources, so it needs the triage package’s dependencies too.

pnpm --dir evals/triage-finding install --frozen-lockfile
pnpm --dir sdk/typescript run build:evals

For Promptfoo triage and SastBench runs, use the triage setup, which also builds the native policy helper for your platform.