Files
codex-security/evals
ianw-oaiandmldangelo-oai e90a30ac25 fix(scan): reduce setup and tool output in Standard scans (#2046)
* fix(scan): reduce setup and tool output in Standard scans

* fix(workbench): keep policy paths optional for scan lookups

* fix(sdk): retain the preflight callback before scan dispatch

* fix(workbench): only advertise scoped inventory for Standard scans

* docs(workbench): explain optional scope report failures

* fix(workbench): preserve the Deep Scan registration response

* test(workbench): use the current scan context request in the Deep Scan fixture

---------

Co-authored-by: mldangelo-oai <269034524+mldangelo-oai@users.noreply.github.com>
2026-11-10 15:26:19 -08:00
..
2026-11-03 17:48:49 -08:00

Evaluations

Start with the deterministic triage checks or the harness tests linked below. These run locally without model calls. Model evaluations are opt-in and consume usage on the account you configure.

  • Triage finding: Promptfoo input contracts, OSS calibration, and SastBench. This suite owns its private package and lockfile.
  • Secret discovery: checks whether the production core audit finds synthetic credentials in source and keeps them in its final findings, with deterministic grading and harness checks.
  • Standard time budget: paired comparison of production Standard scans with and without an advisory time allowance on synthetic vulnerabilities and safe controls, with source-evidence grading.
  • Workflow routing: checks whether repository-installed production skills route natural-language requests to the appropriate workflow, stopping before workflow execution.
  • The skill review fixtures cover pre-install decisions for individual skill files and bundles, including cross-file behavior.

CI runs the deterministic triage, secret-discovery, time-budget, and workflow-routing checks. The MCP suite covers Deep Scan iterations, recovery, and publication.

Build the evaluation tools

For the secret-discovery, Standard time-budget and workflow-routing suites, install the SDK and MCP dependencies, then run these commands from the repository root. The eval type-check includes the triage sources, so it needs the triage package’s dependencies too.

pnpm --dir evals/triage-finding install --frozen-lockfile
pnpm --dir sdk/typescript run build:evals

For Promptfoo triage and SastBench runs, use the triage setup, which also builds the native policy helper for your platform.