mirror of
https://github.com/openai/codex-security.git
synced 2026-11-22 12:23:04 +08:00
* feat(cli): add independently reviewed dependency checks * Compare frozen Python advisory versions as releases * fix(cli): bind unnamed root manifests to observed services * test(cli): evaluate dependency investigations across public fixtures * test(sdk): bound the budget wait after the mock scan starts * test(cli): exercise MCP evidence and missing caller access * Bind graded citations to the approved use evidence --------- Co-authored-by: utsav-oai <348536218+utsav-oai@users.noreply.github.com>
2.4 KiB
2.4 KiB
Evaluations
Start with the deterministic triage checks or the harness tests linked below. These run locally without model calls. Model evaluations are opt-in and consume usage on the account you configure.
- Dependency reachability grades local CVE and package investigations against controlled application builds and measured stack evidence.
- Triage finding: Promptfoo input contracts, OSS calibration, and SastBench. This suite owns its private package and lockfile.
- Secret discovery: checks whether the production core audit finds synthetic credentials in source and keeps them in its final findings, with deterministic grading and harness checks.
- Standard time budget: paired comparison of production Standard scans with and without an advisory time allowance on synthetic vulnerabilities and safe controls, with source-evidence grading.
- Workflow routing: checks whether repository-installed production skills route natural-language requests to the appropriate workflow, stopping before workflow execution.
- The skill review fixtures cover pre-install decisions for individual skill files and bundles, including cross-file behavior.
CI runs the deterministic triage, secret-discovery, time-budget, and workflow-routing checks. The MCP suite covers Deep Scan iterations, recovery, and publication.
Build the evaluation tools
For the secret-discovery, Standard time-budget and workflow-routing suites, install the SDK and MCP dependencies, then run these commands from the repository root. The eval type-check includes the triage sources, so it needs the triage package’s dependencies too.
pnpm --dir evals/triage-finding install --frozen-lockfile
pnpm --dir sdk/typescript run build:evals
For Promptfoo triage and SastBench runs, use the triage setup, which also builds the native policy helper for your platform.