mirror of
https://github.com/openai/codex-security.git
synced 2026-11-22 09:49:39 +08:00
* fix(cli): assess captured stacks against public advisory conditions Resolve advisory records once per captured-stack investigation and extract necessary conditions with supporting quotations. Share the combined conditions with the investigator and proof validator, and freeze the evidence for replay. * refactor(cli): share advisory trigger guidance across investigations * fix(package): include the stack advisory module in the distribution manifest * fix(cli): review captured-stack investigations before publication Verify cited checks for unresolved questions during assessment and finalization. Add independent stack review and share the repository workflow’s one-shot correction loop. * fix(cli): expose each stack report in the command output schema * refactor(cli): use one report format for captured stacks * fix(evals): grade stack reports against their review and advisory evidence
Evaluations
Start with the deterministic triage checks or the harness tests linked below. These run locally without model calls. Model evaluations are opt-in and consume usage on the account you configure.
- Dependency reachability grades local CVE and package investigations against controlled application builds and measured stack evidence.
- Triage finding: Promptfoo input contracts, OSS calibration, and SastBench. This suite owns its private package and lockfile.
- Secret discovery: checks whether the production core audit finds synthetic credentials in source and keeps them in its final findings, with deterministic grading and harness checks.
- Standard time budget: paired comparison of production Standard scans with and without an advisory time allowance on synthetic vulnerabilities and safe controls, with source-evidence grading.
- Workflow routing: checks whether repository-installed production skills route natural-language requests to the appropriate workflow, stopping before workflow execution.
- The skill review fixtures cover pre-install decisions for individual skill files and bundles, including cross-file behavior.
CI runs the deterministic triage, secret-discovery, time-budget, and workflow-routing checks. The MCP suite covers Deep Scan iterations, recovery, and publication.
Build the evaluation tools
For the secret-discovery, Standard time-budget and workflow-routing suites, install the SDK and MCP dependencies, then run these commands from the repository root. The eval type-check includes the triage sources, so it needs the triage package’s dependencies too.
pnpm --dir evals/triage-finding install --frozen-lockfile
pnpm --dir sdk/typescript run build:evals
For Promptfoo triage and SastBench runs, use the triage setup, which also builds the native policy helper for your platform.