Files
codex-security/evals

Evaluations

Model-based evaluations live here so they can evolve with the plugin without being embedded in its source tree or shipped npm runtime.

  • Triage finding: Promptfoo input contracts, OSS calibration, and SastBench. This suite owns its private package and lockfile.
  • Deep reducer: optional model evaluation of real IPC size-limit recovery, pagination, and finding retention. It reuses the MCP test helpers and dependencies.
  • Secret discovery: checks whether the production core audit finds synthetic credentials in source and keeps them in its final findings, with deterministic grading and harness checks.
  • Workflow routing: checks whether repository-installed production skills route natural-language requests to the appropriate workflow, stopping before workflow execution.

Model runs are opt-in. CI runs the deterministic triage, secret-discovery, and workflow-routing helper checks and the real-IPC reducer regression through the normal MCP test suite.