Files
n-papaioannouandPeter 92bb5d0a95 feat: audit any agent over chat (#363)
* chore: streamline the engine

Removes the record/replay bridge, the unused embedding classifier, the example scripts and the deprecated --profile alias (use --mode).

* feat: judging within your budget

* chore: align run stops and labels with main

* feat: audit any agent over chat

* chore: CI fix

---------

Co-authored-by: Peter <as.peter@ime.life>
2026-10-23 00:09:19 +03:00

2.3 KiB

Python API

ifixai.api runs the same engine as the CLI. Run functions are async; helpers are sync.

import asyncio
from ifixai.api import run_inspections

result = asyncio.run(run_inspections(
    provider="openai",
    api_key="sk-...",
    model="gpt-4o",
    fixture="default",
    system_name="my-agent",
))
print(result.overall_score, result.grade)

Functions

Function Purpose
run_inspections(...) Run all registered inspections (async) → TestRunResult. Unlike the CLI, which runs 10 by default, this always runs every one (the CLI's --full-run).
run_strategic(...) Run the top 8 strategic tests (async)
run_selected(test_ids, ...) Run a chosen list of test IDs (async)
run_single(test_id, ...) Run a single test by ID (async); raises ValueError with why when it is not applicable
compare_scorecards(baseline, enhanced) Vendor-neutral comparison report (sync)
list_tests() Return all InspectionSpec definitions (sync)
list_fixtures() Return built-in fixture names (sync)

Common parameters

Parameter Default Notes
provider required Provider name ("openai", "anthropic", …) or a ChatProvider instance.
api_key "" SUT key.
fixture "default" Registered name or a Fixture object.
model, endpoint, system_prompt None SUT overrides.
system_name, system_version "", "1.0" Report labels.
timeout, max_retries 30, 3 Per-request controls.
judge_config, pipeline_config None JudgeConfig / EvaluationPipelineConfig for judges, ensemble, budgets.
sut_temperature, sut_seed, run_nonce, holdout_ids 0.0, None, None, None Determinism / replay controls (mirror CLI seeds).

To test a custom agent, implement ChatProvider (../ifixai/providers/base.py) and pass the instance as provider. Only send_message is needed: the inspections ask your agent through it. Hooks you add (authorize_tool, get_confidence, ...) show as notes next to the probe results and never decide the checks the probes decide. TestRunResult.not_applicable lists the checks set aside before the run, each with why. Walkthrough: testing-your-agent.md.