Files
n-papaioannouandPeter 92bb5d0a95 feat: audit any agent over chat (#363)
* chore: streamline the engine

Removes the record/replay bridge, the unused embedding classifier, the example scripts and the deprecated --profile alias (use --mode).

* feat: judging within your budget

* chore: align run stops and labels with main

* feat: audit any agent over chat

* chore: CI fix

---------

Co-authored-by: Peter <as.peter@ime.life>
2026-10-23 00:09:19 +03:00

45 lines
2.3 KiB
Markdown

# Python API
`ifixai.api` runs the same engine as the [CLI](cli.md). Run functions are async; helpers are sync.
```python
import asyncio
from ifixai.api import run_inspections
result = asyncio.run(run_inspections(
provider="openai",
api_key="sk-...",
model="gpt-4o",
fixture="default",
system_name="my-agent",
))
print(result.overall_score, result.grade)
```
## Functions
| Function | Purpose |
|---|---|
| `run_inspections(...)` | Run all registered inspections (async) → `TestRunResult`. Unlike the CLI, which runs 10 by default, this always runs every one (the CLI's `--full-run`). |
| `run_strategic(...)` | Run the top 8 strategic tests (async) |
| `run_selected(test_ids, ...)` | Run a chosen list of test IDs (async) |
| `run_single(test_id, ...)` | Run a single test by ID (async); raises `ValueError` with why when it is not applicable |
| `compare_scorecards(baseline, enhanced)` | Vendor-neutral comparison report (sync) |
| `list_tests()` | Return all `InspectionSpec` definitions (sync) |
| `list_fixtures()` | Return built-in fixture names (sync) |
## Common parameters
| Parameter | Default | Notes |
|---|---|---|
| `provider` | required | Provider name (`"openai"`, `"anthropic"`, …) or a `ChatProvider` instance. |
| `api_key` | `""` | SUT key. |
| `fixture` | `"default"` | Registered name or a `Fixture` object. |
| `model`, `endpoint`, `system_prompt` | `None` | SUT overrides. |
| `system_name`, `system_version` | `""`, `"1.0"` | Report labels. |
| `timeout`, `max_retries` | `30`, `3` | Per-request controls. |
| `judge_config`, `pipeline_config` | `None` | `JudgeConfig` / `EvaluationPipelineConfig` for judges, ensemble, budgets. |
| `sut_temperature`, `sut_seed`, `run_nonce`, `holdout_ids` | `0.0`, `None`, `None`, `None` | Determinism / replay controls (mirror CLI seeds). |
To test a custom agent, implement `ChatProvider` ([../ifixai/providers/base.py](../ifixai/providers/base.py)) and pass the instance as `provider`. Only `send_message` is needed: the inspections ask your agent through it. Hooks you add (`authorize_tool`, `get_confidence`, ...) show as notes next to the probe results and never decide the checks the probes decide. `TestRunResult.not_applicable` lists the checks set aside before the run, each with why. Walkthrough: [testing-your-agent.md](testing-your-agent.md).