mirror of
https://github.com/ifixai-ai/iFixAi.git
synced 2026-11-21 10:11:54 +08:00
* chore: streamline the engine Removes the record/replay bridge, the unused embedding classifier, the example scripts and the deprecated --profile alias (use --mode). * feat: judging within your budget * chore: align run stops and labels with main * feat: audit any agent over chat * chore: CI fix --------- Co-authored-by: Peter <as.peter@ime.life>
45 lines
2.3 KiB
Markdown
45 lines
2.3 KiB
Markdown
# Python API
|
|
|
|
`ifixai.api` runs the same engine as the [CLI](cli.md). Run functions are async; helpers are sync.
|
|
|
|
```python
|
|
import asyncio
|
|
from ifixai.api import run_inspections
|
|
|
|
result = asyncio.run(run_inspections(
|
|
provider="openai",
|
|
api_key="sk-...",
|
|
model="gpt-4o",
|
|
fixture="default",
|
|
system_name="my-agent",
|
|
))
|
|
print(result.overall_score, result.grade)
|
|
```
|
|
|
|
## Functions
|
|
|
|
| Function | Purpose |
|
|
|---|---|
|
|
| `run_inspections(...)` | Run all registered inspections (async) → `TestRunResult`. Unlike the CLI, which runs 10 by default, this always runs every one (the CLI's `--full-run`). |
|
|
| `run_strategic(...)` | Run the top 8 strategic tests (async) |
|
|
| `run_selected(test_ids, ...)` | Run a chosen list of test IDs (async) |
|
|
| `run_single(test_id, ...)` | Run a single test by ID (async); raises `ValueError` with why when it is not applicable |
|
|
| `compare_scorecards(baseline, enhanced)` | Vendor-neutral comparison report (sync) |
|
|
| `list_tests()` | Return all `InspectionSpec` definitions (sync) |
|
|
| `list_fixtures()` | Return built-in fixture names (sync) |
|
|
|
|
## Common parameters
|
|
|
|
| Parameter | Default | Notes |
|
|
|---|---|---|
|
|
| `provider` | required | Provider name (`"openai"`, `"anthropic"`, …) or a `ChatProvider` instance. |
|
|
| `api_key` | `""` | SUT key. |
|
|
| `fixture` | `"default"` | Registered name or a `Fixture` object. |
|
|
| `model`, `endpoint`, `system_prompt` | `None` | SUT overrides. |
|
|
| `system_name`, `system_version` | `""`, `"1.0"` | Report labels. |
|
|
| `timeout`, `max_retries` | `30`, `3` | Per-request controls. |
|
|
| `judge_config`, `pipeline_config` | `None` | `JudgeConfig` / `EvaluationPipelineConfig` for judges, ensemble, budgets. |
|
|
| `sut_temperature`, `sut_seed`, `run_nonce`, `holdout_ids` | `0.0`, `None`, `None`, `None` | Determinism / replay controls (mirror CLI seeds). |
|
|
|
|
To test a custom agent, implement `ChatProvider` ([../ifixai/providers/base.py](../ifixai/providers/base.py)) and pass the instance as `provider`. Only `send_message` is needed: the inspections ask your agent through it. Hooks you add (`authorize_tool`, `get_confidence`, ...) show as notes next to the probe results and never decide the checks the probes decide. `TestRunResult.not_applicable` lists the checks set aside before the run, each with why. Walkthrough: [testing-your-agent.md](testing-your-agent.md).
|