mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(learn): make live traffic learning use the shared error classifier (#4110)
## Description Live traffic learning (`--learn`) decides whether a tool result failed, and why, with its own copy of the learn error patterns in `memory/traffic_learner.py`. OpenAI-format tool results carry no `is_error` flag, so on the OpenAI/Responses path the learner sniffs the output text. The copy has two bugs: 1. **"exit code 0" counted as an error.** Codex, Grok and opencode append `exit code 0` to the output of every successful shell command. The copy matched any `exit code` text. So every successful Codex shell call through the OpenAI path was recorded as a failure, which can turn into bogus "command X fails, use Y" recovery patterns. 2. **Timeouts and connection errors filed as `runtime_error`.** Every Python exception repr is `XxxError: ...`. The copy checked its generic `Error:` pattern before `TimeoutError`, so `TimeoutError: ...` became `runtime_error`, and it had no connection-error category at all. The shared `classify_error` in `learn/_shared.py` checks those first. **Exactly what changes:** - **Category** (which kind of error) now comes from the shared `headroom.learn._shared.classify_error`. Timeouts become `timeout`, connection errors `connection_error`, and an error that matches no pattern becomes `unknown` instead of `None`. - **Detection** (whether it is an error at all) keeps the old patterns: case-insensitive, over the first 2000 characters. Only the exit-code rule changes: - an exit code or exit status counts as an error only when it is nonzero, so `exit code 0` and `exited with exit code 0` are not errors; - `exit code 2`, `Exit code: 1` and `exited with status 1` still are; - `non-zero` still counts, and so does an `exited with ...` line that carries no code of its own. - The shared `is_error_content` is **not** used for detection. It is narrower (a case-sensitive substring list over 1000 characters) and would miss lowercase `no such file` and `permission denied`, `EACCES`, `EPERM`, `does not exist`, `Exception:`, `non-zero` and `exited with`. - I checked a 55-row truth table from review, over real tool-output strings. Detection matches the old detector on every row except the two `exit code 0` rows, which flip from error to not-an-error. - Category changes on that table: - `TimeoutError: ...` and `rpc error: deadline exceeded` → `timeout`; - `ConnectionError: ...` → `connection_error`; - a traceback mentioning `net/network_utils.py` → `connection_error`. This comes from the shared `network` pattern; see Additional Notes. Refs #4136 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/memory/traffic_learner.py`: - Replace the copied `_ERROR_PATTERNS`/`_classify_error` with these: - `_ERROR_SIGNALS`, the old detection patterns minus the exit-code one; - `_exit_status_is_error`, the nonzero-only exit-code rule. Signed codes such as `-9` count, and `exit code 0` never hides another failure in the same output: a second nonzero code, `exited with signal` or `killed by signal`. Status evidence belongs to its own line: an `exited with ...` line with no code of its own is a failure even when another line says `exit code 0`. A line that ends at `exited with` continues on the next line, so `Process exited with\nexit code 0` stays one zero status. A standalone `Killed` line (the shell's SIGKILL message) counts only when no exit status is reported, since a successful command can print that word, and it is recognized even below the 10-character short-content cut-off; - `_is_error`, the same length and 2000-character window as before. - `on_tool_result` records `classify_error(output).value`. - Both extractors keep calling `_is_error`: the Anthropic fallback sniff and the OpenAI sniff. - `headroom/learn/_shared.py`: `classify_error` maps signal kills (`killed by signal`, a standalone `Killed` or `Killed: 9` line) to `exit_code`, since a shell reports them as status 128+N. They were `unknown`. - `tests/test_memory/test_traffic_learner.py`: - `TestErrorClassification` drives `TrafficLearner` itself. - A parametrized detection table covers: the exit-code-0 rows; signed codes; `exit code 0` next to a nonzero code, a signal or `Killed`; nonzero, `non-zero` and `exited with status 1`; an exit code and a traceback after 1500 characters; the lowercase, `EACCES`, `EPERM`, `does not exist`, `Exception:` and `deadline exceeded` signals; and clean outputs. - A parametrized category table includes `timeout`, `connection_error` and the signal kills. - An end-to-end table runs the OpenAI extractor's result through `on_tool_result` and checks the recorded `is_error` and category. It covers a bare `Killed`, `worker exited with an error` plus a wrapper `exit code 0`, `Killed` plus `exit code 0`, and the controls: clean `exit code 0`, a signal kill plus a wrapper zero, and `-9`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # New tests against origin/main's traffic_learner.py: FAILED tests/test_memory/test_traffic_learner.py::TestErrorClassification::test_error_detection[Process exited with exit code 0\nall tests passed-False] FAILED tests/test_memory/test_traffic_learner.py::TestErrorClassification::test_error_detection[README.md\nsrc\nexit code 0-False] FAILED tests/test_memory/test_traffic_learner.py::TestErrorClassification::test_error_category_recorded[TimeoutError: deadline exceeded after 30s-timeout] FAILED tests/test_memory/test_traffic_learner.py::TestErrorClassification::test_error_category_recorded[ConnectionError: [Errno 111] Connection refused-connection_error] ================= 4 failed, 33 passed, 159 deselected in 0.50s ================= # With this change: $ pytest -q -p no:cacheprovider tests/test_memory/test_traffic_learner.py tests/test_openai_responses_traffic_learner.py tests/test_learn/test_section_lifecycle.py tests/test_learn/test_worktree_merge.py tests/test_learn/test_error_classification.py tests/test_openai_codex_routing.py tests/test_openai_codex_ws_timings.py tests/test_cache_mode_cold_start.py ============================= 290 passed in 39.55s ============================= # The new bare-Killed and wrapper-zero rows against the previous commit (50723621), in a scratch worktree: 4 FAILED. `Killed\nexit code 0` already passed there. # 55-row truth table (detection must equal the old _is_error except the two "exit code 0" rows): mismatches: 0 $ mypy headroom/memory/traffic_learner.py --ignore-missing-imports Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: macOS, Python 3.12.13, repo .venv, branch checkout ate063c64b(re-checked at98f9ea57) (basea9815bbb), temp HOME / HEADROOM_WORKSPACE_DIR / CODEX_HOME, offline. - Exact command / steps: - Ran the shared smoke harness, which starts a real proxy from this checkout against a local fake upstream and sends one Anthropic and one OpenAI request with a compressible tool result. - Ran a direct script that feeds seven OpenAI tool messages through `TrafficLearner.extract_tool_results_from_openai_messages` and then `on_tool_result`, as the proxy does, and prints what the learner recorded. - Ran a script over the reviewer's 55 tool-output strings that compares the new `_is_error` with the old one. - Observed result: - Both proxy requests return HTTP 200 with savings, with no new warnings. - `exit code 0` alone is a success, and so is a `Killed` line printed by a command that reports `exit code 0`. - A signed code (`-9`), a signal kill, or a worker's `exited with an error` is a failure, even next to a wrapper's `exit code 0`. So is a bare `Killed`. - Signal kills are categorized as `exit_code`, and a real error keeps its own category. - 0 detection mismatches outside the intended rows. - Not tested: A live Codex session through the proxy with `--learn`. The smoke run does not send shell tool outputs, so the direct script covers that path. ```text $ bash smoke.sh <worktree> branch: cleanup/memory-cache-traffic-error-classifier @e063c64bhealth: 200 POST /v1/messages -> HTTP200 smoke-ok POST /v1/chat/completions -> HTTP200 smoke-ok stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82 proxy.log WARNING/ERROR lines: 2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you 2 headroom.transforms.kompress_compressor - WARNING - Kompress: 'onnx/kompress-int8-wo.onnx' for chopratejas/kompress-v2-base is not cached and HEADROOM_OFFLINE forbids fetching it (HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that n 4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base 4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection. $ HOME=$TMP/home HEADROOM_WORKSPACE_DIR=$TMP/ws CODEX_HOME=$TMP/codex python run.py 'ok: 3 files\nexit code 0' is_error=False category=None 'command terminated\nexit code -9' is_error=True category=exit_code 'worker killed by signal SIGKILL\nwrapper exit code 0' is_error=True category=exit_code 'Killed' is_error=True category=exit_code 'Killed\nexit code 0' is_error=False category=None 'worker exited with an error\nwrapper exit code 0' is_error=True category=exit_code "ModuleNotFoundError: No module named 'foo'\nexit code 1" is_error=True category=module_not_found ``` ## Runtime Rollout Safety - Rollout-managed feature(s): None. This only affects `--learn` (live traffic learning), which is opt-in. - Minimum rollout channel: Stable; this is a classification bug fix on an opt-in path. - Stable/default behavior changed: - Default proxy behaviour is unchanged. Only runs with `--learn` on see a difference. - Output whose only failure signal is an exit code of 0 is no longer counted as a failure. - Error categories use the shared classifier. - An unmatched error is labelled `unknown`. - Kill switch / disable path: Run without `--learn`. - Unsafe override required: No. - Qualification impact: None. - Rollback path: Revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A. ## Additional Notes Part of maintainer-requested cleanup #4136 (dead code, duplication, debuggability), split into small PRs for review. - Documentation: the behaviour change is recorded in the cleanup docs ledger for the separate docs PR. - Detection and categorisation stay separate on purpose. Making the shared `is_error_content` as broad as this detector would also change the `headroom learn` scanners, so that belongs in its own PR. - Follow-up, pre-existing in `learn/_shared.py`: its `network` pattern files a traceback that mentions a path like `net/network_utils.py` as `connection_error`. - Patterns already persisted with a category from the old copy are left as they are. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
5cac7ab693
commit
e6eaa41179
@@ -69,7 +69,14 @@ _ERROR_PATTERNS: list[tuple[re.Pattern[str], ErrorCategory]] = [
|
||||
ErrorCategory.USER_REJECTED,
|
||||
),
|
||||
(re.compile(r"[Ss]ibling tool call errored", re.I), ErrorCategory.SIBLING_ERROR),
|
||||
(re.compile(r"exit code|non-zero|exited with", re.I), ErrorCategory.EXIT_CODE),
|
||||
# A signal kill is the shell's exit status 128+N, so it shares EXIT_CODE.
|
||||
(
|
||||
re.compile(
|
||||
r"exit code|non-zero|exited with|killed by signal|^\s*Killed(?::\s*\d+)?\s*$",
|
||||
re.I | re.M,
|
||||
),
|
||||
ErrorCategory.EXIT_CODE,
|
||||
),
|
||||
(
|
||||
re.compile(r"BUILD FAILED|compilation error|compile error", re.I),
|
||||
ErrorCategory.BUILD_FAILURE,
|
||||
|
||||
@@ -33,6 +33,8 @@ from enum import Enum
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Any, ClassVar
|
||||
|
||||
from headroom.learn._shared import classify_error
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from headroom.learn.models import ProjectInfo
|
||||
from headroom.memory.backends.local import LocalBackend
|
||||
@@ -207,40 +209,64 @@ def _strip_leading_cd(cmd: str) -> str:
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Error Classification (reused from learn/scanner.py patterns)
|
||||
# Error Detection
|
||||
# =============================================================================
|
||||
|
||||
_ERROR_PATTERNS: list[tuple[re.Pattern[str], str]] = [
|
||||
(
|
||||
re.compile(r"No such file or directory|ENOENT|FileNotFoundError|does not exist", re.I),
|
||||
"file_not_found",
|
||||
),
|
||||
(re.compile(r"ModuleNotFoundError|ImportError|No module named", re.I), "module_not_found"),
|
||||
(re.compile(r"command not found", re.I), "command_not_found"),
|
||||
(re.compile(r"Permission denied|EACCES|EPERM|auto-denied", re.I), "permission_denied"),
|
||||
(re.compile(r"file is too large|too many lines|exceeds.*limit", re.I), "file_too_large"),
|
||||
(re.compile(r"SyntaxError|IndentationError", re.I), "syntax_error"),
|
||||
(re.compile(r"Traceback \(most recent|Exception:|Error:", re.I), "runtime_error"),
|
||||
(re.compile(r"timed? ?out|TimeoutError|deadline exceeded", re.I), "timeout"),
|
||||
(re.compile(r"exit code|non-zero|exited with", re.I), "exit_code"),
|
||||
(re.compile(r"BUILD FAILED|compilation error|compile error", re.I), "build_failure"),
|
||||
]
|
||||
# OpenAI-format tool results carry no is_error flag, so failures are sniffed
|
||||
# from the output. Categories come from headroom.learn._shared.classify_error;
|
||||
# these patterns only decide *whether* the output is an error.
|
||||
_ERROR_SIGNALS: tuple[re.Pattern[str], ...] = (
|
||||
re.compile(r"No such file or directory|ENOENT|FileNotFoundError|does not exist", re.I),
|
||||
re.compile(r"ModuleNotFoundError|ImportError|No module named", re.I),
|
||||
re.compile(r"command not found", re.I),
|
||||
re.compile(r"Permission denied|EACCES|EPERM|auto-denied", re.I),
|
||||
re.compile(r"file is too large|too many lines|exceeds.*limit", re.I),
|
||||
re.compile(r"SyntaxError|IndentationError", re.I),
|
||||
re.compile(r"Traceback \(most recent|Exception:|Error:", re.I),
|
||||
re.compile(r"timed? ?out|TimeoutError|deadline exceeded", re.I),
|
||||
re.compile(r"BUILD FAILED|compilation error|compile error", re.I),
|
||||
)
|
||||
|
||||
# Agent harnesses (Codex, Grok, opencode, ...) append "exit code 0" to every
|
||||
# SUCCESSFUL shell command, so an exit code only signals an error when it is
|
||||
# nonzero (signed codes such as -9 included). Status evidence belongs to its own
|
||||
# line: a zero code never hides a separate explicit failure elsewhere in the
|
||||
# output (a nonzero code, "non-zero", a signal kill, or an "exited with ..." line
|
||||
# that carries no code of its own). A line that ends at "exited with" continues
|
||||
# on the next line ("Process exited with\nexit code 0"), so it is not a failure.
|
||||
# A standalone "Killed" line is the shell's SIGKILL message, but a successful
|
||||
# command can also print that word, so it counts only when no exit status is
|
||||
# reported at all. It is recognized even below the short-content cut-off.
|
||||
_EXIT_STATUS_RE = re.compile(
|
||||
r"\bexit(?:ed)?(?:\s+with)?(?:\s+exit)?\s+(?:code|status)\s*:?\s*([+-]?\d+)", re.I
|
||||
)
|
||||
_EXIT_FAILURE_RE = re.compile(r"non-zero|nonzero|exited with signal|killed by signal", re.I)
|
||||
_KILLED_LINE_RE = re.compile(r"^\s*Killed(?::\s*\d+)?\s*$", re.M)
|
||||
_EXITED_WITH_RE = re.compile(r"exited with(?=\s*\S)", re.I)
|
||||
|
||||
|
||||
def _classify_error(content: str) -> str | None:
|
||||
"""Classify error content. Returns category or None if not an error."""
|
||||
snippet = content[:2000]
|
||||
for pattern, category in _ERROR_PATTERNS:
|
||||
if pattern.search(snippet):
|
||||
return category
|
||||
return None
|
||||
def _exit_status_is_error(snippet: str) -> bool:
|
||||
codes = _EXIT_STATUS_RE.findall(snippet)
|
||||
if any(int(code) != 0 for code in codes) or _EXIT_FAILURE_RE.search(snippet):
|
||||
return True
|
||||
if any(
|
||||
_EXITED_WITH_RE.search(line) and not _EXIT_STATUS_RE.search(line)
|
||||
for line in snippet.splitlines()
|
||||
):
|
||||
return True
|
||||
return not codes and bool(_KILLED_LINE_RE.search(snippet))
|
||||
|
||||
|
||||
def _is_error(content: str) -> bool:
|
||||
"""Quick check if tool output looks like an error."""
|
||||
if not content or len(content) < 10:
|
||||
if not content:
|
||||
return False
|
||||
return _classify_error(content) is not None
|
||||
if len(content) < 10:
|
||||
return bool(_KILLED_LINE_RE.fullmatch(content))
|
||||
snippet = content[:2000]
|
||||
if any(pattern.search(snippet) for pattern in _ERROR_SIGNALS):
|
||||
return True
|
||||
return _exit_status_is_error(snippet)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
@@ -769,7 +795,7 @@ class TrafficLearner:
|
||||
"input": tool_input,
|
||||
"output": tool_output[:2000], # Cap for memory
|
||||
"is_error": is_error,
|
||||
"error_category": _classify_error(tool_output) if is_error else None,
|
||||
"error_category": classify_error(tool_output).value if is_error else None,
|
||||
"timestamp": time.time(),
|
||||
"agent_type": agent_type,
|
||||
}
|
||||
|
||||
@@ -18,10 +18,8 @@ from headroom.memory.traffic_learner import (
|
||||
TrafficLearner,
|
||||
_bash_binaries_match,
|
||||
_bash_first_binary,
|
||||
_classify_error,
|
||||
_commands_related_as_retry,
|
||||
_drop_contradictions,
|
||||
_is_error,
|
||||
_levenshtein,
|
||||
_load_persisted_patterns_from_sqlite,
|
||||
_normalize_bash_for_hash,
|
||||
@@ -40,28 +38,145 @@ UTC = timezone.utc
|
||||
|
||||
|
||||
class TestErrorClassification:
|
||||
def test_file_not_found(self):
|
||||
assert _classify_error("No such file or directory: foo.py") == "file_not_found"
|
||||
assert _classify_error("FileNotFoundError: [Errno 2]") == "file_not_found"
|
||||
"""The learner classifies tool output with the shared learn classifier."""
|
||||
|
||||
def test_command_not_found(self):
|
||||
assert _classify_error("zsh: command not found: ruff") == "command_not_found"
|
||||
@pytest.fixture
|
||||
def learner(self):
|
||||
return TrafficLearner(backend=None, user_id="test-user", min_evidence=1)
|
||||
|
||||
def test_module_not_found(self):
|
||||
assert _classify_error("ModuleNotFoundError: No module named 'foo'") == "module_not_found"
|
||||
@staticmethod
|
||||
def _openai_result(learner: TrafficLearner, output: str) -> dict:
|
||||
messages = [
|
||||
{
|
||||
"role": "assistant",
|
||||
"tool_calls": [
|
||||
{"id": "c1", "function": {"name": "shell", "arguments": '{"command": "ls"}'}}
|
||||
],
|
||||
},
|
||||
{"role": "tool", "tool_call_id": "c1", "content": output},
|
||||
]
|
||||
return learner.extract_tool_results_from_openai_messages(messages)[0]
|
||||
|
||||
def test_permission_denied(self):
|
||||
assert _classify_error("Permission denied: /etc/shadow") == "permission_denied"
|
||||
@pytest.mark.parametrize(
|
||||
("output", "is_error"),
|
||||
[
|
||||
# Successful shell commands from Codex/Grok/opencode end in "exit code 0".
|
||||
("Process exited with exit code 0\nall tests passed", False),
|
||||
("README.md\nsrc\nexit code 0", False),
|
||||
# Nonzero exit codes and other exit-failure wording.
|
||||
("Exit code: 1\nWall time: 0.2s\nOutput:\nfail", True),
|
||||
("exit code 2", True),
|
||||
("Command failed with non-zero status 1", True),
|
||||
("process exited with status 1", True),
|
||||
("x" * 1500 + "\nexit code 3", True),
|
||||
("command terminated\nexit code -9", True),
|
||||
("Process exited with exit code -1", True),
|
||||
# A zero wrapper status must not hide a separate failure.
|
||||
("worker exited with signal 9\nexit code 0", True),
|
||||
("child exit code 2\nwrapper exit code 0", True),
|
||||
("python train.py\nKilled\n", True),
|
||||
# A successful command may print the word; exit code 0 then wins.
|
||||
("grep result:\nKilled\nexit code 0", False),
|
||||
("task killed by signal SIGKILL\nexit code 0", True),
|
||||
# A bare Killed is an error even below the short-content cut-off,
|
||||
# but a reported zero status wins over a printed "Killed".
|
||||
("Killed", True),
|
||||
("Killed\nexit code 0", False),
|
||||
# The wrapper's zero status belongs to its own line.
|
||||
("worker exited with an error\nwrapper exit code 0", True),
|
||||
# One status phrase wrapped across lines is still one zero status.
|
||||
("Process exited with\nexit code 0", False),
|
||||
("Killed: 9", True),
|
||||
# Case-insensitive signals the shared is_error_content would miss.
|
||||
("zsh: no such file or directory: ./run.sh", True),
|
||||
("fatal: path 'x' does not exist in 'HEAD'", True),
|
||||
("open /etc/shadow: permission denied", True),
|
||||
("EACCES: permission denied, open '/x'", True),
|
||||
("Operation not permitted (EPERM) on /x", True),
|
||||
("File content exceeds maximum allowed tokens limit (25000)", True),
|
||||
("file is too large to read", True),
|
||||
("Exception: something broke badly", True),
|
||||
("context deadline exceeded while dialing", True),
|
||||
("x" * 1500 + "\nTraceback (most recent call last):\nValueError: late", True),
|
||||
# Signals the old detector already caught.
|
||||
("No such file or directory: foo.py", True),
|
||||
("ModuleNotFoundError: No module named 'foo'", True),
|
||||
("zsh: command not found: ruff", True),
|
||||
("Tool call auto-denied by policy", True),
|
||||
("IndentationError: unexpected indent here", True),
|
||||
("TimeoutError: deadline exceeded after 30s", True),
|
||||
("Command timed out after 120s", True),
|
||||
("BUILD FAILED in 3s", True),
|
||||
# Not errors.
|
||||
("All tests passed, everything is fine", False),
|
||||
("2 failed, 10 passed in 1.2s", False),
|
||||
("Checked 4 files, no errors found in output", False),
|
||||
("short", False),
|
||||
("", False),
|
||||
],
|
||||
)
|
||||
def test_error_detection(self, learner: TrafficLearner, output: str, is_error: bool):
|
||||
assert self._openai_result(learner, output)["is_error"] is is_error
|
||||
|
||||
def test_not_an_error(self):
|
||||
assert _classify_error("Everything is fine, tests passed!") is None
|
||||
assert _classify_error("") is None
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
("output", "category"),
|
||||
[
|
||||
("No such file or directory: foo.py", "file_not_found"),
|
||||
("zsh: command not found: ruff", "command_not_found"),
|
||||
("ModuleNotFoundError: No module named 'foo'", "module_not_found"),
|
||||
("Permission denied: /etc/shadow", "permission_denied"),
|
||||
("TimeoutError: deadline exceeded after 30s", "timeout"),
|
||||
("ConnectionError: [Errno 111] Connection refused", "connection_error"),
|
||||
("task killed by signal SIGKILL", "exit_code"),
|
||||
("python train.py\nKilled", "exit_code"),
|
||||
("Killed: 9", "exit_code"),
|
||||
],
|
||||
)
|
||||
async def test_error_category_recorded(
|
||||
self, learner: TrafficLearner, output: str, category: str
|
||||
):
|
||||
await learner.on_tool_result(
|
||||
tool_name="Bash", tool_input={"command": "x"}, tool_output=output, is_error=True
|
||||
)
|
||||
assert learner._tool_history[-1]["error_category"] == category
|
||||
|
||||
def test_is_error_helper(self):
|
||||
assert _is_error("No such file or directory")
|
||||
assert not _is_error("All tests passed")
|
||||
assert not _is_error("")
|
||||
assert not _is_error("short")
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
("output", "is_error", "category"),
|
||||
[
|
||||
("Killed", True, "exit_code"),
|
||||
("worker exited with an error\nwrapper exit code 0", True, "exit_code"),
|
||||
("Killed\nexit code 0", False, None),
|
||||
("README.md\nsrc\nexit code 0", False, None),
|
||||
("worker killed by signal SIGKILL\nwrapper exit code 0", True, "exit_code"),
|
||||
("command terminated\nexit code -9", True, "exit_code"),
|
||||
],
|
||||
)
|
||||
async def test_openai_result_recorded_through_on_tool_result(
|
||||
self, learner: TrafficLearner, output: str, is_error: bool, category: str | None
|
||||
):
|
||||
"""End to end: the OpenAI extractor's verdict is what on_tool_result records."""
|
||||
result = self._openai_result(learner, output)
|
||||
await learner.on_tool_result(
|
||||
tool_name=result["tool_name"],
|
||||
tool_input=result["input"],
|
||||
tool_output=result["output"],
|
||||
is_error=result["is_error"],
|
||||
)
|
||||
recorded = learner._tool_history[-1]
|
||||
assert recorded["is_error"] is is_error
|
||||
assert recorded["error_category"] == category
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_success_has_no_error_category(self, learner: TrafficLearner):
|
||||
await learner.on_tool_result(
|
||||
tool_name="Bash",
|
||||
tool_input={"command": "x"},
|
||||
tool_output="Everything is fine, tests passed!",
|
||||
is_error=False,
|
||||
)
|
||||
assert learner._tool_history[-1]["error_category"] is None
|
||||
|
||||
|
||||
# =============================================================================
|
||||
|
||||
Reference in New Issue
Block a user