mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:06:14 +08:00
## Description
The offline session-retention benchmark reports a numeric fact as
retained when its old value is merely a prefix of a changed value. For
example, original `port: 80` is reported retained after compression
changes it to `port: 8080`, `port: 80.5`, or `port: -80`. This
overstates measured fidelity without requiring any model behavior.
Run numeric probes through a bounded numeric-value match before the
generic substring shortcuts. Existing table-format conversion and
sentence punctuation remain supported. A missing numeric fact with a CCR
marker is classified as recoverable instead of retained.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- Prevent longer numbers, decimal continuations, sign changes and
scientific-notation suffixes from satisfying an original numeric probe.
- Skip unsupported exponent prefixes during extraction instead of
falsely marking unchanged scientific notation as lost.
- Add22 regression/control cases; preserve existing nonnumeric probe
matching.
## Testing
- [x] Unit tests pass (focused `pytest` suites below)
- [x] Linting passes (`ruff check .`, Ruff0.17.0)
- [x] Type checking passes (`mypy headroom`, configured hook)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
Before fix (original main17d38e5): new22cases =>15failed,7passed
After fix: test_session_probes.py + test_session_probe_numeric_boundaries.py =>44passed
Required normal commit hooks: plugin sync, Ruff version alignment, merge-conflict check,
Ruff0.17.0, Ruff format, package mypy and Conventional Commit lint =>passed
```
The required Makefile Python test-file selection also passed on this
exact head:177passed,4optional embedding skips and1existing Anthropic
approximation warning. Whole-tree `ruff check .` with Ruff0.17.0 and
`git diff --check` passed. Full repository Python suite, Rust workspace
checks and hosted CI are not claimed by these results. The complete
local pre-push workflow and hosted CI remain separate; the bounded
upload exception and additional validation are disclosed below.
## Real Behavior Proof
- Environment: macOS arm64; Python3.13.5; source package imported
normally; actual public `run_probes` API reads a temporary JSONL file.
Offline mode and isolated workspace/config directories; no source shims,
mocks, provider calls, credentials or model downloads.
- Exact command / steps: run the following public API fixture from the
repository root.
```sh
HEADROOM_OFFLINE=1 HEADROOM_BEACON=off DO_NOT_TRACK=1 python - <<'PYCODE'
import json
import os
import tempfile
from pathlib import Path
with tempfile.TemporaryDirectory() as folder:
os.environ["HEADROOM_WORKSPACE_DIR"] = folder + "/state"
os.environ["HEADROOM_CONFIG_DIR"] = folder + "/config"
from headroom.evals.session_probes import run_probes
records = []
for output in ["port: 8080", "port: 80.5", "port: -80", "| port |\n| 80 |"]:
records.append({
"request_id": str(len(records)), "tokens_before":100, "tokens_after":10,
"original_messages":[{"role":"tool", "content":"port: 80"}],
"compressed_messages":[{"role":"tool", "content":output}],
})
for text in ["port: 80e3", "latency: 0.001e2"]:
records.append({
"tokens_before":100, "tokens_after":10,
"original_messages":[{"role":"tool", "content":text}],
"compressed_messages":[{"role":"tool", "content":text}],
})
Path(folder, "fixture.jsonl").write_text(
"".join(json.dumps(record) + "\n" for record in records),
encoding="utf-8", newline="\n",
)
report = run_probes(Path(folder))
tally = report.aggregate()["numerics"].to_dict()
assert all(event.dims["numerics"].total == 0 for event in report.events[-2:])
print(tally)
assert tally == {"total":4, "retained":1, "recoverable":0, "lost":3}
PYCODE
```
- Observed result: over six JSONL events, before, `total=6 retained=6
recoverable=0 lost=0`; after, `total=4 retained=1 recoverable=0 lost=3`.
The unchanged table-format value is retained; all three altered values
are correctly counted as lost. Two unchanged scientific-notation inputs
generate no partial decimal probe, consistent with the existing unsigned
integer/decimal extraction scope.
- Not tested: real recorded customer sessions, provider/model
evaluation, native Rust integration, hosted CI. This fixes numeric
matching within the existing probe model; it does not claim
association-aware fact scoring or eliminate the documented event-scoped
CCR limitation.
## Runtime Rollout Safety
- Rollout-managed feature(s): existing offline session-probe evaluation.
- Minimum rollout channel: unchanged.
- Stable/default behavior changed: corrected numeric retention counts;
production request compression is unaffected.
- Kill switch / disable path: offline probe evaluation is explicitly
invoked; revert the commit to restore previous scoring.
- Unsafe override required: none.
- Qualification impact: altered numeric values no longer inflate
reported fidelity.
- Rollback path: revert this commit; no persisted schema or
configuration changes.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — no public
API/config added.
- [x] My changes generate no new warnings in the focused validation
- [x] I have added tests that prove my fix is effective
- [ ] New and existing unit tests pass locally with my changes — focused
suites pass; full suite not claimed.
- [x] I did **not** edit CHANGELOG.md
### Additional validation and upload limits
Independent review on final head
`889fbd3f7e1cd92317a2afb718559d532108352a` passes 44 tests and the
six-event actual public JSONL oracle. An earlier candidate introduced an
unchanged-exponent false loss; independent review caught it and the
extraction boundary was corrected before publication. Whole-tree pinned
Ruff0.17.0 passes. The source-built native extension is available; the
Makefile Python test-file selection passes 177 tests with 4 optional
skips and one existing Anthropic approximation warning.
The Rust workspace build was stopped by a disk-reserve guard before
completion; no complete Rust-test/full-pytest/pre-push pass is claimed.
The installed `scripts/install-git-hooks.sh` documents an occasional
`git push --no-verify` upload exception, used for this independently
reviewed Python-only fix after bounded validation. No Rust source or
Cargo dependency changes. Required hosted CI and maintainer review
remain pending. Prepared and independently reviewed with AI assistance.
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4502"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
<!-- security-sync-20261028 -->
### Security baseline sync — 2026-10-28
Current exact head: `4409ba9cd3654e204c312600499811debd05b5c7`. Normal
history merge of upstream `3c6f82b06091206379106f8fe1a54f2400ed7a93`
brings in the maintainer-owned dependency remediation
[#4517](https://github.com/headroomlabs-ai/headroom/pull/4517),
following the prior npm baseline sync. The authored fix and its
test/example files are unchanged by this sync; no additional dependency
policy or runtime logic was introduced. This imports the maintainer
remediation requested by review; it does not establish audit success or
approval.
Validation on this head (macOS arm64, Python3.13.5, offline mode/beacon
off/DNT, existing source-built native extension):
`PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 python -m pytest -p
pytest_asyncio.plugin tests/test_session_probes.py
tests/test_session_probe_numeric_boundaries.py -q` (44 passed). The
six-event actual public `run_probes` JSONL fixture above was rerun:
total=4, retained=1, recoverable=0, lost=3; unchanged exponent inputs
yielded no partial probe.
`PRE_COMMIT_HOME=<existing task cache> pre-commit run --all-files`
passed, including pinned Ruff, whole-package mypy, version and
merge-conflict checks. Normal commit hooks and commitlint passed.
Existing native sources/Cargo dependencies are unchanged. The documented
occasional `git push --no-verify` exception was used for a normal
fast-forward upload; full Rust workspace tests, full pytest and complete
pre-push qualification were not rerun or claimed.
Hosted security checks and maintainer review remain separate gates.
Existing human-readiness declarations are preserved; this status update
is evidence of local validation and a baseline sync, not human approval
or a claim that hosted CI is green.
As of `2026-10-28T17:40:32.107305+00:00`, hosted pip-audit and all five
npm audits passed on this exact head ([security
workflow](https://github.com/headroomlabs-ai/headroom/actions/runs/40737842525/job/122991103291));
cargo audit also passed. Other hosted jobs remain in progress, and the
human review state remains `CHANGES_REQUESTED`. This is not complete CI
qualification or human approval.
Signed-off-by: Rudy Celekli <rudy@gradiahq.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
107 lines
3.3 KiB
Python
107 lines
3.3 KiB
Python
"""An unchanged numeric prefix is not evidence that the original value survived."""
|
|
|
|
import pytest
|
|
|
|
from headroom.evals.session_probes import probe_event
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"compressed",
|
|
[
|
|
"port: 8080",
|
|
'"port": 80.5',
|
|
"port: -80",
|
|
"port: 80e3",
|
|
"| port |\n| 8080 |",
|
|
"| port |\n| 80.5 |",
|
|
"| port |\n| -80 |",
|
|
],
|
|
)
|
|
def test_changed_numeric_value_is_lost(compressed: str) -> None:
|
|
result = probe_event(
|
|
{
|
|
"tokens_before": 100,
|
|
"tokens_after": 10,
|
|
"original_messages": [{"role": "tool", "content": "port: 80"}],
|
|
"compressed_messages": [{"role": "tool", "content": compressed}],
|
|
}
|
|
)
|
|
assert result is not None
|
|
assert result.dims["numerics"].to_dict() == {
|
|
"total": 1,
|
|
"retained": 0,
|
|
"recoverable": 0,
|
|
"lost": 1,
|
|
}
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"compressed", ["port: 80", '"port": 80', "| port |\n| 80 |", "port is 80."]
|
|
)
|
|
def test_exact_numeric_value_remains_retained(compressed: str) -> None:
|
|
result = probe_event(
|
|
{
|
|
"tokens_before": 100,
|
|
"tokens_after": 10,
|
|
"original_messages": [{"role": "tool", "content": "port: 80"}],
|
|
"compressed_messages": [{"role": "tool", "content": compressed}],
|
|
}
|
|
)
|
|
assert result is not None
|
|
assert result.dims["numerics"].retained == 1
|
|
|
|
|
|
def test_changed_value_with_marker_is_recoverable_not_retained() -> None:
|
|
result = probe_event(
|
|
{
|
|
"tokens_before": 100,
|
|
"tokens_after": 10,
|
|
"original_messages": [{"role": "tool", "content": "port: 80"}],
|
|
"compressed_messages": [{"role": "tool", "content": "port: 8080 <<ccr:abc123>>"}],
|
|
}
|
|
)
|
|
assert result is not None
|
|
assert result.dims["numerics"].retained == 0
|
|
assert result.dims["numerics"].recoverable == 1
|
|
|
|
|
|
@pytest.mark.parametrize("number", ["80e3", "80e-3", "80e+3", "0.001e2", "0.001e-2", "80E3"])
|
|
def test_unsupported_exponent_does_not_generate_partial_probe(number: str) -> None:
|
|
from headroom.evals.session_probes import extract_probe_targets
|
|
|
|
text = f"port: {number}"
|
|
assert extract_probe_targets(text)["numerics"] == set()
|
|
result = probe_event(
|
|
{
|
|
"tokens_before": 100,
|
|
"tokens_after": 10,
|
|
"original_messages": [{"role": "tool", "content": text}],
|
|
"compressed_messages": [{"role": "tool", "content": text}],
|
|
}
|
|
)
|
|
assert result is not None
|
|
assert result.dims["numerics"].total == 0
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"original,compressed,retained",
|
|
[
|
|
("port: 8080", "port: 8081", 0),
|
|
("latency: 80.5", "latency: 80.50", 0),
|
|
("latency: 80.5", "latency: 80.5", 1),
|
|
("port: 8080", "port: 8080", 1),
|
|
],
|
|
)
|
|
def test_complete_integer_and_decimal_values(original: str, compressed: str, retained: int) -> None:
|
|
result = probe_event(
|
|
{
|
|
"tokens_before": 100,
|
|
"tokens_after": 10,
|
|
"original_messages": [{"role": "tool", "content": original}],
|
|
"compressed_messages": [{"role": "tool", "content": compressed}],
|
|
}
|
|
)
|
|
assert result is not None
|
|
assert result.dims["numerics"].total == 1
|
|
assert result.dims["numerics"].retained == retained
|