mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(telemetry): give empty-item compressions a stable tool signature (#4396)
## Description
`ToolSignature.from_items([])` hashed a fresh `uuid4()`, so each
empty-items compression created a new entry in
`TelemetryCollector._tool_stats` (unbounded while `HEADROOM_TELEMETRY`
is on). Use a fixed hash: empty outputs carry no structure, so they
share one bucket.
Fixes #4392
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/telemetry/models.py`: empty-items hash is
`sha256(b"empty")[:24]` instead of a per-call uuid4 hash (still 24
chars, still not the literal `"empty"`).
- `tests/test_telemetry.py`: `TestEmptyItemsSignature` (stable hash; 50
empty compressions give one tool stat).
## Testing
- [x] Unit tests pass (`pytest`) (telemetry file only, see Not tested)
- [x] Linting passes (`ruff check .`) (changed files only)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
# source change reverted, new tests only
FAILED tests/test_telemetry.py::TestEmptyItemsSignature::test_empty_signature_is_stable
FAILED tests/test_telemetry.py::TestEmptyItemsSignature::test_repeated_empty_compressions_share_one_tool_stat
E AssertionError: assert 50 == 1
2 failed, 43 deselected
# with the fix
tests/test_telemetry.py .. 2 passed, 43 deselected
tests/test_telemetry.py 45 passed
ruff check headroom/telemetry/models.py tests/test_telemetry.py -> All checks passed!
ruff format --check (same files) -> 2 files already formatted
```
## Real Behavior Proof
- Environment: Python 3.12.15, Ubuntu on WSL2, clone at upstream/main
e995eb9f.
- Exact command / steps: `PYTHONPATH=. python -P -m pytest -q -p
no:cacheprovider tests/test_telemetry.py`, once with only the models.py
change reverted, once with the fix.
- Observed result: 50 tool stats before, 1 after (see above).
- Not tested: mypy; the full test suite. Neighbouring files
test_toin_fixes.py, test_toin_observation_only.py,
test_critical_fixes.py, test_critical_gaps.py, test_proxy_ccr.py show 6
failed / 19 errors both with and without this change on my machine
(CancelledError in test_proxy_ccr, determinism tests in
test_toin_observation_only); I did not investigate them. TOIN behaviour
for empty lists via smart_crusher.py:971 was not exercised. Whether the
collector is bounded elsewhere was not checked.
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: no (only affects telemetry
bookkeeping when telemetry is enabled)
- Kill switch / disable path: HEADROOM_TELEMETRY off
- Unsafe override required: no
- Qualification impact: none expected
- Rollback path: revert this commit
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes (see Not
tested)
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
All empty results now share one signature instead of one per call;
per-tool separation for empty outputs was never possible since the
signature has no fields.
Produced with AI assistance (Claude Code) operated by me (HetCreep); the
reproduction was run on my machine on a clone of the repository.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4396"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
JerrettDavis
parent
755293dd68
commit
a79def4dd1
@@ -214,13 +214,11 @@ class ToolSignature:
|
||||
def from_items(cls, items: list[dict[str, Any]]) -> ToolSignature:
|
||||
"""Create signature from sample items."""
|
||||
if not items:
|
||||
# HIGH FIX: Generate unique hash for empty outputs to prevent
|
||||
# different tools' empty responses from colliding into one pattern.
|
||||
# Use a random component to ensure uniqueness across tool types.
|
||||
import uuid
|
||||
|
||||
# MEDIUM FIX #15: Use 24 chars (96 bits) instead of 16 (64 bits) to reduce collision risk
|
||||
empty_hash = hashlib.sha256(f"empty:{uuid.uuid4()}".encode()).hexdigest()[:24]
|
||||
# Fixed hash (not random): a per-call uuid made every empty result a new
|
||||
# signature, so collector._tool_stats grew without bound. Empty outputs carry
|
||||
# no structure to distinguish, so they share one bucket. Not the literal
|
||||
# "empty", to stay distinct from any caller-supplied string key.
|
||||
empty_hash = hashlib.sha256(b"empty").hexdigest()[:24]
|
||||
return cls(
|
||||
structure_hash=empty_hash,
|
||||
field_count=0,
|
||||
|
||||
@@ -502,6 +502,30 @@ class TestTelemetryCollector:
|
||||
assert len(collector._events) <= 5
|
||||
|
||||
|
||||
class TestEmptyItemsSignature:
|
||||
"""Empty-item compressions must not create a new tool signature each time."""
|
||||
|
||||
def test_empty_signature_is_stable(self):
|
||||
assert (
|
||||
ToolSignature.from_items([]).structure_hash
|
||||
== ToolSignature.from_items([]).structure_hash
|
||||
)
|
||||
|
||||
def test_repeated_empty_compressions_share_one_tool_stat(self):
|
||||
collector = TelemetryCollector()
|
||||
for _ in range(50):
|
||||
collector.record_compression(
|
||||
items=[],
|
||||
original_count=0,
|
||||
compressed_count=0,
|
||||
original_tokens=10,
|
||||
compressed_tokens=10,
|
||||
strategy="none",
|
||||
)
|
||||
|
||||
assert len(collector.get_all_tool_stats()) == 1
|
||||
|
||||
|
||||
class TestTelemetryPersistence:
|
||||
"""Test telemetry persistence to disk."""
|
||||
|
||||
|
||||
Reference in New Issue
Block a user