fix(toin): skip a malformed pattern instead of aborting the store load (#4484)

## Description

One malformed pattern in the TOIN store (`~/.headroom/toin.json` by
default) aborts the whole load, and the next save then writes the
partial set over the file. This fixes part (b) of #4476.

`import_patterns` parses every stored pattern in one loop with no
per-entry handling. When `ToolPattern.from_dict` or the load sanitizer
rejects one entry, the exception leaves the loop. That happens for a
missing `tool_signature_hash`, a bad `field_semantics` or
`query_pattern_frequency` value, or an entry that isn't an object.
`_load_from_backend` catches it and only logs `toin_load_failed`.
Patterns after the bad one are never read, and the next
`record_compression` + save replaces the file with whatever loaded
before it.

Each entry is now handled on its own: parsed, validated, tracked and
merged inside one guard. An entry that raises `AttributeError`,
`TypeError` or `ValueError` at any of those steps is skipped, the rest
load, and one `toin_import_skipped` warning reports how many were
skipped plus the first error. `ToolPattern.from_dict` rejects a value of
the wrong kind before the pattern is accepted. A merge goes into a copy
of the stored pattern that replaces it only when the merge succeeds, so
a failing entry leaves no half-merged pattern behind. I caught those
three types rather than `Exception`, so a real bug in the merge or
bookkeeping still surfaces. Each type is reachable from a stored value
and pinned by a test. `KeyError` is not reachable (neither `from_dict`
subscripts its input), so it isn't caught.

Part (c) of the issue, a store over `max_storage_bytes` being replaced,
needs a maintainer decision (refuse to overwrite, trim, or enforce the
limit at save time), so it isn't changed here.

Refs #4476

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/telemetry/toin.py`:
- `import_patterns` runs each entry's parse (key,
`ToolPattern.from_dict`, `_sanitize_loaded_pattern`), key promotion,
foreign-instance tracking and merge inside one `try`. A rejected entry
is counted and skipped. After the loop, one structured warning is logged
(`event=toin_import_skipped`, `skipped`, `imported`, `error_type`,
`error`), following the `toin_load_failed` / `toin_save_failed` style.
This also covers federated imports, which go through the same function.
- `ToolPattern.from_dict` raises `TypeError` for a field whose value
isn't the kind its annotation names (str / number / bool / list / dict,
from a map built once at import), and for a `seen_instance_hashes` that
isn't a list of strings. Numbers accept int or float, because
`_merge_patterns` stores `optimal_max_items` as a weighted float.
- A merge goes into `_detached_copy(existing)`, which gives the copy its
own lists, dicts and sets. The stored pattern is replaced only when the
merge finishes.
- `tests/test_toin_retention.py`:
- a parametrized end-to-end test saves a 5-pattern store to `tmp_path`
and corrupts the second entry in five ways: no `tool_signature_hash`,
non-object `field_semantics`, non-object `query_pattern_frequency`, a
non-numeric query count, and a non-object entry. It reloads, asserts 4
patterns and one skip warning (`skipped == 1`, `imported == 4`), then
records one more pattern, saves, and asserts the file has the 4
survivors plus the new one;
- the parametrized test also covers a tracking list given as a string, a
list field given as a string, and a count given as a string;
- a foreign store (`instance_id` from another machine) whose first entry
has `seen_instance_hashes: "bad"`: the two healthy entries after it
load, track the foreign instance, and are the two patterns in the file
after `save()`;
- a foreign import whose first entry adds a key and then fails mid-merge
leaves the stored pattern unchanged, and the healthy entry after it
still merges (snapshot through JSON, since `to_dict()` shares the
pattern's containers);
  - float counts, as merges write them, still load with no skip warning;
  - a valid store loads all 5 patterns with no skip warning.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`) - run on the changed files
- [x] Type checking passes (`mypy headroom`) - run on the changed module
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# tests/test_toin_retention.py on main (3fef15d3)
FAILED tests/test_toin_retention.py::test_malformed_pattern_does_not_drop_the_rest_of_the_store[missing_signature_hash]
FAILED tests/test_toin_retention.py::test_malformed_pattern_does_not_drop_the_rest_of_the_store[bad_field_semantics]
FAILED tests/test_toin_retention.py::test_malformed_pattern_does_not_drop_the_rest_of_the_store[bad_query_frequencies]
FAILED tests/test_toin_retention.py::test_malformed_pattern_does_not_drop_the_rest_of_the_store[bad_query_count]
FAILED tests/test_toin_retention.py::test_malformed_pattern_does_not_drop_the_rest_of_the_store[not_a_dict]
5 failed, 7 passed
# each fails with `assert 1 == 4` after `TOIN storage load failed`

# the review cases (second commit), against the first commit's toin.py (c472de22)
FAILED tests/test_toin_retention.py::test_malformed_pattern_does_not_drop_the_rest_of_the_store[tracking_list_is_a_string]
FAILED tests/test_toin_retention.py::test_malformed_pattern_does_not_drop_the_rest_of_the_store[list_field_is_a_string]
FAILED tests/test_toin_retention.py::test_malformed_pattern_does_not_drop_the_rest_of_the_store[count_is_a_string]
FAILED tests/test_toin_retention.py::test_foreign_store_keeps_the_healthy_entries_after_a_malformed_one
FAILED tests/test_toin_retention.py::test_failed_merge_leaves_the_existing_pattern_unchanged
5 failed, 13 passed

# at f3be99a1
18 passed

# mutations of the fix: each must fail at least one test
N1 no field-kind check                          KILLED (2)
N2 int fields accept int only                   KILLED (1)
N3 no tracking-list check                       KILLED (1)
N4 merge in place, not on a copy                KILLED (1)
N5 copy the pattern but share its containers    KILLED (1)
N6 catch TypeError only                         KILLED (3)
N7 drop ValueError                              KILLED (1)
N8 never log the skip                           KILLED (10)
N9 always log                                   KILLED (2)
N10 fall through instead of continue            KILLED (10)
N11 count imported wrong                        KILLED (8)

# all 24 test files that use TOIN (test_toin*.py plus every file referencing it)
main 17d38e56:  113 failed, 396 passed, 2 skipped, 19 errors
f3be99a1:       113 failed, 408 passed, 2 skipped, 19 errors
the 132 failures/errors are the identical set on both

ruff 0.17.0 check / format --check (changed files): All checks passed! / 2 files already formatted
mypy 1.19.1 headroom/telemetry/toin.py: no errors in the changed module
```

The 132 failures are environmental on my machine and identical on
`main`. Most are `TestClient.__init__() got an unexpected keyword
argument 'client'` from the globally installed starlette, in
`test_proxy_loopback_gating.py` and `test_proxy_ccr.py`. The 7 mypy
errors are the Windows-only `os.fchmod` / `os.O_*` ones in
`fileperms.py` and `astgrep.py`, identical on `main`.

## Real Behavior Proof

- Environment: Windows 11, Python 3.11.9, headroom from `main`
(3fef15d3) and from this branch (f3be99a1), a real
`FileSystemTOINBackend` store in a temp directory.
- Exact command / steps: two scripts, run with `HEADROOM_OFFLINE=1
PYTHONPATH=.`. (1) The issue's step (b): record 5 compressions with 5
different `ToolSignature`s and `save()`; delete `tool_signature_hash`
from the second pattern in the JSON file; create a new
`ToolIntelligenceNetwork` on the same path, record one more compression,
`save()`, then count the patterns in the file. (2) The review's case: a
3-pattern store whose first pattern has `seen_instance_hashes: "bad"`
and whose `instance_id` is `"foreign-instance"`; reload, `save()`,
count.
- Observed result: on `main` (1) aborts after the first pattern and the
save shrinks the file from 5 patterns to 2. At c472de22 (2) still
aborted, 1 of 3, and the file went to 1. With this change both skip only
the bad entry, and every healthy pattern survives the save (output
below).
- Not tested: Linux/macOS; part (c) of the issue (unchanged here).

```text
# main (3fef15d3)
(b) saved patterns: 5
WARNING TOIN storage load failed event=toin_load_failed
(b) loaded after malformed entry: 1 of 5
(b) file after next save: 2 patterns (was 5)

# this branch (f3be99a1)
(b) saved patterns: 5
WARNING TOIN skipped malformed patterns on import event=toin_import_skipped
(b) loaded after malformed entry: 4 of 5
(b) file after next save: 5 patterns (was 5)

# review case at c472de22
WARNING TOIN storage load failed event=toin_load_failed
loaded: 1 of 3
file after save: 1 patterns (was 3)

# review case at f3be99a1
WARNING TOIN skipped malformed patterns on import event=toin_import_skipped
loaded: 2 of 3
file after save: 2 patterns (was 3)
```

Cost, from `import_patterns` on a 10,000-pattern export, best of 5 per
run, as the median of six alternating runs against c472de22 (whose
`import_patterns` timing matched `main`). A fresh load takes 74 ms vs 59
ms, about 1.5 µs per pattern for the type checks. Importing into a store
that already holds every key (the merge path) takes 139 ms vs 72 ms,
about 6.7 µs per merge for the copy. A full `deepcopy` cost 373 ms
there, which is why `_detached_copy` copies only the containers.
`import_patterns` runs at startup and on federated imports, not per
request, and a plain startup load doesn't merge.

## Runtime Rollout Safety

- Rollout-managed feature(s): None
- Minimum rollout channel: N/A
- Stable/default behavior changed: Only for a store or import containing
an unreadable entry, including one whose field holds the wrong kind of
value (previously accepted as-is). That entry is skipped, the others
load and survive the next save, and a new `toin_import_skipped` warning
is logged. A valid store loads exactly as before.
- Kill switch / disable path: None added. `HEADROOM_TOIN_BACKEND=none`
still disables TOIN storage.
- Unsafe override required: No
- Qualification impact: None
- Rollback path: Revert the PR's squash commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A: no
documented behavior changes)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- The skipped entry is not written back on the next save. It was
unreadable, so it can't be carried forward as a pattern. The issue
suggests keeping a one-time copy of the original file (e.g.
`toin.json.bad`) before such a save; I left that out as a separate
choice about extra files.
- Not covered, and unchanged from before: a `patterns` value that isn't
an object still fails the whole load. Element types inside a list or
dict field aren't checked, except `seen_instance_hashes`; a bad element
that makes a merge raise is skipped without leaving partial state.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4484"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
Ayush Kumar Jha
2026-10-28 23:35:52 -05:00
committed by GitHub
co-authored by JD Davis
parent b0d62f1d6d
commit 8b628d9685
2 changed files with 238 additions and 27 deletions
+91 -27
View File
@@ -68,7 +68,7 @@ import threading
import time
import warnings
from collections.abc import Callable
from dataclasses import dataclass, field
from dataclasses import dataclass, field, fields
from typing import Any, Final
from .models import FieldSemantics, ToolSignature
@@ -409,9 +409,16 @@ class ToolPattern:
"last_updated",
}
filtered = {k: v for k, v in data.items() if k in valid_fields}
# A value of the wrong kind (a string where a list belongs) would be
# accepted as-is and then iterated or added to when patterns merge.
for name, value in filtered.items():
if not isinstance(value, _PATTERN_FIELD_KINDS.get(name, object)):
raise TypeError(f"{name} is a {type(value).__name__}")
# Handle seen_instance_hashes (serialized without underscore prefix)
seen_hashes = data.get("seen_instance_hashes", [])
if not isinstance(seen_hashes, list) or not all(isinstance(h, str) for h in seen_hashes):
raise TypeError("seen_instance_hashes is not a list of strings")
pattern = cls(**filtered)
pattern._seen_instance_hashes = seen_hashes[:100] # Limit on load
@@ -438,6 +445,38 @@ class ToolPattern:
return pattern
# What each stored ToolPattern field may hold, by the base of its annotation.
# Merges store some counts as floats, so numeric fields accept either.
_KIND_BY_ANNOTATION: Final[dict[str, type | tuple[type, ...]]] = {
"str": str,
"int": (int, float),
"float": (int, float),
"bool": bool,
"list": list,
"dict": dict,
}
_PATTERN_FIELD_KINDS: Final[dict[str, type | tuple[type, ...]]] = {
f.name: _KIND_BY_ANNOTATION.get(str(f.type).split("[")[0], object) for f in fields(ToolPattern)
}
_PATTERN_CONTAINER_FIELDS: Final = tuple(
f.name for f in fields(ToolPattern) if f.default_factory in (list, dict, set)
)
def _detached_copy(pattern: ToolPattern) -> ToolPattern:
"""Copy ``pattern`` so that merging into the copy leaves the original intact.
``_merge_patterns`` reassigns scalars and adds to the pattern's lists, dicts
and sets, so those containers are the parts that need their own copies.
"""
import copy
clone = copy.copy(pattern)
for name in _PATTERN_CONTAINER_FIELDS:
setattr(clone, name, getattr(pattern, name).copy())
return clone
@dataclass
class TOINConfig:
"""Configuration for the Tool Output Intelligence Network."""
@@ -1397,40 +1436,65 @@ class ToolIntelligenceNetwork:
patterns_data = data.get("patterns", {})
source_instance = data.get("instance_id", "unknown")
skipped = 0
first_error: Exception | None = None
with self._lock:
for serialized_key, pattern_dict in patterns_data.items():
key = _deserialize_pattern_key(serialized_key)
imported = ToolPattern.from_dict(pattern_dict)
self._sanitize_loaded_pattern(imported)
# Make sure dataclass fields agree with the dict key — pre-F3
# dumps don't carry tenant_key on the pattern, and pre-B5
# dumps don't carry auth_mode/model_family either; promote
# from the (possibly default) key.
imported.tenant_key = key[0]
imported.auth_mode = key[1]
imported.model_family = key[2]
# Each entry is parsed, tracked and merged into a pattern that is
# only stored once all of that succeeds. One unreadable entry must
# not abort the rest (the store that did load would be saved over
# the whole file) or leave a half-merged pattern behind.
try:
key = _deserialize_pattern_key(serialized_key)
imported = ToolPattern.from_dict(pattern_dict)
self._sanitize_loaded_pattern(imported)
# Make sure dataclass fields agree with the dict key — pre-F3
# dumps don't carry tenant_key on the pattern, and pre-B5
# dumps don't carry auth_mode/model_family either; promote
# from the (possibly default) key.
imported.tenant_key = key[0]
imported.auth_mode = key[1]
imported.model_family = key[2]
if key in self._patterns:
# Merge with existing
self._merge_patterns(self._patterns[key], imported)
else:
# Add new pattern - need to track source instance
self._patterns[key] = imported
existing = self._patterns.get(key)
if existing is not None:
# Merge with existing
pattern = _detached_copy(existing)
self._merge_patterns(pattern, imported)
else:
# Add new pattern - need to track source instance
pattern = imported
# For NEW patterns from another instance, track the source in
# _seen_instance_hashes so user_count reflects cross-user data
if source_instance != self._instance_id:
pattern = self._patterns[key]
if source_instance not in pattern._seen_instance_hashes:
# Limit storage to 100 unique instances to bound memory
if len(pattern._seen_instance_hashes) < 100:
pattern._seen_instance_hashes.append(source_instance)
# CRITICAL: Always increment user_count (even after cap)
pattern.user_count += 1
# For NEW patterns from another instance, track the source in
# _seen_instance_hashes so user_count reflects cross-user data
if source_instance != self._instance_id:
if source_instance not in pattern._seen_instance_hashes:
# Limit storage to 100 unique instances to bound memory
if len(pattern._seen_instance_hashes) < 100:
pattern._seen_instance_hashes.append(source_instance)
# CRITICAL: Always increment user_count (even after cap)
pattern.user_count += 1
except (AttributeError, TypeError, ValueError) as e:
skipped += 1
first_error = first_error or e
continue
self._patterns[key] = pattern
self._prune_patterns_locked()
self._dirty = True
if first_error is not None:
logger.warning(
"TOIN skipped malformed patterns on import",
extra={
"event": "toin_import_skipped",
"skipped": skipped,
"imported": len(patterns_data) - skipped,
"error_type": type(first_error).__name__,
"error": str(first_error),
},
)
def _merge_patterns(self, existing: ToolPattern, imported: ToolPattern) -> None:
"""Merge imported pattern into existing."""
total = existing.sample_size + imported.sample_size
+147
View File
@@ -3,6 +3,9 @@
from __future__ import annotations
import json
import logging
import pytest
from headroom.telemetry.backends.filesystem import FileSystemTOINBackend
from headroom.telemetry.models import ToolSignature
@@ -83,6 +86,150 @@ def test_legacy_raw_query_keys_are_removed_on_load(tmp_path) -> None:
assert pattern.common_query_patterns == ["status:* user:*"]
def _record(toin: ToolIntelligenceNetwork, index: int) -> None:
toin.record_compression(
tool_signature=_signature(index),
original_count=10,
compressed_count=5,
original_tokens=100,
compressed_tokens=50,
strategy="test",
)
def _saved_store(path, count: int) -> dict:
toin = ToolIntelligenceNetwork(TOINConfig(storage_path=str(path), auto_save_interval=0))
for index in range(count):
_record(toin, index)
toin.save()
return json.loads(path.read_text(encoding="utf-8"))
def _skip_warnings(caplog) -> list[logging.LogRecord]:
return [r for r in caplog.records if getattr(r, "event", None) == "toin_import_skipped"]
@pytest.mark.parametrize(
"corrupt",
[
lambda entry: {k: v for k, v in entry.items() if k != "tool_signature_hash"},
lambda entry: {**entry, "field_semantics": {"field_1": "not a dict"}},
lambda entry: {**entry, "query_pattern_frequency": "not a dict"},
lambda entry: {**entry, "query_pattern_frequency": {"status:*": "many"}},
lambda entry: {**entry, "seen_instance_hashes": "bad"},
lambda entry: {**entry, "commonly_retrieved_fields": "abc"},
lambda entry: {**entry, "sample_size": "x"},
lambda entry: "not a pattern",
],
ids=[
"missing_signature_hash",
"bad_field_semantics",
"bad_query_frequencies",
"bad_query_count",
"tracking_list_is_a_string",
"list_field_is_a_string",
"count_is_a_string",
"not_a_dict",
],
)
def test_malformed_pattern_does_not_drop_the_rest_of_the_store(tmp_path, caplog, corrupt) -> None:
path = tmp_path / "toin.json"
data = _saved_store(path, 5)
keys = list(data["patterns"])
data["patterns"][keys[1]] = corrupt(data["patterns"][keys[1]])
path.write_text(json.dumps(data), encoding="utf-8")
with caplog.at_level(logging.WARNING, logger="headroom.telemetry.toin"):
toin = ToolIntelligenceNetwork(TOINConfig(storage_path=str(path), auto_save_interval=0))
assert len(toin._patterns) == 4
[warning] = _skip_warnings(caplog)
assert warning.skipped == 1
assert warning.imported == 4
# The next save used to write only the patterns read before the bad one.
_record(toin, 99)
toin.save()
saved = json.loads(path.read_text(encoding="utf-8"))["patterns"]
assert set(keys) - {keys[1]} <= set(saved)
assert len(saved) == 5
def test_foreign_store_keeps_the_healthy_entries_after_a_malformed_one(tmp_path, caplog) -> None:
path = tmp_path / "toin.json"
data = _saved_store(path, 3)
keys = list(data["patterns"])
# A string tracking list used to pass from_dict and then fail on .append()
# while tracking the foreign source, outside the per-entry guard.
data["patterns"][keys[0]]["seen_instance_hashes"] = "bad"
data["instance_id"] = "foreign-instance"
path.write_text(json.dumps(data), encoding="utf-8")
with caplog.at_level(logging.WARNING, logger="headroom.telemetry.toin"):
toin = ToolIntelligenceNetwork(TOINConfig(storage_path=str(path), auto_save_interval=0))
assert len(toin._patterns) == 2
[warning] = _skip_warnings(caplog)
assert warning.skipped == 1
assert all("foreign-instance" in p._seen_instance_hashes for p in toin._patterns.values())
toin.save()
saved = json.loads(path.read_text(encoding="utf-8"))["patterns"]
assert set(saved) == set(keys[1:])
def test_failed_merge_leaves_the_existing_pattern_unchanged(caplog) -> None:
toin = ToolIntelligenceNetwork(TOINConfig(storage_path=""))
for index in range(2):
_record(toin, index)
exported = toin.export_patterns()
keys = list(exported["patterns"])
# to_dict() shares the pattern's containers, so snapshot through JSON.
before = {key: json.loads(json.dumps(p.to_dict())) for key, p in toin._patterns.items()}
bad = dict(exported["patterns"][keys[0]])
# Passes from_dict, then fails part-way through _merge_patterns: the counts
# and the "field_new" key are already added when the "x" value raises.
bad["field_retrieval_frequency"] = {"field_new": 1, "field_0": "x"}
healthy = dict(exported["patterns"][keys[1]])
with caplog.at_level(logging.WARNING, logger="headroom.telemetry.toin"):
toin.import_patterns(
{"instance_id": "foreign-instance", "patterns": {keys[0]: bad, keys[1]: healthy}}
)
[warning] = _skip_warnings(caplog)
assert warning.skipped == 1
first, second = (toin._patterns[key] for key in before)
assert json.loads(json.dumps(first.to_dict())) == before[next(iter(before))]
assert second.total_compressions == 2
def test_float_counts_written_by_merges_still_load(tmp_path, caplog) -> None:
# _merge_patterns stores optimal_max_items as a weighted float.
path = tmp_path / "toin.json"
data = _saved_store(path, 2)
for entry in data["patterns"].values():
entry["optimal_max_items"] = 37.5
path.write_text(json.dumps(data), encoding="utf-8")
with caplog.at_level(logging.WARNING, logger="headroom.telemetry.toin"):
toin = ToolIntelligenceNetwork(TOINConfig(storage_path=str(path), auto_save_interval=0))
assert len(toin._patterns) == 2
assert _skip_warnings(caplog) == []
def test_valid_store_loads_without_a_skip_warning(tmp_path, caplog) -> None:
path = tmp_path / "toin.json"
_saved_store(path, 5)
with caplog.at_level(logging.WARNING, logger="headroom.telemetry.toin"):
toin = ToolIntelligenceNetwork(TOINConfig(storage_path=str(path), auto_save_interval=0))
assert len(toin._patterns) == 5
assert _skip_warnings(caplog) == []
def test_filesystem_backend_writes_compact_json(tmp_path) -> None:
path = tmp_path / "toin.json"
FileSystemTOINBackend(str(path)).save({"patterns": {"a": {"value": 1}}})