fix(toin): make diagnostic pattern identifiers addressable (#4021)

## Description

Return the full scoped aggregation key, using the issue's option 2,
rather than truncating it. The current key contains tenant, auth mode,
model family, and signature; truncating to 12 characters makes different
patterns share IDs and makes list-to-detail lookup silently return a
different pattern.

Closes #2928.

## Type of Change

- [x] Bug fix
- [x] Documentation update
- [x] Intentional diagnostic contract change: listed `hash` values are
complete scoped identifiers; ambiguous abbreviated lookups now return
409 instead of an arbitrary first match.

## Changes Made

- Return the exact stored scoped key from `/v1/toin/patterns`.
- Resolve exact keys first; allow an abbreviated prefix only if it
matches one pattern. Keep 404 for unknown IDs and return 409 for
ambiguity.
- Capture the complete URL-decoded identifier, including a slash in a
tenant ID, without falling through to an upstream passthrough route.
- Preserve loopback authorization and the existing privacy whitelist.
- Update the TypeScript SDK README to round-trip the listed ID. No SDK
runtime change: `getPattern` already URL-encodes the string.

## Testing

- [x] Unit tests pass (`pytest`), including scoped-ID round-trips and
authorization regressions.
- [x] Changed-file lint, configured pre-commit mypy and manual
production HTTP checks pass.

### Test Output

```text
================== 144 passed, 1 warning in 71.73s (0:01:11) ===================
```

The mandatory pre-push `make ci-precheck` passed: workspace Rust
fmt/clippy/tests, native extension build/import verification, the
complete configured Python gate, and Commitlint. The native gate used
the installed CPU `libonnxruntime.so.1.26.0` explicitly via
`ORT_DYLIB_PATH`, without skipping cached-model assertions.

```text
[build_rust_extension] headroom._core build + install + verify: OK
================== 177 passed, 4 skipped, 1 warning in 9.70s ===================
✅ ci-precheck PASSED — safe to push.
```

```sh
python -m pytest -q tests/test_proxy_loopback_gating.py
```

**144 passed**, one existing Starlette TestClient deprecation warning.
The new consumer regressions produce **2 failed, 2 passed** against the
unchanged baseline (preloaded baseline modules prevent foreign-worktree
conftest imports from selecting the treatment).

Changed-file Ruff, formatting, installed pre-commit `mypy headroom`, and
Commitlint hooks passed. Full-tree Ruff was not green locally because
five unchanged OAuth2 plugin files have executable filesystem
permissions while Git tracks 100644. A separate full dev-environment
mypy run found an unchanged optional LangChain `truthy-function` error;
the isolated configured pre-commit mypy hook passed. No unrelated source
or permissions were changed.

## Real Behavior Proof

- Environment: Linux x86_64, CPython 3.13.14; unchanged
`dea9092e19385f286b7f67a64f4d85de2708d756` and this branch, real
production FastAPI app over loopback HTTP (ports 18792/18793), actual
TOIN instance and public `import_patterns`, no mocked endpoint or fake
TOIN.
- Exact command / steps: import four patterns
`global|unknown|unknown|abc123`, `global|unknown|unknown|abc456`,
`global|payg|claude|abc123`, `other-tenant|unknown|unknown|abc123` with
counts 8, 16, 24, 32; GET `/v1/toin/patterns`; URL-encode each returned
`hash` and GET `/v1/toin/pattern/{hash}`; request the shared prefix
`global|unknown|unknown|abc`.
- Before: four entries produced only three IDs. The entry with count 16
returned count 8 when addressed by its listed ID. The shared prefix
returned 200.
- Observed result: after the fix, four entries produce four distinct
IDs; every detail request returns its own count (8→8, 16→16, 24→24,
32→32); the ambiguous prefix returns **409**. Exact and unique-prefix
lookups remain successful; unknown-prefix regression remains 404.
- Rebased onto current main `1ad92f0cbc3dea27676e253588100f64a2af6994`
and reran the full suite: **144 passed**. A further real HTTP smoke on
port 18794 imported a fifth key,
`tenant/alpha?scope=one#fragment|unknown|unknown|abc123`, URL-encoded
it, and round-tripped all **5/5** counts including **40→40**. The
reserved-character edge case failed before the path-converter fix and
passed after it.
- Not tested: a live remote tenant deployment or the SDK against a cloud
proxy. Existing authorization/privacy tests were exercised locally.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; diagnostic API bug fix.
- Minimum rollout channel: stable; no rollout-managed feature is
enabled.
- Stable/default behavior changed: full scoped list identifiers and
rejection of ambiguous prefixes. This is the selected issue alternative,
not a new raw-signature alias or migration shim.
- Kill switch / disable path: disable access to the loopback diagnostic
API; no new runtime flag.
- Unsafe override required: no.
- Qualification impact: diagnostic IDs only; no compression or billing
change.
- Rollback path: revert this commit; no persistent data migration.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] Existing API and privacy conventions reused; SDK documentation
updated.
- [x] Regression tests cover scoped collisions, list-to-detail
round-trips, exact/unique/ambiguous/missing lookups, and existing
loopback gating.
- [x] `CHANGELOG.md` not edited; Conventional Commit title supplied.

GitHub CI results are not inferred from local checks.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4021"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
Mansurshakh Japarov
2026-10-07 14:07:38 -05:00
committed by GitHub
co-authored by JD Davis
parent cd4367b040
commit def6d41185
3 changed files with 97 additions and 25 deletions
+29 -24
View File
@@ -5935,7 +5935,7 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
limit: Maximum number of patterns to return (default 20)
Response includes for each pattern:
- hash: Truncated tool signature hash (12 chars)
- hash: Full aggregation key identifying one scoped tool pattern
- compressions: Total compression events
- retrievals: Total retrieval events
- retrieval_rate: Percentage of compressions that triggered retrieval
@@ -5959,7 +5959,7 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
patterns_list.append(
{
"hash": sig_hash[:12],
"hash": sig_hash,
"compressions": total_compressions,
"retrievals": total_retrievals,
"retrieval_rate": f"{retrieval_rate:.1%}",
@@ -5979,36 +5979,41 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
return patterns_list[:limit]
@app.get("/v1/toin/pattern/{hash_prefix}", dependencies=[Depends(_require_loopback)])
@app.get("/v1/toin/pattern/{hash_prefix:path}", dependencies=[Depends(_require_loopback)])
async def toin_pattern_detail(hash_prefix: str):
"""Get detailed TOIN pattern info by hash prefix.
"""Get detailed TOIN pattern info by the listed identifier or a unique prefix.
Searches for a pattern where the tool signature hash starts with
the provided prefix. Returns full pattern details if found.
Pass the ``hash`` returned by ``/v1/toin/patterns`` for an exact
lookup. A shorter aggregation-key prefix is accepted only when it
identifies one pattern; ambiguous prefixes return 409.
Path params:
hash_prefix: Beginning of the tool signature hash (min 4 chars recommended)
Response: Full pattern.to_dict() with all learned statistics and recommendations.
Response: Learned statistics without query text or field semantics.
"""
toin = get_toin()
exported = toin.export_patterns()
patterns_data = exported.get("patterns", {})
# Search for pattern with matching hash prefix
for sig_hash, pattern_dict in patterns_data.items():
if sig_hash.startswith(hash_prefix):
# Keep this response aligned with /v1/toin/patterns while
# excluding query text, field semantics, and other internal
# learning state from the detail endpoint.
return {
"compressions": pattern_dict.get("total_compressions", 0),
"retrievals": pattern_dict.get("total_retrievals", 0),
"retrieval_rate": pattern_dict.get("retrieval_rate", 0.0),
"confidence": pattern_dict.get("confidence", 0.0),
"skip_recommended": pattern_dict.get("skip_compression_recommended", False),
"optimal_max_items": pattern_dict.get("optimal_max_items", 20),
}
pattern_dict = patterns_data.get(hash_prefix)
if pattern_dict is None:
matches = (
pattern for key, pattern in patterns_data.items() if key.startswith(hash_prefix)
)
pattern_dict = next(matches, None)
if next(matches, None) is not None:
raise HTTPException(
status_code=409,
detail="Ambiguous TOIN pattern prefix; use a full hash from /v1/toin/patterns",
)
if pattern_dict is not None:
# Keep query text, field semantics, and internal learning state private.
return {
"compressions": pattern_dict.get("total_compressions", 0),
"retrievals": pattern_dict.get("total_retrievals", 0),
"retrieval_rate": pattern_dict.get("retrieval_rate", 0.0),
"confidence": pattern_dict.get("confidence", 0.0),
"skip_recommended": pattern_dict.get("skip_compression_recommended", False),
"optimal_max_items": pattern_dict.get("optimal_max_items", 20),
}
raise HTTPException(
status_code=404, detail=f"No TOIN pattern found with hash starting with: {hash_prefix}"
+8
View File
@@ -279,8 +279,16 @@ const hints = await client.feedback.getHints('list_servers');
// TOIN (Tool Output Intelligence Network)
const toinStats = await client.toin.getStats();
const patterns = await client.toin.getPatterns(20);
const pattern = patterns[0] && await client.toin.getPattern(patterns[0].hash);
```
Each `hash` is the complete scoped aggregation key, including tenant, auth mode,
model family, and tool signature. Pass it back unchanged for an exact lookup;
the client URL-encodes it. Shorter aggregation-key prefixes are accepted only
when they identify one pattern; ambiguous prefixes return HTTP 409 rather than
an arbitrary pattern.
## Configuration Types
Full TypeScript interfaces for every Python config dataclass.
+60 -1
View File
@@ -112,7 +112,9 @@ def test_toin_pattern_detail_whitelists_learned_payload(monkeypatch: pytest.Monk
}
monkeypatch.setattr("headroom.proxy.server.get_toin", lambda: FakeTOIN())
response = _loopback_client().get("/v1/toin/pattern/unknown")
client = _loopback_client()
identifier = client.get("/v1/toin/patterns").json()[0]["hash"]
response = client.get(f"/v1/toin/pattern/{identifier}")
assert response.status_code == 200
assert response.json() == {
@@ -125,6 +127,63 @@ def test_toin_pattern_detail_whitelists_learned_payload(monkeypatch: pytest.Monk
}
@pytest.fixture
def scoped_toin(monkeypatch: pytest.MonkeyPatch, tmp_path):
from headroom.telemetry.toin import TOINConfig, ToolIntelligenceNetwork
toin = ToolIntelligenceNetwork(TOINConfig(storage_path=str(tmp_path / "toin.json")))
patterns = {
"global|unknown|unknown|abc123": 8,
"global|unknown|unknown|abc456": 16,
"global|payg|claude|abc123": 24,
"other-tenant|unknown|unknown|abc123": 32,
"tenant/alpha?scope=one#fragment|unknown|unknown|abc123": 40,
}
toin.import_patterns(
{
"instance_id": "scoped-pattern-regression",
"patterns": {
key: {
"tool_signature_hash": key.rsplit("|", 1)[-1],
"sample_size": count,
"total_compressions": count,
"total_retrievals": 0,
}
for key, count in patterns.items()
},
}
)
monkeypatch.setattr("headroom.proxy.server.get_toin", lambda: toin)
return patterns
def test_listed_toin_identifiers_address_their_own_scoped_pattern(scoped_toin) -> None:
from urllib.parse import quote
client = _loopback_client()
listed = client.get("/v1/toin/patterns").json()
assert {item["hash"] for item in listed} == set(scoped_toin)
for item in listed:
response = client.get(f"/v1/toin/pattern/{quote(item['hash'], safe='')}")
assert response.status_code == 200
assert response.json()["compressions"] == item["compressions"]
@pytest.mark.parametrize(
("prefix", "status", "count"),
[
("global|unknown|unknown|abc", 409, None),
("global|unknown|unknown|abc1", 200, 8),
("missing-pattern", 404, None),
],
)
def test_toin_prefix_lookup_is_unambiguous(scoped_toin, prefix, status, count) -> None:
response = _loopback_client().get(f"/v1/toin/pattern/{prefix}")
assert response.status_code == status
if count is not None:
assert response.json()["compressions"] == count
# Mutating routes reachable from loopback. `require_loopback` cannot stop a
# remote page from POSTing to a known 127.0.0.1 URL: a "simple" cross-origin
# request (Content-Type: text/plain carrying JSON) skips preflight, and the