mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(toin): make diagnostic pattern identifiers addressable (#4021)
## Description Return the full scoped aggregation key, using the issue's option 2, rather than truncating it. The current key contains tenant, auth mode, model family, and signature; truncating to 12 characters makes different patterns share IDs and makes list-to-detail lookup silently return a different pattern. Closes #2928. ## Type of Change - [x] Bug fix - [x] Documentation update - [x] Intentional diagnostic contract change: listed `hash` values are complete scoped identifiers; ambiguous abbreviated lookups now return 409 instead of an arbitrary first match. ## Changes Made - Return the exact stored scoped key from `/v1/toin/patterns`. - Resolve exact keys first; allow an abbreviated prefix only if it matches one pattern. Keep 404 for unknown IDs and return 409 for ambiguity. - Capture the complete URL-decoded identifier, including a slash in a tenant ID, without falling through to an upstream passthrough route. - Preserve loopback authorization and the existing privacy whitelist. - Update the TypeScript SDK README to round-trip the listed ID. No SDK runtime change: `getPattern` already URL-encodes the string. ## Testing - [x] Unit tests pass (`pytest`), including scoped-ID round-trips and authorization regressions. - [x] Changed-file lint, configured pre-commit mypy and manual production HTTP checks pass. ### Test Output ```text ================== 144 passed, 1 warning in 71.73s (0:01:11) =================== ``` The mandatory pre-push `make ci-precheck` passed: workspace Rust fmt/clippy/tests, native extension build/import verification, the complete configured Python gate, and Commitlint. The native gate used the installed CPU `libonnxruntime.so.1.26.0` explicitly via `ORT_DYLIB_PATH`, without skipping cached-model assertions. ```text [build_rust_extension] headroom._core build + install + verify: OK ================== 177 passed, 4 skipped, 1 warning in 9.70s =================== ✅ ci-precheck PASSED — safe to push. ``` ```sh python -m pytest -q tests/test_proxy_loopback_gating.py ``` **144 passed**, one existing Starlette TestClient deprecation warning. The new consumer regressions produce **2 failed, 2 passed** against the unchanged baseline (preloaded baseline modules prevent foreign-worktree conftest imports from selecting the treatment). Changed-file Ruff, formatting, installed pre-commit `mypy headroom`, and Commitlint hooks passed. Full-tree Ruff was not green locally because five unchanged OAuth2 plugin files have executable filesystem permissions while Git tracks 100644. A separate full dev-environment mypy run found an unchanged optional LangChain `truthy-function` error; the isolated configured pre-commit mypy hook passed. No unrelated source or permissions were changed. ## Real Behavior Proof - Environment: Linux x86_64, CPython 3.13.14; unchanged `dea9092e19385f286b7f67a64f4d85de2708d756` and this branch, real production FastAPI app over loopback HTTP (ports 18792/18793), actual TOIN instance and public `import_patterns`, no mocked endpoint or fake TOIN. - Exact command / steps: import four patterns `global|unknown|unknown|abc123`, `global|unknown|unknown|abc456`, `global|payg|claude|abc123`, `other-tenant|unknown|unknown|abc123` with counts 8, 16, 24, 32; GET `/v1/toin/patterns`; URL-encode each returned `hash` and GET `/v1/toin/pattern/{hash}`; request the shared prefix `global|unknown|unknown|abc`. - Before: four entries produced only three IDs. The entry with count 16 returned count 8 when addressed by its listed ID. The shared prefix returned 200. - Observed result: after the fix, four entries produce four distinct IDs; every detail request returns its own count (8→8, 16→16, 24→24, 32→32); the ambiguous prefix returns **409**. Exact and unique-prefix lookups remain successful; unknown-prefix regression remains 404. - Rebased onto current main `1ad92f0cbc3dea27676e253588100f64a2af6994` and reran the full suite: **144 passed**. A further real HTTP smoke on port 18794 imported a fifth key, `tenant/alpha?scope=one#fragment|unknown|unknown|abc123`, URL-encoded it, and round-tripped all **5/5** counts including **40→40**. The reserved-character edge case failed before the path-converter fix and passed after it. - Not tested: a live remote tenant deployment or the SDK against a cloud proxy. Existing authorization/privacy tests were exercised locally. ## Runtime Rollout Safety - Rollout-managed feature(s): none; diagnostic API bug fix. - Minimum rollout channel: stable; no rollout-managed feature is enabled. - Stable/default behavior changed: full scoped list identifiers and rejection of ambiguous prefixes. This is the selected issue alternative, not a new raw-signature alias or migration shim. - Kill switch / disable path: disable access to the loopback diagnostic API; no new runtime flag. - Unsafe override required: no. - Qualification impact: diagnostic IDs only; no compression or billing change. - Rollback path: revert this commit; no persistent data migration. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] Existing API and privacy conventions reused; SDK documentation updated. - [x] Regression tests cover scoped collisions, list-to-detail round-trips, exact/unique/ambiguous/missing lookups, and existing loopback gating. - [x] `CHANGELOG.md` not edited; Conventional Commit title supplied. GitHub CI results are not inferred from local checks. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4021" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
JD Davis
parent
cd4367b040
commit
def6d41185
+29
-24
@@ -5935,7 +5935,7 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
|
||||
limit: Maximum number of patterns to return (default 20)
|
||||
|
||||
Response includes for each pattern:
|
||||
- hash: Truncated tool signature hash (12 chars)
|
||||
- hash: Full aggregation key identifying one scoped tool pattern
|
||||
- compressions: Total compression events
|
||||
- retrievals: Total retrieval events
|
||||
- retrieval_rate: Percentage of compressions that triggered retrieval
|
||||
@@ -5959,7 +5959,7 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
|
||||
|
||||
patterns_list.append(
|
||||
{
|
||||
"hash": sig_hash[:12],
|
||||
"hash": sig_hash,
|
||||
"compressions": total_compressions,
|
||||
"retrievals": total_retrievals,
|
||||
"retrieval_rate": f"{retrieval_rate:.1%}",
|
||||
@@ -5979,36 +5979,41 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
|
||||
|
||||
return patterns_list[:limit]
|
||||
|
||||
@app.get("/v1/toin/pattern/{hash_prefix}", dependencies=[Depends(_require_loopback)])
|
||||
@app.get("/v1/toin/pattern/{hash_prefix:path}", dependencies=[Depends(_require_loopback)])
|
||||
async def toin_pattern_detail(hash_prefix: str):
|
||||
"""Get detailed TOIN pattern info by hash prefix.
|
||||
"""Get detailed TOIN pattern info by the listed identifier or a unique prefix.
|
||||
|
||||
Searches for a pattern where the tool signature hash starts with
|
||||
the provided prefix. Returns full pattern details if found.
|
||||
Pass the ``hash`` returned by ``/v1/toin/patterns`` for an exact
|
||||
lookup. A shorter aggregation-key prefix is accepted only when it
|
||||
identifies one pattern; ambiguous prefixes return 409.
|
||||
|
||||
Path params:
|
||||
hash_prefix: Beginning of the tool signature hash (min 4 chars recommended)
|
||||
|
||||
Response: Full pattern.to_dict() with all learned statistics and recommendations.
|
||||
Response: Learned statistics without query text or field semantics.
|
||||
"""
|
||||
toin = get_toin()
|
||||
exported = toin.export_patterns()
|
||||
patterns_data = exported.get("patterns", {})
|
||||
|
||||
# Search for pattern with matching hash prefix
|
||||
for sig_hash, pattern_dict in patterns_data.items():
|
||||
if sig_hash.startswith(hash_prefix):
|
||||
# Keep this response aligned with /v1/toin/patterns while
|
||||
# excluding query text, field semantics, and other internal
|
||||
# learning state from the detail endpoint.
|
||||
return {
|
||||
"compressions": pattern_dict.get("total_compressions", 0),
|
||||
"retrievals": pattern_dict.get("total_retrievals", 0),
|
||||
"retrieval_rate": pattern_dict.get("retrieval_rate", 0.0),
|
||||
"confidence": pattern_dict.get("confidence", 0.0),
|
||||
"skip_recommended": pattern_dict.get("skip_compression_recommended", False),
|
||||
"optimal_max_items": pattern_dict.get("optimal_max_items", 20),
|
||||
}
|
||||
pattern_dict = patterns_data.get(hash_prefix)
|
||||
if pattern_dict is None:
|
||||
matches = (
|
||||
pattern for key, pattern in patterns_data.items() if key.startswith(hash_prefix)
|
||||
)
|
||||
pattern_dict = next(matches, None)
|
||||
if next(matches, None) is not None:
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail="Ambiguous TOIN pattern prefix; use a full hash from /v1/toin/patterns",
|
||||
)
|
||||
if pattern_dict is not None:
|
||||
# Keep query text, field semantics, and internal learning state private.
|
||||
return {
|
||||
"compressions": pattern_dict.get("total_compressions", 0),
|
||||
"retrievals": pattern_dict.get("total_retrievals", 0),
|
||||
"retrieval_rate": pattern_dict.get("retrieval_rate", 0.0),
|
||||
"confidence": pattern_dict.get("confidence", 0.0),
|
||||
"skip_recommended": pattern_dict.get("skip_compression_recommended", False),
|
||||
"optimal_max_items": pattern_dict.get("optimal_max_items", 20),
|
||||
}
|
||||
|
||||
raise HTTPException(
|
||||
status_code=404, detail=f"No TOIN pattern found with hash starting with: {hash_prefix}"
|
||||
|
||||
@@ -279,8 +279,16 @@ const hints = await client.feedback.getHints('list_servers');
|
||||
// TOIN (Tool Output Intelligence Network)
|
||||
const toinStats = await client.toin.getStats();
|
||||
const patterns = await client.toin.getPatterns(20);
|
||||
const pattern = patterns[0] && await client.toin.getPattern(patterns[0].hash);
|
||||
```
|
||||
|
||||
Each `hash` is the complete scoped aggregation key, including tenant, auth mode,
|
||||
model family, and tool signature. Pass it back unchanged for an exact lookup;
|
||||
the client URL-encodes it. Shorter aggregation-key prefixes are accepted only
|
||||
when they identify one pattern; ambiguous prefixes return HTTP 409 rather than
|
||||
an arbitrary pattern.
|
||||
|
||||
|
||||
## Configuration Types
|
||||
|
||||
Full TypeScript interfaces for every Python config dataclass.
|
||||
|
||||
@@ -112,7 +112,9 @@ def test_toin_pattern_detail_whitelists_learned_payload(monkeypatch: pytest.Monk
|
||||
}
|
||||
|
||||
monkeypatch.setattr("headroom.proxy.server.get_toin", lambda: FakeTOIN())
|
||||
response = _loopback_client().get("/v1/toin/pattern/unknown")
|
||||
client = _loopback_client()
|
||||
identifier = client.get("/v1/toin/patterns").json()[0]["hash"]
|
||||
response = client.get(f"/v1/toin/pattern/{identifier}")
|
||||
|
||||
assert response.status_code == 200
|
||||
assert response.json() == {
|
||||
@@ -125,6 +127,63 @@ def test_toin_pattern_detail_whitelists_learned_payload(monkeypatch: pytest.Monk
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def scoped_toin(monkeypatch: pytest.MonkeyPatch, tmp_path):
|
||||
from headroom.telemetry.toin import TOINConfig, ToolIntelligenceNetwork
|
||||
|
||||
toin = ToolIntelligenceNetwork(TOINConfig(storage_path=str(tmp_path / "toin.json")))
|
||||
patterns = {
|
||||
"global|unknown|unknown|abc123": 8,
|
||||
"global|unknown|unknown|abc456": 16,
|
||||
"global|payg|claude|abc123": 24,
|
||||
"other-tenant|unknown|unknown|abc123": 32,
|
||||
"tenant/alpha?scope=one#fragment|unknown|unknown|abc123": 40,
|
||||
}
|
||||
toin.import_patterns(
|
||||
{
|
||||
"instance_id": "scoped-pattern-regression",
|
||||
"patterns": {
|
||||
key: {
|
||||
"tool_signature_hash": key.rsplit("|", 1)[-1],
|
||||
"sample_size": count,
|
||||
"total_compressions": count,
|
||||
"total_retrievals": 0,
|
||||
}
|
||||
for key, count in patterns.items()
|
||||
},
|
||||
}
|
||||
)
|
||||
monkeypatch.setattr("headroom.proxy.server.get_toin", lambda: toin)
|
||||
return patterns
|
||||
|
||||
|
||||
def test_listed_toin_identifiers_address_their_own_scoped_pattern(scoped_toin) -> None:
|
||||
from urllib.parse import quote
|
||||
|
||||
client = _loopback_client()
|
||||
listed = client.get("/v1/toin/patterns").json()
|
||||
assert {item["hash"] for item in listed} == set(scoped_toin)
|
||||
for item in listed:
|
||||
response = client.get(f"/v1/toin/pattern/{quote(item['hash'], safe='')}")
|
||||
assert response.status_code == 200
|
||||
assert response.json()["compressions"] == item["compressions"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("prefix", "status", "count"),
|
||||
[
|
||||
("global|unknown|unknown|abc", 409, None),
|
||||
("global|unknown|unknown|abc1", 200, 8),
|
||||
("missing-pattern", 404, None),
|
||||
],
|
||||
)
|
||||
def test_toin_prefix_lookup_is_unambiguous(scoped_toin, prefix, status, count) -> None:
|
||||
response = _loopback_client().get(f"/v1/toin/pattern/{prefix}")
|
||||
assert response.status_code == status
|
||||
if count is not None:
|
||||
assert response.json()["compressions"] == count
|
||||
|
||||
|
||||
# Mutating routes reachable from loopback. `require_loopback` cannot stop a
|
||||
# remote page from POSTing to a known 127.0.0.1 URL: a "simple" cross-origin
|
||||
# request (Content-Type: text/plain carrying JSON) skips preflight, and the
|
||||
|
||||
Reference in New Issue
Block a user