mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
feat(pipeline): say whose request an OutcomeSnapshot is (#4698)
## Description
`OutcomeSnapshot` is the one public seam a pipeline extension gets at
`OUTCOME_OBSERVED`, but it says nothing about *whose* request it
measured. An extension that wants to split what it observes (an
exporter, a chargeback job, a budget alert) cannot tell projects,
clients or conversations apart without patching core. This adds four
read-only fields and fills them from values `emit_request_outcome`
already has:
- `attribution`: `{dimension: label}`. For now `project` and `client`,
from the same values the savings tracker books, and only when known.
- `conversation_key`: the handler's key, already on `RequestOutcome`.
- `original_tokens`: the pre-compression size, on the same tokenizer as
`tokens_saved`.
- `output_tokens_saved`: the output-shaping estimate this request
booked.
`attribution` is frozen into a read-only `dict` subclass
(`FrozenLabels`) in `__post_init__`. The dataclass is already frozen,
but a frozen field can still hold a mutable dict, so without this one
extension could rewrite the labels the next one reads. A `dict` subclass
rather than a `MappingProxyType` keeps the snapshot hashable, picklable
and safe for `dataclasses.asdict`, `copy.deepcopy` and `json.dumps`, as
it was before.
It also adds `PipelineExtensionManager.add()`. A proxy extension
(`headroom.proxy_extension`) that also wants lifecycle events is
installed after the manager exists, so it cannot be passed to the
constructor. Adding the same object twice is a no-op.
This is PR 1 of the split proposed in #4676 (generic attribution
labels). It stands on its own, and the later PRs only widen what goes
into `attribution`.
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/pipeline.py`: `OutcomeSnapshot.attribution`,
`conversation_key`, `original_tokens`, `output_tokens_saved`, and
`PipelineExtensionManager.add()`. All new fields have defaults, so
existing constructors are unchanged.
- `headroom/proxy/outcome.py`: `emit_request_outcome` fills the new
fields.
- Tests: the fields and their immutability, `add()` (late attach, no
duplicates) in `tests/test_pipeline_outcome_stage.py`, and end-to-end
through `emit_request_outcome` in
`tests/test_proxy/test_conversation_savings.py`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ pytest tests/test_proxy tests/test_pipeline.py tests/test_pipeline_outcome_stage.py tests/test_5xx_accounting_all_providers.py
615 passed, 7 warnings in 62.72s # upstream/main at bb8d18a5: 608 passed
$ ruff check <changed files> && ruff format --check <changed files> # ruff 0.17.3
All checks passed!
4 files already formatted
$ mypy headroom
No errors in the changed files. The 12 errors reported are in ccr/mcp_server.py,
memory/mcp_server.py, memory/adapters/embedders.py and providers/model_metadata.py,
identical on upstream/main, from local mcp/library versions.
```
## Real Behavior Proof
- Environment: macOS, a Headroom proxy serving as a LAN gateway for
three machines.
- Exact command / steps: a superset of this change (the full #4676
proposal) has run there since 2026-11-06 with an `OUTCOME_OBSERVED`
extension reading these fields.
- Observed result: the extension reads per-request project, client and
conversation from the snapshot with no core patch.
- Not tested: this exact subset was not deployed on its own; it is
covered by the unit tests above.
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: no. The fields are new and additive,
and nothing reads them unless an extension is enabled.
- Kill switch / disable path: n/a (extensions are already opt-in)
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A
## Additional Notes
- No docs change here: the fields are documented on the dataclass.
User-facing attribution docs come with PR 2 of #4676.
- `attribution` uses a mapping rather than more scalar fields so that
later dimensions (host, team, …) need no further change to this
contract.
This commit is contained in:
+58
-2
@@ -5,13 +5,43 @@ from __future__ import annotations
|
||||
import importlib.metadata
|
||||
import logging
|
||||
import os
|
||||
from collections.abc import Iterable
|
||||
from collections.abc import Iterable, Mapping
|
||||
from dataclasses import dataclass, field
|
||||
from enum import Enum
|
||||
from typing import Any, Protocol
|
||||
from typing import Any, NoReturn, Protocol
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class FrozenLabels(dict[str, str]):
|
||||
"""A ``dict`` that refuses changes: what :attr:`OutcomeSnapshot.attribution` holds.
|
||||
|
||||
A subclass of ``dict`` rather than a ``MappingProxyType`` so the snapshot
|
||||
stays what it was before it carried labels: hashable, picklable, and safe
|
||||
for ``dataclasses.asdict``, ``copy.deepcopy`` and ``json.dumps``.
|
||||
"""
|
||||
|
||||
__slots__ = ()
|
||||
|
||||
def _read_only(self, *args: Any, **kwargs: Any) -> NoReturn:
|
||||
raise TypeError("attribution labels are read-only")
|
||||
|
||||
__setitem__ = __delitem__ = __ior__ = _read_only
|
||||
clear = pop = popitem = setdefault = update = _read_only
|
||||
|
||||
def __hash__(self) -> int: # type: ignore[override]
|
||||
return hash(frozenset(self.items()))
|
||||
|
||||
def __reduce__(self) -> tuple[Any, ...]:
|
||||
return (type(self), (dict(self),))
|
||||
|
||||
def __copy__(self) -> FrozenLabels:
|
||||
return self
|
||||
|
||||
def __deepcopy__(self, memo: dict[int, Any]) -> FrozenLabels:
|
||||
return self
|
||||
|
||||
|
||||
ENTRY_POINT_GROUP = "headroom.pipeline_extension"
|
||||
ENV_VAR = "HEADROOM_PIPELINE_EXTENSIONS"
|
||||
|
||||
@@ -86,6 +116,21 @@ class OutcomeSnapshot:
|
||||
#: is re-sent and recompressed every turn, and ``tokens_saved -
|
||||
#: novel_tokens_saved`` is the re-sent history removed again.
|
||||
novel_tokens_saved: int = 0
|
||||
#: Whose request this was: ``{dimension: label}``, for now ``project`` and
|
||||
#: ``client`` when known. Self-reported labels, not authenticated
|
||||
#: identity. Read-only.
|
||||
attribution: Mapping[str, str] = field(default_factory=dict)
|
||||
#: Stable across every turn of one conversation, when the handler knows it.
|
||||
conversation_key: str | None = None
|
||||
#: Pre-compression input size on the same tokenizer as ``tokens_saved``.
|
||||
original_tokens: int = 0
|
||||
#: Estimated output tokens avoided by output shaping on this request.
|
||||
output_tokens_saved: int = 0
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
# Frozen fields can still hold a mutable dict; freeze it so an
|
||||
# extension cannot rewrite the labels another extension will read.
|
||||
object.__setattr__(self, "attribution", FrozenLabels(self.attribution or {}))
|
||||
|
||||
@property
|
||||
def truncated(self) -> bool:
|
||||
@@ -247,6 +292,17 @@ class PipelineExtensionManager:
|
||||
def enabled(self) -> bool:
|
||||
return bool(self._extensions)
|
||||
|
||||
def add(self, extension: Any) -> None:
|
||||
"""Attach an extension after construction.
|
||||
|
||||
For a proxy extension (``headroom.proxy_extension``) that also needs
|
||||
lifecycle events: it is installed after the proxy and its manager
|
||||
exist, so it cannot be passed to the constructor. Adding the same
|
||||
object twice is a no-op.
|
||||
"""
|
||||
if all(existing is not extension for existing in self._extensions):
|
||||
self._extensions.append(extension)
|
||||
|
||||
def emit(
|
||||
self,
|
||||
stage: PipelineStage,
|
||||
|
||||
@@ -665,6 +665,14 @@ async def emit_request_outcome(handler: Any, outcome: RequestOutcome) -> None:
|
||||
transforms_applied=tuple(str(t) for t in (outcome.transforms_applied or ())),
|
||||
tokens_saved=outcome.tokens_saved,
|
||||
novel_tokens_saved=novel_tokens_saved,
|
||||
attribution={
|
||||
dimension: label
|
||||
for dimension, label in (("project", project), ("client", outcome.client))
|
||||
if label
|
||||
},
|
||||
conversation_key=outcome.conversation_key,
|
||||
original_tokens=outcome.original_tokens,
|
||||
output_tokens_saved=output_tokens_saved_est,
|
||||
),
|
||||
)
|
||||
except Exception: # noqa: BLE001 - telemetry must never break a response
|
||||
|
||||
@@ -99,6 +99,69 @@ class TestOutcomeSnapshot:
|
||||
event = mgr.emit(PipelineStage.OUTCOME_OBSERVED, operation="proxy.outcome", outcome=snap)
|
||||
assert event.outcome is snap
|
||||
|
||||
def test_snapshot_carries_attribution_and_conversation(self):
|
||||
snap = OutcomeSnapshot(
|
||||
attribution={"project": "web", "client": "codex"},
|
||||
conversation_key="conv-a",
|
||||
original_tokens=1_000,
|
||||
output_tokens_saved=40,
|
||||
)
|
||||
assert dict(snap.attribution) == {"project": "web", "client": "codex"}
|
||||
assert (snap.conversation_key, snap.original_tokens, snap.output_tokens_saved) == (
|
||||
"conv-a",
|
||||
1_000,
|
||||
40,
|
||||
)
|
||||
|
||||
@pytest.mark.parametrize("labels", [{}, {"project": "web"}])
|
||||
def test_snapshot_still_serializes_copies_and_hashes(self, labels):
|
||||
"""Carrying labels must not cost the snapshot what a frozen dataclass of
|
||||
scalars always had: asdict, deepcopy, pickle, JSON and hashing."""
|
||||
import copy
|
||||
import dataclasses
|
||||
import json
|
||||
import pickle
|
||||
|
||||
snap = OutcomeSnapshot(request_id="r1", attribution=labels)
|
||||
assert dataclasses.asdict(snap)["attribution"] == labels
|
||||
assert json.loads(json.dumps(dataclasses.asdict(snap)))["attribution"] == labels
|
||||
assert copy.deepcopy(snap) == snap
|
||||
restored = pickle.loads(pickle.dumps(snap))
|
||||
assert restored == snap and hash(restored) == hash(snap)
|
||||
with pytest.raises(TypeError):
|
||||
restored.attribution["project"] = "other" # type: ignore[index]
|
||||
|
||||
def test_attribution_labels_are_read_only(self):
|
||||
"""Frozen fields can still hold a mutable dict; one extension must not
|
||||
be able to rewrite the labels the next one reads."""
|
||||
labels = {"project": "web"}
|
||||
snap = OutcomeSnapshot(attribution=labels)
|
||||
with pytest.raises(TypeError):
|
||||
snap.attribution["project"] = "other" # type: ignore[index]
|
||||
labels["project"] = "changed-after"
|
||||
assert snap.attribution["project"] == "web"
|
||||
|
||||
|
||||
class TestAdd:
|
||||
def test_an_extension_added_later_receives_events(self):
|
||||
mgr = PipelineExtensionManager(extensions=[], discover=False)
|
||||
rec = _Recorder()
|
||||
mgr.add(rec)
|
||||
assert mgr.enabled
|
||||
mgr.emit(
|
||||
PipelineStage.OUTCOME_OBSERVED, operation="proxy.outcome", outcome=OutcomeSnapshot()
|
||||
)
|
||||
assert rec.stages() == [PipelineStage.OUTCOME_OBSERVED]
|
||||
|
||||
def test_adding_the_same_extension_twice_is_a_no_op(self):
|
||||
rec = _Recorder()
|
||||
mgr = PipelineExtensionManager(extensions=[rec], discover=False)
|
||||
mgr.add(rec)
|
||||
mgr.emit(
|
||||
PipelineStage.OUTCOME_OBSERVED, operation="proxy.outcome", outcome=OutcomeSnapshot()
|
||||
)
|
||||
assert len(rec.seen) == 1
|
||||
|
||||
|
||||
class TestDiscoveryGate:
|
||||
"""Installing a package must not silently start rewriting live requests."""
|
||||
|
||||
@@ -393,3 +393,27 @@ async def test_pipeline_extensions_read_both_figures_from_the_snapshot() -> None
|
||||
(62_806, 62_806),
|
||||
(76_020, 13_214),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_the_snapshot_says_whose_request_it_was() -> None:
|
||||
"""Project, client, conversation and the pre-compression size reach the
|
||||
extension, so it can attribute what it measures without patching core."""
|
||||
from headroom.pipeline import PipelineExtensionManager, PipelineStage
|
||||
|
||||
seen: list[Any] = []
|
||||
|
||||
class _Recorder:
|
||||
def on_pipeline_event(self, event: Any) -> None:
|
||||
if event.stage is PipelineStage.OUTCOME_OBSERVED:
|
||||
seen.append(event.outcome)
|
||||
|
||||
handler = _Handler()
|
||||
handler.pipeline_extensions = PipelineExtensionManager(extensions=[_Recorder()], discover=False)
|
||||
await emit_request_outcome(handler, _responses_outcome(project="web", client="codex"))
|
||||
await emit_request_outcome(handler, _responses_outcome(request_id="req-2"))
|
||||
|
||||
assert dict(seen[0].attribution) == {"project": "web", "client": "codex"}
|
||||
assert seen[0].conversation_key == "conv-a"
|
||||
assert seen[0].original_tokens == 156_260
|
||||
assert dict(seen[1].attribution) == {}
|
||||
|
||||
Reference in New Issue
Block a user