feat(pipeline): say whose request an OutcomeSnapshot is (#4698)

## Description

`OutcomeSnapshot` is the one public seam a pipeline extension gets at
`OUTCOME_OBSERVED`, but it says nothing about *whose* request it
measured. An extension that wants to split what it observes (an
exporter, a chargeback job, a budget alert) cannot tell projects,
clients or conversations apart without patching core. This adds four
read-only fields and fills them from values `emit_request_outcome`
already has:

- `attribution`: `{dimension: label}`. For now `project` and `client`,
from the same values the savings tracker books, and only when known.
- `conversation_key`: the handler's key, already on `RequestOutcome`.
- `original_tokens`: the pre-compression size, on the same tokenizer as
`tokens_saved`.
- `output_tokens_saved`: the output-shaping estimate this request
booked.

`attribution` is frozen into a read-only `dict` subclass
(`FrozenLabels`) in `__post_init__`. The dataclass is already frozen,
but a frozen field can still hold a mutable dict, so without this one
extension could rewrite the labels the next one reads. A `dict` subclass
rather than a `MappingProxyType` keeps the snapshot hashable, picklable
and safe for `dataclasses.asdict`, `copy.deepcopy` and `json.dumps`, as
it was before.

It also adds `PipelineExtensionManager.add()`. A proxy extension
(`headroom.proxy_extension`) that also wants lifecycle events is
installed after the manager exists, so it cannot be passed to the
constructor. Adding the same object twice is a no-op.

This is PR 1 of the split proposed in #4676 (generic attribution
labels). It stands on its own, and the later PRs only widen what goes
into `attribution`.

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/pipeline.py`: `OutcomeSnapshot.attribution`,
`conversation_key`, `original_tokens`, `output_tokens_saved`, and
`PipelineExtensionManager.add()`. All new fields have defaults, so
existing constructors are unchanged.
- `headroom/proxy/outcome.py`: `emit_request_outcome` fills the new
fields.
- Tests: the fields and their immutability, `add()` (late attach, no
duplicates) in `tests/test_pipeline_outcome_stage.py`, and end-to-end
through `emit_request_outcome` in
`tests/test_proxy/test_conversation_savings.py`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ pytest tests/test_proxy tests/test_pipeline.py tests/test_pipeline_outcome_stage.py tests/test_5xx_accounting_all_providers.py
615 passed, 7 warnings in 62.72s          # upstream/main at bb8d18a5: 608 passed

$ ruff check <changed files> && ruff format --check <changed files>     # ruff 0.17.3
All checks passed!
4 files already formatted

$ mypy headroom
No errors in the changed files. The 12 errors reported are in ccr/mcp_server.py,
memory/mcp_server.py, memory/adapters/embedders.py and providers/model_metadata.py,
identical on upstream/main, from local mcp/library versions.
```

## Real Behavior Proof

- Environment: macOS, a Headroom proxy serving as a LAN gateway for
three machines.
- Exact command / steps: a superset of this change (the full #4676
proposal) has run there since 2026-11-06 with an `OUTCOME_OBSERVED`
extension reading these fields.
- Observed result: the extension reads per-request project, client and
conversation from the snapshot with no core patch.
- Not tested: this exact subset was not deployed on its own; it is
covered by the unit tests above.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: no. The fields are new and additive,
and nothing reads them unless an extension is enabled.
- Kill switch / disable path: n/a (extensions are already opt-in)
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- No docs change here: the fields are documented on the dataclass.
User-facing attribution docs come with PR 2 of #4676.
- `attribution` uses a mapping rather than more scalar fields so that
later dimensions (host, team, …) need no further change to this
contract.
This commit is contained in:
M Barry
2026-11-09 15:15:05 -06:00
committed by GitHub
parent 94a99ed20f
commit a2bcbd101b
4 changed files with 153 additions and 2 deletions
+58 -2
View File
@@ -5,13 +5,43 @@ from __future__ import annotations
import importlib.metadata
import logging
import os
from collections.abc import Iterable
from collections.abc import Iterable, Mapping
from dataclasses import dataclass, field
from enum import Enum
from typing import Any, Protocol
from typing import Any, NoReturn, Protocol
log = logging.getLogger(__name__)
class FrozenLabels(dict[str, str]):
"""A ``dict`` that refuses changes: what :attr:`OutcomeSnapshot.attribution` holds.
A subclass of ``dict`` rather than a ``MappingProxyType`` so the snapshot
stays what it was before it carried labels: hashable, picklable, and safe
for ``dataclasses.asdict``, ``copy.deepcopy`` and ``json.dumps``.
"""
__slots__ = ()
def _read_only(self, *args: Any, **kwargs: Any) -> NoReturn:
raise TypeError("attribution labels are read-only")
__setitem__ = __delitem__ = __ior__ = _read_only
clear = pop = popitem = setdefault = update = _read_only
def __hash__(self) -> int: # type: ignore[override]
return hash(frozenset(self.items()))
def __reduce__(self) -> tuple[Any, ...]:
return (type(self), (dict(self),))
def __copy__(self) -> FrozenLabels:
return self
def __deepcopy__(self, memo: dict[int, Any]) -> FrozenLabels:
return self
ENTRY_POINT_GROUP = "headroom.pipeline_extension"
ENV_VAR = "HEADROOM_PIPELINE_EXTENSIONS"
@@ -86,6 +116,21 @@ class OutcomeSnapshot:
#: is re-sent and recompressed every turn, and ``tokens_saved -
#: novel_tokens_saved`` is the re-sent history removed again.
novel_tokens_saved: int = 0
#: Whose request this was: ``{dimension: label}``, for now ``project`` and
#: ``client`` when known. Self-reported labels, not authenticated
#: identity. Read-only.
attribution: Mapping[str, str] = field(default_factory=dict)
#: Stable across every turn of one conversation, when the handler knows it.
conversation_key: str | None = None
#: Pre-compression input size on the same tokenizer as ``tokens_saved``.
original_tokens: int = 0
#: Estimated output tokens avoided by output shaping on this request.
output_tokens_saved: int = 0
def __post_init__(self) -> None:
# Frozen fields can still hold a mutable dict; freeze it so an
# extension cannot rewrite the labels another extension will read.
object.__setattr__(self, "attribution", FrozenLabels(self.attribution or {}))
@property
def truncated(self) -> bool:
@@ -247,6 +292,17 @@ class PipelineExtensionManager:
def enabled(self) -> bool:
return bool(self._extensions)
def add(self, extension: Any) -> None:
"""Attach an extension after construction.
For a proxy extension (``headroom.proxy_extension``) that also needs
lifecycle events: it is installed after the proxy and its manager
exist, so it cannot be passed to the constructor. Adding the same
object twice is a no-op.
"""
if all(existing is not extension for existing in self._extensions):
self._extensions.append(extension)
def emit(
self,
stage: PipelineStage,
+8
View File
@@ -665,6 +665,14 @@ async def emit_request_outcome(handler: Any, outcome: RequestOutcome) -> None:
transforms_applied=tuple(str(t) for t in (outcome.transforms_applied or ())),
tokens_saved=outcome.tokens_saved,
novel_tokens_saved=novel_tokens_saved,
attribution={
dimension: label
for dimension, label in (("project", project), ("client", outcome.client))
if label
},
conversation_key=outcome.conversation_key,
original_tokens=outcome.original_tokens,
output_tokens_saved=output_tokens_saved_est,
),
)
except Exception: # noqa: BLE001 - telemetry must never break a response
+63
View File
@@ -99,6 +99,69 @@ class TestOutcomeSnapshot:
event = mgr.emit(PipelineStage.OUTCOME_OBSERVED, operation="proxy.outcome", outcome=snap)
assert event.outcome is snap
def test_snapshot_carries_attribution_and_conversation(self):
snap = OutcomeSnapshot(
attribution={"project": "web", "client": "codex"},
conversation_key="conv-a",
original_tokens=1_000,
output_tokens_saved=40,
)
assert dict(snap.attribution) == {"project": "web", "client": "codex"}
assert (snap.conversation_key, snap.original_tokens, snap.output_tokens_saved) == (
"conv-a",
1_000,
40,
)
@pytest.mark.parametrize("labels", [{}, {"project": "web"}])
def test_snapshot_still_serializes_copies_and_hashes(self, labels):
"""Carrying labels must not cost the snapshot what a frozen dataclass of
scalars always had: asdict, deepcopy, pickle, JSON and hashing."""
import copy
import dataclasses
import json
import pickle
snap = OutcomeSnapshot(request_id="r1", attribution=labels)
assert dataclasses.asdict(snap)["attribution"] == labels
assert json.loads(json.dumps(dataclasses.asdict(snap)))["attribution"] == labels
assert copy.deepcopy(snap) == snap
restored = pickle.loads(pickle.dumps(snap))
assert restored == snap and hash(restored) == hash(snap)
with pytest.raises(TypeError):
restored.attribution["project"] = "other" # type: ignore[index]
def test_attribution_labels_are_read_only(self):
"""Frozen fields can still hold a mutable dict; one extension must not
be able to rewrite the labels the next one reads."""
labels = {"project": "web"}
snap = OutcomeSnapshot(attribution=labels)
with pytest.raises(TypeError):
snap.attribution["project"] = "other" # type: ignore[index]
labels["project"] = "changed-after"
assert snap.attribution["project"] == "web"
class TestAdd:
def test_an_extension_added_later_receives_events(self):
mgr = PipelineExtensionManager(extensions=[], discover=False)
rec = _Recorder()
mgr.add(rec)
assert mgr.enabled
mgr.emit(
PipelineStage.OUTCOME_OBSERVED, operation="proxy.outcome", outcome=OutcomeSnapshot()
)
assert rec.stages() == [PipelineStage.OUTCOME_OBSERVED]
def test_adding_the_same_extension_twice_is_a_no_op(self):
rec = _Recorder()
mgr = PipelineExtensionManager(extensions=[rec], discover=False)
mgr.add(rec)
mgr.emit(
PipelineStage.OUTCOME_OBSERVED, operation="proxy.outcome", outcome=OutcomeSnapshot()
)
assert len(rec.seen) == 1
class TestDiscoveryGate:
"""Installing a package must not silently start rewriting live requests."""
@@ -393,3 +393,27 @@ async def test_pipeline_extensions_read_both_figures_from_the_snapshot() -> None
(62_806, 62_806),
(76_020, 13_214),
]
@pytest.mark.asyncio
async def test_the_snapshot_says_whose_request_it_was() -> None:
"""Project, client, conversation and the pre-compression size reach the
extension, so it can attribute what it measures without patching core."""
from headroom.pipeline import PipelineExtensionManager, PipelineStage
seen: list[Any] = []
class _Recorder:
def on_pipeline_event(self, event: Any) -> None:
if event.stage is PipelineStage.OUTCOME_OBSERVED:
seen.append(event.outcome)
handler = _Handler()
handler.pipeline_extensions = PipelineExtensionManager(extensions=[_Recorder()], discover=False)
await emit_request_outcome(handler, _responses_outcome(project="web", client="codex"))
await emit_request_outcome(handler, _responses_outcome(request_id="req-2"))
assert dict(seen[0].attribution) == {"project": "web", "client": "codex"}
assert seen[0].conversation_key == "conv-a"
assert seen[0].original_tokens == 156_260
assert dict(seen[1].attribution) == {}