Commit Graph
5 Commits
Author SHA1 Message Date
Tejas ChopraandClaude Opus 5.5 e4a1ec19be refactor(transforms): deprecate unused compress_with_stats, extract_batch and align_for_cache (#4098)
## Description

`DiffCompressor.compress_with_stats`, `HTMLExtractor.extract_batch` and
`cache_aligner.align_for_cache` are public. The first two are methods on
classes exported from `headroom.transforms`, and the third's docstring
called it a stable public API. None is called anywhere in Headroom, the
plugins or the consumer repos; only tests call them. Each now emits a
`DeprecationWarning` that names the replacement.

Refs #4136

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- `DiffCompressor.compress_with_stats` warns and points to
`DiffCompressor.compress`. Unlike `compress()`, it never wrote its CCR
entry to the compression store, so a retrieval marker in its output
could dangle. The docstring now says so.
- `HTMLExtractor.extract_batch` warns and points to calling
`HTMLExtractor.extract` per page.
- `align_for_cache` warns and points to
`CacheAligner(config).apply(messages, tokenizer)` and the
`stable_prefix_hash:` marker.
- The existing tests now assert the warnings. A new test checks that
`align_for_cache` warns at the caller and still returns the messages and
the hash.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ pytest tests/test_cache_aligner_detector_only.py tests/test_cache_aligner_prefix_stability.py tests/test_transforms/test_html_extractor.py tests/test_transforms/test_diff_compressor_rust_parity.py tests/test_transforms/test_diff_compressor.py
118 passed in 3.87s
$ ruff check / format: All checks passed;  mypy: Success
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, repo `.venv`, this branch's worktree
with the prebuilt `headroom._core`. Temp `HOME`,
`HEADROOM_WORKSPACE_DIR` and `CODEX_HOME`; nothing touched the real
home.
- Exact command / steps: (1) shared smoke harness: a real proxy from
this checkout against a local fake upstream, with one Anthropic and one
OpenAI request carrying a compressible tool result (`smoke.sh
<worktree>`); (2) `python -W always -c 'align_for_cache(...);
DiffCompressor().compress_with_stats(...)'`.
- Observed result: The proxy serves both requests (200) with the same
savings as main. Each deprecated call emits one DeprecationWarning
attributed to the caller and still returns its result. Output below.
- Not tested: Third-party callers; none is known.

```text
branch: cleanup/transforms-deprecate-unused-methods @ a3e57303
health: 200
POST /v1/messages        -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines ($SMOKE/ws/logs/proxy-51205.log):
   2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes.
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.
(smoke dir: $SMOKE)
```

```text
$ python -W always -c "align_for_cache([...]); DiffCompressor().compress_with_stats(...)"
<string>:4: DeprecationWarning: align_for_cache is deprecated and will be removed in a future release; use CacheAligner(config).apply(messages, tokenizer) and read the stable_prefix_hash: marker from markers_inserted.
<string>:5: DeprecationWarning: DiffCompressor.compress_with_stats is deprecated and will be removed in a future release; use DiffCompressor.compress instead.
ok
```

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Not applicable; no rollout-managed behaviour
is touched.
- Stable/default behavior changed: Only a DeprecationWarning when the
deprecated methods are called.
- Kill switch / disable path: Not applicable; the warning is a standard
DeprecationWarning and can be filtered with `warnings`.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this single commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable.

## Additional Notes

Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review.

Documentation changes are not in this PR. They are recorded in the
cleanup docs ledger for the single follow-up docs PR.

Deprecated (not removed): `DiffCompressor.compress_with_stats`,
`HTMLExtractor.extract_batch`, `cache_aligner.align_for_cache`. The
warnings fire on every call, which is standard. Python's default warning
filters show each one once per call site.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 09:03:03 -07:00
chopratejas be5d9f7baa fix(tests): update CacheAligner detector-only tests for F2.2 5-field CompressionPolicy
The two F2.1 fixtures constructed CompressionPolicy with only 2
fields; F2.2 added three required tuning fields (volatile_token_threshold,
max_lossy_ratio, toin_read_only). Updated subscription-disabled fixture to
use Subscription defaults (32 / 0.25 / True) and the PAYG-enabled fixture
to use PAYG defaults (128 / 0.45 / False), mirroring policy_for_mode().

Prefer the fixture update over default-valued fields in the dataclass so
the F2.2 parity-test invariant (every CompressionPolicy is fully specified
per-mode) stays load-bearing.
2026-05-06 14:39:39 -07:00
chopratejas de8e245990 fix(transforms): Python parity port of CompressionPolicy + cache_aligner gate (F2.1 c4/5)
Phase F2.1, commit 4 of 5 (consolidated from the 6-commit plan after
finding the Rust dispatcher gate is a no-op in F2.1 — Subscription
still gets live-zone compression by design, so the only behaviour
change is on the Python detector side).

What lands:

- New `headroom/transforms/compression_policy.py` — hand-mirror of
  `headroom_core::compression_policy::CompressionPolicy`. Two fields,
  `policy_for_mode(AuthMode)` constructor, `policy_default_payg()`
  helper for the enforcement-flag-off path. Source of truth is the
  Rust crate; a parity test guards against silent drift.

- `CacheAligner.should_apply` now reads `kwargs["compression_policy"]`
  and returns `False` when `policy.cache_aligner_enabled is False`.
  This is THE behaviour change for F2.1: subscription requests stop
  triggering volatility warnings and the per-pipeline-instance
  `_previous_prefix_hash` is no longer updated by them.

  Hidden state caveat: the hash field is per-pipeline-instance, not
  per-request. Clearing it on skip would race with concurrent PAYG
  requests on the same pipeline, so we don't. The behaviour is "skip
  silently" — exactly what cache-stability-sensitive callers want.

- 6 new tests in `tests/test_compression_policy.py`:
    - per-mode field assertions (mirror Rust unit tests)
    - `oauth_matches_payg_today` canary for F2.2 divergence
    - immutability check (`@dataclass(frozen=True)`)
    - field-set parity guard against Rust + Python drift
- 2 new tests in `tests/test_cache_aligner_detector_only.py`:
    - subscription policy short-circuits should_apply
    - PAYG policy does NOT short-circuit (sanity)

Note: this commit does NOT yet plumb the policy from the proxy
handlers into `pipeline.apply(...)` kwargs. That happens in c5/5
alongside the flag default flip. Until c5/5 lands, the gate is
reachable but unfired — `kwargs["compression_policy"]` is absent
from every production call site, so `policy.get(...)` returns None
and current behaviour is preserved bit-for-bit.

Verified: 28/28 affected Python tests pass.
2026-05-05 16:49:50 -07:00
chopratejas cf5a71571c fix(security): allowlist GitGuardian-flagged test fixtures
GitGuardian flagged two strings on PR #350 as leaked secrets. Both are
synthetic fixtures, NOT real credentials:

1. tests/test_cache_aligner_detector_only.py:215 — the canonical
   `eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxIn0.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c`
   JWT (header `{"alg":"HS256"}`, payload `{"sub":"1"}`) used to verify
   our `detect_volatile_content` recognises JWT-shaped strings.

2. tests/test_realignment_live_multi_turn.py:1091 — Anthropic-shaped
   tokens whose payloads literally contain "fixture"
   (`sk-ant-api03-payg-fixture`, `sk-ant-oat01-oauth-fixture`,
   `sk-ant-api03-payg-bearer-fixture`). Used to assert the auth-mode
   classifier routes PAYG / OAuth headers correctly. No live API call
   is ever made with these tokens — the test only inspects header
   shape.

Two-layer remediation:

* `.gitguardian.yaml` (new) — explicit allowlist with the literal
  match strings, each tagged with the file it lives in and the
  rationale. Anything else GG flags should be treated as a real
  incident; this file is the audit trail.
* Inline `# ggignore` + `# noqa: S105` comments on each fixture line
  so a reviewer reading the test in isolation sees the intent without
  having to cross-reference the config.

Per-feedback memory: secrets are routed via `.env`; the user's keys
were never in chat or version control. These rows document the
classifier-sweep false positive without weakening the detection rule.
2026-05-02 18:33:22 -07:00
chopratejas 704fb2f19d fix: A2 — system prompt immutable; memory routes to live-zone tail; cache_aligner detector-only
P0-1: Delete `_inject_system_context` from `proxy/server.py`. Memory
context now routes exclusively to the first text block of the latest
non-frozen user message via `_append_context_to_latest_non_frozen_user_turn`
(promoted to the canonical default in handlers/anthropic.py). Mirror
applied to OpenAI Responses API at handlers/openai.py: `body["instructions"]`
is no longer mutated; memory context appends to the latest user item in
`body["input"]`.

P2-23: Replace `headroom/transforms/cache_aligner.py` with a detector-only
implementation. The legacy rewrite path (~400 LOC) is removed. The volatile-
content detector uses no regex — UUIDs via `uuid.UUID`, ISO 8601 via
`datetime.fromisoformat`, JWT shape via base64url segment-count check, hex
hashes via length + `int(token, 16)` validation. Volatile findings surface
through `cache_metrics`/`warnings`/`logger.warning`; the prompt is never
mutated.

Configurability: new env var `HEADROOM_MEMORY_INJECTION_MODE` with values
`live_zone_tail` (default) and `disabled`. No `system_prompt` value — that
path is permanently retired.

Structured logs: every memory injection emits `event=memory_injection`
with `decision`, `bytes_injected`, `query_hash` (BLAKE2b, never raw query),
`session_id`, `request_id`. Auth is never logged.

Tests:
- Add `tests/test_proxy_system_prompt_immutable.py` (7 tests).
- Add `tests/test_cache_aligner_detector_only.py` (20 tests).
- Replace `tests/test_transforms/test_cache_aligner.py` (rewrite-path
  tests, 58 cases) with detector-only behavior.
- Update `tests/test_acceptance.py::TestDateTrap` to pin the new
  detector-only contract.

Acceptance:
- `git grep -n "_inject_system_context\|_inject_to_system_or_instructions" headroom/`
  returns nothing.
- `git grep -n "import re\|from re import" headroom/transforms/cache_aligner.py`
  returns nothing.
- Targeted suite (`test_proxy_system_prompt_immutable.py`,
  `test_cache_aligner_detector_only.py`, `test_proxy_anthropic_cache_stability.py`,
  `test_acceptance.py::TestDateTrap`, `test_memory*.py`, `test_cli/`) green.
2026-05-02 08:34:34 -07:00