mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
feat/https-compose-caddy
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
2800932e18 |
fix(proxy): log the savings and metrics failures that currently vanish (#4084)
## Description Several savings and metrics paths swallow exceptions and log nothing, so when one fails, a customer-facing number can come out wrong with no record of why. This PR adds a log line on each of those paths. The numbers and the control flow are unchanged. Refs #4136 | Where | Before | Now | |---|---|---| | `tool_schema_savings_policy.reconcile_deferred_tokens` | A failure left the tool-search deferral credit as booked, so `/stats` over-reported. Nothing was logged. | Warns once per exception type, giving the booked token count and the fact that `/stats` is over-reported. Every occurrence is also logged at debug. | | `outcome.py` output-savings record/estimate | Booked 0 silently. | Debug, with the request id. | | `server.py` `/stats` `output_reduction` | The section went missing silently. | Debug. | | `server.py` `create_app` saved-settings apply | `settings_store.load()` already warns about an unreadable or corrupt file. Any other error while applying it was swallowed. | Warns with the settings file path and says to fix or remove that file and restart. Startup continues. | | `server.py` security gate, audit record | Debug only. | Warns with the method and path, and says the request is missing from the `headroom.audit` trail. | | `savings_tracker` pricing for an unpriced model | Raised `RuntimeError` only to reach the fallback, and logged nothing. | Returns the fallback directly and logs once per model at debug. The `except` blocks now see only unexpected errors, and log them at debug. | | `savings_tracker._resolve_litellm_model` probes | Two `except Exception: pass` blocks. | Debug. litellm raises a bare `Exception` for a model it does not know, so the catch cannot be narrower. | **No content at any level.** A tokenizer error can quote the client's tool schema, and a settings or pricing error can quote a credential. So every line this PR adds describes a failure with `describe_exception(exc)` from `headroom/log_safety.py`: the exception types and code locations of the cause chain, never the message and never a traceback. Model ids go through `safe_id`, and audit paths are capped the same way. The full text appears only when the operator sets `HEADROOM_DEBUG_DUMP=full`. **Bounded dedupe.** The reconcile guard (keyed by exception type) and the unpriced-model guard (keyed by model) use the shared `WarnOnce`. Each key is logged once, at most a fixed number of keys per hour, followed by one overflow notice. The unpriced-model guard is consulted only when DEBUG is enabled, so a model is not used up while its line is suppressed. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/tool_schema_savings_policy.py`: `_log_reconcile_failure` uses `WarnOnce(32, ...)` and `describe_exception`. - `headroom/proxy/outcome.py`, `headroom/proxy/server.py`, `headroom/proxy/savings_tracker.py`: add the lines in the table above, all through `describe_exception` and `safe_id`. `_log_unpriced_model_once` uses `WarnOnce`. - Tests: - `tests/test_tool_schema_savings_policy.py`: - a tokenizer error quoting a private-schema canary; - every captured DEBUG and WARNING record is rendered with `logging.Formatter()`, which includes `exc_info`, and none contains the canary; - the booked credit stays unchanged; - one warning per exception type, a debug line per failure. - `tests/test_savings_pricing_fallback_logging.py`: - a pricing error and a litellm probe error that quote an API-key canary, with the canary absent from every formatted record; - an unpriced model logs once; - nothing is used up while DEBUG is off. - `tests/test_saved_settings_apply_logging.py`: a settings-apply error that quotes an API-key canary through a real `create_app`. One warning names the file and the exception type, and no formatted record contains the canary. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # New regressions against the previous commit's product files ( |
||
|
|
68359c056c |
fix(proxy): repair tool-search history in place so signed thinking blocks keep their coordinates (#3457)
## Description
`strip_unsupported_tool_search_blocks()` repairs by removing the
unsupportable
`tool_search_tool_result` and its paired `server_tool_use`. When that
pair sits in an assistant
message that also carries **signed thinking blocks** — the normal shape
of Claude Code traffic with
extended thinking on — the removal moves every later block in that
message.
`thinking_block_fingerprint()` keys a thinking block by `(message_index,
block_index, json)`, so
`thinking_blocks_survived_mutation()` returns False and
`select_outbound_body()` forwards the
client's ORIGINAL bytes, discarding every mutation including the repair.
Upstream then 400s on the
exact reference the repair had already found, and because a resume
replays the same transcript the
session is wedged permanently.
Dropping a whole message (`if not kept: continue`) has the same effect
one level up: it shifts
`message_index` for every message after it.
Fix: repair in place. Swap each unsupportable block for a short constant
text block and never
remove a message, so every thinking block keeps its coordinates and its
bytes. Same reasoning and
same shape as the CCR sibling `strip_unsupported_ccr_retrieve_blocks`
right below it ("Neutralize
rather than drop"), which does not have this bug.
Also: dump the request on a **streaming** upstream error, not only on
the buffered path. Claude
Code streams every turn, so `HEADROOM_DEBUG_DUMP` previously captured
nothing for the most common
400s — that blindness is why this took three separate investigations to
pin down. Same gating as
the existing dumps (off by default, never in stateless mode, redacted
unless `=full`).
Closes #3456
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/helpers.py`: `strip_unsupported_tool_search_blocks()`
replaces unsupportable
blocks with a constant `_TOOL_SEARCH_PLACEHOLDER_BLOCK` text block
instead of deleting them; no
message is ever dropped. Return contract unchanged (`(messages, count)`,
original object by
identity when nothing changed), so the call site and its cache/prefix
behaviour are unchanged.
The placeholder is constant so the repaired prefix stays byte-stable
across turns.
- `headroom/proxy/handlers/anthropic.py`: log line now says "repaired
... (replaced with text in
place)" to match the behaviour.
- `headroom/proxy/handlers/streaming.py`: write the same diagnostic dump
the non-streaming handlers
write when an upstream response is >= 400.
- `tests/test_issue_746_tool_search.py`: three tests updated from drop-
to in-place semantics, plus
a new regression test asserting `thinking_block_fingerprint()` is
byte-identical across a repair.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_issue_746_tool_search.py tests/test_proxy/test_tool_search_repair_after_turn_hooks.py -q
63 passed, 1 warning in 3.09s
$ pytest tests -k "tool_search or thinking or body_forward or outbound" -q
208 passed, 26 skipped, 11637 deselected, 1 warning in 16.08s
$ ruff check .
All checks passed!
$ mypy headroom
Success: no issues found in 530 source files
```
## Real Behavior Proof
- Environment: proxy 0.38.0-dev (this branch on `main` @
|
||
|
|
51a3b01174 |
feat(security): pilot hardening — stateless guarantee, model pinning, CI security gate (#1515)
## Description
Engineering hardening derived from the Box vendor security assessment.
Each change turns a "No/Partial" questionnaire answer into a genuine
"Yes" by making the product safer — not by editing the form. The
throughline is Headroom's core promise to enterprise pilots: **it runs
inside the customer's environment and never persists or leaks their
data.** These changes make that provable.
Three themes: (1) a complete **stateless write guarantee** (a stateless
proxy writes nothing to the workspace during serving), (2)
**data-at-rest** protection (no cleartext prompts written on errors),
and (3) **supply-chain integrity** (all model downloads pinned;
SCA/SAST/secret-scanning in CI).
Closes # (no tracking issue)
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
> Note: two deliberate, reversible default changes (not breaking): the
upstream-error debug dump is now off by default
(`HEADROOM_DEBUG_DUMP=1`/`=full` to opt in), and model downloads are
pinned (`HEADROOM_HF_PIN=off` to bypass). All stateless plumbing is a
pure no-op when not stateless.
## Changes Made
- **Stateless writes** — savings tracker + ledger, TOIN (`toin.json`),
and the output-savings recorder now honor stateless (in-memory only);
persistent memory is disabled under stateless with a warning. Added a
process-wide flag `headroom.paths.process_is_stateless()` (also honors
`HEADROOM_STATELESS`).
- **Debug dump** — the Anthropic *and* OpenAI handlers wrote full
requests (cleartext prompts/tools/system) to
`~/.headroom/logs/debug_400/` on every ≥400, even stateless. Now OFF by
default, stateless-aware, with a redacted middle tier; helpers extracted
to `handlers/_debug_dump.py`.
- **Model pinning** — all model downloads pin an immutable commit SHA:
our repos, kompress, image router/SigLIP, the third-party Qdrant memory
embedder (centralized in `onnx_runtime`), and the fastembed relevance
model (via the `revision` kwarg fastembed forwards to
`snapshot_download`). `HEADROOM_HF_PIN=off` bypasses.
- **CI security gate** — new `security.yml`: dependency audit
(pip-audit, scoped to the CVE-free `[all]` set), CodeQL (Python +
JS/TS), and gitleaks secret scanning (binary, MIT-licensed; PR-diff
scoped). `.gitleaks.toml` allowlists SBOM/lockfiles.
- **Dependabot** — extended to Rust (cargo) and npm (TS SDK, plugins,
docs).
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ ruff check <10 changed source files>
All checks passed!
$ mypy <changed source files>
Success: no issues found in 7 source files # + handlers/server: no issues (annotation-unchecked notes only)
$ pytest tests/test_stateless_writers.py tests/test_stateless_toin.py \
tests/test_debug_dump_gating.py tests/test_hf_revision_pinning.py \
tests/test_proxy_savings_history.py tests/test_observability_metrics.py \
tests/test_toin.py tests/test_paths.py -q
================= 176 passed, 6 skipped, 2 warnings in 13.92s ==================
```
New tests (18): `tests/test_stateless_writers.py`,
`tests/test_stateless_toin.py`, `tests/test_debug_dump_gating.py`,
`tests/test_hf_revision_pinning.py`, plus stateless control assertions
in `tests/test_proxy_savings_history.py`. They include the non-stateless
control cases (savings/TOIN still persist) and a regression guard that
fails if any handler writes a debug dump without gating it.
## Real Behavior Proof
- **Environment:** macOS, Python 3.12 (`.venv`); CI on `ubuntu-latest`.
- **Exact command / steps:**
- Stateless guarantee: `SavingsRecorder(tmp/"output_savings.json")` +
`set_process_stateless(True)` → `flush()`; TOIN
`ToolIntelligenceNetwork(TOINConfig(storage_path=""))`;
`create_app(ProxyConfig(memory_enabled=True, stateless=True))`.
- Debug-dump gating: `_debug_dump_mode(SimpleNamespace(stateless=...))`
across env values.
- Model pinning: model SHAs fetched/verified against the live
HuggingFace API; `_resolve_revision` / `_pinned_revision` resolvers
tested.
- **Observed result:** under stateless, no `proxy_savings.json` /
`savings_events.jsonl` / `toin.json` / `output_savings.json` /
`memory.db` is created; `proxy.memory_handler is None`. With
`stateless=False` the control tests confirm each still persists. Debug
dump resolves to `off` by default and is forced off in stateless. CI:
dependency-audit, CodeQL (python + js/ts) pass; secret-scan now runs the
gitleaks binary.
- **Not tested:** `pip-audit` was not run on the local machine (broken
`ensurepip`); CI is the first real run (the committed all-extras grype
scan is clean). The fastembed download path is exercised by CI/runtime,
not in unit tests (the revision resolver is unit-tested).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Additional Notes
- **Concurrency:** `stateless` is a per-process config flag, never
per-request/per-session. Many sessions share one proxy's setting; a
stateless and a stateful proxy are separate OS processes with isolated
state. The one in-process edge (two proxies, different settings —
essentially tests) fails closed to in-memory, so a stateless proxy can
never leak.
- **Memory under stateless** is *disabled* (not in-RAM): the memory
subsystem is multi-component (SQLite + vector + markdown bridge) and a
partial in-RAM mode would be risky; ephemeral containers and
cross-session learning are contradictory. An ephemeral in-RAM memory
mode is a possible follow-up.
- **Docs/CHANGELOG** left unchecked: the two new env vars
(`HEADROOM_DEBUG_DUMP`, `HEADROOM_HF_PIN`) and the stateless behavior
changes are documented in code comments; happy to add user docs + a
CHANGELOG entry if preferred.
- CI deprecation warnings (Node 20, CodeQL Action v3) are GitHub-side
and out of scope here.
|