fix(cache): default CCR payload previews off; harden the log when they are on (#3728)

## A-6 — tool-result content was written to the runtime log by default
(S2)

### RELEASE NOTE — a default changes, and operators lose log content
they may rely on

`HEADROOM_LOG_PAYLOAD_PREVIEW` **now defaults to off.** Operator-visible
impact:

- Each `event=headroom_retrieve` line in
`~/.headroom/logs/proxy-<port>.log` now carries
`"payload_preview":""` and `"payload_preview_chars":0` instead of up to
4096 characters
of the retrieved payload. `payload_chars` (the byte count) is unchanged,
so anything
measuring retrieval size still works; anything grepping the log for
retrieved *content*
  will find nothing until the operator opts in.
- The opt-in is now **explicit and positive**:
`HEADROOM_LOG_PAYLOAD_PREVIEW=1|true|yes|on`.
Previously *any* value other than `0/false/no/off` — including a typo —
meant "on".
- **On Linux and macOS, every Headroom log file is now created `0600`**
(owner-only) — the
runtime log `~/.headroom/logs/proxy-<port>.log`, each of its rotated
backups
(`proxy-8000.log.1`, …), and the `--log-file` JSONL request log. This is
**unconditional**:
it does not depend on `HEADROOM_LOG_PAYLOAD_PREVIEW`, `--log-messages`,
or any other
switch. **A log shipper, sidecar, or collector that reads these files as
a different user
or group will stop being able to read them.** Run the collector as the
same user, or copy
the file out with a privileged job. On Windows nothing changes — see the
Windows note
  below.
- If the runtime log path already exists as a **symlink**, Headroom now
refuses to write the
log at all (with a warning naming the path) instead of writing through
it. An operator who
deliberately symlinks `~/.headroom/logs/proxy-<port>.log` onto another
volume must replace
  it with a real path or point `HEADROOM_WORKSPACE_DIR` at that volume.

### The bug

`_payload_preview_enabled()` returned `True` for an unset variable, so a
stock proxy wrote
up to 4 KB of the retrieved tool-result payload into the always-on
runtime log on every CCR
retrieval. That payload is verbatim tool output: source code, customer
data, and any
credential the three redaction regexes (`*_KEY|TOKEN|SECRET|...`,
`Bearer`/`Basic`, `sk-…`)
do not happen to match. The file was created at the process umask —
`0644` on a normal
developer machine.

**The claim held**, at `headroom/cache/compression_store.py:59`/`:116`
on current `main`
(line numbers unchanged since `89a58fd1`). Demonstrated by the
fail-before run below, where
the new test found the string `ssn=123-45-6789` sitting in a real log
file, mode `0644`.

### The fix

- `headroom/cache/compression_store.py` — `_payload_preview_enabled()`
defaults to `False`
and now requires an affirmative `1/true/yes/on`. The redaction pass is
untouched; it is
now a second line of defence behind an explicit opt-in rather than the
only one.
- `headroom/fileperms.py` (new) — owner-only file creation, and an
explicit
`OWNER_ONLY_SUPPORTED` flag recording *where the mode bits actually
decide read access*.
- `headroom/proxy/helpers.py` — `_OwnerOnlyRotatingFileHandler`, now
used by
`_setup_file_logging` **unconditionally**. `logging` opens the stream
itself at
construction *and on every rollover* with no mode hook, so `_open` is
replaced outright:
the file is opened `O_CREAT|O_APPEND|O_WRONLY|O_NOFOLLOW` at `0600` and
tightened with
`fchmod` on that same descriptor (so an existing `0644` log is fixed,
and the mode cannot
land on a different inode). `rotate()` re-applies the mode to each
backup, and backups
  left `0644` by an older build are tightened when the handler opens.
- `headroom/proxy/request_logger.py` — the `--log-file` JSONL, which
carries whole request
  and response bodies under `--log-messages`, is created the same way.
- `docs/content/docs/proxy.mdx` — the variable was entirely
undocumented; it now has a row
in **Observability**, plus a **Who can read the logs** subsection
stating the POSIX
  guarantee and the Windows non-guarantee in as many words.

### Fail-before / pass-after

Method: commit the work, `git checkout origin/main --
headroom/cache/compression_store.py
headroom/proxy/helpers.py` keeping the new tests, run, then `git
checkout HEAD -- <those
files>`.

**Before (sources reverted to `origin/main`, new tests kept):**

```
__________ test_runtime_log_holds_no_payload_text_at_default_settings __________
tests/test_cache_breakpoint_diagnostics.py:246: in test_runtime_log_holds_no_payload_text_at_default_settings
    assert secret not in text
E   assert 'BEGIN-CUSTO...USTOMER-DATA' not in '2026-09-23 ...MER-DATA"}\n'
E     'BEGIN-CUSTOMER-DAT...s END-CUSTOMER-DATA' is contained here:
E       preview":"BEGIN-CUSTOMER-DATA ssn=123-45-6789 def sekrit(): pass END-CUSTOMER-DATA"}

_____________ test_runtime_log_is_owner_only_when_preview_enabled ______________
tests/test_cache_breakpoint_diagnostics.py:257: in test_runtime_log_is_owner_only_when_preview_enabled
    assert stat.S_IMODE(log_path.stat().st_mode) == 0o600
E   AssertionError: assert 420 == 384        # 0o644 vs 0o600

____ test_runtime_log_hardening_survives_a_pre_existing_world_readable_log _____
E   AssertionError: assert 420 == 384

FAILED tests/test_cache_breakpoint_diagnostics.py::test_payload_preview_disabled_by_default
FAILED tests/test_cache_breakpoint_diagnostics.py::test_payload_preview_stays_off_for_anything_else[]
FAILED tests/test_cache_breakpoint_diagnostics.py::test_payload_preview_stays_off_for_anything_else[maybe]
FAILED tests/test_cache_breakpoint_diagnostics.py::test_payload_preview_stays_off_for_anything_else[  ]
FAILED tests/test_cache_breakpoint_diagnostics.py::test_runtime_log_holds_no_payload_text_at_default_settings
FAILED tests/test_cache_breakpoint_diagnostics.py::test_runtime_log_is_owner_only_when_preview_enabled
FAILED tests/test_cache_breakpoint_diagnostics.py::test_runtime_log_hardening_survives_a_pre_existing_world_readable_log
========================= 7 failed, 17 passed in 1.59s =========================
```

That first failure is the finding itself: on `main`, customer data lands
in a real log file
under default settings.

**After (sources restored):**

```
tests/test_cache_breakpoint_diagnostics.py ........................      [ 24%]
tests/test_compression_store.py ............................................ [100%]
============================= 100 passed in 9.36s ==============================
```

`tests/test_perf_per_port_logs.py` (10 passed) covers the handler swap —
dedup, per-worker
paths, and sequential port changes still behave.

### Tests

New, in `tests/test_cache_breakpoint_diagnostics.py`:

- `test_payload_preview_disabled_by_default` — unset ⇒ byte counts only.
- `test_payload_preview_opt_in_values` /
`test_payload_preview_stays_off_for_anything_else`
— only an affirmative value enables it; a typo (`maybe`, `""`,
whitespace) does not.
- `test_runtime_log_holds_no_payload_text_at_default_settings` —
end-to-end through the
real `_setup_file_logging` handler: a store+retrieve leaves
`event=headroom_retrieve` in
  the file with the right `payload_chars` and **no** payload text.
- `test_runtime_log_is_owner_only_when_preview_enabled` — `0600` on the
created file.
-
`test_runtime_log_hardening_survives_a_pre_existing_world_readable_log`
— a `0644` log
from a pre-hardening run is chmodded, since `O_CREAT`'s mode does not
apply to it.

Updated: the two tests in `tests/test_compression_store.py` that
exercise the preview and
its redaction now opt in explicitly, and
`test_payload_preview_enabled_by_default` is
inverted to `test_payload_preview_disabled_by_default`.

### Notes

- Independent of #3726 (A-5) — both are based on `origin/main`, and they
touch disjoint
  parts of `compression_store.py`.
- A-5's audit found that the runtime log is one of the files
`--stateless` does *not*
suppress, so before this PR a stateless proxy was getting payload text
on disk too.
- `CHANGELOG.md` not touched (CI guard).


---

## Update after review (@JerrettDavis) — both blockers addressed

### 1. Windows: the guarantee is now scoped in code, not just in prose

You were right that this was a control that silently did nothing on a
supported platform.
On NTFS, read access is decided by the file's ACL; `os.chmod` only
toggles the read-only
attribute, so `chmod(0o600)` returns successfully while `stat.S_IMODE`
still reports
`0666` — which is exactly what your run showed.

I did **not** implement a Windows ACL. Python ships no ACL API; doing it
means ~150 lines
of `ctypes` building a `SECURITY_ATTRIBUTES` DACL, which I cannot
execute or test on this
machine, and I would rather not ship unverifiable security code than
ship an untested one.

So the guarantee is scoped honestly, and made loud rather than silent:

- `headroom/fileperms.OWNER_ONLY_SUPPORTED` is `os.name == "posix"`.
`restrict_fd_to_owner`
/ `restrict_path_to_owner` return `False` off POSIX rather than
reporting a protection
  they did not establish.
- `_setup_file_logging` emits **one warning per process** on such a
platform: *"Headroom
cannot create &lt;path&gt; owner-only on this platform: file modes do
not control read
access here, and Headroom does not set an ACL. The runtime log can
contain request and
response content (`--log-messages`, wire debug,
`HEADROOM_LOG_PAYLOAD_PREVIEW`) — protect
the log directory itself."* Logging still works; it just no longer
implies a protection.
- The docs say it in as many words, under **Who can read the logs**: on
Windows the log
inherits the ACL of its parent directory, and the directory is the
access-control
  boundary.
- The tests assert the real platform-conditional behaviour. The mode
assertions carry
`@_posix_only` with that reason;
`test_owner_only_support_matches_what_the_platform_can_enforce`
  asserts the flag equals `os.name == "posix"` **on every platform**;
`test_no_owner_only_claim_is_made_off_posix` runs *only* on Windows and
asserts the
  absence of the claim (the handler still creates the file;
  `restrict_path_to_owner` returns `False`); and
`test_unsupported_platform_says_so_rather_than_silently_not_protecting`
simulates the
unsupported platform on any host so the warning itself is covered by
Linux CI.

**In one line: on Linux and macOS the log is `0600`, enforced. On
Windows Headroom does not
restrict the log, says so at startup and in the docs, and no test claims
otherwise.**

### 2. The permissions no longer key off `HEADROOM_LOG_PAYLOAD_PREVIEW`

Agreed, and taken as stated: **every** Headroom runtime log is created
owner-only, rollover
files included, regardless of which content switch is on. Previews were
never the only
source of request content in that file — `--log-messages` bodies,
wire-debug dumps and
query logging land there too, each behind its own switch, so gating on
any one of them left
the rest writing a sensitive file at the umask. `_setup_file_logging`
now always installs
`_OwnerOnlyRotatingFileHandler` and no longer imports
`_payload_preview_enabled` at all.
I extended it to the `--log-file` JSONL as well, since under
`--log-messages` that is the
file with the whole bodies in it.

Previews still default off — that part of the PR is unchanged.

This **does** break log collection for an operator reading the file as
another user, which
is the trade-off the original PR body ducked. It is now called out
explicitly in the release
note at the top, with the two workarounds.

### Also fixed while in here (found in an internal pass)

- **Rotated backups.** `rotate()` re-applies `0600` to the destination.
`os.rename` happens
to carry the mode across, but a configured `rotator` need not, so the
invariant is
asserted on the result rather than assumed from how it got there.
Backups left `0644` by
  an older build are tightened when the handler opens.
- **`_open` followed symlinks.** The old code pre-created the file and
then delegated to
`super()._open()`, so both the `chmod` and the write followed a symlink
planted at the log
path — an attacker chose the destination *and* the permissions. `_open`
is now replaced
outright and opens with `O_NOFOLLOW` at `0600`, tightening with `fchmod`
on that same
descriptor (no TOCTOU window, and no way for the mode to land on another
inode). A log
path that is *already* a symlink is refused up front with a warning,
which is what carries
the refusal on platforms without `O_NOFOLLOW`. Deliberate decision:
**fail closed** — a
redirected sensitive log is worse than no log, and logging is already
best-effort here.

### Fail-before / pass-after for this round

Baseline is `ec014d64` (the commit you reviewed), not `origin/main`,
since these are new
findings on top of it. Method: commit, `git checkout ec014d64 --
headroom/proxy/helpers.py
headroom/proxy/request_logger.py headroom/cache/compression_store.py`
keeping the new
tests, run, restore.

```
FAILED test_runtime_log_is_owner_only_with_previews_off                       assert 420 == 384
FAILED test_runtime_log_hardening_survives_a_pre_existing_world_readable_log   assert 420 == 384
FAILED test_rotated_backups_are_owner_only                                     assert 420 == 384
FAILED test_pre_existing_backups_are_tightened_when_the_handler_opens          assert 420 == 384
FAILED test_runtime_log_refuses_a_symlinked_path       AssertionError: logging was wired up through the symlink
FAILED test_jsonl_request_log_is_owner_only                                    assert 420 == 384
FAILED test_unsupported_platform_says_so_rather_than_silently_not_protecting
    AttributeError: module 'headroom.proxy.helpers' has no attribute '_owner_only_warning_emitted'
=================== 7 failed, 25 passed, 1 skipped in 1.23s ====================
```

(`420` is `0o644`, `384` is `0o600`. A `predictable_umask` fixture pins
the umask to `0o022`
so "would have been world-readable" is not down to the runner's
environment.)

After, sources restored:

```
tests/test_cache_breakpoint_diagnostics.py tests/test_compression_store.py
tests/test_perf_per_port_logs.py tests/test_proxy/test_request_logger.py
tests/test_proxy_streaming_request_logger.py
======================= 132 passed, 1 skipped in 10.90s ========================
```

The 1 skip is `test_no_owner_only_claim_is_made_off_posix` — it is the
Windows-only half,
and it is the one I cannot run here.

Ruff 0.16.7 `check` and `format --check` are clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
Tejas Chopra
2026-09-24 13:09:06 +02:00
committed by GitHub
co-authored by Claude Opus 5 JD Davis
parent e8a510982a
commit 5cb87bc351
7 changed files with 604 additions and 13 deletions
+25
View File
@@ -296,12 +296,37 @@ Rewrite the upstream model per request — for example, send small, tool-free ca
| `--telemetry` / `HEADROOM_TELEMETRY` | off | **Local-only** usage stats for your own `/stats`, `/metrics`, and dashboard. Nothing leaves the machine. |
| `--log-file` / `HEADROOM_LOG_FILE` | none | JSONL request/response log. |
| `--log-messages` | `false` | Include full message bodies in the log (may contain sensitive data). |
| `HEADROOM_LOG_PAYLOAD_PREVIEW` | `off` | Include up to 4 KB of the retrieved [CCR](/docs/ccr) payload in each `headroom_retrieve` line of the runtime log (`~/.headroom/logs/proxy-<port>.log`). Off by default: the payload is verbatim tool-result content and can carry source code, credentials the redactor does not recognize, or customer data; the log otherwise records byte counts only. Turn it on with `1`/`true`/`yes`/`on`. Do not share a log captured with this on. |
| `HEADROOM_LOG_LEVEL` | `warning` | uvicorn's log level (`critical`, `error`, `warning`, `info`, `debug`, `trace`). Raise to `info` for the per-request access log when diagnosing a deployed proxy. An unrecognized value warns and falls back to `warning`. |
| `HEADROOM_OTEL_METRICS_ENABLED` | `false` | Export OpenTelemetry metrics (`HEADROOM_OTEL_METRICS_ENDPOINT`, …). See [OTLP export](/docs/metrics#opentelemetry-otlp-export). |
| `HEADROOM_LANGFUSE_ENABLED` | `false` | Emit Langfuse traces (`LANGFUSE_PUBLIC_KEY` / `LANGFUSE_SECRET_KEY`). |
See [Metrics](/docs/metrics) for the Prometheus and Grafana setup.
#### Who can read the logs
Headroom writes the runtime log (`~/.headroom/logs/proxy-<port>.log`) and, when
`--log-file` is set, the JSONL request log. Both can hold request and response
content — `--log-messages` bodies, wire-debug dumps, and CCR payload previews
when they are enabled.
- **Linux and macOS:** every log file Headroom creates is `0600`, owner-read/write
only, and so is every rotated backup (`proxy-8000.log.1`, …). This is not
conditional on which content switch is on. If the log path already exists as a
symlink, Headroom refuses to log rather than write through it, and says so.
- **Windows:** **Headroom does not restrict the log file.** Read access on NTFS is
decided by the file's ACL; `chmod` there only toggles the read-only attribute,
and Python has no API for setting an ACL, so Headroom does not claim a
protection it cannot establish. The log inherits the ACL of the directory it
is written into, and the proxy logs a warning at startup saying so. Keep
`%USERPROFILE%\.headroom` (or your chosen `--log-file` path) off shared
volumes and out of directories with broadened permissions.
A log shipper or sidecar running as a different user or group cannot read the
runtime log on Linux/macOS. Run the collector as the same user, or copy the file
with a privileged job.
### Security and networking
| Flag / env | Default | Effect |
+11 -5
View File
@@ -53,9 +53,14 @@ DEFAULT_CCR_TTL_SECONDS = 1800 # session-scale; override via HEADROOM_CCR_TTL_S
CCR_TTL_SECONDS_ENV = "HEADROOM_CCR_TTL_SECONDS"
_RETRIEVAL_LOG_PREVIEW_CHARS = 4096
# Previews carry verbatim tool-result content (post-redaction), which makes
# proxy.log too sensitive for users to share in bug reports. Set to
# 0/false/no/off to log byte counts only.
# Previews carry verbatim tool-result content (post-redaction) — source code,
# credentials the redactor does not recognize, customer data. That is not
# something the always-on runtime log should hold, so previews are OFF unless
# an operator turns them on with 1/true/yes/on; the log then records byte
# counts only. (The log file is created owner-only either way — see
# ``headroom/proxy/helpers.py:_OwnerOnlyRotatingFileHandler`` — because other
# switches put request content in the same file. That is a second line of
# defence, not a reason to log the payload.)
PAYLOAD_PREVIEW_ENV = "HEADROOM_LOG_PAYLOAD_PREVIEW"
_SECRET_KEY_VALUE_RE = re.compile(
r"(?i)\b([A-Z0-9_-]*(?:API[_-]?KEY|TOKEN|SECRET|PASSWORD|CREDENTIAL|AUTH)[A-Z0-9_-]*)"
@@ -114,10 +119,11 @@ def _redact_retrieval_log_payload(payload: str) -> str:
def _payload_preview_enabled() -> bool:
"""True only when an operator has explicitly opted in. Default: off."""
raw = os.environ.get(PAYLOAD_PREVIEW_ENV)
if raw is None:
return True
return raw.strip().lower() not in ("0", "false", "no", "off")
return False
return raw.strip().lower() in ("1", "true", "yes", "on")
def _payload_for_retrieval_log(payload: str) -> dict[str, Any]:
+118
View File
@@ -0,0 +1,118 @@
"""Owner-only creation for the files Headroom writes at runtime.
Headroom's runtime log (``~/.headroom/logs/proxy-<port>.log``) and the optional
JSONL request log can both carry verbatim request and response content: wire
debug dumps, ``--log-messages`` bodies, and — when an operator opts in — CCR
payload previews. None of that should be created at the process umask, which on
a stock developer machine means ``0644``: world-readable.
Scope of the guarantee — read this before citing it in a threat model:
* **POSIX** (Linux, macOS, \\*BSD): enforced. Files are created ``0600`` and
tightened with ``fchmod`` on the already-open descriptor, so a file left
world-readable by an earlier run is fixed, and the mode cannot be applied to
the wrong inode by a path that changed underneath us. ``O_NOFOLLOW`` means a
symlink planted at the path fails the open outright rather than redirecting
the write.
* **Windows**: *not* enforced, and deliberately not claimed. Who may read an
NTFS file is decided by its ACL. ``os.chmod`` on Windows only toggles the
read-only attribute and leaves the ACL untouched, so ``chmod(0o600)`` returns
successfully while ``stat.S_IMODE`` still reports ``0666`` and the file stays
readable per the inherited ACL. Python ships no ACL API, so Headroom does not
pretend to set one: on Windows the log inherits the ACL of its parent
directory and :data:`OWNER_ONLY_SUPPORTED` is ``False``. Callers that create
sensitive files say so once in the log (see
``headroom.proxy.helpers._setup_file_logging``), and operators should treat
the log directory itself as the access-control boundary — keep it under the
user profile, and do not put it on a share.
``O_NOFOLLOW`` does not exist on Windows either, so the symlink protection
there is the explicit :func:`is_symlink` check callers make before opening, not
a kernel-enforced one.
"""
from __future__ import annotations
import os
import stat
from typing import IO, Any
#: Mode for every runtime file Headroom creates that may hold request content.
OWNER_ONLY_MODE = 0o600
#: ``True`` only where the mode bits above actually decide who can read the
#: file. See the module docstring for why Windows is excluded.
OWNER_ONLY_SUPPORTED = os.name == "posix"
def _open_flags() -> int:
flags = os.O_CREAT | os.O_WRONLY | os.O_APPEND
# Refuse to open through a symlink where the platform can enforce it, so a
# planted link cannot redirect either the write or the chmod. Absent on
# Windows, where getattr() leaves the flag out.
flags |= getattr(os, "O_NOFOLLOW", 0)
# Match the builtin open(): keep the OS layer byte-exact and let the text
# wrapper do newline translation, instead of translating twice on Windows.
flags |= getattr(os, "O_BINARY", 0)
return flags
def restrict_fd_to_owner(fd: int) -> bool:
"""Make an open descriptor owner-only. Returns whether it took effect.
Acts on the descriptor, not the path, so there is no window in which the
mode could land on a different file, and no way for a symlink to move it.
Returns ``False`` on platforms where the mode bits do not carry the
guarantee rather than reporting a protection that was not established.
"""
if not OWNER_ONLY_SUPPORTED:
return False
if stat.S_IMODE(os.fstat(fd).st_mode) != OWNER_ONLY_MODE:
os.fchmod(fd, OWNER_ONLY_MODE)
return True
def restrict_path_to_owner(path: str | os.PathLike[str]) -> bool:
"""Make an existing file owner-only. Returns whether it took effect.
For files Headroom did not open itself — rotated log backups, and backups
left behind by an older unhardened run. A path that is a symlink is left
alone: the target is not ours to re-permission.
"""
if not OWNER_ONLY_SUPPORTED:
return False
try:
if os.path.islink(path) or not os.path.exists(path):
return False
if stat.S_IMODE(os.stat(path).st_mode) != OWNER_ONLY_MODE:
os.chmod(path, OWNER_ONLY_MODE)
except OSError:
return False
return True
def open_owner_only(
path: str | os.PathLike[str],
mode: str = "a",
*,
encoding: str | None = None,
errors: str | None = None,
) -> IO[Any]:
"""Open *path* for append, creating it owner-only.
Raises ``OSError`` — which every caller already treats as "logging is
unavailable, carry on" — if the path cannot be opened, including when it is
a symlink on a platform with ``O_NOFOLLOW``. Failing closed is deliberate:
a redirected sensitive log is worse than no log.
"""
fd = os.open(path, _open_flags(), OWNER_ONLY_MODE)
try:
restrict_fd_to_owner(fd)
return open(fd, mode, encoding=encoding, errors=errors, closefd=True)
except BaseException:
try:
os.close(fd)
except OSError:
# open() can have taken and closed the descriptor on its way out.
pass
raise
+98 -2
View File
@@ -20,9 +20,11 @@ import time
from collections import OrderedDict
from collections.abc import Callable, Iterator
from dataclasses import dataclass
from logging.handlers import RotatingFileHandler
from pathlib import Path
from typing import TYPE_CHECKING, Any, Literal, cast
from headroom import fileperms as _fileperms
from headroom import paths as _paths
from headroom.proxy import (
diagnostic_decode_policy,
@@ -1723,6 +1725,79 @@ def _headroom_log_dir() -> Path:
_PROXY_LOG_HANDLER_NAME = "headroom.proxy.file"
class _OwnerOnlyRotatingFileHandler(RotatingFileHandler):
"""RotatingFileHandler whose log file is only readable by its owner.
``logging`` opens the stream itself — once at construction and again for
every rollover — so the process umask would otherwise decide the mode, and
there is no hook to pass one. ``_open`` is replaced outright (rather than
pre-creating the file and delegating) so that the descriptor the handler
writes through is the same one the mode was applied to, and so that
``O_NOFOLLOW`` is in force on the open the stream actually uses.
Three paths, all of which have to hold the guarantee:
* first open — created 0600 instead of at the umask;
* an existing log — ``O_CREAT``'s mode does not apply to a file that
already exists, so the descriptor is tightened with ``fchmod``;
* rollover — ``_open`` runs again for the new base file, and ``rotate``
re-applies the mode to each backup, so ``proxy-8000.log.1`` is no more
readable than ``proxy-8000.log``. Backups left behind by an older,
unhardened build are tightened when the handler is constructed.
The mode bits carry this only on POSIX; see :mod:`headroom.fileperms` for
what is and is not claimed on Windows.
"""
def __init__(self, *args: Any, **kwargs: Any) -> None:
super().__init__(*args, **kwargs)
self._restrict_existing_backups()
def _restrict_existing_backups(self) -> None:
for index in range(1, (self.backupCount or 0) + 1):
_fileperms.restrict_path_to_owner(
self.rotation_filename(f"{self.baseFilename}.{index}")
)
def _open(self): # type: ignore[no-untyped-def]
return _fileperms.open_owner_only(
self.baseFilename,
self.mode,
encoding=self.encoding,
errors=getattr(self, "errors", None),
)
def rotate(self, source: str, dest: str) -> None:
super().rotate(source, dest)
# os.rename carries the mode across, but a configured ``rotator`` (or a
# copy-based one) need not, so the invariant is asserted on the result
# rather than assumed from how it got there.
_fileperms.restrict_path_to_owner(dest)
_owner_only_warning_emitted = False
def _warn_once_if_owner_only_unsupported(log_path: Path) -> None:
"""Say plainly, once per process, when the file cannot be made owner-only.
A security control that quietly does nothing on a supported platform is
worse than no control, so Windows gets told rather than left to assume the
0600 in the docs applies to it.
"""
global _owner_only_warning_emitted
if _fileperms.OWNER_ONLY_SUPPORTED or _owner_only_warning_emitted:
return
_owner_only_warning_emitted = True
logger.warning(
"Headroom cannot create %s owner-only on this platform: file modes do not "
"control read access here, and Headroom does not set an ACL. The runtime log "
"can contain request and response content (--log-messages, wire debug, "
"HEADROOM_LOG_PAYLOAD_PREVIEW) — protect the log directory itself.",
log_path,
)
def _setup_file_logging(
port: int | None = None,
*,
@@ -1737,13 +1812,34 @@ def _setup_file_logging(
The file is keyed by *port* so concurrent instances rotate separate logs.
Multi-worker callers also pass *process_id* so same-port workers cannot
race during rollover. When *port* is omitted the legacy shared name is used.
The log is **always** created owner-only, and so are its rotated backups —
not only when ``HEADROOM_LOG_PAYLOAD_PREVIEW`` is on. Payload previews are
one of several sources of request content in this file: ``--log-messages``
bodies, wire debug dumps and query logging land here too, each behind its
own switch, so keying the file's permissions off any one of them leaves the
others writing a sensitive file at the umask. On POSIX that is enforced;
on Windows it is not — see :mod:`headroom.fileperms`.
"""
from logging.handlers import RotatingFileHandler
handler_cls = _OwnerOnlyRotatingFileHandler
try:
log_dir = _headroom_log_dir()
log_dir.mkdir(parents=True, exist_ok=True)
log_path = _paths.proxy_log_path(port, process_id=process_id)
if log_path.is_symlink():
# Fail closed. A symlink at the log path redirects both the write
# and the mode we set on it, so whoever planted it chooses where
# request content lands and who can read it. O_NOFOLLOW catches
# this in _open too; the explicit check is what carries the
# refusal on platforms without that flag, and lets us say why.
logger.warning(
"Refusing to write the Headroom runtime log: %s is a symlink. "
"Remove it (or point HEADROOM_WORKSPACE_DIR elsewhere) to restore logging.",
log_path,
)
return
_warn_once_if_owner_only_unsupported(log_path)
# Attach to the headroom root logger so all sub-loggers are captured.
# Disable propagation to root to avoid duplicate writes when
# wrap.py redirects stderr to the same log file.
@@ -1763,7 +1859,7 @@ def _setup_file_logging(
]
if any(Path(h.baseFilename) == log_path for h in existing):
return
handler = RotatingFileHandler(
handler = handler_cls(
log_path,
maxBytes=10 * 1024 * 1024, # 10 MB
backupCount=5,
+7 -1
View File
@@ -34,6 +34,7 @@ from typing import TYPE_CHECKING, Any
if TYPE_CHECKING:
from ..memory.tracker import ComponentStats
from headroom import fileperms
from headroom.proxy import request_log_redaction_policy
from headroom.proxy.models import RequestLog
@@ -138,7 +139,12 @@ class RequestLogger:
if self.log_file:
try:
with open(self.log_file, "a") as f:
# Owner-only: with ``log_full_messages`` this file holds whole
# request and response bodies, and it is the caller's chosen
# path rather than one under ~/.headroom, so it must not be
# created at the umask. Symlinked paths fail the open and land
# in the graceful-degradation branch below.
with fileperms.open_owner_only(self.log_file, "a") as f:
log_dict = asdict(entry)
if not self.log_full_messages:
log_dict.pop("request_messages", None)
+341 -3
View File
@@ -9,10 +9,22 @@ Covers the three pieces added for the uncached-tail investigation:
from __future__ import annotations
import logging
import os
import stat
from contextlib import contextmanager
from pathlib import Path
import pytest
from headroom import fileperms
from headroom import paths as _paths
from headroom.cache.compression_store import _payload_for_retrieval_log
from headroom.proxy.handlers.anthropic import AnthropicHandlerMixin
from headroom.proxy.helpers import count_cache_breakpoints, log_cache_breakpoints
from headroom.proxy.helpers import (
_OwnerOnlyRotatingFileHandler,
count_cache_breakpoints,
log_cache_breakpoints,
)
_CC = {"cache_control": {"type": "ephemeral"}}
@@ -125,10 +137,26 @@ def test_payload_preview_disabled_omits_content(monkeypatch) -> None:
assert event["payload_truncated"] is True
def test_payload_preview_enabled_by_default(monkeypatch) -> None:
def test_payload_preview_disabled_by_default(monkeypatch) -> None:
"""Unset means off: the log gets byte counts, never the content."""
monkeypatch.delenv("HEADROOM_LOG_PAYLOAD_PREVIEW", raising=False)
event = _payload_for_retrieval_log("hello world")
assert event["payload_preview"] == "hello world"
assert event["payload_preview"] == ""
assert event["payload_preview_chars"] == 0
assert event["payload_chars"] == len("hello world")
@pytest.mark.parametrize("value", ["1", "true", "TRUE", "yes", "on"])
def test_payload_preview_opt_in_values(monkeypatch, value: str) -> None:
monkeypatch.setenv("HEADROOM_LOG_PAYLOAD_PREVIEW", value)
assert _payload_for_retrieval_log("hello world")["payload_preview"] == "hello world"
@pytest.mark.parametrize("value", ["", "0", "off", "no", "maybe", " "])
def test_payload_preview_stays_off_for_anything_else(monkeypatch, value: str) -> None:
"""Only an explicit opt-in turns previews on — a typo must not."""
monkeypatch.setenv("HEADROOM_LOG_PAYLOAD_PREVIEW", value)
assert _payload_for_retrieval_log("hello world")["payload_preview"] == ""
def test_append_context_skips_breakpointed_text_block() -> None:
@@ -178,3 +206,313 @@ def test_count_cache_breakpoints_tolerates_malformed_shapes() -> None:
empty = count_cache_breakpoints(None, None, None)
assert empty["total"] == 0
assert empty["message_count"] == 0
# --- the runtime log file itself -------------------------------------------
#
# _payload_for_retrieval_log decides what goes into the record;
# _setup_file_logging decides who can read the file it lands in. Both halves
# of the default-off guarantee are checked against a real log on disk.
@contextmanager
def _proxy_log(tmp_path, monkeypatch, port: int):
"""Point the workspace at *tmp_path*, install the real proxy log handler."""
from headroom.proxy.helpers import _PROXY_LOG_HANDLER_NAME, _setup_file_logging
monkeypatch.setenv("HEADROOM_WORKSPACE_DIR", str(tmp_path))
headroom_logger = logging.getLogger("headroom")
before = list(headroom_logger.handlers)
propagate = headroom_logger.propagate
try:
_setup_file_logging(port)
[handler] = [h for h in headroom_logger.handlers if h.name == _PROXY_LOG_HANDLER_NAME]
yield Path(handler.baseFilename)
finally:
for handler in list(headroom_logger.handlers):
if handler not in before:
headroom_logger.removeHandler(handler)
handler.close()
headroom_logger.propagate = propagate
def test_runtime_log_holds_no_payload_text_at_default_settings(tmp_path, monkeypatch) -> None:
"""A retrieval on default settings leaves byte counts in the log, not content."""
from headroom.cache.compression_store import CompressionStore
monkeypatch.delenv("HEADROOM_LOG_PAYLOAD_PREVIEW", raising=False)
secret = "BEGIN-CUSTOMER-DATA ssn=123-45-6789 def sekrit(): pass END-CUSTOMER-DATA"
with _proxy_log(tmp_path, monkeypatch, 18801) as log_path:
store = CompressionStore(enable_feedback=False)
assert store.retrieve(store.store(original=secret, compressed="[compressed]")) is not None
logging.getLogger("headroom").handlers[-1].flush()
text = log_path.read_text(encoding="utf-8")
assert "event=headroom_retrieve" in text, "the retrieval was not logged at all"
assert secret not in text
assert "123-45-6789" not in text
assert f'"payload_chars":{len(secret)}' in text
assert '"payload_preview":""' in text
_posix_only = pytest.mark.skipif(
not fileperms.OWNER_ONLY_SUPPORTED,
reason=(
"asserts POSIX mode bits, which do not control read access on this platform; "
"the scope of the guarantee is asserted instead by "
"test_owner_only_support_matches_what_the_platform_can_enforce and "
"test_no_owner_only_claim_is_made_off_posix"
),
)
@pytest.fixture
def predictable_umask():
"""Pin the umask so "would have been world-readable" is not luck.
Without this the fail-before evidence for these tests depends on whatever
umask the runner happens to have; 0o022 is the stock developer value that
makes an unhardened log 0644.
"""
previous = os.umask(0o022)
try:
yield
finally:
os.umask(previous)
@_posix_only
def test_runtime_log_is_owner_only_with_previews_off(
tmp_path, monkeypatch, predictable_umask
) -> None:
"""The file's permissions are not keyed off the payload-preview switch.
Previews are one of several sources of request content in this file:
``--log-messages`` bodies, wire-debug dumps and query logging land here
too, each behind its own switch. Hardening only when previews are on left
every other combination creating a sensitive log at the umask.
"""
monkeypatch.delenv("HEADROOM_LOG_PAYLOAD_PREVIEW", raising=False)
with _proxy_log(tmp_path, monkeypatch, 18804) as log_path:
assert log_path.exists()
assert stat.S_IMODE(log_path.stat().st_mode) == 0o600
@_posix_only
def test_runtime_log_is_owner_only_when_preview_enabled(tmp_path, monkeypatch) -> None:
"""Opting in to previews hardens the log the previews land in."""
monkeypatch.setenv("HEADROOM_LOG_PAYLOAD_PREVIEW", "1")
with _proxy_log(tmp_path, monkeypatch, 18802) as log_path:
assert log_path.exists()
assert stat.S_IMODE(log_path.stat().st_mode) == 0o600
@_posix_only
def test_runtime_log_hardening_survives_a_pre_existing_world_readable_log(
tmp_path, monkeypatch, predictable_umask
) -> None:
"""O_CREAT's mode does not apply to an existing file; the fchmod must."""
monkeypatch.delenv("HEADROOM_LOG_PAYLOAD_PREVIEW", raising=False)
monkeypatch.setenv("HEADROOM_WORKSPACE_DIR", str(tmp_path))
stale = _paths.proxy_log_path(18803)
stale.parent.mkdir(parents=True, exist_ok=True)
stale.write_text("from an older, unhardened run\n", encoding="utf-8")
stale.chmod(0o644)
with _proxy_log(tmp_path, monkeypatch, 18803) as log_path:
assert stat.S_IMODE(log_path.stat().st_mode) == 0o600
@_posix_only
def test_rotated_backups_are_owner_only(tmp_path, monkeypatch, predictable_umask) -> None:
"""Rotation must not launder the mode away.
``doRollover`` renames the base file and opens a fresh one; a backup that
ends up 0644 exposes exactly the content the base file was protecting, and
keeps exposing it for ``backupCount`` rotations. Exercised through the real
handler the proxy installs (with ``maxBytes`` shrunk so a rollover is
reachable in a test), not a hand-built one, so the handler *selection* is
part of what is asserted.
"""
monkeypatch.delenv("HEADROOM_LOG_PAYLOAD_PREVIEW", raising=False)
with _proxy_log(tmp_path, monkeypatch, 18805) as log_path:
handler = next(
h for h in logging.getLogger("headroom").handlers if h.name == "headroom.proxy.file"
)
handler.maxBytes = 256
for i in range(60):
handler.emit(
logging.LogRecord("headroom", logging.INFO, __file__, i, "x" * 64, None, None)
)
handler.flush()
backups = sorted(log_path.parent.glob(f"{log_path.name}.*"))
assert backups, "no rollover happened — the test did not exercise the path it claims to"
for path in [log_path, *backups]:
assert stat.S_IMODE(path.stat().st_mode) == 0o600, path
@_posix_only
def test_pre_existing_backups_are_tightened_when_the_handler_opens(
tmp_path, predictable_umask
) -> None:
"""Backups written by an older, unhardened build are still on disk."""
log_path = tmp_path / "proxy-18806.log"
stale = tmp_path / "proxy-18806.log.2"
stale.write_text("payload from before the fix\n", encoding="utf-8")
stale.chmod(0o644)
handler = _OwnerOnlyRotatingFileHandler(
log_path, maxBytes=1024, backupCount=5, encoding="utf-8"
)
handler.close()
assert stat.S_IMODE(stale.stat().st_mode) == 0o600
@pytest.mark.skipif(
os.name != "posix",
reason="creating a symlink needs elevation on Windows, and O_NOFOLLOW does not exist there",
)
def test_runtime_log_refuses_a_symlinked_path(tmp_path, monkeypatch) -> None:
"""A planted symlink must not redirect the log — or the mode we set on it."""
from headroom.proxy.helpers import _PROXY_LOG_HANDLER_NAME, _setup_file_logging
monkeypatch.setenv("HEADROOM_WORKSPACE_DIR", str(tmp_path))
log_path = _paths.proxy_log_path(18807)
log_path.parent.mkdir(parents=True, exist_ok=True)
elsewhere = tmp_path / "attacker-readable.log"
elsewhere.write_text("", encoding="utf-8")
elsewhere.chmod(0o666)
log_path.symlink_to(elsewhere)
headroom_logger = logging.getLogger("headroom")
before = list(headroom_logger.handlers)
try:
_setup_file_logging(18807)
attached = [h for h in headroom_logger.handlers if h.name == _PROXY_LOG_HANDLER_NAME]
assert not attached, "logging was wired up through the symlink"
logging.getLogger("headroom").info("a record that must not be written")
finally:
for handler in list(headroom_logger.handlers):
if handler not in before:
headroom_logger.removeHandler(handler)
handler.close()
assert elsewhere.read_text(encoding="utf-8") == ""
assert stat.S_IMODE(elsewhere.stat().st_mode) == 0o666, "the symlink target was chmodded"
@pytest.mark.skipif(
os.name != "posix",
reason="creating a symlink needs elevation on Windows, and O_NOFOLLOW does not exist there",
)
def test_open_owner_only_fails_closed_on_a_symlink(tmp_path) -> None:
"""The kernel-enforced half of the same refusal, at the open() itself."""
target = tmp_path / "target"
target.write_text("", encoding="utf-8")
link = tmp_path / "link"
link.symlink_to(target)
with pytest.raises(OSError):
fileperms.open_owner_only(link).close()
@_posix_only
def test_jsonl_request_log_is_owner_only(tmp_path, predictable_umask) -> None:
"""``--log-file`` with ``--log-messages`` writes whole bodies to this file."""
from headroom.proxy.models import RequestLog
from headroom.proxy.request_logger import RequestLogger
log_file = tmp_path / "requests.jsonl"
entry = RequestLog(
request_id="r1",
timestamp="2026-09-23T00:00:00Z",
provider="anthropic",
model="claude-opus-4-20250514",
input_tokens_original=10,
input_tokens_optimized=8,
output_tokens=2,
tokens_saved=2,
savings_percent=20.0,
optimization_latency_ms=1.0,
total_latency_ms=2.0,
tags={},
cache_hit=False,
transforms_applied=[],
request_messages=[{"role": "user", "content": "ssn=123-45-6789"}],
)
RequestLogger(log_file=str(log_file), log_full_messages=True).log(entry)
assert "123-45-6789" in log_file.read_text(encoding="utf-8")
assert stat.S_IMODE(log_file.stat().st_mode) == 0o600
# --- what the guarantee is, and is not, on Windows -------------------------
def test_owner_only_support_matches_what_the_platform_can_enforce() -> None:
"""The scope is a value in the code, not a claim in a comment.
POSIX mode bits decide read access; on Windows an NTFS ACL does, and
``os.chmod`` there only flips the read-only attribute — a
``chmod(0o600)`` succeeds while ``stat.S_IMODE`` still reports ``0666``.
Python ships no ACL API, so Headroom reports that it cannot make the
promise instead of making it and not keeping it.
"""
assert fileperms.OWNER_ONLY_SUPPORTED is (os.name == "posix")
@pytest.mark.skipif(
fileperms.OWNER_ONLY_SUPPORTED,
reason="asserts the *absence* of the mode guarantee; only meaningful off POSIX",
)
def test_no_owner_only_claim_is_made_off_posix(tmp_path) -> None:
"""On Windows the handler still logs — it just does not claim 0600."""
log_path = tmp_path / "proxy-18808.log"
handler = _OwnerOnlyRotatingFileHandler(
log_path, maxBytes=1024, backupCount=1, encoding="utf-8"
)
handler.close()
assert log_path.exists(), "logging must keep working where hardening cannot"
assert fileperms.restrict_path_to_owner(log_path) is False
def test_unsupported_platform_says_so_rather_than_silently_not_protecting(
tmp_path, monkeypatch
) -> None:
"""Simulates the Windows path on any host, since CI cannot be both.
A control that quietly does nothing on a supported platform is the thing
to avoid, so the operator is told once per process.
"""
from headroom.proxy import helpers as _helpers
monkeypatch.setattr(fileperms, "OWNER_ONLY_SUPPORTED", False)
monkeypatch.setattr(_helpers, "_owner_only_warning_emitted", False)
monkeypatch.setenv("HEADROOM_WORKSPACE_DIR", str(tmp_path))
records: list[logging.LogRecord] = []
sink = logging.Handler()
sink.emit = records.append # type: ignore[method-assign]
emitter = logging.getLogger(_helpers.logger.name)
emitter.addHandler(sink)
headroom_logger = logging.getLogger("headroom")
before = list(headroom_logger.handlers)
try:
_helpers._setup_file_logging(18809)
finally:
emitter.removeHandler(sink)
for handler in list(headroom_logger.handlers):
if handler not in before:
headroom_logger.removeHandler(handler)
handler.close()
warnings = [r for r in records if r.levelno == logging.WARNING]
assert warnings, "the unsupported platform was not reported at all"
message = warnings[0].getMessage()
assert "owner-only" in message
assert "proxy-18809.log" in message
# And only once per process, so it is a notice and not a per-worker flood.
assert _helpers._owner_only_warning_emitted is True
+4 -2
View File
@@ -60,7 +60,8 @@ def _capture_headroom_retrieve_events():
logger.setLevel(previous_level)
def test_retrieve_logs_payload_preview():
def test_retrieve_logs_payload_preview(monkeypatch: pytest.MonkeyPatch):
monkeypatch.setenv("HEADROOM_LOG_PAYLOAD_PREVIEW", "1")
store = CompressionStore(enable_feedback=False)
hash_key = store.store(
original="secret-ish payload for operator debugging",
@@ -85,7 +86,8 @@ def test_retrieve_logs_payload_preview():
assert events[0]["tool_name"] == "tool_a"
def test_retrieve_log_redacts_secret_payload_values():
def test_retrieve_log_redacts_secret_payload_values(monkeypatch: pytest.MonkeyPatch):
monkeypatch.setenv("HEADROOM_LOG_PAYLOAD_PREVIEW", "1")
store = CompressionStore(enable_feedback=False)
hash_key = store.store(
original="OPENAI_API_KEY=sk-proj-secret1234567890 Authorization: Bearer token123456789",