Commit Graph
40 Commits
Author SHA1 Message Date
2649835a51 feat(wrap): make the Serena code-memory MCP opt-in (#4509)
## What changes

`headroom wrap claude|codex|grok|opencode` no longer registers Serena by
default.

| | Before | After |
|---|---|---|
| Default | Serena registered | No code-memory MCP |
| Opt in | (default) | `--code-memory serena` or
`HEADROOM_CODE_MEMORY=serena` (deprecated `--serena` still works) |
| Opt out | `--code-memory none` / `--no-serena` | Still accepted |
| Upgrading user, Serena installed by Headroom | Kept | Removed on next
wrap, with the message "Serena is now off by default; pass --code-memory
serena to keep it" |
| Upgrading user, Serena added by hand | Kept | Kept, no message (shown
only with `--verbose` or an explicit opt-out) |

OpenCode now accepts `--code-memory` and uses the same setup path as the
other agents. Its `--no-serena` flag is hidden and deprecated.

## Why

Measured on 2026-10-27, Serena started exactly as `wrap` registers it:

- It adds 22 tools, about 5.9K tokens of tool definitions on every
request where tools aren't deferred. Its server instruction tells the
model to call `initial_instructions`, which costs another 2K tokens.
- Each launch takes about 6 s cold via `uvx`, plus a background `serena
project index` that starts language servers. It also writes a `.serena/`
folder into the user's repo.
- Serena was registered in 6 of our own Claude Code sessions and called
0 times. Separately, an offline replay of about 108K tool calls found
that symbol-level reads have little to recover for Claude Code.

## Not in this PR

- `headroom.integrations.strands` still defaults
`enable_serena_mcp=True`. It's public integration API, so it needs a
deprecation step instead of a silent flip.
- The opt-in `--serena-instructions` block in CLAUDE.md/AGENTS.md is
never removed, by unwrap or anything else. Users who opted into it keep
instructions that point at tools no longer registered. This predates
this PR and will be fixed separately.
- For Claude Code, Serena entries are per project, so removal happens
the next time `wrap` or `unwrap` runs in each project. The docs say so.

## Tests

- `pytest tests/test_wrap_code_memory.py tests/test_cli
tests/test_mcp_registry`: 1368 passed. 5 failed, and those 5 fail the
same way on `main` in my local venv:
`test_mcp_reconcile.py::test_ordinary_install_does_not_adopt_serena` and
4 in `test_wrap_persistent.py`, all from an environment/install
mismatch.
- New tests:
- The default wrap removes a Serena entry Headroom installed and prints
how to keep it.
- The default wrap leaves a hand-added Serena alone and prints nothing.
  - `--serena --no-serena` resolves to none.
- `wrap opencode --code-memory serena` registers Serena with the `agent`
context.
- `ruff check`, `ruff format --check` and `mypy` are clean on the
changed files.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

## Real Behavior Proof

- **Environment:** macOS (Darwin 27.2), Python 3.12.13, Claude Code
2.1.318, real `claude` CLI doing every `claude mcp add`/`remove`. A
temporary `HOME`, so `~/.claude.json` starts empty, and a fresh git repo
per case.
- **Builds:** "before" = `main` at `12864fc8`; "after" = this PR at
`56557dc9`.
- **Command:** `headroom wrap claude --no-proxy [flags] -- --version`.
Wrap does its full MCP setup, then launches Claude Code, which prints
its version and exits, so no model call is made. `state` is read from
`~/.claude.json` (user scope and this project's local scope) and the
project's `.mcp.json` after each run.

```
== 1. Upgrade: Serena installed by Headroom (project/local scope)
    $ headroom wrap claude --no-proxy -- --version    (cwd=projA, build=12864fc8)
    exit=0
    wrap|   Serena MCP: registered (restart Claude Code if it was already running)
    wrap|     scoped to this project — pass --code-memory-scope user to register it for every Claude Code session
    wrap| 2.1.318 (Claude Code)
    state| user: ['headroom'] | local(projA): ['serena'] | .mcp.json: -
    $ headroom wrap claude --no-proxy -- --version    (cwd=projA, build=56557dc9)
    exit=0
    wrap|   Removed previously-installed Serena MCP (Serena is now off by default; pass --code-memory serena to keep it)
    wrap| 2.1.318 (Claude Code)
    state| user: ['headroom'] | local(projA): - | .mcp.json: -

== 2. Upgrade: Serena added by hand (claude mcp add)
    state| user: ['headroom'] | local(projB): ['serena'] | .mcp.json: -
    $ headroom wrap claude --no-proxy -- --version    (cwd=projB, build=56557dc9)
    exit=0
    wrap| 2.1.318 (Claude Code)
    state| user: ['headroom'] | local(projB): ['serena'] | .mcp.json: -
    $ headroom wrap claude --no-proxy --code-memory none -- --version    (cwd=projB, build=56557dc9)
    exit=0
    wrap|   Serena MCP is present but user-managed — leaving it in place (Headroom only removes entries it installed)
    wrap| 2.1.318 (Claude Code)
    state| user: ['headroom'] | local(projB): ['serena'] | .mcp.json: -

== 3. Explicit opt-in, then rollback to the default
    $ headroom wrap claude --no-proxy --code-memory serena -- --version    (cwd=projC, build=56557dc9)
    exit=0
    wrap|   Serena MCP: registered (restart Claude Code if it was already running)
    wrap|     scoped to this project — pass --code-memory-scope user to register it for every Claude Code session
    wrap| 2.1.318 (Claude Code)
    state| user: ['headroom'] | local(projC): ['serena'] | .mcp.json: -
    $ headroom wrap claude --no-proxy -- --version    (cwd=projC, build=56557dc9)
    exit=0
    wrap|   Removed previously-installed Serena MCP (Serena is now off by default; pass --code-memory serena to keep it)
    wrap| 2.1.318 (Claude Code)
    state| user: ['headroom'] | local(projC): - | .mcp.json: -
    $ headroom wrap claude --no-proxy -- --version    (cwd=projC, build=56557dc9)
    exit=0
    wrap|   Serena MCP: registered (restart Claude Code if it was already running)
    wrap|     scoped to this project — pass --code-memory-scope user to register it for every Claude Code session
    wrap| 2.1.318 (Claude Code)
    state| user: ['headroom'] | local(projC): ['serena'] | .mcp.json: -
    $ headroom wrap claude --no-proxy --serena -- --version    (cwd=projC, build=56557dc9)
    exit=0
    wrap| 2.1.318 (Claude Code)
    state| user: ['headroom'] | local(projC): ['serena'] | .mcp.json: -

== 4. User scope: installed by Headroom with --code-memory-scope user
    $ headroom wrap claude --no-proxy --code-memory-scope user -- --version    (cwd=projD, build=12864fc8)
    exit=0
    wrap|   Serena MCP: registered (restart Claude Code if it was already running)
    wrap|     registered for every Claude Code session on this machine (--code-memory-scope user)
    wrap| 2.1.318 (Claude Code)
    state| user: ['headroom', 'serena'] | local(projD): - | .mcp.json: -
    $ headroom wrap claude --no-proxy -- --version    (cwd=projD, build=56557dc9)
    exit=0
    wrap|   Removed previously-installed Serena MCP (Serena is now off by default; pass --code-memory serena to keep it)
    wrap| 2.1.318 (Claude Code)
    state| user: ['headroom'] | local(projD): - | .mcp.json: -

== 5. User scope opt-in on the PR build
    $ headroom wrap claude --no-proxy --code-memory serena --code-memory-scope user -- --version    (cwd=projE, build=56557dc9)
    exit=0
    wrap|   Serena MCP: registered (restart Claude Code if it was already running)
    wrap|     registered for every Claude Code session on this machine (--code-memory-scope user)
    wrap| 2.1.318 (Claude Code)
    state| user: ['headroom', 'serena'] | local(projE): - | .mcp.json: -
    $ headroom wrap claude --no-proxy -- --version    (cwd=projE, build=56557dc9)
    exit=0
    wrap|   Removed previously-installed Serena MCP (Serena is now off by default; pass --code-memory serena to keep it)
    wrap| 2.1.318 (Claude Code)
    state| user: ['headroom'] | local(projE): - | .mcp.json: -
```

What this shows:
1. A Serena entry Headroom installed on `main` is removed on the first
wrap after upgrading, with the "off by default" message. This holds for
both scopes (cases 1 and 4).
2. A hand-added Serena entry is kept, and the default wrap prints
nothing. `--code-memory none` keeps it and says it's user-managed (case
2).
3. Opting in works with `--code-memory serena`,
`HEADROOM_CODE_MEMORY=serena` and the deprecated `--serena`, at project
scope and user scope (cases 3 and 5). Running the default afterwards
rolls it back.
4. Project scope here means Claude Code's `local` scope, which is where
wrap registers. Wrap never writes `.mcp.json`.

Not tested live: `wrap codex|grok|opencode` (unit tests only), Windows,
and an interactive Claude session that calls Serena tools.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4509"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

## Maintainer post-merge qualification
Published `983b6bb15119a3ecb8f6e20bf014b8bed123e95d` includes upstream
main `e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`. 19 selector/Serena
tests pass,1optional skip; Ruff/format/diff pass. Clean main sync;
contributor selector/ownership code unchanged. Existing real
upgrade/configuration/launch proof requirement remains; jCodeMunch
proposal is unrelated to this source delta. Contributor correctness
logic was not rewritten. Existing Changes Requested decision retained
without duplicate findings. Scoped Windows validation uses the existing
development native extension, not a fresh build. Hosted qualification is
handled by maintainers; nothing merged.


### Maintainer current-main requalification

Exact head `e8e3011187a2ffdfbb6c9932db26b22ec911ab39`, upstream main
`ecbaf86d78cf52d185fbffa306157b9a4cc1b906`.

Mechanically updated from upstream/main
ecbaf86d78 without changing contributor
selector/ownership logic. Reviewed the current discussion and
integration delta. All 29 focused code-memory/Serena-disable/Gemini
tests pass with one optional skip; Ruff0.17.3 lint/format and whitespace
checks pass. Existing Windows HOME/USERPROFILE fixture isolation remains
intact.

The prior upgrade/runtime qualification requirement remains unresolved:
post-patch real configuration/launch evidence must cover an existing
Headroom-owned Serena entry, a hand-managed entry, explicit opt-in, and
Claude project/user scopes, with commands/results and untested-agent
limits. No new author evidence was found to close it. This pass does not
claim those actual launch/upgrade flows were executed; passing unit
tests and the main sync do not establish that deployment behavior. The
hold is not an inherited-main CI defect. Maintainers handle hosted
checks; no contributor CI execution requested. Nothing merged.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `0b49a79cb192b294dde7ecb1c72b75fffa2733d5`, upstream main
`8d3e9e5ca7cd645eb73084b848edbea5fc796e1a`.

Clean mechanical sync with upstream
main8d3e9e5ca7cd645eb73084b848edbea5fc796e1a without changing
contributor selector/ownership logic. Current source, author discussion
and Devin findings reviewed.29focused code-memory/Serena-disable/Gemini
tests pass with1optional skip; changed Python Ruff0.17.3check/format and
whitespace pass. Windows HOME/USERPROFILE fixture isolation preserved.

Existing real upgrade/runtime qualification remains missing: existing
Headroom-owned Serena versus hand-managed entry, explicit opt-in, and
Claude project/user scopes need observed configuration/launch results
with explicit untested-agent limits. No new evidence found to close it;
these launch/upgrade paths were not executed in this pass. Passing unit
tests and clean main integration do not establish deployment behavior.
No new provider/native-build claim or inherited-main pipeline
attribution. Maintainers handle hosted checks; nothing merged.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `30dfb386fa7284c9817bf5d7db0648fb10830055`, upstream main
`7592e681459f14a2041eb5f524a658cb6ad1dd92`.

Clean current-main
sync7592e681459f14a2041eb5f524a658cb6ad1dd92,including HNSW re-index and
memory relation-key fixes after917release-recovery merge. Contributor
selector/ownership logic unchanged; fresh discussion
considered.37wrapper/Serena/memory-regression/Gemini tests pass
with1optional skip;whitespace passes. PriorRuff0.17.3validation applies
to unchanged PR files.

Existing real upgrade/launch evidence hold remains: Headroom-owned
versus hand-managed Serena
entries,explicitoptin,andClaudeproject/userscopes need
observedpostpatchresults. No newevidence closesit;actualupgrade/launch
pathsnotrerun. No freshnativebuild/provider claim.
MaintainershandleCI;nothingmerged.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `e1778a5368626547ad958926d5570edf153182f6`, upstream main
`2d3ee51846435ef5b320eea4806af14e602b0d8c`.

Clean integration with upstream/main
2d3ee51846 after the external 0.43.1
release merge. Exact headroom/crates/tests comparison against previously
reviewed 30dfb386f shows no changes. Re-reviewed generated metadata; 78
release/version/MCP tests pass on this tree and whitespace passes.
Earlier 37 wrapper/Serena/memory/Gemini passes and one optional skip
remain scoped to unchanged inputs.

Existing actual upgrade/launch proof hold remains: Headroom-owned versus
hand-managed Serena entries, explicit opt-in, and Claude project/user
scopes need observed postpatch results. Those runtime paths were not
rerun and no new evidence closes the hold. No contributor ownership
policy changed, no fresh native-build/provider claim. Maintainers handle
checks; nothing merged.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `173c5739d7357ca9398faa41b4d1ce3b72d6c550`, upstream main
`7877241d84c93b3ec487e07324245838ddae85cc`.

Cleanly integrated upstream/main7877241d84c93b3ec487e07324245838ddae85cc
and re-reviewed the wrapper overlap with Hermes registration and port
discovery.35 selector/Serena-disable/Hermes provider and wrapper tests
pass,1optional skip; Ruff and whitespace pass. Contributor
ownership/default policy is unchanged. Existing actual
upgrade/configuration/launch proof requirement remains open for
Headroom-owned versus user-managed entries, explicit opt-in, and Claude
project/user scopes. No runtime proof is inferred from mocks or
native-build claim made. Maintainers handle hosted checks; nothing
merged.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `eb1c3d66669cf71f881f1ba06dc049d36b20da7b`, upstream main
`c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`.

Final post-Dependabot requalification against upstream/main
c01cf07d94. All six Dependabot updates are
now merged. Relative to this branch's freshly tested prepared tree, the
final main sync changes only the three intended Cargo.lock records;
Python/source/test inputs are unchanged. Re-reviewed the integration and
confirmed complete native source/lock/toolchain objects match the tested
combined build and exact fresh native DLL used for this branch's local
tests.

Focused fresh-native qualification: 35 passed, 1 skipped in 2.93s.
Selected real dependency overlays and native DLL paths are recorded in
final4509-fresh-native-tests.log; the DLL provenance/input hashes are in
final-native-artifact-proof.json. No redundant rerun is implied after
this Cargo-only merge. Whitespace passes. Prior source review and scoped
real-behavior evidence remain applicable to unchanged contributor
inputs.

Existing real upgrade/configuration/launch qualification remains open
for Headroom-owned versus user-managed Serena entries, explicit
opt-in/rollback, and Claude project/user scopes. No new author evidence
closes it; contributor ownership/default policy is unchanged.

Maintainers handle hosted checks; no external-provider claim,
contributor semantic rewrite or non-Dependabot merge.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `43bfea8a164c21ab69793b3905fbf1457fef7bc2`, upstream main
`da6d9736cd82074988c7d0a2360b49d9d8b6c0a0`.

Current-main requalification against
da6d9736cd. All69 integration patches
exactly match c01cf07-to-current-main delta after diff-location
normalization; contributor selector/default/Serena ownership logic
unchanged.162 focused code-memory,Serena disable,Hermes,Windows
supervisor,search-compressor tests pass;3 platform/optional skips. All31
changed Python files pass Ruff lint/format; offline lock and whitespace
pass. Native pr3982 DLL verified against complete tracked native-input
hash proof and artifact checksum; no fresh native-build claim.
Author/Devin discussion checked. Existing real
upgrade/configuration/launch qualification remains open: prior
Headroom-owned Serena entry versus user-managed entry, explicit
opt-in/rollback, Claude project/user scopes. No new author evidence
closes it; this run does not assert actual agent launch or deployment
proof. Maintainers handle hosted checks; no contributor policy rewrite
or merge.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `56557dc91725422257f90c3cb5c5f04a665e6043`, upstream main
`0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`.

Clean integration with upstream/main
0ddaa56c21. All 19 integration patches
exactly match the prior-reviewed-main da6d9736 to current-main delta
after diff-location normalization; contributor selector/default/Serena
ownership logic is unchanged. Fresh code-memory/Serena
disable/scope/reconciliation/migration, doctor and incoming
Vertex/backend/CLI tests: 330 passed, 12 skipped, 1 warning. Fourteen
changed Python files pass Ruff lint/format; offline lock and whitespace
checks pass.

Author/Devin discussion read. Existing real upgrade/configuration/launch
qualification remains open: prior Headroom-owned entry versus
user-managed entry, explicit opt-in/rollback, Claude project/user
scopes. This run establishes no actual agent launch or deployment proof.
Static source/test scope and security inspection found no new workflow,
dependency or setup path; the inherited flagged workflow-validation job
is excluded by the exact PR's unchanged workflow paths. Native DLL
reused after complete tracked input-set/hash and artifact-checksum
verification; no fresh build claimed. Maintainers handle exact-head
hosted checks; no PR merged.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-09 12:23:08 -06:00
4a851ed254 fix(codex): accept multi-line inline tables in Codex config.toml (#4436)
## Description

Codex accepts an inline table split across lines, with a trailing comma,
in `~/.codex/config.toml`:

```toml
env = {
  DINGTALK_Client_ID = "**********",
}
```

That is TOML 1.1 syntax. Headroom reads the Codex config with the
standard library `tomllib`, which only implements TOML 1.0 before Python
3.15, so it rejects the file with `Invalid initial character for a key
part`. The result is what the issue shows: `headroom wrap codex` and
`headroom mcp install` refuse to register MCP servers ("is not valid
TOML ... refusing to overwrite") and session routing fails with "could
not read Codex config for session routing", even though Codex itself is
happy with the file.

`tomlkit` is already a runtime dependency (used for Codex config
recovery) and parses this syntax from 0.15.0. 0.13.3 and 0.14.0 both
fail on it (`Empty key at line 1 col 7`).

The fix adds one helper, `loads_codex_config()`, that parses with
`tomllib` first and only on `TOMLDecodeError` retries with `tomlkit`. If
`tomlkit` rejects the text too, it re-raises the original
`tomllib.TOMLDecodeError`, so every caller keeps its existing `except
tomllib.TOMLDecodeError` handling and its error messages unchanged. All
places that parse the user's Codex config now go through it.

The follow-up commit (`dc193870`) fixes a problem the first commit
exposed: once a multi-line inline table parses, a `profiles = {` table
can carry per-profile `model_provider` / `openai_base_url` keys on their
own indented lines, and the line-based "root key" edits in init, the
provider-scope install and persistent wrap mistook those nested lines
for root assignments. Init and install deleted the profile's provider,
and wrap rewrote it to `headroom` and then skipped adding a real root
`model_provider`. Those edits now only touch lines that are actual root
assignments.

Fixes #4378

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/providers/codex/config_toml.py` (new):
`loads_codex_config(content)`, `tomllib` first, `tomlkit` fallback,
original `tomllib` error if both fail.
- Routed the Codex config parse sites through it (15 call sites, 16
parse calls):
- `headroom/cli/wrap.py`: session routing read, the user-content check
and its fallback in `_inject_codex_provider_config`, and the two "never
write a malformed candidate" checks.
- `headroom/cli/init.py`: provider snapshot check, the candidate check
in `_ensure_codex_provider`, and `_codex_features`.
- `headroom/providers/codex/install.py`: `_codex_credential_store`,
`codex_auth_helper_is_referenced`, and the merged-config check before
writing.
- `headroom/providers/codex/project_context.py`: the `sqlite_home`
lookup.
- `headroom/mcp_registry/codex.py`: `_load_toml`, `_unmergeable_reason`,
and both sides of `_only_server_changed`. This registrar class is shared
with Grok, so a Grok `config.toml` with the same syntax is accepted too.
- Left on `tomllib` on purpose: `_table_path` (parses a single `[table]`
header line) and the auth snippet check in `build_codex_auth_config`
(validates text Headroom generates itself).
- `pyproject.toml` / `uv.lock`: raise the `tomlkit` minimum from
`>=0.13.0` to `>=0.15.0`, the first release that parses multi-line
inline tables. The lock already resolved 0.15.0, so `uv lock` only
changed the specifier line.
- Write paths: the user's text is never re-serialized by this change.
The registrar and the wrap/init injectors still append or splice marker
blocks as text, and the existing `tomlkit` round-trip in wrap keeps the
multi-line table byte for byte (checked below).
- Tests:
- `tests/test_provider_codex_config_toml.py`: the helper on the exact
shape from the issue (multi-line, trailing comma, whitespace-only line,
indented `}`), and that invalid TOML still raises
`tomllib.TOMLDecodeError`.
-
`tests/test_mcp_registry/test_codex_registrar.py::test_register_accepts_multiline_inline_table`:
registering into that config succeeds and the user's text is kept as a
byte-for-byte prefix.
-
`tests/test_cli/test_wrap_codex.py::test_codex_session_launch_settings_accept_multiline_inline_table`:
session routing reads that config and leaves the file untouched.
- Follow-up commit `dc193870` (multi-line inline profile overrides):
- `headroom/providers/codex/config_toml.py`:
`root_assignment_lines(content)` returns the indexes of lines that start
a key/value pair at the document root. It stops at the first table
header and skips lines inside a multi-line inline table, array or
string, with a small scanner that tracks quotes, comments and bracket
depth.
- `headroom/cli/init.py` (`_ensure_codex_provider`) and
`headroom/providers/codex/install.py`
(`_strip_root_provider_assignments`) only drop `model_provider` /
`openai_base_url` lines that are root assignments.
- `headroom/cli/wrap.py`: `_redirect_existing_top_level_keys` and
`_has_redirectable_top_level_key` only look at root assignments. Before,
they matched the key on any line, so a `[profiles.work]` section
override was also rewritten to `headroom` and counted as a root key;
that now stays the user's, matching how init, install and
`_strip_codex_headroom_blocks` already treat `[profiles.*]`.
- Tests:
`test_root_assignment_lines_skip_keys_nested_in_multiline_inline_table`,
`test_ensure_codex_provider_preserves_multiline_inline_profile_overrides`
(init),
`test_provider_scope_preserves_multiline_inline_profile_overrides`
(install) and `test_inject_preserves_multiline_inline_profile_overrides`
(wrap).

## Testing

- [x] Unit tests pass (`pytest`) - Codex, MCP registry, init and Grok
suites: 774 passed, 2 failed (the 2 failures are in
`tests/test_codex_responses_passthrough_bytes.py` and fail the same way
on `main` without this change)
- [x] Linting passes (`ruff check .`) - `ruff check` and `ruff format
--check` on the 9 changed Python files
- [x] Type checking passes (`mypy headroom`) - `mypy` on the 6 changed
source files, no issues
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

New end-to-end tests on `main` (fix stashed):

```text
$ pytest tests/test_mcp_registry/test_codex_registrar.py::test_register_accepts_multiline_inline_table tests/test_cli/test_wrap_codex.py::test_codex_session_launch_settings_accept_multiline_inline_table -q
E   AssertionError: .../.codex/config.toml is not valid TOML (Invalid initial character for a key part (at line 3, column 8)); refusing to overwrite. Fix or remove the file, then re-run.
E   click.exceptions.ClickException: could not read Codex config for session routing: Invalid initial character for a key part (at line 5, column 8)
============================== 2 failed in 0.38s ===============================
```

(`tests/test_provider_codex_config_toml.py` cannot import on `main`
because the helper does not exist yet.)

With the fix:

```text
$ pytest tests/test_provider_codex_config_toml.py tests/test_mcp_registry/test_codex_registrar.py::test_register_accepts_multiline_inline_table tests/test_cli/test_wrap_codex.py::test_codex_session_launch_settings_accept_multiline_inline_table -q
============================== 4 passed in 0.56s ===============================

$ pytest tests/test_provider_codex_config_toml.py tests/test_mcp_registry tests/test_cli/test_wrap_codex.py tests/test_cli/test_init_cli.py tests/test_install/test_codex_install.py tests/test_provider_codex*.py tests/test_provider_grok*.py tests/test_codex*.py -q
FAILED tests/test_codex_responses_passthrough_bytes.py::test_unmutated_zstd_post_forwards_decoded_bytes_and_strips_content_encoding - concurrent.futures._base.CancelledError
FAILED tests/test_codex_responses_passthrough_bytes.py::test_unmutated_plain_post_passes_original_bytes_through - concurrent.futures._base.CancelledError
================== 2 failed, 774 passed, 8 warnings in 50.43s ==================

$ pytest tests/test_codex_responses_passthrough_bytes.py -q   # on main, without this change
========================= 2 failed, 1 warning in 4.09s =========================

$ mypy headroom/cli/init.py headroom/cli/wrap.py headroom/mcp_registry/codex.py headroom/providers/codex/install.py headroom/providers/codex/project_context.py headroom/providers/codex/config_toml.py
Success: no issues found in 6 source files
```

Follow-up commit, the three new write-path tests on the first commit
(`dfe93720`, source changes reverted):

```text
$ pytest tests/test_cli/test_init_cli.py tests/test_install/test_codex_install.py tests/test_cli/test_wrap_codex.py -k multiline_inline_profile -q
FAILED tests/test_cli/test_init_cli.py::test_ensure_codex_provider_preserves_multiline_inline_profile_overrides - AssertionError: assert {} == {'model_provi...y.example/v1'}
FAILED tests/test_install/test_codex_install.py::test_provider_scope_preserves_multiline_inline_profile_overrides - AssertionError: assert {} == {'model_provider': 'company'}
FAILED tests/test_cli/test_wrap_codex.py::TestSubscriptionRouting::test_inject_preserves_multiline_inline_profile_overrides - KeyError: 'model_provider'
====================== 3 failed, 220 deselected in 0.56s =======================
```

With `dc193870`:

```text
$ pytest tests/test_provider_codex_config_toml.py tests/test_cli/test_init_cli.py tests/test_install/test_codex_install.py tests/test_cli/test_wrap_codex.py -k "multiline_inline_profile or root_assignment" -q
======================= 4 passed, 222 deselected in 0.27s =======================

$ pytest tests/test_provider_codex_config_toml.py tests/test_cli/test_init_cli.py tests/test_install/ -q
================== 346 passed, 13 skipped, 1 warning in 3.94s ==================
$ pytest tests/test_cli/test_wrap_codex.py tests/test_cli/test_port_alignment.py -q
============================= 167 passed in 6.01s ==============================
$ pytest tests/test_mcp_registry/ tests/test_provider_codex_install.py tests/test_cli/test_install_cli.py tests/test_cli/test_recover_codex.py -q
============================= 364 passed in 4.71s ==============================
$ pytest tests/test_cli/test_wrap_persistent.py tests/test_codex_project_context.py tests/test_cli/test_wrap_stale_marker.py tests/test_cli/test_wrap_dead_marker_selfheal.py tests/test_provider_codex_runtime.py tests/test_install_tool_env_reconcile.py -q
======================== 4 failed, 261 passed in 9.01s =========================
```

The 4 failures are in `tests/test_cli/test_wrap_persistent.py`
(`AttributeError: '_Manifest' object has no attribute 'runtime_kind'`)
and fail the same way with this PR's changes stashed. `ruff check`,
`ruff format --check` (8 files) and `mypy` on the 4 changed source files
are clean.

tomlkit versions on the issue's snippet:

```text
0.13.3 FAIL Empty key at line 1 col 7
0.14.0 FAIL Empty key at line 1 col 7
0.15.0 {'env': {'A': 'x'}}
```

## Real Behavior Proof

- Environment: macOS 26.2 (Apple Silicon), Python 3.14.7, repo venv with
the `mcp` package installed, a throwaway `HOME`/`CODEX_HOME` whose
`config.toml` holds the issue's config:

  ```toml
  [mcp_servers.dingtalk]
  command = "npx"
  env = {
    DINGTALK_Client_ID = "**********",
    
      }
  ```

- Exact command / steps: `HOME=<tmp>/home CODEX_HOME=<tmp>/home/.codex
python -m headroom.cli mcp install --agent codex`, then compare the
start of the resulting file with the original bytes and count
`[mcp_servers.headroom]` tables. Also called
`wrap._inject_codex_provider_config(8787)` and
`init._ensure_codex_provider(path, 8787)` on the same config to check
the write paths.
- Observed result: before (`main`):

  ```text
codex: install failed (failed): <tmp>/home/.codex/config.toml is not
valid TOML (Invalid initial character for a key part (at line 3, column
8)); refusing to overwrite. Fix or remove the file, then re-run.
  ```

  After:

  ```text
  codex: registered (restart the agent if it was already running)
  ```

The user's original bytes are an exact prefix of the new file and one
`[mcp_servers.headroom]` table was appended. The wrap injector printed
`Codex config: injected Headroom provider (WS + HTTP)`, and in both the
wrap and init outputs the user's multi-line table is still present
verbatim, the file parses, `env` is `{'DINGTALK_Client_ID':
'**********'}` and `model_provider` is `headroom`.
- Follow-up commit: with a throwaway `HOME`/`CODEX_HOME` holding

  ```toml
  model_provider = "openai"
  profiles = {
    work = {
      model_provider = "company",
    },
  }
  ```

I ran `init._ensure_codex_provider`, `install.apply_provider_scope`
(provider scope) and `wrap._inject_codex_provider_config` (same config
without the root key) and parsed the result:

  ```text
  == before (dfe93720)
  init:   root = headroom  work = {}
  install: root = headroom  work = {}
  wrap:   root = None  work = {'model_provider': 'headroom'}
  == after (dc193870)
  init:   root = headroom  work = {'model_provider': 'company'}
  install: root = headroom  work = {'model_provider': 'company'}
  wrap:   root = headroom  work = {'model_provider': 'company'}
  ```

- Not tested: a full `headroom wrap codex` run with a live proxy and the
real Codex CLI, and Python 3.10 (where `tomli` is the first parser; the
`tomlkit` fallback is the same there).

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: only for Codex configs `tomllib`
rejects but `tomlkit` 0.15 accepts (TOML 1.1 multi-line inline tables).
Valid TOML 1.0 still goes through `tomllib` exactly as before, and text
both parsers reject fails with the same error as before.
- Kill switch / disable path: none needed; the fallback only runs after
`tomllib` has already failed.
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the two commits

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`, it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- Documentation: no user-facing docs describe which TOML syntax the
Codex config may use, so nothing to update.
- Python 3.15's `tomllib` implements TOML 1.1 and parses this config
directly (as noted in the issue thread), so on 3.15 the fallback simply
never runs.
- `tomlkit` is a bit more permissive than `tomllib` only where TOML 1.1
is; it still rejects duplicate keys and malformed tables, so the
existing "refuse to overwrite an unparseable file" guards still trigger
for genuinely broken configs (covered by the existing registrar tests).
- Dependency review for the `tomlkit` minimum bump (`>=0.13.0,<1.0` to
`>=0.15.0,<1.0`):
- Why this package: it is already a direct runtime dependency (Codex
config recovery and the memory MCP block edit in `wrap.py`), so no new
package enters the tree. Writing a TOML 1.1 parser here is not worth it,
and `tomllib` only gains TOML 1.1 in Python 3.15.
- Who maintains it: the Poetry project (`python-poetry/tomlkit`, MIT),
with Frost Ming and Sébastien Eustace as the main maintainers. It is
actively released: 0.13.3 (2025-06), 0.14.0 (2026-01), 0.15.0 (2026-05),
0.15.1 (2026-07), 0.15.2 (2026-10). No GitHub security advisories are
published for the repo.
- Install surface: unchanged. Pure Python, a single `py3-none-any` wheel
(about 41 KB for 0.15.0), no runtime dependencies, no native code, no
install-time or runtime network access. `uv.lock` already resolved
0.15.0 before this PR, so the locked version and hashes do not change;
only the specifier line moves.
- Why this version: required new functionality. 0.15.0 is the first
release with TOML 1.1 support ("feat(spec): update to toml spec v1.1",
python-poetry/tomlkit#456), which is what parses multi-line inline
tables with trailing commas. 0.13.3 and 0.14.0 both reject the issue's
snippet (`Empty key at line 1 col 7`, output above). 0.15.0 also
includes an upstream parser hang fix (python-poetry/tomlkit#470).

---------

Co-authored-by: Abelo9996 <Abelo9996@users.noreply.github.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-28 23:31:18 -05:00
Tejas ChopraandClaude Opus 5.5 976aa714ef fix(cli): name the broken Codex config, write init configs atomically, keep the identity check (#4129)
## Description

This PR fixes four small CLI problems.

1. **`wrap codex` with a broken `~/.codex/config.toml`.** The warning
looked like this:

   ```
Warning: could not update Codex config: Invalid value (at line 6, column
10)
   ```

The line number pointed into Headroom's own merged text, not the user's
file, and the file was never named. The warning now names the file,
gives the parse error in the user's own lines, and says what to do. The
generic Codex and Grok config warnings now also name the file.

2. **`headroom init` truncated config files while rewriting them.** It
rewrote `~/.claude/settings.json`, Codex `hooks.json` and
`~/.codex/config.toml` with `Path.write_text`. That truncates the file
before writing, so a crash or ENOSPC mid-write left a half-written file.
`init` now uses `fsutil.write_text`, the atomic writer `wrap` already
uses for these same files.

3. **The runtime identity check could be silently skipped.** `init` and
`wrap` each had a `_wait_for_runtime_ready` wrapper. It caught
`TypeError` "for test doubles" and then retried `wait_ready` without
`require_identity=True`. So any real `TypeError` inside `wait_ready`
dropped the runtime identity check. Both wrappers are removed, and
callers now pass `require_identity=True` directly.

4. **A failed hook recovery left no trace.** `_ensure_profile_running`
(the init hook's recovery path) ended in `except Exception: return`. It
now catches only what the start paths raise (`click.ClickException`,
`RuntimeError`, `OSError`, `subprocess.SubprocessError`) and logs one
WARNING after the suppressed-output block: `headroom: could not start
persistent proxy '<profile>': <error>. Check it with `headroom install
status --profile <profile>`.` A start that succeeds but never becomes
ready within 45s logs the same line with `it did not become ready within
45s` as the reason. It goes to stderr; stdout stays clean for the hook.
Any other exception is a bug and now surfaces.

Refs #4136
## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/cli/wrap.py` `_inject_codex_provider_config`: it now parses
the user's own content, with Headroom blocks stripped, before building
the candidate.
- If that content does not parse, it warns `could not update Codex
config <path>: it is not valid TOML (<user's own parse error>); Codex
WebSocket traffic will bypass Headroom. Fix or remove the file, then
re-run.`
- If the content parses only once the Headroom block is removed, the old
repair path runs instead.
- The snapshot still happens first, so `unwrap codex` restores the file
exactly as before.
- `headroom/cli/wrap.py`: the generic Codex and Grok config warnings now
include the file path (`grok_config_paths()`).
- `headroom/cli/init.py`: `_write_json`, `_ensure_codex_provider` and
the Codex feature-block writer now call `fsutil.write_text`.
- `headroom/cli/init.py` and `headroom/cli/wrap.py`: removed
`_wait_for_runtime_ready`; callers call `wait_ready(...,
require_identity=True)` directly.
- `headroom/cli/init.py` `_ensure_profile_running`: narrowed `except
Exception: return` to the start-path exceptions and log the failure
once, as above.
- Tests:
- New `test_invalid_user_config_is_named_and_left_untouched` in
`test_wrap_codex.py`.
- New `test_write_json_keeps_the_existing_file_when_the_write_fails` in
`test_init_cli.py`. It simulates ENOSPC at fsync and checks that the
original file survives and no temp file is left.
- New
`test_ensure_profile_running_never_retries_wait_ready_without_identity`:
its `wait_ready` double raises `TypeError` whenever `require_identity`
is passed. It asserts the error propagates and `wait_ready` was called
once, with `require_identity=True`. It fails if the fallback comes back.
- New `test_ensure_profile_running_logs_a_failed_recovery`: no log on a
clean start; exactly the one WARNING above when `start_supervisor`
raises.
- New
`test_ensure_profile_running_logs_a_recovery_that_never_becomes_ready`:
the start succeeds, the 45s `wait_ready` returns False, and the WARNING
names that reason.
- The `wait_ready` doubles in `test_init_cli.py` and
`test_wrap_persistent.py` now accept, or assert, `require_identity`;
`test_recover_persistent_proxy_routes_docker_task_to_docker_supervisor`
asserts `require_identity=True`.
  - All four new tests fail on origin/main and pass here.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest -q -p no:cacheprovider tests/test_cli/test_init_cli.py tests/test_cli/test_wrap_persistent.py tests/test_cli/test_wrap_codex.py tests/test_cli/test_wrap_grok.py tests/test_cli/test_init_enable_tool_search.py tests/test_cli/test_install_cli.py tests/test_install/test_providers.py tests/test_install/test_runtime.py tests/test_plugin_manifests.py tests/test_cli/test_wrap_helpers.py tests/test_cli/test_port_alignment.py
4 failed, 595 passed in 29.13s
# The 4 failures are in tests/test_cli/test_wrap_persistent.py ("'_Manifest' object has no attribute 'runtime_kind'"). They are pre-existing: clean origin/main gives "4 failed, 59 passed" for that file.

# The new tests against origin/main code:
test_invalid_user_config_is_named_and_left_untouched: 1 failed (old message names no file)
test_write_json_keeps_the_existing_file_when_the_write_fails: 1 failed (DID NOT RAISE: the old write_text never fsyncs and overwrote the file)
test_ensure_profile_running_never_retries_wait_ready_without_identity: 1 failed (DID NOT RAISE: the fallback retried without require_identity)
test_ensure_profile_running_logs_a_failed_recovery: 1 failed (no log record)

$ uvx ruff@0.16.8 check headroom/cli/init.py headroom/cli/wrap.py tests/test_cli/test_init_cli.py tests/test_cli/test_wrap_persistent.py tests/test_cli/test_wrap_codex.py
All checks passed!
$ mypy headroom/cli/init.py headroom/cli/wrap.py --ignore-missing-imports
Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: macOS 25.6, Python 3.12, repo `.venv`.
- Exact command / steps:
1. Ran `python -m headroom.cli wrap codex --prepare-only` with a scratch
`HOME` whose `.codex/config.toml` is `model = [unterminated`. This is
the audit's reproduction, and the new test covers it.
2. Called `_recover_persistent_proxy` and `_restart_persistent_proxy`
directly, with the supervisor and `wait_ready` stubbed, and recorded the
arguments `wait_ready` received.
- Observed result:
1. Before: `Warning: could not update Codex config: Invalid value (at
line 6, column 10)`. After: the warning names
`<HOME>/.codex/config.toml`, reports the error at line 1 (the user's
line), and tells the user to fix or remove the file.
2. Both calls passed `{'timeout_seconds': 45, 'require_identity':
True}`.
- Smoke harness (real proxy started from this checkout against a local
fake upstream; temp HOME and workspace, offline):

```text
branch: cleanup/cli-integrations-config-errors-atomic-init @ 08386d03
health: 200
POST /v1/messages        -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines (<tmp>/ws/logs/proxy-49764.log):
   2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes.
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.
(smoke dir: <tmp>)
```
- Direct runtime command on this PR's own path (temp HOME,
HEADROOM_WORKSPACE_DIR and CODEX_HOME):

```text
## pr5
$ HOME=<tmp>/home HEADROOM_WORKSPACE_DIR=<tmp>/ws CODEX_HOME=<tmp>/codex headroom init hook ensure --profile hookdemo
exit=0
--- stdout

--- stderr
headroom: could not start persistent proxy 'hookdemo': [Errno 2] No such file or directory: 'docker'. Check it with `headroom install status --profile hookdemo`.
## clean
$ HOME=<tmp>/home HEADROOM_WORKSPACE_DIR=<tmp>/ws CODEX_HOME=<tmp>/codex headroom init hook ensure --profile hookdemo
exit=0
--- stdout

--- stderr
## pr5
$ HOME=<tmp>/home HEADROOM_WORKSPACE_DIR=<tmp>/ws CODEX_HOME=<tmp>/codex headroom wrap codex --prepare-only
exit=0
--- stdout
  Warning: could not update Codex config <tmp>/codex/config.toml: it is not valid TOML (Invalid value (at line 1, column 10)); Codex WebSocket traffic will bypass Headroom. Fix or remove the file, then re-run.
--- stderr
## clean
$ HOME=<tmp>/home HEADROOM_WORKSPACE_DIR=<tmp>/ws CODEX_HOME=<tmp>/codex headroom wrap codex --prepare-only
exit=0
--- stdout
  Warning: could not update Codex config: Invalid value (at line 6, column 10)
--- stderr
```
- Not tested: a real disk-full condition; the regression test simulates
ENOSPC at fsync. Windows was not tested either.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A, not rollout-managed.
- Stable/default behavior changed: Yes, in small ways:
- The warning text for an invalid Codex config changes. It stays
non-fatal, and `--prepare-only` still exits 0.
- `init` writes its config files atomically. On Windows they get LF line
endings, the same as `wrap` already writes. New files get mode 0600 from
mkstemp, and existing files keep their mode.
  - `wait_ready` is always called with the identity check.
- A failed init-hook recovery now writes one WARNING line to stderr
instead of nothing.
- Kill switch / disable path: None needed.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review.
- Docs impact is recorded in the cleanup docs ledger; there are no doc
edits in this PR.
- **For the maintainer:** should `wrap codex --prepare-only` exit
non-zero when it cannot inject? `scripts/install.sh` relies on that
command. I left it non-fatal, as before.
- **Test-isolation hazard (separate issue, not fixed here):** while
checking the 4 pre-existing `test_wrap_persistent.py` failures, I added
a `runtime_kind` attribute to its `_Manifest` fixture locally. With that
one change, `test_ensure_proxy_restarts_*` started a **real** `headroom
proxy --port 8787` with the real `HOME`. Those tests do not fully stub
process start.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4129"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 09:06:21 -07:00
a6523b1967 fix(codex): preserve auth.json API keys through Headroom (#3355)
## Description

Closes #3354.

When Codex uses an OpenAI API key stored in auth.json, Headroom's
generated provider block did not configure authentication. Codex
therefore sent requests to the local Headroom proxy without an
Authorization header and received 401 Missing bearer.

This is separate from #3206: that issue covers ChatGPT OAuth
authentication, while this change covers file-backed API-key
authentication.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring

## Changes Made

- Resolve API-key and OAuth credentials from `codex_home_dir()` for both
project and user configurations, honoring `CODEX_HOME`.
- Generate a private, config-specific helper beside `auth.json`; API
keys stay out of generated TOML and Headroom environment variables.
- Configure Codex's supported `auth.command` field for persistent
install, init, and wrap.
- Serialize command and argument paths with literal Unicode and validate
generated TOML before writing provider configurations.
- Abort provider updates with an actionable error if a helper collides
with another file/symlink or cannot be created.
- Use exclusive creation, 0600 permissions on Unix, and content checks
for helper ownership.
- Give project and user configurations independent helper lifetimes.
Retain legacy shared helpers and helpers referenced by
restored/remaining providers or unparseable configurations.
- Preserve unrelated files and the original provider config when
authentication setup fails.
- Remove an unused config-specific helper on uninstall/unwrap.

## Testing

- [x] Focused unit and integration tests pass
- [x] Linting and formatting pass
- [ ] Type checking
- [x] New regression tests added
- [x] Manual runtime testing performed

### Test Output

```text
$ .venv/bin/python -m pytest tests/test_provider_codex_install.py tests/test_install/test_codex_install.py tests/test_cli/test_init_cli.py tests/test_install/test_providers.py tests/test_cli/test_wrap_codex.py -q
284 passed

$ .venv/bin/ruff check .
All checks passed!
$ .venv/bin/ruff format --check .
1744 files already formatted
```

Ruff 0.16.8 matches this branch's CI pin. A focused coverage run reports
93% line/branch coverage for `headroom/providers/codex/install.py` and
covers 44/45 newly added executable lines across the three production
files in this follow-up. This is local focused coverage, not a claim
about the hosted Codecov aggregate. Before the follow-up implementation,
the added local-init/Unicode/helper-isolation regressions produced 6
failures while 118 existing tests passed.

## Real Behavior Proof

- Environment: macOS 26.6.2, Apple Silicon arm64, Python 3.12.11,
Headroom source from this PR, Codex CLI 0.160.0, Sub2API
OpenAI-compatible provider, model `gpt-6-luna`, existing API key in the
user's Codex `auth.json`.
- Setup: start an isolated Headroom proxy on an unused loopback port
with `ProxyConfig(optimize=False, cache_enabled=False,
rate_limit_enabled=False, openai_api_url=<existing upstream>)`. Generate
a temporary provider config with `_ensure_codex_provider(config_path,
port)` while credentials resolve from the user's Codex home. Add `model
= "gpt-6-luna"` and `model_reasoning_effort = "low"` at the document
root. Run the child with its own temporary `CODEX_HOME` and no
`OPENAI_API_KEY` or `OPENAI_BASE_URL` environment variable; its home
contains the generated config, not a copied credential file.
- Exact child command:

```sh
CODEX_HOME=<temporary-home> codex exec --ephemeral --skip-git-repo-check --sandbox read-only --json -C <temporary-workspace> 'Reply with exactly OK. Do not call tools.'
```

Captured after-fix output on 2026-10-07:

```text
Headroom test proxy: ready (optimization disabled)
Codex: 0.160.0; model: gpt-6-luna; auth: generated command, no API-key environment variable
codex exec exit: 0
agent messages: ['OK']
Responses bearer matches auth.json: True
```

The last line comes from an ASGI wrapper that compares the
HTTP/WebSocket Responses request bearer with the file-backed key without
logging either value. The command completed against the real upstream;
no mocked model response was used. The temporary provider config/helper
and test proxy were removed after validation.

- Observed limitation: this upstream rejects WebSocket negotiation with
HTTP 426, so Headroom's existing HTTP streaming fallback completed the
request. This run proves file-backed command authentication and
successful proxy routing, not a successful upstream WebSocket session.
- Not tested in this follow-up: a new Windows/Linux runtime run, full
test suite, native-extension rebuild, or optimization behavior. The
focused Python suite reused the existing local native extension; no Rust
code changed.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: Yes, only when Codex has a
file-backed API key and Headroom writes a provider block.
- Kill switch / disable path: Remove the Headroom-managed provider block
or run the existing Codex unwrap command.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert the generated provider block; auth.json is
unchanged.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of the code
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [x] I did not edit CHANGELOG.md

## Additional Notes

Helpers contain no credential value and read the user's existing
credential file at runtime, so key rotation does not require
reinstalling. Helper names now include a hash of the owning config's
canonical path. Legacy fixed-name helpers are retained conservatively
because other project configurations may still use them.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3355"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-08 10:46:55 -05:00
326079517b fix(codex): make unwrap codex remove headroom init routing (#3920)
## Description

`headroom init -g codex` writes a block into `~/.codex/config.toml` that
points Codex at the local proxy:

```toml
# --- Headroom init provider ---
model_provider = "headroom"
openai_base_url = "http://127.0.0.1:8787/v1"
...
# --- end Headroom init provider ---
```

There was no command to take it out again. `headroom unwrap codex` only
recognises the markers that `headroom wrap codex` writes, so on an
init-written config it prints "Nothing to undo" and then "✓ Codex is no
longer routed through the Headroom proxy." The routing is still there.

This matters most on Windows. `init` relies on Codex session hooks to
start the proxy, and those hooks are disabled upstream on Windows (init
already prints a warning about that). So after a reboot or a proxy
crash, Codex keeps calling `127.0.0.1:8787`, nothing restarts the proxy,
and the obvious undo command claims it worked. That matches the symptom
in #3749: Codex stuck on a dead local proxy until the override was
deleted by hand.

This PR makes `unwrap codex` remove the init block. It also makes the
Windows warning say what it means for the user, and documents how to
remove init routing.

Refs #3749. It doesn't close the issue, which also asks wider questions
about the Windows lifecycle; I'll answer those on the issue.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/cli/wrap.py` (`_restore_codex_provider_config`): if the
config contains the `# --- Headroom init provider ---` marker, strip it
with the existing (already tested) `init._strip_codex_init_block` before
the usual wrap cleanup. The existing `cleaned` / `removed` / thread
re-tag flow then runs unchanged. `_codex_config_has_headroom_markers` is
deliberately not changed, because wrap's pre-wrap backup logic also uses
it.
- `headroom/cli/wrap.py` (`_restore_codex_provider_config`, backup
branch): a pre-wrap snapshot taken *after* `headroom init codex` carries
init's block, so restoring it verbatim left Codex pinned to the proxy
while unwrap printed "Restored". The snapshot is now passed through
`_strip_codex_init_block` too (and the config removed if nothing else is
left). The snapshot is read first and deleted only after the config has
been written, so a failed write leaves it in place for a retry instead
of losing it. Review findings from @JerrettDavis / Devin.
- `headroom/cli/init.py` (new `_strip_codex_root_routing_orphans`): the
orphan-key safety net (`model_provider = "headroom"`, loopback
`openai_base_url`) in both `_strip_codex_init_block` and wrap's
`_strip_codex_headroom_blocks` used to match anywhere in the file, so a
newly-reachable unwrap would delete user-owned `[profiles.*]` overrides.
It now only touches the document root, using the same first-table split
`_ensure_codex_provider` already uses. Neither writer ever puts these
keys inside a table. Review finding from @JerrettDavis / Devin.
- `headroom/cli/init.py` (`_init_codex`): on Windows, after the existing
hook warning, add one message. It says nothing starts the proxy there,
and points to `headroom install apply` for a supervised proxy and to
`headroom unwrap codex` (or deleting the project block) to remove the
routing.
- `docs/content/docs/proxy.mdx`: the "routing is persistent until you
remove it" sentence now says how to remove it, that unwrap leaves init's
Codex `SessionStart` hook in `hooks.json` (routing only), plus the
Windows caveat.
- Tests: five new `unwrap codex` cases: init block plus user content
(user content preserved), init-only file (file removed), init → `wrap
codex --prepare-only` → unwrap (snapshot restored without init routing),
the same sequence with the first unwrap's config write failing (snapshot
kept, a second unwrap restores it cleanly), and init → unwrap with a
`[profiles.work]` that sets `model_provider = "headroom"` + a loopback
`openai_base_url` (profile survives). One assertion added to the
existing Windows init warning test.
`TestSubscriptionRouting::test_strip_cleans_orphaned_openai_base_url`
used a key inside `[profiles.default]` as its "orphan"; it now puts the
orphan at the root and asserts a profile-level key survives.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

Both new tests fail on `main`; the failure reproduces the bug:

```text
tests/test_cli/test_wrap_codex.py FF
E   AssertionError: assert 'Nothing to undo' not in '...'
E         Nothing to undo: ...\.codex\config.toml has no Headroom wrap markers.
E       ✓ Codex is no longer routed through the Headroom proxy.
FAILED tests/test_cli/test_wrap_codex.py::test_unwrap_codex_removes_init_provider_routing
FAILED tests/test_cli/test_wrap_codex.py::test_unwrap_codex_removes_init_only_config_file
```

With this branch:

```text
$ pytest tests/test_cli/test_wrap_codex.py tests/test_cli/test_init_cli.py tests/test_cli/test_wrap_opencode.py -q -n 4
228 passed in 7.50s

$ ruff check .
All checks passed!
$ ruff format --check headroom/cli/wrap.py headroom/cli/init.py tests/test_cli/test_wrap_codex.py tests/test_cli/test_init_cli.py
4 files already formatted
$ mypy headroom --ignore-missing-imports --no-site-packages
headroom\fileperms.py:71: error: Module has no attribute "fchmod"  [attr-defined]
Found 1 error in 1 file (checked 554 source files)
```

The mypy error is the Windows-only `os.fchmod` false positive in an
untouched file; it passes on Linux CI.

The full `tests/test_cli` suite on Windows: 907 passed, 12 failed. Five
of those also fail on a clean `upstream/main` worktree with the same
environment (`test_recover_codex` ×2, `test_wrap_stale_marker`,
`test_unwrap_claude` ×2), so they're pre-existing Windows issues. The
other seven (`test_wrap_persistent`,
`test_install_apply_announces_windows_service_fallback`) only failed
under `-n 6` and pass when run serially on this branch (63 passed).

### Review round (rebased on `upstream/main` 1b7977ce)

The two new regression tests fail without the fix:

```text
E   AssertionError: assert {'model': 'gp... {'work': {}}} == {'model': 'gp....1:9999/v1'}}}
FAILED tests/test_cli/test_wrap_codex.py::test_unwrap_codex_after_init_then_wrap_removes_init_routing
FAILED tests/test_cli/test_wrap_codex.py::test_unwrap_codex_init_cleanup_keeps_profile_overrides
====================== 2 failed, 103 deselected in 1.21s ======================
```

With the fix (serial, Windows):

```text
$ pytest tests/test_cli tests/test_install/test_providers.py -q
FAILED tests/test_cli/test_install_cli.py::test_install_apply_restores_previous_deployment_after_failed_update
FAILED tests/test_cli/test_recover_codex.py::test_discover_dangling_homes_only_returns_codex_homes
FAILED tests/test_cli/test_recover_codex.py::test_recovery_never_writes_through_target_symlinks
FAILED tests/test_cli/test_unwrap_claude.py::test_unwrap_claude_removes_mcp_purges_retired_hook_and_stops_proxy
FAILED tests/test_cli/test_unwrap_claude.py::test_unwrap_claude_stops_claude_owned_persistent_deployment
FAILED tests/test_cli/test_wrap_stale_marker.py::test_signal_handler_unwinds_so_the_restore_can_run
====== 6 failed, 1026 passed, 6 skipped, 17 warnings in 90.47s (0:01:30) ======

$ ruff check headroom/cli/init.py headroom/cli/wrap.py tests/test_cli/test_wrap_codex.py tests/test_cli/test_init_cli.py
All checks passed!
$ ruff format --check (same files)
4 files already formatted
```

All six failures reproduce identically on a clean `upstream/main`
worktree in the same environment (`6 failed, 46 passed, 2 skipped` for
those files), so none come from this branch.

### Review round 2 (rebased on `upstream/main` 0d7d61d5)

`test_unwrap_codex_keeps_snapshot_when_init_cleanup_write_fails` fails
on the previous head (the snapshot was already deleted when the write
raised):

```text
E    +    where exists = WindowsPath('.../.codex/config.toml.headroom-backup').exists
FAILED tests/test_cli/test_wrap_codex.py::test_unwrap_codex_keeps_snapshot_when_init_cleanup_write_fails
====================== 1 failed, 105 deselected in 0.94s ======================
```

With the fix:

```text
$ pytest tests/test_cli/test_wrap_codex.py tests/test_cli/test_init_cli.py -q
188 passed in 9.21s
$ ruff check headroom/cli/wrap.py tests/test_cli/test_wrap_codex.py
All checks passed!
$ ruff format --check headroom/cli/wrap.py tests/test_cli/test_wrap_codex.py
2 files already formatted
$ mypy headroom --ignore-missing-imports --no-site-packages
headroomileperms.py:92: error: Module has no attribute "fchmod"; maybe "chmod"?  [attr-defined]
Found 1 error in 1 file (checked 561 source files)
```

Same Windows-only `os.fchmod` false positive as above.

## Real Behavior Proof

- Environment: Windows 11, Python 3.13, this branch vs. `upstream/main`
d5318ac2, isolated `HOME`/`USERPROFILE`/`CODEX_HOME` in a temp dir.
- Exact command / steps: wrote `model = "gpt-5"` to `config.toml`, added
the init block with `init._ensure_codex_provider(cfg, 8787)` (the same
writer `headroom init -g codex` uses), then ran the real `headroom
unwrap codex --no-stop-proxy` Click command and printed the file.
- Observed result: on `main`, unwrap prints "Nothing to undo ... has no
Headroom wrap markers." and "✓ Codex is no longer routed through the
Headroom proxy.", and the file still has `openai_base_url =
"http://127.0.0.1:8787/v1"`. On this branch it prints "Removed Headroom
block from ...\.codex\config.toml; other content preserved." and the
file is just `model = "gpt-5"`. Review round, same isolated setup with a
`[profiles.work]` table that sets `model_provider = "headroom"` and
`openai_base_url = "http://127.0.0.1:9999/v1"`: (a) `init` → `headroom
wrap codex --prepare-only --port 8787` → `headroom unwrap codex`: before
the fix the restored file still began with the full `# --- Headroom init
provider ---` block (`openai_base_url = "http://127.0.0.1:8787/v1"`);
after it, it prints "Restored prior ... from pre-wrap backup." and the
file is `model = "gpt-5"` plus the untouched profile. (b) `init` →
`unwrap`: before the fix `[profiles.work]` came back empty; after it
both profile keys survive.
- Not tested: running the full `headroom init -g codex` command end to
end (it also writes hooks and a deployment manifest, so I wrote the
block with the same function directly); a real Codex Desktop session;
project-scope `.codex/config.toml` init blocks, which `unwrap codex`
doesn't target (the docs now say to delete that block by hand).

## Runtime Rollout Safety

- Rollout-managed feature(s): none; CLI-only change to `headroom unwrap
codex` and an extra `headroom init codex` message on Windows
- Minimum rollout channel: stable
- Stable/default behavior changed: yes, narrowly. `unwrap codex` now
also removes the `# --- Headroom init provider ---` block from the
active Codex config; previously it left it in place and reported
success. The orphan-key safety net shared with `wrap codex` re-injection
is narrowed to the document root, so it no longer deletes matching keys
inside `[profiles.*]`
- Kill switch / disable path: not needed; the only new action runs when
the user explicitly runs `headroom unwrap codex`
- Unsafe override required: no
- Qualification impact: none; no proxy, compression or request-path code
changed
- Rollback path: revert this commit; `headroom init -g codex` re-adds
the routing if a user wants it back

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- Open PR #3355 also edits `_restore_codex_provider_config` (auth-helper
cleanup). The changes don't overlap logically, but whichever lands
second may need a small rebase.
- Open PR #749 (consolidated `headroom uninstall`) detects Codex only
through wrap markers, so it has the same blind spot for init routing; it
would get this for free by calling `unwrap codex`.
- Not in scope, and a maintainer call: whether `init codex` on Windows
should refuse to write routing, or should set up a supervisor, given
that nothing starts the proxy there.
- Deferred (Devin "existing Codex provider settings lost"): `init codex`
already deletes a user's prior root `model_provider` / `openai_base_url`
when it writes its block, and saves them nowhere. That is pre-existing
`init` behaviour on `main`, which this PR neither introduces nor
worsens, and there is no stored value for unwrap to restore. A clean fix
belongs in `init` (stash the replaced values inside its block, restore
them on unwrap), so I'd rather do it as a follow-up than widen this PR.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3920"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-07 20:59:10 -05:00
Sudhindra Desai 1982b3a773 fix(codex): explain why wrapped Codex runs without the shared server (#3899)
## Description

`headroom wrap codex` passes proxy routing as a `--config` override
(`openai_base_url=…`, or `model_providers.<id>.base_url=…` for custom
providers). This keeps routing scoped to the launched process and leaves
`~/.codex/config.toml` untouched, by design (#1507, #2160). Codex 0.159
runs any session that has command-line overrides in embedded mode, so
wrapped sessions always print `Running without the shared background
server` and never use Codex's shared background server.

Env-only routing can't replace the override. ChatGPT-subscription Codex
ignores `OPENAI_BASE_URL` (see 5c7a5b48 / d54c5b6a), so
`openai_base_url` has to come from config. A shared server also serves
every session at once, so a per-session base URL can't apply to it.

The issue accepts "explicit documentation of the limitation", so this PR
does that:

- `wrap codex` now prints a one-line note at launch that explains
embedded mode and points to the persistent alternative.
- The proxy docs describe the limitation and the alternative: `headroom
init codex` (project) or `headroom init -g codex` (user) writes the
routing to Codex's own config, so a plain `codex` launch keeps the
shared server.

Closes #3888

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/cli/wrap.py`: new `_CODEX_EMBEDDED_MODE_NOTE`.
`_codex_session_launch_settings` appends it to the launch display lines,
so it prints under `Launching CODEX…` for both the built-in `openai`
provider and custom providers.
- `docs/content/docs/proxy.mdx`: new paragraph under "Agent wrapping"
covering why wrapped Codex uses embedded mode, plus the `headroom init
codex` / `headroom init -g codex` alternative and its persistent scope.
- `tests/test_cli/test_wrap_codex.py`: asserts the note for both
provider paths.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`): not run successfully. My
local mypy stops on an unrelated numpy stub syntax error in this env
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ pytest tests/test_cli/test_wrap_codex.py -q
101 passed in 5.68s

$ pytest tests/test_cli -q -k codex
176 passed, 738 deselected in 6.84s

$ ruff check headroom/cli/wrap.py tests/test_cli/test_wrap_codex.py
All checks passed!
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, branch head
- Exact command / steps: `python -c "import headroom.cli.wrap as w;
print(w._CODEX_EMBEDDED_MODE_NOTE)"`
- Observed result: the note prints as expected (below), and the codex
wrap tests assert it appears in the launch display for both the built-in
`openai` provider and custom providers:
  ```text
Note: session-scoped routing runs Codex without its shared background
server (embedded mode). To keep the shared server, route through Codex
config instead: run `headroom init codex` (or `headroom init -g codex`)
once, then start `codex` directly.
  ```
- Not tested: a live `headroom wrap codex` launch against Codex CLI
0.159.2 (Codex isn't installed on my machine). I also didn't verify end
to end that an `init`-configured plain `codex` launch keeps the shared
server, though that path passes no command-line overrides.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: N/A
- Stable/default behavior changed: No. Routing is unchanged; only one
extra banner line is printed.
- Kill switch / disable path: N/A
- Unsafe override required: No
- Qualification impact: None
- Rollback path: revert this PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

A follow-up could skip the `--config` override when Codex's effective
config already routes the selected provider to this proxy, for example
after `headroom init codex`. That would let `wrap codex` itself keep the
shared server. The trade-off is losing the per-project `/p/<project>`
URL prefix for those sessions, so I left it out of this PR. Happy to add
it if maintainers want that behavior.
2026-10-01 12:15:47 -05:00
Bernardo GomesandJD Davis 08dda4cb0b fix(cli): strip unmarked headroom_memory TOML table before injecting (#3490)
## Description

`_inject_memory_mcp_config` (used by `headroom wrap codex`) only guards
against a duplicate `[mcp_servers.headroom_memory]` table via the
`_MEMORY_MCP_MARKER`/`_MEMORY_MCP_END` comment pair it writes. If that
table already exists in a user's `~/.codex/config.toml` **without**
those markers — e.g. hand-copied from the usage example in
`headroom/memory/mcp_server.py`, or left over from before this table
was marker-guarded — the injector appends a second
`[mcp_servers.headroom_memory]` table instead of replacing the
existing one. Two tables sharing the same key is invalid TOML, and
silently breaks Codex's MCP config parsing on startup.

This PR adds a helper that strips any unmarked
`[mcp_servers.headroom_memory]` table before the marker-wrapped
section is appended, mirroring the pattern already used for
`[model_providers.headroom]`.

No issue number to close (found while reviewing the Codex wrap flow).

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Added `_strip_existing_codex_memory_mcp_table()` in
`headroom/cli/wrap.py`, mirroring the existing
`_strip_existing_codex_headroom_provider_table()` helper: it
regex-matches an unmarked `[mcp_servers.headroom_memory]` table and
removes it.
- Called this new helper from `_inject_memory_mcp_config()` right before
appending the marker-wrapped `headroom_memory` section, in the branch
that runs when no `_MEMORY_MCP_MARKER` is found in the existing config.
- Added a regression test, `test_inject_replaces_unmarked_memory_block`,
in `tests/test_cli/test_wrap_codex.py`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ .venv/bin/python -m pytest tests/test_cli/test_wrap_codex.py -q
============================= test session starts ==============================
platform linux -- Python 3.13.15, pytest-9.0.3, pluggy-1.6.0
rootdir: /home/bitter/development/oss/headroom
configfile: pyproject.toml
plugins: anyio-4.12.1, cov-7.0.0, langsmith-0.9.3, respx-0.23.1, asyncio-1.3.0
collected 93 items

tests/test_cli/test_wrap_codex.py ...................................... [ 40%]
.......................................................                  [100%]

======================== 93 passed in 66.82s (0:01:06) =========================

$ .venv/bin/python -m pytest tests/test_cli/ -q
[...]
734 passed, 4 skipped, 1 failed in 92.84s (0:01:32)

FAILED tests/test_cli/test_unwrap_claude.py::test_unwrap_claude_stops_claude_owned_persistent_deployment
```
This one failure is pre-existing on a clean `main` checkout (verified
via `git stash` + rerun before making any change here) and is
unrelated to this PR — it's an assertion on unwrap-claude output
messaging, not touched by this diff.

```text
$ .venv/bin/python -m ruff check headroom/cli/wrap.py tests/test_cli/test_wrap_codex.py
All checks passed!

$ .venv/bin/python -m mypy headroom/cli/wrap.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: local dev checkout, Python 3.13.15, `.venv` from repo
(`pytest 9.0.3`, `ruff`, `mypy`).
- Exact command / steps: Ran `.venv/bin/python -m pytest
tests/test_cli/test_wrap_codex.py::TestCodexMemoryMcpConfig::test_inject_replaces_unmarked_memory_block
-q`, which (1) seeds `~/.codex/config.toml` with an unmarked
`[mcp_servers.headroom_memory]` table plus `[profiles.default]` and an
unrelated `[mcp_servers.other]` table, (2) calls
`_inject_memory_mcp_config("codex-user")` — the same function `headroom
wrap codex` invokes — and (3) reads the resulting `config.toml` back and
parses it with `tomllib.loads`.
- Observed result: With the fix applied, the test passes — exactly one
`[mcp_servers.headroom_memory]` table remains in the file, wrapped in
the `_MEMORY_MCP_MARKER`/`_MEMORY_MCP_END` comments, with the updated
`--user codex-user` value; `[profiles.default]` and
`[mcp_servers.other]` are byte-for-byte untouched;
`tomllib.loads(content)` succeeds and
`parsed["mcp_servers"]["headroom_memory"]["args"][-1] == "codex-user"`.
I additionally reverted just `headroom/cli/wrap.py` to its pre-fix state
(`git checkout e67b3c8a -- headroom/cli/wrap.py`, keeping the new test)
and reran the same test to confirm it fails without the fix: `assert
content.count("[mcp_servers.headroom_memory]") == 1` fails with `assert
2 == 1`, because the injector left the old unmarked table in place and
appended a second, marker-wrapped `[mcp_servers.headroom_memory]` table
below it. Separately, I fed that exact two-table content through
`tomllib.loads()` directly and confirmed it raises
`tomllib.TOMLDecodeError: Cannot declare ('mcp_servers',
'headroom_memory') twice (at line 14, column 29)` — reproducing the
exact failure mode Codex's TOML parser hits on startup. I then restored
the fixed `wrap.py` and reran the test to confirm it passes again.
- Not tested: real `codex` binary startup against the patched config (no
local Codex CLI install in this environment) — covered instead by the
`tomllib.loads` validity assertion, which is what actually breaks
Codex's parser today.

## Runtime Rollout Safety

- Rollout-managed feature(s): none — this only affects the `headroom
wrap codex` CLI subcommand's local config-file writer, not any
server-side or rollout-gated runtime behavior.
- Minimum rollout channel: N/A (CLI-only, no feature flag).
- Stable/default behavior changed: no. Behavior only changes for the
edge case where an unmarked `[mcp_servers.headroom_memory]` table
already exists (previously produced broken/invalid TOML); the normal
marker-found and no-table-at-all paths are unchanged.
- Kill switch / disable path: N/A — no runtime toggle; users can simply
not run `headroom wrap codex` or manually edit `~/.codex/config.toml`.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this commit; the prior (buggy) behavior only
manifested for a narrow, previously-broken edge case, so reverting is
safe.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A —
internal helper, no user-facing docs describe this edge case)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A — CLI/config-file fix, no UI.

## Additional Notes

- "Manual testing performed" is unchecked because I did not run the real
`codex` binary against the patched config in this environment (no Codex
CLI installed here); the regression test's `tomllib.loads` round-trip is
the direct proof that the generated config is valid TOML, which is the
actual failure mode this PR fixes.
- The pre-existing
`test_unwrap_claude_stops_claude_owned_persistent_deployment` failure on
`tests/test_cli/` is unrelated to this change and reproduces identically
on a clean `main` checkout.

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-09-30 11:15:04 -05:00
Raúl 3d3c629a9f fix(proxy): write per-port proxy logs so concurrent instances stop truncating each other (#3204)
## Description

Every proxy instance wrote the same two fixed files:
`~/.headroom/logs/proxy.log` (runtime log, via the `RotatingFileHandler`
in `_setup_file_logging`) and `~/.headroom/logs/proxy-stdio.log`
(subprocess stdout/stderr from `_start_proxy`). The name is a constant
(`paths.py` `_PROXY_LOG_FILE = "proxy.log"`), so N concurrent proxies on
different ports (wrapped copilot, droid, claude; plus any persistent
`install agent` deployment) run N independent rotating handlers over the
*same* file and truncate/rename each other's history.

This fix keys both files by port (`proxy-<port>.log`,
`proxy-stdio-<port>.log`) so each proxy rotates only its own file.
`headroom perf` aggregates every per-port file and still reads a lone
legacy `proxy.log` when that is all that exists (backward compat); the
`proxy-stdio-*` captures (raw stdout/stderr, not PERF records) are
excluded from the parser.

This matters more now that wrap-spawned proxies self-exit after an idle
grace (orphan-proxy watchdog, #3202): the process is gone, so its
per-port log is the only forensic evidence and must survive an unrelated
proxy's rotation.

Closes #3203

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/paths.py`: `proxy_log_path(port=None)` and new
`proxy_stdio_log_path(port=None)` are the single source of truth for the
filenames (`proxy-<port>.log` / `proxy-stdio-<port>.log` with a port,
legacy names without). The `_PROXY_LOG_FILE` constant stays only as the
readers' fallback name.
- `headroom/proxy/helpers.py`: `_setup_file_logging(port=None)` targets
`proxy_log_path(port)`. The duplicate-handler guard runs BEFORE the
`RotatingFileHandler` is constructed (construction opens the file), so a
second `create_app` on a different port in one process cannot leave an
empty stray `proxy-<port>.log` or leak an fd; a handler pointing at a
different port is replaced and closed so per-port keying holds.
- `headroom/proxy/server.py`: `create_app` resolves `config` first, then
calls `_setup_file_logging(config.port)` (the proxy always knows its own
port).
- `headroom/cli/wrap.py`: `_get_log_path(port=None)` /
`_get_proxy_stdio_log_path(port=None)` are per-port; `_start_proxy`
passes its `port`. The stdio filename comes from
`paths.proxy_stdio_log_path` (single source) and its directory from
`_get_log_path`.
- `headroom/perf/analyzer.py`: `parse_log_files` selects PERF logs by a
positive filename match (`_PERF_LOG_FILE_RE` =
`proxy(-<digits>)?.log(.N)?`) rather than a `proxy-stdio` blacklist, so
`proxy-stdio-*.log` and any other `proxy-<word>.log` are never ingested;
ordering is deterministic (mtime, then path).
- `headroom/cli/perf.py`, `wiki/cli.md`: doc/help text updated for
per-port + legacy.
- `tests/test_perf_per_port_logs.py` (new): per-port path helpers; a
second `_setup_file_logging` on a new port reroutes cleanly with no
cross-file leakage (regression for the guard-ordering bug); and
`parse_log_files` aggregates per-port + legacy while excluding stdio and
other non-PERF `proxy-*.log`.
- `tests/test_cli_proxy_env.py`, `tests/test_agent_savings.py`,
`tests/test_cli/test_wrap_{codex,claude_vertex_proxy_env,proxy_detach}.py`:
the `_get_log_path` monkeypatches take the new optional `port`, and the
two stdio-filename assertions/fixtures in `test_cli_proxy_env.py` use
the per-port name.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`) — plus `ruff format --check`
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ .venv/bin/python -m pytest tests/test_perf_per_port_logs.py tests/test_cli_proxy_env.py tests/test_agent_savings.py -q -p no:randomly
115 passed, 1 skipped in 4.44s

$ .venv/bin/python -m pytest tests/test_cli_perf_format.py tests/test_learn/test_analyzer.py -q
107 passed in 4.16s

$ .venv/bin/python -m pytest tests/test_cli/test_wrap_proxy_detach.py tests/test_cli/test_wrap_codex.py tests/test_cli/test_wrap_claude_vertex_proxy_env.py -q
120 passed in 171.19s

$ .venv/bin/ruff check <touched> && .venv/bin/ruff format --check <touched>
All checks passed!

$ .venv/bin/mypy headroom/paths.py headroom/cli/wrap.py headroom/proxy/helpers.py headroom/proxy/server.py headroom/perf/analyzer.py
Success: no issues found in 5 source files
```

Rust gate (`cargo clippy/test/fmt`) not run: Python-only change, no
crate touched.

## Real Behavior Proof

- Environment: macOS 26.5 arm64, Python 3.13, local editable source
build.
- Exact command / steps: `headroom proxy --port 8890` and `headroom
proxy --port 8891` started concurrently; inspected `~/.headroom/logs/`;
then `headroom perf --hours 1 --format json`.
- Observed result: each proxy wrote its own file (`proxy-8890.log`,
`proxy-8891.log`, independent), the pre-existing shared `proxy.log` was
not touched, and `headroom perf` reported `log_files_read: 4`,
aggregating per-port + legacy with no error.
`test_setup_file_logging_reroutes_on_port_change` additionally drives
the production helper twice in one process and asserts each port's
records land only in its own file.
- Not tested: Windows path semantics (filename construction is
platform-independent `pathlib`); a legacy `proxy.log` written by an OLD
binary running beside a new per-port one end to end (readers handle it,
but not exercised live).

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: N/A (no flag; readers accept both layouts).
- Stable/default behavior changed: log filenames change from `proxy.log`
to `proxy-<port>.log`; `headroom perf` reads both, so history is not
orphaned.
- Kill switch / disable path: N/A (no flag).
- Unsafe override required: no.
- Qualification impact: none — readers stay backward compatible with a
lone legacy `proxy.log`.
- Rollback path: revert; a downgraded binary writes legacy `proxy.log`
again and readers still aggregate any per-port files left behind.

Disk growth: each per-port family keeps the prior bound (`maxBytes=10 MB
× backupCount=5` = 60 MB), but there is now one family per distinct port
used instead of one global family. Ports are usually stable (default
8787; wrap auto-selects a nearby free port), so in practice a handful of
families; the theoretical worst case is bounded by the port scan
(`_find_available_port`, ≤100) → ≤ ~6 GB. No automatic pruning; stale
`proxy-<port>.log*` families can be deleted safely (perf reads whatever
remains). A retention policy is intentionally out of scope.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A — log-filename change; no UI.

## Additional Notes

Follow-up (separate SDK PR, not here): the TypeScript SDK helper
`sdk/typescript/src/paths.ts::proxyLogPath()` still returns legacy
`proxy.log` with no port argument and is pinned by `paths.test.ts`. It
mirrors `paths.py`, so it should gain an optional port argument in an
SDK PR (kept separate to stay one-surface, and because that package's
vitest suite is the right gate for it).
2026-09-09 15:54:34 -05:00
b7f342c153 fix(wrap): verify proxy deps before mutating Codex config (#1628)
## Description

\`headroom wrap codex\` now verifies that optional proxy dependencies
(\`headroom-ai[proxy]\`) are installed before mutating Codex
\`config.toml\`. If the check fails, the command exits with the same
error message as \`headroom proxy\` and leaves Codex config untouched.

Fixes #1614 (Bug 1: config mutated before proxy dependency check).

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Extract \`ensure_proxy_dependencies()\` in \`headroom/cli/proxy.py\`
(shared with \`headroom proxy\`)
- Call it at the start of \`wrap codex\` when \`not no_proxy\`, before
config snapshot/injection
- Add regression tests for prepare-only abort, \`--no-proxy\` skip, and
import failure messaging

## Testing

- [x] Unit tests pass (\`pytest\`)
- [x] Linting passes (\`ruff check .\`)
- [ ] Type checking passes (\`mypy headroom\`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

\`\`\`bash
pytest
tests/test_cli/test_wrap_codex.py::test_wrap_codex_aborts_before_mutating_config_when_proxy_deps_missing
\

tests/test_cli/test_wrap_codex.py::test_wrap_codex_skips_proxy_dependency_check_with_no_proxy
\

tests/test_cli/test_wrap_codex.py::test_ensure_proxy_dependencies_exits_when_server_import_fails
-q
# 3 passed
ruff check headroom/cli/wrap.py headroom/cli/proxy.py
tests/test_cli/test_wrap_codex.py
ruff format --check headroom/cli/wrap.py headroom/cli/proxy.py
tests/test_cli/test_wrap_codex.py
\`\`\`

## Real Behavior Proof

Environment: Linux (Ubuntu), Python 3.12, local checkout with
\`PYTHONPATH\` pointed at patched sources.

Exact command / steps:
1. Created a temp \`~/.codex/config.toml\` with \`model_provider =
"openai"\`.
2. Patched \`headroom.cli.wrap.ensure_proxy_dependencies\` to raise
\`SystemExit(1)\` (simulating missing \`[proxy]\` extra).
3. Ran \`headroom wrap codex --prepare-only --no-serena --port 8787\`.

Observed result: exit code 1; \`config.toml\` unchanged; no
\`config.toml.headroom-backup\` created; no \`[mcp_servers.headroom]\`
block written.

Also verified: \`headroom wrap codex --prepare-only --no-proxy ...\`
does not invoke the dependency check.

Not tested: Windows-specific proxy selector behavior (covered separately
in #1655).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did not edit CHANGELOG.md; release notes are generated
automatically

---------

Co-authored-by: syf2211 <syf2211@users.noreply.github.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net>
2026-08-13 11:52:22 -05:00
Tejas Chopra e0ce4b1d48 fix: remove rtk and lean-ctx CLI context tools (#2677)
## Description

Removes both third-party CLI context tools — **rtk** and **lean-ctx** —
and with them the context-tool selector itself. Headroom no longer
downloads, installs or configures either one, and there is no
replacement.

The previous pass (#2344) gated only three entry points inside
`headroom/cli/wrap.py`. That left the feature reachable in practice:

| Gap | Effect |
|---|---|
| `scripts/install.sh:1544`, `install.ps1:1681` | Ran `rtk init --global
--auto-patch` from bash/PowerShell, **bypassing the Python gate
entirely** — `curl \| sh` still wrote a Claude Code `PreToolUse` hook
regardless of `HEADROOM_RTK` |
| `wrap.py` `_setup_context_tool_for_agent` | **`wrap openhands` was
broken by default**: `rtk_required=True` met a gate returning `None` →
`SystemExit(1)`. Invisible because all 8 openhands tests patched
`_ensure_rtk_binary` to a fake path |
| `proxy/helpers.py`, `subscription/tracker.py` | Proxy shelled out to
`rtk gain` from `/stats`, the dashboard and `headroom perf`; the tracker
polled it per contribution (`_RTK_WIRING_DEFAULT = "enabled"`) |
| No cleanup path | Nothing removed artifacts an earlier default had
installed, so a machine that once ran the old default kept rtk in the
loop forever (#1669, #1955) |

Also worth noting: the rtk binary download had **no SHA or signature
verification** — only `rtk --version` as a smoke test.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [x] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

**Removed** — `headroom/rtk/` and `headroom/lean_ctx/` packages,
`headroom/cli/wrap_rtk_metrics.py`, `_selected_context_tool` /
`_setup_context_tool_for_agent` / `_VALID_CONTEXT_TOOLS`, the `--rtk` /
`--no-rtk` / `--no-project-rtk` / `--keep-rtk` flags across all 18 wrap
subcommands, `HEADROOM_RTK*`, the proxy-side `rtk gain` polling, the
dashboard CLI-filtering panel (rows + all 8 `cliFiltering*` Alpine
getters), `paths.rtk_path()` / `lean_ctx_path()`, the SDK path helpers,
`benchmarks/rtk_loop_learn_eval.py`, and the `headroom/rtk/**` CI path
filters.

**Fails loudly, not silently** — `--context-tool` / `--no-context-tool`
/ `HEADROOM_CONTEXT_TOOL` are kept solely to error out. They live in
shell profiles, aliases and CI jobs, and accepting them as a no-op would
read as Headroom having quietly stopped working. The installers reject
them too, which matters more than it looks: their arg parsers forward
the first unknown flag **and everything after it** to the wrapped tool,
so a leftover `--no-rtk` would have silently swallowed a following
`--port` and then been ignored downstream.

**New `headroom/context_tool_cleanup.py`** — deleting the code cannot
help a machine that already ran the old default, since the hooks,
binaries and injected guidance are durable on disk.
`purge_context_tool_artifacts()` runs once per `wrap`/`unwrap` and
removes the registered hook entries, the generated hook scripts, the
Headroom-managed `~/.local/bin` symlinks, the vendored
`~/.headroom/bin/{rtk,lean-ctx}` binaries, the `lean-ctx` MCP server
entry and the marker-fenced instruction blocks. Deliberately
conservative: idempotent, **skips** a malformed config rather than
overwriting it, and only unlinks a symlink resolving inside Headroom's
own bin dir so a user's own build is untouched. It reports on
**stderr**, because `wrap/unwrap openclaw --prepare-only` emit
machine-readable JSON on stdout as their entire contract. Skipped for
`wrap selfheal` (runs from a SessionStart hook; must not race Claude
Code's writer for `~/.claude.json`) and for `--help`, which must stay
read-only.

**Client-config hardening** (discovered while investigating a "corrupted
Serena settings file" report) — `wrap.py` reset a settings file to `{}`
when an existing file would not parse, then wrote that back. One
hand-edited typo or a transient `EACCES`/`EINTR` on a valid file
destroyed the user's `permissions`, `env` and `hooks`, on **every
`headroom wrap claude`**. It now refuses to write. Separately,
`fsutil.write_text` is now atomic (temp file + `fsync` + `os.replace`),
fixing all 14 non-atomic client-config writes at once; it follows
symlinks rather than replacing them (dotfile managers) and preserves an
existing file's mode.

**Deliberately kept** — `rtk` stays in the wrapper-peel list in
`transforms/content_router.py`. It sits beside `sudo`/`env`/`timeout` as
shell-command grammar, so `rtk cat f` is still classified as a file read
for anyone running their own rtk install, which the purge intentionally
leaves alone.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ ruff check headroom/ tests/ e2e/ --exclude headroom/dashboard/templates
All checks passed!

$ ruff format --check headroom/ tests/ e2e/ --exclude headroom/dashboard/templates
1255 files already formatted

$ mypy headroom/
Success: no issues found in 508 source files

$ pytest tests/test_context_tool_cleanup.py -q
11 passed

$ pytest tests/test_fsutil.py -q
12 passed

$ pytest tests/test_cli/test_wrap_codex.py -q            # 89 tests
89 passed in 431.68s
$ pytest tests/test_cli/test_wrap_opencode.py -q
39 passed in 257.46s
$ pytest tests/test_cli/test_wrap_helpers.py -q
45 passed
$ pytest tests/test_paths.py -q
75 passed
$ pytest tests/test_cli/test_unwrap_claude.py -q
14 passed
$ pytest tests/test_proxy_savings_history.py -q
39 passed
$ pytest tests/test_cli/test_wrap_copilot.py -q
27 passed
$ pytest tests/test_cli/test_wrap_zcode.py -q
20 passed
$ pytest tests/test_subscription_tracker.py -q
9 passed
$ pytest tests/test_proxy_dashboard_stats_cache.py -q
5 passed, 1 skipped
```

Repo-wide grep for 14 removed symbols (`headroom.rtk`,
`headroom.lean_ctx`, `_ensure_rtk_binary`, `_selected_context_tool`,
`_get_context_tool_stats`, `rtk_path`, `lean_ctx_path`,
`wrap_rtk_metrics`, `HEADROOM_RTK`, `cli_tokens_avoided`,
`tokens_saved_rtk`, …) across `*.py`, `*.ts`, `*.sh`, `*.ps1`, `*.yml`,
`*.html`: **zero hits**.

Notable test changes: `test_wrap_openhands.py` no longer patches
`_ensure_rtk_binary` and asserts `wrap openhands --prepare-only` exits 0
unpatched — the regression that was previously masked.
`test_wrap_continue.py` and `test_wrap_hintfile_agents.py` were removed
(every test drove RTK instruction injection). A new
`test_subscription_tracker.py::test_load_state_written_before_cli_context_tools_were_removed`
proves a pre-removal `subscription_state.json` still loads.

## Real Behavior Proof

- **Environment:** macOS 15.4 (darwin 25.4.0), Python 3.12.6, Headroom @
this branch, real `~/.headroom` and `~/.claude` on the dev machine.
- **Exact command / steps and observed result:**

```text
# 1. Retired flag fails loudly instead of silently no-op'ing
$ headroom wrap codex --prepare-only --context-tool rtk
Error: CLI context tools (rtk, lean-ctx) have been removed from Headroom: they
rewrote shell commands through a third-party binary Headroom no longer manages.
Drop --context-tool / --no-context-tool and unset HEADROOM_CONTEXT_TOOL;
`headroom wrap` uninstalls what they left behind on first run.

$ HEADROOM_CONTEXT_TOOL=lean-ctx headroom wrap codex --prepare-only
Error: CLI context tools (rtk, lean-ctx) have been removed from Headroom: ...

# 2. install.sh rejects the retired flags (extracted parse_wrap_args harness)
['--no-rtk', '--port', '9999']   rc=1  ERROR: CLI context tools ... Drop --no-rtk
['--context-tool=rtk']           rc=1  ERROR: CLI context tools ... Drop --context-tool
$ bash -n scripts/install.sh   # syntax OK

# 3. Purge ran against the real machine, which had all the orphaned artifacts
$ python -c "from headroom.context_tool_cleanup import purge_context_tool_artifacts; ..."
  removed ~/.headroom/bin/lean-ctx        (51 MB)
  removed ~/.headroom/bin/rtk             (7.7 MB)
  removed ~/.local/bin/rtk                (symlink into ~/.headroom/bin)
  removed ~/.claude/hooks/rtk-rewrite.sh
  removed 8 lean-ctx-* hook scripts
# ~/.claude.json afterwards: 90 top-level keys, 19 projects, mcpServers unchanged
# → ~59 MB reclaimed, no unrelated key touched

# 4. stdout stays machine-readable while the purge reports (planted a fake artifact)
$ headroom wrap openclaw --prepare-only --gateway-provider-id codex >out 2>err
$ cat out
{"enabled":true,"config":{"proxyPort":8787,...}}     # parses as JSON
$ cat err
Retired CLI context tool cleanup: removed /Users/tcms/.headroom/bin/rtk

# 5. --help is inert (planted artifact survives), a real run purges
$ headroom wrap codex --help   → artifact survived: CORRECT
$ headroom wrap openclaw --prepare-only → purged: CORRECT

# 6. MCP purge dry-run against a copy of the real 82 KB ~/.claude.json
top-level keys 90 -> 90;  projects 19 -> 19;  LOST keys: none
all content outside mcpServers byte-identical: True
```

Dashboard rendered via the Playwright test after the panel removal:
"Token Savings" shows only `Proxy 0 (0.0%)` / `Of total wire: 36.86%`,
and "Token Usage" reads Before Compression → Proxy Removed → After
Compression with no "Filtered (this session)" row. Nothing below the
removed panel broke.

- **Not tested:** Windows and Linux (macOS only) — `install.ps1` is
verified by brace-balance and inspection, not executed, since no `pwsh`
is available locally. The wrap e2e suite (`e2e/wrap/run.py`) was updated
but not run; it needs the Docker e2e image. `serena project index`
interaction is exercised in the stacked base PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

**Stacked on #2676** (`tejas/serena-config-bootstrap`) — please merge
that first; this PR's base should then be retargeted to `main`, or it
will read as containing that fix too.

**Breaking-change migration for users:**
- Drop `--rtk`, `--no-rtk`, `--no-project-rtk`, `--keep-rtk`,
`--context-tool`, `--no-context-tool` from any alias, script or CI job,
and unset `HEADROOM_RTK*` / `HEADROOM_CONTEXT_TOOL`. They now error
rather than being ignored, so the failure is immediate and
self-explaining.
- Previously-installed artifacts are purged automatically on the next
`wrap`/`unwrap`; no manual cleanup needed.
- `headroom perf --json` no longer carries a `cli_filtering` key, and
`/stats` no longer returns a `context_tool` section.

**Docs:** `docs/rtk-architecture.md` deleted; RTK/lean-ctx removed from
`README.md`,
`docs/content/docs/{configuration,opencode,grok-build,docker-install,filesystem-contract}.mdx`,
`docs/observability.md` and the matching `wiki/` pages.
`REALIGNMENT/09-phase-G-rtk-observability.md` is marked SUPERSEDED
rather than deleted, to keep the planning record.

**Follow-ups not in scope:** `_emit_wrap_interrupted` was deleted as
dead code — its only caller was the `except KeyboardInterrupt` guarding
the binary download, so with no download there is nothing slow left to
interrupt.
2026-07-30 22:59:41 -07:00
Tejas Chopra 5d23a0aec2 refactor(wrap): retire tokensave; Serena is the code-memory MCP (#2499)
## Description

`tokensave` was a **downloaded third-party Rust binary**
(`aovestdipaperino/tokensave`) that `headroom wrap` registered as a
code-graph MCP server. This removes it entirely and standardises on
**Serena** as the code-memory MCP — which was already the default in
`wrap`. Serena runs on demand via `uvx`, so Headroom no longer downloads
or executes a binary of its own for code memory.

The change is a *removal + safe transition*, not a behaviour flip:
Serena was already the default, so existing users move over
automatically. This PR also folds in a small README repositioning
(Headroom = the proxy; Serena is the recommended companion; RTK/lean-ctx
are third-party tools we don't control), since it's the same
tooling-stack story.

Closes #

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [x] Code refactoring (no functional changes)

## Changes Made

- **Removed the external tool:** `headroom/graph/tokensave_installer.py`
(the download-and-execute path), `build_tokensave_spec`, and the
`_ensure_tokensave_binary` / `_index_tokensave_project` /
`_setup_tokensave_mcp` helpers; dropped the dead `_setup_code_graph`.
- **`--code-memory`** now offers `serena` (default) or `none` — the
`tokensave` choice is gone. The strands `HeadroomBundle` uses Serena as
its (default-on) code-memory MCP.
- **Tombstone / transition (ledger-verified):** `headroom wrap` **and**
`headroom unwrap` remove a previously Headroom-installed `tokensave` MCP
entry so upgrading users stop launching it, and print that the leftover
`~/.local/bin/tokensave` binary and `.tokensave/` folders are safe to
delete. A user-managed `tokensave` entry is left untouched. Mirrors the
existing `codebase-memory-mcp` retirement.
- **Graceful for existing users:** `HEADROOM_CODE_MEMORY=tokensave` and
`--no-tokensave` resolve to Serena instead of erroring; `--no-serena`
now means "no code memory". **No state migration needed** — both tools'
indexes are regenerable caches of the source, so Serena simply
re-indexes.
- **Docs/README:** replaced the "tokensave binary trust model" section
with a Serena note + an "Upgrading from tokensave?" callout; retired the
RTK "first-class part of our stack" framing.
- **Tests:** deleted the tokensave-only test files
(`test_graph_tokensave.py`, `test_cli/test_tokensave_helpers.py`,
`test_cli/test_tokensave_setup.py`), rewrote `test_wrap_code_memory.py`
for the new resolver/dispatch, fixed a codex test that patched a removed
symbol.

Net: **+102 / −1187 lines.**

## Testing

- [x] Unit tests pass (`pytest`) — targeted to the affected areas
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ ruff check headroom/cli/wrap.py headroom/mcp_registry/ headroom/integrations/strands/ tests/test_wrap_code_memory.py tests/test_cli/conftest.py tests/test_cli/test_wrap_codex.py
All checks passed!

$ mypy headroom/cli/wrap.py headroom/mcp_registry headroom/integrations/strands/bundle.py
Success: no issues found in 12 source files

$ pytest tests/test_wrap_code_memory.py tests/test_cli/test_wrap_codex.py \
         tests/test_cli/test_wrap_claude_vertex_proxy_env.py \
         tests/test_cli/test_wrap_claude_finally_unbound.py -q
======================== 120 passed in 97.86s (0:01:37) ========================

$ pytest tests/test_cli --collect-only -q
========================= 713 tests collected in 1.82s =========================   # no import errors after symbol removal
```

## Real Behavior Proof

- **Environment:** macOS (darwin), Python 3.12.6, local `.venv`, on
branch `tejas/remove-tokensave`.
- **Exact command / steps:**
- `python -c "from headroom.integrations.strands.bundle import
HeadroomBundle; b=HeadroomBundle(enable_headroom_mcp=False,
enable_serena_mcp=False); print(len(b.tools))"` → confirms the module
imports after `build_tokensave_spec` removal (the import that my change
would otherwise break).
- CliRunner-driven `wrap codex --prepare-only` (in `test_wrap_codex.py`)
writes `[mcp_servers.serena]` (with `command = "uvx"`, `"--context",
"codex"`) to the codex config and **no** tokensave entry.
- `_resolve_code_memory` unit tests confirm: default → `serena`;
`HEADROOM_CODE_MEMORY=tokensave` → `serena`; `--no-serena` → `none`;
`--code-memory bogus` → `ClickException`.
- **Observed result:** import OK (`tools: 0`); Serena registered,
tokensave absent; resolver behaves as above; 120/120 tests pass.
- **Not tested:** a live `headroom wrap` against a real agent on a
machine with a *previously-installed* tokensave MCP entry — the
tombstone-removal path is covered by unit tests with a fake
registrar/ledger, not an end-to-end run.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title

## Additional Notes

- `--no-tokensave` / `--serena` / `--no-serena` are retained as hidden,
deprecated flags (no-ops or mapped) so existing scripts don't break.
- `--code-graph` is unchanged — it's the proxy's live file-watcher flag
and was never the tokensave MCP; only the dead tokensave hook behind it
was removed.
- `CHANGELOG.md` intentionally left untouched.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-07-22 20:59:24 -07:00
Zhenjia ZHOU f57e959a50 fix(wrap): emit bare dotted keys for Codex --config overrides (#2383)
## Description

`headroom wrap codex` with a custom provider emits `--config` overrides
whose dotted key quotes **every** segment
(`"model_providers"."litellm_prod"."base_url"=…`). Codex's override
parser matches dotted segments literally and silently ignores quoted
ones, so the overrides are dropped, the session keeps the provider's
real `base_url`, and traffic **bypasses Headroom entirely** — the exact
silent-bypass reported in #2358 on Codex 0.144.5.

I reproduced the parser behavior differentially on a local Codex
**0.144.1** (see Real Behavior Proof): a bare key is parsed (Codex
errors on the injected value), the same key quoted produces no error at
all — the override is silently discarded.

Fix: `_codex_dotted_key()` now emits segments **bare** whenever they are
valid bare keys (`[A-Za-z0-9_-]+`, which covers `model_providers`, every
normal provider id, and hyphenated header names like
`X-Headroom-Base-Url`), and quotes only segments where bare emission
would corrupt the dotted path (e.g. a provider id containing a dot).

Fixes #2358.

## Type of Change

- [x] Bug fix (silent proxy bypass for custom-provider Codex wraps)

## Changes Made

- `headroom/cli/wrap.py`: `_codex_dotted_key()` quotes only non-bare-key
segments; docstring explains the observed Codex parser behavior. The
default `openai` path (`openai_base_url=…`, already bare) is unchanged.
- `tests/test_cli/test_wrap_codex.py`: the custom-provider launch test
now pins the bare form for all three overrides (`base_url`,
`supports_websockets`, `env_http_headers.X-Headroom-Base-Url`), plus 2
direct unit tests (bare-when-safe incl. hyphens, quote-only-unsafe).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff` + `mypy`, CI-pinned settings)
- [x] Reproduced the parser behavior on a real Codex CLI first, then
fixed

### Test Output

```text
$ .venv/bin/python -m pytest tests/test_cli/test_wrap_codex.py -q
93 passed

# Before the fix the updated assertions fail (generated args still fully quoted):
#   FAILED ...::test_codex_session_launch_settings_preserve_custom_provider_identity
#   FAILED ...::test_codex_dotted_key_emits_bare_segments_when_safe
#   FAILED ...::test_codex_dotted_key_quotes_only_unsafe_segments

$ ruff check headroom/cli/wrap.py tests/test_cli/test_wrap_codex.py   # All checks passed!
$ ruff format --check <both>                                          # formatted
$ mypy headroom/cli/wrap.py --ignore-missing-imports                  # Success: no issues
```

## Real Behavior Proof

- Environment: macOS (Darwin), Codex CLI **0.144.1**
(`/opt/homebrew/bin/codex`), empty temp `CODEX_HOME` (no auth, no
network side effects), branch `fix/codex-config-bare-keys` off `main`
(`56c7d4a5`).
- Exact command / steps: differential probe of the override parser —
`codex exec --skip-git-repo-check -c 'profile="__nope__"' 'hi'` (bare
key) vs `codex exec --skip-git-repo-check -c '"profile"="__nope__"'
'hi'` (quoted key), each run once against a fresh empty `CODEX_HOME`.
- Observed result: bare key → Codex **parsed the override** and failed
fast on it (`Error: legacy profile = "__nope__" config is no longer
supported…`); quoted key → **no error referencing the override at all**,
Codex proceeded to start a session (banner printed) — the quoted
override was silently discarded. That is precisely the #2358 bypass
mechanism: every generated custom-provider override was quoted, hence
dropped, hence traffic went straight to the real upstream.
- Not tested: an end-to-end wrapped session against a live LiteLLM
upstream on Codex 0.144.5 (the reporter's exact version); the parser
behavior above is version-adjacent (0.144.1) and the argv shape is
pinned by unit tests.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — N/A
(docstring added)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md — N/A

## Additional Notes

- Segments that genuinely need quoting (a provider id with a dot) keep
their quotes: on parsers that ignore quoted segments those overrides
still won't apply, but bare emission would corrupt a *different* key
path, which is strictly worse. Such ids are rare; the common
LiteLLM/custom-provider case is fully bare after this fix.
2026-07-18 09:51:39 -07:00
Tejas ChopraandJerrettDavis 44136ed042 fix(wrap): make RTK opt-in (off by default) across wrap subcommands (#2344)
## Description

RTK CLI-command filtering was set up **by default** across ~16 `wrap`
subcommands (copilot, codex, aider, cursor, cline, continue, goose,
openhands, opencode, grok, omp, openclaude, vibe, …) via `if not
no_rtk:` — so users got rtk hooks / instruction injection without opting
in. `wrap claude` was the lone exception (already gated on
`--context-tool`).

This makes RTK **opt-in (off by default)** everywhere, so Headroom's own
savings are what's measured unless a user explicitly wants rtk.

Closes #

## Type of Change
- [x] Bug fix (behavior change: default flip)

## Changes Made
- **Central gate** `_rtk_opt_in()` — RTK runs only when explicitly
enabled via `--rtk` or `HEADROOM_RTK=1`. Guards the 3 RTK entry points
(`_setup_rtk`, `_ensure_rtk_binary`, `_inject_rtk_instructions`) so they
no-op by default — one small change instead of editing ~30 call sites.
- **`--rtk` opt-in flag** on all 18 tool subcommands via a shared
eager-callback option (`expose_value=False`, sets `HEADROOM_RTK=1`; no
subcommand signature changes).
- `wrap claude`'s legacy `--context-tool` still opts in (mirrored into
the gate).
- **`--no-rtk` kept** as an accepted, now-redundant no-op (back-compat).
- lean-ctx and all non-RTK behavior untouched.

## Testing
```text
pytest tests/test_wrap_rtk_opt_in.py   -> 4 passed
ruff check / format                    -> clean
mypy headroom                          -> Success: no issues found in 504 source files
```
Verified `--rtk` appears in `wrap {claude,codex,copilot} --help`;
`_rtk_opt_in()` is False by default, True with `HEADROOM_RTK=1`; entry
points no-op + write nothing when off.

## Real Behavior Proof
- Env: local `.venv`, click CliRunner.
- Steps: import wrap; assert gate default-off / env-on; assert
`_setup_rtk`/`_ensure_rtk_binary` return None and
`_inject_rtk_instructions` returns False + writes no file when not opted
in; assert `--rtk` in subcommand help.
- Observed: all pass. Not tested: a live end-to-end wrap launch (proxy
spawn).

## Notes
Part 2 of 3 (RTK opt-in). Separate PRs cover the code-graph MCP engine
and the proxy-option cleanup. No `CHANGELOG.md` edit — release-please
generates it from the PR title (per the changelog guard).

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-07-17 16:47:19 -07:00
Rudimar RonsoniandJerrettDavis dec60de976 fix(codex): preserve wrapped sessions and recover state (#2160)
## Description

Closes #2159.

Codex wrappers currently launch against a disposable `CODEX_HOME`, so
session state created during a wrapped run can disappear when that
temporary directory is removed. This change launches Codex against its
durable home, keeps proxy routing process-local, and adds recovery for
retained temporary homes and pinned recovery sources.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Launch Codex against its durable `CODEX_HOME` and apply routing
through process-local config overrides after the actual proxy port is
resolved.
- Preserve custom provider identity and reject providers that cannot be
redirected safely.
- Detect dangling temporary Codex homes before interactive wraps and
offer recovery.
- Add `headroom recover codex` with automatic discovery, repeatable
`--source`, preview, confirmation, retained backups, and rollback on
failure.
- Search Python's temp root, `$TMPDIR`, `/tmp`, `/private/tmp`, and
macOS `/private/var/folders/*/*/T` for retained `headroom-codex-home-*`
directories.
- Reuse `source-pinned/` copies left by interrupted or failed recovery
attempts after the original temporary home has disappeared.
- Report deleted temporary homes still referenced by SQLite rollout
paths without treating paths pasted into prompts or errors as filesystem
evidence.
- Audit the durable thread index, rollout files, and history when no
source remains, including indexed chat counts and history-only orphan
records.
- Normalize legacy localhost `headroom` providers in both SQLite thread
rows and rollout `session_meta`, including retries after an earlier
broken recovery, while preserving user-defined remote providers named
`headroom`.
- Merge compatible config, JSONL, rollout, SQLite, credential, and
regular-file state without propagating deletions or runtime artifacts.
- Rewrite recovered thread rollout paths to the durable home and restore
legacy Headroom thread providers to the active provider.
- Validate SQLite schemas, SQLx migration checksums, integrity, and
foreign keys, and quarantine malformed JSONL.
- Preserve failed targets with an atomic rename before rollback,
avoiding recursive-deletion races with live SQLite runtime files.
- Document discovery, migration, retained backups, rollback behavior,
and the limits of deleted-source recovery.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy`)
- [x] New tests added for new functionality
- [x] Manual testing performed in isolated Docker containers

### Test Output

```text
$ uv run pytest tests/test_cli/test_wrap_codex.py tests/test_cli/test_recover_codex.py -q
122 passed

$ uv run ruff check headroom/cli/recover.py headroom/providers/codex/recovery.py tests/test_cli/test_recover_codex.py
All checks passed!

$ uv run ruff format --check headroom/cli/recover.py headroom/providers/codex/recovery.py tests/test_cli/test_recover_codex.py
3 files already formatted

$ uv run mypy headroom/cli/recover.py headroom/providers/codex/recovery.py
Success: no issues found in 2 source files
```

All validation ran in `ghcr.io/astral-sh/uv:python3.12-bookworm` against
a writable disposable copy of a read-only source mount. Codex was not
installed or launched, and no real user Codex state was read or
modified.

The tests cover multi-root discovery, deleted-reference reporting,
retained pinned-source recovery, durable SQLite path relocation, SQLite
and rollout provider normalization, idempotent repair after an earlier
broken recovery, remote provider preservation, unrelated dangling target
rows, backup retention, atomic rollback, malformed-state quarantine,
SQLite validation, and Windows-safe handle closure.

The repository shim E2E was not launched locally because this recovery
work intentionally avoids launching Codex. Upstream CI exercises wrapper
E2E in isolated environments.

## Real Behavior Proof

- Environment: `ghcr.io/astral-sh/uv:python3.12-bookworm`, Python 3.12,
a writable disposable checkout copied from a read-only source mount, at
head `2d89ecec`.
- Exact command / steps: Run `pytest -q
tests/test_cli/test_wrap_codex.py tests/test_cli/test_recover_codex.py`,
then run `ruff check` and `ruff format --check` against
`headroom/cli/wrap.py`, `headroom/cli/recover.py`,
`headroom/providers/codex/recovery.py`,
`tests/test_cli/test_wrap_codex.py`, and
`tests/test_cli/test_recover_codex.py`.
- Observed result: `122 passed in 10.08s`; Ruff reported `All checks
passed!` and `5 files already formatted`.
- Not tested: Launching a real Codex process or modifying a real user
`CODEX_HOME`; these were intentionally excluded to protect live user
state.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code where the behavior is hard to understand
- [x] I have made corresponding documentation changes
- [x] My changes generate no new warnings
- [x] I have added tests that prove the fix is effective
- [x] New and existing focused unit tests pass with my changes
- [x] I have updated `CHANGELOG.md` if applicable

## Additional Notes

The temporary-home behavior was introduced by #1507 in
`ad9d086f43a664c4c2a19060b847f2e03ce4f6ad`. Related context: #730, #731,
#961, #1034, #1050, #1349, #1853, #1889, #2103, and #2104.

A temporary home that macOS or `TemporaryDirectory` already deleted
cannot be reconstructed unless a retained `source-pinned/` copy exists.
Recovery identifies genuine dangling SQLite paths, audits surviving
durable history, and recovers any retained pinned source it can find.
Prompt text without a rollout cannot reconstruct a full transcript.

The unchecked changelog item is not applicable because this repository
does not require a changelog entry for this fix.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-07-15 19:58:21 +00:00
nangsontayandJerrettDavis e5b3a634df fix(proxy): dedupe Codex WS request logging for accurate mixed-provider dashboards (#2189)
## Description

Running Claude Code (Anthropic) and Codex (OpenAI) against the **same**
Headroom proxy instance on one port produced incorrect, unstable
dashboard data. The proxy core is provider-isolated and
multi-provider-safe by design; the defect was in the observability
layer. The Codex `/v1/responses` **WebSocket** handler was the only path
in the proxy that wrote to the request logger by hand instead of through
the unified `emit_request_outcome` funnel, and it did so twice per
session close: the per-turn funnel record plus an unconditional
cumulative session-summary `RequestLog`. This PR removes the duplicate
summary log so Codex WS emits exactly one request log per turn, matching
the HTTP provider paths.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Dropped the duplicate cumulative session-summary `RequestLog` in the
Codex WS handler while preserving the per-turn `emit_request_outcome`
path.
- Preserved gated `request_messages` and `turn_id` on residual outcomes
so dashboard telemetry keeps the useful attribution without
double-counting tokens.
- Ensured explicit `--anyllm-provider` wins over a leaked
`HEADROOM_ANYLLM_PROVIDER` environment variable.
- Registered retry delay settings that had drifted out of the settings
registry.
- Hardened tests against developer-shell `HEADROOM_*` /
`ANTHROPIC_CUSTOM_HEADERS` leakage and stabilized several focused
proxy/wrap test fixtures.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ .venv/bin/pytest tests/ -q -p no:cacheprovider
8529 passed, 537 skipped, 5831 warnings in 276.25s (0:04:36)

$ .venv/bin/ruff check <touched files>
All checks passed!
```

## Real Behavior Proof

- Environment: macOS (Darwin 25.4.0), Python 3.13.14, pytest 9.0.3, ruff
via project venv, branch `fix/multi-provider-runtime`.
- Exact command / steps: Ran the full test suite without pytest cache
provider and Ruff on all touched files; used `git stash` to confirm the
stale fake-config failures pre-existed this change.
- Observed result: Full suite passed with no failures; Ruff passed;
Codex WS now routes end-of-session logging through
`emit_request_outcome`, emitting one request log per turn with the same
accounting model as Anthropic HTTP turns.
- Not tested: Live simultaneous Claude + Codex dashboard run. `mypy
headroom` was not run to completion; a scoped run reported one
pre-existing `settings_store.py:470` coercion error outside this diff.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A - server-side observability fix; no UI markup changed.

## Additional Notes

- The proxy's multi-provider routing, header/auth isolation, and
per-model cache keying are already correct and unchanged here; only the
WS observability write path was double-counting.
- Architectural assessment:
`plans/reports/research-260714-0004-multi-provider-upstream-compression-report.md`;
root-cause + resolution trail:
`plans/reports/debug-assessment-260714-0011-dashboard-instability-mixed-claude-codex-report.md`.
- No live simultaneous Claude + Codex dashboard run was performed;
validation is from test coverage and code review of the WS logging path.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-07-15 18:18:34 +00:00
d236b27c60 fix(wrap/codex): export the detected custom upstream base URL (#2125)
## Description

`headroom wrap codex` detects a user's custom upstream gateway but never
tells Codex to use it, so the user's gateway key is sent to
`api.openai.com`.

`_inject_codex_provider_config` handles a Codex user who has an
OpenAI-compatible gateway declared in `~/.codex/config.toml`, e.g.

```toml
model_provider = "freemodel"
[model_providers.freemodel]
base_url = "https://api.freemodel.dev"
```

It injects the Headroom provider with `env_http_headers = { ...
"X-Headroom-Base-Url" = "HEADROOM_CODEX_UPSTREAM_BASE_URL" }` and
**returns the preserved upstream URL** so the caller can export it. Its
docstring even says: *"Callers that go on to launch Codex should export
this value into `HEADROOM_CODEX_UPSTREAM_BASE_URL`."*

But `_prepare_codex_wrap_state` called it as a bare statement and
discarded the return, and `_run_codex_wrap` / `_build_codex_launch_env`
only ever set `OPENAI_BASE_URL`. A repo-wide grep confirms
`HEADROOM_CODEX_UPSTREAM_BASE_URL` (`_UPSTREAM_BASE_URL_ENV_VAR`) is
never assigned into any process env — it appears only at its definition
and in that docstring. Since Codex only emits the `X-Headroom-Base-Url`
header when the env var exists, the header is omitted, the proxy's
OpenAI handler falls back to its hardcoded `https://api.openai.com`, and
the user's `freemodel.dev` key is sent to OpenAI, which rejects it.

This is a regression: the wiring existed in the original `#1614` fix
(`_codex_custom_upstream = _inject_codex_provider_config(...)` then
`env[_UPSTREAM_BASE_URL_ENV_VAR] = ...`) and was dropped by a later
refactor that extracted `_prepare_codex_wrap_state`.

## Fix

Restore the wiring: `_prepare_codex_wrap_state` now captures and returns
`_inject_codex_provider_config`'s value, and `_run_codex_wrap` exports
it into the launch env (`env[_UPSTREAM_BASE_URL_ENV_VAR] =
custom_upstream`) when it is non-None and not already set, so a
user-provided value still wins.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/cli/wrap.py`: `_prepare_codex_wrap_state` returns the
detected custom upstream URL; `_run_codex_wrap` exports it into the
launch env (and its display list) when set.
- `tests/test_cli/test_wrap_codex.py`: add
`TestCodexLaunchExportsCustomUpstream` — drives `_run_codex_wrap` with
mocked prepare/launch and asserts the launch env carries
`HEADROOM_CODEX_UPSTREAM_BASE_URL` when a custom upstream is detected,
and does not when there isn't one.
- `CHANGELOG.md`: Bug Fixes entry.

## Testing

- [x] Unit tests pass (`uv run --extra dev pytest
tests/test_cli/test_wrap_codex.py::TestCodexLaunchExportsCustomUpstream
-q`)
- [x] Linting passes (`uvx ruff@0.15.17 check headroom/cli/wrap.py
tests/test_cli/test_wrap_codex.py headroom/memory/factory.py`)
- [x] Type checking passes (`uvx mypy==1.20.2
headroom/memory/factory.py`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ uvx ruff@0.15.17 check headroom/cli/wrap.py tests/test_cli/test_wrap_codex.py
All checks passed!
$ python -m py_compile headroom/cli/wrap.py tests/test_cli/test_wrap_codex.py
OK
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`. Importing
`headroom` pulls in the torch/transformers stack and running the CLI
test locally OOM-kills this box, so I verified the wiring with a
dependency-free script that models prepare -> run -> the proxy's
upstream fallback, and left the full pytest (including the new CLI test)
to CI.
- Exact command / steps: modelled the old flow (inject return discarded)
and the new flow (return exported into the launch env), then applied the
proxy's rule that a missing `HEADROOM_CODEX_UPSTREAM_BASE_URL` falls
back to `api.openai.com`.
- Observed result: old effective upstream is `https://api.openai.com`
(the gateway key is misrouted); new effective upstream is
`https://api.freemodel.dev` (the user's gateway). The new CLI test
asserts the launch env carries the var when a custom upstream is present
and omits it otherwise.
- Not tested: a live Codex process reading the env and emitting the
header; full local `pytest` deferred to CI (OOM, per above).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

Merged current `main` to pick up the repository-wide mypy cache-key
annotation fix, then verified the focused regression locally. the change
threads one return value through two functions and exports it, verified
by the wiring proof and a new CLI test that drives `_run_codex_wrap`
with the heavy prepare/launch steps mocked so only the env-export logic
is exercised.

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
2026-07-14 12:02:24 -04:00
Rudimar Ronsoni c4ddcb93a7 fix(codex): skip sockets in session home overlay (#2104)
## Description

Prevent `headroom wrap codex` from failing when the active `CODEX_HOME`
contains a Unix socket. The session overlay copied every entry with
`shutil.copytree()`, which raises `shutil.Error` when it reaches Git's
`fsmonitor--daemon.ipc` socket.

The overlay now skips socket entries while continuing to copy regular
Codex state and surface unrelated copy errors.

Closes #2103

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Ignore filesystem sockets while seeding the temporary Codex session
home.
- Add a regression test with a real nested `fsmonitor--daemon.ipc`
socket and a regular sibling file.

## Testing

- [x] Focused unit tests pass (`pytest tests/test_cli/test_wrap_codex.py
-q`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New regression test added
- [ ] Manual interactive testing performed

### Test Output

```text
Docker, Linux arm64, Python 3.12.12

pytest tests/test_cli/test_wrap_codex.py -q
88 passed in 5.92s

ruff check .
All checks passed!

ruff format --check .
1191 files already formatted

mypy headroom --ignore-missing-imports
Success: no issues found in 469 source files
```

## Real Behavior Proof

- Environment: isolated Docker container on Linux arm64 with Python
3.12.12 and Rust 1.95.0
- Exact command / steps: bind a real Unix socket at
`vendor_imports/skills/.git/fsmonitor--daemon.ipc`, then enter
`_codex_session_home_overlay()` through the focused pytest regression
- Observed result: the regular sibling file is copied, the socket is
omitted, the source socket remains active, and the overlay exits cleanly
- Not tested: an interactive Codex launch against the live host
`~/.codex`

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] My changes generate no new warnings
- [x] I have added tests that prove the fix is effective
- [x] New and existing focused Codex wrapper tests pass with my changes

## Additional Notes

The filter is intentionally limited to socket entries. Permission errors
and failures involving regular files still propagate from
`shutil.copytree()`.
2026-07-13 10:09:32 -04:00
Rod Boev ad9d086f43 feat(codex): keep wrap routing session-scoped (#1507)
## Description

Keeps Codex wrap routing session-scoped so routing state from one
wrapped session does not leak into another.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Scope Codex wrap routing state to the active session.
- Avoid cross-session routing contamination for wrapped Codex traffic.
- Keep changes focused on wrap/proxy routing behavior.

## Testing

- [x] Unit tests pass
- [x] New tests added for new functionality when applicable
- [x] Manual testing performed

### Test Output

```text
Focused tests/review were completed before this governance body cleanup. The current branch is conflicted and still needs merge resolution before final merge readiness.
```

## Real Behavior Proof

- Environment: Headroom development/review context.
- Exact command / steps: Reviewed session-scoped Codex wrap routing
behavior and existing focused coverage.
- Observed result: Routing state is scoped to the active wrap session
rather than shared globally across sessions.
- Not tested: Current conflicted branch after merge resolution;
conflicts still need to be resolved before merge.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Additional Notes

This body was normalized by a maintainer after approval so the
governance parser reflects the already-reviewed PR state. The PR remains
blocked by merge conflicts.
2026-07-11 11:03:57 -05:00
Rod Boev 372d6c8cd4 fix(wrap): preserve custom Codex provider base_url during proxy injection (#1894)
## Description

Refs #1614 (Bug 2 only; Bug 1's config-mutation ordering is covered by a
separate PR).

`headroom wrap codex` unconditionally pointed the proxy's upstream
OpenAI route at `api.openai.com`, even when the user's Codex config
already declared a custom OpenAI-compatible provider such as
`freemodel.dev`, LiteLLM, or vLLM under `[model_providers.<name>]`. The
proxy then silently rerouted traffic to OpenAI, which rejected the
user's gateway API key, and Codex interpreted the resulting auth
failures as an invalid session.

## Type of Change

- [x] Bug fix

## Changes Made

- `_detect_custom_codex_upstream_base_url` and
`_codex_custom_provider_base_urls` in `headroom/cli/wrap.py` scan the
existing `config.toml` for a user-declared custom `[model_providers.*]`
table, excluding Codex built-ins and Headroom's own table, and return
its `base_url` when the selection is unambiguous: either the top-level
`model_provider` names it directly, or a prior wrap left the original
provider in the `# was: <original>` comment from
`_redirect_existing_top_level_keys`.
- The detector falls back to the sole custom provider when exactly one
candidate exists and no matching top-level selection is present, which
covers the issue repro where the custom table exists without a static
top-level provider pin.
- `_inject_codex_provider_config` now detects that custom upstream
before building the injected provider block. When found, it adds
`X-Headroom-Base-Url` to `env_http_headers`, mapped to
`HEADROOM_CODEX_UPSTREAM_BASE_URL`, matching Codex's env-var-based
header contract.
- `codex()` exports the detected value into
`HEADROOM_CODEX_UPSTREAM_BASE_URL` for the launched Codex process unless
the user already set it. The proxy's OpenAI HTTP handlers already honor
`X-Headroom-Base-Url`, so HTTP `/v1/chat/completions` and
`/v1/responses` requests forward to the preserved gateway instead of the
default OpenAI upstream.

This is scoped to the HTTP request path. Codex's WebSocket transport for
`/v1/responses` resolves its upstream from a separate header-independent
path and keeps the existing behavior.

## Testing

- [x] Focused Codex wrap tests passed locally before PR review: `pytest
tests/test_cli/test_wrap_codex.py -q`
- [x] Broader Codex CLI test selection passed locally before PR review:
`pytest tests/test_cli/ -k codex -q`
- [x] CI lint, format, and type checks passed on PR head `fb769349`.
- [x] CI build, test shards, native wrapper, Docker init, Docker wrap,
security, merge-conflict, and governance jobs passed on PR head
`fb769349`.
- [ ] Live `headroom wrap codex` against a real `freemodel.dev` account
was not run for this PR.

## Test Output

Previously reported local focused test output:

```text
pytest tests/test_cli/test_wrap_codex.py -q
83 passed

pytest tests/test_cli/ -k codex -q
113 passed, 426 deselected
```

Previously reported local static checks:

```text
ruff check
ruff format --check
mypy
```

CI evidence on PR head `fb769349`: GitHub Actions run `28987811482`
completed successfully for CI, including `lint` with `ruff check`, `ruff
format --check`, and `mypy`; `build`; `build-wheel`; four test shards;
`test-agno`; `test-extras`; `test-dashboard-ui`;
`windows-native-wrapper`; `macos-native-wrapper`; and
`docker-native-e2e`. PR Governance run `28987811437` completed
successfully.

## Real Behavior Proof

- Environment: Unit-level Codex config injection using pytest tmp home,
PR head `fb769349`, and GitHub Actions CI run `28987811482`.
- Exact command / steps: With a Codex config containing
`[model_providers.freemodel]`, `base_url = "https://api.freemodel.dev"`,
and `wire_api = "responses"`, call
`_inject_codex_provider_config(8787)`.
- Observed result: The injector returns `https://api.freemodel.dev`; the
injected `[model_providers.headroom]` table contains `env_http_headers =
{ "X-Headroom-Project" = "HEADROOM_PROJECT", "X-Headroom-Base-Url" =
"HEADROOM_CODEX_UPSTREAM_BASE_URL" }`; the user's
`[model_providers.freemodel]` table remains unchanged; and re-running
`_inject_codex_provider_config(9999)` preserves the same upstream while
updating the proxy port.
- Not tested: Live external traffic through a real `freemodel.dev` key,
WebSocket custom-upstream routing, and Bug 1's
dependency-check-before-config-mutation path.

## Review Readiness

- [x] Scope is limited to #1614 Bug 2, custom provider `base_url`
preservation for Codex wrap.
- [x] Bug 1 remains out of scope and is called out separately.
- [x] The changed code uses the existing proxy `X-Headroom-Base-Url`
contract instead of adding a new proxy route.
- [x] Ambiguous multiple custom providers keep prior fallback behavior
instead of guessing.
- [x] A collaborator reviewed and approved the current head after
running the focused Codex wrap tests locally.
- [x] This PR is ready for human review.

## Checklist

- [x] I have performed a self-review.
- [x] Focused tests added and passing.
- [x] Lint, format, and type-check clean on CI.
- [x] No unrelated files changed.
- [x] This PR is ready for human review.
2026-07-09 16:55:00 -05:00
Tejas ChopraandClaude Opus 4.8 68676daa50 feat: ship the coding profile as Headroom's out-of-box default posture (#1893)
Make a bare `headroom proxy` (and the uvicorn factory / argparse main)
default to the cache-mode coding posture instead of requiring users to
set a dozen env vars.

Profile (agent_savings.py):
* "coding" is now the DEFAULT profile (DEFAULT_PROFILE), and it is
rewritten for cache mode: proxy_mode="cache" and
compress_user_messages=True (cache mode compresses the newest
OBSERVATION delta — a user/tool turn — so compress_user must be on or
there is nothing to compress; prefix stability is preserved by the delta
engine, not by refusing to touch user turns).
* AgentSavingsProfile carries the standalone router/handler toggles too
(tool_search, cross_turn_dedup, lossless_then_lossy, protect_reads,
code_aware, effort_router, lossless, min_chars_for_block); proxy_env()
emits them. Defaults preserve current behavior for the other profiles.
* coding sets: tool_search=1, dedupe=1, lossless_then_lossy=1,
protect_reads=1, code_aware=1, effort_router=0, lossless=0,
min_chars_for_block=25. CCR stays ON (no HEADROOM_NO_CCR) so any lossy
loss is recoverable.
* apply_agent_savings_env_defaults() now honors an explicit
HEADROOM_SAVINGS_PROFILE already in the env before falling back to the
default.

Delivery (pollution-free by construction):
* MODE and savings_profile default via INLINE defaults in the config
builders (cache / coding) — no global env mutation, so unit tests that
build config directly keep clean defaults.
* The request-time toggles are seeded into os.environ (setdefault) via
seed_proxy_env_defaults() ONLY at the executable/deployment entries —
run_server() (before serving) and create_app_from_env() (uvicorn
factory) — NOT in the CLI command or any library builder, so CliRunner
tests never leak coding defaults into os.environ across tests.
* CLI code_aware now defaults ON, matching the argparse server path
(degrades to a no-op without tree-sitter).

All explicit user env vars / CLI flags still win (setdefault + `or`
fallbacks). HEADROOM_LOSSLESS was already 0 by default; unchanged.

Tests: coding-profile + CLI-proxy-env tests updated to the new defaults;
1047 passed across the touched areas (only pre-existing memory/env
failures remain).

## Description

<!-- Briefly explain the change and why it is needed. -->

Closes #

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- 

## Testing

<!-- Check what you actually ran, then paste the real command output
below. -->

- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
# Paste relevant command output or artifact links here
```

## Real Behavior Proof

- Environment:
- Exact command / steps:
- Observed result:
- Not tested:

## Review Readiness

- [ ] I have performed a self-review
- [ ] This PR is ready for human review

## Checklist

- [ ] My code follows the project's style guidelines
- [ ] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

Add screenshots to help explain your changes.

## Additional Notes

<!-- Mention any N/A checklist items, tradeoffs, follow-ups, or
maintainer context. -->

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-09 07:49:54 -07:00
GUOHAO LIU b4205c68e6 fix(wrap): replace stale-proxy detection with Vite-style port fallback (#1406)
## Description

Replace the 180-line process-killing approach with a 15-line Vite-style
socket.bind probe: if port is busy (EADDRINUSE or EACCES), try the next
available port.

Pure stdlib -- no /proc, no lsof, no subprocess -- works identically on
Linux/macOS/Windows.

## Problem

When `headroom wrap <agent>` is killed without proper cleanup (window
close, SSH timeout, crash), the background proxy becomes orphaned and
holds the port. The next `headroom wrap` on the same port would wait
30-45 seconds then fail with a confusing error.

This PR takes a simpler, safer approach: find the next available port.
No process detection, no killing.

### Related issues

- **#589** (Port 8787 reserved by Windows) -- partially addressed:
EACCES is now skipped together with EADDRINUSE
- **#804** (Shared proxy killed by exiting session) -- already fixed
upstream via `_live_proxy_clients` marker files; this PR doesn't touch
that code

## Type of Change

- [x] New feature (non-breaking change that adds functionality)
- [x] Code refactoring (no functional changes)

## Changes Made

- `headroom/cli/wrap.py`: Add `_find_available_port()` -- socket.bind
loop that skips EADDRINUSE (busy) and EACCES (reserved/privileged)
ports, returns first available port in range. Replace `_ensure_proxy`
port-bind check with auto-fallback call to `_find_available_port`.
Remove `_ensure_port_free()` call from `_start_proxy()`. Remove 8 dead
functions: `_find_process_on_port`, `_linux_find_process_on_port`,
`_resolve_inode_to_pid`, `_is_headroom_proxy`, `_read_process_cmdline`,
`_kill_process`, `_ensure_port_free`, `_format_unbindable_port_error`.
- `tests/test_cli/test_wrap_helpers.py`: Remove 14 old
`TestEnsurePortFree` tests (mocked /proc parsing, process killing). Add
6 new `TestFindAvailablePort` tests covering: port free, first port
busy, multiple busy, EACCES skipped, unexpected error propagated, range
exhausted.
- `tests/test_cli/test_wrap_persistent.py`: Adapt persistence tests for
`_find_available_port` mock. Rewrite unbindable-port test to use new
error path.

Zero new dependencies. Zero changes to core proxy server, MCP,
compression, or providers.

## Testing

- [x] Unit tests pass (`python -m pytest tests/test_cli/ -v`)
- [x] New tests added for new functionality

### Test Output

```
> python -m pytest tests/test_cli/test_wrap_helpers.py::TestFindAvailablePort -v --no-header
============================= 6 passed ==============================
test_port_free_returns_same PASSED
test_port_busy_finds_next PASSED
test_multiple_busy_ports PASSED
test_propagates_unexpected_error PASSED
test_propagates_eaddrinuse_with_eacces PASSED
test_exhausts_range PASSED

> python -m pytest tests/test_cli/ -q
============================= 445 passed in 8.40s ==============================
```

## Real Behavior Proof

- Environment: Ubuntu 24.04 x86_64, Python 3.12.3
- Exact command / steps: Ran `python -m pytest
tests/test_cli/test_wrap_helpers.py::TestFindAvailablePort -v` -- 6/6
pass for port fallback. Ran full test suite `python -m pytest
tests/test_cli/ -q` -- 445/445 pass.
- Observed result: `_find_available_port(8787)` returns 8787 when free,
8788 when 8787 is busy. EACCES skipped same as EADDRINUSE. Non-retryable
errors (EADDRNOTAVAIL) propagate immediately.
- Not tested: Windows EACCES fallback (no Windows CI runner). macOS port
fallback (no macOS runner). Code path is identical across platforms
(stdlib socket only).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally
2026-07-07 12:10:52 -05:00
Abhay SinghandJD Davis c9d717c13c fix(wrap): remove rtk instructions from Codex AGENTS.md on unwrap (#1604)
## Description

`headroom wrap codex` injects Headroom's marker-fenced rtk instruction
block
into the Codex **global** `AGENTS.md` (`_codex_home_dir() /
"AGENTS.md"`), so
Codex voluntarily prefixes shell commands with `rtk`. But `headroom
unwrap
codex` only restored `config.toml` and cleaned up the MCP/Serena servers
— it
never removed that `AGENTS.md` block.

The result: after unwrapping, a plain `codex` launch still inherits
Headroom's
behavior and keeps trying to run `rtk`. If the managed rtk binary
directory is
no longer on `PATH`, commands fail outright:

```text
rtk : The term 'rtk' is not recognized as the name of a cmdlet, function, script file, or operable program.
Conversation interrupted
```

`unwrap copilot` already calls `_remove_rtk_instructions(...)`; Codex
was simply
missing the same cleanup step.

Closes #1421

## Fix

Call the existing `_remove_rtk_instructions` helper on the Codex global
`AGENTS.md` inside `unwrap_codex`, right after the MCP-server cleanup:

```python
if _remove_rtk_instructions(_codex_home_dir() / "AGENTS.md"):
    click.echo("  Removed Headroom rtk instructions from Codex AGENTS.md.")
```

The helper strips only the marker-fenced block and rewrites the rest of
the
file (deleting it only if nothing else remains), so user-authored
`AGENTS.md`
content is preserved. The call is unconditional and best-effort,
matching the
existing MCP-server cleanup in the same function.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/cli/wrap.py`: `unwrap_codex` now removes the marker-fenced
rtk block from the Codex global `AGENTS.md` via
`_remove_rtk_instructions`, with a status echo.
- `tests/test_cli/test_wrap_codex.py`: regression tests — block removed
on unwrap, surrounding user content preserved, and a no-op when
`AGENTS.md` is absent.
- `CHANGELOG.md`: Bug Fixes entry under Unreleased.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

Before the fix the two removal tests fail (the no-AGENTS.md safety test
passes
either way); after the fix the whole file is green:

```text
# before the fix (wrap.py reverted, tests kept)
FAILED tests/test_cli/test_wrap_codex.py::...::test_unwrap_removes_rtk_block_from_global_agents
FAILED tests/test_cli/test_wrap_codex.py::...::test_unwrap_preserves_user_content_in_global_agents
================= 2 failed, 1 passed, 66 deselected in 1.00s ==================

# after the fix
tests\test_cli\test_wrap_codex.py ......................................
...............................
============================= 69 passed in 7.45s ==============================
```

```text
$ uv run ruff check headroom/cli/wrap.py tests/test_cli/test_wrap_codex.py
All checks passed!
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12.11, headroom built from this
branch (`uv sync --extra dev`).
- Exact command / steps: set `CODEX_HOME` to a temp dir, wrote a user
`AGENTS.md`, injected the rtk block with the same helper `wrap codex`
uses, then ran the real `unwrap codex` command
(`unwrap_codex.callback(port=8787, no_stop_proxy=True)`) and re-read the
file. No mocking of the code under test.
- Observed result: the command printed `Removed Headroom rtk
instructions from Codex AGENTS.md.`, the rtk marker is gone, and the
user's own content survived:

```text
=== AGENTS.md BEFORE unwrap ===
# My rules

Always write tests.

<!-- headroom:rtk-instructions -->
# RTK (Rust Token Killer) - Token-Optimized Commands
...
<!-- /headroom:rtk-instructions -->
rtk marker present before: True

--- running: headroom unwrap codex --no-stop-proxy ---
  Removed Headroom rtk instructions from Codex AGENTS.md.
✓ Codex is no longer routed through the Headroom proxy.

=== AGENTS.md AFTER unwrap ===
# My rules

Always write tests.

rtk marker present after: False
user content preserved: True
```

- Not tested: did not run a full real `codex` binary session end-to-end
(not installed in this environment); the global-`AGENTS.md` state is the
durable thing the bug was about, and it's exercised here for real. Did
not run the full `mypy headroom` pass (one-line cleanup call, no new
types).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

- Single logical change, no new dependencies. Reuses the existing
`_remove_rtk_instructions` helper, so there's no new removal logic to
maintain.
- @chopratejas this mirrors the `unwrap copilot` cleanup; flagging you
since you've been triaging the wrap/unwrap issues.

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-07-02 23:55:41 -05:00
Rudimar Ronsoni ddd4adf911 fix(codex): avoid duplicate headroom provider config (#1431)
## Description

Fixes #1425.

`headroom wrap codex` could leave `~/.codex/config.toml` invalid when
the user already had a `[model_providers.headroom]` table. The previous
duplicate-key handling covered top-level `model_provider` and
`openai_base_url`, but the provider table was still appended as a static
block. That could produce duplicate `env_http_headers` or duplicate
provider-table TOML errors before Codex started.

## Type of Change

- [x] Bug fix (non-breaking change fixes an issue)
- [ ] New feature (non-breaking change adds functionality)
- [ ] Breaking change (fix or feature would cause existing functionality
change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Added a Codex config cleanup helper that removes any pre-existing
`[model_providers.headroom]` table from the working copy before `wrap
codex` appends the managed Headroom provider block.
- Kept unwrap behavior backed by the existing pre-wrap snapshot, so a
custom prior `headroom` provider table is restored byte-for-byte on
`headroom unwrap codex`.
- Added regression tests for TOML validity, a single `env_http_headers`
mapping, one managed `[model_providers.headroom]` table, and unwrap
restoration.

## Testing

- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added to cover the fix

### Test Output

```text
Docker: python:3.12-slim
Command: uv run --frozen --with pytest pytest tests/test_cli/test_wrap_codex.py
Result: 68 passed, 1 warning
```

## Real Behavior Proof

- Environment: disposable Docker container, `python:3.12-slim`, Linux,
Python 3.12.13.
- Exact command / steps: mounted the worktree into `/workspace`,
installed build tools inside the container, then ran `uv run --frozen
--with pytest pytest tests/test_cli/test_wrap_codex.py`.
- Observed result: all Codex wrap tests passed, including the new
regression where an existing `[model_providers.headroom]` table contains
`env_http_headers` before wrapping.
- Not tested: live interactive `headroom wrap codex` launch against a
real user Codex session.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review
2026-06-30 13:42:48 -05:00
Rod Boev 22def93177 fix(mcp): register managed installs with a resolvable headroom command (#1386)
## Description

Managed Headroom installs can register the MCP server with a bare
`headroom mcp serve` command even when the active runtime lives in a
venv outside `PATH`. That leaves Claude and Codex with a registration
they cannot re-launch reliably, and Claude eventually fails with `Failed
to reconnect to headroom: ENOENT`.

This PR reuses the existing runtime command resolver when building the
shared Headroom MCP spec, so the generated registration follows the
active install instead of assuming `headroom` is globally discoverable.
It also updates the shared-builder and registrar tests so the proof rows
now flow through `build_headroom_spec()` and prove the same resolved
command contract on both the Claude CLI path and the Codex TOML path. A
follow-up CI fix keeps the Docker init E2E expectation aligned with that
same resolver-backed contract.

Closes #487

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/mcp_registry/install.py`: build the Headroom MCP server spec
from the canonical runtime command resolver instead of hardcoding
`headroom mcp serve`
- `tests/test_mcp_registry/test_install.py`: cover the shared builder's
direct-binary and module-fallback command shapes
- `tests/test_mcp_registry/test_claude_registrar.py`: prove the Claude
CLI registration forwards the resolved command vector end to end
- `tests/test_mcp_registry/test_codex_registrar.py`: prove the Codex
registrar writes the same resolved command vector into TOML
- `e2e/init/run.py`: derive the Docker init E2E's expected Claude MCP
registration argv from `resolve_headroom_command()` so the CI harness
follows the same runtime contract
- `CHANGELOG.md`: note the managed-install MCP registration fix

## Testing

- [x] Unit tests pass (`uv run pytest
tests/test_mcp_registry/test_install.py -v`, `uv run pytest
tests/test_mcp_registry/test_claude_registrar.py -v`, `uv run pytest
tests/test_mcp_registry/test_codex_registrar.py -v`)
- [x] Linting passes (`uv run ruff check .`)
- [ ] Type checking passes (`uv run mypy headroom`) or explain N/A
truthfully
- [x] New tests added for new functionality when applicable
- [ ] Manual testing performed

### Test Output

```text
$ uv run pytest tests/test_mcp_registry/test_install.py -v
============================= 12 passed in 0.13s ==============================

$ uv run pytest tests/test_mcp_registry/test_claude_registrar.py -v
============================= 24 passed in 0.18s ==============================

$ uv run pytest tests/test_mcp_registry/test_codex_registrar.py -v
============================= 25 passed in 0.20s ==============================

$ uv run python -c "from e2e.init.run import _expected_headroom_mcp_call; print(_expected_headroom_mcp_call('http://127.0.0.1:9011'))"
['mcp', 'add', 'headroom', '-s', 'user', '-e', 'HEADROOM_PROXY_URL=http://127.0.0.1:9011', '--', '.../headroom', 'mcp', 'serve']

$ uv run ruff check e2e/init/run.py
All checks passed!

$ uv run ruff format e2e/init/run.py --check
1 file already formatted

$ uv run ruff check .
All checks passed!

$ uv run ruff format . --check
987 files already formatted
```

`uv run mypy headroom` was not run locally; this repo's focused local
gate for the touched Python registry path is the targeted pytest set
plus Ruff.

## Real Behavior Proof

- Environment: managed-install-safe MCP registration path, Python 3.11+,
no provider required
- Exact command / steps: run the focused MCP registry pytest files,
inspect the captured Claude CLI argv and rendered Codex TOML block, and
verify the Docker init E2E expectation derives its Claude MCP argv from
the same runtime helper
- Observed result: the persisted MCP registration uses a resolvable
command tied to the active Headroom runtime instead of bare `headroom`,
while `HEADROOM_PROXY_URL` handling stays unchanged
- Not tested: full live Claude reconnect against a real managed venv,
unless that is run during implementation

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

- Scoped to the MCP registration slice in `#487`. The RTK hook rewriting
thread from the same issue is intentionally out of scope here.
- `@erikpr1994` isolated the managed-install `ENOENT` failure mode in
the issue thread and narrowed it to the bare-command MCP registration
path.
- If existing owned registrations with the old bare-command contract
need an in-place upgrade path, that should be handled explicitly in the
final diff rather than left implicit.
2026-06-26 23:39:00 -05:00
Shengbo_WangandClaude Opus 4.6 a0cb7982e3 fix(cli): add explicit UTF-8 encoding to file I/O in wrap commands (#1126) (#1164)
## Description

On Windows, `Path.read_text()` and `open()` default to the system locale
encoding (cp1252, GBK, etc.) instead of UTF-8. This causes
`UnicodeDecodeError` when reading or writing instruction files that
contain multi-byte UTF-8 characters such as smart quotes or em dashes.

The RTK instructions block itself contains an em dash (U+2014, `—`), so
`_inject_rtk_instructions` crashes on any non-UTF-8 Windows locale when
writing to `.copilot_instructions`, `.clinerules`, `AGENTS.md`, or
similar hint files.

Closes #1126

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Add `encoding="utf-8"` to all bare `read_text()`, `write_text()`, and
`open()` calls in `headroom/cli/wrap.py` that handle instruction or
config files (18 call sites)
- Update test assertions in `test_wrap_hintfile_agents.py`,
`test_wrap_copilot.py`, and `test_wrap_bridge.py` to read with
`encoding="utf-8"`
- Add `test_inject_rtk_handles_utf8_content` verifying that existing
hint files with smart quotes and em dashes survive RTK injection without
crashing

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_cli/test_wrap_hintfile_agents.py tests/test_cli/test_wrap_copilot.py tests/test_cli/test_wrap_bridge.py -v
47 passed in 1.28s
```

## Real Behavior Proof

- Environment: Windows 11 China (GBK locale), Python 3.11, headroom main
(f03e77b)
- Exact command / steps: python -m pytest
tests/test_cli/test_wrap_hintfile_agents.py -v (on GBK Windows)
- Observed result: Before fix,
test_prepare_only_injects_rtk_into_hintfile fails with
UnicodeDecodeError 'gbk' at position 283 (the em dash in RTK block).
After fix, all 12 hintfile tests pass including new UTF-8 round-trip
test.
- Not tested: no manual `headroom wrap copilot` run against a real
Copilot installation

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

This is the same class of bug reported in #733 (GBK config.toml
corruption). This PR fixes the `wrap.py` call sites; other modules
(`learn/analyzer.py`, `install/providers.py`) have the same pattern and
could benefit from the same treatment in a follow-up.

---------

Signed-off-by: Yiming Zeng <yzeng424@gmail.com>
Signed-off-by: RTCartist <wangshengb@buaa.edu.cn>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-06-26 12:07:03 -05:00
Rod Boev ad7993bf15 fix(codex): stop pinning Codex memory MCP to one project db (#1269)
## Description

Stop `headroom wrap codex --memory` from pinning the global
`headroom_memory` MCP server to one absolute SQLite path. Today the
wrapper writes `--db <wrap-cwd>/.headroom/memory.db` into
`~/.codex/config.toml`, which makes later Codex sessions either reopen a
stale project-local DB or fail with `unable to open database file` when
that original path disappears. This change lets the MCP server use its
existing per-cwd default again, so each Codex session resolves
`.headroom/memory.db` from the active project instead of a serialized
past cwd. Closes #1147

The current Codex-memory config surface was shaped by
https://github.com/chopratejas/headroom/issues/462 and
https://github.com/chopratejas/headroom/issues/730; this PR keeps that
surface project-scoped again instead of globally pinning one DB.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- remove the injected `--db` argument from the global `headroom_memory`
Codex MCP block while keeping `--user` intact
- preserve the wrap-time local `.headroom/memory.db` setup and
Claude-memory import path for the current project
- treat only wrap-owned Codex markers as snapshot-suppression and
unwrap-cleanup signals, so pre-existing named MCP blocks still back up
and restore
- log a startup diagnostic from `headroom.memory.mcp_server` that
records the configured DB path, config source, cwd/project root,
resolved storage scope, path existence/readability, and whether the path
was static or cwd-derived
- add a shared MCP SDK test stub so both the memory MCP and CCR MCP test
surfaces still run in CI when `mcp` is absent
- make the shared MCP stub re-import target modules under the stubbed
dependency set and restore any pre-existing target module object plus
dotted parent-package attribute state after cleanup
- add focused regressions and guard coverage for the persisted Codex
config shape, named-MCP marker backup and restore, the no-backup
memory-only unwrap path, the wrap-memory-then-unwrap cleanup path, the
failed-wrap memory-only cleanup path, the startup-diagnostic path
classification, the shared-store CCR retrieval path, and the shared MCP
stub import lifecycle
- add a `CHANGELOG.md` entry for the user-visible Codex memory scoping
fix

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`uv run mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
uv run pytest tests/test_ccr_mcp_server.py tests/test_memory/test_mcp_server.py tests/test_cli/test_wrap_codex.py tests/test_mcp_stub.py
======================== 78 passed, 1 warning in 5.96s ========================
Pytest warning:
PytestConfigWarning: Unknown config option: asyncio_mode
Pytest post-success atexit noise:
PermissionError: [WinError 5] Access is denied: 'C:\Users\Rod\AppData\Local\Temp\pytest-of-Rod\pytest-current'

uv run ruff check headroom/cli/wrap.py headroom/memory/mcp_server.py tests/_mcp_stub.py tests/test_ccr_mcp_server.py tests/test_cli/test_wrap_codex.py tests/test_mcp_stub.py tests/test_memory/test_mcp_server.py
All checks passed!

uv run ruff format headroom/cli/wrap.py headroom/memory/mcp_server.py tests/_mcp_stub.py tests/test_ccr_mcp_server.py tests/test_cli/test_wrap_codex.py tests/test_mcp_stub.py tests/test_memory/test_mcp_server.py --check
7 files already formatted
```

## Real Behavior Proof

- Environment: isolated temp project directories, a temp Codex home, the
real `wrap codex` and `unwrap codex` CLI commands under pytest, a mocked
missing-`codex` launch path for the failed-wrap cleanup case, and shared
MCP-SDK stubs for the memory MCP and CCR MCP test modules so CI still
exercises those paths without a real `mcp` install.
- Exact command / steps: run `uv run pytest tests/test_ccr_mcp_server.py
tests/test_memory/test_mcp_server.py tests/test_cli/test_wrap_codex.py
tests/test_mcp_stub.py`; prove the persisted config shape with
`TestCodexMemoryMcpConfig::test_inject_omits_db_and_replaces_existing_memory_block`;
prove prepare-only wrap cleanup with
`test_wrap_codex_memory_prepare_only_unwrap_removes_memory_mcp_without_prior_config`;
prove failed-wrap cleanup with
`test_wrap_codex_memory_launch_failure_unwrap_cleans_memory_only_config`;
guard pre-existing named Codex MCP preservation with
`test_memory_only_wrap_restores_preexisting_named_mcp_block` and
`test_memory_only_wrap_without_backup_preserves_named_mcp_block`; prove
the startup diagnostic classifications with
`test_memory_mcp_startup_context_reports_dynamic_project_db` and
`test_memory_mcp_startup_context_reports_static_external_db`; prove the
shared-store CCR retrieval path with
`test_mcp_uses_shared_singleton_store` and
`test_mcp_retrieves_proxy_stored_content`; prove stub import cleanup
with `test_import_module_with_mcp_stub_imports_target_and_cleans_up`,
`test_import_module_with_mcp_stub_reimports_target_and_restores_originals`,
and
`test_import_module_with_mcp_stub_cleans_up_dotted_target_attribute`.
- Observed result: the persisted global `headroom_memory` block now
keeps `--user` but omits `--db`; prepare-only memory setup still
bootstraps the current project's `.headroom/memory.db`; `headroom unwrap
codex --no-stop-proxy` now removes both the prepare-only generated
config and the failed-wrap memory-only config instead of leaving
`[mcp_servers.headroom_memory]` behind; pre-existing named Codex MCP
blocks remain restorable across both normal and no-backup memory-only
unwrap paths because only wrap-owned markers suppress backups or trigger
named-block cleanup; the memory MCP server now logs whether its DB path
came from the cwd default or an explicit static path, along with the
resolved path and scope it will open; CI can exercise both MCP test
modules even when the `mcp` package is absent from the shard
environment, and the shared stub now re-imports target modules under the
stubbed SDK while restoring both dependency and dotted parent-package
target-module import state after cleanup.
- Not tested: full end-to-end interactive Codex CLI launch.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

The code change stays narrowly scoped to Codex memory config
persistence, cleanup, and startup observability. It does not widen into
larger memory-routing redesign or startup-failure recovery logic.
2026-06-23 07:49:07 -05:00
JD Davis b829ceba84 fix(wrap): keep agent savings opt-in (#1294)
## Description

Fixes a regression from #830 where `headroom wrap codex` / `claude` /
`cursor` treated `agent-90` as required even when the user started a
normal proxy without `HEADROOM_SAVINGS_PROFILE`.

A plain `headroom proxy` on port 8787 followed by `headroom wrap codex`
currently reports `Proxy on port 8787 is missing: --savings-profile` and
tries to restart the already-running proxy. The `agent-90` profile was
documented as opt-in, so wrap should only require or inject it when
`HEADROOM_SAVINGS_PROFILE` is explicitly set.

Closes #1293

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Stop defaulting agent wrappers to `agent-90` when
`HEADROOM_SAVINGS_PROFILE` is unset.
- Stop reporting agent-savings config mismatches unless an agent savings
profile was explicitly requested.
- Add regression tests for default wrap startup, explicit profile
forwarding, and reuse/restart behavior around existing proxies.
- Fix repo-wide pre-commit issues found during amend: Windows-safe
`fcntl` typing, an optional env typing issue, OpenCode JSON parser
return typing, and ruff import/format drift.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
> maturin develop -m crates/headroom-py/Cargo.toml
Finished `dev` profile [unoptimized + debuginfo] target(s) in 27.41s
Built wheel for abi3 Python >= 3.10
Installed headroom-ai-0.27.0

> C:\git\headroom\.venv\Scripts\python.exe -c "from headroom._core import hello; print(hello())"
headroom-core

> ruff check .
All checks passed!

> ruff format --check .
913 files already formatted

> C:\git\headroom\.venv\Scripts\python.exe -m mypy headroom
Success: no issues found in 388 source files

> C:\git\headroom\.venv\Scripts\python.exe -m pytest tests/test_agent_savings.py tests/test_cli/test_wrap_persistent.py tests/test_proxy_healthchecks.py tests/test_transforms/test_content_router.py -q
collected 112 items
112 passed, 1 warning in 16.04s

> git diff --check
# no output

> git commit --amend --no-edit
Sync plugin versions.....................................................Passed
ruff.....................................................................Passed
ruff-format..............................................................Passed
mypy.....................................................................Passed
```

## Real Behavior Proof

- Environment: Windows dev checkout, Python 3.13.3 via
`C:\git\headroom\.venv\Scripts\python.exe`, Rust extension built with
`maturin develop -m crates/headroom-py/Cargo.toml`.
- Exact command / steps: reproduced the code path from #830 by
exercising `_ensure_proxy(8787, False, agent_type="codex")` with a
running proxy health payload that has no `savings_profile`, and by
exercising `_start_proxy(8787, agent_type="codex")` with
`HEADROOM_SAVINGS_PROFILE` unset and set.
- Observed result: without `HEADROOM_SAVINGS_PROFILE`, wrap reuses the
running proxy and `_start_proxy` does not inject
`HEADROOM_SAVINGS_PROFILE` or `HEADROOM_TARGET_RATIO`; with
`HEADROOM_SAVINGS_PROFILE=agent-90`, wrap still requires/forwards the
profile and restarts an incompatible proxy.
- Not tested: full end-to-end CLI launch against a live Codex binary.
The focused proxy/wrap tests cover the failing restart/config decision
directly.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A

## Additional Notes

The pytest run emits an existing Windows `cp1252` background-thread
warning while reading subprocess output; the tests still pass.

No documentation or changelog update is included because this restores
the already-documented opt-in behavior for `agent-90`.
2026-06-22 19:06:04 -05:00
Terminal Chai 7c26a54d53 fix(wrap): keep Codex RTK guidance global (#1240)
## Description

Stops `headroom wrap codex` from writing RTK instructions into the
shared project `AGENTS.md`. RTK guidance remains installed in the global
Codex `AGENTS.md`, where it applies only to the user who configured
Headroom.

Closes #1235

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Remove project-level RTK guidance injection from `headroom wrap
codex`.
- Preserve global Codex RTK guidance injection.
- Add a regression test proving an existing project `AGENTS.md` remains
byte-for-byte unchanged.
- Document the fix in the Unreleased changelog.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --extra dev pytest tests/test_cli/test_wrap_codex.py -q
57 passed in 9.54s

$ uv run ruff check headroom/cli/wrap.py tests/test_cli/test_wrap_codex.py
All checks passed!

$ uv run ruff format --check headroom/cli/wrap.py tests/test_cli/test_wrap_codex.py
2 files already formatted

$ uv run mypy headroom/cli/wrap.py
Success: no issues found in 1 source file

$ npx --yes --package=@commitlint/cli --package=@commitlint/config-conventional commitlint --from HEAD~1 --to HEAD --config .commitlintrc.json
exited 0
```

## Real Behavior Proof

- Environment: Windows, Python 3.12.12, locally built Headroom CLI,
isolated project directory, isolated `CODEX_HOME`, and isolated
`HEADROOM_WORKSPACE_DIR`.
- Exact command / steps: created a project `AGENTS.md`, recorded its
SHA-256, then ran `.venv\Scripts\headroom.exe wrap codex --prepare-only
--no-mcp --no-serena` with isolated environment directories and compared
the project hash before and after.
- Observed result: command exited 0; RTK downloaded successfully; the
project `AGENTS.md` hash remained
`2CFF2F420178BFEB9BB863C743805410F2CA30F3F7F70121A8538314CBD0F8B5`; the
global Codex `AGENTS.md` was created and contained the
`headroom:rtk-instructions` marker.
- Not tested: launching an interactive Codex session after preparation;
non-Codex wrapper targets, which are unchanged.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

Not applicable.

## Additional Notes

The repository-wide pre-commit mypy hook reports existing Windows-only
`fcntl` attribute errors in `headroom/subscription/tracker.py` and
`headroom/install/runtime.py`; targeted mypy for the changed module
passes. The plugin-version hook was also verified directly with the
project interpreter and correctly skipped this feature branch.

This pull request includes code written with the assistance of AI. The
changes have not yet been reviewed by a human.
2026-06-21 10:11:06 -07:00
gglucassandClaude Opus 4.8 74ae781644 fix(codex): retag thread providers so history menu stays whole across the proxy boundary (#1034)
## Description

Codex stamps every thread with the `model_provider` it ran under and
filters its
history/projects menu by the currently active provider set. When
Headroom rewrites
Codex's config to route through the custom `headroom` provider, threads
created through
Headroom are tagged `headroom` while native threads keep `openai` — so
the two sets
never appear in the same menu. The visible symptom: enabling Headroom
appears to "lose"
the entire native Codex history, and disabling it hides everything
created while wrapped.

This reconciles the thread tags in Codex's SQLite store alongside the
existing config
edits, so the menu stays whole across the proxy boundary in both
directions:
`openai -> headroom` on enable/wrap, `headroom -> openai` on
revert/unwrap. Only rows
matching the source provider are touched, so third-party providers (e.g.
`anthropic`)
are left alone. The provider key cannot be unified by config — Codex
rejects naming a
custom provider `openai` ("reserved built-in provider IDs") — so
retagging the store is
the only path. A DB-only retag is sufficient: resuming a retagged
session still routes
completions through the active provider; the rollout `.jsonl` files do
not need
rewriting.

Every operation is best-effort: a missing store, a missing `threads`
table, or a corrupt
store is logged and skipped, never raised, so install/uninstall and
wrap/unwrap never
fail on account of the history menu. The store is WAL-mode, so the
update succeeds even
while Codex is running; the short busy timeout only covers a transient
checkpoint lock.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- New `headroom/providers/codex/threads.py`: best-effort retag of Codex
thread provider
tags across both known stores (`<codex_home>/sqlite/state_5.sqlite` for
the GUI and
`<codex_home>/state_5.sqlite` for the CLI/TUI). `retag_to_headroom` /
`retag_to_native`
wrap the directional helper. `codex_home` is passed in by callers (never
re-derived from
  `Path.home()`), so tests stay pointed at a temp dir.
- `providers/codex/install.py`: `apply_provider_scope` calls
`retag_to_headroom` after
writing the provider block; `revert_provider_scope` calls
`retag_to_native` after
  stripping it.
- `cli/wrap.py`: `_inject_codex_provider_config` calls
`retag_to_headroom`;
`unwrap_codex` calls `retag_to_native` once the config restore reports a
  `restored`/`cleaned`/`removed` status.
- Tests: `tests/test_provider_codex_threads.py` (retag direction,
threads-table no-op,
missing/corrupt store best-effort) and a wrap/unwrap round-trip
integration test in
  `tests/test_cli/test_wrap_codex.py`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ uv run --extra dev pytest tests/test_provider_codex_threads.py tests/test_cli/test_wrap_codex.py tests/test_install/test_providers.py -q
... 88 passed, 2 failed
# The 2 failures are TestInjectAvoidsDuplicateTopLevelKeys::* — pre-existing on a clean
# upstream/main checkout, unrelated to this change: they `import tomllib`, which is
# stdlib only on Python 3.11+, and this environment runs Python 3.10.18.

$ uv run --extra dev ruff check headroom/cli/wrap.py headroom/providers/codex/threads.py \
    headroom/providers/codex/install.py tests/test_cli/test_wrap_codex.py tests/test_provider_codex_threads.py
All checks passed!

$ uv run --extra dev mypy headroom/providers/codex/threads.py headroom/providers/codex/install.py
Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: macOS, Codex GUI v148 + Codex CLI, Python 3.10.18.
- Exact command / steps: The root cause and fix were confirmed live in
the Headroom
desktop app, which performs the identical SQLite retag. Connecting Codex
to Headroom
  hid ~140 native (`openai`) threads from the history menu; running
`UPDATE threads SET model_provider='headroom' WHERE
model_provider='openai'` on the live
store (`~/.codex/sqlite/state_5.sqlite`) made the full menu reappear,
and resuming a
retagged session still routed completions through the active provider.
This Python port
is a 1:1 of that logic, exercised by the unit + wrap/unwrap integration
tests above.
- Observed result: full Codex history menu restored across
enable/disable; third-party
provider rows untouched; the real `~/.codex` stores were snapshotted
before/after the
  test run and were not mutated by the tests.
- Not tested: an end-to-end `headroom wrap codex` run against a live
Codex GUI in this CI
environment (no Codex install here); covered instead by the integration
test invoking
  the real `wrap`/`unwrap` Click commands against a temp `$HOME`.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A — behavior is in Codex's own history menu; covered by the proof
above.

## Additional Notes

- Documentation / CHANGELOG: N/A — internal behavior with no user-facing
surface beyond
  the restored menu.
- The 2 failing `TestInjectAvoidsDuplicateTopLevelKeys` tests are
pre-existing on
upstream/main and fail only because this environment runs Python 3.10
(no `tomllib`);
  they are unrelated to this change.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 15:13:21 -05:00
wangxiangyu7andwangxiangyu7 dd22cfd72a fix(wrap): avoid duplicate top-level keys when injecting codex provider (#884)
## Description

`_inject_codex_provider_config` in `headroom/cli/wrap.py`
unconditionally prepended a top-level block to `~/.codex/config.toml`:

```toml
# --- Headroom proxy (auto-injected by headroom wrap codex) ---
model_provider = "headroom"
openai_base_url = "http://127.0.0.1:8787/v1"
# --- end Headroom ---
```

If the user already had a top-level `model_provider` (or
`openai_base_url`), the result was two top-level keys with the same
name. That violates the TOML spec, and Codex refuses to start with
`duplicate key`. This change makes the injector rewrite any pre-existing
top-level `model_provider` / `openai_base_url` in place to the headroom
values (keeping the user's original value in a `# was: …` trailing
comment) and only emit the marker-delimited top-level block for keys the
user has not declared. The pre-wrap snapshot mechanism is unchanged, so
`headroom unwrap codex` still restores the file byte-for-byte.

Closes #883

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/cli/wrap.py`
- New helper `_redirect_existing_top_level_keys(content, port)`:
rewrites existing top-level `model_provider` / `openai_base_url` lines
to the headroom values and preserves the previous value in a trailing `#
was: …` comment.
- New helper `_has_redirectable_top_level_key(content, key)`: cheap
predicate for the two redirectable keys.
- New helper `_build_top_level_block(user_content)`: emits a
marker-delimited block containing only the redirectable keys the user
has **not** already declared (declared ones are rewritten in place
instead, avoiding the TOML duplicate-key error).
- `_inject_codex_provider_config` now rewrites declared keys in place
and only prepends the marker block for the remaining keys;
`requires_openai_auth` handling (#406) is preserved.
- `tests/test_cli/test_wrap_codex.py`
- New class `TestInjectAvoidsDuplicateTopLevelKeys` (TOML-validity after
wrap on a config already declaring a provider, original-value
preservation in a `# was:` comment, idempotent re-wrap with a port
change, marker-block fallback on an empty file, snapshot-based unwrap
restoration). The TOML-validity test parses the wrapped file with
`tomllib.loads`, which fails before the fix and passes after.
- `CHANGELOG.md`
  - Added entry under `## Unreleased` → `### Bug Fixes`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom --ignore-missing-imports`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run pytest tests/test_cli/test_wrap_codex.py -q
======================== 52 passed, 1 warning in 5.28s =========================

$ uv run ruff check headroom/cli/wrap.py tests/test_cli/test_wrap_codex.py
All checks passed!

$ uv run ruff format --check headroom/cli/wrap.py tests/test_cli/test_wrap_codex.py
2 files already formatted

$ mypy headroom --ignore-missing-imports
Success: no issues found in 358 source files
```

## Real Behavior Proof

- Environment: macOS 24.6.0, Python 3.13.3, branch
`fix/wrap-codex-no-duplicate-keys` rebased onto upstream `main`, Codex
CLI config at `~/.codex/config.toml`.
- Exact command / steps: Seed a user config matching the bug report
(`model_provider = "ccswitch"` + `openai_base_url = "…"` +
`[model_providers.ccswitch]`), run the same path `headroom wrap codex`
takes (`_inject_codex_provider_config(8787)`), then parse the result
with `tomllib.loads(...)` and run `headroom unwrap codex`.
- Observed result: On patched code the wrapped `config.toml` parses
cleanly — exactly one `model_provider` and one `openai_base_url` remain
(the user's prior value preserved in a `# was: …` comment) and the
`[model_providers.headroom]` table is present; `unwrap` restores the
file byte-for-byte. On the unpatched code the same file raises
`tomllib.TOMLDecodeError` (duplicate key). Full suite: 52 passed, ruff
lint + format clean (see Test Output).
- Not tested: End-to-end launch of the Codex CLI against a live proxy
(no Codex e2e harness in this sandbox); Windows / `$CODEX_HOME` override
paths (covered only by existing tests).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A — CLI/config change, no UI.

## Additional Notes

`ruff check .`, `ruff format --check .`, and `mypy headroom
--ignore-missing-imports` all pass on the rebased branch. The diff stays
narrow: `headroom/cli/wrap.py` + its tests + a CHANGELOG entry.

Co-authored-by: wangxiangyu7 <wangxiangyu7@lixiang.com>
2026-06-15 11:04:29 -05:00
gglucassandClaude Opus 4.8 8c00f7103c fix(codex): poll /wham/usage for subscription limits (handshake no longer sends x-codex-* headers) (#924)
## Description

Codex's subscription usage window (primary/secondary rate-limit gauges)
stopped populating for ChatGPT-OAuth sessions. This PR restores it by
polling Codex's dedicated usage endpoint instead of relying on response
headers that are no longer sent.

### Why the previous approach no longer works

The existing code populates `CodexRateLimitState` from `x-codex-*`
rate-limit headers captured on the `/v1/responses` WebSocket handshake
(`update_from_headers` at WS accept). That worked when OpenAI returned
`x-codex-primary-used-percent`, `x-codex-primary-window-minutes`, etc.
on the handshake response.

OpenAI has since stopped sending those headers on the ChatGPT WebSocket
handshake. I confirmed this by faithfully replaying a real Plus-account
handshake (both `prewarm` and regular `request_kind`): no `x-codex-*`
headers come back on either. This matches OpenAI's own move to a
dedicated usage endpoint (`GET /backend-api/codex/usage` in CodexApi
mode) and reports such as openai/codex#14728. So `update_from_headers`
now runs on every accept but finds nothing to parse, and the window
silently stays empty.

The headers aren't coming back, so there is nothing to fix in the
parsing path. The data now lives behind a request we have to make
ourselves.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `subscription/codex_rate_limits.py`:
- `parse_codex_usage_payload()` / `update_from_usage_payload()` — map
the `GET /backend-api/wham/usage` JSON (`rate_limit.primary_window` /
`secondary_window` with `used_percent`, `limit_window_seconds`,
`reset_at`; `credits`; `rate_limit_reached_type`) into the existing
`CodexRateLimitState`. `limit_window_seconds` is converted to
window-minutes with the same round-up codex-rs uses (`(secs + 59) //
60`).
- `maybe_schedule_usage_poll()` — fire-and-forget, throttled to one
request per 60s, scoped to ChatGPT sessions (requires both a Bearer
token and `ChatGPT-Account-Id`; API-key traffic is skipped). Uses an
in-flight guard so concurrent accepts don't stack polls. Endpoint is
overridable via `HEADROOM_CODEX_USAGE_URL`.
- `proxy/handlers/openai.py`:
- At the Codex WS accept site, after the now-usually-empty
`update_from_headers` block, schedule the usage poll. Wrapped in
`contextlib.suppress` and fully non-blocking so it can never delay or
fail the WebSocket accept.

The old header-capture path is intentionally left in place as a no-cost
fallback in case OpenAI restores the headers.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed (live `/wham/usage` replay against a Plus
account returned HTTP 200 with the expected schema; payload fixture in
tests mirrors that real shape)

## Test Output

```
$ uv run pytest tests/test_codex_rate_limits.py -q
........................................                                 [100%]
41 passed in 0.16s

$ uv run ruff check headroom/subscription/codex_rate_limits.py headroom/proxy/handlers/openai.py tests/test_codex_rate_limits.py
All checks passed!

$ uv run mypy headroom/subscription/codex_rate_limits.py
Success: no issues found in 1 source file
```

## Additional Notes

- New tests cover: full-payload mapping, window-minutes round-up,
credits balance kept only when `has_credits`, promo object vs string,
empty payload returns `None`, missing `used_percent` skipped,
header-gating (requires Bearer + account-id), poll throttling, and
no-event-loop safety.
- Scoping to `ChatGPT-Account-Id` keeps the poll off API-key traffic,
and the 60s throttle plus in-flight guard bound it to at most one
lightweight GET per minute per running proxy.

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-12 17:03:14 -05:00
Michael Sam 6d3f39f213 feat: add dashboard agent usage stats (#814)
## Description

Add a clear dashboard view for per-agent token usage so end users can
see Cursor, Claude, Codex, and other detected clients with before/after
token counts, tokens saved, and savings percentages. The stats API now
exposes a stable `agent_usage` object that the dashboard renders near
the top of the session view.

Fixes #

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

### New Files

**Tests:**
- `tests/test_dashboard_agent_usage.py` — Covers agent classification,
exact per-request aggregation, and aggregate fallback behavior.

### Modified Files

- `headroom/proxy/server.py` — Adds per-agent usage aggregation to
`/stats` with before tokens, after tokens, output tokens, saved tokens,
savings percentage, source, providers, and models.
- `headroom/dashboard/templates/dashboard.html` — Adds a prominent Agent
Usage panel with totals, coverage status, per-agent token-flow bars,
request counts, before/after tokens, saved tokens, and share of savings.

## Testing

- [x] Unit tests pass: `.venv312/bin/pytest
tests/test_dashboard_agent_usage.py`
- [x] Linting passes: `.venv312/bin/ruff check headroom/proxy/server.py
tests/test_dashboard_agent_usage.py`
- [x] Diff whitespace check passes: `git diff --check
origin/main...HEAD`
- [x] Dashboard smoke render: local proxy on `127.0.0.1:8790`, captured
Chrome headless screenshot of `/dashboard`
- [x] New tests added for new functionality

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing relevant unit tests pass locally with my changes
- [ ] I have made corresponding changes to the documentation
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

The agent usage panel uses exact request-log data when available. If
detailed request logs are empty, it falls back to aggregate
provider/model request counts and labels the coverage as aggregate
fallback so users are not misled.
2026-06-12 14:12:22 -05:00
Focused InstabilityandAsh Rhodes 914a60a2b0 feat(proxy): per-project savings breakdown on the dashboard (claude, codex, aider, copilot, cursor) (#803)
## Summary

Adds a **per-project savings breakdown** to the proxy dashboard,
covering **all wrap-supported agents: Claude Code, Codex, aider,
Copilot, and Cursor**. Spec and rationale in #802 (feature-request issue
— happy to adjust scope per maintainer feedback).

How it works — two attribution channels, by client capability:

**Header channel** (clients that can send custom headers):
- `headroom wrap claude` appends `X-Headroom-Project: <basename(cwd)>`
to `ANTHROPIC_CUSTOM_HEADERS` (a user-supplied x-headroom-project header
always wins; other user headers preserved).
- `headroom wrap codex` extends the injected
`[model_providers.headroom]` block with `env_http_headers = {
"X-Headroom-Project" = "HEADROOM_PROJECT" }` and sets `HEADROOM_PROJECT`
per launch — Codex only sends the header when the env var is set, so the
static config stays inert outside wrap.

**Base-URL prefix channel** (clients that cannot send custom headers):
- The proxy accepts `/p/<url-encoded-name>/...`; middleware strips the
prefix before routing (before Starlette caches the URL) and binds the
project. The explicit header wins over the prefix.
- `headroom wrap aider` points `OPENAI_API_BASE` / `ANTHROPIC_BASE_URL`
at the prefixed URL.
- `headroom wrap copilot` points `COPILOT_PROVIDER_BASE_URL` at the
prefixed URL (BYOK anthropic/openai provider types and the
GitHub-subscription path).
- `headroom wrap cursor` prints the prefixed Override Base URL in its
setup instructions, with a note explaining the attribution.

**Shared plumbing:**
- New `headroom/proxy/project_context.py`: header classification, `/p/`
prefix split/strip, URL-prefix builder, and a contextvar bound per
request (HTTP middleware + WS accept for the Codex responses bridge);
the outcome funnel resolves it (explicit `RequestOutcome.project` wins),
stamps `RequestLog.tags["project"]`, and forwards it through
`PrometheusMetrics.record_request` into the `SavingsTracker`.
- `SavingsTracker`: persisted state gains a `projects` map (requests,
tokens saved, savings USD, input tokens/cost, last activity). Schema v2
→ v3 with transparent forward migration (v2 files load cleanly,
`projects` starts empty). Names sanitized (printable-only, 128-char
cap); map capped at 50 projects, evicting the smallest bucket.
- `/stats` exposes `savings.per_project` (and
`projects`/`projects_limit` inside `persistent_savings`);
`/stats-history` exposes `projects`.
- Dashboard gains a "Per-Project Savings" table mirroring the per-model
table (Alpine `x-text` only — names are user-supplied, no HTML
injection).

**Behavior changes:** none for unattributed traffic — no header and no
prefix means no bucket, aggregate totals exactly as before
(regression-tested: legacy `/stats` and `/stats-history` shapes are
pinned by tests).

## Real behavior proof

**Setup tested on:** macOS (Darwin 25.5.0), Python 3.11.9, `pip install
-e ".[dev]"`, proxy on spare ports with isolated
`HEADROOM_SAVINGS_PATH`; real Claude Code CLI (subscription auth) and
real Codex CLI (ChatGPT auth) as clients.

**Header channel — exact steps:**

```
HEADROOM_SAVINGS_PATH=/tmp/headroom-proof-savings.json .venv/bin/python -m headroom.cli proxy --port 9123  # background
cd /tmp/proof-alpha && headroom wrap claude --port 9123 --no-rtk --no-mcp --no-serena -- -p "Reply with exactly: OK"
cd /tmp/proof-beta  && headroom wrap claude --port 9123 --no-rtk --no-mcp --no-serena -- -p "Reply with exactly: OK"
cd /tmp/proof-beta  && headroom wrap codex  --port 9123 --no-rtk --no-mcp --no-serena -- exec --skip-git-repo-check "Reply with exactly: OK"
```

**Observed** (copied live `/stats` output; all runs replied `OK` through
the proxy):

```json
{
 "proof-beta": {
  "requests": 3, "tokens_saved": 140, "compression_savings_usd": 0.0007,
  "total_input_tokens": 38581, "total_input_cost_usd": 0.360433,
  "last_activity_at": "2026-06-10T09:19:17Z", "savings_percent": 0.36
 },
 "proof-alpha": {
  "requests": 1, "tokens_saved": 0, "compression_savings_usd": 0.0,
  "total_input_tokens": 9050, "total_input_cost_usd": 0.32245,
  "last_activity_at": "2026-06-10T09:18:09Z", "savings_percent": 0.0
 }
}
```

`proof-beta` aggregates one Claude Code turn + one Codex `exec` run
(Codex attribution flows through `env_http_headers`).

**Prefix channel — exact steps** (the same mechanism the
aider/copilot/cursor wraps emit, driven by a real client):

```
.venv/bin/python -m headroom.cli proxy --port 9124  # background, isolated savings path
ANTHROPIC_BASE_URL="http://127.0.0.1:9124/p/aider-style-project" claude -p "Reply with exactly: OK"
```

**Observed:**

```json
{
 "aider-style-project": {
  "requests": 1, "tokens_saved": 0, "compression_savings_usd": 0.0,
  "total_input_tokens": 8571, "total_input_cost_usd": 1.018575,
  "last_activity_at": "2026-06-10T10:10:13Z", "savings_percent": 0.0
 }
}
```

`/stats-history` returned `schema_version: 3` with the same `projects`
keys; `GET /dashboard` HTML contains the new "Per-Project Savings"
table; persisted state survived a tracker reload; a hand-written v2
savings file loaded cleanly with an empty `projects` map.

**What I did NOT test live:** the actual aider/Copilot/Cursor binaries
end-to-end (their wraps emit exactly the prefixed URLs exercised above —
unit tests pin the emitted env/URLs); Codex subscription-mode routing
via the built-in `openai` provider (no provider headers there — such
traffic simply stays unattributed); multi-worker uvicorn; Windows.

## Tests

- `tests/test_proxy_project_savings.py` (17 tests): sanitization, header
classification, `/p/` prefix split + URL-builder round-trip, tracker
aggregation/persistence/migration/cardinality-cap/state-sanitization,
funnel→`/stats` end-to-end, middleware binding for header + prefix +
precedence, plus regression tests pinning legacy
`/stats`/`/stats-history` shape and unattributed-traffic totals.
- Wrap/provider tests extended: `test_cli/test_wrap_helpers.py`,
`test_cli/test_wrap_codex.py`, `test_provider_aider.py`,
`test_provider_cursor.py`, `test_provider_copilot_wrap.py` (prefixed
env/URLs, user-override wins, no duplicate header, TOML block contents,
block strip, setup-line note).
- Full `pytest` suite run locally; `ruff check` + `ruff format` clean on
all touched files.
- `CHANGELOG.md` updated.

## Dependencies

None added or bumped.

Closes #802 (pending maintainer 👍 per CONTRIBUTING — raised there first
with the full spec).

---------

Co-authored-by: Ash Rhodes <ashley.rhodes@king.com>
2026-06-10 21:04:45 -05:00
Gonzalo Zanelli 96abf38b09 fix(codex): respect CODEX_HOME for wrap config (#731)
> ⚠️ This PR was opened using Codex (gpt-5.5 on `xhigh`)

Fixes #730.

## Summary

- centralize Codex config path resolution in `headroom wrap codex` so it
honors `CODEX_HOME` when set
- make the Codex MCP registrar use `$CODEX_HOME/config.toml` instead of
always writing to `~/.codex/config.toml`
- route optional memory MCP and global Codex `AGENTS.md` injection
through the same Codex home helper
- make `headroom unwrap codex` print a warning, but still succeed, when
`CODEX_HOME` is unset and the default Codex config has no Headroom
markers
- add regression coverage for provider injection, prepare-only wrapping,
MCP registration under a custom Codex home, and the ambiguous unwrap
warning
- update `CHANGELOG.md` under `Unreleased > Bug Fixes`

## Real behavior proof

Setup tested on:

- Linux `7.0.10-2-cachyos`
- Python 3.14.3 via `uv`
- local fork branch `fix/codex-home`
- custom Codex home created outside `~/.codex`

Exact command run after the patch:

```bash
tmp_home=$(mktemp -d)
mkdir -p "$tmp_home/codex_custom"
HOME="$tmp_home" USERPROFILE="$tmp_home" CODEX_HOME="$tmp_home/codex_custom" \
  UV_SKIP_WHEEL_FILENAME_CHECK=1 PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1 \
  uv run --with fastapi --with uvicorn --with httpx --with websockets \
  headroom wrap codex --no-context-tool --no-serena --prepare-only --port 8787
find "$tmp_home" -maxdepth 3 -type f -print | sort
sed -n '1,140p' "$tmp_home/codex_custom/config.toml"
test -e "$tmp_home/.codex/config.toml" && echo yes || echo no
HOME="$tmp_home" USERPROFILE="$tmp_home" \
  UV_SKIP_WHEEL_FILENAME_CHECK=1 PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1 \
  uv run --with fastapi --with uvicorn --with httpx --with websockets \
  headroom unwrap codex --no-stop-proxy
```

After-fix evidence + observed result:

Interactive check:

- A full `CODEX_HOME="$HOME/.codex_p" headroom wrap codex --no-serena`
launch was tested locally after the patch and worked as expected.
- The prepare-only proof below shows the same config path behavior
without requiring an interactive Codex session in CI/reviewer
environments.

```text
MCP retrieve tool: registered (restart OpenAI Codex CLI if it was already running)
Codex config: injected Headroom provider (WS + HTTP) into /tmp/tmp.UPUvloGNYE/codex_custom/config.toml

--- files ---
/tmp/tmp.UPUvloGNYE/codex_custom/config.toml
/tmp/tmp.UPUvloGNYE/codex_custom/config.toml.headroom-backup

--- custom config ---
# --- Headroom proxy (auto-injected by headroom wrap codex) ---
model_provider = "headroom"
openai_base_url = "http://127.0.0.1:8787/v1"
# --- end Headroom ---

# --- Headroom MCP server ---
[mcp_servers.headroom]
command = "headroom"
args = ["mcp", "serve"]
# --- end Headroom MCP server ---

# --- Headroom proxy (auto-injected by headroom wrap codex) ---
[model_providers.headroom]
name = "OpenAI via Headroom proxy"
base_url = "http://127.0.0.1:8787/v1"
supports_websockets = true
# --- end Headroom ---

--- default config exists? ---
no

Warning: found no Headroom wrap markers in the default Codex config. If you wrapped Codex with CODEX_HOME, rerun unwrap with the same environment variable, e.g. CODEX_HOME=/path/to/codex-home headroom unwrap codex.
Nothing to undo: .../.codex/config.toml has no Headroom wrap markers.
```

What I did not test:

- Windows/macOS path behavior
- full repository test suite, because this local Python 3.14 environment
hits optional dependency/build constraints outside this patch

## Testing

```bash
UV_SKIP_WHEEL_FILENAME_CHECK=1 PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1 uv run --with pytest --with fastapi --with uvicorn --with httpx --with websockets pytest tests/test_mcp_registry/test_codex_registrar.py tests/test_cli/test_wrap_codex.py -q
UV_SKIP_WHEEL_FILENAME_CHECK=1 PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1 uv run --with ruff ruff check headroom/cli/wrap.py headroom/mcp_registry/codex.py tests/test_cli/test_wrap_codex.py tests/test_mcp_registry/test_codex_registrar.py
UV_SKIP_WHEEL_FILENAME_CHECK=1 PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1 uv run --with ruff ruff format --check headroom/cli/wrap.py headroom/mcp_registry/codex.py tests/test_cli/test_wrap_codex.py tests/test_mcp_registry/test_codex_registrar.py
```

Results:

```text
63 passed, 1 warning in 1.48s
All checks passed!
4 files already formatted
```

Notes:

- Python 3.14 required `PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1` for the
editable build.
- `uv` required `UV_SKIP_WHEEL_FILENAME_CHECK=1` because the existing
`uv.lock` has a GitPython wheel filename/version mismatch.
2026-06-10 18:21:29 -05:00
Tejas Chopra a7160f7eab fix: register serena mcp during wrap 2026-05-09 23:05:59 -07:00
Tejas Chopra eaf5980b4a fix: stabilize codex compression, stats, and proxy lifecycle 2026-05-09 13:47:53 -07:00
Tejas Chopra ac1d11c9a2 fix: sync Codex MCP proxy config during wrap 2026-05-08 19:14:07 -07:00
chopratejas 7fb6d7fefd fix: remove env_key from injected Codex provider — fixes #393
Codex treats env_key as a hard requirement: if the named env var is
absent it throws "Missing environment variable" before the session
starts. Subscription (ChatGPT Plus) users don't have OPENAI_API_KEY
set, so injecting env_key = "OPENAI_API_KEY" blocks them at startup.

With env_key absent, api_key() returns Ok(None) and Codex falls through
to the existing CodexAuth (OAuth for subscription, ApiKey for PAYG) —
both modes authenticate correctly without startup errors.
2026-05-05 14:54:01 -07:00
chopratejas 5c7a5b4857 fix: inject openai_base_url so Codex subscription (ChatGPT plan) routes through proxy
Codex subscription mode uses the built-in openai provider with
chatgpt.com/backend-api/codex as the default base URL, bypassing
OPENAI_BASE_URL and the custom model_provider setting entirely.

Setting openai_base_url in config.toml overrides that default for all
auth modes (API key and ChatGPT subscription), so both traffic paths
now flow through the Headroom proxy.

Also adds orphan-cleanup regex for openai_base_url in
_strip_codex_headroom_blocks and 5 new tests covering subscription
routing behaviour.
2026-05-05 14:33:41 -07:00
JerrettDavisandClaude Opus 4.7 281bc171dc fix(wrap): unwrap codex restores prior config.toml
`headroom wrap codex` injects a `model_provider = "headroom"` block
plus a `[model_providers.headroom]` table into `~/.codex/config.toml`
so Codex routes both HTTP and WebSocket traffic through the proxy. The
matching `unwrap codex` subcommand did not exist, so the injected
block stayed in `config.toml` forever — the moment the proxy stopped,
Codex (CLI and macOS app) started erroring with
`Missing environment variable: OPENAI_API_KEY`, and users had to hand-
edit the file to recover.

Fix:

* `_inject_codex_provider_config` now snapshots the pre-wrap file to
  `~/.codex/config.toml.headroom-backup` before the first modification
  and leaves that snapshot untouched on subsequent wrap runs. The
  injection is also rewritten to use two self-contained marker-
  delimited blocks (top-level key and provider table) so stripping
  them never consumes user content that sits between them.
* `_inject_memory_mcp_config` takes the same snapshot, so
  `wrap codex --memory` without a full provider injection is still
  fully reversible.
* New `_restore_codex_provider_config` helper and `unwrap codex`
  click command:
  * backup present → restore byte-for-byte and delete the backup;
  * backup absent but Headroom block present → strip the block and
    keep surrounding user content;
  * config contained only Headroom content → remove the file so
    Codex falls back to defaults;
  * nothing to undo → safe no-op.

Codex is the only wrap target that modifies a persistent user config
file: claude/aider/cursor/copilot all go through env vars or project-
scoped files only, so this bug was unique to Codex.

Tests:

* `tests/test_cli/test_wrap_codex.py` adds 20 new cases covering the
  strip/snapshot helpers directly, round-trip idempotency of
  wrap → wrap → unwrap, handling of malformed prior configs, and
  end-to-end CliRunner invocations of `headroom wrap codex
  --prepare-only` / `headroom unwrap codex` against a temp `$HOME`.
* All 153 existing `tests/test_cli/` tests continue to pass.

Plugin manifest versions were re-synced from `pyproject.toml` (0.11.2)
by the `sync-plugin-versions` pre-commit hook; the previous values
(0.10.3) had drifted.

Reported by @raenaryl in Discord on 0.6.3; confirmed still broken on
current `main` (0.11.x).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-23 11:13:54 -05:00