Commit Graph
23 Commits
Author SHA1 Message Date
05431b264f fix(install): make install status agree with doctor when the proxy is reachable (#3072) (#3141)
## Description

`headroom install status` could report `Status: stopped` next to
`Healthy: yes`, while `headroom doctor` reported the same proxy as
running (#3072, bug 2). The commands used independent process views:
`install status` trusted PID/container bookkeeping for `Status`, while
both commands used the live HTTP health endpoints to establish health.

This change treats a successful `/readyz` probe as authoritative
evidence that the deployment is running. If the proxy is not ready,
`runtime_status` still distinguishes a running-but-unhealthy process
from a stopped process.

The PR also adds a focused macOS pytest lane for CLI and install code,
closing the Darwin test-coverage gap described in #3065 without
duplicating the full four-shard Linux suite.

Bug 1 from #3072 (`HEADROOM_PORT` being ignored by `install apply` and
`deploy`) was already fixed on `main` by #3085 and is not reimplemented
here.

Closes #3072
Closes #3065

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Updated `headroom/cli/install.py` so `install_status` performs one
readiness probe and reports `Status: running` whenever `/readyz` is
healthy.
- Kept `runtime_status` as the fallback when the proxy is not ready,
preserving the distinction between running-but-unhealthy and stopped.
- Added regression coverage for the formerly contradictory `Status:
stopped` plus `Healthy: yes` output.
- Added regression coverage for the not-ready case where process status
must still come from `runtime_status`.
- Added a `test-macos` CI job that runs the CLI/install pytest subset on
`macos-latest`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_cli/test_install_cli.py -q
============================= test session starts ==============================
platform darwin -- Python 3.11.15, pytest-9.0.3, pluggy-1.6.0
collected 42 items

tests/test_cli/test_install_cli.py ..................................... [ 88%]
.....                                                                    [100%]

============================== 42 passed in 1.79s ==============================

$ uv run --frozen ruff check headroom/cli/install.py tests/test_cli/test_install_cli.py
All checks passed!

$ uv run --frozen ruff format --check headroom/cli/install.py tests/test_cli/test_install_cli.py
2 files already formatted

$ uv run --frozen mypy headroom/cli/install.py --ignore-missing-imports
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS (Apple Silicon), CPython 3.11.15 via `uv`, pytest
9.0.3.
- Exact command / steps: ran `uv run --frozen --extra dev pytest
tests/test_cli/test_install_cli.py -q`. The new regression test invokes
`headroom install status` through Click's `CliRunner` with `/readyz`
returning healthy while PID/runtime bookkeeping returns `stopped`.
- Observed result: the command exits successfully and prints `Status:
running` with `Healthy: yes`; `Status: stopped` is absent. A second test
verifies that when readiness is false, a live runtime still prints
`Status: running` with `Healthy: no`.
- Not tested: an end-to-end persistent-service installation with a real
OS supervisor or Docker daemon. The new macOS CI job itself will be
exercised by GitHub Actions rather than locally.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; this changes diagnostic display
logic and CI coverage only.
- Minimum rollout channel: stable/default patch release.
- Stable/default behavior changed: yes; `install status` now reports a
ready deployment as running even if PID/container bookkeeping says
stopped. Unready deployments retain the existing runtime-status
behavior.
- Kill switch / disable path: no runtime flag; revert this PR if the
readiness precedence causes an unforeseen regression.
- Unsafe override required: none.
- Qualification impact: `tests/test_cli/test_install_cli.py` and the new
`test-macos` CI job must remain green.
- Rollback path: revert commit `f3ae269b`; no persisted state, schema,
manifest, or configuration migration is involved.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable; this is terminal status output and CI configuration.

## Additional Notes

- Documentation was not changed because no command, option,
configuration contract, or documented workflow changed; this corrects
contradictory diagnostic output.
- #2787 was also reviewed as an urgent author-reported issue, but it
already has active implementation PR #2992. This PR deliberately avoids
duplicating or conflicting with that work.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3141"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

### Maintainer current-main requalification

Exact head `f550bf1d05873b20b773d935aa7cb93025b63262`, upstream main
`2d3ee51846435ef5b320eea4806af14e602b0d8c`.

Re-reviewed latest integration head 3c11d4a96 and current discussion;
clean integration with upstream/main
2d3ee51846. Install status still treats a
ready reachable proxy as running, preserving runtime-status fallback
when unhealthy. All 160 install-status/release/version/MCP tests pass;
no PR-specific production-code blocker found in scope.

Downloaded the latest failing native macOS job 126368233730 from run
41765101507: four persistent-wrap recovery tests still failed because
_Manifest/TaskManifest lacked runtime_kind. Repaired test doubles with
the Python runtime declaration and explicitly mocked the readiness
boundary those tests control through health probes; real ownership
classification remains exercised. Parametrized the four affected
recovery cases across Darwin/Linux/Windows by replacing only runtime
module's sys reference, without global platform emulation. All 83
persistent-wrap tests pass, preserving matching deployment recovery,
bound-socket recovery, healthy reuse, and task warning coverage. No
production recovery logic changed.

Changed-file Ruff check/format and whitespace pass. These local platform
variants are not native macOS validation; exact-head hosted macOS lane
must qualify before approval. Existing native extension supports Python
startup, not a fresh native build. Maintainers handle CI; nothing
merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

---------

Co-authored-by: vishal veerareddy <vishalveera.reddy@servicenow.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-05 21:09:29 -06:00
Mansurshakh Japarov 79f2c2781a fix(install): distinguish protocol targets in status (#4040)
## Description

Closes #2615.

`headroom install status` labelled the default backend as the generic
`Backend`, even when an OpenAI target was configured independently. This
made a healthy OpenAI-only deployment look like an Anthropic-only
deployment.

## Type of Change

- [x] Bug fix
- [x] Documentation update

## Changes Made

- Label the fallback/backend choice as `Default backend`.
- Show `Anthropic target` and `OpenAI target` separately as `configured`
or `default`, based on the existing live loopback health configuration.
- Do not print upstream target URLs or their embedded credentials.
- Preserve the existing non-dict health-config guard and all
lifecycle/routing behavior.
- Document the distinction in the existing persistent-installs page.
- Cover configured/default/unknown targets,
missing/non-dict/partial/malformed health configuration, manifest
fallback, credential withholding, and actual HTTP status probes.

## Testing

- Complete install/CLI owner suites: **296 passed, 5 skipped**, one
existing Starlette/httpx deprecation warning.
- Ruff lint and format checks for both changed Python files: passed.
- `mypy headroom/cli/install.py`: passed, one source file checked.
- Installed commit hooks: passed (including whole-project mirrored mypy
and commit lint).
- Required `make ci-precheck`, run by the actual pre-push hook: passed —
Rust fmt, strict Clippy, full workspace tests, native extension
build/import, **177 Python passed / 4 skipped** (one existing
provider-counting warning), and commitlint.
- First pre-push attempt hit the unchanged native
`first_dispatch_never_blocks_on_model_init` 500 ms bound at 592.8 ms.
The isolated test and then the complete unchanged gate passed. No
threshold, test, native source, or gate was modified or skipped.
- Independent installed-LangChain whole-project typing failure at
`chat_model.py:394` is tracked in #4039; this changed CLI file's scoped
check passes. No claim that the CLI change fixes that separate error.
- Hosted Codecov's automated advisory reports 0% patch coverage for the
three display lines. The real CLI before/after runs above exercise them;
no wording-only mock test was added to manufacture coverage. At the
observed SHA, there is no failing Codecov check; remaining hosted jobs
were still in progress.

## Real Behavior Proof

Linux x86_64, Python 3.13.14; base
`3c418f6f0b190ad1060413c0eadd77f20ab3cf46`.

Created a real manifest with the production planner and atomic
persistence API, then ran the actual `headroom install agent run` and
`headroom install status` CLI. HOME, canonical workspace/config roots,
profiles, logs, and ports were isolated under the project verification
directory. No system supervisor or provider configuration was installed
or modified. Compression/caching were disabled through supported raw
proxy flags because this is status-display proof, not compression proof;
the anonymous beacon was disabled.

Before, live `/health` reports `backend=anthropic`, a configured OpenAI
URL, and no configured Anthropic URL; status is running/healthy but ends
with `Backend: anthropic` only.

After, the same configuration reports:

```text
Default backend:  anthropic
Anthropic target: default
OpenAI target:    configured
```

A second actual run with both protocol URLs configured reports both
targets as configured. The intentionally unreachable Anthropic test
endpoint reports `Healthy: no` there; configured is not misrepresented
as reachable/healthy. A fake secret embedded in each verification URL
does not appear in status output. Agents were explicitly terminated
after each observation; no graceful-exit claim.

Not tested: paid upstream requests, a running compression workload,
macOS, Windows, or a real platform supervisor installation.

## Runtime Rollout Safety

No channel, feature flag, routing, auth, or persisted-data format
changes. Only operator display changes. Revert this commit to restore
the old display; deployment profiles remain valid.

## Review Readiness

- [x] Changed consumers and existing health-config contract reviewed
- [x] Actual before/after CLI surface verified

## Checklist

- [x] Existing documentation updated
- [x] No warnings introduced in the changed Python files
- [x] No CHANGELOG edit (release-please owns it)

AI-assisted: omp / GPT-6.1 Sol, without subagents. No hosted-CI claim.


## Follow-up qualification at 2969351e3d

When the health response withholds configuration or omits a target
field, status now reports `unknown` instead of asserting the default.
Malformed target values also remain unknown. Explicit null/empty URL
fields report default; configured string URLs report configured without
exposing their contents.

Reproduce the committed HTTP regressions from a configured repository
Python environment:

```sh
python -m pytest tests/test_cli/test_install_cli.py -k status -q
```

For the actual proxy/CLI replay, save the complete script in [the
maintainer evidence
comment](https://github.com/headroomlabs-ai/headroom/pull/4040#issuecomment-6062359335)
as `verify_status.py` in the repository root, then run `python
verify_status.py`. It creates temporary production manifests and an
actual uvicorn proxy, verifies direct health configuration and
configuration withheld by the real health guard through a relay, and
checks that status never prints URLs/credentials. The same script fails
on the original 39e7d46fa head and passes on this head. This is not a
real Docker networking or OS-supervisor installation claim.

Independent Windows/Python 3.12 verification: 319 install/CLI tests
pass, 2 platform skips, one existing warning; Ruff lint/format and
Linux-platform mypy over 566 files pass. The complete suite also
reproduced one unrelated Windows planner test mismatch: it expects
persistent-service defaults although unchanged Windows planning
intentionally converts that preset to persistent-task (#2552). That test
was deselected in the follow-up suite; no planner source/test changed.
Exact setup, limitations, before/after output assertions, and script are
in the linked evidence.
2026-10-08 10:18:32 -05:00
Mansurshakh JaparovandJD Davis 40955b0eef fix(install): preserve OpenCode edits during reactivation (#4022)
## Description

Reapply, restart, and automatic recovery temporarily remove the Headroom
provider before reactivating it. They must not restore the original
backup and erase user configuration added after installation. Final
removal must still restore the original snapshot.

Closes #4016.

## Type of Change

- [x] Bug fix
- [x] Documentation update

## Changes Made

- Separate transient provider deactivation from final backup
restoration, and propagate the intent through the existing install
registry/manifest lifecycle.
- During reactivation and activation rollback, remove only Headroom's
provider; retain unrelated providers, MCP settings, theme, and the
original backup.
- On true removal, restore and delete the original backup as before.
- Associate backups with the actual selected `opencode.json` or
`opencode.jsonc` filename, and pin edited writes to LF.
- Persist snapshot ownership independently of active mutations,
including activation persistence failures and stop/reload/final-remove.
- Migrate historical JSONC backup names (including custom config names)
before snapshotting/restoration in installer and wrapper callers; do not
claim a sibling JSON installation's backup.
- Accept JSONC comments/trailing commas without erasing user settings;
retain quoted URLs and delimiters.
- Restore the snapshot when a successful replacement drops OpenCode, but
retain it on replacement failure.
- Document preservation during reapply/restart/recovery and
original-snapshot behavior on final removal.

## Testing

- [x] Unit tests pass (`pytest`), including JSON/JSONC and all three
reactivation paths.
- [x] Changed-file lint, configured pre-commit mypy and manual
real-filesystem checks pass.

### Test Output

Real filesystem smoke output for both config suffixes:

```json
[{"final_removal_deletes_backup": true, "final_removal_restores_original": true, "headroom_reactivated": true, "original_backup_unchanged": true, "suffix": ".json", "theme": "edited-after-install", "user_mcp_preserved": true, "user_provider_preserved": true}, {"final_removal_deletes_backup": true, "final_removal_restores_original": true, "headroom_reactivated": true, "original_backup_unchanged": true, "suffix": ".jsonc", "theme": "edited-after-install", "user_mcp_preserved": true, "user_provider_preserved": true}]
```

The mandatory pre-push `make ci-precheck` passed: workspace Rust
fmt/clippy/tests, native extension build/import verification, the
complete configured Python gate, and Commitlint. The native gate
explicitly used the installed CPU `libonnxruntime.so.1.26.0` via
`ORT_DYLIB_PATH`, with cached-model assertions enabled.

```text
[build_rust_extension] headroom._core build + install + verify: OK
================== 177 passed, 4 skipped, 1 warning in 6.31s ===================
✅ ci-precheck PASSED — safe to push.
```

```sh
python -m pytest -q tests/test_providers_opencode_install.py tests/test_cli/test_install_cli.py tests/test_install/test_providers.py
```

Original lifecycle suite: **121 passed**. After the review corrections,
the combined provider/config/wrapper/install-CLI/recovery suite passed
**240 tests in 8.32s**:

```sh
python -m pytest -q tests/test_providers_opencode_install.py tests/test_providers_opencode_config.py tests/test_cli/test_wrap_opencode.py tests/test_cli/test_install_cli.py tests/test_install/test_providers.py tests/test_install_tool_env_reconcile.py
```

The regression suite checks real original bytes and persisted recovery
ownership, not mocked call order or incidental error wording. The
obsolete JSONC-trailing-comma rejection expectation was removed.

Changed-file Ruff, formatting, installed pre-commit `mypy headroom`, and
Commitlint hooks passed. Full-tree Ruff remains blocked locally by
executable permissions on five unchanged OAuth2 plugin files (tracked
100644). A separate full dev-environment mypy run found an unchanged
optional LangChain `truthy-function` error; the configured isolated
pre-commit hook passed. Unrelated code/permissions were not modified.

## Real Behavior Proof

- Environment: Linux x86_64, CPython 3.13.14; unchanged
`dea9092e19385f286b7f67a64f4d85de2708d756` and this branch, real
temporary config files and real installer mutation helpers, no mocked
file mutation or service installed on the workstation.
- Exact command / steps: for each suffix `.json` and `.jsonc`, write an
original theme/provider/MCP config; activate a provider-scoped OpenCode
manifest with `_activate_deployment_mutations`; edit the theme and add a
user provider plus remote MCP; call `_deactivate_deployment_mutations`
then `_activate_deployment_mutations`; inspect real files and the
original backup; finally invoke `revert_mutations(...,
restore_backup=True)`.
- Before: theme reset to `original`; added provider and MCP disappeared.
JSONC's actual-name backup was absent.
- Observed result: after the fix, theme remains `edited-after-install`;
both user additions survive; Headroom is active again; both actual-name
backups remain byte-identical to the original. Final removal restores
the original bytes and deletes the backup for both suffixes.
- Review correction proof against published head `4bc8e50c`: actual
temporary files and persisted manifests cover stop/reload/final-remove
for JSON and JSONC, legacy backup migration for default and custom
filenames, JSONC trailing commas and quoted `//`/`,}` text, successful
target removal, and activation-persistence failure followed by
reload/removal. All eight scenario outcomes pass after the correction;
the published baseline leaves or consumes the wrong snapshots and loses
JSONC edits. The actual `headroom unwrap opencode --no-stop-proxy`
command also restores the historical JSONC snapshot byte-for-byte and
consumes its backup. Only external process/service management and the
explicit persistence-fault boundary are substituted; config, manifest
and snapshot IO are real.
- Second review correction against `1c0855c4`: all four new findings
reproduced on real temporary config/backup files and persisted
manifests. After the fix, failed addition consumes its newly created
snapshot (a later install restores the later user edit); changed config
selection restores the recorded original path; an existing canonical
JSONC snapshot wins without deleting the ambiguous legacy backup; and
restoration failure retains ownership in the active replacement so a
retry after reload finishes restoration. Seven permanent boundary cases
initially gave **6 failures and 1 passing control**; the complete
affected suite now gives **247 passed**. OS supervisor calls and
explicit persistence/restoration faults are the only substituted
boundaries.
- Not tested: a real launchd/systemd supervisor or Windows process on
this Linux host. CLI regression tests exercise apply/restart/recovery
boundaries while replacing service management; the filesystem mutation
smoke uses real production helpers.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; lifecycle data-preservation fix.
- Minimum rollout channel: stable; no rollout-managed feature is
enabled.
- Stable/default behavior changed: transient reactivation stops
reverting unrelated user edits. Final uninstall still restores the
original snapshot.
- Kill switch / disable path: remove the persistent OpenCode
installation using its existing removal flow; no new flag.
- Unsafe override required: no.
- Qualification impact: provider-scoped OpenCode configuration only;
other providers retain their existing removal semantics.
- Rollback path: revert the commits; no deployment manifest schema
change. Backup ownership uses the existing artifact record, and legacy
JSONC snapshots migrate to the selected config's canonical filename.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] Existing registry/lifecycle interfaces reused; persistent-install
docs updated.
- [x] Consumer-visible regressions cover edits, original-backup
invariants, both config suffixes, three reactivation paths, and final
removal.
- [x] `CHANGELOG.md` not edited; Conventional Commit title supplied.

Remote CI results are reported by GitHub, not assumed from local tests.

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-07 17:46:39 -05:00
Rod BoevandJD Davis 44db755ee6 fix(install): replace the launchd wrapper with the proxy listener (#3224)
## Description

The persistent service path can leave a wrapper, listener, native
supervisor, or Docker container with conflicting ownership. Rebuild
runtime ownership so launchd directly owns Python, Docker owns Docker
across service/task presets, status and stop require deployment
identity, readiness cannot be health-only, and failed cleanup preserves
rollback state. Closes #3208

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (feature that would cause existing behavior to
change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Make the launchd-owned Python service exec the listener in place.
- Classify Docker ownership from both runtime and preset, with no
competing native supervisor.
- Route Docker service/task start, restart, recovery, and teardown
through the Docker owner.
- Verify exact process command, port, deployment profile, and runtime
before status or stop.
- Add POSIX, macOS, and Windows process identity fallbacks without
requiring psutil.
- Require identity on readiness and fail closed when cleanup cannot
prove the new owner stopped.
- Persist an inactive previous-deployment snapshot, keep active state
durable before owner start, and make environment/provider mutation
activation transactional.
- Surface cleanup, persistence, supervisor, Docker, and
recovery-snapshot deletion failures with actionable recovery guidance.

## Testing

- [x] Unit tests pass (`uv run --locked pytest
tests/test_provider_package_init.py::test_install_registry_apply_provider_scope_mutations_skips_missing_and_none
tests/test_install/test_providers.py tests/test_cli/test_install_cli.py
tests/test_install/test_runtime.py
tests/test_install/test_supervisors.py tests/test_install/test_paths.py
tests/test_install/test_state.py -q`)
- [x] Linting passes (`uv run --locked ruff check` on changed install,
provider, supervisor, and focused test files)
- [x] Type checking passes (`uv run --locked mypy headroom
--ignore-missing-imports`)
- [x] New tests added for the regression
- [ ] Manual testing performed through live process/supervisor
environments

### Test Output

```text
uv run --locked pytest tests/test_provider_package_init.py::test_install_registry_apply_provider_scope_mutations_skips_missing_and_none tests/test_install/test_providers.py tests/test_cli/test_install_cli.py tests/test_install/test_runtime.py tests/test_install/test_supervisors.py tests/test_install/test_paths.py tests/test_install/test_state.py -q
============================= 197 passed in 1.07s =============================
uv run --locked ruff check .
All checks passed!
uv run --locked ruff format --check .
1534 files already formatted
uv run --locked mypy headroom --ignore-missing-imports
Success: no issues found in 527 source files
git diff --check
passed; the only worktree change is the pre-existing unstaged uv.lock line-ending change
```

## Real Behavior Proof

- Environment: Windows focused process and supervisor tests, with
platform-specific launchd and Docker owner paths represented by fakes.
- Exact command / steps: run the focused lifecycle and state suite
listed above, targeted Ruff and format checks, and the full-head mypy
check.
- Observed result: each persistent mode has one owner; mismatched,
malformed, or unavailable identity cannot authorize cleanup; active
state is durable before owner start; failed cleanup retains recovery
state; mutation rollback preserves unreverted records.
- Not tested: owner-level POSIX exec, a live macOS launchd job, and a
Docker daemon

## Runtime Rollout Safety

- Rollout-managed feature(s): None. This changes the default persistent
install and recovery ownership paths.
- Minimum rollout channel: Stable, after platform-specific CI and
maintainer review.
- Stable/default behavior changed: Yes. Persistent Python service
ownership and Docker supervisor routing change.
- Kill switch / disable path: `headroom uninstall` removes the managed
deployment; users can reinstall with another supported preset or
runtime.
- Unsafe override required: No.
- Qualification impact: launchd, Docker, and Windows identity paths
require their platform-owned environments; local proof uses focused
tests and process/supervisor fakes.
- Rollback path: Revert the rollback-ordering commits in this PR, then
reinstall the prior deployment if needed. Uncertain cleanup retains the
active new manifest and inactive previous snapshot until the new owner
is removed and recovery is completed.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

The changelog is generated by the release pipeline. Live launchd and
Docker-owner behavior remains platform-owned proof; local tests do not
claim those owner environments. Final implementation head: `4004e4a5`.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3224"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-05 08:34:42 -05:00
2e2758de95 Fix macOS install-native CI failure from stale restart manifest test stub (#3779)
The `install-native (macos-latest)` Actions job was failing in the
restart smoke test because the test-local manifest stub no longer
matched the fields required by the current install lifecycle code. This
updates the stub so the workflow exercises the real restart path instead
of failing on fixture drift.

- **Root cause**
- `test_install_restart_waits_for_old_process_before_starting` used a
minimal `Manifest` stub that was missing fields now consumed during
managed-env reconciliation after restart.

- **Change**
- Extend `_restart_race_manifest()` in
`tests/test_cli/test_install_cli.py` to include the current deployment
fields expected by restart:
    - `port`
    - `backend`
    - `targets`
    - `tool_envs`

- **Why this scope**
  - Keeps the fix test-only.
- Preserves production behavior; the failure was caused by an outdated
fixture, not runtime logic.

```python
class Manifest:
    profile = "default"
    preset = "persistent-service"
    runtime_kind = "python"
    supervisor_kind = "service"
    scope = "user"
    port = 8787
    health_url = "http://127.0.0.1:8787/readyz"
    backend = "anthropic"
    targets: list[str] = []
    tool_envs: dict[str, dict[str, str]] = {}
    mutations: list = []
```

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: JerrettDavis <2610199+JerrettDavis@users.noreply.github.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-09-25 02:59:25 +02:00
Radhakrishnan Pachyappan 11c7320bbc fix(install): wait for the old process to stop before restarting (#3670)
## Description

`headroom install restart` reports success but leaves the deployment
stopped. Stopping the supervisor returns before the old proxy finishes
its graceful shutdown, so `/readyz` still answers. `_start_deployment`
treats a ready endpoint as "already running" and returns without
starting anything. The old process then exits and nothing replaces it.
`install start` works afterwards because by then nothing is answering.

Restart now waits (bounded, 15s) for the health endpoint to go down
before starting, and fails instead of reporting success if it never
does.

Closes #3658

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Add `_wait_until_stopped` in `headroom/cli/install.py` and call it
between `_stop_deployment` and `_start_deployment` in `install restart`.
- Add two regression tests: old process lingers after stop (restart must
wait, then start), and old process never stops (restart must fail
without starting or reporting success).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
python -m pytest tests/test_cli/test_install_cli.py -q
42 passed

python -m ruff check headroom/cli/install.py tests/test_cli/test_install_cli.py -> All checks passed
python -m ruff format --check (same files) -> already formatted

Without the source change, test_install_restart_waits_for_old_process_before_starting fails (1 failed).
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.14.3. The issue was reported on
macOS (launchd), which I could not run.
- Exact command / steps: the unit tests above simulate the old process
still answering `/readyz` after stop.
- Observed result: restart waits for the endpoint to go down, then
starts; it errors if the endpoint never goes down.
- Not tested: a real launchd / systemd restart; mypy; the wider test
suite beyond `test_install_cli.py`; the docker and task presets. The
root cause is inferred from the code path and the issue's symptoms, not
reproduced on macOS.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: `install restart` now waits up to 15s
for the old process to stop, and errors if it does not.
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Docs: not applicable (no user-facing docs describe restart timing). Full
unit suite not run locally; only `test_install_cli.py`.
2026-09-24 11:29:03 -05:00
Tejas ChopraandClaude Opus 5 a4cb2bc021 fix(install): apply managed env vars added after a deployment was installed (#3740)
## The bug

`tool_envs` is computed **once**, by `build_manifest` during `headroom
install`, and then stored in the deployment manifest. Every lifecycle
command afterwards re-applies the **stored** map:

```
install start → _require_manifest(profile) → _activate_deployment_mutations(manifest)
             → apply_mutations(manifest) → manifest.tool_envs
```

`build_tool_envs()` is never called again. So any variable added to a
provider's install env later reaches **new installs only** — never an
existing deployment, not on `start`, not on `restart`, and **not on a
Headroom upgrade**. The only cure was reinstalling, which nobody does
for a proxy that is working.

## Why it matters

`ENABLE_TOOL_SEARCH` joined Claude's install env on **2026-06-19**
(#995, GH #746) because pointing Claude Code at a custom
`ANTHROPIC_BASE_URL` makes it **stop deferring MCP tool schemas and
inline every one of them**.

A deployment installed before that date has kept getting the base URL
written *without* the mitigation ever since — the client is pointed at
the proxy **and** has its own tool-schema deferral switched off. That's
the expensive half of the change with none of the benefit, indefinitely.

Measured live against the API on a 15-builtin + 40-MCP-tool surface:

| | input_tokens |
|---|---|
| tool schemas inlined (deferral off) | **16,246** |
| deferral active | **718** |

Paid on every turn, silently, because the manifest is authoritative and
was written before the fix existed.

## The fix

`_reconcile_tool_envs()` rebuilds the expected env from the manifest's
own `port`, `backend` and `targets`, and adds any keys the stored map is
missing. It runs inside `_activate_deployment_mutations()` — the single
choke point every path already goes through (`start`, `restart`,
`_restore_deployment`) — so managed env becomes **self-healing across
upgrades** instead of frozen at install time. Each addition is reported
to the user.

**Missing keys are added; existing values are never overwritten.** A
stored value may legitimately differ from a fresh build — a hand-edited
port, a deployment pinned deliberately — and healing an omission is a
much smaller claim than re-deciding a setting the manifest already
records.

Rebuilding from `manifest.port` also means a deployment on a non-default
port cannot be re-pointed at 8787 by reconciliation. There's a test for
that specifically.

## Testing

7 new tests in `tests/test_install_tool_env_reconcile.py`, including the
pre-#746 manifest case, idempotency, and the non-default-port guard. 199
pass across the install suites.

## Note

This is the class of bug where the code was correct for three months and
every existing deployment kept the old behaviour anyway. Worth assuming
any future addition to a provider's install env had the same fate until
now.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---

### Update after self-review

The reconcile **never ran on the command this PR is about**. `install
start` had:

```python
if probe_ready(manifest.health_url) and not manifest.mutations:
```

Every normally-installed working deployment *has* mutations recorded, so
activation was skipped for exactly the population whose stored env is
stale — the commit message's "not on `start`" was still true after the
patch claiming to fix it.

Added `pending_tool_envs()`, which answers "is anything missing?"
without mutating, so start re-applies when — and only when — there is
something to apply. The new test **fails against the old condition and
passes against the new one** (checked, not assumed). A deployment
already current is still left completely alone.

Also fixed: a `tool_envs` entry that is present but not a dict
(hand-edited manifest; `load_manifest` does not type-check it) raised
`AttributeError` out of a lifecycle command; and a target with no
managed env no longer gets an empty dict written, which rewrote the
manifest and bumped `updated_at` for nothing.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-23 22:38:58 +02:00
Abhay Singh 58f28dc7a6 fix(install): honor HEADROOM_PORT in install apply and deploy (#3085)
## Description

`headroom install apply --preset persistent-service` and `headroom
deploy` ignored an explicit `HEADROOM_PORT` and always configured port
8787, even though `headroom proxy --port` honors `HEADROOM_PORT`. Anyone
running a second instance, or avoiding a port conflict, got a silently
wrong configuration, and the failure is especially confusing because the
override *appears* supported on the direct proxy path.

Root cause: the `--port` options on the `install apply` and `deploy`
commands were declared with a hardcoded `default=8787` and **no**
`envvar` binding:

```python
@click.option("--port", "-p", default=8787, type=int, show_default=True, help="Persistent proxy port.")
```

The proxy command's `--port` already carries `envvar="HEADROOM_PORT"`,
so the two paths disagreed. `build_manifest` /
`_build_deployment_manifest` already thread the `port` argument all the
way through to the generated `HEADROOM_PORT` base-env and the health
URL, so the value was simply never resolved from the environment at the
CLI boundary.

## Fix

Bind both `--port` options to `envvar="HEADROOM_PORT"`, matching the
proxy command. Click resolves the value from the environment when
`--port` is not passed, and an explicit `--port` still wins over the env
var (standard Click precedence: explicit CLI argument over `envvar` over
`default`).

## Scope

This addresses **bug 1** of #3072. Bug 2 (`install status` reporting
`Status: stopped` alongside `Healthy: yes`, disagreeing with `doctor`)
is an unrelated status-reporting concern that the reporter offered a
live repro for; it is left for a separate follow-up rather than bundled
here.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/cli/install.py`: add `envvar="HEADROOM_PORT"` to the
`--port` option on both `install apply` and `deploy` (and note the env
var in each help string), matching `headroom proxy --port`.
- `tests/test_cli/test_install_cli.py`: added
`test_install_apply_honors_headroom_port_env`,
`test_install_apply_explicit_port_overrides_env`, and
`test_deploy_honors_headroom_port_env`, capturing the `port` that
reaches the manifest builder.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy`)
- [x] New tests added

### Test Output

```text
tests/test_cli/test_install_cli.py  40 passed
# uvx ruff@0.15.22 check  -> All checks passed!
# uvx mypy@1.20.2 headroom/cli/install.py -> Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.15.22 and mypy 1.20.2 via uvx.
- Exact command / steps: reverted the source fix and ran the two new
env-var tests to capture the bug (`python -m pytest
tests/test_cli/test_install_cli.py::test_install_apply_honors_headroom_port_env
tests/test_cli/test_install_cli.py::test_deploy_honors_headroom_port_env`
-> both failed with `assert 8787 == 8788`, proving `HEADROOM_PORT=8788`
was dropped); restored the fix; re-ran the full file (`python -m pytest
tests/test_cli/test_install_cli.py` -> 40 passed); then `uvx
ruff@0.15.22 format`, `uvx ruff@0.15.22 check`, and `uvx mypy@1.20.2
headroom/cli/install.py`.
- Observed result: with the fix, `HEADROOM_PORT=8788 headroom install
apply` (and `deploy`) resolves `port=8788` into `build_manifest`, so the
generated service config and `HEADROOM_PORT` base-env use 8788; passing
`--port 9999` alongside the env var still yields 9999.
- Not tested: an end-to-end persistent-service install on a machine with
a running supervisor (the CLI-to-manifest port resolution is verified
through the manifest builder, which already owns the downstream wiring
covered by the existing planner tests).

## Runtime Rollout Safety

- Rollout-managed feature(s): none. This is a CLI option-binding fix on
the install/deploy commands, not a rollout-channel-gated runtime
feature.
- Minimum rollout channel: N/A (no rollout-managed behavior).
- Stable/default behavior changed: only when `HEADROOM_PORT` is set in
the environment. Previously it was ignored (config wired to 8787); now
the install/deploy path honors it, matching `headroom proxy`. With no
`HEADROOM_PORT` set and no `--port`, the default is still 8787, so
existing installs are unaffected.
- Kill switch / disable path: unset `HEADROOM_PORT` (or pass `--port
8787`) to keep the prior port.
- Unsafe override required: no.
- Qualification impact: `install apply` / `deploy` now provision the
proxy on the operator's requested port instead of always 8787, so a
second instance or a port-conflict workaround configures correctly.
- Rollback path: revert this PR; the `--port` options return to ignoring
`HEADROOM_PORT`.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`: it is generated by
release-please from my Conventional Commit PR title

## Additional Notes

Reported by @vsg-prog (split out of #3040 into #3072). The `--port`
option already carried the correct `type`/range validation and threaded
through the manifest builder; the only gap was the missing `envvar`
binding at the CLI boundary.
2026-08-17 20:20:38 -07:00
ddd2a259ec fix(install): consolidate Windows fallback and cleanup safety (#2980)
## Description

Consolidates two fully reviewed installation-safety fixes whose original
PRs can no longer merge under current branch protection: Windows
persistent-service deployments need a supported Task Scheduler fallback,
and legacy context-tool cleanup must never delete user-owned
RTK/lean-ctx artifacts.

Closes #2552
Closes #2817

Supersedes #2600 and #2828 while preserving their authors' commits and
review-driven corrections.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Convert Windows `persistent-service` plans to the supported
`persistent-task` supervisor and make the fallback explicit in CLI
output.
- Restrict context-tool cleanup to artifacts proven to live under
Headroom's managed directory.
- Recognize wrapped, relative, and platform-specific managed commands
without accepting prefixed/path-boundary lookalikes.
- Scope cleanup completion state correctly across projects and alternate
agent homes.
- Stamp cleanup complete only after all managed remnants are settled.
- Preserve the original focused regression suites and behavior-proof
artifact.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run pytest -q tests/test_install/test_planner.py tests/test_install/test_supervisors.py tests/test_cli/test_install_cli.py tests/test_context_tool_cleanup.py tests/test_cli/test_unwrap_claude.py
135 passed in 0.45s

$ uv run ruff check <changed Python and test files>
All checks passed!

$ uv run ruff format --check <changed Python and test files>
8 files already formatted
```

## Real Behavior Proof

- Environment: macOS arm64 for consolidated current-main validation; the
Windows fallback source PR was independently validated on Windows and
includes its captured verification artifact.
- Exact command / steps: run the planner, supervisor, install CLI,
cleanup provenance, and unwrap suites on the rebased combined branch.
- Observed result: 135/135 focused tests pass. Windows service requests
resolve to `persistent-task`; cleanup rejects user-owned and path-prefix
lookalikes while removing managed artifacts.
- Not tested: a fresh privileged Windows host deployment in this local
pass; #2600's accepted review contains the Windows-specific proof.

## Runtime Rollout Safety

- Rollout-managed feature(s): Install supervisor selection and one-time
legacy cleanup.
- Minimum rollout channel: Stable/default; both prevent currently
destructive or nonfunctional install paths.
- Stable/default behavior changed: Windows service requests use Task
Scheduler; cleanup requires managed provenance.
- Kill switch / disable path: Select `persistent-task` explicitly;
cleanup remains bounded by its completion stamp and provenance checks.
- Unsafe override required: No.
- Qualification impact: Windows native install and wrap/unwrap cleanup
suites.
- Rollback path: Revert this PR, restoring the two pre-fix behaviors.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

The Windows verification artifact from #2600 is retained at
`.github/pr-images/issue-2552-windows-fallback-verification.png`.

## Additional Notes

This is intentionally an installation-safety batch rather than two
replacement PRs. Original commit authorship is preserved, and the
combined diff was applied cleanly to current `main` after #2832 and
#1628 landed.

---------

Co-authored-by: Inference1 <68734681+Inference1@users.noreply.github.com>
Co-authored-by: Dennis Alexis Valin Dittrich <dd+github@dr-dittrich.de>
2026-08-13 15:05:45 -05:00
Abhay Singh 82526191a1 fix(cli/install): resolve the deployment profile instead of dead-ending on default (#2832)
## Description

`headroom init` installs its persistent deployment under a non-`default`
profile name (`init-user` for a global-scope install), but every
`headroom install <lifecycle>` subcommand hardcodes `--profile default`.
The docs show those commands without `--profile`, so on a machine set up
by `headroom init` every documented lifecycle command fails while the
real deployment is running fine:

```console
$ headroom install status
Error: No deployment profile named 'default' is installed.

$ headroom install status --profile init-user
Status:  running
Healthy: yes
```

The error named neither the installed profile nor the `--profile` flag,
so there was nothing to lead the user to `init-user`, which exists only
as an internal constant.

When the requested profile is not installed, `_require_manifest` now
resolves the real target instead of dead-ending on a name the user never
chose:

1. an explicit `HEADROOM_DEPLOYMENT_PROFILE` (which the runtime already
exports) wins;
2. otherwise, when `--profile` was left at its `default` default and
exactly one deployment is installed, that one is used;
3. when it still cannot decide, the error lists the installed profiles
and points at `--profile`.

This changes only the not-found path. An installed `default` still loads
exactly as before, and an explicit typo'd `--profile` still fails, now
with a helpful list.

Fixes #2811

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/cli/install.py` (`_require_manifest`): on a manifest miss,
resolve via `HEADROOM_DEPLOYMENT_PROFILE`, then a single installed
deployment when the request is the bare `default`, and otherwise raise
an error that lists installed profiles and points at `--profile`.
Imported `list_manifests` (already present in `headroom.install.state`)
for the enumeration.
- `tests/test_cli/test_install_cli.py`: added
`test_require_manifest_resolves_single_profile_when_default_missing`,
`test_require_manifest_honors_env_profile`, and
`test_require_manifest_lists_installed_profiles_when_ambiguous`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
# Before/after on the exact reported scenario (one installed profile "init-user"):
# ORIGINAL: _require_manifest("default")  -> raises "No deployment profile named 'default' is installed."
# FIXED:    _require_manifest("default")  -> resolves to "init-user"

# Pass-after, install suites:
tests/test_cli/test_install_cli.py         33 passed
tests/test_install/                        174 passed, 1 skipped, 1 pre-existing failure
# the 1 failure is tests/test_install/test_native_installers.py::
#   test_powershell_native_installer_supports_persistent_docker_lifecycle, which
#   runs scripts/install.ps1 and fails identically on clean main with these changes
#   stashed (an environment-specific PowerShell exit, unrelated to this diff).

# uvx ruff@0.15.17 check  -> All checks passed!
# uvx mypy@1.20.2 headroom/cli/install.py -> Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.15.17 and mypy 1.20.2 via uvx.
- Exact command / steps: read `detect`/`_require_manifest` and the
lifecycle command options (`--profile default` at
install.py:720/748/763/775/789/818/830) to confirm the mismatch with
`init.py`'s `_GLOBAL_PROFILE = "init-user"`, then demonstrated
before/after by monkeypatching `load_manifest`/`list_manifests`: on the
original code `_require_manifest("default")` raises "No deployment
profile named 'default' is installed."; with the fix it returns the
single installed manifest (`init-user`). Fail-before via `git stash push
headroom/cli/install.py` and a direct call; pass-after with `git stash
pop` and the install suites (33 passed in the CLI file, 174 passed in
test_install with one pre-existing environment failure).
- Observed result: a bare lifecycle command on an init'd machine now
targets the running deployment instead of failing, matching the
`--profile init-user` command the issue reporter confirmed works. An
explicit `HEADROOM_DEPLOYMENT_PROFILE` selects the target, and an
ambiguous multi-profile machine gets an error naming the installed
profiles and the `--profile` flag.
- Not tested: a full end-to-end `headroom init` then `headroom install
status` on a fresh host (that flow spawns a real deployment and
supervisor). The resolution logic is a pure function verified directly,
and the manifest loading it calls is existing, tested code.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`: it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

The resolution deliberately only triggers on the not-found path and only
auto-selects when a single deployment is installed or an explicit env
profile names one, so it never silently picks the wrong deployment on a
multi-profile host. The docs that show the bare commands
(`docs/content/docs/persistent-installs.mdx`,
`wiki/persistent-installs.md`, `wiki/cli.md`) become correct again
without needing a `--profile` on every line.
2026-08-13 11:51:34 -05:00
Abhay Singh 170b04a74d fix(install): carry upstream-routing env overrides into supervised deployments (#2429)
## Description

Fixes #2240.

`headroom install apply` builds the persistent deployment's environment
from the `HEADROOM_*` family plus any explicit `--env KEY=VALUE`. It
never captured the provider upstream-routing overrides that the
interactive `headroom proxy` reads from the environment through
`resolve_api_overrides` (`ANTHROPIC_TARGET_API_URL` and its
`*_TARGET_API_URL` siblings).

A supervised runner (launchd, systemd, cron, Windows service/task)
starts from a bare environment, so those exports never reach the
persistent proxy. The result: a user who exports
`ANTHROPIC_TARGET_API_URL` pointing at their gateway and runs `install
apply` gets a proxy that silently forwards to the default Anthropic
endpoint instead. That is both a correctness bug and a routing surprise
(traffic and keys can go to the wrong host).

## Fix

Capture the documented `*_TARGET_API_URL` overrides from the current
environment and merge them into the manifest env underneath the explicit
`--env` map, so an explicit `--env` still wins.

Scope notes:

- Only URL overrides are auto-captured. The `*_TARGET_API_HEADERS`
variables can carry bearer tokens, so those are deliberately left to an
explicit `--env` rather than being persisted into the on-disk manifest
implicitly.
- The proxy already resolves these vars correctly at runtime; this only
makes `install apply` hand them to the supervised process the same way
the interactive proxy would inherit them.
- `headroom deploy` (the Docker path) is left unchanged here; this
targets the exact reported `install apply` flow.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/cli/install.py`: add `_PASSTHROUGH_URL_ENV_VARS` and
`_capture_passthrough_env`, and merge the captured overrides under the
parsed `--env` map in `install_apply` before building the manifest.
- `tests/test_cli/test_install_cli.py`: unit test for the capture helper
(skips empty/unrelated vars), plus CliRunner tests that a set
`ANTHROPIC_TARGET_API_URL` reaches `build_manifest`'s env and that an
explicit `--env` overrides the captured value.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_cli/test_install_cli.py -k "capture or captures or overrides" -q
3 passed
$ uvx ruff@0.15.17 check headroom/cli/install.py tests/test_cli/test_install_cli.py
All checks passed!
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/cli/install.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, project venv (`uv sync --extra
proxy`), `uvx ruff@0.15.17` / `uvx mypy@1.20.2`, pytest in the venv.
- Exact command / steps: ran the new CliRunner tests, which export
`ANTHROPIC_TARGET_API_URL` via monkeypatch, invoke `install apply` with
the supervisor side effects stubbed, and capture the kwargs handed to
`build_manifest`. Also called the real `_capture_passthrough_env` and
real `build_manifest` directly to confirm the value lands in
`manifest.base_env`.
- Observed result: with the var exported, `build_manifest` received it
in `extra_env` and `manifest.base_env["ANTHROPIC_TARGET_API_URL"]` held
the gateway URL; with an explicit `--env ANTHROPIC_TARGET_API_URL=...`
the explicit value won; empty and unrelated vars were skipped. Ran
against the actual modules.
- Not tested: a live launchd/systemd run forwarding to a real gateway.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
2026-07-19 22:18:15 -07:00
JD Davis 560ffae103 feat(deploy): Add turnkey deploy command (#1404)
## Description

Adds `headroom deploy` as the turnkey, zero-config local deployment
entrypoint. The command chooses the most capable deployment path it can
verify on the current host, configures detected tools through the
existing persistent-install machinery, starts the proxy, and preserves
the existing rollback behavior if an update fails.

The selection order favors performance first: NVIDIA Docker GPU
passthrough when `nvidia-smi` and Docker's NVIDIA runtime are available,
then plain Docker, then native scheduled recovery, then a detached
Python runtime fallback.

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [x] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- Added the top-level `headroom deploy` command and reused the existing
install manifest/apply/start/rollback path.
- Added conservative runtime selection for GPU Docker, plain Docker,
native schedulers, and detached Python fallback.
- Added Docker runtime support for manifest-driven `--gpus all`
passthrough.
- Added tests for Docker selection, GPU Docker selection, detached
fallback, GPU command rendering, and subprocess wrapper compliance.
- Updated README and persistent-install docs to present the turnkey
deployment flow and performance-first GPU behavior.
- Allowed documented `opencode` targets through `headroom install apply
--target`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Formatting passes (`ruff format --check`)
- [x] Type checking passes in local pre-commit and CI
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
uv run --with pytest --with pytest-asyncio python -m pytest tests/test_cli/test_subprocess_utf8_encoding.py tests/test_cli/test_install_cli.py tests/test_install/test_runtime.py::test_build_runtime_command_for_docker_includes_gpu_passthrough tests/test_install/test_planner.py -q
47 passed in 1.57s

uvx --from ruff==0.15.17 ruff check headroom/cli/install.py headroom/install/runtime.py tests/test_cli/test_install_cli.py tests/test_install/test_runtime.py --output-format concise
All checks passed!

uvx --from ruff==0.15.17 ruff format --check headroom/cli/install.py headroom/install/runtime.py tests/test_cli/test_install_cli.py tests/test_install/test_runtime.py
4 files already formatted
```

GitHub checks are green on the current head.

## Real Behavior Proof

- Environment: Windows local worktree `C:\git\headroom`, Python 3.13 via
`uv`; GitHub Actions Ubuntu/macOS/Windows runners for full PR CI.
- Exact command / steps: Ran the focused deploy/install tests above,
checked the touched Python files with the CI-pinned Ruff version, and
confirmed the current PR head is mergeable with green GitHub checks.
- Observed result: The deploy command, runtime selection, Docker GPU
command rendering, install CLI behavior, and subprocess encoding
coverage all pass locally; the branch is no longer conflicted.
- Not tested: Actual RTX 4090 hardware passthrough on a physical NVIDIA
workstation; the PR tests conservative detection and Docker command
rendering without requiring GPU hardware in CI.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A - CLI/runtime behavior only.

## Additional Notes

CHANGELOG update is not included because this is an unreleased feature
PR and the repository's release tooling owns release notes from
conventional commits.
2026-07-15 18:37:20 +00:00
8f867e4622 fix(install): guard non-dict health config in 'install status' (#2150)
## Description

`headroom install status` crashes with an `AttributeError` when the
probed health endpoint returns a non-dict `config`.

```python
if payload and isinstance(payload, dict):
    click.echo(f"Health URL: {manifest.health_url.replace('/readyz', '/health')}")
    click.echo(f"Backend:    {payload.get('config', {}).get('backend', manifest.backend)}")
```

`payload` is guarded as a dict, but `payload['config']` is not.
`dict.get('config', {})` only substitutes the `{}` default when the key
is **absent** — a present-but-non-dict `config` (`null`, a string, a
list) is returned as-is, and the chained `.get('backend', ...)` then
raises `AttributeError`, crashing the command with a raw traceback.

Reachability: the Headroom proxy normally returns `config` as an object,
so this bites when `install status` probes a port that a different or
older service is occupying (which can emit `config: null` or a
non-object), or a build that emits `config: null`. The correctly-guarded
sibling already exists in the codebase — `wrap.py`'s
`_proxy_health_config` does `config = payload.get("config"); return
config if isinstance(config, dict) else None`.

## Fix

Guard the `config` value with `isinstance(config, dict)` before the
`.get('backend', ...)` lookup, mirroring `_proxy_health_config`. A
non-dict (or missing) `config` falls back to the manifest's backend.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/cli/install.py`: `install status` guards `config` with
`isinstance(config, dict)` before reading `backend`.
- `tests/test_cli/test_install_cli.py`: add
`test_install_status_survives_non_dict_config` (health payload with
`config: null` must not crash; backend falls back to the manifest).
- `CHANGELOG.md`: Bug Fixes entry.

## Testing

- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ uvx ruff@0.15.17 check headroom/cli/install.py tests/test_cli/test_install_cli.py
All checks passed!
$ python -m py_compile headroom/cli/install.py tests/test_cli/test_install_cli.py
OK
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`. Importing
`headroom` pulls in the torch/transformers stack and a full `pytest`
gets OOM-killed on this box, so I verified the access with a
dependency-free script that replicates the old vs guarded lookup, and
left the full pytest (including the new CLI test) to CI.
- Exact command / steps: ran the old `payload.get('config',
{}).get('backend', ...)` and the new guarded lookup against `config`
values of `null`, a string, a list, a proper object, and a missing key.
- Observed result: the old lookup raises `AttributeError` for every
non-dict `config`; the new lookup falls back to the manifest backend for
those and returns the real backend for a proper object (and the
missing-key case is unchanged). The new CLI test drives `install status`
with `probe_json` returning `{"config": null}` and asserts a clean exit
with the manifest backend.
- Not tested: a live foreign service occupying the port; full local
`pytest` deferred to CI (OOM, per above).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

The "unit tests pass locally" and "type checking" boxes are unchecked
because the full suite imports the ML stack, which I can't run in this
environment; the change adds an `isinstance` guard mirroring an existing
sibling, verified by the standalone proof and a new CLI test that reuses
the file's existing `install status` mocking harness.

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
2026-07-14 12:19:58 -04:00
Rod BoevandJerrettDavis 3a39cb99ad install: couple Codex routing to persistent runtime readiness (#2043)
## Description

Persistent install currently writes Codex routing before runtime
readiness, and it leaves that routing in place while the runtime is
stopped or after a failed start. This changes the lifecycle so routing
is applied only after the persistent runtime is ready and reverted
before stop or removal, which prevents Codex from getting stranded on a
dead `127.0.0.1:8787` provider.

Closes #2038

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Apply persistent provider mutations only after runtime readiness.
- Revert persistent provider mutations before stop and remove.
- Clear stale routing before recovery start paths, then reapply after
readiness.
- Add focused install lifecycle tests for the ordering change.

## Testing

- [x] Unit tests pass
- [x] Linting passes
- [ ] Type checking passes
- [x] New tests added for new functionality when applicable
- [ ] Manual testing performed

### Test Output

```text
uv run pytest tests/test_cli/test_install_cli.py -q
23 passed, 1 warning in 0.27s

uv run ruff check headroom/cli/install.py tests/test_cli/test_install_cli.py
All checks passed!

uv run ruff format --check headroom/cli/install.py tests/test_cli/test_install_cli.py
2 files already formatted
```

## Real Behavior Proof

- Environment: Windows, Python 3.12.13, no provider (persistent install
lifecycle change; no live Codex or proxy call)
- Exact command / steps: `uv run pytest
tests/test_cli/test_install_cli.py -q`, then `uv run ruff check` and `uv
run ruff format --check` on `headroom/cli/install.py` and
`tests/test_cli/test_install_cli.py`
- Observed result: 23 install-lifecycle tests pass and lint/format
checks pass, confirming persistent Codex provider mutations are now
applied only after runtime readiness and reverted before stop/remove
- Not tested: a live persistent-runtime start/stop cycle with Codex
routing

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-07-14 11:52:51 -04:00
JD Davis 1d2b76e72e fix: harden persistent install startup (#1851)
## Description

Hardens persistent install startup and proxy compression behavior for
issue #1843. Repeated `headroom install start` / scheduled ensure calls
no longer spawn duplicate runtimes by default, and `/v1/compress` now
fails open on compression timeout instead of returning a 503. The PR
also adds a machine-readable platform feature matrix and app-level
stabilization tests for health, compression functionality, timeout
behavior, and matrix evidence.

Refs #1843

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Wrapped direct persistent deployment starts with the existing
profile-local runtime start lock.
- Made `headroom install start` idempotent when the deployment is
already healthy.
- Added wedged-runtime handling: if a PID is running but `/readyz` does
not recover inside the grace window, stop it before starting again.
- Kept `install agent ensure` inside the already-held lock while
delegating to the shared start helper.
- Changed `/v1/compress` timeout behavior from `503 compression_timeout`
to fail-open `200` with original messages, `compression_skipped: true`,
and `skip_reason: compression_timeout`.
- Added `tests/test_platform_stabilization_functional.py` covering real
FastAPI health/compression routes, successful compression metrics,
timeout fail-open speed, and a real JSON tool payload that reduces
tokens.
- Added `docs/platform-feature-matrix.json` and
`docs/platform-stabilization.md` for Linux/macOS/Windows hardening
coverage and known gaps.
- Strengthened matrix tests so cited local test/workflow paths must
exist.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# Clean source tree without copied Rust extension: functional module is skipped locally, as CI copies _core from the built wheel.
> python -m pytest tests/test_install tests/test_cli/test_install_cli.py tests/test_platform_feature_matrix.py tests/test_platform_stabilization_functional.py -q
collected 120 items / 1 skipped
119 passed, 2 skipped in 17.08s

> python -m ruff check headroom/proxy/handlers/openai.py tests/test_platform_stabilization_functional.py tests/test_platform_feature_matrix.py
All checks passed!

# Local Windows compiled-core proof:
> python -m maturin build --profile ci --out dist-local
Built wheel for abi3 Python >= 3.10 to dist-local\headroom_ai-0.29.0-cp310-abi3-win_amd64.whl

# Copied _core.pyd from the wheel into headroom/ for local route execution, then:
> python -m pytest tests/test_platform_stabilization_functional.py -q
collected 4 items
4 passed in 6.71s

> python -m pytest tests/test_install tests/test_cli/test_install_cli.py tests/test_platform_feature_matrix.py -q
collected 120 items
119 passed, 1 skipped in 17.16s

Commit hooks:
Sync plugin versions.....................................................Passed
check for merge conflicts................................................Passed
ruff.....................................................................Passed
ruff-format..............................................................Passed
mypy.....................................................................Passed
```

## Real Behavior Proof

- Environment: Windows 11, PowerShell, Python 3.13.13, worktree
`C:\git\headroom-stabilization` on branch
`jd/cross-platform-stabilization`.
- Exact command / steps: built the Windows wheel with `maturin`,
extracted `_core.pyd`, ran the new FastAPI route tests and
install/matrix tests listed above, then removed generated artifacts
before committing.
- Observed result: direct start paths now no-op when healthy, skip
spawning when the start lock is contended, and stop a wedged runtime
before restart. `/v1/compress` now returns original messages quickly on
timeout instead of a 503. The real JSON tool-payload smoke test returns
`tokens_before > tokens_after`, `tokens_saved > 0`, `compression_ratio <
1.0`, and non-empty transforms through the public route.
- Not tested: full native Windows persistent process e2e remains blocked
by the upstream CRT/wheel issue already documented in workflows and in
the matrix. No real OS service was installed locally; service manager
behavior is covered by argument-level unit tests.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A

## Additional Notes

CHANGELOG is not updated because this is an unreleased
hardening/test/documentation pass. The platform matrix intentionally
records partial/blocked Windows/macOS e2e gaps instead of claiming full
coverage where the repo cannot currently run it.
2026-07-10 00:40:34 -04:00
Rod Boev 6fb5f3bc3d fix(install): persist --no-http2 override through install apply (#1676)
## Description

`headroom install apply` regenerates the deployment manifest on every
run, and that regeneration silently drops any manually-added
`--no-http2` override. The HTTP/2 workaround itself is already real and
already supported by `headroom proxy`, but persistent installs had no
first-class way to keep it. This PR adds `--no-http2` to `install
apply`, threads it into `build_manifest()`, and persists the flag in
`manifest.proxy_args` so it survives reapply. Closes #1615

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Added `--no-http2` to `headroom install apply`, and forwarded the flag
into `build_manifest()`.
- Extended `headroom/install/planner.py` so `build_manifest(...,
no_http2=True)` persists `--no-http2` into `manifest.proxy_args`.
- Added planner-level regression coverage for both the override path and
the default-preservation path.
- Added CLI-level regression coverage that proves `install apply
--no-http2` forwards correctly and that the help surface advertises the
flag.
- `CHANGELOG.md` intentionally not touched: repo policy generates
changelog entries from conventional commits rather than manual PR edits.

## Testing

- [x] Unit tests pass (`uv run pytest
tests/test_install/test_planner.py` and `uv run pytest
tests/test_cli/test_install_cli.py`)
- [x] Linting passes (`uv run ruff check .` and `uv run ruff format .
--check`)
- [ ] Type checking passes (`uv run mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
> rtk uv run pytest tests/test_install/test_planner.py -k no_http2 -q
collected 7 items / 5 deselected / 2 selected
tests\test_install\test_planner.py ..                                    [100%]
2 passed, 5 deselected in 0.18s

> rtk uv run pytest tests/test_cli/test_install_cli.py -k no_http2 -q
collected 19 items / 17 deselected / 2 selected
tests\test_cli\test_install_cli.py ..                                    [100%]
2 passed, 17 deselected in 0.23s

> rtk uv run pytest tests/test_install/test_runtime.py -q
collected 19 items
tests\test_install\test_runtime.py ..........F........                   [100%]
FAILED tests/test_install/test_runtime.py::test_runtime_start_lock_blocks_another_process
1 failed, 18 passed in 0.44s
(Confirmed pre-existing on unmodified origin/main via `git stash` in this worktree,
identical failure with none of this PR's changes applied. Environment-specific lock-file
flakiness in this sandbox, unrelated to install-manifest persistence; runtime.py was not
touched by this change.)

> rtk uv run ruff check headroom/cli/install.py headroom/install/planner.py tests/test_install/test_planner.py tests/test_cli/test_install_cli.py
All checks passed!

> rtk uv run ruff format --check headroom/cli/install.py headroom/install/planner.py tests/test_install/test_planner.py tests/test_cli/test_install_cli.py
4 files already formatted
```

## Real Behavior Proof

- Environment: local source checkout with `uv` dev environment, using
the existing install CLI and manifest builder, in worktree
`D:\Repos\headroom-pr-1615-persist-install-http2-override`.
- Exact command / steps: ran `headroom install apply --help` through
`CliRunner`, ran a direct `build_manifest(..., no_http2=True)` proof,
and ran the focused planner, CLI, runtime, and lint checks.
- Observed result: on `origin/main`, `install apply --help` lacked
`--no-http2` and `build_manifest(..., no_http2=True)` raised `TypeError:
build_manifest() got an unexpected keyword argument 'no_http2'`; on this
branch, `install apply --help` lists `--no-http2`, `build_manifest(...,
no_http2=True)` returns a manifest whose `proxy_args` contains exactly
one `--no-http2` entry (`['--host', '127.0.0.1', '--port', '8787',
'--mode', 'token', '--backend', 'anthropic', '--telemetry',
'--no-http2']`), persistent installs now preserve the existing HTTP/2
disable flag across `install apply` regeneration, and runtime behavior
still comes entirely from replaying manifest `proxy_args` (`runtime.py`
was not modified).
- Not tested: a full persistent-service supervisor round-trip or full CI
suite locally.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable (not applicable,
changelog entries are generated from conventional commits per repo
policy)

## Additional Notes

This stays scoped to the install-manifest persistence seam only; it does
not revisit HTTP/2 default policy, retry behavior, or proxy transport
construction. Attribution: the implementation shape follows the
persistence pattern already established by #1365, and the remaining
install-layer gap was confirmed by `sarkarsital1959` in the 2026-07-01
comment on #1615.
2026-07-07 11:37:23 -05:00
Tejas Chopra bd76235f5c fix(cli): harden all CLI surfaces + fix docs accuracy (#1491)
## Summary

Full CLI audit + documentation accuracy pass. All 5 commits on this
branch:

### CLI Hardening (4 commits)
- **Clean errors instead of tracebacks**: corrupt manifests, missing
Docker, malformed JSONL, bad `--profile`, invalid env-var values all now
raise `click.ClickException` with helpful messages
- **Range validation**: ~25 numeric flags across 10 files now use
`click.IntRange`/`FloatRange` — `--port 0`, `--hours -1`, `--limit 0`
etc. produce clean usage errors instead of silent wrong behavior
- **Flag combination warnings**: conflicting combos (`--no-rate-limit` +
`--rpm`, `--no-optimize` + `--target-ratio`, `--telemetry` +
`--no-telemetry`) emit yellow warnings on stderr
- **`memory --db-path` default fixed**: was resolving to
`headroom_memory.db` (wrong bare file); now uses project store
`./.headroom/memory.db` if present, else `~/.headroom/memory.db`
- **`memory list --search` + filters**: `--scope`/`--session`/`--since`
were silently ignored when `--search` was also set; now filters are
applied to search results
- **`learn --verbosity --apply` now works**: the output shaper is off by
default (`HEADROOM_OUTPUT_SHAPER`); `--apply` now hot-enables it via
`POST /admin/runtime-env` on a running proxy, or prints explicit `export
HEADROOM_OUTPUT_SHAPER=1` instructions when no proxy is running
- **`perf --hours` overflow**: `1e9` hours no longer raises
`OverflowError`; treated as "all data"
- **`evals memory --categories` invalid input**: `abc,1,2` now raises
`BadParameter` instead of a raw `ValueError` traceback

### Documentation (1 commit, 20 files)

Corrected factual errors found by 3 parallel audit agents across root
docs, wiki, and the published Fumadocs site:

**Critical (caused runtime errors or wrong behavior if followed):**
- `simulation.mdx`: `plan.transforms_applied` -> `plan.transforms`;
`plan.savings_percent` -> computed from available fields (both raised
`AttributeError`)
- `shared-context.mdx`: `import { SharedContext } from "headroom"` ->
`"headroom-ai"` (5x `ImportError`)
- `claude-code-azure-foundry.mdx`: `pip install headroom` -> `pip
install headroom-ai`
- `api-reference.mdx` + `configuration.mdx`: `from headroom import
GoogleProvider` -> `from headroom.providers import GoogleProvider`
- `ccr.mdx`: CCR TTL default 300s -> 1800s (30 min)

**Fabricated flags removed:**
- `wiki/proxy.md` + `wiki/cli.md`: `--no-intelligent-context`,
`--no-intelligent-scoring`, `--no-compress-first` (none exist); replaced
with real CCR flags
- `wiki/configuration.md`: `--no-ccr-responses`, `--no-ccr-expansion`
(none exist); replaced with real flags
- `wiki/troubleshooting.md`, `wiki/metrics.md`,
`docs/troubleshooting.mdx`: `headroom proxy --log-level debug` (flag
doesn't exist)

**Stale content corrected:**
- `llms.txt`: telemetry stated as enabled-by-default (it's opt-in); wrap
list had 5 tools (now 11)
- `README.md`: compatibility matrix added 5 missing `wrap` targets;
`unwrap`, `doctor`, `init`/`install`, savings-analytics now mentioned
- `SECURITY.md`: supported version table showed 0.2.x (current: 0.27.x)
- `wiki/learn.md`: 5 missing flags added; verbosity shaper-off behavior
documented
- `wiki/quickstart.md`: "Configuration Reference" linked to `api.md`
(wrong) -> `configuration.md`
- `CacheAlignerConfig.enabled` default corrected: `True` -> `False`
- `opencode.mdx`: `--port` default wrong ("random") -> 8787; `openai`
backend removed
- `CONTRIBUTING.md`: broken Markdown table cell fixed
- `docs/meta.json`: `claude-code-azure-foundry` added to nav (was
unreachable orphan page)
- `configuration.mdx`: SDK modes vs proxy `--mode` now clearly
distinguished

## Test plan

- [x] `python -m pytest tests/ -x -q` — 857 passed, 0 failures
- [x] 41-combination CLI smoke test (all flag combos across 8 commands)
— 0 tracebacks
- [x] `ruff check` on all modified Python files — clean
- [x] Docs changes are removals/corrections of fabricated or stale
content; no new claims introduced
2026-06-27 14:48:43 -07:00
Vinay Gupta 6c83790680 fix(opencode): write local MCP config (#1381)
## Description

Fixes the OpenCode config corruption reported in #1380 for wrap, MCP
registration, and provider-scope install paths.

OpenCode MCP entries are local stdio servers, not remote HTTP endpoints.
This changes Headroom's OpenCode MCP serialization to write `type:
"local"` with `command: ["headroom", "mcp", "serve"]`, uses OpenCode's
`environment` field for MCP env vars, and still reads the older `env`
key for compatibility.

This also stops provider-only OpenCode config injection from creating a
fake `http://127.0.0.1:<port>/mcp` entry, so `headroom wrap opencode
--no-mcp` no longer leaves `mcp.headroom` behind. Finally, the install
CLI/docs now accept and document `--target opencode` with provider
scope.

This does not change the broader `headroom mcp status/uninstall`
behavior from #1380; that looks like a separate follow-up.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Write OpenCode MCP entries as local stdio config instead of remote
`/mcp` config.
- Use `environment` for OpenCode MCP env vars while continuing to read
legacy `env` entries.
- Stop OpenCode provider injection/persistent provider install from
adding MCP config.
- Keep `--no-mcp` from writing `mcp.headroom` while preserving other MCP
entries such as Serena.
- Allow `headroom install apply --target opencode` at the CLI layer.
- Update OpenCode docs and changelog.

## Testing

- [x] Focused unit tests pass
- [x] Linting passes (`ruff check .`)
- [x] Formatting passes (`ruff format --check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for the fixed behavior
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_mcp_registry_opencode.py tests/test_cli/test_wrap_opencode.py tests/test_providers_opencode_config.py tests/test_providers_opencode_install.py tests/test_cli/test_install_cli.py tests/test_install/test_providers.py
Pytest: 164 passed

$ uvx ruff check .
All checks passed!

$ uvx ruff format --check .
986 files already formatted

$ uvx mypy --config-file pyproject.toml headroom
Success: no issues found in 398 source files
```

## Real Behavior Proof

- Environment: macOS local worktree at
`/Users/vinaygupta/Desktop/git/headroom-fix-opencode-mcp-config`; branch
`fix-opencode-mcp-config`; commit `aea96208`.
- Exact command / steps: ran the focused OpenCode/installer regression
suite plus Ruff lint/format checks and mypy commands shown above.
- Observed result: the focused tests pass and cover OpenCode MCP
serialization as `type: "local"`, `command: ["headroom", "mcp",
"serve"]`, `environment` env vars, `--no-mcp` not writing
`mcp.headroom`, provider-scope install not adding MCP config, and
`install apply --target opencode` being accepted.
- Not tested: full `pytest` locally, because collection requires the
native `headroom._core` extension in this worktree. Attempting the
project runner hit a local native build failure first: `esaxx-rs` failed
compiling `src/esaxx.cpp` with `fatal error: 'cstdint' file not found`.
The broader generic `headroom mcp status/uninstall` behavior from #1380
is intentionally left for a follow-up.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

Scope note: generic `mcp status/uninstall` support from #1380 is
intentionally left as a separate follow-up PR.
2026-06-26 12:23:54 -05:00
Priyanshu Sharma 8da0b4e565 fix(install): guard install_agent_ensure against duplicate runtime spawns (#1301)
## Type of Change

- [x] Bug fix (non-breaking change which fixes an issue)

## Description

`install_agent_ensure` in `cli/install.py` only checked
`probe_ready(health_url)`. If the proxy was alive-but-not-ready (e.g.
during cold start while tokenizers load — ~38s on Windows),
`probe_ready` returned false and it unconditionally called
`_start_deployment` → `start_detached_agent`, spawning a **second
runtime** without:

1. acquiring `acquire_runtime_start_lock`
2. checking `runtime_status`
3. stopping the existing instance

Two proxies then contend for `127.0.0.1:<port>`; only one can bind, and
the deployment ends up wedged (never ready). Every subsequent ensure
spawns yet another runtime → restart storm.

By contrast, the hook path `cli/init.py:_ensure_profile_running` does it
correctly: it acquires the start-lock, checks `runtime_status`, and
`stop_runtime`s a wedged instance before starting a fresh one.

Closes #1151.

## Changes Made

- Added `acquire_runtime_start_lock` to the imports from
`install.runtime` in `headroom/cli/install.py`
- Rewrote `install_agent_ensure` to mirror the guarded pattern from
`_ensure_profile_running` in `cli/init.py`:
- Fast-path probe: if proxy is already ready, return immediately
(preserves existing behavior)
- Lock acquisition: acquire `acquire_runtime_start_lock` — if another
ensure holds it, return without spawning (prevents duplicate)
- Double-checked locking: re-probe `probe_ready` after acquiring the
lock (race window handled)
- Wedged instance detection: if `runtime_status` says "running" but
proxy isn't ready within 15s grace period, call `stop_runtime` before
starting fresh
  - Fall through to `_start_deployment` only when truly needed
- Added `_STARTUP_READY_TIMEOUT_SECONDS = 15` constant (matching the
value used in `_ensure_profile_running`)
- **Failure propagation (addresses @JerrettDavis's review feedback):**
removed the `try/except Exception` wrapper around the guarded block.
`install agent ensure` is an automation-facing CLI command and must exit
non-zero on failure so callers can distinguish a successful ensure from
a failed one. The `init.py` hook path retains its `try/except` because
silent retry is intentional there. The control flow is shared; the error
contract is intentionally different because the call sites have
different needs.
- Added 5 regression tests in `tests/test_cli/test_install_cli.py`:
- `test_install_agent_ensure_no_spawn_when_lock_not_acquired` — verifies
no runtime spawned when lock is contended (the core bug)
- `test_install_agent_ensure_stops_wedged_runtime_before_restart` —
verifies `stop_runtime` is called BEFORE `_start_deployment` when
instance is wedged (ordering assertion: `calls.index("stop") <
calls.index("start_deployment")`)
- `test_install_agent_ensure_starts_when_stopped_and_lock_acquired` —
verifies the normal start path including the real `_start_deployment` →
`start_detached_agent` wiring
- `test_install_agent_ensure_no_duplicate_spawn_after_lock_recheck` —
verifies double-checked locking prevents duplicate when proxy becomes
ready between initial probe and lock acquisition
- `test_install_agent_ensure_propagates_start_deployment_failure` —
**new** regression test for the failure-propagation fix: monkeypatches
`_start_deployment` to raise `click.ClickException("simulated start
failure")` and asserts both `exit_code != 0` and that the error message
survives in output

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```
$ uv run python -m pytest tests/test_cli/test_install_cli.py -v --tb=short

tests/test_cli/test_install_cli.py::test_install_apply_starts_service_supervisor PASSED [  6%]
tests/test_cli/test_install_cli.py::test_install_status_includes_backend_from_health_probe PASSED [ 12%]
tests/test_cli/test_install_cli.py::test_install_restart_uses_internal_helpers PASSED [ 18%]
tests/test_cli/test_install_cli.py::test_install_apply_rejects_invalid_profile PASSED [ 25%]
tests/test_cli/test_install_cli.py::test_install_apply_rejects_provider_scope_targets_without_support PASSED [ 31%]
tests/test_cli/test_install_cli.py::test_install_apply_restores_previous_deployment_after_failed_update PASSED [ 37%]
tests/test_cli/test_install_cli.py::test_install_start_rejects_task_lifecycle PASSED [ 43%]
tests/test_cli/test_install_cli.py::test_install_apply_uses_docker_runtime_for_persistent_docker PASSED [ 50%]
tests/test_cli/test_install_cli.py::test_install_remove_continues_when_runtime_teardown_errors PASSED [ 56%]
tests/test_cli/test_install_cli.py::test_install_agent_ensure_reports_already_healthy PASSED [ 62%]
tests/test_cli/test_install_cli.py::test_install_agent_run_exits_with_foreground_status PASSED [ 68%]
tests/test_cli/test_install_cli.py::test_install_agent_ensure_no_spawn_when_lock_not_acquired PASSED [ 75%]
tests/test_cli/test_install_cli.py::test_install_agent_ensure_stops_wedged_runtime_before_restart PASSED [ 81%]
tests/test_cli/test_install_cli.py::test_install_agent_ensure_starts_when_stopped_and_lock_acquired PASSED [ 87%]
tests/test_cli/test_install_cli.py::test_install_agent_ensure_no_duplicate_spawn_after_lock_recheck PASSED [ 93%]
tests/test_cli/test_install_cli.py::test_install_agent_ensure_propagates_start_deployment_failure PASSED [100%]

============================== 16 passed in 0.29s ==============================
```

```
$ uv run ruff check headroom/cli/install.py tests/test_cli/test_install_cli.py
All checks passed!

$ uv run ruff format --check headroom/cli/install.py tests/test_cli/test_install_cli.py
2 files already formatted

$ uv run mypy headroom/cli/install.py --ignore-missing-imports
Success: no issues found in 1 source file
```

## Real Behavior Proof

- **Environment**: Python 3.11.14, Linux 6.17.0, headroom dev
environment (uv-synced), rebased onto `upstream/main` at `3be2526b`
- **Exact command / steps**: `uv run python -m pytest
tests/test_cli/test_install_cli.py -v --tb=short` (and the ruff + mypy
commands above)
- **Observed result**: All 16 tests pass (11 existing + 5 new regression
tests). The 5 new tests verify: (1) no-spawn when the lock is contended,
(2) `stop_runtime` ordering before `_start_deployment` on a wedged
instance, (3) normal start path, (4) double-checked locking after the
lock is acquired, (5) failure propagation when `_start_deployment`
raises — this last test is the regression for @JerrettDavis's review
feedback. ruff check, ruff format --check, and mypy all pass clean.
- **Not tested**: Live deployment with concurrent `install agent ensure`
invocations on Windows (only unit tests with monkeypatched runtime
functions). The fix mirrors the proven pattern from
`_ensure_profile_running` which is already battle-tested in the init
hook path.

## Review Readiness

- [x] I have performed a self-review of my own code
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A

## Additional Notes

**Triage of labels on this PR:**

- `status: needs author action` — **stale**. The 4 `Real Behavior Proof`
fields (`Environment`, `Exact command / steps`, `Observed result`, `Not
tested`) are all present in this body. The bot snapshot was taken before
the body was filled in. Requesting the label be dropped on the next bot
run.
- `status: ci failing` — **CI env, not caused by this PR.**
`install-native (macos-latest)` and `wrap-native (macos-latest)` fail
during the editable Rust/Python extension build with `ld: library
'clang_rt.osx' not found`, which is before this command path runs.
@JerrettDavis confirmed this is not caused by the PR. All Linux jobs,
all unit/integration/E2E jobs, lint, commitlint, template check, and
Docker E2E jobs are green. `mergeable: MERGEABLE` is the actual gate.

**CHANGELOG:** not updated — this is a single bug fix in an unreleased
section, and the maintainers have not requested CHANGELOG entries for
individual PRs in past PRs in this repo. Happy to add an entry under `##
Unreleased` if requested.
2026-06-24 09:54:28 -05:00
Rudimar RonsoniandRudimar Ronsoni b4571cc346 feat: headroom wrap opencode / unwrap opencode CLI (#1105)
## Summary

This PR implements transparent `headroom wrap opencode` support without
asking users to edit OpenCode provider URLs, choose an extra CLI flag,
or maintain a static provider list.

The wrapper now lives at the runtime transport boundary: OpenCode keeps
its user/provider config, while Headroom intercepts outbound provider
traffic in-process and routes it through the local Headroom proxy.

## What changed

### Transparent OpenCode wrapping

- `headroom wrap opencode` injects the `headroom-opencode` plugin
through `OPENCODE_CONFIG_CONTENT`.
- Existing OpenCode provider URLs are preserved. We do not rewrite user
config URLs to point at Headroom.
- Existing `OPENAI_BASE_URL` and `ANTHROPIC_BASE_URL` env vars are
preserved.
- Local OpenCode traffic, localhost traffic, and Headroom proxy traffic
bypass the shim to avoid loops.

### Runtime transport interception

- Added an OpenCode plugin transport shim that wraps:
  - `globalThis.fetch`
  - `http.request` / `http.get`
  - `https.request` / `https.get`
- External provider calls are routed to the local Headroom proxy.
- The original upstream origin is passed through `x-headroom-base-url`,
so the proxy can forward to the real provider without changing OpenCode
config.
- External `http2.connect` is blocked loudly instead of allowing direct
provider traffic to leak outside Headroom.

### Live provider additions

Provider coverage is no longer based on a static config scan. Because
routing happens at outbound request time, providers added mid-session
are routed through Headroom automatically as long as they use the
covered Node transport paths.

### Subagent and child-process coverage

- The parent OpenCode plugin sets a packaged Node preload shim through
`NODE_OPTIONS=--import=.../hook-shim/handler.js`.
- The transport shim patches `child_process.spawn`, `exec`, `execFile`,
and `fork` so child Node processes receive the Headroom preload even
when OpenCode passes a custom `env`.
- The child-process shim fails closed if it loads without
`HEADROOM_OPENCODE_TRANSPORT_PROXY_URL`.
- This closes the subagent leak path where a child Node process could
otherwise start without Headroom transport interception.

## Why this goes beyond PR #1089

PR #1089 improves OpenCode provider registration, but it still focuses
on provider config shape. This PR moves the enforcement boundary to
runtime transport interception.

This PR goes further because:

- No provider URL rewriting is required.
- New providers added mid-session are covered automatically.
- Subagents and child Node processes inherit the Headroom transport
shim.
- Direct external HTTP/2 paths fail loudly instead of leaking.
- The wrap remains transparent to the user's OpenCode provider config.
- The wrapper is fail-closed for unsupported child-process preload
state.

## Additional robustness fixes

While validating the change in Docker, the full Python suite exposed
unrelated Linux/container robustness issues. These are fixed in this PR
so the suite is green:

- Binary cache handling now treats cache paths under a non-writable
existing parent as unavailable, including when tests run as root in
Docker.
- `release_version.py` honors `MANUAL_VER` before git calls so direct
script execution works outside a `.git` checkout.
- Test logger isolation now resets relevant Headroom child loggers so
proxy logging setup cannot poison later `caplog` tests.
- The scanner missing-path test now uses a guaranteed missing `tmp_path`
child instead of relying on `/nonexistent/path`.

## Validation

All implementation validation was run inside Docker.

- Full Python suite from a fresh Docker copy: `6605 passed, 523
skipped`.
- Ruff on changed Python/OpenCode paths: passed.
- OpenCode plugin typecheck: passed.
- OpenCode plugin tests: `9 passed`.
- OpenCode plugin build: passed.
- Hook shim preload smoke test: passed.

## Notes

This PR intentionally does not add a CLI option. `headroom wrap
opencode` means full wrap. Either Headroom wraps OpenCode transparently,
or the path fails loudly instead of silently leaking provider traffic.

---------

Co-authored-by: Rudimar Ronsoni <6081613+rudironsoni@users.noreply.github.com>
2026-06-22 11:07:12 -05:00
JerrettDavisandCopilot bd242fc62d test: expand persistent install coverage
Add focused regression coverage for install, runtime, provider, state, health, supervisor, and persistent wrap flows so the new persistent deployment surfaces are exercised more thoroughly in CI.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-11 18:24:15 -05:00
JerrettDavisandCopilot 865bef2216 fix: harden persistent install wrappers and review gaps
Align Docker-native wrapper help and runtime behavior with the Python install contract, including persistent deployment metadata, baked install-image defaults, and explicit unsupported wrap targets.

Harden the Python persistent-install path with profile validation, safer provider-scope handling, Windows environment restoration, runtime parity improvements, and rollback-safe apply/update behavior.

Update README, Docker install docs, CI, and focused regressions to cover the Windows BOM failure, wrapper parity, compose coverage, and Docker-native wrap behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-11 18:03:21 -05:00
JerrettDavisandCopilot 21896a095c feat: add persistent install lifecycle management
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-11 13:47:05 -05:00