mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
feat/https-compose-caddy
27
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
6b903cbc81 |
fix(install): stabilize Windows task lifecycle and eliminate console window popup (#4627)
## Description Fixes Windows persistent installation lifecycle and console flash issues: 1. **`start_supervisor()` failure and fallback immediate startup**: When S4U / BootTrigger task registration is rejected without administrator credentials (Issue #3418), Headroom falls back to session-scoped health recovery and intentionally omits the `<profile>-startup` task. `start_supervisor()` now tolerates absent tasks during `schtasks` execution and immediately launches the fallback runtime (`start_detached_agent`) when the startup task is absent, avoiding the 45-second startup timeout. 2. **Watchdog tracking under Task Scheduler supervisor lifecycle**: `_render_windows_vbs_launcher()` executes PowerShell synchronously via `WshShell.Run(..., 0, True)`. This keeps `wscript.exe` running through the full health check lifecycle, allowing Task Scheduler's `IgnoreNew` policy to prevent overlapping executions and enabling `schtasks /End` during `stop_supervisor()` to cleanly terminate in-flight recovery before process exit. 3. **`stop_runtime()` teardown identity race**: Immediately after `TerminateProcess` on Windows, PEB and command-line memory unmap, causing `_process_identity(pid, manifest)` to return `None`. The polling loop now checks `if _process_identity(...) is False:` to tolerate transient `None` during process kernel teardown. 4. **Flashing console/terminal window on health check**: Task Scheduler running under `InteractiveToken` executes `.vbs` actions via `wscript.exe //B //Nologo`, preventing interactive console/terminal window popups. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - **`headroom/install/supervisors.py`**: - In `start_supervisor()`, tolerated absent tasks during `schtasks` enablement and execution; when `<service_name>-startup` is absent, immediately launches `start_detached_agent(manifest.profile)` if the runtime is not yet ready, while continuing to surface genuine permission errors. - In `_render_windows_vbs_launcher()`, set `WaitOnReturn=True` in `WshShell.Run(..., 0, True)` to keep `wscript.exe` alive for the duration of the PowerShell check so Task Scheduler accurately tracks task activity and supports `schtasks /End`. - Updated `_windows_task_xml()` so `.vbs` actions execute through `wscript.exe //B //Nologo`. - Updated `install_supervisor()` fallback path to register the silent VBS launcher under `InteractiveToken`. - **`headroom/install/runtime.py`**: - Changed `_process_identity(pid, manifest) is not True` to `_process_identity(pid, manifest) is False` in the `stop_runtime()` polling loop. - Removed duplicate `if not pid_alive(pid): if not pid_alive(pid):`. - **`tests/test_install/test_supervisors.py`**: - Added coverage in `test_windows_start_tolerates_absent_tasks` asserting immediate detached agent startup when the startup task is absent. - Added `test_windows_start_skips_detached_agent_when_already_ready` and `test_windows_start_surfaces_access_denied`. - Added `test_windows_task_xml_vbs_uses_wscript` and updated `test_render_windows_vbs_launcher` to assert synchronous `, 0, True` execution. - **`tests/test_install/test_runtime.py`**: - Added `test_stop_runtime_tolerates_transient_unknown_identity_during_exit`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --extra proxy pytest tests/test_install/test_supervisors.py tests/test_install/test_runtime.py -q 131 passed, 2 skipped in 17.89s $ uvx ruff check headroom/install/supervisors.py headroom/install/runtime.py tests/test_install/test_supervisors.py tests/test_install/test_runtime.py All checks passed! $ uvx ruff format --check headroom/install/supervisors.py headroom/install/runtime.py tests/test_install/test_supervisors.py tests/test_install/test_runtime.py 4 files already formatted $ uvx --from mypy==1.20.2 mypy headroom --ignore-missing-imports Success: no issues found in headroom/install/supervisors.py and headroom/install/runtime.py ``` ## Real Behavior Proof - Environment: Windows 11 x86_64, Python 3.13.17, headroom 0.41.0 - Exact command / steps: Executed 'headroom install apply' on a standard user account to trigger session-scoped supervisor fallback. Verified Task Scheduler entry and executed 'schtasks /Run /TN headroom-default-health'. Verified immediate daemon startup and teardown via 'headroom install status' and 'headroom install stop'. - Observed result: Task Scheduler registered headroom-default-health with wscript.exe //B //Nologo ensure-headroom.vbs. Background health runs completely silently with zero console window popups. When the startup task is absent, start_supervisor immediately launches the detached agent, and stop_supervisor ends the in-flight task process cleanly. - Not tested: Non-Windows platforms (Linux systemd and macOS launchd paths are untouched and guarded behind platform checks). ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: Stable. - Stable/default behavior changed: On Windows, session-scoped scheduled tasks run through wscript.exe rather than cmd.exe to prevent terminal window popups, and missing startup tasks trigger immediate runtime launch. - Kill switch / disable path: Not applicable (deployment script generation). - Unsafe override required: No. - Qualification impact: None. - Rollback path: Revert this pull request. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ? it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
7b6cb77905 |
fix(install): keep the HuggingFace cache on a writable mount in Docker presets (#4207) (#4209)
## Description With the `persistent-docker` preset, Kompress never loads. The container runs as the host user (`--user 501:20`) with `HOME=/tmp/headroom-home`, but the image never creates that directory. Docker creates it as `root:root 0755` to hold the bind mounts, so the proxy cannot create `~/.cache` and the HuggingFace download of the Kompress model fails with `Permission denied (os error 13)`. The log never shows that error: it reports a missing dependency instead (`Kompress requires onnxruntime or torch`), pointing the operator at reinstalling extras that are already present. Fix: point `HF_HOME` and `XDG_CACHE_HOME` at the bind-mounted, host-owned `.headroom` workspace so the cache is writable regardless of who Docker made the container's `$HOME`; and keep the underlying `OSError` in both `kompress_compressor.py` log sites instead of masking it. Closes #4207 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/install/runtime.py`: `build_runtime_command()` now emits `HF_HOME` and `XDG_CACHE_HOME` as real `--env` flags pointing under the bind-mounted `.headroom` workspace (`<container_home>/.headroom/cache`), and applies them to the persistent-container args path, not only the dev/foreground path. - `headroom/install/runtime.py`: added `_append_cache_env()` and wired it through both command builders. - `scripts/install.sh` and `docker/docker-compose.native.yml`: set `HF_HOME`/`XDG_CACHE_HOME` under the writable `.headroom` mount for the same preset. - `headroom/transforms/kompress_compressor.py`: `_create_onnx_session()` appends the underlying cause to its `No loadable ONNX artifact` message; `_load_kompress()` re-raises the real ONNX failure (chained) instead of reporting a missing dependency when onnxruntime is present but loading failed. ## Testing - [x] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ .venv/bin/python -m pytest tests/test_install/ tests/test_kompress_failsafe.py -q 278 passed, 11 skipped, 1 failed The single failure is tests/test_install/test_runtime.py::test_build_runtime_command_lets_the_host_dashboard_read_its_data which fails with ImportError: FastAPI required in this environment. It fails identically on unmodified main here, so it is a local environment gap, not this change. New regression tests, all passing: - test_build_runtime_command_keeps_hf_cache_on_the_writable_mount - test_build_runtime_command_persistent_path_keeps_hf_cache_on_the_writable_mount - test_no_loadable_artifact_message_carries_the_underlying_error - test_onnx_failure_while_onnxruntime_available_reports_the_real_error - test_missing_dependency_still_reports_install_hint $ bash -n scripts/install.sh # clean $ python -c "import yaml,sys;yaml.safe_load(open('docker/docker-compose.native.yml'))" # clean ``` ## Real Behavior Proof - Environment: Linux, Python venv at `.venv`, repo at this branch. No Docker daemon available on this machine. - Exact command / steps: `pytest tests/test_install/ tests/test_kompress_failsafe.py -q`; asserted the generated `docker run` argv from both `build_runtime_command()` and `build_persistent_container_args()` contains `--env HF_HOME` / `--env XDG_CACHE_HOME` pointing under the `.headroom` mount. - Observed result: all new tests pass; both argv builders now emit the cache env pointing at the writable host-owned mount. - Not tested: the end-to-end container run from the issue's reproduction steps — no Docker daemon here. The fix is verified at the level of the generated `docker run` argv and the unit tests above. ## Runtime Rollout Safety - Rollout-managed feature(s): Kompress proxy model loading under the `persistent-docker` preset. - Minimum rollout channel: all channels that use the persistent-docker preset; the change only affects env vars for cache directories. - Stable/default behavior changed: no. Default (non-persistent) paths are unchanged; persistent paths gain two `--env` flags pointing at an existing writable mount. - Kill switch / disable path: unset/override `HF_HOME`/`XDG_CACHE_HOME`, or run without the persistent-docker preset. - Unsafe override required: no. - Qualification impact: none. - Rollback path: revert this commit; the preset returns to the previous (broken) cache behavior. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes - Docs untouched: no user-facing docs describe `HF_HOME`/`XDG_CACHE_HOME` for this preset, so no documentation change was needed (hence the unchecked box). - No lint/type-check run: `ruff`/`mypy` are not installed in this environment. - No Docker daemon available, so the container-level e2e reproduction was not executed — flagged under "Not tested" above. --------- Co-authored-by: Shivam Maurya <shivam@programmingwithmaurya.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> Co-authored-by: JerrettDavis <jerrett@headroomlabs.ai> |
||
|
|
c191beef59 |
refactor: remove write-only and test-only code in subscription and install (#4123)
## Description This PR removes private code in `subscription/` and `install/` that nothing reads. - `subscription/tracker.py`: `_PERSIST_FILE_ENV`, `_DEFAULT_PERSIST_DIR` and `_DEFAULT_PERSIST_FILE` are left over from before the path moved to `paths.subscription_state_path()`. `_full_tokens` was a counter that was written on every request and never read. It also kept an 8-character OAuth token prefix in memory. - `install/runtime.py`: `_process_matches_runtime` was a test-only wrapper for `_process_identity(...) is True`. Product code calls `_process_identity` directly. Evidence: `rg -n '<name>' headroom tests crates plugins sdk examples benchmarks scripts e2e docs wiki .github pyproject.toml` and the same search over the consumer repos find only the definitions, plus the tests changed here. No `getattr` string spelling exists for any of them. Refs #4136 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - `headroom/subscription/tracker.py`: remove the three persist constants and `_full_tokens`. - `headroom/install/runtime.py`: remove `_process_matches_runtime`. - `tests/test_install/test_runtime.py`: the six asserts now check `_process_identity(...) is True` or `is False` directly. - `tests/test_subscription_tracker.py`: drop the one assert on `_full_tokens`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ pytest -q -p no:cacheprovider tests/test_subscription_tracker.py tests/test_install/test_runtime.py tests/test_network_diff_capture.py tests/test_subscription_window_render.py tests/test_subscription_poll_credential_policy.py tests/test_stateless_no_disk.py tests/test_cli_learn.py # on origin/main ( |
||
|
|
64a661d5b4 |
fix(docker): adds mount for the litellm config dir (#2454)
litellm expects access token in its config dir for github copilot
## Description
<!-- Briefly explain the change and why it is needed. -->
I could not authenticate github copilot via litellm when using
persistent install via docker without mounting litellm's config dir
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- adds mount for litellm's config dir to the proxy container
## Testing
<!-- Check what you actually ran, then paste the real command output
below. -->
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
ruff check .
All checks passed!
mypy headroom/install/runtime.py
Success: no issues found in 1 source file
pytest tests/test_install/test_runtime.py
====================================================================================================== test session starts =======================================================================================================
platform darwin -- Python 3.12.13, pytest-9.0.3, pluggy-1.6.0 -- /Users/derrik.fleming/repos/headroom/.venv/bin/python
cachedir: .pytest_cache
rootdir: /Users/derrik.fleming/repos/headroom
configfile: pyproject.toml
plugins: anyio-4.12.1, langsmith-0.9.3, asyncio-1.3.0, respx-0.23.1, cov-7.0.0
asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 25 items
tests/test_install/test_runtime.py::test_build_runtime_command_for_docker_includes_deployment_env PASSED [ 4%]
tests/test_install/test_runtime.py::test_build_runtime_command_for_docker_includes_gpu_passthrough PASSED [ 8%]
tests/test_install/test_runtime.py::test_build_runtime_command_docker_manifest_env_beats_host_passthrough PASSED [ 12%]
tests/test_install/test_runtime.py::test_build_runtime_command_for_docker_matches_wrapper_parity PASSED [ 16%]
tests/test_install/test_runtime.py::test_build_runtime_command_for_docker_does_not_duplicate_entrypoint PASSED [ 20%]
tests/test_install/test_runtime.py::test_resolve_headroom_command_prefers_headroom_binary PASSED [ 24%]
tests/test_install/test_runtime.py::test_resolve_headroom_command_falls_back_to_python_module PASSED [ 28%]
tests/test_install/test_runtime.py::test_runtime_env_and_mount_source PASSED [ 32%]
tests/test_install/test_runtime.py::test_build_runtime_command_python_and_docker_user PASSED [ 36%]
tests/test_install/test_runtime.py::test_read_pid_handles_invalid_content PASSED [ 40%]
tests/test_install/test_runtime.py::test_write_read_and_clear_pid PASSED [ 44%]
tests/test_install/test_runtime.py::test_runtime_start_lock_is_nonblocking PASSED [ 48%]
tests/test_install/test_runtime.py::test_runtime_start_lock_blocks_another_process PASSED [ 52%]
tests/test_install/test_runtime.py::test_run_foreground_and_detached_helpers PASSED [ 56%]
tests/test_install/test_runtime.py::test_start_detached_agent_closes_parent_log_fd PASSED [ 60%]
tests/test_install/test_runtime.py::test_start_detached_agent_closes_log_fd_when_popen_raises PASSED [ 64%]
tests/test_install/test_runtime.py::test_start_stop_wait_and_runtime_status_branches PASSED [ 68%]
tests/test_install/test_runtime.py::test_stop_runtime_for_docker_stops_and_removes_container PASSED [ 72%]
tests/test_install/test_runtime.py::test_runtime_status_reads_container_and_pid_state PASSED [ 76%]
tests/test_install/test_runtime.py::test_runtime_status_reports_live_pid_without_terminating PASSED [ 80%]
tests/test_install/test_runtime.py::test_runtime_status_survives_winerror87_systemerror PASSED [ 84%]
tests/test_install/test_runtime.py::TestRestartCurrentDeployment::test_foreground_when_no_deployment_env PASSED [ 88%]
tests/test_install/test_runtime.py::TestRestartCurrentDeployment::test_docker_returns_host_command_without_restarting PASSED [ 92%]
tests/test_install/test_runtime.py::TestRestartCurrentDeployment::test_task_mode_detected_and_not_restarted PASSED [ 96%]
tests/test_install/test_runtime.py::TestRestartCurrentDeployment::test_service_spawns_detached_restart PASSED [100%]
======================================================================================================= 25 passed in 0.35s =======================================================================================================
```
## Real Behavior Proof
- Environment: Docker / MacOS
- Exact command / steps: My steps included:
- `headroom install apply --image headroom:local --preset
persistent-docker --scope provider --providers manual --backend
github_copilot`
- send prompt through supported proxy to supported model
- Observed result: litellm cannot authenticate to provider
- Not tested: I didn't test all models from provider, the problem was
apparent. I did not test other providers through litellm.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
Add screenshots to help explain your changes.
## Additional Notes
Maintainer note: existing runtime command tests cover the Docker
command-building path touched here; this PR does not add new test files
or test cases.
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2454"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
Co-authored-by: JD Davis <mxjerrett@gmail.com>
|
||
|
|
44db755ee6 |
fix(install): replace the launchd wrapper with the proxy listener (#3224)
## Description The persistent service path can leave a wrapper, listener, native supervisor, or Docker container with conflicting ownership. Rebuild runtime ownership so launchd directly owns Python, Docker owns Docker across service/task presets, status and stop require deployment identity, readiness cannot be health-only, and failed cleanup preserves rollback state. Closes #3208 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (feature that would cause existing behavior to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Make the launchd-owned Python service exec the listener in place. - Classify Docker ownership from both runtime and preset, with no competing native supervisor. - Route Docker service/task start, restart, recovery, and teardown through the Docker owner. - Verify exact process command, port, deployment profile, and runtime before status or stop. - Add POSIX, macOS, and Windows process identity fallbacks without requiring psutil. - Require identity on readiness and fail closed when cleanup cannot prove the new owner stopped. - Persist an inactive previous-deployment snapshot, keep active state durable before owner start, and make environment/provider mutation activation transactional. - Surface cleanup, persistence, supervisor, Docker, and recovery-snapshot deletion failures with actionable recovery guidance. ## Testing - [x] Unit tests pass (`uv run --locked pytest tests/test_provider_package_init.py::test_install_registry_apply_provider_scope_mutations_skips_missing_and_none tests/test_install/test_providers.py tests/test_cli/test_install_cli.py tests/test_install/test_runtime.py tests/test_install/test_supervisors.py tests/test_install/test_paths.py tests/test_install/test_state.py -q`) - [x] Linting passes (`uv run --locked ruff check` on changed install, provider, supervisor, and focused test files) - [x] Type checking passes (`uv run --locked mypy headroom --ignore-missing-imports`) - [x] New tests added for the regression - [ ] Manual testing performed through live process/supervisor environments ### Test Output ```text uv run --locked pytest tests/test_provider_package_init.py::test_install_registry_apply_provider_scope_mutations_skips_missing_and_none tests/test_install/test_providers.py tests/test_cli/test_install_cli.py tests/test_install/test_runtime.py tests/test_install/test_supervisors.py tests/test_install/test_paths.py tests/test_install/test_state.py -q ============================= 197 passed in 1.07s ============================= uv run --locked ruff check . All checks passed! uv run --locked ruff format --check . 1534 files already formatted uv run --locked mypy headroom --ignore-missing-imports Success: no issues found in 527 source files git diff --check passed; the only worktree change is the pre-existing unstaged uv.lock line-ending change ``` ## Real Behavior Proof - Environment: Windows focused process and supervisor tests, with platform-specific launchd and Docker owner paths represented by fakes. - Exact command / steps: run the focused lifecycle and state suite listed above, targeted Ruff and format checks, and the full-head mypy check. - Observed result: each persistent mode has one owner; mismatched, malformed, or unavailable identity cannot authorize cleanup; active state is durable before owner start; failed cleanup retains recovery state; mutation rollback preserves unreverted records. - Not tested: owner-level POSIX exec, a live macOS launchd job, and a Docker daemon ## Runtime Rollout Safety - Rollout-managed feature(s): None. This changes the default persistent install and recovery ownership paths. - Minimum rollout channel: Stable, after platform-specific CI and maintainer review. - Stable/default behavior changed: Yes. Persistent Python service ownership and Docker supervisor routing change. - Kill switch / disable path: `headroom uninstall` removes the managed deployment; users can reinstall with another supported preset or runtime. - Unsafe override required: No. - Qualification impact: launchd, Docker, and Windows identity paths require their platform-owned environments; local proof uses focused tests and process/supervisor fakes. - Rollback path: Revert the rollback-ordering commits in this PR, then reinstall the prior deployment if needed. Uncertain cleanup retains the active new manifest and inactive previous snapshot until the new owner is removed and recovery is completed. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes The changelog is generated by the release pipeline. Live launchd and Docker-owner behavior remains platform-owned proof; local tests do not claim those owner environments. Final implementation head: `4004e4a5`. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3224" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
ee731d7ea8 |
fix(proxy): refuse token-less non-loopback binds, gate operator routes (#3852)
## Why Four related gaps in the proxy's network gate (findings 01-F1, 01-F2, 01-F5, 01-F9): - **Open bind.** A non-loopback bind with no `HEADROOM_PROXY_TOKEN` only logged a warning, then served the `/v1/*` relay, with the operator's upstream credentials, to every peer that could reach the port. - **Missing peer address.** `is_loopback_host(None)` returned `True`. A Unix-domain socket, an ASGI adapter that leaves `scope["client"]` unset, or a hand-built request therefore turned off the token gate, the `/admin` and `/debug` guards and the WebSocket gate. - **Operator data routes.** `/stats-history` (including CSV export), `/quota`, `/subscription-window` and `/metrics` were served to any network caller. - **WebSocket loopback exemption** checked only the peer IP, not the `Host` header, so it had no DNS-rebinding defence. ## What changes - New `headroom/proxy/bind_policy.py`. A non-loopback bind (`0.0.0.0`, `::`, a LAN address or a hostname) with no token is refused at three places: `create_app` (embedders and the multi-worker factory), `run_server` (one exit with code 2 before uvicorn forks) and the `headroom proxy` CLI (a clear error before the banner, exit code 1). An absent host means uvicorn's default `127.0.0.1`. - `HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1` acknowledges an open bind when the runtime already restricts reachability, such as a container published on `127.0.0.1`. The `proxy_open_bind` warning is still logged on that path, and the banner says the bind is open. - `is_loopback_host(None)` now returns `False` (fail closed), matching `identity.py`. - `/stats-history`, `/quota` and `/subscription-window` answer only: a non-loopback caller that passed the token gate, a loopback caller (peer and `Host` header both loopback), a trusted dashboard client, or, under an acknowledged token-less bind, the container's own default-gateway TCP peer with a loopback `Host` (the host browser reaching a `127.0.0.1`-published container). Everyone else gets 404, as with `/admin`. Settings writes keep their stricter rule with no token short-cut. - `/metrics` answers a non-loopback caller that passed the token gate, a loopback caller, a connecting peer in `HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS` (whatever it forwards), or a resolved client in the dashboard CIDRs. For CIDR peers a hostname `Host` is fine, so Prometheus scrapers work; a cross-origin browser request is still refused. - The token short-cut applies only to non-loopback requests. The gate exempts loopback peers without checking `Host`, so a loopback request has not proven it holds the token and must pass the loopback `Host` check. This blocks a DNS-rebound browser on the operator's machine. - The `/dashboard` shell stays ungated. It is a static template, like the already public `/dashboard/static` assets. Gating it would break the dashboard for containers published on host loopback, where the peer is the bridge gateway and `Host` is `localhost`. The data routes behind it are gated. - The WebSocket loopback exemption requires a loopback peer and a loopback `Host` header. A loopback peer with a foreign `Host` can still connect with the token. - Launchers that always publish on `127.0.0.1` set the acknowledgement themselves: `headroom install` (docker preset), `scripts/install.sh` and `scripts/install.ps1` (through one helper that adds the acknowledgement only together with the `127.0.0.1` publication), and `docker/docker-compose.native.yml`. - Docs: `proxy.mdx` documents the new variable and which routes the token also gates; `docker-install` (docs and wiki) and the `docker-compose.yml` comment show the acknowledgement in the loopback-published example. `e2e/docker-bind-security.sh` sets it and checks that the same launch without it refuses to start. ## Behaviour change (breaking) - `headroom proxy --host 0.0.0.0`, or `HEADROOM_HOST` set to any non-loopback address, without `HEADROOM_PROXY_TOKEN` now refuses to start. The error names both remedies. Set a token, bind loopback, or set `HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1` when the port is already confined. Anyone running the image by hand with `--host 0.0.0.0` and no token must add one of these. - Without a token, `/stats-history`, `/quota`, `/subscription-window` and `/metrics` return 404 to non-loopback callers outside a trusted CIDR. Prometheus scrapers on another host must be in `HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS` or send the token. - Code that relied on a request with no peer address being treated as loopback now gets the non-loopback path. - The default loopback run is unchanged. ## Tests - `tests/test_proxy_bind_policy.py`: the refuse/allow matrix (non-loopback, loopback, unset host, token, acknowledgement values), the warning on the acknowledged path, `create_app`, the `run_server` exit before uvicorn, and the CLI (refusal, `HEADROOM_HOST` from env, acknowledged start, offline banner, start with a token). - `tests/test_proxy_loopback_gating.py`: each data route against non-loopback, loopback, trusted-CIDR and token callers; the dashboard shell stays reachable; the host of an acknowledged, loopback-published container reads the data routes while another bridge peer, a forwarded gateway address and a foreign `Host` do not; CSV export is not served to network callers; `/metrics` across loopback, both CIDR lists, out-of-range peers, cross-origin browsers and token callers, including a trusted gateway relaying a scraper outside its own range. A regression test shows that a token on the proxy does not exempt a loopback peer from the `Host` check; it fails without the fix. - `tests/test_proxy_hardening.py`: the HTTP gate with no peer address, WebSocket `Host` cases, `require_loopback` and `/debug` with no peer. - Installer tests: `headroom install` adds the acknowledgement, and the shell and PowerShell wrappers add it only together with the loopback publication. Launcher-level tests start the proxy with the environment and `--host` that `build_runtime_command()` and `docker-compose.native.yml` produce and check that the host browser (bridge-gateway peer) gets `/stats-history` and another container does not. - Tests that bound `0.0.0.0` for unrelated reasons now set a token or the acknowledgement. Tests that read operator routes use an explicit loopback peer. On `main`, 14 of the gating and hardening tests fail and the bind-policy module does not exist. Manual runs: token-less `--host 0.0.0.0` exits with the refusal from both the CLI and `run_server`; with the acknowledgement, the data routes are 404 to a LAN peer and to a loopback peer with a foreign `Host`; with a token, a LAN peer gets 401 without it and 200 with it; the default loopback bind starts as before. Not tested: a real Unix-domain-socket deployment and a real Prometheus scrape (both covered by tests only). ## Merge order Part of a set. Suggested order: #3852 → #3891 → #3849 → #3860 → #3862 → #3863. This PR goes first: it is the gate the others assume, and it is the release-note event. ## Not in this PR - Enforcing the token for loopback callers when one is configured. Revised 2026-10-02 after review: `/metrics` matches gateway CIDRs against the TCP peer, and the operator data routes serve the host of a loopback-published container (bridge-gateway peer under the launchers' bind acknowledgement) so the default Docker dashboard works. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3852" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
b0c19a25f0 |
fix(security): reject unauthenticated public proxy binds (#3316)
## Description Headroom's published image and maintained Docker launchers could expose the proxy on every host interface without a proxy token. This change moves the security boundary to the deployment artifacts: published images default to a container-loopback listener, supported local launchers publish only on host loopback, and deliberate public deployments retain explicit token protection. The application factory remains transport-neutral. Closes #3283 ## Type of Change - [x] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [x] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Set both published image stages to a container-loopback default while keeping explicit host overrides available. - Changed native Compose and the maintained Bash and PowerShell launchers to use an internal `0.0.0.0` listener with host-loopback publication. - Preserved the top-level Compose token requirement and the existing HTTP and WebSocket credential behavior for deliberate public binds. - Scoped differential-capture token injection to the internal `headroom-proxy` destination and kept sensitive headers redacted in captured records. - Added focused launch-contract, native-core, capture, and Docker publication checks, plus a base/head harness that tolerates a base already containing the fix. - Updated Docker, installer, capture, and mirrored documentation, including an unauthenticated local loopback quickstart and an authenticated deliberate-public example. ## Testing - [x] Focused unit and integration tests pass - [x] Linting and formatting checks pass - [ ] Type checking passes - [x] New tests added for the changed deployment and capture behavior - [ ] Manual testing performed ### Test Output ```text uv run python -m pytest tests/test_proxy_hardening.py tests/test_install/test_native_installers.py tests/test_docker_compose_persistence.py tests/test_network_diff_capture.py tests/test_rust_core_smoke.py -q 76 passed, 1 skipped, 1 warning in 23.45s uv run ruff check tests/test_install/test_native_installers.py tests/test_install/test_runtime.py All checks passed! bash -n e2e/docker-bind-security.sh passed bash e2e/docker-bind-security.sh --base-image headroom-t3283-base --head-image headroom-t3283-head --fixture tests/fixtures/docker-bind-exposure.json --base-ref origin/main base_ref=origin/main ... runtime_slim_config=loopback rust_core=loaded local_completion=200 attacker_loopback=000 ws_status=403 docker bind security: fixture forwarding, loopback isolation, built-image health, slim-image loopback config, and token controls passed ``` ## Real Behavior Proof - Environment: Windows, Docker Desktop 29.6.2, current `origin/main` base, head image, and a real Docker bridge peer. - Exact command / steps: run the focused security/runtime tests, Compose configuration checks, then `bash e2e/docker-bind-security.sh` against the current base and head image definitions. - Observed result: `76 passed, 1 skipped`; the built-image harness passed base/head health, local completion, non-loopback isolation, authenticated public HTTP/WebSocket controls, loopback runtime-slim configuration, and direct Rust-core loading. - Not tested: runtime-slim proxy service health on this host, because its unrelated ONNXRuntime import path exits with code 139; the full hosted matrix and a future published GHCR image remain CI-owned. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Unsafe override required: no - Minimum rollout channel: next release containing the Docker and launcher changes - Stable/default behavior changed: published images and maintained local Docker launchers now use loopback-safe publication defaults - Kill switch / disable path: revert the deployment-artifact changes before release; deliberate public deployments remain explicitly operator-controlled - Qualification impact: Docker users relying on all-interface default publication must opt into public publication and configure the existing proxy token - Deliberate public deployment: use an explicit public host publication, set `HEADROOM_PROXY_TOKEN`, and send the matching bearer or `X-Headroom-Proxy-Token` credential - Local deployment: use the maintained loopback publication and no mandatory token - Rollback path: revert the deployment-artifact and documentation changes together ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove the behavior is effective - [x] Focused tests pass locally with these changes - [x] I did **not** edit `CHANGELOG.md`; release-please generates release notes from the conventional PR title ## Additional Notes The issue fixture remains provenance for the unauthenticated publication. Native `headroom proxy` defaults and the application-level authentication owners are unchanged. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
11c7320bbc |
fix(install): wait for the old process to stop before restarting (#3670)
## Description `headroom install restart` reports success but leaves the deployment stopped. Stopping the supervisor returns before the old proxy finishes its graceful shutdown, so `/readyz` still answers. `_start_deployment` treats a ready endpoint as "already running" and returns without starting anything. The old process then exits and nothing replaces it. `install start` works afterwards because by then nothing is answering. Restart now waits (bounded, 15s) for the health endpoint to go down before starting, and fails instead of reporting success if it never does. Closes #3658 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Add `_wait_until_stopped` in `headroom/cli/install.py` and call it between `_stop_deployment` and `_start_deployment` in `install restart`. - Add two regression tests: old process lingers after stop (restart must wait, then start), and old process never stops (restart must fail without starting or reporting success). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text python -m pytest tests/test_cli/test_install_cli.py -q 42 passed python -m ruff check headroom/cli/install.py tests/test_cli/test_install_cli.py -> All checks passed python -m ruff format --check (same files) -> already formatted Without the source change, test_install_restart_waits_for_old_process_before_starting fails (1 failed). ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.14.3. The issue was reported on macOS (launchd), which I could not run. - Exact command / steps: the unit tests above simulate the old process still answering `/readyz` after stop. - Observed result: restart waits for the endpoint to go down, then starts; it errors if the endpoint never goes down. - Not tested: a real launchd / systemd restart; mypy; the wider test suite beyond `test_install_cli.py`; the docker and task presets. The root cause is inferred from the code path and the issue's symptoms, not reproduced on macOS. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a - Stable/default behavior changed: `install restart` now waits up to 15s for the old process to stop, and errors if it does not. - Kill switch / disable path: n/a - Unsafe override required: no - Qualification impact: none - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes Docs: not applicable (no user-facing docs describe restart timing). Full unit suite not run locally; only `test_install_cli.py`. |
||
|
|
21accecc98 |
fix(install): preserve host file ownership with rootless Podman (#3592)
## Description Rootless Podman versions that honor an image's `USER root` despite `--userns=keep-id` create bind-mounted files as a subordinate host UID. A newly created mode-600 state file then becomes unreadable to the host CLI. Reproduced with Podman 5.4.2 and host UID 1007. Keep `--userns=keep-id` and also select the invoking user's UID/GID with `--user`. This is not the `--user`-without-keep-id mapping that caused #2804. The override is limited to Linux. Docker retains its existing user arguments, macOS Podman keeps its existing keep-id-only behavior, Windows remains unchanged, and the configured image is preserved. Fixes #3569 (the file-ownership defect). The wrapper-script autodetection side note is not changed; its existing environment override remains necessary. ## Type of Change - [x] Bug fix - [x] Documentation update ## Changes Made - Apply the existing host UID/GID selection to Linux Podman as well as Docker, after selecting Podman's keep-id namespace. - Replace the old keep-id-only assertion with a 12-case platform/UID/GID/image matrix, including macOS and Windows regression coverage. - Document the Podman environment override and the limits of recovery for files already owned by another UID. ## Testing - [x] New tests fail before the fix and pass after it (4 cases). - [x] Final focused run (installation subsystem plus the PATH-sensitive MCP case): 162 passed, 6 skipped. - [x] `ruff check .` and `ruff format --check .`: passed. - [x] Repository pre-commit checks, including mypy: passed. - [x] `make ci-precheck-rust` with Rust 1.95.0: passed. - [x] Manual real-behavior testing performed. Full Python run: 12,148 passed, 595 skipped, one failure in `test_ordinary_install_does_not_adopt_serena` because the invocation did not put the venv CLI on PATH. The original checkout fails identically; adding `.venv/bin` to PATH makes it pass. The final focused run above includes this recheck and the platform guard added after the full run. This is a full run plus scoped revalidation, not a claim of one entirely green full-suite invocation. `git push` completed with the installed pre-push hook: the full `make ci-precheck` passed, including a Rust 1.95.0 extension rebuild, the required Python regression set, and commitlint. No hooks were bypassed. ### Test Output ```text # Full Python run (PATH-sensitive failure explained above): 1 failed, 12148 passed, 595 skipped, 6143 warnings in 931.68s # Final installation subsystem and corrected-PATH MCP recheck: 162 passed, 6 skipped in 0.52s # Installed pre-push hook's Python regression set: 175 passed, 4 skipped, 1 warning in 4.09s ✅ ci-precheck PASSED — safe to push. ``` ## Real Behavior Proof - Environment: Debian 13.4, Python 3.13.5, rootless Podman 5.4.2, ordinary user UID/GID 1007. This ran inside a disposable privileged Docker Desktop Linux container to permit nested user namespaces, with no host data directories mounted. Podman used VFS, disabled cgroups and no network. `podman info` reported rootless=true. No provider/model was used. - Exact command / steps: Import the Debian rootfs with `USER root`, run the standalone command below, repeat with `--user 1007:1007`, then execute the real `build_runtime_command` output twice with root and nonroot fixture images. The initial standalone reproduction was: ```sh podman --storage-driver=vfs run --rm --network=none --cgroups=disabled \ --userns=keep-id -v /home/tester/proof:/proof localhost/root-fixture \ sh -c 'id; umask 077; echo written > /proof/state; cat /proc/self/uid_map' stat -c '%u:%g %a %n' /home/tester/proof/state cat /home/tester/proof/state ``` - Observed result: Before the fix, process UID 0, file owner `100000:100000`, mode 600, host read failed with Permission denied. Repeating with `--user 1007:1007` produced a host-readable file owned by `1007:1007`. The end-to-end follow-up called the real `build_runtime_command(DeploymentManifest(...))` and executed its returned command through a docker→podman symlink. The fixture image's entrypoint created a private file/directory and appended to a pre-existing private file. The original source failed the ownership assertion. After the patch, two consecutive runs produced: ```text uid=1007(tester) gid=1007(tester) groups=1007(tester) 0 probe-state 1007 1007 0o600 0 existing 1007 1007 0o600 0 probe-cache 1007 1007 0o700 uid=1007(tester) gid=1007(tester) groups=1007(tester) 1 probe-state 1007 1007 0o600 1 existing 1007 1007 0o600 1 probe-cache 1007 1007 0o700 PASS new files, private directories, append and restart ``` The same end-to-end probe passed with a custom image declaring `USER 1000:1000`, while the host remained UID 1007. Test files left with incorrect ownership by the red run were removed before the green run; this patch does not repair existing ownership. - Not tested: the production Headroom image/proxy startup, Arch/SELinux, rootful Podman, newer Podman releases, or Windows runtime behavior. Python unit/lint checks ran on macOS with Python 3.12.14. Current Podman documentation describes different default USER behavior; the explicit process user is needed for the reproduced 5.4.2 behavior. ## Runtime Rollout Safety - Rollout-managed feature(s): None; this is an installer command-generation bug fix. - Minimum rollout channel: No feature rollout channel applies; use a build containing this patch. - Stable/default behavior changed: Linux Podman receives an explicit invoking UID/GID alongside keep-id. Image names and configuration schemas are unchanged. - Kill switch / disable path: Stop the persistent deployment with `headroom install stop --profile <profile>`; there is no new feature flag. - Unsafe override required: None for this fix. The existing HEADROOM_CONTAINER_RUNTIME=podman override is still needed when a wrapper script cannot be detected. - Qualification impact: Real rootless Podman 5.4.2 file-ownership proof and platform regressions passed; other runtime environments listed above remain untested. - Rollback path: Revert this patch and recreate/restart the deployment. No data migration is involved; the original ownership problem can recur. Files already owned by a subordinate UID require separate recovery. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] Style guidelines followed; hard-to-understand mapping behavior documented - [x] Corresponding documentation updated - [x] Regression tests added - [x] No `CHANGELOG.md` edits --------- Co-authored-by: William Echo <175406538+qinghuanandejiangshi@users.noreply.github.com> |
||
|
|
3488f8d4b5 |
fix(install): use --userns=keep-id under Podman so bind-mount writes don't fail (#2846)
## Description
`build_runtime_command` unconditionally adds `--user <uid>:<gid>` on
non-Windows hosts:
```python
# headroom/install/runtime.py
if not _is_windows():
getuid = getattr(os, "getuid", None)
getgid = getattr(os, "getgid", None)
if callable(getuid) and callable(getgid):
command.extend(["--user", f"{getuid()}:{getgid()}"])
```
That is correct for Docker, where container UIDs equal host UIDs, but
wrong for rootless Podman, where the host user is already mapped to
container UID 0 and the `/etc/subuid` range is mapped to container UIDs
1 and above. Passing `--user $(id -u):$(id -g)` therefore selects a
container UID backed by a subordinate host UID that owns nothing. The
bind-mounted `~/.headroom` appears inside the container as `root:root`
and is unwritable, so every write fails:
```text
PermissionError: [Errno 13] Permission denied: '/tmp/headroom-home/.headroom/memories'
event=proxy_inbound_request_aborted path=/v1/messages reason=PermissionError
```
The proxy still starts and reports healthy, so the failure only surfaces
once a request touches a write path. As the reporter confirmed,
`--userns=keep-id` (or omitting `--user`) fixes it.
The fix detects Podman and uses `--userns=keep-id` instead of `--user`,
which maps the host user to the same UID inside the container and keeps
the bind mounts writable. Docker still gets `--user`, unchanged.
Detection is subprocess-free: it resolves the `docker` binary and checks
its real name for the common `docker -> podman` symlink shim (e.g. NixOS
`/run/current-system/sw/bin/docker -> podman`), with an explicit
`HEADROOM_CONTAINER_RUNTIME` (`podman` / `docker`) override for setups
the symlink heuristic cannot see, such as a wrapper script.
Fixes #2804
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/install/runtime.py`: added `_container_runtime_is_podman()`
(env override, then a `docker`-binary realpath basename check, no
subprocess). In `build_runtime_command`, when Podman is detected the
command uses `--userns=keep-id` instead of `--user <uid>:<gid>`.
- `tests/test_install/test_runtime.py`: pinned the existing docker test
to the Docker path via `HEADROOM_CONTAINER_RUNTIME=docker` and asserted
`--userns=keep-id` is absent there; added
`test_build_runtime_command_podman_uses_keep_id_not_user` asserting the
Podman path drops `--user` and adds `--userns=keep-id`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
# Fail-before (source fix stashed, new test kept):
tests/test_install/test_runtime.py::test_build_runtime_command_podman_uses_keep_id_not_user FAILED
assert "--userns=keep-id" in command
AssertionError: assert '--userns=keep-id' in ['docker', 'run', '--rm', ...]
# Pass-after (fix applied):
tests/test_install/test_runtime.py 26 passed
# Broader install suite (excluding the pre-existing env-specific PowerShell installer test):
tests/test_install/ 142 passed, 1 skipped
# uvx ruff@0.15.17 check -> All checks passed!
# uvx mypy@1.20.2 headroom/install/runtime.py -> Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.15.17 and mypy 1.20.2 via uvx.
- Exact command / steps: confirmed `build_runtime_command` adds `--user`
unconditionally on non-Windows, then drove both runtimes
deterministically via the `HEADROOM_CONTAINER_RUNTIME` override.
Fail-before with `git stash push headroom/install/runtime.py` and
`python -m pytest tests/test_install/test_runtime.py -k
podman_uses_keep_id` (the command still carries `--user`, no keep-id),
pass-after with `git stash pop` and rerunning the file (26 passed).
- Observed result: with Podman detected the docker command now contains
`--userns=keep-id` and no `--user`/`1000:1001`, matching the
`--userns=keep-id` invocation the reporter verified writes successfully;
with Docker it is unchanged (`--user 1000:1001`, no keep-id).
- Not tested: a live rootless-Podman deployment writing to a bind mount
(no Podman in this environment). The command construction is verified
directly, and `--userns=keep-id` is the documented, reporter-confirmed
switch for the rootless-Podman ID-mapping.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`: it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
Detection is intentionally subprocess-free and conservative: it only
diverges from today's behavior when the `docker` binary literally
resolves to a `podman`-named target, or when
`HEADROOM_CONTAINER_RUNTIME` is set. Real Docker installs are untouched.
The override also gives a clean escape hatch in both directions if a
given host's symlink layout hides the runtime. This is the `--user` half
of the persistent-docker + Podman issues; the separate host-memory-path
problem (#2803) is addressed in its own PR.
|
||
|
|
045f3dfe6f |
fix(install): use CREATE_NO_WINDOW instead of DETACHED_PROCESS on Windows (#2527)
## Description On Windows, the detached agent process spawned by `install hook ensure` (and the `install restart` self-spawn) pops up a visible black console window repeatedly, because `DETACHED_PROCESS` makes `CREATE_NO_WINDOW` a no-op per the Win32 process-creation-flags docs. #2521 ## Type of Change - [x] Bug fix (non-breaking change which fixes an issue) ## Changes Made - `headroom/install/runtime.py`: `start_detached_agent()` now uses `CREATE_NO_WINDOW` instead of `DETACHED_PROCESS`, combined with `CREATE_NEW_PROCESS_GROUP` (unchanged detach/isolation semantics, window hidden). - `headroom/install/runtime.py`: `_spawn_detached_restart()` now also sets `CREATE_NO_WINDOW` on Windows (previously had no `creationflags` at all on that platform). - `tests/test_install/test_runtime.py`: updated the Windows branch of `test_start_detached_agent_and_run_foreground` to assert the actual `creationflags` value passed to `Popen`, instead of just monkeypatching an unused `DETACHED_PROCESS` attribute. ## Testing - [x] Added/updated tests - [x] Ran full local test suite ``` $ python -m pytest tests/test_install -q ======================= 137 passed, 1 skipped in 48.68s ======================= $ ruff check headroom/install/runtime.py tests/test_install/test_runtime.py All checks passed! $ ruff format --check headroom/install/runtime.py tests/test_install/test_runtime.py 2 files already formatted $ mypy headroom --ignore-missing-imports Success: no issues found in 506 source files ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.13.11, headroom repo local checkout - Exact command / steps: `python -m pytest tests/test_install/test_runtime.py -q`, plus manual read of `subprocess` Windows creation-flag semantics (`DETACHED_PROCESS` + child console allocation vs `CREATE_NO_WINDOW`) - Observed result: all 25 tests in `test_runtime.py` pass, including the updated assertion that `creationflags == CREATE_NO_WINDOW | CREATE_NEW_PROCESS_GROUP` on the Windows code path - Not tested: did not reproduce the original visible-console-popup repro end-to-end via live Claude Code hook invocation (no environment with the full hook-triggered respawn loop set up in this session); relying on the Win32 docs and the reporter's own local verification of the same flag swap ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
560ffae103 |
feat(deploy): Add turnkey deploy command (#1404)
## Description Adds `headroom deploy` as the turnkey, zero-config local deployment entrypoint. The command chooses the most capable deployment path it can verify on the current host, configures detected tools through the existing persistent-install machinery, starts the proxy, and preserves the existing rollback behavior if an update fails. The selection order favors performance first: NVIDIA Docker GPU passthrough when `nvidia-smi` and Docker's NVIDIA runtime are available, then plain Docker, then native scheduled recovery, then a detached Python runtime fallback. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [x] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - Added the top-level `headroom deploy` command and reused the existing install manifest/apply/start/rollback path. - Added conservative runtime selection for GPU Docker, plain Docker, native schedulers, and detached Python fallback. - Added Docker runtime support for manifest-driven `--gpus all` passthrough. - Added tests for Docker selection, GPU Docker selection, detached fallback, GPU command rendering, and subprocess wrapper compliance. - Updated README and persistent-install docs to present the turnkey deployment flow and performance-first GPU behavior. - Allowed documented `opencode` targets through `headroom install apply --target`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Formatting passes (`ruff format --check`) - [x] Type checking passes in local pre-commit and CI - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text uv run --with pytest --with pytest-asyncio python -m pytest tests/test_cli/test_subprocess_utf8_encoding.py tests/test_cli/test_install_cli.py tests/test_install/test_runtime.py::test_build_runtime_command_for_docker_includes_gpu_passthrough tests/test_install/test_planner.py -q 47 passed in 1.57s uvx --from ruff==0.15.17 ruff check headroom/cli/install.py headroom/install/runtime.py tests/test_cli/test_install_cli.py tests/test_install/test_runtime.py --output-format concise All checks passed! uvx --from ruff==0.15.17 ruff format --check headroom/cli/install.py headroom/install/runtime.py tests/test_cli/test_install_cli.py tests/test_install/test_runtime.py 4 files already formatted ``` GitHub checks are green on the current head. ## Real Behavior Proof - Environment: Windows local worktree `C:\git\headroom`, Python 3.13 via `uv`; GitHub Actions Ubuntu/macOS/Windows runners for full PR CI. - Exact command / steps: Ran the focused deploy/install tests above, checked the touched Python files with the CI-pinned Ruff version, and confirmed the current PR head is mergeable with green GitHub checks. - Observed result: The deploy command, runtime selection, Docker GPU command rendering, install CLI behavior, and subprocess encoding coverage all pass locally; the branch is no longer conflicted. - Not tested: Actual RTX 4090 hardware passthrough on a physical NVIDIA workstation; the PR tests conservative detection and Docker command rendering without requiring GPU hardware in CI. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A - CLI/runtime behavior only. ## Additional Notes CHANGELOG update is not included because this is an unreleased feature PR and the repository's release tooling owns release notes from conventional commits. |
||
|
|
e5b3a634df |
fix(proxy): dedupe Codex WS request logging for accurate mixed-provider dashboards (#2189)
## Description Running Claude Code (Anthropic) and Codex (OpenAI) against the **same** Headroom proxy instance on one port produced incorrect, unstable dashboard data. The proxy core is provider-isolated and multi-provider-safe by design; the defect was in the observability layer. The Codex `/v1/responses` **WebSocket** handler was the only path in the proxy that wrote to the request logger by hand instead of through the unified `emit_request_outcome` funnel, and it did so twice per session close: the per-turn funnel record plus an unconditional cumulative session-summary `RequestLog`. This PR removes the duplicate summary log so Codex WS emits exactly one request log per turn, matching the HTTP provider paths. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Dropped the duplicate cumulative session-summary `RequestLog` in the Codex WS handler while preserving the per-turn `emit_request_outcome` path. - Preserved gated `request_messages` and `turn_id` on residual outcomes so dashboard telemetry keeps the useful attribution without double-counting tokens. - Ensured explicit `--anyllm-provider` wins over a leaked `HEADROOM_ANYLLM_PROVIDER` environment variable. - Registered retry delay settings that had drifted out of the settings registry. - Hardened tests against developer-shell `HEADROOM_*` / `ANTHROPIC_CUSTOM_HEADERS` leakage and stabilized several focused proxy/wrap test fixtures. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ .venv/bin/pytest tests/ -q -p no:cacheprovider 8529 passed, 537 skipped, 5831 warnings in 276.25s (0:04:36) $ .venv/bin/ruff check <touched files> All checks passed! ``` ## Real Behavior Proof - Environment: macOS (Darwin 25.4.0), Python 3.13.14, pytest 9.0.3, ruff via project venv, branch `fix/multi-provider-runtime`. - Exact command / steps: Ran the full test suite without pytest cache provider and Ruff on all touched files; used `git stash` to confirm the stale fake-config failures pre-existed this change. - Observed result: Full suite passed with no failures; Ruff passed; Codex WS now routes end-of-session logging through `emit_request_outcome`, emitting one request log per turn with the same accounting model as Anthropic HTTP turns. - Not tested: Live simultaneous Claude + Codex dashboard run. `mypy headroom` was not run to completion; a scoped run reported one pre-existing `settings_store.py:470` coercion error outside this diff. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A - server-side observability fix; no UI markup changed. ## Additional Notes - The proxy's multi-provider routing, header/auth isolation, and per-model cache keying are already correct and unchanged here; only the WS observability write path was double-counting. - Architectural assessment: `plans/reports/research-260714-0004-multi-provider-upstream-compression-report.md`; root-cause + resolution trail: `plans/reports/debug-assessment-260714-0011-dashboard-instability-mixed-claude-codex-report.md`. - No live simultaneous Claude + Codex dashboard run was performed; validation is from test coverage and code review of the WS logging path. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
c3db8e47f8 |
fix(install): default docker image to headroomlabs-ai GHCR registry (#1867) (#2039)
## Description
The GitHub repository was transferred from `chopratejas/headroom` to
`headroomlabs-ai/headroom`. GitHub 301-redirects transferred repos for
web and git operations, but **GitHub Container Registry (GHCR) does
not** — the old package `ghcr.io/chopratejas/headroom` is now orphaned
and frozen (its `latest` tag stopped advancing at `0.27.0`), while CI
publishes new images to `ghcr.io/headroomlabs-ai/headroom` (the workflow
derives the path from `${{ github.repository }}`).
Headroom's install tooling still defaulted to the dead path, so
`headroom install apply --preset persistent-docker`, `headroom init`,
and the standalone install scripts all pulled a stale `0.27.0` image
instead of the current release.
This changes every docker-image **default** to
`ghcr.io/headroomlabs-ai/headroom:latest`.
Closes #1867
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/install/models.py` — `InstallManifest.image` default.
- `headroom/cli/install.py` — `--image` Click option default.
- `headroom/cli/init.py` — two `InstallManifest(...)` image args.
- `scripts/install.sh` / `scripts/install.ps1` — `IMAGE_DEFAULT` /
`$ImageDefault` plus the `--image` help-text default.
- `docker/docker-compose.native.yml` — image default in both services.
- `tests/test_install/test_planner.py` (4) and
`tests/test_install/test_runtime.py` (5) — updated the assertions that
pinned the old image (including `assert "<image>" in command`), so they
now verify the corrected registry threads through the planner and docker
runtime command.
Scope note: `github.com/chopratejas/...` links and the plugin
marketplace slug are intentionally **not** changed — GitHub redirects
those, so they still work. Only the genuinely-dead GHCR image references
are touched. No new dependency, no new abstraction.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ uv run pytest tests/test_install/ -q
99 passed, 1 skipped in 48.34s
$ uv run ruff check headroom/install/models.py headroom/cli/install.py headroom/cli/init.py
All checks passed!
$ grep -rn "ghcr.io/chopratejas/headroom" headroom/ scripts/ docker/ tests/
# (no source matches — every default now points at headroomlabs-ai)
```
The install-test assertions pinned the old image, so they fail against
the old defaults and pass after the fix — they are the regression guard.
## Real Behavior Proof
- **Environment:** Windows 11, Python 3.13.5, headroom installed from
this branch (editable, via `uv`).
- **Exact command / steps:** The dead-registry claim is verifiable at
the registry level, independent of a release:
```text
docker pull ghcr.io/chopratejas/headroom:latest # old default → 0.27.0
(frozen / orphaned)
docker pull ghcr.io/headroomlabs-ai/headroom:latest # new default →
current release
```
And the install pipeline now emits the correct image (covered by
`tests/test_install/test_runtime.py`, which asserts the resolved docker
command contains `ghcr.io/headroomlabs-ai/headroom:latest`).
- **Observed result:** `grep` confirms no `ghcr.io/chopratejas/headroom`
default remains in code, scripts, compose, or tests;
`tests/test_install/` is green (99 passed) with the corrected image
asserted end-to-end through planner → runtime command.
- **Not tested:** A live `docker pull` of both tags on this specific
machine (no local Docker daemon guaranteed) — the registry difference is
reproducible by anyone running the two `docker pull` commands above; and
an end-to-end `headroom install apply --preset persistent-docker`
against a real Docker host.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Screenshots (if applicable)
N/A
## Additional Notes
- Documentation checklist item is N/A — no user-facing docs surface
beyond the CHANGELOG entry.
- `mypy` left unchecked — not run as part of this verification; the
change is a string-default swap with no type-level surface.
- The `--image` help text and `docker-compose.native.yml` were included
so every user-facing default is consistent; only the dead GHCR image
string was changed.
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
|
b097ef3e25 |
fix(install): don't let host env override the manifest in persistent-docker (#2090)
## Description In persistent-docker deployments a stale host env var can silently override the value the deployment manifest pinned for the container. `build_runtime_command` builds the `docker run` argv in two passes: 1. It emits the manifest's pinned env as `--env NAME=VALUE` (from `base_env` plus the deployment env). 2. It then walks `os.environ` and, for every name matching a `PASSTHROUGH_ENV_PREFIXES` prefix, appends a bare `--env NAME` so the host value is forwarded into the container. A manifest-pinned name and a host-exported name can collide when they share a passthrough prefix. `HEADROOM_BACKEND` is the clearest case: the manifest pins `--env HEADROOM_BACKEND=anthropic` in pass 1, and pass 2 also matches the `HEADROOM_` prefix and appends a bare `--env HEADROOM_BACKEND`. Docker resolves duplicate `--env` flags last-wins, and the bare passthrough comes last, so a stale host export `HEADROOM_BACKEND=anyllm` wins and the container runs a different backend than its deployment config says. `start_persistent_docker` runs the resulting command through `subprocess.run` with the parent process environment, so whatever the operator happened to have exported leaks in and overrides the manifest. The fix skips the bare passthrough for any name the manifest already pins, so the pinned value stands while unrelated host secrets (API keys and so on) are still passed through as before. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/install/runtime.py`: skip the bare `--env NAME` passthrough when `NAME` is already pinned by the manifest (`and name not in runtime_env`). - `tests/test_install/test_runtime.py`: add `test_build_runtime_command_docker_manifest_env_beats_host_passthrough`, which exports a conflicting `HEADROOM_BACKEND` and asserts the command keeps the manifest value and emits no bare passthrough for it. - `CHANGELOG.md`: Bug Fixes entry. ## Testing - [ ] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uvx ruff@0.15.17 format headroom/install/runtime.py tests/test_install/test_runtime.py 2 files left unchanged $ uvx ruff@0.15.17 check headroom/install/runtime.py tests/test_install/test_runtime.py All checks passed! $ python -m py_compile headroom/install/runtime.py tests/test_install/test_runtime.py OK ``` ## Real Behavior Proof - Environment: local checkout, Python 3.11, `uvx ruff@0.15.17`. - Exact command / steps: ran a standalone script that reproduces the two-pass argv build and models Docker's duplicate `--env` last-wins resolution, with the manifest pinning `HEADROOM_BACKEND=anthropic` and the host exporting `HEADROOM_BACKEND=anyllm`. - Observed result: the old build resolves the effective `HEADROOM_BACKEND` to the host value `anyllm` (bare passthrough wins); the new build keeps the manifest value `anthropic` and emits no bare `HEADROOM_BACKEND` token, while a non-pinned passthrough (`ANTHROPIC_API_KEY`) is still forwarded. - Not tested: I did not run the full `pytest` suite locally because it pulls in the ML stack; the new regression test is left for CI. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes The "unit tests pass locally" and "type checking" boxes are unchecked because the full suite imports the ML dependencies, which I can't run in this environment; the change is a pure function over `build_runtime_command`, verified by the standalone proof above and covered by the new regression test for CI. Co-authored-by: Tejas Chopra <chopratejas@gmail.com> |
||
|
|
1d2b76e72e |
fix: harden persistent install startup (#1851)
## Description Hardens persistent install startup and proxy compression behavior for issue #1843. Repeated `headroom install start` / scheduled ensure calls no longer spawn duplicate runtimes by default, and `/v1/compress` now fails open on compression timeout instead of returning a 503. The PR also adds a machine-readable platform feature matrix and app-level stabilization tests for health, compression functionality, timeout behavior, and matrix evidence. Refs #1843 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [x] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Wrapped direct persistent deployment starts with the existing profile-local runtime start lock. - Made `headroom install start` idempotent when the deployment is already healthy. - Added wedged-runtime handling: if a PID is running but `/readyz` does not recover inside the grace window, stop it before starting again. - Kept `install agent ensure` inside the already-held lock while delegating to the shared start helper. - Changed `/v1/compress` timeout behavior from `503 compression_timeout` to fail-open `200` with original messages, `compression_skipped: true`, and `skip_reason: compression_timeout`. - Added `tests/test_platform_stabilization_functional.py` covering real FastAPI health/compression routes, successful compression metrics, timeout fail-open speed, and a real JSON tool payload that reduces tokens. - Added `docs/platform-feature-matrix.json` and `docs/platform-stabilization.md` for Linux/macOS/Windows hardening coverage and known gaps. - Strengthened matrix tests so cited local test/workflow paths must exist. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # Clean source tree without copied Rust extension: functional module is skipped locally, as CI copies _core from the built wheel. > python -m pytest tests/test_install tests/test_cli/test_install_cli.py tests/test_platform_feature_matrix.py tests/test_platform_stabilization_functional.py -q collected 120 items / 1 skipped 119 passed, 2 skipped in 17.08s > python -m ruff check headroom/proxy/handlers/openai.py tests/test_platform_stabilization_functional.py tests/test_platform_feature_matrix.py All checks passed! # Local Windows compiled-core proof: > python -m maturin build --profile ci --out dist-local Built wheel for abi3 Python >= 3.10 to dist-local\headroom_ai-0.29.0-cp310-abi3-win_amd64.whl # Copied _core.pyd from the wheel into headroom/ for local route execution, then: > python -m pytest tests/test_platform_stabilization_functional.py -q collected 4 items 4 passed in 6.71s > python -m pytest tests/test_install tests/test_cli/test_install_cli.py tests/test_platform_feature_matrix.py -q collected 120 items 119 passed, 1 skipped in 17.16s Commit hooks: Sync plugin versions.....................................................Passed check for merge conflicts................................................Passed ruff.....................................................................Passed ruff-format..............................................................Passed mypy.....................................................................Passed ``` ## Real Behavior Proof - Environment: Windows 11, PowerShell, Python 3.13.13, worktree `C:\git\headroom-stabilization` on branch `jd/cross-platform-stabilization`. - Exact command / steps: built the Windows wheel with `maturin`, extracted `_core.pyd`, ran the new FastAPI route tests and install/matrix tests listed above, then removed generated artifacts before committing. - Observed result: direct start paths now no-op when healthy, skip spawning when the start lock is contended, and stop a wedged runtime before restart. `/v1/compress` now returns original messages quickly on timeout instead of a 503. The real JSON tool-payload smoke test returns `tokens_before > tokens_after`, `tokens_saved > 0`, `compression_ratio < 1.0`, and non-empty transforms through the public route. - Not tested: full native Windows persistent process e2e remains blocked by the upstream CRT/wheel issue already documented in workflows and in the matrix. No real OS service was installed locally; service manager behavior is covered by argument-level unit tests. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes CHANGELOG is not updated because this is an unreleased hardening/test/documentation pass. The platform matrix intentionally records partial/blocked Windows/macOS e2e gaps instead of claiming full coverage where the repo cannot currently run it. |
||
|
|
816cb85fa8 |
fix(install): close parent log fd in start_detached_agent (#1576)
## Description
`start_detached_agent()` opens the agent log file and hands it to
`subprocess.Popen` as `stdout`/`stderr`, then returns the process
**without
closing the parent's copy of the file descriptor**. The child inherits
the fd
and writes to it, but the parent keeps its own copy open forever.
The result: every `headroom install start` leaks one file descriptor in
the
parent, and the leaked handle pins the log file open so it can't be
rotated.
On a tight `ulimit` or inside a container, repeated starts can walk
straight
into the fd limit.
```python
# headroom/install/runtime.py — before
log_file = open(log_file_path, "a", encoding="utf-8", errors="replace")
kwargs = {"stdout": log_file, "stderr": log_file, ...}
return subprocess.Popen(command, **kwargs) # parent's log_file never closed
```
The fix closes the parent's copy in a `try/finally` right after `Popen`
returns:
```python
try:
proc = subprocess.Popen(command, **kwargs)
finally:
# The child has inherited the log file descriptor, so the parent's
# copy is dead weight. Closing it (even when Popen raises) avoids
# leaking one fd per `headroom install start` and lets the log file
# be rotated.
log_file.close()
return proc
```
The `finally` is deliberate: it also covers the case where `Popen`
itself
raises (bad executable, fork failure), which would otherwise leak the
just-opened handle. This matches the `with open(...)` pattern already
used by
`run_foreground()` a few lines above.
Closes #1554
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/install/runtime.py`: close the parent's log file descriptor
in a `try/finally` after `subprocess.Popen` in `start_detached_agent()`,
so it is released on the normal path and when `Popen` raises.
- `tests/test_install/test_runtime.py`: add two regression tests — one
for a normal start, one for `Popen` raising — asserting the parent's log
handle is closed afterwards.
- `CHANGELOG.md`: Bug Fixes entry under Unreleased.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
Before the fix (`runtime.py` reverted to its parent commit, new tests
kept) —
the assertion inspects the *actual* log file handle and finds it still
open:
```text
E AssertionError: assert False is True
E + where False = <_io.TextIOWrapper name='...\deploy\demo\runner.log' mode='a' encoding='utf-8'>.closed
FAILED tests/test_install/test_runtime.py::test_start_detached_agent_closes_parent_log_fd
FAILED tests/test_install/test_runtime.py::test_start_detached_agent_closes_log_fd_when_popen_raises
============================== 2 failed in 0.43s ==============================
```
After the fix:
```text
tests\test_install\test_runtime.py ................
====================== 16 passed, 1 deselected in 0.35s =======================
```
(The one deselected test,
`test_runtime_start_lock_blocks_another_process`, is a
pre-existing failure on my Windows box — it fails identically on a clean
checkout of `main` and is unrelated to this change.)
## Real Behavior Proof
- Environment: Windows 11, Python 3.12.11, headroom built from this
branch (`uv sync --extra dev`).
- Exact command / steps: ran the two regression tests, which drive the
real `start_detached_agent` code path — real `open()`, real
`Popen(stdout=...)`, real (or missing) `close()`. Only
`subprocess.Popen` is stubbed so the test never launches an actual
detached agent; the fd-lifecycle bug lives entirely in how the parent
handles its own handle, and that runs for real.
- Observed result: the log file handle the parent passed to `Popen` is
`.closed == False` before the fix and `.closed == True` after — for both
the normal path and the `Popen`-raises path (output above).
- Not tested: I intentionally did not spin up many real detached agents
to watch the OS fd table grow — on Windows that means flashing console
windows and isn't a clean signal anyway. The handle-state assertion on
the real file object is the deterministic equivalent. Did not run the
full `mypy headroom` pass (one-line lifecycle change, no new types).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
- Single logical change, no new dependencies, default behavior otherwise
unchanged.
- Rebased/merged latest `main` to clear a `CHANGELOG.md` conflict.
- @chopratejas this is a sibling of #1555 (the `wrap.py` readiness-loop
handle leak you've got filed). I scoped this PR to #1554 only; happy to
follow up on #1555 separately if you'd like.
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
6b227b9c90 |
fix(install): use Windows-safe PID liveness probe in runtime_status (#1544) (#1560)
## Description `headroom install status` crashed with `OSError: [WinError 87] The parameter is incorrect` on Windows and, worse, tore down the live proxy it was only meant to inspect. `runtime_status()` probed liveness with a bare `os.kill(pid, 0)` guarded only by `except OSError`. Against a detached Windows agent (`DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP`), that call raises WinError 87, which CPython surfaces as a `SystemError` — not an `OSError` — so it escaped the handler, crashed status, and left the deployment dead (PID file removed, port 8787 freed). This mirrors the `os.kill`/`SystemError` fix PR #1315 applied to `cli/wrap.py`. Closes #1544 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added a shared Windows-safe `headroom._subprocess.pid_alive()` helper: rejects non-positive PIDs, prefers `psutil.pid_exists()`, and treats `SystemError` (WinError 87) as "not alive". - `install/runtime.py` `runtime_status()` now delegates to `pid_alive()` instead of an unguarded `os.kill(pid, 0)`. - `install/runtime.py` `stop_runtime()` now also catches `SystemError` to avoid the same crash class on shutdown. - `cli/wrap.py` `_pid_alive()` now delegates to the shared helper, so the marker-cleanup path and the install/runtime status path share one liveness probe (the shared helper the issue asked for). - Added regression tests for the helper and `runtime_status`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ ruff check . All checks passed! $ ruff format --check headroom/_subprocess.py headroom/install/runtime.py headroom/cli/wrap.py tests/test_install/test_runtime.py tests/test_pid_alive.py 5 files already formatted $ mypy headroom --ignore-missing-imports (exit 0) $ pytest tests/test_pid_alive.py tests/test_install/test_runtime.py tests/test_cli/test_wrap_helpers.py tests/test_cli/test_wrap_persistent.py \ --deselect "tests/test_install/test_runtime.py::test_runtime_start_lock_blocks_another_process" 89 passed ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.13, ruff 0.15.17 / mypy 1.20.2 / pytest 9.1.0, psutil 7.2.2, branch `fix/1544-windows-pid-liveness`. - Exact command / steps: ran the four checks above; the new `tests/test_pid_alive.py` injects a `SystemError` (simulated WinError 87) and a stubbed `psutil` to drive both code paths, and `test_runtime_status_*` exercise `runtime_status()` end to end with a PID file present. - Observed result: `runtime_status` returns `"running"` for a live PID without sending any signal (asserted), returns `"stopped"` instead of crashing when the probe raises `SystemError`, and the helper only ever passes signal `0`. All 89 targeted tests pass; ruff/format/mypy clean. - Not tested: the full `headroom install apply --preset persistent-task` detached-agent reproduction against a live proxy was not run end to end; it is instead covered by the deterministic `SystemError`/WinError-87 injection regression tests. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - One pre-existing test, `test_runtime_start_lock_blocks_another_process`, fails on my local Windows checkout **before** these changes too (it asserts cross-process file-lock blocking and depends on `HOME` semantics that differ on Windows). It is unrelated to this fix and is deselected above; it passes on the Linux CI runners. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
feedead077 |
fix(install): stop duplicating ENTRYPOINT in persistent-docker runtime command (#833) (#1348)
## Description `headroom install apply --preset persistent-docker` pulls the image, starts the container, then fails after ~45s with "Deployment 'default' did not become ready after start." The rollback removes the container and manifest, leaving nothing running and no logs. Root cause: the published image already bakes the proxy invocation into its ENTRYPOINT (`Dockerfile`: `ENTRYPOINT ["headroom", "proxy"]`), but `build_runtime_command()` in `headroom/install/runtime.py` re-added `headroom proxy` after the image name. Docker concatenates ENTRYPOINT + args, so the container ran `headroom proxy headroom proxy --host 0.0.0.0 ...` and Click aborted with `Got unexpected extra arguments (headroom proxy)`. The runtime command now appends only the proxy flags after the image name, substituting the all-interface container bind host for the host pair carried in `proxy_args`. Closes #833 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/install/runtime.py`: drop the duplicated `headroom proxy` from the docker `build_runtime_command` output; append only `--host <bind> *proxy_args[2:]`. Extracted `CONTAINER_BIND_HOST` and `_PROXY_ARGS_HOST_PAIR_LEN` named constants. - `tests/test_install/test_runtime.py`: new regression test asserting the args appended after the image name never re-add the `headroom proxy` ENTRYPOINT. - `CHANGELOG.md`: Unreleased → Bug Fixes entry. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_install/ -q 91 passed, 1 skipped in 5.48s $ uv run ruff check headroom/install/runtime.py tests/test_install/test_runtime.py All checks passed! $ uv run mypy headroom/install/runtime.py Success: no issues found in 1 source file ``` #### RED → GREEN proof RED — new test with the prod fix reverted (test kept): ```text E AssertionError: container args re-add the ENTRYPOINT — got ['headroom', 'proxy', '--host', '0.0.0.0', '--port', '8787', '--backend', 'anthropic'] FAILED tests/test_install/test_runtime.py::test_build_runtime_command_for_docker_does_not_duplicate_entrypoint 1 failed in 0.17s ``` GREEN — with the fix applied: ```text tests/test_install/test_runtime.py::test_build_runtime_command_for_docker_does_not_duplicate_entrypoint 1 passed in 0.11s ``` ## Real Behavior Proof - Environment: Linux, Python 3.13, headroom @ this branch. - Exact command / steps: reproduce the exact concatenation Docker performs (ENTRYPOINT `headroom proxy` + the buggy CMD `headroom proxy --host 0.0.0.0 --port 8787`): ```text $ headroom proxy headroom proxy --host 0.0.0.0 --port 8787 Usage: headroom proxy [OPTIONS] Try 'headroom proxy --help' for help. Error: Got unexpected extra arguments (headroom proxy) ``` This is the exact error from the issue. After the fix, `build_runtime_command` appends only the flags after the image name: ```text args after image: ['--host', '0.0.0.0', '--port', '8787', '--backend', 'anthropic'] ``` so the container runs `headroom proxy --host 0.0.0.0 --port 8787 --backend anthropic` (ENTRYPOINT + flags) and Click accepts it. - Observed result: pre-fix Click aborts with the unexpected-arguments error (container crash-loops); post-fix the command line is valid. - Not tested: pulling and running the real `ghcr.io` image end-to-end (requires the published image + Docker host); the failure is fully determined by the generated argv, which is covered above and by the unit test. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes Scope is limited to the docker runtime command construction. The Python (`runtime_kind=python`) path was already correct and is unchanged. Screenshots N/A (CLI-only change). Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
9252d852c5 |
fix(init): guard persistent task startup (#616)
## Description Prevent `headroom init` hooks from spawning duplicate persistent-task runners while a proxy is still starting. Fixes #615 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Problem `_ensure_profile_running()` checked readiness for only one second and then launched `start_detached_agent()` whenever the proxy was not ready yet. When Claude/Codex hooks fired close together, each hook could race through that path and spawn another detached persistent-task runner. ## Changes Made - Add a profile-local, nonblocking runtime start lock around init hook startup. - Re-check readiness after acquiring the lock so late-arriving hooks do not start a duplicate runner. - If a runtime is already alive, wait up to 15 seconds for readiness before stopping and restarting it. - Add regression tests for lock contention, slow startup, and cross-process lock behavior. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ## Test Output ``` UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --frozen --extra dev --extra proxy pytest tests/test_cli/test_init_cli.py tests/test_cli/test_install_cli.py tests/test_cli/test_wrap_persistent.py tests/test_install/test_runtime.py # 89 passed in 0.61s UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --frozen --extra dev --extra proxy ruff check . # All checks passed! UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --frozen --extra dev --extra proxy ruff format --check . # 775 files already formatted UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --frozen --extra dev --extra proxy mypy headroom --ignore-missing-imports # Success: no issues found in 346 source files ``` Manual sandbox check: ``` # before this change: 3 ensure calls spawned 3 detached starts # after this change: 3 ensure calls spawned 1 detached start while the runtime was still starting ``` ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes Docs and CHANGELOG were left unchanged because this is a small runtime bug fix with no user-facing CLI/API change. |
||
|
|
efd2ac1ca4 |
chore: renormalize line endings to LF
`.gitattributes` declares `*.py text eol=lf` and `*.sh text eol=lf`, but 74 files (73 .py, 1 .sh) are stored in the index with CRLF line endings, violating that contract. Every macOS/Linux clone reports these files as "modified" on fresh checkout because git's diff engine sees the stored bytes don't match the attribute contract, even though the working tree and index match byte-for-byte. Running `git add --renormalize .` rewrites each affected blob so the stored form matches the attribute declaration. No semantic changes — every affected file's diff is "N insertions, N deletions" with inserts and deletes being the same lines modulo line endings. Follow-up commit adds `.git-blame-ignore-revs` so `git blame` / GitHub blame skip this mechanical commit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
d4574f4ae2 |
test: avoid global platform leaks in install tests
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
|
38bf3e639c |
test: expand coverage across helper slices
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
|
4a87753713 |
feat(docker): forward HEADROOM_WORKSPACE_DIR and HEADROOM_CONFIG_DIR into containers
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
bd242fc62d |
test: expand persistent install coverage
Add focused regression coverage for install, runtime, provider, state, health, supervisor, and persistent wrap flows so the new persistent deployment surfaces are exercised more thoroughly in CI. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
|
865bef2216 |
fix: harden persistent install wrappers and review gaps
Align Docker-native wrapper help and runtime behavior with the Python install contract, including persistent deployment metadata, baked install-image defaults, and explicit unsupported wrap targets. Harden the Python persistent-install path with profile validation, safer provider-scope handling, Windows environment restoration, runtime parity improvements, and rollback-safe apply/update behavior. Update README, Docker install docs, CI, and focused regressions to cover the Windows BOM failure, wrapper parity, compose coverage, and Docker-native wrap behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
|
21896a095c |
feat: add persistent install lifecycle management
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |