Commit Graph
33 Commits
Author SHA1 Message Date
JD DavisandJerrett Davis b235bc2d27 fix(ci): verify installer provenance in build workflows (#4677)
## Current maintainer status (2026-11-10, current-main refresh)

Published head: `db4bdd71e7ddc1a72316445ef0d16a7d90b62951`, containing
upstream main `a385ea8e60a14c1511d82b21d518421af35deab2` through a clean
mechanical merge. All 13 remediation files are byte-identical to
reviewed head `81ff3ed384afbd404c48fd79d6bed41e8d647a99`; no installer,
action metadata, or checksum selection changed.

Fresh focused Windows checks: 61 passed and 23 platform skips. Fresh WSL
offline bootstrap/hook fixtures: 23 passed, with one pytest
configuration warning. These local checks use fixtures and do not claim
live vendor, provider, or native GUI execution.

Exact-head CI, including Docker-native, and the Init, Wrap, Rust,
governance, changelog, and merge-conflict workflows succeeded. Security
remains FAILURE: the OpenCode npm audit reports two moderate findings
through Effect advisory GHSA-chp3-j5qr-2vfr. The security workflow and
affected OpenCode package manifest/lock are byte-identical to current
main, establishing an inherited dependency finding. This is not an
all-green result or a fresh audit of main. No workflow rerun, dependency
downgrade, or advisory override was performed on this branch.

Independent human review and integration remain required. No
self-approval or merge was performed. Other branch publication and
execution holds remain active. All older qualification below belongs to
its recorded historical heads.

---



## Description

Build validation jobs fetched mutable installer scripts, and the pinned
commitlint action still declared a mutable Docker image tag. This pins
that image digest, verifies workflow-tool archives and cached binaries,
and replaces the Docker bootstrap scripts with pinned, verified sources.


## Type of Change

- [x] Bug fix



## Changes Made

- Preserve Rust 1.95.0, minimal/rustfmt/clippy options, Node 24, npm
12.2.1, and OpenCode 1.17.8.
- Preserve Debian userland architecture detection, Rustup ARM fallback,
and OpenCode x64 AVX2/baseline selection.
- Use an official Node image pinned by manifest digest; retain
npm/corepack trees and explicit runtime libraries.
- Add offline bootstrap tests for failed downloads, corrupt bytes,
unsupported targets, symlinks, and option forwarding.


## Testing

- [x] Offline bootstrap tests pass (stdlib unittest, 16 cases).

- [x] Ruff lint/format pass for the two new test files; shell syntax and
whitespace checks pass.
- [x] Exact-head hosted CI passed, including platform tests and
Docker-native, Init, and Wrap build/container checks.


### Test Output

```text

python3 -m unittest discover -s tests/test_install -p 'test_*bootstrap.py'

................

Ran 16 tests in 0.770s

OK

```



## Real Behavior Proof

- Environment: Ubuntu 24.04 under WSL, exact candidate
4ee38e8619.
- Exact command / steps: Run the unittest discovery command above.
Separately stream the recorded release archives and official Node image
layer through SHA-256 and inspect their tar headers.
- Observed result: Invalid bytes stop before extraction/execution;
fixture platform and CPU cases select the intended archives. Actual
workflow-tool/OpenCode archives and the Node layer match published
digests. The Rustup x64 binary matches its vendor checksum; all nine
embedded target checksums match vendor records.
- Not tested: No live model/provider behavior or unexercised
architecture is claimed. Local bootstrap fixtures use mocks; the hosted
Docker/build checks below exercised the actual pinned installation
paths.


## Runtime Rollout Safety

- Rollout-managed feature(s): None; CI/build bootstrap only.

- Minimum rollout channel: Not applicable.

- Stable/default behavior changed: Build inputs become pinned; proxy
defaults, Rust 1.95.0/components, npm 12.2.1 and OpenCode 1.17.8 stay
unchanged. Node remains on the 24 LTS line with a fixed image digest.
- Kill switch / disable path: Revert the maintenance change through
review; do not bypass checksum failures.
- Unsafe override required: None for this exact reviewed candidate.

- Qualification impact: The exact-head hosted checks below passed.
Existing PR heads retain their separate security/qualification holds
until main is remediated and their own gates are refreshed.
- Rollback path: Revert this maintenance PR if build compatibility
regresses.


## Review Readiness

- [x] I performed a self-review.

- [x] This PR is ready for human review.



## Checklist

- [x] New security regression tests are included.

- [x] CHANGELOG.md is unchanged.



## Additional Notes

Static independent code review found no defects. No merge or
self-approval is requested. Completed historical CI evidence is retained
within its original scope; this change qualifies future installer
execution separately.


Initial hosted validation of b2592de8d stopped at rustup filename
dispatch. Follow-up 4ee38e861 retains the required rustup-init basename
in a private directory and adds the reproducing regression test. All
nine latest applicable hosted workflows passed on the corrected head.


### Hosted validation on 4ee38e861

- [CI: platform tests and Docker-native
build/install/compose/wrap/init](https://github.com/headroomlabs-ai/headroom/actions/runs/42352878789):
success.
- [Wrap
E2E](https://github.com/headroomlabs-ai/headroom/actions/runs/42352878762),
[Init
E2E](https://github.com/headroomlabs-ai/headroom/actions/runs/42352878760),
[Init Native
E2E](https://github.com/headroomlabs-ai/headroom/actions/runs/42352878767),
and
[Security](https://github.com/headroomlabs-ai/headroom/actions/runs/42352878768):
success.
- Governance, changelog, Rust and merge-conflict checks also passed. The
older cancelled governance duplicate was superseded by a successful run.


Independent human review is required. No self-approval or merge has been
performed.




### Local-action metadata coverage follow-up



`38337d8be` adds `.github/actions/**` to both code and workflow filters.
The normal pytest suite now checks the vendored commitlint metadata
structure and immutable Docker digest; this is separate from actionlint
workflow parsing. Six focused CI regression tests pass, including a
filter regression that failed before the fix. Independent delta review
found no issues. `38337d8be` now has successful latest applicable CI,
Wrap E2E, Init E2E, Init Native E2E, Security, Rust, Changelog Guard,
Merge Conflicts, and PR Governance runs. The successful runs above
remain historical evidence for `4ee38e861`.




Current-head
[CI](https://github.com/headroomlabs-ai/headroom/actions/runs/42360858297),
[Wrap](https://github.com/headroomlabs-ai/headroom/actions/runs/42360858271),
[Init](https://github.com/headroomlabs-ai/headroom/actions/runs/42360858296),
[Init
Native](https://github.com/headroomlabs-ai/headroom/actions/runs/42360858178),
and
[Security](https://github.com/headroomlabs-ai/headroom/actions/runs/42360858352)
passed. Independent human review remains required; no self-approval or
merge.


<!-- maintainer-current-main-qualification -->

## Current-main qualification



Published head `9381378ffb07e466acef6a57eb39322aa84d4582` includes
upstream main `bb8d18a56abb07ca7e811235c86ea6b626eb1f8b` and is
mergeable. All nine exact-head workflows succeeded; all 53 jobs are
terminal success or explicit skips, including CI Docker-native,
wrap/init and native-init validation. Local metadata/release checks: 61
passed; 16 Linux-only bootstrap tests skipped on Windows. Remediation
installer/action sources are unchanged by the clean main integration.


Independent human review and merge are required. No self-approval or
merge performed. This PR does not lift the inherited
execution-provenance hold on main or other branches before human
integration.
<!-- /maintainer-current-main-qualification -->



### Maintainer current-main refresh (a2bcbd101)

Published `a20f072a238a1f8de422adf6a25d3f9d664151e4` incorporates
upstream main `a2bcbd101b1391debfd218ec8c63cb72d2bc240a` through a clean
merge. All 13 previously screened CI provenance remediation files are
byte-identical to prior published head
`9381378ffb07e466acef6a57eb39322aa84d4582`; no
installer/action/remediation logic was rewritten. Fresh focused checks:
61 passed, 23 platform-specific shell cases skipped on Windows. No
vendor installer, Docker image or downloaded binary was executed
locally. Earlier Linux bootstrap and hosted evidence retain their
original scope.

Fresh exact-head hosted validation completed successfully: all 10
observed workflow runs succeeded; all 56 checks are terminal successful
or explicitly skipped. Ready for independent human review on this
head/current main. This remediation branch alone is permitted to run its
screened checks; inherited execution/publication holds on main and other
PRs remain until independent human integration. Independent review is
required; no self-approval, merge or release publication.

---------

Co-authored-by: Jerrett Davis <JerrettDavis@users.noreply.github.com>
2026-11-10 21:02:42 -06:00
a5e68deb64 docs(pypi): job-word summary and keywords, and a PyPI-specific README (#4534)
## Description

Agents that install from PyPI read the package page plus the installed
code and nothing else; in both recorded runs they had to read venv
source to learn the result fields and `HEADROOM_OFFLINE`. The GitHub
README is written for a browser: hero image, badges, perk banner,
relative links that break off GitHub, and a first install command that
pulls `[all]`. The summary ("Cut costs by 50-90%") disagrees with the
README's own measured table, and none of the words in the three web
queries that currently return nothing of ours (tool outputs, chat
history, token cost, fastapi) appear in the summary or keywords.

Depends on #4529 for the `/docs/fastapi` link. Takes effect on PyPI with
the next release.

## Type of Change

- [x] Documentation update

## Changes Made

- `README-pypi.md` (new): no-key `compress()` path, zero-network check
with expected output, result fields, offline semantics (beacon is
proxy/MCP-side), coding-agent path, extras explained, "the PyPI
distribution named `headroom` is a different project".
- `pyproject.toml`: `readme = "README-pypi.md"`, job-word `description`
without a percentage, keywords extended with the job words.
- `Dockerfile`, `e2e/wrap/Dockerfile`, `e2e/init/Dockerfile`: `COPY` the
new readme next to `README.md` (all three build the wheel from
`pyproject.toml`, so maturin needs the declared readme in the build
context); `Dockerfile`'s `source_digest` inputs include it too.

## Testing

- [x] Manual testing performed

### Test Output

```text
uvx maturin sdist -o /tmp/sdist: Built source distribution headroom_ai-0.42.0.tar.gz
tar tzf: headroom_ai-0.42.0/README-pypi.md present
PKG-INFO: Summary: Compress tool outputs, logs, JSON and chat history before each LLM call. ... / Description-Content-Type: text/markdown
```

## Real Behavior Proof

- Environment: macOS arm64, `uvx maturin sdist` on this branch.
- Exact command / steps: build the sdist, list it, read `PKG-INFO`.
- Observed result: `headroom_ai-0.42.0/README-pypi.md` is in the
tarball; `PKG-INFO` has the new `Summary:`, `Description-Content-Type:
text/markdown`, the README body and the new keywords. The one-liner in
the file was run against the published 0.42.0 wheel: `20031 -> 95 FATAL
kept: True`.
- Not tested: the Docker images themselves (CI's docker and e2e jobs
cover them); the wheel build (maturin embeds the same `readme` field;
nothing in `.github/workflows` or `scripts/` pins `README.md` as the
readme).

## Runtime Rollout Safety

- Rollout-managed feature(s): none; documentation and metadata only, no
runtime code path changes
- Minimum rollout channel: n/a
- Stable/default behavior changed: no
- Kill switch / disable path: n/a (revert the PR)
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-03 23:37:58 -06:00
Younes BerianeandJD Davis 670767eeb4 fix(docker): pass RUNTIME_HOME=/root for root bake targets (#4445)
## Description

Fixes #4443.

The four root bake targets (`runtime`, `runtime-code`, `runtime-slim`,
`runtime-code-slim`) set `RUNTIME_USER=root` but not `RUNTIME_HOME`, so
the Dockerfile default `/home/nonroot` was used for `VOLUME` and
`WORKDIR` while the process runs with `HOME=/root`. `~/.headroom`
therefore landed in the container layer instead of the declared volume.

Passing `RUNTIME_HOME=/root` fixes the two python-slim targets, but the
two slim targets build the distroless `runtime-slim` stage, which
declared no `ARG RUNTIME_HOME` and no `VOLUME` at all, so the arg was a
silent no-op there. That stage now declares `ARG RUNTIME_HOME` and
`VOLUME ${RUNTIME_HOME}/.headroom`, and since distroless has no shell
for `RUN mkdir`, the state directory is built in a small
`runtime-slim-state` stage and copied in owned by `RUNTIME_USER`.

Closes #4443

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `docker-bake.hcl`: pass `RUNTIME_HOME = "/root"` in the four targets
that set `RUNTIME_USER = "root"`.
- `Dockerfile`: new `runtime-slim-state` stage
(`python:${PYTHON_VERSION}-slim`) that creates `/state/.headroom`; the
distroless `runtime-slim` stage now declares `ARG
RUNTIME_HOME=/home/nonroot`, copies that directory to `${RUNTIME_HOME}/`
with `--chown=${RUNTIME_USER}:${RUNTIME_USER}`, and declares `VOLUME
${RUNTIME_HOME}/.headroom`, matching the python-slim stage.
- `tests/test_release_workflows.py`:
`test_docker_bake_runtime_args_are_declared_by_the_selected_stage` walks
every bake target, resolves the Dockerfile stage it builds (including
inherited parent stages), and checks that each `RUNTIME_*` arg it passes
is declared by that stage, that root targets use `/root`, and that the
stage puts `VOLUME ${RUNTIME_HOME}/.headroom` under the home it was
given.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ pytest tests/test_release_workflows.py -q            # head 59bd12d
58 passed, 1 warning in 6.41s

$ pytest tests/test_release_workflows.py -q -k runtime_args   # new test on upstream main
E   AssertionError: runtime: root target needs /root
1 failed, 57 deselected, 1 warning in 0.77s

$ pytest tests/test_release_workflows.py -q -k runtime_args   # new test on ae30825 (bake fix only)
E   AssertionError: runtime-slim: stage runtime-slim does not declare {'RUNTIME_HOME'}
1 failed, 57 deselected, 1 warning in 0.40s

$ ruff check tests/test_release_workflows.py
All checks passed!
$ ruff format --check tests/test_release_workflows.py
1 file already formatted
```

## Real Behavior Proof

- Environment: local Linux, Python 3.14, pytest (no Docker daemon
locally); Docker images built and run on a GitHub-hosted `ubuntu-latest`
runner (Docker 28.0.4, buildx v0.38.1, `linux/amd64`) by a scratch
workflow in my fork on top of `59bd12d`:
[run](https://github.com/drakeo338/headroom/actions/runs/40957966633),
[workflow](https://github.com/drakeo338/headroom/blob/11ce83618e808688bea5a60cd0545e9bae511d4a/.github/workflows/image-evidence.yml)
- Exact command / steps: ran the new test against upstream `main`,
against `ae30825` (bake args only) and against head `59bd12d` (bake args
plus the `runtime-slim` stage change); full
`tests/test_release_workflows.py` on head. On the runner: `docker buildx
bake -f docker-bake.hcl --print runtime runtime-code runtime-slim
runtime-code-slim`; `docker buildx bake -f docker-bake.hcl --set
'*.platform=linux/amd64' --load` of the same four targets; `docker
inspect` of `Config.User`, `Config.WorkingDir`, `Config.Volumes` per
image; per image, one container writes `/root/.headroom/probe` on a
named volume and a second container on the same volume reads it back,
plus `docker create` without `-v` to list the image's own mounts
- Observed result: the test fails on `main` (root targets have no
`RUNTIME_HOME`), fails on `ae30825` (the distroless `runtime-slim` stage
does not declare `RUNTIME_HOME`, so the slim targets ignore it), passes
on `59bd12d`; all 58 tests in the file pass. The bake plan resolves
`runtime`/`runtime-code` to stage `runtime` and
`runtime-slim`/`runtime-code-slim` to stage `runtime-slim`, all with
`RUNTIME_USER=root RUNTIME_HOME=/root`. Inspected: `runtime` and
`runtime-code` have `User="root" WorkingDir="/root"
Volumes={"/root/.headroom":{}}`; `runtime-slim` and `runtime-code-slim`
have `User="root" WorkingDir="/app" Volumes={"/root/.headroom":{}}`. In
all four the probe written by the first container is read back by the
second, and `docker create` shows `type=volume
destination=/root/.headroom`
- Not tested: mypy (no Python package code changed)

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: yes, the root images keep
`~/.headroom` on the declared volume, and the distroless slim images now
declare that volume
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

N/A

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4445"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-03 22:49:38 -06:00
8d5dde7765 fix(docker): install the redis extra in every image (#4514)
## Description

The 0.42 images ship the replica-shared compress session store, but no
published variant installs the `redis` client. Setting
`HEADROOM_COMPRESS_SESSION_STORE=redis://...` stopped startup with
`ConfigurationError`, and `HEADROOM_CCR_BACKEND=redis` degraded to the
per-process store. This PR adds the `redis` extra to the Dockerfile
default and to all nine `docker-bake.hcl` targets. It locks the
invariant with a release test. It also documents the session fields of
`/v1/compress` and the shared session store.

Closes #4513

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (a fix or feature that changes existing
functionality)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `Dockerfile`, `docker-bake.hcl`: `HEADROOM_EXTRAS` gains `redis` in
the default and in all nine bake targets.
- `tests/test_release_workflows.py`: new
`test_every_docker_variant_includes_the_redis_extra` guards the
Dockerfile default, the bake targets, and the `pyproject.toml` extra.
- `docs/content/docs/installation.mdx`: new `redis` extras row. The
Docker tag table shows the new extras.
- `docs/content/docs/proxy.mdx`, `wiki/proxy.md`: documented the
`session` response fields, `HEADROOM_COMPRESS_SESSION_STORE`, its
failure behavior, and the `session_affinity` interaction.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_release_workflows.py -q
Pytest: 55 passed

$ uvx ruff@0.17.0 check .
All checks passed!
```

## Real Behavior Proof

- Environment: macOS worktree of this repository, Python 3.13.
- Exact command / steps: `python -m pytest
tests/test_release_workflows.py -q` and `uvx ruff@0.17.0 check .`.
- Observed result: 55 tests passed, ruff clean. The new test fails
against the pre-change `Dockerfile`, because its `ARG HEADROOM_EXTRAS`
line lacked `redis`.
- Not tested: the Docker images were not built. The release test asserts
the build args statically, as the sibling Bedrock test does.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. The change touches build args, docs,
and a test.
- Minimum rollout channel: not applicable.
- Stable/default behavior changed: every published image now installs
the `redis` client. With the extra installed,
`HEADROOM_COMPRESS_SESSION_STORE=redis://...` starts instead of failing,
and `HEADROOM_CCR_BACKEND=redis` uses Redis instead of the per-process
fallback.
- Kill switch / disable path: drop `redis` from `HEADROOM_EXTRAS` at
build time, or unset the two environment variables at runtime.
- Unsafe override required: no.
- Qualification impact: none. No runtime code path changed. The `redis`
import stays lazy, so deployments that do not set the variables are
unaffected.
- Rollback path: revert this PR and rebuild.

## Review Readiness

- [x] I performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I performed a self-review of my code
- [x] I commented my code, particularly in hard-to-understand areas
- [x] I made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I added tests that prove my fix is effective or that my feature
works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`. It is generated by
release-please from my Conventional Commit PR title, and a CI guard
enforces this.

## Screenshots (if applicable)

Not applicable.

## Additional Notes

The full test suite and `mypy headroom` were not run. This PR changes no
file under `headroom/`, so I ran the affected suite
(`tests/test_release_workflows.py`) and lint only. The docs half also
answers part of #3184 (documentation of the storage backends). The CI
half of #3184 stays open, and PR #3282 covers it.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4514"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <tejas@headroomlabs.ai>
2026-10-28 11:29:27 -07:00
Tejas ChopraandClaude Fable 5.1 3c6f82b060 deps: require Python 3.11; clear Dependabot alerts in the 3.10 lock fork (#4517)
## Description

Vanta/Dependabot report 64 open alerts on this repo. 59 of them are
pillow, filelock and virtualenv versions that exist only in the Python
3.10 split of `uv.lock`: the patched releases (pillow 13, filelock 4.2)
require Python 3.11, so Dependabot could not open a fix PR. Python 3.10
reached end of life in October 2026 and CI already tests only 3.11, 3.12
and 3.14.

The second commit also fixes the two `security.yml` jobs that fail on
`main` today: pip-audit (litellm CVE-2026-114612) and the e2e npm audit
(three undici advisories published after #4453 for the copy bundled
inside npm).

## Type of Change

- [x] Breaking change: `pip install headroom-ai` now refuses Python 3.10
- [x] Bug fix: security patches

## Changes Made

- `requires-python = ">=3.11"`; drop the 3.10 classifier and the `tomli`
backport dependency
- pillow override and filelock floor lose their `python_version` markers
- mypy `python_version = "3.11"`; the three `tomli` import fallbacks in
`headroom/` become plain `import tomllib`
- ruff `target-version` stays `py310` with a note: moving it raises 213
pyupgrade findings (UP017/UP036/UP041/UP042), a separate mechanical PR
- `uv.lock` re-resolved: pillow 13.0.0, filelock 4.2.4, virtualenv
21.14.13 (single entry each, the 3.10 fork is gone)
- gitpython 3.1.62 → 3.2.1, strands-agents 1.24.0 → 1.62.0, litellm
1.101.0 → 1.101.3
- `e2e/wrap/tools/npm-audit-exceptions.json`: three further undici 6.x
advisories added at the existing exact path and version, expiry
unchanged (2026-11-15)
- README, CONTRIBUTING and the Dockerfile comment updated

Clears 61 of the 64 Dependabot alerts. The remaining 3 (undici,
ip-address) live inside the bundled dependencies of the `npm` package in
`e2e/wrap/tools/package-lock.json`; npm 12.2.1 is the newest release and
still ships them, so there is no bump available yet.

## Dependencies & supply chain

**gitpython 3.1.62 → 3.2.1.** Already a dependency; bumped for a
security patch (CVE-2026-114145). Maintained by the gitpython-developers
org (Sebastian Thiel), monthly-ish releases, long advisory history with
prompt fixes. Install surface unchanged: pure-Python wheel
(`py3-none-any`), one runtime dependency (`gitdb`), no native code, no
install-time or runtime network of its own.

**strands-agents 1.24.0 → 1.62.0.** Already a dependency behind the
`[strands]` extra; bumped for a security patch (CVE-2026-115419, fixed
in 1.43.0; the lock takes the current release). Maintained by AWS
(strands-agents org, `opensource@amazon.com`), weekly releases. Install
surface: pure-Python wheel, no native code. Declared dependencies grow
by two, `httpx` and `pyyaml`, both already in the lock for other
packages, so no new transitive packages are introduced. Runtime network
is unchanged: it calls the model provider the user configures, as
before.

**litellm 1.101.0 → 1.101.3.** Already a core dependency; patch bump for
CVE-2026-114612, the smallest fix that keeps the `filelock>=4.2.1` floor
resolvable (litellm 1.106 caps filelock <4). No change to maintainers or
install surface.

**pillow, filelock, virtualenv.** No new packages; the lock now holds
the single patched version the 3.11+ fork already had.

## Testing

- [x] Linting passes (`ruff check .`, `ruff format --check .`)
- [x] Type checking passes (`mypy headroom`): 1 error, identical to main
(`langchain/chat_model.py:380`)
- [x] Unit tests (targeted, see below)

### Test Output

```text
$ pytest -q tests/test_litellm_optional.py tests/test_dependency_overrides.py tests/test_release_workflows.py tests/test_image_decoder_guard.py tests/test_optional_dependencies.py tests/test_onnx_dependency_contract.py
74 passed in 3.07s

$ grep -rl -E "headroom\.release_version|headroom\.providers\.codex\.project_context|headroom\.cli\.wrap" tests | xargs pytest -q
4 failed, 1043 passed in 28.03s     # same 4 in tests/test_cli/test_wrap_persistent.py fail on main

$ uv run --extra strands --extra dev -m pytest -q tests/integrations/test_strands tests/test_strands_tokenizer.py
18 failed, 87 passed in 52.30s      # the 18 are *Real classes needing live AWS Bedrock creds; main fails the same 18 on 1.24.0

$ uv export --frozen --no-dev --no-emit-project --no-hashes --extra all --format requirements-txt > requirements-prod.txt
$ uvx pip-audit==2.10.1 -r requirements-prod.txt --no-deps --disable-pip
No known vulnerabilities found

$ cd e2e/wrap/tools && npm audit --package-lock-only --json > audit.json && <security.yml check script>
unexpected: []  stale: []  seen: 6
```

## Real Behavior Proof

- Environment: macOS 27.2 arm64; uv-managed CPython 3.11.16 and 3.10;
the lock from this branch.
- Exact command / steps:
  ```text
$ uv run --python 3.11 --all-extras python -c "import sys, platform,
headroom, PIL, filelock, virtualenv, git, strands, litellm,
importlib.metadata as m; ..."
$ uv venv -p 3.10 /tmp/py310-venv && uv pip install --python
/tmp/py310-venv/bin/python --dry-run .
  ```
- Observed result:
  ```text
  macOS-27.2-arm64-arm-64bit
  python 3.11.16
  pillow 13.0.0
  filelock 4.2.4
  virtualenv 21.14.13
  gitpython 3.2.1
  strands-agents 1.62.0
  litellm 1.101.3

(3.10) ... headroom-ai==0.42.0 depends on Python>=3.11, we can conclude
that headroom-ai==0.42.0 cannot be used.
And because only headroom-ai==0.42.0 is available and you require
headroom-ai, we can conclude that your requirements are unsatisfiable.
  ```
- Not tested: Linux and Windows installs (CI wheel matrix covers them);
a live Bedrock run of the strands integration; the Docker image build.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: installs on Python 3.10 are refused
at resolve time; no runtime behavior change on 3.11+
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] New and existing unit tests pass locally with my changes (targeted
runs above; pre-existing failures noted)
- [x] I did **not** edit `CHANGELOG.md`
- [ ] New tests added: n/a, dependency and metadata change; existing
contract tests cover the floor and the pillow override

## Additional Notes

Follow-ups, not in this PR:
- ruff `target-version = "py311"` plus the pyupgrade fixes
- remaining `tomli` fallbacks in `tests/` and the `sys.version_info <
(3, 11)` branches they guard
- `python_version < '3.11'` branches in the onnxruntime markers (now
dead but harmless)
- `pyo3` `abi3-py310` in `Cargo.toml` could move to `abi3-py311`
- the 3 remaining Dependabot alerts (bundled undici/ip-address in npm)
until an npm release replaces them

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-28 07:28:07 -07:00
1be699f68e fix(deps): patch OpenClaw Undici and narrow E2E audit exceptions (#4453)
## Description

Includes the approved #4358 as its prerequisite. This PR targets main so
the complete integration workflows qualify the combined repair; its diff
will shrink to this follow-up once #4358 lands. Replace OpenClaw's
affected Undici copy instead of excepting it, and reduce the E2E audit
exceptions from seven advisory/path pairs to three.

OpenClaw 2026.10.2 lists Undici in `bundleDependencies`. Version
2026.10.6 removes it from that bundle, allowing a targeted, same-major
override to install fixed Undici 8.12.3. npm 12.2.1 still bundles its
own affected copies; this PR retains only those existing, expiring
exceptions.

## Type of Change

- [x] Bug fix

## Changes Made

- Pin the existing E2E OpenClaw CLI to 2026.10.6 and override only its
Undici dependency to 8.12.3.
- Regenerate the E2E lockfile; keep Codex and the npm override
unchanged.
- Remove every OpenClaw exception and the now-stale advisory, preserving
the parent's exact bundled-version checks and 2026-11-15 expiry.
- Update the Dockerfile comment to the actual Node requirement: 24.22.1
in the Node 24 line. Its NodeSource installation already selects that
line.

## Dependency and install-surface assessment

This updates an existing upstream-maintained CLI for a security fix,
rather than adding a replacement package. The fixed Undici remains in
the same major line. The graph is used only by the Docker wrap E2E
harness and is not shipped in Headroom's wheel or SDK. Normal
installation runs OpenClaw's existing lifecycle scripts and fetches
packages from the registry; those scripts were inspected, then exercised
inside a fresh Linux container without host credentials. No application
compression or routing logic changes.

## Testing

- [x] Manual testing performed
- [x] Dependency installation and runtime resolution verified

### Test Output

```text
OpenClaw runtime resolves Undici 8.12.3
OpenClaw 2026.10.6 (fe06181)
Installed audit: only npm bundled copies remain
{ info: 0, low: 0, moderate: 1, high: 1, critical: 0, total: 2 }
parent: 9 audit-guard cases passed
child: 9 audit-guard cases passed
git diff --check: passed
```

The guard accepts the current scoped findings and rejects a new
advisory, wrong path, expired exception, stale exception, audit error,
changed version, non-bundled copy, and missing inherited-advisory entry.
A raw audit remains nonzero for npm's bundled dependencies; this PR does
not claim those packages are patched.

## Real Behavior Proof

- Environment: fresh Linux amd64 Docker container,
`node:24.22.1-bookworm-slim`, Node 24.22.1 / npm 11.19.1, normal `npm
ci` with lifecycle scripts enabled.
- Exact command / steps: copy the two committed package files into a
fresh container volume; run `npm ci --no-audit --no-fund`; resolve
`undici/package.json` using
`createRequire('/work/node_modules/openclaw/package.json')` and assert
version 8.12.3; run `node node_modules/openclaw/openclaw.mjs --version`
and `--help`; run `npm audit --json` and assert that every remaining
affected node is under `node_modules/npm/`.
- Observed result: installation, exact runtime-resolution assertion,
version, and help all succeeded. Both installed-tree and lockfile audits
leave only npm's bundled Undici 6.29.0 and ip-address 10.7.2. The parent
workflow's guard passes with the narrowed exceptions and rejects all
eight negative cases.
- Not tested: a billed provider request or live production deployment.
Hosted Docker agent/integration scenarios have now passed. Hosted
qualification on head 63c3abc5b3 is
complete: all reported checks are terminal with no failures or pending
checks, including the full Security, build, test, and Docker integration
workflows.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; E2E dependency data only.
- Minimum rollout channel: existing E2E test environment.
- Stable/default behavior changed: this follow-up patches only the
harness CLI. The included parent retains its explicitly documented
affected-Pillow decoder guard.
- Kill switch / disable path: revert this dependency commit.
- Unsafe override required: none; the override selects a compatible
fixed Undici 8.x patch. The parent's existing, bounded npm exceptions
remain explicit.
- Qualification impact: package installation, CLI startup, runtime
dependency resolution, and audit guarding were verified in isolation.
- Rollback path: revert the four-file change.

## Review Readiness

- [x] I performed self-review
- [x] This PR is ready for human merge review; the complete hosted check
set passed

## Checklist

- [x] I did not edit CHANGELOG.md.
- [x] The exception scope is reduced, not expanded.
- [x] No contributor correctness logic was modified.

Python lint/type checks and new production unit tests are not applicable
to these dependency/data changes. Independent human approval is
required; this is an author-side qualification record.

---------

Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Tejas Chopra <tejas@headroomlabs.ai>
2026-10-23 23:11:35 -07:00
natoriandJD Davis 1b7977cecd fix: honor configured port in container startup and healthcheck (#2436)
## Description

Deploying on a non-default port produced a working proxy that the
surrounding
tooling reported as broken. Tracing the probe turned up a second defect
upstream of it: the container could not change its port via
`HEADROOM_PORT`
at all.

`headroom proxy --port` is a click option with `envvar="HEADROOM_PORT"`,
and an
explicit CLI argument beats the envvar. The Dockerfile baked `--port
8787` into
CMD, so `HEADROOM_PORT` was inert for a plain `docker run` — while
`docker-compose.yml` already passes `command: ["--host", "0.0.0.0"]`
with no
port, which is the intended shape. The Dockerfile was the outlier.

The two halves have to move together. An earlier revision of this change
fixed
only the HEALTHCHECK; that made the probe follow `HEADROOM_PORT` while
the
listener stayed pinned to 8787, and the container went from misleadingly
healthy to genuinely unhealthy. Evidence in Real Behavior Proof below.

Closes #2432

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `Dockerfile`, both runtime stages: dropped the baked `--port` from CMD
so
`HEADROOM_PORT` stays authoritative. `docker run` with no env still
binds
  8787 (click's default), so the published default is unchanged.
- `Dockerfile` (`runtime-slim-base`): shell-form HEALTHCHECK so
  `${HEADROOM_PORT:-8787}` expands at probe time.
- `Dockerfile` (`runtime-slim`): distroless, no shell — the port is
resolved
  inside Python instead, keeping exec form.
- `headroom/cli/doctor.py`: added `resolve_probe_port()`. `--port` /
`HEADROOM_PORT` wins, then an unambiguous deployment manifest (`default`
profile, or a lone named profile), then 8787. `headroom deploy --port N`
writes N into the deployment's environment, not into the shell `doctor`
runs
in, which is why the manifest tier is needed; with several named
profiles and
  no explicit request there is no non-arbitrary pick, so it stays on the
  default and the existing per-deployment section reports each one.
  `list_manifests()` is now read once and shared with that check.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom --ignore-missing-imports`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

Full suite, run inside this repo's own builder image so the
maturin-compiled
Rust core is the one under test:

```text
9 failed, 9248 passed, 853 skipped, 2230 warnings in 384.56s (0:06:24)
```

All 9 failures reproduce unchanged on `origin/main` (c400f908) in the
same
container. Restricted to just those files, base and branch are
identical:

```text
origin/main c400f908: 13 failed, 119 passed, 15 skipped, 66 warnings in 14.57s
this branch:          13 failed, 119 passed, 15 skipped, 66 warnings in 14.58s
```

They are `test_cli/test_wrap_helpers.py`, `test_image_compression.py`,
`test_memory_bridge.py`, `test_proxy/test_cc_switch_reconciler.py` and
`test_startup_log_noise.py`. Counts differ between the full and
file-scoped
runs because `test_memory_bridge` and `test_startup_log_noise` are
order-dependent — also true on the base.

```text
$ mypy headroom --ignore-missing-imports
pyproject.toml: note: unused section(s): module = ['mlx.*']
Success: no issues found in 505 source files
```

New and changed tests, green on this branch:

```text
tests/test_docker_healthcheck_port.py ......                   [100%]
6 passed in 0.15s

tests/test_cli_doctor.py .............................         [100%]
67 passed in 0.68s
```

Both go red with only the tests applied on top of `origin/main`:

```text
FAILED tests/test_docker_healthcheck_port.py::test_no_healthcheck_hardcodes_the_default_port
FAILED tests/test_docker_healthcheck_port.py::test_every_healthcheck_resolves_the_port_from_the_environment
FAILED tests/test_docker_healthcheck_port.py::test_distroless_stage_healthcheck_stays_shell_free
FAILED tests/test_docker_healthcheck_port.py::test_debian_stage_healthcheck_expands_the_port_variable
FAILED tests/test_docker_healthcheck_port.py::test_no_stage_bakes_a_port_into_cmd
5 failed, 1 passed in 0.23s

tests/test_cli_doctor.py:
E   ImportError: cannot import name 'DEFAULT_PROXY_PORT' from 'headroom.cli.doctor'
```

## Real Behavior Proof

- Environment: macOS 15 (Darwin 25.3.0) arm64, Docker 29.1.3, image
built from this branch with `docker build -t headroom:pr2432 .`; control
is the published `ghcr.io/headroomlabs-ai/headroom:latest`.
- Exact command / steps: started three containers — the patched image
with `HEADROOM_PORT=8789`, the patched image with no env at all, and the
published image with `HEADROOM_PORT=8790` as a control — then read
`docker ps` health and probed `/readyz` from inside each. Full commands
below.
- Observed result: the patched image binds 8789 and its probe follows,
reporting healthy; with no env it still binds 8787, so the published
default is unchanged; the published image ignores `HEADROOM_PORT`
entirely and stays on 8787. Output below.
- Not tested: `headroom deploy` was not driven end to end — the manifest
was written directly rather than produced by the deploy command, so the
manifest *contents* are assumed to match what deploy records (`port` is
the field `headroom install status` already reads). Non-Docker
supervisors were not exercised.

### Containers

```bash
docker run -d --name hr-fixed   -e HEADROOM_PORT=8789 -p 8789:8789 headroom:pr2432
docker run -d --name hr-default                       -p 8791:8787 headroom:pr2432
docker run -d --name hr-control -e HEADROOM_PORT=8790 -p 8790:8790 \
    ghcr.io/headroomlabs-ai/headroom:latest
```

```text
NAMES        STATUS
hr-fixed     Up (healthy)   HEADROOM_PORT=8789 -> listener on 8789
hr-default   Up (healthy)   no env             -> listener on 8787
hr-control   Up (healthy)   HEADROOM_PORT=8790 -> listener on 8787
```

Probing from inside each container confirms which port is actually
serving:

```text
hr-fixed:   8789=OK
hr-default: 8787=OK
hr-control: 8787=OK
```

`hr-control` is the published image ignoring `HEADROOM_PORT` entirely —
the CMD
half of the bug. Baked image config, before and after:

```text
published: ["CMD","curl","--fail","--silent","http://127.0.0.1:8787/readyz"]
this PR:   ["CMD-SHELL","curl --fail --silent \"http://127.0.0.1:${HEADROOM_PORT:-8787}/readyz\""]
```

The distroless stage was verified the same way — `docker build --target
runtime-slim` with `-e HEADROOM_PORT=8795` reports healthy and answers
`GET /readyz` with 200 from the host.

### doctor

Running the real CLI against a seeded `default` deployment manifest
recording
port 8789, with `HEADROOM_PORT` unset in the shell — the reported
situation,
where the port lives in the deployment rather than in the shell:

```text
origin/main c400f908: doctor probed port: 8787
this branch:          doctor probed port: 8789
```

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from
      my Conventional Commit PR title (a CI guard enforces this)

## Screenshots (if applicable)

N/A — no UI surface.

## Additional Notes

"I have made corresponding changes to the documentation" is checked in
the sense
that the `--port` help text now describes the new resolution order; no
prose
docs referenced the baked CMD port. `README.md` and the wiki reference
`headroom proxy --port 8787` as a host command, which is unaffected.

`docker/differential-network-capture/docker-compose.yml` passes `--port
8787`
explicitly in its own `command:`, which still overrides everything and
keeps
working.

Adjacent, deliberately not in this PR: `docker-compose.yml`'s
healthcheck is
hardcoded to 8787 the same way. It is inert today because that file
never sets
`HEADROOM_PORT`, and making it configurable also means parameterising
the
published `ports:` mapping — a separate change. Happy to open a
follow-up if
you want it.

## Runtime Rollout Safety

- Rollout-managed feature(s): Docker runtime port selection and
`headroom doctor` probe-port resolution.
- Minimum rollout channel: Standard Docker image release and persistent
Docker deployments.
- Stable/default behavior changed: No. Unset `HEADROOM_PORT` continues
to use port 8787.
- Kill switch / disable path: No new feature flag; revert the image or
this PR.
- Unsafe override required: No.
- Qualification impact: None. No data or schema migration; existing
port-8787 deployments remain compatible.
- Rollback path: Redeploy the previous image or revert this PR; no
migration cleanup is required.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2436"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-05 08:36:54 -05:00
Rod BoevandJerrettDavis b0c19a25f0 fix(security): reject unauthenticated public proxy binds (#3316)
## Description

Headroom's published image and maintained Docker launchers could expose
the
proxy on every host interface without a proxy token. This change moves
the
security boundary to the deployment artifacts: published images default
to a
container-loopback listener, supported local launchers publish only on
host
loopback, and deliberate public deployments retain explicit token
protection.
The application factory remains transport-neutral.

Closes #3283

## Type of Change

- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [x] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Set both published image stages to a container-loopback default while
keeping explicit host overrides available.
- Changed native Compose and the maintained Bash and PowerShell
launchers to use an internal `0.0.0.0` listener with host-loopback
publication.
- Preserved the top-level Compose token requirement and the existing
HTTP and WebSocket credential behavior for deliberate public binds.
- Scoped differential-capture token injection to the internal
`headroom-proxy` destination and kept sensitive headers redacted in
captured records.
- Added focused launch-contract, native-core, capture, and Docker
publication checks, plus a base/head harness that tolerates a base
already containing the fix.
- Updated Docker, installer, capture, and mirrored documentation,
including an unauthenticated local loopback quickstart and an
authenticated deliberate-public example.

## Testing

- [x] Focused unit and integration tests pass
- [x] Linting and formatting checks pass
- [ ] Type checking passes
- [x] New tests added for the changed deployment and capture behavior
- [ ] Manual testing performed

### Test Output

```text
uv run python -m pytest tests/test_proxy_hardening.py tests/test_install/test_native_installers.py tests/test_docker_compose_persistence.py tests/test_network_diff_capture.py tests/test_rust_core_smoke.py -q
76 passed, 1 skipped, 1 warning in 23.45s

uv run ruff check tests/test_install/test_native_installers.py tests/test_install/test_runtime.py
All checks passed!

bash -n e2e/docker-bind-security.sh
passed

bash e2e/docker-bind-security.sh --base-image headroom-t3283-base --head-image headroom-t3283-head --fixture tests/fixtures/docker-bind-exposure.json --base-ref origin/main
base_ref=origin/main ... runtime_slim_config=loopback rust_core=loaded local_completion=200 attacker_loopback=000 ws_status=403
docker bind security: fixture forwarding, loopback isolation, built-image health, slim-image loopback config, and token controls passed
```

## Real Behavior Proof

- Environment: Windows, Docker Desktop 29.6.2, current `origin/main`
base, head image, and a real Docker bridge peer.
- Exact command / steps: run the focused security/runtime tests, Compose
configuration checks, then `bash e2e/docker-bind-security.sh` against
the current base and head image definitions.
- Observed result: `76 passed, 1 skipped`; the built-image harness
passed base/head health, local completion, non-loopback isolation,
authenticated public HTTP/WebSocket controls, loopback runtime-slim
configuration, and direct Rust-core loading.
- Not tested: runtime-slim proxy service health on this host, because
its unrelated ONNXRuntime import path exits with code 139; the full
hosted matrix and a future published GHCR image remain CI-owned.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Unsafe override required: no
- Minimum rollout channel: next release containing the Docker and
launcher changes
- Stable/default behavior changed: published images and maintained local
Docker launchers now use loopback-safe publication defaults
- Kill switch / disable path: revert the deployment-artifact changes
before release; deliberate public deployments remain explicitly
operator-controlled
- Qualification impact: Docker users relying on all-interface default
publication must opt into public publication and configure the existing
proxy token
- Deliberate public deployment: use an explicit public host publication,
set `HEADROOM_PROXY_TOKEN`, and send the matching bearer or
`X-Headroom-Proxy-Token` credential
- Local deployment: use the maintained loopback publication and no
mandatory token
- Rollback path: revert the deployment-artifact and documentation
changes together

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [x] I have added tests that prove the behavior is effective
- [x] Focused tests pass locally with these changes
- [x] I did **not** edit `CHANGELOG.md`; release-please generates
release notes from the conventional PR title

## Additional Notes

The issue fixture remains provenance for the unauthenticated
publication. Native
`headroom proxy` defaults and the application-level authentication
owners are
unchanged.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-09-24 23:58:46 -05:00
JD Davis eafdf11a2c fix(docker): ship Bedrock auth and current registry (#2982)
## Description

Fixes #1551 and #1692.

Every published Headroom Docker image now installs the existing
`bedrock` extra, so `--backend bedrock` can authenticate with temporary
STS, SSO, and credential-process credentials instead of failing because
`botocore` is absent.

Public Docker instructions now consistently use
`ghcr.io/headroomlabs-ai/headroom`. Several still pointed at the old
personal package, which is frozen at 0.27.0 and caused users to report
that no latest image existed.

## Type of Change

- [x] Bug fix
- [ ] New feature
- [ ] Breaking change
- [x] Documentation update
- [x] Build / CI

## Changes Made

- Add `bedrock` to the standalone Dockerfile default extras.
- Add `bedrock` to all nine root/code/slim/nonroot bake targets.
- Replace obsolete personal GHCR references in README, llms.txt, Compose
guidance, testing guidance, and wiki docs.
- Add release contract tests for Bedrock dependencies and the current
organization registry.

## Testing

- [x] Focused Docker release and Bedrock preflight tests pass.
- [x] Full updater suites pass: 69 tests.
- [x] `uv run ruff check tests/test_release_workflows.py`
- [x] `docker buildx bake --print`
- [x] `git diff --check`

## Real Behavior Proof

Before this change, every published bake target installed only `proxy`
or `proxy,code`, so `AWS_SESSION_TOKEN` selected an unavailable botocore
path. Public copy-paste commands also referenced
`ghcr.io/chopratejas/headroom`, which the existing migration code and
changelog identify as frozen at 0.27.0.

After this change, all nine parsed bake targets install `bedrock`; the
regression resolves that package extra and confirms `boto3` plus
`botocore`. Every public Docker instruction covered by the contract
names `ghcr.io/headroomlabs-ai/headroom`.

## Runtime Rollout Safety

This changes image contents and documentation only; proxy routing and
non-Docker installs are unchanged. Static AWS credentials remain
unaffected. Existing manifests using the deprecated image continue to be
migrated by the established install-state logic. Rollback is a
Docker/bake extras and documentation revert.

## Review Readiness

- [x] Two related Docker blockers batched in one PR
- [x] Regression coverage included
- [x] No unrelated lockfile changes
- [x] Ready for review
2026-08-13 15:06:21 -05:00
JervisandJerrettDavis a63d235e7e Fix Dockerfile (#2337)
## Description

Optimize the Docker build cache to ensure that Rust/Cargo dependencies
are cached correctly when installing Python packages, thereby speeding
up subsequent image builds.

Closes #

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Dockersfile

## Review Readiness

- [ ] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [ ] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-07-17 16:12:20 -07:00
Vinay Gupta 38074888ac fix(docker): report source build version (#1862)
## Description

Closes #1858

Docker/Compose source builds could report stale or misleading version
information: the dashboard initially rendered a hardcoded `v0.3.0`, then
`/health` replaced it with installed package metadata, which can be
stale when building locally from `main` without release metadata in the
image.

This change makes source Docker Compose builds report an explicit
source-build identity, removes the stale dashboard fallback, and keeps
CLI/doctor version checks from treating source-build labels as
release-version drift.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Add `HEADROOM_VERSION` / `HEADROOM_BUILD_VERSION` runtime version
overrides and optional packaged `_build_info.py` metadata.
- Teach Docker Compose source builds to pass a `source-build` sentinel
that the Dockerfile expands to `source-build+g<sha>` when git metadata
is available, or `source-build+sha256.<digest>` otherwise.
- Keep release/published image builds on normal package metadata when
`HEADROOM_BUILD_VERSION` is unset.
- Include only minimal `.git` metadata in the Docker build context so
the source-build label can identify the checkout without copying git
objects.
- Treat source-build labels and raw hashes as non-release labels in
`wrap` and `doctor`, avoiding false stale-proxy restarts and drift
warnings.
- Replace the dashboard hardcoded `0.3.0` fallback with `loading` /
`unknown` and format non-release build labels without a `v` prefix.
- Include the runtime version in proxy startup logs, `/health`,
`/livez`, and OTEL service version reporting.

## Testing

- [x] Unit tests pass (`pytest` in GitHub CI)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
GitHub CI: all checks passing
- CI: build, build-wheel, lint, test shards, test-extras, test-agno, test-dashboard-ui
- Docker: docker-native-e2e, docker-wrap-e2e, docker-init-e2e
- Native wrappers: macOS, Windows, Ubuntu
- Security: CodeQL, gitleaks, pip-audit
- Governance: template, label, merge-conflicts, commitlint

$ HEADROOM_REQUIRE_RUST_CORE=false PYTHONPATH=/Users/vinaygupta/Desktop/git/headroom-fix-1858-version-mismatch pytest tests/test_package_init_lazy.py::test_version_prefers_explicit_build_env tests/test_package_init_lazy.py::test_version_label_helpers_only_prefix_release_versions tests/test_package_init_lazy.py::test_version_uses_packaged_build_metadata tests/test_package_init_lazy.py::test_observability_version_uses_runtime_version tests/test_docker_compose_persistence.py tests/test_cli_doctor.py::TestProxyLiveness::test_up_leaves_source_label_unprefixed tests/test_cli_doctor.py::TestVersionDrift::test_non_release_version_labels_skip_drift_comparison tests/test_cli/test_wrap_persistent.py::test_proxy_version_restart_ignores_non_release_source_labels tests/test_proxy_dashboard_stats_cache.py::test_dashboard_uses_cached_stats_and_lazy_history_feed_polling -q
13 passed, 1 warning

$ uvx ruff==0.15.17 check .
All checks passed!

$ uvx ruff==0.15.17 format --check .
1058 files already formatted

$ uvx mypy==1.20.2 headroom --ignore-missing-imports
Success: no issues found in 407 source files

$ git diff --check
# no output

$ docker compose config
# resolved headroom-proxy build args include HEADROOM_BUILD_VERSION: source-build

$ HEADROOM_BUILD_VERSION=6266a1d docker compose config
# explicit override is preserved as HEADROOM_BUILD_VERSION: 6266a1d

$ docker build --check --build-arg HEADROOM_BUILD_VERSION=source-build .
Check complete, no warnings found.
```

## Real Behavior Proof

- Environment: macOS local checkout, Python 3.13.5, Docker Desktop
builder `desktop-linux`, plus GitHub Actions CI.
- Exact command / steps: `docker compose config`,
`HEADROOM_BUILD_VERSION=6266a1d docker compose config`, and `docker
build --check --build-arg HEADROOM_BUILD_VERSION=source-build .`.
- Observed result: Compose defaults the top-level `headroom-proxy` build
arg to the `source-build` sentinel, preserves explicit overrides, and
Dockerfile syntax/check validation passes for the source-build path.
- Not tested: Full end-to-end release publishing flow; this PR only
changes local/source-build reporting.
- CI proof: GitHub Actions completed successfully across Docker E2E, CI
test shards, lint/type checks, native wrapper checks, security checks,
and PR governance.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally/CI with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A

## Additional Notes

Docs and changelog are N/A for this runtime-reporting bug fix. The PR is
open and ready for review with all GitHub checks passing.
2026-07-08 13:32:04 -05:00
Serge ARADJandClaude 5912d65674 fix(docker): persist session history across container revisions (#1118)
## Description

Session history (savings ledger, memory.db, session stats, telemetry)
stored in `~/.headroom` was lost whenever a new container started —
either a Docker restart or a new Azure Container Apps revision pulling
`:latest`. No volume was mounted for that path, so every run began with
a blank workspace.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **`Dockerfile`** — adds `VOLUME ["/home/nonroot/.headroom"]`. The
directory already exists with correct `nonroot` ownership. Bare `docker
run` now gets an anonymous volume as fallback rather than writing
silently to the ephemeral container layer.
- **`docker-compose.yml`** — mounts named `headroom_workspace` volume at
`/home/nonroot/.headroom` for the `headroom-proxy` service. Named
volumes survive `docker compose pull && docker compose up` on any local
Docker host (Windows, Mac, Linux), matching the pattern already used by
`qdrant_data` and `neo4j_data`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ docker compose pull && docker compose up -d
[+] Pulling 1/1
 ✔ headroom-proxy Pulled  14.2s
[+] Running 3/3
 ✔ Container headroom-neo4j     Running
 ✔ Container headroom-qdrant    Running
 ✔ Container headroom-proxy     Started

$ ls -lh ~/.headroom/
total 56K
-rw-r--r-- 1 nonroot nonroot  18K Jun 18 09:14 proxy_savings.json
-rw-r--r-- 1 nonroot nonroot  12K Jun 18 09:14 memory.db
-rw-r--r-- 1 nonroot nonroot   3K Jun 18 09:14 session_stats.jsonl

$ docker run -d ghcr.io/chopratejas/headroom:latest
a3f7c2e1b849...

$ docker inspect a3f7c2e1b849 | jq '.[].Mounts'
[
  {
    "Type": "volume",
    "Name": "a3f7c2e1b849_headroom_workspace",
    "Source": "/var/lib/docker/volumes/a3f7c2e1b849_headroom_workspace/_data",
    "Destination": "/home/nonroot/.headroom",
    "Mode": "",
    "RW": true,
    "Propagation": ""
  }
]
```

## Real Behavior Proof

- Environment: Docker Desktop 4.x, docker compose v2, linux/amd64
- Exact command / steps: `docker compose pull && docker compose up -d`
- Observed result: `proxy_savings.json` from first run present after
pull+restart with new image digest
- Not tested: Azure Container Apps volume mount (ACA attach tested via
`VOLUME` declaration only; full ACA revision rollout not verified
locally)

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

`docker/docker-compose.native.yml` bind-mounts host `~/.headroom`
directly — unaffected.

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-06-22 11:08:17 -05:00
Yasser Sheikh 0dc2e1cb3f feat(bedrock): cross-region + Converse compression; bundle proxy binary in images (#999)
## Description

The native Bedrock path (Phase D) compresses + signs
Anthropic-on-Bedrock requests, but
two real-world cases slipped through, and the native binary that powers
it was never
shipped. This PR closes those gaps as a focused set of give-backs.

Aligns with the Rust migration plan (see below).

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **Cross-region inference-profile detection** via a new
`bedrock::vendor` module
(`canonical_vendor()`), following the design proposed in #953: strip a
known geo prefix
(`eu.`/`us.`/`apac.`/`global.`) then match the canonical vendor.
Geo-prefixed Anthropic
profiles (`eu.anthropic.…`) now get live-zone compression instead of
being silently
  skipped; geo-prefixed non-Anthropic vendors stay correctly excluded.
- **Converse-body compression (two parts)**:
1. `run_anthropic_compression` no longer bails to passthrough when the
body lacks an
InvokeModel `anthropic_version` envelope; envelope re-emit stays gated
on successful parse.
2. The **live-zone dispatcher now recognizes Bedrock Converse content
blocks**. Converse
blocks carry no `type` discriminator (the variant is the key: `{"text":
…}` vs
Anthropic's `{"type":"text","text":…}`), so real Converse user-message
text was still
passing through uncompressed. A typeless block whose `text` is a JSON
string now routes
through the same surgical text path. Anthropic blocks always carry
`type`, so the
Anthropic path is byte-for-byte unchanged; non-text Converse blocks
(`{"image":…}`,
     `{"toolUse":…}`) stay unrecognized and no-op.
- **Correct `/converse` upstream routing**: the non-streaming handler
resolved the upstream
action from a hard-coded `"invoke"`, so `/converse` requests were
forwarded to Bedrock's
`/invoke` endpoint. It now resolves the action from the inbound path
(`extract_invoke_action`),
mirroring the streaming handler's `extract_streaming_action`. SigV4
signs the same URL it
  forwards, so the signature stays consistent.
- **`aws-config` `sso` feature**: SSO profiles now resolve through the
default credential
chain for SigV4 — the credential chain in `docs/bedrock.md` already
promised SSO; this
  makes the code match.
- **Ship the `headroom-proxy` binary in published images**
(`Dockerfile`): built in the
builder stage (`--locked`, with the cargo registry cache mounted at
`CARGO_HOME`) and
  copied into both the debian and distroless runtime images.
- **Docs** (`docs/bedrock.md`): document cross-region inference profiles
and a "Running the
proxy" section. AWS credentials mount at `/home/nonroot/.aws` (the
default nonroot image
home) where the SDK looks for `~/.aws`, with a note on the root-image
alternative.

## Related issues

- Closes #976 — ship the `headroom-proxy` binary in published images
(this PR implements
  the exact fix proposed there).
- Addresses the **cross-region inference-profile** half of #953 via its
proposed
`canonical_vendor()` design. Non-Anthropic vendor compression parity
(Nova/GLM/MiniMax/
Kimi) is the natural follow-up — `bedrock::vendor` is the shared
resolver it can build on.
- Extends the native Bedrock InvokeModel compression requested in #734
(the Bedrock slice of
  #510) to cross-region profiles and Converse bodies.
- Partially enables #181 (native, Python-free packaging): the native
binary now ships in the
  images, though full Python-free distribution remains out of scope.

## Alignment with the Rust migration plan

Per `docs/spec/022-rust-migration.md`, the migration is **proxy-first**:
`headroom-proxy` is
the deployable Rust artifact, native routes replace Python passthroughs
one at a time
(Stage 4 = provider expansion, Bedrock included), and the binary is
meant to be "built,
tested, and **released together with the Python package**." Two ways
this PR advances that:

- The binary-in-images change makes the codebase do what the spec
already states (ship the
artifact) — closing the gap that forced downstreams to build from
source.
- Hardening the native Bedrock route (cross-region, Converse routing +
body compression) is
exactly the Stage-4 provider-expansion work, keeping the native path at
parity with real
  traffic so it can be the default rather than a passthrough.

## Testing

- [x] Unit tests pass (`cargo test -p headroom-core -p headroom-proxy` —
full suites, 0 failures)
- [x] Linting passes (`cargo clippy -p headroom-core -p headroom-proxy
--all-targets -- -D warnings`)
- [x] Formatting passes (`cargo fmt -- --check`)
- [x] New tests added — `bedrock::vendor` (foundation +
inference-profile matching),
`extract_invoke_action` + converse upstream URL, and live-zone Converse
text-block routing
(`block_has_string_text_field`, converse-vs-anthropic dispatch
equivalence).
- [x] Manual testing performed

### Test Output

```text
$ cargo test -p headroom-core -p headroom-proxy   # all suites: ok, 0 failed
$ cargo clippy -p headroom-core -p headroom-proxy --all-targets -- -D warnings   # Finished, no warnings
$ cargo fmt -- --check                            # clean
# image validation (local, proxy/code extras):
$ docker build --target runtime ...      # debian: /usr/local/bin/headroom-proxy, --help OK
$ docker build --target runtime-slim ... # distroless: binary links + --help OK
```

## Real Behavior Proof

- Environment: native Bedrock proxy against `bedrock-runtime.eu-west-2`,
SSO profile, model
  `eu.anthropic.claude-haiku-4-5-20251001-v1:0`.
- Exact command / steps: POST a large multi-turn Converse body to
`/model/eu.anthropic.claude-haiku-4-5-20251001-v1:0/converse`;
separately build the
`runtime` + `runtime-slim` targets and run
`/usr/local/bin/headroom-proxy --help`.
- Observed result: before — `bedrock_compression_skipped` (geo-prefixed
id not recognized),
forwarded uncompressed to the wrong `/invoke` upstream; after —
geo-prefixed id recognized,
`/converse` forwarded to the `/converse` upstream, live-zone dispatcher
compresses the
Converse user-message text, measurable token savings. Images contain a
runnable
  `headroom-proxy` in both variants.
- Not tested: non-Anthropic vendor compression parity (#953 follow-up);
Converse
`toolResult` nested-text compression (follow-up — only top-level
Converse text blocks
  compress today).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

- An earlier revision flipped the EventStream `Accept` default
(`*/*`/absent → passthrough);
**dropped** — `*/*` is what most clients (incl. reqwest and the proxy's
own metrics tests)
send while expecting SSE, so forcing passthrough breaks the standard SSE
path.
- The binary build adds the native-proxy compile to the image build;
happy to gate it behind
  a build arg if maintainers prefer it opt-in.
- Addressed a Copilot review round: corrected the `/converse` upstream
routing, the stale
`run_anthropic_compression` comment, the Dockerfile cargo cache mount +
`--locked`, and the
  nonroot AWS-credentials docs example.
2026-06-16 09:45:24 -05:00
Patrick AandTejas Chopra 9579567b7d chore(deps): loosen over-pinned constraints and add upper bounds (#538)
## What

Loosen over-pinned Python dependency constraints and add missing upper
bounds in `pyproject.toml`. Also bump the neo4j Docker image and uv
builder version.

## Why

Several dependencies had constraints that either blocked security
patches or allowed silent major-version jumps:

- `litellm==1.82.3` was an exact pin — every security patch release
requires a manual lockfile bump
- `transformers`, `sentence-transformers` had no upper bound and have
already crossed major version boundaries without a constraint gate
- `neo4j>=5.20.0` had no upper cap; the driver has already reached 6.x
in the wild
- `mem0ai>=0.1.100` had a pre-1.0 floor while the locked version is
already 1.0.11
- `langchain-core`, `langchain-openai`, `qdrant-client`, `uvicorn` had
no upper bound on a range with active major-version churn
- `docker-compose.yml` pinned neo4j at `5.15.0`, which is 11 patch
releases behind the current 5.x LTS
- `Dockerfile` pinned uv at `0.11.16`; latest stable is `0.11.18`

## How

Constraint changes only — no code changes, no `uv lock --upgrade`. The
existing locked versions all satisfy the new bounds (we added caps, not
floors). `uv` re-resolved the lockfile to format revision 3 (adds
`upload-time` metadata fields) and cleaned up the defunct `llmlingua`
extra entries.

| Dependency | Before | After |
|---|---|---|
| `litellm` | `==1.82.3` | `>=1.82.3,<2.0` |
| `transformers` | `>=4.30.0` | `>=4.30.0,<6.0` |
| `sentence-transformers` | `>=2.2.0` | `>=2.2.0,<6.0` |
| `neo4j` | `>=5.20.0` | `>=5.20.0,<7.0` |
| `mem0ai` | `>=0.1.100` | `>=1.0.0,<2.0` |
| `langchain-core` | `>=0.2.0` | `>=0.2.0,<4.0` |
| `langchain-openai` | `>=0.1.0` | `>=0.1.0,<2.0` |
| `qdrant-client` | `>=1.9.0` | `>=1.9.0,<2.0` |
| `uvicorn` | `>=0.23.0` | `>=0.23.0,<1.0` |
| neo4j Docker image | `5.15.0` | `5.26` |
| uv (Dockerfile ARG) | `0.11.16` | `0.11.18` |

## Breaking changes

None. All currently installed versions fall within the new ranges.
Installers that previously resolved `litellm` to an older exact pin may
now resolve newer patch releases — which is the desired behavior.

---------

Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
2026-06-08 22:06:24 -08:00
pratikbin 0be2fcb652 chore(docker): remove pinned digests, use floating tags
Digest pinning provided reproducibility at the cost of manual upkeep.
Switching to floating tags (python:3.13-slim, distroless/python3-debian13)
lets Dependabot / CI always pull the latest patched image.
2026-05-28 15:12:11 +05:30
pratikbin e6bf7a03fe fix(docker): upgrade base images to Python 3.13 / debian13
- python:3.11-slim → python:3.13-slim (builder + runtime-slim-base)
- distroless/python3-debian13 already debian13; was mismatched against
  Python 3.11 build stage causing ModuleNotFoundError on pydantic_core
  _pydantic_core.cpython-311 .so in Python 3.13 runtime
- uv 0.6.17 → 0.11.16
- Refresh pinned digests for both base images (2026-05-28)

Affected tags: slim, slim-nonroot, code-slim, code-slim-nonroot
PyO3 0.22.6 (in Cargo.lock) supports Python 3.13.
2026-05-28 13:01:45 +05:30
chopratejas 6f2c0a8400 fix(ci): rustls-everywhere — eliminate openssl-sys from build tree
# Root cause of the wheel-build cascade

We have shipped 5 release-pipeline hot-fixes in 12 hours, each
addressing a different symptom of the same architectural problem:

1. PR #363 — npm artifact downloads + tried `yum openssl-devel`
2. PR #367 — vendored OpenSSL in `headroom-proxy` + dropped Intel mac
3. PR #369 — Debian-cross perl install (`perl` not `libipc-cmd-perl`)
4. PR #370 — moved `openssl/vendored` from headroom-proxy to headroom-py
5. (this PR) — ELIMINATE OpenSSL entirely

Each fix exposed a different missing system package or feature flag in
a different build surface (manylinux x86_64 vs aarch64-cross-Debian vs
macOS Intel vs e2e/wrap Dockerfile vs e2e/init Dockerfile vs main
Dockerfile vs devcontainer). We were playing whack-a-mole because every
Cargo dep change to the OpenSSL surface required matching system-package
updates in 6+ different Dockerfiles and workflows, and the PR-level CI
didn't exercise all of them.

# Why this PR is the structural fix

`fastembed` exposes clean rustls feature flags:
- `hf-hub-rustls-tls`               (replaces default `hf-hub-native-tls`)
- `ort-download-binaries-rustls-tls` (replaces default `…native-tls`)

By disabling fastembed's default features and enabling the rustls
variants explicitly, we remove `native-tls` (and therefore `openssl-sys`,
`openssl`, `openssl-src`, perl modules, OpenSSL build-time deps,
vendored OpenSSL ~30s build cost) from the entire workspace dep tree.

Verified locally:

    $ cargo tree -p headroom-py -i openssl-sys
    error: package ID specification `openssl-sys` did not match any packages

    $ cargo tree -p headroom-py -i native-tls
    error: package ID specification `native-tls` did not match any packages

    $ cargo build --release -p headroom-py
    Finished `release` profile [optimized] target(s) in 25.57s

(Down from 1m+ with vendored OpenSSL.)

# Cleanups enabled by this change

- crates/headroom-py/Cargo.toml — dropped the `openssl/vendored`
  workaround from PR #370.
- crates/headroom-proxy/Cargo.toml — same dep removed.
- e2e/wrap/Dockerfile — dropped `yum install openssl-devel pkgconfig
  perl-IPC-Cmd`. Comment retained explaining why.
- e2e/init/Dockerfile — same.
- Dockerfile (main) — dropped `pkg-config libssl-dev` from apt-get.
- .devcontainer/Dockerfile — dropped `pkg-config libssl-dev`.
- .github/workflows/release.yml — removed the entire before-script-linux
  block (perl install probe + multi-package-manager dispatch + fail-loud
  assertion). No longer needed.

# Regression gate

Three new structural tests in tests/test_release_workflows.py:

- test_no_openssl_sys_in_wheel_build_tree — runs `cargo tree -p <crate>
  -i openssl-sys` for headroom-py / headroom-proxy / headroom-core. If
  openssl-sys reappears (a future native-tls enabler creeping in via a
  new dep), this fails AT PR TIME with an actionable message.
- test_no_native_tls_in_wheel_build_tree — same shape, native-tls is
  the proximate cause.
- test_fastembed_uses_rustls_features — checks the Cargo.toml so a
  future "let me bump fastembed and forget the features" doesn't
  silently re-introduce OpenSSL.

Plus two cleanup gates:
- test_dockerfiles_no_longer_install_openssl_devel
- test_release_yml_does_not_install_openssl_or_perl_for_wheels

All 13 release-workflow tests pass. `make ci-precheck` PASSED.

# What this teaches us about rollouts (per user's ultrathink ask)

The 5-fix cascade exposed three meta-problems:

1. PR checks don't block merges. PR #370 had docker-init-e2e,
   docker-wrap-e2e, docker-native-e2e all FAILED yet got merged.
   Branch protection should require these checks. Operator action
   needed (cannot fix in code).

2. Local validation is misleading. `cargo build -p headroom-py` from
   the workspace root used the workspace lockfile and looked green;
   CI did fresh resolution against headroom-py's manifest alone where
   the feature wasn't enabled. Lesson: verify structural invariants
   with `cargo tree -e features` before trusting that a build "works."

3. 6+ build surfaces with independent system-dep state. Every Cargo
   change required matching updates in 6 places. The structural answer
   (this PR) is to NOT depend on system OpenSSL at all. Where structural
   fixes are not possible, the answer is a single shared
   scripts/install-rust-build-deps.sh — but with this PR there's
   nothing left to install.
2026-05-03 23:26:04 -07:00
chopratejas 2ae57725e7 fix(ci): pre-install rustfmt+clippy components in all Dockerfiles
rust-toolchain.toml at the repo root requests
`components = ["rustfmt", "clippy"]`. When `pip install -e .` invokes
maturin → cargo from inside `/workspace`, rustup auto-detects the
toolchain file and tries to add the missing components on top of the
`--profile minimal` install we did earlier. The install fails with:

    info: downloading component clippy
    info: rolling back changes
    error: failed to install component: 'rustfmt-preview-x86_64-unknown-linux-gnu',
           detected conflict: 'bin/cargo-fmt'

— rustup's auto-component install hits a `bin/cargo-fmt` conflict
inside the toolchain it just installed. The fix is to install the
required components up-front via `-c rustfmt -c clippy`, so the
toolchain matches what rust-toolchain.toml expects on first cargo run
and rustup never needs to mutate it.

Applied to: Dockerfile (main), e2e/init/Dockerfile, e2e/wrap/Dockerfile,
.devcontainer/Dockerfile. Also pinned the main Dockerfile's toolchain
from `stable` to `1.95.0` so all four images now match the lockfile
(prevents drift if rust-toolchain.toml is bumped later).
2026-05-03 13:35:01 -07:00
chopratejas 2a91cbb4b4 refactor: single-wheel maturin build backend (fixes #355)
Eliminates the dual-package architecture that was the root cause of #355.
`pip install headroom-ai` now produces ONE wheel containing both the Python
source (headroom/*.py) and the compiled Rust extension (headroom/_core.so).
No more separate `headroom-core-py` package, no more chicken-and-egg with
PyPI publication, no more wheelhouse / PIP_FIND_LINKS / composite-action
plumbing in CI.

This is the canonical pattern used by cryptography, polars, ruff,
pydantic-core, and other Rust-as-core Python packages. Honors the
"Rust as core engine" direction.

## What changed

- pyproject.toml: `[build-system]` swapped from hatchling to maturin.
  `[tool.hatch.*]` deleted; `[tool.maturin]` added pointing at
  `crates/headroom-py/Cargo.toml` for the cdylib. `python-source = "."`
  picks up the root `headroom/` package directly (dashboard HTML
  templates and other non-Python files included automatically).
- crates/headroom-py/pyproject.toml: deleted. The crate is no longer a
  separate published package; its Cargo.toml stays as the cdylib build
  target invoked via `[tool.maturin] manifest-path`.
- crates/headroom-py/python/: deleted (placeholder layout for the old
  separate package).

## CI updates

- ci.yml: `test` / `test-extras` / `test-agno` jobs simplified — Rust
  toolchain set up before `pip install -e .` (which now invokes maturin
  via build-system). Removed the "build wheel + symlink .so" dance.
  `build` job swapped from `python -m build` (hatch) to
  `maturin build` + `maturin sdist`.
- release.yml: collapsed dual-package matrix into one. New `build-wheels`
  matrix produces cross-platform wheels for cp310/11/12/13 ×
  {linux x86_64, linux aarch64, macos x86_64, macos aarch64}. New
  `collect-dist` aggregator merges artifacts. publish-pypi consumes the
  merged dist.
- init-native-e2e.yml: dropped windows-latest from the matrix —
  upstream `esaxx-rs` (/MT) and `ort-sys` (/MD) link with conflicting
  MSVC C runtime libraries, so the Rust extension cannot build for
  win_amd64 today. Tracked as a follow-up; not a blocker for Linux+macOS.
- headroom-e2e-setup: composite action now sets up Rust toolchain +
  Swatinem/rust-cache before `pip install -e .[proxy]`.
- eval.yml, publish.yml, rust.yml: same pattern — rust toolchain before
  install. rust.yml's wheels job builds from root pyproject.toml (no
  more `-m crates/headroom-py/Cargo.toml`).
- e2e/init/Dockerfile, e2e/wrap/Dockerfile: install rust + maturin in
  the build stage; copy `crates/` + workspace `Cargo.toml/lock` so the
  install can build the extension. Dropped `HEADROOM_REQUIRE_RUST_CORE=false`
  from wrap-e2e — the image now ships the full Rust core.
- Dockerfile (main): simplified — no more Layer 2/3 dance with
  `headroom-core-py` install + symlink. Single `uv pip install` builds
  + installs everything.
- .devcontainer/Dockerfile: rust toolchain + libssl-dev + maturin
  added so `uv sync` builds the extension inside the devcontainer.

## Lockfile + script

- uv.lock: regenerated. No `headroom-core-py` entries remain.
- scripts/build_rust_extension.sh: simplified from a symlink-into-tree
  workaround to a thin wrapper around `pip install -e .`. The maturin
  build-backend handles placement automatically.

## Local validation (all green on macOS aarch64)

1. Clean venv `pip install -e .` → `from headroom._core import …` works.
2. `maturin build --release` → 13.8 MB wheel, 336 files including
   `headroom/_core.cpython-311-darwin.so` (32 MB cdylib) and
   `headroom/dashboard/templates/dashboard.html`.
3. `pip install <wheel>` in fresh venv → import works.
4. Wheel contents verified via `unzip -l`.
5. `pytest tests/test_transforms/test_diff_compressor.py` — 29 passed.
6. `pytest tests/test_relevance.py` — 30 passed.
7. `cargo build --workspace` + `cargo test --workspace` — all green.
8. `make ci-precheck` — 176 Python tests + Rust + commitlint green.

## Migration notes

Users on `pip install headroom-ai` get the Rust core automatically
(linux + macos wheels). sdist installs require rust toolchain available
locally — pip will build via maturin.

Closes #355
Supersedes #357 (workarounds-based fix abandoned in favor of
architectural fix)
2026-05-03 13:16:41 -07:00
chopratejas 4bf559d5b5 fix(ci): install patchelf for maturin wheel-link repair
The Release workflow's multi-arch publish-docker job failed after
78 minutes of QEMU-emulated arm64 cargo compilation. Maturin's
wheel-link repair step needs `patchelf` to bundle external
shared libraries (libssl.so.3, libcrypto.so.3, libzstd.so.1)
into the wheel and rewrite their RPATH:

    🔗 External shared libraries to be copied into the wheel:
        libssl.so.3 => /usr/lib/aarch64-linux-gnu/libssl.so.3
        libzstd.so.1 => /usr/lib/aarch64-linux-gnu/libzstd.so.1.5.7
        libcrypto.so.3 => /usr/lib/aarch64-linux-gnu/libcrypto.so.3
    💥 maturin failed
      Caused by: Failed to execute 'patchelf', did you install it?

Compounding chain:
1. PR #350 added pkg-config + libssl-dev to unblock the cargo build
   (openssl-sys couldn't find OpenSSL headers).
2. That made Cargo dynamically link to libssl.
3. Maturin then needs patchelf to rewrite the wheel's RPATH so the
   bundled .so references resolve at runtime.
4. patchelf was never installed → fail.

Why this didn't surface in PR CI: docker-native-e2e builds only
the host platform (amd64). The Release workflow's docker-bake
builds linux/amd64 + linux/arm64 via setup-qemu-action, and the
arm64 emulation chain hits the patchelf path (different bundling
heuristic from amd64).

Follow-up that's NOT in this hotfix:
The 78-minute QEMU compile is the bigger structural issue. Switching
the Release workflow to native arm64 runners (`runs-on:
ubuntu-24.04-arm`) would cut that to ~5 min. Filing separately.

Run that failed: 25268839539
2026-05-02 22:34:12 -07:00
chopratejas 6b15802c0b fix(ci): A0 verify must run outside /build cwd
The build-stage verify kept failing in PR #350 CI with
"ModuleNotFoundError: No module named 'headroom._core'" even after
the install order was correct. Diagnostic dump (commit 28a4883)
proved why:

    headroom.__file__ = /build/headroom/__init__.py
    headroom.__path__ = ['/build/headroom']

WORKDIR /build puts cwd at the front of sys.path for python -c.
Python resolves `import headroom` to /build/headroom/ — the source
tree COPYd in by Layer 3 — instead of
/usr/local/lib/python3.11/site-packages/headroom/ where the wheel
installed _core.so. The source tree has no _core.so, so the import
falsely fails.

Build-time-only quirk: production startup runs the proxy from a
different cwd where site-packages wins. The customer's box that
motivated A0 was hitting a different failure mode entirely (no
_core.so in the venv at all).

Fix: cd /tmp && python -c ... — /tmp has no headroom/ directory, so
import resolution falls through to site-packages, matching production
order. Removed the diagnostic preamble; it served its purpose.
2026-05-02 20:15:32 -07:00
chopratejas 28a4883421 chore(ci): dump site-packages state before A0 build verify
Diagnostic step in the Dockerfile builder: list site-packages/headroom/
contents, run pip show -f on both headroom-core-py and headroom-ai,
print sys.path and headroom.__path__ before the import-verify. Lets us
see exactly what's on disk when A0's build-time verify keeps failing
in PR #350 CI. Will be removed once the wheel install order issue is
diagnosed.
2026-05-02 20:08:34 -07:00
chopratejas f1aa12cebf fix(ci): reorder Dockerfile so wheel installs before headroom-ai
PR #350 CI: docker-native-e2e's wheel install succeeded but the
build-stage verify (`from headroom._core import hello`) failed with
`ModuleNotFoundError: No module named 'headroom._core'`. Same failure
mode the customer hit in production (Finding #2) — but in CI we have
the full layer trace.

Root cause: the headroom-core-py wheel claims ownership of both
`headroom/__init__.py` (stub from maturin's python-source layout)
AND `headroom/_core.cpython-*.so`. The previous Dockerfile installed
headroom-ai FIRST (which laid down the real `headroom/` tree), then
the wheel SECOND with `--force-reinstall`. pip's --force-reinstall
uninstalls the wheel's previously installed files before reinstalling
— but the wheel's stub `__init__.py` had already overwritten
headroom-ai's at first install. Net result: pip deleted
`headroom/__init__.py` and `headroom/_core.so` ownership records
got into a state where the .so wasn't present after the install.

Fix: swap the order. Install the wheel first (lays down stub
`__init__.py` + `_core.so`), then install headroom-ai (overwrites the
stub with the real `__init__.py` and adds the rest of the
`headroom/` tree). `_core.so` survives because headroom-ai doesn't
claim ownership of it. Drop `--force-reinstall` from the wheel step
since nothing is installing the wheel before it.

This is the exact failure A0 was designed to catch — a deployment
that ships without `_core` working. CI is now serving as a
regression gate for the production install path.

The remaining 3 PR check failures (validate × 3 / Dev Containers)
are environmental: the runner's PyPI mirror (`pypi.netflix.net`)
times out fetching `cuda-bindings==12.9.4` /
`nvidia-cuda-cupti-cu12==12.8.90` / `safetensors==0.7.0`. These come
from `headroom-ai[dev]` → `sentence-transformers` → `torch` → CUDA
deps. Not caused by the realignment branch; the post-create script
needs a `--extra dev-light` profile or the mirror needs the packages
cached. Tracking separately.
2026-05-02 18:55:09 -07:00
chopratejas 55dfc19e1d fix(ci): unblock A0 Docker e2e — install pkg-config + opt out wrap-e2e
Two CI failures introduced by Hotfix-A0's deployment-stage smoke test:

1. docker-native-e2e: the new maturin step in the builder stage failed
   with "Could not find openssl via pkg-config". The workspace
   transitively depends on `openssl-sys` (via reqwest's native-tls
   path in some dep chain). The previous Dockerfile only installed
   `build-essential`/`g++`/`curl`/`ca-certificates` — enough for the
   proxy binary build because cached target/ artefacts already had
   openssl-sys compiled, but the fresh maturin invocation hits a cold
   build and needs the dev headers. Add `pkg-config` + `libssl-dev`.

2. docker-wrap-e2e: this image is a `node:22-bookworm` base that
   installs headroom in editable mode for CLI-routing-only tests
   (aider, codex, openclaw via the wrap subcommand). It deliberately
   does NOT build the Rust extension. After A0, the proxy
   `lifespan` startup refuses to start when `headroom._core` can't
   import — so the wrap-e2e proxy port never opens, the harness's
   /health check times out, and the test fails. The wrap-e2e scope
   doesn't cover compression behaviour, so set
   `HEADROOM_REQUIRE_RUST_CORE=false` to start in degraded
   Python-only mode. Compression is exercised end-to-end by the
   smoke-test and docker-native-e2e jobs which build via the main
   Dockerfile.

The remaining 3 PR check failures (validate * 3) were transient
PyPI download failures (`nvidia-cuda-cupti-cu12==12.8.90`,
`safetensors==0.7.0`) — unrelated to the realignment branch; they
need a re-run, not a code change.
2026-05-02 18:43:09 -07:00
chopratejas 00ab1ea74d fix: A0 — fail-loud rust core deployment smoke test
Production incident (Finding #2 of HEADROOM_PROXY_LOG_FINDINGS_2026_05_03.md):
on this customer's deployment the Rust extension `headroom._core` was
never installed into the runtime Docker image. Diff compression failed
54 times in a single day; "Optimization failed: ModuleNotFoundError" hit
379 times. The failure rate climbed every day and reached ~223/day on
2026-05-03 — effectively 100% of requests on the Rust path. Every Rust
PR we'd merged (MessageScorer, ICM, DiffCompressor, etc.) was providing
zero customer value because the module wasn't loadable at all.

Root cause: the Dockerfile builder stage installed Python deps and the
in-tree `headroom-ai` package but never ran `maturin build` for the
`headroom-py` crate, so the runtime image shipped without `_core.so`.
The Python proxy continued to start because the extension's absence is
caught and routed through Python-only fallbacks that either silently
no-op or raise per-request.

This change makes that mode impossible by default:

* `headroom.proxy.server._check_rust_core()` runs as the first step of
  the FastAPI lifespan. If the import fails it prints a structured
  diagnostic, logs `event=rust_core_missing`, and calls `sys.exit(78)`
  (sysexits.h `EX_CONFIG`). Process supervisors (systemd / k8s /
  docker) treat this as a deliberate config error and stop restart
  loops.
* `HEADROOM_REQUIRE_RUST_CORE=false` is the explicit opt-out for
  Python-only `pip install -e .` developer flows; lifespan logs
  `event=rust_core_disabled` and continues. Any other value (including
  unset) keeps the fail-loud default.
* `/health` now surfaces `rust_core: "loaded" | "disabled" | "missing"`
  (plus `rust_core_error` when non-loaded) so operators can alert on
  the degraded state rather than discovering it via a customer ticket.
* `scripts/build_rust_extension.sh` is the single dev-time path: build
  → install → import-verify with the same `hello()` marker the lifespan
  checks. Failures are loud at every step.
* `Makefile` exposes the script as `make verify-rust-core`.
* `Dockerfile` now installs `rustup` + `maturin`, builds the wheel from
  `crates/headroom-py`, force-installs it into site-packages, and runs
  the same `hello()` import-verify in the build image so a broken build
  fails the docker-build, not the next runtime restart.

Tests:
* `tests/test_rust_core_smoke.py` pins all four contracts:
  - `_core.hello()` returns `"headroom-core"`
  - missing extension + default env → `SystemExit(78)`
  - missing extension + opt-out env → lifespan starts, `/health`
    returns `rust_core: "disabled"` with the underlying error
  - present extension + default env → `("loaded", None)`

Per-finding-#2: ~/Desktop/HEADROOM_PROXY_LOG_FINDINGS_2026_05_03.md.
2026-05-02 17:52:37 -07:00
chopratejasandClaude Opus 4.6 4641e43891 Add supply chain hardening: SBOM, cosign signing, pinned digests, Dependabot
Enterprise supply chain improvements:
- publish.yml: Generate CycloneDX SBOM and attach to GitHub releases
- docker.yml: Sign all Docker images with Sigstore cosign (keyless OIDC)
- Dockerfile: Pin python:3.11-slim and distroless base images with SHA256
  digests to prevent silent upstream changes
- Add .github/dependabot.yml for automated Docker digest, GitHub Actions,
  and pip dependency update PRs

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-15 17:28:14 -07:00
JerrettDavisandCopilot a1beb08d53 feat: add reproducible devcontainers
Add a default devcontainer and a compose-backed memory-stack profile, validate them in CI, and document the contributor workflow.

Also lock the memory-stack dependencies, pin related container tooling, and sync the latest healthcheck shutdown fix for stubbed memory handlers.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-10 12:58:35 -05:00
JerrettDavisandCopilot 7a18b71c0d feat: add proxy healthcheck endpoints
Add /livez and /readyz, keep /health backward-compatible, and wire readiness into Docker artifacts and docs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-10 12:23:23 -05:00
JerrettDavis ca7384402b Fix review regressions in compare, wrap, and Docker 2026-04-05 17:12:48 -05:00
pratikbin 63aebe8235 refactor(docker): migrate to bake with multi-variant distroless images 2026-04-04 21:51:37 +05:30
pratikbin 944b35d78b feat: Dockerfile & ci 2026-04-03 01:02:26 +05:30
chopratejas 55d8fb6361 Fixing the DockerFile to have hnswlib working 2026-02-17 07:06:53 -08:00
Nicola Beghin 1b023e3146 feat: add Docker configuration for containerized deployment
- Add .dockerignore to exclude unnecessary files from Docker builds
- Add Dockerfile to containerize Python application with health checks
- Add docker-compose.yml for easy local development and deployment of headroom proxy service
2026-01-20 00:17:05 +01:00