fix(install): keep the HuggingFace cache on a writable mount in Docker presets (#4207) (#4209)

## Description

With the `persistent-docker` preset, Kompress never loads. The container
runs as the host user (`--user 501:20`) with `HOME=/tmp/headroom-home`,
but the image never creates that directory. Docker creates it as
`root:root 0755` to hold the bind mounts, so the proxy cannot create
`~/.cache` and the HuggingFace download of the Kompress model fails with
`Permission denied (os error 13)`. The log never shows that error: it
reports a missing dependency instead (`Kompress requires onnxruntime or
torch`), pointing the operator at reinstalling extras that are already
present.

Fix: point `HF_HOME` and `XDG_CACHE_HOME` at the bind-mounted,
host-owned `.headroom` workspace so the cache is writable regardless of
who Docker made the container's `$HOME`; and keep the underlying
`OSError` in both `kompress_compressor.py` log sites instead of masking
it.

Closes #4207

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/install/runtime.py`: `build_runtime_command()` now emits
`HF_HOME` and `XDG_CACHE_HOME` as real `--env` flags pointing under the
bind-mounted `.headroom` workspace (`<container_home>/.headroom/cache`),
and applies them to the persistent-container args path, not only the
dev/foreground path.
- `headroom/install/runtime.py`: added `_append_cache_env()` and wired
it through both command builders.
- `scripts/install.sh` and `docker/docker-compose.native.yml`: set
`HF_HOME`/`XDG_CACHE_HOME` under the writable `.headroom` mount for the
same preset.
- `headroom/transforms/kompress_compressor.py`: `_create_onnx_session()`
appends the underlying cause to its `No loadable ONNX artifact` message;
`_load_kompress()` re-raises the real ONNX failure (chained) instead of
reporting a missing dependency when onnxruntime is present but loading
failed.

## Testing

- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ .venv/bin/python -m pytest tests/test_install/ tests/test_kompress_failsafe.py -q
278 passed, 11 skipped, 1 failed

The single failure is tests/test_install/test_runtime.py::test_build_runtime_command_lets_the_host_dashboard_read_its_data
which fails with ImportError: FastAPI required in this environment. It fails identically on unmodified main here,
so it is a local environment gap, not this change.

New regression tests, all passing:
- test_build_runtime_command_keeps_hf_cache_on_the_writable_mount
- test_build_runtime_command_persistent_path_keeps_hf_cache_on_the_writable_mount
- test_no_loadable_artifact_message_carries_the_underlying_error
- test_onnx_failure_while_onnxruntime_available_reports_the_real_error
- test_missing_dependency_still_reports_install_hint

$ bash -n scripts/install.sh          # clean
$ python -c "import yaml,sys;yaml.safe_load(open('docker/docker-compose.native.yml'))"   # clean
```

## Real Behavior Proof

- Environment: Linux, Python venv at `.venv`, repo at this branch. No
Docker daemon available on this machine.
- Exact command / steps: `pytest tests/test_install/
tests/test_kompress_failsafe.py -q`; asserted the generated `docker run`
argv from both `build_runtime_command()` and
`build_persistent_container_args()` contains `--env HF_HOME` / `--env
XDG_CACHE_HOME` pointing under the `.headroom` mount.
- Observed result: all new tests pass; both argv builders now emit the
cache env pointing at the writable host-owned mount.
- Not tested: the end-to-end container run from the issue's reproduction
steps — no Docker daemon here. The fix is verified at the level of the
generated `docker run` argv and the unit tests above.

## Runtime Rollout Safety

- Rollout-managed feature(s): Kompress proxy model loading under the
`persistent-docker` preset.
- Minimum rollout channel: all channels that use the persistent-docker
preset; the change only affects env vars for cache directories.
- Stable/default behavior changed: no. Default (non-persistent) paths
are unchanged; persistent paths gain two `--env` flags pointing at an
existing writable mount.
- Kill switch / disable path: unset/override `HF_HOME`/`XDG_CACHE_HOME`,
or run without the persistent-docker preset.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this commit; the preset returns to the previous
(broken) cache behavior.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- Docs untouched: no user-facing docs describe
`HF_HOME`/`XDG_CACHE_HOME` for this preset, so no documentation change
was needed (hence the unchecked box).
- No lint/type-check run: `ruff`/`mypy` are not installed in this
environment.
- No Docker daemon available, so the container-level e2e reproduction
was not executed — flagged under "Not tested" above.

---------

Co-authored-by: Shivam Maurya <shivam@programmingwithmaurya.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
Co-authored-by: JerrettDavis <jerrett@headroomlabs.ai>
This commit is contained in:
Shivam Maurya
2026-10-24 08:38:28 -07:00
committed by GitHub
co-authored by Shivam Maurya JD Davis JerrettDavis
parent 4fd9cddfd2
commit 7b6cb77905
6 changed files with 139 additions and 1 deletions
+8
View File
@@ -7,6 +7,11 @@ services:
tty: true
environment:
HOME: /tmp/headroom-home
# Docker creates /tmp/headroom-home as root:root, so $HOME is not
# writable by the --user the container runs as (issue #4207). Keep the
# HuggingFace/XDG caches on the bind-mounted workspace instead.
HF_HOME: /tmp/headroom-home/.headroom/cache/huggingface
XDG_CACHE_HOME: /tmp/headroom-home/.headroom/cache
# Canonical Headroom filesystem contract (issue #175). Forwarded into
# the container so the proxy resolves state/config to the bind-mounted
# /tmp/headroom-home/.headroom path. HEADROOM_WORKSPACE (above) remains
@@ -28,6 +33,9 @@ services:
restart: unless-stopped
environment:
HOME: /tmp/headroom-home
# See the `cli` service above (issue #4207).
HF_HOME: /tmp/headroom-home/.headroom/cache/huggingface
XDG_CACHE_HOME: /tmp/headroom-home/.headroom/cache
HEADROOM_HOST: 0.0.0.0
HEADROOM_PROXY_TOKEN: ${HEADROOM_PROXY_TOKEN:-}
# The container binds 0.0.0.0 so the loopback publication below can reach
+12
View File
@@ -160,6 +160,18 @@ def build_runtime_command(manifest: DeploymentManifest) -> list[str]:
container_home,
"--env",
f"HOME={container_home}",
# Docker creates the parent of every bind mount as root:root when it is
# absent from the image, so $HOME itself is NOT writable by the
# --user <host-uid> process (issue #4207). Anything that writes under
# $HOME outside the mounts then fails, which is what breaks the
# HuggingFace download of the Kompress model. Point the HuggingFace and
# XDG caches at the bind-mounted workspace instead: that directory
# belongs to the host user the container runs as, so it is writable,
# and it survives `docker rm` of the container.
"--env",
f"HF_HOME={container_home}/.headroom/cache/huggingface",
"--env",
f"XDG_CACHE_HOME={container_home}/.headroom/cache",
"--env",
"PYTHONUNBUFFERED=1",
# Canonical Headroom filesystem contract (issue #175).
+16 -1
View File
@@ -905,8 +905,9 @@ def _create_onnx_session(
raise refused
if not allow_download and cache_miss:
raise KompressModelNotCached(model_id) from last_err
detail = f": {last_err}" if last_err is not None else ""
raise FileNotFoundError(
f"No loadable ONNX artifact in {model_id}; tried {_onnx_filename_candidates()}"
f"No loadable ONNX artifact in {model_id}; tried {_onnx_filename_candidates()}{detail}"
) from last_err
@@ -1241,6 +1242,7 @@ def _load_kompress(
# Auto mode: preserve stable default behavior. This avoids changing
# compression quality/perf characteristics for existing installs while
# allowing opt-in MPS/CoreML experiments via HEADROOM_KOMPRESS_BACKEND.
onnx_error: Exception | None = None
if _is_onnx_available():
try:
return _load_kompress_onnx(model_id, use_coreml=False, allow_download=allow_download)
@@ -1250,11 +1252,24 @@ def _load_kompress(
if not allow_download:
raise
except Exception as e:
onnx_error = e
logger.warning("ONNX load failed for %s, trying PyTorch: %s", model_id, e)
if _is_pytorch_available():
return _load_kompress_pytorch(model_id, device, allow_download=allow_download)
if onnx_error is not None:
# onnxruntime IS installed and was tried, but loading failed. The
# "requires onnxruntime or torch" message would be a lie that sends the
# operator to reinstall extras that are already present (issue #4207:
# an unwritable $HOME fails the HuggingFace download, which surfaces as
# "No loadable ONNX artifact" and was then reported as a missing
# dependency). Re-raise the real error instead, chained so the original
# traceback is preserved.
raise RuntimeError(
f"Kompress ONNX backend failed for {model_id}: {onnx_error}"
) from onnx_error
raise ImportError(
"Kompress requires onnxruntime or torch. Install with: pip install headroom-ai[proxy]"
)
+6
View File
@@ -122,6 +122,12 @@ append_common_container_args() {
ensure_host_dirs
ref+=(-w /workspace)
ref+=(--env "HOME=${HEADROOM_CONTAINER_HOME}")
# Docker creates ${HEADROOM_CONTAINER_HOME} as root:root when the image has
# no such directory, so $HOME is not writable by the --user <host-uid>
# process (issue #4207). Keep the HuggingFace/XDG caches on the
# bind-mounted workspace, which the host user owns.
ref+=(--env "HF_HOME=${HEADROOM_CONTAINER_HOME}/.headroom/cache/huggingface")
ref+=(--env "XDG_CACHE_HOME=${HEADROOM_CONTAINER_HOME}/.headroom/cache")
ref+=(--env "PYTHONUNBUFFERED=1")
# Canonical Headroom filesystem contract (issue #175) — forward into the
# container so the proxy resolves state/config to the bind-mounted path.
+41
View File
@@ -118,6 +118,47 @@ def test_build_runtime_command_for_docker_includes_deployment_env(
)
def test_build_runtime_command_keeps_hf_cache_on_the_writable_mount(
monkeypatch, tmp_path: Path
) -> None:
"""Issue #4207: ``$HOME`` is created root-owned by Docker but the container
runs as the host uid, so the HuggingFace cache must live on a bind mount the
host user owns. Otherwise the Kompress model download dies with
``Permission denied``.
"""
monkeypatch.setattr(Path, "home", lambda: tmp_path)
manifest = DeploymentManifest(
profile="default",
preset="persistent-docker",
runtime_kind="docker",
supervisor_kind="none",
scope="user",
provider_mode="manual",
targets=["claude"],
port=8787,
host="127.0.0.1",
backend="anthropic",
image="ghcr.io/headroomlabs-ai/headroom:latest",
base_env={"HEADROOM_PORT": "8787"},
proxy_args=["--host", "127.0.0.1", "--port", "8787"],
)
command = build_runtime_command(manifest)
# The cache path sits under the ``~/.headroom`` bind mount, which the host
# user (and therefore the --user the container runs as) can write.
hf_home = "HF_HOME=/tmp/headroom-home/.headroom/cache/huggingface"
xdg_cache = "XDG_CACHE_HOME=/tmp/headroom-home/.headroom/cache"
assert hf_home in command
assert xdg_cache in command
# Both are real --env flags, not stray argv tokens.
assert command[command.index(hf_home) - 1] == "--env"
assert command[command.index(xdg_cache) - 1] == "--env"
# The cache lives under the .headroom mount, so it must not be mounted
# anywhere Docker would create as root.
assert f"{_mount_source(str(tmp_path), '.headroom')}:/tmp/headroom-home/.headroom" in command
def test_build_runtime_command_lets_the_host_dashboard_read_its_data(
monkeypatch, tmp_path: Path
) -> None:
+56
View File
@@ -609,6 +609,62 @@ def test_all_artifacts_failing_at_run_raises_rather_than_returning_a_dead_sessio
kc._create_onnx_session("org/model", ["CPUExecutionProvider"])
def test_no_loadable_artifact_message_carries_the_underlying_error(monkeypatch):
"""Issue #4207: the ONNX loader reported only "No loadable ONNX artifact",
hiding the Permission denied behind it. The message must show the cause.
"""
def fake_artifact(model_id, filename, *, allow_network):
raise PermissionError(13, "Permission denied")
monkeypatch.setattr(kc, "_hf_artifact", fake_artifact)
with pytest.raises(FileNotFoundError) as excinfo:
kc._create_onnx_session("org/model", ["CPUExecutionProvider"])
assert "No loadable ONNX artifact" in str(excinfo.value)
assert "Permission denied" in str(excinfo.value)
def test_onnx_failure_while_onnxruntime_available_reports_the_real_error(monkeypatch):
"""Issue #4207: when onnxruntime IS installed but loading failed, the
compressor must not claim a missing dependency (which points operators at
reinstalling extras that are already present). It must surface the real
failure.
"""
monkeypatch.setattr(kc, "_is_onnx_available", lambda: True)
monkeypatch.setattr(kc, "_is_pytorch_available", lambda: False)
def boom(*args, **kwargs):
raise FileNotFoundError(
"No loadable ONNX artifact in org/model; tried ('a.onnx',): "
"Permission denied (os error 13)"
)
monkeypatch.setattr(kc, "_load_kompress_onnx", boom)
monkeypatch.setattr(kc, "_selected_backend", lambda: "auto")
kc._kompress_cache.pop("org/model", None)
with pytest.raises(RuntimeError) as excinfo:
kc._load_kompress("org/model")
message = str(excinfo.value)
assert "requires onnxruntime or torch" not in message
assert "Permission denied" in message
assert isinstance(excinfo.value.__cause__, FileNotFoundError)
def test_missing_dependency_still_reports_install_hint(monkeypatch):
"""The genuine no-backend case keeps the actionable install hint."""
monkeypatch.setattr(kc, "_is_onnx_available", lambda: False)
monkeypatch.setattr(kc, "_is_pytorch_available", lambda: False)
monkeypatch.setattr(kc, "_selected_backend", lambda: "auto")
kc._kompress_cache.pop("org/model", None)
with pytest.raises(ImportError, match="onnxruntime or torch"):
kc._load_kompress("org/model")
# ── Load-stage timings: say where a slow warm-up spent its time ────────────────