mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description With the `persistent-docker` preset, Kompress never loads. The container runs as the host user (`--user 501:20`) with `HOME=/tmp/headroom-home`, but the image never creates that directory. Docker creates it as `root:root 0755` to hold the bind mounts, so the proxy cannot create `~/.cache` and the HuggingFace download of the Kompress model fails with `Permission denied (os error 13)`. The log never shows that error: it reports a missing dependency instead (`Kompress requires onnxruntime or torch`), pointing the operator at reinstalling extras that are already present. Fix: point `HF_HOME` and `XDG_CACHE_HOME` at the bind-mounted, host-owned `.headroom` workspace so the cache is writable regardless of who Docker made the container's `$HOME`; and keep the underlying `OSError` in both `kompress_compressor.py` log sites instead of masking it. Closes #4207 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/install/runtime.py`: `build_runtime_command()` now emits `HF_HOME` and `XDG_CACHE_HOME` as real `--env` flags pointing under the bind-mounted `.headroom` workspace (`<container_home>/.headroom/cache`), and applies them to the persistent-container args path, not only the dev/foreground path. - `headroom/install/runtime.py`: added `_append_cache_env()` and wired it through both command builders. - `scripts/install.sh` and `docker/docker-compose.native.yml`: set `HF_HOME`/`XDG_CACHE_HOME` under the writable `.headroom` mount for the same preset. - `headroom/transforms/kompress_compressor.py`: `_create_onnx_session()` appends the underlying cause to its `No loadable ONNX artifact` message; `_load_kompress()` re-raises the real ONNX failure (chained) instead of reporting a missing dependency when onnxruntime is present but loading failed. ## Testing - [x] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ .venv/bin/python -m pytest tests/test_install/ tests/test_kompress_failsafe.py -q 278 passed, 11 skipped, 1 failed The single failure is tests/test_install/test_runtime.py::test_build_runtime_command_lets_the_host_dashboard_read_its_data which fails with ImportError: FastAPI required in this environment. It fails identically on unmodified main here, so it is a local environment gap, not this change. New regression tests, all passing: - test_build_runtime_command_keeps_hf_cache_on_the_writable_mount - test_build_runtime_command_persistent_path_keeps_hf_cache_on_the_writable_mount - test_no_loadable_artifact_message_carries_the_underlying_error - test_onnx_failure_while_onnxruntime_available_reports_the_real_error - test_missing_dependency_still_reports_install_hint $ bash -n scripts/install.sh # clean $ python -c "import yaml,sys;yaml.safe_load(open('docker/docker-compose.native.yml'))" # clean ``` ## Real Behavior Proof - Environment: Linux, Python venv at `.venv`, repo at this branch. No Docker daemon available on this machine. - Exact command / steps: `pytest tests/test_install/ tests/test_kompress_failsafe.py -q`; asserted the generated `docker run` argv from both `build_runtime_command()` and `build_persistent_container_args()` contains `--env HF_HOME` / `--env XDG_CACHE_HOME` pointing under the `.headroom` mount. - Observed result: all new tests pass; both argv builders now emit the cache env pointing at the writable host-owned mount. - Not tested: the end-to-end container run from the issue's reproduction steps — no Docker daemon here. The fix is verified at the level of the generated `docker run` argv and the unit tests above. ## Runtime Rollout Safety - Rollout-managed feature(s): Kompress proxy model loading under the `persistent-docker` preset. - Minimum rollout channel: all channels that use the persistent-docker preset; the change only affects env vars for cache directories. - Stable/default behavior changed: no. Default (non-persistent) paths are unchanged; persistent paths gain two `--env` flags pointing at an existing writable mount. - Kill switch / disable path: unset/override `HF_HOME`/`XDG_CACHE_HOME`, or run without the persistent-docker preset. - Unsafe override required: no. - Qualification impact: none. - Rollback path: revert this commit; the preset returns to the previous (broken) cache behavior. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes - Docs untouched: no user-facing docs describe `HF_HOME`/`XDG_CACHE_HOME` for this preset, so no documentation change was needed (hence the unchecked box). - No lint/type-check run: `ruff`/`mypy` are not installed in this environment. - No Docker daemon available, so the container-level e2e reproduction was not executed — flagged under "Not tested" above. --------- Co-authored-by: Shivam Maurya <shivam@programmingwithmaurya.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> Co-authored-by: JerrettDavis <jerrett@headroomlabs.ai>
This commit is contained in:
co-authored by
Shivam Maurya
JD Davis
JerrettDavis
parent
4fd9cddfd2
commit
7b6cb77905
@@ -7,6 +7,11 @@ services:
|
||||
tty: true
|
||||
environment:
|
||||
HOME: /tmp/headroom-home
|
||||
# Docker creates /tmp/headroom-home as root:root, so $HOME is not
|
||||
# writable by the --user the container runs as (issue #4207). Keep the
|
||||
# HuggingFace/XDG caches on the bind-mounted workspace instead.
|
||||
HF_HOME: /tmp/headroom-home/.headroom/cache/huggingface
|
||||
XDG_CACHE_HOME: /tmp/headroom-home/.headroom/cache
|
||||
# Canonical Headroom filesystem contract (issue #175). Forwarded into
|
||||
# the container so the proxy resolves state/config to the bind-mounted
|
||||
# /tmp/headroom-home/.headroom path. HEADROOM_WORKSPACE (above) remains
|
||||
@@ -28,6 +33,9 @@ services:
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
HOME: /tmp/headroom-home
|
||||
# See the `cli` service above (issue #4207).
|
||||
HF_HOME: /tmp/headroom-home/.headroom/cache/huggingface
|
||||
XDG_CACHE_HOME: /tmp/headroom-home/.headroom/cache
|
||||
HEADROOM_HOST: 0.0.0.0
|
||||
HEADROOM_PROXY_TOKEN: ${HEADROOM_PROXY_TOKEN:-}
|
||||
# The container binds 0.0.0.0 so the loopback publication below can reach
|
||||
|
||||
@@ -160,6 +160,18 @@ def build_runtime_command(manifest: DeploymentManifest) -> list[str]:
|
||||
container_home,
|
||||
"--env",
|
||||
f"HOME={container_home}",
|
||||
# Docker creates the parent of every bind mount as root:root when it is
|
||||
# absent from the image, so $HOME itself is NOT writable by the
|
||||
# --user <host-uid> process (issue #4207). Anything that writes under
|
||||
# $HOME outside the mounts then fails, which is what breaks the
|
||||
# HuggingFace download of the Kompress model. Point the HuggingFace and
|
||||
# XDG caches at the bind-mounted workspace instead: that directory
|
||||
# belongs to the host user the container runs as, so it is writable,
|
||||
# and it survives `docker rm` of the container.
|
||||
"--env",
|
||||
f"HF_HOME={container_home}/.headroom/cache/huggingface",
|
||||
"--env",
|
||||
f"XDG_CACHE_HOME={container_home}/.headroom/cache",
|
||||
"--env",
|
||||
"PYTHONUNBUFFERED=1",
|
||||
# Canonical Headroom filesystem contract (issue #175).
|
||||
|
||||
@@ -905,8 +905,9 @@ def _create_onnx_session(
|
||||
raise refused
|
||||
if not allow_download and cache_miss:
|
||||
raise KompressModelNotCached(model_id) from last_err
|
||||
detail = f": {last_err}" if last_err is not None else ""
|
||||
raise FileNotFoundError(
|
||||
f"No loadable ONNX artifact in {model_id}; tried {_onnx_filename_candidates()}"
|
||||
f"No loadable ONNX artifact in {model_id}; tried {_onnx_filename_candidates()}{detail}"
|
||||
) from last_err
|
||||
|
||||
|
||||
@@ -1241,6 +1242,7 @@ def _load_kompress(
|
||||
# Auto mode: preserve stable default behavior. This avoids changing
|
||||
# compression quality/perf characteristics for existing installs while
|
||||
# allowing opt-in MPS/CoreML experiments via HEADROOM_KOMPRESS_BACKEND.
|
||||
onnx_error: Exception | None = None
|
||||
if _is_onnx_available():
|
||||
try:
|
||||
return _load_kompress_onnx(model_id, use_coreml=False, allow_download=allow_download)
|
||||
@@ -1250,11 +1252,24 @@ def _load_kompress(
|
||||
if not allow_download:
|
||||
raise
|
||||
except Exception as e:
|
||||
onnx_error = e
|
||||
logger.warning("ONNX load failed for %s, trying PyTorch: %s", model_id, e)
|
||||
|
||||
if _is_pytorch_available():
|
||||
return _load_kompress_pytorch(model_id, device, allow_download=allow_download)
|
||||
|
||||
if onnx_error is not None:
|
||||
# onnxruntime IS installed and was tried, but loading failed. The
|
||||
# "requires onnxruntime or torch" message would be a lie that sends the
|
||||
# operator to reinstall extras that are already present (issue #4207:
|
||||
# an unwritable $HOME fails the HuggingFace download, which surfaces as
|
||||
# "No loadable ONNX artifact" and was then reported as a missing
|
||||
# dependency). Re-raise the real error instead, chained so the original
|
||||
# traceback is preserved.
|
||||
raise RuntimeError(
|
||||
f"Kompress ONNX backend failed for {model_id}: {onnx_error}"
|
||||
) from onnx_error
|
||||
|
||||
raise ImportError(
|
||||
"Kompress requires onnxruntime or torch. Install with: pip install headroom-ai[proxy]"
|
||||
)
|
||||
|
||||
@@ -122,6 +122,12 @@ append_common_container_args() {
|
||||
ensure_host_dirs
|
||||
ref+=(-w /workspace)
|
||||
ref+=(--env "HOME=${HEADROOM_CONTAINER_HOME}")
|
||||
# Docker creates ${HEADROOM_CONTAINER_HOME} as root:root when the image has
|
||||
# no such directory, so $HOME is not writable by the --user <host-uid>
|
||||
# process (issue #4207). Keep the HuggingFace/XDG caches on the
|
||||
# bind-mounted workspace, which the host user owns.
|
||||
ref+=(--env "HF_HOME=${HEADROOM_CONTAINER_HOME}/.headroom/cache/huggingface")
|
||||
ref+=(--env "XDG_CACHE_HOME=${HEADROOM_CONTAINER_HOME}/.headroom/cache")
|
||||
ref+=(--env "PYTHONUNBUFFERED=1")
|
||||
# Canonical Headroom filesystem contract (issue #175) — forward into the
|
||||
# container so the proxy resolves state/config to the bind-mounted path.
|
||||
|
||||
@@ -118,6 +118,47 @@ def test_build_runtime_command_for_docker_includes_deployment_env(
|
||||
)
|
||||
|
||||
|
||||
def test_build_runtime_command_keeps_hf_cache_on_the_writable_mount(
|
||||
monkeypatch, tmp_path: Path
|
||||
) -> None:
|
||||
"""Issue #4207: ``$HOME`` is created root-owned by Docker but the container
|
||||
runs as the host uid, so the HuggingFace cache must live on a bind mount the
|
||||
host user owns. Otherwise the Kompress model download dies with
|
||||
``Permission denied``.
|
||||
"""
|
||||
monkeypatch.setattr(Path, "home", lambda: tmp_path)
|
||||
manifest = DeploymentManifest(
|
||||
profile="default",
|
||||
preset="persistent-docker",
|
||||
runtime_kind="docker",
|
||||
supervisor_kind="none",
|
||||
scope="user",
|
||||
provider_mode="manual",
|
||||
targets=["claude"],
|
||||
port=8787,
|
||||
host="127.0.0.1",
|
||||
backend="anthropic",
|
||||
image="ghcr.io/headroomlabs-ai/headroom:latest",
|
||||
base_env={"HEADROOM_PORT": "8787"},
|
||||
proxy_args=["--host", "127.0.0.1", "--port", "8787"],
|
||||
)
|
||||
|
||||
command = build_runtime_command(manifest)
|
||||
|
||||
# The cache path sits under the ``~/.headroom`` bind mount, which the host
|
||||
# user (and therefore the --user the container runs as) can write.
|
||||
hf_home = "HF_HOME=/tmp/headroom-home/.headroom/cache/huggingface"
|
||||
xdg_cache = "XDG_CACHE_HOME=/tmp/headroom-home/.headroom/cache"
|
||||
assert hf_home in command
|
||||
assert xdg_cache in command
|
||||
# Both are real --env flags, not stray argv tokens.
|
||||
assert command[command.index(hf_home) - 1] == "--env"
|
||||
assert command[command.index(xdg_cache) - 1] == "--env"
|
||||
# The cache lives under the .headroom mount, so it must not be mounted
|
||||
# anywhere Docker would create as root.
|
||||
assert f"{_mount_source(str(tmp_path), '.headroom')}:/tmp/headroom-home/.headroom" in command
|
||||
|
||||
|
||||
def test_build_runtime_command_lets_the_host_dashboard_read_its_data(
|
||||
monkeypatch, tmp_path: Path
|
||||
) -> None:
|
||||
|
||||
@@ -609,6 +609,62 @@ def test_all_artifacts_failing_at_run_raises_rather_than_returning_a_dead_sessio
|
||||
kc._create_onnx_session("org/model", ["CPUExecutionProvider"])
|
||||
|
||||
|
||||
def test_no_loadable_artifact_message_carries_the_underlying_error(monkeypatch):
|
||||
"""Issue #4207: the ONNX loader reported only "No loadable ONNX artifact",
|
||||
hiding the Permission denied behind it. The message must show the cause.
|
||||
"""
|
||||
|
||||
def fake_artifact(model_id, filename, *, allow_network):
|
||||
raise PermissionError(13, "Permission denied")
|
||||
|
||||
monkeypatch.setattr(kc, "_hf_artifact", fake_artifact)
|
||||
|
||||
with pytest.raises(FileNotFoundError) as excinfo:
|
||||
kc._create_onnx_session("org/model", ["CPUExecutionProvider"])
|
||||
|
||||
assert "No loadable ONNX artifact" in str(excinfo.value)
|
||||
assert "Permission denied" in str(excinfo.value)
|
||||
|
||||
|
||||
def test_onnx_failure_while_onnxruntime_available_reports_the_real_error(monkeypatch):
|
||||
"""Issue #4207: when onnxruntime IS installed but loading failed, the
|
||||
compressor must not claim a missing dependency (which points operators at
|
||||
reinstalling extras that are already present). It must surface the real
|
||||
failure.
|
||||
"""
|
||||
monkeypatch.setattr(kc, "_is_onnx_available", lambda: True)
|
||||
monkeypatch.setattr(kc, "_is_pytorch_available", lambda: False)
|
||||
|
||||
def boom(*args, **kwargs):
|
||||
raise FileNotFoundError(
|
||||
"No loadable ONNX artifact in org/model; tried ('a.onnx',): "
|
||||
"Permission denied (os error 13)"
|
||||
)
|
||||
|
||||
monkeypatch.setattr(kc, "_load_kompress_onnx", boom)
|
||||
monkeypatch.setattr(kc, "_selected_backend", lambda: "auto")
|
||||
kc._kompress_cache.pop("org/model", None)
|
||||
|
||||
with pytest.raises(RuntimeError) as excinfo:
|
||||
kc._load_kompress("org/model")
|
||||
|
||||
message = str(excinfo.value)
|
||||
assert "requires onnxruntime or torch" not in message
|
||||
assert "Permission denied" in message
|
||||
assert isinstance(excinfo.value.__cause__, FileNotFoundError)
|
||||
|
||||
|
||||
def test_missing_dependency_still_reports_install_hint(monkeypatch):
|
||||
"""The genuine no-backend case keeps the actionable install hint."""
|
||||
monkeypatch.setattr(kc, "_is_onnx_available", lambda: False)
|
||||
monkeypatch.setattr(kc, "_is_pytorch_available", lambda: False)
|
||||
monkeypatch.setattr(kc, "_selected_backend", lambda: "auto")
|
||||
kc._kompress_cache.pop("org/model", None)
|
||||
|
||||
with pytest.raises(ImportError, match="onnxruntime or torch"):
|
||||
kc._load_kompress("org/model")
|
||||
|
||||
|
||||
# ── Load-stage timings: say where a slow warm-up spent its time ────────────────
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user