mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(memory): treat a cwd with no UTF-8 form as unresolved instead of raising (#4167)
## Description
`ProjectResolver` raised `UnicodeEncodeError` when the cwd it parsed
from the system prompt contained an unpaired surrogate, which a JSON
decoder produces from an escaped `\ud800`. `_identity_from_cwd` hashes
`normalised.encode("utf-8")`, and the earlier `realpath` fallback passed
the invalid string straight through, so an optional memory-routing step
crashed the request path.
A cwd with no UTF-8 form can't name a real directory, and hashing it
with a lossy encoding (`errors="replace"`/`"surrogatepass"`) could merge
distinct paths onto one store. So `_identity_from_cwd` now returns
`None` for it. That is the same result as an empty cwd: resolution falls
through to the next tier, and if nothing else resolves, project mode
keeps its existing fail-closed "unresolved (no memory)" behavior. Valid
non-ASCII paths are unaffected.
Closes #4162
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/memory/storage_router.py`: `_identity_from_cwd` checks that
the path encodes as UTF-8 after `realpath` normalisation and returns
`None` if it doesn't, then hashes those same bytes. Checking after
`realpath` covers both a surrogate in the input and a symlink that
`realpath` resolves to a non-UTF-8 filename (POSIX returns those as
`\udcXX` escapes). This applies to all three cwd tiers (`x-headroom-cwd`
header, CLI override, system prompt).
- `tests/test_memory_storage_router.py`: four new tests:
- the issue's reproduction resolves to `None` instead of raising
- an unusable CLI-override cwd falls through to a valid system-prompt
cwd
- a valid non-ASCII cwd (`café`) still resolves
- `BackendRouter` in project mode fails closed (`project_key is None`)
for the surrogate cwd
- a cwd that is a symlink to a non-UTF-8 filename resolves to `None`.
This test uses a real symlink and is skipped where the filesystem
rejects such names, such as macOS APFS; it runs on Linux CI.
- the same case with `realpath` returning a `\udcff` escape, so it is
also covered on macOS
## Testing
- [x] Unit tests pass (`pytest`), with unrelated pre-existing failures
noted below
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`): not clean in my
environment, but unchanged by this PR (see below)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_memory_storage_router.py -k "unencodable or non_ascii" -q # before the fix
FAILED tests/test_memory_storage_router.py::test_resolver_unencodable_system_prompt_cwd_does_not_raise
FAILED tests/test_memory_storage_router.py::test_resolver_unencodable_cli_override_falls_through_to_system_prompt
FAILED tests/test_memory_storage_router.py::test_router_project_mode_unencodable_cwd_fails_closed
3 failed, 1 passed, 37 deselected
$ pytest tests/test_memory_storage_router.py tests/test_proxy_handler_helpers.py tests/test_codex_project_context.py tests/test_ccr_context_tracker.py -q
303 passed in 7.88s
$ pytest tests -k memory --ignore=tests/gateway --continue-on-collection-errors -q
2 failed, 1315 passed, 233 skipped, 14928 deselected, 18 errors
$ ruff check headroom/memory/storage_router.py tests/test_memory_storage_router.py
All checks passed!
$ ruff format --check headroom/memory/storage_router.py tests/test_memory_storage_router.py
2 files already formatted
```
The 2 failures
(`test_memory_handler_concurrent_init.py::test_real_localbackend_initializes_via_public_entrypoint`,
`test_proxy_healthchecks.py::test_readyz_reports_memory_backend_when_enabled`)
and the collection errors (missing `respx`, one LangChain import) are
identical on unmodified `main` in my environment. `mypy --python-version
3.12 headroom/memory/storage_router.py` reports the same 11 errors in 4
other files with and without this change, and none in
`storage_router.py`.
## Real Behavior Proof
- Environment: macOS, Python 3.12, editable install at branch head;
Linux container for the symlink check
- Exact command / steps: the issue's reproduction against the real
library, no mocks:
```python
from headroom.memory.storage_router import ProjectResolver,
RequestContext
ctx = RequestContext(headers={}, system_prompt="Primary working
directory: /work/" + chr(0xd800), base_user_id="repro")
print(ProjectResolver().resolve(ctx))
```
- Observed result: on `main` (`976aa714`): `UnicodeEncodeError: 'utf-8'
codec can't encode character '\ud800' in position 6: surrogates not
allowed`. On this branch it prints `None`.
- Symlink case (from review): in a Linux container, a symlink
`/tmp/t/link` pointing to a directory named `b"\xff-target"` gives
`os.path.realpath("/tmp/t/link") == "/tmp/t/\udcff-target"`, and
`.encode("utf-8")` on that raises `UnicodeEncodeError`. That confirms a
valid UTF-8 cwd can still reach the hash with a surrogate, which is why
the check now runs after `realpath`.
- Not tested: a full HTTP request through a running proxy with memory
enabled.
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: N/A
- Stable/default behavior changed: only for a cwd that has no UTF-8
form, which used to raise and now resolves to no project. Every other
cwd gets the same key as before.
- Kill switch / disable path: N/A
- Unsafe override required: No
- Qualification impact: None
- Rollback path: revert this PR
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation: N/A,
internal behavior only
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
Before opening this I searched open and closed PRs for #4162,
"surrogate" and `_identity_from_cwd` and found no existing fix. Open PR
#4166 (for #4163) changes how the system-prompt cwd is extracted, not
how it is hashed, so the two don't overlap.
---------
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
@@ -248,9 +248,21 @@ class ProjectResolver:
|
||||
except (OSError, ValueError):
|
||||
normalised = cwd
|
||||
normalised = normalised.rstrip(os.sep) or os.sep
|
||||
# A path with no UTF-8 form cannot be hashed without a lossy encoding
|
||||
# that could merge distinct directories onto one store. It reaches
|
||||
# here either from the input (an unpaired surrogate, e.g. a
|
||||
# JSON-decoded ``\ud800`` escape) or from ``realpath`` following a
|
||||
# symlink to a non-UTF-8 filename, which POSIX returns as
|
||||
# ``\udcXX`` surrogate escapes. Check after normalising so both are
|
||||
# covered, and treat the cwd as unusable: resolution falls through to
|
||||
# the next tier, or fails closed, instead of raising (#4162).
|
||||
try:
|
||||
encoded = normalised.encode("utf-8")
|
||||
except UnicodeEncodeError:
|
||||
return None
|
||||
basename = os.path.basename(normalised) or "root"
|
||||
safe_basename = cls._sanitize_basename(basename) or "project"
|
||||
digest = hashlib.sha256(normalised.encode("utf-8")).hexdigest()[:16]
|
||||
digest = hashlib.sha256(encoded).hexdigest()[:16]
|
||||
key = f"{safe_basename}-{digest}"
|
||||
return key, basename
|
||||
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
@@ -186,6 +187,77 @@ def test_resolver_returns_none_when_nothing_resolves() -> None:
|
||||
assert out is None
|
||||
|
||||
|
||||
def test_resolver_unencodable_system_prompt_cwd_does_not_raise() -> None:
|
||||
# Issue #4162: a JSON-decoded ``\ud800`` escape leaves an unpaired
|
||||
# surrogate in the cwd. It has no UTF-8 form, so it is not a usable
|
||||
# identity: resolve to "no project" instead of raising.
|
||||
r = ProjectResolver()
|
||||
prompt = "Primary working directory: /work/" + chr(0xD800)
|
||||
assert r.resolve(_ctx(system_prompt=prompt)) is None
|
||||
|
||||
|
||||
def test_resolver_unencodable_cli_override_falls_through_to_system_prompt() -> None:
|
||||
# An unusable cwd in one tier must not block a usable one in a later tier.
|
||||
r = ProjectResolver()
|
||||
out = r.resolve(
|
||||
_ctx(
|
||||
project_root_override="/work/" + chr(0xDC80),
|
||||
system_prompt="Primary working directory: /Users/foo/code/headroom\n",
|
||||
)
|
||||
)
|
||||
assert out is not None
|
||||
_, display = out
|
||||
assert display == "headroom"
|
||||
|
||||
|
||||
@pytest.mark.skipif(os.name != "posix", reason="POSIX undecodable byte filenames and symlinks only")
|
||||
def test_resolver_symlink_to_undecodable_name_does_not_raise(tmp_path: Path) -> None:
|
||||
# The cwd itself is plain UTF-8, but realpath follows the symlink to a
|
||||
# non-UTF-8 filename and returns it as a ``\udcXX`` surrogate escape.
|
||||
# Creating that fixture is a filesystem capability, not product behavior:
|
||||
# skip if it is unavailable (e.g. macOS APFS rejects non-UTF-8 names).
|
||||
# test_resolver_realpath_returning_surrogate_does_not_raise covers the
|
||||
# same path on every platform.
|
||||
base = os.fsencode(tmp_path)
|
||||
try:
|
||||
os.mkdir(base + b"/\xff-target")
|
||||
os.symlink(base + b"/\xff-target", base + b"/link")
|
||||
except (OSError, UnicodeError) as exc:
|
||||
pytest.skip(f"cannot create a non-UTF-8 symlink fixture here: {exc}")
|
||||
prompt = f"Primary working directory: {tmp_path}/link\n"
|
||||
assert ProjectResolver().resolve(_ctx(system_prompt=prompt)) is None
|
||||
|
||||
|
||||
def test_resolver_realpath_returning_surrogate_does_not_raise(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
# Same case as above on any OS: realpath yields what Linux returns for a
|
||||
# symlink to a non-UTF-8 filename.
|
||||
monkeypatch.setattr(os.path, "realpath", lambda _p: "/work/\udcff-target")
|
||||
prompt = "Primary working directory: /work/link\n"
|
||||
assert ProjectResolver().resolve(_ctx(system_prompt=prompt)) is None
|
||||
|
||||
|
||||
def test_resolver_valid_non_ascii_cwd_still_resolves() -> None:
|
||||
r = ProjectResolver()
|
||||
out = r.resolve(_ctx(system_prompt="Primary working directory: /Users/foo/code/café\n"))
|
||||
assert out is not None
|
||||
key, display = out
|
||||
assert display == "café"
|
||||
assert key.startswith("caf")
|
||||
|
||||
|
||||
def test_router_project_mode_unencodable_cwd_fails_closed(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
router = _make_router(tmp_path, MemoryStorageMode.PROJECT, monkeypatch)
|
||||
_, scope = router.backend_for(
|
||||
_ctx(system_prompt="Primary working directory: /work/" + chr(0xD800))
|
||||
)
|
||||
assert scope.mode is MemoryStorageMode.PROJECT
|
||||
assert scope.project_key is None
|
||||
|
||||
|
||||
def test_resolver_same_cwd_yields_stable_key_across_calls() -> None:
|
||||
r = ProjectResolver()
|
||||
k1, _ = r.resolve(_ctx(headers={"x-headroom-cwd": "/Users/foo/code/x"})) # type: ignore[misc]
|
||||
|
||||
Reference in New Issue
Block a user