fix(memory): treat a cwd with no UTF-8 form as unresolved instead of raising (#4167)

## Description

`ProjectResolver` raised `UnicodeEncodeError` when the cwd it parsed
from the system prompt contained an unpaired surrogate, which a JSON
decoder produces from an escaped `\ud800`. `_identity_from_cwd` hashes
`normalised.encode("utf-8")`, and the earlier `realpath` fallback passed
the invalid string straight through, so an optional memory-routing step
crashed the request path.

A cwd with no UTF-8 form can't name a real directory, and hashing it
with a lossy encoding (`errors="replace"`/`"surrogatepass"`) could merge
distinct paths onto one store. So `_identity_from_cwd` now returns
`None` for it. That is the same result as an empty cwd: resolution falls
through to the next tier, and if nothing else resolves, project mode
keeps its existing fail-closed "unresolved (no memory)" behavior. Valid
non-ASCII paths are unaffected.

Closes #4162

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/memory/storage_router.py`: `_identity_from_cwd` checks that
the path encodes as UTF-8 after `realpath` normalisation and returns
`None` if it doesn't, then hashes those same bytes. Checking after
`realpath` covers both a surrogate in the input and a symlink that
`realpath` resolves to a non-UTF-8 filename (POSIX returns those as
`\udcXX` escapes). This applies to all three cwd tiers (`x-headroom-cwd`
header, CLI override, system prompt).
- `tests/test_memory_storage_router.py`: four new tests:
  - the issue's reproduction resolves to `None` instead of raising
- an unusable CLI-override cwd falls through to a valid system-prompt
cwd
  - a valid non-ASCII cwd (`café`) still resolves
- `BackendRouter` in project mode fails closed (`project_key is None`)
for the surrogate cwd
- a cwd that is a symlink to a non-UTF-8 filename resolves to `None`.
This test uses a real symlink and is skipped where the filesystem
rejects such names, such as macOS APFS; it runs on Linux CI.
- the same case with `realpath` returning a `\udcff` escape, so it is
also covered on macOS

## Testing

- [x] Unit tests pass (`pytest`), with unrelated pre-existing failures
noted below
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`): not clean in my
environment, but unchanged by this PR (see below)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_memory_storage_router.py -k "unencodable or non_ascii" -q   # before the fix
FAILED tests/test_memory_storage_router.py::test_resolver_unencodable_system_prompt_cwd_does_not_raise
FAILED tests/test_memory_storage_router.py::test_resolver_unencodable_cli_override_falls_through_to_system_prompt
FAILED tests/test_memory_storage_router.py::test_router_project_mode_unencodable_cwd_fails_closed
3 failed, 1 passed, 37 deselected

$ pytest tests/test_memory_storage_router.py tests/test_proxy_handler_helpers.py tests/test_codex_project_context.py tests/test_ccr_context_tracker.py -q
303 passed in 7.88s

$ pytest tests -k memory --ignore=tests/gateway --continue-on-collection-errors -q
2 failed, 1315 passed, 233 skipped, 14928 deselected, 18 errors

$ ruff check headroom/memory/storage_router.py tests/test_memory_storage_router.py
All checks passed!
$ ruff format --check headroom/memory/storage_router.py tests/test_memory_storage_router.py
2 files already formatted
```

The 2 failures
(`test_memory_handler_concurrent_init.py::test_real_localbackend_initializes_via_public_entrypoint`,
`test_proxy_healthchecks.py::test_readyz_reports_memory_backend_when_enabled`)
and the collection errors (missing `respx`, one LangChain import) are
identical on unmodified `main` in my environment. `mypy --python-version
3.12 headroom/memory/storage_router.py` reports the same 11 errors in 4
other files with and without this change, and none in
`storage_router.py`.

## Real Behavior Proof

- Environment: macOS, Python 3.12, editable install at branch head;
Linux container for the symlink check
- Exact command / steps: the issue's reproduction against the real
library, no mocks:
  ```python
from headroom.memory.storage_router import ProjectResolver,
RequestContext
ctx = RequestContext(headers={}, system_prompt="Primary working
directory: /work/" + chr(0xd800), base_user_id="repro")
  print(ProjectResolver().resolve(ctx))
  ```
- Observed result: on `main` (`976aa714`): `UnicodeEncodeError: 'utf-8'
codec can't encode character '\ud800' in position 6: surrogates not
allowed`. On this branch it prints `None`.
- Symlink case (from review): in a Linux container, a symlink
`/tmp/t/link` pointing to a directory named `b"\xff-target"` gives
`os.path.realpath("/tmp/t/link") == "/tmp/t/\udcff-target"`, and
`.encode("utf-8")` on that raises `UnicodeEncodeError`. That confirms a
valid UTF-8 cwd can still reach the hash with a surrogate, which is why
the check now runs after `realpath`.
- Not tested: a full HTTP request through a running proxy with memory
enabled.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: N/A
- Stable/default behavior changed: only for a cwd that has no UTF-8
form, which used to raise and now resolves to no project. Every other
cwd gets the same key as before.
- Kill switch / disable path: N/A
- Unsafe override required: No
- Qualification impact: None
- Rollback path: revert this PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation: N/A,
internal behavior only
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Before opening this I searched open and closed PRs for #4162,
"surrogate" and `_identity_from_cwd` and found no existing fix. Open PR
#4166 (for #4163) changes how the system-prompt cwd is extracted, not
how it is hashed, so the two don't overlap.

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
SD
2026-10-24 08:44:47 -07:00
committed by GitHub
co-authored by JD Davis
parent f89c9282a8
commit 760c05e1b0
2 changed files with 85 additions and 1 deletions
+13 -1
View File
@@ -248,9 +248,21 @@ class ProjectResolver:
except (OSError, ValueError):
normalised = cwd
normalised = normalised.rstrip(os.sep) or os.sep
# A path with no UTF-8 form cannot be hashed without a lossy encoding
# that could merge distinct directories onto one store. It reaches
# here either from the input (an unpaired surrogate, e.g. a
# JSON-decoded ``\ud800`` escape) or from ``realpath`` following a
# symlink to a non-UTF-8 filename, which POSIX returns as
# ``\udcXX`` surrogate escapes. Check after normalising so both are
# covered, and treat the cwd as unusable: resolution falls through to
# the next tier, or fails closed, instead of raising (#4162).
try:
encoded = normalised.encode("utf-8")
except UnicodeEncodeError:
return None
basename = os.path.basename(normalised) or "root"
safe_basename = cls._sanitize_basename(basename) or "project"
digest = hashlib.sha256(normalised.encode("utf-8")).hexdigest()[:16]
digest = hashlib.sha256(encoded).hexdigest()[:16]
key = f"{safe_basename}-{digest}"
return key, basename
+72
View File
@@ -2,6 +2,7 @@
from __future__ import annotations
import os
from pathlib import Path
import pytest
@@ -186,6 +187,77 @@ def test_resolver_returns_none_when_nothing_resolves() -> None:
assert out is None
def test_resolver_unencodable_system_prompt_cwd_does_not_raise() -> None:
# Issue #4162: a JSON-decoded ``\ud800`` escape leaves an unpaired
# surrogate in the cwd. It has no UTF-8 form, so it is not a usable
# identity: resolve to "no project" instead of raising.
r = ProjectResolver()
prompt = "Primary working directory: /work/" + chr(0xD800)
assert r.resolve(_ctx(system_prompt=prompt)) is None
def test_resolver_unencodable_cli_override_falls_through_to_system_prompt() -> None:
# An unusable cwd in one tier must not block a usable one in a later tier.
r = ProjectResolver()
out = r.resolve(
_ctx(
project_root_override="/work/" + chr(0xDC80),
system_prompt="Primary working directory: /Users/foo/code/headroom\n",
)
)
assert out is not None
_, display = out
assert display == "headroom"
@pytest.mark.skipif(os.name != "posix", reason="POSIX undecodable byte filenames and symlinks only")
def test_resolver_symlink_to_undecodable_name_does_not_raise(tmp_path: Path) -> None:
# The cwd itself is plain UTF-8, but realpath follows the symlink to a
# non-UTF-8 filename and returns it as a ``\udcXX`` surrogate escape.
# Creating that fixture is a filesystem capability, not product behavior:
# skip if it is unavailable (e.g. macOS APFS rejects non-UTF-8 names).
# test_resolver_realpath_returning_surrogate_does_not_raise covers the
# same path on every platform.
base = os.fsencode(tmp_path)
try:
os.mkdir(base + b"/\xff-target")
os.symlink(base + b"/\xff-target", base + b"/link")
except (OSError, UnicodeError) as exc:
pytest.skip(f"cannot create a non-UTF-8 symlink fixture here: {exc}")
prompt = f"Primary working directory: {tmp_path}/link\n"
assert ProjectResolver().resolve(_ctx(system_prompt=prompt)) is None
def test_resolver_realpath_returning_surrogate_does_not_raise(
monkeypatch: pytest.MonkeyPatch,
) -> None:
# Same case as above on any OS: realpath yields what Linux returns for a
# symlink to a non-UTF-8 filename.
monkeypatch.setattr(os.path, "realpath", lambda _p: "/work/\udcff-target")
prompt = "Primary working directory: /work/link\n"
assert ProjectResolver().resolve(_ctx(system_prompt=prompt)) is None
def test_resolver_valid_non_ascii_cwd_still_resolves() -> None:
r = ProjectResolver()
out = r.resolve(_ctx(system_prompt="Primary working directory: /Users/foo/code/café\n"))
assert out is not None
key, display = out
assert display == "café"
assert key.startswith("caf")
def test_router_project_mode_unencodable_cwd_fails_closed(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
router = _make_router(tmp_path, MemoryStorageMode.PROJECT, monkeypatch)
_, scope = router.backend_for(
_ctx(system_prompt="Primary working directory: /work/" + chr(0xD800))
)
assert scope.mode is MemoryStorageMode.PROJECT
assert scope.project_key is None
def test_resolver_same_cwd_yields_stable_key_across_calls() -> None:
r = ProjectResolver()
k1, _ = r.resolve(_ctx(headers={"x-headroom-cwd": "/Users/foo/code/x"})) # type: ignore[misc]