mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(codex): accept multi-line inline tables in Codex config.toml (#4436)
## Description
Codex accepts an inline table split across lines, with a trailing comma,
in `~/.codex/config.toml`:
```toml
env = {
DINGTALK_Client_ID = "**********",
}
```
That is TOML 1.1 syntax. Headroom reads the Codex config with the
standard library `tomllib`, which only implements TOML 1.0 before Python
3.15, so it rejects the file with `Invalid initial character for a key
part`. The result is what the issue shows: `headroom wrap codex` and
`headroom mcp install` refuse to register MCP servers ("is not valid
TOML ... refusing to overwrite") and session routing fails with "could
not read Codex config for session routing", even though Codex itself is
happy with the file.
`tomlkit` is already a runtime dependency (used for Codex config
recovery) and parses this syntax from 0.15.0. 0.13.3 and 0.14.0 both
fail on it (`Empty key at line 1 col 7`).
The fix adds one helper, `loads_codex_config()`, that parses with
`tomllib` first and only on `TOMLDecodeError` retries with `tomlkit`. If
`tomlkit` rejects the text too, it re-raises the original
`tomllib.TOMLDecodeError`, so every caller keeps its existing `except
tomllib.TOMLDecodeError` handling and its error messages unchanged. All
places that parse the user's Codex config now go through it.
The follow-up commit (`dc193870`) fixes a problem the first commit
exposed: once a multi-line inline table parses, a `profiles = {` table
can carry per-profile `model_provider` / `openai_base_url` keys on their
own indented lines, and the line-based "root key" edits in init, the
provider-scope install and persistent wrap mistook those nested lines
for root assignments. Init and install deleted the profile's provider,
and wrap rewrote it to `headroom` and then skipped adding a real root
`model_provider`. Those edits now only touch lines that are actual root
assignments.
Fixes #4378
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/providers/codex/config_toml.py` (new):
`loads_codex_config(content)`, `tomllib` first, `tomlkit` fallback,
original `tomllib` error if both fail.
- Routed the Codex config parse sites through it (15 call sites, 16
parse calls):
- `headroom/cli/wrap.py`: session routing read, the user-content check
and its fallback in `_inject_codex_provider_config`, and the two "never
write a malformed candidate" checks.
- `headroom/cli/init.py`: provider snapshot check, the candidate check
in `_ensure_codex_provider`, and `_codex_features`.
- `headroom/providers/codex/install.py`: `_codex_credential_store`,
`codex_auth_helper_is_referenced`, and the merged-config check before
writing.
- `headroom/providers/codex/project_context.py`: the `sqlite_home`
lookup.
- `headroom/mcp_registry/codex.py`: `_load_toml`, `_unmergeable_reason`,
and both sides of `_only_server_changed`. This registrar class is shared
with Grok, so a Grok `config.toml` with the same syntax is accepted too.
- Left on `tomllib` on purpose: `_table_path` (parses a single `[table]`
header line) and the auth snippet check in `build_codex_auth_config`
(validates text Headroom generates itself).
- `pyproject.toml` / `uv.lock`: raise the `tomlkit` minimum from
`>=0.13.0` to `>=0.15.0`, the first release that parses multi-line
inline tables. The lock already resolved 0.15.0, so `uv lock` only
changed the specifier line.
- Write paths: the user's text is never re-serialized by this change.
The registrar and the wrap/init injectors still append or splice marker
blocks as text, and the existing `tomlkit` round-trip in wrap keeps the
multi-line table byte for byte (checked below).
- Tests:
- `tests/test_provider_codex_config_toml.py`: the helper on the exact
shape from the issue (multi-line, trailing comma, whitespace-only line,
indented `}`), and that invalid TOML still raises
`tomllib.TOMLDecodeError`.
-
`tests/test_mcp_registry/test_codex_registrar.py::test_register_accepts_multiline_inline_table`:
registering into that config succeeds and the user's text is kept as a
byte-for-byte prefix.
-
`tests/test_cli/test_wrap_codex.py::test_codex_session_launch_settings_accept_multiline_inline_table`:
session routing reads that config and leaves the file untouched.
- Follow-up commit `dc193870` (multi-line inline profile overrides):
- `headroom/providers/codex/config_toml.py`:
`root_assignment_lines(content)` returns the indexes of lines that start
a key/value pair at the document root. It stops at the first table
header and skips lines inside a multi-line inline table, array or
string, with a small scanner that tracks quotes, comments and bracket
depth.
- `headroom/cli/init.py` (`_ensure_codex_provider`) and
`headroom/providers/codex/install.py`
(`_strip_root_provider_assignments`) only drop `model_provider` /
`openai_base_url` lines that are root assignments.
- `headroom/cli/wrap.py`: `_redirect_existing_top_level_keys` and
`_has_redirectable_top_level_key` only look at root assignments. Before,
they matched the key on any line, so a `[profiles.work]` section
override was also rewritten to `headroom` and counted as a root key;
that now stays the user's, matching how init, install and
`_strip_codex_headroom_blocks` already treat `[profiles.*]`.
- Tests:
`test_root_assignment_lines_skip_keys_nested_in_multiline_inline_table`,
`test_ensure_codex_provider_preserves_multiline_inline_profile_overrides`
(init),
`test_provider_scope_preserves_multiline_inline_profile_overrides`
(install) and `test_inject_preserves_multiline_inline_profile_overrides`
(wrap).
## Testing
- [x] Unit tests pass (`pytest`) - Codex, MCP registry, init and Grok
suites: 774 passed, 2 failed (the 2 failures are in
`tests/test_codex_responses_passthrough_bytes.py` and fail the same way
on `main` without this change)
- [x] Linting passes (`ruff check .`) - `ruff check` and `ruff format
--check` on the 9 changed Python files
- [x] Type checking passes (`mypy headroom`) - `mypy` on the 6 changed
source files, no issues
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
New end-to-end tests on `main` (fix stashed):
```text
$ pytest tests/test_mcp_registry/test_codex_registrar.py::test_register_accepts_multiline_inline_table tests/test_cli/test_wrap_codex.py::test_codex_session_launch_settings_accept_multiline_inline_table -q
E AssertionError: .../.codex/config.toml is not valid TOML (Invalid initial character for a key part (at line 3, column 8)); refusing to overwrite. Fix or remove the file, then re-run.
E click.exceptions.ClickException: could not read Codex config for session routing: Invalid initial character for a key part (at line 5, column 8)
============================== 2 failed in 0.38s ===============================
```
(`tests/test_provider_codex_config_toml.py` cannot import on `main`
because the helper does not exist yet.)
With the fix:
```text
$ pytest tests/test_provider_codex_config_toml.py tests/test_mcp_registry/test_codex_registrar.py::test_register_accepts_multiline_inline_table tests/test_cli/test_wrap_codex.py::test_codex_session_launch_settings_accept_multiline_inline_table -q
============================== 4 passed in 0.56s ===============================
$ pytest tests/test_provider_codex_config_toml.py tests/test_mcp_registry tests/test_cli/test_wrap_codex.py tests/test_cli/test_init_cli.py tests/test_install/test_codex_install.py tests/test_provider_codex*.py tests/test_provider_grok*.py tests/test_codex*.py -q
FAILED tests/test_codex_responses_passthrough_bytes.py::test_unmutated_zstd_post_forwards_decoded_bytes_and_strips_content_encoding - concurrent.futures._base.CancelledError
FAILED tests/test_codex_responses_passthrough_bytes.py::test_unmutated_plain_post_passes_original_bytes_through - concurrent.futures._base.CancelledError
================== 2 failed, 774 passed, 8 warnings in 50.43s ==================
$ pytest tests/test_codex_responses_passthrough_bytes.py -q # on main, without this change
========================= 2 failed, 1 warning in 4.09s =========================
$ mypy headroom/cli/init.py headroom/cli/wrap.py headroom/mcp_registry/codex.py headroom/providers/codex/install.py headroom/providers/codex/project_context.py headroom/providers/codex/config_toml.py
Success: no issues found in 6 source files
```
Follow-up commit, the three new write-path tests on the first commit
(`dfe93720`, source changes reverted):
```text
$ pytest tests/test_cli/test_init_cli.py tests/test_install/test_codex_install.py tests/test_cli/test_wrap_codex.py -k multiline_inline_profile -q
FAILED tests/test_cli/test_init_cli.py::test_ensure_codex_provider_preserves_multiline_inline_profile_overrides - AssertionError: assert {} == {'model_provi...y.example/v1'}
FAILED tests/test_install/test_codex_install.py::test_provider_scope_preserves_multiline_inline_profile_overrides - AssertionError: assert {} == {'model_provider': 'company'}
FAILED tests/test_cli/test_wrap_codex.py::TestSubscriptionRouting::test_inject_preserves_multiline_inline_profile_overrides - KeyError: 'model_provider'
====================== 3 failed, 220 deselected in 0.56s =======================
```
With `dc193870`:
```text
$ pytest tests/test_provider_codex_config_toml.py tests/test_cli/test_init_cli.py tests/test_install/test_codex_install.py tests/test_cli/test_wrap_codex.py -k "multiline_inline_profile or root_assignment" -q
======================= 4 passed, 222 deselected in 0.27s =======================
$ pytest tests/test_provider_codex_config_toml.py tests/test_cli/test_init_cli.py tests/test_install/ -q
================== 346 passed, 13 skipped, 1 warning in 3.94s ==================
$ pytest tests/test_cli/test_wrap_codex.py tests/test_cli/test_port_alignment.py -q
============================= 167 passed in 6.01s ==============================
$ pytest tests/test_mcp_registry/ tests/test_provider_codex_install.py tests/test_cli/test_install_cli.py tests/test_cli/test_recover_codex.py -q
============================= 364 passed in 4.71s ==============================
$ pytest tests/test_cli/test_wrap_persistent.py tests/test_codex_project_context.py tests/test_cli/test_wrap_stale_marker.py tests/test_cli/test_wrap_dead_marker_selfheal.py tests/test_provider_codex_runtime.py tests/test_install_tool_env_reconcile.py -q
======================== 4 failed, 261 passed in 9.01s =========================
```
The 4 failures are in `tests/test_cli/test_wrap_persistent.py`
(`AttributeError: '_Manifest' object has no attribute 'runtime_kind'`)
and fail the same way with this PR's changes stashed. `ruff check`,
`ruff format --check` (8 files) and `mypy` on the 4 changed source files
are clean.
tomlkit versions on the issue's snippet:
```text
0.13.3 FAIL Empty key at line 1 col 7
0.14.0 FAIL Empty key at line 1 col 7
0.15.0 {'env': {'A': 'x'}}
```
## Real Behavior Proof
- Environment: macOS 26.2 (Apple Silicon), Python 3.14.7, repo venv with
the `mcp` package installed, a throwaway `HOME`/`CODEX_HOME` whose
`config.toml` holds the issue's config:
```toml
[mcp_servers.dingtalk]
command = "npx"
env = {
DINGTALK_Client_ID = "**********",
}
```
- Exact command / steps: `HOME=<tmp>/home CODEX_HOME=<tmp>/home/.codex
python -m headroom.cli mcp install --agent codex`, then compare the
start of the resulting file with the original bytes and count
`[mcp_servers.headroom]` tables. Also called
`wrap._inject_codex_provider_config(8787)` and
`init._ensure_codex_provider(path, 8787)` on the same config to check
the write paths.
- Observed result: before (`main`):
```text
codex: install failed (failed): <tmp>/home/.codex/config.toml is not
valid TOML (Invalid initial character for a key part (at line 3, column
8)); refusing to overwrite. Fix or remove the file, then re-run.
```
After:
```text
codex: registered (restart the agent if it was already running)
```
The user's original bytes are an exact prefix of the new file and one
`[mcp_servers.headroom]` table was appended. The wrap injector printed
`Codex config: injected Headroom provider (WS + HTTP)`, and in both the
wrap and init outputs the user's multi-line table is still present
verbatim, the file parses, `env` is `{'DINGTALK_Client_ID':
'**********'}` and `model_provider` is `headroom`.
- Follow-up commit: with a throwaway `HOME`/`CODEX_HOME` holding
```toml
model_provider = "openai"
profiles = {
work = {
model_provider = "company",
},
}
```
I ran `init._ensure_codex_provider`, `install.apply_provider_scope`
(provider scope) and `wrap._inject_codex_provider_config` (same config
without the root key) and parsed the result:
```text
== before (dfe93720)
init: root = headroom work = {}
install: root = headroom work = {}
wrap: root = None work = {'model_provider': 'headroom'}
== after (dc193870)
init: root = headroom work = {'model_provider': 'company'}
install: root = headroom work = {'model_provider': 'company'}
wrap: root = headroom work = {'model_provider': 'company'}
```
- Not tested: a full `headroom wrap codex` run with a live proxy and the
real Codex CLI, and Python 3.10 (where `tomli` is the first parser; the
`tomlkit` fallback is the same there).
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: only for Codex configs `tomllib`
rejects but `tomlkit` 0.15 accepts (TOML 1.1 multi-line inline tables).
Valid TOML 1.0 still goes through `tomllib` exactly as before, and text
both parsers reject fails with the same error as before.
- Kill switch / disable path: none needed; the fallback only runs after
`tomllib` has already failed.
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the two commits
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`, it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A
## Additional Notes
- Documentation: no user-facing docs describe which TOML syntax the
Codex config may use, so nothing to update.
- Python 3.15's `tomllib` implements TOML 1.1 and parses this config
directly (as noted in the issue thread), so on 3.15 the fallback simply
never runs.
- `tomlkit` is a bit more permissive than `tomllib` only where TOML 1.1
is; it still rejects duplicate keys and malformed tables, so the
existing "refuse to overwrite an unparseable file" guards still trigger
for genuinely broken configs (covered by the existing registrar tests).
- Dependency review for the `tomlkit` minimum bump (`>=0.13.0,<1.0` to
`>=0.15.0,<1.0`):
- Why this package: it is already a direct runtime dependency (Codex
config recovery and the memory MCP block edit in `wrap.py`), so no new
package enters the tree. Writing a TOML 1.1 parser here is not worth it,
and `tomllib` only gains TOML 1.1 in Python 3.15.
- Who maintains it: the Poetry project (`python-poetry/tomlkit`, MIT),
with Frost Ming and Sébastien Eustace as the main maintainers. It is
actively released: 0.13.3 (2025-06), 0.14.0 (2026-01), 0.15.0 (2026-05),
0.15.1 (2026-07), 0.15.2 (2026-10). No GitHub security advisories are
published for the repo.
- Install surface: unchanged. Pure Python, a single `py3-none-any` wheel
(about 41 KB for 0.15.0), no runtime dependencies, no native code, no
install-time or runtime network access. `uv.lock` already resolved
0.15.0 before this PR, so the locked version and hashes do not change;
only the specifier line moves.
- Why this version: required new functionality. 0.15.0 is the first
release with TOML 1.1 support ("feat(spec): update to toml spec v1.1",
python-poetry/tomlkit#456), which is what parses multi-line inline
tables with trailing commas. 0.13.3 and 0.14.0 both reject the issue's
snippet (`Empty key at line 1 col 7`, output above). 0.15.0 also
includes an upstream parser hang fix (python-poetry/tomlkit#470).
---------
Co-authored-by: Abelo9996 <Abelo9996@users.noreply.github.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Abelo9996
JerrettDavis
parent
4ee616453f
commit
4a851ed254
+21
-13
@@ -58,6 +58,11 @@ from headroom.install.state import ManifestError, load_manifest, save_manifest
|
||||
from headroom.install.supervisors import start_supervisor
|
||||
from headroom.providers.claude import TOOL_SEARCH_DEFAULT, TOOL_SEARCH_ENV
|
||||
from headroom.providers.claude.runtime import TOOL_SEARCH_FOUNDRY_DEFAULT
|
||||
from headroom.providers.codex.config_toml import (
|
||||
loads_codex_config,
|
||||
root_assignment_lines,
|
||||
split_toml_lines,
|
||||
)
|
||||
from headroom.providers.codex.install import (
|
||||
CodexAuthConfigError,
|
||||
build_codex_auth_config,
|
||||
@@ -91,6 +96,7 @@ _STARTUP_READY_TIMEOUT_SECONDS = 15
|
||||
# report ready, and every session is permanently cold.
|
||||
_HOOK_ENSURE_TIMEOUT_SECONDS = 60
|
||||
_TOML_TABLE_HEADER_RE = re.compile(r"^[ \t]*(?:\[\[[^\]\r\n]+\]\]|\[[^\]\r\n]+\])[ \t]*(?:#.*)?$")
|
||||
_ROOT_PROVIDER_KEY_RE = re.compile(r"[ \t]*(?:model_provider|openai_base_url)[ \t]*=")
|
||||
_TOML_FEATURES_NAME_RE = r"(?:features|\"features\"|'features')"
|
||||
_TOML_CODEX_HOOKS_NAME_RE = r"(?:codex_hooks|\"codex_hooks\"|'codex_hooks')"
|
||||
_CODEX_FEATURES_TABLE_RE = re.compile(
|
||||
@@ -298,7 +304,7 @@ def _replace_marker_block(
|
||||
# The block carries top-level keys, so it must sit above the first table
|
||||
# header; appended after a table (e.g. [features]) TOML scopes those keys
|
||||
# into that table and Codex rejects the config (#260).
|
||||
lines = content.splitlines()
|
||||
lines = split_toml_lines(content)
|
||||
for index, line in enumerate(lines):
|
||||
if _TOML_TABLE_HEADER_RE.search(line):
|
||||
head = "\n".join(lines[:index]).rstrip()
|
||||
@@ -347,7 +353,7 @@ def _codex_init_provider_snapshot(content: str) -> str | None:
|
||||
snapshot = json.loads(match.group(1))
|
||||
if not isinstance(snapshot, str):
|
||||
raise ValueError("Invalid Headroom init provider snapshot")
|
||||
tomllib.loads(snapshot)
|
||||
loads_codex_config(snapshot)
|
||||
return snapshot
|
||||
|
||||
|
||||
@@ -470,19 +476,21 @@ def _ensure_codex_provider(path: Path, port: int) -> None:
|
||||
# first table header) -- these keys also appear legitimately inside
|
||||
# [profiles.*] tables as per-profile overrides, and stripping them there
|
||||
# silently reroutes the user's profiles to the injected "headroom" default.
|
||||
_first_table = re.search(r"(?m)^[ \t]*\[", content)
|
||||
_split = _first_table.start() if _first_table else len(content)
|
||||
root, rest = content[:_split], content[_split:]
|
||||
root = re.sub(r"(?m)^[ \t]*model_provider[ \t]*=.*\r?\n", "", root)
|
||||
root = re.sub(r"(?m)^[ \t]*openai_base_url[ \t]*=.*\r?\n", "", root)
|
||||
content = root + rest
|
||||
# Lines nested in a multi-line inline table (e.g. ``profiles = {``) are
|
||||
# profile overrides too, so only true root assignments are dropped.
|
||||
_root_lines = root_assignment_lines(content)
|
||||
content = "".join(
|
||||
line
|
||||
for index, line in enumerate(split_toml_lines(content, keepends=True))
|
||||
if index not in _root_lines or not _ROOT_PROVIDER_KEY_RE.match(line)
|
||||
)
|
||||
# The provider block carries top-level keys (model_provider, openai_base_url),
|
||||
# so it must land at the document root rather than after a trailing table (#260).
|
||||
content = _replace_marker_block(
|
||||
content, _CODEX_PROVIDER_MARKER_START, _CODEX_PROVIDER_MARKER_END, block, at_root=True
|
||||
)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
tomllib.loads(content)
|
||||
loads_codex_config(content)
|
||||
fsutil.write_text(path, content)
|
||||
# Codex filters its history menu by the active model_provider, so existing
|
||||
# native threads vanish once we switch to "headroom". Retag them to match the
|
||||
@@ -511,7 +519,7 @@ def _codex_features(content: str) -> dict[str, Any] | None:
|
||||
if not content.strip():
|
||||
return None
|
||||
try:
|
||||
parsed = tomllib.loads(content)
|
||||
parsed = loads_codex_config(content)
|
||||
except tomllib.TOMLDecodeError:
|
||||
return None
|
||||
features = parsed.get("features")
|
||||
@@ -523,7 +531,7 @@ def _codex_features_has_hooks(content: str) -> bool:
|
||||
if features is None:
|
||||
# Keep init resilient for already-invalid user configs; this fallback
|
||||
# only needs to avoid adding a second obvious hooks line.
|
||||
lines = content.splitlines()
|
||||
lines = split_toml_lines(content)
|
||||
features_index = _codex_features_table_index(lines)
|
||||
if features_index is None:
|
||||
return False
|
||||
@@ -538,7 +546,7 @@ def _codex_features_has_hooks(content: str) -> bool:
|
||||
|
||||
|
||||
def _strip_codex_legacy_feature_flag(content: str) -> str:
|
||||
lines = content.splitlines(keepends=True)
|
||||
lines = split_toml_lines(content, keepends=True)
|
||||
retained: list[str] = []
|
||||
in_features = False
|
||||
in_root = True
|
||||
@@ -588,7 +596,7 @@ def _ensure_codex_feature_flag(path: Path) -> None:
|
||||
# the only work.
|
||||
pass
|
||||
else:
|
||||
lines = content.splitlines()
|
||||
lines = split_toml_lines(content)
|
||||
features_index = _codex_features_table_index(lines)
|
||||
if features_index is not None:
|
||||
# Leading blank line matches the normalisation _replace_marker_block
|
||||
|
||||
+25
-11
@@ -110,6 +110,11 @@ from headroom.providers.claude import (
|
||||
)
|
||||
from headroom.providers.claude.runtime import TOOL_SEARCH_FOUNDRY_DEFAULT
|
||||
from headroom.providers.codex import build_launch_env as _build_codex_launch_env
|
||||
from headroom.providers.codex.config_toml import (
|
||||
loads_codex_config,
|
||||
root_assignment_lines,
|
||||
split_toml_lines,
|
||||
)
|
||||
from headroom.providers.codex.install import (
|
||||
CodexAuthConfigError,
|
||||
build_codex_auth_config,
|
||||
@@ -3083,7 +3088,7 @@ def _codex_session_launch_settings(
|
||||
"""Build process-local routing while preserving the selected provider id."""
|
||||
config_file, _ = _codex_config_paths()
|
||||
try:
|
||||
config = tomllib.loads(_read_text(config_file)) if config_file.exists() else {}
|
||||
config = loads_codex_config(_read_text(config_file)) if config_file.exists() else {}
|
||||
except (OSError, tomllib.TOMLDecodeError) as exc:
|
||||
raise click.ClickException(
|
||||
f"could not read Codex config for session routing: {exc}"
|
||||
@@ -3344,19 +3349,28 @@ def _redirect_existing_top_level_keys(content: str, port: int) -> str:
|
||||
|
||||
return _replace
|
||||
|
||||
redirected = content
|
||||
# Only root assignments: the same keys inside [profiles.*] tables or a
|
||||
# multi-line inline ``profiles = {`` table are the user's per-profile routing.
|
||||
lines = split_toml_lines(content, keepends=True)
|
||||
root_lines = sorted(root_assignment_lines(content))
|
||||
for key in _REDIRECTABLE_KEYS:
|
||||
pattern = re.compile(rf'(?m)^[ \t]*{re.escape(key)}[ \t]*=[ \t]*"(?P<value>[^"\n]*)"[^\n]*')
|
||||
redirected = pattern.sub(_make_replacer(key, port), redirected, count=1)
|
||||
return redirected
|
||||
pattern = re.compile(rf'[ \t]*{re.escape(key)}[ \t]*=[ \t]*"(?P<value>[^"\n]*)"[^\n]*')
|
||||
replace = _make_replacer(key, port)
|
||||
for index in root_lines:
|
||||
match = pattern.match(lines[index])
|
||||
if match:
|
||||
lines[index] = replace(match) + lines[index][match.end() :]
|
||||
break
|
||||
return "".join(lines)
|
||||
|
||||
|
||||
def _has_redirectable_top_level_key(content: str, key: str) -> bool:
|
||||
"""Return True if ``content`` declares ``key = "..."`` as a top-level key."""
|
||||
import re # local import to match the module's existing convention
|
||||
|
||||
pattern = re.compile(rf'(?m)^[ \t]*{key}[ \t]*=[ \t]*"[^"\n]*"')
|
||||
return pattern.search(content) is not None
|
||||
pattern = re.compile(rf'[ \t]*{key}[ \t]*=[ \t]*"[^"\n]*"')
|
||||
lines = split_toml_lines(content, keepends=True)
|
||||
return any(pattern.match(lines[index]) for index in root_assignment_lines(content))
|
||||
|
||||
|
||||
def _codex_config_has_headroom_markers(content: str) -> bool:
|
||||
@@ -3715,10 +3729,10 @@ def _inject_codex_provider_config(port: int) -> str | None:
|
||||
# name a line the user never wrote. Only the user's part has to
|
||||
# parse: stripping a stale Headroom block may be what repairs it.
|
||||
try:
|
||||
tomllib.loads(content)
|
||||
loads_codex_config(content)
|
||||
except tomllib.TOMLDecodeError:
|
||||
try:
|
||||
tomllib.loads(original)
|
||||
loads_codex_config(original)
|
||||
reason = "it is no longer valid TOML once Headroom's block is removed"
|
||||
except tomllib.TOMLDecodeError as exc:
|
||||
reason = f"it is not valid TOML ({exc})"
|
||||
@@ -3766,7 +3780,7 @@ def _inject_codex_provider_config(port: int) -> str | None:
|
||||
)
|
||||
|
||||
# Never replace a usable config with a malformed generated candidate.
|
||||
tomllib.loads(content)
|
||||
loads_codex_config(content)
|
||||
_write_text(config_file, content)
|
||||
click.echo(f" Codex config: injected Headroom provider (WS + HTTP) into {config_file}")
|
||||
if custom_upstream_base_url:
|
||||
@@ -4063,7 +4077,7 @@ def _inject_memory_mcp_config(user_id: str) -> None:
|
||||
content = mcp_section
|
||||
|
||||
# Never replace a usable config with a malformed generated candidate.
|
||||
tomllib.loads(content)
|
||||
loads_codex_config(content)
|
||||
_write_text(config_file, content)
|
||||
click.echo(f" Memory MCP: registered in {config_file}")
|
||||
except Exception as e:
|
||||
|
||||
@@ -18,6 +18,7 @@ from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from headroom import fsutil
|
||||
from headroom.providers.codex.config_toml import loads_codex_config, split_toml_lines
|
||||
|
||||
from .base import (
|
||||
MCPRegistrar,
|
||||
@@ -84,7 +85,7 @@ def _evict_foreign_tables(content: str, server_name: str, start: str, end: str)
|
||||
move, so a header-like line inside a multi-line string can split a table
|
||||
wrongly; callers must check the final result with ``_only_server_changed``.
|
||||
"""
|
||||
lines = content.splitlines(keepends=True)
|
||||
lines = split_toml_lines(content, keepends=True)
|
||||
try:
|
||||
i = next(n for n, line in enumerate(lines) if line.rstrip("\r\n") == start)
|
||||
j = next(n for n in range(i + 1, len(lines)) if lines[n].rstrip("\r\n") == end)
|
||||
@@ -121,7 +122,7 @@ def _only_server_changed(old: str, new: str, server_name: str) -> bool:
|
||||
makes this False and the rewrite is refused instead of written.
|
||||
"""
|
||||
try:
|
||||
before, after = tomllib.loads(old), tomllib.loads(new)
|
||||
before, after = loads_codex_config(old), loads_codex_config(new)
|
||||
except tomllib.TOMLDecodeError:
|
||||
return False
|
||||
return _without_server(before, server_name) == _without_server(after, server_name)
|
||||
@@ -255,7 +256,7 @@ class _TomlMarkerRegistrar(MCPRegistrar):
|
||||
# Read via fsutil (UTF-8 with locale fallback) so a config that a
|
||||
# tool wrote in the system locale (e.g. GBK) still parses instead
|
||||
# of failing tomllib's UTF-8 requirement. See #733.
|
||||
data = tomllib.loads(fsutil.read_text(self._config_file))
|
||||
data = loads_codex_config(fsutil.read_text(self._config_file))
|
||||
except (tomllib.TOMLDecodeError, OSError):
|
||||
return {}
|
||||
return data if isinstance(data, dict) else {}
|
||||
@@ -276,7 +277,7 @@ class _TomlMarkerRegistrar(MCPRegistrar):
|
||||
if not raw.strip():
|
||||
return None
|
||||
try:
|
||||
data = tomllib.loads(fsutil.read_text(self._config_file))
|
||||
data = loads_codex_config(fsutil.read_text(self._config_file))
|
||||
except (tomllib.TOMLDecodeError, OSError) as exc:
|
||||
return f"{self._config_file} is not valid TOML ({exc})"
|
||||
if not isinstance(data, dict):
|
||||
|
||||
@@ -0,0 +1,102 @@
|
||||
"""Parse Codex ``config.toml`` with the syntax Codex itself accepts."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from typing import Any, cast
|
||||
|
||||
import tomlkit
|
||||
from tomlkit.exceptions import TOMLKitError
|
||||
|
||||
if sys.version_info >= (3, 11):
|
||||
import tomllib
|
||||
else: # pragma: no cover - exercised only on Python 3.10
|
||||
import tomli as tomllib # type: ignore[no-redef]
|
||||
|
||||
|
||||
def loads_codex_config(content: str) -> dict[str, Any]:
|
||||
"""Parse Codex config text, falling back to ``tomlkit`` for TOML 1.1 syntax.
|
||||
|
||||
Codex accepts TOML 1.1, where an inline table may span several lines and
|
||||
end with a trailing comma (#4378). ``tomllib`` before Python 3.15 only
|
||||
implements TOML 1.0 and rejects that, so retry with ``tomlkit`` (0.15+),
|
||||
which parses it. Text neither parser accepts raises the original
|
||||
``tomllib.TOMLDecodeError``, so callers keep their existing error handling.
|
||||
"""
|
||||
try:
|
||||
return cast("dict[str, Any]", tomllib.loads(content))
|
||||
except tomllib.TOMLDecodeError as exc:
|
||||
try:
|
||||
return cast("dict[str, Any]", tomlkit.parse(content).unwrap())
|
||||
except TOMLKitError:
|
||||
raise exc from None
|
||||
|
||||
|
||||
def split_toml_lines(content: str, keepends: bool = False) -> list[str]:
|
||||
"""Split ``content`` like ``str.splitlines``, but only at TOML newlines.
|
||||
|
||||
TOML ends a line at LF or CRLF only. ``str.splitlines`` also breaks at
|
||||
characters such as U+2028 and U+0085, which TOML allows inside a
|
||||
single-line string or comment, so its pieces are not always TOML lines.
|
||||
"""
|
||||
lines = [line + "\n" for line in content.split("\n")]
|
||||
lines[-1] = lines[-1][:-1]
|
||||
if not lines[-1]:
|
||||
lines.pop()
|
||||
if keepends:
|
||||
return lines
|
||||
return [line.removesuffix("\n").removesuffix("\r") for line in lines]
|
||||
|
||||
|
||||
def root_assignment_lines(content: str) -> set[int]:
|
||||
"""Return indexes of the lines that start a key/value pair at the document root.
|
||||
|
||||
Indexes refer to ``split_toml_lines(content)``. Only lines before
|
||||
the first table header count, and a line inside a multi-line inline table,
|
||||
array, or string is skipped, so a key nested in a multi-line inline table
|
||||
(TOML 1.1, e.g. ``profiles = {\\n work = { model_provider = ... } }``) is
|
||||
never mistaken for a root key.
|
||||
"""
|
||||
roots: set[int] = set()
|
||||
depth = 0
|
||||
open_string: str | None = None # delimiter of an open multi-line string
|
||||
for index, line in enumerate(split_toml_lines(content, keepends=True)):
|
||||
if depth == 0 and open_string is None:
|
||||
stripped = line.lstrip(" \t")
|
||||
if stripped.startswith("["):
|
||||
break # first table header ends the root
|
||||
if stripped and stripped[0] not in "#\r\n":
|
||||
roots.add(index)
|
||||
pos = 0
|
||||
while pos < len(line):
|
||||
if open_string is not None:
|
||||
if line.startswith(open_string, pos):
|
||||
pos += 3
|
||||
# Up to two quotes may directly precede the closing delimiter.
|
||||
for _ in range(2):
|
||||
if pos < len(line) and line[pos] == open_string[0]:
|
||||
pos += 1
|
||||
open_string = None
|
||||
elif open_string == '"""' and line[pos] == "\\":
|
||||
pos += 2
|
||||
else:
|
||||
pos += 1
|
||||
continue
|
||||
char = line[pos]
|
||||
if char == "#":
|
||||
break
|
||||
if line.startswith('"""', pos) or line.startswith("'''", pos):
|
||||
open_string = line[pos : pos + 3]
|
||||
pos += 3
|
||||
elif char in "\"'":
|
||||
pos += 1
|
||||
while pos < len(line) and line[pos] not in (char, "\n"):
|
||||
pos += 2 if char == '"' and line[pos] == "\\" else 1
|
||||
pos += 1
|
||||
else:
|
||||
if char in "[{":
|
||||
depth += 1
|
||||
elif char in "]}" and depth:
|
||||
depth -= 1
|
||||
pos += 1
|
||||
return roots
|
||||
@@ -21,6 +21,7 @@ from headroom._subprocess import run
|
||||
from headroom.install.models import ConfigScope, DeploymentManifest, ManagedMutation, ToolTarget
|
||||
from headroom.install.paths import codex_config_path
|
||||
|
||||
from .config_toml import loads_codex_config, root_assignment_lines, split_toml_lines
|
||||
from .runtime import proxy_base_url
|
||||
from .threads import retag_to_headroom, retag_to_native
|
||||
|
||||
@@ -65,7 +66,7 @@ print(key, end="")
|
||||
|
||||
def _codex_credential_store(config_dir: Path) -> str | None:
|
||||
try:
|
||||
config = tomllib.loads((config_dir / "config.toml").read_text(encoding="utf-8"))
|
||||
config = loads_codex_config((config_dir / "config.toml").read_text(encoding="utf-8"))
|
||||
except (OSError, tomllib.TOMLDecodeError):
|
||||
return None
|
||||
store = config.get("cli_auth_credentials_store")
|
||||
@@ -261,7 +262,7 @@ def codex_auth_helper_is_referenced(content: str, helper_path: str) -> bool | No
|
||||
the helper instead of risking deletion of a pre-existing user file.
|
||||
"""
|
||||
try:
|
||||
document = tomllib.loads(content)
|
||||
document = loads_codex_config(content)
|
||||
except (tomllib.TOMLDecodeError, TypeError):
|
||||
return None
|
||||
|
||||
@@ -358,7 +359,7 @@ def _insert_block_at_root(content: str, block: str) -> str:
|
||||
root instead.
|
||||
"""
|
||||
block = block.strip()
|
||||
lines = content.splitlines()
|
||||
lines = split_toml_lines(content)
|
||||
for index, line in enumerate(lines):
|
||||
if _TOML_TABLE_HEADER_RE.search(line):
|
||||
head = "\n".join(lines[:index]).rstrip()
|
||||
@@ -369,19 +370,18 @@ def _insert_block_at_root(content: str, block: str) -> str:
|
||||
|
||||
|
||||
def _strip_root_provider_assignments(content: str) -> str:
|
||||
"""Remove root provider assignments without touching table-scoped settings."""
|
||||
lines = content.splitlines(keepends=True)
|
||||
kept: list[str] = []
|
||||
in_root = True
|
||||
for line in lines:
|
||||
if in_root and _TOML_TABLE_HEADER_RE.search(line):
|
||||
in_root = False
|
||||
if in_root and (
|
||||
_ROOT_MODEL_PROVIDER_RE.match(line) or _ROOT_OPENAI_BASE_URL_RE.match(line)
|
||||
):
|
||||
continue
|
||||
kept.append(line)
|
||||
return "".join(kept)
|
||||
"""Remove root provider assignments without touching table-scoped settings.
|
||||
|
||||
Keys nested in a multi-line inline table (e.g. a ``profiles = {`` override)
|
||||
are not root assignments and are kept.
|
||||
"""
|
||||
root_lines = root_assignment_lines(content)
|
||||
return "".join(
|
||||
line
|
||||
for index, line in enumerate(split_toml_lines(content, keepends=True))
|
||||
if index not in root_lines
|
||||
or not (_ROOT_MODEL_PROVIDER_RE.match(line) or _ROOT_OPENAI_BASE_URL_RE.match(line))
|
||||
)
|
||||
|
||||
|
||||
def apply_provider_scope(manifest: DeploymentManifest) -> ManagedMutation | None:
|
||||
@@ -411,7 +411,7 @@ def apply_provider_scope(manifest: DeploymentManifest) -> ManagedMutation | None
|
||||
existing = _CODEX_PATTERN.sub("", existing)
|
||||
existing = _strip_root_provider_assignments(existing)
|
||||
merged = _insert_block_at_root(existing, section)
|
||||
tomllib.loads(merged)
|
||||
loads_codex_config(merged)
|
||||
path.write_text(merged, encoding="utf-8")
|
||||
# Pull existing native threads into the headroom-provider menu so Codex's
|
||||
# history list stays whole once it routes through Headroom. Best-effort.
|
||||
|
||||
@@ -17,6 +17,7 @@ from typing import Any, Literal, cast
|
||||
import tomllib
|
||||
|
||||
from headroom.memory.storage_router import ProjectResolver, RequestContext
|
||||
from headroom.providers.codex.config_toml import loads_codex_config
|
||||
from headroom.providers.codex.threads import _codex_state_db_paths
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -295,7 +296,7 @@ class CodexProjectContextResolver:
|
||||
return self._configured_sqlite_home
|
||||
codex_home = Path(os.environ.get("CODEX_HOME", "").strip() or Path.home() / ".codex")
|
||||
try:
|
||||
config = tomllib.loads((codex_home / "config.toml").read_text(encoding="utf-8"))
|
||||
config = loads_codex_config((codex_home / "config.toml").read_text(encoding="utf-8"))
|
||||
configured_sqlite_home = config.get("sqlite_home")
|
||||
if isinstance(configured_sqlite_home, str) and configured_sqlite_home.strip():
|
||||
return Path(configured_sqlite_home).expanduser()
|
||||
|
||||
+1
-1
@@ -62,7 +62,7 @@ dependencies = [
|
||||
# future patched one.
|
||||
"ast-grep-cli>=0.30.0,!=0.44.1",
|
||||
"pyyaml>=6.0", # omp wrap: parse/merge omp's models.yml registry
|
||||
"tomlkit>=0.13.0,<1.0", # Loss-minimizing Codex config recovery
|
||||
"tomlkit>=0.15.0,<1.0", # Loss-minimizing Codex config recovery
|
||||
# Verify upstream TLS through the OS trust store (macOS Keychain, Windows
|
||||
# cert store), where IT installs corporate TLS-inspection roots (Zscaler,
|
||||
# Netskope). Pure Python, no deps; the same library pip uses by default.
|
||||
|
||||
@@ -661,6 +661,54 @@ def test_ensure_codex_provider_preserves_profile_overrides(monkeypatch, tmp_path
|
||||
assert parsed["profiles"]["work"]["openai_base_url"] == "https://azure.example/v1"
|
||||
|
||||
|
||||
def test_ensure_codex_provider_preserves_multiline_inline_profile_overrides(
|
||||
monkeypatch, tmp_path: Path
|
||||
) -> None:
|
||||
"""A profile in a multi-line inline table (TOML 1.1) keeps its provider."""
|
||||
from headroom.providers.codex.config_toml import loads_codex_config
|
||||
|
||||
init_cli, _ = _load_init_module(monkeypatch)
|
||||
path = tmp_path / "config.toml"
|
||||
path.write_text(
|
||||
'model_provider = "openai"\n'
|
||||
"profiles = {\n"
|
||||
" work = {\n"
|
||||
' model_provider = "company",\n'
|
||||
' openai_base_url = "https://company.example/v1",\n'
|
||||
" },\n"
|
||||
"}\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
init_cli._ensure_codex_provider(path, 8787)
|
||||
|
||||
parsed = loads_codex_config(path.read_text(encoding="utf-8"))
|
||||
assert parsed["model_provider"] == "headroom"
|
||||
assert parsed["profiles"]["work"] == {
|
||||
"model_provider": "company",
|
||||
"openai_base_url": "https://company.example/v1",
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("sep", ["
", "\x85"])
|
||||
def test_ensure_codex_provider_keeps_single_line_string_with_unicode_separator(
|
||||
monkeypatch, tmp_path: Path, sep: str
|
||||
) -> None:
|
||||
"""Provider-looking text after U+2028/U+0085 in a string stays in the value."""
|
||||
from headroom.providers.codex.config_toml import loads_codex_config
|
||||
|
||||
init_cli, _ = _load_init_module(monkeypatch)
|
||||
path = tmp_path / "config.toml"
|
||||
notes = f'hello{sep}model_provider = "company"{sep}world'
|
||||
path.write_text(f"notes = '{notes}'\n\n[features]\nhooks = true\n", encoding="utf-8")
|
||||
|
||||
init_cli._ensure_codex_provider(path, 8787)
|
||||
|
||||
parsed = loads_codex_config(path.read_text(encoding="utf-8"))
|
||||
assert parsed["notes"] == notes
|
||||
assert parsed["model_provider"] == "headroom"
|
||||
|
||||
|
||||
def test_ensure_codex_provider_emits_requires_openai_auth_for_chatgpt(
|
||||
monkeypatch, tmp_path: Path
|
||||
) -> None:
|
||||
@@ -707,6 +755,23 @@ def test_ensure_codex_feature_flag_replaces_existing_marker(monkeypatch, tmp_pat
|
||||
assert "codex_hooks" not in content
|
||||
|
||||
|
||||
@pytest.mark.parametrize("sep", ["
", "\x85"])
|
||||
def test_ensure_codex_feature_flag_keeps_single_line_string_with_unicode_separator(
|
||||
monkeypatch, tmp_path: Path, sep: str
|
||||
) -> None:
|
||||
from headroom.providers.codex.config_toml import loads_codex_config
|
||||
|
||||
init_cli, _ = _load_init_module(monkeypatch)
|
||||
path = tmp_path / "config.toml"
|
||||
notes = f"hello{sep}codex_hooks = true{sep}world"
|
||||
path.write_text(f"[features]\nnotes = '{notes}'\n", encoding="utf-8")
|
||||
|
||||
init_cli._ensure_codex_feature_flag(path)
|
||||
|
||||
parsed = loads_codex_config(path.read_text(encoding="utf-8"))
|
||||
assert parsed["features"] == {"notes": notes, "hooks": True}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("custom_home", [False, True])
|
||||
@pytest.mark.parametrize("auth_mode", ["apikey", "chatgpt"])
|
||||
def test_local_codex_provider_reads_credentials_from_user_home(
|
||||
|
||||
@@ -846,6 +846,27 @@ class TestSubscriptionRouting:
|
||||
content = (tmp_path / ".codex" / "config.toml").read_text(encoding="utf-8")
|
||||
assert 'openai_base_url = "http://127.0.0.1:8787/v1"' in content
|
||||
|
||||
def test_inject_preserves_multiline_inline_profile_overrides(
|
||||
self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
||||
) -> None:
|
||||
"""Keys nested in a multi-line inline profiles table are not root keys."""
|
||||
from headroom.providers.codex.config_toml import loads_codex_config
|
||||
|
||||
_set_test_home(monkeypatch, tmp_path)
|
||||
config_file = tmp_path / ".codex" / "config.toml"
|
||||
config_file.parent.mkdir(parents=True)
|
||||
config_file.write_text(
|
||||
'profiles = {\n work = {\n model_provider = "company",\n },\n}\n',
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
wrap_mod._inject_codex_provider_config(8787)
|
||||
|
||||
parsed = loads_codex_config(config_file.read_text(encoding="utf-8"))
|
||||
assert parsed["model_provider"] == "headroom"
|
||||
assert parsed["openai_base_url"] == "http://127.0.0.1:8787/v1"
|
||||
assert parsed["profiles"]["work"] == {"model_provider": "company"}
|
||||
|
||||
def test_inject_emits_requires_openai_auth_for_chatgpt(
|
||||
self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
||||
) -> None:
|
||||
@@ -1138,6 +1159,14 @@ class TestInjectAvoidsDuplicateTopLevelKeys:
|
||||
# the marker-delimited top-level block.
|
||||
assert wrap_mod._redirect_existing_top_level_keys(blank, 8787) == blank
|
||||
|
||||
@pytest.mark.parametrize("sep", ["
", "\x85"])
|
||||
def test_redirect_leaves_provider_text_inside_single_line_string(self, sep: str) -> None:
|
||||
# TOML allows U+2028 and U+0085 inside a single-line string, so the
|
||||
# text after them is still part of the value, not a root assignment.
|
||||
content = f"notes = 'hello{sep}model_provider = \"company\"{sep}world'\n"
|
||||
assert wrap_mod._redirect_existing_top_level_keys(content, 8787) == content
|
||||
assert not wrap_mod._has_redirectable_top_level_key(content, "model_provider")
|
||||
|
||||
def test_inject_preserves_user_value_in_trailing_comment(
|
||||
self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
||||
) -> None:
|
||||
@@ -1564,6 +1593,38 @@ def test_codex_session_launch_settings_keep_routing_process_local(
|
||||
assert config_file.read_text(encoding="utf-8") == original_config
|
||||
|
||||
|
||||
def test_codex_session_launch_settings_accept_multiline_inline_table(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
||||
) -> None:
|
||||
# #4378: Codex accepts an inline table split across lines.
|
||||
_set_test_home(monkeypatch, tmp_path)
|
||||
codex_home = tmp_path / "custom-codex-home"
|
||||
codex_home.mkdir()
|
||||
monkeypatch.setenv("CODEX_HOME", str(codex_home))
|
||||
monkeypatch.setattr(wrap_mod, "_project_name_from_cwd", lambda: None)
|
||||
config_file = codex_home / "config.toml"
|
||||
original_config = (
|
||||
'model = "gpt-5"\n\n'
|
||||
"[mcp_servers.dingtalk]\n"
|
||||
'command = "npx"\n'
|
||||
"env = {\n"
|
||||
' DINGTALK_Client_ID = "**********",\n'
|
||||
" \n"
|
||||
" }\n"
|
||||
)
|
||||
config_file.write_text(original_config, encoding="utf-8")
|
||||
|
||||
args, env, _ = wrap_mod._codex_session_launch_settings(
|
||||
port=9898,
|
||||
codex_args=("exec", "hello"),
|
||||
environ={"CODEX_HOME": str(codex_home)},
|
||||
)
|
||||
|
||||
assert args[:2] == ("--config", 'openai_base_url="http://127.0.0.1:9898/v1"')
|
||||
assert env["OPENAI_BASE_URL"] == "http://127.0.0.1:9898/v1"
|
||||
assert config_file.read_text(encoding="utf-8") == original_config
|
||||
|
||||
|
||||
def test_codex_session_launch_settings_preserve_custom_provider_identity(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
||||
) -> None:
|
||||
|
||||
@@ -13,6 +13,7 @@ else: # pragma: no cover - exercised only on Python 3.10
|
||||
import tomli as tomllib # type: ignore[no-redef]
|
||||
|
||||
from headroom.install.models import DeploymentManifest
|
||||
from headroom.providers.codex.config_toml import loads_codex_config
|
||||
from headroom.providers.codex.install import (
|
||||
CodexAuthConfigError,
|
||||
_codex_login_status,
|
||||
@@ -122,6 +123,44 @@ def test_api_key_auth_is_persisted_as_a_codex_command(tmp_path: Path, monkeypatc
|
||||
assert not helper.exists()
|
||||
|
||||
|
||||
def test_provider_scope_preserves_multiline_inline_profile_overrides(
|
||||
tmp_path: Path, monkeypatch
|
||||
) -> None:
|
||||
config = tmp_path / "config.toml"
|
||||
config.write_text(
|
||||
'model_provider = "openai"\n'
|
||||
"profiles = {\n"
|
||||
" work = {\n"
|
||||
' model_provider = "company",\n'
|
||||
" },\n"
|
||||
"}\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
monkeypatch.setattr("headroom.providers.codex.install.codex_config_path", lambda: config)
|
||||
|
||||
apply_provider_scope(_manifest(tmp_path))
|
||||
|
||||
parsed = loads_codex_config(config.read_text(encoding="utf-8"))
|
||||
assert parsed["model_provider"] == "headroom"
|
||||
assert parsed["profiles"]["work"] == {"model_provider": "company"}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("sep", ["
", "\x85"])
|
||||
def test_provider_scope_keeps_single_line_string_with_unicode_separator(
|
||||
tmp_path: Path, monkeypatch, sep: str
|
||||
) -> None:
|
||||
config = tmp_path / "config.toml"
|
||||
notes = f'hello{sep}model_provider = "company"{sep}world'
|
||||
config.write_text(f"notes = '{notes}'\n\n[features]\nhooks = true\n", encoding="utf-8")
|
||||
monkeypatch.setattr("headroom.providers.codex.install.codex_config_path", lambda: config)
|
||||
|
||||
apply_provider_scope(_manifest(tmp_path))
|
||||
|
||||
parsed = loads_codex_config(config.read_text(encoding="utf-8"))
|
||||
assert parsed["notes"] == notes
|
||||
assert parsed["model_provider"] == "headroom"
|
||||
|
||||
|
||||
def test_persistent_install_helper_collision_preserves_config(tmp_path: Path, monkeypatch) -> None:
|
||||
config = tmp_path / "config.toml"
|
||||
auth = tmp_path / "auth.json"
|
||||
|
||||
@@ -228,6 +228,42 @@ def test_register_appends_to_existing_config_preserves_other_keys(tmp_path: Path
|
||||
assert parsed["mcp_servers"]["headroom"]["command"] == _RESOLVED_COMMAND[0]
|
||||
|
||||
|
||||
def test_evict_foreign_tables_ignores_header_text_inside_string() -> None:
|
||||
# TOML allows U+2028 in a single-line string, so "[foreign]" here is part
|
||||
# of the value, not a table header to move out of the span.
|
||||
from headroom.mcp_registry.codex import _MARKER_END, _MARKER_START, _evict_foreign_tables
|
||||
|
||||
content = (
|
||||
f"{_MARKER_START}\n"
|
||||
"[mcp_servers.headroom]\n"
|
||||
'note = "x
[foreign] #"\n'
|
||||
f"{_MARKER_END}\n"
|
||||
'model = "gpt-5"\n'
|
||||
)
|
||||
assert _evict_foreign_tables(content, "headroom", _MARKER_START, _MARKER_END) == content
|
||||
|
||||
|
||||
def test_register_accepts_multiline_inline_table(tmp_path: Path) -> None:
|
||||
# Codex accepts an inline table split across lines (#4378); registering must
|
||||
# not refuse the file as invalid TOML, and the user's text must be kept as is.
|
||||
cfg = _config_path(tmp_path)
|
||||
cfg.parent.mkdir()
|
||||
original = (
|
||||
"[mcp_servers.dingtalk]\n"
|
||||
'command = "npx"\n'
|
||||
"env = {\n"
|
||||
' DINGTALK_Client_ID = "**********",\n'
|
||||
" \n"
|
||||
" }\n"
|
||||
)
|
||||
cfg.write_text(original)
|
||||
result = _make_registrar(tmp_path).register_server(_spec())
|
||||
assert result.status == RegisterStatus.REGISTERED, result.detail
|
||||
text = cfg.read_text()
|
||||
assert text.startswith(original)
|
||||
assert "[mcp_servers.headroom]" in text
|
||||
|
||||
|
||||
def test_register_includes_env_subtable(tmp_path: Path) -> None:
|
||||
spec = _spec(env={"HEADROOM_PROXY_URL": "http://127.0.0.1:9000"})
|
||||
_make_registrar(tmp_path).register_server(spec)
|
||||
|
||||
@@ -0,0 +1,78 @@
|
||||
"""Tests for parsing Codex config.toml with the syntax Codex accepts."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
|
||||
from headroom.providers.codex.config_toml import (
|
||||
loads_codex_config,
|
||||
root_assignment_lines,
|
||||
split_toml_lines,
|
||||
)
|
||||
|
||||
if sys.version_info >= (3, 11):
|
||||
import tomllib
|
||||
else: # pragma: no cover
|
||||
import tomli as tomllib
|
||||
|
||||
# Shape from #4378: an inline table split across lines with a trailing comma.
|
||||
MULTILINE_INLINE_TABLE = (
|
||||
"[mcp_servers.dingtalk]\n"
|
||||
'command = "npx"\n'
|
||||
"env = {\n"
|
||||
' DINGTALK_Client_ID = "**********",\n'
|
||||
" \n"
|
||||
" }\n"
|
||||
)
|
||||
|
||||
|
||||
def test_loads_codex_config_accepts_multiline_inline_table() -> None:
|
||||
assert loads_codex_config(MULTILINE_INLINE_TABLE) == {
|
||||
"mcp_servers": {"dingtalk": {"command": "npx", "env": {"DINGTALK_Client_ID": "**********"}}}
|
||||
}
|
||||
|
||||
|
||||
def test_loads_codex_config_raises_tomllib_error_for_invalid_toml() -> None:
|
||||
with pytest.raises(tomllib.TOMLDecodeError):
|
||||
loads_codex_config("env = {\n")
|
||||
|
||||
|
||||
def test_root_assignment_lines_skip_keys_nested_in_multiline_inline_table() -> None:
|
||||
content = (
|
||||
'model_provider = "openai"\n' # 0: root
|
||||
"profiles = {\n" # 1: root
|
||||
" work = {\n"
|
||||
' model_provider = "company",\n'
|
||||
" },\n"
|
||||
"}\n"
|
||||
'notes = """\n' # 6: root
|
||||
'model_provider = "in a string"\n'
|
||||
'"""\n'
|
||||
"# comment\n"
|
||||
'openai_base_url = "https://example.test/v1" # {\n' # 10: root
|
||||
"\n"
|
||||
"[profiles.other]\n"
|
||||
'model_provider = "other"\n'
|
||||
)
|
||||
assert root_assignment_lines(content) == {0, 1, 6, 10}
|
||||
|
||||
|
||||
# str.splitlines() also breaks at these. TOML only ends a line at LF or CRLF,
|
||||
# and allows U+2028, U+2029 and U+0085 inside a single-line string.
|
||||
SPLITLINES_ONLY_SEPARATORS = ["
", "
", "\x85", "\x0b", "\x0c", "\x1c", "\x1d", "\x1e"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("sep", SPLITLINES_ONLY_SEPARATORS)
|
||||
def test_root_assignment_lines_keep_single_line_string_whole(sep: str) -> None:
|
||||
content = f"notes = 'hello{sep}model_provider = \"company\"{sep}world'\n"
|
||||
assert root_assignment_lines(content) == {0}
|
||||
|
||||
|
||||
def test_split_toml_lines_breaks_only_at_lf_and_crlf() -> None:
|
||||
content = "a = 'x
y'\r\nb = 1\n\x85c = 2"
|
||||
assert split_toml_lines(content, keepends=True) == ["a = 'x
y'\r\n", "b = 1\n", "\x85c = 2"]
|
||||
assert split_toml_lines(content) == ["a = 'x
y'", "b = 1", "\x85c = 2"]
|
||||
assert split_toml_lines("a = 1\n") == ["a = 1"]
|
||||
assert split_toml_lines("") == []
|
||||
@@ -1800,7 +1800,7 @@ requires-dist = [
|
||||
{ name = "starlette", marker = "extra == 'mcp'", specifier = ">=0.27.0" },
|
||||
{ name = "strands-agents", marker = "extra == 'strands'", specifier = ">=0.1.0" },
|
||||
{ name = "tiktoken", specifier = ">=0.5.0" },
|
||||
{ name = "tomlkit", specifier = ">=0.13.0,<1.0" },
|
||||
{ name = "tomlkit", specifier = ">=0.15.0,<1.0" },
|
||||
{ name = "torch", marker = "sys_platform == 'darwin' and extra == 'pytorch-mps'", specifier = ">=2.12.1" },
|
||||
{ name = "torch", marker = "(platform_machine != 'x86_64' and extra == 'ml') or (sys_platform != 'darwin' and extra == 'ml')", specifier = ">=2.12.1" },
|
||||
{ name = "torch", marker = "(platform_machine != 'x86_64' and extra == 'voice') or (sys_platform != 'darwin' and extra == 'voice')", specifier = ">=2.12.1" },
|
||||
|
||||
Reference in New Issue
Block a user