mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(transforms): preserve YAML block scalar data during comment elision (#4498)
## Description
ConfigCompressor removes whole-line YAML comments only when it believes
no block scalar is present. Its guard misses sequence scalars (`- |-`),
headers with trailing comments (`value: | # script`), and
indentation-before-chomping indicators (`|2-`). Consequently scalar
lines beginning with `#` are removed from the delivered configuration as
if they were comments, even though they are YAML string data.
Recognize these block-scalar headers conservatively and retain the
existing behavior of skipping comment elision for the document. LF and
CRLF are covered. No dependency or default configuration changes.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- Extend the existing YAML block-scalar safety guard to sequence items,
tagged scalars, trailing header comments and either order of
indentation/chomping indicators.
- Add21 public-compressor regression cases, including ordinary-comment
eligibility as a control.
## Testing
- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`, configured pre-commit hook
with `--ignore-missing-imports`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
New regression oracle on unpatched main17d38e5:18failed,3passed
python -m pytest tests/test_config_yaml_scalar_elision.py -q:21passed
Supported existing config tests + new regressions:64passed,6deselected
Scoped ruff check: All checks passed!
Scoped ruff format --check:2files already formatted
```
The six excluded tests need the unavailable compiled headroom._core
extension. An unrestricted run of the existing file on both original
main and this change failed those same six tests with
ModuleNotFoundError; original main43passed and
patchedfile+newtests64passed. Full pytest, repository-wide Ruff and Rust
gates are not claimed here. The configured pre-commit mypy whole-package
check passed, as did scoped Ruff0.17.0 and formatting checks.
## Real Behavior Proof
- Environment: macOS arm64, Python3.13.5; actual public ConfigCompressor
with its real in-memory CompressionStore; no model/provider calls or
mocked modules. HEADROOM_OFFLINE=1, HEADROOM_BEACON=off, DO_NOT_TRACK=1,
HEADROOM_CCR_BACKEND=memory, isolated workspace/config directories.
- Exact command / steps: Construct a valid ConfigMap with `scripts:`, a
sequence item `- |-`,12distinct indented `# command note` lines and
`echo ready`. Call `ConfigCompressor().compress(content)`. Compare all
scalar data bytes and compare input/output objects with
`yaml.safe_load`.
```sh
HEADROOM_OFFLINE=1 HEADROOM_BEACON=off DO_NOT_TRACK=1 HEADROOM_CCR_BACKEND=memory python - <<'PYCODE'
import os
import tempfile
import yaml
with tempfile.TemporaryDirectory() as state:
os.environ["HEADROOM_WORKSPACE_DIR"] = state
os.environ["HEADROOM_CONFIG_DIR"] = state + "/config"
from headroom.transforms.config_compressor import ConfigCompressor
text = "apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: script-config\nscripts:\n - |-\n"
text += "".join(f" # command note {i}: preserve this exact scalar value in the delivered configuration\n" for i in range(12))
text += " echo ready\n"
result = ConfigCompressor().compress(text)
print("modified:", result.was_modified, "elided:", result.lines_elided)
print("scalar preserved:", all(f"# command note {i}:" in result.compressed for i in range(12)))
print("byte exact:", result.compressed == text)
assert yaml.safe_load(result.compressed) == yaml.safe_load(text)
PYCODE
```
- Observed result: Before: modified=true, lines_elided=12,
scalar_data_preserved=false. The real CCR store retained the original,
but the delivered representation lost literal scalar data.
After: modified=false, lines_elided=0, scalar_data_preserved=true, YAML
parsed objects equal=true.
- Not tested: provider/model quality, full ContentRouter/native Rust
integration, hosted CI, Windows runtime. CRLF strings are tested
locally.
## Runtime Rollout Safety
- Rollout-managed feature(s): Existing structured-config comment elision
safety check; no new feature.
- Minimum rollout channel: Existing channel.
- Stable/default behavior changed: Block-scalar configurations affected
by the missed syntax pass through comment elision intact.
- Kill switch / disable path: Existing
ConfigCompressorConfig(enable_ccr=False) disables comment elision.
- Unsafe override required: No.
- Qualification impact: No new capability; conservative refusal of an
unsafe compression path.
- Rollback path: Revert this commit.
### Broader check limits and upload exception
Rust1.95.0 `cargo fmt --all -- --check` and `cargo clippy --workspace --
-D warnings` passed on the unchanged native sources. The cold
workspace-test build was stopped twice by a disk-reserve guard; no full
Rust-test or native-dependent Python-suite pass is claimed. Normal
commit hooks passed, including pinned Ruff0.17.0, whole-package mypy and
commitlint. These patches do not change Rust sources or Cargo
dependencies.
The [installed pre-push
hook](https://github.com/headroomlabs-ai/headroom/blob/17d38e5699b7d7c69ac6addcd762f39e42dd7f00/scripts/install-git-hooks.sh)
documents an occasional `git push --no-verify` upload exception. It was
used for this independently reviewed Python-only fix after the bounded
native-check attempts. Required hosted CI and maintainer review remain
pending; the complete pre-push gate has not passed locally.
Prepared and independently reviewed with AI assistance; actual runtime
proof and its limits are recorded above.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] I did **not** edit CHANGELOG.md
- [x] I have added tests that prove my fix is effective
No new dependency, model download, network/provider call or external
credential is required for the reproduction. Source and runtime
self-review are complete; required hosted CI and maintainer review
remain pending.
### Follow-up native validation
The actual Python extension now builds and imports from the unchanged
native source using Rust1.95.0 and the matching Xcode26.4 SDK. At this
exact PR head, the Makefile pre-push Python test-file selection passes
**177 tests with 4 skips**; one existing Anthropic approximation warning
is reported. Whole-tree pinned Ruff0.17.0 passes. The complete YAML
selection also passes all **70 tests**, including the previously
native-blocked cases. The complete Rust workspace test build was again
stopped by the local disk-reserve guard before completion; this is not a
Rust-test pass. The incomplete task-owned build cache was removed after
all build processes exited, retaining the source-built extension and
logs. Full pytest, complete pre-push success and required hosted CI
remain unclaimed.
<!-- security-sync-20261028 -->
### Security baseline sync — 2026-10-28
Current exact head: `0d2e7262831e68b749e1a0caa2e7ab4e2da6cbf6`. Normal
history merge of upstream `3c6f82b06091206379106f8fe1a54f2400ed7a93`
brings in the maintainer-owned dependency remediation
[#4517](https://github.com/headroomlabs-ai/headroom/pull/4517),
following the prior npm baseline sync. The authored fix and its
test/example files are unchanged by this sync; no additional dependency
policy or runtime logic was introduced. This imports the maintainer
remediation requested by review; it does not establish audit success or
approval.
Validation on this head (macOS arm64, Python3.13.5, offline mode/beacon
off/DNT, existing source-built native extension): `python -m pytest
tests/test_config_yaml_scalar_elision.py
tests/test_transforms_config_compressor.py -q` (70 passed); actual
public ConfigCompressor/CompressionStore YAML fixture rerun: byte-exact
scalar content, zero elided lines, parsed YAML equal.
`PRE_COMMIT_HOME=<existing task cache> pre-commit run --all-files`
passed, including pinned Ruff, whole-package mypy, version and
merge-conflict checks. Normal commit hooks and commitlint passed.
Existing native sources/Cargo dependencies are unchanged. The documented
occasional `git push --no-verify` exception was used for a normal
fast-forward upload; full Rust workspace tests, full pytest and complete
pre-push qualification were not rerun or claimed.
Hosted security checks and maintainer review remain separate gates.
Existing human-readiness declarations are preserved; this status update
is evidence of local validation and a baseline sync, not human approval
or a claim that hosted CI is green.
As of `2026-10-28T17:40:32.107305+00:00`, hosted pip-audit and all five
npm audits passed on this exact head ([security
workflow](https://github.com/headroomlabs-ai/headroom/actions/runs/40732282676/job/122971475120));
cargo audit also passed. Other hosted jobs remain in progress, and the
human review state remains `CHANGES_REQUESTED`. This is not complete CI
qualification or human approval.
Signed-off-by: Rudy Celekli <rudy@gradiahq.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
JD Davis
parent
40cd7e7503
commit
534cec0e14
@@ -53,7 +53,12 @@ _COMMENT_RES = {
|
||||
# Content where whole-line elision is unsafe: a '#' line inside a YAML block
|
||||
# scalar or a TOML multi-line string is data, not a comment. Detection is
|
||||
# deliberately over-broad — when in doubt, Tier 2 stays off.
|
||||
_YAML_BLOCK_SCALAR_RE = re.compile(r":\s*[|>][+-]?\d*\s*$", re.MULTILINE)
|
||||
# Block scalars can be sequence items as well as mapping values; indentation
|
||||
# and chomping indicators may appear in either order, and a header can carry
|
||||
# a trailing comment. Conservatively skip elision for every such header.
|
||||
_YAML_BLOCK_SCALAR_RE = re.compile(
|
||||
r"(?:^|[ \t:])[|>][1-9+-]*[ \t]*(?:#[^\r\n]*)?\r?$", re.MULTILINE
|
||||
)
|
||||
_TOML_MULTILINE_RE = re.compile(r'"""|\'\'\'')
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
"""YAML scalar contents are data even when a line starts with a hash."""
|
||||
|
||||
import pytest
|
||||
|
||||
from headroom.transforms.config_compressor import ConfigCompressor
|
||||
|
||||
|
||||
@pytest.mark.parametrize("newline", ["\n", "\r\n"])
|
||||
@pytest.mark.parametrize(
|
||||
"header",
|
||||
[
|
||||
" - |-",
|
||||
" - >+",
|
||||
" - | # literal script",
|
||||
" - >2- # folded script",
|
||||
" value: | # literal script",
|
||||
" value: >- # folded script",
|
||||
" value: |2-",
|
||||
" value: >2+",
|
||||
" value: |-2",
|
||||
" value: !script |",
|
||||
],
|
||||
)
|
||||
def test_yaml_scalar_hash_lines_survive_compression(header: str, newline: str) -> None:
|
||||
content = (
|
||||
"apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: demo\nscripts:\n"
|
||||
+ header
|
||||
+ "\n"
|
||||
+ "".join(
|
||||
f" # note {i}: this is scalar content, not a removable YAML comment\n"
|
||||
for i in range(12)
|
||||
)
|
||||
+ " echo ready\n"
|
||||
)
|
||||
content = content.replace("\n", newline)
|
||||
result = ConfigCompressor().compress(content)
|
||||
assert result.flavor == "yaml"
|
||||
assert result.lines_elided == 0
|
||||
assert result.ccr_hash is None
|
||||
assert result.compressed == content
|
||||
|
||||
|
||||
def test_plain_yaml_comments_remain_eligible_for_elision() -> None:
|
||||
content = "metadata:\n name: demo\n value: plain\n# ordinary comment\n"
|
||||
assert ConfigCompressor._elision_safe(content, "yaml")
|
||||
Reference in New Issue
Block a user