fix(transforms): preserve YAML block scalar data during comment elision (#4498)

## Description

ConfigCompressor removes whole-line YAML comments only when it believes
no block scalar is present. Its guard misses sequence scalars (`- |-`),
headers with trailing comments (`value: | # script`), and
indentation-before-chomping indicators (`|2-`). Consequently scalar
lines beginning with `#` are removed from the delivered configuration as
if they were comments, even though they are YAML string data.

Recognize these block-scalar headers conservatively and retain the
existing behavior of skipping comment elision for the document. LF and
CRLF are covered. No dependency or default configuration changes.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Extend the existing YAML block-scalar safety guard to sequence items,
tagged scalars, trailing header comments and either order of
indentation/chomping indicators.
- Add21 public-compressor regression cases, including ordinary-comment
eligibility as a control.

## Testing

- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`, configured pre-commit hook
with `--ignore-missing-imports`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
New regression oracle on unpatched main17d38e5:18failed,3passed
python -m pytest tests/test_config_yaml_scalar_elision.py -q:21passed
Supported existing config tests + new regressions:64passed,6deselected
Scoped ruff check: All checks passed!
Scoped ruff format --check:2files already formatted
```

The six excluded tests need the unavailable compiled headroom._core
extension. An unrestricted run of the existing file on both original
main and this change failed those same six tests with
ModuleNotFoundError; original main43passed and
patchedfile+newtests64passed. Full pytest, repository-wide Ruff and Rust
gates are not claimed here. The configured pre-commit mypy whole-package
check passed, as did scoped Ruff0.17.0 and formatting checks.

## Real Behavior Proof

- Environment: macOS arm64, Python3.13.5; actual public ConfigCompressor
with its real in-memory CompressionStore; no model/provider calls or
mocked modules. HEADROOM_OFFLINE=1, HEADROOM_BEACON=off, DO_NOT_TRACK=1,
HEADROOM_CCR_BACKEND=memory, isolated workspace/config directories.
- Exact command / steps: Construct a valid ConfigMap with `scripts:`, a
sequence item `- |-`,12distinct indented `# command note` lines and
`echo ready`. Call `ConfigCompressor().compress(content)`. Compare all
scalar data bytes and compare input/output objects with
`yaml.safe_load`.

```sh
HEADROOM_OFFLINE=1 HEADROOM_BEACON=off DO_NOT_TRACK=1 HEADROOM_CCR_BACKEND=memory python - <<'PYCODE'
import os
import tempfile
import yaml
with tempfile.TemporaryDirectory() as state:
    os.environ["HEADROOM_WORKSPACE_DIR"] = state
    os.environ["HEADROOM_CONFIG_DIR"] = state + "/config"
    from headroom.transforms.config_compressor import ConfigCompressor
    text = "apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: script-config\nscripts:\n  - |-\n"
    text += "".join(f"    # command note {i}: preserve this exact scalar value in the delivered configuration\n" for i in range(12))
    text += "    echo ready\n"
    result = ConfigCompressor().compress(text)
    print("modified:", result.was_modified, "elided:", result.lines_elided)
    print("scalar preserved:", all(f"# command note {i}:" in result.compressed for i in range(12)))
    print("byte exact:", result.compressed == text)
    assert yaml.safe_load(result.compressed) == yaml.safe_load(text)
PYCODE
```

- Observed result: Before: modified=true, lines_elided=12,
scalar_data_preserved=false. The real CCR store retained the original,
but the delivered representation lost literal scalar data.
After: modified=false, lines_elided=0, scalar_data_preserved=true, YAML
parsed objects equal=true.
- Not tested: provider/model quality, full ContentRouter/native Rust
integration, hosted CI, Windows runtime. CRLF strings are tested
locally.

## Runtime Rollout Safety

- Rollout-managed feature(s): Existing structured-config comment elision
safety check; no new feature.
- Minimum rollout channel: Existing channel.
- Stable/default behavior changed: Block-scalar configurations affected
by the missed syntax pass through comment elision intact.
- Kill switch / disable path: Existing
ConfigCompressorConfig(enable_ccr=False) disables comment elision.
- Unsafe override required: No.
- Qualification impact: No new capability; conservative refusal of an
unsafe compression path.
- Rollback path: Revert this commit.


### Broader check limits and upload exception

Rust1.95.0 `cargo fmt --all -- --check` and `cargo clippy --workspace --
-D warnings` passed on the unchanged native sources. The cold
workspace-test build was stopped twice by a disk-reserve guard; no full
Rust-test or native-dependent Python-suite pass is claimed. Normal
commit hooks passed, including pinned Ruff0.17.0, whole-package mypy and
commitlint. These patches do not change Rust sources or Cargo
dependencies.

The [installed pre-push
hook](https://github.com/headroomlabs-ai/headroom/blob/17d38e5699b7d7c69ac6addcd762f39e42dd7f00/scripts/install-git-hooks.sh)
documents an occasional `git push --no-verify` upload exception. It was
used for this independently reviewed Python-only fix after the bounded
native-check attempts. Required hosted CI and maintainer review remain
pending; the complete pre-push gate has not passed locally.

Prepared and independently reviewed with AI assistance; actual runtime
proof and its limits are recorded above.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] I did **not** edit CHANGELOG.md
- [x] I have added tests that prove my fix is effective

No new dependency, model download, network/provider call or external
credential is required for the reproduction. Source and runtime
self-review are complete; required hosted CI and maintainer review
remain pending.


### Follow-up native validation

The actual Python extension now builds and imports from the unchanged
native source using Rust1.95.0 and the matching Xcode26.4 SDK. At this
exact PR head, the Makefile pre-push Python test-file selection passes
**177 tests with 4 skips**; one existing Anthropic approximation warning
is reported. Whole-tree pinned Ruff0.17.0 passes. The complete YAML
selection also passes all **70 tests**, including the previously
native-blocked cases. The complete Rust workspace test build was again
stopped by the local disk-reserve guard before completion; this is not a
Rust-test pass. The incomplete task-owned build cache was removed after
all build processes exited, retaining the source-built extension and
logs. Full pytest, complete pre-push success and required hosted CI
remain unclaimed.


<!-- security-sync-20261028 -->
### Security baseline sync — 2026-10-28

Current exact head: `0d2e7262831e68b749e1a0caa2e7ab4e2da6cbf6`. Normal
history merge of upstream `3c6f82b06091206379106f8fe1a54f2400ed7a93`
brings in the maintainer-owned dependency remediation
[#4517](https://github.com/headroomlabs-ai/headroom/pull/4517),
following the prior npm baseline sync. The authored fix and its
test/example files are unchanged by this sync; no additional dependency
policy or runtime logic was introduced. This imports the maintainer
remediation requested by review; it does not establish audit success or
approval.

Validation on this head (macOS arm64, Python3.13.5, offline mode/beacon
off/DNT, existing source-built native extension): `python -m pytest
tests/test_config_yaml_scalar_elision.py
tests/test_transforms_config_compressor.py -q` (70 passed); actual
public ConfigCompressor/CompressionStore YAML fixture rerun: byte-exact
scalar content, zero elided lines, parsed YAML equal.

`PRE_COMMIT_HOME=<existing task cache> pre-commit run --all-files`
passed, including pinned Ruff, whole-package mypy, version and
merge-conflict checks. Normal commit hooks and commitlint passed.
Existing native sources/Cargo dependencies are unchanged. The documented
occasional `git push --no-verify` exception was used for a normal
fast-forward upload; full Rust workspace tests, full pytest and complete
pre-push qualification were not rerun or claimed.

Hosted security checks and maintainer review remain separate gates.
Existing human-readiness declarations are preserved; this status update
is evidence of local validation and a baseline sync, not human approval
or a claim that hosted CI is green.

As of `2026-10-28T17:40:32.107305+00:00`, hosted pip-audit and all five
npm audits passed on this exact head ([security
workflow](https://github.com/headroomlabs-ai/headroom/actions/runs/40732282676/job/122971475120));
cargo audit also passed. Other hosted jobs remain in progress, and the
human review state remains `CHANGES_REQUESTED`. This is not complete CI
qualification or human approval.

Signed-off-by: Rudy Celekli <rudy@gradiahq.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
Rudy Mizrahi Celekli
2026-10-29 20:11:58 -05:00
committed by GitHub
co-authored by JD Davis
parent 40cd7e7503
commit 534cec0e14
2 changed files with 51 additions and 1 deletions
+6 -1
View File
@@ -53,7 +53,12 @@ _COMMENT_RES = {
# Content where whole-line elision is unsafe: a '#' line inside a YAML block
# scalar or a TOML multi-line string is data, not a comment. Detection is
# deliberately over-broad — when in doubt, Tier 2 stays off.
_YAML_BLOCK_SCALAR_RE = re.compile(r":\s*[|>][+-]?\d*\s*$", re.MULTILINE)
# Block scalars can be sequence items as well as mapping values; indentation
# and chomping indicators may appear in either order, and a header can carry
# a trailing comment. Conservatively skip elision for every such header.
_YAML_BLOCK_SCALAR_RE = re.compile(
r"(?:^|[ \t:])[|>][1-9+-]*[ \t]*(?:#[^\r\n]*)?\r?$", re.MULTILINE
)
_TOML_MULTILINE_RE = re.compile(r'"""|\'\'\'')
+45
View File
@@ -0,0 +1,45 @@
"""YAML scalar contents are data even when a line starts with a hash."""
import pytest
from headroom.transforms.config_compressor import ConfigCompressor
@pytest.mark.parametrize("newline", ["\n", "\r\n"])
@pytest.mark.parametrize(
"header",
[
" - |-",
" - >+",
" - | # literal script",
" - >2- # folded script",
" value: | # literal script",
" value: >- # folded script",
" value: |2-",
" value: >2+",
" value: |-2",
" value: !script |",
],
)
def test_yaml_scalar_hash_lines_survive_compression(header: str, newline: str) -> None:
content = (
"apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: demo\nscripts:\n"
+ header
+ "\n"
+ "".join(
f" # note {i}: this is scalar content, not a removable YAML comment\n"
for i in range(12)
)
+ " echo ready\n"
)
content = content.replace("\n", newline)
result = ConfigCompressor().compress(content)
assert result.flavor == "yaml"
assert result.lines_elided == 0
assert result.ccr_hash is None
assert result.compressed == content
def test_plain_yaml_comments_remain_eligible_for_elision() -> None:
content = "metadata:\n name: demo\n value: plain\n# ordinary comment\n"
assert ConfigCompressor._elision_safe(content, "yaml")