fix(content_detector): detect CMTrace (SCCM/Intune) logs as BUILD_OUTPUT (#3712)

## Description

CMTrace-format logs — what SCCM, MDT and Intune Win32 app/script
deployments
write on Windows — are never detected as `BUILD_OUTPUT`. Two independent
causes,
and the second shadows a fix to the first, so both are fixed together.

Measured on a corpus of 253 real Intune/SCCM deployment logs,
`detect_content_type`
returned `BUILD_OUTPUT` for **zero** of them.

Closes #<!-- issue number if you filed one; otherwise delete this line
-->

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring

## Changes Made

- **Both implementations** — added `^<!\[LOG\[` to `_LOG_PATTERNS` /
  `LOG_PATTERNS`. CMTrace puts its timestamp in an attribute *after* the
  message, so the anchored `^\s*\d{4}-\d{2}-\d{2}`,
`^\s*\[\d{2}:\d{2}:\d{2}\]` and `^={3,}` patterns cannot fire, and a
record
need not contain ERROR/WARN/INFO. `_try_detect_log` returned `None` at
its
  `pattern_matches == 0` guard.
- **Rust only** — ported Python's `_prefix_looks_like_path` as
`prefix_looks_like_path` and applied it in `try_detect_search`. A
CMTrace
  divider contains no whitespace, so `SEARCH_RESULT_PATTERN` runs from
`<![LOG[` through `]LOG]!><time="12`, then `:`, then `00`, then `:` — a
clean
  `file:line:` match. Python rejects it via `_is_search_result_line`'s
path-shape guard (no `<`, `>`, `=`); Rust applied the regex bare. Since
search is dispatched before logs, Rust would otherwise shadow the fix
above.
  Porting the guard, rather than tightening the Rust regex, keeps
`SEARCH_RESULT_PATTERN` byte-identical across the two and converges the
  logic on Python's.
- Tests on both sides: CMTrace multi-line → `BUILD_OUTPUT`, CMTrace
single-line
→ `BUILD_OUTPUT`, the divider is rejected as a search line, plus
regression
  guards that ordinary and extensionless grep output still detect as
`SEARCH_RESULTS`. The Rust suite additionally pins
`prefix_looks_like_path`
  against Python's rules directly.

### Note on the parity harness

The Rust/Python divergence above was **not** caught by
`parity-run --only content_detector`, which reports `matched=21
diffed=0` both
before and after this change. None of the 21 recorded fixtures exercise
a
`<`-prefixed `file:line:` shape, so the drift was invisible to it. Worth
knowing
if the harness is being relied on as the parity guarantee.

### Two further Rust divergences, reported not fixed

Found while porting the guard; left alone to keep this PR to one logical
change.
Happy to open a separate issue or PR:

- Rust has no equivalent of `_GREP_COLON_DASH_PATTERN` or
`_is_grep_context_line` — the `grep -A/-B/-C` context-line handling
added in
  #3580.
- Rust returns on `matching_lines == 0`; Python uses an absolute floor
of
`matching_lines < 2`, added deliberately because the search compressor
drops
  non-matching lines, so a single coincidental match is data loss.

### Why a second, single-line test

Real Intune logs are emitted as one continuous stream with **no line
separators** — a 133 KB production sample held 571 records and 2 line
feeds.
That form never reaches `_try_detect_search` (it exits at
`matching_lines < 2`),
so it fails on Bug 1 alone and lands in `PLAIN_TEXT`. The multi-line
form is
what exercises Bug 2. Both shapes occur, so both are tested.

## Testing

- [x] Unit tests pass (`pytest`) — 12,680 passed; see note on
pre-existing Windows failures below
- [x] Linting passes (`ruff check .`) — `All checks passed!`
- [x] Type checking passes (`mypy headroom`) — 1 pre-existing error in
an unrelated file, none in the files this PR touches
- [x] New tests added for new functionality — 5 Python, 5 Rust
- [ ] Manual testing performed

### Test Output

Python, on 3.13.15 / pytest 9.0.3 — **5 passed:**

```text
tests/test_transforms/test_content_detector_cmtrace.py::test_multiline_cmtrace_detected_as_build_output PASSED   [ 20%]
tests/test_transforms/test_content_detector_cmtrace.py::test_single_line_cmtrace_detected_as_build_output PASSED [ 40%]
tests/test_transforms/test_content_detector_cmtrace.py::test_cmtrace_divider_is_not_treated_as_a_search_result_line PASSED [ 60%]
tests/test_transforms/test_content_detector_cmtrace.py::test_grep_output_still_detected_as_search_results PASSED [ 80%]
tests/test_transforms/test_content_detector_cmtrace.py::test_extensionless_grep_paths_still_detected PASSED      [100%]

================================================== 5 passed in 0.71s ==================================================
```

Rust, `cargo test -p headroom-core content_detector` — **27 passed** (21
pre-existing + 6 new). The 21 pre-existing `content_detector` tests all
still
pass, including `search_results_detected` and `build_output_detected`,
which is
the direct evidence that adding the path-shape guard did not disturb
existing
detection:

```text
running 27 tests
test transforms::content_detector::tests::cmtrace_divider_is_rejected_by_the_path_shape_guard ... ok
test transforms::content_detector::tests::path_shape_guard_matches_python_rules ... ok
test transforms::content_detector::tests::multiline_cmtrace_detects_as_build_output ... ok
test transforms::content_detector::tests::single_line_cmtrace_detects_as_build_output ... ok
test transforms::content_detector::tests::grep_output_still_detects_as_search_results ... ok
test transforms::content_detector::tests::extensionless_grep_paths_still_detected ... ok
test transforms::content_detector::tests::search_results_detected ... ok
test transforms::content_detector::tests::build_output_detected ... ok
[...19 further pre-existing content_detector tests, all ok...]

test result: ok. 27 passed; 0 failed; 0 ignored; 0 measured; 915 filtered out; finished in 0.07s
```

Rust↔Python parity harness over all 21 recorded `content_detector`
fixtures —
**already run, zero divergence**:

```text
$ cargo run -p headroom-parity --bin parity-run -- run --only content_detector
[content_detector] total=21 matched=21 skipped=0 diffed=0
```

Full Python suite — **already run**:

```text
45 failed, 12680 passed, 603 skipped, 6141 warnings, 81 errors in 2743.50s (0:45:43)
```

**The 126 failures/errors are pre-existing and platform-specific, none
related
to this change.** Evidence:

- They span 26 distinct test files. **Not one of them references
`content_detector` or `detect_content_type`** (checked by grep across
every
  failing file).
- 95 of 126 are `PermissionError: [WinError 32] The process cannot
access the
file because it is being used by another process` — Windows SQLite file
  locking, concentrated in `test_sqlite_graph_store.py` (37),
  `test_sqlite_vector_index.py` (18) and `test_memory/` (36).
- The remainder are Windows-platform assumptions: `module 'signal' has
no
attribute 'SIGHUP'`, `WindowsPath` vs POSIX path comparisons, CRLF-vs-LF
in
  `test_owned_asset_encoding.py`, and two missing external binaries
  (`ast-grep`, `difftastic`).

Per CONTRIBUTING ("Test/CI-only PR chasing a known `main` failure —
Don't.
We're tracking it"), these are left alone. Happy to confirm against a
clean
`main` run on request.

Lint — **already run, clean**:

```text
$ py -m uv run ruff check .
All checks passed!
```

Type check — **already run**:

```text
$ py -m uv run mypy headroom
headroom\integrations\langchain\chat_model.py:394: error: Function "RunnableBinding" could always be true in boolean context  [truthy-function]
Found 1 error in 1 file (checked 538 source files)
```

**That error is pre-existing and unrelated.**
`headroom/integrations/langchain/chat_model.py` is not touched by this
PR and contains no reference to
`content_detector` or `detect_content_type`. The two files this PR
changes
produce no mypy errors.

## Real Behavior Proof

- Environment: Windows 11 Enterprise 10.0.26200; Python 3.13.15 with
dependencies resolved from `uv.lock` (`py -m uv sync --extra dev`, uv
0.12.17); cargo/rustc 1.95.0; branch `fix/cmtrace-content-detection`
based on `94206e26` (release 0.38.0). Full detail under *Environment*
below.
- Exact command / steps: `detect_content_type` was run over a 40-record
synthetic CMTrace log three times — with the patch present, with `git
checkout HEAD~1 -- headroom/transforms/content_detector.py` reverting
only the patched file, and with it restored — each run printing whether
the fix was present in the module it actually imported. Full command
list under *Exact command / steps* below.
- Observed result: on identical input, `ContentType.BUILD_OUTPUT` with
the fix present and `ContentType.PLAIN_TEXT` with it reverted, toggled
in both directions and back; classification tracks the file state
exactly. Transcript under *Observed result* below.
- Not tested: end-to-end through the CLI or proxy (detection was
exercised directly, `Manual testing performed` is unchecked above); the
Rust half has unit and parity coverage but no before/after toggle of a
built binary; UTF-16 or BOM-prefixed CMTrace input; non-Windows CMTrace
producers; the 253-log corpus was measured before the patch and not
re-measured after. Detail under *Not tested* below.

### Environment

- Windows 11 Enterprise 10.0.26200
- Python 3.13.15, from `C:\Program Files\Python313`
- uv 0.12.17 (x86_64-pc-windows-msvc), installed via `py -m pip install
uv`
- Dependencies from `uv.lock` via `py -m uv sync --extra dev`
- cargo 1.95.0 / rustc 1.95.0
- Branch `fix/cmtrace-content-detection`, based on `94206e26` (release
0.38.0)

### Exact command / steps

```bash
# environment
py -m pip install uv
py -m uv sync --extra dev

# confirm the suite imports the working tree, not a built copy
py -m uv run python -c "import headroom.transforms.content_detector as m; print(m.__file__)"
#   -> <repo>\headroom\transforms\content_detector.py

# tests
py -m uv run pytest tests/test_transforms/test_content_detector_cmtrace.py -v
py -m uv run pytest
py -m uv run ruff check .
py -m uv run mypy headroom
cargo test -p headroom-core content_detector
cargo run -p headroom-parity --bin parity-run -- run --only content_detector

# before/after on the same input, reverting only the patched file
py -m uv run python -c "<detect command, below>"
git checkout HEAD~1 -- headroom/transforms/content_detector.py
py -m uv run python -c "<same command>"
git checkout HEAD -- headroom/transforms/content_detector.py
py -m uv run python -c "<same command>"
```

The `git checkout` toggle only proves anything if the import resolves to
the
working tree rather than an installed copy, which is why the detect
command
below prints the provenance of the module it imported alongside the
result.

### Observed result

Same input, same command, toggling only the patched file, on a synthetic
40-record CMTrace log containing no real data:

```text
$ py -m uv run python -c "import headroom.transforms.content_detector as m; src=open(m.__file__, encoding='utf-8').read(); print('fix_present =', 'CMTrace record opener' in src); print('result      =', m.detect_content_type(open(r'...\cmtrace-sample.log').read()).content_type)"
fix_present = True
result      = ContentType.BUILD_OUTPUT

$ git checkout HEAD~1 -- headroom/transforms/content_detector.py
$ py -m uv run python -c "...same command..."
fix_present = False
result      = ContentType.PLAIN_TEXT

$ git checkout HEAD -- headroom/transforms/content_detector.py
$ py -m uv run python -c "...same command..."
fix_present = True
result      = ContentType.BUILD_OUTPUT
```

Toggled in both directions and back, and the classification tracks the
file
state exactly.

Note the "before" value is `PLAIN_TEXT`, not `SEARCH_RESULTS`: Python's
existing
path-shape guard already keeps this content off the search path, so it
was
falling through to prose rather than being misrouted. That is the whole
of the
Python-side bug — the log detector simply never fired. The Rust port,
lacking
that guard, *would* have returned `SEARCH_RESULTS` here, which is why
this PR
ports it.

### Supporting analysis, gathered before the patch

Impact analysis against all 21 recorded parity fixtures in
`tests/parity/fixtures/content_detector/`. Each fixture input was
extracted and
run against the search pattern and against the new CMTrace pattern. **No
fixture
changes classification** — the only `search` fixture matches 4 of 4
lines either
way:

```
$ printf '%b\n' "$(...79e6b68a05a1c639.json input...)"
src/main.py:42:def process():
src/util.py:13:    return None
lib/x.py:7:class X:
tests/test_a.py:3:    assert True
matches: 4 of 4
```

Why the bare `file:line:` shape fires on this format at all — on the
same
synthetic log (40 records, 20 dividers):

```
$ grep -cP '^[^\s:]+:\d+:' cmtrace-sample.log
20
$ grep -oP '^[^\s:]+:\d+:' cmtrace-sample.log | head -1
<![LOG[===========================================================================]LOG]!><time="10:00:
```

20 of 40 lines matching gives ratio 0.50 and confidence `0.4 + 0.50*0.6
= 0.70`,
over the 0.6 cutoff. Python discards those matches at the path-shape
guard, so
it lands in `PLAIN_TEXT`; Rust, which had no such guard, is where that
0.70
would have been returned as `SEARCH_RESULTS` — the shadowing this PR
removes.

None of `_LOG_PATTERNS`' existing signals match that same file, each
checked
individually:

```
^\s*\d{4}-\d{2}-\d{2}                     0 matches
^\s*\[\d{2}:\d{2}:\d{2}\]                 0 matches
^={3,}|^-{3,}                             0 matches
\b(ERROR|FAIL|FAILED|FATAL|CRITICAL)\b    0 matches
\b(WARN|WARNING)\b                        0 matches
\b(INFO|DEBUG|TRACE)\b                    0 matches
```

**Downstream real-world measurement:** a tool built on `headroom-core`
processed
253 real Intune/SCCM deployment logs. `LogCompressor` was dispatched **0
times**
across the whole corpus; every log went to the prose path.

### Not tested

- End-to-end through the CLI or proxy. Detection was exercised directly
via
`detect_content_type`; `Manual testing performed` is left unchecked
above.
- A before/after toggle of a built Rust binary. The Rust half is covered
by its
  6 new unit tests and by `parity-run --only content_detector`.
- UTF-16 or BOM-prefixed CMTrace input — out of scope here, detection
operates
  on already-decoded text.
- Non-Windows CMTrace producers.
- Re-measurement of the 253-log corpus after the patch; that figure is a
  pre-patch measurement.

## Runtime Rollout Safety

- Rollout-managed feature(s): none — pure detection logic in
`content_detector`, behind no flag, channel or rollout gate.
- Minimum rollout channel: N/A — not rollout-managed; it ships with the
ordinary release.
- Stable/default behavior changed: yes, narrowly — content beginning
`<![LOG[` now classifies as `BUILD_OUTPUT` instead of `PLAIN_TEXT`
(Python) or `SEARCH_RESULTS` (Rust), and in Rust a `file:line:` prefix
containing `<`, `>` or `=` no longer counts as a search line, which is
what Python has always done. All 21 recorded parity fixtures are
unchanged: `matched=21 diffed=0`.
- Kill switch / disable path: none — there is no runtime toggle;
disabling means reverting the commit.
- Unsafe override required: no.
- Qualification impact: `BUILD_OUTPUT` routes to `LogCompressor` rather
than the prose path, so measured compression ratio on CMTrace input
changes — see Additional Notes, which quantifies it and explains why it
is still the correct classification.
- Rollback path: single-commit revert; no migration, persisted state or
cache format is involved.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — N/A, no
user-facing docs cover detector internals
- [x] My changes generate no new warnings — `ruff check .` clean; `cargo
test` compiled without warnings on the touched crate
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally — 5 Python + 27 Rust;
parity `matched=21 diffed=0`
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

**One thing worth flagging to maintainers, because it changed our own
conclusion.** Correcting detection is necessary but, for this format,
not
sufficient.

We first patched this downstream by routing CMTrace to `BUILD_OUTPUT`.
On
measuring it, we reverted that change: `LogCompressor` returns this
content
**unchanged**. Real CMTrace logs have no line separators, so a
line-oriented
compressor sees one enormous line and has nothing to deduplicate.

| Route | Time | 266 KB CMTrace sample |
|---|---|---|
| `BUILD_OUTPUT` → `LogCompressor` | 0.10 s | 133 KB → 133 KB (**0%**) |
| prose/embedding path | 17.9 s | 266 KB → 9.5 KB (**96%**) |

4 of 5 sampled files saved exactly zero bytes on the log path; across
152 real
CMTrace files the prose path averages 71%.

This PR fixes detection only, because that is the actual bug and it is
correct
on its own terms — CMTrace *is* build output. But `LogCompressor` would
need
record-boundary awareness (splitting on `]LOG]!>` rather than `\n`)
before these
files genuinely benefit. Happy to open a separate issue or PR for that
if
maintainers want it; flagging it here so this change doesn't land and
look like
a regression on compression ratio.

**Not a performance report.** On our corpus these files average 2.7 s;
the slow
files are simply the large ones, regardless of format.

Co-authored-by: PRokicki <prokicki@hemmersbach.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
tiduseQ
2026-09-24 11:12:26 -05:00
committed by GitHub
co-authored by PRokicki JD Davis
parent 195910b218
commit 6feb1fb22a
3 changed files with 261 additions and 1 deletions
@@ -86,6 +86,23 @@ impl DetectionResult {
static SEARCH_RESULT_PATTERN: LazyLock<Regex> =
LazyLock::new(|| Regex::new(r"^[^\s:]+:\d+:").unwrap());
/// Path-shape guard for a matched `file:line:` prefix.
///
/// Ports Python's `_prefix_looks_like_path` (`content_detector.py`), which
/// this implementation was missing. `SEARCH_RESULT_PATTERN` alone also matches
/// ISO-8601 timestamps and XML-ish wrappers, because any run of non-whitespace
/// containing `:<digits>:` satisfies it — including a CMTrace (SCCM/Intune)
/// log record, whose `]LOG]!><time="HH:MM:` follows the message with no
/// intervening space. Misrouting there is not cosmetic: the search compressor
/// keeps only matching lines and drops the rest, so a false positive is data
/// loss.
///
/// Rules out markup tags and `key=value:12:` log prefixes while leaving
/// extensionless paths (`Makefile:12:`, `Dockerfile:3:`) alone.
fn prefix_looks_like_path(prefix: &str) -> bool {
!prefix.contains('<') && !prefix.contains('>') && !prefix.contains('=')
}
/// Diff-header detection. Recognizes:
/// - `git diff` (`diff --git`, `--- a/`)
/// - merge-commit headers (`diff --combined`, `diff --cc`)
@@ -185,6 +202,13 @@ static LOG_PATTERNS: LazyLock<Vec<Regex>> = LazyLock::new(|| {
Regex::new(r"^npm ERR!|^yarn error|^cargo error").unwrap(),
Regex::new(r"Traceback \(most recent call last\)").unwrap(),
Regex::new(r"^\s*at\s+[\w.$]+\(").unwrap(),
// CMTrace record opener — the log format SCCM, MDT and Intune Win32
// app/script deployments write on Windows. Every record is
// `<![LOG[message]LOG]!><time="..." date="..." ...>`, which matches
// none of the patterns above: the timestamp sits in an attribute
// *after* the message, so the anchored date/time/separator patterns
// cannot fire, and a record need not contain ERROR/WARN/INFO.
Regex::new(r"^<!\[LOG\[").unwrap(),
]
});
@@ -389,7 +413,10 @@ fn try_detect_search(content: &str) -> Option<DetectionResult> {
}
let mut matching_lines: u32 = 0;
for line in &lines {
if !line.trim().is_empty() && SEARCH_RESULT_PATTERN.is_match(line) {
if !line.trim().is_empty()
&& SEARCH_RESULT_PATTERN.is_match(line)
&& prefix_looks_like_path(line.split(':').next().unwrap_or(""))
{
matching_lines += 1;
}
}
@@ -766,4 +793,90 @@ func helper() {}
assert_eq!(ContentType::Html.as_str(), "html");
assert_eq!(ContentType::PlainText.as_str(), "text");
}
/// One CMTrace record: `<![LOG[msg]LOG]!><time="..." date="..." ...>`.
fn cmtrace_record(message: &str, sec: usize) -> String {
format!(
"<![LOG[{message}]LOG]!><time=\"10:00:{sec:02}.000-0\" date=\"01-15-2026\" \
component=\"ExampleCorp_SampleApp_1.0.0_x64_B1\" context=\"SYSTEM\" type=\"1\" \
thread=\"4242\" file=\"install.ps1\">"
)
}
/// Dividers alternating with messages — the dividers are what used to make
/// this content look like grep output.
fn cmtrace_records() -> Vec<String> {
let messages = [
"Starting deployment of SampleApp 1.0.0",
"Detection rule evaluated, not installed",
"Downloading package from content source",
"Extracting files to target directory",
"Registering application components",
"Starting service SampleAppSvc",
"Writing uninstall registry entries",
"Deployment completed with exit code 0",
];
let mut records = Vec::new();
for (i, message) in messages.iter().enumerate() {
records.push(cmtrace_record(&"=".repeat(75), i * 2));
records.push(cmtrace_record(message, i * 2 + 1));
}
records
}
#[test]
fn multiline_cmtrace_detects_as_build_output() {
let r = detect_content_type(&cmtrace_records().join("\n"));
assert_eq!(r.content_type, ContentType::BuildOutput);
assert!(r.confidence >= 0.5);
}
#[test]
fn single_line_cmtrace_detects_as_build_output() {
// How Intune actually writes them: records run together with no
// separators. This form never reaches the search detector, so it failed
// purely on the missing LOG_PATTERNS signal.
let r = detect_content_type(&cmtrace_records().concat());
assert_eq!(r.content_type, ContentType::BuildOutput);
}
#[test]
fn cmtrace_divider_is_rejected_by_the_path_shape_guard() {
// The bare pattern *does* match a CMTrace divider - that is the whole
// problem. What rejects it is `prefix_looks_like_path`, ported here
// from Python, which this implementation previously lacked.
let divider = cmtrace_record(&"=".repeat(75), 0);
assert!(SEARCH_RESULT_PATTERN.is_match(&divider));
assert!(!prefix_looks_like_path(
divider.split(':').next().unwrap_or("")
));
}
#[test]
fn path_shape_guard_matches_python_rules() {
// Mirrors Python's `_prefix_looks_like_path`: no `<`, `>` or `=`.
assert!(prefix_looks_like_path("src/main.py"));
assert!(prefix_looks_like_path("Makefile"));
assert!(!prefix_looks_like_path("<current_datetime"));
assert!(!prefix_looks_like_path("key=value"));
}
#[test]
fn grep_output_still_detects_as_search_results() {
let content = "src/main.py:42:def process():\n\
src/util.py:13: return None\n\
lib/x.py:7:class X:\n\
tests/test_a.py:3: assert True";
let r = detect_content_type(content);
assert_eq!(r.content_type, ContentType::SearchResults);
}
#[test]
fn extensionless_grep_paths_still_detected() {
let content = "Makefile:12:\tpytest -q\n\
Dockerfile:3:RUN apt-get update\n\
Jenkinsfile:88: sh 'make test'";
let r = detect_content_type(content);
assert_eq!(r.content_type, ContentType::SearchResults);
}
}
+7
View File
@@ -169,6 +169,13 @@ _LOG_PATTERNS = [
re.compile(r"^\s*at .+\) in .+:line \d+"), # .NET frame with PDB info
re.compile(r"^Caused by: "), # Java exception chain head
re.compile(r"^\s*\.\.\. \d+ more$"), # Java elided-frames summary
# CMTrace record opener -- the log format SCCM, MDT and Intune Win32
# app/script deployments write on Windows. Every record is
# `<![LOG[message]LOG]!><time="..." date="..." ...>`, which matches none of
# the patterns above: the timestamp sits in an attribute *after* the
# message, so the anchored date/time/separator patterns cannot fire, and a
# record need not contain ERROR/WARN/INFO.
re.compile(r"^<!\[LOG\["), # CMTrace (SCCM/Intune) log record
]
@@ -0,0 +1,140 @@
"""CMTrace (SCCM/Intune) deployment logs must detect as ``BUILD_OUTPUT``.
``_LOG_PATTERNS`` had no CMTrace signal. The format puts its timestamp in an
attribute *after* the message, so every anchored date/time/separator pattern
misses, and a record need not contain ERROR/WARN/INFO -- so ``_try_detect_log``
returned ``None`` at its ``pattern_matches == 0`` guard and the content fell
through to ``PLAIN_TEXT``.
A CMTrace record also satisfies the bare ``_SEARCH_RESULT_PATTERN``
(``^[^\\s:]+:\\d+:``), because ``]LOG]!><time="HH:MM:`` follows the message with
no intervening whitespace. Python is already protected from that by
``_is_search_result_line``'s path-shape guard, which rejects angle brackets --
the test below pins that behaviour so the protection cannot be lost. The Rust
port lacked the equivalent guard; this change adds it there.
"""
from __future__ import annotations
from headroom.transforms.content_detector import (
ContentType,
_is_search_result_line,
detect_content_type,
)
_ATTRS = (
'time="10:00:{sec:02d}.000-0" date="01-15-2026" '
'component="ExampleCorp_SampleApp_1.0.0_x64_B1" context="SYSTEM" '
'type="1" thread="4242" file="install.ps1"'
)
_MESSAGES = (
"Starting deployment of SampleApp 1.0.0",
"Detection rule evaluated, not installed",
"Downloading package from content source",
"Extracting files to target directory",
"Registering application components",
"Starting service SampleAppSvc",
"Writing uninstall registry entries",
"Deployment completed with exit code 0",
)
def _record(message: str, sec: int) -> str:
"""Build one CMTrace record.
Args:
message: Text inside the ``<![LOG[...]LOG]!>`` wrapper.
sec: Seconds value for the record's ``time`` attribute.
Returns:
A single CMTrace record, with no trailing newline.
"""
return f"<![LOG[{message}]LOG]!><{_ATTRS.format(sec=sec)}>"
def _cmtrace_records() -> list[str]:
"""Alternate ``===`` divider records with message records.
Dividers are what made this content look like grep output, so they are the
load-bearing half of the fixture, not decoration.
Returns:
Records in emission order, dividers first.
"""
records: list[str] = []
for i, message in enumerate(_MESSAGES):
records.append(_record("=" * 75, i * 2))
records.append(_record(message, i * 2 + 1))
return records
def test_multiline_cmtrace_detected_as_build_output() -> None:
"""Newline-separated CMTrace detects as a log, not as grep output."""
content = "\n".join(_cmtrace_records())
result = detect_content_type(content)
assert result.content_type is ContentType.BUILD_OUTPUT
assert result.confidence >= 0.5
def test_single_line_cmtrace_detected_as_build_output() -> None:
"""Records run together with no separators -- how Intune actually writes them.
Real deployment logs are emitted as one continuous stream: a 133 KB sample
taken from a production estate held 571 records and 2 line feeds. This form
never reached ``_try_detect_search`` (it exits at ``matching_lines < 2``),
so it failed purely on defect 1 and fell through to ``PLAIN_TEXT``.
"""
content = "".join(_cmtrace_records())
result = detect_content_type(content)
assert result.content_type is ContentType.BUILD_OUTPUT
def test_cmtrace_divider_is_not_treated_as_a_search_result_line() -> None:
"""Pin the path-shape guard that keeps CMTrace off the search path.
The bare ``_SEARCH_RESULT_PATTERN`` *does* match a CMTrace divider; what
rejects it is ``_prefix_looks_like_path``'s angle-bracket rule, one level
up. Search is dispatched before logs, so losing that guard would shadow the
``_LOG_PATTERNS`` fix and route these logs to the search compressor -- which
keeps only matching lines and drops the rest, making it data loss rather
than a mis-labelling.
"""
divider = _record("=" * 75, 0)
assert _is_search_result_line(divider) is False
def test_grep_output_still_detected_as_search_results() -> None:
"""Guard against over-tightening: ordinary ``grep -n`` output is unchanged."""
content = "\n".join(
(
"src/main.py:42:def process():",
"src/util.py:13: return None",
"lib/x.py:7:class X:",
"tests/test_a.py:3: assert True",
)
)
result = detect_content_type(content)
assert result.content_type is ContentType.SEARCH_RESULTS
def test_extensionless_grep_paths_still_detected() -> None:
"""Extensionless filenames have no dot to lean on, so they are the tight case."""
content = "\n".join(
(
"Makefile:12:\tpytest -q",
"Dockerfile:3:RUN apt-get update",
"Jenkinsfile:88: sh 'make test'",
)
)
result = detect_content_type(content)
assert result.content_type is ContentType.SEARCH_RESULTS