mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description
This PR removes four compatibility helpers whose only callers were their
own tests:
| Name | What it was |
|---|---|
| `headroom.rollout.current_rollout(environ)` | A wrapper that only
called `resolve_rollout(environ)`. |
| `headroom.rollout.feature_enabled(feature, *, explicit, environ)` | A
wrapper around `resolve_rollout(environ,
requested=...).is_enabled(feature)`. |
| `headroom.rollout.Rollout` | The alias `Rollout = RolloutSnapshot`.
Its own comment called it an undocumented in-branch shim for callers
that were migrating. Those callers have migrated. |
| `headroom.agent_savings.with_target_savings(profile, target_savings)`
| A `dataclasses.replace` helper that nothing called. |
Why this is safe to delete rather than deprecate:
- Neither module has an `__all__`, and these names are not re-exported
anywhere.
- None of the names is documented in `docs/content/docs/`.
`configuration.mdx` documents the rollout environment variables, not
these functions.
- None of the org's consumer repos imports `headroom.rollout` or
`headroom.agent_savings`.
Evidence: a word-boundary search across `headroom`, `crates`, `plugins`,
`sdk`, `examples`, `benchmarks`, `scripts`, `e2e`, `docs`, `wiki`,
`.github`, `pyproject.toml`, `tests` and the consumer repos finds only
the definitions and their tests. The `feature_enabled` hits in
`tests/test_backends/test_litellm_bedrock_prompt_caching.py` are an
unrelated pytest fixture with the same name.
Refs #4136
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)
## Changes Made
- `headroom/rollout.py`: delete `current_rollout`, `feature_enabled` and
the `Rollout` alias.
- `headroom/agent_savings.py`: delete `with_target_savings`, and the
`replace` import it alone used.
- `tests/test_rollout.py`:
`test_snapshot_query_and_compatibility_helpers` becomes
`test_snapshot_query_helpers`. It makes the same assertions through
`resolve_rollout(...)` directly, which is what the helpers wrapped.
- `tests/test_agent_savings.py`: delete
`test_with_target_savings_recomputes_target_ratio`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ pytest -q -p no:cacheprovider tests/test_rollout.py tests/test_agent_savings.py tests/test_backends/test_litellm_bedrock_prompt_caching.py tests/test_config.py
146 passed in 6.83s
$ uvx ruff@0.16.8 check headroom/rollout.py headroom/agent_savings.py tests/test_rollout.py tests/test_agent_savings.py
All checks passed!
$ python -m mypy headroom/rollout.py headroom/agent_savings.py --ignore-missing-imports
Success: no issues found
$ rg -n 'current_rollout|feature_enabled\(|with_target_savings' headroom tests docs wiki
tests/test_backends/test_litellm_bedrock_prompt_caching.py:102:def feature_enabled(monkeypatch: pytest.MonkeyPatch) -> None: # unrelated fixture
```
## Real Behavior Proof
- Environment: macOS, Python 3.12.13, this branch checked out in a
worktree, repo .venv; temp HOME and HEADROOM_WORKSPACE_DIR, offline.
- Exact command / steps: (1) The shared smoke harness `smoke.sh
<worktree>`: it starts a real proxy from this checkout against a local
fake upstream and sends one Anthropic and one OpenAI request with a
compressible tool result. (2) A direct command on this PR's own path:
`HOME=$TMP/home HEADROOM_WORKSPACE_DIR=$TMP/ws
HEADROOM_ROLLOUT_CHANNEL=canary
HEADROOM_FEATURES=tool_result_interceptors python -m headroom.cli
rollout status`.
- Observed result: Both requests return 200 and `/stats` shows savings,
with only the expected offline-mode warnings in proxy.log. The direct
command's output is below.
- Not tested: An outside caller of the removed undocumented helpers;
none exists in the repo or the org's consumer repos.
```text
$ bash smoke.sh <worktree>
branch: cleanup/core-misc-test-only-helpers @ af71b61a
health: 200
POST /v1/messages -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines ($SMOKE_TMP/ws/logs/proxy-50198.log):
2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes.
2 headroom.transforms.kompress_compressor - WARNING - Kompress: 'onnx/kompress-int8-wo.onnx' for chopratejas/kompress-v2-base is not cached and HEADROOM_OFFLINE forbids fetching it (HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.). Compression falls back to the non-ML path.
4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base
4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.
(smoke dir: $SMOKE_TMP)
$ HOME=$TMP/home HEADROOM_WORKSPACE_DIR=$TMP/ws HEADROOM_ROLLOUT_CHANNEL=canary HEADROOM_FEATURES=tool_result_interceptors python -m headroom.cli rollout status
Rollout channel: canary
Policy: 1 (sha256:7752c57ebd846a5cf5f1f3d56c1a1b3e84b69bef54b5e67da6353f51e5dded2f)
Snapshot: sha256:d0a3cdbe4b506547100168eeaba3ac4c1b086876ca1b97209962080146897b2f
Qualification eligible: true
bedrock_openai_prompt_caching: enabled=false decision=not_requested
proxy_output_shaper: enabled=false decision=not_requested
read_maturation: enabled=false decision=not_requested
tool_result_interceptors: enabled=true decision=explicit
exit=0
```
## Runtime Rollout Safety
- Rollout-managed feature(s): None changed. `resolve_rollout`,
`RolloutSnapshot` and `FEATURES` are untouched.
- Minimum rollout channel: N/A; dead-code removal.
- Stable/default behavior changed: No.
- Kill switch / disable path: Unchanged (`HEADROOM_DISABLE_FEATURES`).
- Unsafe override required: No.
- Qualification impact: None. The registry digest and snapshot format
are unchanged.
- Rollback path: Revert this commit.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A
## Additional Notes
Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review.
The parser helpers that were first planned for this PR,
`find_tool_units` and `get_message_content_text`, live in a
consumer-imported module. They are deprecated in a separate PR instead
of being deleted.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
469 lines
16 KiB
Python
469 lines
16 KiB
Python
from __future__ import annotations
|
|
|
|
import json
|
|
import logging
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
from click.testing import CliRunner
|
|
|
|
from headroom.cli.main import main
|
|
from headroom.config import HeadroomConfig
|
|
from headroom.rollout import (
|
|
FEATURES,
|
|
FeatureDecisionReason,
|
|
FeatureSpec,
|
|
RolloutChannel,
|
|
RolloutConfigurationError,
|
|
RolloutSnapshot,
|
|
registry_digest,
|
|
resolve_rollout,
|
|
)
|
|
from headroom.transforms.pipeline import TransformPipeline
|
|
|
|
|
|
def test_default_stable_resolution_is_versioned_and_eligible() -> None:
|
|
snapshot = resolve_rollout({})
|
|
|
|
assert snapshot.channel is RolloutChannel.STABLE
|
|
assert snapshot.schema_version == 1
|
|
assert snapshot.policy_version == "1"
|
|
assert snapshot.qualification_eligible is True
|
|
|
|
|
|
@pytest.mark.parametrize("channel", ["beta", "canary", "dev"])
|
|
def test_valid_rollout_channels(channel: str) -> None:
|
|
assert resolve_rollout({"HEADROOM_ROLLOUT_CHANNEL": channel}).channel.value == channel
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("alias", "expected"),
|
|
[
|
|
("prod", RolloutChannel.STABLE),
|
|
("production", RolloutChannel.STABLE),
|
|
("preview", RolloutChannel.BETA),
|
|
("nightly", RolloutChannel.CANARY),
|
|
("development", RolloutChannel.DEV),
|
|
],
|
|
)
|
|
def test_channel_aliases(alias: str, expected: RolloutChannel) -> None:
|
|
assert RolloutChannel.parse(alias) is expected
|
|
|
|
|
|
def test_strict_channel_configuration_rejects_unknown_input() -> None:
|
|
with pytest.raises(RolloutConfigurationError, match="unknown rollout channel"):
|
|
resolve_rollout({"HEADROOM_ROLLOUT_CHANNEL": "stabel"}, strict=True)
|
|
|
|
|
|
def test_unknown_channel_fails_closed_with_diagnostic(caplog: pytest.LogCaptureFixture) -> None:
|
|
with caplog.at_level(logging.WARNING):
|
|
snapshot = resolve_rollout({"HEADROOM_ROLLOUT_CHANNEL": "stabel"})
|
|
|
|
assert snapshot.channel is RolloutChannel.STABLE
|
|
assert "unknown rollout channel 'stabel'; falling back to 'stable'" in caplog.text
|
|
|
|
|
|
def test_unknown_requested_and_disabled_features_fail_closed_and_warn(
|
|
caplog: pytest.LogCaptureFixture,
|
|
) -> None:
|
|
with caplog.at_level(logging.WARNING):
|
|
snapshot = resolve_rollout(
|
|
{
|
|
"HEADROOM_FEATURES": "typo_requested",
|
|
"HEADROOM_DISABLE_FEATURES": "typo_disabled",
|
|
}
|
|
)
|
|
|
|
assert snapshot.config.requested == frozenset()
|
|
assert snapshot.config.disabled == frozenset()
|
|
assert "typo_requested" in caplog.text
|
|
assert "typo_disabled" in caplog.text
|
|
|
|
|
|
def test_strict_configuration_rejects_unknown_input() -> None:
|
|
with pytest.raises(RolloutConfigurationError, match="unknown rollout feature"):
|
|
resolve_rollout({"HEADROOM_FEATURES": "typo"}, strict=True)
|
|
|
|
|
|
def test_stable_blocks_explicit_canary_feature() -> None:
|
|
snapshot = resolve_rollout(
|
|
{
|
|
"HEADROOM_ROLLOUT_CHANNEL": "stable",
|
|
"HEADROOM_FEATURES": "tool-result-interceptors",
|
|
}
|
|
)
|
|
decision = snapshot.decision("tool_result_interceptors")
|
|
|
|
assert decision.enabled is False
|
|
assert decision.reason is FeatureDecisionReason.BLOCKED_BY_CHANNEL
|
|
|
|
|
|
def test_canary_allows_explicit_request() -> None:
|
|
snapshot = resolve_rollout(
|
|
{
|
|
"HEADROOM_ROLLOUT_CHANNEL": "canary",
|
|
"HEADROOM_FEATURES": "tool_result_interceptors",
|
|
}
|
|
)
|
|
|
|
assert snapshot.decision("tool_result_interceptors").reason is FeatureDecisionReason.EXPLICIT
|
|
|
|
|
|
def test_non_default_feature_remains_off_when_not_requested() -> None:
|
|
decision = resolve_rollout({"HEADROOM_ROLLOUT_CHANNEL": "dev"}).decision(
|
|
"tool_result_interceptors"
|
|
)
|
|
assert decision.enabled is False
|
|
assert decision.reason is FeatureDecisionReason.NOT_REQUESTED
|
|
|
|
|
|
def test_legacy_alias_obeys_channel_and_has_distinct_reason() -> None:
|
|
stable = resolve_rollout(
|
|
{"HEADROOM_ROLLOUT_CHANNEL": "stable", "HEADROOM_INTERCEPT_ENABLED": "1"}
|
|
)
|
|
canary = resolve_rollout(
|
|
{"HEADROOM_ROLLOUT_CHANNEL": "canary", "HEADROOM_INTERCEPT_ENABLED": "1"}
|
|
)
|
|
|
|
assert (
|
|
stable.decision("tool_result_interceptors").reason
|
|
is FeatureDecisionReason.BLOCKED_BY_CHANNEL
|
|
)
|
|
assert canary.decision("tool_result_interceptors").reason is FeatureDecisionReason.LEGACY_ALIAS
|
|
|
|
|
|
@pytest.mark.parametrize("request_source", ["HEADROOM_FEATURES", "HEADROOM_INTERCEPT_ENABLED"])
|
|
def test_disable_beats_explicit_and_legacy_request(request_source: str) -> None:
|
|
snapshot = resolve_rollout(
|
|
{
|
|
"HEADROOM_ROLLOUT_CHANNEL": "canary",
|
|
request_source: "tool_result_interceptors"
|
|
if request_source.endswith("FEATURES")
|
|
else "1",
|
|
"HEADROOM_DISABLE_FEATURES": "tool_result_interceptors",
|
|
}
|
|
)
|
|
decision = snapshot.decision("tool_result_interceptors")
|
|
|
|
assert decision.enabled is False
|
|
assert decision.reason is FeatureDecisionReason.DISABLED
|
|
|
|
|
|
def test_unsafe_override_crosses_channel_and_poisons_qualification() -> None:
|
|
snapshot = resolve_rollout(
|
|
{
|
|
"HEADROOM_ROLLOUT_CHANNEL": "stable",
|
|
"HEADROOM_FEATURES": "tool_result_interceptors",
|
|
"HEADROOM_UNSAFE_ALLOW_UNSTABLE_FEATURES": "1",
|
|
}
|
|
)
|
|
payload = snapshot.to_dict()
|
|
|
|
assert (
|
|
snapshot.decision("tool_result_interceptors").reason
|
|
is FeatureDecisionReason.UNSAFE_OVERRIDE
|
|
)
|
|
assert payload["qualification_eligible"] is False
|
|
assert payload["qualification_ineligible_reason"] == "unsafe_rollout_override_active"
|
|
|
|
|
|
def test_disable_still_beats_unsafe_override() -> None:
|
|
snapshot = resolve_rollout(
|
|
{
|
|
"HEADROOM_FEATURES": "tool_result_interceptors",
|
|
"HEADROOM_DISABLE_FEATURES": "tool_result_interceptors",
|
|
"HEADROOM_UNSAFE_ALLOW_UNSTABLE_FEATURES": "1",
|
|
}
|
|
)
|
|
assert snapshot.decision("tool_result_interceptors").reason is FeatureDecisionReason.DISABLED
|
|
|
|
|
|
def test_live_legacy_reresolution_preserves_channel_and_named_kill_switch() -> None:
|
|
eligible = resolve_rollout({"HEADROOM_ROLLOUT_CHANNEL": "beta"})
|
|
enabled = eligible.with_legacy_env({"HEADROOM_OUTPUT_SHAPER": "1"})
|
|
disabled_again = enabled.with_legacy_env({"HEADROOM_OUTPUT_SHAPER": "0"})
|
|
killed = resolve_rollout(
|
|
{
|
|
"HEADROOM_ROLLOUT_CHANNEL": "beta",
|
|
"HEADROOM_DISABLE_FEATURES": "proxy_output_shaper",
|
|
}
|
|
).with_legacy_env({"HEADROOM_OUTPUT_SHAPER": "1"})
|
|
# ``proxy_output_shaper`` is STABLE and default-on, so it can no longer
|
|
# show a legacy alias being refused by the channel gate. ``read_maturation``
|
|
# is still BETA, so it carries that half of the assertion.
|
|
blocked = resolve_rollout({}).with_legacy_env({"HEADROOM_READ_MATURATION": "1"})
|
|
|
|
assert enabled.decision("proxy_output_shaper").reason is FeatureDecisionReason.LEGACY_ALIAS
|
|
assert disabled_again.decision("proxy_output_shaper").reason is FeatureDecisionReason.DISABLED
|
|
assert killed.decision("proxy_output_shaper").reason is FeatureDecisionReason.DISABLED
|
|
assert blocked.decision("read_maturation").reason is FeatureDecisionReason.BLOCKED_BY_CHANNEL
|
|
assert enabled.snapshot_digest != eligible.snapshot_digest
|
|
|
|
|
|
def test_empty_programmatic_feature_names_are_ignored() -> None:
|
|
snapshot = resolve_rollout({}, requested=["", " "], disabled=[""])
|
|
|
|
assert snapshot.config.requested == frozenset()
|
|
assert snapshot.config.disabled == frozenset()
|
|
|
|
|
|
def test_multi_worker_config_round_trip_preserves_typed_rollout(monkeypatch) -> None:
|
|
from headroom.proxy.models import ProxyConfig
|
|
from headroom.proxy.server import (
|
|
_MULTI_WORKER_CONFIG_ENV,
|
|
_proxy_config_from_env,
|
|
_proxy_config_payload,
|
|
)
|
|
|
|
rollout = resolve_rollout(
|
|
{
|
|
"HEADROOM_ROLLOUT_CHANNEL": "beta",
|
|
"HEADROOM_OUTPUT_SHAPER": "1",
|
|
"HEADROOM_DISABLE_FEATURES": "read_maturation",
|
|
}
|
|
)
|
|
original = ProxyConfig(
|
|
rollout=rollout,
|
|
worker_processes=2,
|
|
profile_seeded_env_keys=frozenset({"HEADROOM_MODE", "HEADROOM_LOSSLESS"}),
|
|
)
|
|
monkeypatch.setenv(_MULTI_WORKER_CONFIG_ENV, json.dumps(_proxy_config_payload(original)))
|
|
|
|
restored = _proxy_config_from_env()
|
|
|
|
assert restored.rollout is not None
|
|
assert restored.rollout.to_internal_dict() == rollout.to_internal_dict()
|
|
assert restored.rollout.is_enabled("proxy_output_shaper") is True
|
|
assert restored.rollout.is_enabled("read_maturation") is False
|
|
assert restored.worker_processes == 2
|
|
assert restored.profile_seeded_env_keys == frozenset({"HEADROOM_MODE", "HEADROOM_LOSSLESS"})
|
|
|
|
|
|
def test_documented_proxy_json_without_internal_snapshot_is_preserved(monkeypatch) -> None:
|
|
from headroom.proxy.server import _MULTI_WORKER_CONFIG_ENV, _proxy_config_from_env
|
|
|
|
monkeypatch.setenv(
|
|
_MULTI_WORKER_CONFIG_ENV,
|
|
json.dumps(
|
|
{
|
|
"port": 39099,
|
|
"rate_limit_enabled": False,
|
|
"proxy_token": "required-token",
|
|
"offline": True,
|
|
}
|
|
),
|
|
)
|
|
|
|
restored = _proxy_config_from_env()
|
|
|
|
assert restored.port == 39099
|
|
assert restored.rate_limit_enabled is False
|
|
assert restored.proxy_token == "required-token"
|
|
assert restored.offline is True
|
|
assert restored.rollout is not None
|
|
|
|
|
|
def test_internal_proxy_json_still_rejects_tampered_rollout_snapshot(monkeypatch) -> None:
|
|
from headroom.proxy.models import ProxyConfig
|
|
from headroom.proxy.server import (
|
|
_MULTI_WORKER_CONFIG_ENV,
|
|
_proxy_config_from_env,
|
|
_proxy_config_payload,
|
|
)
|
|
|
|
payload = _proxy_config_payload(ProxyConfig(port=39099))
|
|
payload["_rollout_snapshot"]["snapshot_digest"] = "sha256:tampered" # type: ignore[index]
|
|
monkeypatch.setenv(_MULTI_WORKER_CONFIG_ENV, json.dumps(payload))
|
|
monkeypatch.setenv("HEADROOM_PORT", "39100")
|
|
|
|
restored = _proxy_config_from_env()
|
|
|
|
assert restored.port == 39100
|
|
assert restored.rollout is not None
|
|
|
|
|
|
@pytest.mark.parametrize("raw_config", ["null", "[]", '"not-an-object"'])
|
|
def test_non_object_proxy_json_falls_back_without_crashing(monkeypatch, raw_config: str) -> None:
|
|
from headroom.proxy.server import _MULTI_WORKER_CONFIG_ENV, _proxy_config_from_env
|
|
|
|
monkeypatch.setenv(_MULTI_WORKER_CONFIG_ENV, raw_config)
|
|
monkeypatch.setenv("HEADROOM_PORT", "39100")
|
|
|
|
restored = _proxy_config_from_env()
|
|
|
|
assert restored.port == 39100
|
|
assert restored.rollout is not None
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("mutation", "message"),
|
|
[
|
|
({"schema_version": 999}, "schema version"),
|
|
({"policy_version": "999"}, "policy version"),
|
|
({"unsafe_allow_unstable": "yes"}, "unsafe override"),
|
|
({"explicit_requested": "proxy_output_shaper"}, "explicit_requested"),
|
|
({"registry_digest": "sha256:tampered"}, "registry digest"),
|
|
({"snapshot_digest": "sha256:tampered"}, "snapshot digest"),
|
|
],
|
|
)
|
|
def test_worker_rollout_handoff_rejects_invalid_or_tampered_state(
|
|
mutation: dict[str, object], message: str
|
|
) -> None:
|
|
payload = resolve_rollout({}).to_internal_dict()
|
|
payload.update(mutation)
|
|
|
|
with pytest.raises(RolloutConfigurationError, match=message):
|
|
RolloutSnapshot.from_internal_dict(payload)
|
|
|
|
|
|
def test_worker_rollout_handoff_rejects_non_object_state() -> None:
|
|
with pytest.raises(RolloutConfigurationError, match="invalid rollout worker snapshot"):
|
|
RolloutSnapshot.from_internal_dict([]) # type: ignore[arg-type]
|
|
|
|
|
|
def test_snapshot_query_helpers() -> None:
|
|
snapshot = resolve_rollout(
|
|
{
|
|
"HEADROOM_ROLLOUT_CHANNEL": "canary",
|
|
"HEADROOM_FEATURES": "tool_result_interceptors",
|
|
"HEADROOM_DISABLE_FEATURES": "read_maturation",
|
|
}
|
|
)
|
|
|
|
assert snapshot.is_available("tool-result-interceptors") is True
|
|
assert snapshot.enabled == frozenset({"tool_result_interceptors"})
|
|
assert snapshot.disabled == frozenset({"read_maturation"})
|
|
assert resolve_rollout(
|
|
{"HEADROOM_ROLLOUT_CHANNEL": "canary"},
|
|
requested=("tool_result_interceptors",),
|
|
).is_enabled("tool_result_interceptors")
|
|
assert not resolve_rollout({}).is_enabled("tool_result_interceptors")
|
|
with pytest.raises(KeyError, match="missing"):
|
|
snapshot.decision("missing")
|
|
|
|
|
|
def test_registry_and_snapshot_digests_are_deterministic_and_policy_sensitive() -> None:
|
|
first = resolve_rollout({"HEADROOM_ROLLOUT_CHANNEL": "canary"})
|
|
second = resolve_rollout({"HEADROOM_ROLLOUT_CHANNEL": "canary"})
|
|
equivalent = dict(reversed(list(FEATURES.items())))
|
|
changed = dict(FEATURES)
|
|
changed["tool_result_interceptors"] = FeatureSpec(
|
|
"tool_result_interceptors", RolloutChannel.BETA
|
|
)
|
|
|
|
assert first.registry_digest == second.registry_digest == registry_digest(equivalent)
|
|
assert first.snapshot_digest == second.snapshot_digest
|
|
assert registry_digest(changed) != first.registry_digest
|
|
assert json.dumps(first.to_dict(), sort_keys=True) == json.dumps(
|
|
second.to_dict(), sort_keys=True
|
|
)
|
|
|
|
|
|
def test_pipeline_uses_config_snapshot_after_environment_mutation(
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
) -> None:
|
|
monkeypatch.delenv("HEADROOM_ROLLOUT_CHANNEL", raising=False)
|
|
monkeypatch.delenv("HEADROOM_FEATURES", raising=False)
|
|
config = HeadroomConfig()
|
|
original_digest = config.rollout.snapshot_digest if config.rollout else None
|
|
monkeypatch.setenv("HEADROOM_ROLLOUT_CHANNEL", "canary")
|
|
monkeypatch.setenv("HEADROOM_FEATURES", "tool_result_interceptors")
|
|
|
|
pipeline = TransformPipeline(config)
|
|
|
|
assert config.rollout is not None
|
|
assert config.rollout.snapshot_digest == original_digest
|
|
assert all(
|
|
type(transform).__name__ != "ToolResultInterceptorTransform"
|
|
for transform in pipeline.transforms
|
|
)
|
|
|
|
|
|
def test_cli_json_status_and_strict_error() -> None:
|
|
runner = CliRunner()
|
|
result = runner.invoke(
|
|
main,
|
|
[
|
|
"rollout",
|
|
"status",
|
|
"--channel",
|
|
"canary",
|
|
"--features",
|
|
"tool_result_interceptors",
|
|
"--json",
|
|
],
|
|
)
|
|
invalid = runner.invoke(main, ["rollout", "status", "--features", "typo", "--json"])
|
|
|
|
assert result.exit_code == 0
|
|
payload = json.loads(result.output)
|
|
assert payload["channel"] == "canary"
|
|
assert "tool_result_interceptors" in [feature["name"] for feature in payload["features"]]
|
|
assert invalid.exit_code != 0
|
|
assert "unknown rollout feature" in invalid.output
|
|
|
|
|
|
def test_cli_human_status_exercises_disable_and_unsafe_options() -> None:
|
|
result = CliRunner().invoke(
|
|
main,
|
|
[
|
|
"rollout",
|
|
"status",
|
|
"--channel",
|
|
"stable",
|
|
"--features",
|
|
"tool_result_interceptors",
|
|
"--disable-features",
|
|
"read_maturation",
|
|
"--unsafe-allow-unstable-features",
|
|
],
|
|
)
|
|
|
|
assert result.exit_code == 0
|
|
assert "Rollout channel: stable" in result.output
|
|
assert "Qualification eligible: false" in result.output
|
|
assert "tool_result_interceptors: enabled=true decision=unsafe_override" in result.output
|
|
assert "read_maturation: enabled=false decision=disabled" in result.output
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("option", "message", "required_channel"),
|
|
[
|
|
("--read-maturation", "--read-maturation is not available", "beta"),
|
|
(
|
|
"--intercept-tool-results",
|
|
"--intercept-tool-results is not available",
|
|
"canary",
|
|
),
|
|
],
|
|
)
|
|
def test_proxy_cli_fails_loudly_when_explicit_feature_is_channel_blocked(
|
|
option: str, message: str, required_channel: str
|
|
) -> None:
|
|
result = CliRunner().invoke(
|
|
main,
|
|
["proxy", option],
|
|
env={"HEADROOM_ROLLOUT_CHANNEL": "stable"},
|
|
)
|
|
|
|
assert result.exit_code == 1
|
|
assert message in result.output
|
|
assert f"HEADROOM_ROLLOUT_CHANNEL={required_channel}" in result.output
|
|
|
|
|
|
def test_shared_python_rust_policy_vectors() -> None:
|
|
vectors = json.loads(
|
|
(Path(__file__).parent / "fixtures" / "rollout_policy_vectors.json").read_text()
|
|
)
|
|
for vector in vectors:
|
|
env = {"HEADROOM_ROLLOUT_CHANNEL": vector["channel"]}
|
|
if vector["requested"]:
|
|
env["HEADROOM_FEATURES"] = "tool_result_interceptors"
|
|
if vector["disabled"]:
|
|
env["HEADROOM_DISABLE_FEATURES"] = "tool_result_interceptors"
|
|
if vector["unsafe"]:
|
|
env["HEADROOM_UNSAFE_ALLOW_UNSTABLE_FEATURES"] = "1"
|
|
decision = resolve_rollout(env).decision("tool_result_interceptors")
|
|
assert decision.enabled is vector["enabled"]
|
|
assert decision.reason.value == vector["decision"]
|