Files
headroom/wiki/troubleshooting.md
eaa16d9f63 fix(proxy): preserve Windows service when Rust core is blocked (#2989)
## Description

When `headroom._core` cannot load, `HEADROOM_REQUIRE_RUST_CORE=false`
should keep the Python proxy available in unoptimized passthrough mode.
A module-level native import made that escape hatch unreachable before
startup could read the setting. This addresses the degraded-service
portion of #2918 and advances #1466; neither issue is closed because
Authenticode/Smart App Control reputation work remains.

## Type of Change

- [x] Bug fix (non-breaking change)
- [x] Documentation update

## Changes Made

- Resolve Rust-backed error-detection registries and patterns lazily
while retaining Rust as the sole source of truth.
- Force `optimize=false` only when the operator explicitly allows a
missing Rust core, keeping traffic available and unmodified.
- Preserve the default fail-loud startup behavior.
- Add Windows Smart App Control, WDAC, antivirus, and Code Integrity
diagnostics.
- Add cold-interpreter coverage for the import boundary and degraded
startup behavior.

## Testing

- [x] Focused tests pass
- [x] Ruff check and formatting pass
- [x] `git diff --check` passes
- [x] New regression tests added

```text
uv run --extra dev pytest -q tests/test_error_detection.py tests/test_rust_core_smoke.py
16 passed in 1.64s (independent exact-head review run)

Additional author verification: 38 focused proxy/core/error-detection/parity/MCP contract tests passed.
```

CI is rerunning on the current head after its merge from `main`; no
failure should be inferred from an unrun or pending job.

## Real Behavior Proof

- Environment: cold CPython 3.12 interpreter with `headroom._core`
marked unavailable and `HEADROOM_REQUIRE_RUST_CORE=false`.
- Exact steps: import `ProxyConfig`/`create_app`, construct the
application with `optimize=True`, and enter the FastAPI lifespan through
the focused degraded-mode tests.
- Observed result: the server module and application construct
successfully; lifespan reports `rust_core=disabled` and changes the
shared config to `optimize=False`. Rust-backed constants still fail
loudly if explicitly used without the extension.
- Not tested: a production Windows host with Smart App Control actively
blocking the shipped `_core.pyd`; the failure boundary is reproduced
deterministically in a child interpreter.

## Runtime Rollout Safety

- Rollout-managed feature: explicit degraded mode controlled by
`HEADROOM_REQUIRE_RUST_CORE=false`.
- Stable/default behavior changed: no; the default still exits loudly
when the required core cannot load.
- Kill switch / disable path: unset the opt-out or set it to true to
restore fail-loud behavior.
- Unsafe override required: no.
- Qualification impact: cold-import, degraded-startup, parity, and
error-detection tests must remain green.
- Rollback path: revert this PR; no persistent data or migration is
involved.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] Code follows project style
- [x] Documentation is updated
- [x] New tests cover the changed behavior
- [x] No manual changelog edit

## Remaining Work

#2918 remains open for Authenticode signing and Smart App Control
reputation. #1466 remains the Windows compatibility umbrella.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-01 16:10:08 -05:00

14 KiB

Troubleshooting Guide

Solutions for common Headroom issues.


Proxy Server Issues

"Proxy won't start"

Symptom: headroom proxy fails or hangs.

Solutions:

# 1. Check if port is already in use
lsof -i :8787
# If something is using the port, either kill it or use a different port

# 2. Try a different port
headroom proxy --port 8788

# 3. Check for missing dependencies
pip install "headroom-ai[proxy]"

# 4. Run with request logging
headroom proxy --log-file ~/.headroom/logs/proxy.jsonl --log-messages

"Connection refused" when calling proxy

Symptom: curl: (7) Failed to connect to localhost port 8787

Solutions:

# 1. Verify proxy is running
curl http://localhost:8787/health

# 2. Check if proxy started on a different port
ps aux | grep headroom

# 3. Check firewall settings (macOS)
sudo pfctl -s rules | grep 8787

"Upstream rejects a beta token the client no longer sends"

Symptom: The upstream API returns an error referencing a beta feature (anthropic-beta header) even though the client is no longer sending that header.

Cause: Headroom's SessionBetaTracker re-injects any anthropic-beta token seen earlier in the same session to preserve prefix-cache stability. Once a token is in the tracker it persists for the rest of the session. Stopping the token on the client side alone is not sufficient.

Solution: Set HEADROOM_BETA_HEADER_STICKY=disabled to pass the client's header value verbatim without accumulation:

export HEADROOM_BETA_HEADER_STICKY=disabled
headroom proxy ...

Alternatively, restarting the proxy process clears the in-memory tracker. See Session Beta Header Tracking for details.


"Proxy returns errors for some requests"

Symptom: Some requests work, others fail with 502/503.

Solutions:

# 1. Check proxy logs for the actual error
headroom proxy --log-file ~/.headroom/logs/proxy.jsonl --log-messages

# 2. Verify API key is set
echo $OPENAI_API_KEY  # or ANTHROPIC_API_KEY

# 3. Test the underlying API directly
curl https://api.openai.com/v1/models -H "Authorization: Bearer $OPENAI_API_KEY"

SDK Issues

"No token savings"

Symptom: stats['session']['tokens_saved_total'] is 0.

Diagnosis:

# 1. Check mode
stats = client.get_stats()
print(f"Mode: {stats['config']['mode']}")  # Should be "optimize"

# 2. Check transforms are enabled
print(f"SmartCrusher: {stats['transforms']['smart_crusher_enabled']}")

# 3. Check if content meets threshold
# SmartCrusher only compresses tool outputs > 200 tokens by default

Solutions:

# 1. Ensure mode is "optimize"
client = HeadroomClient(
    original_client=OpenAI(),
    provider=OpenAIProvider(),
    default_mode="optimize",  # NOT "audit"
)

# 2. Or override per-request
response = client.chat.completions.create(
    model="gpt-4o",
    messages=messages,
    headroom_mode="optimize",
)

# 3. Lower the compression threshold
config = HeadroomConfig()
config.smart_crusher.min_tokens_to_crush = 100  # Default is 200

Why It Might Be 0:

  • Mode is "audit" (observation only)
  • Messages don't contain tool outputs
  • Tool outputs are below the token threshold
  • Data isn't compressible (high uniqueness)

"Compression too aggressive"

Symptom: LLM responses are missing information that was in tool outputs.

Solutions:

# 1. Keep more items
config = HeadroomConfig()
config.smart_crusher.max_items_after_crush = 50  # Default: 15

# 2. Skip compression for specific tools
response = client.chat.completions.create(
    model="gpt-4o",
    messages=messages,
    headroom_tool_profiles={
        "important_tool": {"skip_compression": True},
    },
)

# 3. Disable SmartCrusher entirely
config.smart_crusher.enabled = False

"High latency"

Symptom: Requests take longer than expected.

Diagnosis:

import time
import logging

logging.basicConfig(level=logging.DEBUG)

start = time.time()
response = client.chat.completions.create(...)
print(f"Total time: {time.time() - start:.2f}s")

# Check logs for:
# - "SmartCrusher" timing
# - "EmbeddingScorer" timing (slow if using embeddings)

Solutions:

# 1. Use BM25 instead of embeddings (faster)
config = HeadroomConfig()
config.smart_crusher.relevance.tier = "bm25"  # Default may use embeddings

# 2. Increase threshold to skip small payloads
config.smart_crusher.min_tokens_to_crush = 500

# 3. Disable transforms you don't need
config.cache_aligner.enabled = False
config.rolling_window.enabled = False

"ValidationError on setup"

Symptom: validate_setup() returns errors.

Common Issues:

result = client.validate_setup()
print(result)

# Provider error:
# {"provider": {"ok": False, "error": "No API key"}}
# → Set OPENAI_API_KEY or pass api_key to OpenAI()

# Storage error:
# {"storage": {"ok": False, "error": "unable to open database"}}
# → Check path permissions, use :memory: for testing

# Config error:
# {"config": {"ok": False, "error": "Invalid mode"}}
# → Use "audit" or "optimize" only

Solutions:

# 1. For testing, use in-memory storage
client = HeadroomClient(
    original_client=OpenAI(),
    provider=OpenAIProvider(),
    store_url="sqlite:///:memory:",  # No file created
)

# 2. For temp directory storage
import tempfile
import os

db_path = os.path.join(tempfile.gettempdir(), "headroom.db")
client = HeadroomClient(
    original_client=OpenAI(),
    provider=OpenAIProvider(),
    store_url=f"sqlite:///{db_path}",
)

Import/Installation Issues

"pip install fails with C++ compilation error"

Symptom: Installation fails with an error like:

RuntimeError: Unsupported compiler -- at least C++11 support is needed!
ERROR: Failed building wheel for hnswlib

Cause: headroom-ai depends on hnswlib, a C++ extension that must be compiled from source. Slim environments (Docker slim images, minimal CI runners) lack the required build tools.

Solutions:

# Linux / Debian-based (including Docker)
apt-get install -y build-essential && pip install headroom-ai

# macOS (Xcode command line tools)
xcode-select --install && pip install headroom-ai

In a Dockerfile, install and remove build tools in one layer to keep the image slim:

FROM python:3.11-slim
RUN apt-get update && apt-get install -y --no-install-recommends build-essential \
    && pip install "headroom-ai[proxy]" \
    && apt-get purge -y build-essential && apt-get autoremove -y \
    && rm -rf /var/lib/apt/lists/*

"ModuleNotFoundError: No module named 'headroom'"

# 1. Check it's installed in the right environment
pip show headroom-ai

# 2. If using virtual environment, ensure it's activated
source venv/bin/activate  # or equivalent

# 3. Reinstall
pip install --upgrade headroom-ai

"ImportError: cannot import name 'X' from 'headroom'"

# Check available imports
import headroom

print(dir(headroom))

# Common imports:
from headroom import (
    HeadroomClient,
    OpenAIProvider,
    AnthropicProvider,
    HeadroomConfig,
    # Exceptions
    HeadroomError,
    ConfigurationError,
    ProviderError,
)

"Missing optional dependency"

# For proxy server
pip install "headroom-ai[proxy]"

# For embedding-based relevance scoring
pip install "headroom-ai[relevance]"

# For everything
pip install "headroom-ai[all]"

Provider-Specific Issues

OpenAI: "Invalid API key"

from openai import OpenAI
import os

# Ensure key is set
api_key = os.environ.get("OPENAI_API_KEY")
if not api_key:
    raise ValueError("OPENAI_API_KEY not set")

client = HeadroomClient(
    original_client=OpenAI(api_key=api_key),
    provider=OpenAIProvider(),
)

Anthropic: "Authentication error"

from anthropic import Anthropic
import os

api_key = os.environ.get("ANTHROPIC_API_KEY")
client = HeadroomClient(
    original_client=Anthropic(api_key=api_key),
    provider=AnthropicProvider(),
)

"Unknown model" warnings

# For custom/fine-tuned models, specify context limit
client = HeadroomClient(
    original_client=OpenAI(),
    provider=OpenAIProvider(),
    model_context_limits={
        "ft:gpt-4o-2024-08-06:my-org::abc123": 128000,
        "my-custom-model": 32000,
    },
)

Debugging Techniques

Enable Full Logging

import logging

# See everything
logging.basicConfig(
    level=logging.DEBUG,
    format="%(asctime)s %(name)s %(levelname)s %(message)s",
)

# Or just Headroom logs
logging.getLogger("headroom").setLevel(logging.DEBUG)

Inspect Transform Results

# Use simulate to see what would happen
plan = client.chat.completions.simulate(
    model="gpt-4o",
    messages=messages,
)

print(f"Tokens: {plan.tokens_before} -> {plan.tokens_after}")
print(f"Transforms: {plan.transforms}")
print(f"Waste signals: {plan.waste_signals}")

# See the actual optimized messages
import json

print(json.dumps(plan.messages_optimized, indent=2))

Check Storage Contents

from datetime import datetime, timedelta

# Get recent metrics
metrics = client.get_metrics(
    start_time=datetime.utcnow() - timedelta(hours=1),
    limit=10,
)

for m in metrics:
    print(f"{m.timestamp}: {m.tokens_input_before} -> {m.tokens_input_after}")
    print(f"  Transforms: {m.transforms_applied}")
    if m.error:
        print(f"  ERROR: {m.error}")

Manual Transform Testing

from headroom import SmartCrusher, Tokenizer
from headroom.config import SmartCrusherConfig
import json

# Test compression directly
config = SmartCrusherConfig()
crusher = SmartCrusher(config)
tokenizer = Tokenizer()

messages = [
    {"role": "tool", "content": json.dumps({"items": list(range(100))}), "tool_call_id": "1"}
]

result = crusher.apply(messages, tokenizer)
print(f"Tokens: {result.tokens_before} -> {result.tokens_after}")
print(f"Compressed content: {result.messages[0]['content'][:200]}...")

"Native detector crashes with illegal instruction"

On some older or virtualized x86_64 CPUs, AVX2 may be unavailable. The Magika/ONNX Runtime detector can require AVX2 through its precompiled runtime binary. Headroom skips that detector tier on x86/x86_64 hosts without AVX2 and falls back to non-Magika detection tiers instead of crashing.

If native startup still fails on an older CPU, set:

export HEADROOM_REQUIRE_RUST_CORE=false

Windows: "DLL load failed while importing _core"

Symptoms: the proxy exits at startup with

FATAL: Rust extension `headroom._core` not loadable.
    error: ImportError: DLL load failed while importing _core: ...

and clients pointed at ANTHROPIC_BASE_URL=http://127.0.0.1:8787 get ConnectionRefused.

The compiled extension headroom\_core.pyd ships unsigned. Windows Smart App Control, WDAC, or antivirus can therefore refuse to load it, and Smart App Control has no per-file allowlist: it is a system-wide, one-way switch. Confirm it is the OS blocking the file rather than a broken install:

# Where the extension lives (adjust for your install method)
$core = "$env:APPDATA\uv\tools\headroom-ai\Lib\site-packages\headroom\_core.pyd"
Get-AuthenticodeSignature $core | Format-List Status, StatusMessage

# Code Integrity block events, if any
Get-WinEvent -LogName "Microsoft-Windows-CodeIntegrity/Operational" -MaxEvents 20 |
    Where-Object Id -in 3033, 3077, 3118 | Format-List TimeCreated, Id, Message

A Status: NotSigned plus event 3033 or 3077 naming _core.pyd confirms it.

To keep the proxy serving while the block is in place, start it in degraded mode. The proxy stays up and forwards traffic unoptimized (you lose the token savings, not the connection), /health reports "rust_core": "disabled", and the startup log carries one event=proxy_optimization_disabled warning so the degradation is visible:

$env:HEADROOM_REQUIRE_RUST_CORE = "false"
headroom proxy

For a scheduled-task or service deployment, bake it into the profile so the supervisor passes it to every restart. Supervisors start with a bare environment and will not pick up an export from your shell, so re-run install apply with the options you originally used plus --env:

headroom install apply --preset persistent-task --env HEADROOM_REQUIRE_RUST_CORE=false
headroom install restart

To get compression back you have to let the file load: exclude the headroom package directory in your antivirus, or turn Smart App Control off (Windows Security > App & browser control > Smart App Control). Note that turning it off is irreversible without reinstalling Windows, so prefer degraded mode unless you were going to disable it anyway.


Error Reference

Exception Meaning Solution
ConfigurationError Invalid config values Check config parameters
ProviderError Provider issue (unknown model, etc.) Set model_context_limits
StorageError Database issue Check path/permissions
CompressionError Compression failed Rare - check data format
TokenizationError Token counting failed Check model name
ValidationError Setup validation failed Run validate_setup()

Handling Errors

from headroom import (
    HeadroomClient,
    HeadroomError,
    ConfigurationError,
    StorageError,
)

try:
    client = HeadroomClient(...)
    response = client.chat.completions.create(...)
except ConfigurationError as e:
    print(f"Config issue: {e}")
    print(f"Details: {e.details}")
except StorageError as e:
    print(f"Storage issue: {e}")
    # Headroom continues to work, just without metrics persistence
except HeadroomError as e:
    print(f"Headroom error: {e}")

Getting Help

  1. Enable debug logging and check the output
  2. Use simulate() to see what transforms would apply
  3. Check validate_setup() for configuration issues
  4. File an issue at https://github.com/headroom-sdk/headroom/issues

When filing an issue, include:

  • Headroom version (pip show headroom)
  • Python version
  • Provider (OpenAI/Anthropic)
  • Debug log output
  • Minimal reproduction code