Files
headroom/benchmarks/comprehensive_eval.py
Tejas ChopraandClaude Opus 5 34da21e89e fix(tokenizers): disable trust_remote_code and allowlist HF tokenizer repos (#3723)
## A-1 (S1, remote code execution): HF tokenizer loader ran caller-named
repositories

`headroom/tokenizers/huggingface.py` called
`AutoTokenizer.from_pretrained(...,
trust_remote_code=True)` on an identifier that traces back to a proxied
request body.
`get_tokenizer_name()` ended in *"assume the model name is the tokenizer
name"*, so an
unrecognised `model` field was handed straight to the loader, which
resolves it against the
HuggingFace Hub and — with `trust_remote_code=True` — executes whatever
Python that
repository ships. Publish a repo, name it in a request body, get code
execution in the proxy
process. Nothing constrained the identifier.

### The fix

1. **`trust_remote_code=False` on every path**, passed explicitly rather
than leaning on a
   library default upstream can change.
2. **`ALLOWED_TOKENIZER_REPOS`**, derived from the mappings the product
actually ships
(`MODEL_TO_TOKENIZER.values()`, so adding a family allows it and nothing
drifts in).
`_load_tokenizer` checks it *before* importing transformers: an
unallowlisted name is
refused outright and never becomes a Hub lookup. It fails closed to
character estimation
rather than silently substituting a different vocabulary under the
caller's name.
3. **`get_tokenizer_name()` fails closed.** The fallthrough now returns
`DEFAULT_TOKENIZER`
instead of echoing the caller's string. A full repo id is accepted only
when it is on the
allowlist, and then the value returned is *our* canonical spelling,
never the caller's.

Two layers on purpose: the resolver is the normal path, the loader is
the chokepoint —
`_load_tokenizer` is importable and callable with an arbitrary string.

`benchmarks/comprehensive_eval.py` also passed `trust_remote_code=True`
to
`datasets.load_dataset`. The brief only named the two tokenizer call
sites; that third one is
fixed here too, and the tree-wide test below now holds the line for all
of them.

### Operator escape hatch

`HEADROOM_HF_TOKENIZER_ALLOWLIST` (comma-separated repo ids) lets an
operator admit private
or self-hosted tokenizer repositories. It is deliberately
environment-only — server-side
configuration, never anything a client can reach through a request body
— and repos added
there are still loaded with `trust_remote_code=False`.

### Default change a customer could depend on

An unrecognised model previously resolved to *itself* as a tokenizer
name. In practice a
made-up id failed to resolve on the Hub and counting fell back to
estimation, so most
deployments see no change. Two cases do change:

- A caller who relied on passing an **arbitrary HF repo id** not in
`MODEL_TO_TOKENIZER`
(e.g. a private fine-tune) and getting exact counts now gets the default
tokenizer
  instead. Remedy: add the repo to `HEADROOM_HF_TOKENIZER_ALLOWLIST`.
- Unknown models now resolve to `DEFAULT_TOKENIZER` (Llama 3), so a
deployment with that
vocabulary in its HF cache gets a real BPE count where it previously got
estimation.

Repo ids that already resolved correctly are unchanged. Alias and prefix
matching run first
and are untouched, so existing precedence — including
`mistralai/Mixtral-8x7B-v0.1` hitting
the `mistral` prefix — resolves exactly as before; there is an explicit
test pinning that so
a future reorder has to acknowledge it.

### Tests

- `tests/test_huggingface_tokenizer_allowlist.py` — a crafted `model`
never reaches
`from_pretrained`; every resolution lands inside the shipped set; the
loader refuses an
unallowlisted repo without touching the Hub; `trust_remote_code` is
`False` on both the
cache-only and bounded-network calls; every shipped alias still resolves
to its mapping;
the operator env hatch admits only what it names; and, since matching is
case-insensitive, the string handed to the Hub is the allowlist's own
spelling
  rather than the caller's variant.
- `tests/test_no_trust_remote_code.py` — tree-wide scan over every
git-tracked file
  asserting every `trust_remote_code` assignment is `False`.
- `tests/test_huggingface_tokenizer_timeout.py` — the existing GH #1701
timeout tests used
placeholder repo ids (`some/model`, `slow/model`, `offline/model`) which
the allowlist now
rejects before the timeout logic runs, short-circuiting what those tests
exist to prove.
Switched to real shipped repo ids so they still exercise the
bounded-load path.

### Fail-before / pass-after evidence

Method: commit, `git checkout origin/main --
headroom/tokenizers/huggingface.py
benchmarks/comprehensive_eval.py` (source reverted, new tests kept),
run, then
`git checkout HEAD -- ...` to restore.

**Before (source reverted, tests kept) — 10 failed, 4 passed:**

```
tests/test_huggingface_tokenizer_allowlist.py FFFF..F..FFF               [ 85%]
tests/test_no_trust_remote_code.py FF                                    [100%]

=================================== FAILURES ===================================
E   AssertionError: attacker-named repository was handed to from_pretrained
    assert 'attacker-controlled/headroom-rce-poc' not in ['attacker-controlled/headroom-rce-poc']

E   AssertionError: 'attacker-controlled/headroom-rce-poc' escaped the allowlist
    assert 'attacker-controlled/headroom-rce-poc' in frozenset({'01-ai/Yi-1.5-6B', ...})
     +  where 'attacker-controlled/headroom-rce-poc' = get_tokenizer_name('attacker-controlled/headroom-rce-poc')

E   AttributeError: module 'headroom.tokenizers.huggingface' has no attribute 'DEFAULT_TOKENIZER'

E   AssertionError: trust_remote_code must be False everywhere — it executes repository-supplied Python in this process:
      benchmarks/comprehensive_eval.py:44: trust_remote_code=True,
      headroom/tokenizers/huggingface.py:162: trust_remote_code=True,
      headroom/tokenizers/huggingface.py:184: trust_remote_code=True,

E   AssertionError: assert {'True'} == {'False'}

=========================== short test summary info ============================
FAILED tests/test_huggingface_tokenizer_allowlist.py::test_crafted_model_never_reaches_the_loader
FAILED tests/test_huggingface_tokenizer_allowlist.py::test_crafted_model_resolves_to_a_shipped_tokenizer
FAILED tests/test_huggingface_tokenizer_allowlist.py::test_crafted_model_resolves_to_the_declared_default
FAILED tests/test_huggingface_tokenizer_allowlist.py::test_loader_fetches_the_allowlists_own_spelling
FAILED tests/test_huggingface_tokenizer_allowlist.py::test_remote_code_disabled_on_cache_and_network_paths
FAILED tests/test_huggingface_tokenizer_allowlist.py::test_repo_id_matching_is_case_insensitive_but_returns_our_spelling
FAILED tests/test_huggingface_tokenizer_allowlist.py::test_operator_env_allowlist_admits_a_private_repo
FAILED tests/test_huggingface_tokenizer_allowlist.py::test_empty_operator_env_admits_nothing_extra
FAILED tests/test_no_trust_remote_code.py::test_remote_code_is_never_enabled_anywhere_in_the_tree
FAILED tests/test_no_trust_remote_code.py::test_the_tokenizer_loader_sets_the_flag_explicitly
========================= 10 failed, 4 passed in 1.47s =========================
```

**After (source restored):**

```
tests/test_huggingface_tokenizer_allowlist.py ............               [ 54%]
tests/test_no_trust_remote_code.py ..                                    [ 63%]
tests/test_huggingface_tokenizer_timeout.py ........                     [100%]

============================== 22 passed in 2.73s ==============================
```

Wider tokenizer suites (`test_tokenizers.py`, `test_tokenizer.py`,
`test_tokenizers/`,
`test_tokenizer_encoding_resolution.py`,
`test_tokenizer_selection_coverage.py`,
`test_provider_tokenizer_one_ruler.py`,
`test_tokenizer_count_offload.py`,
`test_strands_tokenizer.py`,
`test_compress_route_tokenizer_by_model.py`,
`test_evals_cjk_tokenization.py`): **166 passed, 14 skipped**.

Lint: `ruff 0.16.7 check .` and `format --check .` both clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-23 20:20:56 +02:00

810 lines
26 KiB
Python

#!/usr/bin/env python3
"""
Comprehensive Headroom Evaluation: Real Data, Real Accuracy
This benchmark uses REAL data from established sources:
1. Berkeley Function Calling Leaderboard (BFCL) - Real API schemas and ground truth
2. HotpotQA - Real Wikipedia passages with verified answers
3. Cached OSS data - Real GitHub issues, code, and logs from popular projects
We measure BOTH:
- Compression ratio (token savings)
- Accuracy preservation (ground truth comparison)
Usage:
pip install datasets # For HuggingFace datasets
export ANTHROPIC_API_KEY=sk-ant-...
python benchmarks/comprehensive_eval.py
"""
import json
import os
import time
from dataclasses import dataclass
from pathlib import Path
from typing import Any
# =============================================================================
# DATA LOADERS - Real data from established sources
# =============================================================================
def load_bfcl_samples(n: int = 20) -> list[dict]:
"""
Load real function calling examples from Berkeley Function Calling Leaderboard.
These are REAL API schemas with ground truth function calls.
"""
try:
from datasets import load_dataset
ds = load_dataset(
"gorilla-llm/Berkeley-Function-Calling-Leaderboard",
"BFCL_v3_live_simple",
split="train",
# BFCL is plain parquet; never execute a dataset repository's own
# loading script, here or anywhere else in the tree.
trust_remote_code=False,
)
samples = []
for i, item in enumerate(ds):
if i >= n:
break
samples.append(
{
"id": f"bfcl_{i}",
"type": "function_calling",
"question": item.get("question", [[]])[0][0]["content"]
if item.get("question")
else "",
"functions": item.get("function", []),
"ground_truth": item.get("ground_truth", []),
"source": "BFCL_v3",
}
)
return samples
except Exception as e:
print(f"Warning: Could not load BFCL dataset: {e}")
return []
def load_hotpotqa_samples(n: int = 20) -> list[dict]:
"""
Load real multi-hop QA examples from HotpotQA.
These are REAL Wikipedia passages with verified answers.
"""
try:
from datasets import load_dataset
ds = load_dataset("hotpotqa/hotpot_qa", "fullwiki", split="validation")
samples = []
for i, item in enumerate(ds):
if i >= n:
break
# Build context from supporting facts
context_parts = []
for title, sentences in zip(item["context"]["title"], item["context"]["sentences"]):
context_parts.append(f"## {title}\n" + "\n".join(sentences))
samples.append(
{
"id": f"hotpot_{i}",
"type": "multi_hop_qa",
"question": item["question"],
"context": "\n\n".join(context_parts),
"ground_truth": item["answer"],
"supporting_facts": item["supporting_facts"],
"source": "HotpotQA",
}
)
return samples
except Exception as e:
print(f"Warning: Could not load HotpotQA dataset: {e}")
return []
def load_real_github_data() -> dict:
"""
Load cached real GitHub data from popular OSS projects.
This includes actual issues, PRs, and code from kubernetes, pytorch, etc.
"""
# Cache file for reproducibility
cache_file = Path(__file__).parent / "data" / "github_cache.json"
if cache_file.exists():
with open(cache_file) as f:
return json.load(f)
# If no cache, return sample structure (would fetch from GitHub API in production)
return {
"issues": [],
"code_snippets": [],
"pull_requests": [],
"error_logs": [],
}
def load_real_logs() -> list[dict]:
"""
Load real production log samples.
These are actual log formats from various systems.
"""
# Real log formats from different systems
return [
# Java Spring Boot logs
{
"type": "java_spring",
"content": """2024-01-15 14:23:45.123 ERROR [http-nio-8080-exec-7] c.e.api.UserController - Failed to process request
org.springframework.dao.DataAccessException: Unable to acquire connection from pool
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:82)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:376)
at com.example.api.UserController.getUser(UserController.java:45)
Caused by: java.sql.SQLException: Cannot get a connection, pool error Timeout waiting for idle object
at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1421)
... 42 more""",
},
# Kubernetes events
{
"type": "kubernetes",
"content": """NAMESPACE LAST SEEN TYPE REASON OBJECT MESSAGE
default 2m Warning FailedScheduling pod/nginx-deployment-5d8b9c7f4-x2k9j 0/3 nodes are available: 3 Insufficient memory
default 5m Normal Scheduled pod/redis-master-0 Successfully assigned default/redis-master-0 to node-2
kube-system 1h Warning NodeNotReady node/node-3 Node node-3 status is now: NodeNotReady
default 30s Normal Pulled pod/api-server-7f8d9c8b5-m4n2p Container image "api-server:v2.1.0" already present on machine""",
},
# Python traceback
{
"type": "python_traceback",
"content": """Traceback (most recent call last):
File "/app/services/payment.py", line 127, in process_payment
result = stripe.PaymentIntent.create(
File "/usr/local/lib/python3.11/site-packages/stripe/api_resources/payment_intent.py", line 87, in create
return cls._static_request("post", url, params=params)
File "/usr/local/lib/python3.11/site-packages/stripe/api_requestor.py", line 298, in request
raise error.CardError(error_data.get("message"), error_data.get("param"), error_data.get("code"))
stripe.error.CardError: Your card was declined. This transaction requires authentication.
Request ID: req_a1b2c3d4e5f6g7h8
Error Code: card_declined
Decline Code: authentication_required""",
},
# nginx access logs
{
"type": "nginx_access",
"content": """192.168.1.100 - - [15/Jan/2024:14:30:45 +0000] "GET /api/v2/users/12345 HTTP/1.1" 200 1543 "https://app.example.com/dashboard" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)"
192.168.1.101 - - [15/Jan/2024:14:30:46 +0000] "POST /api/v2/orders HTTP/1.1" 201 892 "https://app.example.com/checkout" "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"
192.168.1.102 - admin [15/Jan/2024:14:30:47 +0000] "DELETE /api/v2/users/67890 HTTP/1.1" 403 124 "-" "curl/7.81.0"
10.0.0.50 - - [15/Jan/2024:14:30:48 +0000] "GET /health HTTP/1.1" 200 15 "-" "kube-probe/1.25" """,
},
]
def load_real_code_samples() -> list[dict]:
"""
Load real code samples from OSS projects.
These are actual implementations, not synthetic examples.
"""
return [
# Real Python - FastAPI auth middleware pattern
{
"language": "python",
"file": "auth/middleware.py",
"source": "FastAPI patterns",
"content": '''"""Authentication middleware for FastAPI applications."""
from datetime import datetime, timedelta
from typing import Optional
import jwt
from fastapi import HTTPException, Security, Depends
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
from pydantic import BaseModel
class TokenPayload(BaseModel):
sub: str
exp: datetime
iat: datetime
scopes: list[str] = []
class JWTBearer(HTTPBearer):
def __init__(self, auto_error: bool = True):
super().__init__(auto_error=auto_error)
async def __call__(self, credentials: HTTPAuthorizationCredentials = Security(HTTPBearer())):
if not credentials:
raise HTTPException(status_code=403, detail="Invalid authorization code")
if credentials.scheme != "Bearer":
raise HTTPException(status_code=403, detail="Invalid authentication scheme")
return self.verify_jwt(credentials.credentials)
def verify_jwt(self, token: str) -> TokenPayload:
try:
payload = jwt.decode(token, SECRET_KEY, algorithms=[ALGORITHM])
return TokenPayload(**payload)
except jwt.ExpiredSignatureError:
raise HTTPException(status_code=401, detail="Token has expired")
except jwt.JWTError:
raise HTTPException(status_code=403, detail="Could not validate credentials")
def create_access_token(subject: str, scopes: list[str] = [], expires_delta: Optional[timedelta] = None):
expire = datetime.utcnow() + (expires_delta or timedelta(minutes=ACCESS_TOKEN_EXPIRE_MINUTES))
to_encode = {"sub": subject, "exp": expire, "iat": datetime.utcnow(), "scopes": scopes}
return jwt.encode(to_encode, SECRET_KEY, algorithm=ALGORITHM)
async def get_current_user(token: TokenPayload = Depends(JWTBearer())) -> dict:
user = await user_service.get_by_id(token.sub)
if not user:
raise HTTPException(status_code=404, detail="User not found")
return user
''',
},
# Real TypeScript - React hook pattern
{
"language": "typescript",
"file": "hooks/useAsync.ts",
"source": "React patterns",
"content": """import { useState, useCallback, useEffect, useRef } from 'react';
interface AsyncState<T> {
data: T | null;
error: Error | null;
loading: boolean;
}
interface UseAsyncOptions {
immediate?: boolean;
onSuccess?: (data: any) => void;
onError?: (error: Error) => void;
}
export function useAsync<T>(
asyncFunction: (...args: any[]) => Promise<T>,
options: UseAsyncOptions = {}
) {
const { immediate = false, onSuccess, onError } = options;
const [state, setState] = useState<AsyncState<T>>({
data: null,
error: null,
loading: immediate,
});
const mountedRef = useRef(true);
const lastCallId = useRef(0);
const execute = useCallback(
async (...args: any[]) => {
const callId = ++lastCallId.current;
setState(prev => ({ ...prev, loading: true, error: null }));
try {
const result = await asyncFunction(...args);
if (mountedRef.current && callId === lastCallId.current) {
setState({ data: result, error: null, loading: false });
onSuccess?.(result);
}
return result;
} catch (error) {
if (mountedRef.current && callId === lastCallId.current) {
const err = error instanceof Error ? error : new Error(String(error));
setState({ data: null, error: err, loading: false });
onError?.(err);
}
throw error;
}
},
[asyncFunction, onSuccess, onError]
);
useEffect(() => {
if (immediate) execute();
return () => { mountedRef.current = false; };
}, []);
return { ...state, execute, reset: () => setState({ data: null, error: null, loading: false }) };
}
""",
},
# Real Go - HTTP middleware pattern
{
"language": "go",
"file": "middleware/ratelimit.go",
"source": "Go patterns",
"content": """package middleware
import (
"net/http"
"sync"
"time"
"golang.org/x/time/rate"
)
type visitor struct {
limiter *rate.Limiter
lastSeen time.Time
}
type RateLimiter struct {
visitors map[string]*visitor
mu sync.RWMutex
rate rate.Limit
burst int
cleanup time.Duration
}
func NewRateLimiter(r rate.Limit, b int) *RateLimiter {
rl := &RateLimiter{
visitors: make(map[string]*visitor),
rate: r,
burst: b,
cleanup: time.Minute * 3,
}
go rl.cleanupVisitors()
return rl
}
func (rl *RateLimiter) getVisitor(ip string) *rate.Limiter {
rl.mu.Lock()
defer rl.mu.Unlock()
v, exists := rl.visitors[ip]
if !exists {
limiter := rate.NewLimiter(rl.rate, rl.burst)
rl.visitors[ip] = &visitor{limiter: limiter, lastSeen: time.Now()}
return limiter
}
v.lastSeen = time.Now()
return v.limiter
}
func (rl *RateLimiter) cleanupVisitors() {
for {
time.Sleep(rl.cleanup)
rl.mu.Lock()
for ip, v := range rl.visitors {
if time.Since(v.lastSeen) > rl.cleanup {
delete(rl.visitors, ip)
}
}
rl.mu.Unlock()
}
}
func (rl *RateLimiter) Limit(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
ip := r.RemoteAddr
limiter := rl.getVisitor(ip)
if !limiter.Allow() {
http.Error(w, "Rate limit exceeded", http.StatusTooManyRequests)
return
}
next.ServeHTTP(w, r)
})
}
""",
},
]
# =============================================================================
# EVALUATION METRICS
# =============================================================================
@dataclass
class AccuracyResult:
"""Ground truth accuracy measurement."""
exact_match: bool
f1_score: float
contains_answer: bool
def compute_f1(prediction: str, ground_truth: str) -> float:
"""Compute token-level F1 score."""
pred_tokens = set(prediction.lower().split())
truth_tokens = set(ground_truth.lower().split())
if not pred_tokens or not truth_tokens:
return 0.0
common = pred_tokens & truth_tokens
if not common:
return 0.0
precision = len(common) / len(pred_tokens)
recall = len(common) / len(truth_tokens)
return 2 * precision * recall / (precision + recall)
def evaluate_answer(prediction: str, ground_truth: str) -> AccuracyResult:
"""Evaluate prediction against ground truth."""
pred_lower = prediction.lower().strip()
truth_lower = ground_truth.lower().strip()
return AccuracyResult(
exact_match=pred_lower == truth_lower,
f1_score=compute_f1(prediction, ground_truth),
contains_answer=truth_lower in pred_lower,
)
# =============================================================================
# MIXED CONTENT SCENARIOS
# =============================================================================
@dataclass
class Scenario:
"""A test scenario with mixed content types."""
name: str
description: str
tool_outputs: list[dict] # Simulated tool outputs
question: str
ground_truth: str | None = None
validation_fn: Any = None # Custom validation function
def create_sre_scenario() -> Scenario:
"""
Real SRE incident scenario with mixed content:
- Kubernetes events (structured)
- Application logs (semi-structured)
- Stack traces (code)
- Metrics JSON (data)
"""
logs = load_real_logs()
return Scenario(
name="SRE Incident Investigation",
description="Debug a production outage using mixed log types",
tool_outputs=[
{
"tool": "get_kubernetes_events",
"result": logs[1]["content"], # K8s events
},
{
"tool": "get_application_logs",
"result": logs[0]["content"], # Java Spring logs
},
{
"tool": "get_error_details",
"result": logs[2]["content"], # Python traceback
},
{
"tool": "get_metrics",
"result": json.dumps(
{
"cpu_percent": [45, 47, 52, 89, 95, 98, 99, 99],
"memory_mb": [2048, 2100, 2200, 3500, 3800, 3950, 4000, 4000],
"request_latency_p99_ms": [50, 55, 60, 250, 800, 1500, 2000, 2500],
"error_rate_percent": [0.1, 0.1, 0.2, 5.0, 15.0, 25.0, 30.0, 35.0],
"timestamps": [
"14:20",
"14:25",
"14:30",
"14:35",
"14:40",
"14:45",
"14:50",
"14:55",
],
},
indent=2,
),
},
],
question="What is the root cause of this outage? What service is affected and what is the specific error?",
ground_truth="connection pool timeout / database connection exhaustion",
validation_fn=lambda r: any(
term in r.lower()
for term in [
"connection pool",
"timeout",
"database",
"pool error",
"acquire connection",
]
),
)
def create_code_review_scenario() -> Scenario:
"""
Real code review scenario with mixed content:
- Actual code (Python, TypeScript, Go)
- Code diff
- Review comments
"""
code_samples = load_real_code_samples()
return Scenario(
name="Code Review Analysis",
description="Review code across multiple languages and identify patterns",
tool_outputs=[
{
"tool": "get_file_contents",
"file": code_samples[0]["file"],
"result": code_samples[0]["content"],
},
{
"tool": "get_file_contents",
"file": code_samples[1]["file"],
"result": code_samples[1]["content"],
},
{
"tool": "get_file_contents",
"file": code_samples[2]["file"],
"result": code_samples[2]["content"],
},
{
"tool": "get_review_comments",
"result": json.dumps(
[
{
"file": "auth/middleware.py",
"line": 25,
"comment": "Should we add rate limiting here?",
},
{
"file": "hooks/useAsync.ts",
"line": 42,
"comment": "Memory leak risk if component unmounts during fetch",
},
{
"file": "middleware/ratelimit.go",
"line": 55,
"comment": "Consider using sync.Map for better concurrent performance",
},
],
indent=2,
),
},
],
question="What authentication patterns are used across these files? Are there any security concerns?",
ground_truth="JWT Bearer token authentication",
validation_fn=lambda r: any(
term in r.lower() for term in ["jwt", "bearer", "token", "authentication"]
),
)
def create_research_scenario(hotpot_samples: list[dict]) -> Scenario | None:
"""
Real research scenario using HotpotQA data.
Multi-hop reasoning with ground truth answers.
"""
if not hotpot_samples:
return None
sample = hotpot_samples[0]
return Scenario(
name="Research Question Answering",
description="Answer multi-hop question from Wikipedia passages",
tool_outputs=[
{
"tool": "search_wikipedia",
"query": sample["question"],
"result": sample["context"],
},
],
question=sample["question"],
ground_truth=sample["ground_truth"],
validation_fn=lambda r: sample["ground_truth"].lower() in r.lower(),
)
# =============================================================================
# MAIN EVALUATION HARNESS
# =============================================================================
@dataclass
class EvalResult:
"""Result from a single evaluation run."""
scenario_name: str
mode: str # "baseline" or "headroom"
tokens_before: int
tokens_after: int
compression_ratio: float
accuracy_preserved: bool
f1_score: float
latency_ms: float
response: str
def run_scenario_with_headroom(
scenario: Scenario,
model_id: str = "claude-sonnet-4-20250514",
) -> tuple[EvalResult, EvalResult]:
"""Run a scenario with and without Headroom, measure accuracy."""
from agno.agent import Agent
from agno.models.anthropic import Claude
from agno.tools import tool
from headroom.integrations.agno import HeadroomAgnoModel
# Create tools that return our scenario data
tool_data = {t["tool"]: t["result"] for t in scenario.tool_outputs}
@tool(name="search_tool")
def search_tool(query: str) -> str:
"""Search for information."""
# Return all tool outputs concatenated (simulating multiple tool calls)
return "\n\n---\n\n".join(tool_data.values())
# Build the full context
full_context = "\n\n---\n\n".join(tool_data.values())
# Estimate tokens (rough)
baseline_tokens = len(full_context) // 4
# Run with Headroom
base_model = Claude(id=model_id)
headroom_model = HeadroomAgnoModel(wrapped_model=base_model)
agent = Agent(model=headroom_model, tools=[search_tool], markdown=True)
prompt = f"""Based on the following information from various tools:
{full_context}
Question: {scenario.question}
Provide a clear, specific answer."""
start = time.time()
response = agent.run(prompt)
response_text = response.content if hasattr(response, "content") else str(response)
latency = (time.time() - start) * 1000
# Get Headroom stats
stats = headroom_model.get_savings_summary()
tokens_after = stats.get("total_tokens_after", baseline_tokens)
tokens_before = stats.get("total_tokens_before", baseline_tokens)
# Evaluate accuracy
if scenario.ground_truth:
accuracy = evaluate_answer(response_text, scenario.ground_truth)
accuracy_preserved = accuracy.contains_answer or accuracy.f1_score > 0.5
f1 = accuracy.f1_score
elif scenario.validation_fn:
accuracy_preserved = scenario.validation_fn(response_text)
f1 = 1.0 if accuracy_preserved else 0.0
else:
accuracy_preserved = True
f1 = 1.0
compression_ratio = (tokens_before - tokens_after) / tokens_before if tokens_before > 0 else 0
baseline_result = EvalResult(
scenario_name=scenario.name,
mode="baseline",
tokens_before=tokens_before,
tokens_after=tokens_before, # No compression for baseline
compression_ratio=0.0,
accuracy_preserved=True, # Baseline is reference
f1_score=1.0,
latency_ms=0, # Not measured for baseline
response="(baseline - not run separately)",
)
headroom_result = EvalResult(
scenario_name=scenario.name,
mode="headroom",
tokens_before=tokens_before,
tokens_after=tokens_after,
compression_ratio=compression_ratio,
accuracy_preserved=accuracy_preserved,
f1_score=f1,
latency_ms=latency,
response=response_text[:500],
)
return baseline_result, headroom_result
def main():
"""Run comprehensive evaluation."""
print("\n" + "=" * 70)
print(" COMPREHENSIVE HEADROOM EVALUATION")
print(" Real Data | Real Accuracy | Mixed Content")
print("=" * 70)
# Check for API key
if not os.environ.get("ANTHROPIC_API_KEY"):
print("\n ERROR: ANTHROPIC_API_KEY environment variable required")
print(" Set it and re-run: export ANTHROPIC_API_KEY=sk-ant-...")
return
# Load real data
print("\n Loading real datasets...")
bfcl_samples = load_bfcl_samples(5)
print(f" BFCL samples: {len(bfcl_samples)}")
hotpot_samples = load_hotpotqa_samples(5)
print(f" HotpotQA samples: {len(hotpot_samples)}")
# Create scenarios
print("\n Creating test scenarios...")
scenarios = [
create_sre_scenario(),
create_code_review_scenario(),
]
research_scenario = create_research_scenario(hotpot_samples)
if research_scenario:
scenarios.append(research_scenario)
print(f" Total scenarios: {len(scenarios)}")
# Run evaluation
results = []
for scenario in scenarios:
print(f"\n Running: {scenario.name}")
print(f" {scenario.description}")
try:
baseline, headroom = run_scenario_with_headroom(scenario)
results.append((baseline, headroom))
print(
f" Tokens: {headroom.tokens_before:,} → {headroom.tokens_after:,} ({headroom.compression_ratio:.1%} saved)"
)
print(f" Accuracy preserved: {'✓' if headroom.accuracy_preserved else '✗'}")
print(f" F1 score: {headroom.f1_score:.2f}")
except Exception as e:
print(f" ERROR: {e}")
# Summary
print("\n" + "=" * 70)
print(" SUMMARY")
print("=" * 70)
if results:
total_before = sum(h.tokens_before for _, h in results)
total_after = sum(h.tokens_after for _, h in results)
avg_compression = (total_before - total_after) / total_before if total_before > 0 else 0
accuracy_rate = sum(1 for _, h in results if h.accuracy_preserved) / len(results)
avg_f1 = sum(h.f1_score for _, h in results) / len(results)
print(f"""
Scenarios tested: {len(results)}
Total tokens before: {total_before:,}
Total tokens after: {total_after:,}
Average compression: {avg_compression:.1%}
Accuracy preserved: {accuracy_rate:.1%}
Average F1 score: {avg_f1:.2f}
""")
# Save results
output = {
"timestamp": time.strftime("%Y-%m-%d %H:%M:%S"),
"scenarios": [
{
"name": h.scenario_name,
"tokens_before": h.tokens_before,
"tokens_after": h.tokens_after,
"compression_ratio": h.compression_ratio,
"accuracy_preserved": h.accuracy_preserved,
"f1_score": h.f1_score,
}
for _, h in results
],
}
output_file = Path(__file__).parent / "comprehensive_eval_results.json"
with open(output_file, "w") as f:
json.dump(output, f, indent=2)
print(f" Results saved to: {output_file}")
print("=" * 70 + "\n")
if __name__ == "__main__":
main()