feat(vertex): add Gemini 3.8 Flash agent benchmark suite (#3371)

## Summary

Adds a reproducible, end-to-end multi-turn agent benchmark suite for
**Gemini 3.8 Flash** on Google Cloud Vertex AI (Gemini Enterprise Agent
Platform) under `examples/vertex_gemini_benchmark/`.

Compares **Direct Vertex AI** vs **Headroom-proxied Vertex AI** across 4
realistic production agent scenarios:
1. **SRE Incident Root Cause Analysis**: 350+ Kubernetes & microservice
logs + Prometheus metrics (identifying connection pool exhaustion,
cluster endpoint, port, and cascading circuit-breaker trips).
2. **Codebase Security Audit & PR Review**: Multi-file code search
results + git diff (identifying insecure `alg: none` and `bypass_auth`
JWT backdoors).
3. **Enterprise BigQuery Analytics**: 500 tabular transaction rows
(computing summaries and isolating fraudulent accounts and high-spend
outliers).
4. **Multi-Turn RAG & System Architecture Synthesis**: 25 dense
technical chunks regarding agent context management contracts.

### Live Benchmark Results (Gemini 3.8 Flash on Vertex AI `global`)

```text
==================================================================================
 📊 FINAL BENCHMARK SUMMARY: GEMINI-3.8-FLASH ON VERTEX AI
==================================================================================
Scenario                           | Baseline Prompt | Headroom Prompt |  Reduction
----------------------------------------------------------------------------------
SRE Incident Root Cause Analysis   |          51,775 |          13,842 |      73.3%
Codebase Security Audit & PR Review |           6,821 |           4,632 |      32.1%
Enterprise BigQuery Analytics & Outlier Detection |          49,020 |           6,120 |      87.5%
Multi-Turn RAG & Architecture Synthesis |           4,071 |           1,437 |      64.7%
----------------------------------------------------------------------------------
TOTAL / AGGREGATE                  |         111,687 |          26,031 |      76.7%

Metric                             |        Baseline |        Headroom | Delta / Impact
----------------------------------------------------------------------------------
Total Prompt Tokens                |         111,687 |          26,031 | -        76.7%
Total All Tokens                   |         116,639 |          29,168 | -        75.0%
Total Cost ($ USD)                 | $       0.09410 | $       0.02849 | -        69.7%
Avg Latency (ms)                   |          8921ms |          5748ms |        +3173ms
Ground Truth Accuracy (Absolute)   |         100.0% |         100.0% | 100.0% Retained
==================================================================================
```

## Real Behavior Proof

• **Environment**: Linux, Python 3.11, live Google Cloud project
(`alanblount-demo`) with Vertex AI enabled, Application Default
Credentials (`gcloud auth application-default login`).
• **Exact command / steps**: `GCP_PROJECT_ID=alanblount-demo python
examples/vertex_gemini_benchmark/benchmark.py --model gemini-3.8-flash`
• **Observed result**: Headroom proxy started cleanly, executed all 4
multi-turn scenarios on `gemini-3.8-flash`, verified ground truth error
causes and anomalies, recorded a **76.7% prompt token reduction** and
**69.7% cost savings**, exported `results.json`, and shut down cleanly.
• **Not tested**: Windows.

## Runtime Rollout Safety

• **Rollout-managed feature(s)**: None; self-contained benchmark suite
under `examples/`.
• **Minimum rollout channel**: standard release.
• **Stable/default behavior changed**: None; purely adds benchmark
tooling and documentation under `examples/`.
• **Kill switch / disable path**: N/A.
• **Unsafe override required**: None.
• **Qualification impact**: None.
• **Rollback path**: Revert PR.

## Review Readiness

[✓] I have performed a self-review
[✓] This PR is ready for human review

## Checklist

[✓] My code follows the project's style guidelines
[✓] I have performed a self-review of my code
[✓] I have commented my code, particularly in hard-to-understand areas
[✓] I have made corresponding changes to the documentation
[✓] My changes generate no new warnings
[✓] I did not edit `CHANGELOG.md` — it is generated by release-please
from my Conventional Commit PR title (a CI guard enforces this)

## Additional Notes

• Companion to PR #2640 (which adds in-band Vertex diagnostic hints and
fixes).
This commit is contained in:
alan blount
2026-09-06 11:32:19 +08:00
committed by GitHub
parent ee6f9db25b
commit e67b3c8a29
7 changed files with 1257 additions and 0 deletions
+1
View File
@@ -206,6 +206,7 @@ temp/
.benchmarks/
benchmark_results.json
benchmark_results/
examples/vertex_gemini_benchmark/results.json
# DeepEval cache
.deepeval/
+15
View File
@@ -97,6 +97,21 @@ export OPENAI_API_KEY='your-key'
PYTHONPATH=. python -m examples.mcp_demo.run_agent_eval
```
### vertex_gemini_benchmark/
Gemini 3.8 Flash on Google Cloud Vertex AI (Gemini Enterprise Agent Platform) benchmark:
```bash
# Ensure Google Cloud ADC is active
gcloud auth application-default login
export GCP_PROJECT_ID="your-project-id"
# Run comparative benchmark (Direct vs Headroom Proxied)
python examples/vertex_gemini_benchmark/benchmark.py --model gemini-3.8-flash
```
See [vertex_gemini_benchmark/README.md](vertex_gemini_benchmark/README.md) for full benchmark scenarios and methodology.
### strands_bedrock_demo.py
AWS Strands Agents + Bedrock integration demo. Showcases two Headroom integration patterns:
+163
View File
@@ -0,0 +1,163 @@
# Gemini 3.8 Flash on Google Cloud Vertex AI + Headroom Benchmark
A reproducible, real-world benchmark evaluating **Headroom context compression** with **Gemini 3.8 Flash** on **Google Cloud Vertex AI (Gemini Enterprise Agent Platform)**.
---
## 🎯 Why This Matters
When building production agents (coding assistants, SRE incident responders, data analysts, multi-agent frameworks) on Vertex AI, multi-turn tool loops cause **rapid context explosion**:
* **Container & Kubernetes logs** dump hundreds of lines of noise for a single stack trace.
* **Code search & file trees** inflate prompts with repetitive schema structures.
* **Database queries** return large tabular results where only outliers and aggregations matter.
Even with Gemini 3.8 Flash's massive 1M-token context window and fast inference, bloated tool returns:
1. **Drive up inference spend** as conversation histories compound across turns.
2. **Increase Time to First Token (TTFT)** due to large prompt prefill processing.
3. **Dilute attention**, making needle-in-a-haystack reasoning and anomaly isolation harder.
**Headroom** acts as an intelligent, transparent proxy (or in-process SDK layer) that compresses JSON arrays, structured logs, and tables by **40–85%** while strictly preserving schema anchors, recent turns, anomalies, error traces, and ground truth accuracy.
---
## 📊 Live Benchmark Results
Tested live on **Google Cloud Vertex AI** (`global` endpoint) with **`gemini-3.8-flash`**.
Pricing basis: Google Cloud Vertex AI introductory standard rates through 2026-12-31 ($0.75 per 1M prompt tokens, $3.75 per 1M text output tokens; cached input rate is $0.075/M).
| Scenario | Workload Category | Baseline Prompt | Headroom Prompt | Token Reduction | Latency Delta | Baseline Accuracy | Headroom Accuracy | Relative Retention |
| :--- | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **SRE Incident Root Cause** | Kubernetes & Microservice Logs | 51,775 | 13,842 | **-73.3%** | +9.2s faster | 100.0% | 100.0% | **100.0%** (✓ PASS) |
| **Security Audit & PR Review** | Code Search & Git Diffs | 6,821 | 4,632 | **-32.1%** | +1.9s faster | 100.0% | 100.0% | **100.0%** (✓ PASS) |
| **BigQuery Table Analytics** | 500 Tabular Transaction Rows | 49,020 | 6,120 | **-87.5%** | +2.3s faster | 100.0% | 100.0% | **100.0%** (✓ PASS) |
| **Multi-Turn RAG Synthesis** | 25 Dense Specification Chunks | 4,071 | 1,437 | **-64.7%** | -0.7s | 100.0% | 100.0% | **100.0%** (✓ PASS) |
| **TOTAL / AGGREGATE** | **Real-World Agent Trajectory** | **111,687** | **26,031** | **-76.7%** | **+3.2s avg faster** | **100.0%** | **100.0%** | **100.0% Retained** |
### Key Metrics Summary
* **Prompt Tokens Saved**: **85,656 tokens** (**76.7% net reduction**, 111,687 down to 26,031)
* **Total Inference Cost**: **$0.0941 down to $0.0285** (**69.7% cost savings** at standard Vertex rates)
* **Average Latency**: **8.9s down to 5.7s** (**+3.2s faster roundtrip** due to reduced prompt prefill load)
* **Relative Quality Retention**: **100.0%** (zero degradation; Headroom matched baseline extraction of all root causes, database connection pool exhaustion, JWT `alg: none` auth bypass, fraud outliers, and architectural contracts)
---
## 🏗️ Architecture
```
┌───────────────────────────────────────┐
│ Google Cloud Vertex AI │
│ (Gemini Enterprise Agent Platform) │
│ │
┌───────────────────────┐ │ ┌─────────────────────────────────┐ │
│ google-genai Python │ │ │ gemini-3.8-flash │ │
│ SDK Agent / Script │ │ └─────────────────────────────────┘ │
└───────────┬───────────┘ └───────────────────▲───────────────────┘
│ │
│ POST /v1/projects/.../publishers/... │ Compressed
│ (base_url = http://127.0.0.1:8787) │ Payload
▼ │
┌─────────────────────────────────────────────────────────────┴───────────────────┐
│ Headroom Proxy (:8787) │
│ │
│ ┌───────────────────────┐ ┌────────────────────────┐ ┌──────────────────┐ │
│ │ ContentRouter │──▶│ SmartCrusher │──▶│ LogCompressor │ │
│ │ (Format & Role Sieve) │ │ (JSON Array Compactor) │ │ (Error Anchor) │ │
│ └───────────────────────┘ └────────────────────────┘ └──────────────────┘ │
│ │
│ • Preserves Google ADC Bearer Auth Tokens │
│ • Compresses verbose tool arrays & tables │
│ • Preserves 100% of anomalies, errors, and schema anchors │
└─────────────────────────────────────────────────────────────────────────────────┘
```
---
## 🚀 How to Run the Benchmark
### 1. Prerequisites
Ensure you have Google Cloud Application Default Credentials (ADC) configured:
```bash
gcloud auth application-default login
export GCP_PROJECT_ID=$(gcloud config get-value project)
```
Install required dependencies:
```bash
pip install "headroom-ai[proxy]" google-genai
```
### 2. Execute Benchmark
Run the full comparative suite:
```bash
python examples/vertex_gemini_benchmark/benchmark.py --model gemini-3.8-flash
```
#### CLI Options
```text
--project GCP Project ID (defaults to $GCP_PROJECT_ID or gcloud default)
--location Vertex AI location (default: global)
--model Vertex model ID (default: gemini-3.8-flash)
--port Headroom local proxy port (default: 8787)
--thinking-budget Thinking token budget in tokens (default: 0 = standard inference)
--output-json Output file for JSON metrics (default: examples/vertex_gemini_benchmark/results.json)
--social / --no-social Print formatted social media proof point summary (default: on)
```
---
## 💡 Using Headroom with Vertex AI in Your Agent Code
Connecting your `google-genai` agent to Headroom requires **one line** (`http_options`):
```python
from google import genai
# Point the standard SDK at the Headroom proxy
client = genai.Client(
vertexai=True,
project="your-gcp-project-id",
location="global",
http_options={"base_url": "http://127.0.0.1:8787"},
)
response = client.models.generate_content(
model="gemini-3.8-flash",
contents=[
"You are an SRE agent.",
f"Analyze these Kubernetes logs:\n{verbose_json_logs}",
],
)
print(response.text)
```
---
## 📢 Social Post / Proof Point Card
```markdown
🚀 Headroom + Gemini 3.8 Flash on Google Cloud Vertex AI Benchmark
When AI agents run complex multi-turn workflows (SRE debugging, PR reviews, BigQuery analytics), tool output bloat explodes prompt token costs and degrades TTFT.
We ran reproducible end-to-end agent benchmarks comparing Direct Vertex AI vs Headroom-Proxied Vertex AI on gemini-3.8-flash:
📉 Results:
• Prompt Token Reduction: 76.7% (111,687 ➔ 26,031 tokens)
• SRE Log Scenario Reduction: 73.3% (51.7k ➔ 13.8k tokens)
• BigQuery Analytics Scenario: 87.5% (49.0k ➔ 6.1k tokens)
• Total Cost Savings: 69.7% ($0.0941 ➔ $0.0285 at standard Vertex rates)
• Relative Quality Retention: 100.0% (Zero reasoning degradation; 100.0% ground truth retention in both arms)
• Zero Code Changes: Point google-genai SDK http_options.base_url to http://127.0.0.1:8787.
🔗 Full benchmark suite, reproducible scenarios, and code:
https://github.com/headroomlabs-ai/headroom/tree/main/examples/vertex_gemini_benchmark
```
@@ -0,0 +1 @@
"""Vertex AI Gemini Agent Benchmark with Headroom."""
+665
View File
@@ -0,0 +1,665 @@
#!/usr/bin/env python3
"""Vertex AI (Gemini Enterprise Agent Platform) + Headroom Context Compression Benchmark.
Evaluates Gemini 3.8 Flash on Vertex AI across realistic multi-turn agent workloads:
1. Baseline: Direct Vertex AI API calls with raw, uncompressed tool outputs.
2. Headroom: Transparently proxied Vertex AI API calls with intelligent context compression.
Measures:
- Input / Prompt Token Reduction (%)
- Roundtrip Latency & TTFT (ms)
- Total Inference Cost Savings ($ USD)
- Absolute Ground Truth Accuracy (% of expected facts/anomalies identified)
- Relative Quality Retention (% of Baseline accuracy preserved after compression)
"""
from __future__ import annotations
import argparse
import json
import os
import subprocess
import sys
import time
import urllib.error
import urllib.request
from contextlib import suppress
from dataclasses import asdict, dataclass
from pathlib import Path
from typing import Any
# Ensure local package import works
REPO_ROOT = Path(__file__).resolve().parent.parent.parent
if str(REPO_ROOT) not in sys.path:
sys.path.insert(0, str(REPO_ROOT))
try:
from google import genai
from google.genai import types
except ImportError:
print("Error: google-genai SDK is not installed.", file=sys.stderr)
print("Run `pip install google-genai` and try again.", file=sys.stderr)
sys.exit(1)
from examples.vertex_gemini_benchmark.scenarios import ( # noqa: E402
BenchmarkScenario,
get_all_scenarios,
)
# ----------------------------------------------------------------------------
# Constants & Pricing
# ----------------------------------------------------------------------------
DEFAULT_MODEL = "gemini-3.8-flash"
DEFAULT_LOCATION = "global"
DEFAULT_PORT = 8787
# Vertex AI Gemini 3.8 Flash Introductory Standard Pricing (through 2026-12-31):
# - Input tokens: $0.75 per 1M un-cached prompt tokens
# - Text output tokens: $3.75 per 1M candidate tokens
# Source: Google Cloud Vertex AI Pricing documentation (Introductory rates through Dec 31, 2026)
INPUT_PRICE_PER_M = 0.75
OUTPUT_PRICE_PER_M = 3.75
@dataclass
class TrialResult:
scenario_name: str
category: str
mode: str # "Baseline (Direct)" or "Headroom (Optimized)"
prompt_tokens: int
candidate_tokens: int
total_tokens: int
latency_ms: float
cost_usd: float
accuracy_score: float # Absolute ground truth score [0.0 - 1.0]
response_text: str
facts_found: list[str]
anomalies_found: list[str]
@dataclass
class ScenarioComparison:
scenario_name: str
category: str
baseline: TrialResult
headroom: TrialResult
token_savings_pct: float
cost_savings_pct: float
latency_delta_ms: float
quality_retained: bool
relative_accuracy_retention_pct: float
# ----------------------------------------------------------------------------
# Proxy Lifecycle Management
# ----------------------------------------------------------------------------
def start_headroom_proxy(
port: int, region: str, savings_profile: str = "agent-90"
) -> subprocess.Popen[bytes]:
"""Start Headroom proxy as a background process configured for agent compression."""
env = os.environ.copy()
env.setdefault("HEADROOM_LOG", "INFO")
env["HEADROOM_SAVINGS_PROFILE"] = savings_profile
env["HEADROOM_COMPRESS_USER_MESSAGES"] = "1"
env["HEADROOM_NO_MEMORY_TOOLS"] = "1"
env["HEADROOM_NO_MEMORY_CONTEXT"] = "1"
cmd = [
sys.executable,
"-m",
"headroom.cli",
"proxy",
"--backend",
"vertex",
"--region",
region,
"--port",
str(port),
"--no-ccr",
"--no-memory-tools",
"--no-memory-context",
]
log_path = Path("/tmp") / f"headroom_benchmark_proxy_{port}.log"
log_file = log_path.open("wb")
proc = subprocess.Popen(
cmd,
env=env,
stdout=log_file,
stderr=subprocess.STDOUT,
)
return proc
def wait_for_proxy_ready(port: int, timeout_s: float = 35.0) -> None:
"""Poll proxy /readyz until ready."""
url = f"http://127.0.0.1:{port}/readyz"
deadline = time.time() + timeout_s
last_err: Exception | None = None
while time.time() < deadline:
try:
with urllib.request.urlopen(url, timeout=1) as resp:
if resp.status == 200:
return
except (urllib.error.URLError, ConnectionError, TimeoutError) as e:
last_err = e
time.sleep(0.4)
raise RuntimeError(
f"Proxy on port {port} failed to become ready within {timeout_s}s: {last_err}"
)
def stop_proxy(proc: subprocess.Popen[bytes]) -> None:
"""Politely terminate proxy process."""
with suppress(ProcessLookupError):
proc.terminate()
try:
proc.wait(timeout=5)
except subprocess.TimeoutExpired:
proc.kill()
proc.wait(timeout=5)
# ----------------------------------------------------------------------------
# Execution & Scoring
# ----------------------------------------------------------------------------
def build_gemini_contents(scenario: BenchmarkScenario) -> list[types.Content]:
"""Format multi-turn tool interaction into realistic Gemini Content turns."""
contents: list[types.Content] = []
# 1. User query
contents.append(
types.Content(
role="user",
parts=[types.Part.from_text(text=scenario.user_query)],
)
)
# 2. Simulated tool executions and output responses
for tool_out in scenario.tool_outputs:
tool_name = tool_out["tool"]
raw_result = json.dumps(tool_out["result"], indent=2)
# Model turn announcing tool execution
contents.append(
types.Content(
role="model",
parts=[
types.Part.from_text(
text=f"Executing tool `{tool_name}` to retrieve relevant context..."
)
],
)
)
# Tool result returned to model in user context turn
contents.append(
types.Content(
role="user",
parts=[
types.Part.from_text(
text=f"Tool `{tool_name}` output:\n```json\n{raw_result}\n```"
)
],
)
)
# 3. Final instruction turn to trigger synthesis
contents.append(
types.Content(
role="user",
parts=[
types.Part.from_text(
text="Synthesize findings from all tool outputs above. Provide a concise, highly specific response citing all exact root causes, service names, error messages, identifiers, and anomalies."
)
],
)
)
return contents
def evaluate_response_quality(
response_text: str, scenario: BenchmarkScenario
) -> tuple[float, list[str], list[str]]:
"""Evaluate whether the model response accurately captured all ground truth facts & anomalies."""
text_lower = response_text.lower()
text_clean = text_lower.replace("$", "").replace(",", "").replace("-", " ")
def _matches(needle: str) -> bool:
n = needle.lower()
if n in text_lower:
return True
n_clean = n.replace("$", "").replace(",", "").replace("-", " ")
if n_clean in text_clean:
return True
return False
facts_found = [fact for fact in scenario.expected_facts if _matches(fact)]
anomalies_found = [anom for anom in scenario.expected_anomalies if _matches(anom)]
total_expected = len(scenario.expected_facts) + len(scenario.expected_anomalies)
if total_expected == 0:
return 1.0, facts_found, anomalies_found
total_found = len(facts_found) + len(anomalies_found)
score = total_found / total_expected
return score, facts_found, anomalies_found
def calculate_cost(prompt_tokens: int, candidate_tokens: int) -> float:
"""Calculate inference cost in USD for Gemini 3.8 Flash on Vertex."""
return (prompt_tokens * INPUT_PRICE_PER_M + candidate_tokens * OUTPUT_PRICE_PER_M) / 1_000_000.0
def run_trial(
client: genai.Client,
scenario: BenchmarkScenario,
model: str,
mode_name: str,
thinking_budget: int = 0,
) -> TrialResult:
"""Run a single benchmark trial against Gemini on Vertex."""
contents = build_gemini_contents(scenario)
config_kwargs: dict[str, Any] = {
"system_instruction": scenario.system_prompt,
"temperature": 0.1, # Low temperature for deterministic evaluation
}
if thinking_budget > 0:
config_kwargs["thinking_config"] = types.ThinkingConfig(thinking_budget=thinking_budget)
config = types.GenerateContentConfig(**config_kwargs)
start_time = time.perf_counter()
response = client.models.generate_content(
model=model,
contents=contents,
config=config,
)
elapsed_ms = (time.perf_counter() - start_time) * 1000.0
usage = getattr(response, "usage_metadata", None)
prompt_tokens = getattr(usage, "prompt_token_count", 0) if usage else 0
candidate_tokens = getattr(usage, "candidates_token_count", 0) if usage else 0
total_tokens = (
getattr(usage, "total_token_count", prompt_tokens + candidate_tokens) if usage else 0
)
response_text = response.text or ""
score, facts_found, anomalies_found = evaluate_response_quality(response_text, scenario)
cost = calculate_cost(prompt_tokens, candidate_tokens)
return TrialResult(
scenario_name=scenario.name,
category=scenario.category,
mode=mode_name,
prompt_tokens=prompt_tokens,
candidate_tokens=candidate_tokens,
total_tokens=total_tokens,
latency_ms=elapsed_ms,
cost_usd=cost,
accuracy_score=score,
response_text=response_text,
facts_found=facts_found,
anomalies_found=anomalies_found,
)
# ----------------------------------------------------------------------------
# Benchmark Suite Runner
# ----------------------------------------------------------------------------
def run_benchmark_suite(
project_id: str,
location: str = DEFAULT_LOCATION,
model: str = DEFAULT_MODEL,
port: int = DEFAULT_PORT,
thinking_budget: int = 0,
output_json: str | None = None,
social_format: bool = False,
) -> dict[str, Any]:
"""Execute the complete comparative benchmark suite."""
print("=" * 82)
print(" 🚀 VERTEX AI (GEMINI ENTERPRISE AGENT PLATFORM) + HEADROOM BENCHMARK")
print("=" * 82)
print(f" Model: {model}")
print(f" Location: {location}")
print(f" Project: {project_id}")
print(f" Proxy: http://127.0.0.1:{port}")
print(
f" Pricing: ${INPUT_PRICE_PER_M}/M input, ${OUTPUT_PRICE_PER_M}/M output (Introductory rate)"
)
if thinking_budget > 0:
print(f" Thinking: budget={thinking_budget} tokens")
print("=" * 82)
scenarios = get_all_scenarios()
comparisons: list[ScenarioComparison] = []
# Clean any stale proxy processes on this port
try:
subprocess.run(
["pkill", "-f", f"headroom.cli proxy.*{port}"],
check=False,
capture_output=True,
)
time.sleep(0.5)
except Exception:
pass
# 1. Start Headroom Proxy
print("\n[1/4] Spawning Headroom compression proxy ...")
proxy_proc = start_headroom_proxy(port=port, region=location)
try:
wait_for_proxy_ready(port=port, timeout_s=40.0)
print(" ✓ Headroom proxy is active and ready.\n")
# 2. Build Clients
client_baseline = genai.Client(
vertexai=True,
project=project_id,
location=location,
)
client_headroom = genai.Client(
vertexai=True,
project=project_id,
location=location,
http_options={"base_url": f"http://127.0.0.1:{port}"},
)
print("[2/4] Executing Benchmark Scenarios ...\n")
for idx, scenario in enumerate(scenarios, 1):
raw_kb = scenario.total_raw_chars() / 1024.0
print(f"Scenario [{idx}/{len(scenarios)}]: {scenario.name} ({scenario.category})")
print(
f" Payload: ~{raw_kb:.1f} KB raw tool outputs across {len(scenario.tool_outputs)} calls"
)
# Run Baseline
print(" ↳ Running Baseline (Direct Vertex AI) ...", end="", flush=True)
baseline_res = run_trial(
client_baseline,
scenario,
model=model,
mode_name="Baseline (Direct)",
thinking_budget=thinking_budget,
)
print(
f" done ({baseline_res.prompt_tokens:,} prompt tokens, {baseline_res.latency_ms:.0f}ms)"
)
# Run Headroom
print(" ↳ Running Headroom (Context Compressed) ...", end="", flush=True)
headroom_res = run_trial(
client_headroom,
scenario,
model=model,
mode_name="Headroom (Optimized)",
thinking_budget=thinking_budget,
)
print(
f" done ({headroom_res.prompt_tokens:,} prompt tokens, {headroom_res.latency_ms:.0f}ms)"
)
# Calculate savings
token_savings = (
(1.0 - headroom_res.prompt_tokens / baseline_res.prompt_tokens) * 100.0
if baseline_res.prompt_tokens > 0
else 0.0
)
cost_savings = (
(1.0 - headroom_res.cost_usd / baseline_res.cost_usd) * 100.0
if baseline_res.cost_usd > 0
else 0.0
)
latency_delta = baseline_res.latency_ms - headroom_res.latency_ms
quality_ok = headroom_res.accuracy_score >= baseline_res.accuracy_score * 0.90
relative_retention = (
(headroom_res.accuracy_score / baseline_res.accuracy_score) * 100.0
if baseline_res.accuracy_score > 0
else 100.0
)
print(
f" 📊 Results: Prompt Tokens: {baseline_res.prompt_tokens:,} -> {headroom_res.prompt_tokens:,} (-{token_savings:.1f}%)"
)
print(
f" Latency: {baseline_res.latency_ms:.0f}ms -> {headroom_res.latency_ms:.0f}ms ({latency_delta:+.0f}ms)"
)
print(
f" Accuracy: Baseline={baseline_res.accuracy_score:.1%}, Headroom={headroom_res.accuracy_score:.1%} (Retention: {relative_retention:.1f}%, {'✓ PASS' if quality_ok else '✗ DEGRADED'})\n"
)
comparisons.append(
ScenarioComparison(
scenario_name=scenario.name,
category=scenario.category,
baseline=baseline_res,
headroom=headroom_res,
token_savings_pct=token_savings,
cost_savings_pct=cost_savings,
latency_delta_ms=latency_delta,
quality_retained=quality_ok,
relative_accuracy_retention_pct=relative_retention,
)
)
finally:
print("[3/4] Shutting down Headroom proxy ...")
stop_proxy(proxy_proc)
print(" ✓ Proxy shut down cleanly.\n")
# 3. Overall Summary Calculations
total_baseline_prompt = sum(c.baseline.prompt_tokens for c in comparisons)
total_headroom_prompt = sum(c.headroom.prompt_tokens for c in comparisons)
total_baseline_tokens = sum(c.baseline.total_tokens for c in comparisons)
total_headroom_tokens = sum(c.headroom.total_tokens for c in comparisons)
total_baseline_cost = sum(c.baseline.cost_usd for c in comparisons)
total_headroom_cost = sum(c.headroom.cost_usd for c in comparisons)
avg_baseline_latency = sum(c.baseline.latency_ms for c in comparisons) / len(comparisons)
avg_headroom_latency = sum(c.headroom.latency_ms for c in comparisons) / len(comparisons)
avg_baseline_acc = sum(c.baseline.accuracy_score for c in comparisons) / len(comparisons)
avg_headroom_acc = sum(c.headroom.accuracy_score for c in comparisons) / len(comparisons)
overall_token_savings = (
(1.0 - total_headroom_prompt / total_baseline_prompt) * 100.0
if total_baseline_prompt > 0
else 0.0
)
overall_cost_savings = (
(1.0 - total_headroom_cost / total_baseline_cost) * 100.0
if total_baseline_cost > 0
else 0.0
)
overall_relative_retention = (
(avg_headroom_acc / avg_baseline_acc * 100.0) if avg_baseline_acc > 0 else 100.0
)
retention_label = (
"100.0% Retained"
if overall_relative_retention >= 99.9
else f"{overall_relative_retention:.1f}% Retained"
)
# 4. Print Formatted Table
print("=" * 82)
print(f" 📊 FINAL BENCHMARK SUMMARY: {model.upper()} ON VERTEX AI")
print("=" * 82)
print(
f"{'Scenario':<34} | {'Baseline Prompt':>15} | {'Headroom Prompt':>15} | {'Reduction':>10}"
)
print("-" * 82)
for c in comparisons:
print(
f"{c.scenario_name:<34} | {c.baseline.prompt_tokens:>15,} | {c.headroom.prompt_tokens:>15,} | {c.token_savings_pct:>9.1f}%"
)
print("-" * 82)
print(
f"{'TOTAL / AGGREGATE':<34} | {total_baseline_prompt:>15,} | {total_headroom_prompt:>15,} | {overall_token_savings:>9.1f}%\n"
)
print(f"{'Metric':<34} | {'Baseline':>15} | {'Headroom':>15} | {'Delta / Impact':>14}")
print("-" * 82)
print(
f"{'Total Prompt Tokens':<34} | {total_baseline_prompt:>15,} | {total_headroom_prompt:>15,} | -{overall_token_savings:>12.1f}%"
)
print(
f"{'Total All Tokens':<34} | {total_baseline_tokens:>15,} | {total_headroom_tokens:>15,} | -{(1 - total_headroom_tokens / total_baseline_tokens) * 100:>12.1f}%"
)
print(
f"{'Total Cost ($ USD)':<34} | ${total_baseline_cost:>14.5f} | ${total_headroom_cost:>14.5f} | -{overall_cost_savings:>12.1f}%"
)
print(
f"{'Avg Latency (ms)':<34} | {avg_baseline_latency:>13.0f}ms | {avg_headroom_latency:>13.0f}ms | {avg_baseline_latency - avg_headroom_latency:>+12.0f}ms"
)
print(
f"{'Ground Truth Accuracy (Absolute)':<34} | {avg_baseline_acc:>14.1%} | {avg_headroom_acc:>14.1%} | {retention_label:>14}"
)
print("=" * 82)
# 5. Social Post Format
social_text = f"""
🚀 **Headroom + {model} on Google Cloud Vertex AI Benchmark**
When AI agents run complex multi-turn workflows (SRE debugging, PR reviews, BigQuery analytics), tool output bloat explodes prompt token costs and degrades TTFT.
We ran reproducible end-to-end agent benchmarks comparing **Direct Vertex AI** vs **Headroom-Proxied Vertex AI** on `{model}`:
📉 **Results**:
• **Prompt Token Reduction**: **{overall_token_savings:.1f}%** ({total_baseline_prompt:,} ➔ {total_headroom_prompt:,} tokens)
• **Total Cost Savings**: **{overall_cost_savings:.1f}%** (${total_baseline_cost:.4f} ➔ ${total_headroom_cost:.4f})
• **Relative Quality Retention**: **{overall_relative_retention:.1f}%** ({avg_headroom_acc:.1%} Headroom vs {avg_baseline_acc:.1%} Baseline ground truth score)
• **Zero Code Changes**: Point `google-genai` SDK `http_options.base_url` to `http://127.0.0.1:{port}`.
🔗 Full benchmark suite, reproducible scenarios, and code:
https://github.com/headroomlabs-ai/headroom/tree/main/examples/vertex_gemini_benchmark
"""
if social_format:
print("\n" + "=" * 82)
print(" 📢 DEVELOPER SOCIAL POST PROOF POINT")
print("=" * 82)
print(social_text.strip())
print("=" * 82 + "\n")
# 6. JSON Export
result_data = {
"metadata": {
"model": model,
"location": location,
"project_id": project_id,
"timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
"thinking_budget": thinking_budget,
"pricing_source": (
"Google Cloud Vertex AI introductory standard pricing through 2026-12-31: "
f"${INPUT_PRICE_PER_M}/M input tokens, ${OUTPUT_PRICE_PER_M}/M text output tokens."
),
},
"aggregate": {
"total_baseline_prompt_tokens": total_baseline_prompt,
"total_headroom_prompt_tokens": total_headroom_prompt,
"overall_token_savings_pct": overall_token_savings,
"total_baseline_cost_usd": total_baseline_cost,
"total_headroom_cost_usd": total_headroom_cost,
"overall_cost_savings_pct": overall_cost_savings,
"avg_baseline_latency_ms": avg_baseline_latency,
"avg_headroom_latency_ms": avg_headroom_latency,
"avg_baseline_accuracy": avg_baseline_acc,
"avg_headroom_accuracy": avg_headroom_acc,
"overall_relative_retention_pct": overall_relative_retention,
},
"scenarios": [asdict(c) for c in comparisons],
"social_proof_point": social_text.strip(),
}
if output_json:
with open(output_json, "w", encoding="utf-8") as f:
json.dump(result_data, f, indent=2)
print(f"✓ Detailed benchmark results exported to: {output_json}")
return result_data
def main() -> int:
parser = argparse.ArgumentParser(
description="Gemini 3.8 Flash on Vertex AI + Headroom Benchmark"
)
parser.add_argument(
"--project", default=os.environ.get("GCP_PROJECT_ID"), help="GCP Project ID"
)
parser.add_argument(
"--location", default=DEFAULT_LOCATION, help="Vertex location (default: global)"
)
parser.add_argument(
"--model", default=DEFAULT_MODEL, help="Model ID (default: gemini-3.8-flash)"
)
parser.add_argument(
"--port",
type=int,
default=DEFAULT_PORT,
help="Headroom proxy port (default: 8787)",
)
parser.add_argument(
"--thinking-budget",
type=int,
default=0,
help="Thinking token budget (0 = disabled)",
)
parser.add_argument(
"--output-json",
default="examples/vertex_gemini_benchmark/results.json",
help="Path to save output JSON",
)
parser.add_argument(
"--social",
action=argparse.BooleanOptionalAction,
default=True,
help="Print social post text",
)
args = parser.parse_args()
project_id = args.project
if not project_id:
try:
cmd_out = subprocess.check_output(
["gcloud", "config", "get-value", "project"], text=True
).strip()
if cmd_out:
project_id = cmd_out
except Exception:
pass
if not project_id:
print(
"Error: GCP_PROJECT_ID is not set. Specify via --project or set GCP_PROJECT_ID.",
file=sys.stderr,
)
return 1
try:
run_benchmark_suite(
project_id=project_id,
location=args.location,
model=args.model,
port=args.port,
thinking_budget=args.thinking_budget,
output_json=args.output_json,
social_format=args.social,
)
return 0
except Exception as e:
print(f"\n❌ Benchmark failed with error: {e}", file=sys.stderr)
import traceback
traceback.print_exc()
return 2
if __name__ == "__main__":
sys.exit(main())
@@ -0,0 +1,412 @@
"""Realistic agent scenarios with verbose tool outputs for benchmarking Gemini on Vertex AI.
Simulates actual tool-use patterns in production developer and SRE agents:
1. SRE Incident Investigation (Microservice logs, metrics, k8s events)
2. Codebase Security & PR Audit (File tree, code search matches, git diff)
3. Enterprise Analytics & Anomaly Detection (500 database records with anomalies)
4. Multi-turn RAG & Architecture Synthesis (Complex system docs and API schemas)
"""
from __future__ import annotations
import hashlib
import json
import random
from dataclasses import dataclass, field
from typing import Any
@dataclass
class BenchmarkScenario:
"""A realistic multi-turn agent scenario."""
name: str
category: str
description: str
system_prompt: str
user_query: str
tool_outputs: list[dict[str, Any]]
expected_facts: list[str] = field(default_factory=list)
expected_anomalies: list[str] = field(default_factory=list)
def total_raw_chars(self) -> int:
return sum(len(json.dumps(t["result"])) for t in self.tool_outputs)
# ----------------------------------------------------------------------------
# 1. SRE Incident Investigation
# ----------------------------------------------------------------------------
def generate_sre_logs(num_entries: int = 350) -> dict[str, Any]:
"""Generate realistic microservice logs with a buried critical cascade failure."""
random.seed(42)
services = [
"api-gateway",
"auth-service",
"payment-service",
"order-service",
"notification-service",
]
log_levels = ["INFO", "INFO", "INFO", "INFO", "WARN", "DEBUG"]
entries: list[dict[str, Any]] = []
for i in range(num_entries):
svc = random.choice(services)
lvl = random.choice(log_levels)
ts = f"2026-08-31T01:{i // 60:02d}:{i % 60:02d}.{random.randint(100, 999)}Z"
trace_id = hashlib.md5(f"trace-{i}".encode()).hexdigest()[:16]
if i == 142:
lvl = "ERROR"
svc = "payment-service"
msg = "FATAL: Connection pool exhausted for Postgres cluster pg-primary.db.internal:5432. Active: 100/100, Queue: 580 waiters, Timeout: 30000ms."
elif i == 143:
lvl = "ERROR"
svc = "payment-service"
msg = "Unhandled exception: ConnectionPoolExhaustedError: Unable to acquire connection within 30000ms at PaymentProcessor.execute (payment_gateway.py:314)"
elif i == 188:
lvl = "WARN"
svc = "api-gateway"
msg = "Upstream 504 Gateway Timeout from http://payment-service:8080/v2/charge (latency: 30004ms). Circuit breaker trip threshold reached (failure_rate=82%)."
elif lvl == "WARN":
msg = f"High thread pool utilization on {svc} (88% threshold reached)"
else:
msg = f"Handled HTTP request 200 OK for /{svc}/v1/healthcheck [latency={random.randint(2, 45)}ms]"
entries.append(
{
"timestamp": ts,
"level": lvl,
"service": svc,
"message": msg,
"trace_id": trace_id,
"host": f"k8s-pod-{svc}-prod-{random.randint(1, 4)}",
"region": "us-central1",
}
)
return {
"tool": "kubernetes_log_query",
"result": {
"query": "namespace:prod status:>=400 OR level:WARN OR level:ERROR",
"total_lines_scanned": num_entries * 12,
"returned_entries_count": len(entries),
"entries": entries,
},
}
def generate_sre_metrics() -> dict[str, Any]:
"""Generate timeseries metric query results."""
return {
"tool": "prometheus_metrics_query",
"result": {
"query": "sum by (service, status_code) (rate(http_requests_total[5m]))",
"status": "success",
"data": {
"resultType": "matrix",
"result": [
{
"metric": {"service": "payment-service", "status_code": "500"},
"values": [
[1725062400 + i * 60, str(random.randint(45, 120))] for i in range(30)
],
},
{
"metric": {"service": "payment-service", "status_code": "200"},
"values": [
[1725062400 + i * 60, str(random.randint(10, 25))] for i in range(30)
],
},
{
"metric": {"service": "api-gateway", "status_code": "504"},
"values": [
[1725062400 + i * 60, str(random.randint(80, 200))] for i in range(30)
],
},
],
},
},
}
def create_sre_scenario() -> BenchmarkScenario:
return BenchmarkScenario(
name="SRE Incident Root Cause Analysis",
category="DevOps & Reliability",
description="Analyze 350+ multi-service container logs and Prometheus metrics to find cascading incident root cause.",
system_prompt="You are an expert Google Cloud SRE agent diagnosing a production outage. Identify the root cause service, exact error condition, and recommend immediate remediation steps.",
user_query="We are experiencing a surge of 500/504 errors on user checkout. Inspect the attached Kubernetes logs and Prometheus metrics to determine the exact root cause and affected database cluster.",
tool_outputs=[generate_sre_logs(350), generate_sre_metrics()],
expected_facts=[
"payment-service",
"pg-primary.db.internal",
"connection pool",
"5432",
],
expected_anomalies=["connectionpoolexhaustederror", "circuit breaker"],
)
# ----------------------------------------------------------------------------
# 2. Codebase Security & PR Audit
# ----------------------------------------------------------------------------
def generate_code_search_results(num_files: int = 40) -> dict[str, Any]:
"""Generate realistic codebase search output with embedded JWT auth vulnerability."""
random.seed(42)
files = []
for i in range(num_files):
if i == 7:
files.append(
{
"path": "services/auth/jwt_validator.py",
"language": "python",
"size_bytes": 4210,
"matches": [
{
"line_number": 42,
"line_content": "def verify_token(token: str, secret_key: str):",
"context": [
" # Insecure fallback: skip verification when verify=False is passed in header",
" header = jwt.get_unverified_header(token)",
" if header.get('alg') == 'none' or header.get('bypass_auth'):",
" return jwt.decode(token, options={'verify_signature': False})",
" return jwt.decode(token, secret_key, algorithms=['HS256', 'RS256'])",
],
}
],
"score": 0.98,
}
)
else:
files.append(
{
"path": f"services/core/handler_module_{i}.py",
"language": "python",
"size_bytes": random.randint(1500, 9000),
"matches": [
{
"line_number": random.randint(10, 200),
"line_content": f" user_ctx = request.state.get('user_context_{i}')",
"context": [
" # Standard authorization check",
" if not user_ctx or not user_ctx.is_authenticated:",
" raise PermissionDeniedError('Unauthorized')",
],
}
],
"score": round(random.uniform(0.3, 0.7), 2),
}
)
return {
"tool": "ast_code_search",
"result": {
"query": "jwt.decode OR verify_signature: False",
"repository": "enterprise-agent-platform/api-gateway",
"total_matches": len(files),
"files": files,
},
}
def generate_git_diff() -> dict[str, Any]:
"""Generate git diff showing an authentication bypass regression."""
diff_text = """
diff --git a/services/auth/jwt_validator.py b/services/auth/jwt_validator.py
index a1b2c3d..e4f5g6h 100644
--- a/services/auth/jwt_validator.py
+++ b/services/auth/jwt_validator.py
@@ -39,7 +39,10 @@ class JWTValidator:
def verify_token(self, token: str) -> dict:
- return jwt.decode(token, self.secret_key, algorithms=['RS256'])
+ header = jwt.get_unverified_header(token)
+ if header.get('alg') == 'none' or header.get('bypass_auth'):
+ # TODO: Remove testing bypass before production release
+ return jwt.decode(token, options={'verify_signature': False})
+ return jwt.decode(token, self.secret_key, algorithms=['HS256', 'RS256'])
"""
return {
"tool": "git_diff",
"result": {
"base_commit": "main",
"target_commit": "feature/auth-speedup",
"files_changed": 1,
"insertions": 5,
"deletions": 1,
"diff": diff_text.strip(),
},
}
def create_security_audit_scenario() -> BenchmarkScenario:
return BenchmarkScenario(
name="Codebase Security Audit & PR Review",
category="Application Security",
description="Inspect code search results across 40 files and a git diff to detect an authentication signature bypass vulnerability.",
system_prompt="You are a principal software engineer conducting a defensive code review on a pull request. Identify security flaws, cite the exact file and lines, describe the security risk, and propose a secure remediation.",
user_query="Please audit the proposed changes in PR #482 against our codebase search results. Are there any critical security vulnerabilities or auth bypasses introduced? Propose a secure remediation.",
tool_outputs=[generate_code_search_results(40), generate_git_diff()],
expected_facts=["jwt_validator.py", "verify_signature", "alg", "none"],
expected_anomalies=["bypass", "signature"],
)
# ----------------------------------------------------------------------------
# 3. Enterprise Database Analytics & Outlier Detection
# ----------------------------------------------------------------------------
def generate_database_records(num_rows: int = 500) -> dict[str, Any]:
"""Generate 500 rows of user financial/usage metrics with high-value anomalies."""
random.seed(42)
tiers = ["free", "starter", "pro", "enterprise"]
statuses = ["active", "active", "active", "pending", "churned"]
countries = ["US", "DE", "GB", "JP", "FR", "AU", "SG"]
rows: list[dict[str, Any]] = []
for i in range(num_rows):
user_id = f"usr_{10000 + i}"
tier = random.choice(tiers)
status = random.choice(statuses)
country = random.choice(countries)
api_calls = random.randint(100, 25000)
spend = round(random.uniform(10.0, 850.0), 2)
if i == 87:
user_id = "usr_SUSPICIOUS_WHALE_87"
tier = "enterprise"
status = "flagged_review"
spend = 148500.00
api_calls = 9800000
elif i == 312:
user_id = "usr_CREDIT_FRAUD_312"
tier = "starter"
status = "suspended_chargeback"
spend = 92450.75
api_calls = 4500000
rows.append(
{
"user_id": user_id,
"tier": tier,
"status": status,
"country": country,
"monthly_api_calls": api_calls,
"total_spend_usd": spend,
"last_active": f"2026-08-{random.randint(1, 30):02d}",
}
)
return {
"tool": "bigquery_analytics_query",
"result": {
"query": "SELECT user_id, tier, status, country, monthly_api_calls, total_spend_usd, last_active FROM `billing_db.monthly_usage`",
"total_rows_scanned": num_rows * 50,
"row_count": len(rows),
"rows": rows,
},
}
def create_analytics_scenario() -> BenchmarkScenario:
return BenchmarkScenario(
name="Enterprise BigQuery Analytics & Outlier Detection",
category="Data Analytics",
description="Process 500 tabular user metrics rows to compute summary aggregations and detect anomalous/fraudulent billing records.",
system_prompt="You are an enterprise data analyst assistant on Google Cloud. Summarize the dataset and explicitly pinpoint any extreme anomalies, billing outliers, or suspicious accounts.",
user_query="Analyze the query output for our billing database. Find the top anomalous spenders, identify suspicious accounts, and provide summary insights.",
tool_outputs=[generate_database_records(500)],
expected_facts=[
"usr_SUSPICIOUS_WHALE_87",
"usr_CREDIT_FRAUD_312",
"148500",
"92450",
],
expected_anomalies=["flagged_review", "suspended_chargeback"],
)
# ----------------------------------------------------------------------------
# 4. Multi-Turn RAG & System Architecture Synthesis
# ----------------------------------------------------------------------------
def generate_rag_documents(num_chunks: int = 25) -> dict[str, Any]:
"""Generate dense technical architecture documents and API contracts."""
random.seed(42)
docs = []
topics = [
(
"Agent Runtime Memory Bus",
"The Agent Platform Runtime maintains session state across persistent memory buses. For models including Gemini 3.8 Flash, context tokens are billed at regional rates. Long-horizon agent sessions that execute >50 tool calls frequently accumulate over 150k context tokens, causing quadratic latency growth in multi-turn reasoning loops. Automatic prompt compression at the proxy layer reduces token footprint before entering the inference queue.",
),
(
"SmartCrusher Compression Contract",
"SmartCrusher evaluates input JSON arrays by preserving the first N elements (schema anchor), the last N elements (temporal recency), all error/warning records, and generating compact statistical distributions for numerical columns. It achieves 70-90% token reduction with zero loss of semantic grounding.",
),
(
"Vertex AI Regional Routing",
"Vertex AI serves Gemini 3.8 Flash on global and regional endpoints. Application Default Credentials authenticate requests via Google OAuth bearer tokens. Headroom preserves client ADC credentials and transparently forwards streaming tokens.",
),
(
"Tool Execution Idempotency",
"All sidecar tools implementing MCP (Model Context Protocol) must declare idempotency keys. When tool outputs contain tabular payloads or JSON arrays, structured compression engines must preserve primary keys, status codes, and statistical distributions while stripping redundant schema metadata.",
),
]
for i in range(num_chunks):
title, content = topics[i % len(topics)]
docs.append(
{
"doc_id": f"doc_platform_spec_{i + 1:03d}",
"title": f"{title} (Revision {i + 1}.0)",
"section": f"Section {i % 5 + 1}.{i % 3 + 1}",
"author": "Google Cloud AI Architecture Team",
"content": content
+ f" Additional metadata specification chunk #{i + 1} covering protocol validation rules and telemetry hooks.",
"relevance_score": round(1.0 - (i * 0.02), 3),
}
)
return {
"tool": "enterprise_vector_search",
"result": {
"query": "Gemini Agent Platform context compression and SmartCrusher specification",
"total_documents": len(docs),
"documents": docs,
},
}
def create_rag_scenario() -> BenchmarkScenario:
return BenchmarkScenario(
name="Multi-Turn RAG & Architecture Synthesis",
category="Knowledge Retrieval (RAG)",
description="Synthesize architecture specifications from 25 dense vector retrieval chunks regarding agent context management.",
system_prompt="You are an enterprise AI solutions architect. Synthesize the provided technical documentation to answer the user's architectural questions accurately.",
user_query="Explain how context compression optimizes agent performance on Gemini 3.8 Flash and explain how the SmartCrusher algorithm operates.",
tool_outputs=[generate_rag_documents(25)],
expected_facts=[
"SmartCrusher",
"Gemini 3.8 Flash",
"first N",
"last N",
],
expected_anomalies=["quadratic latency", "token reduction"],
)
def get_all_scenarios() -> list[BenchmarkScenario]:
"""Return all benchmark scenarios."""
return [
create_sre_scenario(),
create_security_audit_scenario(),
create_analytics_scenario(),
create_rag_scenario(),
]