Files
alan blount e67b3c8a29 feat(vertex): add Gemini 3.8 Flash agent benchmark suite (#3371)
## Summary

Adds a reproducible, end-to-end multi-turn agent benchmark suite for
**Gemini 3.8 Flash** on Google Cloud Vertex AI (Gemini Enterprise Agent
Platform) under `examples/vertex_gemini_benchmark/`.

Compares **Direct Vertex AI** vs **Headroom-proxied Vertex AI** across 4
realistic production agent scenarios:
1. **SRE Incident Root Cause Analysis**: 350+ Kubernetes & microservice
logs + Prometheus metrics (identifying connection pool exhaustion,
cluster endpoint, port, and cascading circuit-breaker trips).
2. **Codebase Security Audit & PR Review**: Multi-file code search
results + git diff (identifying insecure `alg: none` and `bypass_auth`
JWT backdoors).
3. **Enterprise BigQuery Analytics**: 500 tabular transaction rows
(computing summaries and isolating fraudulent accounts and high-spend
outliers).
4. **Multi-Turn RAG & System Architecture Synthesis**: 25 dense
technical chunks regarding agent context management contracts.

### Live Benchmark Results (Gemini 3.8 Flash on Vertex AI `global`)

```text
==================================================================================
 📊 FINAL BENCHMARK SUMMARY: GEMINI-3.8-FLASH ON VERTEX AI
==================================================================================
Scenario                           | Baseline Prompt | Headroom Prompt |  Reduction
----------------------------------------------------------------------------------
SRE Incident Root Cause Analysis   |          51,775 |          13,842 |      73.3%
Codebase Security Audit & PR Review |           6,821 |           4,632 |      32.1%
Enterprise BigQuery Analytics & Outlier Detection |          49,020 |           6,120 |      87.5%
Multi-Turn RAG & Architecture Synthesis |           4,071 |           1,437 |      64.7%
----------------------------------------------------------------------------------
TOTAL / AGGREGATE                  |         111,687 |          26,031 |      76.7%

Metric                             |        Baseline |        Headroom | Delta / Impact
----------------------------------------------------------------------------------
Total Prompt Tokens                |         111,687 |          26,031 | -        76.7%
Total All Tokens                   |         116,639 |          29,168 | -        75.0%
Total Cost ($ USD)                 | $       0.09410 | $       0.02849 | -        69.7%
Avg Latency (ms)                   |          8921ms |          5748ms |        +3173ms
Ground Truth Accuracy (Absolute)   |         100.0% |         100.0% | 100.0% Retained
==================================================================================
```

## Real Behavior Proof

• **Environment**: Linux, Python 3.11, live Google Cloud project
(`alanblount-demo`) with Vertex AI enabled, Application Default
Credentials (`gcloud auth application-default login`).
• **Exact command / steps**: `GCP_PROJECT_ID=alanblount-demo python
examples/vertex_gemini_benchmark/benchmark.py --model gemini-3.8-flash`
• **Observed result**: Headroom proxy started cleanly, executed all 4
multi-turn scenarios on `gemini-3.8-flash`, verified ground truth error
causes and anomalies, recorded a **76.7% prompt token reduction** and
**69.7% cost savings**, exported `results.json`, and shut down cleanly.
• **Not tested**: Windows.

## Runtime Rollout Safety

• **Rollout-managed feature(s)**: None; self-contained benchmark suite
under `examples/`.
• **Minimum rollout channel**: standard release.
• **Stable/default behavior changed**: None; purely adds benchmark
tooling and documentation under `examples/`.
• **Kill switch / disable path**: N/A.
• **Unsafe override required**: None.
• **Qualification impact**: None.
• **Rollback path**: Revert PR.

## Review Readiness

[✓] I have performed a self-review
[✓] This PR is ready for human review

## Checklist

[✓] My code follows the project's style guidelines
[✓] I have performed a self-review of my code
[✓] I have commented my code, particularly in hard-to-understand areas
[✓] I have made corresponding changes to the documentation
[✓] My changes generate no new warnings
[✓] I did not edit `CHANGELOG.md` — it is generated by release-please
from my Conventional Commit PR title (a CI guard enforces this)

## Additional Notes

• Companion to PR #2640 (which adds in-band Vertex diagnostic hints and
fixes).
2026-09-06 11:32:19 +08:00
..

Gemini 3.8 Flash on Google Cloud Vertex AI + Headroom Benchmark

A reproducible, real-world benchmark evaluating Headroom context compression with Gemini 3.8 Flash on Google Cloud Vertex AI (Gemini Enterprise Agent Platform).


🎯 Why This Matters

When building production agents (coding assistants, SRE incident responders, data analysts, multi-agent frameworks) on Vertex AI, multi-turn tool loops cause rapid context explosion:

  • Container & Kubernetes logs dump hundreds of lines of noise for a single stack trace.
  • Code search & file trees inflate prompts with repetitive schema structures.
  • Database queries return large tabular results where only outliers and aggregations matter.

Even with Gemini 3.8 Flash's massive 1M-token context window and fast inference, bloated tool returns:

  1. Drive up inference spend as conversation histories compound across turns.
  2. Increase Time to First Token (TTFT) due to large prompt prefill processing.
  3. Dilute attention, making needle-in-a-haystack reasoning and anomaly isolation harder.

Headroom acts as an intelligent, transparent proxy (or in-process SDK layer) that compresses JSON arrays, structured logs, and tables by 40–85% while strictly preserving schema anchors, recent turns, anomalies, error traces, and ground truth accuracy.


📊 Live Benchmark Results

Tested live on Google Cloud Vertex AI (global endpoint) with gemini-3.8-flash.

Pricing basis: Google Cloud Vertex AI introductory standard rates through 2026-12-31 ($0.75 per 1M prompt tokens, $3.75 per 1M text output tokens; cached input rate is $0.075/M).

Scenario Workload Category Baseline Prompt Headroom Prompt Token Reduction Latency Delta Baseline Accuracy Headroom Accuracy Relative Retention
SRE Incident Root Cause Kubernetes & Microservice Logs 51,775 13,842 -73.3% +9.2s faster 100.0% 100.0% 100.0% (✓ PASS)
Security Audit & PR Review Code Search & Git Diffs 6,821 4,632 -32.1% +1.9s faster 100.0% 100.0% 100.0% (✓ PASS)
BigQuery Table Analytics 500 Tabular Transaction Rows 49,020 6,120 -87.5% +2.3s faster 100.0% 100.0% 100.0% (✓ PASS)
Multi-Turn RAG Synthesis 25 Dense Specification Chunks 4,071 1,437 -64.7% -0.7s 100.0% 100.0% 100.0% (✓ PASS)
TOTAL / AGGREGATE Real-World Agent Trajectory 111,687 26,031 -76.7% +3.2s avg faster 100.0% 100.0% 100.0% Retained

Key Metrics Summary

  • Prompt Tokens Saved: 85,656 tokens (76.7% net reduction, 111,687 down to 26,031)
  • Total Inference Cost: $0.0941 down to $0.0285 (69.7% cost savings at standard Vertex rates)
  • Average Latency: 8.9s down to 5.7s (+3.2s faster roundtrip due to reduced prompt prefill load)
  • Relative Quality Retention: 100.0% (zero degradation; Headroom matched baseline extraction of all root causes, database connection pool exhaustion, JWT alg: none auth bypass, fraud outliers, and architectural contracts)

🏗️ Architecture

                                          ┌───────────────────────────────────────┐
                                          │ Google Cloud Vertex AI                │
                                          │ (Gemini Enterprise Agent Platform)    │
                                          │                                       │
┌───────────────────────┐                 │  ┌─────────────────────────────────┐  │
│  google-genai Python  │                 │  │        gemini-3.8-flash         │  │
│  SDK Agent / Script   │                 │  └─────────────────────────────────┘  │
└───────────┬───────────┘                 └───────────────────▲───────────────────┘
            │                                                 │
            │  POST /v1/projects/.../publishers/...           │ Compressed
            │  (base_url = http://127.0.0.1:8787)             │ Payload
            ▼                                                 │
┌─────────────────────────────────────────────────────────────┴───────────────────┐
│ Headroom Proxy (:8787)                                                          │
│                                                                                 │
│  ┌───────────────────────┐   ┌────────────────────────┐   ┌──────────────────┐  │
│  │ ContentRouter         │──▶│ SmartCrusher           │──▶│ LogCompressor    │  │
│  │ (Format & Role Sieve) │   │ (JSON Array Compactor) │   │ (Error Anchor)   │  │
│  └───────────────────────┘   └────────────────────────┘   └──────────────────┘  │
│                                                                                 │
│  • Preserves Google ADC Bearer Auth Tokens                                      │
│  • Compresses verbose tool arrays & tables                                      │
│  • Preserves 100% of anomalies, errors, and schema anchors                      │
└─────────────────────────────────────────────────────────────────────────────────┘

🚀 How to Run the Benchmark

1. Prerequisites

Ensure you have Google Cloud Application Default Credentials (ADC) configured:

gcloud auth application-default login
export GCP_PROJECT_ID=$(gcloud config get-value project)

Install required dependencies:

pip install "headroom-ai[proxy]" google-genai

2. Execute Benchmark

Run the full comparative suite:

python examples/vertex_gemini_benchmark/benchmark.py --model gemini-3.8-flash

CLI Options

--project         GCP Project ID (defaults to $GCP_PROJECT_ID or gcloud default)
--location        Vertex AI location (default: global)
--model           Vertex model ID (default: gemini-3.8-flash)
--port            Headroom local proxy port (default: 8787)
--thinking-budget Thinking token budget in tokens (default: 0 = standard inference)
--output-json     Output file for JSON metrics (default: examples/vertex_gemini_benchmark/results.json)
--social / --no-social  Print formatted social media proof point summary (default: on)

💡 Using Headroom with Vertex AI in Your Agent Code

Connecting your google-genai agent to Headroom requires one line (http_options):

from google import genai

# Point the standard SDK at the Headroom proxy
client = genai.Client(
    vertexai=True,
    project="your-gcp-project-id",
    location="global",
    http_options={"base_url": "http://127.0.0.1:8787"},
)

response = client.models.generate_content(
    model="gemini-3.8-flash",
    contents=[
        "You are an SRE agent.",
        f"Analyze these Kubernetes logs:\n{verbose_json_logs}",
    ],
)
print(response.text)

📢 Social Post / Proof Point Card

🚀 Headroom + Gemini 3.8 Flash on Google Cloud Vertex AI Benchmark

When AI agents run complex multi-turn workflows (SRE debugging, PR reviews, BigQuery analytics), tool output bloat explodes prompt token costs and degrades TTFT.

We ran reproducible end-to-end agent benchmarks comparing Direct Vertex AI vs Headroom-Proxied Vertex AI on gemini-3.8-flash:

📉 Results:
• Prompt Token Reduction: 76.7% (111,687 ➔ 26,031 tokens)
• SRE Log Scenario Reduction: 73.3% (51.7k ➔ 13.8k tokens)
• BigQuery Analytics Scenario: 87.5% (49.0k ➔ 6.1k tokens)
• Total Cost Savings: 69.7% ($0.0941 ➔ $0.0285 at standard Vertex rates)
• Relative Quality Retention: 100.0% (Zero reasoning degradation; 100.0% ground truth retention in both arms)
• Zero Code Changes: Point google-genai SDK http_options.base_url to http://127.0.0.1:8787.

🔗 Full benchmark suite, reproducible scenarios, and code:
https://github.com/headroomlabs-ai/headroom/tree/main/examples/vertex_gemini_benchmark