Commit Graph
2 Commits
Author SHA1 Message Date
Tejas ChopraandClaude Opus 5 4bca514acd docs: correct the published site against the code (#3388)
Audited all 52 pages of docs.headroomlabs.ai/docs against source,
executing code rather than reading it. **36 pages changed.** Every
figure that could not be traced to a benchmark, a test, or a committed
artifact was measured or removed — none was replaced with a different
guess.

## The site was telling users things that were not true

**Telemetry.** `proxy.mdx` stated *"nothing is sent to Headroom Labs"*
and that the anonymous beacon *"has been removed from the code."*
Neither is true — `BEACON_DEFAULT_ON = True`
(`headroom/telemetry/beacon.py:56`), `is_beacon_enabled()` is fail-open,
and there is a live POST at `headroom/telemetry/session.py:75`.

`proxy.mdx` and `limitations.mdx` now describe both switches accurately,
say why the beacon exists (compression quality is the product; it shows
when a release regresses a ratio across real workloads), and list the 12
fields the collector allowlists instead of asserting "content-free". The
install id is disclosed for what it is: a random UUID, not a machine
fingerprint, resettable by deleting the file.

**Numbers.** The landing page's "Proof" table claimed 92/92/47/73%
savings from a harness that seeded nothing, so it could not reproduce a
figure even in principle.

| Scenario | Claimed | Measured |
|---|---|---|
| Code search (100 results) | 92% | **21%** |
| SRE incident debugging | 92% | **57%** |
| Codebase exploration | 47% | **42%** |
| GitHub issue triage | 73% | **30%** |

`real_world_agent_benchmark.py` now seeds, and
`benchmarks/index_proof_table.py` measures the same four scenarios
through the real tokenizer and the real `compress()` path with no
network call. Stable across five seeds; the page cites the command.

Latency was wrong in the *other* direction — 189 ms and 943 ms published
against **0.21 ms and 0.80 ms** measured. The docs made the product look
~1000× slower than it is.

**Examples.** Every `agno` example was broken at line one, including
Quick Start: `HeadroomAgnoModel`'s first dataclass field is `id`, so a
positional wrapped model bound to the wrong field and raised
`ValueError`. `api-reference.mdx` — the canonical signature reference —
had 13 broken examples, including three `AttributeError`s in the first
three lines of its `simulate()` block. Four integration pages omitted
the `[proxy]` extra and then told the reader to run `headroom proxy`,
which fails immediately. `ccr.mdx` showed SDK-level retrieval that only
exists in the proxy handlers.

Also: six Prometheus metric names that are never emitted, a Grafana
dashboard filtering on a label no metric carries, a documented deploy
workflow that does not exist, a 365-day retention default the code caps
at 30, and CCR TTL stated as 5 minutes against an actual 1800s.

## Canonical URL bug

`robots.ts`, `sitemap.ts` and `layout.tsx` fell back to a hardcoded
`headroom-docs.vercel.app` when `NEXT_PUBLIC_SITE_URL` was unset — so
the live site served a sitemap and `Host:` on the wrong domain and
pointed crawlers at the wrong canonical. Verified in the build output:
robots and all 53 sitemap URLs now carry `docs.headroomlabs.ai`.

Setting `NEXT_PUBLIC_SITE_URL=https://docs.headroomlabs.ai` in Vercel is
still worth doing so previews match, but the code no longer depends on
it.

## Verification

Four adversarial judges re-checked this work. They confirmed the
landing-page table reproduces digit-for-digit across five seeds, found
**zero cross-page contradictions**, independently regenerated all nine
surviving compression percentages, and proved `headroom_keep_turns` dead
at runtime with a mocked-transport test. They also caught two links this
audit had reversed to the deprecated org, and 18 examples it had missed
— both fixed here.

`next build` passes with all 52 pages prerendered, matching the
pre-change baseline.

## Follow-up, not in this PR

- GitHub org migration (`chopratejas` → `headroomlabs-ai`) across
source, manifests and packaging, plus stale-domain fixes in `llms.txt`,
the dashboard UI and `pyproject.toml`.
- `wiki/` disposition: it is unpublished and unbuilt, and
`wiki/benchmarks.md` carried an invented production-telemetry section
(250+ instances, 1.4 billion tokens saved) that exists nowhere in the
repo.
- `langchain.mdx` imports are dead against LangChain 1.x
(`langchain.memory`, `langchain.retrievers`,
`create_openai_tools_agent`). Needs a pin-vs-rewrite decision.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01HRaEPjQcCSAwD2xbQMNENu

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-02 15:15:37 +05:30
chopratejas c1d2eec588 docs: improve discoverability for AI agents and search crawlers
Several signals AI agents and search engines use to discover and
install a project were misaligned or missing:

* ``docs/app/layout.tsx`` set ``metadataBase`` to
  ``https://chopratejas.github.io/headroom/`` while the live docs run
  on Vercel — every page's ``og:url`` and ``twitter:url`` resolved to
  a URL that returns 404 for ``/llms.txt``. Now points at the live
  Vercel host (overridable via ``NEXT_PUBLIC_SITE_URL`` for a future
  custom domain). Adds explicit ``openGraph`` and ``twitter`` metadata
  so social shares render a card with the project's pitch.
* No ``llms.txt`` at the GitHub repo root. AI agents crawling
  ``github.com/chopratejas/headroom/`` saw only the README. The new
  ``llms.txt`` follows the llmstxt.org convention: 1-line pitch,
  canonical docs links, copy-paste install commands (pip / npm /
  Docker / proxy / ``headroom wrap``), and entry points for the
  library, proxy, MCP server, and SDK integrations. Points at the
  Fumadocs-generated ``/llms.txt`` and ``/llms-full.txt`` for the
  full picture.
* ``pyproject.toml`` ``Documentation`` URL pointed at the GitHub
  README anchor. Updated to point at the docs site so PyPI visitors
  land on searchable docs, and adds an ``AI / LLM Index`` URL
  pointing at the Fumadocs ``/llms.txt``.
* No explicit AI-bot allow list. Added ``docs/app/robots.ts`` (Next
  13+ App Router convention) with explicit allows for GPTBot,
  ClaudeBot, PerplexityBot, Google-Extended, OAI-SearchBot,
  ChatGPT-User, Cohere-AI, CCBot, and Applebot-Extended. Wildcard
  allow as the catch-all. Advertises the sitemap.
* No ``sitemap.xml`` route. Added ``docs/app/sitemap.ts`` that pulls
  every Fumadocs page out of ``source`` (same source backing
  ``/llms.txt``, search, and OG images) so search and AI crawlers
  can enumerate doc pages without scraping HTML.
* README didn't tell AI agents where to look. Added a 2-line
  pointer near the top nav row: read ``/llms.txt`` here, or fetch
  the live index / full docs blob.

Also tightened the GitHub repo description and added five topics
(``claude-code``, ``cursor``, ``tokens``, ``prompt-engineering``,
``typescript``) via ``gh repo edit`` — that's already live on the
repo, not part of this commit.

No Python or Rust code changes; ``make ci-precheck`` was run to
confirm the test slice still passes.
2026-05-13 17:36:06 -07:00