fix(proxy): bound the Bedrock Prometheus model label (#4694)

## Description

The Bedrock handlers in `headroom-proxy` use the `{model_id}` URL path
segment, which the client controls, as the `model` label of
`bedrock_invoke_count_total`, `bedrock_invoke_latency_seconds` and
`bedrock_eventstream_message_count_total`. Every distinct value created
a new series that stayed in the registry until exit. This clips the
label to 128 characters and folds values past 200 distinct ids into
`other`, inside the three `record_*`/`observe_*` helpers in
`observability/prometheus.rs`, so the callers in `bedrock/invoke.rs` and
`bedrock/invoke_streaming.rs` are covered without being touched. The
limits are the ones `headroom/proxy/persistent_metrics.py` already uses
(`MAX_LABEL_LENGTH = 128`, `MAX_TRACKED_MODELS = 200`).

Fixes #4685

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `crates/headroom-proxy/src/observability/prometheus.rs`: add
`bound_label` (clip + fold past a cap) and `bounded_model` (one
process-wide set shared by the three families), and call it at the top
of `record_bedrock_invoke`, `observe_bedrock_invoke_latency` and
`record_bedrock_eventstream_message`.
- Three new unit tests in the same file: many distinct ids stay at most
cap + 1 on all three families, a long id is clipped, and `bound_label`
keeps normal ids exact and folds only past the cap.
- Left out: the issue also suggests recording after validation; that is
a handler-order change and was not needed once the helpers bound the
label.

## Testing

- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

This is a Rust-only change, so the Python checks above do not apply; the
Rust equivalents (`cargo test`, `cargo fmt --check`, `cargo clippy -D
warnings`) were run.

### Test Output

Fail before (the three call-site lines reverted, tests kept; `cargo test
-p headroom-proxy --locked --lib observability::prometheus`):

```text
test observability::prometheus::tests::long_model_id_is_clipped ... FAILED
test observability::prometheus::tests::many_distinct_model_ids_stay_bounded_across_all_families ... FAILED
assertion failed: labels.contains(&"x".repeat(MAX_MODEL_LABEL_LEN))
bedrock_invoke_count_total: 300 distinct model labels
test result: FAILED. 5 passed; 2 failed; 0 ignored; 0 measured; 345 filtered out
```

Pass after:

```text
test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 345 filtered out; finished in 0.28s
```

Full crate suite (`cargo test -p headroom-proxy --locked
--no-fail-fast`, Windows): 579 passed, 1 failed. The one failure is
`integration_path_traversal::forwarder_keeps_requests_under_the_base_prefix`
(`Os { code: 10054, kind: ConnectionReset }`), a Windows-only failure
that is not touched by this change; it was not re-run on a clean
checkout of main in this session.

```text
cargo fmt -p headroom-proxy -- --check   -> exit 0
cargo clippy -p headroom-proxy --all-targets --locked -- -D warnings   -> exit 0
```

## Real Behavior Proof

- Environment: Windows 11, Rust 1.95.0, debug build of the
`headroom-proxy` binary at this commit, listening on 127.0.0.1:18787;
`AWS_EC2_METADATA_DISABLED=true` and no AWS credentials, so Bedrock
requests are unsigned and the closed upstream (127.0.0.1:1) refuses
them; the metric is recorded at handler entry, before the upstream call.
- Exact command / steps: `headroom-proxy --listen 127.0.0.1:18787
--upstream http://127.0.0.1:1 --enable-bedrock-native true
--bedrock-endpoint http://127.0.0.1:1`; then `curl -X POST -d '{}'
http://127.0.0.1:18787/model/proof-model-$i/invoke` for i = 1..300, one
300-character model id, and `anthropic.claude-3-haiku-20240307-v1:0`;
then `GET /metrics` and count the distinct `model` values per family.
- Observed result: 201 distinct `model` values on
`bedrock_invoke_count_total` (200 ids + `other`), 201 on
`bedrock_invoke_latency_seconds_count`;
`bedrock_invoke_count_total{auth_mode="oauth",model="other",region="us-east-1"}
102` (the 100 ids past the cap, plus the 300-character id and the normal
id, which were also sent after the cap was reached); longest label seen
is 15 characters. Before the change the same unit-level flood produced
300 distinct labels.
- Not tested: a control run of the unfixed binary through the same HTTP
steps (the before-state is shown by the failing unit test only); the
streaming route and `bedrock_eventstream_message_count_total` over HTTP
(covered by the unit test, which drives the helper directly); memory
growth or scrape time under load; clipping over HTTP (the 300-character
id arrived after the cap, so it was folded; clipping is covered by
`long_model_id_is_clipped`); a real AWS upstream.

## Runtime Rollout Safety

- Rollout-managed feature(s): none changed; the code sits on the
existing `native_bedrock` route, which this PR does not gate or alter.
- Minimum rollout channel: n/a (no new feature)
- Stable/default behavior changed: yes, narrowly: after 200 distinct
model ids since process start, further new ids appear in the three
Bedrock metrics as `model="other"`, and ids longer than 128 characters
are truncated. Ids seen earlier keep their exact label.
- Kill switch / disable path: none added; it is label bounding only.
Disabling `native_bedrock` removes the metrics' source as before.
- Unsafe override required: no
- Qualification impact: none expected; no request or response path
changes, only metric label values.
- Rollback path: revert this commit; the series count returns to
unbounded growth.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes (one
pre-existing Windows-only failure, see Testing)
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- Documentation: not changed; the module header comment in
`prometheus.rs` still describes the label set, and the new constants
carry their own doc comments.
- The folded set is shared by the three families and is never evicted,
so the ids that arrive first keep their own label for the life of the
process (same behaviour as `bounded_model_key` in the Python tracker). A
maintainer may prefer a different cap or an allow-list of known Bedrock
model ids; this takes the existing project limits.
- `other` is allowed through even if a client literally sends `other`;
it does not use up a slot.

Produced with AI assistance (Claude Code) operated by me (HetCreep); the
tests and proof above were run on my machine against a clone of the
repository.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4694"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
HetCreep
2026-11-09 12:31:11 -06:00
committed by GitHub
co-authored by Claude Opus 5.5
parent d5644cbaca
commit d70a8b76a5
@@ -46,6 +46,7 @@
//! operation while still being available under
//! `RUST_LOG=headroom_proxy::observability=debug`.
use std::collections::HashSet;
use std::sync::OnceLock;
use axum::body::Body;
@@ -64,6 +65,59 @@ use headroom_core::auth_mode::AuthMode;
/// CloudWatch sample dashboards use.
const LATENCY_BUCKETS_SECONDS: &[f64] = &[0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0, 30.0, 60.0];
/// Longest `model` label value kept; mirrors `MAX_LABEL_LENGTH` in
/// `headroom/proxy/persistent_metrics.py`.
const MAX_MODEL_LABEL_LEN: usize = 128;
/// Distinct `model` label values before new ones fold into
/// [`OTHER_MODEL_LABEL`]; mirrors `MAX_TRACKED_MODELS` in
/// `headroom/proxy/persistent_metrics.py`.
const MAX_TRACKED_MODELS: usize = 200;
const OTHER_MODEL_LABEL: &str = "other";
/// Clip `raw` to [`MAX_MODEL_LABEL_LEN`] chars and fold it into
/// [`OTHER_MODEL_LABEL`] once `seen` already holds `cap` values. The
/// model id is the client-supplied URL path segment, so without this
/// every distinct value is a new series that lives until exit.
fn bound_label(seen: &mut HashSet<String>, raw: &str, cap: usize) -> String {
let key: String = raw.chars().take(MAX_MODEL_LABEL_LEN).collect();
if key == OTHER_MODEL_LABEL || seen.contains(&key) {
return key;
}
if seen.len() < cap {
seen.insert(key.clone());
return key;
}
OTHER_MODEL_LABEL.to_string()
}
/// [`bound_label`] over the set shared by all three Bedrock metric
/// families.
fn bounded_model(raw: &str) -> String {
with_seen_models(|seen| bound_label(seen, raw, MAX_TRACKED_MODELS))
}
/// The admitted model ids: one process-wide set.
#[cfg(not(test))]
fn with_seen_models<R>(f: impl FnOnce(&mut HashSet<String>) -> R) -> R {
static SEEN: OnceLock<std::sync::Mutex<HashSet<String>>> = OnceLock::new();
let mut seen = SEEN
.get_or_init(|| std::sync::Mutex::new(HashSet::new()))
.lock()
.unwrap_or_else(|e| e.into_inner());
f(&mut seen)
}
/// Under test the set is per thread. libtest runs each test on its own
/// thread, so a test that fills the cap cannot fold another test's ids
/// to `other`, whatever order or parallelism the runner picks.
#[cfg(test)]
fn with_seen_models<R>(f: impl FnOnce(&mut HashSet<String>) -> R) -> R {
thread_local! {
static SEEN: std::cell::RefCell<HashSet<String>> = Default::default();
}
SEEN.with(|s| f(&mut s.borrow_mut()))
}
/// Lazy singleton registry. Borrowed by `handle_metrics` for
/// scrape rendering and by every metric registration helper.
///
@@ -145,6 +199,8 @@ fn eventstream_counter() -> &'static IntCounterVec {
/// correlate this metric with the request's structured log line via
/// the `request_id` field (callers thread that through).
pub fn record_bedrock_invoke(model: &str, region: &str, auth_mode: AuthMode) {
let model = bounded_model(model);
let model = model.as_str();
invoke_counter()
.with_label_values(&[model, region, auth_mode.as_str()])
.inc();
@@ -164,6 +220,8 @@ pub fn record_bedrock_invoke(model: &str, region: &str, auth_mode: AuthMode) {
/// free of `Instant` types so unit tests can assert on synthetic
/// values.
pub fn observe_bedrock_invoke_latency(model: &str, region: &str, seconds: f64) {
let model = bounded_model(model);
let model = model.as_str();
invoke_latency()
.with_label_values(&[model, region])
.observe(seconds);
@@ -182,6 +240,8 @@ pub fn observe_bedrock_invoke_latency(model: &str, region: &str, seconds: f64) {
/// the message (or `unknown` when the message header was missing,
/// which itself is loud-logged at the call site).
pub fn record_bedrock_eventstream_message(model: &str, region: &str, event_type: &str) {
let model = bounded_model(model);
let model = model.as_str();
eventstream_counter()
.with_label_values(&[model, region, event_type])
.inc();
@@ -351,6 +411,80 @@ mod tests {
);
}
/// Distinct `model="..."` values on lines of `metric` in the scrape.
fn model_labels(metric: &str) -> HashSet<String> {
scrape()
.lines()
.filter(|l| l.starts_with(metric))
.filter_map(|l| l.split("model=\"").nth(1))
.filter_map(|r| r.split('"').next())
.map(str::to_string)
.collect()
}
#[test]
fn many_distinct_model_ids_stay_bounded_across_all_families() {
let known = "anthropic.claude-3-haiku-20240307-v1:0";
assert_eq!(bounded_model(known), known);
for i in 0..(MAX_TRACKED_MODELS + 100) {
let m = format!("flood-model-{i}");
record_bedrock_invoke(&m, "us-flood-1", AuthMode::OAuth);
observe_bedrock_invoke_latency(&m, "us-flood-1", 0.1);
record_bedrock_eventstream_message(&m, "us-flood-1", "chunk");
}
for metric in [
"bedrock_invoke_count_total",
"bedrock_invoke_latency_seconds_count",
"bedrock_eventstream_message_count_total",
] {
// Other tests write straight into the vectors; count only ours.
let n = model_labels(metric)
.iter()
.filter(|l| l.starts_with("flood-model-") || *l == "other")
.count();
assert!(
n <= MAX_TRACKED_MODELS + 1,
"{metric}: {n} distinct model labels"
);
}
assert!(model_labels("bedrock_invoke_count_total").contains("other"));
// Past the cap: an id admitted earlier stays exact, a new one folds.
assert_eq!(bounded_model(known), known);
assert_eq!(bounded_model("brand-new-model"), "other");
// Flood before normal: a test that runs after this one, on its own
// thread, still gets its exact model label.
let fresh = std::thread::spawn(move || bounded_model(known))
.join()
.unwrap();
assert_eq!(fresh, known);
}
#[test]
fn long_model_id_is_clipped() {
let long = "x".repeat(MAX_MODEL_LABEL_LEN + 50);
// Clipping itself, on a fresh set.
let mut seen = HashSet::new();
assert_eq!(
bound_label(&mut seen, &long, MAX_TRACKED_MODELS),
"x".repeat(MAX_MODEL_LABEL_LEN)
);
// Through the metric path the raw id never becomes a label, whichever
// way it is bounded.
record_bedrock_invoke(&long, "us-long-1", AuthMode::OAuth);
assert!(!model_labels("bedrock_invoke_count_total").contains(&long));
}
#[test]
fn bound_label_keeps_normal_ids_and_folds_past_cap() {
let mut seen = HashSet::new();
let id = "anthropic.claude-3-haiku-20240307-v1:0";
assert_eq!(bound_label(&mut seen, id, 2), id);
assert_eq!(bound_label(&mut seen, "b", 2), "b");
assert_eq!(bound_label(&mut seen, "c", 2), "other");
assert_eq!(bound_label(&mut seen, id, 2), id, "known ids stay exact");
assert_eq!(bound_label(&mut seen, "other", 2), "other");
}
#[test]
fn latency_histogram_records_observation() {
observe_bedrock_invoke_latency("anthropic.claude-3-haiku-20240307-v1:0", "us-east-1", 0.42);