fix(proxy): redact upstream error detail and add opt-in /metrics loopback gate (#2589)

## Description

Two response-path information-exposure hardenings for the native Rust
reverse proxy (`crates/headroom-proxy`). Both are defensive and
non-breaking; neither adds a user-facing capability nor changes
forwarding behaviour, and defaults are unchanged.

1. **Redact upstream error detail from client responses.**
`ProxyError::into_response` returned `self.to_string()` as the client
body, which for `reqwest` upstream errors interpolates the upstream
URL/host/port. It now returns a generic, stable message per status while
the full detail is retained in the server-side `tracing::warn!` log.
`413` intentionally stays descriptive (no internal detail; retry-on-413
clients rely on recognising it).
2. **Opt-in loopback gate for `/metrics`.** New
`--metrics-require-loopback` / `HEADROOM_PROXY_METRICS_REQUIRE_LOOPBACK`
(default `false`). When enabled, the Prometheus scrape returns `403` to
non-loopback peers. This is defense-in-depth for deployments that bind a
non-loopback `--listen`. Default-off preserves current behaviour
exactly.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `error.rs`: client-facing bodies are now generic per status; full
detail stays in the log only.
- `config.rs`: add `--metrics-require-loopback` /
`HEADROOM_PROXY_METRICS_REQUIRE_LOOPBACK` (default `false`).
- `proxy.rs`: mount `/metrics` in its own sub-router so the optional
`require_metrics_loopback` layer gates only that route (403 to
non-loopback peers when enabled).
- `RUST_DEV.md`: document the new flag in the configuration table.
- Tests: new `tests/integration_metrics_loopback.rs` plus unit tests for
the gate (v4/v6 loopback and non-loopback) and for error-body redaction.

## Testing

<!-- This PR touches only the Rust `headroom-proxy` crate; the
pytest/ruff/mypy items are Python and N/A here (see Additional Notes).
Validated with cargo on the repo-pinned toolchain. -->

- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ cargo fmt --all -- --check          # clean
$ cargo clippy --workspace -- -D warnings   # 0 warnings
$ cargo test --workspace
test result: ok. 1432 passed; 0 failed; 0 ignored   (aggregate across all suites)
$ make test-parity
[tokenizer/smart_crusher/content_detector/text_crusher/diff/log/search] diffed=0
```

## Real Behavior Proof

- Environment: macOS (Darwin 25.3.0), Rust 1.95.0 (repo-pinned
`rust-toolchain.toml`), `headroom-proxy` debug binary run directly (no
mocks), dead upstream `http://127.0.0.1:9`.
- Exact command / steps: started with
`HEADROOM_PROXY_METRICS_REQUIRE_LOOPBACK=true --listen 0.0.0.0:18787`,
then `curl http://127.0.0.1:18787/metrics`, `curl
http://192.168.1.184:18787/metrics` (LAN IP, non-loopback), and `curl -X
POST http://127.0.0.1:18787/v1/messages -d
'{"model":"x","messages":[]}'`; then restarted with the gate off and
repeated the non-loopback `/metrics` scrape.
- Observed result: loopback `/metrics` returned `200`; non-loopback
`/metrics` returned `403` with body `metrics endpoint restricted to
loopback`; the dead-upstream POST returned `502` with client body
`upstream connection failed` while the server log retained the full
detail (`... error sending request for url
(http://127.0.0.1:9/v1/messages)`), i.e. the upstream URL is logged, not
returned to the client; with the gate off, the non-loopback `/metrics`
scrape returned `200` (default behaviour unchanged).
- Not tested: live cloud upstreams (Bedrock/Vertex/OpenAI/Anthropic real
endpoints); the Python proxy path (unchanged by this PR); Windows/Linux
(macOS only).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A (server-side behaviour; see Real Behavior Proof for command output).

## Additional Notes

- This PR modifies only the Rust `headroom-proxy` crate (plus its docs),
so the Python-oriented Testing checkboxes (`pytest`, `ruff`, `mypy`) are
N/A; validation was via `cargo fmt`/`clippy`/`test` and `make
test-parity` on the repo-pinned toolchain, per the Test Output above.
- No new dependencies. Both changes are non-breaking and leave all
defaults unchanged (the `/metrics` gate is opt-in).

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2589"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
Laura Money
2026-10-05 15:29:31 -05:00
committed by GitHub
co-authored by JerrettDavis
parent e7b3baf236
commit a05717f6ec
5 changed files with 261 additions and 26 deletions
+1
View File
@@ -84,6 +84,7 @@ curl -si http://127.0.0.1:8787/v1/models
| Flag | Env var | Default | Notes |
| --- | --- | --- | --- |
| `--listen` | `HEADROOM_PROXY_LISTEN` | `0.0.0.0:8787` | bind address |
| `--metrics-require-loopback` | `HEADROOM_PROXY_METRICS_REQUIRE_LOOPBACK` | `false` | when set, `/metrics` is served only to loopback peers (403 otherwise); recommended on non-loopback binds |
| `--upstream` | `HEADROOM_PROXY_UPSTREAM` | (required) | base URL the proxy forwards to |
| `--upstream-timeout` | | `600s` | end-to-end request timeout (long for streams) |
| `--upstream-connect-timeout` | | `10s` | TCP/TLS connect timeout |
+28
View File
@@ -276,6 +276,25 @@ pub struct CliArgs {
#[arg(long, env = "HEADROOM_PROXY_LISTEN", default_value = "0.0.0.0:8787")]
pub listen: SocketAddr,
/// Restrict the Prometheus `/metrics` scrape endpoint to loopback
/// clients. Default `false` to preserve existing behaviour (the
/// endpoint is reachable from wherever the proxy is bound). Set to
/// `true` — recommended whenever `--listen` binds a non-loopback
/// address — so `/metrics` (token counts, per-session cache-hit
/// rates, rate-limit gauges) is served only to `127.0.0.1` / `::1`
/// and returns 403 otherwise. This is defense-in-depth alongside
/// firewalling the path.
///
/// Source priority: CLI flag → `HEADROOM_PROXY_METRICS_REQUIRE_LOOPBACK`
/// env var → default (`false`).
#[arg(
long = "metrics-require-loopback",
env = "HEADROOM_PROXY_METRICS_REQUIRE_LOOPBACK",
default_value_t = false,
action = clap::ArgAction::Set,
)]
pub metrics_require_loopback: bool,
/// Upstream base URL the proxy forwards to (e.g. http://127.0.0.1:8788).
/// REQUIRED — there is no default; we want operators to be explicit.
#[arg(long, env = "HEADROOM_PROXY_UPSTREAM")]
@@ -617,6 +636,9 @@ pub struct Config {
/// Runtime rollout state resolved from CLI/env.
pub rollout: RolloutSnapshot,
pub listen: SocketAddr,
/// Gate the Prometheus `/metrics` endpoint to loopback clients.
/// Default `false`; recommend `true` when `listen` is non-loopback.
pub metrics_require_loopback: bool,
pub upstream: Url,
pub upstream_timeout: Duration,
pub upstream_connect_timeout: Duration,
@@ -725,6 +747,7 @@ impl Config {
Self {
rollout: rollout.clone(),
listen: args.listen,
metrics_require_loopback: args.metrics_require_loopback,
upstream: args.upstream,
upstream_timeout: args.upstream_timeout,
upstream_connect_timeout: args.upstream_connect_timeout,
@@ -761,6 +784,11 @@ impl Config {
Self {
rollout: RolloutSnapshot::default(),
listen: "127.0.0.1:0".parse().unwrap(),
// Off by default in tests: the metrics integration tests
// scrape `/metrics` and assert 200, and oneshot-style tests
// carry no `ConnectInfo`. Tests that exercise the gate set
// this to `true` explicitly.
metrics_require_loopback: false,
upstream,
upstream_timeout: Duration::from_secs(60),
upstream_connect_timeout: Duration::from_secs(5),
+81 -18
View File
@@ -55,30 +55,93 @@ impl From<crate::upstream_path::PathError> for ProxyError {
impl IntoResponse for ProxyError {
fn into_response(self) -> Response {
let (status, msg) = match &self {
ProxyError::Upstream(e) if e.is_timeout() => (
StatusCode::GATEWAY_TIMEOUT,
format!("upstream timeout: {e}"),
// Full detail is for operators only. The `Display` impls above
// interpolate the source error (e.g. reqwest's chain), which for
// upstream failures can embed the upstream host/port and other
// internal wiring. We log that detail but return a generic,
// stable body to the client so an error response can't be used
// to fingerprint the upstream. The HTTP status still carries the
// actionable signal for well-behaved clients.
let detail = self.to_string();
let (status, client_msg): (StatusCode, &'static str) = match &self {
ProxyError::Upstream(e) if e.is_timeout() => {
(StatusCode::GATEWAY_TIMEOUT, "upstream timeout")
}
ProxyError::Upstream(e) if e.is_connect() => {
(StatusCode::BAD_GATEWAY, "upstream connection failed")
}
ProxyError::Upstream(_) => (StatusCode::BAD_GATEWAY, "upstream request failed"),
ProxyError::InvalidUpstream(_) => (StatusCode::BAD_GATEWAY, "upstream request failed"),
ProxyError::InvalidHeader(_) => (StatusCode::BAD_REQUEST, "invalid request"),
// PayloadTooLarge stays descriptive: it leaks no internal
// detail (the interpolated value is the client's own size vs
// the configured cap) and clients with retry-on-413 logic
// rely on recognizing it. See the PayloadTooLarge doc comment.
ProxyError::PayloadTooLarge(_) => (
StatusCode::PAYLOAD_TOO_LARGE,
"request body exceeds configured limit",
),
ProxyError::Upstream(e) if e.is_connect() => (
StatusCode::BAD_GATEWAY,
format!("upstream connect error: {e}"),
),
ProxyError::Upstream(_) => (StatusCode::BAD_GATEWAY, self.to_string()),
ProxyError::InvalidUpstream(_) => (StatusCode::BAD_GATEWAY, self.to_string()),
ProxyError::InvalidHeader(_) => (StatusCode::BAD_REQUEST, self.to_string()),
ProxyError::PayloadTooLarge(_) => (StatusCode::PAYLOAD_TOO_LARGE, self.to_string()),
ProxyError::InvalidPath(_) => (StatusCode::BAD_REQUEST, self.to_string()),
ProxyError::WebSocket(_) => (StatusCode::BAD_GATEWAY, self.to_string()),
ProxyError::Io(_) => (StatusCode::INTERNAL_SERVER_ERROR, self.to_string()),
ProxyError::InvalidPath(_) => (StatusCode::BAD_REQUEST, "invalid request"),
ProxyError::WebSocket(_) => (StatusCode::BAD_GATEWAY, "upstream request failed"),
ProxyError::Io(_) => (StatusCode::INTERNAL_SERVER_ERROR, "internal error"),
// CompressionStartup is a startup-time error, not a
// per-request one — but if it ever surfaces in the
// handler path, surface as 500 rather than panic.
ProxyError::CompressionStartup(_) => {
(StatusCode::INTERNAL_SERVER_ERROR, self.to_string())
(StatusCode::INTERNAL_SERVER_ERROR, "internal error")
}
};
tracing::warn!(error = %msg, "proxy error");
(status, msg).into_response()
tracing::warn!(status = %status.as_u16(), error = %detail, "proxy error");
(status, client_msg).into_response()
}
}
#[cfg(test)]
mod tests {
use super::*;
use axum::body::to_bytes;
async fn status_and_body(err: ProxyError) -> (StatusCode, String) {
let resp = err.into_response();
let status = resp.status();
let bytes = to_bytes(resp.into_body(), 1024).await.unwrap();
(status, String::from_utf8_lossy(&bytes).to_string())
}
// The interpolated detail (which for real upstream errors can embed
// the upstream host/port) must NEVER reach the client body. Before
// the hardening these bodies were `self.to_string()` and DID contain
// the detail, so these assertions fail on old code.
#[tokio::test]
async fn invalid_header_detail_is_not_leaked_to_client() {
let (status, body) = status_and_body(ProxyError::InvalidHeader(
"upstream-secret-host:9443".into(),
))
.await;
assert_eq!(status, StatusCode::BAD_REQUEST);
assert_eq!(body, "invalid request");
assert!(!body.contains("upstream-secret-host"));
}
#[tokio::test]
async fn invalid_upstream_detail_is_not_leaked_to_client() {
let (status, body) = status_and_body(ProxyError::InvalidUpstream(
"http://10.0.0.5:8788/internal".into(),
))
.await;
assert_eq!(status, StatusCode::BAD_GATEWAY);
assert_eq!(body, "upstream request failed");
assert!(!body.contains("10.0.0.5"));
}
#[tokio::test]
async fn payload_too_large_stays_descriptive() {
// Deliberately kept descriptive: no internal detail, and
// retry-on-413 clients rely on recognizing it.
let (status, body) =
status_and_body(ProxyError::PayloadTooLarge("body exceeds cap".into())).await;
assert_eq!(status, StatusCode::PAYLOAD_TOO_LARGE);
assert_eq!(body, "request body exceeds configured limit");
}
}
+92 -8
View File
@@ -161,6 +161,36 @@ impl AppState {
}
}
/// Per-route middleware for `/metrics`, attached only when
/// `--metrics-require-loopback` is enabled. Rejects any scrape whose
/// peer address is not loopback (`127.0.0.0/8` / `::1`) with
/// `403 Forbidden`, keeping operational metrics off the wire on a
/// non-loopback bind. Relies on the server being served with
/// `ConnectInfo<SocketAddr>` (see `main.rs`); requests synthesised
/// without connect info (e.g. `oneshot` in unit tests) would fail the
/// `ConnectInfo` extractor — which is one reason the gate defaults off.
async fn require_metrics_loopback(
ConnectInfo(peer): ConnectInfo<SocketAddr>,
req: Request<Body>,
next: axum::middleware::Next,
) -> axum::response::Response {
if peer.ip().is_loopback() {
next.run(req).await
} else {
tracing::warn!(
event = "metrics_scrape_rejected_non_loopback",
peer = %peer,
"rejected /metrics scrape from non-loopback client \
(--metrics-require-loopback is enabled)"
);
(
StatusCode::FORBIDDEN,
"metrics endpoint restricted to loopback",
)
.into_response()
}
}
/// Build the axum app. `/healthz` and `/healthz/upstream` are intercepted;
/// everything else hits the catch-all forwarder. WebSocket upgrades are
/// handled inside the catch-all handler when an `Upgrade: websocket` header
@@ -170,14 +200,6 @@ pub fn build_app(state: AppState) -> Router {
.route("/healthz", get(healthz))
.route("/healthz/upstream", get(healthz_upstream))
.route("/rollout/status", get(rollout_status))
// PR-D3: Prometheus scrape endpoint. Renders the global
// registry in text format. The handler is stateless — no
// `AppState` needed — and idempotent across concurrent
// scrapes (`prometheus`'s registry uses internal locking).
// Mounted unconditionally because it has no dependencies on
// any feature flag; an operator who doesn't want it scraped
// simply firewalls the path.
.route("/metrics", get(crate::observability::handle_metrics))
// PR-C2: explicit POST route for /v1/chat/completions. The
// handler buffers the body and re-injects it into
// `forward_http`, which runs the OpenAI live-zone gate
@@ -212,6 +234,27 @@ pub fn build_app(state: AppState) -> Router {
post(crate::vertex::handle_vertex_predict_dispatch),
);
// PR-D3: Prometheus scrape endpoint. Renders the global registry in
// text format. The handler is stateless — no `AppState` needed — and
// idempotent across concurrent scrapes (`prometheus`'s registry uses
// internal locking). Mounted as its own sub-router so the optional
// loopback gate can wrap ONLY this route. `/metrics` exposes
// operational detail (token counts, per-session cache-hit rates,
// rate-limit gauges); when `--metrics-require-loopback` is set,
// `require_metrics_loopback` rejects non-loopback peers with 403 —
// defense-in-depth alongside firewalling the path. Default off,
// preserving the previous "mounted unconditionally; operator
// firewalls the path" behaviour.
let metrics_router: Router<AppState> = {
let base = Router::new().route("/metrics", get(crate::observability::handle_metrics));
if state.config.metrics_require_loopback {
base.route_layer(axum::middleware::from_fn(require_metrics_loopback))
} else {
base
}
};
router = router.merge(metrics_router);
// PR-D1: native AWS Bedrock InvokeModel route. Mounts only when
// `enable_bedrock_native` is on (default). The handler runs the
// live-zone compressor over Anthropic-shape bodies, signs with
@@ -1697,6 +1740,47 @@ mod tests {
assert_eq!(out.as_str(), "http://up:8080/");
}
// `require_metrics_loopback` gate, driven directly via `oneshot`
// with a manually-injected `ConnectInfo` so BOTH branches are
// covered — including the non-loopback 403 path the real-server
// integration test can't reach from a loopback-only client.
fn metrics_gate_router() -> Router {
Router::new()
.route("/metrics", get(|| async { "metrics-body" }))
.route_layer(axum::middleware::from_fn(require_metrics_loopback))
}
async fn scrape_status_from(peer: &str) -> StatusCode {
use tower::util::ServiceExt;
let mut req = Request::builder()
.method("GET")
.uri("/metrics")
.body(Body::empty())
.unwrap();
let addr: SocketAddr = peer.parse().unwrap();
req.extensions_mut().insert(ConnectInfo(addr));
metrics_gate_router().oneshot(req).await.unwrap().status()
}
#[tokio::test]
async fn metrics_gate_allows_ipv4_loopback() {
assert_eq!(scrape_status_from("127.0.0.1:5555").await, StatusCode::OK);
}
#[tokio::test]
async fn metrics_gate_allows_ipv6_loopback() {
assert_eq!(scrape_status_from("[::1]:5555").await, StatusCode::OK);
}
#[tokio::test]
async fn metrics_gate_rejects_non_loopback_with_403() {
// 203.0.113.0/24 is TEST-NET-3 (RFC 5737) — never loopback.
assert_eq!(
scrape_status_from("203.0.113.7:5555").await,
StatusCode::FORBIDDEN
);
}
#[test]
fn url_build_refuses_to_escape_base_path() {
// `url::Url::set_path` would resolve this to `/tenant-b/v1/messages`.
@@ -0,0 +1,59 @@
//! `/metrics` loopback-gate integration coverage.
//!
//! `--metrics-require-loopback` (`Config::metrics_require_loopback`)
//! restricts the Prometheus scrape endpoint to loopback peers as
//! defense-in-depth for non-loopback binds. The shared harness serves
//! the app over a real `127.0.0.1` listener with
//! `ConnectInfo<SocketAddr>`, so a reqwest client here is itself a
//! loopback peer: with the gate ON the scrape must STILL succeed (200),
//! proving the middleware is wired correctly and never breaks the
//! legitimate local-scrape path. The non-loopback rejection (403) path
//! can't be driven from a loopback-only test client; it is enforced by
//! the middleware's `peer.ip().is_loopback()` check in
//! `crate::proxy::require_metrics_loopback`.
mod common;
use common::start_proxy_with;
/// Gate ON: a loopback client is still allowed through, so the scrape
/// returns 200 with the usual Prometheus descriptor lines.
#[tokio::test]
async fn metrics_gate_allows_loopback_scrape() {
// Upstream is never contacted — `/metrics` is served locally.
let proxy = start_proxy_with("http://127.0.0.1:1", |cfg| {
cfg.metrics_require_loopback = true;
})
.await;
let resp = reqwest::Client::new()
.get(format!("{}/metrics", proxy.url()))
.send()
.await
.expect("metrics scrape");
assert_eq!(
resp.status(),
200,
"loopback client must still pass the metrics loopback gate"
);
let body = resp.text().await.unwrap();
assert!(
body.contains("# HELP") || body.contains("# TYPE"),
"scrape body should carry Prometheus metric descriptors"
);
}
/// Gate OFF (default): unchanged behaviour — the scrape is served.
#[tokio::test]
async fn metrics_default_open_serves_scrape() {
let proxy = start_proxy_with("http://127.0.0.1:1", |_| {}).await;
let resp = reqwest::Client::new()
.get(format!("{}/metrics", proxy.url()))
.send()
.await
.expect("metrics scrape");
assert_eq!(resp.status(), 200);
}