mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 12:05:34 +08:00
fix(proxy): keep the Bedrock credentials provider when startup resolution fails (#4705)
## Description The startup half of #4684, which #4696 left open ("Startup credential-resolution failure recovery remains open on that issue"). `load_bedrock_credentials` (`crates/headroom-proxy/src/main.rs:187` on `bb8d18a5`) built the `BedrockCredentials` source and then returned `Err` if the first `get()` failed, so `main` never attached the source. For the rest of the process every Bedrock request answered `500 bedrock_credentials_missing`, even after credentials became available (an ECS task role or IMDS that was not ready yet, an SSO login done after the proxy started), until a restart. With this PR the loader only fails when the default chain has no provider at all. `main` resolves once to report the state at startup, and attaches the source whether or not that first resolution worked. `BedrockCredentials::get` does not cache a failure, so each Bedrock request asks the provider again until it succeeds, then the #4696 refresh-ahead cache takes over. Requests are still never forwarded unsigned; until credentials arrive they get `500 bedrock_credentials_refresh_failed` instead of `bedrock_credentials_missing`. Fixes #4684 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `crates/headroom-proxy/src/main.rs`: `load_bedrock_credentials` returns the source without resolving; the startup block resolves once, attaches the source either way, and logs `bedrock_credentials_loaded` or `bedrock_credentials_unavailable` (message now says the provider is asked again on each Bedrock request). The no-provider case is unchanged. - `crates/headroom-proxy/src/bedrock/credentials.rs`: test `failed_fetch_is_not_cached` (a provider that fails, then succeeds: the second `get()` returns the credentials and the provider is called twice in total). It pins the property `main.rs` now relies on. - `crates/headroom-proxy/tests/integration_bedrock_startup_credentials.rs`: runs the real binary (`CARGO_BIN_EXE_headroom-proxy`), because the startup decision lives in `main.rs` and the in-process `start_proxy_with_state` harness never runs it. Credentials come from a wiremock ECS container endpoint (`AWS_CONTAINER_CREDENTIALS_FULL_URI`) that answers 503 until switched on; all other `AWS_*` variables are removed and IMDS is disabled. The first invoke must get 500; after the switch the second must get 200 and reach the wiremock upstream signed with the late key. The proxy is killed and its temp file removed on drop. - `crates/headroom-proxy/src/bedrock/invoke.rs`: module doc for the refresh-failure case also names "none resolved yet since startup". ## Testing - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed This is a Rust-only change, so the Python checks above do not apply; the Rust equivalents were run (Windows 11, Rust 1.95.0). ### Test Output ```text cargo test -p headroom-proxy --lib --locked bedrock::credentials test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 349 filtered out cargo fmt -p headroom-proxy -- --check -> exit 0 cargo clippy -p headroom-proxy --all-targets --locked -- -D warnings -> exit 0 ``` Full crate suite (`cargo test -p headroom-proxy --locked --no-fail-fast`, Windows): 585 passed, 1 failed. The one failure is `integration_path_traversal::forwarder_keeps_requests_under_the_base_prefix` (`Os { code: 10054, kind: ConnectionReset }`), a Windows-only failure in a test this change does not touch; it fails the same way in #4694's Windows run. Fail before (`main.rs` from `bb8d18a5`, new integration test kept; `cargo test -p headroom-proxy --locked --test integration_bedrock_startup_credentials`): ```text assertion `left == right` failed: credentials served now: {"error":{"type":"bedrock_credentials_missing","message":"AWS credentials not configured; refusing to forward unsigned"}} left: 500 right: 200 test result: FAILED. 0 passed; 1 failed ``` Pass after: ```text test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out ``` The unit test `failed_fetch_is_not_cached` passes on `main` too (the cache already did not store failures); it pins the property the fix relies on. The integration test above is the fail-before. ## Real Behavior Proof - Environment: Windows 11, Rust 1.95.0, debug builds of the `headroom-proxy` binary at `bb8d18a5` (control) and at this branch, offline. - Exact command / steps: a local ECS container-credentials endpoint (`AWS_CONTAINER_CREDENTIALS_FULL_URI=http://127.0.0.1:<port>/creds`) that answers 503 until a flag file exists and then returns test credentials (`AKIDPROOF4684`, 1 h expiry); empty `AWS_CONFIG_FILE` / `AWS_SHARED_CREDENTIALS_FILE`, `AWS_EC2_METADATA_DISABLED=true`, no other `AWS_*` variables. Proxy: `headroom-proxy --listen 127.0.0.1:<p> --upstream http://127.0.0.1:1 --enable-bedrock-native true --bedrock-endpoint http://127.0.0.1:<recorder>`, where the recorder stores each request's `Authorization` header and answers 200. Request A: `POST /model/anthropic.claude-3-haiku-20240307-v1:0/invoke` before the flag exists. Then create the flag. Request B: the same request. - Observed result: on `bb8d18a5` request B, sent after the credentials endpoint recovered, still got 500 `bedrock_credentials_missing` with 0 upstream hits; on this branch it got 200 and reached the upstream signed with `AKIDPROOF4684`. Details: | | `bb8d18a5` | this branch | |---|---|---| | startup log | `bedrock_credentials_unavailable` | `bedrock_credentials_unavailable` | | A (no credentials yet) | 500 `bedrock_credentials_missing` | 500 `bedrock_credentials_refresh_failed` | | B (credentials now served) | 500 `bedrock_credentials_missing`, 0 upstream hits | 200, 1 upstream hit, `Authorization: AWS4-HMAC-SHA256 Credential=AKIDPROOF4684/.../us-east-1/bedrock/aws4_request`, log `bedrock_credentials_refreshed` | - Not tested: real AWS (IMDS, ECS, SSO); the streaming route over HTTP (it reads the same `bedrock_credentials` source); load. A profile-based source that is broken at startup does not recover with this change: in a first attempt with a `credential_process` profile, the SDK's profile provider failed to build (this crate does not enable the `credentials-process` feature) and then answered "profile file credentials provider initialization error already taken" on later calls, so the SDK keeps that failure itself. Container and IMDS sources are asked again on each call. ## Runtime Rollout Safety - Rollout-managed feature(s): none changed; the existing `native_bedrock` route only. - Minimum rollout channel: n/a (no new feature) - Stable/default behavior changed: yes, narrowly: when credentials are unavailable at startup, Bedrock requests now ask the provider again on each request (and succeed once credentials appear) instead of failing until restart; the error code while unavailable changes from `bedrock_credentials_missing` to `bedrock_credentials_refresh_failed`. When credentials resolve at startup nothing changes. - Kill switch / disable path: none added; disabling `native_bedrock` removes the route as before. - Unsafe override required: no - Qualification impact: none expected; a failed resolution never forwards an unsigned request. - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes (see Testing) - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes - While credentials are unavailable, each Bedrock request runs one provider attempt (for IMDS or a container endpoint, an HTTP call with the SDK's own timeouts) before it gets the 500. That is the cost of recovering without a restart; a maintainer may prefer a backoff or a background retry instead. The issue's own suggestion was "Retrying the initial load would also cover the startup-failure case". Produced with AI assistance (Claude Code) operated by me (HetCreep); the tests and proof above were run on my machine against a clone of the repository. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4705" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
1be54c0b46
commit
5438f0035d
@@ -129,6 +129,50 @@ mod tests {
|
||||
assert_eq!(calls.load(Ordering::SeqCst), 1);
|
||||
}
|
||||
|
||||
/// Fails until `ready` is set, then returns long-lived credentials.
|
||||
#[derive(Debug)]
|
||||
struct LateProvider {
|
||||
ready: Arc<std::sync::atomic::AtomicBool>,
|
||||
calls: Arc<AtomicUsize>,
|
||||
}
|
||||
|
||||
impl ProvideCredentials for LateProvider {
|
||||
fn provide_credentials<'a>(&'a self) -> future::ProvideCredentials<'a>
|
||||
where
|
||||
Self: 'a,
|
||||
{
|
||||
self.calls.fetch_add(1, Ordering::SeqCst);
|
||||
if self.ready.load(Ordering::SeqCst) {
|
||||
future::ProvideCredentials::ready(Ok(Credentials::new(
|
||||
"AKID_LATE",
|
||||
"secret",
|
||||
None,
|
||||
None,
|
||||
"late",
|
||||
)))
|
||||
} else {
|
||||
future::ProvideCredentials::ready(Err(CredentialsError::not_loaded("not yet")))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn failed_fetch_is_not_cached() {
|
||||
// A startup resolution that fails must not stick: main.rs keeps
|
||||
// the source either way and relies on the next get() asking again.
|
||||
let ready = Arc::new(std::sync::atomic::AtomicBool::new(false));
|
||||
let calls = Arc::new(AtomicUsize::new(0));
|
||||
let src = BedrockCredentials::new(LateProvider {
|
||||
ready: ready.clone(),
|
||||
calls: calls.clone(),
|
||||
});
|
||||
assert!(src.get().await.is_err());
|
||||
ready.store(true, Ordering::SeqCst);
|
||||
assert_eq!(src.get().await.unwrap().access_key_id(), "AKID_LATE");
|
||||
assert_eq!(src.get().await.unwrap().access_key_id(), "AKID_LATE");
|
||||
assert_eq!(calls.load(Ordering::SeqCst), 2);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn static_credentials_without_expiry_are_returned() {
|
||||
let src = BedrockCredentials::new(Credentials::new("AKID_STATIC", "s", None, None, "t"));
|
||||
|
||||
@@ -25,7 +25,8 @@
|
||||
//! `event=bedrock_credentials_missing` at WARN, return `500` with
|
||||
//! a JSON error body. NEVER forwards an unsigned request.
|
||||
//! - **Credentials refresh failure** (the provider errors when the
|
||||
//! cached credentials are near expiry): log
|
||||
//! cached credentials are near expiry, or when none have been
|
||||
//! resolved yet since startup): log
|
||||
//! `event=bedrock_credentials_refresh_failed`, return `500`.
|
||||
//! - **Envelope parse failure**: log `event=bedrock_envelope_parse_error`,
|
||||
//! pass the bytes through unchanged. Bedrock will reject anyway,
|
||||
|
||||
@@ -83,16 +83,29 @@ async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
|
||||
// upstream — but the Bedrock invoke handler refuses to forward
|
||||
// unsigned requests when `bedrock_credentials` is `None`
|
||||
// (see `bedrock::invoke::handle_invoke`).
|
||||
// A first resolution that fails keeps the provider too: a failed
|
||||
// fetch is not cached, so each Bedrock request asks it again and
|
||||
// the proxy recovers once credentials appear, without a restart.
|
||||
if config.enable_bedrock_native {
|
||||
match load_bedrock_credentials(&config).await {
|
||||
Ok(source) => {
|
||||
let first = source.get().await;
|
||||
state = state.with_bedrock_credentials_source(source);
|
||||
tracing::info!(
|
||||
event = "bedrock_credentials_loaded",
|
||||
region = %config.bedrock_region,
|
||||
profile = ?config.aws_profile,
|
||||
"AWS credentials resolved for Bedrock SigV4 signing"
|
||||
);
|
||||
match first {
|
||||
Ok(_) => tracing::info!(
|
||||
event = "bedrock_credentials_loaded",
|
||||
region = %config.bedrock_region,
|
||||
profile = ?config.aws_profile,
|
||||
"AWS credentials resolved for Bedrock SigV4 signing"
|
||||
),
|
||||
Err(e) => tracing::warn!(
|
||||
event = "bedrock_credentials_unavailable",
|
||||
region = %config.bedrock_region,
|
||||
profile = ?config.aws_profile,
|
||||
error = %e,
|
||||
"AWS credentials not available at startup; each Bedrock request asks the provider again until it succeeds"
|
||||
),
|
||||
}
|
||||
}
|
||||
Err(e) => {
|
||||
tracing::warn!(
|
||||
@@ -181,9 +194,10 @@ fn init_tracing(level: &str) {
|
||||
/// Uses the `aws-config` default chain (env vars → shared profile
|
||||
/// file → IMDS / ECS task role). Honours `Config::aws_profile` when
|
||||
/// set; otherwise the chain picks up `AWS_PROFILE` from the
|
||||
/// environment automatically. Resolves once here so a missing
|
||||
/// credential is reported at startup; the returned source keeps the
|
||||
/// provider for later refreshes.
|
||||
/// environment automatically. Only fails when the chain has no
|
||||
/// provider at all; the caller resolves once so a missing credential
|
||||
/// is reported at startup, and the returned source keeps the provider
|
||||
/// for later attempts and refreshes.
|
||||
async fn load_bedrock_credentials(
|
||||
config: &Config,
|
||||
) -> Result<headroom_proxy::bedrock::BedrockCredentials, Box<dyn std::error::Error + Send + Sync>> {
|
||||
@@ -198,9 +212,9 @@ async fn load_bedrock_credentials(
|
||||
let creds_provider = aws_config
|
||||
.credentials_provider()
|
||||
.ok_or("no credentials provider configured")?;
|
||||
let source = headroom_proxy::bedrock::BedrockCredentials::new(creds_provider);
|
||||
source.get().await?;
|
||||
Ok(source)
|
||||
Ok(headroom_proxy::bedrock::BedrockCredentials::new(
|
||||
creds_provider,
|
||||
))
|
||||
}
|
||||
|
||||
async fn shutdown_signal() {
|
||||
|
||||
@@ -0,0 +1,154 @@
|
||||
//! Credentials that are missing when the proxy starts and appear later.
|
||||
//!
|
||||
//! The startup sequence lives in `main.rs`, so this test runs the real
|
||||
//! binary. Credentials come from a local ECS container endpoint
|
||||
//! (`AWS_CONTAINER_CREDENTIALS_FULL_URI`) that answers 503 until it is
|
||||
//! switched on; the Bedrock upstream is wiremock. Before the fix the
|
||||
//! failed startup fetch dropped the provider, so the request sent after
|
||||
//! the endpoint recovered still got `bedrock_credentials_missing`.
|
||||
|
||||
use std::net::TcpListener;
|
||||
use std::process::{Child, Command, Stdio};
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::sync::Arc;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use serde_json::json;
|
||||
use wiremock::matchers::{method, path};
|
||||
use wiremock::{Mock, MockServer, Request, Respond, ResponseTemplate};
|
||||
|
||||
const TEST_MODEL: &str = "anthropic.claude-3-haiku-20240307-v1:0";
|
||||
|
||||
/// ECS credentials endpoint: 503 until `ready` is set.
|
||||
struct LateCredentials {
|
||||
ready: Arc<AtomicBool>,
|
||||
}
|
||||
|
||||
impl Respond for LateCredentials {
|
||||
fn respond(&self, _request: &Request) -> ResponseTemplate {
|
||||
if !self.ready.load(Ordering::SeqCst) {
|
||||
return ResponseTemplate::new(503);
|
||||
}
|
||||
ResponseTemplate::new(200).set_body_json(json!({
|
||||
"AccessKeyId": "AKIDSTARTUPTEST",
|
||||
"SecretAccessKey": "secret",
|
||||
"Token": "session-token",
|
||||
"Expiration": "2099-01-01T00:00:00Z",
|
||||
}))
|
||||
}
|
||||
}
|
||||
|
||||
/// Kills the proxy and removes its empty AWS file when the test ends,
|
||||
/// pass or fail.
|
||||
struct ProxyProcess(Child, std::path::PathBuf);
|
||||
|
||||
impl Drop for ProxyProcess {
|
||||
fn drop(&mut self) {
|
||||
let _ = self.0.kill();
|
||||
let _ = self.0.wait();
|
||||
let _ = std::fs::remove_file(&self.1);
|
||||
}
|
||||
}
|
||||
|
||||
fn free_port() -> u16 {
|
||||
TcpListener::bind("127.0.0.1:0")
|
||||
.unwrap()
|
||||
.local_addr()
|
||||
.unwrap()
|
||||
.port()
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn credentials_that_appear_after_startup_are_used() {
|
||||
let ready = Arc::new(AtomicBool::new(false));
|
||||
let creds = MockServer::start().await;
|
||||
Mock::given(method("GET"))
|
||||
.and(path("/creds"))
|
||||
.respond_with(LateCredentials {
|
||||
ready: ready.clone(),
|
||||
})
|
||||
.mount(&creds)
|
||||
.await;
|
||||
let upstream = MockServer::start().await;
|
||||
Mock::given(method("POST"))
|
||||
.respond_with(
|
||||
ResponseTemplate::new(200)
|
||||
.insert_header("content-type", "application/json")
|
||||
.set_body_string(r#"{"id":"msg_x","content":[]}"#),
|
||||
)
|
||||
.mount(&upstream)
|
||||
.await;
|
||||
|
||||
let empty = std::env::temp_dir().join(format!("hr-empty-aws-{}", std::process::id()));
|
||||
std::fs::write(&empty, "").unwrap();
|
||||
let port = free_port();
|
||||
let mut cmd = Command::new(env!("CARGO_BIN_EXE_headroom-proxy"));
|
||||
for (key, _) in std::env::vars() {
|
||||
if key.starts_with("AWS_") {
|
||||
cmd.env_remove(key);
|
||||
}
|
||||
}
|
||||
cmd.env("AWS_CONFIG_FILE", &empty)
|
||||
.env("AWS_SHARED_CREDENTIALS_FILE", &empty)
|
||||
.env("AWS_EC2_METADATA_DISABLED", "true")
|
||||
.env(
|
||||
"AWS_CONTAINER_CREDENTIALS_FULL_URI",
|
||||
format!("{}/creds", creds.uri()),
|
||||
)
|
||||
.args([
|
||||
"--listen",
|
||||
&format!("127.0.0.1:{port}"),
|
||||
"--upstream",
|
||||
"http://127.0.0.1:1",
|
||||
"--enable-bedrock-native",
|
||||
"true",
|
||||
"--bedrock-endpoint",
|
||||
&upstream.uri(),
|
||||
])
|
||||
.stdout(Stdio::null())
|
||||
.stderr(Stdio::null());
|
||||
let _proxy = ProxyProcess(cmd.spawn().expect("spawn headroom-proxy"), empty.clone());
|
||||
|
||||
let client = reqwest::Client::new();
|
||||
let base = format!("http://127.0.0.1:{port}");
|
||||
let deadline = Instant::now() + Duration::from_secs(60);
|
||||
while client.get(format!("{base}/healthz")).send().await.is_err() {
|
||||
assert!(Instant::now() < deadline, "proxy did not start");
|
||||
tokio::time::sleep(Duration::from_millis(100)).await;
|
||||
}
|
||||
|
||||
let body = serde_json::to_vec(&json!({
|
||||
"anthropic_version": "bedrock-2023-05-31",
|
||||
"max_tokens": 16,
|
||||
"messages": [{"role": "user", "content": "hi"}]
|
||||
}))
|
||||
.unwrap();
|
||||
let invoke = || {
|
||||
client
|
||||
.post(format!("{base}/model/{TEST_MODEL}/invoke"))
|
||||
.header("content-type", "application/json")
|
||||
.body(body.clone())
|
||||
.send()
|
||||
};
|
||||
|
||||
let before = invoke().await.unwrap();
|
||||
assert_eq!(before.status(), 500, "no credentials yet");
|
||||
|
||||
ready.store(true, Ordering::SeqCst);
|
||||
let after = invoke().await.unwrap();
|
||||
let status = after.status();
|
||||
let text = after.text().await.unwrap();
|
||||
assert_eq!(status, 200, "credentials served now: {text}");
|
||||
|
||||
let signed = upstream.received_requests().await.unwrap();
|
||||
assert_eq!(signed.len(), 1);
|
||||
let auth = signed[0]
|
||||
.headers
|
||||
.get("authorization")
|
||||
.and_then(|v| v.to_str().ok())
|
||||
.unwrap_or_default();
|
||||
assert!(
|
||||
auth.contains("Credential=AKIDSTARTUPTEST/"),
|
||||
"signed with the late credentials: {auth}"
|
||||
);
|
||||
}
|
||||
Reference in New Issue
Block a user