fix(proxy): keep the Bedrock credentials provider when startup resolution fails (#4705)

## Description

The startup half of #4684, which #4696 left open ("Startup
credential-resolution failure recovery remains open on that issue").

`load_bedrock_credentials` (`crates/headroom-proxy/src/main.rs:187` on
`bb8d18a5`) built the `BedrockCredentials` source and then returned
`Err` if the first `get()` failed, so `main` never attached the source.
For the rest of the process every Bedrock request answered `500
bedrock_credentials_missing`, even after credentials became available
(an ECS task role or IMDS that was not ready yet, an SSO login done
after the proxy started), until a restart.

With this PR the loader only fails when the default chain has no
provider at all. `main` resolves once to report the state at startup,
and attaches the source whether or not that first resolution worked.
`BedrockCredentials::get` does not cache a failure, so each Bedrock
request asks the provider again until it succeeds, then the #4696
refresh-ahead cache takes over. Requests are still never forwarded
unsigned; until credentials arrive they get `500
bedrock_credentials_refresh_failed` instead of
`bedrock_credentials_missing`.

Fixes #4684

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `crates/headroom-proxy/src/main.rs`: `load_bedrock_credentials`
returns the source without resolving; the startup block resolves once,
attaches the source either way, and logs `bedrock_credentials_loaded` or
`bedrock_credentials_unavailable` (message now says the provider is
asked again on each Bedrock request). The no-provider case is unchanged.
- `crates/headroom-proxy/src/bedrock/credentials.rs`: test
`failed_fetch_is_not_cached` (a provider that fails, then succeeds: the
second `get()` returns the credentials and the provider is called twice
in total). It pins the property `main.rs` now relies on.
-
`crates/headroom-proxy/tests/integration_bedrock_startup_credentials.rs`:
runs the real binary (`CARGO_BIN_EXE_headroom-proxy`), because the
startup decision lives in `main.rs` and the in-process
`start_proxy_with_state` harness never runs it. Credentials come from a
wiremock ECS container endpoint (`AWS_CONTAINER_CREDENTIALS_FULL_URI`)
that answers 503 until switched on; all other `AWS_*` variables are
removed and IMDS is disabled. The first invoke must get 500; after the
switch the second must get 200 and reach the wiremock upstream signed
with the late key. The proxy is killed and its temp file removed on
drop.
- `crates/headroom-proxy/src/bedrock/invoke.rs`: module doc for the
refresh-failure case also names "none resolved yet since startup".

## Testing

- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

This is a Rust-only change, so the Python checks above do not apply; the
Rust equivalents were run (Windows 11, Rust 1.95.0).

### Test Output

```text
cargo test -p headroom-proxy --lib --locked bedrock::credentials
test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 349 filtered out

cargo fmt -p headroom-proxy -- --check                               -> exit 0
cargo clippy -p headroom-proxy --all-targets --locked -- -D warnings -> exit 0
```

Full crate suite (`cargo test -p headroom-proxy --locked
--no-fail-fast`, Windows): 585 passed, 1 failed. The one failure is
`integration_path_traversal::forwarder_keeps_requests_under_the_base_prefix`
(`Os { code: 10054, kind: ConnectionReset }`), a Windows-only failure in
a test this change does not touch; it fails the same way in #4694's
Windows run.

Fail before (`main.rs` from `bb8d18a5`, new integration test kept;
`cargo test -p headroom-proxy --locked --test
integration_bedrock_startup_credentials`):

```text
assertion `left == right` failed: credentials served now: {"error":{"type":"bedrock_credentials_missing","message":"AWS credentials not configured; refusing to forward unsigned"}}
  left: 500
 right: 200
test result: FAILED. 0 passed; 1 failed
```

Pass after:

```text
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
```

The unit test `failed_fetch_is_not_cached` passes on `main` too (the
cache already did not store failures); it pins the property the fix
relies on. The integration test above is the fail-before.

## Real Behavior Proof

- Environment: Windows 11, Rust 1.95.0, debug builds of the
`headroom-proxy` binary at `bb8d18a5` (control) and at this branch,
offline.
- Exact command / steps: a local ECS container-credentials endpoint
(`AWS_CONTAINER_CREDENTIALS_FULL_URI=http://127.0.0.1:<port>/creds`)
that answers 503 until a flag file exists and then returns test
credentials (`AKIDPROOF4684`, 1 h expiry); empty `AWS_CONFIG_FILE` /
`AWS_SHARED_CREDENTIALS_FILE`, `AWS_EC2_METADATA_DISABLED=true`, no
other `AWS_*` variables. Proxy: `headroom-proxy --listen 127.0.0.1:<p>
--upstream http://127.0.0.1:1 --enable-bedrock-native true
--bedrock-endpoint http://127.0.0.1:<recorder>`, where the recorder
stores each request's `Authorization` header and answers 200. Request A:
`POST /model/anthropic.claude-3-haiku-20240307-v1:0/invoke` before the
flag exists. Then create the flag. Request B: the same request.
- Observed result: on `bb8d18a5` request B, sent after the credentials
endpoint recovered, still got 500 `bedrock_credentials_missing` with 0
upstream hits; on this branch it got 200 and reached the upstream signed
with `AKIDPROOF4684`. Details:

  | | `bb8d18a5` | this branch |
  |---|---|---|
| startup log | `bedrock_credentials_unavailable` |
`bedrock_credentials_unavailable` |
| A (no credentials yet) | 500 `bedrock_credentials_missing` | 500
`bedrock_credentials_refresh_failed` |
| B (credentials now served) | 500 `bedrock_credentials_missing`, 0
upstream hits | 200, 1 upstream hit, `Authorization: AWS4-HMAC-SHA256
Credential=AKIDPROOF4684/.../us-east-1/bedrock/aws4_request`, log
`bedrock_credentials_refreshed` |

- Not tested: real AWS (IMDS, ECS, SSO); the streaming route over HTTP
(it reads the same `bedrock_credentials` source); load. A profile-based
source that is broken at startup does not recover with this change: in a
first attempt with a `credential_process` profile, the SDK's profile
provider failed to build (this crate does not enable the
`credentials-process` feature) and then answered "profile file
credentials provider initialization error already taken" on later calls,
so the SDK keeps that failure itself. Container and IMDS sources are
asked again on each call.

## Runtime Rollout Safety

- Rollout-managed feature(s): none changed; the existing
`native_bedrock` route only.
- Minimum rollout channel: n/a (no new feature)
- Stable/default behavior changed: yes, narrowly: when credentials are
unavailable at startup, Bedrock requests now ask the provider again on
each request (and succeed once credentials appear) instead of failing
until restart; the error code while unavailable changes from
`bedrock_credentials_missing` to `bedrock_credentials_refresh_failed`.
When credentials resolve at startup nothing changes.
- Kill switch / disable path: none added; disabling `native_bedrock`
removes the route as before.
- Unsafe override required: no
- Qualification impact: none expected; a failed resolution never
forwards an unsigned request.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes (see
Testing)
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- While credentials are unavailable, each Bedrock request runs one
provider attempt (for IMDS or a container endpoint, an HTTP call with
the SDK's own timeouts) before it gets the 500. That is the cost of
recovering without a restart; a maintainer may prefer a backoff or a
background retry instead. The issue's own suggestion was "Retrying the
initial load would also cover the startup-failure case".

Produced with AI assistance (Claude Code) operated by me (HetCreep); the
tests and proof above were run on my machine against a clone of the
repository.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4705"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
HetCreep
2026-11-09 12:32:18 -06:00
committed by GitHub
co-authored by Claude Opus 5.5
parent 1be54c0b46
commit 5438f0035d
4 changed files with 226 additions and 13 deletions
@@ -129,6 +129,50 @@ mod tests {
assert_eq!(calls.load(Ordering::SeqCst), 1);
}
/// Fails until `ready` is set, then returns long-lived credentials.
#[derive(Debug)]
struct LateProvider {
ready: Arc<std::sync::atomic::AtomicBool>,
calls: Arc<AtomicUsize>,
}
impl ProvideCredentials for LateProvider {
fn provide_credentials<'a>(&'a self) -> future::ProvideCredentials<'a>
where
Self: 'a,
{
self.calls.fetch_add(1, Ordering::SeqCst);
if self.ready.load(Ordering::SeqCst) {
future::ProvideCredentials::ready(Ok(Credentials::new(
"AKID_LATE",
"secret",
None,
None,
"late",
)))
} else {
future::ProvideCredentials::ready(Err(CredentialsError::not_loaded("not yet")))
}
}
}
#[tokio::test]
async fn failed_fetch_is_not_cached() {
// A startup resolution that fails must not stick: main.rs keeps
// the source either way and relies on the next get() asking again.
let ready = Arc::new(std::sync::atomic::AtomicBool::new(false));
let calls = Arc::new(AtomicUsize::new(0));
let src = BedrockCredentials::new(LateProvider {
ready: ready.clone(),
calls: calls.clone(),
});
assert!(src.get().await.is_err());
ready.store(true, Ordering::SeqCst);
assert_eq!(src.get().await.unwrap().access_key_id(), "AKID_LATE");
assert_eq!(src.get().await.unwrap().access_key_id(), "AKID_LATE");
assert_eq!(calls.load(Ordering::SeqCst), 2);
}
#[tokio::test]
async fn static_credentials_without_expiry_are_returned() {
let src = BedrockCredentials::new(Credentials::new("AKID_STATIC", "s", None, None, "t"));
+2 -1
View File
@@ -25,7 +25,8 @@
//! `event=bedrock_credentials_missing` at WARN, return `500` with
//! a JSON error body. NEVER forwards an unsigned request.
//! - **Credentials refresh failure** (the provider errors when the
//! cached credentials are near expiry): log
//! cached credentials are near expiry, or when none have been
//! resolved yet since startup): log
//! `event=bedrock_credentials_refresh_failed`, return `500`.
//! - **Envelope parse failure**: log `event=bedrock_envelope_parse_error`,
//! pass the bytes through unchanged. Bedrock will reject anyway,
+26 -12
View File
@@ -83,16 +83,29 @@ async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
// upstream — but the Bedrock invoke handler refuses to forward
// unsigned requests when `bedrock_credentials` is `None`
// (see `bedrock::invoke::handle_invoke`).
// A first resolution that fails keeps the provider too: a failed
// fetch is not cached, so each Bedrock request asks it again and
// the proxy recovers once credentials appear, without a restart.
if config.enable_bedrock_native {
match load_bedrock_credentials(&config).await {
Ok(source) => {
let first = source.get().await;
state = state.with_bedrock_credentials_source(source);
tracing::info!(
event = "bedrock_credentials_loaded",
region = %config.bedrock_region,
profile = ?config.aws_profile,
"AWS credentials resolved for Bedrock SigV4 signing"
);
match first {
Ok(_) => tracing::info!(
event = "bedrock_credentials_loaded",
region = %config.bedrock_region,
profile = ?config.aws_profile,
"AWS credentials resolved for Bedrock SigV4 signing"
),
Err(e) => tracing::warn!(
event = "bedrock_credentials_unavailable",
region = %config.bedrock_region,
profile = ?config.aws_profile,
error = %e,
"AWS credentials not available at startup; each Bedrock request asks the provider again until it succeeds"
),
}
}
Err(e) => {
tracing::warn!(
@@ -181,9 +194,10 @@ fn init_tracing(level: &str) {
/// Uses the `aws-config` default chain (env vars → shared profile
/// file → IMDS / ECS task role). Honours `Config::aws_profile` when
/// set; otherwise the chain picks up `AWS_PROFILE` from the
/// environment automatically. Resolves once here so a missing
/// credential is reported at startup; the returned source keeps the
/// provider for later refreshes.
/// environment automatically. Only fails when the chain has no
/// provider at all; the caller resolves once so a missing credential
/// is reported at startup, and the returned source keeps the provider
/// for later attempts and refreshes.
async fn load_bedrock_credentials(
config: &Config,
) -> Result<headroom_proxy::bedrock::BedrockCredentials, Box<dyn std::error::Error + Send + Sync>> {
@@ -198,9 +212,9 @@ async fn load_bedrock_credentials(
let creds_provider = aws_config
.credentials_provider()
.ok_or("no credentials provider configured")?;
let source = headroom_proxy::bedrock::BedrockCredentials::new(creds_provider);
source.get().await?;
Ok(source)
Ok(headroom_proxy::bedrock::BedrockCredentials::new(
creds_provider,
))
}
async fn shutdown_signal() {
@@ -0,0 +1,154 @@
//! Credentials that are missing when the proxy starts and appear later.
//!
//! The startup sequence lives in `main.rs`, so this test runs the real
//! binary. Credentials come from a local ECS container endpoint
//! (`AWS_CONTAINER_CREDENTIALS_FULL_URI`) that answers 503 until it is
//! switched on; the Bedrock upstream is wiremock. Before the fix the
//! failed startup fetch dropped the provider, so the request sent after
//! the endpoint recovered still got `bedrock_credentials_missing`.
use std::net::TcpListener;
use std::process::{Child, Command, Stdio};
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
use std::time::{Duration, Instant};
use serde_json::json;
use wiremock::matchers::{method, path};
use wiremock::{Mock, MockServer, Request, Respond, ResponseTemplate};
const TEST_MODEL: &str = "anthropic.claude-3-haiku-20240307-v1:0";
/// ECS credentials endpoint: 503 until `ready` is set.
struct LateCredentials {
ready: Arc<AtomicBool>,
}
impl Respond for LateCredentials {
fn respond(&self, _request: &Request) -> ResponseTemplate {
if !self.ready.load(Ordering::SeqCst) {
return ResponseTemplate::new(503);
}
ResponseTemplate::new(200).set_body_json(json!({
"AccessKeyId": "AKIDSTARTUPTEST",
"SecretAccessKey": "secret",
"Token": "session-token",
"Expiration": "2099-01-01T00:00:00Z",
}))
}
}
/// Kills the proxy and removes its empty AWS file when the test ends,
/// pass or fail.
struct ProxyProcess(Child, std::path::PathBuf);
impl Drop for ProxyProcess {
fn drop(&mut self) {
let _ = self.0.kill();
let _ = self.0.wait();
let _ = std::fs::remove_file(&self.1);
}
}
fn free_port() -> u16 {
TcpListener::bind("127.0.0.1:0")
.unwrap()
.local_addr()
.unwrap()
.port()
}
#[tokio::test]
async fn credentials_that_appear_after_startup_are_used() {
let ready = Arc::new(AtomicBool::new(false));
let creds = MockServer::start().await;
Mock::given(method("GET"))
.and(path("/creds"))
.respond_with(LateCredentials {
ready: ready.clone(),
})
.mount(&creds)
.await;
let upstream = MockServer::start().await;
Mock::given(method("POST"))
.respond_with(
ResponseTemplate::new(200)
.insert_header("content-type", "application/json")
.set_body_string(r#"{"id":"msg_x","content":[]}"#),
)
.mount(&upstream)
.await;
let empty = std::env::temp_dir().join(format!("hr-empty-aws-{}", std::process::id()));
std::fs::write(&empty, "").unwrap();
let port = free_port();
let mut cmd = Command::new(env!("CARGO_BIN_EXE_headroom-proxy"));
for (key, _) in std::env::vars() {
if key.starts_with("AWS_") {
cmd.env_remove(key);
}
}
cmd.env("AWS_CONFIG_FILE", &empty)
.env("AWS_SHARED_CREDENTIALS_FILE", &empty)
.env("AWS_EC2_METADATA_DISABLED", "true")
.env(
"AWS_CONTAINER_CREDENTIALS_FULL_URI",
format!("{}/creds", creds.uri()),
)
.args([
"--listen",
&format!("127.0.0.1:{port}"),
"--upstream",
"http://127.0.0.1:1",
"--enable-bedrock-native",
"true",
"--bedrock-endpoint",
&upstream.uri(),
])
.stdout(Stdio::null())
.stderr(Stdio::null());
let _proxy = ProxyProcess(cmd.spawn().expect("spawn headroom-proxy"), empty.clone());
let client = reqwest::Client::new();
let base = format!("http://127.0.0.1:{port}");
let deadline = Instant::now() + Duration::from_secs(60);
while client.get(format!("{base}/healthz")).send().await.is_err() {
assert!(Instant::now() < deadline, "proxy did not start");
tokio::time::sleep(Duration::from_millis(100)).await;
}
let body = serde_json::to_vec(&json!({
"anthropic_version": "bedrock-2023-05-31",
"max_tokens": 16,
"messages": [{"role": "user", "content": "hi"}]
}))
.unwrap();
let invoke = || {
client
.post(format!("{base}/model/{TEST_MODEL}/invoke"))
.header("content-type", "application/json")
.body(body.clone())
.send()
};
let before = invoke().await.unwrap();
assert_eq!(before.status(), 500, "no credentials yet");
ready.store(true, Ordering::SeqCst);
let after = invoke().await.unwrap();
let status = after.status();
let text = after.text().await.unwrap();
assert_eq!(status, 200, "credentials served now: {text}");
let signed = upstream.received_requests().await.unwrap();
assert_eq!(signed.len(), 1);
let auth = signed[0]
.headers
.get("authorization")
.and_then(|v| v.to_str().ok())
.unwrap_or_default();
assert!(
auth.contains("Credential=AKIDSTARTUPTEST/"),
"signed with the late credentials: {auth}"
);
}