mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 12:05:34 +08:00
fix(proxy): refresh Bedrock AWS credentials before they expire (#4696)
## Description
`headroom-proxy` resolved AWS credentials once at startup and signed
every Bedrock request with that one `Credentials` value. Temporary
credentials (IMDS, ECS task role, SSO, assumed role) expire, so after
expiry every Bedrock request was signed with a dead key until the
process was restarted.
This keeps the credentials provider and resolves credentials per request
through a small cache that asks the provider again once the cached
credentials are within 60 seconds of their expiry, the same policy as
`vertex/adc.rs` (`REFRESH_AHEAD_SECS = 60`).
Why a cache here and not just `provide_credentials()` per request: the
provider returned by `SdkConfig::credentials_provider()` does not cache.
In aws-config 1.13.0, `ConfigLoader::load` builds it as
`SharedCredentialsProvider::new(DefaultCredentialsChain::builder()...build().await)`
(`src/lib.rs:974-983`), and `DefaultCredentialsChain::credentials` calls
the provider chain directly
(`src/default_provider/credentials.rs:72-77`; the struct holds only
`provider_chain`, built at `:175-197`). The SDK caches in the separate
`identity_cache` (`src/lib.rs:1060-1064`, `IdentityCache::lazy()`),
which SDK clients use and this proxy does not. Calling the chain on
every request would hit IMDS / the ECS endpoint / SSO each time.
Partially addresses #4684: refreshes credentials after a successful
initial resolution. Startup credential-resolution failure recovery
remains open on that issue.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- New `bedrock/credentials.rs`: `BedrockCredentials` holds a
`SharedCredentialsProvider` and the last `Credentials` it returned.
`get()` returns the cached value unless it is within 60s of expiry,
otherwise asks the provider again (lock held across the fetch, so
concurrent requests share one refresh). Credentials with no expiry
(long-lived keys) are cached for good. Logs
`event=bedrock_credentials_refreshed` per fetch.
- `AppState.bedrock_credentials` is now
`Option<Arc<BedrockCredentials>>`.
`with_bedrock_credentials(Credentials)` keeps its signature (wraps the
static value, so existing tests are unchanged); new
`with_bedrock_credentials_source(BedrockCredentials)`.
- `bedrock/invoke.rs` and `bedrock/invoke_streaming.rs` call
`get().await` per request. A refresh error returns 500 with
`event=bedrock_credentials_refresh_failed` and never forwards unsigned.
The `None` path (`bedrock_credentials_missing`) is unchanged.
- `main.rs`: `load_bedrock_credentials` still resolves once at startup
(fills the cache) so a missing provider or failed resolve is reported at
startup exactly as before (`bedrock_credentials_unavailable`, Bedrock
routes 5xx); on success it now keeps the provider instead of the one
value.
- Tests: 3 unit tests in `bedrock/credentials.rs` and a new
`tests/integration_bedrock_credentials_refresh.rs` (both invoke routes)
using a test `ProvideCredentials` impl that issues `AKID_<n>` with a
1-second expiry and counts calls. No real AWS.
## Testing
- [ ] Unit tests pass (`pytest`) - not applicable, Rust-only change
- [ ] Linting passes (`ruff check .`) - not applicable; `cargo fmt
--check` and `cargo clippy -D warnings` run instead
- [ ] Type checking passes (`mypy headroom`) - not applicable
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
Fail before (source reverted, new test file kept; the test needs the new
API, so it fails at compile time):
```text
error[E0432]: unresolved import `headroom_proxy::bedrock::BedrockCredentials`
--> crates\headroom-proxy\tests\integration_bedrock_credentials_refresh.rs:13:5
error[E0599]: no method named `with_bedrock_credentials_source` found for struct `AppState` in the current scope
--> crates\headroom-proxy\tests\integration_bedrock_credentials_refresh.rs:66:15
error: could not compile `headroom-proxy` (test "integration_bedrock_credentials_refresh") due to 2 previous errors
```
The runtime fail-before is in the Real Behavior Proof below (unpatched
binary signs the second request with the expired key).
Pass after:
```text
$ cargo test -p headroom-proxy --locked --test integration_bedrock_credentials_refresh
test invoke_streaming_signs_with_refreshed_credentials_after_expiry ... ok
test invoke_signs_with_refreshed_credentials_after_expiry ... ok
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.33s
$ cargo test -p headroom-proxy --locked --lib bedrock::credentials
test bedrock::credentials::tests::static_credentials_without_expiry_are_returned ... ok
test bedrock::credentials::tests::expiring_credentials_are_replaced_on_next_get ... ok
test bedrock::credentials::tests::long_lived_credentials_are_cached ... ok
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 349 filtered out; finished in 0.09s
```
Full crate suite (`cargo test -p headroom-proxy --locked
--no-fail-fast`): 42 test binaries, 581 passed, 1 failed. The one
failure is
`integration_path_traversal::forwarder_keeps_requests_under_the_base_prefix`
(`Os { code: 10054, kind: ConnectionReset, ... }` at
`integration_path_traversal.rs:48`, a raw-TCP read on the catch-all
forwarder). It is a known Windows-only failure on unmodified `main` and
does not touch the Bedrock path; I did not re-run it on `main` for this
PR.
```text
$ cargo fmt -p headroom-proxy -- --check
(no output, exit 0)
$ cargo clippy -p headroom-proxy --all-targets --locked -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 18s
```
## Real Behavior Proof
- Environment: Windows 11, Rust 1.95.0 (pinned by rust-toolchain.toml),
debug build of the `headroom-proxy` binary from `main` at 12864fc8
(before) and from this branch (after). No real AWS: a local Python HTTP
server on 127.0.0.1 stands in for the ECS container-credentials endpoint
(every call issues a new fake key `AKIDFAKE<n>` with `Expiration` 3
seconds later) and for Bedrock runtime (logs which access key id signed
each request). Real AWS config was shut out: `AWS_CONFIG_FILE` and
`AWS_SHARED_CREDENTIALS_FILE` pointed at an empty file,
`AWS_EC2_METADATA_DISABLED=true`, no `AWS_*` key variables set, so the
real `aws-config` default chain resolved through its ECS provider via
`AWS_CONTAINER_CREDENTIALS_FULL_URI=http://127.0.0.1:18923/creds`.
- Exact command / steps: `headroom-proxy --listen 127.0.0.1:18924
--upstream http://127.0.0.1:18923 --bedrock-endpoint
http://127.0.0.1:18923` with the environment above, then two `curl -X
POST
http://127.0.0.1:18924/model/anthropic.claude-3-haiku-20240307-v1:0/invoke`
with a minimal Anthropic-on-Bedrock body, 5 seconds apart (so the first
key has expired before the second request); run once with each binary.
- Observed result: before the patch the credentials endpoint is called
once at startup and both requests are signed with `AKIDFAKE1`, the
second one about 6 seconds after a 3-second expiry (`creds-endpoint call
1: issued AKIDFAKE1` / `bedrock-mock .../invoke: signed with AKIDFAKE1`
/ `bedrock-mock .../invoke: signed with AKIDFAKE1`). After the patch:
`creds-endpoint call 1: issued AKIDFAKE1` (startup) / `creds-endpoint
call 2: issued AKIDFAKE2` / `bedrock-mock .../invoke: signed with
AKIDFAKE2` / `creds-endpoint call 3: issued AKIDFAKE3` / `bedrock-mock
.../invoke: signed with AKIDFAKE3`, both requests HTTP 200, proxy log
shows 1 `bedrock_credentials_loaded` and 3
`bedrock_credentials_refreshed`. (The first request also refreshes
because a 3-second credential is already inside the 60-second
refresh-ahead window.)
- Not tested: real AWS, STS, IMDS, SSO or an assumed role; a real
signature check by Bedrock (the mock only reads the key id from
`Authorization`); the streaming route through the binary (covered by the
integration test only); the refresh-failure 500 path through the binary;
behaviour under concurrent requests during a refresh; Linux/macOS.
## Runtime Rollout Safety
- Rollout-managed feature(s): none; this changes how the existing native
Bedrock route obtains credentials.
- Minimum rollout channel: n/a (no flag; applies wherever
`--enable-bedrock-native` is on, which is the default).
- Stable/default behavior changed: yes, for the better on long-running
processes: credentials are re-resolved from the same provider chain
within 60s of expiry instead of never. With non-expiring credentials
(static access keys) behaviour is unchanged (one resolve at startup,
cached). Startup behaviour on missing credentials is unchanged. New
error case: if a refresh fails, the request gets 500
`bedrock_credentials_refresh_failed` instead of being signed with an
expired key.
- Kill switch / disable path: `--enable-bedrock-native false` (existing)
disables the route; no separate switch for the refresh.
- Unsafe override required: no.
- Qualification impact: none expected; Bedrock tests use the unchanged
`with_bedrock_credentials(Credentials)` helper.
- Rollback path: revert this commit.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (no
user-facing docs describe the credential lifetime; module docs updated)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes (except
the known Windows-only `integration_path_traversal` failure noted above)
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
- Out of scope: the inbound-auth side of the Bedrock route (how the
proxy authenticates or classifies incoming client requests,
`bedrock/auth_mode_layer.rs`) is not touched.
- Left for a maintainer decision: when the startup resolve fails, the
proxy still keeps no provider and the Bedrock routes 5xx until restart,
as before (the issue mentions this too). Keeping the provider and
retrying per request would fix that, but it would also make every
Bedrock request on a box with no AWS credentials walk the whole chain
(IMDS timeouts included), so I left the startup behaviour as it is.
- When a refresh fails while the cached credentials have not yet
actually expired (inside the 60s window), this returns 500 rather than
using the still-valid cached value, matching the project's
no-silent-fallback rule. Easy to change if you prefer the fallback.
- Produced with AI assistance (Claude Code) operated by me (HetCreep);
the tests and proof above were run on my machine against a clone of the
repository.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4696"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
08e66222c9
commit
adc6724966
@@ -0,0 +1,138 @@
|
||||
//! AWS credentials for Bedrock SigV4 signing, refreshed before expiry.
|
||||
//!
|
||||
//! The `aws-config` default chain can return temporary credentials
|
||||
//! (IMDS, ECS task role, SSO, assumed role). Holding one resolved
|
||||
//! `Credentials` value for the life of the process means every request
|
||||
//! after expiry is signed with a dead key. This source keeps the
|
||||
//! provider and the last credentials it returned, and asks the
|
||||
//! provider again once the cached value is within
|
||||
//! [`REFRESH_AHEAD_SECS`] of its expiry (the same policy as
|
||||
//! `vertex::adc`). Credentials without an expiry (long-lived access
|
||||
//! keys, or the static test credentials) are cached for good.
|
||||
//!
|
||||
//! The provider returned by `SdkConfig::credentials_provider()` does
|
||||
//! not cache on its own (the SDK caches in the client's identity
|
||||
//! cache, which this proxy does not use), so the cache lives here.
|
||||
|
||||
use std::time::{Duration, SystemTime};
|
||||
|
||||
use aws_credential_types::provider::error::CredentialsError;
|
||||
use aws_credential_types::provider::{ProvideCredentials, SharedCredentialsProvider};
|
||||
use aws_credential_types::Credentials;
|
||||
use tokio::sync::Mutex;
|
||||
|
||||
/// Refresh credentials this many seconds before their expiry.
|
||||
pub const REFRESH_AHEAD_SECS: u64 = 60;
|
||||
|
||||
/// A credentials provider plus the last credentials it returned.
|
||||
#[derive(Debug)]
|
||||
pub struct BedrockCredentials {
|
||||
provider: SharedCredentialsProvider,
|
||||
cached: Mutex<Option<Credentials>>,
|
||||
}
|
||||
|
||||
impl BedrockCredentials {
|
||||
pub fn new(provider: impl ProvideCredentials + 'static) -> Self {
|
||||
Self {
|
||||
provider: SharedCredentialsProvider::new(provider),
|
||||
cached: Mutex::new(None),
|
||||
}
|
||||
}
|
||||
|
||||
/// Return credentials that are not within the refresh-ahead window
|
||||
/// of their expiry, asking the provider again when needed. The lock
|
||||
/// is held across the fetch so concurrent requests share one refresh.
|
||||
pub async fn get(&self) -> Result<Credentials, CredentialsError> {
|
||||
let mut guard = self.cached.lock().await;
|
||||
if let Some(c) = guard.as_ref() {
|
||||
if fresh(c) {
|
||||
return Ok(c.clone());
|
||||
}
|
||||
}
|
||||
let creds = self.provider.provide_credentials().await?;
|
||||
tracing::info!(
|
||||
event = "bedrock_credentials_refreshed",
|
||||
expires_in_secs = creds
|
||||
.expiry()
|
||||
.and_then(|e| e.duration_since(SystemTime::now()).ok())
|
||||
.map(|d| d.as_secs()),
|
||||
);
|
||||
*guard = Some(creds.clone());
|
||||
Ok(creds)
|
||||
}
|
||||
}
|
||||
|
||||
fn fresh(c: &Credentials) -> bool {
|
||||
match c.expiry() {
|
||||
None => true,
|
||||
Some(expiry) => match expiry.duration_since(SystemTime::now()) {
|
||||
Ok(remaining) => remaining > Duration::from_secs(REFRESH_AHEAD_SECS),
|
||||
Err(_) => false,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use aws_credential_types::provider::future;
|
||||
use std::sync::atomic::{AtomicUsize, Ordering};
|
||||
use std::sync::Arc;
|
||||
|
||||
/// Returns `AKID_<n>` on the n-th call, expiring `lifetime` from now.
|
||||
#[derive(Debug)]
|
||||
struct CountingProvider {
|
||||
calls: Arc<AtomicUsize>,
|
||||
lifetime: Duration,
|
||||
}
|
||||
|
||||
impl ProvideCredentials for CountingProvider {
|
||||
fn provide_credentials<'a>(&'a self) -> future::ProvideCredentials<'a>
|
||||
where
|
||||
Self: 'a,
|
||||
{
|
||||
let n = self.calls.fetch_add(1, Ordering::SeqCst) + 1;
|
||||
future::ProvideCredentials::ready(Ok(Credentials::new(
|
||||
format!("AKID_{n}"),
|
||||
"secret",
|
||||
Some("token".into()),
|
||||
Some(SystemTime::now() + self.lifetime),
|
||||
"counting",
|
||||
)))
|
||||
}
|
||||
}
|
||||
|
||||
fn source(lifetime: Duration) -> (BedrockCredentials, Arc<AtomicUsize>) {
|
||||
let calls = Arc::new(AtomicUsize::new(0));
|
||||
let provider = CountingProvider {
|
||||
calls: calls.clone(),
|
||||
lifetime,
|
||||
};
|
||||
(BedrockCredentials::new(provider), calls)
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn expiring_credentials_are_replaced_on_next_get() {
|
||||
// 1s of life is inside the 60s refresh-ahead window.
|
||||
let (src, calls) = source(Duration::from_secs(1));
|
||||
assert_eq!(src.get().await.unwrap().access_key_id(), "AKID_1");
|
||||
assert_eq!(src.get().await.unwrap().access_key_id(), "AKID_2");
|
||||
assert_eq!(calls.load(Ordering::SeqCst), 2);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn long_lived_credentials_are_cached() {
|
||||
let (src, calls) = source(Duration::from_secs(3600));
|
||||
for _ in 0..3 {
|
||||
assert_eq!(src.get().await.unwrap().access_key_id(), "AKID_1");
|
||||
}
|
||||
assert_eq!(calls.load(Ordering::SeqCst), 1);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn static_credentials_without_expiry_are_returned() {
|
||||
let src = BedrockCredentials::new(Credentials::new("AKID_STATIC", "s", None, None, "t"));
|
||||
assert_eq!(src.get().await.unwrap().access_key_id(), "AKID_STATIC");
|
||||
assert_eq!(src.get().await.unwrap().access_key_id(), "AKID_STATIC");
|
||||
}
|
||||
}
|
||||
@@ -24,6 +24,9 @@
|
||||
//! - **Missing credentials**: log
|
||||
//! `event=bedrock_credentials_missing` at WARN, return `500` with
|
||||
//! a JSON error body. NEVER forwards an unsigned request.
|
||||
//! - **Credentials refresh failure** (the provider errors when the
|
||||
//! cached credentials are near expiry): log
|
||||
//! `event=bedrock_credentials_refresh_failed`, return `500`.
|
||||
//! - **Envelope parse failure**: log `event=bedrock_envelope_parse_error`,
|
||||
//! pass the bytes through unchanged. Bedrock will reject anyway,
|
||||
//! but the failure is the customer's, not ours — we just route.
|
||||
@@ -264,7 +267,7 @@ pub async fn handle_invoke(
|
||||
};
|
||||
|
||||
// Resolve credentials. No silent fallback: missing creds → 5xx.
|
||||
let creds = match state.bedrock_credentials.as_ref() {
|
||||
let creds_source = match state.bedrock_credentials.as_ref() {
|
||||
Some(c) => c.clone(),
|
||||
None => {
|
||||
tracing::warn!(
|
||||
@@ -281,6 +284,24 @@ pub async fn handle_invoke(
|
||||
);
|
||||
}
|
||||
};
|
||||
let creds = match creds_source.get().await {
|
||||
Ok(c) => c,
|
||||
Err(e) => {
|
||||
tracing::error!(
|
||||
event = "bedrock_credentials_refresh_failed",
|
||||
request_id = %request_id,
|
||||
model_id = %model_id,
|
||||
error = %e,
|
||||
"bedrock invoke: AWS credentials refresh failed; refusing to forward unsigned"
|
||||
);
|
||||
capture::finalize_rejected(pending_stats.take());
|
||||
return error_response(
|
||||
StatusCode::INTERNAL_SERVER_ERROR,
|
||||
"bedrock_credentials_refresh_failed",
|
||||
&e.to_string(),
|
||||
);
|
||||
}
|
||||
};
|
||||
|
||||
// Build the headers we sign + forward. Start from the inbound
|
||||
// headers, drop the ones the upstream client manages, then sign.
|
||||
@@ -294,7 +315,7 @@ pub async fn handle_invoke(
|
||||
method: method.as_str(),
|
||||
url: &upstream_url,
|
||||
region: &state.config.bedrock_region,
|
||||
credentials: creds.as_ref(),
|
||||
credentials: &creds,
|
||||
body: &outbound_body,
|
||||
extra_signed_headers: &extra_signed_refs,
|
||||
time: SystemTime::now(),
|
||||
|
||||
@@ -235,7 +235,7 @@ pub async fn handle_invoke_streaming(
|
||||
};
|
||||
|
||||
// 3. Resolve credentials. No silent fallback.
|
||||
let creds = match state.bedrock_credentials.as_ref() {
|
||||
let creds_source = match state.bedrock_credentials.as_ref() {
|
||||
Some(c) => c.clone(),
|
||||
None => {
|
||||
tracing::warn!(
|
||||
@@ -252,6 +252,24 @@ pub async fn handle_invoke_streaming(
|
||||
);
|
||||
}
|
||||
};
|
||||
let creds = match creds_source.get().await {
|
||||
Ok(c) => c,
|
||||
Err(e) => {
|
||||
tracing::error!(
|
||||
event = "bedrock_credentials_refresh_failed",
|
||||
request_id = %request_id,
|
||||
model_id = %model_id,
|
||||
error = %e,
|
||||
"bedrock invoke-streaming: AWS credentials refresh failed; refusing to forward unsigned"
|
||||
);
|
||||
capture::finalize_rejected(pending_stats.take());
|
||||
return error_response(
|
||||
StatusCode::INTERNAL_SERVER_ERROR,
|
||||
"bedrock_credentials_refresh_failed",
|
||||
&e.to_string(),
|
||||
);
|
||||
}
|
||||
};
|
||||
|
||||
// 4. Build the headers we sign + forward.
|
||||
let extra_signed: Vec<(String, String)> = collect_signed_headers(&headers, &upstream_url);
|
||||
@@ -264,7 +282,7 @@ pub async fn handle_invoke_streaming(
|
||||
method: method.as_str(),
|
||||
url: &upstream_url,
|
||||
region: &state.config.bedrock_region,
|
||||
credentials: creds.as_ref(),
|
||||
credentials: &creds,
|
||||
body: &outbound_body,
|
||||
extra_signed_headers: &extra_signed_refs,
|
||||
time: SystemTime::now(),
|
||||
|
||||
@@ -46,6 +46,7 @@
|
||||
//! [`eventstream_to_sse`].
|
||||
|
||||
pub mod auth_mode_layer;
|
||||
pub mod credentials;
|
||||
pub mod envelope;
|
||||
pub mod eventstream;
|
||||
pub mod eventstream_to_sse;
|
||||
@@ -64,6 +65,7 @@ pub(crate) struct CompressionStats {
|
||||
}
|
||||
|
||||
pub use auth_mode_layer::classify_and_attach_auth_mode;
|
||||
pub use credentials::BedrockCredentials;
|
||||
pub use envelope::{BedrockEnvelope, EnvelopeError};
|
||||
pub use eventstream::{
|
||||
parse as parse_eventstream, CrcValidation, EventStreamMessage, EventStreamParser, HeaderValue,
|
||||
|
||||
@@ -77,15 +77,16 @@ async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
|
||||
let mut state = AppState::new(config.clone())?;
|
||||
|
||||
// PR-D1: resolve AWS credentials at startup via the `aws-config`
|
||||
// default chain. Loaded once so per-request signing is cheap.
|
||||
// default chain. The provider is kept and asked again before the
|
||||
// cached credentials expire (`bedrock::credentials`).
|
||||
// Failure is NOT fatal — the proxy may run in front of a non-AWS
|
||||
// upstream — but the Bedrock invoke handler refuses to forward
|
||||
// unsigned requests when `bedrock_credentials` is `None`
|
||||
// (see `bedrock::invoke::handle_invoke`).
|
||||
if config.enable_bedrock_native {
|
||||
match load_bedrock_credentials(&config).await {
|
||||
Ok(creds) => {
|
||||
state = state.with_bedrock_credentials(creds);
|
||||
Ok(source) => {
|
||||
state = state.with_bedrock_credentials_source(source);
|
||||
tracing::info!(
|
||||
event = "bedrock_credentials_loaded",
|
||||
region = %config.bedrock_region,
|
||||
@@ -180,12 +181,13 @@ fn init_tracing(level: &str) {
|
||||
/// Uses the `aws-config` default chain (env vars → shared profile
|
||||
/// file → IMDS / ECS task role). Honours `Config::aws_profile` when
|
||||
/// set; otherwise the chain picks up `AWS_PROFILE` from the
|
||||
/// environment automatically.
|
||||
/// environment automatically. Resolves once here so a missing
|
||||
/// credential is reported at startup; the returned source keeps the
|
||||
/// provider for later refreshes.
|
||||
async fn load_bedrock_credentials(
|
||||
config: &Config,
|
||||
) -> Result<aws_credential_types::Credentials, Box<dyn std::error::Error + Send + Sync>> {
|
||||
) -> Result<headroom_proxy::bedrock::BedrockCredentials, Box<dyn std::error::Error + Send + Sync>> {
|
||||
use aws_config::BehaviorVersion;
|
||||
use aws_credential_types::provider::ProvideCredentials;
|
||||
|
||||
let mut loader = aws_config::defaults(BehaviorVersion::latest())
|
||||
.region(aws_config::Region::new(config.bedrock_region.clone()));
|
||||
@@ -196,8 +198,9 @@ async fn load_bedrock_credentials(
|
||||
let creds_provider = aws_config
|
||||
.credentials_provider()
|
||||
.ok_or("no credentials provider configured")?;
|
||||
let creds = creds_provider.provide_credentials().await?;
|
||||
Ok(creds)
|
||||
let source = headroom_proxy::bedrock::BedrockCredentials::new(creds_provider);
|
||||
source.get().await?;
|
||||
Ok(source)
|
||||
}
|
||||
|
||||
async fn shutdown_signal() {
|
||||
|
||||
@@ -54,14 +54,15 @@ use headroom_core::compression_policy::CompressionPolicy;
|
||||
pub struct AppState {
|
||||
pub config: Arc<Config>,
|
||||
pub client: reqwest::Client,
|
||||
/// PR-D1: AWS credentials resolved at startup via the
|
||||
/// `aws-config` default chain. `None` when the proxy boots
|
||||
/// PR-D1: AWS credentials source (the `aws-config` default chain,
|
||||
/// asked again before the cached credentials expire; see
|
||||
/// `bedrock::credentials`). `None` when the proxy boots
|
||||
/// without AWS creds available (operator running locally
|
||||
/// against a non-Bedrock upstream); the Bedrock invoke handler
|
||||
/// returns 5xx with a structured `event=bedrock_credentials_missing`
|
||||
/// log so failures are LOUD — no silent fallback to unsigned
|
||||
/// requests.
|
||||
pub bedrock_credentials: Option<Arc<aws_credential_types::Credentials>>,
|
||||
pub bedrock_credentials: Option<Arc<crate::bedrock::BedrockCredentials>>,
|
||||
/// PR-E6: per-session structural-hash LRU for the cache-bust
|
||||
/// drift detector. Bounded to 1000 sessions in production. The
|
||||
/// detector is read-only — observing it never mutates the
|
||||
@@ -161,8 +162,17 @@ impl AppState {
|
||||
/// modified state; intended to be chained off `AppState::new`.
|
||||
/// Tests that don't exercise the Bedrock route can leave
|
||||
/// credentials unset (the catch-all paths never read them).
|
||||
pub fn with_bedrock_credentials(mut self, creds: aws_credential_types::Credentials) -> Self {
|
||||
self.bedrock_credentials = Some(Arc::new(creds));
|
||||
pub fn with_bedrock_credentials(self, creds: aws_credential_types::Credentials) -> Self {
|
||||
self.with_bedrock_credentials_source(crate::bedrock::BedrockCredentials::new(creds))
|
||||
}
|
||||
|
||||
/// Attach a credentials source that is asked again before the
|
||||
/// cached credentials expire (temporary IMDS / ECS / SSO creds).
|
||||
pub fn with_bedrock_credentials_source(
|
||||
mut self,
|
||||
source: crate::bedrock::BedrockCredentials,
|
||||
) -> Self {
|
||||
self.bedrock_credentials = Some(Arc::new(source));
|
||||
self
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,114 @@
|
||||
//! Temporary AWS credentials are refreshed between Bedrock requests.
|
||||
//!
|
||||
//! A test provider hands out `AKID_<n>` credentials that expire one
|
||||
//! second after issue (inside the refresh-ahead window), so each
|
||||
//! request must ask the provider again and be signed with the new key.
|
||||
//! No real AWS: the upstream is wiremock.
|
||||
|
||||
mod common;
|
||||
|
||||
use aws_credential_types::provider::{future, ProvideCredentials};
|
||||
use aws_credential_types::Credentials;
|
||||
use common::start_proxy_with_state;
|
||||
use headroom_proxy::bedrock::BedrockCredentials;
|
||||
use serde_json::json;
|
||||
use std::sync::atomic::{AtomicUsize, Ordering};
|
||||
use std::sync::Arc;
|
||||
use std::time::{Duration, SystemTime};
|
||||
use url::Url;
|
||||
use wiremock::matchers::method;
|
||||
use wiremock::{Mock, MockServer, ResponseTemplate};
|
||||
|
||||
const TEST_MODEL: &str = "anthropic.claude-3-haiku-20240307-v1:0";
|
||||
|
||||
#[derive(Debug)]
|
||||
struct ShortLivedProvider {
|
||||
calls: Arc<AtomicUsize>,
|
||||
}
|
||||
|
||||
impl ProvideCredentials for ShortLivedProvider {
|
||||
fn provide_credentials<'a>(&'a self) -> future::ProvideCredentials<'a>
|
||||
where
|
||||
Self: 'a,
|
||||
{
|
||||
let n = self.calls.fetch_add(1, Ordering::SeqCst) + 1;
|
||||
future::ProvideCredentials::ready(Ok(Credentials::new(
|
||||
format!("AKID_{n}"),
|
||||
"secret",
|
||||
Some("session-token".into()),
|
||||
Some(SystemTime::now() + Duration::from_secs(1)),
|
||||
"short-lived-test",
|
||||
)))
|
||||
}
|
||||
}
|
||||
|
||||
async fn signed_key_ids(action: &str) -> (Vec<String>, usize) {
|
||||
let upstream = MockServer::start().await;
|
||||
Mock::given(method("POST"))
|
||||
.respond_with(
|
||||
ResponseTemplate::new(200)
|
||||
.insert_header("content-type", "application/json")
|
||||
.set_body_string(r#"{"id":"msg_x","content":[]}"#),
|
||||
)
|
||||
.mount(&upstream)
|
||||
.await;
|
||||
let calls = Arc::new(AtomicUsize::new(0));
|
||||
let provider = ShortLivedProvider {
|
||||
calls: calls.clone(),
|
||||
};
|
||||
let endpoint: Url = upstream.uri().parse().unwrap();
|
||||
let proxy = start_proxy_with_state(
|
||||
&upstream.uri(),
|
||||
|c| {
|
||||
c.bedrock_endpoint = Some(endpoint);
|
||||
c.compression_mode = headroom_proxy::config::CompressionMode::Off;
|
||||
},
|
||||
|s| s.with_bedrock_credentials_source(BedrockCredentials::new(provider)),
|
||||
)
|
||||
.await;
|
||||
|
||||
let body = serde_json::to_vec(&json!({
|
||||
"anthropic_version": "bedrock-2023-05-31",
|
||||
"max_tokens": 16,
|
||||
"messages": [{"role": "user", "content": "hi"}]
|
||||
}))
|
||||
.unwrap();
|
||||
for _ in 0..2 {
|
||||
let resp = reqwest::Client::new()
|
||||
.post(format!("{}/model/{TEST_MODEL}/{action}", proxy.url()))
|
||||
.header("content-type", "application/json")
|
||||
.body(body.clone())
|
||||
.send()
|
||||
.await
|
||||
.unwrap();
|
||||
let _ = resp.bytes().await;
|
||||
}
|
||||
proxy.shutdown().await;
|
||||
|
||||
let keys = upstream
|
||||
.received_requests()
|
||||
.await
|
||||
.unwrap()
|
||||
.iter()
|
||||
.map(|r| {
|
||||
let auth = r.headers.get("authorization").unwrap().to_str().unwrap();
|
||||
let start = auth.find("Credential=").unwrap() + "Credential=".len();
|
||||
auth[start..].split('/').next().unwrap().to_string()
|
||||
})
|
||||
.collect();
|
||||
(keys, calls.load(Ordering::SeqCst))
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn invoke_signs_with_refreshed_credentials_after_expiry() {
|
||||
let (keys, calls) = signed_key_ids("invoke").await;
|
||||
assert_eq!(keys, vec!["AKID_1", "AKID_2"]);
|
||||
assert_eq!(calls, 2);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn invoke_streaming_signs_with_refreshed_credentials_after_expiry() {
|
||||
let (keys, calls) = signed_key_ids("invoke-with-response-stream").await;
|
||||
assert_eq!(keys, vec!["AKID_1", "AKID_2"]);
|
||||
assert_eq!(calls, 2);
|
||||
}
|
||||
Reference in New Issue
Block a user