Commit Graph
2 Commits
Author SHA1 Message Date
Rod Boev 871bbde37a fix(proxy): reject Anthropic batch operations on Copilot (#3318)
## Description

Anthropic batch routes followed the resolved Anthropic target to GitHub
Copilot, where the batch surface isn't available, and returned an opaque
upstream failure. This change returns a stable unsupported response
before batch parsing or mutation while preserving normal Copilot-hosted
messages and explicit non-Copilot Anthropic overrides.

Closes #3278

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Added one handler-level capability check for Copilot-resolved
Anthropic batch routes.
- Applied the response consistently to batch create, list, get, cancel,
and results routes.
- Added regressions for Copilot batch routes and explicit Anthropic
overrides. Existing provider-registry coverage continues to verify
normal message target resolution.

## Testing

- [x] Unit tests pass (`uv run pytest
tests/test_proxy_batch_integration.py tests/test_copilot_auth.py
tests/test_provider_registry.py -q`)
- [x] Linting passes (`uv run ruff check
headroom/proxy/handlers/anthropic.py
tests/test_proxy_batch_integration.py tests/test_copilot_auth.py`)
- [ ] Type checking passes (`uv run mypy headroom`)
- [x] New tests added for new functionality when applicable
- [ ] Manual testing performed

### Test Output

```text
uv run python -m pytest tests/test_proxy_batch_integration.py tests/test_copilot_auth.py tests/test_provider_registry.py -q
139 passed, 14 skipped, 1 warning in 7.78s

uv run ruff check headroom/proxy/handlers/anthropic.py tests/test_proxy_batch_integration.py tests/test_copilot_auth.py
All checks passed!

git diff --check
completed without whitespace errors
```

## Real Behavior Proof

- Environment: Windows, focused Headroom test client, no live provider
credential
- Exact command / steps: send Anthropic batch create and the remaining
batch routes with the resolved target set to a Copilot host, then repeat
with an explicit non-Copilot Anthropic target
- Observed result: the five registered Anthropic batch routes return
HTTP 501 with
`{"type":"error","error":{"type":"api_error","message":"Anthropic batch
operations are not supported for Copilot targets."}}` before body
parsing or upstream access; with OpenAI forced to
`https://api.githubcopilot.com`, the explicit
`https://api.anthropic.com` override forwards to
`https://api.anthropic.com/v1/messages/batches`. The focused run
reported `139 passed, 14 skipped, 1 warning` and the route test reported
`6 passed, 17 deselected, 1 warning`.
- Not tested: live Copilot tenant

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Unsafe override required: no
- Minimum rollout channel: standard release after focused CI checks pass
- Stable/default behavior changed: Copilot-resolved Anthropic batch
routes now return a deterministic unsupported response before upstream
access
- Kill switch / disable path: revert the handler guard or disable the
surrounding batch route configuration
- Qualification impact: ordinary Copilot messages and explicit Anthropic
targets keep their existing routing paths
- Rollback path: revert the handler guard and its focused regressions;
no persisted-state migration is required

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

This follows the Anthropic message-target routing introduced in [PR
#3258](https://github.com/headroomlabs-ai/headroom/pull/3258). Headroom
generates changelog entries from conventional commits. The provider's
hosted batch capability remains an external contract; local proof covers
Headroom's deterministic guard and route behavior.
2026-09-24 23:56:55 -05:00
chopratejas 95fd6d8688 Add multi-provider batch API support with CCR post-processing
This commit adds comprehensive batch API support for all three major LLM
providers (Anthropic, OpenAI, Google/Gemini) with integrated CCR
(Compress-Cache-Retrieve) functionality for asynchronous batch processing.

## Batch CCR Post-Processing Architecture

When batch APIs are used, responses are processed asynchronously. If the
model calls the CCR retrieval tool (`headroom_retrieve`) within a batch
response, the system now handles this automatically:

1. **Batch Submit**: Request context (messages, tools, model) is stored
   in BatchContextStore keyed by batch_id
2. **Batch Results**: When results are retrieved, CCR tool calls are
   detected in the responses
3. **Continuation**: For each CCR tool call, the system executes local
   retrieval and makes a continuation API call to complete the response
4. **Result Update**: The batch result is updated with the complete
   response, transparent to the caller

## New Components

- `headroom/ccr/batch_store.py`: TTL-based context storage for batch
  requests, enabling CCR retrieval during result processing
- `headroom/ccr/batch_processor.py`: Processes batch results, detects
  CCR tool calls across all provider formats, executes continuations

## Provider Support

### Anthropic
- POST /v1/messages/batches (create with compression)
- GET /v1/messages/batches (list)
- GET /v1/messages/batches/{id} (status)
- GET /v1/messages/batches/{id}/results (with CCR post-processing)

### OpenAI
- POST /v1/batches (create)
- GET /v1/batches (list)
- GET /v1/batches/{id} (status)
- Batch file upload/download support

### Google/Gemini
- Native API support: /v1beta/models/{model}:generateContent
- Batch API: /v1beta/models/{model}:batchGenerateContent
- Token counting: /v1beta/models/{model}:countTokens
- OpenAI-compatible endpoint support

## CCR Enhancements

- Added Google/Gemini format support to response_handler.py
- Extended tool_injection.py with multiple marker patterns for different
  compressors (SmartCrusher, TextCompressor, LogCompressor, etc.)
- Added Google functionCall/functionResponse handling

## Proxy Server Updates

- Added Gemini native API handlers alongside OpenAI-compatible endpoints
- Integrated batch context storage on submission
- Added batch result processing with CCR continuation
- Rate limiting and metrics tracking for all providers

## Test Coverage

Added comprehensive integration tests (all skip gracefully without API keys):
- test_proxy_batch_integration.py: Anthropic and OpenAI batch APIs
- test_proxy_gemini_integration.py: Gemini via OpenAI-compatible endpoint
- test_proxy_gemini_native_integration.py: Gemini native API
- test_proxy_count_tokens_integration.py: Token counting endpoint
- test_proxy_openai_responses_integration.py: OpenAI responses API
- test_proxy_passthrough_integration.py: Passthrough endpoints

## Compression Results (Real API Testing)

- Token savings: 83-98% on tool result content
- CCR tool injection: Working across all providers
- Model behavior: OpenAI gpt-4o-mini successfully called headroom_retrieve
  when presented with compressed data, proving end-to-end CCR functionality
2026-01-24 11:41:18 -08:00