Commit Graph
26 Commits
Author SHA1 Message Date
1dc7a183f5 fix(copilot): honour GitHub's advertised Business/Enterprise host by default (#4155)
## Description

Supersedes #3403 (parked, stacked on the conflicting #3402) as a
standalone change on `main`.

GitHub's token exchange advertises a per-plan chat host in
`endpoints.api`: `api.business.githubcopilot.com` for Business seats and
`api.enterprise.githubcopilot.com` for Enterprise seats, and GitHub's
own clients route with it. Since #2441 Headroom has folded both into the
generic `api.githubcopilot.com`, because the segmented hosts lagged on
newer Claude models at the time.

That fold breaks enterprises on GitHub's [subscription-based network
routing](https://docs.github.com/en/copilot/how-tos/administer-copilot/manage-for-enterprise/manage-access/manage-network-access):
their firewall allows only the plan host, the generic host is
unreachable, and the failure looks like a random connection error. It
also made the VS Code guide's Enterprise section wrong, which said
Headroom uses the advertised URL.

This PR honours the Business and Enterprise advertised hosts by default
and adds `GITHUB_COPILOT_USE_ADVERTISED_HOST=0` (also
`false`/`no`/`off`) to restore the generic-host fold for seats where the
segmented host still lags. An explicit `GITHUB_COPILOT_API_URL` or
enterprise domain still wins over both. The individual host keeps being
folded (#610 is unchanged).

Refs #2441, #610, #3352 (an Enterprise seat behind a corporate firewall;
the inline-completions 401 there is #4144).

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Documentation update

## Changes Made

- `headroom/copilot_auth.py`:
`_subscription_api_url_from_user_info_payload` returns the advertised
host for the two plan hosts unless `GITHUB_COPILOT_USE_ADVERTISED_HOST`
opts out, and logs the chosen host with the opt-out next to it. That
line is emitted where the host is resolved: the `wrap` process for the
subscription lanes, which does not write to `proxy.log`. The dedicated
proxy's `/health` (`config.openai_api_url`) is where the chosen host
shows up. The generic and individual hosts still normalize to
`DEFAULT_API_URL`. `resolve_copilot_api_url`'s docstring no longer
claims the advertised host is ignored on purpose. Every subscription
lane (`wrap vscode`, `wrap copilot --subscription`, OpenCode
`--copilot-subscription`) already pins `resolution.api_url` into the
dedicated proxy as `--openai-api-url` and `GITHUB_COPILOT_API_URL`, so
they pick this up without changes; `is_copilot_api_url`,
`copilot_completions_base_url` and `_github_host` already treat
`*.githubcopilot.com` as a public Copilot host (verified by an
independent review of the diff).
- `headroom/cli/wrap.py`: `wrap copilot` help text describes the
per-plan routing for `--subscription`; the implicit OAuth lane still
uses the generic host (it performs no token exchange), which the text
now says.
- Docs: `docs/content/docs/vscode-copilot.mdx` Enterprise section is now
true and names the opt-out and the firewall case;
`docs/content/docs/corporate-networks.mdx` lists the plan hosts;
`TESTING-copilot-subscription.md` host section split by lane:
`--subscription` / `wrap vscode` / OpenCode honour the plan host, the
implicit OAuth `wrap copilot` lane is explicitly generic-host (it
performs no token exchange at launch, and `resolve_copilot_api_url` is
pinned network-free by an existing test), and its "open an issue for
auto-detection" paragraph removed.
- Tests: Business and Enterprise honoured; individual and generic still
folded; opt-out restores the fold; explicit pin wins; `wrap copilot
--subscription` pins the advertised host; the implicit OAuth lane stays
on the generic host even when user-info advertises the Enterprise host;
OpenCode subscription lane expectation updated. Three existing tests
that asserted the fold were updated to the new contract.

Follow-ups, not in this PR: `headroom doctor --network` still probes
only `api.githubcopilot.com` (it should probe the resolved plan host);
the implicit OAuth `wrap copilot` lane could resolve the plan host
through the token exchange too.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy`)
- [x] New tests added for new functionality
- [x] Manual testing performed (live Copilot Business seat, 2026-10-13;
see Live verification)

### Test Output

```text
$ pytest tests/test_copilot_auth.py tests/test_cli/test_wrap_copilot.py tests/test_cli/test_wrap_vscode.py \
    tests/test_copilot_vscode_completions_routing.py tests/test_provider_copilot_vscode_config.py \
    tests/test_cli/test_wrap_opencode.py tests/test_proxy_copilot_auth_hooks.py tests/test_provider_proxy_routes.py \
    tests/test_copilot_integration_id_hmac.py tests/test_proxy/test_anthropic_copilot_upstream_auth.py \
    tests/test_copilot_native_mode.py tests/test_copilot_subscription_smoke.py tests/test_cli/test_copilot_auth.py \
    tests/test_cli/test_wrap_persistent.py
449 passed, 4 deselected
# the 4 deselected tests (test_wrap_persistent.py::*recover*) fail identically on the base commit 976aa714
# with AttributeError: 'TaskManifest' object has no attribute 'runtime_kind'; unrelated to this change

$ uvx ruff@0.16.8 check . && uvx ruff@0.16.8 format --check .
All checks passed! / 1777 files already formatted
$ mypy headroom/copilot_auth.py headroom/cli/wrap.py --ignore-missing-imports
Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, repo venv, native core loaded.
- Exact command / steps: the suites above, with the new regressions
failing when `_subscription_api_url_from_user_info_payload` is reverted
to the fold.
- Observed result: the token-exchange and user-info producers, `wrap
vscode`, `wrap copilot --subscription` and the OpenCode lane all hand
the dedicated proxy `https://api.enterprise.githubcopilot.com` (or
`.business.`) as both `--openai-api-url` and `GITHUB_COPILOT_API_URL`;
completions still resolve to `copilot-proxy.githubusercontent.com`.
- Live seat: see "Live verification" below. Not tested: an
Enterprise-plan seat (`api.enterprise.`; same code branch as Business),
and the `wrap copilot --subscription` lane (the test seat has the
Copilot CLI policy off).

### Verification needed before release

No Business or Enterprise seat is available to us. Carrier (Enterprise
seats behind a subscription-routed firewall) runs, on this branch, with
`SSL_CERT_FILE` unset and no `GITHUB_COPILOT_API_URL` exported:

```bash
headroom copilot-auth login            # device flow; approve in the browser
headroom wrap vscode                   # leave running; note the port it prints
curl -s http://127.0.0.1:<port>/health | python3 -m json.tool | grep openai_api_url
#   expected: "openai_api_url": "https://api.enterprise.githubcopilot.com"
# then send one Copilot Chat turn in VS Code (any model) and one inline completion
headroom wrap copilot --subscription -- --model claude-sonnet-4.6 -s -p "Reply with exactly: OK"
#   expected: the banner shows COPILOT_PROVIDER_API_URL=https://api.enterprise.githubcopilot.com and the reply is OK
```

`curl -s http://127.0.0.1:<port>/health` on the dedicated proxy should
report `"openai_api_url": "https://api.enterprise.githubcopilot.com"`
(the "advertised plan host" log line is printed by the wrap process, not
written to `proxy.log`). If a model that the generic host serves is
rejected on the plan host (`400 ... model is not supported`), the #2455
condition still holds for that seat: set
`GITHUB_COPILOT_USE_ADVERTISED_HOST=0`, restart the wrapper, and report
the model and the response body on this PR before it is released.

### Live verification (2026-10-13, Copilot Business seat)

Branch merged with #4154 on current `main`, isolated
`HEADROOM_WORKSPACE_DIR`, `SSL_CERT_FILE` and `GITHUB_COPILOT_API_URL`
unset, `headroom copilot-auth login` device flow.

- `/copilot_internal/user` for the seat: `copilot_plan: business`,
`endpoints.api: https://api.business.githubcopilot.com`. (For about
three hours after the seat was assigned GitHub still advertised
`api.individual.`; the plan host appeared on its own.)
- Resolution: default → `https://api.business.githubcopilot.com`;
`GITHUB_COPILOT_USE_ADVERTISED_HOST=0` →
`https://api.githubcopilot.com`; explicit `GITHUB_COPILOT_API_URL` wins.
`main`'s `copilot_auth.py` resolves the same seat to
`https://api.githubcopilot.com`.
- `headroom wrap vscode`: dedicated proxy started with `--openai-api-url
https://api.business.githubcopilot.com`; `/health` reports it.
- Model sweep through the proxy, one turn per picker model on each
endpoint it lists (plus `gpt-4o-mini`): **36/36 returned 200** on the
plan host (Claude Haiku/Sonnet/Opus 4.x–5.5 on `/v1/messages` and
`/chat/completions`, Gemini 3.7/3.8/4, GPT-5.x/6.x and Grok on
`/responses`, `gpt-5-mini`, `gpt-4o-mini`). No `model_not_supported`, so
the #2441/#2455 lag does not reproduce on Business today.
- Real VS Code (Copilot Chat 0.69): Ask mode on Auto (`gpt-6-luna`),
Claude Sonnet 5.5 chat (`/v1/messages`), and Agent mode with tool calls
(wrote and ran a pytest file) all went to
`api.business.githubcopilot.com`; every Copilot upstream call in the
proxy log used the plan host, none the generic host.
- Found while testing, not caused by this PR and present on `main`:
ghost text / next edits (`copilot-nes-*` on `/chat/completions`) are
sent to CAPI and get `400 model_not_supported` on both the plan host and
the generic host. Fixed separately in #4188.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; env-gated.
- Minimum rollout channel: stable.
- Stable/default behavior changed: yes. Business and Enterprise
subscription sessions now route to their advertised plan host instead of
`api.githubcopilot.com`. Individual seats, explicit pins and GHE domains
are unchanged.
- Kill switch / disable path: `GITHUB_COPILOT_USE_ADVERTISED_HOST=0`
restores the previous fold without a rollback; `GITHUB_COPILOT_API_URL`
pins any host.
- Unsafe override required: no.
- Qualification impact: adds plan-host routing coverage to the Copilot
auth and wrap suites.
- Rollback path: revert the commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-13 15:20:50 -07:00
Rajat NagdaandJerrettDavis 2b2dc1b2c9 fix(copilot): read timezone-naive token expiry as UTC, not host local time (#3216)
## Description

`_parse_expiry` in `headroom/copilot_auth.py` parses an ISO-8601 token
expiry with `datetime.fromisoformat(...).timestamp()`. For a
**timezone-naive** value (no `Z`, no offset), `datetime.timestamp()`
interprets the datetime in the **host's local zone**, so the same expiry
resolves to a different epoch on any non-UTC machine.

`_entry_expired` compares that epoch against `time.time()` (UTC), so the
naive expiry is judged in the wrong frame:

- On a host **ahead** of UTC (e.g. `Asia/Kolkata`, +05:30) a still-valid
  token is treated as already expired — needless re-auth churn.
- On a host **behind** UTC (e.g. US zones) an already-expired token is
  trusted past its expiry, surfacing later as upstream `401`s.

Naive ISO expiries reach this path from the documented
`GITHUB_COPILOT_API_TOKEN_EXPIRES_AT` env var and from credential-store
payloads (`expires_at` / `expiry` / `expires`). The fix treats a naive
datetime as UTC, matching every other ISO parser in the codebase
(`telemetry/traffic_learner._parse_iso_timestamp`,
`proxy/memory_rank_policy.parse_memory_created_at`).

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `_parse_expiry`: when `datetime.fromisoformat` yields a naive
datetime,
  attach `timezone.utc` before calling `.timestamp()`. Aware values and
  the integer/float epoch paths are unchanged.
- Added regression tests that force a non-UTC host zone so they fail on
  any machine before the fix.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom --ignore-missing-imports`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_copilot_auth.py -q
110 passed, 1 warning in 0.81s

# The two naive-path tests fail before the fix (Z-suffix path already worked):
$ git stash push -- headroom/copilot_auth.py && \
  python -m pytest tests/test_copilot_auth.py -q -k "naive_iso_as_utc or entry_expired_uses_utc or zulu_suffix"
tests/test_copilot_auth.py:1635: assert naive == aware
E   assert 1767205800.0 == 1767225600.0
FAILED ...::test_parse_expiry_reads_naive_iso_as_utc_not_local_time
FAILED ...::test_entry_expired_uses_utc_for_naive_iso_expiry
2 failed, 1 passed, 107 deselected

$ python -m ruff check headroom/copilot_auth.py tests/test_copilot_auth.py
All checks passed!

$ mypy headroom --ignore-missing-imports
Success: no issues found in 527 source files
```

## Real Behavior Proof

- Environment: macOS (Darwin arm64), Python 3.13, `TZ=Asia/Kolkata`
(UTC+05:30). Operator sets a naive ISO expiry via the documented
`GITHUB_COPILOT_API_TOKEN_EXPIRES_AT` env var, 1 hour in the future
(UTC).
- Exact command / steps: with `TZ=Asia/Kolkata` and `time.tzset()`,
parse the env value with `copilot_auth._parse_expiry(<naive iso>)` and
evaluate `copilot_auth._entry_expired({"expires_at": <naive iso>})`;
compare the parsed epoch against the intended UTC epoch. Run once with
the fix stashed (before) and once with it applied (after).
- Observed result: BEFORE the fix the naive expiry was misread as local
time and the live token was reported expired; AFTER the fix it resolves
to UTC and is correctly live.
  ```text
  BEFORE fix (same host, naive ISO expiry 1h in the future):
    parsed epoch : 1787451474
    expected epoch: 1787471275
    skew seconds : -19801   (== -05:30, interpreted as local time)
_entry_expired says token is expired? True (WRONG: token is 1h in the
future)

  AFTER fix:
    parsed epoch : 1787471264
    expected epoch: 1787471265
    skew seconds : -1   (sub-second isoformat truncation; == correct)
    _entry_expired says token is expired? False  (correct)
  ```
- Not tested: live GitHub Copilot token exchange against real
credentials (no account handy); the reachable env-var and
credential-payload paths are exercised directly above.

## Runtime Rollout Safety

- Rollout-managed feature(s): none — unconditional correctness fix in
the token-expiry parser.
- Minimum rollout channel: N/A
- Stable/default behavior changed: only for tz-naive ISO expiries, which
were previously misread; UTC hosts see no change.
- Kill switch / disable path: N/A
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A — no
doc surface changed)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-01 16:03:43 -05:00
rNoz 81a8a28da3 fix(copilot): defer keychain auth lookup (#2739)
## Description

Defers macOS Keychain and Linux Secret Service OAuth discovery until
non-platform Copilot credentials fail during `headroom wrap copilot
--subscription`. This prevents needless macOS Keychain prompts when
saved Headroom OAuth authentication already resolves. Closes #2738.

## Type of Change

- [x] Bug fix (non-breaking change fixes an issue)
- [ ] New feature (non-breaking change adds functionality)
- [ ] Breaking change (fix or feature would cause existing functionality
change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/copilot_auth.py`: split subscription credential resolution
into a non-platform pass and a platform-secret-store fallback pass.
- `tests/test_copilot_auth.py`: cover successful saved-token resolution
without Keychain access, Keychain fallback, and duplicate-token
avoidance.
- `tests/test_copilot_subscription_smoke.py`: preserve the
generic-token-to-Keychain fallback smoke test for the new two-pass
candidate-discovery API.
- Merged current upstream main to pick up the audited cryptography
50.0.0 lock; latest head is `ec306470`.

## Testing

- [x] Unit tests pass for changed behavior
- [x] Linting passes
- [x] Type checking passes
- [x] New tests added
- [x] Manual testing performed

### Test Output

```text
UV_NO_SYNC=1 uv run --no-sync pytest tests/test_copilot_auth.py tests/test_copilot_macos_keychain.py tests/test_copilot_linux_secret.py tests/test_copilot_subscription_smoke.py tests/test_cli/test_wrap_copilot.py -q
158 passed

UV_NO_SYNC=1 uv run --no-sync ruff format --check .
1333 files already formatted

UV_NO_SYNC=1 uv run --no-sync ruff check .
All checks passed

UV_NO_SYNC=1 uv run --no-sync mypy headroom
Success: no issues found in 508 source files
```

## Real Behavior Proof

- Environment: macOS 26.5 (25F71), Python 3.14.5, Headroom topic
worktree `rnoz/fix-copilot-keychain-fallback`, GitHub Copilot CLI
1.0.77, saved Headroom OAuth credential present.
- Exact command / steps: commands below

```text
cd /Users/rnozal/projects/headroom/.worktrees/fix-copilot-keychain-fallback
UV_NO_SYNC=1 uv run --no-sync headroom wrap copilot --subscription --port 8795 -- --model claude-opus-4.8 --prompt 'Reply with exactly: OK' --silent --allow-all-tools --allow-all-paths --allow-all-urls
```

- Observed result: the command printed `OK`, launched against
`http://127.0.0.1:8795/p/fix-copilot-keychain-fallback/v1`, and no macOS
Keychain authorization dialog appeared.
- Not tested: an expired or revoked saved token falling back to a real
macOS Keychain item, Linux Secret Service, Windows Credential Manager,
and GitHub Enterprise credential hosts.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review
- [x] I commented my code, particularly in hard-to-understand areas (not
needed, the code is self-explanatory)
- [x] I made corresponding changes to documentation (not needed, no
user-facing API changes)
- [x] My changes generate no new warnings
- [x] I added tests to prove my fix is effective
- [x] New and existing relevant unit tests pass locally
- [x] I did **not** edit `CHANGELOG.md`

## Screenshots (if applicable)

Not applicable.

## Additional Notes

The full `pytest -q` run was attempted after installing the missing
`litellm` test dependency. It reached execution but failed in five
unrelated existing tests: four Kompress readiness tests depended on an
already-initialized PyTorch backend, and one Windows-only selector-loop
test failed under Python 3.14 on macOS. The changed Copilot suites and
the live wrapped-Copilot proof passed.
2026-10-01 00:09:41 -05:00
66258c40a4 fix: make Headroom work behind Zscaler and other TLS-inspecting networks (#3831)
## Description

Customers behind **Zscaler** (and Netskope, Palo Alto, Cisco Secure
Access, …) report that Headroom "cannot work": every upstream request
fails with `CERTIFICATE_VERIFY_FAILED`, and the agent only sees a bare
502.

These gateways inspect TLS by re-signing every certificate with a
company root. IT installs that root in the **OS certificate store**
(macOS Keychain, Windows cert store), which is why browsers, `curl` and
Claude Code keep working. Headroom never read that store:

- The httpx upstream client trusted certifi only.
- On macOS, the stdlib default reads the static file
`/etc/ssl/cert.pem`, not the Keychain.
- The Rust proxy trusted Mozilla's webpki roots only.

Only 3 of about 40 outbound call sites used the shared CA helper. We had
been patching one path at a time (#741, #998, #1308, #3244, #3344,
#3358, #810, #2029). This PR sets **one trust policy, based on the OS
store, used everywhere.**

Closes #3358

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

**1. Verify upstream TLS through the OS store by default**
(`headroom/proxy/ssl_context.py`)
- New `HEADROOM_CERT_STORE`, default `system,bundled` (same shape as
Claude Code's `CLAUDE_CODE_CERT_STORE`). It uses
[`truststore`](https://truststore.readthedocs.io/), which pip has used
by default since 24.2, plus certifi's public roots.
- The same policy covers the HTTP upstream client, both WebSocket
upstreams, and the `urlopen` paths (Copilot auth).
- **The builders now always return a concrete, verifying `SSLContext`**,
never a `bool`. The legacy default is built explicitly as httpx's
`verify=True` equivalent (certifi).
- OS-store contexts are cached per configuration. They set
`VERIFY_X509_PARTIAL_CHAIN` so an exported intermediate still works as a
trust anchor (matching `create_default_context()` on 3.13). On macOS and
Windows the OS verifies the chain, so the Python 3.13 strict-mode
rejection (#1308) no longer happens.
- `HEADROOM_CA_BUNDLE` and `NODE_EXTRA_CA_CERTS` **add** roots.
`SSL_CERT_FILE` and `REQUESTS_CA_BUNDLE` keep their **replacement**
meaning and turn the OS store off.
- `HEADROOM_CERT_STORE=bundled` restores the old behavior. The aliases
`bundle`, `certifi`, `os` and `native` are also accepted.
- `truststore.inject_into_ssl()` (covers `huggingface_hub`, tiktoken and
OTEL) runs only in processes the Headroom CLI owns. The CLI marks them
with `HEADROOM_PROCESS_TRUST=1`, and proxy workers inherit it.
Applications that embed Headroom keep their own process-wide TLS
behavior.
- `truststore` is added as a core dependency: pure Python, with no
dependencies of its own.

**2. Errors that say what's wrong**
(`headroom/proxy/tls_diagnostics.py`)
- When the certificate check fails, the Anthropic, streaming (SSE) and
OpenAI passthrough errors now name who signed the certificate and give
the fix. For example: *"…The certificate was issued by 'O=Zscaler Inc.,
CN=Zscaler Intermediate Root CA': Zscaler is inspecting this connection.
Fix (pick one): …"*
- The issuer comes from a separate handshake that inspects the
certificate only:
  - It sends no data and requires **TLS ≥ 1.2**.
- It has a 4-second timeout and is cached for 5 minutes per host. A
burst of failures shares a single handshake.
  - It uses the same environment-variable proxies as the real client.
  - It targets the host that actually failed.
- **Security:** on the request path it skips hosts that resolve to
non-public addresses. It also connects to the exact address it checked,
instead of looking the host up again. A per-request upstream therefore
can't be used to read an internal service's certificate or to rebind
DNS.

**3. `headroom doctor`**
- A new **tls trust** row reports the running proxy's policy, read from
`/health`.
- A **proxy env** warning appears when `HTTP(S)_PROXY` is set but
`NO_PROXY` doesn't exempt loopback.
- `--network` (plus a repeatable `--network-url`) probes the provider
and model hosts. For each it shows the certificate issuer, whether
Headroom trusts it, and whether a gateway block page came back. On
failure it prints a paste-ready request for IT.

**4. `headroom wrap`**
- Merges `NO_PROXY` and `no_proxy`, appends `127.0.0.1,localhost,::1`,
and writes the result to both, so a corporate proxy doesn't capture the
agent's traffic to Headroom.
- The notice goes to stderr, because the JSON that `--prepare-only`
prints on stdout is parsed by `install.sh`.

**5. Rust proxy** (`crates/headroom-proxy`)
- reqwest now enables `rustls-tls-native-roots`, and
`HEADROOM_CA_BUNDLE` / `NODE_EXTRA_CA_CERTS` add roots.
- **Fixes a bug already on `main`: `wss://` upstreams panicked.** Both
ring and aws-lc-rs are compiled into this binary, so
`rustls::ClientConfig::builder()`, which
`tokio_tungstenite::connect_async` calls, panics with "no process-level
CryptoProvider available". WebSocket upstreams now use an explicit
config: the ring provider, webpki + OS + additive roots, and ALPN
`http/1.1`.

**6. Docs**
- New page **Corporate Networks**: trust settings, HTTP proxies, a
domain table for IT, and notes on Caution/Isolate, SSE buffering,
process bypass, tenant restrictions and `HF_ENDPOINT` mirrors.
- README section updated, correcting a wrong claim about which trust
store the Rust core uses.

**7. `uv.lock`**
- `main`'s lockfile plus `truststore` only. The diff is 11 lines: the
new package entry and two lines in `headroom-ai`'s dependency lists.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_corporate_tls.py tests/test_ssl_context.py tests/test_copilot_auth.py tests/test_cli_doctor.py -q
============================= 278 passed in 2.88s ==============================

$ pytest <117 related files: doctor, wrap, streaming, codex, copilot, ssl, upstream, websocket, passthrough> -q
====== 4 failed, 1876 passed, 25 skipped, 7 warnings in 132.42s (0:02:12) ======
# the 4 failures: tests/test_codex_live.py ordering issue, identical on main (see Additional Notes)

$ cargo test -p headroom-proxy
passed=478 failed=0 suites=37
$ cargo clippy -p headroom-proxy --all-targets -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s)

$ ruff check .                                            -> All checks passed!
$ ruff format --check .                                   -> 1660 files already formatted
$ mypy headroom --ignore-missing-imports                  -> Success: no issues found in 544 source files
$ uv export --frozen --no-dev --no-hashes | grep truststore -> truststore==0.10.4
```

`tests/test_corporate_tls.py` runs a **fake Zscaler** HTTPS server. Its
root is `O=Zscaler Inc.` with **non-critical basicConstraints**, like
the real 2014 root. The tests check that:

- certifi alone rejects it;
- the default policy plus `HEADROOM_CA_BUNDLE` or `NODE_EXTRA_CA_CERTS`
accepts it;
- the probe names Zscaler, and the request path skips private hosts;
- a streaming request surfaces the actionable message in a 502 SSE
error;
- a burst of failures shares one handshake;
- the notices and JSON output from `wrap` go to the right streams;
- process injection is gated as described.

## Real Behavior Proof

- **Environment:**
  - macOS (Darwin 25.6, Apple Silicon), Python 3.12.13 venv
  - Python 3.13.15 via `uv run --python 3.13`
  - Linux `python:3.13-slim` in Docker
  - Rust proxy built from this branch
- **Steps and results:**
1. **Standalone fake-Zscaler script** (same root as above) on Python
3.13:
- old stdlib path, `ssl.create_default_context(cafile=root)` →
`CERTIFICATE_VERIFY_FAILED: Missing Authority Key Identifier`
     - new `truststore` context + root → `200`
     - identical results on macOS and in `python:3.13-slim`.
2. **`headroom doctor --network`** from this branch on an uninspected
network: every endpoint shows `reachable, certificate trusted`
(api.anthropic.com 401, api.openai.com 401, api.githubcopilot.com 400,
huggingface.co 200, openaipublic 404); the tls trust row shows `OS
certificate store + certifi`.
3. **Rust `wss://` connection** (temporary test) to
`wss://echo.websocket.org`:
- `tokio_tungstenite::connect_async` as on `main` → **panic** (no
process-level CryptoProvider)
     - `connect_async_tls_with_config` with the new config → `Ok(())`
- **Not tested:**
  - A real Zscaler, Netskope or Palo Alto tenant.
- Windows. `truststore` supports the Windows cert store, but no Windows
run was done here.
- A root installed into the OS store. That needs admin rights, so the
additive-bundle path stood in for it; it goes through the same OS
verifier.

## Runtime Rollout Safety

- **Rollout-managed feature(s):** none. This is trust-store selection,
not a rollout-gated feature.
- **Minimum rollout channel:** n/a.
- **Stable/default behavior changed:** yes. Upstream TLS now also trusts
the OS certificate store; certifi is still loaded, so public endpoints
behave the same. The Rust `wss://` upstream no longer panics.
- **Kill switch / disable path:** `HEADROOM_CERT_STORE=bundled` restores
certifi-only trust. An explicit `SSL_CERT_FILE` / `REQUESTS_CA_BUNDLE`
behaves exactly as before.
- **Unsafe override required:** none. Nothing here disables certificate
verification.
- **Qualification impact:** none expected. Default behavior on
uninspected networks is unchanged.
- **Rollback path:** revert the PR, or set
`HEADROOM_CERT_STORE=bundled`.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

n/a (CLI and proxy behavior).

## Additional Notes

**Review response** (@JerrettDavis):
- **CodeQL:** the 5 alerts that were still open on `efe173c9` had two
causes.
- **Certificate-validation alerts:** CodeQL traced a `None` returned by
the optional-context helpers through the `configured or default`
fallback into `verify=`. All contexts are now built by a single
`_build_context()`, and every branch returns a verifying `SSLContext`.
`build_httpx_verify()` is typed `-> ssl.SSLContext`.
- **TLS-version alerts:** every context the module creates now sets an
explicit `minimum_version = TLSv1_2`. That includes the
certificate-inspection probe, whose no-validation scope stays confined
to that handshake behind the public-address / environment-proxy CONNECT
guard. The test server also requires TLS 1.2, and the host assertion in
the test is a regex.
  - **Verified locally with CodeQL 2.27.1** before pushing:
- the same 5 alerts reproduce on `efe173c9`, and 0 remain on the fix
commit;
- `python-security-extended` over every Python file this PR changes
finds the same 14 pre-existing alerts on `main`'s versions and on the
branch's versions, so the PR adds none.
- **`uv.lock`:** rebuilt as `main`'s lockfile plus `truststore` only (11
lines). Re-locking with current uv also rewrites markers across the
file, because `main`'s lockfile predates the current `pyproject.toml`.
That is left for a separate lock refresh.

**Earlier review:** a high-effort code review of this branch produced 10
findings plus 3 minor notes. Commit `fix: address review findings…`
resolves 9 of the 10 and two of the notes; the third note (lockfile
markers) checked out as a false alarm. The remaining finding is the last
follow-up below.

**Follow-ups (not in this PR):**
- SSE keep-alives and HTTP/1.1 fallback when a gateway buffers a stream.
- A `wrap` pre-flight that steps aside when the upstream is unreachable.
- The Rust core's fastembed/hf-hub embedding download, which hard-codes
WebPki roots (needs an hf-hub patch; BM25 is the fallback, and
`HF_ENDPOINT` is documented).
- A CI job with a fake root installed in the OS store on macOS and
Windows.
- Map certificate failures once, in a shared layer, so batch, Gemini and
the other handlers get the TLS hint too.

**Known unrelated failures:**
- 4 `tests/test_codex_live.py` tests fail when run after other files in
this set, both on `main` and on this branch. They pass when run on their
own.
- After merging the latest `main` (thanks for the refresh),
`tests/test_cli/test_wrap_helpers.py::test_start_proxy_strips_ambient_worker_configuration`
and `::test_start_proxy_timeout_kills_failed_new_process` fail locally
with `TypeError: <lambda>() takes 0 positional arguments but 1 was
given`. They fail the same way on a clean checkout of current `main`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-09-28 18:06:09 -07:00
Abhay Singh 9b8cae8448 fix(security): create credential files private instead of narrowing after write (#3215)
## Description

Three code paths persist sensitive local data by writing the file first
and narrowing its permissions afterward:

```python
path.write_text(...)   # created at the umask default — typically 0o644
path.chmod(0o600)      # narrowed only now
```

Between those two calls the file is group/world readable, and if the
process dies in the gap (or the `chmod` raises and is swallowed) it
stays that way permanently. On a shared host a local unprivileged user
can read the contents in that window. Affected files:

- `headroom/copilot_auth.py` — the GitHub Copilot **OAuth refresh
token** (the most sensitive of the three).
- `headroom/providers/codex/recovery.py` — the backup manifest
describing the recovered account's file layout.
- `headroom/cache/backends/sqlite.py` — the CCR originals database,
which holds raw tool output (file contents, command output). Here it is
worse: `sqlite3.connect` creates the db file itself at the umask
default, and the `chmod` runs only after the first `DELETE`/`commit`, so
the window spans real writes.

The repo already knows the correct primitive —
`headroom/telemetry/session.py` writes its install-id with
`os.open(path, O_WRONLY|O_CREAT|O_TRUNC, 0o600)`, which creates the file
private from birth (umask can only clear permission bits, never add
them, and `0o600` has none set for group/other).

This is **low severity**: local-only, POSIX-only (Windows does not use
these bits), and not remotely exploitable. It is a defense-in-depth
hardening of files that are meant to be private and are already
`chmod`ed — this just closes the creation window.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/copilot_auth.py` / `headroom/providers/codex/recovery.py`:
added a module-local `_write_private_text(path, text)` that narrows any
pre-existing file to `0o600` first (so a legacy world-readable file is
not left wide while new secret content is written into it), then writes
through `os.open(..., O_WRONLY|O_CREAT|O_TRUNC, 0o600)` so new files are
private from birth, and `chmod`s once more after. The two token/manifest
writers now call it instead of `write_text` + `chmod`.
- `headroom/cache/backends/sqlite.py`: added `_ensure_private(path)` and
call it in `_open()` **before** `sqlite3.connect`, so the db file is
created (or an existing one narrowed) at `0o600` before sqlite opens it.
The existing post-commit `chmod` loop over `-wal`/`-shm` is kept: those
are created by sqlite during writes, and narrowing them after commit is
unchanged behavior.
- `tests/test_copilot_auth.py`, `tests/test_ccr_sqlite_backend.py`,
`tests/test_cli/test_recover_codex.py`: added **born-private**
regressions for all three writers. Asserting the *final* mode would pass
on the old code too (its trailing `chmod` always ends at `0o600`), so
each test neutralizes every `chmod` and forces `umask(0o022)`, leaving
the mode to reflect only how the file was *created* — `0o600` for
`os.open(..., 0o600)`, `0o644` for the old create-then-narrow. Plus a
content-roundtrip test confirming the token still writes and reads back.
The permission assertions are POSIX-gated (`skipif os.name == 'nt'`).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality

### Test Output

```text
tests/test_copilot_auth.py         ->  128/131 passed (born-private POSIX test skipped on the Windows dev box)
tests/test_ccr_sqlite_backend.py   ->  born-private + existing tests pass (POSIX test skipped on Windows)
tests/test_cli/test_recover_codex.py -> born-private manifest test passes on POSIX (skipped on Windows)
uvx ruff@0.16.2 check <changed files>  ->  All checks passed!
uvx mypy@1.20.2 headroom/copilot_auth.py headroom/providers/codex/recovery.py headroom/cache/backends/sqlite.py  ->  Success: no issues found in 3 source files
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.16.2 and mypy 1.20.2 via uvx.
- Exact command / steps: verified all three writers round-trip content
correctly. The permission tests are genuine red->green: with every
`chmod` neutralized and `umask(0o022)` forced, the new `os.open(...,
0o600)` path yields a `0o600` file while the old create-then-`chmod`
path would leave `0o644` (the forced umask makes this independent of the
runner's ambient umask). These assertions are POSIX-only; on this
Windows dev box they are skipped (Windows does not apply POSIX
permission bits) and run on Linux CI.
- Observed result: content is written and read back identically; no
behavior change beyond the file being private from creation.
- Not tested: the POSIX permission-bit assertions did not execute on
this Windows host (they are `skipif os.name == 'nt'` and run on
Linux/macOS CI); no live GitHub OAuth login or real codex recovery run —
the writers/helpers are exercised directly.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. These are local file writes (auth
token, backup manifest, CCR cache db), not rollout-channel-gated runtime
features.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: no. Files end up at the same `0o600`
they were `chmod`ed to before; only the creation window is closed.
Content and read paths are unchanged.
- Kill switch / disable path: N/A.
- Unsafe override required: no.
- Qualification impact: none. On Windows the `chmod`/`os.open` mode is a
no-op exactly as the prior `chmod` was.
- Rollback path: revert this PR; the writers return to `write_text` +
`chmod`.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A:
internal behavior, unchanged semantics)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

The `-wal`/`-shm` sidecar files remain narrowed after the first commit
rather than pre-created private, deliberately: pre-creating them risks
sqlite WAL-format edge cases, and the durable store (the main db) —
which is what persists between runs and holds the originals — is now
private before sqlite ever opens it.
2026-09-25 10:48:09 -05:00
Rod Boev 871bbde37a fix(proxy): reject Anthropic batch operations on Copilot (#3318)
## Description

Anthropic batch routes followed the resolved Anthropic target to GitHub
Copilot, where the batch surface isn't available, and returned an opaque
upstream failure. This change returns a stable unsupported response
before batch parsing or mutation while preserving normal Copilot-hosted
messages and explicit non-Copilot Anthropic overrides.

Closes #3278

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Added one handler-level capability check for Copilot-resolved
Anthropic batch routes.
- Applied the response consistently to batch create, list, get, cancel,
and results routes.
- Added regressions for Copilot batch routes and explicit Anthropic
overrides. Existing provider-registry coverage continues to verify
normal message target resolution.

## Testing

- [x] Unit tests pass (`uv run pytest
tests/test_proxy_batch_integration.py tests/test_copilot_auth.py
tests/test_provider_registry.py -q`)
- [x] Linting passes (`uv run ruff check
headroom/proxy/handlers/anthropic.py
tests/test_proxy_batch_integration.py tests/test_copilot_auth.py`)
- [ ] Type checking passes (`uv run mypy headroom`)
- [x] New tests added for new functionality when applicable
- [ ] Manual testing performed

### Test Output

```text
uv run python -m pytest tests/test_proxy_batch_integration.py tests/test_copilot_auth.py tests/test_provider_registry.py -q
139 passed, 14 skipped, 1 warning in 7.78s

uv run ruff check headroom/proxy/handlers/anthropic.py tests/test_proxy_batch_integration.py tests/test_copilot_auth.py
All checks passed!

git diff --check
completed without whitespace errors
```

## Real Behavior Proof

- Environment: Windows, focused Headroom test client, no live provider
credential
- Exact command / steps: send Anthropic batch create and the remaining
batch routes with the resolved target set to a Copilot host, then repeat
with an explicit non-Copilot Anthropic target
- Observed result: the five registered Anthropic batch routes return
HTTP 501 with
`{"type":"error","error":{"type":"api_error","message":"Anthropic batch
operations are not supported for Copilot targets."}}` before body
parsing or upstream access; with OpenAI forced to
`https://api.githubcopilot.com`, the explicit
`https://api.anthropic.com` override forwards to
`https://api.anthropic.com/v1/messages/batches`. The focused run
reported `139 passed, 14 skipped, 1 warning` and the route test reported
`6 passed, 17 deselected, 1 warning`.
- Not tested: live Copilot tenant

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Unsafe override required: no
- Minimum rollout channel: standard release after focused CI checks pass
- Stable/default behavior changed: Copilot-resolved Anthropic batch
routes now return a deterministic unsupported response before upstream
access
- Kill switch / disable path: revert the handler guard or disable the
surrounding batch route configuration
- Qualification impact: ordinary Copilot messages and explicit Anthropic
targets keep their existing routing paths
- Rollback path: revert the handler guard and its focused regressions;
no persisted-state migration is required

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

This follows the Anthropic message-target routing introduced in [PR
#3258](https://github.com/headroomlabs-ai/headroom/pull/3258). Headroom
generates changelog entries from conventional commits. The provider's
hosted batch capability remains an external contract; local proof covers
Headroom's deterministic guard and route behavior.
2026-09-24 23:56:55 -05:00
JD Davis 36cc800162 fix(copilot): honor corporate TLS for token refresh (#3246)
## Description

Copilot OAuth/device-auth, user-info, and short-lived token exchange
requests used
`urllib.request.urlopen` directly, bypassing the corporate CA and X.509
strictness
configuration already applied to Headroom's upstream HTTP client. Reuse
that TLS
resolver for every Copilot GitHub request so token refresh works behind
TLS inspection.

Closes #3244

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Added a `urlopen` adapter for Headroom's existing corporate TLS
resolver.
- Routed Copilot device authorization, user-info, and token exchange
through it.
- Added a regression test proving token exchange receives the configured
TLS context.

## Testing

- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
pytest tests/test_copilot_auth.py tests/test_ssl_context.py tests/test_copilot_vscode_completions_routing.py -q
202 passed in 2.45s

ruff check . --exclude .codex-worktrees
All checks passed!

ruff format --check . --exclude .codex-worktrees
1449 files already formatted

mypy headroom/copilot_auth.py headroom/proxy/ssl_context.py
Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12.13, OpenSSL 3.5.0; local HTTPS
server signed by a private test CA; `REQUESTS_CA_BUNDLE` set to that CA.
The exercised request path is the same adapter used by Copilot token
exchange.
- Exact command / steps: generated a one-day localhost certificate,
started an in-process TLS HTTP server, set only `REQUESTS_CA_BUNDLE` to
the private CA, and called
`headroom.copilot_auth._urlopen(Request(local_https_url), timeout=5)`.
- Observed result: `corporate_ca_https_status=200` and
`response_body=ok`.
- Not tested: a real Cisco/Zscaler interception appliance, macOS, or a
live GitHub Copilot Business token (no corporate network/account is
available locally).

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: Only Copilot GitHub requests when a
custom CA or `HEADROOM_TLS_STRICT=0` produces an explicit TLS context;
default `urlopen` behavior remains unchanged otherwise.
- Kill switch / disable path: Unset `SSL_CERT_FILE`,
`REQUESTS_CA_BUNDLE`, or `NODE_EXTRA_CA_CERTS` and leave
`HEADROOM_TLS_STRICT` enabled.
- Unsafe override required: No.
- Qualification impact: Restores existing documented corporate TLS
settings for Copilot authentication traffic.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing relevant unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A.

## Additional Notes

The issue attributes token exchange to the Rust extension, but current
`main` performs it in Python via `urllib`. The direct `urllib` path was
the trust-configuration gap. Full-suite execution was also started
locally; unrelated environment-dependent failures appeared outside the
changed Copilot/TLS scope, while all focused tests pass.
2026-08-25 21:37:12 -05:00
397803a942 fix(copilot): bind the minted token to the integration ID we forward (#3164)
## Description

Reported from a Copilot CLI session:

```
[CopilotCLISession] Failed to fetch models: Error: 401 "unauthorized:
    unable to validate HMAC for the given Copilot-Integration-ID"
[CopilotCLISession] Proxy URL configured (authType=hmac), skipping
    client-side token validation
```

GitHub **binds a Copilot API token to the `Copilot-Integration-Id` it
was minted under** and verifies the pairing with an HMAC. Present a
token minted for integration A alongside a header naming integration B,
and you get exactly this error.

`apply_copilot_api_auth` applied the integration ID with *set-default*
semantics — `_set_header_default` returns early when the header is
already present — **before** deciding whose token to use:

```python
for name, value in _copilot_chat_header_defaults().items():
    _set_header_default(resolved, name, value)   # ← never overwrites
...
if incoming_auth and _is_forwardable_copilot_bearer_token(...):
    return resolved                               # client's token kept
...
token = await get_copilot_token_provider().get_api_token()   # ← REPLACED
```

The client always sends an ID, so when Headroom replaced the token — the
common case, logged as `incoming token not suitable (kind=unknown), will
replace` — the request left carrying **the client's integration ID next
to Headroom's token**, minted under `vscode-chat` via
`_copilot_token_exchange_headers`. A Copilot CLI session does not
identify as `vscode-chat`.

The second log line is why nothing caught it sooner: seeing a proxy URL,
the Copilot client reports `authType=hmac` and **skips its own token
validation**, deferring to the proxy. Nobody validates the pairing until
GitHub rejects it.

**Why this matters beyond one 401:** the failing call is *model
discovery*. When it fails the client falls back to its built-in model
list — which is why a user's selected model never appeared in telemetry
and all traffic surfaced as `gpt-4o-mini`.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

Restores one invariant: **the credential and the integration ID leave
together.**

- **Mint under the client's ID** rather than the proxy's default, so
GitHub's usage attribution keeps pointing at the surface that actually
made the call.
- **Overwrite the forwarded header to match what we minted** — but only
on the replace path. The pass-through branch returns earlier and keeps
the client's own ID beside the client's own token, which is equally a
matched pair.
- **Key the token cache by integration ID.** A single slot would hand a
`vscode-chat` token to a CLI session and reproduce the same 401 straight
from cache.

Two existing contracts deliberately preserved:

- Resolution order is **client header > `GITHUB_COPILOT_INTEGRATION_ID`
> built-in default**. The env var configures the *default* this proxy
sends; it does not override a client that stated its own identity.
Pinned by the existing
`test_apply_copilot_api_auth_preserves_existing_copilot_headers` (whose
fixture literally names the value `should-not-override`).
- The overwrite writes through the client's **existing key**, so a
lowercase `copilot-integration-id` does not gain a second capitalised
variant beside it — pinned by the existing
`..._preserves_existing_headers_case_insensitively`.

Existing test stubs for `get_api_token` gained the new keyword — the
same signature-drift hazard this repo just hit in
`RemoteKompressCompressor` (#3162).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ pytest tests/ -q -k copilot
338 passed, 8 skipped

$ pytest tests/ -q          # this branch
6 failed, 11386 passed, 587 skipped in 425.40s

All 6 also fail on clean origin/main, same machine — pre-existing, not regressions:
  test_graceful_shutdown.py::test_run_server_installs_cancelled_error_filter
  test_learn/test_integration.py::TestCodexIntegration::test_full_pipeline
  test_release_workflows.py::test_no_native_tls_in_wheel_build_tree
  test_providers/test_deepseek.py::...  (3 litellm pricing tests)

$ ruff check headroom/
All checks passed!

$ mypy headroom/copilot_auth.py
0 errors
```

12 new tests: the mint/forward pairing, the pass-through branch keeping
the client's pair untouched, no duplicate case-variant header,
resolution order in both directions, blank/absent client values,
non-Copilot upstreams untouched, and per-integration cache isolation.

## Real Behavior Proof

- **Environment:** macOS, Python 3.12.13, branch on `origin/main` @
`a3821378`.
- **Exact command / steps:** drove `apply_copilot_api_auth` with the
reported shape — an unusable client bearer plus `Copilot-Integration-Id:
copilot-cli-chat` against `api.githubcopilot.com` — and compared the ID
the token would be **minted under** (via
`_copilot_token_exchange_headers`) against the ID actually
**forwarded**. Run against the same script before and after the change,
with `PYTHONPATH` pinned to the worktree.
- **Observed result:**

```
########## PRE-FIX ##########
  token minted under : vscode-chat
  header forwarded   : copilot-cli-chat
  -> GitHub would REJECT (401 HMAC)

########## POST-FIX ##########
  token minted under : copilot-cli-chat
  header forwarded   : copilot-cli-chat
  -> GitHub would ACCEPT
```

- **Not tested:** no live call to GitHub's CAPI — the HMAC is validated
server-side by GitHub and cannot be exercised offline. The claim
verified here is that the two halves now agree; that GitHub accepts a
correctly-paired credential is inferred from its error message, not
observed. **Worth one live Copilot CLI run before shipping to a
reporter.** The `GITHUB_COPILOT_API_TOKEN` path is also unchanged: an
externally-supplied token was minted under an integration this proxy
cannot know, so it is passed through as before.

## Runtime Rollout Safety

- **Rollout-managed feature(s):** none.
- **Minimum rollout channel:** n/a.
- **Stable/default behavior changed:** requests where Headroom replaces
the token now forward the integration ID the replacement was minted
under. For a client sending `vscode-chat` (VS Code, the previous
default) nothing changes at all — the resolved value is identical.
- **Kill switch / disable path:** setting
`GITHUB_COPILOT_INTEGRATION_ID` pins the value used for clients that
send none; clients that send one are unaffected either way.
- **Unsafe override required:** none.
- **Qualification impact:** model discovery should stop 401ing for
non-VS-Code Copilot surfaces, which restores the real model list.
- **Rollback path:** revert the commit; behavior returns to minting
under `vscode-chat` regardless of caller.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines

Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-20 22:11:42 -07:00
JD DavisandJerrettDavis 007446c73a feat(copilot): proxy VS Code models transparently (#2687)
## Description

Make Headroom a transparent proxy for VS Code GitHub Copilot. Users keep
using Copilot's normal model picker—GPT-4.1, Claude Sonnet, Claude Opus,
and other models in their entitlement—while Headroom silently forwards
the selected model instead of registering or requiring a separate
"Headroom" model.

This also fixes GitHub's device OAuth exchange by sending form-encoded
request bodies, matching the endpoint contract.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Add `headroom wrap vscode` to start a Copilot-seeded subscription
proxy and safely configure VS Code's shipped Copilot proxy override.
- Add `headroom unwrap vscode` for reversible cleanup.
- Preserve VS Code's selected model by changing only the proxy URL/auth
override; no custom model is registered and no model preference is
written.
- Support stable VS Code settings locations on macOS, Windows, and
Linux, plus `--settings-file` for Insiders, portable, and other
installations.
- Edit JSONC settings with a marker-owned block while preserving
unrelated bytes, comments, ordering, and trailing commas.
- Refuse malformed markers, invalid JSONC, or unmanaged existing Copilot
overrides instead of overwriting user configuration.
- Fix SIGINT cleanup so the managed settings block is removed and normal
shutdown exits successfully.
- Fix Copilot device OAuth start/poll requests to use
`application/x-www-form-urlencoded`.
- Add a compatibility matrix, setup/removal flow, credential behavior,
remote-development guidance, enterprise notes, troubleshooting, and
verification documentation.

## Testing

- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ .venv/bin/pytest -q tests/test_provider_copilot_vscode_config.py tests/test_cli/test_wrap_vscode.py tests/test_cli/test_wrap_helpers.py tests/test_copilot_auth.py tests/test_cli/test_copilot_auth.py tests/test_provider_copilot_wrap.py tests/test_cli/test_wrap_copilot.py tests/test_copilot_provider_label.py tests/test_copilot_subscription_smoke.py
244 passed in 0.59s

$ .venv/bin/ruff check <changed Python files and tests>
All checks passed!

$ .venv/bin/mypy headroom/providers/copilot/vscode.py
Success: no issues found in 1 source file

$ cd docs && npm run types:check
fumadocs-mdx && next typegen && tsc --noEmit
# exited 0

$ git diff --check
# exited 0
```

The full 10,179-test suite was also sampled through approximately 83%,
but was stopped because of its runtime. It exposed existing failures in
`test_recover_codex.py`, `test_wrap_stale_marker.py`, and
`test_proxy_health.py`; therefore the broad `pytest`, repository-wide
Ruff, and repository-wide mypy boxes are intentionally not checked.

## Real Behavior Proof

- Environment: macOS arm64, VS Code 1.131.0, built-in GitHub Copilot
0.59.0, Headroom 0.33.1-dev.
- Exact command / steps:
  1. Completed `headroom copilot login` with GitHub's device flow.
  2. Ran `.venv/bin/headroom wrap vscode --port 8788`.
3. Confirmed VS Code retained its ordinary Copilot model catalog and
made `GET /models` through Headroom with `GitHubCopilotChat/0.59.0` and
`editor-version: vscode/1.131.0`.
4. Sent native Copilot `/p/headroom/chat/completions` requests through
the same endpoint using `gpt-4.1`, `claude-sonnet-4.6`, and
`claude-opus-4.7`.
- Observed result:
  - All three completion requests returned HTTP 200.
- GPT-4.1 resolved upstream to `gpt-4.1-2025-04-14`; Sonnet and Opus
retained their exact selected IDs.
  - All returned the requested exact marker content.
- VS Code's settings contained only the Headroom proxy URL and token
auth override—no Headroom model or model-selection setting.
- The proxy health endpoint remained ready with `openai_api_url` set to
`https://api.githubcopilot.com`.
- Not tested:
- Physical Windows or Linux hosts (their path/config behavior is covered
by unit tests).
- WSL, dev containers, SSH remotes, VS Code Insiders, or enterprise
Copilot deployments end-to-end.
  - Every model in the live Copilot catalog.
- A fully submitted chat from VS Code's UI automation; the real
extension's catalog request and native completion paths were verified
separately.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing targeted unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable; this integration intentionally has no separate UI or
model entry.

## Additional Notes

The integration uses VS Code Copilot's shipped advanced/debug proxy
endpoint seam. The managed settings block is deliberately narrow and
reversible. Remote extension hosts may need their own reachable
proxy/configuration as documented.

---------

Co-authored-by: JerrettDavis <2610199+JerrettDavis@users.noreply.github.com>
2026-08-03 04:42:48 -07:00
Rod Boev 4a8157fa0a fix(copilot): derive GHE credential host from API URL (#800) (#2511)
## Description

GHE Copilot credential discovery falls back straight to `github.com`
when `GITHUB_COPILOT_HOST` is unset, even if the documented
`GITHUB_COPILOT_API_URL` points at an enterprise host. This change keeps
explicit-host precedence, then reuses the configured enterprise domain
or a normalized custom API URL hostname for credential lookup, so
Windows, macOS, Linux, GH CLI, and credential-file discovery search the
same custom host instead of the public default.

Closes #800.

Attribution:
https://github.com/headroomlabs-ai/headroom/issues/800#issuecomment-5044382263
narrowed the shared credential-host mismatch.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds new functionality)
- [ ] Breaking change
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring

## Changes Made

- Preserve explicit-host precedence, then fall back to the configured
enterprise domain or a normalized custom API URL hostname only when the
configured value is usable.
- Normalize `api.` and `copilot-api.` prefixes before routing credential
lookup, while keeping exact and segmented GitHub-hosted public domains
plus public enterprise or malformed enterprise or API configuration
fallback on `github.com`.
- Add focused coverage for the base/head reproduction, explicit-host
precedence, configured-enterprise precedence, public-enterprise,
malformed-enterprise, and invalid-port fallback, prefixed-host
normalization, adjacent-host exclusion, and GH CLI plus keychain
forwarding.

## Testing

- [x] Unit tests pass (`uv run pytest tests/test_copilot_auth.py -q`)
- [x] Linting passes
- [x] New tests added for new functionality when applicable
- [ ] Manual testing performed

### Test Output

```text
uv run pytest tests/test_copilot_auth.py -q
105 passed in 0.58s

uvx --from ruff==0.15.17 ruff check headroom/copilot_auth.py tests/test_copilot_auth.py
All checks passed!

uvx --from ruff==0.15.17 ruff format --check headroom/copilot_auth.py tests/test_copilot_auth.py
2 files already formatted

git diff --check
clean
```

## Real Behavior Proof

- Environment: Windows, isolated temporary credential file, local
`origin/main` checkout plus this branch
- Exact command / steps: With only
`GITHUB_COPILOT_API_URL=https://api.ghe.example.com:8443/copilot` set
and all other token sources disabled, run the same credential-file
discovery reproduction against `origin/main` and this branch.
- Observed result: `origin/main` selected `github.com` and resolved no
token; the review branch selected `ghe.example.com` and resolved
`gho-ghe`.
- Not tested: live GitHub Enterprise Copilot tenant

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did not edit `CHANGELOG.md`; Headroom generates release notes
from the PR title

## Additional Notes

The change does not alter API routing, token exchange, discovery order,
or credential matching breadth, and it keeps the live tenant claim out
of the PR body until an enterprise user reruns it.
2026-07-23 15:44:33 -07:00
Rod Boev 2eca5ee114 fix(copilot): normalize subscription API routing (#2441) (#2455)
## Description

PR https://github.com/headroomlabs-ai/headroom/pull/2445 added the
missing OpenCode subscription path, but the shared Copilot subscription
resolver still lets Business and Enterprise payload hosts route through
segmented `*.githubcopilot.com` domains and still drops an explicit
`GITHUB_COPILOT_API_URL` pin on two resolution paths. This follow-up
moves the final hosted-route decision back into the shared resolver,
normalizes `api.business.githubcopilot.com` and
`api.enterprise.githubcopilot.com` to the generic host by default, and
makes the explicit pin win on token exchange, explicit API token, and
Copilot-token candidate resolution. Both `headroom wrap copilot
--subscription` and `headroom wrap opencode --copilot-subscription`
inherit the same fix because they already consume the same
`CopilotSubscriptionTokenResolution.api_url`. Refs #2441.

Attribution:
https://github.com/headroomlabs-ai/headroom/pull/2445#issuecomment-5026212395
reported and narrowed the Business or Enterprise regression, and
https://github.com/headroomlabs-ai/headroom/pull/2445#issuecomment-5026863498
scoped the shared-resolver follow-up that this change implements.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Centralize subscription hosted-route selection so explicit
`GITHUB_COPILOT_API_URL` pins win on token exchange, explicit API token,
and Copilot-token candidate resolution.
- Normalize `api.business.githubcopilot.com` and
`api.enterprise.githubcopilot.com` to `https://api.githubcopilot.com` by
default, extending the existing individual-seat normalization.
- Extend focused auth and wrapper tests so both subscription wrappers
prove the corrected shared resolver output and the private-proxy
isolation contract stays intact.

## Testing

- [x] Unit tests pass (`uv run pytest tests/test_copilot_auth.py -q`)
- [x] Unit tests pass (`uv run pytest
tests/test_cli/test_wrap_copilot.py -q`)
- [x] Unit tests pass (`uv run pytest
tests/test_cli/test_wrap_opencode.py -q`)
- [x] Unit tests pass (`uv run pytest
tests/test_cli/test_wrap_persistent.py -q`)
- [x] Linting passes (`uv run ruff check .`)
- [x] Formatting check passes (`uv run ruff format . --check`)
- [x] New tests added for new functionality when applicable
- [ ] Manual testing performed

### Test Output

```text
uv run pytest tests/test_copilot_auth.py -q -> 84 passed
uv run pytest tests/test_cli/test_wrap_copilot.py -q -> 31 passed
uv run pytest tests/test_cli/test_wrap_opencode.py -q -> 44 passed in 143.46s
uv run pytest tests/test_cli/test_wrap_persistent.py -q -> 31 passed
uv run ruff check . -> All checks passed!
uv run ruff format . --check -> 1331 files already formatted
```

## Real Behavior Proof

- Environment: Windows
- Exact command / steps: Run the focused auth, Copilot wrapper, OpenCode
wrapper, and persistent-proxy pytest files after implementing the shared
resolver change, then ask lucasp1337 to rerun the Business or Enterprise
`--copilot-subscription` scenario from PR
https://github.com/headroomlabs-ai/headroom/pull/2445#issuecomment-5026212395
on a real seat.
- Observed result: Focused auth and wrapper pytest runs passed locally,
including the enterprise-host exchange reproduction row, explicit-pin
precedence on all three producer paths, both subscription wrapper
routes, and the private-proxy isolation regression. Live Business or
Enterprise success stays behind reporter retest.
- Not tested: live Business or Enterprise tenant run

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

No `CHANGELOG.md` edit is needed because Headroom generates release
notes from conventional commits.

Risk for maintainers: PR
https://github.com/headroomlabs-ai/headroom/pull/641 manually validated
a Business seat against the GitHub-returned hosted domain in June on
`gpt-5.4`, so generic-by-default could affect tenants that genuinely
require a dedicated host. This follow-up keeps the documented escape
hatch intact by making `GITHUB_COPILOT_API_URL` win on every path.

Live-seat proof boundary: lucasp1337 offered to retest on a Business or
Enterprise seat in PR
https://github.com/headroomlabs-ai/headroom/pull/2445#issuecomment-5026212395.
Keep any live success claim behind that rerun.
2026-07-20 17:15:57 -07:00
Abhay Singh c400f90810 fix(copilot): preserve /v1 for the Anthropic /v1/messages endpoint (#2409) (#2414)
## Description

Fixes #2409.

GitHub Copilot Claude requests routed through Headroom return `404 page
not found`. Copilot serves Claude models at `/v1/messages`, but Headroom
forwards them to `/messages`, so the upstream 404s (observed on
OpenCode's GitHub Copilot provider for `claude-haiku-4.5` /
`claude-sonnet-4.6`, Headroom 0.32.0).

## Root cause

`build_copilot_upstream_url` strips the `/v1` prefix from every Copilot
path:

```python
if normalized_path.startswith("/v1/"):
    normalized_path = normalized_path[3:]
```

That is correct for Copilot's **OpenAI-compatible** surface, which has
no `/v1` (`/chat/completions`, `/responses`, `/embeddings`). But
Copilot's **Anthropic** surface for Claude models is `/v1/messages` —
with the `/v1`. Stripping it produces
`https://api.githubcopilot.com/messages`, which 404s. Confirmed against
the current code:

```text
build_copilot_upstream_url("https://api.githubcopilot.com", "/v1/messages")
  -> "https://api.githubcopilot.com/messages"   # 404
```

## Fix

Keep `/v1` for the messages endpoint; still strip it for the OpenAI
paths:

```python
if normalized_path.startswith("/v1/") and not normalized_path.startswith("/v1/messages"):
    normalized_path = normalized_path[3:]
```

Now `/v1/messages` (and `/v1/messages/batches`) route to
`.../v1/messages`, while `/v1/chat/completions` -> `/chat/completions`
and `/v1/responses` -> `/responses` are unchanged, on both the public
and GHE Copilot hosts. Non-Copilot upstreams are untouched.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/copilot_auth.py`: exclude `/v1/messages` from the
`/v1`-strip in `build_copilot_upstream_url`.
- `tests/test_copilot_auth.py`: assert `/v1/messages` (+ batches, + GHE
host) keep `/v1` while the OpenAI paths still strip it.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ uvx ruff@0.15.17 check headroom/copilot_auth.py tests/test_copilot_auth.py
All checks passed!
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/copilot_auth.py
Success: no issues found in 1 source file
# copilot_auth is import-light, so I ran the real function in the project venv
# (uv sync): /v1/messages -> .../v1/messages, /v1/chat/completions -> /chat/completions.
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, project venv (`uv sync --extra
proxy`), `uvx ruff@0.15.17` / `uvx mypy@1.20.2`.
- Exact command / steps: called the real `build_copilot_upstream_url`
before and after the change for `/v1/messages`, `/v1/messages/batches`,
`/v1/chat/completions`, `/v1/responses`, `/v1/embeddings`, and a
non-Copilot host.
- Observed result: before, `/v1/messages` -> `.../messages` (the 404);
after, `.../v1/messages`. Batches keep `/v1` too; the OpenAI paths still
strip `/v1` (`/chat/completions`, `/responses`, `/embeddings`);
`https://api.anthropic.com/v1/messages` is unchanged. Ran against the
actual module.
- Not tested: a live OpenCode -> Copilot Claude round trip; the added
unit tests assert the URL construction directly.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

`copilot_auth` is a light module, so I verified the fix against the real
function in the venv (output above) in addition to the unit tests. Scope
is limited to the messages endpoint (the reported 404); every other
Copilot path is byte-identical to before.
2026-07-19 22:19:32 -07:00
nangsontayandJerrettDavis e5b3a634df fix(proxy): dedupe Codex WS request logging for accurate mixed-provider dashboards (#2189)
## Description

Running Claude Code (Anthropic) and Codex (OpenAI) against the **same**
Headroom proxy instance on one port produced incorrect, unstable
dashboard data. The proxy core is provider-isolated and
multi-provider-safe by design; the defect was in the observability
layer. The Codex `/v1/responses` **WebSocket** handler was the only path
in the proxy that wrote to the request logger by hand instead of through
the unified `emit_request_outcome` funnel, and it did so twice per
session close: the per-turn funnel record plus an unconditional
cumulative session-summary `RequestLog`. This PR removes the duplicate
summary log so Codex WS emits exactly one request log per turn, matching
the HTTP provider paths.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Dropped the duplicate cumulative session-summary `RequestLog` in the
Codex WS handler while preserving the per-turn `emit_request_outcome`
path.
- Preserved gated `request_messages` and `turn_id` on residual outcomes
so dashboard telemetry keeps the useful attribution without
double-counting tokens.
- Ensured explicit `--anyllm-provider` wins over a leaked
`HEADROOM_ANYLLM_PROVIDER` environment variable.
- Registered retry delay settings that had drifted out of the settings
registry.
- Hardened tests against developer-shell `HEADROOM_*` /
`ANTHROPIC_CUSTOM_HEADERS` leakage and stabilized several focused
proxy/wrap test fixtures.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ .venv/bin/pytest tests/ -q -p no:cacheprovider
8529 passed, 537 skipped, 5831 warnings in 276.25s (0:04:36)

$ .venv/bin/ruff check <touched files>
All checks passed!
```

## Real Behavior Proof

- Environment: macOS (Darwin 25.4.0), Python 3.13.14, pytest 9.0.3, ruff
via project venv, branch `fix/multi-provider-runtime`.
- Exact command / steps: Ran the full test suite without pytest cache
provider and Ruff on all touched files; used `git stash` to confirm the
stale fake-config failures pre-existed this change.
- Observed result: Full suite passed with no failures; Ruff passed;
Codex WS now routes end-of-session logging through
`emit_request_outcome`, emitting one request log per turn with the same
accounting model as Anthropic HTTP turns.
- Not tested: Live simultaneous Claude + Codex dashboard run. `mypy
headroom` was not run to completion; a scoped run reported one
pre-existing `settings_store.py:470` coercion error outside this diff.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A - server-side observability fix; no UI markup changed.

## Additional Notes

- The proxy's multi-provider routing, header/auth isolation, and
per-model cache keying are already correct and unchanged here; only the
WS observability write path was double-counting.
- Architectural assessment:
`plans/reports/research-260714-0004-multi-provider-upstream-compression-report.md`;
root-cause + resolution trail:
`plans/reports/debug-assessment-260714-0011-dashboard-instability-mixed-claude-codex-report.md`.
- No live simultaneous Claude + Codex dashboard run was performed;
validation is from test coverage and code review of the WS logging path.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-07-15 18:18:34 +00:00
5dbe3314a1 fix(auth): support GitHub Enterprise Copilot OAuth domain (#2192)
## Description

Adds GitHub Enterprise OAuth domain support for Copilot auth. When
`GITHUB_COPILOT_ENTERPRISE_URL` is set, the default OAuth domain
resolves to that enterprise host; explicit `--domain` values still take
precedence.

Closes #1152

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/copilot_auth.py`: derive the default OAuth domain from
`GITHUB_COPILOT_ENTERPRISE_URL` when present, falling back to
`github.com` for unset or blank values.
- `headroom/cli/copilot_auth.py`: keep explicit CLI domain overrides
authoritative even when the enterprise env var is set.
- `README.md`: document the enterprise OAuth environment setting and
precedence.
- Added regression tests for enterprise URL handling, blank/unset
fallback, and explicit CLI override precedence.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
uv run pytest tests/test_copilot_auth.py tests/test_cli/test_copilot_auth.py -q
69 passed in 1.05s

uvx ruff@0.15.17 check headroom/cli/copilot_auth.py headroom/copilot_auth.py tests/test_cli/test_copilot_auth.py tests/test_copilot_auth.py
All checks passed!

uvx ruff@0.15.17 format --check headroom/cli/copilot_auth.py headroom/copilot_auth.py tests/test_cli/test_copilot_auth.py tests/test_copilot_auth.py
4 files already formatted
```

## Real Behavior Proof

- Environment: Windows 11 development checkout, Python 3.13.3, with
focused Copilot auth tests using monkeypatched enterprise env vars.
- Exact command / steps: ran the Copilot auth unit/CLI tests plus ruff
check and format-check against the touched auth files and tests.
- Observed result:
`GITHUB_COPILOT_ENTERPRISE_URL=https://ghe.example.com` resolves
`default_oauth_domain()` to `ghe.example.com`; unset/blank enterprise
env vars fall back to `github.com`; and `headroom copilot-auth login
--domain github.com` still honors the explicit override when enterprise
env vars are set.
- Not tested: live OAuth against a real GitHub Enterprise Server
instance.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A - CLI/auth behavior and README update.

## Additional Notes

`GITHUB_COPILOT_ENTERPRISE_URL` takes precedence over
`GITHUB_COPILOT_ENTERPRISE_DOMAIN`; explicit `--domain` remains
authoritative for the login command.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
2026-07-14 16:07:17 -04:00
Rod Boev 4364eb8dc4 fix(copilot): refresh wrapped subscription tokens (#2156) (#2182)
## Description

`headroom wrap copilot --subscription` currently validates a Copilot
subscription credential once at launch, exchanges it once, and then pins
that short-lived API token into the proxy as an explicit override. When
the token expires, long-lived wrapped sessions start returning
`transient_auth_error` and then a final HTTP 401 until the entire
wrapped session is restarted.

This PR keeps the validated launch token for first-request determinism,
carries reusable OAuth refresh material into the proxy, and refreshes
inside `CopilotTokenProvider` when the seeded token is expired. The
explicit `GITHUB_COPILOT_API_TOKEN` override path stays unchanged when
no reusable OAuth token exists. The configured `GITHUB_COPILOT_API_URL`
also stays pinned across refresh, matching the current wrap contract.

Closes #2156.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Extended the Copilot subscription token resolution path to carry
reusable OAuth refresh material and expiry metadata instead of
discarding it after the wrap-time exchange.
- Reworked `CopilotTokenProvider.get_api_token()` so it seeds the
wrapper-validated launch token once for the first request, then
refreshes through the existing exchange path when that token is expired
and reusable OAuth material exists.
- Rejected non-finite seeded expiry values such as `inf`, so malformed
`GITHUB_COPILOT_API_TOKEN_EXPIRES_AT` inputs cannot pin a stale launch
token forever.
- Preserved the explicit `GITHUB_COPILOT_API_TOKEN` override path when
no reusable OAuth token exists, so non-refreshable overrides keep
today's fixed behavior.
- Kept explicitly configured `GITHUB_COPILOT_API_URL` values pinned
across refresh rather than adopting a refreshed payload's host.
- Replaced wrapper-managed seeded `tid_` bearer passthrough with the
refresh-aware provider path, so the wrapped CLI no longer bypasses
expiry refresh just because it keeps sending the launch token back to
the proxy.
- Started a dedicated local proxy instance whenever a
subscription-seeded session targets a shared or persistent proxy port,
so per-session refresh material is not silently dropped on healthy-proxy
reuse or cross-wired between concurrent sessions.
- Scrubbed inherited Copilot refresh-seed environment variables from
both the Copilot child env and the proxy subprocess env before
re-injecting the explicit launch-time values.
- Added focused auth, wrap, proxy-env, and proxy-reuse regression
coverage for expiry refresh, non-finite expiry rejection, session-local
proxy isolation, explicit-override preservation, exchange-flag
independence, configured API URL pinning, and secret handling.

## Testing

- [x] Unit tests pass (`uv run pytest tests/test_copilot_auth.py
tests/test_cli/test_wrap_copilot.py tests/test_cli_proxy_env.py
tests/test_cli/test_wrap_persistent.py -q`)
- [x] Linting passes (`uv run ruff check headroom/copilot_auth.py
headroom/cli/wrap.py tests/test_copilot_auth.py
tests/test_cli/test_wrap_copilot.py tests/test_cli_proxy_env.py
tests/test_cli/test_wrap_persistent.py`)
- [x] Formatting passes (`uv run ruff format --check
headroom/copilot_auth.py headroom/cli/wrap.py tests/test_copilot_auth.py
tests/test_cli/test_wrap_copilot.py tests/test_cli_proxy_env.py
tests/test_cli/test_wrap_persistent.py`)
- [x] Type checking passes (`uv run mypy headroom/copilot_auth.py`)
- [x] New tests added for new functionality when applicable
- [ ] Manual testing performed

### Test Output

```text
195 passed, 1 skipped in 3.14s
```

```text
All checks passed!
```

```text
6 files already formatted
```

```text
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Python 3.12.13, `uv`, no live Copilot credentials.
- Exact command / steps: run the focused auth, wrap, proxy-env, and
proxy-reuse regression suite after seeding an expired launch token plus
reusable OAuth refresh material, then rerun lint and format checks on
the touched files.
- Observed result: base reproduces the bug because the explicit-token
branch never refreshes, accepts non-finite expiry inputs, and
shared-proxy reuse can keep the wrong per-session refresh seed alive;
head refreshes through the reusable OAuth token, rejects non-finite
seeded expiry, replaces the wrapper-managed seeded bearer instead of
blindly passing it through, preserves the valid-seed fast path and the
fixed override path when no refresh material exists, keeps the
configured API URL pinned across refresh, starts a dedicated local proxy
when the requested port already belongs to a shared or persistent proxy,
and keeps the reusable OAuth token confined to explicit proxy launch env
only. The focused suite passed with `195 passed, 1 skipped in 3.14s`.
- Not tested: live business-subscription session past the provider's
real token-expiry window.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

- Scope stays provider-local. The fix remains inside
`headroom/copilot_auth.py` and the Copilot wrap handoff in
`headroom/cli/wrap.py`; it does not add generic 401 retry logic to
provider-neutral proxy layers.
- The line that disables token exchange for the Copilot CLI child env is
unchanged because it never reached the proxy env and was not the root
cause.
- Subscription-seeded sessions now get a dedicated local proxy whenever
the requested port already belongs to a shared or persistent proxy;
existing shared proxies are left alone to avoid disrupting attached
wrappers.
- Live provider confirmation still needs maintainer or reporter
validation because that truth is owned by GitHub's real subscription
APIs, not by local stubs.
- Headroom's release pipeline generates changelog entries from
conventional commits, so `CHANGELOG.md` is intentionally untouched.
2026-07-14 11:52:43 -04:00
yehsufandJerrettDavis f52ca19db1 fix(copilot-auth): stop discarding the caller's valid Copilot auth token (#1879)
## Description

`apply_copilot_api_auth()` treats any incoming
`gho_`/`ghs_`/`ghp_`/`github_pat_`-prefixed GitHub OAuth bearer token as
"not a suitable Copilot API token" (`_is_copilot_api_token()`) and
silently replaces it with a token independently fetched/exchanged by
Headroom itself, even when the caller already sent its own valid,
correctly-entitled Copilot credential.

This closes headroomlabs-ai/headroom#1813, which reports the exact same
defect from OpenCode's native GitHub Copilot integration: Headroom
replaces its native `gho_` bearer token, changing the effective
client/integrator lane Copilot's backend sees and breaking model
discovery/inference parity with native (non-proxied) behavior.

I hit the same root cause independently, through a different trigger
path, while building a new dedicated GitHub Copilot CLI redirect
integration (so Copilot CLI traffic gets the same full Headroom pipeline
treatment Claude Code already gets): a live Copilot CLI session's own
`gho_`-prefixed token worked end-to-end for model `claude-sonnet-5` when
sent directly to Copilot's real API, but the exact same request got `400
model_not_supported` once routed through Headroom, because Headroom
silently swapped in a different, independently-exchanged token with
different (lesser) model entitlements.

Closes #1813

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Added `_is_forwardable_copilot_bearer_token()` in
`headroom/copilot_auth.py`: a new, narrowly-scoped helper used only by
`apply_copilot_api_auth()` (the chat-completion/inference auth path)
that accepts both short-lived Copilot API tokens (`tid_`) AND GitHub
OAuth tokens (`gho_`/`ghs_`/`ghp_`/`github_pat_`) as forwardable, since
live evidence (this PR + #1813) shows both are valid, correctly-entitled
Copilot bearer credentials when supplied directly by a Copilot-aware
client.
- **Deliberately did NOT touch `_is_copilot_api_token()`** — that helper
is used by a separate, unrelated code path
(`resolve_subscription_bearer_token_details()`, for Copilot
subscription/user-info API resolution) with different endpoint
requirements. Broadening it in place would have changed
subscription-resolution behavior as an unintended side effect; the new
helper keeps that concern isolated.
- `apply_copilot_api_auth()` now calls
`_is_forwardable_copilot_bearer_token()` instead of
`_is_copilot_api_token()` when deciding whether to forward the caller's
existing bearer token unchanged. Non-Copilot-shaped or
blank/whitespace-only tokens still fall through to Headroom's own
fetch/exchange, unchanged.
- Fixed a companion `/v1/responses` WebSocket ordering bug in
`headroom/proxy/handlers/openai.py` (filed as
headroomlabs-ai/headroom#1880): the `OPENAI_API_KEY` fallback used a
stale `_lower_headers` snapshot computed before
`apply_copilot_api_auth()` ran, and ran *before* it. Once the upstream
is known, Copilot auth is now resolved first, and the fallback checks
live header state instead — this matters more now that a
correctly-forwarded Copilot bearer token must not be shadowed by an
unrelated OpenAI-key fallback injected earlier in the flow.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_copilot_auth.py tests/test_proxy_copilot_auth_hooks.py tests/test_provider_copilot_wrap.py -k "not falls_back_to_gh_cli" -q
============================= test session starts ==============================
platform darwin -- Python 3.12.12, pytest-9.0.3, pluggy-1.6.0
collected 95 items / 1 deselected / 94 selected

tests/test_copilot_auth.py ............................................. [ 47%]
............                                                             [ 60%]
tests/test_proxy_copilot_auth_hooks.py ...                               [ 63%]
tests/test_provider_copilot_wrap.py ..................................   [100%]

======================= 94 passed, 1 deselected in 0.52s =======================

# The 1 deselected test (test_read_cached_oauth_token_falls_back_to_gh_cli) is a
# pre-existing, environment-dependent failure unrelated to this change: it leaks a
# real gho_ token from this machine's own authenticated `gh` CLI keychain instead of
# using the mocked one. Confirmed identical on main before this change.

$ pytest tests/test_openai_codex_ws_lifecycle.py tests/test_openai_codex_routing.py tests/test_proxy_openai_responses_integration.py tests/test_codex_ws_compression_scheduler.py tests/test_openai_codex_ws_timings.py -q
================== 53 passed, 15 skipped, 1 warning in 16.19s ==================
# (15 skips are pre-existing, environment-gated, unrelated to this change)

$ ruff check headroom/copilot_auth.py headroom/proxy/handlers/openai.py tests/test_copilot_auth.py
All checks passed!
```

## Real Behavior Proof

- Environment: real GitHub Copilot CLI session (`gho_`-prefixed OAuth
token), routed through a dedicated Headroom redirect-proxy integration
(new, currently opt-in/staging-only work, not part of this PR) that
forwards Copilot CLI's `/v1/messages` and `/chat/completions` traffic
through Headroom.
- Exact command / steps: live Copilot CLI request for model
`claude-sonnet-5`, sent once directly to
`api.business.githubcopilot.com` and once routed through Headroom with
this fix applied.
- Observed result: both paths return `200 OK` with identical token/cost
usage. Before this fix, the proxied path returned `400
model_not_supported` for the exact same request, because Headroom
substituted a different, less-entitled token for the client's own valid
`gho_` token.
- Not tested: the companion WS-ordering fix (`/v1/responses` fallback
ordering) was validated via the existing WS lifecycle/routing/timing
unit test suites (see Test Output) rather than a live WebSocket session
— I don't have a live client that exercises that exact
`OPENAI_API_KEY`-fallback-into-Copilot-upstream path end-to-end.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A

## Additional Notes

This PR was rebased onto current `main` after it turned out
`_is_copilot_api_token()` had already been introduced upstream (#557,
merged before this PR was opened) with the opposite assumption — that
`gho_`/`ghs_`/`ghp_`/`github_pat_` tokens always need exchanging and
must not be forwarded. That assumption is exactly what #1813 reports as
broken. Rather than reverting that classification wholesale (my original
approach in this branch), this version keeps it intact for its original
caller and adds a second, narrowly-scoped helper for the
inference/chat-completion auth path specifically — see "Changes Made"
above for the reasoning.

mypy is left unchecked above only because I didn't have it
configured/runnable in my local environment for this pass — happy to run
it if a maintainer flags a concern, or CI will catch it.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-07-09 12:46:35 -05:00
Rod Boev afd9cbdfaf fix(copilot): normalize subscription routing host (#1836)
## Description

`headroom wrap copilot --subscription` can currently trust the
token-exchange host for individual Copilot seats, which routes newer
responses-API models like `gpt-5.4` to
`api.individual.githubcopilot.com` and reproduces the transient `502`
retry loop from issue #1694. This normalizes that public individual-seat
host back to the generic Copilot API host while preserving dedicated
business or explicitly pinned hosts. Closes #1694.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Normalized exchanged Copilot subscription hosts through the existing
public-host classifier instead of trusting the raw token-exchange
payload.
- Added a regression proving `api.individual.githubcopilot.com`
downgrades to `https://api.githubcopilot.com` for subscription routing.
- Added a wrap-level regression proving subscription launches export the
normalized host into the proxy env.
- Preserved business-host and explicit `GITHUB_COPILOT_API_URL` routing
behavior.

## Testing

- [x] Unit tests pass (`uv run pytest tests/test_copilot_auth.py -q`)
- [x] Unit tests pass (`uv run pytest
tests/test_cli/test_wrap_copilot.py -q`)
- [x] Linting passes (`uv run ruff check headroom/copilot_auth.py
tests/test_copilot_auth.py tests/test_cli/test_wrap_copilot.py`)
- [ ] Type checking passes (`uv run mypy headroom`)
- [x] New tests added for new functionality when applicable
- [ ] Manual testing performed

### Test Output

```text
uv run pytest tests/test_copilot_auth.py -q
57 passed, 1 warning in 0.34s

uv run pytest tests/test_cli/test_wrap_copilot.py -q
31 passed, 1 warning in 0.32s

uv run ruff check headroom/copilot_auth.py tests/test_copilot_auth.py tests/test_cli/test_wrap_copilot.py
All checks passed!

uv run ruff format --check headroom/copilot_auth.py tests/test_copilot_auth.py tests/test_cli/test_wrap_copilot.py
3 files already formatted
```

## Real Behavior Proof

- Environment: Windows, Python `uv` environment, mocked Copilot
token-exchange and wrap launch surfaces.
- Exact command / steps: run the focused Copilot auth and wrap
regression tests after teaching subscription token-exchange routing to
normalize the public individual-seat host.
- Observed result: exchanged subscription tokens that advertise
`https://api.individual.githubcopilot.com` now route through
`https://api.githubcopilot.com`, while business-host and explicit-host
pin cases stay unchanged.
- Not tested: a live GitHub Copilot subscription request against the
upstream service.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

This is intentionally scoped to host selection for exchanged Copilot
subscription tokens. It does not change token discovery, token pinning,
or non-subscription OAuth routing.
2026-07-06 06:23:48 -07:00
Ralf Escher 16c638bc21 fix: recover persistent proxy feature checks and reject non-Copilot exchange URL (#1465)
## Description

This PR fixes two related reliability issues in Copilot
wrap/subscription flows:

1. Recovered persistent proxy instances could be reused too early,
before validating requested feature-sensitive config (especially
`openai_api_url`), which could lead to wrong upstream routing.
2. Subscription token-exchange payloads could provide a non-Copilot API
URL; this is now rejected and we safely fall back to user-info/default
Copilot endpoint resolution.

Related: #488

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Updated persistent proxy recover path to:
- continue into feature checks when feature-sensitive options are
requested
- restart persistent deployment when config is missing/mismatched after
recovery
  - keep historical fast return for plain recover-only calls
- Hardened subscription exchange URL resolution:
  - accept exchange `api_url` only when it is a Copilot host
  - log warning and fall back when non-Copilot host is provided
- Added regression tests for:
- recovered persistent proxy feature mismatch and config-unavailable
restart behavior
  - non-Copilot exchange host rejection with/without user-info fallback

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ python -m pytest -q tests/test_copilot_auth.py tests/test_cli/test_wrap_persistent.py
============================= test session starts =============================
platform win32 -- Python 3.12.8, pytest-9.1.1, pluggy-1.6.0
rootdir: C:\Users\ralf.escher\Documents\headroom
collected 82 items

tests\test_copilot_auth.py ............................................. [ 54%]
...........                                                              [ 68%]
tests\test_cli\test_wrap_persistent.py ..........................        [100%]

============================= 82 passed in 1.60s ==============================
```

## Real Behavior Proof

- Environment:
  - Windows
  - Python 3.12.8
  - Local Headroom branch with this patch
  - Copilot subscription route through local proxy

- Exact command / steps:
  1. Start local proxy and run Copilot wrap in subscription mode.
  2. Execute chat-completions requests through proxy.
  3. Inspect runtime proxy logs for outbound target and inbound status.
  4. Run focused regression tests:
- `python -m pytest -q tests/test_copilot_auth.py
tests/test_cli/test_wrap_persistent.py`

- Observed result:
  - Outbound requests routed to Copilot business host:
    - `path=https://api.business.githubcopilot.com/chat/completions`
  - Successful proxy responses observed:
    - `path=/v1/chat/completions status=200`
  - Model activity logged during successful requests:
    - `PERF model=gpt-4.1 ...`
  - Regression tests pass (`82 passed`), covering both fixes.

- Not tested:
  - Full repository test suite
  - Full lint/typecheck across entire project
  - Non-Windows runtime verification in this run

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

- This PR intentionally excludes incidental local edits to
`.github/copilot-instructions.md`.
- Scope is limited to this bug fix and regression coverage; linked as
related work to #488.
2026-06-28 15:26:38 -07:00
ParideboyandClaude Opus 4.8 d633e8172c fix(windows): pin UTF-8 encoding on text-mode subprocess calls (#1311)
Fixes #1310.

## Description

On Windows, `headroom` startup crashes a subprocess reader thread:

```
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 7894: character maps to <undefined>
  ... subprocess.py _readerthread -> buffer.append(fh.read())
  ... encodings/cp1252.py
```

Text-mode `subprocess` calls omit `encoding=`, so Python decodes child
output with the locale codec (**cp1252** on Windows). Children that emit
UTF-8 ??? `cbm index_repository` (indexing sources with chars like
`???`/`???`), `claude mcp get/add`, the memory-sync process ??? produce
bytes invalid in cp1252 and kill the reader thread. Linux/macOS default
to UTF-8, so it's invisible there.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Add `encoding="utf-8", errors="replace"` to every text-mode
(`text=True` / `universal_newlines=True`) subprocess call in the
`headroom/` package (~50 call sites; several already had it).
- `errors="replace"` (not `ignore`) so corrupt bytes surface as `???`
rather than vanishing from parsed output.
- Add `tests/test_cli/test_subprocess_utf8_encoding.py`: an AST guard
asserting every text-mode subprocess call pins `encoding=`. The runtime
crash can't reproduce on UTF-8 CI, so the invariant is enforced at the
source level instead.

## Testing

- [x] Unit tests pass (`pytest`)
- New guard test passes (validates 51 call sites).
- `tests/test_install`, `tests/test_cli/test_mcp.py`,
`tests/test_mcp_registry` pass.
(`test_runtime_start_lock_blocks_another_process` fails on this Windows
box, but it fails identically on unmodified `main` ??? a pre-existing
`msvcrt` lock flake, unrelated.)

### Test Output

```text
> python -m pytest tests/test_cli/test_subprocess_utf8_encoding.py -q
1 passed in 0.12s

> python -m pytest tests/test_install/ tests/test_cli/test_mcp.py tests/test_mcp_registry/ -q
133 passed, 2 skipped in 15.34s
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.13.13.
- Exact command / steps: Started `headroom` without `PYTHONUTF8=1` on a
repo with UTF-8 chars in indexable files. Observed the
`UnicodeDecodeError` crash. Applied the fix (pinning `encoding="utf-8"`
on all text-mode subprocess calls). Re-ran. No crash. The AST guard
enforces the invariant on CI (which runs UTF-8 locales and cannot
reproduce the cp1252 crash natively).
- Observed result: Subprocess reader threads no longer crash on UTF-8
output under cp1252 locale.
- Not tested: All third-party tools that `headroom` shells out to; each
was given `errors="replace"` as a safety net.

## Workaround for affected users (before fix is deployed)

`PYTHONUTF8=1` (PowerShell: `$env:PYTHONUTF8=1; headroom ...`).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-23 12:52:49 -05:00
Joel Belangerandjbelanger 0b4a4bd483 fix: support Copilot Business subscription auth (#641)
## Description

Adds a first-party `headroom copilot-auth login` flow for Copilot
subscription
mode and uses the resulting Copilot OAuth token to perform GitHub's
Copilot
token exchange before launching the wrapped Copilot CLI.

This fixes Business/Enterprise Cloud accounts where a generic
GitHub/Copilot
token can read Copilot account metadata but is rejected by the Copilot
token
exchange endpoint. It also avoids treating GitHub.com Enterprise Cloud
account
URLs such as `github.com/enterprises/acme` as API hostnames.

Fixes #635
Related: #488, #610
Builds on #576

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [x] Documentation update
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Adds `headroom copilot-auth login` and `headroom copilot-auth status`.
- Stores a Headroom-specific Copilot OAuth token under Headroom's state
dir.
- Exchanges reusable Copilot OAuth tokens with Copilot Chat-compatible
headers before subscription-mode launch.
- Carries the resolved Copilot API endpoint into `headroom wrap copilot
--subscription`.
- Handles GitHub.com Enterprise Cloud URLs without synthesizing invalid
`api.github.com/enterprises/...` hosts.
- Adds focused unit tests and README guidance for subscription login.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```console
ruff check headroom/copilot_auth.py headroom/cli/copilot_auth.py headroom/cli/main.py headroom/cli/__init__.py headroom/cli/wrap.py tests/test_copilot_auth.py tests/test_cli/test_copilot_auth.py tests/test_cli/test_wrap_copilot.py tests/test_copilot_subscription_smoke.py
# All checks passed!

ruff format --check headroom/copilot_auth.py headroom/cli/copilot_auth.py headroom/cli/main.py headroom/cli/__init__.py headroom/cli/wrap.py tests/test_copilot_auth.py tests/test_cli/test_copilot_auth.py tests/test_cli/test_wrap_copilot.py tests/test_copilot_subscription_smoke.py
# 9 files already formatted

python -m py_compile headroom/copilot_auth.py headroom/cli/copilot_auth.py headroom/cli/main.py headroom/cli/__init__.py headroom/cli/wrap.py tests/test_copilot_auth.py tests/test_cli/test_copilot_auth.py tests/test_cli/test_wrap_copilot.py tests/test_copilot_subscription_smoke.py

uv run --no-project --with pytest --with pytest-asyncio --with click --with rich --with opentelemetry-api --with pydantic --with tiktoken --with 'litellm==1.82.3' --with fastapi --with uvicorn --with 'httpx[http2]' --with openai --with mcp --with magika --with zstandard --with websockets --with onnxruntime --with transformers --with watchdog --with sqlite-vec pytest tests/test_copilot_auth.py tests/test_cli/test_copilot_auth.py tests/test_cli/test_wrap_copilot.py tests/test_cli_proxy_env.py tests/test_copilot_subscription_smoke.py
# 127 passed
```

Local note: `uv run pytest ...` against the project currently fails
before
running tests because `uv.lock` has an unrelated `gitpython`
wheel/version
mismatch.

## Manual Validation

I tested this with an existing GitHub Copilot Business subscription
associated with a GitHub.com Enterprise Cloud account.

The Enterprise Cloud value I tested was in the form:

```text
github.com/enterprises/<enterprise>
```

The tested flow was:

```text
headroom copilot-auth login
headroom wrap copilot --subscription -- --model gpt-5.4
```

This validated that Headroom does not treat
github.com/enterprises/<enterprise> as a Copilot API hostname. Instead,
token exchange uses GitHub.com and Headroom routes subscription-mode
traffic to the Copilot API endpoint returned by GitHub for the signed-in
account.

I did not test this with GitHub Enterprise Server or a custom enterprise
domain such as ghe.example.com.

No tokens, request IDs, or organization-specific identifiers are
included in this PR.

## Real Behavior Proof

- Environment: macOS Darwin, Python 3.12.7, local checkout on
`codex/copilot-business-auth`.
- Exact command / steps: Ran `headroom copilot-auth login`, then
launched `headroom wrap copilot --subscription -- --model gpt-5.4` with
a GitHub Copilot Business subscription tied to a GitHub.com Enterprise
Cloud account.
- Observed result: Headroom did not treat
`github.com/enterprises/<enterprise>` as a Copilot API hostname; token
exchange used GitHub.com and subscription traffic was routed to the
Copilot API endpoint returned for the signed-in account. The latest
focused Copilot auth/proxy tests pass locally (`127 passed`).
- Not tested: GitHub Enterprise Server or custom enterprise domains such
as `ghe.example.com`; Windows Credential Manager integration still needs
confirmation from someone on Windows.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing targeted unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A

## Additional Notes

Acknowledgement: the OAuth/token-exchange behavior was informed by
`anomalyco/opencode-copilot-auth` by Aiden Cline.

No tokens are printed by the new login/status commands; only a short
SHA-256
fingerprint is displayed for troubleshooting.

The interactive login is included because the missing piece is not just
an
Enterprise URL or routing hint. For GitHub.com Enterprise Cloud
accounts,
URLs like `github.com/enterprises/acme` identify the enterprise account
but
are not Copilot API hostnames; token exchange still happens through
GitHub.com
and then returns the account-specific Copilot API endpoint. A
command-line
enterprise argument can help for true GitHub Enterprise
Server/custom-domain
deployments, but it cannot produce the Copilot OAuth token class that
the
token-exchange endpoint accepts.

Ideally, Headroom would avoid an extra interactive login and reuse an
existing
GitHub/Copilot CLI session everywhere. In practice, some
reusable-looking
tokens can read Copilot account metadata but are rejected by Copilot
token
exchange, which leaves Business/Enterprise Cloud users with missing
model
catalogs. The explicit login command is the smallest independent way to
obtain
and persist the token needed for that exchange without asking users to
pass a
secret on the command line.

---------

Co-authored-by: jbelanger <your-username@users.noreply.github.com>
2026-06-12 20:46:38 -05:00
Abhinav KauravandAbhinav Kaurav 6b0c09ffd5 fix: harden Copilot API auth token handling (#557)
## Description
Improve Copilot API authentication behavior by correctly handling
incoming bearer tokens and ensuring required Copilot headers are
present.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Added token classification logic to distinguish short-lived Copilot
API tokens (`tid_`) from GitHub OAuth tokens.
- Updated auth flow to pass through valid existing Copilot API bearer
tokens and replace unsuitable bearer tokens.
- Added default `Copilot-Integration-Id` and `editor-version` headers
when missing.
- Improved Windows credential lookup to consider both GitHub CLI (`gh:`)
and Copilot CLI credential target prefixes.
- Added regression tests for pass-through, replacement, header
injection, and token prefix classification.

## Testing

Describe the tests you ran to verify your changes:

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

## Test Output

```text
pytest -q tests/test_copilot_auth.py
29 passed in 0.40s
```

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A

## Additional Notes

This change is intentionally scoped to auth behavior and tests in:
- `headroom/copilot_auth.py`
- `tests/test_copilot_auth.py`

---------

Co-authored-by: Abhinav Kaurav <abhinav.kaurav@e2open.com>
2026-06-11 12:57:48 -05:00
Tejas Chopra 6ed43027b7 style(copilot): ruff-format test_copilot_auth.py
Parenthesize a multi-line conditional lambda so 'ruff format --check .'
passes (the original commit added it unformatted).
2026-06-04 00:22:04 -07:00
Tejas Chopra ff4a0c6bc6 fix(copilot): support subscription auth through Headroom
Route GitHub Copilot CLI subscription traffic through the Headroom
OpenAI-compatible proxy path and resolve the account-specific Copilot API
endpoint before launch.

Add source-aware Copilot token discovery for explicit Copilot env vars,
macOS Keychain, Windows Credential Manager, Linux Secret Service, credential
files, and generic GitHub fallbacks. Validate subscription candidates against
GitHub Copilot user metadata so generic GH_TOKEN/GITHUB_TOKEN values do not
shadow Copilot CLI auth.

Document the subscription command and platform status in README: macOS
Keychain auth reuse has been smoke-tested, while Windows, Linux, Docker, and
CI auth-discovery paths still need real OS validation.

Tests: .venv/bin/python -m pytest tests/test_copilot_auth.py
tests/test_copilot_macos_keychain.py tests/test_copilot_linux_secret.py
tests/test_cli/test_wrap_copilot.py tests/test_cli/test_wrap_persistent.py
tests/test_proxy_copilot_auth_hooks.py
2026-06-02 21:24:47 -07:00
JerrettDavisandCopilot dcb3e8bdcb test: cover copilot auth branches
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-22 00:23:28 -05:00
JerrettDavisandCopilot d60cf7914c fix: support live copilot oauth runtime
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-21 22:48:57 -05:00
JerrettDavisandCopilot 7989581350 fix: support copilot oauth sessions
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-21 22:11:30 -05:00