Files
headroom/tests/test_client_store_url.py
Satya BharadwajandJD Davis 47fd8cb3ba fix(client): store metrics at HeadroomConfig.store_url (#4270)
## Description

`HeadroomClient` ignored `HeadroomConfig.store_url`. The constructor
copied `config.store_url` into `self._store_url`, but then opened
storage with its own `store_url` argument:

```python
if config.store_url:
    self._store_url = config.store_url   # saved here
...
self._storage = create_storage(store_url)  # but storage uses the argument
```

So `HeadroomClient(...,
config=HeadroomConfig(store_url="sqlite:///mine.db"))` wrote metrics to
`<system temp>/headroom.db`, and `mine.db` was never created.
`validate_setup()` reads `self._store_url`, so it reported `mine.db` as
the storage URL. The client said it stored metrics in one place and
actually stored them in another. `get_metrics()` / `get_summary()` also
read from the temp database, so anything reading `mine.db` directly (for
example `generate_report(store_url="sqlite:///mine.db")`) found nothing.

The constructor docstring says the config "takes precedence over
individual settings like store_url".

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/client.py`: storage is opened with `self._store_url`, the
URL the constructor already resolves and reports.
- `headroom/client.py`: the client no longer writes its resolved URL
back into a caller-supplied config (the old `else:
self._config.store_url = store_url`).
- **Why:** a caller can reuse one `HeadroomConfig` object for several
clients. Client A would fill `shared.store_url` with its own URL, and
client B would then read that as an explicit config value and store into
A's database despite passing its own `store_url=`. (Reported by Devin
review; the write-back was harmless on `main` only because the old
truthy default meant it never ran and storage ignored the config
anyway.)
- The client keeps its resolved URL in `self._store_url`, which is all
it uses. A client built without a config still records the URL on the
`HeadroomConfig` it creates for itself.
- `headroom/config.py`: `HeadroomConfig.store_url` now defaults to
`None` instead of `"sqlite:///headroom.db"`. `None` means "let the
client decide": its `store_url` argument, else `headroom.db` in the
system temp directory.
- **Why the default had to change:** the old default was always truthy,
so with only the one-line fix every
`HeadroomClient(config=HeadroomConfig())` would have started writing
`headroom.db` into the current working directory. v0.2.0 (90d3aea4,
"Fixed SQLite default path (now uses temp directory)") deliberately
moved the client default from `./headroom.db` to the temp directory, but
didn't update this field. It was never read for storage, so it never
mattered until now.
- Nothing else reads `HeadroomConfig.store_url` (`grep -rn "\.store_url"
headroom`: only `client.py` and `testing/harness.py`, which sets it
explicitly).
- `tests/test_config.py`: the default-values test expects `store_url is
None`.
- `tests/test_client_store_url.py`, 5 tests (each one isolates the
system temp directory):
- `config.store_url` is where storage opens and the file is created, and
`validate_setup()` reports the same URL.
- `config.store_url` wins over the `store_url` argument (per the
docstring).
- A config with no `store_url` uses the `store_url` argument, and the
config is left unchanged.
- One default config shared by two clients with different `store_url=`
arguments: each client stores in its own database, and the shared config
is left unchanged.
- A default `HeadroomConfig()` with no argument stores in the temp
directory and creates nothing in the working directory.

**Resulting precedence:** `config.store_url` > `store_url=` argument >
`<system temp>/headroom.db`. Callers who pass only `store_url=` (no
config), or no storage settings at all, see no change.

## Testing

- [x] Unit tests pass (`pytest`): the new file plus every test file that
references `HeadroomClient`, `store_url` or `HeadroomConfig(` (17 files,
run one by one), not the full suite. The Strands and Agno test files
skip entirely in my env because those extras aren't installed
- [x] Linting passes (`ruff check .`): on the changed files
- [ ] Type checking passes (`mypy headroom`): ran `mypy
headroom/client.py headroom/config.py` only
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

New tests without the fix (`client.py` and `config.py` reverted to
`main`):

```text
$ uv run pytest tests/test_client_store_url.py -q
FAILED tests/test_client_store_url.py::test_config_store_url_is_where_metrics_are_stored
FAILED tests/test_client_store_url.py::test_config_store_url_takes_precedence_over_the_argument
FAILED tests/test_client_store_url.py::test_argument_is_used_when_config_has_no_store_url
FAILED tests/test_client_store_url.py::test_shared_config_does_not_leak_one_clients_url_into_another
========================= 4 failed, 1 passed in 1.72s ==========================
```

The first two fail on the storage path (`.../system-temp/headroom.db`
and `.../from-argument.db` instead of the configured database). The last
two store in the right place on `main` but fail on the config-default
assertion, since `main`'s default is `"sqlite:///headroom.db"`. The one
pass is `test_default_config_stores_in_temp_dir_not_cwd`, which is
expected: it guards that default callers keep the temp-directory
location, so it holds before and after.

The shared-config test against this PR's first commit (f96bc8e2, before
the write-back was removed) fails on the leak itself:

```text
E   AssertionError: assert PosixPath('/private/var/.../a.db') == PosixPath('/private/var/.../b.db')
FAILED tests/test_client_store_url.py::test_shared_config_does_not_leak_one_clients_url_into_another
```

With the fix:

```text
$ uv run pytest tests/test_client_store_url.py tests/test_config.py -q
============================== 41 passed in 2.20s ==============================

$ for f in <17 test files referencing HeadroomClient / store_url / HeadroomConfig(>; do uv run pytest "$f" -q; done
220 passed, 0 failed (summed across files; 9 skipped in test_chat_model.py)

$ uv run ruff check headroom/client.py headroom/config.py tests/test_client_store_url.py tests/test_config.py
All checks passed!
$ uv run ruff format --check (same files)
4 files already formatted
$ uv run mypy headroom/client.py headroom/config.py
Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: macOS 27.0.1 (arm64), Python 3.13.15, `uv sync --extra
dev --extra proxy`, openai 2.41.0, `HeadroomClient` against a local stub
OpenAI upstream.
- Exact command / steps: wrapped the real OpenAI SDK client in
`HeadroomClient(original_client=OpenAI(base_url=<stub>),
provider=OpenAIProvider(),
config=HeadroomConfig(store_url="sqlite:///<work>/mine.db"))`, sent one
`chat.completions.create` request, then counted rows in the `requests`
table of each candidate database, on `main` and on this branch.
- Stub upstream: a small `ThreadingHTTPServer` returning a canned chat
completion.
- The script pointed `tempfile.tempdir` at `<work>/system-temp` and
`chdir`-ed into `<work>/cwd`, so all three possible locations are
isolated and checkable.
- Observed result: on `main` the request's metrics row landed in the
temp-directory database while `validate_setup()` reported `mine.db`; on
this branch it lands in `mine.db`. Nothing is written to the working
directory in either case. Separately, two clients built from one shared
default config with `store_url=` `a.db` and `b.db` store in `a.db` and
`b.db` respectively, and the shared config's `store_url` stays `None`.

  | | `main` | this branch |
  |---|---|---|
| `validate_setup()["storage"]["url"]` | `sqlite:///<work>/mine.db` |
`sqlite:///<work>/mine.db` |
  | rows in `mine.db` | file never created | 1 |
  | rows in `<temp>/headroom.db` | 1 | file never created |
  | `headroom.db` in working directory | no | no |

- Not tested: a live upstream provider; `jsonl://` and plugin storage
URLs (they go through the same `create_storage` call, only the URL
passed in changed).

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: N/A
- Stable/default behavior changed: no for callers who don't set
`HeadroomConfig.store_url`: the default stays `<system
temp>/headroom.db`, and a `store_url=` argument is honored as before.
Callers who set `config.store_url` now get metrics stored there, which
is what they configured.
- Kill switch / disable path: N/A
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A: the
docs already describe the temp-directory default and `store_url` as the
way to change it)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

`HeadroomConfig().store_url` now reads `None` instead of
`"sqlite:///headroom.db"`, and a client no longer fills it in on a
caller-supplied config. That old value never controlled where anything
was stored, so code that read it was already getting a path nothing
wrote to.

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-28 17:57:20 -05:00

105 lines
3.3 KiB
Python

"""``HeadroomConfig.store_url`` must decide where ``HeadroomClient`` stores metrics.
The client copied ``config.store_url`` into ``self._store_url`` (which
``validate_setup()`` reports) but opened storage with its own ``store_url``
argument, so metrics went to the temp-directory default while the client
reported the configured path.
"""
from __future__ import annotations
import tempfile
from pathlib import Path
from unittest.mock import MagicMock
import pytest
from headroom.client import HeadroomClient
from headroom.config import HeadroomConfig
from headroom.providers.openai import OpenAIProvider
@pytest.fixture
def temp_dir(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
"""Point the client's temp-directory default at an isolated folder."""
temp = tmp_path / "system-temp"
temp.mkdir()
monkeypatch.setattr(tempfile, "tempdir", str(temp))
return temp
def _client(**kwargs: object) -> HeadroomClient:
return HeadroomClient(
original_client=MagicMock(),
provider=OpenAIProvider(),
enable_cache_optimizer=False,
**kwargs,
)
def _storage_path(client: HeadroomClient) -> Path:
# "sqlite:////abs/path" parses to "//abs/path"; resolve() normalizes it.
return Path(client._storage.db_path).resolve()
def test_config_store_url_is_where_metrics_are_stored(tmp_path: Path, temp_dir: Path) -> None:
db = tmp_path / "mine.db"
client = _client(config=HeadroomConfig(store_url=f"sqlite:///{db}"))
assert _storage_path(client) == db.resolve()
assert db.exists()
assert not (temp_dir / "headroom.db").exists()
assert client.validate_setup()["storage"]["url"] == f"sqlite:///{db}"
def test_config_store_url_takes_precedence_over_the_argument(
tmp_path: Path, temp_dir: Path
) -> None:
from_config = tmp_path / "from-config.db"
from_argument = tmp_path / "from-argument.db"
client = _client(
store_url=f"sqlite:///{from_argument}",
config=HeadroomConfig(store_url=f"sqlite:///{from_config}"),
)
assert _storage_path(client) == from_config.resolve()
assert not from_argument.exists()
def test_argument_is_used_when_config_has_no_store_url(tmp_path: Path, temp_dir: Path) -> None:
from_argument = tmp_path / "from-argument.db"
config = HeadroomConfig()
client = _client(store_url=f"sqlite:///{from_argument}", config=config)
assert _storage_path(client) == from_argument.resolve()
assert config.store_url is None
def test_shared_config_does_not_leak_one_clients_url_into_another(
tmp_path: Path, temp_dir: Path
) -> None:
"""One config object reused for two clients, each with its own database."""
shared = HeadroomConfig()
a_db = tmp_path / "a.db"
b_db = tmp_path / "b.db"
client_a = _client(store_url=f"sqlite:///{a_db}", config=shared)
client_b = _client(store_url=f"sqlite:///{b_db}", config=shared)
assert _storage_path(client_a) == a_db.resolve()
assert _storage_path(client_b) == b_db.resolve()
assert shared.store_url is None
def test_default_config_stores_in_temp_dir_not_cwd(
tmp_path: Path, temp_dir: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
cwd = tmp_path / "cwd"
cwd.mkdir()
monkeypatch.chdir(cwd)
client = _client(config=HeadroomConfig())
assert _storage_path(client) == (temp_dir / "headroom.db").resolve()
assert not (cwd / "headroom.db").exists()