mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:06:14 +08:00
## Description
`HeadroomClient` ignored `HeadroomConfig.store_url`. The constructor
copied `config.store_url` into `self._store_url`, but then opened
storage with its own `store_url` argument:
```python
if config.store_url:
self._store_url = config.store_url # saved here
...
self._storage = create_storage(store_url) # but storage uses the argument
```
So `HeadroomClient(...,
config=HeadroomConfig(store_url="sqlite:///mine.db"))` wrote metrics to
`<system temp>/headroom.db`, and `mine.db` was never created.
`validate_setup()` reads `self._store_url`, so it reported `mine.db` as
the storage URL. The client said it stored metrics in one place and
actually stored them in another. `get_metrics()` / `get_summary()` also
read from the temp database, so anything reading `mine.db` directly (for
example `generate_report(store_url="sqlite:///mine.db")`) found nothing.
The constructor docstring says the config "takes precedence over
individual settings like store_url".
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/client.py`: storage is opened with `self._store_url`, the
URL the constructor already resolves and reports.
- `headroom/client.py`: the client no longer writes its resolved URL
back into a caller-supplied config (the old `else:
self._config.store_url = store_url`).
- **Why:** a caller can reuse one `HeadroomConfig` object for several
clients. Client A would fill `shared.store_url` with its own URL, and
client B would then read that as an explicit config value and store into
A's database despite passing its own `store_url=`. (Reported by Devin
review; the write-back was harmless on `main` only because the old
truthy default meant it never ran and storage ignored the config
anyway.)
- The client keeps its resolved URL in `self._store_url`, which is all
it uses. A client built without a config still records the URL on the
`HeadroomConfig` it creates for itself.
- `headroom/config.py`: `HeadroomConfig.store_url` now defaults to
`None` instead of `"sqlite:///headroom.db"`. `None` means "let the
client decide": its `store_url` argument, else `headroom.db` in the
system temp directory.
- **Why the default had to change:** the old default was always truthy,
so with only the one-line fix every
`HeadroomClient(config=HeadroomConfig())` would have started writing
`headroom.db` into the current working directory. v0.2.0 (90d3aea4,
"Fixed SQLite default path (now uses temp directory)") deliberately
moved the client default from `./headroom.db` to the temp directory, but
didn't update this field. It was never read for storage, so it never
mattered until now.
- Nothing else reads `HeadroomConfig.store_url` (`grep -rn "\.store_url"
headroom`: only `client.py` and `testing/harness.py`, which sets it
explicitly).
- `tests/test_config.py`: the default-values test expects `store_url is
None`.
- `tests/test_client_store_url.py`, 5 tests (each one isolates the
system temp directory):
- `config.store_url` is where storage opens and the file is created, and
`validate_setup()` reports the same URL.
- `config.store_url` wins over the `store_url` argument (per the
docstring).
- A config with no `store_url` uses the `store_url` argument, and the
config is left unchanged.
- One default config shared by two clients with different `store_url=`
arguments: each client stores in its own database, and the shared config
is left unchanged.
- A default `HeadroomConfig()` with no argument stores in the temp
directory and creates nothing in the working directory.
**Resulting precedence:** `config.store_url` > `store_url=` argument >
`<system temp>/headroom.db`. Callers who pass only `store_url=` (no
config), or no storage settings at all, see no change.
## Testing
- [x] Unit tests pass (`pytest`): the new file plus every test file that
references `HeadroomClient`, `store_url` or `HeadroomConfig(` (17 files,
run one by one), not the full suite. The Strands and Agno test files
skip entirely in my env because those extras aren't installed
- [x] Linting passes (`ruff check .`): on the changed files
- [ ] Type checking passes (`mypy headroom`): ran `mypy
headroom/client.py headroom/config.py` only
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
New tests without the fix (`client.py` and `config.py` reverted to
`main`):
```text
$ uv run pytest tests/test_client_store_url.py -q
FAILED tests/test_client_store_url.py::test_config_store_url_is_where_metrics_are_stored
FAILED tests/test_client_store_url.py::test_config_store_url_takes_precedence_over_the_argument
FAILED tests/test_client_store_url.py::test_argument_is_used_when_config_has_no_store_url
FAILED tests/test_client_store_url.py::test_shared_config_does_not_leak_one_clients_url_into_another
========================= 4 failed, 1 passed in 1.72s ==========================
```
The first two fail on the storage path (`.../system-temp/headroom.db`
and `.../from-argument.db` instead of the configured database). The last
two store in the right place on `main` but fail on the config-default
assertion, since `main`'s default is `"sqlite:///headroom.db"`. The one
pass is `test_default_config_stores_in_temp_dir_not_cwd`, which is
expected: it guards that default callers keep the temp-directory
location, so it holds before and after.
The shared-config test against this PR's first commit (f96bc8e2, before
the write-back was removed) fails on the leak itself:
```text
E AssertionError: assert PosixPath('/private/var/.../a.db') == PosixPath('/private/var/.../b.db')
FAILED tests/test_client_store_url.py::test_shared_config_does_not_leak_one_clients_url_into_another
```
With the fix:
```text
$ uv run pytest tests/test_client_store_url.py tests/test_config.py -q
============================== 41 passed in 2.20s ==============================
$ for f in <17 test files referencing HeadroomClient / store_url / HeadroomConfig(>; do uv run pytest "$f" -q; done
220 passed, 0 failed (summed across files; 9 skipped in test_chat_model.py)
$ uv run ruff check headroom/client.py headroom/config.py tests/test_client_store_url.py tests/test_config.py
All checks passed!
$ uv run ruff format --check (same files)
4 files already formatted
$ uv run mypy headroom/client.py headroom/config.py
Success: no issues found in 2 source files
```
## Real Behavior Proof
- Environment: macOS 27.0.1 (arm64), Python 3.13.15, `uv sync --extra
dev --extra proxy`, openai 2.41.0, `HeadroomClient` against a local stub
OpenAI upstream.
- Exact command / steps: wrapped the real OpenAI SDK client in
`HeadroomClient(original_client=OpenAI(base_url=<stub>),
provider=OpenAIProvider(),
config=HeadroomConfig(store_url="sqlite:///<work>/mine.db"))`, sent one
`chat.completions.create` request, then counted rows in the `requests`
table of each candidate database, on `main` and on this branch.
- Stub upstream: a small `ThreadingHTTPServer` returning a canned chat
completion.
- The script pointed `tempfile.tempdir` at `<work>/system-temp` and
`chdir`-ed into `<work>/cwd`, so all three possible locations are
isolated and checkable.
- Observed result: on `main` the request's metrics row landed in the
temp-directory database while `validate_setup()` reported `mine.db`; on
this branch it lands in `mine.db`. Nothing is written to the working
directory in either case. Separately, two clients built from one shared
default config with `store_url=` `a.db` and `b.db` store in `a.db` and
`b.db` respectively, and the shared config's `store_url` stays `None`.
| | `main` | this branch |
|---|---|---|
| `validate_setup()["storage"]["url"]` | `sqlite:///<work>/mine.db` |
`sqlite:///<work>/mine.db` |
| rows in `mine.db` | file never created | 1 |
| rows in `<temp>/headroom.db` | 1 | file never created |
| `headroom.db` in working directory | no | no |
- Not tested: a live upstream provider; `jsonl://` and plugin storage
URLs (they go through the same `create_storage` call, only the URL
passed in changed).
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: N/A
- Stable/default behavior changed: no for callers who don't set
`HeadroomConfig.store_url`: the default stays `<system
temp>/headroom.db`, and a `store_url=` argument is honored as before.
Callers who set `config.store_url` now get metrics stored there, which
is what they configured.
- Kill switch / disable path: N/A
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A: the
docs already describe the temp-directory default and `store_url` as the
way to change it)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
`HeadroomConfig().store_url` now reads `None` instead of
`"sqlite:///headroom.db"`, and a client no longer fills it in on a
caller-supplied config. That old value never controlled where anything
was stored, so code that read it was already getting a path nothing
wrote to.
---------
Co-authored-by: JD Davis <mxjerrett@gmail.com>
105 lines
3.3 KiB
Python
105 lines
3.3 KiB
Python
"""``HeadroomConfig.store_url`` must decide where ``HeadroomClient`` stores metrics.
|
|
|
|
The client copied ``config.store_url`` into ``self._store_url`` (which
|
|
``validate_setup()`` reports) but opened storage with its own ``store_url``
|
|
argument, so metrics went to the temp-directory default while the client
|
|
reported the configured path.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import tempfile
|
|
from pathlib import Path
|
|
from unittest.mock import MagicMock
|
|
|
|
import pytest
|
|
|
|
from headroom.client import HeadroomClient
|
|
from headroom.config import HeadroomConfig
|
|
from headroom.providers.openai import OpenAIProvider
|
|
|
|
|
|
@pytest.fixture
|
|
def temp_dir(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
|
|
"""Point the client's temp-directory default at an isolated folder."""
|
|
temp = tmp_path / "system-temp"
|
|
temp.mkdir()
|
|
monkeypatch.setattr(tempfile, "tempdir", str(temp))
|
|
return temp
|
|
|
|
|
|
def _client(**kwargs: object) -> HeadroomClient:
|
|
return HeadroomClient(
|
|
original_client=MagicMock(),
|
|
provider=OpenAIProvider(),
|
|
enable_cache_optimizer=False,
|
|
**kwargs,
|
|
)
|
|
|
|
|
|
def _storage_path(client: HeadroomClient) -> Path:
|
|
# "sqlite:////abs/path" parses to "//abs/path"; resolve() normalizes it.
|
|
return Path(client._storage.db_path).resolve()
|
|
|
|
|
|
def test_config_store_url_is_where_metrics_are_stored(tmp_path: Path, temp_dir: Path) -> None:
|
|
db = tmp_path / "mine.db"
|
|
client = _client(config=HeadroomConfig(store_url=f"sqlite:///{db}"))
|
|
|
|
assert _storage_path(client) == db.resolve()
|
|
assert db.exists()
|
|
assert not (temp_dir / "headroom.db").exists()
|
|
assert client.validate_setup()["storage"]["url"] == f"sqlite:///{db}"
|
|
|
|
|
|
def test_config_store_url_takes_precedence_over_the_argument(
|
|
tmp_path: Path, temp_dir: Path
|
|
) -> None:
|
|
from_config = tmp_path / "from-config.db"
|
|
from_argument = tmp_path / "from-argument.db"
|
|
client = _client(
|
|
store_url=f"sqlite:///{from_argument}",
|
|
config=HeadroomConfig(store_url=f"sqlite:///{from_config}"),
|
|
)
|
|
|
|
assert _storage_path(client) == from_config.resolve()
|
|
assert not from_argument.exists()
|
|
|
|
|
|
def test_argument_is_used_when_config_has_no_store_url(tmp_path: Path, temp_dir: Path) -> None:
|
|
from_argument = tmp_path / "from-argument.db"
|
|
config = HeadroomConfig()
|
|
client = _client(store_url=f"sqlite:///{from_argument}", config=config)
|
|
|
|
assert _storage_path(client) == from_argument.resolve()
|
|
assert config.store_url is None
|
|
|
|
|
|
def test_shared_config_does_not_leak_one_clients_url_into_another(
|
|
tmp_path: Path, temp_dir: Path
|
|
) -> None:
|
|
"""One config object reused for two clients, each with its own database."""
|
|
shared = HeadroomConfig()
|
|
a_db = tmp_path / "a.db"
|
|
b_db = tmp_path / "b.db"
|
|
|
|
client_a = _client(store_url=f"sqlite:///{a_db}", config=shared)
|
|
client_b = _client(store_url=f"sqlite:///{b_db}", config=shared)
|
|
|
|
assert _storage_path(client_a) == a_db.resolve()
|
|
assert _storage_path(client_b) == b_db.resolve()
|
|
assert shared.store_url is None
|
|
|
|
|
|
def test_default_config_stores_in_temp_dir_not_cwd(
|
|
tmp_path: Path, temp_dir: Path, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
cwd = tmp_path / "cwd"
|
|
cwd.mkdir()
|
|
monkeypatch.chdir(cwd)
|
|
|
|
client = _client(config=HeadroomConfig())
|
|
|
|
assert _storage_path(client) == (temp_dir / "headroom.db").resolve()
|
|
assert not (cwd / "headroom.db").exists()
|