mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description
The LangChain, CrewAI and AutoGen tool wrappers each had their own
verbatim copy of `ToolCompressionMetrics`, `ToolMetricsCollector` and
the `_record_metrics` body. The UTC timestamp fix (#3117) had to land in
all three. `OptimizationMetrics` was also byte-identical in the
LangChain, Agno and Strands model wrappers.
This PR moves them into one stdlib-only module,
`headroom/integrations/_metrics.py`. Every integration re-exports the
names, so all existing import paths still work
(`headroom.integrations.*`, `.langchain`, `.crewai`, `.autogen`,
`.agno`, `.strands`, and each `agents` / `model` module).
Refs #4136
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring
## Changes Made
- New `headroom/integrations/_metrics.py`: `OptimizationMetrics`,
`ToolCompressionMetrics`, `ToolMetricsCollector` and the private helper
`_record_tool_metrics(collector, tool_name, original, compressed,
was_compressed, log)`.
- `langchain/agents.py`, `crewai/agents.py`, `autogen/agents.py`: drop
their copies and import the shared ones. Each module keeps its own
`_global_metrics`, `get_tool_metrics`, `reset_tool_metrics`, its
module-local `compress_tool_result` import (tests patch it per module)
and its `_record_metrics` (now a delegate with the same signature).
- Each wrapper passes its own module logger, so the savings line keeps
its logger name. The LangChain line moved from an f-string to `%`-style.
The text is identical: `{r:.1%}` and `"%.1f%%" % (r*100)` matched on
200,000 random ratios.
- `langchain/chat_model.py`, `agno/model.py`, `strands/model.py`: drop
their `OptimizationMetrics` copies and import the shared one.
- Tests: drop the CrewAI and AutoGen copies of the dataclass and
collector tests (9 tests).
`tests/test_integrations/langchain/test_agents.py` covers the same cases
(create, required fields, empty summary, add and summary, cap at 1000)
against the now-shared class.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
# repo venv: langchain/test_agents, test_extended, timestamps_utc, crewai, autogen, langchain/test_chat_model, agno/test_model, strands/test_model_unit
origin/main: 165 passed, 65 skipped
this branch: 165 passed, 56 skipped (the 9 removed tests were skipped here: no crewai/autogen in the repo venv)
# scratch venv with crewai 1.15.27 + autogen-agentchat 0.7.5 (these files never run in CI)
$ pytest tests/test_integrations/crewai tests/test_integrations/autogen tests/test_integration_metric_timestamps_utc.py
origin/main: 28 passed
this branch: 19 passed (9 removed)
# scratch venv with strands-agents: tests/integrations/test_strands/test_model_unit.py test_hooks_unit.py
origin/main: 61 passed this branch: 61 passed
# tests/test_integrations/agno/test_model.py
origin/main: 54 passed, 5 skipped this branch: 54 passed, 5 skipped
$ uvx ruff@0.16.8 check <touched files> -> All checks passed!
$ uvx ruff@0.16.8 format <touched files> -> unchanged
$ mypy <7 touched product files> --ignore-missing-imports
headroom/integrations/langchain/chat_model.py:380: error: Function "RunnableBinding" could always be true [truthy-function]
(pre-existing: the same error on origin/main, on an untouched line)
```
## Real Behavior Proof
- Environment: macOS, Python 3.12, repo venv plus a scratch venv with
crewai 1.15.27, autogen-agentchat 0.7.5 and strands-agents.
- Exact command / steps: ran the test sets above on origin/main and on
this branch. Then `python -c "import headroom.integrations as I;
print(I.ToolMetricsCollector is I.CrewAIToolMetricsCollector is
I.AutoGenToolMetricsCollector, I.OptimizationMetrics is
I.AgnoOptimizationMetrics)"`.
- Observed result: same pass counts, minus the 9 removed tests. Prints
`True True`. All public names and aliases still import from their old
paths.
- Smoke harness (real proxy started from this checkout against a local
fake upstream; temp HOME and workspace, offline):
```text
branch: cleanup/cli-integrations-shared-tool-metrics @ 8b12da71
health: 200
POST /v1/messages -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines (<tmp>/ws/logs/proxy-63816.log):
2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes.
2 headroom.transforms.kompress_compressor - WARNING - Kompress: 'onnx/kompress-int8-wo.onnx' for chopratejas/kompress-v2-base is not cached and HEADROOM_OFFLINE forbids fetching it (HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.). Compression falls back to the non-ML path.
4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base
4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.
(smoke dir: <tmp>)
```
- Direct runtime command on this PR's own path (temp HOME,
HEADROOM_WORKSPACE_DIR and CODEX_HOME):
```text
$ python metrics_run.py # real HeadroomToolWrapper around a LangChain tool returning 300 JSON rows
## pr11
INFO headroom.integrations.langchain.agents: HeadroomToolWrapper[list_items]: 14480 -> 12681 chars (1799 saved, 87.6% of original)
langchain summary: {'total_invocations': 1, 'total_compressions': 1}
same class across frameworks: True
__module__: headroom.integrations._metrics
crewai/autogen collectors untouched: 0 0
## clean
INFO headroom.integrations.langchain.agents: HeadroomToolWrapper[list_items]: 14480 -> 12681 chars (1799 saved, 87.6% of original)
langchain summary: {'total_invocations': 1, 'total_compressions': 1}
same class across frameworks: False
__module__: headroom.integrations.langchain.agents
crewai/autogen collectors untouched: 0 0
```
- Not tested: a live agent run against a real model in each framework.
## Runtime Rollout Safety
- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a, library refactor.
- Stable/default behavior changed: Yes, class identity only (see
Additional Notes). Same fields, same summary output, same log text and
logger names.
- Kill switch / disable path: n/a.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this commit.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A
## Additional Notes
Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review.
- **User-visible change: the per-framework classes are now one class.**
- `ToolCompressionMetrics` and `ToolMetricsCollector` are the same
object in `langchain`, `crewai` and `autogen`
(`CrewAIToolCompressionMetrics`, `AutoGenToolCompressionMetrics` and the
other aliases included). `OptimizationMetrics` is the same object in
`langchain`, `agno` (`AgnoOptimizationMetrics`) and `strands`.
- So `isinstance` and `==` between, e.g., a CrewAI and a LangChain
`ToolCompressionMetrics` now return True; before, both returned False
(the dataclass `__eq__` compares `__class__`).
- `__module__` is now `headroom.integrations._metrics`. Pickles made
with this version reference that path, so an older headroom cannot load
them; old pickles still load, because the old paths are re-exported.
- Fields, signatures, methods and `repr` are unchanged. No in-repo or
known consumer code depends on the classes being distinct.
- New names: the module `headroom.integrations._metrics` and the helper
`_record_tool_metrics`, imported under that private name in the three
`agents` modules. No new public name.
- The stdlib helpers `dataclass`, `field`, `datetime` and `timezone` are
no longer attributes of the three `agents` modules, and `dataclass` is
no longer one of `chat_model` / `strands.model`. These were incidental
imports, not API.
- Docs need no change (`ToolMetricsCollector` in crewai.mdx and
autogen.mdx still works as written).
- No new tests and no docs edits: the moved code is covered by the
existing tests.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4132"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->