fix(integrations): record metric timestamps in UTC, not naive local time (#3117)

## Description

Several framework integrations stamp their metric records with a naive
local `datetime.now()`:

- `OptimizationMetrics` in `langchain/chat_model.py` (two sites)
- `ToolCompressionMetrics` in `autogen/agents.py`, `crewai/agents.py`,
`langchain/agents.py`

Everywhere else in the codebase these metric timestamps are UTC: the
SDK's core recorder `headroom/client.py` uses
`datetime.now(timezone.utc).replace(tzinfo=None)`, and the agno and
strands adapters build the same metric classes with
`datetime.now(timezone.utc)`. `langchain/langsmith.py` then exports
`metrics.timestamp.isoformat()` as an absolute telemetry field
(`headroom.optimization_timestamp`).

A naive local `datetime.now()` records the local wall-clock time with no
offset. Serialized and read downstream as UTC, it is wrong by the local
UTC offset for every user not on UTC (and can produce negative "time
ago" values, the same class as the previously fixed #2910). This aligns
the metric sites to UTC.

The `start_time` / `end_time` calls in `langchain/chat_model.py` are
deliberately left as naive `datetime.now()`: they are only ever
subtracted from each other to compute a duration, where the offset
cancels, so UTC is unnecessary there.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/integrations/autogen/agents.py`,
`headroom/integrations/crewai/agents.py`,
`headroom/integrations/langchain/agents.py`:
`ToolCompressionMetrics(timestamp=...)` now uses
`datetime.now(timezone.utc)`.
- `headroom/integrations/langchain/chat_model.py`: the two
`OptimizationMetrics(timestamp=...)` sites now use
`datetime.now(timezone.utc)`; the four `start_time`/`end_time` duration
calls are unchanged.
- Added `timezone` to the `datetime` import in each of those four files.
- `tests/test_integration_metric_timestamps_utc.py`: new regression test
driving the three framework-free recording seams (autogen's standalone
`_record_metrics`; crewai and langchain via an unbound `_record_metrics`
call with a duck-typed self) and asserting the recorded timestamp is
UTC-aware.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality

### Test Output

```text
tests/test_integration_metric_timestamps_utc.py  ->  3 passed
uvx ruff@0.16.2 check <changed files>  ->  All checks passed!
uvx mypy@1.20.2 <four integration modules>  ->  Success: no issues found in 4 source files
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.16.2 and mypy 1.20.2 via uvx.
- Exact command / steps: reverted the three tested sites to
`datetime.now()` and ran `python -m pytest
tests/test_integration_metric_timestamps_utc.py` -> 3 failed (`assert
ts.tzinfo is not None` -> the recorded timestamp was
`datetime.datetime(2026, 8, 18, 23, 42, 45, ...)` with `tzinfo=None`,
i.e. naive local time); restored `datetime.now(timezone.utc)` and re-ran
-> 3 passed. The adapters import their frameworks lazily, so the
recording functions run without autogen/crewai/langchain installed.
- Observed result: after the fix, each adapter records a UTC-aware
timestamp (`utcoffset() == timedelta(0)`); before, it recorded a naive
local datetime.
- Not tested: the two `OptimizationMetrics` sites in
`chat_model._optimize_messages` (they require a live pipeline + a
wrapped LangChain model to reach); they are the identical one-line
change and are the sites the LangSmith exporter serializes absolutely.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. This is metric-timestamp
construction in the framework adapters, not a rollout-channel-gated
runtime feature.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: integration metric timestamps are now
UTC-aware instead of naive local time. Duration math
(`start_time`/`end_time`) is unchanged.
- Kill switch / disable path: N/A.
- Unsafe override required: no.
- Qualification impact: LangSmith optimization timestamps (and any
absolute use of these metrics) are now correct for non-UTC users.
- Rollback path: revert this PR; the metric sites return to naive
`datetime.now()`.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A:
internal metric fields, no user-facing docs)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

I scoped this to the metric-record timestamps and intentionally left the
duration-only `datetime.now()` calls naive, since converting them would
add no value and the offset cancels in subtraction. Complements the
writer fix in the sibling PR for `utils.format_timestamp`
(aware-datetime handling); the two are independent.
This commit is contained in:
Abhay Singh
2026-09-30 23:56:09 -05:00
committed by GitHub
parent c072251c10
commit 76ef2c3bc9
6 changed files with 114 additions and 11 deletions
+2 -2
View File
@@ -36,7 +36,7 @@ import asyncio
import functools
import logging
from dataclasses import dataclass, field
from datetime import datetime
from datetime import datetime, timezone
from typing import Any
try:
@@ -226,7 +226,7 @@ def _record_metrics(
metric = ToolCompressionMetrics(
tool_name=tool_name,
timestamp=datetime.now(),
timestamp=datetime.now(timezone.utc),
chars_before=chars_before,
chars_after=chars_after,
chars_saved=max(0, chars_saved),
+2 -2
View File
@@ -26,7 +26,7 @@ from __future__ import annotations
import logging
from dataclasses import dataclass, field
from datetime import datetime
from datetime import datetime, timezone
from typing import Any
try:
@@ -294,7 +294,7 @@ class HeadroomToolWrapper(BaseTool): # type: ignore[misc]
metric = ToolCompressionMetrics(
tool_name=self.name,
timestamp=datetime.now(),
timestamp=datetime.now(timezone.utc),
chars_before=chars_before,
chars_after=chars_after,
chars_saved=max(0, chars_saved),
+2 -2
View File
@@ -25,7 +25,7 @@ from __future__ import annotations
import logging
from dataclasses import dataclass, field
from datetime import datetime
from datetime import datetime, timezone
from typing import Any
# LangChain imports - these are optional dependencies
@@ -289,7 +289,7 @@ class HeadroomToolWrapper:
metric = ToolCompressionMetrics(
tool_name=self.name,
timestamp=datetime.now(),
timestamp=datetime.now(timezone.utc),
chars_before=chars_before,
chars_after=chars_after,
chars_saved=max(0, chars_saved),
@@ -33,7 +33,7 @@ import json
import logging
from collections.abc import AsyncIterator, Iterator, Sequence
from dataclasses import dataclass
from datetime import datetime
from datetime import datetime, timezone
from typing import Any
from uuid import UUID, uuid4
@@ -350,7 +350,7 @@ class HeadroomChatModel(BaseChatModel):
# Create metrics
metrics = OptimizationMetrics(
request_id=request_id,
timestamp=datetime.now(),
timestamp=datetime.now(timezone.utc),
tokens_before=result.tokens_before,
tokens_after=result.tokens_after,
tokens_saved=result.tokens_before - result.tokens_after,
@@ -920,7 +920,7 @@ class HeadroomRunnable:
# Track metrics
metrics = OptimizationMetrics(
request_id=str(uuid4()),
timestamp=datetime.now(),
timestamp=datetime.now(timezone.utc),
tokens_before=result.tokens_before,
tokens_after=result.tokens_after,
tokens_saved=result.tokens_before - result.tokens_after,
+2 -2
View File
@@ -39,7 +39,7 @@ from __future__ import annotations
import logging
import os
from dataclasses import dataclass, field
from datetime import datetime
from datetime import datetime, timezone
from typing import Any
from uuid import UUID
@@ -86,7 +86,7 @@ class PendingMetrics:
tokens_saved: int
savings_percent: float
transforms_applied: list[str]
timestamp: datetime = field(default_factory=datetime.now)
timestamp: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
class HeadroomLangSmithCallbackHandler(BaseCallbackHandler):
@@ -0,0 +1,103 @@
"""Integration metric timestamps must be recorded in UTC, not local time.
The SDK's canonical recorder (`headroom/client.py`) and the agno/strands
adapters stamp metric timestamps in UTC, and `langchain/langsmith.py` exports a
metric timestamp via `.isoformat()` as the absolute `headroom.optimization_timestamp`
telemetry field. The autogen/crewai/langchain tool wrappers, and the langsmith
`PendingMetrics` default, used a naive local `datetime.now()`, which records the
local wall-clock time with no offset — read downstream as UTC, it is wrong by the
local UTC offset for every non-UTC user.
These adapters import their frameworks lazily, so the recording and serialization
seams are exercised directly here without installing autogen/crewai/langchain.
"""
from __future__ import annotations
from datetime import datetime, timedelta
from types import SimpleNamespace
from headroom.integrations.autogen.agents import ToolMetricsCollector as AutogenCollector
from headroom.integrations.autogen.agents import _record_metrics as autogen_record
from headroom.integrations.crewai.agents import HeadroomToolWrapper as CrewaiWrapper
from headroom.integrations.crewai.agents import ToolMetricsCollector as CrewaiCollector
from headroom.integrations.langchain.agents import HeadroomToolWrapper as LangchainWrapper
from headroom.integrations.langchain.agents import ToolMetricsCollector as LangchainCollector
def _assert_utc(ts: object) -> None:
assert isinstance(ts, datetime)
# Aware and anchored to UTC (naive local time would have tzinfo is None).
assert ts.tzinfo is not None
assert ts.utcoffset() == timedelta(0)
# Serialization contract: langchain/langsmith exports this value verbatim via
# ``metrics.timestamp.isoformat()``, so the persisted string must carry an
# explicit UTC offset — otherwise a downstream reader silently treats it as
# local time. A naive datetime serializes with no offset at all, and an aware
# ``isoformat()`` yields a valid ``+00:00`` string (never the malformed
# ``+00:00Z`` that appending "Z" to an aware datetime would produce). This
# path does not go through ``headroom.utils.format_timestamp``, so it is
# correct independently of that helper.
serialized = ts.isoformat()
assert serialized.endswith("+00:00"), serialized
# And it round-trips through a strict parser unchanged.
assert datetime.fromisoformat(serialized) == ts
def test_autogen_record_metrics_timestamp_is_utc() -> None:
collector = AutogenCollector()
autogen_record(collector, "search", "the original tool output", "short", True)
_assert_utc(collector.metrics[0].timestamp)
def test_crewai_record_metrics_timestamp_is_utc() -> None:
collector = CrewaiCollector()
wrapper = SimpleNamespace(name="search", _metrics=collector)
# Unbound call with a duck-typed self avoids constructing the CrewAI BaseTool.
CrewaiWrapper._record_metrics(wrapper, "the original tool output", "short", True)
_assert_utc(collector.metrics[0].timestamp)
def test_langchain_record_metrics_timestamp_is_utc() -> None:
collector = LangchainCollector()
wrapper = SimpleNamespace(name="search", _metrics=collector)
LangchainWrapper._record_metrics(wrapper, "the original tool output", "short", True)
_assert_utc(collector.metrics[0].timestamp)
def test_langsmith_exported_optimization_timestamp_is_utc() -> None:
"""The actual telemetry serialization seam must emit an aware UTC offset.
``HeadroomLangSmithCallbackHandler._attach_metrics_to_run`` is the real
persistence/format path: it serializes ``PendingMetrics.timestamp`` via
``.isoformat()`` into the exported ``headroom.optimization_timestamp``
LangSmith metadata field. Regression: ``PendingMetrics`` defaulted its
timestamp to a naive local ``datetime.now()``, so the exported string
carried no offset and a downstream reader silently mis-read it as UTC.
The handler imports LangChain lazily and the export seam does not touch
the framework, so it is exercised unbound with a duck-typed ``self`` —
no langchain/langsmith install required.
"""
from headroom.integrations.langchain.langsmith import (
HeadroomLangSmithCallbackHandler,
PendingMetrics,
)
# The default-constructed timestamp (the recording seam) is UTC-aware.
metrics = PendingMetrics(
tokens_before=100,
tokens_after=40,
tokens_saved=60,
savings_percent=60.0,
transforms_applied=["code"],
)
_assert_utc(metrics.timestamp)
# Drive the real export code path (metrics.timestamp.isoformat()).
handler = SimpleNamespace(_run_metrics={}, _client=None, _auto_update=False)
HeadroomLangSmithCallbackHandler._attach_metrics_to_run(handler, "run-1", metrics)
serialized = handler._run_metrics["run-1"]["headroom.optimization_timestamp"]
assert serialized.endswith("+00:00"), serialized
assert datetime.fromisoformat(serialized).utcoffset() == timedelta(0)