mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(integrations): record metric timestamps in UTC, not naive local time (#3117)
## Description Several framework integrations stamp their metric records with a naive local `datetime.now()`: - `OptimizationMetrics` in `langchain/chat_model.py` (two sites) - `ToolCompressionMetrics` in `autogen/agents.py`, `crewai/agents.py`, `langchain/agents.py` Everywhere else in the codebase these metric timestamps are UTC: the SDK's core recorder `headroom/client.py` uses `datetime.now(timezone.utc).replace(tzinfo=None)`, and the agno and strands adapters build the same metric classes with `datetime.now(timezone.utc)`. `langchain/langsmith.py` then exports `metrics.timestamp.isoformat()` as an absolute telemetry field (`headroom.optimization_timestamp`). A naive local `datetime.now()` records the local wall-clock time with no offset. Serialized and read downstream as UTC, it is wrong by the local UTC offset for every user not on UTC (and can produce negative "time ago" values, the same class as the previously fixed #2910). This aligns the metric sites to UTC. The `start_time` / `end_time` calls in `langchain/chat_model.py` are deliberately left as naive `datetime.now()`: they are only ever subtracted from each other to compute a duration, where the offset cancels, so UTC is unnecessary there. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/integrations/autogen/agents.py`, `headroom/integrations/crewai/agents.py`, `headroom/integrations/langchain/agents.py`: `ToolCompressionMetrics(timestamp=...)` now uses `datetime.now(timezone.utc)`. - `headroom/integrations/langchain/chat_model.py`: the two `OptimizationMetrics(timestamp=...)` sites now use `datetime.now(timezone.utc)`; the four `start_time`/`end_time` duration calls are unchanged. - Added `timezone` to the `datetime` import in each of those four files. - `tests/test_integration_metric_timestamps_utc.py`: new regression test driving the three framework-free recording seams (autogen's standalone `_record_metrics`; crewai and langchain via an unbound `_record_metrics` call with a duck-typed self) and asserting the recorded timestamp is UTC-aware. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality ### Test Output ```text tests/test_integration_metric_timestamps_utc.py -> 3 passed uvx ruff@0.16.2 check <changed files> -> All checks passed! uvx mypy@1.20.2 <four integration modules> -> Success: no issues found in 4 source files ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1, ruff 0.16.2 and mypy 1.20.2 via uvx. - Exact command / steps: reverted the three tested sites to `datetime.now()` and ran `python -m pytest tests/test_integration_metric_timestamps_utc.py` -> 3 failed (`assert ts.tzinfo is not None` -> the recorded timestamp was `datetime.datetime(2026, 8, 18, 23, 42, 45, ...)` with `tzinfo=None`, i.e. naive local time); restored `datetime.now(timezone.utc)` and re-ran -> 3 passed. The adapters import their frameworks lazily, so the recording functions run without autogen/crewai/langchain installed. - Observed result: after the fix, each adapter records a UTC-aware timestamp (`utcoffset() == timedelta(0)`); before, it recorded a naive local datetime. - Not tested: the two `OptimizationMetrics` sites in `chat_model._optimize_messages` (they require a live pipeline + a wrapped LangChain model to reach); they are the identical one-line change and are the sites the LangSmith exporter serializes absolutely. ## Runtime Rollout Safety - Rollout-managed feature(s): none. This is metric-timestamp construction in the framework adapters, not a rollout-channel-gated runtime feature. - Minimum rollout channel: N/A. - Stable/default behavior changed: integration metric timestamps are now UTC-aware instead of naive local time. Duration math (`start_time`/`end_time`) is unchanged. - Kill switch / disable path: N/A. - Unsafe override required: no. - Qualification impact: LangSmith optimization timestamps (and any absolute use of these metrics) are now correct for non-UTC users. - Rollback path: revert this PR; the metric sites return to naive `datetime.now()`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation (N/A: internal metric fields, no user-facing docs) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes I scoped this to the metric-record timestamps and intentionally left the duration-only `datetime.now()` calls naive, since converting them would add no value and the offset cancels in subtraction. Complements the writer fix in the sibling PR for `utils.format_timestamp` (aware-datetime handling); the two are independent.
This commit is contained in:
@@ -36,7 +36,7 @@ import asyncio
|
||||
import functools
|
||||
import logging
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any
|
||||
|
||||
try:
|
||||
@@ -226,7 +226,7 @@ def _record_metrics(
|
||||
|
||||
metric = ToolCompressionMetrics(
|
||||
tool_name=tool_name,
|
||||
timestamp=datetime.now(),
|
||||
timestamp=datetime.now(timezone.utc),
|
||||
chars_before=chars_before,
|
||||
chars_after=chars_after,
|
||||
chars_saved=max(0, chars_saved),
|
||||
|
||||
@@ -26,7 +26,7 @@ from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any
|
||||
|
||||
try:
|
||||
@@ -294,7 +294,7 @@ class HeadroomToolWrapper(BaseTool): # type: ignore[misc]
|
||||
|
||||
metric = ToolCompressionMetrics(
|
||||
tool_name=self.name,
|
||||
timestamp=datetime.now(),
|
||||
timestamp=datetime.now(timezone.utc),
|
||||
chars_before=chars_before,
|
||||
chars_after=chars_after,
|
||||
chars_saved=max(0, chars_saved),
|
||||
|
||||
@@ -25,7 +25,7 @@ from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any
|
||||
|
||||
# LangChain imports - these are optional dependencies
|
||||
@@ -289,7 +289,7 @@ class HeadroomToolWrapper:
|
||||
|
||||
metric = ToolCompressionMetrics(
|
||||
tool_name=self.name,
|
||||
timestamp=datetime.now(),
|
||||
timestamp=datetime.now(timezone.utc),
|
||||
chars_before=chars_before,
|
||||
chars_after=chars_after,
|
||||
chars_saved=max(0, chars_saved),
|
||||
|
||||
@@ -33,7 +33,7 @@ import json
|
||||
import logging
|
||||
from collections.abc import AsyncIterator, Iterator, Sequence
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
@@ -350,7 +350,7 @@ class HeadroomChatModel(BaseChatModel):
|
||||
# Create metrics
|
||||
metrics = OptimizationMetrics(
|
||||
request_id=request_id,
|
||||
timestamp=datetime.now(),
|
||||
timestamp=datetime.now(timezone.utc),
|
||||
tokens_before=result.tokens_before,
|
||||
tokens_after=result.tokens_after,
|
||||
tokens_saved=result.tokens_before - result.tokens_after,
|
||||
@@ -920,7 +920,7 @@ class HeadroomRunnable:
|
||||
# Track metrics
|
||||
metrics = OptimizationMetrics(
|
||||
request_id=str(uuid4()),
|
||||
timestamp=datetime.now(),
|
||||
timestamp=datetime.now(timezone.utc),
|
||||
tokens_before=result.tokens_before,
|
||||
tokens_after=result.tokens_after,
|
||||
tokens_saved=result.tokens_before - result.tokens_after,
|
||||
|
||||
@@ -39,7 +39,7 @@ from __future__ import annotations
|
||||
import logging
|
||||
import os
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any
|
||||
from uuid import UUID
|
||||
|
||||
@@ -86,7 +86,7 @@ class PendingMetrics:
|
||||
tokens_saved: int
|
||||
savings_percent: float
|
||||
transforms_applied: list[str]
|
||||
timestamp: datetime = field(default_factory=datetime.now)
|
||||
timestamp: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
|
||||
|
||||
|
||||
class HeadroomLangSmithCallbackHandler(BaseCallbackHandler):
|
||||
|
||||
@@ -0,0 +1,103 @@
|
||||
"""Integration metric timestamps must be recorded in UTC, not local time.
|
||||
|
||||
The SDK's canonical recorder (`headroom/client.py`) and the agno/strands
|
||||
adapters stamp metric timestamps in UTC, and `langchain/langsmith.py` exports a
|
||||
metric timestamp via `.isoformat()` as the absolute `headroom.optimization_timestamp`
|
||||
telemetry field. The autogen/crewai/langchain tool wrappers, and the langsmith
|
||||
`PendingMetrics` default, used a naive local `datetime.now()`, which records the
|
||||
local wall-clock time with no offset — read downstream as UTC, it is wrong by the
|
||||
local UTC offset for every non-UTC user.
|
||||
|
||||
These adapters import their frameworks lazily, so the recording and serialization
|
||||
seams are exercised directly here without installing autogen/crewai/langchain.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timedelta
|
||||
from types import SimpleNamespace
|
||||
|
||||
from headroom.integrations.autogen.agents import ToolMetricsCollector as AutogenCollector
|
||||
from headroom.integrations.autogen.agents import _record_metrics as autogen_record
|
||||
from headroom.integrations.crewai.agents import HeadroomToolWrapper as CrewaiWrapper
|
||||
from headroom.integrations.crewai.agents import ToolMetricsCollector as CrewaiCollector
|
||||
from headroom.integrations.langchain.agents import HeadroomToolWrapper as LangchainWrapper
|
||||
from headroom.integrations.langchain.agents import ToolMetricsCollector as LangchainCollector
|
||||
|
||||
|
||||
def _assert_utc(ts: object) -> None:
|
||||
assert isinstance(ts, datetime)
|
||||
# Aware and anchored to UTC (naive local time would have tzinfo is None).
|
||||
assert ts.tzinfo is not None
|
||||
assert ts.utcoffset() == timedelta(0)
|
||||
# Serialization contract: langchain/langsmith exports this value verbatim via
|
||||
# ``metrics.timestamp.isoformat()``, so the persisted string must carry an
|
||||
# explicit UTC offset — otherwise a downstream reader silently treats it as
|
||||
# local time. A naive datetime serializes with no offset at all, and an aware
|
||||
# ``isoformat()`` yields a valid ``+00:00`` string (never the malformed
|
||||
# ``+00:00Z`` that appending "Z" to an aware datetime would produce). This
|
||||
# path does not go through ``headroom.utils.format_timestamp``, so it is
|
||||
# correct independently of that helper.
|
||||
serialized = ts.isoformat()
|
||||
assert serialized.endswith("+00:00"), serialized
|
||||
# And it round-trips through a strict parser unchanged.
|
||||
assert datetime.fromisoformat(serialized) == ts
|
||||
|
||||
|
||||
def test_autogen_record_metrics_timestamp_is_utc() -> None:
|
||||
collector = AutogenCollector()
|
||||
autogen_record(collector, "search", "the original tool output", "short", True)
|
||||
_assert_utc(collector.metrics[0].timestamp)
|
||||
|
||||
|
||||
def test_crewai_record_metrics_timestamp_is_utc() -> None:
|
||||
collector = CrewaiCollector()
|
||||
wrapper = SimpleNamespace(name="search", _metrics=collector)
|
||||
# Unbound call with a duck-typed self avoids constructing the CrewAI BaseTool.
|
||||
CrewaiWrapper._record_metrics(wrapper, "the original tool output", "short", True)
|
||||
_assert_utc(collector.metrics[0].timestamp)
|
||||
|
||||
|
||||
def test_langchain_record_metrics_timestamp_is_utc() -> None:
|
||||
collector = LangchainCollector()
|
||||
wrapper = SimpleNamespace(name="search", _metrics=collector)
|
||||
LangchainWrapper._record_metrics(wrapper, "the original tool output", "short", True)
|
||||
_assert_utc(collector.metrics[0].timestamp)
|
||||
|
||||
|
||||
def test_langsmith_exported_optimization_timestamp_is_utc() -> None:
|
||||
"""The actual telemetry serialization seam must emit an aware UTC offset.
|
||||
|
||||
``HeadroomLangSmithCallbackHandler._attach_metrics_to_run`` is the real
|
||||
persistence/format path: it serializes ``PendingMetrics.timestamp`` via
|
||||
``.isoformat()`` into the exported ``headroom.optimization_timestamp``
|
||||
LangSmith metadata field. Regression: ``PendingMetrics`` defaulted its
|
||||
timestamp to a naive local ``datetime.now()``, so the exported string
|
||||
carried no offset and a downstream reader silently mis-read it as UTC.
|
||||
|
||||
The handler imports LangChain lazily and the export seam does not touch
|
||||
the framework, so it is exercised unbound with a duck-typed ``self`` —
|
||||
no langchain/langsmith install required.
|
||||
"""
|
||||
from headroom.integrations.langchain.langsmith import (
|
||||
HeadroomLangSmithCallbackHandler,
|
||||
PendingMetrics,
|
||||
)
|
||||
|
||||
# The default-constructed timestamp (the recording seam) is UTC-aware.
|
||||
metrics = PendingMetrics(
|
||||
tokens_before=100,
|
||||
tokens_after=40,
|
||||
tokens_saved=60,
|
||||
savings_percent=60.0,
|
||||
transforms_applied=["code"],
|
||||
)
|
||||
_assert_utc(metrics.timestamp)
|
||||
|
||||
# Drive the real export code path (metrics.timestamp.isoformat()).
|
||||
handler = SimpleNamespace(_run_metrics={}, _client=None, _auto_update=False)
|
||||
HeadroomLangSmithCallbackHandler._attach_metrics_to_run(handler, "run-1", metrics)
|
||||
|
||||
serialized = handler._run_metrics["run-1"]["headroom.optimization_timestamp"]
|
||||
assert serialized.endswith("+00:00"), serialized
|
||||
assert datetime.fromisoformat(serialized).utcoffset() == timedelta(0)
|
||||
Reference in New Issue
Block a user