Commit Graph
3 Commits
Author SHA1 Message Date
Tejas ChopraandClaude Opus 5.5 2800932e18 fix(proxy): log the savings and metrics failures that currently vanish (#4084)
## Description

Several savings and metrics paths swallow exceptions and log nothing, so
when one fails, a customer-facing number can come out wrong with no
record of why. This PR adds a log line on each of those paths. The
numbers and the control flow are unchanged.

Refs #4136

| Where | Before | Now |
|---|---|---|
| `tool_schema_savings_policy.reconcile_deferred_tokens` | A failure
left the tool-search deferral credit as booked, so `/stats`
over-reported. Nothing was logged. | Warns once per exception type,
giving the booked token count and the fact that `/stats` is
over-reported. Every occurrence is also logged at debug. |
| `outcome.py` output-savings record/estimate | Booked 0 silently. |
Debug, with the request id. |
| `server.py` `/stats` `output_reduction` | The section went missing
silently. | Debug. |
| `server.py` `create_app` saved-settings apply |
`settings_store.load()` already warns about an unreadable or corrupt
file. Any other error while applying it was swallowed. | Warns with the
settings file path and says to fix or remove that file and restart.
Startup continues. |
| `server.py` security gate, audit record | Debug only. | Warns with the
method and path, and says the request is missing from the
`headroom.audit` trail. |
| `savings_tracker` pricing for an unpriced model | Raised
`RuntimeError` only to reach the fallback, and logged nothing. | Returns
the fallback directly and logs once per model at debug. The `except`
blocks now see only unexpected errors, and log them at debug. |
| `savings_tracker._resolve_litellm_model` probes | Two `except
Exception: pass` blocks. | Debug. litellm raises a bare `Exception` for
a model it does not know, so the catch cannot be narrower. |

**No content at any level.** A tokenizer error can quote the client's
tool schema, and a settings or pricing error can quote a credential. So
every line this PR adds describes a failure with
`describe_exception(exc)` from `headroom/log_safety.py`: the exception
types and code locations of the cause chain, never the message and never
a traceback. Model ids go through `safe_id`, and audit paths are capped
the same way. The full text appears only when the operator sets
`HEADROOM_DEBUG_DUMP=full`.

**Bounded dedupe.** The reconcile guard (keyed by exception type) and
the unpriced-model guard (keyed by model) use the shared `WarnOnce`.
Each key is logged once, at most a fixed number of keys per hour,
followed by one overflow notice. The unpriced-model guard is consulted
only when DEBUG is enabled, so a model is not used up while its line is
suppressed.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/tool_schema_savings_policy.py`:
`_log_reconcile_failure` uses `WarnOnce(32, ...)` and
`describe_exception`.
- `headroom/proxy/outcome.py`, `headroom/proxy/server.py`,
`headroom/proxy/savings_tracker.py`: add the lines in the table above,
all through `describe_exception` and `safe_id`.
`_log_unpriced_model_once` uses `WarnOnce`.
- Tests:
  - `tests/test_tool_schema_savings_policy.py`:
    - a tokenizer error quoting a private-schema canary;
- every captured DEBUG and WARNING record is rendered with
`logging.Formatter()`, which includes `exc_info`, and none contains the
canary;
    - the booked credit stays unchanged;
    - one warning per exception type, a debug line per failure.
  - `tests/test_savings_pricing_fallback_logging.py`:
- a pricing error and a litellm probe error that quote an API-key
canary, with the canary absent from every formatted record;
    - an unpriced model logs once;
    - nothing is used up while DEBUG is off.
- `tests/test_saved_settings_apply_logging.py`: a settings-apply error
that quotes an API-key canary through a real `create_app`. One warning
names the file and the exception type, and no formatted record contains
the canary.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# New regressions against the previous commit's product files (c5850f02), in a scratch worktree:
FAILED tests/test_tool_schema_savings_policy.py::test_reconcile_failure_keeps_private_schema_out_of_every_log_record
  E   assert 'PRIVATE-SCHEMA-CANARY-7f3a' not in "tool-schema...NARY-7f3a'}]"
FAILED tests/test_tool_schema_savings_policy.py::test_reconcile_warns_again_for_a_new_exception_type
FAILED tests/test_savings_pricing_fallback_logging.py::test_unpriced_model_logs_once
FAILED tests/test_savings_pricing_fallback_logging.py::test_credential_bearing_pricing_error_is_described_not_quoted
FAILED tests/test_savings_pricing_fallback_logging.py::test_unpriced_model_not_marked_when_debug_disabled
FAILED tests/test_saved_settings_apply_logging.py::test_settings_apply_failure_warns_without_credential
=================== 6 failed, 10 passed, 1 warning in 3.54s ====================
# This branch:
$ pytest -q -p no:cacheprovider tests/test_tool_schema_savings_policy.py tests/test_savings_pricing_fallback_logging.py tests/test_saved_settings_apply_logging.py
======================== 16 passed, 1 warning in 2.77s =========================
$ pytest -q -p no:cacheprovider <every test file referencing the touched estimators, reconcile_deferred_tokens, settings_store, record_admin_action or log_safety> + the savings/stats/output-shaper files
======================= 401 passed, 1 warning in 44.33s ========================
$ uvx ruff@0.16.8 check <touched files> && uvx ruff@0.16.8 format --check headroom/proxy
All checks passed!
$ mypy headroom/proxy/tool_schema_savings_policy.py headroom/proxy/savings_tracker.py headroom/proxy/server.py headroom/proxy/outcome.py --ignore-missing-imports
Success: no issues found in 4 source files
```

`tests/test_cost_card_attribution.py` and
`tests/test_savings_provider_matrix.py` also fail at collection on a
clean `origin/main`, because the local litellm build prices none of
their candidate models. I left them out of the runs above.

## Real Behavior Proof

- Environment: macOS, Python 3.12.13, this branch at d3834fea, the repo
.venv, a temp `HOME`, `HEADROOM_WORKSPACE_DIR` and `CODEX_HOME`, and
`HEADROOM_OFFLINE=1`.
- Exact command / steps:
- (1) The shared smoke harness, `smoke.sh <worktree>`. It runs a real
`headroom proxy` from this checkout against a local fake upstream, sends
one Anthropic and one OpenAI request that each carry a compressible tool
result, reads `/stats`, and greps `proxy.log`.
  - (2) A script that forces each new path:
- `settings_store.apply_to_environ` raises `RuntimeError("value
sk-secret-should-not-appear")` under `create_app()`;
- `reconcile_deferred_tokens` is called twice with a counter that raises
`RuntimeError("... secret-schema-text")`;
- the unknown model `local-llama-7b` is priced three times with the
savings-tracker logger at DEBUG.
- Observed result:

```text
(1) branch: cleanup/proxy-core-b-savings-logging @ d3834fea
health: 200
POST /v1/messages        -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82

(2) WARNING headroom.proxy: Could not apply saved settings from <tmp>/ws/settings.json (RuntimeError at pcb_4084_proof.py:16 in _boom <- headroom/proxy/server.py:3141 in create_app); continuing with env and defaults. Fix or remove that file and restart the proxy.
WARNING headroom.proxy.tool_schema_savings_policy: tool-schema deferral reconcile failed (RuntimeError at pcb_4084_proof.py:31 in _broken <- headroom/proxy/tool_schema_savings_policy.py:146 in reconcile_deferred_tokens); credit left at 100 booked tokens, so /stats over-reports tool-search savings for this request. Please report this as a bug.
DEBUG headroom.proxy.savings_tracker: litellm cannot price model='local-llama-7b' as given: BadRequestError at get_llm_provider_logic.py:515 in get_llm_provider <- get_llm_provider_logic.py:528 in get_llm_provider <- cost_calculator.py:472 in cost_per_token
DEBUG headroom.proxy.savings_tracker: No litellm input_cost_per_token for model='local-llama-7b'; savings and cost use the fallback rate
```

  - Neither secret string appears in any line.
  - The second reconcile failure does not warn again.
  - The unpriced model is logged once across three calls.
- Not tested:
- Forcing the audit-record failure inside a running proxy. It uses the
same `describe_exception` pattern.
- Seeing the debug lines in a running proxy's `proxy.log`. That depends
on #4137, which makes `HEADROOM_LOG_LEVEL=debug` reach `headroom.*`.
Until it lands, these debug lines show only when the `headroom` logger
is set to DEBUG programmatically, as in the script.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A; logging only.
- Stable/default behavior changed: No. Every dollar value is the same as
before, and an unpriced model now gets its fallback returned directly
instead of through a raise and catch, with an identical result. Only the
log output changes.
- Kill switch / disable path: None needed. Warnings fire only on a real
failure, and they are deduplicated with a bound.
- Unsafe override required: No. `HEADROOM_DEBUG_DUMP=full` is the
existing explicit opt-in for full exception text.
- Qualification impact: None.
- Rollback path: Revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review. Docs for the new
warnings are collected for the single follow-up docs PR.

Depends on #4138; its files (`headroom/log_safety.py`,
`handlers/_debug_dump.py` and their tests) show in this diff until it
merges.

This PR's own diff, excluding #4138's files, is +377/−20, and 256 of
those added lines are tests.

`cost._warn_pricing_once` is not reused for unpriced models because it
logs at WARNING, and it shares its set with the live cost path, which
already warns once per unpriceable model.

For an unknown model, cache savings book 0 while compression savings use
the $3/M fallback. This PR leaves that unchanged for a maintainer
decision.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 11:19:35 -07:00
68359c056c fix(proxy): repair tool-search history in place so signed thinking blocks keep their coordinates (#3457)
## Description

`strip_unsupported_tool_search_blocks()` repairs by removing the
unsupportable
`tool_search_tool_result` and its paired `server_tool_use`. When that
pair sits in an assistant
message that also carries **signed thinking blocks** — the normal shape
of Claude Code traffic with
extended thinking on — the removal moves every later block in that
message.
`thinking_block_fingerprint()` keys a thinking block by `(message_index,
block_index, json)`, so
`thinking_blocks_survived_mutation()` returns False and
`select_outbound_body()` forwards the
client's ORIGINAL bytes, discarding every mutation including the repair.
Upstream then 400s on the
exact reference the repair had already found, and because a resume
replays the same transcript the
session is wedged permanently.

Dropping a whole message (`if not kept: continue`) has the same effect
one level up: it shifts
`message_index` for every message after it.

Fix: repair in place. Swap each unsupportable block for a short constant
text block and never
remove a message, so every thinking block keeps its coordinates and its
bytes. Same reasoning and
same shape as the CCR sibling `strip_unsupported_ccr_retrieve_blocks`
right below it ("Neutralize
rather than drop"), which does not have this bug.

Also: dump the request on a **streaming** upstream error, not only on
the buffered path. Claude
Code streams every turn, so `HEADROOM_DEBUG_DUMP` previously captured
nothing for the most common
400s — that blindness is why this took three separate investigations to
pin down. Same gating as
the existing dumps (off by default, never in stateless mode, redacted
unless `=full`).

Closes #3456

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/helpers.py`: `strip_unsupported_tool_search_blocks()`
replaces unsupportable
blocks with a constant `_TOOL_SEARCH_PLACEHOLDER_BLOCK` text block
instead of deleting them; no
message is ever dropped. Return contract unchanged (`(messages, count)`,
original object by
identity when nothing changed), so the call site and its cache/prefix
behaviour are unchanged.
The placeholder is constant so the repaired prefix stays byte-stable
across turns.
- `headroom/proxy/handlers/anthropic.py`: log line now says "repaired
... (replaced with text in
  place)" to match the behaviour.
- `headroom/proxy/handlers/streaming.py`: write the same diagnostic dump
the non-streaming handlers
  write when an upstream response is >= 400.
- `tests/test_issue_746_tool_search.py`: three tests updated from drop-
to in-place semantics, plus
a new regression test asserting `thinking_block_fingerprint()` is
byte-identical across a repair.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_issue_746_tool_search.py tests/test_proxy/test_tool_search_repair_after_turn_hooks.py -q
63 passed, 1 warning in 3.09s

$ pytest tests -k "tool_search or thinking or body_forward or outbound" -q
208 passed, 26 skipped, 11637 deselected, 1 warning in 16.08s

$ ruff check .
All checks passed!

$ mypy headroom
Success: no issues found in 530 source files
```

## Real Behavior Proof

- Environment: proxy 0.38.0-dev (this branch on `main` @ e67b3c8a),
Python 3.14, Fedora 44,
`HEADROOM_MODE=token HEADROOM_BACKEND=anthropic`, upstream
`api.anthropic.com`, live Claude Code
2.1.263 session (claude-opus-5, extended thinking on, ~533 messages / 19
MB body / 2436 client
  tools).
- Exact command / steps: reproduce the wedged session, capture the real
outbound body with `HEADROOM_DEBUG_DUMP=full`, then replay that body
through the helper before and after this change.
1. The session had looped on `API Error: 400 Tool reference
'mcp__claude_ai_Instacart__cart' not
found in available tools` for every turn — a claude.ai connector that no
longer advertised
that tool, while 18 `tool_search_tool_result` blocks in the transcript
still referenced it.
2. Captured the real outbound body with `HEADROOM_DEBUG_DUMP=full` plus
the new streaming dump.
3. Replayed the captured body through the helper, before and after this
change.
- Observed result: before, the repair was computed and then discarded
and upstream still returned 400; after, the same body repairs with every
thinking-block fingerprint byte-identical and the wedged live session
answers normally again.
- Before: proxy logged `Tool search: dropped 2 unsupportable history
block(s)` and
`body_mutated=true mutation_reasons=...,tool_search_history_repair`,
upstream still returned
400. Instrumented at the call site: `TSDIAG tools=2437 search_tools=1
ts_blocks=18
    unsat_blocks=1 msgs=533` — detected, repaired, discarded.
  - After, replaying the captured 19 MB body through the helper:
    ```
no search tool (client sent all): repaired=36 unsatisfied_refs_left=0
msgs=533 thinking_fingerprint_identical=True
with injected search tool: repaired=2 unsatisfied_refs_left=0 msgs=533
thinking_fingerprint_identical=True
    ```
- Live: restarted the proxy on this branch and prompted the same wedged
session — it answered
normally on the first turn, full history preserved, and has kept
answering with the repair
    firing each turn and no further 400s.
- Not tested: the OpenAI/Codex handlers (untouched by the helper change;
they do not carry
Anthropic signed thinking blocks), and Copilot/third-party
Anthropic-compatible upstreams, where
client-originated `tool_search_tool_*` entries are stripped before this
stage.

## Runtime Rollout Safety

- Rollout-managed feature(s): none — this is a correctness fix inside
the existing, unconditional
  tool-search history repair (#2807 / #2889).
- Minimum rollout channel: n/a (no flag).
- Stable/default behavior changed: yes, narrowly — an unsupportable
tool-search block is now
forwarded as a short text block instead of being removed, and an
assistant turn that consisted
only of tool-search bookkeeping keeps its message slot as text blocks
instead of disappearing.
Both shapes are valid to the API; the previous shapes were valid too,
they just could not survive
  the signed-thinking passthrough.
- Kill switch / disable path: `HEADROOM_TOOL_SEARCH=0` still disables
Headroom's own deferral
injection; the repair itself is deliberately unconditional (transcripts
poisoned before the flag
  was turned off must still recover), unchanged by this PR.
- Unsafe override required: no.
- Qualification impact: none expected; token accounting is unaffected
beyond a few tokens of
placeholder text per repaired block (repairs are rare and the text is
~10 tokens).
- Rollback path: revert this commit — behaviour returns to removal, i.e.
to the bug.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
2026-09-13 08:04:31 -07:00
Tejas Chopra 51a3b01174 feat(security): pilot hardening — stateless guarantee, model pinning, CI security gate (#1515)
## Description

Engineering hardening derived from the Box vendor security assessment.
Each change turns a "No/Partial" questionnaire answer into a genuine
"Yes" by making the product safer — not by editing the form. The
throughline is Headroom's core promise to enterprise pilots: **it runs
inside the customer's environment and never persists or leaks their
data.** These changes make that provable.

Three themes: (1) a complete **stateless write guarantee** (a stateless
proxy writes nothing to the workspace during serving), (2)
**data-at-rest** protection (no cleartext prompts written on errors),
and (3) **supply-chain integrity** (all model downloads pinned;
SCA/SAST/secret-scanning in CI).

Closes # (no tracking issue)

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

> Note: two deliberate, reversible default changes (not breaking): the
upstream-error debug dump is now off by default
(`HEADROOM_DEBUG_DUMP=1`/`=full` to opt in), and model downloads are
pinned (`HEADROOM_HF_PIN=off` to bypass). All stateless plumbing is a
pure no-op when not stateless.

## Changes Made

- **Stateless writes** — savings tracker + ledger, TOIN (`toin.json`),
and the output-savings recorder now honor stateless (in-memory only);
persistent memory is disabled under stateless with a warning. Added a
process-wide flag `headroom.paths.process_is_stateless()` (also honors
`HEADROOM_STATELESS`).
- **Debug dump** — the Anthropic *and* OpenAI handlers wrote full
requests (cleartext prompts/tools/system) to
`~/.headroom/logs/debug_400/` on every ≥400, even stateless. Now OFF by
default, stateless-aware, with a redacted middle tier; helpers extracted
to `handlers/_debug_dump.py`.
- **Model pinning** — all model downloads pin an immutable commit SHA:
our repos, kompress, image router/SigLIP, the third-party Qdrant memory
embedder (centralized in `onnx_runtime`), and the fastembed relevance
model (via the `revision` kwarg fastembed forwards to
`snapshot_download`). `HEADROOM_HF_PIN=off` bypasses.
- **CI security gate** — new `security.yml`: dependency audit
(pip-audit, scoped to the CVE-free `[all]` set), CodeQL (Python +
JS/TS), and gitleaks secret scanning (binary, MIT-licensed; PR-diff
scoped). `.gitleaks.toml` allowlists SBOM/lockfiles.
- **Dependabot** — extended to Rust (cargo) and npm (TS SDK, plugins,
docs).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ ruff check <10 changed source files>
All checks passed!

$ mypy <changed source files>
Success: no issues found in 7 source files   # + handlers/server: no issues (annotation-unchecked notes only)

$ pytest tests/test_stateless_writers.py tests/test_stateless_toin.py \
    tests/test_debug_dump_gating.py tests/test_hf_revision_pinning.py \
    tests/test_proxy_savings_history.py tests/test_observability_metrics.py \
    tests/test_toin.py tests/test_paths.py -q
================= 176 passed, 6 skipped, 2 warnings in 13.92s ==================
```

New tests (18): `tests/test_stateless_writers.py`,
`tests/test_stateless_toin.py`, `tests/test_debug_dump_gating.py`,
`tests/test_hf_revision_pinning.py`, plus stateless control assertions
in `tests/test_proxy_savings_history.py`. They include the non-stateless
control cases (savings/TOIN still persist) and a regression guard that
fails if any handler writes a debug dump without gating it.

## Real Behavior Proof

- **Environment:** macOS, Python 3.12 (`.venv`); CI on `ubuntu-latest`.
- **Exact command / steps:**
- Stateless guarantee: `SavingsRecorder(tmp/"output_savings.json")` +
`set_process_stateless(True)` → `flush()`; TOIN
`ToolIntelligenceNetwork(TOINConfig(storage_path=""))`;
`create_app(ProxyConfig(memory_enabled=True, stateless=True))`.
- Debug-dump gating: `_debug_dump_mode(SimpleNamespace(stateless=...))`
across env values.
- Model pinning: model SHAs fetched/verified against the live
HuggingFace API; `_resolve_revision` / `_pinned_revision` resolvers
tested.
- **Observed result:** under stateless, no `proxy_savings.json` /
`savings_events.jsonl` / `toin.json` / `output_savings.json` /
`memory.db` is created; `proxy.memory_handler is None`. With
`stateless=False` the control tests confirm each still persists. Debug
dump resolves to `off` by default and is forced off in stateless. CI:
dependency-audit, CodeQL (python + js/ts) pass; secret-scan now runs the
gitleaks binary.
- **Not tested:** `pip-audit` was not run on the local machine (broken
`ensurepip`); CI is the first real run (the committed all-extras grype
scan is clean). The fastembed download path is exercised by CI/runtime,
not in unit tests (the revision resolver is unit-tested).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

- **Concurrency:** `stateless` is a per-process config flag, never
per-request/per-session. Many sessions share one proxy's setting; a
stateless and a stateful proxy are separate OS processes with isolated
state. The one in-process edge (two proxies, different settings —
essentially tests) fails closed to in-memory, so a stateless proxy can
never leak.
- **Memory under stateless** is *disabled* (not in-RAM): the memory
subsystem is multi-component (SQLite + vector + markdown bridge) and a
partial in-RAM mode would be risky; ephemeral containers and
cross-session learning are contradictory. An ephemeral in-RAM memory
mode is a possible follow-up.
- **Docs/CHANGELOG** left unchecked: the two new env vars
(`HEADROOM_DEBUG_DUMP`, `HEADROOM_HF_PIN`) and the stateless behavior
changes are documented in code comments; happy to add user docs + a
CHANGELOG entry if preferred.
- CI deprecation warnings (Node 20, CodeQL Action v3) are GitHub-side
and out of scope here.
2026-06-27 17:44:10 -07:00