57 Commits
Author SHA1 Message Date
SToneX a4445e39a4 ci: run the catalog validator
scripts/catalog_validate.py failed on main unnoticed because CI never ran it; only the unit tests
exercised pieces of it. Run it after the import-boundary check, under the same docs-only skip, so
a catalog data PR that breaks a rule fails the build.
2026-09-27 18:59:24 +08:00
b4ed9253d7 feat(catalog): serve herus13 Apify actors on the platform key with capped billing (#657)
* fix(catalog): keep Apify actor starts off treg's shared key

apify.web.scrape.job.start is priced free because the run bills later by the
actor's own pricing, and nothing on the shared-key path meters that run. On
treg's key it let any caller run any actor on treg's Apify account at no
charge. It now needs the team's own Apify key; job.status and job.results
stay open because per-team ownership already scopes them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat(catalog): add Weibo, TikTok Shop, Lazada and Google Maps Apify actors

Combines #641-#644 into one catalog change: nine run-sync entries over four
herus13 actors, plus the lazada platform. The six Weibo entries serve on
treg's key at a price observed on it. Lazada, Google Maps and TikTok Shop are
own-key only: their actors bill run compute or a per-GB start fee that a
per-result price cannot meter.

Co-authored-by: Herus13 <bootforge.ai@gmail.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(catalog): require own keys for Weibo actor runs

* test(frontend): isolate landing interactions from continuous WebGL rendering

* feat(call): let platform_request pin query parameters

Some upstreams take their spend bounds as query options rather than body
fields (Apify's maxTotalChargeUsd, memory and timeout run options). A
queryParams pin must be sent exactly once and is compared as the pinned
value's type; own credentials keep the upstream contract.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(settle): bill Apify platform calls by returned dataset rows

run-sync-get-dataset-items answers a bare array, so every Apify per_result
call settled at its estimate whatever it returned. Count the rows, add an
optional per-row-independent call_fee for the actor's start or compute
charge, and require maxItems (1-200) on the platform key so the hold is the
worst case. Apify's usageTotalUsd lags a finished run by minutes, so the
response body is the settlement evidence.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat(catalog): serve the herus13 Apify actors on treg's key

Weibo, Google Maps, TikTok Shop and Lazada now settle on the platform key by
counted dataset rows plus a flat call_fee (actor start, or Lazada's run
compute). memory and timeout are pinned so the fee is fixed and a run ends
before Apify's synchronous wait; TikTok Shop is held to keyword search.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(settle): bound Apify platform calls by maxTotalChargeUsd

maxItems does not bind pay-per-event actors whose own input sets the row
count, so a one-row hold could settle thousands of rows. Require the
maxTotalChargeUsd run option Apify enforces (at most $1), hold it plus
call_fee, accept only the run options each once in plain ASCII, and bill the
hold when a run reaches its cap. Pin meta-ads enrichment off and bill the
LinkedIn actor-start event on treg's key.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat(catalog): cap herus13 Apify runs by maxTotalChargeUsd on treg's key

Row notes name maxTotalChargeUsd as the enforced spend limit; Lazada's
compute fee is 0.015 under a 180-second timeout pin.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* docs(catalog): name maxTotalChargeUsd as the Apify spend cap

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(settle): treat an Apify run within two rows of its hold as capped

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(call): require a short timeout and a three-row cap on Apify platform runs

Past Apify's 300-second synchronous wait a run answers 408 and keeps billing,
so every Apify per_result platform call now names timeout <= 280. A cap under
call_fee plus three rows would bill an empty answer in full, so it is refused.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(call): keep Apify platform runs inside every wait

treg's upstream read timeout (call_timeout_s, 180 s) and the MCP client's
120 s end the call before a 280 s run finishes, releasing the hold unbilled
while the run keeps billing. Bound timeout to 90 s and 30 s under
call_timeout_s.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat(catalog): pin herus13 Apify runs to a 90-second timeout on treg's key

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(settle): bill a timed-out Apify platform run at its hold

A run that outlives its own timeout answers 400 run-failed with no rows, but
Apify billed its events up to the caller's cap and the run id in that body
reads the dataset. The caller chose the run's size and timeout, so the hold
settles instead of releasing. The minimum-cap check compares micro-USD.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* docs(catalog): tell Lazada callers to keep platform runs small

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(settle): release timed-out Apify runs; the account must stay Restricted

Billing the whole cap on a TIMED-OUT 400 overcharged callers: a run that
timed out after its start event cost Apify $0.00005 and would have billed the
full cap. The loss treg absorbs stays bounded by the $1 cap, and keeping the
Apify account's resource access Restricted stops anyone reading the unbilled
run's rows by id. Examples now show the 90-second platform timeout.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* docs(money): name disconnects as a bounded Apify loss; call_fee wording

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* docs(catalog): record Lazada's 0.05 minimum cap and 10-product floor

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(asynctasks): keep the Bright Data and CompanyEnrich platform keys the merge dropped

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(catalog): verify every Apify per-result price on the platform key

Each row's test_request ran on 2026-09-26 and Apify's chargedEventCounts
matched its rate card. The TikTok ad library actor returns rows again, so it
is verified with a captured example.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat(catalog): route Google Maps and Weibo post detail through Apify

Adapters let treg.google.serp.maps and treg.weibo.post.detail choose the
Apify actors, with the run's spend cap, timeout and memory fixed so the
platform guard accepts the child call.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(catalog): drop Lazada's compute fee and keep account details out of notes

The actor's 2026-09-25 pricing no longer bills run compute to the caller (a
79-second run showed no platform usage), so its call_fee over-charged every
call. Notes describe observations by price tier, not by the account that
made them, and carry no run ids.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* docs(catalog): state tiered Apify observations without the account's tier

Notes for plan-tiered actors record the events billed and that they matched
the rate card for the key's tier, not the dollar figure that would name it;
the repeated Weibo observation and the owner-meter notes go.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(settle): bill LinkedIn's actor-start once per query it runs

The LinkedIn jobs actor bills its actor-start event for every job title x
location searched, so a flat call_fee under-billed any multi-query call.
cost.call_fee_per names the body arrays whose lengths multiply the fee. The
apify.yaml header no longer names a plan or calls the TikTok actor broken.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* docs(money): describe call_fee_per

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(catalog): accept only declared fields on LinkedIn job search

The actor bills an actor-start for every geo id it searches, and geoIds was
undeclared, so it passed through unbilled. The row now takes its declared
filters only (salary, easyApply, under10Applicants and industryIds added);
places go in locations, which call_fee_per counts.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* docs(catalog): leave headroom above the Apify spend cap

A run whose charges land exactly on maxTotalChargeUsd can be aborted by
Apify and answer 400 with no rows, which releases the hold. Lazada's
10-product floor costs exactly its 0.05 minimum cap, so its example now uses
0.06 and the notes say to set the cap above the expected spend.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Herus13 <bootforge.ai@gmail.com>
2026-09-26 15:04:26 +10:00
UncleCode 3ee1948c64 Merge origin/main into dev/hub: the tool hub, behind TREG_HUB_ENABLED
Brings main's 86 commits (dashboard boot and loading, legacy dashboard removal, test pruning,
overflow and routing fixes) together with the tool hub branch.

Conflicts, both sides kept unless noted:
- the legacy dashboard stays deleted, as on main;
- App.vue and the dashboard state: main's search page and loading states plus the hub pages;
- ci.yml: main's Postgres job, with the hub tests added to its list;
- dev-local.sh: main's server environment plus the hub flag passthrough;
- test_call_application_contract.py, test_marketplace_call.py: main's pruned files plus the
  hub branch's sync `settle: usage` test.

Not conflicts: main and the hub branch fixed the same CompanyEnrich empty-page billing; main's
rule runs first, so the hub branch's copy and its test are dropped. The Listing-tab test reads
the Vue source instead of the deleted legacy page.
2026-09-26 07:32:53 +08:00
SToneX b279e6fb51 chore(web): remove the legacy dashboard and its rollout switch
Every visitor has been on the compiled Dashboard since the 100% rollout, so
the frozen legacy snapshot, its asset route, the per-account selection and
the three TREG_DASHBOARD_ROLLOUT_* settings go. Catalog pages and shared
links no longer look up the session to pick a frontend, /search is always
served, and the app-version stamp is the bundle hash alone. Rollback is now
a deploy of the previous build.
2026-09-25 21:26:40 +08:00
SToneX aa6c150591 ci(tests): run the Postgres job in parallel, one database per worker
Under TREG_TEST_DB_URL each xdist worker now creates and uses its own
database, so the Postgres job no longer has to run serially. Without it,
every sqlite test process gets its own pid-named file, removed at exit,
so two runs in one checkout no longer wipe each other's tables.

Also move test_orgs_isolation.py and test_orgs_mgmt.py into the Postgres
test list: they had landed inside the failure-diagnostics psql script and
never ran on Postgres.
2026-09-25 21:17:25 +08:00
UncleCode 986d5ffa57 merge: main into dev/hub, hub migrations renumbered 0044-0049
Brings main's 24 commits since the 2026-09-21 merge: the new Vue dashboard in frontend/ with
the account-based rollout (legacy frozen as src/treg/web/dashboard-legacy/), Olostep and
Keenable, routed web search, TrestleIQ, call_media and resources_list on MCP, provider
resources, the search log.

Conflicts, every one "both sides added": imports in call/service.py and routers/web.py; the
error-owner table in call/types.py; dev-local.sh keeps the TREG_HUB_ENABLED passthrough in
front of main's new SERVER_ENV; the legacy dashboard keeps both the Resources and the Hub
nav entries and view names; test_mcp.py lists main's two new tools and the hub's three;
test_marketplace_call.py keeps both new test blocks; .gitignore keeps both. MAP.md and
docs/context/README.md regenerated. tests/test_dashboard_markup.py removed, as on main.

The hub's six migrations move from 0041-0046 to 0044-0049 above main's 0043; a fresh database
upgrades to one head, 0049.
2026-09-24 12:38:40 +08:00
shehjad-dev 854bed643a build(package): exclude demo videos from PyPI artifacts 2026-09-23 13:02:23 +06:00
SToneX 55727e790d build(web): replace vendored libraries with CDN and npm dependencies 2026-09-23 12:29:53 +08:00
SToneX 675c7f2722 feat(dashboard): add account-based frontend rollout and rollback 2026-09-22 18:51:51 +08:00
SToneX 698898cf14 refactor(dashboard): extract Vue app and integrate production builds 2026-09-22 17:05:52 +08:00
yewenhai bbd3b91fa1 feat(db): add optional read replica datasource 2026-09-21 10:55:32 +08:00
UncleCode acecf1bc19 Merge main into dev/hub (150 commits) — migrations renumbered, two breakages fixed
main had moved 150 commits ahead. Nine files conflicted; every one keeps BOTH sides, except the
catalog search hint, which keeps the hub branch and calls main's new `_run_hint` helper for a
catalog row. The app serves the hub routes and main's new media routes; the worker keeps both its
hub and arena subcommands.

Migration numbers collided a second time: main took 0033-0038, so the six hub revisions move to
0039-0044 and chain onto main's 0038. Single head 0044; the chain applies clean on a fresh sqlite
with both hub switch columns present. The numbers named in architecture/hub.md follow.

Two real breakages the merge caused, both fixed here:

1. Money. main made the signup credit verified-only (`claim_signup_promo`), and `POST /users` is
   legacy registration, which leaves the user UNVERIFIED. Every hub test buyer therefore started at
   zero and each paid run answered 402. New shared helper `conftest.funded_user()` grants the team
   1_000_000 micro; used at the 19 call sites that have to pay. The tests that deliberately exercise
   the out-of-money path are untouched.

2. Identity. main added `ApiKey | None` to `Caller` for managed keys. The hub's scheduled check
   builds a Caller directly and crashed with a TypeError. It now passes `api_key=None`: a scheduled
   check is not an API-key call, it runs as the maker's membership.

Green on the merged tree: full suite 4070 passed, 8 skipped; agent pages + seo 228; import-linter
14/14; plugin check OK; alembic parity 4. `hub_enabled` stays False by default, and with the flag
absent every hub route answers 404 with a valid request, the agent files carry no hub text, and
catalog search returns no hub row.
2026-09-16 12:20:57 +08:00
shehjad-dev 5c986612ed fix(auth): integrate main and protect managed-key CLI state 2026-09-14 11:53:06 +06:00
SToneX 8112c47f00 fix(teams): cap accounts at ten owned teams (#458) 2026-09-11 14:58:43 +08:00
UncleCode c49d648f5d Merge branch 'main' into dev/hub (8.4)
Bring the team's work since 2026-09-09 into the hub branch: the review/feedback system, the arena
insights, the R2 body storage and cache admission for the archive, the shared key-value store, and
the SSRF CGNAT fix. 108 commits, 233 files.

The one real conflict was the migration numbers: both branches used 0026-0030 for different
changes. main's chain is 0026 (callreview) through 0032 (archive body storage); the five hub
migrations are renumbered to 0033-0037 and rechained onto 0032, so there is a single head 0037.
Everything else was additive and kept from both sides: the HubTool/HubRun models beside
CallReview/FeedbackHandling, the `hub` domain and the key-value store both in the AGENTS.md tables,
the hub CLI family beside `treg review`, the hub MCP verbs beside `review` in the tool sets, the
quickjs and obstore/redis dependencies together. mcp_feedback.py was removed on main; the deletion
is kept. The docs fragment's migration numbers were updated.

Full suite 3508 passed; agent pages + SEO 216; boundaries 14/14; the migration chain a single head
0037 on Postgres 16; the plugin staleness guard green.
2026-09-10 20:19:31 +08:00
UncleCode a707ad85ea fix(hub): the scheduled-check worker test gives itself a secret key; run the hub files on Postgres in CI
8.3, the full suite on Postgres 16: the migration chain runs up to head, down to base and up
again, and the CI Postgres file list (plus the three hub files) is green serially, 954 tests.

Two things this found:
- `treg-worker hub check` runs the server's startup guard `verify_db()`, which refuses a
  non-SQLite database with no TREG_SECRET_KEY. A real worker always has the key; the test now
  sets one before calling `_hub_check`. On SQLite (each test file its own database) the guard saw
  SQLite and passed, which hid it; on one shared Postgres database in the serial run it did not.
- the same test scoped its assertion to its own tool with `--only`, because the check walks every
  live tool in the database and a shared database keeps the tools earlier files published.

CI: the Postgres job now also runs tests/test_hub.py, tests/test_hub_sandbox.py and
tests/test_mcp.py, so the hub is covered on Postgres on every push.
2026-09-10 19:31:30 +08:00
SToneX 9f67130169 feat(archive): add R2 body storage and gated double writes 2026-09-10 19:05:57 +08:00
SToneX ea829e3355 fix(cache): admit only confirmed useful results and preserve history 2026-09-10 18:29:27 +08:00
Jason Zhou 28b5470988 fix(arena): integrate main and verify deployment readiness 2026-09-10 14:20:58 +10:00
shehjad-dev f08baf1a9d feat(auth): add managed API key controls 2026-09-07 23:31:44 +06:00
SToneX c87e4135dc fix(ci): scan the complete checked-out history for secrets 2026-09-07 20:33:18 +08:00
SToneX 0baf2735e2 feat(feedback): add private agent feedback intake 2026-09-07 16:14:04 +08:00
SToneXandCursor Agent 6a68eb0ee8 test: document xdist local default and drop tautological pins (#346)
Daily local runs now match CI (`pytest -n auto` via --with). Two catalog
refresh=True parses and the per-parametrize AST walk in the pool-isolation
guard were paying ~25-30s for no extra coverage. High-confidence tautologies
that only pinned tables living inside the test file are gone; gateway blame
now reads production `_BLAME_BY_KIND`. Visual CSS substring pins on the
dashboard were dropped; trapped-dialog, confirm-slug, masked referrer and
empty platPrice stay.

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
2026-09-05 22:34:28 +08:00
SToneX 8031c83d99 docs(agents): contract-first AGENTS.md and a pinned uv floor instead of a lock ban (#335)
* docs(agents): AGENTS.md is the single guide, contract first, no state snapshots

CLAUDE.md now only imports AGENTS.md so Claude Code, Codex and Cursor read one file.

AGENTS.md is rewritten around what an agent cannot learn from the code:

- Non-negotiables move to the top and are corrected against the code. "Own key
  always wins, never metered, never routed or overflowed" is promoted to rule 1.
  The relay rule now scopes to plain /call/ and names routed endpoints and
  overflow as wrappers, since route.py injects `_treg` into the body and overflow
  adds X-Treg-Served-Via; the old wording contradicted both. The hold rule
  defines "hold"; the pool rule says why reserve and settle are two transactions;
  the ledger rule records that there is deliberately no refund entry.
- The dataplane write allowlist becomes guidance that points at
  tests/test_call_architecture.py as the authority instead of a prose copy.
- Enforcement gap inventories, per-file commit lists, contract-by-contract
  recaps and the CLI-module list are removed: they duplicate pyproject, the
  import-boundaries fragment and the tests, and rot without a drift check.
- Endpoint and provider counts are removed (three files carried three values).
- Duplicated statements (own-key, faithful relay, keep-four-in-step, money-only
  path, relay guard, secrets) are stated once each.
- Sections reordered: contract, where the truth lives, architecture,
  development, working agreement, and a closing section for user-facing copy.

* build(uv): pin required-version instead of banning `uv lock`

uv.lock is revision 3, first written by uv 0.8.4. An older uv reads it fine but
rewrites it to revision 2 on any touch, dropping every upload-time field: the
~650-line no-op diff the old "always --frozen, never uv sync or uv lock" rule
worked around. That rule also told agents to hand-edit the lock, which --frozen
would then install unchecked.

- pyproject.toml: `[tool.uv] required-version = ">=0.12"`, so an old uv refuses
  to run instead of rewriting the lock.
- ci.yml: `--frozen` becomes `--locked`, so a stale lock fails CI instead of
  being installed silently. The comment about the team's older uv is gone.
- CONTRIBUTING.md names the floor; AGENTS.md replaces the ban with "change
  dependencies through uv add or uv lock, never by hand".
- docs/context/architecture/import-boundaries.md describes the CI step as it
  now runs.

Verified: uv lock --check, uv sync --locked and uv run --locked lint-imports
(12 kept, 0 broken) on uv 0.12.3.
2026-09-05 14:52:29 +08:00
SToneX ccd44827ff feat(money): settle async metered calls at terminal state
Add the expand-only async task record and data-derived settlement basis, defer tier-4 holds until provider terminal state, and complete them through a multi-instance-safe worker. Archive terminal JSON, expose read-only reconciliation, add AIGC platform key slots and Render scheduling, and pin SQLite/Postgres behavior with E2E coverage.

The table migration ships with behavior because older code ignores it while the new request path cannot retain a hold safely without the durable record.

Live Postgres verification: Replicate FLUX Schnell settled a no-key team success at 3000 micro-USD. A real Seedance task accepted with an unreachable input image reached terminal failure and released its 72000-micro hold in full. Both terminal JSON responses were archived; catalog-priced provider spend was USD 0.003 total.
2026-09-01 19:50:40 +08:00
SToneX 093b256775 fix: logger names are contracts, labeler globs follow the moved files
The PR4 sweep renamed three loggers to module paths - treg.ledger, treg.proxy, and
treg.billing became treg.domain.money, treg.infra.upstream.relay, and
treg.application.billing - an undeclared behavior change on exactly the settle/release,
cancel-release, and cap-check failure paths where external log filters and alerts hook in.
Reverted to the short functional names the rest of the pipeline uses (treg.idempotency,
treg.capacity, ...): a logger name is an observability contract and does not follow code
location. Also: the area:proxy labeler glob pointed at deleted proxy.py (the label could
never fire again) and area:auth-secrets listed the two deleted shims for injection and
session signing; the stripe dependency comment and a money.md passage named deleted files.
2026-08-31 10:15:59 +08:00
SToneX dd214b82f8 refactor(db): delete the legacy migration path - startup now verifies schema without writes 2026-08-30 18:03:28 +08:00
UncleCode 4ac7a0fe9b ci(postgres): dump pg_stat_activity + ungranted locks when the serial job fails
Three CI-only hangs at archive drains were undiagnosable from Python stacks alone. On the next
failure this prints the database's view: session states (idle-in-transaction = the leak), wait
events, and ungranted locks joined to their holders' queries. Temporary; delete once understood.
2026-08-29 08:21:52 +08:00
SToneX 2dc86168b9 fix: make overflow spend updates atomic 2026-08-28 21:58:48 +08:00
UncleCode b8b3fad6c8 merge main into dev/archive — the archive survives the call-kernel refactor
Main gained 107 commits since this branch was cut, including the phase-4 call-kernel refactor
that moved the entire call path out of api.py into application/call/ and the catalog store into
domain/catalog/. The archive's two hooks were re-transplanted rather than force-merged:

- The serve hook and the recorder now live in application/call/service.py at the kernel's
  relay/buffer stretch, adapted to its types (UpstreamResponse with status/raw_headers instead
  of a fastapi Response; a _served_response builder puts a stored answer in the relay's own
  clothes with content-length and the cache headers). The cached telemetry tag rides the
  kernel's audit build; the /calls serializer keeps the cached column.
- The catalog cache field moved with the store into domain/catalog/store.py; catalog_store.py is
  main's forwarding shim, untouched.
- config/bootstrap/admin/ci conflicts were both-sides additions — both kept.
- tests/test_archive.py patches the kernel's relay reference (call_service.relay), following
  main's own test convention; test_marketplace_call's binding tests came from main already
  pointing at oauth_providers.platform_bindings.

Proof on the merged tree: full suite 2079 passed (2 skipped; test_agent_pages excluded — known
TREG_PUBLIC_URL local condition), import boundaries 10/10 kept, and a LIVE end-to-end run:
real CoinGecko call, then a real cached hit through the new kernel (X-Treg-Cache: hit, age 2s,
billed 290 micro like live), panel serving, report counting the hit.
2026-08-28 16:24:42 +08:00
SToneX f9761acae3 test(postgres): truncate data between test cases 2026-08-27 14:44:48 +08:00
SToneX faf84f175a ci: dump stacks when the postgres job stalls 2026-08-27 14:04:35 +08:00
UncleCode 2d26a6786f feat(archive): tables + skeleton — PR 1 of the cache/archive
The archive keeps every metered platform answer, versioned with its timestamp; the cache is its
newest fresh layer, served (in a later PR) instead of a vendor call. This first slice is shape
only — nothing writes or reads it yet, and production behavior is unchanged.

- src/treg/archive.py: the mode gate (TREG_ARCHIVE_MODE off|shadow|serve, unknown degrades to
  off — a typo must disable, never enable), the eligibility policy (actions never; per-entry
  catalog cache field forbidden|transient|archive, absent ⇒ forbidden), and the canonical cache
  key (sorted query, canonical-JSON body hash, only Accept/Accept-Language from headers —
  credentials cannot enter: injection happens after the key is taken).
- models.py: ArchiveKey (unique key_hash, AIMD timer state, change statistics, learned volatile
  paths, heat) and ArchiveSnapshot (verbatim bytes, content-hash dedup via body_of, unique
  (key_id, version)). Platform-scoped by design — own-key traffic never enters, so the privacy
  line is drawn at write time. Bodies in Postgres per the IdempotentCall precedent.
- alembic 0002: both tables; parity with the models is enforced by test_alembic_baseline.
- config.py: archive_mode setting, default off.
- tests/test_archive.py (21 tests): mode degradation, policy refusal-by-default, key
  canonicalization, round-trip + dedup + uniqueness on the running engine; added to CI's serial
  Postgres job.
- docs/context/architecture/archive.md: the fragment, status building, with the explicit
  do-not-document-what-is-not-built note for the later slices.

Money is out of scope by decision: no archive code imports ledger/billing, ever. Full suite:
2007 passed; the 63 test_agent_pages failures are the known TREG_PUBLIC_URL local condition and
fail identically on main.
2026-08-27 11:52:44 +08:00
SToneX 9e695e7118 test(ci): make cancellation stalls diagnosable 2026-08-27 10:21:56 +08:00
SToneX 3dff4e71bd fix(call): release cancelled calls before re-raising 2026-08-27 10:20:02 +08:00
SToneX 4a714529ff fix(api): use naive UTC for database timestamps 2026-08-26 08:02:29 +08:00
SToneX 84791be713 Merge branch 'test/phase0-surface-and-pg' into refactor/phase1-assembly
# Conflicts:
#	.agents/skills/tools-registry-context/MAP.md
#	docs/context/README.md
#	docs/context/architecture/data-model.md
2026-08-25 20:09:48 +08:00
SToneX 3e639a8cfe Merge branch 'test/call-matrix' into test/phase0-surface-and-pg
Regenerate routes.json for main's additions: /tools/{service}, /pricing,
/sitetrack.js, and the flattened /use-cases/{job} URLs. openapi.json is
unchanged because page routes never enter the schema. ci.yml keeps both
sides' jobs: test-postgres (ours) and the restored gitleaks (main's).
2026-08-25 20:03:14 +08:00
SToneX 5d9c20e030 chore(architecture): enforce import boundaries
Correct the relocated MCP lifespan condition in the surface snapshot without changing runtime behavior.
2026-08-25 14:36:20 +08:00
SToneX 0c219bfc31 feat(db): add Alembic schema baseline 2026-08-25 12:21:16 +08:00
SToneX 85594b8343 ci: test critical paths on postgres 2026-08-25 10:55:07 +08:00
UncleCode c59b402733 ci: restore the gitleaks job that branch protection requires
main requires a status check named `gitleaks`. The CI rewrite in 528383d dropped the job while
leaving the requirement in place, so since then every pull request has waited on a check that
could never report, and only an admin bypass moved anything to main. .gitleaks.toml stayed in the
repo the whole time, still saying 'we run gitleaks in CI (see .github/workflows/ci.yml)'.

Restored exactly as it was — pinned gitleaks 8.21.2, fetch-depth 0, the repo's own allowlist
config — plus a 10-minute timeout, matching the reason `test` has one. It never skips: the file's
own docs-only note already records that a skipped required check blocks a merge forever.

Verified before committing: the same command scans 585 commits and 17.36 MB on today's main and
on the open SEO branch, and reports no leaks on both.
2026-08-24 13:27:50 +08:00
Jason ZhouandClaude Opus 5 d1c789d073 fix(catalog): honor retired markers on /call/, and mark TikHub's 50 dead routes
hcl-software burned 47 calls in one day on tikhub.x.linkedin-web-search-people,
a route TikHub removed when it collapsed linkedin/web into an eight-route
linkedin/web_v2. The 2026-08-17 sweep already knew 50 catalogued paths were gone
and left them served.

PR #105's retired/broken markers turned out never to have reached main — it was
merged into fix/refusal-audit, which is itself unmerged — so the marker fields,
the discovery filter and the direct-lookup story are (re)introduced here, plus
the piece #105 never had: /call/ now refuses a marked id with its migration
story instead of relaying into a dead upstream. Marking alone would not have
helped, because an agent calling a cached id never reads a browse surface.

- 410 + refused_by=retired, so a retirement is its own class in the daily report
  rather than falling into "request" via the dict default.
- platform_eligible refuses a marked row: a retained price explains a cached id,
  it is not an offer treg may spend its own key against.
- 50 TikHub routes marked (9 with a real successor, 41 with none), the LinkedIn
  people-search capability rescued onto JustOneAPI, and the live-but-uncatalogued
  linkedin/web_v2/get_post_comments added.
- scripts/catalog_drift.py compares catalogued path+method against providers'
  published OpenAPI documents — no credential, no provider call. The root cause
  was that nothing re-checked a spec for drift; the first signal was a user
  hitting a 404 in the wild.

Verified against the LIVE TikHub spec (Codex ran without network and could only
use the 2026-08-17 snapshot): 0 dead-path, 0 method-rot, 50 acknowledged,
0 restored. Suite 1767 passed; each new defence confirmed to fail when reverted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-19 12:47:55 +10:00
UncleCode 528383d127 ci: parallel tests, superseded-run cancellation, a 15-minute cap, docs-only skip
The diagnosis, from run logs: the suite itself is healthy (1,757 tests, none slower than 4.5s,
average 42ms) — the pain was SERIAL execution amplified by GitHub's shared-runner lottery. One
throttled run showed 68 seconds before the first test started and 17 stalls spread evenly through
the run; nothing in our code was the bottleneck.

Four measures, each commented in the workflow where it lives:

  * pytest -n auto — the suite parallelizes cleanly once each xdist worker gets its OWN sqlite
    file (conftest: the first parallel attempt scored 1,022 errors from twelve workers dropping
    each other's tables in one shared treg-test.db, exactly as the old comment warned). xdist
    arrives via `uv run --with`, deliberately not the lockfile: it is a CI-only concern and the
    team's older uv rewrites uv.lock's entire format on any touch.
  * concurrency + cancel-in-progress — a new push kills the outdated run (two were overlapped the
    day this landed).
  * timeout-minutes: 15 — a lottery-loser run dies loudly; a re-run lands on a healthier machine.
  * docs-only PUSHES skip the suite; pull requests ALWAYS run, because a skipped required check
    would block merging forever.

Measured locally: serial 140s, 4 workers 74s, 12 workers 63s (a ~50s floor: per-worker startup
plus the longest chain). Projection for GitHub's 4-core runners at their measured 2.4x-per-core
slowness: about 3 minutes end to end on a healthy runner, 9-10 capped at 15 on a throttled one,
~20 seconds for a docs push. This push is the first live measurement.
2026-08-18 13:48:57 +08:00
Jason ZhouandClaude Fable 5 8ca54c0c6f feat: treg.to is the canonical domain — treg.superdesign.dev becomes the legacy alias
public_url/email_from defaults, render.yaml, CLI fallbacks, packaging (npm/plugin/pyproject),
web pages (+canonical tag), docs and context fragments all move to https://treg.to.

The legacy host keeps serving the FULL API forever — installed CLIs, skill.md files and
.mcp.json configs in the wild hold Bearer tokens pointed at it, and HTTP clients strip
Authorization on cross-host redirects. Only browser-facing marketing pages 301 to treg.to
(new middleware + tests).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-13 11:56:20 +10:00
Jason ZhouandClaude Fable 5 4eccbfdd37 chore: repo renamed to superdesigndev/treg — update all self-references
GitHub redirects the old slug, but every URL we publish (pyproject,
npm package, plugin manifest, issue templates, web pages, llms.txt)
now says the real name. PyPI package name stays tools-registry.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-11 20:54:30 +10:00
UncleCode f02473c343 Merge pull request #27 from superdesigndev/dependabot/github_actions/actions/stale-10
chore(deps): bump actions/stale from 9 to 10
2026-07-27 15:10:47 +08:00
UncleCode 1cd244af4e Merge pull request #25 from superdesigndev/dependabot/github_actions/astral-sh/setup-uv-7
chore(deps): bump astral-sh/setup-uv from 5 to 7
2026-07-27 15:10:42 +08:00
dependabot[bot] 86f26af0c9 chore(deps): bump actions/stale from 9 to 10
Bumps [actions/stale](https://github.com/actions/stale) from 9 to 10.
- [Release notes](https://github.com/actions/stale/releases)
- [Changelog](https://github.com/actions/stale/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/stale/compare/v9...v10)

---
updated-dependencies:
- dependency-name: actions/stale
  dependency-version: '10'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-07-22 22:54:01 +00:00
dependabot[bot] 57937eb676 chore(deps): bump astral-sh/setup-uv from 5 to 7
Bumps [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) from 5 to 7.
- [Release notes](https://github.com/astral-sh/setup-uv/releases)
- [Commits](https://github.com/astral-sh/setup-uv/compare/v5...v7)

---
updated-dependencies:
- dependency-name: astral-sh/setup-uv
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-07-22 22:53:56 +00:00