treg/docs/context/ops/deploy.md

title, status, sources, related

title	status	sources	related
Running & deploying the server	shipped	pyproject.toml hatch_build.py scripts/build-dashboard.sh scripts/build-web.sh scripts/frontend-e2e-server.sh src/treg/__main__.py src/treg/maintenance.py src/treg/alembic/env.py src/treg/alembic/versions/0034_managed_api_keys.py src/treg/alembic/versions/0035_default_key_generation.py src/treg/alembic/versions/0036_activity_key_indexes.py src/treg/worker.py src/treg/web/selfhost.sh src/treg/config.py src/treg/infra/db.py src/treg/email.py src/treg/audit.py scripts/dev-local.sh deploy/render.example.yaml	ops/capacity.md architecture/data-model.md architecture/ads-conversions.md foundation/charter.md

Running & deploying

This fragment documents behavior every operator needs to run or self-host a registry. It does not describe the private topology, live settings, account funding, incidents or rollout state of the hosted treg.to service. Superdesign operators use the private treg.to deployment runbook.

Fish Audio shared-key capacity monitoring requires both TREG_PLATFORM_KEY_FISHAUDIO and TREG_PLATFORM_FISHAUDIO_WORKSPACE_ID. The latter is the Fish workspace selector used only by the free API-credit probe; without it, capacity remains unknown rather than reading the unrelated personal wallet.

Entry point (__main__.py)

python -m treg upgrade runs the explicit release phase. maintenance._upgrade_schema() runs alembic upgrade head for an empty or stamped database. A non-empty unstamped database is refused without writes: adoption ended with 0.14.x, so the operator must install tools-registry[server]==0.14.*, run python -m treg upgrade there once, then continue onward. A database stamped at a revision this build does not know is also refused because explicit upgrade may not cross the rollback floor. The ordered, idempotent release-task registry runs only after schema success and never provisions a local user.

The default python -m treg serve path runs the same upgrade phase, calls api._bootstrap_single_user(), disposes the async engine inside the pre-serve event loop, and then calls uvicorn.run("treg.api:app", host="0.0.0.0", port=int($PORT or 18790)). --reload is optional. It honors $PORT for platforms that route and health-check an injected port. python -m treg keygen prints a Fernet key for TREG_SECRET_KEY without importing the server maintenance stack. treg.api:app is bootstrap.create_app(role="all").

The FastAPI lifespan calls read-only verify_db(). It refuses an unstamped database or a known revision behind head and directs raw-ASGI operators to python -m treg upgrade. An unknown newer revision means the code is older than the schema: startup warns and serves because additive revisions tolerate rollback. A revision marked contract = True is the documented hard rollback floor. Every role startup manifest is schema-write-free.

Both pre-Uvicorn entry paths dispose the engine before their asyncio.run() loop closes. The next event loop therefore creates fresh pooled connections instead of receiving connections bound to a closed maintenance loop. Calling maintenance.upgrade() directly does not dispose the engine.

Schema upgrade safety

• Managed-key rollback floor: revision 0034 adds key controls, audit rows, Activity snapshots, and a hash-only backfill for existing membership credentials. It is marked contract = True because old code cannot enforce newly stored disable or revoke state. The migration is additive and uses SQL that works on SQLite and Postgres.
• Alembic is authoritative: migration scripts ship inside src/treg/alembic/ in the wheel. maintenance._alembic_config() resolves that installed package resource, supplies the escaped configured URL, and runs Alembic in a worker thread.
• The adoption floor is final. An unstamped existing database must pass through release 0.14.x. Current code does not inspect, repair or stamp it.
• Startup is read-only. verify_db() checks the stamped revision through packaged Alembic metadata. It never creates tables, stamps versions or runs release tasks. Worker commands use the same check.
• Schema changes are revision-only. An autogenerate drift guard requires Alembic head and SQLModel.metadata to match exactly. reset_db() uses create_all only to build a missing test schema and stamps it directly at head.
• A missing encryption key fails loudly on a real database. If TREG_SECRET_KEY is empty and database_url is not SQLite, verify_db() raises. On SQLite development it logs a warning.

For PostgreSQL, migrations set bounded lock and statement timeouts. A contended ordinary DDL migration must fail cleanly rather than queue production traffic behind an exclusive lock: while an ALTER TABLE waits for its ACCESS EXCLUSIVE lock, every later query on that table queues behind it, so the 5 s lock_timeout is the longest stall a deploy may inflict and no revision may raise it for an ALTER. A deploy waits longer by retrying instead: maintenance re-runs alembic upgrade head up to LOCK_RETRY_ATTEMPTS times after a lock timeout, pausing a jittered few seconds between attempts so the table's short transactions can drain, and env.py commits each revision on its own (transaction_per_migration) so a retry resumes at the revision that timed out. Any other error fails the deploy at once. The one sanctioned exception to the 5 s cap is CREATE INDEX CONCURRENTLY in its own revision: its lock blocks nobody while it waits, so such a revision owns its longer timeouts and must detect and rebuild an invalid index left by interruption.

Hot-table ALTERs stay cheap to retry when they sit in their own revision, add only nullable columns without defaults (metadata-only in PostgreSQL) and leave backfills and NOT NULL to later steps.

Deploy schema changes before starting application or worker code that expects the new revision. A platform whose scheduled workers update independently must sequence them accordingly.

Database pools

POOL_SPECS in infra/db.py defines three PostgreSQL pools:

pool	maker	serves
api	session_maker	request handlers
admin	admin_session_maker	/admin/* handlers
background	background_session_maker	audit, archive and other background work

Each class can exhaust only its own slots. Minor pools deliberately have no overflow because an unbounded escape hatch defeats the bulkhead. A new class of work gets an appropriate pool, not only a module-local semaphore. tests/test_db_pool_isolation.py pins the routing.

Every process opens its own pools and runs its own in-process background tasks. Calculate the full connection peak from pool capacities, process count and the maximum number of overlapping instances during a rolling deployment. Keep that peak below the database connection ceiling with headroom.

TREG_DB_POOL_OVERRIDES accepts entries such as "admin.pool_size=4,background.pool_size=12" and patches POOL_SPECS at startup. Unknown pools, unknown field names, non-integers and out-of-range values are logged and skipped. In particular, SQLAlchemy treats some zero and negative values as unlimited, so the parser must not pass them through. Live values and production sizing belong in the private operator runbook.

bootstrap.pool_gauge samples infra/db.pool_snapshot() and emits per-pool peak, capacity and headroom. Resize from measurements, one pool at a time. A larger pool does not repair a slow scan, lock queue, disk-bound database or missing index.

All pools use pool_pre_ping=True, pool_recycle=300 and pool_timeout=5. A request that gets no slot in time receives 503 {"treg_saturated": true} with Retry-After: 2 through bootstrap_handlers._pool_saturated. A /call/ holds no connection during the upstream round trip.

SQLite aliases all three makers to one engine. It has no pool to protect and file-level write locks it cannot share, so separate engines would only manufacture lock failures. Tests pin routing rather than SQLite isolation.

Optional read replica

TREG_READ_DATABASE_URL adds a datasource exposed as read_session_maker. Its supported drivers match the project's SQLite / PostgreSQL support: sqlite+aiosqlite and postgresql+asyncpg. Bare postgres:// and postgresql:// URLs normalize to postgresql+asyncpg, as for the primary; other read URL drivers fail settings validation. This does not add support for MySQL or other SQLAlchemy dialects. When empty, the read maker aliases session_maker, with the primary's existing transaction behavior (including writes) and no additional pool. Connections are opened only when used.

A configured URL always gets an independent engine, even when it names the primary database:

• PostgreSQL uses a read pool defaulting to two connections with no overflow; TREG_DB_POOL_OVERRIDES accepts read.pool_size and read.max_overflow. Each connection sets default_transaction_read_only=on through asyncpg's server_settings.
• SQLite retains the driver's default pool behavior and sets PRAGMA query_only=ON on every new read connection, including after reconnection. The primary's connections remain writable. For an existing file, sqlite+aiosqlite:///file:replica.db?mode=ro&uri=true additionally opens the file read-only and fails if it is missing. A plain SQLite URL retains SQLite's usual file-opening behavior, including creating a missing file; query_only guards SQL changes, not file creation. A separate in-memory SQLite URL starts empty and cannot serve as a copy of the primary.

These connection settings guard accidental writes; they are not an authorization boundary and can be disabled by deliberate SQL. Use a physical replica/read-only database role or filesystem access controls as appropriate. Connection/query failures propagate without primary fallback.

pool_snapshot() includes a separate read entry for a configured PostgreSQL datasource; SQLite engines are omitted as for the primary. connection_budget() describes only the primary pools; budget the read pool against its target database, including process count and deployment overlap. If both URLs target the same server, add both budgets against that server's limit. dispose_engine() also disposes the read engine. Schema upgrades, startup verification and test schema resets continue to target the primary.

This datasource is opt-in infrastructure: no application query or worker currently uses it. Adopting callers must tolerate replica lag and keep writes, cursor advancement and concurrency control on the primary. Operators must provision and synchronize a compatible schema and data; setting a URL does not establish replication, translate dialect-specific queries, or migrate any Cron job's workload. Configure it in a private local .env or the hosting service's environment; .env.example contains no secrets.

Configuration (config.py)

Settings uses the TREG_ prefix, reads .env, and is cached by get_settings().

• database_url defaults to local SQLite. A validator rewrites bare postgres:// and postgresql:// URLs to postgresql+asyncpg://.
• secret_key is the Fernet key. An empty value creates an ephemeral key suitable only for local development.
• public_url builds OAuth callbacks and public links. Self-hosters must set it to their canonical external URL.
• admin_token is the cross-tenant super-admin bearer. Empty disables this environment path. Keep it long and secret.
• OAuth provider clients use <provider>_client_id and <provider>_client_secret settings. Empty credentials make that provider unconfigured instead of failing halfway through consent.
• oauth_review_pending is a comma-separated set of provider-registry review keys. Its hosted value is operational state and is maintained privately.
• promo_grant_micro controls the once-per-verified-user signup grant. Zero pauses new automatic grants without changing existing balances.
• blocked_email_domains is the complete comma-separated blocklist. Empty blocks no domains. It is a sign-in and team-creation speed bump, not a substitute for suspending an existing abusive user.
• run_proof gates release of a shared key to the root-installed local runner. Empty refuses shared local runs while leaving owner-key runs available.
• run_allowed_bins is the command allowlist for server-side CLI execution.
• run_rlimits, run_cpu_seconds and run_fsize_mb bound server-run resource use where POSIX resource limits are available.
• proxy_ssrf_check enables the call-time SSRF guard and is on by default.
• claude_connector_enabled gates the catalog-only connector at /mcp/v2/.
• connect_demo_enabled gates the developer OAuth test page. Leave it off on public deployments.
• intercom_app_id and intercom_secret enable optional support chat. Empty disables it.
• resend_api_key and email_from enable transactional email. The sender must use a domain verified with the operator's mail provider.

Registry-owned OAuth applications, advertising conversion credentials and platform provider keys are optional deployment capabilities. Their names and binding behavior are documented with their owning subsystems. Credentials must stay in the deployment secret store. TREG_PLATFORM_PROVIDERS is the shared-serving allow-list. Most providers also require a configured platform key. A live-verified free endpoint declared platform_auth: anonymous needs only the allow-list because treg injects no provider credential.

Safe local mode

single_user and single_user_token_file support curl {BASE}/selfhost.sh | sh. The default serve pre-phase idempotently creates the local identity and personal team and writes a stable token with mode 0600. The token is re-minted only if its file is deleted.

This behavior is gated by single_user_ok, which requires both a local SQLite database and a loopback public_url. A stray TREG_SINGLE_USER=true on a public PostgreSQL deployment therefore does nothing.

email_dev_mode is similarly guarded. A login code is exposed only when the flag is true and the database is local SQLite. Hosted deployments must still leave it false.

Web service and generic Render example

The redesigned homepage ships to all homepage visitors; its rollback, like the Dashboard's, is a code revert or a deploy of the previous build.

GET /app, the catalog pages and shared links all serve the Vite-built Vue application. There is no frontend switch to configure; a Dashboard rollback is a deploy of the previous build. See frontend/README.md. The frontend is authored in frontend/ within the same repository. GET / retains the existing landing behavior. Dashboard assets, tutorials, agent files and installer assets ship with the wheel. Hosted-page MP4 demos remain in Git checkout deployments but are excluded from published wheels and source archives; a server installed from PyPI serves the product surfaces without those optional marketing videos.

Run bash scripts/build-dashboard.sh before building a distributable Python package. Hatch's build hook rejects a wheel or sdist without the dashboard entry and includes the generated assets; editable installs remain Python-only. Node and npm are build tools, not runtime services. TREG_FRONTEND_DEV=true serves the authored entry with Vite scripts on local port 5173 and is accepted only with SQLite and a loopback public URL. scripts/dev-local.sh up manages both processes. For browser previews from another device, build the dashboard and start or restart the local stack with TREG_FRONTEND_DEV=false; compiled assets then use the same origin on port 18790. The browser tests' server (scripts/frontend-e2e-server.sh) runs on TREG_E2E_PORT, which frontend/playwright.config.ts sets to a port the OS reports free at the start of each run, so runs in parallel worktrees or beside a treg serve proxy (default 18791) never share a port.

A feature flag set in the calling shell (TREG_HUB_ENABLED=1 scripts/dev-local.sh up) does not reach the server on its own: the script starts the server inside a tmux session, and a tmux session inherits the tmux server's environment, not the calling client's — the server would start and answer every hub route 404 with no error. dev-local.sh expands a fixed passthrough list (currently TREG_HUB_ENABLED) in its own process and bakes the value into the command string tmux runs; add a flag to that list to pass another one through. The script's own three env vars (TREG_EMAIL_DEV_MODE, TREG_CONNECT_DEMO_ENABLED, TREG_DATABASE_URL) are appended last so they always win over anything passed through.

deploy/render.example.yaml is a generic self-hosting example. It creates one web service and one PostgreSQL database, builds with bash scripts/build-web.sh (frontend build followed by the locked Python install), runs python -m treg upgrade before serving, starts python -m treg, and checks /meta. Copy it into the operator's own deployment repository and change resource names, region, plans, public URL and integrations.

Build from the lock file

A deployment built from a checkout must install the dependency set uv.lock records, which is the set CI tests (uv sync --locked in .github/workflows/ci.yml). pip install ".[server]" does not read the lock: it resolves every open range in pyproject.toml afresh, so a build made after a dependency publishes a breaking release ships that release with no commit in this repository, and the suite that passed on the lock proves nothing about it. --locked refuses a stale lock instead of resolving, which makes a forgotten uv lock a failed build rather than a silent drift; --no-dev leaves the test tooling out; --active installs into the virtual environment the platform already activated for the start command. Every process that shares a database (web service and scheduled workers) must build the same way, or they run different dependency sets against one schema.

Render adds uv to its Python runtime when uv.lock is present at the service root, but at an older version than [tool.uv] required-version accepts; set UV_VERSION on the service to a version that satisfies it. Upgrading a dependency is a uv lock --upgrade-package <name> (or uv lock --upgrade) commit that CI tests before it can reach a build.

Installing the published wheel (pip install "tools-registry[server]") is a different path: a wheel carries no lock, so that operator pins versions in their own requirements file.

The published source archive is built by Hatchling from the checkout. The Hatch target exclusions in pyproject.toml keep local linked worktrees, root-level working plans and previews, evidence, databases, environment files and hosted-page MP4 demos out of public artifacts. Release validation inspects both archive contents; a clean Git diff alone is not sufficient.

The example is deliberately not the treg.to production Blueprint. The hosted topology and settings are private operational state.

Audit and archive back-pressure

audit.record_call enqueues rows off the request path. One batching writer per process drains them through the background pool. _MAX_PENDING bounds queued rows; overload sheds audit work instead of growing memory without limit. A rejected batch retries row by row so one malformed row costs only that row. Unknown model fields are dropped with an error log rather than silently discarding the whole record.

Archive recording has independent task-count and byte limits. _MAX_PENDING_BYTES bounds bodies retained for database writes, and archive_r2_max_pending_bytes separately bounds object-store work. These are admission budgets, not an RSS ceiling: SDK buffers, compression and mandatory terminal evidence require additional memory headroom.

Spooled settlement evidence (spooled_response, proxy-model.md) writes large metered answers to the process temp directory, not RAM. TREG_SPOOL_BUDGET_BYTES (default 512 MiB per process) bounds that disk use, TREG_SPOOL_MAX_BYTES (64 MiB) one answer, TREG_SPOOL_PARSE_CONCURRENCY (2) the transient parse memory (up to about three times the body each), and TREG_SPOOL_DIR overrides the directory. Size the budget against the host's ephemeral disk: a platform that evicts an instance over its local-storage allowance turns an oversized budget into a restart. Production observations and incident history live in the private database-capacity runbook.

Hosted feature switches

Several settings alter optional shared-service behavior without changing the underlying public contract:

• TREG_PLATFORM_DAILY_CAP_USD is the default per-UTC-day shared-provider spend limit for a team that has not set its own; 0 (the default) is no limit. A team's own setting wins in either direction.
• TREG_OAUTH_BILLED_PROVIDERS names OAuth providers whose upstream bill lands on the registry operator. Own-app connections are never metered by this switch.
• TREG_ROUTED_DISCOVERY controls whether discovery leads callers to routed capability tools. It does not delete or make existing endpoint IDs uncallable.
• TREG_PLATFORM_PROVIDERS enables use of configured platform credentials.
• TREG_OVERFLOW_MODE and TREG_OVERFLOW_DAILY_BUDGET_USD control same-vendor overflow. See capacity.
• TREG_SEARCH_EXPERIMENT (off | shadow | interleave) runs the discovery experiment on the MCP search tools; it is also its kill switch. Needs TREG_TYPESAFE_API_KEY; TREG_TYPESAFE_TIMEOUT_S bounds what the judge may add to a search. Off by default, and off whenever the key is empty.
• TREG_FIND_ENGINE (v1 | v2 | shadow) picks what /catalog/find answers with (see find); shadow serves v1 and logs v2 beside it, at a second judge request per find. Rolling back is setting it to v1.
• TREG_FIND_EMBED_API_KEY turns on v2's semantic channel; with the default OpenRouter TREG_FIND_EMBED_URL an empty value uses TREG_PLATFORM_KEY_OPENROUTER. Card vectors are cached in the archive's object store when archive R2 is on, else rebuilt by each process.

Exact treg.to values, funded accounts and rollout instructions live in the private provider-capacity runbook.

Worker commands

treg-worker is a console script in the [server] extra. It hosts scheduled maintainer commands without importing the heavy database stack into the light treg CLI.

• treg-worker capacity sweep collects provider capacity and writes snapshots.
• treg-worker overflow verify obtains route evidence and spends real money when configured.
• treg-worker overflow sync derives enabled overflow routes from current evidence.
• treg-worker asynctasks settle completes durable holds for asynchronous upstream operations.
• treg-worker arena insights folds new audit rows into the rolling Arena aggregate (--max-seconds, default 110, bounds one pass; schedule it every two minutes). It requires the archive object-store settings when R2 reads are enabled, opens the same client lifecycle as the web service, and flushes read analytics before exiting.
• treg-worker catalog stats folds new audit rows into per-endpoint, per-day reliability buckets (--max-rows, default 500,000, bounds one pass; schedule it every few minutes). The catalog keeps computing observations live until this command has caught up once, so it can be scheduled after the application deploys, and a self-hosted registry that never schedules it loses nothing.
• treg-worker jev xboost runs the /jev launch-radar demo once a day: it calls treg's own /call/ API with TREG_JEV_TREG_TOKEN (a member token of the demo team, so the spend is an ordinary bill) and jev through the Vercel AI Gateway (TREG_AI_GATEWAY_API_KEY), and stores the run under Ephemeral for the page. Both variables also belong on the web service, which needs them for the visitor judge endpoint.
• treg-worker admin purge-evidence blanks failed-call evidence past the 14-day retention window (--batch-size, default 5000, rows per transaction; schedule it daily). GET /admin/errors is read-only and already withholds evidence past the window, so an unscheduled purge keeps the old bytes in the database but never shows them.
• treg-worker hub check [--only <tool_id>] [--json] runs every live hub tool's check.json once, as its maker (spends the maker's balance at step prices); a no-op with exit 0 when hub_enabled is off. Schedule it every 6 hours; health is derived from the last three runs.

The two analytics commands exist so that no web process aggregates the audit table beside the money path; callrecord is read only through the persisted cursors they own. Workers call read-only verify_db() before work and must run against a compatible schema. They need only the credentials and configuration required by their job (the two analytics commands need only the database URL). Hosting schedules, service wiring and manual production procedures belong in the private deployment runbook.

Managed-key rollout uses revisions 0034 through 0036. Apply the key controls and generation before the separate concurrent Activity index build. The index migration allows 180 seconds for lock waits and 600 seconds per statement, then restores 5/120 seconds. A failed concurrent build can leave an invalid index; retrying 0036 removes and rebuilds only that invalid index. Do not deploy server code older than 0034 after key disable or revoke state has been recorded. The package version follows current main; this branch does not publish a release.

For hosted rollout, release and verify the compatible CLI before deploying the managed-key server. The served installer installs from PyPI, so changing the server alone does not make treg update install the new client. Old browser login and saved-token calls remain usable; affected email and team-change requests receive an update instruction before their local state can be replaced.