openrig/docs/reference/stream-classifier-worker.md

Occupant-owned stream classification and shadow capture (experimental)

Experimental in 0.6.0, optional and off by default. The Jev classification path and shadow capture may be incomplete or not work, and no accuracy or production-reliability claim is made. OpenRig's delivery, watchdog and routing decisions do not use their output. If they do not behave as described here, please open an issue or send a pull request (CONTRIBUTING.md).

This source provides a bounded receiver for one real classifier occupant. It does not start an agent, register a watchdog job, or enable capture by default. Taxonomy, questions and decisions belong to the occupant; the daemon supplies source facts and enforces its existing lease, execution and idempotency fences. An explicit optional Jev mode supplies advisory decisions through the same worker. Experimental labels have no demonstrated accuracy or calibrated confidence and do not replace delivery, watchdog, or routing decisions.

Optional foreground Jev experiment

With no configuration, the experiment is off. Choose an explicit local JSON path in an existing private directory; the file contains only enabled state and bounds:

rig project experimental status --config ./experiment.json --json
rig project experimental enable --config ./experiment.json --max-requests 3 --timeout-ms 10000 --json
rig project experimental disable --config ./experiment.json --json

Enable does not start processing, capture, a daemon or an agent. Status and disable make no provider request. The invoking process needs OPENROUTER_API_KEY in its environment for an enabled run; this entry never searches credential files or copies credentials into configuration. Missing credentials fail before attempts or provider work. Normal key provisioning remains with the operator.

The selected route is https://openrouter.ai/api/alpha/decisions, pinned to typesafe/jev-1.13, without provider fallback or HTTP redirects. A request declares maximum provider prices of $0.05 per million prompt tokens, zero completion and request price. These are admission ceilings, not a price quote, spend estimate or credential-access claim. Each foreground run allows 1–20 requests (default 3), each with a 1–30 second deadline (default 10), 24 KiB request and 256 KiB response limits. Failed requests consume their count. Nothing retries in the background.

To process stream items from the actual selected occupant:

rig project wake --project PROJECT --taxonomy ./taxonomy.yaml \
  --classifier-version experimental-jev-1.13 --evidence-epoch OWNER_EPOCH \
  --limit 3 --experiment ./experiment.json --json

Use the taxonomy format described below. Each criterion must have a nonempty string description; __unknown__ is reserved. Questions stay outside the daemon. The request sends the selected item's body, supplied taxonomy questions/criteria, canonical scope IDs and current roster destinations to OpenRouter. Select inputs appropriate for that external processing. It does not transmit the credential in evidence or store it in the classification. The output binds current source, taxonomy, candidate version, lease, occupant generation and attempt execution. No action or confidence label is inferred. Duplicate judgment remains manual.

The smaller of --limit and the configured request maximum bounds the wake. Unknown answers remain null; a wholly unknown answer abstains. Invalid model, provider, answer keys, choices, probabilities or response bytes reject the whole answer. Probability sums retain the strict 0.001 tolerance; there is no rounding repair or partial salvage. Unavailable/invalid provider work stops that run with no classification write. Inspect the returned attempt state: terminal abstentions are not reopened, and failed/unknown daemon writes must be reconciled through the existing ledger. Never change the evidence epoch merely to retry an experiment.

Disable is checked before each call and before applying its result. A separate disable command cannot cancel an already forwarded remote request; its late result is discarded, with the existing deadline bounding the wait. Ctrl-C or SIGTERM requests immediate cancellation of the foreground run. An uncooperative request remains reported pending and cannot cause a late write or a second call in that run. Remote cancellation or refund is not proven by local cancellation. The command registers no wake, returns no automatic continuation, and requires an explicit later invocation to do more work.

For selected observations from an existing shadow archive:

rig project experimental capture --config ./experiment.json \
  --input ./selected-shadow.jsonl --output ./new-advisory-results.jsonl --json

This reads at most 8 MiB and considers only the first configured maximum number of records. It never captures a terminal. The input remains unchanged; output is exclusively created at mode 0600. Keep prior output and choose another filename only for a deliberately selected new run. It sends captured post-screen text to the same provider, and retains the original attempt/node/occupant/pane binding and capture hash beside the answer. Missing capture or binding is unavailable and makes no provider call. State answers are experimental hints; delivery remains INDETERMINATE because screen text cannot prove submitted/consumed/effect and the archive does not retain the original sent text. No result feeds back into live transport or inventory. Failed runs preserve already written output; inspect unavailable, remaining, call count and stop reason rather than treating a file as proof of success.

One bounded wake

From the selected occupant, read candidates using the configured project ID:

rig project candidates --project PROJECT --taxonomy /private/taxonomy.yaml \
  --classifier-version VERSION --evidence-epoch OWNER_EPOCH --limit 20 --json

The result includes eligible stream IDs, authored scope IDs with exact source paths/hashes, the current running roster of the occupant's rig, observation time, and a candidate version. Taxonomy YAML supplies version and fields.{kind,urgency,maturity,area} with a question and values mapping. These questions are returned to the occupant; draft values are not verified truth. Confidence has no selectable values until calibration selects that contract. Destination choices are the current roster sessions plus explicit pool. Null/omitted destination remains unknown; it is not converted into pool. The candidate hash binds the selectable values as well as source and taxonomy hashes. Packets prepared before this pool correction require fresh preparation; it does not reopen historical attempts or change the evidence epoch.

Read each eligible item with rig stream show ITEM --json. Duplicate candidates are the latest 100 nonarchived items with exact body-hash evidence references; their preview is capped at 2,000 characters and marked when truncated. Read the complete candidate before selecting a duplicate. A retrieval miss means unknown. Directory names, issue IDs and bare membership never supply scope IDs. Source errors are explicit, and partial source snapshots cause abstention.

Write an occupant-owned decision file (at most 100 unique item IDs, at most 1 MiB):

{
  "candidateSetVersion": "sha256:<version returned by candidates>",
  "decisions": [{
    "streamItemId": "<eligible ID>",
    "bodyHash": "sha256:<SHA256 of the exact UTF-8 item body>",
    "decision": { "kind": "classify", "labels": {
      "classificationType": "<selected taxonomy value>",
      "scopeRef": "<selected canonical mission or slice ID>",
      "needsHuman": null
    }}
  }]
}

An explicit abstention is {"kind":"abstain","reason":"why unknown"}. Other label keys are classificationUrgency, classificationMaturity, classificationDestination, area, and classificationConfidence. Omitted or null labels stay unknown. A duplicate needs both duplicateOfStreamItemId and the matching duplicateEvidenceRef from the candidates, plus occupant judgment that it really is a duplicate. An existing item alone is not that judgment.

rig project wake --project PROJECT --taxonomy /private/taxonomy.yaml \
  --classifier-version VERSION --evidence-epoch OWNER_EPOCH --limit 20 \
  --decisions /private/decisions.json --json

This command actually invokes one StreamClassificationWorker.wake() through the existing HTTP services. It derives sender identity through DaemonClient, requires a current running occupant, and carries the captured generation through each lease/attempt/classification request. It never invents a daemon holder. Existing direct HTTP callers keep their documented sender-provenance behavior.

The command refreshes source candidates before acquiring a lease. Missing or stale decision packets return unavailable without starting attempts; refresh preparation if the roster, scopes, taxonomy or recent stream changed. During a wake, an item with no exact body-bound decision, missing sources, or a selected candidate that is unavailable abstains. This is deliberately strict; continuous source churn can require repeated preparation and is not a throughput claim.

Each wake processes at most --limit items (1–100; default 20), starting with the oldest eligible work rather than relying on a lossy notification cursor. Durable terminal attempts stay excluded. A new candidate version does not authorize another pass: only an owner-selected evidence epoch changes that attempt identity. Retain the candidates output beside the decisions and result so its version is reproducible. The CLI does not keep a second local ledger.

The result carries per-item outcomes, moreEligible, nextWakeAt, and a watchdog wakeRequest descriptor with the real session/generation and a receiving instruction. registered:false is literal. A periodic-reminder message asks the occupant to prepare and run this entry; delivery of that message is not execution. Registration/placement remains an explicit later action. Honor nextWakeAt: lease maintenance uses TTL/3 opportunities; lease loss or an unavailable service backs off 60 seconds. The daemon's retry/exhaustion ledger remains authoritative. The ordinary client bounds each request at five seconds; a timed-out write has an unknown outcome until the ledger is read, and is never reported as written.

There is no provider in the manual-decision path. The reusable asynchronous worker has one pending classifier slot; timeout requests cancellation but cannot force it. An uncooperative classifier blocks further calls in that instance and late results cannot write. That in-memory pending slot is not cross-process cancellation.

Separate bounded shadow drain

The production transport and inventory probes accept the same optional observer. They record only captures already taken, with the node, occupant, pane and capture target bound before awaits. The ordinary cheap inventory path still takes no capture. Retained-no-write is a separate event, never delivery evidence.

Activation requires explicitly supplied OPENRIG_SHADOW_CAPTURE JSON at a later authorized daemon start. Absence or invalid configuration leaves capture disabled. This document supplies no live destination or activation instruction. Required fields and engineering ceilings are:

Field	Meaning	Maximum
destination	New absolute JSONL file in a canonical, current-user-owned 0700 directory	Explicit path
maxRecords	Total reserved sink records	100,000
maxBytes	Total reserved sink bytes, including newlines	1 GiB
capacity	Queued observation count	1,000
maxQueuedBytes	Serialized bytes waiting in the queue	8 MiB
maxObservationBytes	Serialized bytes per observation	1 MiB

All numeric fields are required positive safe integers. These are engineering capacity limits, not approved corpus sizes or classification thresholds. One in-flight drain can hold up to 64 additional observations, bounded by the queue's byte allowance; queue plus drain can therefore retain up to twice that allowance. The standalone observer defaults are 256 entries, 4 MiB queued and 256 KiB per observation; explicit production configuration supplies each value.

rig project shadow-status only inspects configuration/counters. rig project shadow-drain asks the existing HTTP service to drain up to 64 rows; it requires an actor and never enables capture. No background drain timer is added. rig project shadow-stop disables new enqueueing immediately, finishes any active drain and the finite retained queue, then closes the sink. Repeated stop is idempotent; it cannot enable capture or delete the existing archive. Existing quota and sink-failure losses remain counted. A slow disk can leave stop pending; inspect status after a client timeout instead of assuming the flush completed. Enabling capture again requires a separately authorized daemon start with a new destination; enabling the local provider experiment does not enable capture. The file is exclusively created with mode 0600. An existing file is refused, never overwritten, appended, rotated or deleted. A restart requires a new selected destination. Private-file creation currently requires POSIX ownership support.

The hot path only performs bounded synchronous enqueue work; it never awaits disk or drain completion. Slow sinks leave one drain pending; concurrent drains do not start another writer. Overflow drops new observations. Capacity exhaustion stops the sink. Errors stop writes, count losses and preserve any partial file; failed batches are not replayed. reservedRecords/Bytes include attempted writes, while completedRecords/Bytes count successful appends. drained means removed from the observer, not necessarily persisted. Queue count drops, byte-bound drops, record errors, missing/unavailable captures and sink failures remain separate.

Live collection, private destination/retention selection, corpus labels, hosted calibration, native placement, package/install proof and independent visible / submitted / consumed / effect evidence remain outside this source entry.