pi/packages/chord

@earendil-works/chord

Chord is an application-composition runtime for systems assembled from plugins/extensions. It provides facets, services, replicated state, and a pluggable remote-service boundary. It is developed as a standalone package in the Pi monorepo, but it is not a Pi package: it does not depend on any other Pi workspace package and can be used by unrelated applications.

What Chord is for

A single application feature may need to run in several environments: for example, an agent worker, a terminal UI, and a remote WebUI. Chord provides the generic machinery to write such extensions in a way that is both delightful for humans as well as agents.

The design has a few connected pieces:

• Plugins are synchronous setup units that declare the services they provide and require. After every plugin has declared its shape, a host validates the complete dependency graph, binds services, activates providers before consumers, and disposes resources in reverse dependency order. These units are called facets.

• Facets are parts of a plugin. Each facet is bundled up separately and runs in the process or environment where it's supposed to run. You can use facets to split a plugin into separate pieces that need to be loaded into different processes and environments (think backend, browser, TUI etc.)

• Services are typed, stable tokens with either one provider (singleton) or dynamic keyed instances (keyed). A service can be process-local, with an unrestricted JavaScript contract, or remotely exposable. Consumers retain a stable facade while a provider disconnects or is replaced.

• Replicated state exposes authoritative state to local and remote connected consumers. Producers publish atomic overlay transactions with change(context, callback); consumers receive complete immutable values. Draft proxies exist only during the callback and become unusable afterward. Preparation materializes a structurally shared immutable candidate and one exact decoded operation batch, while each remote client/state stream owns independent path-codec state. Replicas become unready on disconnect or replacement until rehydrated.

• Delta tracking records and coalesces operations over tracked plain JSON. It preserves common string and array operations, supports durable base batches, and validates untrusted operations as they are applied. Batches guarantee convergence but are not canonical or necessarily minimal.

• Remote service sources advertise services available outside a facet host and open bindings for the services its facets require. Bindings carry logical calls and subscriptions through an application-supplied adapter. Chord requires strict-JSON arguments, results, snapshots, updates, and catalogues, but does not prescribe framing, routing, transport, or an application wire envelope. JsonRepresentation<T> derives a wire-safe type for application data with unknown payloads, while isJsonValue() validates received values at an adapter boundary. Symmetric RPC peers are planned as one optional implementation of this boundary.

• Context Chord provides a Go-like context system for cancellation and invocation-scoped application values. Applications can carry permissions or telemetry through those values without Chord depending on either.

The current runtime exports service tokens, singleton and keyed providers, remote bindings, replicated state, facet hosts, and facet loaders from @earendil-works/chord. Import public types and general runtime APIs from the package root. Context constants and functions live in @earendil-works/chord/context because their generic names should not pollute the root API. Chord-owned identifiers use the chord.* namespace and its reserved service prefix is $chord.*.

Remote service adapters

Chord owns its transport-independent service wire grammar. Consumer adapters use createServiceCatalogueCall(), createServiceSubscribeCall(), and createServiceUnsubscribeCall() for $chord.service control calls. createRemoteServiceEndpoint() handles those calls for one provider consumer, including subscription activation and cleanup. parseServiceCall(), parseServiceCatalogue(), and the decoded/wire snapshot and update parsers validate Chord semantics after an adapter has established a strict-JSON boundary. RemoteServiceErrorCode and REMOTE_SERVICE_ERROR_CODES define the service errors that may cross that boundary.

Replicated state operations use one createServiceStateEncoder() at the provider side and one createServiceStateDecoder() at the consumer side for each subscription. Those registries create an independent Delta path dictionary for every instance/member state and reset it on replacement, unavailability, close, or fresh hydration. Applications may place these values inside any routing, request, response, or event envelope; Chord does not prescribe that outer protocol.

Provider subscriptions atomically capture a snapshot and buffer subsequent updates until activation. Activation and reentrant publication use the same FIFO. The provider retains at most 100 pending updates per subscription, excluding the running delivery. Adding update 101 replaces the entire pending queue with { type: "reset", snapshot }: a current full subscription snapshot whose state members contain [["r", value]] and their new sequence baselines. It also captures current instance membership, so discarded spawn/close/replacement events cannot leave stale instances behind. Live keyed generations retain their existing handles. The reset carries the context of the publication that triggered overflow.

Consumers and codecs must handle this explicit reset before accepting subsequent ordinary deltas. A root operation in an ordinary state update does not permit a sequence gap. Resets restart the subscription's path dictionaries, and later deltas must be contiguous from each reset baseline. Transport adapters must preserve snapshot/reset/update order; they must not drop encoded delta batches or assume their own asynchronous queues are bounded by the provider's queue.

Tracking JSON deltas

Import the standalone transactional tracker from @earendil-works/chord/delta:

import { applyImmutable, track } from "@earendil-works/chord/delta";

const tracker = track({ output: "", count: 0 });
const change = tracker.beginChange();
change.state.output += "done\n";
change.state.count += 1;
const prepared = change.prepare();

tracker.adopt(prepared);
const replica = applyImmutable(prepared.base, prepared.ops);

tracker.value is always the latest adopted immutable revision. Preparation does not change authority; adoption validates the preparation and swaps the root pointer. Assigned containers are copied by value and unchanged subtrees may be shared between revisions.

The tracker uses trusted immutable ownership rather than defensive copying or freezing. track(initial), prepareReplace(value), replicatedState(initial), and replace(context, value) take ownership of alias-free strict-JSON roots without walking them. Callers must not mutate transferred or published data. Values placed through drafts are already copied, so that walk also rejects non-strict JSON before the draft changes. Published values are not frozen. In-process loopback consumers may share their containers with the provider; mutating a consumed value violates the contract and can corrupt authority. Clone or serialize at any mutable trust boundary.

Replicated state provides the same model through a callback:

const initial = { output: "", count: 0 };
const status = env.replicatedState(initial); // transfers ownership of initial
status.change(context, (draft) => {
	draft.output += "done\n";
	draft.count += 1;
});

A successful change() publishes exactly one atomic revision. If its callback throws, the original value and sequence remain unchanged. Draft handles become unusable when the callback returns. Chord emits string append and front-truncate operations, array splices and permutations, sets, and deletes; large edit sets may fold into a complete replacement. Remote connection plumbing encodes each batch independently for every client/state pairing.

Public state subscriptions

state.subscribe(async (value, context, delivery) => { ... }) serializes callbacks independently for each subscription. Hydration starts immediately when a value is available, and its returned promise must settle before updates start. Synchronous callbacks still run synchronously. Read the captured value inside an asynchronous callback: state.value may already refer to a later revision.

Each subscription retains at most 100 pending complete-value deliveries, excluding the running callback. On overflow, only the newest pending value, context, and delivery metadata are retained; a not-yet-started initial hydration is preserved. Thus public delivery sequences may skip. This is a frame-count policy, not a byte limit. Internal exact-operation subscriptions remain synchronous and receive every revision, independently of slow public callbacks.

The returned unsubscribe function immediately discards pending work and prevents new callbacks. It neither aborts nor waits for the running callback. Synchronous throws and promise rejections are observed, reported, and do not stop other subscriptions or subsequent callbacks. Attached sources and remote bindings use their onError handler; local mutable states (and attached sources without a handler) report failures by throwing in a microtask. Unsubscribing does not hide a later rejection from the running callback.

The standalone Delta guide defines the complete ownership, lifecycle, operation, and replica contracts.

Bundling and loading facets

@earendil-works/chord/bundler uses esbuild to turn ESM or TypeScript application entries into independent, content-addressed CommonJS files. The package-level API reads plugin identity and build configuration from package.json, then applies facet path conventions supplied by the host application:

{
  "name": "@example/my-plugin",
  "version": "1.0.0",
  "type": "module",
  "peerDependencies": {
    "@earendil-works/chord": "^0.84.4"
  },
  "chord": {
    "facets": {
      "worker": "./src/custom-worker.ts",
      "presentation": false
    }
  }
}

import { bundleFacetPackage } from "@earendil-works/chord/bundler";

await bundleFacetPackage({
	packagePath: "/path/to/my-plugin",
	outdir: "/application-owned/plugin-builds/my-plugin",
	defaultFacets: {
		worker: "src/worker.ts",
		presentation: "src/presentation.ts",
	},
});

Existing conventional files become entries unless chord.facets overrides or disables them. Peer dependencies are externalized and resolved against the host when loading. Chord never installs dependencies or runs package lifecycle scripts. bundleFacets() remains available as the lower-level API for callers that already have explicit plugin identity and entry mappings.

The output directory contains one .cjs file per entry plus chord-facets.json. Load one application-selected entry through the Node-only loader:

import { createFacetBundleLoader } from "@earendil-works/chord/node";

const loader = createFacetBundleLoader({
	manifestPath: "/application-owned/plugin-builds/my-plugin/chord-facets.json",
	entry: "worker",
	resolveExternal: (specifier) => import.meta.resolve(specifier),
});
const loaded = await loader.load();

Each load() verifies SHA-256 integrity and compiles the CommonJS body directly with node:vm instead of putting the plugin into Node's CommonJS or ESM module cache. Externals are resolved by the host and loaded through a restricted require; esbuild lowers dynamic imports so they use the same path. Disposing a retired generation releases the loader's facet references, making its compiled code eligible for garbage collection once plugin-owned resources are also gone.

For transport to another Node host, readFacetBundleArtifact() packages one verified manifest entry with its source, and createFacetBundleArtifactLoader() materializes fresh temporary generations while resolving externals against the receiving host.

To reload, load a candidate, pass its facets to FacetHost.reload(), dispose the candidate on failure, and dispose the retired LoadedFacets only after a successful cutover. The host activates and validates the candidate while the currently active providers remain routed, then replaces each singleton directly without an unavailable interval. Stable service handles therefore do not become disconnected during an ordinary reload. Keyed instances remain incarnation-specific and replacements receive fresh generations. The bundler writes a complete temporary directory before replacing the previous output, so loaders do not observe partially built generations.

See PLANNING.md for the broader RPC and generation-loading architecture.