Transition the AX architecture from asynchronous event queues (Redis Streams
and the `ax-controller` worker pool) to direct, synchronous reconciliation in
`ax-server` guarded by fine-grained distributed locks.
Key changes:
- Implement per-resource locking (`lock:task`, `lock:workspace`, `lock:model`)
using Redis distributed locks (`SET NX PX` with token validation) and an
in-memory locker for testing.
- Move Substrate reconciliation directly into `ax-server` for Task, Workspace,
and Model lifecycle methods (`Create`, `Resume`, `Suspend`, `Delete`).
- Decommission `ax-controller`, `deploy/ax-controller.yaml`, and Redis Stream
event publishing/consuming interfaces.
- Simplify `ax delete` and `ax watch` in the CLI to eliminate two-phase
polling loops now that operations complete synchronously.
- Update `demo.sh`, build configurations (`Makefile`, `.ko.yaml`), and
documentation to reflect the single-binary control plane architecture.
Refine the Model section in docs/concepts.md to directly describe it as a
declarative configuration resource for model providers, parameters, and
credentials rather than using colloquial phrasing.
Fixes#356
* Replace UpdateTask with CreateTask and enforce immutability
Replaced UpdateTask RPC with CreateTask. Tasks are now
immutable once created; modifying an existing task returns a
FailedPrecondition error.
- Renamed UpdateTask/UpdateTaskRequest to CreateTask/CreateTaskRequest.
- Checked existence in Server.CreateTask to prevent mutation.
- Updated CLI apply logic, tests, and documentation.
* refactor: remove TaskSpec.Suspend and default new tasks to suspended state
* chore: update default snapshots bucket to snapshot-substrate-test-ax-substrate
ax apply accepted any metadata.name, but Substrate requires actor,
ActorTemplate and atespace names to be lowercase RFC 1123 labels. A name
such as Task-With-Caps was therefore stored, and only failed seconds
later with an asynchronous ActorCreationFailed condition that the user
had to dig out of ax describe or the controller logs.
Validate metadata.name (and metadata.atespace when set) against
[a-z0-9]([-a-z0-9]*[a-z0-9])? with a 63 character limit for Task,
Workspace and Model alike, and return InvalidArgument at apply time.
Fixes#370
Remove the Gateway resource across the API, controller, server, CLI,
store, and documentation.
Gateway introduced an unnecessary layer and architectural bifurcation
between Agent Substrate and AX. By removing it, AX aligns directly with
Substrate's native primitives and networking rather than maintaining an
overlapping ingress/routing abstraction.
- Reserved protobuf field 8 and removed Gateway RPCs and messages.
- Cleaned up controller reconciler, server endpoints, and store models.
- Updated CLI, documentation, and example manifests.
AX is undergoing a significant redesign, moving away from a single CLI with
an embedded Python harness toward a general-purpose orchestration layer for
agentic tasks. The repository is reorganized around three binaries: ax-server
exposes a gRPC API for submitting Task manifests, ax-controller runs as a
horizontally scaled reconciler that consumes work from Redis Streams, and
ax-task-runner executes tasks inside sandboxed workers. Task state now lives
in Redis rather than Kubernetes CRDs so the system can handle millions of
short-lived tasks without straining etcd. The legacy Python harness, ATE
client, SQL event log, and skill examples are removed, and new design,
concepts, manifest, networking, runner, and sandbox docs plus Kubernetes
deploy manifests describe the new architecture.
So we don't accidentally promote this option. There are also rough edges
in this implementation that we don't have time to fix, e.g. the use of
kubectl.
This change introduces an EPP to create/resume actor the incoming
conversation_id. We also introduce an ax-shell image and a demo pod for
testing purposes.
Follow up changes:
- Suspend the AX server automatically after the request is sent.
- Explore failure cases and how we can ensure suspension safely even if
AX server crashes.
- EventLogService.Fork shouldn't autogenerate a dest_conversation_id.
dest_conversation_id should always be supplied from the client for
routing.
This change also removes the RegisterAgent RPC. Actor
resumption/suspension requires a valid conversation_id for all RPCs and
RegisterAgent no longer fits into this new constraint.
This PR adds Colab agents into AX. The main use case covered:
1. From a local python script, run it with Colab VM, optionally download
output images, and convert it to a notebook and save in Google Drive.
2. From a remote notebook in Google Drive, run it with Colab VM, get the
output in AX and optionally download output images.
Note: This is only available in gLinux for internal Google version of
Colab CLI. Expect the OSS version coming soon. Instructions at
docs/colab-agent.md.
The changes introduce a conversation log to delegate the history storage
and retrieval to AX. Instead of treating every task/execution as a new
isolated event, AX now supports conversational history.
Conversational history is implemented as a layer above the execution
log. Executions are still logged separately and with much more rich
context that enables resumption within the lifetime of an execution.
With this change, controller natively can identify an execution that is
waiting for resumption from the conversation log. Clients don't need to
know about the underlying execution identifier.
The change also removed the WIP forking capability until we stabilize
the event log schema and the concepts. We will implement it as a follow
up change.
`ExecRequest` is updated to support conversation identifier and
`last_seen_seq`. Sequence numbers will be used to be able to resume
playing from a moment client has since the last. If client connection
disappears, we will still be able to communicate the outputs during a
reconnect.
Some follow-up changes after this change is merged:
- Support for `last_seen_seq`, `seq` number in the ExecResponse.
- Implementation of forking.
- Potential schema changes to conversation_log and execution_log.
- More README updates.
- Rewrite of the trace viewer tool with the new schema once the logs are
more stable.
- Controller will introduce its own OutputHandler to emit additional
fields, e.g. `seq`.
- Remove redundancy between conversation_log and execution_log.
JBD