Make Task manifests fully immutable once created, removing `UpdateTask`
in favor of `CreateTask` and driving lifecycle transitions (suspend,
resume, delete) through dedicated RPCs and event stream processing.
- Proto: Replace `UpdateTask` with `CreateTask` in AX service.
- Proto: Remove mutable `suspend` field from `TaskSpec` and reserve field 2.
- API / Server: Implement `CreateTask` with duplicate name rejection
(`codes.AlreadyExists`). Update `SuspendTask`, `ResumeTask`, and `DeleteTask`
to transition status phase and publish lifecycle events.
- Store: Remove redundant `SetTaskSuspended` and `MarkTaskDeleting` methods;
standardize on `CreateTask`, `UpdateTaskStatus`, `DeleteTask`, and
`PublishEvent`.
- Controller: Decouple spec reconciliation from task suspension; introduce
`ReconcileSuspend` to handle checkpointing and pausing actors on Substrate.
- CLI: Update `ax apply` to submit tasks via `CreateTask`.
- Docs: Update design docs and specifications to reflect task immutability.
AX is undergoing a significant redesign, moving away from a single CLI with
an embedded Python harness toward a general-purpose orchestration layer for
agentic tasks. The repository is reorganized around three binaries: ax-server
exposes a gRPC API for submitting Task manifests, ax-controller runs as a
horizontally scaled reconciler that consumes work from Redis Streams, and
ax-task-runner executes tasks inside sandboxed workers. Task state now lives
in Redis rather than Kubernetes CRDs so the system can handle millions of
short-lived tasks without straining etcd. The legacy Python harness, ATE
client, SQL event log, and skill examples are removed, and new design,
concepts, manifest, networking, runner, and sandbox docs plus Kubernetes
deploy manifests describe the new architecture.
We want AX to be compatible with the Interactions API protocol as much as possible. This PR aligns the proto with Steps and with the Interactions semantics. We will follow up with more changes to make it more compatible with Interactions while preserving a `conversation_id` based history model.
`ConversationService` is removed for now until we know what kind of data operations we want to provide for event log management.
Refactor telemetry configuration and resolve deprecated GCP metadata calls:
- Replace deprecated `metadata.ProjectID()` and `metadata.OnGCE()` with context-aware versions.
- Update `telemetry.SetTraceProvider` to accept variadic `otlptracegrpc.Option`s and remove the hardcoded endpoint parameter.
- Add `telemetry.WithGCPCredentials()` helper to encapsulate GCP-specific credentials.
- Isolate GCP-specific server telemetry configuration into a new `serve_gcp.go` file.
Fix premature gRPC stream cancellation in the harness execution clients:
- Update the client receive loop to drain the stream until `io.EOF` after receiving the `HarnessEnd` frame, preventing `CANCELLED` errors in telemetry.
- Encapsulate the stream draining and message dispatching logic into a shared `drainStream` helper in a new `stream.go` file.
- Clean up unused `io` imports in `antigravity.go` and `substrate.go`.
Follow-up:
- End-to-end trace context propagation to HarnessService deployments
- End-to-end trace context propagation to Substrate calls
* Cleanup the repo and move to controller2
* Migrate controller and registry to internal/controller2
- Remove the legacy harness build tag
* refactor: move GeminiConfig definition from internal/config to internal/gemini package
* test: replace controller registry with dummy implementation and update dependency constraints in go.mod
* chore: remove harness build and test steps from GitHub Actions workflow
- Register antigravity harness
- Don't reset the agent_id, we want to stick to the same agent
- Make sure registry isn't nil at controller2.New
This is a follow up for #33.
- Make changes to the testagents for a realistic sub agent demo
- Sanitize agent names, replace - with _ to avoid malformation errors
- Increase the idle networking timeout across different configs, set a
high GKE ingress timeout to enable long running operations
- Register AX_TEST_AGENTS with description and name
- Make changes to the symptom prompt and disable bash tool improve the
selection quality for the demo
- Move inflight tracking to server to avoid double suspension
- Make the resumption command to honor --server if set
Follow up:
- Remove AX_TEST_AGENTS once we don't need them.
- Work on Gemini planner to support better selection among the agents
and the bash tool.
We will make conversation actors to be forkable from snapshots in the
upcoming weeks. Renaming this service so it represents more than just
event log operations.
The dummyAgent test fixture defined a HealthCheck method that was never
part of the agent.Agent interface and had no callers. Remove it so the
fixture matches the actual interface (Connect, Close) and nothing else.
Two related fork fixes uncovered while testing `ax fork --src-seq`:
1. controller.Fork: validate src_seq exists in source. Previously,
passing a non-existent src_seq (e.g. a typo or stale checkpoint)
silently degraded to "fork all events", which is misleading. Mirror the
validation tryResuming already does for --last-seq and return "src_seq N
not found in conversation X".
2. server.ForkConversation: update the README to require destiation
conversation id. this is required for Substrate integration
Tests:
- controller: TestController_Fork_SrcSeqNotFound covers the new
validation and asserts no events are appended on failure.
This change introduces an EPP to create/resume actor the incoming
conversation_id. We also introduce an ax-shell image and a demo pod for
testing purposes.
Follow up changes:
- Suspend the AX server automatically after the request is sent.
- Explore failure cases and how we can ensure suspension safely even if
AX server crashes.
- EventLogService.Fork shouldn't autogenerate a dest_conversation_id.
dest_conversation_id should always be supplied from the client for
routing.
This change also removes the RegisterAgent RPC. Actor
resumption/suspension requires a valid conversation_id for all RPCs and
RegisterAgent no longer fits into this new constraint.
This PR adds a new conversation_id field in AgentMessage.
Also add the conversationID to the signature of Agent.Connect and
Executor.Exec so that we can:
1. propagate conversationID from controller to sub-agent, like remote
agent, a2a agent
2. propagate conversationID to substrate worker
The conversationID is unused by anything yet. It will be picked up by
a2a agents in follow up PRs, and possibly substrate agent as well.
1. Conversation Forking
Protocol: Added Fork RPC to EventLogService with support for specifying
source conversation, sequence number (checkpoint), and optional
destination ID.
Controller: Implemented the Fork method to clone events while preserving
shared execution references.
CLI: Added root-level ax fork command. It supports both remote mode (via
--server) and "headless" local mode (reading ax.yaml or custom config
via --config).
2. Trace Viewer Fixes
Display Correct ID: Fixed the trace viewer to display the actual
Conversation ID in the terminal output and UI header instead of the
random root execution ID.
Chronological Ordering: Fixed a bug where execution cards were sorted
alphabetically by their random UUID ExecID. They are now sorted
chronologically based on the order they occurred in the conversation.
3. Testing
Added unit tests for the forking logic in the controller.
Added integration tests for the server-side RPC.
Co-authored-by: anj-s <anjalisridhar@google.com>
This change introduces AXEventLogService to allow AX clients to be able
to operate event log operations. As a part of this change, we are also
allowing clients to be able delete a conversation (including all of its
sub executions).
Follow up:
- We should introduce an `ax eventlog` to make it possible to trigger
some of these operations directly from the CLI.
- Move `ax fork` to `ax eventlog fork`.
We need to return an error if an agent becomes available even though
being registered as an agent. Health check based selection causes
quality regression that are hard to debug and isn't the expected
behavior.
Depends on https://github.com/google-gemini/ax/pull/183.
Change also removes the expectations that Content types should include
some whitespace. For token by token streaming, we will use ContentBlock
and ax exec can behave differently.
The changes introduce a conversation log to delegate the history storage
and retrieval to AX. Instead of treating every task/execution as a new
isolated event, AX now supports conversational history.
Conversational history is implemented as a layer above the execution
log. Executions are still logged separately and with much more rich
context that enables resumption within the lifetime of an execution.
With this change, controller natively can identify an execution that is
waiting for resumption from the conversation log. Clients don't need to
know about the underlying execution identifier.
The change also removed the WIP forking capability until we stabilize
the event log schema and the concepts. We will implement it as a follow
up change.
`ExecRequest` is updated to support conversation identifier and
`last_seen_seq`. Sequence numbers will be used to be able to resume
playing from a moment client has since the last. If client connection
disappears, we will still be able to communicate the outputs during a
reconnect.
Some follow-up changes after this change is merged:
- Support for `last_seen_seq`, `seq` number in the ExecResponse.
- Implementation of forking.
- Potential schema changes to conversation_log and execution_log.
- More README updates.
- Rewrite of the trace viewer tool with the new schema once the logs are
more stable.
- Controller will introduce its own OutputHandler to emit additional
fields, e.g. `seq`.
- Remove redundancy between conversation_log and execution_log.
This PR enforces a mandatory 5-second health check timeout when
registering new remote agents in the controller registry. It also
updates the `TestRegistry_GracefulShutdown` testing suite to properly
attach workload agents to a mock server.
### Implementation Detail
If an agent fails the health check (e.g. offline during initialization),
the registration is *not* rejected. Instead, the server registers the
agent with its `Healthy` flag set to `false`, logs a clear connection
`Warning`, and informs the CLI that it will be retried in the
background. This ensures that the AX Server can still successfully start
up and load its initial `ax.yaml` configuration even if some remote
agents are strictly offline at boot time.
### Validation
Verified the implementation successfully against the AX Server via `ax
register` CUJs. Here are the logs for the E2E verification:
**CLI Registration Output:**
```text
======================================
TEST 1: Registration (Remote agent offline)
=> Attempting to register an offline agent on port 50053...
Agent registered successfully, but represents an unreachable address at localhost:50053.
It will be retried in the background.
======================================
TEST 2: Registration (Remote agent online)
=> Attempting to register an online agent on port 50051...
Agent test-remote-online registered successfully.
```
**AX Server Logs Tracking Status:**
```text
2026/03/27 23:35:22 Starting AX server at :8494...
2026/03/27 23:35:24 Warning: remote agent test-remote-offline is unreachable or unhealthy during registration: health check failed: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:50053: connect: connection refused"
```
Interactions Content doesn't include a "role" and we introduced this
field in our own copy. Remove "role" from Content in preparation to
switch to the actual Contual message.
This change introduces AgentService.Connect RPC and replaces Process.
- Connect allows bidirectionally executing agents.
- Connect needs to be started with AgentStart and need to be followed
with AgentOutputs.
- Thought the repo, task is renamed to execution.
- SQLite schema is updated accordingly.
- agent.Task is removed because AgentStart represents the required
fields to start a new agent call.
We are replacing the current loop executor with a new internal execution
framework. This update refactors the majority of the project's core
logic to improve stability and performance.
We introduce a new Task Executor, moving away from simple
request-response agent loops toward a more structured, stateful
execution model. The core of this update is the addition of a Task
struct and a TaskExecutor interface, which allow agents to trigger
sub-tasks and manage complex workflows. This change enables better
recursion and delegation between different agents, such as the testing
coding agent and specialized worker agents. The new executor framework
supports confirmation propagation and leaves the confirmation handling
to the agent implementation.
The update introduces a new persistent Event Log system to track every
step of a task's execution. By recording inputs, outputs, and state
changes to a newline-delimited JSON file, the system can now "rehydrate"
sessions at any point. This allows the executor to resume from a
consistent state after a crash or restart, and provides a durable
history of agent actions.
The change refactors the Controller and various agent implementations to
support these new interfaces. It includes updates to the CLI trigger
tool to handle interactive confirmation prompts more effectively and
manage conversation history across task boundaries. Comprehensive tests
for the new Task Manager, including fan-out execution and resume
capabilities, are also included to ensure reliability in concurrent
environments.
The change also introduces a "gemini" agent to make it easier to make
Gemini calls without having to write any new code. We default to the
Flash models but can follow up with some configuration options to
override the default settings.
Trigger now also allows starting execution from any available agent, `$
trigger --agent <agent_id>`.
The new capabilities can be explorable and testable with the
internal/testagent.
Follow up changes after this change is merged:
- Fixing Gemini planner and skills support
- gar trigger will be rename to gar exec
- We will remove the session IDs and replace them with IDs/task IDs
- Exec over the wire, changes to AgentService
- Task log in SQLite
- Trace view tool to be able to inspect the logs
minor change to allow graceful shutdown for server
tested:
lhuan-mac:forkSession-cleanup lhuan$ ./gar serve
--dangerously-skip-approvals
2026/02/17 10:30:03 Starting GAR server at :8494...
^C2026/02/17 10:30:05
1. server.triggerSession - remove checkPointId, destinationSourceId from
this RPC.
3. cmd / trigger.go - correspondingly remove checkpointId,
destinationSourceId from gar trigger
4. user CUJ: user should explicitly "gar fork" then "gar trigger"
support forkSession from a specific checkpoint
tested with unitTest + manual test
```
lhuan-mac:forkSession lhuan$ ./gar trigger --input "test_fork" --session session678 --checkpoint 11c9717c-f273-4770-8755-5785bfae2b30
[STATE_RUNNING] TEST
[STATE_RUNNING] TEST
[STATE_RUNNING] TEST_FORK
```
Jaana Dogan