John T. MyersandJohn Myers
679fe4c334
* fix(policy): bind reviews to applicable candidates Build and validate the exact effective-policy candidate before approval, bind review to live policy/provider/credential inputs, and preserve inspected endpoint contracts during mechanistic expansion. Closes #2821 Signed-off-by: John Myers <johntmyers@users.noreply.github.com> * fix(policy): canonicalize advisor review inputs Serialize nested protobuf maps in stable key order for proposal review tokens and effective-policy hashes. Narrow reused multi-port endpoint contracts to the denied port so advisor proposals cannot widen binary access. Add regressions for both cases. Signed-off-by: John Myers <johntmyers@users.noreply.github.com> * test(e2e): keep advisor sandbox running Create the issue 2821 regression sandbox detached with a durable canonical main process so policy denial, approval, and hot-reload checks run before lifecycle exit. Signed-off-by: John Myers <johntmyers@users.noreply.github.com> * fix(policy): apply reviewed draft batches atomically Signed-off-by: John Myers <johntmyers@users.noreply.github.com> --------- Signed-off-by: John Myers <johntmyers@users.noreply.github.com> Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
2.5 KiB
Policy Advisor end-to-end test
Deterministic, no-LLM exercise of the agent-driven policy loop:
- Start a sandbox with a read-only GitHub L7 policy.
- From inside the sandbox, attempt a GitHub contents PUT and assert OpenShell
returns a structured
policy_denied403. - Submit a narrow
addRuleproposal throughhttp://policy.local/v1/proposals. - Approve the draft from the host and retry until the write succeeds.
This proves the proxy, the structured deny body, the policy.local HTTP API,
the gateway proposal path, and the hot-reload of approved rules — without
involving an LLM. The user-facing demo (examples/agent-driven-policy-management/)
runs the same loop with Codex driving from inside the sandbox.
Run it
Run against an ephemeral Docker gateway:
DEMO_GITHUB_OWNER=<your-handle> \
DEMO_GITHUB_REPO=openshell-policy-demo \
e2e/with-docker-gateway.sh bash -lc '
target/debug/openshell settings set --global \
--key agent_policy_proposals_enabled \
--value true \
--yes
OPENSHELL_BIN="$PWD/target/debug/openshell" bash e2e/policy-advisor/test.sh
'
To keep the sandbox for debugging, start a local gateway first with
mise run gateway:docker, then run:
target/debug/openshell settings set --global \
--key agent_policy_proposals_enabled \
--value true \
--yes
OPENSHELL_GATEWAY=docker-dev \
OPENSHELL_BIN="$PWD/target/debug/openshell" \
DEMO_KEEP_SANDBOX=1 \
DEMO_GITHUB_OWNER=<your-handle> \
DEMO_GITHUB_REPO=openshell-policy-demo \
bash e2e/policy-advisor/test.sh
Requires Docker, agent_policy_proposals_enabled=true, and a GitHub token with
contents write on the repository. The test auto-resolves the token from
DEMO_GITHUB_TOKEN, GITHUB_TOKEN, GH_TOKEN, or gh auth token.
Mechanistic smoke
Lightweight regression for the L4 CONNECT deny → mechanistic chunk pipeline. No GitHub token or LLM required.
mise run e2e:mechanistic-smoke
Or manually against a running gateway with agent_policy_proposals_enabled=true:
OPENSHELL_BIN=target/debug/openshell bash e2e/policy-advisor/mechanistic-smoke.sh
The #2821 regression additionally verifies that a denial on an existing inspected endpoint becomes a binary expansion, auto-approves, hot-reloads, and does not downgrade the endpoint to L4:
mise run e2e:mechanistic-existing-endpoint