OpenShell/e2e/policy-advisor

Policy Advisor end-to-end test

Deterministic, no-LLM exercise of the agent-driven policy loop:

1. Start a sandbox with a read-only GitHub L7 policy.
2. From inside the sandbox, attempt a GitHub contents PUT and assert OpenShell returns a structured policy_denied 403.
3. Submit a narrow addRule proposal through http://policy.local/v1/proposals.
4. Approve the draft from the host and retry until the write succeeds.

This proves the proxy, the structured deny body, the policy.local HTTP API, the gateway proposal path, and the hot-reload of approved rules — without involving an LLM. The user-facing demo (examples/agent-driven-policy-management/) runs the same loop with Codex driving from inside the sandbox.

Run it

Run against an ephemeral Docker gateway:

DEMO_GITHUB_OWNER=<your-handle> \
DEMO_GITHUB_REPO=openshell-policy-demo \
e2e/with-docker-gateway.sh bash -lc '
  target/debug/openshell settings set --global \
    --key agent_policy_proposals_enabled \
    --value true \
    --yes
  OPENSHELL_BIN="$PWD/target/debug/openshell" bash e2e/policy-advisor/test.sh
'

To keep the sandbox for debugging, start a local gateway first with mise run gateway:docker, then run:

target/debug/openshell settings set --global \
  --key agent_policy_proposals_enabled \
  --value true \
  --yes

OPENSHELL_GATEWAY=docker-dev \
OPENSHELL_BIN="$PWD/target/debug/openshell" \
DEMO_KEEP_SANDBOX=1 \
DEMO_GITHUB_OWNER=<your-handle> \
DEMO_GITHUB_REPO=openshell-policy-demo \
bash e2e/policy-advisor/test.sh

Requires Docker, agent_policy_proposals_enabled=true, and a GitHub token with contents write on the repository. The test auto-resolves the token from DEMO_GITHUB_TOKEN, GITHUB_TOKEN, GH_TOKEN, or gh auth token.

Conformance coverage

The policy-advisor/mechanistic-proposal and policy-advisor/new-hostname-proposal conformance scenarios check draft generation for a denied IP address and for a hostname absent from policy. The policy-advisor/sandbox-local scenario uses policy.local to inspect policy, submit a narrow permission request, and read the resulting proposal. Run them against a configured gateway with --openshell-bin pointing to the CLI under test:

openshell-conformance run \
  policy-advisor/mechanistic-proposal \
  policy-advisor/new-hostname-proposal \
  policy-advisor/sandbox-local \
  --openshell-bin target/debug/openshell

Run openshell-conformance list to see all scenario names. A manual Integration Tests workflow run can select the policy-advisor testsuite to run only these three scenarios against an installed candidate. Set artifact-run-id to the candidate build's workflow run ID and test-matrix to:

[{"environment":"ubuntu-docker-rootful","installer":"binaries","testsuite":"policy-advisor"}]

The GitHub write test above and the regressions below still exercise distinct proposal review, approval, and hot-reload behavior.

The #2821 regression additionally verifies that a denial on an existing inspected endpoint becomes a binary expansion, auto-approves, hot-reloads, and does not downgrade the endpoint to L4:

mise run e2e:mechanistic-existing-endpoint