Mesut Oezdil
7174983ff1
* fix(sandbox): add mechanistic smoke test for L4 deny and document the L4/L7 split The old smoke script exercised an L7 PUT which hung because the denial aggregator is only wired to L4 CONNECT denies, not L7 enforcement. Add mechanistic-smoke.sh which triggers an L4 deny, waits for the aggregator to flush, and asserts a pending chunk appears under openshell rule get --status pending. Document the intentional L4-only scope of the mechanistic mapper in architecture/sandbox.md. Fixes #1333 Signed-off-by: mesutoezdil <mesudozdil@gmail.com> * refactor(smoke): remove redundant variable inits and merge double step call Signed-off-by: mesutoezdil <mesudozdil@gmail.com> * fix(smoke): wire mechanistic smoke into mise and guard TMP_DIR - Initialize TMP_DIR before trap to prevent unbound variable on early exit - Add e2e:mechanistic-smoke mise task with gateway setup - Document mechanistic smoke in policy-advisor README * test(proxy): verify L4 deny enqueues a DenialEvent Signed-off-by: mesutoezdil <mesudozdil@gmail.com> * fix(proxy): remove unnecessary path qualifications in L4 denial smoke test --------- Signed-off-by: mesutoezdil <mesudozdil@gmail.com>
2.2 KiB
Policy Advisor end-to-end test
Deterministic, no-LLM exercise of the agent-driven policy loop:
- Start a sandbox with a read-only GitHub L7 policy.
- From inside the sandbox, attempt a GitHub contents PUT and assert OpenShell
returns a structured
policy_denied403. - Submit a narrow
addRuleproposal throughhttp://policy.local/v1/proposals. - Approve the draft from the host and retry until the write succeeds.
This proves the proxy, the structured deny body, the policy.local HTTP API,
the gateway proposal path, and the hot-reload of approved rules — without
involving an LLM. The user-facing demo (examples/agent-driven-policy-management/)
runs the same loop with Codex driving from inside the sandbox.
Run it
Run against an ephemeral Docker gateway:
DEMO_GITHUB_OWNER=<your-handle> \
DEMO_GITHUB_REPO=openshell-policy-demo \
e2e/with-docker-gateway.sh bash -lc '
target/debug/openshell settings set --global \
--key agent_policy_proposals_enabled \
--value true \
--yes
OPENSHELL_BIN="$PWD/target/debug/openshell" bash e2e/policy-advisor/test.sh
'
To keep the sandbox for debugging, start a local gateway first with
mise run gateway:docker, then run:
target/debug/openshell settings set --global \
--key agent_policy_proposals_enabled \
--value true \
--yes
OPENSHELL_GATEWAY=docker-dev \
OPENSHELL_BIN="$PWD/target/debug/openshell" \
DEMO_KEEP_SANDBOX=1 \
DEMO_GITHUB_OWNER=<your-handle> \
DEMO_GITHUB_REPO=openshell-policy-demo \
bash e2e/policy-advisor/test.sh
Requires Docker, agent_policy_proposals_enabled=true, and a GitHub token with
contents write on the repository. The test auto-resolves the token from
DEMO_GITHUB_TOKEN, GITHUB_TOKEN, GH_TOKEN, or gh auth token.
Mechanistic smoke
Lightweight regression for the L4 CONNECT deny → mechanistic chunk pipeline. No GitHub token or LLM required.
mise run e2e:mechanistic-smoke
Or manually against a running gateway with agent_policy_proposals_enabled=true:
OPENSHELL_BIN=target/debug/openshell bash e2e/policy-advisor/mechanistic-smoke.sh