fix(conformance): address Windows review feedback

Signed-off-by: Evan Lezar <elezar@nvidia.com>

.agents/skills/build-openshell-mxc-windows/SKILL.md

@@ -50,6 +50,8 @@ In scope:
 - Building x64 and ARM64 release binaries for `openshell-gateway` and
   `openshell`.
 - Running workspace tests on a native x64 or ARM64 host.
+- Running the complete conformance suite against the mock MXC gateway and
+  reporting unsupported operations without blocking hosted Windows CI.
 - Running focused unsupported-driver contract tests.
 - Running the shipped Ollama and cloud-inference MXC examples against the
   in-process `wxc` mock and a local API stub.
@@ -161,6 +163,7 @@ mise run --skip-tools windows:check:arm64
 mise run --skip-tools windows:build:x64
 mise run --skip-tools windows:build:arm64
 mise run --skip-tools windows:test:x64
+mise run --skip-tools windows:test:conformance:x64
 mise run --skip-tools windows:test:unsupported:x64
 mise run --skip-tools windows:e2e:mxc:inference-mock:x64
 mise run --skip-tools windows:e2e:mxc:provider-mock
@@ -247,12 +250,13 @@ release binaries, and run the inference, provider-credential, OCSF audit, and
 aggregate shipped examples through their mock tasks. Pushes to `main` and
 manual dispatches run the same lint and test commands in a cache-seed job,
 followed by a dependent release-binary build and mock-example job. The seed and
-PR jobs use the same cache namespaces. Merge
-queues do not run this workflow. Main/manual seed and build jobs use job-level
-`continue-on-error: true`; opt-in PR jobs report failures normally. Applying
-the label alone does not start a run: re-run all jobs in the current mirror
-push run, or push a new mirrored commit. The binaries are not uploaded or
-published.
+PR jobs use the same cache namespaces. Merge queues do not run this workflow.
+Main/manual seed and build jobs use job-level `continue-on-error: true`;
+opt-in PR jobs report failures normally, except that the full mock-MXC
+conformance step is advisory on every hosted lane so known capability gaps are
+archived without blocking the job. Applying the label alone does not start a
+run: re-run all jobs in the current mirror push run, or push a new mirrored
+commit. The binaries are not uploaded or published.
 
 The ARM64 check/build steps in this x64-host contract are cross-builds. The
 wrapper discovers and adds host-native LLVM and Ninja to `PATH`, requires the
@@ -292,6 +296,8 @@ crypto dependency builds.
 | `windows:build:arm64` | Release-builds `openshell-gateway.exe`, `openshell.exe`, and `openshell-supervisor-relay.exe` for ARM64. |
 | `windows:test:x64` | Runs native x64 workspace tests with `--no-fail-fast`, excluding unsupported Windows packages as top-level workspace targets. |
 | `windows:test:arm64` | Runs native ARM64 workspace tests with `--no-fail-fast` and the same package exclusions. Rejects non-ARM64 hosts. |
+| `windows:test:conformance:x64` | Runs every conformance scenario against the mock MXC gateway on native x64, then exits nonzero if any scenario failed. Hosted CI treats that result as advisory and uploads the complete report. |
+| `windows:test:conformance:arm64` | Runs the same complete mock-MXC report natively on ARM64. Rejects non-ARM64 hosts. Hosted CI treats scenario failures as advisory. |
 | `windows:test:unsupported:x64` | Re-runs focused `openshell-gateway` tests for unsupported Windows driver behavior. |
 | `windows:test:unsupported:arm64` | Re-runs the same focused contracts natively on ARM64. Rejects non-ARM64 hosts. |
 | `windows:test:mxc-real:x64` | Runs the serial, ignored real-`wxc-exec` integration suite natively on x64 through the MSVC wrapper. Rejects non-x64 hosts. |
@@ -356,6 +362,9 @@ When reporting `windows:ci`, distinguish these categories:
 - The focused unsupported-contract re-run.
 - The architecture-matched MXC inference, provider-credential, OCSF audit, and
   aggregate mock tasks and their explicit wiring-only limitation.
+- The complete mock-MXC conformance report, separating passed scenarios from
+  expected capability gaps. Do not report an advisory hosted run as full
+  conformance.
 - Explicit Cargo ignored tests, usually ignored doc examples.
 - Tests hidden by `#[cfg(not(target_os = "windows"))]`; these often appear as
   `running 0 tests`, not as ignored tests.
@@ -373,6 +382,11 @@ Useful log files:
 | `build-aarch64-pc-windows-msvc-release.log` | ARM64 release build output. |
 | `test-x86_64-pc-windows-msvc.log` | Full native x64 workspace test output. |
 | `test-aarch64-pc-windows-msvc.log` | Full native ARM64 workspace test output. |
+| `build-<target>-conformance.log` | Build output for the CLI, gateway, and standalone conformance runner. |
+| `test-<target>-conformance.log` | Complete mock-MXC conformance scenario report. |
+| `test-<target>-conformance.err.log` | Conformance runner standard error. |
+| `test-<target>-conformance-gateway.log` | Mock-MXC gateway standard output. |
+| `test-<target>-conformance-gateway.err.log` | Mock-MXC gateway standard error. |
 | `test-x86_64-pc-windows-msvc-unsupported-*.log` | Focused unsupported-driver contract output. |
 | `test-aarch64-pc-windows-msvc-unsupported-*.log` | Focused native ARM64 contract output. |
 | `test-x86_64-pc-windows-msvc-mxc-real.log` | Native x64 real-MXC integration output. |

.agents/skills/build-openshell-mxc-windows/reference.md

@@ -30,6 +30,8 @@ mise run --skip-tools windows:build:x64
 mise run --skip-tools windows:build:arm64
 mise run --skip-tools windows:test:x64
 mise run --skip-tools windows:test:arm64
+mise run --skip-tools windows:test:conformance:x64
+mise run --skip-tools windows:test:conformance:arm64
 mise run --skip-tools windows:test:unsupported:x64
 mise run --skip-tools windows:test:unsupported:arm64
 mise run --skip-tools windows:ci
@@ -53,6 +55,12 @@ if ($arch -eq [System.Runtime.InteropServices.Architecture]::Arm64) {
 The native test tasks reject a target that does not match the host architecture.
 Do not report x64 compatibility-under-emulation coverage from an ARM64 run.
 
+The conformance tasks start a temporary gateway with the in-process mock MXC
+backend and run every atomic scenario through the standalone conformance
+runner. The task exits nonzero when any scenario fails so local runs retain an
+accurate result. Hosted Windows CI marks only this step as advisory and uploads
+the runner and gateway logs even when known MXC capability gaps fail.
+
 The wrapper adds missing rustup targets and preserves an inherited
 `RUSTC_WRAPPER` when the command is available. Otherwise, it warns and clears
 the setting. It does not install Visual Studio, Rust, Docker, Kubernetes,
@@ -198,6 +206,9 @@ Use the log summaries from:
 |---|---|
 | `test-x86_64-pc-windows-msvc.log` | Full x64 workspace test pass. |
 | `test-aarch64-pc-windows-msvc.log` | Full native ARM64 workspace test pass. |
+| `test-<target>-conformance.log` | Complete mock-MXC conformance report. |
+| `test-<target>-conformance.err.log` | Conformance runner standard error. |
+| `test-<target>-conformance-gateway*.log` | Mock-MXC gateway output and errors. |
 | `test-x86_64-pc-windows-msvc-unsupported-*.log` | Focused unsupported-contract re-runs and filtered counts. |
 | `test-aarch64-pc-windows-msvc-unsupported-*.log` | Focused native ARM64 re-runs and filtered counts. |
 
@@ -209,6 +220,7 @@ Separate:
 - filtered out
 - cfg-gated zero-test targets
 - package-level excludes
+- conformance scenarios that pass versus capability gaps that fail
 
 Package-level excludes are not printed as ignored tests by Cargo.
 

.github/workflows/windows-msvc.yml

@@ -99,6 +99,7 @@ jobs:
       - name: Test
         run: mise run --skip-tools windows:test:${{ matrix.arch }}
       - name: Full MXC conformance
+        continue-on-error: true
         run: mise run --skip-tools windows:test:conformance:${{ matrix.arch }}
       - name: Upload MXC conformance report
         if: always()
@@ -185,6 +186,7 @@ jobs:
       - name: Test
         run: mise run --skip-tools windows:test:${{ matrix.arch }}
       - name: Full MXC conformance
+        continue-on-error: true
         run: mise run --skip-tools windows:test:conformance:${{ matrix.arch }}
       - name: Upload MXC conformance report
         if: always()

CONTRIBUTING.md

@@ -385,6 +385,7 @@ install the Windows compiler toolchain.
 | `windows:lint:<x64\|arm64>` | Run Clippy for the Windows-supported workspace on the selected target architecture. |
 | `windows:build:<x64\|arm64>` | Build the three Windows release executables. |
 | `windows:test:<x64\|arm64>` | Run the workspace suite natively; the target must match the host architecture. |
+| `windows:test:conformance:<x64\|arm64>` | Run the complete conformance suite against the mock MXC gateway and report every unsupported operation. The task exits nonzero when any scenario fails; hosted CI treats the report as advisory and always uploads its logs. |
 | `windows:test:unsupported:<x64\|arm64>` | Run the focused unsupported-driver contracts. |
 | `windows:test:mxc-real:<x64\|arm64>` | Run the probe-gated real-`wxc-exec` integration suite on the matching host. |
 | `windows:artifacts` | Report sizes and SHA256 hashes for release artifacts. |

architecture/windows.md

@@ -249,7 +249,10 @@ Windows validation separates source correctness from host capability:
 - Hosted Windows CI runs the complete archive conformance suite against the
   in-process MXC mock through the standalone `openshell-conformance` runner.
   Every atomic scenario runs so unsupported operations appear as explicit
-  failures in the archived report instead of being skipped.
+  failures in the archived report instead of being skipped. This report is
+  advisory while those capability gaps remain: its scenario failures do not
+  fail the surrounding Windows job, and CI always uploads the runner and
+  gateway logs.
   The mock verifies public CLI, gateway, driver, lifecycle, and policy wiring;
   it does not claim Windows OS enforcement.
 - Mock MXC E2E validates gateway, CLI, driver, lifecycle, and policy wiring but

crates/openshell-conformance/src/scenarios/sandbox_lifecycle.rs

@@ -21,6 +21,12 @@ struct SandboxState {
     phase: String,
 }
 
+#[derive(Debug, Deserialize)]
+struct SandboxListPage {
+    sandboxes: Vec<SandboxState>,
+    next_page_token: String,
+}
+
 /// Certify sandbox stop, start, and deletion lifecycle behavior.
 pub const SANDBOX_LIFECYCLE_SCENARIO: Scenario = Scenario {
     name: "sandbox/lifecycle",
@@ -246,22 +252,60 @@ async fn wait_for_absence(
             &poll_step,
             TRANSITION_TIMEOUT,
             TRANSITION_INTERVAL,
-            async move |runner| {
-                let result = runner
-                    .step(format!("{step}/get"))
-                    .description(format!("sandbox '{sandbox_name}' is no longer retrievable"))
-                    .with_timeout(COMMAND_TIMEOUT)
-                    .run(&["sandbox", "get", &sandbox_name, "--output", "json"])
-                    .await;
-                match result {
-                    Ok(result) if !result.success() => Poll::Ready(()),
-                    Ok(_) => {
-                        Poll::Pending(format!("sandbox '{sandbox_name}' is still retrievable"))
-                    }
-                    Err(error) => Poll::Pending(error.to_string()),
-                }
+            async move |runner| match sandbox_is_listed(runner, &sandbox_name, &step).await {
+                Ok(false) => Poll::Ready(()),
+                Ok(true) => Poll::Pending(format!("sandbox '{sandbox_name}' is still retrievable")),
+                Err(error) => Poll::Pending(error),
             },
         )
         .await
         .map_err(|error| error.to_string())
 }
+
+async fn sandbox_is_listed(
+    runner: &OpenShellRunner,
+    sandbox_name: &str,
+    step: &str,
+) -> Result<bool, String> {
+    let mut page_token = String::new();
+    let mut page = 0u32;
+    loop {
+        let result = runner
+            .step(format!("{step}/list/{page}"))
+            .description(format!(
+                "sandbox list confirms whether '{sandbox_name}' still exists"
+            ))
+            .with_timeout(COMMAND_TIMEOUT)
+            .run(&[
+                "sandbox",
+                "list",
+                "--page-size",
+                "1000",
+                "--page-token",
+                &page_token,
+                "--output",
+                "json",
+            ])
+            .await
+            .map_err(|error| error.to_string())?;
+        result.require_success()?;
+
+        let response = result
+            .json::<SandboxListPage>()
+            .map_err(|error| error.to_string())?;
+        if response
+            .sandboxes
+            .iter()
+            .any(|sandbox| sandbox.name == sandbox_name)
+        {
+            return Ok(true);
+        }
+        if response.next_page_token.is_empty() {
+            return Ok(false);
+        }
+        page_token = response.next_page_token;
+        page = page
+            .checked_add(1)
+            .ok_or_else(|| "sandbox list page counter overflowed".to_string())?;
+    }
+}

tasks/windows.toml

@@ -56,12 +56,12 @@ run = "echo 'windows:* tasks require a Windows MSVC host' && exit 1"
 run_windows = "powershell -NoProfile -ExecutionPolicy Bypass -File tasks/scripts/windows-msvc.ps1 test-precommit aarch64-pc-windows-msvc"
 
 ["windows:test:conformance:x64"]
-description = "Run all openshell-conformance scenarios against a mock MXC gateway on native Windows x64"
+description = "Report all openshell-conformance scenarios against a mock MXC gateway on native Windows x64"
 run = "echo 'windows:* tasks require a Windows MSVC host' && exit 1"
 run_windows = "powershell -NoProfile -ExecutionPolicy Bypass -File tasks/scripts/windows-msvc.ps1 test-conformance x86_64-pc-windows-msvc"
 
 ["windows:test:conformance:arm64"]
-description = "Run all openshell-conformance scenarios against a mock MXC gateway on native Windows ARM64"
+description = "Report all openshell-conformance scenarios against a mock MXC gateway on native Windows ARM64"
 run = "echo 'windows:* tasks require a Windows MSVC host' && exit 1"
 run_windows = "powershell -NoProfile -ExecutionPolicy Bypass -File tasks/scripts/windows-msvc.ps1 test-conformance aarch64-pc-windows-msvc"