feat: support default-hidden experimental settings (#13980)

## Thinking Path

> - Paperclip is the open source control plane for AI-agent companies.
> - Operators can hide settings that their users must not change.
> - The server shares the effective restrictions with the UI and
settings API.
> - An explicit list must change whenever a new experimental flag is
added.
> - This pull request adds a wildcard with named exceptions to the
existing setting.
> - Core applies the policy to its current catalog, so new flags stay
hidden automatically.

## Linked Issues or Issue Description

Related: #11823 introduced settings visibility. #13907 added workspace
isolation visibility. I searched related PRs and issues and found no
duplicate wildcard implementation.

**What existing behavior does this improve?**

Operator control of experimental setting visibility through
`PAPERCLIP_HIDDEN_SETTINGS`.

**Subsystem affected**

Shared settings policy and its existing server health and mutation
consumers.

**Current behavior**

Operators must name every hidden experimental toggle. A new Core flag
can become visible until the operator updates that list.

**Proposed behavior**

`instance.experimental.*` hides current and future experimental toggles.
Entries such as `!instance.experimental.enableEnvironments` leave named
controls available. Explicit hidden keys and the hidden parent page take
precedence over exceptions.

**Reason and benefit**

Operators can maintain a short list of allowed controls instead of a
second copy of Core's full feature catalog.

**Breaking changes**

Existing explicit lists and unset configuration keep their behavior. The
new syntax is opt-in. Older images ignore it, so operators must retain
explicit restrictions until those images are upgraded. Visibility does
not change feature values.

## What Changed

- Expand the wildcard into concrete catalog keys in the shared parser.
- Limit exceptions to known experimental controls and preserve explicit
restrictions in either input order.
- Test a synthetic future catalog addition, duplicate and invalid
entries, API rejection, same-value echoes, and the effective health
payload.
- Document the syntax and the transition for deployments with mixed
image versions.

## Verification

- Targeted parser, future-catalog, health, and settings-route tests
pass: 95 tests across four files.
- `pnpm -r typecheck` passes, including Rust checks, with the installed
Cargo directory on PATH.
- `pnpm build` passes.
- All current-head CI gates pass, including the full test shards, Rust,
build, browser E2E, and canary dry run:
https://github.com/paperclipai/paperclip/actions/runs/36083292578. One
unchanged runtime-exposure cold-start test passed on its first retry.
- The full local `pnpm test:run` did not pass on macOS/Node 25: the
first server group reported 13,356 passed, 18 failed, and 99 skipped,
with six failed files (including two failed suite setups). Failures were
in unchanged runtime/company skill cache, chat/email connector fixtures,
embedded-Postgres setup, and workspace cleanup tests. A standalone
filesystem probe reproduced the read-only-directory rename permission
failure. Missing connector fixture paths, database startup failures, and
two integration assertions also occurred; the remaining local groups
were not reached after this group failed. The corresponding CI lanes all
pass. These local failures are not claimed as fixed by this PR.
- Browser suites were not run because this changes the shared policy,
not UI rendering or browser workflows. Health payload and route tests
cover the shared UI/API contract.

## Risks

A malformed exception remains hidden and is reported as unknown.
Exceptions cannot override an explicit hidden toggle or parent page.
Older images ignore wildcard syntax; keep their explicit list during a
mixed-version rollout. No schema or feature-value changes are included.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, tool use, and code
execution. The exact runtime variant and context-window size are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>

docs/deploy/environment-variables.md

@@ -100,6 +100,13 @@ Daytona snapshot for future leases.
 - Any experimental toggle: `instance.experimental.<flagKey>` (e.g.
   `instance.experimental.enableSmokeLab`) — the card disappears and
   value-changing writes are rejected.
+- All current and future experimental toggles: `instance.experimental.*`.
+  Add `!instance.experimental.<flagKey>` entries to leave specific controls
+  available. The server expands this policy against its own feature catalog,
+  so new toggles stay hidden without an environment change. The Experimental
+  page remains available. Exceptions only apply to the wildcard; an explicit
+  hidden toggle or `instance.experimental` page restriction always wins,
+  regardless of entry order. Unknown exceptions are logged and ignored.
 - Any top-level company settings page: `company.members`, `company.invites`,
   `company.secrets`, `company.export`, `company.import` — removed from the
   settings sidebar, tab bar, and routing (the company General page is the
@@ -132,6 +139,24 @@ is identical to earlier releases. Hiding a toggle does not change its value;
 pair hiding with the desired default where it matters (for general settings,
 see [Operator setting defaults](#operator-setting-defaults)).
 
+For example, this allows only the Environments control and keeps the Plugins
+settings page hidden:
+
+```sh
+PAPERCLIP_HIDDEN_SETTINGS='instance.plugins,instance.experimental.*,!instance.experimental.enableEnvironments'
+```
+
+`GET /api/health` returns the expanded concrete keys in `hiddenSettings`.
+The UI and settings API use the same restrictions. Reads and same-value
+echoes remain allowed; changing a hidden value returns
+`403 settings_operator_managed`.
+
+Older images that predate wildcard support ignore the wildcard and exceptions.
+Keep their explicit hidden-toggle entries during an upgrade, or upgrade all
+images before replacing an explicit list. Once every image supports this
+syntax, the wildcard and its exceptions are sufficient. A recognized exception
+without a wildcard has no effect.
+
 ### Operator setting defaults
 
 `PAPERCLIP_SETTING_DEFAULTS` takes a JSON object whose fields come from the

packages/shared/src/index.ts

@@ -2675,6 +2675,7 @@ export {
   type InstanceFeatureKey,
 } from "./feature-catalog.js";
 export {
+  EXPERIMENTAL_SETTINGS_WILDCARD,
   HIDEABLE_COMPANY_PAGES,
   HIDEABLE_COMPANY_SECTIONS,
   HIDEABLE_GENERAL_SECTIONS,

packages/shared/src/settings-visibility-catalog-growth.test.ts

@@ -0,0 +1,17 @@
+import { expect, it, vi } from "vitest";
+
+// Model a future Core release without changing the operator's policy.
+vi.mock("./feature-catalog.js", async (importOriginal) => {
+  const catalog = await importOriginal<typeof import("./feature-catalog.js")>();
+  return { ...catalog, INSTANCE_FEATURE_KEYS: [...catalog.INSTANCE_FEATURE_KEYS, "enableFutureFeature"] };
+});
+
+import { parseHiddenSettingsList } from "./settings-visibility.js";
+
+it("hides an added catalog feature without an operator configuration change", () => {
+  const parsed = parseHiddenSettingsList("instance.plugins,instance.experimental.*,!instance.experimental.enableEnvironments");
+  expect(parsed.unknown).toEqual([]);
+  expect(parsed.hidden).toContain("instance.experimental.enableFutureFeature");
+  expect(parsed.hidden).toContain("instance.plugins");
+  expect(parsed.hidden).not.toContain("instance.experimental.enableEnvironments");
+});

packages/shared/src/settings-visibility.test.ts

@@ -49,6 +49,48 @@ describe("hideable setting keys", () => {
 });
 
 describe("parseHiddenSettingsList", () => {
+  it("expands the experimental wildcard into concrete keys without hiding the page", () => {
+    const parsed = parseHiddenSettingsList("instance.experimental.*");
+    expect(parsed).toEqual({ hidden: INSTANCE_FEATURE_KEYS.map(experimentalSettingKey), unknown: [] });
+    expect(parsed.hidden).not.toContain("instance.experimental");
+  });
+
+  it("allows only named exceptions, independent of order or duplicates", () => {
+    const exception = "!instance.experimental.enableEnvironments";
+    for (const raw of [`instance.experimental.*, ${exception}, ${exception}`, `${exception},instance.experimental.*`]) {
+      const parsed = parseHiddenSettingsList(raw);
+      expect(parsed).toEqual({
+        hidden: INSTANCE_FEATURE_KEYS.filter((key) => key !== "enableEnvironments").map(experimentalSettingKey),
+        unknown: [],
+      });
+    }
+  });
+
+  it("keeps explicit hides and parent page restrictions stronger than exceptions", () => {
+    for (const restriction of ["instance.experimental.enableEnvironments", "instance.experimental"]) {
+      for (const raw of [
+        `instance.experimental.*,!instance.experimental.enableEnvironments,${restriction}`,
+        `${restriction},!instance.experimental.enableEnvironments,instance.experimental.*`,
+      ]) {
+        const { hidden } = parseHiddenSettingsList(raw);
+        expect(hidesExperimentalSetting(new Set(hidden), "enableEnvironments")).toBe(true);
+        expect(new Set(hidden).size).toBe(hidden.length);
+      }
+    }
+  });
+
+  it("ignores unknown or out-of-scope exceptions without opening any controls", () => {
+    const parsed = parseHiddenSettingsList("instance.experimental.*,!instance.experimental.enableTypo,!instance.plugins,!instance.experimental,!instance.experimental.*");
+    expect(parsed.hidden).toEqual(INSTANCE_FEATURE_KEYS.map(experimentalSettingKey));
+    expect(parsed.unknown).toEqual(["!instance.experimental.enableTypo", "!instance.plugins", "!instance.experimental", "!instance.experimental.*"]);
+  });
+
+  it("does not use exceptions to override individual restrictions without a wildcard", () => {
+    expect(parseHiddenSettingsList("!instance.experimental.enableEnvironments").hidden).toEqual([]);
+    expect(parseHiddenSettingsList("instance.experimental.enableEnvironments,!instance.experimental.enableEnvironments").hidden)
+      .toEqual(["instance.experimental.enableEnvironments"]);
+  });
+
   it("accepts workspace controls independently of experimental flags", () => {
     expect(parseHiddenSettingsList("workspaces.isolation")).toEqual({ hidden: ["workspaces.isolation"], unknown: [] });
     expect(hidesExperimentalSetting(new Set(["workspaces.isolation"]), "enableIsolatedWorkspaces")).toBe(false);

packages/shared/src/settings-visibility.ts

@@ -6,7 +6,8 @@ import { INSTANCE_FEATURE_KEYS, type InstanceFeatureKey } from "./feature-catalo
  * A hosting operator (a managed cloud, an internal shared server) can hide
  * settings surfaces that do not apply to their deployment by setting the
  * `PAPERCLIP_HIDDEN_SETTINGS` environment variable to a comma-separated
- * list of keys from this registry. Hiding a surface removes it from the UI
+ * list of keys from this registry. An experimental wildcard with named
+ * exceptions can also hide future controls automatically. Hiding a surface removes it from the UI
  * (nav, routes, page sections). Surfaces backed by instance-level mutation
  * routes are also floored with a 403 carrying
  * `SETTINGS_OPERATOR_MANAGED_ERROR_CODE`: the Access, Plugins, and Adapters
@@ -111,7 +112,7 @@ export type HideableSettingKey =
   | HideableGeneralSection
   | HideableExperimentalSetting;
 
-/** Every key `PAPERCLIP_HIDDEN_SETTINGS` accepts. */
+/** Concrete setting keys; the parser also accepts the experimental wildcard and exceptions. */
 export const HIDEABLE_SETTING_KEYS: readonly HideableSettingKey[] = [
   ...HIDEABLE_WORKSPACE_SECTIONS,
   ...HIDEABLE_INSTANCE_PAGES,
@@ -124,30 +125,50 @@ export const HIDEABLE_SETTING_KEYS: readonly HideableSettingKey[] = [
 /** Stable 403 code for writes to operator-hidden settings. */
 export const SETTINGS_OPERATOR_MANAGED_ERROR_CODE = "settings_operator_managed";
 
+/** Hide current and future experimental controls, with optional !key exceptions. */
+export const EXPERIMENTAL_SETTINGS_WILDCARD = "instance.experimental.*";
+
 export interface ParsedHiddenSettings {
-  /** Recognized keys, deduplicated, in input order. */
+  /** Concrete keys, deduplicated; wildcard-derived keys follow in catalog order. */
   hidden: HideableSettingKey[];
   /** Unrecognized entries, for the caller to warn about. */
   unknown: string[];
 }
 
-/** Parse a `PAPERCLIP_HIDDEN_SETTINGS`-style comma-separated list. */
+/**
+ * Parse operator settings into concrete keys for both the UI and API.
+ * `instance.experimental.*` hides every catalog control except entries such as
+ * `!instance.experimental.enableEnvironments`. Exceptions only affect the
+ * wildcard; an explicit hidden key or hidden parent page always wins.
+ */
 export function parseHiddenSettingsList(raw: string | undefined): ParsedHiddenSettings {
   const hidden: HideableSettingKey[] = [];
   const unknown: string[] = [];
   if (!raw) return { hidden, unknown };
   const known = new Set<string>(HIDEABLE_SETTING_KEYS);
   const seen = new Set<string>();
+  const exceptions = new Set<string>();
+  let hideExperimental = false;
   for (const part of raw.split(",")) {
     const key = part.trim();
     if (!key || seen.has(key)) continue;
     seen.add(key);
-    if (known.has(key)) {
+    if (key === EXPERIMENTAL_SETTINGS_WILDCARD) {
+      hideExperimental = true;
+    } else if (key.startsWith("!instance.experimental.") && known.has(key.slice(1))) {
+      exceptions.add(key.slice(1));
+    } else if (known.has(key)) {
       hidden.push(key as HideableSettingKey);
     } else {
       unknown.push(key);
     }
   }
+  if (hideExperimental) {
+    for (const feature of INSTANCE_FEATURE_KEYS) {
+      const key = experimentalSettingKey(feature);
+      if (!exceptions.has(key) && !seen.has(key)) hidden.push(key);
+    }
+  }
   return { hidden, unknown };
 }
 

server/src/__tests__/health.test.ts

@@ -136,6 +136,22 @@ describe("GET /health", () => {
     expect(Object.prototype.hasOwnProperty.call(res.body, "hiddenSettings")).toBe(false);
   });
 
+  it("publishes concrete wildcard restrictions to the UI and refreshes changed exceptions", async () => {
+    const env = { PAPERCLIP_HIDDEN_SETTINGS: "instance.plugins,instance.experimental.*,!instance.experimental.enableEnvironments" };
+    const app = createApp(undefined, testServerInfo, undefined, env);
+    const first = await request(app).get("/health");
+    expect(first.status).toBe(200);
+    expect(first.body.hiddenSettings).toContain("instance.plugins");
+    expect(first.body.hiddenSettings).toContain("instance.experimental.enableMemoryConnectors");
+    expect(first.body.hiddenSettings).not.toContain("instance.experimental.enableEnvironments");
+    expect(first.body.hiddenSettings.some((key: string) => key.includes("*") || key.startsWith("!"))).toBe(false);
+
+    env.PAPERCLIP_HIDDEN_SETTINGS = "instance.experimental.*,!instance.experimental.enableMemoryConnectors";
+    const second = await request(app).get("/health");
+    expect(second.body.hiddenSettings).toContain("instance.experimental.enableEnvironments");
+    expect(second.body.hiddenSettings).not.toContain("instance.experimental.enableMemoryConnectors");
+  });
+
   it("returns 200 when the database probe succeeds", async () => {
     const db = {
       execute: vi.fn().mockResolvedValue([{ "?column?": 1 }]),

server/src/__tests__/instance-settings-routes.test.ts

@@ -853,6 +853,40 @@ describe("instance settings routes", () => {
       expect(mockInstanceSettingsService.updateExperimental).not.toHaveBeenCalled();
     });
 
+    it("enforces a wildcard allowlist at the API and preserves hidden values", async () => {
+      process.env.PAPERCLIP_HIDDEN_SETTINGS =
+        "instance.experimental.*,!instance.experimental.enableIsolatedWorkspaces";
+      const app = await createApp(adminActor);
+
+      const rejected = await request(app)
+        .patch("/api/instance/settings/experimental")
+        .send({ enableEnvironments: true, enableIsolatedWorkspaces: true });
+      expect(rejected.status).toBe(403);
+      expect(rejected.body.details).toMatchObject({ code: "settings_operator_managed" });
+      expect(mockInstanceSettingsService.updateExperimental).not.toHaveBeenCalled();
+
+      // Existing full-form clients may echo hidden values without changing them.
+      const allowed = await request(app)
+        .patch("/api/instance/settings/experimental")
+        .send({ enableEnvironments: false, enableIsolatedWorkspaces: true });
+      expect(allowed.status).toBe(200);
+      expect(mockInstanceSettingsService.updateExperimental).toHaveBeenCalledWith({
+        enableEnvironments: false, enableIsolatedWorkspaces: true,
+      });
+    });
+
+    it("does not let an allowlist exception bypass an explicit API restriction", async () => {
+      process.env.PAPERCLIP_HIDDEN_SETTINGS =
+        "instance.experimental.*,!instance.experimental.enableEnvironments,instance.experimental.enableEnvironments";
+      const app = await createApp(adminActor);
+      const res = await request(app)
+        .patch("/api/instance/settings/experimental")
+        .send({ enableEnvironments: true });
+      expect(res.status).toBe(403);
+      expect(res.body.details).toMatchObject({ code: "settings_operator_managed" });
+      expect(mockInstanceSettingsService.updateExperimental).not.toHaveBeenCalled();
+    });
+
     it("allows writes to non-hidden experimental toggles while others are hidden", async () => {
       process.env.PAPERCLIP_HIDDEN_SETTINGS =
         "instance.experimental.enableEnvironments,instance.experimental.enableServerInfoDebugView";