fix: repair lessons, harden security, and gate lesson tests and quiz bias (#480)

* fix(phase-13/06): bootstrap sys.path so the documented test command runs The lesson doc says to run unittest discovery over code/tests, but the test imported main with no path setup, so discovery from the lesson root failed with ImportError. Insert the code directory on sys.path the way the later protocol lessons already do. Passes from the lesson root and from code/. * fix(phase-13/07): bootstrap sys.path so the documented test command runs Discovery over code/tests failed with ImportError because the test imported main with no path setup. Insert the code directory on sys.path to match the later protocol lessons. * fix(phase-13/08): bootstrap sys.path so the documented test command runs The documented discovery command raised ImportError on import main. Add the sys.path bootstrap used by the sibling lessons so the test runs from the lesson root and from code/. * fix(phase-13/09): bootstrap sys.path so the documented test command runs The documented discovery command raised ImportError on import main. Add the sys.path bootstrap used by the sibling lessons so the test runs from the lesson root and from code/. * fix(phase-19/29): make the fixture tests directory an importable package The demo fixture ships a namespace-style tests directory with no __init__.py, so a regular tests package elsewhere on sys.path could shadow it and the in-repo test runner failed to import the fixture module. Add an empty __init__.py so the fixture tests resolve regardless of what else is installed. * fix(phase-19/39): bound generation by chars and gate the demo on loss decode_response replaces invalid UTF-8 bytes, so a 4-byte generation can re-encode to more than 4 bytes; the test now bounds the decoded character count, which is the real invariant that generate enforces. The demo's success gate keyed on exact-match improvement, which the documented padding mask makes unreachable, so it now checks that training loss decreased. * fix(phase-19/59): test patch-projection grad via a nonzero reduction Summing the CLS token straight out of the final LayerNorm is an algebraic zero for any input, so the gradient reaching the patch projection was always zero and the test failed for a reason unrelated to the encoder. Reduce with a sum of squares, which the demo already uses, so the test exercises a real gradient path. * fix(phase-19/47): load checkpoints weights-only, jail shard paths torch.load ran with weights_only=False, so restoring an untrusted checkpoint executes whatever the pickle names. Store RNG state as primitives so it survives the weights-only loader, load every payload with weights_only=True, reject shard paths that resolve outside the checkpoint directory, and raise ValueError on integrity failures instead of assert. Add regression tests for a pickled object and a shard path escape, and update the docs, skill, and quiz. * fix(phase-11/09): guard run_code with an AST check, not a blocklist The substring blocklist read the snippet as text, so an attribute chain such as the object-subclass walk reached the real interpreter through __globals__. Parse the snippet and walk the tree, rejecting import statements, dunder attribute access, and unsafe builtin names by structure. Add regression tests for the escape, and make the tool description and docs state plainly that this is a teaching filter, not real isolation. Align the TypeScript port's wording and block the constructor and globalThis gadgets in its blocklist. * fix(site): serve markdown at / before the static file wins The homepage markdown-negotiation rewrite lost to the static index.html, so requesting text/markdown at / returned cached HTML. Move it into a legacy route, which is evaluated before the filesystem. Add a readiness-test guard that fails if any negotiation rewrite is shadowed by a static file, and assert the root route. * fix(phase-19/86): parse nested sequences in the stdlib YAML fallback The PyYAML-free fallback stopped gathering a rule's lines at the first nested sequence item, so a rule with an any_of block lost that block and every field after it, and the engine rejected the rule for a missing explanation. The gather now ends on indentation alone, so a deeper sequence stays part of its rule. Verified the fallback matches PyYAML byte for byte on rules.yml, which restores this lesson and the end-to-end safety gate that composes it on a standard Python install. * test(ci): run each lesson's own tests on push and pull request CI executed the certification labs but never the 523 lessons, so a lesson that could not import its own module reached main unnoticed. Adds scripts/run_lesson_tests.py, which discovers each lesson's tests across the four layouts in the repo and runs them the way the lesson docs say, and a lesson-tests job that runs it. The runner installs no scientific dependencies: a lesson that needs one is skipped by scanning its source, so the job stays green while the stdlib lessons run for real. Documents the command in CONTRIBUTING. * test(ci): gate quiz answer-length bias and ratchet it down The quizzes de-bias answer position but not answer length: on 84% of questions the correct option is the longest by a wide margin, so a reader can guess it without knowing the material. Adds scripts/check_quiz_bias.py, which measures the share of length-biased questions and, in --check, fails when a change pushes the rate above a baseline set to today's level. New or edited quizzes cannot add bias, and the baseline ratchets down as quizzes are rebalanced. Wires it into the curriculum workflow next to the position gate and documents the rule in AGENTS.md. Reducing the existing rate is a separate content pass; this stops it getting worse and makes it measurable. * fix(phase-11/09): add module docstring, honest TS wording, ast import Address review: give function_calling.py a module docstring like the sibling lessons, align the TypeScript run_code tool description with the honest wording already used on the Python side (a teaching filter, not real isolation), and add the missing ast import to the docs example's import block so the example runs. * fix(phase-19/39): fail the demo on a non-finite final loss The success gate compared the final loss to the first with >=, so a NaN final loss made the comparison false and the demo exited 0. Require the final loss to be finite before comparing, so a diverged run is reported as a failure. * docs(phase-19/47): note the torch 2.6 requirement for weights_only Before torch 2.6 the weights_only loader had a known bypass (CVE-2025-32434), so the security guarantee this lesson relies on holds only from 2.6 on. Say so next to the weights_only explanation. * fix(site): match the root markdown Accept header case-insensitively Media types are case-insensitive, so a client sending Text/Markdown should still reach the markdown route. Add the (?i) flag to the root route's Accept matcher. * fix(ci): fail a lesson when a real test fails next to a missing dep The lesson-test runner skipped a suite whenever its output mentioned a missing optional module, which could hide a genuine assertion failure in the same run. Only skip when the output shows no test failure alongside the missing module. * fix(ci): fail the quiz-bias check on an unreadable quiz file The scan skipped a quiz.json it could not parse and carried on, so a malformed file would leave the gate green on an incomplete scan. Collect read and parse errors, report each file, and exit non-zero when any are found.
2026-10-02 01:54:39 +08:00 · 2026-09-24 17:27:49 +05:30
parent 0285d9bd92
commit 8050434e5c
25 changed files with 474 additions and 37 deletions
@@ -9,10 +9,12 @@ on:
      - "skills/**"
      - ".claude/skills/**"
      - "scripts/audit_lessons.py"
+      - "scripts/run_lesson_tests.py"
      - "scripts/audit_certifications.py"
      - "scripts/backfill_certification_references.py"
      - "scripts/debias_certification_questions.py"
      - "scripts/debias_quizzes.py"
+      - "scripts/check_quiz_bias.py"
      - "scripts/build_readme_i18n.py"
      - "scripts/readme_translations.py"
      - "scripts/build_catalog.py"
@@ -48,10 +50,12 @@ on:
      - "skills/**"
      - ".claude/skills/**"
      - "scripts/audit_lessons.py"
+      - "scripts/run_lesson_tests.py"
      - "scripts/audit_certifications.py"
      - "scripts/backfill_certification_references.py"
      - "scripts/debias_certification_questions.py"
      - "scripts/debias_quizzes.py"
+      - "scripts/check_quiz_bias.py"
      - "scripts/build_readme_i18n.py"
      - "scripts/readme_translations.py"
      - "scripts/build_catalog.py"
@@ -126,6 +130,8 @@ jobs:
        run: node --test site/test_ui_i18n.js
      - name: quiz answer positions are de-biased
        run: python3 scripts/debias_quizzes.py --check
+      - name: quiz answer lengths do not give away the answer
+        run: python3 scripts/check_quiz_bias.py --check
      - name: certification answer positions are de-biased
        run: python3 scripts/debias_certification_questions.py --check
      - name: README translations are in sync with English
@@ -140,6 +146,23 @@ jobs:
        # repo auto-loads them in Claude Code. They must never diverge.
        run: diff -r skills .claude/skills

+  lesson-tests:
+    name: lesson tests (stdlib)
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5  # v4
+        with:
+          persist-credentials: false
+      - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065  # v5
+        with:
+          python-version: "3.12"
+      - name: run each lesson's own tests
+        # No scientific dependencies are installed here, so lessons that need
+        # one (torch, numpy, and the rest of the allowlist in the script) are
+        # skipped, not failed. The stdlib lessons run for real, which catches
+        # the class of breakage where a lesson cannot import its own module.
+        run: python3 scripts/run_lesson_tests.py
+
  readme-counts-sync:
    name: README counts auto-fix (main only)
    runs-on: ubuntu-latest
@@ -104,6 +104,8 @@ The `**Languages:**` field must match the languages with a `main.*` file in `cod

 Exactly 6 questions: 1 pre + 3 check + 2 post. `correct` is zero-indexed. The site renderer only understands this shape — legacy `q/choices/answer` schemas crash silently.

+Keep the distractors comparable in length to the correct option. When the correct answer is the longest by a wide margin, a reader can guess it without knowing the material. `scripts/check_quiz_bias.py --check` gates this, and `scripts/debias_quizzes.py` spreads the correct option across positions.
+
 ### Claude certification contract

 Certification lessons under `certifications/claude/lessons/` follow the same
@@ -151,7 +151,7 @@ names, logos, links, and tier assignments are managed by the maintainer. See
 1. Fork the repository
 2. Create a feature branch (`git checkout -b add-lesson-phase3-gradient-descent`)
 3. Make your changes
-4. Ensure all code runs
+4. Ensure all code runs. Run `python3 scripts/run_lesson_tests.py` to execute every lesson's own tests; lessons whose tests need a scientific dependency you have not installed are skipped, the rest run.
 5. Submit a pull request with a clear description

 ## Code of Conduct
@@ -1,3 +1,7 @@
+"""Function calling from scratch: a tool registry, the model-to-tool dispatch
+loop, and guarded tool implementations. See docs/en.md for the walkthrough."""
+
+import ast
 import json
 import math
 import re
@@ -101,10 +105,18 @@ def read_file(path):
 def run_code(code, language="python"):
    if language != "python":
        return {"error": True, "message": f"Language '{language}' not supported. Only 'python' is available."}
-    forbidden = ["import os", "import sys", "import subprocess", "exec(", "eval(", "__import__", "open("]
-    for pattern in forbidden:
-        if pattern in code:
-            return {"error": True, "message": f"Forbidden operation: {pattern}", "code": "SECURITY_VIOLATION"}
+    try:
+        tree = ast.parse(code)
+    except SyntaxError as e:
+        return {"error": True, "message": f"SyntaxError: {e}", "code": "SYNTAX_ERROR"}
+    unsafe_names = {"exec", "eval", "compile", "__import__", "open", "globals", "locals", "vars", "getattr", "setattr", "delattr"}
+    for node in ast.walk(tree):
+        if isinstance(node, (ast.Import, ast.ImportFrom)):
+            return {"error": True, "message": "Forbidden operation: import is not allowed", "code": "SECURITY_VIOLATION"}
+        if isinstance(node, ast.Attribute) and node.attr.startswith("__") and node.attr.endswith("__"):
+            return {"error": True, "message": "Forbidden operation: dunder attribute access is not allowed", "code": "SECURITY_VIOLATION"}
+        if isinstance(node, ast.Name) and node.id in unsafe_names:
+            return {"error": True, "message": f"Forbidden operation: {node.id} is not allowed", "code": "SECURITY_VIOLATION"}
    try:
        local_vars = {}
        exec(
@@ -184,7 +196,7 @@ def register_all_tools():
    )
    register_tool(
        "run_code",
-        "Execute Python code in a sandboxed environment. Set a 'result' variable to return output.",
+        "Run a small Python snippet behind a static-analysis guard and a restricted interpreter. This is a teaching filter, not real isolation. Set a 'result' variable to return output.",
        {
            "type": "object",
            "properties": {
@@ -141,7 +141,7 @@ function runCode(args: Readonly<Record<string, JsonValue>>): JsonValue {
  if (language !== "javascript") {
    return { error: true, message: "Language '" + language + "' not supported." };
  }
-  const FORBIDDEN = ["require(", "process.", "fs.", "child_process", "import ", "eval(", "Function("];
+  const FORBIDDEN = ["require(", "process.", "fs.", "child_process", "import ", "eval(", "Function(", "constructor", "globalThis"];
  for (const p of FORBIDDEN) {
    if (code.includes(p)) {
      return { error: true, message: "Forbidden operation: " + p, code: "SECURITY_VIOLATION" };
@@ -209,7 +209,7 @@ function registerAllTools(): void {
  );
  registerTool(
    "run_code",
-    "Execute JavaScript in a sandbox. Assign to 'result' to return output.",
+    "Run a small JavaScript snippet behind a denylist and a restricted evaluator. This is a teaching filter, not real isolation. Assign to 'result' to return output.",
    {
      type: "object",
      properties: {
@@ -0,0 +1,55 @@
+import unittest
+import sys
+from pathlib import Path
+
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+
+from function_calling import run_code, calculator, read_file
+
+
+class RunCodeGuardTests(unittest.TestCase):
+    def test_benign_code_runs(self):
+        out = run_code("result = sum(range(1, 101))")
+        self.assertTrue(out.get("success"))
+        self.assertEqual(out["result"], 5050)
+
+    def test_math_helper_available(self):
+        out = run_code("result = round(math.sqrt(2), 3)")
+        self.assertTrue(out.get("success"))
+        self.assertEqual(out["result"], 1.414)
+
+    def test_import_statement_blocked(self):
+        out = run_code("import os\nresult = os.getpid()")
+        self.assertEqual(out.get("code"), "SECURITY_VIOLATION")
+
+    def test_from_import_blocked(self):
+        out = run_code("from os import getpid\nresult = getpid()")
+        self.assertEqual(out.get("code"), "SECURITY_VIOLATION")
+
+    def test_dunder_attribute_chain_blocked(self):
+        payload = "result = ().__class__.__base__.__subclasses__()"
+        out = run_code(payload)
+        self.assertEqual(out.get("code"), "SECURITY_VIOLATION")
+
+    def test_unsafe_builtin_name_blocked(self):
+        out = run_code("result = eval('1 + 1')")
+        self.assertEqual(out.get("code"), "SECURITY_VIOLATION")
+
+    def test_syntax_error_reported(self):
+        out = run_code("result = (")
+        self.assertEqual(out.get("code"), "SYNTAX_ERROR")
+
+
+class OtherToolGuardTests(unittest.TestCase):
+    def test_calculator_rejects_non_math_characters(self):
+        out = calculator("__import__('os').system('ls')")
+        self.assertTrue(out.get("error"))
+
+    def test_read_file_rejects_traversal(self):
+        out = read_file("../../etc/passwd")
+        self.assertEqual(out.get("code"), "FORBIDDEN")
+
+
+if __name__ == "__main__":
+    unittest.main()
@@ -182,6 +182,7 @@ mx-tool-call-loop
 Build a registry that stores tool definitions and their implementations. Each tool has a JSON Schema definition (what the model sees) and a Python function (what your code executes).

 ```python
+import ast
 import json
 import math
 import time
@@ -290,10 +291,18 @@ def read_file(path):
 def run_code(code, language="python"):
    if language != "python":
        return {"error": True, "message": f"Language '{language}' not supported. Only 'python' is available."}
-    forbidden = ["import os", "import sys", "import subprocess", "exec(", "eval(", "__import__", "open("]
-    for pattern in forbidden:
-        if pattern in code:
-            return {"error": True, "message": f"Forbidden operation: {pattern}", "code": "SECURITY_VIOLATION"}
+    try:
+        tree = ast.parse(code)
+    except SyntaxError as e:
+        return {"error": True, "message": f"SyntaxError: {e}", "code": "SYNTAX_ERROR"}
+    unsafe_names = {"exec", "eval", "compile", "__import__", "open", "globals", "locals", "vars", "getattr", "setattr", "delattr"}
+    for node in ast.walk(tree):
+        if isinstance(node, (ast.Import, ast.ImportFrom)):
+            return {"error": True, "message": "Forbidden operation: import is not allowed", "code": "SECURITY_VIOLATION"}
+        if isinstance(node, ast.Attribute) and node.attr.startswith("__") and node.attr.endswith("__"):
+            return {"error": True, "message": "Forbidden operation: dunder attribute access is not allowed", "code": "SECURITY_VIOLATION"}
+        if isinstance(node, ast.Name) and node.id in unsafe_names:
+            return {"error": True, "message": f"Forbidden operation: {node.id} is not allowed", "code": "SECURITY_VIOLATION"}
    try:
        local_vars = {}
        exec(code, {"__builtins__": {"print": print, "range": range, "len": len, "str": str, "int": int, "float": float, "list": list, "dict": dict, "sum": sum, "min": min, "max": max, "abs": abs, "round": round, "sorted": sorted, "enumerate": enumerate, "zip": zip, "map": map, "filter": filter, "math": math}}, local_vars)
@@ -303,6 +312,8 @@ def run_code(code, language="python"):
        return {"error": True, "message": f"{type(e).__name__}: {e}"}
 ```

+A substring blocklist reads the code as text, so it misses anything the string match does not literally spell. Parsing the code into a syntax tree and walking it lets the guard reject `import` statements, dunder attribute access (the `__class__` and `__globals__` chains that reach back to the real interpreter), and unsafe builtin names by structure rather than spelling. Even so, treat this as a teaching filter, not a real boundary. Any in-process guard shares the interpreter with the code it runs, and a determined caller can still find reachable objects. Production systems run untrusted code in a separate process or container (a subprocess with dropped privileges, gVisor, Firecracker, or a hosted code runner), where an escape lands the attacker in a throwaway box instead of your service.
+
 ### Step 3: Register All Tools

 ```python
@@ -328,7 +339,7 @@ def register_all_tools():
        read_file,
    )
    register_tool(
-        "run_code", "Execute Python code in a sandboxed environment. Set a 'result' variable to return output.",
+        "run_code", "Run a small Python snippet behind a static-analysis guard and a restricted interpreter. This is a teaching filter, not real isolation. Set a 'result' variable to return output.",
        {"type": "object", "properties": {"code": {"type": "string", "description": "Python code to execute"}, "language": {"type": "string", "enum": ["python"], "description": "Programming language"}}, "required": ["code"]},
        run_code,
    )
@@ -1,4 +1,9 @@
+import sys
 import unittest
+from pathlib import Path
+
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))

 import main

@@ -1,9 +1,13 @@
 import json
 import shutil
 import subprocess
+import sys
 import unittest
 from pathlib import Path

+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+
 import main


@@ -1,4 +1,9 @@
+import sys
 import unittest
+from pathlib import Path
+
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))

 import main

@@ -7,6 +7,9 @@ import urllib.error
 import urllib.request
 from pathlib import Path

+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+
 import main


@@ -662,8 +662,12 @@ def run_demo(cfg: Optional[SFTConfig] = None) -> int:
    print("")
    print(f"FINAL EXACT MATCH = {report.final_em:.3f}  (baseline was {initial_em:.3f})")

-    if report.final_em <= initial_em:
-        print("ERROR: training did not improve EM over the untrained baseline", file=sys.stderr)
+    if (
+        len(report.losses) < 2
+        or not math.isfinite(report.losses[-1])
+        or report.losses[-1] >= report.losses[0]
+    ):
+        print("ERROR: training loss did not decrease over the run", file=sys.stderr)
        return 1
    return 0

@@ -144,8 +144,7 @@ class GenerateTests(unittest.TestCase):
        model = build_model(cfg)
        out = generate(model, tok, "Hi.", max_len=cfg.max_len, max_new_tokens=4)
        self.assertIsInstance(out, str)
-        # At most max_new_tokens bytes (the function may stop earlier).
-        self.assertLessEqual(len(out.encode("utf-8")), 4)
+        self.assertLessEqual(len(out), 4)

    def test_temperature_zero_is_deterministic(self) -> None:
        cfg = SFTConfig(hidden=32, heads=2, depth=1, max_len=24, seed=1)
@@ -75,9 +75,10 @@ def synthetic_loader(batch_size: int, num_batches: int, in_dim: int, out_dim: in


 def capture_rng_state() -> Dict[str, Any]:
+    name, keys, pos, has_gauss, cached_gaussian = np.random.get_state()
    state: Dict[str, Any] = {
        "python": random.getstate(),
-        "numpy": np.random.get_state(),
+        "numpy": (name, keys.tolist(), int(pos), int(has_gauss), float(cached_gaussian)),
        "torch_cpu": torch.get_rng_state().tolist(),
    }
    if torch.cuda.is_available():
@@ -197,8 +198,9 @@ def load_checkpoint(
    optimizer: torch.optim.Optimizer,
    scheduler: torch.optim.lr_scheduler._LRScheduler,
 ) -> TrainState:
-    payload = torch.load(path, map_location="cpu", weights_only=False)
-    assert payload["schema"].startswith("ckpt"), f"unknown schema {payload['schema']}"
+    payload = torch.load(path, map_location="cpu", weights_only=True)
+    if not str(payload.get("schema", "")).startswith("ckpt"):
+        raise ValueError(f"unknown schema {payload.get('schema')}")
    model.load_state_dict(payload["model"])
    optimizer.load_state_dict(payload["optimizer"])
    scheduler.load_state_dict(payload["scheduler"])
@@ -292,15 +294,21 @@ def load_sharded_checkpoint(
    expected_sha = index["meta_sha256"]
    meta_path = ckpt_dir / "meta.pt"
    actual_sha = file_sha256(meta_path)
-    assert actual_sha == expected_sha, f"meta sha mismatch: {actual_sha} != {expected_sha}"
-    meta = torch.load(meta_path, map_location="cpu", weights_only=False)
+    if actual_sha != expected_sha:
+        raise ValueError(f"meta sha mismatch: {actual_sha} != {expected_sha}")
+    meta = torch.load(meta_path, map_location="cpu", weights_only=True)
+    root = ckpt_dir.resolve()
    merged: Dict[str, torch.Tensor] = {}
    for shard in meta["shards"]:
-        shard_path = ckpt_dir / shard["path"]
+        shard_path = (ckpt_dir / shard["path"]).resolve()
+        if not shard_path.is_relative_to(root):
+            raise ValueError(f"shard path escapes the checkpoint directory: {shard['path']}")
        actual = file_sha256(shard_path)
-        assert actual == shard["sha256"], f"shard sha mismatch: {shard['path']}"
-        body = torch.load(shard_path, map_location="cpu", weights_only=False)
-        assert body["schema"] == SHARD_SCHEMA
+        if actual != shard["sha256"]:
+            raise ValueError(f"shard sha mismatch: {shard['path']}")
+        body = torch.load(shard_path, map_location="cpu", weights_only=True)
+        if body["schema"] != SHARD_SCHEMA:
+            raise ValueError(f"unknown shard schema {body['schema']}")
        merged.update(body["tensors"])
    model.load_state_dict(merged)
    optimizer.load_state_dict(meta["optimizer"])
@@ -3,6 +3,7 @@
 from __future__ import annotations

 import json
+import pickle
 import sys
 import tempfile
 import unittest
@@ -17,6 +18,10 @@ sys.path.insert(0, str(HERE))
 import main as ckpt


+class _ArbitraryObject:
+    pass
+
+
 def _build_components(total_steps: int = 10, lr: float = 0.01):
    model = ckpt.make_model(8, 12, 4)
    opt, sched = ckpt.make_optimizer_and_scheduler(model, lr=lr, total_steps=total_steps)
@@ -79,6 +84,14 @@ class CheckpointResumeTests(unittest.TestCase):
            )
            self.assertLess(result["max_loss_diff_after_resume"], 1e-5)

+    def test_load_refuses_pickled_objects(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            target = Path(tmp) / "ckpt.pt"
+            torch.save({"schema": "ckpt.v1", "payload": _ArbitraryObject()}, target)
+            model, opt, sched = _build_components(total_steps=4)
+            with self.assertRaises(pickle.UnpicklingError):
+                ckpt.load_checkpoint(target, model, opt, sched)
+

 class ShardedCheckpointTests(unittest.TestCase):
    def test_sharded_round_trip(self):
@@ -106,7 +119,24 @@ class ShardedCheckpointTests(unittest.TestCase):
            data = tampered.read_bytes()
            tampered.write_bytes(data + b"\x00")
            model2, opt2, sched2 = _build_components(total_steps=4)
-            with self.assertRaises(AssertionError):
+            with self.assertRaises(ValueError):
+                ckpt.load_sharded_checkpoint(Path(tmp), model2, opt2, sched2)
+
+    def test_shard_path_outside_checkpoint_dir_is_rejected(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            ckpt.seed_everything(0)
+            model, opt, sched = _build_components(total_steps=4)
+            state = ckpt.TrainState(step=1, epoch=0, batch_in_epoch=1, losses=[0.9])
+            meta = ckpt.save_sharded_checkpoint(model, opt, sched, state, Path(tmp), num_shards=2)
+            meta["shards"][0]["path"] = "../outside.pt"
+            meta_path = Path(tmp) / "meta.pt"
+            ckpt.atomic_save(meta, meta_path)
+            index_path = Path(tmp) / "index.json"
+            index = json.loads(index_path.read_text())
+            index["meta_sha256"] = ckpt.file_sha256(meta_path)
+            ckpt.atomic_write_json(index, index_path)
+            model2, opt2, sched2 = _build_components(total_steps=4)
+            with self.assertRaises(ValueError):
                ckpt.load_sharded_checkpoint(Path(tmp), model2, opt2, sched2)


@@ -90,7 +90,7 @@ cc-atomic-checkpoint

 ### Step 1: capture and restore RNG state

-`capture_rng_state` returns a dict with Python's `random.getstate`, NumPy's `np.random.get_state`, and PyTorch CPU and CUDA RNG bytes. `restore_rng_state` reverses it. The CPU tensor is a uint8 byte buffer that PyTorch's RNG knows how to consume.
+`capture_rng_state` returns a dict with Python's `random.getstate`, NumPy's `np.random.get_state`, and PyTorch CPU and CUDA RNG bytes. Every piece is stored as plain Python numbers, tuples, and lists (NumPy's key array goes through `tolist()`), so the loader in Step 3 can read it back without unpickling arbitrary objects. `restore_rng_state` reverses it. The CPU tensor is a uint8 byte buffer that PyTorch's RNG knows how to consume.

 ### Step 2: atomic save

@@ -100,9 +100,11 @@ cc-atomic-checkpoint

 `save_checkpoint` packages the model, optimizer, scheduler, train state, and RNG into one dict. `load_checkpoint` reverses it and returns a `TrainState`. The schema field is the upgrade hook: future format changes bump the version string and the loader dispatches.

+`load_checkpoint` calls `torch.load(..., weights_only=True)`. A `.pt` file is a pickle, and unpickling an untrusted file with `weights_only=False` runs whatever code the file names. The weights-only loader accepts tensors and primitive containers and rejects everything else, which is why Step 1 keeps the RNG state in plain lists. Integrity checks raise `ValueError` instead of using `assert`, because `python -O` strips asserts. Use torch 2.6 or newer: before that release `weights_only=True` had a known bypass (CVE-2025-32434), so the guarantee this lesson relies on holds only from 2.6 on.
+
 ### Step 4: sharded variant

-`save_sharded_checkpoint` round-robins the parameter keys across N shards, writes each shard with its own atomic save, writes a meta file with optimizer and scheduler and train state, and writes the JSON index with shard sha256s. `load_sharded_checkpoint` verifies every shard before merging.
+`save_sharded_checkpoint` round-robins the parameter keys across N shards, writes each shard with its own atomic save, writes a meta file with optimizer and scheduler and train state, and writes the JSON index with shard sha256s. `load_sharded_checkpoint` verifies every shard before merging and refuses any shard path that resolves outside the checkpoint directory.

 ### Step 5: resume demo

@@ -120,8 +122,9 @@ The single-file and sharded demos both assert max-diff under 1e-4. The summary l

 Production training stacks ship checkpointing as part of the trainer. The shape is the same: model + optimizer + scheduler + counters + RNG, written atomically, named by step so the latest is easy to find. Sharded layouts power large model loading with parallel reads; the index.json is what makes that work.

-Three patterns to enforce:
+Four patterns to enforce:

+- **Load with `weights_only=True`.** A checkpoint pulled from a shared drive or a download is untrusted input. The weights-only loader keeps a malicious file from running code on the machine that resumes.
 - **Schema is a string in the payload.** Migrations branch on it. Without it you cannot evolve the format without breaking old runs.
 - **Sha256 every shard.** A silently truncated download is the worst kind of bug; the loader fails fast or it fails late.
 - **Keep checkpoint cadence honest.** Save every N steps and every wallclock-minute, whichever is shorter. Otherwise the long step that crashes wastes a full window of work.
@@ -151,7 +154,7 @@ Three patterns to enforce:
 ## Further Reading

 - POSIX `rename` semantics for the atomicity claim that `os.replace` relies on.
- PyTorch documentation on `torch.save` and `torch.load`, including `map_location` for cross-device restores.
+- PyTorch documentation on `torch.save` and `torch.load`, including `map_location` for cross-device restores and `weights_only` for loading untrusted files.
 - Phase 19 lesson 46 covers the gradient accumulation that this lesson's checkpoint payload survives across.
 - Phase 19 lesson 48 covers the distributed wrappers whose state dict format this scheme accommodates.
 - The Linux kernel `fsync` documentation for the durability guarantee behind atomic rename.
@@ -36,7 +36,8 @@ Any training run longer than the wallclock cap of the cluster, any run that must
 - `model.shard-NNN.pt` per shard, round robin on keys or split by parameter group.
 - `meta.pt` carries optimizer, scheduler, train state, RNG, and the shard manifest.
 - `index.json` carries `sha256` for every shard and for `meta.pt`.
- Loader verifies every hash before merging.
+- Loader verifies every hash before merging and rejects shard paths outside the checkpoint directory.
+- Load every file with `torch.load(path, map_location="cpu", weights_only=True)`. Keep RNG state as plain lists so it survives the weights-only loader.

 ## Mid-epoch resume

@@ -50,3 +51,4 @@ Any training run longer than the wallclock cap of the cluster, any run that must
 - Forgetting RNG: resumed loss diverges from baseline. Run the demo's assertion.
 - Forgetting optimizer state: next step lurches. Same diff blows up.
 - Pruning the wrong checkpoint: keep last K plus best.
+- Loading with `weights_only=False`: a `.pt` file is a pickle, so an untrusted checkpoint runs code on load.
@@ -36,7 +36,7 @@
        "To compress the file"
      ],
      "correct": 0,
-      "explanation": "load_sharded_checkpoint asserts each shard's actual hash matches the recorded one and the meta file's hash too. Silent corruption is the worst failure mode."
+      "explanation": "load_sharded_checkpoint checks each shard's actual hash against the recorded one, and the meta file's hash too, and raises before merging on any mismatch. Silent corruption is the worst failure mode."
    },
    {
      "stage": "check",
@@ -74,7 +74,7 @@ class TestViTStack(unittest.TestCase):
        enc = VisionEncoder(cfg)
        img = torch.randn(1, 3, cfg.image_size, cfg.image_size)
        _, cls = enc(img)
-        cls.sum().backward()
+        cls.pow(2).sum().backward()
        grad = enc.front.patch.proj.weight.grad
        self.assertIsNotNone(grad)
        self.assertGreater(grad.norm().item(), 0.0)
@@ -154,7 +154,7 @@ def _parse_sequence(lines: list[str], start: int, indent: int) -> tuple[list[Any
            synthetic = " " * child_indent + rest
            j = i + 1
            extra_lines = []
-            while j < len(lines) and _indent_of(lines[j]) > indent and not lines[j].lstrip().startswith("- "):
+            while j < len(lines) and _indent_of(lines[j]) > indent:
                extra_lines.append(lines[j])
                j += 1
            block = [synthetic] + extra_lines
@@ -0,0 +1,119 @@
+#!/usr/bin/env python3
+"""Measure and gate answer-length bias in the lesson quizzes.
+
+A multiple-choice question is guessable when the correct option is written more
+fully than every distractor: a reader can pick the longest answer without
+knowing the material. `scripts/debias_quizzes.py` already spreads the correct
+option across positions; this measures the separate length tell and keeps it
+from getting worse.
+
+A question is counted as length-biased when its correct option is strictly the
+longest and is at least MIN_RATIO times the length of its longest distractor.
+
+Usage:
+    python3 scripts/check_quiz_bias.py            # report the distribution
+    python3 scripts/check_quiz_bias.py --report   # also list the worst offenders
+    python3 scripts/check_quiz_bias.py --check     # fail if the rate rose above the baseline
+"""
+import argparse
+import collections
+import glob
+import json
+import sys
+
+QUIZ_GLOB = "phases/*/*/quiz.json"
+MIN_RATIO = 1.25
+
+# The share of length-biased questions on main today. The gate fails when a
+# change pushes the rate above this, so new quizzes cannot add bias. Lower it as
+# quizzes are rebalanced so the ceiling ratchets down and never drifts back up.
+BASELINE_RATE = 0.84
+
+
+def questions_in(data):
+    if isinstance(data, list):
+        return data
+    if isinstance(data, dict):
+        return data.get("questions", [])
+    return []
+
+
+def is_length_biased(question):
+    options = question.get("options")
+    correct = question.get("correct")
+    if not isinstance(options, list) or not isinstance(correct, int):
+        return None
+    if not (0 <= correct < len(options)) or len(options) < 2:
+        return None
+    lengths = [len(str(option).strip()) for option in options]
+    correct_length = lengths[correct]
+    longest_distractor = max(lengths[i] for i in range(len(lengths)) if i != correct)
+    if longest_distractor == 0:
+        return True
+    return correct_length > longest_distractor and correct_length / longest_distractor >= MIN_RATIO
+
+
+def scan():
+    total = 0
+    biased = 0
+    per_phase = collections.defaultdict(lambda: [0, 0])
+    offenders = []
+    errors = []
+    for path in sorted(glob.glob(QUIZ_GLOB)):
+        phase = path.split("/")[1]
+        try:
+            data = json.load(open(path, encoding="utf-8"))
+        except (OSError, json.JSONDecodeError) as exc:
+            errors.append((path, str(exc)))
+            continue
+        for index, question in enumerate(questions_in(data)):
+            flag = is_length_biased(question)
+            if flag is None:
+                continue
+            total += 1
+            per_phase[phase][1] += 1
+            if flag:
+                biased += 1
+                per_phase[phase][0] += 1
+                offenders.append((path, index, str(question.get("question", ""))[:70]))
+    return total, biased, per_phase, offenders, errors
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--check", action="store_true")
+    parser.add_argument("--report", action="store_true")
+    args = parser.parse_args()
+
+    total, biased, per_phase, offenders, errors = scan()
+    if errors:
+        for path, message in errors:
+            print(f"could not read {path}: {message}", file=sys.stderr)
+        return 1
+    if total == 0:
+        print("no quiz questions found")
+        return 0
+    rate = biased / total
+    print(f"{biased}/{total} questions are length-biased ({rate:.1%}); baseline {BASELINE_RATE:.0%}")
+
+    if args.report:
+        print("\nper phase:")
+        for phase, (b, n) in sorted(per_phase.items(), key=lambda kv: -kv[1][0]):
+            print(f"  {phase:44} {b:4}/{n:4}  {b / n:5.1%}")
+        print(f"\nworst offenders ({min(len(offenders), 30)} of {len(offenders)}):")
+        for path, index, text in offenders[:30]:
+            print(f"  {path} [q{index}] {text}")
+
+    if args.check and rate > BASELINE_RATE:
+        print(
+            f"\nlength-bias rate {rate:.1%} is above the baseline {BASELINE_RATE:.0%}. "
+            "Rebalance the new or edited quizzes so distractors are comparable in "
+            "length to the correct option.",
+            file=sys.stderr,
+        )
+        return 1
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())
@@ -0,0 +1,137 @@
+#!/usr/bin/env python3
+"""Run each lesson's own test suite and report pass, skip, or fail.
+
+CI installs no heavy scientific dependencies, so a lesson whose test needs one
+(torch, numpy, and the rest of the allowlist below) is reported as skipped
+rather than failed. Any other import error, including a lesson that cannot
+import its own module, is a failure: that is the class of breakage this runner
+exists to catch. Exit status is non-zero when any lesson fails.
+
+Usage:
+    python3 scripts/run_lesson_tests.py [--timeout SECONDS] [--phase NN-slug]
+"""
+
+from __future__ import annotations
+
+import argparse
+import importlib.util
+import re
+import subprocess
+import sys
+from pathlib import Path
+
+
+ROOT = Path(__file__).resolve().parents[1]
+PHASES = ROOT / "phases"
+
+OPTIONAL_DEPS = {
+    "torch", "torchvision", "torchaudio", "numpy", "scipy", "pandas",
+    "matplotlib", "sklearn", "transformers", "datasets", "tokenizers",
+    "safetensors", "h5py", "zstandard", "tiktoken", "sentencepiece",
+    "sacrebleu", "nltk", "jax", "jaxlib", "flax", "diffusers", "accelerate",
+    "umap", "optuna", "gymnasium", "gym", "faiss", "peft", "bitsandbytes",
+    "einops", "wandb", "PIL", "cv2", "librosa", "soundfile", "networkx",
+}
+
+MISSING_MODULE = re.compile(r"No module named '([\w.]+)'")
+IMPORT_LINE = re.compile(r"^\s*(?:import|from)\s+([\w.]+)", re.MULTILINE)
+
+
+def missing_optional_dep(lesson: Path) -> str | None:
+    """Return an optional dependency the lesson needs but cannot import.
+
+    Some lessons import an optional package inside a try/except and degrade
+    instead of raising, so scanning the source is more reliable than watching
+    for a ModuleNotFoundError at run time.
+    """
+    seen: set[str] = set()
+    for source in lesson.rglob("*.py"):
+        try:
+            text = source.read_text(encoding="utf-8")
+        except OSError:
+            continue
+        for name in IMPORT_LINE.findall(text):
+            seen.add(name.split(".")[0])
+    for dep in sorted(seen & OPTIONAL_DEPS):
+        if importlib.util.find_spec(dep) is None:
+            return dep
+    return None
+
+
+def discover_targets(phase_filter: str | None):
+    """Yield (label, cwd, argv) for each lesson test suite on disk."""
+    for lesson in sorted(PHASES.glob("*/*")):
+        if not lesson.is_dir() or lesson.name.startswith("."):
+            continue
+        if phase_filter and lesson.parent.name != phase_filter:
+            continue
+        code = lesson / "code"
+        rel = lesson.relative_to(ROOT)
+        if (code / "tests").is_dir() and list((code / "tests").glob("test_*.py")):
+            yield f"{rel} (discover code/tests)", lesson, lesson, ["-m", "unittest", "discover", "-s", "code/tests"]
+        for loose in sorted(code.glob("test_*.py")):
+            yield f"{rel}/code/{loose.name}", lesson, code, ["-m", "unittest", loose.stem]
+        if (code / "tests.py").is_file():
+            yield f"{rel}/code/tests.py", lesson, code, ["-m", "unittest", "tests"]
+        if (lesson / "tests").is_dir() and list((lesson / "tests").glob("*.py")):
+            yield f"{rel} (discover tests)", lesson, lesson, ["-m", "unittest", "discover", "-s", "tests"]
+
+
+def classify(output: str) -> tuple[str, str]:
+    has_test_failure = "AssertionError" in output or "FAILED (failures=" in output
+    if not has_test_failure:
+        match = MISSING_MODULE.search(output)
+        if match and match.group(1).split(".")[0] in OPTIONAL_DEPS:
+            return "skip", f"needs {match.group(1)}"
+    return "fail", ""
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--timeout", type=int, default=120)
+    parser.add_argument("--phase", default=None)
+    args = parser.parse_args()
+
+    passed, skipped, failed = [], [], []
+    for label, lesson, cwd, argv in discover_targets(args.phase):
+        dep = missing_optional_dep(lesson)
+        if dep is not None:
+            skipped.append((label, f"needs {dep}"))
+            print(f"SKIP {label}: needs {dep}")
+            continue
+        try:
+            proc = subprocess.run(
+                [sys.executable, *argv],
+                cwd=str(cwd),
+                capture_output=True,
+                text=True,
+                timeout=args.timeout,
+            )
+        except subprocess.TimeoutExpired:
+            failed.append((label, f"timed out after {args.timeout}s"))
+            print(f"FAIL {label}: timed out after {args.timeout}s")
+            continue
+        if proc.returncode == 0:
+            passed.append(label)
+            print(f"PASS {label}")
+            continue
+        kind, reason = classify(proc.stderr + proc.stdout)
+        if kind == "skip":
+            skipped.append((label, reason))
+            print(f"SKIP {label}: {reason}")
+        else:
+            tail = (proc.stderr.strip() or proc.stdout.strip()).splitlines()[-1:] or [""]
+            failed.append((label, tail[0]))
+            print(f"FAIL {label}: {tail[0]}")
+
+    print(f"\n{len(passed)} passed, {len(skipped)} skipped, {len(failed)} failed")
+    if failed:
+        print("\nFailures:")
+        for label, reason in failed:
+            print(f"  {label}: {reason}")
+        return 1
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())
@@ -69,6 +69,15 @@ def main() -> None:
    negotiator_rewrites = [r for r in rewrites if r.get("destination", "").startswith("/api/markdown")]
    assert negotiator_rewrites, "markdown negotiation rewrite is missing"
    assert all("accept" in h["key"].lower() for r in negotiator_rewrites for h in r["has"])
+    shadowed = [
+        r["source"]
+        for r in negotiator_rewrites
+        if any(
+            candidate.is_file()
+            for candidate in (SITE / r["source"].strip("/") / "index.html", SITE / r["source"].strip("/"))
+        )
+    ]
+    assert not shadowed, f"a static file wins over these negotiation rewrites: {shadowed}"

    route_rewrites = {rewrite["source"]: rewrite["destination"] for rewrite in rewrites}
    assert route_rewrites["/lesson"] == "/api/lesson"
@@ -87,6 +96,12 @@ def main() -> None:
        "methods": ["GET", "HEAD"],
        "dest": "/api/certification?legacy=1",
    }
+    root_route = legacy_routes["/"]
+    assert root_route["dest"] == "/api/markdown?path=/"
+    assert any(
+        h["type"] == "header" and h["key"].lower() == "accept" and "text/markdown" in h["value"]
+        for h in root_route["has"]
+    )

    headers = config["headers"]
    llms_header = next(h for h in headers if h["source"] == "/llms.txt")
@@ -5,12 +5,12 @@
  "installCommand": "echo skip",
  "routes": [
    { "src": "/lesson\\.html", "methods": ["GET", "HEAD"], "dest": "/api/lesson?legacy=1" },
-    { "src": "/certification\\.html", "methods": ["GET", "HEAD"], "dest": "/api/certification?legacy=1" }
+    { "src": "/certification\\.html", "methods": ["GET", "HEAD"], "dest": "/api/certification?legacy=1" },
+    { "src": "/", "methods": ["GET", "HEAD"], "has": [{ "type": "header", "key": "accept", "value": "(?i).*text/markdown.*" }], "dest": "/api/markdown?path=/" }
  ],
  "rewrites": [
    { "source": "/lesson", "destination": "/api/lesson" },
    { "source": "/certification", "destination": "/api/certification" },
-    { "source": "/", "has": [{ "type": "header", "key": "accept", "value": ".*" }], "destination": "/api/markdown?path=/" },
    { "source": "/about", "has": [{ "type": "header", "key": "accept", "value": ".*" }], "destination": "/api/markdown?path=/about" },
    { "source": "/catalog", "has": [{ "type": "header", "key": "accept", "value": ".*" }], "destination": "/api/markdown?path=/catalog" },
    { "source": "/glossary", "has": [{ "type": "header", "key": "accept", "value": ".*" }], "destination": "/api/markdown?path=/glossary" },