Files
ai-engineering-from-scratch/phases/10-llms-from-scratch/10-evaluation
Rohit Ghumare a05d392590 fix: repair broken lesson references and guard homepage storage (#495)
* fix(site): guard theme storage access on the homepage and about page

Reading or writing localStorage throws a SecurityError when storage is
blocked (strict privacy settings, some embedded webviews). site/app.js
read the saved theme outside any try/catch, before it registered its
DOMContentLoaded handler, so the throw stopped the whole homepage from
initializing. The about page's inline theme script had the same
unguarded read and write.

Wrap all four accesses the way the other pages already do and fall
back to the system theme. Bump the app.js cache key so browsers pick
up the fix, and give app.js its own release constant in the cache-key
test.

Fixes #490

* fix(lessons): repair dataset, model, and tool references that no longer resolve

Lesson snippets pointed at resources that are gone or never existed, so
they fail when a student runs them:

- wikimedia/wikipedia only ships 20231101.* configs; 20220301.en is gone
- MMAU-Pro lives at gamma-lab-umd/MMAU-Pro, with audio_path and answer
  fields; open-ended rows are filtered out of the exact-match score
- meta-llama/Llama-3-70B-Instruct is meta-llama/Meta-Llama-3-70B-Instruct
- Depth Anything V2 for transformers is
  depth-anything/Depth-Anything-V2-Large-hf
- microsoft/deberta-v3-large-mnli and microsoft/BEATs-base are not on
  the Hub; BEATs checkpoints ship with microsoft/unilm
- sayakpaul/sd-lora-ghibli does not exist; use a published SD 1.5
  Ghibli LoRA with its trigger phrase
- Qwen/Qwen3-0.6B-spec and meta-llama/Llama-3.2-1B-Instruct-spec are
  not real draft models
- gemini-3-pro is not a Gemini model code, gemini-1.5-pro is shut down,
  and the google.generativeai SDK reached end of life on 2025-11-30
- vLLM replaced --speculative-model and --num-speculative-tokens with
  --speculative-config, and --dtype has no float8_e4m3fn choice
- convert_hf_to_gguf.py cannot write q4_k_m; llama-quantize does
- AutoGPTQ and AutoAWQ are archived; GPTQModel and LLM Compressor are
  the maintained successors, and vLLM reads their quantization method
  from the checkpoint config

Fixes #493

* fix(lessons): replace dead reference links

A link check over every lesson found 46 references that return 404 or
410 or no longer resolve. Each replacement was fetched and matched
against the cited title or content:

- pages that moved on the same site: vLLM, librosa, Letta, Apollo
  Research, Stability AI, NVIDIA, NeurIPS, the MCP spec, the Claude
  docs, the A2A spec, the Julia docs, W&B, Baseten, and Arena (formerly
  LMSYS Chatbot Arena, whose old domain no longer resolves)
- papers and books pointed at DOIs or publisher pages: Friedman on
  gradient boosting, Golub and Van Loan, Kuttruff, Littman's thesis,
  Milne and Witten, and Zave and Jackson (whose DOI was wrong)
- Wayback snapshots where the source only survives in the archive:
  Poynton's color space tour and a model-routing article
- citations of pages that never existed replaced with the real source:
  the DINOv2 paper, the MMAU-Pro project page, Anthropic's Contextual
  Retrieval post, the February 2026 risk report, and the correct
  Anthropic alignment post
- removed where no source exists: a Spinning Up DQN page, the
  andrewgarst/agentic_harness repo, and an Akira blog post

URLs in code-file reference headers get the same treatment.

* chore(site): rebuild data.js

* Revert "chore(site): rebuild data.js"

The main-only CI job rebuilds site/data.js after merge, so the PR
should not carry it; a stale "Last built" line would conflict.

This reverts commit e78e5def.

* fix(lessons): correct claims flagged in review

Each finding was checked against its primary source before changing
anything:

- vLLM: the v0.18.0 feature matrix marks speculative decoding as
  compatible with chunked prefill (and incompatible with LoRA), so the
  "draft model plus chunked prefill does not compile" gotcha was false.
  It is replaced in both lessons, their skills, the scheduler and
  EAGLE-3 diagrams, and the quiz question that asserted it
- alignment faking: the cited Anthropic post tests interrogation
  training, scratchpad length penalties, and process supervision, not
  a compliance-gap loss or faithful-CoT training. The section, learning
  objective, exercise, reference, skill, diagram, and quiz now match it
- Gemini: Google limits the 2.5 models to existing users and points new
  projects to 3.5 Flash-Lite or 3.8 Flash, so both examples use
  gemini-3.8-flash (GA, caching supported, 1M context), and the
  context comparison names GPT-4o's 128K window instead of Claude
- vLLM FP8: --quantization fp8_per_tensor, which the 0.30.0 release
  accepts and main recommends over the deprecated fp8
- GGUF: the converter has no K-quant output; it does write f32 and
  ternary files, which the old wording ruled out
- MMAU-Pro: the exact-match loop is labeled a sanity check, with the
  official evaluator (embedding match, LLM judge, regex rules) named
- Depth Anything: the pipeline example is labeled V2, with the separate
  depth_anything_3 package described for V3, and the missing numpy
  import added
- audio classification: Step 5 is titled for the AST example it runs,
  with BEATs loading described beneath it

The repo's quiz de-biaser moved one rewritten question's correct answer
to its assigned position.

* fix(lessons): drop the speculative decoding and LoRA incompatibility

The v0.18.0 docs matrix marks speculative decoding with LoRA as
unsupported, but vLLM has supported LoRA with speculative decoding on
the V1 GPU engine since vllm-project/vllm#21068 (merged 2025-11-08), so
calling the pair incompatible steers learners away from valid setups.
Remove the claim from both serving lessons, their skills, and both
diagrams. The rollout skill's hard reject now names an incompatibility
the speculative-decoding docs do list: pipeline parallelism on vLLM
0.15.0 or earlier.
2026-09-27 15:38:59 +05:30
..