mirror of
https://github.com/rohitg00/ai-engineering-from-scratch.git
synced 2026-10-02 01:54:39 +08:00
Rohit Ghumare
dda194f840
Every "Test Your Understanding" quiz placed the correct answer in option B.
Across the 2026 questions in 338 quiz files the correct answer sat at index 1
in 61.5% of cases (uniform would be ~25%), and 107 files had every answer at B,
making the quizzes guessable without reading them.
scripts/debias_quizzes.py rewrites each question's option order with a
deterministic, content-seeded permutation and updates the correct index to
follow the moved answer. It is idempotent: options are canonicalised to a sorted
base before permuting, so re-running produces byte-identical output. Questions
whose options reference each other by position ("all of the above", "both A and
B") are left untouched. The correct-answer value, the option set, and every
explanation are preserved exactly; only order and the index change.
Result: A 23.8% / B 26.3% / C 23.5% / D 26.4%.
The script doubles as a CI guard: `--check` exits non-zero if any quiz is not
de-biased, wired into the curriculum workflow so new lessons cannot regress.
Fixes #368
79 lines
3.2 KiB
JSON
79 lines
3.2 KiB
JSON
{
|
|
"lesson": "80-checkpoint-sharded-resume",
|
|
"title": "Sharded Checkpoint and Atomic Resume",
|
|
"questions": [
|
|
{
|
|
"stage": "pre",
|
|
"question": "Why is gather-then-write the wrong checkpoint pattern at scale?",
|
|
"options": [
|
|
"All state passes through rank 0; write bandwidth is the slowest single link, not the aggregate, so a TB-scale write takes hours",
|
|
"It needs special hardware",
|
|
"It uses too much CPU",
|
|
"It looks ugly"
|
|
],
|
|
"correct": 0,
|
|
"explanation": "Sharded write scales with cluster size: 64 ranks writing in parallel finish in 1/64 the time of one rank writing the gathered state."
|
|
},
|
|
{
|
|
"stage": "pre",
|
|
"question": "What does the manifest record that the shard files cannot?",
|
|
"options": [
|
|
"world_size, schema version, per-shard sha256, and shard ownership; gives the loader a contract to validate before touching state",
|
|
"Hostname",
|
|
"Random numbers",
|
|
"Filesystem name"
|
|
],
|
|
"correct": 0,
|
|
"explanation": "Without the manifest the loader has to guess which shard belongs to which rank and trust the file count is right."
|
|
},
|
|
{
|
|
"stage": "check",
|
|
"question": "Why write to <name>.tmp and rename instead of writing directly?",
|
|
"options": [
|
|
"Required by torch",
|
|
"Faster",
|
|
"Saves memory",
|
|
"POSIX rename within a filesystem is atomic: a crash mid-write leaves the previous file untouched. Direct writes can leave half-finished checkpoints that poison the next resume"
|
|
],
|
|
"correct": 3,
|
|
"explanation": "Without atomic write a partial save plus an updated manifest pointing at it corrupts state on the next load."
|
|
},
|
|
{
|
|
"stage": "check",
|
|
"question": "What does sha256 per shard defend against?",
|
|
"options": [
|
|
"Slowdown",
|
|
"Network attacks",
|
|
"Partial or corrupted writes; verifying the hash on load catches truncated shards before they touch model state",
|
|
"Memory leak"
|
|
],
|
|
"correct": 2,
|
|
"explanation": "Without sha256 verification a truncated shard loads silently and corrupts the optimiser, surfacing later as NaN loss."
|
|
},
|
|
{
|
|
"stage": "check",
|
|
"question": "Why does the load path reject a world_size mismatch loudly?",
|
|
"options": [
|
|
"Resuming a 4-rank shard layout on 8 ranks silently misassigns shards; loud rejection forces the operator to use the cross-world-size rebalance path",
|
|
"Hardware needs it",
|
|
"Easier to code",
|
|
"Cosmetic"
|
|
],
|
|
"correct": 0,
|
|
"explanation": "The cross-world-size rebalance is a separate operation; silently splicing wrong sized shards corrupts state."
|
|
},
|
|
{
|
|
"stage": "post",
|
|
"question": "Why does production rotate checkpoints (keep last K)?",
|
|
"options": [
|
|
"Compression",
|
|
"Aesthetic",
|
|
"It does not",
|
|
"Without rotation the disk fills mid-run and the next checkpoint fails; rotation deletes the oldest before writing the new one, bounding disk usage"
|
|
],
|
|
"correct": 3,
|
|
"explanation": "Real runs keep 3-5 recent checkpoints; the oldest is dropped before each new save so the disk budget is fixed."
|
|
}
|
|
]
|
|
}
|