mirror of
https://github.com/rohitg00/ai-engineering-from-scratch.git
synced 2026-10-02 01:54:39 +08:00
Rohit Ghumare
dda194f840
Every "Test Your Understanding" quiz placed the correct answer in option B.
Across the 2026 questions in 338 quiz files the correct answer sat at index 1
in 61.5% of cases (uniform would be ~25%), and 107 files had every answer at B,
making the quizzes guessable without reading them.
scripts/debias_quizzes.py rewrites each question's option order with a
deterministic, content-seeded permutation and updates the correct index to
follow the moved answer. It is idempotent: options are canonicalised to a sorted
base before permuting, so re-running produces byte-identical output. Questions
whose options reference each other by position ("all of the above", "both A and
B") are left untouched. The correct-answer value, the option set, and every
explanation are preserved exactly; only order and the index change.
Result: A 23.8% / B 26.3% / C 23.5% / D 26.4%.
The script doubles as a CI guard: `--check` exits non-zero if any quiz is not
de-biased, wired into the curriculum workflow so new lessons cannot regress.
Fixes #368
79 lines
3.4 KiB
JSON
79 lines
3.4 KiB
JSON
{
|
|
"lesson": "42-large-corpus-downloader",
|
|
"title": "Large Corpus Downloader",
|
|
"questions": [
|
|
{
|
|
"stage": "pre",
|
|
"question": "Why does the downloader keep a separate .partial.json checkpoint per shard?",
|
|
"options": [
|
|
"To make the cache directory look organized",
|
|
"So a resumed download can verify the sha256 prefix over the verified bytes before appending, making silent corruption impossible",
|
|
"To store the URL in case the shard file moves",
|
|
"Because urllib requires it"
|
|
],
|
|
"correct": 1,
|
|
"explanation": "The checkpoint records verified_bytes and the sha256 of those bytes; resume only proceeds when the on-disk hash matches."
|
|
},
|
|
{
|
|
"stage": "check",
|
|
"question": "What does the LSH band scheme buy that exact-hash dedup does not?",
|
|
"options": [
|
|
"Constant memory usage",
|
|
"Lossless compression",
|
|
"Sub-linear lookup that flags near-duplicates with high Jaccard similarity even when the documents differ by a few tokens",
|
|
"Faster sha256 computation"
|
|
],
|
|
"correct": 2,
|
|
"explanation": "MinHash plus LSH catches paraphrases and boilerplate variants that exact-hash dedup walks past."
|
|
},
|
|
{
|
|
"stage": "check",
|
|
"question": "Why is the checkpoint written before the bytes are appended, not after?",
|
|
"options": [
|
|
"If the process dies between the write and the checkpoint update, the next resume reads a checkpoint that is behind the bytes and silently re-appends, corrupting the file",
|
|
"It saves disk space",
|
|
"It is convenient",
|
|
"It is faster"
|
|
],
|
|
"correct": 0,
|
|
"explanation": "Checkpoint-first matches a write-ahead log: a crash leaves the verified offset trailing or equal to the file, never ahead."
|
|
},
|
|
{
|
|
"stage": "check",
|
|
"question": "What is the role of the manifest sha256 lock file?",
|
|
"options": [
|
|
"It speeds up reads",
|
|
"It is a backup of the manifest",
|
|
"Downstream stages refuse to start unless the manifest content matches the pinned hash, closing the silent attack surface where an attacker edits one file",
|
|
"It records the date"
|
|
],
|
|
"correct": 2,
|
|
"explanation": "Pinning the manifest hash makes the link between code and data a content-addressed contract."
|
|
},
|
|
{
|
|
"stage": "post",
|
|
"question": "Why is dedup placed upstream of tokenization rather than after?",
|
|
"options": [
|
|
"Tokenizers reject duplicates",
|
|
"Tokenization is expensive; running it twice on the same document doubles cost for no loss-curve benefit, so dropping duplicates first is strictly cheaper",
|
|
"The tokenizer needs a vocabulary first",
|
|
"The tokenizer cannot read JSONL"
|
|
],
|
|
"correct": 1,
|
|
"explanation": "Dedup is the first stage where each duplicate-dropped document saves the tokenizer one pass."
|
|
},
|
|
{
|
|
"stage": "post",
|
|
"question": "What does tombstoning a dropped duplicate preserve that a silent delete does not?",
|
|
"options": [
|
|
"Compression ratio",
|
|
"Disk space",
|
|
"Manifest sha256 stability",
|
|
"The link between the duplicate and the keeper it collided with, plus an audit trail that survives a later threshold change"
|
|
],
|
|
"correct": 3,
|
|
"explanation": "Tombstones let a future pass revisit the dedup threshold; deletes lose the evidence."
|
|
}
|
|
]
|
|
}
|