mirror of
https://github.com/rohitg00/ai-engineering-from-scratch.git
synced 2026-10-02 01:54:39 +08:00
Rohit Ghumare
dda194f840
Every "Test Your Understanding" quiz placed the correct answer in option B.
Across the 2026 questions in 338 quiz files the correct answer sat at index 1
in 61.5% of cases (uniform would be ~25%), and 107 files had every answer at B,
making the quizzes guessable without reading them.
scripts/debias_quizzes.py rewrites each question's option order with a
deterministic, content-seeded permutation and updates the correct index to
follow the moved answer. It is idempotent: options are canonicalised to a sorted
base before permuting, so re-running produces byte-identical output. Questions
whose options reference each other by position ("all of the above", "both A and
B") are left untouched. The correct-answer value, the option set, and every
explanation are preserved exactly; only order and the index change.
Result: A 23.8% / B 26.3% / C 23.5% / D 26.4%.
The script doubles as a CI guard: `--check` exits non-zero if any quiz is not
de-biased, wired into the curriculum workflow so new lessons cannot regress.
Fixes #368
103 lines
4.0 KiB
JSON
103 lines
4.0 KiB
JSON
{
|
|
"lesson": "11-machine-translation",
|
|
"title": "Machine Translation",
|
|
"questions": [
|
|
{
|
|
"stage": "pre",
|
|
"question": "What does BLEU measure?",
|
|
"options": [
|
|
"N-gram precision (typically 1-4) between hypothesis and reference, with a brevity penalty",
|
|
"Embedding cosine similarity",
|
|
"Language identification accuracy",
|
|
"Character-level F-score"
|
|
],
|
|
"correct": 0,
|
|
"explanation": "BLEU is the geometric mean of 1-4-gram precision against references, plus a brevity penalty."
|
|
},
|
|
{
|
|
"stage": "pre",
|
|
"question": "Why use sacrebleu instead of rolling your own BLEU?",
|
|
"options": [
|
|
"It runs on GPU",
|
|
"It is more accurate",
|
|
"It normalizes tokenization so scores are comparable across papers and runs",
|
|
"It supports streaming"
|
|
],
|
|
"correct": 2,
|
|
"explanation": "sacrebleu freezes tokenization, removing a common source of incomparable BLEU numbers."
|
|
},
|
|
{
|
|
"stage": "check",
|
|
"question": "Which NLLB-specific setting controls the target language during decoding?",
|
|
"options": [
|
|
"src_lang",
|
|
"forced_bos_token_id set to the target language code's token id",
|
|
"num_beams",
|
|
"length_penalty"
|
|
],
|
|
"correct": 1,
|
|
"explanation": "NLLB forces the first decoded token to a target-language code via forced_bos_token_id."
|
|
},
|
|
{
|
|
"stage": "check",
|
|
"question": "Which metric family is the 2026 default for production MT quality where labeled data exists?",
|
|
"options": [
|
|
"Token edit distance",
|
|
"Latency",
|
|
"BLEU alone",
|
|
"Learned metrics such as COMET (and BERTScore/BLEURT) trained on human judgment"
|
|
],
|
|
"correct": 3,
|
|
"explanation": "Learned metrics like COMET correlate more strongly with human judgment than BLEU/chrF alone."
|
|
},
|
|
{
|
|
"stage": "check",
|
|
"question": "When does chrF tend to be more informative than BLEU?",
|
|
"options": [
|
|
"For morphologically rich languages where character-level matches catch inflectional variants BLEU misses",
|
|
"When using beam search",
|
|
"On very short sentences",
|
|
"Whenever a reference exists"
|
|
],
|
|
"correct": 0,
|
|
"explanation": "Character F-score captures partial morphological matches that word-level BLEU undercounts."
|
|
},
|
|
{
|
|
"stage": "post",
|
|
"question": "What is off-target generation in multilingual MT?",
|
|
"options": [
|
|
"Output that drops named entities",
|
|
"The model decodes into the wrong target language (e.g. NLLB outputting Spanish when French was requested)",
|
|
"Output that is too short",
|
|
"Output that misses punctuation"
|
|
],
|
|
"correct": 1,
|
|
"explanation": "Off-target generation is common on rare language pairs; a post-translation language-ID check catches it."
|
|
},
|
|
{
|
|
"stage": "post",
|
|
"question": "Why does fine-tuning on a few thousand high-quality domain pairs often beat much larger noisy web data?",
|
|
"options": [
|
|
"Quality and domain match dominate volume; noisy parallel data introduces drift and hallucination",
|
|
"Smaller datasets train faster",
|
|
"Web data is illegal to use",
|
|
"Larger data overflows GPU memory"
|
|
],
|
|
"correct": 0,
|
|
"explanation": "Clean domain-aligned pairs are the largest production lever; noisy data degrades adaptation."
|
|
},
|
|
{
|
|
"stage": "post",
|
|
"question": "When is an LLM (e.g. GPT-4) likely to outperform a specialized MT model in 2026?",
|
|
"options": [
|
|
"Latency-critical browser translation",
|
|
"Idiomatic content, long context, stylistic adaptation via prompting, or content requiring tone control",
|
|
"Highest throughput batch translation",
|
|
"Small-language pairs with millions of parallel sentences"
|
|
],
|
|
"correct": 1,
|
|
"explanation": "LLMs win on idiomatic, long-context, or style-controlled translation; specialized MT wins on throughput and latency."
|
|
}
|
|
]
|
|
}
|