mirror of
https://github.com/rohitg00/ai-engineering-from-scratch.git
synced 2026-10-02 01:54:39 +08:00
Rohit Ghumare
dda194f840
Every "Test Your Understanding" quiz placed the correct answer in option B.
Across the 2026 questions in 338 quiz files the correct answer sat at index 1
in 61.5% of cases (uniform would be ~25%), and 107 files had every answer at B,
making the quizzes guessable without reading them.
scripts/debias_quizzes.py rewrites each question's option order with a
deterministic, content-seeded permutation and updates the correct index to
follow the moved answer. It is idempotent: options are canonicalised to a sorted
base before permuting, so re-running produces byte-identical output. Questions
whose options reference each other by position ("all of the above", "both A and
B") are left untouched. The correct-answer value, the option set, and every
explanation are preserved exactly; only order and the index change.
Result: A 23.8% / B 26.3% / C 23.5% / D 26.4%.
The script doubles as a CI guard: `--check` exits non-zero if any quiz is not
de-biased, wired into the curriculum workflow so new lessons cannot regress.
Fixes #368
103 lines
4.0 KiB
JSON
103 lines
4.0 KiB
JSON
{
|
|
"lesson": "21-nli-textual-entailment",
|
|
"title": "Natural Language Inference — Textual Entailment",
|
|
"questions": [
|
|
{
|
|
"stage": "pre",
|
|
"question": "What three labels does NLI assign to a (premise, hypothesis) pair?",
|
|
"options": [
|
|
"True / false / unknown",
|
|
"Cause / effect / unrelated",
|
|
"Entailment / contradiction / neutral",
|
|
"Positive / negative / neutral"
|
|
],
|
|
"correct": 2,
|
|
"explanation": "NLI is a 3-way classification over entailment, contradiction, and neutral."
|
|
},
|
|
{
|
|
"stage": "pre",
|
|
"question": "How is NLI used as a zero-shot text classifier?",
|
|
"options": [
|
|
"By prompting an LLM",
|
|
"Verbalize each candidate label as a hypothesis (e.g. 'This text is about sports') and pick the label with the highest entailment score",
|
|
"By computing TF-IDF",
|
|
"By averaging embeddings"
|
|
],
|
|
"correct": 1,
|
|
"explanation": "NLI-as-classifier turns labels into hypotheses; the model picks the max-entailment label."
|
|
},
|
|
{
|
|
"stage": "check",
|
|
"question": "Why is NLI a faithfulness check for RAG outputs?",
|
|
"options": [
|
|
"It is cheap",
|
|
"Checking whether the retrieved context entails each answer claim is exactly the formulation NLI was trained on",
|
|
"It is multilingual",
|
|
"It uses tokenizers"
|
|
],
|
|
"correct": 1,
|
|
"explanation": "Hallucination = answer claims not entailed by retrieved context; NLI directly measures entailment."
|
|
},
|
|
{
|
|
"stage": "check",
|
|
"question": "What does the hypothesis-only baseline expose?",
|
|
"options": [
|
|
"Tokenizer drift",
|
|
"Slow inference",
|
|
"Multilingual gaps",
|
|
"Datasets where the hypothesis alone (without the premise) is predictive of the label, signalling label leakage"
|
|
],
|
|
"correct": 3,
|
|
"explanation": "A high hypothesis-only score on SNLI revealed annotation artifacts; useful for debugging your data."
|
|
},
|
|
{
|
|
"stage": "check",
|
|
"question": "Which NLI model family tops 2026 leaderboards as the standard workhorse?",
|
|
"options": [
|
|
"DeBERTa-v3 variants fine-tuned on MNLI/FEVER/ANLI",
|
|
"Plain Word2Vec",
|
|
"GPT-2",
|
|
"fastText"
|
|
],
|
|
"correct": 0,
|
|
"explanation": "DeBERTa-v3 fine-tuned on MNLI and related corpora is the open NLI workhorse in 2026."
|
|
},
|
|
{
|
|
"stage": "post",
|
|
"question": "Why do sentence-level NLI models drop accuracy on document-length premises?",
|
|
"options": [
|
|
"They were trained on short premises and fail at multi-sentence and multi-hop inference; DocNLI-tuned models handle longer inputs",
|
|
"Larger inputs run slower",
|
|
"Cosine similarity decays",
|
|
"Documents trigger tokenizer drift"
|
|
],
|
|
"correct": 0,
|
|
"explanation": "Training distribution mismatch: single-sentence NLI models lose 20+ F1 on document-length inputs."
|
|
},
|
|
{
|
|
"stage": "post",
|
|
"question": "Why can zero-shot accuracy swing 10+ points based on the hypothesis template?",
|
|
"options": [
|
|
"Templates change the label set",
|
|
"Models are sensitive to phrasing; e.g. 'This text is about {label}' vs '{label}' alone shifts entailment probabilities",
|
|
"Templates change tokenizer behavior",
|
|
"Templates affect model weights"
|
|
],
|
|
"correct": 1,
|
|
"explanation": "Template wording materially shifts entailment scores; tune it on a small held-out set."
|
|
},
|
|
{
|
|
"stage": "post",
|
|
"question": "What is a safe limit to claim about NLI for hallucination detection?",
|
|
"options": [
|
|
"It reduces hallucination as a faithfulness signal but does not eliminate it; combine with retrieval recall and human review",
|
|
"It only works on English",
|
|
"It eliminates hallucination",
|
|
"It requires LLMs"
|
|
],
|
|
"correct": 0,
|
|
"explanation": "NLI is a useful signal but not a complete solution; pair with retrieval metrics and human spot-checks."
|
|
}
|
|
]
|
|
}
|