mirror of
https://github.com/rohitg00/ai-engineering-from-scratch.git
synced 2026-10-02 01:54:39 +08:00
Rohit Ghumare
dda194f840
Every "Test Your Understanding" quiz placed the correct answer in option B.
Across the 2026 questions in 338 quiz files the correct answer sat at index 1
in 61.5% of cases (uniform would be ~25%), and 107 files had every answer at B,
making the quizzes guessable without reading them.
scripts/debias_quizzes.py rewrites each question's option order with a
deterministic, content-seeded permutation and updates the correct index to
follow the moved answer. It is idempotent: options are canonicalised to a sorted
base before permuting, so re-running produces byte-identical output. Questions
whose options reference each other by position ("all of the above", "both A and
B") are left untouched. The correct-answer value, the option set, and every
explanation are preserved exactly; only order and the index change.
Result: A 23.8% / B 26.3% / C 23.5% / D 26.4%.
The script doubles as a CI guard: `--check` exits non-zero if any quiz is not
de-biased, wired into the curriculum workflow so new lessons cannot regress.
Fixes #368
79 lines
2.9 KiB
JSON
79 lines
2.9 KiB
JSON
{
|
|
"lesson": "01-instruction-following-alignment-signal",
|
|
"title": "Instruction-Following as Alignment Signal",
|
|
"questions": [
|
|
{
|
|
"stage": "pre",
|
|
"question": "Why does a raw pre-trained language model often respond to 'write a Python function that reverses a list' with another prompt instead of code?",
|
|
"options": [
|
|
"Its temperature defaults to zero",
|
|
"Its vocabulary lacks Python tokens",
|
|
"It is trained to complete web-style text, where prompts continue with more prompts",
|
|
"It cannot represent function signatures without fine-tuning"
|
|
],
|
|
"correct": 2,
|
|
"explanation": ""
|
|
},
|
|
{
|
|
"stage": "check",
|
|
"question": "Which loss does the InstructGPT reward model use over pairwise preference labels?",
|
|
"options": [
|
|
"Cross-entropy between predicted and true reward",
|
|
"Mean squared error on labeler scores",
|
|
"Hinge loss with a fixed margin",
|
|
"Bradley-Terry: -log sigmoid(r(x, y_w) - r(x, y_l))"
|
|
],
|
|
"correct": 3,
|
|
"explanation": ""
|
|
},
|
|
{
|
|
"stage": "check",
|
|
"question": "What is the KL penalty in stage 3 of InstructGPT primarily protecting against?",
|
|
"options": [
|
|
"Catastrophic forgetting of tokenizer statistics",
|
|
"The optimizer finding adversarial high-reward strings that exploit the reward model",
|
|
"Overflowing context windows during sampling",
|
|
"Numerical instability in PPO gradient updates"
|
|
],
|
|
"correct": 1,
|
|
"explanation": ""
|
|
},
|
|
{
|
|
"stage": "check",
|
|
"question": "What problem does PPO-ptx mix into the RL objective to mitigate?",
|
|
"options": [
|
|
"The alignment tax: regression on benchmarks RLHF did not target",
|
|
"Tokenizer mismatch between SFT and RM",
|
|
"The KL term becoming negative",
|
|
"The reward model drifting during PPO"
|
|
],
|
|
"correct": 0,
|
|
"explanation": ""
|
|
},
|
|
{
|
|
"stage": "post",
|
|
"question": "A 1.3B InstructGPT was preferred over the 175B base GPT-3 about 70% of the time. What does this say about alignment and capability?",
|
|
"options": [
|
|
"Alignment is a different axis from capability, and the base model sets the capability floor",
|
|
"Preference rates above 50% prove the proxy reward equals human values",
|
|
"Capability and alignment are the same axis at scale",
|
|
"Bigger base models always lose to small aligned models"
|
|
],
|
|
"correct": 0,
|
|
"explanation": ""
|
|
},
|
|
{
|
|
"stage": "post",
|
|
"question": "Which of the following is the reward model in InstructGPT initialized from?",
|
|
"options": [
|
|
"The base pre-trained model with the LM head intact",
|
|
"The SFT model with the LM head replaced by a scalar head",
|
|
"A separate encoder-only transformer",
|
|
"A random scalar regressor with no pretraining"
|
|
],
|
|
"correct": 1,
|
|
"explanation": ""
|
|
}
|
|
]
|
|
}
|