mirror of
https://github.com/rohitg00/ai-engineering-from-scratch.git
synced 2026-10-02 01:54:39 +08:00
Rohit Ghumare
dda194f840
Every "Test Your Understanding" quiz placed the correct answer in option B.
Across the 2026 questions in 338 quiz files the correct answer sat at index 1
in 61.5% of cases (uniform would be ~25%), and 107 files had every answer at B,
making the quizzes guessable without reading them.
scripts/debias_quizzes.py rewrites each question's option order with a
deterministic, content-seeded permutation and updates the correct index to
follow the moved answer. It is idempotent: options are canonicalised to a sorted
base before permuting, so re-running produces byte-identical output. Questions
whose options reference each other by position ("all of the above", "both A and
B") are left untouched. The correct-answer value, the option set, and every
explanation are preserved exactly; only order and the index change.
Result: A 23.8% / B 26.3% / C 23.5% / D 26.4%.
The script doubles as a CI guard: `--check` exits non-zero if any quiz is not
de-biased, wired into the curriculum workflow so new lessons cannot regress.
Fixes #368
79 lines
2.7 KiB
JSON
79 lines
2.7 KiB
JSON
{
|
|
"lesson": "20-shadow-canary-progressive",
|
|
"title": "Shadow Traffic, Canary Rollout, and Progressive Deployment for LLMs",
|
|
"questions": [
|
|
{
|
|
"stage": "pre",
|
|
"question": "What is the right way to order shadow, canary, and A/B testing for an LLM rollout?",
|
|
"options": [
|
|
"Skip shadow entirely",
|
|
"Shadow (zero-impact compare), then canary (live traffic progressive with gates), then A/B for distinct alternatives once stability is confirmed",
|
|
"Canary first, then shadow, then A/B",
|
|
"A/B first, then shadow, then canary"
|
|
],
|
|
"correct": 1,
|
|
"explanation": ""
|
|
},
|
|
{
|
|
"stage": "check",
|
|
"question": "Which set of five metrics does the lesson gate canary progressions on?",
|
|
"options": [
|
|
"GPU temp, fan speed, queue depth, cost, latency",
|
|
"Just throughput",
|
|
"Latency percentiles, cost per request, error/refusal rate, output length distribution, user-feedback rate",
|
|
"Accuracy on offline eval only"
|
|
],
|
|
"correct": 2,
|
|
"explanation": ""
|
|
},
|
|
{
|
|
"stage": "check",
|
|
"question": "Roughly how much run-to-run accuracy variance does the lesson cite for identical inputs on LLMs?",
|
|
"options": [
|
|
"Under 0.1%",
|
|
"Up to about 15%, due to GPU FP non-associativity plus batch-size variance",
|
|
"Always 50%",
|
|
"Identical outputs run-to-run"
|
|
],
|
|
"correct": 1,
|
|
"explanation": ""
|
|
},
|
|
{
|
|
"stage": "check",
|
|
"question": "What is shadow mode for, in the lesson's framing?",
|
|
"options": [
|
|
"A complete quality test that replaces evals",
|
|
"A smoke test catching cost blow-ups, length regressions, refusal changes, and hard errors — not a quality guarantee",
|
|
"Replacement for rollback",
|
|
"Final production rollout step"
|
|
],
|
|
"correct": 1,
|
|
"explanation": ""
|
|
},
|
|
{
|
|
"stage": "post",
|
|
"question": "What is the correct rollback design per the lesson?",
|
|
"options": [
|
|
"Manual SSH to each pod",
|
|
"Redeploy with new model digest, taking hours",
|
|
"Flip a policy flag and revert the pinned model digest in seconds — no redeploy",
|
|
"Wait for the next release window"
|
|
],
|
|
"correct": 2,
|
|
"explanation": ""
|
|
},
|
|
{
|
|
"stage": "post",
|
|
"question": "Why is cost listed as a gate alongside latency and quality?",
|
|
"options": [
|
|
"A 20% better model can be 3x more expensive per call; shipping that without a cost gate breaks unit economics",
|
|
"Cost is the same across providers",
|
|
"Cost is automatically capped by every provider",
|
|
"Cost is a vanity metric"
|
|
],
|
|
"correct": 0,
|
|
"explanation": ""
|
|
}
|
|
]
|
|
}
|