mirror of
https://github.com/rohitg00/ai-engineering-from-scratch.git
synced 2026-10-02 01:54:39 +08:00
Rohit Ghumare
dda194f840
Every "Test Your Understanding" quiz placed the correct answer in option B.
Across the 2026 questions in 338 quiz files the correct answer sat at index 1
in 61.5% of cases (uniform would be ~25%), and 107 files had every answer at B,
making the quizzes guessable without reading them.
scripts/debias_quizzes.py rewrites each question's option order with a
deterministic, content-seeded permutation and updates the correct index to
follow the moved answer. It is idempotent: options are canonicalised to a sorted
base before permuting, so re-running produces byte-identical output. Questions
whose options reference each other by position ("all of the above", "both A and
B") are left untouched. The correct-answer value, the option set, and every
explanation are preserved exactly; only order and the index change.
Result: A 23.8% / B 26.3% / C 23.5% / D 26.4%.
The script doubles as a CI guard: `--check` exits non-zero if any quiz is not
de-biased, wired into the curriculum workflow so new lessons cannot regress.
Fixes #368
79 lines
2.7 KiB
JSON
79 lines
2.7 KiB
JSON
{
|
|
"lesson": "15-batch-apis",
|
|
"title": "Batch APIs — the 50% Discount as Industry Standard",
|
|
"questions": [
|
|
{
|
|
"stage": "pre",
|
|
"question": "What is the common batch-API offer across OpenAI, Anthropic, and Google in 2026?",
|
|
"options": [
|
|
"90% discount with 7-day turnaround",
|
|
"50% discount with 24-hour turnaround",
|
|
"10% discount with 1-hour turnaround",
|
|
"Free if under 1k tokens"
|
|
],
|
|
"correct": 1,
|
|
"explanation": ""
|
|
},
|
|
{
|
|
"stage": "check",
|
|
"question": "What does \"24-hour turnaround\" actually guarantee in the lesson's framing?",
|
|
"options": [
|
|
"24h is the cache TTL",
|
|
"The provider promises to return within 24 hours, with typical P50 around 2-6 hours",
|
|
"The batch always takes 24 hours",
|
|
"Only batches under 1k requests qualify"
|
|
],
|
|
"correct": 1,
|
|
"explanation": ""
|
|
},
|
|
{
|
|
"stage": "check",
|
|
"question": "How does stacking batch with cached input change the bill versus synchronous uncached on a shared-system-prompt workload?",
|
|
"options": [
|
|
"It only helps if the model is on Vertex",
|
|
"It can drop to roughly 10% of the synchronous-uncached baseline",
|
|
"It increases cost by 50%",
|
|
"It has no effect because caching is automatic"
|
|
],
|
|
"correct": 1,
|
|
"explanation": ""
|
|
},
|
|
{
|
|
"stage": "check",
|
|
"question": "Which workload-triage lane is wrong to default to in 2026 for content pipelines and offline labeling?",
|
|
"options": [
|
|
"Hybrid batch-and-cache",
|
|
"Interactive, because it sounds urgent",
|
|
"Batch, because the user does not see a 24h delay",
|
|
"Semi-interactive with async queue"
|
|
],
|
|
"correct": 1,
|
|
"explanation": ""
|
|
},
|
|
{
|
|
"stage": "post",
|
|
"question": "What is the output-schema trap across providers?",
|
|
"options": [
|
|
"All providers use the same OpenAI JSONL format",
|
|
"Batch file formats differ per provider (OpenAI JSONL, Anthropic JSONL, Vertex BigQuery/GCS), so a portable client needs per-provider adapters",
|
|
"Vertex requires Parquet only",
|
|
"JSONL is unsupported by Anthropic"
|
|
],
|
|
"correct": 1,
|
|
"explanation": ""
|
|
},
|
|
{
|
|
"stage": "post",
|
|
"question": "Per the lesson, what is the simplest decision rule for triaging a workload to batch?",
|
|
"options": [
|
|
"If the prompt is under 1k tokens, batch it",
|
|
"If the user wouldn't notice a 24-hour delivery, always batch (and stack caching)",
|
|
"Batch only when the gateway requires it",
|
|
"If it uses tools, batch it"
|
|
],
|
|
"correct": 1,
|
|
"explanation": ""
|
|
}
|
|
]
|
|
}
|