mirror of
https://github.com/rohitg00/ai-engineering-from-scratch.git
synced 2026-10-02 01:54:39 +08:00
Rohit Ghumare
dda194f840
Every "Test Your Understanding" quiz placed the correct answer in option B.
Across the 2026 questions in 338 quiz files the correct answer sat at index 1
in 61.5% of cases (uniform would be ~25%), and 107 files had every answer at B,
making the quizzes guessable without reading them.
scripts/debias_quizzes.py rewrites each question's option order with a
deterministic, content-seeded permutation and updates the correct index to
follow the moved answer. It is idempotent: options are canonicalised to a sorted
base before permuting, so re-running produces byte-identical output. Questions
whose options reference each other by position ("all of the above", "both A and
B") are left untouched. The correct-answer value, the option set, and every
explanation are preserved exactly; only order and the index change.
Result: A 23.8% / B 26.3% / C 23.5% / D 26.4%.
The script doubles as a CI guard: `--check` exits non-zero if any quiz is not
de-biased, wired into the curriculum workflow so new lessons cannot regress.
Fixes #368
103 lines
4.1 KiB
JSON
103 lines
4.1 KiB
JSON
{
|
|
"lesson": "17-chatbots-rule-to-neural",
|
|
"title": "Chatbots — Rule-Based to Neural to LLM Agents",
|
|
"questions": [
|
|
{
|
|
"stage": "pre",
|
|
"question": "What does a slot-filling state machine do in a rule-based chatbot?",
|
|
"options": [
|
|
"Picks the most fluent reply",
|
|
"Embeds the user message",
|
|
"Tracks which required parameters (date, destination, amount) are still missing and asks for them in sequence",
|
|
"Detects sarcasm"
|
|
],
|
|
"correct": 2,
|
|
"explanation": "Slot filling iteratively collects the structured parameters a task handler needs."
|
|
},
|
|
{
|
|
"stage": "pre",
|
|
"question": "Why is retrieval-based chat resistant to hallucination?",
|
|
"options": [
|
|
"It uses BM25",
|
|
"It uses embeddings",
|
|
"It rejects all queries",
|
|
"It returns a canned response from a curated set rather than generating new text"
|
|
],
|
|
"correct": 3,
|
|
"explanation": "Retrieval surfaces pre-written answers; no generation means no fabricated content."
|
|
},
|
|
{
|
|
"stage": "check",
|
|
"question": "What defines an LLM agent loop versus a single-shot LLM call?",
|
|
"options": [
|
|
"A controller that interleaves LLM calls with tool invocations until the model returns a final answer or the step budget is hit",
|
|
"Use of softmax",
|
|
"Use of greedy decoding",
|
|
"Bigger context window"
|
|
],
|
|
"correct": 0,
|
|
"explanation": "Agents add a plan-act-observe loop with tool calls and a termination condition."
|
|
},
|
|
{
|
|
"stage": "check",
|
|
"question": "Why is hybrid routing (rules + retrieval + LLM agent) the 2026 production default?",
|
|
"options": [
|
|
"It avoids embeddings",
|
|
"No single architecture handles every request well; rules cover destructive actions, retrieval covers FAQ, agents handle ambiguous open-ended queries",
|
|
"It is cheaper to maintain",
|
|
"It removes the need for evaluation"
|
|
],
|
|
"correct": 1,
|
|
"explanation": "Hybrid systems use deterministic rules for risky actions and reserve LLM agents for open-ended queries."
|
|
},
|
|
{
|
|
"stage": "check",
|
|
"question": "What is prompt injection?",
|
|
"options": [
|
|
"A type of tokenizer attack",
|
|
"User-supplied (direct) or document-supplied (indirect) text that tries to override the system prompt or hijack the agent's behavior",
|
|
"A SQL injection variant only",
|
|
"Injecting tokens into embeddings"
|
|
],
|
|
"correct": 1,
|
|
"explanation": "Prompt injection rewrites the agent's behavior via untrusted text in user input or tool outputs."
|
|
},
|
|
{
|
|
"stage": "post",
|
|
"question": "Which OWASP Top 10 (LLM Apps 2025) risk is ranked LLM01?",
|
|
"options": [
|
|
"SQL injection",
|
|
"Broken access control",
|
|
"Insecure deserialization",
|
|
"Prompt injection (direct and indirect)"
|
|
],
|
|
"correct": 3,
|
|
"explanation": "Prompt injection is LLM01 in the OWASP LLM Apps Top 10 (2025)."
|
|
},
|
|
{
|
|
"stage": "post",
|
|
"question": "What mitigation pattern reduces indirect prompt injection by separating planning from execution?",
|
|
"options": [
|
|
"Plan-Verify-Execute: the agent plans first, verifies each action against the plan, then executes — preventing tool outputs from injecting new unplanned actions",
|
|
"Shorter context",
|
|
"Lower temperature",
|
|
"Bigger model"
|
|
],
|
|
"correct": 0,
|
|
"explanation": "PVE checks each step against the agreed plan, so injected instructions from tool outputs are rejected."
|
|
},
|
|
{
|
|
"stage": "post",
|
|
"question": "Why must destructive actions (payments, deletions) route through a structured flow even in an LLM-agent system?",
|
|
"options": [
|
|
"Tools cannot be called",
|
|
"Beam search is unsafe",
|
|
"Confident fabrication, prompt injection, and scope creep mean the LLM cannot be the sole authority for irreversible side effects",
|
|
"LLMs are slow"
|
|
],
|
|
"correct": 2,
|
|
"explanation": "Hallucination and injection make irreversible actions through pure LLM agents unsafe; require deterministic confirmation flows."
|
|
}
|
|
]
|
|
}
|