mirror of
https://github.com/rohitg00/ai-engineering-from-scratch.git
synced 2026-10-02 01:54:39 +08:00
Rohit Ghumare
dda194f840
Every "Test Your Understanding" quiz placed the correct answer in option B.
Across the 2026 questions in 338 quiz files the correct answer sat at index 1
in 61.5% of cases (uniform would be ~25%), and 107 files had every answer at B,
making the quizzes guessable without reading them.
scripts/debias_quizzes.py rewrites each question's option order with a
deterministic, content-seeded permutation and updates the correct index to
follow the moved answer. It is idempotent: options are canonicalised to a sorted
base before permuting, so re-running produces byte-identical output. Questions
whose options reference each other by position ("all of the above", "both A and
B") are left untouched. The correct-answer value, the option set, and every
explanation are preserved exactly; only order and the index change.
Result: A 23.8% / B 26.3% / C 23.5% / D 26.4%.
The script doubles as a CI guard: `--check` exits non-zero if any quiz is not
de-biased, wired into the curriculum workflow so new lessons cannot regress.
Fixes #368
91 lines
3.4 KiB
JSON
91 lines
3.4 KiB
JSON
{
|
|
"lesson": "32-minimal-agent-workbench",
|
|
"title": "The Minimal Agent Workbench",
|
|
"questions": [
|
|
{
|
|
"stage": "pre",
|
|
"question": "What three files form the smallest useful workbench?",
|
|
"options": [
|
|
"model.py, prompts.py, tools.py",
|
|
"Dockerfile, Makefile, .gitignore",
|
|
"AGENTS.md (router), agent_state.json (state), task_board.json (queue)",
|
|
"README.md, CHANGELOG.md, LICENSE"
|
|
],
|
|
"correct": 2,
|
|
"explanation": "A short router, durable state, and a task queue are the floor."
|
|
},
|
|
{
|
|
"stage": "pre",
|
|
"question": "What is the lesson's framing of AGENTS.md?",
|
|
"options": [
|
|
"A secret kept out of the repo",
|
|
"A pure prompt-cache key",
|
|
"A short router that points at deeper docs and the state and board",
|
|
"A 3000-line onboarding manual"
|
|
],
|
|
"correct": 2,
|
|
"explanation": "AGENTS.md is a router, not a manual; long manuals get ignored."
|
|
},
|
|
{
|
|
"stage": "check",
|
|
"question": "What does Augment Code's data say about a good AGENTS.md?",
|
|
"options": [
|
|
"A good AGENTS.md gives a quality jump equivalent to upgrading from Haiku to Opus; a bad one is worse than no file",
|
|
"Length always helps",
|
|
"Only Claude reads it",
|
|
"It has no measurable effect"
|
|
],
|
|
"correct": 0,
|
|
"explanation": "Augment Code's measurement: best files are model-upgrade-shaped, worst are worse than nothing."
|
|
},
|
|
{
|
|
"stage": "check",
|
|
"question": "Why does the lesson recommend file-backed state over chat-history state?",
|
|
"options": [
|
|
"Files are smaller",
|
|
"Disk is cheaper",
|
|
"Chat history is volatile; the file survives session death, conversation trimming, and tool resets",
|
|
"It is required by SOC 2"
|
|
],
|
|
"correct": 2,
|
|
"explanation": "Sessions die and chat gets trimmed; the file is the durable system of record."
|
|
},
|
|
{
|
|
"stage": "check",
|
|
"question": "What does the lesson say happens when conflicting instructions land in AGENTS.md?",
|
|
"options": [
|
|
"Cost rises slightly",
|
|
"Nothing",
|
|
"Latency drops",
|
|
"Conflicting instructions silently drop the agent from interactive to greedy mode (AMBIG-SWE: 48.8% to 28% resolve rate)"
|
|
],
|
|
"correct": 3,
|
|
"explanation": "AMBIG-SWE measured a large resolve-rate drop when contradictions appear; number priorities instead."
|
|
},
|
|
{
|
|
"stage": "post",
|
|
"question": "Why does the lesson recommend cross-tool symlinks like CLAUDE.md -> AGENTS.md?",
|
|
"options": [
|
|
"To satisfy MIT",
|
|
"To pass auditing",
|
|
"So a single source of truth fans out to every coding agent without forking",
|
|
"To save bytes"
|
|
],
|
|
"correct": 2,
|
|
"explanation": "Symlinks (or Nx-style generators) keep one canonical source across Claude Code, Codex, Cursor, Copilot, etc."
|
|
},
|
|
{
|
|
"stage": "post",
|
|
"question": "What pattern do nested AGENTS.md files follow?",
|
|
"options": [
|
|
"Only the root file is read",
|
|
"Walk from the working file toward the repo root, concatenate every AGENTS.md found on the way (nearest wins; sub-directories extend root)",
|
|
"Random selection",
|
|
"Alphabetical merge"
|
|
],
|
|
"correct": 1,
|
|
"explanation": "OpenAI ships 88 AGENTS.md files across its main repo; tools concatenate nearest-up-tree."
|
|
}
|
|
]
|
|
}
|