mirror of
https://github.com/rohitg00/ai-engineering-from-scratch.git
synced 2026-10-02 01:54:39 +08:00
Rohit Ghumare
dda194f840
Every "Test Your Understanding" quiz placed the correct answer in option B.
Across the 2026 questions in 338 quiz files the correct answer sat at index 1
in 61.5% of cases (uniform would be ~25%), and 107 files had every answer at B,
making the quizzes guessable without reading them.
scripts/debias_quizzes.py rewrites each question's option order with a
deterministic, content-seeded permutation and updates the correct index to
follow the moved answer. It is idempotent: options are canonicalised to a sorted
base before permuting, so re-running produces byte-identical output. Questions
whose options reference each other by position ("all of the above", "both A and
B") are left untouched. The correct-answer value, the option set, and every
explanation are preserved exactly; only order and the index change.
Result: A 23.8% / B 26.3% / C 23.5% / D 26.4%.
The script doubles as a CI guard: `--check` exits non-zero if any quiz is not
de-biased, wired into the curriculum workflow so new lessons cannot regress.
Fixes #368
79 lines
2.9 KiB
JSON
79 lines
2.9 KiB
JSON
{
|
|
"lesson": "23-sre-for-ai",
|
|
"title": "SRE for AI — Multi-Agent Incident Response, Runbooks, Predictive Detection",
|
|
"questions": [
|
|
{
|
|
"stage": "pre",
|
|
"question": "What multi-agent shape does the lesson recommend for AI SRE?",
|
|
"options": [
|
|
"Supervisor agent that breaks the incident into sub-queries for specialized log, metric, and runbook agents, then synthesizes and presents to a human",
|
|
"One monolithic agent owning everything",
|
|
"Random selection of one of three agents",
|
|
"Two agents in series with no supervisor"
|
|
],
|
|
"correct": 0,
|
|
"explanation": ""
|
|
},
|
|
{
|
|
"stage": "check",
|
|
"question": "Which auto-remediation set does the lesson call safe?",
|
|
"options": [
|
|
"Modifying IAM policies",
|
|
"Altering databases",
|
|
"Re-architecting service topology",
|
|
"Restart pod, revert a specific deploy, scale a pool within pre-approved bounds, enable a pre-approved feature flag"
|
|
],
|
|
"correct": 3,
|
|
"explanation": ""
|
|
},
|
|
{
|
|
"stage": "check",
|
|
"question": "How does NeuBird Hawkeye use adversarial evaluation to filter hallucinated root causes?",
|
|
"options": [
|
|
"Picks the higher-confidence model's answer always",
|
|
"Runs the same model twice on the same input",
|
|
"Two models independently analyze the same incident; agreement = high confidence, disagreement = escalate to human with both hypotheses",
|
|
"Uses GAN-style training"
|
|
],
|
|
"correct": 2,
|
|
"explanation": ""
|
|
},
|
|
{
|
|
"stage": "check",
|
|
"question": "What does operational memory solve in AI SRE?",
|
|
"options": [
|
|
"Loss of tribal knowledge when teams turn over — runbooks and post-mortems live in a vector DB that agents retrieve on every incident",
|
|
"Network egress filtering",
|
|
"Cold start of inference pods",
|
|
"Token cost attribution"
|
|
],
|
|
"correct": 0,
|
|
"explanation": ""
|
|
},
|
|
{
|
|
"stage": "post",
|
|
"question": "What MIT 2025 result does the lesson cite for pre-incident prediction?",
|
|
"options": [
|
|
"Predictions remain unsolved",
|
|
"100% prediction with 1-second lead",
|
|
"89% of outages predicted 10-15 minutes early using logs + GPU temps + API error patterns",
|
|
"10% with no lead time"
|
|
],
|
|
"correct": 2,
|
|
"explanation": ""
|
|
},
|
|
{
|
|
"stage": "post",
|
|
"question": "What operational constraint does the lesson stress about predictive detection?",
|
|
"options": [
|
|
"Predictions replace runbooks",
|
|
"Predictions without actuation are just dashboards — the operational question is what action (pre-drain, page, auto-scale) the prediction triggers",
|
|
"Predictions should never be wired to action",
|
|
"Predictions are always accurate"
|
|
],
|
|
"correct": 1,
|
|
"explanation": ""
|
|
}
|
|
]
|
|
}
|