mirror of
https://github.com/rohitg00/ai-engineering-from-scratch.git
synced 2026-10-02 01:54:39 +08:00
Rohit Ghumare
dda194f840
Every "Test Your Understanding" quiz placed the correct answer in option B.
Across the 2026 questions in 338 quiz files the correct answer sat at index 1
in 61.5% of cases (uniform would be ~25%), and 107 files had every answer at B,
making the quizzes guessable without reading them.
scripts/debias_quizzes.py rewrites each question's option order with a
deterministic, content-seeded permutation and updates the correct index to
follow the moved answer. It is idempotent: options are canonicalised to a sorted
base before permuting, so re-running produces byte-identical output. Questions
whose options reference each other by position ("all of the above", "both A and
B") are left untouched. The correct-answer value, the option set, and every
explanation are preserved exactly; only order and the index change.
Result: A 23.8% / B 26.3% / C 23.5% / D 26.4%.
The script doubles as a CI guard: `--check` exits non-zero if any quiz is not
de-biased, wired into the curriculum workflow so new lessons cannot regress.
Fixes #368
79 lines
2.4 KiB
JSON
79 lines
2.4 KiB
JSON
{
|
|
"lesson": "18-frontier-safety-frameworks-rsp-pf-fsf",
|
|
"title": "Frontier Safety Frameworks - RSP, PF, FSF",
|
|
"questions": [
|
|
{
|
|
"stage": "pre",
|
|
"question": "What does Anthropic's ASL structure model itself on?",
|
|
"options": [
|
|
"EU AI Act categories",
|
|
"Biosafety levels (BSL)",
|
|
"Internet RFC tiers",
|
|
"ITIL incident severities"
|
|
],
|
|
"correct": 1,
|
|
"explanation": ""
|
|
},
|
|
{
|
|
"stage": "check",
|
|
"question": "Which OpenAI Preparedness Framework v2 criterion captures 'harm occurs fast or cannot be undone'?",
|
|
"options": [
|
|
"Net-new",
|
|
"Plausible",
|
|
"Measurable",
|
|
"Instantaneous-or-irremediable"
|
|
],
|
|
"correct": 3,
|
|
"explanation": ""
|
|
},
|
|
{
|
|
"stage": "check",
|
|
"question": "Which Critical Capability Level was added to DeepMind's FSF v3.0 (September 2025)?",
|
|
"options": [
|
|
"Harmful Manipulation",
|
|
"Cyber Uplift",
|
|
"Bioweapon Uplift",
|
|
"ML R&D Acceleration"
|
|
],
|
|
"correct": 0,
|
|
"explanation": ""
|
|
},
|
|
{
|
|
"stage": "check",
|
|
"question": "What is a 'competitor-adjustment clause' in the 2025+ frontier safety frameworks?",
|
|
"options": [
|
|
"A provision allowing a lab to reduce safety requirements if peer labs ship without comparable safeguards",
|
|
"An export-control exemption",
|
|
"A model-card disclosure requirement",
|
|
"A pricing rule for API access"
|
|
],
|
|
"correct": 0,
|
|
"explanation": ""
|
|
},
|
|
{
|
|
"stage": "post",
|
|
"question": "What are the three standard pillars of a safety case as described in the lesson?",
|
|
"options": [
|
|
"Helpful, honest, harmless",
|
|
"Monitoring, illegibility, incapability",
|
|
"Speed, accuracy, cost",
|
|
"Inner, outer, mesa"
|
|
],
|
|
"correct": 1,
|
|
"explanation": ""
|
|
},
|
|
{
|
|
"stage": "post",
|
|
"question": "Which structural alignment exists across Anthropic's 'Capability Thresholds,' OpenAI's 'High Capability thresholds,' and DeepMind's 'Critical Capability Levels'?",
|
|
"options": [
|
|
"None are publicly documented",
|
|
"Structural alignment: tiered frontier-capability thresholds with published evaluation criteria, even though terminology differs",
|
|
"Identical terminology and numeric cutoffs",
|
|
"All three use the exact same evaluations"
|
|
],
|
|
"correct": 1,
|
|
"explanation": ""
|
|
}
|
|
]
|
|
}
|