mirror of
https://github.com/rohitg00/ai-engineering-from-scratch.git
synced 2026-10-02 01:54:39 +08:00
Rohit Ghumare
dda194f840
Every "Test Your Understanding" quiz placed the correct answer in option B.
Across the 2026 questions in 338 quiz files the correct answer sat at index 1
in 61.5% of cases (uniform would be ~25%), and 107 files had every answer at B,
making the quizzes guessable without reading them.
scripts/debias_quizzes.py rewrites each question's option order with a
deterministic, content-seeded permutation and updates the correct index to
follow the moved answer. It is idempotent: options are canonicalised to a sorted
base before permuting, so re-running produces byte-identical output. Questions
whose options reference each other by position ("all of the above", "both A and
B") are left untouched. The correct-answer value, the option set, and every
explanation are preserved exactly; only order and the index change.
Result: A 23.8% / B 26.3% / C 23.5% / D 26.4%.
The script doubles as a CI guard: `--check` exits non-zero if any quiz is not
de-biased, wired into the curriculum workflow so new lessons cannot regress.
Fixes #368
79 lines
2.7 KiB
JSON
79 lines
2.7 KiB
JSON
{
|
|
"lesson": "25-security-secrets-audit",
|
|
"title": "Security — Secrets, API Key Rotation, Audit Logs, Guardrails",
|
|
"questions": [
|
|
{
|
|
"stage": "pre",
|
|
"question": "What is the 2026 standard pattern for LLM service credentials?",
|
|
"options": [
|
|
"Hardcode API keys in config files for speed",
|
|
"Email the key to each engineer",
|
|
"Store keys in a Slack channel",
|
|
"Centralized vault pulled by an AI gateway at runtime via IAM role; rotate in vault and all apps pick up in minutes"
|
|
],
|
|
"correct": 3,
|
|
"explanation": ""
|
|
},
|
|
{
|
|
"stage": "check",
|
|
"question": "What rotation cadence does the lesson recommend for API keys, vault root tokens, and CI/CD credentials?",
|
|
"options": [
|
|
"Within 90 days, automated where possible, logged and tracked when manual",
|
|
"Only when leaked",
|
|
"Never",
|
|
"Every 5 years"
|
|
],
|
|
"correct": 0,
|
|
"explanation": ""
|
|
},
|
|
{
|
|
"stage": "check",
|
|
"question": "Why is consistent tokenization (Mesh approach) used for PII scrubbing?",
|
|
"options": [
|
|
"It encrypts the prompt to the model",
|
|
"It is required by ISO 27001",
|
|
"Same source value maps to the same placeholder, so the LLM preserves code and relationship semantics across the prompt",
|
|
"It uses less memory than regex"
|
|
],
|
|
"correct": 2,
|
|
"explanation": ""
|
|
},
|
|
{
|
|
"stage": "check",
|
|
"question": "What egress posture does the lesson recommend for LLM service subnets?",
|
|
"options": [
|
|
"Whitelist a small set of domains (api.openai.com, api.anthropic.com, vector DB, vault) and drop everything else, with an allowlist-only DNS resolver",
|
|
"Block all egress including providers",
|
|
"Allow DNS but block HTTP",
|
|
"Allow all outbound traffic"
|
|
],
|
|
"correct": 0,
|
|
"explanation": ""
|
|
},
|
|
{
|
|
"stage": "post",
|
|
"question": "What did the 2026 Vercel supply-chain incident teach about CI/CD credentials?",
|
|
"options": [
|
|
"CI/CD credentials are low-risk and can stay in env files",
|
|
"CI/CD secrets cannot be stolen",
|
|
"Vercel was unaffected",
|
|
"CI/CD credentials are prod-equivalent — store in vault, scope narrowly, rotate aggressively"
|
|
],
|
|
"correct": 3,
|
|
"explanation": ""
|
|
},
|
|
{
|
|
"stage": "post",
|
|
"question": "Which audit log fields does the lesson recommend keeping for every LLM call?",
|
|
"options": [
|
|
"Only the cost",
|
|
"Timestamp, user/tenant, prompt hash (not raw), model + version, token counts, cost, response hash, any guardrail trips",
|
|
"Just the raw prompt",
|
|
"Only the response"
|
|
],
|
|
"correct": 1,
|
|
"explanation": ""
|
|
}
|
|
]
|
|
}
|