mirror of
https://github.com/rohitg00/ai-engineering-from-scratch.git
synced 2026-10-02 01:54:39 +08:00
Rohit Ghumare
dda194f840
Every "Test Your Understanding" quiz placed the correct answer in option B.
Across the 2026 questions in 338 quiz files the correct answer sat at index 1
in 61.5% of cases (uniform would be ~25%), and 107 files had every answer at B,
making the quizzes guessable without reading them.
scripts/debias_quizzes.py rewrites each question's option order with a
deterministic, content-seeded permutation and updates the correct index to
follow the moved answer. It is idempotent: options are canonicalised to a sorted
base before permuting, so re-running produces byte-identical output. Questions
whose options reference each other by position ("all of the above", "both A and
B") are left untouched. The correct-answer value, the option set, and every
explanation are preserved exactly; only order and the index change.
Result: A 23.8% / B 26.3% / C 23.5% / D 26.4%.
The script doubles as a CI guard: `--check` exits non-zero if any quiz is not
de-biased, wired into the curriculum workflow so new lessons cannot regress.
Fixes #368
91 lines
3.2 KiB
JSON
91 lines
3.2 KiB
JSON
{
|
|
"lesson": "08-memory-blocks-sleep-time-compute",
|
|
"title": "Memory Blocks and Sleep-Time Compute",
|
|
"questions": [
|
|
{
|
|
"stage": "pre",
|
|
"question": "What are Letta's three memory tiers?",
|
|
"options": [
|
|
"RAM, swap, disk",
|
|
"Cache, KV, archival",
|
|
"Core, recall, archival",
|
|
"Working, episodic, semantic"
|
|
],
|
|
"correct": 2,
|
|
"explanation": "Letta uses core (always visible), recall (conversation history), and archival (external) tiers."
|
|
},
|
|
{
|
|
"stage": "pre",
|
|
"question": "Which production problem does sleep-time compute target?",
|
|
"options": [
|
|
"Lower embedding cost",
|
|
"Faster JSON parsing",
|
|
"Higher accuracy on math problems",
|
|
"Tail latency from doing memory consolidation on the critical path"
|
|
],
|
|
"correct": 3,
|
|
"explanation": "Sleep-time moves prune/summarize/reconcile off the user-facing path, so primary responses stay fast."
|
|
},
|
|
{
|
|
"stage": "check",
|
|
"question": "Which property is NOT a memory block field in Letta?",
|
|
"options": [
|
|
"limit",
|
|
"embedding_model_version",
|
|
"label",
|
|
"value"
|
|
],
|
|
"correct": 1,
|
|
"explanation": "Blocks carry id, label, value, limit, description; embedding model version is not part of the block schema."
|
|
},
|
|
{
|
|
"stage": "check",
|
|
"question": "Why can the sleep-time agent run a stronger model than the primary?",
|
|
"options": [
|
|
"Memory ops cost half tokens",
|
|
"It receives a private API key",
|
|
"It is off the critical path, so it is not latency-constrained",
|
|
"It is exempt from rate limits"
|
|
],
|
|
"correct": 2,
|
|
"explanation": "Because it does not block user responses, the sleep-time agent can be slower and more expensive."
|
|
},
|
|
{
|
|
"stage": "check",
|
|
"question": "What pattern do the Human and Persona blocks generalize to?",
|
|
"options": [
|
|
"OS processes",
|
|
"Arbitrary user-defined typed editable blocks (Task, Project, Safety, ...)",
|
|
"JSON-RPC channels",
|
|
"Vector embeddings"
|
|
],
|
|
"correct": 1,
|
|
"explanation": "Letta generalizes the two MemGPT blocks to any user-defined block with id, label, value, limit, description."
|
|
},
|
|
{
|
|
"stage": "post",
|
|
"question": "What is silent drift in this pattern?",
|
|
"options": [
|
|
"A primary agent never seeing that the sleep-time agent rewrote a block underneath it; fix with versioning and visible diffs",
|
|
"Embedding model upgrades",
|
|
"Slow disk writes",
|
|
"Rate-limit jitter"
|
|
],
|
|
"correct": 0,
|
|
"explanation": "Versioning blocks and surfacing diffs in the trace makes sleep-time rewrites visible to the primary loop."
|
|
},
|
|
{
|
|
"stage": "post",
|
|
"question": "What replaced inline `Thought:` tokens and the send_message/heartbeat pattern in Letta V1?",
|
|
"options": [
|
|
"Native reasoning emitted on a separate channel and passed through turns",
|
|
"A second LLM dedicated to thoughts",
|
|
"A bigger system prompt",
|
|
"Manual user-typed thoughts"
|
|
],
|
|
"correct": 0,
|
|
"explanation": "Letta V1 (letta_v1_agent) uses provider-level native reasoning, not prompt-shaped thoughts."
|
|
}
|
|
]
|
|
}
|