🧠 AI Engineering from Scratch

From linear algebra to autonomous agent swarms. learn AI with AI, then ship the tools.

🧭 Quick Navigation

🚀 Get Started  ·  🤖 AI-Native  ·  🗺️ The Journey  ·  🧰 Toolkit  ·  📚 Glossary  ·  🛣️ Roadmap  ·  🤝 Contribute  ·  🌐 Website

---

💬 "84% of students already use AI tools. Only 18% feel prepared to use them professionally.

This course closes that gap."

272+ lessons. 20 phases. ~306 hours. From linear algebra to autonomous agent swarms. Python, TypeScript, Rust, Julia. Every lesson produces something reusable: prompts, skills, agents, and MCP servers.

You don't just learn AI. You learn AI with AI. Then you build real things. Then you ship tools others can use.

🆚 Why This Course?

📺 Traditional Courses	🧠 This Course
Scope One slice (NLP or Vision or Agents)	Scope 🌍 Everything — math · ML · DL · NLP · vision · speech · transformers · LLMs · agents · swarms
Languages Python only	Languages 🐍 Python · 🟦 TypeScript · 🦀 Rust · 🟣 Julia
Output "I learned something"	Output 📦 A portfolio of tools, prompts, skills, and agents you can install
Depth Surface-level or theory-heavy	Depth 🔬 Build from scratch first, then use frameworks
Format Videos you watch	Format 💻 Runnable code + docs + web app + AI-powered quizzes
Style Passive consumption	Style 🤖 AI-native — Claude Code skills test you as you go

---

🤖 AI-Native Learning

This isn't a course you watch. It's a course you use with your AI coding agent.

🎯 Learn with AI, not just about AI

# 🧪 Find where to start based on what you already know
/find-your-level

# ✅ Quiz yourself after completing a phase
/check-understanding 3

# 📦 Every lesson produces a reusable artifact
ls phases/03-deep-learning-core/05-loss-functions/outputs/
# ├── prompt-loss-function-selector.md
# └── prompt-loss-debugger.md

🛠️ Built-in Claude Code Skills

🎴 Skill	⚡ What it does
	🧭 10-question quiz that maps your knowledge to a starting phase and builds a personalized path with hour estimates
	📝 Per-phase quiz (8 questions) with feedback and specific lessons to review

🚢 Every Lesson Ships Something

Other courses end with "congratulations, you learned X." Our lessons end with a reusable tool:

📝 Prompts Paste into any AI assistant for expert-level help

🎴 Skills Install into Claude Code, Cursor, or any agent

🤖 Agents Deploy as autonomous workers

🔌 MCP Servers Plug into any MCP-compatible AI app

277-term searchable glossary. Full lesson catalog. ~306 hours of content with per-lesson time estimates.
🌐 Browse the website →

---

🗺️ The Journey

20 phases · 272+ lessons · click any phase to expand

Legend: hands-on implementation  ·  concept + intuition

12 lessons 🛠️ Get your environment ready for everything that follows. # Lesson Type Lang 01 Dev Environment 🐍 🟦 🦀 02 Git & Collaboration — 03 GPU Setup & Cloud 🐍 04 APIs & Keys 🐍 🟦 05 Jupyter Notebooks 🐍 06 Python Environments 🐍 07 Docker for AI 🐍 08 Editor Setup — 09 Data Management 🐍 10 Terminal & Shell — 11 Linux for AI — 12 Debugging & Profiling 🐍

🟣 Phase 1 — Math Foundations  22 lessons  The intuition behind every AI algorithm, through code.

#	Lesson	Type	Lang
01	Linear Algebra Intuition		🐍 🟣
02	Vectors, Matrices & Operations		🐍 🟣
03	Matrix Transformations & Eigenvalues		🐍 🟣
04	Calculus for ML: Derivatives & Gradients		🐍
05	Chain Rule & Automatic Differentiation		🐍
06	Probability & Distributions		🐍
07	Bayes' Theorem & Statistical Thinking		🐍
08	Optimization: Gradient Descent Family		🐍
09	Information Theory: Entropy, KL Divergence		🐍
10	Dimensionality Reduction: PCA, t-SNE, UMAP		🐍
11	Singular Value Decomposition		🐍 🟣
12	Tensor Operations		🐍
13	Numerical Stability		🐍
14	Norms & Distances		🐍
15	Statistics for ML		🐍
16	Sampling Methods		🐍
17	Linear Systems		🐍
18	Convex Optimization		🐍
19	Complex Numbers for AI		🐍
20	The Fourier Transform		🐍
21	Graph Theory for ML		🐍
22	Stochastic Processes		🐍

🔵 Phase 2 — ML Fundamentals  18 lessons  Classical ML — still the backbone of most production AI.

#	Lesson	Type	Lang
01	What Is Machine Learning		🐍
02	Linear Regression from Scratch		🐍
03	Logistic Regression & Classification		🐍
04	Decision Trees & Random Forests		🐍
05	Support Vector Machines		🐍
06	KNN & Distance Metrics		🐍
07	Unsupervised Learning: K-Means, DBSCAN		🐍
08	Feature Engineering & Selection		🐍
09	Model Evaluation: Metrics, Cross-Validation		🐍
10	Bias, Variance & the Learning Curve		🐍
11	Ensemble Methods: Boosting, Bagging, Stacking		🐍
12	Hyperparameter Tuning		🐍
13	ML Pipelines & Experiment Tracking		🐍
14	Naive Bayes		🐍
15	Time Series Fundamentals		🐍
16	Anomaly Detection		🐍
17	Handling Imbalanced Data		🐍
18	Feature Selection		🐍

🟢 Phase 3 — Deep Learning Core  13 lessons  Neural networks from first principles. No frameworks until you build one.

#	Lesson	Type	Lang
01	The Perceptron: Where It All Started		🐍
02	Multi-Layer Networks & Forward Pass		🐍
03	Backpropagation from Scratch		🐍
04	Activation Functions: ReLU, Sigmoid, GELU & Why		🐍
05	Loss Functions: MSE, Cross-Entropy, Contrastive		🐍
06	Optimizers: SGD, Momentum, Adam, AdamW		🐍
07	Regularization: Dropout, Weight Decay, BatchNorm		🐍
08	Weight Initialization & Training Stability		🐍
09	Learning Rate Schedules & Warmup		🐍
10	Build Your Own Mini Framework		🐍
11	Introduction to PyTorch		🐍
12	Introduction to JAX		🐍
13	Debugging Neural Networks		🐍

🟠 Phase 4 — Computer Vision  28 lessons  From pixels to understanding — image, video, 3D, VLMs, and world models.

#	Lesson	Type	Lang
01	Image Fundamentals: Pixels, Channels, Color Spaces		🐍
02	Convolutions from Scratch		🐍
03	CNNs: LeNet to ResNet		🐍
04	Image Classification		🐍
05	Transfer Learning & Fine-Tuning		🐍
06	Object Detection — YOLO from Scratch		🐍
07	Semantic Segmentation — U-Net		🐍
08	Instance Segmentation — Mask R-CNN		🐍
09	Image Generation — GANs		🐍
10	Image Generation — Diffusion Models		🐍
11	Stable Diffusion — Architecture & Fine-Tuning		🐍
12	Video Understanding — Temporal Modeling		🐍
13	3D Vision: Point Clouds, NeRFs		🐍
14	Vision Transformers (ViT)		🐍
15	Real-Time Vision: Edge Deployment		🐍 🦀
16	Build a Complete Vision Pipeline		🐍
17	Self-Supervised Vision — SimCLR, DINO, MAE		🐍
18	Open-Vocabulary Vision — CLIP		🐍
19	OCR & Document Understanding		🐍
20	Image Retrieval & Metric Learning		🐍
21	Keypoint Detection & Pose Estimation		🐍
22	3D Gaussian Splatting from Scratch		🐍
23	Diffusion Transformers & Rectified Flow		🐍
24	SAM 3 & Open-Vocabulary Segmentation		🐍
25	Vision-Language Models (ViT-MLP-LLM)		🐍
26	Monocular Depth & Geometry Estimation		🐍
27	Multi-Object Tracking & Video Memory		🐍
28	World Models & Video Diffusion		🐍

🔴 Phase 5 — NLP: Foundations to Advanced  29 lessons  Language is the interface to intelligence.

#	Lesson	Type	Lang
01	Text Processing: Tokenization, Stemming, Lemmatization		🐍
02	Bag of Words, TF-IDF & Text Representation		🐍
03	Word Embeddings: Word2Vec from Scratch		🐍
04	GloVe, FastText & Subword Embeddings		🐍
05	Sentiment Analysis		🐍
06	Named Entity Recognition (NER)		🐍
07	POS Tagging & Syntactic Parsing		🐍
08	Text Classification — CNNs & RNNs for Text		🐍
09	Sequence-to-Sequence Models		🐍
10	Attention Mechanism — The Breakthrough		🐍
11	Machine Translation		🐍
12	Text Summarization		🐍
13	Question Answering Systems		🐍
14	Information Retrieval & Search		🐍
15	Topic Modeling: LDA, BERTopic		🐍
16	Text Generation		🐍
17	Chatbots: Rule-Based to Neural		🐍
18	Multilingual NLP		🐍
19	Subword Tokenization: BPE, WordPiece, Unigram, SentencePiece		🐍
20	Structured Outputs & Constrained Decoding		🐍
21	NLI & Textual Entailment		🐍
22	Embedding Models Deep Dive		🐍
23	Chunking Strategies for RAG		🐍
24	Coreference Resolution		🐍
25	Entity Linking & Disambiguation		🐍
26	Relation Extraction & Knowledge Graph Construction		🐍
27	LLM Evaluation: RAGAS, DeepEval, G-Eval		🐍
28	Long-Context Evaluation: NIAH, RULER, LongBench, MRCR		🐍
29	Dialogue State Tracking		🐍

🟢 Phase 6 — Speech & Audio  17 lessons  Hear, understand, speak.

#	Lesson	Type	Lang
01	Audio Fundamentals: Waveforms, Sampling, FFT		🐍
02	Spectrograms, Mel Scale & Audio Features		🐍
03	Audio Classification		🐍
04	Speech Recognition (ASR)		🐍
05	Whisper: Architecture & Fine-Tuning		🐍
06	Speaker Recognition & Verification		🐍
07	Text-to-Speech (TTS)		🐍
08	Voice Cloning & Voice Conversion		🐍
09	Music Generation		🐍
10	Audio-Language Models		🐍
11	Real-Time Audio Processing		🐍 🦀
12	Build a Voice Assistant Pipeline		🐍
13	Neural Audio Codecs — EnCodec, SNAC, Mimi, DAC		🐍
14	Voice Activity Detection & Turn-Taking		🐍
15	Streaming Speech-to-Speech — Moshi, Hibiki		🐍
16	Voice Anti-Spoofing & Audio Watermarking		🐍
17	Audio Evaluation — WER, MOS, MMAU, Leaderboards		🐍

🟢 Phase 7 — Transformers Deep Dive  14 lessons  The architecture that changed everything.

#	Lesson	Type	Lang
01	Why Transformers: The Problems with RNNs		🐍
02	Self-Attention from Scratch		🐍
03	Multi-Head Attention		🐍
04	Positional Encoding: Sinusoidal, RoPE, ALiBi		🐍
05	The Full Transformer: Encoder + Decoder		🐍
06	BERT — Masked Language Modeling		🐍
07	GPT — Causal Language Modeling		🐍
08	T5, BART — Encoder-Decoder Models		🐍
09	Vision Transformers (ViT)		🐍
10	Audio Transformers — Whisper Architecture		🐍
11	Mixture of Experts (MoE)		🐍
12	KV Cache, Flash Attention & Inference Optimization		🐍
13	Scaling Laws		🐍
14	Build a Transformer from Scratch		🐍

💗 Phase 8 — Generative AI  14 lessons  Create images, video, audio, 3D, and more.

#	Lesson	Type	Lang
01	Generative Models: Taxonomy & History		🐍
02	Autoencoders & VAE		🐍
03	GANs: Generator vs Discriminator		🐍
04	Conditional GANs & Pix2Pix		🐍
05	StyleGAN		🐍
06	Diffusion Models — DDPM from Scratch		🐍
07	Latent Diffusion & Stable Diffusion		🐍
08	ControlNet, LoRA & Conditioning		🐍
09	Inpainting, Outpainting & Editing		🐍
10	Video Generation		🐍
11	Audio Generation		🐍
12	3D Generation		🐍
13	Flow Matching & Rectified Flows		🐍
14	Evaluation: FID, CLIP Score		🐍

🟣 Phase 9 — Reinforcement Learning  12 lessons  The foundation of RLHF and game-playing AI.

#	Lesson	Type	Lang
01	MDPs, States, Actions & Rewards		🐍
02	Dynamic Programming		🐍
03	Monte Carlo Methods		🐍
04	Q-Learning, SARSA		🐍
05	Deep Q-Networks (DQN)		🐍
06	Policy Gradients — REINFORCE		🐍
07	Actor-Critic — A2C, A3C		🐍
08	PPO		🐍
09	Reward Modeling & RLHF		🐍
10	Multi-Agent RL		🐍
11	Sim-to-Real Transfer		🐍
12	RL for Games		🐍

🟧 Phase 10 — LLMs from Scratch  14 lessons  Build, train, and understand large language models.

#	Lesson	Type	Lang
01	Tokenizers: BPE, WordPiece, SentencePiece		🐍
02	Building a Tokenizer from Scratch		🐍
03	Data Pipelines for Pre-Training		🐍
04	Pre-Training a Mini GPT (124M)		🐍
05	Distributed Training, FSDP, DeepSpeed		🐍
06	Instruction Tuning — SFT		🐍
07	RLHF — Reward Model + PPO		🐍
08	DPO — Direct Preference Optimization		🐍
09	Constitutional AI & Self-Improvement		🐍
10	Evaluation — Benchmarks, Evals		🐍
11	Quantization: INT8, GPTQ, AWQ, GGUF		🐍 🦀
12	Inference Optimization		🐍
13	Building a Complete LLM Pipeline		🐍
14	Open Models: Architecture Walkthroughs		🐍

🟥 Phase 11 — LLM Engineering  15 lessons  Put LLMs to work in production.

#	Lesson	Type	Lang
01	Prompt Engineering: Techniques & Patterns		🐍
02	Few-Shot, CoT, Tree-of-Thought		🐍
03	Structured Outputs		🐍 🟦
04	Embeddings & Vector Representations		🐍
05	Context Engineering		🐍 🟦
06	RAG: Retrieval-Augmented Generation		🐍 🟦
07	Advanced RAG: Chunking, Reranking		🐍
08	Fine-Tuning with LoRA & QLoRA		🐍
09	Function Calling & Tool Use		🐍
10	Evaluation & Testing		🐍
11	Caching, Rate Limiting & Cost		🐍
12	Guardrails & Safety		🐍
13	Building a Production LLM App		🐍
14	Model Context Protocol (MCP)		🐍
15	Prompt Caching & Context Caching		🐍

🟩 Phase 12 — Multimodal AI  11 lessons  See, hear, read, and reason across modalities.

#	Lesson	Type	Lang
01	Multimodal Representations		—
02	CLIP: Vision + Language		🐍
03	Vision-Language Models		🐍
04	Audio-Language Models		🐍
05	Document Understanding		🐍
06	Video-Language Models		🐍
07	Multimodal RAG		🐍 🟦
08	Multimodal Agents		🐍 🟦
09	Text-to-Image Pipelines		🐍
10	Text-to-Video Pipelines		🐍
11	Any-to-Any Models		🐍

🟦 Phase 13 — Tools & Protocols  10 lessons  The interfaces between AI and the real world.

#	Lesson	Type	Lang
01	Function Calling Deep Dive		🐍 🟦
02	Tool Use Patterns		🟦
03	MCP: Model Context Protocol		—
04	Building MCP Servers		🟦 🐍
05	Building MCP Clients		🟦 🐍
06	MCP Resources, Prompts & Sampling		🟦
07	Structured Output Schemas		🟦 🐍
08	API Design for AI		🟦
09	Browser Automation & Web Agents		🟦
10	Build a Complete Tool Ecosystem		🟦 🐍

🟧 Phase 14 — Agent Engineering  15 lessons  Build agents from first principles.

#	Lesson	Type	Lang
01	The Agent Loop		🐍 🟦
02	Tool Dispatch & Registration		🟦
03	Planning: TodoWrite, DAGs		🟦
04	Memory: Short-Term, Long-Term, Episodic		🟦 🐍
05	Context Window Management		🟦
06	Context Compression & Summarization		🟦
07	Subagents: Delegation		🟦
08	Skills & Knowledge Loading		🟦
09	Permissions, Sandboxing & Safety		🟦 🦀
10	File-Based Task Systems		🟦
11	Background Task Execution		🟦
12	Error Recovery & Self-Healing		🟦
13	Hooks: PreToolUse, PostToolUse		🟦
14	Eval-Driven Agent Development		🐍 🟦
15	Build a Complete AI Agent		🟦

⬜ Phase 15 — Autonomous Systems  11 lessons  Agents that run without human intervention safely.

#	Lesson	Type	Lang
01	What Makes a System Autonomous		—
02	Autonomous Loops		🟦 🐍
03	Self-Healing Agents		🟦
04	AutoResearch: Autonomous Research		🟦 🐍
05	Eval-Driven Loops		🟦
06	Human-in-the-Loop		🟦
07	Continuous Agents		🟦
08	Cost-Aware Autonomous Systems		🟦
09	Monitoring & Observability		🟦 🦀
10	Safety Boundaries		🟦
11	Build an Autonomous Coding Agent		🟦

🟩 Phase 16 — Multi-Agent & Swarms  14 lessons  Coordination, emergence, and collective intelligence.

#	Lesson	Type	Lang
01	Why Multi-Agent		—
02	Agent Teams: Roles & Delegation		🟦
03	Communication Protocols		🟦
04	Shared State & Coordination		🟦 🦀
05	Message Passing & Mailboxes		🟦
06	Task Markets		🟦
07	Consensus Algorithms		🟦 🦀
08	Swarm Intelligence		🐍 🟦
09	Agent Economies		🟦
10	Worktree Isolation		🟦
11	Hierarchical Swarms		🟦
12	Self-Organizing Systems		🟦 🦀
13	DAG-Based Orchestration		🟦 🦀
14	Build an Autonomous Swarm		🟦 🦀

⬛ Phase 17 — Infrastructure & Production  11 lessons  Ship AI to the real world.

#	Lesson	Type	Lang
01	Model Serving		🐍
02	Docker for AI Workloads		🐍 🦀
03	Kubernetes for AI		🐍
04	Edge Deployment: ONNX, WASM		🐍 🦀
05	Observability		🟦 🦀
06	Cost Optimization		🟦
07	CI/CD for ML		🐍
08	A/B Testing & Feature Flags		🐍 🟦
09	Data Pipelines		🐍 🦀
10	Security: Red Teaming, Defense		🐍 🟦
11	Build a Production AI Platform		🐍 🟦 🦀

🟪 Phase 18 — Ethics, Safety & Alignment  6 lessons  Build AI that helps humanity. Not optional.

#	Lesson	Type	Lang
01	AI Ethics: Bias, Fairness		—
02	Alignment: What & Why		—
03	Red Teaming & Adversarial Testing		🐍
04	Responsible AI Frameworks		—
05	Privacy: Differential Privacy, FL		🐍
06	Interpretability: SHAP, Attention		🐍

🏆 Phase 19 — Capstone Projects  5 projects  Prove everything you learned.

#	Project	Combines	Lang
01	🤖 Build a Mini GPT & Chat Interface	Phases 1, 3, 7, 10	🐍 🟦
02	🔍 Build a Multimodal RAG System	Phases 5, 11, 12, 13	🐍 🟦
03	🧪 Build an Autonomous Research Agent	Phases 14, 15, 6	🟦 🐍
04	👥 Build a Multi-Agent Dev Team	Phases 14, 15, 16, 17	🟦 🦀
05	🚀 Build a Production AI Platform	All phases	🐍 🟦 🦀

---

🧰 Course Output: The Toolkit

Other courses give you a certificate. This one gives you a toolkit.

Every lesson produces a reusable artifact — a prompt, skill, agent, or MCP server you can install and use immediately. By the end of the course you have:

outputs/
├── 📝 prompts/         Prompt templates for every AI task
├── 🎴 skills/          SKILL.md files for AI coding agents
├── 🤖 agents/          Agent definitions ready to deploy
└── 🔌 mcp-servers/     MCP servers you built during the course

💡 Install them with SkillKit. Plug them into Claude Code, Cursor, or any AI agent. These are real tools, not homework.

---

📐 How Each Lesson Works

phases/XX-phase-name/NN-lesson-name/
├── 💻 code/           Runnable implementations (Python, TS, Rust, Julia)
├── 📖 docs/
│   └── en.md          Lesson documentation
└── 📦 outputs/        Prompts, skills, agents produced by this lesson

🔄 Every lesson follows 6 steps

Step	What happens
🎯 Motto	One-line core idea that sticks
❓ Problem	A concrete scenario where not knowing this hurts
🧠 Concept	Mermaid diagrams and intuition — no code yet
🔨 Build It	Implement from scratch in pure Python. No frameworks.
⚙️ Use It	Same thing with PyTorch, sklearn, or the real tool
🚢 Ship It	The prompt, skill, or agent this lesson produces

🔑 The Build It / Use It split is the key. You understand what the framework does because you built it yourself first.

---

🚀 Getting Started

🅰️ Option A — Just start reading

Pick any completed lesson from the website or expand any phase above.

🅱️ Option B — Clone and run

git clone https://github.com/rohitg00/ai-engineering-from-scratch.git
cd ai-engineering-from-scratch

python phases/01-math-foundations/01-linear-algebra-intuition/code/vectors.py

🅲 Option C — Find your level (recommended) ⭐

If you already know some ML/DL, don't start from Phase 1. Use the built-in assessment:

# In Claude Code:
/find-your-level

This 10-question quiz maps your knowledge to a starting phase and builds a personalized path with hour estimates.

✅ Prerequisites

• You can write code (Python or any language)
• You want to understand how AI actually works, not just call APIs

👤 Who This Is For

🧑‍💻 You are...	🚪 Start at...	⏱️ Time to complete
🌱 New to programming + AI	Phase 0 (Setup)	~306 hours
🐍 Know Python, new to ML	Phase 1 (Math)	~270 hours
📊 Know ML, new to DL	Phase 3 (Deep Learning)	~200 hours
🧠 Know DL, want LLMs/agents	Phase 10 (LLMs from Scratch)	~100 hours
🚀 Senior eng, want agents only	Phase 14 (Agent Engineering)	~60 hours

---

📰 Why This Matters Now

📈 The Industry Signal "The hottest new programming language is English." — Andrej Karpathy (tweet) "Software engineering is being remade in front of our eyes." — Boris Cherny, creator of Claude Code "Models will keep getting better. The skill that compounds is knowing what to build." — Industry consensus, 2026

📚 Foundational Papers Covered 📄 Attention Is All You Need (Vaswani et al., 2017) → Phase 7 📄 GPT-3: Language Models are Few-Shot Learners → Phase 10 📄 Denoising Diffusion Probabilistic Models → Phase 8 📄 InstructGPT / RLHF → Phase 10 📄 Direct Preference Optimization (DPO) → Phase 10 📄 Chain-of-Thought Prompting → Phase 11 📄 ReAct: Reasoning + Acting in LLMs → Phase 14 📄 MCP: Model Context Protocol (Anthropic) → Phase 13

---

🤝 Contributing

We welcome contributions of all kinds — new lessons, translations, fixes, and outputs.

📋 Want to...	👉 Read
Contribute a lesson or fix	CONTRIBUTING.md
Fork for your team or school	FORKING.md
See the lesson template	LESSON_TEMPLATE.md
Track progress	ROADMAP.md
Code of conduct	CODE_OF_CONDUCT.md

---

⭐ Star History

🌟 If this helped you, please star the repo! It keeps the project alive.

💚 Built with care by Rohit Ghumare and the community.

_{📜 MIT License — Use it however you want. Fork it. Teach it. Sell it. Ship it.}

_{✨ From linear algebra to autonomous agent swarms — one lesson at a time. ✨}