mirror of
https://github.com/rohitg00/ai-engineering-from-scratch.git
synced 2026-10-02 01:54:39 +08:00
Merge PR #56: phase-08 generative AI 14 lessons
This commit is contained in:
Rohit Ghumare
@@ -423,20 +423,20 @@ Other courses end with *"congratulations, you learned X."* Our lessons end with
|
||||
|
||||
| # | Lesson | Type | Lang |
|
||||
|:---:|--------|:----:|------|
|
||||
| 01 | Generative Models: Taxonomy & History |  | — |
|
||||
| 02 | Autoencoders & VAE |  | 🐍 |
|
||||
| 03 | GANs: Generator vs Discriminator |  | 🐍 |
|
||||
| 04 | Conditional GANs & Pix2Pix |  | 🐍 |
|
||||
| 05 | StyleGAN |  | 🐍 |
|
||||
| 06 | Diffusion Models — DDPM from Scratch |  | 🐍 |
|
||||
| 07 | Latent Diffusion & Stable Diffusion |  | 🐍 |
|
||||
| 08 | ControlNet, LoRA & Conditioning |  | 🐍 |
|
||||
| 09 | Inpainting, Outpainting & Editing |  | 🐍 |
|
||||
| 10 | Video Generation |  | 🐍 |
|
||||
| 11 | Audio Generation |  | 🐍 |
|
||||
| 12 | 3D Generation |  | 🐍 |
|
||||
| 13 | Flow Matching & Rectified Flows |  | 🐍 |
|
||||
| 14 | Evaluation: FID, CLIP Score |  | 🐍 |
|
||||
| 01 | [Generative Models: Taxonomy & History](phases/08-generative-ai/01-generative-models-taxonomy-history/) |  | 🐍 |
|
||||
| 02 | [Autoencoders & VAE](phases/08-generative-ai/02-autoencoders-vae/) |  | 🐍 |
|
||||
| 03 | [GANs: Generator vs Discriminator](phases/08-generative-ai/03-gans-generator-discriminator/) |  | 🐍 |
|
||||
| 04 | [Conditional GANs & Pix2Pix](phases/08-generative-ai/04-conditional-gans-pix2pix/) |  | 🐍 |
|
||||
| 05 | [StyleGAN](phases/08-generative-ai/05-stylegan/) |  | 🐍 |
|
||||
| 06 | [Diffusion Models — DDPM from Scratch](phases/08-generative-ai/06-diffusion-ddpm-from-scratch/) |  | 🐍 |
|
||||
| 07 | [Latent Diffusion & Stable Diffusion](phases/08-generative-ai/07-latent-diffusion-stable-diffusion/) |  | 🐍 |
|
||||
| 08 | [ControlNet, LoRA & Conditioning](phases/08-generative-ai/08-controlnet-lora-conditioning/) |  | 🐍 |
|
||||
| 09 | [Inpainting, Outpainting & Editing](phases/08-generative-ai/09-inpainting-outpainting-editing/) |  | 🐍 |
|
||||
| 10 | [Video Generation](phases/08-generative-ai/10-video-generation/) |  | 🐍 |
|
||||
| 11 | [Audio Generation](phases/08-generative-ai/11-audio-generation/) |  | 🐍 |
|
||||
| 12 | [3D Generation](phases/08-generative-ai/12-3d-generation/) |  | 🐍 |
|
||||
| 13 | [Flow Matching & Rectified Flows](phases/08-generative-ai/13-flow-matching-rectified-flows/) |  | 🐍 |
|
||||
| 14 | [Evaluation: FID, CLIP Score](phases/08-generative-ai/14-evaluation-fid-clip-score/) |  | 🐍 |
|
||||
|
||||
</details>
|
||||
|
||||
|
||||
+15
-15
@@ -199,24 +199,24 @@ Progress tracking for every phase and lesson.
|
||||
| 13 | [Scaling Laws](phases/07-transformers-deep-dive/13-scaling-laws) | ✅ | ~45 min |
|
||||
| 14 | [Build a Transformer from Scratch — The Capstone](phases/07-transformers-deep-dive/14-build-a-transformer-capstone) | ✅ | ~120 min |
|
||||
|
||||
## Phase 8: Generative AI — ⬚ (~14 hours)
|
||||
## Phase 8: Generative AI — ✅ (~14 hours)
|
||||
|
||||
| # | Lesson | Status | Est. |
|
||||
|---|--------|--------|------|
|
||||
| 01 | Generative Models — Taxonomy & History | ⬚ | ~45 min |
|
||||
| 02 | Autoencoders & VAE | ⬚ | ~75 min |
|
||||
| 03 | GANs — Generator vs Discriminator | ⬚ | ~75 min |
|
||||
| 04 | Conditional GANs & Pix2Pix | ⬚ | ~75 min |
|
||||
| 05 | StyleGAN | ⬚ | ~45 min |
|
||||
| 06 | Diffusion Models — DDPM from Scratch | ⬚ | ~75 min |
|
||||
| 07 | Latent Diffusion & Stable Diffusion | ⬚ | ~75 min |
|
||||
| 08 | ControlNet, LoRA & Image Conditioning | ⬚ | ~75 min |
|
||||
| 09 | Inpainting, Outpainting & Image Editing | ⬚ | ~75 min |
|
||||
| 10 | Video Generation | ⬚ | ~45 min |
|
||||
| 11 | Audio Generation | ⬚ | ~45 min |
|
||||
| 12 | 3D Generation | ⬚ | ~45 min |
|
||||
| 13 | Flow Matching & Rectified Flows | ⬚ | ~45 min |
|
||||
| 14 | Evaluation — FID, CLIP Score, Human Preference | ⬚ | ~45 min |
|
||||
| 01 | [Generative Models — Taxonomy & History](phases/08-generative-ai/01-generative-models-taxonomy-history/) | ✅ | ~45 min |
|
||||
| 02 | [Autoencoders & VAE](phases/08-generative-ai/02-autoencoders-vae/) | ✅ | ~75 min |
|
||||
| 03 | [GANs — Generator vs Discriminator](phases/08-generative-ai/03-gans-generator-discriminator/) | ✅ | ~75 min |
|
||||
| 04 | [Conditional GANs & Pix2Pix](phases/08-generative-ai/04-conditional-gans-pix2pix/) | ✅ | ~75 min |
|
||||
| 05 | [StyleGAN](phases/08-generative-ai/05-stylegan/) | ✅ | ~45 min |
|
||||
| 06 | [Diffusion Models — DDPM from Scratch](phases/08-generative-ai/06-diffusion-ddpm-from-scratch/) | ✅ | ~75 min |
|
||||
| 07 | [Latent Diffusion & Stable Diffusion](phases/08-generative-ai/07-latent-diffusion-stable-diffusion/) | ✅ | ~75 min |
|
||||
| 08 | [ControlNet, LoRA & Image Conditioning](phases/08-generative-ai/08-controlnet-lora-conditioning/) | ✅ | ~75 min |
|
||||
| 09 | [Inpainting, Outpainting & Image Editing](phases/08-generative-ai/09-inpainting-outpainting-editing/) | ✅ | ~75 min |
|
||||
| 10 | [Video Generation](phases/08-generative-ai/10-video-generation/) | ✅ | ~45 min |
|
||||
| 11 | [Audio Generation](phases/08-generative-ai/11-audio-generation/) | ✅ | ~45 min |
|
||||
| 12 | [3D Generation](phases/08-generative-ai/12-3d-generation/) | ✅ | ~45 min |
|
||||
| 13 | [Flow Matching & Rectified Flows](phases/08-generative-ai/13-flow-matching-rectified-flows/) | ✅ | ~45 min |
|
||||
| 14 | [Evaluation — FID, CLIP Score, Human Preference](phases/08-generative-ai/14-evaluation-fid-clip-score/) | ✅ | ~45 min |
|
||||
|
||||
## Phase 9: Reinforcement Learning — ✅ (~13 hours)
|
||||
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 520" font-family="Georgia, 'Times New Roman', serif">
|
||||
<defs>
|
||||
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto">
|
||||
<path d="M0,0 L10,5 L0,10 z" fill="#1a1a1a"/>
|
||||
</marker>
|
||||
<style>
|
||||
.box { fill: #faf6ef; stroke: #1a1a1a; stroke-width: 1.5; }
|
||||
.hot { fill: #fff1d6; stroke: #c0392b; stroke-width: 1.5; }
|
||||
.label { font-size: 14px; font-weight: 600; fill: #1a1a1a; }
|
||||
.content { font-size: 12px; fill: #333; }
|
||||
.mono { font-size: 11px; fill: #333; font-family: 'Menlo', monospace; }
|
||||
.caption { font-size: 11px; fill: #555; font-style: italic; }
|
||||
.title { font-size: 16px; font-weight: 700; fill: #1a1a1a; }
|
||||
</style>
|
||||
</defs>
|
||||
|
||||
<text x="450" y="30" text-anchor="middle" class="title">five families of generative models</text>
|
||||
<text x="450" y="50" text-anchor="middle" class="caption">split by what they model and how they sample</text>
|
||||
|
||||
<rect x="380" y="70" width="140" height="44" class="box"/>
|
||||
<text x="450" y="90" text-anchor="middle" class="label">p_data(x)</text>
|
||||
<text x="450" y="106" text-anchor="middle" class="caption">unknown, want sampler</text>
|
||||
|
||||
<line x1="450" y1="114" x2="180" y2="150" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
<line x1="450" y1="114" x2="340" y2="150" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
<line x1="450" y1="114" x2="500" y2="150" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
<line x1="450" y1="114" x2="660" y2="150" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
<line x1="450" y1="114" x2="820" y2="150" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
|
||||
<rect x="60" y="155" width="230" height="180" class="box"/>
|
||||
<text x="175" y="178" text-anchor="middle" class="label">1. explicit, tractable</text>
|
||||
<text x="175" y="200" text-anchor="middle" class="mono">p(x) = ∏ p(x_i | x_<i)</text>
|
||||
<text x="175" y="225" text-anchor="middle" class="content">autoregressive: GPT, PixelCNN</text>
|
||||
<text x="175" y="245" text-anchor="middle" class="content">flows: Glow, RealNVP</text>
|
||||
<text x="175" y="275" text-anchor="middle" class="caption">exact log p(x)</text>
|
||||
<text x="175" y="293" text-anchor="middle" class="caption">slow sequential inference</text>
|
||||
<text x="175" y="311" text-anchor="middle" class="caption">architecture restrictions (flows)</text>
|
||||
|
||||
<rect x="305" y="155" width="230" height="180" class="hot"/>
|
||||
<text x="420" y="178" text-anchor="middle" class="label">2. explicit, approximate</text>
|
||||
<text x="420" y="200" text-anchor="middle" class="mono">maximize ELBO ≤ log p(x)</text>
|
||||
<text x="420" y="225" text-anchor="middle" class="content">VAE: encoder + decoder</text>
|
||||
<text x="420" y="245" text-anchor="middle" class="content">diffusion: DDPM, SD3</text>
|
||||
<text x="420" y="275" text-anchor="middle" class="caption">dominant in 2026</text>
|
||||
<text x="420" y="293" text-anchor="middle" class="caption">iterative sampling (20-50 steps)</text>
|
||||
<text x="420" y="311" text-anchor="middle" class="caption">scales to text, image, video, 3D</text>
|
||||
|
||||
<rect x="550" y="155" width="230" height="180" class="box"/>
|
||||
<text x="665" y="178" text-anchor="middle" class="label">3. implicit density</text>
|
||||
<text x="665" y="200" text-anchor="middle" class="mono">G(z) -> x, D(x) -> real/fake</text>
|
||||
<text x="665" y="225" text-anchor="middle" class="content">GAN, StyleGAN, Pix2Pix</text>
|
||||
<text x="665" y="255" text-anchor="middle" class="caption">one-shot sampling</text>
|
||||
<text x="665" y="273" text-anchor="middle" class="caption">no log p(x) at all</text>
|
||||
<text x="665" y="291" text-anchor="middle" class="caption">training instability, mode collapse</text>
|
||||
<text x="665" y="309" text-anchor="middle" class="caption">still SOTA for narrow photoreal</text>
|
||||
|
||||
<rect x="60" y="350" width="350" height="150" class="box"/>
|
||||
<text x="235" y="373" text-anchor="middle" class="label">4. score / continuous-time</text>
|
||||
<text x="235" y="395" text-anchor="middle" class="mono">learn s(x) = ∇_x log p(x)</text>
|
||||
<text x="235" y="418" text-anchor="middle" class="content">score SDE, flow matching</text>
|
||||
<text x="235" y="438" text-anchor="middle" class="content">rectified flow, consistency models</text>
|
||||
<text x="235" y="465" text-anchor="middle" class="caption">simulation-free training</text>
|
||||
<text x="235" y="483" text-anchor="middle" class="caption">straight paths => 1-4 step sampling</text>
|
||||
|
||||
<rect x="430" y="350" width="350" height="150" class="box"/>
|
||||
<text x="605" y="373" text-anchor="middle" class="label">5. tokens + AR transformer</text>
|
||||
<text x="605" y="395" text-anchor="middle" class="mono">VQ-VAE + transformer over tokens</text>
|
||||
<text x="605" y="418" text-anchor="middle" class="content">DALL-E 1, Parti, MuseNet</text>
|
||||
<text x="605" y="438" text-anchor="middle" class="content">AudioLM, VALL-E, MusicGen, Sora patches</text>
|
||||
<text x="605" y="465" text-anchor="middle" class="caption">reuse LLM stack</text>
|
||||
<text x="605" y="483" text-anchor="middle" class="caption">quality bounded by tokenizer</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 5.1 KiB |
@@ -0,0 +1,103 @@
|
||||
import math
|
||||
import random
|
||||
|
||||
|
||||
def sample_mixture(n, rng):
|
||||
"""Two-mode Gaussian mixture. Mode A at -2 (sigma 0.6), mode B at +2 (sigma 0.9)."""
|
||||
samples = []
|
||||
for _ in range(n):
|
||||
if rng.random() < 0.4:
|
||||
samples.append(rng.gauss(-2.0, 0.6))
|
||||
else:
|
||||
samples.append(rng.gauss(2.0, 0.9))
|
||||
return samples
|
||||
|
||||
|
||||
def histogram_density(samples, x, bin_width=0.25):
|
||||
"""Explicit density via histogram. Returns p(x) as (count in bin) / (n * bin_width)."""
|
||||
n = len(samples)
|
||||
lo, hi = x - bin_width / 2, x + bin_width / 2
|
||||
count = sum(1 for s in samples if lo <= s < hi)
|
||||
return count / (n * bin_width)
|
||||
|
||||
|
||||
def kde_density(samples, x, bandwidth=0.3):
|
||||
"""Approximate density via Gaussian kernel density estimate."""
|
||||
n = len(samples)
|
||||
total = 0.0
|
||||
for s in samples:
|
||||
u = (x - s) / bandwidth
|
||||
total += math.exp(-0.5 * u * u) / math.sqrt(2 * math.pi)
|
||||
return total / (n * bandwidth)
|
||||
|
||||
|
||||
def implicit_generator(samples, k, rng):
|
||||
"""Implicit generator: sample a training point and add tiny noise. No p(x)."""
|
||||
out = []
|
||||
for _ in range(k):
|
||||
base = rng.choice(samples)
|
||||
out.append(base + rng.gauss(0.0, 0.1))
|
||||
return out
|
||||
|
||||
|
||||
def integrate_density(density_fn, samples, lo, hi, steps=200):
|
||||
"""Trapezoid-rule integration of a density over [lo, hi]."""
|
||||
xs = [lo + (hi - lo) * i / steps for i in range(steps + 1)]
|
||||
total = 0.0
|
||||
for i in range(steps):
|
||||
a, b = xs[i], xs[i + 1]
|
||||
total += 0.5 * (density_fn(samples, a) + density_fn(samples, b)) * (b - a)
|
||||
return total
|
||||
|
||||
|
||||
def ascii_histogram(samples, lo=-5.0, hi=5.0, bins=40, height=12):
|
||||
"""Tiny text histogram so you can see the two modes without a plotting lib."""
|
||||
width = (hi - lo) / bins
|
||||
counts = [0] * bins
|
||||
for s in samples:
|
||||
if lo <= s < hi:
|
||||
counts[int((s - lo) / width)] += 1
|
||||
peak = max(counts) or 1
|
||||
rows = []
|
||||
for row in range(height, 0, -1):
|
||||
threshold = peak * row / height
|
||||
line = "".join("#" if c >= threshold else " " for c in counts)
|
||||
rows.append(line)
|
||||
rows.append("-" * bins)
|
||||
rows.append(f"{lo:<.1f}" + " " * (bins - 8) + f"{hi:>.1f}")
|
||||
return "\n".join(rows)
|
||||
|
||||
|
||||
def main():
|
||||
rng = random.Random(42)
|
||||
samples = sample_mixture(2000, rng)
|
||||
|
||||
print("=== 2000 samples from a two-mode Gaussian mixture ===")
|
||||
print(ascii_histogram(samples))
|
||||
print()
|
||||
|
||||
query = 0.0
|
||||
print(f"evaluate p(x={query}) three ways:")
|
||||
print(f" histogram density: {histogram_density(samples, query):.4f}")
|
||||
print(f" kernel density: {kde_density(samples, query):.4f}")
|
||||
print(f" implicit generator: N/A (only samples, no density)")
|
||||
print()
|
||||
|
||||
p_hist = integrate_density(histogram_density, samples, -0.5, 0.5)
|
||||
p_kde = integrate_density(kde_density, samples, -0.5, 0.5)
|
||||
print(f"integrate p(x in [-0.5, 0.5]):")
|
||||
print(f" histogram: {p_hist:.3f}")
|
||||
print(f" kde: {p_kde:.3f}")
|
||||
print()
|
||||
|
||||
new_samples = implicit_generator(samples, 10, rng)
|
||||
print("10 new samples from the implicit (GAN-ish) generator:")
|
||||
print(" " + ", ".join(f"{s:+.2f}" for s in new_samples))
|
||||
print()
|
||||
|
||||
print("takeaway: explicit density (buckets 1-2 in the doc) lets you answer")
|
||||
print("'how likely is this point?'. implicit (bucket 3) does not.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,134 @@
|
||||
# Generative Models — Taxonomy & History
|
||||
|
||||
> Every image model, text model, video model, and 3D model fits in one of five buckets. Pick the wrong bucket and you will fight the math for weeks. Pick the right one and the field's last twelve years of progress stacks cleanly in your head.
|
||||
|
||||
**Type:** Learn
|
||||
**Languages:** Python
|
||||
**Prerequisites:** Phase 2 (ML Fundamentals), Phase 3 (Deep Learning Core), Phase 7 · 14 (Transformers)
|
||||
**Time:** ~45 minutes
|
||||
|
||||
## The Problem
|
||||
|
||||
A generative model does one job: given training samples drawn from some unknown distribution `p_data(x)`, output new samples that look like they came from the same distribution. Faces, sentences, MIDI files, protein structures — all the same problem if you squint.
|
||||
|
||||
The rub is that `p_data` lives in a space with millions of dimensions (a 512x512 RGB image is ~786k dimensions), the samples sit on a thin manifold inside that space, and you only have maybe 10M examples. Brute-forcing the density is hopeless. Every generative model is a compromise that trades one hard problem for a slightly less hard one.
|
||||
|
||||
Five families have survived the last twelve years. Knowing which compromise each family makes tells you why it wins on some tasks and collapses on others.
|
||||
|
||||
## The Concept
|
||||
|
||||
|
||||
|
||||
**1. Explicit density, tractable.** Write `log p(x)` as a sum you can actually evaluate. Autoregressive models (PixelCNN, WaveNet, GPT) factorize `p(x) = ∏ p(x_i | x_<i)`. Normalizing flows (RealNVP, Glow) build `p(x)` as an invertible transform of a simple base. Pro: exact likelihood, clean training loss. Con: autoregressive inference is sequential (slow for long sequences), flows need invertible architectures (architecturally restrictive).
|
||||
|
||||
**2. Explicit density, approximate.** Bound `log p(x)` from below (ELBO) and optimize the bound. VAEs (Kingma 2013) use an encoder-decoder with a variational posterior. Diffusion models (DDPM, Ho 2020) train a denoiser that implicitly optimizes a weighted ELBO. Diffusion is the dominant image, video, and 3D backbone in 2026.
|
||||
|
||||
**3. Implicit density.** Skip density entirely; learn a generator `G(z)` that produces samples and a discriminator `D(x)` that tells real from fake. GANs (Goodfellow 2014). Fast at inference (one forward pass) but notoriously unstable during training. StyleGAN 1/2/3 remain state of the art for fixed-domain photorealism (faces, bedrooms) even in 2026.
|
||||
|
||||
**4. Score-based / continuous-time.** Learn the gradient of the log-density `∇_x log p(x)` (the score) directly. Song & Ermon (2019) showed score matching generalizes diffusion to an SDE. Flow matching (Lipman 2023) is the 2024-2026 hotness: simulate-free training, straighter paths, 4-10x faster sampling than DDPM. Stable Diffusion 3, Flux, AudioCraft 2 all use flow matching.
|
||||
|
||||
**5. Token-based autoregressive over discrete codes.** Compress high-dim data with a VQ-VAE or residual quantizer into a short sequence of discrete tokens, then use a Transformer to model the token sequence. Parti, MuseNet, AudioLM, VALL-E, Sora's patch tokenizer all use this. This is bucket 1 plus a learned tokenizer.
|
||||
|
||||
## A brief history
|
||||
|
||||
| Year | Model | Why it mattered |
|
||||
|------|-------|-----------------|
|
||||
| 2013 | VAE (Kingma) | First deep generative model with a usable training loss. |
|
||||
| 2014 | GAN (Goodfellow) | Implicit density, no likelihood — shockingly sharp samples. |
|
||||
| 2015 | DRAW, PixelCNN | Sequential image generation. |
|
||||
| 2017 | Glow, RealNVP | Invertible flows; exact likelihood with depth. |
|
||||
| 2017 | Progressive GAN | First megapixel faces. |
|
||||
| 2019 | StyleGAN / StyleGAN2 | Photorealistic faces still hard to beat for that one domain. |
|
||||
| 2020 | DDPM (Ho) | Diffusion becomes practical. |
|
||||
| 2021 | CLIP, DALL-E 1, VQGAN | Text-to-image goes mainstream. |
|
||||
| 2022 | Imagen, Stable Diffusion 1, DALL-E 2 | Latent diffusion + text conditioning = commodity. |
|
||||
| 2022 | ControlNet, LoRA | Fine control over pretrained diffusion. |
|
||||
| 2023 | SDXL, Midjourney v5, Flow matching | Scale + better training dynamics. |
|
||||
| 2024 | Sora, Stable Diffusion 3, Flux.1 | Video diffusion; flow matching wins. |
|
||||
| 2025 | Veo 2, Kling 1.5, Runway Gen-3, Nano Banana | Production-grade video. |
|
||||
| 2026 | Consistency + Rectified Flow | One-step sampling from diffusion backbones. |
|
||||
|
||||
## The five-question triage
|
||||
|
||||
When a new generative model paper drops, answer these five questions before reading the method section.
|
||||
|
||||
1. **What is being modeled?** Pixels, latents, discrete tokens, 3D Gaussians, meshes, waveforms?
|
||||
2. **Is the density explicit or implicit?** Do they write down `log p(x)`?
|
||||
3. **Sampling: one-shot or iterative?** Iterative means slower inference; one-shot usually means adversarial or distilled.
|
||||
4. **Conditioning: unconditional, class, text, image, pose?** This determines the loss and architecture scaffolding.
|
||||
5. **Evaluation: FID, CLIP score, IS, human preference, task accuracy?** Each has known failure modes (see Lesson 14).
|
||||
|
||||
You will re-answer these five for every lesson in this phase. By the end, they will be reflex.
|
||||
|
||||
## Build It
|
||||
|
||||
The code for this lesson is a lightweight visualization: fit a 1-D mixture-of-Gaussians from samples using three toy approaches (kernel density, discrete histogram, and a nearest-sample "GAN-ish" generator) so you can see the difference between explicit vs implicit density on a problem you can print on one screen.
|
||||
|
||||
Run `code/main.py`. It draws 2000 samples from a two-mode Gaussian mixture, then prints:
|
||||
|
||||
```
|
||||
explicit density (histogram): p(x in [-0.5, 0.5]) ≈ 0.38
|
||||
approximate density (KDE): p(x in [-0.5, 0.5]) ≈ 0.41
|
||||
implicit (nearest-sample gen): 20 new samples printed, no p(x)
|
||||
```
|
||||
|
||||
Notice: the first two let you ask "how likely is this point?" The third cannot. This is the *explicit vs implicit* distinction that will matter for every future lesson.
|
||||
|
||||
## Use It
|
||||
|
||||
Which family, for which task, in 2026?
|
||||
|
||||
| Task | Best family | Why |
|
||||
|------|-------------|-----|
|
||||
| Photoreal faces, narrow domain | StyleGAN 2/3 | Still sharpest, fastest inference. |
|
||||
| General text-to-image | Latent diffusion + flow matching | SD3, Flux.1, DALL-E 3. |
|
||||
| Fast text-to-image | Rectified flow + distillation | SDXL-Turbo, SD3-Turbo, LCM. |
|
||||
| Text-to-video | Diffusion Transformer + flow matching | Sora, Veo 2, Kling. |
|
||||
| Speech + music | Token-based AR (AudioLM, VALL-E, MusicGen) or flow matching (AudioCraft 2) | Discrete tokens scale cheaply. |
|
||||
| 3D scenes | Gaussian Splatting fit, diffusion prior | 3D-GS for reconstruction, diffusion for novel-view. |
|
||||
| Density estimation (no sampling) | Flows | Only family with exact `log p(x)`. |
|
||||
| Simulation / physics | Flow matching, score SDE | Straight-line paths, smooth vector fields. |
|
||||
|
||||
## Ship It
|
||||
|
||||
Save as `outputs/skill-model-chooser.md`.
|
||||
|
||||
The skill takes a task description and outputs: (1) which family to use, (2) a ranked list of three open and three hosted options, (3) the likely failure mode you should watch for, and (4) a compute/time budget.
|
||||
|
||||
## Exercises
|
||||
|
||||
1. **Easy.** For each of these five products, identify the family and backbone: ChatGPT image, Midjourney v7, Sora, Runway Gen-3, ElevenLabs. Evidence should be from public technical reports.
|
||||
2. **Medium.** The paper you are about to read tomorrow claims 100x faster sampling than diffusion. Write down three questions to check whether the speedup survives conditioning and high resolution.
|
||||
3. **Hard.** Take one domain you care about (e.g. protein structure, CAD, molecules, trajectories). Answer the five-question triage for the current SOTA model in that domain and sketch what a better model would change.
|
||||
|
||||
## Key Terms
|
||||
|
||||
| Term | What people say | What it actually means |
|
||||
|------|-----------------|-----------------------|
|
||||
| Generative model | "It makes new stuff" | Learns a sampler for `p_data(x)`, optionally exposes `log p(x)`. |
|
||||
| Explicit density | "You can evaluate it" | Model provides a closed-form or tractable `log p(x)`. |
|
||||
| Implicit density | "GAN-style" | Only a sampler — no way to evaluate `p(x)` of a given point. |
|
||||
| ELBO | "Evidence lower bound" | A tractable lower bound on `log p(x)`; VAEs and diffusion optimize it. |
|
||||
| Score | "Gradient of log-density" | `∇_x log p(x)`; diffusion and SDE models learn this field. |
|
||||
| Manifold hypothesis | "Data lives on a surface" | High-dim data concentrates on a low-dim manifold; why dimensionality reduction works. |
|
||||
| Autoregressive | "Predict the next piece" | Factorize joint as product of conditionals. |
|
||||
| Latent | "Compressed code" | Low-dim representation from which a decoder can reconstruct the input. |
|
||||
|
||||
## Production note: five families, five inference shapes
|
||||
|
||||
Each family maps to a different inference-server cost curve. production-inference literature frames LLM inference as prefill + decode; the same decomposition applies here:
|
||||
|
||||
- **Autoregressive (bucket 1 and 5).** Sequential decode dominates latency; KV-cache, continuous batching, and speculative decoding all apply directly.
|
||||
- **VAE / diffusion / flow-matching (buckets 2 and 4).** There is no decode in the LLM sense. Cost = `num_steps × step_cost`, and the `step_cost` is a transformer or U-Net forward at the full latent resolution. The production knobs are step count (DDIM / DPM-Solver / distillation), batch size, and precision (bf16 / fp8 / int4).
|
||||
- **GAN (bucket 3).** One forward pass. No schedule, no KV-cache. TTFT ≈ total latency. This is why StyleGAN still wins on narrow-domain UX.
|
||||
|
||||
When you see "faster than diffusion" in a paper abstract, translate it to "fewer steps × same step cost" or "same steps × cheaper step cost". Everything else is marketing.
|
||||
|
||||
## Further Reading
|
||||
|
||||
- [Goodfellow et al. (2014). Generative Adversarial Nets](https://arxiv.org/abs/1406.2661) — the GAN paper.
|
||||
- [Kingma & Welling (2013). Auto-Encoding Variational Bayes](https://arxiv.org/abs/1312.6114) — the VAE paper.
|
||||
- [Ho, Jain, Abbeel (2020). Denoising Diffusion Probabilistic Models](https://arxiv.org/abs/2006.11239) — the DDPM paper.
|
||||
- [Song et al. (2021). Score-Based Generative Modeling through SDEs](https://arxiv.org/abs/2011.13456) — diffusion as an SDE.
|
||||
- [Lipman et al. (2023). Flow Matching for Generative Modeling](https://arxiv.org/abs/2210.02747) — the flow matching paper.
|
||||
- [Esser et al. (2024). Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://arxiv.org/abs/2403.03206) — Stable Diffusion 3.
|
||||
+18
@@ -0,0 +1,18 @@
|
||||
---
|
||||
name: generative-model-chooser
|
||||
description: Pick a generative-model family, backbone, and hosted alternative for a given task and budget.
|
||||
version: 1.0.0
|
||||
phase: 8
|
||||
lesson: 01
|
||||
tags: [generative, taxonomy]
|
||||
---
|
||||
|
||||
Given a task description (modality, domain, latency budget, compute budget, conditioning signal), output:
|
||||
|
||||
1. Family. Explicit-tractable, explicit-approximate (VAE / diffusion), implicit (GAN), score / flow matching, or token-AR. One-sentence reason tied to the modality + latency.
|
||||
2. Backbone + open reference. One pretrained open-weights model the user can fine-tune today (e.g. Stable Diffusion 3, Flux.1-dev, AudioCraft 2, StyleGAN3, 3D Gaussian Splatting).
|
||||
3. Hosted alternatives. Three production APIs ranked by quality / cost / latency trade-off (fal.ai, Replicate, Stability, Runway, Veo, Kling, ElevenLabs, etc.).
|
||||
4. Failure mode. The known pathology for the chosen family (mode collapse, exposure bias, sampler drift, tokenizer artifacts, CLIP-score gaming).
|
||||
5. Budget. Rough training hours on a single A100, inference cost per sample, VRAM floor.
|
||||
|
||||
Refuse to recommend a GAN when the task requires likelihood scoring. Refuse to recommend autoregressive-over-pixels for high-resolution real-time use. Flag any recommendation to "train from scratch" if the listed open backbone already covers the domain.
|
||||
@@ -0,0 +1,84 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 480" font-family="Georgia, 'Times New Roman', serif">
|
||||
<defs>
|
||||
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto">
|
||||
<path d="M0,0 L10,5 L0,10 z" fill="#1a1a1a"/>
|
||||
</marker>
|
||||
<style>
|
||||
.box { fill: #faf6ef; stroke: #1a1a1a; stroke-width: 1.5; }
|
||||
.hot { fill: #fff1d6; stroke: #c0392b; stroke-width: 1.5; }
|
||||
.cold { fill: #eaf4ff; stroke: #2c5f8c; stroke-width: 1.5; }
|
||||
.label { font-size: 14px; font-weight: 600; fill: #1a1a1a; }
|
||||
.content { font-size: 12px; fill: #333; }
|
||||
.mono { font-size: 12px; fill: #333; font-family: 'Menlo', monospace; }
|
||||
.caption { font-size: 11px; fill: #555; font-style: italic; }
|
||||
.title { font-size: 16px; font-weight: 700; fill: #1a1a1a; }
|
||||
</style>
|
||||
</defs>
|
||||
|
||||
<text x="450" y="28" text-anchor="middle" class="title">autoencoder vs VAE — the one trick</text>
|
||||
|
||||
<!-- plain AE -->
|
||||
<text x="180" y="70" text-anchor="middle" class="label">plain autoencoder</text>
|
||||
<rect x="40" y="85" width="60" height="50" class="box"/>
|
||||
<text x="70" y="115" text-anchor="middle" class="content">x</text>
|
||||
<line x1="100" y1="110" x2="140" y2="110" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
|
||||
<rect x="140" y="85" width="80" height="50" class="box"/>
|
||||
<text x="180" y="115" text-anchor="middle" class="content">encoder</text>
|
||||
<line x1="220" y1="110" x2="260" y2="110" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
|
||||
<rect x="260" y="85" width="40" height="50" class="box"/>
|
||||
<text x="280" y="115" text-anchor="middle" class="content">z</text>
|
||||
<line x1="300" y1="110" x2="320" y2="110" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
|
||||
<text x="180" y="170" text-anchor="middle" class="caption">loss = ||x - x̂||²</text>
|
||||
<text x="180" y="188" text-anchor="middle" class="caption">z-space: lumpy, not a distribution</text>
|
||||
<text x="180" y="206" text-anchor="middle" class="caption">sample random z -> garbage out</text>
|
||||
|
||||
<!-- VAE -->
|
||||
<text x="630" y="70" text-anchor="middle" class="label">variational autoencoder</text>
|
||||
|
||||
<rect x="360" y="85" width="60" height="50" class="box"/>
|
||||
<text x="390" y="115" text-anchor="middle" class="content">x</text>
|
||||
<line x1="420" y1="110" x2="460" y2="110" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
|
||||
<rect x="460" y="85" width="90" height="50" class="box"/>
|
||||
<text x="505" y="115" text-anchor="middle" class="content">encoder</text>
|
||||
|
||||
<line x1="550" y1="100" x2="585" y2="85" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
<line x1="550" y1="125" x2="585" y2="140" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
|
||||
<rect x="585" y="65" width="60" height="36" class="hot"/>
|
||||
<text x="615" y="88" text-anchor="middle" class="mono">μ(x)</text>
|
||||
<rect x="585" y="128" width="80" height="36" class="hot"/>
|
||||
<text x="625" y="151" text-anchor="middle" class="mono">log σ²(x)</text>
|
||||
|
||||
<rect x="700" y="85" width="120" height="60" class="cold"/>
|
||||
<text x="760" y="108" text-anchor="middle" class="mono">z = μ + σ·ε</text>
|
||||
<text x="760" y="128" text-anchor="middle" class="caption">reparameterize</text>
|
||||
<text x="760" y="142" text-anchor="middle" class="caption">ε ~ N(0, I)</text>
|
||||
|
||||
<line x1="650" y1="85" x2="700" y2="100" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
<line x1="665" y1="146" x2="700" y2="130" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
|
||||
<!-- decoder + loss -->
|
||||
<line x1="760" y1="145" x2="760" y2="190" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
<rect x="700" y="195" width="120" height="45" class="box"/>
|
||||
<text x="760" y="220" text-anchor="middle" class="content">decoder</text>
|
||||
<line x1="760" y1="240" x2="760" y2="280" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
<rect x="720" y="285" width="80" height="40" class="box"/>
|
||||
<text x="760" y="309" text-anchor="middle" class="content">x̂</text>
|
||||
|
||||
<!-- loss box -->
|
||||
<rect x="40" y="260" width="540" height="140" class="box"/>
|
||||
<text x="310" y="284" text-anchor="middle" class="label">loss = reconstruction + β · KL</text>
|
||||
<text x="310" y="312" text-anchor="middle" class="mono">||x - x̂||² + ½ Σ( σ² + μ² - logσ² - 1 )</text>
|
||||
<text x="310" y="340" text-anchor="middle" class="caption">recon term: push x̂ → x</text>
|
||||
<text x="310" y="360" text-anchor="middle" class="caption">KL term: push q(z|x) → N(0, I)</text>
|
||||
<text x="310" y="380" text-anchor="middle" class="caption">β knob trades sharpness for well-shaped latent</text>
|
||||
|
||||
<!-- sampling -->
|
||||
<rect x="40" y="420" width="820" height="50" class="hot"/>
|
||||
<text x="450" y="438" text-anchor="middle" class="label">inference: sample z ~ N(0, I), forward through decoder → new x̂</text>
|
||||
<text x="450" y="458" text-anchor="middle" class="caption">one forward pass; no iteration; decoder is the whole generator</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 5.0 KiB |
@@ -0,0 +1,201 @@
|
||||
import math
|
||||
import random
|
||||
|
||||
|
||||
def matmul(W, x):
|
||||
return [sum(w * xi for w, xi in zip(row, x)) for row in W]
|
||||
|
||||
|
||||
def add(a, b):
|
||||
return [x + y for x, y in zip(a, b)]
|
||||
|
||||
|
||||
def tanh(v):
|
||||
return [math.tanh(x) for x in v]
|
||||
|
||||
|
||||
def tanh_grad(h):
|
||||
return [1 - x * x for x in h]
|
||||
|
||||
|
||||
def randn_matrix(rows, cols, rng, scale=0.2):
|
||||
return [[rng.gauss(0, scale) for _ in range(cols)] for _ in range(rows)]
|
||||
|
||||
|
||||
def init_vae(in_dim, hidden, z_dim, rng):
|
||||
return {
|
||||
"enc": {
|
||||
"W1": randn_matrix(hidden, in_dim, rng),
|
||||
"b1": [0.0] * hidden,
|
||||
"W_mu": randn_matrix(z_dim, hidden, rng),
|
||||
"b_mu": [0.0] * z_dim,
|
||||
"W_sig": randn_matrix(z_dim, hidden, rng),
|
||||
"b_sig": [0.0] * z_dim,
|
||||
},
|
||||
"dec": {
|
||||
"W1": randn_matrix(hidden, z_dim, rng),
|
||||
"b1": [0.0] * hidden,
|
||||
"W_out": randn_matrix(in_dim, hidden, rng),
|
||||
"b_out": [0.0] * in_dim,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def clamp(v, lo, hi):
|
||||
return [max(lo, min(hi, x)) for x in v]
|
||||
|
||||
|
||||
def forward(x, params, eps):
|
||||
"""Forward pass with a fixed epsilon for the reparameterization."""
|
||||
enc, dec = params["enc"], params["dec"]
|
||||
h_enc = tanh(add(matmul(enc["W1"], x), enc["b1"]))
|
||||
mu = add(matmul(enc["W_mu"], h_enc), enc["b_mu"])
|
||||
log_sigma2 = clamp(add(matmul(enc["W_sig"], h_enc), enc["b_sig"]), -6, 6)
|
||||
sigma = [math.exp(0.5 * lv) for lv in log_sigma2]
|
||||
z = [m + s * e for m, s, e in zip(mu, sigma, eps)]
|
||||
h_dec = tanh(add(matmul(dec["W1"], z), dec["b1"]))
|
||||
x_hat = add(matmul(dec["W_out"], h_dec), dec["b_out"])
|
||||
return {
|
||||
"h_enc": h_enc, "mu": mu, "log_sigma2": log_sigma2,
|
||||
"sigma": sigma, "z": z, "h_dec": h_dec, "x_hat": x_hat,
|
||||
}
|
||||
|
||||
|
||||
def loss_value(x, fwd, beta):
|
||||
recon = sum((a - b) ** 2 for a, b in zip(x, fwd["x_hat"]))
|
||||
kl = 0.5 * sum(math.exp(lv) + m * m - lv - 1
|
||||
for m, lv in zip(fwd["mu"], fwd["log_sigma2"]))
|
||||
return recon + beta * kl, recon, kl
|
||||
|
||||
|
||||
def backward(x, fwd, params, beta):
|
||||
"""Hand-written backprop. Returns gradient dict matching params shape."""
|
||||
enc, dec = params["enc"], params["dec"]
|
||||
mu, log_sigma2, sigma = fwd["mu"], fwd["log_sigma2"], fwd["sigma"]
|
||||
z, h_dec, h_enc = fwd["z"], fwd["h_dec"], fwd["h_enc"]
|
||||
x_hat = fwd["x_hat"]
|
||||
|
||||
grads = {"enc": {}, "dec": {}}
|
||||
|
||||
# d recon / d x_hat = 2(x_hat - x)
|
||||
d_x_hat = [2 * (a - b) for a, b in zip(x_hat, x)]
|
||||
|
||||
# decoder: x_hat = W_out @ h_dec + b_out
|
||||
grads["dec"]["b_out"] = d_x_hat[:]
|
||||
grads["dec"]["W_out"] = [[d * h for h in h_dec] for d in d_x_hat]
|
||||
|
||||
# d loss / d h_dec = W_out^T @ d_x_hat
|
||||
d_h_dec = [sum(dec["W_out"][i][j] * d_x_hat[i] for i in range(len(d_x_hat)))
|
||||
for j in range(len(h_dec))]
|
||||
# through tanh
|
||||
d_pre_dec = [dg * g for dg, g in zip(d_h_dec, tanh_grad(h_dec))]
|
||||
grads["dec"]["b1"] = d_pre_dec[:]
|
||||
grads["dec"]["W1"] = [[d * zi for zi in z] for d in d_pre_dec]
|
||||
|
||||
# d loss / d z = W1_dec^T @ d_pre_dec
|
||||
d_z = [sum(dec["W1"][i][j] * d_pre_dec[i] for i in range(len(d_pre_dec)))
|
||||
for j in range(len(z))]
|
||||
|
||||
# reparameterization: z = mu + sigma * eps, sigma = exp(0.5 * log_sigma2)
|
||||
# d z / d mu = 1, d z / d log_sigma2 = 0.5 * sigma * eps
|
||||
d_mu_recon = d_z[:]
|
||||
eps_used = [(z[i] - mu[i]) / max(sigma[i], 1e-8) for i in range(len(z))]
|
||||
d_lv_recon = [0.5 * d_z[i] * sigma[i] * eps_used[i] for i in range(len(z))]
|
||||
|
||||
# KL term: 0.5 * sum(exp(lv) + mu^2 - lv - 1)
|
||||
# d KL / d mu = mu ; d KL / d lv = 0.5 * (exp(lv) - 1)
|
||||
d_mu = [d_mu_recon[i] + beta * mu[i] for i in range(len(mu))]
|
||||
d_lv = [d_lv_recon[i] + beta * 0.5 * (math.exp(log_sigma2[i]) - 1)
|
||||
for i in range(len(mu))]
|
||||
|
||||
grads["enc"]["b_mu"] = d_mu[:]
|
||||
grads["enc"]["W_mu"] = [[d * h for h in h_enc] for d in d_mu]
|
||||
grads["enc"]["b_sig"] = d_lv[:]
|
||||
grads["enc"]["W_sig"] = [[d * h for h in h_enc] for d in d_lv]
|
||||
|
||||
# d loss / d h_enc from both mu and log_sigma2 paths
|
||||
d_h_enc = [0.0] * len(h_enc)
|
||||
for j in range(len(h_enc)):
|
||||
for i in range(len(d_mu)):
|
||||
d_h_enc[j] += enc["W_mu"][i][j] * d_mu[i]
|
||||
d_h_enc[j] += enc["W_sig"][i][j] * d_lv[i]
|
||||
d_pre_enc = [dg * g for dg, g in zip(d_h_enc, tanh_grad(h_enc))]
|
||||
grads["enc"]["b1"] = d_pre_enc[:]
|
||||
grads["enc"]["W1"] = [[d * xi for xi in x] for d in d_pre_enc]
|
||||
return grads
|
||||
|
||||
|
||||
def apply_update(params, grads, lr):
|
||||
for part in ("enc", "dec"):
|
||||
for name, tensor in params[part].items():
|
||||
g = grads[part][name]
|
||||
if isinstance(tensor[0], list):
|
||||
for i, row in enumerate(tensor):
|
||||
for j in range(len(row)):
|
||||
row[j] -= lr * g[i][j]
|
||||
else:
|
||||
for i in range(len(tensor)):
|
||||
tensor[i] -= lr * g[i]
|
||||
|
||||
|
||||
def sample_mixture(n, d, rng):
|
||||
data = []
|
||||
for _ in range(n):
|
||||
if rng.random() < 0.5:
|
||||
center = [1.0] * (d // 2) + [-1.0] * (d - d // 2)
|
||||
else:
|
||||
center = [-1.0] * (d // 2) + [1.0] * (d - d // 2)
|
||||
data.append([c + rng.gauss(0, 0.2) for c in center])
|
||||
return data
|
||||
|
||||
|
||||
def mean(xs):
|
||||
return sum(xs) / max(len(xs), 1)
|
||||
|
||||
|
||||
def main():
|
||||
rng = random.Random(7)
|
||||
in_dim, hidden, z_dim = 8, 10, 2
|
||||
params = init_vae(in_dim, hidden, z_dim, rng)
|
||||
data = sample_mixture(60, in_dim, rng)
|
||||
|
||||
beta = 0.2
|
||||
lr = 0.01
|
||||
print(f"=== training tiny VAE: {in_dim}-D input, {z_dim}-D latent, beta={beta} ===")
|
||||
for epoch in range(40):
|
||||
losses, recons, kls = [], [], []
|
||||
for x in data:
|
||||
eps = [rng.gauss(0, 1) for _ in range(z_dim)]
|
||||
fwd = forward(x, params, eps)
|
||||
total, recon, kl = loss_value(x, fwd, beta)
|
||||
grads = backward(x, fwd, params, beta)
|
||||
apply_update(params, grads, lr)
|
||||
losses.append(total); recons.append(recon); kls.append(kl)
|
||||
if (epoch + 1) % 5 == 0:
|
||||
print(f"epoch {epoch+1:2d}: loss {mean(losses):.3f} recon {mean(recons):.3f} KL {mean(kls):.3f}")
|
||||
|
||||
print()
|
||||
print("=== reconstruction on held-out sample ===")
|
||||
x_test = sample_mixture(1, in_dim, rng)[0]
|
||||
eps = [0.0] * z_dim
|
||||
fwd = forward(x_test, params, eps)
|
||||
mse = sum((a - b) ** 2 for a, b in zip(x_test, fwd["x_hat"]))
|
||||
print(" x =", [f"{v:+.2f}" for v in x_test])
|
||||
print(" x_hat =", [f"{v:+.2f}" for v in fwd["x_hat"]])
|
||||
print(f" mse = {mse:.3f}")
|
||||
|
||||
print()
|
||||
print("=== samples from N(0, I) through decoder ===")
|
||||
for _ in range(4):
|
||||
z = [rng.gauss(0, 1) for _ in range(z_dim)]
|
||||
h = tanh(add(matmul(params["dec"]["W1"], z), params["dec"]["b1"]))
|
||||
x_hat = add(matmul(params["dec"]["W_out"], h), params["dec"]["b_out"])
|
||||
print(f" z={[f'{zi:+.2f}' for zi in z]} -> x_hat={[f'{v:+.2f}' for v in x_hat]}")
|
||||
|
||||
print()
|
||||
print("takeaway: decoder turns N(0, I) samples into structured 8-D vectors")
|
||||
print(" that resemble the two-cluster training data.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,152 @@
|
||||
# Autoencoders & Variational Autoencoders (VAE)
|
||||
|
||||
> A plain autoencoder compresses then reconstructs. It memorizes. It does not generate. Add one trick — force the code to look Gaussian — and you get a sampler. That single trick, the reparameterization of `z = μ + σ·ε`, is why every latent-diffusion and flow-matching image model you use in 2026 has a VAE at the input.
|
||||
|
||||
**Type:** Build
|
||||
**Languages:** Python
|
||||
**Prerequisites:** Phase 3 · 02 (Backprop), Phase 3 · 07 (CNNs), Phase 8 · 01 (Taxonomy)
|
||||
**Time:** ~75 minutes
|
||||
|
||||
## The Problem
|
||||
|
||||
Compress a 784-pixel MNIST digit to a 16-number code, then reconstruct. A plain autoencoder will ace reconstruction MSE but the code space is a lumpy mess. Pick a random point in the code space, decode it, and you get noise. It has no sampler. It is a compression model dressed up.
|
||||
|
||||
What you actually want is: (a) the code space is a clean, smooth distribution you can sample from — say an isotropic Gaussian `N(0, I)`, (b) decoding any sample produces a plausible digit, and (c) the encoder and decoder still compress well. Three goals, one architecture, one loss.
|
||||
|
||||
Kingma's 2013 VAE solves this by training the encoder to output a *distribution* `q(z|x) = N(μ(x), σ(x)²)`, pulling that distribution toward the prior `N(0, I)` via a KL penalty, and then sampling `z` from `q(z|x)` before decoding. At inference time, drop the encoder, sample `z ~ N(0, I)`, decode. The KL penalty is what forces the code space to be structured.
|
||||
|
||||
In 2026 VAEs rarely ship standalone — they have been outclassed by diffusion for raw image quality — but they are the encoder of choice for every latent-diffusion model (SD 1/2/XL/3, Flux, AudioCraft). Learn the VAE and you learn the invisible first layer of every image pipeline you use.
|
||||
|
||||
## The Concept
|
||||
|
||||
|
||||
|
||||
**Autoencoder.** `z = encoder(x)`, `x̂ = decoder(z)`, loss = `||x - x̂||²`. Code space unstructured.
|
||||
|
||||
**VAE encoder.** Outputs two vectors: `μ(x)` and `log σ²(x)`. These define `q(z|x) = N(μ, diag(σ²))`.
|
||||
|
||||
**Reparameterization trick.** Sampling from `q(z|x)` is not differentiable. Rewrite the sample as `z = μ + σ·ε` where `ε ~ N(0, I)`. Now `z` is a deterministic function of `(μ, σ)` plus a non-parameter noise — gradients flow through `μ` and `σ`.
|
||||
|
||||
**Loss.** Evidence Lower BOund (ELBO), two terms:
|
||||
|
||||
```
|
||||
loss = reconstruction + β · KL[q(z|x) || N(0, I)]
|
||||
= ||x - x̂||² + β · Σ_i ( σ_i² + μ_i² - log σ_i² - 1 ) / 2
|
||||
```
|
||||
|
||||
Reconstruction pushes `x̂` toward `x`. KL pushes `q(z|x)` toward the prior. They trade off. Small β (<1) = sharper samples, code space less Gaussian. Large β (>1) = cleaner code space, blurrier samples. β-VAE (Higgins 2017) made this knob famous and kicked off disentanglement research.
|
||||
|
||||
**Sampling.** At inference: draw `z ~ N(0, I)`, forward through decoder. One forward pass — no iterative sampling like diffusion.
|
||||
|
||||
## Build It
|
||||
|
||||
`code/main.py` implements a tiny VAE without numpy or torch. Input is 8-dimensional synthetic data drawn from a 2-component Gaussian mixture in 8-D. Encoder and decoder are single hidden-layer MLPs. We implement tanh activation, forward pass, loss, and a hand-written backward pass. Not production — pedagogy.
|
||||
|
||||
### Step 1: encoder forward
|
||||
|
||||
```python
|
||||
def encode(x, enc):
|
||||
h = tanh(add(matmul(enc["W1"], x), enc["b1"]))
|
||||
mu = add(matmul(enc["W_mu"], h), enc["b_mu"])
|
||||
log_sigma2 = add(matmul(enc["W_sig"], h), enc["b_sig"])
|
||||
return mu, log_sigma2
|
||||
```
|
||||
|
||||
`log σ²` instead of `σ` so the network output is unconstrained (softplus of σ is a trap — gradients die at σ ≈ 0).
|
||||
|
||||
### Step 2: reparameterize and decode
|
||||
|
||||
```python
|
||||
def reparameterize(mu, log_sigma2, rng):
|
||||
eps = [rng.gauss(0, 1) for _ in mu]
|
||||
sigma = [math.exp(0.5 * lv) for lv in log_sigma2]
|
||||
return [m + s * e for m, s, e in zip(mu, sigma, eps)]
|
||||
|
||||
def decode(z, dec):
|
||||
h = tanh(add(matmul(dec["W1"], z), dec["b1"]))
|
||||
return add(matmul(dec["W_out"], h), dec["b_out"])
|
||||
```
|
||||
|
||||
### Step 3: the ELBO
|
||||
|
||||
```python
|
||||
def elbo(x, x_hat, mu, log_sigma2, beta=1.0):
|
||||
recon = sum((a - b) ** 2 for a, b in zip(x, x_hat))
|
||||
kl = 0.5 * sum(math.exp(lv) + m * m - lv - 1 for m, lv in zip(mu, log_sigma2))
|
||||
return recon + beta * kl, recon, kl
|
||||
```
|
||||
|
||||
Exact closed-form KL because both distributions are Gaussian. Do not integrate numerically. People still ship code with monte-carlo KL estimates in 2026 — it is 3x slower for no reason.
|
||||
|
||||
### Step 4: generate
|
||||
|
||||
```python
|
||||
def sample(dec, z_dim, rng):
|
||||
z = [rng.gauss(0, 1) for _ in range(z_dim)]
|
||||
return decode(z, dec)
|
||||
```
|
||||
|
||||
That is the generative model. Five lines.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Posterior collapse.** KL term drives `q(z|x) → N(0, I)` so aggressively that `z` carries no info about `x`. Fix: β-annealing (start β=0, ramp to 1), free bits, or skip the KL on inactive dimensions.
|
||||
- **Blurry samples.** The Gaussian decoder likelihood implies MSE reconstruction, which is Bayes-optimal for L2 (the mean) — the mean of a set of plausible digits is a fuzzy digit. Fix: discrete decoder (VQ-VAE, NVAE), or use the VAE only as an encoder and stack diffusion on the latents (this is what Stable Diffusion does).
|
||||
- **β too large, too early.** See posterior collapse. Start at β≈0.01 and ramp.
|
||||
- **Latent dim too small.** 16-D works for MNIST, 256-D for ImageNet 256², 2048-D for ImageNet 1024². Stable Diffusion's VAE compresses 512×512×3 → 64×64×4 (32x downsample factor in spatial area, 32x in channels).
|
||||
|
||||
## Use It
|
||||
|
||||
The 2026 VAE stack:
|
||||
|
||||
| Situation | Pick |
|
||||
|-----------|------|
|
||||
| Image-latent encoder for diffusion | Stable Diffusion VAE (`sd-vae-ft-ema`) or Flux VAE |
|
||||
| Audio-latent encoder | Encodec (Meta), SoundStream, or DAC (Descript) |
|
||||
| Video latents | Sora's spatiotemporal patches, Latte VAE, WAN VAE |
|
||||
| Disentangled representation learning | β-VAE, FactorVAE, TCVAE |
|
||||
| Discrete latents (for transformer modelling) | VQ-VAE, RVQ (ResidualVQ) |
|
||||
| Continuous latents for generation | Plain VAE, then condition a flow/diffusion model in that latent space |
|
||||
|
||||
A latent-diffusion model is a VAE with a diffusion model living between encoder and decoder. The VAE does coarse compression, the diffusion model does the heavy lifting. Same pattern for video (VAE + video-diffusion DiT) and audio (Encodec + MusicGen transformer).
|
||||
|
||||
## Ship It
|
||||
|
||||
Save `outputs/skill-vae-trainer.md`.
|
||||
|
||||
Skill takes: dataset profile + latent-dim target + downstream use (reconstruction, sampling, or latent-diffusion input) and outputs: architecture choice (plain/β/VQ/RVQ), β schedule, latent dim, decoder likelihood (Gaussian vs categorical), and evaluation plan (recon MSE, KL per dim, Fréchet distance between `q(z|x)` and `N(0, I)`).
|
||||
|
||||
## Exercises
|
||||
|
||||
1. **Easy.** Change `β` in `code/main.py` to `0.01`, `0.1`, `1.0`, `5.0`. Record the final reconstruction MSE and KL. Which β is Pareto-best for your synthetic data?
|
||||
2. **Medium.** Replace the Gaussian decoder likelihood with a Bernoulli likelihood (cross-entropy loss). Compare sample quality on a binarized version of the same synthetic data.
|
||||
3. **Hard.** Extend `code/main.py` into a mini VQ-VAE: replace the continuous `z` with a nearest-neighbour lookup in a codebook of K=32 entries. Compare reconstruction MSE and report how many codebook entries get used (codebook collapse is real).
|
||||
|
||||
## Key Terms
|
||||
|
||||
| Term | What people say | What it actually means |
|
||||
|------|-----------------|-----------------------|
|
||||
| Autoencoder | Encode-decode network | `x → z → x̂`, learn MSE. Not generative. |
|
||||
| VAE | AE with a sampler | Encoder outputs a distribution, KL penalty shapes code space. |
|
||||
| ELBO | Evidence lower bound | `log p(x) ≥ recon - KL[q(z|x) \|\| p(z)]`; tight when `q = p(z|x)`. |
|
||||
| Reparameterization | `z = μ + σ·ε` | Rewrites stochastic node as deterministic + pure noise. Enables backprop through sampling. |
|
||||
| Prior | `p(z)` | Target distribution for the latent, typically `N(0, I)`. |
|
||||
| Posterior collapse | "KL term wins" | Encoder ignores `x`, outputs the prior; decoder must hallucinate. |
|
||||
| β-VAE | Tunable KL weight | `loss = recon + β·KL`. Higher β = more disentangled but blurrier. |
|
||||
| VQ-VAE | Discrete latent | Replace continuous `z` with nearest codebook vector; enables transformer modelling. |
|
||||
|
||||
## Production note: the VAE is the hottest path in a diffusion server
|
||||
|
||||
In a Stable Diffusion / Flux / SD3 pipeline the VAE is called twice per request — once to encode (if doing img2img / inpainting) and once to decode. At 1024² the decoder pass is often the single largest activation-memory peak in the whole pipeline because it upsamples `128×128×16` latents back to `1024×1024×3`. Two practical consequences:
|
||||
|
||||
- **Slice or tile the decode.** `diffusers` exposes `pipe.vae.enable_slicing()` and `pipe.vae.enable_tiling()`. Tiling trades a small seam artifact for `O(tile²)` memory instead of `O(H·W)`. Essential for 1024²+ on consumer GPUs.
|
||||
- **bf16 decoder, fp32 numerics for the final resize.** The SD 1.x VAE was released in fp32 and *silently produces NaNs* when cast to fp16 at 1024²+. SDXL ships `madebyollin/sdxl-vae-fp16-fix` — always prefer the fp16-fix variant or use bf16.
|
||||
|
||||
## Further Reading
|
||||
|
||||
- [Kingma & Welling (2013). Auto-Encoding Variational Bayes](https://arxiv.org/abs/1312.6114) — the VAE paper.
|
||||
- [Higgins et al. (2017). β-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework](https://openreview.net/forum?id=Sy2fzU9gl) — disentangled β-VAE.
|
||||
- [van den Oord et al. (2017). Neural Discrete Representation Learning](https://arxiv.org/abs/1711.00937) — VQ-VAE.
|
||||
- [Vahdat & Kautz (2021). NVAE: A Deep Hierarchical Variational Autoencoder](https://arxiv.org/abs/2007.03898) — state-of-the-art image VAE.
|
||||
- [Rombach et al. (2022). High-Resolution Image Synthesis with Latent Diffusion Models](https://arxiv.org/abs/2112.10752) — Stable Diffusion; VAE as encoder.
|
||||
- [Défossez et al. (2022). High Fidelity Neural Audio Compression](https://arxiv.org/abs/2210.13438) — Encodec, the audio VAE standard.
|
||||
@@ -0,0 +1,18 @@
|
||||
---
|
||||
name: vae-trainer
|
||||
description: Specify VAE architecture, latent size, beta schedule, and eval plan for a given dataset and downstream use.
|
||||
version: 1.0.0
|
||||
phase: 8
|
||||
lesson: 02
|
||||
tags: [vae, latent, generative]
|
||||
---
|
||||
|
||||
Given a dataset profile (modality, resolution, dataset size) and the downstream use (reconstruction only, sampling, or input-encoder for a latent-diffusion or token-AR model), output:
|
||||
|
||||
1. Variant. Plain VAE, beta-VAE, VQ-VAE, RVQ (residual), or NVAE. One-sentence reason tied to modality and downstream use.
|
||||
2. Architecture. Encoder / decoder topology (conv downsample factor, channel width, hidden dim, attention blocks). Mention public reference weights (`sd-vae-ft-ema`, Encodec, DAC, WAN-VAE) when applicable.
|
||||
3. Latent dim. Spatial and channel dims. Total bits per sample. Compression ratio vs the raw data.
|
||||
4. Beta schedule. Warmup ramp, final value, and free-bits threshold if used.
|
||||
5. Eval plan. Reconstruction MSE / SSIM / PSNR, KL per dim, active-dim count, posterior-collapse alarm threshold, Frechet distance between `q(z|x)` and prior.
|
||||
|
||||
Refuse to ship a VAE with beta > 0.5 at training start (posterior collapse). Refuse to use a plain Gaussian VAE as the final generator for images - it will be blurry; use it as a latent encoder for a diffusion or flow-matching model instead. Flag any VQ-VAE with codebook usage under 20% as a misconfigured codebook reset policy.
|
||||
@@ -0,0 +1,71 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 460" font-family="Georgia, 'Times New Roman', serif">
|
||||
<defs>
|
||||
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto">
|
||||
<path d="M0,0 L10,5 L0,10 z" fill="#1a1a1a"/>
|
||||
</marker>
|
||||
<style>
|
||||
.box { fill: #faf6ef; stroke: #1a1a1a; stroke-width: 1.5; }
|
||||
.hot { fill: #fff1d6; stroke: #c0392b; stroke-width: 1.5; }
|
||||
.cold { fill: #eaf4ff; stroke: #2c5f8c; stroke-width: 1.5; }
|
||||
.label { font-size: 14px; font-weight: 600; fill: #1a1a1a; }
|
||||
.content { font-size: 12px; fill: #333; }
|
||||
.mono { font-size: 12px; fill: #333; font-family: 'Menlo', monospace; }
|
||||
.caption { font-size: 11px; fill: #555; font-style: italic; }
|
||||
.title { font-size: 16px; font-weight: 700; fill: #1a1a1a; }
|
||||
</style>
|
||||
</defs>
|
||||
|
||||
<text x="450" y="28" text-anchor="middle" class="title">adversarial training: two networks, one loss</text>
|
||||
|
||||
<!-- noise -->
|
||||
<rect x="40" y="100" width="80" height="50" class="box"/>
|
||||
<text x="80" y="128" text-anchor="middle" class="content">z ~ N(0, I)</text>
|
||||
<line x1="120" y1="125" x2="170" y2="125" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
|
||||
<!-- generator -->
|
||||
<rect x="170" y="90" width="120" height="70" class="cold"/>
|
||||
<text x="230" y="118" text-anchor="middle" class="label">G(z)</text>
|
||||
<text x="230" y="138" text-anchor="middle" class="caption">generator</text>
|
||||
|
||||
<line x1="290" y1="125" x2="340" y2="125" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
|
||||
<!-- fake sample -->
|
||||
<rect x="340" y="100" width="80" height="50" class="box"/>
|
||||
<text x="380" y="128" text-anchor="middle" class="content">x̂ (fake)</text>
|
||||
|
||||
<!-- real data -->
|
||||
<rect x="340" y="220" width="80" height="50" class="box"/>
|
||||
<text x="380" y="248" text-anchor="middle" class="content">x (real)</text>
|
||||
|
||||
<!-- discriminator -->
|
||||
<line x1="420" y1="125" x2="470" y2="140" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
<line x1="420" y1="245" x2="470" y2="220" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
|
||||
<rect x="470" y="135" width="130" height="90" class="hot"/>
|
||||
<text x="535" y="170" text-anchor="middle" class="label">D(x)</text>
|
||||
<text x="535" y="195" text-anchor="middle" class="caption">discriminator</text>
|
||||
<text x="535" y="215" text-anchor="middle" class="caption">real? 0-1</text>
|
||||
|
||||
<line x1="600" y1="180" x2="660" y2="180" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
|
||||
<!-- loss -->
|
||||
<rect x="660" y="150" width="200" height="60" class="box"/>
|
||||
<text x="760" y="180" text-anchor="middle" class="label">BCE loss</text>
|
||||
<text x="760" y="200" text-anchor="middle" class="caption">real -> 1, fake -> 0</text>
|
||||
|
||||
<!-- objective -->
|
||||
<rect x="40" y="300" width="820" height="50" class="box"/>
|
||||
<text x="450" y="320" text-anchor="middle" class="label">minimax: min_G max_D E_real[log D(x)] + E_fake[log(1 - D(G(z)))]</text>
|
||||
<text x="450" y="340" text-anchor="middle" class="caption">use non-saturating G loss -log D(G(z)) to avoid vanishing gradients</text>
|
||||
|
||||
<!-- failures -->
|
||||
<rect x="40" y="370" width="400" height="70" class="hot"/>
|
||||
<text x="240" y="392" text-anchor="middle" class="label">failure: D wins</text>
|
||||
<text x="240" y="412" text-anchor="middle" class="caption">D(fake) → 0, gradient to G vanishes</text>
|
||||
<text x="240" y="428" text-anchor="middle" class="caption">fix: cut D lr, add input noise, WGAN</text>
|
||||
|
||||
<rect x="460" y="370" width="400" height="70" class="hot"/>
|
||||
<text x="660" y="392" text-anchor="middle" class="label">failure: G collapses</text>
|
||||
<text x="660" y="412" text-anchor="middle" class="caption">G outputs one mode, D can't penalize</text>
|
||||
<text x="660" y="428" text-anchor="middle" class="caption">fix: minibatch disc, spectral norm, PacGAN</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 3.8 KiB |
@@ -0,0 +1,191 @@
|
||||
import math
|
||||
import random
|
||||
|
||||
|
||||
def sigmoid(x):
|
||||
if x >= 0:
|
||||
z = math.exp(-x)
|
||||
return 1 / (1 + z)
|
||||
z = math.exp(x)
|
||||
return z / (1 + z)
|
||||
|
||||
|
||||
def leaky_relu(x, a=0.2):
|
||||
return x if x > 0 else a * x
|
||||
|
||||
|
||||
def leaky_grad(x, a=0.2):
|
||||
return 1.0 if x > 0 else a
|
||||
|
||||
|
||||
def randn_matrix(rows, cols, rng, scale=0.3):
|
||||
return [[rng.gauss(0, scale) for _ in range(cols)] for _ in range(rows)]
|
||||
|
||||
|
||||
def matmul(W, x):
|
||||
return [sum(w * xi for w, xi in zip(row, x)) for row in W]
|
||||
|
||||
|
||||
def add(a, b):
|
||||
return [x + y for x, y in zip(a, b)]
|
||||
|
||||
|
||||
def init_mlp(in_dim, hidden, out_dim, rng):
|
||||
return {
|
||||
"W1": randn_matrix(hidden, in_dim, rng),
|
||||
"b1": [0.0] * hidden,
|
||||
"W2": randn_matrix(out_dim, hidden, rng),
|
||||
"b2": [0.0] * out_dim,
|
||||
}
|
||||
|
||||
|
||||
def forward_g(z, G):
|
||||
pre1 = add(matmul(G["W1"], z), G["b1"])
|
||||
h = [leaky_relu(v) for v in pre1]
|
||||
pre2 = add(matmul(G["W2"], h), G["b2"])
|
||||
return pre2, h, pre1
|
||||
|
||||
|
||||
def forward_d(x, D):
|
||||
pre1 = add(matmul(D["W1"], x), D["b1"])
|
||||
h = [leaky_relu(v) for v in pre1]
|
||||
pre2 = add(matmul(D["W2"], h), D["b2"])
|
||||
return sigmoid(pre2[0]), h, pre1, pre2[0]
|
||||
|
||||
|
||||
def sample_real(n, rng):
|
||||
out = []
|
||||
for _ in range(n):
|
||||
if rng.random() < 0.5:
|
||||
out.append([rng.gauss(-2.0, 0.4)])
|
||||
else:
|
||||
out.append([rng.gauss(2.0, 0.4)])
|
||||
return out
|
||||
|
||||
|
||||
def sample_noise(n, z_dim, rng):
|
||||
return [[rng.gauss(0, 1) for _ in range(z_dim)] for _ in range(n)]
|
||||
|
||||
|
||||
def update_d(reals, fakes, D, lr):
|
||||
"""Gradient step on D to maximize log D(x) + log(1 - D(G(z)))."""
|
||||
grads = {k: None for k in D}
|
||||
for part in D:
|
||||
if isinstance(D[part][0], list):
|
||||
grads[part] = [[0.0] * len(D[part][0]) for _ in D[part]]
|
||||
else:
|
||||
grads[part] = [0.0] * len(D[part])
|
||||
|
||||
def accumulate(x, target):
|
||||
p, h, pre1, pre2 = forward_d(x, D)
|
||||
dL_dpre2 = p - target
|
||||
grads["b2"][0] += dL_dpre2
|
||||
for j in range(len(h)):
|
||||
grads["W2"][0][j] += dL_dpre2 * h[j]
|
||||
dh = [D["W2"][0][j] * dL_dpre2 for j in range(len(h))]
|
||||
dpre1 = [dh[j] * leaky_grad(pre1[j]) for j in range(len(h))]
|
||||
for j in range(len(h)):
|
||||
grads["b1"][j] += dpre1[j]
|
||||
for k in range(len(x)):
|
||||
grads["W1"][j][k] += dpre1[j] * x[k]
|
||||
|
||||
for x in reals:
|
||||
accumulate(x, 1.0)
|
||||
for x in fakes:
|
||||
accumulate(x, 0.0)
|
||||
|
||||
n = len(reals) + len(fakes)
|
||||
for part in D:
|
||||
if isinstance(D[part][0], list):
|
||||
for i in range(len(D[part])):
|
||||
for j in range(len(D[part][i])):
|
||||
D[part][i][j] -= lr * grads[part][i][j] / n
|
||||
else:
|
||||
for i in range(len(D[part])):
|
||||
D[part][i] -= lr * grads[part][i] / n
|
||||
|
||||
|
||||
def update_g(noise_batch, G, D, lr):
|
||||
"""Non-saturating G loss: maximize log D(G(z)). Gradient flows through both."""
|
||||
grads = {k: None for k in G}
|
||||
for part in G:
|
||||
if isinstance(G[part][0], list):
|
||||
grads[part] = [[0.0] * len(G[part][0]) for _ in G[part]]
|
||||
else:
|
||||
grads[part] = [0.0] * len(G[part])
|
||||
|
||||
for z in noise_batch:
|
||||
x_hat, g_h, g_pre1 = forward_g(z, G)
|
||||
p, d_h, d_pre1, d_pre2 = forward_d(x_hat, D)
|
||||
# dL/dpre2_D where L = -log(p) is -(1/p) * p*(1-p) = p - 1
|
||||
dL_dpre2 = p - 1.0
|
||||
# back through D to get dL / d x_hat
|
||||
dh_D = [D["W2"][0][j] * dL_dpre2 for j in range(len(d_h))]
|
||||
dpre1_D = [dh_D[j] * leaky_grad(d_pre1[j]) for j in range(len(d_h))]
|
||||
dL_dxhat = [0.0] * len(x_hat)
|
||||
for j in range(len(d_h)):
|
||||
for k in range(len(x_hat)):
|
||||
dL_dxhat[k] += D["W1"][j][k] * dpre1_D[j]
|
||||
# now back through G
|
||||
grads["b2"] = [grads["b2"][i] + dL_dxhat[i] for i in range(len(x_hat))]
|
||||
for i in range(len(x_hat)):
|
||||
for j in range(len(g_h)):
|
||||
grads["W2"][i][j] += dL_dxhat[i] * g_h[j]
|
||||
dh_G = [sum(G["W2"][i][j] * dL_dxhat[i] for i in range(len(x_hat)))
|
||||
for j in range(len(g_h))]
|
||||
dpre1_G = [dh_G[j] * leaky_grad(g_pre1[j]) for j in range(len(g_h))]
|
||||
for j in range(len(g_h)):
|
||||
grads["b1"][j] += dpre1_G[j]
|
||||
for k in range(len(z)):
|
||||
grads["W1"][j][k] += dpre1_G[j] * z[k]
|
||||
|
||||
n = len(noise_batch)
|
||||
for part in G:
|
||||
if isinstance(G[part][0], list):
|
||||
for i in range(len(G[part])):
|
||||
for j in range(len(G[part][i])):
|
||||
G[part][i][j] -= lr * grads[part][i][j] / n
|
||||
else:
|
||||
for i in range(len(G[part])):
|
||||
G[part][i] -= lr * grads[part][i] / n
|
||||
|
||||
|
||||
def mean(xs):
|
||||
return sum(xs) / max(len(xs), 1)
|
||||
|
||||
|
||||
def main():
|
||||
rng = random.Random(1)
|
||||
z_dim, hidden = 4, 16
|
||||
G = init_mlp(z_dim, hidden, 1, rng)
|
||||
D = init_mlp(1, hidden, 1, rng)
|
||||
|
||||
batch, g_lr, d_lr = 32, 0.02, 0.01
|
||||
print("=== training 1-D GAN on two-mode Gaussian mixture ===")
|
||||
for step in range(1, 801):
|
||||
reals = sample_real(batch, rng)
|
||||
noise = sample_noise(batch, z_dim, rng)
|
||||
fakes = [forward_g(z, G)[0] for z in noise]
|
||||
update_d(reals, fakes, D, d_lr)
|
||||
|
||||
noise = sample_noise(batch, z_dim, rng)
|
||||
update_g(noise, G, D, g_lr)
|
||||
|
||||
if step % 100 == 0:
|
||||
probe_fakes = [forward_g(z, G)[0][0] for z in sample_noise(400, z_dim, rng)]
|
||||
mode_a = sum(1 for v in probe_fakes if v < 0)
|
||||
mode_b = 400 - mode_a
|
||||
d_real = mean([forward_d(x, D)[0] for x in sample_real(100, rng)])
|
||||
d_fake = mean([forward_d([v], D)[0] for v in probe_fakes])
|
||||
warn = " [!] mode collapse" if min(mode_a, mode_b) < 50 else ""
|
||||
print(f"step {step:4d}: D(real)={d_real:.2f} D(fake)={d_fake:.2f} "
|
||||
f"modeA={mode_a:3d} modeB={mode_b:3d}{warn}")
|
||||
|
||||
print()
|
||||
print("=== final 10 generator samples ===")
|
||||
for z in sample_noise(10, z_dim, rng):
|
||||
print(f" G(z) = {forward_g(z, G)[0][0]:+.2f}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,163 @@
|
||||
# GANs — Generator vs Discriminator
|
||||
|
||||
> Goodfellow's trick in 2014 was to skip density entirely. Two networks. One makes fakes. One catches them. They fight until the fakes are indistinguishable from real. It shouldn't work. It often doesn't. When it does, the samples are still the sharpest in the literature for narrow domains.
|
||||
|
||||
**Type:** Build
|
||||
**Languages:** Python
|
||||
**Prerequisites:** Phase 3 · 02 (Backprop), Phase 3 · 08 (Optimizers), Phase 8 · 02 (VAE)
|
||||
**Time:** ~75 minutes
|
||||
|
||||
## The Problem
|
||||
|
||||
VAEs produce blurry samples because their MSE decoder loss is Bayes-optimal for the *mean* image — and the mean of many plausible digits is a fuzzy digit. You want a loss that rewards *plausibility*, not pixel-wise proximity to any one target. There is no closed-form for plausibility. You have to learn it.
|
||||
|
||||
Goodfellow's idea: train a classifier `D(x)` to distinguish real images from fakes. Train a generator `G(z)` to fool `D`. The loss signal for `G` is whatever `D` currently thinks makes something look real. This signal updates as `G` improves, chasing a moving target. If both networks converge, `G` has learned the data distribution without ever writing down `log p(x)`.
|
||||
|
||||
This is adversarial training. The math is a minimax game:
|
||||
|
||||
```
|
||||
min_G max_D E_real[log D(x)] + E_fake[log(1 - D(G(z)))]
|
||||
```
|
||||
|
||||
In 2026 GANs are no longer the SOTA generator (diffusion and flow matching ate that crown). But StyleGAN 2/3 remain the sharpest face models ever shipped, GAN discriminators are used as *perceptual losses* in diffusion training, and adversarial training powers the fast 1-step distillations (SDXL-Turbo, SD3-Turbo, LCM) that let you ship real-time diffusion.
|
||||
|
||||
## The Concept
|
||||
|
||||
|
||||
|
||||
**Generator `G(z)`.** Maps a noise vector `z ~ N(0, I)` to a sample `x̂`. A decoder-shaped network (dense or transposed conv).
|
||||
|
||||
**Discriminator `D(x)`.** Maps a sample to a scalar probability (or score). Real → 1, fake → 0.
|
||||
|
||||
**Loss.** Two alternating updates:
|
||||
|
||||
- **Train `D`:** `loss_D = -[ log D(x) + log(1 - D(G(z))) ]`. Binary cross-entropy on real=1, fake=0.
|
||||
- **Train `G`:** `loss_G = -log D(G(z))`. This is the *non-saturating* form Goodfellow used (original `log(1 - D(G(z)))` saturates and kills gradients when `D` is confident).
|
||||
|
||||
**Training loop.** One step of `D`, one step of `G`. Repeat.
|
||||
|
||||
**Why it works.** If `G` perfectly matches `p_data`, then `D` cannot do better than chance and outputs 0.5 everywhere; `G` gets no more gradient. Equilibrium.
|
||||
|
||||
**Why it breaks.** Mode collapse (`G` finds one mode `D` can't classify and mints it forever), vanishing gradient (`D` learns too fast and `log D` saturates), training instability (learning rates, batch sizes, anything).
|
||||
|
||||
## Variants that made GANs work
|
||||
|
||||
| Year | Innovation | Fix |
|
||||
|------|------------|-----|
|
||||
| 2015 | DCGAN | Conv/deconv, batch norm, LeakyReLU — the first stable architecture. |
|
||||
| 2017 | WGAN, WGAN-GP | Replace BCE with Wasserstein distance + gradient penalty. Fixes vanishing gradient. |
|
||||
| 2017 | Spectral normalization | Lipschitz-bound the discriminator. Still used in 2026 discriminators. |
|
||||
| 2018 | Progressive GAN | Train low-res first, add layers. First megapixel results. |
|
||||
| 2019 | StyleGAN / StyleGAN2 | Mapping network + adaptive instance norm. State of the art for fixed-domain photorealism. |
|
||||
| 2021 | StyleGAN3 | Alias-free, translation-equivariant — still the face gold standard in 2026. |
|
||||
| 2022 | StyleGAN-XL | Conditional, class-aware, larger scale. |
|
||||
| 2024 | R3GAN | Rebrands with stronger regularization; works on 1024² without tricks. |
|
||||
|
||||
## Build It
|
||||
|
||||
`code/main.py` trains a tiny GAN on 1-D data: a mixture of two Gaussians. Generator and discriminator are single-hidden-layer MLPs. We implement forward, backward, and the minimax loop by hand. The goal is to see the two key failure modes (mode collapse + vanishing gradient) as they happen.
|
||||
|
||||
### Step 1: non-saturating loss
|
||||
|
||||
The vanilla Goodfellow loss `log(1 - D(G(z)))` goes to 0 when D classifies G's fake as fake with high confidence. At that point the gradient for G is basically zero — G cannot improve. The non-saturating form `-log D(G(z))` has the opposite asymptote: it blows up when D is confident, giving G a strong signal.
|
||||
|
||||
```python
|
||||
def g_loss(d_fake):
|
||||
# maximize log D(G(z)) <=> minimize -log D(G(z))
|
||||
return -sum(math.log(max(p, 1e-8)) for p in d_fake) / len(d_fake)
|
||||
```
|
||||
|
||||
### Step 2: one discriminator step per generator step
|
||||
|
||||
```python
|
||||
for step in range(steps):
|
||||
# train D
|
||||
real_batch = sample_real(batch_size)
|
||||
fake_batch = [G(z) for z in sample_noise(batch_size)]
|
||||
update_D(real_batch, fake_batch)
|
||||
|
||||
# train G
|
||||
fake_batch = [G(z) for z in sample_noise(batch_size)] # fresh fakes
|
||||
update_G(fake_batch)
|
||||
```
|
||||
|
||||
Fresh fakes for G, otherwise gradients are stale.
|
||||
|
||||
### Step 3: watch for mode collapse
|
||||
|
||||
```python
|
||||
if step % 200 == 0:
|
||||
samples = [G(z) for z in sample_noise(500)]
|
||||
mode_a = sum(1 for s in samples if s < 0)
|
||||
mode_b = 500 - mode_a
|
||||
if min(mode_a, mode_b) < 50:
|
||||
print(" [!] mode collapse: one mode is starved")
|
||||
```
|
||||
|
||||
The canonical symptom: one of the two real modes stops being generated. The discriminator stops correcting it because it's never seen as a fake.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Discriminator too strong.** Cut D's learning rate by 2-5x, or add instance/layer noise. If D reaches >95% accuracy, G is dead.
|
||||
- **Generator memorizes a mode.** Add noise to D inputs, use a minibatch-discriminator layer, or switch to WGAN-GP.
|
||||
- **Batch norm leaking statistics.** Real batch + fake batch flowing through the same BN layer mixes their statistics. Use instance norm or spectral norm instead.
|
||||
- **Inception-score gaming.** FID and IS are noisy at low sample counts. Use ≥10k samples at eval.
|
||||
- **One-shot sampling is a lie for conditional tasks.** You still need CFG scales, truncation tricks, and re-sampling to get usable outputs.
|
||||
|
||||
## Use It
|
||||
|
||||
The 2026 GAN stack:
|
||||
|
||||
| Situation | Pick |
|
||||
|-----------|------|
|
||||
| Photoreal human faces, fixed pose | StyleGAN3 (sharpest, smallest) |
|
||||
| Anime / stylized faces | StyleGAN-XL or Stable Diffusion LoRA |
|
||||
| Image-to-image translation | Pix2Pix / CycleGAN (Phase 8 · 04) or ControlNet (Phase 8 · 08) |
|
||||
| Fast 1-step text-to-image | Adversarial distillation of diffusion (SDXL-Turbo, SD3-Turbo) |
|
||||
| Perceptual loss inside a diffusion trainer | Small GAN discriminator on image crops |
|
||||
| Anything multi-modal, open-ended | Don't — use diffusion or flow matching |
|
||||
|
||||
GANs are sharp but narrow. Once your domain opens up — photos, arbitrary text prompts, video — switch to diffusion. The adversarial trick lives on as a component (perceptual losses, distillation), not a standalone generator.
|
||||
|
||||
## Ship It
|
||||
|
||||
Save `outputs/skill-gan-debugger.md`. Skill takes a failing GAN run (loss curves, sample grid, dataset size) and outputs a ranked list of likely causes, one-line fixes, and a rerun protocol.
|
||||
|
||||
## Exercises
|
||||
|
||||
1. **Easy.** Run `code/main.py` with the stock settings. Then set `D_LR = 5 * G_LR` and rerun. How fast does G's loss collapse to a constant?
|
||||
2. **Medium.** Replace the Goodfellow BCE loss with the WGAN loss: `loss_D = E[D(fake)] - E[D(real)]`, `loss_G = -E[D(fake)]`, and clip D's weights to `[-0.01, 0.01]`. Is training more stable? Compare wall-clock convergence.
|
||||
3. **Hard.** Extend the 1-D example to 2-D data (mixture of 8 Gaussians on a ring). Track how many of the 8 modes the generator captures at steps 1k, 5k, 10k. Implement minibatch discrimination and re-measure.
|
||||
|
||||
## Key Terms
|
||||
|
||||
| Term | What people say | What it actually means |
|
||||
|------|-----------------|-----------------------|
|
||||
| Generator | "G" | Noise-to-sample network, `G: z → x̂`. |
|
||||
| Discriminator | "D" | Classifier `D: x → [0, 1]`, real vs fake. |
|
||||
| Minimax | "The game" | `min_G max_D` of a joint objective. |
|
||||
| Non-saturating loss | "The fix" | Use `-log D(G(z))` for G instead of `log(1 - D(G(z)))`. |
|
||||
| Mode collapse | "G memorized one thing" | Generator produces few distinct outputs despite diverse data. |
|
||||
| WGAN | "Wasserstein" | Replace BCE with Earth-Mover distance + gradient penalty; smoother gradient. |
|
||||
| Spectral norm | "Lipschitz trick" | Constrain D's weight norms to bound its slope; stabilizes training. |
|
||||
| StyleGAN | "The one that works" | Mapping network + AdaIN; best-in-class for faces, still in 2026. |
|
||||
|
||||
## Production note: one-shot inference is GAN's lasting advantage
|
||||
|
||||
GANs no longer win on sample quality for open-domain generation, but they still win on inference cost. In production-inference literature vocabulary a GAN has:
|
||||
|
||||
- **No prefill, no decode stages.** A single `G(z)` forward pass. TTFT ≈ total latency.
|
||||
- **No KV-cache pressure.** The only state is the weights. Batch size is bounded by activation memory, not cache.
|
||||
- **Trivial continuous batching.** Since every request takes the same fixed FLOPs, a static batch at the server's target occupancy is usually optimal. No in-flight scheduler needed.
|
||||
|
||||
This is why GAN distillation (SDXL-Turbo, SD3-Turbo, ADD, LCM) is the dominant technique for fast text-to-image in 2026: it collapses a 20-50-step diffusion pipeline into 1-4 GAN-style forward passes while keeping the distribution of a diffusion base. The adversarial loss survives as a training-time knob for turning slow generators into fast ones.
|
||||
|
||||
## Further Reading
|
||||
|
||||
- [Goodfellow et al. (2014). Generative Adversarial Nets](https://arxiv.org/abs/1406.2661) — the original GAN paper.
|
||||
- [Radford et al. (2015). Unsupervised Representation Learning with DCGAN](https://arxiv.org/abs/1511.06434) — the first stable architecture.
|
||||
- [Arjovsky, Chintala, Bottou (2017). Wasserstein GAN](https://arxiv.org/abs/1701.07875) — WGAN.
|
||||
- [Miyato et al. (2018). Spectral Normalization for GANs](https://arxiv.org/abs/1802.05957) — SN.
|
||||
- [Karras et al. (2020). Analyzing and Improving the Image Quality of StyleGAN](https://arxiv.org/abs/1912.04958) — StyleGAN2.
|
||||
- [Karras et al. (2021). Alias-Free Generative Adversarial Networks](https://arxiv.org/abs/2106.12423) — StyleGAN3.
|
||||
- [Sauer et al. (2023). Adversarial Diffusion Distillation](https://arxiv.org/abs/2311.17042) — SDXL-Turbo.
|
||||
@@ -0,0 +1,18 @@
|
||||
---
|
||||
name: gan-debugger
|
||||
description: Diagnose failing GAN training from loss curves and sample grids; prescribe one-line fixes.
|
||||
version: 1.0.0
|
||||
phase: 8
|
||||
lesson: 03
|
||||
tags: [gan, adversarial, debugging]
|
||||
---
|
||||
|
||||
Given a failing GAN run (D and G loss curves, sample grid, dataset size, optimizer config), output:
|
||||
|
||||
1. Diagnosis. One root cause from: mode collapse, D too strong, D too weak, vanishing gradient, batch-norm leakage, overfit D, learning-rate mismatch, bad init.
|
||||
2. Evidence. Pointer to the telltale in the loss curves or samples (e.g. "D(fake) < 0.05 by step 500 = D too strong").
|
||||
3. Fix. One concrete change. Examples: `lr_D = lr_G / 2`, replace BN with IN, add spectral norm to D, switch to WGAN-GP with lambda=10, cut batch size by 2, add 0.1 Gaussian noise to D inputs.
|
||||
4. Rerun protocol. Seeds to try, number of steps before re-evaluation, acceptance criterion (e.g. "FID drops below baseline by step 20k").
|
||||
5. Fallback. If the fix doesn't land in one rerun, what to try next. Usually: switch architecture (StyleGAN, R3GAN) or switch paradigm (diffusion, flow matching) if dataset is too diverse.
|
||||
|
||||
Refuse to recommend increasing G learning rate when D is already saturated. Refuse to add regularization to G when the real failure is D - fix D first. Flag any run that shows training collapse within 100 steps as likely bad init or lr blowup, not a deep algorithmic issue.
|
||||
@@ -0,0 +1,90 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 500" font-family="Georgia, 'Times New Roman', serif">
|
||||
<defs>
|
||||
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto">
|
||||
<path d="M0,0 L10,5 L0,10 z" fill="#1a1a1a"/>
|
||||
</marker>
|
||||
<style>
|
||||
.box { fill: #faf6ef; stroke: #1a1a1a; stroke-width: 1.5; }
|
||||
.hot { fill: #fff1d6; stroke: #c0392b; stroke-width: 1.5; }
|
||||
.cold { fill: #eaf4ff; stroke: #2c5f8c; stroke-width: 1.5; }
|
||||
.label { font-size: 14px; font-weight: 600; fill: #1a1a1a; }
|
||||
.content { font-size: 12px; fill: #333; }
|
||||
.mono { font-size: 12px; fill: #333; font-family: 'Menlo', monospace; }
|
||||
.caption { font-size: 11px; fill: #555; font-style: italic; }
|
||||
.title { font-size: 16px; font-weight: 700; fill: #1a1a1a; }
|
||||
</style>
|
||||
</defs>
|
||||
|
||||
<text x="450" y="28" text-anchor="middle" class="title">Pix2Pix — U-Net generator, PatchGAN discriminator</text>
|
||||
|
||||
<!-- input image -->
|
||||
<rect x="30" y="90" width="90" height="70" class="box"/>
|
||||
<text x="75" y="120" text-anchor="middle" class="content">input x</text>
|
||||
<text x="75" y="140" text-anchor="middle" class="caption">edge map</text>
|
||||
|
||||
<!-- U-Net generator -->
|
||||
<rect x="140" y="60" width="420" height="140" class="cold"/>
|
||||
<text x="350" y="82" text-anchor="middle" class="label">U-Net generator G(x)</text>
|
||||
|
||||
<!-- encoder blocks -->
|
||||
<rect x="160" y="100" width="40" height="80" class="box"/>
|
||||
<rect x="210" y="110" width="40" height="60" class="box"/>
|
||||
<rect x="260" y="120" width="40" height="40" class="box"/>
|
||||
<rect x="310" y="130" width="40" height="20" class="hot"/>
|
||||
<!-- decoder blocks -->
|
||||
<rect x="360" y="120" width="40" height="40" class="box"/>
|
||||
<rect x="410" y="110" width="40" height="60" class="box"/>
|
||||
<rect x="460" y="100" width="40" height="80" class="box"/>
|
||||
<rect x="510" y="95" width="30" height="90" class="box"/>
|
||||
|
||||
<!-- skip connections -->
|
||||
<path d="M 180,100 Q 270,65 510,100" fill="none" stroke="#c0392b" stroke-width="1.2" stroke-dasharray="4,3"/>
|
||||
<path d="M 230,110 Q 310,80 460,110" fill="none" stroke="#c0392b" stroke-width="1.2" stroke-dasharray="4,3"/>
|
||||
<path d="M 280,120 Q 330,95 410,120" fill="none" stroke="#c0392b" stroke-width="1.2" stroke-dasharray="4,3"/>
|
||||
|
||||
<text x="350" y="195" text-anchor="middle" class="caption">skip connections preserve high-freq detail</text>
|
||||
|
||||
<line x1="540" y1="130" x2="580" y2="130" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
|
||||
<!-- output -->
|
||||
<rect x="580" y="95" width="90" height="70" class="box"/>
|
||||
<text x="625" y="125" text-anchor="middle" class="content">output ŷ</text>
|
||||
<text x="625" y="145" text-anchor="middle" class="caption">photo</text>
|
||||
|
||||
<!-- real target -->
|
||||
<rect x="680" y="95" width="90" height="70" class="box"/>
|
||||
<text x="725" y="125" text-anchor="middle" class="content">target y</text>
|
||||
|
||||
<!-- L1 loss -->
|
||||
<rect x="780" y="95" width="80" height="70" class="hot"/>
|
||||
<text x="820" y="125" text-anchor="middle" class="mono">L1</text>
|
||||
<text x="820" y="145" text-anchor="middle" class="caption">λ = 100</text>
|
||||
|
||||
<line x1="670" y1="130" x2="680" y2="130" stroke="#1a1a1a" stroke-width="1.2"/>
|
||||
<line x1="770" y1="130" x2="780" y2="130" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
|
||||
<!-- PatchGAN D -->
|
||||
<rect x="30" y="250" width="220" height="100" class="hot"/>
|
||||
<text x="140" y="273" text-anchor="middle" class="label">PatchGAN D(x, ŷ)</text>
|
||||
<text x="140" y="295" text-anchor="middle" class="caption">output is an N × N grid</text>
|
||||
<text x="140" y="315" text-anchor="middle" class="caption">each cell judges ~70×70 patch</text>
|
||||
<text x="140" y="335" text-anchor="middle" class="caption">averaged → real / fake score</text>
|
||||
|
||||
<line x1="75" y1="160" x2="75" y2="250" stroke="#1a1a1a" stroke-width="1.2"/>
|
||||
<line x1="75" y1="250" x2="30" y2="280" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
<line x1="625" y1="165" x2="250" y2="280" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
|
||||
<!-- objective -->
|
||||
<rect x="290" y="250" width="570" height="100" class="box"/>
|
||||
<text x="575" y="275" text-anchor="middle" class="label">objective</text>
|
||||
<text x="575" y="298" text-anchor="middle" class="mono">L_G = -log D(x, G(x)) + 100 · ||y - G(x)||_1</text>
|
||||
<text x="575" y="320" text-anchor="middle" class="mono">L_D = -log D(x, y) - log (1 - D(x, G(x)))</text>
|
||||
<text x="575" y="340" text-anchor="middle" class="caption">L1 stabilizes + sharp edges; adv term fights blur</text>
|
||||
|
||||
<!-- CycleGAN callout -->
|
||||
<rect x="30" y="380" width="830" height="100" class="box"/>
|
||||
<text x="445" y="403" text-anchor="middle" class="label">CycleGAN (unpaired): add a second G and a cycle-consistency loss</text>
|
||||
<text x="445" y="425" text-anchor="middle" class="mono">G: X -> Y, F: Y -> X, loss += ||F(G(x)) - x||_1 + ||G(F(y)) - y||_1</text>
|
||||
<text x="445" y="447" text-anchor="middle" class="caption">no paired data needed; horses <-> zebras, summer <-> winter</text>
|
||||
<text x="445" y="465" text-anchor="middle" class="caption">2026: mostly superseded by ControlNet + IP-Adapter over diffusion</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 5.1 KiB |
@@ -0,0 +1,199 @@
|
||||
import math
|
||||
import random
|
||||
|
||||
|
||||
def sigmoid(x):
|
||||
if x >= 0:
|
||||
z = math.exp(-x)
|
||||
return 1 / (1 + z)
|
||||
z = math.exp(x)
|
||||
return z / (1 + z)
|
||||
|
||||
|
||||
def leaky(x, a=0.2):
|
||||
return x if x > 0 else a * x
|
||||
|
||||
|
||||
def leaky_grad(x, a=0.2):
|
||||
return 1.0 if x > 0 else a
|
||||
|
||||
|
||||
def randn_matrix(rows, cols, rng, scale=0.3):
|
||||
return [[rng.gauss(0, scale) for _ in range(cols)] for _ in range(rows)]
|
||||
|
||||
|
||||
def matmul(W, x):
|
||||
return [sum(w * xi for w, xi in zip(row, x)) for row in W]
|
||||
|
||||
|
||||
def add(a, b):
|
||||
return [x + y for x, y in zip(a, b)]
|
||||
|
||||
|
||||
def one_hot(c, num):
|
||||
v = [0.0] * num
|
||||
v[c] = 1.0
|
||||
return v
|
||||
|
||||
|
||||
def init_mlp(in_dim, hidden, out_dim, rng):
|
||||
return {
|
||||
"W1": randn_matrix(hidden, in_dim, rng),
|
||||
"b1": [0.0] * hidden,
|
||||
"W2": randn_matrix(out_dim, hidden, rng),
|
||||
"b2": [0.0] * out_dim,
|
||||
}
|
||||
|
||||
|
||||
def g_forward(z, c, G, num_classes):
|
||||
inp = z + one_hot(c, num_classes)
|
||||
pre1 = add(matmul(G["W1"], inp), G["b1"])
|
||||
h = [leaky(v) for v in pre1]
|
||||
out = add(matmul(G["W2"], h), G["b2"])
|
||||
return out, h, pre1, inp
|
||||
|
||||
|
||||
def d_forward(x, c, D, num_classes):
|
||||
inp = x + one_hot(c, num_classes)
|
||||
pre1 = add(matmul(D["W1"], inp), D["b1"])
|
||||
h = [leaky(v) for v in pre1]
|
||||
logit = add(matmul(D["W2"], h), D["b2"])[0]
|
||||
return sigmoid(logit), h, pre1, inp, logit
|
||||
|
||||
|
||||
def sample_real_conditional(n, num_classes, rng):
|
||||
out = []
|
||||
for _ in range(n):
|
||||
c = rng.randrange(num_classes)
|
||||
if c == 0:
|
||||
x = rng.gauss(-2.0, 0.3)
|
||||
else:
|
||||
x = rng.gauss(2.0, 0.3)
|
||||
out.append(([x], c))
|
||||
return out
|
||||
|
||||
|
||||
def update_d(reals, fakes, D, num_classes, lr):
|
||||
grads = init_grads(D)
|
||||
for (x, c) in reals:
|
||||
accumulate_d_grad(x, c, 1.0, D, num_classes, grads)
|
||||
for (x, c) in fakes:
|
||||
accumulate_d_grad(x, c, 0.0, D, num_classes, grads)
|
||||
n = len(reals) + len(fakes)
|
||||
apply_grads(D, grads, lr, n)
|
||||
|
||||
|
||||
def accumulate_d_grad(x, c, target, D, num_classes, grads):
|
||||
p, h, pre1, inp, _ = d_forward(x, c, D, num_classes)
|
||||
dL_dpre2 = p - target
|
||||
grads["b2"][0] += dL_dpre2
|
||||
for j in range(len(h)):
|
||||
grads["W2"][0][j] += dL_dpre2 * h[j]
|
||||
dh = [D["W2"][0][j] * dL_dpre2 for j in range(len(h))]
|
||||
dpre1 = [dh[j] * leaky_grad(pre1[j]) for j in range(len(h))]
|
||||
for j in range(len(h)):
|
||||
grads["b1"][j] += dpre1[j]
|
||||
for k in range(len(inp)):
|
||||
grads["W1"][j][k] += dpre1[j] * inp[k]
|
||||
|
||||
|
||||
def update_g(noise, cs, G, D, num_classes, lr, l1_w=0.0, targets=None):
|
||||
"""Non-saturating G loss + optional conditional L1 toward a target."""
|
||||
grads = init_grads(G)
|
||||
for i, z in enumerate(noise):
|
||||
c = cs[i]
|
||||
x_hat, g_h, g_pre1, g_inp = g_forward(z, c, G, num_classes)
|
||||
p, d_h, d_pre1, d_inp, d_logit = d_forward(x_hat, c, D, num_classes)
|
||||
dL_dpre2 = p - 1.0
|
||||
dh_D = [D["W2"][0][j] * dL_dpre2 for j in range(len(d_h))]
|
||||
dpre1_D = [dh_D[j] * leaky_grad(d_pre1[j]) for j in range(len(d_h))]
|
||||
dL_dxhat = [0.0] * len(x_hat)
|
||||
for j in range(len(d_h)):
|
||||
for k in range(len(x_hat)):
|
||||
dL_dxhat[k] += D["W1"][j][k] * dpre1_D[j]
|
||||
if l1_w > 0 and targets is not None:
|
||||
for k in range(len(x_hat)):
|
||||
dL_dxhat[k] += l1_w * (1.0 if x_hat[k] > targets[i][k] else -1.0)
|
||||
grads["b2"] = [grads["b2"][i] + dL_dxhat[i] for i in range(len(x_hat))]
|
||||
for a in range(len(x_hat)):
|
||||
for b in range(len(g_h)):
|
||||
grads["W2"][a][b] += dL_dxhat[a] * g_h[b]
|
||||
dh_G = [sum(G["W2"][a][b] * dL_dxhat[a] for a in range(len(x_hat)))
|
||||
for b in range(len(g_h))]
|
||||
dpre1_G = [dh_G[j] * leaky_grad(g_pre1[j]) for j in range(len(g_h))]
|
||||
for j in range(len(g_h)):
|
||||
grads["b1"][j] += dpre1_G[j]
|
||||
for k in range(len(g_inp)):
|
||||
grads["W1"][j][k] += dpre1_G[j] * g_inp[k]
|
||||
apply_grads(G, grads, lr, len(noise))
|
||||
|
||||
|
||||
def init_grads(net):
|
||||
grads = {}
|
||||
for k, v in net.items():
|
||||
if isinstance(v[0], list):
|
||||
grads[k] = [[0.0] * len(v[0]) for _ in v]
|
||||
else:
|
||||
grads[k] = [0.0] * len(v)
|
||||
return grads
|
||||
|
||||
|
||||
def apply_grads(net, grads, lr, n):
|
||||
for k, v in net.items():
|
||||
if isinstance(v[0], list):
|
||||
for i in range(len(v)):
|
||||
for j in range(len(v[i])):
|
||||
v[i][j] -= lr * grads[k][i][j] / n
|
||||
else:
|
||||
for i in range(len(v)):
|
||||
v[i] -= lr * grads[k][i] / n
|
||||
|
||||
|
||||
def mean(xs):
|
||||
return sum(xs) / max(len(xs), 1)
|
||||
|
||||
|
||||
def main():
|
||||
rng = random.Random(5)
|
||||
num_classes, z_dim, hidden = 2, 4, 16
|
||||
G = init_mlp(z_dim + num_classes, hidden, 1, rng)
|
||||
D = init_mlp(1 + num_classes, hidden, 1, rng)
|
||||
|
||||
batch, g_lr, d_lr = 32, 0.02, 0.01
|
||||
print("=== conditional GAN on two-mode mixture (class 0 -> -2, class 1 -> +2) ===")
|
||||
for step in range(1, 601):
|
||||
reals = sample_real_conditional(batch, num_classes, rng)
|
||||
cs = [c for _, c in reals]
|
||||
noise = [[rng.gauss(0, 1) for _ in range(z_dim)] for _ in range(batch)]
|
||||
fakes = [(g_forward(noise[i], cs[i], G, num_classes)[0], cs[i]) for i in range(batch)]
|
||||
update_d(reals, fakes, D, num_classes, d_lr)
|
||||
|
||||
noise = [[rng.gauss(0, 1) for _ in range(z_dim)] for _ in range(batch)]
|
||||
cs = [rng.randrange(num_classes) for _ in range(batch)]
|
||||
update_g(noise, cs, G, D, num_classes, g_lr)
|
||||
|
||||
if step % 150 == 0:
|
||||
probes = {c: [] for c in range(num_classes)}
|
||||
for _ in range(300):
|
||||
c = rng.randrange(num_classes)
|
||||
z = [rng.gauss(0, 1) for _ in range(z_dim)]
|
||||
probes[c].append(g_forward(z, c, G, num_classes)[0][0])
|
||||
line = f"step {step:4d}:"
|
||||
for c in range(num_classes):
|
||||
line += f" class {c}: mean {mean(probes[c]):+.2f} (n={len(probes[c])})"
|
||||
print(line)
|
||||
|
||||
print()
|
||||
print("=== sampling per class ===")
|
||||
for c in range(num_classes):
|
||||
z_batch = [[rng.gauss(0, 1) for _ in range(z_dim)] for _ in range(6)]
|
||||
outs = [g_forward(z, c, G, num_classes)[0][0] for z in z_batch]
|
||||
print(f" class {c}: " + " ".join(f"{v:+.2f}" for v in outs))
|
||||
|
||||
print()
|
||||
print("takeaway: G(z, c) learns a class-specific sampler.")
|
||||
print(" same architecture, one extra input, totally different samples.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,147 @@
|
||||
# Conditional GANs & Pix2Pix
|
||||
|
||||
> The first big unlock of 2014-2017 was controlling what a GAN makes. Attach a label, or an image, or a sentence. Pix2Pix did the image version and it still beats every generic text-to-image model on narrow image-to-image tasks.
|
||||
|
||||
**Type:** Build
|
||||
**Languages:** Python
|
||||
**Prerequisites:** Phase 8 · 03 (GANs), Phase 4 · 06 (U-Net), Phase 3 · 07 (CNNs)
|
||||
**Time:** ~75 minutes
|
||||
|
||||
## The Problem
|
||||
|
||||
An unconditional GAN samples arbitrary faces. Useful for a demo, useless in production. You want: *map a sketch to a photo*, *map a map to an aerial photo*, *map a daytime scene to nighttime*, *colorize a grayscale image*. In all of these, you are given an input image `x` and must output `y` with some semantic correspondence. There are many plausible `y`s per `x`. Mean-squared error flattens them into mush. An adversarial loss doesn't, because "looks real" is sharp.
|
||||
|
||||
Conditional GAN (Mirza & Osindero, 2014) adds a condition `c` as an input to both `G` and `D`. Pix2Pix (Isola et al., 2017) specialized this: condition is a full input image, generator is a U-Net, discriminator is a *patch-based* classifier (PatchGAN), and loss is adversarial + L1. That recipe outperforms from-scratch text-to-image models on narrow image-to-image domains even in 2026 because it is trained on *paired data* — you have exactly the signal you need.
|
||||
|
||||
## The Concept
|
||||
|
||||
|
||||
|
||||
**Conditional G.** `G(x, z) → y`. In Pix2Pix, `z` is dropout inside G (no input noise — Isola found explicit noise got ignored).
|
||||
|
||||
**Conditional D.** `D(x, y) → [0, 1]`. Input is the *pair* (condition, output). This is the key difference: D must judge whether `y` is consistent with `x`, not just whether `y` looks real.
|
||||
|
||||
**U-Net generator.** Encoder-decoder with skip connections across the bottleneck. Critical for tasks where input and output share low-level structure (edges, silhouette). Without the skips, high-frequency detail vanishes.
|
||||
|
||||
**PatchGAN discriminator.** Instead of outputting a single real/fake score, D outputs an `N×N` grid where each cell judges a receptive field of ~70×70 pixels. Averaged. This is a Markov random field assumption: realism is local. Much faster to train, fewer parameters, sharper output.
|
||||
|
||||
**Loss.**
|
||||
|
||||
```
|
||||
loss_G = -log D(x, G(x)) + λ · ||y - G(x)||_1
|
||||
loss_D = -log D(x, y) - log (1 - D(x, G(x)))
|
||||
```
|
||||
|
||||
The L1 term stabilizes training and pushes G toward the known target. L1 gives sharper edges than L2 (medians, not means). `λ = 100` was the Pix2Pix default.
|
||||
|
||||
## CycleGAN — when you don't have pairs
|
||||
|
||||
Pix2Pix needs paired `(x, y)` data. CycleGAN (Zhu et al., 2017) drops this requirement at the cost of an extra loss: the *cycle consistency* loss. Two generators `G: X → Y` and `F: Y → X`. Train them so `F(G(x)) ≈ x` and `G(F(y)) ≈ y`. This lets you translate horses to zebras, summer to winter, without paired examples.
|
||||
|
||||
In 2026, unpaired image-to-image is mostly done via diffusion (ControlNet, IP-Adapter) rather than CycleGAN, but the cycle-consistency idea survives in almost every unpaired domain adaptation paper.
|
||||
|
||||
## Build It
|
||||
|
||||
`code/main.py` implements a tiny conditional GAN on 1-D data. The condition `c` is a class label (0 or 1). The task: produce a sample from the conditional distribution for the given class.
|
||||
|
||||
### Step 1: append condition to both G and D inputs
|
||||
|
||||
```python
|
||||
def G(z, c, params):
|
||||
return mlp(concat([z, one_hot(c)]), params)
|
||||
|
||||
def D(x, c, params):
|
||||
return mlp(concat([x, one_hot(c)]), params)
|
||||
```
|
||||
|
||||
One-hot encoding is the simplest way. Larger models use learned embeddings, FiLM modulation, or cross-attention.
|
||||
|
||||
### Step 2: train conditional
|
||||
|
||||
```python
|
||||
for step in range(steps):
|
||||
x, c = sample_real_conditional()
|
||||
noise = sample_noise()
|
||||
update_D(x_real=x, x_fake=G(noise, c), c=c)
|
||||
update_G(noise, c)
|
||||
```
|
||||
|
||||
The generator must match the real distribution *for the given condition*, not the marginal.
|
||||
|
||||
### Step 3: verify per-class output
|
||||
|
||||
```python
|
||||
for c in [0, 1]:
|
||||
samples = [G(noise, c) for noise in batch]
|
||||
mean_c = mean(samples)
|
||||
assert_near(mean_c, real_mean_for_class_c)
|
||||
```
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Condition ignored.** G learns to marginalize, D never penalizes because condition signal is weak. Fix: condition D more aggressively (early layer, not just late), use projection discriminator (Miyato & Koyama 2018).
|
||||
- **L1 weight too low.** G drifts to arbitrary real-looking outputs, not faithful ones. Start λ≈100 for Pix2Pix-style tasks.
|
||||
- **L1 weight too high.** G produces blurry outputs because L1 is still an L_p norm. Anneal down once training stabilizes.
|
||||
- **Ground-truth leakage in D.** Concatenate `(x, y)` as D input, not just `y`. Without this D cannot check consistency.
|
||||
- **Mode collapse per class.** Each class can collapse independently. Run class-conditional diversity checks.
|
||||
|
||||
## Use It
|
||||
|
||||
2026 state of image-to-image tasks:
|
||||
|
||||
| Task | Best approach |
|
||||
|------|---------------|
|
||||
| Sketch → photo, same domain, paired data | Pix2Pix / Pix2PixHD (still fast, still sharp) |
|
||||
| Sketch → photo, unpaired | ControlNet with a Scribble conditioning model |
|
||||
| Semantic seg → photo | SPADE / GauGAN2 or SD + ControlNet-Seg |
|
||||
| Style transfer | Diffusion with IP-Adapter or LoRA; GAN methods are legacy |
|
||||
| Depth → photo | ControlNet-Depth over Stable Diffusion |
|
||||
| Super-resolution | Real-ESRGAN (GAN), ESRGAN-Plus, or SD-Upscale (diffusion) |
|
||||
| Colorization | ColTran, diffusion-based colorizers, or Pix2Pix-color |
|
||||
| Daytime → nighttime, seasons, weather | CycleGAN or ControlNet-based |
|
||||
|
||||
Pix2Pix remains the right tool when (a) you have thousands of paired examples, (b) the task is narrow and repeatable, and (c) you need fast inference. On generic open-domain tasks, diffusion wins.
|
||||
|
||||
## Ship It
|
||||
|
||||
Save `outputs/skill-img2img-chooser.md`. Skill takes a task description, data availability (paired vs unpaired, N samples), and latency/quality budget, then outputs: approach (Pix2Pix, CycleGAN, ControlNet variant, SDXL + IP-Adapter), training data requirements, inference cost, and eval protocol (LPIPS, FID, task-specific).
|
||||
|
||||
## Exercises
|
||||
|
||||
1. **Easy.** Modify `code/main.py` to add a third class. Confirm G still maps each class's noise to the correct mode.
|
||||
2. **Medium.** Replace L1 with a perceptual-style loss in the 1-D setting (e.g. a small frozen D acting as feature extractor). Does it change sharpness of the conditional distribution?
|
||||
3. **Hard.** Sketch a CycleGAN in the 1-D setting: two distributions, two generators, cycle loss. Show that it learns to map between them with no paired data.
|
||||
|
||||
## Key Terms
|
||||
|
||||
| Term | What people say | What it actually means |
|
||||
|------|-----------------|-----------------------|
|
||||
| Conditional GAN | "GAN with labels" | G(z, c), D(x, c). Both networks see the condition. |
|
||||
| Pix2Pix | "Image-to-image GAN" | Paired cGAN with U-Net G and PatchGAN D + L1 loss. |
|
||||
| U-Net | "Encoder-decoder with skips" | Symmetric conv network; skips preserve high-freq. |
|
||||
| PatchGAN | "Local-realism classifier" | D outputs per-patch score instead of global score. |
|
||||
| CycleGAN | "Unpaired image translation" | Two G's + cycle-consistency loss; no paired data. |
|
||||
| SPADE | "GauGAN" | Normalizes intermediate activations with the semantic map; segmentation-to-image. |
|
||||
| FiLM | "Feature-wise linear modulation" | Per-feature affine transform from the condition; cheap conditioning. |
|
||||
|
||||
## Production note: Pix2Pix as a latency-bound baseline
|
||||
|
||||
When you have paired data and a narrow task (sketch → render, semantic map → photo, day → night), Pix2Pix's one-shot inference beats diffusion by an order of magnitude on latency. The production comparison is usually:
|
||||
|
||||
| Path | Steps | Typical latency at 512² on a single L4 |
|
||||
|------|-------|----------------------------------------|
|
||||
| Pix2Pix (U-Net forward) | 1 | ~30 ms |
|
||||
| SD-Inpaint or SD-Img2Img | 20 | ~1.2 s |
|
||||
| SDXL-Turbo Img2Img | 1-4 | ~0.15-0.35 s |
|
||||
| ControlNet + SDXL base | 20-30 | ~3-5 s |
|
||||
|
||||
Pix2Pix wins on throughput in static batches (every request is the same FLOPs). Diffusion wins on quality and generalization. The modern play is often to ship a Pix2Pix-style distilled model for the narrow task and a diffusion fallback for tail inputs.
|
||||
|
||||
## Further Reading
|
||||
|
||||
- [Mirza & Osindero (2014). Conditional Generative Adversarial Nets](https://arxiv.org/abs/1411.1784) — the cGAN paper.
|
||||
- [Isola et al. (2017). Image-to-Image Translation with Conditional Adversarial Networks](https://arxiv.org/abs/1611.07004) — Pix2Pix.
|
||||
- [Zhu et al. (2017). Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks](https://arxiv.org/abs/1703.10593) — CycleGAN.
|
||||
- [Wang et al. (2018). High-Resolution Image Synthesis with Conditional GANs](https://arxiv.org/abs/1711.11585) — Pix2PixHD.
|
||||
- [Park et al. (2019). Semantic Image Synthesis with Spatially-Adaptive Normalization](https://arxiv.org/abs/1903.07291) — SPADE / GauGAN.
|
||||
- [Miyato & Koyama (2018). cGANs with Projection Discriminator](https://arxiv.org/abs/1802.05637) — the projection D.
|
||||
@@ -0,0 +1,18 @@
|
||||
---
|
||||
name: img2img-chooser
|
||||
description: Pick an image-to-image approach given paired vs unpaired data, domain specificity, and latency budget.
|
||||
version: 1.0.0
|
||||
phase: 8
|
||||
lesson: 04
|
||||
tags: [pix2pix, img2img, conditional]
|
||||
---
|
||||
|
||||
Given a task description (source domain, target domain, data availability - paired/unpaired/N samples, latency budget, quality bar), output:
|
||||
|
||||
1. Approach. Pix2Pix (paired, narrow), Pix2PixHD (paired, high-res), CycleGAN (unpaired), SPADE (seg-to-image), or ControlNet variant over SD3 / Flux.1 (general, open-domain).
|
||||
2. Training data spec. Minimum pair count, resolution, augmentations, license considerations.
|
||||
3. Architecture. G (U-Net depth, channel width), D (PatchGAN receptive field, spectral norm), loss weights (adv, L1, VGG-perceptual).
|
||||
4. Inference latency. Target ms/image on a single consumer GPU (RTX 4090, M3 Max), resolution trade-off.
|
||||
5. Eval. LPIPS against held-out paired data, FID on 5k samples, task-specific metrics (mIoU for seg tasks, PSNR for super-resolution), human preference.
|
||||
|
||||
Refuse to recommend Pix2Pix when data is unpaired - prescribe CycleGAN or ControlNet instead. Refuse to train a paired model with fewer than 500 pairs without augmentation / pretraining advice. Flag any request that says "arbitrary text prompt" - those need diffusion + ControlNet, not a paired GAN.
|
||||
@@ -0,0 +1,83 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 520" font-family="Georgia, 'Times New Roman', serif">
|
||||
<defs>
|
||||
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto">
|
||||
<path d="M0,0 L10,5 L0,10 z" fill="#1a1a1a"/>
|
||||
</marker>
|
||||
<style>
|
||||
.box { fill: #faf6ef; stroke: #1a1a1a; stroke-width: 1.5; }
|
||||
.hot { fill: #fff1d6; stroke: #c0392b; stroke-width: 1.5; }
|
||||
.cold { fill: #eaf4ff; stroke: #2c5f8c; stroke-width: 1.5; }
|
||||
.label { font-size: 14px; font-weight: 600; fill: #1a1a1a; }
|
||||
.content { font-size: 12px; fill: #333; }
|
||||
.mono { font-size: 12px; fill: #333; font-family: 'Menlo', monospace; }
|
||||
.caption { font-size: 11px; fill: #555; font-style: italic; }
|
||||
.title { font-size: 16px; font-weight: 700; fill: #1a1a1a; }
|
||||
</style>
|
||||
</defs>
|
||||
|
||||
<text x="450" y="28" text-anchor="middle" class="title">StyleGAN: mapping network + AdaIN + per-layer noise</text>
|
||||
|
||||
<!-- mapping network -->
|
||||
<rect x="30" y="90" width="80" height="40" class="box"/>
|
||||
<text x="70" y="114" text-anchor="middle" class="mono">z ~ N(0, I)</text>
|
||||
|
||||
<line x1="110" y1="110" x2="140" y2="110" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
|
||||
<rect x="140" y="80" width="130" height="60" class="cold"/>
|
||||
<text x="205" y="108" text-anchor="middle" class="label">mapping f(z)</text>
|
||||
<text x="205" y="126" text-anchor="middle" class="caption">8-layer MLP</text>
|
||||
|
||||
<line x1="270" y1="110" x2="310" y2="110" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
|
||||
<rect x="310" y="90" width="60" height="40" class="hot"/>
|
||||
<text x="340" y="114" text-anchor="middle" class="mono">w ∈ W</text>
|
||||
|
||||
<!-- synthesis -->
|
||||
<rect x="380" y="60" width="470" height="260" class="box"/>
|
||||
<text x="615" y="82" text-anchor="middle" class="label">synthesis g(const, w, noise)</text>
|
||||
|
||||
<rect x="400" y="100" width="80" height="30" class="hot"/>
|
||||
<text x="440" y="120" text-anchor="middle" class="mono">const 4×4×512</text>
|
||||
|
||||
<line x1="440" y1="130" x2="440" y2="160" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
<rect x="400" y="160" width="80" height="30" class="box"/>
|
||||
<text x="440" y="180" text-anchor="middle" class="content">conv 3×3</text>
|
||||
|
||||
<line x1="440" y1="190" x2="440" y2="220" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
<rect x="400" y="220" width="80" height="30" class="cold"/>
|
||||
<text x="440" y="240" text-anchor="middle" class="content">AdaIN(w)</text>
|
||||
|
||||
<line x1="480" y1="235" x2="530" y2="235" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
<rect x="530" y="220" width="80" height="30" class="box"/>
|
||||
<text x="570" y="240" text-anchor="middle" class="content">+ noise</text>
|
||||
|
||||
<line x1="610" y1="235" x2="660" y2="235" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
<rect x="660" y="220" width="60" height="30" class="box"/>
|
||||
<text x="690" y="240" text-anchor="middle" class="content">up 2x</text>
|
||||
|
||||
<line x1="720" y1="235" x2="760" y2="235" stroke="#1a1a1a" stroke-width="1.2" stroke-dasharray="3,3"/>
|
||||
<text x="790" y="240" text-anchor="middle" class="caption">...repeat to 1024</text>
|
||||
|
||||
<!-- AdaIN formula -->
|
||||
<rect x="380" y="280" width="470" height="30" class="hot"/>
|
||||
<text x="615" y="300" text-anchor="middle" class="mono">AdaIN(x, w) = scale(w) · (x - μ) / σ + bias(w)</text>
|
||||
|
||||
<!-- w injection arrows -->
|
||||
<path d="M 340,130 Q 360,200 400,235" fill="none" stroke="#2c5f8c" stroke-width="1.2" stroke-dasharray="3,3" marker-end="url(#arrow)"/>
|
||||
|
||||
<!-- truncation -->
|
||||
<rect x="30" y="360" width="400" height="130" class="box"/>
|
||||
<text x="230" y="385" text-anchor="middle" class="label">truncation trick</text>
|
||||
<text x="230" y="408" text-anchor="middle" class="mono">w′ = w̄ + ψ · (w - w̄)</text>
|
||||
<text x="230" y="432" text-anchor="middle" class="caption">ψ = 1.0 → full diversity, occasional glitches</text>
|
||||
<text x="230" y="452" text-anchor="middle" class="caption">ψ = 0.7 → default demo setting</text>
|
||||
<text x="230" y="472" text-anchor="middle" class="caption">ψ = 0.0 → mean image, no variation</text>
|
||||
|
||||
<!-- version evolution -->
|
||||
<rect x="460" y="360" width="400" height="130" class="cold"/>
|
||||
<text x="660" y="385" text-anchor="middle" class="label">v1 → v2 → v3</text>
|
||||
<text x="660" y="408" text-anchor="middle" class="caption">v2: weight demodulation, no droplets</text>
|
||||
<text x="660" y="428" text-anchor="middle" class="caption">v3: alias-free conv, no texture sticking</text>
|
||||
<text x="660" y="448" text-anchor="middle" class="caption">XL: conditional ImageNet</text>
|
||||
<text x="660" y="468" text-anchor="middle" class="caption">R3GAN (2024): minimal recipe, 20x fewer params</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 4.7 KiB |
@@ -0,0 +1,129 @@
|
||||
import math
|
||||
import random
|
||||
|
||||
|
||||
def leaky(x, a=0.2):
|
||||
return x if x > 0 else a * x
|
||||
|
||||
|
||||
def randn_matrix(rows, cols, rng, scale=0.3):
|
||||
return [[rng.gauss(0, scale) for _ in range(cols)] for _ in range(rows)]
|
||||
|
||||
|
||||
def matmul(W, x):
|
||||
return [sum(w * xi for w, xi in zip(row, x)) for row in W]
|
||||
|
||||
|
||||
def add(a, b):
|
||||
return [x + y for x, y in zip(a, b)]
|
||||
|
||||
|
||||
def mean_std(xs):
|
||||
m = sum(xs) / len(xs)
|
||||
v = sum((x - m) ** 2 for x in xs) / len(xs)
|
||||
return m, math.sqrt(v + 1e-8)
|
||||
|
||||
|
||||
def adain(features, scale, bias):
|
||||
m, s = mean_std(features)
|
||||
return [scale * (f - m) / s + bias for f in features]
|
||||
|
||||
|
||||
def mapping(z, layers):
|
||||
h = z
|
||||
for W, b in layers:
|
||||
pre = add(matmul(W, h), b)
|
||||
h = [leaky(x) for x in pre]
|
||||
return h
|
||||
|
||||
|
||||
def init_mapping(z_dim, w_dim, depth, rng):
|
||||
layers = []
|
||||
dims = [z_dim] + [w_dim] * depth
|
||||
for i in range(depth):
|
||||
layers.append((randn_matrix(dims[i + 1], dims[i], rng), [0.0] * dims[i + 1]))
|
||||
return layers
|
||||
|
||||
|
||||
def stylegan_forward(w, const, synth, noise_sigma, rng, adain_on=True):
|
||||
"""Very small 'synthesis' network: three resolution blocks on a 4-channel constant."""
|
||||
h = list(const)
|
||||
for i in range(3):
|
||||
W = synth[f"W{i}"]
|
||||
b = synth[f"b{i}"]
|
||||
pre = add(matmul(W, h), b)
|
||||
h = [leaky(x) for x in pre]
|
||||
if adain_on:
|
||||
scale = sum(synth[f"scale{i}"][j] * w[j] for j in range(len(w)))
|
||||
bias = sum(synth[f"bias{i}"][j] * w[j] for j in range(len(w)))
|
||||
h = adain(h, scale, bias)
|
||||
if noise_sigma > 0:
|
||||
h = [x + noise_sigma * rng.gauss(0, 1) for x in h]
|
||||
return h
|
||||
|
||||
|
||||
def init_synth(hidden, w_dim, rng):
|
||||
synth = {}
|
||||
for i in range(3):
|
||||
synth[f"W{i}"] = randn_matrix(hidden, hidden, rng)
|
||||
synth[f"b{i}"] = [0.0] * hidden
|
||||
synth[f"scale{i}"] = [rng.gauss(0, 0.3) for _ in range(w_dim)]
|
||||
synth[f"bias{i}"] = [rng.gauss(0, 0.3) for _ in range(w_dim)]
|
||||
return synth
|
||||
|
||||
|
||||
def main():
|
||||
rng = random.Random(3)
|
||||
z_dim, w_dim, hidden = 8, 8, 6
|
||||
|
||||
mapping_net = init_mapping(z_dim, w_dim, depth=4, rng=rng)
|
||||
synth = init_synth(hidden, w_dim, rng)
|
||||
const = [rng.gauss(0, 0.3) for _ in range(hidden)]
|
||||
|
||||
print("=== compare: style inputs via AdaIN vs no AdaIN ===")
|
||||
print("sample 5 random z, look at std of output under each mode")
|
||||
|
||||
for mode in [True, False]:
|
||||
outs = []
|
||||
for _ in range(5):
|
||||
z = [rng.gauss(0, 1) for _ in range(z_dim)]
|
||||
w = mapping(z, mapping_net)
|
||||
h = stylegan_forward(w, const, synth, 0.0, rng, adain_on=mode)
|
||||
outs.append(h)
|
||||
flat = [v for row in outs for v in row]
|
||||
m, s = mean_std(flat)
|
||||
label = "with AdaIN" if mode else "no AdaIN "
|
||||
print(f" {label}: mean {m:+.3f} std {s:.3f}")
|
||||
|
||||
print()
|
||||
print("=== truncation trick: sample many w, take mean, interpolate ===")
|
||||
ws = []
|
||||
for _ in range(200):
|
||||
z = [rng.gauss(0, 1) for _ in range(z_dim)]
|
||||
ws.append(mapping(z, mapping_net))
|
||||
w_bar = [sum(w[i] for w in ws) / len(ws) for i in range(w_dim)]
|
||||
|
||||
z_test = [rng.gauss(0, 1) for _ in range(z_dim)]
|
||||
w_test = mapping(z_test, mapping_net)
|
||||
|
||||
for psi in [0.0, 0.5, 0.7, 1.0]:
|
||||
w_psi = [w_bar[i] + psi * (w_test[i] - w_bar[i]) for i in range(w_dim)]
|
||||
h = stylegan_forward(w_psi, const, synth, 0.0, rng, adain_on=True)
|
||||
print(f" psi={psi:.1f}: output = {[f'{v:+.2f}' for v in h]}")
|
||||
|
||||
print()
|
||||
print("=== per-layer noise injection (pose fixed, stochastic detail changes) ===")
|
||||
z_fixed = [rng.gauss(0, 1) for _ in range(z_dim)]
|
||||
w_fixed = mapping(z_fixed, mapping_net)
|
||||
for seed in range(3):
|
||||
rng_local = random.Random(seed)
|
||||
h = stylegan_forward(w_fixed, const, synth, 0.1, rng_local, adain_on=True)
|
||||
print(f" seed {seed}: {[f'{v:+.2f}' for v in h]}")
|
||||
|
||||
print()
|
||||
print("notice: with the same w, outputs vary slightly with noise seed.")
|
||||
print(" that is the stochastic-detail vs global-style split.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,144 @@
|
||||
# StyleGAN
|
||||
|
||||
> Most generators stir `z` into every layer at the same time. StyleGAN split it apart: first map `z` to an intermediate `w`, then *inject* `w` at every resolution level through AdaIN. That single change untangled the latent space and made photorealistic faces a solved problem for seven years running.
|
||||
|
||||
**Type:** Build
|
||||
**Languages:** Python
|
||||
**Prerequisites:** Phase 8 · 03 (GANs), Phase 4 · 08 (Normalization), Phase 3 · 07 (CNNs)
|
||||
**Time:** ~45 minutes
|
||||
|
||||
## The Problem
|
||||
|
||||
A DCGAN maps `z` to an image through a stack of transposed convolutions. The problem: `z` controls everything — pose, lighting, identity, background — entangled together. Move along one axis of `z`, all four change. You cannot ask the model "same person, different pose" because the representation does not factor that way.
|
||||
|
||||
Karras et al. (2019, NVIDIA) proposed: stop feeding `z` directly into conv layers. Feed a constant `4×4×512` tensor as the network input. Learn an 8-layer MLP that maps `z ∈ Z → w ∈ W`. Inject `w` at every resolution via *adaptive instance normalization* (AdaIN): normalize each conv feature map, then scale and shift by affine projections of `w`. Add per-layer noise for stochastic detail (skin pores, hair strands).
|
||||
|
||||
The result: `W` has roughly orthogonal axes for "high-level style" (pose, identity) vs "fine style" (lighting, color). You can swap styles between two images by using image A's `w` for the low-resolution levels and image B's `w` for the high. This unlocked editing, cross-domain stylization, and the entire "StyleGAN-inversion" line of research.
|
||||
|
||||
## The Concept
|
||||
|
||||
|
||||
|
||||
**Mapping network.** `f: Z → W`, an 8-layer MLP. `Z = N(0, I)^512`. `W` is not forced to be Gaussian — it learns a data-adapted shape.
|
||||
|
||||
**Synthesis network.** Starts from a learned constant `4×4×512`. Each resolution block: `upsample → conv → AdaIN(w_i) → noise → conv → AdaIN(w_i) → noise`. Resolutions double: 4, 8, 16, 32, 64, 128, 256, 512, 1024.
|
||||
|
||||
**AdaIN.**
|
||||
|
||||
```
|
||||
AdaIN(x, y) = y_scale · (x - mean(x)) / std(x) + y_bias
|
||||
```
|
||||
|
||||
where `y_scale` and `y_bias` come from affine projections of `w`. Normalize per feature map, then restyle. "Style" here is the first- and second-order statistics of the feature map.
|
||||
|
||||
**Per-layer noise.** Single-channel Gaussian noise added to each feature map, scaled by a learned per-channel factor. Controls stochastic detail without affecting global structure.
|
||||
|
||||
**Truncation trick.** At inference, sample `z`, compute `w = mapping(z)`, then `w' = ŵ + ψ·(w - ŵ)` where `ŵ` is the mean `w` over many samples. `ψ < 1` trades diversity for quality. Almost every StyleGAN demo uses `ψ ≈ 0.7`.
|
||||
|
||||
## StyleGAN 1 → 2 → 3
|
||||
|
||||
| Version | Year | Innovation |
|
||||
|---------|------|------------|
|
||||
| StyleGAN | 2019 | Mapping network + AdaIN + noise + progressive growing. |
|
||||
| StyleGAN2 | 2020 | Weight demodulation replaces AdaIN (fixes droplet artifacts); skip/residual architecture; path-length regularization. |
|
||||
| StyleGAN3 | 2021 | Alias-free convolution + equivariant kernels; eliminates texture sticking to pixel grid. |
|
||||
| StyleGAN-XL | 2022 | Class-conditional, 1024², ImageNet. |
|
||||
| R3GAN | 2024 | Rebrands with stronger reg; closes gap to diffusion on FFHQ-1024 with 20x fewer params. |
|
||||
|
||||
In 2026 StyleGAN3 remains the default for (a) narrow-domain photorealism at high FPS, (b) few-shot domain adaptation (train on a new dataset with 100 images, freeze mapping), (c) inversion-based editing (find the `w` that reconstructs a real photo, then edit that `w`). For open-domain text-to-image, it is not the tool — diffusion is.
|
||||
|
||||
## Build It
|
||||
|
||||
`code/main.py` implements a toy "style-GAN lite" in 1-D: a mapping MLP, a synthesis function that takes a learned constant vector and modulates it with `w`-derived scale/bias, and per-layer noise. It shows that injecting `w` via affine-modulation matches or beats concatenating `z` into the generator's input.
|
||||
|
||||
### Step 1: mapping network
|
||||
|
||||
```python
|
||||
def mapping(z, M):
|
||||
h = z
|
||||
for i in range(num_layers):
|
||||
h = leaky_relu(add(matmul(M[f"W{i}"], h), M[f"b{i}"]))
|
||||
return h
|
||||
```
|
||||
|
||||
### Step 2: adaptive instance normalization
|
||||
|
||||
```python
|
||||
def adain(x, w_scale, w_bias):
|
||||
mu = mean(x)
|
||||
sd = std(x)
|
||||
x_norm = [(xi - mu) / (sd + 1e-8) for xi in x]
|
||||
return [w_scale * xi + w_bias for xi in x_norm]
|
||||
```
|
||||
|
||||
Per-feature-map scale and bias come from `w` via linear projection.
|
||||
|
||||
### Step 3: per-layer noise
|
||||
|
||||
```python
|
||||
def add_noise(x, sigma, rng):
|
||||
return [xi + sigma * rng.gauss(0, 1) for xi in x]
|
||||
```
|
||||
|
||||
Sigma per-channel is learnable.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Droplet artifacts.** StyleGAN 1 produced a blobby droplet in the feature maps because AdaIN zeroed out mean. StyleGAN 2's weight demodulation fixes it by scaling the convolution weights instead.
|
||||
- **Texture sticking.** StyleGAN 1 and 2 textures followed pixel coordinates, not object coordinates (visible when interpolating). StyleGAN 3's alias-free convolutions fix this with windowed sinc filters.
|
||||
- **Mode coverage.** Truncation `ψ < 0.7` looks clean but samples from a narrow cone; use `ψ = 1.0` if you need diversity.
|
||||
- **Inversion is lossy.** Inverting a real photo into `W` is usually done through optimization or an encoder (e4e, ReStyle, HyperStyle). Results drift over many iterations.
|
||||
|
||||
## Use It
|
||||
|
||||
| Use case | Approach |
|
||||
|----------|----------|
|
||||
| Photoreal human faces (anime, product, narrow) | StyleGAN3 FFHQ / custom fine-tune |
|
||||
| Face editing from a photo | e4e inversion + StyleSpace / InterFaceGAN directions |
|
||||
| Face swap / reenactment | StyleGAN + encoder + blending |
|
||||
| Avatar pipelines | StyleGAN3 w/ ADA for low-data fine-tune |
|
||||
| Domain adaptation from a few images | Freeze mapping network, fine-tune synthesis |
|
||||
| Multi-modal or text-conditioned generation | Don't — use diffusion |
|
||||
|
||||
For product-grade demos where the answer is "photo of a person's face", StyleGAN beats diffusion on inference cost (single forward pass, <10ms on a 4090) and sharpness for the same quality bar.
|
||||
|
||||
## Ship It
|
||||
|
||||
Save `outputs/skill-stylegan-inversion.md`. Skill takes a real photo and outputs: inversion method (e4e / ReStyle / HyperStyle), expected latent loss, editing budget (how far in `W` you can move before artifacts), and a list of known-good editing directions (age, expression, pose).
|
||||
|
||||
## Exercises
|
||||
|
||||
1. **Easy.** Run `code/main.py` with `adain_on=True` and `adain_on=False`. Compare the spread of outputs for a fixed latent vs perturbed latent.
|
||||
2. **Medium.** Implement mixing regularization: for a training batch, compute `w_a`, `w_b`, and apply `w_a` for the first half of synthesis and `w_b` for the second half. Does the decoder learn disentangled styles?
|
||||
3. **Hard.** Take a pretrained StyleGAN3 FFHQ model (ffhq-1024.pkl). Find the `w` direction that controls "smile" by training an SVM on labelled samples; report how far you can push before identity drifts.
|
||||
|
||||
## Key Terms
|
||||
|
||||
| Term | What people say | What it actually means |
|
||||
|------|-----------------|-----------------------|
|
||||
| Mapping network | "The MLP" | `f: Z → W`, 8 layers, decouples latent geometry from data statistics. |
|
||||
| W space | "The style space" | Output of the mapping network; roughly disentangled. |
|
||||
| AdaIN | "Adaptive instance norm" | Normalize feature map, then scale + shift by `w`-projection. |
|
||||
| Truncation trick | "Psi" | `w = mean + ψ·(w - mean)`, ψ<1 trades diversity for quality. |
|
||||
| Path-length regularization | "PL reg" | Penalizes large changes in image per unit change in `w`; makes `W` smoother. |
|
||||
| Weight demodulation | "The StyleGAN2 fix" | Normalize conv weights instead of activations; kills droplet artifacts. |
|
||||
| Alias-free | "StyleGAN3's trick" | Windowed sinc filters; eliminates texture sticking to the pixel grid. |
|
||||
| Inversion | "Find w for a real image" | Optimize or encode `x → w` so `G(w) ≈ x`. |
|
||||
|
||||
## Production note: why StyleGAN still ships in 2026
|
||||
|
||||
StyleGAN3 on a 4090 generates a 1024² FFHQ face in under 10 ms — `num_steps = 1`, no VAE decode, no cross-attention pass. In production terms this is the floor latency for any image generator. A 50-step SDXL + VAE-decode pipeline at the same resolution is ~3 seconds. That is a **300× gap**, and for narrow-domain products (avatar services, ID document pipelines, stock face generation) it wins on TCO.
|
||||
|
||||
Two operational consequences:
|
||||
|
||||
- **No scheduler, no batcher.** Static batch at the target occupancy is optimal. Continuous batching (essential for LLMs and diffusion) provides zero benefit because every request takes the same FLOPs.
|
||||
- **Truncation `ψ` is the safety knob.** `ψ < 0.7` samples from a narrow cone of the mapping network's range. This is the only lever the serving layer has over sample variance. Lower `ψ` at peak load, raise it for premium users.
|
||||
|
||||
## Further Reading
|
||||
|
||||
- [Karras et al. (2019). A Style-Based Generator Architecture for GANs](https://arxiv.org/abs/1812.04948) — StyleGAN.
|
||||
- [Karras et al. (2020). Analyzing and Improving the Image Quality of StyleGAN](https://arxiv.org/abs/1912.04958) — StyleGAN2.
|
||||
- [Karras et al. (2021). Alias-Free Generative Adversarial Networks](https://arxiv.org/abs/2106.12423) — StyleGAN3.
|
||||
- [Tov et al. (2021). Designing an Encoder for StyleGAN Image Manipulation](https://arxiv.org/abs/2102.02766) — e4e inversion.
|
||||
- [Sauer et al. (2022). StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets](https://arxiv.org/abs/2202.00273) — StyleGAN-XL.
|
||||
- [Huang et al. (2024). R3GAN: The GAN is dead; long live the GAN!](https://arxiv.org/abs/2501.05441) — modern minimal GAN recipe.
|
||||
@@ -0,0 +1,18 @@
|
||||
---
|
||||
name: stylegan-inversion
|
||||
description: Choose an inversion and editing pipeline for a pretrained StyleGAN over a real photo.
|
||||
version: 1.0.0
|
||||
phase: 8
|
||||
lesson: 05
|
||||
tags: [stylegan, inversion, editing]
|
||||
---
|
||||
|
||||
Given a real photo + pretrained StyleGAN checkpoint (FFHQ-1024, StyleGAN-XL, a custom fine-tune) and target edit (age, smile, pose, hair, identity preservation), output:
|
||||
|
||||
1. Inversion method. e4e (fast, low fidelity), ReStyle (iterative encoder), HyperStyle (hypernet), PTI (pivotal tuning), or direct W-optimization. One-sentence reason tied to fidelity vs speed.
|
||||
2. Target space. W, W+, or StyleSpace. Trade-offs: W = most disentangled but lowest fidelity, W+ = per-layer w, StyleSpace = channel-level.
|
||||
3. Editing direction. Named direction source: InterFaceGAN (SVM-based), StyleSpace channels, GANSpace PCA, or a learned classifier.
|
||||
4. Fidelity budget. LPIPS threshold before identity drift; rollback heuristic.
|
||||
5. Eval. ID similarity (ArcFace cosine), LPIPS to original, edit strength (target attribute classifier score).
|
||||
|
||||
Refuse any pipeline that edits directly in Z (entangled). Refuse large edits (>1.5 sigma in W) without identity checks. Flag requests that need open-domain editing (e.g. "make him a cartoon") - those require diffusion + IP-Adapter, not StyleGAN.
|
||||
@@ -0,0 +1,68 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 500" font-family="Georgia, 'Times New Roman', serif">
|
||||
<defs>
|
||||
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto">
|
||||
<path d="M0,0 L10,5 L0,10 z" fill="#1a1a1a"/>
|
||||
</marker>
|
||||
<style>
|
||||
.box { fill: #faf6ef; stroke: #1a1a1a; stroke-width: 1.5; }
|
||||
.hot { fill: #fff1d6; stroke: #c0392b; stroke-width: 1.5; }
|
||||
.cold { fill: #eaf4ff; stroke: #2c5f8c; stroke-width: 1.5; }
|
||||
.label { font-size: 14px; font-weight: 600; fill: #1a1a1a; }
|
||||
.content { font-size: 12px; fill: #333; }
|
||||
.mono { font-size: 12px; fill: #333; font-family: 'Menlo', monospace; }
|
||||
.caption { font-size: 11px; fill: #555; font-style: italic; }
|
||||
.title { font-size: 16px; font-weight: 700; fill: #1a1a1a; }
|
||||
</style>
|
||||
</defs>
|
||||
|
||||
<text x="450" y="28" text-anchor="middle" class="title">DDPM: one net predicts noise, reversal does the rest</text>
|
||||
|
||||
<!-- forward chain -->
|
||||
<text x="80" y="80" class="label">forward q: add noise</text>
|
||||
<rect x="40" y="95" width="80" height="50" class="box"/>
|
||||
<text x="80" y="125" text-anchor="middle" class="mono">x_0</text>
|
||||
<line x1="120" y1="120" x2="180" y2="120" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
<rect x="180" y="95" width="80" height="50" class="box"/>
|
||||
<text x="220" y="125" text-anchor="middle" class="mono">x_1</text>
|
||||
<line x1="260" y1="120" x2="310" y2="120" stroke="#1a1a1a" stroke-width="1.2" stroke-dasharray="3,3"/>
|
||||
<text x="335" y="125" class="caption">...</text>
|
||||
<line x1="360" y1="120" x2="410" y2="120" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
<rect x="410" y="95" width="80" height="50" class="box"/>
|
||||
<text x="450" y="125" text-anchor="middle" class="mono">x_t</text>
|
||||
<line x1="490" y1="120" x2="550" y2="120" stroke="#1a1a1a" stroke-width="1.2" stroke-dasharray="3,3"/>
|
||||
<text x="575" y="125" class="caption">...</text>
|
||||
<line x1="600" y1="120" x2="660" y2="120" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
<rect x="660" y="95" width="80" height="50" class="hot"/>
|
||||
<text x="700" y="120" text-anchor="middle" class="mono">x_T</text>
|
||||
<text x="700" y="138" text-anchor="middle" class="caption">~ N(0, I)</text>
|
||||
|
||||
<text x="450" y="170" text-anchor="middle" class="mono">q(x_t | x_0) = N(√(ᾱ_t) · x_0, (1 - ᾱ_t) I)</text>
|
||||
<text x="450" y="190" text-anchor="middle" class="caption">closed form: jump to any t in one shot</text>
|
||||
|
||||
<!-- reverse chain -->
|
||||
<text x="80" y="230" class="label">reverse p_θ: denoise</text>
|
||||
<rect x="660" y="245" width="80" height="50" class="hot"/>
|
||||
<text x="700" y="275" text-anchor="middle" class="mono">x_T</text>
|
||||
<line x1="660" y1="270" x2="600" y2="270" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
<rect x="520" y="245" width="80" height="50" class="box"/>
|
||||
<text x="560" y="275" text-anchor="middle" class="mono">x_{T-1}</text>
|
||||
<line x1="520" y1="270" x2="460" y2="270" stroke="#1a1a1a" stroke-width="1.2" stroke-dasharray="3,3"/>
|
||||
<text x="435" y="275" class="caption">...</text>
|
||||
<line x1="410" y1="270" x2="350" y2="270" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
<rect x="270" y="245" width="80" height="50" class="box"/>
|
||||
<text x="310" y="275" text-anchor="middle" class="mono">x_1</text>
|
||||
<line x1="270" y1="270" x2="210" y2="270" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
<rect x="130" y="245" width="80" height="50" class="cold"/>
|
||||
<text x="170" y="275" text-anchor="middle" class="mono">x_0 (sample)</text>
|
||||
|
||||
<text x="450" y="320" text-anchor="middle" class="mono">x_{t-1} = (1/√α_t) ( x_t - (β_t / √(1-ᾱ_t)) · ε_θ(x_t, t) ) + σ_t · z</text>
|
||||
<text x="450" y="340" text-anchor="middle" class="caption">subtract the predicted noise, rescale, re-inject a bit of fresh noise</text>
|
||||
|
||||
<!-- loss -->
|
||||
<rect x="40" y="370" width="820" height="60" class="hot"/>
|
||||
<text x="450" y="395" text-anchor="middle" class="label">training loss</text>
|
||||
<text x="450" y="420" text-anchor="middle" class="mono">L = E_{x_0, t, ε} ||ε - ε_θ(√ᾱ_t · x_0 + √(1-ᾱ_t) · ε, t)||²</text>
|
||||
|
||||
<text x="450" y="460" text-anchor="middle" class="caption">one net, one MSE loss, no minimax, no KL divergence in the training loop</text>
|
||||
<text x="450" y="480" text-anchor="middle" class="caption">scales unchanged to images, video, audio, 3D Gaussians</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 4.5 KiB |
@@ -0,0 +1,181 @@
|
||||
import math
|
||||
import random
|
||||
|
||||
|
||||
def sin_embed(t, T, dim=8):
|
||||
"""Sinusoidal timestep embedding."""
|
||||
out = []
|
||||
half = dim // 2
|
||||
for i in range(half):
|
||||
freq = 1.0 / (10000 ** (i / max(half - 1, 1)))
|
||||
out.append(math.sin(t * freq))
|
||||
out.append(math.cos(t * freq))
|
||||
return out[:dim]
|
||||
|
||||
|
||||
def tanh(v):
|
||||
return [math.tanh(x) for x in v]
|
||||
|
||||
|
||||
def tanh_grad(h):
|
||||
return [1 - x * x for x in h]
|
||||
|
||||
|
||||
def matmul(W, x):
|
||||
return [sum(w * xi for w, xi in zip(row, x)) for row in W]
|
||||
|
||||
|
||||
def add(a, b):
|
||||
return [x + y for x, y in zip(a, b)]
|
||||
|
||||
|
||||
def randn_matrix(rows, cols, rng, scale=0.3):
|
||||
return [[rng.gauss(0, scale) for _ in range(cols)] for _ in range(rows)]
|
||||
|
||||
|
||||
def init_net(x_dim, t_dim, hidden, rng):
|
||||
return {
|
||||
"W1": randn_matrix(hidden, x_dim + t_dim, rng),
|
||||
"b1": [0.0] * hidden,
|
||||
"W2": randn_matrix(hidden, hidden, rng),
|
||||
"b2": [0.0] * hidden,
|
||||
"W3": randn_matrix(x_dim, hidden, rng),
|
||||
"b3": [0.0] * x_dim,
|
||||
}
|
||||
|
||||
|
||||
def forward(x_t, t_embed, net):
|
||||
inp = x_t + t_embed
|
||||
pre1 = add(matmul(net["W1"], inp), net["b1"])
|
||||
h1 = tanh(pre1)
|
||||
pre2 = add(matmul(net["W2"], h1), net["b2"])
|
||||
h2 = tanh(pre2)
|
||||
eps_hat = add(matmul(net["W3"], h2), net["b3"])
|
||||
return eps_hat, {"inp": inp, "h1": h1, "h2": h2, "pre1": pre1, "pre2": pre2}
|
||||
|
||||
|
||||
def backward(target_eps, eps_hat, cache, net):
|
||||
grads = {k: None for k in net}
|
||||
for part in net:
|
||||
if isinstance(net[part][0], list):
|
||||
grads[part] = [[0.0] * len(net[part][0]) for _ in net[part]]
|
||||
else:
|
||||
grads[part] = [0.0] * len(net[part])
|
||||
|
||||
d_out = [2 * (a - b) for a, b in zip(eps_hat, target_eps)]
|
||||
for i in range(len(d_out)):
|
||||
grads["b3"][i] += d_out[i]
|
||||
for j in range(len(cache["h2"])):
|
||||
grads["W3"][i][j] += d_out[i] * cache["h2"][j]
|
||||
d_h2 = [sum(net["W3"][i][j] * d_out[i] for i in range(len(d_out)))
|
||||
for j in range(len(cache["h2"]))]
|
||||
d_pre2 = [d_h2[j] * tanh_grad(cache["h2"])[j] for j in range(len(cache["h2"]))]
|
||||
for j in range(len(cache["h2"])):
|
||||
grads["b2"][j] += d_pre2[j]
|
||||
for k in range(len(cache["h1"])):
|
||||
grads["W2"][j][k] += d_pre2[j] * cache["h1"][k]
|
||||
d_h1 = [sum(net["W2"][j][k] * d_pre2[j] for j in range(len(cache["h2"])))
|
||||
for k in range(len(cache["h1"]))]
|
||||
d_pre1 = [d_h1[j] * tanh_grad(cache["h1"])[j] for j in range(len(cache["h1"]))]
|
||||
for j in range(len(cache["h1"])):
|
||||
grads["b1"][j] += d_pre1[j]
|
||||
for k in range(len(cache["inp"])):
|
||||
grads["W1"][j][k] += d_pre1[j] * cache["inp"][k]
|
||||
return grads
|
||||
|
||||
|
||||
def apply_update(net, grads, lr):
|
||||
for k, v in net.items():
|
||||
if isinstance(v[0], list):
|
||||
for i in range(len(v)):
|
||||
for j in range(len(v[i])):
|
||||
v[i][j] -= lr * grads[k][i][j]
|
||||
else:
|
||||
for i in range(len(v)):
|
||||
v[i] -= lr * grads[k][i]
|
||||
|
||||
|
||||
def make_schedule(T):
|
||||
betas = [1e-4 + (0.02 - 1e-4) * t / (T - 1) for t in range(T)]
|
||||
alphas = [1 - b for b in betas]
|
||||
alpha_bars, cum = [], 1.0
|
||||
for a in alphas:
|
||||
cum *= a
|
||||
alpha_bars.append(cum)
|
||||
return betas, alphas, alpha_bars
|
||||
|
||||
|
||||
def sample_data(rng):
|
||||
return rng.gauss(-2.0, 0.4) if rng.random() < 0.5 else rng.gauss(2.0, 0.4)
|
||||
|
||||
|
||||
def train(net, alpha_bars, T, steps, lr, t_dim, rng):
|
||||
for step in range(steps):
|
||||
x0 = sample_data(rng)
|
||||
t = rng.randrange(T)
|
||||
a_bar = alpha_bars[t]
|
||||
eps = rng.gauss(0, 1)
|
||||
x_t = math.sqrt(a_bar) * x0 + math.sqrt(1 - a_bar) * eps
|
||||
t_emb = sin_embed(t, T, t_dim)
|
||||
eps_hat, cache = forward([x_t], t_emb, net)
|
||||
grads = backward([eps], eps_hat, cache, net)
|
||||
apply_update(net, grads, lr)
|
||||
if (step + 1) % 500 == 0:
|
||||
loss = (eps_hat[0] - eps) ** 2
|
||||
print(f"step {step+1:5d}: loss {loss:.4f}")
|
||||
|
||||
|
||||
def sample(net, alphas, alpha_bars, T, t_dim, rng):
|
||||
x = rng.gauss(0, 1)
|
||||
for t in range(T - 1, -1, -1):
|
||||
t_emb = sin_embed(t, T, t_dim)
|
||||
eps_hat, _ = forward([x], t_emb, net)
|
||||
beta_t = 1 - alphas[t]
|
||||
mean = (x - beta_t / math.sqrt(1 - alpha_bars[t]) * eps_hat[0]) / math.sqrt(alphas[t])
|
||||
if t > 0:
|
||||
x = mean + math.sqrt(beta_t) * rng.gauss(0, 1)
|
||||
else:
|
||||
x = mean
|
||||
return x
|
||||
|
||||
|
||||
def histogram(samples, lo=-5.0, hi=5.0, bins=30):
|
||||
width = (hi - lo) / bins
|
||||
counts = [0] * bins
|
||||
for s in samples:
|
||||
if lo <= s < hi:
|
||||
counts[int((s - lo) / width)] += 1
|
||||
peak = max(counts) or 1
|
||||
height = 8
|
||||
rows = []
|
||||
for r in range(height, 0, -1):
|
||||
thr = peak * r / height
|
||||
rows.append("".join("#" if c >= thr else " " for c in counts))
|
||||
rows.append("-" * bins)
|
||||
return "\n".join(rows)
|
||||
|
||||
|
||||
def main():
|
||||
rng = random.Random(13)
|
||||
T, t_dim, hidden = 40, 8, 24
|
||||
_, alphas, alpha_bars = make_schedule(T)
|
||||
net = init_net(1, t_dim, hidden, rng)
|
||||
|
||||
print("=== training DDPM on two-mode 1-D mixture ===")
|
||||
train(net, alpha_bars, T, steps=4000, lr=0.01, t_dim=t_dim, rng=rng)
|
||||
|
||||
print()
|
||||
print("=== sampling ===")
|
||||
samples = [sample(net, alphas, alpha_bars, T, t_dim, rng) for _ in range(500)]
|
||||
print(histogram(samples))
|
||||
m = sum(samples) / len(samples)
|
||||
pos = sum(1 for s in samples if s > 0)
|
||||
print(f"mean {m:+.3f}, modeA(<0)={500-pos}, modeB(>0)={pos}")
|
||||
|
||||
print()
|
||||
print("takeaway: trained noise predictor + reverse chain reproduces both modes.")
|
||||
print(" same loss function that scales to images, video, 3D.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,181 @@
|
||||
# Diffusion Models — DDPM from Scratch
|
||||
|
||||
> Ho, Jain, Abbeel (2020) gave the field a recipe it could not quit. Destroy the data with noise over a thousand small steps. Train one neural net to predict the noise. Reverse the process at inference. Today every mainstream image, video, 3D, and music model runs on this loop, possibly with flow matching or consistency tricks on top.
|
||||
|
||||
**Type:** Build
|
||||
**Languages:** Python
|
||||
**Prerequisites:** Phase 3 · 02 (Backprop), Phase 8 · 02 (VAE)
|
||||
**Time:** ~75 minutes
|
||||
|
||||
## The Problem
|
||||
|
||||
You want a sampler for `p_data(x)`. GANs play a minimax game that often diverges. VAEs produce blurry samples from a Gaussian decoder. What you really want is a training objective that is (a) a single stable loss (no saddle point, no minimax), (b) a lower bound on `log p(x)` (so you have likelihoods), and (c) samples that match SOTA quality.
|
||||
|
||||
Sohl-Dickstein et al. (2015) had a theoretical answer: define a Markov chain `q(x_t | x_{t-1})` that gradually adds Gaussian noise, and train a reverse chain `p_θ(x_{t-1} | x_t)` to denoise. Ho, Jain, Abbeel (2020) showed the loss could be simplified to one line — predict the noise — and cleaned up the math. In 2020 this was a curiosity. In 2021 it produced state-of-the-art samples. In 2022 it became Stable Diffusion. In 2026 it is the substrate.
|
||||
|
||||
## The Concept
|
||||
|
||||
|
||||
|
||||
**Forward process `q`.** Add Gaussian noise in `T` small steps. The closed form — the reason the math is tractable — is that the cumulative step is also Gaussian:
|
||||
|
||||
```
|
||||
q(x_t | x_0) = N( sqrt(α̅_t) · x_0, (1 - α̅_t) · I )
|
||||
```
|
||||
|
||||
where `α̅_t = ∏_{s=1..t} (1 - β_s)` for a schedule of `β_t`. Pick `β_t` from 1e-4 to 0.02 linearly over T=1000 steps and `x_T` is approximately `N(0, I)`.
|
||||
|
||||
**Reverse process `p_θ`.** Learn a neural net `ε_θ(x_t, t)` that predicts the noise that was added. Given `x_t`, denoise by:
|
||||
|
||||
```
|
||||
x_{t-1} = (1 / sqrt(α_t)) · ( x_t - (β_t / sqrt(1 - α̅_t)) · ε_θ(x_t, t) ) + σ_t · z
|
||||
```
|
||||
|
||||
where `σ_t` is either `sqrt(β_t)` or a learned variance. The expression is ugly but it is just algebra — solving for `x_{t-1}` given the posterior `q(x_{t-1} | x_t, x_0)` and substituting `x_0` with its noise-predicted estimate.
|
||||
|
||||
**Training loss.**
|
||||
|
||||
```
|
||||
L_simple = E_{x_0, t, ε} [ || ε - ε_θ( sqrt(α̅_t) · x_0 + sqrt(1 - α̅_t) · ε, t ) ||² ]
|
||||
```
|
||||
|
||||
Sample `x_0` from data, pick a random `t`, sample `ε ~ N(0, I)`, compute the noisy `x_t` in one shot via the closed form, and regress on the noise. One loss, no minimax, no KL, no reparameterization tricks.
|
||||
|
||||
**Sampling.** Start `x_T ~ N(0, I)`. Iterate the reverse step from `t = T` to `1`. Done.
|
||||
|
||||
## Why it works
|
||||
|
||||
Three intuitions:
|
||||
|
||||
1. **Denoising is easy; generating is hard.** At `t=T`, the data is pure noise — the net has to solve a trivial problem. At `t=0`, the net only has to clean up a few pixels. At intermediate `t`, the problem is hard but the net has many gradients flowing through the same weights from every noise level.
|
||||
|
||||
2. **Score matching in disguise.** Vincent (2011) proved that predicting the noise is equivalent to estimating `∇_x log q(x_t | x_0)`, the *score*. The reverse SDE uses this score to walk up the density gradient — a guided random walk toward high-probability regions.
|
||||
|
||||
3. **The ELBO reduces to simple MSE.** The full variational lower bound has a KL term per timestep. With DDPM's parameterization those KL terms simplify to MSE on noise prediction with specific coefficients; Ho dropped the coefficients (calling it "simple" loss) and quality *improved*.
|
||||
|
||||
## Build It
|
||||
|
||||
`code/main.py` implements a 1-D DDPM. Data is a two-mode mixture. The "net" is a tiny MLP that takes `(x_t, t)` and outputs predicted noise. Training is the one-line loss. Sampling iterates the reverse chain.
|
||||
|
||||
### Step 1: the forward schedule (closed form)
|
||||
|
||||
```python
|
||||
betas = [1e-4 + (0.02 - 1e-4) * t / (T - 1) for t in range(T)]
|
||||
alphas = [1 - b for b in betas]
|
||||
alpha_bars = []
|
||||
cum = 1.0
|
||||
for a in alphas:
|
||||
cum *= a
|
||||
alpha_bars.append(cum)
|
||||
```
|
||||
|
||||
### Step 2: sample `x_t` in one shot
|
||||
|
||||
```python
|
||||
def forward_sample(x0, t, alpha_bars, rng):
|
||||
a_bar = alpha_bars[t]
|
||||
eps = rng.gauss(0, 1)
|
||||
x_t = math.sqrt(a_bar) * x0 + math.sqrt(1 - a_bar) * eps
|
||||
return x_t, eps
|
||||
```
|
||||
|
||||
### Step 3: one training step
|
||||
|
||||
```python
|
||||
def train_step(x0, model, alpha_bars, rng):
|
||||
t = rng.randrange(T)
|
||||
x_t, eps = forward_sample(x0, t, alpha_bars, rng)
|
||||
eps_hat = model_forward(model, x_t, t)
|
||||
loss = (eps - eps_hat) ** 2
|
||||
return loss, gradient_step(model, ...)
|
||||
```
|
||||
|
||||
### Step 4: reverse sampling
|
||||
|
||||
```python
|
||||
def sample(model, alpha_bars, T, rng):
|
||||
x = rng.gauss(0, 1)
|
||||
for t in range(T - 1, -1, -1):
|
||||
eps_hat = model_forward(model, x, t)
|
||||
beta_t = 1 - alphas[t]
|
||||
x = (x - beta_t / math.sqrt(1 - alpha_bars[t]) * eps_hat) / math.sqrt(alphas[t])
|
||||
if t > 0:
|
||||
x += math.sqrt(beta_t) * rng.gauss(0, 1)
|
||||
return x
|
||||
```
|
||||
|
||||
For a 1-D problem with 40 timesteps and a 24-unit MLP, this learns the two-mode mixture in ~200 epochs.
|
||||
|
||||
## Time conditioning
|
||||
|
||||
The net needs to know which timestep it is denoising. Two standard options:
|
||||
|
||||
- **Sinusoidal embedding.** Like Transformer positional encoding. `embed(t) = [sin(t/ω_0), cos(t/ω_0), sin(t/ω_1), ...]`. Pass through an MLP, broadcast into the net.
|
||||
- **Film / group-norm conditioning.** Project embedding to per-channel scale/bias (FiLM) at each block.
|
||||
|
||||
Our toy code uses sinusoidal → concat. Production U-Nets use FiLM.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Schedule matters a lot.** Linear `β` is the DDPM default but cosine schedule (Nichol & Dhariwal, 2021) gives better FID for the same compute. Switch schedules if quality plateaus.
|
||||
- **Timestep embedding is fragile.** Passing raw `t` as a float works for toy 1-D but fails for images; always use a proper embedding.
|
||||
- **V-prediction vs ε-prediction.** For narrow regimes (very small or very large t), `ε` has poor signal-to-noise. V-prediction (`v = α·ε - σ·x`) is more stable; SDXL, SD3, and Flux use it.
|
||||
- **Classifier-free guidance.** At inference, compute both conditional and unconditional `ε`, then `ε_cfg = (1 + w) · ε_cond - w · ε_uncond` with `w ≈ 3-7`. Covered in Lesson 08.
|
||||
- **1000 steps is a lot.** Production uses DDIM (20-50 steps), DPM-Solver (10-20 steps), or distillation (1-4 steps). See Lesson 12.
|
||||
|
||||
## Use It
|
||||
|
||||
| Role | Typical stack in 2026 |
|
||||
|------|-----------------------|
|
||||
| Image pixel-space diffusion (small, toy) | DDPM + U-Net |
|
||||
| Image latent diffusion | VAE encoder + U-Net or DiT (Lesson 07) |
|
||||
| Video latent diffusion | Spatiotemporal DiT (Sora, Veo, WAN) |
|
||||
| Audio latent diffusion | Encodec + diffusion transformer |
|
||||
| Science (molecules, proteins, physics) | Equivariant diffusion (EDM, RFdiffusion, AlphaFold3) |
|
||||
|
||||
Diffusion is the universal generative backbone. Flow matching (Lesson 13) is the 2024-2026 competitor that usually wins on inference speed for the same quality.
|
||||
|
||||
## Ship It
|
||||
|
||||
Save `outputs/skill-diffusion-trainer.md`. Skill takes a dataset + compute budget and outputs: schedule (linear/cosine/sigmoid), prediction target (ε/v/x), number of steps, guidance scale, sampler family, and an eval protocol.
|
||||
|
||||
## Exercises
|
||||
|
||||
1. **Easy.** Change T from 40 to 10 in `code/main.py`. How does sample quality (visual histogram of outputs) degrade? At what T does the two-mode structure collapse?
|
||||
2. **Medium.** Switch from ε-prediction to v-prediction. Re-derive the reverse step. Compare final sample quality.
|
||||
3. **Hard.** Add classifier-free guidance. Condition on a class label `c ∈ {0, 1}`, drop it 10% of the time during training, and at sampling time use `ε = (1+w)·ε_cond - w·ε_uncond`. Measure the conditional-mode-hit rate at `w = 0, 1, 3, 7`.
|
||||
|
||||
## Key Terms
|
||||
|
||||
| Term | What people say | What it actually means |
|
||||
|------|-----------------|-----------------------|
|
||||
| Forward process | "Adding noise" | Fixed Markov chain `q(x_t | x_{t-1})` that destroys the data. |
|
||||
| Reverse process | "Denoising" | Learned chain `p_θ(x_{t-1} | x_t)` that reconstructs the data. |
|
||||
| β schedule | "The noise ladder" | Per-step variance; linear, cosine, or sigmoid. |
|
||||
| α̅ | "Alpha bar" | Cumulative product `∏(1 - β)`; gives closed-form `x_t` from `x_0`. |
|
||||
| Simple loss | "MSE on noise" | `||ε - ε_θ(x_t, t)||²`; all variational derivations collapse to this. |
|
||||
| ε-prediction | "Predict noise" | Output is the noise added; standard DDPM. |
|
||||
| V-prediction | "Predict velocity" | Output is `α·ε - σ·x`; better conditioning across t. |
|
||||
| DDPM | "The paper" | Ho et al. 2020; linear β, 1000 steps, U-Net. |
|
||||
| DDIM | "Deterministic sampler" | Non-Markov sampler, 20-50 steps, same training objective. |
|
||||
| Classifier-free guidance | "CFG" | Mix conditional and unconditional noise predictions to amplify conditioning. |
|
||||
|
||||
## Production note: diffusion inference is a step-count problem
|
||||
|
||||
The DDPM paper runs T=1000 reverse steps. Nobody ships that in production. Every real inference stack picks one of three strategies — and each maps cleanly to production framing of "where is the latency coming from":
|
||||
|
||||
1. **Faster sampler, same model.** DDIM (20-50 steps), DPM-Solver++ (10-20), UniPC (8-16). Drop-in replacement of the reverse loop; the trained `ε_θ` weights are untouched. Cuts latency 20-50×.
|
||||
2. **Distillation.** Train a student to match the teacher in fewer steps: Progressive Distillation (2 → 1), Consistency Models (arbitrary → 1-4), LCM, SDXL-Turbo, SD3-Turbo. Cuts latency another 5-10×, requires retraining.
|
||||
3. **Caching and compilation.** `torch.compile(unet, mode="reduce-overhead")`, TensorRT-LLM's diffusion backends, `xformers`/SDPA attention, bf16 weights. Cuts per-step latency ~2×. Stacks with (1) and (2).
|
||||
|
||||
For a production diffusion server the budget conversation is the same as production literature describes for LLMs: latency is `num_steps × step_cost + VAE_decode`, throughput is `batch_size × (num_steps × step_cost)^-1`. TTFT is small (one step); TPOT-equivalent is the full response time because image generation is "all-at-once" from the user's perspective.
|
||||
|
||||
## Further Reading
|
||||
|
||||
- [Sohl-Dickstein et al. (2015). Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://arxiv.org/abs/1503.03585) — the diffusion paper, ahead of its time.
|
||||
- [Ho, Jain, Abbeel (2020). Denoising Diffusion Probabilistic Models](https://arxiv.org/abs/2006.11239) — DDPM.
|
||||
- [Song, Meng, Ermon (2021). Denoising Diffusion Implicit Models](https://arxiv.org/abs/2010.02502) — DDIM, fewer steps.
|
||||
- [Nichol & Dhariwal (2021). Improved DDPM](https://arxiv.org/abs/2102.09672) — cosine schedule, learned variance.
|
||||
- [Dhariwal & Nichol (2021). Diffusion Models Beat GANs on Image Synthesis](https://arxiv.org/abs/2105.05233) — classifier guidance.
|
||||
- [Ho & Salimans (2022). Classifier-Free Diffusion Guidance](https://arxiv.org/abs/2207.12598) — CFG.
|
||||
- [Karras et al. (2022). Elucidating the Design Space of Diffusion-Based Generative Models (EDM)](https://arxiv.org/abs/2206.00364) — unified notation, cleanest recipe.
|
||||
+18
@@ -0,0 +1,18 @@
|
||||
---
|
||||
name: diffusion-trainer
|
||||
description: Configure a diffusion training run: schedule, prediction target, sampler, and eval plan.
|
||||
version: 1.0.0
|
||||
phase: 8
|
||||
lesson: 06
|
||||
tags: [diffusion, ddpm, training]
|
||||
---
|
||||
|
||||
Given a dataset profile (modality, resolution, dataset size), compute budget (GPU hours, VRAM floor), and quality bar (FID target or downstream use), output:
|
||||
|
||||
1. Schedule. Linear, cosine (Nichol), or sigmoid. Number of steps T (1000 for DDPM baseline; 256 for faster variants).
|
||||
2. Prediction target. epsilon, v-prediction, or x_0. Reason tied to resolution and signal-to-noise across the schedule.
|
||||
3. Architecture. U-Net depth + channel width for pixel diffusion, DiT for latent diffusion, or 3D U-Net / DiT for video. Include time embedding scheme (sinusoidal + MLP, FiLM, or AdaLN).
|
||||
4. Sampler. DDIM (20-50 steps), DPM-Solver++ (10-20), Euler-A (creative), or distilled 1-4-step. Include guidance scale (CFG w) recommendation.
|
||||
5. Eval plan. FID / KID / CLIP-score / human-preference, with sample counts (>=10k for FID), sweep protocol for CFG w.
|
||||
|
||||
Refuse to recommend training pixel-space diffusion at >=256x256 when latent diffusion achieves the same quality at 1/16th the FLOPs. Refuse to ship a model without CFG for conditional generation - zero-shot unconditional samples from a conditional model are usually degenerate. Flag any schedule with beta_T > 0.1 as likely to produce saturated or unstable training.
|
||||
+84
@@ -0,0 +1,84 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 520" font-family="Georgia, 'Times New Roman', serif">
|
||||
<defs>
|
||||
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto">
|
||||
<path d="M0,0 L10,5 L0,10 z" fill="#1a1a1a"/>
|
||||
</marker>
|
||||
<style>
|
||||
.box { fill: #faf6ef; stroke: #1a1a1a; stroke-width: 1.5; }
|
||||
.hot { fill: #fff1d6; stroke: #c0392b; stroke-width: 1.5; }
|
||||
.cold { fill: #eaf4ff; stroke: #2c5f8c; stroke-width: 1.5; }
|
||||
.label { font-size: 14px; font-weight: 600; fill: #1a1a1a; }
|
||||
.content { font-size: 12px; fill: #333; }
|
||||
.mono { font-size: 12px; fill: #333; font-family: 'Menlo', monospace; }
|
||||
.caption { font-size: 11px; fill: #555; font-style: italic; }
|
||||
.title { font-size: 16px; font-weight: 700; fill: #1a1a1a; }
|
||||
</style>
|
||||
</defs>
|
||||
|
||||
<text x="450" y="28" text-anchor="middle" class="title">latent diffusion = VAE + diffusion, separately trained</text>
|
||||
|
||||
<!-- stage 1 -->
|
||||
<rect x="30" y="60" width="840" height="130" class="box"/>
|
||||
<text x="450" y="82" text-anchor="middle" class="label">stage 1: train VAE (encoder + decoder), freeze</text>
|
||||
|
||||
<rect x="60" y="100" width="100" height="70" class="box"/>
|
||||
<text x="110" y="130" text-anchor="middle" class="content">image x</text>
|
||||
<text x="110" y="148" text-anchor="middle" class="caption">512 × 512 × 3</text>
|
||||
|
||||
<line x1="160" y1="135" x2="200" y2="135" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
|
||||
<rect x="200" y="100" width="120" height="70" class="cold"/>
|
||||
<text x="260" y="130" text-anchor="middle" class="content">encoder E</text>
|
||||
|
||||
<line x1="320" y1="135" x2="360" y2="135" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
|
||||
<rect x="360" y="100" width="120" height="70" class="hot"/>
|
||||
<text x="420" y="128" text-anchor="middle" class="content">latent z</text>
|
||||
<text x="420" y="146" text-anchor="middle" class="caption">64 × 64 × 4 (1/16 of pixels)</text>
|
||||
|
||||
<line x1="480" y1="135" x2="520" y2="135" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
|
||||
<rect x="520" y="100" width="120" height="70" class="cold"/>
|
||||
<text x="580" y="130" text-anchor="middle" class="content">decoder D</text>
|
||||
|
||||
<line x1="640" y1="135" x2="680" y2="135" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
|
||||
<rect x="680" y="100" width="150" height="70" class="box"/>
|
||||
<text x="755" y="130" text-anchor="middle" class="content">x̂ (reconstruction)</text>
|
||||
<text x="755" y="148" text-anchor="middle" class="caption">L1 + LPIPS + GAN</text>
|
||||
|
||||
<!-- stage 2 -->
|
||||
<rect x="30" y="210" width="840" height="190" class="box"/>
|
||||
<text x="450" y="232" text-anchor="middle" class="label">stage 2: train diffusion on z-space</text>
|
||||
|
||||
<rect x="80" y="250" width="110" height="80" class="hot"/>
|
||||
<text x="135" y="280" text-anchor="middle" class="mono">z_T ~ N(0, I)</text>
|
||||
|
||||
<line x1="190" y1="290" x2="240" y2="290" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
|
||||
<rect x="240" y="250" width="300" height="80" class="cold"/>
|
||||
<text x="390" y="278" text-anchor="middle" class="label">U-Net / DiT</text>
|
||||
<text x="390" y="298" text-anchor="middle" class="mono">ε_θ(z_t, t, text_embed)</text>
|
||||
<text x="390" y="318" text-anchor="middle" class="caption">iterate T->0</text>
|
||||
|
||||
<line x1="540" y1="290" x2="590" y2="290" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
|
||||
<rect x="590" y="250" width="120" height="80" class="hot"/>
|
||||
<text x="650" y="290" text-anchor="middle" class="mono">z_0</text>
|
||||
|
||||
<line x1="710" y1="290" x2="750" y2="290" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
|
||||
<rect x="750" y="250" width="110" height="80" class="box"/>
|
||||
<text x="805" y="285" text-anchor="middle" class="content">D(z_0)</text>
|
||||
<text x="805" y="303" text-anchor="middle" class="caption">decoded image</text>
|
||||
|
||||
<!-- text conditioning -->
|
||||
<rect x="80" y="345" width="460" height="45" class="box"/>
|
||||
<text x="310" y="370" text-anchor="middle" class="caption">text encoder (CLIP / T5) -> cross-attention in each U-Net block</text>
|
||||
|
||||
<!-- loss -->
|
||||
<rect x="30" y="415" width="840" height="80" class="hot"/>
|
||||
<text x="450" y="438" text-anchor="middle" class="label">same loss as pixel-space DDPM: L = E || ε - ε_θ(z_t, t, c) ||²</text>
|
||||
<text x="450" y="462" text-anchor="middle" class="caption">~64x fewer FLOPs than pixel diffusion for the same quality</text>
|
||||
<text x="450" y="482" text-anchor="middle" class="caption">CFG: ε_cfg = (1+w) · ε_cond - w · ε_uncond (w ≈ 3-7)</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 4.6 KiB |
@@ -0,0 +1,184 @@
|
||||
import math
|
||||
import random
|
||||
|
||||
|
||||
def sin_embed(t, T, dim=8):
|
||||
out = []
|
||||
half = dim // 2
|
||||
for i in range(half):
|
||||
freq = 1.0 / (10000 ** (i / max(half - 1, 1)))
|
||||
out.append(math.sin(t * freq))
|
||||
out.append(math.cos(t * freq))
|
||||
return out[:dim]
|
||||
|
||||
|
||||
def one_hot(c, num):
|
||||
v = [0.0] * num
|
||||
v[c] = 1.0
|
||||
return v
|
||||
|
||||
|
||||
def tanh(v):
|
||||
return [math.tanh(x) for x in v]
|
||||
|
||||
|
||||
def tanh_grad(h):
|
||||
return [1 - x * x for x in h]
|
||||
|
||||
|
||||
def matmul(W, x):
|
||||
return [sum(w * xi for w, xi in zip(row, x)) for row in W]
|
||||
|
||||
|
||||
def add(a, b):
|
||||
return [x + y for x, y in zip(a, b)]
|
||||
|
||||
|
||||
def randn_matrix(rows, cols, rng, scale=0.3):
|
||||
return [[rng.gauss(0, scale) for _ in range(cols)] for _ in range(rows)]
|
||||
|
||||
|
||||
NULL_CLASS = 2
|
||||
|
||||
|
||||
def init_net(x_dim, t_dim, c_dim, hidden, rng):
|
||||
return {
|
||||
"W1": randn_matrix(hidden, x_dim + t_dim + c_dim, rng),
|
||||
"b1": [0.0] * hidden,
|
||||
"W2": randn_matrix(hidden, hidden, rng),
|
||||
"b2": [0.0] * hidden,
|
||||
"W3": randn_matrix(x_dim, hidden, rng),
|
||||
"b3": [0.0] * x_dim,
|
||||
}
|
||||
|
||||
|
||||
def forward(x_t, t_emb, c_emb, net):
|
||||
inp = x_t + t_emb + c_emb
|
||||
pre1 = add(matmul(net["W1"], inp), net["b1"])
|
||||
h1 = tanh(pre1)
|
||||
pre2 = add(matmul(net["W2"], h1), net["b2"])
|
||||
h2 = tanh(pre2)
|
||||
out = add(matmul(net["W3"], h2), net["b3"])
|
||||
return out, {"inp": inp, "h1": h1, "h2": h2}
|
||||
|
||||
|
||||
def backward(target, out, cache, net):
|
||||
grads = {k: None for k in net}
|
||||
for part in net:
|
||||
if isinstance(net[part][0], list):
|
||||
grads[part] = [[0.0] * len(net[part][0]) for _ in net[part]]
|
||||
else:
|
||||
grads[part] = [0.0] * len(net[part])
|
||||
d_out = [2 * (a - b) for a, b in zip(out, target)]
|
||||
for i in range(len(d_out)):
|
||||
grads["b3"][i] += d_out[i]
|
||||
for j in range(len(cache["h2"])):
|
||||
grads["W3"][i][j] += d_out[i] * cache["h2"][j]
|
||||
d_h2 = [sum(net["W3"][i][j] * d_out[i] for i in range(len(d_out)))
|
||||
for j in range(len(cache["h2"]))]
|
||||
d_pre2 = [d_h2[j] * tanh_grad(cache["h2"])[j] for j in range(len(cache["h2"]))]
|
||||
for j in range(len(cache["h2"])):
|
||||
grads["b2"][j] += d_pre2[j]
|
||||
for k in range(len(cache["h1"])):
|
||||
grads["W2"][j][k] += d_pre2[j] * cache["h1"][k]
|
||||
d_h1 = [sum(net["W2"][j][k] * d_pre2[j] for j in range(len(cache["h2"])))
|
||||
for k in range(len(cache["h1"]))]
|
||||
d_pre1 = [d_h1[j] * tanh_grad(cache["h1"])[j] for j in range(len(cache["h1"]))]
|
||||
for j in range(len(cache["h1"])):
|
||||
grads["b1"][j] += d_pre1[j]
|
||||
for k in range(len(cache["inp"])):
|
||||
grads["W1"][j][k] += d_pre1[j] * cache["inp"][k]
|
||||
return grads
|
||||
|
||||
|
||||
def apply(net, grads, lr):
|
||||
for k, v in net.items():
|
||||
if isinstance(v[0], list):
|
||||
for i in range(len(v)):
|
||||
for j in range(len(v[i])):
|
||||
v[i][j] -= lr * grads[k][i][j]
|
||||
else:
|
||||
for i in range(len(v)):
|
||||
v[i] -= lr * grads[k][i]
|
||||
|
||||
|
||||
def make_schedule(T):
|
||||
betas = [1e-4 + (0.02 - 1e-4) * t / (T - 1) for t in range(T)]
|
||||
alphas = [1 - b for b in betas]
|
||||
bars, cum = [], 1.0
|
||||
for a in alphas:
|
||||
cum *= a
|
||||
bars.append(cum)
|
||||
return alphas, bars
|
||||
|
||||
|
||||
def encode(x):
|
||||
return x * 0.5
|
||||
|
||||
|
||||
def decode(z):
|
||||
return z * 2.0
|
||||
|
||||
|
||||
def sample_data(rng):
|
||||
c = rng.randrange(2)
|
||||
x = rng.gauss(-2.0 if c == 0 else 2.0, 0.4)
|
||||
return x, c
|
||||
|
||||
|
||||
def main():
|
||||
rng = random.Random(11)
|
||||
T, t_dim, hidden = 40, 8, 32
|
||||
num_classes_inc_null = 3
|
||||
alphas, alpha_bars = make_schedule(T)
|
||||
net = init_net(1, t_dim, num_classes_inc_null, hidden, rng)
|
||||
|
||||
print("=== training class-conditional latent diffusion with CFG dropout ===")
|
||||
for step in range(4000):
|
||||
x0, c = sample_data(rng)
|
||||
z0 = encode(x0)
|
||||
t = rng.randrange(T)
|
||||
eps = rng.gauss(0, 1)
|
||||
z_t = math.sqrt(alpha_bars[t]) * z0 + math.sqrt(1 - alpha_bars[t]) * eps
|
||||
use_c = NULL_CLASS if rng.random() < 0.1 else c
|
||||
c_emb = one_hot(use_c, num_classes_inc_null)
|
||||
t_emb = sin_embed(t, T, t_dim)
|
||||
out, cache = forward([z_t], t_emb, c_emb, net)
|
||||
grads = backward([eps], out, cache, net)
|
||||
apply(net, grads, 0.01)
|
||||
if (step + 1) % 1000 == 0:
|
||||
print(f" step {step+1:5d}")
|
||||
|
||||
def sample(c_target, w):
|
||||
z = rng.gauss(0, 1)
|
||||
for t in range(T - 1, -1, -1):
|
||||
t_emb = sin_embed(t, T, t_dim)
|
||||
eps_c, _ = forward([z], t_emb, one_hot(c_target, num_classes_inc_null), net)
|
||||
eps_u, _ = forward([z], t_emb, one_hot(NULL_CLASS, num_classes_inc_null), net)
|
||||
eps_cfg = (1 + w) * eps_c[0] - w * eps_u[0]
|
||||
beta_t = 1 - alphas[t]
|
||||
mean = (z - beta_t / math.sqrt(1 - alpha_bars[t]) * eps_cfg) / math.sqrt(alphas[t])
|
||||
if t > 0:
|
||||
z = mean + math.sqrt(beta_t) * rng.gauss(0, 1)
|
||||
else:
|
||||
z = mean
|
||||
return decode(z)
|
||||
|
||||
print()
|
||||
print("=== CFG sweep: per-class mean over 200 samples ===")
|
||||
for w in [0.0, 1.0, 3.0, 7.0]:
|
||||
samples = {0: [], 1: []}
|
||||
for _ in range(200):
|
||||
c = rng.randrange(2)
|
||||
samples[c].append(sample(c, w))
|
||||
m0 = sum(samples[0]) / len(samples[0])
|
||||
m1 = sum(samples[1]) / len(samples[1])
|
||||
print(f" w={w:.1f}: class 0 mean {m0:+.2f} class 1 mean {m1:+.2f}")
|
||||
|
||||
print()
|
||||
print("takeaway: same DDPM loss, just running on encoded z.")
|
||||
print(" CFG scales conditioning strength without retraining.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,145 @@
|
||||
# Latent Diffusion & Stable Diffusion
|
||||
|
||||
> Pixel-space diffusion on 512×512 images is a computational war crime. Rombach et al. (2022) noticed that you do not need all 786k dimensions to generate an image — you need enough to capture semantic structure, and a separate decoder for the rest. Run diffusion inside a VAE's latent space. That one idea is Stable Diffusion.
|
||||
|
||||
**Type:** Build
|
||||
**Languages:** Python
|
||||
**Prerequisites:** Phase 8 · 02 (VAE), Phase 8 · 06 (DDPM), Phase 7 · 09 (ViT)
|
||||
**Time:** ~75 minutes
|
||||
|
||||
## The Problem
|
||||
|
||||
Pixel-space diffusion at 512² means the U-Net runs on tensors of shape `[B, 3, 512, 512]`. Each sampling step is ~100 GFLOPS for a 500M-param U-Net. Fifty steps is 5 TFLOPS per image. Train on a billion images and the compute bill is absurd.
|
||||
|
||||
Most of those FLOPs go to pushing perceptually unimportant details through the net — the high-frequency texture that a lossy VAE could compress away. Rombach's idea: train a VAE once (the *first stage*), freeze it, and run diffusion entirely in the 4-channel 64×64 latent space (the *second stage*). Same U-Net. 1/16th the pixels. ~64x fewer FLOPs for comparable quality.
|
||||
|
||||
This is the Stable Diffusion recipe. SD 1.x / 2.x used an 860M U-Net over `64×64×4` latents, SDXL used a 2.6B U-Net over `128×128×4`, SD3 swapped the U-Net for a Diffusion Transformer (DiT) with flow matching. Flux.1-dev (Black Forest Labs, 2024) ships a 12B-param DiT-MMDiT. All run on the same two-stage substrate.
|
||||
|
||||
## The Concept
|
||||
|
||||
|
||||
|
||||
**Two stages, separately trained.**
|
||||
|
||||
1. **Stage 1 — VAE.** Encoder `E(x) → z`, decoder `D(z) → x`. Target compression: 8× downsample in each spatial axis + adjust channels so total latent size is ~1/16th of pixel count. Loss = reconstruction (L1 + LPIPS perceptual) + KL (small weight so `z` isn't forced too Gaussian, because we do not need exact sampling from `z`). Often trained with an adversarial loss so decoded images are sharp.
|
||||
|
||||
2. **Stage 2 — diffusion on `z`.** Treat `z = E(x_real)` as the data. Train a U-Net (or DiT) to denoise `z_t`. At inference: sample `z_0` via diffusion, then `x = D(z_0)`.
|
||||
|
||||
**Text conditioning.** Two additional components. A frozen text encoder (CLIP-L for SD 1.x, CLIP-L+OpenCLIP-G for SD 2/XL, T5-XXL for SD3 and Flux). A cross-attention injection: every U-Net block takes `[Q = image features, K = V = text tokens]` and mixes them in. The tokens are the only way text influences the image.
|
||||
|
||||
**The loss function is identical to Lesson 06.** Same DDPM / flow matching MSE on noise. You just swap the data domain.
|
||||
|
||||
## Architecture variants
|
||||
|
||||
| Model | Year | Backbone | Latent shape | Text encoder | Params |
|
||||
|-------|------|----------|--------------|--------------|--------|
|
||||
| SD 1.5 | 2022 | U-Net | 64×64×4 | CLIP-L (77 tokens) | 860M |
|
||||
| SD 2.1 | 2022 | U-Net | 64×64×4 | OpenCLIP-H | 865M |
|
||||
| SDXL | 2023 | U-Net + refiner | 128×128×4 | CLIP-L + OpenCLIP-G | 2.6B + 6.6B |
|
||||
| SDXL-Turbo | 2023 | Distilled | 128×128×4 | same | 1-4 step sampling |
|
||||
| SD3 | 2024 | MMDiT (multimodal DiT) | 128×128×16 | T5-XXL + CLIP-L + CLIP-G | 2B / 8B |
|
||||
| Flux.1-dev | 2024 | MMDiT | 128×128×16 | T5-XXL + CLIP-L | 12B |
|
||||
| Flux.1-schnell | 2024 | MMDiT distilled | 128×128×16 | T5-XXL + CLIP-L | 12B, 1-4 step |
|
||||
|
||||
The trend: replace U-Net with DiT (transformer over latent patches), scale the text encoder (T5 beats CLIP for prompt adherence), increase latent channels (4 → 16 gives more detail headroom).
|
||||
|
||||
## Build It
|
||||
|
||||
`code/main.py` stacks a toy 1-D "VAE" (identity encoder + decoder, for demonstration; a real VAE would be a conv net) on top of the DDPM from Lesson 06 and adds class conditioning with classifier-free guidance. It shows that the same diffusion loss works whether you run on raw 1-D values or on encoded values — the key insight.
|
||||
|
||||
### Step 1: encoder/decoder
|
||||
|
||||
```python
|
||||
def encode(x): return x * 0.5 # toy "compression" to smaller scale
|
||||
def decode(z): return z * 2.0
|
||||
```
|
||||
|
||||
A real VAE has trained weights. For pedagogy, this linear map is enough to show that diffusion operates on `z` without caring about the original data space.
|
||||
|
||||
### Step 2: diffusion in `z`-space
|
||||
|
||||
Same DDPM as Lesson 06. The data the net sees is `z = E(x)`. After sampling `z_0`, decode with `D(z_0)`.
|
||||
|
||||
### Step 3: classifier-free guidance
|
||||
|
||||
During training, drop the class label 10% of the time (replace with a null token). At inference, compute both `ε_cond` and `ε_uncond`, then:
|
||||
|
||||
```python
|
||||
eps_cfg = (1 + w) * eps_cond - w * eps_uncond
|
||||
```
|
||||
|
||||
`w = 0` = no guidance (full diversity), `w = 3` = default, `w = 7+` = saturated / over-sharp.
|
||||
|
||||
### Step 4: text conditioning (concept, not code)
|
||||
|
||||
Replace the class label with a frozen text encoder output. Feed the text embedding to the U-Net via cross-attention:
|
||||
|
||||
```python
|
||||
h = h + CrossAttention(Q=h, K=text_embed, V=text_embed)
|
||||
```
|
||||
|
||||
This is the only substantive difference between a class-conditional diffusion model and Stable Diffusion.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **VAE-scale mismatch.** SD 1.x VAEs have a scaling constant (`scaling_factor ≈ 0.18215`) applied after encoding. Forgetting this makes the U-Net train on latents with wildly wrong variance. Every checkpoint ships one.
|
||||
- **Text encoder silently wrong.** SD3 needs T5-XXL with >=128 tokens, and the fallback to CLIP-only is lossy. Always check `use_t5=True` or prompt fidelity craters.
|
||||
- **Mixing latent spaces.** SDXL, SD3, Flux all use different VAEs. A LoRA trained on SDXL latents will not work on SD3. Hugging Face diffusers 0.30+ refuses to load mismatched checkpoints.
|
||||
- **CFG too high.** `w > 10` produces saturated, oily images and over-fits the prompt at the cost of diversity. The sweet spot is `w = 3-7`.
|
||||
- **Negative prompts leaking.** Empty negative prompt becomes the null token; a filled negative prompt becomes the `ε_uncond`. These are not the same; some pipelines silently default to the null.
|
||||
|
||||
## Use It
|
||||
|
||||
Production stacks in 2026:
|
||||
|
||||
| Target | Recommended backbone |
|
||||
|--------|----------------------|
|
||||
| Narrow domain, paired data, training a model from scratch | SDXL fine-tune (LoRA / full) — fastest to ship |
|
||||
| Open-domain text-to-image, open weights | Flux.1-dev (12B, Apache / non-commercial) or SD3.5-Large |
|
||||
| Fastest inference, open weights | Flux.1-schnell (1-4 step, Apache) or SDXL-Lightning |
|
||||
| Best prompt adherence, hosted | GPT-Image / DALL-E 3 (still), Midjourney v7, Imagen 4 |
|
||||
| Edit workflows | Flux.1-Kontext (Dec 2024) — natively accepts image + text |
|
||||
| Research, baseline | SD 1.5 — ancient but well-studied |
|
||||
|
||||
## Ship It
|
||||
|
||||
Save `outputs/skill-sd-prompter.md`. Skill takes a text prompt + target style and outputs: model + checkpoint, CFG scale, sampler, negative prompt, resolution, optional ControlNet/IP-Adapter combo, and a per-step QA checklist.
|
||||
|
||||
## Exercises
|
||||
|
||||
1. **Easy.** Run `code/main.py` with guidance `w ∈ {0, 1, 3, 7, 15}`. Record mean sample by class. At what `w` do the class means diverge past the real data means?
|
||||
2. **Medium.** Swap the toy linear encoder for a tanh-MLP encoder/decoder pair with a reconstruction loss. Retrain diffusion on the new latents. Does sample quality change?
|
||||
3. **Hard.** Set up a real Stable Diffusion inference with diffusers: load `sdxl-base`, run 30 Euler steps with CFG=7, time it. Now switch to `sdxl-turbo` with 4 steps and CFG=0. Same subject, different quality — describe what changed and why.
|
||||
|
||||
## Key Terms
|
||||
|
||||
| Term | What people say | What it actually means |
|
||||
|------|-----------------|-----------------------|
|
||||
| First stage | "The VAE" | Trained encoder/decoder pair; compresses 512² to 64². |
|
||||
| Second stage | "The U-Net" | Diffusion model over the latent space. |
|
||||
| CFG | "Guidance scale" | `(1+w)·ε_cond - w·ε_uncond`; tunes conditioning strength. |
|
||||
| Null token | "Empty prompt embed" | Unconditional embed used for `ε_uncond`. |
|
||||
| Cross-attention | "How text gets in" | Each U-Net block attends to text tokens as K and V. |
|
||||
| DiT | "Diffusion Transformer" | Replace U-Net with a transformer over latent patches; scales better. |
|
||||
| MMDiT | "Multi-modal DiT" | SD3's architecture: text and image streams with joint attention. |
|
||||
| VAE scaling factor | "Magic number" | Divides latents by ~5.4 so diffusion operates in unit-variance space. |
|
||||
|
||||
## Production note: running Flux-12B on an 8GB consumer GPU
|
||||
|
||||
the reference Flux integration is the canonical "I have a consumer GPU, can I ship this?" recipe. The trick is the same three-knob recipe production inference literature lists applied to a diffusion DiT:
|
||||
|
||||
1. **Staggered loading.** Flux has three networks that never need to coexist in VRAM: T5-XXL text encoder (~10 GB in fp32), CLIP-L (small), the 12B MMDiT, and the VAE. Encode the prompt first, *delete* the encoders, load the DiT, denoise, *delete* the DiT, load the VAE, decode. Consumer 8GB GPUs only fit one stage at a time.
|
||||
2. **4-bit quantization via bitsandbytes.** `BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16)` on both the T5 encoder and the DiT. Cuts memory 8×, quality drop is imperceptible for text-to-image per Aritra's benchmarks (linked in the notebook).
|
||||
3. **CPU offload.** `pipe.enable_model_cpu_offload()` auto-swaps modules between CPU and GPU as each forward pass advances. Adds 10-20% latency but makes the pipeline run at all.
|
||||
|
||||
The memory accounting is: `10 GB T5 / 8 = 1.25 GB` quantized, `12 B params × 0.5 bytes = ~6 GB` quantized DiT, plus activations. In stas00's terms this is the extreme-end of TP=1 inference — no model parallelism, maximum quantization. For production you'd run TP=2 or TP=4 on H100s; for a single dev laptop, this is the recipe.
|
||||
|
||||
## Further Reading
|
||||
|
||||
- [Rombach et al. (2022). High-Resolution Image Synthesis with Latent Diffusion Models](https://arxiv.org/abs/2112.10752) — Stable Diffusion.
|
||||
- [Podell et al. (2023). SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://arxiv.org/abs/2307.01952) — SDXL.
|
||||
- [Peebles & Xie (2023). Scalable Diffusion Models with Transformers (DiT)](https://arxiv.org/abs/2212.09748) — DiT.
|
||||
- [Esser et al. (2024). Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://arxiv.org/abs/2403.03206) — SD3, MMDiT.
|
||||
- [Ho & Salimans (2022). Classifier-Free Diffusion Guidance](https://arxiv.org/abs/2207.12598) — CFG.
|
||||
- [Labs (2024). Flux.1 — Black Forest Labs announcement](https://blackforestlabs.ai/announcing-black-forest-labs/) — Flux.1 family.
|
||||
- [Hugging Face Diffusers docs](https://huggingface.co/docs/diffusers/index) — reference implementation for every checkpoint above.
|
||||
+18
@@ -0,0 +1,18 @@
|
||||
---
|
||||
name: sd-prompter
|
||||
description: Configure Stable Diffusion / Flux inference for a given prompt, style, and quality bar.
|
||||
version: 1.0.0
|
||||
phase: 8
|
||||
lesson: 07
|
||||
tags: [stable-diffusion, flux, latent-diffusion]
|
||||
---
|
||||
|
||||
Given a prompt, target style, and quality bar (fast preview / portfolio quality / print-ready), output:
|
||||
|
||||
1. Model + checkpoint. SD 1.5 (legacy tools), SDXL-base + refiner, SDXL-Turbo (fast), SD3.5-Large, Flux.1-dev (best open), Flux.1-schnell (fast open), or a hosted API (DALL-E 3, Imagen 4, Midjourney v7). One-sentence reason.
|
||||
2. Sampler. Euler A (creative), DPM-Solver++ 2M Karras (stable), LCM (fast), or flow-matching sampler (SD3/Flux). Include step count.
|
||||
3. CFG scale. 0 for turbo / LCM, 3-4 for Flux, 5-7 for SDXL, 7-10 for SD1.5. Document the trade-off.
|
||||
4. Add-ons. ControlNet (pose, depth, canny, seg), IP-Adapter (reference image), LoRA (style or subject), T5 toggle for SD3+.
|
||||
5. Negative prompt. Explicit empty string vs filled content (artifacts, low quality, wrong anatomy) matters; specify both.
|
||||
|
||||
Refuse CFG > 10 for SDXL+ (saturated outputs). Refuse > 50 sampler steps on non-legacy checkpoints (quality plateaus by 30). Refuse to mix LoRAs trained on different base models (SD 1.5 LoRA on SDXL is silently broken). Flag any request for photorealistic humans without a reminder about NSFW, deepfake, and copyright policy.
|
||||
@@ -0,0 +1,103 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 540" font-family="Georgia, 'Times New Roman', serif">
|
||||
<defs>
|
||||
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto">
|
||||
<path d="M0,0 L10,5 L0,10 z" fill="#1a1a1a"/>
|
||||
</marker>
|
||||
<style>
|
||||
.box { fill: #faf6ef; stroke: #1a1a1a; stroke-width: 1.5; }
|
||||
.hot { fill: #fff1d6; stroke: #c0392b; stroke-width: 1.5; }
|
||||
.cold { fill: #eaf4ff; stroke: #2c5f8c; stroke-width: 1.5; }
|
||||
.frozen { fill: #ecece8; stroke: #666; stroke-width: 1; stroke-dasharray: 4,3; }
|
||||
.label { font-size: 14px; font-weight: 600; fill: #1a1a1a; }
|
||||
.content { font-size: 12px; fill: #333; }
|
||||
.mono { font-size: 12px; fill: #333; font-family: 'Menlo', monospace; }
|
||||
.caption { font-size: 11px; fill: #555; font-style: italic; }
|
||||
.title { font-size: 16px; font-weight: 700; fill: #1a1a1a; }
|
||||
</style>
|
||||
</defs>
|
||||
|
||||
<text x="450" y="28" text-anchor="middle" class="title">ControlNet clones the encoder; LoRA adds a low-rank delta</text>
|
||||
|
||||
<!-- ControlNet -->
|
||||
<text x="100" y="70" class="label">ControlNet</text>
|
||||
<rect x="40" y="80" width="120" height="200" class="frozen"/>
|
||||
<text x="100" y="100" text-anchor="middle" class="caption">frozen SD U-Net</text>
|
||||
<rect x="55" y="115" width="90" height="35" class="box"/>
|
||||
<text x="100" y="138" text-anchor="middle" class="content">encoder</text>
|
||||
<rect x="55" y="155" width="90" height="35" class="box"/>
|
||||
<text x="100" y="178" text-anchor="middle" class="content">bottleneck</text>
|
||||
<rect x="55" y="195" width="90" height="35" class="box"/>
|
||||
<text x="100" y="218" text-anchor="middle" class="content">decoder</text>
|
||||
<rect x="55" y="235" width="90" height="35" class="box"/>
|
||||
<text x="100" y="258" text-anchor="middle" class="content">decoder</text>
|
||||
|
||||
<rect x="200" y="80" width="140" height="200" class="cold"/>
|
||||
<text x="270" y="100" text-anchor="middle" class="caption">ControlNet clone (trainable)</text>
|
||||
<rect x="220" y="115" width="100" height="35" class="box"/>
|
||||
<text x="270" y="138" text-anchor="middle" class="content">enc copy</text>
|
||||
<rect x="220" y="155" width="100" height="35" class="box"/>
|
||||
<text x="270" y="178" text-anchor="middle" class="content">bottleneck copy</text>
|
||||
|
||||
<rect x="220" y="230" width="100" height="40" class="hot"/>
|
||||
<text x="270" y="253" text-anchor="middle" class="mono">depth map</text>
|
||||
|
||||
<line x1="270" y1="230" x2="270" y2="190" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
|
||||
<line x1="320" y1="170" x2="360" y2="200" stroke="#c0392b" stroke-width="1.5" marker-end="url(#arrow)"/>
|
||||
<text x="380" y="180" class="mono">zero-conv</text>
|
||||
<line x1="320" y1="130" x2="145" y2="215" stroke="#c0392b" stroke-width="1.2" stroke-dasharray="3,3"/>
|
||||
<line x1="320" y1="170" x2="145" y2="255" stroke="#c0392b" stroke-width="1.2" stroke-dasharray="3,3"/>
|
||||
|
||||
<text x="180" y="310" text-anchor="middle" class="caption">zero-conv init => starts as identity; learns a delta</text>
|
||||
|
||||
<!-- LoRA -->
|
||||
<text x="590" y="70" class="label">LoRA</text>
|
||||
<rect x="500" y="80" width="360" height="220" class="box"/>
|
||||
<text x="680" y="102" text-anchor="middle" class="mono">W' = W + α · B · A</text>
|
||||
|
||||
<rect x="540" y="130" width="70" height="80" class="frozen"/>
|
||||
<text x="575" y="175" text-anchor="middle" class="content">W</text>
|
||||
<text x="575" y="195" text-anchor="middle" class="caption">d × d, frozen</text>
|
||||
|
||||
<text x="630" y="175" class="mono">+</text>
|
||||
|
||||
<rect x="655" y="140" width="60" height="60" class="cold"/>
|
||||
<text x="685" y="175" text-anchor="middle" class="content">B</text>
|
||||
<text x="685" y="195" text-anchor="middle" class="caption">d × r</text>
|
||||
|
||||
<rect x="730" y="150" width="80" height="40" class="cold"/>
|
||||
<text x="770" y="175" text-anchor="middle" class="content">A</text>
|
||||
<text x="770" y="195" text-anchor="middle" class="caption">r × d</text>
|
||||
|
||||
<text x="680" y="240" text-anchor="middle" class="caption">r = 4-16 typical; rank-r compression</text>
|
||||
<text x="680" y="260" text-anchor="middle" class="caption">params: 2 · d · r instead of d²</text>
|
||||
<text x="680" y="280" text-anchor="middle" class="caption">runtime knob: α ∈ [0.5, 1.5]</text>
|
||||
|
||||
<!-- composability -->
|
||||
<rect x="40" y="340" width="820" height="180" class="box"/>
|
||||
<text x="450" y="365" text-anchor="middle" class="label">composability in 2026 pipelines</text>
|
||||
|
||||
<rect x="60" y="380" width="180" height="120" class="cold"/>
|
||||
<text x="150" y="402" text-anchor="middle" class="content">ControlNet</text>
|
||||
<text x="150" y="422" text-anchor="middle" class="caption">spatial (pose, depth,</text>
|
||||
<text x="150" y="438" text-anchor="middle" class="caption">edges, scribble, seg)</text>
|
||||
<text x="150" y="470" text-anchor="middle" class="caption">70-360 MB per modality</text>
|
||||
|
||||
<rect x="260" y="380" width="180" height="120" class="cold"/>
|
||||
<text x="350" y="402" text-anchor="middle" class="content">LoRA</text>
|
||||
<text x="350" y="422" text-anchor="middle" class="caption">style, subject, concept</text>
|
||||
<text x="350" y="438" text-anchor="middle" class="caption">20-200 MB</text>
|
||||
<text x="350" y="470" text-anchor="middle" class="caption">stack multiple with α scaling</text>
|
||||
|
||||
<rect x="460" y="380" width="180" height="120" class="cold"/>
|
||||
<text x="550" y="402" text-anchor="middle" class="content">IP-Adapter</text>
|
||||
<text x="550" y="422" text-anchor="middle" class="caption">reference image as condition</text>
|
||||
<text x="550" y="438" text-anchor="middle" class="caption">via CLIP image tokens</text>
|
||||
<text x="550" y="470" text-anchor="middle" class="caption">~20 MB</text>
|
||||
|
||||
<rect x="660" y="380" width="180" height="120" class="cold"/>
|
||||
<text x="750" y="402" text-anchor="middle" class="content">DreamBooth</text>
|
||||
<text x="750" y="422" text-anchor="middle" class="caption">full fine-tune of base</text>
|
||||
<text x="750" y="438" text-anchor="middle" class="caption">strongest identity</text>
|
||||
<text x="750" y="470" text-anchor="middle" class="caption">2-5 GB</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 6.0 KiB |
@@ -0,0 +1,111 @@
|
||||
import math
|
||||
import random
|
||||
|
||||
|
||||
def matmul_mat_vec(M, v):
|
||||
return [sum(M[i][j] * v[j] for j in range(len(v))) for i in range(len(M))]
|
||||
|
||||
|
||||
def outer(u, v):
|
||||
return [[u[i] * v[j] for j in range(len(v))] for i in range(len(u))]
|
||||
|
||||
|
||||
def zeros(rows, cols):
|
||||
return [[0.0] * cols for _ in range(rows)]
|
||||
|
||||
|
||||
def randn_matrix(rows, cols, rng, scale=0.3):
|
||||
return [[rng.gauss(0, scale) for _ in range(cols)] for _ in range(rows)]
|
||||
|
||||
|
||||
def lora_forward(W_frozen, A, B, x, alpha=1.0):
|
||||
"""Compute (W + alpha * B @ A) @ x."""
|
||||
base = matmul_mat_vec(W_frozen, x)
|
||||
Ax = matmul_mat_vec(A, x)
|
||||
BAx = matmul_mat_vec(B, Ax)
|
||||
return [base[i] + alpha * BAx[i] for i in range(len(base))]
|
||||
|
||||
|
||||
def train_lora(W_frozen, W_target, r, rng, steps=4000, lr=0.01):
|
||||
d = len(W_frozen)
|
||||
A = randn_matrix(r, d, rng, scale=0.2)
|
||||
B = [[0.0] * r for _ in range(d)]
|
||||
for step in range(steps):
|
||||
x = [rng.gauss(0, 1) for _ in range(d)]
|
||||
target = matmul_mat_vec(W_target, x)
|
||||
pred = lora_forward(W_frozen, A, B, x)
|
||||
err = [pred[i] - target[i] for i in range(d)]
|
||||
Ax = matmul_mat_vec(A, x)
|
||||
for i in range(d):
|
||||
for k in range(r):
|
||||
grad_B = err[i] * Ax[k]
|
||||
B[i][k] -= lr * grad_B
|
||||
for k in range(r):
|
||||
for j in range(d):
|
||||
grad_A = sum(err[i] * B[i][k] for i in range(d)) * x[j]
|
||||
A[k][j] -= lr * grad_A
|
||||
total_err = 0.0
|
||||
n = 500
|
||||
for _ in range(n):
|
||||
x = [rng.gauss(0, 1) for _ in range(d)]
|
||||
target = matmul_mat_vec(W_target, x)
|
||||
pred = lora_forward(W_frozen, A, B, x)
|
||||
total_err += sum((a - b) ** 2 for a, b in zip(target, pred))
|
||||
return total_err / n
|
||||
|
||||
|
||||
def controlnet_toy(steps, rng):
|
||||
"""Learn a gated side-network that conditions on an extra signal."""
|
||||
# base: f_base(x) = x (frozen)
|
||||
# side: f_side(x, c) = c (learnable weight w_side)
|
||||
# gated: out = f_base + gate * w_side * c
|
||||
w_side = rng.gauss(0, 0.1)
|
||||
gate = 0.0 # zero-conv init
|
||||
lr = 0.03
|
||||
trace = []
|
||||
for step in range(steps):
|
||||
x = rng.gauss(0, 1)
|
||||
c = rng.choice([-1.0, 1.0])
|
||||
target = x + 0.7 * c # the "true" signal we want
|
||||
pred = x + gate * w_side * c
|
||||
err = pred - target
|
||||
grad_gate = 2 * err * w_side * c
|
||||
grad_wside = 2 * err * gate * c
|
||||
gate -= lr * grad_gate
|
||||
w_side -= lr * grad_wside
|
||||
if (step + 1) % 100 == 0:
|
||||
trace.append((step + 1, gate, w_side))
|
||||
return trace
|
||||
|
||||
|
||||
def main():
|
||||
rng = random.Random(17)
|
||||
d = 6
|
||||
W_frozen = randn_matrix(d, d, rng, scale=0.5)
|
||||
delta = rng.choice([1, 2, 3])
|
||||
delta_matrix = zeros(d, d)
|
||||
u = [rng.gauss(0, 1) for _ in range(d)]
|
||||
v = [rng.gauss(0, 1) for _ in range(d)]
|
||||
for i in range(d):
|
||||
for j in range(d):
|
||||
delta_matrix[i][j] = u[i] * v[j] * 0.5
|
||||
W_target = [[W_frozen[i][j] + delta_matrix[i][j] for j in range(d)] for i in range(d)]
|
||||
|
||||
print("=== LoRA: approximate a known rank-1 delta ===")
|
||||
for r in [1, 2, 4]:
|
||||
err = train_lora(W_frozen, W_target, r=r, rng=random.Random(2 * r))
|
||||
print(f" rank r={r}: residual MSE {err:.5f}")
|
||||
|
||||
print()
|
||||
print("=== ControlNet-lite: zero-initialized gate on a side signal ===")
|
||||
trace = controlnet_toy(steps=800, rng=rng)
|
||||
for step, gate, wside in trace[::2][:6]:
|
||||
print(f" step {step:4d}: gate={gate:+.3f} w_side={wside:+.3f}")
|
||||
|
||||
print()
|
||||
print("takeaway: LoRA needs rank >= true delta rank to converge exactly.")
|
||||
print(" ControlNet-lite gate ramps from 0 as the side signal proves useful.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,156 @@
|
||||
# ControlNet, LoRA & Conditioning
|
||||
|
||||
> Text alone is a clumsy control signal. ControlNet lets you clone a pretrained diffusion model and steer it with a depth map, pose skeleton, scribble, or edge image. LoRA lets you fine-tune a 2B-parameter model by training 10 million parameters. Together they turned Stable Diffusion from a toy into the 2026 image pipeline that ships at every agency.
|
||||
|
||||
**Type:** Build
|
||||
**Languages:** Python
|
||||
**Prerequisites:** Phase 8 · 07 (Latent Diffusion), Phase 10 (LLMs from Scratch — for LoRA foundation)
|
||||
**Time:** ~75 minutes
|
||||
|
||||
## The Problem
|
||||
|
||||
A prompt like "a woman in a red dress walking a dog on a busy street" gives the model no information about *where* the dog is, *what pose* the woman is in, or *the perspective* of the street. Text pins down about 10% of what you need to specify an image. The rest is visual and cannot be described efficiently in words.
|
||||
|
||||
Training a new conditional model from scratch for every signal (pose, depth, canny, segmentation) is prohibitive. You want to keep the 2.6B-param SDXL backbone frozen, attach a small side-network that reads the conditioning, and have it nudge the backbone's intermediate features. That is ControlNet.
|
||||
|
||||
You also want to teach the model new concepts (your face, your product, your style) without retraining the full model. You want a 100x smaller delta. That is LoRA — low-rank adapters that plug into existing attention weights.
|
||||
|
||||
ControlNet + LoRA + text = the 2026 practitioner's toolkit. Most production image pipelines layer 2-5 LoRAs, 1-3 ControlNets, and an IP-Adapter on top of an SDXL / SD3 / Flux base.
|
||||
|
||||
## The Concept
|
||||
|
||||
|
||||
|
||||
### ControlNet (Zhang et al., 2023)
|
||||
|
||||
Take a pretrained SD. *Clone* the encoder half of the U-Net. Freeze the original. Train the clone to accept an extra conditioning input (edges, depth, pose). Connect the clone back to the decoder half of the original with *zero-convolution* skip connections (1×1 convs initialized to zero — start as a no-op, learn a delta).
|
||||
|
||||
```
|
||||
SD U-Net decoder: ... ← orig_enc_features + zero_conv(controlnet_enc(condition))
|
||||
```
|
||||
|
||||
Zero-conv init means ControlNet starts as identity — no harm even before training. Train on 1M (prompt, condition, image) triples with the standard diffusion loss.
|
||||
|
||||
Per-modality ControlNets ship as small side models (~360M for SDXL, ~70M for SD 1.5). You can compose them at inference:
|
||||
|
||||
```
|
||||
features += weight_a * control_a(depth) + weight_b * control_b(pose)
|
||||
```
|
||||
|
||||
### LoRA (Hu et al., 2021)
|
||||
|
||||
For any linear layer `W ∈ R^{d×d}` in the model, freeze `W` and add a low-rank delta:
|
||||
|
||||
```
|
||||
W' = W + ΔW, ΔW = B @ A, A ∈ R^{r×d}, B ∈ R^{d×r}
|
||||
```
|
||||
|
||||
with `r << d`. Rank 4-16 is standard for attention, rank 64-128 for heavy fine-tunes. Number of new parameters: `2 · d · r` instead of `d²`. For SDXL attention with `d=640`, `r=16`: 20k params per adapter instead of 410k — a 20x reduction. Across the whole model: a LoRA is usually 20-200MB vs the base 5GB.
|
||||
|
||||
At inference you can scale the LoRA: `W' = W + α · B @ A`. `α = 0.5-1.5` is normal. Multiple LoRAs stack additively (with the usual caveat that they interact in non-linear ways).
|
||||
|
||||
### IP-Adapter (Ye et al., 2023)
|
||||
|
||||
A tiny adapter that accepts an *image* as conditioning (alongside text). Uses the CLIP image encoder to produce image tokens, injects them into cross-attention alongside text tokens. ~20MB per base model. Lets you do "generate an image in the style of this reference" without a LoRA.
|
||||
|
||||
## Composability matrix
|
||||
|
||||
| Tool | What it controls | Size | When to use |
|
||||
|------|------------------|------|-------------|
|
||||
| ControlNet | Spatial structure (pose, depth, edges) | 70-360MB | Exact layout, composition |
|
||||
| LoRA | Style, subject, concept | 20-200MB | Personalization, style |
|
||||
| IP-Adapter | Style or subject from reference image | 20MB | No text can describe the look |
|
||||
| Textual Inversion | Single concept as a new token | 10KB | Legacy, mostly replaced by LoRA |
|
||||
| DreamBooth | Full fine-tune on a subject | 2-5GB | Strong identity, high compute |
|
||||
| T2I-Adapter | Lighter ControlNet alternative | 70MB | Edge devices, inference budget |
|
||||
|
||||
ControlNet ≈ spatial. LoRA ≈ semantic. Use both.
|
||||
|
||||
## Build It
|
||||
|
||||
`code/main.py` simulates the two mechanisms on 1-D:
|
||||
|
||||
1. **LoRA.** A pretrained linear layer `W`. Freeze it. Train a low-rank `B @ A` such that `W + BA` matches a target linear layer. Show that `r = 1` is enough to learn a rank-1 correction perfectly.
|
||||
|
||||
2. **ControlNet-lite.** A "frozen base" predictor and a "side network" that reads an extra signal. The side network's output is gated by a learnable scalar initialized to zero (our version of zero-conv). Train and watch the gate ramp up.
|
||||
|
||||
### Step 1: LoRA math
|
||||
|
||||
```python
|
||||
def lora(W, A, B, x, alpha=1.0):
|
||||
# W is frozen; A, B are the trainable low-rank factors.
|
||||
return [W[i][j] * x[j] for i, j in ...] + alpha * (B @ (A @ x))
|
||||
```
|
||||
|
||||
### Step 2: zero-init side network
|
||||
|
||||
```python
|
||||
side_out = control_net(x, condition)
|
||||
gated = gate * side_out # gate initialized to 0
|
||||
h = base(x) + gated
|
||||
```
|
||||
|
||||
At step 0 the output is identical to base. Early training updates `gate` slowly — no catastrophic drift.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Over-scaling LoRAs.** `α = 2` or `α = 3` is a common "make it stronger" hack that produces over-stylized / broken outputs. Keep `α ≤ 1.5`.
|
||||
- **ControlNet weight conflict.** Using a Pose ControlNet at weight 1.0 and a Depth ControlNet at weight 1.0 usually overshoots. Sum of weights ≈ 1.0 is a safe default.
|
||||
- **LoRA on the wrong base.** SDXL LoRAs silently no-op on SD 1.5 because the attention dimensions do not match. Diffusers will warn in 0.30+.
|
||||
- **Textual Inversion drift.** Tokens trained on one checkpoint drift badly on another. LoRA is more portable.
|
||||
- **LoRA weight-merging and storage.** You can bake a LoRA into the base model weights for faster inference (no runtime addition), but you lose the ability to scale `α` at runtime. Keep both versions.
|
||||
|
||||
## Use It
|
||||
|
||||
| Goal | 2026 pipeline |
|
||||
|------|---------------|
|
||||
| Reproduce a brand's art style | LoRA trained on ~30 curated images at rank 32 |
|
||||
| Put my face in a generated image | DreamBooth or LoRA + IP-Adapter-FaceID |
|
||||
| Specific pose + prompt | ControlNet-Openpose + SDXL + text |
|
||||
| Depth-aware composition | ControlNet-Depth + SD3 |
|
||||
| Reference + prompt | IP-Adapter + text |
|
||||
| Exact layout | ControlNet-Scribble or ControlNet-Canny |
|
||||
| Background replace | ControlNet-Seg + Inpainting (Lesson 09) |
|
||||
| Fast 1-step style | LCM-LoRA on SDXL-Turbo |
|
||||
|
||||
## Ship It
|
||||
|
||||
Save `outputs/skill-sd-toolkit-composer.md`. Skill takes a task (input assets: prompt, optional reference image, optional pose, optional depth, optional scribble) and outputs the tool stack, weights, and a reproducible seed protocol.
|
||||
|
||||
## Exercises
|
||||
|
||||
1. **Easy.** In `code/main.py`, vary the LoRA rank `r` from 1 to 4. At what rank does the LoRA exactly match a rank-2 target delta?
|
||||
2. **Medium.** Train two separate LoRAs on two target transforms. Load them together and show their additive interaction. When does the interaction break linearity?
|
||||
3. **Hard.** Use diffusers to stack: SDXL-base + Canny-ControlNet (weight 0.8) + a style LoRA (α 0.8) + IP-Adapter (weight 0.6). Measure FID-vs-prompt-adherence trade-off as the stack weights vary.
|
||||
|
||||
## Key Terms
|
||||
|
||||
| Term | What people say | What it actually means |
|
||||
|------|-----------------|-----------------------|
|
||||
| ControlNet | "Spatial control" | Cloned encoder + zero-conv skips; reads a conditioning image. |
|
||||
| Zero convolution | "Starts as identity" | 1×1 conv initialized to zero; ControlNet starts as no-op. |
|
||||
| LoRA | "Low-rank adapter" | `W + B @ A`, `r << d`; 100x fewer params than a full fine-tune. |
|
||||
| rank r | "The knob" | LoRA compression; 4-16 typical, 64+ for heavy personalization. |
|
||||
| α | "LoRA strength" | Runtime scaling of the LoRA delta. |
|
||||
| IP-Adapter | "Reference image" | Small image-conditioning adapter via CLIP-image tokens. |
|
||||
| DreamBooth | "Full subject fine-tune" | Train the full model on ~30 images of a subject. |
|
||||
| Textual Inversion | "New token" | Learn a new word embedding only; legacy, mostly replaced. |
|
||||
|
||||
## Production note: LoRA swaps, ControlNet lanes, multi-tenant serving
|
||||
|
||||
A real text-to-image SaaS serves hundreds of LoRAs and a dozen ControlNets over the same base checkpoint. The serving problem looks a lot like LLM multi-tenancy (the production literature covers the LLM case under continuous batching and LoRAX / S-LoRA):
|
||||
|
||||
- **Hot-swap LoRAs, do not merge.** Merging `W' = W + α·B·A` into the base gives ~3-5% faster per-step inference but freezes `α` and the base. Keep LoRAs hot in VRAM as rank-r deltas; diffusers exposes `pipe.load_lora_weights()` + `pipe.set_adapters([...], adapter_weights=[...])` for per-request activation. Swap cost is the `2 · d · r · num_layers` weights — MB-scale, sub-second.
|
||||
- **ControlNet as a second attention lane.** The cloned encoder runs in parallel with the base. Two ControlNets at weight 1.0 each = two extra forward passes per step, not one merged pass. Batch-size headroom drops quadratically. Budget for ~1.5× step cost per active ControlNet.
|
||||
- **Quantized LoRAs too.** If you quantized the base (see Lesson 07, Flux on 8GB), the LoRA delta also quantizes cleanly to 8-bit or 4-bit. QLoRA-style loading lets you stack 5-10 LoRAs on top of a 4-bit Flux base without blowing memory.
|
||||
|
||||
Flux-specific: Niels' Flux-on-8GB notebook quantizes the base to 4-bit; stacking a style LoRA (`pipe.load_lora_weights("user/style-lora")`) on that quantized base at `weight_name="pytorch_lora_weights.safetensors"` still works. This is the recipe most SaaS agencies ship in 2026.
|
||||
|
||||
## Further Reading
|
||||
|
||||
- [Zhang, Rao, Agrawala (2023). Adding Conditional Control to Text-to-Image Diffusion Models](https://arxiv.org/abs/2302.05543) — ControlNet.
|
||||
- [Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2106.09685) — LoRA (originally for LLMs; ports to diffusion).
|
||||
- [Ye et al. (2023). IP-Adapter: Text Compatible Image Prompt Adapter](https://arxiv.org/abs/2308.06721) — IP-Adapter.
|
||||
- [Mou et al. (2023). T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability](https://arxiv.org/abs/2302.08453) — lighter alternative to ControlNet.
|
||||
- [Ruiz et al. (2023). DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://arxiv.org/abs/2208.12242) — DreamBooth.
|
||||
- [HuggingFace Diffusers — ControlNet / LoRA / IP-Adapter docs](https://huggingface.co/docs/diffusers/training/controlnet) — reference pipelines.
|
||||
+19
@@ -0,0 +1,19 @@
|
||||
---
|
||||
name: sd-toolkit-composer
|
||||
description: Compose ControlNets, LoRAs, and IP-Adapters on top of an SD / Flux base for a given set of inputs.
|
||||
version: 1.0.0
|
||||
phase: 8
|
||||
lesson: 08
|
||||
tags: [controlnet, lora, ip-adapter, diffusion]
|
||||
---
|
||||
|
||||
Given a task (target image), inputs (prompt, reference image, pose / depth / scribble / seg, subject identity), and base model (SDXL, SD3.5, Flux.1-dev), output:
|
||||
|
||||
1. ControlNet stack. Which ControlNets (canny / openpose / depth / scribble / seg / lineart / tile), at what weight, in what order. Max sum of weights <= 1.5.
|
||||
2. LoRA stack. Named LoRAs, rank, alpha. Warn when alpha > 1.5 or multiple LoRAs target the same concept.
|
||||
3. IP-Adapter. None, plain, or FaceID variant; weight 0.4-0.8 typical.
|
||||
4. Text prompt + negative prompt. Keyword order, token budget, negative scaffolding.
|
||||
5. Sampler + CFG + seed. Euler A / DPM-Solver++ / LCM; CFG scale tied to base. Reproducible seed protocol.
|
||||
6. QA checklist. Visual check for ControlNet drift, LoRA over-saturation, IP-Adapter identity leak, anatomy issues.
|
||||
|
||||
Refuse to stack a SD 1.5 LoRA on an SDXL base (dimension mismatch). Refuse to run 3+ ControlNets at weight 1.0 each (feature collision). Flag any SD 1.5 recommendation when the user has GPU budget for SDXL or Flux. Flag LoRA identity training on < 10 images as likely to overfit.
|
||||
@@ -0,0 +1,60 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 500" font-family="Georgia, 'Times New Roman', serif">
|
||||
<defs>
|
||||
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto">
|
||||
<path d="M0,0 L10,5 L0,10 z" fill="#1a1a1a"/>
|
||||
</marker>
|
||||
<style>
|
||||
.box { fill: #faf6ef; stroke: #1a1a1a; stroke-width: 1.5; }
|
||||
.hot { fill: #fff1d6; stroke: #c0392b; stroke-width: 1.5; }
|
||||
.cold { fill: #eaf4ff; stroke: #2c5f8c; stroke-width: 1.5; }
|
||||
.mask { fill: #2c3e50; stroke: #1a1a1a; stroke-width: 1.5; }
|
||||
.label { font-size: 14px; font-weight: 600; fill: #1a1a1a; }
|
||||
.content { font-size: 12px; fill: #333; }
|
||||
.mono { font-size: 12px; fill: #333; font-family: 'Menlo', monospace; }
|
||||
.caption { font-size: 11px; fill: #555; font-style: italic; }
|
||||
.title { font-size: 16px; font-weight: 700; fill: #1a1a1a; }
|
||||
</style>
|
||||
</defs>
|
||||
|
||||
<text x="450" y="28" text-anchor="middle" class="title">inpainting vs outpainting vs SDEdit</text>
|
||||
|
||||
<!-- inpainting panel -->
|
||||
<text x="165" y="70" text-anchor="middle" class="label">inpainting</text>
|
||||
<rect x="40" y="80" width="250" height="130" class="box"/>
|
||||
<rect x="55" y="95" width="220" height="100" class="cold"/>
|
||||
<rect x="120" y="120" width="80" height="50" class="mask"/>
|
||||
<text x="160" y="230" text-anchor="middle" class="caption">mask inside, pin outside</text>
|
||||
<text x="160" y="248" text-anchor="middle" class="caption">9-channel U-Net:</text>
|
||||
<text x="160" y="264" text-anchor="middle" class="mono">noisy | encoded_src | mask</text>
|
||||
|
||||
<!-- outpainting -->
|
||||
<text x="450" y="70" text-anchor="middle" class="label">outpainting</text>
|
||||
<rect x="310" y="80" width="260" height="130" class="box"/>
|
||||
<rect x="410" y="110" width="70" height="70" class="cold"/>
|
||||
<rect x="325" y="95" width="85" height="100" class="mask"/>
|
||||
<rect x="480" y="95" width="85" height="100" class="mask"/>
|
||||
<text x="450" y="230" text-anchor="middle" class="caption">invert the mask</text>
|
||||
<text x="450" y="248" text-anchor="middle" class="caption">extend beyond the canvas</text>
|
||||
<text x="450" y="264" text-anchor="middle" class="caption">same model, same loss</text>
|
||||
|
||||
<!-- SDEdit -->
|
||||
<text x="720" y="70" text-anchor="middle" class="label">SDEdit (no retraining)</text>
|
||||
<rect x="590" y="80" width="270" height="130" class="box"/>
|
||||
<text x="725" y="110" text-anchor="middle" class="mono">x_0 -> add noise to t -> denoise</text>
|
||||
<text x="725" y="135" text-anchor="middle" class="caption">t/T = 0.3 → minor edits</text>
|
||||
<text x="725" y="153" text-anchor="middle" class="caption">t/T = 0.6 → moderate edits</text>
|
||||
<text x="725" y="171" text-anchor="middle" class="caption">t/T = 0.9 → near-random</text>
|
||||
<text x="725" y="195" text-anchor="middle" class="caption">no mask, just noise-level slider</text>
|
||||
|
||||
<!-- pipeline flow -->
|
||||
<rect x="40" y="310" width="820" height="100" class="hot"/>
|
||||
<text x="450" y="335" text-anchor="middle" class="label">inpainting inference loop</text>
|
||||
<text x="450" y="358" text-anchor="middle" class="mono">for t = T .. 1: x_t[masked] = denoise; x_t[unmasked] = noise(clean_source, t)</text>
|
||||
<text x="450" y="380" text-anchor="middle" class="caption">replace unmasked region with a fresh forward-diffused clean image each step</text>
|
||||
<text x="450" y="398" text-anchor="middle" class="caption">final step: pin unmasked pixels to the clean source exactly</text>
|
||||
|
||||
<!-- tools -->
|
||||
<rect x="40" y="425" width="820" height="60" class="box"/>
|
||||
<text x="450" y="448" text-anchor="middle" class="label">2026 editing stack</text>
|
||||
<text x="450" y="470" text-anchor="middle" class="caption">SAM 2 mask → SD-Inpaint / Flux-Fill / GPT-Image Edit → Flux-Kontext for instruction edits</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 3.7 KiB |
@@ -0,0 +1,197 @@
|
||||
import math
|
||||
import random
|
||||
|
||||
|
||||
def sin_embed(t, T, dim=8):
|
||||
out = []
|
||||
half = dim // 2
|
||||
for i in range(half):
|
||||
freq = 1.0 / (10000 ** (i / max(half - 1, 1)))
|
||||
out.append(math.sin(t * freq))
|
||||
out.append(math.cos(t * freq))
|
||||
return out[:dim]
|
||||
|
||||
|
||||
def tanh(v):
|
||||
return [math.tanh(x) for x in v]
|
||||
|
||||
|
||||
def tanh_grad(h):
|
||||
return [1 - x * x for x in h]
|
||||
|
||||
|
||||
def matmul(W, x):
|
||||
return [sum(w * xi for w, xi in zip(row, x)) for row in W]
|
||||
|
||||
|
||||
def add(a, b):
|
||||
return [x + y for x, y in zip(a, b)]
|
||||
|
||||
|
||||
def randn_matrix(rows, cols, rng, scale=0.3):
|
||||
return [[rng.gauss(0, scale) for _ in range(cols)] for _ in range(rows)]
|
||||
|
||||
|
||||
def init_net(x_dim, t_dim, hidden, rng):
|
||||
return {
|
||||
"W1": randn_matrix(hidden, x_dim + t_dim, rng),
|
||||
"b1": [0.0] * hidden,
|
||||
"W2": randn_matrix(hidden, hidden, rng),
|
||||
"b2": [0.0] * hidden,
|
||||
"W3": randn_matrix(x_dim, hidden, rng),
|
||||
"b3": [0.0] * x_dim,
|
||||
}
|
||||
|
||||
|
||||
def forward(x_t, t_emb, net):
|
||||
inp = list(x_t) + list(t_emb)
|
||||
pre1 = add(matmul(net["W1"], inp), net["b1"])
|
||||
h1 = tanh(pre1)
|
||||
pre2 = add(matmul(net["W2"], h1), net["b2"])
|
||||
h2 = tanh(pre2)
|
||||
out = add(matmul(net["W3"], h2), net["b3"])
|
||||
return out, {"inp": inp, "h1": h1, "h2": h2}
|
||||
|
||||
|
||||
def backward(target, out, cache, net):
|
||||
grads = {k: None for k in net}
|
||||
for p in net:
|
||||
if isinstance(net[p][0], list):
|
||||
grads[p] = [[0.0] * len(net[p][0]) for _ in net[p]]
|
||||
else:
|
||||
grads[p] = [0.0] * len(net[p])
|
||||
d_out = [2 * (a - b) for a, b in zip(out, target)]
|
||||
for i in range(len(d_out)):
|
||||
grads["b3"][i] += d_out[i]
|
||||
for j in range(len(cache["h2"])):
|
||||
grads["W3"][i][j] += d_out[i] * cache["h2"][j]
|
||||
d_h2 = [sum(net["W3"][i][j] * d_out[i] for i in range(len(d_out)))
|
||||
for j in range(len(cache["h2"]))]
|
||||
d_pre2 = [d_h2[j] * tanh_grad(cache["h2"])[j] for j in range(len(cache["h2"]))]
|
||||
for j in range(len(cache["h2"])):
|
||||
grads["b2"][j] += d_pre2[j]
|
||||
for k in range(len(cache["h1"])):
|
||||
grads["W2"][j][k] += d_pre2[j] * cache["h1"][k]
|
||||
d_h1 = [sum(net["W2"][j][k] * d_pre2[j] for j in range(len(cache["h2"])))
|
||||
for k in range(len(cache["h1"]))]
|
||||
d_pre1 = [d_h1[j] * tanh_grad(cache["h1"])[j] for j in range(len(cache["h1"]))]
|
||||
for j in range(len(cache["h1"])):
|
||||
grads["b1"][j] += d_pre1[j]
|
||||
for k in range(len(cache["inp"])):
|
||||
grads["W1"][j][k] += d_pre1[j] * cache["inp"][k]
|
||||
return grads
|
||||
|
||||
|
||||
def apply(net, grads, lr):
|
||||
for k, v in net.items():
|
||||
if isinstance(v[0], list):
|
||||
for i in range(len(v)):
|
||||
for j in range(len(v[i])):
|
||||
v[i][j] -= lr * grads[k][i][j]
|
||||
else:
|
||||
for i in range(len(v)):
|
||||
v[i] -= lr * grads[k][i]
|
||||
|
||||
|
||||
def make_schedule(T):
|
||||
betas = [1e-4 + (0.02 - 1e-4) * t / (T - 1) for t in range(T)]
|
||||
alphas = [1 - b for b in betas]
|
||||
bars, cum = [], 1.0
|
||||
for a in alphas:
|
||||
cum *= a
|
||||
bars.append(cum)
|
||||
return alphas, bars
|
||||
|
||||
|
||||
def sample_data(rng, d=5):
|
||||
cluster = rng.choice([0, 1])
|
||||
center = [-1.0 if cluster == 0 else 1.0] * d
|
||||
return [c + rng.gauss(0, 0.2) for c in center], cluster
|
||||
|
||||
|
||||
def train(net, alpha_bars, T, steps, lr, t_dim, d, rng):
|
||||
for step in range(steps):
|
||||
x0, _ = sample_data(rng, d)
|
||||
t = rng.randrange(T)
|
||||
eps = [rng.gauss(0, 1) for _ in range(d)]
|
||||
a_bar = alpha_bars[t]
|
||||
x_t = [math.sqrt(a_bar) * x0[i] + math.sqrt(1 - a_bar) * eps[i] for i in range(d)]
|
||||
t_emb = sin_embed(t, T, t_dim)
|
||||
out, cache = forward(x_t, t_emb, net)
|
||||
grads = backward(eps, out, cache, net)
|
||||
apply(net, grads, lr)
|
||||
|
||||
|
||||
def sample_unconditional(net, alphas, alpha_bars, T, t_dim, d, rng):
|
||||
x = [rng.gauss(0, 1) for _ in range(d)]
|
||||
for t in range(T - 1, -1, -1):
|
||||
t_emb = sin_embed(t, T, t_dim)
|
||||
eps_hat, _ = forward(x, t_emb, net)
|
||||
beta_t = 1 - alphas[t]
|
||||
mean = [(x[i] - beta_t / math.sqrt(1 - alpha_bars[t]) * eps_hat[i]) / math.sqrt(alphas[t])
|
||||
for i in range(d)]
|
||||
if t > 0:
|
||||
x = [mean[i] + math.sqrt(beta_t) * rng.gauss(0, 1) for i in range(d)]
|
||||
else:
|
||||
x = mean
|
||||
return x
|
||||
|
||||
|
||||
def inpaint(net, alphas, alpha_bars, T, t_dim, d, clean, mask, rng):
|
||||
"""mask[i] == True means that dim is to be regenerated. Unmasked dims pinned to clean."""
|
||||
x = [rng.gauss(0, 1) for _ in range(d)]
|
||||
for t in range(T - 1, -1, -1):
|
||||
a_bar = alpha_bars[t]
|
||||
for i in range(d):
|
||||
if not mask[i]:
|
||||
x[i] = math.sqrt(a_bar) * clean[i] + math.sqrt(1 - a_bar) * rng.gauss(0, 1)
|
||||
t_emb = sin_embed(t, T, t_dim)
|
||||
eps_hat, _ = forward(x, t_emb, net)
|
||||
beta_t = 1 - alphas[t]
|
||||
mean = [(x[i] - beta_t / math.sqrt(1 - alpha_bars[t]) * eps_hat[i]) / math.sqrt(alphas[t])
|
||||
for i in range(d)]
|
||||
if t > 0:
|
||||
x = [mean[i] + math.sqrt(beta_t) * rng.gauss(0, 1) for i in range(d)]
|
||||
else:
|
||||
x = mean
|
||||
for i in range(d):
|
||||
if not mask[i]:
|
||||
x[i] = clean[i]
|
||||
return x
|
||||
|
||||
|
||||
def main():
|
||||
rng = random.Random(5)
|
||||
T, t_dim, hidden, d = 40, 8, 32, 5
|
||||
alphas, alpha_bars = make_schedule(T)
|
||||
net = init_net(d, t_dim, hidden, rng)
|
||||
|
||||
print("=== training 5-D DDPM on two-cluster mixture ===")
|
||||
train(net, alpha_bars, T, steps=5000, lr=0.01, t_dim=t_dim, d=d, rng=rng)
|
||||
|
||||
print()
|
||||
print("=== inpainting: pin dims 0-2, regenerate dims 3-4 ===")
|
||||
for trial in range(5):
|
||||
clean, cluster = sample_data(rng, d)
|
||||
mask = [False, False, False, True, True]
|
||||
out = inpaint(net, alphas, alpha_bars, T, t_dim, d, clean, mask, rng)
|
||||
label = "neg cluster" if cluster == 0 else "pos cluster"
|
||||
print(f" {label}: pinned={[f'{clean[i]:+.2f}' for i in range(3)]} "
|
||||
f"filled={[f'{out[i]:+.2f}' for i in range(3, 5)]}")
|
||||
|
||||
print()
|
||||
print("=== outpainting (mask dims 0-1, pin 2-4) ===")
|
||||
for trial in range(3):
|
||||
clean, cluster = sample_data(rng, d)
|
||||
mask = [True, True, False, False, False]
|
||||
out = inpaint(net, alphas, alpha_bars, T, t_dim, d, clean, mask, rng)
|
||||
print(f" pinned tail=[{clean[2]:+.2f}, {clean[3]:+.2f}, {clean[4]:+.2f}] "
|
||||
f"filled head=[{out[0]:+.2f}, {out[1]:+.2f}]")
|
||||
|
||||
print()
|
||||
print("takeaway: the filled dims match the cluster sign of the pinned dims.")
|
||||
print(" that is why inpainting looks coherent with the surroundings.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,156 @@
|
||||
# Inpainting, Outpainting & Image Editing
|
||||
|
||||
> Text-to-image makes new things. Inpainting fixes old ones. In production, 70% of billable image work is editing — swap a background, remove a logo, extend the canvas, regenerate a hand. Inpainting is where diffusion earns its keep.
|
||||
|
||||
**Type:** Build
|
||||
**Languages:** Python
|
||||
**Prerequisites:** Phase 8 · 07 (Latent Diffusion), Phase 8 · 08 (ControlNet & LoRA)
|
||||
**Time:** ~75 minutes
|
||||
|
||||
## The Problem
|
||||
|
||||
A client sends a perfect product photo with a distracting sign in the background. You want to erase the sign and leave everything else pixel-identical. You cannot run text-to-image from scratch — the result will have a different color, different lighting, different product angle. You want to regenerate *only* the masked region, and you want the regeneration to respect the surrounding context.
|
||||
|
||||
That is inpainting. Variants:
|
||||
|
||||
- **Inpainting.** Regenerate inside a mask, keep outside pixels.
|
||||
- **Outpainting.** Regenerate outside a mask (or beyond the canvas), keep inside.
|
||||
- **Image editing.** Regenerate the whole image but keep semantic or structural fidelity to the original (SDEdit, InstructPix2Pix).
|
||||
|
||||
Every diffusion pipeline in 2026 ships an inpainting mode. Flux.1-Fill, Stable Diffusion Inpaint, SDXL-Inpaint, DALL-E 3 Edit. They work on the same principle.
|
||||
|
||||
## The Concept
|
||||
|
||||
|
||||
|
||||
### The naive approach (and why it's wrong)
|
||||
|
||||
Run standard text-to-image with a mask. At each sampling step, replace the unmasked region of the noisy latent with the forward-diffused clean image. It works... badly. Boundary artifacts bleed through because the model has no information about what is in the masked region.
|
||||
|
||||
### The proper inpainting model
|
||||
|
||||
Train a modified U-Net that takes 9 input channels instead of 4:
|
||||
|
||||
```
|
||||
input = concat([ noisy_latent (4ch), encoded_image (4ch), mask (1ch) ], dim=channel)
|
||||
```
|
||||
|
||||
The extra channels are a copy of the VAE-encoded source image plus a single-channel mask. At training time, you randomly mask regions of the image and train the model to denoise only the masked region while the unmasked region is given as a clean conditioning signal. At inference, the model can "see" what surrounds the masked region and produces coherent completions.
|
||||
|
||||
SD-Inpaint, SDXL-Inpaint, Flux-Fill all use this 9-channel (or analog) input. Diffusers `StableDiffusionInpaintPipeline`, `FluxFillPipeline`.
|
||||
|
||||
### SDEdit (Meng et al., 2022) — free editing
|
||||
|
||||
Add noise to the source image up to some intermediate `t`, then run the reverse chain from `t` down to 0 with a new prompt. No retraining. The choice of starting `t` trades fidelity for creative freedom:
|
||||
|
||||
- `t/T = 0.3` → nearly identical to source, small stylistic changes
|
||||
- `t/T = 0.6` → moderate edits, preserves coarse structure
|
||||
- `t/T = 0.9` → generated from near-noise, minimal source preservation
|
||||
|
||||
### InstructPix2Pix (Brooks et al., 2023)
|
||||
|
||||
Fine-tune a diffusion model on `(input_image, instruction, output_image)` triples. At inference, condition on both the input image and a text instruction ("make it sunset", "add a dragon"). Two CFG scales: image scale and text scale.
|
||||
|
||||
### RePaint (Lugmayr et al., 2022)
|
||||
|
||||
Keep a standard unconditional diffusion model. At each reverse step, resample — jump back to a noisier state occasionally and regenerate. Avoids boundary artifacts. Used when you don't have a trained inpainting model.
|
||||
|
||||
## Build It
|
||||
|
||||
`code/main.py` implements a toy 1-D inpainting scheme on 5-dimensional data. We train a DDPM on 5-D mixture data where each sample is 5 floats from one of two clusters. At inference, we "mask" 2 of the 5 dimensions, inject the noisy-forward version of the unmasked three at each step, and regenerate only the masked dimensions.
|
||||
|
||||
### Step 1: 5-D DDPM data
|
||||
|
||||
```python
|
||||
def sample_data(rng):
|
||||
cluster = rng.choice([0, 1])
|
||||
center = [-1.0] * 5 if cluster == 0 else [1.0] * 5
|
||||
return [c + rng.gauss(0, 0.2) for c in center], cluster
|
||||
```
|
||||
|
||||
### Step 2: train denoiser over all 5 dims
|
||||
|
||||
Standard DDPM. Net outputs 5-D noise prediction for 5-D noisy input.
|
||||
|
||||
### Step 3: at inference, mask-aware reverse
|
||||
|
||||
```python
|
||||
def inpaint_step(x_t, mask, clean_image, alpha_bars, t, rng):
|
||||
# replace unmasked dims with a freshly noised version of the clean source
|
||||
a_bar = alpha_bars[t]
|
||||
for i in range(len(x_t)):
|
||||
if not mask[i]:
|
||||
x_t[i] = math.sqrt(a_bar) * clean_image[i] + math.sqrt(1 - a_bar) * rng.gauss(0, 1)
|
||||
# ...then run the normal reverse step on x_t
|
||||
```
|
||||
|
||||
This is the naive approach and it works on toy 1-D data. Real image inpainting uses the 9-channel input because texture coherence matters more.
|
||||
|
||||
### Step 4: outpainting
|
||||
|
||||
Outpainting is inpainting with the mask inverted: mask the new (previously non-existent) canvas, fill the rest with the original. Identical training objective.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Seams.** The naive approach leaves visible boundaries because gradient info doesn't flow across the mask. Fix: dilate the mask by 8-16 pixels, or use a proper inpainting model.
|
||||
- **Mask leakage.** If the conditioning image's unmasked region is low-quality or noisy, it pollutes the generation inside the mask. Denoise or blur slightly.
|
||||
- **CFG interacts with mask size.** High CFG on a small mask = saturated patch. Reduce CFG for small edits.
|
||||
- **SDEdit fidelity cliff.** Going from `t/T = 0.5` to `t/T = 0.6` can lose the subject's identity. Sweep and checkpoint.
|
||||
- **Prompt mismatch.** The prompt should describe the *whole* image, not just the new content. "A cat sitting on a chair" not "a cat".
|
||||
|
||||
## Use It
|
||||
|
||||
| Task | Pipeline |
|
||||
|------|----------|
|
||||
| Remove object, small mask | SD-Inpaint or Flux-Fill, standard prompt |
|
||||
| Replace sky | SD-Inpaint + "blue sky at sunset" |
|
||||
| Extend canvas | SDXL outpaint mode (8px feather) or Flux-Fill with outpaint mask |
|
||||
| Regenerate hand / face | SD-Inpaint with prompt re-describing the subject + ControlNet-Openpose |
|
||||
| Change style of one region | SDEdit at `t/T=0.5` on masked region |
|
||||
| "Make it sunset" | InstructPix2Pix or Flux-Kontext |
|
||||
| Background replacement | SAM mask → SD-Inpaint |
|
||||
| Ultra-high-fidelity | Flux-Fill or GPT-Image (hosted) for hardest cases |
|
||||
|
||||
SAM (Meta's Segment Anything, 2023) + diffusion inpaint is the 2026 background-removal pipeline. SAM 2 (2024) works on video.
|
||||
|
||||
## Ship It
|
||||
|
||||
Save `outputs/skill-editing-pipeline.md`. Skill takes an original image + edit description + optional mask (or SAM prompt) and outputs: mask-generation approach, base model, CFG scales (image + text), SDEdit-t or inpainting mode, and QA checklist.
|
||||
|
||||
## Exercises
|
||||
|
||||
1. **Easy.** In `code/main.py`, vary the fraction of dimensions masked from 0.2 to 0.8. At what fraction does the inpaint quality (residual in masked dims) equal unconditional generation?
|
||||
2. **Medium.** Implement RePaint: at every 10th reverse step, jump back 5 steps (add noise) and re-denoise. Measure whether it reduces boundary residual at the mask edge.
|
||||
3. **Hard.** Use Hugging Face diffusers to compare: SD 1.5 Inpaint + ControlNet-Openpose vs Flux.1-Fill on 20 face-regeneration tasks. Score pose adherence and identity preservation separately.
|
||||
|
||||
## Key Terms
|
||||
|
||||
| Term | What people say | What it actually means |
|
||||
|------|-----------------|-----------------------|
|
||||
| Inpainting | "Fill the hole" | Regenerate inside a mask; keep outside pixels. |
|
||||
| Outpainting | "Extend the canvas" | Regenerate outside the canvas; keep inside. |
|
||||
| 9-channel U-Net | "Proper inpainting model" | U-Net with `noisy | encoded-source | mask` as input. |
|
||||
| SDEdit | "Img2img with noise level" | Noise to time `t`, denoise with new prompt. |
|
||||
| InstructPix2Pix | "Text-only edits" | Fine-tuned diffusion on (image, instruction, output) triples. |
|
||||
| RePaint | "No retraining" | Re-noise periodically during reverse to reduce seams. |
|
||||
| SAM | "Segment Anything" | Mask generator by clicks or boxes; pairs with inpaint. |
|
||||
| Flux-Kontext | "Edit with context" | Flux variant that accepts a reference image + instruction for edits. |
|
||||
|
||||
## Production note: edit pipelines are latency-sensitive
|
||||
|
||||
Users editing an image expect sub-5-second round trips. A 30-step SDXL-Inpaint at 1024² is 3-4 s on an L4, plus SAM mask generation (~200 ms) and VAE encode/decode (~500 ms combined). In production framing, this is TTFT-bound rather than throughput-bound — batch 1, low concurrency, minimize every stage:
|
||||
|
||||
- **SAM-H is the slow one.** SAM-H at 1024² is ~200 ms; SAM-ViT-B is ~40 ms with minor quality loss. SAM 2 (video) adds temporal overhead; do not use it for single-image edits.
|
||||
- **Skip the encode when possible.** `pipe.image_processor.preprocess(img)` encodes to latents. If you have the latents from the previous generation (typical in iterative-edit UIs), pass them directly via `latents=...` to skip one VAE encode.
|
||||
- **Mask dilation matters for throughput too.** A small mask means most of the U-Net forward pass is wasted (the unmasked pixels are clamped anyway). `diffusers`' `StableDiffusionInpaintPipeline` runs the full U-Net regardless; only the 9-channel proper-inpaint variants exploit masked compute.
|
||||
- **Flux-Kontext is the 2025 answer.** Single forward pass over `(source_image, instruction)` — no separate mask, no SDEdit noise sweep. On an H100 it ships an edit in ~1.5 s. The architectural lesson: collapse the stages.
|
||||
|
||||
## Further Reading
|
||||
|
||||
- [Lugmayr et al. (2022). RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://arxiv.org/abs/2201.09865) — training-free inpainting.
|
||||
- [Meng et al. (2022). SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://arxiv.org/abs/2108.01073) — SDEdit.
|
||||
- [Brooks, Holynski, Efros (2023). InstructPix2Pix](https://arxiv.org/abs/2211.09800) — text-instruction editing.
|
||||
- [Kirillov et al. (2023). Segment Anything](https://arxiv.org/abs/2304.02643) — SAM, the mask source.
|
||||
- [Ravi et al. (2024). SAM 2: Segment Anything in Images and Videos](https://arxiv.org/abs/2408.00714) — video SAM.
|
||||
- [Hertz et al. (2022). Prompt-to-Prompt Image Editing with Cross-Attention Control](https://arxiv.org/abs/2208.01626) — attention-level editing.
|
||||
- [Black Forest Labs (2024). Flux.1-Fill and Flux.1-Kontext](https://blackforestlabs.ai/flux-1-tools/) — 2024 tooling.
|
||||
+18
@@ -0,0 +1,18 @@
|
||||
---
|
||||
name: editing-pipeline
|
||||
description: Plan an image-editing pipeline from source + edit description to a ready-to-ship output.
|
||||
version: 1.0.0
|
||||
phase: 8
|
||||
lesson: 09
|
||||
tags: [inpaint, outpaint, edit, sam]
|
||||
---
|
||||
|
||||
Given source image, target edit (remove X, replace Y with Z, extend canvas, restyle region, change season / time-of-day), and quality bar (draft / portfolio / print), output:
|
||||
|
||||
1. Mask strategy. Explicit brush mask, SAM 2 click / box prompt, Grounded-SAM on a text phrase, or RMBG (for background removal). One-sentence reason.
|
||||
2. Base model + mode. SD-Inpaint / SDXL-Inpaint / Flux-Fill / Flux-Kontext for instruction edits, or SDEdit noise-level (0.3 / 0.6 / 0.9) if no mask.
|
||||
3. Prompt scaffolding. Describe the whole image after edit, not only the new content. Include negative prompt.
|
||||
4. CFG + strength + feather. Mask feather 8-16 px; CFG ~5-7 for SDXL-inpaint, 3-4 for Flux. Strength 0.8-1.0 for full regenerate, 0.3-0.5 for preserve.
|
||||
5. Guardrails. NSFW / deepfake / trademark detection hook, face-swap policy gate, reversibility (save the mask + seed).
|
||||
|
||||
Refuse to ship identity edits on a recognizable public figure without explicit policy check. Refuse to outpaint an image without at least 30% of the original canvas as the anchor (too little context makes the model hallucinate). Flag any SDEdit run with t/T > 0.7 and fidelity target "preserve subject" as a likely mismatch.
|
||||
@@ -0,0 +1,70 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 500" font-family="Georgia, 'Times New Roman', serif">
|
||||
<defs>
|
||||
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto">
|
||||
<path d="M0,0 L10,5 L0,10 z" fill="#1a1a1a"/>
|
||||
</marker>
|
||||
<style>
|
||||
.box { fill: #faf6ef; stroke: #1a1a1a; stroke-width: 1.5; }
|
||||
.hot { fill: #fff1d6; stroke: #c0392b; stroke-width: 1.5; }
|
||||
.cold { fill: #eaf4ff; stroke: #2c5f8c; stroke-width: 1.5; }
|
||||
.label { font-size: 14px; font-weight: 600; fill: #1a1a1a; }
|
||||
.content { font-size: 12px; fill: #333; }
|
||||
.mono { font-size: 12px; fill: #333; font-family: 'Menlo', monospace; }
|
||||
.caption { font-size: 11px; fill: #555; font-style: italic; }
|
||||
.title { font-size: 16px; font-weight: 700; fill: #1a1a1a; }
|
||||
</style>
|
||||
</defs>
|
||||
|
||||
<text x="450" y="28" text-anchor="middle" class="title">video diffusion: patchify, DiT, decode</text>
|
||||
|
||||
<!-- raw video -->
|
||||
<text x="80" y="70" class="label">raw video</text>
|
||||
<g>
|
||||
<rect x="30" y="85" width="30" height="30" class="box"/>
|
||||
<rect x="65" y="85" width="30" height="30" class="box"/>
|
||||
<rect x="100" y="85" width="30" height="30" class="box"/>
|
||||
<rect x="135" y="85" width="30" height="30" class="box"/>
|
||||
<rect x="170" y="85" width="30" height="30" class="box"/>
|
||||
<text x="115" y="135" text-anchor="middle" class="caption">T × H × W × 3 (240 frames @ 1080p)</text>
|
||||
</g>
|
||||
|
||||
<line x1="210" y1="100" x2="260" y2="100" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
|
||||
<!-- 3D VAE -->
|
||||
<rect x="260" y="75" width="140" height="60" class="cold"/>
|
||||
<text x="330" y="105" text-anchor="middle" class="content">3-D VAE encoder</text>
|
||||
<text x="330" y="125" text-anchor="middle" class="caption">spatiotemporal latent</text>
|
||||
|
||||
<line x1="400" y1="100" x2="450" y2="100" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
|
||||
<!-- patchify -->
|
||||
<rect x="450" y="75" width="140" height="60" class="hot"/>
|
||||
<text x="520" y="105" text-anchor="middle" class="content">patchify</text>
|
||||
<text x="520" y="125" text-anchor="middle" class="caption">t_p × h_p × w_p blocks</text>
|
||||
|
||||
<line x1="590" y1="100" x2="640" y2="100" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
|
||||
<!-- DiT -->
|
||||
<rect x="640" y="60" width="220" height="90" class="cold"/>
|
||||
<text x="750" y="85" text-anchor="middle" class="label">spatiotemporal DiT</text>
|
||||
<text x="750" y="105" text-anchor="middle" class="caption">factorized: spatial then temporal attn</text>
|
||||
<text x="750" y="125" text-anchor="middle" class="caption">cross-attn to T5-XXL text</text>
|
||||
|
||||
<!-- loss -->
|
||||
<rect x="30" y="180" width="830" height="60" class="hot"/>
|
||||
<text x="445" y="203" text-anchor="middle" class="label">same DDPM loss over spatiotemporal latents</text>
|
||||
<text x="445" y="225" text-anchor="middle" class="mono">L = E || ε - ε_θ( z_t, t, text, first_frame_opt ) ||²</text>
|
||||
|
||||
<!-- temporal coherence panel -->
|
||||
<rect x="30" y="260" width="400" height="220" class="box"/>
|
||||
<text x="230" y="285" text-anchor="middle" class="label">flicker: independent per-frame sampling</text>
|
||||
<path d="M 60,370 L 100,340 L 140,410 L 180,320 L 220,390 L 260,335 L 300,405 L 340,340 L 380,395"
|
||||
fill="none" stroke="#c0392b" stroke-width="1.5"/>
|
||||
<text x="230" y="460" text-anchor="middle" class="caption">per-frame noise is independent => jagged motion</text>
|
||||
|
||||
<rect x="460" y="260" width="400" height="220" class="box"/>
|
||||
<text x="660" y="285" text-anchor="middle" class="label">coherent: joint sequence diffusion</text>
|
||||
<path d="M 490,370 Q 540,355 580,365 T 680,370 T 780,380 T 840,395"
|
||||
fill="none" stroke="#2c5f8c" stroke-width="1.5"/>
|
||||
<text x="660" y="460" text-anchor="middle" class="caption">shared noise + temporal attention => smooth motion</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 3.8 KiB |
@@ -0,0 +1,208 @@
|
||||
import math
|
||||
import random
|
||||
|
||||
|
||||
def sin_embed(t, dim=8):
|
||||
out = []
|
||||
half = dim // 2
|
||||
for i in range(half):
|
||||
freq = 1.0 / (10000 ** (i / max(half - 1, 1)))
|
||||
out.append(math.sin(t * freq))
|
||||
out.append(math.cos(t * freq))
|
||||
return out[:dim]
|
||||
|
||||
|
||||
def tanh(v):
|
||||
return [math.tanh(x) for x in v]
|
||||
|
||||
|
||||
def tanh_grad(h):
|
||||
return [1 - x * x for x in h]
|
||||
|
||||
|
||||
def matmul(W, x):
|
||||
return [sum(w * xi for w, xi in zip(row, x)) for row in W]
|
||||
|
||||
|
||||
def add(a, b):
|
||||
return [x + y for x, y in zip(a, b)]
|
||||
|
||||
|
||||
def randn_matrix(rows, cols, rng, scale=0.3):
|
||||
return [[rng.gauss(0, scale) for _ in range(cols)] for _ in range(rows)]
|
||||
|
||||
|
||||
T_FRAMES = 6
|
||||
POS_DIM = 4
|
||||
|
||||
|
||||
def make_video(rng):
|
||||
"""1-D 'video': smooth trajectory of T_FRAMES values."""
|
||||
base = rng.gauss(0, 1)
|
||||
slope = rng.gauss(0, 0.3)
|
||||
return [base + slope * t + rng.gauss(0, 0.05) for t in range(T_FRAMES)]
|
||||
|
||||
|
||||
def patchify_with_pos(video):
|
||||
"""Each 'patch' here is one frame value + its time position embedding."""
|
||||
out = []
|
||||
for t in range(T_FRAMES):
|
||||
pe = sin_embed(t, POS_DIM)
|
||||
out.append([video[t]] + pe)
|
||||
return out # list of (1 + POS_DIM) vectors
|
||||
|
||||
|
||||
def flatten(patches):
|
||||
return [v for patch in patches for v in patch]
|
||||
|
||||
|
||||
def init_net(in_dim, hidden, out_dim, rng):
|
||||
return {
|
||||
"W1": randn_matrix(hidden, in_dim, rng),
|
||||
"b1": [0.0] * hidden,
|
||||
"W2": randn_matrix(hidden, hidden, rng),
|
||||
"b2": [0.0] * hidden,
|
||||
"W3": randn_matrix(out_dim, hidden, rng),
|
||||
"b3": [0.0] * out_dim,
|
||||
}
|
||||
|
||||
|
||||
def forward(x, t_emb, net):
|
||||
inp = list(x) + list(t_emb)
|
||||
pre1 = add(matmul(net["W1"], inp), net["b1"])
|
||||
h1 = tanh(pre1)
|
||||
pre2 = add(matmul(net["W2"], h1), net["b2"])
|
||||
h2 = tanh(pre2)
|
||||
out = add(matmul(net["W3"], h2), net["b3"])
|
||||
return out, {"inp": inp, "h1": h1, "h2": h2}
|
||||
|
||||
|
||||
def backward(target, out, cache, net):
|
||||
grads = {k: None for k in net}
|
||||
for p in net:
|
||||
if isinstance(net[p][0], list):
|
||||
grads[p] = [[0.0] * len(net[p][0]) for _ in net[p]]
|
||||
else:
|
||||
grads[p] = [0.0] * len(net[p])
|
||||
d_out = [2 * (a - b) for a, b in zip(out, target)]
|
||||
for i in range(len(d_out)):
|
||||
grads["b3"][i] += d_out[i]
|
||||
for j in range(len(cache["h2"])):
|
||||
grads["W3"][i][j] += d_out[i] * cache["h2"][j]
|
||||
d_h2 = [sum(net["W3"][i][j] * d_out[i] for i in range(len(d_out)))
|
||||
for j in range(len(cache["h2"]))]
|
||||
d_pre2 = [d_h2[j] * tanh_grad(cache["h2"])[j] for j in range(len(cache["h2"]))]
|
||||
for j in range(len(cache["h2"])):
|
||||
grads["b2"][j] += d_pre2[j]
|
||||
for k in range(len(cache["h1"])):
|
||||
grads["W2"][j][k] += d_pre2[j] * cache["h1"][k]
|
||||
d_h1 = [sum(net["W2"][j][k] * d_pre2[j] for j in range(len(cache["h2"])))
|
||||
for k in range(len(cache["h1"]))]
|
||||
d_pre1 = [d_h1[j] * tanh_grad(cache["h1"])[j] for j in range(len(cache["h1"]))]
|
||||
for j in range(len(cache["h1"])):
|
||||
grads["b1"][j] += d_pre1[j]
|
||||
for k in range(len(cache["inp"])):
|
||||
grads["W1"][j][k] += d_pre1[j] * cache["inp"][k]
|
||||
return grads
|
||||
|
||||
|
||||
def apply(net, grads, lr):
|
||||
for k, v in net.items():
|
||||
if isinstance(v[0], list):
|
||||
for i in range(len(v)):
|
||||
for j in range(len(v[i])):
|
||||
v[i][j] -= lr * grads[k][i][j]
|
||||
else:
|
||||
for i in range(len(v)):
|
||||
v[i] -= lr * grads[k][i]
|
||||
|
||||
|
||||
def make_schedule(T):
|
||||
betas = [1e-4 + (0.02 - 1e-4) * t / (T - 1) for t in range(T)]
|
||||
alphas = [1 - b for b in betas]
|
||||
bars, cum = [], 1.0
|
||||
for a in alphas:
|
||||
cum *= a
|
||||
bars.append(cum)
|
||||
return alphas, bars
|
||||
|
||||
|
||||
def train_joint(net, alpha_bars, T, t_dim, steps, lr, rng):
|
||||
"""Joint sampling: denoiser sees all frames + their time positions simultaneously."""
|
||||
for step in range(steps):
|
||||
video = make_video(rng)
|
||||
t = rng.randrange(T)
|
||||
eps = [rng.gauss(0, 1) for _ in range(T_FRAMES)]
|
||||
a_bar = alpha_bars[t]
|
||||
noisy = [math.sqrt(a_bar) * video[i] + math.sqrt(1 - a_bar) * eps[i]
|
||||
for i in range(T_FRAMES)]
|
||||
patches = patchify_with_pos(noisy)
|
||||
x_flat = flatten(patches)
|
||||
t_emb = sin_embed(t, t_dim)
|
||||
out, cache = forward(x_flat, t_emb, net)
|
||||
grads = backward(eps, out, cache, net)
|
||||
apply(net, grads, lr)
|
||||
|
||||
|
||||
def sample_joint(net, alphas, alpha_bars, T, t_dim, rng):
|
||||
x = [rng.gauss(0, 1) for _ in range(T_FRAMES)]
|
||||
for t in range(T - 1, -1, -1):
|
||||
patches = patchify_with_pos(x)
|
||||
x_flat = flatten(patches)
|
||||
t_emb = sin_embed(t, t_dim)
|
||||
eps_hat, _ = forward(x_flat, t_emb, net)
|
||||
beta_t = 1 - alphas[t]
|
||||
new_x = [(x[i] - beta_t / math.sqrt(1 - alpha_bars[t]) * eps_hat[i]) / math.sqrt(alphas[t])
|
||||
for i in range(T_FRAMES)]
|
||||
if t > 0:
|
||||
x = [new_x[i] + math.sqrt(beta_t) * rng.gauss(0, 1) for i in range(T_FRAMES)]
|
||||
else:
|
||||
x = new_x
|
||||
return x
|
||||
|
||||
|
||||
def independent_per_frame(T_frames, rng):
|
||||
"""Baseline: sample each frame independently from a random walk."""
|
||||
return [rng.gauss(0, 1) + 0.3 * t for t in range(T_frames)]
|
||||
|
||||
|
||||
def frame_deltas(video):
|
||||
return [abs(video[i + 1] - video[i]) for i in range(len(video) - 1)]
|
||||
|
||||
|
||||
def main():
|
||||
rng = random.Random(21)
|
||||
T, t_dim, hidden = 40, 8, 48
|
||||
alphas, alpha_bars = make_schedule(T)
|
||||
net = init_net(T_FRAMES * (1 + POS_DIM) + t_dim, hidden, T_FRAMES, rng)
|
||||
|
||||
print(f"=== training joint video DDPM: {T_FRAMES} frames per clip ===")
|
||||
train_joint(net, alpha_bars, T, t_dim, steps=3000, lr=0.01, rng=rng)
|
||||
|
||||
print()
|
||||
print("=== 5 clips, joint sampling (coherent) ===")
|
||||
joint_deltas = []
|
||||
for i in range(5):
|
||||
clip = sample_joint(net, alphas, alpha_bars, T, t_dim, rng)
|
||||
deltas = frame_deltas(clip)
|
||||
joint_deltas.extend(deltas)
|
||||
print(f" clip {i}: " + " ".join(f"{v:+.2f}" for v in clip))
|
||||
|
||||
print()
|
||||
print("=== 5 clips, independent per-frame (flicker baseline) ===")
|
||||
indep_deltas = []
|
||||
for i in range(5):
|
||||
clip = independent_per_frame(T_FRAMES, rng)
|
||||
deltas = frame_deltas(clip)
|
||||
indep_deltas.extend(deltas)
|
||||
print(f" clip {i}: " + " ".join(f"{v:+.2f}" for v in clip))
|
||||
|
||||
avg_joint = sum(joint_deltas) / len(joint_deltas)
|
||||
avg_indep = sum(indep_deltas) / len(indep_deltas)
|
||||
print()
|
||||
print(f"avg frame-to-frame delta: joint={avg_joint:.2f} independent={avg_indep:.2f}")
|
||||
print("joint sampling produces smoother motion (smaller deltas).")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,154 @@
|
||||
# Video Generation
|
||||
|
||||
> An image is a 2-D tensor. A video is a 3-D one. The theory is the same; the compute is 10-100x harder. OpenAI's Sora (Feb 2024) proved it was possible. By 2026 Veo 2, Kling 1.5, Runway Gen-3, Pika 2.0, and WAN 2.2 ship production video from text at 1080p — and the open-weights stack (CogVideoX, HunyuanVideo, Mochi-1, WAN 2.2) is 12 months behind.
|
||||
|
||||
**Type:** Build
|
||||
**Languages:** Python
|
||||
**Prerequisites:** Phase 8 · 07 (Latent Diffusion), Phase 7 · 09 (ViT), Phase 8 · 06 (DDPM)
|
||||
**Time:** ~45 minutes
|
||||
|
||||
## The Problem
|
||||
|
||||
A 10-second 1080p video at 24fps is 240 frames of 1920×1080×3 pixels. That's ~1.5 GB of raw data per clip. Pixel-space diffusion is infeasible. You need:
|
||||
|
||||
1. **Spatiotemporal compression.** A VAE that encodes videos, not frames, into a sequence of spatial-temporal patches.
|
||||
2. **Temporal coherence.** Frames need to share content, lighting, and object identity over seconds. The net has to model motion.
|
||||
3. **Compute budget.** Video training is 10-100x more expensive than image for the same model size.
|
||||
4. **Conditioning.** Text, image (first-frame), audio, or another video. Most production models accept all four.
|
||||
|
||||
The architecture that solved this is the **Diffusion Transformer (DiT)** applied to spatiotemporal patches, trained on huge (prompt, caption, video) datasets. Same diffusion loss as Lesson 06.
|
||||
|
||||
## The Concept
|
||||
|
||||
|
||||
|
||||
### Patchify
|
||||
|
||||
Encode the video with a 3D VAE (learned spatiotemporal compression). The latent is shape `[T_latent, H_latent, W_latent, C_latent]`. Split into patches of size `[t_p, h_p, w_p]`. For Sora-style models, `t_p = 1` (per-frame patches) or `t_p = 2` (every two frames). A 10-second 1080p video compresses to ~20,000-100,000 patches.
|
||||
|
||||
### Spatiotemporal DiT
|
||||
|
||||
A transformer processes the flat sequence of patches. Each patch has a 3D positional embedding (time + y + x). Attention is usually factorized:
|
||||
|
||||
- **Spatial attention** within each frame's patches.
|
||||
- **Temporal attention** across frames at the same spatial location.
|
||||
- **Full 3D attention** is 16-100x more expensive; used only at low resolution or in research.
|
||||
|
||||
### Text conditioning
|
||||
|
||||
Cross-attention with a large text encoder (T5-XXL for Sora, CogVideoX-5B uses T5-XXL). Long prompts matter — Sora's training set had GPT-generated dense re-captions averaging 200 tokens per clip.
|
||||
|
||||
### Training
|
||||
|
||||
Standard diffusion loss (ε or v prediction) over spatiotemporal latents. Data: web video + ~100M curated clips + synthetic text captions. Compute: 10,000+ GPU hours for even a small research run; Sora-scale is 100,000+.
|
||||
|
||||
## The 2026 production landscape
|
||||
|
||||
| Model | Date | Max duration | Max res | Open weights? | Notable |
|
||||
|-------|------|--------------|---------|---------------|---------|
|
||||
| Sora (OpenAI) | 2024-02 | 60s | 1080p | No | First model to show world simulator properties at scale |
|
||||
| Sora Turbo | 2024-12 | 20s | 1080p | No | Production Sora at 5x faster inference |
|
||||
| Veo 2 (Google) | 2024-12 | 8s | 4K | No | Highest quality + physics in 2025 |
|
||||
| Veo 3 | 2025 Q3 | 15s | 4K | No | Native audio and stronger camera control |
|
||||
| Kling 1.5 / 2.1 (Kuaishou) | 2024-2025 | 10s | 1080p | No | Best human motion in 2025 Q1 |
|
||||
| Runway Gen-3 Alpha | 2024-06 | 10s | 768p | No | Professional video tools on top |
|
||||
| Pika 2.0 | 2024-10 | 5s | 1080p | No | Strongest character consistency |
|
||||
| CogVideoX (THUDM) | 2024 | 10s | 720p | Yes (2B, 5B) | First open 5B-scale video |
|
||||
| HunyuanVideo (Tencent) | 2024-12 | 5s | 720p | Yes (13B) | Open SOTA late 2024 |
|
||||
| Mochi-1 (Genmo) | 2024-10 | 5.4s | 480p | Yes (10B) | Most permissively licensed |
|
||||
| WAN 2.2 (Alibaba) | 2025-07 | 5s | 720p | Yes | Strongest open model mid-2025 |
|
||||
|
||||
Open weights are closing the gap faster than in the image space: HunyuanVideo + WAN 2.2 LoRAs already power most open-source workflows by mid-2026.
|
||||
|
||||
## Build It
|
||||
|
||||
`code/main.py` simulates the core spatiotemporal DiT idea: patchify a small synthetic video, add a per-patch position embedding, and denoise the whole sequence with a transformer-style attention over patches. No numpy; pure Python. We show that temporal coherence emerges even in 1-D when adjacent-frame patches share a denoiser and position embeddings.
|
||||
|
||||
### Step 1: patchify a synthetic 1-D "video"
|
||||
|
||||
```python
|
||||
def make_video(T_frames=8, rng=None):
|
||||
# a "video" is a sequence of 1-D values following a smooth trajectory
|
||||
base = rng.gauss(0, 1)
|
||||
return [base + 0.3 * t + rng.gauss(0, 0.1) for t in range(T_frames)]
|
||||
```
|
||||
|
||||
### Step 2: position embedding per frame
|
||||
|
||||
```python
|
||||
def pos_embed(t, dim):
|
||||
return sinusoidal(t, dim)
|
||||
```
|
||||
|
||||
### Step 3: denoiser sees the whole sequence
|
||||
|
||||
Instead of denoising each frame independently, our tiny net concatenates all frame values + their position embeddings and predicts the noise for all frames jointly.
|
||||
|
||||
### Step 4: temporal coherence test
|
||||
|
||||
After training, sample a video. Measure the frame-to-frame delta. If the model has learned temporal structure, the deltas stay smaller than sampling each frame independently.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Independent per-frame sampling = flicker.** If you run image diffusion on each frame separately, the output flickers because each frame's noise is independent. Video diffusion fixes this by coupling the frames through attention or shared noise.
|
||||
- **Naive 3D attention = OOM.** Full 3D attention on a 10-second 1080p latent is hundreds of billions of operations. Factorize into spatial + temporal.
|
||||
- **Data captioning matters more than size.** Sora's main upgrade over prior work was training on ~10x more detailed captions (GPT-4 re-labelled clips). OpenAI's technical report is explicit on this.
|
||||
- **First-frame conditioning.** Most production models also accept an image as the first frame. This is "image-to-video" mode; training includes this variant.
|
||||
- **Physics drift.** Long clips (>10s) accumulate subtle inconsistencies. Sliding-window generation + keyframe anchoring helps.
|
||||
|
||||
## Use It
|
||||
|
||||
| Use case | 2026 pick |
|
||||
|----------|-----------|
|
||||
| Highest-quality text-to-video, hosted | Veo 3 or Sora |
|
||||
| Camera-controlled cinematic | Runway Gen-3 with motion brushes |
|
||||
| Character consistency across clips | Pika 2.0 or Kling 2.1 |
|
||||
| Open weights, fast fine-tune | WAN 2.2 + LoRA |
|
||||
| Image-to-video | WAN 2.2-I2V, Kling 2.1 I2V, or Runway |
|
||||
| Audio-to-video lip sync | Veo 3 (native audio) or a dedicated lip-sync model |
|
||||
| Video editing | Runway Act-Two, Kling Motion Brush, Flux-Kontext (still-frame) |
|
||||
|
||||
Cost per second of video at quality parity has dropped 20x between 2024 and 2026.
|
||||
|
||||
## Ship It
|
||||
|
||||
Save `outputs/skill-video-brief.md`. Skill takes a video brief (duration, aspect ratio, style, camera plan, subject consistency, audio) and outputs: model + hosting, prompt scaffolding (camera language, subject description, motion descriptors), seed + reproducibility protocol, and a frame-level QA checklist.
|
||||
|
||||
## Exercises
|
||||
|
||||
1. **Easy.** In `code/main.py`, compare frame-to-frame delta for (a) independent per-frame sampling, (b) joint sequence sampling. Report the mean and variance of the deltas.
|
||||
2. **Medium.** Add a first-frame condition: pin frame 0 to a given value and sample the rest. Measure how the pinned value propagates.
|
||||
3. **Hard.** Use HuggingFace diffusers to run CogVideoX-2B on a local GPU. Time 20 inference steps at 720p for a 6-second clip. Profile the spatiotemporal attention to identify the bottleneck.
|
||||
|
||||
## Key Terms
|
||||
|
||||
| Term | What people say | What it actually means |
|
||||
|------|-----------------|-----------------------|
|
||||
| Video VAE | "3-D VAE" | Encoder that compresses `(T, H, W, C)` → spatiotemporal latent. |
|
||||
| Patches | "The tokens" | Fixed-size 3-D blocks of the latent; input to the DiT. |
|
||||
| Factorized attention | "Spatial + temporal" | Run attention over space, then over time; skip full 3-D attention. |
|
||||
| Image-to-video (I2V) | "Animate this photo" | Model takes an image + text, outputs a video that starts from it. |
|
||||
| Keyframe conditioning | "Anchor frames" | Pin specific frames to control the video's arc. |
|
||||
| Motion brush | "Directional hint" | UI input where the user paints motion vectors onto the image. |
|
||||
| Re-captioning | "Dense captions" | Using an LLM to re-label training clips with detailed prompts. |
|
||||
| Flicker | "Temporal artifact" | Frame-to-frame inconsistency; fixed with coupled denoising. |
|
||||
|
||||
## Production note: video latents are a memory-bandwidth problem
|
||||
|
||||
A 10-second 1080p clip at 24 fps is 240 frames × 1920 × 1080 × 3 ≈ 1.5 GB of raw pixels. After a 4× video VAE compression (`2 × spatial × 2 × temporal`) the latent is ~100 MB per request. Run this through a spatiotemporal DiT for 30 steps at batch 1 and you are moving ~3 GB/step through HBM — memory bandwidth, not FLOPs, is the bottleneck.
|
||||
|
||||
Three production knobs, all straight from production-inference literature inference chapter:
|
||||
|
||||
- **TP across the DiT.** Text-to-video models are routinely ≥10B params. TP=4 across 4 H100s is standard; PP=2 × TP=2 for 405B-class models. Latency per step drops roughly linearly with TP up to the all-reduce wall.
|
||||
- **Frame batching = continuous batching.** At generation time, video is conceptually a batch of frames linked by attention. Continuous batching (in-flight scheduling) applies: start rendering frame `t+1` while frame `t-1` is being returned, if the model architecture allows sliding-window generation.
|
||||
- **Clip-level prefill cache.** For image-to-video, the first-frame conditioning is analogous to an LLM's prompt prefill: compute it once, reuse across the temporal decoder passes. This is effectively a KV-cache for video.
|
||||
|
||||
## Further Reading
|
||||
|
||||
- [Brooks et al. (2024). Video generation models as world simulators](https://openai.com/index/video-generation-models-as-world-simulators/) — Sora technical report.
|
||||
- [Yang et al. (2024). CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://arxiv.org/abs/2408.06072) — CogVideoX.
|
||||
- [Kong et al. (2024). HunyuanVideo: A Systematic Framework for Large Video Generative Models](https://arxiv.org/abs/2412.03603) — HunyuanVideo.
|
||||
- [Genmo (2024). Mochi-1 Technical Report](https://www.genmo.ai/blog/mochi) — Mochi-1.
|
||||
- [Alibaba (2025). WAN 2.2](https://wanvideo.io/) — open SOTA mid-2025.
|
||||
- [Ho, Salimans, Gritsenko et al. (2022). Video Diffusion Models](https://arxiv.org/abs/2204.03458) — the seminal video diffusion paper.
|
||||
- [Blattmann et al. (2023). Align your Latents (Video LDM)](https://arxiv.org/abs/2304.08818) — Stable Video Diffusion's ancestor.
|
||||
@@ -0,0 +1,19 @@
|
||||
---
|
||||
name: video-brief
|
||||
description: Translate a video brief into a model + prompt + shot plan for a 2026 video generator.
|
||||
version: 1.0.0
|
||||
phase: 8
|
||||
lesson: 10
|
||||
tags: [video, diffusion, sora, veo, kling]
|
||||
---
|
||||
|
||||
Given a video brief (duration, aspect ratio, style, subject, camera plan, audio needs, fidelity bar, budget), output:
|
||||
|
||||
1. Model + hosting. Sora, Veo 3, Kling 2.1, Runway Gen-3, Pika 2.0, CogVideoX, HunyuanVideo, WAN 2.2, or Mochi-1. One-sentence reason tied to duration / quality / license.
|
||||
2. Prompt scaffolding. (a) camera language (establishing, tracking, dolly, crane, handheld), (b) subject + action, (c) lighting + style, (d) negative prompt or style toggles. Aim for 50-150 tokens for Sora, 20-60 for Runway.
|
||||
3. Shot plan. Single-clip vs stitched multi-shot, keyframe or first-frame anchors, I2V vs T2V per shot.
|
||||
4. Seed + reproducibility. Per-shot seed, version pin, tooling repo.
|
||||
5. QA checklist. Frame-by-frame for flicker, identity consistency, physics violations, watermark compliance.
|
||||
6. Audio. Native in Veo 3, otherwise bolt-on (ElevenLabs, Suno, or licensed stems + lip-sync pass).
|
||||
|
||||
Refuse to promise > 10s of continuous motion at 1080p on a free tier (Pika / Kling / Runway cap at 10s; longer runs are stitched). Refuse to generate likenesses of real people without a release. Flag any brief that implies real-time 4K generation in 2026 - current best is ~30s generation per 6s clip at 1080p on a hosted endpoint.
|
||||
@@ -0,0 +1,70 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 520" font-family="Georgia, 'Times New Roman', serif">
|
||||
<defs>
|
||||
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto">
|
||||
<path d="M0,0 L10,5 L0,10 z" fill="#1a1a1a"/>
|
||||
</marker>
|
||||
<style>
|
||||
.box { fill: #faf6ef; stroke: #1a1a1a; stroke-width: 1.5; }
|
||||
.hot { fill: #fff1d6; stroke: #c0392b; stroke-width: 1.5; }
|
||||
.cold { fill: #eaf4ff; stroke: #2c5f8c; stroke-width: 1.5; }
|
||||
.label { font-size: 14px; font-weight: 600; fill: #1a1a1a; }
|
||||
.content { font-size: 12px; fill: #333; }
|
||||
.mono { font-size: 12px; fill: #333; font-family: 'Menlo', monospace; }
|
||||
.caption { font-size: 11px; fill: #555; font-style: italic; }
|
||||
.title { font-size: 16px; font-weight: 700; fill: #1a1a1a; }
|
||||
</style>
|
||||
</defs>
|
||||
|
||||
<text x="450" y="28" text-anchor="middle" class="title">audio gen: codec tokens + transformer or diffusion</text>
|
||||
|
||||
<!-- waveform -->
|
||||
<rect x="30" y="70" width="140" height="60" class="box"/>
|
||||
<text x="100" y="98" text-anchor="middle" class="content">waveform</text>
|
||||
<text x="100" y="116" text-anchor="middle" class="caption">24 kHz, 1-D</text>
|
||||
|
||||
<line x1="170" y1="100" x2="210" y2="100" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
|
||||
<!-- codec encoder -->
|
||||
<rect x="210" y="70" width="160" height="60" class="cold"/>
|
||||
<text x="290" y="98" text-anchor="middle" class="content">codec encoder</text>
|
||||
<text x="290" y="116" text-anchor="middle" class="caption">Encodec / DAC / SoundStream</text>
|
||||
|
||||
<line x1="370" y1="100" x2="410" y2="100" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
|
||||
<!-- tokens -->
|
||||
<rect x="410" y="60" width="220" height="80" class="hot"/>
|
||||
<text x="520" y="85" text-anchor="middle" class="label">RVQ tokens</text>
|
||||
<text x="520" y="105" text-anchor="middle" class="mono">K × 75 Hz indices</text>
|
||||
<text x="520" y="125" text-anchor="middle" class="caption">8 codebooks typical</text>
|
||||
|
||||
<line x1="630" y1="100" x2="670" y2="100" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
|
||||
<rect x="670" y="70" width="180" height="60" class="cold"/>
|
||||
<text x="760" y="98" text-anchor="middle" class="content">codec decoder</text>
|
||||
<text x="760" y="116" text-anchor="middle" class="caption">tokens → wav</text>
|
||||
|
||||
<!-- two generators -->
|
||||
<text x="230" y="190" text-anchor="middle" class="label">token-AR path (MusicGen, VALL-E)</text>
|
||||
<rect x="40" y="200" width="380" height="130" class="cold"/>
|
||||
<text x="230" y="225" text-anchor="middle" class="content">decoder-only transformer</text>
|
||||
<text x="230" y="243" text-anchor="middle" class="mono">p(t_n | t_<n, text_prompt, voice_prompt)</text>
|
||||
<text x="230" y="265" text-anchor="middle" class="caption">streams naturally (~200 ms TTFB)</text>
|
||||
<text x="230" y="285" text-anchor="middle" class="caption">delayed-parallel: K offset streams</text>
|
||||
<text x="230" y="305" text-anchor="middle" class="caption">dominates speech in 2026</text>
|
||||
|
||||
<text x="670" y="190" text-anchor="middle" class="label">diffusion / flow path (Stable Audio, AudioLDM)</text>
|
||||
<rect x="480" y="200" width="380" height="130" class="cold"/>
|
||||
<text x="670" y="225" text-anchor="middle" class="content">DiT on audio latents</text>
|
||||
<text x="670" y="243" text-anchor="middle" class="mono">x_t → x_0 via flow matching</text>
|
||||
<text x="670" y="265" text-anchor="middle" class="caption">faster total time for long clips</text>
|
||||
<text x="670" y="285" text-anchor="middle" class="caption">cleaner for music at >=30 s</text>
|
||||
<text x="670" y="305" text-anchor="middle" class="caption">dominates music generation in 2026</text>
|
||||
|
||||
<!-- 2026 stack -->
|
||||
<rect x="30" y="360" width="830" height="140" class="box"/>
|
||||
<text x="445" y="385" text-anchor="middle" class="label">2026 production stack</text>
|
||||
<text x="445" y="410" text-anchor="middle" class="caption">TTS: ElevenLabs V3, OpenAI TTS, GPT-4o realtime, NaturalSpeech 3</text>
|
||||
<text x="445" y="430" text-anchor="middle" class="caption">Music: Suno v4, Udio, Stable Audio 2.5, MusicGen 3.3B</text>
|
||||
<text x="445" y="450" text-anchor="middle" class="caption">SFX: AudioCraft 2, ElevenLabs SFX, Stable Audio Open</text>
|
||||
<text x="445" y="475" text-anchor="middle" class="caption">Voice clone: XTTS v2 (open), ElevenLabs Pro (consent-verified)</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 4.3 KiB |
@@ -0,0 +1,99 @@
|
||||
import math
|
||||
import random
|
||||
|
||||
|
||||
VOCAB = 16
|
||||
NUM_STYLES = 2
|
||||
|
||||
|
||||
def make_tokens(style, length, rng):
|
||||
"""Synthetic 'audio token' sequences by style."""
|
||||
if style == 0: # alternating, speech-like
|
||||
return [(i + rng.randint(0, 1)) % VOCAB for i in range(length)]
|
||||
return [(i * 3 + rng.randint(0, 1)) % VOCAB for i in range(length)]
|
||||
|
||||
|
||||
def init_counts():
|
||||
return [[[1.0 for _ in range(VOCAB)] for _ in range(VOCAB)] for _ in range(NUM_STYLES)]
|
||||
|
||||
|
||||
def update_counts(counts, sequence, style):
|
||||
for i in range(len(sequence) - 1):
|
||||
counts[style][sequence[i]][sequence[i + 1]] += 1.0
|
||||
|
||||
|
||||
def probs(counts, style, prev_tok):
|
||||
row = counts[style][prev_tok]
|
||||
total = sum(row)
|
||||
return [x / total for x in row]
|
||||
|
||||
|
||||
def entropy(p):
|
||||
return -sum(pi * math.log(max(pi, 1e-10)) for pi in p)
|
||||
|
||||
|
||||
def sample_from(p, rng):
|
||||
r = rng.random()
|
||||
acc = 0.0
|
||||
for i, pi in enumerate(p):
|
||||
acc += pi
|
||||
if r <= acc:
|
||||
return i
|
||||
return len(p) - 1
|
||||
|
||||
|
||||
def generate(counts, style, start, length, rng, temperature=1.0):
|
||||
out = [start]
|
||||
for _ in range(length - 1):
|
||||
p = probs(counts, style, out[-1])
|
||||
if temperature != 1.0:
|
||||
p = [pi ** (1 / temperature) for pi in p]
|
||||
total = sum(p)
|
||||
p = [x / total for x in p]
|
||||
out.append(sample_from(p, rng))
|
||||
return out
|
||||
|
||||
|
||||
def main():
|
||||
rng = random.Random(42)
|
||||
counts = init_counts()
|
||||
|
||||
print("=== training codec-token bigram per style on 500 sequences each ===")
|
||||
for _ in range(500):
|
||||
for style in range(NUM_STYLES):
|
||||
seq = make_tokens(style, length=20, rng=rng)
|
||||
update_counts(counts, seq, style)
|
||||
|
||||
print()
|
||||
print("=== generate 20 tokens per style, start=0 ===")
|
||||
for style in range(NUM_STYLES):
|
||||
label = "speech-like (alternating)" if style == 0 else "music-like (ramp)"
|
||||
print(f"\nstyle {style}: {label}")
|
||||
for temp in [0.7, 1.0]:
|
||||
out = generate(counts, style, start=0, length=20, rng=rng, temperature=temp)
|
||||
print(f" temp {temp:.1f}: {out}")
|
||||
|
||||
print()
|
||||
print("=== entropy at each position for style 0 conditional on token 5 ===")
|
||||
p = probs(counts, 0, 5)
|
||||
top3 = sorted(range(VOCAB), key=lambda i: -p[i])[:3]
|
||||
print(f" p(next | style=0, prev=5): H = {entropy(p):.3f}")
|
||||
print(f" top-3: {[(i, round(p[i], 3)) for i in top3]}")
|
||||
|
||||
print()
|
||||
print("=== VALL-E-style prompt continuation ===")
|
||||
prompt = make_tokens(0, length=5, rng=rng)[:5]
|
||||
print(f" 3-second voice prompt (tokens): {prompt}")
|
||||
continuation = list(prompt)
|
||||
for _ in range(15):
|
||||
p = probs(counts, 0, continuation[-1])
|
||||
continuation.append(sample_from(p, rng))
|
||||
print(f" continuation: {continuation}")
|
||||
|
||||
print()
|
||||
print("takeaway: tokens + transformer = entire TTS / music generation substrate.")
|
||||
print(" RVQ of Encodec / DAC makes real audio fit in the same loop.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,144 @@
|
||||
# Audio Generation
|
||||
|
||||
> Audio is a 1-D signal at 16-48 kHz. A five-second clip is 80-240k samples. No transformer attends to that sequence directly. The solution for every production audio model in 2026 is the same: a neural codec (Encodec, SoundStream, DAC) compresses audio to discrete tokens at 50-75 Hz, and a transformer or diffusion model generates tokens.
|
||||
|
||||
**Type:** Build
|
||||
**Languages:** Python
|
||||
**Prerequisites:** Phase 6 · 02 (Audio Features), Phase 6 · 04 (ASR), Phase 8 · 06 (DDPM)
|
||||
**Time:** ~45 minutes
|
||||
|
||||
## The Problem
|
||||
|
||||
Three audio generation tasks:
|
||||
|
||||
1. **Text-to-speech.** Given text, produce speech. Clean speech is narrow-band and has strong phonetic structure — solved well by transformer-over-tokens. VALL-E (Microsoft), NaturalSpeech 3, ElevenLabs, OpenAI TTS.
|
||||
2. **Music generation.** Given a prompt (text, melody, chord progression, genre), produce music. Much broader distribution. MusicGen (Meta), Stable Audio 2.5, Suno v4, Udio, Riffusion.
|
||||
3. **Audio effects / sound design.** Given a prompt, produce ambient sound or Foley. AudioGen, AudioLDM 2, Stable Audio Open.
|
||||
|
||||
All three run on the same substrate: neural audio codec + token-AR or diffusion generator.
|
||||
|
||||
## The Concept
|
||||
|
||||
|
||||
|
||||
### Neural audio codecs
|
||||
|
||||
Encodec (Meta, 2022), SoundStream (Google, 2021), Descript Audio Codec (DAC, 2023). A convolutional encoder compresses waveform to a per-timestep vector; residual vector quantization (RVQ) converts each vector to a cascade of K codebook indices. Decoder reverses it. 24 kHz audio at 2 kbps using 8 RVQ codebooks at 75 Hz = 600 tokens/sec.
|
||||
|
||||
```
|
||||
waveform (16000 samples/sec)
|
||||
└─ encoder conv ─┐
|
||||
├─ RVQ layer 1 → indices at 75 Hz
|
||||
├─ RVQ layer 2 → indices at 75 Hz
|
||||
├─ ...
|
||||
└─ RVQ layer 8
|
||||
```
|
||||
|
||||
### Two generative paradigms on top
|
||||
|
||||
**Token-autoregressive.** Flatten RVQ tokens into a sequence, run a decoder-only transformer. MusicGen uses "delayed parallel" to emit K codebook streams in parallel with per-stream offsets. VALL-E generates speech tokens from a text prompt + 3-second voice sample.
|
||||
|
||||
**Latent diffusion.** Pack codec tokens as continuous latents or model them with categorical diffusion. Stable Audio 2.5 uses flow matching on continuous audio latents. AudioLDM 2 uses text-to-mel-to-audio diffusion.
|
||||
|
||||
The 2024-2026 trend: flow matching is winning for music (faster inference, cleaner samples) while token-AR still dominates speech because it is naturally causal and streams well.
|
||||
|
||||
## Production landscape
|
||||
|
||||
| System | Task | Backbone | Latency |
|
||||
|--------|------|----------|---------|
|
||||
| ElevenLabs V3 | TTS | Token-AR + neural vocoder | ~300ms first token |
|
||||
| OpenAI GPT-4o audio | Full-duplex speech | End-to-end multimodal AR | ~200ms |
|
||||
| NaturalSpeech 3 | TTS | Latent flow matching | Non-streaming |
|
||||
| Stable Audio 2.5 | Music / SFX | DiT + flow matching on audio latents | ~10s for 1-minute clip |
|
||||
| Suno v4 | Full songs | Undisclosed; token-AR suspected | ~30s per song |
|
||||
| Udio v1.5 | Full songs | Undisclosed | ~30s per song |
|
||||
| MusicGen 3.3B | Music | Token-AR on Encodec 32kHz | Real-time |
|
||||
| AudioCraft 2 | Music + SFX | Flow matching | ~5s for 5s clip |
|
||||
| Riffusion v2 | Music | Spectrogram diffusion | ~10s |
|
||||
|
||||
## Build It
|
||||
|
||||
`code/main.py` simulates the core idea: train a tiny next-token transformer on synthetic "audio token" sequences generated from two distinct "styles" (alternating low and high tokens for style A, monotonic ramp for style B). Condition on style and sample.
|
||||
|
||||
### Step 1: synthetic audio tokens
|
||||
|
||||
```python
|
||||
def make_tokens(style, length, vocab_size, rng):
|
||||
if style == 0: # "speech-like": alternating
|
||||
return [i % vocab_size for i in range(length)]
|
||||
# "music-like": ramp
|
||||
return [(i * 3) % vocab_size for i in range(length)]
|
||||
```
|
||||
|
||||
### Step 2: train a tiny token predictor
|
||||
|
||||
A bigram-style predictor conditioned on style. The point is the pattern: codec tokens → cross-entropy training → autoregressive sampling.
|
||||
|
||||
### Step 3: sample conditionally
|
||||
|
||||
Given the style token and a starting token, sample the next token from the predicted distribution. Continue for 20-40 tokens.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Codec quality caps output quality.** If the codec can't represent a sound faithfully, no amount of generator quality helps. DAC is the current open best.
|
||||
- **RVQ error accumulation.** Each RVQ layer models the residual of the previous. Errors on layer 1 propagate. Sampling with temperature 0 on higher layers helps.
|
||||
- **Musical structure.** 30 seconds of tokens is 20k+ tokens at 75 Hz. Hard for transformers. MusicGen uses sliding window + prompt continuation; Stable Audio uses shorter clips + crossfading.
|
||||
- **Artifacts at boundaries.** Crossfading between generated clips needs careful overlap-add.
|
||||
- **Clean-data appetite.** Music generators need tens of thousands of hours of licensed music. The Suno / Udio RIAA lawsuit (2024) brought this to the surface.
|
||||
- **Voice cloning ethics.** A 3-second sample plus a text prompt is enough for VALL-E / XTTS / ElevenLabs to clone a voice. Every production model needs abuse detection + opt-out lists.
|
||||
|
||||
## Use It
|
||||
|
||||
| Task | 2026 stack |
|
||||
|------|------------|
|
||||
| Commercial TTS | ElevenLabs, OpenAI TTS, or Azure Neural |
|
||||
| Voice cloning (consent-verified) | XTTS v2 (open) or ElevenLabs Pro |
|
||||
| Background music, fast | Stable Audio 2.5 API, Suno, or Udio |
|
||||
| Music with lyrics | Suno v4 or Udio v1.5 |
|
||||
| Sound effects / Foley | AudioCraft 2, ElevenLabs SFX, or Stable Audio Open |
|
||||
| Real-time voice agent | GPT-4o realtime or Gemini Live |
|
||||
| Open-weights music research | MusicGen 3.3B, Stable Audio Open 1.0, AudioLDM 2 |
|
||||
| Dubbing / translation | HeyGen, ElevenLabs Dubbing |
|
||||
|
||||
## Ship It
|
||||
|
||||
Save `outputs/skill-audio-brief.md`. Skill takes an audio brief (task, duration, style, voice, license) and outputs: model + hosting, prompt format (genre tags, style descriptors, structural markers), codec + generator + vocoder chain, seed protocol, and eval plan (MOS / CLAP score / CER for TTS / user A/B).
|
||||
|
||||
## Exercises
|
||||
|
||||
1. **Easy.** Run `code/main.py` and set style explicitly. Verify the generated sequences match the style's pattern.
|
||||
2. **Medium.** Add delayed parallel decoding: simulate 2 streams of tokens that must stay offset by 1 step. Train a joint predictor.
|
||||
3. **Hard.** Use HuggingFace transformers to run MusicGen-small locally. Generate a 10-second clip with three different prompts; A/B for style adherence.
|
||||
|
||||
## Key Terms
|
||||
|
||||
| Term | What people say | What it actually means |
|
||||
|------|-----------------|-----------------------|
|
||||
| Codec | "Neural compression" | Encoder / decoder for audio; typical output is 50-75 Hz tokens. |
|
||||
| RVQ | "Residual VQ" | Cascade of K quantizers; each models the residual of the previous. |
|
||||
| Token | "One codec symbol" | Discrete index into a codebook; 1024 or 2048 typical. |
|
||||
| Delayed parallel | "Offset codebooks" | Emit K token streams with staggered offsets to reduce sequence length. |
|
||||
| Flow matching | "The 2024 win for audio" | Straighter-path alternative to diffusion; faster sampling. |
|
||||
| Voice prompt | "3-second sample" | Speaker embedding or token prefix that steers the cloned voice. |
|
||||
| Mel spectrogram | "The visual" | Log-magnitude perceptual spectrogram; used by many TTS systems. |
|
||||
| Vocoder | "Mel to wave" | Neural component that converts mel spectrograms back to audio. |
|
||||
|
||||
## Production note: audio is a streaming problem
|
||||
|
||||
Audio is the one output modality users expect to arrive *as it is generated*, not all-at-once. In production terms this means TPOT matters (Time Per Output Token) because the user's listening speed is the target throughput — not their reading speed. For 16kHz audio tokenized at ~75 tokens/second (Encodec), the server must generate ≥75 tokens/sec per user to keep playback smooth.
|
||||
|
||||
Two architectural consequences:
|
||||
|
||||
- **Flow-matching audio models cannot stream trivially.** Stable Audio 2.5 and AudioCraft 2 render a fixed clip length in one pass. To stream, you chunk the clip and overlap boundaries — think sliding-window diffusion — adding 100-300ms of latency overhead vs a codec AR model.
|
||||
|
||||
If the product is "live voice chat" or "real-time music continuation", pick the codec AR path. If it is "render a 30-second clip on submit", flow-matching wins on quality and total latency.
|
||||
|
||||
## Further Reading
|
||||
|
||||
- [Défossez et al. (2022). Encodec: High Fidelity Neural Audio Compression](https://arxiv.org/abs/2210.13438) — the codec standard.
|
||||
- [Zeghidour et al. (2021). SoundStream](https://arxiv.org/abs/2107.03312) — the first widely used neural audio codec.
|
||||
- [Kumar et al. (2023). High-Fidelity Audio Compression with Improved RVQGAN (DAC)](https://arxiv.org/abs/2306.06546) — DAC.
|
||||
- [Wang et al. (2023). Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (VALL-E)](https://arxiv.org/abs/2301.02111) — VALL-E.
|
||||
- [Copet et al. (2023). Simple and Controllable Music Generation (MusicGen)](https://arxiv.org/abs/2306.05284) — MusicGen.
|
||||
- [Liu et al. (2023). AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining](https://arxiv.org/abs/2308.05734) — AudioLDM 2.
|
||||
- [Stability AI (2024). Stable Audio 2.5](https://stability.ai/news/introducing-stable-audio-2-5) — 2025 text-to-music with flow matching.
|
||||
@@ -0,0 +1,19 @@
|
||||
---
|
||||
name: audio-brief
|
||||
description: Translate an audio brief into a model + prompt + eval plan across TTS, music, and SFX.
|
||||
version: 1.0.0
|
||||
phase: 8
|
||||
lesson: 11
|
||||
tags: [audio, tts, music, sfx, codec]
|
||||
---
|
||||
|
||||
Given an audio brief (task: TTS / music / SFX / voice clone, duration, style, voice or genre, license constraints, real-time or offline, quality bar), output:
|
||||
|
||||
1. Model + hosting. ElevenLabs V3, OpenAI TTS, XTTS v2, Suno v4, Udio, Stable Audio 2.5, MusicGen 3.3B, AudioCraft 2, or GPT-4o realtime. One-sentence reason.
|
||||
2. Prompt format. TTS: text + voice prompt (3-10 s sample or voice ID) + emotion / pace tags. Music: genre + instrumentation + mood + BPM + structural markers. SFX: onomatopoeia + source + duration hint.
|
||||
3. Codec + generator + vocoder chain. Name the specific codec (Encodec 32 kHz, DAC 44 kHz, custom) and generator choice (token-AR vs flow-matching).
|
||||
4. Seed + reproducibility. Seed pin, version pin, prompt hash.
|
||||
5. Eval. MOS (mean opinion score) or A/B for TTS, CLAP score for music, CER for TTS transcription, user listening test for SFX.
|
||||
6. Guardrails. Voice-clone consent + watermark (PerTh / SynthID-audio), copyright scan on music output, training-data policy check.
|
||||
|
||||
Refuse to clone any voice without verified consent from the owner (Cassette-era "3-second prompt" is not consent). Refuse to ship music with unlicensed reference material. Flag any real-time target < 200 ms that does not use a streaming token-AR model - diffusion-based audio cannot meet sub-300 ms TTFB in 2026.
|
||||
@@ -0,0 +1,79 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 520" font-family="Georgia, 'Times New Roman', serif">
|
||||
<defs>
|
||||
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto">
|
||||
<path d="M0,0 L10,5 L0,10 z" fill="#1a1a1a"/>
|
||||
</marker>
|
||||
<style>
|
||||
.box { fill: #faf6ef; stroke: #1a1a1a; stroke-width: 1.5; }
|
||||
.hot { fill: #fff1d6; stroke: #c0392b; stroke-width: 1.5; }
|
||||
.cold { fill: #eaf4ff; stroke: #2c5f8c; stroke-width: 1.5; }
|
||||
.label { font-size: 14px; font-weight: 600; fill: #1a1a1a; }
|
||||
.content { font-size: 12px; fill: #333; }
|
||||
.mono { font-size: 12px; fill: #333; font-family: 'Menlo', monospace; }
|
||||
.caption { font-size: 11px; fill: #555; font-style: italic; }
|
||||
.title { font-size: 16px; font-weight: 700; fill: #1a1a1a; }
|
||||
</style>
|
||||
</defs>
|
||||
|
||||
<text x="450" y="28" text-anchor="middle" class="title">text / image -> 3D in 2026</text>
|
||||
|
||||
<!-- input -->
|
||||
<rect x="30" y="70" width="140" height="60" class="box"/>
|
||||
<text x="100" y="95" text-anchor="middle" class="content">prompt or image</text>
|
||||
<text x="100" y="115" text-anchor="middle" class="caption">text, 1 photo, or 3-16 photos</text>
|
||||
|
||||
<line x1="170" y1="100" x2="210" y2="100" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
|
||||
<!-- multi-view diffusion -->
|
||||
<rect x="210" y="70" width="220" height="60" class="cold"/>
|
||||
<text x="320" y="95" text-anchor="middle" class="content">multi-view diffusion</text>
|
||||
<text x="320" y="115" text-anchor="middle" class="caption">SV3D, CAT3D, MVDream, Zero123</text>
|
||||
|
||||
<line x1="430" y1="100" x2="470" y2="100" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
|
||||
<!-- views -->
|
||||
<g>
|
||||
<rect x="480" y="75" width="25" height="25" class="hot"/>
|
||||
<rect x="510" y="75" width="25" height="25" class="hot"/>
|
||||
<rect x="540" y="75" width="25" height="25" class="hot"/>
|
||||
<rect x="570" y="75" width="25" height="25" class="hot"/>
|
||||
<rect x="480" y="105" width="25" height="25" class="hot"/>
|
||||
<rect x="510" y="105" width="25" height="25" class="hot"/>
|
||||
<rect x="540" y="105" width="25" height="25" class="hot"/>
|
||||
<rect x="570" y="105" width="25" height="25" class="hot"/>
|
||||
</g>
|
||||
<text x="540" y="148" text-anchor="middle" class="caption">8 consistent views</text>
|
||||
|
||||
<line x1="605" y1="100" x2="650" y2="100" stroke="#1a1a1a" stroke-width="1.2" marker-end="url(#arrow)"/>
|
||||
|
||||
<!-- 3D fit -->
|
||||
<rect x="650" y="70" width="220" height="60" class="cold"/>
|
||||
<text x="760" y="95" text-anchor="middle" class="content">3D fit</text>
|
||||
<text x="760" y="115" text-anchor="middle" class="caption">Gaussian splat or mesh extract</text>
|
||||
|
||||
<!-- Gaussian splat example -->
|
||||
<text x="180" y="200" class="label">3D Gaussian Splatting (Kerbl 2023)</text>
|
||||
<rect x="40" y="215" width="380" height="160" class="box"/>
|
||||
<circle cx="120" cy="290" r="25" fill="#c0392b" opacity="0.6"/>
|
||||
<circle cx="160" cy="280" r="20" fill="#c0392b" opacity="0.5"/>
|
||||
<circle cx="190" cy="305" r="30" fill="#c0392b" opacity="0.5"/>
|
||||
<circle cx="230" cy="290" r="22" fill="#c0392b" opacity="0.6"/>
|
||||
<circle cx="275" cy="310" r="28" fill="#c0392b" opacity="0.5"/>
|
||||
<circle cx="320" cy="295" r="18" fill="#c0392b" opacity="0.7"/>
|
||||
<text x="230" y="355" text-anchor="middle" class="caption">~1M Gaussians per scene, differentiable render, 100 fps</text>
|
||||
|
||||
<!-- Direct text-to-mesh -->
|
||||
<text x="670" y="200" class="label">direct text/image to mesh</text>
|
||||
<rect x="480" y="215" width="380" height="160" class="box"/>
|
||||
<text x="670" y="248" text-anchor="middle" class="content">Meshy 4, Rodin Gen-1.5, Hunyuan3D 2.0</text>
|
||||
<text x="670" y="275" text-anchor="middle" class="caption">output: PBR mesh (albedo, roughness,</text>
|
||||
<text x="670" y="293" text-anchor="middle" class="caption">metallic, normal)</text>
|
||||
<text x="670" y="320" text-anchor="middle" class="caption">→ direct import into Unity, Unreal, Blender</text>
|
||||
<text x="670" y="340" text-anchor="middle" class="caption">30s - 60s per asset on hosted endpoints</text>
|
||||
|
||||
<!-- time progression -->
|
||||
<rect x="40" y="410" width="820" height="90" class="hot"/>
|
||||
<text x="450" y="435" text-anchor="middle" class="label">2022 → 2026 time per asset</text>
|
||||
<text x="450" y="458" text-anchor="middle" class="caption">DreamFusion 2022: 1 hour; LRM 2023: 5s; TripoSR 2024: 1s; Meshy 4 2025: 30s with PBR</text>
|
||||
<text x="450" y="480" text-anchor="middle" class="caption">NeRF → 3D-GS → direct generative mesh: quality up, time down 100x</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 4.5 KiB |
@@ -0,0 +1,105 @@
|
||||
import math
|
||||
import random
|
||||
|
||||
|
||||
SIZE = 12 # small image grid for speed
|
||||
|
||||
|
||||
def make_target(size):
|
||||
"""Target: a smooth bright blob in the upper-left, dimmer one in the lower-right."""
|
||||
target = [[0.0] * size for _ in range(size)]
|
||||
for y in range(size):
|
||||
for x in range(size):
|
||||
d1 = ((x - 3) ** 2 + (y - 3) ** 2) / 6.0
|
||||
d2 = ((x - 8) ** 2 + (y - 8) ** 2) / 8.0
|
||||
target[y][x] = math.exp(-d1) + 0.5 * math.exp(-d2)
|
||||
return target
|
||||
|
||||
|
||||
def init_gaussians(n, rng):
|
||||
return [{
|
||||
"pos": [rng.uniform(2, SIZE - 2), rng.uniform(2, SIZE - 2)],
|
||||
"sigma": rng.uniform(0.8, 2.5),
|
||||
"color": rng.uniform(0.2, 0.8),
|
||||
} for _ in range(n)]
|
||||
|
||||
|
||||
def gaussian_value(x, y, g):
|
||||
dx = x - g["pos"][0]
|
||||
dy = y - g["pos"][1]
|
||||
d2 = dx * dx + dy * dy
|
||||
return g["color"] * math.exp(-d2 / (2 * g["sigma"] ** 2))
|
||||
|
||||
|
||||
def render(gaussians):
|
||||
img = [[0.0] * SIZE for _ in range(SIZE)]
|
||||
for y in range(SIZE):
|
||||
for x in range(SIZE):
|
||||
for g in gaussians:
|
||||
img[y][x] += gaussian_value(x, y, g)
|
||||
return img
|
||||
|
||||
|
||||
def mse(a, b):
|
||||
total = 0.0
|
||||
for y in range(SIZE):
|
||||
for x in range(SIZE):
|
||||
total += (a[y][x] - b[y][x]) ** 2
|
||||
return total / (SIZE * SIZE)
|
||||
|
||||
|
||||
def finite_diff_step(gaussians, target, lr, eps=0.1):
|
||||
base = render(gaussians)
|
||||
base_loss = mse(base, target)
|
||||
for g in gaussians:
|
||||
for key in ("pos", "sigma", "color"):
|
||||
if isinstance(g[key], list):
|
||||
for i in range(len(g[key])):
|
||||
g[key][i] += eps
|
||||
up = mse(render(gaussians), target)
|
||||
g[key][i] -= eps
|
||||
grad = (up - base_loss) / eps
|
||||
g[key][i] -= lr * grad
|
||||
else:
|
||||
g[key] += eps
|
||||
up = mse(render(gaussians), target)
|
||||
g[key] -= eps
|
||||
grad = (up - base_loss) / eps
|
||||
g[key] -= lr * grad
|
||||
return base_loss
|
||||
|
||||
|
||||
def ascii_img(img, chars=" .:;+*#@"):
|
||||
peak = max(max(row) for row in img) or 1.0
|
||||
lines = []
|
||||
for row in img:
|
||||
line = "".join(chars[min(len(chars) - 1, int(v / peak * (len(chars) - 1)))]
|
||||
for v in row)
|
||||
lines.append(line)
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def main():
|
||||
rng = random.Random(23)
|
||||
target = make_target(SIZE)
|
||||
|
||||
print("=== target image ===")
|
||||
print(ascii_img(target))
|
||||
print()
|
||||
|
||||
for n in [2, 4, 8]:
|
||||
rng_local = random.Random(7 + n)
|
||||
gaussians = init_gaussians(n, rng_local)
|
||||
print(f"=== fit {n} Gaussians ===")
|
||||
for step in range(30):
|
||||
loss = finite_diff_step(gaussians, target, lr=0.5, eps=0.2)
|
||||
print(f"final loss (MSE): {loss:.4f}")
|
||||
print(ascii_img(render(gaussians)))
|
||||
print()
|
||||
|
||||
print("takeaway: a few differentiable Gaussians can approximate smooth targets.")
|
||||
print(" scale to 1M splats in 3D, render via alpha compositing = 3D-GS.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,163 @@
|
||||
# 3D Generation
|
||||
|
||||
> 3D is the modality where 2D-to-3D leverage is strongest. The 2023 breakthrough was 3D Gaussian Splatting. The 2024-2026 generative push layers multi-view diffusion + 3D reconstruction on top to produce objects and scenes from a single prompt or photo.
|
||||
|
||||
**Type:** Learn
|
||||
**Languages:** Python
|
||||
**Prerequisites:** Phase 4 (Vision), Phase 8 · 07 (Latent Diffusion)
|
||||
**Time:** ~45 minutes
|
||||
|
||||
## The Problem
|
||||
|
||||
3D content is painful:
|
||||
|
||||
- **Representation.** Meshes, point clouds, voxel grids, signed distance fields (SDFs), neural radiance fields (NeRFs), 3D Gaussians. Each has trade-offs.
|
||||
- **Data scarcity.** ImageNet has 14M images. The largest clean 3D dataset (Objaverse-XL, 2023) has ~10M objects, most low quality.
|
||||
- **Memory.** A 512³ voxel grid is 128M voxels; a useful scene NeRF needs 1M samples/ray. Generation is harder than reconstruction.
|
||||
- **Supervision.** For a 2D image you have the pixels. For 3D you usually have a handful of 2D views and have to lift to 3D.
|
||||
|
||||
The 2026 stack separates the two problems. First, generate *2D multi-view images* with a diffusion model. Second, fit a *3D representation* (usually Gaussian splatting) to those images.
|
||||
|
||||
## The Concept
|
||||
|
||||
|
||||
|
||||
### Representation: 3D Gaussian Splatting (Kerbl et al., 2023)
|
||||
|
||||
Represent a scene as a cloud of ~1M 3D Gaussians. Each has 59 parameters: position (3), covariance (6, or quaternion 4 + scale 3), opacity (1), spherical-harmonics color (48 at degree 3, 3 at degree 0).
|
||||
|
||||
Rendering = projection + alpha-compositing. Fast (~100 fps at 1080p on a 4090). Differentiable. Fit by gradient descent against ground-truth photos. A scene fits in 5-30 minutes on a consumer GPU.
|
||||
|
||||
Two 2023-2024 innovations on top:
|
||||
- **Generative Gaussian splats.** Models like LGM, LRM, InstantMesh predict a Gaussian cloud directly from one or a few images.
|
||||
- **4D Gaussian Splatting.** Gaussians with per-frame offsets for dynamic scenes.
|
||||
|
||||
### Multi-view diffusion
|
||||
|
||||
Fine-tune a pretrained image diffusion model to generate multiple consistent views of the same object from a text prompt or single image. Zero123 (Liu et al., 2023), MVDream (Shi et al., 2023), SV3D (Stability, 2024), CAT3D (Google, 2024). Usually output 4-16 views around the object, lifted to 3D via Gaussian splatting or NeRF.
|
||||
|
||||
### Text-to-3D pipelines
|
||||
|
||||
| Model | Input | Output | Time |
|
||||
|-------|-------|--------|------|
|
||||
| DreamFusion (2022) | text | NeRF via SDS | ~1 hour per asset |
|
||||
| Magic3D | text | mesh + texture | ~40 min |
|
||||
| Shap-E (OpenAI, 2023) | text | implicit 3D | ~1 min |
|
||||
| SJC / ProlificDreamer | text | NeRF / mesh | ~30 min |
|
||||
| LRM (Meta, 2023) | image | triplane | ~5 s |
|
||||
| InstantMesh (2024) | image | mesh | ~10 s |
|
||||
| SV3D (Stability, 2024) | image | novel views | ~2 min |
|
||||
| CAT3D (Google, 2024) | 1-64 images | 3D NeRF | ~1 min |
|
||||
| TripoSR (2024) | image | mesh | ~1 s |
|
||||
| Meshy 4 (2025) | text + image | PBR mesh | ~30 s |
|
||||
| Rodin Gen-1.5 (2025) | text + image | PBR mesh | ~60 s |
|
||||
| Tencent Hunyuan3D 2.0 (2025) | image | mesh | ~30 s |
|
||||
|
||||
2025-2026 direction: direct text-to-mesh models with PBR materials suitable for game engines. Multi-view diffusion intermediate step is still the best-performing recipe for general objects.
|
||||
|
||||
### NeRF (for context)
|
||||
|
||||
Neural Radiance Field (Mildenhall et al., 2020). A tiny MLP takes `(x, y, z, view direction)` and outputs `(color, density)`. Render by integrating along rays. Beats mesh-based novel-view synthesis in quality but is 100-1000x slower to render. Superseded by Gaussian splatting for most real-time use but still dominant in research.
|
||||
|
||||
## Build It
|
||||
|
||||
`code/main.py` implements a toy 2D "Gaussian splatting" fit: represent a synthetic target image (a smooth gradient) as a sum of 2D Gaussian splats. Optimize positions, colors, and covariances by gradient descent to match the target. You see the two core operations: forward render (splat + alpha-composite) and fit by gradient descent.
|
||||
|
||||
### Step 1: 2D Gaussian splat
|
||||
|
||||
```python
|
||||
def gaussian_at(x, y, gaussian):
|
||||
px, py = gaussian["pos"]
|
||||
sigma = gaussian["sigma"]
|
||||
d2 = (x - px) ** 2 + (y - py) ** 2
|
||||
return math.exp(-d2 / (2 * sigma * sigma))
|
||||
```
|
||||
|
||||
### Step 2: render by summing splats
|
||||
|
||||
```python
|
||||
def render(image_size, gaussians):
|
||||
img = [[0.0] * image_size for _ in range(image_size)]
|
||||
for g in gaussians:
|
||||
for y in range(image_size):
|
||||
for x in range(image_size):
|
||||
img[y][x] += g["color"] * gaussian_at(x, y, g)
|
||||
return img
|
||||
```
|
||||
|
||||
Real 3D Gaussian splatting sorts Gaussians by depth and alpha-composites in order. Our 2D toy just sums.
|
||||
|
||||
### Step 3: fit by gradient descent
|
||||
|
||||
```python
|
||||
for step in range(steps):
|
||||
pred = render(size, gaussians)
|
||||
loss = mse(pred, target)
|
||||
gradients = compute_grads(pred, target, gaussians)
|
||||
update(gaussians, gradients, lr)
|
||||
```
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **View inconsistency.** If you generate 4 views independently and they disagree about object structure, the 3D fit is blurry. Fix: multi-view diffusion with shared attention.
|
||||
- **Back-side hallucination.** Single-image → 3D has to invent the unseen side. Quality varies wildly.
|
||||
- **Gaussian splat explosion.** Unconstrained training grows to 10M splats and overfits. Densification + pruning heuristics (from 3D-GS original paper) are essential.
|
||||
- **Topology issues.** Meshes from implicit fields (SDFs) often have holes or self-intersections. Run a remesher (e.g. blender's voxel remesh) before shipping.
|
||||
- **License of training data.** Objaverse has mixed licenses; commercial use varies per model.
|
||||
|
||||
## Use It
|
||||
|
||||
| Task | 2026 pick |
|
||||
|------|-----------|
|
||||
| Scene reconstruction from photos | Gaussian splatting (3DGS, Gsplat, Scaniverse) |
|
||||
| Text-to-3D object for games | Meshy 4 or Rodin Gen-1.5 (PBR output) |
|
||||
| Image-to-3D | Hunyuan3D 2.0, TripoSR, InstantMesh |
|
||||
| Novel-view synthesis from few images | CAT3D, SV3D |
|
||||
| Dynamic scene reconstruction | 4D Gaussian Splatting |
|
||||
| Avatar / clothed human | Gaussian Avatar, HUGS |
|
||||
| Research / SOTA | Whatever dropped last week |
|
||||
|
||||
For shipping production 3D in a game or e-commerce pipeline: Meshy 4 or Rodin Gen-1.5 output PBR meshes that go straight into Unity / Unreal.
|
||||
|
||||
## Ship It
|
||||
|
||||
Save `outputs/skill-3d-pipeline.md`. Skill takes a 3D brief (input: text / one image / few images; output: mesh / splat / NeRF; usage: render / game / VR) and outputs: pipeline (multi-view diffusion + fit, or direct mesh model), base model, iteration budget, topology post-processing, material channels needed.
|
||||
|
||||
## Exercises
|
||||
|
||||
1. **Easy.** Run `code/main.py` with 4, 16, 64 Gaussians. Report final MSE vs target.
|
||||
2. **Medium.** Extend to color Gaussians (RGB). Confirm reconstruction matches the target color pattern.
|
||||
3. **Hard.** Using gsplat or Nerfstudio, reconstruct a real object from a 50-photo capture. Report fit time and final SSIM on held-out views.
|
||||
|
||||
## Key Terms
|
||||
|
||||
| Term | What people say | What it actually means |
|
||||
|------|-----------------|-----------------------|
|
||||
| 3D Gaussian Splatting | "3DGS" | Scene as a cloud of 3D Gaussians; differentiable alpha-composite render. |
|
||||
| NeRF | "Neural radiance field" | MLP that outputs color + density at a 3D point; render by ray integration. |
|
||||
| Triplane | "Three 2-D planes" | Factor 3D into three 2-D axis-aligned feature grids; cheaper than volumetric. |
|
||||
| SDS | "Score distillation sampling" | Train 3D model by using 2D-diffusion score as pseudo-gradient. |
|
||||
| Multi-view diffusion | "Many views at once" | Diffusion model that outputs a batch of consistent camera views. |
|
||||
| PBR | "Physically-based rendering" | Material with albedo, roughness, metallic, normal channels. |
|
||||
| Densification | "Grow splats" | 3DGS training heuristic: split / clone splats in high-gradient regions. |
|
||||
|
||||
## Production note: 3D has no shared substrate yet
|
||||
|
||||
Unlike image (latent diffusion + DiT) and video (spatiotemporal DiT), 3D has no single dominant runtime in 2026. The production decision tree forks on the representation:
|
||||
|
||||
- **NeRF / triplane.** Inference is ray-marching + an MLP forward per sample. A 512² render requires millions of MLP forwards. Batch the ray samples aggressively; SDPA/xformers applies.
|
||||
- **Multi-view diffusion + LRM reconstruction.** Two-stage pipeline. Stage 1 (multi-view DiT) is a diffusion server just like Lesson 07. Stage 2 (LRM transformer) is a one-shot forward pass over the views. The overall latency profile is "diffusion + one-shot" — pick per-stage serving primitives accordingly.
|
||||
- **SDS / DreamFusion.** Per-asset optimization, not inference. Build jobs, not request handlers.
|
||||
|
||||
For most 2026 products, the right answer is "run a multi-view diffusion model on request, reconstruct to 3DGS asynchronously, serve the 3DGS for real-time viewing". This splits the workload cleanly between a GPU-inference server (fast) and an offline optimizer (slow).
|
||||
|
||||
## Further Reading
|
||||
|
||||
- [Mildenhall et al. (2020). NeRF: Representing Scenes as Neural Radiance Fields](https://arxiv.org/abs/2003.08934) — NeRF.
|
||||
- [Kerbl et al. (2023). 3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://arxiv.org/abs/2308.04079) — 3DGS.
|
||||
- [Poole et al. (2022). DreamFusion: Text-to-3D using 2D Diffusion](https://arxiv.org/abs/2209.14988) — SDS.
|
||||
- [Liu et al. (2023). Zero-1-to-3: Zero-shot One Image to 3D Object](https://arxiv.org/abs/2303.11328) — Zero123.
|
||||
- [Shi et al. (2023). MVDream](https://arxiv.org/abs/2308.16512) — multi-view diffusion.
|
||||
- [Hong et al. (2023). LRM: Large Reconstruction Model for Single Image to 3D](https://arxiv.org/abs/2311.04400) — LRM.
|
||||
- [Gao et al. (2024). CAT3D: Create Anything in 3D with Multi-View Diffusion Models](https://arxiv.org/abs/2405.10314) — CAT3D.
|
||||
- [Stability AI (2024). Stable Video 3D (SV3D)](https://stability.ai/research/sv3d) — SV3D.
|
||||
@@ -0,0 +1,19 @@
|
||||
---
|
||||
name: 3d-pipeline
|
||||
description: Choose a 3D generation or reconstruction pipeline given input type, output format, and use case.
|
||||
version: 1.0.0
|
||||
phase: 8
|
||||
lesson: 12
|
||||
tags: [3d, gaussian-splatting, nerf, mesh]
|
||||
---
|
||||
|
||||
Given inputs (text prompt / one image / few images / photo capture / video), target output (mesh / Gaussian splat / NeRF / point cloud), and use case (real-time render, game engine, AR / VR, cinematic), output:
|
||||
|
||||
1. Pipeline. (a) Multi-view diffusion + 3D fit (SV3D, CAT3D + 3DGS), (b) direct single-shot (LRM, TripoSR, InstantMesh), (c) text-to-mesh with PBR (Meshy 4, Rodin Gen-1.5, Hunyuan3D 2.0), (d) photo capture + 3DGS (Gsplat, Postshot, Scaniverse).
|
||||
2. Base model + hosting. Named model + open / hosted. Include license relevance for commercial use.
|
||||
3. Iteration budget. Expected time to first output, iteration cost, refinement strategy.
|
||||
4. Topology + materials. Remesh pass needed? PBR channel requirements (albedo, roughness, metallic, normal)? UV layout automated or manual?
|
||||
5. Eval. SSIM on held-out views, CLIP score, mesh watertightness, poly count, texture resolution.
|
||||
6. Platform target. Unity / Unreal / Blender / web (three.js / Babylon) / AR (USDZ / glb).
|
||||
|
||||
Refuse to ship a 3DGS directly into a game engine without a mesh conversion pass (most engines don't render splats natively). Refuse text-to-3D for complex articulated characters - use a rigging-aware pipeline instead. Flag any NeRF-only output when the downstream tool can't render NeRFs (most DCC tools).
|
||||
@@ -0,0 +1,59 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 500" font-family="Georgia, 'Times New Roman', serif">
|
||||
<defs>
|
||||
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto">
|
||||
<path d="M0,0 L10,5 L0,10 z" fill="#1a1a1a"/>
|
||||
</marker>
|
||||
<style>
|
||||
.box { fill: #faf6ef; stroke: #1a1a1a; stroke-width: 1.5; }
|
||||
.hot { fill: #fff1d6; stroke: #c0392b; stroke-width: 1.5; }
|
||||
.cold { fill: #eaf4ff; stroke: #2c5f8c; stroke-width: 1.5; }
|
||||
.label { font-size: 14px; font-weight: 600; fill: #1a1a1a; }
|
||||
.content { font-size: 12px; fill: #333; }
|
||||
.mono { font-size: 12px; fill: #333; font-family: 'Menlo', monospace; }
|
||||
.caption { font-size: 11px; fill: #555; font-style: italic; }
|
||||
.title { font-size: 16px; font-weight: 700; fill: #1a1a1a; }
|
||||
</style>
|
||||
</defs>
|
||||
|
||||
<text x="450" y="28" text-anchor="middle" class="title">flow matching: train on a straight line</text>
|
||||
|
||||
<!-- DDPM vs FM comparison -->
|
||||
<text x="225" y="70" text-anchor="middle" class="label">DDPM: curved path</text>
|
||||
<rect x="50" y="90" width="350" height="170" class="box"/>
|
||||
|
||||
<!-- data cluster -->
|
||||
<circle cx="95" cy="210" r="18" fill="#c0392b" opacity="0.4"/>
|
||||
<text x="95" y="240" text-anchor="middle" class="caption">x_0 (data)</text>
|
||||
<!-- noise -->
|
||||
<circle cx="365" cy="130" r="18" fill="#2c5f8c" opacity="0.4"/>
|
||||
<text x="365" y="110" text-anchor="middle" class="caption">x_T ~ N(0, I)</text>
|
||||
|
||||
<path d="M 95,210 Q 160,160 200,200 T 280,150 T 365,130" fill="none" stroke="#1a1a1a" stroke-width="1.5"/>
|
||||
<text x="225" y="190" text-anchor="middle" class="caption">1000-step SDE</text>
|
||||
<text x="225" y="253" text-anchor="middle" class="caption">DDIM collapses to ~50 steps</text>
|
||||
|
||||
<text x="665" y="70" text-anchor="middle" class="label">flow matching: straight line</text>
|
||||
<rect x="490" y="90" width="380" height="170" class="box"/>
|
||||
|
||||
<circle cx="535" cy="210" r="18" fill="#c0392b" opacity="0.4"/>
|
||||
<text x="535" y="240" text-anchor="middle" class="caption">x_0</text>
|
||||
<circle cx="835" cy="130" r="18" fill="#2c5f8c" opacity="0.4"/>
|
||||
<text x="835" y="110" text-anchor="middle" class="caption">x_1 ~ N(0, I)</text>
|
||||
|
||||
<line x1="535" y1="210" x2="835" y2="130" stroke="#1a1a1a" stroke-width="2"/>
|
||||
<text x="685" y="190" text-anchor="middle" class="mono">x_t = t · x_1 + (1-t) · x_0</text>
|
||||
<text x="685" y="253" text-anchor="middle" class="caption">2-8 Euler steps at inference</text>
|
||||
|
||||
<!-- loss -->
|
||||
<rect x="50" y="290" width="820" height="70" class="hot"/>
|
||||
<text x="460" y="315" text-anchor="middle" class="label">loss = || v_θ(x_t, t) - (x_1 - x_0) ||²</text>
|
||||
<text x="460" y="340" text-anchor="middle" class="caption">simulation-free training; target is a constant vector along the straight line</text>
|
||||
|
||||
<!-- rectified flow -->
|
||||
<rect x="50" y="380" width="820" height="110" class="cold"/>
|
||||
<text x="460" y="402" text-anchor="middle" class="label">rectified flow: iteratively straighten</text>
|
||||
<text x="460" y="425" text-anchor="middle" class="caption">1. train v_1 with random (x_0, x_1) pairs</text>
|
||||
<text x="460" y="445" text-anchor="middle" class="caption">2. sample paired (x_0, x_1) by integrating v_1 -> pairs are ODE-matched</text>
|
||||
<text x="460" y="465" text-anchor="middle" class="caption">3. retrain v_2 on those pairs -> genuinely straighter flow</text>
|
||||
<text x="460" y="482" text-anchor="middle" class="caption">2 reflow iterations enable 1-step inference (SDXL-Turbo, SD3-Turbo, Flux-schnell)</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 3.5 KiB |
@@ -0,0 +1,148 @@
|
||||
import math
|
||||
import random
|
||||
|
||||
|
||||
def tanh(v):
|
||||
return [math.tanh(x) for x in v]
|
||||
|
||||
|
||||
def tanh_grad(h):
|
||||
return [1 - x * x for x in h]
|
||||
|
||||
|
||||
def matmul(W, x):
|
||||
return [sum(w * xi for w, xi in zip(row, x)) for row in W]
|
||||
|
||||
|
||||
def add(a, b):
|
||||
return [x + y for x, y in zip(a, b)]
|
||||
|
||||
|
||||
def randn_matrix(rows, cols, rng, scale=0.3):
|
||||
return [[rng.gauss(0, scale) for _ in range(cols)] for _ in range(rows)]
|
||||
|
||||
|
||||
def init_net(in_dim, hidden, out_dim, rng):
|
||||
return {
|
||||
"W1": randn_matrix(hidden, in_dim, rng),
|
||||
"b1": [0.0] * hidden,
|
||||
"W2": randn_matrix(hidden, hidden, rng),
|
||||
"b2": [0.0] * hidden,
|
||||
"W3": randn_matrix(out_dim, hidden, rng),
|
||||
"b3": [0.0] * out_dim,
|
||||
}
|
||||
|
||||
|
||||
def forward(x, t, net):
|
||||
inp = [x, t, t * t, math.sin(2 * math.pi * t), math.cos(2 * math.pi * t)]
|
||||
pre1 = add(matmul(net["W1"], inp), net["b1"])
|
||||
h1 = tanh(pre1)
|
||||
pre2 = add(matmul(net["W2"], h1), net["b2"])
|
||||
h2 = tanh(pre2)
|
||||
out = add(matmul(net["W3"], h2), net["b3"])
|
||||
return out[0], {"inp": inp, "h1": h1, "h2": h2}
|
||||
|
||||
|
||||
def backward(target, out, cache, net):
|
||||
grads = {k: None for k in net}
|
||||
for p in net:
|
||||
if isinstance(net[p][0], list):
|
||||
grads[p] = [[0.0] * len(net[p][0]) for _ in net[p]]
|
||||
else:
|
||||
grads[p] = [0.0] * len(net[p])
|
||||
d_out = 2 * (out - target)
|
||||
grads["b3"][0] += d_out
|
||||
for j in range(len(cache["h2"])):
|
||||
grads["W3"][0][j] += d_out * cache["h2"][j]
|
||||
d_h2 = [net["W3"][0][j] * d_out for j in range(len(cache["h2"]))]
|
||||
d_pre2 = [d_h2[j] * tanh_grad(cache["h2"])[j] for j in range(len(cache["h2"]))]
|
||||
for j in range(len(cache["h2"])):
|
||||
grads["b2"][j] += d_pre2[j]
|
||||
for k in range(len(cache["h1"])):
|
||||
grads["W2"][j][k] += d_pre2[j] * cache["h1"][k]
|
||||
d_h1 = [sum(net["W2"][j][k] * d_pre2[j] for j in range(len(cache["h2"])))
|
||||
for k in range(len(cache["h1"]))]
|
||||
d_pre1 = [d_h1[j] * tanh_grad(cache["h1"])[j] for j in range(len(cache["h1"]))]
|
||||
for j in range(len(cache["h1"])):
|
||||
grads["b1"][j] += d_pre1[j]
|
||||
for k in range(len(cache["inp"])):
|
||||
grads["W1"][j][k] += d_pre1[j] * cache["inp"][k]
|
||||
return grads
|
||||
|
||||
|
||||
def apply(net, grads, lr):
|
||||
for k, v in net.items():
|
||||
if isinstance(v[0], list):
|
||||
for i in range(len(v)):
|
||||
for j in range(len(v[i])):
|
||||
v[i][j] -= lr * grads[k][i][j]
|
||||
else:
|
||||
for i in range(len(v)):
|
||||
v[i] -= lr * grads[k][i]
|
||||
|
||||
|
||||
def sample_data(rng):
|
||||
return rng.gauss(-2.0, 0.3) if rng.random() < 0.5 else rng.gauss(2.0, 0.3)
|
||||
|
||||
|
||||
def train(net, steps, lr, rng):
|
||||
for _ in range(steps):
|
||||
x0 = sample_data(rng)
|
||||
x1 = rng.gauss(0, 1)
|
||||
t = rng.random()
|
||||
x_t = t * x1 + (1 - t) * x0
|
||||
target = x1 - x0
|
||||
pred, cache = forward(x_t, t, net)
|
||||
grads = backward(target, pred, cache, net)
|
||||
apply(net, grads, lr)
|
||||
|
||||
|
||||
def sample(net, num_steps, rng):
|
||||
x = rng.gauss(0, 1)
|
||||
dt = 1.0 / num_steps
|
||||
for i in range(num_steps):
|
||||
t = 1.0 - i * dt
|
||||
v, _ = forward(x, t, net)
|
||||
x -= dt * v
|
||||
return x
|
||||
|
||||
|
||||
def histogram(samples, lo=-5.0, hi=5.0, bins=30):
|
||||
width = (hi - lo) / bins
|
||||
counts = [0] * bins
|
||||
for s in samples:
|
||||
if lo <= s < hi:
|
||||
counts[int((s - lo) / width)] += 1
|
||||
peak = max(counts) or 1
|
||||
height = 6
|
||||
rows = []
|
||||
for r in range(height, 0, -1):
|
||||
thr = peak * r / height
|
||||
rows.append("".join("#" if c >= thr else " " for c in counts))
|
||||
rows.append("-" * bins)
|
||||
return "\n".join(rows)
|
||||
|
||||
|
||||
def main():
|
||||
rng = random.Random(31)
|
||||
net = init_net(in_dim=5, hidden=24, out_dim=1, rng=rng)
|
||||
|
||||
print("=== training flow matching on two-mode mixture ===")
|
||||
train(net, steps=6000, lr=0.01, rng=rng)
|
||||
|
||||
print()
|
||||
for num_steps in [1, 2, 4, 8, 20]:
|
||||
samples = [sample(net, num_steps, rng) for _ in range(500)]
|
||||
m = sum(samples) / len(samples)
|
||||
pos = sum(1 for s in samples if s > 0)
|
||||
print(f"=== {num_steps}-step Euler integration ===")
|
||||
print(histogram(samples))
|
||||
print(f"mean {m:+.2f}, left mode = {500 - pos}, right mode = {pos}")
|
||||
print()
|
||||
|
||||
print("takeaway: straight-line flow matching lets Euler work at 4-8 steps.")
|
||||
print(" DDPM needs 20+ for similar quality in this toy.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,176 @@
|
||||
# Flow Matching & Rectified Flows
|
||||
|
||||
> Diffusion models take 20-50 sampling steps because they walk a curved path from noise to data. Flow matching (Lipman et al., 2023) and rectified flow (Liu et al., 2022) trained straight paths. Straighter paths mean fewer steps mean faster inference. Stable Diffusion 3, Flux.1, and AudioCraft 2 all switched to flow matching in 2024.
|
||||
|
||||
**Type:** Build
|
||||
**Languages:** Python
|
||||
**Prerequisites:** Phase 8 · 06 (DDPM), Phase 1 · Calculus
|
||||
**Time:** ~45 minutes
|
||||
|
||||
## The Problem
|
||||
|
||||
DDPM's reverse process is a 1000-step stochastic walk from `N(0, I)` back to the data distribution. DDIM collapsed it to 20-50 deterministic steps. You want fewer steps — ideally one. The blocker is that the ODE solving the reverse process is stiff; the path is curved.
|
||||
|
||||
If you could train the model such that the path from noise to data was a *straight line*, a single Euler step from `t=1` to `t=0` would work. Flow matching builds this directly: define a straight-line interpolation from `x_1 ∼ N(0, I)` to `x_0 ∼ data`, train a vector field `v_θ(x, t)` to match its time derivative, integrate at inference.
|
||||
|
||||
Rectified flow (Liu 2022) goes further: iteratively straighten the paths with a reflow procedure that produces a progressively closer-to-linear ODE. After two reflow iterations, a 2-step sampler matches 50-step DDPM quality.
|
||||
|
||||
## The Concept
|
||||
|
||||
|
||||
|
||||
### Straight-line flow
|
||||
|
||||
Define:
|
||||
|
||||
```
|
||||
x_t = t · x_1 + (1 - t) · x_0, t ∈ [0, 1]
|
||||
```
|
||||
|
||||
where `x_0 ~ data` and `x_1 ~ N(0, I)`. The time derivative along this straight line is constant:
|
||||
|
||||
```
|
||||
dx_t / dt = x_1 - x_0
|
||||
```
|
||||
|
||||
Define a neural vector field `v_θ(x_t, t)` and train it to match this derivative:
|
||||
|
||||
```
|
||||
L = E_{x_0, x_1, t} || v_θ(x_t, t) - (x_1 - x_0) ||²
|
||||
```
|
||||
|
||||
This is the **conditional flow matching** loss (Lipman 2023). Training is simulation-free: you never unroll the ODE. Just sample `(x_0, x_1, t)` and regress.
|
||||
|
||||
### Sampling
|
||||
|
||||
At inference, integrate the learned vector field *backwards* in time:
|
||||
|
||||
```
|
||||
x_{t-Δt} = x_t - Δt · v_θ(x_t, t)
|
||||
```
|
||||
|
||||
Start at `x_1 ~ N(0, I)`, Euler-step down to `t=0`.
|
||||
|
||||
### Rectified flow (Liu 2022)
|
||||
|
||||
Straight-line flow works but the learned paths are *not actually straight* — they curve because many `x_0`s can map to the same `x_1`. Rectified flow's reflow step:
|
||||
|
||||
1. Train flow model v_1 with random pairings.
|
||||
2. Sample N pairs `(x_1, x_0)` by integrating v_1 from `x_1` to its landing `x_0`.
|
||||
3. Train v_2 on those paired examples. Because the pairs are now "ODE-matched", the straight-line interpolant between them is genuinely flatter.
|
||||
4. Repeat.
|
||||
|
||||
In practice 2 reflow iterations get you to near-linear, enabling 2-4 step inference. SDXL-Turbo, SD3-Turbo, LCM are all distilled-from-flow-matching models.
|
||||
|
||||
### Why this won for images in 2024
|
||||
|
||||
Three reasons:
|
||||
|
||||
1. **Simulation-free training** — no ODE unrolling during training, trivial to implement.
|
||||
2. **Better loss geometry** — straight paths have consistent signal-to-noise, whereas DDPM ε-loss has bad SNR at edges of the schedule.
|
||||
3. **Faster inference** — 4-8 steps at SDXL-Turbo quality; 1 step with consistency distillation.
|
||||
|
||||
## Flow matching vs DDPM — the exact connection
|
||||
|
||||
Flow matching with a Gaussian-conditional path is diffusion *with a specific noise schedule*. Pick the `x_t = α(t) x_0 + σ(t) x_1` schedule and flow matching recovers Stratonovich-reformulated diffusion with `v = α'·x_0 - σ'·x_1`. The two are algebraically equivalent for Gaussian paths.
|
||||
|
||||
What flow matching added: the *clarity* of the target (a plain velocity), a cleaner loss, and the license to experiment with non-Gaussian interpolants.
|
||||
|
||||
## Build It
|
||||
|
||||
`code/main.py` implements 1-D flow matching on a two-mode Gaussian mixture. The vector field `v_θ(x, t)` is a tiny MLP trained with the straight-line target. At inference, integrate 1, 2, 4, and 20 Euler steps and compare sample quality.
|
||||
|
||||
### Step 1: training loss
|
||||
|
||||
```python
|
||||
def train_step(x0, net, rng, lr):
|
||||
x1 = rng.gauss(0, 1)
|
||||
t = rng.random()
|
||||
x_t = t * x1 + (1 - t) * x0
|
||||
target = x1 - x0
|
||||
pred = net_forward(x_t, t)
|
||||
loss = (pred - target) ** 2
|
||||
# backprop + update
|
||||
```
|
||||
|
||||
### Step 2: multi-step inference
|
||||
|
||||
```python
|
||||
def sample(net, num_steps):
|
||||
x = rng.gauss(0, 1)
|
||||
for i in range(num_steps):
|
||||
t = 1.0 - i / num_steps
|
||||
dt = 1.0 / num_steps
|
||||
x -= dt * net_forward(x, t)
|
||||
return x
|
||||
```
|
||||
|
||||
### Step 3: compare step counts
|
||||
|
||||
Expect the 4-step sampler to already match the 20-step quality — a big deal for latency.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Time parameterization.** Flow matching uses `t ∈ [0, 1]` with `t=0` at data, `t=1` at noise. DDPM uses `t ∈ [0, T]` with `t=0` at data, `t=T` at noise. Same direction, different scale. Papers get this wrong constantly.
|
||||
- **Schedule choice.** Rectified flow's straight line is "the" flow-matching schedule, but you can use cosine or logit-normal t-sampling (SD3 does this) for better scale coverage.
|
||||
- **Reflow cost.** Generating the paired dataset for reflow is a full inference pass per sample. Only do reflow when you really need 1-2 step inference.
|
||||
- **Classifier-free guidance still applies.** Just swap ε for v in the linear combination: `v_cfg = (1+w) v_cond - w v_uncond`.
|
||||
|
||||
## Use It
|
||||
|
||||
| Use case | 2026 stack |
|
||||
|----------|-----------|
|
||||
| Text-to-image, best quality | Flow matching: SD3, Flux.1-dev |
|
||||
| Text-to-image, 1-4 steps | Distilled flow matching: Flux.1-schnell, SD3-Turbo, SDXL-Turbo |
|
||||
| Real-time inference | Consistency distillation from a flow-matched base (LCM, PCM) |
|
||||
| Audio generation | Flow matching: Stable Audio 2.5, AudioCraft 2 |
|
||||
| Video generation | Flow matching mixed with diffusion (Sora, Veo, Stable Video) |
|
||||
| Science / physics (particle trajectories, molecules) | Flow matching + equivariant vector field |
|
||||
|
||||
Whenever a paper says "faster than diffusion" in 2025-2026, it is almost always flow matching + distillation.
|
||||
|
||||
## Ship It
|
||||
|
||||
Save `outputs/skill-fm-tuner.md`. Skill takes a diffusion-style model spec and converts it to a flow-matching training config: schedule choice, time sampling distribution (uniform / logit-normal), optimizer, reflow plan, target step count, eval protocol.
|
||||
|
||||
## Exercises
|
||||
|
||||
1. **Easy.** Run `code/main.py` and compare 1-step vs 20-step MSE vs the true data distribution.
|
||||
2. **Medium.** Switch from uniform `t` sampling to logit-normal (concentrates sampling at mid-t). Does the model quality improve?
|
||||
3. **Hard.** Implement one reflow iteration: generate paired (x_0, x_1) by integrating the first model, train a second model on the pairs, and compare 1-step sample quality.
|
||||
|
||||
## Key Terms
|
||||
|
||||
| Term | What people say | What it actually means |
|
||||
|------|-----------------|-----------------------|
|
||||
| Flow matching | "Straight-line diffusion" | Train `v_θ(x, t)` to match `x_1 - x_0` along an interpolant. |
|
||||
| Rectified flow | "Reflow" | Iterative procedure that straightens learned flows. |
|
||||
| Velocity field | "v_θ" | Output of the model — the direction to move `x_t`. |
|
||||
| Straight-line interpolant | "The path" | `x_t = (1-t)·x_0 + t·x_1`; trivial target derivative. |
|
||||
| Euler sampler | "1st order ODE solver" | Simplest integrator; works well when paths are straight. |
|
||||
| Logit-normal t | "SD3 sampling" | Concentrate `t` sampling toward mid-values where gradients are strongest. |
|
||||
| Consistency distillation | "1-step sampler" | Train a student to map any `x_t` directly to `x_0`. |
|
||||
| CFG with velocity | "v-CFG" | `v_cfg = (1+w) v_cond - w v_uncond`; same trick, new variable. |
|
||||
|
||||
## Production note: Flux.1-schnell is flow matching at its fastest
|
||||
|
||||
Flow matching's production win is Flux.1-schnell — a flow-matched DiT distilled to 1-4 inference steps while keeping Flux-dev-grade quality. Niels' "Run Flux on an 8GB machine" notebook is the reference deployment recipe: T5 + CLIP encode, quantized MMDiT denoise (in 4 steps for schnell vs 50 for dev), VAE decode. The cost accounting:
|
||||
|
||||
| Variant | Steps | Latency at 1024² on L4 | Total FLOPs (relative) |
|
||||
|---------|-------|------------------------|------------------------|
|
||||
| Flux.1-dev (raw) | 50 | ~15 s | 1.0× |
|
||||
| Flux.1-schnell | 4 | ~1.2 s | 0.08× (12× faster) |
|
||||
| SDXL-base | 30 | ~4 s | 0.25× |
|
||||
| SDXL-Lightning 2-step | 2 | ~0.3 s | 0.03× |
|
||||
|
||||
The production rule: **flow-matched base + distillation = the 2026 default for fast text-to-image.** Every major vendor ships this combo: SD3-Turbo (SD3 + flow + distillation), Flux-schnell (Flux-dev + rectified-flow straightening), CogView-4-Flash. Pure diffusion bases exist only for legacy checkpoints.
|
||||
|
||||
## Further Reading
|
||||
|
||||
- [Liu, Gong, Liu (2022). Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://arxiv.org/abs/2209.03003) — rectified flow.
|
||||
- [Lipman et al. (2023). Flow Matching for Generative Modeling](https://arxiv.org/abs/2210.02747) — flow matching.
|
||||
- [Esser et al. (2024). Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://arxiv.org/abs/2403.03206) — SD3, rectified flow at scale.
|
||||
- [Albergo, Vanden-Eijnden (2023). Stochastic Interpolants](https://arxiv.org/abs/2303.08797) — general framework that covers FM + diffusion.
|
||||
- [Song et al. (2023). Consistency Models](https://arxiv.org/abs/2303.01469) — 1-step distillation of diffusion / flow.
|
||||
- [Sauer et al. (2023). Adversarial Diffusion Distillation (SDXL-Turbo)](https://arxiv.org/abs/2311.17042) — turbo variant.
|
||||
- [Black Forest Labs (2024). Flux.1 models](https://blackforestlabs.ai/announcing-black-forest-labs/) — flow matching in production.
|
||||
@@ -0,0 +1,20 @@
|
||||
---
|
||||
name: fm-tuner
|
||||
description: Convert a diffusion training plan into a flow-matching / rectified-flow config.
|
||||
version: 1.0.0
|
||||
phase: 8
|
||||
lesson: 13
|
||||
tags: [flow-matching, rectified-flow, diffusion]
|
||||
---
|
||||
|
||||
Given a diffusion-style training plan (data, compute, schedule, target step count, quality bar), output a flow-matching equivalent:
|
||||
|
||||
1. Schedule + interpolant. Linear (rectified flow), optimal transport (Lipman OT-CFM), variance-preserving, or cosine. One-sentence reason.
|
||||
2. Time sampling. Uniform, logit-normal (SD3), or mode-weighted. Warn when uniform sampling at 1000 Hz wastes capacity at endpoints.
|
||||
3. Target. Velocity v = x_1 - x_0 (rectified flow) or alpha'(t)x_1 + sigma'(t)x_0 (CFM). State which.
|
||||
4. Optimizer + lr warmup. Include AdamW with beta2 = 0.95 for stability at transformer scale.
|
||||
5. Reflow plan. Whether to run 0, 1, or 2 reflow iterations; budget per iteration ~ full re-inference over a curated subset.
|
||||
6. Step counts. Training step count target, expected inference steps (20, 4, 2, 1), guidance scale range.
|
||||
7. Eval. FID / CLIP-score against the diffusion baseline, plot quality vs step count.
|
||||
|
||||
Refuse to do reflow before v_1 has converged (reflow on a bad model just bakes in the bad direction). Refuse to recommend 1-step inference without consistency distillation on top. Flag any flow-matching model that targets > 20 step inference - if you need that many steps, you wasted the reformulation.
|
||||
@@ -0,0 +1,64 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 520" font-family="Georgia, 'Times New Roman', serif">
|
||||
<defs>
|
||||
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto">
|
||||
<path d="M0,0 L10,5 L0,10 z" fill="#1a1a1a"/>
|
||||
</marker>
|
||||
<style>
|
||||
.box { fill: #faf6ef; stroke: #1a1a1a; stroke-width: 1.5; }
|
||||
.hot { fill: #fff1d6; stroke: #c0392b; stroke-width: 1.5; }
|
||||
.cold { fill: #eaf4ff; stroke: #2c5f8c; stroke-width: 1.5; }
|
||||
.label { font-size: 14px; font-weight: 600; fill: #1a1a1a; }
|
||||
.content { font-size: 12px; fill: #333; }
|
||||
.mono { font-size: 12px; fill: #333; font-family: 'Menlo', monospace; }
|
||||
.caption { font-size: 11px; fill: #555; font-style: italic; }
|
||||
.title { font-size: 16px; font-weight: 700; fill: #1a1a1a; }
|
||||
</style>
|
||||
</defs>
|
||||
|
||||
<text x="450" y="28" text-anchor="middle" class="title">three-pillar evaluation of generative models</text>
|
||||
|
||||
<!-- FID column -->
|
||||
<rect x="30" y="60" width="270" height="200" class="box"/>
|
||||
<text x="165" y="85" text-anchor="middle" class="label">FID (sample quality)</text>
|
||||
<text x="165" y="110" text-anchor="middle" class="mono">||μ_r - μ_g||² + Tr(Σ_r + Σ_g - 2√Σ_rΣ_g)</text>
|
||||
<text x="165" y="140" text-anchor="middle" class="caption">Fréchet distance between</text>
|
||||
<text x="165" y="158" text-anchor="middle" class="caption">Gaussian fits in Inception-v3 space</text>
|
||||
<text x="165" y="188" text-anchor="middle" class="caption">use N ≥ 10k or numbers lie</text>
|
||||
<text x="165" y="206" text-anchor="middle" class="caption">ImageNet-biased; use FD-DINO or</text>
|
||||
<text x="165" y="224" text-anchor="middle" class="caption">CMMD out of domain</text>
|
||||
<text x="165" y="252" text-anchor="middle" class="caption">lower = better</text>
|
||||
|
||||
<!-- CLIP column -->
|
||||
<rect x="315" y="60" width="270" height="200" class="box"/>
|
||||
<text x="450" y="85" text-anchor="middle" class="label">CLIP score (adherence)</text>
|
||||
<text x="450" y="110" text-anchor="middle" class="mono">cos( CLIP_image(x), CLIP_text(p) )</text>
|
||||
<text x="450" y="140" text-anchor="middle" class="caption">how well does output match prompt?</text>
|
||||
<text x="450" y="170" text-anchor="middle" class="caption">compositional failures slip through</text>
|
||||
<text x="450" y="188" text-anchor="middle" class="caption">short prompts score higher mechanically</text>
|
||||
<text x="450" y="206" text-anchor="middle" class="caption">CMMD + VQA tests cover CLIP blind spots</text>
|
||||
<text x="450" y="252" text-anchor="middle" class="caption">higher = better</text>
|
||||
|
||||
<!-- preference column -->
|
||||
<rect x="600" y="60" width="270" height="200" class="box"/>
|
||||
<text x="735" y="85" text-anchor="middle" class="label">human preference (truth)</text>
|
||||
<text x="735" y="110" text-anchor="middle" class="mono">Elo from pairwise wins</text>
|
||||
<text x="735" y="140" text-anchor="middle" class="caption">A vs B on same prompt</text>
|
||||
<text x="735" y="158" text-anchor="middle" class="caption">HPSv2 / ImageReward / PickScore</text>
|
||||
<text x="735" y="188" text-anchor="middle" class="caption">as automated proxies</text>
|
||||
<text x="735" y="206" text-anchor="middle" class="caption">Chatbot-Arena-style image arenas</text>
|
||||
<text x="735" y="252" text-anchor="middle" class="caption">higher win rate = better</text>
|
||||
|
||||
<!-- failure modes -->
|
||||
<rect x="30" y="285" width="840" height="100" class="hot"/>
|
||||
<text x="450" y="310" text-anchor="middle" class="label">each metric has a known game</text>
|
||||
<text x="450" y="333" text-anchor="middle" class="caption">FID: overfit Inception prior → low FID, same quality</text>
|
||||
<text x="450" y="353" text-anchor="middle" class="caption">CLIP: saturate with 'masterpiece, 4k' → inflated score</text>
|
||||
<text x="450" y="373" text-anchor="middle" class="caption">preference: prompt set overlap with training → rigged</text>
|
||||
|
||||
<!-- production report -->
|
||||
<rect x="30" y="410" width="840" height="100" class="cold"/>
|
||||
<text x="450" y="435" text-anchor="middle" class="label">production eval report</text>
|
||||
<text x="450" y="458" text-anchor="middle" class="caption">1. FID on 10-30k + 2. CLIP / CMMD on the same pool +</text>
|
||||
<text x="450" y="478" text-anchor="middle" class="caption">3. 200+ blind human-pair Elo + 4. qualitative failure audit on 50 outputs</text>
|
||||
<text x="450" y="498" text-anchor="middle" class="caption">one metric = marketing; four = evidence</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 4.4 KiB |
@@ -0,0 +1,146 @@
|
||||
import math
|
||||
import random
|
||||
|
||||
|
||||
def mean_vec(vectors):
|
||||
d = len(vectors[0])
|
||||
n = len(vectors)
|
||||
return [sum(v[i] for v in vectors) / n for i in range(d)]
|
||||
|
||||
|
||||
def covariance(vectors, mu):
|
||||
d = len(mu)
|
||||
n = len(vectors)
|
||||
cov = [[0.0] * d for _ in range(d)]
|
||||
for v in vectors:
|
||||
for i in range(d):
|
||||
for j in range(d):
|
||||
cov[i][j] += (v[i] - mu[i]) * (v[j] - mu[j])
|
||||
return [[cov[i][j] / max(n - 1, 1) for j in range(d)] for i in range(d)]
|
||||
|
||||
|
||||
def trace(M):
|
||||
return sum(M[i][i] for i in range(len(M)))
|
||||
|
||||
|
||||
def matmul(A, B):
|
||||
n = len(A)
|
||||
p = len(B[0])
|
||||
m = len(B)
|
||||
out = [[0.0] * p for _ in range(n)]
|
||||
for i in range(n):
|
||||
for k in range(m):
|
||||
for j in range(p):
|
||||
out[i][j] += A[i][k] * B[k][j]
|
||||
return out
|
||||
|
||||
|
||||
def jacobi_sqrt(M, iters=30):
|
||||
"""Matrix square root by Denman-Beavers iteration (stable for PSD M)."""
|
||||
n = len(M)
|
||||
Y = [row[:] for row in M]
|
||||
Z = [[1.0 if i == j else 0.0 for j in range(n)] for i in range(n)]
|
||||
for _ in range(iters):
|
||||
Y_inv = inverse(Y)
|
||||
Z_inv = inverse(Z)
|
||||
Y = [[(Y[i][j] + Z_inv[i][j]) / 2 for j in range(n)] for i in range(n)]
|
||||
Z = [[(Z[i][j] + Y_inv[i][j]) / 2 for j in range(n)] for i in range(n)]
|
||||
return Y
|
||||
|
||||
|
||||
def inverse(M):
|
||||
n = len(M)
|
||||
A = [row[:] + [1.0 if i == j else 0.0 for j in range(n)] for i, row in enumerate(M)]
|
||||
for col in range(n):
|
||||
pivot = col
|
||||
for r in range(col + 1, n):
|
||||
if abs(A[r][col]) > abs(A[pivot][col]):
|
||||
pivot = r
|
||||
A[col], A[pivot] = A[pivot], A[col]
|
||||
piv = A[col][col]
|
||||
if abs(piv) < 1e-12:
|
||||
piv = 1e-12
|
||||
for j in range(2 * n):
|
||||
A[col][j] /= piv
|
||||
for r in range(n):
|
||||
if r == col: continue
|
||||
factor = A[r][col]
|
||||
for j in range(2 * n):
|
||||
A[r][j] -= factor * A[col][j]
|
||||
return [row[n:] for row in A]
|
||||
|
||||
|
||||
def fid(real_features, gen_features):
|
||||
mu_r = mean_vec(real_features)
|
||||
mu_g = mean_vec(gen_features)
|
||||
cov_r = covariance(real_features, mu_r)
|
||||
cov_g = covariance(gen_features, mu_g)
|
||||
mean_sq = sum((a - b) ** 2 for a, b in zip(mu_r, mu_g))
|
||||
prod = matmul(cov_r, cov_g)
|
||||
sqrt_prod = jacobi_sqrt(prod)
|
||||
return mean_sq + trace(cov_r) + trace(cov_g) - 2 * trace(sqrt_prod)
|
||||
|
||||
|
||||
def clip_like(a, b):
|
||||
dot = sum(x * y for x, y in zip(a, b))
|
||||
na = math.sqrt(sum(x * x for x in a))
|
||||
nb = math.sqrt(sum(x * x for x in b))
|
||||
return dot / max(na * nb, 1e-8)
|
||||
|
||||
|
||||
def elo_update(r_a, r_b, winner, k=32):
|
||||
expected_a = 1 / (1 + 10 ** ((r_b - r_a) / 400))
|
||||
actual_a = 1.0 if winner == "a" else 0.0
|
||||
delta = k * (actual_a - expected_a)
|
||||
return r_a + delta, r_b - delta
|
||||
|
||||
|
||||
def make_features(center, n, d, rng, scale=0.4):
|
||||
return [[center + rng.gauss(0, scale) for _ in range(d)] for _ in range(n)]
|
||||
|
||||
|
||||
def main():
|
||||
rng = random.Random(29)
|
||||
d = 4
|
||||
|
||||
print("=== FID bias at small N ===")
|
||||
for n in [50, 200, 1000]:
|
||||
real = make_features(0.0, n, d, rng)
|
||||
gen = make_features(0.0, n, d, rng) # same distribution
|
||||
score = fid(real, gen)
|
||||
print(f" N={n:5d}: FID (identical distributions) = {score:.4f} (lower = more similar)")
|
||||
|
||||
print(" -> FID should be 0 for identical distributions but is biased up at small N")
|
||||
print()
|
||||
|
||||
print("=== FID separates different distributions ===")
|
||||
real = make_features(0.0, 500, d, rng)
|
||||
for shift in [0.0, 0.2, 0.5, 1.0]:
|
||||
gen = make_features(shift, 500, d, rng)
|
||||
score = fid(real, gen)
|
||||
print(f" shift={shift:.1f}: FID = {score:.3f}")
|
||||
|
||||
print()
|
||||
print("=== CLIP-like cosine similarity ===")
|
||||
prompt = [1.0, 0.5, -0.2, 0.3]
|
||||
for image_center in [1.0, 0.5, 0.0, -0.5]:
|
||||
image = [image_center + rng.gauss(0, 0.1) for _ in range(d)]
|
||||
score = clip_like(image, prompt)
|
||||
print(f" image center {image_center:+.1f}: CLIP-like score = {score:+.3f}")
|
||||
|
||||
print()
|
||||
print("=== Elo from synthetic A/B preferences ===")
|
||||
r_a, r_b = 1000, 1000
|
||||
for i in range(200):
|
||||
# Suppose model A wins 70% of the time
|
||||
winner = "a" if rng.random() < 0.7 else "b"
|
||||
r_a, r_b = elo_update(r_a, r_b, winner)
|
||||
print(f" after 200 pairs (A wins 70%): r_A = {r_a:.0f}, r_B = {r_b:.0f}")
|
||||
|
||||
print()
|
||||
print("takeaway: FID is a distance; CLIP is an adherence score; Elo aggregates preferences.")
|
||||
print(" production evaluation uses all three plus qualitative failure audits.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,183 @@
|
||||
# Evaluation — FID, CLIP Score, Human Preference
|
||||
|
||||
> Every generative model leaderboard cites FID, CLIP score, and a win rate from a human-preference arena. Each number has a failure mode a determined researcher can game. If you do not know the failure modes, you cannot tell a real improvement from a gaming run.
|
||||
|
||||
**Type:** Build
|
||||
**Languages:** Python
|
||||
**Prerequisites:** Phase 8 · 01 (Taxonomy), Phase 2 · 04 (Evaluation Metrics)
|
||||
**Time:** ~45 minutes
|
||||
|
||||
## The Problem
|
||||
|
||||
A generative model is judged on *sample quality* and *conditioning adherence*. Neither has a closed-form measure. Your model has to render 10,000 images; something has to assign them numbers; you have to trust the numbers across model families, across resolutions, across architectures. Three metrics survived the 2014-2026 gauntlet:
|
||||
|
||||
- **FID (Fréchet Inception Distance).** A distance between two distributions — real and generated — in an Inception network's feature space. Lower is better.
|
||||
- **CLIP score.** Cosine similarity between a generated image's CLIP-image embedding and a prompt's CLIP-text embedding. Higher is better. Measures prompt adherence.
|
||||
- **Human preference.** Pit two models head-to-head on the same prompt, have humans (or a GPT-4-class model) pick the better one, aggregate to an Elo score.
|
||||
|
||||
You will also see: IS (inception score, largely retired), KID, CMMD, ImageReward, PickScore, HPSv2, MJHQ-30k. Each corrects for one failure of the previous.
|
||||
|
||||
## The Concept
|
||||
|
||||
|
||||
|
||||
### FID — sample quality
|
||||
|
||||
Heusel et al. (2017). Steps:
|
||||
|
||||
1. Extract Inception-v3 features (2048-D) for N real images and N generated.
|
||||
2. Fit a Gaussian to each pool: compute mean `μ_r, μ_g` and covariance `Σ_r, Σ_g`.
|
||||
3. FID = `||μ_r - μ_g||² + Tr(Σ_r + Σ_g - 2 · (Σ_r · Σ_g)^0.5)`.
|
||||
|
||||
Interpretation: Fréchet distance between two multivariate Gaussians in feature space. Lower = more similar distributions.
|
||||
|
||||
Failure modes:
|
||||
- **Biased on small N.** FID is mean-squared over the feature distribution — small N under-estimates covariance, gives falsely low FID. Always use N ≥ 10,000.
|
||||
- **Inception-dependent.** Inception-v3 was trained on ImageNet. Domains far from ImageNet (faces, art, text images) produce meaningless FID. Use a domain-specific feature extractor.
|
||||
- **Gaming.** Overfitting to the Inception prior gives low FID without visual quality improvement. Beat it with CMMD (below).
|
||||
|
||||
### CLIP score — prompt adherence
|
||||
|
||||
Radford et al. (2021). For a generated image + prompt:
|
||||
|
||||
```
|
||||
clip_score = cos_sim( CLIP_image(x_gen), CLIP_text(prompt) )
|
||||
```
|
||||
|
||||
Average across 30k generated images → a scalar comparable between models.
|
||||
|
||||
Failure modes:
|
||||
- **CLIP's own blind spots.** CLIP has weak compositional reasoning ("a red cube on a blue sphere" often fails). Models can rank well on CLIP score without really following complex prompts.
|
||||
- **Short prompt bias.** Short prompts have more CLIP-image matches in the wild. Longer prompts have lower CLIP scores mechanically.
|
||||
- **Prompt gaming.** Including "high quality, 4k, masterpiece" in the prompt inflates CLIP score without improving image-text binding.
|
||||
|
||||
CMMD (Jayasumana et al., 2024) fixes some of these: uses CLIP features instead of Inception, maximum-mean discrepancy instead of Fréchet. Better at detecting subtle quality differences.
|
||||
|
||||
### Human preference — the ground truth
|
||||
|
||||
Pick a pool of prompts. Generate with model A and model B. Show pairs to humans (or a strong LLM judge). Aggregate wins into an Elo or Bradley-Terry score. Benchmarks:
|
||||
|
||||
- **PartiPrompts (Google)**: 1,600 diverse prompts, 12 categories.
|
||||
- **HPSv2**: 107k human annotations, widely used as automated proxy.
|
||||
- **ImageReward**: 137k prompt-image preference pairs, MIT-licensed.
|
||||
- **PickScore**: trained on Pick-a-Pic 2.6M preferences.
|
||||
- **Chatbot-Arena-style image arenas**: https://imagearena.ai/ and others.
|
||||
|
||||
Failure modes:
|
||||
- **Judge variance.** Non-experts have different preferences than experts. Use both.
|
||||
- **Prompt distribution.** Cherry-picked prompts favor one family. Always document.
|
||||
- **LLM-judge reward hacking.** GPT-4-judge gets fooled by pretty-but-wrong outputs. Triangulate with human.
|
||||
|
||||
## Use together
|
||||
|
||||
A production eval report should include:
|
||||
|
||||
1. FID on 10-30k samples against a held-out real distribution (sample quality).
|
||||
2. CLIP score / CMMD on the same samples vs their prompts (adherence).
|
||||
3. Win rate in a blinded arena vs the previous model (overall preference).
|
||||
4. Failure mode analysis: 50 randomly sampled outputs, flagged for known issues (hand anatomy, text rendering, consistent object count).
|
||||
|
||||
Any single metric is a lie. Three corroborating metrics + qualitative review are a claim.
|
||||
|
||||
## Build It
|
||||
|
||||
`code/main.py` implements FID, CLIP-score-like, and Elo aggregation on synthetic "feature vectors" (we use 4-D vectors as stand-ins for Inception features). You see:
|
||||
|
||||
- FID computation on a small N and on a large N — the bias.
|
||||
- "CLIP score" as cosine similarity between feature pools.
|
||||
- Elo update rule from a synthetic preference stream.
|
||||
|
||||
### Step 1: FID in four lines
|
||||
|
||||
```python
|
||||
def fid(real_features, gen_features):
|
||||
mu_r, cov_r = mean_and_cov(real_features)
|
||||
mu_g, cov_g = mean_and_cov(gen_features)
|
||||
mean_diff = sum((a - b) ** 2 for a, b in zip(mu_r, mu_g))
|
||||
trace_term = trace(cov_r) + trace(cov_g) - 2 * sqrt_cov_product(cov_r, cov_g)
|
||||
return mean_diff + trace_term
|
||||
```
|
||||
|
||||
### Step 2: CLIP-style cosine-similarity
|
||||
|
||||
```python
|
||||
def clip_like(image_feat, text_feat):
|
||||
dot = sum(a * b for a, b in zip(image_feat, text_feat))
|
||||
norm = math.sqrt(dot_self(image_feat) * dot_self(text_feat))
|
||||
return dot / max(norm, 1e-8)
|
||||
```
|
||||
|
||||
### Step 3: Elo aggregation
|
||||
|
||||
```python
|
||||
def elo_update(r_a, r_b, winner, k=32):
|
||||
expected_a = 1 / (1 + 10 ** ((r_b - r_a) / 400))
|
||||
actual_a = 1.0 if winner == "a" else 0.0
|
||||
r_a_new = r_a + k * (actual_a - expected_a)
|
||||
r_b_new = r_b - k * (actual_a - expected_a)
|
||||
return r_a_new, r_b_new
|
||||
```
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **FID at N=1000.** Heuristic is unreliable under N=10k. Papers reporting low-N FID are gaming.
|
||||
- **Comparing FID across resolutions.** Inception's 299×299 resize changes the feature distribution. Compare at matched resolution only.
|
||||
- **Reporting one seed.** Run 3 seeds minimum. Report std.
|
||||
- **CLIP score inflation via negative prompts.** Some pipelines boost CLIP by over-fitting the prompt. Check for visual saturation.
|
||||
- **Elo bias from prompt overlap.** If both models saw a benchmark prompt during training, Elo is meaningless. Use held-out prompt sets.
|
||||
- **Human eval paid-crowd skew.** Prolific, MTurk annotators skew younger / tech-friendly. Mix with recruited art/design experts.
|
||||
|
||||
## Use It
|
||||
|
||||
Production eval protocol in 2026:
|
||||
|
||||
| Pillar | Minimum | Recommended |
|
||||
|--------|---------|-------------|
|
||||
| Sample quality | FID on 10k vs held-out real | + CMMD on 5k + FID on subset per category |
|
||||
| Prompt adherence | CLIP score on 30k | + HPSv2 + ImageReward + VQA-style question answering |
|
||||
| Preference | 200 blinded pairs vs baseline | + 2000 paired human + LLM-judge + Chatbot Arena |
|
||||
| Failure analysis | 50 hand-flagged | 500 hand-flagged + automated safety classifier |
|
||||
|
||||
All four pillars in one report = claim. Any one alone = marketing.
|
||||
|
||||
## Ship It
|
||||
|
||||
Save `outputs/skill-eval-report.md`. Skill takes a new model checkpoint + baseline and outputs a full eval plan: sample sizes, metrics, failure-mode probes, sign-off criteria.
|
||||
|
||||
## Exercises
|
||||
|
||||
1. **Easy.** Run `code/main.py`. Compare FID at N=100 vs N=1000 on the same synthetic distributions. Report bias magnitude.
|
||||
2. **Medium.** Implement CMMD from synthetic CLIP-style features (see Jayasumana et al., 2024 for the formula). Compare sensitivity to quality differences vs FID.
|
||||
3. **Hard.** Replicate the HPSv2 setup: take 1000 image-prompt pairs from a subset of Pick-a-Pic, fine-tune a small CLIP-based scorer on the preferences, and measure its agreement with a held-out set.
|
||||
|
||||
## Key Terms
|
||||
|
||||
| Term | What people say | What it actually means |
|
||||
|------|-----------------|-----------------------|
|
||||
| FID | "Fréchet Inception Distance" | Fréchet distance of Gaussian fits to real vs gen Inception features. |
|
||||
| CLIP score | "Text-image similarity" | Cosine similarity between CLIP image and text embeddings. |
|
||||
| CMMD | "FID's replacement" | CLIP-feature MMD; less biased, no Gaussian assumption. |
|
||||
| IS | "Inception score" | Exp KL(p(y|x) || p(y)); correlates poorly on modern models, retired. |
|
||||
| HPSv2 / ImageReward / PickScore | "Learned preference proxies" | Small models trained on human preferences; used as automatic judges. |
|
||||
| Elo | "Chess rating" | Bradley-Terry aggregation of pairwise wins. |
|
||||
| PartiPrompts | "The benchmark prompt set" | 1,600 Google-curated prompts across 12 categories. |
|
||||
| FD-DINO | "Self-sup replacement" | FD using DINOv2 features; better for out-of-ImageNet domains. |
|
||||
|
||||
## Production note: evaluation is an inference workload too
|
||||
|
||||
Running FID on 10k samples means generating 10k images. For a 50-step SDXL base at 1024² on a single L4, that is ~11 hours of single-request inference. Evaluation budgets are real, and the framing is exactly the offline-inference scenario (maximize throughput, ignore TTFT):
|
||||
|
||||
- **Batch hard, forget latency.** Offline eval = static batching at the largest size that fits in memory. `pipe(...).images` with `num_images_per_prompt=8` on an 80GB H100 runs 4-6× faster wall-clock than single-request.
|
||||
- **Cache the real features.** The Inception (FID) or CLIP (CLIP-score, CMMD) feature extraction over the real reference set is run *once*, stored as a `.npz`. Do not recompute per eval.
|
||||
|
||||
For CI / regression gates: run FID + CLIP score on a 500-sample subset per PR (~30 min); run full 10k FID + HPSv2 + Elo nightly.
|
||||
|
||||
## Further Reading
|
||||
|
||||
- [Heusel et al. (2017). GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium (FID)](https://arxiv.org/abs/1706.08500) — FID paper.
|
||||
- [Jayasumana et al. (2024). Rethinking FID: Towards a Better Evaluation Metric for Image Generation (CMMD)](https://arxiv.org/abs/2401.09603) — CMMD.
|
||||
- [Radford et al. (2021). Learning Transferable Visual Models from Natural Language Supervision (CLIP)](https://arxiv.org/abs/2103.00020) — CLIP.
|
||||
- [Wu et al. (2023). HPSv2: A Comprehensive Human Preference Score](https://arxiv.org/abs/2306.09341) — HPSv2.
|
||||
- [Xu et al. (2023). ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation](https://arxiv.org/abs/2304.05977) — ImageReward.
|
||||
- [Yu et al. (2023). Scaling Autoregressive Models for Content-Rich Text-to-Image Generation (Parti + PartiPrompts)](https://arxiv.org/abs/2206.10789) — PartiPrompts.
|
||||
- [Stein et al. (2023). Exposing flaws of generative model evaluation metrics](https://arxiv.org/abs/2306.04675) — failure-mode survey.
|
||||
@@ -0,0 +1,19 @@
|
||||
---
|
||||
name: eval-report
|
||||
description: Plan a full generative-model evaluation: sample quality, adherence, preference, failure audit.
|
||||
version: 1.0.0
|
||||
phase: 8
|
||||
lesson: 14
|
||||
tags: [evaluation, fid, clip, elo]
|
||||
---
|
||||
|
||||
Given a new generative-model checkpoint, a reference baseline, and a modality (image / video / audio / 3D), output a full eval plan:
|
||||
|
||||
1. Sample quality. FID / FD-DINO / CMMD on 10-30k samples vs held-out real set. Matched resolution. Report 3-seed mean +/- std.
|
||||
2. Adherence. CLIP score / CMMD on prompt-image pairs. Include HPSv2 + ImageReward + PickScore for text-to-image. For video, add vision-language metrics (V-Eval). For audio, CLAP + MOS.
|
||||
3. Pairwise preference. Blinded A/B on 200-2000 prompts vs baseline. Human + LLM-judge + PartiPrompts coverage.
|
||||
4. Category breakdown. Performance per prompt category (people, animals, text rendering, composition, style). Flag regressions per category even if global metrics improve.
|
||||
5. Safety / misuse. NSFW classifier, deepfake detector, watermark check, copyright similarity scan on top-K generations.
|
||||
6. Sign-off. Explicit gate: FID within +5% of baseline OR >55% human win rate OR documented qualitative advantage. No single-metric claims.
|
||||
|
||||
Refuse to report FID at N < 5000. Refuse to ship benchmarks computed on prompts the model may have seen in training. Refuse to report only LLM-judge results without human cross-check. Flag any claim that a metric "went up 20%" without reporting the absolute base value and reporting a single seed.
|
||||
@@ -2,4 +2,23 @@
|
||||
|
||||
> Create images, video, audio, 3D, and more.
|
||||
|
||||
See [ROADMAP.md](../../ROADMAP.md) for the full lesson plan.
|
||||
14 lessons, ~14 hours total. Each lesson ships: a 180-230 line doc, a runnable stdlib Python demo, a diagram, and a named skill for your agent.
|
||||
|
||||
| # | Lesson | Time |
|
||||
|---|--------|------|
|
||||
| 01 | [Generative Models — Taxonomy & History](01-generative-models-taxonomy-history/) | ~45 min |
|
||||
| 02 | [Autoencoders & VAE](02-autoencoders-vae/) | ~75 min |
|
||||
| 03 | [GANs — Generator vs Discriminator](03-gans-generator-discriminator/) | ~75 min |
|
||||
| 04 | [Conditional GANs & Pix2Pix](04-conditional-gans-pix2pix/) | ~75 min |
|
||||
| 05 | [StyleGAN](05-stylegan/) | ~45 min |
|
||||
| 06 | [Diffusion Models — DDPM from Scratch](06-diffusion-ddpm-from-scratch/) | ~75 min |
|
||||
| 07 | [Latent Diffusion & Stable Diffusion](07-latent-diffusion-stable-diffusion/) | ~75 min |
|
||||
| 08 | [ControlNet, LoRA & Conditioning](08-controlnet-lora-conditioning/) | ~75 min |
|
||||
| 09 | [Inpainting, Outpainting & Editing](09-inpainting-outpainting-editing/) | ~75 min |
|
||||
| 10 | [Video Generation](10-video-generation/) | ~45 min |
|
||||
| 11 | [Audio Generation](11-audio-generation/) | ~45 min |
|
||||
| 12 | [3D Generation](12-3d-generation/) | ~45 min |
|
||||
| 13 | [Flow Matching & Rectified Flows](13-flow-matching-rectified-flows/) | ~45 min |
|
||||
| 14 | [Evaluation — FID, CLIP Score, Human Preference](14-evaluation-fid-clip-score/) | ~45 min |
|
||||
|
||||
See [ROADMAP.md](../../ROADMAP.md) for the full cross-phase plan.
|
||||
|
||||
+44
-30
@@ -1,5 +1,5 @@
|
||||
// Auto-generated by build.js — do not edit manually.
|
||||
// Last built: 2026-04-23T09:21:29.186Z
|
||||
// Last built: 2026-04-23T09:23:19.743Z
|
||||
|
||||
const PHASES = [
|
||||
{
|
||||
@@ -1140,92 +1140,106 @@ const PHASES = [
|
||||
{
|
||||
"id": 8,
|
||||
"name": "Generative AI",
|
||||
"status": "planned",
|
||||
"status": "complete",
|
||||
"desc": "Create images, video, audio, 3D, and more.",
|
||||
"lessons": [
|
||||
{
|
||||
"name": "Generative Models: Taxonomy & History",
|
||||
"status": "planned",
|
||||
"status": "complete",
|
||||
"type": "Learn",
|
||||
"lang": "—"
|
||||
"lang": "Python",
|
||||
"url": "https://github.com/rohitg00/ai-engineering-from-scratch/tree/main/phases/08-generative-ai/01-generative-models-taxonomy-history/"
|
||||
},
|
||||
{
|
||||
"name": "Autoencoders & VAE",
|
||||
"status": "planned",
|
||||
"status": "complete",
|
||||
"type": "Build",
|
||||
"lang": "Python"
|
||||
"lang": "Python",
|
||||
"url": "https://github.com/rohitg00/ai-engineering-from-scratch/tree/main/phases/08-generative-ai/02-autoencoders-vae/"
|
||||
},
|
||||
{
|
||||
"name": "GANs: Generator vs Discriminator",
|
||||
"status": "planned",
|
||||
"status": "complete",
|
||||
"type": "Build",
|
||||
"lang": "Python"
|
||||
"lang": "Python",
|
||||
"url": "https://github.com/rohitg00/ai-engineering-from-scratch/tree/main/phases/08-generative-ai/03-gans-generator-discriminator/"
|
||||
},
|
||||
{
|
||||
"name": "Conditional GANs & Pix2Pix",
|
||||
"status": "planned",
|
||||
"status": "complete",
|
||||
"type": "Build",
|
||||
"lang": "Python"
|
||||
"lang": "Python",
|
||||
"url": "https://github.com/rohitg00/ai-engineering-from-scratch/tree/main/phases/08-generative-ai/04-conditional-gans-pix2pix/"
|
||||
},
|
||||
{
|
||||
"name": "StyleGAN",
|
||||
"status": "planned",
|
||||
"status": "complete",
|
||||
"type": "Build",
|
||||
"lang": "Python"
|
||||
"lang": "Python",
|
||||
"url": "https://github.com/rohitg00/ai-engineering-from-scratch/tree/main/phases/08-generative-ai/05-stylegan/"
|
||||
},
|
||||
{
|
||||
"name": "Diffusion Models — DDPM from Scratch",
|
||||
"status": "planned",
|
||||
"status": "complete",
|
||||
"type": "Build",
|
||||
"lang": "Python"
|
||||
"lang": "Python",
|
||||
"url": "https://github.com/rohitg00/ai-engineering-from-scratch/tree/main/phases/08-generative-ai/06-diffusion-ddpm-from-scratch/"
|
||||
},
|
||||
{
|
||||
"name": "Latent Diffusion & Stable Diffusion",
|
||||
"status": "planned",
|
||||
"status": "complete",
|
||||
"type": "Build",
|
||||
"lang": "Python"
|
||||
"lang": "Python",
|
||||
"url": "https://github.com/rohitg00/ai-engineering-from-scratch/tree/main/phases/08-generative-ai/07-latent-diffusion-stable-diffusion/"
|
||||
},
|
||||
{
|
||||
"name": "ControlNet, LoRA & Conditioning",
|
||||
"status": "planned",
|
||||
"status": "complete",
|
||||
"type": "Build",
|
||||
"lang": "Python"
|
||||
"lang": "Python",
|
||||
"url": "https://github.com/rohitg00/ai-engineering-from-scratch/tree/main/phases/08-generative-ai/08-controlnet-lora-conditioning/"
|
||||
},
|
||||
{
|
||||
"name": "Inpainting, Outpainting & Editing",
|
||||
"status": "planned",
|
||||
"status": "complete",
|
||||
"type": "Build",
|
||||
"lang": "Python"
|
||||
"lang": "Python",
|
||||
"url": "https://github.com/rohitg00/ai-engineering-from-scratch/tree/main/phases/08-generative-ai/09-inpainting-outpainting-editing/"
|
||||
},
|
||||
{
|
||||
"name": "Video Generation",
|
||||
"status": "planned",
|
||||
"status": "complete",
|
||||
"type": "Build",
|
||||
"lang": "Python"
|
||||
"lang": "Python",
|
||||
"url": "https://github.com/rohitg00/ai-engineering-from-scratch/tree/main/phases/08-generative-ai/10-video-generation/"
|
||||
},
|
||||
{
|
||||
"name": "Audio Generation",
|
||||
"status": "planned",
|
||||
"status": "complete",
|
||||
"type": "Build",
|
||||
"lang": "Python"
|
||||
"lang": "Python",
|
||||
"url": "https://github.com/rohitg00/ai-engineering-from-scratch/tree/main/phases/08-generative-ai/11-audio-generation/"
|
||||
},
|
||||
{
|
||||
"name": "3D Generation",
|
||||
"status": "planned",
|
||||
"status": "complete",
|
||||
"type": "Build",
|
||||
"lang": "Python"
|
||||
"lang": "Python",
|
||||
"url": "https://github.com/rohitg00/ai-engineering-from-scratch/tree/main/phases/08-generative-ai/12-3d-generation/"
|
||||
},
|
||||
{
|
||||
"name": "Flow Matching & Rectified Flows",
|
||||
"status": "planned",
|
||||
"status": "complete",
|
||||
"type": "Build",
|
||||
"lang": "Python"
|
||||
"lang": "Python",
|
||||
"url": "https://github.com/rohitg00/ai-engineering-from-scratch/tree/main/phases/08-generative-ai/13-flow-matching-rectified-flows/"
|
||||
},
|
||||
{
|
||||
"name": "Evaluation: FID, CLIP Score",
|
||||
"status": "planned",
|
||||
"status": "complete",
|
||||
"type": "Build",
|
||||
"lang": "Python"
|
||||
"lang": "Python",
|
||||
"url": "https://github.com/rohitg00/ai-engineering-from-scratch/tree/main/phases/08-generative-ai/14-evaluation-fid-clip-score/"
|
||||
}
|
||||
]
|
||||
},
|
||||
|
||||
Reference in New Issue
Block a user