Model-Optimizer/plugins/modelopt/skills/quant-recipe-search/references/recipe_iteration.md

# Recipe Iteration Reference

## Problem

Quantization recipe search is a constrained optimization loop:

- Improve the user's chosen objective: compute/throughput, memory/latency, or a
  custom score.
- Keep every benchmark within the accepted accuracy-loss threshold. Default:
  less than 1 percentage point versus the matching BF16/FP16 baseline.
- Keep accuracy, verbosity/token usage, runtime behavior, and active cost
  separate until the final decision.
- Treat generated checkpoints as candidates. Evaluation and comparison decide
  whether a candidate is useful.

Ask for missing objective, primary quantization family, and benchmark set before
planning candidates.

## Search Space

Write the search space before launching PTQ. A recipe is a combination of these
axes, not just a numeric format.

| Axis | Choices To Try | Notes |
| --- | --- | --- |
| Numeric format | FP8/W8A8, NVFP4/W4A4, W4A16 NVFP4, INT4/AWQ, mixed formats | Keep FP8/W8A8 as a near-lossless baseline unless it is the user's target. |
| Calibration/search algorithm | Max, MSE, GPTQ, AWQ, AutoQuant scoring, calibration data variants | Algorithm choice is independent from numeric format. |
| Selection method | Manual/heuristic, sensitivity-guided manual, AutoQuant, hybrid | Record how each candidate was selected. |
| Module family | Attention, MLP, MoE experts, routers/gates, embeddings, `lm_head`, adapters, vision encoders | Change one major family at a time for ablations. |
| Layer position | First/last transformer-layer counts or explicit ordinal ranges kept in BF16 | Treat as a controlled ablation, not a default. Start with first 3-4 and last 1-2 layers when evidence supports it. |
| Runtime constraints | Fused attention groups, fused MoE expert projections, backend-supported formats | Do not mix incompatible quantization inside a fused runtime group. |
| Calibration budget | Dataset mix, sample count, sequence length, batch size | Vary deliberately and record the budget. |

### Objective Axis

Use the user's objective to decide which candidates are worth testing.

- **Compute / throughput:** typical data-center target. Favor activation
  quantization such as NVFP4 or FP8 when the runtime has fast kernels.
- **Memory / latency:** typical edge or memory-pressure target. Favor W4A16 or
  weight-only recipes when they preserve accuracy. Prefer active bytes per
  forward/decode path over total checkpoint size for routed or sparse models.
- **Custom:** use the user-provided score, for example checkpoint size, latency
  at a fixed concurrency, or product-specific memory budget.

If the user needs multiple objectives, maintain separate tables or define an
explicit weighted score.

### Numeric Format Axis

Common starting families:

- FP8/W8A8: near-lossless baseline or explicit primary target.
- NVFP4/W4A4: low-bit candidate family when activation quantization is part of
  the target.
- W4A16 NVFP4: weight-only NVFP4 family for accuracy-preserving memory/latency
  searches.
- INT4/AWQ: weight-only low-bit family for low-batch memory/latency targets.
- Mixed formats: examples include NVFP4+FP8, W4A16 NVFP4 with FP8 attention, or
  model-specific recipe fragments.

KV-cache dtype, parser settings, token caps, and backend flags are runtime/eval
controls unless the user explicitly makes them part of the recipe objective.

### Calibration Algorithm Axis

Try calibration/search algorithms as independent recipe variants:

- Max calibration: fast baseline for many FP8/NVFP4 formats.
- MSE calibration: try when max calibration loses accuracy for low-bit weights
  or sensitive layers.
- GPTQ: try for weight-quantized candidates where correction cost is acceptable.
- AWQ: try for INT4 or other weight-only candidates.
- AutoQuant scoring: use KL-divergence or gradient-based scoring when available
  to rank layers/modules and produce sensitivity reports.

### Selection Method Axis

Choose which modules get which format by one of these methods:

- Manual/heuristic: use prior experience, module-family cost, and controlled
  ablations.
- Sensitivity-guided manual: generate or recover an AutoQuant sensitivity report,
  then protect sensitive families or quantize low-sensitivity high-cost families.
- AutoQuant: search per-layer/per-module selections under constraints such as
  target bits, active-cost objective, or allowed formats. When AutoQuant is
  available, include at least one AutoQuant-generated candidate in the portfolio
  so its trade-off can be compared against manual recipes.
- Hybrid: start from AutoQuant, then override known runtime constraints or
  known-sensitive fused groups manually.

### Layer Position Axis

Use positional exclusions when sensitivity results or model behavior suggest
that boundary transformer layers are unusually sensitive. Do not assume every
model benefits from them.

- Resolve ordinal positions from the model's transformer block sequence rather
  than assuming a model-specific module path.
- Start with controlled candidates that keep the first 3 or 4 layers, the last
  1 or 2 layers, or both ranges in BF16. Include zero exclusion as the control.
- Change one boundary or count at a time when identifying the useful range.
- For manual candidates, exclude the resolved block paths in the recipe. For
  AutoQuant or hybrid candidates, pass the positional exclusions into the
  selected implementation when supported; otherwise apply them as a manual
  override after selection and record that constraint.
- Keep fused runtime groups internally compatible. Expand an exclusion to the
  complete fused group when the backend cannot mix precisions within it.
- Compare each excluded-layer candidate with the same BF16 baseline, benchmarks,
  and acceptance threshold used for the rest of the search.

## Design Workflow

1. Recover existing evidence:
   - Result tables, checkpoints, recipe logs, AutoQuant states, sensitivity
     reports, and active jobs.
   - Use `monitor`, `launching-evals`, and `compare-results` for execution
     state and metric provenance.

2. Define the target:
   - Objective, primary quantization family, benchmark set, accuracy-loss
     threshold, cost metric, and calibration budget.
   - Include scale storage and other quantization metadata in cost estimates.

3. Pick baselines:
   - BF16/FP16 baseline.
   - FP8/W8A8 near-lossless baseline, unless FP8 is the final target.
   - Existing production recipe if one exists.

4. Pick first candidates:
   - Start from `modelopt_recipes` when ModelOpt is available.
   - Prefer model-specific recipes, then general PTQ presets, then recipe
     fragments.
   - Add one AutoQuant candidate in the requested primary family when AutoQuant
     is available. Treat it as the expected best-search path, but validate it.
   - Add at least one manual or sensitivity-guided candidate for comparison and
     as a fallback if AutoQuant misses the benchmark frontier or produces a
     runtime-incompatible recipe.
   - Add positional BF16 exclusion candidates only when sensitivity or observed
     behavior warrants the ablation.

5. Generate and validate:
   - Delegate checkpoint generation and validation to `ptq`.
   - Check checkpoint coverage, quantization metadata, and expected module
     coverage.
   - Pipe-clean serving only after checkpoint validation passes.

6. Scale evaluation:
   - Run cheap screen benchmarks first.
   - Expand only candidates that pass screen evals and runtime gates.

## Runtime Fusion Rules

Search by module family, but respect modules fused by the target runtime.

- vLLM Qwen linear attention can fuse `linear_attn.in_proj_qkv` and
  `linear_attn.in_proj_z` into `linear_attn.in_proj_qkvz`; do not mix formats or
  algorithms across those shards unless the runtime supports it.
- Fused MoE kernels can couple expert projections such as gate/up (`w1`/`w3`, or
  equivalent names); treat each fused expert group as one recipe unit unless
  deployment confirms mixed formats are supported.
- Positional exclusions must preserve these grouping rules. If one boundary
  layer intersects a fused group, keep the whole group at a compatible
  precision or choose another boundary.
- If a checkpoint is valid but deployment fails due to missing support, classify
  it as checkpoint-quality, recipe/runtime compatibility, or deployment
  implementation. For deployment implementation, try small patches or flags via
  `deployment` / `debug` before rejecting the recipe.

## Iteration Loop

Use this loop after each candidate:

1. Update the portfolio table with recipe axes, active cost, checkpoint path,
   eval logs, accuracy, verbosity, and decision.
2. Compare against BF16/FP16 and FP8/W8A8 baselines.
3. If accuracy drops:
   - Protect sensitive module families.
   - Try MSE, GPTQ, or AWQ variants.
   - Use AutoQuant sensitivity to choose manual overrides.
   - Test first-layer, last-layer, or combined BF16 exclusion candidates when
     sensitivity or model behavior points to boundary layers.
4. If performance or active cost is insufficient:
   - Quantize the next high-cost active family.
   - Try a more aggressive format.
   - Revisit the active-cost objective or AutoQuant constraints.
5. If verbosity changes:
   - Inspect output samples and generation stats.
   - Verify parser, token cap, sampling, backend, and KV-cache settings did not
     change.
6. If results are close or noisy:
   - Rerun before labeling a benchmark regression.
7. If AutoQuant gives repeated recipes:
   - Check achieved bits and recipe hashes.
   - Adjust objective, allowed formats, or constraints before larger sweeps.
8. If AutoQuant underperforms manual recipes:
   - Compare the AutoQuant sensitivity report against manual ablation results.
   - Check whether AutoQuant protected high-active-cost modules, excluded the
     wrong families, optimized checkpoint size instead of active cost, or hit
     runtime-fusion constraints.
   - Keep the manual recipe in the table and use AutoQuant sensitivity to design
     the next hybrid/manual candidate.

Promote a recipe only when validated comparison shows it satisfies the user's
objective and benchmark threshold.

## Delegating To Existing Skills

Do not reimplement workflows that existing skills own:

| Need | Use |
| --- | --- |
| Generate/check a quantized checkpoint | `ptq` |
| Serve a checkpoint or test backend flags | `deployment` |
| Create or submit NEL configs | `evaluation` |
| Resume/debug/analyze live eval runs | `launching-evals` |
| Track active Slurm/NEL jobs | `monitor` |
| Fetch MLflow artifacts | `accessing-mlflow` |
| Compute baseline-vs-candidate deltas | `compare-results` |

Before launching PTQ in a ModelOpt repo, use the `ptq` skill; its current recipe
paths and validation gates are authoritative.

## ModelOpt Starting Points

When ModelOpt is available, start from `modelopt_recipes`:

1. Check model-specific recipes first, for example
   `modelopt_recipes/model_type/<model_family>/ptq/`.
2. Check general PTQ recipes and presets.
3. Use recipe fragments to build controlled manual variants.
4. Summarize include/exclude coverage before calibration. If a pattern misses the
   intended layer family, fix the recipe before launching.

Useful starting candidates:

- Compute/throughput: FP8/W8A8, NVFP4/W4A4, mixed NVFP4+FP8 with activation
  quantization.
- Memory/latency: W4A16 NVFP4, weight-only NVFP4, or W4A16 mixed with FP8 for
  sensitive modules.
- MoE: experts-only or MLP-only recipes, then expand based on sensitivity and
  active-routing cost.

## Candidate Record

For every candidate, record:

- Objective and acceptance threshold.
- Numeric formats and module-family coverage.
- First/last BF16 exclusion counts or ordinal ranges.
- Calibration/search algorithm and calibration data budget.
- Selection method: manual, sensitivity-guided, AutoQuant, or hybrid.
- Whether the candidate came from AutoQuant, manual ablation, or a hybrid
  override, so AutoQuant and manual trade-offs can be compared directly.
- Runtime fusion assumptions.
- Active bytes/token estimate including scales.
- Checkpoint path and eval/log paths.
- Accuracy and verbosity metrics.
- Decision and next action.