mirror of
https://github.com/NVIDIA/Model-Optimizer.git
synced 2026-10-02 03:14:52 +08:00
deployment skill: add SGLang cookbook cross-check (analog of recipes.vllm.ai) (#1685)
### What does this PR do? Type of change: documentation (agent skill) The `deployment` and `evaluation` skills cross-check vLLM launch commands against `recipes.vllm.ai`, but the SGLang reference had **no** equivalent authoritative recipe source. This adds the **SGLang cookbook** (`docs.sglang.io/cookbook`) — the direct SGLang analog of `recipes.vllm.ai`, using the same `(hw, variant, quant, strategy, nodes)` variant-tuple command generator — as the cross-check source for SGLang deployments. Changes (canonical `.agents/` source; `.claude/` are symlinks): - **`SKILL.md`**: cookbook cross-check note under the SGLang quick-start, mirroring the existing vLLM cu130/sm_103 block — covers the JS-rendered (Mintlify) page, the raw-markdown fallback in `sgl-project/sglang` `docs_new/`, the URL-fragment variant selection, and the SM120 (RTX PRO 6000) nightly-image gotcha. - **`references/sglang.md`**: a new authoritative-recipe section, a MoE/FP4 backend flag matrix (`flashinfer_mxfp4` vs `marlin`, `deepep` vs `megamoe`), and per-hardware notes (Blackwell / Hopper / RTX PRO 6000). ### Usage N/A — documentation only. Agents now cross-check SGLang launch flags against `docs.sglang.io/cookbook/<category>/<org>/<model>`, fetching the raw markdown at `github.com/sgl-project/sglang/blob/main/docs_new/cookbook/<category>/<org>/<model>.mdx` when the page is JS-rendered. ### Testing Markdown-only change to skill docs; no code paths affected. ### Before your PR is "*Ready for review*" - Is this change backward compatible?: ✅ - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: N/A - Did you write any new necessary tests?: N/A (docs only) - Did you update Changelog?: N/A (agent skill docs, not a library feature/API change) - Did you get Claude approval on this PR?: ❌ (pending) 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Added authoritative cookbook cross-check guidance for SGLang deployments and how to select model variants via recipe parameters. * Explained how to fetch raw cookbook content when rendered pages omit commands and which flag areas the cookbook governs. * Expanded backend/strategy guidance for MoE, dispatch modes, and runtime tuning (batching, request sizing, CUDA-graph). * Added hardware-specific notes including RTX PRO 6000 nightly-image requirement and processor-specific recommendations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Zhiyu Cheng <zhiyuc@nvidia.com>
This commit is contained in:
@@ -148,6 +148,18 @@ python -m sglang.launch_server \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
For NVFP4 checkpoints, use `--quantization modelopt_fp4`.
|
||||
|
||||
> **Cross-check SGLang launch flags via the SGLang cookbook** (the SGLang analog
|
||||
> of `recipes.vllm.ai`): `docs.sglang.io/cookbook/<category>/<org>/<model>` (e.g.
|
||||
> `.../autoregressive/DeepSeek/DeepSeek-V4`) — authoritative for parallelism, MoE
|
||||
> backends, strategy flags, Docker image, and min version. Select the variant via
|
||||
> the URL fragment `#hw=...&variant=...&quant=...&strategy=...&nodes=...`. The
|
||||
> page is **JS-rendered** — fetch the raw markdown at
|
||||
> `raw.githubusercontent.com/sgl-project/sglang/main/docs_new/cookbook/<category>/<org>/<model>.mdx`.
|
||||
> SM120 (RTX PRO 6000) needs the `lmsysorg/sglang:dev` nightly (`:latest` lacks
|
||||
> SM120). See `references/sglang.md` for the full backend/flag matrix.
|
||||
|
||||
#### TRT-LLM (direct)
|
||||
|
||||
```python
|
||||
|
||||
@@ -1,5 +1,28 @@
|
||||
# SGLang Deployment Reference
|
||||
|
||||
## Authoritative recipe source — the SGLang cookbook
|
||||
|
||||
For any non-trivial model (large MoE, multi-node, Blackwell FP4), **cross-check
|
||||
the launch command against the SGLang cookbook** before hand-rolling flags. It
|
||||
is the SGLang analog of `recipes.vllm.ai`: a verified command generator keyed on
|
||||
a `(hw, variant, quant, strategy, nodes)` tuple.
|
||||
|
||||
- **URL:** `docs.sglang.io/cookbook/<category>/<org>/<model>` — e.g.
|
||||
`docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V4`. Select the
|
||||
variant via the URL fragment, e.g.
|
||||
`#hw=b200&variant=flash&quant=fp4&strategy=low-latency&nodes=single`.
|
||||
- **JS-rendered (Mintlify).** A plain fetch usually misses the commands. Fetch
|
||||
the **raw markdown** instead:
|
||||
`raw.githubusercontent.com/sgl-project/sglang/main/docs_new/cookbook/<category>/<org>/<model>.mdx`
|
||||
(the URL path maps directly onto the directory tree). The old
|
||||
`sgl-project/sgl-cookbook` repo is archived — the live source is `docs_new/`
|
||||
in the main `sgl-project/sglang` repo.
|
||||
- **Authoritative for:** parallelism layout (`--tp` / `--ep`, plus
|
||||
multi-node/data-parallel settings), MoE backends,
|
||||
strategy-driven flags (MTP, CUDA-graph batch sizing), Docker image tag, and the
|
||||
minimum SGLang version for the chosen variant. Pull the layout/flags from the
|
||||
cookbook, then adapt to the GPUs you actually have.
|
||||
|
||||
## Requirements
|
||||
|
||||
- SGLang >= 0.4.10
|
||||
@@ -72,6 +95,35 @@ Reference: `examples/specdec_bench/specdec_bench/models/sglang.py`
|
||||
| `--cuda-graph-max-bs` | Max batch size for CUDA graphs |
|
||||
| `--attention-backend` | `flashinfer` (default) or `triton` |
|
||||
|
||||
## MoE / FP4 backend flags (Blackwell vs Hopper)
|
||||
|
||||
For large MoE models the cookbook recipe is authoritative — these are the flags
|
||||
it selects per hardware. Quantized DeepSeek-style checkpoints are FP4 experts +
|
||||
FP8 attention/dense.
|
||||
|
||||
| Flag | Use |
|
||||
|------|-----|
|
||||
| `--moe-runner-backend flashinfer_mxfp4` | Default FP4 MoE runner on Blackwell (SM100/SM103) |
|
||||
| `--moe-runner-backend marlin` | Hopper W4A16 FP4 MoE runner |
|
||||
| `--moe-a2a-backend deepep` | Default expert all-to-all backend |
|
||||
| `--moe-a2a-backend megamoe` | Blackwell-only, high-throughput strategy |
|
||||
| `--deepep-mode auto\|normal\|low_latency` | DeepEP dispatch mode |
|
||||
|
||||
Strategy (low-latency / balanced / high-throughput) tunes
|
||||
`--cuda-graph-max-bs`, `--max-running-requests`, and MTP (Multi-Token
|
||||
Prediction) draft steps/tokens — take these from the cookbook variant rather
|
||||
than guessing.
|
||||
|
||||
## Hardware notes
|
||||
|
||||
- **Blackwell B200/B300/GB200/GB300 (SM100/SM103):** use the
|
||||
`flashinfer_mxfp4` MoE runner; `megamoe` only for the high-throughput
|
||||
strategy.
|
||||
- **Hopper (H100/H200):** FP4 runs via Marlin W4A16 kernels. Converted FP8
|
||||
checkpoints (`sgl-project/DeepSeek-V4-*-FP8`) exist for richer parallelism.
|
||||
- **RTX PRO 6000 (SM120):** `:latest` does **not** support SM120 — use the
|
||||
`lmsysorg/sglang:dev` nightly image.
|
||||
|
||||
## Common Issues
|
||||
|
||||
| Issue | Fix |
|
||||
|
||||
Reference in New Issue
Block a user