57 Commits
Author SHA1 Message Date
Michael 4250360dfc [AMD] ci: run ROCm CI on the ROCm 10 MI35X image (#4514) 2026-10-28 22:40:50 -07:00
HEJIAN SANGandClaude Opus 5.5 8855b832e4 Pass packed-document boundaries to HF attention so FSDP runs on FlashAttention-4 (#4316)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 11:29:44 -07:00
HEJIAN SANGandClaude Opus 5.5 47c0899eae [2/9] Add docker/Dockerfile.tpu: miles on TPU v7x from sglang-torchtpu's release image (#4268)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 10:17:43 -07:00
Zhichen Zeng 843938ba14 [3/3] ci: track training MFU in the metric history (#4098) 2026-10-27 00:36:28 -07:00
Jiajun Li b5f051f862 ci: rebalance eight-GPU tests from H200 to H100 (#4332) 2026-10-23 12:23:12 -07:00
Jiajun Li 37325a3d95 Install SGLang's radix_transfer_plugins in the CUDA image (#4329) 2026-10-23 11:04:15 -07:00
Xinyu JiangandZhiyao Jiang 1e4ad549a1 [AMD] ci: bump ROCm SGLang to v0.5.22 and expand MI350 nightly coverage (#4202)
Co-authored-by: Zhiyao Jiang <jessicajiang324@gmail.com>
2026-10-19 18:23:06 -07:00
Jiajun Li 0309247cea fix(ci): remove the default raw reward upper gate (#4160) 2026-10-17 20:55:01 -07:00
Jiajun Li 1647bd3b25 Bump sglang to v0.5.22 (#4125) 2026-10-16 17:43:49 -07:00
Jiajun Li ff9853c0b1 ci: gate performance benchmarks on est_time with a perf label (#4033) 2026-10-14 16:12:38 -07:00
Jiajun Li 64842375d6 fix(ci): make the default grad_norm gate a 4x sanity band (#4094) 2026-10-14 14:51:58 -07:00
Jiajun Li aa5dbf0b22 feat(ci): key metric history by metric and step (#4089) 2026-10-14 13:16:41 -07:00
Jiajun Li ddb6113545 ci: keep NVSHMEM on NVLink on every RoCE-only runner (#4088) 2026-10-14 12:56:34 -07:00
Jiajun Li 3f5c754c58 feat(ci): enforce metric history automatically after ten goldens (#4023) 2026-10-14 12:50:15 -07:00
Jiajun Li 2766ab0ac1 ci: move ten 4x H200 tests that fit on two GPUs to 2x H200 (#4053) 2026-10-13 23:31:46 -07:00
Jiajun Li 77309a60d9 ci: move slow CPU tests to tagged Stage B registrations (#4008) 2026-10-10 15:55:38 -07:00
Jiajun Li 9bdc3daaca ci: queue Hopper and ROCm shards without matrix throttling (#4006) 2026-10-10 14:26:22 -07:00
Jiajun Li ccc8009ade ci: split Hopper jobs to reduce failed-job rerun cost (#4005) 2026-10-10 02:41:27 -07:00
Jiajun Li 3a05ce560e fix(ci): gate slow Kubernetes and rollout CPU tests by domain (#3993) 2026-10-09 02:24:55 -07:00
Yueming Yuan dba9001cc6 ci: schedule B200 tests by minimum GPU count (#3983)
Schedule B200 tests with their minimum GPU count and dynamically lease free devices. Preserve test coverage, isolate concurrent jobs, and protect GPUs held by surviving job containers.

Bound admission waits with per-batch job budgets while preserving per-file execution timeouts.
2026-10-08 22:33:45 -07:00
Jiajun Li 6f94a01372 fix(ci): match H200 shards to runner capacity (#3990) 2026-10-08 19:25:54 -07:00
Zhichen Zeng 49b57dffea [1/8] refactor(kernels): move quant, MoE, activation, norm, HC and RoPE kernels to miles/kernels (#3605) 2026-10-08 12:57:20 -07:00
Kaixi Hou f81d52a40e Add support for Vera Rubin (#3700) 2026-10-08 11:07:31 -07:00
Jiajun LiandClaude Opus 5.5 49464b3cd1 ci: offload the 8x H200 suite to H100, B200 and 4x H200 (#3956)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 00:33:50 -07:00
Jiajun LiandClaude Opus 5.5 233a0cbe41 ci: write the rules for adding a CI test in .claude/rules (#3953)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 00:33:50 -07:00
Jiajun Li a0b2171b02 fix(ci): keep 4-GPU H200 DeepEP on NVLink for RoCE runners (#3955) 2026-10-07 22:47:11 -07:00
Jiajun Li 9350dda7ca feat(ci): rerun PR workflows when labels are removed (#3978) 2026-10-07 22:46:44 -07:00
Jiajun Li 1e1966293e fix(ci): cancel file tests and ROCm after PR merge (#3927) 2026-10-06 14:14:07 -07:00
Jiajun Li ae1d5fa60c fix(ci): skip GPU runners for empty test selections (#3902) 2026-10-06 13:36:48 -07:00
MichaelandXinyu Jiang 3d388872ac [AMD] docker: add rocm10-mi30x (gfx942) image variant (#3858)
Co-authored-by: Xinyu Jiang <xinyuj2@andrew.cmu.edu>
2026-10-05 15:49:56 -07:00
Yueming Yuan a095490625 ci: allow Megatron Bridge version overrides (#3897) 2026-10-02 19:37:28 -07:00
Jiajun LiandYueming Yuan 53f5d94224 build: retire CUDA 12 Docker builds (#3895)
Co-authored-by: Yueming Yuan <yym022502@gmail.com>
2026-10-02 11:26:28 -07:00
Jiajun Li 64772f50b2 build: bump CUDA 13 images to SGLang v0.5.21 (#3884) 2026-10-02 00:00:16 -07:00
Jiajun Li 919a6502fe ci(wheels): notify Lark when builds or publishing fail (#3881) 2026-10-01 15:32:18 -07:00
Jiajun LiandClaude Opus 5.5 1f92079cfc ci(wheels): add a workflow that rebuilds the x86_64 TE wheels from the fork (#3871)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 15:23:36 -07:00
Jiajun Li 957e11642f fix(ci): preserve image cuDNN during dependency resolution (#3880) 2026-10-01 15:22:28 -07:00
Jiajun Li 55ec3d870f test(e2e): rework e2e CI coverage on top of the H200 migration (#3699) 2026-09-30 15:54:37 -07:00
Xinyu JiangandZhiyao Jiang 874b3e2756 [AMD] docker: build the ROCm images on the sglang v0.5.20 release images (#3873)
Co-authored-by: Zhiyao Jiang <jessicajiang324@gmail.com>
2026-09-30 00:10:03 -07:00
Jiajun Li 9e4260de04 Use the base image's Mooncake instead of reinstalling a dev wheel (#3738) 2026-09-27 21:55:14 -07:00
Jiajun LiandClaude Opus 5.5 f84b1496e1 test(e2e): move 8x H100 e2e tests to H200 stages and widen parallelism coverage (#3660)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-27 20:41:43 -07:00
fzyzcjy cee2e4b143 Cover the ray x rpc worker comm path in CI (#3048) 2026-09-26 20:32:29 +08:00
fzyzcjy 2ff226b1da Install kubectl and helm into the training image (#2191) 2026-09-26 17:42:02 +08:00
Yueming Yuan 3d01116aee release: gate on CPU and CUDA CI only (#3705) 2026-09-25 18:40:18 -06:00
Yueming Yuan 9e5cfd143a release: publish CUDA 13 only and fingerprint configured wheels (#3703) 2026-09-25 01:37:23 -07:00
Xinyu JiangandZhiyao Jiang e30b8b6dd6 [AMD] replace the ROCm 7.2.0 image with ROCm 7.2.4 (#3326)
Co-authored-by: Zhiyao Jiang <jessicajiang324@gmail.com>
2026-09-24 03:59:24 -04:00
Jiajun Li 84ebad0569 ci(docker): notify Lark when an automatic image build fails (#3639) 2026-09-23 15:33:32 -07:00
Jiajun Li d2b66cab68 ci(docker): poll upstream every 10 minutes and rebuild at least every 12h (#3630) 2026-09-23 15:29:41 -07:00
Jiajun Li 8b4d79bf50 ci: run PR Test on non-main-base PRs only with a run-ci label (#3636) 2026-09-23 15:27:23 -07:00
Yueming Yuan a66064f190 fix(ci): install test dependencies before training pins (#3580) 2026-09-21 17:08:17 -07:00
Zhiyao JiangandXinyu Jiang 8d22a2c345 docs(AMD): Add an AMD ROCm page and fix the AMD image and hardware notes (#3209)
Co-authored-by: Xinyu Jiang <xinyuj2@andrew.cmu.edu>
2026-09-17 18:13:19 -07:00