98 Commits
Author SHA1 Message Date
Xinyu JiangandZhiyao Jiang 1e4ad549a1 [AMD] ci: bump ROCm SGLang to v0.5.22 and expand MI350 nightly coverage (#4202)
Co-authored-by: Zhiyao Jiang <jessicajiang324@gmail.com>
2026-10-19 18:23:06 -07:00
fzyzcjy 168c882f97 Hold the rollout after each all-gather fault until the victim heals (#4178) 2026-10-19 11:57:23 +08:00
fzyzcjy 87de59af2f Add a timed sleep fault action that resumes after a fixed duration (#4177) 2026-10-19 11:56:18 +08:00
fzyzcjy 67182baaf6 Verify fault hook dispatch evidence and surviving trainer progress (#3854) 2026-10-19 11:45:28 +08:00
fzyzcjy e6ab7955b6 Deliver a process fault to another worker through the api server (#3839) 2026-10-19 11:25:09 +08:00
fzyzcjy 32b658aad2 Add stop, freeze and deadlock process fault actions (#3832) 2026-10-19 11:18:45 +08:00
fzyzcjy 400845d5ce Park the orchestrator through a fault hook and retire --ci-ft-test-actions (#3826) 2026-10-19 11:11:43 +08:00
fzyzcjy c82a030154 Stop and start a trainer cell through fault hooks (#3825) 2026-10-19 11:10:31 +08:00
fzyzcjy 023029f60d Crash the trainer through a fault hook instead of the declared test action (#3824) 2026-10-19 11:09:27 +08:00
Jiajun Li 0309247cea fix(ci): remove the default raw reward upper gate (#4160) 2026-10-17 20:55:01 -07:00
Jiajun Li 1647bd3b25 Bump sglang to v0.5.22 (#4125) 2026-10-16 17:43:49 -07:00
Jiajun Li ff9853c0b1 ci: gate performance benchmarks on est_time with a perf label (#4033) 2026-10-14 16:12:38 -07:00
Jiajun Li 64842375d6 fix(ci): make the default grad_norm gate a 4x sanity band (#4094) 2026-10-14 14:51:58 -07:00
Zhichen Zeng 23d3d38094 feat(skills): pr-ci-targeted run only the CI test files a PR reaches (#4002) 2026-10-14 13:47:02 -07:00
Jiajun Li aa5dbf0b22 feat(ci): key metric history by metric and step (#4089) 2026-10-14 13:16:41 -07:00
Jiajun Li ddb6113545 ci: keep NVSHMEM on NVLink on every RoCE-only runner (#4088) 2026-10-14 12:56:34 -07:00
Jiajun Li 3f5c754c58 feat(ci): enforce metric history automatically after ten goldens (#4023) 2026-10-14 12:50:15 -07:00
fzyzcjy 0c03ff789d Select the legacy buffer only for partial rollout (#3381) 2026-10-14 20:46:07 +08:00
Jiajun Li 2766ab0ac1 ci: move ten 4x H200 tests that fit on two GPUs to 2x H200 (#4053) 2026-10-13 23:31:46 -07:00
Jiajun Li 77309a60d9 ci: move slow CPU tests to tagged Stage B registrations (#4008) 2026-10-10 15:55:38 -07:00
Jiajun Li 9bdc3daaca ci: queue Hopper and ROCm shards without matrix throttling (#4006) 2026-10-10 14:26:22 -07:00
Jiajun Li ccc8009ade ci: split Hopper jobs to reduce failed-job rerun cost (#4005) 2026-10-10 02:41:27 -07:00
Jiajun Li 3a05ce560e fix(ci): gate slow Kubernetes and rollout CPU tests by domain (#3993) 2026-10-09 02:24:55 -07:00
Yueming Yuan dba9001cc6 ci: schedule B200 tests by minimum GPU count (#3983)
Schedule B200 tests with their minimum GPU count and dynamically lease free devices. Preserve test coverage, isolate concurrent jobs, and protect GPUs held by surviving job containers.

Bound admission waits with per-batch job budgets while preserving per-file execution timeouts.
2026-10-08 22:33:45 -07:00
Jiajun Li 6f94a01372 fix(ci): match H200 shards to runner capacity (#3990) 2026-10-08 19:25:54 -07:00
Zhichen Zeng 49b57dffea [1/8] refactor(kernels): move quant, MoE, activation, norm, HC and RoPE kernels to miles/kernels (#3605) 2026-10-08 12:57:20 -07:00
Kaixi Hou f81d52a40e Add support for Vera Rubin (#3700) 2026-10-08 11:07:31 -07:00
Jiajun LiandClaude Opus 5.5 49464b3cd1 ci: offload the 8x H200 suite to H100, B200 and 4x H200 (#3956)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 00:33:50 -07:00
Jiajun LiandClaude Opus 5.5 233a0cbe41 ci: write the rules for adding a CI test in .claude/rules (#3953)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 00:33:50 -07:00
Jiajun Li a0b2171b02 fix(ci): keep 4-GPU H200 DeepEP on NVLink for RoCE runners (#3955) 2026-10-07 22:47:11 -07:00
Jiajun Li 9350dda7ca feat(ci): rerun PR workflows when labels are removed (#3978) 2026-10-07 22:46:44 -07:00
Jiajun Li 1e1966293e fix(ci): cancel file tests and ROCm after PR merge (#3927) 2026-10-06 14:14:07 -07:00
Jiajun Li ae1d5fa60c fix(ci): skip GPU runners for empty test selections (#3902) 2026-10-06 13:36:48 -07:00
MichaelandXinyu Jiang 3d388872ac [AMD] docker: add rocm10-mi30x (gfx942) image variant (#3858)
Co-authored-by: Xinyu Jiang <xinyuj2@andrew.cmu.edu>
2026-10-05 15:49:56 -07:00
Yueming Yuan a095490625 ci: allow Megatron Bridge version overrides (#3897) 2026-10-02 19:37:28 -07:00
Jiajun LiandYueming Yuan 53f5d94224 build: retire CUDA 12 Docker builds (#3895)
Co-authored-by: Yueming Yuan <yym022502@gmail.com>
2026-10-02 11:26:28 -07:00
Jiajun Li 64772f50b2 build: bump CUDA 13 images to SGLang v0.5.21 (#3884) 2026-10-02 00:00:16 -07:00
Zhichen Zeng 2339f2571e training_utils: the shared layer for every training backend (#3248) 2026-10-01 20:02:43 -07:00
Zhichen Zeng fe9fe8d484 feat (trainer): trainer refactor with torchtitan backend support (#3171) 2026-10-01 20:02:42 -07:00
Jiajun Li 919a6502fe ci(wheels): notify Lark when builds or publishing fail (#3881) 2026-10-01 15:32:18 -07:00
Jiajun LiandClaude Opus 5.5 1f92079cfc ci(wheels): add a workflow that rebuilds the x86_64 TE wheels from the fork (#3871)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 15:23:36 -07:00
Jiajun Li 957e11642f fix(ci): preserve image cuDNN during dependency resolution (#3880) 2026-10-01 15:22:28 -07:00
Jiajun Li 55ec3d870f test(e2e): rework e2e CI coverage on top of the H200 migration (#3699) 2026-09-30 15:54:37 -07:00
Xinyu JiangandZhiyao Jiang 874b3e2756 [AMD] docker: build the ROCm images on the sglang v0.5.20 release images (#3873)
Co-authored-by: Zhiyao Jiang <jessicajiang324@gmail.com>
2026-09-30 00:10:03 -07:00
Jiajun Li 9e4260de04 Use the base image's Mooncake instead of reinstalling a dev wheel (#3738) 2026-09-27 21:55:14 -07:00
Jiajun LiandClaude Opus 5.5 f84b1496e1 test(e2e): move 8x H100 e2e tests to H200 stages and widen parallelism coverage (#3660)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-27 20:41:43 -07:00
fzyzcjy 126cbc0064 Save every sample of a multi-policy run, keyed by policy (#3074) 2026-09-26 20:44:51 +08:00
fzyzcjy cee2e4b143 Cover the ray x rpc worker comm path in CI (#3048) 2026-09-26 20:32:29 +08:00
fzyzcjy 6c5e42dc8a Zero-pad the indices in cell ids and worker names (#3023) 2026-09-26 20:19:41 +08:00
fzyzcjy e1d6c2049e Let an injection put a run to sleep at an exact step (#3012) 2026-09-26 20:14:12 +08:00