Xinyu Jiang and Zhiyao Jiang
1e4ad549a1
[AMD] ci: bump ROCm SGLang to v0.5.22 and expand MI350 nightly coverage ( #4202 )
...
Co-authored-by: Zhiyao Jiang <jessicajiang324@gmail.com >
2026-10-19 18:23:06 -07:00
fzyzcjy
168c882f97
Hold the rollout after each all-gather fault until the victim heals ( #4178 )
2026-10-19 11:57:23 +08:00
fzyzcjy
87de59af2f
Add a timed sleep fault action that resumes after a fixed duration ( #4177 )
2026-10-19 11:56:18 +08:00
fzyzcjy
67182baaf6
Verify fault hook dispatch evidence and surviving trainer progress ( #3854 )
2026-10-19 11:45:28 +08:00
fzyzcjy
e6ab7955b6
Deliver a process fault to another worker through the api server ( #3839 )
2026-10-19 11:25:09 +08:00
fzyzcjy
32b658aad2
Add stop, freeze and deadlock process fault actions ( #3832 )
2026-10-19 11:18:45 +08:00
fzyzcjy
400845d5ce
Park the orchestrator through a fault hook and retire --ci-ft-test-actions ( #3826 )
2026-10-19 11:11:43 +08:00
fzyzcjy
c82a030154
Stop and start a trainer cell through fault hooks ( #3825 )
2026-10-19 11:10:31 +08:00
fzyzcjy
023029f60d
Crash the trainer through a fault hook instead of the declared test action ( #3824 )
2026-10-19 11:09:27 +08:00
Jiajun Li
0309247cea
fix(ci): remove the default raw reward upper gate ( #4160 )
2026-10-17 20:55:01 -07:00
Jiajun Li
1647bd3b25
Bump sglang to v0.5.22 ( #4125 )
2026-10-16 17:43:49 -07:00
Jiajun Li
ff9853c0b1
ci: gate performance benchmarks on est_time with a perf label ( #4033 )
2026-10-14 16:12:38 -07:00
Jiajun Li
64842375d6
fix(ci): make the default grad_norm gate a 4x sanity band ( #4094 )
2026-10-14 14:51:58 -07:00
Zhichen Zeng
23d3d38094
feat(skills): pr-ci-targeted run only the CI test files a PR reaches ( #4002 )
2026-10-14 13:47:02 -07:00
Jiajun Li
aa5dbf0b22
feat(ci): key metric history by metric and step ( #4089 )
2026-10-14 13:16:41 -07:00
Jiajun Li
ddb6113545
ci: keep NVSHMEM on NVLink on every RoCE-only runner ( #4088 )
2026-10-14 12:56:34 -07:00
Jiajun Li
3f5c754c58
feat(ci): enforce metric history automatically after ten goldens ( #4023 )
2026-10-14 12:50:15 -07:00
fzyzcjy
0c03ff789d
Select the legacy buffer only for partial rollout ( #3381 )
2026-10-14 20:46:07 +08:00
Jiajun Li
2766ab0ac1
ci: move ten 4x H200 tests that fit on two GPUs to 2x H200 ( #4053 )
2026-10-13 23:31:46 -07:00
Jiajun Li
77309a60d9
ci: move slow CPU tests to tagged Stage B registrations ( #4008 )
2026-10-10 15:55:38 -07:00
Jiajun Li
9bdc3daaca
ci: queue Hopper and ROCm shards without matrix throttling ( #4006 )
2026-10-10 14:26:22 -07:00
Jiajun Li
ccc8009ade
ci: split Hopper jobs to reduce failed-job rerun cost ( #4005 )
2026-10-10 02:41:27 -07:00
Jiajun Li
3a05ce560e
fix(ci): gate slow Kubernetes and rollout CPU tests by domain ( #3993 )
2026-10-09 02:24:55 -07:00
Yueming Yuan
dba9001cc6
ci: schedule B200 tests by minimum GPU count ( #3983 )
...
Schedule B200 tests with their minimum GPU count and dynamically lease free devices. Preserve test coverage, isolate concurrent jobs, and protect GPUs held by surviving job containers.
Bound admission waits with per-batch job budgets while preserving per-file execution timeouts.
2026-10-08 22:33:45 -07:00
Jiajun Li
6f94a01372
fix(ci): match H200 shards to runner capacity ( #3990 )
2026-10-08 19:25:54 -07:00
Zhichen Zeng
49b57dffea
[1/8] refactor(kernels): move quant, MoE, activation, norm, HC and RoPE kernels to miles/kernels ( #3605 )
2026-10-08 12:57:20 -07:00
Kaixi Hou
f81d52a40e
Add support for Vera Rubin ( #3700 )
2026-10-08 11:07:31 -07:00
Jiajun Li and Claude Opus 5.5
49464b3cd1
ci: offload the 8x H200 suite to H100, B200 and 4x H200 ( #3956 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-08 00:33:50 -07:00
Jiajun Li and Claude Opus 5.5
233a0cbe41
ci: write the rules for adding a CI test in .claude/rules ( #3953 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-08 00:33:50 -07:00
Jiajun Li
a0b2171b02
fix(ci): keep 4-GPU H200 DeepEP on NVLink for RoCE runners ( #3955 )
2026-10-07 22:47:11 -07:00
Jiajun Li
9350dda7ca
feat(ci): rerun PR workflows when labels are removed ( #3978 )
2026-10-07 22:46:44 -07:00
Jiajun Li
1e1966293e
fix(ci): cancel file tests and ROCm after PR merge ( #3927 )
2026-10-06 14:14:07 -07:00
Jiajun Li
ae1d5fa60c
fix(ci): skip GPU runners for empty test selections ( #3902 )
2026-10-06 13:36:48 -07:00
Michael and Xinyu Jiang
3d388872ac
[AMD] docker: add rocm10-mi30x (gfx942) image variant ( #3858 )
...
Co-authored-by: Xinyu Jiang <xinyuj2@andrew.cmu.edu >
2026-10-05 15:49:56 -07:00
Yueming Yuan
a095490625
ci: allow Megatron Bridge version overrides ( #3897 )
2026-10-02 19:37:28 -07:00
Jiajun Li and Yueming Yuan
53f5d94224
build: retire CUDA 12 Docker builds ( #3895 )
...
Co-authored-by: Yueming Yuan <yym022502@gmail.com >
2026-10-02 11:26:28 -07:00
Jiajun Li
64772f50b2
build: bump CUDA 13 images to SGLang v0.5.21 ( #3884 )
2026-10-02 00:00:16 -07:00
Zhichen Zeng
2339f2571e
training_utils: the shared layer for every training backend ( #3248 )
2026-10-01 20:02:43 -07:00
Zhichen Zeng
fe9fe8d484
feat (trainer): trainer refactor with torchtitan backend support ( #3171 )
2026-10-01 20:02:42 -07:00
Jiajun Li
919a6502fe
ci(wheels): notify Lark when builds or publishing fail ( #3881 )
2026-10-01 15:32:18 -07:00
Jiajun Li and Claude Opus 5.5
1f92079cfc
ci(wheels): add a workflow that rebuilds the x86_64 TE wheels from the fork ( #3871 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-01 15:23:36 -07:00
Jiajun Li
957e11642f
fix(ci): preserve image cuDNN during dependency resolution ( #3880 )
2026-10-01 15:22:28 -07:00
Jiajun Li
55ec3d870f
test(e2e): rework e2e CI coverage on top of the H200 migration ( #3699 )
2026-09-30 15:54:37 -07:00
Xinyu Jiang and Zhiyao Jiang
874b3e2756
[AMD] docker: build the ROCm images on the sglang v0.5.20 release images ( #3873 )
...
Co-authored-by: Zhiyao Jiang <jessicajiang324@gmail.com >
2026-09-30 00:10:03 -07:00
Jiajun Li
9e4260de04
Use the base image's Mooncake instead of reinstalling a dev wheel ( #3738 )
2026-09-27 21:55:14 -07:00
Jiajun Li and Claude Opus 5.5
f84b1496e1
test(e2e): move 8x H100 e2e tests to H200 stages and widen parallelism coverage ( #3660 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-27 20:41:43 -07:00
fzyzcjy
126cbc0064
Save every sample of a multi-policy run, keyed by policy ( #3074 )
2026-09-26 20:44:51 +08:00
fzyzcjy
cee2e4b143
Cover the ray x rpc worker comm path in CI ( #3048 )
2026-09-26 20:32:29 +08:00
fzyzcjy
6c5e42dc8a
Zero-pad the indices in cell ids and worker names ( #3023 )
2026-09-26 20:19:41 +08:00
fzyzcjy
e1d6c2049e
Let an injection put a run to sleep at an exact step ( #3012 )
2026-09-26 20:14:12 +08:00