* [BugFix] Track GEMM accumulator writes in warp-specialization liveness collector
* [BugFix] Track scale-factor regions and region extents in gemm liveness
T.atomic_load and T.atomic_store previously accepted every order in the shared memory-order map, including operation-illegal combinations that trigger device-side assertions in libcu++.
Validate memory orders against operation-specific allow lists before lowering, report invalid values with a frontend ValueError, and update the documentation and regression coverage accordingly.
Fixes#2574
* [BugFix] Return previous value for atomic min/max/add (scalar + vectorized)
`atomic_min`/`atomic_max` (scalar) and `atomic_addx2`/`atomic_addx4`
(vectorized) typed their call with a non-void return under
`return_prev=True` but always selected the void `*_elem_op`, so codegen
emitted an empty binding (`prev = ;`) and nvcc failed.
Select the `*_ret_elem_op` variant under `return_prev`, as `atomic_add`
already does:
- min/max: the ret ops, codegen handlers, and device templates
(AtomicMaxRet/AtomicMinRet) already existed.
- addx2/addx4 (float32): register the ret ops (builtin.cc/.h) and add
codegen handlers emitting AtomicAddx2Ret/AtomicAddx4Ret (already in
the device headers) as expressions; the return is typed as the packed
2-/4-lane vector so the store matches.
- addx2 (fp16/bf16): the ret store LHS is `uint1`; codegen wraps the
half2/bf162 result with `tl::to_uint1`.
- addx4 (fp16/bf16): add AtomicAddx4Ret(half_t*)/(bfloat16_t*) device
overloads that run two AtomicAddx2Ret halves and return them packed as
uint2 (a fp16x4 is stored as uint2).
The 16-bit x4 path is per-pair atomic (two half2 atomics), not
whole-quad single-atomic -- hardware has no vector atomic wider than
`.f16x2` for 16-bit types. This matches the existing fp32-x4 scalar
fallback, and the docstring only promises "quad-width atomic addition".
Adds tests across scalar/x2/x4 and float32/fp16/bf16.
Fixes#2667
* minor fix for missing conditiions
---------
Co-authored-by: LeiWang1999 <leiwang1999@outlook.com>
`T.infinity` aborted with an internal `LOG(FATAL)` ("Cannot decide
infinity for type float8_e5m2") because `infinity_op` had no fp8 branch.
e5m2 is the only fp8 format with a representable inf, so add a branch
returning +inf, and emit it in codegen via its bit pattern
(`::bitcast(0x7c)`, mirroring the tfloat32 path) since inf/NaN cannot be
spelled as a float literal for the fp8 constructor. Other fp8 formats
(e4m3*, e3m4, e8m0, finite fnuz incl. e5m2fnuz) have no inf and keep the
fatal. Adds e5m2 to the infinity test.
* feat(autotuner): support per-config pass_configs with code deduplication
- Extract _merge_pass_configs_into_compile_args helper to deduplicate
pass_configs merge logic across grouped_compile.py and tuner.py
- Replace json.dumps uniformity check with direct equality comparison
- Extract _make_jit_compile_func factory to unify lazy/eager jit_compile
- Fix decorator-mode cache isolation: use compile() directly when
per_config_pass_configs is present to avoid JIT cache key collision
- Add example and test for per-config pass_configs tuning
* docs: add docstrings to meet coverage threshold
* refactor(autotuner): move pass_configs resolution to tuner so grouped compile handles uniform configs
* fix(autotuner): define _PASS_CONFIGS_KEY constant and fix jit_compile race condition
- Replace all magic string "__pass_configs__" with _PASS_CONFIGS_KEY constant
- Move _pass_configs_lock to cover both branches in jit_compile to prevent
concurrent reads of temporarily mutated pass_configs
* refactor(autotuner): introduce type aliases for compile-unit bookkeeping
- Add ConfigArg, UnitItem, UnitResult, CompileUnit type aliases
- Replace all long tuple type annotations with aliases
- Expand example configs to include num_stages=[1,2] and enable_grouped_compile
---------
Co-authored-by: daniu.ld <daniu.ld@alibaba-inc.com>
T.dp4a previously accepted arbitrary buffer dtypes and passed them to backend DP4A implementations, which could silently reinterpret the operand bytes and produce incorrect results.
Require both inputs to use int8 and the accumulator to use int32. Add regression coverage for valid Buffer and BufferLoad forms and for invalid input and accumulator dtypes.
Fixes#2600
* Add experimental IKET CUDA backend hooks
* Keep IKET APIs under namespace
* Use real Perfetto screenshot for IKET example
* Show IKET events in Perfetto timeline screenshot
* Add IKET validation overhead example
* Remove IKET validation overhead example
* Fix IKET lint issues
* Move IKET integration to CUDA tools
* Add tools documentation and fix workflow regressions
---------
Co-authored-by: LeiWang1999 <leiwang1999@outlook.com>
* [Feature] Add pass_visualizer: structure-tree pass browser
Add an interactive, pass-by-pass IR structure-tree visualizer under
tilelang/utils/pass_visualizer. It complements the existing text-level
pass_diff tool by rendering the SBlock structure tree and expanding tile
ops by field name, with per-class operator highlighting (tile op / sync /
lowered hardware intrinsic) across the CUDA lowering prologue.
Includes a CLI entry point, an example gemm_relu kernel, a pytest suite,
and a docs section contrasting it with pass_diff.
Co-Authored-By: Claude <noreply@anthropic.com>
* Move pass_visualizer to tilelang/tools and fix prologue + review issues
- Move package from tilelang/utils to tilelang/tools per review
- Add missing MaterializeKernelLaunch pass (fixes CUDA CI modulo-by-zero)
- Escape "</" in embedded JSON to prevent <script> injection in HTML report
- Use explicit utf-8 encoding for source/output file I/O
- Guard _fmt_shape against symbolic shapes and validate module spec
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* [CUDA] Add T.__fns intrinsic for find-nth-set bit
Expose CUDA __fns in TileLang for direct lookup of the k-th active lane in a 32-bit bitmask, complementing the existing T.__ffs helper.
* Add builtin intrinsics for shared ldst and atomics
* [BugFix] Skip source-compilation options when exporting LLVM module cache
* Add regression test
* Refactor kernel cache export handling and improve test coverage
- Renamed `_get_compile_args` to `_get_source_compile_args` for clarity.
- Introduced `_get_export_link_args` to manage export link arguments.
- Updated `_safe_write_executable` to accept export kwargs directly.
- Enhanced `TVMFFIKernelCache` to determine export kwargs based on target host.
- Modified tests to validate new export behavior and ensure source options are correctly handled for different targets.
---------
Co-authored-by: LeiWang1999 <leiwang1999@outlook.com>
Use PTX v4 atomics for 16-bit CUDA atomic_addx4
The initial fp16/bf16 atomic_addx4 fix avoided the generic float4 path by splitting each operation into two AtomicAddx2 calls. That fixed correctness, but it still used the simulated implementation on Hopper where PTX provides native vector atomics.
Add sm90+ half_t and bfloat16_t AtomicAddx4 overloads backed by the PTX vector forms atom.global.v4.f16.add.noftz and atom.global.v4.bf16.add.noftz, including the existing memory-order variants. Keep the x2 fallback for pre-sm90 targets because PTX vector f16/bf16 atomics require sm90 or newer.
Keep the CUDA language regression focused on fp16/bf16 atomic_addx4 codegen and offset coverage.
Co-authored-by: dingsg <shengge.ding@enflame-tech.com>
* Fix physical wait counts for async pipeline groups
* Add commit group tracking for async pipeline in inject_pipeline.cc
- Introduced buffer_to_commit_group mapping and commit_group_count in AsyncStateGlobal.
- Enhanced wait handling for tail consumers in async pipeline to ensure correct physical wait counts.
- Updated tests to validate fine-grained physical waits and tail drain waits for async pipeline.
_tir_u32_to_int_to_float decoded signed sub-word fields from uint32 storage as unsigned values because it only masked the selected field before casting. As a result, negative encodings such as signed int4 `0xF` decoded as `15.0` instead of `-1.0`.
Sign-extend the extracted field through int32 before casting to the requested float dtype, matching the existing packed signed-int decode helper.
Add CUDA regression coverage for signed int2, int4, and int8 values decoded from uint32 storage.
Fixes#2482
* Add split-k for example_warp_specialize_flashmla
* Rotate QK issue to end of main loop to allow more latency hiding with TMA issue
* During rescale, read max score from REG instead of SMEM
* Move K_pe_shared_1 load
* Use stmatrix for SP0 write to reduce SMEM bank conflict
* Support arbitrary SMEM layout in MMA lowering
* Remove runtime guard for increase_descriptor_offset
* [Refactor] Remove unsupported tl_gemm and tl_gemm_sp operations
* Removed references to tl_gemm and tl_gemm_sp from various files, including code generation and built-in definitions, as these operations are currently unsupported.
* Updated related comments and diagnostics to reflect the removal of these operations, ensuring clarity in the codebase.
* This change simplifies the handling of unsupported operations and improves overall code maintainability.
* Reduce CUDA template include overhead
* Fix CUDA reduce fast min max helpers
* lint fix
* Fix CUDA swizzle ceil div helper
* Restore CUDA wait_wgmma intrinsic helper
* Fix CUDA cython TMA host adapter include
* Remove unnecessary check for Cutlass fast math header in CUDA code generation
* Update LibraryGenerator to correctly handle CUDA gencode flags for shared-library compilation, ensuring compatibility with target architecture and avoiding Hopper-only instruction issues.
* Refactor smem_ptr_to_uint function to use __cvta_generic_to_shared for improved clarity and performance.
* Refactor example scripts and CUDA math intrinsics for improved performance and clarity
- Updated `example_dequant_gemm_fp4_hopper.py` to directly call `run_regression_perf()` and print latency, removing unused argument parsing code.
- Modified `example_topk.py` to specify the backend as "cupti" in the benchmarking function.
- Enhanced CUDA code generation by introducing `RequiresTileLangMathHeader` to manage math header dependencies for specific functions.
- Updated barrier operations in `barrier.h` to use the correct syntax for shared memory barriers.
- Improved testing for bfloat16 math intrinsics by replacing pytest skips with decorators for CUDA availability checks.
* Refactor CUDA code for improved readability and consistency
- Reformatted the `RequiresTileLangMathHeader` function for better readability.
- Updated assembly syntax in `barrier.h` for clarity and consistency.
- Cleaned up whitespace in the bfloat16 math intrinsics test file to enhance code cleanliness.
* Remove unnecessary whitespace in CUDA code for improved readability
* Add TileLang intrinsic for prefetching TMA descriptor
- Introduced `prefetch_tma_descriptor` intrinsic in C++ and updated relevant files to use `call_intrin` instead of `call_extern`.
- Updated CUDA code generation to handle the new intrinsic correctly.
- Modified tests to ensure the new intrinsic is called appropriately and that no `call_extern` is used for internal TileLang operations.
- Enhanced documentation in SKILL.md regarding the addition of new TileLang intrinsics.
* Add bfloat16 hexp overload and update common.h for compatibility
- Introduced `hexp` overload for `bfloat16_t` in `common.h` to bridge TileLang and CUDA's handling of exponential functions.
- Updated tests to verify the presence of the new `hexp` definition in `common.h`.
- Enhanced CUDA code generation to support the new intrinsic for better performance and compatibility.
* Update common.h for CUDA compatibility with aligned double4 types
- Adjusted preprocessor condition in `common.h` to maintain compatibility with CUDA versions prior to 13 by ensuring proper definition of aligned double4 types.
- This change enhances the compatibility of the code with older NVRTC built-in vector headers.
* [BugFix] Reject unsupported T.serial access to thread-distributed fragments with clear diagnostics
A T.serial loop runs sequentially inside each CUDA thread, but a fragment's
elements are distributed across threads, so indexing a fragment by a serial
loop variable has no valid thread-ownership mapping. Three fuzzer reports share
this root cause:
- #2393: serial reduce of a GEMM-output fragment -> unhelpful internal error
("contains inner var j").
- #2395: serial write into a fragment then T.copy out -> nvcc "identifier
undefined" (an owner-thread guard built from serial loop vars was emitted
outside the loop that binds them).
- #2396: serial write of a fragment to global -> silently wrong (threads
race-write each cell).
AddWrapperForSingleBufStore wraps bare fragment stores in a degenerate
T.parallel(1) and only rejected constant non-zero fragment indices, so a
dynamic (e.g. serial-loop) index slipped through and lowered to broken code
(#2395/#2396). Reject any fragment index that is not the constant all-zero
index in this fallback path, with a message pointing to T.Parallel /
T.reduce_sum. Rephrase the existing inner-var layout check (#2393) to explain
the cause instead of dumping "contains inner var".
Also collect every access to a buffer in a statement (not just the last) so the
diagnostic is not bypassed when one fragment is accessed at both a constant and
a variable index in the same statement; without it such a case falls through to
a cryptic downstream StructuralEqual internal error. This is a completeness
hardening only, not required by the three issues above.
Fixes#2393Fixes#2395Fixes#2396
* Generalize fragment diagnostic to non-parallel loops
inner_vars_ is populated for every non-parallel For (serial, unroll,
vectorized), so the message must not hard-code T.serial. Reword to
"non-parallel loop variable" per review feedback.
* Narrow fix scope to #2393 only
Remove fragment index validation from AddWrapperForSingleBufStore and
delete tests for #2395/#2396. These issues cannot be caught at this pass
stage because T.grid and T.serial are indistinguishable (both appear as
ForKind.SERIAL before LowerTileOp).
Keep only the ParallelOpNode diagnostic improvement which successfully
catches #2393.
* Add CUDA target code attr support
* Add CUDA target normalization support
Introduced a new function to normalize CUDA targets, ensuring that the target is correctly identified and transformed based on the detected architecture. Added a test to verify that the bare CUDA target uses the detected architecture accurately.
* Add cross-host CUDA binary cache
* Nest host kernel cache under version namespace
* [BugFix] Fix T.Persistent dropping tiles when last dim is not a multiple of group_size (#2433)
* Update src/ir.cc
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Add CUDA target code attr support
* Add CUDA target normalization support
Introduced a new function to normalize CUDA targets, ensuring that the target is correctly identified and transformed based on the detected architecture. Added a test to verify that the bare CUDA target uses the detected architecture accurately.
T.fill of an int8 buffer with a negative scalar (other than -1) silently
wrote only 1 of every 4 elements correctly and clobbered the other 3 to
-1. The vectorized fill packs four int8 lanes into one 32-bit store via
make_int(v,v,v,v); a negative signed char is integer-promoted to int and
sign-extended before the shifts/OR, so its high 1-bits flood the
neighbouring lanes (e.g. make_int(-7,-7,-7,-7) returned 0xFFFFFFF9 =
[-7,-1,-1,-1]).
Build the packed value from explicit unsigned bytes so the sign-extension
bits cannot leak across lanes. Add an int8 negative-fill regression test.
Fixes#2427
* Add stmatrix m16n8 support
* lint
* Refactor stmatrix code generation to handle optional shape argument
Updated the handling of the shape argument in the stmatrix function across CUDA, CuTeDSL, and HIP code generators. The shape is now checked as the last argument, and the payload start index is adjusted accordingly. This change improves consistency and clarity in the code generation process for tile language operations.
* Refactor stmatrix function naming for consistency
Updated the naming convention of stmatrix functions in CUDA and related files to improve clarity and consistency. The function names now follow a uniform pattern, placing the shape descriptor before the axis identifier. This change enhances readability and aligns with the recent updates in code generation logic.
* Enhance extern call statement printing in CUDA codegen
Updated the `VisitExpr_` method in `codegen_cuda.cc` to improve the clarity of extern call statements. The function now includes comments indicating that the transformation, number, and shape are encoded in the helper name, and only the shared memory pointer and value operands are printed. This change aims to enhance code readability and maintainability.
* Refactor extern call expression printing in CUDA codegen
Introduced a helper function to streamline the printing of extern call expressions in the `VisitExpr_` method of `codegen_cuda.cc`. This change enhances code readability by reducing redundancy and improving the clarity of the generated output for specific function calls, such as `tl::pack_b8x4`.
* Refactor shape argument handling in stmatrix code generation
Updated the handling of the shape argument in the `VisitExpr_` method for both CUDA and CuTeDSL code generators. The shape is now checked as the fourth argument instead of the last, improving clarity and consistency in extern call statement printing. This change enhances the robustness of the code generation process for tile language operations.
* Refactor shape argument handling in stmatrix code generation
Updated the handling of the shape argument in the `VisitExpr_` method for both CUDA and CuTeDSL code generators. The shape is now checked as the last argument and validated accordingly, improving clarity and consistency in the code generation process for tile language operations.
* Refactor tilelang lower function context management
Updated the context management in the `test_tilelang_copy_uses_stmatrix_m16n8_for_sm100_int8_shared_store` test by removing the unnecessary `PassContext` wrapper. This change simplifies the code and enhances clarity in the test setup for tile language operations.
---------
Co-authored-by: LeiWang1999 <leiwang1999@outlook.com>
morluto