* [BugFix] Always vectorize `T.Parallel` loops
* Fix CI failures
* Restore always-vectorize accidentally reverted by the CI-fix commit
f447919b captured a transient working-tree state from a concurrent
baseline-vs-patched benchmark flip, reverting the gate removal in src/
while keeping the new-policy tests: CI then ran the old gated compiler
against them (scalar var-loop -> no float2; positional if/for check
broken by the copy loops the larger tile introduces).
Bring back the parameter-less LowerParallelLoop and rewrite the issue
1734 assertion as containment: the guarded loop must sit inside the
hoisted if block, which holds on both sm90 and sm100 regardless of how
many copy loops precede it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
_tir_packed_to_unsigned_convert_with_zeros performed the zero-point subtraction in the unsigned packed storage dtype before casting to the requested output dtype. Values below the zero-point therefore wrapped modulo the storage width.
Cast the unpacked field and zero-point to the output dtype before subtraction so asymmetric dequantization preserves negative values.
Add CUDA regression coverage for uint32-packed 4-bit values decoded with a non-zero zero-point.
Fixes#2947
* [BugFix][Hopper] Select the widest legal WGMMA N instead of gcd
block_N extents that are legal single WGMMA widths but not powers of two
(96, 112, 160, 176) were decomposed into n32 / n16 instructions, because
the instruction width came from gcd(warp_col_tiles, 256). wgmma.h already
instantiates every multiple of 8 up to 256, so only the width selection
was the constraint.
Select the widest legal width that tiles warp_col_tiles exactly, keeping
gcd as the fallback. Widening is restricted to multiples of 16: the
N % 16 == 8 class already miscomputes (#2593), and the integer instruction
tables only instantiate multiples of 16 (plus 8 and 24). The result is
never narrower than before, so no existing shape regresses.
The sparse generator carried the same line and now shares the helper.
* [Testing] Cover T.gemm_sp at the widened WGMMA N values
The width selector is shared with the sparse generator, so assert the emitted
wgmma_sp instruction and the sparse numeric result for block_N in {96, 112,
160} as well. Verified as a real regression: with the selector reverted the
sparse path emits n32 / n16 / n32 exactly like the dense one.
The test added in #2899 lowers with an explicit `tvm.target.Target("cuda")`, which
reaches `tilelang/cuda/transform/__init__.py`. In a `USE_CUDA=OFF` build only the
"ALWAYS" CUDA sources compile, so `AnnotateDeviceBoundTmaCopies` is never registered
and the test fails on ROCm:
AttributeError: module 'tilelang.cuda._ffi_api' has no attribute
'AnnotateDeviceBoundTmaCopies'
Same pattern as #2863. `requires_cuda` is the weakest sufficient gate: the test is
compile-only (`enable_device_compile=False`), so it needs no particular GPU
generation and keeps running on the NVIDIA leg.
Verified in CI: SKIPPED on the gfx942 leg, PASSED on CUDA-auto.
* [BugFix] Handle vectorized SelectNode in codegen_cuda
* address review comments
* [BugFix] Fix cuda codegen vectorized SelectNode when condition is a scalar
A large number of tests import CUDA-only internals directly (for example
`tilelang.cuda.pipeline.CUDAPassPipelineBodyPrologue`) or exercise
NVIDIA-specific features such as TMA, LDGSTG, mbarrier, warp specialization,
tcgen05 and tfloat32. `src/cuda/CMakeLists.txt` only compiles a small
"ALWAYS" subset when `USE_CUDA=OFF`, so those FFI symbols are never
registered and the tests fail with:
AttributeError: module 'tilelang.cuda._ffi_api' has no attribute
'AnnotateDeviceBoundTmaCopies'
This is not ROCm-specific -- the same tests fail on any build without the
CUDA backend, including Metal and CPU-only.
Add `@tilelang.testing.requires_cuda` to those tests, plus three
portability fixes that keep coverage on non-CUDA backends instead of
skipping it:
* `test_tilelang_transform_layout_inference.py`: CUDA emits
`__launch_bounds__(N, 1)` while HIP emits `__launch_bounds__(N)`.
Match both.
* `test_tilelang_transform_pipeline_planning.py`: PipelinePlanning
deliberately strips pipeline annotations on ROCm targets other than
gfx950 (see the guard in src/transform/pipeline_planning.cc), so assert
the documented sequential fallback there.
* `test_tilelang_kernel_gemm.py`: the two f32f32f32 cases use `T.tfloat32`,
which is NVIDIA-only, so narrow `requires_cuda_or_cdna` to
`requires_cuda`.
Also refresh three stale RDNA tests: gfx12 is now a supported generation
(`get_arch` accepts 11 and 12), so they can no longer assert that gfx1200
is rejected. Force an unsupported generation instead, which is the only
way to reach the guard now that `target_is_rdna` reports only gfx11/gfx12.
Verified on 8x MI300X (gfx942, ROCm 7.2 container, torch nightly rocm7.2):
the ROCm suite goes from 105 failures to 0 with no loss of passing tests.
* fix: resolve issue 2537
* test: pin default-search regression tests to the MMA backend
On sm_90 SelectInst reroutes M>=64 / warps%4==0 shapes to the WGMMA
search, so two tests written against the default search exercised the
wrong code path on Hopper CI. The FullCol M=192 N=104 numeric test
additionally hit a pre-existing, partition-independent WGMMA
descriptor-layer limitation (N=104 fits no canonical GMMA swizzle atom).
Pin both to the MMA path via tl.disable_wgmma; the WGMMA search keeps
its own dedicated tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* [BugFix] Reject contracting shared-buffer layouts in T.annotate_layout
T.annotate_layout could accept a contracting many-to-one forward map on a
shared buffer. makeBufferWithLayout then misread the smaller output extent as
a replication factor, rewrote the buffer to a higher rank, and eventually
failed with an unrelated BufferStore dimensionality ICHECK.
Add a pigeonhole check for static layout shapes before the replication branch.
When the layout output extent is smaller than its input extent, raise a
user-facing ValueError that identifies the buffer and reports both extents.
Keep identity, permutation, and output-growing layouts unchanged.
This check intentionally targets contracting maps; general injectivity
analysis for maps with larger output bounding boxes is outside this change.
Fixes#2632
* [Layout] Enforce injective shared-buffer annotations
Add a LayoutNode injectivity check with an exact fallback for static domains, and reuse it for FragmentNode thread and replication coordinates. Validate shared-buffer annotations after layout reshaping so sparse many-to-one maps are rejected with a useful collision diagnostic while padded injective maps remain valid.
---------
Co-authored-by: LeiWang1999 <leiwang1999@outlook.com>
* [BugFix] Pack 32-lane 8-bit CUDA vectors correctly
* [Test] Avoid SM100-only store in int8 broadcast regression
* [BugFix] Move int8x32 packing into CUDA templates
T.dp4a previously accepted arbitrary buffer dtypes and passed them to backend DP4A implementations, which could silently reinterpret the operand bytes and produce incorrect results.
Require both inputs to use int8 and the accumulator to use int32. Add regression coverage for valid Buffer and BufferLoad forms and for invalid input and accumulator dtypes.
Fixes#2600
_tir_u32_to_int_to_float decoded signed sub-word fields from uint32 storage as unsigned values because it only masked the selected field before casting. As a result, negative encodings such as signed int4 `0xF` decoded as `15.0` instead of `-1.0`.
Sign-extend the extracted field through int32 before casting to the requested float dtype, matching the existing packed signed-int decode helper.
Add CUDA regression coverage for signed int2, int4, and int8 values decoded from uint32 storage.
Fixes#2482
* [BugFix] Reject unsupported T.serial access to thread-distributed fragments with clear diagnostics
A T.serial loop runs sequentially inside each CUDA thread, but a fragment's
elements are distributed across threads, so indexing a fragment by a serial
loop variable has no valid thread-ownership mapping. Three fuzzer reports share
this root cause:
- #2393: serial reduce of a GEMM-output fragment -> unhelpful internal error
("contains inner var j").
- #2395: serial write into a fragment then T.copy out -> nvcc "identifier
undefined" (an owner-thread guard built from serial loop vars was emitted
outside the loop that binds them).
- #2396: serial write of a fragment to global -> silently wrong (threads
race-write each cell).
AddWrapperForSingleBufStore wraps bare fragment stores in a degenerate
T.parallel(1) and only rejected constant non-zero fragment indices, so a
dynamic (e.g. serial-loop) index slipped through and lowered to broken code
(#2395/#2396). Reject any fragment index that is not the constant all-zero
index in this fallback path, with a message pointing to T.Parallel /
T.reduce_sum. Rephrase the existing inner-var layout check (#2393) to explain
the cause instead of dumping "contains inner var".
Also collect every access to a buffer in a statement (not just the last) so the
diagnostic is not bypassed when one fragment is accessed at both a constant and
a variable index in the same statement; without it such a case falls through to
a cryptic downstream StructuralEqual internal error. This is a completeness
hardening only, not required by the three issues above.
Fixes#2393Fixes#2395Fixes#2396
* Generalize fragment diagnostic to non-parallel loops
inner_vars_ is populated for every non-parallel For (serial, unroll,
vectorized), so the message must not hard-code T.serial. Reword to
"non-parallel loop variable" per review feedback.
* Narrow fix scope to #2393 only
Remove fragment index validation from AddWrapperForSingleBufStore and
delete tests for #2395/#2396. These issues cannot be caught at this pass
stage because T.grid and T.serial are indistinguishable (both appear as
ForKind.SERIAL before LowerTileOp).
Keep only the ParallelOpNode diagnostic improvement which successfully
catches #2393.
* [BugFix] Fix T.Persistent dropping tiles when last dim is not a multiple of group_size (#2433)
* Update src/ir.cc
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Fix eager JIT packed FP4 shape binding
* Add logical shape and stride calculations for buffers in eager builder
This update introduces helper functions to compute logical shape and stride values for buffers, enhancing the handling of packed storage factors. The changes streamline the process of determining dimensions and strides in the context of tensor operations, improving overall efficiency and clarity in the eager execution flow.
---------
Co-authored-by: LeiWang1999 <leiwang1999@outlook.com>
* Place auto WS producers in first warp group
* [Transform] Update layout inference to bind thread range for fully replicated fragments; add tests for TMA fragment mask copy and producer-consumer warp specialization
* [Refactor] Update thread variable calculation to use zero-based indexing across GEMM operations
* [Transform] Enhance simplification process with ContextSingletonFloorDivFolder and add tests for floor division simplification
* [Refactor] Simplify tensor conversion in dequantization functions and improve performance in GEMM examples
* [Backend] Refactor Pipeline to support different backends
Introduce Pipeline abstraction in backend/pipeline.py with per-backend
registration. Each backend (cuda, hip, c, llvm) now registers its own
compilation pass pipeline. engine/lower.py resolves the pipeline from
the target instead of hardcoding phase imports.
* enhance the structure
* fuse two stage
* fix after rebase
* update template pass pipeline
* pipeline
* fix test
* fix test
* fix cuda pipeline prologue
* [Fix] Suppress spurious __syncthreads() between T.ws blocks (#2192) and add T.named_barrier_arrive (#2188)
Issue #2192 - BugFix: Unexpected CTA-wide __syncthreads() between T.ws producer/consumer
- Root cause: FindConflict() in thread_storage_sync.cc did not check for mutually
exclusive thread constraints. When T.ws(0) (tx<128) reads and T.ws(1) (tx>=128)
writes the same shared buffer, no thread satisfies both constraints, so no
__syncthreads() is needed — mbarriers already handle the synchronization.
- Fix: add mutual-exclusivity check (prove NOT(P AND C)) early in FindConflict().
If the access constraints are mutually exclusive (disjoint thread sets), return
false (no conflict) immediately, before any index-range analysis.
- Measured improvement: barrierpipe_stage2 gemm latency 44ms → 25ms on H200.
Issue #2188 - Feature: expose native T.named_barrier_arrive(barrier_id, thread_count)
- Adds tl.named_barrier_arrive intrinsic op (bar.arrive PTX, one-sided arrive).
- Registers op in builtin.cc, declares in builtin.h.
- Adds tl::__named_barrier_arrive<id, cnt>() template in common.h.
- Adds codegen case in codegen_cuda.cc.
- Exposes T.named_barrier_arrive() in tilelang/language/builtin.py.
- Adds regression test testing/python/issue/test_tilelang_issue_2188_2192.py.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* lint: clang-format + ruff fixes after #2188/#2192 changes
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* test: simplify issue 2188/2192 test, keep only #2192, add requires_cuda
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [Fix] Make named_barrier_arrive accept variable barrier_id/thread_count
* Update documentation for named barriers and remove obsolete test file
- Clarified documentation for `named_barrier_arrive` in `builtin.h` and `builtin.py` to specify the barrier index range.
- Removed obsolete regression test for issues #2188 and #2192 as it is no longer needed.
* lint fix
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: LeiWang1999 <leiwang1999@outlook.com>
* [Bugfix] Fix issue 2123 access_ptr lowering
Add the issue 2123 regression test and keep the access_ptr base BufferLoad intact through safe-memory legalization and lowering.
Closes#2123
* Deduplicate access_ptr base load handling
Move shared tl.access_ptr BufferLoad base visitation into a common transform helper so LowerAccessPtr and safe-memory legalization stay aligned.
Extend the issue 2123 regression with a direct LowerAccessPtr pass check while keeping the LowerAndLegalize pipeline coverage.
* Format access_ptr helper header
* Narrow access_ptr helper scope
* Share access_ptr base helper
* Address access_ptr helper review comments
* [Transform] Preserve access_ptr buffer loads in safe memory legalization
* Refactor AccessPtrInfo return structure in legalize_safe_memory_access.cc
Updated the return structure of AccessPtrInfo to improve readability and maintainability. The logic for handling the tvm_access_ptr operation has been streamlined, ensuring clearer checks and consistent formatting of returned values.
* fix
---------
Co-authored-by: LeiWang1999 <leiwang1999@outlook.com>
* [Feature] Implement Hoist Loop-Invariant If Pass
- Added a new pass to hoist loop-invariant if statements out of loops, improving optimization opportunities.
- Introduced classes for collecting and checking conditions, as well as for rewriting statements.
- Integrated the new pass into the optimization pipeline and provided a corresponding API for usage.
* Add tilelang semantics guide to programming guides (#1740)
* Add CallNodeChecker to Loop Unswitching Pass and integrate tilelang cache disabling in benchmark script
- Introduced CallNodeChecker class to identify CallNode expressions in loop conditions, enhancing loop-invariant checks.
- Updated IsLoopInvariant function to reject conditions containing CallNodes, preventing potential side effects.
- Added tilelang.disable_cache() in benchmark_mha_sink_fwd.py to optimize performance during benchmarking.
* lint fix
* lint fix
* Enhance Loop Unswitching Pass with Let Binding Support
- Added support for Let-bound variables in the WrittenBufferReadChecker to improve buffer read checks.
- Introduced UsesLoopVarThroughLetBindings function to check if conditions depend on loop variables through Let bindings.
- Updated IsLoopInvariant function to account for Let bindings when determining loop invariance.
- Enhanced HoistableIfFinder to track Let bindings for variables bound to BufferLoad expressions.
- Added debug print statements in the OptimizeForTarget function to visualize the module state before and after loop unswitching.
* fix
* lint fix
* Enhance Loop Unswitching with CallNode Exclusion Check
- Added CallCheckerExcludingIf class to ensure function calls outside of hoisted if statements are identified, preventing potential synchronization issues during loop unswitching.
- Updated loop unswitching logic to incorporate the new call checker, enhancing safety and correctness.
- Integrated debug print statements in OptimizeForTarget to visualize module state before and after loop unswitching.
- Disabled tilelang cache in the benchmark script for improved performance.
* Update TVM subproject to latest commit and remove debug print statements in OptimizeForTarget function
* Add support for disabling loop unswitching optimization
- Introduced a new configuration option `tl.disable_loop_unswitching` to allow users to disable the loop unswitching optimization.
- Updated the Loop Unswitching pass to check this configuration and return the original function if the option is enabled.
- Added relevant documentation in the PassConfigKey enumeration for clarity.
* update tvm
* Refactor: Update TileLang to use tirx instead of tir
- Removed references to `tvm.tir` and replaced them with `tvm.tirx` across various files, including examples and backend operations.
- Updated target configuration in documentation to reflect the new usage of target config dictionaries instead of CLI-style strings.
- Cleaned up `pyproject.toml` and `load_tvm.cmake` by removing obsolete paths.
- Enhanced examples for dequantization and GEMM operations to utilize the new `tirx` constructs.
- Adjusted various backend operations to ensure compatibility with the new `tirx` namespace.
* pass fix
* Update submodule 'tvm' to latest commit f24c7b19
* Apply formatting fixes
* Update CMakeLists.txt to set HIDE_PRIVATE_SYMBOLS for TileLang and modify target_host initialization in lower.py for compatibility with TVM's target handling.
* Refactor CMakeLists.txt and kernel_cache.py to update target names and library dependencies for TVM integration. Adjusted output targets and library names for better compatibility across platforms.
* Update repair-wheel-command in pyproject.toml to exclude libz3.so with wildcard for better compatibility with dependency packages.
* Fix macOS Z3 dylib lookup
* Move macOS Z3 install-name fix to install step
* Update TVM submodule for runtime import fix
* Update ROCm backend for TVM module factory changes
* Fix build-tree Z3 runtime lookup
* Fix macOS build-tree Z3 lookup
* Support legacy target option strings
* Normalize CUDA target in transform test
* Qualify Metal stream FFI lookup
* Fix PyPI Z3 packaging links
* Update TVM for Windows DLL lookup
* Register Windows dependency DLL directories
* Bump tvm-ffi to match tirx
* tvm update
* Apply lint fixes
* Enhance reduction operations with support for nan propagation and vectorization. Introduce new functions for preferred vectorized size and update existing reduction logic to handle packed operations for bfloat16 and float16 types. Add nan-aware min and max operations in CUDA and ROCm backends, and update related tests to validate functionality.
* Refactor ReduceLowerer: Remove unnecessary buffer allocation logic for packed operations in reduce.h
* Normalize PyPI Z3 dylib before linking
* Drop legacy target option string support
* Fix type reference for barrier annotations in CUDA copy operations. Updated `BufferLoadNode` to `tirx::BufferLoadNode` in `GetBarrier` and `LowerCluster` methods to ensure compatibility with recent changes in the TIR API.
---------
Co-authored-by: Yu Cheng <yu.cheng@pku.edu.cn>
Co-authored-by: Yichen Yan <oraluben@outlook.com>
