Commit Graph
99 Commits
Author SHA1 Message Date
YYDHandSiriusNEO a2e02cc5cd [Bugfix][CUDA] Fix UB in packed 8-bit vector stores (#3092)
* [Bugfix][CUDA] Fix UB in packed 8-bit vector stores

* test(cuda): check int8x32 field initialization

---------

Co-authored-by: SiriusNEO <chaofan@deepseek.com>
2026-08-27 19:47:41 +08:00
Wenzhe WangandSiriusNEO 469c5d5cb8 [Bugfix] Validate T.gemm k_pack arguments (#3094)
* [Bugfix] Validate T.gemm k_pack arguments

* [Bugfix] Improve T.gemm k_pack diagnostic

* [Bugfix] Accept integral T.gemm k_pack values

* [Bugfix] Simplify T.gemm k_pack validation

---------

Co-authored-by: SiriusNEO <chaofan@deepseek.com>
2026-08-27 19:33:26 +08:00
Sam Sui e8b9da92d2 [BugFix][CUDA] Support int4x2 and uint4x2 codegen (#3036)
* fix(cuda): support int4x2 and uint4x2 codegen (#2994, #2995)

* fix(cuda): handle packed int4 lane access

* fix(cuda): handle packed 4-bit x2 construction

* fix(cuda): load packed x2 gathers by logical index

* fix(cuda): validate packed int4 vector alignment

* fix(cuda): preserve packed vector load fallback
2026-08-25 14:48:53 +08:00
Zupeng Wang f436610f7d [Language] Honor declared strides in pointer helpers (#3073)
Signed-off-by: Zupeng Wang <71580390+zupengwang@users.noreply.github.com>
2026-08-24 01:16:09 +08:00
Diao Han 78a4ff6587 [BugFix][Hopper] Select the widest legal WGMMA N instead of gcd (#2931)
* [BugFix][Hopper] Select the widest legal WGMMA N instead of gcd

block_N extents that are legal single WGMMA widths but not powers of two
(96, 112, 160, 176) were decomposed into n32 / n16 instructions, because
the instruction width came from gcd(warp_col_tiles, 256). wgmma.h already
instantiates every multiple of 8 up to 256, so only the width selection
was the constraint.

Select the widest legal width that tiles warp_col_tiles exactly, keeping
gcd as the fallback. Widening is restricted to multiples of 16: the
N % 16 == 8 class already miscomputes (#2593), and the integer instruction
tables only instantiate multiples of 16 (plus 8 and 24). The result is
never narrower than before, so no existing shape regresses.

The sparse generator carried the same line and now shares the helper.

* [Testing] Cover T.gemm_sp at the widened WGMMA N values

The width selector is shared with the sparse generator, so assert the emitted
wgmma_sp instruction and the sparse numeric result for block_N in {96, 112,
160} as well. Verified as a real regression: with the selector reverted the
sparse path emits n32 / n16 / n32 exactly like the dense one.
2026-08-11 17:20:51 +08:00
edragainandSiriusNEO 7cabeed7c9 [BugFix] Fix failed ties-away-from-zero round compile on bfloat16/float8 (#2873)
* [BugFix] Fix failed ties-away-from-zero round compile on bfloat16/float8

* add helper function on hip

* [Test] Check ties-away rounding helper codegen

---------

Co-authored-by: SiriusNEO <chaofan@deepseek.com>
2026-08-10 16:44:34 +08:00
Yuxi Chi 7fc47066cd [CuTeDSL] Port backend intrinsics to CUTLASS DSL primitives (#2871)
* Port CuTeDSL backend to CUTLASS primitives

* Update CuTeDSL dependency to CUTLASS DSL 4.7.0

* Fix CuTeDSL dynamic-shape wrapper generation
2026-08-08 18:00:57 +08:00
Chaofan Lin 27224884f9 [Testing] Run portable regression tests on auto targets (#2914)
[Testing] Remove stale CUDA gates from portable tests
2026-08-07 15:07:26 +08:00
andyluo7 02cf41f178 [Testing] Gate the issue #2883 regression test on CUDA (#2908)
The test added in #2899 lowers with an explicit `tvm.target.Target("cuda")`, which
reaches `tilelang/cuda/transform/__init__.py`. In a `USE_CUDA=OFF` build only the
"ALWAYS" CUDA sources compile, so `AnnotateDeviceBoundTmaCopies` is never registered
and the test fails on ROCm:

    AttributeError: module 'tilelang.cuda._ffi_api' has no attribute
                    'AnnotateDeviceBoundTmaCopies'

Same pattern as #2863. `requires_cuda` is the weakest sufficient gate: the test is
compile-only (`enable_device_compile=False`), so it needs no particular GPU
generation and keeps running on the NVIDIA leg.

Verified in CI: SKIPPED on the gfx942 leg, PASSED on CUDA-auto.
2026-08-06 19:25:53 -07:00
Chaofan Lin f779a4ef8c [Test] Add regression test for issue #2883 (#2899) 2026-08-06 16:52:03 +08:00
Jinpeng Wang 6873e19da2 [BugFix] Handle vectorized SelectNode in codegen_cuda (#2843)
* [BugFix] Handle vectorized SelectNode in codegen_cuda

* address review comments

* [BugFix] Fix cuda codegen vectorized SelectNode when condition is a scalar
2026-08-05 13:51:31 +08:00
andyluo7 dd9c20dbf7 [Testing] Gate CUDA-only tests so non-CUDA backends can run the suite (#2863)
A large number of tests import CUDA-only internals directly (for example
`tilelang.cuda.pipeline.CUDAPassPipelineBodyPrologue`) or exercise
NVIDIA-specific features such as TMA, LDGSTG, mbarrier, warp specialization,
tcgen05 and tfloat32. `src/cuda/CMakeLists.txt` only compiles a small
"ALWAYS" subset when `USE_CUDA=OFF`, so those FFI symbols are never
registered and the tests fail with:

    AttributeError: module 'tilelang.cuda._ffi_api' has no attribute
                    'AnnotateDeviceBoundTmaCopies'

This is not ROCm-specific -- the same tests fail on any build without the
CUDA backend, including Metal and CPU-only.

Add `@tilelang.testing.requires_cuda` to those tests, plus three
portability fixes that keep coverage on non-CUDA backends instead of
skipping it:

* `test_tilelang_transform_layout_inference.py`: CUDA emits
  `__launch_bounds__(N, 1)` while HIP emits `__launch_bounds__(N)`.
  Match both.
* `test_tilelang_transform_pipeline_planning.py`: PipelinePlanning
  deliberately strips pipeline annotations on ROCm targets other than
  gfx950 (see the guard in src/transform/pipeline_planning.cc), so assert
  the documented sequential fallback there.
* `test_tilelang_kernel_gemm.py`: the two f32f32f32 cases use `T.tfloat32`,
  which is NVIDIA-only, so narrow `requires_cuda_or_cdna` to
  `requires_cuda`.

Also refresh three stale RDNA tests: gfx12 is now a supported generation
(`get_arch` accepts 11 and 12), so they can no longer assert that gfx1200
is rejected. Force an unsupported generation instead, which is the only
way to reach the guard now that `target_is_rdna` reports only gfx11/gfx12.

Verified on 8x MI300X (gfx942, ROCm 7.2 container, torch nightly rocm7.2):
the ROCm suite goes from 105 failures to 0 with no loss of passing tests.
2026-08-04 18:22:40 +08:00
jpwang 03645a6483 [BugFix] Use masked warpsync in in-warp allreduce (#2865)
* [BugFix] Use masked warpsync in in-warp allreduce

* Add more tests to make coderabbit happy
2026-08-04 17:51:23 +08:00
jpwang 4be24b75bc [BugFix] Fix absmax and abssum for uint dtypes (#2845)
* [BugFix] Fix absmax and abssum for uint dtypes

* use parametrize tests

* Resolve review comments
2026-08-03 17:16:55 +08:00
Ruinan LiandClaude Fable 5 5b1f321847 [BugFix] Reject uncovered warp partitions in T.gemm instead of producing silently wrong results (#2724)
* fix: resolve issue 2537

* test: pin default-search regression tests to the MMA backend

On sm_90 SelectInst reroutes M>=64 / warps%4==0 shapes to the WGMMA
search, so two tests written against the default search exercised the
wrong code path on Hopper CI. The FullCol M=192 N=104 numeric test
additionally hit a pre-existing, partition-independent WGMMA
descriptor-layer limitation (N=104 fits no canonical GMMA swizzle atom).
Pin both to the MMA path via tl.disable_wgmma; the WGMMA search keeps
its own dedicated tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-30 13:53:03 +08:00
Tong WU 32e02e6c72 [Fix] Refine architecture guards (#2790)
* fix: refine TCGEN05 architecture guards

* test: consolidate CUDA intrinsic architecture guards

* test: remove CUDA device helper architecture guard tests
2026-07-29 15:21:50 +08:00
Tong WU b5e3eb93ef [Enhancement] More compile-time guards for architecture-specific CUDA intrinsics (#2781)
fix: guard architecture-specific CUDA intrinsics
2026-07-27 21:11:59 +08:00
Chaofan Lin 8bb3300d60 [BugFix] Preserve if condition evaluation during fan-out (#2764)
* [BugFix] Preserve if condition evaluation during fan-out

* [BugFix] Avoid snapshotting replayable if conditions
2026-07-25 14:30:13 +08:00
Chaofan Lin e9240d6816 [Refactor] Move example-only helpers out of tilelang package (#2761) 2026-07-24 18:03:34 +08:00
Zheng YuandZheng Yu 1591d368db [BugFix] Preserve re-evaluation of mutable if conditions (#2744)
Avoid merging ifs with mutable conditions

Co-authored-by: Zheng Yu <17133261+cla7aye15I4nd@users.noreply.github.com>
2026-07-24 17:09:08 +08:00
xiaoyouPREG 192ddea60f [BugFix] Gate TMEM and TMA builtins by CUDA architecture (#2743) 2026-07-23 14:22:05 +08:00
ChennesandLeiWang1999 a443dde926 [BugFix] Reject contracting shared-buffer layouts in T.annotate_layout (#2719)
* [BugFix] Reject contracting shared-buffer layouts in T.annotate_layout

T.annotate_layout could accept a contracting many-to-one forward map on a
shared buffer. makeBufferWithLayout then misread the smaller output extent as
a replication factor, rewrote the buffer to a higher rank, and eventually
failed with an unrelated BufferStore dimensionality ICHECK.

Add a pigeonhole check for static layout shapes before the replication branch.
When the layout output extent is smaller than its input extent, raise a
user-facing ValueError that identifies the buffer and reports both extents.
Keep identity, permutation, and output-growing layouts unchanged.

This check intentionally targets contracting maps; general injectivity
analysis for maps with larger output bounding boxes is outside this change.

Fixes #2632

* [Layout] Enforce injective shared-buffer annotations

Add a LayoutNode injectivity check with an exact fallback for static domains, and reuse it for FragmentNode thread and replication coordinates. Validate shared-buffer annotations after layout reshaping so sparse many-to-one maps are rejected with a useful collision diagnostic while padded injective maps remain valid.

---------

Co-authored-by: LeiWang1999 <leiwang1999@outlook.com>
2026-07-23 11:21:48 +08:00
Lei Wang dff136d4da [CUDA][Pipeline] Fix 1D TMA selection for versioned layouts (#2737)
Fix pipelined 1D TMA layout detection
2026-07-22 21:38:18 +08:00
Liu Yunuo f8d8cd4b14 [BugFix] Use blockDim as workspace stride in batch AllReduce (#2621)
* fix: use blockDim as workspace stride in batch AllReduce

* add test
2026-07-21 18:46:14 +08:00
lianjinfeng2003 ac576c6356 [BugFix] Fix FP4 dequant symbolic exponent clamp (#2656)
* [BugFix] Fix FP4 dequant symbolic exponent clamp

* [BugFix] Align quantization regression
2026-07-21 18:23:49 +08:00
xiaoyouPREG ffeda9f394 [BugFix] Pack 32-lane 8-bit CUDA vectors correctly (#2701)
* [BugFix] Pack 32-lane 8-bit CUDA vectors correctly

* [Test] Avoid SM100-only store in int8 broadcast regression

* [BugFix] Move int8x32 packing into CUDA templates
2026-07-21 13:36:03 +08:00
wcxandLeiWang1999 8cdd4d62f3 [BugFix] Prevent partial 1-D TMA stores from bypassing bounds checks (#2716)
* [BugFix] Prevent partial 1-D TMA stores from bypassing bounds checks

Signed-off-by: UnsettingGalaxy <chenxuw641@gmail.com>

* [BugFix] Prevent partial 1-D TMA stores from bypassing bounds checks

Signed-off-by: UnsettingGalaxy <chenxuw641@gmail.com>

* [BugFix] Derive TMA copy bounds during selection

Remove buffer_oob from generic layout inference arguments and derive copy-region containment from the active analyzer in both inference and lowering. Use descriptor TMA for partial tiles and add rank-1 tail regression coverage.

---------

Signed-off-by: UnsettingGalaxy <chenxuw641@gmail.com>
Co-authored-by: LeiWang1999 <leiwang1999@outlook.com>
2026-07-20 19:42:49 +08:00
Liu Yunuo 0c7f14c3f9 [BugFix] Fix WGMMA C-store layout for multiple warpgroups along M (#2663)
* fix wgmma store layout

* fix wgmma_sp the same way, update testing requirements
2026-07-17 17:09:22 +08:00
xiaoyouPREG 9754ac4471 [BugFix] Preserve bf16 NaN and Inf during RNE packing (#2690) 2026-07-17 13:50:39 +08:00
Lyscoria 31755e7138 [BugFix] Support runtime-dependent vector negative indices (#2654) 2026-07-16 18:30:46 +08:00
Chennes c8c49d5025 [BugFix] Reject invalid dtypes in T.dp4a (#2652)
T.dp4a previously accepted arbitrary buffer dtypes and passed them to backend DP4A implementations, which could silently reinterpret the operand bytes and produce incorrect results.

Require both inputs to use int8 and the accumulator to use int32. Add regression coverage for valid Buffer and BufferLoad forms and for invalid input and accumulator dtypes.

Fixes #2600
2026-07-13 17:13:24 +08:00
Chennes e2c2c207e6 [BugFix] Sign-extend packed uint32 signed decode (#2500)
_tir_u32_to_int_to_float decoded signed sub-word fields from uint32 storage as unsigned values because it only masked the selected field before casting. As a result, negative encodings such as signed int4 `0xF` decoded as `15.0` instead of `-1.0`.

Sign-extend the extracted field through int32 before casting to the requested float dtype, matching the existing packed signed-int decode helper.

Add CUDA regression coverage for signed int2, int4, and int8 values decoded from uint32 storage.

Fixes #2482
2026-07-02 13:12:56 +08:00
Chennes ed00dfcd7f [BugFix] Improve diagnostic for T.serial fragment access (#2462)
* [BugFix] Reject unsupported T.serial access to thread-distributed fragments with clear diagnostics

A T.serial loop runs sequentially inside each CUDA thread, but a fragment's
elements are distributed across threads, so indexing a fragment by a serial
loop variable has no valid thread-ownership mapping. Three fuzzer reports share
this root cause:

- #2393: serial reduce of a GEMM-output fragment -> unhelpful internal error
  ("contains inner var j").
- #2395: serial write into a fragment then T.copy out -> nvcc "identifier
  undefined" (an owner-thread guard built from serial loop vars was emitted
  outside the loop that binds them).
- #2396: serial write of a fragment to global -> silently wrong (threads
  race-write each cell).

AddWrapperForSingleBufStore wraps bare fragment stores in a degenerate
T.parallel(1) and only rejected constant non-zero fragment indices, so a
dynamic (e.g. serial-loop) index slipped through and lowered to broken code
(#2395/#2396). Reject any fragment index that is not the constant all-zero
index in this fallback path, with a message pointing to T.Parallel /
T.reduce_sum. Rephrase the existing inner-var layout check (#2393) to explain
the cause instead of dumping "contains inner var".

Also collect every access to a buffer in a statement (not just the last) so the
diagnostic is not bypassed when one fragment is accessed at both a constant and
a variable index in the same statement; without it such a case falls through to
a cryptic downstream StructuralEqual internal error. This is a completeness
hardening only, not required by the three issues above.

Fixes #2393
Fixes #2395
Fixes #2396

* Generalize fragment diagnostic to non-parallel loops

inner_vars_ is populated for every non-parallel For (serial, unroll,
vectorized), so the message must not hard-code T.serial. Reword to
"non-parallel loop variable" per review feedback.

* Narrow fix scope to #2393 only

Remove fragment index validation from AddWrapperForSingleBufStore and
delete tests for #2395/#2396. These issues cannot be caught at this pass
stage because T.grid and T.serial are indistinguishable (both appear as
ForKind.SERIAL before LowerTileOp).

Keep only the ParallelOpNode diagnostic improvement which successfully
catches #2393.
2026-06-28 13:05:51 +08:00
Ftoandcoderabbitai[bot] ab083cb3c6 [BugFix] Fix T.Persistent dropping tiles when last dim is not a multiple of group_size (#2455)
* [BugFix] Fix T.Persistent dropping tiles when last dim is not a multiple of group_size (#2433)

* Update src/ir.cc

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2026-06-26 14:12:53 +08:00
Lei Wang dfc7e953ed [Pipeline] Simplify async copy lowering (#2444)
* Clean up async copy pipeline lowering

* FIX

* enhance

* refactor pipeline
2026-06-24 15:11:05 +08:00
Chaofan Lin 2913ad3f00 [Backend] Cleanup Metal Codegen, split AsyncCopy lowering and common target utils (#2361)
* [Backend] Cleanup Metal Codegen and split common target utils

* [Backend] Preserve PTX async copy pass docs

* [Backend] Split backend CMake source ownership

* [Backend] Split target warp size helpers
2026-06-09 20:20:25 +08:00
Lei Wang 66ef211815 [TIR][Transform] Partition parallel loops with fragment access (#2340)
Fix fragment parallel loop partitioning
2026-06-05 14:37:46 +08:00
Tong WUandLeiWang1999 c2998920cc [BugFix] Fix eager JIT sub-btye shape binding (#2334)
* Fix eager JIT packed FP4 shape binding

* Add logical shape and stride calculations for buffers in eager builder

This update introduces helper functions to compute logical shape and stride values for buffers, enhancing the handling of packed storage factors. The changes streamline the process of determining dimensions and strides in the context of tensor operations, improving overall efficiency and clarity in the eager execution flow.

---------

Co-authored-by: LeiWang1999 <leiwang1999@outlook.com>
2026-06-05 13:53:12 +08:00
Lei Wang d988dd0491 [Transform] Place auto WS producers in first warp group (#2315)
* Place auto WS producers in first warp group

* [Transform] Update layout inference to bind thread range for fully replicated fragments; add tests for TMA fragment mask copy and producer-consumer warp specialization

* [Refactor] Update thread variable calculation to use zero-based indexing across GEMM operations

* [Transform] Enhance simplification process with ContextSingletonFloorDivFolder and add tests for floor division simplification

* [Refactor] Simplify tensor conversion in dequantization functions and improve performance in GEMM examples
2026-06-04 14:47:52 +08:00
Chaofan Lin 8d315be580 [Backend][Transform] Move backend-specific transforms into separate namespaces (#2297)
* [Backend][Transform] Move backend-specific transforms into separate namespaces

* cleanup

* fix

* Restore transform pass docstrings
2026-06-02 11:25:10 +08:00
Chaofan Lin 4f0e127de2 [Backend] Promote PassPipeline to backend sub-folder and cleanup Metal Leftover (#2291)
* [Backend] Promote PassPipeline to backend sub-folder and fix metal transform position

* cleanup metal gemm
2026-05-28 16:44:10 +08:00
Chaofan Lin 0a9b651244 [Backend] Refactor Transform Pipeline to support different backends (#2189)
* [Backend] Refactor Pipeline to support different backends

Introduce Pipeline abstraction in backend/pipeline.py with per-backend
registration. Each backend (cuda, hip, c, llvm) now registers its own
compilation pass pipeline. engine/lower.py resolves the pipeline from
the target instead of hardcoding phase imports.

* enhance the structure

* fuse two stage

* fix after rebase

* update template pass pipeline

* pipeline

* fix test

* fix test

* fix cuda pipeline prologue
2026-05-28 04:11:19 +08:00
c66cadf87f [Feature] Support named barrier arrive (#2194)
* [Fix] Suppress spurious __syncthreads() between T.ws blocks (#2192) and add T.named_barrier_arrive (#2188)

Issue #2192 - BugFix: Unexpected CTA-wide __syncthreads() between T.ws producer/consumer
  - Root cause: FindConflict() in thread_storage_sync.cc did not check for mutually
    exclusive thread constraints. When T.ws(0) (tx<128) reads and T.ws(1) (tx>=128)
    writes the same shared buffer, no thread satisfies both constraints, so no
    __syncthreads() is needed — mbarriers already handle the synchronization.
  - Fix: add mutual-exclusivity check (prove NOT(P AND C)) early in FindConflict().
    If the access constraints are mutually exclusive (disjoint thread sets), return
    false (no conflict) immediately, before any index-range analysis.
  - Measured improvement: barrierpipe_stage2 gemm latency 44ms → 25ms on H200.

Issue #2188 - Feature: expose native T.named_barrier_arrive(barrier_id, thread_count)
  - Adds tl.named_barrier_arrive intrinsic op (bar.arrive PTX, one-sided arrive).
  - Registers op in builtin.cc, declares in builtin.h.
  - Adds tl::__named_barrier_arrive<id, cnt>() template in common.h.
  - Adds codegen case in codegen_cuda.cc.
  - Exposes T.named_barrier_arrive() in tilelang/language/builtin.py.
  - Adds regression test testing/python/issue/test_tilelang_issue_2188_2192.py.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* lint: clang-format + ruff fixes after #2188/#2192 changes

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* test: simplify issue 2188/2192 test, keep only #2192, add requires_cuda

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* [Fix] Make named_barrier_arrive accept variable barrier_id/thread_count

* Update documentation for named barriers and remove obsolete test file

- Clarified documentation for `named_barrier_arrive` in `builtin.h` and `builtin.py` to specify the barrier index range.
- Removed obsolete regression test for issues #2188 and #2192 as it is no longer needed.

* lint fix

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: LeiWang1999 <leiwang1999@outlook.com>
2026-05-26 12:57:52 +08:00
VitalyRandLeiWang1999 b1083d72b1 Fix atomic_load access_ptr lowering for dynamic indices (#2157)
* [Bugfix] Fix issue 2123 access_ptr lowering

Add the issue 2123 regression test and keep the access_ptr base BufferLoad intact through safe-memory legalization and lowering.

Closes #2123

* Deduplicate access_ptr base load handling

Move shared tl.access_ptr BufferLoad base visitation into a common transform helper so LowerAccessPtr and safe-memory legalization stay aligned.

Extend the issue 2123 regression with a direct LowerAccessPtr pass check while keeping the LowerAndLegalize pipeline coverage.

* Format access_ptr helper header

* Narrow access_ptr helper scope

* Share access_ptr base helper

* Address access_ptr helper review comments

* [Transform] Preserve access_ptr buffer loads in safe memory legalization

* Refactor AccessPtrInfo return structure in legalize_safe_memory_access.cc

Updated the return structure of AccessPtrInfo to improve readability and maintainability. The logic for handling the tvm_access_ptr operation has been streamlined, ensuring clearer checks and consistent formatting of returned values.

* fix

---------

Co-authored-by: LeiWang1999 <leiwang1999@outlook.com>
2026-05-25 13:20:47 +08:00
b939fa0137 [TIR][IR] Update to use tirx (#2216)
* [Feature] Implement Hoist Loop-Invariant If Pass

- Added a new pass to hoist loop-invariant if statements out of loops, improving optimization opportunities.
- Introduced classes for collecting and checking conditions, as well as for rewriting statements.
- Integrated the new pass into the optimization pipeline and provided a corresponding API for usage.

* Add tilelang semantics guide to programming guides (#1740)

* Add CallNodeChecker to Loop Unswitching Pass and integrate tilelang cache disabling in benchmark script

- Introduced CallNodeChecker class to identify CallNode expressions in loop conditions, enhancing loop-invariant checks.
- Updated IsLoopInvariant function to reject conditions containing CallNodes, preventing potential side effects.
- Added tilelang.disable_cache() in benchmark_mha_sink_fwd.py to optimize performance during benchmarking.

* lint fix

* lint fix

* Enhance Loop Unswitching Pass with Let Binding Support

- Added support for Let-bound variables in the WrittenBufferReadChecker to improve buffer read checks.
- Introduced UsesLoopVarThroughLetBindings function to check if conditions depend on loop variables through Let bindings.
- Updated IsLoopInvariant function to account for Let bindings when determining loop invariance.
- Enhanced HoistableIfFinder to track Let bindings for variables bound to BufferLoad expressions.
- Added debug print statements in the OptimizeForTarget function to visualize the module state before and after loop unswitching.

* fix

* lint fix

* Enhance Loop Unswitching with CallNode Exclusion Check

- Added CallCheckerExcludingIf class to ensure function calls outside of hoisted if statements are identified, preventing potential synchronization issues during loop unswitching.
- Updated loop unswitching logic to incorporate the new call checker, enhancing safety and correctness.
- Integrated debug print statements in OptimizeForTarget to visualize module state before and after loop unswitching.
- Disabled tilelang cache in the benchmark script for improved performance.

* Update TVM subproject to latest commit and remove debug print statements in OptimizeForTarget function

* Add support for disabling loop unswitching optimization

- Introduced a new configuration option `tl.disable_loop_unswitching` to allow users to disable the loop unswitching optimization.
- Updated the Loop Unswitching pass to check this configuration and return the original function if the option is enabled.
- Added relevant documentation in the PassConfigKey enumeration for clarity.

* update tvm

* Refactor: Update TileLang to use tirx instead of tir

- Removed references to `tvm.tir` and replaced them with `tvm.tirx` across various files, including examples and backend operations.
- Updated target configuration in documentation to reflect the new usage of target config dictionaries instead of CLI-style strings.
- Cleaned up `pyproject.toml` and `load_tvm.cmake` by removing obsolete paths.
- Enhanced examples for dequantization and GEMM operations to utilize the new `tirx` constructs.
- Adjusted various backend operations to ensure compatibility with the new `tirx` namespace.

* pass fix

* Update submodule 'tvm' to latest commit f24c7b19

* Apply formatting fixes

* Update CMakeLists.txt to set HIDE_PRIVATE_SYMBOLS for TileLang and modify target_host initialization in lower.py for compatibility with TVM's target handling.

* Refactor CMakeLists.txt and kernel_cache.py to update target names and library dependencies for TVM integration. Adjusted output targets and library names for better compatibility across platforms.

* Update repair-wheel-command in pyproject.toml to exclude libz3.so with wildcard for better compatibility with dependency packages.

* Fix macOS Z3 dylib lookup

* Move macOS Z3 install-name fix to install step

* Update TVM submodule for runtime import fix

* Update ROCm backend for TVM module factory changes

* Fix build-tree Z3 runtime lookup

* Fix macOS build-tree Z3 lookup

* Support legacy target option strings

* Normalize CUDA target in transform test

* Qualify Metal stream FFI lookup

* Fix PyPI Z3 packaging links

* Update TVM for Windows DLL lookup

* Register Windows dependency DLL directories

* Bump tvm-ffi to match tirx

* tvm update

* Apply lint fixes

* Enhance reduction operations with support for nan propagation and vectorization. Introduce new functions for preferred vectorized size and update existing reduction logic to handle packed operations for bfloat16 and float16 types. Add nan-aware min and max operations in CUDA and ROCm backends, and update related tests to validate functionality.

* Refactor ReduceLowerer: Remove unnecessary buffer allocation logic for packed operations in reduce.h

* Normalize PyPI Z3 dylib before linking

* Drop legacy target option string support

* Fix type reference for barrier annotations in CUDA copy operations. Updated `BufferLoadNode` to `tirx::BufferLoadNode` in `GetBarrier` and `LowerCluster` methods to ensure compatibility with recent changes in the TIR API.

---------

Co-authored-by: Yu Cheng <yu.cheng@pku.edu.cn>
Co-authored-by: Yichen Yan <oraluben@outlook.com>
2026-05-20 15:38:13 +08:00
Lei Wang 256fbc6e60 [Pipeline] Fix guarded TMA pipeline handling (#2224)
* Fix guarded TMA pipeline handling

* Apply ruff fixes for latest main
2026-05-19 15:53:27 +08:00
f11954cb36 [Backend] Refactor gemm_sp (#2048)
* [Refactor] refactor gemm_sp following 5d729eee

* [Doc] update doc

* [Refactor] remove gemm_sp CUTLASS templates

* [cute] add mma sp tempaltes for sm80

* [templates] use templates in codegen

* Add WGMMA_SP templates

* Pass layout_map to py lowering

* Fix type

* Add wgmma.sp checking

* Implement wgmma_sp_ss

* Add wgmma_sp_rs

* Add sparse selector

* Fix layout and param pass

* Fix transpose metadata

* Add mma.sp fp8

* Fix integer rounding

* Fix metadata layout

* Update compress and test cases

* Remove debug print statements from sparse compress and wgmma emitter

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Clean up gemm_sp test file

- Remove unused imports (tvm, _TORCH_DTYPE_TO_STR)
- Remove print("pass") calls
- Normalize all parametrize tuples to compact aligned format

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Remove unused tvm import from gemm_sp_wgmma

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Update examples, tests, and docs for new compress() API

- Replace removed compress_sm90 with compress() in sparse_tensorcore
  example and test_compress_utils
- Fix example_gemm_sp.py: remove ARCH_INFO and make_cutlass_metadata_layout;
  compress() now produces natural-layout int16 metadata on all architectures
- Update matmul_sparse.md to reflect the simplified compress() API and
  remove the CUTLASS layout annotation requirement

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Refactor sparse constants into sparse_config.py and clean up examples

- Add tilelang/utils/sparse_config.py as single source of truth for
  SPARSE_PARAMS, E_FACTOR_MAP, E_REPLICATE_FACTOR, and get_e_factor()
- Remove inline E_FACTOR_MAP/E_REPLICATE_FACTOR from SparseTensorCoreIntrinEmitter;
  import from sparse_config.py with backward-compatible class aliases
- Update compress() API: remove transposed/block_k/arch params, use T.int16 metadata
- Delete example_custom_compress.py; move torch_compress reference to
  testing/python/utils/test_compress_utils.py with float8 .view() support
- Update examples (example_gemm_sp, tilelang_example_sparse_tensorcore) and
  test files to use new compress() API and get_e_factor()

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Compute e_factor and e_replicate_factor instead of hardcoding tables

- Remove E_FACTOR_MAP and E_REPLICATE_FACTOR dicts from sparse_config.py
- get_e_factor: derived from SPARSE_PARAMS group size and meta_dtype.bits
  (hardware always packs 4 bits per sparsity group)
- get_e_replicate_factor: derived from a_dtype.bits (<=8 bit → 1, else → 2)
- SparseTensorCoreIntrinEmitter uses the two functions directly instead of
  class-level dict aliases

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Use DataType keys in SPARSE_PARAMS

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Move sparse_config.py from utils/ to intrinsics/

The constants (SPARSE_PARAMS, get_e_factor, get_e_replicate_factor) are
hardware-level parameters primarily consumed by mma_sp_macro_generator;
intrinsics/ is the natural home. utils/sparse.py re-exports them for
backward compatibility.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Rename sparse_config.py to sparse_params.py

"params" better reflects the content: hardware sparsity format parameters
(SPARSE_PARAMS, get_e_factor, get_e_replicate_factor), not runtime config.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Fix sparse.py breaking TIR type hints by removing future annotations

from __future__ import annotations makes all annotations lazy strings,
which breaks TIR prim_func's get_type_hints() on the inner kernel.
Use Optional[torch.dtype] instead of the union syntax to avoid needing it.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Fix compress utils

* Update example

* Add a compress benchmark

* Remove print

* Remove unused MetaType

* Polish

* Refactor gemm_sp op

* Update doc && remove unused file

* Fix bug spotted by coderabbit

* Fix

* Add note

---------

Co-authored-by: LeiWang1999 <leiwang1999@outlook.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-19 13:33:41 +08:00
Chenhao Xu fa76eaf014 Allow variable barrier id in T.sync_threads() (#2197)
* fix: allow variable barrier id in T.sync_threads()

* ruff format
2026-05-18 14:50:43 +08:00
Kasper Nielsen d135bd1cd2 fix: TMA alignment to 1024 bytes on Blackwell (#2134)
* fix: TMA alignment to 1024 bytes on Blackwell

Previously 1024 byte alignemnt was guarded only by `IsHopper`.
This commit changes it to be guarded by `TargetHasBulkCopy` with the
same threshold as Hopper is using.

The kernel added in the test was failing on my RTX5090 GPU.

* Fix lint
2026-05-04 00:12:02 +08:00
74fc9806fb [Refactor] Refactor DecoupleTypeCast Pass (#2026)
* draft

* fix lint

* fix lint

* Improve pass comments

* treat loop-invariant memory access as scalar in vectorization

* Add unit test

* fix lint

* fix b[i] = a[i] + a[i+32] case

* Minor fix

* Tighten DecoupleTypeCast cast detection and add RMW regression test

Skip BufferLoad/BufferStore indices when searching for Cast nodes so an
index-type conversion does not spuriously trigger the decoupling
transformation. Clarify the load-replacement table in visit_for_ — store
entries must feed into it so RMW loads map to the store-side cast buffer —
and cover the a[i] = a[i] + a[i+32] case with a regression test.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Keep loop-invariant stores in memory bucket during vectorize planning

Commit 03bb0706 diverted all loop-invariant global/shared accesses into
the local_fragment bucket, but that bucket's constraint is dropped by the
has_global_or_shared_buffer strategy. ComputeBufferVectorSize already
returns 1 for a reduction-like store such as shared[tx] += a[...+j], yet
that 1 was silently lost, so vectorization proceeded with vector_size=2
and emitted two scalar writes to the same shared[tx] — the second
clobbered the first and dropped a lane of the accumulation.

Only loop-invariant loads (genuine broadcast reads) are safe to divert;
stores must stay in the memory bucket so their vector_size=1 constraint
is honored and the loop is left scalar.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Remove issue 1106 test

Each thread only touches its own shared[tx] slot, so no __syncthreads
is actually required and the test asserts an over-conservative behavior.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: LeiWang1999 <leiwang1999@outlook.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 15:24:33 +08:00