The kernel cache and the autotuner result cache loaded params.pkl with
cloudpickle.load(), so a writable cache directory allowed arbitrary code
execution on the next cache hit. Kernel params are plain data, so store
them as JSON via TVM's reflection (tvm.ir.save_json / load_json), which
rebuilds IR nodes without executing code. Entries without params.json are
treated as incomplete and rebuilt.
Fixes#2817
PrimExpr overloads unary minus and invert but not unary plus, so
`Out[i] = +A[i]` raised TypeError at trace time. Route unary plus through
the builder, the same way `not` already is, and return PrimExpr operands
unchanged; Python values keep their native unary plus.
Fixes#2991
* [BugFix][Blackwell] Fix 256-bit vectorized codegen for fp4/int4/fp16
256-bit vectorization (sm_100+) broke nvcc compilation for several
packed dtypes because every (dtype x lanes) combination relied on
hand-written per-type glue:
- fp4 x 64 lanes: predicated copy materializes a typed vector temporary,
which needs `fp4_e2_64_t::operator=(ulonglong4)` (never defined; the
operator was copy-pasted onto the 128-bit fp4_e2_32_t with a broken
body) and `make_fp4_e2_64_t` (constructors stop at 32 lanes).
- int4/uint4 x 64 lanes: PrintType emitted the nonexistent CUDA type
`(u)int8`, and the packed-nibble broadcast only handled 16/32 lanes.
- fp16 x 16 lanes: broadcast calls tl::pack_float16x4 whose parameters
were `__half`, but codegen prints fp16 scalars as cutlass::half_t
(the bf16 variant already took bfloat16_t).
Instead of patching each cell, make the contracts generic:
- tl::load_global_256 / _conditional now return the pointee type
(static_assert sizeof == 32) and bit-cast internally, so assignment
type-checks for any 32-byte vector struct. The hand-written
operator=(ulonglong4) on fp8_e{4,5,8}_32_t become dead and are
removed, along with the misplaced fp4_e2_32_t one. The T.ldg256
builtin keeps its uint32x8 result via an explicit ulonglong4 cast.
- New tl::broadcast<V>(unit) replicates a packed unit across any vector
width; fp4 broadcast now emits one packed fp4x2 byte + tl::broadcast
for every lane count instead of a 64-arg make_ call.
- int4x64 maps to (u)longlong4 (like int8x32) and its broadcast packs
two 8-nibble fields per 64-bit component.
Adds a compile matrix test (12 dtypes x {copy, predicated copy, fill}
x {128b, 256b}) plus numeric checks for the previously broken cells, so
the next dtype/width gap fails in CI instead of on a user's machine.
On < sm_100 targets the 256b params compile at 128 bits and stay green.
* [Refactor][CUDA] Replace per-lane-count vector constructors with variadic packers
Follow-up to #3153: the make_<type>_<N>_t constructor ladders (5 for fp4,
15 for fp8) and several BroadcastNode special cases enumerated the same
(dtype x lanes) matrix by hand, so every new dtype or vector width needed
more glue. Replace the enumeration with two generic templates:
- tl::make_vec<V>(e0, e1, ...): builds a vector struct from one scalar
element per lane (byte-sized or larger elements).
- tl::make_fp4_vec<V>(e0, e1, ...): same for nibble-packed fp4.
and route the codegen through them:
- PrintVecConstructor emits tl::make_fp4_vec<V> / tl::make_vec<V> for
fp4/fp8 vectors, covering the Shuffle and element-wise assembly paths
(whose previous output for these types did not even compile).
- fp8 and int8 broadcasts of side-effect-free values emit
tl::broadcast<carrier>(byte) instead of per-lane make_ calls; the int4
16/32/64-lane packed broadcasts collapse into a single
tl::broadcast branch.
- Side-effecting broadcast values (e.g. casts of rng calls) keep their
evaluate-once-per-lane semantics: they fall through to the generic
per-lane path, guarded via tirx::SideEffect. The int32 rng broadcast
path and the fp16/bf16/f32x8 paths are intentionally untouched.
The make_fp4_e2_{2..32}_t and make_fp8_{e4,e5,e8}_{2..32}_t ladders are
deleted; the char-packing make_int* overloads in common.h stay because
the int8 Shuffle path still assembles vectors element-wise.
Verified on sm_103a: the 12-dtype x 3-op x 2-width compile matrix stays
green, new numeric tests cover non-constant fp4/fp8/int8 scalar fills,
and the ldg/ldst/vectorize/vectorized_cast/quantize suites pass
(228 tests) with TILELANG_DISABLE_CACHE=1.
* [Test] Update issue #2480 assertions for the tl::broadcast emission
The int8x32 broadcast is now emitted as tl::broadcast<(u)longlong4>(byte)
instead of make_(u)longlong4 with 32 scalar args; the exact-value checks
that guard the original issue (upper 32 bits of each field zeroed) are
unchanged.
`for ... else` and `while ... else` inside a kernel traced without error
while the `else` body was silently dropped from the IR. Raise a clear
NotImplementedError from the AST rewriter instead; `if ... else` is
unaffected.
Fixes#2946
* [BugFix] Always vectorize `T.Parallel` loops
* Fix CI failures
* Restore always-vectorize accidentally reverted by the CI-fix commit
f447919b captured a transient working-tree state from a concurrent
baseline-vs-patched benchmark flip, reverting the gate removal in src/
while keeping the new-policy tests: CI then ran the old gated compiler
against them (scalar var-loop -> no float2; positional if/for check
broken by the copy loops the larger tile introduces).
Bring back the parameter-less LowerParallelLoop and rewrite the issue
1734 assertion as containment: the guarded loop must sit inside the
hoisted if block, which holds on both sm90 and sm100 regardless of how
many copy loops precede it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
_tir_packed_to_unsigned_convert_with_zeros performed the zero-point subtraction in the unsigned packed storage dtype before casting to the requested output dtype. Values below the zero-point therefore wrapped modulo the storage width.
Cast the unpacked field and zero-point to the output dtype before subtraction so asymmetric dequantization preserves negative values.
Add CUDA regression coverage for uint32-packed 4-bit values decoded with a non-zero zero-point.
Fixes#2947
* [BugFix][Hopper] Select the widest legal WGMMA N instead of gcd
block_N extents that are legal single WGMMA widths but not powers of two
(96, 112, 160, 176) were decomposed into n32 / n16 instructions, because
the instruction width came from gcd(warp_col_tiles, 256). wgmma.h already
instantiates every multiple of 8 up to 256, so only the width selection
was the constraint.
Select the widest legal width that tiles warp_col_tiles exactly, keeping
gcd as the fallback. Widening is restricted to multiples of 16: the
N % 16 == 8 class already miscomputes (#2593), and the integer instruction
tables only instantiate multiples of 16 (plus 8 and 24). The result is
never narrower than before, so no existing shape regresses.
The sparse generator carried the same line and now shares the helper.
* [Testing] Cover T.gemm_sp at the widened WGMMA N values
The width selector is shared with the sparse generator, so assert the emitted
wgmma_sp instruction and the sparse numeric result for block_N in {96, 112,
160} as well. Verified as a real regression: with the selector reverted the
sparse path emits n32 / n16 / n32 exactly like the dense one.
The test added in #2899 lowers with an explicit `tvm.target.Target("cuda")`, which
reaches `tilelang/cuda/transform/__init__.py`. In a `USE_CUDA=OFF` build only the
"ALWAYS" CUDA sources compile, so `AnnotateDeviceBoundTmaCopies` is never registered
and the test fails on ROCm:
AttributeError: module 'tilelang.cuda._ffi_api' has no attribute
'AnnotateDeviceBoundTmaCopies'
Same pattern as #2863. `requires_cuda` is the weakest sufficient gate: the test is
compile-only (`enable_device_compile=False`), so it needs no particular GPU
generation and keeps running on the NVIDIA leg.
Verified in CI: SKIPPED on the gfx942 leg, PASSED on CUDA-auto.
* [BugFix] Handle vectorized SelectNode in codegen_cuda
* address review comments
* [BugFix] Fix cuda codegen vectorized SelectNode when condition is a scalar
A large number of tests import CUDA-only internals directly (for example
`tilelang.cuda.pipeline.CUDAPassPipelineBodyPrologue`) or exercise
NVIDIA-specific features such as TMA, LDGSTG, mbarrier, warp specialization,
tcgen05 and tfloat32. `src/cuda/CMakeLists.txt` only compiles a small
"ALWAYS" subset when `USE_CUDA=OFF`, so those FFI symbols are never
registered and the tests fail with:
AttributeError: module 'tilelang.cuda._ffi_api' has no attribute
'AnnotateDeviceBoundTmaCopies'
This is not ROCm-specific -- the same tests fail on any build without the
CUDA backend, including Metal and CPU-only.
Add `@tilelang.testing.requires_cuda` to those tests, plus three
portability fixes that keep coverage on non-CUDA backends instead of
skipping it:
* `test_tilelang_transform_layout_inference.py`: CUDA emits
`__launch_bounds__(N, 1)` while HIP emits `__launch_bounds__(N)`.
Match both.
* `test_tilelang_transform_pipeline_planning.py`: PipelinePlanning
deliberately strips pipeline annotations on ROCm targets other than
gfx950 (see the guard in src/transform/pipeline_planning.cc), so assert
the documented sequential fallback there.
* `test_tilelang_kernel_gemm.py`: the two f32f32f32 cases use `T.tfloat32`,
which is NVIDIA-only, so narrow `requires_cuda_or_cdna` to
`requires_cuda`.
Also refresh three stale RDNA tests: gfx12 is now a supported generation
(`get_arch` accepts 11 and 12), so they can no longer assert that gfx1200
is rejected. Force an unsupported generation instead, which is the only
way to reach the guard now that `target_is_rdna` reports only gfx11/gfx12.
Verified on 8x MI300X (gfx942, ROCm 7.2 container, torch nightly rocm7.2):
the ROCm suite goes from 105 failures to 0 with no loss of passing tests.
* fix: resolve issue 2537
* test: pin default-search regression tests to the MMA backend
On sm_90 SelectInst reroutes M>=64 / warps%4==0 shapes to the WGMMA
search, so two tests written against the default search exercised the
wrong code path on Hopper CI. The FullCol M=192 N=104 numeric test
additionally hit a pre-existing, partition-independent WGMMA
descriptor-layer limitation (N=104 fits no canonical GMMA swizzle atom).
Pin both to the MMA path via tl.disable_wgmma; the WGMMA search keeps
its own dedicated tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* [BugFix] Reject contracting shared-buffer layouts in T.annotate_layout
T.annotate_layout could accept a contracting many-to-one forward map on a
shared buffer. makeBufferWithLayout then misread the smaller output extent as
a replication factor, rewrote the buffer to a higher rank, and eventually
failed with an unrelated BufferStore dimensionality ICHECK.
Add a pigeonhole check for static layout shapes before the replication branch.
When the layout output extent is smaller than its input extent, raise a
user-facing ValueError that identifies the buffer and reports both extents.
Keep identity, permutation, and output-growing layouts unchanged.
This check intentionally targets contracting maps; general injectivity
analysis for maps with larger output bounding boxes is outside this change.
Fixes#2632
* [Layout] Enforce injective shared-buffer annotations
Add a LayoutNode injectivity check with an exact fallback for static domains, and reuse it for FragmentNode thread and replication coordinates. Validate shared-buffer annotations after layout reshaping so sparse many-to-one maps are rejected with a useful collision diagnostic while padded injective maps remain valid.
---------
Co-authored-by: LeiWang1999 <leiwang1999@outlook.com>
* [BugFix] Pack 32-lane 8-bit CUDA vectors correctly
* [Test] Avoid SM100-only store in int8 broadcast regression
* [BugFix] Move int8x32 packing into CUDA templates
T.dp4a previously accepted arbitrary buffer dtypes and passed them to backend DP4A implementations, which could silently reinterpret the operand bytes and produce incorrect results.
Require both inputs to use int8 and the accumulator to use int32. Add regression coverage for valid Buffer and BufferLoad forms and for invalid input and accumulator dtypes.
Fixes#2600
_tir_u32_to_int_to_float decoded signed sub-word fields from uint32 storage as unsigned values because it only masked the selected field before casting. As a result, negative encodings such as signed int4 `0xF` decoded as `15.0` instead of `-1.0`.
Sign-extend the extracted field through int32 before casting to the requested float dtype, matching the existing packed signed-int decode helper.
Add CUDA regression coverage for signed int2, int4, and int8 values decoded from uint32 storage.
Fixes#2482
* [BugFix] Reject unsupported T.serial access to thread-distributed fragments with clear diagnostics
A T.serial loop runs sequentially inside each CUDA thread, but a fragment's
elements are distributed across threads, so indexing a fragment by a serial
loop variable has no valid thread-ownership mapping. Three fuzzer reports share
this root cause:
- #2393: serial reduce of a GEMM-output fragment -> unhelpful internal error
("contains inner var j").
- #2395: serial write into a fragment then T.copy out -> nvcc "identifier
undefined" (an owner-thread guard built from serial loop vars was emitted
outside the loop that binds them).
- #2396: serial write of a fragment to global -> silently wrong (threads
race-write each cell).
AddWrapperForSingleBufStore wraps bare fragment stores in a degenerate
T.parallel(1) and only rejected constant non-zero fragment indices, so a
dynamic (e.g. serial-loop) index slipped through and lowered to broken code
(#2395/#2396). Reject any fragment index that is not the constant all-zero
index in this fallback path, with a message pointing to T.Parallel /
T.reduce_sum. Rephrase the existing inner-var layout check (#2393) to explain
the cause instead of dumping "contains inner var".
Also collect every access to a buffer in a statement (not just the last) so the
diagnostic is not bypassed when one fragment is accessed at both a constant and
a variable index in the same statement; without it such a case falls through to
a cryptic downstream StructuralEqual internal error. This is a completeness
hardening only, not required by the three issues above.
Fixes#2393Fixes#2395Fixes#2396
* Generalize fragment diagnostic to non-parallel loops
inner_vars_ is populated for every non-parallel For (serial, unroll,
vectorized), so the message must not hard-code T.serial. Reword to
"non-parallel loop variable" per review feedback.
* Narrow fix scope to #2393 only
Remove fragment index validation from AddWrapperForSingleBufStore and
delete tests for #2395/#2396. These issues cannot be caught at this pass
stage because T.grid and T.serial are indistinguishable (both appear as
ForKind.SERIAL before LowerTileOp).
Keep only the ParallelOpNode diagnostic improvement which successfully
catches #2393.
* [BugFix] Fix T.Persistent dropping tiles when last dim is not a multiple of group_size (#2433)
* Update src/ir.cc
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Fix eager JIT packed FP4 shape binding
* Add logical shape and stride calculations for buffers in eager builder
This update introduces helper functions to compute logical shape and stride values for buffers, enhancing the handling of packed storage factors. The changes streamline the process of determining dimensions and strides in the context of tensor operations, improving overall efficiency and clarity in the eager execution flow.
---------
Co-authored-by: LeiWang1999 <leiwang1999@outlook.com>
* Place auto WS producers in first warp group
* [Transform] Update layout inference to bind thread range for fully replicated fragments; add tests for TMA fragment mask copy and producer-consumer warp specialization
* [Refactor] Update thread variable calculation to use zero-based indexing across GEMM operations
* [Transform] Enhance simplification process with ContextSingletonFloorDivFolder and add tests for floor division simplification
* [Refactor] Simplify tensor conversion in dequantization functions and improve performance in GEMM examples
Rishabh Sinha