LowerSharedBarrier lowers T.alloc_barrier() to hardware mbarrier code (the
cutlass Barrier type, tl::tl_shuffle_elect, tl::fence_barrier_init), all of
which only exist on sm_90+. On pre-Hopper targets (e.g. sm_86) the frontend
accepted the call silently and nvcc later failed with cryptic
"identifier 'Barrier' is undefined" / "namespace 'tl' has no member ..."
errors.
Add a static compute-capability guard in CUDAPassPipelineBody, before
LowerSharedBarrier: if the target is < sm_90 and the module uses a shared
barrier, raise a clear ValueError naming the offending primitives and pointing
to __syncthreads()/named barriers as the pre-Hopper alternative.
Fixes the sm_86 alloc_barrier compile crash.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix eager JIT packed FP4 shape binding
* Add logical shape and stride calculations for buffers in eager builder
This update introduces helper functions to compute logical shape and stride values for buffers, enhancing the handling of packed storage factors. The changes streamline the process of determining dimensions and strides in the context of tensor operations, improving overall efficiency and clarity in the eager execution flow.
---------
Co-authored-by: LeiWang1999 <leiwang1999@outlook.com>
* Add TMA copy for packed global FP4 to unpacked SMEM.
Wire IsValidTMACopyDtypePair and ALIGN16B tensor map descriptors for
float4_e2m1fn global to float4_e2m1_unpacked shared transfers, and
extend language tests for TMA and SIMT copy restrictions.
* Extend tcgen05 MMA metadata for FP4 f8f6f4 and dtype helpers.
* Refactor tcgen05_gemm_blockscaled API and lowering.
* Use separate A/B dtypes in GEMM tileop emitters across backends.
* Add maint/gemm correctness evaluation scripts and update gemm_sm100 examples.
* Fix profiler assert_allclose to compare outputs only.
* lint
* canonicalize data types
* add dpgm fp8-fp4 kernel
* resolve comments
* Refine FP4 unpacked SMEM dtype handling
Treat float4_e2m1_unpacked as a SMEM/TMA-only storage tag instead of a general vector/register dtype. Remove vectorized Python aliases and keep codegen uses constrained to shared-memory storage/reference paths that lower to uint8_t.
Allow packed FP4 global tensors to TMA-load into unpacked shared memory with the ALIGN16B TensorMap format, while keeping unpacked storage out of TMA store paths. Use payload-bit accounting for global strides and mbarrier transactions.
Add focused dtype and TMA descriptor tests, and validate the real tcgen05 fp4-unpacked GEMM paths.
* Restrict FP4 unpack TMA to loads
* Hide vectorized unpacked FP4 language aliases
* Update TVM for unpacked FP4 symbol exports
* Rebind TVM submodule to merged unpacked FP4 export fix
* Enable cache for persistent tcgen5 GEMM example
* Place auto WS producers in first warp group
* [Transform] Update layout inference to bind thread range for fully replicated fragments; add tests for TMA fragment mask copy and producer-consumer warp specialization
* [Refactor] Update thread variable calculation to use zero-based indexing across GEMM operations
* [Transform] Enhance simplification process with ContextSingletonFloorDivFolder and add tests for floor division simplification
* [Refactor] Simplify tensor conversion in dequantization functions and improve performance in GEMM examples
* [AMD][RDNA4] Fix RDNA4 (gfx1201 / Wave32) CI failures
- reduce.h: Replace hardcoded wave64 assumptions with runtime
__builtin_amdgcn_wavefrontsize() in SharedReduceWarp and AllReduce
- scan.h: Add SEG=32 support in InclusiveScan/CumSum/CumMax for RDNA wave32
- finalize_reducer.cc: Use TargetGetWarpSize() instead of hardcoded 64
- carver/arch/__init__.py + rdna.py: Add gfx12 (RDNA4) support alongside gfx11
- wmma_macro_generator.py: Fix k_pack indexing in ldmatrix_a/b for k_pack > 1
These changes correspond to PR #2210 (Fix RDNA4 gfx1201/Wave32 CI Failures),
rebased cleanly onto upstream main (a8d93798).
* [AMD][HIP] Fix HIPMath dispatch for vector dtype in T.Parallel
When BLOCK_M > threads, TileLang generates vector types (e.g. float32x4)
for T.Parallel loops. HIPMath::operator() only handled scalar types, so
vector dtypes fell through to return causing Unresolved call errors
in codegen for math intrinsics like tirx.exp2.
Fix: add a vector-type early-return that recurses on element_of(), so
float32xN correctly resolves to exp2f the same as scalar float32.
This complements the tirx.* registration fixes in #2282 and enables
Online Softmax on gfx1151 with BM > threads (e.g. BM=1024, TH=256).
* [Review] Fix 3 issues found in code review
1. tilelang/carver/arch/__init__.py: error message was not updated
when gfx12 support was added — still said gfx11 only. Now says
gfx11/gfx12 targets only, consistent with rdna.py.
2. src/tl_templates/hip/reduce.h: the static_assert(Threads % kWarpSize == 0)
was removed when kWarpSize became a runtime value, leaving no safety
check. Added equivalent runtime assert().
3. src/tl_templates/hip/scan.h: run_auto() else branch silently fell
through to SEG=64 for any non-32 wavefront size. Added assert to
catch unsupported wavefront sizes early.
* update codes for format checking
* [Review] Address review feedback and rebase on upstream main
- intrin_rule_hip.cc: remove redundant HIPMath scalar recursion; upstream
PR #2286 already handles vector dtypes via DispatchPureExternScalarized
at the IR level (correct for HIP packed carrier types)
- wmma_macro_generator.py: rename l/r to row_off/col_off to fix Ruff E741
ambiguous variable name warnings
- scan.h: guard run_auto() branches with threads>=32 / threads>=64 so a
threads==32 kernel on Wave64 doesn't compute TILE=0 and divide by zero
- reduce.h: replace assert (stripped in Release) with unconditional early
return when Threads % kWarpSize != 0, preventing silent num_warps==0
* update codes for format checking
* fix
---------
Co-authored-by: LeiWang1999 <leiwang1999@outlook.com>
* Scalarize vector math intrinsics for HIP codegen
* Clarify HIP fp32 vector-math comment in scalarizer
* fix comment
* Scalarize HIP vector math via PrintCallExtern, not generic Shuffle
The generic Shuffle re-pack emits one ctor arg per logical lane, which is
wrong for HIP packed carrier types (float32x8->ulonglong4, float16x4->uint2).
Emit a vector call_pure_extern with the element-derived extern name instead
and let CodeGenTileLangHIP::PrintCallExtern store each lane (PrintVecElemStore),
which is carrier-aware.
* Add HIP vector math scalarizer test
* JIT-decorate scalarizer test helper
[BugFix][Transform] Freshen buffer defs after loop unroll
UnrollLoop clones each loop body with Substitute, which preserves Buffer object identity for definitions inside the body. This can define the same DeclBuffer handle in multiple unrolled copies after flattening and violate TIR SSA. Freshen buffer definitions per cloned body while preserving DeclBuffer backing data aliases.
Co-authored-by: LeiWang1999 <leiwang1999@outlook.com>
* [Backend] Refactor Pipeline to support different backends
Introduce Pipeline abstraction in backend/pipeline.py with per-backend
registration. Each backend (cuda, hip, c, llvm) now registers its own
compilation pass pipeline. engine/lower.py resolves the pipeline from
the target instead of hardcoding phase imports.
* enhance the structure
* fuse two stage
* fix after rebase
* update template pass pipeline
* pipeline
* fix test
* fix test
* fix cuda pipeline prologue
* Add float4_e2m1_unpacked dtype
* Fix TIR cast for float4_e2m1_unpacked dtype.
Route custom[float4_e2m1_unpacked] through Float4E2M1Unpacked FFI
so T.float4_e2m1_unpacked(0.0) works in prim_func builders.
* Limit float4_e2m1_unpacked vector lanes and fix TMA transaction bytes.
Cap unpacked FP4 vector dtypes at x32 and count TMA/mbarrier
transactions using 4-bit payload size (0.5 byte per element).
* lint
* [Refactor] Extract shared-access analysis into a reusable header
- Move AccessEntry/AccessType definitions from thread_storage_sync.cc to
src/transform/common/shared_access_analysis.h so other passes (notably
MergeSharedMemoryAllocations) can reuse the same precise access
representation.
- Add tl.debug_thread_storage_sync_summary pass-config option for optional
ThreadSync planner summary logging.
- Add regression tests covering explicit shared sync deduplication and
shared.dyn alias-aware dependency planning.
* [Feature] Rewrite MergeSharedMemoryAllocations with precise access analysis
- Build per-access live ranges using shared_access_analysis utilities so
branch-alternative dataflow and barrier-driven epoch boundaries are
modeled directly instead of through whole-buffer approximations.
- Plan arena layout starting from arena_size_const so allocations whose
starting offsets are non-zero no longer collide.
- Preserve single-slot semantics for mutually exclusive branch writes
followed by a post-dominating read (fixes the K_norm_pairs_shared style
miscompile where stage-1 alternatives were split across epochs).
- Cover BufferRegion block reads/writes, tvm_access_ptr, tl.access_ptr,
cp_async branches, loop-carried buffers and partial-subregion liveness
cases.
- Keep aggressive-merge disabled under warp specialization and retain
whole-buffer liveness as the conservative default.
- Add 18 transform regressions in
test_tilelang_transform_merge_shared_memory_allocations.py covering
basic merge, sync, branch, cp_async, BufferRegion and opaque-pointer
paths.
* [Cleanup] Remove dead FreeList/FreeBlock/ActiveInterval helpers
These helpers were superseded by the segment-aware LinearScanPack
implementation that performs interval-overlap walking directly. They
have no remaining callers after the rewrite; remove them to reduce
maintenance burden.
* [Test] Cover tail-allocation arena growth scenario
Add a regression case mirroring upstream b4c913be (#2106): when the only
free slot reusable by an incoming buffer sits at the arena tail and is
smaller than the request, the arena must grow by the deficit only, not
push the new buffer past the freed slot.
The legacy free-list-based packer would emit a 384 B arena for this
pattern (X=128, Y=64-then-freed, Z=192), bumping Z to offset 192. The
interval-based LinearScanPack reuses Y's offset and grows the tail to
fit Z, producing a 320 B arena -- a 16.7%% reduction that matches the
peak-live lower bound.
* [Feature] Add EpochGraph and use per-epoch liveness to relax aliasing
Introduce src/transform/common/epoch_graph.h, a control-flow-aware
builder that partitions a flattened StmtEntry sequence into hard-sync
and cp_async-wait epochs across nested For/IfThenElse scopes, exposing
loop-carry, branch-join and parent-edge structure required for sound
liveness analysis.
Wire MergeSharedMemoryAllocations to compute per-buffer live epochs as
the intersection of:
* forward def-reach: epochs reachable from any write of the buffer
along epoch-graph successors;
* backward use-reach: epochs that can reach any read of the buffer
along predecessors.
A buffer is live in epoch e iff e is in both sets, modeling 'data
written upstream is consumed downstream' rather than the legacy
[start_stmt, end_stmt) bounding box. The result feeds two relaxers:
* LinearScanPack: two intervals whose live-epoch sets are disjoint
may share an offset even if their stmt ranges overlap.
* PlanMemory: the symmetric overlap probe consults the same set
before forcing a non-aliased layout.
Hard syncs (kSync) and cp_async waits are treated as epoch boundaries;
ptx_cp_async_wait is conservatively classified as a hard barrier.
IfThenElse joins take the union of branch live sets so a buffer
written in only one branch remains live at the join.
The whole-buffer EpochGraph + liveness pipeline always runs; verbose
[MSMA-EPOCH-GRAPH] / [MSMA-EPOCH-LIVE] / [MSMA-EPOCH-ACCESS] /
[MSMA-EPOCHSEG] / [MSMA-STAGE0-RAW] dumps remain gated behind the
existing tl.debug_merge_shared_memory_allocations pass-config flag.
Also remove SplitLocalSegmentsAtEpochSyncs, a Phase-1 helper that was
defined but never wired into the planner; the new liveness pipeline
subsumes its role.
Validated on a real lowbit MHA decode kernel
(seq_lens=[(3,96),(2,64)], heads=4, dim=64, page_size=32, bits=4):
arena drops from 35,232 B (upstream baseline) to 31,136 B, an 11.6%%
reduction driven by K_shared and V_shared sharing one offset thanks
to disjoint live-epoch sets.
Add testing/python/transform/test_tilelang_transform_epoch_graph.py
covering the five S1.4 epoch-graph topology cases (sequential
__syncthreads, ptx_wait_group, For body/back/exit edges, IfThenElse
branch_join, nested For). Existing
test_tilelang_transform_merge_shared_memory_allocations.py and
test_tilelang_transform_thread_sync.py suites continue to pass.
* [Feature] Gate per-epoch alias relaxer on annotated layout equality
Two shared-memory buffers carrying distinct layouts under
`block.annotations[attr::kLayoutMap]` (e.g. via `T.annotate_layout(...)`)
must not share storage even when per-epoch dataflow proves their
liveness sets disjoint: downstream layout-driven access lowering can
emit indexing arithmetic that is only sound under each buffer's
original layout, so naively aliasing them risks silent miscompiles.
Add a `LayoutSignatureCollector` that harvests the per-Var `Layout`
entries from `Block` annotations and a `LayoutAliasIncompatible`
helper that returns true iff both candidates have non-trivial layouts
and the layouts are not structurally equal. The two relaxer call
sites in `SharedMemoryRewriter` (`LinearScanPack` and the post-pack
overlap audit) consult the helper before dropping a conflict, so the
legacy decision is preserved exactly when the gate fires.
Buffers without a layout entry remain layout-agnostic and continue to
participate in arena packing, keeping the existing reuse for the
common un-annotated case. Verified that the lowbit reference kernel
arena footprints stay at 31136/61856/61808/61792 B and the merge /
thread-sync / epoch-graph trio passes 49/49.
* [Test] Add transform regression that fails on upstream MergeSharedMemoryAllocations
Add test_general_cond_branch_alternatives_share_offset_with_postdom_buffer
to lock the per-epoch / branch-alternative analysis behavior on a
mutually-exclusive if/else with a runtime cond plus a post-dominator
buffer after a sync. Upstream's linear-scan plan keeps the two branch
buffers at distinct offsets (128-byte arena); the per-epoch relaxer
collapses them onto the same offset and reuses it for the
post-dominator buffer (64-byte arena, 50% smaller).
* [Example] Demo MergeSharedMemoryAllocations branch-alternative arena reuse (64 vs 128 B)
Add runnable demo under examples/merge_shared_memory_allocations/
that constructs a three-buffer PrimFunc with mutually-exclusive if/else
branches, lowers through the merge pass, and reads the merged arena
size from the dyn_shared_memory_buf attr. The demo also compiles the
kernel end-to-end and validates output correctness with torch.allclose.
On this branch: arena = 64 B (saved 128 B / 66.7% vs naive 192).
On upstream main (1fbd994c): arena = 128 B (-50% worse).
* [Test] Add end-to-end torch correctness regression for merge pass
Add two tests (test_merged_branch_kernel_generates_correct_results,
test_merged_sync_phased_buffers_generate_correct_results) that compile
the merged kernel and verify bit-exact output via torch.allclose,
ensuring the arena packing does not alter numerical results.
* [Test] Add torch correctness and dynamic-buffer fallback regression
- test_merged_branch_kernel_generates_correct_results: compiles a
merged branch-alternative kernel with torch.allclose validation.
- test_merged_sync_phased_buffers_generate_correct_results: sync-phased
two-buffer kernel end-to-end check.
- test_dynamic_size_buffer_fallback_does_not_incorrectly_alias:
verifies that a symbolic-extent shared.dyn buffer correctly aliases
with a sync-separated constant-size buffer without unsafe overlap.
* [Fix] Disable per-epoch relaxer for loop-live buffers to avoid pipelined iteration overlap
* [Fix] Fallback to structural_segments_by_var for buffers missing planner segments to avoid buffer_byte_offsets crash and shared memory bloat
* Merge branch 'main' into merge_pass and fix merge comflicts
* [Fix] Resolve post-merge artifacts in CMakeLists, hipcc, and deepseek example
- Remove duplicate TILELANG_OUTPUT_TARGETS in CMakeLists.txt
- Fix duplicate _resolve_artifact_paths function in hipcc.py
- Add missing G=kv_group and fix assert in sparse_mla_fwd_pipelined.py
- Add ruff noqa pragma to suppress lint warnings
* [Style] Apply clang-format to modified source files
* [Pipeline] Refactor software pipeline transforms
* enhance
* Refine pipeline handling for guarded binds
* Refactor IfStmtBindingRewriter and related classes for improved access analysis
- Removed unused Target parameter from IfStmtBindingRewriter.
- Introduced IfStmtAccessCollector to handle buffer access collection.
- Updated access analysis methods to utilize the new collector.
- Cleaned up includes and removed obsolete functions in access_analysis.
- Enhanced pipeline utility headers with necessary includes for consistency.
* [Fix] Suppress spurious __syncthreads() between T.ws blocks (#2192) and add T.named_barrier_arrive (#2188)
Issue #2192 - BugFix: Unexpected CTA-wide __syncthreads() between T.ws producer/consumer
- Root cause: FindConflict() in thread_storage_sync.cc did not check for mutually
exclusive thread constraints. When T.ws(0) (tx<128) reads and T.ws(1) (tx>=128)
writes the same shared buffer, no thread satisfies both constraints, so no
__syncthreads() is needed — mbarriers already handle the synchronization.
- Fix: add mutual-exclusivity check (prove NOT(P AND C)) early in FindConflict().
If the access constraints are mutually exclusive (disjoint thread sets), return
false (no conflict) immediately, before any index-range analysis.
- Measured improvement: barrierpipe_stage2 gemm latency 44ms → 25ms on H200.
Issue #2188 - Feature: expose native T.named_barrier_arrive(barrier_id, thread_count)
- Adds tl.named_barrier_arrive intrinsic op (bar.arrive PTX, one-sided arrive).
- Registers op in builtin.cc, declares in builtin.h.
- Adds tl::__named_barrier_arrive<id, cnt>() template in common.h.
- Adds codegen case in codegen_cuda.cc.
- Exposes T.named_barrier_arrive() in tilelang/language/builtin.py.
- Adds regression test testing/python/issue/test_tilelang_issue_2188_2192.py.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* lint: clang-format + ruff fixes after #2188/#2192 changes
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* test: simplify issue 2188/2192 test, keep only #2192, add requires_cuda
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [Fix] Make named_barrier_arrive accept variable barrier_id/thread_count
* Update documentation for named barriers and remove obsolete test file
- Clarified documentation for `named_barrier_arrive` in `builtin.h` and `builtin.py` to specify the barrier index range.
- Removed obsolete regression test for issues #2188 and #2192 as it is no longer needed.
* lint fix
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: LeiWang1999 <leiwang1999@outlook.com>
* Optimize disk cache source loading
* Add tests for kernel cache frontend hit and improve memory cache handling
- Introduced a new test to validate the round-trip functionality of real primitive functions in the kernel cache.
- Enhanced the memory cache logic to prevent overwriting existing entries.
- Improved error handling when tagging kernel cache entries with logging for better debugging.
* [Cache] Refine lazy source restore
Guard memory-cache writes after compile, make cache tagging failures diagnosable, and document the lazy frontend-cache gate.
Wrap cached source text/path in CachedTextSource and add a real PrimFunc frontend-cache round-trip regression test.
---------
Co-authored-by: LeiWang1999 <leiwang1999@outlook.com>
* feat(tma_copy): add thread_extent parameter for per-warp TMA copy
Add a `thread_extent` parameter to `T.tma_copy()` that controls the
elect size for TMA leader thread selection. Defaults to full block
size, can be set to warp_size (32) for per-warp TMA operations.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor(tma_copy): rename leader election scope parameter
* remove: delete test_per_warp_tma.py and its associated tests; add per_warp_tma_copy_kernel and block_tma_copy_kernel tests in test_tilelang_language_tma_copy.py
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: LeiWang1999 <leiwang1999@outlook.com>
* [Bugfix] Fix issue 2123 access_ptr lowering
Add the issue 2123 regression test and keep the access_ptr base BufferLoad intact through safe-memory legalization and lowering.
Closes#2123
* Deduplicate access_ptr base load handling
Move shared tl.access_ptr BufferLoad base visitation into a common transform helper so LowerAccessPtr and safe-memory legalization stay aligned.
Extend the issue 2123 regression with a direct LowerAccessPtr pass check while keeping the LowerAndLegalize pipeline coverage.
* Format access_ptr helper header
* Narrow access_ptr helper scope
* Share access_ptr base helper
* Address access_ptr helper review comments
* [Transform] Preserve access_ptr buffer loads in safe memory legalization
* Refactor AccessPtrInfo return structure in legalize_safe_memory_access.cc
Updated the return structure of AccessPtrInfo to improve readability and maintainability. The logic for handling the tvm_access_ptr operation has been streamlined, ensuring clearer checks and consistent formatting of returned values.
* fix
---------
Co-authored-by: LeiWang1999 <leiwang1999@outlook.com>
* add amd gfx950 mxfp4 support
* update the codes for format checking
* [AMD][CDNA4] Address review comments for MXFP4 gfx950 support
Four issues from CodeRabbit review fixed:
1. hip_fp4.h: Fix FP4 E2M1 denormal decoder round-trip bug.
The decoder returned 0.25f for mant==1 (exp==0) but the encoder
maps nibble 1 to 0.5f, breaking float→fp4→float round-trips.
Fix: return 0.5f to match the encoder and LUT tables.
2. codegen_hip.cc: Fix UB address-of-prvalue in FP4→float16 cast.
The generated code took &(__tl_cvt_fp4x2_to_half2(...)), which
takes the address of a temporary (C++ UB). Fix: materialize the
uint1 return value into a local variable first, matching the
pattern already used in the float32/bfloat16 cast paths.
3. codegen_hip.cc: Extend fp4_pair_cast to cover FP4x16 and FP4x32.
Previously only 2/4/8-lane FP4 casts were handled; x16/x32 fell
through silently to generic vector code that cannot index fp4_e2_*
aggregates. The pairwise uint8_t byte logic is correct for any
even lane count, so extend to include 16 and 32 lanes.
4. mxfp.py: Narrow bare except to expected import-time errors.
The catch-all except Exception silently swallowed all errors from
target_is_gfx950(), causing HIP targets to fall back to CUDA/PTX.
Fix: catch only ImportError/ModuleNotFoundError/AttributeError.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* update for format checking
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: LeiWang1999 <leiwang1999@outlook.com>
* [Metal] Add Metal GEMM support with simdgroup_matrix MMA
Add T.gemm support for Apple Metal using simdgroup_matrix 8x8 operations
(simdgroup_load/store/multiply_accumulate). Works on all Apple Silicon
(M1-M5) without requiring a TVM fork.
Key changes:
- codegen_metal.cc/h: Fork TVM Metal codegen to tilelang with
simdgroup intrinsic emission and 128-bit vectorized copy
- gemm_metal.py: GemmMetal tile operator for sharedxshared GEMM
- metal_macro_generator.py: MPSIntrinEmitter for simdgroup MMA macros
- metal_fragment_to_simdgroup.py: Pass rewrites local.fragment GEMM
accumulators to metal.simdgroup scope before layout inference
- LowerSIMDGroupCopy in copy.cc for fragment->device simdgroup_store
24 Metal tests (codegen cross-platform + correctness on device).
* Move Metal buffer helpers to Metal backend
* fix(metal): remove layout_inference bypass, fix build & clarify docs
- Remove duplicate BuildTileLangMetal in rt_mod_metal.cc (fixes linker error)
- Remove TargetIsMetal bypass in layout_inference.cc. The bypass is
unnecessary because MetalFragmentToSimdgroup already converts all
GEMM accumulator fragment buffers to metal.simdgroup scope before
LayoutInference runs, so IsFragmentBuffer never matches them.
- Clarify docstrings for MetalFragmentToSimdgroup pass explaining why
it runs before LayoutInference (simdgroup matrices are opaque and
have no explicit thread-level layout).
- Restore is_metal_simdgroup check in GemmMetal.lower() so the GEMM
lowering correctly handles buffers already converted to simdgroup
scope by the pass.
* fix: tir->tirx migration build fixes for metal backend
- Replace tir::Buffer with Buffer, tir:: with tirx:: in metal backend
- Update codegen_metal to use AllocBufferNode instead of AllocateNode
- Create runtime/metal/metal_module.h with MetalModuleCreate factory
- Fix ICHECK -> TVM_FFI_ICHECK macros
- Fix 'from tvm import tir' -> 'from tvm import tirx as tir' in Python
- Restore tilelang/backend/__init__.py for metal gemm registration
- Add metal backend import to tilelang/__init__.py
- Fix metal_fragment_to_simdgroup.py: Block->SBlock, Allocate->AllocBuffer
* fix: improve metal_fragment_to_simdgroup buffer remapping
- Add AllocBuffer node handling for standalone allocations
- Remap buffer references in SBlock reads/writes/match_buffers
- Add _remap_buffer_region and _remap_match_buffer helpers
* lint
* fix: add full-body var subst in MetalFragmentToSimdgroup to reach gemm call_intrin args
The per-SBlock substitute() in _rewrite_scope missed Var references
buried inside BufferRegion objects nested in call_intrin arguments (e.g.,
the GEMM op's C buffer). This leaked the old local.fragment-typed C_local
var into the subsequent gemm lowering, producing a duplicate variable that
MakePackedAPI flagged as undefined.
Fix: apply tir.stmt_functor.substitute() to the entire body before the
ir_transform pass, so all Var references — including those inside opaque
Call args — are remapped to the metal.simdgroup-typed replacements.
Also remove the redundant per-SBlock substitute() and the duplicate
_remap_buffer function.
* fix: add missing <cmath> include for std::isinf/isnan in codegen_metal
* [Metal] Fix review findings: dedup warp partition, kernel_only, barrier comment, immutable gemm ops
- Extract shared ComputeSquareWarpPartition to src/backend/metal/op/utils.h,
used by both gemm.cc (Square policy) and copy.cc (simdgroup store lowering).
- Implement kernel_only parameter in MetalKernelAdapter.get_kernel_source.
- Add comment documenting that Metal's simdgroup_barrier synchronizes the
full threadgroup (no per-simdgroup barrier exists).
- Replace module-level mutable global in metal_fragment_to_simdgroup.py
with @lru_cache and remove CUDA-only gemm ops from the set.
* lint
* Refactor MetalFragmentToSimdgroup import path in LowerAndLegalize function
* Remove metal_fragment_to_simdgroup.py file and update import in metal/__init__.py
---------
Co-authored-by: LeiWang1999 <leiwang1999@outlook.com>
[CuTeDSL] Integrate host codegen call sites
Drive CuTeDSL launcher generation from lowered host tvm_call_packed call sites so multi-kernel wrappers preserve host IR order and repeated launches can be represented.
Expose separate device, host, and generated module sources through the CuTeDSL adapter and persist them separately in the kernel/autotune caches.
Normalize bare cutedsl targets through TileLang CUDA arch detection so Hopper runners use sm_90a for CuTeDSL compilation instead of the generic sm_90 Target("cuda") result.
Add CuTeDSL host-codegen integration tests covering multi-call launcher ordering, source accessors, cache restore behavior, and bare target architecture normalization.
Co-authored-by: Jayce <jayce.su@enflame-tech.com>
Co-authored-by: LeiWang1999 <leiwang1999@outlook.com>
* [Example] Migrate eligible examples to eager style
* fix wrong shape annotation in gemm_split_k examples
* fix incorrect dtype in gemm_streamk example
* fix
* revert some changes
* Refactor attention functions to use new parameter structures and improve readability. Updated `example_amd_flash_attn_bwd.py` and `example_amd_flash_attn_fwd.py` to define input parameters more clearly. Enhanced shared memory usage in `flashattn_fwd` and `flashattn_sink` functions across various examples, ensuring compatibility with new configurations. Adjusted kernel definitions and added necessary checks for GPU capabilities.
* lint fix
* FIX
---------
Co-authored-by: LeiWang1999 <leiwang1999@outlook.com>
* Preserve shared memory aliases after dynamic merge
* Enhance SharedMemoryRewriter to support alias preservation and buffer remapping during shared memory allocation merging
* lint fix
* [Feature] Introduce T.tfloat32
* fix: change tfloat32 type code to 130
* minor fix
* update tvm to latest
* minor fix
* minor fix
* update tvm
* add conversion from __nv_bfloat16 to tfloat32_t, update mhc example
* minor fix
* enable wgmma support for tfloat32, update gdn example
* update tf32 gemm test
* Add TL_DISABLE_MERGE_SHARED_MEMORY_ALLOCATIONS pass config
* Rename to TL_DISABLE_SHARED_MEMORY_REUSE and implement disable_reuse in C++
Instead of skipping MergeSharedMemoryAllocations entirely, add a disable_reuse
flag that still merges allocations into one but lays out each buffer sequentially
without lifetime-based sharing.
* Fix build: use AllocBufferNode instead of tir::AllocateNode in PlanSequentialLayout
* Add test for TL_DISABLE_SHARED_MEMORY_REUSE pass config
Lei Wang