* [Transform] Never classify side-effecting binds as replayable
IsReplayableScalarBind (introduced in #2245) decides whether a scalar
bind may be dissolved and its value re-evaluated at every use site
(IfStmtBinding, default-on) or replayed across pipeline stages
(PipelinePlanning, InjectSoftwarePipeline). The predicate checked the
dtype and read/write disjointness, but never the value's effect kind,
so a bind like
pos = T.atomic_add(counter[0], 1, return_prev=True)
used at two sites (bound check + store index) was inlined into both use
sites, executing the atomic once per use site instead of exactly once
and silently corrupting compaction/histogram/top-k kernels since
v0.1.11. The atomic intrinsic is correctly registered kOpaque; its
pointer argument enters the expression through tvm_access_ptr, which
the access collector records as a read, so the write-disjointness check
passed and the bind was declared replayable.
Reject any bind whose value has SideEffect > kReadState, mirroring the
effect guard StmtSimplifier already applies to let inlining. Read-only
values remain replayable; read consistency is enforced by the existing
write-set disjointness check.
The shipped examples/deepseek_v32/topk_selector.py uses exactly this
pattern and is miscompiled on v0.1.11/v0.1.12; its test only printed
recall and could not fail, so it also gains a recall assertion.
Regression tests:
- transform: IfStmtBinding must keep a bind whose value is an atomic
RMW returning the previous value.
- language (CUDA): compaction kernel using atomic_add(return_prev=True)
must contain exactly one AtomicAddRet and produce a permutation with
counter == N.
* [Transform] Validate scalar bind replay safety
Track access-pointer write masks during bind and pipeline analysis, and reject unsafe precomputed replay annotations during pipeline injection.
Add regressions for atomic target reads and side-effecting replay masks.
* Remove recall calculation from top-k selector
Removed recall calculation and assertion for top-k selection.
---------
Co-authored-by: LeiWang1999 <leiwang1999@outlook.com>
Co-authored-by: Lei Wang <34334180+LeiWang1999@users.noreply.github.com>
* [BugFix] Track GEMM accumulator writes in warp-specialization liveness collector
* [BugFix] Track scale-factor regions and region extents in gemm liveness
* [BugFix] Return previous value for atomic min/max/add (scalar + vectorized)
`atomic_min`/`atomic_max` (scalar) and `atomic_addx2`/`atomic_addx4`
(vectorized) typed their call with a non-void return under
`return_prev=True` but always selected the void `*_elem_op`, so codegen
emitted an empty binding (`prev = ;`) and nvcc failed.
Select the `*_ret_elem_op` variant under `return_prev`, as `atomic_add`
already does:
- min/max: the ret ops, codegen handlers, and device templates
(AtomicMaxRet/AtomicMinRet) already existed.
- addx2/addx4 (float32): register the ret ops (builtin.cc/.h) and add
codegen handlers emitting AtomicAddx2Ret/AtomicAddx4Ret (already in
the device headers) as expressions; the return is typed as the packed
2-/4-lane vector so the store matches.
- addx2 (fp16/bf16): the ret store LHS is `uint1`; codegen wraps the
half2/bf162 result with `tl::to_uint1`.
- addx4 (fp16/bf16): add AtomicAddx4Ret(half_t*)/(bfloat16_t*) device
overloads that run two AtomicAddx2Ret halves and return them packed as
uint2 (a fp16x4 is stored as uint2).
The 16-bit x4 path is per-pair atomic (two half2 atomics), not
whole-quad single-atomic -- hardware has no vector atomic wider than
`.f16x2` for 16-bit types. This matches the existing fp32-x4 scalar
fallback, and the docstring only promises "quad-width atomic addition".
Adds tests across scalar/x2/x4 and float32/fp16/bf16.
Fixes#2667
* minor fix for missing conditiions
---------
Co-authored-by: LeiWang1999 <leiwang1999@outlook.com>
* Fix physical wait counts for async pipeline groups
* Add commit group tracking for async pipeline in inject_pipeline.cc
- Introduced buffer_to_commit_group mapping and commit_group_count in AsyncStateGlobal.
- Enhanced wait handling for tail consumers in async pipeline to ensure correct physical wait counts.
- Updated tests to validate fine-grained physical waits and tail drain waits for async pipeline.
* [Refactor] Remove unsupported tl_gemm and tl_gemm_sp operations
* Removed references to tl_gemm and tl_gemm_sp from various files, including code generation and built-in definitions, as these operations are currently unsupported.
* Updated related comments and diagnostics to reflect the removal of these operations, ensuring clarity in the codebase.
* This change simplifies the handling of unsupported operations and improves overall code maintainability.
* Reduce CUDA template include overhead
* Fix CUDA reduce fast min max helpers
* lint fix
* Fix CUDA swizzle ceil div helper
* Restore CUDA wait_wgmma intrinsic helper
* Fix CUDA cython TMA host adapter include
* Remove unnecessary check for Cutlass fast math header in CUDA code generation
* Update LibraryGenerator to correctly handle CUDA gencode flags for shared-library compilation, ensuring compatibility with target architecture and avoiding Hopper-only instruction issues.
* Refactor smem_ptr_to_uint function to use __cvta_generic_to_shared for improved clarity and performance.
* Refactor example scripts and CUDA math intrinsics for improved performance and clarity
- Updated `example_dequant_gemm_fp4_hopper.py` to directly call `run_regression_perf()` and print latency, removing unused argument parsing code.
- Modified `example_topk.py` to specify the backend as "cupti" in the benchmarking function.
- Enhanced CUDA code generation by introducing `RequiresTileLangMathHeader` to manage math header dependencies for specific functions.
- Updated barrier operations in `barrier.h` to use the correct syntax for shared memory barriers.
- Improved testing for bfloat16 math intrinsics by replacing pytest skips with decorators for CUDA availability checks.
* Refactor CUDA code for improved readability and consistency
- Reformatted the `RequiresTileLangMathHeader` function for better readability.
- Updated assembly syntax in `barrier.h` for clarity and consistency.
- Cleaned up whitespace in the bfloat16 math intrinsics test file to enhance code cleanliness.
* Remove unnecessary whitespace in CUDA code for improved readability
* Add TileLang intrinsic for prefetching TMA descriptor
- Introduced `prefetch_tma_descriptor` intrinsic in C++ and updated relevant files to use `call_intrin` instead of `call_extern`.
- Updated CUDA code generation to handle the new intrinsic correctly.
- Modified tests to ensure the new intrinsic is called appropriately and that no `call_extern` is used for internal TileLang operations.
- Enhanced documentation in SKILL.md regarding the addition of new TileLang intrinsics.
* Add bfloat16 hexp overload and update common.h for compatibility
- Introduced `hexp` overload for `bfloat16_t` in `common.h` to bridge TileLang and CUDA's handling of exponential functions.
- Updated tests to verify the presence of the new `hexp` definition in `common.h`.
- Enhanced CUDA code generation to support the new intrinsic for better performance and compatibility.
* Update common.h for CUDA compatibility with aligned double4 types
- Adjusted preprocessor condition in `common.h` to maintain compatibility with CUDA versions prior to 13 by ensuring proper definition of aligned double4 types.
- This change enhances the compatibility of the code with older NVRTC built-in vector headers.
* Refactor kernel launch materialization
* [Refactor] Improve git commit ID retrieval and version labeling logic
* Enhance the get_git_commit_id function to check for the existence of the .git directory before attempting to retrieve the commit hash, ensuring more robust behavior.
* Update the dynamic_metadata function to conditionally set the NO_VERSION_LABEL environment variable based on the presence of the .git directory.
* Modify the GitHub Actions workflow to unset the NO_VERSION_LABEL environment variable during the installation and version check of the package, allowing for accurate version labeling in SDist builds.
* [Refactor] Update WebGPU backend registration
* Import the WebGPU package to automatically register its pass pipeline.
* Remove the explicit registration of the WebGPU pass pipeline, streamlining the backend setup process.
* Place auto WS producers in first warp group
* [Transform] Update layout inference to bind thread range for fully replicated fragments; add tests for TMA fragment mask copy and producer-consumer warp specialization
* [Refactor] Update thread variable calculation to use zero-based indexing across GEMM operations
* [Transform] Enhance simplification process with ContextSingletonFloorDivFolder and add tests for floor division simplification
* [Refactor] Simplify tensor conversion in dequantization functions and improve performance in GEMM examples
[BugFix][Transform] Freshen buffer defs after loop unroll
UnrollLoop clones each loop body with Substitute, which preserves Buffer object identity for definitions inside the body. This can define the same DeclBuffer handle in multiple unrolled copies after flattening and violate TIR SSA. Freshen buffer definitions per cloned body while preserving DeclBuffer backing data aliases.
Co-authored-by: LeiWang1999 <leiwang1999@outlook.com>
* [Backend] Refactor Pipeline to support different backends
Introduce Pipeline abstraction in backend/pipeline.py with per-backend
registration. Each backend (cuda, hip, c, llvm) now registers its own
compilation pass pipeline. engine/lower.py resolves the pipeline from
the target instead of hardcoding phase imports.
* enhance the structure
* fuse two stage
* fix after rebase
* update template pass pipeline
* pipeline
* fix test
* fix test
* fix cuda pipeline prologue
* [Refactor] Extract shared-access analysis into a reusable header
- Move AccessEntry/AccessType definitions from thread_storage_sync.cc to
src/transform/common/shared_access_analysis.h so other passes (notably
MergeSharedMemoryAllocations) can reuse the same precise access
representation.
- Add tl.debug_thread_storage_sync_summary pass-config option for optional
ThreadSync planner summary logging.
- Add regression tests covering explicit shared sync deduplication and
shared.dyn alias-aware dependency planning.
* [Feature] Rewrite MergeSharedMemoryAllocations with precise access analysis
- Build per-access live ranges using shared_access_analysis utilities so
branch-alternative dataflow and barrier-driven epoch boundaries are
modeled directly instead of through whole-buffer approximations.
- Plan arena layout starting from arena_size_const so allocations whose
starting offsets are non-zero no longer collide.
- Preserve single-slot semantics for mutually exclusive branch writes
followed by a post-dominating read (fixes the K_norm_pairs_shared style
miscompile where stage-1 alternatives were split across epochs).
- Cover BufferRegion block reads/writes, tvm_access_ptr, tl.access_ptr,
cp_async branches, loop-carried buffers and partial-subregion liveness
cases.
- Keep aggressive-merge disabled under warp specialization and retain
whole-buffer liveness as the conservative default.
- Add 18 transform regressions in
test_tilelang_transform_merge_shared_memory_allocations.py covering
basic merge, sync, branch, cp_async, BufferRegion and opaque-pointer
paths.
* [Cleanup] Remove dead FreeList/FreeBlock/ActiveInterval helpers
These helpers were superseded by the segment-aware LinearScanPack
implementation that performs interval-overlap walking directly. They
have no remaining callers after the rewrite; remove them to reduce
maintenance burden.
* [Test] Cover tail-allocation arena growth scenario
Add a regression case mirroring upstream b4c913be (#2106): when the only
free slot reusable by an incoming buffer sits at the arena tail and is
smaller than the request, the arena must grow by the deficit only, not
push the new buffer past the freed slot.
The legacy free-list-based packer would emit a 384 B arena for this
pattern (X=128, Y=64-then-freed, Z=192), bumping Z to offset 192. The
interval-based LinearScanPack reuses Y's offset and grows the tail to
fit Z, producing a 320 B arena -- a 16.7%% reduction that matches the
peak-live lower bound.
* [Feature] Add EpochGraph and use per-epoch liveness to relax aliasing
Introduce src/transform/common/epoch_graph.h, a control-flow-aware
builder that partitions a flattened StmtEntry sequence into hard-sync
and cp_async-wait epochs across nested For/IfThenElse scopes, exposing
loop-carry, branch-join and parent-edge structure required for sound
liveness analysis.
Wire MergeSharedMemoryAllocations to compute per-buffer live epochs as
the intersection of:
* forward def-reach: epochs reachable from any write of the buffer
along epoch-graph successors;
* backward use-reach: epochs that can reach any read of the buffer
along predecessors.
A buffer is live in epoch e iff e is in both sets, modeling 'data
written upstream is consumed downstream' rather than the legacy
[start_stmt, end_stmt) bounding box. The result feeds two relaxers:
* LinearScanPack: two intervals whose live-epoch sets are disjoint
may share an offset even if their stmt ranges overlap.
* PlanMemory: the symmetric overlap probe consults the same set
before forcing a non-aliased layout.
Hard syncs (kSync) and cp_async waits are treated as epoch boundaries;
ptx_cp_async_wait is conservatively classified as a hard barrier.
IfThenElse joins take the union of branch live sets so a buffer
written in only one branch remains live at the join.
The whole-buffer EpochGraph + liveness pipeline always runs; verbose
[MSMA-EPOCH-GRAPH] / [MSMA-EPOCH-LIVE] / [MSMA-EPOCH-ACCESS] /
[MSMA-EPOCHSEG] / [MSMA-STAGE0-RAW] dumps remain gated behind the
existing tl.debug_merge_shared_memory_allocations pass-config flag.
Also remove SplitLocalSegmentsAtEpochSyncs, a Phase-1 helper that was
defined but never wired into the planner; the new liveness pipeline
subsumes its role.
Validated on a real lowbit MHA decode kernel
(seq_lens=[(3,96),(2,64)], heads=4, dim=64, page_size=32, bits=4):
arena drops from 35,232 B (upstream baseline) to 31,136 B, an 11.6%%
reduction driven by K_shared and V_shared sharing one offset thanks
to disjoint live-epoch sets.
Add testing/python/transform/test_tilelang_transform_epoch_graph.py
covering the five S1.4 epoch-graph topology cases (sequential
__syncthreads, ptx_wait_group, For body/back/exit edges, IfThenElse
branch_join, nested For). Existing
test_tilelang_transform_merge_shared_memory_allocations.py and
test_tilelang_transform_thread_sync.py suites continue to pass.
* [Feature] Gate per-epoch alias relaxer on annotated layout equality
Two shared-memory buffers carrying distinct layouts under
`block.annotations[attr::kLayoutMap]` (e.g. via `T.annotate_layout(...)`)
must not share storage even when per-epoch dataflow proves their
liveness sets disjoint: downstream layout-driven access lowering can
emit indexing arithmetic that is only sound under each buffer's
original layout, so naively aliasing them risks silent miscompiles.
Add a `LayoutSignatureCollector` that harvests the per-Var `Layout`
entries from `Block` annotations and a `LayoutAliasIncompatible`
helper that returns true iff both candidates have non-trivial layouts
and the layouts are not structurally equal. The two relaxer call
sites in `SharedMemoryRewriter` (`LinearScanPack` and the post-pack
overlap audit) consult the helper before dropping a conflict, so the
legacy decision is preserved exactly when the gate fires.
Buffers without a layout entry remain layout-agnostic and continue to
participate in arena packing, keeping the existing reuse for the
common un-annotated case. Verified that the lowbit reference kernel
arena footprints stay at 31136/61856/61808/61792 B and the merge /
thread-sync / epoch-graph trio passes 49/49.
* [Test] Add transform regression that fails on upstream MergeSharedMemoryAllocations
Add test_general_cond_branch_alternatives_share_offset_with_postdom_buffer
to lock the per-epoch / branch-alternative analysis behavior on a
mutually-exclusive if/else with a runtime cond plus a post-dominator
buffer after a sync. Upstream's linear-scan plan keeps the two branch
buffers at distinct offsets (128-byte arena); the per-epoch relaxer
collapses them onto the same offset and reuses it for the
post-dominator buffer (64-byte arena, 50% smaller).
* [Example] Demo MergeSharedMemoryAllocations branch-alternative arena reuse (64 vs 128 B)
Add runnable demo under examples/merge_shared_memory_allocations/
that constructs a three-buffer PrimFunc with mutually-exclusive if/else
branches, lowers through the merge pass, and reads the merged arena
size from the dyn_shared_memory_buf attr. The demo also compiles the
kernel end-to-end and validates output correctness with torch.allclose.
On this branch: arena = 64 B (saved 128 B / 66.7% vs naive 192).
On upstream main (1fbd994c): arena = 128 B (-50% worse).
* [Test] Add end-to-end torch correctness regression for merge pass
Add two tests (test_merged_branch_kernel_generates_correct_results,
test_merged_sync_phased_buffers_generate_correct_results) that compile
the merged kernel and verify bit-exact output via torch.allclose,
ensuring the arena packing does not alter numerical results.
* [Test] Add torch correctness and dynamic-buffer fallback regression
- test_merged_branch_kernel_generates_correct_results: compiles a
merged branch-alternative kernel with torch.allclose validation.
- test_merged_sync_phased_buffers_generate_correct_results: sync-phased
two-buffer kernel end-to-end check.
- test_dynamic_size_buffer_fallback_does_not_incorrectly_alias:
verifies that a symbolic-extent shared.dyn buffer correctly aliases
with a sync-separated constant-size buffer without unsafe overlap.
* [Fix] Disable per-epoch relaxer for loop-live buffers to avoid pipelined iteration overlap
* [Fix] Fallback to structural_segments_by_var for buffers missing planner segments to avoid buffer_byte_offsets crash and shared memory bloat
* Merge branch 'main' into merge_pass and fix merge comflicts
* [Fix] Resolve post-merge artifacts in CMakeLists, hipcc, and deepseek example
- Remove duplicate TILELANG_OUTPUT_TARGETS in CMakeLists.txt
- Fix duplicate _resolve_artifact_paths function in hipcc.py
- Add missing G=kv_group and fix assert in sparse_mla_fwd_pipelined.py
- Add ruff noqa pragma to suppress lint warnings
* [Style] Apply clang-format to modified source files
* [Pipeline] Refactor software pipeline transforms
* enhance
* Refine pipeline handling for guarded binds
* Refactor IfStmtBindingRewriter and related classes for improved access analysis
- Removed unused Target parameter from IfStmtBindingRewriter.
- Introduced IfStmtAccessCollector to handle buffer access collection.
- Updated access analysis methods to utilize the new collector.
- Cleaned up includes and removed obsolete functions in access_analysis.
- Enhanced pipeline utility headers with necessary includes for consistency.
* [Bugfix] Fix issue 2123 access_ptr lowering
Add the issue 2123 regression test and keep the access_ptr base BufferLoad intact through safe-memory legalization and lowering.
Closes#2123
* Deduplicate access_ptr base load handling
Move shared tl.access_ptr BufferLoad base visitation into a common transform helper so LowerAccessPtr and safe-memory legalization stay aligned.
Extend the issue 2123 regression with a direct LowerAccessPtr pass check while keeping the LowerAndLegalize pipeline coverage.
* Format access_ptr helper header
* Narrow access_ptr helper scope
* Share access_ptr base helper
* Address access_ptr helper review comments
* [Transform] Preserve access_ptr buffer loads in safe memory legalization
* Refactor AccessPtrInfo return structure in legalize_safe_memory_access.cc
Updated the return structure of AccessPtrInfo to improve readability and maintainability. The logic for handling the tvm_access_ptr operation has been streamlined, ensuring clearer checks and consistent formatting of returned values.
* fix
---------
Co-authored-by: LeiWang1999 <leiwang1999@outlook.com>
* [Example] Migrate eligible examples to eager style
* fix wrong shape annotation in gemm_split_k examples
* fix incorrect dtype in gemm_streamk example
* fix
* revert some changes
* Refactor attention functions to use new parameter structures and improve readability. Updated `example_amd_flash_attn_bwd.py` and `example_amd_flash_attn_fwd.py` to define input parameters more clearly. Enhanced shared memory usage in `flashattn_fwd` and `flashattn_sink` functions across various examples, ensuring compatibility with new configurations. Adjusted kernel definitions and added necessary checks for GPU capabilities.
* lint fix
* FIX
---------
Co-authored-by: LeiWang1999 <leiwang1999@outlook.com>
* Preserve shared memory aliases after dynamic merge
* Enhance SharedMemoryRewriter to support alias preservation and buffer remapping during shared memory allocation merging
* lint fix
* Add TL_DISABLE_MERGE_SHARED_MEMORY_ALLOCATIONS pass config
* Rename to TL_DISABLE_SHARED_MEMORY_REUSE and implement disable_reuse in C++
Instead of skipping MergeSharedMemoryAllocations entirely, add a disable_reuse
flag that still merges allocations into one but lays out each buffer sequentially
without lifetime-based sharing.
* Fix build: use AllocBufferNode instead of tir::AllocateNode in PlanSequentialLayout
* Add test for TL_DISABLE_SHARED_MEMORY_REUSE pass config
morluto