mirror of
https://github.com/tile-ai/tilelang.git
synced 2026-10-24 07:04:00 +08:00
6873e19da2b267d3cb7d0305561ba0bed63feae3
21
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chennes
|
7fd95363cc |
[CUDA] Extend the GEMM FMA fallback to SM75 (#2811)
On SM75, SelectInst routed GEMMs to the plain MMA path even when their dtype combination had no native Turing mma.sync atom. In particular, fp32, fp64, and bf16 selected SM80-only atoms: bf16 failed an nvcc static_assert, while fp32 and fp64 compiled into runtime traps that invalidated the CUDA context. Add AllowTuringMma alongside the Volta check introduced in #2339 and select the existing cuda.fma implementation when SM75 has no native atom. Keep the check dtype-only because the SM75 emitter supports the same operand scopes and transposes as the generic MMA path. Valid f16 combinations remain on native MMA rather than being demoted to the slower and less accurate serial FMA fallback. Add compile-only dispatch checks pinned to an explicit sm_75 target, so they run on any CUDA runner, verifying that fp32, fp64, and bf16 use the fallback while f16 remains on MMA. Add execution checks gated on compute capability 7.5. |
||
|
cjandcaojian5
|
2a06036faa |
[BugFix] Reject mixed packed x2 operand dtypes (#2802)
* Reject mixed packed x2 operand dtypes * Cover mixed first fma2 operand --------- Co-authored-by: caojian5 <caojian5@huawei.com> |
||
|
Yongqi Zhuo
|
1cb4d4f3ac |
[CUDA] Support arbitrary TMEM layouts (#2785)
* [CUDA] Support arbitrary TMEM layouts * Fix bugs |
||
|
wcxandSiriusNEO
|
51fbfc7ee6 |
[BugFix] Fallback non-16B cluster bulk copies (#2683)
* [BugFix] Fallback non-16B cluster bulk copies * test: tolerate typed cluster copy size diagnostics --------- Co-authored-by: SiriusNEO <chaofan@deepseek.com> |
||
|
cklxx
|
2a17fffd44 |
[TileOP] Add SM70 GEMM FMA fallback (#2339)
* [TileOP] Add SM70 GEMM FMA fallback * [TileOP] GemmFMA: drop redundant pre-staging sync and add docstrings Address PR #2279 review: - coderabbitai nit: the T.sync_threads() before the cooperative A/B stage loads is unnecessary because A_stage / B_stage are freshly allocated shared buffers — no other thread holds references to them, and the later sync before the GEMM compute is sufficient. - Docstring Coverage pre-merge check: add module-, helper-, method-, and prim_func-level docstrings so coverage clears the 80% threshold. No behavior change; the prim_func still stages through shared memory and accumulates via scalar FMA. SM70 pytest trio still expected to pass. * [TileOP] Harden SM70 GemmFMA fallback * [TileOP] Fix SM70 FMA staging dtypes * [TileOP] Tighten SM70 MMA dispatch for transposed A |
||
|
Lei Wang
|
30aac1acdd |
[CUDA][Reduce] Fix packed AllReduce workspace pointer (#2778)
Update the TVM submodule for vector-typed access pointer lowering and keep generated reduction workspace accesses read-write. Add an SM100 regression for packed float32 reductions across warps. |
||
|
Yongqi Zhuo
|
1e07514959 | [CUDA] Support fp32x2 ops as reducers (#2637) | ||
|
Lei Wang
|
465c6c5e8c |
[CUDA] Reduce template include overhead (#2474)
* [Refactor] Remove unsupported tl_gemm and tl_gemm_sp operations * Removed references to tl_gemm and tl_gemm_sp from various files, including code generation and built-in definitions, as these operations are currently unsupported. * Updated related comments and diagnostics to reflect the removal of these operations, ensuring clarity in the codebase. * This change simplifies the handling of unsupported operations and improves overall code maintainability. * Reduce CUDA template include overhead * Fix CUDA reduce fast min max helpers * lint fix * Fix CUDA swizzle ceil div helper * Restore CUDA wait_wgmma intrinsic helper * Fix CUDA cython TMA host adapter include * Remove unnecessary check for Cutlass fast math header in CUDA code generation * Update LibraryGenerator to correctly handle CUDA gencode flags for shared-library compilation, ensuring compatibility with target architecture and avoiding Hopper-only instruction issues. * Refactor smem_ptr_to_uint function to use __cvta_generic_to_shared for improved clarity and performance. * Refactor example scripts and CUDA math intrinsics for improved performance and clarity - Updated `example_dequant_gemm_fp4_hopper.py` to directly call `run_regression_perf()` and print latency, removing unused argument parsing code. - Modified `example_topk.py` to specify the backend as "cupti" in the benchmarking function. - Enhanced CUDA code generation by introducing `RequiresTileLangMathHeader` to manage math header dependencies for specific functions. - Updated barrier operations in `barrier.h` to use the correct syntax for shared memory barriers. - Improved testing for bfloat16 math intrinsics by replacing pytest skips with decorators for CUDA availability checks. * Refactor CUDA code for improved readability and consistency - Reformatted the `RequiresTileLangMathHeader` function for better readability. - Updated assembly syntax in `barrier.h` for clarity and consistency. - Cleaned up whitespace in the bfloat16 math intrinsics test file to enhance code cleanliness. * Remove unnecessary whitespace in CUDA code for improved readability * Add TileLang intrinsic for prefetching TMA descriptor - Introduced `prefetch_tma_descriptor` intrinsic in C++ and updated relevant files to use `call_intrin` instead of `call_extern`. - Updated CUDA code generation to handle the new intrinsic correctly. - Modified tests to ensure the new intrinsic is called appropriately and that no `call_extern` is used for internal TileLang operations. - Enhanced documentation in SKILL.md regarding the addition of new TileLang intrinsics. * Add bfloat16 hexp overload and update common.h for compatibility - Introduced `hexp` overload for `bfloat16_t` in `common.h` to bridge TileLang and CUDA's handling of exponential functions. - Updated tests to verify the presence of the new `hexp` definition in `common.h`. - Enhanced CUDA code generation to support the new intrinsic for better performance and compatibility. * Update common.h for CUDA compatibility with aligned double4 types - Adjusted preprocessor condition in `common.h` to maintain compatibility with CUDA versions prior to 13 by ensuring proper definition of aligned double4 types. - This change enhances the compatibility of the code with older NVRTC built-in vector headers. |
||
|
Chennes
|
c9e5112332 |
[BugFix] Fix bf16 CUDA exp self-recursion (#2402)
`hexp` is #define'd to `cutlass::fast_exp`, so the hand-written `cutlass::fast_exp(bfloat16_t)` overload's `return ::hexp(x)` expanded into a call to itself: bf16 `T.exp` recursed and returned garbage. Route through float, mirroring cutlass's own `half_t fast_exp` (cutlass/fast_math.h), so the call resolves to `cutlass::fast_exp(float)` (= ::expf). Verified against torch.exp using the #2383 repro. Fixes #2383. |
||
|
Chennes
|
b7489e2608 |
[CUDA] Add SM75 MMA dispatchers for FP16 accumulation and UINT8 (#2392)
The SM75 GEMM emitter derives MMA shapes from input dtype bits alone, so it already emits the (16,8,8) shape for fp16->fp16 accumulation (identical to the registered fp16->fp32 case) and the (8,8,16) shape for uint8 inputs (identical to signed int8). But the C++ dispatcher table only covered the three configs tested by #2198 (fp16->fp32, int8, int4), so requesting fp16->fp16-accum or uint8 GEMM on Turing failed at nvcc with an opaque `tl::mma_sync: unsupported configuration` static_assert. Add the two missing tl::detail SM75 atoms and register their dispatchers, following the precedent set by #2198's hand-written int4/uint4 structs (CuTe ships only fp16->fp32 and signed-int8 SM75 atoms). The uint8 atom uses the non-saturating PTX form to match the existing signed-int8 path. Verified on a TITAN RTX (SM75): emitted PTX is mma.sync.aligned.m16n8k8.row.col.f16.f16.f16.f16 and mma.sync.aligned.m8n8k16.row.col.s32.u8.u8.s32, and end-to-end GEMM numerics match torch. |
||
|
赖泉laiquanandMAC
|
17c4b384df |
Fix bf16 CUDA rsqrt overload (#2386)
fix: add bf16 CUDA rsqrt overload Co-authored-by: MAC <mac@MacBook-Air.local> |
||
|
Lei Wang
|
d238837022 |
[Transform] Materialize kernel launch per backend (#2387)
* Refactor kernel launch materialization * [Refactor] Improve git commit ID retrieval and version labeling logic * Enhance the get_git_commit_id function to check for the existence of the .git directory before attempting to retrieve the commit hash, ensuring more robust behavior. * Update the dynamic_metadata function to conditionally set the NO_VERSION_LABEL environment variable based on the presence of the .git directory. * Modify the GitHub Actions workflow to unset the NO_VERSION_LABEL environment variable during the installation and version check of the package, allowing for accurate version labeling in SDist builds. * [Refactor] Update WebGPU backend registration * Import the WebGPU package to automatically register its pass pipeline. * Remove the explicit registration of the WebGPU pass pipeline, streamlining the backend setup process. |
||
|
|
b939fa0137 |
[TIR][IR] Update to use tirx (#2216)
* [Feature] Implement Hoist Loop-Invariant If Pass - Added a new pass to hoist loop-invariant if statements out of loops, improving optimization opportunities. - Introduced classes for collecting and checking conditions, as well as for rewriting statements. - Integrated the new pass into the optimization pipeline and provided a corresponding API for usage. * Add tilelang semantics guide to programming guides (#1740) * Add CallNodeChecker to Loop Unswitching Pass and integrate tilelang cache disabling in benchmark script - Introduced CallNodeChecker class to identify CallNode expressions in loop conditions, enhancing loop-invariant checks. - Updated IsLoopInvariant function to reject conditions containing CallNodes, preventing potential side effects. - Added tilelang.disable_cache() in benchmark_mha_sink_fwd.py to optimize performance during benchmarking. * lint fix * lint fix * Enhance Loop Unswitching Pass with Let Binding Support - Added support for Let-bound variables in the WrittenBufferReadChecker to improve buffer read checks. - Introduced UsesLoopVarThroughLetBindings function to check if conditions depend on loop variables through Let bindings. - Updated IsLoopInvariant function to account for Let bindings when determining loop invariance. - Enhanced HoistableIfFinder to track Let bindings for variables bound to BufferLoad expressions. - Added debug print statements in the OptimizeForTarget function to visualize the module state before and after loop unswitching. * fix * lint fix * Enhance Loop Unswitching with CallNode Exclusion Check - Added CallCheckerExcludingIf class to ensure function calls outside of hoisted if statements are identified, preventing potential synchronization issues during loop unswitching. - Updated loop unswitching logic to incorporate the new call checker, enhancing safety and correctness. - Integrated debug print statements in OptimizeForTarget to visualize module state before and after loop unswitching. - Disabled tilelang cache in the benchmark script for improved performance. * Update TVM subproject to latest commit and remove debug print statements in OptimizeForTarget function * Add support for disabling loop unswitching optimization - Introduced a new configuration option `tl.disable_loop_unswitching` to allow users to disable the loop unswitching optimization. - Updated the Loop Unswitching pass to check this configuration and return the original function if the option is enabled. - Added relevant documentation in the PassConfigKey enumeration for clarity. * update tvm * Refactor: Update TileLang to use tirx instead of tir - Removed references to `tvm.tir` and replaced them with `tvm.tirx` across various files, including examples and backend operations. - Updated target configuration in documentation to reflect the new usage of target config dictionaries instead of CLI-style strings. - Cleaned up `pyproject.toml` and `load_tvm.cmake` by removing obsolete paths. - Enhanced examples for dequantization and GEMM operations to utilize the new `tirx` constructs. - Adjusted various backend operations to ensure compatibility with the new `tirx` namespace. * pass fix * Update submodule 'tvm' to latest commit f24c7b19 * Apply formatting fixes * Update CMakeLists.txt to set HIDE_PRIVATE_SYMBOLS for TileLang and modify target_host initialization in lower.py for compatibility with TVM's target handling. * Refactor CMakeLists.txt and kernel_cache.py to update target names and library dependencies for TVM integration. Adjusted output targets and library names for better compatibility across platforms. * Update repair-wheel-command in pyproject.toml to exclude libz3.so with wildcard for better compatibility with dependency packages. * Fix macOS Z3 dylib lookup * Move macOS Z3 install-name fix to install step * Update TVM submodule for runtime import fix * Update ROCm backend for TVM module factory changes * Fix build-tree Z3 runtime lookup * Fix macOS build-tree Z3 lookup * Support legacy target option strings * Normalize CUDA target in transform test * Qualify Metal stream FFI lookup * Fix PyPI Z3 packaging links * Update TVM for Windows DLL lookup * Register Windows dependency DLL directories * Bump tvm-ffi to match tirx * tvm update * Apply lint fixes * Enhance reduction operations with support for nan propagation and vectorization. Introduce new functions for preferred vectorized size and update existing reduction logic to handle packed operations for bfloat16 and float16 types. Add nan-aware min and max operations in CUDA and ROCm backends, and update related tests to validate functionality. * Refactor ReduceLowerer: Remove unnecessary buffer allocation logic for packed operations in reduce.h * Normalize PyPI Z3 dylib before linking * Drop legacy target option string support * Fix type reference for barrier annotations in CUDA copy operations. Updated `BufferLoadNode` to `tirx::BufferLoadNode` in `GetBarrier` and `LowerCluster` methods to ensure compatibility with recent changes in the TIR API. --------- Co-authored-by: Yu Cheng <yu.cheng@pku.edu.cn> Co-authored-by: Yichen Yan <oraluben@outlook.com> |
||
|
|
cd02a30c46 |
[Feature] Add T.copy_cluster to support TMA multicast and SM-to-SM cluster copy (#1908)
* [Feature] Extend t.copy to support TMA multicast and SM-to-SM cluster copy Add multicast copy (tma_load_multicast) and shared-memory-to-shared-memory cluster copy (tma_store_cluster / ptx_cluster_store) to the t.copy operator. Multicast copy: - A single CTA issues tma_load_multicast to broadcast a tile to multiple CTAs in the cluster simultaneously; other masked CTAs receive passively. - New `cluster_mask` parameter on T.copy() controls which CTAs participate. SM-to-SM cluster copy: - Fast path via cp.async.bulk.shared::cluster with mbarrier synchronisation (new `remote_barrier` parameter on T.copy()). - Slow path via cooperative_groups::map_shared_rank for element-by-element fallback (new `dst_block` parameter on T.copy()). Supporting changes: - New builtins: tma_load_multicast, tma_store_cluster, ptx_cluster_store, cluster_sync, get_cluster_id, get_cluster_block_rank, get_cluster_block_nums, mbarrier_arrive (cluster-scope). - Codegen (codegen_cuda.cc) and device templates (barrier.h, copy_sm90.h) for all new intrinsics; unified mbarrier API via tl::mbarrier_* free functions. - inject_tma_barrier: handle tma_load_multicast; distinguish thread guards from cluster-rank conditions for correct expect_tx injection. - lower_hopper_intrin: migrate barrier allocation to builtin::create_barriers; hoist user ptx_init_barrier_thread_count calls alongside compiler barriers. - warp_specialized_rewriter: support tma_load_multicast as producer op; UserBarrierInitExtractor to separate user barrier inits from body; ragged (dynamic-extent) pipeline prefix counter; fix nested IfThenElse state corruption in shuffle-elect optimisation. - multi_version_buffer_rewriter: ragged pipeline support with runtime prefix counter for correct ping-pong buffer versioning. - thread_storage_sync: reserve fixed barrier IDs (kProducer/kConsumer) for 256-thread CTA warp-specialised split to prevent deadlocks. - pipeline_planning: defensive handling of non-BufferLoad mbarrier args. - Shared memory alignment bumped to 128 bytes for TMA. Co-authored-by: Guangda Sun <2012661711@qq.com> * [Docs] Add programming guide for Cluster TMA features Add docs/programming_guides/cluster_tma.md covering the two new T.copy extensions introduced in the t_copy_extend feature branch: - TMA multicast (cluster_mask): explains how a single TMA transaction broadcasts one global tile to multiple CTAs in a cluster simultaneously, with API usage, a per-rank behaviour table, and a complete code example. - SM-to-SM cluster copy (dst_block / remote_barrier): documents the fast path (cp.async.bulk.shared::cluster + mbarrier, single-thread async DMA) and the slow path (map_shared_rank element-by-element SIMT fallback), including the synchronisation contract for source and destination CTAs. Also covers cluster helper builtins (T.get_cluster_block_rank, T.cluster_sync, T.mbarrier_init/arrive/wait_parity, etc.) and a split-K sketch combining both features end-to-end. * change cluster_size -> cluster_dims * fix merge conflict * fix TILELANG CHECK bug * unify cluster function * fix pre-commit errors * docs(cluster_tma): clarify multicast non-issuer behavior in lowering note * fix(copy): gate dst_block cluster lowering on target capability * fix(cuda/cluster): hard-trap when cluster intrinsics are unavailable instead of assert(false)+dummy return. * fix inject_tma & lower_hopper & lower_tile bugs * [warp specialize]Fix mbarrier retarget for tma_load_multicast in warp-specialized producer path * fix(tma-barrier): correct arrive thread count under equality-guarded threadIdx * fix: overlapped TMA barrier IDs * fix: multicast checks * minor fix * minor fix * fix: tma_load_multicast() in the stage-local mbarrier rewrite * minor fix * format * minor fix * fix: Track allocation from every hoisted barrier init. * fix: Scope the 128-byte alignment to kernels that actually use TMA. * fix: Assert the tl::tma_store_cluster lowering, not just the output. * format fix * fix: Don't assume both if arms transfer the same number of bytes. * fix: Don't wrap the kept pipelined loop in a SeqStmt before the For post-processing runs. * rm mbarrier_init and use alloc_cluster_barrier * fix: Don't make cluster copy a global exemption for unrelated TMA loads. * fix: Don't drop dependency modeling for handle-based mbarrier_wait_parity(). * fix: Don't hoist extracted barrier-init if statements to the block root. * fix(copy): honor remapped dst layout in cluster-copy slow path * fix(copy): gate cluster TMA fast path on provable contiguity * fix(copy): add barrier completion for SIMT cluster-copy fallback * fix: remove dup cluster_sync * fix: remove not used _get_mbarrier * fix: remove mbarrier related code in codegen_cuda.cc and barrier.h * fix: codegen_cuda.cc: reuse functions in cluster.h and remove dependency on cg * fix testting * fix(transform): prevent ragged_prefix free var from breaking MakePackedAPI * minor fix * format * fix inject tma barrier: fix crash and correct mbarrier thread-count injection * revert inject tma * fix(inject_tma_barrier): avoid SIGSEGV and correctly infer barrier arrive thread counts * fix: add tma_load_multicast same as tma_load * T.copy_cluster * fix: renaming cluster mbarrier_arrive to ptx_arrive_cluster_barerier * fix: remove mbarrier_arrive in builtin.cc * test_tma_dsmem: fallback test * [Feature] Multi-TMA fallback for non-contiguous T.copy_cluster regions Previously, a non-contiguous copy region (e.g. a 2-D slice where the inner dim does not span the full buffer width) with a remote_barrier fell back to element-wise SIMT stores, losing TMA throughput. This commit introduces a "multi-TMA" middle path in LowerClusterCopy: the ND copy region is recursively decomposed into individual contiguous rows, each emitting one tma_store_cluster call. The mbarrier arrive_count is updated to N (the number of rows) so the destination CTA's wait() still works correctly without any API change. Key changes: - src/op/copy.cc: add static MakeTMARows() that recursively splits an ND region into contiguous rows; static extents are unrolled at compile time, symbolic extents produce TIR For loops. - src/op/operator.h: add UpdateBarrierArriveCallback (Var → PrimExpr) to LowerArgs so LowerClusterCopy can propagate the new arrive_count. - src/transform/lower_tile_op.cc: collect arrive_count overrides in barrier_arrive_updates_ and apply them to the barrier_init block annotation after visiting the block body, before LowerSharedBarrier consumes it. - testing/python/cuda/test_tma_dsmem.py: replace the old SIMT-barrier test with test_store_cluster_multi_tma_barrier (2-D, M rows) and add test_store_cluster_3d_multi_tma (3-D, D×M rows) to cover the two-level recursive decomposition. * bug fix * fix: unified continuous check * fix doc * fix: remote redundant annotation converting * fix: remove unused code * fix: remove unused code * rm unused comments * minor fix * fix: rm unused helper func * minor * minor * Clean up copy cluster merge formatting * Reduce copy cluster PR churn --------- Co-authored-by: Guangda Sun <2012661711@qq.com> Co-authored-by: LeiWang1999 <leiwang1999@outlook.com> |
||
|
ベアトリーチェandLeiWang1999
|
e9b7e21de9 |
[CUDA] Add native SM75 MMA GEMM support for FP16, INT8 and INT4 (#2198)
* [CUDA] Add SM75 MMA GEMM support * [CUDA] Add SM75 INT4 MMA GEMM support * [Lint] Apply pre-commit formatting * [CUDA] Address SM75 MMA review feedback * [CUDA] Fix SM75 MMA emitter review issues * [CUDA] Update SM75 MMA emitter tests and refactor TensorCoreIntrinEmitter to remove Turing-specific logic * [Testing] Add main entry point to CUDA MMA and kernel tests * [CMake] Update CMakeLists.txt to ensure proper CUDA compiler detection and project declaration * [CUDA] Enhance SM75 GEMM support by adding implementation and updating tests * lint fix --------- Co-authored-by: LeiWang1999 <leiwang1999@outlook.com> |
||
|
Lei WangandClaude Opus 4.6
|
3ee0988c86 |
[Refactor][Pipeline] Run pipeline rewriting before layout inference and stabilize tiled WS (#2002)
* Refactor optimization pipeline in phase.py to streamline barrier handling
- Consolidated the handling of shared barriers and pipeline planning by removing redundant conditional checks.
- Ensured that `LowerSharedBarrier`, `PipelinePlanning`, and `InjectSoftwarePipeline` are consistently applied, enhancing the clarity and efficiency of the optimization process.
This change improves the maintainability of the code while preserving existing functionality.
* Enhance tile operation handling with instruction annotation and refactor gemm.h
- Added a new `InstructionAnnotation` pass to annotate tile operations with their instruction kind before layout inference, improving the optimization pipeline's ability to reason about instruction mixes.
- Refactored `gemm.h` to move the `allowTcgen5Mma` and `allowWgmma` methods under the private section, enhancing code organization and encapsulation.
These changes improve the clarity and maintainability of the code while preserving existing functionality.
* Enhance multi-version buffer handling and introduce tile-level warp specialization
- Updated `multi_version_buffer_rewriter.cc` to improve read/write access detection for tile operations by analyzing `tl.tileop.region` calls, ensuring accurate buffer access tracking.
- Modified `phase.py` to integrate `ProducerConsumerWarpSpecializedTiled` before layout inference, allowing for high-level tile-op IR transformations that enhance producer/consumer splits.
- Added a new `ProducerConsumerWarpSpecializedTiled` function in `__init__.py` to facilitate tile-level warp specialization, improving the optimization pipeline's efficiency.
These changes enhance the handling of multi-version buffers and optimize the transformation process for tiled operations.
* Enable tiled WS for stage-1 pipelines
* Move Hopper pipeline planning before layout inference
* Run pipeline rewriting before layout inference
* Add pipeline refactor WIP: subtree_modified_ guard, debug prints, and plan
- inject_pipeline.cc: guard reads/writes recalculation with subtree_modified_
flag to prevent local.var buffer promotion to kernel parameters
- phase.py: add temporary debug prints after PipelinePlanning and
InjectSoftwarePipeline (to be removed during CI fix work)
- example_group_per_split_token_cast_to_fp8.py: add disable_cache for debugging
- docs/plan.md: implementation plan for fixing CI test failures
- draft.md: original design draft
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Remove debug artifacts: prints in phase.py, disable_cache in example
- Remove print("After PipelinePlanning"), print(mod),
print("After InjectSoftwarePipeline"), print(mod) from
LowerAndLegalize in tilelang/engine/phase.py
- Remove tilelang.disable_cache() from
examples/cast/example_group_per_split_token_cast_to_fp8.py
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Fix LayoutInference crash on multi-versioned pipeline buffers
After InjectSoftwarePipeline, multi-versioned buffers share the same
data Var as the original but have an extra leading dimension (num_stages).
LayoutInference's alias propagation and annotation handling tried to
Reshape layouts between these buffers, which failed because the total
element counts differ.
Guard three Reshape call sites in layout_inference.cc to skip sibling
buffers whose total storage size is incompatible with the source layout.
This lets multi-versioned buffers get their own layout inference instead
of inheriting an incompatible layout from the original buffer.
Fixes compilation failures in dequantize_gemm, GDN, and other kernels
that use software pipelining with shared memory buffers.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Strip pipeline annotations on WS fallback for TMA kernels
When ProducerConsumerWarpSpecializedTiled identifies a TMA kernel as a
warp-specialization candidate but the tiled rewriter cannot handle it
(e.g., conditional loop bodies like sparse block masks), the fallback
previously returned the original function with num_stages annotations
intact. PipelinePlanning and InjectSoftwarePipeline would then generate
non-WS TMA pipeline code with broken barrier phase tracking for
conditional pipeline bodies (barrier waits outside conditionals cause
deadlocks when the condition is false).
Fix: on WS fallback, strip num_stages annotations from pipeline loops
so that the pipeline passes skip the function. The kernel runs
unpipelined but correctly.
Fixes CUDA_ERROR_LAUNCH_FAILED in blocksparse_gemm and related TMA
kernels with conditional loop bodies.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Add guarded phase-counter support to tiled WS pass
Port PhaseCounter and StageExprReplacer from the legacy
ProducerConsumerWarpSpecialized pass into the tiled WS pass to handle
conditional loop bodies (e.g., sparse block masks).
When the pipeline loop body is wrapped in an IfThenElse without else:
1. Unwrap the condition before classifying statements
2. Create separate producer/consumer PhaseCounters (local int32 buffers)
3. Use counter-based stage/parity expressions instead of loop-variable
4. Wrap producer and consumer bodies in the original condition
5. Increment counters at end of each guarded iteration
6. Rewrite shared-buffer stage indices via StageExprReplacer
This ensures barrier parity stays correct when iterations are
conditionally skipped, fixing CUDA_ERROR_LAUNCH_FAILED in
blocksparse_gemm and related TMA kernels with conditional execution.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Add GemmSPNode support to pipeline planning and injection
Add GemmSPNode handling to:
- inject_pipeline.cc AddReadsWritesForTileOp: model A, E, B as reads
and C as write (E is the sparse metadata buffer)
- pipeline_planning.cc: same access model for dependency analysis
This makes sparse GEMM visible to the pipeline machinery for correct
stage assignment and buffer multi-versioning. However, the tile-op's
consumer-side buffer accesses still don't get stage-indexed because
the pipeline body rewriter can't rewrite high-level tile-op Call
arguments (an architectural limitation of running InjectSoftwarePipeline
before LowerTileOp).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Fix GemmSPNode::Lower to use MakeAccessPtrFromRegion
Replace whole-buffer access_ptr(1) calls with MakeAccessPtrFromRegion
for A, B, C, and E buffers in sparse GEMM lowering. This preserves
stage-specific region offsets from pipeline multi-versioning, matching
the dense GemmNode::Lower pattern.
CUDA output now shows correct stage-indexed consumer accesses:
gemm_sp_ss(..., (k%3)*8192, (k%3)*8192+27648, C_local, (k%3)*2048+49152)
instead of always using stage-0 offsets.
Note: gemm_sp still produces incorrect results because the kernel needs
warp specialization but TiledWSCandidate::Check doesn't recognize it
as a TMA candidate. The non-WS pipeline path generates structurally
different code from the reference WS path.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Fix gemm_sp and seer_attention regressions
Three changes in producer_consumer_ws_tiled.cc:
1. Require num_stages >= 2 for WS candidacy (was >= 1). Single-stage
kernels like seer_attention don't need WS and the transformation
produces incorrect results for them.
2. Add HasTmaPipeline() check to detect TMA kernels with pipeline
annotations that are rejected by the full WS candidate check (e.g.,
kernels with manual layout annotations like gemm_sp).
3. Strip num_stages annotations for rejected TMA pipeline kernels to
prevent InjectSoftwarePipeline from generating broken non-WS TMA
pipeline code.
One change in gemm_sp.cc:
- Use MakeAccessPtrFromRegion for A, B, C, E buffer access pointers
instead of whole-buffer access_ptr. This preserves stage-specific
region offsets from pipeline multi-versioning.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Fix WS conditional body unwrapping for complex loop bodies
Only unwrap IfThenElse wrapper when the then-branch is a simple flat
sequence of tile-op Evaluate calls. Skip unwrapping for complex bodies
with LetStmt, For, or other control flow that could break variable
scoping when split into producer/consumer for WS.
Fixes variable-used-before-definition error in
blocksparse_attention sparse_gqa_decode_varlen_indice, which has a
conditional loop body containing LetStmt bindings.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Revert num_stages >= 2 to >= 1 for WS candidacy
The >= 2 threshold broke test_num_stages_one_pure_tma_keeps_auto_warp_specialize.
Pure TMA kernels with num_stages=1 should still be WS candidates.
The seer_attention issue (num_stages=1 with manual layout) is handled
by the has_manual_layout_ check, not the num_stages threshold.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Propagate manual layouts onto MVB-expanded buffers via Layout::Expand
When LayoutInference encounters an MVB-expanded buffer (with leading
stage dimensions) whose trailing dimensions match the original layout,
use Layout::Expand to propagate the manual layout instead of rejecting
or skipping the buffer. Applied to all 3 layout propagation paths:
annotated layout map, alias propagation, and finalization.
Also remove the blanket !has_manual_layout_ WS candidate rejection
since manual layouts now survive onto versioned shared buffers via
Layout::Expand.
Fixes test_sparse_ws_regular_metadata_copy_stays_in_producer.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Extend conditional body unwrapping for LetStmt chains and restore
has_manual_layout_ guard
1. LetStmt chain peeling: when IfThenElse then_case starts with
LetStmt bindings, peel them and append to let_bindings before
checking the simple-body guard. This allows WS for conditional
bodies with variable definitions (e.g., sparse attention patterns).
2. Restore !has_manual_layout_ in WS candidacy check: removing it
caused dequant_groupedgemm_bf16_mxfp4_hopper to fail because
MXFP4 layouts don't survive MVB expansion. The Layout::Expand fix
handles sparse metadata layouts but not all manual layout types.
3. Layout::Expand propagation (from Round 6) remains in place for
future use when MVB learns to handle all manual layout types.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Remove is_simple_body guard from conditional WS unwrapping
The simple-body guard only accepted flat Evaluate sequences inside
IfThenElse, blocking legitimate WS for complex conditional bodies
like sparse flash attention (T.clear, T.reduce_max, T.gemm inside
the guard). The LetStmt peeling already handles variable scoping.
Fixes test_pure_tma_consumer_local_init_does_not_leak_into_producer
and test_sparse_ws_regular_metadata_copy_stays_in_producer.
The remaining test_mixed_tma_cp_async_shared_stage_barriers failure
is a pre-existing issue on the original branch: the tiled WS pass
produces SIMT copies instead of cp.async because LowerPTXAsyncCopy
is not in the pass pipeline (the comment at phase.py:308 says it
runs earlier but no actual call exists on either branch or reference).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Preserve pipeline context for WS producer cp.async injection
Wrap the WS producer loop body in a kPipelineContextNumStages AttrStmt
so that LowerTileOp's pipelined_depth_ is > 0 when processing SIMT
producer copies. This enables InjectPTXAsyncCopy to generate cp.async
for global-to-shared copies in the WS producer branch.
Without this, the WS rewriter strips all pipeline annotations from
the rewritten loops, causing LowerTileOp to skip cp.async injection
for SIMT producers. The consumer loop stays annotation-free since it
doesn't need cp.async.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Fix mixed barrier protocol, consumer-init sinking, restore manual-layout guard
1. Mixed TMA+cp.async barrier: use ptx_cp_async_barrier_noinc for
forward barrier arrival in mixed producer groups, matching the
reference producer_consumer_ws.cc protocol.
2. Consumer-only pre-loop init sinking: in ReplacePipelineLoopInStmt,
guard pre-loop siblings as consumer-only when they're not classified
as producer (TMA/SIMT/cp.async). Fragment init (T.fill, T.clear)
and local buffer init are placed in the consumer branch instead of
the shared prelude.
3. Restored blanket has_manual_layout_ guard: the dtype-based heuristic
to distinguish sparse metadata from MXFP4 layouts doesn't work
because both use uint8. Dequant_groupedgemm_bf16_mxfp4_hopper
requires the guard to prevent broken WS.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Narrow consumer-init sinking, targeted manual-layout check, per-group barrier
1. Consumer-init sinking: only sink pre-loop stmts that are FillNode
writing to fragment/local buffers. Keep block_mask setup and shared
state in the shared prelude.
2. Manual-layout: attempt targeted check that only rejects when TMA
copy destinations match layout_map entries. Collect layout_map vars
and compare against TMA copy destinations.
3. Per-group cp.async barrier: use group-level cp.async flag (single
group for now) instead of function-wide boolean.
The 3 WS issue tests still fail because the layout_map annotation
parsing falls through to the conservative rejection path.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Fix layout_map parsing with Var→Buffer mapping for manual-layout check
The layout_map annotation uses Map<Var, Layout> before LayoutInference
(not Map<Buffer, Layout>). Parse as Map<ObjectRef, ObjectRef> and
handle both key types: Buffer (post-inference) and Var (pre-inference).
For Var keys, look up the corresponding alloc_buffer by data Var match.
Compare collected manual-layout buffers against pipeline copy
destinations: reject only when a manually-laid-out buffer is also
a producer copy target (TMA/SIMT/cp.async) inside the pipeline.
This allows sparse metadata (E_shared, SIMT-copied) onto the WS path
while rejecting dequant MXFP4 (B_shared, SIMT-copied with swizzle).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Fix SEGFAULT in manual-layout check, use DetectSwizzleMode
1. Fixed SEGFAULT: removed dangling layout_map_layouts_ vector that
was never populated, causing OOB access. Now stores Buffer+Layout
pairs in layout_map_entries_.
2. Use DetectSwizzleMode to distinguish swizzled layouts (MXFP4,
incompatible with MVB) from non-swizzled (sparse metadata, safe).
Swizzled layouts reject WS candidacy; non-swizzled layouts allow it.
3. Removed debug LOG(WARNING) from hot path.
4. Parse layout_map annotation keys as both Buffer and Var (via
Map<ObjectRef, ObjectRef>), resolving Var keys to alloc_buffers.
Results: sparse metadata WS test PASSES, dequant PASSES (protected).
Down to 2 failures: mixed barrier pattern + consumer init sinking.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Simplify consumer-init sinking to Evaluate-only classification
Sink pre-loop Evaluate nodes classified as kConsumer into consumer
branch. Keep For loops (block_mask setup), producer copies, and other
control flow in the shared prelude. This is simpler and safer than
the FillNode scope check approach.
The test_pure_tma_consumer_local_init test still fails because the
T.fill statements are at a different structural level than the
SeqStmt where the pipeline loop lives. Fixing this requires deeper
IR structure analysis.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Sink consumer-only pre-loop init into WS consumer branch
Extract consumer-only pre-loop Evaluate statements (T.fill on
fragments) from the shared prelude and prepend them to the consumer
branch inside the WS if/else structure. This ensures fragment init
like acc_o, logsum, scores_max appears only in the consumer branch,
not in the shared prelude or producer branch.
Uses a two-pass approach: first ReplacePipelineLoopInStmt extracts
consumer-only stmts into extracted_consumer_init_, then the WS body
is rebuilt with the extracted stmts prepended to the consumer branch.
Fixes test_pure_tma_consumer_local_init_does_not_leak_into_producer.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Finalize pipeline refactor fixes
* Refactor CopyNode stride checks for TMA bulk load/store
- Extracted the global stride validation logic into a new static method `CheckGlobalStrides` for better readability and reusability.
- Updated `CheckBulkLoad` and `CheckBulkStore` methods to utilize the new stride checking function, improving code clarity and maintainability.
- Enhanced documentation for the new method to clarify its purpose and requirements.
* Remove redundant copy direction notes from documentation in `copy.h` for stride checks. This simplifies the comments while maintaining clarity on TMA requirements.
* refactor
* layout related fix
* refactor mbarrier with software pipeline.
* remove legacy pass
* Enhance TMA barrier handling and merging logic
- Updated `CopyToTmaCopyRewriter` to conditionally emit arrive barriers based on the last TMA copy.
- Refactored barrier creation to allow merging of TMA barriers when conditions are met, reducing the number of barriers needed.
- Adjusted consumer and producer logic to accommodate merged barriers, ensuring correct synchronization across TMA copies.
- Improved documentation and comments for clarity on the new barrier merging behavior.
* Update example_mhc_post.py to disable main execution and add test function call
- Commented out the main function call in `example_mhc_post.py`.
- Added a call to `tilelang.disable_cache()` and a new `test(n=4096, h=2560)` function for testing purposes.
* Enhance pipeline barrier handling and TMA copy detection
- Updated `RewritePipelineTmaBarriers` to accept additional parameters for loop variable and stage count, improving barrier synchronization.
- Modified `HandleTileOp` to ensure only user-defined T.copy patterns are marked for pipeline barriers, preventing interference with T.tma_copy.
- Adjusted test cases to enforce specific CUDA compute version requirements for better compatibility and consistency.
* lint fix
* Enhance convolution example and TMA handling
- Added kernel source printing in `example_convolution.py` for debugging.
- Disabled the main function call and invoked `run_regression_perf()` to streamline execution.
- Introduced `MakeTmaLeaderCondition` function in `copy.cc` to improve TMA leader-thread condition handling.
- Updated `LowerBulkCopy` and `LowerBulkCopy1D` methods to utilize the new TMA leader condition for better thread management.
- Enhanced `Conv2DIm2ColOpNode` to support barrier annotations, improving synchronization in TMA operations.
* enhance
* Remove example_dequant_groupedgemm_bf16_mxfp4_hopper.py and clean up phase.py by removing unnecessary whitespace. Enhance TMA handling in producer_consumer_ws_tiled.cc to improve barrier synchronization and streamline TMA copy operations.
* fix
* fix
* Eliminate MVB(barrier_only=true) late fixup from OptimizeForTarget
Move pipeline barrier ownership into InjectSoftwarePipeline: create
pipeline_mbar[num_stages] at final expanded size instead of
pipeline_mbar[1] that required late MVB expansion.
Key changes:
- RewritePipelineTmaBarriers creates barriers at num_stages size
- Barrier indices use FloorMod(loop_var - loop_min, num_stages)
- barrier_init has num_stages entries (one per slot, arrive_count=1)
- CopyToTmaCopyRewriter accepts PrimExpr barrier_id (was int)
- phase.py: remove MVB(barrier_only=True), unify both paths under
PlanAndUpdateBufferAllocationLocation
- Fix pre-existing test expectation for tma_copies annotation
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Fix Codex review issues: num_stages=1 regression, im2col pipeline barriers, test fixes
1. Fix num_stages=1 regression: use tl_pipelined_num_stages annotation
for barrier sizing instead of max_stage+1. Gate barrier creation on
pipeline_depth > 1 so num_stages=1 kernels don't get multi-versioned
pipeline barriers.
2. Extend RewritePipelineTmaBarriers to handle c2d_im2col: annotate
im2col calls with pipeline barrier in CopyToTmaCopyRewriter. Add
im2col TMA recognition to PipelinePlanning. Fix im2col Lower() to
respect emit_arrive annotation from pipeline barriers.
3. Fix test_simple_pipeline: use annotation-checking approach instead of
structural equality with hardcoded tma_copies annotation that fails
on non-Hopper targets.
4. Remove dead kPipelineMVBStageExpr/kPipelineMVBParityExpr/
kPipelineMVBContextNumStages emission from inject_pipeline.cc since
pipeline barriers are now created at final size.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Fix depth-1 barrier ownership, add regression tests, remove dead kPipelineMVB*
1. Fix depth-1 ownership gap: always create shared pipeline barrier for
TMA copies even when pipeline_depth=1. This prevents LowerTileOp from
allocating separate per-copy internal barriers, keeping num_stages=1
kernels at pipeline_mbar[1] (single slot).
2. Add checked-in lowering regression tests:
- non-WS num_stages=3 TMA GEMM → asserts pipeline_mbar[3], no fallback
- non-WS num_stages=1 TMA GEMM → asserts pipeline_mbar[1], no multi-slot
- non-WS num_stages=3 im2col → asserts pipeline_mbar[3] feeds tma_load_im2col
3. Remove dead kPipelineMVB* definitions from pipeline_utils.h and
consumption code from multi_version_buffer_rewriter.cc (stacks,
explicit parity/version index logic, attr stripping).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Fix GemmWMMA.lower() signature and autotuner cache backwards compatibility
[P1] Add missing mbar_phase_expr parameter to GemmWMMA.lower() to match
the interface expected by GemmPy.lower() dispatch. Without this, any
RDNA kernel using the WMMA path would fail with TypeError.
[P2] Make out_idx.json loading optional in autotuner load_from_disk().
Older cache directories don't have this file; fallback to
compile_args.out_idx when the file is absent.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Add .humanize to gitignore
* Fix HIP swizzle codegen and CPU scalar GEMM region handling
[P2] Update codegen_hip.cc to parse tvm_tuple(device_func, panel_size)
format for threadblock_swizzle_pattern, matching the annotation format
now emitted by T.use_swizzle(). Previously expected StringImmNode which
would ICHECK-fail.
[P2] Fix gemm_scalar.py: clear only the output tile region (not the
whole buffer), and use the last two dimensions from regions to handle
rank>2 buffers with leading singleton dimensions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Fix HIP codegen to parse tvm_tuple swizzle annotation format
Update codegen_hip.cc threadblock_swizzle_pattern handler to parse
tvm_tuple(device_func, panel_size) format, matching CUDA and cutedsl
codegens. Previously expected StringImmNode which ICHECK-fails with
the current annotation format.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Restore kPipelineMVB* annotations — still needed by WS tiled full MVB
The kPipelineMVBStageExpr/ParityExpr/ContextNumStages annotations and
their MVB consumption code cannot be removed: they are emitted by
EmitImpl for ALL pipelines and consumed by the full MVB call inside
ProducerConsumerWarpSpecializedTiled. Removing them broke non-TMA
pipeline kernels like mhc_post that go through the WS path.
Restores: annotation emission in inject_pipeline.cc, constant
definitions in pipeline_utils.h, and consumption code in
multi_version_buffer_rewriter.cc.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Unify barrier multi-versioning in InjectSoftwarePipeline (Direction A)
Move all pipeline barrier multi-versioning into InjectSoftwarePipeline
via ExpandPipelineBarriers, eliminating the need for late
MVB(barrier_only=true) in OptimizeForTarget.
ExpandPipelineBarriers runs before BuildPipeline and handles:
- ISP-created pipeline_mbar (for non-WS TMA pipelines)
- User-written T.alloc_barrier (for manual WS pipelines like softpipe)
Key design: only barriers with explicit ptx_arrive_barrier calls OR
ISP-created local barriers are expanded. Barriers whose arrival is
managed internally by tile-ops (e.g., tcgen05 MMA arrive) are left
unchanged. This distinguishes pipeline sync barriers from hardware-
managed barriers.
Buffer expansion: barrier[N] -> barrier[N * num_stages]
Index rewriting: barrier[idx] -> barrier[stage_expr * N + idx]
Parity rewriting: user_parity -> (iteration_block + offset) % 2
barrier_init replication: [c0,c1] -> [c0,c1,c0,c1] for num_stages=2
Expanded buffers propagate to outer blocks via pending_buffer_remap_.
barrier_init annotations are replicated in VisitStmt_(BlockNode).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Fix Python 3.9 compat: replace star expression in index with tuple concat
* fix
* fix
* revert changes
* fix
* enhance pipeline
* enhance pipeline
* implement descriptor reuse pass
* Refactor pipeline management and enhance async copy handling
- Updated `example_mhc_pre.py` to replace the main function call with a test function for better testing flexibility.
- Modified `example_gqa_decode.py` to disable argument parsing and added latency measurement for regression performance.
- Enhanced `copy.cc` and `copy.h` with new checks for pipeline-managed cp.async synchronization and improved async copy handling.
- Updated `inject_pipeline.cc` and `pipeline_planning.cc` to refine the handling of tile operations and global/shared buffer checks.
- Added tests to ensure correct behavior of async pipeline and descriptor allocation reuse.
This commit improves the overall pipeline management and async copy capabilities, ensuring better performance and flexibility in the codebase.
* enhance pipeline
* enhance pipeline
* enhance pipeline
* enhance pipeline
* enhance pipeline
* Revert "enhance pipeline"
This reverts commit
|
||
|
Lei WangandZhiwen Mo
|
cabdb86f25 |
[codex] Fuse packed x2 mul-add into fma2 in CUDA codegen (#2017)
* fuse packed x2 mul-add into fma2 * document packed x2 fma2 fusion rationale --------- Co-authored-by: Zhiwen Mo <zm125@ic.ac.uk> |
||
|
Lei Wang
|
4f75940612 |
[BugFix] Enhance CUDA vectorization for binary operations (#2015)
Enhance CUDA vectorization for binary operations by supporting wider tensor dimensions. Update kernel generation to accommodate variable width, improving auto-vectorization for float32 types. Add tests for width-8 scenarios to ensure correct emission of packed operations. |
||
|
|
6fc3afa2a3 |
[BugFix] Fix auto vectorization for binary operations after wider copy instructions (#1986)
* fix f32x2 vectorize for wider shape * clean code * fix: use string::operator+= to satisfy clang-tidy performance check Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Freebase6912 <amid-gauze-racing@duck.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Chenhao Xu
|
c31ff58a5f |
Unified packed x2 intrinsics with multi-dtype support and bug fixes (#1978)
* fix: bits product to have lanes included * fix: fix T.view to view non vectorized type to vectorized type with same total bytes * feat: bfloat16 vec type along with all vectorized ops * fix: fix tests |
||
|
Lei WangandZhiwen Mo
|
6d10d1602c |
[CUDA][Feature] Add packed FP32x2 math intrinsics and auto vectorized support (#1839)
* [Feature] Add packed FP32x2 math intrinsics and support for CUDA/HIP This commit introduces new packed FP32x2 math operations: fadd2, fmul2, and fma2, which leverage PTX instructions on supported architectures. The changes include: - Definitions and implementations of fadd2, fmul2, and fma2 in both CUDA and HIP codegen files. - New Python API functions for these operations, ensuring they validate input types and handle fallbacks for unsupported architectures. - Documentation updates to reflect the new intrinsics in the math language module. These enhancements improve performance for vectorized floating-point operations in TileLang. * revert hip changes * annotate cuda only --------- Co-authored-by: Zhiwen Mo <zm125@ic.ac.uk> |