* upgrade cutlass to upstream v3.8.0
* Implement fp8 gemm and add example script
* Fix dtype retrieval with map_torch_type for fp8 inputs
* Disable vectorization of fp8 values
* Make MMA declaration compatible with cutlass 3.4.0+
* Add test for fp8 T.gemm
* fix indent
* fix indent
* Add copyright and license header
* Add copyright and license header
* lint fix
* Refactor matmul_nt and assert_matmul_correctness functions for improved readability by consolidating parameter definitions and adjusting formatting.
* clang format lint
---------
Co-authored-by: Lei Wang <34334180+LeiWang1999@users.noreply.github.com>
Co-authored-by: LeiWang1999 <leiwang1999@outlook.com>
* Add kernel caching mechanism to TileLang
- Implement a new `cached` function in `tilelang/cache/__init__.py` to cache and reuse compiled kernels
- Expose the `cached` function in the main `tilelang/__init__.py`
- Add a test case for cached matrix multiplication in `testing/python/cache/test_tilelang_cache_matmul.py`
- Provide a `clear_cache()` function to reset the kernel cache when needed
* Refactor kernel caching test and implementation
- Simplify the `cached` function in `tilelang/cache/__init__.py`
- Update test script `test_tilelang_cache_matmul.py` to use `tilelang.testing.main()`
- Remove unnecessary whitespace and improve code formatting
* Update import for `cached` function in MHA examples
- Modify import statement in `example_mha_bwd.py` and `test_tilelang_kernel_mha_bwd.py`
- Change import from `tilelang.profiler import cached` to `tilelang import cached`
- Align with recent refactoring of kernel caching mechanism
* Refactor `cached` function signature in kernel caching
- Update function signature to use keyword-only arguments for `target` and `target_host`
- Improve parameter order and readability of the `cached` decorator
- Maintain existing functionality while enhancing function definition
* [Refactor] Update BitBLAS Benchmark with TileLang Carver Imports and Roller Hints Generation
- Replace BitBLAS imports with TileLang Carver imports in benchmark_matmul.py
- Modify roller hints generation using new TileLang Carver template and utility functions
- Update get_roller_hints_from_func to handle None cases and improve return logic
- Adjust DefaultPolicy to handle different codegen dictionary formats
* [Refactor] Update Thread Binding and Import Statements in TileLang Kernels
- Replace T.thread_binding() with T.get_thread_binding() across multiple kernel test files
- Update import statements for MMA layout and macro generator in dequantize GEMM and FP8 examples
- Move map_torch_type utility function to tilelang.utils.tensor
- Remove unnecessary imports and improve code organization
* Refactor Native Sparse Attention Example with Enhanced Triton Kernel
- Update parallel_nsa_fwd_kernel to support more flexible sparse attention computation
- Add support for block counts and offsets in the Triton kernel
- Modify kernel grid and computation logic for improved performance
- Update example script to use naive_nsa_simple reference implementation
- Improve type hints and kernel configuration
* Add Native Sparse Attention Examples with Tilelang and Triton Implementations
- Introduce new example scripts for native sparse attention:
* example_tilelang_nsa_fwd.py: Forward pass implementation using TileLang
* example_tilelang_nsa_decode.py: Decoding-specific sparse attention implementation
* example_triton_nsa_fwd.py: Triton-based sparse attention forward pass
- Update reference.py with naive implementations for sparse attention
- Support different sparse attention scenarios including forward pass and inference
- Add comprehensive testing and validation against reference implementations
* lint fix
* Add Variable-Length Native Sparse Attention Examples for TileLang and Triton
- Introduce new example scripts for variable-length native sparse attention:
* example_tilelang_nsa_fwd_varlen.py: TileLang implementation with variable sequence lengths
* example_triton_nsa_fwd_varlen.py: Triton implementation with variable sequence lengths
- Update reference.py to support variable-length sparse attention scenarios
- Enhance existing sparse attention implementations to handle variable-length inputs
- Add comprehensive testing and validation for variable-length sparse attention
* Refactor Native Sparse Attention Examples: Code Style and Formatting Improvements
- Standardize function and parameter formatting across NSA example files
- Improve code readability by adjusting indentation and line breaks
- Enhance type hints and parameter alignment
- Remove unnecessary whitespaces and optimize imports
- Maintain consistent code style across TileLang and Triton implementations
* [Refactor] Update BitBLAS Benchmark with TileLang Carver Imports and Roller Hints Generation
- Replace BitBLAS imports with TileLang Carver imports in benchmark_matmul.py
- Modify roller hints generation using new TileLang Carver template and utility functions
- Update get_roller_hints_from_func to handle None cases and improve return logic
- Adjust DefaultPolicy to handle different codegen dictionary formats
* [Refactor] Update Thread Binding and Import Statements in TileLang Kernels
- Replace T.thread_binding() with T.get_thread_binding() across multiple kernel test files
- Update import statements for MMA layout and macro generator in dequantize GEMM and FP8 examples
- Move map_torch_type utility function to tilelang.utils.tensor
- Remove unnecessary imports and improve code organization
* Fix debug print buffer template for unsigned char type
- Update debug_print_buffer_value template specialization for unsigned char
- Modify test_tilelang_debug_print.py to include additional dtype tests
- Add test case for uint8 dtype in debug print buffer function
* Refactor debug print buffer template formatting for unsigned char
- Improve code formatting for debug_print_buffer_value template specialization
- Adjust line breaks and indentation for better readability
- Maintain consistent code style with other template specializations
* Extract map_torch_type utility function to tilelang.utils.tensor
- Move map_torch_type function from multiple test files to a centralized location
- Import map_torch_type from tilelang.utils.tensor in kernel test files
- Improve code reusability by creating a shared utility function for type mapping
* Add buffer dtype mapping for Cython kernel adapter
- Introduce buffer_dtype_map in CythonKernelAdapter to track buffer variable dtypes
- Add _process_buffer_dtype method to extract dtype information from TIR function
- Update CythonKernelWrapper to support setting and validating buffer dtypes
- Enhance type checking during kernel execution with dtype verification
- Improve logging message for Cython JIT adapter compilation
* Add static shape mapping for Cython kernel adapter
- Introduce static_shape_map in CythonKernelAdapter to track buffer variable static shapes
- Add _process_static_shape method to extract static shape information from TIR function
- Update CythonKernelWrapper to support setting and validating static shapes
- Enhance type checking during kernel execution with static shape verification
* Add Multi-Head Attention (MHA) Backward Pass Test for TileLang Kernel
- Implement comprehensive test for Multi-Head Attention backward pass
- Support both causal and non-causal attention scenarios
- Add reference implementation for comparing kernel outputs
- Test different batch sizes, head counts, sequence lengths, and head dimensions
- Verify forward and backward pass correctness using torch.testing.assert_close
* Set random seed for MHA backward pass test
- Add random seed initialization for consistent test reproducibility
- Use tilelang.testing.set_random_seed(42) to ensure deterministic test results
* Change default log level from WARNING to INFO in TileLang initialization
* Refactor Flash Attention Variable-Length MHA Example with Cython Backend Support
- Update `example_mha_fwd_varlen.py` to use Cython backend for kernel compilation
- Remove unused imports and simplify function signature
- Modify `flashattn` function to handle max sequence length as a separate argument
- Update kernel call to include max sequence length parameter
- Improve code readability and remove commented-out code
- Add print statement to confirm successful assertion
* Refactor code formatting in TileLang lowering and example files
- Improve line breaks and code formatting in `lower.py`, `wrapper.py`, and `tensor.py`
- Simplify line breaks and reduce unnecessary whitespace
- Enhance code readability by adjusting indentation and line breaks
- Update example MHA forward pass script with cleaner tensor initialization
* Update TileLang kernel test with import path changes for MMA layout and macro generator
- Modify import statements in test_tilelang_kernel_dequantize_gemm.py
- Replace bitblas imports with tilelang.intrinsics imports for MMA-related utilities
- Update main function to use tilelang.testing.main()
* Add Block Sparse Attention Examples for TileLang and Triton
- Implement block sparse attention kernels for both TileLang and Triton
- Add utility functions for generating sparse attention masks using top-k and threshold methods
- Support causal and variable-length attention scenarios
- Include test cases for different sequence length configurations
- Demonstrate block-level sparse attention with configurable parameters
* Refactor Block Sparse Attention Examples with Code Style Improvements
- Improve code formatting in block_sparse_attn_tilelang.py and block_sparse_attn_triton.py
- Enhance readability by adjusting line breaks and indentation
- Simplify kernel and function calls with better formatting
- Add whitespace and line break improvements for better code clarity
* Enhance Layout Plotting with Multi-Replication and Dynamic Visualization
- Update plot_layout function to support multiple replications in thread and value mapping
- Improve thread and value mapping to handle replicated layouts
- Dynamically adjust figure size and legend positioning
- Add print statements for saved plot file paths
- Modify example fragment_mma_load_a.py to uncomment and enable warp and block layout plotting
* Refactor AtomicAdd functions in CUDA common header
- Implement a generic template for AtomicAdd function
- Specialize templates for half_t, bfloat16_t, and pointer types
- Reorganize and clean up existing AtomicAdd implementations
- Improve type handling and conversion in atomic operations
* Remove unused import in MHA backward test file
- Remove unnecessary argparse import from test_tilelang_kenrel_mha_bwd.py
- Add blank line for improved code formatting
- Minor code cleanup in test file
* Change default log level from WARNING to INFO in TileLang initialization
* Refactor Flash Attention Variable-Length MHA Example with Cython Backend Support
- Update `example_mha_fwd_varlen.py` to use Cython backend for kernel compilation
- Remove unused imports and simplify function signature
- Modify `flashattn` function to handle max sequence length as a separate argument
- Update kernel call to include max sequence length parameter
- Improve code readability and remove commented-out code
- Add print statement to confirm successful assertion
* Refactor code formatting in TileLang lowering and example files
- Improve line breaks and code formatting in `lower.py`, `wrapper.py`, and `tensor.py`
- Simplify line breaks and reduce unnecessary whitespace
- Enhance code readability by adjusting indentation and line breaks
- Update example MHA forward pass script with cleaner tensor initialization
* Update TileLang kernel test with import path changes for MMA layout and macro generator
- Modify import statements in test_tilelang_kernel_dequantize_gemm.py
- Replace bitblas imports with tilelang.intrinsics imports for MMA-related utilities
- Update main function to use tilelang.testing.main()
* Add Block Sparse Attention Examples for TileLang and Triton
- Implement block sparse attention kernels for both TileLang and Triton
- Add utility functions for generating sparse attention masks using top-k and threshold methods
- Support causal and variable-length attention scenarios
- Include test cases for different sequence length configurations
- Demonstrate block-level sparse attention with configurable parameters
* Refactor Block Sparse Attention Examples with Code Style Improvements
- Improve code formatting in block_sparse_attn_tilelang.py and block_sparse_attn_triton.py
- Enhance readability by adjusting line breaks and indentation
- Simplify kernel and function calls with better formatting
- Add whitespace and line break improvements for better code clarity
* refactor code
* enhance tutorial
* Enhance error handling and code generation in CUDA and TileLang components
This commit introduces several improvements across multiple files:
- Added more informative error messages in GEMM layout checks
- Updated CUDA codegen to support more flexible function signature generation
- Improved TMA descriptor initialization and kernel dispatch logic
- Refined library generation and source code parsing utilities
- Enhanced error handling in various adapter and wrapper classes
* Add thread tag validation for warp specialization
Introduce a ThreadTagChecker to validate that a PrimFunc only uses threadIdx.x before applying warp specialization. This prevents unintended transformations on kernels with complex thread binding and provides a clear warning to users about potential issues with warp specialization.
* Update TileLang Profiling and Compilation in Flash Decoding Examples
Refactor the profiling and compilation workflow in two flash decoding example scripts:
- Replace `tilelang.lower()` and `tilelang.Profiler()` with `tilelang.compile()`
- Simplify profiler initialization using `get_profiler()`
- Update method calls to use the new profiler and compiled kernel objects
- Maintain existing performance benchmarking and validation logic
* Refactor and clean up code formatting in TileLang testing and adapter modules
This commit includes several code style and formatting improvements:
- Adjust whitespace and line breaks in test files
- Improve code formatting in CUDA source wrapper and adapter utilities
- Enhance readability of function calls and argument handling
- Remove unnecessary whitespace and standardize indentation
- Simplify function signatures and argument parsing
* Refactor CUDA codegen and improve code formatting
This commit includes several improvements to CUDA code generation and formatting:
- Enhance function signature generation in CodeGenTileLangCUDA
- Improve code formatting and readability in CUDA-related files
- Simplify parameter handling and type annotations
- Clean up whitespace and line breaks in codegen and layout files
---------
Co-authored-by: Ubuntu <dlisuser@h100testl730RPS.xu5snccwrbtejcqqalluoku5hb.xx.internal.cloudapp.net>
* [Feature] Add CTypes JIT kernel support for dynamic shapes and multi-stream execution
- Enhance CtypesKernelAdapter to handle dynamic symbolic shapes
- Add support for multi-stream kernel execution in CTypes backend
- Implement dynamic shape handling in test_tilelang_jit_gemm_ctypes.py
- Add symbolic shape utility function in tilelang.language
- Update profiler to improve flexibility in benchmark selection
* Remove redundant thread binding in GEMM kernel implementations
- Remove unnecessary `thread_binding` line in GEMM kernel functions
- Clean up code in `examples/gemm/README.md` and `testing/python/kernel/test_tilelang_kernel_int4_gemm_mma.py`
- Enhance code readability by removing redundant thread binding annotation
* Fix indentation in int4 GEMM kernel test file
- Correct indentation for function calls in `test_tilelang_kernel_int4_gemm_mma.py`
- Remove extra indentation in `mma_emitter.ldmatrix_a()` and `mma_emitter.ldmatrix_b()` calls
- Improve code formatting for better readability
* [CI][Test] Add test cases for tilelang transform MultiVersionBuffer and WarpSpecialized
* Relax the mismatch ratio restrictions in the flash_linear_attention and mha tests
* [Enhancement] Add VectorizeLoop function and update imports for compatibility
* [CI][Test] Improve test cases for vectorization and fix typos in parser comments
* lint fix
* Fix incorrect module reference for VectorizeLoop transformation
* Refactor vectorize_loop transformation by removing unused extent mutation logic
* [Enhancement] Add support for FP8 data types and global barriers in CUDA codegen
* Fix formatting in CUDA FP8 header file for consistency
* Refactor CI workflow to use 'tilelang_ci' virtual environment and update CUDA type printing for better clarity
* Update submodule 'tvm' to latest commit for improved functionality
* Refactor execution backend references from 'dl_pack' to 'dlpack' for consistency and clarity; add apply_simplify function to simplify PrimFunc or IRModule.
* Refactor CUDA code for improved readability; clean up formatting and remove unnecessary whitespace in multiple files.
* Refactor import statement in test_tilelang_kernel_dequantize_gemm.py to use 'tilelang.language' for consistency
* Add CUDA requirements to FP8 test cases and update references for clarity
* Add a blank line for improved readability in test_tilelang_kernel_fp8_gemm_mma.py
* Fix data type in reference result calculation for consistency in test_tilelang_kernel_gemm_mma_intrinsic.py
* Add CUDA requirements and FP8 test cases for matmul and gemv simulations
* Remove debug print statements and use tilelang's testing assertion for result validation in test_tilelang_kernel_gemm_mma_intrinsic.py
* Remove outdated comment regarding FP8 tests in test_tilelang_kernel_gemv_simt.py
* Add BF16 support to matrix multiplication and introduce corresponding test cases
* Add a blank line for improved readability in BF16 GEMM test
* Update acknowledgements in README to include supervision by Zhi Yang at Peking University
* [Enhancement] Add VectorizeLoop function and update imports for compatibility
* [CI][Test] Improve test cases for vectorization and fix typos in parser comments
* lint fix
* Fix incorrect module reference for VectorizeLoop transformation
* Refactor vectorize_loop transformation by removing unused extent mutation logic
* [Enhancement] Add support for FP8 data types and global barriers in CUDA codegen
* Fix formatting in CUDA FP8 header file for consistency
* Refactor CI workflow to use 'tilelang_ci' virtual environment and update CUDA type printing for better clarity
* Update submodule 'tvm' to latest commit for improved functionality
* Refactor execution backend references from 'dl_pack' to 'dlpack' for consistency and clarity; add apply_simplify function to simplify PrimFunc or IRModule.
* Refactor CUDA code for improved readability; clean up formatting and remove unnecessary whitespace in multiple files.
* Refactor import statement in test_tilelang_kernel_dequantize_gemm.py to use 'tilelang.language' for consistency
* Add CUDA requirements to FP8 test cases and update references for clarity
* Add a blank line for improved readability in test_tilelang_kernel_fp8_gemm_mma.py
* Fix data type in reference result calculation for consistency in test_tilelang_kernel_gemm_mma_intrinsic.py
* Add CUDA requirements and FP8 test cases for matmul and gemv simulations
* Remove debug print statements and use tilelang's testing assertion for result validation in test_tilelang_kernel_gemm_mma_intrinsic.py
* Remove outdated comment regarding FP8 tests in test_tilelang_kernel_gemv_simt.py
* implement jit test case
* [Dev] implement auto tune test case for matrix multiplication
* Implement test for legalize memory access and vectorized loop
* lint fix
* introduce run_once
* Refactor callback function names for consistency and improve code readability
* enhance documentations
* lint fix
* lint fix
* lint fix
* lint fix
* fix formatting issues in rt_mod_hip.cc
* add random seed initialization for deterministic testing
* [Dev] Add FlashDecoding example
* [CI][Test] Add test cases for tilelang kernel convolution
* [CI][Test] Add test cases for tilelang kernel FlashAttention
* Reduce the number of stages to ensure the shared memory allocation is valid
* Temporarily remove the dim128 case
* lint
* update einops in requirements-dev.txt
* update einops in requirements-test.txt
* remove einops in requirements-dev.txt
