* remove llvm build
* [Refactor] Update kernel compilation and profiling in examples
- Replaced `tilelang.lower` with `tilelang.compile` in multiple example scripts to streamline kernel compilation.
- Updated profiling calls to utilize the new `get_profiler` method, enhancing performance measurement consistency.
- Adjusted assertions and benchmarking methods to align with the new profiling structure across various examples, ensuring correctness and clarity in performance evaluations.
* lint fix
* License Update
* [Refactor] Improve code formatting and documentation in CUDA header and HIP runtime files
- Adjusted formatting in `cuda.h` for better readability, including alignment of comments and struct fields.
- Cleaned up whitespace and improved comment clarity in `rt_mod_hip.cc` to enhance code maintainability.
* [Refactor] Enhance formatting and clarity in CUDA header and HIP runtime files
- Improved comment alignment and readability in `cuda.h`.
- Cleaned up whitespace and formatting in `rt_mod_hip.cc` to enhance maintainability.
* lint fix
* lint fix
* lint fix
* lint fix
* fix
* License update
* [Enhancement] Update JITKernel to use artifact for kernel source
- Assigned the generated artifact to `self.artifact` for better management.
- Updated kernel source references to use `artifact.kernel_source` for consistency in execution backend handling.
* lint fix
* Add @tilelang.testing.requires_llvm decorator to vectorization tests
* Enhance setup.py and env.py for library management
- Added functionality to remove original files after copying in CMakeBuild.
- Updated TVM_LIBRARY_PATH in env.py to include the PyPI build library path for better integration.
* Refactor TVM_LIBRARY_PATH assignment for improved readability in env.py
* Refactor CMakeBuild file handling in setup.py
- Added a check to ensure the target library directory exists before copying .so files.
- Improved the logic for creating the target directory and copying files to enhance robustness.
* bugfix
* Rename BuildTLDebug to BuildTileLangCUDAWithoutCompile and update registration. Add @tilelang.testing.requires_llvm decorator to multiple tests for LLVM requirement.
* lint fix
* Enhance TileLang code generation by adding support for device code generation without compilation. Updated `host_codegen` and `device_codegen` functions to include new transformations and registration for `tilelang_hip_without_compile`. Refactored JIT kernel adapters to accommodate host and device modules, improving overall integration and flexibility.
* lint fix
* Add support for C target in device code generation
- Updated `device_codegen_without_compile` to include handling for the C target by registering the `tilelang_cpp` function.
* [Enhancement] Implement auto-clear cache feature based on environment variable
* Added TILELANG_CLEAR_CACHE environment variable to control cache clearing.
* Updated CI workflow to set TILELANG_CLEAR_CACHE during testing.
* Modified cache initialization to clear cache if TILELANG_CLEAR_CACHE is set to true.
* [Refactor] Update kernel invocation and import paths in tests and cache
* Changed kernel invocation in `test_tilelang_kernel_dequantize_gemm.py` to return the result.
* Updated import statements in `test_tilelang_kernel_int4_gemm_mma.py` to use `bitblas` instead of `tilelang`.
* Refactored paths for artifact and parameters in `kernel_cache.py` for better maintainability.
* [Refactor] Clean up whitespace and improve code formatting in kernel_cache.py
* Removed unnecessary blank lines and adjusted spacing for better readability in the KernelCache class.
* Enhanced overall code formatting to align with project standards.
* [Enhancement] Add bfloat16 test case and improve kernel caching logic
* Introduced a new test case for bfloat16 matrix multiplication in `test_tilelang_kernel_gemm_mma_intrinsic.py`.
* Updated `KernelCache` to handle multiple kernel source files and improve error handling during saving and loading.
* Refactored `JITKernel` to support instantiation from a database, enhancing flexibility in kernel management.
* Adjusted `CtypesKernelAdapter` and `CythonKernelAdapter` to utilize the new kernel loading mechanism from the database.
* Improved code formatting and readability across several files.
* lint fix
* Update bfloat16 matrix multiplication test case to use larger dimensions for improved coverage
* [Refactor] Update KernelParam integration across modules
- Replaced instances of TensorType with KernelParam in various modules to standardize parameter handling.
- Updated JITKernel, BaseKernelAdapter, and CythonKernelAdapter to utilize KernelParam for improved type consistency.
- Enhanced Profiler class to include KernelParam in its parameters, ensuring better integration with the new parameter structure.
- Adjusted tensor handling in utility functions to accommodate the new KernelParam type, improving overall code clarity and maintainability.
- Updated copyright headers to reflect the correct organization.
* [Refactor] Clean up whitespace in kernel, profiler, and tensor modules
- Added blank lines for improved readability in kernel.py, __init__.py, and tensor.py.
- Enhanced code clarity by ensuring consistent formatting across these modules.
* [Enhancement] Add detailed docstrings to KernelParam and Profiler classes
- Enhanced KernelParam class with comprehensive docstrings for better understanding of its purpose and methods.
- Updated Profiler class to include detailed docstrings for its attributes and methods, improving code documentation and usability.
- Removed unused do_bench function to streamline the profiler module and improve clarity.
* [Refactor] Update type hints in do_bench function and clean up whitespace in profiler module
- Changed type hints for grad_to_none and quantiles parameters in do_bench function to use Optional for better clarity.
- Added a blank line in __init__.py for improved readability and consistency in the profiler module.
* [Refactor] Update type hint in do_bench function for consistency
- Changed the return type hint in the do_bench function from a union type to a more explicit List type for better clarity and consistency in type annotations.
* [Refactor] Update return type hint in do_bench function for clarity
- Changed the return type hint in the do_bench function from a union type to Union[float, List[float]] for improved clarity and consistency in type annotations.
* [Enhancement] Add func property to Profiler class for adapter access
- Introduced a new property `func` in the Profiler class to provide access to the adapter, ensuring that the adapter is set before retrieval. This enhancement improves the usability of the Profiler class by allowing easier access to the adapter functionality.
* [Refactor] Update kernel compilation and profiling in tests
- Replaced instances of `TL.lower` and `TL.Profiler` with `tilelang.compile` and the new profiler interface across multiple test files.
- Enhanced the kernel compilation process to utilize the updated API, improving consistency and maintainability in the testing framework.
- Updated assertions to use the new profiler methods for better clarity and functionality in performance testing.
* [Refactor] Simplify kernel invocation and remove unused parameters in tests
- Updated the kernel invocation in `test_tilelang_dynamic_symbolic.py` to directly assign the result to `C`, improving clarity.
- Removed the `execution_backend` parameter from `tilelang.compile` calls in `test_tilelang_jit_callback.py` and `test_tilelang_jit_gemm.py` for consistency with the updated API.
- Commented out the call to `tilelang.testing.main()` in `test_tilelang_jit_callback.py` and replaced it with a direct call to `test_gemm_jit_kernel()` to streamline test execution.
- Adjusted the dtype mapping in `TorchDLPackKernelAdapter` to use the parameter's dtype directly, enhancing code simplicity.
* [Refactor] Remove unused imports in test files for cleaner code
- Eliminated unnecessary imports of `tilelang` as `TL` in various test files to enhance code clarity and maintainability.
- Updated multiple test files to streamline the codebase and reduce potential confusion from unused references.
* [Refactor] Simplify kernel invocation in tilelang kernel test
- Updated the kernel invocation in `test_tilelang_kernel_bf16_gemm_mma.py` to directly assign the result to `C`, enhancing code clarity and consistency with recent changes in the API.
* [Refactor] Simplify kernel invocation in tilelang kernel tests
- Updated kernel invocations in multiple test files to directly assign the result to `C`, improving code clarity and consistency with the updated API.
- Removed unnecessary initialization of `C` as a zero tensor, streamlining the code further.
* [Refactor] Update kernel invocation in tilelang transform tests
- Replaced the use of `TL.Profiler` with `tilelang.compile` in `test_tilelang_transform_simplify.py`, enhancing code clarity and consistency with the updated API.
- Streamlined the kernel invocation process by directly assigning the result to `C`, improving readability and maintainability of the test code.
- Introduce TMAStoreArrive and TMAStoreWait operations for CUDA TMA store synchronization
- Add new builtin operations in op/builtin.cc and op/builtin.h
- Implement TMAStoreSyncInjector to automatically inject TMA store synchronization calls
- Update CUDA codegen to support new TMA store synchronization intrinsics
- Add Python language bindings for new TMA store synchronization operations
* [Refactor] Add SetMaxNRegCollector to Improve Register Hint Handling in Warp Specialized Rewriter
- Introduce `SetMaxNRegCollector` to collect register hints from SetMaxNReg calls
- Modify `WarpSpecializedRewriter` to use collected register hints for producer and consumer code
- Add validation checks for register hint values in the collector
- Remove SetMaxNReg calls during code transformation
- Enhance flexibility of register allocation in warp specialized rewriting
* temporary remove check in lower_hopper_intrin
* [Dev] Add RetNet Linear Attention example
* [Dev] Add WgmmaSync rewriter for pipelined WGMMA operations and add MHA WGMMA pipelined example (FA3-like scheduling)
This commit introduces a new transformation pass `RewriteWgmmaSync` to optimize warp group matrix multiply accumulate (WGMMA) operations in the TileLang compiler:
- Implemented `WgmmaSyncRewriter` in `src/transform/wgmma_sync_rewriter.cc`
- Added pass registration for `RewriteWgmmaSync`
- Updated `tilelang/engine/phase.py` to include the new transformation pass
- Updated `tilelang/transform/__init__.py` to expose the new pass
The rewriter intelligently manages synchronization and dependencies between WGMMA operations, improving pipeline efficiency for complex matrix multiplication kernels.
* [Bugfix] Fix bug in ThreadTagChecker for warp specialization
Improve thread tag validation in warp specialized rewriter to prevent unintended transformations:
- Add more precise checks for threadIdx.y and threadIdx.z
- Validate thread extent to ensure only single-extent thread bindings are allowed
- Prevent warp specialization for multi-extent thread bindings in y and z dimensions
* lint
* [CI] Add TMA descriptor attribute to transformed module in test case
* bump version into v0.1.0
* [Enhancement] Add custom develop command for editable installs and update .gitignore
* [Documentation] Update README to include system dependencies installation instructions
* [Build] Update setup.py to support library file copying for both release and develop modes
* [Build] Refactor library file copying logic in setup.py
* [Documentation] Remove unnecessary install section header in Installation.md
* [Build] Add tox configuration and local distribution script for multi-Python version support
* [Build] Improve git submodule update function with better error handling
* [Build] Update LLVM configuration path in ROCm installation script
* [Build] Add .tox/ to .gitignore for tox testing environment
* [Build] Add support for TVM prebuild path configuration in CMakeLists.txt
* [Cleanup] Remove unused TVM runtime error codes header
* [Cleanup] Fix TVM grid constant type reference in CUDA module
* [Cleanup] Remove unused customized_code function from IR module
* [Feature] Add TileLang thread synchronization and storage access analysis passes
* [Build] Reorder DLL search path directories for more flexible library loading
* [Refactor] Improve thread synchronization and library path handling
- Rename ThreadSync and TileLangThreadSync functions in C++ code
- Update Python docstring for ThreadSync with more detailed description
- Reorder library path detection in tilelang environment setup
- Minor comment and code cleanup in CUDA and warp specialization modules
* [Refactor] Improve thread synchronization code style and formatting
- Standardize pointer type spacing in storage_access.h and storage_access.cc
- Update whitespace and indentation in thread_storage_sync.cc
- Reorder include statements in thread_partial_sync.cc
- Minor code formatting improvements across thread synchronization files
* [Refactor] Fix global function registration for ThreadSync
- Correct global function registration to use ThreadSync instead of TileLangThreadSync
- Update TVM global registration to match recent refactoring efforts
* [Refactor] Simplify ThreadSync global function registration
- Remove unnecessary whitespace in global function registration
- Compact the TVM global registration line for ThreadSync
* [CI][Test] Add test cases for tilelang transform MultiVersionBuffer and WarpSpecialized
* Relax the mismatch ratio restrictions in the flash_linear_attention and mha tests
* [Dev] Add FlashDecoding example
* [CI][Test] Add test cases for tilelang kernel convolution
* [CI][Test] Add test cases for tilelang kernel FlashAttention
* Reduce the number of stages to ensure the shared memory allocation is valid
* Temporarily remove the dim128 case
* lint
* update einops in requirements-dev.txt
* update einops in requirements-test.txt
* remove einops in requirements-dev.txt
* [CI][Test] Add test cases for tilelang transform ClusterPlanning
* [CI][Test] Add test cases for tilelang transform LowerHopperIntrin
* [CI][Test] Add test cases for tilelang transform InjectFenceProxy
* [Enhancement] Add VectorizeLoop function and update imports for compatibility
* [CI][Test] Improve test cases for vectorization and fix typos in parser comments
* lint fix
* Fix incorrect module reference for VectorizeLoop transformation
* Refactor vectorize_loop transformation by removing unused extent mutation logic
* [Dev] Add FlashDecoding example
* [CI][Test] Add test cases for tilelang kernel convolution
* [CI][Test] Add test cases for tilelang kernel FlashAttention
* Reduce the number of stages to ensure the shared memory allocation is valid
* Temporarily remove the dim128 case
* lint
* update einops in requirements-dev.txt
* update einops in requirements-test.txt
* remove einops in requirements-dev.txt
* [CI][Test] Add test cases for tilelang transform ClusterPlanning
* [CI][Test] Add test cases for tilelang transform LowerHopperIntrin
* [Dev] Add FlashDecoding example
* [CI][Test] Add test cases for tilelang kernel convolution
* [CI][Test] Add test cases for tilelang kernel FlashAttention
* Reduce the number of stages to ensure the shared memory allocation is valid
* Temporarily remove the dim128 case
* lint
* update einops in requirements-dev.txt
* update einops in requirements-test.txt
* remove einops in requirements-dev.txt
* [CI][Test] Add test cases for tilelang transform ClusterPlanning
* implement jit test case
* [Dev] implement auto tune test case for matrix multiplication
* Implement test for legalize memory access and vectorized loop
* lint fix
Lei Wang