feat: add moss-image support (#2148 )

chore: sync ggml and remove obsolete conv1d workarounds (#2147 )
fix: handle UTF-8 file paths on Windows (#2145 )
2026-10-17 05:39:54 +08:00 · 2026-10-17 03:38:38 +08:00 · 2026-10-17 02:39:51 +08:00 · 2026-10-17 00:50:42 +08:00 · 2026-10-17 00:50:09 +08:00 · 2026-10-16 02:42:07 +08:00
42 changed files with 1074 additions and 159 deletions
@@ -15,6 +15,7 @@ API and command-line option may change frequently.***

 ## 🔥Important News

+* **2026/10/16** 🚀 stable-diffusion.cpp adds **Day-0 support for FLUX.3 Image Klein / Klein Base**
 * **2026/09/20** 🚀 stable-diffusion.cpp adds **Day-0 support for Qwen-Image-2.1**
 * **2026/08/20** 🚀 stable-diffusion.cpp now supports **LTX-2.5**
 * **2026/08/04** 🚀 stable-diffusion.cpp adds **Day-1 support for MiniMax-H3**
@@ -44,6 +45,8 @@ API and command-line option may change frequently.***
    - [SD3/SD3.5](./docs/sd3.md)
    - [FLUX.1-dev/FLUX.1-schnell](./docs/flux.md)
    - [FLUX.2-dev/FLUX.2-klein](./docs/flux2.md)
+    - [FLUX.3 Image Klein / Klein Base](./docs/flux3.md)
+    - [MOSS-Image-Base / MOSS-Image-SFT](./docs/moss_image.md)
    - [Lens](./docs/lens.md)
    - [Chroma](./docs/chroma.md)
    - [Chroma1-Radiance](./docs/chroma_radiance.md)
@@ -14,6 +14,7 @@ Depending on the architecture, different models handle reference images differen
 | [**Qwen Image Edit**](./qwen_image_edit.md) | `qwen` |
 | **Qwen Image LAYERED** | `qwen_layered` |
 | [**Flux.2 [Dev] / Flux.2 [Klein]**](./flux2.md) | `flux2` |
+| [**FLUX.3 Image Klein / Klein Base**](./flux3.md) | `flux2` |
 | [**Boogu Image Edit**](./boogu_image.md) | `z_image_omni` |
 | **Krea2 (Community Edit LoRAs)** | `krea2_ostris_edit` |
 | [**Mage-Flow-Edit**](./mage_flow.md#image-editing) | `mage_flow` |
@@ -0,0 +1,48 @@
+# How to Use
+
+FLUX.3 Image Klein uses a FLUX.3 diffusion transformer, the FLUX.2 VAE, and Qwen3-VL 4B as the LLM text encoder.
+
+## Download weights
+
+- Download FLUX.3 Image Klein
+    - safetensors: https://huggingface.co/black-forest-labs/flux-3-image-klein/tree/main
+- Download FLUX.3 Image Klein Base
+    - safetensors: https://huggingface.co/black-forest-labs/flux-3-image-klein-base/tree/main
+- Download vae
+    - safetensors: https://huggingface.co/black-forest-labs/FLUX.2-dev/tree/main
+    - The `ae.safetensors` included with FLUX.3 can also be used.
+- Download Qwen3-VL 4B
+    - safetensors: https://huggingface.co/Comfy-Org/Krea-2/tree/main/text_encoders
+    - gguf: https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct-GGUF/tree/main
+
+Use the root `model.safetensors` for the diffusion model.
+
+## Examples
+
+### FLUX.3 Image Klein
+
+The distilled model uses 16 steps and CFG 1.0.
+
+```
+.\bin\Release\sd-cli.exe --diffusion-model ..\models\diffusion_models\flux-3-image-klein.safetensors --llm ..\models\text_encoders\Qwen3-VL-4B-Instruct-Q4_K_M.gguf --vae ..\models\vae\flux2_ae.safetensors -p "a lovely cat holding a sign says 'flux3.cpp'" --cfg-scale 1.0 --steps 16 --sampling-method euler -v --offload-to-cpu
+```
+
+<img src="../assets/flux3/flux3-image-klein.png" />
+
+```
+.\bin\Release\sd-cli.exe --diffusion-model  ..\models\diffusion_models\flux-3-image-klein.safetensors --vae ..\models\vae\flux2_ae.safetensors --llm  ..\models\text_encoders\Qwen3-VL-4B-Instruct-Q4_K_M.gguf -r ..\assets\flux\flux1-dev-q8_0.png -p "change 'flux.cpp' to 'flux3-image-klein.cpp'" --cfg-scale 1.0 --sampling-method euler -v  --offload-to-cpu --steps 16
+```
+
+<img src="../assets/flux3/flux3-image-klein-edit.png" />
+
+Repeat `-r` to use multiple reference images, up to 10. See [Image Editing](./edit.md) for reference-image preprocessing options.
+
+### FLUX.3 Image Klein Base
+
+```
+.\bin\Release\sd-cli.exe --diffusion-model ..\models\diffusion_models\flux-3-image-klein-base.safetensors --llm ..\models\text_encoders\Qwen3-VL-4B-Instruct-Q4_K_M.gguf --vae ..\models\vae\flux2_ae.safetensors -p "a lovely cat holding a sign says 'flux3.cpp'" --cfg-scale 3.5 --steps 20 --sampling-method euler -v --offload-to-cpu
+```
+
+<img src="../assets/flux3/flux3-image-klein-base.png" />
+
+The default discrete scheduler uses the upstream time shift of 6.93 for both models. Use `--flow-shift` to override it.
@@ -0,0 +1,34 @@
+# How to Use
+
+MOSS-Image uses a 5B diffusion transformer, the FLUX.2 VAE, and Qwen3.5 9B as the LLM text encoder. Both Base and SFT are supported. Prompt enhancement is not included.
+
+## Download weights
+
+- Download MOSS-Image Base or SFT
+    - safetensors: https://huggingface.co/Comfy-Org/MOSS-Image/tree/main/diffusion_models
+- Download vae
+    - safetensors: https://huggingface.co/Comfy-Org/MOSS-Image/tree/main/vae
+- Download Qwen3.5 9B
+    - gguf: https://huggingface.co/unsloth/Qwen3.5-9B-GGUF/tree/main
+- Download `tokenizer.json`
+    - https://huggingface.co/OpenMOSS-Team/MOSS-Image-Base/tree/main/tokenizer
+
+Pass the external `tokenizer.json` with `--tokenizer`; Qwen3.5 uses a different vocabulary from the embedded Qwen tokenizer.
+
+## Examples
+
+### MOSS-Image Base
+
+```
+.\bin\Release\sd-cli.exe --diffusion-model ..\models\diffusion_models\moss_image_base_int8_convrot_attn.safetensors --llm ..\models\text_encoders\Qwen3.5-9B-Q4_K_M.gguf --vae ..\models\vae\flux2_ae.safetensors --tokenizer ..\models\tokenizers\tokenizer_qwen3.5.json --cfg-scale 3.0 -p "a lovely cat holding a sign says 'moss-image.cpp'" -v --offload-to-cpu -H 1024 -W 1024
+```
+
+### MOSS-Image SFT
+
+```
+.\bin\Release\sd-cli.exe --diffusion-model ..\models\diffusion_models\moss_image_sft_int8_convrot_attn.safetensors --llm ..\models\text_encoders\Qwen3.5-9B-Q4_K_M.gguf --vae ..\models\vae\flux2_ae.safetensors --tokenizer ..\models\tokenizers\tokenizer_qwen3.5.json --cfg-scale 3.0 -p "a lovely cat holding a sign says 'moss-image.cpp'" -v --offload-to-cpu -H 1024 -W 1024
+```
+
+<img src="../assets/moss/sft.png" />
+
+Use dimensions divisible by 16. The default discrete scheduler uses the upstream time shift of 5. Use `--flow-shift` to override it. The negative CFG branch skips DiT layer 22 (zero-based), matching the upstream pipeline.
@@ -4,6 +4,7 @@
 #include <cctype>
 #include <cstdint>
 #include <cstring>
+#include <filesystem>
 #include <fstream>
 #include <iomanip>
 #include <iostream>
@@ -276,7 +277,7 @@ namespace {
    }

    bool read_file(const std::string& path, std::vector<uint8_t>& data, std::string& error) {
-        std::ifstream fin(path, std::ios::binary);
+        std::ifstream fin(std::filesystem::u8path(path), std::ios::binary);
        if (!fin) {
            error = "failed to open file: " + path;
            return false;
@@ -188,7 +188,7 @@ struct SDCliParams {
            if (++index >= argc) {
                return -1;
            }
-            imatrix_in.push_back(argv[index]);
+            imatrix_in.push_back(argv_to_utf8(index, argv));
            return 1;
        };

@@ -323,13 +323,14 @@ void sd_log_cb(enum sd_log_level_t level, const char* log, void* data) {
 bool load_images_from_dir(const std::string dir,
                          std::vector<SDImageOwner>& images,
                          int max_image_num = 0) {
-    if (!fs::exists(dir) || !fs::is_directory(dir)) {
+    const fs::path dir_path = fs::u8path(dir);
+    if (!fs::exists(dir_path) || !fs::is_directory(dir_path)) {
        LOG_ERROR("'%s' is not a valid directory\n", dir.c_str());
        return false;
    }

    std::vector<fs::directory_entry> entries;
-    for (const auto& entry : fs::directory_iterator(dir)) {
+    for (const auto& entry : fs::directory_iterator(dir_path)) {
        if (entry.is_regular_file()) {
            entries.push_back(entry);
        }
@@ -337,12 +338,12 @@ bool load_images_from_dir(const std::string dir,

    std::sort(entries.begin(), entries.end(),
              [](const fs::directory_entry& a, const fs::directory_entry& b) {
-                  return a.path().filename().string() < b.path().filename().string();
+                  return a.path().filename().u8string() < b.path().filename().u8string();
              });

    for (const auto& entry : entries) {
-        std::string path = entry.path().string();
-        std::string ext  = entry.path().extension().string();
+        std::string path = entry.path().u8string();
+        std::string ext  = entry.path().extension().u8string();
        std::transform(ext.begin(), ext.end(), ext.begin(), ::tolower);

        if (ext == ".jpg" || ext == ".jpeg" || ext == ".png" || ext == ".bmp" || ext == ".webp") {
@@ -397,19 +398,19 @@ void step_callback(int step, int frame_count, sd_image_t* image, bool is_noisy,
    // is_noisy is set to true if the preview corresponds to noisy latents, false if it's denoised latents
    // unused in this app, it will either be always noisy or always denoised here
    if (frame_count == 1) {
-        fs::path path = cli_params->preview_path;
-        if (encoded_image_format_from_path(path.string()) == EncodedImageFormat::UNKNOWN)
+        fs::path path = fs::u8path(cli_params->preview_path);
+        if (encoded_image_format_from_path(path.u8string()) == EncodedImageFormat::UNKNOWN)
            path += ".png";
-        if (std::regex_search(path.string(), format_specifier_regex))
-            path = fs::path(format_frame_idx(path.string(), continuous_preview_counter++));
-        if (!write_image_to_file(path.string(),
+        if (std::regex_search(path.u8string(), format_specifier_regex))
+            path = fs::u8path(format_frame_idx(path.u8string(), continuous_preview_counter++));
+        if (!write_image_to_file(path.u8string(),
                                 image->data,
                                 image->width,
                                 image->height,
                                 image->channel,
                                 "",
                                 cli_params->compression_quality)) {
-            LOG_ERROR("save preview image to '%s' failed", path.string().c_str());
+            LOG_ERROR("save preview image to '%s' failed", path.u8string().c_str());
        }
    } else {
        int fps = cli_params->preview_method == PREVIEW_PROJ ? cli_params->preview_fps / 4 : cli_params->preview_fps;
@@ -420,12 +421,12 @@ void step_callback(int step, int frame_count, sd_image_t* image, bool is_noisy,
 }

 static fs::path get_video_audio_sidecar_path(const SDCliParams& cli_params) {
-    fs::path out_path     = cli_params.output_path;
+    fs::path out_path     = fs::u8path(cli_params.output_path);
    fs::path base_path    = out_path;
    fs::path ext          = out_path.has_extension() ? out_path.extension() : fs::path{};
-    std::string ext_lower = ext.string();
+    std::string ext_lower = ext.u8string();
    std::transform(ext_lower.begin(), ext_lower.end(), ext_lower.begin(), ::tolower);
-    const EncodedImageFormat output_format = encoded_image_format_from_path(out_path.string());
+    const EncodedImageFormat output_format = encoded_image_format_from_path(out_path.u8string());
    if (!ext.empty()) {
        if (output_format == EncodedImageFormat::JPEG ||
            output_format == EncodedImageFormat::PNG ||
@@ -450,14 +451,14 @@ bool save_results(const SDCliParams& cli_params,
    }

    namespace fs      = std::filesystem;
-    fs::path out_path = cli_params.output_path;
+    fs::path out_path = fs::u8path(cli_params.output_path);

    if (!out_path.parent_path().empty()) {
        std::error_code ec;
        fs::create_directories(out_path.parent_path(), ec);
        if (ec) {
            LOG_ERROR("failed to create directory '%s': %s",
-                      out_path.parent_path().string().c_str(), ec.message().c_str());
+                      out_path.parent_path().u8string().c_str(), ec.message().c_str());
            return false;
        }
    }
@@ -465,9 +466,9 @@ bool save_results(const SDCliParams& cli_params,
    fs::path base_path = out_path;
    fs::path ext       = out_path.has_extension() ? out_path.extension() : fs::path{};

-    std::string ext_lower = ext.string();
+    std::string ext_lower = ext.u8string();
    std::transform(ext_lower.begin(), ext_lower.end(), ext_lower.begin(), ::tolower);
-    const EncodedImageFormat output_format = encoded_image_format_from_path(out_path.string());
+    const EncodedImageFormat output_format = encoded_image_format_from_path(out_path.u8string());
    if (!ext.empty()) {
        if (output_format == EncodedImageFormat::JPEG ||
            output_format == EncodedImageFormat::PNG ||
@@ -493,8 +494,8 @@ bool save_results(const SDCliParams& cli_params,
        std::string params          = gen_params.embed_image_metadata
                                          ? get_image_params(ctx_params, gen_params, metadata_seed, cli_params.mode)
                                          : "";
-        const bool ok               = write_image_to_file(path.string(), img.data, img.width, img.height, img.channel, params, cli_params.compression_quality);
-        LOG_INFO("save result image %d to '%s' (%s)", idx, path.string().c_str(), ok ? "success" : "failure");
+        const bool ok               = write_image_to_file(path.u8string(), img.data, img.width, img.height, img.channel, params, cli_params.compression_quality);
+        LOG_INFO("save result image %d to '%s' (%s)", idx, path.u8string().c_str(), ok ? "success" : "failure");
        return ok;
    };

@@ -502,14 +503,14 @@ bool save_results(const SDCliParams& cli_params,
        if (generated_audio == nullptr) {
            return;
        }
-        if (write_wav_to_file(wav_path.string(),
+        if (write_wav_to_file(wav_path.u8string(),
                              generated_audio->data,
                              generated_audio->sample_count,
                              generated_audio->channels,
                              generated_audio->sample_rate)) {
-            LOG_INFO("save result audio to '%s'", wav_path.string().c_str());
+            LOG_INFO("save result audio to '%s'", wav_path.u8string().c_str());
        } else {
-            LOG_WARN("failed to save result audio to '%s'", wav_path.string().c_str());
+            LOG_WARN("failed to save result audio to '%s'", wav_path.u8string().c_str());
        }
    };

@@ -522,7 +523,7 @@ bool save_results(const SDCliParams& cli_params,
        pattern += ext;

        for (int i = 0; i < num_results; ++i) {
-            fs::path img_path = format_frame_idx(pattern.string(), output_begin_idx + i);
+            fs::path img_path = fs::u8path(format_frame_idx(pattern.u8string(), output_begin_idx + i));
            if (write_image(img_path, i)) {
                sucessful_reults++;
            }
@@ -540,11 +541,11 @@ bool save_results(const SDCliParams& cli_params,

        fs::path video_path = base_path;
        video_path += ext;
-        std::string final_ext_lower = ext.string();
+        std::string final_ext_lower = ext.u8string();
        std::transform(final_ext_lower.begin(), final_ext_lower.end(), final_ext_lower.begin(), ::tolower);
        const bool mux_audio = generated_audio != nullptr && (final_ext_lower == ".avi" || final_ext_lower == ".webm");
-        if (create_video_from_sd_images(video_path.string().c_str(), results, num_results, gen_params.fps, cli_params.compression_quality, mux_audio ? generated_audio : nullptr, params) == 0) {
-            LOG_INFO("save result video to '%s'", video_path.string().c_str());
+        if (create_video_from_sd_images(video_path.u8string().c_str(), results, num_results, gen_params.fps, cli_params.compression_quality, mux_audio ? generated_audio : nullptr, params) == 0) {
+            LOG_INFO("save result video to '%s'", video_path.u8string().c_str());
            if (generated_audio != nullptr && !mux_audio) {
                fs::path wav_path = video_path;
                wav_path.replace_extension(".wav");
@@ -552,7 +553,7 @@ bool save_results(const SDCliParams& cli_params,
            }
            return true;
        } else {
-            LOG_ERROR("Failed to save result video to '%s'", video_path.string().c_str());
+            LOG_ERROR("Failed to save result video to '%s'", video_path.u8string().c_str());
            return false;
        }
    }
@@ -701,7 +702,7 @@ int main(int argc, const char* argv[]) {
    LOG_DEBUG("%s", gen_params.to_string().c_str());

    if (!cli_params.imatrix_out.empty()) {
-        if (fs::exists(cli_params.imatrix_out) &&
+        if (fs::exists(fs::u8path(cli_params.imatrix_out)) &&
            std::find(cli_params.imatrix_in.begin(), cli_params.imatrix_in.end(), cli_params.imatrix_out) == cli_params.imatrix_in.end()) {
            LOG_WARN("imatrix file '%s' already exists and will be overwritten", cli_params.imatrix_out.c_str());
        }
@@ -75,7 +75,7 @@ static std::string utf16_to_utf8(const std::wstring& wstr) {
    return utf8;
 }

-static std::string argv_to_utf8(int index, const char** argv) {
+std::string argv_to_utf8(int index, const char** argv) {
    (void)argv;
    int argc;
    wchar_t** argv_w = CommandLineToArgvW(GetCommandLineW(), &argc);
@@ -91,7 +91,7 @@ static std::string argv_to_utf8(int index, const char** argv) {
 }

 #else  // Linux / macOS
-static std::string argv_to_utf8(int index, const char** argv) {
+std::string argv_to_utf8(int index, const char** argv) {
    return std::string(argv[index]);
 }

@@ -839,16 +839,17 @@ ArgOptions SDContextParams::get_options() {
 void SDContextParams::build_embedding_map() {
    static const std::vector<std::string> valid_ext = {".gguf", ".safetensors", ".pt", ".ckpt"};

-    if (!fs::exists(embedding_dir) || !fs::is_directory(embedding_dir)) {
+    const fs::path embedding_dir_path = fs::u8path(embedding_dir);
+    if (!fs::exists(embedding_dir_path) || !fs::is_directory(embedding_dir_path)) {
        return;
    }

-    for (auto& p : fs::directory_iterator(embedding_dir)) {
+    for (auto& p : fs::directory_iterator(embedding_dir_path)) {
        if (!p.is_regular_file())
            continue;

        auto path       = p.path();
-        std::string ext = path.extension().string();
+        std::string ext = path.extension().u8string();

        bool valid = false;
        for (auto& e : valid_ext) {
@@ -860,8 +861,8 @@ void SDContextParams::build_embedding_map() {
        if (!valid)
            continue;

-        std::string key   = path.stem().string();
-        std::string value = path.string();
+        std::string key   = path.stem().u8string();
+        std::string value = path.u8string();

        embedding_map[key] = value;
    }
@@ -1572,7 +1573,7 @@ ArgOptions SDGenerationParams::get_options() {
        if (++index >= argc) {
            return -1;
        }
-        ref_image_paths.push_back(argv[index]);
+        ref_image_paths.push_back(argv_to_utf8(index, argv));
        return 1;
    };

@@ -1580,7 +1581,7 @@ ArgOptions SDGenerationParams::get_options() {
        if (++index >= argc) {
            return -1;
        }
-        ref_video_paths.push_back(argv[index]);
+        ref_video_paths.push_back(argv_to_utf8(index, argv));
        return 1;
    };

@@ -1588,7 +1589,7 @@ ArgOptions SDGenerationParams::get_options() {
        if (++index >= argc) {
            return -1;
        }
-        ref_video_audio_paths.push_back(argv[index]);
+        ref_video_audio_paths.push_back(argv_to_utf8(index, argv));
        return 1;
    };

@@ -1596,7 +1597,7 @@ ArgOptions SDGenerationParams::get_options() {
        if (++index >= argc) {
            return -1;
        }
-        ref_audio_paths.push_back(argv[index]);
+        ref_audio_paths.push_back(argv_to_utf8(index, argv));
        return 1;
    };

@@ -1604,7 +1605,7 @@ ArgOptions SDGenerationParams::get_options() {
        if (++index >= argc) {
            return -1;
        }
-        ref_audio_paths.push_back(argv[index]);
+        ref_audio_paths.push_back(argv_to_utf8(index, argv));
        return 1;
    };

@@ -1703,15 +1704,15 @@ ArgOptions SDGenerationParams::get_options() {
        if (++index >= argc) {
            return -1;
        }
-        const char* arg = argv[index];
-        std::ifstream f(arg, std::ios::binary);
+        const std::string arg = argv_to_utf8(index, argv);
+        std::ifstream f(fs::u8path(arg), std::ios::binary);
        try {
            prompt = std::string(std::istreambuf_iterator<char>{f}, {});
        } catch (const std::ios_base::failure&) {
            f.setstate(std::ios_base::failbit);
        }
        if (f.fail()) {
-            LOG_ERROR("error: failed to read prompt file '%s'\n", arg);
+            LOG_ERROR("error: failed to read prompt file '%s'\n", arg.c_str());
            return -1;
        }
        return 1;
@@ -1721,15 +1722,15 @@ ArgOptions SDGenerationParams::get_options() {
        if (++index >= argc) {
            return -1;
        }
-        const char* arg = argv[index];
-        std::ifstream f(arg, std::ios::binary);
+        const std::string arg = argv_to_utf8(index, argv);
+        std::ifstream f(fs::u8path(arg), std::ios::binary);
        try {
            negative_prompt = std::string(std::istreambuf_iterator<char>{f}, {});
        } catch (const std::ios_base::failure&) {
            f.setstate(std::ios_base::failbit);
        }
        if (f.fail()) {
-            LOG_ERROR("error: failed to read negative prompt file '%s'\n", arg);
+            LOG_ERROR("error: failed to read negative prompt file '%s'\n", arg.c_str());
            return -1;
        }
        return 1;
@@ -2050,22 +2051,22 @@ static bool resolve_model_file_from_dir(const std::string& model_name,
    if (model_name.empty() ||
        model_name.find('/') != std::string::npos ||
        model_name.find('\\') != std::string::npos ||
-        fs::path(model_name).has_root_path() ||
+        fs::u8path(model_name).has_root_path() ||
        ends_with_valid_ext()) {
        LOG_ERROR("%s must be a model name without path or extension: %s", label, model_name.c_str());
        return false;
    }

-    fs::path model_dir_path = model_dir;
+    fs::path model_dir_path = fs::u8path(model_dir);
    for (const auto& ext : valid_ext) {
-        fs::path try_path = model_dir_path / (model_name + ext);
+        fs::path try_path = model_dir_path / fs::u8path(model_name + ext);
        if (fs::exists(try_path) && fs::is_regular_file(try_path)) {
-            resolved_path = try_path.lexically_normal().string();
+            resolved_path = try_path.lexically_normal().u8string();
            return true;
        }
    }

-    LOG_ERROR("can not find %s %s in %s", label, model_name.c_str(), model_dir_path.lexically_normal().string().c_str());
+    LOG_ERROR("can not find %s %s in %s", label, model_name.c_str(), model_dir_path.lexically_normal().u8string().c_str());
    return false;
 }

@@ -2359,10 +2360,18 @@ bool SDGenerationParams::extract_and_remove_lora(const std::string& lora_model_d
        }

        fs::path final_path;
-        if (is_absolute_path(raw_path)) {
-            final_path = raw_path;
-        } else {
-            final_path = fs::path(lora_model_dir) / raw_path;
+        try {
+            if (is_absolute_path(raw_path)) {
+                final_path = fs::u8path(raw_path);
+            } else {
+                final_path = fs::u8path(lora_model_dir) / fs::u8path(raw_path);
+            }
+        } catch (const std::exception&) {  // not UTF-8, e.g. a prompt file in another encoding
+            LOG_WARN("invalid lora path %s", raw_path.c_str());
+            tmp     = m.suffix().str();
+            prompt  = std::regex_replace(prompt, re, "", std::regex_constants::format_first_only);
+            missing = true;
+            continue;
        }
        if (!fs::exists(final_path)) {
            bool found = false;
@@ -2376,7 +2385,7 @@ bool SDGenerationParams::extract_and_remove_lora(const std::string& lora_model_d
                }
            }
            if (!found) {
-                LOG_WARN("can not found lora %s", final_path.lexically_normal().string().c_str());
+                LOG_WARN("can not found lora %s", final_path.lexically_normal().u8string().c_str());
                tmp     = m.suffix().str();
                prompt  = std::regex_replace(prompt, re, "", std::regex_constants::format_first_only);
                missing = true;
@@ -2384,7 +2393,7 @@ bool SDGenerationParams::extract_and_remove_lora(const std::string& lora_model_d
            }
        }

-        const std::string key = final_path.lexically_normal().string();
+        const std::string key = final_path.lexically_normal().u8string();

        if (is_high_noise)
            high_noise_lora_map[key] += mul;
@@ -12,6 +12,10 @@
 #include "resource_owners.hpp"
 #include "stable-diffusion.h"

+// The command-line argument at `index` as UTF-8 (on Windows, read again from
+// the UTF-16 command line: argv itself is in the ANSI code page).
+std::string argv_to_utf8(int index, const char** argv);
+
 #define SAFE_STR(s) ((s) ? (s) : "")
 #define BOOL_STR(b) ((b) ? "true" : "false")

@@ -15,10 +15,12 @@

 #define STB_IMAGE_IMPLEMENTATION
 #define STB_IMAGE_STATIC
+#define STBI_WINDOWS_UTF8
 #include "stb_image.h"

 #define STB_IMAGE_WRITE_IMPLEMENTATION
 #define STB_IMAGE_WRITE_STATIC
+#define STBIW_WINDOWS_UTF8
 #include "stb_image_write.h"

 #define STB_IMAGE_RESIZE_IMPLEMENTATION
@@ -147,7 +149,7 @@ private:
 #endif

 bool read_binary_file_bytes(const char* path, std::vector<uint8_t>& data) {
-    std::ifstream fin(fs::path(path), std::ios::binary);
+    std::ifstream fin(fs::u8path(path), std::ios::binary);
    if (!fin) {
        return false;
    }
@@ -170,7 +172,7 @@ bool read_binary_file_bytes(const char* path, std::vector<uint8_t>& data) {
 }

 bool write_binary_file_bytes(const std::string& path, const std::vector<uint8_t>& data) {
-    std::ofstream fout(fs::path(path), std::ios::binary);
+    std::ofstream fout(fs::u8path(path), std::ios::binary);
    if (!fout) {
        return false;
    }
@@ -714,7 +716,7 @@ static std::pair<uint64_t, uint64_t> audio_sample_range_for_video_frame(const sd
 }

 EncodedImageFormat encoded_image_format_from_path(const std::string& path) {
-    std::string ext = fs::path(path).extension().string();
+    std::string ext = fs::u8path(path).extension().u8string();
    std::transform(ext.begin(), ext.end(), ext.begin(), ::tolower);

    if (ext == ".jpg" || ext == ".jpeg" || ext == ".jpe") {
@@ -1413,7 +1415,7 @@ bool write_wav_to_file(const std::string& path,
        return false;
    }

-    std::ofstream file(path, std::ios::binary);
+    std::ofstream file(fs::u8path(path), std::ios::binary);
    if (!file.is_open()) {
        return false;
    }
@@ -1475,7 +1477,7 @@ bool load_wav_from_file(const std::string& path,
    sample_rate = 0;
    channels    = 0;

-    std::ifstream file(path, std::ios::binary);
+    std::ifstream file(fs::u8path(path), std::ios::binary);
    uint8_t riff_header[12];
    if (!file.read(reinterpret_cast<char*>(riff_header), sizeof(riff_header)) ||
        std::memcmp(riff_header, "RIFF", 4) != 0 ||
@@ -1,5 +1,6 @@
 #include "routes.h"

+#include <filesystem>
 #include <fstream>
 #include <iterator>

@@ -7,7 +8,7 @@ void register_index_endpoints(httplib::Server& svr, const SDSvrParams& svr_param
    const std::string serve_html_path = svr_params.serve_html_path;
    svr.Get("/", [serve_html_path, index_html](const httplib::Request&, httplib::Response& res) {
        if (!serve_html_path.empty()) {
-            std::ifstream file(serve_html_path);
+            std::ifstream file(std::filesystem::u8path(serve_html_path));
            if (file) {
                std::string content((std::istreambuf_iterator<char>(file)), std::istreambuf_iterator<char>());
                res.set_content(content, "text/html");
@@ -28,10 +28,10 @@ static std::string extract_and_remove_sd_cpp_extra_args(std::string& text) {
 static fs::path resolve_display_model_path(const ServerRuntime& runtime) {
    const auto& ctx = *runtime.ctx_params;
    if (!ctx.model_path.empty()) {
-        return fs::path(ctx.model_path);
+        return fs::u8path(ctx.model_path);
    }
    if (!ctx.diffusion_model_path.empty()) {
-        return fs::path(ctx.diffusion_model_path);
+        return fs::u8path(ctx.diffusion_model_path);
    }
    return {};
 }
@@ -583,9 +583,9 @@ void register_sdapi_endpoints(httplib::Server& svr, ServerRuntime& rt) {
    svr.Get("/sdapi/v1/sd-models", [runtime](const httplib::Request&, httplib::Response& res) {
        fs::path model_path = resolve_display_model_path(*runtime);
        json entry;
-        entry["title"]      = model_path.stem();
-        entry["model_name"] = model_path.stem();
-        entry["filename"]   = model_path.filename();
+        entry["title"]      = model_path.stem().u8string();
+        entry["model_name"] = model_path.stem().u8string();
+        entry["filename"]   = model_path.filename().u8string();
        entry["hash"]       = "8888888888";
        entry["sha256"]     = "8888888888888888888888888888888888888888888888888888888888888888";
        entry["config"]     = nullptr;
@@ -598,7 +598,7 @@ void register_sdapi_endpoints(httplib::Server& svr, ServerRuntime& rt) {
        fs::path model_path = resolve_display_model_path(*runtime);
        json r;
        r["samples_format"]      = "png";
-        r["sd_model_checkpoint"] = model_path.stem();
+        r["sd_model_checkpoint"] = model_path.stem().u8string();
        res.set_content(r.dump(), "application/json");
    });

@@ -90,10 +90,10 @@ static json make_vae_tiling_json(const sd_tiling_params_t& params) {
 static fs::path resolve_display_model_path(const ServerRuntime& runtime) {
    const auto& ctx = *runtime.ctx_params;
    if (!ctx.model_path.empty()) {
-        return fs::path(ctx.model_path);
+        return fs::u8path(ctx.model_path);
    }
    if (!ctx.diffusion_model_path.empty()) {
-        return fs::path(ctx.diffusion_model_path);
+        return fs::u8path(ctx.diffusion_model_path);
    }
    return {};
 }
@@ -22,7 +22,7 @@ static std::string lower_ascii(std::string value) {
 }

 static bool is_supported_model_ext(const fs::path& p) {
-    auto ext = lower_ascii(p.extension().string());
+    auto ext = lower_ascii(p.extension().u8string());
    return ext == ".gguf" || ext == ".pt" || ext == ".pth" || ext == ".safetensors";
 }

@@ -226,7 +226,7 @@ bool SDSvrParams::validate() {
        return false;
    }

-    if (!serve_html_path.empty() && !fs::exists(serve_html_path)) {
+    if (!serve_html_path.empty() && !fs::exists(fs::u8path(serve_html_path))) {
        LOG_ERROR("error: serve_html_path file does not exist: %s", serve_html_path.c_str());
        return false;
    }
@@ -260,7 +260,7 @@ void refresh_lora_cache(ServerRuntime& rt) {
        return;
    }

-    fs::path lora_dir = rt.ctx_params->lora_model_dir;
+    fs::path lora_dir = fs::u8path(rt.ctx_params->lora_model_dir);
    std::error_code ec;
    if (fs::exists(lora_dir, ec) && !ec && fs::is_directory(lora_dir, ec) && !ec) {
        try {
@@ -293,13 +293,13 @@ void refresh_lora_cache(ServerRuntime& rt) {
                it.increment(ec);
            }
        } catch (const std::exception& e) {
-            LOG_WARN("error while scanning lora directory '%s': %s", lora_dir.string().c_str(), e.what());
+            LOG_WARN("error while scanning lora directory '%s': %s", lora_dir.u8string().c_str(), e.what());
            return;
        }
    }

    if (ec) {
-        LOG_WARN("error while scanning lora directory '%s': %s", lora_dir.string().c_str(), ec.message().c_str());
+        LOG_WARN("error while scanning lora directory '%s': %s", lora_dir.u8string().c_str(), ec.message().c_str());
        return;
    }

@@ -328,7 +328,7 @@ void refresh_upscaler_cache(ServerRuntime& rt) {
        previous_cache = *rt.upscaler_cache;
    }

-    fs::path upscaler_dir = rt.ctx_params->hires_upscalers_dir;
+    fs::path upscaler_dir = fs::u8path(rt.ctx_params->hires_upscalers_dir);
    std::error_code ec;
    if (fs::exists(upscaler_dir, ec) && !ec && fs::is_directory(upscaler_dir, ec) && !ec) {
        try {
@@ -384,13 +384,13 @@ void refresh_upscaler_cache(ServerRuntime& rt) {
                it.increment(ec);
            }
        } catch (const std::exception& e) {
-            LOG_WARN("error while scanning upscalers directory '%s': %s", upscaler_dir.string().c_str(), e.what());
+            LOG_WARN("error while scanning upscalers directory '%s': %s", upscaler_dir.u8string().c_str(), e.what());
            return;
        }
    }

    if (ec) {
-        LOG_WARN("error while scanning upscalers directory '%s': %s", upscaler_dir.string().c_str(), ec.message().c_str());
+        LOG_WARN("error while scanning upscalers directory '%s': %s", upscaler_dir.u8string().c_str(), ec.message().c_str());
        return;
    }

@@ -1995,6 +1995,9 @@ struct LLMEmbedder : public Conditioner {
            if (sd_version_is_pid(version)) {
                throw std::runtime_error("PiD requires an external Gemma 2 tokenizer.json; pass --tokenizer FILE or set sd_ctx_params_t::tokenizer");
            }
+            if (version == VERSION_MOSS_IMAGE) {
+                throw std::runtime_error("MOSS-Image requires the official Qwen3.5 tokenizer.json; pass --tokenizer FILE or set sd_ctx_params_t::tokenizer");
+            }
        }
        LLM::LLMArch arch = LLM::LLMArch::QWEN2_5_VL;
        if (version == VERSION_FLUX2) {
@@ -2006,6 +2009,7 @@ struct LLMEmbedder : public Conditioner {
        } else if (sd_version_is_pid(version)) {
            arch = LLM::LLMArch::GEMMA2_2B;
        } else if (version == VERSION_QWEN_IMAGE_2_1 ||
+                   sd_version_is_flux3(version) ||
                   sd_version_is_lingbot_video(version) ||
                   sd_version_is_ideogram4(version) ||
                   sd_version_is_boogu_image(version) ||
@@ -2017,6 +2021,8 @@ struct LLMEmbedder : public Conditioner {
            arch = LLM::LLMArch::QWEN3_VL;
        } else if (sd_version_is_z_image(version) || sd_version_is_z_image_l2p(version) || version == VERSION_OVIS_IMAGE || version == VERSION_FLUX2_KLEIN) {
            arch = LLM::LLMArch::QWEN3;
+        } else if (version == VERSION_MOSS_IMAGE) {
+            arch = LLM::LLMArch::QWEN3_5;
        }
        llm        = std::make_shared<LLM::LLMRunner>(arch,
                                               backend,
@@ -2031,6 +2037,8 @@ struct LLMEmbedder : public Conditioner {
            pad_id = 199999;
        } else if (arch == LLM::LLMArch::GEMMA2_2B) {
            pad_id = 0;
+        } else if (arch == LLM::LLMArch::QWEN3_5) {
+            pad_id = 248044;
        }
        tokenizer = tokenizers.create(TokenizerConfig::MAIN, llm->config.vocab_size, pad_id);
        if (!tokenizer) {
@@ -2201,6 +2209,9 @@ struct LLMEmbedder : public Conditioner {
            weights.insert(weights.end(), suffix_tokens.size(), 1.f);
        }

+        if (sd_version_is_flux3(version)) {
+            min_length = std::min(max_length, ((tokens.size() + 79) / 80) * 80);
+        }
        std::vector<float> mask;
        tokenizer->pad_tokens(tokens, &weights, &mask, min_length, max_length);

@@ -2254,6 +2265,15 @@ struct LLMEmbedder : public Conditioner {
            }
        }

+        sd::Tensor<int32_t> position_ids;
+        if (sd_version_is_flux3(version)) {
+            position_ids = sd::Tensor<int32_t>::zeros({static_cast<int64_t>(tokens.size() * 4)});
+            for (size_t axis = 0; axis < 4; ++axis) {
+                for (size_t i = 0; i < tokens.size(); ++i) {
+                    position_ids[axis * tokens.size() + i] = mask[i] == 0.f ? 1 : static_cast<int32_t>(i);
+                }
+            }
+        }
        auto hidden_states = llm->compute(n_threads,
                                          input_ids,
                                          attention_mask,
@@ -2262,7 +2282,8 @@ struct LLMEmbedder : public Conditioner {
                                          false,
                                          false,
                                          deepstack_image_embeds,
-                                          image_grids);
+                                          image_grids,
+                                          position_ids);
        if (hidden_states.empty()) {
            LOG_ERROR("LLM prompt encoding failed");
            return {};
@@ -3025,6 +3046,20 @@ struct LLMEmbedder : public Conditioner {

                prompt += "<|im_end|>\n<|im_start|>assistant\n";
            }
+        } else if (version == VERSION_MOSS_IMAGE) {
+            prompt_template_encode_start_idx = 0;
+            max_length                       = 4096;
+            prompt                           = "<|im_start|>user\n" + conditioner_params.text + "<|im_end|>\n<|im_start|>assistant\n<think>\n";
+            prompt_attn_range                = {0, 0};
+        } else if (sd_version_is_flux3(version)) {
+            prompt_template_encode_start_idx = 0;
+            max_length                       = 4096;
+            out_layers                       = {4, 8, 12, 16, 20, 24, 28, 32};
+            prompt                           = "<|im_start|>user\n";
+            prompt_attn_range.first          = static_cast<int>(prompt.size());
+            prompt += conditioner_params.text;
+            prompt_attn_range.second = static_cast<int>(prompt.size());
+            prompt += "<|im_end|>\n<|im_start|>assistant\n";
        } else if (version == VERSION_FLUX2_KLEIN) {
            prompt_template_encode_start_idx = 0;
            min_length                       = 512;
@@ -3,6 +3,7 @@
 #include <cstdint>
 #include <cstring>
 #include <exception>
+#include <filesystem>
 #include <fstream>
 #include <map>
 #include <memory>
@@ -108,7 +109,7 @@ static bool preallocate_output_file(const std::string& output_path, uint64_t fil
        return true;
    }

-    std::fstream file(output_path, std::ios::binary | std::ios::in | std::ios::out);
+    std::fstream file(std::filesystem::u8path(output_path), std::ios::binary | std::ios::in | std::ios::out);
    if (!file.is_open()) {
        if (error != nullptr) {
            *error = "failed to open output file '" + output_path + "' for preallocation";
@@ -251,7 +252,7 @@ static bool stream_tensor_data(ModelLoader& model_loader,

    for (int worker = 0; worker < n_threads; worker++) {
        workers.emplace_back([&]() {
-            std::fstream output_file(output_path, std::ios::binary | std::ios::in | std::ios::out);
+            std::fstream output_file(std::filesystem::u8path(output_path), std::ios::binary | std::ios::in | std::ios::out);
            if (!output_file.is_open()) {
                fail("failed to open output file '" + output_path + "' for tensor writing");
                return;
@@ -344,22 +344,15 @@ ggml_tensor* ggml_ext_pad(ggml_context* ctx,
 }

 static ggml_tensor* conv_1d(ggml_context* ctx, ggml_tensor* x, ggml_tensor* w, int s0, int p0, int d0, bool force_prec_f32) {
-    ggml_tensor* result;
-    if (force_prec_f32) {
-        ggml_tensor* patches = ggml_im2col(ctx, w, x, s0, 0, p0, 0, d0, 0, false, GGML_TYPE_F32);
-        result               = ggml_mul_mat(ctx,
-                                            ggml_reshape_2d(ctx, patches, patches->ne[0], patches->ne[2] * patches->ne[1]),
-                                            ggml_reshape_2d(ctx, w, w->ne[0] * w->ne[1], w->ne[2]));
-        result               = ggml_reshape_3d(ctx, result, patches->ne[1], w->ne[2], patches->ne[2]);
-    } else {
-        result = ggml_conv_1d(ctx, w, x, s0, p0, d0);
+    if (!force_prec_f32) {
+        return ggml_conv_1d(ctx, w, x, s0, p0, d0);
    }
-    if (x->ne[2] > 1) {
-        // mul_mat packs positions and batches before output channels: [OL, N, OC].
-        result = ggml_reshape_3d(ctx, result, result->ne[0], x->ne[2], w->ne[2]);
-        result = ggml_cont(ctx, ggml_permute(ctx, result, 0, 2, 1, 3));
-    }
-    return result;
+    ggml_tensor* patches = ggml_im2col(ctx, w, x, s0, 0, p0, 0, d0, 0, false, GGML_TYPE_F32);
+    ggml_tensor* result  = ggml_mul_mat(ctx,
+                                        ggml_reshape_2d(ctx, patches, patches->ne[0], patches->ne[2] * patches->ne[1]),
+                                        ggml_reshape_2d(ctx, w, w->ne[0] * w->ne[1], w->ne[2]));
+    result               = ggml_reshape_3d(ctx, result, patches->ne[1], patches->ne[2], w->ne[2]);
+    return ggml_cont(ctx, ggml_permute(ctx, result, 0, 2, 1, 3));
 }

 ggml_tensor* ggml_ext_conv_1d(ggml_context* ctx,
@@ -3,6 +3,7 @@
 #include <algorithm>
 #include <cstdlib>
 #include <cstring>
+#include <filesystem>
 #include <fstream>

 #include "core/ggml_extend_backend.h"
@@ -100,7 +101,7 @@ void ggml_ext_tensor_diff(
 }

 ggml_tensor* load_tensor_from_file(ggml_context* ctx, const std::string& file_path) {
-    std::ifstream file(file_path, std::ios::binary);
+    std::ifstream file(std::filesystem::u8path(file_path), std::ios::binary);
    if (!file.is_open()) {
        LOG_ERROR("failed to open '%s'", file_path.c_str());
        return nullptr;
@@ -3,6 +3,7 @@

 #include <array>
 #include <cstring>
+#include <filesystem>
 #include <fstream>
 #include <stdexcept>
 #include <string>
@@ -105,7 +106,7 @@ namespace sd {

    template <typename T>
    inline Tensor<T> load_tensor_from_file_as_tensor(const std::string& file_path) {
-        std::ifstream file(file_path, std::ios::binary);
+        std::ifstream file(std::filesystem::u8path(file_path), std::ios::binary);
        if (!file.is_open()) {
            throw std::runtime_error("failed to open tensor file: " + file_path);
        }
@@ -7,6 +7,7 @@
 #include <cstdlib>
 #include <cstring>
 #include <exception>
+#include <filesystem>
 #include <fstream>
 #include <locale>
 #include <regex>
@@ -123,12 +124,12 @@ int round_up_to(int value, int base) {
 #include <windows.h>

 bool file_exists(const std::string& filename) {
-    DWORD attributes = GetFileAttributesA(filename.c_str());
+    DWORD attributes = GetFileAttributesW(std::filesystem::u8path(filename).c_str());
    return (attributes != INVALID_FILE_ATTRIBUTES && !(attributes & FILE_ATTRIBUTE_DIRECTORY));
 }

 bool is_directory(const std::string& path) {
-    DWORD attributes = GetFileAttributesA(path.c_str());
+    DWORD attributes = GetFileAttributesW(std::filesystem::u8path(path).c_str());
    return (attributes != INVALID_FILE_ATTRIBUTES && (attributes & FILE_ATTRIBUTE_DIRECTORY));
 }

@@ -152,8 +153,8 @@ std::unique_ptr<MmapWrapper> MmapWrapper::create(const std::string& filename, bo
    void* mapped_data = nullptr;
    size_t file_size  = 0;

-    HANDLE file_handle = CreateFileA(
-        filename.c_str(),
+    HANDLE file_handle = CreateFileW(
+        std::filesystem::u8path(filename).c_str(),
        GENERIC_READ,
        FILE_SHARE_READ,
        nullptr,
@@ -66,6 +66,8 @@ enum SDVersion {
    VERSION_MING_IMAGE,
    VERSION_Z_IMAGE_L2P,
    VERSION_IRIS,
+    VERSION_FLUX3,
+    VERSION_MOSS_IMAGE,
    VERSION_COUNT,
 };

@@ -118,6 +120,10 @@ static inline bool sd_version_is_flux(SDVersion version) {
    return false;
 }

+static inline bool sd_version_is_flux3(SDVersion version) {
+    return version == VERSION_FLUX3;
+}
+
 static inline bool sd_version_is_flux2(SDVersion version) {
    if (version == VERSION_FLUX2 || version == VERSION_FLUX2_KLEIN) {
        return true;
@@ -280,7 +286,14 @@ static inline bool sd_version_uses_flux_vae(SDVersion version) {
 }

 static inline bool sd_version_uses_flux2_vae(SDVersion version) {
-    if (sd_version_is_flux2(version) || sd_version_is_ernie_image(version) || sd_version_is_lens(version) || sd_version_is_ideogram4(version) || sd_version_is_sefi_image(version) || sd_version_is_llada_image(version)) {
+    if (sd_version_is_flux2(version) ||
+        sd_version_is_flux3(version) ||
+        sd_version_is_ernie_image(version) ||
+        sd_version_is_lens(version) ||
+        sd_version_is_ideogram4(version) ||
+        sd_version_is_sefi_image(version) ||
+        sd_version_is_llada_image(version) ||
+        version == VERSION_MOSS_IMAGE) {
        return true;
    }
    return false;
@@ -311,6 +324,7 @@ static inline bool sd_version_is_inpaint(SDVersion version) {
 static inline bool sd_version_is_dit(SDVersion version) {
    if (sd_version_is_flux(version) ||
        sd_version_is_flux2(version) ||
+        sd_version_is_flux3(version) ||
        sd_version_is_ltxav(version) ||
        sd_version_is_minimax_h3(version) ||
        sd_version_is_sd3(version) ||
@@ -336,7 +350,8 @@ static inline bool sd_version_is_dit(SDVersion version) {
        sd_version_is_mage_flow(version) ||
        sd_version_is_sensenova_u1(version) ||
        sd_version_is_pixart(version) ||
-        version == VERSION_IRIS) {
+        version == VERSION_IRIS ||
+        version == VERSION_MOSS_IMAGE) {
        return true;
    }
    return false;
@@ -13,8 +13,8 @@
 #include <vector>

 #include "core/ggml_extend.h"
-#include "core/ggml_runner.h"
 #include "core/ggml_extend_backend.h"
+#include "core/ggml_runner.h"
 #include "model.h"

 class GGMLBlock {
@@ -0,0 +1,256 @@
+#ifndef __SD_MODEL_DIFFUSION_FLUX3_HPP__
+#define __SD_MODEL_DIFFUSION_FLUX3_HPP__
+
+#include "model/diffusion/flux.hpp"
+
+namespace Flux3 {
+
+    struct Flux3Config {
+        int64_t hidden_size = 3072;
+        int64_t context_dim = 20480;
+        int64_t channels    = 128;
+        int64_t mlp_dim     = 9216;
+        int heads           = 24;
+        int depth           = 5;
+        int single_depth    = 28;
+        bool flow_map       = false;
+
+        static Flux3Config detect_from_weights(const String2TensorStorage& tensors, const std::string& prefix) {
+            Flux3Config config;
+            int depth = 0, single_depth = 0;
+            bool detected = false;
+            for (const auto& [name, tensor] : tensors) {
+                if (!starts_with(name, prefix + ".")) {
+                    continue;
+                }
+                std::string key = name.substr(prefix.size() + 1);
+                if (key == "txt_in.weight") {
+                    detected           = true;
+                    config.context_dim = tensor.ne[0];
+                    config.hidden_size = tensor.ne[1];
+                } else if (key == "emb_in.image.weight") {
+                    config.channels = tensor.ne[0];
+                } else if (key == "single_blocks.0.mlp_out.weight") {
+                    config.mlp_dim = tensor.ne[0];
+                } else if (key == "ref_time_in.in_layer.weight") {
+                    config.flow_map = true;
+                }
+                if (starts_with(key, "txt_mode_blocks.")) {
+                    depth = std::max(depth, atoi(key.substr(16).c_str()) + 1);
+                }
+                if (starts_with(key, "single_blocks.")) {
+                    single_depth = std::max(single_depth, atoi(key.substr(14).c_str()) + 1);
+                }
+            }
+            if (depth > 0)
+                config.depth = depth;
+            if (single_depth > 0)
+                config.single_depth = single_depth;
+            config.heads = static_cast<int>(config.hidden_size / 128);
+            if (detected) {
+                LOG_VERBOSE("flux3: hidden_size = %" PRId64 ", context_dim = %" PRId64 ", depth = %d, single_depth = %d, flow_map = %s",
+                            config.hidden_size, config.context_dim, config.depth, config.single_depth, config.flow_map ? "true" : "false");
+            }
+            return config;
+        }
+    };
+
+    static ggml_tensor* token_slice(ggml_context* ctx, ggml_tensor* x, int64_t start, int64_t length) {
+        return ggml_view_3d(ctx, x, x->ne[0], length, x->ne[2], x->nb[1], x->nb[2], start * x->nb[1]);
+    }
+
+    struct Block : public GGMLBlock {
+        int heads;
+
+        Block(const Flux3Config& config)
+            : heads(config.heads) {
+            for (const auto* name : {"q_proj", "k_proj", "v_proj", "attn_out"}) {
+                blocks[name] = std::make_shared<Linear>(config.hidden_size, config.hidden_size, false);
+            }
+            blocks["mlp_in"]  = std::make_shared<Linear>(config.hidden_size, 2 * config.mlp_dim, false);
+            blocks["mlp_out"] = std::make_shared<Linear>(config.mlp_dim, config.hidden_size, false);
+            blocks["norm"]    = std::make_shared<Flux::QKNorm>(config.hidden_size / heads);
+        }
+
+        ggml_tensor* linear(GGMLRunnerContext* ctx, const std::string& name, ggml_tensor* x) {
+            return std::dynamic_pointer_cast<Linear>(blocks[name])->forward(ctx, x);
+        }
+
+        ggml_tensor* forward(GGMLRunnerContext* ctx, ggml_tensor* x, ggml_tensor* pe, const std::vector<Flux::ModulationOut>& mods, const std::vector<int64_t>& lengths) {
+            auto gc            = ctx->ggml_ctx;
+            ggml_tensor* x_mod = nullptr;
+            int64_t offset     = 0;
+            for (size_t i = 0; i < lengths.size(); ++i) {
+                auto part = token_slice(gc, x, offset, lengths[i]);
+                part      = Flux::modulate(gc, ggml_norm(gc, part, 1e-6f), mods[i].shift, mods[i].scale);
+                x_mod     = x_mod ? ggml_concat(gc, x_mod, part, 1) : part;
+                offset += lengths[i];
+            }
+            auto split = [&](ggml_tensor* v) {
+                return ggml_reshape_4d(gc, v, x->ne[0] / heads, heads, x->ne[1], x->ne[2]);
+            };
+            auto norm        = std::dynamic_pointer_cast<Flux::QKNorm>(blocks["norm"]);
+            auto q           = norm->query_norm(ctx, split(linear(ctx, "q_proj", x_mod)));
+            auto k           = norm->key_norm(ctx, split(linear(ctx, "k_proj", x_mod)));
+            auto v           = split(linear(ctx, "v_proj", x_mod));
+            auto attn        = Rope::attention(ctx, q, k, v, pe, nullptr);
+            auto mlp         = ggml_ext_silu_act(gc, linear(ctx, "mlp_in", x_mod));
+            auto update      = ggml_add(gc, linear(ctx, "attn_out", attn), linear(ctx, "mlp_out", mlp));
+            ggml_tensor* out = nullptr;
+            offset           = 0;
+            for (size_t i = 0; i < lengths.size(); ++i) {
+                auto part = ggml_mul(gc, token_slice(gc, update, offset, lengths[i]), mods[i].gate);
+                part      = ggml_add(gc, token_slice(gc, x, offset, lengths[i]), part);
+                out       = out ? ggml_concat(gc, out, part, 1) : part;
+                offset += lengths[i];
+            }
+            return out;
+        }
+    };
+
+    struct Model : public GGMLBlock {
+        Flux3Config config;
+
+        Model(const Flux3Config& config)
+            : config(config) {
+            blocks["txt_in"]  = std::make_shared<Linear>(config.context_dim, config.hidden_size, false);
+            blocks["time_in"] = std::make_shared<Flux::MLPEmbedder>(256, config.hidden_size, false);
+            if (config.flow_map) {
+                blocks["ref_time_in"] = std::make_shared<Flux::MLPEmbedder>(256, config.hidden_size, false);
+            }
+            for (const std::string mode : {"image", "image_cond", "txt"}) {
+                blocks["early_stream_modulations." + mode]  = std::make_shared<Flux::Modulation>(config.hidden_size, false, false);
+                blocks["single_stream_modulations." + mode] = std::make_shared<Flux::Modulation>(config.hidden_size, false, false);
+                const std::string base                      = mode == "txt" ? "txt_mode_blocks." : "content_mode_blocks." + mode + ".";
+                for (int i = 0; i < config.depth; ++i) {
+                    blocks[base + std::to_string(i)] = std::make_shared<Block>(config);
+                }
+                if (mode != "txt") {
+                    blocks["emb_in." + mode] = std::make_shared<Linear>(config.channels, config.hidden_size, false);
+                }
+            }
+            for (int i = 0; i < config.single_depth; ++i) {
+                blocks["single_blocks." + std::to_string(i)] = std::make_shared<Block>(config);
+            }
+            blocks["final_layer.image"] = std::make_shared<Flux::LastLayer>(config.hidden_size, 1, config.channels, false, false);
+        }
+
+        ggml_tensor* forward(GGMLRunnerContext* ctx, ggml_tensor* image, ggml_tensor* text, ggml_tensor* reference, ggml_tensor* timestep, ggml_tensor* zero, ggml_tensor* target, ggml_tensor* pe) {
+            auto gc             = ctx->ggml_ctx;
+            const int64_t width = image->ne[0], height = image->ne[1];
+            auto time_in   = std::dynamic_pointer_cast<Flux::MLPEmbedder>(blocks["time_in"]);
+            auto image_vec = time_in->forward(ctx, ggml_ext_timestep_embedding(gc, timestep, 256, 10000, 1000.f));
+            auto clean_vec = time_in->forward(ctx, ggml_ext_timestep_embedding(gc, zero, 256, 10000, 1000.f));
+            if (config.flow_map) {
+                auto ref_time = std::dynamic_pointer_cast<Flux::MLPEmbedder>(blocks["ref_time_in"]);
+                auto ref_vec  = ref_time->forward(ctx, ggml_ext_timestep_embedding(gc, target, 256, 10000, 1000.f));
+                image_vec     = ggml_add(gc, image_vec, ref_vec);
+                clean_vec     = ggml_add(gc, clean_vec, ref_vec);
+            }
+            std::vector<std::string> modes{"image"};
+            std::vector<ggml_tensor*> inputs{DiT::patchify(gc, image, 1, 1)};
+            if (reference) {
+                modes.push_back("image_cond");
+                inputs.push_back(reference);
+            }
+            modes.push_back("txt");
+            inputs.push_back(text);
+            std::vector<Flux::ModulationOut> single_mods;
+            std::vector<int64_t> lengths;
+            ggml_tensor* joint = nullptr;
+            int64_t offset     = 0;
+            for (size_t m = 0; m < modes.size(); ++m) {
+                const auto& mode = modes[m];
+                auto vec         = m == 0 ? image_vec : clean_vec;
+                auto early       = std::dynamic_pointer_cast<Flux::Modulation>(blocks["early_stream_modulations." + mode])->forward(ctx, vec)[0];
+                single_mods.push_back(std::dynamic_pointer_cast<Flux::Modulation>(blocks["single_stream_modulations." + mode])->forward(ctx, vec)[0]);
+                lengths.push_back(inputs[m]->ne[1]);
+                auto h                 = std::dynamic_pointer_cast<Linear>(blocks[mode == "txt" ? "txt_in" : "emb_in." + mode])->forward(ctx, inputs[m]);
+                auto mode_pe           = ggml_view_4d(gc, pe, 2, 2, 64, lengths.back(), pe->nb[1], pe->nb[2], pe->nb[3], offset * pe->nb[3]);
+                const std::string base = mode == "txt" ? "txt_mode_blocks." : "content_mode_blocks." + mode + ".";
+                for (int i = 0; i < config.depth; ++i) {
+                    h = std::dynamic_pointer_cast<Block>(blocks[base + std::to_string(i)])->forward(ctx, h, mode_pe, {early}, {lengths.back()});
+                }
+                joint = joint ? ggml_concat(gc, joint, h, 1) : h;
+                offset += lengths.back();
+            }
+            for (int i = 0; i < config.single_depth; ++i) {
+                joint = std::dynamic_pointer_cast<Block>(blocks["single_blocks." + std::to_string(i)])->forward(ctx, joint, pe, single_mods, lengths);
+            }
+            auto out = std::dynamic_pointer_cast<Flux::LastLayer>(blocks["final_layer.image"])->forward(ctx, token_slice(gc, joint, 0, lengths[0]), image_vec);
+            return DiT::unpatchify(gc, out, height, width, 1, 1);
+        }
+    };
+
+    struct Flux3Runner : public DiffusionModelRunner {
+        Flux3Config config;
+        Model model;
+        std::vector<float> pe_values;
+        sd::Tensor<float> zero = sd::Tensor<float>::zeros({1});
+
+        Flux3Runner(ggml_backend_t backend, const String2TensorStorage& tensors, const std::string& prefix, std::shared_ptr<RunnerWeightManager> weight_manager = nullptr)
+            : DiffusionModelRunner(backend, prefix, weight_manager),
+              config(Flux3Config::detect_from_weights(tensors, prefix)),
+              model(config) {
+            model.init(params_ctx, tensors, prefix);
+        }
+
+        std::string get_desc() override { return "flux3"; }
+        void get_param_tensors(std::map<std::string, ggml_tensor*>& tensors, const std::string& prefix) override {
+            model.get_param_tensors(tensors, prefix);
+        }
+
+        sd::Tensor<float> compute(int n_threads, const DiffusionParams& params) override {
+            if (params.ref_latents && params.ref_image_params.pass_to_dit && params.ref_latents->size() > 10) {
+                LOG_ERROR("FLUX.3 supports at most 10 reference images");
+                return {};
+            }
+            const auto* extra = diffusion_extra_as<Flux3DiffusionExtra>(params);
+            GGML_ASSERT(params.x && params.context && params.timesteps);
+            GGML_ASSERT(!config.flow_map || extra->target_timestep);
+            auto get_graph = [&]() {
+                auto gf   = new_graph_custom(FLUX_GRAPH_SIZE);
+                auto x    = make_input(*params.x);
+                auto text = make_input(*params.context);
+                GGML_ASSERT(x->ne[3] == 1 && text->ne[2] == 1);
+                Rope::Embedding embedding;
+                auto append_image = [&](int64_t w, int64_t h, float index) {
+                    embedding.positions.append_image(static_cast<int>(h), static_cast<int>(w), 1, 0.5f, 0.5f);
+                    for (int64_t y = 0; y < h; ++y) {
+                        for (int64_t x = 0; x < w; ++x) {
+                            embedding.ids.push_back({0.f, y * 0.5f, x * 0.5f, index});
+                        }
+                    }
+                };
+                append_image(x->ne[0], x->ne[1], 0.f);
+                ggml_tensor* references = nullptr;
+                if (params.ref_latents && params.ref_image_params.pass_to_dit) {
+                    int index = 0;
+                    for (const auto& ref : *params.ref_latents) {
+                        auto r = make_input(ref);
+                        append_image(r->ne[0], r->ne[1], -float(++index));
+                        r          = DiT::patchify(compute_ctx, r, 1, 1);
+                        references = references ? ggml_concat(compute_ctx, references, r, 1) : r;
+                    }
+                }
+                for (int64_t i = 0; i < text->ne[1]; ++i) {
+                    embedding.ids.push_back({0.f, 0.f, 0.f, static_cast<float>(i)});
+                }
+                embedding.positions.append_tokens(text->ne[1]);
+                embedding.values = Rope::embed_nd(embedding.ids, 1, 10000.f, {32, 32, 32, 32}, embedding.layout, &embedding.frequencies);
+                pe_values        = finish_rope_pe(std::move(embedding));
+                auto pe          = ggml_new_tensor_4d(compute_ctx, GGML_TYPE_F32, 2, 2, 64, pe_values.size() / 256);
+                set_backend_tensor_data(pe, pe_values.data());
+                auto runner_ctx = get_context();
+                auto out        = model.forward(&runner_ctx, x, text, references, make_input(*params.timesteps), make_input(zero),
+                                         config.flow_map ? make_input(*extra->target_timestep) : nullptr, pe);
+                ggml_build_forward_expand(gf, out);
+                return gf;
+            };
+            return restore_trailing_singleton_dims(GGMLRunner::compute(get_graph, n_threads, false), params.x->dim());
+        }
+    };
+
+}  // namespace Flux3
+
+#endif  // __SD_MODEL_DIFFUSION_FLUX3_HPP__
@@ -64,6 +64,10 @@ struct FluxDiffusionExtra {
    float pulid_id_weight               = 1.0f;
 };

+struct Flux3DiffusionExtra {
+    const sd::Tensor<float>* target_timestep = nullptr;
+};
+
 struct AnimaDiffusionExtra {
    const sd::Tensor<int32_t>* t5_ids   = nullptr;
    const sd::Tensor<float>* t5_weights = nullptr;
@@ -149,6 +153,7 @@ using DiffusionExtraParams = std::variant<std::monostate,
                                          UNetDiffusionExtra,
                                          SkipLayerDiffusionExtra,
                                          FluxDiffusionExtra,
+                                          Flux3DiffusionExtra,
                                          AnimaDiffusionExtra,
                                          QwenImage21DiffusionExtra,
                                          WanDiffusionExtra,
@@ -0,0 +1,262 @@
+#ifndef __SD_MODEL_DIFFUSION_MOSS_IMAGE_HPP__
+#define __SD_MODEL_DIFFUSION_MOSS_IMAGE_HPP__
+
+#include <algorithm>
+#include <cinttypes>
+
+#include "model/common/ggml_block.hpp"
+#include "model/common/rope.hpp"
+#include "model/diffusion/dit.hpp"
+#include "model/diffusion/model.hpp"
+#include "model_loader.h"
+
+namespace MOSSImage {
+    struct MOSSImageConfig {
+        int64_t dim      = 3072;
+        int64_t ffn_dim  = 8192;
+        int64_t text_dim = 4096;
+        int64_t channels = 128;
+        int64_t head_dim = 128;
+        int64_t depth    = 42;
+        int64_t freq_dim = 256;
+
+        static MOSSImageConfig detect_from_weights(const String2TensorStorage& weights, const std::string& prefix) {
+            MOSSImageConfig config;
+            int64_t depth = 0;
+            for (const auto& [name, tensor] : weights) {
+                if (!starts_with(name, prefix + ".")) {
+                    continue;
+                }
+                auto key = name.substr(prefix.size() + 1);
+                if (key == "video_embedder.weight") {
+                    config.channels = tensor.ne[0];
+                    config.dim      = tensor.ne[1];
+                } else if (key == "text_embedder.proj.weight") {
+                    config.text_dim = tensor.ne[0];
+                } else if (key == "layers.0.proj.ffn_down.weight") {
+                    config.ffn_dim = tensor.ne[0];
+                } else if (key == "layers.0.q_norm.weight") {
+                    config.head_dim = tensor.ne[0];
+                } else if (key == "time_embedding.0.weight") {
+                    config.freq_dim = tensor.ne[0];
+                }
+                if (starts_with(key, "layers.")) {
+                    depth = std::max<int64_t>(depth, std::stoi(key.substr(7)) + 1);
+                }
+            }
+            if (depth) {
+                config.depth = depth;
+                LOG_VERBOSE("moss_image: dim = %" PRId64 ", ffn_dim = %" PRId64 ", depth = %" PRId64 ", head_dim = %" PRId64,
+                            config.dim, config.ffn_dim, config.depth, config.head_dim);
+            }
+            return config;
+        }
+    };
+
+    inline ggml_tensor* situ(ggml_context* c, ggml_tensor* hidden, ggml_tensor* gate) {
+        auto a = ggml_scale(c, ggml_tanh(c, ggml_scale(c, gate, 0.25f)), 4.f);
+        a      = ggml_mul(c, a, ggml_sigmoid(c, gate));
+        auto b = ggml_scale(c, ggml_tanh(c, ggml_scale(c, hidden, 1.f / 25.f)), 25.f);
+        return ggml_mul(c, a, b);
+    }
+
+    struct TextBlock : public GGMLBlock {
+        MOSSImageConfig config;
+
+        TextBlock(const MOSSImageConfig& config)
+            : config(config) {
+            blocks["norm1"]    = std::make_shared<RMSNorm>(config.dim, 1e-6f);
+            blocks["norm2"]    = std::make_shared<RMSNorm>(config.dim, 1e-6f);
+            blocks["q_norm"]   = std::make_shared<RMSNorm>(config.head_dim, 1e-6f);
+            blocks["k_norm"]   = std::make_shared<RMSNorm>(config.head_dim, 1e-6f);
+            blocks["qkv"]      = std::make_shared<Linear>(config.dim, config.dim * 3, true);
+            blocks["proj"]     = std::make_shared<Linear>(config.dim, config.dim, true);
+            blocks["ffn_up"]   = std::make_shared<Linear>(config.dim, config.ffn_dim, true);
+            blocks["ffn_gate"] = std::make_shared<Linear>(config.dim, config.ffn_dim, true);
+            blocks["ffn_down"] = std::make_shared<Linear>(config.ffn_dim, config.dim, true);
+        }
+
+        ggml_tensor* forward(GGMLRunnerContext* ctx, ggml_tensor* x, ggml_tensor* pe) {
+            auto c      = ctx->ggml_ctx;
+            auto linear = [&](const char* name, ggml_tensor* h) { return std::dynamic_pointer_cast<Linear>(blocks[name])->forward(ctx, h); };
+            auto norm   = [&](const char* name, ggml_tensor* h) { return std::dynamic_pointer_cast<RMSNorm>(blocks[name])->forward(ctx, h); };
+            auto y      = norm("norm1", x);
+            auto qkv    = linear("qkv", y);
+            auto split  = [&](int i) {
+                return ggml_reshape_4d(c, ggml_ext_slice(c, qkv, 0, i * config.dim, (i + 1) * config.dim), config.head_dim, config.dim / config.head_dim, x->ne[1], x->ne[2]);
+            };
+            auto q = norm("q_norm", split(0));
+            auto k = norm("k_norm", split(1));
+            auto v = split(2);
+            y      = Rope::attention(ctx, q, k, v, pe, nullptr, 1.f, false);
+            y      = linear("proj", y);
+            x      = ggml_add(c, x, y);
+            y      = norm("norm2", x);
+            y      = situ(c, linear("ffn_up", y), linear("ffn_gate", y));
+            y      = linear("ffn_down", y);
+            return ggml_add(c, x, y);
+        }
+    };
+
+    struct Block : public GGMLBlock {
+        MOSSImageConfig config;
+
+        Block(const MOSSImageConfig& config)
+            : config(config) {
+            blocks["norm1"]            = std::make_shared<RMSNorm>(config.dim, 1e-6f);
+            blocks["norm2"]            = std::make_shared<RMSNorm>(config.dim, 1e-6f);
+            blocks["q_norm"]           = std::make_shared<RMSNorm>(config.head_dim, 1e-6f);
+            blocks["k_norm"]           = std::make_shared<RMSNorm>(config.head_dim, 1e-6f);
+            blocks["proj.qkv"]         = std::make_shared<Linear>(config.dim, config.dim * 3, false);
+            blocks["proj.o"]           = std::make_shared<Linear>(config.dim, config.dim, false);
+            blocks["proj.ffn_up_gate"] = std::make_shared<Linear>(config.dim, config.ffn_dim * 2, false);
+            blocks["proj.ffn_down"]    = std::make_shared<Linear>(config.ffn_dim, config.dim, false);
+        }
+
+        ggml_tensor* forward(GGMLRunnerContext* ctx, ggml_tensor* x, ggml_tensor* pe, ggml_tensor* mod) {
+            auto c      = ctx->ggml_ctx;
+            auto linear = [&](const char* name, ggml_tensor* h) { return std::dynamic_pointer_cast<Linear>(blocks[name])->forward(ctx, h); };
+            auto norm   = [&](const char* name, ggml_tensor* h) { return std::dynamic_pointer_cast<RMSNorm>(blocks[name])->forward(ctx, h); };
+            auto part   = [&](int i) { return ggml_ext_slice(c, mod, 0, i * config.dim, (i + 1) * config.dim); };
+            auto y      = norm("norm1", x);
+            y           = ggml_add(c, ggml_add(c, y, ggml_mul(c, y, part(1))), part(0));
+            auto qkv    = linear("proj.qkv", y);
+            auto split  = [&](int i) {
+                return ggml_reshape_4d(c, ggml_ext_slice(c, qkv, 0, i * config.dim, (i + 1) * config.dim), config.head_dim, config.dim / config.head_dim, x->ne[1], x->ne[2]);
+            };
+            auto q       = norm("q_norm", split(0));
+            auto k       = norm("k_norm", split(1));
+            auto v       = split(2);
+            y            = Rope::attention(ctx, q, k, v, pe, nullptr, 1.f, false);
+            y            = linear("proj.o", y);
+            y            = ggml_mul(c, y, part(2));
+            x            = ggml_add(c, x, y);
+            y            = norm("norm2", x);
+            y            = ggml_add(c, ggml_add(c, y, ggml_mul(c, y, part(4))), part(3));
+            auto up_gate = linear("proj.ffn_up_gate", y);
+            y            = situ(c, ggml_ext_slice(c, up_gate, 0, 0, config.ffn_dim), ggml_ext_slice(c, up_gate, 0, config.ffn_dim, 2 * config.ffn_dim));
+            y            = linear("proj.ffn_down", y);
+            y            = ggml_mul(c, y, part(5));
+            return ggml_add(c, x, y);
+        }
+    };
+
+    struct Model : public GGMLBlock {
+        MOSSImageConfig config;
+
+        Model(const MOSSImageConfig& config)
+            : config(config) {
+            blocks["video_embedder"]     = std::make_shared<Linear>(config.channels, config.dim, true);
+            blocks["text_embedder.proj"] = std::make_shared<Linear>(config.text_dim, config.dim, true);
+            for (int i = 0; i < 2; ++i) {
+                blocks["text_embedder.blocks." + std::to_string(i)] = std::make_shared<TextBlock>(config);
+            }
+            blocks["time_embedding.0"]        = std::make_shared<Linear>(config.freq_dim, config.dim, true);
+            blocks["time_embedding.2"]        = std::make_shared<Linear>(config.dim, config.dim, true);
+            blocks["shared_block_modulation"] = std::make_shared<Linear>(config.dim, 6 * config.dim, true);
+            blocks["video_head_modulation"]   = std::make_shared<Linear>(config.dim, 2 * config.dim, true);
+            for (int64_t i = 0; i < config.depth; ++i) {
+                blocks["layers." + std::to_string(i)] = std::make_shared<Block>(config);
+            }
+            blocks["final_norm_video"]   = std::make_shared<RMSNorm>(config.dim, 1e-6f);
+            blocks["final_linear_video"] = std::make_shared<Linear>(config.dim, config.channels, false);
+        }
+
+        ggml_tensor* forward(GGMLRunnerContext* ctx, ggml_tensor* image, ggml_tensor* time, ggml_tensor* text, ggml_tensor* pe, const std::vector<int>& skip) {
+            auto c            = ctx->ggml_ctx;
+            auto linear       = [&](const char* name, ggml_tensor* h) { return std::dynamic_pointer_cast<Linear>(blocks[name])->forward(ctx, h); };
+            int64_t image_len = image->ne[0] * image->ne[1];
+            auto x            = linear("video_embedder", DiT::patchify(c, image, 1, 1));
+            text              = linear("text_embedder.proj", text);
+            auto text_pe      = ggml_ext_slice(c, pe, 3, image_len, image_len + text->ne[1]);
+            for (int i = 0; i < 2; ++i) {
+                text = std::dynamic_pointer_cast<TextBlock>(blocks["text_embedder.blocks." + std::to_string(i)])->forward(ctx, text, text_pe);
+            }
+            time      = linear("time_embedding.2", ggml_silu(c, linear("time_embedding.0", time)));
+            auto mod  = linear("shared_block_modulation", time);
+            auto head = linear("video_head_modulation", time);
+            x         = ggml_concat(c, x, text, 1);
+            for (int64_t i = 0; i < config.depth; ++i) {
+                if (std::find(skip.begin(), skip.end(), i) == skip.end()) {
+                    x = std::dynamic_pointer_cast<Block>(blocks["layers." + std::to_string(i)])->forward(ctx, x, pe, mod);
+                    sd::ggml_graph_cut::mark_graph_cut(x, "moss_image.layers." + std::to_string(i), "x");
+                }
+            }
+            x          = ggml_ext_slice(c, x, 1, 0, image_len);
+            x          = std::dynamic_pointer_cast<RMSNorm>(blocks["final_norm_video"])->forward(ctx, x);
+            auto shift = ggml_ext_slice(c, head, 0, 0, config.dim);
+            auto scale = ggml_ext_slice(c, head, 0, config.dim, 2 * config.dim);
+            x          = ggml_add(c, ggml_add(c, x, ggml_mul(c, x, scale)), shift);
+            x          = linear("final_linear_video", x);
+            return DiT::unpatchify(c, x, image->ne[1], image->ne[0], 1, 1);
+        }
+    };
+
+    struct MOSSImageRunner : public DiffusionModelRunner {
+        MOSSImageConfig config;
+        Model model;
+        std::vector<float> pe_data, time_data;
+
+        MOSSImageRunner(ggml_backend_t backend, const String2TensorStorage& weights, const std::string& prefix, std::shared_ptr<RunnerWeightManager> weight_manager = nullptr)
+            : DiffusionModelRunner(backend, prefix, weight_manager), config(MOSSImageConfig::detect_from_weights(weights, prefix)), model(config) {
+            model.init(params_ctx, weights, prefix);
+        }
+
+        std::string get_desc() override { return "MOSS-Image"; }
+        void get_param_tensors(std::map<std::string, ggml_tensor*>& tensors, const std::string& prefix) override {
+            model.get_param_tensors(tensors, prefix);
+        }
+
+        sd::Tensor<float> compute(int n_threads, const DiffusionParams& inputs) override {
+            if (!inputs.x || !inputs.context || !inputs.timesteps || inputs.context->empty()) {
+                LOG_ERROR("MOSS-Image requires latents, timesteps and text features");
+                return {};
+            }
+            std::vector<int> skip;
+            if (auto extra = std::get_if<SkipLayerDiffusionExtra>(&inputs.extra)) {
+                if (extra->skip_layers) {
+                    skip = *extra->skip_layers;
+                }
+            }
+            auto graph = [&]() {
+                auto gf   = new_graph_custom(65536);
+                auto x    = make_input(*inputs.x);
+                auto text = make_input(*inputs.context);
+                GGML_ASSERT(x->ne[3] == 1 && x->ne[2] == config.channels && text->ne[0] == config.text_dim);
+                int64_t h = x->ne[1], w = x->ne[0], nt = text->ne[1];
+                Rope::Embedding embedding;
+                embedding.positions.append_image(static_cast<int>(h), static_cast<int>(w));
+                embedding.positions.images.back().height_axis = 2;
+                embedding.positions.images.back().width_axis  = 3;
+                embedding.positions.append_tokens(nt);
+                for (int64_t token = 0; token < h * w + nt; ++token) {
+                    float pos[4] = {0.f, float(token - h * w), 0.f, 0.f};
+                    if (token < h * w) {
+                        pos[1] = float(nt);
+                        pos[2] = float(token / w) - float(h - 1) / 2.f;
+                        pos[3] = float(token % w) - float(w - 1) / 2.f;
+                    }
+                    embedding.ids.push_back({pos[0], pos[1], pos[2], pos[3]});
+                }
+                embedding.values = Rope::embed_nd(embedding.ids, 1, 10000.f, std::vector<int>(4, static_cast<int>(config.head_dim / 4)), embedding.layout, &embedding.frequencies);
+                pe_data          = finish_rope_pe(std::move(embedding));
+                time_data.resize(config.freq_dim);
+                for (int64_t i = 0; i < config.freq_dim / 2; ++i) {
+                    double angle                       = inputs.timesteps->values()[0] * std::pow(10000.0, -double(i) / double(config.freq_dim / 2));
+                    time_data[i]                       = float(std::cos(angle));
+                    time_data[i + config.freq_dim / 2] = float(std::sin(angle));
+                }
+                auto pe   = ggml_new_tensor_4d(compute_ctx, GGML_TYPE_F32, 2, 2, config.head_dim / 2, h * w + nt);
+                auto time = ggml_new_tensor_3d(compute_ctx, GGML_TYPE_F32, config.freq_dim, 1, 1);
+                set_backend_tensor_data(pe, pe_data.data());
+                set_backend_tensor_data(time, time_data.data());
+                auto ctx = get_context();
+                ggml_build_forward_expand(gf, model.forward(&ctx, x, time, text, pe, skip));
+                return gf;
+            };
+            return restore_trailing_singleton_dims(GGMLRunner::compute(graph, n_threads, false), inputs.x->dim());
+        }
+    };
+}  // namespace MOSSImage
+
+#endif  // __SD_MODEL_DIFFUSION_MOSS_IMAGE_HPP__
@@ -43,6 +43,7 @@ namespace LLM {
        QWEN2_5_VL,
        QWEN3,
        QWEN3_VL,
+        QWEN3_5,
        MISTRAL_SMALL_3_2,
        MINISTRAL_3_3B,
        GEMMA3_12B,
@@ -59,6 +60,7 @@ namespace LLM {
        "qwen2.5vl",
        "qwen3",
        "qwen3vl",
+        "qwen3.5",
        "mistral_small3.2",
        "ministral3.3b",
        "gemma3_12b",
@@ -121,6 +123,7 @@ namespace LLM {
        bool qk_norm                    = false;
        bool final_norm                 = true;
        bool rms_norm_add               = false;
+        bool qwen35_gguf                = false;
        bool normalize_input            = false;
        int64_t vocab_size              = 152064;
        int64_t max_position_embeddings = 128000;
@@ -134,6 +137,11 @@ namespace LLM {
        bool qkv_fused              = false;
        bool bidirectional          = false;
        float partial_rotary        = 1.f;
+        int linear_head_dim         = 128;
+        int linear_key_heads        = 16;
+        int linear_value_heads      = 32;
+        int linear_conv_kernel      = 4;
+        std::set<int> linear_attention_layers;

        // DeepSeek-V3-style grouped-sigmoid MoE routing (LLaDA2)
        int64_t moe_intermediate_size = 0;
@@ -167,6 +175,19 @@ namespace LLM {
                config.num_kv_heads = 8;
                config.qkv_bias     = false;
                config.rms_norm_eps = 1e-5f;
+            } else if (arch == LLMArch::QWEN3_5) {
+                config.hidden_size             = 4096;
+                config.intermediate_size       = 12288;
+                config.head_dim                = 256;
+                config.num_heads               = 16;
+                config.num_kv_heads            = 4;
+                config.qkv_bias                = false;
+                config.qk_norm                 = true;
+                config.rms_norm_add            = true;
+                config.partial_rotary          = 0.25f;
+                config.rope_thetas             = {10000000.f};
+                config.max_position_embeddings = 262144;
+                config.vocab_size              = 248320;
            } else if (arch == LLMArch::QWEN3 || arch == LLMArch::QWEN3_VL) {
                config.head_dim     = 128;
                config.num_heads    = 32;
@@ -365,6 +386,41 @@ namespace LLM {
            if ((arch == LLMArch::QWEN3 || arch == LLMArch::QWEN3_VL) && config.num_layers == 28) {
                config.num_heads = 16;
            }
+            if (arch == LLMArch::QWEN3_5) {
+                config.qwen35_gguf  = tensor_storage_map.count(prefix + ".model.layers.0.linear_attn.A") > 0;
+                config.rms_norm_add = !config.qwen35_gguf;
+                for (const auto& [name, tensor] : tensor_storage_map) {
+                    if (!starts_with(name, prefix + ".")) {
+                        continue;
+                    }
+                    if (ends_with(name, "self_attn.q_norm.weight")) {
+                        config.head_dim = static_cast<int>(tensor.ne[0]);
+                    } else if (ends_with(name, "linear_attn.norm.weight")) {
+                        config.linear_head_dim = static_cast<int>(tensor.ne[0]);
+                    } else if (ends_with(name, "linear_attn.conv1d.weight")) {
+                        config.linear_conv_kernel = static_cast<int>(tensor.ne[0]);
+                        const auto pos            = name.find("layers.");
+                        config.linear_attention_layers.insert(std::stoi(name.substr(pos + 7)));
+                    }
+                }
+                for (const auto& [name, tensor] : tensor_storage_map) {
+                    if (!starts_with(name, prefix + ".")) {
+                        continue;
+                    }
+                    if (ends_with(name, "self_attn.q_proj.weight")) {
+                        config.num_heads = static_cast<int>(tensor.ne[1] / (2 * config.head_dim));
+                    } else if (ends_with(name, "self_attn.k_proj.weight")) {
+                        config.num_kv_heads = static_cast<int>(tensor.ne[1] / config.head_dim);
+                    } else if (ends_with(name, "linear_attn.in_proj_z.weight")) {
+                        config.linear_value_heads = static_cast<int>(tensor.ne[1] / config.linear_head_dim);
+                    }
+                }
+                for (const auto& [name, tensor] : tensor_storage_map) {
+                    if (starts_with(name, prefix + ".") && ends_with(name, "linear_attn.in_proj_qkv.weight")) {
+                        config.linear_key_heads = static_cast<int>((tensor.ne[1] / config.linear_head_dim - config.linear_value_heads) / 2);
+                    }
+                }
+            }
            if (arch == LLMArch::QWEN3_VL &&
                (config.num_layers == 50 || config.num_layers == 64) &&
                config.hidden_size == 5120) {
@@ -1390,6 +1446,81 @@ namespace LLM {
        }
    };

+    struct Qwen35DeltaNet : public UnaryBlock {
+        int head_dim;
+        int key_heads;
+        int value_heads;
+        int conv_dim;
+        int kernel;
+        bool gguf_layout;
+
+        Qwen35DeltaNet(const LLMConfig& config)
+            : head_dim(config.linear_head_dim), key_heads(config.linear_key_heads), value_heads(config.linear_value_heads), conv_dim(head_dim * (key_heads * 2 + value_heads)), kernel(config.linear_conv_kernel), gguf_layout(config.qwen35_gguf) {
+            auto hidden_size      = config.hidden_size;
+            blocks["in_proj_qkv"] = std::make_shared<Linear>(hidden_size, conv_dim, false);
+            blocks["in_proj_z"]   = std::make_shared<Linear>(hidden_size, head_dim * value_heads, false);
+            blocks["in_proj_a"]   = std::make_shared<Linear>(hidden_size, value_heads, false);
+            blocks["in_proj_b"]   = std::make_shared<Linear>(hidden_size, value_heads, false);
+            blocks["out_proj"]    = std::make_shared<Linear>(head_dim * value_heads, hidden_size, false);
+            blocks["norm"]        = std::make_shared<LLMRMSNorm>(head_dim, 1e-6f);
+        }
+
+        void init_params(ggml_context* ctx, const String2TensorStorage& = {}, std::string = "") override {
+            params["conv1d.weight"]             = gguf_layout ? ggml_new_tensor_2d(ctx, GGML_TYPE_F32, kernel, conv_dim) : ggml_new_tensor_3d(ctx, GGML_TYPE_F32, kernel, 1, conv_dim);
+            params[gguf_layout ? "A" : "A_log"] = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, value_heads);
+            params["dt_bias"]                   = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, value_heads);
+        }
+
+        ggml_tensor* forward(GGMLRunnerContext* ctx, ggml_tensor* x) override {
+            auto c          = ctx->ggml_ctx;
+            const int64_t n = x->ne[1];
+            GGML_ASSERT(x->ne[2] == 1);
+            auto linear        = [&](const char* name) { return std::dynamic_pointer_cast<Linear>(blocks[name])->forward(ctx, x); };
+            auto qkv           = linear("in_proj_qkv");
+            auto padded        = ggml_pad_ext(c, qkv, 0, 0, kernel - 1, 0, 0, 0, 0, 0);
+            ggml_tensor* mixed = nullptr;
+            // Causal depthwise convolution, with zero state for every independent prompt.
+            for (int i = 0; i < kernel; ++i) {
+                auto w    = ggml_cont(c, ggml_view_2d(c, params["conv1d.weight"], 1, conv_dim, kernel * sizeof(float), i * sizeof(float)));
+                w         = ggml_reshape_1d(c, w, conv_dim);
+                auto part = ggml_mul(c, ggml_ext_slice(c, padded, 1, i, i + n), w);
+                mixed     = mixed ? ggml_add(c, mixed, part) : part;
+            }
+            mixed      = ggml_silu(c, mixed);
+            auto heads = [&](int start, int count) {
+                auto h = ggml_ext_slice(c, mixed, 0, start, start + head_dim * count);
+                return ggml_reshape_4d(c, h, head_dim, count, n, 1);
+            };
+            auto repeat_keys = [&](ggml_tensor* h) {
+                h = ggml_scale(c, ggml_rms_norm(c, h, 1e-6f / head_dim), 1.f / std::sqrt(float(head_dim)));
+                // llama.cpp reorders V heads and stores negative exp(A_log) and offset norms.
+                if (gguf_layout) {
+                    return h;
+                }
+                // ggml GDN broadcasts heads cyclically; Qwen uses adjacent repeated heads.
+                h          = ggml_reshape_4d(c, h, head_dim, 1, key_heads, n);
+                auto shape = ggml_new_tensor_4d(c, GGML_TYPE_F32, head_dim, value_heads / key_heads, key_heads, n);
+                return ggml_reshape_4d(c, ggml_repeat(c, h, shape), head_dim, value_heads, n, 1);
+            };
+            auto q     = repeat_keys(heads(0, key_heads));
+            auto k     = repeat_keys(heads(head_dim * key_heads, key_heads));
+            auto v     = heads(2 * head_dim * key_heads, value_heads);
+            auto beta  = ggml_reshape_4d(c, ggml_sigmoid(c, linear("in_proj_b")), 1, value_heads, n, 1);
+            auto decay = ggml_softplus(c, ggml_add(c, linear("in_proj_a"), params["dt_bias"]));
+            auto a     = gguf_layout ? params["A"] : ggml_neg(c, ggml_exp(c, params["A_log"]));
+            decay      = ggml_mul(c, decay, a);
+            decay      = ggml_reshape_4d(c, decay, 1, value_heads, n, 1);
+            auto state = ggml_ext_zeros(c, head_dim, head_dim, value_heads, 1);
+            auto out   = ggml_gated_delta_net(c, q, k, v, decay, beta, state, 1);
+            out        = ggml_view_3d(c, out, head_dim, value_heads, n, head_dim * sizeof(float), head_dim * value_heads * sizeof(float), 0);
+            out        = std::dynamic_pointer_cast<LLMRMSNorm>(blocks["norm"])->forward(ctx, out);
+            auto z     = ggml_reshape_3d(c, linear("in_proj_z"), head_dim, value_heads, n);
+            out        = ggml_mul(c, out, ggml_silu(c, z));
+            out        = ggml_reshape_3d(c, out, head_dim * value_heads, n, 1);
+            return std::dynamic_pointer_cast<Linear>(blocks["out_proj"])->forward(ctx, out);
+        }
+    };
+
    struct Attention : public GGMLBlock {
    protected:
        LLMArch arch;
@@ -1439,7 +1570,7 @@ namespace LLM {
                GGML_ASSERT(!k_eq_v);
                blocks["query_key_value"] = std::make_shared<Linear>(config.hidden_size, (num_heads + num_kv_heads * 2) * head_dim, config.qkv_bias);
            } else {
-                blocks["q_proj"] = std::make_shared<Linear>(config.hidden_size, num_heads * head_dim, config.qkv_bias);
+                blocks["q_proj"] = std::make_shared<Linear>(config.hidden_size, num_heads * head_dim * (arch == LLMArch::QWEN3_5 ? 2 : 1), config.qkv_bias);
                blocks["k_proj"] = std::make_shared<Linear>(config.hidden_size, num_kv_heads * head_dim, config.qkv_bias);
                if (!k_eq_v) {
                    blocks["v_proj"] = std::make_shared<Linear>(config.hidden_size, num_kv_heads * head_dim, config.qkv_bias);
@@ -1502,6 +1633,13 @@ namespace LLM {
                v = k_eq_v ? k : v_proj->forward(ctx, x);  // [N, n_token, num_kv_heads*head_dim]
            }

+            ggml_tensor* output_gate = nullptr;
+            if (arch == LLMArch::QWEN3_5) {
+                q           = ggml_reshape_4d(ctx->ggml_ctx, q, head_dim * 2, num_heads, n_token, N);
+                output_gate = ggml_ext_slice(ctx->ggml_ctx, q, 0, head_dim, head_dim * 2);
+                output_gate = ggml_reshape_3d(ctx->ggml_ctx, output_gate, head_dim * num_heads, n_token, N);
+                q           = ggml_ext_slice(ctx->ggml_ctx, q, 0, 0, head_dim);
+            }
            q = ggml_reshape_4d(ctx->ggml_ctx, q, head_dim, num_heads, n_token, N);     // [N, n_token, num_heads, head_dim]
            k = ggml_reshape_4d(ctx->ggml_ctx, k, head_dim, num_kv_heads, n_token, N);  // [N, n_token, num_kv_heads, head_dim]
            v = ggml_reshape_4d(ctx->ggml_ctx, v, head_dim, num_kv_heads, n_token, N);  // [N, n_token, num_kv_heads, head_dim]
@@ -1658,7 +1796,7 @@ namespace LLM {
                                  1.f,
                                  32.f,
                                  1.f);
-            } else if (arch == LLMArch::LLADA2_MOE) {
+            } else if (arch == LLMArch::LLADA2_MOE || arch == LLMArch::QWEN3_5) {
                // LLaDA2 slices the head (query[..., :rotary_dim]) instead of zero-padding
                // inv_freq like gemma does, so rotate_half pairs i with i + rotary_dim/2 and the
                // frequencies use rotary_dim as the exponent denominator. Passing n_dims =
@@ -1734,6 +1872,9 @@ namespace LLM {
                x = ggml_ext_attention_ext(ctx, q, k, v, num_heads, attention_mask, true, ctx->flash_attn_enabled);  // [N, n_token, hidden_size]
            }

+            if (output_gate) {
+                x = ggml_mul(ctx->ggml_ctx, x, ggml_sigmoid(ctx->ggml_ctx, output_gate));
+            }
            x = out_proj->forward(ctx, x);  // [N, n_token, hidden_size]
            return x;
        }
@@ -1783,7 +1924,11 @@ namespace LLM {
            if (!config.sliding_attention.empty()) {
                sliding_attention = config.sliding_attention[layer_index % config.sliding_attention.size()];
            }
-            blocks["self_attn"] = std::make_shared<Attention>(config, sliding_attention == 0);
+            if (config.arch == LLMArch::QWEN3_5 && config.linear_attention_layers.count(layer_index)) {
+                blocks["linear_attn"] = std::make_shared<Qwen35DeltaNet>(config);
+            } else {
+                blocks["self_attn"] = std::make_shared<Attention>(config, sliding_attention == 0);
+            }
            if (config.arch == LLMArch::GPT_OSS_20B) {
                blocks["mlp"] = std::make_shared<GPTOSSMLP>(config);
            } else if ((config.arch == LLMArch::LLADA2_MOE || config.arch == LLMArch::BAILING_MOE) && layer_index >= config.first_k_dense_replace) {
@@ -1813,7 +1958,7 @@ namespace LLM {
                             ggml_tensor* rope_cos               = nullptr,
                             ggml_tensor* rope_sin               = nullptr) {
            // x: [N, n_token, hidden_size]
-            auto self_attn                                  = std::dynamic_pointer_cast<Attention>(blocks["self_attn"]);
+            auto self_attn                                  = blocks.count("self_attn") ? std::dynamic_pointer_cast<Attention>(blocks["self_attn"]) : nullptr;
            auto input_layernorm                            = std::dynamic_pointer_cast<LLMRMSNorm>(blocks["input_layernorm"]);
            auto pre_ffw_norm                               = std::dynamic_pointer_cast<LLMRMSNorm>(blocks[pre_ffw_norm_name]);
            std::shared_ptr<LLMRMSNorm> post_attention_norm = nullptr;
@@ -1833,7 +1978,11 @@ namespace LLM {

            auto residual = x;
            x             = input_layernorm->forward(ctx, x);
-            x             = self_attn->forward(ctx, x, input_pos, block_attention_mask, rope_index, rope_cos, rope_sin);
+            if (blocks.count("linear_attn")) {
+                x = std::dynamic_pointer_cast<Qwen35DeltaNet>(blocks["linear_attn"])->forward(ctx, x);
+            } else {
+                x = self_attn->forward(ctx, x, input_pos, block_attention_mask, rope_index, rope_cos, rope_sin);
+            }
            if (post_attention_norm != nullptr) {
                x = post_attention_norm->forward(ctx, x);
            }
@@ -2338,7 +2487,8 @@ namespace LLM {
                                 const DeepStackImageEmbeds& deepstack_image_embeds_tensor,
                                 const std::vector<ImageGrid>& image_grids,
                                 std::set<int> out_layers,
-                                 bool return_all_hidden_states = false) {
+                                 bool return_all_hidden_states           = false,
+                                 const sd::Tensor<int32_t>& position_ids = {}) {
            ggml_cgraph* gf        = new_graph_custom(LLM_GRAPH_SIZE);
            ggml_tensor* input_ids = make_input(input_ids_tensor);
            std::vector<std::pair<int, ggml_tensor*>> image_embeds;
@@ -2359,6 +2509,7 @@ namespace LLM {
            if (config.arch == LLMArch::MISTRAL_SMALL_3_2 ||
                config.arch == LLMArch::MINISTRAL_3_3B ||
                config.arch == LLMArch::QWEN3 ||
+                config.arch == LLMArch::QWEN3_5 ||
                config.arch == LLMArch::GEMMA3_12B ||
                config.arch == LLMArch::GEMMA4_12B ||
                config.arch == LLMArch::GEMMA2_2B ||
@@ -2402,6 +2553,10 @@ namespace LLM {
                }
            }

+            if (!position_ids.empty()) {
+                GGML_ASSERT(position_ids.numel() == static_cast<int64_t>(input_pos_vec.size()));
+                input_pos_vec = position_ids.values();
+            }
            auto input_pos = ggml_new_tensor_1d(compute_ctx,
                                                GGML_TYPE_I32,
                                                input_pos_vec.size());
@@ -2475,7 +2630,8 @@ namespace LLM {
                                  bool return_all_hidden_states                      = false,
                                  bool auto_runner_end                               = true,
                                  const DeepStackImageEmbeds& deepstack_image_embeds = {},
-                                  const std::vector<ImageGrid>& image_grids          = {}) {
+                                  const std::vector<ImageGrid>& image_grids          = {},
+                                  const sd::Tensor<int32_t>& position_ids            = {}) {
            auto get_graph = [&]() -> ggml_cgraph* {
                return build_graph(input_ids,
                                   attention_mask,
@@ -2483,7 +2639,8 @@ namespace LLM {
                                   deepstack_image_embeds,
                                   image_grids,
                                   out_layers,
-                                   return_all_hidden_states);
+                                   return_all_hidden_states,
+                                   position_ids);
            };
            return restore_trailing_singleton_dims(GGMLRunner::compute(get_graph, n_threads, auto_runner_end),
                                                   input_ids.dim() + 1);
@@ -363,33 +363,22 @@ namespace MiniMaxH3 {
            auto pre       = std::dynamic_pointer_cast<AudioAttentionProjection>(blocks["pre_block"]);
            auto mean_proj = std::dynamic_pointer_cast<LTXV::Conv1D>(blocks["mean_proj"]);

-            // GGML's batched conv1d storage interleaves the stream dimension
-            // with output channels. Subsequent layers then read stereo samples
-            // as adjacent feature channels. Run each mono stream independently,
-            // matching PyTorch's reshape(B*S, 1, samples), and concatenate only
-            // the completed normalized latents.
            const int64_t streams = waveform->ne[2] * waveform->ne[3];
            waveform              = ggml_reshape_3d(ctx->ggml_ctx,
                                                    waveform,
                                                    waveform->ne[0],
                                                    1,
                                                    streams);
-            ggml_tensor* stereo_z = nullptr;
-            for (int64_t stream = 0; stream < streams; ++stream) {
-                auto mono = ggml_ext_slice(ctx->ggml_ctx, waveform, 2, stream, stream + 1);
-                auto x    = encoder->forward(ctx, mono);
-                x         = ggml_cont(ctx->ggml_ctx, ggml_permute(ctx->ggml_ctx, x, 1, 0, 2, 3));
-                x         = pre->forward(ctx, x);
-                x         = ggml_cont(ctx->ggml_ctx, ggml_permute(ctx->ggml_ctx, x, 1, 0, 2, 3));
-                auto z    = mean_proj->forward(ctx, x);
+            auto x                = encoder->forward(ctx, waveform);
+            x                     = ggml_cont(ctx->ggml_ctx, ggml_permute(ctx->ggml_ctx, x, 1, 0, 2, 3));
+            x                     = pre->forward(ctx, x);
+            x                     = ggml_cont(ctx->ggml_ctx, ggml_permute(ctx->ggml_ctx, x, 1, 0, 2, 3));
+            auto z                = mean_proj->forward(ctx, x);

-                auto mean = ggml_reshape_4d(ctx->ggml_ctx, params["latents_mean"], 1, kLatentChannels, 1, 1);
-                auto std  = ggml_reshape_4d(ctx->ggml_ctx, params["latents_std"], 1, kLatentChannels, 1, 1);
-                z         = ggml_div(ctx->ggml_ctx, ggml_sub(ctx->ggml_ctx, z, mean), std);
-                z         = ggml_cont(ctx->ggml_ctx, ggml_permute(ctx->ggml_ctx, z, 0, 2, 1, 3));
-                stereo_z  = stereo_z == nullptr ? z : ggml_concat(ctx->ggml_ctx, stereo_z, z, 1);
-            }
-            return stereo_z;
+            auto mean = ggml_reshape_4d(ctx->ggml_ctx, params["latents_mean"], 1, kLatentChannels, 1, 1);
+            auto std  = ggml_reshape_4d(ctx->ggml_ctx, params["latents_std"], 1, kLatentChannels, 1, 1);
+            z         = ggml_div(ctx->ggml_ctx, ggml_sub(ctx->ggml_ctx, z, mean), std);
+            return ggml_cont(ctx->ggml_ctx, ggml_permute(ctx->ggml_ctx, z, 0, 2, 1, 3));
        }

        ggml_tensor* decode(GGMLRunnerContext* ctx, ggml_tensor* latent) {
@@ -3,6 +3,7 @@
 #include <algorithm>
 #include <cstdint>
 #include <cstdio>
+#include <filesystem>
 #include <fstream>
 #include <ostream>
 #include <string>
@@ -19,7 +20,7 @@ static void set_error(std::string* error, const std::string& message) {
 }

 bool is_gguf_file(const std::string& file_path) {
-    std::ifstream file(file_path, std::ios::binary);
+    std::ifstream file(std::filesystem::u8path(file_path), std::ios::binary);
    if (!file.is_open()) {
        return false;
    }
@@ -178,7 +179,7 @@ bool GGUFStreamingWriter::write_metadata(const std::string& file_path,
    }

    LOG_INFO("trying to save tensors to %s", file_path.c_str());
-    FILE* file = fopen(file_path.c_str(), "wb+");
+    FILE* file = ggml_fopen(file_path.c_str(), "wb+");
    if (file == nullptr) {
        set_error(error, "failed to open output file '" + file_path + "'");
        close();
@@ -2,6 +2,7 @@
 #define __SD_MODEL_IO_GGUF_READER_EXT_H__

 #include <cstdint>
+#include <filesystem>
 #include <fstream>
 #include <string>
 #include <vector>
@@ -184,7 +185,7 @@ private:

 public:
    bool load(const std::string& file_path) {
-        std::ifstream fin(file_path, std::ios::binary | std::ios::ate);
+        std::ifstream fin(std::filesystem::u8path(file_path), std::ios::binary | std::ios::ate);
        if (!fin) {
            LOG_ERROR("failed to open '%s'", file_path.c_str());
            return false;
@@ -26,15 +26,15 @@ static void set_error(std::string* error, const std::string& message) {
 }

 static std::string resolve_index_shard_path(const std::string& index_path, const std::string& shard_path) {
-    fs::path shard_fs_path(shard_path);
+    fs::path shard_fs_path = fs::u8path(shard_path);
    if (shard_fs_path.is_absolute()) {
-        return shard_fs_path.lexically_normal().string();
+        return shard_fs_path.lexically_normal().u8string();
    }
-    return (fs::path(index_path).parent_path() / shard_fs_path).lexically_normal().string();
+    return (fs::u8path(index_path).parent_path() / shard_fs_path).lexically_normal().u8string();
 }

 bool is_safetensors_file(const std::string& file_path) {
-    std::ifstream file(file_path, std::ios::binary);
+    std::ifstream file(fs::u8path(file_path), std::ios::binary);
    if (!file.is_open()) {
        return false;
    }
@@ -153,7 +153,7 @@ bool read_safetensors_file(const std::string& file_path,
                           std::string* error,
                           std::map<std::string, std::string>* metadata,
                           std::set<std::string>* tensor_names) {
-    std::ifstream file(file_path, std::ios::binary);
+    std::ifstream file(fs::u8path(file_path), std::ios::binary);
    if (!file.is_open()) {
        set_error(error, "failed to open '" + file_path + "'");
        return false;
@@ -418,7 +418,7 @@ bool read_safetensors_index_file(const std::string& file_path,
                                 std::string* error) {
    shard_paths.clear();

-    std::ifstream file(file_path);
+    std::ifstream file(fs::u8path(file_path));
    if (!file.is_open()) {
        set_error(error, "failed to open '" + file_path + "'");
        return false;
@@ -522,7 +522,7 @@ bool write_safetensors_file(const std::string& file_path,

    const std::string header_str = header.dump();

-    std::ofstream file(file_path, std::ios::binary);
+    std::ofstream file(fs::u8path(file_path), std::ios::binary);
    if (!file.is_open()) {
        set_error(error, "failed to open '" + file_path + "' for writing");
        return false;
@@ -596,7 +596,7 @@ bool SafetensorsStreamingWriter::write_metadata(const std::string& file_path,
    data_start_                  = ST_HEADER_SIZE_LEN + header_str.size();

    LOG_INFO("trying to save tensors to %s", file_path.c_str());
-    std::ofstream file(file_path, std::ios::binary | std::ios::trunc);
+    std::ofstream file(fs::u8path(file_path), std::ios::binary | std::ios::trunc);
    if (!file.is_open()) {
        set_error(error, "failed to open '" + file_path + "' for writing");
        return false;
@@ -2,6 +2,7 @@

 #include <algorithm>
 #include <cstdint>
+#include <filesystem>
 #include <fstream>
 #include <string>
 #include <unordered_map>
@@ -98,7 +99,7 @@ static std::string torch_legacy_diagnostics(const std::string& file_path, const
 bool read_torch_legacy_file(const std::string& file_path,
                            std::vector<TensorStorage>& tensor_storages,
                            std::string* error) {
-    std::ifstream file(file_path, std::ios::binary);
+    std::ifstream file(std::filesystem::u8path(file_path), std::ios::binary);
    if (!file.is_open()) {
        set_error(error, "failed to open '" + file_path + "'");
        return false;
@@ -4,6 +4,7 @@
 #include <cinttypes>
 #include <cstdarg>
 #include <cstdlib>
+#include <filesystem>
 #include <fstream>
 #include <functional>
 #include <mutex>
@@ -333,7 +334,7 @@ bool ModelLoader::init_from_safetensors_index_file(const std::string& file_path,
        }
    }

-    std::ifstream index_file(file_path);
+    std::ifstream index_file(std::filesystem::u8path(file_path));
    const auto index = nlohmann::json::parse(index_file);
    for (const auto& entry : index.at("weight_map").items()) {
        const auto expected = (std::filesystem::u8path(file_path).parent_path() /
@@ -438,6 +439,16 @@ bool ModelLoader::init_from_diffusers_file(const std::string& file_path, const s
 SDVersion ModelLoader::get_sd_version() const {
    TensorStorage token_embedding_weight, input_block_weight, context_ebedding_weight;

+    if (tensor_storage_map.find("model.diffusion_model.emb_in.image.weight") != tensor_storage_map.end() &&
+        tensor_storage_map.find("model.diffusion_model.txt_mode_blocks.0.q_proj.weight") != tensor_storage_map.end()) {
+        return VERSION_FLUX3;
+    }
+
+    if (tensor_storage_map.count("model.diffusion_model.video_embedder.weight") &&
+        tensor_storage_map.count("model.diffusion_model.shared_block_modulation.weight")) {
+        return VERSION_MOSS_IMAGE;
+    }
+
    bool has_multiple_encoders = false;
    bool is_unet               = false;

@@ -1145,7 +1156,7 @@ bool ModelLoader::load_tensors(on_new_tensor_cb_t on_new_tensor_cb,
                        return;
                    }
                } else if (!mmapped) {
-                    file.open(file_path, std::ios::binary);
+                    file.open(std::filesystem::u8path(file_path), std::ios::binary);
                    if (!file.is_open()) {
                        LOG_ERROR("failed to open '%s'", file_path.c_str());
                        failed = true;
@@ -1663,6 +1663,26 @@ std::string convert_tensor_name(std::string name, SDVersion version) {

    replace_with_prefix_map(name, prefix_map);

+    if (version == VERSION_MOSS_IMAGE) {
+        const std::string te_prefix = "text_encoders.llm.";
+        if (starts_with(name, te_prefix + "blk.")) {
+            static const std::vector<std::pair<std::string, std::string>> qwen35_name_map = {
+                {"attn_post_norm.", "post_attention_layernorm."},
+                {"post_attention_norm.", "post_attention_layernorm."},
+                {"attn_qkv.", "linear_attn.in_proj_qkv."},
+                {"attn_gate.", "linear_attn.in_proj_z."},
+                {"ssm_conv1d.", "linear_attn.conv1d."},
+                {"ssm_dt.bias", "linear_attn.dt_bias"},
+                {"ssm_norm.", "linear_attn.norm."},
+                {"ssm_out.", "linear_attn.out_proj."},
+                {"ssm_alpha.", "linear_attn.in_proj_a."},
+                {"ssm_beta.", "linear_attn.in_proj_b."},
+                {"ssm_a", "linear_attn.A"},
+            };
+            replace_with_name_map(name, qwen35_name_map);
+        }
+    }
+
    if (version == VERSION_QWEN_IMAGE_2_1 || sd_version_is_boogu_image(version) || sd_version_is_krea2(version) || sd_version_is_mage_flow(version) || sd_version_is_minimax_h3(version)) {
        const std::string hf_vision_prefix = "text_encoders.llm.model.visual.";
        if (starts_with(name, hf_vision_prefix)) {
@@ -111,6 +111,8 @@ const char* model_version_to_str[] = {
    "Ming-Image",
    "Z-Image L2P",
    "Iris-3B",
+    "Flux.3 Image Klein",
+    "MOSS-Image",
 };

 static_assert(VERSION_COUNT == sizeof(model_version_to_str) / sizeof(model_version_to_str[0]),
@@ -1489,9 +1491,10 @@ bool StableDiffusionGGML::build_denoiser() {
                   sd_version_is_boogu_image(version) ||
                   sd_version_is_pid(version) ||
                   sd_version_is_ideogram4(version) ||
-                   version == VERSION_IRIS) {
+                   version == VERSION_IRIS ||
+                   version == VERSION_MOSS_IMAGE) {
            pred_type = FLOW_PRED;
-            if (sd_version_is_wan(version)) {
+            if (sd_version_is_wan(version) || version == VERSION_MOSS_IMAGE) {
                default_flow_shift = 5.f;
            } else if (sd_version_is_hunyuan_video(version)) {
                default_flow_shift = 7.f;
@@ -1514,6 +1517,9 @@ bool StableDiffusionGGML::build_denoiser() {
            } else {
                default_flow_shift = 3.f;
            }
+        } else if (sd_version_is_flux3(version)) {
+            pred_type          = FLUX_FLOW_PRED;
+            default_flow_shift = 6.93f;
        } else if (sd_version_is_flux(version) ||
                   sd_version_is_flux2(version) ||
                   sd_version_is_longcat(version) ||
@@ -1579,7 +1585,11 @@ bool StableDiffusionGGML::build_denoiser() {
        }
        case FLUX_FLOW_PRED: {
            LOG_INFO("running in Flux FLOW mode");
-            denoiser = std::make_shared<FluxFlowDenoiser>();
+            if (sd_version_is_flux3(version)) {
+                denoiser = std::make_shared<Flux3FlowDenoiser>();
+            } else {
+                denoiser = std::make_shared<FluxFlowDenoiser>();
+            }
            break;
        }
        case SEFI_FLOW_PRED: {
@@ -2558,6 +2568,13 @@ sd::Tensor<float> StableDiffusionGGML::sample(const std::shared_ptr<DiffusionMod
        if (sd_version_is_hunyuan_video(version) && step + 1 < sigmas.size()) {
            hunyuan_timestep_r_tensor = sd::Tensor<float>::from_vector({sigmas[step + 1]});
        }
+        sd::Tensor<float> flux3_target_timestep;
+        if (version == VERSION_FLUX3) {
+            // The sign marks intermediate evaluations; the magnitude is the one-based step.
+            const size_t target_index = static_cast<size_t>(std::abs(step));
+            GGML_ASSERT(target_index > 0 && target_index < sigmas.size());
+            flux3_target_timestep = sd::Tensor<float>::from_vector({sigmas[target_index]});
+        }
        sd::Tensor<float> noised_input = x * c_in;
        if (apply_denoise_mask && (version == VERSION_WAN2_2_TI2V || sd_version_is_ltxav(version) || sd_version_is_lingbot_video(version))) {
            noised_input = noised_input * denoise_mask + sampling_init_latent * (1.0f - denoise_mask);
@@ -2629,11 +2646,13 @@ sd::Tensor<float> StableDiffusionGGML::sample(const std::shared_ptr<DiffusionMod
                    unet_extra.ip_scale   = ip_adapter_strength;
                }
                diffusion_params.extra = unet_extra;
-            } else if (sd_version_is_sd3(version)) {
+            } else if (sd_version_is_sd3(version) || version == VERSION_MOSS_IMAGE) {
                diffusion_params.extra = SkipLayerDiffusionExtra{local_skip_layers};
            } else if (sd_version_is_flux(version) || sd_version_is_flux2(version) || sd_version_is_longcat(version) || sd_version_is_sefi_image(version)) {
                diffusion_params.extra = FluxDiffusionExtra{&guidance_tensor,
                                                            local_skip_layers};
+            } else if (sd_version_is_flux3(version)) {
+                diffusion_params.extra = Flux3DiffusionExtra{flux3_target_timestep.empty() ? nullptr : &flux3_target_timestep};
            } else if (version == VERSION_QWEN_IMAGE_2_1) {
                diffusion_params.extra = QwenImage21DiffusionExtra{&condition.c_token_types};
            } else if (sd_version_is_anima(version)) {
@@ -2856,9 +2875,12 @@ sd::Tensor<float> StableDiffusionGGML::sample(const std::shared_ptr<DiffusionMod
        if (uncond_out.empty() && !uncond.empty()) {
            if (!skip_uncond) {
                const std::vector<int>* uncond_skip_layers = nullptr;
+                const std::vector<int> moss_skip_layers    = {22};
                if (is_skiplayer_step && slg_uncond) {
                    LOG_VERBOSE("Skipping layers at uncond step %d\n", step);
                    uncond_skip_layers = &skip_layer_guidance.layers();
+                } else if (version == VERSION_MOSS_IMAGE) {
+                    uncond_skip_layers = &moss_skip_layers;
                }
                uncond_out = run_condition(uncond,
                                           uncond.c_concat.empty() ? nullptr : &uncond.c_concat,
@@ -3180,6 +3202,8 @@ std::string StableDiffusionGGML::get_default_ref_image_preset(SDVersion version)
        return "longcat";
    } else if (sd_version_is_flux(version)) {
        return "flux_kontext";
+    } else if (sd_version_is_flux3(version)) {
+        return "flux2";
    } else if (sd_version_is_flux2(version) || sd_version_is_sefi_image(version)) {
        return "flux2";
    } else if (version == VERSION_QWEN_IMAGE_LAYERED) {
@@ -15,6 +15,7 @@
 #include "model/diffusion/control.hpp"
 #include "model/diffusion/ernie_image.hpp"
 #include "model/diffusion/flux.hpp"
+#include "model/diffusion/flux3.hpp"
 #include "model/diffusion/hidream_o1.hpp"
 #include "model/diffusion/hunyuan.hpp"
 #include "model/diffusion/ideogram4.hpp"
@@ -30,6 +31,7 @@
 #include "model/diffusion/minit2i.hpp"
 #include "model/diffusion/mmdit.hpp"
 #include "model/diffusion/model.hpp"
+#include "model/diffusion/moss_image.hpp"
 #include "model/diffusion/pid.hpp"
 #include "model/diffusion/pixart.hpp"
 #include "model/diffusion/qwen_image.hpp"
@@ -173,6 +175,11 @@ namespace sd::model_builders {
                                                                  version,
                                                                  weight_manager,
                                                                  sd_ctx_params->model_args);
+        } else if (sd_version_is_flux3(version)) {
+            result.conditioner = std::make_shared<LLMEmbedder>(ctx.backends.runtime_backend(SDBackendModule::TE),
+                                                               tensor_storage_map, version, "", false, weight_manager, tokenizers);
+            result.diffusion   = std::make_shared<Flux3::Flux3Runner>(ctx.backends.runtime_backend(SDBackendModule::DIFFUSION),
+                                                                    tensor_storage_map, "model.diffusion_model", weight_manager);
        } else if (sd_version_is_flux2(version) || sd_version_is_sefi_image(version)) {
            bool is_chroma     = false;
            result.conditioner = std::make_shared<LLMEmbedder>(ctx.backends.runtime_backend(SDBackendModule::TE),
@@ -462,6 +469,11 @@ namespace sd::model_builders {
                                                               tensor_storage_map, version, "", false, weight_manager, tokenizers);
            result.diffusion   = std::make_shared<Iris::IrisRunner>(ctx.backends.runtime_backend(SDBackendModule::DIFFUSION),
                                                                  tensor_storage_map, "model.diffusion_model", weight_manager);
+        } else if (version == VERSION_MOSS_IMAGE) {
+            result.conditioner = std::make_shared<LLMEmbedder>(ctx.backends.runtime_backend(SDBackendModule::TE),
+                                                               tensor_storage_map, version, "", false, weight_manager, tokenizers);
+            result.diffusion   = std::make_shared<MOSSImage::MOSSImageRunner>(ctx.backends.runtime_backend(SDBackendModule::DIFFUSION),
+                                                                            tensor_storage_map, "model.diffusion_model", weight_manager);
        } else {  // SD1.x SD2.x SDXL
            std::map<std::string, std::string> embbeding_map;
            for (uint32_t i = 0; i < sd_ctx_params->embedding_count; i++) {
@@ -1452,6 +1452,29 @@ struct FluxFlowDenoiser : public DiscreteFlowDenoiser {
    }
 };

+struct Flux3FlowDenoiser : public FluxFlowDenoiser {
+    float t_to_sigma(float t) override {
+        float u = (t + 1.f) / TIMESTEPS;
+        return shift * u / (1.f + (shift - 1.f) * u);
+    }
+
+    std::vector<float> get_sigmas(uint32_t n, int image_seq_len, scheduler_t scheduler_type, SDVersion version, const char* extra_sample_args = nullptr) override {
+        if (scheduler_type != DISCRETE_SCHEDULER) {
+            return Denoiser::get_sigmas(n, image_seq_len, scheduler_type, version, extra_sample_args);
+        }
+        if (n == 0) {
+            return {};
+        }
+        std::vector<float> sigmas;
+        sigmas.reserve(n + 1);
+        for (uint32_t i = 0; i <= n; ++i) {
+            float u = 1.f - static_cast<float>(i) / n;
+            sigmas.push_back(shift * u / (1.f + (shift - 1.f) * u));
+        }
+        return sigmas;
+    }
+};
+
 struct SefiFlowDenoiser;

 struct SefiFlowDenoiser : public FluxFlowDenoiser {
@@ -10,6 +10,7 @@
 #include <cmath>
 #include <cstdlib>
 #include <cstring>
+#include <filesystem>

 static IMatrixCollector imatrix_collector;

@@ -227,7 +228,7 @@ void IMatrixCollector::save_imatrix(std::string fname, int ncall) const {
        LOG_WARN("storing only %zu out of %zu entries\n", to_store.size(), stats_.size());
    }

-    std::ofstream out(fname, std::ios::binary);
+    std::ofstream out(std::filesystem::u8path(fname), std::ios::binary);
    out.write((const char*)&n_entries, sizeof(n_entries));
    for (const auto& name : to_store) {
        const auto& stat = stats_.at(name);
@@ -251,7 +252,7 @@ void IMatrixCollector::save_imatrix(std::string fname, int ncall) const {
 }

 bool IMatrixCollector::load_imatrix(const char* fname) {
-    std::ifstream in(fname, std::ios::binary);
+    std::ifstream in(std::filesystem::u8path(fname), std::ios::binary);
    if (!in) {
        LOG_ERROR("failed to open %s\n", fname);
        return false;
@@ -4,6 +4,7 @@
 #include <array>
 #include <climits>
 #include <cstdlib>
+#include <filesystem>
 #include <fstream>
 #include <queue>
 #include <sstream>
@@ -495,7 +496,7 @@ struct HFTokenizer::Impl {

 HFTokenizer::HFTokenizer(const std::string& path)
    : impl_(new Impl) {
-    std::ifstream stream(path, std::ios::binary);
+    std::ifstream stream(std::filesystem::u8path(path), std::ios::binary);
    tokenizer_require(stream.good(), "cannot open " + path);
    TokenizerJSON config;
    stream >> config;
Author	SHA1	Message	Date
leejet	68285df319	feat: add moss-image support (#2148 )	2026-10-17 03:38:38 +08:00
leejet	af0fc72c96	chore: sync ggml and remove obsolete conv1d workarounds (#2147 )	2026-10-17 02:39:51 +08:00
therealshulgin	2f016e8e30	fix: handle UTF-8 file paths on Windows (#2145 )	2026-10-17 00:50:42 +08:00
therealshulgin	8c442fb0ad	fix: open model files with UTF-8 paths on Windows (#2144 )	2026-10-17 00:50:09 +08:00
leejet	8cd731e7b4	feat: add FLUX.3 Image Klein and Klein Base support (#2143 )	2026-10-16 02:42:07 +08:00