stable-diffusion.cpp/docs/model_config.md

Model Configuration Conventions

This document describes model configuration structs, weight-based configuration detection, and the definitions used to integrate models into the runtime.

Config Types

Model configuration should live in a model-specific *Config struct.

Examples:

• ZImageConfig
• UNetConfig
• MMDiTConfig
• LLMConfig

Preserve established acronym casing in type names, such as UNet, MMDiT, LLM, VAE, and T5.

Place the config struct near the top of the model header, before the main model blocks and runner types that consume it.

Config Variables

Variables and members that hold a config should be named config.

Examples:

UNetConfig config;
UnetModelBlock unet;

MMDiTRunner(...)
    : DiffusionModelRunner(backend, params_backend, prefix),
      config(MMDiTConfig::detect_from_weights(tensor_storage_map, prefix)),
      mmdit(config) {
}

Avoid alternate names such as params, params_cfg, model_params, or model-specific aliases unless an existing public API requires them.

Weight Detection

If a model can derive configuration from loaded weight metadata, expose that logic as a static method on the config type:

static XxxConfig detect_from_weights(const String2TensorStorage& tensor_storage_map,
                                     const std::string& prefix);

Additional selector arguments are allowed when required by an existing model family, for example SDVersion version or an architecture enum:

static UNetConfig detect_from_weights(const String2TensorStorage& tensor_storage_map,
                                      const std::string& prefix,
                                      SDVersion version = VERSION_SD1);

Use TensorStorage metadata, especially n_dims and ne, to infer shapes. Do not load or parse tensor data for config detection.

Detection should respect prefix. For nested weights, construct full names from prefix + "." + suffix or filter entries with starts_with(name, prefix).

Do not add persistent config fields such as inferred_from_weights only to record whether detection happened. If the function needs to decide whether to print a verbose line, keep that as local control flow inside detect_from_weights.

Logging

When config values are inferred from weights, print one LOG_VERBOSE line at the end of detect_from_weights.

Example:

LOG_VERBOSE("llm: num_layers = %" PRId64 ", vocab_size = %" PRId64 ", hidden_size = %" PRId64 ", intermediate_size = %" PRId64,
            config.num_layers,
            config.vocab_size,
            config.hidden_size,
            config.intermediate_size);

Only print the config detection log when the function actually inferred values from weights. Do not duplicate the same config summary in runner constructors or model loading code.

Use the correct format specifiers for field types, such as %" PRId64 " for int64_t and %d for int.

Runner And Model Responsibilities

Runners should detect the config once and pass it into the model block:

struct XxxRunner : public DiffusionModelRunner {
    XxxConfig config;
    XxxModel model;

    XxxRunner(..., const String2TensorStorage& tensor_storage_map, const std::string prefix)
        : DiffusionModelRunner(backend, params_backend, prefix),
          config(XxxConfig::detect_from_weights(tensor_storage_map, prefix)),
          model(config) {
        model.init(params_ctx, tensor_storage_map, prefix);
    }
};

Model blocks should consume config directly instead of re-scanning weights in their constructors. Keep config-derived behavior centralized in the config struct.

If a model has no weight-derived config today, it may still provide detect_from_weights for API consistency, but it should not print a config detection log unless it actually derives values from weights.

Model Integration Definitions

Every SDVersion has an explicit ModelDefinition in src/model/integrations/. Related variants share an integration file and runner factory. Network blocks and their weight-derived configuration remain under src/model/diffusion/; reusable text encoder implementations remain under src/model/te/. Weight-name conversion remains in src/name_conversion.cpp.

To add a model:

1. Add its identity to SDVersion without renumbering existing values.
2. Implement the network and *Config::detect_from_weights following the conventions above, or reuse an existing runner.
3. Add a definition in its integration file, including its component factory, traits, geometry, and sampling policy.
4. Add the family's explicit registration in src/model_registry.cpp. For a new variant of a registered family, add its definition to that family's list.
5. Add version detection signatures and any required weight-name mappings in src/model_loader.cpp and src/name_conversion.cpp respectively. Update shared conditioning or runtime code only when existing behavior cannot serve the model.

Use the existing family integration as the starting point for a variant. For a new family, src/model/integrations/moss_image.cpp is a compact example of a conditioner/diffusion factory and its definition. Definitions collect model construction and defaults; they do not replace version detection, name conversion, or model-specific inference algorithms.

The registry checks for missing, duplicate, and invalid version registrations. Explicit references keep integrations reachable in static-library builds. The existing CMake glob includes integration sources automatically. No legacy builder, display-name table, or prediction-default fallback needs updating.

Detection And Weight Conversion

Version detection remains in ModelLoader::get_sd_version() in src/model_loader.cpp. Add model signatures there and preserve the order of overlapping checks. Detection uses tensor names and metadata, and can run before and after name conversion, so signatures must work at both stages. Do not load tensor data to identify an architecture. The registry looks up definitions by the detected version; it does not perform detection.

Weight-name mappings and their version-based dispatch remain centralized in src/name_conversion.cpp. Add new mappings there when needed. Inference, GGUF export, quantization, and LoRA conversion share the convert_tensor_name entry point; preserve its prefix normalization and component conversion order.

Components And Conditioning

The core factory receives model_builders::Context, the tokenizer configuration, and the output runners. Backend checks, optional IP-Adapter setup, scale overrides, and tokenizer usage validation remain in the common builder. A factory returns failure if any required additional backend cannot initialize.

LLM architecture selection, tokenizer requirements, prompt preparation, and encoding remain centralized in src/conditioning/conditioner.hpp. Integration factories instantiate the conditioner with the model version; models requiring new LLM conditioning behavior still need changes in that shared implementation.

vae_kind is the single source of truth for the VAE family; compatibility helpers derive their results from it. KL, FLUX, and FLUX2 share the AutoEncoderKL implementation while retaining their distinct latent formats. The related traits have separate responsibilities:

• supports_video_generation describes the model's video generation capability.
• video_tae selects TinyVideoAutoEncoder instead of TinyImageAutoEncoder. Image models can use a video TAE, so this is independent of video generation.
• fake_vae selects FakeVAE for pixel-space models. It is separate from vae_kind, which also controls latent-format compatibility helpers.
• vae_format_override allows PiD's VAE format override.

configure_vae handles model-specific KL adjustments such as PixArt's scale and SDXL's convolution scale. VAE compatibility traits also drive latent transformations and previews through the existing helper functions.

Sampling And Geometry

ModelSamplingPolicy groups the defaults first: default_prediction, default_flow_shift, default_sample_method, and default_scheduler. Set only the values that differ from the struct defaults. prediction_from_weights and flow_shift_from_weights preserve checkpoint-dependent defaults during automatic prediction selection; probe_v_prediction enables SD2's runtime v-prediction probe. Explicit prediction settings take precedence over automatic selection. MOSS's default unconditional skip layer stays in the diffusion engine, with active unconditional SLG taking precedence.

The resolved prediction_t selects the denoiser directly. Flux3, MiniMax H3 AV, and Ming-Image use FLUX3_FLOW_PRED, H3_AV_FLOW_PRED, and MING_IMAGE_FLOW_PRED; their CLI overrides are flux3_flow, h3_av_flow, and ming_image_flow. Explicit sd3_flow always selects DiscreteFlowDenoiser, and flux_flow always selects FluxFlowDenoiser, regardless of model version. Automatic selection retains each model's default implementation. New prediction values are appended before PREDICTION_COUNT, preserving existing valid values; clients using the automatic-selection sentinel must rebuild with the new header.

ModelGeometry declares diffusion_down_factor, latent_channels, and image_channels. ref_image_preset defaults to "default"; set it only for models that need another reference-image preset.

The common pipeline executes these policies; specialized video, multimodal, cache, and extension algorithms remain in their existing runtime components. Model-specific diffusion extra parameters are assembled in the diffusion engine. Video frame conversion and alignment also remain in the diffusion engine. get_vae_scale_factor() retains PiD's pixel-space special case and otherwise uses the VAE's scale factor.

Validate migrations against the previous implementation using synthetic metadata, name conversions, prompt preparation, default/override selection, and focused builds. Real generation checks are still needed to validate numerical behavior when changing network or encoding algorithms.