spirula-studio/docs/codegen.md

Code generation

Generated trees are marked linguist-generated / linguist-vendored in .gitattributes and are committed. A fresh checkout therefore builds with no Python at all — the dev build scripts run codegen when python3 is available and fall back to the committed files when it isn't.

Every generator reads C++/CUDA sources. None reads Python, and none should be added: the training config used to be generated from the Python dataclasses and is now hand-written in src/config/TrainConfig.h.

Generated locations:

src/generated/       device-math headers from Slang
src/backend/api/     backend module forwarders
src/instantiations/  kernel instantiation TUs

All generators run from the repo root:

python3 tools/codegen/generate_headers.py
python3 tools/codegen/generate_kernel_instantiation.py
python3 tools/codegen/generate_backend_api.py
# generate_vulkan_stubs.py takes arguments; see below

The one exception is src/shaders/build.bash, which runs from src/ and needs slangc on PATH. The dev build scripts do NOT run it. Editing anything under src/shaders/ and skipping it leaves the two backends computing different things: Vulkan compiles the Slang, CUDA compiles the committed src/generated/*.cuh.

---

generate_headers.py — .cu → .cuh declarations

Scans the .cu files listed in HEADER_SOURCES for functions preceded by the marker

/*[AutoHeaderGeneratorExport]*/
void my_kernel_launcher(...) { ... }

and rewrites the declaration section of the matching <Name>.cuh.

Invariants:

1. Each .cuh has a splitter line:

   /* == AUTO HEADER GENERATOR - DO NOT EDIT THIS LINE OR ANYTHING BELOW THIS LINE == */
   

   Everything above it is hand-written (includes, types, preamble) and preserved. Everything below is regenerated. Never edit below it.
2. #if 0 blocks are stripped before scanning, so a disabled function is not exported.
3. Sources are listed explicitly. HEADER_SOURCES in the script maps each header stem to the ordered list of .cu files that declare into it. One header may be fed by many sources, and a source file's name is therefore free — name it after what it does. A listed file that does not exist raises FileNotFoundError, so a rename cannot silently drop declarations.
4. A new kernel family needs an entry in HEADER_SOURCES.
5. Declaration sections must stay CUDA-include-free — they must parse under -DSS_BACKEND_VULKAN with no CUDA toolkit present. Use backend/api/BackendTypes.h for vector types.

Splitting a large .cu (the sanctioned workflow)

Because of invariant 3, splitting is nearly free:

# carve the sections out along the // ==== banners, naming each part after
# what it does -- not after the file it came from:
#   PixelWise.cu -> ImageConvert.cu, ImageColorOps.cu, DepthGeometry.cu,
#                   ImageDistort.cu, ImageWarp.cu, GtDepthNormalWarp.cu, Ppisp.cu
# put the shared preamble in <Family>Common.cuh and shared device helpers in a
# descriptively named header (BilinearSample.cuh), then:
#   - add the new files to HEADER_SOURCES['PixelWise']
python3 tools/codegen/generate_headers.py   # PixelWise.cuh's contents are unchanged

No build file changes: both build systems glob the kernel directories through cmake/sources.txt (CMake with CONFIGURE_DEPENDS, so a new file is picked up without a manual reconfigure). No caller changes: the header keeps its name and its declaration set.

Naming rules when splitting:

• Name each part after its function, not its origin — ImageWarp.cu, not PixelWise.Warp.cu.
• <Name>_kernel.cuh is reserved for device bodies consumed by generate_kernel_instantiation.py. A plain shared-helper header gets a descriptive name instead.
• <Family>Common.cuh for the shared preamble (slang namespace includes, host-side view helpers).

---

generate_kernel_instantiation.py — explicit template instantiations

Reads kernel declarations out of src/kernels/**/*.cuh and emits one small TU per (primitive × camera model × SH degree × …) combination into src/instantiations/ (currently 111 files). This keeps each nvcc edge small and parallelizable rather than instantiating everything in one TU.

---

The training config — not generated

src/config/TrainConfig.h is hand-written and is the training config's single source of truth. It holds:

• SS_CONFIG_FIELDS(X) — one X-macro row per flag, (type, member, default, section, tier, choices). The flag's NAME and help sentence are translated and live in src/i18n/catalog/TrainFields.h, keyed by member and resolved at compile time;
• struct TrainConfig — expanded from that same table, so a field cannot exist in one and not the other;
• kTrainPresets + train_apply_preset() — one branch per preset.

The table is expanded by the CLI's generic parser and --help printer, the GUI's "All Options" editor, TrainerCore's config.json dump and the resume path's config.json reader. Add a row and the flag appears in all of them.

The CLI flag is member stringified (--sh-degree sets sh_degree; - and _ are interchangeable), so a flag name cannot drift from its member. The config.json key is the flag, spelled the same way, at the top level: the file is flat. It used to nest under the group column, which made a presentational choice part of an on-disk format — moving a flag to another heading moved its key, and the reader (spirula mesh, --resume) silently fell back to the default when it could not find it. section and tier are display metadata only and can be reshuffled freely.

It is hand-written: nothing generates it, and nothing is generated from it except the struct and the tables SS_CONFIG_FIELDS expands into.

---

generate_backend_api.py — backend module headers

Emits src/backend/api/*.h (Projection.h, Rasterization.h, PixelWise.h, …) as forwarders that #include the relevant per-kernel .cuh headers, grouped by subsystem.

Forwarding rather than copying declarations is deliberate: it keeps one definition of every preamble type, so a TU may include both a module header and an individual .cuh without ODR violations.

---

generate_vulkan_stubs.py — link-probe for unported kernels

python3 tools/codegen/generate_vulkan_stubs.py <build-vulkan-dir> <output.cpp>

Link-probes csrc_portable's objects against libss_backend_vulkan.a, collects undefined C++ symbols, finds each function's declaration in the per-kernel .cuh headers, and emits one throwing definition per function (backend/vulkan/kernels/TrainingStubs.gen.cpp). This lets the portable engine link today and lose stubs one module at a time as ports land.

Symbols with no header declaration (class methods, templates) are listed in the generated file's header comment for manual stubbing in TrainingStubsManual.cpp.

Rerun it whenever the engine gains a new kernel call, or when a port lands — a ported symbol would otherwise be defined twice.

---

Non-script generation done by CMake

• viewer.html → app_generated/viewer_html.h (byte array), so spirula train and the GUI serve the viewer from a self-contained binary. Regenerated at configure time when the HTML changes. Viewer.cpp has a dev override so HTML edits don't require a rebuild.
• Slang → SPIR-V blobs → vk_shaders_embedded.cpp (backend/vulkan/shaders/SpirvShaders.cmake + backend/vulkan/shaders/spirv_tool.cpp). SPIR-V is never committed; one slangc edge per blob, with capability variants (.atomicadd, .noint64, .int8) compiled per entry point.