diff --git a/AGENTS.md b/AGENTS.md index cecc0899..c52a4551 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -175,6 +175,8 @@ src/ │ seeding -> step loop -> eval. Both the CLI and │ the GUI drive this; it lives in the engine │ library (cmake/sources.txt), not the app targets. +│ TrainForecast.{h,cpp} beside it is the run's ETA +│ and VRAM forecast -- docs/notes/train-forecast.md ├── config/ TrainConfig.h — the training config's single source │ of truth: one X-macro row per flag, hand-written. │ TrainConfigJson.h is the one flat-JSON encoding @@ -584,7 +586,10 @@ no ceremony — do not ask, do not leave a note saying you removed it. `SS_POOL_ALIAS_POISON=1` fills the arena at every phase switch so a read that outlives its phase becomes NaNs a parity test catches, and `SS_POOL_ALIAS=0` turns the whole thing off. Read - `docs/notes/vram-splat-x-img.md` before adding a row. + `docs/notes/vram-splat-x-img.md` before adding a row. A buffer whose + length follows the LIVE splat count goes in `POOL_LIVE_SPLAT_TABLE`, so it + is sized for `cap_max` once: the trainer's VRAM forecast assumes only + `splat x img` grows during a run. - **`SS_PROFILE=1`** enables the per-stage backend timing breakdown (H2D / D2H / D2D / memset / device / host), header-only, both backends, plus a per-category VRAM breakdown after any run that trained. What the biggest diff --git a/cmake/sources.txt b/cmake/sources.txt index 3274c819..9017a8bc 100644 --- a/cmake/sources.txt +++ b/cmake/sources.txt @@ -39,6 +39,7 @@ src/data/*.cpp src/data/parsers/*.cpp src/app/webviewer/*.cpp src/app/TrainerCore.cpp +src/app/TrainForecast.cpp src/app/EvalMetrics.cpp src/external/*.cpp src/external/*.c @@ -55,6 +56,7 @@ src/data/*.cpp src/data/parsers/*.cpp src/app/webviewer/*.cpp src/app/TrainerCore.cpp +src/app/TrainForecast.cpp src/app/EvalMetrics.cpp src/external/*.cpp src/external/*.c diff --git a/docs/notes/train-forecast.md b/docs/notes/train-forecast.md new file mode 100644 index 00000000..a9ed9450 --- /dev/null +++ b/docs/notes/train-forecast.md @@ -0,0 +1,99 @@ +# Training ETA and VRAM forecast + +`app/TrainForecast.{h,cpp}` predicts, while a run trains, how long it has left +and how much device memory it will peak at. `TrainerSession::train()` feeds it +one sample per step; the CLI's progress line, `/progress`, the GUI's status +strip and the VRAM bar's hover card read it. `train_forecast_test` checks both +against synthetic runs whose answers are known, and `SS_FORECAST_LOG=1` prints +its state every 100 steps for checking it against a real run. + +Both models lean on the same fact: until the model reaches `cap_max`, the +splat count still to come is **known**. `engine_densify_step` grows it by +`densify_grows_at` / `densify_target` (`engine/EngineConfig.h`), and +`SplatSchedule` replays that same rule forward. The SH degree schedule +(`min(step / sh_degree_warmup_every, sh_degree)`) is known the same way. So +nothing extrapolates the count or the degree -- only what they cost. + +## ETA + +A step costs `a + (b0 + b1 * k) * N^gamma`: `N` in millions of splats, `k` +the SH coefficients the step trains (0, 3, 8, 15), `a` everything that does +not scale with the model (loss, bilateral grids, PPISP, data loading, host +overhead). Per window of `refine_every` steps -- so every window holds the +same share of densify steps -- a Kalman filter updates `[a, b0, b1, c]` from: + +- the window's wall time per step, outliers past 3x the median dropped (the + pipeline builds when the SH degree steps up are one-offs); +- the GPU time of the splat stages, `c + (b0 + b1 * k) * N^gamma`. + `SplatStageTimer` (`engine/EngineStepTiming.cpp`) brackets the forward, the + raster / projection backward, the optimizer and densify with timestamp + events on one step in ten; on Vulkan every bracket is a queue submission. + +Those stages are NOT all per-splat: rasterization has a large per-pixel part, +~45% of their time on bonsai at 1/4 resolution and nearly all of it at 1/2. +Treating their time as `b * N` overshot the ETA there by +130%. Hence the +intercept `c`: the GPU measurement pins the slope once the count moves (it +has less host noise than the wall clock), and says nothing about it before. +While the count holds still the slope is a prior -- half the splat stages' +time, +-70% -- and the ETA's sigma says so. + +No single `gamma` fits: whole-run fits gave 1.0 on bonsai at 1/4 resolution, +0.85 at 1/2 and 0.8 on garden. So three filters run, at 0.75, 0.875 and 1.0, +each scored on how well it predicted each window before seeing it +(discounted by 0.9 a window), and the ETA is the weighted mixture. Updates +are gated at 2 sigma with a 10% floor: the laptop GPU these were measured on +ran whole 100-step windows 40-60% slow, four times in one run. + +The ETA also adds the checkpoint saves still to come, at the seconds per +million splats the saves so far took; a save is excluded from its step's +wall time for that reason. + +Replayed on three recorded runs, error in the time left: + +| step | 300 | 1000 | 2000 | 3000 | 4000 | 5000 | +|---|---|---|---|---|---|---| +| bonsai 1/4, naive (last 100 steps) | -66% | -65% | -59% | -50% | -29% | +3% | +| bonsai 1/4, model | -25% | -14% | -10% | -12% | -4% | +4% | +| bonsai 1/2, naive | -62% | -63% | -61% | -54% | -43% | -36% | +| bonsai 1/2, model | +27% | +55% | +16% | -10% | -3% | -1% | +| garden 1/4, naive | -23% | -25% | -23% | -30% | -15% | -24% | +| garden 1/4, model | +34% | +22% | +23% | +30% | 0% | -3% | + +The naive estimate is biased low everywhere the count still grows; the model +is not biased, and its early error is what the prior cannot know. + +## VRAM + +Only `splat x img` grows during training (docs/notes/vram-splat-x-img.md), +so ours at a future step is everything else now plus the growth, projected. +Two measurements decide what "the growth" is: + +- It is read off the process **high-water**, not the pool. A step that + overflows the alias arena holds private buffers beside it until the next + phase switch, so the real peak sits above the steady cap: 179 against 156 + MiB on bonsai at 1/4, from one step near the end. +- The first densify step allocates `splat x img` scratch of its own -- 10 -> + 55 MiB on garden, count unchanged -- so the fit starts after it. Before + that the forecast is provisional: growth all per splat, +-70%, plus 15% +- + 8% of the splat category for the live-splat scratch the step reserves + (14.9% measured on bonsai). + +The growth is `g0 + g1 * N`, a Kalman filter over `[g0, g1]` from one +high-water measurement per window, starting at `g0 = 0`: fitting a fixed +chunk from the first windows under-read bonsai's peak by 14 sigma, while +putting it all per splat over-reads garden's (its 55 MiB chunk) by up to +36% -- the safe side. Two terms widen the projection: the expected largest +per-step demand over the draws still to come at each count, minus what a +window already saw (the pool keeps the largest draw, and a view that sees +more of the model sets it); and 0.3 relative sigma per e-fold the count is +extrapolated. + +"Others" is device use minus ours (on CUDA that includes the context), an +exponential average with its spread. The chance of running out is the normal +tail of `peak + others` above 99% of the device -- the last percent is +fragmentation. Low below 10%, medium below 50%, high above; the trainer logs a +warning the first time the risk reaches each of medium and high. + +On the three runs, from the first densify step on, the projected peak stayed +within 2 sigma or above the truth, by at most 8% (bonsai 1/4), 10% (bonsai +1/2) and 36% (garden) of it. diff --git a/docs/notes/vram-splat-x-img.md b/docs/notes/vram-splat-x-img.md index 5ced0007..b99abb6d 100644 --- a/docs/notes/vram-splat-x-img.md +++ b/docs/notes/vram-splat-x-img.md @@ -15,6 +15,21 @@ Three numbers describe a step, and every buffer below is one of them: | `nnz` | pairs that survive the visibility test | 15.1M | | `n_isects` | (splat, tile) pairs the raster consumes | 22M - 36M | +## Only this category grows + +Splat buffers are sized for `cap_max`, so everything else a run allocates is +fixed once the first densify step has run -- which is what lets the trainer +forecast its peak from this category alone. Scratch requested per LIVE splat +(the densify draws, `raster_bwd.accum_weight`, `fused_proj_bwd.cam_bounds`) +is Splat too: `POOL_LIVE_SPLAT_TABLE` in `core/PoolSlots.h` makes the pool +allocate it for `cap_max` on first use, scaling the request by cap / live, and +`POOL_SPLAT_DRAW_TABLE` gives the two index lists whose length is a draw count +`cap_max` entries outright. Measured on bonsai at 1/4 resolution, 228k -> 666k +splats: splat stays at 1042.6 MiB (revised) / 1019.7 MiB (MCMC); without the +tables it climbs 11 MiB over the same run and `other` another 15 MiB. A new buffer whose length +follows the live count goes in one of those two tables, or the forecast +under-reads the peak. + ## What each buffer costs Per element, after the 2026-08-30 pass: diff --git a/src/app/TrainForecast.cpp b/src/app/TrainForecast.cpp new file mode 100644 index 00000000..eb69bfd5 --- /dev/null +++ b/src/app/TrainForecast.cpp @@ -0,0 +1,520 @@ +// TrainForecast.cpp -- see TrainForecast.h and docs/notes/train-forecast.md. + +#include "app/TrainForecast.h" + +#include +#include + +namespace spirula { + +namespace { + +// One scalar measurement y = h . x with variance r. `gate` > 0 inflates r +// until the innovation is at most `gate` sigma, so one bad window nudges the +// state instead of yanking it. +template +void kf_update(double (&x)[D], double (&P)[D][D], const double (&h)[D], double y, + double r, double gate = 0.0) { + double ph[D]; + double s = r, pred = 0.0; + for (int i = 0; i < D; ++i) { + ph[i] = 0.0; + for (int j = 0; j < D; ++j) ph[i] += P[i][j] * h[j]; + s += h[i] * ph[i]; + pred += h[i] * x[i]; + } + if (!(s > 0.0)) return; + const double innov = y - pred; + if (gate > 0.0 && innov * innov > gate * gate * s) s = innov * innov / (gate * gate); + for (int i = 0; i < D; ++i) x[i] += ph[i] / s * innov; + for (int i = 0; i < D; ++i) + for (int j = 0; j < D; ++j) P[i][j] -= ph[i] * ph[j] / s; +} + +template +double quad(const double (&P)[D][D], const double (&h)[D]) { + double q = 0.0; + for (int i = 0; i < D; ++i) + for (int j = 0; j < D; ++j) q += h[i] * P[i][j] * h[j]; + return std::max(0.0, q); +} + +double norm_cdf(double z) { return 0.5 * std::erfc(-z / std::sqrt(2.0)); } +double norm_pdf(double z) { + return std::exp(-0.5 * z * z) / std::sqrt(2.0 * 3.14159265358979323846); +} + +// Acklam's rational approximation, relative error < 1.2e-9. +double norm_quantile(double p) { + static const double a[] = {-3.969683028665376e+01, 2.209460984245205e+02, + -2.759285104469687e+02, 1.383577518672690e+02, + -3.066479806614716e+01, 2.506628277459239e+00}; + static const double b[] = {-5.447609879822406e+01, 1.615858368580409e+02, + -1.556989798598866e+02, 6.680131188771972e+01, + -1.328068155288572e+01}; + static const double c[] = {-7.784894002430293e-03, -3.223964580411365e-01, + -2.400758277161838e+00, -2.549732539343734e+00, + 4.374664141464968e+00, 2.938163982698783e+00}; + static const double d[] = {7.784695709041462e-03, 3.224671290700398e-01, + 2.445134137142996e+00, 3.754408661907416e+00}; + p = std::min(std::max(p, 1e-12), 1.0 - 1e-12); + if (p < 0.02425) { + const double q = std::sqrt(-2.0 * std::log(p)); + return (((((c[0] * q + c[1]) * q + c[2]) * q + c[3]) * q + c[4]) * q + c[5]) / + ((((d[0] * q + d[1]) * q + d[2]) * q + d[3]) * q + 1.0); + } + if (p > 1.0 - 0.02425) return -norm_quantile(1.0 - p); + const double q = p - 0.5, r = q * q; + return (((((a[0] * r + a[1]) * r + a[2]) * r + a[3]) * r + a[4]) * r + a[5]) * q / + (((((b[0] * r + b[1]) * r + b[2]) * r + b[3]) * r + b[4]) * r + 1.0); +} + +// Expected largest of n standard normals (Blom's plotting position). +double expected_max(double n) { + if (n <= 1.0) return 0.0; + return norm_quantile((n - 0.375) / (n + 0.25)); +} + +// max(c, X) for X ~ N(mu, sigma^2), moment-matched back to a normal. +void max_with(double mu, double sigma, double c, double& mean, double& sd) { + if (sigma <= 0.0) { mean = std::max(mu, c); sd = 0.0; return; } + const double d = (mu - c) / sigma; + const double P = norm_cdf(d), phi = norm_pdf(d); + mean = mu * P + sigma * phi + c * (1.0 - P); + const double m2 = (mu * mu + sigma * sigma) * P + (mu + c) * sigma * phi + + c * c * (1.0 - P); + sd = std::sqrt(std::max(0.0, m2 - mean * mean)); +} + +// Mean and its variance, over the samples within 3x the median: a pipeline +// build when the SH degree steps up is a one-off, not what later steps cost. +void robust_mean(const std::vector& v, double& mean, double& var) { + std::vector s = v; + std::nth_element(s.begin(), s.begin() + s.size() / 2, s.end()); + const double cut = 3.0 * s[s.size() / 2]; + double sum = 0.0, sq = 0.0; + int k = 0; + for (double x : v) + if (x <= cut) { sum += x; sq += x * x; ++k; } + mean = sum / k; + var = std::max(0.0, sq / k - mean * mean) / k; +} + +// Live-splat scratch the first densify step allocates for cap_max, as a +// share of the Splat category before it: 14.9% on bonsai (revised, SH 3). +constexpr double kFirstDensifySplatShare = 0.15; +constexpr double kFirstDensifySplatSigma = 0.08; + +// Pipeline creation and first-touch allocation: bonsai's first 100 steps ran +// 17% slower than the next 500 at the same splat count. +constexpr int kWarmupSteps = 50; + +// How the splat stages' cost scales with the count. 1.0 fit bonsai at 1/4 +// resolution, 0.85 bonsai at 1/2 and 0.8 garden at 1/4; each run weighs them. +constexpr double kGammas[3] = {0.75, 0.875, 1.0}; + +// Picked by replaying bonsai at 1/4 and 1/2 and garden at 1/4 resolution; +// docs/notes/train-forecast.md has the errors each one bought. +constexpr double kTimeFloor = 0.1; // wall-time noise floor, per window +constexpr double kTimeDrift = 0.05; // state drift, per window +constexpr double kGate = 2.0; // innovations past this many sigma shrink +constexpr double kForget = 0.9; // per window, of each exponent's score +// Growth starts all per splat (g0 = 0): fitting a fixed chunk under-read +// bonsai's peak by 14 sigma. +constexpr double kMemFloor = 0.05; // high-water noise floor +constexpr double kMemDrift = 0.02; // [g0, g1] drift, per window +constexpr double kChunkSigma = 0.5; // prior sigmas, as shares of the growth +constexpr double kPerSplatSigma = 0.7; +constexpr double kReach = 0.3; // extra sigma per e-fold of extrapolation + +} // namespace + +// ================ +// SplatSchedule +// ================ + +int64_t SplatSchedule::final_count(int from, int64_t live) const { + int64_t last = live; + segments(from, live, [&](int, int, int64_t n) { last = n; }); + return last; +} + +int SplatSchedule::first_densify(int from) const { + if (_d.refine_every <= 0) return -1; + int g = std::max(from, _d.refine_start_iter + 1); + g = (g + _d.refine_every - 1) / _d.refine_every * _d.refine_every; + return densify_grows_at(_d, g, _total) ? g : -1; +} + +// ================ +// TrainForecast: time +// ================ + +void TrainForecast::reset(const ForecastSetup& s) { + std::lock_guard lk(_mu); + _setup = s; + const int every = s.schedule.refine_every(); + _window_len = std::min(500, std::max(20, every > 0 ? every : 100)); + _warmup_left = kWarmupSteps; + _tw = Window{}; + _time_init = false; + for (int i = 0; i < 3; ++i) { + _tm[i] = TimeModel{}; + _tm[i].gamma = kGammas[i]; + } + _fallback_wall = 0.0; + _fallback_n = 0; + _save_per_msplat = 0.0; + _saves = 0; + _mw = MemWindow{}; + _mem_init = false; + _demand_sd = 0.0; + _ours_hw = _grow_hw = 0.0; + _history.clear(); + _projection.clear(); + _history_stride = 1; + _history_skip = 0; + _others_mean = _others_sq = 0.0; + _others_n = 0; + _vram = VramForecast{}; +} + +int TrainForecast::sh_coeffs(int step) const { + const int every = _setup.sh_degree_every; + const int d = every > 0 ? std::min(step / every, _setup.sh_degree) : _setup.sh_degree; + return (d + 1) * (d + 1) - 1; +} + +void TrainForecast::add_step(int step, double wall_s, int64_t splats, + double splat_gpu_s) { + std::lock_guard lk(_mu); + // The first step builds the pipelines: seconds, not milliseconds. + if (_warmup_left < kWarmupSteps) { + _fallback_wall += wall_s; + ++_fallback_n; + } + if (_warmup_left > 0) { --_warmup_left; return; } + _tw.wall.push_back(wall_s); + _tw.gpu.push_back(splat_gpu_s); + _tw.msplats.push_back((double)splats / 1e6); + _tw.coeffs.push_back(sh_coeffs(step)); + if ((int)_tw.wall.size() >= _window_len) close_time_window(); +} + +void TrainForecast::close_time_window() { + Window w; + std::swap(w, _tw); + double y = 0.0, var_y = 0.0, g = 0.0, var_g = 0.0; + robust_mean(w.wall, y, var_y); + // Laptop GPUs change clocks for whole windows at a time: bonsai at 1/2 + // resolution ran 40-60% slow for 100 steps at a stretch, four times. + var_y += (kTimeFloor * y) * (kTimeFloor * y); + std::vector timed; + for (double v : w.gpu) + if (v >= 0.0) timed.push_back(v); + const bool has_gpu = timed.size() >= 2; + if (has_gpu) { + robust_mean(timed, g, var_g); + var_g += (0.05 * g) * (0.05 * g); + } + for (TimeModel& t : _tm) + update_time_model(t, w, y, var_y, g, var_g, has_gpu, !_time_init); + _time_init = true; +} + +void TrainForecast::update_time_model(TimeModel& t, const Window& w, double y, + double var_y, double g, double var_g, + bool timed, bool first) { + double n = 0.0, nk = 0.0; + for (size_t i = 0; i < w.msplats.size(); ++i) { + const double load = std::pow(w.msplats[i], t.gamma); + n += load; + nk += load * w.coeffs[i]; + } + n = std::max(1e-9, n / w.msplats.size()); + nk /= w.msplats.size(); + if (first) { + // Until the count moves, nothing measures how cost scales with it. + // Half the splat stages' GPU time is the prior, +-70%: their per-pixel + // share measured ~45% at 1/4 resolution and ~100% at 1/2 on bonsai. + const double splat_part = timed ? 0.5 * std::min(g, y) : 0.5 * y; + t.x[1] = splat_part / n; + // The full SH band (15 coefficients) doubling the per-splat cost, + // +-100%: whole-run fits on bonsai gave +127% at 1/4 resolution. + t.x[2] = t.x[1] / 15.0; + t.x[0] = y - splat_part; + t.x[3] = timed ? g - splat_part : 0.0; + for (auto& row : t.P) for (double& p : row) p = 0.0; + t.P[0][0] = (0.5 * y) * (0.5 * y); + t.P[1][1] = (0.7 * t.x[1]) * (0.7 * t.x[1]); + t.P[2][2] = t.x[2] * t.x[2]; + t.P[3][3] = (0.5 * g) * (0.5 * g) + 1e-12; + } else { + // Drift: splats shrink and the screen footprint changes as training runs. + for (int i = 0; i < 4; ++i) t.P[i][i] += (kTimeDrift * t.x[i]) * (kTimeDrift * t.x[i]) + 1e-12; + // Each exponent is scored on how well it predicted this window's wall + // time before seeing it; the clamp keeps one bad window from deciding. + const double pred = t.x[0] + t.x[1] * n + t.x[2] * nk; + const double s = quad(t.P, {1.0, n, nk, 0.0}) + var_y; + const double z2 = std::min((y - pred) * (y - pred) / s, 4.0 * kGate * kGate); + t.score = kForget * t.score - 0.5 * (z2 + std::log(s)); + } + kf_update(t.x, t.P, {1.0, n, nk, 0.0}, y, var_y, kGate); + // The splat stages' GPU time has less host noise than the wall clock, so + // it pins the slope sooner once the count moves -- through its own + // intercept c, the per-pixel work inside those same stages. + if (timed) kf_update(t.x, t.P, {0.0, n, nk, 1.0}, g, var_g, kGate); + for (double& v : t.x) v = std::max(0.0, v); +} + +void TrainForecast::add_save(double seconds, int64_t splats) { + std::lock_guard lk(_mu); + if (splats <= 0) return; + const double per = seconds / ((double)splats / 1e6); + ++_saves; + _save_per_msplat += (per - _save_per_msplat) / _saves; +} + +EtaForecast TrainForecast::eta(int next_step, int64_t live) const { + std::lock_guard lk(_mu); + EtaForecast out; + const SplatSchedule& sch = _setup.schedule; + const int T = sch.total_steps(); + if (next_step >= T) { out.seconds = out.sigma = 0.0; return out; } + + const int sps = _setup.steps_per_save; + const int every = _setup.sh_degree_every; + double steps = 0.0, save_msplat = 0.0; + double load[3] = {0, 0, 0}, load_k[3] = {0, 0, 0}; + int64_t last = live; + sch.segments(next_step, live, [&](int f, int e, int64_t n) { + const double m = (double)n / 1e6; + steps += e - f; + for (int i = 0; i < 3; ++i) { + const double l = std::pow(m, _tm[i].gamma); + load[i] += (e - f) * l; + for (int t = f; t < e;) { + const int end = every > 0 && t / every < _setup.sh_degree + ? std::min(e, (t / every + 1) * every) : e; + load_k[i] += (end - t) * l * sh_coeffs(t); + t = end; + } + } + if (sps > 0) { + const int lo = std::max(f, 1); + if (e > lo) save_msplat += ((e - 1) / sps - (lo - 1) / sps) * m; + } + last = n; + }); + if (sps != 0) save_msplat += (double)last / 1e6; + + if (_time_init) { + double best = _tm[0].score; + for (const TimeModel& t : _tm) best = std::max(best, t.score); + double wsum = 0.0, mean = 0.0, second = 0.0; + for (int i = 0; i < 3; ++i) { + const TimeModel& t = _tm[i]; + const double wi = std::exp(t.score - best); + const double sec = t.x[0] * steps + t.x[1] * load[i] + t.x[2] * load_k[i]; + const double var = quad(t.P, {steps, load[i], load_k[i], 0.0}); + wsum += wi; + mean += wi * sec; + second += wi * (var + sec * sec); + } + out.seconds = mean / wsum; + out.sigma = std::sqrt(std::max(0.0, second / wsum - out.seconds * out.seconds)); + } else if (_fallback_n > 0) { + out.seconds = _fallback_wall / _fallback_n * steps; + } else { + return out; + } + if (_saves > 0) out.seconds += _save_per_msplat * save_msplat; + return out; +} + +// ================ +// TrainForecast: VRAM +// ================ + +void TrainForecast::add_memory(const MemorySample& m) { + std::lock_guard lk(_mu); + const int sxi = (int)VramCategory::SplatXImg; + if (m.has_used && m.has_process) { + const double o = (double)m.used_bytes - (double)m.process_bytes; + const double a = _others_n == 0 ? 1.0 : 0.05; + _others_mean += a * (o - _others_mean); + _others_sq += a * (o * o - _others_sq); + ++_others_n; + } + if (m.splats_ran > 0 && m.pool_used[sxi] > 0) { + const double lr = std::log((double)m.pool_used[sxi] / (double)m.splats_ran); + _mw.log_ratio += lr; + _mw.log_ratio_sq += lr * lr; + } + // What grows is measured off the process high-water, not the pool: a step + // that overflows the alias arena holds private buffers beside it for a + // while, and that transient -- not the steady cap -- is the real peak. + double pool = (double)m.scratch; + for (size_t c : m.pool_cap) pool += (double)c; + const double ours = m.has_process ? (double)m.process_bytes : pool; + const double steady = (double)m.pool_cap[sxi] + (double)m.scratch; + _ours_hw = std::max(_ours_hw, ours); + _grow_hw = std::max(steady, _ours_hw - (ours - steady)); + if (++_mw.steps >= _window_len) close_mem_window(m); + refresh_vram(m); +} + +void TrainForecast::close_mem_window(const MemorySample& m) { + const MemWindow w = _mw; + _mw = MemWindow{}; + if (w.steps >= 2) { + const double mean = w.log_ratio / w.steps; + const double sd = + std::sqrt(std::max(0.0, w.log_ratio_sq / w.steps - mean * mean)); + _demand_sd = _demand_sd > 0.0 ? 0.7 * _demand_sd + 0.3 * sd : sd; + } + // The first densify step allocates splat x img scratch of its own -- 10 + // -> 55 MiB on garden with the count unchanged -- so nothing before it + // says how the rest grows. + const int first = _setup.schedule.first_densify(_setup.start_step); + if (_grow_hw <= 0.0 || m.splats_ran <= 0 || (first >= 0 && m.step <= first)) + return; + const double n = (double)m.splats_ran / 1e6; + if (!_mem_init) { + _g[0] = 0.0; + _g[1] = _grow_hw / n; + _G[0][0] = (kChunkSigma * _grow_hw) * (kChunkSigma * _grow_hw); + _G[1][1] = (kPerSplatSigma * _g[1]) * (kPerSplatSigma * _g[1]); + _G[0][1] = _G[1][0] = 0.0; + _mem_init = true; + } else { + _G[0][0] += (kMemDrift * _g[0]) * (kMemDrift * _g[0]); + _G[1][1] += (kMemDrift * _g[1]) * (kMemDrift * _g[1]); + } + kf_update(_g, _G, {1.0, n}, _grow_hw, (kMemFloor * _grow_hw) * (kMemFloor * _grow_hw), 3.0); + _g[0] = std::max(0.0, _g[0]); + _g[1] = std::max(0.0, _g[1]); +} + +void TrainForecast::refresh_vram(const MemorySample& m) { + VramForecast v; + const int nc = (int)VramCategory::Count; + const int sxi = (int)VramCategory::SplatXImg; + double pool_total = (double)m.scratch; + for (int c = 0; c < nc; ++c) { + v.category[c] = (double)m.pool_cap[c]; + pool_total += (double)m.pool_cap[c]; + } + v.scratch = (double)m.scratch; + const double ours = m.has_process ? (double)m.process_bytes : pool_total; + v.unpooled = std::max(0.0, ours - pool_total); + v.ours_bytes = ours; + v.total_bytes = m.has_total ? (double)m.total_bytes : 0.0; + if (_others_n > 0) { + v.others_bytes = std::max(0.0, _others_mean); + v.others_sigma = std::max( + std::sqrt(std::max(0.0, _others_sq - _others_mean * _others_mean)), + 0.01 * v.total_bytes); + } + + if (_history_skip++ % _history_stride == 0) + _history.push_back({m.step, ours, v.others_bytes}); + if (_history.size() > 1024) { + std::vector half; + for (size_t i = 0; i < _history.size(); i += 2) half.push_back(_history[i]); + _history.swap(half); + _history_stride *= 2; + } + + const double fixed = ours - (v.category[sxi] + v.scratch); + const SplatSchedule& sch = _setup.schedule; + const int next = m.step + 1; + const int first = sch.first_densify(_setup.start_step); + v.provisional = first >= next; + const double extra_mean = + v.provisional ? kFirstDensifySplatShare * v.category[(int)VramCategory::Splat] : 0.0; + const double extra_sd = + v.provisional ? kFirstDensifySplatSigma * v.category[(int)VramCategory::Splat] : 0.0; + + if (next >= sch.total_steps() || m.splats_ran <= 0 || _grow_hw <= 0.0) { + v.peak_mean = _ours_hw; + v.valid = next >= sch.total_steps(); + _projection.clear(); + _vram = std::move(v); + return; + } + + // Before the filter starts: the prior it will start from. + const double n_now = (double)m.splats_ran / 1e6; + double g[2] = {_g[0], _g[1]}; + double G[2][2] = {{_G[0][0], _G[0][1]}, {_G[1][0], _G[1][1]}}; + if (!_mem_init) { + g[0] = 0.0; + g[1] = _grow_hw / n_now; + G[0][0] = G[0][1] = G[1][0] = 0.0; + G[1][1] = (kPerSplatSigma * g[1]) * (kPerSplatSigma * g[1]); + } + + const double C = std::max(1, _setup.distinct_batches); + const double seen = expected_max(std::min(_window_len, C)); + double hw_mean = _grow_hw, hw_sd = 0.0; + auto band = [&](int step, double g_mean, double g_sd) { + const bool extra = v.provisional && step >= first; + const double mean = fixed + g_mean + (extra ? extra_mean : 0.0); + const double sd = std::sqrt(g_sd * g_sd + (extra ? extra_sd * extra_sd : 0.0)); + v.projection.push_back({step, mean, sd}); + }; + band(next, hw_mean, 0.0); + sch.segments(next, m.splats_next, [&](int f, int e, int64_t count) { + const double n = (double)count / 1e6; + const double per_splat = g[1] * n; + // The pool keeps the largest draw: more draws at one count, larger peak. + const double corr = std::max( + 0.0, _demand_sd * (expected_max(std::min(e - f, C)) - seen)); + // Bytes per splat drift as splats shrink; the further the count is + // extrapolated, the less the fit so far says. + const double reach = kReach * std::log(n / n_now) * per_splat; + const double g_mean = g[0] + per_splat * (1.0 + corr); + const double g_sd = std::sqrt(quad(G, {1.0, n}) + reach * reach + + 0.25 * corr * corr * per_splat * per_splat); + double mean, sd; + max_with(g_mean, g_sd, hw_mean, mean, sd); + const double keep = hw_sd > 0.0 + ? 1.0 - norm_cdf((g_mean - hw_mean) / std::max(g_sd, 1.0)) : 0.0; + sd = std::sqrt(sd * sd + keep * keep * hw_sd * hw_sd); + if (f > next) band(f, hw_mean, hw_sd); + hw_mean = mean; + hw_sd = sd; + band(e - 1, hw_mean, hw_sd); + }); + + v.valid = true; + v.grow_peak_mean = hw_mean; + v.peak_mean = std::max(_ours_hw, v.projection.back().mean); + v.peak_sigma = v.projection.back().sigma; + if (v.total_bytes > 0.0) { + // Allocator fragmentation keeps the last percent out of reach. + const double limit = 0.99 * v.total_bytes - v.others_bytes; + const double sd = std::sqrt(v.peak_sigma * v.peak_sigma + + v.others_sigma * v.others_sigma); + v.p_oom = sd > 0.0 ? 1.0 - norm_cdf((limit - v.peak_mean) / sd) + : (v.peak_mean > limit ? 1.0 : 0.0); + v.risk = v.p_oom < 0.1 ? OomRisk::Low + : v.p_oom < 0.5 ? OomRisk::Medium : OomRisk::High; + } + _projection.swap(v.projection); + v.projection.clear(); + _vram = std::move(v); +} + +VramForecast TrainForecast::vram(bool series) const { + std::lock_guard lk(_mu); + VramForecast out = _vram; + if (series) { + out.history = _history; + out.projection = _projection; + } + return out; +} + +} // namespace spirula diff --git a/src/app/TrainForecast.h b/src/app/TrainForecast.h new file mode 100644 index 00000000..2ba2462f --- /dev/null +++ b/src/app/TrainForecast.h @@ -0,0 +1,196 @@ +#pragma once + +// TrainForecast -- where a training run is heading: the wall time it has left +// and the VRAM it will peak at, fitted online from the steps already run and +// the densify schedule still to come. The models and the measurements behind +// their constants: docs/notes/train-forecast.md. + +#include "core/PoolSlots.h" +#include "engine/EngineConfig.h" + +#include +#include +#include +#include +#include + +namespace spirula { + +// The splat count each step trains with, replayed from the densify schedule. +class SplatSchedule { +public: + SplatSchedule() = default; + SplatSchedule(const DensifyConfig& d, int total_steps, int64_t cap) + : _d(d), _total(total_steps), _cap(cap) {} + + // f(first, end, n): steps [first, end) run with n splats, covering + // [from, total_steps) given `live` splats entering step `from`. + template + void segments(int from, int64_t live, F&& f) const; + + int64_t final_count(int from, int64_t live) const; + // The first densify step >= from, or -1. It runs even at the cap: it + // relocates, and allocates the live-splat scratch. + int first_densify(int from) const; + int total_steps() const { return _total; } + int refine_every() const { return _d.refine_every; } + int64_t cap() const { return _cap; } + +private: + DensifyConfig _d; + int _total = 0; + int64_t _cap = 0; +}; + +enum class OomRisk : uint8_t { Unknown, Low, Medium, High }; + +struct EtaForecast { + double seconds = -1.0; // < 0 while unknown + double sigma = -1.0; +}; + +// All byte figures are device memory. "Ours" is this process; "others" is +// everything else on the device, which includes the backend's own context. +struct VramForecast { + bool valid = false; + // No densify step has run yet, so the live-splat scratch the first one + // allocates is an estimate rather than a measurement. + bool provisional = false; + double total_bytes = 0.0; + double ours_bytes = 0.0; + double others_bytes = 0.0; + double others_sigma = 0.0; + + struct Sample { int step; double ours; double others; }; + std::vector history; + struct Band { int step; double mean; double sigma; }; + std::vector projection; // ours, from the next step to the end + + double peak_mean = 0.0; // ours at the last step + double peak_sigma = 0.0; + double p_oom = 0.0; // chance ours + others exceeds the device + OomRisk risk = OomRisk::Unknown; + + // Ours now, split by pool category, plus scratch and what the pool does + // not account for; and the splat x img share projected at the peak. + std::array category{}; + double scratch = 0.0; + double unpooled = 0.0; + double grow_peak_mean = 0.0; +}; + +struct ForecastSetup { + SplatSchedule schedule; + int start_step = 0; + int steps_per_save = 0; // 0: never saves; < 0: only the final one + int distinct_batches = 1; // views a step can draw, for the peak draw + int sh_degree = 0; // the step's SH degree is + int sh_degree_every = 0; // min(step / sh_degree_every, sh_degree) +}; + +struct MemorySample { + int step = 0; + int64_t splats_ran = 0; // what the step trained + int64_t splats_next = 0; // what the next one will, after densify + std::array pool_used{}; + std::array pool_cap{}; + size_t scratch = 0; + bool has_process = false, has_used = false, has_total = false; + uint64_t process_bytes = 0, used_bytes = 0, total_bytes = 0; +}; + +class TrainForecast { +public: + void reset(const ForecastSetup& s); + + // A finished step: its wall time without any checkpoint save, the splats + // it ran with, and the GPU seconds of its splat-proportional stages + // (< 0 when not timed). + void add_step(int step, double wall_s, int64_t splats, double splat_gpu_s); + void add_save(double seconds, int64_t splats); + void add_memory(const MemorySample& m); + + // Time left from `next_step` with `live` splats entering it. + EtaForecast eta(int next_step, int64_t live) const; + // Without the history and projection series when `series` is false. + VramForecast vram(bool series = true) const; + +private: + struct Window { + std::vector wall, gpu; // gpu < 0: step not timed + std::vector msplats; + std::vector coeffs; + }; + // Step time = a + (b0 + b1 * sh_coeffs) * msplats^gamma, and the splat + // stages' GPU time = c + the same term: a Kalman filter over + // [a, b0, b1, c], one per exponent, weighted by how well each predicted. + struct TimeModel { + double gamma = 1.0; + double x[4] = {0, 0, 0, 0}; + double P[4][4] = {}; + double score = 0.0; // discounted log-likelihood + }; + struct MemWindow { + int steps = 0; + double log_ratio = 0.0, log_ratio_sq = 0.0; + }; + + int sh_coeffs(int step) const; + void close_time_window(); + void update_time_model(TimeModel& t, const Window& w, double y, double var_y, + double g, double var_g, bool timed, bool first); + void close_mem_window(const MemorySample& m); + void refresh_vram(const MemorySample& m); + + mutable std::mutex _mu; + ForecastSetup _setup; + int _window_len = 100; + int _warmup_left = 0; + + Window _tw; + bool _time_init = false; + std::array _tm; + double _fallback_wall = 0.0; + int _fallback_n = 0; + double _save_per_msplat = 0.0; + int _saves = 0; + + // splat x img + scratch, at its high-water = g0 + g1 * msplats: a Kalman + // filter over [g0, g1], started once the first densify step has run. + MemWindow _mw; + bool _mem_init = false; + double _g[2] = {0, 0}; + double _G[2][2] = {}; + double _demand_sd = 0.0; + double _ours_hw = 0.0, _grow_hw = 0.0; + + std::vector _history; + std::vector _projection; + int _history_stride = 1; + int _history_skip = 0; + double _others_mean = 0.0, _others_sq = 0.0; + int _others_n = 0; + VramForecast _vram; +}; + +template +void SplatSchedule::segments(int from, int64_t live, F&& f) const { + int s = from; + int64_t n = live; + const int stop = std::min( + _total, std::max(_d.refine_stop_iter, _total - _d.refine_stop_num_iter)); + if (_d.refine_every > 0 && n < _cap) { + int g = std::max(s, _d.refine_start_iter + 1); + g = (g + _d.refine_every - 1) / _d.refine_every * _d.refine_every; + for (; g < stop && n < _cap; g += _d.refine_every) { + const int64_t next = densify_target(_d, n, _cap); + if (next == n) break; + f(s, g + 1, n); + n = next; + s = g + 1; + } + } + if (s < _total) f(s, _total, n); +} + +} // namespace spirula diff --git a/src/app/TrainerCore.cpp b/src/app/TrainerCore.cpp index 4fe750ca..0bc16c3c 100644 --- a/src/app/TrainerCore.cpp +++ b/src/app/TrainerCore.cpp @@ -1,6 +1,9 @@ // TrainerCore.cpp -- see TrainerCore.h. #include "app/TrainerCore.h" +#include "backend/api/BackendRuntime.h" +#include "core/Env.h" +#include "core/Tensor.h" #include "data/SceneTransform.h" #include "app/EvalMetrics.h" #include "checkpoint/Adapt.h" @@ -1128,6 +1131,7 @@ void TrainerSession::setup_engine() { // Batch-size policy. double n_batch = std::max((double)num_train / std::max(cfg.max_batch_per_epoch, 1), 1.0); int train_bs = std::max(1, (int)(n_batch + 0.5)); + _batches_per_epoch = (int)std::max(1, (num_train + train_bs - 1) / train_bs); int val_bs = 1; if (num_val > 0) val_bs = std::max(1, (int)std::ceil(n_batch * (double)num_val / (double)num_train)); @@ -1396,10 +1400,45 @@ double TrainerSession::avg_step_latency() const { } double TrainerSession::eta_seconds() const { - const double avg = avg_step_latency(); const int step = cur_step.load(); - if (avg < 0.0 || step <= 0) return -1.0; - return std::max(0, cfg.num_iterations - step) * avg; + if (step <= 0) return -1.0; + return _forecast.eta(step, _live_splats.load()).seconds; +} + +void TrainerSession::observe_memory(int step, int64_t splats_ran) { + MemorySample m; + m.step = step; + m.splats_ran = splats_ran; + m.splats_next = _live_splats.load(); + const DevicePool::CategoryBytes pool = DevicePool::global().category_bytes(); + for (int c = 0; c < (int)VramCategory::Count; ++c) { + m.pool_used[c] = pool.used[c]; + m.pool_cap[c] = pool.cap[c]; + } + m.scratch = engine_get_scratch_bytes(); + const backend::MemoryUsage mu = backend::memory_usage(); + m.has_process = mu.has_process; + m.has_used = mu.has_used; + m.has_total = mu.has_total; + m.process_bytes = mu.process_bytes; + m.used_bytes = mu.used_bytes; + m.total_bytes = mu.total_bytes; + _forecast.add_memory(m); + + // Once per level, so a run that stays at risk says it once. + const VramForecast v = _forecast.vram(false); + if (!v.valid || (int)v.risk <= (int)_warned_risk) return; + _warned_risk = v.risk; + auto gib = [](double b) { + char s[32]; + std::snprintf(s, sizeof s, "%.2f", b / (1024.0 * 1024.0 * 1024.0)); + return std::string(s); + }; + char pct[16]; + std::snprintf(pct, sizeof pct, "%.0f", v.p_oom * 100.0); + log(lfmt(lmsg::vram_forecast_warn, + {gib(v.peak_mean), gib(v.peak_sigma), + gib(0.99 * v.total_bytes - v.others_bytes), std::string(pct)})); } void TrainerSession::train(const TrainerCallbacks& cb) { @@ -1410,6 +1449,20 @@ void TrainerSession::train(const TrainerCallbacks& cb) { _paused_s = 0.0; } + { + ForecastSetup fs; + fs.schedule = SplatSchedule(build_step_config(cfg, st, start_step).densify, + cfg.num_iterations, engine_get_max_num_splats()); + fs.start_step = start_step; + fs.steps_per_save = cfg.steps_per_save; + fs.distinct_batches = _batches_per_epoch; + fs.sh_degree = cfg.sh_degree; + fs.sh_degree_every = cfg.sh_degree_warmup_every; + _forecast.reset(fs); + _live_splats = engine_get_cur_num_splats(); + _warned_risk = OomRisk::Low; + } + int step = start_step; for (; step < cfg.num_iterations; step++) { // Pause gate + render-fairness yield: give viewer render workers an @@ -1430,15 +1483,27 @@ void TrainerSession::train(const TrainerCallbacks& cb) { std::map losses; std::string data_error; + double save_s = 0.0, splat_gpu_s = -1.0; + int64_t splats_ran = 0; + // One step in ten: on Vulkan each bracket is a queue submission. + const bool timed = step % 10 == 0; { std::lock_guard lk(engine_mutex); - if (step > 0 && cfg.steps_per_save > 0 && step % cfg.steps_per_save == 0) + if (step > 0 && cfg.steps_per_save > 0 && step % cfg.steps_per_save == 0) { + const auto t0 = std::chrono::steady_clock::now(); save_checkpoint(step); + save_s = std::chrono::duration( + std::chrono::steady_clock::now() - t0).count(); + } + splats_ran = engine_get_cur_num_splats(); + if (timed) engine_step_timing_arm(); try { losses = train_step(step); } catch (const DataDecodeError& e) { data_error = e.what(); } + if (timed) splat_gpu_s = engine_step_timing_read(); + _live_splats = engine_get_cur_num_splats(); } // Asking outside the lock: the front end may sit on this for minutes // while the user puts the dataset back, and the viewport still wants @@ -1464,6 +1529,24 @@ void TrainerSession::train(const TrainerCallbacks& cb) { _step_latencies.push_back(latency); if (_step_latencies.size() > 100) _step_latencies.pop_front(); } + _forecast.add_step(step, latency - save_s, splats_ran, splat_gpu_s); + if (save_s > 0.0) _forecast.add_save(save_s, splats_ran); + observe_memory(step, splats_ran); + // SS_FORECAST_LOG=1: the forecast's state every 100 steps, English, + // for checking it against how the run actually ends. + static const bool forecast_log = spirula::env("FORECAST_LOG") != nullptr; + if (forecast_log && (step + 1) % 100 == 0) { + const EtaForecast e = _forecast.eta(step + 1, _live_splats.load()); + const VramForecast v = _forecast.vram(false); + const double gib = 1024.0 * 1024.0 * 1024.0; + std::fprintf(stderr, + "[forecast] step %d eta %.1f +- %.1f s ours %.3f GiB peak %.3f +- %.3f GiB" + " others %.3f total %.3f p_oom %.3f%s\n", + step + 1, e.seconds, e.sigma, + v.ours_bytes / gib, + v.peak_mean / gib, v.peak_sigma / gib, v.others_bytes / gib, + v.total_bytes / gib, v.p_oom, v.provisional ? " provisional" : ""); + } if (cb.on_step) { TrainerProgress p; @@ -1507,21 +1590,31 @@ std::string TrainerSession::progress_json() { double elapsed = elapsed_seconds(); double avg = avg_step_latency(); double eta = eta_seconds(); + const VramForecast v = _forecast.vram(false); + static const char* const kRisk[] = {"unknown", "low", "medium", "high"}; + char tail[160]; + if (v.valid) + std::snprintf(tail, sizeof tail, + ", \"vram_peak_bytes\": %.0f, \"vram_peak_sigma\": %.0f, " + "\"oom_probability\": %.4f, \"oom_risk\": \"%s\"}", + v.peak_mean, v.peak_sigma, v.p_oom, kRisk[(int)v.risk]); + else + std::snprintf(tail, sizeof tail, ", \"oom_risk\": \"unknown\"}"); char buf[256]; if (eta >= 0.0) { std::snprintf(buf, sizeof buf, "{\"step\": %d, \"total_steps\": %d, \"elapsed_time\": %.3f, " - "\"eta\": %.3f, \"latency_ms\": %.3f, \"paused\": %s}", + "\"eta\": %.3f, \"latency_ms\": %.3f, \"paused\": %s", step, cfg.num_iterations, elapsed, eta, avg * 1000.0, paused.load() ? "true" : "false"); } else { std::snprintf(buf, sizeof buf, "{\"step\": %d, \"total_steps\": %d, \"elapsed_time\": %.3f, " - "\"eta\": null, \"latency_ms\": null, \"paused\": %s}", + "\"eta\": null, \"latency_ms\": null, \"paused\": %s", step, cfg.num_iterations, elapsed, paused.load() ? "true" : "false"); } - return buf; + return std::string(buf) + tail; } ViewerRenderConfig TrainerSession::make_viewer_config() const { diff --git a/src/app/TrainerCore.h b/src/app/TrainerCore.h index 8d149755..2ce3aa6c 100644 --- a/src/app/TrainerCore.h +++ b/src/app/TrainerCore.h @@ -18,6 +18,7 @@ // setup_engine() calls engine_reset(), so a fresh session can follow a // finished one in the same process (the GUI's "train again" path). +#include "app/TrainForecast.h" #include "engine/Engine.h" #include "core/ColorSpace.h" #include "data/DatasetParser.h" @@ -298,9 +299,11 @@ public: // train() starts, frozen once it returns. double elapsed_seconds() const; - // Remaining wall clock over the last 100 steps' average, or -1 before - // the first step lands. + // Remaining wall clock, from TrainForecast's step-time model and the + // densify schedule still to come; -1 before the first step lands. double eta_seconds() const; + // The run's ETA and VRAM forecast, fed by train(). + const TrainForecast& forecast() const { return _forecast; } // The /progress response body. std::string progress_json(); @@ -333,6 +336,12 @@ private: mutable std::mutex _progress_mutex; // guards the latency window std::deque _step_latencies; // last 100, seconds bool _diverged_loss_reported = false; + + void observe_memory(int step, int64_t splats_ran); + TrainForecast _forecast; + int _batches_per_epoch = 1; + std::atomic _live_splats{0}; + OomRisk _warned_risk = OomRisk::Low; }; } // namespace spirula diff --git a/src/app/gui/GuiApp.cpp b/src/app/gui/GuiApp.cpp index 5b144dab..563abb13 100644 --- a/src/app/gui/GuiApp.cpp +++ b/src/app/gui/GuiApp.cpp @@ -18,6 +18,7 @@ #include "app/gui/Subprocess.h" #include "mesh/MeshImport.h" #include "app/gui/Ui.h" +#include "app/gui/VramForecastView.h" #include "i18n/Locale.h" #include "i18n/catalog/Brand.h" @@ -9355,8 +9356,35 @@ void GuiApp::draw_vram_readout(float x0, float avail) { format_gib(m.total_bytes) + " GiB" : "VRAM " + part(m.has_process, m.process_bytes) + " GiB"; + // A run in progress (or just finished) has a forecast: its risk goes in + // front of the bar, and hovering either shows the projection behind it. + spirula::TrainerSession* session = nullptr; + { + const TrainRunner::Phase ph = _runner.phase(); + if ((ph == TrainRunner::Phase::Training || ph == TrainRunner::Phase::Done) && + _runner.engine_ready()) + session = _runner.session(); + } + const spirula::VramForecast brief = + session ? session->forecast().vram(false) : spirula::VramForecast{}; + const spirula::i18n::Msg* risk = session ? oom_risk_label(brief.risk) : nullptr; + auto hover = [&] { + if (!session) { ui::help_on_hover(msg::vram_help); return; } + if (!ImGui::IsItemHovered(ImGuiHoveredFlags_DelayShort | + ImGuiHoveredFlags_NoSharedDelay) || + !ImGui::BeginTooltip()) + return; + ImGui::PushTextWrapPos(px(480.0f)); + ui::TextDisabled(msg::vram_help); + ImGui::PopTextWrapPos(); + ImGui::Separator(); + vram_forecast_card(session->forecast().vram(), session->cfg.num_iterations); + ImGui::EndTooltip(); + }; + const ImGuiStyle& st = ImGui::GetStyle(); - const float text_w = ImGui::CalcTextSize(label.c_str()).x; + float text_w = ImGui::CalcTextSize(label.c_str()).x; + if (risk) text_w += ImGui::CalcTextSize(risk->get()).x + st.ItemSpacing.x; ImGui::SameLine(); // The bar is the first thing to give when the row is short -- the numbers // beside it say everything it does. Without this the readout ran off the @@ -9370,6 +9398,11 @@ void GuiApp::draw_vram_readout(float x0, float avail) { } if (target > ImGui::GetCursorPosX()) ImGui::SetCursorPosX(target); + if (risk) { + ui::TextColored(oom_risk_color(brief.risk), *risk); + hover(); + ImGui::SameLine(); + } if (bar_w > 0.0f) { const float h = ImGui::GetTextLineHeight(); const ImVec2 p = ImGui::GetCursorScreenPos(); @@ -9389,11 +9422,11 @@ void GuiApp::draw_vram_readout(float x0, float avail) { dl->AddRect(p, ImVec2(p.x + bar_w, p.y + h), ImGui::GetColorU32(ImGuiCol_Border), r); ui::InvisibleButtonRaw("##vram", ImVec2(bar_w, h)); - ui::help_on_hover(msg::vram_help); + hover(); ImGui::SameLine(0.0f, gap); } ui::TextColoredRaw(sized ? kDim : color, label); - ui::help_on_hover(msg::vram_help); + hover(); } // --------------------------------------------------------------------------- diff --git a/src/app/gui/TrainRunner.cpp b/src/app/gui/TrainRunner.cpp index 2793c681..fdd94f5b 100644 --- a/src/app/gui/TrainRunner.cpp +++ b/src/app/gui/TrainRunner.cpp @@ -48,10 +48,7 @@ double TrainRunner::avg_step_latency() { } double TrainRunner::eta_seconds() { - std::lock_guard lk(_mu); - const double avg = avg_latency_locked(); - if (avg < 0.0 || _latest.total_steps <= 0) return -1.0; - return avg * std::max(0, _latest.total_steps - (_latest.step + 1)); + return _session ? _session->eta_seconds() : -1.0; } double TrainRunner::elapsed_seconds() { diff --git a/src/app/gui/TrainRunner.h b/src/app/gui/TrainRunner.h index 38ed50a4..dc005bae 100644 --- a/src/app/gui/TrainRunner.h +++ b/src/app/gui/TrainRunner.h @@ -88,11 +88,11 @@ public: // Latest per-step progress (copy). spirula::TrainerProgress latest_progress(); - // Mean over the last 100 steps, which is also what the ETA is built from. - // A single step's latency swings several-fold with whether a viewer - // render landed on it, so it is not a number to put on screen. + // Mean over the last 100 steps. A single step's latency swings + // several-fold with whether a viewer render landed on it, so it is not a + // number to put on screen. double avg_step_latency(); - double eta_seconds(); // < 0 when unknown + double eta_seconds(); // TrainerSession::eta_seconds; < 0 when unknown // Time spent in the step loop, pauses excluded: < 0 before a session // exists, 0 until the loop starts, frozen once it ends. double elapsed_seconds(); diff --git a/src/app/gui/VramForecastView.cpp b/src/app/gui/VramForecastView.cpp new file mode 100644 index 00000000..4cc9779d --- /dev/null +++ b/src/app/gui/VramForecastView.cpp @@ -0,0 +1,257 @@ +// VramForecastView.cpp -- see VramForecastView.h. + +#include "app/gui/VramForecastView.h" + +#include "i18n/catalog/Gui.h" + +#include +#include +#include +#include + +namespace gui { + +namespace msg = spirula::i18n::msg::gui; +using spirula::OomRisk; +using spirula::VramForecast; + +namespace { + +// The status strip's own pressure colours (GuiApp.cpp). +const ImVec4 kOk(0.35f, 0.85f, 0.45f, 1.0f); +const ImVec4 kWarn(0.95f, 0.75f, 0.30f, 1.0f); +const ImVec4 kErr(1.0f, 0.42f, 0.42f, 1.0f); +const ImVec4 kOthers(0.55f, 0.55f, 0.60f, 1.0f); + +constexpr double kGiB = 1024.0 * 1024.0 * 1024.0; + +std::string gib(double bytes) { + char s[32]; + std::snprintf(s, sizeof s, "%.2f", bytes / kGiB); + return s; +} + +ImU32 with_alpha(ImVec4 c, float a) { + c.w = a; + return ImGui::GetColorU32(c); +} + +void dashed_line(ImDrawList* dl, ImVec2 a, ImVec2 b, ImU32 col, float thick) { + const float dx = b.x - a.x, dy = b.y - a.y; + const float len = std::sqrt(dx * dx + dy * dy); + const float dash = px(4.0f); + for (float t = 0.0f; t < len; t += 2.0f * dash) { + const float t1 = std::min(len, t + dash); + dl->AddLine(ImVec2(a.x + dx * t / len, a.y + dy * t / len), + ImVec2(a.x + dx * t1 / len, a.y + dy * t1 / len), col, thick); + } +} + +void swatch(ImU32 col) { + const float h = ImGui::GetTextLineHeight(); + const ImVec2 p = ImGui::GetCursorScreenPos(); + ImGui::GetWindowDrawList()->AddRectFilled( + ImVec2(p.x, p.y + h * 0.25f), ImVec2(p.x + h * 0.5f, p.y + h * 0.75f), col); + ImGui::Dummy(ImVec2(h * 0.5f, h)); + ImGui::SameLine(0.0f, ImGui::GetStyle().ItemInnerSpacing.x); +} + +// A gridline spacing of 10^k, 2*10^k or 5*10^k GiB that draws at most five. +double tick_step(double max_gib) { + double step = std::pow(10.0, std::floor(std::log10(std::max(max_gib, 1e-3)))); + for (double m : {1.0, 2.0, 5.0, 10.0}) + if (max_gib / (step * m) <= 5.0) return step * m; + return step * 10.0; +} + +void chart(const VramForecast& v, int total_steps) { + ImDrawList* dl = ImGui::GetWindowDrawList(); + const ImGuiStyle& st = ImGui::GetStyle(); + const float line = ImGui::GetTextLineHeight(); + const float W = px(460.0f), H = px(170.0f); + const ImVec2 p0 = ImGui::GetCursorScreenPos(); + ImGui::Dummy(ImVec2(W, H + line + st.ItemInnerSpacing.y)); + + double y_max = v.total_bytes; + for (const auto& s : v.history) y_max = std::max(y_max, s.ours + s.others); + for (const auto& b : v.projection) + y_max = std::max(y_max, v.others_bytes + b.mean + 2.0 * b.sigma); + y_max = std::max(y_max * 1.05, 1.0); + const double T = std::max(1, total_steps); + + const float label_w = ImGui::CalcTextSize("00.0").x + st.ItemInnerSpacing.x; + const ImVec2 r0(p0.x + label_w, p0.y); + const ImVec2 r1(p0.x + W, p0.y + H); + auto X = [&](double step) { + return r0.x + (r1.x - r0.x) * (float)std::min(1.0, std::max(0.0, step / T)); + }; + auto Y = [&](double bytes) { + return r1.y - (r1.y - r0.y) * (float)std::min(1.0, std::max(0.0, bytes / y_max)); + }; + + dl->AddRectFilled(r0, r1, ImGui::GetColorU32(ImGuiCol_FrameBg)); + const ImU32 grid = ImGui::GetColorU32(ImGuiCol_Border); + const ImU32 text_dim = ImGui::GetColorU32(ImGuiCol_TextDisabled); + const double tick = tick_step(y_max / kGiB); + for (double g = 0.0; g * kGiB <= y_max; g += tick) { + const float y = Y(g * kGiB); + dl->AddLine(ImVec2(r0.x, y), ImVec2(r1.x, y), grid); + char s[16]; + std::snprintf(s, sizeof s, tick < 1.0 ? "%.1f" : "%.0f", g); + const ImVec2 sz = ImGui::CalcTextSize(s); + dl->AddText(ImVec2(r0.x - sz.x - st.ItemInnerSpacing.x, y - sz.y * 0.5f), + text_dim, s); + } + + // Other programs at the bottom, this run stacked on top: the gap to the + // capacity line is what the run has left. + const ImU32 others_fill = with_alpha(kOthers, 0.35f); + for (size_t i = 1; i < v.history.size(); ++i) { + const auto& a = v.history[i - 1]; + const auto& b = v.history[i]; + dl->AddQuadFilled(ImVec2(X(a.step), Y(0)), ImVec2(X(a.step), Y(a.others)), + ImVec2(X(b.step), Y(b.others)), ImVec2(X(b.step), Y(0)), + others_fill); + } + const double last_step = v.history.empty() ? 0.0 : v.history.back().step; + if (!v.projection.empty()) + dl->AddRectFilled(ImVec2(X(last_step), Y(v.others_bytes)), ImVec2(r1.x, Y(0)), + others_fill); + + const ImVec4 run = ImGui::GetStyleColorVec4(ImGuiCol_PlotLines); + for (size_t i = 1; i < v.projection.size(); ++i) { + const auto& a = v.projection[i - 1]; + const auto& b = v.projection[i]; + const double o = v.others_bytes; + dl->AddQuadFilled(ImVec2(X(a.step), Y(o + a.mean + 2.0 * a.sigma)), + ImVec2(X(b.step), Y(o + b.mean + 2.0 * b.sigma)), + ImVec2(X(b.step), Y(o + b.mean - 2.0 * b.sigma)), + ImVec2(X(a.step), Y(o + a.mean - 2.0 * a.sigma)), + with_alpha(run, 0.22f)); + dashed_line(dl, ImVec2(X(a.step), Y(o + a.mean)), + ImVec2(X(b.step), Y(o + b.mean)), with_alpha(run, 0.9f), px(1.5f)); + } + for (size_t i = 1; i < v.history.size(); ++i) { + const auto& a = v.history[i - 1]; + const auto& b = v.history[i]; + dl->AddLine(ImVec2(X(a.step), Y(a.ours + a.others)), + ImVec2(X(b.step), Y(b.ours + b.others)), + ImGui::GetColorU32(run), px(2.0f)); + } + if (!v.history.empty()) + dl->AddLine(ImVec2(X(last_step), r0.y), ImVec2(X(last_step), r1.y), grid); + if (v.total_bytes > 0.0) + dl->AddLine(ImVec2(r0.x, Y(v.total_bytes)), ImVec2(r1.x, Y(v.total_bytes)), + ImGui::GetColorU32(kErr), px(1.5f)); + dl->AddRect(r0, r1, grid); + + char s[32]; + dl->AddText(ImVec2(r0.x, r1.y + st.ItemInnerSpacing.y), text_dim, "0"); + std::snprintf(s, sizeof s, "%d", total_steps); + dl->AddText(ImVec2(r1.x - ImGui::CalcTextSize(s).x, r1.y + st.ItemInnerSpacing.y), + text_dim, s); + + swatch(ImGui::GetColorU32(run)); + ui::TextDisabled(msg::vram_legend_run); + ImGui::SameLine(); + swatch(with_alpha(run, 0.35f)); + ui::TextDisabled(msg::vram_legend_projected); + ImGui::SameLine(); + swatch(others_fill); + ui::TextDisabled(msg::vram_legend_others); + ImGui::SameLine(); + swatch(ImGui::GetColorU32(kErr)); + ui::TextDisabled(msg::vram_legend_capacity); +} + +void breakdown(const VramForecast& v) { + struct Row { const spirula::i18n::Msg* label; double now; double peak; ImVec4 color; }; + const ImVec4 bar = ImGui::GetStyleColorVec4(ImGuiCol_PlotHistogram); + const auto cat = [&](VramCategory c) { return v.category[(int)c]; }; + const double sxi = cat(VramCategory::SplatXImg); + const double sxi_peak = std::max(sxi, v.grow_peak_mean - v.scratch); + const Row rows[] = { + {&msg::vram_cat_splat, cat(VramCategory::Splat), 0.0, bar}, + {&msg::vram_cat_splat_x_img, sxi, v.valid ? sxi_peak : 0.0, bar}, + {&msg::vram_cat_image, cat(VramCategory::Image), 0.0, bar}, + {&msg::vram_cat_appearance, cat(VramCategory::Appearance), 0.0, bar}, + {&msg::vram_cat_viewer, cat(VramCategory::Viewer), 0.0, bar}, + {&msg::vram_cat_other, cat(VramCategory::Other), 0.0, bar}, + {&msg::vram_cat_scratch, v.scratch, 0.0, bar}, + {&msg::vram_cat_unpooled, v.unpooled, 0.0, bar}, + {&msg::vram_legend_others, v.others_bytes, 0.0, kOthers}, + }; + const double shown = 1024.0 * 1024.0; // below 1 MiB a row is noise + double scale = 1.0; + float label_w = 0.0f; + for (const Row& r : rows) { + if (std::max(r.now, r.peak) < shown) continue; + scale = std::max(scale, std::max(r.now, r.peak)); + label_w = std::max(label_w, ImGui::CalcTextSize(r.label->get()).x); + } + + ImDrawList* dl = ImGui::GetWindowDrawList(); + const ImGuiStyle& st = ImGui::GetStyle(); + const float bar_w = px(220.0f); + const float h = ImGui::GetTextLineHeight(); + bool faded = false; + for (const Row& r : rows) { + if (std::max(r.now, r.peak) < shown) continue; + ui::Text(*r.label); + ImGui::SameLine(label_w + st.ItemSpacing.x * 2.0f); + const ImVec2 p = ImGui::GetCursorScreenPos(); + const float w_now = bar_w * (float)(r.now / scale); + const float w_peak = bar_w * (float)(r.peak / scale); + dl->AddRectFilled(p, ImVec2(p.x + bar_w, p.y + h), + ImGui::GetColorU32(ImGuiCol_FrameBg)); + if (w_peak > w_now) { + dl->AddRectFilled(ImVec2(p.x + w_now, p.y), ImVec2(p.x + w_peak, p.y + h), + with_alpha(r.color, 0.3f)); + faded = true; + } + dl->AddRectFilled(p, ImVec2(p.x + w_now, p.y + h), ImGui::GetColorU32(r.color)); + ImGui::Dummy(ImVec2(bar_w, h)); + ImGui::SameLine(); + if (r.peak > r.now) + ui::TextRaw(gib(r.now) + " -> " + gib(r.peak)); + else + ui::TextRaw(gib(r.now)); + } + if (faded) ui::TextDisabled(msg::vram_breakdown_growth); +} + +} // namespace + +const spirula::i18n::Msg* oom_risk_label(OomRisk r) { + switch (r) { + case OomRisk::Low: return &msg::oom_risk_low; + case OomRisk::Medium: return &msg::oom_risk_medium; + case OomRisk::High: return &msg::oom_risk_high; + default: return nullptr; + } +} + +ImVec4 oom_risk_color(OomRisk r) { + return r == OomRisk::High ? kErr : r == OomRisk::Medium ? kWarn : kOk; +} + +void vram_forecast_card(const VramForecast& v, int total_steps) { + ui::Text(msg::vram_chart_title); + chart(v, total_steps); + if (v.valid && !v.projection.empty() && v.total_bytes > 0.0) { + char pct[16]; + std::snprintf(pct, sizeof pct, "%.0f", v.p_oom * 100.0); + ui::TextColoredWrapped(oom_risk_color(v.risk), msg::vram_chart_peak, + {gib(v.peak_mean), gib(v.peak_sigma), + gib(0.99 * v.total_bytes - v.others_bytes), + std::string(pct)}); + if (v.provisional) ui::TextDisabledWrapped(msg::vram_chart_provisional); + } else if (!v.valid) { + ui::TextDisabledWrapped(msg::vram_chart_waiting); + } + ImGui::Spacing(); + ui::Text(msg::vram_breakdown_title); + breakdown(v); +} + +} // namespace gui diff --git a/src/app/gui/VramForecastView.h b/src/app/gui/VramForecastView.h new file mode 100644 index 00000000..fbaff7cf --- /dev/null +++ b/src/app/gui/VramForecastView.h @@ -0,0 +1,19 @@ +#pragma once + +// The trainer's VRAM forecast on screen: the out-of-memory risk tag beside the +// status strip's VRAM bar, and the card that bar shows on hover. + +#include "app/TrainForecast.h" +#include "app/gui/Ui.h" + +namespace gui { + +// nullptr while the risk is unknown. +const spirula::i18n::Msg* oom_risk_label(spirula::OomRisk r); +ImVec4 oom_risk_color(spirula::OomRisk r); + +// Memory over the run with its projection, then this run's usage by category. +// Call inside a tooltip. +void vram_forecast_card(const spirula::VramForecast& v, int total_steps); + +} // namespace gui diff --git a/src/backend/cuda/BackendRuntimeCuda.h b/src/backend/cuda/BackendRuntimeCuda.h index abd2928c..5feb25e5 100644 --- a/src/backend/cuda/BackendRuntimeCuda.h +++ b/src/backend/cuda/BackendRuntimeCuda.h @@ -22,13 +22,15 @@ namespace backend { // counter/table are shared across all translation units. Allocation is pooled // upstream (Tensor.h), so this map is touched rarely, not per frame. namespace detail { +// Leaked on purpose: DevicePool::global() is constructed before the first +// device_malloc, so it is destroyed after these and frees through them. inline std::mutex& alloc_mutex() { - static std::mutex m; - return m; + static std::mutex* m = new std::mutex(); + return *m; } inline std::unordered_map& alloc_sizes() { - static std::unordered_map m; - return m; + static auto* m = new std::unordered_map(); + return *m; } inline std::atomic& device_bytes() { static std::atomic v{0}; diff --git a/src/backend/tests/train_forecast_test.cpp b/src/backend/tests/train_forecast_test.cpp new file mode 100644 index 00000000..a41286b0 --- /dev/null +++ b/src/backend/tests/train_forecast_test.cpp @@ -0,0 +1,197 @@ +// TrainForecast against synthetic runs whose true time and VRAM are known. + +#include "app/TrainForecast.h" + +#include +#include +#include +#include + +using namespace spirula; + +static int g_fail = 0; +#define CHECK(cond, ...) \ + do { \ + if (!(cond)) { \ + std::printf("FAIL %s:%d: ", __func__, __LINE__); \ + std::printf(__VA_ARGS__); \ + std::printf("\n"); \ + ++g_fail; \ + } \ + } while (0) + +static DensifyConfig densify(int start, int stop, int every, float growth) { + DensifyConfig d; + d.refine_start_iter = start; + d.refine_stop_iter = stop; + d.refine_stop_num_iter = 1 << 20; // leaves refine_stop_iter in charge + d.refine_every = every; + d.growth_factor = growth; + return d; +} + +// The replay must agree with stepping densify_grows_at / densify_target. +static void schedule_matches_engine_rule() { + const DensifyConfig d = densify(500, 15000, 100, 1.05f); + const int T = 30000; + const int64_t cap = 1000000; + const SplatSchedule sch(d, T, cap); + for (int from : {0, 499, 501, 7300, 20000}) { + int64_t n = 200000; + for (int s = 0; s < from; ++s) + if (densify_grows_at(d, s, T)) n = densify_target(d, n, cap); + std::vector want(T - from); + int64_t m = n; + for (int s = from; s < T; ++s) { + want[s - from] = m; + if (densify_grows_at(d, s, T)) m = densify_target(d, m, cap); + } + int covered = from; + bool ok = true; + sch.segments(from, n, [&](int f, int e, int64_t k) { + ok = ok && f == covered; + for (int s = f; s < e; ++s) ok = ok && want[s - from] == k; + covered = e; + }); + CHECK(ok && covered == T, "segments diverge from the engine rule, from=%d", from); + CHECK(sch.final_count(from, n) == m, "final count %lld vs %lld", + (long long)sch.final_count(from, n), (long long)m); + } + CHECK(sch.first_densify(0) == 600, "first densify %d", sch.first_densify(0)); + CHECK(sch.first_densify(15000) == -1, "densify past the stop"); +} + +// Step time = 20 ms + (40 ms + 3 ms per SH coefficient) per Msplat: the +// naive mean underestimates badly while the count and the SH degree are still +// growing, the model must not. +static void eta_tracks_growth() { + const DensifyConfig d = densify(500, 15000, 100, 1.05f); + const int T = 30000; + const int64_t cap = 3000000; + const SplatSchedule sch(d, T, cap); + ForecastSetup fs; + fs.schedule = sch; + fs.sh_degree = 3; + fs.sh_degree_every = 1000; + TrainForecast fc; + fc.reset(fs); + std::mt19937 rng(1); + std::normal_distribution noise(1.0, 0.1); + const double a = 0.020, b = 0.040, b_coeff = 0.003; + auto coeffs = [](int s) { + const int d = std::min(s / 1000, 3); + return (d + 1) * (d + 1) - 1; + }; + + std::vector n_at(T); + sch.segments(0, 300000, [&](int f, int e, int64_t n) { + for (int s = f; s < e; ++s) n_at[s] = n; + }); + std::vector t_at(T); + for (int s = 0; s < T; ++s) + t_at[s] = a + (b + b_coeff * coeffs(s)) * n_at[s] / 1e6; + auto remaining = [&](int from) { + double sum = 0.0; + for (int s = from; s < T; ++s) sum += t_at[s]; + return sum; + }; + + double recent = 0.0; + for (int s = 0; s < T; ++s) { + const double wall = t_at[s] * noise(rng); + const double gpu = s % 10 == 0 ? (t_at[s] - a) * noise(rng) : -1.0; + fc.add_step(s, wall, n_at[s], gpu); + recent = s < 100 ? wall : recent + (wall - recent) / 100.0; + if (s == 400 || s == 1500 || s == 2500 || s == 8000) { + const double truth = remaining(s + 1); + const EtaForecast e = fc.eta(s + 1, n_at[s + 1]); + const double naive = recent * (T - s - 1); + std::printf(" eta @%d: model %.0f s (sigma %.0f), naive %.0f s, truth %.0f s\n", + s, e.seconds, e.sigma, naive, truth); + // Before the count moves only the prior speaks, so all it can be + // held to is its own sigma -- and beating the naive estimate. + const bool grown = s > 1000; + // Past ~5300 the count sits at the cap and naive is exact too. + if (s < 5000) + CHECK(std::fabs(e.seconds - truth) < std::fabs(naive - truth), + "eta at step %d no better than naive", s); + CHECK(std::fabs(e.seconds - truth) < + (grown ? std::max(0.1 * truth, 2.0 * e.sigma) : 3.0 * e.sigma), + "eta at step %d off by %.1f%%", s, 100.0 * (e.seconds / truth - 1.0)); + } + } +} + +// splat x img follows 60 bytes per splat with per-step spread; the rest is 1 GiB. +static VramForecast run_vram(double others_gib, int stop_at) { + const DensifyConfig d = densify(500, 15000, 100, 1.05f); + const int T = 30000; + const int64_t cap = 3000000; + const SplatSchedule sch(d, T, cap); + ForecastSetup fs; + fs.schedule = sch; + fs.distinct_batches = 200; + TrainForecast fc; + fc.reset(fs); + std::mt19937 rng(7); + std::vector view_ratio(200); + std::lognormal_distribution spread(0.0, 0.2); + for (double& r : view_ratio) r = 60.0 * spread(rng); + std::uniform_int_distribution pick(0, 199); + + const double gib = 1024.0 * 1024.0 * 1024.0; + std::vector n_at(T + 1); + sch.segments(0, 300000, [&](int f, int e, int64_t n) { + for (int s = f; s < e; ++s) n_at[s] = n; + }); + n_at[T] = n_at[T - 1]; + size_t hw = 0; + VramForecast at_stop; + for (int s = 0; s < T; ++s) { + const size_t used = (size_t)(n_at[s] * view_ratio[pick(rng)]); + hw = std::max(hw, used); + MemorySample m; + m.step = s; + m.splats_ran = n_at[s]; + m.splats_next = n_at[s + 1]; + m.pool_used[(int)VramCategory::SplatXImg] = used; + m.pool_cap[(int)VramCategory::SplatXImg] = hw; + m.pool_cap[(int)VramCategory::Splat] = (size_t)gib; + m.has_process = m.has_used = m.has_total = true; + m.process_bytes = (uint64_t)gib + hw; + m.used_bytes = m.process_bytes + (uint64_t)(others_gib * gib); + m.total_bytes = (uint64_t)(8.0 * gib); + fc.add_memory(m); + if (s == stop_at) at_stop = fc.vram(); + } + const double truth = gib + (double)hw; + std::printf(" vram @%d: peak %.3f +- %.3f GiB, truth %.3f GiB, p_oom %.3f\n", + stop_at, at_stop.peak_mean / gib, at_stop.peak_sigma / gib, + truth / gib, at_stop.p_oom); + CHECK(at_stop.valid, "no forecast at step %d", stop_at); + CHECK(std::fabs(at_stop.peak_mean - truth) < 3.0 * at_stop.peak_sigma + 0.02 * truth, + "peak off: %.3f vs %.3f GiB", at_stop.peak_mean / gib, truth / gib); + return at_stop; +} + +static void vram_predicts_peak() { + const VramForecast early = run_vram(1.0, 1000); + CHECK(early.risk == OomRisk::Low, "risk %d with 6 GiB spare", (int)early.risk); + const VramForecast late = run_vram(1.0, 10000); + CHECK(late.peak_sigma <= early.peak_sigma, "sigma grew with more data"); + // 8 GiB device, 6.8 GiB elsewhere: 1.2 GiB for a ~1.2 GiB peak. + const VramForecast tight = run_vram(6.8, 1000); + CHECK(tight.risk >= OomRisk::Medium, "risk %d on a full device", (int)tight.risk); +} + +int main() { + schedule_matches_engine_rule(); + eta_tracks_growth(); + vram_predicts_peak(); + if (g_fail) { + std::printf("train_forecast_test: %d failure(s)\n", g_fail); + return 1; + } + std::printf("train_forecast_test: OK\n"); + return 0; +} diff --git a/src/core/PoolSlots.h b/src/core/PoolSlots.h index 894438f0..a00684cb 100644 --- a/src/core/PoolSlots.h +++ b/src/core/PoolSlots.h @@ -50,9 +50,9 @@ // other. enum class VramCategory : uint8_t { Splat = 0, // per-splat params, gradients, optimizer state, densify aux - // Scratch sized by (camera, gaussian), visible or splat-tile PAIRS, so it - // grows with the batch too. A buffer sized [N] is Splat however late it is - // allocated -- docs/notes/vram-splat-x-img.md. + // Scratch sized by (camera, gaussian), visible or splat-tile PAIRS: the + // one category that grows during training. Splat is sized for cap_max + // (POOL_LIVE_SPLAT_TABLE) -- docs/notes/vram-splat-x-img.md. SplatXImg, Image, // per-image buffers: render outputs, GT, loss maps, img grads Appearance, // bilagrid / background-SH / PPISP / color-space / color-shift @@ -316,27 +316,27 @@ enum class SaveClass : uint8_t { /* ---- densify scratch ---- */ \ X(DensifyQuantileTemp , "densify_quantile_temp", Other , Never) \ X(DensifyMapNorm , "densify_map_norm", Other , Never) \ - X(DensifyScoreGather , "densify_score_gather", Other , Never) \ + X(DensifyScoreGather , "densify_score_gather", Splat , Never) \ X(DensifyScoreClip , "densify_score_clip", Other , Never) \ X(DensifyClipScale , "densify_clip_scale", Other , Never) \ X(DensifyUpdateWeight , "densify_update_weight", Other , Never) \ - X(DensifyWswrSortingValues , "densify_wswr_sorting_values", Other , Never) \ - X(DensifyWswrOutIdx , "densify_wswr_out_idx", Other , Never) \ - X(DensifyWswrKeysOut , "densify_wswr_keys_out", Other , Never) \ - X(DensifyWswrIndicesIn , "densify_wswr_indices_in", Other , Never) \ - X(DensifyWswrIndicesOut , "densify_wswr_indices_out", Other , Never) \ + X(DensifyWswrSortingValues , "densify_wswr_sorting_values", Splat , Never) \ + X(DensifyWswrOutIdx , "densify_wswr_out_idx", Splat , Never) \ + X(DensifyWswrKeysOut , "densify_wswr_keys_out", Splat , Never) \ + X(DensifyWswrIndicesIn , "densify_wswr_indices_in", Splat , Never) \ + X(DensifyWswrIndicesOut , "densify_wswr_indices_out", Splat , Never) \ X(DensifyWswrEligible , "densify_wswr_eligible", Other , Never) \ - X(DensifyRelocMask , "densify_reloc_mask", Other , Never) \ + X(DensifyRelocMask , "densify_reloc_mask", Splat , Never) \ X(DensifyRelocCount , "densify_reloc_count", Other , Never) \ - X(DensifyRelocDstIndices , "densify_reloc_dst_indices", Other , Never) \ - X(DensifyMcmcSampleProbs , "densify_mcmc_sample_probs", Other , Never) \ - X(DensifyMcmcSampleProbsCumsum , "densify_mcmc_sample_probs_cumsum", Other , Never) \ - X(DensifyMcmcIndexMap , "densify_mcmc_index_map", Other , Never) \ - X(DensifyMcmcNIdxBuffer , "densify_mcmc_n_idx_buffer", Other , Never) \ - X(DensifyMcmcAddSampleProbs , "densify_mcmc_add_sample_probs", Other , Never) \ - X(DensifyMcmcAddSampleProbsCumsum, "densify_mcmc_add_sample_probs_cumsum",Other , Never) \ - X(DensifyMcmcAddIndexMap , "densify_mcmc_add_index_map", Other , Never) \ - X(DensifyMcmcAddNIdxBuffer , "densify_mcmc_add_n_idx_buffer", Other , Never) \ + X(DensifyRelocDstIndices , "densify_reloc_dst_indices", Splat , Never) \ + X(DensifyMcmcSampleProbs , "densify_mcmc_sample_probs", Splat , Never) \ + X(DensifyMcmcSampleProbsCumsum , "densify_mcmc_sample_probs_cumsum", Splat , Never) \ + X(DensifyMcmcIndexMap , "densify_mcmc_index_map", Splat , Never) \ + X(DensifyMcmcNIdxBuffer , "densify_mcmc_n_idx_buffer", Splat , Never) \ + X(DensifyMcmcAddSampleProbs , "densify_mcmc_add_sample_probs", Splat , Never) \ + X(DensifyMcmcAddSampleProbsCumsum, "densify_mcmc_add_sample_probs_cumsum",Splat , Never) \ + X(DensifyMcmcAddIndexMap , "densify_mcmc_add_index_map", Splat , Never) \ + X(DensifyMcmcAddNIdxBuffer , "densify_mcmc_add_n_idx_buffer", Splat , Never) \ X(DensifyRobustResid , "densify_robust_resid", Other , Never) \ X(DensifyTukeyC , "densify_tukey_c", Other , Never) \ /* ---- viewer cache/scratch ---- */ \ @@ -510,6 +510,70 @@ constexpr bool ce_alias_rows_unique() { static_assert(ce_alias_rows_unique(), "POOL_ALIAS_TABLE: a slot is listed twice"); +// ---- Slots sized by the live splat count ---------------------------------- + +// Requested at a size proportional to the LIVE count, so the pool allocates +// them for cap_max on first use: Splat then stops growing as the model +// densifies, which the training VRAM forecast assumes. +#define POOL_LIVE_SPLAT_TABLE(X) \ + X(EngSubbatchAccumWeightSum) X(EngDensifyOversizeWeight) \ + X(RasterBwdAccumWeight) X(RasterBwdVWorld) \ + X(FusedProjBwdCamBounds) X(DensifyScoreGather) \ + X(DensifyWswrSortingValues) X(DensifyWswrKeysOut) \ + X(DensifyWswrIndicesIn) X(DensifyWswrIndicesOut) \ + X(DensifyRelocMask) X(DensifyRelocDstIndices) \ + X(DensifyMcmcSampleProbs) X(DensifyMcmcSampleProbsCumsum) \ + X(DensifyMcmcIndexMap) X(DensifyMcmcNIdxBuffer) \ + X(DensifyMcmcAddSampleProbs) X(DensifyMcmcAddSampleProbsCumsum) \ + X(DensifyMcmcAddNIdxBuffer) + +// One int32 per drawn splat: the draw's size varies from call to call and is +// bounded only by the live count, so these get cap_max elements outright. +#define POOL_SPLAT_DRAW_TABLE(X) \ + X(DensifyWswrOutIdx) X(DensifyMcmcAddIndexMap) + +enum class LiveSizing : uint8_t { None, Proportional, Draw }; + +inline constexpr PoolSlot kLiveSplatSlots[] = { +#define X(name) PoolSlot::name, + POOL_LIVE_SPLAT_TABLE(X) +#undef X +}; +inline constexpr PoolSlot kSplatDrawSlots[] = { +#define X(name) PoolSlot::name, + POOL_SPLAT_DRAW_TABLE(X) +#undef X +}; + +constexpr LiveSizing slot_live_sizing(PoolSlot s) { + for (PoolSlot r : kLiveSplatSlots) + if (r == s) return LiveSizing::Proportional; + for (PoolSlot r : kSplatDrawSlots) + if (r == s) return LiveSizing::Draw; + return LiveSizing::None; +} + +constexpr bool ce_live_splat_rows_are_splat() { + for (PoolSlot r : kLiveSplatSlots) + if (slot_category(r) != VramCategory::Splat || slot_phase(r) != PoolPhase::None) + return false; + for (PoolSlot r : kSplatDrawSlots) + if (slot_category(r) != VramCategory::Splat || slot_phase(r) != PoolPhase::None) + return false; + return true; +} +static_assert(ce_live_splat_rows_are_splat(), + "POOL_LIVE_SPLAT_TABLE: rows must be Splat and own their memory"); + +// DevicePool::category_bytes() books the whole arena to SplatXImg. +constexpr bool ce_alias_rows_grow() { + for (const AliasRow& r : kAliasRows) + if (slot_category(r.slot) != VramCategory::SplatXImg) return false; + return true; +} +static_assert(ce_alias_rows_grow(), + "POOL_ALIAS_TABLE: an arena-backed slot must be SplatXImg"); + constexpr const char* to_string(PoolPhase p) { switch (p) { case PoolPhase::None: return "none"; diff --git a/src/core/Tensor.h b/src/core/Tensor.h index 8b92bdb3..77d4f516 100644 --- a/src/core/Tensor.h +++ b/src/core/Tensor.h @@ -8,6 +8,7 @@ #include "backend/api/BackendRuntime.h" #include "core/Env.h" +#include #include #include #include @@ -181,6 +182,8 @@ class DevicePool { Arena _arena; PoolPhase _phase = PoolPhase::None; uint32_t _epoch = 0; // bumped per begin_phase; 0 never matches a Slot + int64_t _live_splats = 0; + int64_t _cap_splats = 0; // Guards _slots / _dyn / _arena against concurrent acquire. It does NOT // make the phase protocol thread-safe: two threads in different phases // would fight over the arena, which the engine mutex is what prevents. @@ -219,8 +222,9 @@ class DevicePool { } // Grow-if-needed on a Slot already selected by the caller (mutex held). + // `reserve` > n allocates room for that many elements up front. template - static T* _acquire_into(Slot& slot, size_t n) { + static T* _acquire_into(Slot& slot, size_t n, size_t reserve = 0) { size_t bytes = n * sizeof(T); // An arena slice is not capacity this slot may reuse; drop it first. if (!slot.owns) { slot.ptr = nullptr; slot.cap_bytes = 0; slot.owns = true; } @@ -230,12 +234,13 @@ class DevicePool { // in a consistent empty state rather than {null ptr, stale cap}. slot.ptr = nullptr; slot.cap_bytes = 0; - slot.ptr = backend::device_malloc_checked(bytes); - slot.cap_bytes = bytes; + const size_t alloc = std::max(bytes, reserve * sizeof(T)); + slot.ptr = backend::device_malloc_checked(alloc); + slot.cap_bytes = alloc; // SS_POISON_POOL=1: fresh device memory holds whatever the // driver left, so a read-before-write is silent where that is // zero and catastrophic where it is not. NaN makes it show. - if (_poison_pool()) backend::memset_sync(slot.ptr, 0xff, bytes); + if (_poison_pool()) backend::memset_sync(slot.ptr, 0xff, alloc); } slot.owns = true; slot.used_bytes = bytes; @@ -333,10 +338,27 @@ public: T* acquire(PoolKey key, size_t n) { std::lock_guard lock(_mu); const PoolSlot slot = pool_key_slot(key); + const LiveSizing live = slot_live_sizing(slot); + if (live != LiveSizing::None && _live_splats > 0 && + _cap_splats > _live_splats && n > 0) { + const size_t reserve = live == LiveSizing::Draw + ? (size_t)_cap_splats + : (size_t)((double)n * (double)_cap_splats / (double)_live_splats + 1.0); + return _acquire_into(_slots[key], n, reserve); + } return _acquire_checked(_slots[key], n, slot_phase(slot), slot_name(slot)); } + // The engine's live and cap splat counts, which size the slots in + // POOL_LIVE_SPLAT_TABLE and POOL_SPLAT_DRAW_TABLE; the engine updates + // them wherever it changes either. 0 disables the reservation. + void set_splat_counts(int64_t live, int64_t cap) { + std::lock_guard lock(_mu); + _live_splats = live; + _cap_splats = cap; + } + // Convenience: acquire a (slot, sub) buffer. sub defaults to the main slot. template T* acquire(PoolSlot slot, size_t n, uint32_t sub = kSubMain) { @@ -401,6 +423,7 @@ public: if (_arena.ptr) backend::device_free(_arena.ptr); _arena = Arena{}; _phase = PoolPhase::None; + _live_splats = _cap_splats = 0; } // Total bytes allocated (capacity, not logical size). @@ -412,6 +435,30 @@ public: return total; } + // Per category, without building names: cheap enough to call every step. + struct CategoryBytes { + size_t used[(int)VramCategory::Count] = {}; + size_t cap[(int)VramCategory::Count] = {}; + }; + CategoryBytes category_bytes() const { + std::lock_guard lock(_mu); + CategoryBytes out; + for (uint32_t i = 0; i < _slots.size(); ++i) { + const Slot& s = _slots[i]; + if (s.cap_bytes == 0 && s.used_bytes == 0) continue; + const int c = (int)slot_category(pool_key_slot(i)); + out.used[c] += s.used_bytes; + out.cap[c] += s.cap_bytes; + } + for (const auto& kv : _dyn) { + const int c = (int)kv.second.cat; + out.used[c] += kv.second.slot.used_bytes; + out.cap[c] += kv.second.slot.cap_bytes; + } + out.cap[(int)VramCategory::SplatXImg] += _arena.cap; + return out; + } + // Per-slot breakdown: [(name, used_bytes, cap_bytes), ...]. Names are // reconstructed byte-identically to the old string keys (slot_name + sub // suffix), so existing consumers keep working. diff --git a/src/engine/Engine.h b/src/engine/Engine.h index 3bbcea5d..b05cfd5d 100644 --- a/src/engine/Engine.h +++ b/src/engine/Engine.h @@ -679,6 +679,12 @@ std::vector> engine_get_pool_breakdown_categorized(); size_t engine_get_scratch_bytes(); +// GPU seconds the next step spends in its splat stages (forward, raster and +// projection backward, optimizer, densify) for the trainer's ETA model. Arm +// before the step, read after; < 0 if not armed or the device has no timer. +void engine_step_timing_arm(); +double engine_step_timing_read(); + // Formatted per-category pool report (SS_PROFILE). Reads the pool's // high-water capacities, so it is a peak, not an instantaneous figure. std::string engine_vram_report(); diff --git a/src/engine/EngineCheckpoint.cpp b/src/engine/EngineCheckpoint.cpp index e3d0485b..a8afb1fb 100644 --- a/src/engine/EngineCheckpoint.cpp +++ b/src/engine/EngineCheckpoint.cpp @@ -431,6 +431,7 @@ int engine_load_checkpoint(std::string input_dir) { // Install runtime scalars + optimizer layout from the checkpoint. s.cur_num_splats = cur_n; s.max_num_splats = max_n; + DevicePool::global().set_splat_counts(cur_n, max_n); s.num_sh = num_sh; s.sh_degree = sh_degree; s.packed = (packed != 0); diff --git a/src/engine/EngineConfig.h b/src/engine/EngineConfig.h index b3b343fc..0b65a9aa 100644 --- a/src/engine/EngineConfig.h +++ b/src/engine/EngineConfig.h @@ -9,7 +9,9 @@ #include "kernels/loss/PerPixelLoss.cuh" // LossWeightIndex #include "kernels/pixelwise/PixelWise.cuh" // PPISPRegLossIndex +#include #include +#include // Bundles the scalars engine_compute_loss_backward takes, for the call path @@ -240,6 +242,17 @@ struct DensifyConfig { int las_split_opacity_k_warmup = 4500; }; +// The schedule engine_densify_step grows the model on; the trainer's ETA and +// VRAM forecasts replay it. +inline bool densify_grows_at(const DensifyConfig& c, int step, int max_steps) { + return c.refine_every > 0 && step > c.refine_start_iter && + step % c.refine_every == 0 && + step < std::max(c.refine_stop_iter, max_steps - c.refine_stop_num_iter); +} +inline int64_t densify_target(const DensifyConfig& c, int64_t cur, int64_t cap) { + return std::max(cur, std::min(cap, (int64_t)(c.growth_factor * (float)cur))); +} + // Per-type Adam LR + TV regularization weight. lr <= 0 disables the channel // for the current step (so a single config covers "enabled but skipped" too). diff --git a/src/engine/EngineDensify.cpp b/src/engine/EngineDensify.cpp index 7e82b635..b1af2999 100644 --- a/src/engine/EngineDensify.cpp +++ b/src/engine/EngineDensify.cpp @@ -90,7 +90,7 @@ int engine_densify_step(int step, int max_steps, const DensifyConfig& cfg) { bool densify_ongoing = (step < std::max(cfg.refine_stop_iter, max_steps - cfg.refine_stop_num_iter)); - bool do_densify = densify_ongoing && (step > cfg.refine_start_iter && step % cfg.refine_every == 0); + bool do_densify = densify_grows_at(cfg, step, max_steps); float progress = ((float)step + 0.5f) / (float)max_steps; // Use pool-backed DeviceVector/DeviceTensor from Buffers directly @@ -340,7 +340,7 @@ int engine_densify_step(int step, int max_steps, const DensifyConfig& cfg) { ); // Add more splats - int64_t n_target = std::min(max_num_splats, (int64_t)(cfg.growth_factor * cur_num_splats)); + int64_t n_target = densify_target(cfg, cur_num_splats, max_num_splats); num_added = (int)std::max((int64_t)0, n_target - cur_num_splats); if (num_added > 0) { // The oversize channel draws first, so its candidate set is the @@ -406,7 +406,7 @@ int engine_densify_step(int step, int max_steps, const DensifyConfig& cfg) { , engine().region.weight); // MCMC sample add - int64_t n_target = std::min(max_num_splats, (int64_t)(cfg.growth_factor * cur_num_splats)); + int64_t n_target = densify_target(cfg, cur_num_splats, max_num_splats); num_added = (int)std::max((int64_t)0, n_target - cur_num_splats); if (num_added > 0) { add_splats_mcmc_tensor( @@ -449,5 +449,6 @@ int engine_densify_step(int step, int max_steps, const DensifyConfig& cfg) { } engine().cur_num_splats = cur_num_splats + num_added; + DevicePool::global().set_splat_counts(engine().cur_num_splats, engine().max_num_splats); return num_added; } diff --git a/src/engine/EngineInternal.h b/src/engine/EngineInternal.h index 0a8bdba9..1e10e837 100644 --- a/src/engine/EngineInternal.h +++ b/src/engine/EngineInternal.h @@ -185,3 +185,13 @@ void engine_ensure_optim_state(int sh_optim_bits, int sh_value_bits, float* _engine_ppisp_reg_loss_into( const std::array& loss_weights, bool compute_grad); + +// Brackets one of a step's splat stages while engine_step_timing_arm() is in +// force. Defined in EngineStepTiming.cpp. +struct SplatStageTimer { + SplatStageTimer(); + ~SplatStageTimer(); + SplatStageTimer(const SplatStageTimer&) = delete; + SplatStageTimer& operator=(const SplatStageTimer&) = delete; + int _pair = -1; +}; diff --git a/src/engine/EngineLoss.cpp b/src/engine/EngineLoss.cpp index 2ab7ac47..68f7608e 100644 --- a/src/engine/EngineLoss.cpp +++ b/src/engine/EngineLoss.cpp @@ -146,6 +146,7 @@ static void _engine_raster_proj_backward( TorchTensorView v_depth_dist = _tv_null(), TorchTensorView v_normal_dist = _tv_null() ) { + SplatStageTimer stage_timer; RenderOutput::TensorTuple v_render_outputs = std::make_tuple( DeviceTensor3D(v_render_rgb), DeviceTensor3D(v_render_depth), diff --git a/src/engine/EngineScenes.cpp b/src/engine/EngineScenes.cpp index b9dc36ec..d0f7e523 100644 --- a/src/engine/EngineScenes.cpp +++ b/src/engine/EngineScenes.cpp @@ -180,6 +180,7 @@ void engine_scene_activate(int slot) { engine().world = s.world; engine().cur_num_splats = s.cur_num_splats; engine().max_num_splats = s.max_num_splats; + DevicePool::global().set_splat_counts(engine().cur_num_splats, engine().max_num_splats); engine().num_sh = s.num_sh; if (g_cs_slot != slot || g_cs_gen != s.cs_gen) { engine_init_color_space(s.cs_enabled, s.cs_transfer, s.cs_is_linear, diff --git a/src/engine/EngineSetup.cpp b/src/engine/EngineSetup.cpp index 7cba5855..3a1272c2 100644 --- a/src/engine/EngineSetup.cpp +++ b/src/engine/EngineSetup.cpp @@ -29,6 +29,7 @@ void set_data_3dgs( engine().cur_num_splats = num_splats; engine().max_num_splats = max_num_splats; + DevicePool::global().set_splat_counts(engine().cur_num_splats, engine().max_num_splats); auto sh_shape = std::get<2>(features_sh); engine().num_sh = (sh_shape.size() >= 2) ? (int)sh_shape[1] : 0; diff --git a/src/engine/EngineStepTiming.cpp b/src/engine/EngineStepTiming.cpp new file mode 100644 index 00000000..a96f2ee2 --- /dev/null +++ b/src/engine/EngineStepTiming.cpp @@ -0,0 +1,59 @@ +// GPU time of a step's splat-proportional stages, for the trainer's ETA model. + +#include "engine/Engine.h" +#include "engine/EngineInternal.h" + +#include "backend/api/BackendRuntime.h" + +#include +#include + +namespace { + +struct StageTiming { + bool armed = false; + // Reused across steps; one pair per bracketed stage of the armed step. + std::vector> pairs; + size_t used = 0; +}; + +StageTiming& timing() { + static StageTiming t; + return t; +} + +} // namespace + +SplatStageTimer::SplatStageTimer() { + StageTiming& t = timing(); + if (!t.armed) return; + if (t.used == t.pairs.size()) + t.pairs.emplace_back(backend::event_create(true), backend::event_create(true)); + _pair = (int)t.used++; + backend::event_record(t.pairs[_pair].first); +} + +SplatStageTimer::~SplatStageTimer() { + if (_pair < 0) return; + backend::event_record(timing().pairs[_pair].second); +} + +void engine_step_timing_arm() { + StageTiming& t = timing(); + t.armed = true; + t.used = 0; +} + +double engine_step_timing_read() { + StageTiming& t = timing(); + if (!t.armed) return -1.0; + t.armed = false; + if (t.used == 0) return -1.0; + backend::event_synchronize(t.pairs[t.used - 1].second); + double ms = 0.0; + for (size_t i = 0; i < t.used; ++i) + ms += backend::event_elapsed_ms(t.pairs[i].first, t.pairs[i].second); + t.used = 0; + // 0 means the device has no timestamps, not that the stages were free. + return ms > 0.0 ? ms * 1e-3 : -1.0; +} diff --git a/src/engine/EngineTrainStep.cpp b/src/engine/EngineTrainStep.cpp index 547f6658..19ebb5a5 100644 --- a/src/engine/EngineTrainStep.cpp +++ b/src/engine/EngineTrainStep.cpp @@ -139,7 +139,10 @@ static std::map _engine_step_fwd_bwd_only( engine().ppisp.enabled && engine().ppisp.cur_run_before_color_space; engine().background.match_luma_pending = cfg.background.match_luminance; - forward_3dgs(primitive, sh_degree, packed, /*output_median=*/false, (int)dist_type); + { + SplatStageTimer stage_timer; + forward_3dgs(primitive, sh_degree, packed, /*output_median=*/false, (int)dist_type); + } // PPISP already ran inside the forward in the before-color-space order. const bool ppisp_after = engine().ppisp.enabled && @@ -186,7 +189,10 @@ static void _engine_step_optim_and_densify( const EngineStepConfig& cfg, std::map& loss_dict ) { - engine_optim_step(step, cfg.optim); + { + SplatStageTimer stage_timer; + engine_optim_step(step, cfg.optim); + } if (engine().background.enabled) { engine_background_optim_step(step, cfg.background); @@ -226,7 +232,11 @@ static void _engine_step_optim_and_densify( ppisp_reg_readout.issue(losses_buf); } - int num_added = engine_densify_step(step, max_steps, cfg.densify); + int num_added = 0; + { + SplatStageTimer stage_timer; + num_added = engine_densify_step(step, max_steps, cfg.densify); + } loss_dict["num_added"] = (float)num_added; loss_dict["cur_num_splats"] = (float)engine().cur_num_splats; diff --git a/src/i18n/catalog/Gui.h b/src/i18n/catalog/Gui.h index 13851105..2c8db8de 100644 --- a/src/i18n/catalog/Gui.h +++ b/src/i18n/catalog/Gui.h @@ -2207,6 +2207,184 @@ SS_MSG(vram_help, "/ aygıtın kapasitesi. “?”, arka ucun o değeri sorgulayamadığı " "anlamına gelir.")); +// The VRAM forecast: the risk tag beside the bar, and its hover card. +SS_MSG(oom_risk_low, + EN("OOM risk: low"), JA("メモリ不足リスク: 低"), ZH_HANS("显存不足风险：低"), + ZH_HANT("顯示記憶體不足風險：低"), KO("메모리 부족 위험: 낮음"), + DE("OOM-Risiko: gering"), FR("Risque de saturation : faible"), + ES("Riesgo de falta de memoria: bajo"), PT("Risco de falta de memória: baixo"), + IT("Rischio memoria esaurita: basso"), NL("Risico geheugentekort: laag"), + RU("Риск нехватки памяти: низкий"), TR("Bellek yetmeme riski: düşük")); + +SS_MSG(oom_risk_medium, + EN("OOM risk: medium"), JA("メモリ不足リスク: 中"), ZH_HANS("显存不足风险：中"), + ZH_HANT("顯示記憶體不足風險：中"), KO("메모리 부족 위험: 보통"), + DE("OOM-Risiko: mittel"), FR("Risque de saturation : moyen"), + ES("Riesgo de falta de memoria: medio"), PT("Risco de falta de memória: médio"), + IT("Rischio memoria esaurita: medio"), NL("Risico geheugentekort: middel"), + RU("Риск нехватки памяти: средний"), TR("Bellek yetmeme riski: orta")); + +SS_MSG(oom_risk_high, + EN("OOM risk: high"), JA("メモリ不足リスク: 高"), ZH_HANS("显存不足风险：高"), + ZH_HANT("顯示記憶體不足風險：高"), KO("메모리 부족 위험: 높음"), + DE("OOM-Risiko: hoch"), FR("Risque de saturation : élevé"), + ES("Riesgo de falta de memoria: alto"), PT("Risco de falta de memória: alto"), + IT("Rischio memoria esaurita: alto"), NL("Risico geheugentekort: hoog"), + RU("Риск нехватки памяти: высокий"), TR("Bellek yetmeme riski: yüksek")); + +SS_MSG(vram_chart_title, + EN("GPU memory over the run (GiB)"), JA("学習中の GPU メモリ（GiB）"), + ZH_HANS("训练过程中的显存（GiB）"), ZH_HANT("訓練過程中的顯示記憶體（GiB）"), + KO("학습 중 GPU 메모리(GiB)"), DE("Grafikspeicher im Verlauf (GiB)"), + FR("Mémoire GPU au fil de l'entraînement (Gio)"), + ES("Memoria de GPU durante el entrenamiento (GiB)"), + PT("Memória da GPU ao longo do treinamento (GiB)"), + IT("Memoria GPU durante l'addestramento (GiB)"), + NL("GPU-geheugen tijdens de training (GiB)"), + RU("Видеопамять по ходу обучения (ГиБ)"), TR("Eğitim boyunca GPU belleği (GiB)")); + +SS_MSG(vram_legend_run, + EN("this run"), JA("この学習"), ZH_HANS("本次训练"), ZH_HANT("本次訓練"), + KO("이 학습"), DE("dieses Training"), FR("cet entraînement"), + ES("este entrenamiento"), PT("este treinamento"), IT("questo addestramento"), + NL("deze training"), RU("это обучение"), TR("bu eğitim")); + +SS_MSG(vram_legend_projected, + EN("projected (95% band)"), JA("予測（95% 範囲）"), ZH_HANS("预测（95% 区间）"), + ZH_HANT("預測（95% 區間）"), KO("예측(95% 범위)"), DE("Prognose (95-%-Band)"), + FR("prévision (bande à 95 %)"), ES("previsión (banda del 95 %)"), + PT("previsão (faixa de 95%)"), IT("previsione (banda al 95%)"), + NL("prognose (95%-band)"), RU("прогноз (полоса 95 %)"), TR("tahmin (%95 aralığı)")); + +SS_MSG(vram_legend_others, + EN("other programs"), JA("他のプログラム"), ZH_HANS("其他程序"), ZH_HANT("其他程式"), + KO("다른 프로그램"), DE("andere Programme"), FR("autres programmes"), + ES("otros programas"), PT("outros programas"), IT("altri programmi"), + NL("andere programma's"), RU("другие программы"), TR("diğer programlar")); + +SS_MSG(vram_legend_capacity, + EN("device capacity"), JA("デバイスの容量"), ZH_HANS("设备容量"), ZH_HANT("裝置容量"), + KO("장치 용량"), DE("Kapazität des Geräts"), FR("capacité du périphérique"), + ES("capacidad del dispositivo"), PT("capacidade do dispositivo"), + IT("capacità del dispositivo"), NL("capaciteit van het apparaat"), + RU("объём устройства"), TR("aygıt kapasitesi")); + +SS_MSG(vram_chart_peak, + EN("Projected peak: {0} ± {1} GiB free for training: {2} GiB chance of running out: {3}%"), + JA("予測ピーク: {0} ± {1} GiB 学習に使える量: {2} GiB 不足する確率: {3}%"), + ZH_HANS("预计峰值：{0} ± {1} GiB 可供训练：{2} GiB 耗尽的概率：{3}%"), + ZH_HANT("預計峰值：{0} ± {1} GiB 可供訓練：{2} GiB 耗盡的機率：{3}%"), + KO("예상 최대치: {0} ± {1} GiB 학습에 쓸 수 있는 양: {2} GiB 부족할 확률: {3}%"), + DE("Erwartete Spitze: {0} ± {1} GiB für das Training frei: {2} GiB Wahrscheinlichkeit, dass er ausgeht: {3} %"), + FR("Pic prévu : {0} ± {1} Gio disponible pour l'entraînement : {2} Gio probabilité de saturation : {3} %"), + ES("Pico previsto: {0} ± {1} GiB libre para entrenar: {2} GiB probabilidad de quedarse sin memoria: {3} %"), + PT("Pico previsto: {0} ± {1} GiB livre para o treinamento: {2} GiB chance de faltar memória: {3}%"), + IT("Picco previsto: {0} ± {1} GiB libera per l'addestramento: {2} GiB probabilità di esaurirla: {3}%"), + NL("Verwachte piek: {0} ± {1} GiB vrij voor training: {2} GiB kans op tekort: {3}%"), + RU("Ожидаемый пик: {0} ± {1} ГиБ доступно для обучения: {2} ГиБ вероятность нехватки: {3} %"), + TR("Beklenen tepe: {0} ± {1} GiB eğitim için boş: {2} GiB yetmeme olasılığı: %{3}")); + +SS_MSG(vram_chart_provisional, + EN("The estimate firms up once the first densification step has run."), + JA("最初の高密度化ステップが済むと、推定の精度が上がります。"), + ZH_HANS("第一次加密步骤运行后，估计会更准确。"), + ZH_HANT("第一次加密步驟執行後，估計會更準確。"), + KO("첫 번째 밀집화 단계가 끝나면 추정이 더 정확해집니다."), + DE("Die Schätzung wird genauer, sobald der erste Verdichtungsschritt gelaufen ist."), + FR("L'estimation se précise après la première étape de densification."), + ES("La estimación se afina en cuanto se ejecuta el primer paso de densificación."), + PT("A estimativa fica mais precisa depois do primeiro passo de densificação."), + IT("La stima si affina dopo il primo passo di densificazione."), + NL("De schatting wordt nauwkeuriger zodra de eerste verdichtingsstap is uitgevoerd."), + RU("Оценка уточнится после первого шага уплотнения."), + TR("İlk yoğunlaştırma adımı çalıştıktan sonra tahmin netleşir.")); + +SS_MSG(vram_chart_waiting, + EN("The projection appears after the first steps have been measured."), + JA("最初のステップを計測すると予測が表示されます。"), + ZH_HANS("测量完最初的若干步后会显示预测。"), + ZH_HANT("量測完最初的若干步後會顯示預測。"), + KO("처음 몇 단계를 측정하면 예측이 표시됩니다."), + DE("Die Prognose erscheint, sobald die ersten Schritte gemessen sind."), + FR("La prévision apparaît une fois les premières étapes mesurées."), + ES("La previsión aparece cuando se han medido los primeros pasos."), + PT("A previsão aparece depois que os primeiros passos são medidos."), + IT("La previsione compare dopo che i primi passi sono stati misurati."), + NL("De prognose verschijnt zodra de eerste stappen gemeten zijn."), + RU("Прогноз появится, когда будут измерены первые шаги."), + TR("Tahmin, ilk adımlar ölçüldükten sonra görünür.")); + +SS_MSG(vram_breakdown_title, + EN("This run by category (GiB)"), JA("この学習の内訳（GiB）"), + ZH_HANS("本次训练按类别（GiB）"), ZH_HANT("本次訓練按類別（GiB）"), + KO("이 학습의 항목별 사용량(GiB)"), DE("Dieses Training nach Kategorie (GiB)"), + FR("Cet entraînement par catégorie (Gio)"), ES("Este entrenamiento por categoría (GiB)"), + PT("Este treinamento por categoria (GiB)"), IT("Questo addestramento per categoria (GiB)"), + NL("Deze training per categorie (GiB)"), RU("Это обучение по категориям (ГиБ)"), + TR("Bu eğitim, kategoriye göre (GiB)")); + +SS_MSG(vram_breakdown_growth, + EN("Faded: growth still to come, up to the projected peak."), + JA("薄い部分: 予測ピークまでにこれから増える分。"), + ZH_HANS("浅色部分：到预计峰值前还会增加的量。"), + ZH_HANT("淺色部分：到預計峰值前還會增加的量。"), + KO("흐린 부분: 예상 최대치까지 앞으로 늘어날 양."), + DE("Blass: der Zuwachs, der bis zur erwarteten Spitze noch kommt."), + FR("En pâle : la croissance encore à venir, jusqu'au pic prévu."), + ES("Atenuado: el crecimiento que aún falta hasta el pico previsto."), + PT("Esmaecido: o crescimento que ainda virá, até o pico previsto."), + IT("Sbiadito: la crescita ancora da venire, fino al picco previsto."), + NL("Vaag: de groei die nog komt, tot de verwachte piek."), + RU("Бледным: рост, который ещё впереди, до ожидаемого пика."), + TR("Soluk: beklenen tepeye kadar daha gelecek artış.")); + +SS_MSG(vram_cat_splat, + EN("Splats"), JA("スプラット"), ZH_HANS("泼溅"), ZH_HANT("潑濺"), KO("스플랫"), + DE("Splats"), FR("Splats"), ES("Splats"), PT("Splats"), IT("Splat"), + NL("Splats"), RU("Сплаты"), TR("Splat'ler")); + +SS_MSG(vram_cat_splat_x_img, + EN("Splats × images"), JA("スプラット × 画像"), ZH_HANS("泼溅 × 图像"), + ZH_HANT("潑濺 × 影像"), KO("스플랫 × 이미지"), DE("Splats × Bilder"), + FR("Splats × images"), ES("Splats × imágenes"), PT("Splats × imagens"), + IT("Splat × immagini"), NL("Splats × beelden"), RU("Сплаты × изображения"), + TR("Splat × görüntü")); + +SS_MSG(vram_cat_image, + EN("Images"), JA("画像"), ZH_HANS("图像"), ZH_HANT("影像"), KO("이미지"), + DE("Bilder"), FR("Images"), ES("Imágenes"), PT("Imagens"), IT("Immagini"), + NL("Beelden"), RU("Изображения"), TR("Görüntüler")); + +SS_MSG(vram_cat_appearance, + EN("Appearance"), JA("外観補正"), ZH_HANS("外观校正"), ZH_HANT("外觀校正"), + KO("외관 보정"), DE("Erscheinungsbild"), FR("Apparence"), ES("Apariencia"), + PT("Aparência"), IT("Aspetto"), NL("Uiterlijk"), RU("Внешний вид"), + TR("Görünüm")); + +SS_MSG(vram_cat_viewer, + EN("Viewer"), JA("ビューア"), ZH_HANS("查看器"), ZH_HANT("檢視器"), KO("뷰어"), + DE("Betrachter"), FR("Visionneuse"), ES("Visor"), PT("Visualizador"), + IT("Visualizzatore"), NL("Viewer"), RU("Просмотр"), TR("Görüntüleyici")); + +SS_MSG(vram_cat_other, + EN("Other"), JA("その他"), ZH_HANS("其他"), ZH_HANT("其他"), KO("기타"), + DE("Sonstiges"), FR("Autre"), ES("Otros"), PT("Outros"), IT("Altro"), + NL("Overig"), RU("Прочее"), TR("Diğer")); + +SS_MSG(vram_cat_scratch, + EN("Sort scratch"), JA("ソート用の作業領域"), ZH_HANS("排序临时缓冲"), + ZH_HANT("排序暫存緩衝"), KO("정렬 작업 공간"), DE("Sortierpuffer"), + FR("Tampon de tri"), ES("Búfer de ordenación"), PT("Buffer de ordenação"), + IT("Buffer di ordinamento"), NL("Sorteerbuffer"), RU("Буфер сортировки"), + TR("Sıralama tamponu")); + +SS_MSG(vram_cat_unpooled, + EN("Backend and staging"), JA("バックエンドと転送用"), ZH_HANS("后端与中转"), + ZH_HANT("後端與中轉"), KO("백엔드와 전송용"), DE("Backend und Staging"), + FR("Backend et transfert"), ES("Backend y transferencia"), + PT("Backend e transferência"), IT("Backend e trasferimento"), + NL("Backend en staging"), RU("Бэкенд и передача"), TR("Arka uç ve aktarım")); + // =========================================================================== // Log panel // =========================================================================== diff --git a/src/i18n/catalog/Log.h b/src/i18n/catalog/Log.h index 1425d338..e7ff966e 100644 --- a/src/i18n/catalog/Log.h +++ b/src/i18n/catalog/Log.h @@ -1725,6 +1725,45 @@ SS_MSG(train_finished, RU("Обучение завершено. Шагов: {0} Время: {1}"), TR("Eğitim tamamlandı. Adım: {0} Süre: {1}")); +SS_MSG(vram_forecast_warn, + EN("Warning: training may run out of GPU memory. Projected peak: {0} ± {1} GiB " + "free for training: {2} GiB chance of running out: {3}%. Lower --cap-max, " + "or close other programs using the GPU."), + JA("警告: 学習中に GPU メモリが不足する可能性があります。予測ピーク: {0} ± {1} GiB " + "学習に使える量: {2} GiB 不足する確率: {3}%。--cap-max を下げるか、GPU を" + "使っている他のプログラムを閉じてください。"), + ZH_HANS("警告：训练可能会耗尽显存。预计峰值：{0} ± {1} GiB 可供训练：{2} GiB " + "耗尽的概率：{3}%。请调低 --cap-max，或关闭其他占用 GPU 的程序。"), + ZH_HANT("警告：訓練可能會耗盡顯示記憶體。預計峰值：{0} ± {1} GiB 可供訓練：{2} GiB " + "耗盡的機率：{3}%。請調低 --cap-max，或關閉其他佔用 GPU 的程式。"), + KO("경고: 학습 중 GPU 메모리가 부족할 수 있습니다. 예상 최대치: {0} ± {1} GiB " + "학습에 쓸 수 있는 양: {2} GiB 부족할 확률: {3}%. --cap-max를 낮추거나 GPU를 " + "쓰는 다른 프로그램을 닫으세요."), + DE("Warnung: Dem Training kann der Grafikspeicher ausgehen. Erwartete Spitze: " + "{0} ± {1} GiB für das Training frei: {2} GiB Wahrscheinlichkeit: {3} %. " + "Senken Sie --cap-max oder schließen Sie andere Programme, die die GPU nutzen."), + FR("Avertissement : l'entraînement risque de manquer de mémoire GPU. Pic prévu : " + "{0} ± {1} Gio disponible pour l'entraînement : {2} Gio probabilité : {3} %. " + "Réduisez --cap-max ou fermez les autres programmes qui utilisent le GPU."), + ES("Aviso: el entrenamiento puede quedarse sin memoria de GPU. Pico previsto: " + "{0} ± {1} GiB libre para entrenar: {2} GiB probabilidad: {3} %. Reduzca " + "--cap-max o cierre otros programas que usen la GPU."), + PT("Aviso: o treinamento pode ficar sem memória de GPU. Pico previsto: {0} ± {1} GiB " + "livre para o treinamento: {2} GiB probabilidade: {3}%. Reduza --cap-max ou " + "feche outros programas que usam a GPU."), + IT("Attenzione: l'addestramento potrebbe esaurire la memoria GPU. Picco previsto: " + "{0} ± {1} GiB libera per l'addestramento: {2} GiB probabilità: {3}%. Riduci " + "--cap-max o chiudi gli altri programmi che usano la GPU."), + NL("Waarschuwing: de training kan zonder GPU-geheugen komen te zitten. Verwachte piek: " + "{0} ± {1} GiB vrij voor training: {2} GiB kans: {3}%. Verlaag --cap-max of " + "sluit andere programma's die de GPU gebruiken."), + RU("Предупреждение: обучению может не хватить видеопамяти. Ожидаемый пик: " + "{0} ± {1} ГиБ доступно для обучения: {2} ГиБ вероятность нехватки: {3} %. " + "Уменьшите --cap-max или закройте другие программы, использующие GPU."), + TR("Uyarı: eğitimin GPU belleği yetmeyebilir. Beklenen tepe: {0} ± {1} GiB " + "eğitim için boş: {2} GiB yetmeme olasılığı: %{3}. --cap-max değerini düşürün " + "ya da GPU kullanan diğer programları kapatın.")); + SS_MSG(partition_applied, EN("Partition part {0}: cameras {1} (core {2}, ring {3}), seed points {4}"), JA("分割パート {0}: カメラ {1}（コア {2}、リング {3}）、初期点 {4}"),