mirror of
https://github.com/harry7557558/spirula-studio.git
synced 2026-10-02 02:44:54 +08:00
Merge remote-tracking branch 'refs/remotes/origin/master'
This commit is contained in:
Harry Chen
@@ -175,6 +175,8 @@ src/
|
||||
│ seeding -> step loop -> eval. Both the CLI and
|
||||
│ the GUI drive this; it lives in the engine
|
||||
│ library (cmake/sources.txt), not the app targets.
|
||||
│ TrainForecast.{h,cpp} beside it is the run's ETA
|
||||
│ and VRAM forecast -- docs/notes/train-forecast.md
|
||||
├── config/ TrainConfig.h — the training config's single source
|
||||
│ of truth: one X-macro row per flag, hand-written.
|
||||
│ TrainConfigJson.h is the one flat-JSON encoding
|
||||
@@ -584,7 +586,10 @@ no ceremony — do not ask, do not leave a note saying you removed it.
|
||||
`SS_POOL_ALIAS_POISON=1` fills the arena at every phase switch so a read
|
||||
that outlives its phase becomes NaNs a parity test catches, and
|
||||
`SS_POOL_ALIAS=0` turns the whole thing off. Read
|
||||
`docs/notes/vram-splat-x-img.md` before adding a row.
|
||||
`docs/notes/vram-splat-x-img.md` before adding a row. A buffer whose
|
||||
length follows the LIVE splat count goes in `POOL_LIVE_SPLAT_TABLE`, so it
|
||||
is sized for `cap_max` once: the trainer's VRAM forecast assumes only
|
||||
`splat x img` grows during a run.
|
||||
- **`SS_PROFILE=1`** enables the per-stage backend timing breakdown
|
||||
(H2D / D2H / D2D / memset / device / host), header-only, both backends, plus
|
||||
a per-category VRAM breakdown after any run that trained. What the biggest
|
||||
|
||||
@@ -39,6 +39,7 @@ src/data/*.cpp
|
||||
src/data/parsers/*.cpp
|
||||
src/app/webviewer/*.cpp
|
||||
src/app/TrainerCore.cpp
|
||||
src/app/TrainForecast.cpp
|
||||
src/app/EvalMetrics.cpp
|
||||
src/external/*.cpp
|
||||
src/external/*.c
|
||||
@@ -55,6 +56,7 @@ src/data/*.cpp
|
||||
src/data/parsers/*.cpp
|
||||
src/app/webviewer/*.cpp
|
||||
src/app/TrainerCore.cpp
|
||||
src/app/TrainForecast.cpp
|
||||
src/app/EvalMetrics.cpp
|
||||
src/external/*.cpp
|
||||
src/external/*.c
|
||||
|
||||
@@ -0,0 +1,99 @@
|
||||
# Training ETA and VRAM forecast
|
||||
|
||||
`app/TrainForecast.{h,cpp}` predicts, while a run trains, how long it has left
|
||||
and how much device memory it will peak at. `TrainerSession::train()` feeds it
|
||||
one sample per step; the CLI's progress line, `/progress`, the GUI's status
|
||||
strip and the VRAM bar's hover card read it. `train_forecast_test` checks both
|
||||
against synthetic runs whose answers are known, and `SS_FORECAST_LOG=1` prints
|
||||
its state every 100 steps for checking it against a real run.
|
||||
|
||||
Both models lean on the same fact: until the model reaches `cap_max`, the
|
||||
splat count still to come is **known**. `engine_densify_step` grows it by
|
||||
`densify_grows_at` / `densify_target` (`engine/EngineConfig.h`), and
|
||||
`SplatSchedule` replays that same rule forward. The SH degree schedule
|
||||
(`min(step / sh_degree_warmup_every, sh_degree)`) is known the same way. So
|
||||
nothing extrapolates the count or the degree -- only what they cost.
|
||||
|
||||
## ETA
|
||||
|
||||
A step costs `a + (b0 + b1 * k) * N^gamma`: `N` in millions of splats, `k`
|
||||
the SH coefficients the step trains (0, 3, 8, 15), `a` everything that does
|
||||
not scale with the model (loss, bilateral grids, PPISP, data loading, host
|
||||
overhead). Per window of `refine_every` steps -- so every window holds the
|
||||
same share of densify steps -- a Kalman filter updates `[a, b0, b1, c]` from:
|
||||
|
||||
- the window's wall time per step, outliers past 3x the median dropped (the
|
||||
pipeline builds when the SH degree steps up are one-offs);
|
||||
- the GPU time of the splat stages, `c + (b0 + b1 * k) * N^gamma`.
|
||||
`SplatStageTimer` (`engine/EngineStepTiming.cpp`) brackets the forward, the
|
||||
raster / projection backward, the optimizer and densify with timestamp
|
||||
events on one step in ten; on Vulkan every bracket is a queue submission.
|
||||
|
||||
Those stages are NOT all per-splat: rasterization has a large per-pixel part,
|
||||
~45% of their time on bonsai at 1/4 resolution and nearly all of it at 1/2.
|
||||
Treating their time as `b * N` overshot the ETA there by +130%. Hence the
|
||||
intercept `c`: the GPU measurement pins the slope once the count moves (it
|
||||
has less host noise than the wall clock), and says nothing about it before.
|
||||
While the count holds still the slope is a prior -- half the splat stages'
|
||||
time, +-70% -- and the ETA's sigma says so.
|
||||
|
||||
No single `gamma` fits: whole-run fits gave 1.0 on bonsai at 1/4 resolution,
|
||||
0.85 at 1/2 and 0.8 on garden. So three filters run, at 0.75, 0.875 and 1.0,
|
||||
each scored on how well it predicted each window before seeing it
|
||||
(discounted by 0.9 a window), and the ETA is the weighted mixture. Updates
|
||||
are gated at 2 sigma with a 10% floor: the laptop GPU these were measured on
|
||||
ran whole 100-step windows 40-60% slow, four times in one run.
|
||||
|
||||
The ETA also adds the checkpoint saves still to come, at the seconds per
|
||||
million splats the saves so far took; a save is excluded from its step's
|
||||
wall time for that reason.
|
||||
|
||||
Replayed on three recorded runs, error in the time left:
|
||||
|
||||
| step | 300 | 1000 | 2000 | 3000 | 4000 | 5000 |
|
||||
|---|---|---|---|---|---|---|
|
||||
| bonsai 1/4, naive (last 100 steps) | -66% | -65% | -59% | -50% | -29% | +3% |
|
||||
| bonsai 1/4, model | -25% | -14% | -10% | -12% | -4% | +4% |
|
||||
| bonsai 1/2, naive | -62% | -63% | -61% | -54% | -43% | -36% |
|
||||
| bonsai 1/2, model | +27% | +55% | +16% | -10% | -3% | -1% |
|
||||
| garden 1/4, naive | -23% | -25% | -23% | -30% | -15% | -24% |
|
||||
| garden 1/4, model | +34% | +22% | +23% | +30% | 0% | -3% |
|
||||
|
||||
The naive estimate is biased low everywhere the count still grows; the model
|
||||
is not biased, and its early error is what the prior cannot know.
|
||||
|
||||
## VRAM
|
||||
|
||||
Only `splat x img` grows during training (docs/notes/vram-splat-x-img.md),
|
||||
so ours at a future step is everything else now plus the growth, projected.
|
||||
Two measurements decide what "the growth" is:
|
||||
|
||||
- It is read off the process **high-water**, not the pool. A step that
|
||||
overflows the alias arena holds private buffers beside it until the next
|
||||
phase switch, so the real peak sits above the steady cap: 179 against 156
|
||||
MiB on bonsai at 1/4, from one step near the end.
|
||||
- The first densify step allocates `splat x img` scratch of its own -- 10 ->
|
||||
55 MiB on garden, count unchanged -- so the fit starts after it. Before
|
||||
that the forecast is provisional: growth all per splat, +-70%, plus 15% +-
|
||||
8% of the splat category for the live-splat scratch the step reserves
|
||||
(14.9% measured on bonsai).
|
||||
|
||||
The growth is `g0 + g1 * N`, a Kalman filter over `[g0, g1]` from one
|
||||
high-water measurement per window, starting at `g0 = 0`: fitting a fixed
|
||||
chunk from the first windows under-read bonsai's peak by 14 sigma, while
|
||||
putting it all per splat over-reads garden's (its 55 MiB chunk) by up to
|
||||
36% -- the safe side. Two terms widen the projection: the expected largest
|
||||
per-step demand over the draws still to come at each count, minus what a
|
||||
window already saw (the pool keeps the largest draw, and a view that sees
|
||||
more of the model sets it); and 0.3 relative sigma per e-fold the count is
|
||||
extrapolated.
|
||||
|
||||
"Others" is device use minus ours (on CUDA that includes the context), an
|
||||
exponential average with its spread. The chance of running out is the normal
|
||||
tail of `peak + others` above 99% of the device -- the last percent is
|
||||
fragmentation. Low below 10%, medium below 50%, high above; the trainer logs a
|
||||
warning the first time the risk reaches each of medium and high.
|
||||
|
||||
On the three runs, from the first densify step on, the projected peak stayed
|
||||
within 2 sigma or above the truth, by at most 8% (bonsai 1/4), 10% (bonsai
|
||||
1/2) and 36% (garden) of it.
|
||||
@@ -15,6 +15,21 @@ Three numbers describe a step, and every buffer below is one of them:
|
||||
| `nnz` | pairs that survive the visibility test | 15.1M |
|
||||
| `n_isects` | (splat, tile) pairs the raster consumes | 22M - 36M |
|
||||
|
||||
## Only this category grows
|
||||
|
||||
Splat buffers are sized for `cap_max`, so everything else a run allocates is
|
||||
fixed once the first densify step has run -- which is what lets the trainer
|
||||
forecast its peak from this category alone. Scratch requested per LIVE splat
|
||||
(the densify draws, `raster_bwd.accum_weight`, `fused_proj_bwd.cam_bounds`)
|
||||
is Splat too: `POOL_LIVE_SPLAT_TABLE` in `core/PoolSlots.h` makes the pool
|
||||
allocate it for `cap_max` on first use, scaling the request by cap / live, and
|
||||
`POOL_SPLAT_DRAW_TABLE` gives the two index lists whose length is a draw count
|
||||
`cap_max` entries outright. Measured on bonsai at 1/4 resolution, 228k -> 666k
|
||||
splats: splat stays at 1042.6 MiB (revised) / 1019.7 MiB (MCMC); without the
|
||||
tables it climbs 11 MiB over the same run and `other` another 15 MiB. A new buffer whose length
|
||||
follows the live count goes in one of those two tables, or the forecast
|
||||
under-reads the peak.
|
||||
|
||||
## What each buffer costs
|
||||
|
||||
Per element, after the 2026-08-30 pass:
|
||||
|
||||
@@ -0,0 +1,520 @@
|
||||
// TrainForecast.cpp -- see TrainForecast.h and docs/notes/train-forecast.md.
|
||||
|
||||
#include "app/TrainForecast.h"
|
||||
|
||||
#include <algorithm>
|
||||
#include <cmath>
|
||||
|
||||
namespace spirula {
|
||||
|
||||
namespace {
|
||||
|
||||
// One scalar measurement y = h . x with variance r. `gate` > 0 inflates r
|
||||
// until the innovation is at most `gate` sigma, so one bad window nudges the
|
||||
// state instead of yanking it.
|
||||
template <int D>
|
||||
void kf_update(double (&x)[D], double (&P)[D][D], const double (&h)[D], double y,
|
||||
double r, double gate = 0.0) {
|
||||
double ph[D];
|
||||
double s = r, pred = 0.0;
|
||||
for (int i = 0; i < D; ++i) {
|
||||
ph[i] = 0.0;
|
||||
for (int j = 0; j < D; ++j) ph[i] += P[i][j] * h[j];
|
||||
s += h[i] * ph[i];
|
||||
pred += h[i] * x[i];
|
||||
}
|
||||
if (!(s > 0.0)) return;
|
||||
const double innov = y - pred;
|
||||
if (gate > 0.0 && innov * innov > gate * gate * s) s = innov * innov / (gate * gate);
|
||||
for (int i = 0; i < D; ++i) x[i] += ph[i] / s * innov;
|
||||
for (int i = 0; i < D; ++i)
|
||||
for (int j = 0; j < D; ++j) P[i][j] -= ph[i] * ph[j] / s;
|
||||
}
|
||||
|
||||
template <int D>
|
||||
double quad(const double (&P)[D][D], const double (&h)[D]) {
|
||||
double q = 0.0;
|
||||
for (int i = 0; i < D; ++i)
|
||||
for (int j = 0; j < D; ++j) q += h[i] * P[i][j] * h[j];
|
||||
return std::max(0.0, q);
|
||||
}
|
||||
|
||||
double norm_cdf(double z) { return 0.5 * std::erfc(-z / std::sqrt(2.0)); }
|
||||
double norm_pdf(double z) {
|
||||
return std::exp(-0.5 * z * z) / std::sqrt(2.0 * 3.14159265358979323846);
|
||||
}
|
||||
|
||||
// Acklam's rational approximation, relative error < 1.2e-9.
|
||||
double norm_quantile(double p) {
|
||||
static const double a[] = {-3.969683028665376e+01, 2.209460984245205e+02,
|
||||
-2.759285104469687e+02, 1.383577518672690e+02,
|
||||
-3.066479806614716e+01, 2.506628277459239e+00};
|
||||
static const double b[] = {-5.447609879822406e+01, 1.615858368580409e+02,
|
||||
-1.556989798598866e+02, 6.680131188771972e+01,
|
||||
-1.328068155288572e+01};
|
||||
static const double c[] = {-7.784894002430293e-03, -3.223964580411365e-01,
|
||||
-2.400758277161838e+00, -2.549732539343734e+00,
|
||||
4.374664141464968e+00, 2.938163982698783e+00};
|
||||
static const double d[] = {7.784695709041462e-03, 3.224671290700398e-01,
|
||||
2.445134137142996e+00, 3.754408661907416e+00};
|
||||
p = std::min(std::max(p, 1e-12), 1.0 - 1e-12);
|
||||
if (p < 0.02425) {
|
||||
const double q = std::sqrt(-2.0 * std::log(p));
|
||||
return (((((c[0] * q + c[1]) * q + c[2]) * q + c[3]) * q + c[4]) * q + c[5]) /
|
||||
((((d[0] * q + d[1]) * q + d[2]) * q + d[3]) * q + 1.0);
|
||||
}
|
||||
if (p > 1.0 - 0.02425) return -norm_quantile(1.0 - p);
|
||||
const double q = p - 0.5, r = q * q;
|
||||
return (((((a[0] * r + a[1]) * r + a[2]) * r + a[3]) * r + a[4]) * r + a[5]) * q /
|
||||
(((((b[0] * r + b[1]) * r + b[2]) * r + b[3]) * r + b[4]) * r + 1.0);
|
||||
}
|
||||
|
||||
// Expected largest of n standard normals (Blom's plotting position).
|
||||
double expected_max(double n) {
|
||||
if (n <= 1.0) return 0.0;
|
||||
return norm_quantile((n - 0.375) / (n + 0.25));
|
||||
}
|
||||
|
||||
// max(c, X) for X ~ N(mu, sigma^2), moment-matched back to a normal.
|
||||
void max_with(double mu, double sigma, double c, double& mean, double& sd) {
|
||||
if (sigma <= 0.0) { mean = std::max(mu, c); sd = 0.0; return; }
|
||||
const double d = (mu - c) / sigma;
|
||||
const double P = norm_cdf(d), phi = norm_pdf(d);
|
||||
mean = mu * P + sigma * phi + c * (1.0 - P);
|
||||
const double m2 = (mu * mu + sigma * sigma) * P + (mu + c) * sigma * phi +
|
||||
c * c * (1.0 - P);
|
||||
sd = std::sqrt(std::max(0.0, m2 - mean * mean));
|
||||
}
|
||||
|
||||
// Mean and its variance, over the samples within 3x the median: a pipeline
|
||||
// build when the SH degree steps up is a one-off, not what later steps cost.
|
||||
void robust_mean(const std::vector<double>& v, double& mean, double& var) {
|
||||
std::vector<double> s = v;
|
||||
std::nth_element(s.begin(), s.begin() + s.size() / 2, s.end());
|
||||
const double cut = 3.0 * s[s.size() / 2];
|
||||
double sum = 0.0, sq = 0.0;
|
||||
int k = 0;
|
||||
for (double x : v)
|
||||
if (x <= cut) { sum += x; sq += x * x; ++k; }
|
||||
mean = sum / k;
|
||||
var = std::max(0.0, sq / k - mean * mean) / k;
|
||||
}
|
||||
|
||||
// Live-splat scratch the first densify step allocates for cap_max, as a
|
||||
// share of the Splat category before it: 14.9% on bonsai (revised, SH 3).
|
||||
constexpr double kFirstDensifySplatShare = 0.15;
|
||||
constexpr double kFirstDensifySplatSigma = 0.08;
|
||||
|
||||
// Pipeline creation and first-touch allocation: bonsai's first 100 steps ran
|
||||
// 17% slower than the next 500 at the same splat count.
|
||||
constexpr int kWarmupSteps = 50;
|
||||
|
||||
// How the splat stages' cost scales with the count. 1.0 fit bonsai at 1/4
|
||||
// resolution, 0.85 bonsai at 1/2 and 0.8 garden at 1/4; each run weighs them.
|
||||
constexpr double kGammas[3] = {0.75, 0.875, 1.0};
|
||||
|
||||
// Picked by replaying bonsai at 1/4 and 1/2 and garden at 1/4 resolution;
|
||||
// docs/notes/train-forecast.md has the errors each one bought.
|
||||
constexpr double kTimeFloor = 0.1; // wall-time noise floor, per window
|
||||
constexpr double kTimeDrift = 0.05; // state drift, per window
|
||||
constexpr double kGate = 2.0; // innovations past this many sigma shrink
|
||||
constexpr double kForget = 0.9; // per window, of each exponent's score
|
||||
// Growth starts all per splat (g0 = 0): fitting a fixed chunk under-read
|
||||
// bonsai's peak by 14 sigma.
|
||||
constexpr double kMemFloor = 0.05; // high-water noise floor
|
||||
constexpr double kMemDrift = 0.02; // [g0, g1] drift, per window
|
||||
constexpr double kChunkSigma = 0.5; // prior sigmas, as shares of the growth
|
||||
constexpr double kPerSplatSigma = 0.7;
|
||||
constexpr double kReach = 0.3; // extra sigma per e-fold of extrapolation
|
||||
|
||||
} // namespace
|
||||
|
||||
// ================
|
||||
// SplatSchedule
|
||||
// ================
|
||||
|
||||
int64_t SplatSchedule::final_count(int from, int64_t live) const {
|
||||
int64_t last = live;
|
||||
segments(from, live, [&](int, int, int64_t n) { last = n; });
|
||||
return last;
|
||||
}
|
||||
|
||||
int SplatSchedule::first_densify(int from) const {
|
||||
if (_d.refine_every <= 0) return -1;
|
||||
int g = std::max(from, _d.refine_start_iter + 1);
|
||||
g = (g + _d.refine_every - 1) / _d.refine_every * _d.refine_every;
|
||||
return densify_grows_at(_d, g, _total) ? g : -1;
|
||||
}
|
||||
|
||||
// ================
|
||||
// TrainForecast: time
|
||||
// ================
|
||||
|
||||
void TrainForecast::reset(const ForecastSetup& s) {
|
||||
std::lock_guard<std::mutex> lk(_mu);
|
||||
_setup = s;
|
||||
const int every = s.schedule.refine_every();
|
||||
_window_len = std::min(500, std::max(20, every > 0 ? every : 100));
|
||||
_warmup_left = kWarmupSteps;
|
||||
_tw = Window{};
|
||||
_time_init = false;
|
||||
for (int i = 0; i < 3; ++i) {
|
||||
_tm[i] = TimeModel{};
|
||||
_tm[i].gamma = kGammas[i];
|
||||
}
|
||||
_fallback_wall = 0.0;
|
||||
_fallback_n = 0;
|
||||
_save_per_msplat = 0.0;
|
||||
_saves = 0;
|
||||
_mw = MemWindow{};
|
||||
_mem_init = false;
|
||||
_demand_sd = 0.0;
|
||||
_ours_hw = _grow_hw = 0.0;
|
||||
_history.clear();
|
||||
_projection.clear();
|
||||
_history_stride = 1;
|
||||
_history_skip = 0;
|
||||
_others_mean = _others_sq = 0.0;
|
||||
_others_n = 0;
|
||||
_vram = VramForecast{};
|
||||
}
|
||||
|
||||
int TrainForecast::sh_coeffs(int step) const {
|
||||
const int every = _setup.sh_degree_every;
|
||||
const int d = every > 0 ? std::min(step / every, _setup.sh_degree) : _setup.sh_degree;
|
||||
return (d + 1) * (d + 1) - 1;
|
||||
}
|
||||
|
||||
void TrainForecast::add_step(int step, double wall_s, int64_t splats,
|
||||
double splat_gpu_s) {
|
||||
std::lock_guard<std::mutex> lk(_mu);
|
||||
// The first step builds the pipelines: seconds, not milliseconds.
|
||||
if (_warmup_left < kWarmupSteps) {
|
||||
_fallback_wall += wall_s;
|
||||
++_fallback_n;
|
||||
}
|
||||
if (_warmup_left > 0) { --_warmup_left; return; }
|
||||
_tw.wall.push_back(wall_s);
|
||||
_tw.gpu.push_back(splat_gpu_s);
|
||||
_tw.msplats.push_back((double)splats / 1e6);
|
||||
_tw.coeffs.push_back(sh_coeffs(step));
|
||||
if ((int)_tw.wall.size() >= _window_len) close_time_window();
|
||||
}
|
||||
|
||||
void TrainForecast::close_time_window() {
|
||||
Window w;
|
||||
std::swap(w, _tw);
|
||||
double y = 0.0, var_y = 0.0, g = 0.0, var_g = 0.0;
|
||||
robust_mean(w.wall, y, var_y);
|
||||
// Laptop GPUs change clocks for whole windows at a time: bonsai at 1/2
|
||||
// resolution ran 40-60% slow for 100 steps at a stretch, four times.
|
||||
var_y += (kTimeFloor * y) * (kTimeFloor * y);
|
||||
std::vector<double> timed;
|
||||
for (double v : w.gpu)
|
||||
if (v >= 0.0) timed.push_back(v);
|
||||
const bool has_gpu = timed.size() >= 2;
|
||||
if (has_gpu) {
|
||||
robust_mean(timed, g, var_g);
|
||||
var_g += (0.05 * g) * (0.05 * g);
|
||||
}
|
||||
for (TimeModel& t : _tm)
|
||||
update_time_model(t, w, y, var_y, g, var_g, has_gpu, !_time_init);
|
||||
_time_init = true;
|
||||
}
|
||||
|
||||
void TrainForecast::update_time_model(TimeModel& t, const Window& w, double y,
|
||||
double var_y, double g, double var_g,
|
||||
bool timed, bool first) {
|
||||
double n = 0.0, nk = 0.0;
|
||||
for (size_t i = 0; i < w.msplats.size(); ++i) {
|
||||
const double load = std::pow(w.msplats[i], t.gamma);
|
||||
n += load;
|
||||
nk += load * w.coeffs[i];
|
||||
}
|
||||
n = std::max(1e-9, n / w.msplats.size());
|
||||
nk /= w.msplats.size();
|
||||
if (first) {
|
||||
// Until the count moves, nothing measures how cost scales with it.
|
||||
// Half the splat stages' GPU time is the prior, +-70%: their per-pixel
|
||||
// share measured ~45% at 1/4 resolution and ~100% at 1/2 on bonsai.
|
||||
const double splat_part = timed ? 0.5 * std::min(g, y) : 0.5 * y;
|
||||
t.x[1] = splat_part / n;
|
||||
// The full SH band (15 coefficients) doubling the per-splat cost,
|
||||
// +-100%: whole-run fits on bonsai gave +127% at 1/4 resolution.
|
||||
t.x[2] = t.x[1] / 15.0;
|
||||
t.x[0] = y - splat_part;
|
||||
t.x[3] = timed ? g - splat_part : 0.0;
|
||||
for (auto& row : t.P) for (double& p : row) p = 0.0;
|
||||
t.P[0][0] = (0.5 * y) * (0.5 * y);
|
||||
t.P[1][1] = (0.7 * t.x[1]) * (0.7 * t.x[1]);
|
||||
t.P[2][2] = t.x[2] * t.x[2];
|
||||
t.P[3][3] = (0.5 * g) * (0.5 * g) + 1e-12;
|
||||
} else {
|
||||
// Drift: splats shrink and the screen footprint changes as training runs.
|
||||
for (int i = 0; i < 4; ++i) t.P[i][i] += (kTimeDrift * t.x[i]) * (kTimeDrift * t.x[i]) + 1e-12;
|
||||
// Each exponent is scored on how well it predicted this window's wall
|
||||
// time before seeing it; the clamp keeps one bad window from deciding.
|
||||
const double pred = t.x[0] + t.x[1] * n + t.x[2] * nk;
|
||||
const double s = quad(t.P, {1.0, n, nk, 0.0}) + var_y;
|
||||
const double z2 = std::min((y - pred) * (y - pred) / s, 4.0 * kGate * kGate);
|
||||
t.score = kForget * t.score - 0.5 * (z2 + std::log(s));
|
||||
}
|
||||
kf_update(t.x, t.P, {1.0, n, nk, 0.0}, y, var_y, kGate);
|
||||
// The splat stages' GPU time has less host noise than the wall clock, so
|
||||
// it pins the slope sooner once the count moves -- through its own
|
||||
// intercept c, the per-pixel work inside those same stages.
|
||||
if (timed) kf_update(t.x, t.P, {0.0, n, nk, 1.0}, g, var_g, kGate);
|
||||
for (double& v : t.x) v = std::max(0.0, v);
|
||||
}
|
||||
|
||||
void TrainForecast::add_save(double seconds, int64_t splats) {
|
||||
std::lock_guard<std::mutex> lk(_mu);
|
||||
if (splats <= 0) return;
|
||||
const double per = seconds / ((double)splats / 1e6);
|
||||
++_saves;
|
||||
_save_per_msplat += (per - _save_per_msplat) / _saves;
|
||||
}
|
||||
|
||||
EtaForecast TrainForecast::eta(int next_step, int64_t live) const {
|
||||
std::lock_guard<std::mutex> lk(_mu);
|
||||
EtaForecast out;
|
||||
const SplatSchedule& sch = _setup.schedule;
|
||||
const int T = sch.total_steps();
|
||||
if (next_step >= T) { out.seconds = out.sigma = 0.0; return out; }
|
||||
|
||||
const int sps = _setup.steps_per_save;
|
||||
const int every = _setup.sh_degree_every;
|
||||
double steps = 0.0, save_msplat = 0.0;
|
||||
double load[3] = {0, 0, 0}, load_k[3] = {0, 0, 0};
|
||||
int64_t last = live;
|
||||
sch.segments(next_step, live, [&](int f, int e, int64_t n) {
|
||||
const double m = (double)n / 1e6;
|
||||
steps += e - f;
|
||||
for (int i = 0; i < 3; ++i) {
|
||||
const double l = std::pow(m, _tm[i].gamma);
|
||||
load[i] += (e - f) * l;
|
||||
for (int t = f; t < e;) {
|
||||
const int end = every > 0 && t / every < _setup.sh_degree
|
||||
? std::min(e, (t / every + 1) * every) : e;
|
||||
load_k[i] += (end - t) * l * sh_coeffs(t);
|
||||
t = end;
|
||||
}
|
||||
}
|
||||
if (sps > 0) {
|
||||
const int lo = std::max(f, 1);
|
||||
if (e > lo) save_msplat += ((e - 1) / sps - (lo - 1) / sps) * m;
|
||||
}
|
||||
last = n;
|
||||
});
|
||||
if (sps != 0) save_msplat += (double)last / 1e6;
|
||||
|
||||
if (_time_init) {
|
||||
double best = _tm[0].score;
|
||||
for (const TimeModel& t : _tm) best = std::max(best, t.score);
|
||||
double wsum = 0.0, mean = 0.0, second = 0.0;
|
||||
for (int i = 0; i < 3; ++i) {
|
||||
const TimeModel& t = _tm[i];
|
||||
const double wi = std::exp(t.score - best);
|
||||
const double sec = t.x[0] * steps + t.x[1] * load[i] + t.x[2] * load_k[i];
|
||||
const double var = quad(t.P, {steps, load[i], load_k[i], 0.0});
|
||||
wsum += wi;
|
||||
mean += wi * sec;
|
||||
second += wi * (var + sec * sec);
|
||||
}
|
||||
out.seconds = mean / wsum;
|
||||
out.sigma = std::sqrt(std::max(0.0, second / wsum - out.seconds * out.seconds));
|
||||
} else if (_fallback_n > 0) {
|
||||
out.seconds = _fallback_wall / _fallback_n * steps;
|
||||
} else {
|
||||
return out;
|
||||
}
|
||||
if (_saves > 0) out.seconds += _save_per_msplat * save_msplat;
|
||||
return out;
|
||||
}
|
||||
|
||||
// ================
|
||||
// TrainForecast: VRAM
|
||||
// ================
|
||||
|
||||
void TrainForecast::add_memory(const MemorySample& m) {
|
||||
std::lock_guard<std::mutex> lk(_mu);
|
||||
const int sxi = (int)VramCategory::SplatXImg;
|
||||
if (m.has_used && m.has_process) {
|
||||
const double o = (double)m.used_bytes - (double)m.process_bytes;
|
||||
const double a = _others_n == 0 ? 1.0 : 0.05;
|
||||
_others_mean += a * (o - _others_mean);
|
||||
_others_sq += a * (o * o - _others_sq);
|
||||
++_others_n;
|
||||
}
|
||||
if (m.splats_ran > 0 && m.pool_used[sxi] > 0) {
|
||||
const double lr = std::log((double)m.pool_used[sxi] / (double)m.splats_ran);
|
||||
_mw.log_ratio += lr;
|
||||
_mw.log_ratio_sq += lr * lr;
|
||||
}
|
||||
// What grows is measured off the process high-water, not the pool: a step
|
||||
// that overflows the alias arena holds private buffers beside it for a
|
||||
// while, and that transient -- not the steady cap -- is the real peak.
|
||||
double pool = (double)m.scratch;
|
||||
for (size_t c : m.pool_cap) pool += (double)c;
|
||||
const double ours = m.has_process ? (double)m.process_bytes : pool;
|
||||
const double steady = (double)m.pool_cap[sxi] + (double)m.scratch;
|
||||
_ours_hw = std::max(_ours_hw, ours);
|
||||
_grow_hw = std::max(steady, _ours_hw - (ours - steady));
|
||||
if (++_mw.steps >= _window_len) close_mem_window(m);
|
||||
refresh_vram(m);
|
||||
}
|
||||
|
||||
void TrainForecast::close_mem_window(const MemorySample& m) {
|
||||
const MemWindow w = _mw;
|
||||
_mw = MemWindow{};
|
||||
if (w.steps >= 2) {
|
||||
const double mean = w.log_ratio / w.steps;
|
||||
const double sd =
|
||||
std::sqrt(std::max(0.0, w.log_ratio_sq / w.steps - mean * mean));
|
||||
_demand_sd = _demand_sd > 0.0 ? 0.7 * _demand_sd + 0.3 * sd : sd;
|
||||
}
|
||||
// The first densify step allocates splat x img scratch of its own -- 10
|
||||
// -> 55 MiB on garden with the count unchanged -- so nothing before it
|
||||
// says how the rest grows.
|
||||
const int first = _setup.schedule.first_densify(_setup.start_step);
|
||||
if (_grow_hw <= 0.0 || m.splats_ran <= 0 || (first >= 0 && m.step <= first))
|
||||
return;
|
||||
const double n = (double)m.splats_ran / 1e6;
|
||||
if (!_mem_init) {
|
||||
_g[0] = 0.0;
|
||||
_g[1] = _grow_hw / n;
|
||||
_G[0][0] = (kChunkSigma * _grow_hw) * (kChunkSigma * _grow_hw);
|
||||
_G[1][1] = (kPerSplatSigma * _g[1]) * (kPerSplatSigma * _g[1]);
|
||||
_G[0][1] = _G[1][0] = 0.0;
|
||||
_mem_init = true;
|
||||
} else {
|
||||
_G[0][0] += (kMemDrift * _g[0]) * (kMemDrift * _g[0]);
|
||||
_G[1][1] += (kMemDrift * _g[1]) * (kMemDrift * _g[1]);
|
||||
}
|
||||
kf_update(_g, _G, {1.0, n}, _grow_hw, (kMemFloor * _grow_hw) * (kMemFloor * _grow_hw), 3.0);
|
||||
_g[0] = std::max(0.0, _g[0]);
|
||||
_g[1] = std::max(0.0, _g[1]);
|
||||
}
|
||||
|
||||
void TrainForecast::refresh_vram(const MemorySample& m) {
|
||||
VramForecast v;
|
||||
const int nc = (int)VramCategory::Count;
|
||||
const int sxi = (int)VramCategory::SplatXImg;
|
||||
double pool_total = (double)m.scratch;
|
||||
for (int c = 0; c < nc; ++c) {
|
||||
v.category[c] = (double)m.pool_cap[c];
|
||||
pool_total += (double)m.pool_cap[c];
|
||||
}
|
||||
v.scratch = (double)m.scratch;
|
||||
const double ours = m.has_process ? (double)m.process_bytes : pool_total;
|
||||
v.unpooled = std::max(0.0, ours - pool_total);
|
||||
v.ours_bytes = ours;
|
||||
v.total_bytes = m.has_total ? (double)m.total_bytes : 0.0;
|
||||
if (_others_n > 0) {
|
||||
v.others_bytes = std::max(0.0, _others_mean);
|
||||
v.others_sigma = std::max(
|
||||
std::sqrt(std::max(0.0, _others_sq - _others_mean * _others_mean)),
|
||||
0.01 * v.total_bytes);
|
||||
}
|
||||
|
||||
if (_history_skip++ % _history_stride == 0)
|
||||
_history.push_back({m.step, ours, v.others_bytes});
|
||||
if (_history.size() > 1024) {
|
||||
std::vector<VramForecast::Sample> half;
|
||||
for (size_t i = 0; i < _history.size(); i += 2) half.push_back(_history[i]);
|
||||
_history.swap(half);
|
||||
_history_stride *= 2;
|
||||
}
|
||||
|
||||
const double fixed = ours - (v.category[sxi] + v.scratch);
|
||||
const SplatSchedule& sch = _setup.schedule;
|
||||
const int next = m.step + 1;
|
||||
const int first = sch.first_densify(_setup.start_step);
|
||||
v.provisional = first >= next;
|
||||
const double extra_mean =
|
||||
v.provisional ? kFirstDensifySplatShare * v.category[(int)VramCategory::Splat] : 0.0;
|
||||
const double extra_sd =
|
||||
v.provisional ? kFirstDensifySplatSigma * v.category[(int)VramCategory::Splat] : 0.0;
|
||||
|
||||
if (next >= sch.total_steps() || m.splats_ran <= 0 || _grow_hw <= 0.0) {
|
||||
v.peak_mean = _ours_hw;
|
||||
v.valid = next >= sch.total_steps();
|
||||
_projection.clear();
|
||||
_vram = std::move(v);
|
||||
return;
|
||||
}
|
||||
|
||||
// Before the filter starts: the prior it will start from.
|
||||
const double n_now = (double)m.splats_ran / 1e6;
|
||||
double g[2] = {_g[0], _g[1]};
|
||||
double G[2][2] = {{_G[0][0], _G[0][1]}, {_G[1][0], _G[1][1]}};
|
||||
if (!_mem_init) {
|
||||
g[0] = 0.0;
|
||||
g[1] = _grow_hw / n_now;
|
||||
G[0][0] = G[0][1] = G[1][0] = 0.0;
|
||||
G[1][1] = (kPerSplatSigma * g[1]) * (kPerSplatSigma * g[1]);
|
||||
}
|
||||
|
||||
const double C = std::max(1, _setup.distinct_batches);
|
||||
const double seen = expected_max(std::min<double>(_window_len, C));
|
||||
double hw_mean = _grow_hw, hw_sd = 0.0;
|
||||
auto band = [&](int step, double g_mean, double g_sd) {
|
||||
const bool extra = v.provisional && step >= first;
|
||||
const double mean = fixed + g_mean + (extra ? extra_mean : 0.0);
|
||||
const double sd = std::sqrt(g_sd * g_sd + (extra ? extra_sd * extra_sd : 0.0));
|
||||
v.projection.push_back({step, mean, sd});
|
||||
};
|
||||
band(next, hw_mean, 0.0);
|
||||
sch.segments(next, m.splats_next, [&](int f, int e, int64_t count) {
|
||||
const double n = (double)count / 1e6;
|
||||
const double per_splat = g[1] * n;
|
||||
// The pool keeps the largest draw: more draws at one count, larger peak.
|
||||
const double corr = std::max(
|
||||
0.0, _demand_sd * (expected_max(std::min<double>(e - f, C)) - seen));
|
||||
// Bytes per splat drift as splats shrink; the further the count is
|
||||
// extrapolated, the less the fit so far says.
|
||||
const double reach = kReach * std::log(n / n_now) * per_splat;
|
||||
const double g_mean = g[0] + per_splat * (1.0 + corr);
|
||||
const double g_sd = std::sqrt(quad(G, {1.0, n}) + reach * reach +
|
||||
0.25 * corr * corr * per_splat * per_splat);
|
||||
double mean, sd;
|
||||
max_with(g_mean, g_sd, hw_mean, mean, sd);
|
||||
const double keep = hw_sd > 0.0
|
||||
? 1.0 - norm_cdf((g_mean - hw_mean) / std::max(g_sd, 1.0)) : 0.0;
|
||||
sd = std::sqrt(sd * sd + keep * keep * hw_sd * hw_sd);
|
||||
if (f > next) band(f, hw_mean, hw_sd);
|
||||
hw_mean = mean;
|
||||
hw_sd = sd;
|
||||
band(e - 1, hw_mean, hw_sd);
|
||||
});
|
||||
|
||||
v.valid = true;
|
||||
v.grow_peak_mean = hw_mean;
|
||||
v.peak_mean = std::max(_ours_hw, v.projection.back().mean);
|
||||
v.peak_sigma = v.projection.back().sigma;
|
||||
if (v.total_bytes > 0.0) {
|
||||
// Allocator fragmentation keeps the last percent out of reach.
|
||||
const double limit = 0.99 * v.total_bytes - v.others_bytes;
|
||||
const double sd = std::sqrt(v.peak_sigma * v.peak_sigma +
|
||||
v.others_sigma * v.others_sigma);
|
||||
v.p_oom = sd > 0.0 ? 1.0 - norm_cdf((limit - v.peak_mean) / sd)
|
||||
: (v.peak_mean > limit ? 1.0 : 0.0);
|
||||
v.risk = v.p_oom < 0.1 ? OomRisk::Low
|
||||
: v.p_oom < 0.5 ? OomRisk::Medium : OomRisk::High;
|
||||
}
|
||||
_projection.swap(v.projection);
|
||||
v.projection.clear();
|
||||
_vram = std::move(v);
|
||||
}
|
||||
|
||||
VramForecast TrainForecast::vram(bool series) const {
|
||||
std::lock_guard<std::mutex> lk(_mu);
|
||||
VramForecast out = _vram;
|
||||
if (series) {
|
||||
out.history = _history;
|
||||
out.projection = _projection;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
} // namespace spirula
|
||||
@@ -0,0 +1,196 @@
|
||||
#pragma once
|
||||
|
||||
// TrainForecast -- where a training run is heading: the wall time it has left
|
||||
// and the VRAM it will peak at, fitted online from the steps already run and
|
||||
// the densify schedule still to come. The models and the measurements behind
|
||||
// their constants: docs/notes/train-forecast.md.
|
||||
|
||||
#include "core/PoolSlots.h"
|
||||
#include "engine/EngineConfig.h"
|
||||
|
||||
#include <algorithm>
|
||||
#include <array>
|
||||
#include <cstdint>
|
||||
#include <mutex>
|
||||
#include <vector>
|
||||
|
||||
namespace spirula {
|
||||
|
||||
// The splat count each step trains with, replayed from the densify schedule.
|
||||
class SplatSchedule {
|
||||
public:
|
||||
SplatSchedule() = default;
|
||||
SplatSchedule(const DensifyConfig& d, int total_steps, int64_t cap)
|
||||
: _d(d), _total(total_steps), _cap(cap) {}
|
||||
|
||||
// f(first, end, n): steps [first, end) run with n splats, covering
|
||||
// [from, total_steps) given `live` splats entering step `from`.
|
||||
template <class F>
|
||||
void segments(int from, int64_t live, F&& f) const;
|
||||
|
||||
int64_t final_count(int from, int64_t live) const;
|
||||
// The first densify step >= from, or -1. It runs even at the cap: it
|
||||
// relocates, and allocates the live-splat scratch.
|
||||
int first_densify(int from) const;
|
||||
int total_steps() const { return _total; }
|
||||
int refine_every() const { return _d.refine_every; }
|
||||
int64_t cap() const { return _cap; }
|
||||
|
||||
private:
|
||||
DensifyConfig _d;
|
||||
int _total = 0;
|
||||
int64_t _cap = 0;
|
||||
};
|
||||
|
||||
enum class OomRisk : uint8_t { Unknown, Low, Medium, High };
|
||||
|
||||
struct EtaForecast {
|
||||
double seconds = -1.0; // < 0 while unknown
|
||||
double sigma = -1.0;
|
||||
};
|
||||
|
||||
// All byte figures are device memory. "Ours" is this process; "others" is
|
||||
// everything else on the device, which includes the backend's own context.
|
||||
struct VramForecast {
|
||||
bool valid = false;
|
||||
// No densify step has run yet, so the live-splat scratch the first one
|
||||
// allocates is an estimate rather than a measurement.
|
||||
bool provisional = false;
|
||||
double total_bytes = 0.0;
|
||||
double ours_bytes = 0.0;
|
||||
double others_bytes = 0.0;
|
||||
double others_sigma = 0.0;
|
||||
|
||||
struct Sample { int step; double ours; double others; };
|
||||
std::vector<Sample> history;
|
||||
struct Band { int step; double mean; double sigma; };
|
||||
std::vector<Band> projection; // ours, from the next step to the end
|
||||
|
||||
double peak_mean = 0.0; // ours at the last step
|
||||
double peak_sigma = 0.0;
|
||||
double p_oom = 0.0; // chance ours + others exceeds the device
|
||||
OomRisk risk = OomRisk::Unknown;
|
||||
|
||||
// Ours now, split by pool category, plus scratch and what the pool does
|
||||
// not account for; and the splat x img share projected at the peak.
|
||||
std::array<double, (int)VramCategory::Count> category{};
|
||||
double scratch = 0.0;
|
||||
double unpooled = 0.0;
|
||||
double grow_peak_mean = 0.0;
|
||||
};
|
||||
|
||||
struct ForecastSetup {
|
||||
SplatSchedule schedule;
|
||||
int start_step = 0;
|
||||
int steps_per_save = 0; // 0: never saves; < 0: only the final one
|
||||
int distinct_batches = 1; // views a step can draw, for the peak draw
|
||||
int sh_degree = 0; // the step's SH degree is
|
||||
int sh_degree_every = 0; // min(step / sh_degree_every, sh_degree)
|
||||
};
|
||||
|
||||
struct MemorySample {
|
||||
int step = 0;
|
||||
int64_t splats_ran = 0; // what the step trained
|
||||
int64_t splats_next = 0; // what the next one will, after densify
|
||||
std::array<size_t, (int)VramCategory::Count> pool_used{};
|
||||
std::array<size_t, (int)VramCategory::Count> pool_cap{};
|
||||
size_t scratch = 0;
|
||||
bool has_process = false, has_used = false, has_total = false;
|
||||
uint64_t process_bytes = 0, used_bytes = 0, total_bytes = 0;
|
||||
};
|
||||
|
||||
class TrainForecast {
|
||||
public:
|
||||
void reset(const ForecastSetup& s);
|
||||
|
||||
// A finished step: its wall time without any checkpoint save, the splats
|
||||
// it ran with, and the GPU seconds of its splat-proportional stages
|
||||
// (< 0 when not timed).
|
||||
void add_step(int step, double wall_s, int64_t splats, double splat_gpu_s);
|
||||
void add_save(double seconds, int64_t splats);
|
||||
void add_memory(const MemorySample& m);
|
||||
|
||||
// Time left from `next_step` with `live` splats entering it.
|
||||
EtaForecast eta(int next_step, int64_t live) const;
|
||||
// Without the history and projection series when `series` is false.
|
||||
VramForecast vram(bool series = true) const;
|
||||
|
||||
private:
|
||||
struct Window {
|
||||
std::vector<double> wall, gpu; // gpu < 0: step not timed
|
||||
std::vector<double> msplats;
|
||||
std::vector<int> coeffs;
|
||||
};
|
||||
// Step time = a + (b0 + b1 * sh_coeffs) * msplats^gamma, and the splat
|
||||
// stages' GPU time = c + the same term: a Kalman filter over
|
||||
// [a, b0, b1, c], one per exponent, weighted by how well each predicted.
|
||||
struct TimeModel {
|
||||
double gamma = 1.0;
|
||||
double x[4] = {0, 0, 0, 0};
|
||||
double P[4][4] = {};
|
||||
double score = 0.0; // discounted log-likelihood
|
||||
};
|
||||
struct MemWindow {
|
||||
int steps = 0;
|
||||
double log_ratio = 0.0, log_ratio_sq = 0.0;
|
||||
};
|
||||
|
||||
int sh_coeffs(int step) const;
|
||||
void close_time_window();
|
||||
void update_time_model(TimeModel& t, const Window& w, double y, double var_y,
|
||||
double g, double var_g, bool timed, bool first);
|
||||
void close_mem_window(const MemorySample& m);
|
||||
void refresh_vram(const MemorySample& m);
|
||||
|
||||
mutable std::mutex _mu;
|
||||
ForecastSetup _setup;
|
||||
int _window_len = 100;
|
||||
int _warmup_left = 0;
|
||||
|
||||
Window _tw;
|
||||
bool _time_init = false;
|
||||
std::array<TimeModel, 3> _tm;
|
||||
double _fallback_wall = 0.0;
|
||||
int _fallback_n = 0;
|
||||
double _save_per_msplat = 0.0;
|
||||
int _saves = 0;
|
||||
|
||||
// splat x img + scratch, at its high-water = g0 + g1 * msplats: a Kalman
|
||||
// filter over [g0, g1], started once the first densify step has run.
|
||||
MemWindow _mw;
|
||||
bool _mem_init = false;
|
||||
double _g[2] = {0, 0};
|
||||
double _G[2][2] = {};
|
||||
double _demand_sd = 0.0;
|
||||
double _ours_hw = 0.0, _grow_hw = 0.0;
|
||||
|
||||
std::vector<VramForecast::Sample> _history;
|
||||
std::vector<VramForecast::Band> _projection;
|
||||
int _history_stride = 1;
|
||||
int _history_skip = 0;
|
||||
double _others_mean = 0.0, _others_sq = 0.0;
|
||||
int _others_n = 0;
|
||||
VramForecast _vram;
|
||||
};
|
||||
|
||||
template <class F>
|
||||
void SplatSchedule::segments(int from, int64_t live, F&& f) const {
|
||||
int s = from;
|
||||
int64_t n = live;
|
||||
const int stop = std::min(
|
||||
_total, std::max(_d.refine_stop_iter, _total - _d.refine_stop_num_iter));
|
||||
if (_d.refine_every > 0 && n < _cap) {
|
||||
int g = std::max(s, _d.refine_start_iter + 1);
|
||||
g = (g + _d.refine_every - 1) / _d.refine_every * _d.refine_every;
|
||||
for (; g < stop && n < _cap; g += _d.refine_every) {
|
||||
const int64_t next = densify_target(_d, n, _cap);
|
||||
if (next == n) break;
|
||||
f(s, g + 1, n);
|
||||
n = next;
|
||||
s = g + 1;
|
||||
}
|
||||
}
|
||||
if (s < _total) f(s, _total, n);
|
||||
}
|
||||
|
||||
} // namespace spirula
|
||||
+100
-7
@@ -1,6 +1,9 @@
|
||||
// TrainerCore.cpp -- see TrainerCore.h.
|
||||
|
||||
#include "app/TrainerCore.h"
|
||||
#include "backend/api/BackendRuntime.h"
|
||||
#include "core/Env.h"
|
||||
#include "core/Tensor.h"
|
||||
#include "data/SceneTransform.h"
|
||||
#include "app/EvalMetrics.h"
|
||||
#include "checkpoint/Adapt.h"
|
||||
@@ -1128,6 +1131,7 @@ void TrainerSession::setup_engine() {
|
||||
// Batch-size policy.
|
||||
double n_batch = std::max((double)num_train / std::max(cfg.max_batch_per_epoch, 1), 1.0);
|
||||
int train_bs = std::max(1, (int)(n_batch + 0.5));
|
||||
_batches_per_epoch = (int)std::max<int64_t>(1, (num_train + train_bs - 1) / train_bs);
|
||||
int val_bs = 1;
|
||||
if (num_val > 0)
|
||||
val_bs = std::max(1, (int)std::ceil(n_batch * (double)num_val / (double)num_train));
|
||||
@@ -1396,10 +1400,45 @@ double TrainerSession::avg_step_latency() const {
|
||||
}
|
||||
|
||||
double TrainerSession::eta_seconds() const {
|
||||
const double avg = avg_step_latency();
|
||||
const int step = cur_step.load();
|
||||
if (avg < 0.0 || step <= 0) return -1.0;
|
||||
return std::max(0, cfg.num_iterations - step) * avg;
|
||||
if (step <= 0) return -1.0;
|
||||
return _forecast.eta(step, _live_splats.load()).seconds;
|
||||
}
|
||||
|
||||
void TrainerSession::observe_memory(int step, int64_t splats_ran) {
|
||||
MemorySample m;
|
||||
m.step = step;
|
||||
m.splats_ran = splats_ran;
|
||||
m.splats_next = _live_splats.load();
|
||||
const DevicePool::CategoryBytes pool = DevicePool::global().category_bytes();
|
||||
for (int c = 0; c < (int)VramCategory::Count; ++c) {
|
||||
m.pool_used[c] = pool.used[c];
|
||||
m.pool_cap[c] = pool.cap[c];
|
||||
}
|
||||
m.scratch = engine_get_scratch_bytes();
|
||||
const backend::MemoryUsage mu = backend::memory_usage();
|
||||
m.has_process = mu.has_process;
|
||||
m.has_used = mu.has_used;
|
||||
m.has_total = mu.has_total;
|
||||
m.process_bytes = mu.process_bytes;
|
||||
m.used_bytes = mu.used_bytes;
|
||||
m.total_bytes = mu.total_bytes;
|
||||
_forecast.add_memory(m);
|
||||
|
||||
// Once per level, so a run that stays at risk says it once.
|
||||
const VramForecast v = _forecast.vram(false);
|
||||
if (!v.valid || (int)v.risk <= (int)_warned_risk) return;
|
||||
_warned_risk = v.risk;
|
||||
auto gib = [](double b) {
|
||||
char s[32];
|
||||
std::snprintf(s, sizeof s, "%.2f", b / (1024.0 * 1024.0 * 1024.0));
|
||||
return std::string(s);
|
||||
};
|
||||
char pct[16];
|
||||
std::snprintf(pct, sizeof pct, "%.0f", v.p_oom * 100.0);
|
||||
log(lfmt(lmsg::vram_forecast_warn,
|
||||
{gib(v.peak_mean), gib(v.peak_sigma),
|
||||
gib(0.99 * v.total_bytes - v.others_bytes), std::string(pct)}));
|
||||
}
|
||||
|
||||
void TrainerSession::train(const TrainerCallbacks& cb) {
|
||||
@@ -1410,6 +1449,20 @@ void TrainerSession::train(const TrainerCallbacks& cb) {
|
||||
_paused_s = 0.0;
|
||||
}
|
||||
|
||||
{
|
||||
ForecastSetup fs;
|
||||
fs.schedule = SplatSchedule(build_step_config(cfg, st, start_step).densify,
|
||||
cfg.num_iterations, engine_get_max_num_splats());
|
||||
fs.start_step = start_step;
|
||||
fs.steps_per_save = cfg.steps_per_save;
|
||||
fs.distinct_batches = _batches_per_epoch;
|
||||
fs.sh_degree = cfg.sh_degree;
|
||||
fs.sh_degree_every = cfg.sh_degree_warmup_every;
|
||||
_forecast.reset(fs);
|
||||
_live_splats = engine_get_cur_num_splats();
|
||||
_warned_risk = OomRisk::Low;
|
||||
}
|
||||
|
||||
int step = start_step;
|
||||
for (; step < cfg.num_iterations; step++) {
|
||||
// Pause gate + render-fairness yield: give viewer render workers an
|
||||
@@ -1430,15 +1483,27 @@ void TrainerSession::train(const TrainerCallbacks& cb) {
|
||||
|
||||
std::map<std::string, float> losses;
|
||||
std::string data_error;
|
||||
double save_s = 0.0, splat_gpu_s = -1.0;
|
||||
int64_t splats_ran = 0;
|
||||
// One step in ten: on Vulkan each bracket is a queue submission.
|
||||
const bool timed = step % 10 == 0;
|
||||
{
|
||||
std::lock_guard<std::mutex> lk(engine_mutex);
|
||||
if (step > 0 && cfg.steps_per_save > 0 && step % cfg.steps_per_save == 0)
|
||||
if (step > 0 && cfg.steps_per_save > 0 && step % cfg.steps_per_save == 0) {
|
||||
const auto t0 = std::chrono::steady_clock::now();
|
||||
save_checkpoint(step);
|
||||
save_s = std::chrono::duration<double>(
|
||||
std::chrono::steady_clock::now() - t0).count();
|
||||
}
|
||||
splats_ran = engine_get_cur_num_splats();
|
||||
if (timed) engine_step_timing_arm();
|
||||
try {
|
||||
losses = train_step(step);
|
||||
} catch (const DataDecodeError& e) {
|
||||
data_error = e.what();
|
||||
}
|
||||
if (timed) splat_gpu_s = engine_step_timing_read();
|
||||
_live_splats = engine_get_cur_num_splats();
|
||||
}
|
||||
// Asking outside the lock: the front end may sit on this for minutes
|
||||
// while the user puts the dataset back, and the viewport still wants
|
||||
@@ -1464,6 +1529,24 @@ void TrainerSession::train(const TrainerCallbacks& cb) {
|
||||
_step_latencies.push_back(latency);
|
||||
if (_step_latencies.size() > 100) _step_latencies.pop_front();
|
||||
}
|
||||
_forecast.add_step(step, latency - save_s, splats_ran, splat_gpu_s);
|
||||
if (save_s > 0.0) _forecast.add_save(save_s, splats_ran);
|
||||
observe_memory(step, splats_ran);
|
||||
// SS_FORECAST_LOG=1: the forecast's state every 100 steps, English,
|
||||
// for checking it against how the run actually ends.
|
||||
static const bool forecast_log = spirula::env("FORECAST_LOG") != nullptr;
|
||||
if (forecast_log && (step + 1) % 100 == 0) {
|
||||
const EtaForecast e = _forecast.eta(step + 1, _live_splats.load());
|
||||
const VramForecast v = _forecast.vram(false);
|
||||
const double gib = 1024.0 * 1024.0 * 1024.0;
|
||||
std::fprintf(stderr,
|
||||
"[forecast] step %d eta %.1f +- %.1f s ours %.3f GiB peak %.3f +- %.3f GiB"
|
||||
" others %.3f total %.3f p_oom %.3f%s\n",
|
||||
step + 1, e.seconds, e.sigma,
|
||||
v.ours_bytes / gib,
|
||||
v.peak_mean / gib, v.peak_sigma / gib, v.others_bytes / gib,
|
||||
v.total_bytes / gib, v.p_oom, v.provisional ? " provisional" : "");
|
||||
}
|
||||
|
||||
if (cb.on_step) {
|
||||
TrainerProgress p;
|
||||
@@ -1507,21 +1590,31 @@ std::string TrainerSession::progress_json() {
|
||||
double elapsed = elapsed_seconds();
|
||||
double avg = avg_step_latency();
|
||||
double eta = eta_seconds();
|
||||
const VramForecast v = _forecast.vram(false);
|
||||
static const char* const kRisk[] = {"unknown", "low", "medium", "high"};
|
||||
char tail[160];
|
||||
if (v.valid)
|
||||
std::snprintf(tail, sizeof tail,
|
||||
", \"vram_peak_bytes\": %.0f, \"vram_peak_sigma\": %.0f, "
|
||||
"\"oom_probability\": %.4f, \"oom_risk\": \"%s\"}",
|
||||
v.peak_mean, v.peak_sigma, v.p_oom, kRisk[(int)v.risk]);
|
||||
else
|
||||
std::snprintf(tail, sizeof tail, ", \"oom_risk\": \"unknown\"}");
|
||||
char buf[256];
|
||||
if (eta >= 0.0) {
|
||||
std::snprintf(buf, sizeof buf,
|
||||
"{\"step\": %d, \"total_steps\": %d, \"elapsed_time\": %.3f, "
|
||||
"\"eta\": %.3f, \"latency_ms\": %.3f, \"paused\": %s}",
|
||||
"\"eta\": %.3f, \"latency_ms\": %.3f, \"paused\": %s",
|
||||
step, cfg.num_iterations, elapsed, eta, avg * 1000.0,
|
||||
paused.load() ? "true" : "false");
|
||||
} else {
|
||||
std::snprintf(buf, sizeof buf,
|
||||
"{\"step\": %d, \"total_steps\": %d, \"elapsed_time\": %.3f, "
|
||||
"\"eta\": null, \"latency_ms\": null, \"paused\": %s}",
|
||||
"\"eta\": null, \"latency_ms\": null, \"paused\": %s",
|
||||
step, cfg.num_iterations, elapsed,
|
||||
paused.load() ? "true" : "false");
|
||||
}
|
||||
return buf;
|
||||
return std::string(buf) + tail;
|
||||
}
|
||||
|
||||
ViewerRenderConfig TrainerSession::make_viewer_config() const {
|
||||
|
||||
+11
-2
@@ -18,6 +18,7 @@
|
||||
// setup_engine() calls engine_reset(), so a fresh session can follow a
|
||||
// finished one in the same process (the GUI's "train again" path).
|
||||
|
||||
#include "app/TrainForecast.h"
|
||||
#include "engine/Engine.h"
|
||||
#include "core/ColorSpace.h"
|
||||
#include "data/DatasetParser.h"
|
||||
@@ -298,9 +299,11 @@ public:
|
||||
// train() starts, frozen once it returns.
|
||||
double elapsed_seconds() const;
|
||||
|
||||
// Remaining wall clock over the last 100 steps' average, or -1 before
|
||||
// the first step lands.
|
||||
// Remaining wall clock, from TrainForecast's step-time model and the
|
||||
// densify schedule still to come; -1 before the first step lands.
|
||||
double eta_seconds() const;
|
||||
// The run's ETA and VRAM forecast, fed by train().
|
||||
const TrainForecast& forecast() const { return _forecast; }
|
||||
|
||||
// The /progress response body.
|
||||
std::string progress_json();
|
||||
@@ -333,6 +336,12 @@ private:
|
||||
mutable std::mutex _progress_mutex; // guards the latency window
|
||||
std::deque<double> _step_latencies; // last 100, seconds
|
||||
bool _diverged_loss_reported = false;
|
||||
|
||||
void observe_memory(int step, int64_t splats_ran);
|
||||
TrainForecast _forecast;
|
||||
int _batches_per_epoch = 1;
|
||||
std::atomic<int64_t> _live_splats{0};
|
||||
OomRisk _warned_risk = OomRisk::Low;
|
||||
};
|
||||
|
||||
} // namespace spirula
|
||||
|
||||
+36
-3
@@ -18,6 +18,7 @@
|
||||
#include "app/gui/Subprocess.h"
|
||||
#include "mesh/MeshImport.h"
|
||||
#include "app/gui/Ui.h"
|
||||
#include "app/gui/VramForecastView.h"
|
||||
|
||||
#include "i18n/Locale.h"
|
||||
#include "i18n/catalog/Brand.h"
|
||||
@@ -9355,8 +9356,35 @@ void GuiApp::draw_vram_readout(float x0, float avail) {
|
||||
format_gib(m.total_bytes) + " GiB"
|
||||
: "VRAM " + part(m.has_process, m.process_bytes) + " GiB";
|
||||
|
||||
// A run in progress (or just finished) has a forecast: its risk goes in
|
||||
// front of the bar, and hovering either shows the projection behind it.
|
||||
spirula::TrainerSession* session = nullptr;
|
||||
{
|
||||
const TrainRunner::Phase ph = _runner.phase();
|
||||
if ((ph == TrainRunner::Phase::Training || ph == TrainRunner::Phase::Done) &&
|
||||
_runner.engine_ready())
|
||||
session = _runner.session();
|
||||
}
|
||||
const spirula::VramForecast brief =
|
||||
session ? session->forecast().vram(false) : spirula::VramForecast{};
|
||||
const spirula::i18n::Msg* risk = session ? oom_risk_label(brief.risk) : nullptr;
|
||||
auto hover = [&] {
|
||||
if (!session) { ui::help_on_hover(msg::vram_help); return; }
|
||||
if (!ImGui::IsItemHovered(ImGuiHoveredFlags_DelayShort |
|
||||
ImGuiHoveredFlags_NoSharedDelay) ||
|
||||
!ImGui::BeginTooltip())
|
||||
return;
|
||||
ImGui::PushTextWrapPos(px(480.0f));
|
||||
ui::TextDisabled(msg::vram_help);
|
||||
ImGui::PopTextWrapPos();
|
||||
ImGui::Separator();
|
||||
vram_forecast_card(session->forecast().vram(), session->cfg.num_iterations);
|
||||
ImGui::EndTooltip();
|
||||
};
|
||||
|
||||
const ImGuiStyle& st = ImGui::GetStyle();
|
||||
const float text_w = ImGui::CalcTextSize(label.c_str()).x;
|
||||
float text_w = ImGui::CalcTextSize(label.c_str()).x;
|
||||
if (risk) text_w += ImGui::CalcTextSize(risk->get()).x + st.ItemSpacing.x;
|
||||
ImGui::SameLine();
|
||||
// The bar is the first thing to give when the row is short -- the numbers
|
||||
// beside it say everything it does. Without this the readout ran off the
|
||||
@@ -9370,6 +9398,11 @@ void GuiApp::draw_vram_readout(float x0, float avail) {
|
||||
}
|
||||
if (target > ImGui::GetCursorPosX()) ImGui::SetCursorPosX(target);
|
||||
|
||||
if (risk) {
|
||||
ui::TextColored(oom_risk_color(brief.risk), *risk);
|
||||
hover();
|
||||
ImGui::SameLine();
|
||||
}
|
||||
if (bar_w > 0.0f) {
|
||||
const float h = ImGui::GetTextLineHeight();
|
||||
const ImVec2 p = ImGui::GetCursorScreenPos();
|
||||
@@ -9389,11 +9422,11 @@ void GuiApp::draw_vram_readout(float x0, float avail) {
|
||||
dl->AddRect(p, ImVec2(p.x + bar_w, p.y + h),
|
||||
ImGui::GetColorU32(ImGuiCol_Border), r);
|
||||
ui::InvisibleButtonRaw("##vram", ImVec2(bar_w, h));
|
||||
ui::help_on_hover(msg::vram_help);
|
||||
hover();
|
||||
ImGui::SameLine(0.0f, gap);
|
||||
}
|
||||
ui::TextColoredRaw(sized ? kDim : color, label);
|
||||
ui::help_on_hover(msg::vram_help);
|
||||
hover();
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
@@ -48,10 +48,7 @@ double TrainRunner::avg_step_latency() {
|
||||
}
|
||||
|
||||
double TrainRunner::eta_seconds() {
|
||||
std::lock_guard<std::mutex> lk(_mu);
|
||||
const double avg = avg_latency_locked();
|
||||
if (avg < 0.0 || _latest.total_steps <= 0) return -1.0;
|
||||
return avg * std::max(0, _latest.total_steps - (_latest.step + 1));
|
||||
return _session ? _session->eta_seconds() : -1.0;
|
||||
}
|
||||
|
||||
double TrainRunner::elapsed_seconds() {
|
||||
|
||||
@@ -88,11 +88,11 @@ public:
|
||||
|
||||
// Latest per-step progress (copy).
|
||||
spirula::TrainerProgress latest_progress();
|
||||
// Mean over the last 100 steps, which is also what the ETA is built from.
|
||||
// A single step's latency swings several-fold with whether a viewer
|
||||
// render landed on it, so it is not a number to put on screen.
|
||||
// Mean over the last 100 steps. A single step's latency swings
|
||||
// several-fold with whether a viewer render landed on it, so it is not a
|
||||
// number to put on screen.
|
||||
double avg_step_latency();
|
||||
double eta_seconds(); // < 0 when unknown
|
||||
double eta_seconds(); // TrainerSession::eta_seconds; < 0 when unknown
|
||||
// Time spent in the step loop, pauses excluded: < 0 before a session
|
||||
// exists, 0 until the loop starts, frozen once it ends.
|
||||
double elapsed_seconds();
|
||||
|
||||
@@ -0,0 +1,257 @@
|
||||
// VramForecastView.cpp -- see VramForecastView.h.
|
||||
|
||||
#include "app/gui/VramForecastView.h"
|
||||
|
||||
#include "i18n/catalog/Gui.h"
|
||||
|
||||
#include <algorithm>
|
||||
#include <cmath>
|
||||
#include <cstdio>
|
||||
#include <string>
|
||||
|
||||
namespace gui {
|
||||
|
||||
namespace msg = spirula::i18n::msg::gui;
|
||||
using spirula::OomRisk;
|
||||
using spirula::VramForecast;
|
||||
|
||||
namespace {
|
||||
|
||||
// The status strip's own pressure colours (GuiApp.cpp).
|
||||
const ImVec4 kOk(0.35f, 0.85f, 0.45f, 1.0f);
|
||||
const ImVec4 kWarn(0.95f, 0.75f, 0.30f, 1.0f);
|
||||
const ImVec4 kErr(1.0f, 0.42f, 0.42f, 1.0f);
|
||||
const ImVec4 kOthers(0.55f, 0.55f, 0.60f, 1.0f);
|
||||
|
||||
constexpr double kGiB = 1024.0 * 1024.0 * 1024.0;
|
||||
|
||||
std::string gib(double bytes) {
|
||||
char s[32];
|
||||
std::snprintf(s, sizeof s, "%.2f", bytes / kGiB);
|
||||
return s;
|
||||
}
|
||||
|
||||
ImU32 with_alpha(ImVec4 c, float a) {
|
||||
c.w = a;
|
||||
return ImGui::GetColorU32(c);
|
||||
}
|
||||
|
||||
void dashed_line(ImDrawList* dl, ImVec2 a, ImVec2 b, ImU32 col, float thick) {
|
||||
const float dx = b.x - a.x, dy = b.y - a.y;
|
||||
const float len = std::sqrt(dx * dx + dy * dy);
|
||||
const float dash = px(4.0f);
|
||||
for (float t = 0.0f; t < len; t += 2.0f * dash) {
|
||||
const float t1 = std::min(len, t + dash);
|
||||
dl->AddLine(ImVec2(a.x + dx * t / len, a.y + dy * t / len),
|
||||
ImVec2(a.x + dx * t1 / len, a.y + dy * t1 / len), col, thick);
|
||||
}
|
||||
}
|
||||
|
||||
void swatch(ImU32 col) {
|
||||
const float h = ImGui::GetTextLineHeight();
|
||||
const ImVec2 p = ImGui::GetCursorScreenPos();
|
||||
ImGui::GetWindowDrawList()->AddRectFilled(
|
||||
ImVec2(p.x, p.y + h * 0.25f), ImVec2(p.x + h * 0.5f, p.y + h * 0.75f), col);
|
||||
ImGui::Dummy(ImVec2(h * 0.5f, h));
|
||||
ImGui::SameLine(0.0f, ImGui::GetStyle().ItemInnerSpacing.x);
|
||||
}
|
||||
|
||||
// A gridline spacing of 10^k, 2*10^k or 5*10^k GiB that draws at most five.
|
||||
double tick_step(double max_gib) {
|
||||
double step = std::pow(10.0, std::floor(std::log10(std::max(max_gib, 1e-3))));
|
||||
for (double m : {1.0, 2.0, 5.0, 10.0})
|
||||
if (max_gib / (step * m) <= 5.0) return step * m;
|
||||
return step * 10.0;
|
||||
}
|
||||
|
||||
void chart(const VramForecast& v, int total_steps) {
|
||||
ImDrawList* dl = ImGui::GetWindowDrawList();
|
||||
const ImGuiStyle& st = ImGui::GetStyle();
|
||||
const float line = ImGui::GetTextLineHeight();
|
||||
const float W = px(460.0f), H = px(170.0f);
|
||||
const ImVec2 p0 = ImGui::GetCursorScreenPos();
|
||||
ImGui::Dummy(ImVec2(W, H + line + st.ItemInnerSpacing.y));
|
||||
|
||||
double y_max = v.total_bytes;
|
||||
for (const auto& s : v.history) y_max = std::max(y_max, s.ours + s.others);
|
||||
for (const auto& b : v.projection)
|
||||
y_max = std::max(y_max, v.others_bytes + b.mean + 2.0 * b.sigma);
|
||||
y_max = std::max(y_max * 1.05, 1.0);
|
||||
const double T = std::max(1, total_steps);
|
||||
|
||||
const float label_w = ImGui::CalcTextSize("00.0").x + st.ItemInnerSpacing.x;
|
||||
const ImVec2 r0(p0.x + label_w, p0.y);
|
||||
const ImVec2 r1(p0.x + W, p0.y + H);
|
||||
auto X = [&](double step) {
|
||||
return r0.x + (r1.x - r0.x) * (float)std::min(1.0, std::max(0.0, step / T));
|
||||
};
|
||||
auto Y = [&](double bytes) {
|
||||
return r1.y - (r1.y - r0.y) * (float)std::min(1.0, std::max(0.0, bytes / y_max));
|
||||
};
|
||||
|
||||
dl->AddRectFilled(r0, r1, ImGui::GetColorU32(ImGuiCol_FrameBg));
|
||||
const ImU32 grid = ImGui::GetColorU32(ImGuiCol_Border);
|
||||
const ImU32 text_dim = ImGui::GetColorU32(ImGuiCol_TextDisabled);
|
||||
const double tick = tick_step(y_max / kGiB);
|
||||
for (double g = 0.0; g * kGiB <= y_max; g += tick) {
|
||||
const float y = Y(g * kGiB);
|
||||
dl->AddLine(ImVec2(r0.x, y), ImVec2(r1.x, y), grid);
|
||||
char s[16];
|
||||
std::snprintf(s, sizeof s, tick < 1.0 ? "%.1f" : "%.0f", g);
|
||||
const ImVec2 sz = ImGui::CalcTextSize(s);
|
||||
dl->AddText(ImVec2(r0.x - sz.x - st.ItemInnerSpacing.x, y - sz.y * 0.5f),
|
||||
text_dim, s);
|
||||
}
|
||||
|
||||
// Other programs at the bottom, this run stacked on top: the gap to the
|
||||
// capacity line is what the run has left.
|
||||
const ImU32 others_fill = with_alpha(kOthers, 0.35f);
|
||||
for (size_t i = 1; i < v.history.size(); ++i) {
|
||||
const auto& a = v.history[i - 1];
|
||||
const auto& b = v.history[i];
|
||||
dl->AddQuadFilled(ImVec2(X(a.step), Y(0)), ImVec2(X(a.step), Y(a.others)),
|
||||
ImVec2(X(b.step), Y(b.others)), ImVec2(X(b.step), Y(0)),
|
||||
others_fill);
|
||||
}
|
||||
const double last_step = v.history.empty() ? 0.0 : v.history.back().step;
|
||||
if (!v.projection.empty())
|
||||
dl->AddRectFilled(ImVec2(X(last_step), Y(v.others_bytes)), ImVec2(r1.x, Y(0)),
|
||||
others_fill);
|
||||
|
||||
const ImVec4 run = ImGui::GetStyleColorVec4(ImGuiCol_PlotLines);
|
||||
for (size_t i = 1; i < v.projection.size(); ++i) {
|
||||
const auto& a = v.projection[i - 1];
|
||||
const auto& b = v.projection[i];
|
||||
const double o = v.others_bytes;
|
||||
dl->AddQuadFilled(ImVec2(X(a.step), Y(o + a.mean + 2.0 * a.sigma)),
|
||||
ImVec2(X(b.step), Y(o + b.mean + 2.0 * b.sigma)),
|
||||
ImVec2(X(b.step), Y(o + b.mean - 2.0 * b.sigma)),
|
||||
ImVec2(X(a.step), Y(o + a.mean - 2.0 * a.sigma)),
|
||||
with_alpha(run, 0.22f));
|
||||
dashed_line(dl, ImVec2(X(a.step), Y(o + a.mean)),
|
||||
ImVec2(X(b.step), Y(o + b.mean)), with_alpha(run, 0.9f), px(1.5f));
|
||||
}
|
||||
for (size_t i = 1; i < v.history.size(); ++i) {
|
||||
const auto& a = v.history[i - 1];
|
||||
const auto& b = v.history[i];
|
||||
dl->AddLine(ImVec2(X(a.step), Y(a.ours + a.others)),
|
||||
ImVec2(X(b.step), Y(b.ours + b.others)),
|
||||
ImGui::GetColorU32(run), px(2.0f));
|
||||
}
|
||||
if (!v.history.empty())
|
||||
dl->AddLine(ImVec2(X(last_step), r0.y), ImVec2(X(last_step), r1.y), grid);
|
||||
if (v.total_bytes > 0.0)
|
||||
dl->AddLine(ImVec2(r0.x, Y(v.total_bytes)), ImVec2(r1.x, Y(v.total_bytes)),
|
||||
ImGui::GetColorU32(kErr), px(1.5f));
|
||||
dl->AddRect(r0, r1, grid);
|
||||
|
||||
char s[32];
|
||||
dl->AddText(ImVec2(r0.x, r1.y + st.ItemInnerSpacing.y), text_dim, "0");
|
||||
std::snprintf(s, sizeof s, "%d", total_steps);
|
||||
dl->AddText(ImVec2(r1.x - ImGui::CalcTextSize(s).x, r1.y + st.ItemInnerSpacing.y),
|
||||
text_dim, s);
|
||||
|
||||
swatch(ImGui::GetColorU32(run));
|
||||
ui::TextDisabled(msg::vram_legend_run);
|
||||
ImGui::SameLine();
|
||||
swatch(with_alpha(run, 0.35f));
|
||||
ui::TextDisabled(msg::vram_legend_projected);
|
||||
ImGui::SameLine();
|
||||
swatch(others_fill);
|
||||
ui::TextDisabled(msg::vram_legend_others);
|
||||
ImGui::SameLine();
|
||||
swatch(ImGui::GetColorU32(kErr));
|
||||
ui::TextDisabled(msg::vram_legend_capacity);
|
||||
}
|
||||
|
||||
void breakdown(const VramForecast& v) {
|
||||
struct Row { const spirula::i18n::Msg* label; double now; double peak; ImVec4 color; };
|
||||
const ImVec4 bar = ImGui::GetStyleColorVec4(ImGuiCol_PlotHistogram);
|
||||
const auto cat = [&](VramCategory c) { return v.category[(int)c]; };
|
||||
const double sxi = cat(VramCategory::SplatXImg);
|
||||
const double sxi_peak = std::max(sxi, v.grow_peak_mean - v.scratch);
|
||||
const Row rows[] = {
|
||||
{&msg::vram_cat_splat, cat(VramCategory::Splat), 0.0, bar},
|
||||
{&msg::vram_cat_splat_x_img, sxi, v.valid ? sxi_peak : 0.0, bar},
|
||||
{&msg::vram_cat_image, cat(VramCategory::Image), 0.0, bar},
|
||||
{&msg::vram_cat_appearance, cat(VramCategory::Appearance), 0.0, bar},
|
||||
{&msg::vram_cat_viewer, cat(VramCategory::Viewer), 0.0, bar},
|
||||
{&msg::vram_cat_other, cat(VramCategory::Other), 0.0, bar},
|
||||
{&msg::vram_cat_scratch, v.scratch, 0.0, bar},
|
||||
{&msg::vram_cat_unpooled, v.unpooled, 0.0, bar},
|
||||
{&msg::vram_legend_others, v.others_bytes, 0.0, kOthers},
|
||||
};
|
||||
const double shown = 1024.0 * 1024.0; // below 1 MiB a row is noise
|
||||
double scale = 1.0;
|
||||
float label_w = 0.0f;
|
||||
for (const Row& r : rows) {
|
||||
if (std::max(r.now, r.peak) < shown) continue;
|
||||
scale = std::max(scale, std::max(r.now, r.peak));
|
||||
label_w = std::max(label_w, ImGui::CalcTextSize(r.label->get()).x);
|
||||
}
|
||||
|
||||
ImDrawList* dl = ImGui::GetWindowDrawList();
|
||||
const ImGuiStyle& st = ImGui::GetStyle();
|
||||
const float bar_w = px(220.0f);
|
||||
const float h = ImGui::GetTextLineHeight();
|
||||
bool faded = false;
|
||||
for (const Row& r : rows) {
|
||||
if (std::max(r.now, r.peak) < shown) continue;
|
||||
ui::Text(*r.label);
|
||||
ImGui::SameLine(label_w + st.ItemSpacing.x * 2.0f);
|
||||
const ImVec2 p = ImGui::GetCursorScreenPos();
|
||||
const float w_now = bar_w * (float)(r.now / scale);
|
||||
const float w_peak = bar_w * (float)(r.peak / scale);
|
||||
dl->AddRectFilled(p, ImVec2(p.x + bar_w, p.y + h),
|
||||
ImGui::GetColorU32(ImGuiCol_FrameBg));
|
||||
if (w_peak > w_now) {
|
||||
dl->AddRectFilled(ImVec2(p.x + w_now, p.y), ImVec2(p.x + w_peak, p.y + h),
|
||||
with_alpha(r.color, 0.3f));
|
||||
faded = true;
|
||||
}
|
||||
dl->AddRectFilled(p, ImVec2(p.x + w_now, p.y + h), ImGui::GetColorU32(r.color));
|
||||
ImGui::Dummy(ImVec2(bar_w, h));
|
||||
ImGui::SameLine();
|
||||
if (r.peak > r.now)
|
||||
ui::TextRaw(gib(r.now) + " -> " + gib(r.peak));
|
||||
else
|
||||
ui::TextRaw(gib(r.now));
|
||||
}
|
||||
if (faded) ui::TextDisabled(msg::vram_breakdown_growth);
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
const spirula::i18n::Msg* oom_risk_label(OomRisk r) {
|
||||
switch (r) {
|
||||
case OomRisk::Low: return &msg::oom_risk_low;
|
||||
case OomRisk::Medium: return &msg::oom_risk_medium;
|
||||
case OomRisk::High: return &msg::oom_risk_high;
|
||||
default: return nullptr;
|
||||
}
|
||||
}
|
||||
|
||||
ImVec4 oom_risk_color(OomRisk r) {
|
||||
return r == OomRisk::High ? kErr : r == OomRisk::Medium ? kWarn : kOk;
|
||||
}
|
||||
|
||||
void vram_forecast_card(const VramForecast& v, int total_steps) {
|
||||
ui::Text(msg::vram_chart_title);
|
||||
chart(v, total_steps);
|
||||
if (v.valid && !v.projection.empty() && v.total_bytes > 0.0) {
|
||||
char pct[16];
|
||||
std::snprintf(pct, sizeof pct, "%.0f", v.p_oom * 100.0);
|
||||
ui::TextColoredWrapped(oom_risk_color(v.risk), msg::vram_chart_peak,
|
||||
{gib(v.peak_mean), gib(v.peak_sigma),
|
||||
gib(0.99 * v.total_bytes - v.others_bytes),
|
||||
std::string(pct)});
|
||||
if (v.provisional) ui::TextDisabledWrapped(msg::vram_chart_provisional);
|
||||
} else if (!v.valid) {
|
||||
ui::TextDisabledWrapped(msg::vram_chart_waiting);
|
||||
}
|
||||
ImGui::Spacing();
|
||||
ui::Text(msg::vram_breakdown_title);
|
||||
breakdown(v);
|
||||
}
|
||||
|
||||
} // namespace gui
|
||||
@@ -0,0 +1,19 @@
|
||||
#pragma once
|
||||
|
||||
// The trainer's VRAM forecast on screen: the out-of-memory risk tag beside the
|
||||
// status strip's VRAM bar, and the card that bar shows on hover.
|
||||
|
||||
#include "app/TrainForecast.h"
|
||||
#include "app/gui/Ui.h"
|
||||
|
||||
namespace gui {
|
||||
|
||||
// nullptr while the risk is unknown.
|
||||
const spirula::i18n::Msg* oom_risk_label(spirula::OomRisk r);
|
||||
ImVec4 oom_risk_color(spirula::OomRisk r);
|
||||
|
||||
// Memory over the run with its projection, then this run's usage by category.
|
||||
// Call inside a tooltip.
|
||||
void vram_forecast_card(const spirula::VramForecast& v, int total_steps);
|
||||
|
||||
} // namespace gui
|
||||
@@ -22,13 +22,15 @@ namespace backend {
|
||||
// counter/table are shared across all translation units. Allocation is pooled
|
||||
// upstream (Tensor.h), so this map is touched rarely, not per frame.
|
||||
namespace detail {
|
||||
// Leaked on purpose: DevicePool::global() is constructed before the first
|
||||
// device_malloc, so it is destroyed after these and frees through them.
|
||||
inline std::mutex& alloc_mutex() {
|
||||
static std::mutex m;
|
||||
return m;
|
||||
static std::mutex* m = new std::mutex();
|
||||
return *m;
|
||||
}
|
||||
inline std::unordered_map<void*, size_t>& alloc_sizes() {
|
||||
static std::unordered_map<void*, size_t> m;
|
||||
return m;
|
||||
static auto* m = new std::unordered_map<void*, size_t>();
|
||||
return *m;
|
||||
}
|
||||
inline std::atomic<uint64_t>& device_bytes() {
|
||||
static std::atomic<uint64_t> v{0};
|
||||
|
||||
@@ -0,0 +1,197 @@
|
||||
// TrainForecast against synthetic runs whose true time and VRAM are known.
|
||||
|
||||
#include "app/TrainForecast.h"
|
||||
|
||||
#include <algorithm>
|
||||
#include <cmath>
|
||||
#include <cstdio>
|
||||
#include <random>
|
||||
|
||||
using namespace spirula;
|
||||
|
||||
static int g_fail = 0;
|
||||
#define CHECK(cond, ...) \
|
||||
do { \
|
||||
if (!(cond)) { \
|
||||
std::printf("FAIL %s:%d: ", __func__, __LINE__); \
|
||||
std::printf(__VA_ARGS__); \
|
||||
std::printf("\n"); \
|
||||
++g_fail; \
|
||||
} \
|
||||
} while (0)
|
||||
|
||||
static DensifyConfig densify(int start, int stop, int every, float growth) {
|
||||
DensifyConfig d;
|
||||
d.refine_start_iter = start;
|
||||
d.refine_stop_iter = stop;
|
||||
d.refine_stop_num_iter = 1 << 20; // leaves refine_stop_iter in charge
|
||||
d.refine_every = every;
|
||||
d.growth_factor = growth;
|
||||
return d;
|
||||
}
|
||||
|
||||
// The replay must agree with stepping densify_grows_at / densify_target.
|
||||
static void schedule_matches_engine_rule() {
|
||||
const DensifyConfig d = densify(500, 15000, 100, 1.05f);
|
||||
const int T = 30000;
|
||||
const int64_t cap = 1000000;
|
||||
const SplatSchedule sch(d, T, cap);
|
||||
for (int from : {0, 499, 501, 7300, 20000}) {
|
||||
int64_t n = 200000;
|
||||
for (int s = 0; s < from; ++s)
|
||||
if (densify_grows_at(d, s, T)) n = densify_target(d, n, cap);
|
||||
std::vector<int64_t> want(T - from);
|
||||
int64_t m = n;
|
||||
for (int s = from; s < T; ++s) {
|
||||
want[s - from] = m;
|
||||
if (densify_grows_at(d, s, T)) m = densify_target(d, m, cap);
|
||||
}
|
||||
int covered = from;
|
||||
bool ok = true;
|
||||
sch.segments(from, n, [&](int f, int e, int64_t k) {
|
||||
ok = ok && f == covered;
|
||||
for (int s = f; s < e; ++s) ok = ok && want[s - from] == k;
|
||||
covered = e;
|
||||
});
|
||||
CHECK(ok && covered == T, "segments diverge from the engine rule, from=%d", from);
|
||||
CHECK(sch.final_count(from, n) == m, "final count %lld vs %lld",
|
||||
(long long)sch.final_count(from, n), (long long)m);
|
||||
}
|
||||
CHECK(sch.first_densify(0) == 600, "first densify %d", sch.first_densify(0));
|
||||
CHECK(sch.first_densify(15000) == -1, "densify past the stop");
|
||||
}
|
||||
|
||||
// Step time = 20 ms + (40 ms + 3 ms per SH coefficient) per Msplat: the
|
||||
// naive mean underestimates badly while the count and the SH degree are still
|
||||
// growing, the model must not.
|
||||
static void eta_tracks_growth() {
|
||||
const DensifyConfig d = densify(500, 15000, 100, 1.05f);
|
||||
const int T = 30000;
|
||||
const int64_t cap = 3000000;
|
||||
const SplatSchedule sch(d, T, cap);
|
||||
ForecastSetup fs;
|
||||
fs.schedule = sch;
|
||||
fs.sh_degree = 3;
|
||||
fs.sh_degree_every = 1000;
|
||||
TrainForecast fc;
|
||||
fc.reset(fs);
|
||||
std::mt19937 rng(1);
|
||||
std::normal_distribution<double> noise(1.0, 0.1);
|
||||
const double a = 0.020, b = 0.040, b_coeff = 0.003;
|
||||
auto coeffs = [](int s) {
|
||||
const int d = std::min(s / 1000, 3);
|
||||
return (d + 1) * (d + 1) - 1;
|
||||
};
|
||||
|
||||
std::vector<int64_t> n_at(T);
|
||||
sch.segments(0, 300000, [&](int f, int e, int64_t n) {
|
||||
for (int s = f; s < e; ++s) n_at[s] = n;
|
||||
});
|
||||
std::vector<double> t_at(T);
|
||||
for (int s = 0; s < T; ++s)
|
||||
t_at[s] = a + (b + b_coeff * coeffs(s)) * n_at[s] / 1e6;
|
||||
auto remaining = [&](int from) {
|
||||
double sum = 0.0;
|
||||
for (int s = from; s < T; ++s) sum += t_at[s];
|
||||
return sum;
|
||||
};
|
||||
|
||||
double recent = 0.0;
|
||||
for (int s = 0; s < T; ++s) {
|
||||
const double wall = t_at[s] * noise(rng);
|
||||
const double gpu = s % 10 == 0 ? (t_at[s] - a) * noise(rng) : -1.0;
|
||||
fc.add_step(s, wall, n_at[s], gpu);
|
||||
recent = s < 100 ? wall : recent + (wall - recent) / 100.0;
|
||||
if (s == 400 || s == 1500 || s == 2500 || s == 8000) {
|
||||
const double truth = remaining(s + 1);
|
||||
const EtaForecast e = fc.eta(s + 1, n_at[s + 1]);
|
||||
const double naive = recent * (T - s - 1);
|
||||
std::printf(" eta @%d: model %.0f s (sigma %.0f), naive %.0f s, truth %.0f s\n",
|
||||
s, e.seconds, e.sigma, naive, truth);
|
||||
// Before the count moves only the prior speaks, so all it can be
|
||||
// held to is its own sigma -- and beating the naive estimate.
|
||||
const bool grown = s > 1000;
|
||||
// Past ~5300 the count sits at the cap and naive is exact too.
|
||||
if (s < 5000)
|
||||
CHECK(std::fabs(e.seconds - truth) < std::fabs(naive - truth),
|
||||
"eta at step %d no better than naive", s);
|
||||
CHECK(std::fabs(e.seconds - truth) <
|
||||
(grown ? std::max(0.1 * truth, 2.0 * e.sigma) : 3.0 * e.sigma),
|
||||
"eta at step %d off by %.1f%%", s, 100.0 * (e.seconds / truth - 1.0));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// splat x img follows 60 bytes per splat with per-step spread; the rest is 1 GiB.
|
||||
static VramForecast run_vram(double others_gib, int stop_at) {
|
||||
const DensifyConfig d = densify(500, 15000, 100, 1.05f);
|
||||
const int T = 30000;
|
||||
const int64_t cap = 3000000;
|
||||
const SplatSchedule sch(d, T, cap);
|
||||
ForecastSetup fs;
|
||||
fs.schedule = sch;
|
||||
fs.distinct_batches = 200;
|
||||
TrainForecast fc;
|
||||
fc.reset(fs);
|
||||
std::mt19937 rng(7);
|
||||
std::vector<double> view_ratio(200);
|
||||
std::lognormal_distribution<double> spread(0.0, 0.2);
|
||||
for (double& r : view_ratio) r = 60.0 * spread(rng);
|
||||
std::uniform_int_distribution<int> pick(0, 199);
|
||||
|
||||
const double gib = 1024.0 * 1024.0 * 1024.0;
|
||||
std::vector<int64_t> n_at(T + 1);
|
||||
sch.segments(0, 300000, [&](int f, int e, int64_t n) {
|
||||
for (int s = f; s < e; ++s) n_at[s] = n;
|
||||
});
|
||||
n_at[T] = n_at[T - 1];
|
||||
size_t hw = 0;
|
||||
VramForecast at_stop;
|
||||
for (int s = 0; s < T; ++s) {
|
||||
const size_t used = (size_t)(n_at[s] * view_ratio[pick(rng)]);
|
||||
hw = std::max(hw, used);
|
||||
MemorySample m;
|
||||
m.step = s;
|
||||
m.splats_ran = n_at[s];
|
||||
m.splats_next = n_at[s + 1];
|
||||
m.pool_used[(int)VramCategory::SplatXImg] = used;
|
||||
m.pool_cap[(int)VramCategory::SplatXImg] = hw;
|
||||
m.pool_cap[(int)VramCategory::Splat] = (size_t)gib;
|
||||
m.has_process = m.has_used = m.has_total = true;
|
||||
m.process_bytes = (uint64_t)gib + hw;
|
||||
m.used_bytes = m.process_bytes + (uint64_t)(others_gib * gib);
|
||||
m.total_bytes = (uint64_t)(8.0 * gib);
|
||||
fc.add_memory(m);
|
||||
if (s == stop_at) at_stop = fc.vram();
|
||||
}
|
||||
const double truth = gib + (double)hw;
|
||||
std::printf(" vram @%d: peak %.3f +- %.3f GiB, truth %.3f GiB, p_oom %.3f\n",
|
||||
stop_at, at_stop.peak_mean / gib, at_stop.peak_sigma / gib,
|
||||
truth / gib, at_stop.p_oom);
|
||||
CHECK(at_stop.valid, "no forecast at step %d", stop_at);
|
||||
CHECK(std::fabs(at_stop.peak_mean - truth) < 3.0 * at_stop.peak_sigma + 0.02 * truth,
|
||||
"peak off: %.3f vs %.3f GiB", at_stop.peak_mean / gib, truth / gib);
|
||||
return at_stop;
|
||||
}
|
||||
|
||||
static void vram_predicts_peak() {
|
||||
const VramForecast early = run_vram(1.0, 1000);
|
||||
CHECK(early.risk == OomRisk::Low, "risk %d with 6 GiB spare", (int)early.risk);
|
||||
const VramForecast late = run_vram(1.0, 10000);
|
||||
CHECK(late.peak_sigma <= early.peak_sigma, "sigma grew with more data");
|
||||
// 8 GiB device, 6.8 GiB elsewhere: 1.2 GiB for a ~1.2 GiB peak.
|
||||
const VramForecast tight = run_vram(6.8, 1000);
|
||||
CHECK(tight.risk >= OomRisk::Medium, "risk %d on a full device", (int)tight.risk);
|
||||
}
|
||||
|
||||
int main() {
|
||||
schedule_matches_engine_rule();
|
||||
eta_tracks_growth();
|
||||
vram_predicts_peak();
|
||||
if (g_fail) {
|
||||
std::printf("train_forecast_test: %d failure(s)\n", g_fail);
|
||||
return 1;
|
||||
}
|
||||
std::printf("train_forecast_test: OK\n");
|
||||
return 0;
|
||||
}
|
||||
+83
-19
@@ -50,9 +50,9 @@
|
||||
// other.
|
||||
enum class VramCategory : uint8_t {
|
||||
Splat = 0, // per-splat params, gradients, optimizer state, densify aux
|
||||
// Scratch sized by (camera, gaussian), visible or splat-tile PAIRS, so it
|
||||
// grows with the batch too. A buffer sized [N] is Splat however late it is
|
||||
// allocated -- docs/notes/vram-splat-x-img.md.
|
||||
// Scratch sized by (camera, gaussian), visible or splat-tile PAIRS: the
|
||||
// one category that grows during training. Splat is sized for cap_max
|
||||
// (POOL_LIVE_SPLAT_TABLE) -- docs/notes/vram-splat-x-img.md.
|
||||
SplatXImg,
|
||||
Image, // per-image buffers: render outputs, GT, loss maps, img grads
|
||||
Appearance, // bilagrid / background-SH / PPISP / color-space / color-shift
|
||||
@@ -316,27 +316,27 @@ enum class SaveClass : uint8_t {
|
||||
/* ---- densify scratch ---- */ \
|
||||
X(DensifyQuantileTemp , "densify_quantile_temp", Other , Never) \
|
||||
X(DensifyMapNorm , "densify_map_norm", Other , Never) \
|
||||
X(DensifyScoreGather , "densify_score_gather", Other , Never) \
|
||||
X(DensifyScoreGather , "densify_score_gather", Splat , Never) \
|
||||
X(DensifyScoreClip , "densify_score_clip", Other , Never) \
|
||||
X(DensifyClipScale , "densify_clip_scale", Other , Never) \
|
||||
X(DensifyUpdateWeight , "densify_update_weight", Other , Never) \
|
||||
X(DensifyWswrSortingValues , "densify_wswr_sorting_values", Other , Never) \
|
||||
X(DensifyWswrOutIdx , "densify_wswr_out_idx", Other , Never) \
|
||||
X(DensifyWswrKeysOut , "densify_wswr_keys_out", Other , Never) \
|
||||
X(DensifyWswrIndicesIn , "densify_wswr_indices_in", Other , Never) \
|
||||
X(DensifyWswrIndicesOut , "densify_wswr_indices_out", Other , Never) \
|
||||
X(DensifyWswrSortingValues , "densify_wswr_sorting_values", Splat , Never) \
|
||||
X(DensifyWswrOutIdx , "densify_wswr_out_idx", Splat , Never) \
|
||||
X(DensifyWswrKeysOut , "densify_wswr_keys_out", Splat , Never) \
|
||||
X(DensifyWswrIndicesIn , "densify_wswr_indices_in", Splat , Never) \
|
||||
X(DensifyWswrIndicesOut , "densify_wswr_indices_out", Splat , Never) \
|
||||
X(DensifyWswrEligible , "densify_wswr_eligible", Other , Never) \
|
||||
X(DensifyRelocMask , "densify_reloc_mask", Other , Never) \
|
||||
X(DensifyRelocMask , "densify_reloc_mask", Splat , Never) \
|
||||
X(DensifyRelocCount , "densify_reloc_count", Other , Never) \
|
||||
X(DensifyRelocDstIndices , "densify_reloc_dst_indices", Other , Never) \
|
||||
X(DensifyMcmcSampleProbs , "densify_mcmc_sample_probs", Other , Never) \
|
||||
X(DensifyMcmcSampleProbsCumsum , "densify_mcmc_sample_probs_cumsum", Other , Never) \
|
||||
X(DensifyMcmcIndexMap , "densify_mcmc_index_map", Other , Never) \
|
||||
X(DensifyMcmcNIdxBuffer , "densify_mcmc_n_idx_buffer", Other , Never) \
|
||||
X(DensifyMcmcAddSampleProbs , "densify_mcmc_add_sample_probs", Other , Never) \
|
||||
X(DensifyMcmcAddSampleProbsCumsum, "densify_mcmc_add_sample_probs_cumsum",Other , Never) \
|
||||
X(DensifyMcmcAddIndexMap , "densify_mcmc_add_index_map", Other , Never) \
|
||||
X(DensifyMcmcAddNIdxBuffer , "densify_mcmc_add_n_idx_buffer", Other , Never) \
|
||||
X(DensifyRelocDstIndices , "densify_reloc_dst_indices", Splat , Never) \
|
||||
X(DensifyMcmcSampleProbs , "densify_mcmc_sample_probs", Splat , Never) \
|
||||
X(DensifyMcmcSampleProbsCumsum , "densify_mcmc_sample_probs_cumsum", Splat , Never) \
|
||||
X(DensifyMcmcIndexMap , "densify_mcmc_index_map", Splat , Never) \
|
||||
X(DensifyMcmcNIdxBuffer , "densify_mcmc_n_idx_buffer", Splat , Never) \
|
||||
X(DensifyMcmcAddSampleProbs , "densify_mcmc_add_sample_probs", Splat , Never) \
|
||||
X(DensifyMcmcAddSampleProbsCumsum, "densify_mcmc_add_sample_probs_cumsum",Splat , Never) \
|
||||
X(DensifyMcmcAddIndexMap , "densify_mcmc_add_index_map", Splat , Never) \
|
||||
X(DensifyMcmcAddNIdxBuffer , "densify_mcmc_add_n_idx_buffer", Splat , Never) \
|
||||
X(DensifyRobustResid , "densify_robust_resid", Other , Never) \
|
||||
X(DensifyTukeyC , "densify_tukey_c", Other , Never) \
|
||||
/* ---- viewer cache/scratch ---- */ \
|
||||
@@ -510,6 +510,70 @@ constexpr bool ce_alias_rows_unique() {
|
||||
static_assert(ce_alias_rows_unique(),
|
||||
"POOL_ALIAS_TABLE: a slot is listed twice");
|
||||
|
||||
// ---- Slots sized by the live splat count ----------------------------------
|
||||
|
||||
// Requested at a size proportional to the LIVE count, so the pool allocates
|
||||
// them for cap_max on first use: Splat then stops growing as the model
|
||||
// densifies, which the training VRAM forecast assumes.
|
||||
#define POOL_LIVE_SPLAT_TABLE(X) \
|
||||
X(EngSubbatchAccumWeightSum) X(EngDensifyOversizeWeight) \
|
||||
X(RasterBwdAccumWeight) X(RasterBwdVWorld) \
|
||||
X(FusedProjBwdCamBounds) X(DensifyScoreGather) \
|
||||
X(DensifyWswrSortingValues) X(DensifyWswrKeysOut) \
|
||||
X(DensifyWswrIndicesIn) X(DensifyWswrIndicesOut) \
|
||||
X(DensifyRelocMask) X(DensifyRelocDstIndices) \
|
||||
X(DensifyMcmcSampleProbs) X(DensifyMcmcSampleProbsCumsum) \
|
||||
X(DensifyMcmcIndexMap) X(DensifyMcmcNIdxBuffer) \
|
||||
X(DensifyMcmcAddSampleProbs) X(DensifyMcmcAddSampleProbsCumsum) \
|
||||
X(DensifyMcmcAddNIdxBuffer)
|
||||
|
||||
// One int32 per drawn splat: the draw's size varies from call to call and is
|
||||
// bounded only by the live count, so these get cap_max elements outright.
|
||||
#define POOL_SPLAT_DRAW_TABLE(X) \
|
||||
X(DensifyWswrOutIdx) X(DensifyMcmcAddIndexMap)
|
||||
|
||||
enum class LiveSizing : uint8_t { None, Proportional, Draw };
|
||||
|
||||
inline constexpr PoolSlot kLiveSplatSlots[] = {
|
||||
#define X(name) PoolSlot::name,
|
||||
POOL_LIVE_SPLAT_TABLE(X)
|
||||
#undef X
|
||||
};
|
||||
inline constexpr PoolSlot kSplatDrawSlots[] = {
|
||||
#define X(name) PoolSlot::name,
|
||||
POOL_SPLAT_DRAW_TABLE(X)
|
||||
#undef X
|
||||
};
|
||||
|
||||
constexpr LiveSizing slot_live_sizing(PoolSlot s) {
|
||||
for (PoolSlot r : kLiveSplatSlots)
|
||||
if (r == s) return LiveSizing::Proportional;
|
||||
for (PoolSlot r : kSplatDrawSlots)
|
||||
if (r == s) return LiveSizing::Draw;
|
||||
return LiveSizing::None;
|
||||
}
|
||||
|
||||
constexpr bool ce_live_splat_rows_are_splat() {
|
||||
for (PoolSlot r : kLiveSplatSlots)
|
||||
if (slot_category(r) != VramCategory::Splat || slot_phase(r) != PoolPhase::None)
|
||||
return false;
|
||||
for (PoolSlot r : kSplatDrawSlots)
|
||||
if (slot_category(r) != VramCategory::Splat || slot_phase(r) != PoolPhase::None)
|
||||
return false;
|
||||
return true;
|
||||
}
|
||||
static_assert(ce_live_splat_rows_are_splat(),
|
||||
"POOL_LIVE_SPLAT_TABLE: rows must be Splat and own their memory");
|
||||
|
||||
// DevicePool::category_bytes() books the whole arena to SplatXImg.
|
||||
constexpr bool ce_alias_rows_grow() {
|
||||
for (const AliasRow& r : kAliasRows)
|
||||
if (slot_category(r.slot) != VramCategory::SplatXImg) return false;
|
||||
return true;
|
||||
}
|
||||
static_assert(ce_alias_rows_grow(),
|
||||
"POOL_ALIAS_TABLE: an arena-backed slot must be SplatXImg");
|
||||
|
||||
constexpr const char* to_string(PoolPhase p) {
|
||||
switch (p) {
|
||||
case PoolPhase::None: return "none";
|
||||
|
||||
+51
-4
@@ -8,6 +8,7 @@
|
||||
#include "backend/api/BackendRuntime.h"
|
||||
#include "core/Env.h"
|
||||
|
||||
#include <algorithm>
|
||||
#include <mutex>
|
||||
#include <tuple>
|
||||
#include <type_traits>
|
||||
@@ -181,6 +182,8 @@ class DevicePool {
|
||||
Arena _arena;
|
||||
PoolPhase _phase = PoolPhase::None;
|
||||
uint32_t _epoch = 0; // bumped per begin_phase; 0 never matches a Slot
|
||||
int64_t _live_splats = 0;
|
||||
int64_t _cap_splats = 0;
|
||||
// Guards _slots / _dyn / _arena against concurrent acquire. It does NOT
|
||||
// make the phase protocol thread-safe: two threads in different phases
|
||||
// would fight over the arena, which the engine mutex is what prevents.
|
||||
@@ -219,8 +222,9 @@ class DevicePool {
|
||||
}
|
||||
|
||||
// Grow-if-needed on a Slot already selected by the caller (mutex held).
|
||||
// `reserve` > n allocates room for that many elements up front.
|
||||
template<typename T>
|
||||
static T* _acquire_into(Slot& slot, size_t n) {
|
||||
static T* _acquire_into(Slot& slot, size_t n, size_t reserve = 0) {
|
||||
size_t bytes = n * sizeof(T);
|
||||
// An arena slice is not capacity this slot may reuse; drop it first.
|
||||
if (!slot.owns) { slot.ptr = nullptr; slot.cap_bytes = 0; slot.owns = true; }
|
||||
@@ -230,12 +234,13 @@ class DevicePool {
|
||||
// in a consistent empty state rather than {null ptr, stale cap}.
|
||||
slot.ptr = nullptr;
|
||||
slot.cap_bytes = 0;
|
||||
slot.ptr = backend::device_malloc_checked(bytes);
|
||||
slot.cap_bytes = bytes;
|
||||
const size_t alloc = std::max(bytes, reserve * sizeof(T));
|
||||
slot.ptr = backend::device_malloc_checked(alloc);
|
||||
slot.cap_bytes = alloc;
|
||||
// SS_POISON_POOL=1: fresh device memory holds whatever the
|
||||
// driver left, so a read-before-write is silent where that is
|
||||
// zero and catastrophic where it is not. NaN makes it show.
|
||||
if (_poison_pool()) backend::memset_sync(slot.ptr, 0xff, bytes);
|
||||
if (_poison_pool()) backend::memset_sync(slot.ptr, 0xff, alloc);
|
||||
}
|
||||
slot.owns = true;
|
||||
slot.used_bytes = bytes;
|
||||
@@ -333,10 +338,27 @@ public:
|
||||
T* acquire(PoolKey key, size_t n) {
|
||||
std::lock_guard<std::mutex> lock(_mu);
|
||||
const PoolSlot slot = pool_key_slot(key);
|
||||
const LiveSizing live = slot_live_sizing(slot);
|
||||
if (live != LiveSizing::None && _live_splats > 0 &&
|
||||
_cap_splats > _live_splats && n > 0) {
|
||||
const size_t reserve = live == LiveSizing::Draw
|
||||
? (size_t)_cap_splats
|
||||
: (size_t)((double)n * (double)_cap_splats / (double)_live_splats + 1.0);
|
||||
return _acquire_into<T>(_slots[key], n, reserve);
|
||||
}
|
||||
return _acquire_checked<T>(_slots[key], n, slot_phase(slot),
|
||||
slot_name(slot));
|
||||
}
|
||||
|
||||
// The engine's live and cap splat counts, which size the slots in
|
||||
// POOL_LIVE_SPLAT_TABLE and POOL_SPLAT_DRAW_TABLE; the engine updates
|
||||
// them wherever it changes either. 0 disables the reservation.
|
||||
void set_splat_counts(int64_t live, int64_t cap) {
|
||||
std::lock_guard<std::mutex> lock(_mu);
|
||||
_live_splats = live;
|
||||
_cap_splats = cap;
|
||||
}
|
||||
|
||||
// Convenience: acquire a (slot, sub) buffer. sub defaults to the main slot.
|
||||
template<typename T>
|
||||
T* acquire(PoolSlot slot, size_t n, uint32_t sub = kSubMain) {
|
||||
@@ -401,6 +423,7 @@ public:
|
||||
if (_arena.ptr) backend::device_free(_arena.ptr);
|
||||
_arena = Arena{};
|
||||
_phase = PoolPhase::None;
|
||||
_live_splats = _cap_splats = 0;
|
||||
}
|
||||
|
||||
// Total bytes allocated (capacity, not logical size).
|
||||
@@ -412,6 +435,30 @@ public:
|
||||
return total;
|
||||
}
|
||||
|
||||
// Per category, without building names: cheap enough to call every step.
|
||||
struct CategoryBytes {
|
||||
size_t used[(int)VramCategory::Count] = {};
|
||||
size_t cap[(int)VramCategory::Count] = {};
|
||||
};
|
||||
CategoryBytes category_bytes() const {
|
||||
std::lock_guard<std::mutex> lock(_mu);
|
||||
CategoryBytes out;
|
||||
for (uint32_t i = 0; i < _slots.size(); ++i) {
|
||||
const Slot& s = _slots[i];
|
||||
if (s.cap_bytes == 0 && s.used_bytes == 0) continue;
|
||||
const int c = (int)slot_category(pool_key_slot(i));
|
||||
out.used[c] += s.used_bytes;
|
||||
out.cap[c] += s.cap_bytes;
|
||||
}
|
||||
for (const auto& kv : _dyn) {
|
||||
const int c = (int)kv.second.cat;
|
||||
out.used[c] += kv.second.slot.used_bytes;
|
||||
out.cap[c] += kv.second.slot.cap_bytes;
|
||||
}
|
||||
out.cap[(int)VramCategory::SplatXImg] += _arena.cap;
|
||||
return out;
|
||||
}
|
||||
|
||||
// Per-slot breakdown: [(name, used_bytes, cap_bytes), ...]. Names are
|
||||
// reconstructed byte-identically to the old string keys (slot_name + sub
|
||||
// suffix), so existing consumers keep working.
|
||||
|
||||
@@ -679,6 +679,12 @@ std::vector<std::tuple<std::string, std::string, size_t, size_t>>
|
||||
engine_get_pool_breakdown_categorized();
|
||||
size_t engine_get_scratch_bytes();
|
||||
|
||||
// GPU seconds the next step spends in its splat stages (forward, raster and
|
||||
// projection backward, optimizer, densify) for the trainer's ETA model. Arm
|
||||
// before the step, read after; < 0 if not armed or the device has no timer.
|
||||
void engine_step_timing_arm();
|
||||
double engine_step_timing_read();
|
||||
|
||||
// Formatted per-category pool report (SS_PROFILE). Reads the pool's
|
||||
// high-water capacities, so it is a peak, not an instantaneous figure.
|
||||
std::string engine_vram_report();
|
||||
|
||||
@@ -431,6 +431,7 @@ int engine_load_checkpoint(std::string input_dir) {
|
||||
// Install runtime scalars + optimizer layout from the checkpoint.
|
||||
s.cur_num_splats = cur_n;
|
||||
s.max_num_splats = max_n;
|
||||
DevicePool::global().set_splat_counts(cur_n, max_n);
|
||||
s.num_sh = num_sh;
|
||||
s.sh_degree = sh_degree;
|
||||
s.packed = (packed != 0);
|
||||
|
||||
@@ -9,7 +9,9 @@
|
||||
#include "kernels/loss/PerPixelLoss.cuh" // LossWeightIndex
|
||||
#include "kernels/pixelwise/PixelWise.cuh" // PPISPRegLossIndex
|
||||
|
||||
#include <algorithm>
|
||||
#include <array>
|
||||
#include <cstdint>
|
||||
|
||||
|
||||
// Bundles the scalars engine_compute_loss_backward takes, for the call path
|
||||
@@ -240,6 +242,17 @@ struct DensifyConfig {
|
||||
int las_split_opacity_k_warmup = 4500;
|
||||
};
|
||||
|
||||
// The schedule engine_densify_step grows the model on; the trainer's ETA and
|
||||
// VRAM forecasts replay it.
|
||||
inline bool densify_grows_at(const DensifyConfig& c, int step, int max_steps) {
|
||||
return c.refine_every > 0 && step > c.refine_start_iter &&
|
||||
step % c.refine_every == 0 &&
|
||||
step < std::max(c.refine_stop_iter, max_steps - c.refine_stop_num_iter);
|
||||
}
|
||||
inline int64_t densify_target(const DensifyConfig& c, int64_t cur, int64_t cap) {
|
||||
return std::max(cur, std::min(cap, (int64_t)(c.growth_factor * (float)cur)));
|
||||
}
|
||||
|
||||
|
||||
// Per-type Adam LR + TV regularization weight. lr <= 0 disables the channel
|
||||
// for the current step (so a single config covers "enabled but skipped" too).
|
||||
|
||||
@@ -90,7 +90,7 @@ int engine_densify_step(int step, int max_steps, const DensifyConfig& cfg) {
|
||||
bool densify_ongoing =
|
||||
(step < std::max(cfg.refine_stop_iter,
|
||||
max_steps - cfg.refine_stop_num_iter));
|
||||
bool do_densify = densify_ongoing && (step > cfg.refine_start_iter && step % cfg.refine_every == 0);
|
||||
bool do_densify = densify_grows_at(cfg, step, max_steps);
|
||||
float progress = ((float)step + 0.5f) / (float)max_steps;
|
||||
|
||||
// Use pool-backed DeviceVector/DeviceTensor from Buffers directly
|
||||
@@ -340,7 +340,7 @@ int engine_densify_step(int step, int max_steps, const DensifyConfig& cfg) {
|
||||
);
|
||||
|
||||
// Add more splats
|
||||
int64_t n_target = std::min(max_num_splats, (int64_t)(cfg.growth_factor * cur_num_splats));
|
||||
int64_t n_target = densify_target(cfg, cur_num_splats, max_num_splats);
|
||||
num_added = (int)std::max((int64_t)0, n_target - cur_num_splats);
|
||||
if (num_added > 0) {
|
||||
// The oversize channel draws first, so its candidate set is the
|
||||
@@ -406,7 +406,7 @@ int engine_densify_step(int step, int max_steps, const DensifyConfig& cfg) {
|
||||
, engine().region.weight);
|
||||
|
||||
// MCMC sample add
|
||||
int64_t n_target = std::min(max_num_splats, (int64_t)(cfg.growth_factor * cur_num_splats));
|
||||
int64_t n_target = densify_target(cfg, cur_num_splats, max_num_splats);
|
||||
num_added = (int)std::max((int64_t)0, n_target - cur_num_splats);
|
||||
if (num_added > 0) {
|
||||
add_splats_mcmc_tensor(
|
||||
@@ -449,5 +449,6 @@ int engine_densify_step(int step, int max_steps, const DensifyConfig& cfg) {
|
||||
}
|
||||
|
||||
engine().cur_num_splats = cur_num_splats + num_added;
|
||||
DevicePool::global().set_splat_counts(engine().cur_num_splats, engine().max_num_splats);
|
||||
return num_added;
|
||||
}
|
||||
|
||||
@@ -185,3 +185,13 @@ void engine_ensure_optim_state(int sh_optim_bits, int sh_value_bits,
|
||||
float* _engine_ppisp_reg_loss_into(
|
||||
const std::array<float, (int)PPISPRegLossIndex::length>& loss_weights,
|
||||
bool compute_grad);
|
||||
|
||||
// Brackets one of a step's splat stages while engine_step_timing_arm() is in
|
||||
// force. Defined in EngineStepTiming.cpp.
|
||||
struct SplatStageTimer {
|
||||
SplatStageTimer();
|
||||
~SplatStageTimer();
|
||||
SplatStageTimer(const SplatStageTimer&) = delete;
|
||||
SplatStageTimer& operator=(const SplatStageTimer&) = delete;
|
||||
int _pair = -1;
|
||||
};
|
||||
|
||||
@@ -146,6 +146,7 @@ static void _engine_raster_proj_backward(
|
||||
TorchTensorView v_depth_dist = _tv_null(),
|
||||
TorchTensorView v_normal_dist = _tv_null()
|
||||
) {
|
||||
SplatStageTimer stage_timer;
|
||||
RenderOutput::TensorTuple v_render_outputs = std::make_tuple(
|
||||
DeviceTensor3D<float3>(v_render_rgb),
|
||||
DeviceTensor3D<float>(v_render_depth),
|
||||
|
||||
@@ -180,6 +180,7 @@ void engine_scene_activate(int slot) {
|
||||
engine().world = s.world;
|
||||
engine().cur_num_splats = s.cur_num_splats;
|
||||
engine().max_num_splats = s.max_num_splats;
|
||||
DevicePool::global().set_splat_counts(engine().cur_num_splats, engine().max_num_splats);
|
||||
engine().num_sh = s.num_sh;
|
||||
if (g_cs_slot != slot || g_cs_gen != s.cs_gen) {
|
||||
engine_init_color_space(s.cs_enabled, s.cs_transfer, s.cs_is_linear,
|
||||
|
||||
@@ -29,6 +29,7 @@ void set_data_3dgs(
|
||||
|
||||
engine().cur_num_splats = num_splats;
|
||||
engine().max_num_splats = max_num_splats;
|
||||
DevicePool::global().set_splat_counts(engine().cur_num_splats, engine().max_num_splats);
|
||||
|
||||
auto sh_shape = std::get<2>(features_sh);
|
||||
engine().num_sh = (sh_shape.size() >= 2) ? (int)sh_shape[1] : 0;
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
// GPU time of a step's splat-proportional stages, for the trainer's ETA model.
|
||||
|
||||
#include "engine/Engine.h"
|
||||
#include "engine/EngineInternal.h"
|
||||
|
||||
#include "backend/api/BackendRuntime.h"
|
||||
|
||||
#include <utility>
|
||||
#include <vector>
|
||||
|
||||
namespace {
|
||||
|
||||
struct StageTiming {
|
||||
bool armed = false;
|
||||
// Reused across steps; one pair per bracketed stage of the armed step.
|
||||
std::vector<std::pair<backend::Event*, backend::Event*>> pairs;
|
||||
size_t used = 0;
|
||||
};
|
||||
|
||||
StageTiming& timing() {
|
||||
static StageTiming t;
|
||||
return t;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
SplatStageTimer::SplatStageTimer() {
|
||||
StageTiming& t = timing();
|
||||
if (!t.armed) return;
|
||||
if (t.used == t.pairs.size())
|
||||
t.pairs.emplace_back(backend::event_create(true), backend::event_create(true));
|
||||
_pair = (int)t.used++;
|
||||
backend::event_record(t.pairs[_pair].first);
|
||||
}
|
||||
|
||||
SplatStageTimer::~SplatStageTimer() {
|
||||
if (_pair < 0) return;
|
||||
backend::event_record(timing().pairs[_pair].second);
|
||||
}
|
||||
|
||||
void engine_step_timing_arm() {
|
||||
StageTiming& t = timing();
|
||||
t.armed = true;
|
||||
t.used = 0;
|
||||
}
|
||||
|
||||
double engine_step_timing_read() {
|
||||
StageTiming& t = timing();
|
||||
if (!t.armed) return -1.0;
|
||||
t.armed = false;
|
||||
if (t.used == 0) return -1.0;
|
||||
backend::event_synchronize(t.pairs[t.used - 1].second);
|
||||
double ms = 0.0;
|
||||
for (size_t i = 0; i < t.used; ++i)
|
||||
ms += backend::event_elapsed_ms(t.pairs[i].first, t.pairs[i].second);
|
||||
t.used = 0;
|
||||
// 0 means the device has no timestamps, not that the stages were free.
|
||||
return ms > 0.0 ? ms * 1e-3 : -1.0;
|
||||
}
|
||||
@@ -139,7 +139,10 @@ static std::map<std::string, float> _engine_step_fwd_bwd_only(
|
||||
engine().ppisp.enabled && engine().ppisp.cur_run_before_color_space;
|
||||
engine().background.match_luma_pending = cfg.background.match_luminance;
|
||||
|
||||
forward_3dgs(primitive, sh_degree, packed, /*output_median=*/false, (int)dist_type);
|
||||
{
|
||||
SplatStageTimer stage_timer;
|
||||
forward_3dgs(primitive, sh_degree, packed, /*output_median=*/false, (int)dist_type);
|
||||
}
|
||||
|
||||
// PPISP already ran inside the forward in the before-color-space order.
|
||||
const bool ppisp_after = engine().ppisp.enabled &&
|
||||
@@ -186,7 +189,10 @@ static void _engine_step_optim_and_densify(
|
||||
const EngineStepConfig& cfg,
|
||||
std::map<std::string, float>& loss_dict
|
||||
) {
|
||||
engine_optim_step(step, cfg.optim);
|
||||
{
|
||||
SplatStageTimer stage_timer;
|
||||
engine_optim_step(step, cfg.optim);
|
||||
}
|
||||
|
||||
if (engine().background.enabled) {
|
||||
engine_background_optim_step(step, cfg.background);
|
||||
@@ -226,7 +232,11 @@ static void _engine_step_optim_and_densify(
|
||||
ppisp_reg_readout.issue(losses_buf);
|
||||
}
|
||||
|
||||
int num_added = engine_densify_step(step, max_steps, cfg.densify);
|
||||
int num_added = 0;
|
||||
{
|
||||
SplatStageTimer stage_timer;
|
||||
num_added = engine_densify_step(step, max_steps, cfg.densify);
|
||||
}
|
||||
|
||||
loss_dict["num_added"] = (float)num_added;
|
||||
loss_dict["cur_num_splats"] = (float)engine().cur_num_splats;
|
||||
|
||||
@@ -2207,6 +2207,184 @@ SS_MSG(vram_help,
|
||||
"/ aygıtın kapasitesi. “?”, arka ucun o değeri sorgulayamadığı "
|
||||
"anlamına gelir."));
|
||||
|
||||
// The VRAM forecast: the risk tag beside the bar, and its hover card.
|
||||
SS_MSG(oom_risk_low,
|
||||
EN("OOM risk: low"), JA("メモリ不足リスク: 低"), ZH_HANS("显存不足风险：低"),
|
||||
ZH_HANT("顯示記憶體不足風險：低"), KO("메모리 부족 위험: 낮음"),
|
||||
DE("OOM-Risiko: gering"), FR("Risque de saturation : faible"),
|
||||
ES("Riesgo de falta de memoria: bajo"), PT("Risco de falta de memória: baixo"),
|
||||
IT("Rischio memoria esaurita: basso"), NL("Risico geheugentekort: laag"),
|
||||
RU("Риск нехватки памяти: низкий"), TR("Bellek yetmeme riski: düşük"));
|
||||
|
||||
SS_MSG(oom_risk_medium,
|
||||
EN("OOM risk: medium"), JA("メモリ不足リスク: 中"), ZH_HANS("显存不足风险：中"),
|
||||
ZH_HANT("顯示記憶體不足風險：中"), KO("메모리 부족 위험: 보통"),
|
||||
DE("OOM-Risiko: mittel"), FR("Risque de saturation : moyen"),
|
||||
ES("Riesgo de falta de memoria: medio"), PT("Risco de falta de memória: médio"),
|
||||
IT("Rischio memoria esaurita: medio"), NL("Risico geheugentekort: middel"),
|
||||
RU("Риск нехватки памяти: средний"), TR("Bellek yetmeme riski: orta"));
|
||||
|
||||
SS_MSG(oom_risk_high,
|
||||
EN("OOM risk: high"), JA("メモリ不足リスク: 高"), ZH_HANS("显存不足风险：高"),
|
||||
ZH_HANT("顯示記憶體不足風險：高"), KO("메모리 부족 위험: 높음"),
|
||||
DE("OOM-Risiko: hoch"), FR("Risque de saturation : élevé"),
|
||||
ES("Riesgo de falta de memoria: alto"), PT("Risco de falta de memória: alto"),
|
||||
IT("Rischio memoria esaurita: alto"), NL("Risico geheugentekort: hoog"),
|
||||
RU("Риск нехватки памяти: высокий"), TR("Bellek yetmeme riski: yüksek"));
|
||||
|
||||
SS_MSG(vram_chart_title,
|
||||
EN("GPU memory over the run (GiB)"), JA("学習中の GPU メモリ（GiB）"),
|
||||
ZH_HANS("训练过程中的显存（GiB）"), ZH_HANT("訓練過程中的顯示記憶體（GiB）"),
|
||||
KO("학습 중 GPU 메모리(GiB)"), DE("Grafikspeicher im Verlauf (GiB)"),
|
||||
FR("Mémoire GPU au fil de l'entraînement (Gio)"),
|
||||
ES("Memoria de GPU durante el entrenamiento (GiB)"),
|
||||
PT("Memória da GPU ao longo do treinamento (GiB)"),
|
||||
IT("Memoria GPU durante l'addestramento (GiB)"),
|
||||
NL("GPU-geheugen tijdens de training (GiB)"),
|
||||
RU("Видеопамять по ходу обучения (ГиБ)"), TR("Eğitim boyunca GPU belleği (GiB)"));
|
||||
|
||||
SS_MSG(vram_legend_run,
|
||||
EN("this run"), JA("この学習"), ZH_HANS("本次训练"), ZH_HANT("本次訓練"),
|
||||
KO("이 학습"), DE("dieses Training"), FR("cet entraînement"),
|
||||
ES("este entrenamiento"), PT("este treinamento"), IT("questo addestramento"),
|
||||
NL("deze training"), RU("это обучение"), TR("bu eğitim"));
|
||||
|
||||
SS_MSG(vram_legend_projected,
|
||||
EN("projected (95% band)"), JA("予測（95% 範囲）"), ZH_HANS("预测（95% 区间）"),
|
||||
ZH_HANT("預測（95% 區間）"), KO("예측(95% 범위)"), DE("Prognose (95-%-Band)"),
|
||||
FR("prévision (bande à 95 %)"), ES("previsión (banda del 95 %)"),
|
||||
PT("previsão (faixa de 95%)"), IT("previsione (banda al 95%)"),
|
||||
NL("prognose (95%-band)"), RU("прогноз (полоса 95 %)"), TR("tahmin (%95 aralığı)"));
|
||||
|
||||
SS_MSG(vram_legend_others,
|
||||
EN("other programs"), JA("他のプログラム"), ZH_HANS("其他程序"), ZH_HANT("其他程式"),
|
||||
KO("다른 프로그램"), DE("andere Programme"), FR("autres programmes"),
|
||||
ES("otros programas"), PT("outros programas"), IT("altri programmi"),
|
||||
NL("andere programma's"), RU("другие программы"), TR("diğer programlar"));
|
||||
|
||||
SS_MSG(vram_legend_capacity,
|
||||
EN("device capacity"), JA("デバイスの容量"), ZH_HANS("设备容量"), ZH_HANT("裝置容量"),
|
||||
KO("장치 용량"), DE("Kapazität des Geräts"), FR("capacité du périphérique"),
|
||||
ES("capacidad del dispositivo"), PT("capacidade do dispositivo"),
|
||||
IT("capacità del dispositivo"), NL("capaciteit van het apparaat"),
|
||||
RU("объём устройства"), TR("aygıt kapasitesi"));
|
||||
|
||||
SS_MSG(vram_chart_peak,
|
||||
EN("Projected peak: {0} ± {1} GiB free for training: {2} GiB chance of running out: {3}%"),
|
||||
JA("予測ピーク: {0} ± {1} GiB 学習に使える量: {2} GiB 不足する確率: {3}%"),
|
||||
ZH_HANS("预计峰值：{0} ± {1} GiB 可供训练：{2} GiB 耗尽的概率：{3}%"),
|
||||
ZH_HANT("預計峰值：{0} ± {1} GiB 可供訓練：{2} GiB 耗盡的機率：{3}%"),
|
||||
KO("예상 최대치: {0} ± {1} GiB 학습에 쓸 수 있는 양: {2} GiB 부족할 확률: {3}%"),
|
||||
DE("Erwartete Spitze: {0} ± {1} GiB für das Training frei: {2} GiB Wahrscheinlichkeit, dass er ausgeht: {3} %"),
|
||||
FR("Pic prévu : {0} ± {1} Gio disponible pour l'entraînement : {2} Gio probabilité de saturation : {3} %"),
|
||||
ES("Pico previsto: {0} ± {1} GiB libre para entrenar: {2} GiB probabilidad de quedarse sin memoria: {3} %"),
|
||||
PT("Pico previsto: {0} ± {1} GiB livre para o treinamento: {2} GiB chance de faltar memória: {3}%"),
|
||||
IT("Picco previsto: {0} ± {1} GiB libera per l'addestramento: {2} GiB probabilità di esaurirla: {3}%"),
|
||||
NL("Verwachte piek: {0} ± {1} GiB vrij voor training: {2} GiB kans op tekort: {3}%"),
|
||||
RU("Ожидаемый пик: {0} ± {1} ГиБ доступно для обучения: {2} ГиБ вероятность нехватки: {3} %"),
|
||||
TR("Beklenen tepe: {0} ± {1} GiB eğitim için boş: {2} GiB yetmeme olasılığı: %{3}"));
|
||||
|
||||
SS_MSG(vram_chart_provisional,
|
||||
EN("The estimate firms up once the first densification step has run."),
|
||||
JA("最初の高密度化ステップが済むと、推定の精度が上がります。"),
|
||||
ZH_HANS("第一次加密步骤运行后，估计会更准确。"),
|
||||
ZH_HANT("第一次加密步驟執行後，估計會更準確。"),
|
||||
KO("첫 번째 밀집화 단계가 끝나면 추정이 더 정확해집니다."),
|
||||
DE("Die Schätzung wird genauer, sobald der erste Verdichtungsschritt gelaufen ist."),
|
||||
FR("L'estimation se précise après la première étape de densification."),
|
||||
ES("La estimación se afina en cuanto se ejecuta el primer paso de densificación."),
|
||||
PT("A estimativa fica mais precisa depois do primeiro passo de densificação."),
|
||||
IT("La stima si affina dopo il primo passo di densificazione."),
|
||||
NL("De schatting wordt nauwkeuriger zodra de eerste verdichtingsstap is uitgevoerd."),
|
||||
RU("Оценка уточнится после первого шага уплотнения."),
|
||||
TR("İlk yoğunlaştırma adımı çalıştıktan sonra tahmin netleşir."));
|
||||
|
||||
SS_MSG(vram_chart_waiting,
|
||||
EN("The projection appears after the first steps have been measured."),
|
||||
JA("最初のステップを計測すると予測が表示されます。"),
|
||||
ZH_HANS("测量完最初的若干步后会显示预测。"),
|
||||
ZH_HANT("量測完最初的若干步後會顯示預測。"),
|
||||
KO("처음 몇 단계를 측정하면 예측이 표시됩니다."),
|
||||
DE("Die Prognose erscheint, sobald die ersten Schritte gemessen sind."),
|
||||
FR("La prévision apparaît une fois les premières étapes mesurées."),
|
||||
ES("La previsión aparece cuando se han medido los primeros pasos."),
|
||||
PT("A previsão aparece depois que os primeiros passos são medidos."),
|
||||
IT("La previsione compare dopo che i primi passi sono stati misurati."),
|
||||
NL("De prognose verschijnt zodra de eerste stappen gemeten zijn."),
|
||||
RU("Прогноз появится, когда будут измерены первые шаги."),
|
||||
TR("Tahmin, ilk adımlar ölçüldükten sonra görünür."));
|
||||
|
||||
SS_MSG(vram_breakdown_title,
|
||||
EN("This run by category (GiB)"), JA("この学習の内訳（GiB）"),
|
||||
ZH_HANS("本次训练按类别（GiB）"), ZH_HANT("本次訓練按類別（GiB）"),
|
||||
KO("이 학습의 항목별 사용량(GiB)"), DE("Dieses Training nach Kategorie (GiB)"),
|
||||
FR("Cet entraînement par catégorie (Gio)"), ES("Este entrenamiento por categoría (GiB)"),
|
||||
PT("Este treinamento por categoria (GiB)"), IT("Questo addestramento per categoria (GiB)"),
|
||||
NL("Deze training per categorie (GiB)"), RU("Это обучение по категориям (ГиБ)"),
|
||||
TR("Bu eğitim, kategoriye göre (GiB)"));
|
||||
|
||||
SS_MSG(vram_breakdown_growth,
|
||||
EN("Faded: growth still to come, up to the projected peak."),
|
||||
JA("薄い部分: 予測ピークまでにこれから増える分。"),
|
||||
ZH_HANS("浅色部分：到预计峰值前还会增加的量。"),
|
||||
ZH_HANT("淺色部分：到預計峰值前還會增加的量。"),
|
||||
KO("흐린 부분: 예상 최대치까지 앞으로 늘어날 양."),
|
||||
DE("Blass: der Zuwachs, der bis zur erwarteten Spitze noch kommt."),
|
||||
FR("En pâle : la croissance encore à venir, jusqu'au pic prévu."),
|
||||
ES("Atenuado: el crecimiento que aún falta hasta el pico previsto."),
|
||||
PT("Esmaecido: o crescimento que ainda virá, até o pico previsto."),
|
||||
IT("Sbiadito: la crescita ancora da venire, fino al picco previsto."),
|
||||
NL("Vaag: de groei die nog komt, tot de verwachte piek."),
|
||||
RU("Бледным: рост, который ещё впереди, до ожидаемого пика."),
|
||||
TR("Soluk: beklenen tepeye kadar daha gelecek artış."));
|
||||
|
||||
SS_MSG(vram_cat_splat,
|
||||
EN("Splats"), JA("スプラット"), ZH_HANS("泼溅"), ZH_HANT("潑濺"), KO("스플랫"),
|
||||
DE("Splats"), FR("Splats"), ES("Splats"), PT("Splats"), IT("Splat"),
|
||||
NL("Splats"), RU("Сплаты"), TR("Splat'ler"));
|
||||
|
||||
SS_MSG(vram_cat_splat_x_img,
|
||||
EN("Splats × images"), JA("スプラット × 画像"), ZH_HANS("泼溅 × 图像"),
|
||||
ZH_HANT("潑濺 × 影像"), KO("스플랫 × 이미지"), DE("Splats × Bilder"),
|
||||
FR("Splats × images"), ES("Splats × imágenes"), PT("Splats × imagens"),
|
||||
IT("Splat × immagini"), NL("Splats × beelden"), RU("Сплаты × изображения"),
|
||||
TR("Splat × görüntü"));
|
||||
|
||||
SS_MSG(vram_cat_image,
|
||||
EN("Images"), JA("画像"), ZH_HANS("图像"), ZH_HANT("影像"), KO("이미지"),
|
||||
DE("Bilder"), FR("Images"), ES("Imágenes"), PT("Imagens"), IT("Immagini"),
|
||||
NL("Beelden"), RU("Изображения"), TR("Görüntüler"));
|
||||
|
||||
SS_MSG(vram_cat_appearance,
|
||||
EN("Appearance"), JA("外観補正"), ZH_HANS("外观校正"), ZH_HANT("外觀校正"),
|
||||
KO("외관 보정"), DE("Erscheinungsbild"), FR("Apparence"), ES("Apariencia"),
|
||||
PT("Aparência"), IT("Aspetto"), NL("Uiterlijk"), RU("Внешний вид"),
|
||||
TR("Görünüm"));
|
||||
|
||||
SS_MSG(vram_cat_viewer,
|
||||
EN("Viewer"), JA("ビューア"), ZH_HANS("查看器"), ZH_HANT("檢視器"), KO("뷰어"),
|
||||
DE("Betrachter"), FR("Visionneuse"), ES("Visor"), PT("Visualizador"),
|
||||
IT("Visualizzatore"), NL("Viewer"), RU("Просмотр"), TR("Görüntüleyici"));
|
||||
|
||||
SS_MSG(vram_cat_other,
|
||||
EN("Other"), JA("その他"), ZH_HANS("其他"), ZH_HANT("其他"), KO("기타"),
|
||||
DE("Sonstiges"), FR("Autre"), ES("Otros"), PT("Outros"), IT("Altro"),
|
||||
NL("Overig"), RU("Прочее"), TR("Diğer"));
|
||||
|
||||
SS_MSG(vram_cat_scratch,
|
||||
EN("Sort scratch"), JA("ソート用の作業領域"), ZH_HANS("排序临时缓冲"),
|
||||
ZH_HANT("排序暫存緩衝"), KO("정렬 작업 공간"), DE("Sortierpuffer"),
|
||||
FR("Tampon de tri"), ES("Búfer de ordenación"), PT("Buffer de ordenação"),
|
||||
IT("Buffer di ordinamento"), NL("Sorteerbuffer"), RU("Буфер сортировки"),
|
||||
TR("Sıralama tamponu"));
|
||||
|
||||
SS_MSG(vram_cat_unpooled,
|
||||
EN("Backend and staging"), JA("バックエンドと転送用"), ZH_HANS("后端与中转"),
|
||||
ZH_HANT("後端與中轉"), KO("백엔드와 전송용"), DE("Backend und Staging"),
|
||||
FR("Backend et transfert"), ES("Backend y transferencia"),
|
||||
PT("Backend e transferência"), IT("Backend e trasferimento"),
|
||||
NL("Backend en staging"), RU("Бэкенд и передача"), TR("Arka uç ve aktarım"));
|
||||
|
||||
// ===========================================================================
|
||||
// Log panel
|
||||
// ===========================================================================
|
||||
|
||||
@@ -1725,6 +1725,45 @@ SS_MSG(train_finished,
|
||||
RU("Обучение завершено. Шагов: {0} Время: {1}"),
|
||||
TR("Eğitim tamamlandı. Adım: {0} Süre: {1}"));
|
||||
|
||||
SS_MSG(vram_forecast_warn,
|
||||
EN("Warning: training may run out of GPU memory. Projected peak: {0} ± {1} GiB "
|
||||
"free for training: {2} GiB chance of running out: {3}%. Lower --cap-max, "
|
||||
"or close other programs using the GPU."),
|
||||
JA("警告: 学習中に GPU メモリが不足する可能性があります。予測ピーク: {0} ± {1} GiB "
|
||||
"学習に使える量: {2} GiB 不足する確率: {3}%。--cap-max を下げるか、GPU を"
|
||||
"使っている他のプログラムを閉じてください。"),
|
||||
ZH_HANS("警告：训练可能会耗尽显存。预计峰值：{0} ± {1} GiB 可供训练：{2} GiB "
|
||||
"耗尽的概率：{3}%。请调低 --cap-max，或关闭其他占用 GPU 的程序。"),
|
||||
ZH_HANT("警告：訓練可能會耗盡顯示記憶體。預計峰值：{0} ± {1} GiB 可供訓練：{2} GiB "
|
||||
"耗盡的機率：{3}%。請調低 --cap-max，或關閉其他佔用 GPU 的程式。"),
|
||||
KO("경고: 학습 중 GPU 메모리가 부족할 수 있습니다. 예상 최대치: {0} ± {1} GiB "
|
||||
"학습에 쓸 수 있는 양: {2} GiB 부족할 확률: {3}%. --cap-max를 낮추거나 GPU를 "
|
||||
"쓰는 다른 프로그램을 닫으세요."),
|
||||
DE("Warnung: Dem Training kann der Grafikspeicher ausgehen. Erwartete Spitze: "
|
||||
"{0} ± {1} GiB für das Training frei: {2} GiB Wahrscheinlichkeit: {3} %. "
|
||||
"Senken Sie --cap-max oder schließen Sie andere Programme, die die GPU nutzen."),
|
||||
FR("Avertissement : l'entraînement risque de manquer de mémoire GPU. Pic prévu : "
|
||||
"{0} ± {1} Gio disponible pour l'entraînement : {2} Gio probabilité : {3} %. "
|
||||
"Réduisez --cap-max ou fermez les autres programmes qui utilisent le GPU."),
|
||||
ES("Aviso: el entrenamiento puede quedarse sin memoria de GPU. Pico previsto: "
|
||||
"{0} ± {1} GiB libre para entrenar: {2} GiB probabilidad: {3} %. Reduzca "
|
||||
"--cap-max o cierre otros programas que usen la GPU."),
|
||||
PT("Aviso: o treinamento pode ficar sem memória de GPU. Pico previsto: {0} ± {1} GiB "
|
||||
"livre para o treinamento: {2} GiB probabilidade: {3}%. Reduza --cap-max ou "
|
||||
"feche outros programas que usam a GPU."),
|
||||
IT("Attenzione: l'addestramento potrebbe esaurire la memoria GPU. Picco previsto: "
|
||||
"{0} ± {1} GiB libera per l'addestramento: {2} GiB probabilità: {3}%. Riduci "
|
||||
"--cap-max o chiudi gli altri programmi che usano la GPU."),
|
||||
NL("Waarschuwing: de training kan zonder GPU-geheugen komen te zitten. Verwachte piek: "
|
||||
"{0} ± {1} GiB vrij voor training: {2} GiB kans: {3}%. Verlaag --cap-max of "
|
||||
"sluit andere programma's die de GPU gebruiken."),
|
||||
RU("Предупреждение: обучению может не хватить видеопамяти. Ожидаемый пик: "
|
||||
"{0} ± {1} ГиБ доступно для обучения: {2} ГиБ вероятность нехватки: {3} %. "
|
||||
"Уменьшите --cap-max или закройте другие программы, использующие GPU."),
|
||||
TR("Uyarı: eğitimin GPU belleği yetmeyebilir. Beklenen tepe: {0} ± {1} GiB "
|
||||
"eğitim için boş: {2} GiB yetmeme olasılığı: %{3}. --cap-max değerini düşürün "
|
||||
"ya da GPU kullanan diğer programları kapatın."));
|
||||
|
||||
SS_MSG(partition_applied,
|
||||
EN("Partition part {0}: cameras {1} (core {2}, ring {3}), seed points {4}"),
|
||||
JA("分割パート {0}: カメラ {1}（コア {2}、リング {3}）、初期点 {4}"),
|
||||
|
||||
Reference in New Issue
Block a user