Files
jakeaturner ac437f8429 fix(gpu): detect Ollama CPU fallback on AMD and offer the right fix
GpuPassthroughRemediationProvider returned early unless the NVIDIA
runtime was registered with Docker. AMD doesn't register one, so the
log-based CPU-fallback detection added in v1.35.0-rc.1 never ran on
AMD boxes. On a Phoenix1 iGPU running ollama:rocm with /dev/kfd and
/dev/dri passed through, Ollama logged library=cpu while gpuHealth
reported status "ok".

The gpuHealth side had its own bug. hasNvidiaRuntime and
hasRocmRuntime were set only inside the lspci probe branch. When lspci
returned a plausible controller, the else branch reported "ok" without
probing and left both flags false. A blind reinstall would not have
helped either: gfx1103 needs HSA_OVERRIDE_GFX_VERSION=11.0.0, and
without the gfx marker or ai.amdHsaOverride a reinstall rebuilds the
same CPU-bound container.

- Add resolveExpectedGpuVendor(). It reads the nomad_ollama container
  config first (nvidia DeviceRequest, ROCm image tag, /dev/kfd), then
  the nvidia runtime, then the AMD GPU type when no container exists.
  /dev/dri alone is not treated as AMD. A CPU container on an AMD box
  is trusted as deliberate (#1232 fallback, acceleration disabled).
- Gate the provider on the expected vendor. NVIDIA auto-reinstall is
  unchanged but now skips when the container requests a GPU with no
  runtime registered. AMD is detect-only: the provider logs whether a
  reinstall would change the container and never reinstalls itself.
- In gpuHealth, set the runtime flags up front and let Ollama's
  startup log decide health whenever a GPU is expected, whatever lspci
  returned. A CPU-only log now reports passthrough_failed and skips
  the nvidia-smi exec. lspci stays the display source when clean.
- Parse the first GPU "inference compute" line instead of the last
  line, which could be the CPU fallback device, and recognize Vulkan.
  Move the startup-log reader into ollama_compute so both callers
  share it.
- Report amdReinstallWouldChange, amdGfxTarget, currentHsaOverride and
  suggestedHsaOverride on AMD passthrough_failed. The gfx target is
  parsed from Ollama's "amdgpu is not supported" line (best-effort).
- Add GpuPassthroughAlert, shared by Settings > System and Settings >
  Models. When a reinstall would rebuild the same AMD container, it
  offers "Fix: Set GFX Override" instead of a reinstall. The dialog
  saves ai.amdHsaOverride and then reinstalls.
- Expose ai.amdHsaOverride through the settings API. The value lands
  in the container env, so only x.y.z, "none" or empty is accepted.
- Silence the HSA override resolver's logging on the gpuHealth poll.

The #1325 case where no GPU runtime is expected and lspci reports a
controller still returns "ok" unprobed.

Fixes #1344
Refs #1325, #1232
2026-09-29 15:54:41 -07:00

175 lines
4.2 KiB
TypeScript

import { Systeminformation } from 'systeminformation'
export type GpuHealthStatus = {
status: 'ok' | 'passthrough_failed' | 'no_gpu' | 'ollama_not_installed'
hasNvidiaRuntime: boolean
hasRocmRuntime: boolean
ollamaGpuAccessible: boolean
gpuVendor?: 'nvidia' | 'amd'
/**
* AMD passthrough_failed only. Whether a reinstall would change the running
* container. When false, a reinstall rebuilds the same CPU-bound container and
* the fix is an HSA override (ai.amdHsaOverride).
*/
amdReinstallWouldChange?: boolean
/** gfx target Ollama reported for an unsupported AMD GPU, e.g. 'gfx1103'. */
amdGfxTarget?: string
/** HSA_OVERRIDE_GFX_VERSION the running container has, if any. */
currentHsaOverride?: string
/** Override the gfx mapping recommends for amdGfxTarget, if it knows one. */
suggestedHsaOverride?: string
}
export type SystemInformationResponse = {
cpu: Systeminformation.CpuData
mem: Systeminformation.MemData
os: Systeminformation.OsData
disk: NomadDiskInfo[]
currentLoad: Systeminformation.CurrentLoadData
fsSize: Systeminformation.FsSizeData[]
uptime: Systeminformation.TimeData
graphics: Systeminformation.GraphicsData
gpuHealth?: GpuHealthStatus
}
// Type inferrence is not working properly with usePage and shared props, so we define this type manually
export type UsePageProps = {
appVersion: string
environment: string
}
export type LSBlockDevice = {
name: string
size: string
type: string
model: string | null
serial: string | null
vendor: string | null
rota: boolean | null
tran: string | null
children?: LSBlockDevice[]
}
export type NomadDiskInfoRaw = {
diskLayout: {
blockdevices: LSBlockDevice[]
}
fsSize: {
fs: string
size: number
used: number
available: number
use: number
mount: string
}[]
}
export type NomadDiskInfo = {
name: string
model: string
vendor: string
rota: boolean
tran: string
size: string
totalUsed: number
totalSize: number
percentUsed: number
filesystems: {
fs: string
mount: string
used: number
size: number
percentUsed: number
}[]
}
export type SystemUpdateStatus = {
stage: 'idle' | 'starting' | 'pulling' | 'pulled' | 'recreating' | 'complete' | 'error'
progress: number
message: string
timestamp: string
}
export type CheckLatestVersionResult = {
success: boolean,
updateAvailable: boolean,
currentVersion: string,
latestVersion: string,
message?: string
}
export type AutoUpdateEligibleTarget = {
version: string
tag: string
publishedAt: string
}
export type AutoUpdateStatus = {
enabled: boolean
windowStart: string
windowEnd: string
cooloffHours: number
currentVersion: string
withinWindow: boolean
eligibleTarget: AutoUpdateEligibleTarget | null
lastAttemptAt: string | null
lastResult: string | null
lastError: string | null
consecutiveFailures: number
autoDisabledReason: string | null
}
export type AppAutoUpdateAppStatus = {
service_name: string
friendly_name: string | null
auto_update_enabled: boolean
current_version: string
available_update_version: string | null
first_seen_at: string | null
eligible: boolean
reason: string
cooloff_remaining_hours: number | null
consecutive_failures: number
auto_disabled_reason: string | null
}
export type AppAutoUpdateStatus = {
enabled: boolean
windowStart: string
windowEnd: string
cooloffHours: number
withinWindow: boolean
lastAttemptAt: string | null
lastResult: string | null
apps: AppAutoUpdateAppStatus[]
}
export type ContentAutoUpdateResourceStatus = {
resource_id: string
resource_type: 'zim' | 'map'
current_version: string
available_update_version: string | null
size_bytes: number | null
eligible: boolean
reason: string
cooloff_remaining_hours: number | null
exceeds_cap: boolean
consecutive_failures: number
auto_disabled_reason: string | null
}
export type ContentAutoUpdateStatus = {
enabled: boolean
windowStart: string
windowEnd: string
cooloffHours: number
maxBytesPerWindow: number
withinWindow: boolean
windowBytesUsed: number
lastAttemptAt: string | null
lastResult: string | null
lastError: string | null
autoDisabledReason: string | null
resources: ContentAutoUpdateResourceStatus[]
}