Training Memory Monitoring

CUDA allocator sampling and drift warnings shared by TorchLean training loops.

structure TorchLean.Trainer.Memory.State : Type

State carried by the CUDA-memory drift detector used by sustained training runs.

• firstStep : ℕ
• firstAvailableBytes : ℕ

  Estimated allocator headroom at the first sample, not guaranteed free memory.

• warned : Bool

def TorchLean.Trainer.Memory.instReprState.repr : State → ℕ → Std.Format

@[instance_reducible]

instance TorchLean.Trainer.Memory.instReprState : Repr State

def TorchLean.Trainer.Memory.cadence (options : Runtime.Autograd.Torch.Config) (steps requested : ℕ) : ℕ

Resolve an explicit CUDA-memory cadence, or enable periodic sampling for very long runs.

def TorchLean.Trainer.Memory.sample (options : Runtime.Autograd.Torch.Config) (watchEvery totalSteps done : ℕ) (state? : Option State) : IO (Option State)

Sample the CUDA allocator and warn when estimated allocation headroom keeps decreasing.

The estimate adds driver-free bytes to reserved-minus-allocated bytes, then caps that sum by the remaining configured memory-fraction budget. This avoids treating cache growth alone as a leak. Native allocations include library workspaces that can outlive TorchLean's buffer owners. Reserved storage is not necessarily reclaimable, samples are not atomic, and the fraction-to-byte conversion is approximate. The forecast is a drift warning, not a guarantee of successful allocation or a prediction of the exact step where allocation fails.