Differentiable DQN Objectives

These losses build backend-generic autograd programs for value learning. Bellman targets are detached inside the objective: a DQN update fits the online prediction to a fixed target, even when the caller computed that target through a differentiable network.

Huber loss bounds the derivative with respect to each TD residual. It limits the influence of large Bellman errors without bounding the parameter gradient or guaranteeing that training is stable.

def Runtime.RL.DQN.Autograd.huberTDLoss {α : Type} [TorchLean.Storage α] [Context α] {m : Type → Type} [Monad m] [Autograd.Torch.Ops m α] {s : Spec.Shape} (prediction target : Autograd.Model.RefTy m α s) (delta : α := 1) (reduction : TorchLean.Loss.Reduction := TorchLean.Loss.Reduction.mean) : m (Autograd.Model.RefTy m α Spec.Shape.scalar)

Huber loss between predictions and detached Bellman targets.

For a positive delta, the elementwise loss is d² / 2 when |d| ≤ delta and delta * (|d| - delta / 2) otherwise, where d = prediction - target. The target receives zero gradient. reduction applies to all entries of the prediction tensor.

This is Huber loss, whose outer derivative has magnitude delta. For delta ≠ 1, it differs from Smooth L1 loss by a factor of delta.

def Runtime.RL.DQN.Autograd.actionHuberLossBatch {α : Type} [TorchLean.Storage α] [Context α] {m : Type → Type} [Monad m] [Autograd.Torch.Ops m α] {batch nActions : ℕ} [NeZero batch] [NeZero nActions] (qValues actionOneHot : Autograd.Model.RefTy m α (Spec.Shape.dim batch (Spec.Shape.dim nActions Spec.Shape.scalar))) (target : Autograd.Model.RefTy m α (Spec.Shape.dim batch Spec.Shape.scalar)) (delta : α := 1) : m (Autograd.Model.RefTy m α Spec.Shape.scalar)

Mean DQN Huber loss for a batch of Q vectors, one-hot actions, and scalar Bellman targets.

qValues and actionOneHot have shape (batch, nActions); target has shape (batch). Each row of actionOneHot must select exactly one action. Actions and targets are detached, so only the selected online Q values receive gradients. Reduction averages over transitions; adding unused actions does not rescale the loss.

Pass a positive delta. Target construction, termination masking, and target-network updates remain the caller's responsibility.