Deep Value-Learning Objectives

This module packages the core scalar objectives / targets behind common deep RL algorithms:

• DQN and Double DQN,
• DDPG-style actor / critic objectives,
• TD3 clipped double critics,
• SAC entropy-regularized targets and actor objectives.

Compute a bootstrap target, then pass it to loss with the selected action. DDPG uses the ordinary Core.discountedBackup for its critic and negates the critic value for its actor. A TD residual is the target minus the selected tensor entry. Replay, target-network synchronization and optimizer orchestration live in higher-level modules.

Primary references:

• Mnih et al., "Human-level control through deep reinforcement learning" (2015): https://doi.org/10.1038/nature14236
• van Hasselt, Guez, and Silver, "Deep Reinforcement Learning with Double Q-learning" (2016): https://arxiv.org/abs/1509.06461
• Lillicrap et al., "Continuous Control with Deep Reinforcement Learning" (2015): https://arxiv.org/abs/1509.02971
• Fujimoto et al., "Addressing Function Approximation Error in Actor-Critic Methods" (2018): https://arxiv.org/abs/1802.09477
• Haarnoja et al., "Soft Actor-Critic" (2018): https://arxiv.org/abs/1801.01290

def Runtime.RL.ValueLearning.maximum {α : Type} [TorchLean.Storage α] [Context α] {nActions : ℕ} (qValues : TorchLean.Tensor α [nActions]) : α

Maximum Q-value in a vector, defaulting to 0 when nActions = 0.

def Runtime.RL.ValueLearning.dqnTarget {α : Type} [TorchLean.Storage α] [Context α] {nActions : ℕ} (reward gamma : α) (done : Bool) (nextQTarget : TorchLean.Tensor α [nActions]) : α

DQN bootstrap target r + γ max_a Q_target(s', a).

def Runtime.RL.ValueLearning.doubleDqnTarget {α : Type} [TorchLean.Storage α] [Context α] {nActions : ℕ} (reward gamma : α) (done : Bool) (nextQOnline nextQTarget : TorchLean.Tensor α [nActions]) : α

Double DQN target: select with the online network, evaluate with the target network.

def Runtime.RL.ValueLearning.loss {α : Type} [TorchLean.Storage α] [Context α] {nActions : ℕ} (q : TorchLean.Tensor α [nActions]) (action : Fin nActions) (target : α) (error : α → α → α := Core.squaredError) : α

Loss for a selected action against a supplied target.

Compute the target separately with dqnTarget, doubleDqnTarget, or another bootstrap rule. The default is squared error; pass error to use Huber loss or another scalar objective.

def Runtime.RL.ValueLearning.td3Target {α : Type} [Context α] (reward gamma nextCritic1 nextCritic2 : α) (done : Bool := false) : α

TD3 clipped-double target using the minimum of the two target critics.

def Runtime.RL.ValueLearning.sacTarget {α : Type} [Context α] (reward gamma nextCritic1 nextCritic2 logProb temperature : α) (done : Bool := false) : α

SAC entropy-regularized soft target: r + γ (min(Q1', Q2') - α * log π(a'|s')).

def Runtime.RL.ValueLearning.sacActorObjective {α : Type} [Context α] (critic1 critic2 logProb temperature : α) : α

SAC actor objective: minimize α * log π(a|s) - min(Q1, Q2).