DQN Replay Mini-Example

This example runs the runtime pieces used by an off-policy DQN-style update:

1. construct typed transitions;
2. insert them into a bounded replay buffer;
3. sample a minibatch;
4. evaluate a DQN minibatch loss from caller-provided online/target Q-functions.

The Q-functions are hand-written closures rather than neural networks, so the example stays focused on replay buffers and minibatch losses. A full trainable DQN run can later swap those closures for typed graph TorchLean models and an optimizer step.

Run from the repo root through the maintained example runner:

lake exe torchlean dqn_replay

References:

• Mnih et al., "Human-level control through deep reinforcement learning" (2015): https://doi.org/10.1038/nature14236
• Lin, "Self-Improving Reactive Agents Based on Reinforcement Learning, Planning and Teaching" (1992), early experience replay.

def NN.Examples.Models.RL.DqnReplay.exeName : String

Subcommand name.

@[reducible, inline]

abbrev NN.Examples.Models.RL.DqnReplay.observation : Spec.Shape

Two observation features, small enough that the replay buffer's contents can be printed in full.

@[reducible, inline]

abbrev NN.Examples.Models.RL.DqnReplay.actionCount : ℕ

Three discrete actions.

def NN.Examples.Models.RL.DqnReplay.obsA : TorchLean.Tensor Float observation

A compact two-feature observation.

def NN.Examples.Models.RL.DqnReplay.obsB : TorchLean.Tensor Float observation

A second observation used as the next state.

def NN.Examples.Models.RL.DqnReplay.transitionA : Runtime.RL.Core.Transition Float observation actionCount

One typed transition inserted into the replay buffer.

def NN.Examples.Models.RL.DqnReplay.transitionB : Runtime.RL.Core.Transition Float observation actionCount

A second transition, marked terminal, so the sample contains both bootstrap modes.

def NN.Examples.Models.RL.DqnReplay.onlineQ (obs : TorchLean.Tensor Float observation) : TorchLean.Tensor Float [actionCount]

Compact online Q-function used by the example.

def NN.Examples.Models.RL.DqnReplay.targetQ (obs : TorchLean.Tensor Float observation) : TorchLean.Tensor Float [actionCount]

Compact target Q-function used by the example.

def NN.Examples.Models.RL.DqnReplay.run : IO Unit

Build a replay buffer, sample a minibatch, and compute DQN losses.

def NN.Examples.Models.RL.DqnReplay.usage : String

Command-line help for the replay-buffer mini-example.

def NN.Examples.Models.RL.DqnReplay.main (args : List String) : IO UInt32

Runner entrypoint used by lake exe torchlean dqn_replay.