Neural-network operations for the eager engine.

This file implements runtime nodes such as dropout, normalization, attention, and recurrent/sequence building blocks on top of the core tensor operation layer.

def Runtime.Autograd.Tape.layerNorm {α : Type} [TorchLean.Storage α] [Context α] [DecidableRel fun (x1 x2 : α) => x1 > x2] {seqLen embedDim : ℕ} (h_seq_pos : seqLen > 0) (h_embed_pos : embedDim > 0) (t : Tape α) (xId gammaId betaId : ℕ) (epsilon : α := TorchLean.normalizationEpsilon) : Result (Tape α × ℕ)

Layer normalization for (seqLen, embedDim) tensors.

This records a single node whose backward returns gradients for x, gamma, and beta. PyTorch comparison: torch.nn.LayerNorm(embedDim) (applied per token) / functional.layer_norm.

def Runtime.Autograd.Tape.batchNorm {α : Type} [TorchLean.Storage α] [Context α] [DecidableRel fun (x1 x2 : α) => x1 > x2] {channels : ℕ} {sSpatial : Spec.Shape} (hWellFormed : (Spec.Shape.dim channels sSpatial).wellFormed) (t : Tape α) (xId gammaId betaId : ℕ) (epsilon : α := TorchLean.normalizationEpsilon) : Result (Tape α × ℕ)

Batch normalization over every spatial axis of a channel-first tensor.

def Runtime.Autograd.Tape.multiHeadAttention {α : Type} [TorchLean.Storage α] [Context α] [DecidableRel fun (x1 x2 : α) => x1 > x2] {n numHeads dModel headDim : ℕ} (h1 : n ≠ 0) (t : Tape α) (wqId wkId wvId woId xId : ℕ) (mask : Option (TorchLean.Tensor Bool [n, n]) := none) : Result (Tape α × ℕ)

Multi-head self-attention.

This is a shape-specialized attention primitive used by transformer-style models. It depends on an optional boolean (n,n) mask and returns the attended output of shape (n,dModel).

PyTorch comparison: similar to torch.nn.MultiheadAttention / scaled dot-product attention.