TorchLean API

NN.Spec.Module.Normalization

Normalization module wrappers #

This file wraps selected normalization specs as NNModuleSpecs for composition/export.

For LayerNormModuleSpec, we require (defaulted) proofs that dimensions are positive. This matches the spec-level intent: normalization divides by the number of features and uses variance/standard deviation, so degenerate "zero-width" cases are excluded when we want clean theorems.

PyTorch mental picture: nn.LayerNorm(embedDim) applied at each timestep, with weight=gamma and bias=beta.

def Spec.LayerNormModuleSpec {α : Type} [Context α] (seqLen embedDim : ) (gamma beta : Tensor α (Shape.dim embedDim Shape.scalar)) (h_seq_pos : seqLen > 0) (h_embed_pos : embedDim > 0) :

LayerNorm over the last dimension, wrapped as an NNModuleSpec.

Instances For