Vision Layers #
This file provides named-field layer records for spatial operators. Tensors remain ordinary
arbitrary-rank tensors; each operator states the trailing axes it consumes, while leading records
any axes mapped pointwise by the layer.
Spatial layers #
Configuration shared by arbitrary-dimensional convolution layers.
- outChannels : ℕ
Number of output channels.
Kernel extent along each spatial axis.
Step along each spatial axis.
Symmetric zero-padding along each spatial axis.
Every kernel extent is positive.
Every stride is positive.
- seedKernel : ℕ
Seed for deterministic kernel initialization.
- seedBias : ℕ
Seed for deterministic bias initialization.
- kernelInit : Runtime.Autograd.Torch.Init.Scheme
Initialization scheme for the kernel weights.
Instances For
Apply an arbitrary-dimensional convolution to the channel and spatial suffix of a tensor.
The input suffix is (inChannels, spatial...). Any axes in leading are preserved; internally
they are flattened into one runtime batch and restored after the convolution.
Instances For
Configuration shared by arbitrary-dimensional pooling layers.
Window extent along each spatial axis.
Step along each spatial axis.
Symmetric padding along each spatial axis.
Every window extent is positive.
Every stride is positive.
Instances For
Apply max pooling to the channel and spatial suffix of a tensor.
Instances For
Apply average pooling to the channel and spatial suffix of a tensor.
Instances For
Global average pooling over every spatial axis, preserving the leading axes and channels.
Instances For
LayerNorm configuration for batched (batch x seqLen x embedDim) tensors.
PyTorch analogue: torch.nn.LayerNorm.
See https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html.
- seedGamma : ℕ
Seed for deterministic initialization of
gamma(scale). - seedBeta : ℕ
Seed for deterministic initialization of
beta(shift).
Instances For
Layer normalization over (batch × seqLen × embedDim) tensors, with explicit positivity proofs.
This matches the common Transformer usage: normalize each token’s embedDim-vector independently,
with learnable scale/shift parameters gamma and beta.
PyTorch analogue: torch.nn.LayerNorm(embedDim) applied to a tensor of shape
(batch, seqLen, embedDim).
Call nn.layerNorm when NeZero can discharge the positivity proofs automatically.
Instances For
Layer normalization over (batch × seqLen × embedDim) tensors.
This normalizes each embedDim-vector (per batch element, per sequence position), and applies
learned affine parameters gamma and beta.
PyTorch analogue: torch.nn.LayerNorm(embedDim) on a tensor shaped (batch, seqLen, embedDim).
Implementation note:
TorchLean uses NeZero to ensure seqLen and embedDim are positive, avoiding degenerate shapes.
Instances For
RMSNorm configuration for batched (batch x seqLen x embedDim) tensors.
This is a common alternative to LayerNorm in modern transformer architectures.
- seedGamma : ℕ
Seed for deterministic initialization of
gamma(scale).
Instances For
RMS normalization over (batch × seqLen × embedDim) tensors, with explicit positivity proofs.
This is like LayerNorm but without mean subtraction: we scale by the root-mean-square over the
embedDim axis, and apply a learned scale gamma.
PyTorch analogue: many libraries provide an RMSNorm(embedDim) module; conceptually it is applied
to tensors shaped (batch, seqLen, embedDim).
Call nn.rmsNorm when NeZero can discharge the positivity proofs automatically.
Instances For
RMS normalization over (batch × seqLen × embedDim) tensors.
This normalizes by the root-mean-square over the embedDim axis (per batch element, per position),
then applies a learned scale gamma.
Implementation note:
TorchLean uses NeZero to ensure seqLen and embedDim are positive, avoiding degenerate shapes.
Instances For
Parameter initialization for affine channel normalization.
Instances For
A checked reshape layer used internally to flatten and restore spatial axes.
Instances For
Apply a channel-first normalization kernel after adding its singleton trailing axis.
Instances For
Flatten arbitrary spatial axes to the channel-first kernel representation.
Instances For
Restore the original spatial axes after channel normalization.
Instances For
Batch normalization over (leading..., channels, spatial...) for any spatial rank.
Instances For
Instance normalization over (leading..., channels, spatial...) for any spatial rank.
Instances For
Group normalization over (leading..., channels, spatial...) for any spatial rank.
Instances For
Multi-head self-attention configuration.
PyTorch analogue: torch.nn.MultiheadAttention (conceptually).
See https://pytorch.org/docs/stable/generated/torch.nn.MultiheadAttention.html.
- numHeads : ℕ
Number of attention heads.
- headDim : ℕ
Per-head embedding dimension.
- seedW : ℕ
Base seed for deterministic parameter initialization.
- weightInit? : Option Runtime.Autograd.Torch.Init.Scheme
Projection-weight initialization.
noneretains Xavier-uniform initialization. - outputWeightInit? : Option Runtime.Autograd.Torch.Init.Scheme
Optional initializer for the output projection.
This is separate because deep residual stacks commonly scale the projection that writes back to the residual stream. When omitted,
weightInit?is used. - outputBias : Bool
Add a trainable bias after the output projection.
Instances For
Multi-head self-attention with an explicit nonzero sequence length proof.
If mask is provided, it is a boolean attention mask of shape (n × n) (e.g. causal masking).
Instances For
Multi-head self-attention using NeZero to hide the nonzero sequence length proof.
If mask is provided, it is a boolean attention mask of shape (n × n) (e.g. causal masking).