TorchLean NN: Activation and Shape Layers

def Runtime.Autograd.Model.Layers.relu {s : Spec.Shape} : Layer s s

ReLU activation layer (no parameters).

PyTorch analogues: torch.nn.ReLU / torch.nn.functional.relu.

def Runtime.Autograd.Model.Layers.silu {s : Spec.Shape} : Layer s s

SiLU (a.k.a. swish) activation layer (no parameters).

PyTorch analogues: torch.nn.SiLU / torch.nn.functional.silu.

def Runtime.Autograd.Model.Layers.gelu {s : Spec.Shape} : Layer s s

GELU activation layer (no parameters), using the tanh approximation.

PyTorch analogues: torch.nn.GELU(approximate='tanh') / torch.nn.functional.gelu(x, approximate='tanh'). PyTorch's default nn.GELU() uses the exact erf form, which TorchLean does not implement.

def Runtime.Autograd.Model.Layers.sigmoid {s : Spec.Shape} : Layer s s

Sigmoid activation layer (no parameters).

PyTorch analogy: torch.sigmoid.

def Runtime.Autograd.Model.Layers.tanh {s : Spec.Shape} : Layer s s

Hyperbolic tangent activation layer (no parameters).

PyTorch analogy: torch.tanh.

def Runtime.Autograd.Model.Layers.softmax {s : Spec.Shape} (axis : ℕ) [Spec.Shape.AxisInBounds axis s] : Layer s s

Shape-preserving softmax layer along axis.

def Runtime.Autograd.Model.Layers.logSoftmax {s : Spec.Shape} (axis : ℕ) [Spec.Shape.AxisInBounds axis s] : Layer s s

Shape-preserving stable log-softmax layer along axis.

def Runtime.Autograd.Model.Layers.square {s : Spec.Shape} : Layer s s

Pointwise square x ↦ x^2 (no parameters).

PyTorch analogy: torch.square(x) / x.square().

def Runtime.Autograd.Model.Layers.sum {s : Spec.Shape} : Layer s Spec.Shape.scalar

Sum-reduce all elements of the input to a scalar (no parameters).

PyTorch analogy: x.sum().

def Runtime.Autograd.Model.Layers.flatten {s : Spec.Shape} : Layer s [s.size]

Flatten any tensor to a 1D vector of length Spec.Shape.size s (no parameters).

PyTorch analogy: torch.flatten(x) or x.reshape(-1).

def Runtime.Autograd.Model.Layers.dropout {s : Spec.Shape} (p : Float) (seed : ℕ := 0) : Layer s s

Dropout layer controlled by Mode.

• In Mode.train, randomly zeroes entries with probability p.
• In Mode.eval, it is the identity.
• At p = 1, training returns zero exactly rather than evaluating the undefined scale 1 / (1-p).

We store p as a scalar parameter tensor (with requiresGrad := false) so it can be threaded through the unified parameter list without being optimized.

PyTorch analogy: torch.nn.Dropout(p) / torch.nn.functional.dropout(x, p, training=...).