Transformer Blocks

This module defines the configuration records for Transformer encoder blocks and stacks. The seeded constructors nn.transformerEncoderBlock and nn.transformerEncoderStack live in NN.API.Seeded.

structure TorchLean.nn.TransformerEncoder.Block.Config : Type

Config record for transformerEncoderBlock.

Separating the config as a structure makes it easier to write readable examples and keep seed management deterministic.

• headCount : ℕ

  Number of attention heads. Must be positive.

• headWidth : ℕ

  Per-head embedding dimension. Must be positive.

• feedForwardWidth : ℕ

  Hidden dimension of the feed-forward network.

• activation : Activation.Kind

  Activation used in the feed-forward network.

• dropout? : Option Float

  Dropout on the attention and feed-forward outputs before their residual additions.

  This controls the two residual-output sites. Set attentionDropout? and feedForwardDropout? as well when all four Transformer dropout sites are wanted.

• normalizeFirst : Bool

  Normalize before attention and feed-forward sublayers instead of after each residual.

• attentionOutputBias : Bool

  Add a trainable bias after the attention output projection.

• attentionInputBias : Bool

  Add a separate bias to each query, key, and value projection.

• attentionDropout? : Option Float

  Drop softmax attention probabilities before they weight the value vectors.

• feedForwardDropout? : Option Float

  Drop activated feed-forward hidden units before the second affine map.

• weightInitialization? : Option Init.Scheme

  Attention and feed-forward weight initialization. none keeps each layer's default.

• residualOutputInitialization? : Option Init.Scheme

  Initializer for the attention and feed-forward projections that write to residual streams.

  When omitted, weightInitialization? is used. The separate field supports depth-scaled residual initialization without imposing that convention on every Transformer.

structure TorchLean.nn.TransformerEncoder.Stack.Config : Type

Configuration for a stack of Transformer encoder blocks.

Initialization seeds are allocated by nn.build; they are deliberately not part of model configuration.

• layerCount : ℕ

  Number of encoder blocks.

• block : Block.Config

  Shared configuration for each block.

def TorchLean.nn.TransformerEncoder.Block.Config.validate (config : Config) (kind : String := "TransformerEncoder") : Except String Unit

Validate the shared Transformer block template.

Validation belongs to the configuration itself rather than to an instantiated block so an empty stack cannot silently accept settings that would fail as soon as its depth changes.