Attention

Multi-head self-attention configuration.

structure TorchLean.nn.MultiHeadAttention.Config : Type

Multi-head self-attention configuration.

PyTorch analogue: torch.nn.MultiheadAttention (conceptually). See https://pytorch.org/docs/stable/generated/torch.nn.MultiheadAttention.html.

modelWidth belongs to the input and output shape, while this configuration controls the internal attention width headCount * headWidth. Query, key, and value projections map from modelWidth into that internal width; the output projection maps back to modelWidth. The two widths therefore do not need to be equal.

• headCount : ℕ

  Number of attention heads. Must be positive.

• headWidth : ℕ

  Per-head embedding dimension. Must be positive.

• weightInitialization? : Option Init.Scheme

  Projection-weight initialization. none retains Xavier-uniform initialization.

• outputWeightInitialization? : Option Init.Scheme

  Optional initializer for the output projection.

  This is separate because deep residual stacks commonly scale the projection that writes back to the residual stream. When omitted, weightInitialization? is used.

• outputBias : Bool

  Add a trainable bias after the output projection.

• inputBias : Bool

  Add independent trainable biases to the query, key, and value projections.

  Each bias has width headCount * headWidth. The default preserves the original bias-free projection layout; setting both this field and outputBias represents four affine projections.

• dropout? : Option Float

  Drop attention probabilities after softmax and before multiplication by values.

  This is separate from dropout on the projected attention output in a Transformer block. Evaluation leaves the probabilities unchanged; none adds no dropout state or seed draw.

def TorchLean.nn.MultiHeadAttention.Config.validate (config : Config) (sequenceLength modelWidth : ℕ) : Except String Unit

Validate attention dimensions and projection initializers.