Low-Rank Adapters

LoRA represents a linear-weight update as two smaller matrices. For a base weight $W : \mathbb{R}^{d_{in}\times d_{out}}$, an adapter of rank $r$ uses an input factor $A : \mathbb{R}^{d_{in}\times r}$ and an output factor $B : \mathbb{R}^{r\times d_{out}}$:

$$W_{eff}=W+sAB.$$

The matrix orientation agrees with TorchLean's row-batch linear layers. This module defines the typed update and its action on a batch; the training code decides which parameters to optimize.

Reference: Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models” (2021), https://arxiv.org/abs/2106.09685.

structure TorchLean.Adapters.LoRA.Parameters (α : Type) [Storage α] (inputWidth rank outputWidth : ℕ) : Type

LoRA factors for a linear weight of shape inputWidth × outputWidth.

• inputFactor : Tensor α [inputWidth, rank]

  Projection from the input dimension to the adapter rank.

• outputFactor : Tensor α [rank, outputWidth]

  Projection from the adapter rank to the output dimension.

def TorchLean.Adapters.LoRA.weightUpdate {α : Type} [Storage α] [Add α] [Mul α] [Zero α] {inputWidth rank outputWidth : ℕ} (parameters : Parameters α inputWidth rank outputWidth) (scale : α) : Tensor α [inputWidth, outputWidth]

The scaled low-rank update $sAB$.

def TorchLean.Adapters.LoRA.effectiveWeight {α : Type} [Storage α] [Add α] [Mul α] [Sub α] [Zero α] {inputWidth rank outputWidth : ℕ} (baseWeight : Tensor α [inputWidth, outputWidth]) (parameters : Parameters α inputWidth rank outputWidth) (scale : α) : Tensor α [inputWidth, outputWidth]

Add a LoRA update to a base linear weight.

def TorchLean.Adapters.LoRA.linear {α : Type} [Storage α] [Add α] [Mul α] [Sub α] [Zero α] {batchShape : Shape} {inputWidth rank outputWidth : ℕ} (input : Tensor α (batchShape.appendDim inputWidth)) (baseWeight : Tensor α [inputWidth, outputWidth]) (parameters : Parameters α inputWidth rank outputWidth) (scale : α) : Tensor α (batchShape.appendDim outputWidth)

Apply a linear map whose weight is augmented by a LoRA update over an arbitrary batch shape.