LoRA Adapters

LoRA is a parameter-efficient adapter for linear layers: instead of updating the full weight matrix W, training learns two small matrices A and B and uses

W_eff = W + scale * (A * B).

The convention in this file matches the rest of TorchLean's linear specs, where row-batch inputs multiply a weight matrix on the right. If a base linear layer uses W : inDim × outDim, then:

• A : inDim × rank maps the input dimension into the adapter rank,
• B : rank × outDim maps the rank dimension back to the output dimension,
• loraDelta A B scale : inDim × outDim is the trainable low-rank delta.

Reference: Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models”, https://arxiv.org/abs/2106.09685.

structure NN.API.Adapters.LoRA.Params (α : Type) (inDim rank outDim : ℕ) : Type

LoRA adapter parameters for a linear weight matrix of shape inDim × outDim.

The usual LoRA scaling is alpha / rank; TorchLean keeps the final scalar as an explicit scale argument so callers can choose that convention, a schedule, or a test value.

• A : Spec.Tensor α (Spec.Shape.dim inDim (Spec.Shape.dim rank Spec.Shape.scalar))

  Down projection from the model input dimension into the adapter rank.

• B : Spec.Tensor α (Spec.Shape.dim rank (Spec.Shape.dim outDim Spec.Shape.scalar))

  Up projection from the adapter rank into the model output dimension.

def NN.API.Adapters.LoRA.delta {α : Type} [Add α] [Mul α] [Zero α] {inDim rank outDim : ℕ} (p : Params α inDim rank outDim) (scale : α) : Spec.Tensor α (Spec.Shape.dim inDim (Spec.Shape.dim outDim Spec.Shape.scalar))

The low-rank matrix scale * (A * B) added to a base linear weight.

def NN.API.Adapters.LoRA.effectiveWeight {α : Type} [Add α] [Mul α] [Sub α] [Zero α] {inDim rank outDim : ℕ} (base : Spec.Tensor α (Spec.Shape.dim inDim (Spec.Shape.dim outDim Spec.Shape.scalar))) (p : Params α inDim rank outDim) (scale : α) : Spec.Tensor α (Spec.Shape.dim inDim (Spec.Shape.dim outDim Spec.Shape.scalar))

Apply a LoRA adapter to a base linear weight matrix.

def NN.API.Adapters.LoRA.linear {α : Type} [Add α] [Mul α] [Sub α] [Zero α] {batch inDim rank outDim : ℕ} (x : Spec.Tensor α (Spec.Shape.dim batch (Spec.Shape.dim inDim Spec.Shape.scalar))) (base : Spec.Tensor α (Spec.Shape.dim inDim (Spec.Shape.dim outDim Spec.Shape.scalar))) (p : Params α inDim rank outDim) (scale : α) : Spec.Tensor α (Spec.Shape.dim batch (Spec.Shape.dim outDim Spec.Shape.scalar))

Run a batched linear projection using the base weight plus the LoRA delta.