Transformer Text Example

Runnable torchlean transformer example. It reads a local text corpus, builds a short sequence reconstruction sample, and trains one transformer encoder block on that real text window.

The reusable model wiring is exposed as TorchLean.nn.models.TransformerEncoder. This command stays small so attention, normalization, the optimizer, logging, and CUDA execution remain easy to test regularly.

python3 scripts/datasets/download_example_data.py --tiny-shakespeare
lake build -R -K cuda=true && lake exe torchlean transformer --cuda --tiny-shakespeare --steps 1

def NN.Examples.Models.Sequence.Transformer.exeName : String

CLI subcommand name used in terminal banners and error messages.

def NN.Examples.Models.Sequence.Transformer.defaultLogJson : System.FilePath

Default JSON loss-curve path for this command.

def NN.Examples.Models.Sequence.Transformer.batch : ℕ

Number of rows in the typed encoder batch.

def NN.Examples.Models.Sequence.Transformer.seqLen : ℕ

Short reconstruction window for the quick encoder training run.

def NN.Examples.Models.Sequence.Transformer.dModel : ℕ

Transformer feature width.

def NN.Examples.Models.Sequence.Transformer.numHeads : ℕ

Number of attention heads.

def NN.Examples.Models.Sequence.Transformer.headDim : ℕ

Per-head width; numHeads * headDim matches dModel.

def NN.Examples.Models.Sequence.Transformer.ffnHidden : ℕ

Feed-forward hidden width inside the encoder block.

def NN.Examples.Models.Sequence.Transformer.cfg : TorchLean.nn.models.TransformerEncoderConfig

API-level encoder configuration shared by shapes and the constructor.

@[reducible, inline]

abbrev NN.Examples.Models.Sequence.Transformer.σ : Shape

Input shape: a batch of sequence rows with dModel features per token.

@[reducible, inline]

abbrev NN.Examples.Models.Sequence.Transformer.τ : Shape

Output shape matches the input because this command trains a reconstruction objective.

def NN.Examples.Models.Sequence.Transformer.model : TorchLean.nn.M (TorchLean.nn.Sequential σ τ)

One reusable transformer encoder block from the public model API.

def NN.Examples.Models.Sequence.Transformer.sample (corpus : String) : TorchLean.SupervisedSample Float σ τ

Build one reconstruction sample from the loaded corpus prefix.

def NN.Examples.Models.Sequence.Transformer.train (opts : TorchLean.Options) (corpusFlags : RealData.TextCorpusFlags) (flags : TorchLean.ModelZoo.LoggedTrainFlags) : IO Unit

Train the Transformer encoder with the public Trainer surface.

def NN.Examples.Models.Sequence.Transformer.main (args : List String) : IO UInt32

CLI entrypoint for the Transformer encoder text command.