Causal Transformer Text Example

Runnable torchlean transformer example. It reads a local text corpus, builds a shifted next-byte feature sample, and trains a compact causal Transformer on that real text window.

This command uses the same public causal model family as the GPT examples. Future tokens are masked, so a position cannot read the byte it is being trained to predict.

python3 scripts/datasets/download_example_data.py --tiny-shakespeare
lake -R -K cuda=true exe torchlean transformer --device cuda --tiny-shakespeare --steps 1

def NN.Examples.Models.Sequence.Transformer.exeName : String

CLI subcommand name used in terminal banners and error messages.

def NN.Examples.Models.Sequence.Transformer.defaultLogPath : System.FilePath

Default JSON loss-curve path for this command.

def NN.Examples.Models.Sequence.Transformer.contextLength : ℕ

Short multi-token window for the quick encoder training run.

def NN.Examples.Models.Sequence.Transformer.modelWidth : ℕ

Transformer feature width.

def NN.Examples.Models.Sequence.Transformer.defaultWindows : ℕ

Default number of distinct corpus windows exposed to the trainer.

theorem NN.Examples.Models.Sequence.Transformer.instNeZeroNatModelWidth : NeZero modelWidth

def NN.Examples.Models.Sequence.Transformer.byteBucket (id : ℕ) : Fin modelWidth

Compact byte vocabulary: encode byte id b as b % 4; collisions are intentional.

def NN.Examples.Models.Sequence.Transformer.attentionHeads : ℕ

Number of attention heads.

def NN.Examples.Models.Sequence.Transformer.attentionHeadWidth : ℕ

Per-head width; attentionHeads * attentionHeadWidth = modelWidth.

def NN.Examples.Models.Sequence.Transformer.feedForwardWidth : ℕ

Feed-forward hidden width inside the encoder block.

@[reducible, inline]

abbrev NN.Examples.Models.Sequence.Transformer.modelConfig : TorchLean.nn.models.CausalTransformer.Config

Causal language-model configuration.

@[reducible, inline]

abbrev NN.Examples.Models.Sequence.Transformer.input : Spec.Shape

Input shape: batched one-hot byte buckets.

@[reducible, inline]

abbrev NN.Examples.Models.Sequence.Transformer.output : Spec.Shape

Output shape: one next-byte logit row per token position.

def NN.Examples.Models.Sequence.Transformer.model : TorchLean.nn.Builder (TorchLean.nn.Sequential input output)

Compact causal Transformer used by the runnable text example.

def NN.Examples.Models.Sequence.Transformer.samples (corpus : String) (windows : ℕ) : TorchLean.Data.SampleStream (TorchLean.Sample.Supervised Float input output)

Build a finite next-byte dataset from evenly spaced corpus windows.

def NN.Examples.Models.Sequence.Transformer.train (runtime : TorchLean.Runtime.Config) (data : RealData.TextWindowFlags) (flags : TorchLean.CLI.Training.OptimizerOptions) : IO Unit

Train the causal Transformer with the public Trainer surface.

def NN.Examples.Models.Sequence.Transformer.main (args : List String) : IO UInt32

CLI entrypoint for the causal Transformer text command.