GPT-2-Style Causal Language Model Example

Runnable torchlean gpt2 example. It builds a GPT-2-style causal transformer over byte-level tokens, with optional real text input from tiny-shakespeare or --data-file PATH.

For the simplest "Karpathy-style single text file" path, use torchlean chargpt (character-level tokenizer). This gpt2 command is byte-level and shows the Transformer block wiring and save/reload loop.

python3 scripts/datasets/download_example_data.py --tiny-shakespeare
lake -R -K cuda=true exe torchlean gpt2 --device cuda --tiny-shakespeare --steps 1 --windows 1 \
  --generate 0

def NN.Examples.Models.Sequence.Gpt2.exeName : String

CLI subcommand name used in terminal banners and error messages.

def NN.Examples.Models.Sequence.Gpt2.defaultLogPath : System.FilePath

Default JSON loss-curve path for this command.

def NN.Examples.Models.Sequence.Gpt2.usage : String

Complete command help, including the text and training flags parsed after runtime selection.

def NN.Examples.Models.Sequence.Gpt2.batchSize : ℕ

Batch size for the byte-level causal Transformer.

theorem NN.Examples.Models.Sequence.Gpt2.instNeZeroNatBatchSize : NeZero batchSize

def NN.Examples.Models.Sequence.Gpt2.contextLength : ℕ

Prompt/target window length for the runnable GPT example.

def NN.Examples.Models.Sequence.Gpt2.vocabularySize : ℕ

Byte vocabulary width used by the one-hot tokenizer.

theorem NN.Examples.Models.Sequence.Gpt2.instNeZeroNatVocabularySize : NeZero vocabularySize

def NN.Examples.Models.Sequence.Gpt2.byteIndex (id : ℕ) : Fin vocabularySize

Embed a byte id in the complete 256-entry byte vocabulary.

def NN.Examples.Models.Sequence.Gpt2.attentionHeads : ℕ

Number of attention heads in the miniature Transformer block.

def NN.Examples.Models.Sequence.Gpt2.attentionHeadWidth : ℕ

Per-head embedding width.

def NN.Examples.Models.Sequence.Gpt2.modelWidth : ℕ

Transformer embedding width.

def NN.Examples.Models.Sequence.Gpt2.feedForwardWidth : ℕ

Hidden width of the feed-forward sublayer.

def NN.Examples.Models.Sequence.Gpt2.transformerLayers : ℕ

Number of Transformer encoder blocks.

theorem NN.Examples.Models.Sequence.Gpt2.instNeZeroNatContextLength : NeZero contextLength

theorem NN.Examples.Models.Sequence.Gpt2.instNeZeroNatModelWidth : NeZero modelWidth

@[reducible, inline]

abbrev NN.Examples.Models.Sequence.Gpt2.input : Spec.Shape

Input shape: batched byte-level one-hot token windows.

@[reducible, inline]

abbrev NN.Examples.Models.Sequence.Gpt2.output : Spec.Shape

Output shape: one vocabulary-logit row for every input token position.

def NN.Examples.Models.Sequence.Gpt2.model : TorchLean.nn.Builder (TorchLean.nn.Sequential input output)

Public GPT-style causal Transformer constructor specialized to the byte-level config.

structure NN.Examples.Models.Sequence.Gpt2.Options : Type

Command-local controls for GPT training, checkpointing, generation, and the prompt loop.

• training : TorchLean.CLI.Training.OptimizerOptions

  Optimizer, step, batching, and logging controls.

• generation : TorchLean.text.GenerationOptions

  Prompt and token-sampling policy.

• window : TorchLean.text.WindowOptions

  Number of corpus windows exposed to training.

• checkpoint : TorchLean.text.CheckpointOptions

  Optional checkpoint input and output paths.

• interaction : TorchLean.text.InteractiveOptions

  Terminal prompt-loop policy.

def NN.Examples.Models.Sequence.Gpt2.instReprOptions.repr : Options → ℕ → Std.Format

@[instance_reducible]

instance NN.Examples.Models.Sequence.Gpt2.instReprOptions : Repr Options

def NN.Examples.Models.Sequence.Gpt2.Options.parse (args : List String) (defaultSteps : ℕ) : Except String (Options × List String)

Parse the byte-level GPT command's training and generation flags.

def NN.Examples.Models.Sequence.Gpt2.batchSampleFromTokenIds (idsByBatch : TorchLean.Tensor ℕ [batchSize, contextLength + 1]) : TorchLean.Sample.Supervised Float input output

Build a batch sample from exactly one token window per batch row.

def NN.Examples.Models.Sequence.Gpt2.takeInputText (args : List String) : IO (String × List String)

Parse GPT-2-specific data flags and return the training corpus plus remaining runtime flags.

def NN.Examples.Models.Sequence.Gpt2.tokenWindowIds (prompt : String) (offset : ℕ) : TorchLean.Tensor ℕ [contextLength]

Byte-token window used for reporting prompt/target text.

def NN.Examples.Models.Sequence.Gpt2.printPredictionReport (label prompt : String) (logits : TorchLean.Tensor Float output) : IO Unit

Print a compact before/after language-model report for the first batch row.

def NN.Examples.Models.Sequence.Gpt2.inputTensorFromIds (ids : TorchLean.Tensor ℕ [contextLength]) : TorchLean.Tensor Float input

Convert byte ids into the typed batched one-hot input tensor used for generation.

@[reducible, inline]

abbrev NN.Examples.Models.Sequence.Gpt2.Predictor : Type

Fitted byte-level GPT predictor.

Training, saved-checkpoint inference, and future optimized runners all provide this one closure. Generation only needs a logit-producing function; it does not depend on where the logits came from.

def NN.Examples.Models.Sequence.Gpt2.generateSampledFromIds {promptLength : ℕ} (predict : Predictor) (promptTokens : TorchLean.Tensor ℕ [promptLength]) (steps : ℕ) (temperature : Float) (topK seed repeatWindow : ℕ) (repeatPenalty : Float) (asciiOnly : Bool) : IO (TorchLean.Tensor ℕ [promptLength + steps])

Autoregressively extend byte token ids using a trained byte-level GPT model.

def NN.Examples.Models.Sequence.Gpt2.generateSampled (predict : Predictor) (prompt : String) (steps : ℕ) (temperature : Float) (topK seed repeatWindow : ℕ) (repeatPenalty : Float) (asciiOnly : Bool) : IO (TorchLean.Tensor ℕ [(TorchLean.text.Tokenizer.byte.encode prompt).size + steps])

Encode a string prompt and autoregressively extend it.

def NN.Examples.Models.Sequence.Gpt2.samplesFromCorpus (corpus : String) (windows : ℕ) : TorchLean.Data.SampleStream (TorchLean.Sample.Supervised Float input output)

Build a finite training set from approximately evenly spaced corpus windows.

def NN.Examples.Models.Sequence.Gpt2.interactiveLoopFloat (predict : Predictor) (options : Options) : IO Unit

Interactive prompt loop for the in-memory Float model.

Each line is appended to the current byte context, decoded through the trained local model, and then kept as context for the next prompt unless the user clears it.

partial def NN.Examples.Models.Sequence.Gpt2.interactiveLoopFloat.loop (predict : Predictor) (options : Options) (stdin : IO.FS.Stream) {count : ℕ} (ctx : TorchLean.Tensor ℕ [count]) : IO Unit

def NN.Examples.Models.Sequence.Gpt2.trainAndDecode (runtime : TorchLean.Runtime.Config) (corpus : String) (options : Options) : IO (Float × Float × String)

Train the byte-level model and decode a prediction report.

Inputs and reports use host Float, while RunConfig.arithmetic selects the arithmetic used by the trainer. The command fixes that runtime choice to native execution.

def NN.Examples.Models.Sequence.Gpt2.main (args : List String) : IO UInt32

CLI entrypoint for byte-level GPT training, sampling, logging, and checkpointing.