Char-GPT (minGPT-style) Example

This example follows the character-level Transformer from Andrej Karpathy's "Let's build GPT: from scratch, in code, spelled out" lecture:

• build an alphabet (itos) from the training text,
• build a stoi tokenizer from that alphabet,
• train a compact causal Transformer to predict the next character,
• sample text continuations from a prompt.

The karpathy preset follows the lecture configuration: batch size 64, context length 256, width 384, six attention heads, six pre-normalized Transformer blocks, ReLU feed-forward layers, dropout 0.2, AdamW, and 5,000 updates. The CUDA command executes the numerical path in float32. TorchLean applies dropout to the attention and feed-forward sublayer outputs; unlike the lecture code, it does not yet apply a second dropout to the attention weights themselves. All dimensions remain command-line choices.

Training draws a fresh deterministic batch of corpus windows at every step. The windows are built on demand, so a long run does not retain thousands of large one-hot tensors in host memory.

Unlike the introductory model examples, this file intentionally uses the low-level Module API. The indexed objective receives token IDs while the model parameters and loss use floating-point storage, so it crosses a mixed-dtype boundary that the homogeneous Trainer interface does not express. Ordinary floating-point models should use Trainer.new and trainer.train.

Quick check:

scripts/lake.sh -Kcuda=true build torchlean:exe
scripts/lake.sh -Kcuda=true exe torchlean chargpt --device cuda --tiny-shakespeare --preset smoke

Full lecture experiment:

scripts/lake.sh -Kcuda=true exe torchlean chargpt --device cuda --tiny-shakespeare --preset karpathy

Reference: https://github.com/karpathy/ng-video-lecture/blob/master/gpt.py.

def NN.Examples.Models.Sequence.CharGpt.exeName : String

CLI subcommand name used in terminal banners and error messages.

def NN.Examples.Models.Sequence.CharGpt.takeInputText (args : List String) : IO (String × List String)

Parse corpus flags and return the UTF-8 training text plus remaining CLI arguments.

def NN.Examples.Models.Sequence.CharGpt.buildAlphabet (s : String) : Array Char

Build a deterministic character alphabet from the corpus.

def NN.Examples.Models.Sequence.CharGpt.defaultLogPath : System.FilePath

Default JSON loss-curve path for this command.

structure NN.Examples.Models.Sequence.CharGpt.Preset : Type

Architecture and evaluation controls independent of the corpus and runtime device.

• modelWidth : ℕ
• attentionHeads : ℕ
• transformerLayers : ℕ
• dropoutProbability : Float
• trainingSteps : ℕ
• batchSize : ℕ
• contextLength : ℕ
• lr : Float
• evalEvery : ℕ
• evaluationBatches : ℕ
• generationLength : ℕ

@[instance_reducible]

instance NN.Examples.Models.Sequence.CharGpt.instReprPreset : Repr Preset

def NN.Examples.Models.Sequence.CharGpt.instReprPreset.repr : Preset → ℕ → Std.Format

def NN.Examples.Models.Sequence.CharGpt.Preset.smoke : Preset

Fast configuration used to validate the complete training and generation path.

def NN.Examples.Models.Sequence.CharGpt.Preset.karpathy : Preset

Hyperparameters from Karpathy's final Tiny Shakespeare lecture model.

def NN.Examples.Models.Sequence.CharGpt.Preset.parse (name : String) : Except String Preset

Resolve a --preset name. smoke is a short local run; karpathy reproduces the hyperparameters from the nanoGPT character-level Shakespeare run.

def NN.Examples.Models.Sequence.CharGpt.usage : String

Help text for character-level GPT training.

structure NN.Examples.Models.Sequence.CharGpt.Options : Type

Command-local controls for CharGPT training, checkpointing, and generation.

• training : TorchLean.CLI.Training.OptimizerOptions

  Optimizer, step, batching, and logging controls.

• generation : TorchLean.text.GenerationOptions

  Prompt and token-sampling policy.

• checkpoint : TorchLean.text.CheckpointOptions

  Optional checkpoint input and output paths.

• contextLength : ℕ

  Context length in characters.

def NN.Examples.Models.Sequence.CharGpt.instReprOptions.repr : Options → ℕ → Std.Format

@[instance_reducible]

instance NN.Examples.Models.Sequence.CharGpt.instReprOptions : Repr Options

def NN.Examples.Models.Sequence.CharGpt.Options.parse (args : List String) (defaults : Preset) (generationDefaults : TorchLean.text.GenerationOptions) : Except String (Options × List String)

Parse training controls after the experiment preset has supplied defaults.

@[reducible, inline]

abbrev NN.Examples.Models.Sequence.CharGpt.Predictor (α : Type) [TorchLean.Storage α] (batchSize contextLength vocabularySize : ℕ) : Type

Fitted predictor for a runtime-sized character GPT model.

def NN.Examples.Models.Sequence.CharGpt.generate {α : Type} [TorchLean.Storage α] {promptLength : ℕ} (toFloat : α → Float) (batchSize contextLength vocabularySize : ℕ) [NeZero vocabularySize] (predict : Predictor α batchSize contextLength vocabularySize) (promptTokens : TorchLean.Tensor (Fin vocabularySize) [promptLength]) (options : TorchLean.text.GenerationOptions) (allowToken : Fin vocabularySize → Bool := fun (x : Fin vocabularySize) => true) (paddingTokenId : Fin vocabularySize := 0) : IO (TorchLean.Tensor (Fin vocabularySize) [promptLength + options.newTokenCount])

Autoregressively extend character token ids using a trained CharGPT model.

def NN.Examples.Models.Sequence.CharGpt.main (args : List String) : IO UInt32

CLI entrypoint for character-level GPT training and sampling.