Char-GPT (minGPT-style) Example

This example follows the character-level Transformer from Andrej Karpathy's "Let's build GPT: from scratch, in code, spelled out" lecture:

• build an alphabet (itos) from the training text,
• build a stoi tokenizer from that alphabet,
• train a compact causal Transformer to predict the next character,
• sample text continuations from a prompt.

The karpathy preset follows the lecture configuration: batch size 64, context length 256, width 384, six attention heads, six pre-normalized Transformer blocks, ReLU feed-forward layers, dropout 0.2, AdamW, and 5,000 updates. The CUDA command executes the numerical path in float32. TorchLean applies dropout to the attention and feed-forward sublayer outputs; unlike the lecture code, it does not yet apply a second dropout to the attention weights themselves. All dimensions remain command-line choices.

Training draws a fresh deterministic batch of corpus windows at every step. The windows are built on demand, so a long run does not retain thousands of large one-hot tensors in host memory.

Quick check:

lake -R -K cuda=true build torchlean:exe
lake -R -K cuda=true exe torchlean chargpt --device cuda --tiny-shakespeare --preset smoke

Full lecture experiment:

lake -R -K cuda=true exe torchlean chargpt --device cuda --tiny-shakespeare --preset karpathy

Reference: https://github.com/karpathy/ng-video-lecture/blob/master/gpt.py.

def NN.Examples.Models.Sequence.CharGpt.exeName : String

CLI subcommand name used in terminal banners and error messages.

def NN.Examples.Models.Sequence.CharGpt.takeInputText (args : List String) : IO (String × List String)

Parse corpus flags and return the UTF-8 training text plus remaining CLI arguments.

def NN.Examples.Models.Sequence.CharGpt.buildAlphabet (s : String) : Array Char

Build a deterministic character alphabet from the corpus.

def NN.Examples.Models.Sequence.CharGpt.defaultLogJson : System.FilePath

Default JSON loss-curve path for this command.

structure NN.Examples.Models.Sequence.CharGpt.ExperimentConfig : Type

Architecture and evaluation controls independent of the corpus and runtime device.

• width : ℕ
• heads : ℕ
• layers : ℕ
• dropout : Float
• steps : ℕ
• batch : ℕ
• seqLen : ℕ
• learningRate : Float
• evalEvery : ℕ
• evalIters : ℕ
• generate : ℕ

@[implicit_reducible]

instance NN.Examples.Models.Sequence.CharGpt.instReprExperimentConfig : Repr ExperimentConfig

def NN.Examples.Models.Sequence.CharGpt.instReprExperimentConfig.repr : ExperimentConfig → ℕ → Std.Format

def NN.Examples.Models.Sequence.CharGpt.ExperimentConfig.smoke : ExperimentConfig

Fast configuration used to validate the complete training and generation path.

def NN.Examples.Models.Sequence.CharGpt.ExperimentConfig.karpathy : ExperimentConfig

Hyperparameters from Karpathy's final Tiny Shakespeare lecture model.

def NN.Examples.Models.Sequence.CharGpt.ExperimentConfig.ofName (name : String) : Except String ExperimentConfig

def NN.Examples.Models.Sequence.CharGpt.usage : String

Help text for character-level GPT training.

def NN.Examples.Models.Sequence.CharGpt.escapeCharIdsForDisplay (t : TorchLean.text.Tokenizer) (ids : List ℕ) : String

Decode token ids for terminal output with control characters escaped.

def NN.Examples.Models.Sequence.CharGpt.asciiAllowed (c : Char) : Bool

Printable-ASCII generation filter used by --ascii-only.

@[reducible, inline]

abbrev NN.Examples.Models.Sequence.CharGpt.Predictor (batch seqLen vocab : ℕ) : Type

Fitted predictor for a runtime-sized character GPT model.

def NN.Examples.Models.Sequence.CharGpt.generateSampledFromIds (batch seqLen vocab : ℕ) (predict : Predictor batch seqLen vocab) (promptIds : List ℕ) (steps : ℕ) (temperature : Float) (topK seed repeatWindow : ℕ) (repeatPenalty : Float) (allowId : ℕ → Bool := fun (x : ℕ) => true) (padId : ℕ := 0) : IO (List ℕ)

Autoregressively extend character token ids using a trained CharGPT model.

def NN.Examples.Models.Sequence.CharGpt.main (args : List String) : IO UInt32

CLI entrypoint for character-level GPT training and sampling.