Text Datasets

Causal language-model sample and dataset constructors.

def TorchLean.Data.CausalLM.tokenSample {β : Type} [Storage β] (batchShape : Shape) (sequenceLength : ℕ) (window : Tensor β (batchShape.appendDim (sequenceLength + 1))) : Sample.Supervised β (batchShape.appendDim sequenceLength) (batchShape.appendDim sequenceLength)

Split exact (sequenceLength + 1) token windows into causal-LM input and target tensors.

For a window $[t_0,t_1,\ldots,t_{\mathrm{sequenceLength}}]$, the model input is $[t_0,\ldots,t_{\mathrm{sequenceLength}-1}]$ and the target is $[t_1,\ldots,t_{\mathrm{sequenceLength}}]$. Corpus readers perform any requested padding before constructing this tensor, so the split itself cannot invent or discard tokens.

def TorchLean.Data.CausalLM.oneHotInputs {α : Type} [Storage α] [Zero α] [One α] {shape : Shape} (vocabularySize : ℕ) (tokens : Tensor (Fin vocabularySize) shape) : Tensor α (shape.appendDim vocabularySize)

One-hot encode every bounded token id along a new final vocabulary dimension.

def TorchLean.Data.CausalLM.oneHotSample {α : Type} [Storage α] [Zero α] [One α] (batchShape : Shape) (sequenceLength vocabularySize : ℕ) (tokens : Tensor (Fin vocabularySize) (batchShape.appendDim (sequenceLength + 1))) : Sample.Supervised α ((batchShape.appendDim sequenceLength).appendDim vocabularySize) ((batchShape.appendDim sequenceLength).appendDim vocabularySize)

Build a one-hot causal-language-model sample over an arbitrary batch shape.

The input and target both append sequence and vocabulary axes to batchShape.

def TorchLean.Data.CausalLM.oneHotBatch {tokenCount : ℕ} {α : Type} [Storage α] [Zero α] [One α] (batchSize sequenceLength vocabularySize : ℕ) (tokens : Tensor ℕ [tokenCount]) (seed step : ℕ) (paddingTokenId : ℕ := 0) : Except String (Sample.Supervised α [batchSize, sequenceLength, vocabularySize] [batchSize, sequenceLength, vocabularySize])

Build a batched one-hot causal-language-model sample from a token tensor by choosing one deterministic (sequenceLength + 1) window per batch row.

Use this for GPT-style trainers that keep a tokenized corpus in memory and derive each batch from the same (tokens, seed, step) rule.

def TorchLean.Data.CausalLM.tokenBatch {tokenCount : ℕ} (vocabularySize batchSize sequenceLength : ℕ) (tokens : Tensor ℕ [tokenCount]) (seed step : ℕ) (paddingTokenId : ℕ := 0) : Except String (Sample.Supervised (Fin vocabularySize) [batchSize, sequenceLength] [batchSize, sequenceLength])

Build an indexed-token causal-language-model batch from a tensor corpus.

The result contains the input and next-token target as separate bounded-index tensors. Validation happens once at this corpus boundary; model code therefore has no out-of-range token case.

def TorchLean.Data.CausalLM.byteSample {α : Type} [Storage α] [Zero α] [One α] (sequenceLength vocabularySize : ℕ) (encodeToken : ℕ → Fin vocabularySize) (text : String) : Sample.Supervised α [sequenceLength, vocabularySize] [sequenceLength, vocabularySize]

Build one unbatched one-hot causal-language-model sample from a text corpus string.

This takes one (sequenceLength + 1) byte window from the UTF-8 bytes of text, converts it to one-hot input/target matrices, and casts the result into the runtime-selected arithmetic representation.

def TorchLean.Data.CausalLM.byteSamples {α : Type} [Storage α] [Zero α] [One α] (sequenceLength vocabularySize : ℕ) (encodeToken : ℕ → Fin vocabularySize) (windowCount : ℕ) (text : String) (paddingTokenId : ℕ := 0) : SampleStream (Sample.Supervised α [sequenceLength, vocabularySize] [sequenceLength, vocabularySize])

Build a finite causal-language-model dataset from approximately evenly spaced byte windows.

Offsets are measured in UTF-8 bytes, matching byteSample. Short corpora remain total because byteTokenWindow pads beyond the end, while windowCount = 0 returns an empty dataset.

def TorchLean.Data.CausalLM.byteBatch {α : Type} [Storage α] [Zero α] [One α] (batchSize sequenceLength vocabularySize : ℕ) (encodeToken : ℕ → Fin vocabularySize) (text : String) : Sample.Supervised α [batchSize, sequenceLength, vocabularySize] [batchSize, sequenceLength, vocabularySize]

Build one fixed-batch one-hot causal-language-model sample from a text corpus string by repeating the same text window across every batch row.