Text Datasets

Causal language-model sample and dataset constructors.

def TorchLean.Data.CausalLM.tokenSample {β : Type} [Storage β] (batchShape : Shape) (sequenceLength : ℕ) (window : Tensor β (batchShape.appendDim (sequenceLength + 1))) : Sample.Supervised β (batchShape.appendDim sequenceLength) (batchShape.appendDim sequenceLength)

Split exact (sequenceLength + 1) token windows into causal-LM input and target tensors.

For a window $[t_0,t_1,\ldots,t_{\mathrm{sequenceLength}}]$, the model input is $[t_0,\ldots,t_{\mathrm{sequenceLength}-1}]$ and the target is $[t_1,\ldots,t_{\mathrm{sequenceLength}}]$. Corpus readers perform any requested padding before constructing this tensor, so the split itself cannot invent or discard tokens.

def TorchLean.Data.CausalLM.oneHotSample {α : Type} [Storage α] [Zero α] [One α] (batchShape : Shape) (sequenceLength vocabularySize : ℕ) (tokens : Tensor (Fin vocabularySize) (batchShape.appendDim (sequenceLength + 1))) : Sample.Supervised α ((batchShape.appendDim sequenceLength).appendDim vocabularySize) ((batchShape.appendDim sequenceLength).appendDim vocabularySize)

Build a one-hot causal-language-model sample over an arbitrary batch shape.

The input and target both append sequence and vocabulary axes to batchShape.

def TorchLean.Data.CausalLM.tokenBatch {tokenCount : ℕ} (vocabularySize batchSize sequenceLength : ℕ) (tokens : Tensor ℕ [tokenCount]) (seed step : ℕ) (paddingTokenId : ℕ := 0) : Except String (Sample.Supervised (Fin vocabularySize) [batchSize, sequenceLength] [batchSize, sequenceLength])

Build an indexed-token causal-language-model batch from a tensor corpus.

The result contains the input and next-token target as separate bounded-index tensors. Validation happens once at this corpus boundary; model code therefore has no out-of-range token case.

def TorchLean.Data.CausalLM.byteSample {α : Type} [Storage α] [Zero α] [One α] (sequenceLength vocabularySize : ℕ) (encodeToken : ℕ → Fin vocabularySize) (text : String) : Sample.Supervised α [sequenceLength, vocabularySize] [sequenceLength, vocabularySize]

Build one unbatched one-hot causal-language-model sample from a text corpus string.

This takes one (sequenceLength + 1) byte window from the UTF-8 bytes of text and constructs one-hot input/target matrices directly in the selected scalar type α.

def TorchLean.Data.CausalLM.byteSamples {α : Type} [Storage α] [Zero α] [One α] (sequenceLength vocabularySize : ℕ) (encodeToken : ℕ → Fin vocabularySize) (windowCount : ℕ) (text : String) (paddingTokenId : ℕ := 0) : SampleStream (Sample.Supervised α [sequenceLength, vocabularySize] [sequenceLength, vocabularySize])

Build an indexed stream of causal-language-model samples from approximately evenly spaced byte windows.

Offsets are measured in UTF-8 bytes, matching byteSample. Short corpora remain total because byteTokenWindow pads beyond the end, while windowCount = 0 returns an empty stream. Samples are constructed on access. For supervised training, select α := Float and wrap the result with Data.fromStream.

def TorchLean.Data.CausalLM.byteBatch {α : Type} [Storage α] [Zero α] [One α] (batchSize sequenceLength vocabularySize : ℕ) (encodeToken : ℕ → Fin vocabularySize) (text : String) : Sample.Supervised α [batchSize, sequenceLength, vocabularySize] [batchSize, sequenceLength, vocabularySize]

Build one fixed-batch one-hot causal-language-model sample from a text corpus string by repeating the same text window across every batch row.