Text Datasets #
Causal language-model sample and dataset constructors.
Build a batched one-hot causal-language-model sample by repeating one token window across every batch row.
The token list represents a seqLen + 1 window. Shorter lists are padded and longer lists are
truncated by the causal-LM construction.
Instances For
Build a batched one-hot causal-language-model sample from one token window per batch row.
Use this for GPT-style examples that already know the per-row (seqLen + 1) token window they want
each batch row to see.
Instances For
Build a batched one-hot causal-language-model sample from an array of per-row token windows.
Rows past the end of the array use the explicit fallback window, so partial-batch behavior stays
visible at the call site.
Instances For
Build a batched one-hot causal-language-model sample from a token array by choosing one
deterministic (seqLen + 1) window per batch row.
Use this for GPT-style trainers that keep a tokenized corpus in memory and derive each batch from
the same (tokens, seed, step) rule.
Instances For
Flatten one (seqLen + 1) token window into causal-LM (x, y) id lists.
For a window $[t_0,t_1,\ldots,t_{\mathrm{seqLen}}]$, the model input is $[t_0,\ldots,t_{\mathrm{seqLen}-1}]$ and the target is $[t_1,\ldots,t_{\mathrm{seqLen}}]$. Short windows are padded rather than rejected so small corpora can still exercise the training loop.
Instances For
Materialize row weights for a flattened (batch × seqLen) objective.
Extra values are ignored and missing values receive weight zero. The masked sample builder below
constructs exactly batch * seqLen entries.
Instances For
Build an indexed-token causal-language-model batch from an array-backed corpus.
The result contains the input and next-token target as separate Nat tensors. Keeping their scalar
type discrete avoids one-hot expansion and prevents token ids from entering floating-point
autograd.
Instances For
Read the target-mask row paired with a causal-language-model window.
The input window begins at offset, while its first prediction target is the following token.
Consequently the returned row starts at targetMask[offset + 1].
Instances For
Build a token-id causal-language-model batch with an explicit target mask.
targetMask[i] states whether token tokens[i] should contribute when it is used as a next-token
target. The mask therefore shifts with the labels: a row containing
tokens[offset], ..., tokens[offset + seqLen] receives target weights from
targetMask[offset + 1], ..., targetMask[offset + seqLen].
Active rows receive weight 1 / activeCount, so the resulting weighted cross entropy is a mean over
exactly the selected targets. If a sampled batch has no active target, all weights are zero and the
loss is zero. A shorter mask is treated as false beyond its end.
Instances For
Build one unbatched one-hot causal-language-model sample directly from a token list.
The token list represents a seqLen + 1 window. Shorter lists are padded and longer lists are
truncated by the causal-LM construction.
Instances For
Build one unbatched one-hot causal-language-model sample from a text corpus string.
This takes one (seqLen + 1) byte window from the UTF-8 bytes of input, converts it to one-hot
x/y matrices, and casts the result into the runtime-selected scalar.
Instances For
Build one fixed-batch one-hot causal-language-model sample from a text corpus string by repeating the same text window across every batch row.
Instances For
Build a runtime-polymorphic dataset containing one unbatched causal-language-model sample from a text corpus string.
Instances For
Build a runtime-polymorphic dataset containing one causal-language-model sample repeated across a fixed batch axis.
Use this when the model itself owns the batch dimension but the example naturally starts from one text window.