7.2.1.1. The Architecture🔗

The command constructs a CausalTransformerConfig from the runtime options:

D=32,\qquad H=4,\qquad D_h=D/H=8,\qquad L=2.

For a batch of token windows I\in\mathbb{N}^{B\times T}, the model first checks that every token id is smaller than V$ and gathers embeddings

E[I]\in\mathbb{R}^{B\times T\times D},

adds a learned positional table, and applies two pre-normalized Transformer blocks. In one block,

\begin{aligned} Z_1&=X+\operatorname{Dropout} \left(\operatorname{MHA}(\operatorname{LN}(X))\right),\\ Z_2&=Z_1+\operatorname{Dropout} \left(W_2\,\rho(W_1\operatorname{LN}(Z_1)+b_1)+b_2\right). \end{aligned}

The final layer normalization and linear projection produce logits

\operatorname{logits}\in\mathbb{R}^{B\times T\times V}.

The training target is the same token window shifted by one position. Cross entropy at location t$ therefore uses the target character x_{t+1}. Token ids and targets enter the runtime as Tensor Nat; they are not transported through floating-point tensors. The reusable GPT API also supports tying the output projection to the embedding table. This CharGPT command uses an independent output head.

The mask is semantic, not merely a convenient floating-point bias:

j>i\quad\Longrightarrow\quad \operatorname{attentionWeight}_{i,j}=0.

The source of the shared architecture is NN/API/Models/Gpt2.lean. The corpus split, configuration presets, evaluation loop, generation, and checkpoint handling are in NN/Examples/Models/Sequence/CharGpt.lean.

7.2.1.2. Make The Run Your Own🔗

Every structural parameter in the smoke preset can be overridden:

lake -R -K cuda=true exe torchlean chargpt --device cuda \
  --tiny-shakespeare --preset smoke \
  --width 64 --heads 4 --layers 3 --seq-len 64 \
  --batch 8 --steps 20 --eval-every 5 --eval-iters 4

Two constraints are worth testing deliberately.

First, --heads must divide --width, because the model width is split into equal head dimensions. Try --width 30 --heads 4; the command rejects the configuration before training.

Second, increasing seq-len increases attention work quadratically:

\operatorname{cost}_{\mathrm{attention}} =O(BHT^2D_h).

Increasing width mostly changes matrix multiplication; increasing context changes both the score matrix and its stored autograd state. This is why a “larger Transformer” is not described by parameter count alone.

The karpathy preset records the well-known lecture configuration: batch 64, context 256, width 384, six heads, six blocks, dropout 0.2, and 5,000 updates. It is a real long-running experiment, not a quick check:

lake -R -K cuda=true exe torchlean chargpt --device cuda \
  --tiny-shakespeare --preset karpathy

TorchLean follows the architecture and hyperparameter lineage of Karpathy's lecture model, but the runtime and implementation are TorchLean's. It does not claim bit-for-bit identity with the Python program or with a pretrained GPT-2 checkpoint.

7.2.2.1. Residual Geometry🔗

The current ResNet application crops each image to 3\times8\times8, lifts it to four hidden channels, applies two shape-preserving residual blocks, globally averages the spatial axes, and emits ten logits.

At every residual join, both branches have shape

1\times4\times8\times8.

That equality is part of model construction. The optimizer never sees a malformed residual block. It may still see a numerically poor model, a bad label, or an incorrect unproved kernel; shape typing solves one problem rather than pretending to solve all of them.

7.2.2.2. Patch Geometry🔗

The ViT application uses a convolution to create patch embeddings, reshapes the patch grid into a token sequence, and applies one Transformer encoder block. If the patch output grid is H'\times W', the token count is

N=H'W'.

The conversion

B\times D\times H'\times W' \longrightarrow B\times N\times D

is an explicit layer in the reusable ViT model. It is not a hidden view whose correctness depends on remembering which axis currently stores channels.

7.2.2.3. A Useful Comparison🔗

Try ten updates on both commands:

lake exe torchlean resnet --device cpu --n-total 8 --steps 10
lake exe torchlean vit --device cpu --n-total 8 --steps 10

Do not compare the final losses as if this were a controlled architecture benchmark. The examples are intentionally compact and their initialization and capacity differ. Instead, compare:

• model summaries and parameter shapes;

• the residual join versus the spatial-to-token conversion;

• the backend capsules printed by adding --show-backend;

• the JSON metadata written by --log.

That comparison teaches more about TorchLean's architecture than a single accuracy number.

7.2.3.1. What The Spectral Layer Computes🔗

Let v\in\mathbb R^{N\times C} be a field with C latent channels. The spectral branch computes a discrete Fourier transform, retains m modes at each end of the real spectrum, multiplies those modes by learned complex weights, and transforms back:

\widehat v_k =\sum_{j=0}^{N-1}v_j e^{-2\pi i jk/N},

\widehat y_k =R_{\theta,k}\widehat v_k \quad\text{for retained }k,\qquad y=\mathcal F^{-1}(\widehat y).

A pointwise linear branch is added before the activation. The command uses grid N=32, width 8, eight retained modes on each side, and one spectral residual block.

The reusable FNO constructor states the grid and mode constraints independently of the backend. The Burgers application chooses between:

• a portable dense multidimensional DFT path;

• a fused CUDA real-FFT autograd operation backed by cuFFT.

The output line spectral path=fused cuFFT RFFT autograd op is therefore part of the scientific record. It says which external numerical provider produced the transform. It does not turn cuFFT into a Lean-proved implementation.

7.2.3.2. Change The Evidence, Not Only The Runtime🔗

Run the same small dataset on CPU:

lake exe torchlean fno1d_burgers --device cpu \
  --steps 1 --train-rows 4 --test-rows 2 --eval-rows 2

The command reports spectral path=portable dense multidimensional DFT. The model contract and dataset remain the same, but the numerical provider changes. This is precisely the kind of comparison for which backend capsules are useful: provider, reduction policy, layout, and evidence can change without silently changing the model's mathematical interface.