3. Runtime, Autograd, and Interop🔗

Training introduces state and hardware. Parameters change, tape nodes save intermediate values, and optimizer buffers accumulate history. None of that appears in the clean equation

f_\theta(x)=W_2\,\operatorname{ReLU}(W_1x+b_1)+b_2.

We will follow one step through the runtime, then move the same operation to compiled execution, CUDA, and LibTorch.

1. 3.1. Choosing How A Model Runs
2. 3.2. Inside The Backend Planner
3. 3.3. Differentiation By Example
4. 3.4. Differentiable Scientific Models
5. 3.5. What Actually Runs
6. 3.6. PyTorch Round Trip