3. Runtime, Autograd, and Interop🔗

Training changes the parameters in the forward map

f_\theta(x)=W_2\,\operatorname{ReLU}(W_1x+b_1)+b_2.

To compute an update, reverse mode needs intermediate values from the forward pass, and the optimizer may need state from earlier updates. As the model runs, the autograd tape retains that information. CUDA and LibTorch add a choice of implementation for the numerical operations.

At the ReLU, reverse mode needs to know which pre-activations were positive. At a linear layer, it needs the input to form the weight gradient and the weight matrix to propagate sensitivity back toward the input. A forward pass therefore leaves information that a later backward pass will use. Execution choices affect where that information lives, how long it stays available, and which operations can consume it.

Scalar arithmetic and execution strategy are separate choices. Recording a typed graph doesn't by itself choose CUDA, and moving tensors to a device doesn't change the mathematical definition of the loss. Interoperation adds another concrete question: whether the imported weights, layouts, and operation conventions describe the same model. Following one forward and backward calculation through these interfaces makes their roles easier to distinguish.