3. Runtime, Autograd, and Interop🔗

Training changes the parameters in the forward map

f_\theta(x)=W_2\,\operatorname{ReLU}(W_1x+b_1)+b_2.

To compute an update, reverse mode needs intermediate values from the forward pass, and the optimizer may need state from earlier updates. Eager execution and typed graph execution retain that information differently. CUDA and LibTorch add a choice of implementation for the numerical operations.

At the ReLU, reverse mode needs to know which pre-activations were positive. At a linear layer, it needs the input to form the weight gradient and the weight matrix to propagate sensitivity back toward the input. A forward pass therefore leaves information that a later backward pass will use. Execution choices affect where that information lives, how long it stays available, and which operations can consume it.

The scalar arithmetic and the execution strategy are separate choices. Recording a typed graph does not by itself choose CUDA, and moving tensors to a device does not change the mathematical definition of the loss. Interoperation adds another concrete question: whether the imported weights, layouts, and operation conventions describe the same model. Following one forward and backward calculation through these interfaces makes their roles easier to distinguish.