Recurrent Models #
RNN, GRU, and LSTM sequence models with a linear projection at every time step.
Configuration for an RNN, GRU, or LSTM followed by a time-distributed linear head.
The model consumes a fixed-length sequence. Constructors accept any batchShape for batches,
ensembles, or other pointwise collections of sequences.
- sequenceLength : ℕ
Number of time steps.
- inputWidth : ℕ
Number of features presented at each time step.
- outputWidth : ℕ
Number of features produced at each time step.
Instances For
Dimension checks shared by the recurrent model family.
kind is the label that appears in the error message, so a failure names the model the user asked
for rather than this helper. Keeping the checks here is why Recurrent.Config.validate and the
sequence-to-sequence variants cannot drift into reporting different messages for the same mistake.
Instances For
Input tensor shape: batchShape × sequenceLength × inputWidth.
Instances For
Output tensor shape: batchShape × sequenceLength × outputWidth.
Instances For
Vanilla RNN core plus time-distributed linear head:
rnn(sequenceLength, inputWidth, hiddenWidth) → linear(hiddenWidth, outputWidth).
Instances For
Gated recurrent unit plus time-distributed linear head:
gru(sequenceLength, inputWidth, hiddenWidth) → linear(hiddenWidth, outputWidth).
By default the core uses the Cho-style reset-before convention. With reset gate $r_t$, update gate $z_t$ and previous hidden state $h_{t-1}$, its candidate and update are
$$ n_t = \tanh\!\left(W_{nx}x_t + W_{nh}(r_t \odot h_{t-1}) + b_n\right), \qquad h_t = (1-z_t)\odot n_t + z_t\odot h_{t-1}. $$
Choose convention := .resetAfter for PyTorch's candidate equation:
$$ n_t = \tanh\!\left(W_{nx}x_t + b_{nx} + r_t\odot(W_{nh}h_{t-1} + b_{nh})\right). $$
Here the reset acts after the recurrent affine map, including its bias. Moving the reset across
a general recurrent matrix changes the function, and the reset-dependent bias cannot generally
be folded into a constant bias. Repacking PyTorch weights alone therefore does not preserve a
general PyTorch GRU's behavior. The reset-after constructor implements this different recurrence
directly and stores weight_ih, weight_hh, bias_ih, bias_hh in PyTorch's packed gate order.
The reset-before core stores a matrix and one bias for each gate, in reset, update, candidate order.
Each matrix has shape [hiddenWidth, inputWidth + hiddenWidth], with input columns followed by
hidden columns.
PyTorch stores separate input and recurrent matrices and biases, each packed in the same gate
order. Its reset and update bias pairs can be added for forward evaluation; the candidate differs
by the reset placement above.
Input and output shapes are batchShape ++ [sequenceLength, inputWidth] and
batchShape ++ [sequenceLength, outputWidth]. The core weights are shared across batch entries,
and the same linear head projects every hidden state. Each call starts every sequence at zero
hidden state and returns all projected time steps. There is no initial-state argument or separate
final-state result, and hidden state is not carried between calls.
Instances For
LSTM core plus time-distributed linear head:
lstm(sequenceLength, inputWidth, hiddenWidth) → linear(hiddenWidth, outputWidth).