BugZoo: constant normalization slices #

Normalization layers should have a boring answer on a constant slice. If every value in the slice being normalized is the same finite value x, then the slice mean is x, the variance is zero, and the normalized activations are zero.

For affine normalization layers this gives the contract:

$$ \operatorname{normalize}([x,x,\ldots])=\beta. $$

The scale/weight gradient for that slice is also zero, because it is multiplied by the normalized activation. This applies to the mathematical core behind LayerNorm, GroupNorm, InstanceNorm, and BatchNorm; those layers differ mainly in which axes define the slice.

The paired Python reproducer checks this contract against PyTorch normalization kernels on large constant tensors.

source

TorchLean's scalar normalization core sends a constant normalized slice to the affine bias.

This is the pointwise representative of the GroupNorm/InstanceNorm/BatchNorm constant-slice contract: once the slice statistics are $\mathrm{mean}=x$ and $\mathrm{variance}=0$, the normalized contribution is zero and only beta remains.

source

theorem NN.Examples.BugZoo.ConstantNormalizationSlice.constant_slice_scale_grad_zero (dy x epsilon : ℝ) :

dy * ((x - x) / MathFunctions.sqrt (max (0 + epsilon) 0)) = 0

The scale gradient contribution from a constant normalized slice is zero.