Code QuizIntermediate

Weight Scale and Exploding Values

Find why this initialization causes exploding activations and gradients.

Codepython
import torch.nn as nn

layer = nn.Linear(256, 256)
nn.init.normal_(layer.weight, mean=0.0, std=10.0)

What is the bug that leads to exploding gradients?