About the causes and mitigation methods of vanishing/exploding gradients

Cause

As we all know, a neural network is a giant network formed by stacking many different linear/nonlinear layers. Its function expression is very complex and difficult to optimize; however, chain derivation and gradient descent methods can be used to solve this problem. While chain derivation brings solutions, it also brings some problems, that is, gradient disappearance/explosion.

Gradient explosion

Generally speaking, gradient explosion occurs in deep networks, or if the initial value of the weight of the network is set too large, then gradient explosion is prone to occur when performing gradient descent. From a mathematical point of view, the larger the number of layers of the neural network, the more times the chain is found, and the final gradient will change exponentially, making it easy for the value between [0,1] to become close to 0.

Gradient disappears

Also in deep networks, there may be situations where the gradient approaches 0 and is close to disappearing; or it may be due to defects in the loss function itself, such as the common sigmod function, where the gradient is very small away from the original position. Same as above. In mathematics, due to the exponential amplification effect, it is easy for values ​​greater than 1 to approach infinity, causing the gradient to explode.

Summary

The inherent shortcomings of gradient descent will lead to different parameter update speeds in different layers of the neural network.

Current improvement methods

1. Pre-training plus fine-tuning: The capsule proposed by Hinton (which completely abandons backpropagation) trains a layer of hidden nodes each time. During training, the output of the hidden node of the previous layer is used as input, and the output of the hidden node of this layer is used as the input of the hidden node of the next layer. This process is "layer-by-layer pre-training"; after the pre-training is completed, the entire network is fine-tuned (fine-tune).

2. Gradient clipping and regularization; set a threshold for the gradient and force the gradient to be within a reasonable range; weight regularization, the more common ones are L1 regularization and L2 regularization; if gradient explosion occurs, the final optimization goal will be very large, and the penalty term will also be very large, thereby achieving the effect of suppressing the explosion.

3. batchnorm: namely "batch normalization",

4. Use the residual structure; the residual network can easily build hundreds of layers without the problem of the single-line gradient disappearing too fast. The reason lies in the shortcut part.

The first factor $\frac{\partial loss}{\partial x L}$ represents the gradient of the loss function reaching L. The 1 in the parentheses indicates that the short-circuit mechanism can propagate the gradient losslessly, while the other residual gradient needs to go through the layer with weights, and the gradient is not passed directly. The residual gradient will not be all -1 by chance, and even if it is small, the presence of 1 will not cause the gradient to disappear. So residual learning will be easier.

6. LSTM: There is a "gate" structure in LSTM, which can "remember" the "residual memory" left over from previous trainings, that is, modify the network structure

7. Using certain parameter initialization methods, such as initializing to an identity matrix, can help avoid gradient disappearance.