Adam optimization method memory usage

#Adam optimization method memory usage

---

Generally speaking, Adam uses more information, so it requires more memory. There was an experiment. When the Optimizer selects SGD, the Batch Size can be selected to be 20, but when it is replaced with Adam, the Batch Size can only be selected to be 2.

Definition of moment

Moment comes from the definition of physics, it is the product of distance and physical quantity, and reflects the spatial distribution of objects;

In mathematics, a moment represents a measure of data: $E(|X - c|^k)$ is called the $k$ moment of $X$ with respect to $c$.

In particular, $c = 0$ is called the $k$-order origin moment, and $c = E(X)$ is called the center distance.

More specifically, $E(|X - c_1|^p \cdot |Y - c_2|^q)$ is called the $q + p$ order moment of $X, Y$ about $c_1, c_2$.

If $c_1 = c_2 = 0$, it is called $p + q$ order mixed origin moment

If $c_1 = E(X), c_2 = E(Y)$ , it is called $p + q$ order mixed center distance

Exponential moving average

The exponential moving average method is referred to as the exponential smoothing method. It uses the last actual value and predicted value (estimated value), performs different weighted distributions on them, and obtains an exponential smoothing value as a prediction method for the next period's predicted value. Its prediction formula is:

$$X_t= α X_{t-1} + (1 - α) S_{t}, (0<α<1)\\ X_t is the t-th predicted value, S_{t} is the actual value; X_{t-1} is the last predicted value; α is the weighting coefficient.$$

Usually, the first-order exponential smoothing method is not used directly for prediction, but the second-order exponential smoothing method is used to find the smoothing coefficient, establish a prediction model, and then make predictions. Cubic or more exponential smoothing methods are suitable for almost all time series predictions.

Adam principle (full name: adaptive moment estimation optimizer)

The most critical part is: use exponentially weighted moving average to estimate the momentum and second moment of the gradient, state variables:

$$v_t = \beta_1 v_{t -1} +(1 - \beta_1)g_t\\ s_t = \beta_2 s_{t - 1} + (1 - \beta_2)g_t^2$$

It can be found that the upper term $v_t$ is actually the first-order moment of the gradient, and the lower term $s_t$ is the second-order moment of the gradient. Generally speaking, $\beta_1 = 0.9, \beta_2 = 0.999$, that is to say, the variance estimate moves much slower than the momentum estimate movement.

Bias in corrected moment estimates

$$\hat{v_t} = \frac{v_t}{1 - \beta_1^t}\\ \hat{s_t} = \frac{s_t}{1 - \beta_2^t}$$

Update equation:

$$g_t^` = \frac{ \hat{v_t}}{\sqrt{\hat{s_t}} + \epsilon }\\ x_t = x_{t - 1} - \eta \cdot g_t^`$$

Adam is essentially RMSprop with a momentum term, which uses the first-order moment estimate and second-order moment estimate of the gradient to dynamically adjust

The $g_t^{'}$ here is regarded as normalization. The possible understanding is: $g_t^`$ only cares about the direction, the norm does not matter, anyway there is a learning rate $\eta$

Understand

In fact, each training is done in batches, and a part of the overall sample is selected as the sample for this training. Then using the gradient of this small sample as the gradient of the overall sample is moment estimation (it has never been clear where the moment of adaptive moment estimation is reflected). Then, if you directly use the sample gradient as the overall gradient, you will find some problems, and that is the variance. Obviously, the batch sample extracted is only a part of the overall sample, so the gradient of the batch sample cannot be exactly the gradient of the overall sample. The gradient of the batch sample obeys a random distribution, and its expectation is the overall gradient, but this random distribution has variance. (balabalal, nonsense)

The embodiment of Adam's adaptability is exponential smoothing, which alleviates drastic changes in the gradient direction and makes the convergence path smoother.

Memory usage

Since more information is used, the memory usage will naturally be more, and it is obvious that the upper bound of Batch_Size for batch training will be smaller.

Adam uses parameters, gradients, momentum and second-order momentum, so its memory footprint is four times the data