The difference between BatchNorm and Dropout in training and testing

2026/08/282003 words5 min read

"TLDR: This article explores the differences between BatchNorm and Dropout during the training and testing phases, and explains their respective roles. BatchNorm calculates the global mean and variance through sliding average, which is used to speed up numerical stability and accelerate the training process. At the same time, it retains these parameters for use during the testing phase. Dropout randomly freezes neuron output during the training process to reduce overfitting, and scales the output results by 1-r times during testing to ensure the stability of the activation function input. The article also points out that the essence of Dropout is the idea of Bagging, which reduces the variance of the model by randomly selecting different neurons."

#The difference between BatchNorm and Dropout during training and testing

BatchNorm

Each small batch training will be standardized, and the distribution of values will be adjusted to have a mean of 0 and a variance of 1 to ensure the stability of the values and speed up training.

At the same time, Batch Norm will maintain a global mean and variance, which are calculated through moving average.

$movingMean = movingMean * (moment) + (1 - moment) * batchNrom$

$movingStd = movingStd * (moment) + (1 - moment) * batchStd$

These two global parameters will be reserved for testing.

Dropout

During the training process, the output of the neuron will be frozen according to a certain probability $r$ . In actual operation, all the output of the neuron is set to 0, which can effectively avoid overfitting.

During testing, all neurons will participate in the calculation. In order to ensure the stability of the input value of the activation function, the output result needs to be scaled by $1 - r$ .

Proof: The activation function depends on the input signal strength. During training, the expectation of signal strength is $E_1(x) = \sum_i (1-r) Y(i)$ , which means the probability that the $i$ th neuron can output is $1- r$ , and the output strength is $Y(i)$ . During testing, all neurons can output, and the expectation of signal strength is $E_2(x) = \sum_i Y(i)$ , in order to ensure equal signal strength, so $(1 - r) E_2 = E_1$ , so the value needs to be scaled $1 - r$ times during testing.

The essence of Dropout is the idea of Bagging.

Randomly selecting different neurons is equivalent to training different models. This diversity can reduce the variance of the model.

The so-called variance, large variance means it is too complex, too sensitive to the data set, too dependent on certain features, and performs poorly on the test set

KV Cache

This article introduces the importance of KV Cache in LLM (Large Language Model). It is a key step in accelerating operations to ensure that the inference speed is not affected when the input text grows during the conversation. The article explains the Self Attention mechanism and the concept of causal mask in detail, and discusses how KV Cache optimizes the prediction process by caching calculation results.

2026/08/28·6 min read

Common PEFT method principles

This article discusses LLM model fine-tuning methods trained on large-scale general data sets, especially how to adapt to specific tasks through methods such as adapters, prefixes, and prompts. In particular, the low-rank adaptive (LoRA) technology proposed by Microsoft is introduced. This method reduces the amount of calculations by decomposing the update matrix into the product of two low-rank matrices, and achieves similar effects to full fine-tuning. The article also discusses the specific principles, applications and challenges of LoRA.

BatchNorm

Dropout

Related

KV Cache

Common PEFT method principles