Classification problems often use CE instead of MSE

Machine learning is usually divided into two categories: regression and classification. The former commonly uses mean square error (MSE), and the latter commonly uses cross entropy (CE). The predecessors must have their reasons for doing this. Let’s analyze it from several angles.

Distribution angle

MSE as a loss function actually has an a priori assumption: the error obeys the standard Gaussian distribution. Here is a simple derivation of the origin of the MSE loss function:

It is nothing more than three steps: assuming distribution, maximum likelihood estimation, and deriving the loss function.

Assume that the error $\epsilon$ between the predicted value and the actual value obeys the standard Gaussian distribution, that is, $\epsilon \sim N(0, 1)$ , and because $\epsilon = f(X) - Y$ , so $Y \sim N(f(X), 1)$

The probability density function of $Y$ is $p(Y) = \frac{1}{\sqrt{2 \pi }} \exp [- \frac{(f(X) - Y)^2}{2} ]$.

Then use the maximum likelihood estimation method: $\log L = \sum_{i = 1} \log p(Y_i) = -\frac{n}{2} \log(2\pi) - \frac{1}{2} \sum_{i = 1} [Y_i - f(X_i)]^2$

At this time, the MSE loss function was derived

A little review here

1. If it is a Lasso loss function, it is equivalent to the maximum posterior estimate of the parameters obeying the Laplace distribution;

2. If it is the loss function of ridge regression, it is equivalent to the maximum posterior estimate of the parameters obeying Gaussian distribution;

And CE assumes that the data obeys Bernoulli distribution/polynomial distribution

The Bernoulli distribution means doing N trials. The probability of success for each trial is $p$ and the probability of failure is $1- p$. Then the expectation of the number of successes is $N \cdot p$, the variance is $N \cdot p \cdot (1-p)$, and the probability density function is $f(x) = p^x (1 - p)^{1 -x}$

Then in the two classification task, given the data set $D$, then for each sample $D_i$, the probability density function is $p^x(1 - p)^{1 -x}$. We still use the likelihood estimation, which means we can get $\log L = \sum_{i = 1} \log[p^{x_i} ( 1- p)^{1 - x_i}] = \sum_{i = 1} x_i \log p + (1 - x_i) \log(1 - p)$ .

At this time, we will derive the situation except the CE loss function in the second classification.

Therefore, from a priori point of view, in classification tasks, it is more realistic to assume that the data obeys the Bernoulli distribution. Therefore, CE is often used in classification tasks, while MSE is often used in regression tasks.

Let’s discuss it again. mse assumes that the data distribution is Gaussian distribution. However, in reality, long-tail distribution is also very common, and the effect is not good if MSE is used directly. Therefore, sometimes when doing data EDA, it is found that the data does not conform to the Gaussian distribution and needs to be corrected, such as using logarithmic transformation or boxcox transformation.

Optimization angle

From an optimization perspective, if the classification problem uses MSE as the loss function, it is a non-convex optimization and it is difficult to find the optimal solution.

$$L = \frac{1}{N} \sum_{i = 1}[f(x_i) - y_i]^2\\ = \frac{1}{N} \sum{i = 1}[y_i - \sigma(z)] \\ = \frac{1}{N} \sum{i = 1}[y_i - \frac{1}{1 + \exp(-wx)}]$$

This is a non-convex function.