Posts

This article introduces the relationship between KL divergence and cross entropy, pointing out that the expectation of information amount is $-log(p)$, and entropy is the sum of expectations. KL divergence measures the difference between two probability distributions and explains their asymmetry and asymmetry. Furthermore, the article explores the relationship between information entropy and coding, and how to use Shannon's coding theorem to calculate entropy, thereby deriving the concept of average coding length. Finally, the article discusses the application of conditional entropy and maximum likelihood estimation in cross-entropy loss functions, emphasizing another understanding of cross-entropy as a loss function.

This article explores the causes and mitigation methods of vanishing/exploding gradients in neural networks. First of all, the article points out that the gradient descent method will encounter the problem of gradient disappearance or explosion when dealing with complex networks, which is mainly caused by chain derivation and gradient explosion/disappearance. Next, the article lists several common improvement methods: pre-training plus fine-tuning, gradient clipping and regularization, using batch normalization (batchnorm), residual structure, and LSTM, etc. These methods aim to optimize the training process of neural networks, reduce the risk of vanishing or exploding gradients, and improve model performance.

This article introduces common problems and solutions encountered when using PyInstaller to package Python programs. First, the article provides an explanation of the basic commands and parameters of PyInstaller, such as -Fw for generating a single executable file and hiding the console window, and --add-data for adding resource files. Next, the article discusses two main problems: one is the problem that the packaged program is too large due to the use of Anaconda, which is recommended to be solved by creating a virtual environment and installing only necessary libraries; the other is the problem of missing resource files after packaging, and two solutions are proposed: one is to write a hook script to specify the resource folder or file that needs to be included; the other is to use the --add-data option to directly specify the data path and target location to be added in the command line.

At a time when Vibe Coding is so hot that it's almost a topic of national conversation, saying these things is bound to draw plenty of opposition. But this article isn't meant to dismiss Vibe Coding—it is indeed a significant technological advancement. What I really want to discuss is: **why do so many people today have nearly boundless expectations for it?** Perhaps a few years from now, when this wave has passed, future generations digging through the archives and revisiting today's discussions will see things differently.

AI Summary: Why do most neural networks first expand dimensions and then reduce them? I came across a very rigorous theoretical analysis on Zhihu: https://www.zhihu.com/question/1956421703728076399/answer/2035006358764905189. Although I couldn't fully understand it, I'll briefly note it down here.