DIN model

2026/08/284951 words11 min read

"TLDR: This article introduces the DIN model, which uses an attention mechanism to measure the relevance of each item in a user's history to the target item. The article elaborates on the principles of DIN, the computation method of the attention mechanism, strategies for initializing model parameters, and approaches to addressing cold-start and data imbalance issues."

Main Idea:

User interests are diverse, and a user's current interest is only related to a portion of their historical behaviors.

DIN uses an attention mechanism to measure the relevance between each item in the user's history and the target item. Historical behaviors with high relevance receive higher weights, while irrelevant historical behaviors can be ignored.

Principle

Specifically, the model takes three types of input: user features (e.g., age, gender, etc.), user historical behaviors (previously clicked items), and target item features (product information).

Here is an example, assuming there is a user (u) and a target item (i):

User (u) features might be $([age: 25, gender: male, location: Shanghai])$
User (u)'s historical behavior sequence might be $([item_1, item_2, item_3])$
Target item (i) features might be $([category: electronics, brand: Samsung, price: 5000])$
If user (u) clicked item (i), the label is 1; if not clicked, it is 0

First, the system converts all features into dense embedding vectors.

Attention Mechanism

Through the attention mechanism, the system computes the relevance between each historical behavior and the target item. Each element in the user's historical behavior sequence [item1, item2, item3] represents a user behavior, and each behavior is converted into an embedding vector $[e_1, e_2, e_3]$ . Then, the relevance score between each $e_i$ and the target $Item_i$ is calculated. These relevance scores are multiplied by the original historical behavior embeddings and summed to obtain the user interest representation $\boldsymbol{v}_{U}(A)$ .

The calculation formula is as follows: $\boldsymbol{v}_{U}(A)=f\left(\boldsymbol{v}_{A}, \boldsymbol{e}_{1}, \boldsymbol{e}_{2}, \ldots, \boldsymbol{e}_{H}\right)=\sum_{j=1}^{H} a\left(\boldsymbol{e}_{j}, \boldsymbol{v}_{A}\right) \boldsymbol{e}_{j}$ .

Here, $\{ \boldsymbol{e}_{1}, \boldsymbol{e}_{2}, \ldots, \boldsymbol{e}_{H}\}$ are the historical behavior feature embeddings of user U, $\boldsymbol{v}_{A}$ represents the embedding vector of candidate ad A, and $a(e_j,v_A)=w_j$ denotes the degree of relevance between the historical behavior item and the current ad A. $a(\cdot)$ represents a feedforward neural network, i.e., the attention mechanism.

It is worth noting that in addition to the historical behavior vectors and the candidate ad vector, the input also includes their outer product operation, which helps the model perform explicit relevance modeling.

It should be particularly noted that the sum of these weights is not 1. To be precise, standardized weights are not used here; instead, the relevance scores are directly used as weights (i.e., the scores before softmax). This preserves the intensity information of user interest.

Another point to note is that historical behaviors have a sequential nature, and the length of historical behavior features may vary across different users. Since neural networks require sequences of equal length, we typically use padding operations (filling with 0) to make sequences reach the maximum length. During computation, a mask is used to mark the padded positions to ensure computational accuracy.

Dice

Formula:

\begin{align} dice(X) &= p(X) \cdot X + (1 - p(X)) \cdot \alpha X (where, p(X) = \frac{1}{1 + \exp(- \frac{X - E(X)}{\sqrt{Var(X)} + \epsilon})})\\ & = sigmoid(BN(X)) \cdot X + (1 - sigmoid(BN(X))) \cdot \alpha X \end{align}

In essence, it is a further transformation applied on top of Batch Normalization (BN).

Initializing Model Parameters

This is a common problem faced by deep learning models. It is typically alleviated through Xavier initialization or He initialization to mitigate gradient vanishing or explosion issues.

Xavier initialization: Samples the weight matrix from $N(0, \sigma)$ distribution, where $\sigma = \sqrt{2}{n_{in} + n_{out}}$
He initialization: Proposed to alleviate the gradient vanishing problem of ReLU, it samples the weight matrix from $N(0, \sigma)$ distribution, where $\sigma = \sqrt{2}{n_{in}}$ , and $n_{in}$ denotes the number of input nodes of the matrix. Since ReLU tends to produce zero gradients in small-value regions, this issue is mitigated by increasing the standard deviation of the weights.

Cold Start

The solution is to first use content-based recommendation or rule-based recommendation, and then switch to DIN once sufficient data has been accumulated.

Data Imbalance Problem

In the search ad recommendation domain, a user clicking an item usually indicates genuine interest, but not clicking does not necessarily mean disinterest—it could simply be overlooked. In the collected data, positive samples (interested) are often far fewer than negative samples (not interested). Solutions include: oversampling/undersampling, or applying smaller weights to majority-class samples and larger weights to minority-class samples in the loss function.

KV Cache

This article introduces the importance of KV Cache in LLM (Large Language Model). It is a key step in accelerating operations to ensure that the inference speed is not affected when the input text grows during the conversation. The article explains the Self Attention mechanism and the concept of causal mask in detail, and discusses how KV Cache optimizes the prediction process by caching calculation results.

2026/08/28·6 min read

Common PEFT method principles

This article discusses LLM model fine-tuning methods trained on large-scale general data sets, especially how to adapt to specific tasks through methods such as adapters, prefixes, and prompts. In particular, the low-rank adaptive (LoRA) technology proposed by Microsoft is introduced. This method reduces the amount of calculations by decomposing the update matrix into the product of two low-rank matrices, and achieves similar effects to full fine-tuning. The article also discusses the specific principles, applications and challenges of LoRA.