Collaborative filtering model

Collaborative filtering is the most primitive recommendation algorithm, and even today when deep learning dominates the field, it still holds significant learning value.

User CF

Similar users tend to like the same items.

Advantages: UserCF recommends based on user similarity, which is highly social in nature and well-suited for news scenarios.

Disadvantages: In internet scenarios, the number of users far exceeds the number of items.

1. The storage cost of the user similarity matrix is very high; as the number of users grows, the space complexity of the user similarity matrix increases at a rate of $O(N^2)$.
2. User historical data is extremely sparse. For users with only a few purchase behaviors, the accuracy of finding similar users is very low, making UserCF unsuitable for scenarios where positive feedback is difficult to obtain (low-frequency scenarios such as hotel booking and large-ticket item purchases).

Item CF

Obtain an $m \times n$ matrix of interactions between $m$ users and $n$ items, calculate the similarity between different items column-wise to derive an $n \times n$ similarity matrix. For the items in a user's positive feedback, find the Top k similar items to generate a recommendation list.

Matrix Factorization

A serious drawback of CF: popular items have a strong head effect, causing them to be similar to a large number of items, while tail items, due to sparse feature vectors, rarely become similar to other items and are therefore seldom recommended.

Ultimately, the sparsity of item vectors leads to deficiencies in recommendations. Matrix factorization uses denser latent vectors to uncover the implicit interests and latent features of users and items.

The smaller the value of $k$, the less information the latent vectors contain, and the higher the model's generalization;

The larger the value of $k$, the stronger the expressive power of the latent vectors, but the generalization correspondingly decreases.

In addition, the value of $k$ is also directly related to the computational complexity of solving the matrix factorization.

In specific applications, the value of $k$ should be determined through multiple experiments to find a balance point between recommendation effectiveness and engineering cost.

Matrix Factorization Process

• Eigenvalue decomposition
• Singular value decomposition (SVD)
• Gradient descent

Eigenvalue Decomposition

Only suitable for decomposing square matrices. The User-Item matrix is not square, so this is ruled out.

Singular Value Decomposition

$M = U \Sigma V^T$, where $U_{m \times m}$ and $V_{n \times n}$ are orthogonal matrices, and $\Sigma_{m \times n}$ is a diagonal matrix.

1. Singular value decomposition requires the original co-occurrence matrix to be dense, but in internet scenarios, most users have very few behavioral records, making the User-Item matrix sparse.
2. The time complexity of singular value decomposition is $O(M \times N^2)$, which is unacceptable.

Gradient Descent Method

We use $q_i$ to denote the feature vector of user $i$, and $p_u$ to denote the feature vector of $Item_u$. Then the relevance $r_{u, i}$ between user $i$ and $Item_u$ is $r_{u, i} = q_i^T p_u$.

The training loss is then:

$$\begin{align} SSE = \sum_{u, i} (r_{u, i} - q_i^T p_u)^2 + \lambda (||q_i|| + ||p_u||)^2 \end{align}$$

Here, $r_{u, i}$ comes from the true rating labels of users for items in the co-occurrence matrix, and the $\lambda$ part is the regularization term.

Advantages:

• Strong generalization capability: it alleviates the data sparsity problem to some extent.
• Better scalability and flexibility: the final output of matrix factorization is user and item latent vectors, which aligns closely with the Embedding concept in deep learning (not exactly the same, but essentially no difference).

Disadvantages:

• It still cannot incorporate user features (Age, gender), item features (price, rating, category), or contextual features, losing a considerable amount of useful information.