Wide&Deep model and DCN model

Wide & Deep Principle

There are many blogs on the Internet explaining that the Wide part provides memory capabilities and learns simple feature patterns, while the Deep part provides pan-Chinese capabilities and learns complex feature patterns. It seems a bit confusing, but it’s very clear if you look directly at the Wide & Deep code:

class WideAndDeepModel(nn.Module):

    def __init__(self, categorical_dims, numerical_dim, embedding_dim, hidden_layers):

        super(WideAndDeepModel, self).__init__()

        #Wide section

        self.wide = nn.Linear(len(categorical_dims) + numerical_dim, 1)

        

        # Deep part

        self.embeddings = nn.ModuleList([

            nn.Embedding(dim, embedding_dim) for dim in categorical_dims

        ])

        deep_input_dim = len(categorical_dims) * embedding_dim + numerical_dim

        layers = []

        for dim in hidden_layers:

            layers.append(nn.Linear(deep_input_dim, dim))

            layers.append(nn.ReLU())

            deep_input_dim = dim

        layers.append(nn.Linear(deep_input_dim, 1))

        self.deep = nn.Sequential(*layers)



    def forward(self, categorical_data, numerical_data):

        #Wide section

        wide_input = torch.cat([categorical_data.float(), numerical_data], dim=1)

        wide_output = self.wide(wide_input)

        

        # Deep part

        embeddings = [emb(categorical_data[:, i]) for i, emb in enumerate(self.embeddings)]

        deep_input = torch.cat(embeddings + [numerical_data], dim=1)

        deep_output = self.deep(deep_input)

        

        # Combine Wide and Deep output

        output = torch.sigmoid(wide_output + deep_output)

        return output

• The Wide part is a short answer Linear linear layer, a very simple and direct linear transformation;

• The Deep part is a slightly more complex MLP, with a Relu activation function between linear layers, just a deeper neural network;

• The output of both Wide and Deep parts are thrown into the sigmoid function for normalization;

It is easy to understand in this way. Shallow neural networks learn shallow features, and deep neural networks learn complex features.

In fact, it is better to understand in CV. CNN is used to learn the classification of cats and dogs. The shallow network learns simple features such as color and gray intensity, and the deep network learns complex features such as contours.

Wide & Deep FAQ

Deep & Cross model

The Deep & Cross model is the most common and simple evolved version of the Wide & Deep model, which is to change the Wide part into a Cross network.

In fact, there will be many improvements in the future, basically replacing the weak Wide part with more diverse cross-operations, such as Cross, FM and other modules. (Traditional recommendation models and deep learning models join forces at this moment)

Let’s look at DCN specifically:

• Deep part: Needless to say, basically unchanged, it is still a DNN (deep learning neural network, responsible for learning complex non-linear combinations of features)

• Wide part: Cross Network (cross network, responsible for explicitly modeling the intersection between features and capturing the interaction between high-order features).

The respective outputs of these two networks are then fused, usually through a simple weighted sum or simple concatenation, and the final output is passed through a linear layer and activation function to obtain the prediction result (exactly the same as the original Wide & Deep)

Cross Network

For the input vector $x$, the $l$ layer of the crossover network can be expressed as: $x_l = x_0 (W_l x_{l - 1})^T + b_l + x_{l - 1}$, so that the intersection between different features can be learned

The reason why each second-order crossover part is operated with $x_0$ is to retain more of the original display features and avoid learning all deep implicit features after multi-layer crossover (this is what the Deep part is responsible for)

I'm too lazy to look at the formula, so just look at the Wide (Cross) part of the model structure diagram.

Code implementation:

class DCN(nn.Module):

    def __init__(self, categorical_dims, numerical_dim, embedding_dim, hidden_layers, cross_layers):

        super(DCN, self).__init__()

        # Embedding layer, used to map category features to low-dimensional dense space

        self.embeddings = nn.ModuleList([

            nn.Embedding(dim, embedding_dim) for dim in categorical_dims

        ])

        

        # Cross Network section

        input_dim = len(categorical_dims) * embedding_dim + numerical_dim

        self.cross_layers = nn.ModuleList([

            nn.Linear(input_dim, input_dim) for _ in range(cross_layers)

        ])

        

        # Deep Network section

        deep_layers = []

        for dim in hidden_layers:

            deep_layers.append(nn.Linear(input_dim, dim))

            deep_layers.append(nn.ReLU())

            input_dim = dim

        self.deep = nn.Sequential(*deep_layers)

        

        # Output layer

        self.output = nn.Linear(hidden_layers[-1] + len(categorical_dims) * embedding_dim + numerical_dim, 1)



    def forward(self, categorical_data, numerical_data):

        # Embedding layer: Encode categorical features

        embeddings = [emb(categorical_data[:, i]) for i, emb in enumerate(self.embeddings)]

        embed_concat = torch.cat(embeddings, dim=1)

        

        # Concatenate embeddings and numerical features as input

        x = torch.cat([embed_concat, numerical_data], dim=1) # [batch, Category_num * Dim + Numeric_num]

        

        # Cross Network section

        x_cross = x.clone() # [batch, Category_num * Dim + Numeric_num]

        for layer in self.cross_layers:

            x_cross = x + layer(x_cross) # Explicit cross between layers

        

        # Deep Network section

        x_deep = self.deep(x)

        

        # Combine Cross and Deep output

        combined = torch.cat([x_cross, x_deep], dim=1)

        output = torch.sigmoid(self.output(combined))

        return output