DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems (2020-08-19T00:00:00.000000Z)

TL;DR

This work proposes an improved framework DCN-V2, which is simple, can be easily adopted as building blocks, and has delivered significant offline accuracy and online business metrics gains across many web-scale learning to rank systems at Google.

Abstract

Learning effective feature crosses is the key behind building recommender systems. However, the sparse and large feature space requires exhaustive search to identify effective crosses. Deep & Cross Network (DCN) was proposed to automatically and efficiently learn bounded-degree predictive feature interactions. Unfortunately, in models that serve web-scale traffic with billions of training examples, DCN showed limited expressiveness in its cross network at learning more predictive feature interactions. Despite significant research progress made, many deep learning models in production still rely on traditional feed-forward neural networks to learn feature crosses inefficiently. In light of the pros/cons of DCN and existing feature interaction learning approaches, we propose an improved framework DCN-V2 to make DCN more practical in large-scale industrial settings. In a comprehensive experimental study with extensive hyper-parameter search and model tuning, we observed that DCN-V2 approaches outperform all the state-of-the-art algorithms on popular benchmark datasets. The improved DCN-V2 is more expressive yet remains cost efficient at feature interaction learning, especially when coupled with a mixture of low-rank architecture. DCN-V2 is simple, can be easily adopted as building blocks, and has delivered significant offline accuracy and online business metrics gains across many web-scale learning to rank systems at Google. Our code and tutorial are open-sourced as part of TensorFlow Recommenders (TFRS)1.

Authors

Lichan Hong

3 papers

Ruoxi Wang

2 papers

Ed H. Chi

6 papers

TL;DR

Abstract

Authors

References61 items

Recommender Systems

Neural Collaborative Filtering vs. Matrix Factorization Revisited

A Metric Learning Reality Check

Interpretable Click-Through Rate Prediction through Hierarchical Attention

Adaptive Factorization Network: Learning Adaptive-Order Feature Interactions

SNR: Sub-Network Routing for Flexible Parameter Sharing in Multi-Task Learning

Are we really making much progress? A worrying analysis of recent neural recommendation approaches

Deep Learning Recommendation Model for Personalization and Recommendation Systems

AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks

Adaptive Mixture of Low-Rank Factorizations for Compact Neural Modeling

A multiscale neural network based on hierarchical nested bases

Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts

xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems

Latent Cross: Making Use of Context in Recurrent Recommender Systems

Learning Sparse Neural Networks through L0 Regularization

Deep & Cross Network for Ad Click Predictions

Neural Factorization Machines for Sparse Predictive Analytics

Attentional Factorization Machines: Learning the Weight of Feature Interactions via Attention Networks

On Compressing Deep Models by Low Rank and Sparse Decomposition

Attention is All you Need

DeepFM: A Factorization-Machine based Neural Network for CTR Prediction

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

Categorical Reparameterization with Gumbel-Softmax

Product-Based Neural Networks for User Response Prediction

Deep Crossing: Web-Scale Modeling without Manually Crafted Combinatorial Features

Wide & Deep Learning for Recommender Systems

Residual Networks Behave Like Ensembles of Relatively Shallow Networks

Block Basis Factorization for Scalable Kernel Evaluation

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

Adam: A Method for Stochastic Optimization

Recommender Systems: Kembellec/Recommender Systems

Practical Lessons from Predicting Clicks on Ads at Facebook

Learning Polynomials with Neural Networks

Speeding up Convolutional Neural Networks with Low Rank Expansions

Deep learning in neural networks: An overview

Learning Factored Representations in a Deep Mixture of Experts

Counterfactual reasoning and learning systems: the example of computational advertising

Factorization Machines with libFM

Feature engineering in Context-Dependent Deep Neural Networks for conversational speech transcription

Extensions of recurrent neural network language model

Factorization Machines

Computational advertising and recommender systems

Learning to rank: from pairwise approach to listwise approach

On the Nyström Method for Approximating a Gram Matrix for Improved Kernel-Based Learning

Recommender systems in e-commerce

Long Short-Term Memory

Adaptive Mixtures of Local Experts

Backpropagation Applied to Handwritten Zip Code Recognition

Learning internal representations by error propagation

FINDING STRUCTURE WITH RANDOMNESS : PROBABILISTIC ALGORITHMS FOR CONSTRUCTING

Matrix Computations

Evaluating collaborative filtering recommender systems

Face recognition: a convolutional neural-network approach

Neural Networks for Optimal Approximation of Smooth and Analytic Functions

WayneHubbard,andLawrenceDJackel

Matrix computations

RQ2 How does the feature-interaction component of each baseline perform without integrating with DNN?

each order, we randomly selected 20 cross terms from x ∈ R 50 . Fig. 4 shows

RQ3 How does the proposed mDCN approaches compare to the baselines? Could we achieve healthier trade-off between model accuracy and cost through mDCN and the mixture of low-rank DCN?

RQ1 When would feature cross learning methods become more efficient than ReLU-based DNNs?

RQ3 How does the settings in DCN-V2 affect model quality?

Field of Study

Journal Information

Name

Venue Information

Name

Type

URL

Alternate Names