AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks (2018-10-29T00:00:00.000000Z)

TL;DR

An effective and efficient method called the AutoInt to automatically learn the high-order feature interactions of input features and map both the numerical and categorical features into the same low-dimensional space is proposed.

Abstract

Click-through rate (CTR) prediction, which aims to predict the probability of a user clicking on an ad or an item, is critical to many online applications such as online advertising and recommender systems. The problem is very challenging since (1) the input features (e.g., the user id, user age, item id, item category) are usually sparse and high-dimensional, and (2) an effective prediction relies on high-order combinatorial features (a.k.a. cross features), which are very time-consuming to hand-craft by domain experts and are impossible to be enumerated. Therefore, there have been efforts in finding low-dimensional representations of the sparse and high-dimensional raw features and their meaningful combinations. In this paper, we propose an effective and efficient method called the AutoInt to automatically learn the high-order feature interactions of input features. Our proposed algorithm is very general, which can be applied to both numerical and categorical input features. Specifically, we map both the numerical and categorical features into the same low-dimensional space. Afterwards, a multi-head self-attentive neural network with residual connections is proposed to explicitly model the feature interactions in the low-dimensional space. With different layers of the multi-head self-attentive neural networks, different orders of feature combinations of input features can be modeled. The whole model can be efficiently fit on large-scale raw data in an end-to-end fashion. Experimental results on four real-world datasets show that our proposed approach not only outperforms existing state-of-the-art approaches for prediction but also offers good explainability. Code is available at: \urlhttps://github.com/DeepGraphLearning/RecommenderSystems.

References45 items

Session-Based Social Recommendation via Dynamic Graph Attention Networks

NAIS: Neural Attentive Item Similarity Model for Recommendation

TEM: Tree-enhanced Embedding Model for Explainable Recommendation

xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems

Latent Cross: Making Use of Context in Recurrent Recommender Systems

Graph Attention Networks

Deep & Cross Network for Ad Click Predictions

Neural Factorization Machines for Sparse Predictive Analytics

Attentional Factorization Machines: Learning the Weight of Feature Interactions via Attention Networks

Deep Interest Network for Click-Through Rate Prediction

Attention is All you Need

Exponential Machines

GB-CENT: Gradient Boosted Categorical Embedding and Numerical Trees

Deep Embedding Forest: Forest-based Serving with Deep Embedding Features

DeepFM: A Factorization-Machine based Neural Network for CTR Prediction

A Structured Self-attentive Sentence Embedding

Product-Based Neural Networks for User Response Prediction

Field-aware Factorization Machines for CTR Prediction

Deep Neural Networks for YouTube Recommendations

Deep Crossing: Web-Scale Modeling without Manually Crafted Combinatorial Features

Higher-Order Factorization Machines

Wide & Deep Learning for Recommender Systems

Polynomial Networks and Factorization Machines: New Insights and Efficient Training Algorithms

Key-Value Memory Networks for Directly Reading Documents

Predicting ad click-through rates via feature-based fully coupled interaction tensor factorization

Deep Learning over Multi-field Categorical Data - - A Case Study on User Response Prediction

Deep Residual Learning for Image Recognition

A Neural Attention Model for Abstractive Sentence Summarization

End-To-End Memory Networks

Adam: A Method for Stochastic Optimization

Gradient boosting factorization machines

Neural Machine Translation by Jointly Learning to Align and Translate

Practical Lessons from Predicting Clicks on Ads at Facebook

Predicting response in mobile advertising with hierarchical importance-aware factorization machine

Ad click prediction: a view from the trenches

Representation Learning: A Review and New Perspectives

Unsupervised learning of hierarchical representations with convolutional deep belief networks

Fast context-aware recommendations with factorization machines

Factorization Machines

Web-Scale Bayesian Click-Through rate Prediction for Sponsored Search Advertising in Microsoft's Bing Search Engine

Factorizing personalized Markov chains for next-basket recommendation

Predicting clicks: estimating the click-through rate for new ads

Dropout: a simple way to prevent neural networks from overfitting

This Paper Is Included in the Proceedings of the 12th Usenix Symposium on Operating Systems Design and Implementation (osdi '16). Tensorflow: a System for Large-scale Machine Learning Tensorflow: a System for Large-scale Machine Learning

RQ3 What are the dependency structures between different features? Is our proposed model explainable?