NGBoost: Natural Gradient Boosting for Probabilistic Prediction (2019-10-08T00:00:00.000000Z)

TL;DR

NGBoost generalizes gradient boosting to probabilistic regression by treating the parameters of the conditional distribution as targets for a multiparameter boosting algorithm, and shows how the Natural Gradient is required to correct the training dynamics of the authors' multiparameters boosting approach.

Abstract

We present Natural Gradient Boosting (NGBoost), an algorithm for generic probabilistic prediction via gradient boosting. Typical regression models return a point estimate, conditional on covariates, but probabilistic regression models output a full probability distribution over the outcome space, conditional on the covariates. This allows for predictive uncertainty estimation -- crucial in applications like healthcare and weather forecasting. NGBoost generalizes gradient boosting to probabilistic regression by treating the parameters of the conditional distribution as targets for a multiparameter boosting algorithm. Furthermore, we show how the Natural Gradient is required to correct the training dynamics of our multiparameter boosting approach. NGBoost can be used with any base learner, any family of distributions with continuous parameters, and any scoring rule. NGBoost matches or exceeds the performance of existing methods for probabilistic prediction while offering additional benefits in flexibility, scalability, and usability. An open-source implementation is available at this http URL.

Authors

Tony Duan

3 papers

D. Ding

3 papers

A. Ng

22 papers

TL;DR

Abstract

Authors

References43 items

Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift

Estimation Methods for Nonhomogeneous Regression Models: Minimum Continuous Ranked Probability Score versus Maximum Likelihood

Accurate Uncertainties for Deep Learning Using Calibrated Regression

Countdown Regression: Sharp and Calibrated Survival Predictions

Distributional regression forests for probabilistic precipitation forecasting in complex terrain

Accelerating Natural Gradient with Higher-Order Invariance

LightGBM: A Highly Efficient Gradient Boosting Decision Tree

Improving palliative care with deep learning

On Calibration of Modern Neural Networks

Concrete Dropout

What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?

Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles

XGBoost: A Scalable Tree Boosting System

Weight Uncertainty in Neural Network

Mondrian Forests for Large-Scale Regression when Uncertainty Matters

Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

Weight Uncertainty in Neural Networks

Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks

New Insights and Perspectives on the Natural Gradient Method

Identifying and attacking the saddle point problem in high-dimensional non-convex optimization

Theory and applications of proper scoring rules

Contrasting probabilistic scoring rules

Practical Variational Inference for Neural Networks

Scikit-learn: Machine Learning in Python

Variational Learning of Inducing Variables in Sparse Gaussian Processes

BART: Bayesian Additive Regression Trees

Generalized Additive Models for Location Scale and Shape (GAMLSS) in R

Strictly Proper Scoring Rules, Prediction, and Estimation

The geometry of proper scoring rules

Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning)

Gaussian Processes in Machine Learning

Greedy function approximation: A gradient boosting machine.

Natural Gradient Works Efficiently in Learning

Learning Generative Samplers using Relaxed Injective Flow

Machine learning meets economics

Probabilistic Forecasting

NGBoost: Natural Gradient Boosting for Probabilistic Prediction Weiss

BMC Bioinformatics BioMed Central Methodology article Flexible boosting of accelerated failure time models

Bayesian learning for neural networks

Natural Gradient Works Eciently in Learning

LightGBM: A Highly Eﬀicient Gradient Boosting Decision Tree

We present a generalization of the Natural Gradient to other scoring rules such as CRPS (sec 2.2)

Anand Avati*, Daisy Yi Ding Ar-tiﬁcial Conference on Machine Learning , ICML’15

Field of Study

Venue Information

Name

Type

URL

Alternate Names