methodology-5

Mutual Information Estimation

3260 papers • 126 benchmarks • 313 datasets

To estimate mutual information from samples, specially for high-dimensional variables.

(Image credit: Papersgraph)

Benchmarks

These leaderboards are used to track progress in mutual-information-estimation-5

Trend

Dataset

Best Model

Actions

No benchmarks available.

Libraries

i

Use these libraries to find mutual-information-estimation-5 models and implementations

Datasets

No datasets available.

Subtasks

No subtasks available.

Most implemented papers

Learning deep representations by mutual information estimation and maximization

Yoshua Bengio, R. Devon Hjelm, A. Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Adam Trischler•Sun Aug 19 2018

It is shown that structure matters: incorporating knowledge about locality in the input into the objective can significantly improve a representation’s suitability for downstream tasks and is an important step towards flexible formulations of representation learning objectives for specific end-goals.

2872

Content

0

Paper Graph

Graph Representation Learning via Aggregation Enhancement

Chaim Baskin, Evgenii Zheltonozhskii, Maxim Fishman, Almog David, Ron Banner, A. Mendelson•Sat Jan 29 2022

This work highlights the potential of KR to advance the field of graph representation learning and enhance the performance of GNNs, using KR loss as the primary loss in self-supervised settings or as a regularization term in supervised settings.

0 0

Paper Graph

Estimating Mutual Information for Discrete-Continuous Mixtures

Sewoong Oh, P. Viswanath, Weihao Gao, Sreeram Kannan•Mon Sep 18 2017

This paper designs a novel estimator for mutual information of discrete-continuous mixtures and proves that the proposed estimator is consistent, and provides numerical experiments suggesting superiority of this estimator compared to other heuristics.

170 0

Paper Graph

Scalable Mutual Information Estimation Using Dependence Graphs

M. Noshad, A. Hero•Fri Jan 26 2018

EDGE is the first non-parametric MI estimator that can achieve paramet- ric MSE rates with linear time complexity and the utility of EDGE is illustrated for the analysis of the information plane (IP) in deep learning.

80 0

Paper Graph

Practical and Consistent Estimation of f-Divergences

I. Tolstikhin, C. Riquelme, J. Djolonga, O. Bousquet, Paul K. Rubenstein•Tue Apr 30 2019

This work proposes and study an estimator that can be easily implemented, works well in high dimensions, and enjoys faster rates of convergence, and discusses its direct implications for total correlation, entropy, and mutual information estimation.

48 0

Paper Graph

Deep Learning for Channel Coding via Neural Mutual Information Estimation

Rick Fritschek, Rafael F. Schaefer, G. Wunder•Wed Mar 06 2019

This work uses a recently proposed neural estimator of mutual information to optimize the encoder for a maximized mutual information, only relying on channel samples, and shows that this approach achieves the same performance as state-of-the-art end-to-end learning with perfect channel model knowledge.

48 0

Paper Graph

Empowerment-driven Exploration using Mutual Information Estimation

Navneet Kumar•Wed Oct 10 2018

It is demonstrated that an empowerment driven agent is able to improve significantly the score of a baseline DQN agent on the game of Montezuma's Revenge.

10 0

Paper Graph

Better Long-Range Dependency By Bootstrapping A Mutual Information Regularizer

Peng Xu, Yanshuai Cao•Tue Apr 30 2019

A novel regularizer is developed to improve the learning of long-range dependency of sequence data and shows how the `next sentence prediction (classification)' heuristic can be derived in a principled way from the authors' mutual information estimation framework, and be further extended to maximize the mutual information of sequence variables.

2 0

Paper Graph

Neural Entropic Estimation: A faster path to mutual information estimation

Chung Chan, Ali Al-Bashabsheh, Hingpang Huang, Michael Lim, D. Tam, Chao Zhao•Wed May 29 2019

It is argued that the seemingly redundant intermediate step of entropy estimation allows one to improve the convergence by an appropriate reference distribution and faster convergence is possible by choosing the uniform distribution as the reference distribution instead.

25 0

Paper Graph

CCMI : Classifier based Conditional Mutual Information Estimation

Sreeram Kannan, Sudipto Mukherjee, Himanshu Asnani•Tue Jun 04 2019

This paper introduces an estimator for KL-Divergence based on the likelihood ratio by training a classifier to distinguish the observed joint distribution from the product distribution and shows how to construct several CMI estimators using this basic divergence estimator by drawing ideas from conditional generative models.

87 0

Paper Graph

Adding a benchmark result helps the community track progress.