A Survey on Large-Scale Machine Learning (2020-08-10T00:00:00.000000Z)

TL;DR

A systematic survey on existing LML methods is offered to provide a blueprint for the future developments of this area and categorize the methods in each perspective according to their targeted scenarios and introduce representative methods in line with intrinsic strategies.

Abstract

Machine learning can provide deep insights into data, allowing machines to make high-quality predictions and having been widely used in real-world applications, such as text mining, visual classification, and recommender systems. However, most sophisticated machine learning approaches suffer from huge time costs when operating on large-scale data. This issue calls for the need of Large-scale Machine Learning (LML), which aims to learn patterns from big data with comparable performance efficiently. In this paper, we offer a systematic survey on existing LML methods to provide a blueprint for the future developments of this area. We first divide these LML methods according to the ways of improving the scalability: 1) model simplification on computational complexities, 2) optimization approximation on computational efficiency, and 3) computation parallelism on computational capabilities. Then we categorize the methods in each perspective according to their targeted scenarios and introduce representative methods in line with intrinsic strategies. Lastly, we analyze their limitations and discuss potential directions as well as open issues that are promising to address in the future.

Authors

Meng Wang

1 papers

Weijie Fu

1 papers

Xiangnan He

12 papers

TL;DR

Abstract

Authors

References246 items

Big Data Analytics: A Survey

Shark

AdderNet: Do We Really Need Multiplications in Deep Learning?

PyTorch: An Imperative Style, High-Performance Deep Learning Library

PS2: Parameter Server on Spark

A Survey of Optimization Methods From a Machine Learning Perspective

DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression

SLIDE : In Defense of Smart Algorithms over Hardware Acceleration for Large-Scale Deep Learning Systems

Towards Federated Learning at Scale: System Design

Optimal mini-batch and step sizes for SAGA

PowerLyra: Differentiated Graph Computation and Partitioning on Skewed Graphs

Deformable ConvNets V2: More Deformable, Better Results

Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead

Fast Matrix Factorization With Nonuniform Weights on Missing Data

GradiVeQ: Vector Quantization for Bandwidth-Efficient Gradient Aggregation in Distributed CNN Training

Gandiva: Introspective Cluster Scheduling for Deep Learning

Large Scale GAN Training for High Fidelity Natural Image Synthesis

AuTO: scaling deep reinforcement learning for datacenter-scale automatic traffic optimization

Scalable Active Learning by Approximated Error Reduction

Improved Large-Scale Graph Learning through Ridge Spectral Sparsification

Hierarchical Graph Representation Learning with Differentiable Pooling

Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimization

Neural Ordinary Differential Equations

SketchML: Accelerating Distributed Machine Learning with Data Sketches

DimBoost: Boosting Gradient Boosting Decision Tree to Higher Dimensions

cpSGD: Communication-efficient and differentially-private distributed SGD

Adversarially Robust Generalization Requires More Data

Quantum Machine Learning

Angel: a new large-scale machine learning system

Snap ML: A Hierarchical Framework for Machine Learning

Horovod: fast and easy distributed deep learning in TensorFlow

On the Convergence of Adam and Beyond

To understand deep learning we need to understand kernel learning

Deep Learning with Python

Ray: A Distributed Framework for Emerging AI Applications

Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training

LightGBM: A Highly Efficient Gradient Boosting Decision Tree

Don't Decay the Learning Rate, Increase the Batch Size

Practical Secure Aggregation for Privacy-Preserving Machine Learning

A Survey of Model Compression and Acceleration for Deep Neural Networks

FLAG: Faster Learning on Anchor Graph with Label Predictor Optimization

ZipML: Training Linear Models with End-to-End Low Precision, and a Little Bit of Deep Learning

Theoretical Insights Into the Optimization Landscape of Over-Parameterized Shallow Neural Networks

ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices

A Survey on Deep Learning in Big Data

Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour

TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning

Learning on Big Graph: Label Inference and Regularization with Anchor Hierarchy

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

Interpretable Learning for Self-Driving Cars by Visualizing Causal Attention

Deformable Convolutional Networks

Diving into the shallows: a computational perspective on large-scale shallow learning

Bounded activation functions for enhanced training stability of deep neural networks on visual pattern recognition problems

A Generic Coordinate Descent Framework for Learning from Implicit Feedback

QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding

A Primer on Coordinate Descent Algorithms

Asynchronous Stochastic Gradient Descent with Delay Compensation

An overview of gradient descent optimization algorithms

Data mining with big data

Efficient softmax approximation for GPUs

SystemML: Declarative Machine Learning on Spark

Parallel Dual Coordinate Descent Method for Large-scale Linear Classification in Multi-core Environments

Towards Comprehensive Traffic Forecasting in Cloud Computing: Design and Application

High-Level Programming Abstractions for Distributed Graph Processing

Discrete Collaborative Filtering

Scalable Semi-Supervised Learning by Efficient Anchor Graph Regularization

Adaptive Sampling for SGD by Exploiting Side Information

Optimization Methods for Large-Scale Machine Learning

A Survey on Learning to Hash

Computing Web-scale Topic Models using an Asynchronous Parameter Server

XGBoost: A Scalable Tree Boosting System

Big Data-Survey

vDNN: Virtualized deep neural networks for scalable, memory-efficient neural network design

Incorporating Nesterov Momentum into Adam

Bridging the Gap between Stochastic Gradient MCMC and Stochastic Optimization

Accelerated Proximal Gradient Methods for Nonconvex Programming