Compute Trends Across Three Eras of Machine Learning

Published in

IEEE International Joint Conference on Neural N...(2022)

External Links:

Generate Graph DownloadPDF

TL;DR

This paper curates a dataset with the training compute of 123 milestone ML systems, 3× larger than previous such datasets, and frames the trends in compute in in three eras - the Pre Deep Learning Era, the Deep learning Era, and the Large-Scale Era.

Abstract

Compute, data, and algorithmic advances are the three fundamental factors that drive progress in modern Machine Learning (ML). In this paper we study trends in the most readily quantified factor - compute. We make three novel contributions: (1) we curate a dataset with the training compute of 123 milestone ML systems, 3× larger than previous such datasets. (2) We frame the trends in compute in in three eras - the Pre Deep Learning Era, the Deep Learning Era, and the Large-Scale Era, based on our identification of a novel trend emerging around 2015. (3) We find a Deep Learning Era compute doubling time of around 6 months, significantly longer than previous findings. Overall, our work highlights the fast-growing compute requirements for training advanced ML systems.

Authors

J. Sevilla

1 papers

Lennart Heim

1 papers

A. Ho

1 papers

References181 items

Flamingo: a Visual Language Model for Few-Shot Learning

PaLM: Scaling Language Modeling with Pathways

Training Compute-Optimal Large Language Models

Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model

LaMDA: Language Models for Dialog Applications

Compute Trends Across Three Eras of Machine Learning

Published in

IEEE International Joint Conference on Neural N...(2022)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

Authors

J. Sevilla

1 papers

Lennart Heim

1 papers

A. Ho

1 papers

References181 items

Flamingo: a Visual Language Model for Few-Shot Learning

PaLM: Scaling Language Modeling with Pathways

Training Compute-Optimal Large Language Models

Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model

LaMDA: Language Models for Dialog Applications

T. Besiroglu

1 papers

Marius Hobbhahn

1 papers

Pablo Villalobos

1 papers

Generative Adversarial Networks

Varuna: scalable, low-cost training of massive deep learning models

Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learning

M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining

Primer: Searching for Efficient Transformers for Language Modeling

Compute and Energy Consumption Trends in Deep Learning Inference

The semiconductor shortage and its implication for euro area trade, production and prices

HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units

Scaling Vision Transformers

CogView: Mastering Text-to-Image Generation via Transformers

PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation

Software-hardware co-design for fast and scalable training of deep learning recommendation models

Scaling Scaling Laws with Board Games

Self-supervised Pretraining of Visual Features in the Wild

Learning Transferable Visual Models From Natural Language Supervision

Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

AraGPT2: Pre-Trained Transformer for Arabic Language Generation

CPM: A Large-scale Generative Chinese Pre-trained Language Model

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

The Computational Limits of Deep Learning

GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

Denoising Diffusion Probabilistic Models

Language Models are Few-Shot Learners

The Cost of Training NLP Models: A Concise Overview

Meta Pseudo Labels

ProGen: Language Modeling for Protein Generation

Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers

Scaling Laws for Neural Language Models

Improved protein structure prediction using potentials from deep learning

Dota 2 with Large Scale Deep Reinforcement Learning

KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation

Grandmaster level in StarCraft II using multi-agent reinforcement learning

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Solving Rubik's Cube with a Robot Hand

A Constructive Prediction of the Generalization Error Across Scales

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

Once for All: Train One Network and Specialize it for Efficient Deployment

Deep Learning Recommendation Model for Personalization and Recommendation Systems

AlphaX: eXploring Neural Architectures with Deep Neural Networks and Monte Carlo Tree Search

Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing

The Hanabi Challenge: A New Frontier for AI Research

GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism

ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware

Large Scale GAN Training for High Fidelity Natural Image Synthesis

MnasNet: Platform-Aware Neural Architecture Search for Mobile

YOLOv3: An Incremental Improvement

The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approaches

Regularized Evolution for Image Classifier Architecture Search

IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures

Applied Machine Learning at Facebook: A Datacenter Infrastructure Perspective

Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm

Progressive Neural Architecture Search

Deep Learning Scaling is Predictable, Empirically

Decoupled Weight Decay Regularization

Mastering the game of Go without human knowledge

Libratus: The Superhuman AI for No-Limit Poker

Revisiting Unreasonable Effectiveness of Data in Deep Learning Era

Attention is All you Need

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

Neural Architecture Search with Reinforcement Learning

Xception: Deep Learning with Depthwise Separable Convolutions

Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation

Layer Normalization

R-FCN: Object Detection via Region-based Fully Convolutional Networks

Mastering the game of Go with deep neural networks and tree search

Deep Residual Learning for Image Recognition

Deep Speech 2 : End-to-End Speech Recognition in English and Mandarin

Continuous control with deep reinforcement learning

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

Adam: A Method for Stochastic Optimization

Going deeper with convolutions

Sequence to Sequence Learning with Neural Networks

Very Deep Convolutional Networks for Large-Scale Image Recognition

Neural Machine Translation by Jointly Learning to Align and Translate

Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition

Auto-Encoding Variational Bayes

Playing Atari with Deep Reinforcement Learning

Translating Embeddings for Modeling Multi-relational Data

Visualizing and Understanding Convolutional Networks

Distributed Representations of Words and Phrases and their Compositionality

Mitosis Detection in Breast Cancer Histology Images with Deep Neural Networks

ImageNet classification with deep convolutional neural networks

Multi-column deep neural network for traffic sign classification

Improving neural networks by preventing co-adaptation of feature detectors

Multi-column deep neural networks for image classification

Understanding the difficulty of training deep feedforward neural networks

Deep, Big, Simple Neural Nets for Handwritten Digit Recognition

Large-scale deep unsupervised learning using graphics processors

Fast support vector machine training and classification on graphics processors

Animal research: Mighty mouse

High Performance Convolutional Neural Networks for Document Processing

100

Using GPUs for machine learning algorithms

101

2005 Special Issue: Framewise phoneme classification with bidirectional LSTM and other neural network architectures

102

A Neural Probabilistic Language Model

103

An RNN-based prosodic information synthesizer for Mandarin text-to-speech

104

Long Short-Term Memory

105

Neural network-based face detection

106

Some Studies in Machine Learning Using the Game of Checkers

107

Practical issues in temporal difference learning

108

Backpropagation Applied to Handwritten Zip Code Recognition

109

Learning representations by back-propagating errors

110

Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position

111

The perceptron: a probabilistic model for information storage and organization in the brain.

112

FLOP/s)? We expect most of the compute estimates to be accurate within a factor of about two based on some comparisons we did between different estimation methods

113

Language modelling at scale. Retrieved 2022-02-23, from https:// deepmind.com/blog/article/language-modelling-at-scale

114

2019, February). Better Language Models and Their Implications

115

Report: Microsoft Handed OpenAI $500m in Azure Credits.

116

Human-level performance in first-person multiplayer games with population-based deep reinforcement learning

117

How much longer can computing power drive artificial intelligence progress? Technical report, Center for Security and Technology

118

Announcing GPT-NeoX-20B

120

Computer Progress

121

Estimating training compute of Deep Learning models. Retrieved from https://docs.google.com/document/d/ 1J2BX9jkE5nN5EA1zYRN0lHhdCf1YkiFERc_nwiYqCOA

122

AI and Compute Trend Isn’t Predictive of What Is Happening.

123

An Analysis on the Crisis of “Chips shortage” in Automobile Industry ——Based on the Double Influence of COVID-19 and Trade Friction

124

Aggregating Nested Transformers

125

DALL-E: CREATING IMAGES FROM TEXT

126

Review of Parameter Counts in Machine Learning. Alignment Forum (blog). https://www.alignmentforum.org/posts/GzoWcYibWYwJva8aL/ parameter-counts-in-machine-learning

127

GPT-J-6B: 6B JAX-Based Transformer

128

Ford to Shut Some N. American Plants for Few Weeks on Chip Shortage.

129

Why the Global Chip Shortage is Making It So Hard to Buy a PS5

130

emphParameter Counts in Machine Learning

131

OpenAI API Pricing

132

Review of \emphParameter Counts in Machine Learning. (Published: Alignment Forum (blog)

133

Google Trained a Trillion-Parameter AI Language Model.

134

Taiwan’s drought is exposing just how much water chipmakers like TSMC use (and reuse)

135

The Chip Shortage Seems to Be Impacting AI Workloads in the Cloud

136

models has created challenges unique to the Large-Scale Era

137

2019 with an estimated cost of $12M (K

139

DeepMind achieved StarCraft II GrandMaster Level, but at what cost?

140

OpenAI’s GPT-3 Language Model: A Technical Overview

141

GPU Shortages Hit Nvidia’s Data Center Business: Not Enough $15,000+ GPUs.

142

How much did AlphaGo Zero cost. https://www.yuzeh.com/data/agz-cost.html

143

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

144

ObjectNet: A large-scale bias-controlled dataset for pushing the limits of object recognition models

145

Designing and implementing algorithms that do this efficiently can be

146

2019, February). The Hanabi Challenge: A New Frontier for AI Research. Retrieved 2022-02-23, from https://arxiv.org/abs/ 1902.00506v2 doi: 10.1016/j.artint.2019.103216

147

Emergent Tool Use from Multi-Agent Interaction

148

2019, November). Review of AI and Compute Addendum: Compute Used in Older Headline Results. (Published: OpenAI Blog

149

Review of AI and Compute

150

Improving Language Understanding with Unsupervised Learning

151

DeepStack: Expert-Level Artificial Intelligence in No-Limit Poker

153

Adaptive Switching Circuits

154

ALVINN, an autonomous land vehicle in a neural network

155

We ﬁnd that our conclusions from the post are still supported by the record-setting models

156

Learning about speech

157

FUNDAMENTAL TECHNOLOGIES IN MODERN SPEECH RECOGNITION Digital Object Identifier 10.1109/MSP.2012.2205597

158

2012)’s deep CNNs (which again predates AlexNet) also beat

159

Trends in different domains Different domains rely on different architectures and we can naively expect them to be subject to different scaling laws

160

Other types of ML models, such as support-vector machines

161

2010b, 2011) made substantial improvements over the previous state-of-the-art results on

162

Recurrent neural network based language model

163

surpassed all competition at the time on the Wall Street Journal task, an NLP

164

deep architectures with efficient learning algorithms

165

Deep Learning for Speech Recognition and Related Applications

166

PROC OF THE IEEE NOVEMBER Gradient Based Learning Applied to Document Recognition

167

A bitter lesson.

168

Gradient-based learning applied to document recognition

169

Parallel Networks that Learn to Pronounce English Text

171

Review of The Future of Integrated Electronics. Electronics Magazine

172

Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence Flexible, High Performance Convolutional Neural Networks for Image Classification

174

Retrieved 2022-02-23

175

Trends in record setting models

178

Pandemonium: A Paradigm for Learning | AITopics

179

Announcing AI21 Studio and Jurassic-1 Language Models

180

Towards a Conversational Agent that Can Chat About

181

Hardware-Und Nachrichten-Links Des 30./31. Oktober 2021

Field of Study

Computer Science

Journal Information

Name

2022 International Joint Conference on Neural Networks (IJCNN)

Page

1-8

Venue Information

Name

IEEE International Joint Conference on Neural Network

Type

conference

URL

http://www.wikicfp.com/cfp/program?id=1573

Alternate Names

IJCNN
IEEE Int Jt Conf Neural Netw
Int Jt Conf Neural Netw
International Joint Conference on Neural Network