MIOpen: An Open Source Library For Deep Learning Primitives (2019-09-30T00:00:00.000000Z)

TL;DR

This paper introduces MIOpen and provides details about the internal workings of the library and supported features, as well as implementing fusion to optimize for memory bandwidth and GPU launch overheads, and implementing different algorithms to optimize convolutions for different filter and input sizes.

Abstract

Deep Learning has established itself to be a common occurrence in the business lexicon. The unprecedented success of deep learning in recent years can be attributed to: an abundance of data, availability of gargantuan compute capabilities offered by GPUs, and adoption of open-source philosophy by the researchers and industry. Deep neural networks can be decomposed into a series of different operators. MIOpen, AMD's open-source deep learning primitives library for GPUs, provides highly optimized implementations of such operators, shielding researchers from internal implementation details and hence, accelerating the time to discovery. This paper introduces MIOpen and provides details about the internal workings of the library and supported features. MIOpen innovates on several fronts, such as implementing fusion to optimize for memory bandwidth and GPU launch overheads, providing an auto-tuning infrastructure to overcome the large design space of problem configurations, and implementing different algorithms to optimize convolutions for different filter and input sizes. MIOpen is one of the first libraries to publicly support the bfloat16 data-type for convolutions, allowing efficient training at lower precision without the loss of accuracy.

Authors

Chao Liu

2 papers

Jehandad Khan

1 papers

Paul Fultz

1 papers

TL;DR

Abstract

Authors

References55 items

PyTorch: An Imperative Style, High-Performance Deep Learning Library

Reducing Uncertainty in Undersampled MRI Reconstruction With Active Acquisition

Glow: Graph Lowering Compiler Techniques for Neural Networks

Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions

TVM: An Automated End-to-End Optimizing Compiler for Deep Learning

Intel nGraph: An Intermediate Representation, Compiler, and Executor for Deep Learning

The tensor algebra compiler

ImageNet Training in Minutes

Simple Recurrent Units for Highly Parallelizable Recurrence

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

Xception: Deep Learning with Depthwise Separable Convolutions

Deep Roots: Improving CNN Efficiency with Hierarchical Filter Groups

Theano: A Python framework for fast computation of mathematical expressions

Simit

A guide to convolution arithmetic for deep learning

Deep Residual Learning for Image Recognition

MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems

Fast Algorithms for Convolutional Neural Networks

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Fully convolutional networks for semantic segmentation

cuDNN: Efficient Primitives for Deep Learning

Going deeper with convolutions

Caffe: Convolutional Architecture for Fast Feature Embedding

Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation

Halide: a language and compiler for optimizing parallelism, locality, and recomputation in image processing pipelines

Speech recognition with deep recurrent neural networks

ImageNet classification with deep convolutional neural networks

ACCESS: Smart scheduling for asymmetric cache CMPs

Automating the generation of composed linear algebra kernels

The OpenCL specification

A Novel Connectionist System for Unconstrained Handwriting Recognition

Improving predictive inference under covariate shift by weighting the log-likelihood function

OoLALA: an object oriented analysis and design of numerical linear algebra

Recurrent nets that time and count

Automatically Tuned Linear Algebra Software

Convolutional networks for images, speech, and time series

FFTW: an adaptive software architecture for the FFT

Long Short-Term Memory

Learning representations by back-propagating errors

Neural networks and physical systems with emergent collective computational abilities.

Basic Linear Algebra Subprograms for Fortran Usage

MLIR Primer: A Compiler Infrastructure for the End of Moore’s Law

TVM: An Automated End-to-End Optimizing Compiler for Deep Learning

PyTorch: Tensors and dynamic neural networks in python with strong GPU acceleration

Long short-term memory recurrent neural network architectures for large scale acoustic modeling

Rigid-motion scattering for image classification

Implementing Neural Networks Efficiently

This Paper Is Included in the Proceedings of the 12th Usenix Symposium on Operating Systems Design and Implementation (osdi '16). Tensorflow: a System for Large-scale Machine Learning Tensorflow: a System for Large-scale Machine Learning

“AMD GCN ISA,”

“MIOpen: Documentation,”

“AMD HIP,”

“TensorFlow RNN cell,”

ROCm - Open Source Platform for HPC and Ultrascale GPU Computing

MLIR: Multi-level Intermediate Representation for Compiler Infrastructure

Clang: a C language family frontend for LLVM

Field of Study

Journal Information

Name

Volume