DORY: Automatic End-to-End Deployment of Real-World DNNs on Low-Cost IoT MCUs (2020-08-17T00:00:00.000000Z)

TL;DR

This work proposes DORY (Deployment Oriented to memoRY) – an automatic tool to deploy DNNs on low cost MCUs with typically less than 1MB of on-chip SRAM memory and releases all the developments – the DORY framework, the optimized backend kernels, and the related heuristics – as open-source software.

Abstract

The deployment of Deep Neural Networks (DNNs) on end-nodes at the extreme edge of the Internet-of-Things is a critical enabler to support pervasive Deep Learning-enhanced applications. Low-Cost MCU-based end-nodes have limited on-chip memory and often replace caches with scratchpads, to reduce area overheads and increase energy efficiency – requiring explicit DMA-based memory transfers between different levels of the memory hierarchy. Mapping modern DNNs on these systems requires aggressive topology-dependent tiling and double-buffering. In this work, we propose DORY (Deployment Oriented to memoRY) – an automatic tool to deploy DNNs on low cost MCUs with typically less than 1MB of on-chip SRAM memory. DORY abstracts tiling as a Constraint Programming (CP) problem: it maximizes L1 memory utilization under the topological constraints imposed by each DNN layer. Then, it generates ANSI C code to orchestrate off- and on-chip transfers and computation phases. Furthermore, to maximize speed, DORY augments the CP formulation with heuristics promoting performance-effective tile sizes. As a case study for DORY, we target GreenWaves Technologies GAP8, one of the most advanced parallel ultra-low power MCU-class devices on the market. On this device, DORY achieves up to 2.5× better MAC/cycle than the GreenWaves proprietary software solution and 18.1× better than the state-of-the-art result on an STM32-H743 MCU on single layers. Using our tool, GAP-8 can perform end-to-end inference of a 1.0-MobileNet-128 network consuming just 63 pJ/MAC on average @ 4.3 fps – 15.4× better than an STM32-H743. We release all our developments – the DORY framework, the optimized backend kernels, and the related heuristics – as open-source software.

References62 items

TensorFlow Lite Micro: Embedded Machine Learning on TinyML Systems

MCUNet: Tiny Deep Learning on IoT Devices

MAESTRO: A Data-Centric Approach to Understand Reuse, Performance, and Hardware Cost of DNN Mappings

Technical Report: NEMO DNN Quantization for Deployment Model

CMix-NN: Mixed Low-Precision CNN Library for Memory-Constrained Edge Devices

XpulpNN: Accelerating Quantized Neural Networks on RISC-V Processors Through ISA Extensions

Larq: An Open-Source Library for Training Binarized Neural Networks

Robust Real-Time Embedded EMG Recognition Framework Using Temporal Convolutional Networks on a Multicore IoT Processor

FANN-on-MCU: An Open-Source Toolkit for Energy-Efficient Neural Network Inference at the Edge of the Internet of Things

dMazeRunner

PULP-NN: accelerating quantized neural networks on parallel ultra-low-power RISC-V processors

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

Mr.Wolf: An Energy-Precision Scalable Parallel Ultra Low Power SoC for IoT Edge Processing

Timeloop: A Systematic Approach to DNN Accelerator Evaluation

Interstellar: Using Halide's Scheduling Language to Analyze DNN Accelerators

Work-in-Progress: Quantized NNs as the Definitive Solution for Inference on Low-Power ARM MCUs?

GAP-8: A RISC-V SoC for AI at the Edge of the IoT

An Overview of Internet of Things (IoT) and Data Analytics in Agriculture: Benefits and Challenges

IFQ-Net: Integrated Fixed-Point Quantization Networks for Embedded Vision

A 64-mW DNN-Based Visual Navigation Engine for Autonomous Nano-Drones

PACT: Parameterized Clipping Activation for Quantized Neural Networks

CMSIS-NN: Efficient Neural Network Kernels for Arm Cortex-M CPUs

MobileNetV2: Inverted Residuals and Linear Bottlenecks

μDMA: An autonomous I/O subsystem for IoT end-nodes

Machine learning for Internet of Things data analysis: A survey

Optimal Tiling Strategy for Memory Bandwidth Reduction for CNNs

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

UAV-Based IoT Platform: A Crowd Surveillance Use Case

14.1 A 2.9TOPS/W deep convolutional neural network SoC in FD-SOI 28nm for intelligent embedded systems

An IoT Endpoint System-on-Chip for Secure and Energy-Efficient Near-Sensor Analytics

A near-threshold RISC-V core with DSP extensions for scalable IoT Endpoint Devices

Eyeriss: A Spatial Architecture for Energy-Efficient Dataflow for Convolutional Neural Networks

PULP: A Ultra-Low Power Parallel Accelerator for Energy-Efficient and Flexible Embedded Vision

Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding

ADRENALINE: An OpenVX Environment to Optimize Embedded Vision Applications on Many-core Accelerators

A ultra-low-energy convolution engine for fast brain-inspired vision in multicore clusters

Ultra-low-latency lightweight DMA for tightly coupled multi-core clusters

Memory-centric accelerator design for Convolutional Neural Networks

Optimizing two-dimensional DMA transfers for scratchpad Based MPSoCs platforms

A fully-synthesizable single-cycle interconnection network for Shared-L1 processor clusters

Design, Automation & Test in Europe Conference & Exhibition, DATE 2021, Grenoble, France, February 1-5, 2021

The Memory Challenge in Ultra-Low Power Deep Learning

Data Movement Is All You Need: A Case Study of Transformer Networks

Cypress DRAM

GAP8 Auto-tiler

Paper Award

Hardware for machine learning: Challenges and opportunities

X-CUBE-AI

Arm helium

Tensor

Available: https

the lifetime counter of L 2 y,i is set to the lifetime of the activation buffer

ST Microelectronics

NXP. Nxp lpc4300

stage 3. L3-tile w = OFF, L3-tile y = ON

the allocator deallocates the last L 2 w,i − 2 buffer on the corner

L 2 y,i , L 2 w,i are allocated in order in the selected corner (with L 2 w,i nearest to the pointer)

L3-tile w = ON

one of the two corners of the stack is selected depending on a begin_end ﬂag that is switched at each new weight allocation

His research interests cover hardware and software optimization for low power and high efﬁciency embedded systems, parallel programming for multicore architectures and virtual prototyping

the allocator checks if L 2 y,i − 2 has its lifetime counter set to 0; if so, it

received the PhD from the University of Bologna, Italy