Challenges in Deploying Machine Learning: A Survey of Case Studies (2020-11-18T00:00:00.000000Z)

TL;DR

By mapping found challenges to the steps of the machine learning deployment workflow, it is shown that practitioners face issues at each stage of the deployment process.

Abstract

In recent years, machine learning has transitioned from a field of academic research interest to a field capable of solving real-world business problems. However, the deployment of machine learning models in production systems can present a number of issues and concerns. This survey reviews published reports of deploying machine learning solutions in a variety of use cases, industries, and applications and extracts practical considerations corresponding to stages of the machine learning deployment workflow. By mapping found challenges to the steps of the machine learning deployment workflow, we show that practitioners face issues at each stage of the deployment process. The goal of this article is to lay out a research agenda to explore approaches addressing these challenges.

Authors

Andrei Paleyes

2 papers

Raoul-Gabriel Urma

1 papers

Neil D. Lawrence

4 papers

TL;DR

Abstract

Authors

References178 items

ARTIFICIAL INTELLIGENCE FOR THE REAL WORLD

AutoML: A Perspective where Industry Meets Academy

Towards better data discovery and collection with flow-based programming

Challenges of real-world reinforcement learning: definitions, benchmarks and analysis

On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜

On the Co-evolution of ML Pipelines and Source Code - Empirical Study of DVC Projects

“Brilliant AI Doctor” in Rural Clinics: Challenges in AI-Powered Clinical Decision Support System Deployment

Fairness in Machine Learning

Denoising Multi-Source Weak Supervision for Neural Text Classification

On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice

Artificial intelligence in the creative industries: a review

Gear-Induced Concept Drift in Marine Images and Its Effect on Deep Learning Classification

Monitoring and explainability of models in production

An Empirical Analysis of Backward Compatibility in Machine Learning Systems

Review of Concept Drift Detection Method for Industrial Process Modeling

Technology readiness levels for machine learning systems

Adaptation Strategies for Automated Machine Learning on Evolving Data

Beyond Accuracy: Behavioral Testing of NLP Models with CheckList

Analyzing concept drift: A case study in the financial sector

Data Engineering for Data Analytics: A Classification of the Issues, and Case Studies

The Creation and Detection of Deepfakes

The Cost of Training NLP Models: A Concise Overview

Regulatory responses to medical machine learning

Using Small Business Banking Data for Explainable Credit Risk Scoring

Energy and Policy Considerations for Modern Deep Learning Research

5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

Adversarial Machine Learning-Industry Perspectives

Bridging the Gap Between AI and Healthcare Sides: Towards Developing Clinically Relevant AI-Powered Diagnosis Systems

Perceptions of artificial intelligence in healthcare: findings from a qualitative survey study among actors in France

A Survey of Deep Learning Applications to Autonomous Vehicle Control

Taking Ethics, Fairness, and Bias Seriously in Machine Learning for Disaster Risk Management

How can Automated Machine Learning Help Business Data Science Teams?

Failure Modes in Machine Learning Systems

"The human body is a black box": supporting clinical decision-making with deep learning

When Machine Learning Meets Wireless Cellular Networks: Deployment, Challenges, and Applications

Machine learning based concept drift detection for predictive maintenance

Confident Learning: Estimating Uncertainty in Dataset Labels

A Survey on Active Learning and Human-in-the-Loop Deep Learning for Medical Image Analysis

Explainable machine learning in deployment

Automated deep learning design for medical image classification by health-care professionals with no coding experience: a feasibility study.

Learning a Unified Embedding for Visual Search at Pinterest

150 Successful Machine Learning Models: 6 Lessons Learned at Booking.com

Enabling Onboard Detection of Events of Scientific Interest for the Europa Clipper Spacecraft

Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff

Understanding artificial intelligence ethics and safety

Energy and Policy Considerations for Deep Learning in NLP

Automatic Discovery of Privacy–Utility Pareto Fronts

Assuring the Machine Learning Lifecycle

AIOps: Real-World Challenges and Research Innovations

Software Engineering for Machine Learning: A Case Study

Challenges of Real-World Reinforcement Learning

Continual Learning in Practice

Data Lifecycle Challenges in Production Machine Learning

Snorkel DryBell: A Case Study in Deploying Weak Supervision at Industrial Scale

Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead

Snuba: Automating Weak Supervision to Label Training Data

Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift

Implementation and On-Orbit Testing Results of a Space Communications Cognitive Engine

Applying Deep Learning to Airbnb Search

Model Cards for Model Reporting

Hardware-Aware Machine Learning: Modeling and Optimization

Assessing gender bias in machine translation: a case study with Google Translate

Deploying Machine Learning Models for Public Policy: A Framework

Algorithms that remember: model inversion attacks and data protection law

Auto-Keras: An Efficient Neural Architecture Search System

Crowdsourcing Real-Time Viral Disease and Pest Information: A Case of Nation-Wide Cassava Disease Surveillance in a Developing Country

[Journal First] Data Scientists in Software Teams: State of the Art and Challenges

Data Scientists in Software Teams: State of the Art and Challenges

Adapting to Concept Drift in Credit Card Transaction Data Streams Using Contextual Bandits and Decision Trees

Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learning

SoK: Security and Privacy in Machine Learning

Datasheets for datasets

Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification

Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning

Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy

A Review of Deep Learning Methods and Applications for Unmanned Aerial Vehicles