Technology readiness levels for machine learning systems (2020-06-21T00:00:00.000000Z)

TL;DR

The authors introduce a framework that defines a principled process of machine learning system formation, from research to production, for various domains and data scenarios, and elucidate with use-cases from physics research to computer vision apps to medical diagnostics.

Abstract

The development and deployment of machine learning systems can be executed easily with modern tools, but the process is typically rushed and means-to-an-end. Lack of diligence can lead to technical debt, scope creep and misaligned objectives, model misuse and failures, and expensive consequences. Engineering systems, on the other hand, follow well-defined processes and testing standards to streamline development for high-quality, reliable results. The extreme is spacecraft systems, with mission critical measures and robustness throughout the process. Drawing on experience in both spacecraft engineering and machine learning (research through product across domain areas), we’ve developed a proven systems engineering approach for machine learning and artificial intelligence: the Machine Learning Technology Readiness Levels framework defines a principled process to ensure robust, reliable, and responsible systems while being streamlined for machine learning workflows, including key distinctions from traditional software engineering, and a lingua franca for people across teams and organizations to work collaboratively on machine learning and artificial intelligence technologies. Here we describe the framework and elucidate with use-cases from physics research to computer vision apps to medical diagnostics. The development of machine learning systems has to ensure their robustness and reliability. The authors introduce a framework that defines a principled process of machine learning system formation, from research to production, for various domains and data scenarios.

References92 items

Unity Perception: Generate Synthetic Data for Computer Vision

Challenges in Deploying Machine Learning: A Survey of Case Studies

Towards Compliant Data Management Systems for Healthcare ML

Split-Treatment Analysis to Rank Heterogeneous Causal Effects for Prospective Interventions

Learnings from Frontier Development Lab and SpaceML - AI Accelerators for NASA and ESA

Underspecification Presents Challenges for Credibility in Modern Machine Learning

Bayesian Workflow.

Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure

Regulatory Frameworks for Development and Evaluation of Artificial Intelligence–Based Diagnostic Imaging Algorithms: Summary and Recommendations

Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension

Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans

Improving the accuracy of medical diagnosis with causal machine learning

Data governance: Organizing data for trustworthy Artificial Intelligence

Technology Readiness Levels for AI & ML

Manifolds for Unsupervised Visual Anomaly Detection

Beyond Accuracy: Behavioral Testing of NLP Models with CheckList

Collider bias undermines our understanding of COVID-19 disease risk and severity

Understanding and Visualizing Data Iteration in Machine Learning

Sense and Sensitivity Analysis: Simple Post-Hoc Analysis of Bias Due to Unobserved Confounding

Bias in data‐driven artificial intelligence systems—An introductory survey

Bias in Data-driven AI Systems - An Introductory Survey

Closing the AI accountability gap: defining an end-to-end framework for internal algorithmic auditing

Diagnosing bias in data-driven algorithms for healthcare

Lessons from archives: strategies for collecting sociocultural data in machine learning

Failure Modes in Machine Learning Systems

The frontier of simulation-based inference

Dissecting racial bias in an algorithm used to manage the health of populations

An Annotation Saved is an Annotation Earned: Using Fully Synthetic Training for Object Detection

Explainable machine learning in deployment

A Survey on Bias and Fairness in Machine Learning

Federated Learning: Challenges, Methods, and Future Directions

Etalumis: Bringing Probabilistic Programming to Scientific Simulators at Scale

Understanding artificial intelligence ethics and safety

Software Engineering for Machine Learning: A Case Study

An Annotation Saved is an Annotation Earned: Using Fully Synthetic Training for Object Instance Detection

Artificial intelligence, bias and clinical safety

A generic framework for privacy preserving deep learning

Model Cards for Model Reporting

Towards trustable machine learning

An Introduction to Probabilistic Programming

Unity: A General Platform for Intelligent Agents

Comparative Study of OKR and KPI

A survey of southern hemisphere meteor showers

Datasheets for datasets

Theoretical Impediments to Machine Learning With Seven Sparks from the Causal Revolution

Harnessing the Power of Real‐World Evidence (RWE): A Checklist to Ensure Regulatory‐Grade Data Quality

The ML test score: A rubric for ML production readiness and technical debt reduction

What is a case study?

Generating Natural Adversarial Examples

Agile Software Development Methods: Review and Analysis

Deep Reinforcement Learning that Matters

Concrete Problems for Autonomous Vehicle Safety: Advantages of Bayesian Deep Learning

Hybrid software and system development in practice: waterfall, scrum, and beyond

Towards Deep Learning Models Resistant to Adversarial Attacks

Bias and High-Dimensional Adjustment in Observational Studies of Peer Effects

Double/Debiased Machine Learning for Treatment and Structural Parameters

Double-adjustment in propensity score matching analysis: choosing a threshold for considering residual imbalance

Reliable Decision Support using Counterfactual Models

Domain randomization for transferring deep neural networks from simulation to the real world

Deep Bayesian Active Learning with Image Data

Hidden Technical Debt in Machine Learning Systems

The Impact of Debriefing on Future Performance of Projects

Recommendations on the use and design of risk matrices

Probabilistic machine learning and artificial intelligence

Power to the People: The Role of Humans in Interactive Machine Learning

Managing Innovation in Architecturally Hierarchical Systems: Three Switchback Mechanisms That Impact Practice

Build, Compute, Critique, Repeat: Data Analysis with Latent Variable Models

CAMS: Cameras for Allsky Meteor Surveillance to establish minor meteor showers

Testing and validating machine learning classifiers by metamorphic testing

Event generation with SHERPA 1.1

NASA Systems Engineering Handbook

Exploratory Data Mining and Data Cleaning

Agile Software Development

Active Learning with Statistical Models

Verifying and Validating Software Requirements and Design Specifications

Summary and Recommendations

Data Validation for Machine Learning

The dawn of the deep tech ecosystem

A guide for the responsible design and implementation of AI systems in the public sector

A new typology design of performance metrics to measure errors in machine learning regression algorithms. Interdiscip

Searching for Long-Period Comets with Deep Learning Tools

Taking the Human Out of the Loop: A Review of Bayesian Optimization

What’s your ML test score? A rubric for ML production systems

a reviewof bayesian optimization

google,我,萨娜

Defense acquisition guidebook

CRISP-DM 1.0: Step-by-step data mining guide

NASA systems engineering handbook

The use of misclassification costs to learn rule-based decision support models for cost-effective hospital admission strategies.

Probabilistic Logic and the Synthesis of Reliable Organisms from Unreliable Components

Gall , Ece Kamar , Nachiappan Nagappan , Besmira Nushi , and Thomas Zimmermann