From Motor Control to Team Play in Simulated Humanoid Football (2021-05-25T00:00:00.000000Z)

TL;DR

This work optimized teams of agents to play simulated football via reinforcement learning, constraining the solution space to that of plausible movements learned using human motion capture data, resulting in a team of coordinated humanoid football players that exhibit complex behavior at different scales, quantified by a range of analysis and statistics.

Abstract

Learning to combine control at the level of joint torques with longer-term goal-directed behavior is a long-standing challenge for physically embodied artificial agents. Intelligent behavior in the physical world unfolds across multiple spatial and temporal scales: Although movements are ultimately executed at the level of instantaneous muscle tensions or joint torques, they must be selected to serve goals that are defined on much longer time scales and that often involve complex interactions with the environment and other agents. Recent research has demonstrated the potential of learning-based approaches applied to the respective problems of complex movement, long-term planning, and multiagent coordination. However, their integration traditionally required the design and optimization of independent subsystems and remains challenging. In this work, we tackled the integration of motor control and long-horizon decision-making in the context of simulated humanoid football, which requires agile motor control and multiagent coordination. We optimized teams of agents to play simulated football via reinforcement learning, constraining the solution space to that of plausible movements learned using human motion capture data. They were trained to maximize several environment rewards and to imitate pretrained football-specific skills if doing so led to improved performance. The result is a team of coordinated humanoid football players that exhibit complex behavior at different scales, quantified by a range of analysis and statistics, including those used in real-world sport analytics. Our work constitutes a complete demonstration of learned integrated decision-making at multiple scales in a multiagent setting.

References165 items

Control strategies for physically simulated characters performing two-player competitive sports

AMP

Game Plan: What AI can do for Football, and What Football can do for AI

Game-Theoretic Multiagent Reinforcement Learning

Behavior Priors for Efficient Reinforcement Learning

Learning quadrupedal locomotion over challenging terrain

Deep reinforcement learning for modeling human locomotion control in neuromechanical simulation

Reinforced Grounded Action Transformation for Sim-to-Real Transfer

Data-efficient Hindsight Off-policy Option Learning

dm_control: Software and Tasks for Continuous Control

Real World Games Look Like Spinning Tops

Learning Agile Robotic Locomotion Skills by Imitating Animals

Hierarchical motor control in mammals and machines

Catch & Carry

DReCon

Learning predict-and-simulate policies from unorganized human motion data

Grandmaster level in StarCraft II using multi-agent reinforcement learning

Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning

Solving Rubik's Cube with a Robot Hand

Emergent Tool Use From Multi-Agent Autocurricula

Learning to Sit: Synthesizing Human-Chair Interactions via Hierarchical Control

Stochastic games

Google Research Football: A Novel Reinforcement Learning Environment

Scalable muscle-actuated human simulation and control

Learning to Run Faster in a Humanoid Robot Soccer Environment Through Reinforcement Learning

B-Human 2019 - Complex Team Play Under Natural Lighting Conditions

Compositional Transfer in Hierarchical Reinforcement Learning

MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies

Information asymmetry in KL-regularized RL

Iterative Reinforcement Learning Based Design of Dynamic Locomotion Skills for Cassie

Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Research

Emergent Coordination Through Competition

Open-ended Learning in Symmetric Zero-sum Games

Learning agile and dynamic motor skills for legged robots

Physics-based motion capture imitation with deep reinforcement learning

A survey and critique of multiagent deep reinforcement learning

Near-Optimal Representation Learning for Hierarchical Reinforcement Learning

Neural probabilistic motor primitives for humanoid control

Hierarchical visuomotor control of humanoids

Learning basketball dribbling skills using trajectory optimization and deep reinforcement learning

Human-level performance in 3D multiplayer games with population-based reinforcement learning

Re-evaluating evaluation

Data-Efficient Hierarchical Reinforcement Learning

Latent Space Policies for Hierarchical Reinforcement Learning

DeepMimic

Learning by Playing - Solving Sparse Reward Tasks from Scratch

Diversity is All You Need: Learning Skills without a Reward Function

Maximum a Posteriori Policy Optimisation

Learning an Embedding Space for Transferable Robot Skills

Population Based Training of Neural Networks

Meta Learning Shared Hierarchies

DDCO: Discovery of Deep Continuous Options for Robot Learning from Demonstrations

Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments

Emergent Complexity via Multi-Agent Competition

Proximal Policy Optimization Algorithms

DeepLoco

Distral: Robust multitask reinforcement learning

Learning human behaviors from motion capture by adversarial imitation

Emergence of Locomotion Behaviours in Rich Environments

Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments

Counterfactual Multi-Agent Policy Gradients

Multi-Level Discovery of Deep Options

Emergence of Grounded Compositional Language in Multi-Agent Populations

FeUdal Networks for Hierarchical Reinforcement Learning

Grounded Action Transformation for Robot Learning in Simulation

DeepStack: Expert-level artificial intelligence in heads-up no-limit poker

Variational Intrinsic Control

Learning and Transfer of Modulated Locomotor Controllers

The Option-Critic Architecture

Safe and Efficient Off-Policy Reinforcement Learning

Learning Multiagent Communication with Backpropagation

Learning to Communicate with Deep Multi-Agent Reinforcement Learning

Asynchronous Methods for Deep Reinforcement Learning

Mastering the game of Go with deep neural networks and tree search

Interactive Control of Diverse Complex Characters with Neural Networks

Knowing when not to swing: EEG evidence that enhanced perception–action coupling underlies baseball batter expertise

Deep Reinforcement Learning in Parameterized Action Space

Learning Continuous Control Policies by Stochastic Value Gradients

Continuous control with deep reinforcement learning

Optimization-based locomotion planning, estimation, and control design for the atlas humanoid robot

Online control of simulated humanoids using particle belief propagation

High-Dimensional Continuous Control Using Generalized Advantage Estimation

Anticipation in Soccer: A Systematic Review

Motor skill learning between selection and execution

Human-level control through deep reinforcement learning

Hierarchical Relative Entropy Policy Search

20 years later: deliberate practice and the development of expertise in sport

Probabilistic Movement Primitives

STAC: Simultaneous tracking and calibration

Humanoid robots learning to walk faster: from the real world to simulation and back

MuJoCo: A physics engine for model-based control

Synthesis and stabilization of complex behaviors through online trajectory optimization

Evolution of Cooperation among Mammalian Carnivores and Its Relevance to Hominin Evolution

Developmental activities and the acquisition of superior anticipation and decision making in soccer players

Terrain runner

Multiagent Learning: Basics, Challenges, and Prospects

Independent reinforcement learners in cooperative Markov games: a survey regarding coordination problems

Computational Mechanisms of Sensorimotor Control

On optimizing interdependent skills: a case study in simulated 3D humanoid robot soccer

100

The world of independent learners is not markovian

101

Learning, planning, and control for quadruped locomotion over challenging terrain

102

Cognitive concepts in autonomous soccer playing robots

103

Generalized biped walking control

104

Reinforcement learning for robot soccer

105

Learning Complementary Multiagent Behaviors: A Case Study

106

Learning to dribble on a real robot by success and failure

107

A Comprehensive Survey of Multiagent Reinforcement Learning

108

Sparse but not ‘Grandmother-cell’ coding in the medial temporal lobe

109

Model-Based Reinforcement Learning in a Complex Domain

110

SIMBICON: simple biped locomotion control

111

Autonomous Learning of Stable Quadruped Locomotion

112

Half Field Offense in RoboCup Soccer: A Multiagent Reinforcement Learning Case Study

113

The Chin Pinch: A Case Study in Skill Learning on a Legged Robot

114

Observational Modeling Effects for Movement Dynamics and Movement Outcome Measures Across Differing Task Constraints: A Meta-Analysis

115

Joint action: bodies and minds moving together

116

Cooperative Multi-Agent Learning: The State of the Art

117

Invariant visual representation by single neurons in the human brain

118

Machine Learning for Fast Quadrupedal Locomotion

119

Policy gradient reinforcement learning for fast quadrupedal locomotion

120

Reinforcement Learning in Large State Spaces

121

Composable controllers for physics-based character animation

122

The Prefrontal Cortex—An Update Time Is of the Essence

123

The Complexity of Decentralized Control of Markov Decision Processes

124

An Algorithm for Distributed Reinforcement Learning in Cooperative Multi-Agent Systems

125

Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning

126

Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition

127

Team-partitioned, opaque-transition reinforcement learning

128

The Dynamics of Reinforcement Learning in Cooperative Multiagent Systems

129

Reinforcement Learning with Hierarchies of Machines

130

Long Short-Term Memory

131

HQ-Learning

132

Layered Learning in Multiagent Systems

133

RoboCup: The Robot World Cup Initiative

134

Some Studies in Machine Learning Using the Game of Checkers

135

Temporal Difference Learning and TD-Gammon

136

Evolving virtual creatures

137

The role of deliberate practice in the acquisition of expert performance.

138

A reference model architecture for intelligent systems design

139

Feudal Reinforcement Learning

140

Combining Specialized Reasoners and General Purpose Planners: A Case Study

141

Unified Theories of Cognition

142

Algorithmic motion planning in robotics

143

Legged Robots That Balance

144

Arms races between and within species

145

Scripts, plans, goals and understanding: an inquiry into human knowledge structures

146

CoMic: Complementary Task Learning & Mimicry for Reusable Skills

147

UT Austin Villa: RoboCup 2019 3D Simulation League Competition and Technical Challenge Champions

148

Overlapping layered learning

149

Beyond expected goals

150

DeepStack: Expert-Level Artificial Intelligence in No-Limit Poker

151

Nando de Freitas, and Nicolas Heess. Robust imitation of diverse behaviors

152

Deep Learning

153

Learning Powerful Kicks on the Aibo ERS-7: The Quest for a Striker

154

On Progress in RoboCup: The Simulation League Showcase

155

Reinforcement learning in the robocup-soccer keepaway

156

Recent Advances in Hierarchical Reinforcement Learning

157

Discrete event dynamic systems: Theory and applications.

158

Karlsruhe Brainstormers - A Reinforcement Learning Approach to Robotic Soccer

159

Layered learning in multiagent systems - a winning approach to robotic soccer

160

Neural sequence chunkers

161

Hierarchical organization of motor programs.

162

The rating of chessplayers, past and present

163

The problem of serial order in behavior

164

Tezak, Jerry 34 From Motor Control to Team Play in Simulated Humanoid

165