Win-Fail Action Recognition

Abstract

Current video/action understanding systems have demonstrated impressive performance on large recognition tasks. However, they might be limiting themselves to learning to recognize spatiotemporal patterns, rather than attempting to thoroughly understand the actions. To spur progress in the direction of a more comprehensive understanding of videos, we introduce the task of win-fail action recognition — differentiating between successful and failed attempts at various activities. We introduce a first of its kind paired win-fail action understanding dataset with samples from the following domains: “General Stunts,” “Internet Wins-Fails,” “Trick Shots,” & “Party Games.” Unlike existing action recognition datasets, intra-class variation is high making the task challenging, yet feasible. Using a battery of experiments, including a novel video retrieval test, we systematically analyze the characteristics of our win-fail task/dataset, and determine its suitability to serve as a video understanding problem benchmark. While current prototypical action recognition methods work well on our task/dataset, they still leave a large gap to achieve high performance. We hope to motivate more work towards the true understanding of actions/videos. Dataset will be available from: https://github.com/ParitoshParmar/Win-Fail-Action-Recognition.