Iterative Contrast-Classify For Semi-supervised Temporal Action Segmentation (2021-12-02T00:00:00.000000Z)

TL;DR

This work proposes a novel way to learn frame-wise representations from temporal convolutional networks (TCNs) by clustering input features with added time-proximity conditions and multi-resolution similarity by merging representation learning with conventional supervised learning.

Abstract

Temporal action segmentation classifies the action of each frame in (long) video sequences. Due to the high cost of frame-wise labeling, we propose the first semi-supervised method for temporal action segmentation. Our method hinges on unsupervised representation learning, which, for temporal action segmentation, poses unique challenges. Actions in untrimmed videos vary in length and have unknown labels and start/end times. Ordering of actions across videos may also vary. We propose a novel way to learn frame-wise representations from temporal convolutional networks (TCNs) by clustering input features with added time-proximity conditions and multi-resolution similarity. By merging representation learning with conventional supervised learning, we develop an "Iterative Contrast-Classify (ICC)'' semi-supervised learning scheme. With more labelled data, ICC progressively improves in performance; ICC semi-supervised learning, with 40% labelled videos, performs similarly to fully-supervised counterparts. Our ICC improves MoF by {+1.8, +5.6, +2.5}% on Breakfast, 50Salads, and GTEA respectively for 100% labelled videos.

Authors

Angela Yao

10 papers

Dipika Singhania

4 papers

R. Rahaman

1 papers

TL;DR

Abstract

Authors

References47 items

Coarse to Fine Multi-Resolution Temporal Convolutional Network

A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning

Pretrained Equivariant Features Improve Unsupervised Landmark Discovery

Action Shuffle Alternating Learning for Unsupervised Action Segmentation

Temporally-Weighted Hierarchical Clustering for Unsupervised Action Segmentation

Temporal Action Segmentation from Timestamp Supervision

Semi-Supervised Action Recognition with Temporal Contrastive Learning

Contrastive Transformation for Self-supervised Correspondence Learning

Can Temporal Information Help with Contrastive Self-Supervised Learning?

Cycle-Contrast for Self-Supervised Video Representation Learning

Spatiotemporal Contrastive Video Representation Learning

Unsupervised Learning of Visual Features by Contrasting Cluster Assignments

Momentum contrastive learning for few-shot COVID-19 diagnosis from chest CT images

Temporal Aggregate Representations for Long-Range Video Understanding

Supervised Contrastive Learning

SF-Net: Single-Frame Supervision for Temporal Action Localization

Action Segmentation With Joint Self-Supervised Temporal Domain Adaptation

Temporal Contrastive Pretraining for Video Action Recognition

A Simple Framework for Contrastive Learning of Visual Representations

Joint Visual-Temporal Embedding for Unsupervised Learning of Actions in Untrimmed Sequences

End-to-End Learning of Visual Representations From Uncurated Instructional Videos

Self-Supervised Learning by Cross-Modal Audio-Video Clustering

Momentum Contrast for Unsupervised Visual Representation Learning

Weakly Supervised Energy-Based Learning for Action Segmentation

Semi-Supervised Semantic Segmentation With High- and Low-Level Consistency

A Hybrid RNN-HMM Approach for Weakly Supervised Temporal Action Segmentation

Unsupervised Learning of Action Classes With Continuous Temporal Embedding

Fast Weakly Supervised Action Segmentation Using Mutual Consistency

MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation

Efficient Parameter-Free Clustering Using First Neighbor Relations

D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentation

Temporal Deformable Residual Networks for Action Segmentation in Videos

NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learning

Weakly-Supervised Action Segmentation with Iterative Soft Boundary Assignment

Unsupervised Learning and Segmentation of Complex Activities from Video

Adversarial Learning for Semi-supervised Semantic Segmentation

UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction

Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset

Time-Contrastive Networks: Self-Supervised Learning from Video

Temporal Convolutional Networks for Action Segmentation and Detection

What's the Point: Semantic Segmentation with Point Supervision

The Language of Actions: Recovering the Syntax and Semantics of Goal-Directed Human Activities

Combining embedded accelerometers with computer vision for recognizing food preparation activities

Learning to recognize objects in egocentric activities

Dimensionality Reduction by Learning an Invariant Mapping

Boundary-Aware Cascade Networks for Temporal Action Segmentation

Author manuscript, published in "International Conference on Computer Vision (2013)" Action Recognition with Improved Trajectories

Field of Study

Venue Information

Name

Type

URL

Alternate Names