ActNetFormer: Transformer-ResNet Hybrid Method for Semi-Supervised Action Recognition in Videos (2024-04-09T00:00:00.000000Z)

TL;DR

This work proposes a novel approach using Cross-Architecture Pseudo-Labeling with contrastive learning for semi-supervised action recognition, combining pseudo-labeling with contrastive learning for effective learning from both types of samples.

Abstract

Human action or activity recognition in videos is a fundamental task in computer vision with applications in surveillance and monitoring, self-driving cars, sports analytics, human-robot interaction and many more. Traditional supervised methods require large annotated datasets for training, which are expensive and time-consuming to acquire. This work proposes a novel approach using Cross-Architecture Pseudo-Labeling with contrastive learning for semi-supervised action recognition. Our framework leverages both labeled and unlabelled data to robustly learn action representations in videos, combining pseudo-labeling with contrastive learning for effective learning from both types of samples. We introduce a novel cross-architecture approach where 3D Convolutional Neural Networks (3D CNNs) and video transformers (VIT) are utilised to capture different aspects of action representations; hence we call it ActNetFormer. The 3D CNNs excel at capturing spatial features and local dependencies in the temporal domain, while VIT excels at capturing long-range dependencies across frames. By integrating these complementary architectures within the ActNetFormer framework, our approach can effectively capture both local and global contextual information of an action. This comprehensive representation learning enables the model to achieve better performance in semi-supervised action recognition tasks by leveraging the strengths of each of these architectures. Experimental results on standard action recognition datasets demonstrate that our approach performs better than the existing methods, achieving state-of-the-art performance with only a fraction of labeled data. The official website of this work is available at: https://github.com/rana2149/ActNetFormer.

References39 items

Neighbor-Guided Consistent and Contrastive Learning for Semi-Supervised Action Recognition

Semi-Supervised Action Recognition From Temporal Augmentation Using Curriculum Learning

DANet: Semi-supervised differentiated auxiliaries guided network for video action recognition

TP-VIT: A Two-Pathway Vision Transformer for Video Action Recognition

Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels

Understanding Deep Contrastive Learning via Coordinate-wise Optimization

Max-Margin Contrastive Learning

Cross-Model Pseudo-Labeling for Semi-Supervised Action Recognition

Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-supervised Action Recognition

Learning from Temporal Gradient for Semi-supervised Action Recognition

FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling

Deep convolutional neural model for human activities recognition in a sequence of video by combining multiple CNN streams

A transfer learning-based efficient spatiotemporal human action recognition framework for long and overlapping action classes

Multiview Pseudo-Labeling for Semi-supervised Learning from Video

SimPLE: Similar Pseudo Label Exploitation for Semi-Supervised Classification

ViViT: A Video Vision Transformer

Is Space-Time Attention All You Need for Video Understanding?

Semi-Supervised Action Recognition with Temporal Contrastive Learning

TCLR: Temporal Contrastive Learning for Video Representation

Training data-efficient image transformers & distillation through attention

A Comprehensive Study of Deep Video Action Recognition

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Contrastive Representation Learning: A Framework and Review

A survey on video-based Human Action Recognition: recent updates, datasets, challenges, and applications

Augmented Skeleton Based Contrastive Action Learning with Momentum LSTM for Unsupervised Action Recognition

Video Representation Learning with Visual Tempo Consistency

What makes for good views for contrastive learning

A Simple Framework for Contrastive Learning of Visual Representations

FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence

AutoAugment: Learning Augmentation Strategies From Data

SlowFast Networks for Video Recognition

Non-local Neural Networks

The Kinetics Human Action Video Dataset

Dropout as data augmentation

UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild

Boosted multi-class semi-supervised learning for human action recognition

Evaluation of semi-supervised learning method on action recognition

The primary model effectively learns from the auxiliary model through the proposed cross-architecture strategy

Cross-architecture pseudo-labeling using a different architecture can improve the primary model’s performance