Unified Fully and Timestamp Supervised Temporal Action Segmentation via Sequence to Sequence Translation

Published in

European Conference on Computer Vision(2022)

External Links:

Generate Graph DownloadPDF

TL;DR

This paper introduces a unified framework for video action segmentation via sequence to sequence (seq2seq) translation in a fully and timestamp supervised setup, and extends the framework to the timestamp supervised setting via the proposed constrained k-medoids algorithm to generate pseudo-segmentations.

Abstract

This paper introduces a unified framework for video action segmentation via sequence to sequence (seq2seq) translation in a fully and timestamp supervised setup. In contrast to current state-of-the-art frame-level prediction methods, we view action segmentation as a seq2seq translation task, i.e., mapping a sequence of video frames to a sequence of action segments. Our proposed method involves a series of modifications and auxiliary loss functions on the standard Transformer seq2seq translation model to cope with long input sequences opposed to short output sequences and relatively few videos. We incorporate an auxiliary supervision signal for the encoder via a frame-wise loss and propose a separate alignment decoder for an implicit duration prediction. Finally, we extend our framework to the timestamp supervised setting via our proposed constrained k-medoids algorithm to generate pseudo-segmentations. Our proposed framework performs consistently on both fully and timestamp supervised settings, outperforming or competing state-of-the-art on several datasets. Our code is publicly available at https://github.com/boschresearch/UVAST.

Authors

Zico Kolter

3 papers

Juergen Gall

14 papers

Nadine Behrmann

1 papers

References42 items

Hierarchical Transformers Are More Efficient Language Models

ASFormer: Transformer for Action Segmentation

FIFA: Fast Inference Approximation for Action Segmentation

Long-Short Transformer: Efficient Transformers for Language and Vision

Coarse to Fine Multi-Resolution Temporal Convolutional Network

Unified Fully and Timestamp Supervised Temporal Action Segmentation via Sequence to Sequence Translation

Published in

European Conference on Computer Vision(2022)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

Authors

Zico Kolter

3 papers

Juergen Gall

14 papers

Nadine Behrmann

1 papers

References42 items

Hierarchical Transformers Are More Efficient Language Models

ASFormer: Transformer for Action Segmentation

FIFA: Fast Inference Approximation for Action Segmentation

Long-Short Transformer: Efficient Transformers for Language and Vision

Coarse to Fine Multi-Resolution Temporal Convolutional Network

S. Golestaneh

1 papers

TL;DR

Abstract

Authors

References42 items

Hierarchical Transformers Are More Efficient Language Models

ASFormer: Transformer for Action Segmentation

FIFA: Fast Inference Approximation for Action Segmentation

Long-Short Transformer: Efficient Transformers for Language and Vision

Coarse to Fine Multi-Resolution Temporal Convolutional Network

TL;DR

Abstract

Authors

References42 items

Hierarchical Transformers Are More Efficient Language Models

ASFormer: Transformer for Action Segmentation

FIFA: Fast Inference Approximation for Action Segmentation

Long-Short Transformer: Efficient Transformers for Language and Vision

Coarse to Fine Multi-Resolution Temporal Convolutional Network

ViViT: A Video Vision Transformer

Temporal Action Segmentation from Timestamp Supervision

Is Space-Time Attention All You Need for Video Understanding?

Global2Local: Efficient Structure Search for Video Action Segmentation

Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers

Temporal Relational Modeling with Self-Supervision for Action Segmentation

MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers

Long Range Arena: A Benchmark for Efficient Transformers

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Alleviating Over-segmentation Errors by Detecting Action Boundaries

Improving Action Segmentation via Graph-Based Temporal Reasoning

Longformer: The Long-Document Transformer

Action Segmentation With Joint Self-Supervised Temporal Domain Adaptation

Weakly Supervised Energy-Based Learning for Action Segmentation

Fine-grained Action Segmentation using the Semi-Supervised Action GAN

A Hybrid RNN-HMM Approach for Weakly Supervised Temporal Action Segmentation

Fast Weakly Supervised Action Segmentation Using Mutual Consistency

MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation

Transformer-XL: Attentive Language Models beyond a Fixed-Length Context

Temporal Deformable Residual Networks for Action Segmentation in Videos

NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learning

Weakly-Supervised Action Segmentation with Iterative Soft Boundary Assignment

Attention is All you Need

Action Sets: Weakly Supervised Action Segmentation Without Ordering Constraints

Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset

Temporal Convolutional Networks for Action Segmentation and Detection

Connectionist Temporal Modeling for Weakly Supervised Action Labeling

Temporal Action Detection Using a Statistical Language Model

An end-to-end generative framework for video segmentation and recognition

Weakly Supervised Action Labeling in Videos under Ordering Constraints

The Language of Actions: Recovering the Syntax and Semantics of Goal-Directed Human Activities

Combining embedded accelerometers with computer vision for recognizing food preparation activities

Learning latent temporal structure for complex event detection

A database for fine grained activity detection of cooking activities

Learning to recognize objects in egocentric activities

Boundary-Aware Cascade Networks for Temporal Action Segmentation

Fast saliency based pooling of fisher encoded dense trajectories

Field of Study

Venue Information

Name

Type

URL

Alternate Names