The Multimodal Information Based Speech Processing (Misp) 2022 Challenge: Audio-Visual Diarization And Recognition (2023-03-11T00:00:00.000000Z)

TL;DR

The dataset, track settings, and baselines of the MISP2022 challenge are introduced, and analyses of experiments and examples indicate the good performance of AVDR baseline system, and the potential difficulties in this challenge due to, e.g., the far-field video quality, the presence of TV noise in the background, andThe indistinguishable speakers.

Abstract

The Multi-modal Information based Speech Processing (MISP) challenge aims to extend the application of signal processing technology in specific scenarios by promoting the research into wake-up words, speaker diarization, speech recognition, and other technologies. The MISP2022 challenge has two tracks: 1) audio-visual speaker diarization (AVSD), aiming to solve "who spoken when" using both audio and visual data; 2) a novel audio-visual diarization and recognition (AVDR) task that focuses on addressing "who spoken what when" with audio-visual speaker diarization results. Both tracks focus on the Chinese language, and use far-field audio and video in real home-tv scenarios: 2-6 people communicating each other with TV noise in the background. This paper introduces the dataset, track settings, and baselines of the MISP2022 challenge. Our analyses of experiments and examples indicate the good performance of AVDR baseline system, and the potential difficulties in this challenge due to, e.g., the far-field video quality, the presence of TV noise in the background, and the indistinguishable speakers.

Authors

References34 items

Audio-Visual Speech Recognition in MISP2021 Challenge: Dataset Release and Deep Analysis

End-to-End Audio-Visual Neural Speaker Diarization

Channel-Wise AV-Fusion Attention for Multi-Channel Audio-Visual Speech Recognition

The Sjtu System For Multimodal Information Based Speech Processing Challenge 2021

The First Multimodal Information Based Speech Processing (Misp) Challenge: Data, Tasks, Baselines And Results

The USTC-Ximalaya System for the ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription (M2met) Challenge

End-To-End Audio-Visual Speech Recognition with Conformers

Bayesian HMM clustering of x-vector sequences (VBx) in speaker diarization: theory, implementation and analysis on standard tasks

DOVER-Lap: A Method for Combining Overlap-Aware Diarization Outputs

Third DIHARD Challenge Evaluation Plan

Conformer: Convolution-augmented Transformer for Speech Recognition

The STC System for the CHiME-6 Challenge

The JHU Multi-Microphone Multi-Speaker ASR System for the CHiME-6 Challenge

CHiME-6 Challenge: Tackling Multispeaker Speech Recognition for Unsegmented Recordings

Self-Supervised Learning for Audio-Visual Speaker Diarization

Lipreading Using Temporal Convolutional Networks

Multimodal Speaker Diarization Using a Pre-Trained Audio-Visual Synchronization Model

CN-Celeb: A Challenging Chinese Speaker Recognition Dataset

MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation

Deep Audio-Visual Speech Recognition

Gating Neural Network for Large Vocabulary Audiovisual Speech Recognition

VoxCeleb2: Deep Speaker Recognition

X-Vectors: Robust DNN Embeddings for Speaker Recognition

VoxCeleb: A Large-Scale Speaker Identification Dataset

Lip Reading Sentences in the Wild

Audio-Visual Speaker Diarization Based on Spatiotemporal Bayesian Fusion

Speech Perception as a Multimodal Phenomenon

Acoustic Beamforming for Speaker Diarization of Meetings

THE RICH TRANSCRIPTION 2004 SPRING MEETING RECOGNITION EVALUATION

Quantitative association of vocal-tract and facial behavior

NARA-WPE: A Python package for weighted prediction error dereverberation in Numpy and Tensorflow for online and offline processing

Multimodal Speaker Diarization

The Kaldi Speech Recognition Toolkit

license agreement with IEEE. Restrictions apply