TL
TL;DR
There have been various music visualization schemes with different purposes, such as to visualize the emotion of music by selecting photos, to implement an active listening interface by visualizing structure or progress of music, or to create media art performance.
Abstract
The goal of this paper to generate a visually appealing video that responds to music with a neural network so that each frame of the video reflects the musical characteristics of the corresponding audio clip. To achieve the goal, we propose a neural music visualizer directly mapping deep music embeddings to style embeddings of StyleGAN, named TraumerAI, which consists of a music auto-tagging model using short-chunk CNN and StyleGAN2 pre-trained on WikiArt dataset. Rather than establishing an objective metric between musical and visual semantics, we manually labeled the pairs in a subjective manner. An annotator listened to 100 music clips of 10 seconds long and selected an image that suits the music among the 200 StyleGAN-generated examples. Based on the collected data, we trained a simple transfer function that converts an audio embedding to a style embedding. The generated examples show that the mapping between audio and video makes a certain level of intra-segment similarity and inter-segment dissimilarity.
References15 items
1
Crossing You in Style: Cross-modal Style Transfer from Music to Visual Arts
2
Evaluation of CNN-based Automatic Music Tagging Models
3
Analyzing and Improving the Image Quality of StyleGAN
4
A Style-Based Generator Architecture for Generative Adversarial Networks
5