Rhythm is a Dancer: Music-Driven Motion Synthesis with Global Structure
Overview
In this work, we present a music-driven neural framework that generates realistic human motions, which are rich, avoid repetitions, and jointly form a global structure that respects the culture of a specific dance genre. We illustrate examples of various dance genre, where we demonstrate choreography control and editing in a number of applications.
Abstract
Synthesizing human motion with a global structure, such as a choreography, is a challenging task. Existing methods tend to concentrate on local smooth pose transitions and neglect the global context or the theme of the motion. In this work, we present a music-driven motion synthesis framework that generates long-term sequences of human motions which are synchronized with the input beats, and jointly form a global structure that respects a specific dance genre. In addition, our framework enables generation of diverse motions that are controlled by the content of the music, and not only by the beat. Our music-driven dance synthesis framework is a hierarchical system that consists of three levels: pose, motif, and choreography. The pose level consists of an LSTM component that generates temporally coherent sequences of poses. The motif level guides sets of consecutive poses to form a movement that belongs to a specific distribution using a novel motion perceptual-loss. And the choreography level selects the order of the performed movements and drives the system to follow the global structure of a dance genre. Our results demonstrate the effectiveness of our music-driven framework to generate natural and consistent movements on various dance types, having control over the content of the synthesized motions, and respecting the overall structure of the dance.
Architecture
Figure 1: Our framework consists of three main components: pose, motif and choreography, where each is responsible for a different resolution in the motion synthesis process. The pose level generates a sequence of temporal coherent poses; the motif level ensures that a sequence of poses creates a short movement that follows a specific motion motif; the choreography level takes care of the global, long-term, structure of the dance by generating the sequence of motifs.
Video
Contributions
- A framework that generates music-guided dance motion that respects the global structure of the dance genre.
- A novel motion perceptual loss to guide the synthesis to follow some high-level motion representations (motion motifs), and control the global culture of the generated motion.
- Our method follows the music's content, such as the tempo, to ensure music-to-motion synchronization, and the spectrum, to enforce diversity in the style of the generated movements.
BibTeX
@article{Aristidou:2023:RhythmDancer,
author = {Aristidou, Andreas and Yiannakidis, Anastasios and Aberman, Kfir and Cohen-Or, Daniel and Shamir, Ariel and Chrysanthou, Yiorgos},
title = {Rhythm is a Dancer: Music-Driven Motion Synthesis with Global Structure},
journal = {IEEE Transactions on Visualization and Computer Graphics},
issue_date = {August 2023},
volume = {29},
number = {8},
month = {aug},
url = {https://doi.org/10.1109/TVCG.2022.3163676},
pages = {3519-3534},
doi = {10.1109/TVCG.2022.3163676},
publisher = {IEEE Computer Society},
address = {},
year = {2023}
}
Acknowledgements
This work has received funding from the University of Cyprus. It has also received funding from the European Union's Horizon 2020 Research and Innovation Programme under Grant Agreement No 739578 and the Government of the Republic of Cyprus through the Deputy Ministry of Research, Innovation and Digital Policy.
