VideoComposer

Controllable technology for video content creation.

VideoComposer: Compositional Video Synthesis
with Motion Controllability

Xiang Wang1*, Hangjie Yuan1*, Shiwei Zhang1*, Dayou Chen1*, Jiuniu Wang1,
Yingya Zhang1, Yujun Shen2, Deli Zhao1, Jingren Zhou1

1Alibaba Group, 2Ant Group

The pursuit of controllability as a higher standard of visual content creation has yielded remarkable progress in customizable image synthesis. However, achieving controllable video synthesis remains challenging due to the large variation of temporal dynamics and the requirement of cross-frame temporal consistency. Based on the paradigm of compositional generation, this work presents VideoCompoer that allows users to flexibly compose a video with textual conditions, spatial conditions, and more importantly temporal conditions. Specifically, considering the characteristic of video data, we introduce the motion vector from compressed videos as an explicit control signal to provide guidance regarding temporal dynamics. In addition, we develop a Spatio-Temporal Condition encoder (STC-encoder) that serves as a unified interface to effectively incorporate the spatial and temporal relations of sequential inputs, with which the model could make better use of temporal conditions and hence achieve higher inter-frame consistency. Extensive experimental results suggest that VideoCompoer is able to control the spatial and temporal patterns simultaneously within a synthesized video in various forms, such as text description, sketch sequence, reference video, or even simply hand-crafted motions. Code and models will be made publicly available.

Overview: Summary of the Generated Videos
The real source of art is your imagination, and VideoComposer is the solution for bringing it to life.
Ability 1: Compositional Video Synthesis
You can generate videos flexibly in any style that you can imagine.
Ability 2: Compositional image-to-video synthesis
Give me a picture, and I will describe a beautiful, vivid and living world for you.
Ability 3: Compositional video inpainting
Give me a broken time, and I will restore it to its most beautiful moment for you.
Ability 4: Compositional sketch-to-video generation
Give me a little hint, and I will showcase the beauty within your heart.
Ability 5: Versatile motion control using hand-crafted motions
Tell me how beauty moves with simple strokes, and I will bring it to life for you.
Ability 6: Motion Transfer
If it is beautiful, I will create an even more beautiful one for you.
Ability 7: Video Translation
If you want it to be better and become the way you want it to be, just tell me in a nutshell.
Ability 8: Video Style Transfer
Seemingly nothing is impossible, if you dare to imagine and dare to act.
Ability 9: Compositional sketch sequence-to-video generation
If it is beautiful, then we can also have it.
Ability 10: Compositional depth sequence-to-video generation
Give me some simple sketches, and I will turn them into the most beautiful moments for you.
Image

If you are seeking an exhilarating challenge and the chance to collaborate with AIGC and large-scale pretraining, then you have come to the right place. We are searching for talented, motivated, and imaginative researchers to join our team. If you are interested, please don't hesitate to send us your resume via email yingya.zyy@alibaba-inc.com