Tell Me What Happened: Unifying Text-guided Video Completion
via Multimodal Masked Video Generation


Tsu-Jui Fu1   Licheng Yu2   Ning Zhang2   Cheng-Yang Fu2
Jong-Chyi Su3   William Yang Wang1   Sean Bell2
1UC Santa Barbara   2Meta   3NEC Lab
Conference on Computer Vision and Pattern Recognition (CVPR) 2023


Abstract

Generating a video given the first several static frames is challenging as it anticipates reasonable future frames with temporal coherence. Besides video prediction, the ability to rewind from the last frame or infilling between the head and tail is also crucial, but they have rarely been explored for video completion. Since there could be different outcomes from the hints of just a few frames, a system that can follow natural language to perform video completion may significantly improve controllability. Inspired by this, we introduce a novel task, text-guided video completion (TVC), which requests the model to generate a video from partial frames guided by an instruction. We then propose Multimodal Masked Video Generation (MMVG) to address this TVC task. During training, MMVG discretizes the video frames into visual tokens and masks most of them to perform video completion from any time point. At inference time, a single MMVG model can address all 3 cases of TVC, including video prediction, rewind, and infilling, by applying corresponding masking conditions. We evaluate MMVG in various video scenarios, including egocentric, animation, and gaming. Extensive experimental results indicate that MMVG is effective in generating high-quality visual appearances with text guidance for TVC.


Qualitative Results

👇 press the tab for different tasks and datasets


Kitchen (1282)
First Frame w/o Text Text TATS Ours GT
open cupboard
turn on tap
put down cloth
put plate on counter
put down cup
pour water in pot
rinse plate
open fridge
place pan
close lid

Flintstones (1282)
First Frame w/o Text Text TATS Ours GT
They are looking at each other while
Fred is speaking.
Barney is in the
living room, talking.
Wilma is in a car
and talking about something.
Fred is talking
to Barney and
turning his head.
Betty is talking in
the living room
and nodding.
Fred is walking
across a room,
talking to himself.
Betty is quickly opening and closing
a window blind.
Fred is walking outside while he
is looking behind.
Wilma gestures
with one hand
while talking.
Barney is in the
car talking and laughing.

MUGEN (1282)
First Frame w/o Text Text TATS Ours GT
Mugen lands on a snail to crush it. It walks right to collect coins and turns left.
Mugen runs from left to right, collects coin, jumps to hit face, and jumps to down.
Mugen walks to
the right across a platform. It picks up a gem and a coin before crushing a worm.
Mugen jumps over
a gear and then collects a coin.
Mugen jumps and moves left. And
then collects three coins and a gem.
Mugen runs from right to left and it collect coins there. It saw a ladybug so it runs from left to right.
Mugen jumps down and up to collect a coin. Then it jumps left from right to collect the coins.
Mugen climbs up the ladder and moves to the right. It jumps again and gets killed.
Mugen walks to the right, gets a gem, and then jumps up to a platform for coins.
Mugen jumps down to the middle floor to knock the worm and jumps down to the ground floor.

Kitchen (1282)
Last Frame Text Ours GT
turn tap water off
open oven
put down pepper
close drawer
add olive oil

Flintstones (1282)
Last Frame Text Ours GT
Betty is in the
living room
reading the paper.
Wilma is riding in
the car and talking.
Barney is talking
at the window.
Fred stands over
the fence to have
a glimpse.
Wilma is standing
in the kitchen. She turns her head
then she speaks.

MUGEN (1282)
Last Frame Text Ours GT
Mugen runs from right to left and
gets the coins.
Mugen runs from left to right, collects some coins, and kills a snail. And it jumps down.
Mugen jumps to stage. It runs from
left to right, collects
a coin, and jumps
over a worm. Then
it jumps up.
Mugen jumps over
a box and then
towards a coin.
Mugen jumps up
and gets the coins.

Kitchen (1282)
First Frame Last Frame Text Ours GT
take plate
turn down hob
take lid off
grab salt
put plates in sink

Flintstones (1282)
First Frame Last Frame Text Ours GT
Barney is trying
to sing in a
living room.
Wilma is talking
and holds a
finger over her lips.
Fred and Barney speak out loud.
Wilma looks back
at Betty who is speaking then she turns her head.
Fred and Wilma
are dancing
across a room.

MUGEN (1282)
First Frame Last Frame Text Ours GT
Mugen runs to
left. Then collects
a coin and a gem.
Mugen jumps up
the stage. It runs
from left to right
and jumps on a worm.
Mugen jumps
down the ladder
and jumps up. It
collects a gem.
Mugen climbs up a
ladder. It jumps onto
a stack of boxes, drops down, and is killed by a worm.
Mugen climbs a ladder onto a platform and kills a worm. It then walks to the right and towards a gem.

Kitchen (1282)
K Frames Text K=1 K=2 K=3 K=4 GT
open drawer
put bowl in saucepan
put glove back
place pepper shaker
rinse knife

Flintstones (1282)
K Frames Text K=1 K=2 K=3 K=4 GT
Wilma is talking on the phone.
Fred and Barney shake hands across the rock wall.
Betty and Wilma
are peeking their heads into a room.
Betty and barney are dancing in a room.
Fred angrily points
at Wilma while
he talks to her.

MUGEN (1282)
K Frames Text K=1 K=2 K=3 K=4 GT
Mugen runs to the right and jumps to an upper level, where
it collects a coin.
Mugen walks to the right to get a gem. It then walks past the ladybug and jumps.
Mugen runs to the right, jumps onto
a platform, and collects coins.
Mugen walks. And then it jumps.
Mugen goes from
the left and
collects coins.

Flintstones (1282)
First Frame Text 1 Output 1 Text 2 Output 2
Fred speaks to a phone and laughs.
Fred hangs up a phone and laughs.
Fred is holding
a bone beside
his head.
Fred is moving a
bone up and down
beside his head.
Wilma and Betty
are walking
through a room.
Wilma and Betty
are speaking
in a room.

MUGEN (1282)
First Frame Text 1 Output 1 Text 2 Output 2
Mugen jumps from left to right to an upper platform.
Mugen runs right
to left and
collects a gem.
Mugen runs to
the right. It jumps
down the ground.
Mugen runs to
the right. It jumps
landing on a face.
Mugen jumps.
Mugen jumps and gets a gem and coin.

MUGEN (1282)
Last Frame Text 1 Output 1 Text 2 Output 2
Mugen walks to the right and then jumps over a mouse to collect coins.
Mugen walks to the right and then jumps to collect coins.
Mugen jumps and moves to left.
Mugen jumps down from a ladder and moves to left.
Mugen moves from right to left and climbs onto a ladder.
Mugen moves from left to right and climbs onto a ladder.

MUGEN (1282)
First Frame Last Frame Text 1 Output 1 Text 2 Output 2
Mugen jumps down
a ladder, collects coins, and then
jumps down.
Mugen jumps down
a ladder, collects coins, and then
jumps up and down.
Mugen collects coins. Then it runs from
left to right.
Mugen collects coins and jumps on face. Then it runs from
left to right.
Mugen jumps to get the coin. It then keeps walking and drops
off the platform.
Mugen jumps to get the coin. It then jumps over a gear and drops off the platform.


UCF-101 (1282)

BAIR (642)

UCF-101 (1282)
Typing Writing on Board Knitting Pull Up Mixing
Front Crawl Skiing Yo Yo Playing Guitar Surfing

WebVid (3842)
wash hand cut chicken
with knife		 boy writes
in notebook		 sailboat on
horizon		 cloudscape
time-lapse
type on
laptop keyboard		 downtown city with traffic car cook goulash
soup		 wind turbine flame and wood
pour coffee
into cup		 walk outdoors
on beach		 high-speed railway link busy city of
times square		 young team
busy discusses
green sea turtle swims and relaxes man trains on exercise machine flags flatter on strong wind beautiful girl swings river flows under old stone bridge
rotates apple lollipop sunset on the baltic sea woman unveils curtain elephant shakes his head hand of man playing guitar
designer mixes paint underwater of coral reef drive on
empty road		 child rides
a bike		 shave his face
in bathroom

TGIF (3842)
shove food in his mouth tilt her head hug each other surf in the ocean sing a song
perform in front of audiences skateboard down the hill move his lips talk and smile stick her tongue out
kiss each other ice skating snowboard on a mountain walk palm tree is blowing
cut a mini pizza swing his club slap their hands together flying above the clouds run along
a track