ActBERT: Learning Global-Local Video-Text Representations Linchao - - PowerPoint PPT Presentation

ActBERT: Learning Global-Local Video-Text Representations

Linchao Zhu

Self-supervised pretraining

• Single modal pretraining
• Image
• Jigsaw, CPC, MoCO, SimCLR
• Video
• Shuffle and Learn, Video GAN
• Text
• Word2Vec, GPT, BERT
• Multi-modal pretraining
• Image-text
• Vilbert, LXMERT, VisualBERT, VLBERT, UNITER, Unified VLP
• Video-text
• HowTo100M, VideoBERT, CBT, MIL-NCE

Video and text pre-training

• Instructional videos
• A natural source for video and text representation

learning

• Instructions are available from ASR
• Diverse domains
• Cooking
• Assembling furniture

Video and text pre-training

• Howto100M

Miech et al., HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips, ICCV 2019

Video and text pre-training

• VideoBERT

Sun et al., VideoBERT: A Joint Model for Video and Language Representation Learning, ICCV 2019.

ActBERT

• Decouple verbs and nouns
• Add spinach -> verb: “add”, noun: “spinach”
• Verb label is extracted from the description.
• Train a 3D ConvNet for verb classification
• Object label can be produced from a pre-

trained Faster R-CNN.

Tangled transformer

Rotate shrimp balls

Clip-level feature Language model

Joint embedding / BERT Rotate shrimp balls

Action feature Language model

ActBERT with Tangled Transformer

Region feature

Tangled transformer

Add & Norm Feed forward Add & Norm Multi-head Attention Q K V hl+1

w

Action inputs Region inputs Linguistic inputs

Transformer

Q K V Q K V

hl+1

r

hl+1

a

hl+1

w

• transformer
• transformer

w

a

r

• transformer

Action inputs Region inputs Linguistic inputs

Tangled transformer

Tangled transformer

Q K V Q K V

MHA

Q K V

hl+1

r

hl+1

a

hl+1

w

• transformer
• transformer

w

a

r

• transformer

Action inputs Region inputs Linguistic inputs

MHA: Multi-head Attention

• transformer: word transformer

w

• transformer: action transformer

a

r

• transformer: region transformer

Tangled transformer

Q K V Q K V

MHA

Q K V

hl+1

r

hl+1

a

hl+1

w

• transformer
• transformer

w

a

r

• transformer

Action inputs Region inputs Linguistic inputs

MHA: Multi-head Attention

• transformer: word transformer

w

• transformer: action transformer

a

r

• transformer: region transformer

Tangled transformer

Q K V Q K V K

MHA MHA

Q Q K V V

hl

a

hl

r

hl+1

r

hl+1

a

hl+1

w

• transformer
• transformer

w

a

r

• transformer

Action inputs Region inputs Linguistic inputs

MHA: Multi-head Attention

• transformer: word transformer

w

• transformer: action transformer

a

r

• transformer: region transformer

ActBERT

Rotate shrimp balls.

…

Add spinach.

…

ActBERT

Rotate shrimp balls.

…

Action feature

Add spinach.

…

Action feature Rotate Add

3D CNN 3D CNN

ActBERT

Rotate shrimp balls.

…

Action feature

Add spinach.

…

Action feature

… …

Region feature

… …

Region feature Rotate Add

3D CNN 3D CNN

Proposal Generation

ActBERT

Visual (action) embedding Global stacked frames Local object regions

Spatial position encoding

Token embedding [ACT]

[REGION]

[ACT]

[REGION]

ActBERT

Visual (action) embedding

…

Global stacked frames Local object regions

Spatial position encoding

Token embedding [CLS] rotate shrimp [SEP] [ACT]

[REGION]

[ACT]

[REGION]

[SEP]

…

ActBERT

Segment embedding

SA SA SA SB SA SA SB SB SB

Visual (action) embedding

… …

Global stacked frames Local object regions

Spatial position encoding

Token embedding [CLS] rotate shrimp [SEP] [ACT]

[REGION]

[ACT]

[REGION]

[SEP]

…

ActBERT

Position embedding

1 2 3 4 5 6 7 8

Segment embedding

SA SA SA SB SA SA SB SB SB

Visual (action) embedding

… … …

Global stacked frames Local object regions

Spatial position encoding

Token embedding [CLS] rotate shrimp [SEP] [ACT]

[REGION]

[ACT]

[REGION]

[SEP]

…

ActBERT

Position embedding

1 2 3 4 5 6 7 8

Segment embedding

SA SA SA SB SA SA SB SB SB

Visual (action) embedding rotate [CLS] [SEP] shrimp Nouni rotate add [SEP]

… … …

Nounj

…

ActBERT

Masked language modeling Global stacked frames Local object regions

Spatial position encoding

Token embedding [CLS] rotate [MASK] [SEP] [ACT]

[REGION]

[ACT]

[REGION]

[SEP]

…

ActBERT

Position embedding

1 2 3 4 5 6 7 8

Segment embedding

SA SA SA SB SA SA SB SB SB

Visual (action) embedding rotate [CLS] [SEP] shrimp Nouni rotate add [SEP]

… … …

Nounj

…

ActBERT

Masked action (verb) classification Masked language modeling Global stacked frames Local object regions

Spatial position encoding

Token embedding [CLS] rotate [MASK] [SEP] [ACT]

[REGION]

[ACT]

[REGION]

[SEP]

…

ActBERT

Position embedding

1 2 3 4 5 6 7 8

Segment embedding

SA SA SA SB SA SA SB SB SB

Visual (action) embedding rotate [CLS] [SEP] shrimp Nouni rotate add [SEP]

… … …

Nounj

…

ActBERT

Masked action (verb) classification Masked object (noun) classification Masked language modeling Global stacked frames Local object regions

Spatial position encoding

Token embedding [CLS] rotate [MASK] [SEP] [ACT]

[REGION]

[ACT]

[REGION]

[SEP]

…

ActBERT

Position embedding

1 2 3 4 5 6 7 8

Segment embedding

SA SA SA SB SA SA SB SB SB

Visual (action) embedding rotate [CLS] [SEP] shrimp Nouni rotate add [SEP]

… … …

Nounj

…

ActBERT

Masked action (verb) classification Masked object (noun) classification Masked language modeling Global stacked frames Local object regions

Spatial position encoding

Cross-modal matching Token embedding [CLS] rotate [MASK] [SEP] [ACT]

[REGION]

[ACT]

[REGION]

[SEP]

…

Evaluation

• Pre-training
• Howto100M - 136M clips, 1.221M videos.
• Downstream evaluation
• Video captioning on YouCook2 (+1.7 on METEOR)
• Action segmentation on COIN (+22.65)
• Action step localization on CrossTask (+7.8)
• Text-video clip retrieval on YouCook2 (+5.4 on R@1), MSR-VTT (+1.1 on R@1)
• Video question answering on MSR-VTT (+3.1)

Thank you!