Beyond the Pixels: Exploring the Effect of Video File Corruptions - - PowerPoint PPT Presentation

▶
beyond the pixels exploring the effect of video file
SMART_READER_LITE
LIVE PREVIEW

Beyond the Pixels: Exploring the Effect of Video File Corruptions - - PowerPoint PPT Presentation

Jun 04, 2023 141 likes •282 views

Beyond the Pixels: Exploring the Effect of Video File Corruptions on Model Robustness Trenton Chang Daniel Y. Fu Yixuan Li Christopher R Stanford University Workshop on Adversarial Robustness in the Real World, ECCV 2020

slide-1
SLIDE 1

Beyond the Pixels: Exploring the Effect of Video File Corruptions

  • n Model Robustness

Workshop on Adversarial Robustness in the Real World, ECCV 2020

Trenton Chang Daniel Y. Fu Yixuan Li Christopher Ré

Stanford University

slide-2
SLIDE 2
  • Video robustness: why file corruption?
  • Setup
  • Model evaluation
  • Simulating file corruptions
  • Results
  • Effect of file corruption on model performance
  • Qualitative analysis of corrupted videos
  • Quantitative analysis of corrupted videos

Out utli line

2

slide-3
SLIDE 3

Vi Video rob

  • bustness:

: not not jus just a a pix pixel-space pr proble lem

Golf Action recognition model ? Video file

We apply corruptions here

?

Previous work studies perturbations to video pixels

3

slide-4
SLIDE 4

Goa

  • al: Sim

imulate rea eal-world file file corr

  • rruptions and

measure th their effect on

  • n vid

ideo mod

  • del rob
  • bustness.

H.264/AVC Decoding Video file

We apply corruptions here

Action recognition model

How does the model perform?

Corrupt pted ed Vi Videos eos in Pixel el Spac ace Table Tennis Basketball Playing Flute Fencing

4

slide-5
SLIDE 5

Setup

5

slide-6
SLIDE 6

Pre-trained ResNet-18 Fine-tune on UCF101/ HMDB51

Mod

  • del

l eval aluatio ion pi pipeli line

Evaluate model

  • n corrupted

data Convert to MP4 w/ H.264 codec Simulate file corruptions

6

slide-7
SLIDE 7

Co Conti tiguous corruption

replace with random bits bitstream index

Ra Random corruption

flipped bit flipped bit flipped bit flipped bit bitstream index

Sim imula latin ing tw two

  • ty

types of

  • f fi

file le corr

  • rruptio

ions

Experiments: vary total length of orange segments (corruption proportion)

7

slide-8
SLIDE 8

Results & Discussion

8

slide-9
SLIDE 9

Accu ccuracy drops as cor

  • rruption proportion in

increases

Model accuracy drops as corruption proportion increases

9

slide-10
SLIDE 10

Mod

  • del

l er error

  • rs cor
  • rrela

late e wit ith cor

  • rruption pr

proportio ion

correct incorrect

Model prediction:

Clips that look worse (more corrupted) tend to be classified incorrectly

10

slide-11
SLIDE 11

Mod

  • del err

rrors corr

  • rrelate with

ith more visu visually dis istorted clip lips giv given con

  • nstant corr
  • rruption proportion and str

trategy

correct incorrect

Model prediction:

More visibly distorted = more likely to be incorrect

✓

Correct

X

Incorrect

11

slide-12
SLIDE 12

In Incorrectly clas classified exam amples ar are e mor

  • re quan

antitatively dis istorted under pix ixel-space Eucl clidean dis istance

12

slide-13
SLIDE 13

Sum ummary ry

  • As proportion of file corrupted goes up, accuracy goes down
  • Clips that look more distorted tend to be classified incorrectly
  • Clips that are more distorted under pixel-space Euclidean distance

tend to be classified incorrectly

Contact: tchang97@cs.stanford.edu

13