Beyond the Pixels: Exploring the Effect of Video File Corruptions - PowerPoint PPT Presentation

Beyond the Pixels: Exploring the Effect of Video File Corruptions on Model Robustness Trenton Chang Daniel Y. Fu Yixuan Li Christopher Ré Stanford University Workshop on Adversarial Robustness in the Real World, ECCV 2020

Out utli line • Video robustness: why file corruption? • Setup • Model evaluation • Simulating file corruptions • Results • Effect of file corruption on model performance • Qualitative analysis of corrupted videos • Quantitative analysis of corrupted videos 2

Video rob Vi obustness: : not not jus just a a pix pixel-space pr proble lem ? model ? Action Video file recognition Golf We apply corruptions here Previous work studies perturbations to video pixels 3

Goa oal: Sim imulate rea eal-world file file corr orruptions and measure th their effect on on vid ideo mod odel rob obustness. Corrupt pted ed Vi Videos eos in Pixel el Spac ace Action H.264/AVC Basketball Table Tennis Video file recognition Decoding model How does the Fencing Playing Flute model perform? We apply corruptions here 4

Setup 5

Mod odel l eval aluatio ion pi pipeli line Fine-tune on Pre-trained UCF101/ ResNet-18 HMDB51 Evaluate model on corrupted data Simulate file Convert to MP4 corruptions w/ H.264 codec 6

Sim imula latin ing tw two o ty types of of fi file le corr orruptio ions Random corruption Ra bitstream index flipped bit flipped bit flipped bit flipped bit Co Conti tiguous corruption bitstream index replace with random bits Experiments: vary total length of orange segments (corruption proportion) 7

Results & Discussion 8

Accu ccuracy drops as cor orruption proportion in increases Model accuracy drops as corruption proportion increases 9

Mod odel l er error ors cor orrela late e wit ith cor orruption pr proportio ion Clips that look worse (more corrupted) tend to be classified incorrectly correct Model prediction: 10 incorrect

Mod odel err rrors corr orrelate with ith more visu visually dis istorted clip lips giv given con onstant corr orruption proportion and str trategy ✓ Correct X Incorrect More visibly distorted = more likely to be incorrect correct Model prediction: 11 incorrect

In Incorrectly clas classified exam amples ar are e mor ore quan antitatively dis istorted under pix ixel-space Eucl clidean dis istance 12

Sum ummary ry • As proportion of file corrupted goes up, accuracy goes down • Clips that look more distorted tend to be classified incorrectly • Clips that are more distorted under pixel-space Euclidean distance tend to be classified incorrectly Contact: tchang97@cs.stanford.edu 13