VIDEO UNDERSTANDING @ TWITTER C O U R T E S Y O F C O R T E X - PowerPoint PPT Presentation

VIDEO UNDERSTANDING @ TWITTER C O U R T E S Y O F C O R T E X

USER PROTECTION T W I T T E R C O R T E X T W I T T E R C O R T E X

CONTENT UNDERSTANDING T W I T T E R C O R T E X T W I T T E R C O R T E X

T W I T T E R C O R T E X

CONTENT INDEXING T W I T T E R C O R T E X

Two dogs sitting on a event poster couch A red car parked in front of a building T W I T T E R C O R T E X

CONTENT RELEVANCE T W I T T E R C O R T E X

T W I T T E R C O R T E X

USER PROTECTION CONTENT UNDERSTANDING CONTENT INDEXING CONTENT RELEVANCE T W I T T E R C O R T E X

VIDEOS — THE COMBINATORIAL CHALLENGE T W I T T E R C O R T E X

T W I T T E R C O R T E X

T W I T T E R C O R T E X

T W I T T E R C O R T E X

Twitter Images Vine Periscope T W I T T E R C O R T E X

FORWARD OR BACKWARD ? T W I T T E R C O R T E X

CLOCKWISE OR COUNTER-CLOCKWISE ? MOTION: FORWARD OR BACKWARD ? T W I T T E R C O R T E X

MULTIMODAL � � � VIDEO � � � AUDIO � � � TEXT T W I T T E R C O R T E X

0.93933 0.43433 George Toderici and Sudheendra Vijayanarasimhan http://googleresearch.blogspot.com/2015/04/beyond-short-snippets-deep-networks- for.html T W I T T E R C O R T E X

T W I T T E R C O R T E X

RESEARCH ANSWERS T W I T T E R C O R T E X

CNN VARIATIONS VGG SIMONYAN AND ZISSERMAN TWITTER CORTEX T W I T T E R C O R T E X

CNN VARIATIONS Inception Net SZEGEDY AND AL. TWITTER CORTEX T W I T T E R C O R T E X

CNN VARIATIONS Residual Nets KAIMING HE AND AL. TWITTER CORTEX T W I T T E R C O R T E X

RECURRENT NEURAL NETWORKS TWITTER CORTEX T W I T T E R C O R T E X

T W I T T E R C O R T E X

? T W I T T E R C O R T E X

CORTEX SOLUTIONS T W I T T E R C O R T E X

TAXONOMY EFFORT FAST PROTOTYPING SCALABILITY EASY DEPLOYMENT T W I T T E R C O R T E X

TAXONOMY EFFORT FAST PROTOTYPING SCALABILITY EASY DEPLOYMENT T W I T T E R C O R T E X

TAXONOMY EFFORT PRODUCT � MANAGER Defines a taxonomy LABELING � TRAINING MODEL DEPLOYMENT T W I T T E R C O R T E X

TAXONOMY EFFORT PRODUCT � MANAGER Defines a taxonomy LABELING � TRAINING MODEL DEPLOYMENT T W I T T E R C O R T E X

TAXONOMY EFFORT FAST PROTOTYPING SCALABILITY EASY DEPLOYMENT T W I T T E R C O R T E X

CORTEX PROCESS 1. Data gathering 4. Production Service 
 & Traffic 3. Model Training 2. Labeling T W I T T E R C O R T E X

CORTEX REAL PROCESS D MODEL PROTOTYPING L E PROJECT IDEA A Hey, let’s build a model that B B generates video depending U on tweet contents and user’s E mood! G L G I I N N PREVIOUS SLIDE G MODEL TRAINING G That means you’re lucky T W I T T E R C O R T E X

SOFTWARE NEEDS HARDWARE DATA Low latency storage PROCESSING DATABASE Data download g n i d o c e d o e d i v d DISTRIBUTED e s s e r p TRAINING m o C Training speed NETWORK VIDEO CONVOLUTIONAL CLASSIFICATION LAYERS Computation + Memory hungry CPU Variable temporal sequence FULLY CONNECTED word dictionary LAYERS MULTI-GPU Memory hungry LSTM LOOKUP TABLE T W I T T E R C O R T E X

SOFTWARE NEEDS HARDWARE DATA Low latency storage PROCESSING DATABASE Data download g n i d o c e d o e d i v d DISTRIBUTED e s s e r p TRAINING m o C Training speed NETWORK VIDEO CONVOLUTIONAL CLASSIFICATION LAYERS Computation + Memory hungry CPU Variable temporal sequence FULLY CONNECTED word dictionary LAYERS MULTI-GPU Memory hungry LSTM LOOKUP TABLE T W I T T E R C O R T E X

CORTEX 1.0 T W I T T E R C O R T E X

Target Output Input FORWARD Parameters FORWARD Loss output GradParameters BACKWARD GradOutput ACCGRADPARAMETERS nn.Criterion GradInput BACKWARD nn.Module T W I T T E R C O R T E X

Input GradInput Parameters GradParameters AUTOGRAD FUNCTION Target Loss output Some other torch-autograd function* stuff T W I T T E R C O R T E X

CASCADED DEEP NETS FOR BINARY CLASSIFICATION —— DYNAMIC DAG torch-autograd function T W I T T E R C O R T E X

GITHUB.COM/TWITTER/TORCH-AUTOGRAD T W I T T E R C O R T E X

TAXONOMY EFFORT FAST PROTOTYPING SCALABILITY EASY DEPLOYMENT T W I T T E R C O R T E X

LARGE SCALE DATASET T W I T T E R C O R T E X

SOFTWARE NEEDS HARDWARE DATA Low latency storage PROCESSING DATABASE Data download g n i d o c e d o e d i v d DISTRIBUTED e s s e r p TRAINING m o C Training speed NETWORK VIDEO CONVOLUTIONAL CLASSIFICATION LAYERS Computation + Memory hungry CPU Variable temporal sequence FULLY CONNECTED word dictionary LAYERS MULTI-GPU Memory hungry LSTM LOOKUP TABLE T W I T T E R C O R T E X

DATA STORAGES THREADS MODEL INPUTS HDFS Download/Decode 1 Download/Decode 2 Download/Decode 2 VIDEO BATCH BLOB STORAGE 64*N*3*256*256 … … . . . . ……… Download/Decode 64 Download/Decode 64 T W I T T E R C O R T E X

DATA STORAGES THREADS MODEL INPUTS HDFS Download/Decode 1 Download/Decode 2 Download/Decode 2 VIDEO BATCH BLOB STORAGE 64*N*3*256*256 … . . ……… Download/Decode 64 Download/Decode 64 T W I T T E R C O R T E X

UNLIMITED SAMPLING CAPABILITY T W I T T E R C O R T E X

GITHUB.COM/TWITTER/TORCH-DATASET T W I T T E R C O R T E X

LARGE SCALE TRAINING T W I T T E R C O R T E X

SOFTWARE NEEDS HARDWARE DATA Low latency storage PROCESSING DATABASE Data download g n i d o c e d o e d i v d DISTRIBUTED e s s e r p TRAINING m o C Training speed NETWORK VIDEO CONVOLUTIONAL CLASSIFICATION LAYERS Computation + Memory hungry CPU Variable temporal sequence FULLY CONNECTED word dictionary LAYERS MULTI-GPU Memory hungry LSTM LOOKUP TABLE T W I T T E R C O R T E X

GITHUB.COM/TWITTER/TORCH-DISTLEARN T W I T T E R C O R T E X

LARGE SCALE INFERENCE T W I T T E R C O R T E X

T W I T T E R C O R T E X

PREDICTIONS GPU CLUSTER T W I T T E R C O R T E X

TAXONOMY EFFORT FAST PROTOTYPING SCALABILITY EASY DEPLOYMENT T W I T T E R C O R T E X

ANYBODY CAN TRAIN PLAY DEPLOY T W I T T E R C O R T E X

T W I T T E R C O R T E X

NEW PROJECT PROTOTYPING LARGE SCALE DEPLOYMENT CLUSTER T W I T T E R C O R T E X

NEW PROJECT PROTOTYPING < 1 WEEKS LARGE SCALE DEPLOYMENT CLUSTER T W I T T E R C O R T E X

THANK YOU !! T W I T T E R C O R T E X

SLIDES SKETCH • Twitter Cortex mission statements: • User protection • Content Understanding • Content indexing • Content relevance T W I T T E R C O R T E X

SLIDES SKETCH • Focus on video: how is it challenging? • Pixels distribution (darker, blurry) • Combinatorial challenge (motion, streaming, size, mulitmodal) T W I T T E R C O R T E X

SLIDES SKETCH • Focus on video: what research tells us • CNNs • RNNs • Multimodal? T W I T T E R C O R T E X

SLIDES SKETCH • Focus on video: Cortex industrial answer • Smart taxonomy [organization] (fast labeling, feedback) • Fast Prototyping [software] (torch-autograd) • Scalability [software + hardware] (torch-dataset, torch-distlearn) • Fast deployment [software] (embedded torch, no extra step) T W I T T E R C O R T E X