VIDEO UNDERSTANDING @ TWITTER C O U R T E S Y O F C O R T E X - - PowerPoint PPT Presentation

▶
video understanding twitter
SMART_READER_LITE
LIVE PREVIEW

VIDEO UNDERSTANDING @ TWITTER C O U R T E S Y O F C O R T E X - - PowerPoint PPT Presentation

Sep 17, 2022 266 likes •918 views

VIDEO UNDERSTANDING @ TWITTER C O U R T E S Y O F C O R T E X USER PROTECTION T W I T T E R C O R T E X T W I T T E R C O R T E X CONTENT UNDERSTANDING T W I T T E R C O R T E X T W I T T E R C O R T E X T W I T T E R C O R T E X

slide-1
SLIDE 1

C O U R T E S Y O F C O R T E X

VIDEO UNDERSTANDING @ TWITTER

slide-2
SLIDE 2

T W I T T E R C O R T E X

USER PROTECTION

T W I T T E R C O R T E X

slide-3
SLIDE 3

T W I T T E R C O R T E X

CONTENT UNDERSTANDING

T W I T T E R C O R T E X

slide-4
SLIDE 4

T W I T T E R C O R T E X

slide-5
SLIDE 5

CONTENT INDEXING

T W I T T E R C O R T E X

slide-6
SLIDE 6

T W I T T E R C O R T E X

event poster Two dogs sitting on a couch A red car parked in front of a building

slide-7
SLIDE 7

CONTENT RELEVANCE

T W I T T E R C O R T E X

slide-8
SLIDE 8

T W I T T E R C O R T E X

slide-9
SLIDE 9

USER PROTECTION CONTENT UNDERSTANDING CONTENT INDEXING CONTENT RELEVANCE

T W I T T E R C O R T E X

slide-10
SLIDE 10

VIDEOS — THE COMBINATORIAL CHALLENGE

T W I T T E R C O R T E X

slide-11
SLIDE 11

T W I T T E R C O R T E X

slide-12
SLIDE 12

T W I T T E R C O R T E X

slide-13
SLIDE 13

T W I T T E R C O R T E X

slide-14
SLIDE 14

T W I T T E R C O R T E X

Twitter Images Vine Periscope

slide-15
SLIDE 15

T W I T T E R C O R T E X

FORWARD OR BACKWARD ?

slide-16
SLIDE 16

T W I T T E R C O R T E X

MOTION: FORWARD OR BACKWARD ? CLOCKWISE OR COUNTER-CLOCKWISE ?

slide-17
SLIDE 17

T W I T T E R C O R T E X

MULTIMODAL

  • VIDEO
  • AUDIO
  • TEXT
slide-18
SLIDE 18

T W I T T E R C O R T E X

George Toderici and Sudheendra Vijayanarasimhan http://googleresearch.blogspot.com/2015/04/beyond-short-snippets-deep-networks- for.html 0.93933 0.43433

slide-19
SLIDE 19

T W I T T E R C O R T E X

slide-20
SLIDE 20

RESEARCH ANSWERS

T W I T T E R C O R T E X

slide-21
SLIDE 21

TWITTER CORTEX

CNN VARIATIONS

VGG SIMONYAN AND ZISSERMAN

T W I T T E R C O R T E X

slide-22
SLIDE 22

TWITTER CORTEX

Inception Net SZEGEDY AND AL.

CNN VARIATIONS

T W I T T E R C O R T E X

slide-23
SLIDE 23

TWITTER CORTEX

CNN VARIATIONS

Residual Nets KAIMING HE AND AL.

T W I T T E R C O R T E X

slide-24
SLIDE 24

TWITTER CORTEX

RECURRENT NEURAL NETWORKS

T W I T T E R C O R T E X

slide-25
SLIDE 25

T W I T T E R C O R T E X

slide-26
SLIDE 26

T W I T T E R C O R T E X

?

slide-27
SLIDE 27

CORTEX SOLUTIONS

T W I T T E R C O R T E X

slide-28
SLIDE 28

TAXONOMY EFFORT FAST PROTOTYPING SCALABILITY EASY DEPLOYMENT

T W I T T E R C O R T E X

slide-29
SLIDE 29

TAXONOMY EFFORT FAST PROTOTYPING SCALABILITY EASY DEPLOYMENT

T W I T T E R C O R T E X

slide-30
SLIDE 30

TAXONOMY EFFORT

  • T W I T T E R C O R T E X

PRODUCT MANAGER

Defines a taxonomy

LABELING

MODEL DEPLOYMENT

TRAINING

slide-31
SLIDE 31

TAXONOMY EFFORT

  • T W I T T E R C O R T E X

PRODUCT MANAGER

Defines a taxonomy

LABELING

MODEL DEPLOYMENT

TRAINING

slide-32
SLIDE 32

TAXONOMY EFFORT FAST PROTOTYPING SCALABILITY EASY DEPLOYMENT

T W I T T E R C O R T E X

slide-33
SLIDE 33

CORTEX PROCESS

T W I T T E R C O R T E X

  • 1. Data gathering
  • 2. Labeling
  • 3. Model Training
  • 4. Production Service


& Traffic

slide-34
SLIDE 34

CORTEX REAL PROCESS

T W I T T E R C O R T E X

PROJECT IDEA

Hey, let’s build a model that generates video depending

  • n tweet contents and user’s

mood!

MODEL PROTOTYPING MODEL TRAINING L A B E L I N G D E B U G G I N G PREVIOUS SLIDE

That means you’re lucky

slide-35
SLIDE 35

T W I T T E R C O R T E X

NEEDS SOFTWARE HARDWARE

VIDEO CLASSIFICATION

CPU NETWORK DATA PROCESSING

Data download

DATABASE

Low latency storage

DISTRIBUTED TRAINING

Training speed C
  • m
p r e s s e d v i d e
  • d
e c
  • d
i n g

CONVOLUTIONAL LAYERS FULLY CONNECTED LAYERS LSTM

Variable temporal sequence

MULTI-GPU

Computation + Memory hungry Memory hungry

LOOKUP TABLE

word dictionary
slide-36
SLIDE 36

T W I T T E R C O R T E X

NEEDS SOFTWARE HARDWARE

VIDEO CLASSIFICATION

CPU NETWORK DATA PROCESSING

Data download

DATABASE

Low latency storage

DISTRIBUTED TRAINING

Training speed C
  • m
p r e s s e d v i d e
  • d
e c
  • d
i n g

CONVOLUTIONAL LAYERS FULLY CONNECTED LAYERS LSTM

Variable temporal sequence

MULTI-GPU

Computation + Memory hungry Memory hungry

LOOKUP TABLE

word dictionary
slide-37
SLIDE 37

CORTEX 1.0

T W I T T E R C O R T E X

slide-38
SLIDE 38

T W I T T E R C O R T E X

FORWARD BACKWARD ACCGRADPARAMETERS

Input Output Target

FORWARD

Loss output

BACKWARD

GradOutput Parameters GradInput GradParameters

nn.Module nn.Criterion

slide-39
SLIDE 39

T W I T T E R C O R T E X

AUTOGRAD FUNCTION

Input Loss output Parameters GradInput GradParameters

torch-autograd function*

Target Some other stuff

slide-40
SLIDE 40

T W I T T E R C O R T E X

CASCADED DEEP NETS FOR BINARY CLASSIFICATION —— DYNAMIC DAG

torch-autograd function

slide-41
SLIDE 41

GITHUB.COM/TWITTER/TORCH-AUTOGRAD

T W I T T E R C O R T E X

slide-42
SLIDE 42

TAXONOMY EFFORT FAST PROTOTYPING SCALABILITY EASY DEPLOYMENT

T W I T T E R C O R T E X

slide-43
SLIDE 43

LARGE SCALE DATASET

T W I T T E R C O R T E X

slide-44
SLIDE 44

T W I T T E R C O R T E X

NEEDS SOFTWARE HARDWARE

VIDEO CLASSIFICATION

CPU NETWORK DATA PROCESSING

Data download

DATABASE

Low latency storage

DISTRIBUTED TRAINING

Training speed C
  • m
p r e s s e d v i d e
  • d
e c
  • d
i n g

CONVOLUTIONAL LAYERS FULLY CONNECTED LAYERS LSTM

Variable temporal sequence

MULTI-GPU

Computation + Memory hungry Memory hungry

LOOKUP TABLE

word dictionary
slide-45
SLIDE 45

T W I T T E R C O R T E X

DATA STORAGES THREADS MODEL INPUTS

HDFS BLOB STORAGE ……… Download/Decode 1

… . .

Download/Decode 2 Download/Decode 64 Download/Decode 2 Download/Decode 64 VIDEO BATCH 64*N*3*256*256

… . .

slide-46
SLIDE 46

T W I T T E R C O R T E X

DATA STORAGES THREADS MODEL INPUTS

HDFS BLOB STORAGE ……… Download/Decode 1

… . .

Download/Decode 2 Download/Decode 64 Download/Decode 2 Download/Decode 64 VIDEO BATCH 64*N*3*256*256

slide-47
SLIDE 47

T W I T T E R C O R T E X

UNLIMITED SAMPLING CAPABILITY

slide-48
SLIDE 48

T W I T T E R C O R T E X

GITHUB.COM/TWITTER/TORCH-DATASET

slide-49
SLIDE 49

LARGE SCALE TRAINING

T W I T T E R C O R T E X

slide-50
SLIDE 50

T W I T T E R C O R T E X

NEEDS SOFTWARE HARDWARE

VIDEO CLASSIFICATION

CPU NETWORK DATA PROCESSING

Data download

DATABASE

Low latency storage

DISTRIBUTED TRAINING

Training speed C
  • m
p r e s s e d v i d e
  • d
e c
  • d
i n g

CONVOLUTIONAL LAYERS FULLY CONNECTED LAYERS LSTM

Variable temporal sequence

MULTI-GPU

Computation + Memory hungry Memory hungry

LOOKUP TABLE

word dictionary
slide-51
SLIDE 51

GITHUB.COM/TWITTER/TORCH-DISTLEARN

T W I T T E R C O R T E X

slide-52
SLIDE 52

LARGE SCALE INFERENCE

T W I T T E R C O R T E X

slide-53
SLIDE 53

T W I T T E R C O R T E X

slide-54
SLIDE 54

T W I T T E R C O R T E X

PREDICTIONS GPU CLUSTER

slide-55
SLIDE 55

TAXONOMY EFFORT FAST PROTOTYPING SCALABILITY EASY DEPLOYMENT

T W I T T E R C O R T E X

slide-56
SLIDE 56

T W I T T E R C O R T E X

ANYBODY CAN TRAIN PLAY DEPLOY

slide-57
SLIDE 57

T W I T T E R C O R T E X

slide-58
SLIDE 58

T W I T T E R C O R T E X

NEW PROJECT PROTOTYPING DEPLOYMENT LARGE SCALE CLUSTER

slide-59
SLIDE 59

T W I T T E R C O R T E X

NEW PROJECT PROTOTYPING DEPLOYMENT LARGE SCALE CLUSTER

< 1 WEEKS

slide-60
SLIDE 60

THANK YOU !!

T W I T T E R C O R T E X

slide-61
SLIDE 61

SLIDES SKETCH

  • Twitter Cortex mission statements:
  • User protection
  • Content Understanding
  • Content indexing
  • Content relevance

T W I T T E R C O R T E X

slide-62
SLIDE 62

SLIDES SKETCH

  • Focus on video: how is it challenging?
  • Pixels distribution (darker, blurry)
  • Combinatorial challenge (motion, streaming, size, mulitmodal)

T W I T T E R C O R T E X

slide-63
SLIDE 63

SLIDES SKETCH

  • Focus on video: what research tells us
  • CNNs
  • RNNs
  • Multimodal?

T W I T T E R C O R T E X

slide-64
SLIDE 64

SLIDES SKETCH

  • Focus on video: Cortex industrial answer
  • Smart taxonomy [organization] (fast labeling, feedback)
  • Fast Prototyping [software] (torch-autograd)
  • Scalability [software + hardware] (torch-dataset, torch-distlearn)
  • Fast deployment [software] (embedded torch, no extra step)

T W I T T E R C O R T E X