Microsoft AI & Research Traditional IR Keyword based Search - - PowerPoint PPT Presentation

▶
microsoft ai research traditional ir
SMART_READER_LITE
LIVE PREVIEW

Microsoft AI & Research Traditional IR Keyword based Search - - PowerPoint PPT Presentation

Apr 14, 2023 239 likes •714 views

Microsoft AI & Research Traditional IR Keyword based Search AUTB streams Inverted index User Engagement Natural Language User clicks, Search Inverted index metawords Voice, Vision AI Context-Based Deep Learning Vectors Search

slide-1
SLIDE 1

Microsoft AI & Research

slide-2
SLIDE 2
slide-3
SLIDE 3
slide-4
SLIDE 4
slide-5
SLIDE 5
slide-6
SLIDE 6

Keyword based Search Natural Language Search Voice, Vision Context-Based Search Traditional IR

AUTB streams Inverted index

User Engagement

User clicks, metawords Inverted index

AI

Deep Learning Vectors

slide-7
SLIDE 7
slide-8
SLIDE 8
slide-9
SLIDE 9
slide-10
SLIDE 10
slide-11
SLIDE 11
slide-12
SLIDE 12
slide-13
SLIDE 13
slide-14
SLIDE 14

AI Conversational Search

slide-15
SLIDE 15
slide-16
SLIDE 16

 Key

ey Cha halle llenges nges

slide-17
SLIDE 17
slide-18
SLIDE 18
slide-19
SLIDE 19
slide-20
SLIDE 20
slide-21
SLIDE 21
slide-22
SLIDE 22
slide-23
SLIDE 23

Ideas Experiments Trained models Shipped models

slide-24
SLIDE 24
slide-25
SLIDE 25

 Trad

ade-offs ffs – A Vi Visu sual al T ax axonomy nomy

slide-26
SLIDE 26

“Maximum bar” metrics Model latency (milliseconds @ 99) Cost of HW ($/ops) Error rate (1%) Model footprint Throughput (@ given 99% latency) Cost efficiency (ops/$) Model accuracy (99%) Model fit “Minimum bar” metrics

slide-27
SLIDE 27

Model latency Cost of HW Model error rate

slide-28
SLIDE 28

Model latency Cost of HW Model error rate

slide-29
SLIDE 29

Latency Cost of HW Model error rate

slide-30
SLIDE 30

Latency Cost of HW Model error rate

slide-31
SLIDE 31

Model error rate Cost Latency

slide-32
SLIDE 32

 Cost

t to serve: : BERT

slide-33
SLIDE 33
slide-34
SLIDE 34
slide-35
SLIDE 35

 Mod

  • del

el fi fit: : Tempora mporal/ l/sp spatia atial l br break eakdo down: wn: wor word-base ased d RNNs

slide-36
SLIDE 36

Case study: Typical word-based RNNs

  • Hidden dimensions = ~100
  • Input size = 100 (or more)
  • Output size = 100
  • Bi-directional
  • Time steps = 100+ (one timestep per word in a

sentence/paragraph) Problem:

  • Native TensorFlow execution is slow
  • 20K kernel invocations
  • 5 us/invocation
  • ~100 ms delay
  • We need a custom implementation [1]
  • TensorFlow custom op for RNN execution
  • Collaboration with Nvidia

Diagram source: https://devblogs.nvidia.com/optimizing-recurrent-neural-networks- cudnn-5

Model fit technique #1: T emporal/spatial breakdown

[1] - S81039 – GTC 2018: Accelerating Machine Reading Comprehension Models Using GPUs

slide-37
SLIDE 37

cuDNN DNN Blockpe ckpersist sistent nt Batch 1 Batch 5 Batch 1 Batch 5 K80 9.30 .30 6.52 .52 1.77 77 1.87 .87 M4 M40 2.78 78 2.90 .90 0.95 .95 1.00 .00 P4 P4 0.25 .25 1.06 .06 0.68 .68 0.72 72

  • Time is averaged over back-to-back 100 iterations (different input data)
  • 40 timesteps

1) For K80 and M40 we used CUDNN_RNN_ALGO_STANDARD, since grid persistent approach is supported starting from Pascal. 2) For P4 we chose CUDNN_RNN_ALGO_PERSIST_STATIC, and it is more than 2x faster than block persistent. Block-persistent RNNs results – joint collaboration with NVidia

Latency Cost of HW Model accuracy

slide-38
SLIDE 38

GRNN

0.5 1 1.5 2 2.5 3 1 5 10 15 20 25 30 35 40 45 50

Latency (ms) Batch Size

CharRNN

* GRNN: Low-Latency and Scalable RNN Inference on GPUs. Connor Holmes, Daniel Mawhirter, Yuxiong He, Feng Yan, Bo Wu. To Appear @ Eurosys 2019.

2 4 6 8 10 12 14 16 18 20 10 20 30 40 50

Latency (ms) Batch Size

Text Classification

GRNN cuDNN Persistent cuDNN Traditional / TensorRT

slide-39
SLIDE 39

 Mode

del l fi fit: : Qu Quantizat antization: ion: ELMO O

slide-40
SLIDE 40
  • Case study: ELMO [1]
  • Deep contextualized word representation that models

both (1) complex characteristics of word use (e.g., syntax and semantics), and (2) how these uses vary across linguistic contexts (i.e., to model polysemy).

  • Characteristics
  • Main block: Very large LSTM, 4-layer
  • Hidden states = 4096
  • Hard to fit in memory
  • We can improve model fit through quantization
  • Latency greatly improved
  • We might need more expensive HW (INT8 support is
  • nly in V100)
  • Model accuracy suffers

Latency Cost of HW Model error rate [1] - https://allennlp.org/elmo

slide-41
SLIDE 41

Projection bi-LSTM from the last stage dominates the execution time 2 layer bi-LSTM, 512 input, 4096 hidden, 512 projection Batch 1, 100 timesteps, takes about 70 ms on P40 in FP32 when using cuDNN Reduce precision to INT8 to meet latency requirements Custom implementation (Joint collaboration with Nvidia): 1) Single cuBLAS call per layer for input GEMMs for all timesteps (done in FP32) 2) Elementwise ops kernel in FP32 3) Custom MatVec kernel for recurrent part (at batch 1)

  • Input / Output vectors are FP32
  • Recurrent weights are quantized once
  • All math is FP32, so older GPU architectures can be used as well

We choose conservative quantization approach, keep as much as possible in FP32

slide-42
SLIDE 42
  • Only hidden and projection weights are quantized, input weights and all bias are in FP32
  • In the quantization scheme (e.g. https://arxiv.org/abs/1609.08144v2 ), each row has a separate scale factor

rowMax = max(abs(row)) scale = 127.0 / rowMax quantRow = int(round(row * scale))

Results at batch 1

cuDNN NN FP32, 32, ms ms Custom

  • m INT8,

8, ms ms P40, ECC ON 70.0 20.7 V100, ECC ON 41.9 11.9

slide-43
SLIDE 43
slide-44
SLIDE 44
slide-45
SLIDE 45
slide-46
SLIDE 46