Intelligent Chatbot on WeChat WeChat AI NLP 2017.05.09 WeCh We - - PowerPoint PPT Presentation

intelligent chatbot on wechat
SMART_READER_LITE
LIVE PREVIEW

Intelligent Chatbot on WeChat WeChat AI NLP 2017.05.09 WeCh We - - PowerPoint PPT Presentation

Intelligent Chatbot on WeChat WeChat AI NLP 2017.05.09 WeCh We Chat is is the he le leading ding mob obil ile e so socia ial l ne network work in in Ch Chin ina. In In 6 6 ye years, rs, We WeCh Chat at has gained llion


slide-1
SLIDE 1

Intelligent Chatbot on WeChat

WeChat AI NLP 2017.05.09

slide-2
SLIDE 2

889 889 mi million llion mo

monthly nthly act active users ive users

600 m 00 mil illion lion We

WeChat Chat Pa Pay user y users

10 10 mill million ion Of

Offic ficial Acco ial Accounts unts

200 th thous usand and de

devel velopers

  • pers

We WeCh Chat is is the he le leading ding mob

  • bil

ile e so socia ial l ne network work in in Ch Chin

  • ina. In

In 6 6 ye years, rs, We WeCh Chat at has gained…

Data: : Tencent Financial Repor

  • rts
slide-3
SLIDE 3

3

50% of users spend more

than 1 hour on WeChat every day

Data: Penguin Intelligence

slide-4
SLIDE 4

WeChat is not just a mobile messaging app. It’s a new lifestyle, connecting people with people, services, devices and more.

WeChat Overview

The WeChat Lifestyle

slide-5
SLIDE 5

Red Pocket Jan 27 – Feb 01 46 Billion Emoji Jan 27 – Feb 01 16 Billion Voice Call Jan 27 – Jan 28 2.1 Billion minutes

Chinese New Year 2017

slide-6
SLIDE 6

6

The new way for businesses to interact with their customers.

Powered by WeChat

slide-7
SLIDE 7

7

Messaging (Can be automated) Account management

Service Accounts

China Merchant Bank case

China Merchants Bank

Over 10 million followers Open an account Pay bill/loan Receive payment notifications Receive CRM promotions

Powered by WeChat

slide-8
SLIDE 8

8

Messaging Account management

Service Accounts

China Southern Airlines case

China Southern Airlines

Buy Tickets Check-in Choose seats Flight status update Frequent flyer services

Powered by WeChat

slide-9
SLIDE 9

… 4.3

2012.10 Voice Search

4.5

2013.2 Voice Reminder Shake Music

5.0

2013.8 Voice Input Scan Cover/Word

5.2

2013.10 Voice to Text

5.3

2014.1 Shake TV

5.4

2014.6 Scan Cards Smart Open Platform

6.0

2014.12 Voice Print

6.2

2015.4 Data Mining

Now

Amber Platform BOTs Platform

WeChat AI

Growth in 6 years

slide-10
SLIDE 10

WeChat Amber Platform

Highlights Applications

  • Data/model parallelism
  • Flexible resource management and scheduling
  • Compile the graphs
  • Best-effort concurrent operations
  • Limited memory reuse
  • Consistent data streaming
  • Kernels merge
  • Machine Learning
  • Deep Learning
  • Data Mining

Experiments – Google Net

slide-11
SLIDE 11

Speech Recognition

Features Applications

  • End-to-end deep learning
  • Above 95% accuracy
  • Cloud based & embedded ASR
  • User defined vocabulary and

grammar

  • WeChat voice input
  • Keyword spotting
  • Speech retrieval
  • Large vocabulary continuou

s speech recognition

Infrastructure

  • Clusters+CUDA+MPI
  • Latency control with infiniband
  • Training with Tesla M40
  • Inference with Tesla P4
slide-12
SLIDE 12

Image Recognition

  • OCR
  • Identity Card Recognition

– Key personal information extraction and verification

  • Image Understanding

– Classify tens of millions of images daily – Supports 3 levels and 1,000 categories – CNN/RNN/LSTM – End-to-end deep learning

Applications Algorithms

slide-13
SLIDE 13

Chatbot Examples

  • WeBank
  • WeChat official account
  • Tencent games
  • Xiao‘er Mechanical Monk

Chatbot on WeChat

  • Natural to server customers
  • Powerful for users to acquire

service, information, knowledge, etc.

slide-14
SLIDE 14

Work Flow of Wechat Chatbot

Question

Question Parsing Question Understanding Output

Rule Match

QnA

Chitter Chat Model

Answer Ranking

Answer

Context

Answer Candidates

Knowledge Base

slide-15
SLIDE 15

Chatbot Architecture in Progress

Question

Question Parsing Question Understanding Output

Rule Match

QnA

Document Content

Chitter Chat Model

Answer Ranking

Answer Sentiment Analysis

Sentiment Analysis Output

Context

Answer Candidates Personalization Knowledge Graph

Under development

  • Sentiment analysis
  • Knowledge graph
  • Doc-chat
  • Personalization
  • Expose the platform to public
slide-16
SLIDE 16

Conversational Chatbot

How can be happy? Why I’m so busy?

slide-17
SLIDE 17

Hard Problems for Conversational Chatbot

Questio stion n Unde ders rstan tandi ding ng:

  • 干啥呐?(what are you doing?)
  • 干啥的?(what is your job?)
  • 你哪里好?(why you think you are good?)
  • 你在哪里? (where are you?)
  • 你师父呢?(where is your master?)
  • 师父在忙

(master is busy)

  • 他在忙啥? (what is he

he doing?)

  • 闻何法啊? (how do you practice Dharma?)
  • 破除我执

(being not obsessive)

  • 如何破除呢? (how

how?) Knowle wledg dge e Re Repr present esentatio tion: n:

  • Notaria

ial l ce certifi ficat cates es, executed in the ma mainla land nd, and to be used in Hong g Kong g Spe peci cial l Adm dministr istrative tive Re Regi gion

  • n, shall be acknowledged by

the Consu sular lar Depa partment ment of th f the Minis istry ry of f Fo Foreig eign Aff ffairs rs of the People's Republic of China

  • 转心 (transform the heart),就是心里要去拿起一

个正确的东西,否则心在烦恼(affliction)中时是 很难转动的。要不断培养自己的发心(bodhicitta- samutpada) ,让它越来越宽广,越来越清净, 烦恼自然就越来越少。恨(hatred)也好,念 (obsession)也好,都是妄想(delusion) ,消耗心力、 迷障未来。 Answ swer er Genera eratio tion: n: avoid trivial and boring answers

  • 忙呢

(busy now)

  • 你忙

(take your time)

  • 再见

(see you later)

  • 狗狗很可爱

(dogs are cute) 是很可爱 (yes, they are cute)

slide-18
SLIDE 18

Sentence Modeling by Recurrent Neural Network

x0 x1 x2 x3 xn

Embedding Layer

V0 V1 V2 V3

Vn

h0 h1 h2 hn h3

V0 V1 V2 V3 V4 V5 V6 V7 V8 x0 x1 x2 x3 x8

Embedding Layer

x4 x5 x6 x7

slide-19
SLIDE 19

Anaphora Resolution

Inpu put: q: q: cu current ent qu query c: c: co contenxt nxt Ou Outpu put: q': current query after anaphora resolution H: H: replace pronouns in the current query with noun phrases in the context About 5% of the total queries Examples: C1: 你是陈奕迅粉丝吗? (are you a fan of Eason Chan? ) C2: 更喜欢张学友 (I like Jacky Cheung more) q : 为什么更喜欢他? (Why like him more?) q ‘: 为什么更喜欢张学友 (Why like Jacky Cheung more?)

q'= = H(q,C)

C1 : 你住哪儿? (where do you live? ) C2 : 不二寺。 (Bu’er Temple ) q : 那在哪儿? (Where is it? ) q ‘ : 不二寺在哪儿? (Where is Bu’er Temple ? )

slide-20
SLIDE 20

模型建立 代消解

Context Query

陈奕迅 粉丝 更 喜欢 张学友

为什么 更 喜欢 他

) | (

m ax

为什么更喜欢他 张学友 P P 

“他”(him) “张学友”(Jacky Chueng) q' = 为什么更喜欢张学友

RNN for Anaphora Resolution

Example: C1: 你是陈奕迅粉丝吗? C2: 更喜欢张学友 q : 为什么更喜欢他? q ‘: 为什么更喜欢张学友

  • 100K training data
  • Accuracy: 90%
  • Majority of the errors are

caused by the mistakes of entity tagging A bad case: C1: 你认识贤三吗? C2: 当然认识。 q : 他是你什么人? q ': 三是你什么人?

slide-21
SLIDE 21

Query Complement

Inpu put: q: current query c: context Ou Outpu put: q': current query after query complement H: H: complete the current query with information in the context About 15% of the total queries Examples: C1: 那你会发表情包吗? (can you send emojis? ) C2: 一般不发 (usually I don’t send emojis) q :为什么? (Why?) q ‘: 为什么不发表情包 (Why not send emojis?)

q'= = H(q,C)

C1 :讲个故事给我听 (tell me a story ) C2 :等我学会了给你讲哦 。 (I’ll tell you a story once I learn how to) q :我等着 (I’m waiting) q ‘ :我等着听故事 (I’m waiting for the story)

slide-22
SLIDE 22

模型建立 代消解

RNN for Query Completiontt

Training Sample: C1:讲个故事给我听 C2:等我学会了给你讲哦 。 q :我等着 q ‘:我等着听故事

  • 100,000 training instances
  • Accuracy: 70%
  • Increased the engagement of

Xian’er Mechanical Monk by 11%

我 等 着 听 故 我 等 着 听

讲 个 故 事 给 我 听 _E_ 等 ...

... ... x y

slide-23
SLIDE 23

部分结果展示

你去问问师父喜欢你吗 不会的,问你师父去 什么时候问必要

Query Completion Results in Real Dialogs

Does your master like you? Need to ask him ask Ask your master if he likes you

slide-24
SLIDE 24

部分结果展示

Sentence Similarity Computation

Unsupervised word embedding approach is not good enough

Sentence 0 Sentence 1

Similarity based

  • n Word

Embedding

Similar Enough? 你是谁 (who are you) 我是谁 (who am I) 0.93 No 我爱你 (I love you) 你爱我 (you love me) 0.89 No 吃饭了吗 (Do you have lunch?) 吃饭了 (just had lunch) 0.84 No 你干嘛的 (what is your job?) 你干嘛呢 (who are you doing?) 0.93 No 有轮回吗? (Is

reincarnation true?)

轮回有结束吗 (will the cycle of

life end?)

0.73 No 会不会轮回 (will reincarnation

happen?)

会不会轮回结束 (Will

reincarnation end?)

0.84 No 随喜您 (you did it well) 您做的很好 (you did it well) 0.20 Yes

slide-25
SLIDE 25

Supervised Learning for Sentence Similarity Feature Embedding Model

  • Sentence features

unigrams bi-grams

  • Comparison Features

word pairs from two sentences each edit operations 什么 含义 vs. s. 什么 意思

match-什么-什么 replace-含义-意思

RNN for sentence similarity

Question 0 Question 1

slide-26
SLIDE 26

Sentence Similarity Results

Models Models Accurac acy Unsuper ervi vised ed word embeddi ding 0.63 0.63 RNN RNN + c cosine similar arit ity 0.65 0.65 RNN + M MLP 0.6878 0.6878 CNN + M MLP 0.6968 0.6968 RNN + T Tensor 0.728 0.728 Feature re Embeddi ding ng 0.75 0.75

220,000 sentence pairs for training 20,000 for testing

slide-27
SLIDE 27

Response Generation

  • Generative model is used if no match from knowledge base
  • Neural Network based methods for response generation

28

  • Motivated by neural network based methods for translation

One sentence in Language A One sentence in Language B Input Sentence Response Sentence

Translation

Response Generation

slide-28
SLIDE 28

Neural Network based Methods for Response Generation

  • Motivated by neural network based methods for translation

29

Training data: Objective:

slide-29
SLIDE 29

NN based Methods for Response Generation

30

slide-30
SLIDE 30

Dialogue vs. Translation

31

  • Dialogue corpus is different from translation corpus
  • The response diversity problem exists in dialogue

corpus

slide-31
SLIDE 31

Diversity

32 For question: What's up?

The normal I am OK. I am fine.

  • Mr. Shelton

Bazinga!

  • Mr. Trump

You are fired!

slide-32
SLIDE 32
  • In our experimental corpus, more than 60 different responses

exist to the post “You are so silly”

  • No!
  • You are!
  • Why?
  • Don’t say that
  • Many different responses usually correspond to the same post

Diversity

slide-33
SLIDE 33

Issues on Response Diversity

  • Only return the safe and generic answers, i.e. the one with the

highest probability

  • Cannot recognize good but low probability answers

34

Responses with high probabilities

Good responses, but occur not frequently Bad responses

slide-34
SLIDE 34

Response-Style Modeling

36

slide-35
SLIDE 35

37

A diverter is developed to generate the mechanism distribution of an input post

Encoder-Diverter-Decoder

slide-36
SLIDE 36

39

  • Training
  • 815, 852 pairs of post and response
  • 775, 852 are for training,
  • 40, 000 are for model validation.
  • T

esting

  • We randomly select 300 posts from about 15 million posts
  • Every baseline model generates 5 response
  • Use human judgment the evaluate the model performance

Experiment

slide-37
SLIDE 37

41

the diversity of the response is increased by 1.7 times, and the accuracy is increased by 9.8%

Experiment Results

slide-38
SLIDE 38

42

Example Output

slide-39
SLIDE 39

Future Work

  • Making use of more knowledge sources

knowledge graph article content

  • Unsupervised machine learning
  • Open the service to the public

knowledge management model tuning chatbot customization

43

slide-40
SLIDE 40

Voice & Audio Natural Language Processing Machine Learning Image & Video

WeChat AI are hiring now!

AI@tencent.com niucheng@tencent.com Beijing, Guangzhou, Shenzhen, Palo Alto

Machine Translation

/通用格式 /通用格式 /通用格式 /通用格式 /通用格式

/通用格式 /通用格式 /通用格式 /通用格式 /通用格式

samples/sec

batch_size

speed, 4 gpus

amber mxnet tf

slide-41
SLIDE 41

Thanks

WeC eChat hat A.I .I. . NLP LP