Multi-Turn Response Selection for Chatbots with Deep Attention - - PowerPoint PPT Presentation

Multi-Turn Response Selection for Chatbots with Deep Attention Matching Network

𝑌𝑗𝑏𝑜𝑕𝑧𝑏𝑜𝑕 𝑎ℎ𝑝𝑣∗ , 𝑀𝑣 𝑀𝑗∗, 𝐸𝑏𝑦𝑗𝑏𝑜𝑕 𝐸𝑝𝑜𝑕, 𝑍𝑗 𝑀𝑗𝑣, 𝑍𝑗𝑜𝑕 𝐷ℎ𝑓𝑜, 𝑋𝑏𝑧𝑜𝑓 𝑌𝑗𝑜 𝑎ℎ𝑏𝑝, 𝐸𝑗𝑏𝑜ℎ𝑏𝑗 𝑍𝑣 and 𝐼𝑣𝑏 𝑋𝑣

Baidu – Natural Language Processing

Background

Chatbots

• Naturally and consistently converse with

human-beings on open-domain topics.

• Data-driven
• Retrieval-based method
• Generation-based method
• System ensemble

Context-Response Matching

𝑞(𝑗𝑡_𝑕𝑝𝑝𝑒|𝑑, 𝑠)

Cont

• ntext

xt 𝒅 Respons ponse 𝒔

Cont

• ntext

xt 𝒅

Response 𝑠

A

Response 𝑠

B

Response 𝑠

C

Response 𝑠

D

Dialogue Corpus Retrieval System

Retrieval-based Chatbot

input Search Rank

𝑞(𝑗𝑡_𝑕𝑝𝑝𝑒|𝑑, 𝑠)

Response 𝑠

A

Response 𝑠

C

Response 𝑠

D

Response 𝑠

B

An information retrieval approach to short text conversation. Ji et al., 2014

Adversarial Dialogue Generation

Dialogue Corpus

Generator 𝐻(𝑠∗|𝑑) Real Conversation Discriminator

𝑞(𝑗𝑡_𝑕𝑝𝑝𝑒|𝑑, 𝑠)

Sampled <c, r>

Adversarial Learning for Neural Dialogue Generation. Li et al., EMNLP-2017

Challenges

• Detecting truly matched segment pairs across context and

response.

• Segment pairs could be matched at different granularities.
• Segment pairs, across context and response, could be matched

because of textual relevance or semantic dependency. Cha hallenge nges

Conversation Context Speaker A: Hi I am looking to see what packages are installed on my system, I don’t see a path, is the list being held somewhere else? Speaker B: Try dpkg - get-selections Speaker A: What is that like? A database for packages instead of a flat file structure? Speaker B: dpkg is the debian package manager - get-selections simply shows you what packages are handed by it Response of Speaker A: No clue what do you need it for, its just reassurance as I don’t know the debian package manager

Conversation Context Speaker A: Hi I am looking to see what packages are installed on my system, I don’t see a path, is the list being held somewhere else? Speaker B: Try dpkg - get-selections Speaker A: What is that like? A database for packages instead of a flat file structure? Speaker B: dpkg is the debian package manager - get-selections simply shows you what packages are handed by it Response of Speaker A: No clue what do you need it for, its just reassurance as I don’t know the debian package manager

Matchi hing w ng with s h sur urface t text xt

Conversation Context Speaker A: Hi I am looking to see what packages are installed on my system, I don’t see a path, is the list being held somewhere else? Speaker B: Try dpkg - get-selections Speaker A: What is that like? A database for packages instead of a flat file structure? Speaker B: dpkg is the debian package manager - get-selections simply shows you what packages are handed by it Response of Speaker A: No clue what do you need it for, its just reassurance as I don’t know the debian package manager

Matchi hing w ng with de h depe pende ndenc ncy

Conversation Context Speaker A: Hi I am looking to see what packages are installed on my system, I don’t see a path, is the list being held somewhere else? Speaker B: Try dpkg - get-selections Speaker A: What is that like? A database for packages instead of a flat file structure? Speaker B: dpkg is the debian package manager - get-selections simply shows you what packages are handed by it Response of Speaker A: No clue what do you need it for, its just reassurance as I don’t know the debian package manager

Motivation

Use GRU/LSTM to encode segments and match context with response only considering textual relevance.

Previ vious

• us w

wor

• rks

ks

• Se

Self-Attent ntion

• n: Using intra-attention of utterance/response to

gradually construct multi-grained semantic representations.

• Cros
• ss-Attent

ntion

• n: Using attention across context and response to

match with dependency information.

Mot

• tiva

vation

• n

Conversation Context Speaker A: Hi I am looking to see what packages are installed on my system, I don’t see a path, is the list being held somewhere else? Speaker B: Try dpkg - get-selections Speaker A: What is that like? A database for packages instead of a flat file structure? Speaker B: dpkg is the debian package manager - get-selections simply shows you what packages are handed by it Response of Speaker A: No clue what do you need it for, its just reassurance as I don’t know the debian package manager

Conversation Context Speaker A: Hi I am looking to see what packages are installed on my system, I don’t see a path, is the list being held somewhere else? Speaker B: Try dpkg - get-selections Speaker A: What is that like? A database for packages instead of a flat file structure? Speaker B: dpkg is the debian package manager - get-selections simply shows you what packages are handed by it Response of Speaker A: No clue what do you need it for, its just reassurance as I don’t know the debian package manager

Conversation Context Speaker A: Hi I am looking to see what packages are installed on my system, I don’t see a path, is the list being held somewhere else? Speaker B: Try dpkg - get-selections Speaker A: What is that like? A database for packages instead of a flat file structure? Speaker B: dpkg is the debian package manager - get-selections simply shows you what packages are handed by it Response of Speaker A: No clue what do you need it for, its just reassurance as I don’t know the debian package manager

Conversation Context Speaker A: Hi I am looking to see what packages are installed on my system, I don’t see a path, is the list being held somewhere else? Speaker B: Try dpkg - get-selections Speaker A: What is that like? A database for packages instead of a flat file structure? Speaker B: dpkg is the debian package manager - get-selections simply shows you what packages are handed by it Response of Speaker A: No clue what do you need it for, its just reassurance as I don’t know the debian package manager

Conversation Context Speaker A: Hi I am looking to see what packages are installed on my system, I don’t see a path, is the list being held somewhere else? Speaker B: Try dpkg - get-selections Speaker A: What is that like? A database for packages instead of a flat file structure? Speaker B: dpkg is the debian package manager - get-selections simply shows you what packages are handed by it Response of Speaker A: No clue what do you need it for, its just reassurance as I don’t know the debian package manager

Conversation Context Speaker A: Hi I am looking to see what packages are installed on my system, I don’t see a path, is the list being held somewhere else? Speaker B: Try dpkg - get-selections Speaker A: What is that like? A database for packages instead of a flat file structure? Speaker B: dpkg is the debian package manager - get-selections simply shows you what packages are handed by it Response of Speaker A: No clue what do you need it for, its just reassurance as I don’t know the debian package manager

Conversation Context Speaker A: Hi I am looking to see what packages are installed on my system, I don’t see a path, is the list being held somewhere else? Speaker B: Try dpkg - get-selections Speaker A: What is that like? A database for packages instead of a flat file structure? Speaker B: dpkg is the debian package manager - get-selections simply shows you what packages are handed by it Response of Speaker A: No clue what do you need it for, its just reassurance as I don’t know the debian package manager

dkpg dkpg + ` ``it” it + + ` ``dpkg dpkg”

Conversation Context Speaker A: Hi I am looking to see what packages are installed on my system, I don’t see a path, is the list being held somewhere else? Speaker B: Try dpkg - get-selections Speaker A: What is that like? A database for packages instead of a flat file structure? Speaker B: dpkg is the debian package manager - get-selections simply shows you what packages are handed by it Response of Speaker A: No clue what do you need it for, its just reassurance as I don’t know the debian package manager

dkpg dkpg + ` ``it” it + + ` ``dpkg dpkg”

Attentive Module

que query Attent ntion

• n

Weight ghted d Sum um ke key va value ue Sum um & Nor

• rm

Fe Feed-For

• rward

Sum um & Nor

• rm
• Attent

ntive veModul

• dule(Q,K

,K,V ,V)

Vatt = softmax(Q · KT √dk )V

Qatt = LayerNorm(Vatt + Q)

FFN(Qatt) =max(0, QattW1 + b1)W2 + b2

LayerNorm(FFN(Qatt) + Qatt)

Input nput

Q = [e0, ..., enQ−1] V = [e0, ..., enV−1] K = [e0, ..., enK−1]

• Capture structures across Q and K-V
• Composite semantic representations of captured

structures with input embedding Sum ummary

Attention is All You Need. Vaswani et al., NIPS-2017

Wor

• rd

Embe beddi dding ng

𝒔 𝒗𝟐 𝒗𝒐 𝒗𝒋

Wor

• rd

Embe beddi dding ng

Repr present ntation

• n

Modul

• dule

𝒔 𝒗𝟐 𝒗𝒐 𝒗𝒋

Ui

R

Mul ulti-gr graine ned Repr present ntations

• ns

=            U0

i = ui,

... Ul

i = AttentiveModule(Ul−1 i

, Ul−1

i

, Ul−1

i

), ... UL

i = AttentiveModule(UL−1 i

, UL−1

i

, UL−1

i

)           

phr phrase-leve vel wor

• rd-leve

vel sent nt.-leve vel

=            R0 = r, ... Rl = AttentiveModule(Rl−1, Rl−1, Rl−1), ... RL = AttentiveModule(RL−1, RL−1, RL−1)           

Wor

• rd

Embe beddi dding ng

Repr present ntation

• n

Modul

• dule

𝒔

Wor

• rd-wor
• rd

d Matchi hing ng with C h Cros

• ss-Attent

ntion

• n

𝒗𝟐 𝒗𝒐 𝒗𝒋

Ui

R

Mul ulti-gr graine ned Repr present ntations

• ns

Mui,r

self

Mui,r

cross

Wor

• rd

Embe beddi dding ng

Repr present ntation

• n

Modul

• dule

𝒔

Wor

• rd-wor
• rd

d Matchi hing ng with C h Cros

• ss-Attent

ntion

• n

𝒗𝟐 𝒗𝒐 𝒗𝒋

Ui

R

Mul ulti-gr graine ned Repr present ntations

• ns

Mui,r

self

Mui,r

cross Se Self lf-Attent ntion M

• n Match

Mui,r,l

self = {Ul i[k]T · Rl[t]}nui×nr

Match w h with h text xtua ual i inf nfor

• rmation
• n

Wor

• rd

Embe beddi dding ng

Repr present ntation

• n

Modul

• dule

𝒔

Wor

• rd-wor
• rd

d Matchi hing ng with C h Cros

• ss-Attent

ntion

• n

𝒗𝟐 𝒗𝒐 𝒗𝒋

Ui

R

Mul ulti-gr graine ned Repr present ntations

• ns

Mui,r

self

Mui,r

cross Se Self lf-Attent ntion M

• n Match

Cros

• ss-Attent

ntion M

• n Match

Mui,r,l

self = {Ul i[k]T · Rl[t]}nui×nr

Match w h with t h text xtua ual i inf nfor

• rmation
• n

Match w h with h de depe pende ndenc ncy i y inf nfor

• rmation
• n

Mui,r,l

cross = { e

U

l i[k]T · e

R

l[t]}nui×nr

e U

l i = AttentiveModule(Ul i, Rl, Rl)

e R

l = AttentiveModule(Rl, Ul i, Ul i)

Wor

• rd

Embe beddi dding ng

Repr present ntation

• n

Modul

• dule

𝒔

Wor

• rd-wor
• rd

d Matchi hing ng with C h Cros

• ss-Attent

ntion

• n

𝒗𝟐 𝒗𝒐 𝒗𝒋

Matchi hing ng Scor

• re

g(c,r)

3D 3D M Matchi hing ng Image ge Q Q

Ui

R

Mul ulti-gr graine ned Repr present ntations

• ns

Mui,r

self

Mui,r

cross

Wor

• rd

Embe beddi dding ng

Repr present ntation

• n

Modul

• dule

𝒔

Wor

• rd-wor
• rd

d Matchi hing ng with C h Cros

• ss-Attent

ntion

• n

𝒗𝟐 𝒗𝒐 𝒗𝒋

Matchi hing ng Scor

• re

g(c,r)

3D 3D M Matchi hing ng Image ge Q Q

Ui

R

Mul ulti-gr graine ned Repr present ntations

• ns

Mui,r

self

Mui,r

cross

Mul ulti-gr graine ned matching, including both text xtua ual and de depe pende ndenc ncy information, of a segment pair across context and response

Pixe xel

Ubunt buntu Cor

• rpus

pus V1 Douba

• uban Conve
• nversation
• n

Cor

• rpus

pus Tr Train De Dev Te Test Tr Train De Dev Te Test # c # cont

• ntext

xt-respons ponse-pa pairs 1M 1M 50k 50k 50k 50k 1M 1M 50k 50k 10k 10k # c # candi ndida dates pe per c cont

• ntext

xt 2 10 10 10 10 2 2 10 10 # pos # positive ve c candi ndida dates pe per c cont

• ntext

xt 1 1 1 1.18 1.18

• Min. # t
• n. # tur

urns ns pe per c cont

• ntext

xt 3 3 3 3 Max.

• x. # t

# tur urns ns pe per c cont

• ntext

xt 19 19 98 98 91 91 45 45

• Avg. # t
• vg. # tur

urns ns pe per c cont

• ntext

xt 7.71 7.71 6.69 6.69 6.75 6.75 6.45 6.45

• Avg. #w
• vg. #wor
• rds

ds pe per ut utteranc nce 10.34 10.34 18.56 18.56 18.50 18.50 20.74 20.74

Experiment

• Ubuntu Corpus
• One-one multi-turn conversation
• Ubuntu troubleshooting
• Douban Corpus
• One-one multi-turn conversation
• Open domain topics
• Task
• Given multi-turn context and serval

response candidates

• Select the best candidate based on

matching score

Experiment

• DAM setup
• Test stacking 3-7 self-attention layers
• Comparison
• Seque

quent ntial M Matchi hing N ng Networ

• rk (

k (SMN) (Wu et al., ACL-2017), Mul ulti-vi view M Matchi hing ng (Zhou et al., EMNLP-2016), DL2R 2R (Yan et al., SIGIR-2016), Dua ualEnc ncode

• der (Rowe et al., SigDial-2015)
• Ablation
• 𝐸𝐵𝑁

LMNOP : without stacked self-attention

• 𝐸𝐵𝑁QROP : only using the last layer of stacked self-attention
• 𝐸𝐵𝑁OSQL : only using self-attention-match
• 𝐸𝐵𝑁TNUOO : only using cross-attention-match

L(·) = − X

(c,r,y)∈D

g(c, r)y + (1 − g(c, r))(1 − y)

Evaluation Results

no clue what do you need it for. its just reassurance as i dont know the debain package manager hi i am looking to see what packages are installed

• n

my system i.1 dont see.1 a path is the list being held somewhere else

response

self−attention−match in stack 0

response

no clue what do you need it for. its just reassurance as i dont know the debain package manager hi i am looking to see what packages are installed

• n

my system i.1 dont see.1 a path is the list being held somewhere else

turn 0

self−attention−match in stack 2

response

no clue what do you need it for. its just reassurance as i dont know the debain package manager hi i am looking to see what packages are installed

• n

my system i.1 dont see.1 a path is the list being held somewhere else

self−attention−match in stack 4

response

Self-Attention Match Visualization

Stack-0 Stack-2 Stack-4

Cross-Attention Match Visualization

no clue what do you need it for. its just reassurance as i dont know the debain package manager hi i am looking to see what packages are installed

• n

my system i.1 dont see.1 a path is the list being held somewhere else

turn 0 response

self−attention−match in stack 4

response

no clue what do you need it for. its just reassurance as i dont know the debain package manager hi i am looking to see what packages are installed

• n

my system i.1 dont see.1 a path is the list being held somewhere else

turn 0 response

cross−attention−match in stack 4

Self-Attention Match Cross-Attention Match

Summary

• We propose a novel deep attention matching network for multi-turn

response selection that entirely based on attention.

• We use stacked self-attention to construct multi-grained semantic

representations.

• We use cross-attention to match context with its candidate response

considering both textual and dependency information

Thanks

• Code & Data: https://github.com/baidu/Dialogue/DAM
• Our Authors