How Far are We from Effective Context Modeling? An Exploratory Study - - PowerPoint PPT Presentation

The 29th International Joint Conference on Artificial Intelligence (IJCAI-20)

How Far are We from Effective Context Modeling?

An Exploratory Study on Semantic Parsing in Context

Qian Liu, Bei Chen, Jiaqi Guo, Jian-Guang Lou, Bin Zhou, Dongmei Zhang

How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context

Semantic Parsing

CARS_NAMES MakeId Model Make CARS_DATA Id MPG Horsepower

SELECT Id FROM CARS_DATA ORDER BY Horsepower DESC LIMIT 1

What is id of the car with the max horsepower? Semantic parsing aims to translate a natural language sentence into its corresponding executable programming language, which relieves users from the burden of learning techniques behind the programming language.

How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context

Semantic Parsing in Context

SELECT Id FROM CARS_DATA ORDER BY Horsepower DESC LIMIT 1

What is id of the car with the max horsepower?

SELECT Id FROM CARS_DATA ORDER BY MPG DESC LIMIT 1

How about with the max mpg? CARS_NAMES MakeId Model Make CARS_DATA Id MPG Horsepower Users prefer to interact with systems in a dialogue, where users are allowed to ask context-dependent questions, which arises the task of Semantic Parsing in Context (SPC).

How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context

Semantic Parsing in Context

SELECT Id FROM CARS_DATA ORDER BY Horsepower DESC LIMIT 1

What is id of the car with the max horsepower?

SELECT Id FROM CARS_DATA ORDER BY MPG DESC LIMIT 1

How about with the max mpg? Show its Make!

SELECT T1.Make FROM CAR_NAMES AS T1 JOIN CARS_DATA AS T2 ON T1.MakeId = T2.Id ORDER BY T2.MPG DESC LIMIT 1

Recent Questions as Context Precedent SQL as Context

How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context

Grammar-based Semantic Parser

→ → → → →

…

→ → Attention

Encoder Encoder Encoder

…

Encoder

…

Encoder

SELECT Id FROM CARS_DATA ORDER BY Horsepower DESC LIMIT 1 Ques SQL Action

Start→Root, Root→Select Order, Select→Agg, Agg→max Col Tab, Col→ Id, Tab→CARS DATA, Order→desc limit Agg, Agg→none Col Tab, Col→Horsepower, Tab→CARS DATA

Abstract Syntax Tree

How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context

Recent Questions as Context

CONCAT TURN [Suhr et al., 2018] GATE [Zhang et al., 2018]

Recent ℎ questions 𝐲𝑗−ℎ, 𝐲𝑗−ℎ+1, … , 𝐲𝑗−1 & Current NL question 𝐲𝑗 Corresponding SQL 𝐳𝑗 Input Output

How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context

Precedent SQL as Context

Precedent SQL 𝐳𝑗−1 & Current NL question 𝐲𝑗 Corresponding SQL 𝐳𝑗 Input Output ACTION COPY [Zhang et al., 2019] TREE COPY [Suhr et al., 2018] SQL Attn

How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context

Precedent SQL as Context

Precedent SQL 𝐳𝑗−1 & Current NL question 𝐲𝑗 Corresponding SQL 𝐳𝑗 Input Output

SELECT Id FROM CARS_DATA ORDER BY Horsepower DESC LIMIT 1 SELECT Id FROM CARS_DATA ORDER BY MPG DESC LIMIT 1

Large Overlap

How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context

Precedent SQL as Context

Precedent SQL 𝐳𝑗−1 & Current NL question 𝐲𝑗 Corresponding SQL 𝐳𝑗 Input Output ACTION COPY [Zhang et al., 2019] TREE COPY [Suhr et al., 2018] SQL Attn

How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context

Research Question

How Far are We from Effective Context Modeling?

🤕

How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context

Experiments

• Dataset [Complex, Multi-Table, Cross-Domain]
• SParC [Yu et al., 2019b]: Train/Dev = 3034/422 dialogues, average turn number is 3.0
• CoSQL [Yu et al., 2019a]: Train/Dev = 2164/292 dialogues, average turn number is 5.2
• Metrics
• SQL Exact Set Match: Question Match, Interaction Match, Turn 𝑗 Match

CONCAT model V.S. SOTA methods ?

Comparison of different context modeling methods ? Context models for different contextual phenomena ?

How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context

CONCAT model V.S. SOTA methods

• Our CONCAT model outperforms baselines by a large margin, achieving

new SOTA performances on both datasets.

✓ SyntaxSQL-con [Yu et al., 2018] is analogous to TURN ✓ CD-Seq2Seq [Suhr et al., 2018] is analogous to TURN + TREE COPY ✓ EditSQL [Zhang et al., 2019] is analogous to TURN + SQL ATTN + ACTION COPY

How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context

Comparison of Different Methods

• None of the 13 methods can be

consistently superior to the others. ✓ Recent questions as context ✓ Precedent SQL as context ✓ Combination methods

13 Context Modeling Methods

• We need more effective context models.

✓ Simple/naive methods work well (e.g. CONACT, TURN, and ACTION COPY)

How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context

Comparison on Contextual Phenomena

What is id of the car with the max horsepower? How about with the max mpg? Show its Make!

Ellipsis Coreference

How many cars in total ?

Semantically Complete

How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context

Comparison on Coreference

Bridging Anaphora Show the version number for all templates. What is the smallest value? Definite Noun Phrases Which country has a head of state named Beatrix? What languages are spoken in that country? One Anaphora Order the pets by age. How much does each one weigh? Demonstrative Pronoun Which students have pets? Of those, whose last name is smith? Possessive Determiner How many highschoolers are liked by someone else? What are their names?

• All methods perform relatively well on types:

✓ Bridging Anaphora ✓ Demonstrative Pronoun ✓ Possessive Determiner ✓ They refer more to the “Whole” Antecedent

• All methods struggle with types:

✓ Definite Noun Phrases ✓ One Anaphora ✓ They refer more to the “Partial” Antecedent

How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context

Comparison on Ellipsis

• Performance Continuation > Substitution
• For Substitution

✓ Performance Implicit > Explicit ✓ Performance Operator > Schema

Continuation What are all the flight numbers? Which land in Aberdeen? Substitution Explicit What is id of the car with the max horsepower? How about with the max MPG? Implicit Find the names of museums opened before 2010. How about after? Schema How many losers participated in the Australian Open? Winners? Operator Who was the last student to register? Who was the first to register?

How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context

Conclusion How Far are We from Effective Context Modeling?

• A long way to go

✓ In the cross-domain setting, existing models are not as effective as expected.

• What we can do

✓ incorporating common sense for better pronouns inference ✓ modeling contextual clues in a more explicit manner ✓ debug models in a fine-grained level

Thanks

Paper: How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context Code: https://github.com/microsoft/ContextualSP We are hiring interns: di-recruit@microsoft.com