Learning Household Task Knowledge from WikiHow Descriptions Yilun - - PowerPoint PPT Presentation

▶

learning household task knowledge from wikihow

Learning Household Task Knowledge from WikiHow Descriptions Yilun - - PowerPoint PPT Presentation

Oct 10, 2023 481 likes •798 views

Learning Household Task Knowledge from WikiHow Descriptions Yilun Zhou, Julie A. Shah, Steven Schockaert Introduction Learning Household Task Knowledge from WikiHow Descriptions. Y. Zhou, J. A. Shah, S. Schockaert Introduction Learning

slide-1

SLIDE 1

Learning Household Task Knowledge from WikiHow Descriptions

Yilun Zhou, Julie A. Shah, Steven Schockaert

slide-2

SLIDE 2

Learning Household Task Knowledge from WikiHow Descriptions. Y. Zhou, J. A. Shah, S. Schockaert

Introduction

slide-3

SLIDE 3

Learning Household Task Knowledge from WikiHow Descriptions. Y. Zhou, J. A. Shah, S. Schockaert

Introduction

slide-4

SLIDE 4

Learning Household Task Knowledge from WikiHow Descriptions. Y. Zhou, J. A. Shah, S. Schockaert

Introduction

Can you bring me the bread and milk?

slide-5

SLIDE 5

Learning Household Task Knowledge from WikiHow Descriptions. Y. Zhou, J. A. Shah, S. Schockaert

Introduction

Can you bring me the scissors?

slide-6

SLIDE 6

Learning Household Task Knowledge from WikiHow Descriptions. Y. Zhou, J. A. Shah, S. Schockaert

Introduction

Can you make a cup of coffee for me?

slide-7

SLIDE 7

Learning Household Task Knowledge from WikiHow Descriptions. Y. Zhou, J. A. Shah, S. Schockaert

Related Work

Knowledge representation

○ ConceptNet [Speer, Chin, and Havasi] ○ KnowRob

Script knowledge learning
WikiHow mining

slide-8

SLIDE 8

Learning Household Task Knowledge from WikiHow Descriptions. Y. Zhou, J. A. Shah, S. Schockaert

Related Work

Knowledge representation

○ ConceptNet ○ KnowRob [Tenorth and Beetz]

Script knowledge learning
WikiHow mining

slide-9

SLIDE 9

Learning Household Task Knowledge from WikiHow Descriptions. Y. Zhou, J. A. Shah, S. Schockaert

Related Work

Knowledge representation
Script knowledge learning

○ [Chambers and Jurafsky] ○ [Modi and Tidov] ○ [Pichotta and Mooney]

WikiHow mining

slide-10

SLIDE 10

Learning Household Task Knowledge from WikiHow Descriptions. Y. Zhou, J. A. Shah, S. Schockaert

Related Work

Knowledge representation
Script knowledge learning
WikiHow mining

○ [Chu, Tandon, and Weikum] ○ [Park and Nezhad]

slide-11

SLIDE 11

Learning Household Task Knowledge from WikiHow Descriptions. Y. Zhou, J. A. Shah, S. Schockaert

Related Work

Knowledge representation
Script knowledge learning
WikiHow mining

○ [Chu, Tandon, and Weikum] ○ [Park and Nezhad]

slide-12

SLIDE 12

Learning Household Task Knowledge from WikiHow Descriptions. Y. Zhou, J. A. Shah, S. Schockaert

Dataset Description

slide-13

SLIDE 13

Learning Household Task Knowledge from WikiHow Descriptions. Y. Zhou, J. A. Shah, S. Schockaert

Dataset Description

slide-14

SLIDE 14

Learning Household Task Knowledge from WikiHow Descriptions. Y. Zhou, J. A. Shah, S. Schockaert

Dataset Description

slide-15

SLIDE 15

Learning Household Task Knowledge from WikiHow Descriptions. Y. Zhou, J. A. Shah, S. Schockaert

Dataset Description

Title

Step gist Step explanation

slide-16

SLIDE 16

Learning Household Task Knowledge from WikiHow Descriptions. Y. Zhou, J. A. Shah, S. Schockaert

Dataset Description

slide-17

SLIDE 17

Learning Household Task Knowledge from WikiHow Descriptions. Y. Zhou, J. A. Shah, S. Schockaert

Dataset Description

slide-18

SLIDE 18

Learning Household Task Knowledge from WikiHow Descriptions. Y. Zhou, J. A. Shah, S. Schockaert

Dataset Description

slide-19

SLIDE 19

Learning Household Task Knowledge from WikiHow Descriptions. Y. Zhou, J. A. Shah, S. Schockaert

Architecture

Text A Embedding A Prediction task 1 Prediction task 2 Text B Embedding B

slide-20

SLIDE 20

Learning Household Task Knowledge from WikiHow Descriptions. Y. Zhou, J. A. Shah, S. Schockaert

Architecture

Text A Embedding A Prediction task 1 Prediction task 2 Text B Embedding B

slide-21

SLIDE 21

Learning Household Task Knowledge from WikiHow Descriptions. Y. Zhou, J. A. Shah, S. Schockaert

Architecture

Text A Embedding A Prediction task 1 Prediction task 2 Text B Embedding B

slide-22

SLIDE 22

Learning Household Task Knowledge from WikiHow Descriptions. Y. Zhou, J. A. Shah, S. Schockaert

Architecture

Text A Embedding A Prediction task 1 Prediction task 2 Text B Embedding B

slide-23

SLIDE 23

Learning Household Task Knowledge from WikiHow Descriptions. Y. Zhou, J. A. Shah, S. Schockaert

Results

slide-24

SLIDE 24

Learning Household Task Knowledge from WikiHow Descriptions. Y. Zhou, J. A. Shah, S. Schockaert

Ambiguous Ordering

slide-25

SLIDE 25

Learning Household Task Knowledge from WikiHow Descriptions. Y. Zhou, J. A. Shah, S. Schockaert

xij = 1 represents that step i occurs strictly before step j
wij = log Pr(i before j)

Integer Programming for Ordering

slide-26

SLIDE 26

Learning Household Task Knowledge from WikiHow Descriptions. Y. Zhou, J. A. Shah, S. Schockaert

Integer Programming for Ordering

slide-27

SLIDE 27

Learning Household Task Knowledge from WikiHow Descriptions. Y. Zhou, J. A. Shah, S. Schockaert

Embedding Visualization

slide-28

SLIDE 28

Learning Household Task Knowledge from WikiHow Descriptions. Y. Zhou, J. A. Shah, S. Schockaert

Embedding Visualization

slide-29

SLIDE 29

Learning Household Task Knowledge from WikiHow Descriptions. Y. Zhou, J. A. Shah, S. Schockaert

Conclusion and Future work

Learn embedding representation from unstructured natural language
Integer programming for order inference

slide-30

SLIDE 30

Learning Household Task Knowledge from WikiHow Descriptions. Y. Zhou, J. A. Shah, S. Schockaert

Conclusion and Future work

Learn embedding representation from unstructured natural language
Integer programming for order inference
Consider more problems to learn jointly

○ Predict if a certain tool is needed for a certain task ○ Predict if two steps are from two tasks are equivalent

Text grounding

○ Map from steps/tasks to images/videos ○ Generate 3D trajectories for step execution