CMP722 ADVANCED COMPUTER VISION Lecture #5 Language and Vision - PowerPoint PPT Presentation

Representing meaning Is there a red shape above a circle? 57

Representing meaning Is there a red shape above a circle? 58

Sets encode meaning Is there a red shape above a circle? 59

Sets encode meaning Is there a red shape above a circle? 60

Set transformations encode meaning Is there a red shape above a circle? 61

Set transformations encode meaning Is there a red shape above a circle? 62

Sentence meanings are computations Is there a red shape above a circle? exists and red above circle 63

Sentence meanings are computations Is there a red shape above a circle? exists and red above circle 64

Computations are built from set functions exists ↦ red and ↦ true exists ↦ red above above circle 65

…or relaxed to vector functions 0.0 0.9 1.0 exists ↦ red and ↦ true exists ↦ red above above circle 66

Composing vector functions exists ↦ red and ↦ true exists ↦ red above above circle 67

Composing vector functions exists ↦ red and ↦ true exists ↦ red above above circle 68

Composing vector functions ↦ exists red and ↦ true red above exists ↦ circle above 69

Compositions of vector functions are neural nets ↦ ↦ true ↦ 70

Compositions of vector functions are neural nets ↦ exists red and ↦ true red above exists ↦ circle above 71

Outline yes Is there a red shape above a circle? exists and red above ↦ circle red ↦ true exists ↦ above 72

Outline yes Is there a red shape above a circle? exists and red above ↦ circle red ↦ true exists ↦ above 73

Outline yes Is there a red shape above a circle? exists and red above ↦ circle red ↦ true exists ↦ above 74

Outline yes Is there a red shape above a circle? exists and red above ↦ circle red ↦ true exists ↦ above 75

Outline yes Is there a red shape above a circle? exists and red above ↦ circle red ↦ true exists ↦ above 76

Anatomy of a module above 77

Anatomy of a module red color 78

The [find] module red 79

The [find] module necktie [Xu et al. 2015] 80

The [find] module 0.9 Columbia 0.1 Cooper 0.8 Myrtle Beach city nam ame typ ype coast co astal al Co Columbia ci city no no Cooper Co rive ver yes ye Myr yrtle ci city ye yes Beach ach 81

The [find] module red 82

The [find] module red 83

The [find] module red red 84

The [find] module 0.9 red red 85

The [find] module 0.9 red red 86

The [find] module 0.1 red red 87

The [describe] module red color 88

The [describe] module necktie what 89

The [describe] module red color 90

The [describe] module red . . . color 91

The [describe] module red . . . color 92

What modules do we need? Is there a red shape above a circle? What color is the triangle? Who is running in the grass? What cities are south of San Diego? 93

A module for predicates Is there a red shape above a circle? [find] What color is the triangle? Who is running in the grass? What cities are south of San Diego? 94

A module for relations Is there a red shape above a circle? [relate] What color is the triangle? Who is running in the grass? What cities are south of San Diego? [find] 95

Module inventory Is there a red shape above a circle? What color is the triangle? Who is running in the grass? [and] [describe] What cities are south of San Diego? [find] [relate] [exists] 96

Outline yes Is there a red shape above a circle? exists and red above ↦ circle red ↦ true exists ↦ above 97

Learning yes blue exists color and right_of red above circle circle What color is the shape right of a Is there a red shape above a circle? circle? 98

Learning yes blue What color is the shape right of a Is there a red shape above a circle? circle? 99

Parameter tying yes blue circle circle What color is the shape right of a Is there a red shape above a circle? circle? 100