Experiment presentation by Nayan Singhal Motivation Human - - PowerPoint PPT Presentation

▶
experiment presentation by nayan singhal motivation human
SMART_READER_LITE
LIVE PREVIEW

Experiment presentation by Nayan Singhal Motivation Human - - PowerPoint PPT Presentation

Jul 09, 2023 377 likes •880 views

Situation Recognition: Visual Semantic Role Labelling for Image Understanding Mark Yatskar, Luke Zettlemoyer, Ali Farhadi Experiment presentation by Nayan Singhal Motivation Human understanding of image Verbs in English language.

slide-1
SLIDE 1

Situation Recognition: Visual Semantic Role Labelling for Image Understanding

Mark Yatskar, Luke Zettlemoyer, Ali Farhadi

Experiment presentation by Nayan Singhal

slide-2
SLIDE 2

Motivation

  • Human understanding of image
  • Verbs in English language.
slide-3
SLIDE 3

Approach

  • CRF with CNN
  • Log linear Loss

CNN CRF

1024

CARRYING AGENT WOMAN ITEM

JAR

AGENTPART HEAD PLACE OUTDOOR

slide-4
SLIDE 4

How object plays role in image understanding?

slide-5
SLIDE 5

neighboring images

slide-6
SLIDE 6

Remove Cliff

neighboring images Removing Cliff

slide-7
SLIDE 7

Remove person

neighboring images Removing Man

slide-8
SLIDE 8

Remove Sky

neighboring images Removing Sky

slide-9
SLIDE 9

Image (2)

neighboring images

slide-10
SLIDE 10

Remove Person

neighboring images Removing man

slide-11
SLIDE 11

Remove Background

neighboring images Removing Sky and Man

slide-12
SLIDE 12

Conclusion

Each object plays a significant role in image understanding.

slide-13
SLIDE 13

Experiment

1) Analyzing Failure Cases 2) Different moods of faces

slide-14
SLIDE 14

Expt 1: Analyzing Failure Cases

slide-15
SLIDE 15

Object Recognition (1)

Imsitu Result

slide-16
SLIDE 16

Object Recognition (2)

Imsitu Result

slide-17
SLIDE 17

Object Recognition (3)

Imsitu Result

slide-18
SLIDE 18

Object Recognition (4)

Imsitu Result

slide-19
SLIDE 19

Object Recognition (5)

Imsitu Result

slide-20
SLIDE 20

Object Recognition (6)

Imsitu Result

slide-21
SLIDE 21

Why is it happening? Are these images difficult to categorize?

slide-22
SLIDE 22

Let’s analyze these with ImageNet

slide-23
SLIDE 23

Object Recognition (1)

Imsitu Result

slide-24
SLIDE 24

ImageNet classification

slide-25
SLIDE 25

Object Recognition (2)

Imsitu Result

slide-26
SLIDE 26

Object Recognition (2)

ImageNet classification

slide-27
SLIDE 27

Object Recognition (3)

Imsitu Result

slide-28
SLIDE 28

Object Recognition (3)

ImageNet classification

slide-29
SLIDE 29

Object Recognition (4)

Imsitu Result

slide-30
SLIDE 30

Object Recognition (4)

ImageNet classification

slide-31
SLIDE 31

Object Recognition (5)

Imsitu Result

slide-32
SLIDE 32

Object Recognition (5)

ImageNet classification

slide-33
SLIDE 33

Object Recognition (6)

Imsitu Result

slide-34
SLIDE 34

Object Recognition (6)

ImageNet classification

slide-35
SLIDE 35

Object Recognition (1)

Verb Role Noun Potential Labels

A (Verb Role Noun Potential) + B (Labels)

Post Processing: Slot Noun

slide-36
SLIDE 36

Object Recognition (1)

Imsitu Result

slide-37
SLIDE 37

Object Recognition (1)

VGG Verb Potential Verb Role Noun Potential Imagenet Labels

Preprocessing

slide-38
SLIDE 38

Future Work

  • Add labels in preprocessing.
slide-39
SLIDE 39

Exp 2: Different moods

  • Laughing
  • Smiling
  • Frowning
  • Grimacing
  • Winking
  • Squinting
  • Shouting
  • Puckering

Laughing Smiling Frowning Puckering Squinting Winking

slide-40
SLIDE 40

Success Case

Smiling Agent place man

  • 0.35967

Laughing Agent place man

  • 0.35777

Shouting Agent place man

  • 0.37531
slide-41
SLIDE 41

Success Case

Frowning Agent place man

  • 0.24378

Grimacing Agent place man

  • 0.21052
slide-42
SLIDE 42

Failure Case

Winking Agent place man

  • 0.20954

Puckering Agent place woman

  • 0.21052
slide-43
SLIDE 43

Test Images (25)

  • Conclusion: Detect different moods of faces with slight

variation.

slide-44
SLIDE 44

Some Interest Categorization

slide-45
SLIDE 45

Some Interest Categorization

Camouflaging Agent frog Hiding Item pebble Place

  • Camouflaging

Agent

  • wl

Hiding Item tree Place

  • utdoors
slide-46
SLIDE 46

neighboring images

slide-47
SLIDE 47

neighboring images

slide-48
SLIDE 48

Thank You

slide-49
SLIDE 49

No Agent(2)

Watering Agent Person Tool Bucket Place garden Shredding Agent Person Tool Shreder Item paper Place