Zero-shot Task Transfer Vineeth N Balasubramanian Dept of Computer - - PowerPoint PPT Presentation

▶
zero shot task transfer
SMART_READER_LITE
LIVE PREVIEW

Zero-shot Task Transfer Vineeth N Balasubramanian Dept of Computer - - PowerPoint PPT Presentation

Nov 29, 2023 364 likes •717 views

Zero-shot Task Transfer Vineeth N Balasubramanian Dept of Computer Science & Engineering Indian Institute of Technology, Hyderabad (Joint work with Arghya Pal, PhD student) CVPR 2019 (Oral) Our Groups Research Grad-CAM++: Generalized

slide-1
SLIDE 1

Zero-shot Task Transfer

Vineeth N Balasubramanian Dept of Computer Science & Engineering Indian Institute of Technology, Hyderabad (Joint work with Arghya Pal, PhD student) CVPR 2019 (Oral)

slide-2
SLIDE 2

Our Group’s Research

slide-3
SLIDE 3

Grad-CAM++: Generalized Visual Explanations

Chattopadhyay, Sarkar, Howlader, Balasubramanian, WACV 2018

slide-4
SLIDE 4

Grad-CAM++: Generalized Visual Explanations

Chattopadhyay, Sarkar, Howlader, Balasubramanian, WACV 2018

slide-5
SLIDE 5

Grad-CAM++: Generalized Visual Explanations

Chattopadhyay, Sarkar, Howlader, Balasubramanian, WACV 2018

slide-6
SLIDE 6

Causal NN Attributions

Chattopadhyay, Manupriya, Sarkar, Balasubramanian, arXiv 2019

Neural network as a SCM

slide-7
SLIDE 7

Causal NN Attributions

Chattopadhyay, Manupriya, Sarkar, Balasubramanian, arXiv 2019

slide-8
SLIDE 8

Causal NN Attributions

Chattopadhyay, Manupriya, Sarkar, Balasubramanian, arXiv 2019

slide-9
SLIDE 9

Zero-shot Task Transfer

Zero-shot Task Transfer

slide-10
SLIDE 10

Tasks

❖ Vision tasks:

■

■ Object recognition ■ Depth ■ Edge detection ■ Pose estimation ■ ...

Zamir et al., CVPR 2018

slide-11
SLIDE 11

Tasks

❖ Relation among vision tasks

Zamir et al., CVPR 2018

slide-12
SLIDE 12

Tasks

❖ Taskonomy CVPR 2018 (Best Paper)

➢ 26 Vision tasks ➢ Sampled set of tasks and not an exhaustive list

Zamir et al., CVPR 2018

slide-13
SLIDE 13

Tasks

Vision tasks are often related to each other. How to leverage?

Key Takeaway

slide-14
SLIDE 14

Zero-shot Task Transfer

Zero-shot Task Transfer

slide-15
SLIDE 15

Zero-shot Classification: A Review

❖ Object recognition for a set of categories for which we have no training examples ➢ 𝓩 = {y1, y2, … , ym} classes with training samples ➢ 𝓪 = {z1, z2, … , zn} classes with no training samples ➢ Learn a classification model: H : 𝓨 → (𝓪 union 𝓩)

slide-16
SLIDE 16

Zero-shot Classification: A Review

❖ For each class z ϵ 𝓪 and y ϵ 𝓩:

➢ attribute representations az , ay ϵ 𝓑 are available

slide-17
SLIDE 17

Tasks

Vision tasks are often related to each other

Zero-shot classification

If relation exists among classes, new classes can be detected based on attribute representation without the need for a new training phase / ground truth

Key Takeaway

slide-18
SLIDE 18

Zero-shot Task Transfer: Motivation

  • Vision tasks:

○ Expensive ○ May require special sensors ○ Lesser amounts of labeled data leads to poorly performing models

Pal, Balasubramanian, Zero-shot Task Transfer, CVPR 2019

zero-shot classification → zero-shot task transfer

slide-19
SLIDE 19

Zero-shot Task Transfer

  • Consider K tasks, i.e. 𝓤 = {𝓤1, 𝓤2, … , 𝓤K}
  • Model parameters lie on a meta-manifold ℳθ
  • On meta manifold; Task 𝓤 is equivalent to model parameter θ

Pal, Balasubramanian, Zero-shot Task Transfer, CVPR 2019

slide-20
SLIDE 20

Zero-shot Task Transfer

  • Ground truth available for first m tasks

○ 𝓤known = {𝓤1, 𝓤2, … , 𝓤m} ○ Corresponding model parameters, {θ 𝓤 i : i = 1, … , m}, on meta manifold ℳ known

  • No knowledge of ground truth for the zero-shot tasks

○ 𝓤zero = {𝓤(m+1), 𝓤(m+2), … , 𝓤K}

Pal, Balasubramanian, Zero-shot Task Transfer, CVPR 2019

slide-21
SLIDE 21

Zero-shot Task Transfer: Idea

○ Learn a meta-learning function Fw (·) ○ Fw (·) regresses unknown zero-shot model parameters from known model parameters

Pal, Balasubramanian, Zero-shot Task Transfer, CVPR 2019

slide-22
SLIDE 22

Task Transfer Net (TTNet)

Pal, Balasubramanian, Zero-shot Task Transfer, CVPR 2019

slide-23
SLIDE 23

Task Correlation Matrix

slide-24
SLIDE 24

More on Task Correlation

slide-25
SLIDE 25

Task Correlation Matrix

  • We get task correlation matrix from 30

annotators

  • Annotators are asked to give task

correlation label on a scale of {+3, +2, +1, 0, −1}

○ +3 denotes self relation ○ +2 describes strong relation ○ +1 implies weak relation ○ 0 to mention abstain ○ −1 to denote no relation between two tasks

Note: Our framework is not limited to crowdsourced task correlation. Any other method to compute task correlation will work

slide-26
SLIDE 26

Results - Surface Normal Estimation

TTNet6 Source Tasks: Autoencoding, Scene Class, 3D key point, Reshading, Vanishing Pt, Colorization Zero-Shot Task: Surface Normal

slide-27
SLIDE 27

Results - Depth Estimation

Ref: Arghya Pal, Vineeth N Balasubramanian, Zero-shot Task Transfer, CVPR 2019 Oral

TTNet6 (same model, only change in gamma values) Source Tasks: Same as previous Zero-Shot Task: Depth Estimation

slide-28
SLIDE 28

Results - Camera Pose Estimation

TTNet6 (same model, only change in gamma values) Source Tasks: Same as previous Zero-Shot Task: Camera Pose Estimation

slide-29
SLIDE 29

Why better than Supervised Learning?

slide-30
SLIDE 30

Zero shot to known task transfer

slide-31
SLIDE 31

How many source tasks do we need?

slide-32
SLIDE 32

Different Choices of Zero-shot tasks

Ref: Arghya Pal, Vineeth N Balasubramanian, Zero-shot Task Transfer, CVPR 2019 Oral

slide-33
SLIDE 33

Performance on Other Datasets:

Ref: Arghya Pal, Vineeth N Balasubramanian, Zero-shot Task Transfer, CVPR 2019 Oral Object detection on COCO-Stuff dataset

slide-34
SLIDE 34

Thank you!