Transfer Learnin ing and Domain in Adaptatio ion Prof. - - PowerPoint PPT Presentation

▶
transfer learnin ing and domain in adaptatio ion
SMART_READER_LITE
LIVE PREVIEW

Transfer Learnin ing and Domain in Adaptatio ion Prof. - - PowerPoint PPT Presentation

Feb 20, 2023 322 likes •566 views

Transfer Learnin ing and Domain in Adaptatio ion Prof. Leal-Taix and Prof. Niessner 1 Big iggest Cri riticis ism of f Computer Vis ision Works on constructed datasets, but not in the real world and thats also true for

slide-1
SLIDE 1

Transfer Learnin ing and Domain in Adaptatio ion

  • Prof. Leal-Taixé and Prof. Niessner

1

slide-2
SLIDE 2

Big iggest Cri riticis ism of f Computer Vis ision

Works on constructed datasets, but not in the real world…

… and that’s also true for deep learning

  • Prof. Leal-Taixé and Prof. Niessner

2

slide-3
SLIDE 3

E.g .g., ., Mult lti-Dataset Eff fforts

  • Prof. Leal-Taixé and Prof. Niessner

3

Robust Vision Challenge: CVPR’18 [Geiger/Niessner/Pollefeys/Rother et al.]

slide-4
SLIDE 4

Tra ransfer r Learn rnin ing & Domain in Adaptatio ion

  • Task

– Image Classification – Image Segmentation – Object Instance Segmentation – …

  • Domain

– Real data

  • Real != real: webcam model 1 vs webcam model 2; day vs night

– Synthetic data

  • E.g., rasterization vs

– …

  • Prof. Leal-Taixé and Prof. Niessner

4

slide-5
SLIDE 5

Tra ransfer r Learn rnin ing & Domain in Adaptatio ion

  • Prof. Leal-Taixé and Prof. Niessner

5 Source: wikipedia

slide-6
SLIDE 6

Tra ransfer Learning

Same domain, different task

  • Pre-trained Image Net (visual domain of real images)

– Train on image classification

  • Fine-tune on new task

– E.g., semantic image segmentation – > keep ‘backbone the same, fine-tune ‘head’ layers – > assumption: visual features generalize within domain

  • Prof. Leal-Taixé and Prof. Niessner

6

slide-7
SLIDE 7

Tra ransfer Learning

Same task, different domain

  • Pre-trained Image Net (visual domain of real images)

– Train on image classification

  • Fine-tune on new task

– Now need to train *entire* network, cuz input features will be different – Training only a few layers at the end is less likely to fundamentally solve it

  • Prof. Leal-Taixé and Prof. Niessner

7

slide-8
SLIDE 8

Fin ine Tunin ing

  • How much labeled data in the target domain?

– Zero-shot learning – One-shot learning – Few-shot learning

  • Just ‘throwing in as much data as we can’ seems

somewhat unsatisfactory…

  • Prof. Leal-Taixé and Prof. Niessner

8

slide-9
SLIDE 9

Domain in Adaption

  • Prof. Leal-Taixé and Prof. Niessner

9

slide-10
SLIDE 10

Applic icatio ions to diff ifferent types of f domain in shift ift

From simulated to real control From dataset to dataset From RGB to depth From CAD models to real images Slide Credit: Kate Saenko

slide-11
SLIDE 11

backpack chair bike Source Data + Labels Unlabeled Target Data ?

Classifier

classification loss

Advers rsari rial domain adaptatio ion

conv1 conv5 fc 6 fc 7 conv1 conv5 fc 6 fc 7

Slide Credit: Kate Saenko

slide-12
SLIDE 12

backpack chair bike Source Data + Labels Unlabeled Target Data ?

Encoder Classifier Encoder

classification loss

Advers rsari rial domain adaptatio ion

can be shared Slide Credit: Kate Saenko

slide-13
SLIDE 13

backpack chair bike Source Data + Labels Unlabeled Target Data ?

Encoder Classifier Encoder

classification loss

Advers rsari rial domain adaptatio ion

Discriminator

Adversarial loss

can be shared Slide Credit: Kate Saenko

slide-14
SLIDE 14

backpack chair bike Source Data + Labels Unlabeled Target Data ?

Encoder Classifier Encoder

classification loss

Advers rsari rial domain adaptatio ion

Discriminator

Adversarial loss

can be shared Slide Credit: Kate Saenko

slide-15
SLIDE 15

Before domain confusion After domain confusion

Results on City ityscapes to SF adaptatio ion

FCNs in the Wild: Pixel-level Adversarial and Constraint-based Adaptation, Judy Hoffman, Dequan Wang, Fisher Yu, Trevor Darrell, Arxiv 2016

slide-16
SLIDE 16

Cycle-Consis istent Adversaria ial Domain in Adaptatio ion

  • Prof. Leal-Taixé and Prof. Niessner

16

CyCADA [Hoffman et al. 2018]

slide-17
SLIDE 17

Cycle-Consis istent Adversaria ial Domain in Adaptatio ion

  • Prof. Leal-Taixé and Prof. Niessner

17

CyCADA [Hoffman et al. 2018]

slide-18
SLIDE 18

Exam

  • Slides provide additional references (use them)
  • Look up the important papers that we discussed
  • Understanding of

– high-level concepts – underlying math – architecture design

  • Prof. Leal-Taixé and Prof. Niessner

18

slide-19
SLIDE 19

Admin inistrative

  • Deadline for final projects

– Wed d Fe Feb 6th

th, 11

11:59 :59pm – Submission via moodle – Submission must contain

  • Code (results must be replicable)
  • 2-3 pages of final report (at most 1 page of text, rest results;

i.e., images and tables)

  • Use CVPR templates:

http://cvpr2019.thecvf.com/submission/main_conference/ author_guidelines

  • Prof. Leal-Taixé and Prof. Niessner

19

slide-20
SLIDE 20

Admin inistrative

  • Poster presentation

– Fri Friday Fe Feb 8th

th, 1p

1pm-3pm 3pm – Location:

  • Magistrale (preliminary – will update if it changes)
  • In the area next to the back entrance (parking lot direction)

– Poster stands will be provided – You need to print posters yourself (poster@in.tum.de) – Hang posters 15 mins before presentation session starts

  • Prof. Leal-Taixé and Prof. Niessner

20

slide-21
SLIDE 21

Guest Speakers

  • Oriol Vinyals:

– https://ai.google/research/people/OriolVinyals – Time: Ja January 31 31st

st, 6pm – 8pm

– Location: HS-1 (CS building – the big one)

  • Prof. Leal-Taixé and Prof. Niessner

21

slide-22
SLIDE 22

Next Lectures

This was the last lecture 

  • Prof. Leal-Taixé and Prof. Niessner

22