Self-Supervised Feature Learning by Learning to Spot Artifacts - - PowerPoint PPT Presentation

▶
self supervised feature
SMART_READER_LITE
LIVE PREVIEW

Self-Supervised Feature Learning by Learning to Spot Artifacts - - PowerPoint PPT Presentation

Oct 16, 2022 135 likes •389 views

Self-Supervised Feature Learning by Learning to Spot Artifacts Wonbin Kim Self-Supervised Learning To exploit different labelings that are freely available besides or within visual data, and to use them as intrinsic reward signals to

slide-1
SLIDE 1

Self-Supervised Feature Learning by Learning to Spot Artifacts

Wonbin Kim

slide-2
SLIDE 2

Self-Supervised Learning

2

  • “To exploit different labelings that are freely available

besides or within visual data, and to use them as intrinsic reward signals to learn general-purpose features.”[5]

Object Detection, Semantic Segmentation, Classification

slide-3
SLIDE 3

Shortcut

3

  • “Trivial” and undesirable Solution to

Self-Supervised Learning

  • Low-level statistics
  • Boundary Pattern or Texture
  • Edge continuity
  • The pixel intensity/color distribution
  • Chromatic aberration

Wikipedia : Industrial_National_Bank_Building

slide-4
SLIDE 4

Chromatic aberration

4

  • It cause different wavelengths of light to have differing focal lengths.

Wikipedia : Chromatic aberration

slide-5
SLIDE 5

Motivation

5

“There is nothing noble in being superior to your fellow man; true nobility is being superior to your former self.”

  • Ernest Hemingway

“There is nothing noble in being superior to your fellow man; true nobility is being to your former self.”

  • Ernest Hemingway
  • What words will come to the blank?
slide-6
SLIDE 6
  • 1. Context prediction

6

Carl Doersch et al., "Unsupervised visual representation learning by context prediction.“, ICCV, (2015)

slide-7
SLIDE 7
  • 1. Context prediction – avoiding shortcuts

7

Carl Doersch et al., "Unsupervised visual representation learning by context prediction.“, ICCV, (2015)

  • Sampling with gap and jittering
  • Color dropping
slide-8
SLIDE 8
  • 1. Context prediction[6]

8

Carl Doersch et al., "Unsupervised visual representation learning by context prediction.“, ICCV, (2015)

slide-9
SLIDE 9
  • 2. Inpainting

9

Deepak Pathak et al. "Context encoders: Feature learning by inpainting." ICCV, (2016)

slide-10
SLIDE 10
  • 2. Inpainting

1

  • Loss function
  • Reconstruction loss

ℒ𝑠𝑓𝑑 𝑦 = 𝜇𝑠𝑓𝑑 ෡ 𝑁 ⊙ 𝑦 − 𝐺 1 − ෡ 𝑁 ⊙ 𝑦

2 2

  • Adversarial loss

ℒ𝑏𝑒𝑤 = max

𝐸

𝔽𝑦∈𝜓 log 𝐸 𝑦 + log 1 − 𝐸 𝐺 1 − ෡ 𝑁 ⊙

  • Aggregated Loss

ℒ = 𝜇𝑠𝑓𝑑ℒ𝑠𝑓𝑑 + 𝜇𝑏𝑒𝑤ℒ𝑏𝑒𝑤

slide-11
SLIDE 11
  • 2. Inpainting

1 1

Deepak Pathak et al. "Context encoders: Feature learning by inpainting." ICCV, (2016)

slide-12
SLIDE 12
  • 2. Inpainting

1 2

Deepak Pathak et al. "Context encoders: Feature learning by inpainting." ICCV, (2016)

slide-13
SLIDE 13
  • 2. Inpainting

1 3

Deepak Pathak et al. "Context encoders: Feature learning by inpainting." ICCV, (2016)

slide-14
SLIDE 14
  • 2. Inpainting

1 4

Deepak Pathak et al. "Context encoders: Feature learning by inpainting." ICCV, (2016)

slide-15
SLIDE 15
  • 3. Solving Jigsaw puzzle

15

Mehdi Noroozi et el., "Unsupervised learning of visual representations by solving jigsaw puzzles." ECCV, 2016.

slide-16
SLIDE 16
  • 3. Solving Jigsaw puzzle

16

Mehdi Noroozi et el., "Unsupervised learning of visual representations by solving jigsaw puzzles." ECCV, 2016.

slide-17
SLIDE 17
  • 3. Solving Jigsaw puzzle

17

Mehdi Noroozi et el., "Unsupervised learning of visual representations by solving jigsaw puzzles." ECCV, 2016.

slide-18
SLIDE 18
  • 3. Solving Jigsaw puzzle

18

Mehdi Noroozi et el., "Unsupervised learning of visual representations by solving jigsaw puzzles." ECCV, 2016.

slide-19
SLIDE 19
  • 3. Distortion

19

Simon Jenni et el., "Self-Supervised Feature Learning by Learning to Spot Artifacts." CVPR (2018).

slide-20
SLIDE 20
  • 4. Distortion

20

Simon Jenni et el., "Self-Supervised Feature Learning by Learning to Spot Artifacts." CVPR (2018).

slide-21
SLIDE 21
  • 4. Distortion

21

Simon Jenni et el., "Self-Supervised Feature Learning by Learning to Spot Artifacts." CVPR (2018).

  • Decoder with Repair Network.
  • i=1

𝐸1 = 𝜚 𝑦 + 1 − Ω ⊙ 𝑆1 ෠ 𝜚 𝑦 , Where ෠ 𝜚 x = Ω ⊙ 𝜚 x + (1 − Ω) ⊙ 𝑣 ∗ 𝜚 𝑦

  • i=2, 3, 4

𝐸𝑗 = 𝐸𝑗−1 + 1 − 𝑉𝑗−1 Ω ⊙ 𝑆𝑗 𝐸𝑗−1

slide-22
SLIDE 22
  • 4. Distortion

22

Simon Jenni et el., "Self-Supervised Feature Learning by Learning to Spot Artifacts." CVPR (2018).

  • Loss function
  • Discriminator adversarial Loss : {real , corruption}

ℒ𝑑𝑚𝑏𝑡𝑡 = min

𝑆 max 𝐷

෍

𝐲~p 𝐲

log 𝐷𝑑𝑚𝑏𝑡𝑡 𝐸 𝜚 𝐲 + log 1 − 𝐷𝑑𝑚𝑏𝑡𝑡 ෡ 𝐸 ෡ 𝜚 𝐲 .

  • Discriminator binary classification Loss : mask prediction

ℒ𝑛𝑏𝑡𝑙 = min

𝐷 ෍ ො 𝐲

෍

𝑗𝑘

Ω𝑗𝑘 log 𝜏 𝐷𝑗𝑘

𝑛𝑏𝑡𝑙 ො

𝐲 + 1 − Ω𝑗𝑘 log 1 − 𝜏 𝐷𝑗𝑘

𝑛𝑏𝑡𝑙

෡ 𝐸 ෡ 𝜚 𝐲

slide-23
SLIDE 23
  • 4. Distortion

23

Simon Jenni et el., "Self-Supervised Feature Learning by Learning to Spot Artifacts." CVPR (2018).

slide-24
SLIDE 24

Reference

  • [1] Jenni, Simon, and Paolo Favaro. "Self-Supervised Feature Learning by

Learning to Spot Artifacts." CVPR (2018).

  • [2] Pathak, Deepak, et al. "Context encoders: Feature learning by inpainting."

Proceedings of the IEEE Conference on Computer Vision and Pattern

  • Recognition. 2016.
  • [3] Noroozi, Mehdi, and Paolo Favaro. "Unsupervised learning of visual

representations by solving jigsaw puzzles." European Conference on Computer

  • Vision. Springer, Cham, 2016.
  • [4] Doersch, Carl, Abhinav Gupta, and Alexei A. Efros. "Unsupervised visual

representation learning by context prediction." Proceedings of the IEEE International Conference on Computer Vision. 2015.

  • [5] Wikipedia : Chromatic aberration;

https://en.wikipedia.org/wiki/Chromatic_aberration

24