INTERPOLATION IN LIGHT FIELDS WITH OVERLAPPING PATCHES D. C. Schedl - - PowerPoint PPT Presentation

▶
interpolation in light fields with overlapping patches
SMART_READER_LITE
LIVE PREVIEW

INTERPOLATION IN LIGHT FIELDS WITH OVERLAPPING PATCHES D. C. Schedl - - PowerPoint PPT Presentation

Feb 25, 2024 457 likes •788 views

OPTIMIZED SAMPLING FOR VIEW INTERPOLATION IN LIGHT FIELDS WITH OVERLAPPING PATCHES D. C. Schedl 1 and O. Bimber 1 1 Institute of Computer Graphics, david.schedl@jku.at and oliver.bimber@jku.at MOTIVATION uniform sparse [ours] coded sparse

slide-1
SLIDE 1

OPTIMIZED SAMPLING FOR VIEW INTERPOLATION IN LIGHT FIELDS WITH OVERLAPPING PATCHES

  • D. C. Schedl1 and O. Bimber1

1Institute of Computer Graphics, david.schedl@jku.at and oliver.bimber@jku.at

slide-2
SLIDE 2

MOTIVATION

uniform sparse (64 samples) dense (225 samples) [ours] coded sparse (225 from 64 samples)

  • D. C. Schedl & O. Bimber / Optimized Sampling for View Interpolation in Light Fields with Overlapping Patches

2

slide-3
SLIDE 3

RELATED WORK

 Coded sampling with local dictionaries [Schedl et al., ICCP 2015] [Schedl et al., CVIU 2017]

  • D. C. Schedl & O. Bimber / Optimized Sampling for View Interpolation in Light Fields with Overlapping Patches

3

slide-4
SLIDE 4

RELATED WORK

 Coded sampling with local dictionaries  Compressed Sensing [Marwah et al., TOG 2013] [Cao et al., Opt. Express 2014]

  • D. C. Schedl & O. Bimber / Optimized Sampling for View Interpolation in Light Fields with Overlapping Patches

4

slide-5
SLIDE 5

RELATED WORK

 Coded sampling with local dictionaries  Compressed Sensing  Depth-based view interpolation  Learning-based methods [Kalantari et al., TOG 2016]

  • D. C. Schedl & O. Bimber / Optimized Sampling for View Interpolation in Light Fields with Overlapping Patches

5

slide-6
SLIDE 6
  • D. C. Schedl & O. Bimber / Optimized Sampling for View Interpolation in Light Fields with Overlapping Patches

RELATED WORK

 Coded sampling with local dictionaries  Compressed Sensing  Depth-based view interpolation  Learning-based methods  Other [Vagharshakyan et al., PAMI 2015] [Shi et al., TOG 2014]

6

slide-7
SLIDE 7

CONTRIBUTIONS

 New sampling quality metric  A reduced search space for sampling mask estimation  An enhanced upsampling technique supporting maximal patch overlaps

  • D. C. Schedl & O. Bimber / Optimized Sampling for View Interpolation in Light Fields with Overlapping Patches

7

slide-8
SLIDE 8

SAMPLING QUALITY METRIC

  • D. C. Schedl & O. Bimber / Optimized Sampling for View Interpolation in Light Fields with Overlapping Patches

8

minimum distance

slide-9
SLIDE 9

SAMPLING QUALITY METRIC

  • D. C. Schedl & O. Bimber / Optimized Sampling for View Interpolation in Light Fields with Overlapping Patches

9

minimum distance extrap. interpolation

slide-10
SLIDE 10

SAMPLING QUALITY METRIC

  • D. C. Schedl & O. Bimber / Optimized Sampling for View Interpolation in Light Fields with Overlapping Patches

10

extrap. interpolation

  • urs
slide-11
SLIDE 11

SAMPLING QUALITY METRIC

  • D. C. Schedl & O. Bimber / Optimized Sampling for View Interpolation in Light Fields with Overlapping Patches

11

slide-12
SLIDE 12

SAMPLING QUALITY METRIC

sampling mask

  • D. C. Schedl & O. Bimber / Optimized Sampling for View Interpolation in Light Fields with Overlapping Patches

12

slide-13
SLIDE 13

SAMPLING PATTERN ESTIMATION

Constraints:

 regular  symmetric

sampling mask (64 samples; 15x15 grid)

  • D. C. Schedl & O. Bimber / Optimized Sampling for View Interpolation in Light Fields with Overlapping Patches

13

slide-14
SLIDE 14

SAMPLING PATTERN ESTIMATION

sampling mask (only guidance)

  • D. C. Schedl & O. Bimber / Optimized Sampling for View Interpolation in Light Fields with Overlapping Patches

14

basis grid permutations

slide-15
SLIDE 15

SAMPLING PATTERN ESTIMATION

basis grid permutations

  • D. C. Schedl & O. Bimber / Optimized Sampling for View Interpolation in Light Fields with Overlapping Patches

15

0.188 0.184 0.191 E:

slide-16
SLIDE 16

SAMPLING PATTERN ESTIMATION

  • D. C. Schedl & O. Bimber / Optimized Sampling for View Interpolation in Light Fields with Overlapping Patches

16

final sampling mask

0.188 0.184 0.191

basis grid permutations

E:

slide-17
SLIDE 17

RECONSTRUCTION

17

sampling mask … dictionary

  • D. C. Schedl & O. Bimber / Optimized Sampling for View Interpolation in Light Fields with Overlapping Patches
slide-18
SLIDE 18

RECONSTRUCTION

18

sampling mask … reconstructed light-field … sub-sampled light-field

  • D. C. Schedl & O. Bimber / Optimized Sampling for View Interpolation in Light Fields with Overlapping Patches
slide-19
SLIDE 19

RECONSTRUCTION

19

  • D. C. Schedl & O. Bimber / Optimized Sampling for View Interpolation in Light Fields with Overlapping Patches

sampling mask

slide-20
SLIDE 20

RECONSTRUCTION

20

reconstructed light-field

  • D. C. Schedl & O. Bimber / Optimized Sampling for View Interpolation in Light Fields with Overlapping Patches
slide-21
SLIDE 21

RESULTS: SAMPLING MASKS

21

Schedl ’17 Ours Schedl ’15 low (good) high (bad) 64 72 69 48

  • D. C. Schedl & O. Bimber / Optimized Sampling for View Interpolation in Light Fields with Overlapping Patches

unsupported unsupported unsupported

slide-22
SLIDE 22

RESULTS

 Table

22

Scenes (N) Marwah '13 Shi '14 Schedl '15 Kalantari '16 Schedl '17 Ours Amethyst (64) 37.77dB

  • 40.11dB

41.86dB 42.08dB Lego (64) 28.79dB

  • 32.87dB

35.63dB 37.26dB Lego (48)

  • 33.86dB

35.75dB Cave (64) 26.51dB

  • 30.99dB

38.57dB 41.08dB Alley (64) 36.58dB

  • 43.23dB

43.83dB 44.35dB Amethyst (72)

  • 36.40dB
  • 42.18dB

42.55dB Tarot (72)

  • 30.19dB
  • 37.81dB

39.20dB Amethyst (69)

  • 41.91dB
  • 42.07dB

42.43dB Tarot (69)

  • 34.09dB
  • 37.88dB

39.04dB Tarot (48)

  • 35.96dB

37.54dB Cave (69)

  • 29.96dB
  • 39.14dB

41.41dB Alley (69)

  • 41.36dB
  • 44.24dB

45.20dB

  • D. C. Schedl & O. Bimber / Optimized Sampling for View Interpolation in Light Fields with Overlapping Patches
slide-23
SLIDE 23

Schedl ’17 (64) Ours (64) Reference (225)

  • D. C. Schedl & O. Bimber / Optimized Sampling for View Interpolation in Light Fields with Overlapping Patches

23

Lego:

slide-24
SLIDE 24

Kalantari’ 16 (64) Ours (64) Reference (225)

  • D. C. Schedl & O. Bimber / Optimized Sampling for View Interpolation in Light Fields with Overlapping Patches

24

Cave:

slide-25
SLIDE 25

Schedl’ 15 (69) Ours (69) Reference (225)

  • D. C. Schedl & O. Bimber / Optimized Sampling for View Interpolation in Light Fields with Overlapping Patches

25

Tarot:

slide-26
SLIDE 26

Shi '14 (72) Ours (72) Reference (225)

  • D. C. Schedl & O. Bimber / Optimized Sampling for View Interpolation in Light Fields with Overlapping Patches

26

Amethyst:

slide-27
SLIDE 27

Marwah’13 (64) Ours (64) Reference (225)

  • D. C. Schedl & O. Bimber / Optimized Sampling for View Interpolation in Light Fields with Overlapping Patches

27

Alley:

slide-28
SLIDE 28

LIMITATIONS / FUTURE WORK

 Time: 40h – 5 days on NVIDIA Tesla V100 GPU

  • D. C. Schedl & O. Bimber / Optimized Sampling for View Interpolation in Light Fields with Overlapping Patches

29

slide-29
SLIDE 29

LIMITATIONS / FUTURE WORK

 Time: 40h – 5 days on NVIDIA Tesla V100 GPU  Other light-field camera designs

  • D. C. Schedl & O. Bimber / Optimized Sampling for View Interpolation in Light Fields with Overlapping Patches

30

slide-30
SLIDE 30

LIMITATIONS / FUTURE WORK

 Time: 40h – 5 days on NVIDIA Tesla V100 GPU  Other light-field camera designs  Other fields (e.g. image-based relighting)

  • D. C. Schedl & O. Bimber / Optimized Sampling for View Interpolation in Light Fields with Overlapping Patches

31

slide-31
SLIDE 31

JOHANNES KEPLER UNIVERSITY LINZ Altenberger Str. 69 4040 Linz, Austria www.jku.at

More information: www.jku.at/cg Contact: david.schedl@jku.at and oliver.bimber@jku.at This project was funded by FWF (P 28581-N33) Schedl ’17 (64) Ours (64) Reference (225) Kalantari’ 16 (64) Ours (64) Reference (225)