Single Image Portrait Relighting Tiancheng Sun 1 , Jonathan T. Barron - - PowerPoint PPT Presentation

▶
single image portrait relighting
SMART_READER_LITE
LIVE PREVIEW

Single Image Portrait Relighting Tiancheng Sun 1 , Jonathan T. Barron - - PowerPoint PPT Presentation

Sep 02, 2023 129 likes •1.65k views

Single Image Portrait Relighting Tiancheng Sun 1 , Jonathan T. Barron 2 , Yun-Ta Tsai 2 , Zexiang Xu 1 , Xueming Yu 3 , Graham Fyffe 3 , Christoph Rhemann 3 , Jay Busch 3 , Paul Debevec 3 , Ravi Ramamoorthi 1 1 University of California, San Diego, 2

slide-1
SLIDE 1

Single Image Portrait Relighting

Tiancheng Sun1, Jonathan T. Barron2, Yun-Ta Tsai2, Zexiang Xu1, Xueming Yu3, Graham Fyffe3, Christoph Rhemann3, Jay Busch3, Paul Debevec3, Ravi Ramamoorthi1

1University of California, San Diego, 2Google Research, 3Google

slide-2
SLIDE 2

Photography & Recording Allowed

slide-3
SLIDE 3

Overview

slide-4
SLIDE 4

Overview

light from the back

slide-5
SLIDE 5

Overview

light from the back shadow on the face

slide-6
SLIDE 6

Overview

want to add dramatic lighting light from the back shadow on the face

slide-7
SLIDE 7

Overview

want to add dramatic lighting light from the back shadow on the face

Change the lighting of any portrait after capture using post-processing algorithm

slide-8
SLIDE 8

Overview

slide-9
SLIDE 9

Overview

slide-10
SLIDE 10

Portrait Relighting System

Overview

slide-11
SLIDE 11

Portrait Relighting System

Overview

another lighting

slide-12
SLIDE 12

Portrait Relighting System

Overview

another lighting

slide-13
SLIDE 13

Overview

Portrait Relighting System

slide-14
SLIDE 14

Overview

Portrait Relighting System

slide-15
SLIDE 15

Overview

Portrait Relighting System

I want to rotate the lighting a little bit.

slide-16
SLIDE 16

Overview

Portrait Relighting System

I want to rotate the lighting a little bit.

slide-17
SLIDE 17

Previous work

slide-18
SLIDE 18

Previous work

  • Light Stage

Debevec, Paul, et al. "Acquiring the reflectance field of a human face." SIGGRAPH 2000.

slide-19
SLIDE 19

Previous work

  • Light Stage

Debevec, Paul, et al. "Acquiring the reflectance field of a human face." SIGGRAPH 2000.

slide-20
SLIDE 20

Previous work

  • Light Stage

Debevec, Paul, et al. "Acquiring the reflectance field of a human face." SIGGRAPH 2000.

slide-21
SLIDE 21

Previous work

  • Light Stage

Debevec, Paul, et al. "Acquiring the reflectance field of a human face." SIGGRAPH 2000.

: capture ~100 images and do image-based relighting

slide-22
SLIDE 22

Previous work

slide-23
SLIDE 23
  • Deep image-based relighting

Xu, Zexiang, et al. "Deep image-based relighting from optimal sparse samples." SIGGRAPH 2018

Previous work

slide-24
SLIDE 24
  • Deep image-based relighting

Xu, Zexiang, et al. "Deep image-based relighting from optimal sparse samples." SIGGRAPH 2018

Previous work

slide-25
SLIDE 25
  • Deep image-based relighting

Xu, Zexiang, et al. "Deep image-based relighting from optimal sparse samples." SIGGRAPH 2018

Previous work

slide-26
SLIDE 26
  • Deep image-based relighting

Xu, Zexiang, et al. "Deep image-based relighting from optimal sparse samples." SIGGRAPH 2018

capture 5 images and do relighting via neural network

Previous work

slide-27
SLIDE 27

Previous work

slide-28
SLIDE 28
  • Portrait lighting transfer

Shu, Zhixin, et al. "Portrait lighting transfer using a mass transport approach." SIGGRAPH 2018

Previous work

slide-29
SLIDE 29
  • Portrait lighting transfer

Shu, Zhixin, et al. "Portrait lighting transfer using a mass transport approach." SIGGRAPH 2018

Previous work

slide-30
SLIDE 30
  • Portrait lighting transfer

Shu, Zhixin, et al. "Portrait lighting transfer using a mass transport approach." SIGGRAPH 2018

transfer lighting from one portrait to another

Previous work

slide-31
SLIDE 31

Previous work

slide-32
SLIDE 32

Normal Shading Albedo Relit

  • SfSNet

Sengupta, Soumyadip, et al. "SfSNet: Learning Shape, Reflectance and Illuminance of Faces in the Wild'." CVPR 2018.

Previous work

slide-33
SLIDE 33

Normal Shading Albedo Relit

  • SfSNet

Sengupta, Soumyadip, et al. "SfSNet: Learning Shape, Reflectance and Illuminance of Faces in the Wild'." CVPR 2018.

: deep intrinsic decomposition mostly trained on synthetic faces

Previous work

slide-34
SLIDE 34

Normal Shading Albedo Relit

  • SfSNet

Sengupta, Soumyadip, et al. "SfSNet: Learning Shape, Reflectance and Illuminance of Faces in the Wild'." CVPR 2018.

: deep intrinsic decomposition mostly trained on synthetic faces

Previous work

slide-35
SLIDE 35

Overview

slide-36
SLIDE 36
  • Goal: practical relighting on single portrait image

Overview

slide-37
SLIDE 37
  • Goal: practical relighting on single portrait image
  • Practical in detail:
  • Robust to the pose and camera view
  • Work well on natural lightings
  • Adapt to high-resolution images
  • Run at interactive rate

Overview

slide-38
SLIDE 38
  • Goal: practical relighting on single portrait image
  • Practical in detail:
  • Robust to the pose and camera view
  • Work well on natural lightings
  • Adapt to high-resolution images
  • Run at interactive rate
  • Solution: Deep Neural Network + Real Face Data.

Overview

slide-39
SLIDE 39

Method

slide-40
SLIDE 40

Method

portrait under lighting A

slide-41
SLIDE 41

Method

Portrait Relighting System (Neural Network)

portrait under lighting A

slide-42
SLIDE 42

Method

Portrait Relighting System (Neural Network)

portrait under lighting A lighting B

slide-43
SLIDE 43

Method

Portrait Relighting System (Neural Network)

portrait under lighting A portrait under lighting B lighting B

slide-44
SLIDE 44

Method

Portrait Relighting System (Neural Network)

portrait under lighting A portrait under lighting B lighting A lighting B

slide-45
SLIDE 45

Method

Portrait Relighting System (Neural Network)

portrait under lighting A portrait under lighting B lighting A lighting B

slide-46
SLIDE 46

Method

Portrait Relighting System (Neural Network)

portrait under lighting A portrait under lighting B lighting A lighting B

slide-47
SLIDE 47

Method

Portrait Relighting System (Neural Network)

portrait under lighting A portrait under lighting B lighting A lighting B

slide-48
SLIDE 48

Method

Portrait Relighting System (Neural Network)

portrait under lighting A portrait under lighting B lighting A lighting B

How can we get the portrait pair for training?

slide-49
SLIDE 49

Method: Data

slide-50
SLIDE 50

Method: Data

Light Stage

slide-51
SLIDE 51

Method: Data

Light Stage One-Light-At-a-Time scans (OLAT)

slide-52
SLIDE 52

Method: Data

slide-53
SLIDE 53

Method: Data

Debevec, Paul, et al. "Acquiring the reflectance field of a human face." SIGGRAPH 2000.

slide-54
SLIDE 54

Method: Data

captured OLAT captured OLAT

Debevec, Paul, et al. "Acquiring the reflectance field of a human face." SIGGRAPH 2000.

slide-55
SLIDE 55

lighting

Method: Data

captured OLAT captured OLAT

Debevec, Paul, et al. "Acquiring the reflectance field of a human face." SIGGRAPH 2000.

slide-56
SLIDE 56

lighting

Method: Data

captured OLAT captured OLAT latitude-longitude representation

Debevec, Paul, et al. "Acquiring the reflectance field of a human face." SIGGRAPH 2000.

slide-57
SLIDE 57

lighting

Method: Data

captured OLAT captured OLAT latitude-longitude representation

Debevec, Paul, et al. "Acquiring the reflectance field of a human face." SIGGRAPH 2000.

slide-58
SLIDE 58

lighting

Method: Data

captured OLAT captured OLAT latitude-longitude representation

Debevec, Paul, et al. "Acquiring the reflectance field of a human face." SIGGRAPH 2000.

slide-59
SLIDE 59

lighting

x

x + + …… =

Method: Data

captured OLAT captured OLAT latitude-longitude representation

Debevec, Paul, et al. "Acquiring the reflectance field of a human face." SIGGRAPH 2000.

slide-60
SLIDE 60

lighting

x

x + + …… =

Method: Data

captured OLAT captured OLAT relit image (background removed) latitude-longitude representation

Debevec, Paul, et al. "Acquiring the reflectance field of a human face." SIGGRAPH 2000.

slide-61
SLIDE 61

lighting

x

x + + …… =

Method: Data

captured OLAT captured OLAT relit image (background removed) latitude-longitude representation

Wadhwa, Neal, et al. "Synthetic depth-of-field with a single-camera mobile phone." SIGGRAPH 2018 Debevec, Paul, et al. "Acquiring the reflectance field of a human face." SIGGRAPH 2000.

slide-62
SLIDE 62

Method: Data

slide-63
SLIDE 63
  • OLAT images
  • 22 people (18 training, 4 validation), each 3~5 facial expressions

Method: Data

slide-64
SLIDE 64
  • OLAT images
  • 22 people (18 training, 4 validation), each 3~5 facial expressions
  • Each OLAT captured with 7 cameras in 6 seconds.

Method: Data

slide-65
SLIDE 65
  • OLAT images
  • 22 people (18 training, 4 validation), each 3~5 facial expressions
  • Each OLAT captured with 7 cameras in 6 seconds.
  • HDR lighting environments

Method: Data

slide-66
SLIDE 66
  • OLAT images
  • 22 people (18 training, 4 validation), each 3~5 facial expressions
  • Each OLAT captured with 7 cameras in 6 seconds.
  • HDR lighting environments
  • ~2000 indoor HDR lighting

from Laval Dataset

Method: Data

slide-67
SLIDE 67
  • OLAT images
  • 22 people (18 training, 4 validation), each 3~5 facial expressions
  • Each OLAT captured with 7 cameras in 6 seconds.
  • HDR lighting environments
  • ~2000 indoor HDR lighting

from Laval Dataset

  • ~1000 outdoor HDR

lighting from the web

Method: Data

slide-68
SLIDE 68
  • OLAT images
  • 22 people (18 training, 4 validation), each 3~5 facial expressions
  • Each OLAT captured with 7 cameras in 6 seconds.
  • HDR lighting environments
  • ~2000 indoor HDR lighting

from Laval Dataset

  • ~1000 outdoor HDR

lighting from the web

Method: Data

  • Total: 226,800 portrait and lighting pairs for training
slide-69
SLIDE 69

Method: Training

slide-70
SLIDE 70

Method: Training

Encoder Decoder Bottleneck

slide-71
SLIDE 71

Method: Training

Encoder Decoder Bottleneck

  • Task 1: Complete relighting
slide-72
SLIDE 72

Method: Training

Encoder Decoder Bottleneck source image

  • Task 1: Complete relighting
slide-73
SLIDE 73

Method: Training

Encoder Decoder Bottleneck source light source image

  • Task 1: Complete relighting
slide-74
SLIDE 74

Method: Training

Encoder Decoder Bottleneck source light target light source image

  • Task 1: Complete relighting
slide-75
SLIDE 75

Method: Training

Encoder Decoder Bottleneck source light target light source image target image

  • Task 1: Complete relighting
slide-76
SLIDE 76

Method: Training

Encoder Decoder Bottleneck source light target light source image target image

  • Task 1: Complete relighting

L1 loss Log L1 loss

slide-77
SLIDE 77

Method: Training

Encoder Decoder Bottleneck

slide-78
SLIDE 78

Method: Training

  • Task 2: Illumination retargeting

Encoder Decoder Bottleneck

slide-79
SLIDE 79

Method: Training

  • Task 2: Illumination retargeting

Encoder Decoder Bottleneck source image

slide-80
SLIDE 80

Method: Training

  • Task 2: Illumination retargeting

Encoder Decoder Bottleneck source light source image

slide-81
SLIDE 81

Method: Training

  • Task 2: Illumination retargeting

Encoder Decoder Bottleneck source light source image Rotate

slide-82
SLIDE 82

Method: Training

  • Task 2: Illumination retargeting

Encoder Decoder Bottleneck source light target light source image Rotate

slide-83
SLIDE 83

Method: Training

  • Task 2: Illumination retargeting

Encoder Decoder Bottleneck source light target light source image target image Rotate

slide-84
SLIDE 84

Method: Training

  • Task 2: Illumination retargeting

Encoder Decoder Bottleneck source light target light source image target image Rotate

L1 loss Log L1 loss

slide-85
SLIDE 85

Method: Training

slide-86
SLIDE 86

Method: Training

  • Network structure
  • U-Net

k x k conv layer concatenation

k

weighted average tiling

k

k-dimensional input/label

k

k-dimensional activation

256 x 256 128 x 128 64 x 64 32 x 32 16 x 16 Spatial Resolution:

loss

slide-87
SLIDE 87

Method: Training

  • Network structure
  • U-Net

k x k conv layer concatenation

k

weighted average tiling

k

k-dimensional input/label

k

k-dimensional activation

256 x 256 128 x 128 64 x 64 32 x 32 16 x 16 Spatial Resolution:

loss

Source Image

64 64 3 128 128 3 256 256 3 512 512 3 512 3 512 3

Output Source Light

16 x 32 x 3 3 29 7 16 x 32 x 1 3 3 3 3 3 3

True Source Light

16 x 32 x 3 16 x 32 x 3

slide-88
SLIDE 88

Method: Training

  • Network structure
  • U-Net
  • Predict and feed in light at bottleneck

k x k conv layer concatenation

k

weighted average tiling

k

k-dimensional input/label

k

k-dimensional activation

256 x 256 128 x 128 64 x 64 32 x 32 16 x 16 Spatial Resolution:

loss

Source Image

64 64 3 128 128 3 256 256 3 512 512 3 512 3 512 3

Output Source Light

16 x 32 x 3 3 29 7 16 x 32 x 1 3 3 3 3 3 3

True Source Light

16 x 32 x 3 16 x 32 x 3

slide-89
SLIDE 89

Method: Training

  • Network structure
  • U-Net
  • Predict and feed in light at bottleneck

confidence learning module

k x k conv layer concatenation

k

weighted average tiling

k

k-dimensional input/label

k

k-dimensional activation

256 x 256 128 x 128 64 x 64 32 x 32 16 x 16 Spatial Resolution:

loss

Source Image

64 64 3 128 128 3 256 256 3 512 512 3 512 3 512 3

Output Source Light

16 x 32 x 3 3 29 7 16 x 32 x 1 3 3 3 3 3 3

True Source Light

16 x 32 x 3 16 x 32 x 3

slide-90
SLIDE 90

Method: Training

  • Network structure
  • U-Net
  • Predict and feed in light at bottleneck

confidence learning module

k x k conv layer concatenation

k

weighted average tiling

k

k-dimensional input/label

k

k-dimensional activation

256 x 256 128 x 128 64 x 64 32 x 32 16 x 16 Spatial Resolution:

loss

Source Image

64 64 3 128 128 3 256 256 3 512 512 3 512 3 512 3

Output Source Light

16 x 32 x 3 3 29 7 16 x 32 x 1 3 3 3 3 3 3

True Source Light

16 x 32 x 3 16 x 32 x 3

3 512

Target Light

3 32 32 3 512 256 512 3 512 512 256 3 256 128 3 128 64 3 64 512 256 256 128 128 64 64 3 3 3 3

16 x 32 x 3

True Target Image Output Target Image

3 3

slide-91
SLIDE 91

Method: Training

slide-92
SLIDE 92

Method: Training

  • Confidence learning
slide-93
SLIDE 93

Method: Training

  • Confidence learning
slide-94
SLIDE 94

Method: Training

Several conv layers

  • Confidence learning
slide-95
SLIDE 95

Method: Training

Several conv layers

  • Confidence learning
slide-96
SLIDE 96

Method: Training

Several conv layers resolution

  • f the light
  • Confidence learning
slide-97
SLIDE 97

Method: Training

Several conv layers resolution

  • f the light

Light prediction on each image patch

  • Confidence learning
slide-98
SLIDE 98

Method: Training

Several conv layers resolution

  • f the light

Light prediction on each image patch Confidence of prediction on each image patch

  • Confidence learning
  • Predict the confidence of light prediction
slide-99
SLIDE 99

Method: Training

Several conv layers resolution

  • f the light

Light prediction on each image patch Confidence of prediction on each image patch

* =

  • Confidence learning
  • Predict the confidence of light prediction

Reshape

slide-100
SLIDE 100

Method: Training

Several conv layers resolution

  • f the light

Light prediction on each image patch Confidence of prediction on each image patch

* =

  • Confidence learning
  • Predict the confidence of light prediction
  • Allow network to say “I don’t know”

Reshape

slide-101
SLIDE 101

Results: Validation

slide-102
SLIDE 102

Results: Validation

➤ Relit images for complete relighting

slide-103
SLIDE 103

Results: Validation

➤ Relit images for complete relighting

source image

slide-104
SLIDE 104

Results: Validation

➤ Relit images for complete relighting

source image

slide-105
SLIDE 105

Results: Validation

➤ Relit images for complete relighting

source image

slide-106
SLIDE 106

Results: Validation

➤ Relit images for complete relighting

source image target image

slide-107
SLIDE 107

Results: Validation

➤ Relit images for complete relighting

source image target image

  • urs

[Barron & Malik 2015][Sengupta et al. 2018] [Li et al. 2018]

slide-108
SLIDE 108

Results: Validation

➤ Relit images for complete relighting

source image target image

  • urs

[Barron & Malik 2015][Sengupta et al. 2018] [Li et al. 2018]

slide-109
SLIDE 109

Results: Validation

slide-110
SLIDE 110

Results: Validation

slide-111
SLIDE 111

Results: Validation

slide-112
SLIDE 112

Results: Validation

➤ Relight images with

predicted light as target light

Encoder Decoder Bottleneck

slide-113
SLIDE 113

Results: Validation

➤ Relight images with

predicted light as target light

source image

Encoder Decoder Bottleneck

slide-114
SLIDE 114

Results: Validation

➤ Relight images with

predicted light as target light

source image with self-supervision without self-supervision

Encoder Decoder Bottleneck

slide-115
SLIDE 115

Results: Validation

slide-116
SLIDE 116

Results: Validation

slide-117
SLIDE 117

Results: Validation

slide-118
SLIDE 118

Results: Validation

➤ Comparison with portrait lighting transfer

slide-119
SLIDE 119

Results: Validation

➤ Comparison with portrait lighting transfer

reference image

slide-120
SLIDE 120

Results: Validation

➤ Comparison with portrait lighting transfer

source image reference image

slide-121
SLIDE 121

Results: Validation

➤ Comparison with portrait lighting transfer

source image reference image

Extract light from reference

slide-122
SLIDE 122

Results: Validation

➤ Comparison with portrait lighting transfer

source image reference image

Extract light from reference Apply to source image

slide-123
SLIDE 123

Results: Validation

➤ Comparison with portrait lighting transfer

source image groundtruth

  • urs

reference image

Extract light from reference Apply to source image

slide-124
SLIDE 124

Results: Validation

➤ Comparison with portrait lighting transfer

source image groundtruth

  • urs

[Shih et al. 2014] [Shu et al. 2018] reference image

Extract light from reference Apply to source image

slide-125
SLIDE 125

Results: Validation

slide-126
SLIDE 126

Results: Validation

➤ Evaluation on lighting prediction

slide-127
SLIDE 127

Results: Validation

➤ Evaluation on lighting prediction

source image ground truth

slide-128
SLIDE 128

Results: Validation

➤ Evaluation on lighting prediction

source image ground truth

  • urs
  • urs w/o

confidence learning

slide-129
SLIDE 129

Results: Validation

➤ Evaluation on lighting prediction

source image ground truth

  • urs
  • urs w/o

confidence learning

[Barron & Malik 2015] [Sengupta et al. 2018]

slide-130
SLIDE 130

Results: Images in the wild

slide-131
SLIDE 131

Results: Images in the wild

Input Image Relit Image

slide-132
SLIDE 132

Results: Images in the wild

Input Image Relit Image

slide-133
SLIDE 133

Results: Images in the wild

Input Image Relit Image Input Image Relit Image

slide-134
SLIDE 134

Results: Images in the wild

Input Image Relit Image Input Image Relit Image

slide-135
SLIDE 135

Results: Images in the wild

slide-136
SLIDE 136

Results: Images in the wild

slide-137
SLIDE 137

Results: Images in the wild

slide-138
SLIDE 138

Results: Images in the wild

slide-139
SLIDE 139

Limitations

slide-140
SLIDE 140

Limitations

Input Image Relit Image

  • Complex shadows
  • Specular highlights
  • Overexposed pixels
slide-141
SLIDE 141

Limitations

Input Image Relit Image

  • Complex shadows
  • Specular highlights
  • Overexposed pixels
  • Over-smoothing
slide-142
SLIDE 142

Limitations

Input Image Relit Image

  • Complex shadows
  • Specular highlights
  • Overexposed pixels
  • Over-smoothing
  • Unseen high-saturation

color

slide-143
SLIDE 143

Conclusion

slide-144
SLIDE 144

Conclusion

  • Learn the relighting function on portraits using Light Stage data
slide-145
SLIDE 145

Conclusion

  • Learn the relighting function on portraits using Light Stage data
  • Take home message:
slide-146
SLIDE 146

Conclusion

  • Learn the relighting function on portraits using Light Stage data
  • Take home message:
  • For human faces, use real data.
slide-147
SLIDE 147

Conclusion

  • Learn the relighting function on portraits using Light Stage data
  • Take home message:
  • For human faces, use real data.
  • End-to-end training vs assuming models.
slide-148
SLIDE 148

Acknowledgement

  • This work was funded in part by a Jacobs Fellowship, the Ronald L.

Graham Chair, and the UC San Diego Center for Visual Computing.

  • Thanks to Zhixin Shu and Yichang Shih for the help on baseline

algorithms

  • Thanks to Jean-François Lalonde for providing the indoor lighting

dataset.

  • Thanks to Peter Denny for coordinating dataset capture.
  • Thanks to all the anonymous volunteers in the dataset from

Google, UCSD and UCLA

slide-149
SLIDE 149

Also presenting in poster session today at 12:15-1:15.

Single Image Portrait Relighting

slide-150
SLIDE 150

Single Image Portrait Relighting

input image generated portraits under new illuminations

slide-151
SLIDE 151

Single Image Portrait Relighting

input image generated portraits under new illuminations