CraGL Creativity and Graphics Lab Background: Digital Painting - - PowerPoint PPT Presentation

▶
cragl
SMART_READER_LITE
LIVE PREVIEW

CraGL Creativity and Graphics Lab Background: Digital Painting - - PowerPoint PPT Presentation

Dec 13, 2022 392 likes •1.4k views

Decomposing Time-Lapse Paintings into Layers Jianchao Tan George Mason University Marek Dvoro k Czech Technical University in Prague Daniel S kora Czech Technical University in Prague Yotam Gingold George Mason University CraGL

slide-1
SLIDE 1

Decomposing Time-Lapse Paintings into Layers

Jianchao Tan George Mason University Marek Dvorožňák Czech Technical University in Prague Yotam Gingold Daniel Sýkora Czech Technical University in Prague George Mason University

Creativity and Graphics Lab

CraGL

slide-2
SLIDE 2

Background: Digital Painting

[Angela Sasser, https://www.artstation.com/artwork/nariko-heavenly-guardian]

slide-3
SLIDE 3

Background: Digital Painting

[Angela Sasser, https://www.artstation.com/artwork/nariko-heavenly-guardian]

Layers

slide-4
SLIDE 4

are RGBA images Layers

Background: Digital Painting

[Angela Sasser, https://www.artstation.com/artwork/nariko-heavenly-guardian]

slide-5
SLIDE 5

Background: Digital Painting

[Angela Sasser, https://www.artstation.com/artwork/nariko-heavenly-guardian]

slide-6
SLIDE 6

Motivation

  • Physical paintings are hard to edit.
slide-7
SLIDE 7

Motivation

  • What if we have a time lapse video?
slide-8
SLIDE 8

Motivation

  • What if we have a time lapse video?
slide-9
SLIDE 9

Goal

  • Decompose a time-lapse painting video into layers
slide-10
SLIDE 10

Goal

  • Decompose a time-lapse painting video into layers
slide-11
SLIDE 11

Goal

  • Decompose a time-lapse painting video into layers
slide-12
SLIDE 12

Challenges

  • Preprocessing:
slide-13
SLIDE 13

Challenges

  • Preprocessing:

painter

slide-14
SLIDE 14

Challenges

  • Preprocessing:

painter shadows

slide-15
SLIDE 15

Challenges

  • Preprocessing:

color shift painter shadows

slide-16
SLIDE 16

Challenges

  • Preprocessing:

color shift painter shadows lighting

slide-17
SLIDE 17

Challenges

  • Recovering paint layers

before after

slide-18
SLIDE 18

Challenges

  • Recovering paint layers

before after

color change

slide-19
SLIDE 19

Related Work

  • Interacting with editing history
  • Su et al. [2009], VisTrails [2009], McCann and Pollard [2009; 2012], Grossman et al. [2010], Noris

et al. [2012], Denning and Pellacini [2013] , Chen et al. [2014], Matzen and Snavely [2014], Karsch et al. [2014]. Chronicle [Grossman et al. 2010]

slide-20
SLIDE 20

Related Work

  • Decomposing edits
  • Xu et al. [2006], Amati and Brostow [2010], Fu et al. [2011], Hu et al. [2013], Richardt et al. [2014].

Inverse Image Editing [Hu et al. 2013]

slide-21
SLIDE 21

Related Work

  • Image matting
  • Smith and Blinn [1996], Zongker et al. [1999], Farid and Adelson [1999], Szeliski et al. [2000],

Levin et al. [2006; 2007] Blue Screen Matting [Smith and Blinn 1996]

slide-22
SLIDE 22

Pipeline

Preprocess Extract Layers Edit Input

slide-23
SLIDE 23

Pipeline

Preprocess Extract Layers Edit Input

slide-24
SLIDE 24

Pipeline

Preprocess Extract Layers Edit Input

slide-25
SLIDE 25

Pipeline

Preprocess Extract Layers Edit Input

slide-26
SLIDE 26

Pipeline

Preprocess Extract Layers Edit Input

slide-27
SLIDE 27

Pipeline

Preprocess Extract Layers Edit Input

slide-28
SLIDE 28

Pipeline

Preprocess Edit Input Extract Layers

slide-29
SLIDE 29

Pipeline

Preprocess Edit Input Extract Layers

slide-30
SLIDE 30

Pipeline

Preprocess Extract Layers Input Edit

slide-31
SLIDE 31

Pipeline

Preprocess Extract Layers Input Edit

slide-32
SLIDE 32

Pipeline

Preprocess Extract Layers Input Edit

slide-33
SLIDE 33

Pipeline

Preprocess Extract Layers Input Edit

slide-34
SLIDE 34

Preprocessing Overview

slide-35
SLIDE 35

Preprocessing Overview

slide-36
SLIDE 36

Preprocessing Overview

time

slide-37
SLIDE 37

time

Preprocessing Overview

slide-38
SLIDE 38

time

Preprocessing Overview

The value of an unblocked pixel should be piecewise constant in time (stable)

slide-39
SLIDE 39

time

Preprocessing Overview

The value of an unblocked pixel should be piecewise constant in time (stable) Identical sequences of stable frames provide checkpoints for the painting progress

slide-40
SLIDE 40

Preprocessing

time

slide-41
SLIDE 41

Preprocessing

time

slide-42
SLIDE 42

Preprocessing

time

slide-43
SLIDE 43

Preprocessing

time

slide-44
SLIDE 44

Preprocessing

  • See paper for:
  • illumination
  • color shift
  • noise removal
  • 1D L0 smoothing

and
 bilateral filtering

slide-45
SLIDE 45

Preprocessing

  • See paper for:
  • illumination
  • color shift
  • noise removal
  • 1D L0 smoothing

and
 bilateral filtering

slide-46
SLIDE 46

?

Recovering Layers

before after + =

slide-47
SLIDE 47
  • paque solution
  • ur solution

?

Recovering Layers

before after + =

slide-48
SLIDE 48

Recovering Layers

slide-49
SLIDE 49

Recovering Layers

Model Porter-Duff (1983) Kubelka-Munk (1931)

slide-50
SLIDE 50

Recovering Layers

Model Porter-Duff (1983) Kubelka-Munk (1931) The standard for: digital compositing physical compositing

slide-51
SLIDE 51

Recovering Layers

Model Porter-Duff (1983) Kubelka-Munk (1931) The standard for: digital compositing physical compositing Compositing operation: Linear Non-linear

slide-52
SLIDE 52

Recovering Layers

Model Porter-Duff (1983) Kubelka-Munk (1931) The standard for: digital compositing physical compositing Compositing operation: Linear Non-linear Used in graphics: Almost everywhere Occasionally

Lu et al. [2014], …

slide-53
SLIDE 53

Porter-Duff Model

  • “Over” operator:

After = Before · (1 − α) + Paint · α

slide-54
SLIDE 54

Porter-Duff Model

  • “Over” operator:

Before

After = Before · (1 − α) + Paint · α

slide-55
SLIDE 55

Porter-Duff Model

  • “Over” operator:

Before Paint

After = Before · (1 − α) + Paint · α

slide-56
SLIDE 56

Porter-Duff Model

  • “Over” operator:

Before Paint After

After = Before · (1 − α) + Paint · α

slide-57
SLIDE 57

Porter-Duff Model

  • “Over” operator:

Before Paint After

After = Before · (1 − α) + Paint · α

unknown

slide-58
SLIDE 58

Porter-Duff Model

before after

slide-59
SLIDE 59

Porter-Duff Model

before after RGB Color Space

slide-60
SLIDE 60

Porter-Duff Model

before after RGB Color Space

before after

slide-61
SLIDE 61

Porter-Duff Model

before after RGB Color Space

before after

I N VALI D VALID INVALID

before after

slide-62
SLIDE 62

Porter-Duff Model

before after Find solution that minimizes alpha RGB Color Space

before after

I N VALI D VALID INVALID

before after

slide-63
SLIDE 63

Porter-Duff Model

before after Find solution that minimizes alpha RGB Color Space

before after

I N VALI D VALID INVALID

before after

I N VALI D VALID INVALID

before after paint

slide-64
SLIDE 64

Porter-Duff Model

Layer (RGBA) before after + =

slide-65
SLIDE 65

before after

Kubelka-Munk Model

  • Layer model (mixing model can be found in paper)
slide-66
SLIDE 66
  • Layer model (mixing model can be found in paper)

Kubelka-Munk Model

canvas Transmittancecanvas: Reflectancecanvas:

thickness

slide-67
SLIDE 67
  • Layer model (mixing model can be found in paper)

Kubelka-Munk Model

canvas Transmittancecanvas: Reflectancecanvas:

before

thickness

slide-68
SLIDE 68
  • Layer model (mixing model can be found in paper)

Kubelka-Munk Model

canvas Transmittancecanvas: Reflectancecanvas:

before

paint Transmittancepaint: Reflectancepaint: ? ?

thickness

slide-69
SLIDE 69
  • Layer model (mixing model can be found in paper)

Kubelka-Munk Model

canvas paint

thickness

Reflectanceoverall:

slide-70
SLIDE 70
  • Layer model (mixing model can be found in paper)

Kubelka-Munk Model

canvas paint

thickness

after

Reflectanceoverall:

slide-71
SLIDE 71

? ?

  • Layer model (mixing model can be found in paper)

Kubelka-Munk Model

paint Transmittancepaint: Reflectancepaint:

slide-72
SLIDE 72

? ?

  • Layer model (mixing model can be found in paper)

Kubelka-Munk Model

paint Transmittancepaint: Reflectancepaint: Find solution that maximizes Transmittancepaint

slide-73
SLIDE 73

? ?

  • Layer model (mixing model can be found in paper)

Kubelka-Munk Model

paint Transmittancepaint: Reflectancepaint: Find solution that maximizes Transmittancepaint

recovered recovered

slide-74
SLIDE 74

Kubelka-Munk Model

before after Layer (on white canvas) Transmittance Reflectance

slide-75
SLIDE 75

Results Overview

slide-76
SLIDE 76

Results Overview

slide-77
SLIDE 77

Editing

  • Temporal-Spatial Selection:
slide-78
SLIDE 78

Editing

  • Coloring using Time Gradient :
slide-79
SLIDE 79

Editing

slide-80
SLIDE 80

Editing

slide-81
SLIDE 81

Editing

slide-82
SLIDE 82

Conclusion

  • A preprocessing method to get a clean, albedo video
slide-83
SLIDE 83

Conclusion

  • A preprocessing method to get a clean, albedo video
slide-84
SLIDE 84

Conclusion

  • Two types of solutions for extracting translucent layers
slide-85
SLIDE 85

Conclusion

  • Two types of solutions for extracting translucent layers
slide-86
SLIDE 86

Conclusion

  • Useful layers for editing
slide-87
SLIDE 87

Conclusion

  • Useful layers for editing
slide-88
SLIDE 88

Future Work

slide-89
SLIDE 89

Future Work

  • Camera and canvas calibration.
slide-90
SLIDE 90

Future Work

  • Camera and canvas calibration.
slide-91
SLIDE 91

Future Work

  • Camera and canvas calibration.
  • Single image layer extraction?
slide-92
SLIDE 92

Future Work

  • Camera and canvas calibration.
  • Single image layer extraction?
  • Apply layer data into more systems.
  • WetPaint [Bonanni et al. 2009]
  • Chronicle [Grossman et al. 2010]
  • …
slide-93
SLIDE 93

Future Work

  • Camera and canvas calibration.
  • Single image layer extraction?
  • Apply layer data into more systems.
  • WetPaint [Bonanni et al. 2009]
  • Chronicle [Grossman et al. 2010]
  • …
  • Apply our technique to art education.
slide-94
SLIDE 94

Thank You!

  • Contact Information
  • Jianchao Tan: jtan8@gmu.edu
  • Marek Dvorožňák: dvoromar@fel.cvut.cz
  • Daniel Sýkora: sykorad@fel.cvut.cz
  • Yotam Gingold: ygingold@gmu.edu
  • Project Website: https://cs.gmu.edu/~ygingold/timemap/
  • Artists: Marcello Barenghi, Matyáš Veselý, Dani Jones, semisecretsoftware (YouTube)
  • Sponsors:
  • United States National Science Foundation, Google.
  • Technology Agency of the Czech Republic, Czech Science Foundation, Grant Agency of the

Czech Technical University in Prague

slide-95
SLIDE 95

P-D and K-M Comparison

P-D K-M

Layers Input

slide-96
SLIDE 96

P-D and K-M Comparison

P-D K-M

Layers Input

slide-97
SLIDE 97

P-D and K-M Comparison

P-D K-M

Layers Input

slide-98
SLIDE 98

P-D and K-M Comparison

P-D K-M

Layers Input

slide-99
SLIDE 99

Preprocessing Comparison

0.1 0.1 1.5 0.3 0.3 3.0 0.5 0.5 4.5 0.7 0.7 6.0 0.9 0.9 7.5

[Godbehere et al. 2012] [Zivkovic and van der Heijden 2006]

Our method Repaired frame

slide-100
SLIDE 100

Closest-Paint Method