Corpus-based Visual Synthesis: An Approach to Artistic Stylization P - - PowerPoint PPT Presentation

▶
corpus based visual synthesis an approach to artistic
SMART_READER_LITE
LIVE PREVIEW

Corpus-based Visual Synthesis: An Approach to Artistic Stylization P - - PowerPoint PPT Presentation

Sep 24, 2022 278 likes •1.01k views

Corpus Images Source Image Stylization Corpus-based Visual Synthesis: An Approach to Artistic Stylization P a r a g K . M i t a l 1 M i c k G r i e r s o n 1 T i m S m i t h 2 1 Department of Computing , Goldsmit hs, University of London 2

slide-1
SLIDE 1

A C M S y m p o s i u m O n A p p l i e d P e r c e p t i o n | 2 3 A u g u s t 2 0 1 3

Corpus-based Visual Synthesis: An Approach to Artistic Stylization

P a r a g K . M i t a l 1 M i c k G r i e r s o n 1 T i m S m i t h 2

Source Image Corpus Images Stylization

1Department of Computing , Goldsmit hs, University of London 2Department of Psychological Sciences , Bir k b e ck , University of London

slide-2
SLIDE 2

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71 2

Representation

✤ Perception must be supported by pre-

attentive representations

✤ Abstract description of objects/scenes ✤ Numerous theories: ✤ Gestalts ✤ Geons (Biederman) ✤ Object files (Wolfe) ✤ Proto-objects (Rensink) ✤ Indexicals (Pylyshyn) ✤ Shapes (Marr) ✤ Streams (Bregman)

slide-3
SLIDE 3

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71 3

Representation

✤ Perception must be supported by pre-

attentive representations

✤ Abstract description of objects/scenes ✤ Numerous theories: ✤ Gestalts ✤ Geons (Biederman) ✤ Object files (Wolfe) ✤ Proto-objects (Rensink) ✤ Indexicals (Pylyshyn) ✤ Shapes (Marr) ✤ Streams (Bregman)

slide-4
SLIDE 4

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71

Representation

4

✤ Perception must be supported by pre-

attentive representations

✤ Abstract description of objects/scenes ✤ Numerous theories: ✤ Gestalts ✤ Geons (Biederman) ✤ Object files (Wolfe) ✤ Proto-objects (Rensink) ✤ Indexicals (Pylyshyn) ✤ Shapes (Marr) ✤ Streams (Bregman)

slide-5
SLIDE 5

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71

Representation

5

✤ Perception must be supported by pre-

attentive representations

✤ Abstract description of objects/scenes ✤ Numerous theories: ✤ Gestalts ✤ Geons (Biederman) ✤ Object files (Wolfe) ✤ Proto-objects (Rensink) ✤ Indexicals (Pylyshyn) ✤ Shapes (Marr) ✤ Streams (Bregman)

slide-6
SLIDE 6

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71

Representation

6

✤ Perception must be supported by pre-

attentive representations

✤ Abstract description of objects/scenes ✤ Numerous theories: ✤ Gestalts ✤ Geons (Biederman) ✤ Object files (Wolfe) ✤ Proto-objects (Rensink) ✤ Indexicals (Pylyshyn) ✤ Shapes (Marr) ✤ Streams (Bregman)

slide-7
SLIDE 7

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71

Representation

7

✤ Perception must be supported by pre-

attentive representations

✤ Abstract description of objects/scenes ✤ Numerous theories: ✤ Gestalts ✤ Geons (Biederman) ✤ Object files (Wolfe) ✤ Proto-objects (Rensink) ✤ Indexicals (Pylyshyn) ✤ Shapes (Marr) ✤ Streams (Bregman)

slide-8
SLIDE 8

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71

Representation

8

✤ Artists are well aware of the role of

abstract representations in perception

✤ Influence how we look at and where we

look in a scene

✤ Art Movements ✤ Impressionism ✤ Pointilism ✤ Cubism ✤ Orphism ✤ Expressionism ✤ Abstract-Expressionism

slide-9
SLIDE 9

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71

Representation

9

✤ Artists are well aware of the role of

abstract representations in perception

✤ Influence how we look at and where we

look in a scene

✤ Art Movements ✤ Impressionism ✤ Pointilism ✤ Cubism ✤ Orphism ✤ Expressionism ✤ Abstract-Expressionism

slide-10
SLIDE 10

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71

Representation

10

✤ Artists are well aware of the role of

abstract representations in perception

✤ Influence how we look at and where we

look in a scene

✤ Art Movements ✤ Impressionism ✤ Pointilism ✤ Cubism ✤ Orphism ✤ Expressionism ✤ Abstract-Expressionism

slide-11
SLIDE 11

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71

Representation

11

✤ Artists are well aware of the role of

abstract representations in perception

✤ Influence how we look at and where we

look in a scene

✤ Art Movements ✤ Impressionism ✤ Pointilism ✤ Cubism ✤ Orphism ✤ Expressionism ✤ Abstract-Expressionism

slide-12
SLIDE 12

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71

Representation

12

✤ Artists are well aware of the role of

abstract representations in perception

✤ Influence how we look at and where we

look in a scene

✤ Art Movements ✤ Impressionism ✤ Pointilism ✤ Cubism ✤ Orphism ✤ Expressionism ✤ Abstract-Expressionism

slide-13
SLIDE 13

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71

Representation

13

✤ Artists are well aware of the role of

abstract representations in perception

✤ Influence how we look at and where we

look in a scene

✤ Art Movements ✤ Impressionism ✤ Pointilism ✤ Cubism ✤ Orphism ✤ Expressionism ✤ Abstract-Expressionism

slide-14
SLIDE 14

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71

Representation

14

✤ Artists are well aware of the role of

abstract representations in perception

✤ Influence how we look at and where we

look in a scene

✤ Art Movements ✤ Impressionism ✤ Pointilism ✤ Cubism ✤ Orphism ✤ Expressionism ✤ Abstract-Expressionism

slide-15
SLIDE 15

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71

Related Work

15

✤ Filtering/Clustering ✤ Example-based Images ✤ Texture-transfer/Patch-based ✤ Dictionary methods/Collage approaches

Kyprianidis, J. et al., 2012. State of the “Art”: A Taxonomy of Artistic Stylization Techniques for Images and Video. IEEE transactions on Visualization and Computer Graphics. Available at: http://ieeexplore.ieee.org/xpls/abs_all.jsp? arnumber=6243138

slide-16
SLIDE 16

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71

Problem Statement

16

✤ Our approach ✤ Build corpus of abstract representations

from user chosen images

✤ Match target image’s abstract

representations to nearest ones in corpus

✤ Synthesize target image using closest

matches

✤ Interact with a simple set of parameters

effecting representation detection and synthesis

✤ Create automated artistic

stylizations of images/videos using an understanding of the role of abstract representations in art and perception

✤ Allow for a range of styles

through a simple set of parameters

✤ Needs to be fast in order to

explore different styles quickly / run in real-time

slide-17
SLIDE 17

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71

Outline

✤ CBVS Framework ✤ Results for Images/Videos ✤ Extensions ✤ Memory Mosaicing ✤ Augmented Reality Hallucinations ✤ PhotoSynthesizer (iOS app) ✤ Conclusion

17

slide-18
SLIDE 18

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71

Outline

✤ CBVS Framework ✤ Results for Images/Videos ✤ Extensions ✤ Memory Mosaicing ✤ Augmented Reality Hallucinations ✤ PhotoSynthesizer (iOS app) ✤ Conclusion

18

slide-19
SLIDE 19

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71

CBVS Framework

19

✤ Build corpus of abstract representations

from user chosen images

✤ Match target image’s abstract

representations to nearest ones in corpus

✤ Synthesize target image using closest

matches

✤ Interact with a simple set of parameters

effecting representation detection and synthesis

RC = {R1, R2, ..., RNC}

slide-20
SLIDE 20

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71

Building the Corpus

20

✤ Need to represent: ✤ Sparse/Dense strokes ✤ Small/Large strokes ✤ Watershed? ✤ Posterization? ✤ Mean-Shift?

slide-21
SLIDE 21

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71

Building the Corpus

21

✤ Maximally Stable Color Regions ✤ No need for multiple scale detections ✤ Implicit ordering of regions ✤ Simple set of parameters for

discovering sparse/dense small/ large strokes

✤ Fast/Robust across multiple views

(used in video tracking)

✤ Similar process to the unconscious

representations as theorized before

slide-22
SLIDE 22

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71 22

slide-23
SLIDE 23

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71 23

slide-24
SLIDE 24

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71 24

slide-25
SLIDE 25

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71 25

slide-26
SLIDE 26

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71 26

slide-27
SLIDE 27

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71 27

slide-28
SLIDE 28

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71 28

slide-29
SLIDE 29

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71 29

Increasing timesteps = Denser detection More expressive corpus

slide-30
SLIDE 30

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71

CBVS Framework

30

✤ Build corpus of abstract representations

from user chosen images

✤ Match target image’s abstract

representations to nearest ones in corpus

✤ Synthesize target image using closest

matches

✤ Interact with a simple set of parameters

effecting representation detection and synthesis

RC = {R1, R2, ..., RNC} RT = {R1, R2, ..., RNT }

?

slide-31
SLIDE 31

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71

Matching

31

✤ Need to match regions in target to similar

  • nes in corpus

✤ Describe Shape and Color ✤ Use Euclidean distance for shape values ✤ Use Perceptual distance for color values

(CIEDE2000 formula)

✤ Nearest neighbor matching

slide-32
SLIDE 32

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71 32

✤ Need to match regions in target to similar

  • nes in corpus

✤ Describe Shape and Color ✤ Use Euclidean distance for shape values ✤ Use Perceptual distance for color values

(CIEDE2000 formula)

✤ Nearest neighbor matching

dRi = ⇣ µ00, η11, η20, η02, L, a∗, b∗⌘

ηij = µij µ(1+ i+j

2 )

00

Matching

slide-33
SLIDE 33

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71 33

✤ Need to match regions in target to similar

  • nes in corpus

✤ Describe Shape and Color ✤ Use Euclidean distance for shape values ✤ Use Perceptual distance for color values

(CIEDE2000 formula)

✤ Nearest neighbor matching

d(Rt, Rc) = ds(Rt, Rc) + dc(Rt, Rc)

Matching

slide-34
SLIDE 34

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71

CBVS Framework

34

✤ Build corpus of abstract representations

from user chosen images

✤ Match target image’s abstract

representations to nearest ones in corpus

✤ Synthesize target image using closest

matches

✤ Interact with a simple set of parameters

effecting representation detection and synthesis

slide-35
SLIDE 35

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71

Synthesis

35

✤ From largest to smallest target region ✤ Find nearest neighbor ✤ Translate ✤ Rotate ✤ Scale ✤ Blend

d(Rt, Rc) = ds(Rt, Rc) + dc(Rt, Rc)

slide-36
SLIDE 36

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71

Synthesis

36

T = centroidRt − centroidRc

✤ From largest to smallest target region ✤ Find nearest neighbor ✤ Translate ✤ Rotate ✤ Scale ✤ Blend

slide-37
SLIDE 37

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71

Synthesis

37

Θ = 1 2 ∗ arctan 2 ∗ µ11

µ00 µ20 µ00 − µ02 µ00

✤ From largest to smallest target region ✤ Find nearest neighbor ✤ Translate ✤ Rotate ✤ Scale ✤ Blend

slide-38
SLIDE 38

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71

Synthesis

38

Sx = widthRT widthRC Sy = heightRT heightRC

✤ From largest to smallest target region ✤ Find nearest neighbor ✤ Translate ✤ Rotate ✤ Scale ✤ Blend

slide-39
SLIDE 39

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71

Synthesis

39

✤ From largest to smallest target region ✤ Find nearest neighbor ✤ Translate ✤ Rotate ✤ Scale ✤ Blend

slide-40
SLIDE 40

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71

Interaction

✤ Spatial blending ✤ Temporal blending ✤ Motion tracking ✤ Timesteps ✤ Minimum region size ✤ Maximum region size ✤ Blending radius

40

Discrete Parameters Continuous Parameters

slide-41
SLIDE 41

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71

Spatial Blending

41

slide-42
SLIDE 42

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71

Temporal Blending

42

slide-43
SLIDE 43

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71

Temporal Blending

43

slide-44
SLIDE 44

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71 44

Increasing timesteps = Denser layers, More expressive

Timesteps

slide-45
SLIDE 45

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71 45

Decreasing minimum size = Finer brush strokes

Minimum Size

slide-46
SLIDE 46

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71 46

Increasing radius = More of source texture

Blending Radius

slide-47
SLIDE 47

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71

Outline

✤ CBVS Framework ✤ Results for Images/Videos ✤ Extensions ✤ Memory Mosaicing ✤ Augmented Reality Hallucinations ✤ Conclusion

47

slide-48
SLIDE 48

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71 48

Target Image Corpus Synthesis

slide-49
SLIDE 49

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71 49

slide-50
SLIDE 50

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71 50

Target Image Corpus Synthesis

slide-51
SLIDE 51

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71 51

slide-52
SLIDE 52

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71 52

Target Image Corpus Synthesis

slide-53
SLIDE 53

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71 53

slide-54
SLIDE 54

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71 54

Target Image Corpus

slide-55
SLIDE 55

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71 55

slide-56
SLIDE 56

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71 56

slide-57
SLIDE 57

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71 57

slide-58
SLIDE 58

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71 58

slide-59
SLIDE 59

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71 59

slide-60
SLIDE 60

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71

Outline

✤ CBVS Framework ✤ Results for Images/Videos ✤ Extensions ✤ Memory Mosaicing ✤ Augmented Reality Hallucinations ✤ PhotoSynthesizer (iOS app) ✤ Conclusion

60

slide-61
SLIDE 61

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71

Memory Mosaicing

✤ Dynamic target (movie or webcam) ✤ Aggregate corpus over time using target,

retaining only most recent N objects

✤ Only allow learning of objects with

distance greater than threshold

61

slide-62
SLIDE 62

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71 62

Live demo

slide-63
SLIDE 63

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71

Outline

✤ CBVS Framework ✤ Results for Images/Videos ✤ Extensions ✤ Memory Mosaicing ✤ Augmented Reality Hallucinations ✤ PhotoSynthesizer (iOS app) ✤ Conclusion

63

slide-64
SLIDE 64

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71

Augmented Reality Hallucinations

✤ Memory Mosaicing ✤ Uses Augmented Reality headset ✤ Exhibited during the Victoria & Albert

Museum’s Digital Design Weekend co- located during the London Design Festival, 15,000 participants

✤ Short questionnaire for participants

rating their experience (21 participants

  • nly)

64

slide-65
SLIDE 65

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71 65

slide-66
SLIDE 66

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71

Outline

✤ CBVS Framework ✤ Results for Images/Videos ✤ Extensions ✤ Memory Mosaicing ✤ Augmented Reality Hallucinations ✤ PhotoSynthesizer (iOS app) ✤ Conclusion

66

slide-67
SLIDE 67

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71

PhotoSynthesizer

✤ Free iOS app allows user to synthesize

target image

✤ No interaction besides selecting target

and corpus

✤ Reveals synthesis process as painting

regions over time

✤ Reached Top 50 app in Photo & Video in

many countries

67

slide-68
SLIDE 68

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71

PhotoSynthesizer

68

slide-69
SLIDE 69

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71 69

Live demo

slide-70
SLIDE 70

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71

Outline

✤ CBVS Framework ✤ Results for Images/Videos ✤ Extensions ✤ Memory Mosaicing ✤ Augmented Reality Hallucinations ✤ PhotoSynthesizer (iOS app) ✤ Conclusion

70

slide-71
SLIDE 71

Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com

/ 71

Conclusion

✤ Simple shape representation affords

range of stylizations and a range of non/ real-time applications

✤ Expressive control in a few parameters ✤ Future? ✤ Better method of evaluation ✤ Better metrics for shape description ✤ Better temporal coherence

71