SLIDE 1
Artistic Stylization and Rendering
Aaron Hertzmann Adobe Research San Francisco
SLIDE 2 class Nullspace implements Constants, Cloneable { /** The rows of the nullspace */ Vector rows = new Vector(); /** A list of the variables currently contained in the nullspace */ Vector variables = new Vector(); /** Add a constraint to the nullspace * * @param c The new constraint * @return True if the new constraint is already consistent with the * existing nullspace */ boolean add(Constraint c) { // Convert the Constraint into a Row // do this first to combine equivalent angles; might zero Row newRow = new Row(c); // Check if c contains any variables that the nullspace doesn't // If so, add them for(int i=0;i<newRow.sources.size();i++) { Object src = newRow.sources.elementAt(i); if (src instanceof AngleMeasure) src = ((AngleMeasure)src).getEquivalent(); if (variables.indexOf(src) < 0) addVariable(src); } int nk = rows.size(); // n-k = num vars - num constraints int[] Nx = new int[nk]; boolean zero = true; int pivot = -1; // compute N * x, where N is the nullspace and x is the new row for(int i=0;i<nk;i++) { Nx[i] = Row.dot((Row)rows.elementAt(i),newRow); if (Nx[i] != 0) { zero = false; pivot = i; } } // test if the new constraint was already consistent if (zero) return true;
SLIDE 3
SLIDE 4
SLIDE 5
SLIDE 6
SLIDE 9
SLIDE 10
SLIDE 11
SLIDE 12 Hertzmann, SIGGRAPH 1998
SLIDE 13 Hertzmann, NPAR 2000
SLIDE 14
SLIDE 15
Non-photorealistic rendering: computer graphics and animation inspired by natural artistic media
SLIDE 16
- 1. Scientific models for art
Research goals
SLIDE 19 The development of art and technology have always gone hand-in-hand
SLIDE 20 3D Non-Photorealistic Rendering
Smooth surface Occluding contours Stylized rendering
SLIDE 21 Occluding Contours
Weiss 1966
SLIDE 22 Suggestive Contours
DeCarlo et al. SIGGRAPH 2003
Camera view Contours Contours+SC
SLIDE 23 Studies on line drawing
Cole et al. SIGGRAPH 2008
SLIDE 24 Stylized Contour Algorithms
[Hertzmann and Zorin 2000] [Grabli et al. 2010] [Kalnins et al. 2003] [Buchholz et al. 2011] [Eisemann et al. 2008]
SLIDE 25 Bénard et al. NPAR 2012
SLIDE 26
Disney’s Paperman
SLIDE 27 Procedural methods
Pro: lovely results, very controllable Cons: hard to design styles,
complex to implement
SLIDE 28
What is texture?
SLIDE 29
SLIDE 30
What is texture?
SLIDE 31 Early Texture models
Haralick 1973
SLIDE 32 Higher-Order Statistics
Portilla and Simoncelli 2000
SLIDE 33 Higher-Order Statistics
Portilla and Simoncelli 2000
SLIDE 34 Higher-Order Statistics
Portilla and Simoncelli 2000
SLIDE 35 Higher-Order Statistics
Portilla and Simoncelli 1999
SLIDE 36 Patch-Based Texture
Efros and Leung 1999
Input texture Output texture
SLIDE 37 Patch-Based Texture
Efros and Leung 1999
Input texture Output texture
SLIDE 38 Input texture Output texture
Efros and Leung 1999, Wei and Levoy 2000
SLIDE 39 Results
Efros and Leung 1999, Wei and Levoy 2000
SLIDE 40
Curve stylization
SLIDE 41 Curve Propagation
Frame 1
?
Frame 2
SLIDE 42 Image Analogies
Goal: Process an image by example
?
: : ::
B B’ A A’
Hertzmann et al. SIGGRAPH 2001
SLIDE 43 :
A’
: ::
A B’ B
Hertzmann et al. SIGGRAPH 2001
SLIDE 44 : :
A’ A B’ B
::
Hertzmann et al. SIGGRAPH 2001
SLIDE 45 Blur
A A’ B B’
Hertzmann et al. SIGGRAPH 2001
SLIDE 46 Superresolution
Hertzmann et al. SIGGRAPH 2001
SLIDE 47 Texture transfer
A A’ (same texture) B
Closer to texture Closer to photo
B’s
SLIDE 48
SLIDE 49 Color channels
Input image Luminance Color channels
SLIDE 50 Color channels
Luminance Blurry color
SLIDE 51 Color channels
Blurry luminance Color channels
SLIDE 52 Color transfer
Example luminance Input luminance Input photo Input colors Output luminance
+
Output
image
SLIDE 53
SLIDE 54
SLIDE 55
SLIDE 56
SLIDE 57
SLIDE 59
SLIDE 60
SLIDE 61
:
A A’
:
B B’
::
SLIDE 62
:
A A’ B
::
B’
:
SLIDE 63
SLIDE 64
SLIDE 65 Image Analogies for Animation
Bénard et al. SIGGRAPH 2013
SLIDE 66 Illumination-Guided Example-Based Stylization of 3D Renderings
StyLit
1 CTU in Prague, FEE 2Adobe
Research
Jakub Fišer 1
Eli Shechtman 2 Ondřej Jamriška 1 Paul Asente 2 Daniel Sýkora 1 Michal Lukáč 1 Jingwan Lu 2
SLIDE 67
Neural texture
SLIDE 68
Can we model statistical textures with neural networks?
SLIDE 69 Texture synthesis
Gatys et al., NIPS 2015
SLIDE 70 Texture synthesis
Gatys et al., NIPS 2015
SLIDE 71
Neural stylization
SLIDE 72
Neural Style Transfer
SLIDE 73
Results
SLIDE 74
SLIDE 75 Where are we?
Procedural NPR Patch-Based (Analogies) Neural How do we get the best of each?
SLIDE 76
Adding control to neural stylization
SLIDE 77 Color Control - Color Preservation
Gatys et al., arXiv 2016
SLIDE 78
Color Preservation
SLIDE 79
Color Preservation
SLIDE 80 Color Control - Luminance Style Transfer
(a) (b)
Stylize
Gatys et al., arXiv 2016
SLIDE 81
Color Preservation
SLIDE 82 Spatial Control
Gatys et al., arXiv 2016
SLIDE 83 Spatial Control
Gatys et al., arXiv 2016
SLIDE 84 Spatial Control
Guidance Channels No control
SLIDE 85 Spatial Control
Gatys et al., arXiv 2016
SLIDE 86
Neural animation
SLIDE 87 Where are we?
Procedural NPR Patch-Based (Analogies) Neural Open question: How do we get the best of each?
SLIDE 88 Open problems
How do we author images? Learning style from large datasets Detailed control of style Creating 3D animation Making the details look good Make the fast methods better What is style? What is texture?
