Virtual Cinematography Theory and Practice for Automatic Real-Time - - PowerPoint PPT Presentation
Virtual Cinematography Theory and Practice for Automatic Real-Time Camera Control and Directing Liwei He Microsoft Research http://research.microsoft.com/users/lhe Motivation There are 3 elements in computer graphics lights, scene
Virtual Cinematography
Theory and Practice for Automatic Real-Time Camera Control and Directing
Liwei He
Microsoft Research http://research.microsoft.com/users/lhe
There are 3 elements in computer graphics
– lights, scene objects, and – the camera (focus of this talk)
Camera control is hard (7 DOFs)
– position (3), direction (3), field of view (1)
We may learn from cinematography
Low-level responsibilities
– Cameraman
– Film editor
High-level responsibilities
– Script writer and director
Input: camera position and direction Output: camera transformation Application: low-level graphics library
Input: show both A and B, follow C, etc. Output: camera position and direction Application: 3D game, computer
Input: high-level user directions
– show a conversation – show a car chase scene
Output: a sequence of level 2 camera spec Applications: graphical chat, tele-
conferencing, Virtual Reality games
Encodes cinematographic expertise
–an interesting problem in itself
Provides an interface that is
–real-time –visually entertaining –more informative
Eyepeep BARG07011 Feldegast
Microsoft V-Chat Lunar Island
Eyepeep says "really" Feldegast says "yup" BARG07011 has joined the conversation Eyepeep says "im still older" Feldegast says "lol" DGeste309 does something silly. BARG07011 DGerste309 Eyepeep Feldegast GrenDayGod
A B
A B
A B
A B
A B
A B
pan(A) track(B)
Don't cross the line of interest Avoid jump cuts Let the actor lead Break movement
1 2
Don't cross the line of interest Avoid jump cuts Let the actor lead Break movement
Don't cross the line of interest Avoid jump cuts Let the actor lead Break movement
A B 1 2 3 4
2 3 1
A B A B A
4
B
2 4 3 1 B A C
4 3 2
A B C A B
1
A B C
x y z Universe
The universe space
The eye space (z-axis is the look at direction) x y z Universe z Eye x y
The screen space (screen is ctan(fov/2) away from eye point) x y z Eye Screen
+1 +1
Given eye position Pfrom, a look at direction Want a rotation matrix R and a translation to transform a vector from universe space to eye space x y
T ˆ T ˆ
x y z Universe Pfrom Eye
x y z Universe Pfrom Eye
But this is not unique
T ˆ
x y
x y z Universe Pfrom Eye
But this is not unique
T ˆ
x y
x y z Universe Pfrom Eye
But this is not unique
T ˆ
y x
x y z Universe Pfrom Eye
We will need to specify an up vector , usually [0,1,0]
U ˆ U ˆ T ˆ
x y z Universe Pfrom Eye
In eye space Z-axis is X-axis is Y-axis is Rotation matrix Translation is -Pfrom
T ˆ Z U V ˆ ˆ ˆ × =
V T U ˆ ˆ ˆ × = "
T U V R ˆ ˆ ˆ
3 3
=
×
U ˆ U ˆ! V ˆ T ˆ
x y z Universe Pfrom Eye Pat
U ˆ! V ˆ T ˆ
Given eye position Pfrom, look at point Pat, and Pat in screen space [xat, yat] Want In universe space
T ˆ A ˆ
) ( ˆ
from at
P P unit A − =
In screen space: H = [xat, yat, ctan(fov/2)] Solve from the following equations: x y z Eye Screen
+1 +1
Pfrom Pat H
1 ˆ ˆ ˆ ˆ ] , 1 , [ ˆ ˆ ˆ ] 1 , , [ ˆ = ! ⋅ ! ⋅ = ⋅ ⋅ = ⋅ U T U A H T A H
T ˆ
x y z Universe Pfrom Eye PB PA
U ˆ! V ˆ T ˆ
Given PA,PB in universe space and eye space, and distance of eye position Pfrom and Pa. Want Pfrom and T
ˆ A ˆ
Use numerical method:
[xat,yat] method
R, getting a new Pfrom approximation
x y z Eye Screen
+1 +1
Pfrom PB H PA
Jim Blinn’s CG&A ’88 article
–Where am I? What am I looking at?
Steve Drucker’s Interactive 3D ’92, ’94,
–Cast camera positioning as an optimization problem
Real-time Application Virtual Cinematographer Renderer
– ( subject, verb, object )
– camera specifications – acting hints
Real-time Application Virtual Cinematographer Renderer
– ( subject, verb, object )
– camera specifications – acting hints
Real-time Application Virtual Cinematographer Renderer
– ( subject, verb, object )
– camera specifications – acting hints
Real-time Application Virtual Cinematographer Renderer
Actor’s mind Loneliness Thirst Boredom
Walk, converse, look around,
drink, etc.
Can be controlled by the users
Server Client Client Client User actions Events
Geometric placement of specific
Choose the side of the line of interest Influence acting
A B
A B
ext(B,A) ext(A,B) apex(A,B) int(B)
A B
pan(A) track(B)
Geometric placement of specific
Choose the side of the line of interest Influence acting
1 2
Geometric placement of specific
Choose the side of the line of interest Influence acting
A B A’ B’
Dave Christianson’s AAAI ’96 paper
–Declarative camera control for automatic cinematography –An off-line algorithm –Need to know all events ahead –Use plan algorithms in AI
Capture a particular type of scene Register relevant events Select shot types Determine the transitions between shots
Lookaround Master Converse Drinking Moving 3Talk 2Talk
Avoids exponential growth of states Allows expertise to be reused Provides robustness
track(C) pan(A) ext(A,B) ext(B,A) apex(A,B) apex(A,B)
track(C) pan(A) ext(A,B) ext(B,A) apex(A,B) apex(A,B) T > 8
track(C) pan(A) ext(A,B) ext(B,A) apex(A,B) apex(A,B) T > 8
track(C) pan(A) ext(A,B) ext(B,A) apex(A,B) apex(A,B) D < 10
track(C) pan(A) ext(A,B) ext(B,A) apex(A,B) apex(A,B) D < 5
apex(A,B) D < 5 apex(A,B) pan(A) ext(B,A) T > 8 ext(A,B) D < 10 T > 8
int(C) int(C) ext1to2
2Talk(A,B)
int(C) int(C) ext1to2
2Talk(A,B)
ext(B,A) ext(B,A) ext(A,B) ext(A,B)
2Talk
Real-time camera control
– lightweight – automatic camera placement – automatic shot transitions
Do a reasonable job
– not to replace human in computer animation
Multiple cameras per classroom
–Lecturer cameras –Audience camera
Use audio and vision techniques to
–Track the lecturer –Activate the audience camera
Virtual Cinematography
Theory and Practice for Automatic Real-Time Camera Control and Directing
Liwei He
Microsoft Research http://research.microsoft.com/users/lhe