Introduction to Game Programming Introduction to Game Programming - - PDF document

SLIDE 1

Introduction to Game Programming Introduction to Game Programming Autumn 2017 Autumn 2017 Autumn 2017 Autumn 2017

• 04. Graphics for games
• 04. Graphics for games

Juha Vihavainen Juha Vihavainen Juha Vihavainen Juha Vihavainen University of Helsinki University of Helsinki

Outline Outline

• Creating and drawing game entities

Creating and drawing game entities from 2D images ( from 2D images (sprites sprites) to 3D models ) to 3D models

• from 2D images (

from 2D images (sprites sprites) to 3D models ) to 3D models

• the position, dimensions, and visual data of game

the position, dimensions, and visual data of game

• bjects represented for computer graphics
• bjects represented for computer graphics
• 3D models ~ triangle meshes ~ geometry/shape

3D models ~ triangle meshes ~ geometry/shape (plus other info) (plus other info) On rendering pipeline On rendering pipeline

• On rendering pipeline

On rendering pipeline

• Creating and using

Creating and using virtual virtual (synthetic) (synthetic) in in-world world camera camera

2 3.3.2017 3.3.2017 Juha Vihavainen / University of Helsinki Juha Vihavainen / University of Helsinki

SLIDE 2

Literature on 3D graphics Literature on 3D graphics

• (Madhav, 2014)

(Madhav, 2014) Ch.

• Ch. 2 2D

2D Graphics Graphics, Ch. , Ch. 4 3D 3D Graphics, Graphics, Ch.

• Ch. 8

8 Cameras Cameras (Gregory, 2014) Ch. (Gregory, 2014) Ch. 10 10 The Rendering Engine, The Rendering Engine, pp. 443

• pp. 443-541;

541; 10.2 10.2

• (Gregory, 2014) Ch.

(Gregory, 2014) Ch. 10 10 The Rendering Engine, The Rendering Engine, pp. 443

• pp. 443-541;

541; 10.2 10.2 The Rendering Pipeline, p. 489. The Rendering Pipeline, p. 489.

• Dunn Fletcher, Parberry Ian:

Dunn Fletcher, Parberry Ian: 3D Math Primer for Graphics and 3D Math Primer for Graphics and Game Development Game Development. A K Peters/CRC Press, 2011. . A K Peters/CRC Press, 2011.

• Edward Angel, Dave Shreiner

Edward Angel, Dave Shreiner, Interactive Computer Graphics: A , Interactive Computer Graphics: A Top Top-Down Approach with Shader Down Approach with Shader-Based OpenGL, Based OpenGL, 6th Ed. Addison 6th Ed. Addison- Wesley, 2011. Wesley, 2011. Wesley, 2011. Wesley, 2011.

• Tomas Akenine

Tomas Akenine-Moller, Eric Haines, Naty Hoffman, Moller, Eric Haines, Naty Hoffman, Real Real-Time Time Rendering Rendering, 3rd Ed. A K Peters/CRC Press, 2008. , 3rd Ed. A K Peters/CRC Press, 2008.

• Junghyan Han,

Junghyan Han, 3D Graphics for Game Programming 3D Graphics for Game Programming. CRC, 2011 . CRC, 2011

3.3.2017 3.3.2017 Juha Vihavainen / University of Helsinki Juha Vihavainen / University of Helsinki 3

Motivation for linear algebra Motivation for linear algebra

• A game is (essentially) a mathematical model of a virtual world

A game is (essentially) a mathematical model of a virtual world simulated on a computer simulated on a computer a game engine needs to keep track of the positions, orientations, and a game engine needs to keep track of the positions, orientations, and

• a game engine needs to keep track of the positions, orientations, and

a game engine needs to keep track of the positions, orientations, and scales of objects, animate them in the game world, and transform scales of objects, animate them in the game world, and transform them into screen space so they can be rendered on screen them into screen space so they can be rendered on screen

• 3D objects are (almost always) made up of triangles, the vertices of

3D objects are (almost always) made up of triangles, the vertices of which are represented by vectors/points which are represented by vectors/points

• Mathematics pervades everything in game development

Mathematics pervades everything in game development games make use many branches of mathematics, e.g., trigonometry, games make use many branches of mathematics, e.g., trigonometry,

• games make use many branches of mathematics, e.g., trigonometry,

games make use many branches of mathematics, e.g., trigonometry, algebra, and calculus (often probability, too) algebra, and calculus (often probability, too)

• the most important kind of mathematics for a game programmer is

the most important kind of mathematics for a game programmer is vector vector and and matrix matrix math math, i.e., , i.e., linear algebra linear algebra

3.3.2017 3.3.2017 Juha Vihavainen / University of Helsinki Juha Vihavainen / University of Helsinki 4

SLIDE 3

Some 3D terminology Some 3D terminology

• Rasterization

Rasterization

• set of algorithms that draw 3D objects into a 2D

set of algorithms that draw 3D objects into a 2D color color buffer buffer (part of (part of frame buffer frame buffer with other data) with other data) buffer buffer (part of (part of frame buffer frame buffer with other data) with other data)

• Modern computers use a

Modern computers use a graphics processing unit graphics processing unit (GPU) that (GPU) that can do most rasterization can do most rasterization

• but need to tell the GPU

but need to tell the GPU what what we want to draw and we want to draw and how how

• Due to limited resources, can't achieve photo

Due to limited resources, can't achieve photo-realistic images realistic images Due to limited resources, can't achieve photo Due to limited resources, can't achieve photo-realistic images realistic images

• graphic

graphic (visual visual) artifact artifact = some unwanted result = some unwanted result (anomaly) from digital image processing (of using (anomaly) from digital image processing (of using approximate algorithms, or imprecise or corrupted data) approximate algorithms, or imprecise or corrupted data)

3.3.2017 3.3.2017 5 Juha Vihavainen / University of Helsinki Juha Vihavainen / University of Helsinki

Types of art assets Types of art assets

(Joseph Hocking, Unity in Action, 2015, p.70) textures + shaders

3.3.2017 3.3.2017 Juha Vihavainen / University of Helsinki Juha Vihavainen / University of Helsinki 6

SLIDE 4

A graphics system (sketch) A graphics system (sketch)

general appl.

• Consists of:

Consists of: input devices, CPU, GPU, memory, frame buffer, output input devices, CPU, GPU, memory, frame buffer, output

matrix & other computations

• input devices, CPU, GPU, memory, frame buffer, output

input devices, CPU, GPU, memory, frame buffer, output devices devices

• E.g., PCs, workstations, mobile phones, video game consoles,

E.g., PCs, workstations, mobile phones, video game consoles, GPS systems, etc. GPS systems, etc.

3.3.2017 3.3.2017 Juha Vihavainen / University of Helsinki Juha Vihavainen / University of Helsinki 7

CRT monitor basics CRT monitor basics

• 2D image = an array of

2D image = an array of picture elements picture elements known as known as pixels pixels

• for color displays, each pixel contains a

for color displays, each pixel contains a red red, , green green, and , and blue blue sub sub-pixel pixel an an RGBA RGBA value value - for Red, Green, and Blue plus for Red, Green, and Blue plus Alpha Alpha

• an

an RGBA RGBA value value - for Red, Green, and Blue plus for Red, Green, and Blue plus Alpha Alpha (alpha = 0 => (alpha = 0 => totally transparent image/part totally transparent image/part)

• Resolution width x height determines the number of pixels

Resolution width x height determines the number of pixels

• 300 x 200 means each row (or scan line) has 300 pixels

300 x 200 means each row (or scan line) has 300 pixels and there's a total of 200 rows. and there's a total of 200 rows.

• CRTs used an electron gun to activate the various pixels

CRTs used an electron gun to activate the various pixels

• Modern display technologies:

Modern display technologies: plasma plasma, , LCD LCD (liquid liquid-crystal crystal display display), etc. replaced ), etc. replaced CRT displays in (most) applications CRT displays in (most) applications

3.3.2017 3.3.2017 8 Juha Vihavainen / University of Helsinki Juha Vihavainen / University of Helsinki

SLIDE 5

CRT monitor basics (historically) CRT monitor basics (historically)

• Electron gun draws one scan

Electron gun draws one scan line at a time line at a time

• When it gets to bottom

When it gets to bottom-right right

Basic CRT drawing Basic CRT drawing

When it gets to bottom When it gets to bottom-right right corner, it takes some time to get corner, it takes some time to get the aim back to top left (vertical the aim back to top left (vertical blank interval or VBLANK) blank interval or VBLANK)

• a VSYNC

a VSYNC signal signal reports that reports that the drawing is complete the drawing is complete

• LCD displays require the timing

LCD displays require the timing signals of their own: VSYNC signals of their own: VSYNC signals of their own: VSYNC signals of their own: VSYNC (reset row pointer to top), (reset row pointer to top), HSYNC (reset column pointer HSYNC (reset column pointer to edge), LCDCLK (LCD clock to edge), LCDCLK (LCD clock to control display refresh rate) to control display refresh rate)

3.3.2017 3.3.2017 9 Juha Vihavainen / University of Helsinki Juha Vihavainen / University of Helsinki

Color buffer Color buffer

• A buffer is memory that

A buffer is memory that stores the color data for all stores the color data for all the pixels on the screen the pixels on the screen

Screen tearing caused by updating video data while drawing

the pixels on the screen the pixels on the screen

• With only a single color

With only a single color buffer, writing to it while buffer, writing to it while still displaying the old image still displaying the old image results in screen tearing results in screen tearing

• Meshes up

Meshes up

data while drawing

new image

• Meshes up

Meshes up info from two or more info from two or more frames in a single screen frames in a single screen draw draw

3.3.2017 3.3.2017 10 10 Juha Vihavainen / University of Helsinki Juha Vihavainen / University of Helsinki

• ld contents

SLIDE 6

Double buffering (as a game pattern) Double buffering (as a game pattern)

• Use two color buffers

Use two color buffers A and and B:

• while buffer

while buffer A is written to, display buffer is written to, display buffer B

• then next frame, write to

then next frame, write to B and display and display A

• An interrupt service routine can run to modify data in the video display

An interrupt service routine can run to modify data in the video display

• An interrupt service routine can run to modify data in the video display

An interrupt service routine can run to modify data in the video display memory while it is not being read (by a special signal from the device) memory while it is not being read (by a special signal from the device)

• Double Buffer

Double Buffer game programming pattern game programming pattern

• cause a series of sequential operations to appear instantaneous or

cause a series of sequential operations to appear instantaneous or simultaneous simultaneous (i.e., as (i.e., as atomic atomic)

• e.g., in game graphics,

e.g., in game graphics, the scene must update smoothly and quickly, the scene must update smoothly and quickly, displaying a series of complete frames, each appearing instantly displaying a series of complete frames, each appearing instantly displaying a series of complete frames, each appearing instantly displaying a series of complete frames, each appearing instantly

• Sometimes, even

Sometimes, even triple buffering triple buffering (three color buffers) (three color buffers)

• if the back buffer becomes filled while display is not ready => start

if the back buffer becomes filled while display is not ready => start filling a third buffer filling a third buffer

• potential drawback:

potential drawback: input lag input lag

3.3.2017 3.3.2017 11 11 Juha Vihavainen / University of Helsinki Juha Vihavainen / University of Helsinki

What is a graphic engine? What is a graphic engine?

• An API for real

An API for real-time 2D and 3D applications (games) time 2D and 3D applications (games)

• ften runs the "
• ften runs the "main loop

main loop" " => => a a software framework software framework Provides Provides development environment development environment for programmers for programmers

• Provides

Provides development environment development environment for programmers for programmers

• may target multiple platforms, e.g., PC, consoles,

may target multiple platforms, e.g., PC, consoles, smartphones, tablets smartphones, tablets

• A graphics engine provides a level of abstraction

A graphics engine provides a level of abstraction

• wraps

wraps DirectX, OpenGL DirectX, OpenGL or

• r Vulkan

Vulkan inside the provided API inside the provided API API calls may still closely mirror API calls may still closely mirror the low the low-level C level C API API

• API calls may still closely mirror

API calls may still closely mirror the low the low-level C level C API API

• shader programs

shader programs are written and run on the GPU are written and run on the GPU

12 12 3.3.2017 3.3.2017 Juha Vihavainen / University of Helsinki Juha Vihavainen / University of Helsinki

SLIDE 7

"Getting started is easier.. Getting started is easier.."

• Writing graphic applications

Writing graphic applications can be hard to get started can be hard to get started

• create

create your " your "graphics device graphics device"

• lots

lots of low

• f low-level

level initialization initialization and set and set-up up code code

• handle input and

handle input and other events (in a platform

• ther events (in a platform-independent way)

independent way)

• handle input and

handle input and other events (in a platform

• ther events (in a platform-independent way)

independent way)

• load/unload

load/unload various file formats various file formats of graphic/audio resources

• f graphic/audio resources
• A graphics engine

A graphics engine tries to tries to hide (most of) the complexity hide (most of) the complexity

• the framework provides already running "empty"

the framework provides already running "empty" application application

• Ideally, tools are integrated (XNA/MonoGame, Unreal, Unity)

Ideally, tools are integrated (XNA/MonoGame, Unreal, Unity)

games are written and debugged within an IDE or builder games are written and debugged within an IDE or builder

• games are written and debugged within an IDE or builder

games are written and debugged within an IDE or builder

• separate project kinds/builds for consoles, smartphones

separate project kinds/builds for consoles, smartphones

• content pipeline

content pipeline modifies assets (textures, 3D models) to a form more modifies assets (textures, 3D models) to a form more suitable for the game program (e.g., part of suitable for the game program (e.g., part of import import or

• r build phase

build phase)

• plus

plus audio tools to author, organize, and play audio assets, etc audio tools to author, organize, and play audio assets, etc

13 13 3.3.2017 3.3.2017 Juha Vihavainen / University of Helsinki Juha Vihavainen / University of Helsinki

Making a 2D game entity (" Making a 2D game entity ("sprite sprite") ")

• A"sprite" is often used to mean an element of a 2D

A"sprite" is often used to mean an element of a 2D game game

• usually

usually both a texture and a position on the both a texture and a position on the screen (state) screen (state)

• the texture is often shared but the position changes

the texture is often shared but the position changes possibly using " possibly using "speed speed" variable (say, " variable (say, new new vector vector2 (3, 3) 2 (3, 3))

• possibly using "

possibly using "speed speed" variable (say, " variable (say, new new vector vector2 (3, 3) 2 (3, 3))

• The position of a 2D object on the screen is given as coordinates

The position of a 2D object on the screen is given as coordinates based on screen pixels based on screen pixels

• some appropriate window size (e.g., 800

some appropriate window size (e.g., 800 * 480 pixels) 480 pixels)

• these define the range for display coordinates

these define the range for display coordinates Specify Specify source rectangle source rectangle (from a (from a sprite sheet sprite sheet) if want to draw ) if want to draw

• Specify

Specify source rectangle source rectangle (from a (from a sprite sheet sprite sheet) if want to draw ) if want to draw

• nly a part of an image
• nly a part of an image
• Scale an image (uniform, non

Scale an image (uniform, non-uniform along uniform along x and and y, rectangle y, rectangle)

• Sprite depth

Sprite depth (float float) between 0 (= front) and 1 (= back) ) between 0 (= front) and 1 (= back)

• We can draw things off the screen but then nothing is seen

We can draw things off the screen but then nothing is seen

14 14 3.3.2017 3.3.2017 Juha Vihavainen / University of Helsinki Juha Vihavainen / University of Helsinki

SLIDE 8

Implementation of sprites Implementation of sprites

• Sprites are 2D visual objects that represent

Sprites are 2D visual objects that represent

• game characters

game characters

• ther dynamic objects
• ther dynamic objects

for simple games, backgrounds, too for simple games, backgrounds, too

• for simple games, backgrounds, too

for simple games, backgrounds, too

• The corresponding image format can vary depending on the platform

The corresponding image format can vary depending on the platform (PNG, TGA, and PVR are potential options) (PNG, TGA, and PVR are potential options) class Sprite class Sprite ImageFile image; ImageFile image; // // (current current) ) texture texture int drawOrder; int drawOrder; // // using an using an int int value, here value, here int drawOrder; int drawOrder; // // using an using an int int value, here value, here int x, y; int x, y; // // specify location specify location (e.g., (e.g., top left corner top left corner) function Draw ( ) function Draw ( ) // draw the // draw the image image at the correct at the correct (x, y) . . ) . .

3.3.2017 3.3.2017 15 15 Juha Vihavainen / University of Helsinki Juha Vihavainen / University of Helsinki

Painter's algorithm Painter's algorithm

• Sprites are drawn from back to front: nearest images are placed at the top

Sprites are drawn from back to front: nearest images are placed at the top so that background images become hidden (more or less) so that background images become hidden (more or less)

Painter Painter’s algorithm applied to a 2D s algorithm applied to a 2D space scene space scene

3.3.2017 3.3.2017 16 16 Juha Vihavainen / University of Helsinki Juha Vihavainen / University of Helsinki

SLIDE 9

From 2D to 3D From 2D to 3D

• Definitely

Definitely more more complex than 2D complex than 2D graphics graphics

• 2D is

2D is 2D is 2D is

• like painting on a canvas: placement of sprites on a 2D screen

like painting on a canvas: placement of sprites on a 2D screen

• 3D simulates

3D simulates

• more

more-or

• r-less "solid" game entities located in a 3D world

less "solid" game entities located in a 3D world

• recording them with a freely

recording them with a freely-

• moving video camera

moving video camera

17 17 3.3.2017 3.3.2017 Juha Vihavainen / University of Helsinki Juha Vihavainen / University of Helsinki

Left Left- vs.

• vs. right

right-handed handed coordinate systems coordinate systems

left-handed right-handed

• item

item

3.3.2017 3.3.2017 Juha Vihavainen / University of Helsinki Juha Vihavainen / University of Helsinki 18 18

DirectX Unity OpenGL XNA/MonoGame Unreal (z-up world )

SLIDE 10

"3D scene 3D scene" in computer graphics " in computer graphics

• A 3D

A 3D scene scene is is

• "geometry" (triangles, lines, points, and more)

"geometry" (triangles, lines, points, and more)

• material

material surface properties of geometry surface properties of geometry material material surface properties of geometry surface properties of geometry

• how

how to draw it (a to draw it (a shader shader program) program)

• textures

textures (images to glue onto the geometry) (images to glue onto the geometry)

• light sources and their properties

light sources and their properties

• A triangle consists of 3 vertices

A triangle consists of 3 vertices

• a vertex is 3D position, and may include normals and more

a vertex is 3D position, and may include normals and more

• To take a picture, we need a

To take a picture, we need a camera camera – a a virtual virtual one

• ne
• decides what will end up in the final image

decides what will end up in the final image

3.3.2017 3.3.2017 Juha Vihavainen / University of Helsinki Juha Vihavainen / University of Helsinki 19 19

"Virtual" (synthetic) 3D camera "Virtual" (synthetic) 3D camera

• Defined by

Defined by position position, , direction direction vector, vector, up up vector, vector, field of view field of view (angle angle), ), near near and and far far plane plane

• Used by

Used by OpenGL OpenGL, , DirectX DirectX, ray tracing image processing, etc. , ray tracing image processing, etc.

• Create the image of those shapes located inside the frustum

Create the image of those shapes located inside the frustum

near far

world 2D projection

3.3.2017 3.3.2017 Juha Vihavainen / University of Helsinki Juha Vihavainen / University of Helsinki 20 20

point dir

fov(angle)

SLIDE 11

3D camera (cont.) 3D camera (cont.)

• Can have objects in a scene that aren’t (currently) visible

Can have objects in a scene that aren’t (currently) visible

• bjects can be rotated around three axes
• bjects can be rotated around three axes

a freely moving camera can also be rotated around all 3 axes a freely moving camera can also be rotated around all 3 axes

• a freely moving camera can also be rotated around all 3 axes

a freely moving camera can also be rotated around all 3 axes

• the camera determines what shows up on screen

the camera determines what shows up on screen

• f course, if the camera is pointing in the opposite direction
• f course, if the camera is pointing in the opposite direction

from an object, we won't see it from an object, we won't see it

21 21 3.3.2017 3.3.2017 Juha Vihavainen / University of Helsinki Juha Vihavainen / University of Helsinki

Models in 3D games Models in 3D games

• 3D games consist of large numbers of visible objects

3D games consist of large numbers of visible objects

• In practice, these objects are

In practice, these objects are not not created by game code by defining created by game code by defining each individual primitive (triangle) each individual primitive (triangle) each individual primitive (triangle) each individual primitive (triangle)

• Instead, artists create objects for a game world, game designers

Instead, artists create objects for a game world, game designers focus on behavior of those objects, and programmers mostly focus on behavior of those objects, and programmers mostly concentrate on support systems concentrate on support systems

• Special 3D modeling tools are used, e.g.,

Special 3D modeling tools are used, e.g.,

• Blender

Blender, 3ds , 3ds Max Max, , Maya

• Maya. .

. .

• Blender

Blender, 3ds , 3ds Max Max, , Maya

• Maya. .

. .

• modeling tools permit separate editing of models

modeling tools permit separate editing of models

• plug

plug-in tools allow models to be saved in a number of different in tools allow models to be saved in a number of different file formats file formats

22 22 3.3.2017 3.3.2017 Juha Vihavainen / University of Helsinki Juha Vihavainen / University of Helsinki

SLIDE 12

A sphere made of triangles A sphere made of triangles

3.3.2017 3.3.2017 Juha Vihavainen / University of Helsinki Juha Vihavainen / University of Helsinki 23 23

Drawing the triangles in GPU Drawing the triangles in GPU

• In 3D graphics engines, all rendering is ultimately handled by the GPU

In 3D graphics engines, all rendering is ultimately handled by the GPU and and shader shader programs programs

• shaders

shaders are small special are small special-purpose programs that execute inside the purpose programs that execute inside the GPU and define how the data received from a game program is GPU and define how the data received from a game program is processed in the programmable stages of the rendering pipeline processed in the programmable stages of the rendering pipeline processed in the programmable stages of the rendering pipeline processed in the programmable stages of the rendering pipeline

• shader programs are defined using a

shader programs are defined using a high level shader language high level shader language (e.g., Microsoft's (e.g., Microsoft's HLSL HLSL, OpenGL , OpenGL GLSL GLSL)

• Vulkan supports shaders in a bytecode format, called

Vulkan supports shaders in a bytecode format, called SPIR SPIR-V, as V, as

• pposed to human
• pposed to human-readable syntax like GLSL

readable syntax like GLSL and HLSL

and HLSL

• provide

provide wide range of visual effects wide range of visual effects not not part of this part of this course course

• not

not part of this part of this course course

• see (Gregory) for explanations and

see (Gregory) for explanations and examples of GPU programming examples of GPU programming

• Luckily, the game IDE / tools often

Luckily, the game IDE / tools often provide many provide many predefined predefined default shaders default shaders

24 24 3.3.2017 3.3.2017 Juha Vihavainen / University of Helsinki Juha Vihavainen / University of Helsinki

SLIDE 13

Rendering data Rendering data

• Shared data between the CPU and the GPU

Shared data between the CPU and the GPU

• 3D model data: streams of vertices specifying triangles..

3D model data: streams of vertices specifying triangles..

• matrices

matrices that represent object transforms and skeletal that represent object transforms and skeletal animation animation

• lighting parameters, and

lighting parameters, and

• ther kinds of "shader constants" used for computations
• ther kinds of "shader constants" used for computations
• Data (almost exclusively) produced and managed by the GPU

Data (almost exclusively) produced and managed by the GPU vertex and index buffers, frame buffers vertex and index buffers, frame buffers

• vertex and index buffers, frame buffers

vertex and index buffers, frame buffers

3.3.2017 3.3.2017 Juha Vihavainen / University of Helsinki Juha Vihavainen / University of Helsinki 25 25

Drawing shapes as sets of Drawing shapes as sets of triangles triangles

• 3D shapes (in games) are (ultimately) composed of triangles

3D shapes (in games) are (ultimately) composed of triangles

• A

A triangle triangle has 3 points, one for each corner has 3 points, one for each corner A A triangle triangle has 3 points, one for each corner has 3 points, one for each corner

• is the minimum number of

is the minimum number of vertices vertices (points) to define a (points) to define a plane plane

• these vertices need to be sent to

these vertices need to be sent to GPU GPU for processing for processing vert01 = new vert01 = new vertexPositionColor vertexPositionColor (new (new vector vector3 (0, 1, 0), 3 (0, 1, 0), Color.Blue); Color.Blue); represents the represents the location location of

• f one
• ne vertex in a triangle

vertex in a triangle many alternatives many alternatives

• represents the

represents the location location of

• f one
• ne vertex in a triangle

vertex in a triangle

• defines the

defines the color color of a vertex (to be interpolated over)

• f a vertex (to be interpolated over)
• A vertex structure may include other relevant info processed by

A vertex structure may include other relevant info processed by shader programs.. shader programs..

26 26 3.3.2017 3.3.2017 Juha Vihavainen / University of Helsinki Juha Vihavainen / University of Helsinki

SLIDE 14

Typical Typical vertex vertex structures (used by graphics API) structures (used by graphics API)

1.

. VertexPositionColor VertexPositionColor

• x,

, y, , z plus a plus a color color for each vertex for each vertex

• useful for basic shapes, basic colors

useful for basic shapes, basic colors 2.

• 2. VertexPositionTexture

VertexPositionTexture 2.

• 2. VertexPositionTexture

VertexPositionTexture

• x, y,

x, y, z plus relative plus relative u, , v coordinates into a texture (bitmap) coordinates into a texture (bitmap)

• can overlay a bitmap texture onto a shape

can overlay a bitmap texture onto a shape 3.

• 3. VertexPositionNormalTexture

VertexPositionNormalTexture

• x,

, y, , z plus plus u, , v, coordinates plus a , coordinates plus a normal normal vector vector

• normal permits calculating lighting effects

normal permits calculating lighting effects 4.

• 4. VertexPositionColorTexture

VertexPositionColorTexture 4.

• 4. VertexPositionColorTexture

VertexPositionColorTexture

• x,

, y, , z plus plus u, , v coordinates plus coordinates plus color color

• color information changes color of bitmap texture

color information changes color of bitmap texture

• can potentially reuse same texture in different contexts

can potentially reuse same texture in different contexts

27 27 3.3.2017 3.3.2017 Juha Vihavainen / University of Helsinki Juha Vihavainen / University of Helsinki

What are What are texture coordinates texture coordinates?

• vertexPositionTexture

vertexPositionTexture takes two parameters takes two parameters

• vector3

vector3 represents the position ( represents the position (x,y,z) of a vertex ) of a vertex

• vector2

vector2 represents relative texture coordinates represents relative texture coordinates u and and v

• vector2

vector2 represents relative texture coordinates represents relative texture coordinates u and and v telling which image point ( telling which image point (texel texel) hits the vertex ) hits the vertex

• u

u is a horizontal coordinate in the range 0.0 . . 1.0 is a horizontal coordinate in the range 0.0 . . 1.0

• v is a vertical coordinate in the range 0.0 . . 1.0

is a vertical coordinate in the range 0.0 . . 1.0

• top

top-left corner = { 0, 0 } left corner = { 0, 0 }

• a point exactly in the middle of a texture = { 0.5, 0.5 }

a point exactly in the middle of a texture = { 0.5, 0.5 }

• a point exactly in the middle of a texture = { 0.5, 0.5 }

a point exactly in the middle of a texture = { 0.5, 0.5 }

• bottom

bottom-right corner = { 1, 1 } right corner = { 1, 1 }

• Note that these relative (

Note that these relative (u, , v) coordinates are now independent of ) coordinates are now independent of the size of the image (good for potential scaling or other changes) the size of the image (good for potential scaling or other changes)

28 28 3.3.2017 3.3.2017 Juha Vihavainen / University of Helsinki Juha Vihavainen / University of Helsinki

SLIDE 15

What is a What is a virtual game camera virtual game camera

• In game graphics, a

In game graphics, a virtual camera virtual camera is implemented by software to is implemented by software to simulate the way a real camera (or an eye) would work in real simulate the way a real camera (or an eye) would work in real- world situations world situations In the game/engine, the virtual camera is made up from In the game/engine, the virtual camera is made up from

• In the game/engine, the virtual camera is made up from

In the game/engine, the virtual camera is made up from mathematical calculations that determine how objects will be mathematical calculations that determine how objects will be finally rendered based on the location and angle of the virtual finally rendered based on the location and angle of the virtual camera in the program camera in the program

• 3D is like a video camera

3D is like a video camera recording recording - what camera sees what camera sees (inside its (inside its viewing viewing frustum frustum) is ) is

3.3.2017 3.3.2017 Juha Vihavainen / University of Helsinki Juha Vihavainen / University of Helsinki 29 29

(inside its (inside its viewing viewing frustum frustum) is ) is shown on screen shown on screen

Implementing Implementing 3D graphics system (GPU) 3D graphics system (GPU)

• Vertex

Vertex and and index index buffers buffers are used to stream lots of 3D points are used to stream lots of 3D points (vertices) to the graphics device for rendering (vertices) to the graphics device for rendering

• indices

indices are just indirect pointers to are just indirect pointers to shared shared vertices (to save space vertices (to save space and bandwidth) and bandwidth)

• Separately must tell

Separately must tell how how consequent vertices are to be interpreted as consequent vertices are to be interpreted as primitive shapes (e.g., triangles forming a 3D object): whether primitive shapes (e.g., triangles forming a 3D object): whether adjacent adjacent triangles use triangles use shared shared vertices or not ( vertices or not (lists lists, , strips strips, , fans fans)

• Shaders

Shaders and related data structures (registers, memory) in GPU and related data structures (registers, memory) in GPU

• processing of vertices, primitives, and pixels

processing of vertices, primitives, and pixels

• nowadays, always need to use shaders (default/custom)

nowadays, always need to use shaders (default/custom)

• nowadays, always need to use shaders (default/custom)

nowadays, always need to use shaders (default/custom)

• apply matrix transformations (

apply matrix transformations (Transform Transform component) to vertices component) to vertices

• translation

translation (i.e., movement), (i.e., movement), rotation rotation, and , and scale scale

• apply colors and textures to triangles

apply colors and textures to triangles

• specify lighting, its location (plus other properties)

specify lighting, its location (plus other properties)

30 30 3.3.2017 3.3.2017 Juha Vihavainen / University of Helsinki Juha Vihavainen / University of Helsinki

SLIDE 16

Painter's algorithm (again) Painter's algorithm (again)

• Painter's algorithm doesn't work well for 3D scenes

Painter's algorithm doesn't work well for 3D scenes

• We would have to sort every triangle/pixel back to front

We would have to sort every triangle/pixel back to front worse, the camera is dynamic, so we'd have to re worse, the camera is dynamic, so we'd have to re-sort! sort!

• worse, the camera is dynamic, so we'd have to re

worse, the camera is dynamic, so we'd have to re-sort! sort!

• In 3D, there is a possibility

In 3D, there is a possibility that no one triangle is the that no one triangle is the furthest back furthest back The painter's algorithm has a lot of overdraw The painter's algorithm has a lot of overdraw

Overlapping Overlapping-

• triangles

triangles-failure failure case case

• The painter's algorithm has a lot of overdraw

The painter's algorithm has a lot of overdraw

• unnecessary redrawing the same pixel multiple times per

unnecessary redrawing the same pixel multiple times per frame frame => => inefficient inefficient

3.3.2017 3.3.2017 31 31 Juha Vihavainen / University of Helsinki Juha Vihavainen / University of Helsinki

Solution: Z Solution: Z-buffering buffering

• Which elements (parts) of a rendered

Which elements (parts) of a rendered scene are visible, and which are hidden scene are visible, and which are hidden

• Create a new screen

Create a new screen-sized buffer called sized buffer called the z the z-buffer ( buffer (depth buffering depth buffering), to ), to the z the z-buffer ( buffer (depth buffering depth buffering), to ), to determine determine what is what is closer to the camera closer to the camera

• It stores the

It stores the depth depth (z coordinate) of coordinate) of every pixel that is drawn (or actually, of every pixel that is drawn (or actually, of the the last last one)

• ne)
• As we draw pixels in the scene, only

As we draw pixels in the scene, only draw it if the depth stored in the z draw it if the depth stored in the z-buffer buffer draw it if the depth stored in the z draw it if the depth stored in the z-buffer buffer is greater than the depth of the current is greater than the depth of the current pixel (and so located further out in the pixel (and so located further out in the scene) => pixels can be drawn in any scene) => pixels can be drawn in any appropriate appropriate order, simplifying the

• rder, simplifying the
• verall processing of triangles
• verall processing of triangles

3.3.2017 3.3.2017 32 32 Juha Vihavainen / University of Helsinki Juha Vihavainen / University of Helsinki

A sample scene and its corresponding z-buffer (Wikipedia)

SLIDE 17

On the On the rendering pipeline rendering pipeline

• Note: "

Note: "rasterization rasterization" " - a scene with 3D models (shapes) converted a scene with 3D models (shapes) converted into a raster image (consisting of pixels or dots) into a raster image (consisting of pixels or dots)

• need only to represent the surface/outer boundary of an object

need only to represent the surface/outer boundary of an object the most popular technique for producing real the most popular technique for producing real-time 3D computer time 3D computer

• the most popular technique for producing real

the most popular technique for producing real-time 3D computer time 3D computer graphics; nearly all visual models used in game graphics graphics; nearly all visual models used in game graphics

• colors and textures can be applied

colors and textures can be applied

• A pipeline is a sequence of stages operating in a fixed order (and

A pipeline is a sequence of stages operating in a fixed order (and potentially potentially in parallel in parallel), e.g. ), e.g.

• vertex processing

vertex processing, , rasterization rasterization, , pixel pixel (fragment fragment) processing, .. processing, .. vertex processing vertex processing, , rasterization rasterization, , pixel pixel (fragment fragment) processing, .. processing, ..

• each stage receives its input from the prior stage and sends its

each stage receives its input from the prior stage and sends its

• utput to the subsequent stage
• utput to the subsequent stage
• A 3D application (game) sends to the GPU a sequence of vertices

A 3D application (game) sends to the GPU a sequence of vertices batched into geometric " batched into geometric "primitives primitives", typically triangles (in buffers) ", typically triangles (in buffers)

Juha Vihavainen / University of Helsinki Juha Vihavainen / University of Helsinki 33 33 3.3.2017 3.3.2017

[Gregory, 10.2 The Rendering Pipeline, p. 489] [Gregory, 10.2 The Rendering Pipeline, p. 489]

Here, the Here, the pipeline pipeline includes tools used to create scenes and game entities includes tools used to create scenes and game entities 1. 1. Tools stage Tools stage (offline): defining geometry and surface properties (offline): defining geometry and surface properties (materials), animations, audio resources, etc. (materials), animations, audio resources, etc. 2. 2. Asset conditioning Asset conditioning (offline): geometry and materials are processed (offline): geometry and materials are processed 2. 2. Asset conditioning Asset conditioning (offline): geometry and materials are processed (offline): geometry and materials are processed by asset conditioning pipeline (ACP) into an engine by asset conditioning pipeline (ACP) into an engine-ready format ready format 3. 3. Application stage Application stage (CPU): potentially visible mesh instances are (CPU): potentially visible mesh instances are identified and submitted to the graphics hardware along with their identified and submitted to the graphics hardware along with their materials for rendering materials for rendering 4. 4. Geometry processing Geometry processing (GPU): vertices are transformed and lit and (GPU): vertices are transformed and lit and projected into (homogeneous) clip space; triangles are processed by projected into (homogeneous) clip space; triangles are processed by projected into (homogeneous) clip space; triangles are processed by projected into (homogeneous) clip space; triangles are processed by the optional geometry shader and then clipped to the frustum (lines the optional geometry shader and then clipped to the frustum (lines and surfaces outside the view volume are removed) and surfaces outside the view volume are removed) 5. 5. Rasterization stage Rasterization stage (GPU): triangles are converted into fragments (GPU): triangles are converted into fragments that are shaded, passed through various tests (z test, alpha test, that are shaded, passed through various tests (z test, alpha test, stencil test, etc.) and finally blended into the frame buffer stencil test, etc.) and finally blended into the frame buffer

3.3.2017 3.3.2017 Juha Vihavainen / University of Helsinki Juha Vihavainen / University of Helsinki 34 34

SLIDE 18

The rendering pipeline (simplified overview) The rendering pipeline (simplified overview)

(1) (offline) (2) (offlline) ACP = Asset (3) (CPU) (4) (GPU) ACP = Asset Conditioning Pipeline fragment = potential pixel (OpenGL)

3.3.2017 3.3.2017 Juha Vihavainen / University of Helsinki Juha Vihavainen / University of Helsinki 35 35

(5) (GPU)

<= <= For explanations, see the For explanations, see the previous previous slide slide

[Greg, Fig. 10.41, p. 491]

Rendering pipeline Rendering pipeline transforms data transforms data

• The

The tools and asset conditioning tools and asset conditioning stages deal with meshes and stages deal with meshes and materials materials

• The

The application application stage deals in terms of mesh instances and stage deals in terms of mesh instances and submeshes, each of which is associated with a single material submeshes, each of which is associated with a single material

• During the

During the geometry geometry stage, each submesh is broken down into stage, each submesh is broken down into individual vertices, which are processed largely in parallel; individual vertices, which are processed largely in parallel; at its conclusion, the triangles are reconstructed from the fully at its conclusion, the triangles are reconstructed from the fully transformed and shaded vertices transformed and shaded vertices

• In the

In the rasterization rasterization stage, each triangle is broken into fragments, stage, each triangle is broken into fragments, and these fragments are either discarded, or they are eventually and these fragments are either discarded, or they are eventually written into the frame buffer as colors written into the frame buffer as colors

3.3.2017 3.3.2017 Juha Vihavainen / University of Helsinki Juha Vihavainen / University of Helsinki 36 36

SLIDE 19

Z-buffering in buffering in GPU GPU hardware hardware

• When using z

hen using z-buffering, we buffering, we don't don't draw triangles draw triangles back back-to to-front front, or , or many draws will become waisted (when overridden) many draws will become waisted (when overridden)

• Transparent

ransparent objects, such as water, need some special arrangements

• bjects, such as water, need some special arrangements

first irst draw all non draw all non-transparent objects, and transparent objects, and

• first

irst draw all non draw all non-transparent objects, and transparent objects, and

• then

then transparent ones ( transparent ones (partly partly disabling z disabling z-buffering buffering - to blend) to blend)

• therwise transparent objects would totally hide objects
• therwise transparent objects would totally hide objects

behind them (background opaque ones would not be drawn behind them (background opaque ones would not be drawn)

• Occlusion

Occlusion culling culling tehniques tehniques eliminate, at application stage, entire eliminate, at application stage, entire scene objects that cannot be visible on a current scene objects that cannot be visible on a current frame frame

3.3.2017 3.3.2017 37 37 Juha Vihavainen / University of Helsinki Juha Vihavainen / University of Helsinki

scene objects that cannot be visible on a current scene objects that cannot be visible on a current frame frame

• e.g., a

e.g., a tree totally occluded by a tree totally occluded by a building building

• uses

uses spatial partitioning spatial partitioning for for efficiency efficiency

• An algorithm in pseudo

An algorithm in pseudo-language, see next slide language, see next slide // // clear clear the frame and depth buffers the frame and depth buffers fillColorBuffer fillColorBuffer (backgroundColor); (backgroundColor); fillDepthBuffer (infinity); fillDepthBuffer (infinity); // // outer loop iterates over all the primitives

• uter loop iterates over all the primitives (usually triangles

usually triangles) for each geometric primitive do for each geometric primitive do // // first rasterize the primitive . . first rasterize the primitive . .

(3D Math Primer for Graphics and Game Development, 2011), p. 346

// // first rasterize the primitive . . first rasterize the primitive . . for each pixel x , y in the projection of the primitive do for each pixel x , y in the projection of the primitive do // // test the depth buffer , to see if a closer pixel has been written test the depth buffer , to see if a closer pixel has been written float primDepth = getDepthOfPrimitiveAtPixel (x , y) ; float primDepth = getDepthOfPrimitiveAtPixel (x , y) ; // // pixel of this primitive is obscured , discard it pixel of this primitive is obscured , discard it if primDepth > readDepthBuffer (x, y) continue; if primDepth > readDepthBuffer (x, y) continue; // // determine primitive color at this pixel determine primitive color at this pixel Color c = getColorOfPrimitiveAtPixel (x, y) Color c = getColorOfPrimitiveAtPixel (x, y) // // update the color and depth buffers update the color and depth buffers writeColorBuffer writeColorBuffer (x, y, c) (x, y, c) writeDepthBuffer (x, y, primDepth) writeDepthBuffer (x, y, primDepth)