Graphics 2014 Linear Algebra II Linear Maps & Matrices Linear - - PowerPoint PPT Presentation

Graphics 2014

Linear Algebra II

Linear Maps & Matrices

core topics important

CORE

Linear Maps & Matrices

Linear Combinations

Linear Combinations as Mappings

• Fix vectors 𝐲1, … , 𝐲𝑜 ∈ ℝ𝑛.
• Factors λ1, … , λ𝑜 → 𝐳

x1 Linear Combinations x2 2x2 + x1

𝐳 = λ𝑗𝐲𝑗

𝑜 𝑗=1 Algebra

Linear Mappings

Linear Map

• Fix vectors

𝐲1, … , 𝐲𝑜 ∈ ℝ𝑛

• Input coordinates

λ1, … , λ𝑜

• Output vector

𝐳 ∈ ℝ𝑛

Linear Combinatio tion

λ1 ⋯ λ𝑜

fixed: x1,… , xn

𝐳 = λ𝑗𝐲𝑗

𝑜 𝑗=1

Map λ1, … , λ𝑜 → 𝐳 is called a linear map

𝐳

Linear Mappings

Linear Map

• Fix vectors

𝐲1, … , 𝐲𝑜 ∈ ℝ𝑛

• Input coordinates

λ1, … , λ𝑜

• Output vector

𝐳 ∈ ℝ𝑛

Linear Combinatio tion

λ1 ⋯ λ𝑜

fixed: x1,… , xn

𝐳 = λ𝑗𝐲𝑗

𝑜 𝑗=1

Map λ1, … , λ𝑜 → 𝐳 is called a linear map

Input Vectors

𝐳

Linear Mappings

Linear Combinatio tion 𝛍 = 𝜇1 ⋮ λ𝑜

fixed: x1, … , xn ∈ ℝ𝑛

𝐳 = 𝑧1 ⋮ 𝑧𝑛 𝜇1 𝜇2 y = 𝜇1x1 + 𝜇2x2

How the machine works

𝐳 = λ𝑗𝐲𝑗

𝑜 𝑗=1

= | | 𝑦1 ⋯ 𝑦𝑜 | | ⋅ 𝜇1 ⋮ λ𝑜 = λ𝑗 𝑦1,𝑗 ⋮ 𝑦𝑛,𝑗

𝑜 𝑗=1

= 𝑦1,1 ⋯ 𝑦1,𝑜 ⋮ ⋮ 𝑦𝑛,1 ⋯ 𝑦𝑛,𝑜 ⋅ 𝜇1 ⋮ λ𝑜

Matrix Representation

Matrix Representation

𝐳 = λ𝑗𝐲𝑗

𝑜 𝑗=1

= | | 𝐲1 ⋯ 𝐲𝑜 | | ⋅ 𝜇1 ⋮ λ𝑜 = λ𝑗 𝑦1,𝑗 ⋮ 𝑦𝑛,𝑗

𝑜 𝑗=1

= 𝑦1,1 ⋯ 𝑦1,𝑜 ⋮ ⋮ 𝑦𝑛,1 ⋯ 𝑦𝑛,𝑜 ⋅ 𝜇1 ⋮ λ𝑜

Short

𝐳 = 𝐘 ⋅ 𝛍

Matrix

𝐘 = 𝑦1,1 ⋯ 𝑦1,𝑜 ⋮ ⋮ 𝑦𝑛,1 ⋯ 𝑦𝑛,𝑜 Vectors 𝛍 = 𝜇1 ⋮ λ𝑜 , 𝐳 = 𝑧1 ⋮ 𝑧𝑛

Convention

Taken from Textbook [Shirley et al.]

• Matrix elements

𝑦𝑠𝑝𝑥,𝑑𝑝𝑚𝑣𝑛𝑜

• Row first, then column
• “y”-coordinate of the array first

(unintuitive, but common convention)

𝑦1,1 ⋯ 𝑦1,𝑜 ⋮ ⋮ 𝑦𝑛,1 ⋯ 𝑦𝑛,𝑜

𝑛 𝑠𝑝𝑥𝑡 𝑜 𝑑𝑝𝑚𝑣𝑛𝑜𝑡

Matrix Representation

Matrix-vector product Construction

• Maps from ℝ𝑜 → ℝ𝑛
• 𝛍 ∈ ℝ𝑜
• 𝐲𝑗 ∈ ℝ𝑛 ⇒ 𝐳 ∈ ℝ𝑛
• Columns of 𝐘 = images of the basis vectors of ℝ𝑜

𝐳 𝛍 = | | 𝐲1 ⋯ 𝐲𝑜 | | ⋅ 𝜇1 ⋮ λ𝑜

Example

Example: rotation matrix

        1         1

𝐍𝑠𝑝𝑢 = cos 𝛽 − sin 𝛽 sin 𝛽 cos 𝛽

cos 𝛽 sin 𝛽 − sin 𝛽 cos 𝛽

General Matrix Product (Notation)

Algebraic rule:

• Vector-matrix product:

𝐍 ⋅ 𝐲 = 𝐳

=

𝐳 𝐍 𝐲

General Matrix Product (Notation)

Algebraic rule:

• Vector-matrix product:

°

𝐍 ⋅ 𝐲 = 𝐳

°

𝐳 𝐍 𝐲

× × × × ∑ ∑ × × × ×

General Matrix Product (Notation)

Algebraic rule:

• Vector-matrix product:

°

𝐍 ⋅ 𝐲 = 𝐳 𝐳 𝐍 𝐲

× × × × ∑ ∑ × × × ×

° °

General Matrix Product (Notation)

Algebraic rule:

• Vector-matrix product:

°

𝐍 ⋅ 𝐲 = 𝐳 𝐳 𝐍 𝐲

× × × × ∑ ∑ × × × ×

° ° °

General Matrix Product (Notation)

Algebraic rule:

• Vector-matrix product:

°

𝐍 ⋅ 𝐲 = 𝐳 𝐳 𝐍 𝐲

× × × × ∑ × × × × ∑

° ° ° °

Matrix Representation

Matrix-Vector Multiplication

𝑦1,1 ⋯ 𝑦1,1 ⋮ ⋮ 𝑦𝑛,1 ⋯ 𝑦𝑛,𝑜 ⋅ 𝜇1 ⋮ λ𝑜 ≔ 𝜇𝑗 𝑦1,𝑗 ⋮ 𝑦𝑛,𝑗

𝑜 𝑗=1

= 𝜇1 ⋅ 𝑦1,1 + ⋯ + 𝜇𝑜 ⋅ 𝑦1,𝑜 ⋮ 𝜇1 ⋅ 𝑦𝑛,1 + ⋯ + 𝜇𝑜 ⋅ 𝑦𝑛,𝑜 °

basic topics study completely

BASIC

Standard Transformations

Identity Transform

Example: identity matrix

        1         1

𝐍𝑗𝑒𝑓𝑜𝑢𝑗𝑢𝑧 = 𝐉 = 1 1

        1         1

General case 𝐉: ℝ𝑜 → ℝ𝑜, 𝐉 = 1 ⋯ 1 ⋮ ⋱ ⋮ ⋯ 1

1 1

Scaling (Center = Origin)

𝜇 𝜇

General case 𝐓𝜇: ℝ𝑜 → ℝ𝑜, 𝐓𝜇 = 𝜇 ⋯ 𝜇 ⋮ ⋱ ⋮ ⋯ 𝜇

1 1

Non-Uniform Scaling

𝜇1 𝜇2

General case 𝐓𝛍: ℝ𝑜 → ℝ𝑜, 𝐓𝛍 = 𝜇1 ⋯ 𝜇2 ⋮ ⋱ ⋮ ⋯ 𝜇3

Rotation (2D)

        1         1

𝐍𝑠𝑝𝑢 = cos 𝛽 − sin 𝛽 sin 𝛽 cos 𝛽

cos 𝛽 sin 𝛽 − sin 𝛽 cos 𝛽

Rotation (3D)

𝐰 = 𝑦 𝑧 𝑨

𝐳 𝐲 𝐩𝐬𝐣𝐡𝐣𝐨 𝐴

𝑊 = ℝ3

𝐳 𝐩𝐬𝐣𝐡𝐣𝐨 𝐴 𝐲 𝐩𝐬𝐣𝐡𝐣𝐨 𝐴 𝐲 𝐳 𝐩𝐬𝐣𝐡𝐣𝐨 𝐴 𝐲 𝐳

Rotation (3D)

𝐳 𝐩𝐬𝐣𝐡𝐣𝐨 𝐴 𝐲 𝐩𝐬𝐣𝐡𝐣𝐨 𝐴 𝐲 𝐳 𝐩𝐬𝐣𝐡𝐣𝐨 𝐴 𝐲 𝐳 𝐒𝑦 = 1 cos 𝛽 − sin 𝛽 sin 𝛽 cos 𝛽 𝐒𝑨 = cos 𝛽 − sin 𝛽 sin 𝛽 cos 𝛽 1 𝐒𝑧 = cos 𝛽 − sin 𝛽 1 sin 𝛽 cos 𝛽

Reflection

General case 𝐓𝜇: ℝ𝑜 → ℝ𝑜, 𝐓𝜇 = 1 ⋯ −1 ⋮ ⋱ ⋮ ⋯ 1

1 1 −1 1 𝐍𝑠𝑓𝑔𝑚 = −1 1 Reflection Axis

Shearing

1 1

𝐍𝑡ℎ𝑓𝑏𝑠 = 1 𝜇 1

0.5 1

General Case

You can combine all of these Example: General axis of rotation

• First rotate rotation axis to x-axis
• Rotate around x
• Rotate back

Question

• How to combine multiple matrix multiplications?

basic topics study completely

BASIC

Combining Transformations

Matrix Products

Matrix Multiplication

Execute multiple linear maps,

• ne after another
• Written as product
• 𝐂 ⋅ 𝐁 ⋅ 𝐲:
• Apply 𝐁 to 𝐲 first
• Then 𝐂
• 𝐂 ⋅ 𝐁 is again a matrix

How does it work?

Consider 𝐂 ⋅ 𝐁 :

• Rotate first (𝐁)
• Then scale (𝐂)

        1         1

cos 𝛽 sin 𝛽 − sin 𝛽 cos 𝛽

1 1 2 2

𝐁 𝐂

How does it work?

How to compute 𝐂 ⋅ 𝐁 ?

• Transform basis vectors
• Transform again

cos 𝛽 sin 𝛽 − sin 𝛽 cos 𝛽 𝐁

𝐂 ⋅ 𝐁

2cos 𝛽 2sin 𝛽 − 2sin 𝛽 2cos 𝛽

Matrix product: 𝐁

Matrix Multiplication

𝐂 𝐛1 𝐛4

column 4

𝐛3

column 3

𝐛2

column 2 column 1

𝐁

Matrix Multiplication

Matrix product: 𝐂

°

Matrix Multiplication

General matrix products:

• 𝐂 ⋅ 𝐁: possible if

#Row(𝐁) = #Columns(𝐂)

°

𝐁 𝐂

𝐁 = 𝑏1,1 ⋯ 𝑏1,𝑜 ⋮ ⋮ 𝑏𝑛,1 ⋯ 𝑏𝑛,𝑜 𝐂 = 𝑐1,1 ⋯ 𝑐1,𝑛 ⋮ ⋮ 𝑐𝑙,1 ⋯ 𝑐𝑙,𝑛

𝑙 𝑛 𝑜 𝑜 𝑛 𝑙

𝐒 = 𝑠

1,1

⋯ 𝑠

1,𝑜

⋮ ⋮ 𝑠

𝑙,1

⋯ 𝑠

𝑙,𝑜

𝐒 = 𝐂 ⋅ 𝐁 𝑠

𝑗,𝑘 = 𝑏𝑟,𝑘 ⋅ 𝑐𝑗,𝑟 𝑛 𝑟=1

Rules for Matrix Multiplication

Matrix-Multiplication

• Associative

𝐁 ⋅ 𝐂 ⋅ 𝐃 = 𝐁 ⋅ 𝐂 ⋅ 𝐃

• Includes vector-multiplication

𝐁 ⋅ 𝐂 ⋅ 𝐰 = 𝐁 ⋅ 𝐂 ⋅ 𝐰

• In general, not commutative:

It might be that 𝐁 ⋅ 𝐂 ≠ 𝐂 ⋅ 𝐁

• Linear

𝐁 ⋅ 𝐰 + 𝐱 = 𝐁 ⋅ 𝐰 + 𝐁 ⋅ 𝐱 𝐁 ⋅ 𝜇 ⋅ 𝐰 = 𝜇 ⋅ 𝐁 ⋅ 𝐰

(Remark: linearity is used to define linear maps axiomatically)

𝜇 ∈ ℝ 𝐁, 𝐂, 𝐃 - matrices 𝐰, 𝐱 - vectors

Settings

core topics important

CORE

Reversing Transformations

Matrix Inversion

Inverse Matrix

Can we find the inverse matrix?

• “Undo effect”
• Formally

𝐍−1 ⋅ 𝐍 = 𝐉

Inverse Matrix

Examples

• Rotation matrix

𝐍𝑠𝑝𝑢 = cos 𝛽 − sin 𝛽 sin 𝛽 cos 𝛽

• Inverse?

cos 𝛽 sin 𝛽 − sin 𝛽 cos 𝛽 𝐍𝑠𝑝𝑢 cos(−𝛽) sin(−𝛽) 𝐍𝑠𝑝𝑢

−1

𝛽 −𝛽

𝐍𝑠𝑝𝑢

−1 = cos(−𝛽)

− sin(−𝛽) sin(−𝛽) cos(−𝛽)

− sin(−𝛽) cos(−𝛽)

Inverse Matrix

Examples

• Null matrix

𝟏 = 0

• Inverse?

𝟏

Inverse Matrix

Examples

• Projection matrix (remove x-component)

𝐍𝑞𝑠𝑘 = 0

1

• Inverse?

1 𝐍𝑞𝑠𝑘

Inverse Matrix

Examples

• Projection matrix (remove x-component)

𝐍𝑔𝑏𝑜𝑑𝑧 = 2

1 4 2

• Inverse?

2 4 1 2 𝐍𝑔𝑏𝑜𝑑𝑧

Invertible Matrices

Invertible matrices

• Are always square (#rows = #columns)
• In addition
• Columns are linearly independent

Equivalent characterizations:

• Square and rows are linearly independent
• Columns form basis of vector space
• Rows form basis of vector space

Invertible Matrices

Rank

• Number of linearly independent columns
• Dimension of span{𝐝𝐩𝐦𝐯𝐧𝐨_𝐰𝐟𝐝𝐮𝐩𝐬𝐭}

Theorem

• Rank = number of linearly independent rows

Full rank

• rank(𝐍) = dim

(𝑊)

• Then: 𝐍 is invertible

Linear Systems of Equations

First consider simpler case

• Say, we know that

𝐍 ⋅ 𝐲 = 𝐳

• Square matrix 𝐍 ∈ ℝ𝑒×𝑒
• Vectors 𝐲, 𝐳 ∈ ℝ𝑒×𝑒

Knowns & Unknowns

• We are given 𝐍, 𝐳
• We should compute 𝐲
• Linear system of equations

Linear Systems of Equations

Linear System of Equations

𝐍 ⋅ 𝐲 = 𝐳 ⇔

𝑛1,1 ⋯ 𝑛1,𝑒 ⋮ ⋮ 𝑛𝑒,1 ⋯ 𝑛𝑒,𝑒 ⋅ 𝑦1 ⋮ 𝑦𝑒 = 𝑧1 ⋮ 𝑧𝑒

⇔

𝑛1,1𝑦1 + ⋯ + 𝑛1,𝑒 𝑦𝑒 = 𝑧1 𝑛2,1𝑦1 + ⋯ + 𝑛2,𝑒 𝑦𝑒 = 𝑧2 ⋮ 𝑛𝑒,1𝑦1 + ⋯ + 𝑛𝑒,𝑒 𝑦𝑒 = 𝑧𝑒

and and

Gaussian Elimination

Linear System

∧ 𝑛1,1𝑦1 + ⋯ + 𝑛1,𝑒 𝑦𝑒 = 𝑧1 ∧ 𝑛2,1𝑦1 + ⋯ + 𝑛2,𝑒 𝑦𝑒 = 𝑧2 ⋮ ∧ 𝑛𝑒,1𝑦1 + ⋯ + 𝑛𝑒,𝑒 𝑦𝑒 = 𝑧𝑒

Row Operations

• Swap rows 𝑠

𝑗, 𝑠 𝑘

• Scale row 𝑠

𝑗 by factor 𝜇 ≠ 0

• Add multiple of row 𝑠

𝑗 to row 𝑠 𝑘, 𝑗 ≠ 𝑘

(i.e., 𝑠

𝑗 += 𝜇𝑠 𝑘)

Convert to Upper Triangle Matrix

=

𝐳 𝐍 𝐲

= = =

0 0

(use row-operations)

Convert to Diagonal Matrix

=

𝐳 𝐍 𝐲

= = =

0 0

=

0 0

=

0 0

1 1 1 1 𝑧1

′/m1,1 ′

𝑧2

′/m2,2 ′

𝑧3

′/m3,3 ′

𝑧4

′/m4,4 ′

𝑦1 𝑦2 𝑦3 𝑦4

(use row-operations)

𝑛1,1

′

𝑛2,2

′

𝑛3,3

′

𝑛4,4

′

𝑧1

′

𝑧2

′

𝑧3

′

𝑧4

′

𝑦1 𝑦2 𝑦3 𝑦4

Gauss-Algorithm

Gauss-Algorithm

• Substract rows to cancel front-coefficient
• Create upper triangle matrix first
• Then create diagonal matrix
• If current row starts with 0
• Swap with another row
• If all rows start with 0: matrix not invertible
• Diagonal form: Solution can be read-off
• Data structure
• Modify matrix M, “right-hand-side” y.
• x remains unknown (no change)

Matrix Inverse

Solve for

𝐍 ⋅ 𝐲1 = 1 ⋮ , 𝐍 ⋅ 𝐲2 = 1 ⋮ , … , 𝐍 ⋅ 𝐲𝑒 = ⋮ 1

• The resulting 𝐲1, 𝐲2, … , 𝐲𝑒 are the columns of 𝐍−1:

𝐍−1 = | | 𝐲1 ⋯ 𝐲𝑒 | |

Matrix Inverse

Algorithm

• Simultaneous Gaussian elimination
• Start as follows:
• Handle all right-hand sides simultaneously
• After Gauss-algorithm, the right-hand matrix

is the inverse

=

0 0

1 1 1 1

𝐍 𝐲 𝐉

Alternative: Kramer’s Rule

Small Matrices

• Direct formula based on determinants
• “Kramer’s rule”
• (more later)
• Naive implementation has run-time 𝒫(𝑒!)

– Gauss: 𝒫(𝑒3)

• Not advised for 𝑒 > 3

basic topics study completely

BASIC

More Vector Operations:

Scalar Products

Additional Vector Operations

Length of Vectors

𝐰2 = 𝟓. 𝟑cm

“length” or “norm” ‖𝐰‖ yields real number ≥ 0

𝐰1 = 𝟑. 𝟒cm 𝐰1 𝐰2

Additional Vector Operations

Angle between Vectors

𝛽 = ∠ 𝐰1, 𝐰2 = 𝟒𝟒°

angle ∠ 𝐰1, 𝐰2 yields real number 0, … , 2𝜌 = [0, … , 360°)

𝐰1 𝐰2

𝛽

Additional Vector Operations

Angle between Vectors

right angles 𝐰1 𝐰2

90°

Additional Vector Operations

Projection Projection: determine length of 𝐰 along direction of 𝐱

𝐰 𝐱

90°

𝐰 prj on 𝐱

Additional Vector Operations

Scalar Product*)

𝐰 ⋅ 𝐱 = 𝐰 ⋅ 𝐱 ⋅ cos ∠(𝐰, 𝐱)

*) also known as inner product

• r dot-product

also: 𝐰, 𝐱

90°

𝐰 𝐱

Signature

• ut
• perato

tor ∗

Scalar Product

(dot product, inner-product)

in

42.0

in

Additional Vector Operations

Scalar Product*)

90°

𝐰 ⋅ 𝐱 = 𝐰 ⋅ 𝐱 ⋅ cos ∠(𝐰, 𝐱)

*) also known as inner product

• r dot-product

also: 𝐰, 𝐱 𝐰 𝐱

Additional Vector Operations

Scalar Product*)

90°

𝐰 ⋅ 𝐱 = 𝐰 ⋅ 𝐱 ⋅ cos ∠(𝐰, 𝐱)

Comprises: length, projection, angles

*) also known as inner product

• r dot-product

𝐰 𝐱

Additional Vector Operations

𝐰 ⋅ 𝐱 = 𝐰 ⋅ 𝐱 ⋅ cos ∠(𝐰, 𝐱)

Comprises: length, projection, angles

Length: 𝐰 = 𝐰 ⋅ 𝐰 Angle: ∠ 𝐰, 𝐱 = arccos 𝐰 ⋅ 𝐱 Projection: „𝐰 prj on 𝐱” = 𝐰⋅𝐱

𝐱

basic topics study completely

BASIC

Algebraic Representation

(Implementation)

Scalar Product

Scalar Product*)

𝐰 𝐱

90°

𝐰 ⋅ 𝐱 = 𝑤1 𝑤2 ⋅ 𝑥1 𝑥2 ≔ 𝑤1 ⋅ 𝑥1 + 𝑤2 ⋅ 𝑥2 Theorem: 𝐰 ⋅ 𝐱 = 𝐰 ⋅ 𝐱 ⋅ cos ∠(𝐰, 𝐱)

Scalar Product

Scalar product 𝐰 ⋅ 𝐱 = 𝑤1 𝑤2 ⋅ 𝑥1 𝑥2 𝐰 𝐰 = 3 2 𝐱 = 1 2 𝐱

Scalar Product

2D Scalar product 𝐰 ⋅ 𝐱 = 𝑤1 𝑤2 ⋅ 𝑥1 𝑥2 ≔ 𝑤1 ⋅ 𝑥1 + 𝑤2 ⋅ 𝑥2 d-dim scalar product 𝐰 ⋅ 𝐱 = 𝑤1 ⋮ 𝑤𝑒 ⋅ 𝑥1 ⋮ 𝑥𝑒 ≔ 𝑤1 ⋅ 𝑥1 + ⋯ + 𝑤𝑒 ⋅ 𝑥𝑒

Algebraic Properties

Properties

• Symmetry (commutativity)

𝐯, 𝐰 = 𝐰, 𝐯

• Bilinearity

𝜇𝐰, 𝐱 = 𝜇 𝐰, 𝐱 = 𝐰, 𝜇𝐱 𝐯 + 𝐰, 𝐱 = 𝐯, 𝐱 + 𝐰, 𝐱

(symmetry: same for second argument)

• Positive definite

𝐯, 𝐯 ≥ 0, 𝐯, 𝐯 = 𝟏 ⇒ 𝐯 = 𝟏

These three: axiomatic definition

𝜇 ∈ ℝ 𝐯, 𝐰, 𝐱 ∈ ℝ𝑒

Settings

Attention!

Do not mix

• Scalar-vector product
• Inner (scalar) product

In general 𝐲, 𝐳 ⋅ 𝐴 ≠ 𝐲 ⋅ 𝐳, 𝐴 Beware of notation: 𝐲 ⋅ 𝐳 ⋅ 𝐴 ≠ 𝐲 ⋅ 𝐳 ⋅ 𝐴

(no violation of associativity: different operations; details later)

core topics important

CORE

Applications of the Scalar Product

Applications

Obvious applications

• Measuring length
• Measuring angles
• Projections

More complex applications

• Creating orthogonal (90°) pairs of vectors
• Creating orthogonal bases

Projection

Scalar Product*)

90°

𝐰 ⋅ 𝐱 = 𝐰 ⋅ 𝐱 ⋅ cos ∠(𝐰, 𝐱) 𝐰 𝐱

Projection

Scalar Product*)

90°

𝐰 𝐱 𝐱n 𝐱n = 𝐱 𝐱 = 𝐱 𝐱, 𝐱

projection prj.-vector

Projection: 𝐰 ⋅

𝐱 𝐱⋅𝐱

Prj.-Vector: 𝐰,

𝐱 𝐱,𝐱

⋅

𝐱 𝐱,𝐱

= 𝐰, 𝐱 ⋅

𝐱 𝐱,𝐱

Orthogonalization

Scalar Product*)

90°

𝐰 𝐱 𝐱n = 𝐱 𝐱 = 𝐱 𝐱, 𝐱

projection prj.-vector

Orthogonalize 𝐰 wrt. 𝐱:

𝐰′ = 𝐰 − 𝐰, 𝐱 ⋅ 𝐱 𝐱, 𝐱 𝐰′

Orthogonalization

Scalar Product*)

90°

𝐰 𝐱 𝐰′

Orthogonalize 𝐰 wrt. 𝐱:

𝐰′ = 𝐰 − 𝐰, 𝐱 ⋅ 𝐱 𝐱, 𝐱

Gram-Schmidt Orthogonalization

Orthogonal basis

• All vectors in 90° angle to each other

𝐜𝑗, 𝐜𝑘 = 0 for 𝑗 ≠ 𝑘

Create orthogonal bases

• Start with arbitrary one
• Orthogonalize 𝐜2 by 𝐜1
• Orthogonalize 𝐜3 by 𝐜1, then by 𝐜2
• Orthogonalize 𝐜4 by 𝐜1, then by 𝐜2, then by 𝐜3
• ...

Orthonormal Basis

Orthonormal bases

• Orthogonal and all vectors have unit length

Computation

• Orthogonalize first
• Then scale each vector 𝐜𝑗 by 1/ 𝐜𝑗 .

Matrices

Orthogonal Matrices

• A matrix with orthonormal columns

is called orthogonal matrix

• Yes, this terminology is not quite logical...

Orthogonal Matrices are always

• Rotation matrices
• Or reflection matrices
• Or products of the two

core topics important

CORE

Further Operations

Cross Product

Cross-Product: Exists Only For 3D Vectors!

• 𝐲, 𝐳 ∈ ℝ3
• 𝐲 × 𝐳 =

𝑦1 𝑦2 𝑦3 × 𝑧1 𝑧2 𝑧3 ≔ 𝑦2𝑧3 − 𝑦3𝑧2 𝑦3𝑧1 − 𝑦1𝑧3 𝑦1𝑧2 − 𝑦2𝑧1

Geometrically: Theorem

• 𝐲 × 𝐳 orthogonal to 𝐲, 𝐳
• Right-handed system 𝐲, 𝐳, 𝐲 × 𝐳
• 𝐲 × 𝐳

= 𝐲 ⋅ 𝐳 ⋅ sin∠ 𝐲, 𝐳

y x x  y ‖x  y‖

Cross-Product Properties

Bilinearity

• Distributive:

𝐯 × 𝐰 + 𝐱 = 𝐯 × 𝐰 + 𝐯 × 𝐱

• Scalar-Mult.:

𝜇𝐯 × 𝐰 = 𝐯 × 𝜇𝐰 = 𝜇 𝐯 × 𝐰

But beware of

• Anti-Commutative: 𝐯 × 𝐰 = −𝐰 × 𝐯
• Not associative;

we can have 𝐯 × 𝐰 × 𝐱 ≠ 𝐯 × 𝐰 × 𝐱

Determinants

Determinants

• Square matrix M
• det(M) = |M| = volume of parallelepiped
• f column vectors

v2 v1 det 𝐍

𝐍 = | | | 𝐰1 𝐰2 𝐰3 | | |

v3

Determinants

Sign:

• Positive for right handed coordinates
• Negative for left-handed coordinates

v2 v1 det 𝐍 > 0

𝐍 = | | | 𝐰1 𝐰2 𝐰3 | | |

v3 v1 v2 det 𝐍′

< 0 𝐍′ = | | | 𝐰2 𝐰1 𝐰3 | | |

v3

negative determinant → map contains reflection

Properties

A few properties:

• det(A) det(B) = det(A⋅B)
• det(𝜇A) = 𝜇d det(A) (d  d matrix A)
• det(A-1) = det(A)-1
• det(AT) = det(A)
• det 𝐁 ≠ 0 ⇔ 𝐁 invertible
• Efficient computation using Gaussian elimination

sign flips! → reflections cancel each

• ther (parity)

Computing Determinants

𝑏 𝑐 𝑑 𝑒 𝑓 𝑔 𝑕 ℎ 𝑗 = +𝑏 𝑓 𝑔 ℎ 𝑗 − 𝑐 𝑒 𝑔 𝑕 𝑗 + 𝑑 𝑒 𝑓 𝑕 ℎ

Recursive Formula

• Sum over first row
• Multiply element there

with subdeterminant

• Subdeterminant :

Leave out row and column

• f selected element
• Recursion ends with |a|= a
• Alternate signs +/−/+/−/…

𝑏 𝑐 𝑑 𝑒 𝑓 𝑔 𝑕 ℎ 𝑗 𝑒 𝑔 𝑕 𝑗

subdeterminants

+𝑏 −𝑐 +𝑑 𝑒 𝑓 𝑔 𝑕 ℎ 𝑗

signs

Beware of 𝒫 𝑒𝑗𝑛! complexity

+ − + |a|= a

Computing Determinants

Result in 3D Case

det 𝑏 𝑐 𝑑 𝑒 𝑓 𝑔 𝑕 ℎ 𝑗 = 𝑏𝑓𝑗 + 𝑐𝑔𝑕 + 𝑑𝑒ℎ − 𝑑𝑓𝑕 − 𝑐𝑒𝑗 − 𝑏𝑔ℎ

Solving Linear Systems

Consider 𝐁 ⋅ 𝐲 = 𝐜

• Invertible matrix 𝐁 ∈ ℝ𝑒×𝑒
• Known vector 𝐜 ∈ ℝ𝑒
• Unknown vector 𝐲 ∈ ℝ𝑒

Solution with Determinants (Cramar’s rule): 𝑦𝑗 = det 𝐁𝑗 det 𝐁

𝐁𝑗 = | | | 𝐰1 ⋯ 𝐜 ⋯ 𝐰3 | | |

column 𝑗

advanced topics main ideas

ADV

Addendum

Matrix Algebra

Matrix Algebra

Define three operations

• Matrix addition

𝑏1,1 ⋯ 𝑏1,𝑜 ⋮ ⋱ ⋮ 𝑏𝑛,1 ⋯ 𝑏𝑛,𝑜 + 𝑐1,1 ⋯ 𝑐1,𝑜 ⋮ ⋱ ⋮ 𝑐𝑛,1 ⋯ 𝑐𝑛,𝑜 = 𝑏1,1 + 𝑐1,1 ⋯ 𝑏1,𝑜 + 𝑐1,𝑜 ⋮ ⋱ ⋮ 𝑏𝑛,1 + 𝑐𝑛,1 ⋯ 𝑏𝑛,𝑜 + 𝑐𝑛,𝑜

• Scalar matrix multiplication

𝜇 ⋅ 𝑏1,1 ⋯ 𝑏1,𝑜 ⋮ ⋱ ⋮ 𝑏𝑛,1 ⋯ 𝑏𝑛,𝑜 = 𝜇 ⋅ 𝑏1,1 ⋯ 𝜇 ⋅ 𝑏1,𝑜 ⋮ ⋱ ⋮ 𝜇 ⋅ 𝑏𝑛,1 ⋯ 𝜇 ⋅ 𝑏𝑛,𝑜

• Matrix-matrix multiplication

𝑏1,1 ⋯ 𝑏1,𝑜 ⋮ ⋱ ⋮ 𝑏𝑛,1 ⋯ 𝑏𝑛,𝑜 ⋅ 𝑐1,1 ⋯ 𝑐1,𝑛 ⋮ ⋱ ⋮ 𝑐𝑙,1 ⋯ 𝑐𝑙,𝑛 = ⋱ ⋰ 𝑏𝑟,𝑘 ⋅ 𝑐𝑗,𝑟

𝑙 𝑟=1

⋰ ⋱

Transposition

Matrix Transposition

• Swap rows and columns
• Formally:

⋱ ⋅ ⋰ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋰ ⋅ ⋱

T

= ⋱ ⋅ ⋅ ⋅ ⋰ ⋅ ⋅ ⋅ ⋅ ⋰ ⋅ ⋅ ⋅ ⋱

T =

𝑏𝑗,𝑘 𝑏𝑘,𝑗

Vectors

Vectors

• Column matrices
• Matrix-Vector product consistent

Co-Vectors

• “projectors”, “dual vectors”,

“linear forms”, “row vectors”

• Vectors to be projected on

Transposition

• Convert vectors into projectors and vice versa

𝐲 ∈ ℝ𝑒 𝐳T ∈ ℝ𝑒

Vectors

Inner product (as a generalized “projection”)

• Matrix-product 𝐝𝐩𝐦𝐯𝐧𝐨 ⋅ 𝐬𝐩𝐱

„𝐲 ⋅ 𝐳“ = 𝐲, 𝐳 = 𝐲T ⋅ 𝐳

• People use all three notations
• Meaning of “ ⋅ ” clear from context

𝐲T ⋅ 𝐳 → ℝ

Matrix-Vector Products

Two Interpretations

• Linear combination of column vectors
• Projection on row (co-)vectors

°

𝐍 ⋅ 𝐲 = 𝐳 𝐳 𝐍 𝐲

⋅ + ⋅ + ⋅ + ⋅

° ° ° °

⋅ ⋅ ⋅ ⋅ ⋅ =

Matrix Algebra

We can add and scalar multiply

• Matrices and vectors (special case)

We can matrix-multiply

• Matrices with other matrices

(execute one-after-another)

• Vectors in certain cases (next)

We can “divide” by some (not all) matrices

• Determine inverse matrix
• Full-rank, square matrices only

Algebraic Rules: Addition

Addition: like real numbers (“commutative group”)

• Prerequisites:
• Number of rows match
• Number of columns match
• Associative:

𝐁 + 𝐂 + 𝐃 = 𝐁 + 𝐂 + 𝐃

• Commutative: 𝐁 + 𝐂 = 𝐂 + 𝐁
• Subtraction:

𝐁 + −𝐁 = 𝟏

• Neutral Op.:

𝐁 + 𝟏 = 𝐁

𝐁, 𝐂, 𝐃 ∈ ℝ𝑜×𝑛

(matrices, same size)

Settings

Algebraic Rules: Scalar Multiplication

Scalar Multiplication: Vector space

• Prerequisites:
• Always possible
• Repeated Scaling: 𝜇 𝜈𝐁 = 𝜇𝜈 𝐁
• Neutral Operation: 1 ⋅ 𝐁 = 𝐁
• Distributivity 1:

𝜇(𝐁 + 𝐂) = 𝜇𝐁 + 𝜇𝐂

• Distributivity 2:

𝜇 + 𝜈 𝐁 = 𝜇𝐁 + 𝜈𝐁

So far:

• Matrices form vector space
• Just different notation, same semantics!

𝜇 ∈ ℝ 𝐁, 𝐂 ∈ ℝ𝑜×𝑛

(same size)

Settings

Algebraic Rules: Multiplication

Multiplication: Non-Commutative Ring / Group

• Prerequisites:
• Number of columns right

= number of rows left

• Associative:

𝐁 ⋅ 𝐂 ⋅ 𝐃 = 𝐁 ⋅ 𝐂 ⋅ 𝐃

• Not commutative: often 𝐁 ⋅ 𝐂 ≠ 𝐂 ⋅ 𝐁
• Neutral Op.:

𝐁 ⋅ 𝐉 = 𝐁

• Inverse:

𝐁 ⋅ 𝐁−1 = 𝐉

• Additional prerequisite:

– Matrix must be square! – Matrix must have full rank

Set of invertible matrices: 𝐻𝑀 𝑒 ⊂ ℝ𝑒×𝑒 “general linear group”

Algebraic Rules: Multiplication

Multiplication: Non-Commutative Ring / Group

• Prerequisites:
• Number of columns right

= number of rows left

• Associative:

𝐁 ⋅ 𝐂 ⋅ 𝐃 = 𝐁 ⋅ 𝐂 ⋅ 𝐃

• Not commutative: often 𝐁 ⋅ 𝐂 ≠ 𝐂 ⋅ 𝐁
• Neutral Op.:

𝐁 ⋅ 𝐉 = 𝐁

• Inverse:

𝐁 ⋅ 𝐁−1 = 𝐉

• Additional prerequisite:

– Matrix must be square! – Matrix must have full rank

Set of invertible matrices: 𝐻𝑀 𝑒 ⊂ ℝ𝑒×𝑒 “general linear group”

𝐁 ∈ ℝ𝑜×𝑛 𝐂 ∈ ℝ𝑛×𝑙 𝐃 ∈ ℝ𝑙×𝑚

Settings

Transposition Rules

Transposition

• Addition:

𝐁 + 𝐂 T = 𝐁T + 𝐂T = 𝐂T + 𝐁T

• Scalar-mult.:

𝜇𝐁 T = 𝜇𝐁T

• Multiplication:

𝐁 ⋅ 𝐂 T = 𝐂T ⋅ 𝐁T

• Self-inverse:

𝐁T T = 𝐁

• (Inversion:)

𝐁 ⋅ 𝐂 −1 = 𝐂−1 ⋅ 𝐁−1

• Inverse-transp.:

𝐁T −1 = 𝐁−1 T

• Othogonality:

𝐁T = 𝐁−1 ⇔ 𝐁 is orthogonal

Matrix Multiplication

Matrix Multiplication 𝐁 ⋅ 𝐂 = − 𝐛1 − ⋮ − 𝐛𝑒 − ⋅ | | 𝐜1 ⋯ 𝐜𝑒 | | = ⋱ ⋰ 𝐛𝑗, 𝐜𝑘 ⋰ ⋱

• Scalar products of rows and columns

Orthogonal Matrices

Othogonal Matrices

• (i.e., column vectors orthonormal)

𝐍𝑈 = 𝐍−1

• Proof: previous slide.

Scalar Product

Matrix Algebra:

• Scalar product is a special case

𝐲, 𝐳 = 𝐲T ⋅ 𝐳

• Caution when mixing with scalar-vector product!

𝐲, 𝐳 ⋅ 𝐴 ≠ 𝐲 ⋅ 𝐳, 𝐴 𝐲T ⋅ 𝐳 ⋅ 𝐴 ≠ 𝐲 ⋅ 𝐳T ⋅ 𝐴

⋅ ⋅ ≠

Scalar multiplication not a matrix-product!

Scalar Product

NOT OK

⋅ ⋅

OK

Scalar Product

What does work:

• Associativity with outer product

𝐲 ⋅ 𝐳, 𝐴 = 𝐲 ⋅ 𝐳T ⋅ 𝐴 = 𝐲 ⋅ 𝐳T ⋅ 𝐴

⋅ ⋅ =

advanced topics main ideas

ADV

Addendum

Axiomatic Mathematics

(This is not a core topic of the course; material is provided just for your information.)

“Class Diagram” for Real Numbers

field

binary operation tion magma semi-grou

• up

monoid group Abeli lian group

• perator +
• perator •
• rdered

ed field Real Numbers rs

Real Numbers

field

binary operation ion

binary operation: template <set T, operator ○> T operator”○”(T, T) throws DoesNotCompute

magma

closed binary operation: T operator”○”(T, T) no-exceptions

semi mi-gr group

• up

associativity: (A ○ B) ○ C = A ○ (B ○ C)

monoid

identity element “id”: id ○ A = A ○ id = A

group

inverse “T-1”: A ○ A-1 = A-1 ○ A = id

abelia ian group

commutativity: A ○ B = B ○ A

• perator +
• perator •

set with two operations template<set F> F operator+(F, F) F operator*(F, F)

• rdered

ed field

full order: template<set F> bool operator<(F, F)

Real Numbers rs

completeness: “all Cauchy series converge”

advanced topics main ideas

ADV

Structure: Vector Space

Vector Spaces

Vector space:

• Set of vectors V
• Based on field F (we use only F = ℝ)
• Two operations:
• Adding vectors u = v + w (u, v, w  V)
• Scaling vectors w = v (u  V,   F)

Vector Spaces

Vector space axioms:

• Vector addition – Abelian group:
• ∀𝐯, 𝐰, 𝐱 ∈ V: 𝐯 + 𝐰 + 𝐱 = 𝐯 + 𝐰 + 𝐱
• ∀𝐯, 𝐰 ∈ V: 𝐯 + 𝐰 = 𝐰 + 𝐯
• ∃𝟏 ∈ V: ∀𝐰 ∈ V: 𝐰 + 𝟏 = 𝐰
• ∀𝐰 ∈ V: ∃"−v" ∈ V: v +(−v) = 𝟏
• Compatibility with scalar multiplication:
• ∀𝐰 ∈ V, 𝜇, 𝜈 ∈ 𝐺: 𝜇 𝜈𝐯 = 𝜇𝜈 𝐯
• ∀𝐰 ∈ V: 1 ⋅ 𝐰 = 𝐰
• ∀𝐰, 𝐱 ∈ V, 𝜇 ∈ 𝐺: 𝜇(𝐰 + 𝐱) = 𝜇𝐰 + 𝜇𝐱
• ∀𝐰 ∈ V, 𝜇, 𝜈 ∈ 𝐺: 𝜇 + 𝜈 𝐰 = 𝜇𝐰 + 𝜈v

Settings

𝑊: vector space 𝐺: field (e.g., ℝ)

Properties

Some differences to our definition

• Abstract vector spaces can have infinite dimension
• For example: The set of all functions

𝑔: ℝ → ℝ forms an ∞-dimensional vector space

• But they always have a basis

→ coordinate representation

• We can use other fields than ℝ, such as ℂ or finite

fields such as (ℤ mod 𝑞, 𝑞 prime)

• We can recognize them before we have a

coordinate representation

Theorem

Theorem (“Basis-Isomorphism”)

• Any finite-dimensional vector space can be

represented by columns of numbers

• Use the 𝑒 coordinates of the 𝑒 basis vectors (dim= 𝑒)

Our definition makes sense

• Special case

advanced topics main ideas

ADV

Structure: Scalar Product

Scalar Product

Aximatic Definition: Scalar Product

• Function
• two vector arguments (input)
• one scalar output
• 𝑐: 𝑊 × 𝑊 → 𝐺

– think 𝑐 == “𝑝𝑞𝑓𝑠𝑏𝑢𝑝𝑠 ∘”

• 𝑊 is a vector space, F is a field (such as ℝ)

𝑊: vector space 𝐺: field (e.g., ℝ)

Settings

Axiomatic Definition: Scalar Product

Properties

• Symmetry

𝑐 𝐯, 𝐰 = 𝑐 𝐰, 𝐯

• Bilinearity

𝑐 𝐯 + 𝜇𝐰, 𝐱 = 𝑐 𝐯, 𝐱 + 𝑐 𝜇𝐰, 𝐱

(linearity in second argument follows from symmetry)

• Positive definite

𝑐 𝐯, 𝐯 ≥ 0, 𝑐 𝐯, 𝐯 = 𝟏 ⇒ 𝒗 = 𝟏

Symmetric, positive-definite, bilinear function

𝜇 ∈ 𝐺 𝐯, 𝐰, 𝐱 ∈ 𝑊

Settings

General Scalar Product

Theorem

• In a finite-dimensional vector space, any scalar

product has the following form: 𝑐 𝐲, 𝐳 = 𝐍𝐲 ⋅ 𝐍𝐳 = 𝐲T 𝐍T𝐍 𝐳

• “ ⋅ ” is the standard scalar product as we defined it
• M is a square matrix with linearly-independent columns

– I.e., M transforms to a different coordinate frame

Our definition still makes sense…

• Special case: undistorted coordinates
• General scalar products can take non-standard

coordinate frames into account

advanced topics main ideas

ADV

Structure: Linear Map

Definition of Linear Maps

Axioms

• Linear Map: A function

𝐁: 𝑊

1 → V2

maps from one vector space (𝑊

1) to another (𝑊 2)

• Linearity requires

𝐁 𝐰 + 𝐱 = 𝐁 ⋅ 𝐰 + 𝐁 ⋅ 𝐱 𝐁 ⋅ 𝜇 ⋅ 𝐰 = 𝜇 ⋅ 𝐁 ⋅ 𝐰

Theorem

• Linear maps in finite-dimensional vector spaces

can always be represented by matrices

• Our definition makes sense: special case

𝐁 – linear map 𝐰 ∈ 𝑊

1 - vector

Settings