Matrix Arithmetic (Multidimensional Math) Shaina Race, PhD - - PowerPoint PPT Presentation

▶
matrix arithmetic
SMART_READER_LITE
LIVE PREVIEW

Matrix Arithmetic (Multidimensional Math) Shaina Race, PhD - - PowerPoint PPT Presentation

Aug 13, 2023 400 likes •1.2k views

Matrix Arithmetic (Multidimensional Math) Shaina Race, PhD Institute for Advanced Analytics. Element-wise Operations T ABLE OF C ONTENTS Linear Combinations of Matrices and Vectors. Vector Multiplication Inner products and Matrix-Vector

slide-1
SLIDE 1

Shaina Race, PhD Institute for Advanced Analytics.

Matrix Arithmetic

(Multidimensional Math)

slide-2
SLIDE 2

TABLE OF CONTENTS

Element-wise Operations

Linear Combinations of Matrices and Vectors.

Matrix Multiplication

Inner product and linear combination viewpoint

Vector Multiplication

The Outer Product

Vector Multiplication

Inner products and Matrix-Vector Multiplication

slide-3
SLIDE 3
  • Two matrices/vectors can be added/subtracted if

and only if they have the same size

  • Then simply add/subtract corresponding elements

Matrix Addition/Subtraction

slide-4
SLIDE 4

Matrix Addition/Subtraction

( ( (

(

( ( (

(

+ =(

(

+ + + + + + + + + + + + + + +

! " # # $ ## ! " # # $ ##

! " # # $ ##

A B A+B

(A + B)ij = Aij + Bij

Aij Bij (A + B)ij

(Element-wise)

slide-5
SLIDE 5

Example: Matrix Addition/Subtraction

slide-6
SLIDE 6

Scalar Multiplication

(

(

α

! " # # $ ##

M

(αM)ij = αMij

( (

=

α α α α α α α α α α α α α α α

! " # # $ ##

αM

(Element-wise)

slide-7
SLIDE 7

Geometric Look

Vector addition and scalar multiplication

slide-8
SLIDE 8

Points <—> Vectors

a

a = 1 2 ⎛ ⎝ ⎜ ⎞ ⎠ ⎟

Vectors have both direction and magnitude Direction arrow points from

  • rigin to the coordinate point

Magnitude is the length of that arrow #pythagoras

slide-9
SLIDE 9

Scalar Multiplication (Geometrically)

2a a

  • 0.5a
slide-10
SLIDE 10

Vector Addition (Geometrically)

a b a+b b a addition is still commutative

slide-11
SLIDE 11

Example: Centering the data

Average/Mean (Centroid)

(x1,x2)

x1 x2

slide-12
SLIDE 12

Example: Centering the data

x1 x2

slide-13
SLIDE 13

Example: Centering the data

x1 x2 New mean is the origin (0,0)

slide-14
SLIDE 14

Linear Combinations

slide-15
SLIDE 15

Linear Combinations


 A linear combination of vectors is a just weighted sum:

α1v1 +α 2v2 +…+α pv p

Scalar Coefficients ⍺i Vectors vi

slide-16
SLIDE 16
  • The simplest linear combination might involve

columns of the identity matrix (elementary vectors):

  • Picture this linear combination as a “breakdown into

parts” where the parts give directions along the 3 coordinate axes.

Elementary Linear Combinations

slide-17
SLIDE 17

Linear Combinations (Geometrically)

a b

  • 3

a-3b

slide-18
SLIDE 18

Linear Combinations (Geometrically)

(axis 1) (axis 2) (axis 3)

slide-19
SLIDE 19

Example: Linear Combination of Matrices

slide-20
SLIDE 20

TABLE OF CONTENTS

Element-wise Operations

Linear Combinations of Matrices and Vectors.

Vector Multiplication

Inner products and Matrix-Vector Multiplication

Matrix Multiplication

Inner product and linear combination viewpoint

Vector Multiplication

The Outer Product

slide-21
SLIDE 21
  • Throughout this course, unless otherwise specified,

all vectors are assumed to be columns.

  • Simplifies notation because if x is a column vector:

then we can automatically assume that xT is a row vector:

Notation: Column vs. Row Vectors

x = x1 x2 ! xn ⎛ ⎝ ⎜ ⎜ ⎜ ⎜ ⎞ ⎠ ⎟ ⎟ ⎟ ⎟

xT = (x1 x2 … xn)

slide-22
SLIDE 22
  • The vector inner product is the multiplication of a row

vector times a column vector.

  • It is known across broader sciences as the ‘dot product’.
  • The result of this product is a scalar.

Vector Inner Product

=

slide-23
SLIDE 23

(

(

( (

1 2 3 4 5 6

! " # # $ ##

aT

! " # # $ ##

b =

aTb =

i=1 n

∑aibi

Inner Product (row x column)

slide-24
SLIDE 24

( (

1 2 3 4 5 6

* * * * * *

+ + + + + =

Inner Product (row x column)

a and b must have the same number of elements.

slide-25
SLIDE 25

Examples: Inner Product

slide-26
SLIDE 26

Examples: Inner Product

slide-27
SLIDE 27

Check your Understanding

slide-28
SLIDE 28

Check your Understanding SOLUTION

slide-29
SLIDE 29

Matrix-Vector Multiplication

Inner Product View (I-P View)

slide-30
SLIDE 30

Matrix-Vector Multiplication (I-P view)

( (

( (

1 2 3

slide-31
SLIDE 31

( (

( (

Sizes must match up! 1 2 3

Matrix-Vector Multiplication (I-P view)

slide-32
SLIDE 32

( (

1 2 3

Matrix-Vector Multiplication (I-P view)

slide-33
SLIDE 33

( (

1 2 3

Matrix-Vector Multiplication (I-P view)

slide-34
SLIDE 34

( ( ( (

( (

=

Matrix-Vector Multiplication (I-P view)

slide-35
SLIDE 35

Example: Matrix-Vector Products

slide-36
SLIDE 36

Matrix-Vector Multiplication

Linear Combination View (L-C View)

slide-37
SLIDE 37

( (

( (

1 2 3

Matrix-Vector Multiplication (L-C view)

slide-38
SLIDE 38

( ( ( ( ( (

+ + = (

(

1 2 3

Matrix-Vector Multiplication (L-C view)

slide-39
SLIDE 39

⎛ ⎝ ⎜ ⎞ ⎠ ⎟

Example: Linear Combination View

2 −1 5 ⎛ ⎝ ⎜ ⎜ ⎜ ⎞ ⎠ ⎟ ⎟ ⎟ 3 4 1 ⎛ ⎝ ⎜ ⎜ ⎜ ⎞ ⎠ ⎟ ⎟ ⎟

3 2

slide-40
SLIDE 40

Example: Linear Combination View

2 −1 5 ⎛ ⎝ ⎜ ⎜ ⎜ ⎞ ⎠ ⎟ ⎟ ⎟ 3 4 1 ⎛ ⎝ ⎜ ⎜ ⎜ ⎞ ⎠ ⎟ ⎟ ⎟

3 2 + =

slide-41
SLIDE 41

TABLE OF CONTENTS

Element-wise Operations

Linear Combinations of Matrices and Vectors.

Matrix Multiplication

Inner product and linear combination viewpoint

Vector Multiplication

The Outer Product

Vector Multiplication

Inner products and Matrix-Vector Multiplication

slide-42
SLIDE 42
  • Matrix multiplication is NOT commutative.

AB≠BA

  • Matrix multiplication is only defined for dimension-

compatible matrices

Matrix-Matrix Multiplication

slide-43
SLIDE 43
  • If A and B are dimension compatible, then we

compute the product AB by multiplying every row

  • f A by every column of B (inner products).
  • The (i,j)th entry of the product AB is the ith row of

A multiplied by the jth column of B

Matrix-Matrix Multiplication (I-P View)

slide-44
SLIDE 44

A and B are dimension compatible for the product AB if the number of columns in A is equal to the number of rows in B

Matrix-Matrix Multiplication (I-P View)

(

(

( (

1 2 3 6 5 4 9 8 7 A S

( (

1 2 3 6 5 4 9 8 7 A S 3 4 7 S =

! " # # $ ## ! " # # $ ## ! " # # $ ##

A B (AB) 4 x 3 3 x 4 x 4 4

slide-45
SLIDE 45

(

(

( (

1 2 3 6 5 4 9 8 7 A S =

(

(

1 2 3 6 5 4 9 8 7 A S 3 4 7 S

(AB)ij = Ai!B! j

A2!

(AB)23

B!3

Matrix-Matrix Multiplication (I-P View)

slide-46
SLIDE 46

Example: Matrix-Matrix Multiplication

slide-47
SLIDE 47

Check Your Understanding

slide-48
SLIDE 48

Check your Understanding SOLUTION

slide-49
SLIDE 49

Check Your Understanding

slide-50
SLIDE 50

Check your Understanding SOLUTION

slide-51
SLIDE 51
  • Very important to remember that
  • As we see in previous exercise, common to be able

to compute product AB when the reverse product, BA, is not even defined.

  • Even when both products are possible, almost never

the case that AB = BA.

NOT Commutative

Matrix multiplication is NOT commutative!

slide-52
SLIDE 52

Diagonal Scaling

Multiplication by a diagonal matrix

slide-53
SLIDE 53

Multiplication by a diagonal matrix

The net effect is that the rows of A are scaled by the corresponding diagonal element of D

slide-54
SLIDE 54

Multiplication by a diagonal matrix

Rather than computing DA, what if we instead put the diagonal matrix on the right hand side and compute AD?

AD =

(Exercise)

slide-55
SLIDE 55

Matrix-Matrix Multiplication

As a Collection of Linear Combinations (L-C View)

slide-56
SLIDE 56
  • Just a collection of matrix-vector products 


(linear combinations) with different coefficients.

  • Each linear combination involves the same set of

vectors (the green columns) with different coefficients (the purple columns).

Matrix-Matrix Multiplication (L-C View)

slide-57
SLIDE 57

Matrix-Matrix Multiplication (L-C View)

(

(

( (

1 2 3 6 5 4 9 8 7 A S =

(

(

1 2 3 6 5 4 9 8 7 A S 3 4 7 S

slide-58
SLIDE 58

Matrix-Matrix Multiplication (L-C View)

(

(

( (

1 2 3 6 5 4 9 8 7 A S = =

( (

1 2 3 6 5 4 9 8 7 A S 3 4 7 S

( (

( (

slide-59
SLIDE 59

Matrix-Matrix Multiplication (L-C View)

(

(

( (

1 2 3 6 5 4 9 8 7 A S = =

( (

1 2 3 6 5 4 9 8 7 A S 3 4 7 S

( (

( (

( (

+ +

i=1 n

∑

a11 ⎛ ⎝ ⎜ ⎜ ⎜ ⎜ ⎞ ⎠ ⎟ ⎟ ⎟ ⎟

slide-60
SLIDE 60

Matrix-Matrix Multiplication (L-C View)

(

(

( (

1 2 3 6 5 4 9 8 7 A S = =

( (

1 2 3 6 5 4 9 8 7 A S 3 4 7 S

( (

( (

( (

+ +

slide-61
SLIDE 61

Matrix-Matrix Multiplication (L-C View)

(

(

( (

1 2 3 6 5 4 9 8 7 A S = =

( (

1 2 3 6 5 4 9 8 7 A S 3 4 7 S

( (

( (

( (

+ +

slide-62
SLIDE 62
  • Just a collection of matrix-vector products 


(linear combinations) with different coefficients.

  • Each linear combination involves the same set of

vectors (the green columns) with different coefficients (the purple columns).

  • This has important implications!

Matrix-Matrix Multiplication (L-C View)

slide-63
SLIDE 63

TABLE OF CONTENTS

Element-wise Operations

Linear Combinations of Matrices and Vectors.

Matrix Multiplication

Inner product and linear combination viewpoint

Vector Multiplication

The Outer Product

Vector Multiplication

Inner products and Matrix-Vector Multiplication

slide-64
SLIDE 64
  • The vector outer product is the multiplication of a

column vector times a row vector.

  • For any column/row this product is possible
  • The result of this product is a matrix!

Vector Outer Product

= (mx1) (1xn) (mxn)

slide-65
SLIDE 65

(

(

( (

1 2 3 4 5

! " # # $ ##

aT

! " # # $ ##

b

Outer Product (column x row)

=(

(

baT

! " # # $ ##

slide-66
SLIDE 66

(

(

( (

! " # # $ ##

aT

! " # # $ ##

b =

Outer Product (column x row)

(

(

baT

! " # # $ ##

slide-67
SLIDE 67

(

(

( (

! " # # $ ##

aT

! " # # $ ##

b =

Outer Product (column x row)

(

(

baT

! " # # $ ##

slide-68
SLIDE 68

Example: Outer Product

slide-69
SLIDE 69

From the previous example, you can see that the rows of an outer product are necessarily multiples of each other.

Outer Product has rank 1

(

(

( (

! " # # $ ##

aT

! " # # $ ##

b =(

(

baT

! " # # $ ##

slide-70
SLIDE 70

Matrix-Matrix Multiplication

As a Sum of Outer Products (O-P View)

slide-71
SLIDE 71

We can write the product AB as a sum of outer products of columns of A(mxn) and rows of B(nxp)

Matrix-Matrix Multiplication (O-P View)

AB = A!iBi!

i=1 n

∑

This view decomposes the product AB into the sum of n matrices, each of which has rank 1 (discussed later).

slide-72
SLIDE 72

Challenge Puzzle

  • Suppose we have 1,000 individuals that have been

divided into 5 different groups each year for 20 years.

  • We need to make a 1000x1000 matrix C where
  • The data currently has 1000 rows and 5x20 = 100

binary columns indicating whether each individual was a member of each group (yLgK: yearLgroupK): 
 (y1g1, y1g2, y1g3, y1g4, y1g5, y2g1, … y20g5)

  • Can we use what we’ve just learned to help us here?

Cij = # times person i grouped with person j

slide-73
SLIDE 73

Special Cases of Matrix Multiplication

The Identity and the Inverse

slide-74
SLIDE 74
  • The identity matrix, ‘I’, is to matrices what the

number ‘1’ is to scalars.

  • It is the multiplicative identity.
  • For any matrix (or vector) A, multiplying A by the

(appropriately sized) identity matrix on either side does not change A:

The Identity Matrix

AI = IA = A

I4 = 1 1 1 1 ⎛ ⎝ ⎜ ⎜ ⎜ ⎜ ⎞ ⎠ ⎟ ⎟ ⎟ ⎟

slide-75
SLIDE 75
  • For certain square matrices, A, an inverse matrix written

A-1, exists such that:

  • A MATRIX MUST BE SQUARE TO HAVE AN INVERSE.
  • NOT ALL SQUARE MATRICES HAVE AN INVERSE
  • Only full-rank, square matrices are invertible. (more on this later)
  • For now, understand that the inverse matrix serves like the

multiplicative inverse in scalar algebra:

  • Multiplying a matrix by its inverse (if it exists) yields the

multiplicative identity, I

The Matrix Inverse

slide-76
SLIDE 76
  • A square matrix which has an inverse is equivalently

called:

  • Non-singular
  • Invertible
  • Full Rank

The Matrix Inverse

slide-77
SLIDE 77

Don’t Cancel That!!

slide-78
SLIDE 78

Cancellation implies inversion

Canceling numbers in scalar algebra: Canceling matrices in linear algebra:

Inverses can help solve an equation…

WHEN THEY EXIST!