20/11/2019 20/11/2019 2 Outline Introduction to Linear Algebra - - PDF document

20/11/2019 1

Leonardo de Knegt

Introduction to Linear Algebra

Outline

20/11/2019 2

• Algebra
• Solving equations
• Matrices
• Types of matrices
• Matrix operations
• Systems of linear equations
• Linear regression
• State-space models

Algebra x Arythmetics

20/11/2019 3

• Arythmetics
• Greek Arythmos: to count
• Computation of numbers
• Operations: sum, subtraction, multiplication, division
• Arythmetic expression: 7 + 3 = 3 + 7
• Algebra
• Arabic Al-jabr: the reunion of broken parts
• Combines unknown quantities and numbers
• Problem solving using generalizable rules
• Algebraic equation: a + b = b + a

Solving equations

20/11/2019 4

• Let a ≠ 0 and b be known real numbers, and x be

an unknown real number.

• If, for some reason, we know that

∙ we say that we have an equation.

• We can solve the equation

∙

• b

Solving equations

• Example

20/11/2019 5

Farmer Hansen has delivered milk to the dairy last week.

• He delivered 10.000kg
• He has received a total of 23.000 DKK as payment.

What is the price for 1 kg of milk? (Cheat: ∙ )

• 10.000
• 23.000
• So 10.000 ∙ 23.000, therefore

23.000 10.000 (, )*

• What is a matrix?
• Consider the system

3 + 6- + 2. 0 /1 + 5- + 23. 0 /2- + 4. 0

• We write the values of x1, x2 and x3 in matrix form:

3 6 2 /1 5 23 /2 4

Which is a 3x3 matrix (Notation: #89 : #;9)

20/11/2019 6

x1 x2 x3

20/11/2019 2

Matrices

• Another example

3 6 2 /1 5 23 3 /1 6 5 2 23 Symbolically we can express them as

• .
• .
• .

.-

20/11/2019 7

2x3 3x2

Types of matrices

A matrix of dimension : is called a quadratic (or rectangular) matrix: A matrix of dimension 1 : is called a row vector: A matrix of dimension : 1 is called a column vector

20/11/2019 8

Matrix operations Addition

• Two matrices may be added if they are of equal

dimensions (say : ) From the axioms of real numbers, it follows that the commutative law is valid for matrix operation: + +

20/11/2019 9

Matrix operations

Class exercise Additive identity Does a set of : matrices have a ‘zero’ element 0 so that for any : + 0 If yes, what does it look like?

20/11/2019 10

2 /1 15 5 4 /7 8 12 23 + = 2 /1 15 5 4 /7 8 12 23

Matrix operations

Like for real numbers, every matrix , has an additive inverse, , such that + 0, and, for the additive inverse, b /a Resulting in the ‘zero’ or ‘null’ matrix.

20/11/2019 11

2 /1 15 5 4 /7 8 12 23 + /2 1 /15 /5 /4 7 /8 /12 /23 =

Matrix operations

20/11/2019 12

Multiplication

• Two matrices ( and ) may be multiplied if a has the

same number of columns as b has rows

An element in the product is calculated as the product of a row and a column 5 4 3 6 1 2 2 3 2 1 2 4 3 2 1

21 30 15 24 22 26

A 3 x 3 matrix multiplied with a 3 x 2 matrix

a b

ra x ca rb x cb

20/11/2019 3

Matrix operations

Class exercise Does the commutative law a*b = b*a apply to matrices, as it applies to real numbers?

• Yes
• No
• Not always
• Maaaaybe...?

Why?

20/11/2019 13

Identical matrices Identity matrix Zero matrix

Matrix operations

Multiplicative identity Does the set of matrices have a ‘one’ element ?, so that if ? is an : matrix, then for any : matrix , ? ∙ If yes:

• What must the values of n necessarily be?
• What are the elements of ? – what does the matrix look like?

Does a ‘one’ element ?- exist such that for any matrix of given dimension, ∙ ?- If yes: Same questions as before.

20/11/2019 14

2 /1 15 5 4 /7 8 12 23 * 1 1 1 = 2 /1 15 5 4 /7 8 12 23

Matrix operations

Other operations

• A real number may be multiplied with a matrix

2*a =

• The transpose @ (often denoted as A or B) of a matrix is

formed by changing the columns to rows. Visualize it as a folding around the diagonal.

20/11/2019 15

2 ∗ 1 2 3 4 5 6

• 2

4 6 8 10 12

′ 1 3 5 2 4 6

Exercise C.1.1 – Appendix C

In R:

• A = matrix(c(2,-3,1,2,0,-1),2)

2 1 /3 2 /1

• B = matrix(c(1,2,-1,1,-1,0),3)

1 1 2 /1 /1

• Sum: A+B
• Multiplication: A%*%B
• Transposition: t(A)
• Inverse: solve(A)
• Determinant: det(A)

20/11/2019 16

Matrix operations

Multiplicative inverse I Does every matrix that are non-zero ( E 0) have a multiplicative inverse, , such that ∙ ? ? Only under certain conditions:

• Matrix is quadratic
• Matrix is non-singular
• is singular if and only if FG 0 where FG is the

determinant of .

• Many quadratic matrices are singular!

20/11/2019 17

Determinant

• The determinant of a quadratic matrix is a real

number.

• Calculation of the determinant is rather

complicated for large dimensions.

• The determinant of a 2 x 2 matrix:
• The determinant of a 3 x 3 matrix:

20/11/2019 4

Matrix operations

The inverse matrix A 2x2 matrix a is inverted by

• 1
• /-

/-

• Example

1 3 2

• 1

1 3 2 /3 /2 1 1 /6 /3 /2 1

• 1

2 1 3 / 1 6

20/11/2019 19

The inverse matrix Example of a 3x3 matrix

20/11/2019 20

Matrix operations Exercise C.1.2 – Appendix C

In R:

• A = matrix(c(2,-3,1,2,0,-1),2)

2 1 /3 2 /1

• B = matrix(c(1,2,-1,1,-1,0),3)

1 1 2 /1 /1

• Sum: A+B
• Multiplication: A%*%B
• Transposition: t(A)
• Inverse: solve(A)
• Determinant: det(A)

20/11/2019 21

WARNING

NOT POSSIBLE to:

• Divide by a matrix
• Sum a matrix with a number

Why?

20/11/2019 22

• Because they enable us to express very complex relations

in a very compact way.

• Because they enable us to solve large systems of linear

equations.

Systems of linear equations

Old McDonalds has a farm…

• On his farm he has some sheep, but he has forgotten how
• many. Let us denote the number as x1 .
• On his farm he has some geese, but he has forgotten how
• many. Let us denote the number as x2 .
• He has no other animals, and the other day he counted the

number of heads of his animals. The number was 25. He knows that sheep and geese have one head each, so he set up the following equation:

• 1x1 + 1x2 = 25
• He also counted the number of legs, and it was 70. He

knows that a sheep has 4 legs and a goose has 2 legs, so he set up the following equation:

• 4x1 + 2x2 = 70

20/11/2019 23

Systems of linear equations

• Old McDonalds Farm
• We have two equations with two unknowns:
• 1x1 + 1x2 = 25
• 4x1 + 2x2 = 70
• Define the following matrix a and the (column-) vectors x

and b

• We may then express the two equations as one matrix

equation:

20/11/2019 24

20/11/2019 5

Systems of linear equations

• Old McDonalds Farm

Remember: This solution is valid whether it’s a system of 2 equations or a million (which is not unrealistic). In R: a = matrix(c(1,1,4,2),2) b = (matrix(c(25,70),2) x = (solve(a))%*%b Solution: Solving this system we get

• 10

15 , meaning 10 sheep and 15 geese.

20/11/2019 25

Linear regression and matrices I

• In a study of children born in Berkeley 1928-29

the height and weight of 10 18-year old girls were measured.

• It is reasonable to assume that the weight Yi

depends on the height xi according to the following linear regression model:

• Yi = β0 + β1xi + εi where,
• β0 and β1 are unknown parameters
• The εi are N(0, σ2)

20/11/2019 26

Linear regression and matrices II

Let us define the following matrices

• We may then write our model in matrix notation simply as:

Y = xβ + ε

20/11/2019 27

Linear regression and matrices III

The least squares estimate of β is Define the vector of predictions as: Then an estimate,s2 , for the residual variance σ2 is:

• Where n = 10 is the number of observations and k = 2 is the number of parameters

estimated.

Yielding

20/11/2019 28

Complex relations

20/11/2019 29

• Madsen et al. (2005) performed an on-line monitoring of

the water intake of piglets. The water intake Yt at time t was expressed as

• Where
• Simple, but …

Complex relations II

20/11/2019 30

F, θt and wt are of dimension 25 x 1, G and Wt are of dimension 25 x 25.

• The value of θ

θ θ θt is what we try to estimate.

20/11/2019 6

Matrices and state-space models

Let’s do exercise C.1.3. a and b together, as an example.

20/11/2019 31