1. Foundations of Numerics from Advanced Mathematics 1. Foundations - - PowerPoint PPT Presentation

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

• 1. Foundations of Numerics from Advanced Mathematics
• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 1 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

The main purpose of this first chapter (about 4 weeks) is to recall those topics from your Advanced Mathematics courses (linear algebra, calculus, stochastics) typical for the first two years of bachelor’s programs in science and engineering that are of particular importance for numerical algorithms and, hence, for the whole CSE master’s program. We do this,

• since you can hardly go successfully through a thorough numerical education

without these foundations;

• since we made the experience that the CSE freshmen’s mathematical

backgrounds are quite heterogeneous (and not always at hand ...);

• since TUM’s CSE program has a methodological (i. e. mathematical and

informatical) point of view that goes beyond the usual and widespread engineering approach and way of thinking;

• and since the two numerics courses have been the most serious roadblock for

CSE students since the program’s launch (too high failure rates – something we want to reduce without touching the level of education). If you are familiar with all this stuff, don’t feel bored – just consider this as a warm-up to the numerical contents to be discussed later on in this course.

• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 2 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

We also changed the name of the courses from “Numerical Analysis” to “Numerical Programming”, to indicate that there are mathematical topics to be addressed, but with a clear focus on algorithmics, programming, and applications (instead of proofs etc.). This introductory part won’t be a complete lecture with all explanations etc. Rather, it will be a “guided tour” through important topics, mentioning notions and buzzwords that should have some meaning for you. If they don’t, you know that you have to close the gaps as soon as possible, with the help of the references provided or by doing additional exercises etc. Also use the tutorials to refresh your knowledge!

• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 3 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

1.1. Mathematical Essentials and Notation Symbols and Notions

Everyone familiar with

• the symbol ∞;
• the symbols ∃, ∃1, and ∀ (so-called quantifiers);
• the symbols

n

X

i=1

and Y

i=k

;

• the notions maximum, minimum, infimum, and supremum;
• Kronecker symbol δij;
• the Landau symbol O(N), O(h2)
• the symbol ⇒;
• the meaning of sufficient and necessary;
• the meaning of iff: sufficient and necessary;
• the meaning of associative, commutative, and distributive?
• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 4 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Numbers

• Booleans: true/false; logical operations; relations of logics to set theory (see

below)

• natural numbers, integers N, Z: factorials; binomial coefficients; Pascal’s triangle
• rational numbers Q: countable/non-countable
• real numbers R:

– field property (allows for arithmetic operations) – order property (allows for comparison) – completeness property (each interval nesting defines exactly one real number) – supremum/infimum property

• Q is dense in R
• different classes of irrational numbers:

√ 2, e, ...

• complex numbers C: imaginary unit i, Re, and Im; conjugate complex
• fundamental theorem of algebra: each polynomial of degree n with complex

coefficients has at least one complex root

• what else can be said of roots of polynomials?
• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 5 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Sets

• notions of sets, subsets, and elements
• set operations: union, intersection, difference, complement
• symbols ∈, ⊂, ⊆
• power set
• Cartesian product of sets
• appearances:

– explicit {1, 2, 3, ...} – implicit {x ∈ R : f(x) = 0}

• already here a bit of topology: open sets, closed sets, bounded sets, compact

sets

• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 6 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Relations

• definition: relation R between two sets A and B as a subset of A × B:

R ⊆ A × B

• notation: aRb or (a, b) ∈ R
• important examples for A = B: <, ≤, >, ≥, ...
• properties of relations:

– transitive – reflexive – symmetric – asymmetric – antisymmetric – connex

• notion of an equivalence relation
• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 7 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Mappings and Functions

• mapping or function (here used in a synonymous way)

f : A → B : ∀x ∈ A ∃1y ∈ B such that y = f(x); write x → y

• properties of mappings:

– injective – surjective – bijective

• f−1(x) = ?
• inverse mapping: existence and meaning
• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 8 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Building Blocks ...

... of a math course / book / presentation:

• definition: new notions etc. are defined and, thus, introduced
• theorem / proposition:

– a central statement, typically consisting of conditions and a conclusion (“if this and that holds, then the following is valid ...”) – the more restrictions are made, the more can be concluded (but also the less general the statements are)

• lemma: similar to a theorem w.r.t. its structure, but usually only an auxiliary

statement of minor importance by itself (that marks just a step on the way to a theorem, e.g.)

• corollary: a statement that follows immediately from a previous theorem etc.
• proof: a precise argumentation showing clearly that a theorem, lemma, or

corollary is correct Note that all this is typically formulated as general and generic as possible – a fact which is frequently misinterpreted as “not concrete” or “without practical relevance”.

• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 9 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

A Short Remark on Proofs

Why proofs – or how much of proofs?

• Proofs are the essence of mathematical argumentation – they make the latter

rigorous.

• Proofs are a permanent source of misunderstandings and problems:

– math professors often do not want to do anything without proofs – even in courses for non-mathematicians – non-math students often think that only the results or statements are relevant, but not the proofs (which they suppose to be something for hardcore mathematicians only) – note that both points of view are problematic

• hence: proofs for non-mathematicians (such as CSE students)?

– yes, if the way of proving a statement helps to understand it – no, if just for itself (i.e. just to prove it)

• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 10 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Standard Proof Techniques

• forward: A ⇒ B ⇒ C ⇒ D
• by contradiction (“what if”): ¬D ⇒ ... ⇒ ¬A
• by counterexample: to refute the assertion that all students are smart, just find
• ne stupid and the job is done
• complete search: to prove that all students are smart, check them all
• mathematical / complete induction: show the statement for n = 1, and show

the conclusion from n to n + 1 (does it work for the smart student example?)

• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 11 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

1.2. Linear Algebra Mathematical Structures

• a mathematical structure consists of one or several sets and one or several
• perations defined on the set(s)
• special elements:

– neutral element (of an operation) – inverse element (of some element x)

• a group: a structure to add and subtract
• a field: a structure to add, subtract, multiply, and divide
• a vector space: a set with additional properties, allows for addition and

multiplication with scalars

• note: sometimes, the association with classical (geometric) vectors is helpful,

sometimes it is more harmful

• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 12 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Vector Spaces

• a linear combination of vectors
• linear (in)dependence of a set of vectors
• the span of a set of vectors
• a basis of a vector space

– definition? – why do we need a basis? – is a vector’s basis representation unique? – is there only one basis for a vector space?

• the dimension of a vector space
• does infinite dimensionality exist?
• important applications:

– (analytic) geometry – numerical and functional analysis: function spaces are vector spaces (frequently named after mathematicians: Banach spaces, Hilbert spaces, Sobolev spaces, ...)

• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 13 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Linear Mappings

• definition in the vector space context; notion of a homomorphism
• image and kernel of a homomorphism
• matrices, transposed and Hermitian of a matrix
• relations of matrices and homomorphisms
• meaning of injective, surjective, and bijective for a matrix; rank of a matrix
• meaning of the matrix columns for the underlying mapping
• matrices and systems of linear equations
• basis transformation and coordinate transformation
• mono-, epi, iso-, endo-, and automorphisms
• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 14 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Determinants

• definition
• properties
• meaning
• occurrences
• Cramer’s rule
• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 15 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Eigenvalues

• notions of eigenvalue, eigenvector, and spectrum
• similar matrices A, B:

∃S : B = SAS−1 (i.e.: A and B as two basis representations of the same endomorphism)

• resulting objective: look for the best / cheapest representation (diagonal form)
• important: matrix A is diagonalizable iff there is a basis consisting of

eigenvectors only

• characteristic polynomial, its roots are the eigenvalues
• Jordan normal form
• important:

– spectrum characterizes a matrix – many situations / applications where eigenvalues are crucial

• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 16 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Scalar Products and Vector Norms

• notions of a linear form and a bilinear form
• scalar product: a positive-definite symmetric bilinear form
• examples of vector spaces and scalar products
• vector norms:

– definition: positivity, homogeneity, triangle inequality – meaning of triangle inequality – examples: Euclidean, maximum, and sum norm

• normed vector spaces
• Cauchy-Schwarz inequality
• notions of orthogonality and orthonormality
• turning a basis into an orthonormal one: Gram-Schmidt orthogonalization
• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 17 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Matrix Norms

• definition:

– properties corresponding to those of vector norms – plus sub-multiplicativity: AB ≤ A · B – plus consistency Ax ≤ A · x

• matrix norms can be induced from corresponding vector norms: Euclidean,

maximum, sum A := max

x=1 Ax

• alternative: completely new definition, for example Frobenius norm (consider

matrix as a vector, then take Euclidean norm)

• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 18 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Classes of Matrices

• symmetric: A = AT
• skew-symmetric: A = −AT
• Hermitian: A = AH = ¯

AT

• s.p.d. (symmetric positive definite): xT Ax > 0 ∀x = 0
• orthogonal: A−1 = AT (the whole spectrum has modulus 1)
• unitary: A−1 = AH (the whole spectrum has modulus 1)
• normal: AAT = AT A or AAH = AHA, resp. (for those and only those matrices

there exists an orthonormal basis of eigenvectors)

• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 19 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

1.3. Calculus Functions Revisited

• notions of a function, its range, and its image
• graph of a function
• isolines and isosurfaces
• sums and products of functions
• composition of functions
• inverse of a function: when existing?
• simple properties: (strictly) monotonous
• explicit and implicit definition
• parametrized representations (curves, ...)
• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 20 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Continuity

• remember the ε and the δ!
• definition of local (“in x0”) and global continuity (“∀x”)
• what about sums, products, quotients, ... of continuous functions?
• what about compositions of continuous functions?
• what about continuity of the inverse?
• intermediate value theorem
• continuous functions on compact sets – maximum and minimum value
• uniform continuity
• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 21 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Limits

• meaning of ε → 0 and N → ∞ and x → ∞
• accumulation point of a set
• limit (value) of a set
• limits from the left or from the right, respectively: f(x+), f(x−)
• limits at infinity: limx→∞ f(x)
• infinite limits: f(x) → ∞
• how can discontinuities look like?

– jumps: f(x+) = f(x−) – holes: f(x+) = f(x−) = f(x) – second kind: f(x) = 0 in x = 0 and f(x) = sin ` 1

x

´ elsewhere

• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 22 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Sequences

• definition of a sequence: a function f defined on N
• if f(n) = an, write

(an)

• r

a1, a2, a3, ...

• bounded / monotonously increasing / monotonously decreasing sequences
• notion of convergence of a sequence: existence of a limit for n → ∞
• Cauchy sequence
• subsequences
• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 23 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Series

• notion of an (infinite) series

– elements of a series – partial sums of a series – convergence defined by convergence of the sequence of the partial sums – convergence and absolute convergence

• examples:

– geometric series: P∞

k=1 xk = 1 1−x

– harmonic series: P∞

k=1 1 k = ∞

– alternating harmonic series: P∞

k=1(−1)k−1 1 k = ln(2)

• criteria for convergence: quotient and root criterion
• power series: P∞

k=0 ak(z − a)k

– coefficients ak and centre point a – radius of convergence R: absolute convergence for |z − a| < R – identity theorem for power series

• re-arrangement
• sums of series, nested series, products of series (Cauchy product)
• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 24 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Differentiation

• first step: functions f of one real variable, complex values allowed
• derivative or differential quotient of f:

– defined via limit process of difference quotients – write f′ or ˙ f or df

dx

– geometric meaning? – local and global differentiability – derivative from the left / from the right

• rules for the daily work:

– derivative of f + g, fg, and f/g? – derivative of f(g) (chain rule)? – derivative of the inverse function?

• higher derivatives f(k)(x); meaning
• notion of continuous differentiability
• smoothness of a function
• space of k-times continuously differentiable functions: Ck
• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 25 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Differential Calculus of one Real Variable

• notion of a global/local minimum/maximum
• local extrema and the first derivative
• mean value theorem:

∃ξ ∈ (a, b) : f′(ξ) = f(b) − f(a) b − a

• monotonous behaviour and the first derivative
• local extrema and the second derivative
• rule of de l’Hospital
• notions of convexity and concavity
• convexity/concavity and the second derivative
• notion of a turning point
• turning points and the second derivative
• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 26 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Function Classes (1)

• polynomials

– definition, degree, sums and products, division with rest, identity theorem, roots and their multiplicity

• rational functions

– poles and their multiplicity, partial fraction decomposition

• exponential function and logarithm

– characterising law of the exponential function: exp(s + t) = exp(s) · exp(t)

• r

y′ = y (functional equation of natural growth) – series expansion of the exponential function, speed of growth – natural logarithm as exp’s inverse: y = exp(x) = ex, x = ln(y) – functional equation: ln(xy) = ln(x) + ln(y) – exponential function and logarithm for general basis a: ax := ex ln a, loga(y) := ln y ln a

• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 27 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Function Classes (2)

• hyperbolic functions

– cosh(z), sinh(z), ...

• trigonometric functions

– sin(x), cos(x): solutions of y(2) + y = 0 – geometric meaning? – Euler’s formula: eix = cos(x) + i · sin(x) – derivatives, addition theorem – periodicity – series expansion

• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 28 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Integral Calculus of one Variable

• Riemann integral, upper and lower sums
• approximation by staircase functions
• properties:

– linearity – monotonicity

• mean value theorem:

ξ ∈ (a, b) : Z b

a

f(x)dx = (b − a) · f(ξ)

• main theorem of differential and integral calculus:

– define F(x) := R x

a f(t)dt

– then R b

a f(t)dt = F(b) − F(a)

• rules for everyday work:

– partial integration: Z uv′dx = uv − Z vu′dx – substitution: Z b

a

f(t(x))t′(x)dx = Z t(b)

t(a)

f(t)dt

• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 29 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Local Approximation: Taylor Polynomials and Series

• local approximation of functions with polynomials
• generalization of the tangent approximation used for the definition of the derivative
• Taylor polynomials:

– let f be n-times differentiable in a – we look for a polynomial T with T (k) = f(k) for k = 0, 1, ..., n – obviously: T(x) :=

n

X

k=0

1 k! f(k)(a)(x − a)k – unique, degree n, write Tnf(x; a) – remainder Rn+1(x) := f(x) − Tnf(x : a) Rn+1(x) = f(n+1)(ξ) (n + 1)! (x − a)n+1

• Taylor series:

– for infinitely differentiable functions (exp, sin, cos, ...) – sum up to ∞ instead of n only – examples

• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 30 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Global Approximation: Uniform Convergence

• convergence of sequences of functions fn defined on D:
• pointwise: for each x ∈ D; then

f(x) := lim

n→∞ fn(x)

defines a function – problems: are properties such as continuity or differentiability inherited from the fn to f, and how to calculate derivatives or integrals of f? – i.e., can the order of limit processes be changed?

• therefore the notion of uniform convergence:

– definition: fn − fD → 0 for n → ∞ – with that, the inheritance and change-order problems from above are solved! – criteria: Cauchy, ...

• approximation theorem of Weierstrass: each continuous function f on a

compact set can be arbitrarily well approximated with some polynomial

• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 31 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Simple Differential Equations

• notion of a differential equation

– ordinary: one variable – partial: more than one variable (several spatial dimensions or space and time)

• examples:

– growth: ˙ y = k · y or ˙ y = k(t, y) · y – oscillation: ¨ y + y = 0 or similar

• example of an analytic solution strategy: separation of variables

y′ = g(x) · h(y), y(x) = y0 – formal separation: dy h(y) = g(x)dx – integration of the left and right side – some requirements for applicability

• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 32 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Periodic Functions

• target now: periodic functions, period typically 2π
• trigonometric polynomials
• definition:

T(x) :=

n

X

k=−n

ckeikx = a0 2 +

n

X

k=1

(ak cos(kx) + bk sin(kx)) (coefficients ck, ak, and bk are unique)

• formula for the coefficients:

ck = 1 2π Z 2π T(x)e−ikxdx

• T is real iff all ak, bk are real iff ck = c−k
• Weierstrass: 2π-periodic continuous functions can be arbitrarily well

approximated by trigonometric polynomials

• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 33 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Fourier Series

• consider vector space of 2π-periodic complex functions f on R
• Fourier coefficients:

ˆ f(k) := 1 2π Z 2π f(x)e−ikxdx

• Fourier polynomial:

Snf(x) :=

n

X

k=−n

ˆ f(k)eikx

• Fourier series:

∞

X

−∞

ˆ f(k)eikx

• sine-cosine representation of Snf:

Snf(x) = a0 2 +

n

X

k=1

(ak cos(kx) + bk sin(kx))

• coefficients:

ak = ˆ f(k) + ˆ f(−k) = 1 π Z π

−π

f(x) cos(kx)dx bk = i( ˆ f(k) − ˆ f(−k)) = 1 π Z π

−π

f(x) sin(kx)dx

• all ak vanish for odd f, all bk vanish for even f
• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 34 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Functions of Several Variables

• f now defined on Rn or a subset of it
• notion of differentiability: now via existence of a linear map, the differential
• directional derivatives
• partial derivatives
• prominent differentiability criterion: existence and continuity of all partial

derivatives

• the gradient of a scalar function f and its interpretation
• the Jacobian of a vector-valued function f
• mean value theorem
• higher partial derivatives, Taylor approximation, Hessian
• local minima and maxima, criteria
• saddle points
• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 35 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Integration over Domains

• a huge field, from which we only mention a few results
• theorem of Fubini:

– shows that, in many cases, a multi-dimensional integration domain can be tackled dimension by dimension – statement (we neglect the requirements, for which more integration theory is needed): Z

X×Y

f(x, y)d(x, y) = Z

Y

„Z

X

f(x, y)dx « dy = Z

X

„Z

Y

f(x, y)dy « dx – related to Cavalieri’s principle – will also be of relevance for numerical quadrature

• transformation theorem:

– a generalisation of integration by substitution – statement, again without requirements: Z

U

f(T(x)) · ‚ ‚det T ′(x) ‚ ‚ dx = Z

V

f(y)dy – allows for a change of the coordinate system (polar coordinates), e.g.

• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 36 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Gauss Theorem

• we further generalise integration, now allowing for integration over hyper-surfaces

(a sphere, e.g.)

• this is important for the physical modelling in many scenarios (heat flux through a

pot’s surface, ...)

• the famous Gauss theorem allows to combine integrals over volumes and

surfaces, which occurs in the derivation of many physical models (conservation laws) and, hence, is of special relevance for CSE

• prerequisites:

– a vector field: a vector-valued function on Rn (example: the velocity field in fluid mechanics) – the divergence of a vector field F: div F(x) =

n

X

i=1

∂iFi(x)

• finally the Gauss theorem:

– several regularity assumptions needed Z

G

div Fdx = Z

∂G

F dS

• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 37 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

1.4. Stochastics and Statistics

Not a primary ingredient of CSE, but you should have at hand at least some basic knowledge (for things to come later: Monte-Carlo-based simulations, stochastic PDE, ...):

• probability theory: subfield of mathematics providing the apparatus to formalise

random events and studying what can be derived within that formalism; typical question: if the world looks this and that, what can we say concerning the results

• f a random experiment?
• mathematical statistics: not that firmly connected to mathematics; descriptive

statistics deals with the presentation of huge data sets, inductive statistics deals with conclusions from measured or observed values of random variables to their underlying properties

• stochastics: sometimes used in a synonymous way to probability theory,

sometimes as a notion covering probability theory and statistics

• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 38 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Combinatorics

Combinatorics deals with counting possible cases of a certain characteristics in a discrete and finite universe. The most famous and important example: Select k ≤ n elements from a set of n elements. Remember the four cases:

• with repetition, ordered:

nk

• without repetition, ordered:

nk =

k−1

Y

i=0

(n − i) = n! (n − k)!

• without repetition, unordered:

“n k ” = n! (n − k)!k!

• with repetition, unordered:

“n + k − 1 k ”

• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 39 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Discrete Probability Spaces: Events (1)

• random experiments have results ωi from a discrete (possibly infinite) result set

Ω = {ω1, ...}

• events as subsets of this result set: A, B ⊆ Ω
• A ∩ B, A ∪ B, ¯

A, A \ B

• elementary events {ωi} with probabilities pi
• rules: ∀i : pi ∈ [0, 1], P

Ω pi = 1

• handling probabilities:

p( ¯ A) = 1−p(A), p(A∪B) = p(A)+p(B)−p(A∩B), A ⊆ B ⇒ p(A) ≤ p(B), ...

• conditional probabilities (check that p(.|B) are probabilities):

p(A|B) := p(A ∩ B) p(B)

• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 40 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Discrete Probability Spaces: Events (2)

• multiplication theorem:

p(A1 ∩ ... ∩ An) = p(A1) · p(A2|A1) · p(A3|A1 ∩ A2) · ... · p(An|A1 ∩ ... ∩ An−1)

• theorem of total probability for pairwise disjoint Ai:

p(B) = X

i

p(B|Ai) · p(Ai)

• theorem of Bayes:

p(Aj|B) = p(Aj ∩ B) p(B) = p(B|Aj) · p(Aj) P

i p(B|Ai) · p(Ai)

• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 41 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Independence of Events

• intuitive definition: two events A, B are independent, if there is no influence

among each other: p(A|B) = p(A), p(B|A) = p(B)

• formal definition: independence, if

p(A ∩ B) = p(A) · p(B)

• examples?
• independence of a set of events A1, ..., An:

p @\

i∈I

Ai 1 A = Y

i∈I

p(Ai) ∀I ⊆ {1, ..., n}

• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 42 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Discrete Random Variables

• definition: X : Ω → R
• X defines events:

“X = x” := {ωi ∈ Ω : X(ωi) = x}, ...

• discrete density (function):

fX(x) := p(X = x)

• discrete distribution (function):

FX(x) := p(X ≤ x)

• expectation or mean value:

E(X) := X

x

x · p(X = x)

• variance:

V (X) := E ` (X − E(X))2´

• rules how to work with E(X) and V (X):

E(aX + b) = aE(X) + b, V (aX + b) = a2V (X), ...

• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 43 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Important Discrete Distributions

• Bernoulli distribution:

– binary random variable – results 1 and 0 with probabilities p and 1 − p, resp.

• Binomial distribution

– sum of n Bernoulli variables – p(X = k) = `n

k

´ pk(1 − p)n−k

• geometric distribution

– repeat a Bernoulli experiment until success (result 1) – p(X = i) = p(1 − p)i−1

• Poisson distribution

– counts events – possible results 0, 1, 2, 3, ... – density fX(i) =

e−λλi i!

• exercise: think about examples, calculate E(X) and V (X) for all
• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 44 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Continuous Probability Spaces

• motivation:

– think about the differences of discrete and continuous realities and probability spaces

• sums get integrals now:

– expectation value: E(X) := Z ∞

−∞

tfX(t)dt – density: p(A) := Z

A

fX(t)dt, Z ∞

−∞

fX(t)dt = 1 – distribution: FX(x) := Z x

−∞

f(t)dt – properties of fX and FX?

• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 45 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Important Continuous Distributions

• uniform distribution

– no differences in probability – density: fX(x) = 

1 b−a

for x ∈ [a, b] ,

• therwise.
• normal or Gaussian distribution

– frequent in reality, especially in limit cases, see slide on asymptotics – density: fX(x) = 1 √ 2πσ · e− (x−µ)2

2σ2

, FX(x) =: Φ(x) ∼ N(µ, σ)

• negative exponential distribution

– describes time between events – density: fX(x) =  λ · e−λx if x ≥ 0 ,

• therwise
• exercise: think about examples and calculate E(X) and V (X) for all
• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 46 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Asymptotics (1)

• inequality of Markov:

p(X ≥ t) ≤ E(X) t .

• inequality of Chebyshev:

p(| X − E(X) |≥ t) ≤ V (X) t2 .

• theorem of large numbers:

– given random variables Xi, i = 1, 2, ..., iid (independent and identically distributed) with mean value µ and variance σ2 – define arithmetic average Zn of X1, ..., Xn: Zn := 1 n ·

n

X

i=1

Xi . – given ε, δ > 0 and n ∈ N with n ≥

σ2 εδ2

– then it holds p(| Zn − µ |≥ δ) ≤ ε

• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 47 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Asymptotics (2)

• central limit theorem:

– given random variables Xi, i = 1, 2, ..., iid with mean value µ and variance σ2 > 0 – define Yn, n = 1, 2, ... as n-th partial sum of Xi: Yn :=

n

X

i=1

Xi – let Zn be the normalized random variable of Yn defined as Zn := Yn − nµ σ√n – Then: All Zn are asymptotically standardized normally distributed, i. e. lim

n→∞ Zn

∼ N(0, 1) Particularly, it holds for the series of distribution functions FZn of Zn lim

n→∞ FZn(x)

= Φ(x) ∀x ∈ R

• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 48 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Inductive Statistics (1)

• (random) sample:

– repeat a random experiment – record results as an iid sequence (ωi) or (xi), resp.

• objective: learn about reality with the help of the sample
• first technique: estimators: Y

:= g(X1, ..., Xn) – properties: unbiased, variance-reducing, efficient, consistent – examples: Y := ¯ X := 1 n

n

X

i=1

Xi for E(X) Y := S2 := 1 n − 1

n

X

i=1

(Xi − ¯ X)2 for V (X)

• maximum likelihood principle and estimators
• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 49 of 50

Mathematical Essentials and Notation Linear Algebra Calculus Stochastics and Statistics

Inductive Statistics (2)

• second technique: confidence intervals

– construct two estimators U1, U2 to define an interval for an unknown parameter θ (E(X), V (X), ...): p(U1 ≤ θ ≤ U2) ≥ 1 − α

• third technique: tests

– hypothesis H0 and alternative H1 on the size of an unknown parameter E(X) = 27.2, p < 0.9, ... – make a sample – determine a range where to reject the hypothesis – error types: error of first kind, error of second kind – maximum risk of first kind: significance level of the test – exercise: study simple examples

• 1. Foundations of Numerics from Advanced Mathematics

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 50 of 50