Chapter 1: Probability Theory (a recap) STK4011/9011: Statistical - - PowerPoint PPT Presentation

Chapter 1: Probability Theory (a recap)

STK4011/9011: Statistical Inference Theory

Johan Pensar

STK4011/9011: Statistical Inference Theory Chapter 1: Probability Theory (a recap) 1 / 11

Overview

1

Set Theory

2

Probability Function

3

Conditional Probability

4

Independence

5

Random Variables

6

Distribution Functions Covers parts of Sec 1.1–1.2 and most of Sec 1.3–1.6 in CB.

STK4011/9011: Statistical Inference Theory Chapter 1: Probability Theory (a recap) 2 / 11

Set Theory - Sample Space

One of the main objectives of a statistician is to draw conclusions about a population of

• bjects through experiments.

A critical step in such experiments is to identify the possible outcomes, or sample space, which is defined as a set S. The sample space can be countable (possibly finite) or uncountable. Examples:

Tossing a six-sided die: S = {1, 2, 3, 4, 5, 6}. Tossing two coins: S = {HH, HT, TH, TT} (ordered) or S = {HH, HT, TT} (unordered). Reaction time: S = (0, ∞).

STK4011/9011: Statistical Inference Theory Chapter 1: Probability Theory (a recap) 3 / 11

Set Theory - Event

For a given sample space, we are interested in collections of possible outcomes, or events. An event, A, is simply a subset of the sample space, that is, A ⊆ S. Given two events, A, B ⊆ S, there are some elementary operations:

Union: A ∪ B = {x : x ∈ A or x ∈ B}. Intersection: A ∩ B = {x : x ∈ A and x ∈ B}. Complement: Ac = {x : x ∈ A}.

The elementary operations can be combined in a similar way as addition and multiplication (see Thm 1.1.4 in CB).

STK4011/9011: Statistical Inference Theory Chapter 1: Probability Theory (a recap) 4 / 11

Set Theory - Partition of the Sample Space

Events A and B are disjoint (or mutually exclusive) if A ∩ B = ∅. Events A1, A2, . . . are pairwise disjoint (or mutually exclusive) if Ai ∩ Aj = ∅ for all i = j. If events A1, A2, . . . are pairwise disjoint and ∪∞

i=1Ai = S, then the events form a partition

• f S.

Example: The events Ai = [i, i + 1), i = 0, 1, 2, . . . form a partition of S = [0, ∞).

STK4011/9011: Statistical Inference Theory Chapter 1: Probability Theory (a recap) 5 / 11

Probability Function

For each event A in the sample space S, we want to assign a probability P(A) ∈ [0, 1] (through a so-called probability function). More formally: Given a sample space S and an associated sigma algebra* B, a probability function is a function P with domain B that satisfies the following properties:

1

P(A) ≥ 0 for all A ∈ B.

2

P(S) = 1.

3

If A1, A2, . . . ∈ B are pairwise disjoint, then P(∪∞

i=1Ai) = ∞ i=1 P(Ai).

The above properties are referred to as the Kolmogorov axioms. Using these basic properties, many useful properties of the probability function can be derived (see Sec 1.2.2 in CB).

STK4011/9011: Statistical Inference Theory Chapter 1: Probability Theory (a recap) 6 / 11

Conditional Probability

If A and B are events in S, and P(B) > 0, then the conditional probability of A given B, denoted by P(A | B), is P(A | B) = P(A ∩ B) P(B) . Bayes’ rule: Let A1, A2, . . . be a partition of the sample space, and let B be any set. Then, for each i = 1, 2, . . . ,: P(Ai | B) = P(B | Ai)P(Ai) ∞

j=1 P(B | Aj)P(Aj).

STK4011/9011: Statistical Inference Theory Chapter 1: Probability Theory (a recap) 7 / 11

Independence

In some cases, the occurrence of an event A has no effect on the probability of another event B. Formally: Two events, A and B, are statistically independent if P(A ∩ B) = P(A)P(B). Furthermore: A collection of events A1, . . . , An are mutually independent if for any subcollection Ai1, . . . , Aik, we have P(∩k

j=1Aij) = k

• j=1

P(Aij).

STK4011/9011: Statistical Inference Theory Chapter 1: Probability Theory (a recap) 8 / 11

Random Variables

A random variable X is a function that maps the original sample space S into the real numbers, X:S → R.

X: outcome space of X. X = xi or xi: X has taken on the value xi.

Example - Coin tossing: X:{heads, tails} → {0, 1}. In most experiments, it makes more sense to deal with a summary variable rather than having to consider all the elements in the original sample space. Example - Sum of two dice: X:{(1, 1), (1, 2), . . . , (6, 6)} → {2, ..., 12}.

STK4011/9011: Statistical Inference Theory Chapter 1: Probability Theory (a recap) 9 / 11

Cumulative Distribution Function

For a random variable X, the cumulative distribution function (cdf) is denoted by FX(x), and defined by FX(x) = PX(X ≤ x), for all x. A cdf satisfies certain properties: F(x) is a cdf iff the following conditions hold:

1

limx→−∞ F(x) = 0 and limx→∞ F(x) = 1.

2

F(x) is a nondecreasing function.

3

F(x) is right-continuous.

A random variable is continuous if FX(x) is a continuous function and discrete if FX(x) is a step function.

STK4011/9011: Statistical Inference Theory Chapter 1: Probability Theory (a recap) 10 / 11

Probability Mass and Density Functions

The probability mass function (pmf) of a discrete random variable X is given by fX(x) = P(X = x), for all x. The probability density function (pdf) of a continuous random variable X is the function fX(x) that satisfies FX(x) = x

−∞

fX(t) dt, for all x. A function fX(x) is a pmf (or pdf) of a random variable X iff

1

fX(x) ≥ 0 for all x.

2

• x fX(x) = 1 or

∞

−∞ fX(x) dx = 1.

STK4011/9011: Statistical Inference Theory Chapter 1: Probability Theory (a recap) 11 / 11