Chapter 1: Probability Theory (a recap) STK4011/9011: Statistical - PowerPoint PPT Presentation

Chapter 1: Probability Theory (a recap) STK4011/9011: Statistical Inference Theory Johan Pensar STK4011/9011: Statistical Inference Theory Chapter 1: Probability Theory (a recap) 1 / 11

Overview Set Theory 1 Probability Function 2 Conditional Probability 3 Independence 4 Random Variables 5 Distribution Functions 6 Covers parts of Sec 1.1–1.2 and most of Sec 1.3–1.6 in CB. STK4011/9011: Statistical Inference Theory Chapter 1: Probability Theory (a recap) 2 / 11

Set Theory - Sample Space One of the main objectives of a statistician is to draw conclusions about a population of objects through experiments. A critical step in such experiments is to identify the possible outcomes, or sample space , which is defined as a set S . The sample space can be countable (possibly finite) or uncountable. Examples: Tossing a six-sided die: S = { 1 , 2 , 3 , 4 , 5 , 6 } . Tossing two coins: S = { HH , HT , TH , TT } (ordered) or S = { HH , HT , TT } (unordered). Reaction time: S = (0 , ∞ ). STK4011/9011: Statistical Inference Theory Chapter 1: Probability Theory (a recap) 3 / 11

Set Theory - Event For a given sample space, we are interested in collections of possible outcomes, or events . An event, A , is simply a subset of the sample space, that is, A ⊆ S . Given two events, A , B ⊆ S , there are some elementary operations: Union: A ∪ B = { x : x ∈ A or x ∈ B } . Intersection: A ∩ B = { x : x ∈ A and x ∈ B } . Complement: A c = { x : x �∈ A } . The elementary operations can be combined in a similar way as addition and multiplication (see Thm 1.1.4 in CB). STK4011/9011: Statistical Inference Theory Chapter 1: Probability Theory (a recap) 4 / 11

Set Theory - Partition of the Sample Space Events A and B are disjoint (or mutually exclusive) if A ∩ B = ∅ . Events A 1 , A 2 , . . . are pairwise disjoint (or mutually exclusive) if A i ∩ A j = ∅ for all i � = j . If events A 1 , A 2 , . . . are pairwise disjoint and ∪ ∞ i =1 A i = S , then the events form a partition of S . Example: The events A i = [ i , i + 1) , i = 0 , 1 , 2 , . . . form a partition of S = [0 , ∞ ). STK4011/9011: Statistical Inference Theory Chapter 1: Probability Theory (a recap) 5 / 11

Probability Function For each event A in the sample space S , we want to assign a probability P ( A ) ∈ [0 , 1] (through a so-called probability function). More formally: Given a sample space S and an associated sigma algebra* B , a probability function is a function P with domain B that satisfies the following properties: P ( A ) ≥ 0 for all A ∈ B . 1 P ( S ) = 1. 2 i =1 A i ) = � ∞ If A 1 , A 2 , . . . ∈ B are pairwise disjoint, then P ( ∪ ∞ i =1 P ( A i ). 3 The above properties are referred to as the Kolmogorov axioms. Using these basic properties, many useful properties of the probability function can be derived (see Sec 1.2.2 in CB). STK4011/9011: Statistical Inference Theory Chapter 1: Probability Theory (a recap) 6 / 11

Conditional Probability If A and B are events in S , and P ( B ) > 0, then the conditional probability of A given B , denoted by P ( A | B ), is P ( A | B ) = P ( A ∩ B ) . P ( B ) Bayes’ rule: Let A 1 , A 2 , . . . be a partition of the sample space, and let B be any set. Then, for each i = 1 , 2 , . . . , : P ( B | A i ) P ( A i ) P ( A i | B ) = j =1 P ( B | A j ) P ( A j ) . � ∞ STK4011/9011: Statistical Inference Theory Chapter 1: Probability Theory (a recap) 7 / 11

Independence In some cases, the occurrence of an event A has no effect on the probability of another event B . Formally: Two events, A and B , are statistically independent if P ( A ∩ B ) = P ( A ) P ( B ) . Furthermore: A collection of events A 1 , . . . , A n are mutually independent if for any subcollection A i 1 , . . . , A i k , we have k P ( ∩ k � j =1 A i j ) = P ( A i j ) . j =1 STK4011/9011: Statistical Inference Theory Chapter 1: Probability Theory (a recap) 8 / 11

Random Variables A random variable X is a function that maps the original sample space S into the real numbers, X : S → R . X : outcome space of X . X = x i or x i : X has taken on the value x i . Example - Coin tossing: X : { heads , tails } → { 0 , 1 } . In most experiments, it makes more sense to deal with a summary variable rather than having to consider all the elements in the original sample space. Example - Sum of two dice: X : { (1 , 1) , (1 , 2) , . . . , (6 , 6) } → { 2 , ..., 12 } . STK4011/9011: Statistical Inference Theory Chapter 1: Probability Theory (a recap) 9 / 11

Cumulative Distribution Function For a random variable X , the cumulative distribution function (cdf) is denoted by F X ( x ), and defined by F X ( x ) = P X ( X ≤ x ) , for all x . A cdf satisfies certain properties: F ( x ) is a cdf iff the following conditions hold: lim x →−∞ F ( x ) = 0 and lim x →∞ F ( x ) = 1. 1 F ( x ) is a nondecreasing function. 2 F ( x ) is right-continuous. 3 A random variable is continuous if F X ( x ) is a continuous function and discrete if F X ( x ) is a step function. STK4011/9011: Statistical Inference Theory Chapter 1: Probability Theory (a recap) 10 / 11

Probability Mass and Density Functions The probability mass function (pmf) of a discrete random variable X is given by f X ( x ) = P ( X = x ) , for all x . The probability density function (pdf) of a continuous random variable X is the function f X ( x ) that satisfies � x F X ( x ) = f X ( t ) dt , for all x . −∞ A function f X ( x ) is a pmf (or pdf) of a random variable X iff f X ( x ) ≥ 0 for all x . 1 � ∞ � x f X ( x ) = 1 or −∞ f X ( x ) dx = 1. 2 STK4011/9011: Statistical Inference Theory Chapter 1: Probability Theory (a recap) 11 / 11