Probability basics DS GA 1002 Statistical and Mathematical Models - PowerPoint PPT Presentation

Probability basics DS GA 1002 Statistical and Mathematical Models http://www.cims.nyu.edu/~cfgranda/pages/DSGA1002_fall16 Carlos Fernandez-Granda

Probability spaces Conditional probability Independence

General approach Probabilistic modeling 1. Model phenomenon of interest as an experiment with several (possibly infinite) mutually exclusive outcomes 2. Group these outcomes in sets called events 3. Assign probabilities to the different events

Probability space A probability space is a triple (Ω , F , P ) consisting of ◮ A sample space Ω , which contains all possible outcomes of the experiment ◮ A set of events F , which must be a σ algebra ◮ A probability measure P that assigns probabilities to the events in F

Sample space Sample spaces can be ◮ Discrete: coin toss, score of a basketball game, number of people that show up at a party . . . ◮ Continuous: intervals of R or R n used to model time, position, temperature, . . .

σ -algebra A σ -algebra F is a collection of sets in Ω such that 1. If a set S ∈ F then S c ∈ F 2. If the sets S 1 , S 2 ∈ F , then S 1 ∪ S 2 ∈ F Also infinite sequences; if S 1 , S 2 , . . . ∈ F then ∪ ∞ i = 1 S i ∈ F 3. Ω ∈ F

Basketball game ◮ Cleveland Cavaliers are playing the Golden State Warriors ◮ Sample space Ω := { Cavs 1 − Warriors 0 , Cavs 0 − Warriors 1 , . . . , Cavs 101 − Warriors 97 , . . . } . ◮ Several possible σ algebras ◮ If we want high granularity we can choose the power set of scores ◮ If we only care who wins F := { Cavs win , Warriors win , Cavs or Warriors win , ∅}

Probability measure Function over the sets in F such that 1. P ( S ) ≥ 0 for any event S ∈ F 2. If S 1 , S 2 ∈ F are disjoint then P ( S 1 ∪ S 2 ) = P ( S 1 ) + P ( S 2 ) Also countably infinite sequences of disjoint sets: S 1 , S 2 , . . . ∈ F n � � � n →∞ ∪ n lim = lim P ( S i ) P i = 1 S i n →∞ i = 1 3. P (Ω) = 1

Properties of a probability measure ◮ P ( ∅ ) = 0 ◮ If A ⊆ B then P ( A ) ≤ P ( B ) ◮ P ( A ∪ B ) = P ( A ) + P ( B ) − P ( A ∩ B )

Important ◮ Probability measure only assigns probabilities to events in the σ algebra ◮ Simpler σ algebras can make our life easy P ( Cavs win ) = 1 2 P ( Warriors win ) = 1 2 P ( Cavs or Warriors win ) = 1 P ( ∅ ) = 0

Probability spaces Conditional probability Independence

Definition The conditional probability of an event S ′ ∈ F given S is := P ( S ′ ∩ S ) � � S ′ | S P P ( S ) P ( ·| S ) is a valid probability measure

Example: Flights and rain Probabilistic model for late arrivals at an airport Ω = { late and rain , late and no rain , on time and rain , on time and no rain } F = power set of Ω , P ( late , no rain ) = 2 P ( on time , no rain ) = 14 20 , 20 , P ( late , rain ) = 3 P ( on time , rain ) = 1 20 , 20 P ( late | rain ) ?

Chain rule For any pair of events A and B P ( A ∩ B ) = P ( A ) P ( B | A ) = P ( B ) P ( A | B ) For any sequence of events S 1 , S 2 , S 3 , . . . P ( ∩ i S i ) = P ( S 1 ) P ( S 2 | S 1 ) P ( S 3 | S 1 ∩ S 2 ) . . . � � � S i | ∩ i − 1 = P j = 1 S j i

Law of Total Probability If A 1 , A 2 , . . . ∈ F is a partition of Ω ◮ A i and A j are disjoint if i � = j ◮ Ω = ∪ i A i For any set S ∈ F � � P ( S ) = P ( S ∩ A i ) = P ( A i ) P ( S | A i ) i i

Example: Flights and rain (continued) P ( rain ) = 0 . 2 P ( late | rain ) = 0 . 75 P ( late | no rain ) = 0 . 125 P ( late ) ?

Important! P ( A | B ) � = P ( B | A )

Bayes’ Rule Let A 1 , A 2 , . . . ∈ F be a partition of Ω For any set S ∈ F P ( A i ) P ( S | A i ) P ( A i | S ) = � j P ( S | A j ) P ( A j )

Example: Flights and rain (continued) P ( rain ) = 0 . 2 P ( late | rain ) = 0 . 75 P ( late | no rain ) = 0 . 125 P ( rain | late ) ?

Example: Flights and rain (continued) P ( rain | late )

Example: Flights and rain (continued) P ( rain | late ) = P ( rain , late ) P ( late )

Example: Flights and rain (continued) P ( rain | late ) = P ( rain , late ) P ( late ) P ( late | rain ) P ( rain ) = P ( late | rain ) P ( rain ) + P ( late | no rain ) P ( no rain )

Example: Flights and rain (continued) P ( rain | late ) = P ( rain , late ) P ( late ) P ( late | rain ) P ( rain ) = P ( late | rain ) P ( rain ) + P ( late | no rain ) P ( no rain ) 0 . 75 · 0 . 2 = 0 . 75 · 0 . 2 + 0 . 125 · 0 . 8 = 0 . 6

Probability spaces Conditional probability Independence

Definition Two sets A , B are independent if P ( A | B ) = P ( A ) or equivalently P ( A ∩ B ) = P ( A ) P ( B )

Conditional independence A , B are conditionally independent given C if P ( A | B , C ) = P ( A | C ) where P ( A | B , C ) := P ( A | B ∩ C ) , or equivalently P ( A ∩ B | C ) = P ( A | C ) P ( B | C )

Conditional independence does not imply independence Probabilistic model for taxi availability, flight delay and weather P ( rain ) = 0 . 2 P ( late | rain ) = 0 . 75 P ( late | no rain ) = 0 . 125 P ( taxi | rain ) = 0 . 1 P ( taxi | no rain ) = 0 . 6 Given rain and no rain , late and taxi are conditionally independent Are they also independent? P ( taxi ) = P ( taxi | late ) ?

Conditional independence does not imply independence P ( taxi )

Conditional independence does not imply independence P ( taxi ) = P ( taxi | rain ) P ( rain ) + P ( taxi | no rain ) P ( no rain )

Conditional independence does not imply independence P ( taxi ) = P ( taxi | rain ) P ( rain ) + P ( taxi | no rain ) P ( no rain ) = 0 . 1 · 0 . 2 + 0 . 6 · 0 . 8 = 0 . 5

Conditional independence does not imply independence P ( taxi ) = P ( taxi | rain ) P ( rain ) + P ( taxi | no rain ) P ( no rain ) = 0 . 1 · 0 . 2 + 0 . 6 · 0 . 8 = 0 . 5 P ( taxi | late )

Conditional independence does not imply independence P ( taxi ) = P ( taxi | rain ) P ( rain ) + P ( taxi | no rain ) P ( no rain ) = 0 . 1 · 0 . 2 + 0 . 6 · 0 . 8 = 0 . 5 P ( taxi | late ) = P ( taxi , late ) P ( late )

Conditional independence does not imply independence P ( taxi ) = P ( taxi | rain ) P ( rain ) + P ( taxi | no rain ) P ( no rain ) = 0 . 1 · 0 . 2 + 0 . 6 · 0 . 8 = 0 . 5 P ( taxi | late ) = P ( taxi , late ) P ( late ) = P ( taxi , late , rain ) + P ( taxi , late , no rain ) P ( late )

Conditional independence does not imply independence P ( taxi ) = P ( taxi | rain ) P ( rain ) + P ( taxi | no rain ) P ( no rain ) = 0 . 1 · 0 . 2 + 0 . 6 · 0 . 8 = 0 . 5 P ( taxi | late ) = P ( taxi , late ) P ( late ) = P ( taxi , late , rain ) + P ( taxi , late , no rain ) P ( late ) = P ( t | l , r ) P ( l | r ) P ( r ) + P ( t | l , no r ) P ( l | no r ) P ( no r ) P ( l )

Conditional independence does not imply independence P ( taxi ) = P ( taxi | rain ) P ( rain ) + P ( taxi | no rain ) P ( no rain ) = 0 . 1 · 0 . 2 + 0 . 6 · 0 . 8 = 0 . 5 P ( taxi | late ) = P ( taxi , late ) P ( late ) = P ( taxi , late , rain ) + P ( taxi , late , no rain ) P ( late ) = P ( t | l , r ) P ( l | r ) P ( r ) + P ( t | l , no r ) P ( l | no r ) P ( no r ) P ( l ) = P ( taxi | r ) P ( late | r ) P ( r ) + P ( taxi | no r ) P ( late | no r ) P ( no r ) P ( late )

Conditional independence does not imply independence P ( taxi ) = P ( taxi | rain ) P ( rain ) + P ( taxi | no rain ) P ( no rain ) = 0 . 1 · 0 . 2 + 0 . 6 · 0 . 8 = 0 . 5 P ( taxi | late ) = P ( taxi , late ) P ( late ) = P ( taxi , late , rain ) + P ( taxi , late , no rain ) P ( late ) = P ( t | l , r ) P ( l | r ) P ( r ) + P ( t | l , no r ) P ( l | no r ) P ( no r ) P ( l ) = P ( taxi | r ) P ( late | r ) P ( r ) + P ( taxi | no r ) P ( late | no r ) P ( no r ) P ( late ) = 0 . 1 · 0 . 75 · 0 . 2 + 0 . 6 · 0 . 125 · 0 . 8 = 0 . 3 0 . 25

Conditional independence does not imply independence P ( taxi ) = P ( taxi | rain ) P ( rain ) + P ( taxi | no rain ) P ( no rain ) = 0 . 1 · 0 . 2 + 0 . 6 · 0 . 8 = 0 . 5 P ( taxi | late ) = P ( taxi , late ) P ( late ) = P ( taxi , late , rain ) + P ( taxi , late , no rain ) P ( late ) = P ( t | l , r ) P ( l | r ) P ( r ) + P ( t | l , no r ) P ( l | no r ) P ( no r ) P ( l ) = P ( taxi | r ) P ( late | r ) P ( r ) + P ( taxi | no r ) P ( late | no r ) P ( no r ) P ( late ) = 0 . 1 · 0 . 75 · 0 . 2 + 0 . 6 · 0 . 125 · 0 . 8 = 0 . 3 0 . 25 They are not independent

Independence does not imply conditional independence Probabilistic model for mechanical problems, weather and delays P ( rain ) = 0 . 2 P ( late | rain ) = 0 . 75 P ( late | no rain ) = 0 . 125 P ( problem ) = 0 . 1 P ( late | problem ) = 0 . 7 P ( late | no problem ) = 0 . 2 P ( late | no rain , problem ) = 0 . 5 problem and no rain are independent Are they also conditionally independent given late ? P ( problem | late , no rain ) = P ( problem | late ) ?

Independence does not imply conditional independence P ( problem | late )

Independence does not imply conditional independence P ( problem | late ) = P ( late , problem ) P ( late )

Independence does not imply conditional independence P ( problem | late ) = P ( late , problem ) P ( late ) P ( late | p ) P ( p ) = P ( late | p ) P ( p ) + P ( late | no p ) P ( no p )

Independence does not imply conditional independence P ( problem | late ) = P ( late , problem ) P ( late ) P ( late | p ) P ( p ) = P ( late | p ) P ( p ) + P ( late | no p ) P ( no p ) 0 . 7 · 0 . 1 = 0 . 7 · 0 . 1 + 0 . 2 · 0 . 9 = 0 . 28