Lecture 1: Probability and Counting Ziyu Shao School of Information - PowerPoint PPT Presentation

Lecture 1: Probability and Counting Ziyu Shao School of Information Science and Technology ShanghaiTech University September 21, 2018 Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 1 / 47

Outline Good Books for Linear Algebra 1 Naive Definition of Probability 2 Counting 3 Definition of Probability 4 Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 2 / 47

Outline Good Books for Linear Algebra 1 Naive Definition of Probability 2 Counting 3 Definition of Probability 4 Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 3 / 47

Introduction to Linear Algebra Gilbert Strang Introduction to Linear Algebra (5 th Edition) Wellesley-Cambridge Press, 2016. http://math.mit.edu/ ~gs/linearalgebra/ Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 4 / 47

Coding The Matrix Philip N. Klein Coding The Matrix: Linear Algebra Through Computer Science Applications Newtonian Press, 2013 codingthematrix.com Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 5 / 47

Introduction to Applied Linear Algebra Stephen Boyd & Lieven Vandenberghe Introduction to Applied Linear Algebra: Vectors, Matrices, and Least Squares Cambridge University Press, 2018. http://vmls-book. stanford.edu/ Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 6 / 47

Practical Linear Algebra Gerald Farin & Dianne Hansford Practical Linear Algebra: A Geometry Toolbox (3 rd Edition) CRC Press, 2014. Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 7 / 47

Linear Algebra Done Right Sheldon Axler Linear Algebra Done Right (3 rd Edition) Springer, 2015. Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 8 / 47

No Bullshit Guide to Linear Algebra Ivan Savov No Bullshit Guide to Linear Algebra (2 nd Edition) Minireference Co., 2017. Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 9 / 47

Outline Good Books for Linear Algebra 1 Naive Definition of Probability 2 Counting 3 Definition of Probability 4 Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 10 / 47

Set = A set is a collection of objects. Given two sets A , B , key concepts include empty set : ; A is a subset of B: A ✓ B me union of A and B : A [ B intersection of A and B : A \ B - complement of A : A c ~ De Morgan’s laws : ( A [ B ) c = A c \ B c ( A \ B ) c = A c [ B c Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 11 / 47

Venn Diagram Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 12 / 47

Sample Space & Event e- The sample space S of an experiment: the set of all possible ⇒ outcomes of the experiment. ef1,2,...q}_ C) An event A is a subset of the sample space S . A={ 1. 2,3>4,5 } _- A occurred if the actual outcome is in A . - 13=156.8 ,q } z 2 I 67 4 5 8 9 Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 13 / 47

Example: Coin flips 1,111k¥ A coin is flipped 10 times. Writing Heads as 1 and Tails as 0. Then An outcome is a sequence ( s 1 , s 2 , . . . , s 10 ) with s j 2 { 0 , 1 } . The sample space: the set of all such sequences. D - A j : the event that the j th flip is Head. - B : the event that at least one flip was Head. ( B = S 10 j =1 A j ) ~ - -- C : the event that all the flips were Heads. ( C = \ 10 j =1 A j ) mm - D : the event that there were at least two consecutive Heads. ÷ ( D = S 9 j =1 ( A j \ A j +1 )) - Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 14 / 47

Translation Between English & Sets ÷ = - -r - r me Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 15 / 47

Naive Definition of Probability Assumption 1: finite sample space Assumption 2: all outcomes occur equally likely Definition Let A be an event for an experiment with a finite sample space S. The naive probability of A is P naive ( A ) = | A | | S | = number of outcomes favorable to A . total number of outcomes in S - Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 16 / 47

Outline Good Books for Linear Algebra 1 Naive Definition of Probability 2 Counting 3 Definition of Probability 4 Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 17 / 47

Multiplication Rule You buy an ice cream cone with several choices: cone: cake or wa ffl e == flavor: chocolate, vanilla, or strawberry - - 0 0 Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 18 / 47

Multiplication Rule in General Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 19 / 47

Sampling With Replacement . # ' ×*L× Ik }=n@ . ... Theorem Consider n objects and making k choices from them, one at a time . - with replacement (i.e., choosing a certain object does not preclude it from being chosen again). Then there are n k possible outcomes. Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 20 / 47

Sampling Without Replacement . = ° I # I n man , } hen # nt 2 ¥n - Theorem Consider n objects and making k choices from them, one at a time . without replacement (i.e., choosing a certain object preclude it from being chosen again). Then there are n ( n � 1) · · · ( n � k + 1) possible outcomes for k  n (and 0 possibilities for k > n). Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 21 / 47

Example: Birthday Problem There are k people in a room. Assume each person’s birthday is - equally likely to be any of the 365 days of the year (we exclude February 29), and that people’s birthdays are independent (we . assume there are no twins in the room). What is the probability that two or more people in the group have the same birthday? Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 22 / 47

Solution of Birthday Problem k O 10=1 365 > - k£365 � 2 � . " have the A birthday ' ' 32 same = AC " " two People = no ' ' - . i i"ii= k ( without match ) tputc ) PLA ) ( = = # ofugs assign birthdays to k . . ( 365 k = without replacer . ) , , = 1 . @ 65 . k replacement ) , 365.364 -365£ - ( 365 KTH . . . - 1- = Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 23 / 47

Solution of Birthday Problem l : I : 1 I 23 Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 24 / 47

&@ 2@ Generalized Birthday Problem . . " A " 32 man = . ... . - ktl ) i ) - NCN vithon .cn i. . - . . - . . 1- - PLA ) =p = MR sample nvtmveplucem Approximations . Int ) 1 ' In .tn p=x a- . . Each of k people has a random number (“birthday”) drawn from - - . .ch#n)EeIxEL n values (“days”). , ] tunhtas = . mm If the probability that at least two people have the same number is 50%, then k ⇡ 1 . 18 p n . . ,.e_a@ . EE etn.EE ' Fit . . d- .tk i ) CHZT . . = f- e- £i# =L - a ' e easy be PCA 1=0.5 . Fini k=Fn .i8Jn ⇒ knOcJ# al - Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 25 / 47

Application: Hash Table - A commonly used data structure for fast information retrieval ÷ Example: store people’s name. For each people x , a hash function h is computed. h ( x ): the location that will be used to store x 0 s name. - 0 0 , " ~ . Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 26 / 47

Hash Collision ÷ Collision: x 6 = y , but h ( x ) = h ( y ) ( � 1 locations has � 2 names stored there) Given k people (di ff erent names) and n locations, what is the - probability of occurrence of hash collision? - Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 27 / 47

Solution of Hash Collision Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 28 / 47

Binomial Coe ffi cient no C ! Definition 0 � n � For any nonnegative integers k and n , the binomial coe ffi cient , k read as “ n choose k ”, is the number of subsets of size k for a set of ncn-D.gg?h# size n . Theorem For k  n, we have 0 ✓ n ◆ = n ( n � 1) · · · ( n � k + 1) n ! = k k ! k !( n � k )! Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 29 / 47

Example: Bose-Einstein 'nteg÷ 1,2€ ( ¥ }n° ) ' Xj jthobgaes C # chosen ) : . =k Xn Xitxzt + . . . nonnegative - How many ways are there to choose k times from a set of n objects with replacement, if order doesn’t matter (we only care about how =L - many times each object was chosen, not the order in which they were chosen)? Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 30 / 47