Lecture 1: Probability and Counting Ziyu Shao School of Information - - PowerPoint PPT Presentation

▶
lecture 1 probability and counting
SMART_READER_LITE
LIVE PREVIEW

Lecture 1: Probability and Counting Ziyu Shao School of Information - - PowerPoint PPT Presentation

Mar 13, 2024 270 likes •762 views

Lecture 1: Probability and Counting Ziyu Shao School of Information Science and Technology ShanghaiTech University September 21, 2018 Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 1 / 47 Outline Good Books

slide-1
SLIDE 1

Lecture 1: Probability and Counting

Ziyu Shao School of Information Science and Technology ShanghaiTech University September 21, 2018 Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 1 / 47
slide-2
SLIDE 2

Outline

1 Good Books for Linear Algebra 2 Naive Definition of Probability 3 Counting 4 Definition of Probability Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 2 / 47
slide-3
SLIDE 3

Outline

1 Good Books for Linear Algebra 2 Naive Definition of Probability 3 Counting 4 Definition of Probability Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 3 / 47
slide-4
SLIDE 4

Introduction to Linear Algebra

Gilbert Strang Introduction to Linear Algebra (5th Edition) Wellesley-Cambridge Press, 2016. http://math.mit.edu/ ~gs/linearalgebra/ Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 4 / 47
slide-5
SLIDE 5

Coding The Matrix

Philip N. Klein Coding The Matrix: Linear Algebra Through Computer Science Applications Newtonian Press, 2013 codingthematrix.com Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 5 / 47
slide-6
SLIDE 6

Introduction to Applied Linear Algebra

Stephen Boyd & Lieven Vandenberghe Introduction to Applied Linear Algebra: Vectors, Matrices, and Least Squares Cambridge University Press, 2018. http://vmls-book. stanford.edu/ Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 6 / 47
slide-7
SLIDE 7

Practical Linear Algebra

Gerald Farin & Dianne Hansford Practical Linear Algebra: A Geometry Toolbox (3rd Edition) CRC Press, 2014. Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 7 / 47
slide-8
SLIDE 8

Linear Algebra Done Right

Sheldon Axler Linear Algebra Done Right (3rd Edition) Springer, 2015. Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 8 / 47
slide-9
SLIDE 9

No Bullshit Guide to Linear Algebra

Ivan Savov No Bullshit Guide to Linear Algebra (2nd Edition) Minireference Co., 2017. Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 9 / 47
slide-10
SLIDE 10

Outline

1 Good Books for Linear Algebra 2 Naive Definition of Probability 3 Counting 4 Definition of Probability Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 10 / 47
slide-11
SLIDE 11

Set

A set is a collection of objects. Given two sets A, B, key concepts include empty set: ; A is a subset of B: A ✓ B union of A and B: A [ B intersection of A and B: A \ B complement of A: Ac De Morgan’s laws: (A [ B)c = Ac \ Bc (A \ B)c = Ac [ Bc Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 11 / 47

=

me

  • ~
slide-12
SLIDE 12

Venn Diagram

Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 12 / 47
slide-13
SLIDE 13

Sample Space & Event

The sample space S of an experiment: the set of all possible
  • utcomes of the experiment.
An event A is a subset of the sample space S. A occurred if the actual outcome is in A. Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 13 / 47 e-

⇒

ef1,2,...q}_

C)

_- A={ 1. 2,3>4,5 }
  • I
2 z 13=156.8 ,q } 4 5

67

8 9
slide-14
SLIDE 14

Example: Coin flips

A coin is flipped 10 times. Writing Heads as 1 and Tails as 0. Then An outcome is a sequence (s1, s2, . . . , s10) with sj 2 {0, 1}. The sample space: the set of all such sequences. Aj: the event that the jth flip is Head. B: the event that at least one flip was Head. (B = S10 j=1 Aj) C: the event that all the flips were Heads. (C = \10 j=1Aj) D: the event that there were at least two consecutive Heads. (D = S9 j=1(Aj \ Aj+1)) Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 14 / 47

1,111k¥

  • D
  • ~
  • mm
  • ÷
slide-15
SLIDE 15

Translation Between English & Sets

Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 15 / 47

÷

  • =
r
  • r
me
slide-16
SLIDE 16

Naive Definition of Probability

Assumption 1: finite sample space Assumption 2: all outcomes occur equally likely Definition Let A be an event for an experiment with a finite sample space S. The naive probability of A is Pnaive(A) = |A| |S| = number of outcomes favorable to A total number of outcomes in S . Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 16 / 47
slide-17
SLIDE 17

Outline

1 Good Books for Linear Algebra 2 Naive Definition of Probability 3 Counting 4 Definition of Probability Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 17 / 47
slide-18
SLIDE 18

Multiplication Rule

You buy an ice cream cone with several choices: cone: cake or waffle flavor: chocolate, vanilla, or strawberry Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 18 / 47
  • ==
slide-19
SLIDE 19

Multiplication Rule in General

Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 19 / 47
slide-20
SLIDE 20

Sampling With Replacement

Theorem Consider n objects and making k choices from them, one at a time with replacement (i.e., choosing a certain object does not preclude it from being chosen again). Then there are nk possible outcomes. Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 20 / 47

.

# ' ×*L× . ...

Ik

}=n@
  • .
slide-21
SLIDE 21

Sampling Without Replacement

Theorem Consider n objects and making k choices from them, one at a time without replacement (i.e., choosing a certain object preclude it from being chosen again). Then there are n(n 1) · · · (n k + 1) possible
  • utcomes for k  n (and 0 possibilities for k > n).
Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 21 / 47

.

I ° # I n hen # 2 nt
  • ¥n
man , } .

=

slide-22
SLIDE 22

Example: Birthday Problem

There are k people in a room. Assume each person’s birthday is equally likely to be any of the 365 days of the year (we exclude February 29), and that people’s birthdays are independent (we assume there are no twins in the room). What is the probability that two or more people in the group have the same birthday? Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 22 / 47
  • .
slide-23
SLIDE 23

Solution of Birthday Problem

Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 23 / 47 O k > 365 10=1
  • 2
k£365 . A = " 32 have the same birthday ' ' AC = " no two People
  • '
' . " i k ( without match ) PLA ) = tputc ) = (

i"ii=

# ofugs assign birthdays to k . . ( 365 , k , without replacer . ) = 1 .

=

@ 65 . k , replacement ) 365.364 . . .
  • ( 365
  • KTH
= 1-
  • 365£
slide-24
SLIDE 24

Solution of Birthday Problem

Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 24 / 47 l : I : 1 I 23
slide-25
SLIDE 25

Generalized Birthday Problem

Each of k people has a random number (“birthday”) drawn from n values (“days”). If the probability that at least two people have the same number is 50%, then k ⇡ 1.18pn. Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 25 / 47 A = " 32 . ... man " .

&@ 2@

. . .
  • i.
. vithon . NCN
  • i )
. . .cn
  • ktl )
PLA ) = 1-
  • =p
  • sample
nvtmveplucem MR Approximations ) 1 a- In .tn . . . Int '

p=x

  • mm
= tunhtas . . .ch#n)EeIxEL , ]

.

Fit etn.EE . . . EE ' = f- e- d- CHZT . .tk . i ) PCA 1=0.5 be =L ' e
  • £i#
a

,.e_a@

  • easy
⇒ k=Fn . Fini al .i8Jn knOcJ#
slide-26
SLIDE 26

Application: Hash Table

A commonly used data structure for fast information retrieval Example: store people’s name. For each people x, a hash function h is computed. h(x): the location that will be used to store x0s name. Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 26 / 47
  • ÷
  • ,
" . ~
slide-27
SLIDE 27

Hash Collision

Collision: x 6= y, but h(x) = h(y) ( 1 locations has 2 names stored there) Given k people (different names) and n locations, what is the probability of occurrence of hash collision? Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 27 / 47

÷

slide-28
SLIDE 28

Solution of Hash Collision

Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 28 / 47
slide-29
SLIDE 29

Binomial Coefficient

Definition For any nonnegative integers k and n, the binomial coefficient n k
  • ,
read as “n choose k”, is the number of subsets of size k for a set of size n. Theorem For k  n, we have ✓n k ◆ = n(n 1) · · · (n k + 1) k! = n! k!(n k)! Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 29 / 47

C!

no

ncn-D.gg?h#

slide-30
SLIDE 30

Example: Bose-Einstein

How many ways are there to choose k times from a set of n objects with replacement, if order doesn’t matter (we only care about how many times each object was chosen, not the order in which they were chosen)? Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 30 / 47

1,2€

Xj : jthobgaes C # . chosen ) ( ¥ '

}n°)

Xitxzt . . . + Xn =k
  • nonnegative

'nteg÷

  • =L
slide-31
SLIDE 31

Equivalent Problem

Theorem There are r1 n1
  • distinct positive integer-valued vectors
(x1, x2, . . . , xn) satisfying the equation x1 + x2 + · · · + xn = r, xi > 0, i = 1, 2, . . . , n. Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 31 / 47 r
  • t##
  • 6
'

!

. : . 1 ' .0 . . . . O a a a . f- ( spaces . =
  • mum
' i. ° ! ° ! ° ° . < a / 2 1 2 2
slide-32
SLIDE 32

Bose-Einstein Counting

Theorem There are n+k1 n1
  • distinct nonnegative integer-valued vectors
(x1, x2, . . . , xn) satisfying the equation x1 + x2 + · · · + xn = k, xi 0, i = 1, 2, . . . , n. Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 32 / 47
  • XI

T.ci#Ye31&Y,-yz+..iYn=kt=

positive integers

and

slide-33
SLIDE 33

Example: Multinomial Expansion

How many terms are there in the multinomial expansion of (x1 + x2 + . . . + xr)n? Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 33 / 47 ( Xntxn )2 =×±?tziE¥i+xI . a For each item , ( Xi

"xin?..xT)

  • *
. Ng 30 C j=l , .ir )
  • Nitnzt
. . . thran @ coeftia .

cnn.jcnnmj.i.FI#=N!n

! ni !n2 ! ... nh 3 ( Xitxzt . .tXn , " = £ T.hr?Xin'Xzn2...xrnrni.nL...nrZ0n,+nrttnr=n # of <⇒ ni-nu.mn#nonhegatueintegensow+._

("F#.

slide-34
SLIDE 34

Summary of Counting

Choose k objects out of n objects, the number of possible ways: Order Matters Order Not Matter with replacement nk n+k1 k
  • without replacement
n(n 1) · · · (n k + 1) n k
  • Ziyu Shao (ShanghaiTech)
Lecture 1: Probability and Counting September 21, 2018 34 / 47

TO

08

slide-35
SLIDE 35

Outline

1 Good Books for Linear Algebra 2 Naive Definition of Probability 3 Counting 4 Definition of Probability Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 35 / 47
slide-36
SLIDE 36

General Definition of Probability

Definition A probability space consists of a sample space S and a probability function P which takes an event A ✓ S as input and returns P(A), a real number between 0 and 1, as output. The function P must satisfy the following axioms: 1 P(;) = 0, P(S) = 1. 2 If A1, A2, . . . are disjoint events, then P( 1 [ j=1 Aj) = 1 X j=1 P(Aj) (Saying that these events are disjoint means that they are mutually exclusive: Ai \ Aj = ; for i 6= j. Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 36 / 47 =
  • n

=

=

  • Countableinfiniter
slide-37
SLIDE 37

Interpretation of Probability

Complementary views: The frequentist view: probability represents a long-run frequency
  • ver a large number of repetitions of an experiment.
I if we say a coin has probability 1/2 of Heads, that means the coin would land Heads 50% of the time if we tossed it over and
  • ver and over.
The Bayesian view: probability represents a degree of belief about the event in question. I So we can assign probabilities to hypotheses like “candidate A will win the election” or “the defendant is guilty” even if it isn’t possible to repeat the same election or the same crime over and
  • ver again.
Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 37 / 47

÷

slide-38
SLIDE 38

Properties of Probability

Probability has the following properties, for any events A and B: 1 P(Ac) = 1 P(A). 2 If A ✓ B, then P(A)  P(B). 3 P(A S B) = P(A) + P(B) P(A \ B). Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 38 / 47 EO . O PCEUAI ) =p(s)=| 2

€0

B=A0CBnA= PLBKPCA ) t PCBNAC ) PCAHPLAC ) =l⇐ ZPLA )
  • ⇐
  • spotting

€,*@!o

" a " "

IEia5YYYI¥→

, ✓ p(BnA9=PcB1-PCAnB=
slide-39
SLIDE 39

Example: Bonferroni’s Inequality

Theorem For any n events A1, . . . , An, we have P(A1 \ A2 \ · · · \ An) P(A1) + P(A2) + · · · + P(An) (n 1). Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 39 / 47 Lt 's aus F- € w n 't i

pts

path ✓ 2 n=2 PCAinAz)>PLAittPlA= ✓ L⇒ 1 3 P(Ai)tPLA2)-PLAinA= PCA.UA#

:

Tamai ) =p( Fi 0¥ . . .uEn )

÷

tend

¥n¥¥

.it?ItEiIIII.n.J

= n
  • [
PCAD
  • 1
. . . + Plan >)

↳

slide-40
SLIDE 40

Example: Boole’s Inequality

Theorem For any events A1, A2, . . ., we have P( 1 [ i=1 Ai)  1 X i=1 P(Ai). Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 40 / 47

¥ AL

⇒ Sequences of disjoint Sets . Al U As = A , U ( Az n AT ) ÷ .

And

U Az = Aid U ( Az n AIID = Al U ( Az n AT ) U ( A 3 n

At

A i U.A.su . . . U An = A , U ( Azn AT ) U . . . . (1 CAN NATE . , )

died

. . . . p ( Ia ) = P ( I AIEA) = ftp.#InTadIDeTIPas .
slide-41
SLIDE 41

Inclusion-Exclusion Formula

For any events A1, . . . , An: P( n [ i=1 Ai) = X i P(Ai) X i<j P(Ai \ Aj) + X i<j<k P(Ai \ Aj \ Ak) + . . . + (1)n+1P(A1 \ · · · \ An). Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 41 / 47
  • =

it

'
slide-42
SLIDE 42

Example: De Montmort’s Matching Problem

Consider a well-shuffled deck of n cards, labeled 1 through n. You flip
  • ver the cards one by one, saying the numbers 1 through n as you do
  • so. You win the game if, at some point, the number you say aloud is
the same as the number on the card being flipped over (for example, if the 7th card in the deck has the label 7). What is the probability of winning? Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 42 / 47

=

slide-43
SLIDE 43

Solution to De Montmort’s Matching Problem

Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 43 / 47 A ; = " c # Card in the deck has label : " .

=p

( A,UAz..vAn)_ Winton E*=t×+z¥ ...

.ee/,nnIPlAi)=+nkc=.#b=tn

]

+ " '

Yana

,

,=÷e÷⇒

a

unless

.IS?iiIlIYPlAinAInAk)=#h=ai

, , = 1- et :
  • bitten 'Az
. . . nah )

=n!E¥InI÷EEI¥I÷¥I÷I¥y÷Ii::I¥ta÷*TDI

slide-44
SLIDE 44

Summary 1: Events & Numbers

Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 44 / 47

00

slide-45
SLIDE 45

Summary 2: Probability Space

Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 45 / 47
  • 8 ?
slide-46
SLIDE 46

Summary 3: The Role of Probability & Statistics

A framework for analyzing phenomena with uncertain outcomes: Rules for consistent reasoning Used for predictions and decisions Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 46 / 47

⇐

slide-47
SLIDE 47

Reading Assignment

Next lecture we will study conditional probability. Please read corresponding chapters in textbooks: Chapter 2 of BH Chapter 1 of BT Ziyu Shao (ShanghaiTech) Lecture 1: Probability and Counting September 21, 2018 47 / 47

KG

.

g-