Probability*and*Statistics* ! for*Computer*Science** - - PowerPoint PPT Presentation
Probability*and*Statistics* ! for*Computer*Science** "Sta&s&cal!thinking!will!one!day! be!as!necessary!for!efficient! ci&zenship!as!the!ability!to!read! and!write."!H.!G.!Wells ! Credit:!wikipedia!
!
Probability*and*Statistics* for*Computer*Science**
"Sta&s&cal!thinking!will!one!day! be!as!necessary!for!efficient! ci&zenship!as!the!ability!to!read! and!write."!H.!G.!Wells!
Hongye!Liu,!Teaching!Assistant!Prof,!CS361,!UIUC,!03.12.2020! Credit:!wikipedia!
Midterm1**
* Grading
is
done
.*
Grades
will
be published
today
.*
Points
will
be
curved
given
it 's
relatively harder
than last
semester
.*
You 're
welcome
to
come
to
hour
today
to discuss
about
it .
MidTerm1
Score Frequency 60 80 100 120 140 10 20 30stole
112.849 - r
' ' 5 '9.724last
semester
Std
median
= 124mean
=L 12.956 Last*time*
Review!of!variance,!sample!mean! Sum!and!difference!between!
variables!of!normal!distribu&ons!
Hypothesis!test!of!equality!of!two!
sample!means!
ChiRsquare!test!
Contents*
Review!of!sta&s&cal!inference! Inferring!probability!model!from!
data!
Maximum!likelihood!es&mate! Confidence!interval!for!MLE! Bayesian!inference!
1
Categories*of*Statistical* inference**
Sta&s&cal!inference!includes!
Drawing!conclusion!from!samples! Assessing!the!significance!of!evidence!
for!a!hypothesis!
Inferring!the!parameters!of!
probabilis&c!model!from!data!
Contents*
Review!of!sta&s&cal!inference! Inferring(probability(model(from(
data(
Maximum!likelihood!es&mate! Confidence!interval!for!MLE! Bayesian!inference!
Motivation:*binomial*example*
Suppose!we!have!a!coin!with!unknown!
probability!of!coming!up!heads!
We!toss!it!N!&mes!and!observe!k!heads! We!know!that!this!data!comes!from!a!
binomial!distribu&on!
What!is!your!best!es&mate!of!the!probability!
Credit:!David!Varodayan!
pix-ks-fkjpkci.PT
"
µ= Is
pot getting
head ?
15-5
Motivation:*geometric*example*
Suppose!we!have!a!die!with!unknown!
probability!of!coming!up!six!
We!roll!it!and!it!comes!up!six!for!the!first!
&me!on!the!kth!roll!
We!know!that!this!data!comes!from!a!
geometric!distribu&on!
What!is!your!best!es&mate!of!the!probability!
Credit:!David!Varodayan!
Motivation:*Poisson*example*
Suppose!we!have!data!on!the!number!of!babies!
born!each!hour!in!a!large!hospital!
!We!can!assume!the!data!comes!from!a!Poisson!
distribu&on!
What!is!your!best!es&mate!of!the!intensity!λ?!
Credit:!David!Varodayan!
hour(
1( 2(
…!
N"
#!of!babies! k1# k2#
…!
kN#
!
known
④
T
T
T
K ,
kN
Atm known
The*parameter*estimation*problem*
Suppose!we!have!a!dataset!that!we!know!comes!from!
a!distribu&on!(ie.!Binomial,!Geometric,!or!Poisson,!etc.)!
What!is!the!best!es&mate!of!the!parameters!(θ!or!θs)!
Examples:!
For!binomial!and!geometric!distribu&on,!θ(=!p!(probability!of!
success)!
For!Poisson!and!exponen&al!distribu&ons,!θ(=!λ!(intensity)! For!normal!distribu&ons,!θ(could!be!μ!or!σ2.#
mu
n
n
Maximum*likelihood*estimation*(MLE)*
We!write!the!probability!of!seeing!the!data!D!
given!parameter!θ!!
The!likelihood(func,on!!!!!!!!!!is!not!a!
probability!distribu&on!
The!maximum(likelihood(es,mate((MLE)!of!
θ!is!! !
L(θ) = P(D|θ)
L(θ)
ˆ θ = arg max
θ
L(θ)
mm
'
up!
"
P H T
Ky
Pata
Yammerer
Why*is*L(θ)*not*a*probability*distribution?*
A.!!It!doesn’t!give!the!probability!of!all!the! possible!θ!values.!! B.!Don’t!know!whether!the!sum!or!integral!of!!!!!!!!!!! for!all!possible!θ!values!is!one!or!not.!! C.!Both.! L(θ)
40)
I
O is
not
a
random variable
Why*is*L(θ)*not*a*probability*distribution?*
A.!!It!doesn’t!give!the!probability!of!all!the! possible!θ!values.!! B.!Don’t!know!whether!the!sum!or!integral!of!!!!!!!!!!! for!all!possible!θ!values!is!one!or!not.!! C.!Both.! L(θ)
Likelihood*function:*binomial*example*
Suppose!we!have!a!coin!with!unknown!
probability!of!coming!up!heads!
We!toss!it!N!&mes!and!observe!k!heads! We!know!that!this!data!comes!from!a!binomial!
distribu&on!
What!is!the!likelihood!func&on!!!!!!!!!!!!!!!!!!!!!!!!!!!?!
!
L(θ) = P(D|θ)
Likelihood*function:*binomial*example*
Suppose!we!have!a!coin!with!unknown!
probability!of!coming!up!heads!
We!toss!it!N!&mes!and!observe!k!heads! We!know!that!this!data!comes!from!a!binomial!
distribu&on!
What!is!the!likelihood!func&on!!!!!!!!!!!!!!!!!!!!!!!!!!!?!
!
L(θ) = P(D|θ)
L(θ) = N k
replace
P
with
MLE*derivation:*binomial*example*
L(θ) = N k
ˆ θ = arg max
θ
L(θ)
In!order!to!find:! ! We!set:!!
dL(θ) dθ = 0
.
MLE*derivation:*binomial*example*
L(θ) = N k
MLE*derivation:*binomial*example*
L(θ) = N k
d dθL(θ) = N k
(c
. fi . tr ) " MLE*derivation:*binomial*example*
L(θ) = N k
d dθL(θ) = N k
kθk−1(1 − θ)N−k = θk(N − k)(1 − θ)N−k−1
MLE*derivation:*binomial*example*
L(θ) = N k
d dθL(θ) = N k
kθk−1(1 − θ)N−k = θk(N − k)(1 − θ)N−k−1 k − kθ = Nθ − kθ
MLE*derivation:*binomial*example*
L(θ) = N k
d dθL(θ) = N k
kθk−1(1 − θ)N−k = θk(N − k)(1 − θ)N−k−1 k − kθ = Nθ − kθ
ˆ θ = k N
The(MLE(of(p(
maximized at E
O
ECKf- NP
Likelihood*function:*geometric*example*
Suppose!we!have!a!die!with!unknown!probability!
We!roll!it!and!it!comes!up!six!for!the!first!&me!on!
the!kth!roll!
We!know!that!this!data!comes!from!a!geometric!
distribu&on!
What!is!the!likelihood!func&on!!!!!!!!!!!!!!!!!!!!!!!!!!!?!
Assume(θ(is(p.! !
L(θ) = P(D|θ)
MLE*derivation:*geometric*example*
L(θ) = (1 − θ)k−1θ
pot
MLE*derivation:*geometric*example*
L(θ) = (1 − θ)k−1θ
d dθL(θ) = (1 − θ)k−1 − (k − 1)(1 − θ)k−2θ = 0
f- e f u
fit.
Titi
¥
MLE*derivation:*geometric*example*
L(θ) = (1 − θ)k−1θ
(1 − θ)k−1 = (k − 1)(1 − θ)k−2θ
d dθL(θ) = (1 − θ)k−1 − (k − 1)(1 − θ)k−2θ = 0
I - O
= ( K - 1) O
p at sett's
l
head is
i
⇒
'
='T
MLE*derivation:*geometric*example*
L(θ) = (1 − θ)k−1θ
(1 − θ)k−1 = (k − 1)(1 − θ)k−2θ
d dθL(θ) = (1 − θ)k−1 − (k − 1)(1 − θ)k−2θ = 0
1 − θ = kθ − θ
MLE*derivation:*geometric*example*
L(θ) = (1 − θ)k−1θ
(1 − θ)k−1 = (k − 1)(1 − θ)k−2θ
d dθL(θ) = (1 − θ)k−1 − (k − 1)(1 − θ)k−2θ = 0
1 − θ = kθ − θ
ˆ θ = 1 k
The(MLE(of(p(
Geometric
dis r - .
Eckl = ¥
=I
6
MLE*with*data*from*IID*trials*
If!the!dataset!!!!!!!!!!!!!!!!!!comes!from!IID!trials! Each!xi!!is!one!observed!result!from!an!IID!trial!
D = {x}
L(θ) = P(D|θ) =
P(xi|θ)
"
""
Xi C- D
Q:*MLE*with*data*from*IID*trials*
If!the!dataset!!!!!!!!!!!!!!!!!!comes!from!IID!trials! Why!is!the!above!func&on!defined!by!the!product?!
!A.!IID!samples!are!independent! !B.!Each!trial!has!iden&cal!probability!func&on! !C.!Both.!
D = {x}
L(θ) = P(D|θ) =
P(xi|θ)
Xi GD
Q:*MLE*with*data*from*IID*trials*
If!the!dataset!!!!!!!!!!!!!!!!!!comes!from!IID!trials! Why!is!the!above!func&on!defined!by!the!product?!
!A.!IID!samples!are!independent! !B.!Each!trial!has!iden&cal!probability!func&on! !C.!Both.!
D = {x}
L(θ) = P(D|θ) =
P(xi|θ)
MLE*with*data*from*IID*trials*
If!the!dataset!!!!!!!!!!!!!!!!!!comes!from!IID!trials! The!likelihood!func&on!is!hard!to!differen&ate!in!
general,!except!for!the!binomial!and!geometric! cases.!
Clever!trick:!take!the!(natural)!log!
D = {x}
L(θ) = P(D|θ) =
P(xi|θ)
LogJlikelihood*function*
Since!log!is!a!strictly!increasing!func&on!
!
So!we!can!aim!to!maximize!the!logClikelihood(
func,on(
The!logRlikelihood!func&on!is!usually!much!easier!
to!differen&ate!
ˆ θ = arg max
θ
L(θ) = arg max
θ
logL(θ)
logL(θ) = logP(D|θ) = log
P(xi|θ) =
logP(xi|θ)
LogJlikelihood*function:*Poisson*example*
Suppose!we!have!data!on!the!number!of!babies!
born!each!hour!in!a!large!hospital!
!We!can!assume!the!data!comes!from!a!Poisson!
distribu&on!λ!
What!is!the!log!likelihood!func&on!!!!!!!!!!!!!!!!!!!!!?!
hour(
1( 2(
…!
N"
#!of!babies! k1# k2#
…!
kN#
!
LogL(θ)
LogJlikelihood*function:*Poisson*example*
L(θ) =
N
e−θθki ki!
log L(θ) = log (
N
e−θθki ki! ) =
N
log(e−θθki ki! )
=
N
(−θ + ki logθ − log ki!)
PC kilos-E!?
. O
two
MLE*:*Poisson*example*
LogL(θ) =
N
(−θ + ki logθ − log ki!)
"
= II. title
I
= II,
c- It
In
=
E-I
6
MLE*:*Poisson*example*
d dθlog L(θ) = 0 ⇒
N
(−1 + ki θ − 0) = 0 LogL(θ) =
N
(−θ + ki logθ − log ki!)
MLE*:*Poisson*example*
d dθlog L(θ) = 0 ⇒
N
(−1 + ki θ − 0) = 0 −N + N
i ki
θ = 0 LogL(θ) =
N
(−θ + ki logθ − log ki!)
"
T ! Ki
D= -
d
N
for a
Poisson
Poisson
B.
'mo
( ( O)
= IT 401 Luca) ↳ co)
'
. iinoictoi"
. MLE*:*Poisson*example*
d dθlog L(θ) = 0 ⇒
N
(−1 + ki θ − 0) = 0 −N + N
i ki
θ = 0
ˆ θ = N
i ki
N
The(MLE(of(λ(
LogL(θ) =
N
(−θ + ki logθ − log ki!)
MLE*for*normal*distribution*
Suppose!we!model!the!dataset!!!!!!!!!!!!!!!!!!as!
normally!distributed!!
What!should!be!the!likelihood!func&on?!Is!the!
method!of!modeling!the!same!as!for!the!Poisson! distribu&on?! !A.!!!Yes!!!!!!!!B.!No!
D = {x}
MLE*for*normal*distribution*
Suppose!we!model!the!dataset!!!!!!!!!!!!!!!!!!as!
normally!distributed!!
What!should!be!the!likelihood!func&on?!Is!the!
method!of!modeling!the!same!as!for!the!Poisson! distribu&on?!Yes(and(No.!The!idea!is!similar!but! the!normal!distribu&on!is!con&nuous,!we!need!to! use!the!probability(density!instead.!
D = {x}
N
¥, R
→ II, P
p→density
fun at.by
MLE*for*normal*distribution*
Suppose!we!model!the!dataset!!!!!!!!!!!!!!!!!!as!
normally!distributed!!
The!likelihood!func&on!of!a!normal!distribu&on:!
D = {x}
L(µ, σ) =
n
1 √ 2πσexp(−(xi − µ)2 2σ2 )
*
x
n
p
I'
U
G
p -
anax!mish
L (µ ,6)
MLE*for*normal*distribution*
Suppose!we!model!the!dataset!!!!!!!!!!!!!!!!!!as!
normally!distributed!!
There!are!two!parameters!to!es&mate:!μ!and!σ#
If!we!fix!σ(and!set!θ=!μ#
""
If!we!fix!μ#and!set!θ=!σ!
ˆ θ = 1 N
N
xi ˆ θ =
N
N
(xi − µ)2
D = {x}
"
i
Efx]
A
DX
= µ
( (M , 5 )
d KCMO) ⇒
fco , is -_ o
fix G
→-
i
dm
tix n → dD⇒ { face,nI=o d5
in
E
correction
:in
ele
case of
fixing
0 ,
using log Likelihood function fi
doesn't depend
6
.in the
case of fixing u ,
we
assume
we
know u
using Ei
asestimation of u
✓ Drawbacks*of*MLE*
Maximizing!some!likelihood!or!logRlikelihood!
func&on!is!mathema&cally!hard!
!If!there!are!very!few!data!items,!the!MLE!
es&mate!maybe!very!unreliable!
If!we!observe!3!heads!in!10!coin!tosses,!should!we!
accept!that!p(heads)=!0.3!?!
If!we!observe!0!heads!in!2!coin!tosses,!should!we!
accept!that!p(heads)=!0!?!
Confidence*intervals*for*MLE*estimates*
An!MLE!parameter!es&mate!!!!!!depends!on!the!
data!that!was!observed!
We!can!construct!a!confidence!interval!for!!!!!using!
the!parametric!bootstrap!
Use!the!distribu&on!with!parameter!!!!!!!to!generate!
a!large!number!of!bootstrap!samples!
From!each!“synthe&c”!dataset,!reRes&mate!the!
parameter!using!MLE!
Use!the!histogram!of!these!reRes&mates!to!
construct!a!confidence!interval!
ˆ θ ˆ θ ˆ θ
Bayesian*inference*
In!MLE,!we!maximized!the!likelihood!func&on!! In!Bayesian!inference,!we!will!maximize!the!posterior,!
which!is!the!probability!of!the!parameters!θ!given!the!
Unlike!!!!!!!!!,!the!posterior!is!a!probability!distribu&on! The!value!of!θ!that!maximizes!!!!!!!!!!!!!!!!is!called!the!
maximum(a(posterior((MAP)!es&mate!!
L(θ) = P(D|θ)
ˆ θ
P(θ|D)
P(θ|D)
L(θ)
The*prior*
From!Bayes!rule!
P(θ|D) = P(D|θ)P(θ) P(D) ∝ P(D|θ)P(θ)
The*prior*
From!Bayes!rule!
P(θ|D) = P(D|θ)P(θ) P(D) ∝ P(D|θ)P(θ)
The*prior*
From!Bayes!rule! The!probability!of!the!data!P(D)!is!a!constant,!which!
doesn’t!maher!for!differen&a&on.!
Bayesian!inference!allows!us!to!include!prior!beliefs!
about!θ!in!the!prior!!!!!!!!!!,!which!is!useful!
When!we!have!reasonable!beliefs,!such!as!a!coin!can!not!
have!P(heads)!=!0!
When!there!isn’t!much!data!
P(θ)
P(θ|D) = P(D|θ)P(θ) P(D) ∝ P(D|θ)P(θ)
Q.*Why*is*the*posterior*a*probability*
distribution?**
!A.!!It!is!defined!so!that!it!gives!the!probability! !of!θ!for!all!possibility!condi&oned!on!the!data! !B.!The!sum!or!integral!of!such!condi&onal! !probability!should!be!1! !C.!Both!
Q.*Why*is*the*posterior*a*probability*
distribution?**
!A.!!It!is!defined!so!that!it!gives!the!probability! !of!θ!for!all!possibility!condi&oned!on!the!data! !B.!The!sum!or!integral!of!such!condi&onal! !probability!should!be!1! !C.!Both!
Assignments*
Read!Chapter!9!of!the!textbook! Next!&me:!Bayesian!inference!
!
Additional*References*
✺ Robert!V.!Hogg,!Elliot!A.!Tanis!and!Dale!L.!
Zimmerman.!“Probability!and!Sta&s&cal! Inference”!!
Morris!H.!Degroot!and!Mark!J.!Schervish!
"Probability!and!Sta&s&cs”!
See*you*next*time*
See You!