CSE 312 Final Review: Section AA CSE 312 TAs December 8, 2011 CSE - - PowerPoint PPT Presentation

CSE 312 Final Review: Section AA

CSE 312 TAs December 8, 2011

CSE 312 Final Review: Section AA

General Information

CSE 312 Final Review: Section AA

General Information

Comprehensive Midterm

CSE 312 Final Review: Section AA

General Information

Comprehensive Midterm Heavily weighted toward material after the midterm

CSE 312 Final Review: Section AA

Pre-Midterm Material

CSE 312 Final Review: Section AA

Pre-Midterm Material

Basic Counting Principles

CSE 312 Final Review: Section AA

Pre-Midterm Material

Basic Counting Principles

Pigeonhole Principle

CSE 312 Final Review: Section AA

Pre-Midterm Material

Basic Counting Principles

Pigeonhole Principle Inclusion Exclusion

CSE 312 Final Review: Section AA

Pre-Midterm Material

Basic Counting Principles

Pigeonhole Principle Inclusion Exclusion Counting the Complement

CSE 312 Final Review: Section AA

Pre-Midterm Material

Basic Counting Principles

Pigeonhole Principle Inclusion Exclusion Counting the Complement Using symmetry

CSE 312 Final Review: Section AA

Pre-Midterm Material

Basic Counting Principles

Pigeonhole Principle Inclusion Exclusion Counting the Complement Using symmetry

Conditional Probability

CSE 312 Final Review: Section AA

Pre-Midterm Material

Basic Counting Principles

Pigeonhole Principle Inclusion Exclusion Counting the Complement Using symmetry

Conditional Probability

P(A | B) = P(AB)

P(B)

CSE 312 Final Review: Section AA

Pre-Midterm Material

Basic Counting Principles

Pigeonhole Principle Inclusion Exclusion Counting the Complement Using symmetry

Conditional Probability

P(A | B) = P(AB)

P(B)

Law of Total Probability: P(A) = P(A | B) · P(B) + P(A |B) · P(B)

CSE 312 Final Review: Section AA

Pre-Midterm Material

Basic Counting Principles

Pigeonhole Principle Inclusion Exclusion Counting the Complement Using symmetry

Conditional Probability

P(A | B) = P(AB)

P(B)

Law of Total Probability: P(A) = P(A | B) · P(B) + P(A |B) · P(B) Bayes’ Theorem: P(A | B) = P(B | A) · P(A) P(B)

CSE 312 Final Review: Section AA

Pre-Midterm Material

Basic Counting Principles

Pigeonhole Principle Inclusion Exclusion Counting the Complement Using symmetry

Conditional Probability

P(A | B) = P(AB)

P(B)

Law of Total Probability: P(A) = P(A | B) · P(B) + P(A |B) · P(B) Bayes’ Theorem: P(A | B) = P(B | A) · P(A) P(B) Network Failure Questions

CSE 312 Final Review: Section AA

Pre-Midterm Material

CSE 312 Final Review: Section AA

Pre-Midterm Material

Independence

CSE 312 Final Review: Section AA

Pre-Midterm Material

Independence

E and F are independent if P(EF) = P(E)P(F)

CSE 312 Final Review: Section AA

Pre-Midterm Material

Independence

E and F are independent if P(EF) = P(E)P(F) E and F are independent if P(E | F) = P(E) and P(F | E) = P(F)

CSE 312 Final Review: Section AA

Pre-Midterm Material

Independence

E and F are independent if P(EF) = P(E)P(F) E and F are independent if P(E | F) = P(E) and P(F | E) = P(F) Events E1, . . . En are independent if for every subset S of events P

• i∈S

Ei

• =
• i∈S

P(Ei)

CSE 312 Final Review: Section AA

Pre-Midterm Material

Independence

E and F are independent if P(EF) = P(E)P(F) E and F are independent if P(E | F) = P(E) and P(F | E) = P(F) Events E1, . . . En are independent if for every subset S of events P

• i∈S

Ei

• =
• i∈S

P(Ei) Biased coin example from Lecture 5, slide 7

CSE 312 Final Review: Section AA

Pre-Midterm Material

Independence

E and F are independent if P(EF) = P(E)P(F) E and F are independent if P(E | F) = P(E) and P(F | E) = P(F) Events E1, . . . En are independent if for every subset S of events P

• i∈S

Ei

• =
• i∈S

P(Ei) Biased coin example from Lecture 5, slide 7 If E and F are independent and G is an arbitrary event then in general P(EF | G) = P(E | G) · P(F | G)

CSE 312 Final Review: Section AA

Pre-Midterm Material

Independence

E and F are independent if P(EF) = P(E)P(F) E and F are independent if P(E | F) = P(E) and P(F | E) = P(F) Events E1, . . . En are independent if for every subset S of events P

• i∈S

Ei

• =
• i∈S

P(Ei) Biased coin example from Lecture 5, slide 7 If E and F are independent and G is an arbitrary event then in general P(EF | G) = P(E | G) · P(F | G) For any given G, equality in the above statement means that E and F are Conditionally Independent given G

CSE 312 Final Review: Section AA

Distributions

CSE 312 Final Review: Section AA

Distributions

Know the mean and variance for:

CSE 312 Final Review: Section AA

Distributions

Know the mean and variance for:

Uniform distribution

CSE 312 Final Review: Section AA

Distributions

Know the mean and variance for:

Uniform distribution Normal distribution

CSE 312 Final Review: Section AA

Distributions

Know the mean and variance for:

Uniform distribution Normal distribution Geometric distribution

CSE 312 Final Review: Section AA

Distributions

Know the mean and variance for:

Uniform distribution Normal distribution Geometric distribution Binomial distribution

CSE 312 Final Review: Section AA

Distributions

Know the mean and variance for:

Uniform distribution Normal distribution Geometric distribution Binomial distribution Poisson distribution

CSE 312 Final Review: Section AA

Distributions

Know the mean and variance for:

Uniform distribution Normal distribution Geometric distribution Binomial distribution Poisson distribution Hypergeometric distribution

CSE 312 Final Review: Section AA

Distributions

Know the mean and variance for:

Uniform distribution Normal distribution Geometric distribution Binomial distribution Poisson distribution Hypergeometric distribution

Remember Linearity of Expectation and other useful facts (e.g. Var[aX + b] = a2Var[X]; in general Var[X + Y ] = Var[X] + Var[Y ]).

CSE 312 Final Review: Section AA

Distributions

Know the mean and variance for:

Uniform distribution Normal distribution Geometric distribution Binomial distribution Poisson distribution Hypergeometric distribution

Remember Linearity of Expectation and other useful facts (e.g. Var[aX + b] = a2Var[X]; in general Var[X + Y ] = Var[X] + Var[Y ]). Remember: For any a, P(X = a) = 0 (the probability that a continuous R.V. falls at a specific point is 0!)

CSE 312 Final Review: Section AA

Distributions

Know the mean and variance for:

Uniform distribution Normal distribution Geometric distribution Binomial distribution Poisson distribution Hypergeometric distribution

Remember Linearity of Expectation and other useful facts (e.g. Var[aX + b] = a2Var[X]; in general Var[X + Y ] = Var[X] + Var[Y ]). Remember: For any a, P(X = a) = 0 (the probability that a continuous R.V. falls at a specific point is 0!) Expectation is now an integral: E[X] = ∞

−∞ x · f (x)dx

CSE 312 Final Review: Section AA

Distributions

Know the mean and variance for:

Uniform distribution Normal distribution Geometric distribution Binomial distribution Poisson distribution Hypergeometric distribution

Remember Linearity of Expectation and other useful facts (e.g. Var[aX + b] = a2Var[X]; in general Var[X + Y ] = Var[X] + Var[Y ]). Remember: For any a, P(X = a) = 0 (the probability that a continuous R.V. falls at a specific point is 0!) Expectation is now an integral: E[X] = ∞

−∞ x · f (x)dx

Use normal approximation when applicable.

CSE 312 Final Review: Section AA

Central Limit Theorem

CSE 312 Final Review: Section AA

Central Limit Theorem

Central Limit Theorem: Consider i.i.d. (independent, identically distributed) random variables X1, X2, . . . . Xi has µ = E[Xi] and σ2 = Var[Xi]. Then, as n → ∞ X1 + · · · + Xn − nµ σ√n → N(0, 1) Alternatively X = 1 n

n

• i=1

Xi ≈ N

• µ, σ2

n

• CSE 312 Final Review: Section AA

Tail Bounds

CSE 312 Final Review: Section AA

Tail Bounds

Markov’s Inequality: If X is a non-negative random variable, then for every α > 0, we have P(X ≥ α) ≤ E[X] α

CSE 312 Final Review: Section AA

Tail Bounds

Markov’s Inequality: If X is a non-negative random variable, then for every α > 0, we have P(X ≥ α) ≤ E[X] α Corollary P(X ≥ αE[X]) ≤ 1 α

CSE 312 Final Review: Section AA

Tail Bounds

Markov’s Inequality: If X is a non-negative random variable, then for every α > 0, we have P(X ≥ α) ≤ E[X] α Corollary P(X ≥ αE[X]) ≤ 1 α Chebyshev’s Inequality: If Y is an arbitrary random variable with E[Y ] = µ, then, for any α > 0, P(|Y − µ| ≥ α) ≤ Var[Y ] α2

CSE 312 Final Review: Section AA

Tail Bounds

CSE 312 Final Review: Section AA

Tail Bounds

Chernoff Bounds: Suppose X is drawn from Bin(n, p) and µ = E[X] = pn Then, for any 0 < δ < 1 P(X > (1 + δ)µ) ≤ e− δ2µ

2

P(X < (1 − δ)µ) ≤ e− δ2µ

3 CSE 312 Final Review: Section AA

Law of Large Numbers

CSE 312 Final Review: Section AA

Law of Large Numbers

Weak Law of Large Numbers: Let X be the empirical mean

• f i.i.d.s X1, . . . , Xn. For any ǫ > 0, as n → ∞

Pr

• X − µ
• > ǫ
• → 0

CSE 312 Final Review: Section AA

Law of Large Numbers

Weak Law of Large Numbers: Let X be the empirical mean

• f i.i.d.s X1, . . . , Xn. For any ǫ > 0, as n → ∞

Pr

• X − µ
• > ǫ
• → 0

Strong Law of Large Numbers: Same hypotheses Pr

• lim

n→∞

X1 + · · · + Xn n = µ

• = 1

CSE 312 Final Review: Section AA

Random Facts

CSE 312 Final Review: Section AA

Random Facts

If X and Y are R.V.s from the same distribution, then Z = X + Y isn’t necessarily from the same distribution as X and Y .

CSE 312 Final Review: Section AA

Random Facts

If X and Y are R.V.s from the same distribution, then Z = X + Y isn’t necessarily from the same distribution as X and Y . If X and Y are both normal, then so is X + Y .

CSE 312 Final Review: Section AA

MLEs

CSE 312 Final Review: Section AA

MLEs

Write an expression for the likelihood.

CSE 312 Final Review: Section AA

MLEs

Write an expression for the likelihood. Convert this into the log likelihood.

CSE 312 Final Review: Section AA

MLEs

Write an expression for the likelihood. Convert this into the log likelihood. Take derivatives to find the maximum.

CSE 312 Final Review: Section AA

MLEs

Write an expression for the likelihood. Convert this into the log likelihood. Take derivatives to find the maximum. Verify that this is indeed a maximum.

CSE 312 Final Review: Section AA

MLEs

Write an expression for the likelihood. Convert this into the log likelihood. Take derivatives to find the maximum. Verify that this is indeed a maximum. See Lecture 11 for worked examples.

CSE 312 Final Review: Section AA

Expectation-Maximization

CSE 312 Final Review: Section AA

Expectation-Maximization

E-step: Computes the log-likelihood using current parameters.

CSE 312 Final Review: Section AA

Expectation-Maximization

E-step: Computes the log-likelihood using current parameters. M-step: Maximize the expected log-likelihood, changing the parameters.

CSE 312 Final Review: Section AA

Expectation-Maximization

E-step: Computes the log-likelihood using current parameters. M-step: Maximize the expected log-likelihood, changing the parameters. Iterated until convergence is achieved.

CSE 312 Final Review: Section AA

Hypothesis Testing

CSE 312 Final Review: Section AA

Hypothesis Testing

H0 is the null hypothesis

CSE 312 Final Review: Section AA

Hypothesis Testing

H0 is the null hypothesis H1 is the alternative hypothesis

CSE 312 Final Review: Section AA

Hypothesis Testing

H0 is the null hypothesis H1 is the alternative hypothesis Likelihood Ratio = L1

L0 where L0 is the likelihood of H0 and L1

is the likelihood of H1.

CSE 312 Final Review: Section AA

Hypothesis Testing

H0 is the null hypothesis H1 is the alternative hypothesis Likelihood Ratio = L1

L0 where L0 is the likelihood of H0 and L1

is the likelihood of H1. Saying that alternative hypothesis is 5 times more likely than the null hypothesis means that L1

L0 ≥ 5.

CSE 312 Final Review: Section AA

Hypothesis Testing

H0 is the null hypothesis H1 is the alternative hypothesis Likelihood Ratio = L1

L0 where L0 is the likelihood of H0 and L1

is the likelihood of H1. Saying that alternative hypothesis is 5 times more likely than the null hypothesis means that L1

L0 ≥ 5.

Decision rule: When to reject H0.

CSE 312 Final Review: Section AA

Hypothesis Testing

H0 is the null hypothesis H1 is the alternative hypothesis Likelihood Ratio = L1

L0 where L0 is the likelihood of H0 and L1

is the likelihood of H1. Saying that alternative hypothesis is 5 times more likely than the null hypothesis means that L1

L0 ≥ 5.

Decision rule: When to reject H0. α = P(rejected H0 but H0 was true)

CSE 312 Final Review: Section AA

Hypothesis Testing

H0 is the null hypothesis H1 is the alternative hypothesis Likelihood Ratio = L1

L0 where L0 is the likelihood of H0 and L1

is the likelihood of H1. Saying that alternative hypothesis is 5 times more likely than the null hypothesis means that L1

L0 ≥ 5.

Decision rule: When to reject H0. α = P(rejected H0 but H0 was true) β = P(accept H0 but H1 was true)

CSE 312 Final Review: Section AA

Algorithms

CSE 312 Final Review: Section AA

Algorithms

Better algorithms usually trump better hardware, but both are needed for progress

CSE 312 Final Review: Section AA

Algorithms

Better algorithms usually trump better hardware, but both are needed for progress Some problems cannot be solved (e.g. the Halting Problem)

CSE 312 Final Review: Section AA

Algorithms

Better algorithms usually trump better hardware, but both are needed for progress Some problems cannot be solved (e.g. the Halting Problem) Other (intractable) problems cannot (yet?) be solved in a reasonable amount of time (e.g. Integer Factorization)

CSE 312 Final Review: Section AA

Sequence Alignment

CSE 312 Final Review: Section AA

Sequence Alignment

Brute force solution take at least 2n

n

• computations of the

sequence score, which is exponential time.

CSE 312 Final Review: Section AA

Sequence Alignment

Brute force solution take at least 2n

n

• computations of the

sequence score, which is exponential time. Dynamic Programming decrease computation to O(n2).

CSE 312 Final Review: Section AA

Sequence Alignment

Brute force solution take at least 2n

n

• computations of the

sequence score, which is exponential time. Dynamic Programming decrease computation to O(n2). IDEA:

CSE 312 Final Review: Section AA

Sequence Alignment

Brute force solution take at least 2n

n

• computations of the

sequence score, which is exponential time. Dynamic Programming decrease computation to O(n2). IDEA: Store prior computations so that future computations can do table look-ups.

CSE 312 Final Review: Section AA

Sequence Alignment

Brute force solution take at least 2n

n

• computations of the

sequence score, which is exponential time. Dynamic Programming decrease computation to O(n2). IDEA: Store prior computations so that future computations can do table look-ups. Backtrace Algorithm: Start at the bottom right of the matrix and find which neighboring cells could have transitioned to the current cell under the cost function, σ. Time Bound (O(n2))

CSE 312 Final Review: Section AA

Sequence Alignment

Brute force solution take at least 2n

n

• computations of the

sequence score, which is exponential time. Dynamic Programming decrease computation to O(n2). IDEA: Store prior computations so that future computations can do table look-ups. Backtrace Algorithm: Start at the bottom right of the matrix and find which neighboring cells could have transitioned to the current cell under the cost function, σ. Time Bound (O(n2)) See Lecture 15 for a worked example.

CSE 312 Final Review: Section AA

P vs. NP

CSE 312 Final Review: Section AA

P vs. NP

Some problems cannot be computed (e.g. The Halting Problem).

CSE 312 Final Review: Section AA

P vs. NP

Some problems cannot be computed (e.g. The Halting Problem). Some problems can be computed but take a long time (e.g. SAT, 3-SAT, 3-coloring).

CSE 312 Final Review: Section AA

P vs. NP

Some problems cannot be computed (e.g. The Halting Problem). Some problems can be computed but take a long time (e.g. SAT, 3-SAT, 3-coloring). P - a solution is computable in polynomial time

CSE 312 Final Review: Section AA

P vs. NP

Some problems cannot be computed (e.g. The Halting Problem). Some problems can be computed but take a long time (e.g. SAT, 3-SAT, 3-coloring). P - a solution is computable in polynomial time NP - a solution can be verified in polynomial time, given a hint that is polynomial in the input length

CSE 312 Final Review: Section AA

P vs. NP

Some problems cannot be computed (e.g. The Halting Problem). Some problems can be computed but take a long time (e.g. SAT, 3-SAT, 3-coloring). P - a solution is computable in polynomial time NP - a solution can be verified in polynomial time, given a hint that is polynomial in the input length A ≤p B means that if you have a fast algorithm for B, you have a fast algorithm for A.

CSE 312 Final Review: Section AA

P vs. NP

Some problems cannot be computed (e.g. The Halting Problem). Some problems can be computed but take a long time (e.g. SAT, 3-SAT, 3-coloring). P - a solution is computable in polynomial time NP - a solution can be verified in polynomial time, given a hint that is polynomial in the input length A ≤p B means that if you have a fast algorithm for B, you have a fast algorithm for A. NP-complete - In NP and as hard as the hardest problem in NP

CSE 312 Final Review: Section AA

P vs. NP

Some problems cannot be computed (e.g. The Halting Problem). Some problems can be computed but take a long time (e.g. SAT, 3-SAT, 3-coloring). P - a solution is computable in polynomial time NP - a solution can be verified in polynomial time, given a hint that is polynomial in the input length A ≤p B means that if you have a fast algorithm for B, you have a fast algorithm for A. NP-complete - In NP and as hard as the hardest problem in NP Any fast solution to an NP-complete problem would yield a fast solution to all problems in NP.

CSE 312 Final Review: Section AA