The Detection of Defective Members of Large Populations November - - PowerPoint PPT Presentation

▶
the detection of defective members of large populations
SMART_READER_LITE
LIVE PREVIEW

The Detection of Defective Members of Large Populations November - - PowerPoint PPT Presentation

Oct 17, 2023 252 likes •1.22k views

The Detection of Defective Members of Large Populations November 21, 2019 This is me PhD Student at Stanford, ex-engineer Outline The paper What makes the paper work? How its ideas can be reused Group Testing The setting is World

slide-1
SLIDE 1

The Detection of Defective Members of Large Populations

November 21, 2019

slide-2
SLIDE 2

This is me

PhD Student at Stanford, ex-engineer

slide-3
SLIDE 3
slide-4
SLIDE 4

Outline

  • The paper
  • What makes the paper work?
  • How its ideas can be reused
slide-5
SLIDE 5
slide-6
SLIDE 6
slide-7
SLIDE 7

Group Testing

The setting is World War II…

slide-8
SLIDE 8

🤡

Group Testing

The setting is World War II…

slide-9
SLIDE 9

🤡 🤡

Group Testing

The setting is World War II…

slide-10
SLIDE 10

🤡 🤡 🤡

Group Testing

The setting is World War II…

slide-11
SLIDE 11

🤡 🤡 🤡 🤡

Group Testing

The setting is World War II…

slide-12
SLIDE 12

🤡 🤡 🤡 🤡 🤡

Group Testing

The setting is World War II…

slide-13
SLIDE 13

🤡 🤡 🤡 🤡 🤡 🤡

Group Testing

The setting is World War II…

slide-14
SLIDE 14

🤡 🤡 🤡 🤡 🤡 🤡 🤡

Group Testing

The setting is World War II…

slide-15
SLIDE 15

🤡 🤡 🤡 🤡 🤡 🤡

Sick :(

🤣 🤡

Group Testing

The setting is World War II…

slide-16
SLIDE 16

Group Testing

🤡 🤡 🤡 🤡 🤡 🤡

Sick :(

🤣 🤡 💊

slide-17
SLIDE 17

Group Testing

🤡 🤡 🤡 🤡 🤡 🤡

Sick :(

🤣 🤡 💊

slide-18
SLIDE 18

Don’t need individual tests

🤡 🤡 🤡 🤡 🤡 🤡 🤣 🤡

slide-19
SLIDE 19 Ok

Don’t need individual tests

🤡 🤡 🤡 🤡 🤡 🤡 🤣 🤡

slide-20
SLIDE 20

Sick :(

Ok

Don’t need individual tests

🤡 🤡 🤡 🤡 🤡 🤡 🤣 🤡

slide-21
SLIDE 21 Ok

Sick :(

Ok

Don’t need individual tests

🤡 🤡 🤡 🤡 🤡 🤡 🤣 🤡

slide-22
SLIDE 22 Ok

Sick :(

Ok

Don’t need individual tests

🤡 🤡 🤡 🤡 🤡 🤡 🤣 🤡

We know this person is sick
slide-23
SLIDE 23 Ok

Sick :(

Ok

Need to carefully design tests

🤡 🤡 🤡 🤡 🤣 🤡 🤡 🤡

We can’t distinguish these two
slide-24
SLIDE 24 Ok

Sick :(

Ok

Need to carefully design tests

🤡 🤡 🤡 🤡 🤡 🤣 🤡 🤡

We can’t distinguish these two
slide-25
SLIDE 25

Group Testing Problem

We have n items, at most s of which are “sick.” Definition: A test returns whether a subset of items includes any sick items or not. Problem: Construct a set of tests which can identify a worst-case set of at most s sick items.

slide-26
SLIDE 26

A better design

If every column is unique, we win

🤡 🤡 🤡 🤡 🤡 🤡 🤡

slide-27
SLIDE 27

A better design

If every column is unique, we win

🤡 🤡 🤡 🤣 🤡 🤡 🤡

Ok Ok Sick

slide-28
SLIDE 28

A better design

If every column is unique, we win

🤡 🤡 🤡 🤣 🤡 🤡 🤡

Ok Sick Sick

slide-29
SLIDE 29

A better design

If every column is unique, we win

🤡 🤡 🤡 🤣 🤡 🤡 🤡

1 1 1 1 1 1 1 1 1 1 1 1

Ok Sick Ok

slide-30
SLIDE 30

What we just saw

If there is one sick person, we can find them non-adaptively with log n tests!

slide-31
SLIDE 31

Dorfman’s Construction

This seems hard, so let’s just do something totally random Will show this works with decent probability and O(s2 log n) tests

slide-32
SLIDE 32

Why is s2log n tests cool?

100 80 60 40 20
slide-33
SLIDE 33

Why is s2log n tests cool?

100 80 60 40 20 Way fewer tests!
slide-34
SLIDE 34

Dorfman’s construction 🤡 🤡 🤡 🤡 🤡 🤡 🤡

Include with probability 1/s
slide-35
SLIDE 35

Dorfman’s construction 🤡 🤡 🤡 🤡 🤡 🤡 🤡

Include with probability 1/s
slide-36
SLIDE 36

Dorfman’s construction 🤡 🤡 🤡 🤡 🤡 🤡 🤡

Include with probability 1/s
slide-37
SLIDE 37

Dorfman’s construction 🤡 🤡 🤡 🤡 🤡 🤡 🤡

Include with probability 1/s
slide-38
SLIDE 38

Dorfman’s construction 🤡 🤡 🤡 🤡 🤡 🤡 🤡

Include with probability 1/s
slide-39
SLIDE 39

Dorfman’s construction 🤡 🤡 🤡 🤡 🤡 🤡 🤡

Include with probability 1/s
slide-40
SLIDE 40

Dorfman’s construction 🤡 🤡 🤡 🤡 🤡 🤡 🤡

Include with probability 1/s
slide-41
SLIDE 41

Dorfman’s construction 🤡 🤡 🤡 🤡 🤡 🤡 🤡

Include with probability 1/s
slide-42
SLIDE 42

Dorfman’s construction 🤡 🤡 🤡 🤡 🤡 🤡 🤡

Include with probability 1/s
slide-43
SLIDE 43

Ok Sick Ok

First idea: finding healthy people

🤡 🤡 🤡 🤣 🤡 🤡 🤡

slide-44
SLIDE 44

Ok Sick Ok

First idea: finding healthy people

🤡 🤡 🤡 🤣 🤡 🤡 🤡

These tests pass

}

slide-45
SLIDE 45

Ok Sick Ok

First idea: finding healthy people

🤡 🤡 🤡 🤣 🤡 🤡 🤡

These tests pass These people cannot be sick!

}

slide-46
SLIDE 46

First idea: finding healthy people

For each set of sick people, need to be able to prove each other person is healthy

Should not be in the test

🤣 🤯 😸 🤖 🤡 😏

Should be in test

🥴 🤔 🙄

slide-47
SLIDE 47

First idea: finding healthy people

For each set of sick people, need to be able to prove each other person is healthy

Should not be in the test

🤣 🤯 😸 🤖 🤡 😏

Should be in test

🥴 🤔 🙄

Math time!

slide-48
SLIDE 48

First idea: finding healthy people

For each set of sick people, need to be able to prove each other person is healthy

Should not be in the test

🤣 🤯 😸 🤖 🤡 😏

Should be in test

🥴 🤔 🙄

What is the probability this happens? P(none in test) =

slide-49
SLIDE 49

First idea: finding healthy people

For each set of sick people, need to be able to prove each other person is healthy

Should not be in the test

🤣 🤯 😸 🤖 🤡 😏

Should be in test

🥴 🤔 🙄

What is the probability this happens? P(none in test) =

in test w/ p. 1/s
slide-50
SLIDE 50

First idea: finding healthy people

For each set of sick people, need to be able to prove each other person is healthy

Should not be in the test

🤣 🤯 😸 🤖 🤡 😏

Should be in test

🥴 🤔 🙄

What is the probability this happens? P(none in test) =

in test w/ p. 1/s in test w/ p. 1/s
slide-51
SLIDE 51

First idea: finding healthy people

For each set of sick people, need to be able to prove each other person is healthy

Should not be in the test

🤣 🤯 😸 🤖 🤡 😏

Should be in test

🥴 🤔 🙄

What is the probability this happens? P(none in test) =

in test w/ p. 1/s in test w/ p. 1/s in test w/ p. 1/s
slide-52
SLIDE 52

First idea: finding healthy people

For each set of sick people, need to be able to prove each other person is healthy

Should not be in the test

🤣 🤯 😸 🤖 🤡 😏

Should be in test

🥴 🤔 🙄

What is the probability this happens? P(none in test) = (1-1/s)s

in test w/ p. 1/s in test w/ p. 1/s in test w/ p. 1/s
slide-53
SLIDE 53

First idea: finding healthy people

For each set of sick people, need to be able to prove each other person is healthy

Should not be in the test

🤣 🤯 😸 🤖 🤡 😏

Should be in test

🥴 🤔 🙄

What is the probability this happens? P(none in test) = (1-1/s)s ≈ e-s/s

in test w/ p. 1/s in test w/ p. 1/s in test w/ p. 1/s
slide-54
SLIDE 54

First idea: finding healthy people

For each set of sick people, need to be able to prove each other person is healthy

Should not be in the test

🤣 🤯 😸 🤖 🤡 😏

Should be in test

🥴 🤔 🙄

What is the probability this happens? P(none in test) = (1-1/s)s ≈ e-s/s ≈ 1/3

in test w/ p. 1/s in test w/ p. 1/s in test w/ p. 1/s
slide-55
SLIDE 55

First idea: finding healthy people

For each set of sick people, need to be able to prove each other person is healthy

Should not be in the test

🤣 🤯 😸 🤖 🤡 😏

Should be in test

🥴 🤔 🙄

What is the probability this happens? P(none in test) = (1-1/s)s ≈ e-s/s ≈ 1/3

in test w/ p. 1/s in test w/ p. 1/s in test w/ p. 1/s

Idea: not too many sick people, so pretty good probability of missing ‘em all

slide-56
SLIDE 56

First idea: finding healthy people

For each set of sick people, need to be able to prove each other person is healthy

Not in test w/ probability 1/3

🤣 🤯 😸 🤖 🤡 😏 🥴 🤔 🙄

Should be in test

Need this person in test

slide-57
SLIDE 57

First idea: finding healthy people

For each set of sick people, need to be able to prove each other person is healthy

Not in test w/ probability 1/3

🤣 🤯 😸 🤖 🤡 😏 🥴 🤔 🙄

Should be in test

What is the probability this happens? P(🙄 in test) = 1/s

Need this person in test

slide-58
SLIDE 58

First idea: finding healthy people

For each set of sick people, need to be able to prove each other person is healthy

Not in test w/ probability 1/3

🤣 🤯 😸 🤖 🤡 😏 🥴 🤔 🙄

Should be in test

What is the probability the test works? P(none in test and 🙄 in test) ≈ 1/3s

slide-59
SLIDE 59

Repeating tests 🤡 🤔 🙄 😏

Works with probability 1/3s Works with probability 1/3s …

🤣 🤯 😸

slide-60
SLIDE 60

Repeating tests 🤡 🤔 🙄 😏

Works with probability 1/3s Works with probability 1/3s …

🤣 🤯 😸

What is the probability no test works? P(no test works) = (1-1/3s)T

slide-61
SLIDE 61

Repeating tests 🤡 🤔 🙄 😏

Works with probability 1/3s Works with probability 1/3s …

🤣 🤯 😸

What is the probability no test works? P(no test works) = (1-1/3s)T ≈ e-T/3s

slide-62
SLIDE 62

Repeating tests 🤡 🤔 🙄 😏

Works with probability 1/3s Works with probability 1/3s …

🤣 🤯 😸

What is the probability no test works? P(no test works) = (1-1/3s)T ≈ e-T/3s ≈ n-2s T = 6s2logn

slide-63
SLIDE 63

Union bound 🤡 🤔 🙄 😏 🤣 🤯 😸

We just saw P(no test works for 🙄 and 🤣,🤯,😸) ≈ n-2s

slide-64
SLIDE 64

Union bound 🤡 🤔 🙄 😏 🤣 🤯 😸

We just saw P(no test works for 🙄 and 🤣,🤯,😸) ≈ n-2s But we haven’t dealt with 🤡,🤔,😏

slide-65
SLIDE 65

Dorfman’s Construction

It works! With good probability! And very few tests!!

slide-66
SLIDE 66

Why did this work?

slide-67
SLIDE 67

Key components

1. Not too many things to find

  • 2. Get information about many things in each test
  • 3. Can do something random
slide-68
SLIDE 68

Coin weighting Compressed Sensing Traitor Tracing Streaming Algorithms Johnson- Lindenstrauss Mastermind Network Tomography Finding wifi users IP Traceback Error correcting codes Multicast Message Authentication

slide-69
SLIDE 69

Coin weighting Compressed Sensing Traitor Tracing Streaming Algorithms Johnson- Lindenstrauss Mastermind Network Tomography Finding wifi users IP Traceback Error correcting codes Multicast Message Authentication

slide-70
SLIDE 70

Coin weighting Compressed Sensing Traitor Tracing Streaming Algorithms Johnson- Lindenstrauss Mastermind Network Tomography Finding wifi users IP Traceback Error correcting codes Multicast Message Authentication

Joint work with Mary Wootters

slide-71
SLIDE 71
slide-72
SLIDE 72

A network

slide-73
SLIDE 73

A network, failing

🔦

slide-74
SLIDE 74

Finding failures

🔦

💍

slide-75
SLIDE 75

Finding failures

🔦

slide-76
SLIDE 76

Finding failures

🔦

💍

slide-77
SLIDE 77

Finding failures

🔦

💍

slide-78
SLIDE 78

Tomography problem

We have a graph G=(V,E) with n edges, at most s edges are sick. Definition: A graph-constrained test returns whether any edges in a connected subset of edges are sick or not. Problem: Construct a set of graph-constrained tests which can identify any set of at most s sick edges.

slide-79
SLIDE 79

This seems tricky

Which is sick?

💍 🔦

slide-80
SLIDE 80

This seems tricky

Theorem [Harvey et al 2007]: For the line graph on n nodes, about n/2 tests required Proof: Each neighboring pair of edges must be separated by some test. Each test is a path and can only separate two pairs. There are about n pairs.

slide-81
SLIDE 81

Key components?

1. Not too many things to find

  • 2. Get information about many things in each test
  • 3. Can do something random
slide-82
SLIDE 82

Key components?

1. Not too many things to find

  • 2. Get information about many things in each test
  • 3. Can do something random

👎

slide-83
SLIDE 83

Key components?

1. Not too many things to find

  • 2. Get information about many things in each test
  • 3. Can do something random

👎 👎

slide-84
SLIDE 84

Key components?

1. Not too many things to find

  • 2. Get information about many things in each test
  • 3. Can do something random

👎 👎 👎

slide-85
SLIDE 85

Our informal result

If a graph is sufficiently well-enough connected, we can find any set of s sick edges using O(s2 log n) tests

slide-86
SLIDE 86

Our informal result

If a graph is sufficiently well-enough connected, we can find any set of s sick edges using O(s2 log n) tests

Same as group testing

slide-87
SLIDE 87

Algorithm

For 1…2s2log n:

  • Include each edge with probability

p ~ 1/s

  • Use large connected components

as tests

14 15 17 22 19 24 25 26 16 18 20 21 23 27 3 4 7 8 9 12 13 1 11 2 6 5 10
slide-88
SLIDE 88

Example: K4 (6 edges)

slide-89
SLIDE 89

Example: K4 (6 edges)

slide-90
SLIDE 90

Example: K10 (45 edges)

slide-91
SLIDE 91

Coin weighting Compressed Sensing Traitor Tracing Streaming Algorithms Johnson- Lindenstrauss Mastermind Network Tomography Finding wifi users IP Traceback Error correcting codes Multicast Message Authentication

slide-92
SLIDE 92

Coin weighting Compressed Sensing Traitor Tracing Streaming Algorithms Johnson- Lindenstrauss Mastermind Network Tomography Finding wifi users IP Traceback Error correcting codes Multicast Message Authentication

slide-93
SLIDE 93

Key components

1. Not too many things to find

  • 2. Get information about many things in each test
  • 3. Can do something random
slide-94
SLIDE 94
slide-95
SLIDE 95

Thanks!