SLIDE 1
The Multiclass Prediction Problem
Repeatedly
1 See x 2 Predict ˆ
y ∈ {1, ..., K}
3 See y
Logarithmic Time Prediction John Langford Microsoft Research - - PowerPoint PPT Presentation
Logarithmic Time Prediction John Langford Microsoft Research - - PowerPoint PPT Presentation
Logarithmic Time Prediction John Langford Microsoft Research DIMACS Workshop on Big Data through the Lens of Sublinear Algorithms The Multiclass Prediction Problem Repeatedly 1 See x 2 Predict y { 1 , ..., K } 3 See y The Multiclass
SLIDE 2
SLIDE 3
The Multiclass Prediction Problem
Repeatedly
1 See x 2 Predict ˆ
y ∈ {1, ..., K}
3 See y
Goal: Find h(x) minimizing error rate: Pr
(x,y)∼D(h(x) = y)
with h(x) fast.
SLIDE 4
Why?
SLIDE 5
Why?
SLIDE 6
Trick #1
K is small
SLIDE 7
Trick #2: A hierarchy exists
SLIDE 8
Trick #2: A hierarchy exists
So use Trick #1 repeatedly.
SLIDE 9
Trick #3: Shared representation
SLIDE 10
Trick #3: Shared representation
Very helpful... but computation in the last layer can still blow up.
SLIDE 11
Trick #4: “Structured Prediction”
SLIDE 12
Trick #4: “Structured Prediction”
But what if the structure is unclear?
SLIDE 13
Trick #5: GPU
SLIDE 14
Trick #5: GPU
4 Teraflops is great... yet still burns energy.
SLIDE 15
How fast can we hope to go?
SLIDE 16
How fast can we hope to go?
Theorem: There exists multiclass classification problems where achieving 0 error rate requires Ω(log K) time to train or test per example.
SLIDE 17
How fast can we hope to go?
Theorem: There exists multiclass classification problems where achieving 0 error rate requires Ω(log K) time to train or test per example. Proof: By construction Pick y ∼ U(1, ..., K)
SLIDE 18
How fast can we hope to go?
Theorem: There exists multiclass classification problems where achieving 0 error rate requires Ω(log K) time to train or test per example. Proof: By construction Pick y ∼ U(1, ..., K) Any prediction algorithm outputting less than log2 K bits loses with constant probability. Any training algorithm reading an example requires Ω(log2 K) time.
SLIDE 19
Can we predict in time O(log2 K)?
1 10 100 1000 10000 100000 10 100 1000 10000 100000 1e+06 Benefit K Computational Advantage of Log Time K / log(K)
SLIDE 20
Not it #1: Sparse Error Correcting Output Codes
1 Create O(log K) binary vectors biy of length K
SLIDE 21
Not it #1: Sparse Error Correcting Output Codes
1 Create O(log K) binary vectors biy of length K 2 Train O(log K) binary classifiers hi to minimize error rate:
Prx,y(hi(x) = biy)
SLIDE 22
Not it #1: Sparse Error Correcting Output Codes
1 Create O(log K) binary vectors biy of length K 2 Train O(log K) binary classifiers hi to minimize error rate:
Prx,y(hi(x) = biy)
3 Predict by finding y with minimal error.
SLIDE 23
Not it #1: Sparse Error Correcting Output Codes
1 Create O(log K) binary vectors biy of length K 2 Train O(log K) binary classifiers hi to minimize error rate:
Prx,y(hi(x) = biy)
3 Predict by finding y with minimal error.
Prediction is Ω(K)
SLIDE 24
Not it #2: Hierarchy Construction
1 Build confusion matrix of errors.
SLIDE 25
Not it #2: Hierarchy Construction
1 Build confusion matrix of errors. 2 Recursive partition to create hierarchy.
SLIDE 26
Not it #2: Hierarchy Construction
1 Build confusion matrix of errors. 2 Recursive partition to create hierarchy. 3 Apply hierarchy solution.
SLIDE 27
Not it #2: Hierarchy Construction
1 Build confusion matrix of errors. 2 Recursive partition to create hierarchy. 3 Apply hierarchy solution.
Training is Ω(K) or worse.
SLIDE 28
Not it #3: Unnormalized learning
Train K regressors by For each example (x, y)
1 Train regressor y with (x, 1).
SLIDE 29
Not it #3: Unnormalized learning
Train K regressors by For each example (x, y)
1 Train regressor y with (x, 1). 2 Pick y′ = y uniformly at random. 3 Train regressor y′ with (x, −1).
SLIDE 30
Not it #3: Unnormalized learning
Train K regressors by For each example (x, y)
1 Train regressor y with (x, 1). 2 Pick y′ = y uniformly at random. 3 Train regressor y′ with (x, −1).
Prediction is still Ω(K).
SLIDE 31
Can we predict in time O(log2 K)?
SLIDE 32
Is logarithmic time even possible?
P(y=1) = .4 P(y=2) = .3 P(y=3) = .3 1 2 3 1 v {2,3} 2 v 3
P({2, 3}) > P(1) ⇒ lose for divide and conquer
SLIDE 33
Filter Trees [BLR09]
P(y=1) = .4 P(y=2) = .3 P(y=3) = .3 1 2 3 1 v {2,3} 2 v 3
1 Learn 2v3 first 2 Throw away all error examples 3 Learn 1 v Survivors
Theorem: For all multiclass problems, for all binary classifiers, Multiclass Regret ≤ Average Binary Regret * log(K)
SLIDE 34
Can you make it robust?
1 2 4 5 6 7 8 3 Winner
SLIDE 35
Can you make it robust?
1 2 4 5 6 7 8 3 Winners
SLIDE 36
Can you make it robust?
1 2 4 5 6 7 8 3 Winners
SLIDE 37
Can you make it robust?
1 2 4 5 6 7 8 3 Winners
Theorem: [BLR09] For all multiclass problems, for all binary classifiers, a log(K)-correcting tournament satisfies: Multiclass Regret ≤ Average Binary Regret * 5.5 Determined best paper prize for ICML2012 (area chair decisions).
SLIDE 38
How do you learn structure?
Not all partitions are equally difficult. Compare {1, 7}v{3, 8} to {1, 8}v{3, 7} What is better?
SLIDE 39
How do you learn structure?
Not all partitions are equally difficult. Compare {1, 7}v{3, 8} to {1, 8}v{3, 7} What is better? [BWG10]: Better to confuse near leaves than near root. Intuition: the root predictor tends to be overconstrained while the leafwards predictors are less constrained.
SLIDE 40
The Partitioning Problem [CL14]
Given a set of n examples each with one of K labels, find a partitioner h that maximizes: Ex,y| Pr(h(x) = 1, y) − Pr(h(x) = 1) Pr(y)|
SLIDE 41
The Partitioning Problem [CL14]
Given a set of n examples each with one of K labels, find a partitioner h that maximizes: Ex
- y
Pr(y)| Pr(h(x) = 1|x ∈ Xy) − Pr(h(x) = 1)| where Xy is the set of x associated with y.
SLIDE 42
The Partitioning Problem [CL14]
Given a set of n examples each with one of K labels, find a partitioner h that maximizes: Nonconvex for any symmetric hypothesis class (ouch)
SLIDE 43
Bottom Up doesn’t work
1 2 3
Suppose you use linear representations.
SLIDE 44
Bottom Up doesn’t work
1 2 3
Suppose you use linear representations. Suppose you first build a 1v3 predictor.
SLIDE 45
Bottom Up doesn’t work
1 2 3
Suppose you use linear representations. Suppose you first build a 1v3 predictor. Suppose you then build a 2v{1v3} predictor. You lose.
SLIDE 46
Does partitioning recurse well?
Theorem: If at every node n, Ex,y| Pr(h(x) = 1, y) − Pr(h(x) = 1) Pr(y)| > γ then after 1 ǫ 4(1−γ)2 ln k
γ2
splits, the multiclass error is less than ǫ.
SLIDE 47
Online Partitioning
Relax the optimization criteria: Ex,y
- Ex|y [ˆ
y(x)] − Ex [ˆ y(x)]
- ... and approximate with running average
SLIDE 48
Online Partitioning
Relax the optimization criteria: Ex,y
- Ex|y [ˆ
y(x)] − Ex [ˆ y(x)]
- ... and approximate with running average
Let e = 0 and for all y, ey = 0, ny = 0 For each example (x, y)
1 if ey < e then b = −1 else b = 1 2 Update w using (x, b) 3 ny ← ny + 1 4 ey ← (ny−1)ey
ny
+ ˆ
y(x) ny
5 e ← (t−1)e
t
+ ˆ
y(x) t
Apply recursively to construct a tree structure.
SLIDE 49
Accuracy for a fixed training time
0.001 0.01 0.1 1 26 isolet 105 sector 1000 aloi 21841 imagenet 105033 ODP accuracy number of classes LOMtree vs one-against-all LOMtree OAA
SLIDE 50
Test Error %, optimized, no train-time constraint
10 20 30 40 50 60 70 80 90 100 Isolet Sector Aloi Imagenet ODP Test Error % Performance of Log-time algorithms Rand Filter LOM
SLIDE 51
Test Error %, optimized, no train-time constraint
10 20 30 40 50 60 70 80 90 100 Isolet Sector Aloi Imagenet ODP Test Error % Compared to OAA Rand Filter LOM OAA
SLIDE 52
Classes vs Test time ratio
6 8 10 12 14 16 2 4 6 8 10 12 log2(number of classes) log2(time ratio) LOMtree vs one−against−all
SLIDE 53
Can we predict in time O(log2 K)?
SLIDE 54
Can we predict in time O(log2 K)?
What is the right way to achieve consistency and dynamic partition?
SLIDE 55
Can we predict in time O(log2 K)?
What is the right way to achieve consistency and dynamic partition? How can you balance representation complexity and sample complexity?
SLIDE 56