Lossless or Quantized Boosting with Integer Arithmetic Richard Nock - - PowerPoint PPT Presentation

Lossless or Quantized Boosting with Integer Arithmetic

Robert C. Williamson Richard Nock

• Many constraints in today-ML (e.g. for privacy, at-the-edge, distributed or deep) 



 — generic: integer encoding, small set of operations, quantisation (+ accuracy)


• The shortest path to solutions: hammering existing SOTA for new constraints



 — does not go without uncertainty or loss in SOTA guarantees


• Alternative: “replace current ML algorithms with [constraint-friendly] ones”



 — some great stories in supervised ML start 
 from the same ground, a “nice” loss function…

Lossless or Quantized boosting with Integer Arithmetic — Nock and Williamson 2

The big picture

(NeurIPS’18 PPML workshop) cryptographic (SVM, Boosting, etc.) (may need a
 big hammer)

Lossless or Quantized boosting with Integer Arithmetic — Nock and Williamson 3

…so we created a new loss that fits to the constraints

Lossless or Quantized boosting with Integer Arithmetic — Nock and Williamson 4

• Loss for class probability estimation: , where
• Can be defined from partial losses
• Ex: square loss has &

• (pointwise) Bayes risk: , proper if in inf
• Ex: square loss has (Gini entropy), proper

• Real valued classification via a link giving
• Proper loss canonical if link implicit, given by
• Ex: square loss has , proper canonical

Statistical properties of a good loss

`

sq

ψ(y, z) = (1/4) · (1 − 2yz)2

`ψ(y, z)

.

= `(y, −1(z))

` : Y × (0, 1) → R

`(y, u)

.

= Jy = 1K · `1(u) + Jy = −1K · `−1(u)

`

sq

1 (u) .

= (1/2) · (1 − u)2

`

sq

−1(u) .

= (1/2) · u2

L(⇡)

.

= infc EY∼π`(Y, c)

Y

.

= {−1, 1}

π

L

sq(π) = (1/2) · π(1 − π)

ψ : [0, 1] → R

ψ

.

= −L0

(concave) (convex)

F(yz)

“surrogate”

Lossless or Quantized boosting with Integer Arithmetic — Nock and Williamson 5

• 1- Statistics: strictly proper canonical,
• 2- Statistics: link with image spanning the full ,
• 3- Optimisation: strictly convex 2x differentiable,
• 4- Learning: mirror update entails +, -, /, *, |.|
• Ex: 1 rules out exp loss, 2 rules out square loss, 4 rules out log loss, etc., so

no popular loss fits…

Desiderata for our loss, summarised

ℝ

0.2 0.4 0.6 0.8 1

• 4
• 3
• 2
• 1

1 2 3 4 square loss logistic loss Q-loss

⋄

F

ψ−1

z ⇧ u ψ−1(z + ψ(u))

• partial losses: The Q-loss is defined from the following partial losses,

Lossless or Quantized boosting with Integer Arithmetic — Nock and Williamson 6

The Q-loss

" ∈ (0, 1/2), % > 0

`Q

y (u) .

= % · ✓ log ⇣u " ⌘ + Jy = 1K · ✓ −2 + 1 u ◆◆

u ≤ 1/2

`Q

y (u) .

= `Q

−y(1 − u)

u > 1/2

Pointwise Bayes risk:

LQ(u) = % · ✓ log ✓err(u) " ◆ + 1 − 2 · err(u) ◆

h(z)

.

= 0 ∨ −z

err(u)

.

= u ∧ (1 − u)

Surrogate:

F Q(z) = −% · log " − % · log ✓ 2 + |z| % ◆ + h(z)

Mirror update:

z ⇧ u = % · err(u) + H (z · err(u) + % · (1 2u)) 2% · err(u) + |z · err(u) + % · (1 2u)|

Theorem:

(can be simplified)

The Q-loss fits all four constraints

• Boosting 1: yields a boosting-compliant algorithm for (weak/strong learning)

• If inputs are rational numbers, the exact solution can be computed using

integer arithmetic (lossless solution)

• Sufficient condition on weight quantization to keep boosting convergence

• Boosting 2: pointwise Bayes risk yields optimal boosting rate for decision trees

• Experiments:

Lossless or Quantized boosting with Integer Arithmetic — Nock and Williamson 7

Learning with the Q-loss

⋄

LQ

Efficient adaptive weight quantization scheme

2 bits 3 bits 4 bits 5 bits 6 bits

F Q

Lossless or Quantized boosting with Integer Arithmetic — Nock and Williamson 8

Thank you !

(more on achieving lossless boosting by choosing a loss ? poster # 194)