Linear Quantization by Yining Wang Effective Resistance Sampling - - PowerPoint PPT Presentation

ICASSP 2018, Calgary, Canada

Linear Quantization by Effective Resistance Sampling

Yining Wang

Carnegie Mellon University

Joint work with Aarti Singh

QUANTIZED LINEAR SENSING

❖ The linear model:

✴ X: n by p “design” matrix, with full knowledge ✴ y: n-dim vector, the sensing result ✴ β0: p-dim unknown signal to be recovered

y = Xβ0

QUANTIZED LINEAR SENSING

❖ The linear model: ❖ The quantized sensing problem:

✴ Measurements of y cannot be made in arbitrary precision ✴ A total of k bits allocated to each measurement yi ✴ Each yi rounded to the nearest integer with ki binary bits.

y = Xβ0 e yi = 2−(ki−1) · round h 2ki−1 yi M i

QUANTIZED LINEAR SENSING

❖ The linear model: ❖ The quantized sensing problem:

✴ Measurements of y cannot be made in arbitrary precision ✴ A total of k bits allocated to each measurement yi ✴ Each yi rounded to the nearest integer with ki binary bits.

y = Xβ0 e yi = 2−(ki−1) · round h 2ki−1 yi M i

Range of y

QUANTIZED LINEAR SENSING

❖ The linear model: ❖ The quantized sensing problem:

✴ Measurements of y cannot be made in arbitrary precision

❖ Example applications:

✴ Brain activity measurements: total signal strength limited ✴ Distributed sensing: signal communication limited

y = Xβ0

QUANTIZED LINEAR SENSING

❖ The linear model: ❖ The quantized sensing problem: ❖ Question: how to allocate measurement bits to achieve

the best statistical efficiency? y = Xβ0 e yi = 2−(ki−1) · round h 2ki−1 yi M i

DITHERING

❖ “Dithering”:

✴ Introducing artificial noise for independent statistical error ✴ Equivalent model:

e yi = 2−(ki−1) · round h 2ki−1 ⇣ yi M + δi ⌘i e yi = hxi, β0i + εi E[εi] = 0 E[ε2

i ] ≤ 4−(ki+1)M 2

DITHERING

❖ “Dithering”:

✴ Introducing artificial noise for independent statistical error ✴ Equivalent model:

e yi = 2−(ki−1) · round h 2ki−1 ⇣ yi M + δi ⌘i e yi = hxi, β0i + εi E[εi] = 0 E[ε2

i ] ≤ 4−(ki+1)M 2

Uniform noise between two values

WEIGHTED OLS

❖ “Dithering”: ❖ Weighted Ordinary Least Squares (OLS)

e yi = 2−(ki−1) · round h 2ki−1 ⇣ yi M + δi ⌘i E[ε2

i ] ≤ 4−(ki+1)M 2

= hxi, β0i + εi b βk = (X>WX)1X>W e y

WEIGHTED OLS

❖ “Dithering”: ❖ Weighted Ordinary Least Squares (OLS)

e yi = 2−(ki−1) · round h 2ki−1 ⇣ yi M + δi ⌘i E[ε2

i ] ≤ 4−(ki+1)M 2

= hxi, β0i + εi b βk = (X>WX)1X>W e y

W = diag(w1, w2, · · · , wn)

= diag(4k1+1, 4k2+1, · · · , 4kn+1)

WEIGHTED OLS

❖ Weighted Ordinary Least Squares (OLS) ❖ Optimal quantization:

b βk = (X>WX)1X>W e y Ekb βk β0k2

2  M 2 · tr

" n X

i=1

4ki+1xix>

i

#1 min

k tr[X>WX]1

s.t. k1 + · · · + kn ≤ k, ki ∈ N

WEIGHTED OLS

❖ Weighted Ordinary Least Squares (OLS) ❖ Optimal quantization:

b βk = (X>WX)1X>W e y Ekb βk β0k2

2  M 2 · tr

" n X

i=1

4ki+1xix>

i

#1 X>WX min

k tr[X>WX]1

s.t. k1 + · · · + kn ≤ k, ki ∈ N

WEIGHTED OLS

❖ Weighted Ordinary Least Squares (OLS) ❖ Optimal quantization:

b βk = (X>WX)1X>W e y Ekb βk β0k2

2  M 2 · tr

" n X

i=1

4ki+1xix>

i

#1 X>WX min

k tr[X>WX]1

s.t. k1 + · · · + kn ≤ k, ki ∈ N

Combinatorial… hard!

CONTINUOUS RELAXATION

❖ Continuously relaxed optimal quantization: ❖ Still a challenging problem…

✴ Non-convexity of objectives!

min

k tr[X>WX]1

s.t. k1 + · · · + kn ≤ k, ki ∈ N

CONTINUOUS RELAXATION

❖ Continuously relaxed optimal quantization: ❖ Still a challenging problem…

✴ Non-convexity of objectives!

min

k tr[X>WX]1

s.t. k1 + · · · + kn ≤ k, ki ∈ N ki ∈ R+

CONTINUOUS RELAXATION

❖ Continuously relaxed optimal quantization: ❖ A re-formulation:

min

k tr[X>WX]1

s.t. k1 + · · · + kn ≤ k, ki ∈ N ki ∈ R+ min tr " n X

i=1

4ki+1xix>

i

#1 s.t.

n

X

i=1

ki ≤ k min tr " n X

i=1

wixix>

i

#1 s.t.

n

X

i=1

log4(wi) − 1 ≤ k

CONTINUOUS RELAXATION

❖ Continuously relaxed optimal quantization: ❖ A re-formulation:

min

k tr[X>WX]1

s.t. k1 + · · · + kn ≤ k, ki ∈ N ki ∈ R+ min tr " n X

i=1

4ki+1xix>

i

#1 s.t.

n

X

i=1

ki ≤ k min tr " n X

i=1

wixix>

i

#1 s.t.

n

X

i=1

log4(wi) − 1 ≤ k

Convex objective

CONTINUOUS RELAXATION

❖ Continuously relaxed optimal quantization: ❖ A re-formulation:

min

k tr[X>WX]1

s.t. k1 + · · · + kn ≤ k, ki ∈ N ki ∈ R+ min tr " n X

i=1

4ki+1xix>

i

#1 s.t.

n

X

i=1

ki ≤ k min tr " n X

i=1

wixix>

i

#1 s.t.

n

X

i=1

log4(wi) − 1 ≤ k

Convex objective Non-convex feasible set

CONTINUOUS RELAXATION

❖ Continuously relaxed optimal quantization: ❖ A re-formulation:

min

k tr[X>WX]1

s.t. k1 + · · · + kn ≤ k, ki ∈ N ki ∈ R+ min tr " n X

i=1

wixix>

i

#1 s.t.

n

X

i=1

log4(wi) − 1 ≤ k min tr " n X

i=1

wixix>

i

#1 + λ " n X

i=1

log4(wi) − (n − k) #

CONTINUOUS RELAXATION

❖ Continuously relaxed optimal quantization: ❖ A re-formulation:

min

k tr[X>WX]1

s.t. k1 + · · · + kn ≤ k, ki ∈ N ki ∈ R+ min tr " n X

i=1

wixix>

i

#1 s.t.

n

X

i=1

log4(wi) − 1 ≤ k min tr " n X

i=1

wixix>

i

#1 + λ " n X

i=1

log4(wi) − (n − k) #

Convex objective

CONTINUOUS RELAXATION

❖ Continuously relaxed optimal quantization: ❖ A re-formulation:

min

k tr[X>WX]1

s.t. k1 + · · · + kn ≤ k, ki ∈ N ki ∈ R+ min tr " n X

i=1

wixix>

i

#1 s.t.

n

X

i=1

log4(wi) − 1 ≤ k min tr " n X

i=1

wixix>

i

#1 + λ " n X

i=1

log4(wi) − (n − k) #

Convex objective concave objective

CONTINUOUS RELAXATION

❖ Continuously relaxed optimal quantization: ❖ A re-formulation:

✴ DC (Difference of Convex functions) programming:

min

k tr[X>WX]1

s.t. k1 + · · · + kn ≤ k, ki ∈ N ki ∈ R+ min tr " n X

i=1

wixix>

i

#1 − λ " −

n

X

i=1

log4(wi) + (n − k) #

ROUNDING / SPARSIFICATION

❖ Continuously relaxed optimal quantization: ❖ How to obtain integral solutions? “Sparsify” k

✴ Idea 1: round to the nearest integer ✴ Problem: might cause objective to increase significantly

min

k tr[X>WX]1

s.t. k1 + · · · + kn ≤ k, ki ∈ N ki ∈ R+

ROUNDING / SPARSIFICATION

❖ Continuously relaxed optimal quantization: ❖ How to obtain integral solutions? “Sparsify” k

✴ Idea 2: simple sampling

❖

Sample i from the distribution normalized by k

❖

k(i) = k(i) + 1

✴ Problem: slow convergence (require large budget k)

min

k tr[X>WX]1

s.t. k1 + · · · + kn ≤ k, ki ∈ N ki ∈ R+

ROUNDING / SPARSIFICATION

❖ Continuously relaxed optimal quantization: ❖ How to obtain integral solutions? “Sparsify” k

✴ Idea 3: effective resistance sampling ✴ Advantage: fast convergence (k independent of condition

numbers of X or W.

min

k tr[X>WX]1

s.t. k1 + · · · + kn ≤ k, ki ∈ N ki ∈ R+ t ∼ pt ∝ 4kt+1`t

ROUNDING / SPARSIFICATION

❖ Continuously relaxed optimal quantization: ❖ How to obtain integral solutions? “Sparsify” k

✴ Idea 3: effective resistance sampling ✴ Advantage: fast convergence (k independent of condition

numbers of X or W.

min

k tr[X>WX]1

s.t. k1 + · · · + kn ≤ k, ki ∈ N ki ∈ R+ t ∼ pt ∝ 4kt+1`t

Effective resistance:

`t = x>

t [W ⇤]1xt

OPEN QUESTIONS

❖ Most important question: how to solve (continuous) ❖ Some ideas:

✴ Is the objective quasi-convex or directional convex? ✴ Are local minima also global, or approximately global?

❖

Escaping saddle point methods?

✴ Are there adequate convex relaxations?

min tr " n X

i=1

4ki+1xix>

i

#1 s.t.

n

X

i=1

ki ≤ k

Thank you! Questions