Dim imensionality ty Redu eduction: Th Theoretic ical Ana - - PowerPoint PPT Presentation

Dim imensionality ty Redu eduction: Th Theoretic ical Ana nalysis of Pr Practi tical Mea easu sures

Nova Fandina Hebrew University Joint work with Yair Bartal, Hebrew University Ofer Neiman, Ben Gurion University

1

Outl utline

• Measuring the Quality of Embedding
• in theory : worst case distortion analysis
• in practice: average case distortion measures
• in between: theoretical analysis of practical measures

(for dimensionality reduction methods)

• Our Results
• upper bounds
• lower bounds
• approximating optimal embedding

2

Measuring the Quality ty of Embeddin ing: g: in theory

Basic question in metric embedding theory (informally) Given metric spaces 𝑌 and 𝑍, embed 𝑌 into 𝑍, with small error on the distances How well it can be done? In theory: “well” traditionally means to minimize distortion of the worst pair

3

Definition For an embedding 𝑔: 𝑌 → 𝑍 , for a pair of points 𝑣 ≠ 𝑤 ∈ 𝑌

• 𝑓𝑦𝑞𝑏𝑜𝑡𝑔 𝑣, 𝑤 =

𝑒𝑍 𝑔 𝑣 ,𝑔 𝑤 𝑒𝑌 𝑣,𝑤

, 𝑑𝑝𝑜𝑢𝑠

𝑔 𝑣, 𝑤 = 𝑒𝑌 𝑣,𝑤 𝑒𝑍 𝑔 𝑣 ,𝑔 𝑤

• 𝑒𝑗𝑡𝑢𝑝𝑠𝑢𝑗𝑝𝑜 𝑔 = 𝑛𝑏𝑦𝑣≠𝑤∈𝑌 𝑓𝑦𝑞𝑏𝑜𝑡𝑔 𝑣, 𝑤

⋅ 𝑛𝑏𝑦𝑣≠𝑤∈𝑌{𝑑𝑝𝑜𝑢𝑠

𝑔 (𝑣, 𝑤)}

Mea easuring the Quality ty of Embeddin ing: in pract ctice ce

Demand for the worst case guarantee is too strong: The quality of a method in practical applications is its average performance over all pairs

• Yuval Shavitt and Tomer Tankel. Big-bang simulation for embedding network distances in Euclidean space.

IEEE/ACM Trans. Netw., 12(6), 2004.

• P. Sharma, Z. Xu, S. Banerjee, and S. Lee. Estimating network proximity and
• latency. Computer Communication Review, 36(3), 2006.
• P. J. F. Groenen, R. Mathar, and W. J. Heiser. The majorization approach to multidimensional scaling for

minkowski distances. Journal of Classification, 12(1), 1995.

• J. F. Vera, W. J. Heiser, and A. Murillo. Global optimization in any minkowski metric: A permutation-

translation simulated annealing algorithm for multidimensional scaling. Journal of Classification, 24(2), 2007.

• A. Censi and D. Scaramuzza. Calibration by correlation using metric embedding from nonmetric similarities.

IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(10), 2013.

• C. Lumezanu and N. Spring. Measurement manipulation and space selection in network coordinates. The

28th International Conference on Distributed Computing Systems, 2008.

• S. Chatterjee, B. Neff, and P. Kumar. Instant approximate 1-center on road networks via embeddings. In

Proceedings of the 19th International Conference on Advances in Geographic Information Systems, GIS ’11, 2011.

• S. Lee, Z. Zhang, S. Sahu, and D. Saha. On suitability of Euclidean embedding for host-based network

coordinate systems. IEEE/ACM Trans. Netw., 18(1), 2010.

• L. Chennuru Vankadara and U. von Luxburg. Measures of distortion for machine learning. Advances in Neural

Information Processing Systems, Curran Associates, Inc., 2018.

Just a small sample from googolplex number of such studies

4

Measuring the Quality ty of Embeddin ing: g: in pract ctice ce

Moments of Distortion and Relative Error

For 𝑔: 𝑌 → 𝑍, for a pair 𝑣 ≠ 𝑤 ∈ 𝑌, 𝑒𝑗𝑡𝑢𝑔 𝑣, 𝑤 ≔ max{𝑓𝑦𝑞𝑏𝑜𝑡𝑔 𝑣, 𝑤 , 𝑑𝑝𝑜𝑢𝑠𝑏𝑑𝑢𝑔(𝑣, 𝑤)}

ℓ𝒓-di distortio ion ABN[06]

For 𝑔: 𝑌 → 𝑍, for a distribution Π over pairs of 𝑌, 𝑟 ≥ 1

Relative Error Measure [In many papers]

5

ℓ𝑟

Π -dist(f)= 𝐹𝛲

𝑒𝑗𝑡𝑢𝑔 𝑣, 𝑤

𝑟 1/𝑟

𝑆𝐹𝑁𝑟

(Π) = 𝐹𝛲

|𝑒𝑗𝑡𝑢𝑔 𝑣, 𝑤 − 1|

𝑟 1/𝑟

Measuring the Quality ty of Embeddin ing: g: in pract ctice ce

Additive Distortion Measures [MDS: optimally embed a given finite X into a k-dim Euclidean space, for a given k] For a pair 𝑣 ≠ 𝑤 ∈ 𝑌, 𝑒𝑣𝑤 = 𝑒𝑌 𝑣, 𝑤 , መ 𝑒𝑣𝑤 = 𝑒𝑍 𝑔 𝑣 , 𝑔 𝑤

6

𝐹𝑜𝑓𝑠𝑕𝑧𝑟 𝑔 = EΠ | መ 𝑒𝑣𝑤 − 𝑒𝑣𝑤| 𝑒𝑣𝑤

𝑟 1/𝑟

𝑇𝑢𝑠𝑓𝑡𝑡𝑟 𝑔 = 𝐹Π[|𝑒𝑣𝑤 − መ 𝑒𝑣𝑤|𝑟] 𝐹Π[ 𝑒𝑣𝑤 𝑟]

1/𝑟

𝑇𝑢𝑠𝑓𝑡𝑡𝑟

∗ 𝑔 =

𝐹Π[|𝑒𝑣𝑤 − መ 𝑒𝑣𝑤|𝑟] 𝐹Π[ መ 𝑒𝑣𝑤

𝑟] 1/𝑟

𝑆𝐹𝑁𝑟 𝑔 = EΠ | መ 𝑒𝑣𝑤 − 𝑒𝑣𝑤| min{𝑒𝑣𝑤, መ 𝑒𝑣𝑤}

𝑟 1/𝑟

Measuring the Quality ty of Embeddin ing: g: in pract ctice ce

𝝉-distortion ML motivated, in [VvL18]

“Necessary properties for ML applications”

• translation invariance
• scale invariance
• monotonicity
• robustness (outliers, noise)
• incorporation of probability

7

➢ Almost nothing is known in terms of rigorous analysis

𝜏 − 𝑒𝑗𝑡𝑢(Π)𝑟,𝑠 𝑔 = EΠ 𝑓𝑦𝑏𝑞𝑜𝑡𝑔 𝑣, 𝑤 ℓ𝑠

𝑉 − 𝑓𝑦𝑞𝑏𝑜𝑡 𝑔 − 1 𝑟 1/𝑟

➢ Many heuristics for

• ptimizing these measures
• ℓ𝑠

(U) − 𝑓𝑦𝑞𝑏𝑜𝑡 𝑔 = 𝐹U[(𝑓𝑦𝑞𝑏𝑜𝑡𝑔 𝑣, 𝑤 𝑠)]

• ℓ𝑠

(U) − 𝑑𝑝𝑜𝑢𝑠 𝑔 = 𝐹U[(𝑑𝑝𝑜𝑢𝑠 𝑔 𝑣, 𝑤 𝑠)]

Measuring the Quality ty of Embeddin ing: g: in betw tween

Bridging the gap between theory and practice outlook [CD06] Optimizing is NP-hard for 𝑇𝑢𝑠𝑓𝑡𝑡𝑟 and 𝑙 = 1

8

𝛽(𝑙, 𝑟)-Dimension Reduction Given a dimension bound 𝐥 ≥ 𝟐 and 𝐫 ≥ 𝟐, what is the least 𝛃(𝐥, 𝐫) such that every finite subset of Euclidean space embeds into 𝐥 dim. with 𝐍𝐟𝐛𝐭𝐯𝐬𝐟𝐫 ≤ 𝛃(𝐥, 𝐫)? General Metrics (MDS) For a given finite 𝑌 and 𝑙 ≥ 1, compute the optimal embedding of 𝑌 into k-dim Euclidean space, minimizing a particular 𝑁𝑓𝑏𝑡𝑣𝑠𝑓𝑟. General Metrics: Approximating the Optimal Embedding For a given finite 𝑌 and for 𝑙 ≥ 1, compute an embedding of X into k-dim Euclidean space that approximates the best possible embedding, for a given 𝑁𝑓𝑏𝑡𝑣𝑠𝑓𝑟.

Our ur Resu sult lts: : upper boun

• unds

s previous results

Previous results: worst case distortion JL[84] Every 𝑜-point 𝑌 ∈ ℓ2

𝑒 embeds into ℓ2 𝑙 with distortion 𝑃 𝑜

2 𝑙 (log 𝑜)/𝑙

Mat[90] There is 𝑊 ∈ ℓ2

𝑙+1 such that any 𝑔: 𝑊 → ℓ2 𝑙 must have distortion 𝑜Ω(1/𝑙)

▪ distortion(f) ≤ (ℓ∞-dist)2 ▪ For every 𝑔: 𝑌 → 𝑍 (scalable) there is g: 𝑌 → 𝑍 with ℓ∞-𝐞𝐣𝐭𝐮 𝐡 = 𝐞𝐣𝐭𝐮 𝐠 What about the 𝑁𝑓𝑏𝑡𝑣𝑠𝑓𝑟 guarantees for 𝑟 < ∞?

9

𝛽(𝑙, 𝑟)-Dimension Reduction Given a dimension bound 𝐥 ≥ 𝟐 and 𝐫 ≥ 𝟐, what is the least 𝛃(𝐥, 𝐫) such that every finite subset of Euclidean space embeds into 𝐥 dim. with 𝐍𝐟𝐛𝐭𝐯𝐬𝐟𝐫 ≤ 𝛃(𝐥, 𝐫)?

Our ur Resu esult lts: upper boun bounds s JL transform: IM implementation

The answer to the 𝛽(𝑙, 𝑟)-Dim. Reduction question is, essentially, the JL transform [JL84] Projection onto a random subspace of dim. 𝒍 = 𝑷(𝐦𝐩𝐡 𝒐 /𝝑𝟑), with const. prob. 𝒆𝒋𝒕𝒖 𝒈 = 𝟐 + 𝝑 [IM 98] 𝑈 is a matrix of size 𝑙 × 𝑒 with indep. entries sampled from 𝑂(0,1). The embedding 𝑔: 𝑌 → ℓ2

𝑙 is

𝑔 𝑦 = 1/ 𝑙 ⋅ 𝑈(𝑦)

• The JL transform of IM98 provides constant upper bounds for all 𝑁𝑓𝑏𝑡𝑣𝑠𝑓𝑟

The bounds are almost optimal

• Other popular implementations of JL do not work for ℓ𝑟-dist and for 𝑆𝐹𝑁𝑟
• PCA may produce an embedding of extremely poor quality for all the measures

(this does not happen to the JL)

10

[tight, LN16]

Our ur Resu sult lts: : upper boun

• unds

s other implementations of JL

[Achl03] The entries of T are uniform indep. from {±1} [DKS10,KN10, AL10] Sparse/Fast: particular distr. from {±1,0} Constant bounds cannot be achieved using the above implementations

▪ ℓ𝑟

Π -dist(f) = 𝐹𝛲

𝑒𝑗𝑡𝑢𝑔 𝑣, 𝑤

𝑟 1/𝑟

, 𝑆𝐹𝑁𝑟

(Π) = 𝐹𝛲

|𝑒𝑗𝑡𝑢𝑔 𝑣, 𝑤 − 1|

𝑟 1/𝑟

▪ 𝑒𝑗𝑡𝑢𝑔 𝑣, 𝑤 = max(𝑓𝑦𝑞𝑏𝑜𝑡𝑔 𝑣, 𝑤 , 𝑑𝑝𝑜𝑢𝑠𝑏𝑑𝑢𝑔(𝑣, 𝑤))

➢ 𝑈 𝑓1, … , 𝑓𝑒 = {𝑑𝑝𝑚𝑣𝑛𝑜𝑡 𝑝𝑔 𝑈}. The number of different columns is 𝑡𝑙 < 𝑒

11

Observation If a linear transformation 𝑈: 𝑆𝑒 → 𝑆𝑙 samples its entries form a discrete set of values of size 𝑡 ≤ 𝑒1/𝑙, then applying it on a standard basis of 𝑆𝑒 results in ℓ𝑟-dist, 𝑆𝐹𝑁𝑟 = ∞.

Our ur Resu sult lts: upper boun

• unds

s limitation of PCA

PCA/c-MDS For a given finite 𝑌 ∈ ℓ2

𝑒 and a given integer 𝑙 ≥ 1, computes the

best rank 𝑙- approx. to 𝑌: A projection 𝑄 onto the 𝑙- dim subspace spanned by largest eigenvectors of the covariance matrix, with the smallest σ𝑣∈𝑌 𝑣 − 𝑄 𝑣

2

▪ 𝑔: 𝑌 → ℓ2

𝑙 with optimal σ𝑣≠𝑤∈𝑌 (𝑒𝑣𝑤 2 − መ

𝑒𝑣𝑤

2 ) over all projections

▪ Often misused: “minimizing 𝑇𝑢𝑠𝑓𝑡𝑡2 over all embeddings into 𝑙-dim” ▪ Actually, PCA does not minimize any of the mentioned measures

12

Our ur Resu esult lts: upper bou

• unds

s Bad metric for PCA

• The metric is in 𝑒 dimensional Euclidean space, for any 𝑒 large enough
• Fix some 𝛽 < 1, and 𝑟 ≥ 1
• Consider the standard basis vectors 𝑓1, … , 𝑓𝑒
• For each vector 𝑓𝑗, let 𝑌𝑗 be the set of

1 𝛽𝑗 𝑟

copies of vector 𝛽𝑗 ⋅ 𝑓𝑗, and let 𝑍

𝑗 be the set of the same size of the antipodal vector −𝛽𝑗 ⋅ 𝑓𝑗

13

𝑧 𝑦 𝑨

• 𝑇𝑢𝑠𝑓𝑡𝑡2

2 measure: σ𝑣≠𝑤∈𝑌 𝑒𝑣𝑤− ෠ 𝑒𝑣𝑤

2

σ𝑣≠𝑤∈𝑌 𝑒𝑣𝑤

2

• σ𝑣≠𝑤 𝑒𝑣𝑤

2 ≈ 𝑒 ⋅ 1 𝛽𝑒 2

• pairs between 𝑌𝑗 and 𝑌

𝑘, 𝑗 < 𝑘 contribute: 1 𝛽2𝑗 ⋅ 1 𝛽2𝑘 ⋅ (𝛽2𝑗+𝛽2𝑘) ≤ 1 𝛽2𝑗 ⋅ 1 𝛽2𝑘 ⋅ 2𝛽2𝑗 ≤ 2 1 𝛽2𝑘 1 𝛽2𝑗 ⋅ 1 𝛽2𝑘 ⋅ (𝛽2𝑗+𝛽2𝑘) ≥ 1 𝛽2𝑘

i j 𝛽 𝛽

𝛽2

𝛽3

Our ur Resu esult lts: upper boun bounds s limitation of PCA

෍

𝑣≠𝑤

𝑒𝑣𝑤

2 ≈ 𝑒 ⋅

1 𝛽𝑒

2

➢ PCA projects onto 𝑡𝑞𝑏𝑜{𝑓1, 𝑓2, … , 𝑓𝑙} taking 𝑙 < 0.99𝑒 ⦁ ℓ𝒓-dist/𝑆𝐹𝑁𝑟 = ∞ ➢ The JL embedding has bounded measures: 𝛽 𝑙 → 0, as 𝑙 increases

14

𝑧 𝑦 𝑨

• Error contribution: 𝑗 < 𝑘
• ≈

1 𝛽2𝑗 ⋅ 1 𝛽2𝑘

(𝛽2𝑗+𝛽2𝑘) − 𝛽𝑗 − 𝛽𝑘

2

≈1/α2𝑗 for 𝑗 < 𝑘, in total ≈

1 𝛽𝑒−1 2

• 𝑻𝒖𝒔𝒇𝒕𝒕𝟑 ≤

Τ 𝜷 𝒆𝟐/𝟑

• Error contribution: ≥ 𝑒 − 𝑙

1/𝛽𝑒 2

• 𝑇𝑢𝑠𝑓𝑡𝑡2 ≥ Ω 1
• Is not better than a naïve algo:

any non-expansive embedding i j

2𝛽3

2𝛽2

Our ur Resu esult lts: upper bou

• unds

s moment analysis of JL transform

The bounds are almost tight in most of the ranges of values of 𝑟 Proo

• of

f (for for 𝐫 < 𝐥) For a given dist. Π over pairs of 𝑌, for a given 𝑟 ≥ 1

𝐹

𝑔

ℓ𝑟

(Π) − 𝑒𝑗𝑡𝑢 𝑔 𝑟

= 𝐹

𝑔

𝐹 𝑣,𝑤 ∼Π 𝑒𝑗𝑡𝑢𝑔 𝑣, 𝑤

𝑟

= 𝐹 u,v ∼ Π 𝐹

𝑔

𝑒𝑗𝑡𝑢𝑔 𝑣, 𝑤

𝑟

15

Theorem (Moment analysis of JL transform) There is a map (JL or normalized JL) 𝑔: 𝑌 → ℓ2

𝑙 s.t. for a given 𝑟 ≥ 1 with const. prob.

ℓ𝒓-dist(f) =

1 ≤ 𝑟 < 𝑙 𝑙 ≤ 𝑟 ≤ 𝑙/4 𝑙/4 ≤ 𝑟 ≤ 𝑙 𝑟 = 𝑙 𝑙 ≤ 𝑟 ≤ ∞ 1 + 𝑃 1 𝑙 1 + 𝑃 𝑟 𝑙 − 𝑟 𝑙 𝑙 − 𝑟

𝑃(1/𝑟)

𝑃 log𝑜

1/𝑙

𝑜

𝑃 1 𝑙−1 𝑟

For every 𝑣 ≠ 𝑤 ∈ 𝑌 , estimate 𝐹𝑔 𝑒𝑗𝑡𝑢𝑔 𝑣, 𝑤

𝑟

= 𝐹𝑔 max

𝑔 𝑣 −𝑔 𝑤 𝑣−𝑤

,

𝑣−𝑤 𝑔 𝑣 −𝑔 𝑤 𝑟

Since 𝑔 is a linear map, for any 𝑨 ∈ ℝ𝑒, with 𝑨 = 1 estimate 𝐹𝑔 max 𝑔 𝑨

𝑟, 1 𝑔 𝑨

𝑟

• 𝑔 𝑨 = 1/ 𝑙 ⋅ 𝑈 𝑨 = 1/ 𝑙 ⋅ < 𝑨, 𝑈

1 >, … , < 𝑨, 𝑈 𝑙 > = 1/ 𝑙 (𝑍 1, … , 𝑍 𝑙),

for 𝑍

𝑗 ∼ 𝑂(0,1)

• 𝑔 𝑨

𝑟 = ( 𝑔 𝑨 2)𝑟/2 = 𝑌 𝑙 𝑟/2

, where X ∼ 𝜓𝑙

2

16

• 𝐹

𝑔 max

𝑔 𝑨

𝑟, 1 𝑔 𝑨

𝑟

≤ 𝐹𝑌∼𝜓𝑙

2

Τ 𝑌 𝑙 𝑟/2 + 𝐹𝑌∼𝜓𝑙

2

Τ 𝑙 𝑌 𝑟/2

• 𝐹𝑌∼𝜓𝑙

2

Τ 𝑙 𝑌 𝑟/2 = Gamma function: Γ 𝑢 = ׬

∞ 𝑦𝑢−1𝑓−𝑦 𝑒𝑦, Γ 𝑢 + 1 = 𝑢 Γ(𝑢)

• 𝐹

𝑔

ℓ𝑟 − 𝑑𝑝𝑜𝑢𝑠 𝑔

𝑟

= 1 + 𝑃

𝑟 𝑙−𝑟 𝑟

, for all 𝑟 < 𝑙

• 𝐹

𝑔

ℓ𝑟 − 𝑓𝑦𝑞𝑏𝑜𝑡 𝑔

𝑟

= 1 + 𝑃

𝑟 𝑙 𝑟

, for all 𝑟 ≥ 1

• 𝐹

𝑔 ℓ𝑟 − 𝑒𝑗𝑡𝑢(𝑔) ≤ 2

1 𝑟 1 + 𝑃

𝑟 𝑙−𝑟

≤ 1 +

1 𝑟

1 + 𝑃

𝑟 𝑙−𝑟

= 1 + 𝑃(1/ 𝑙), taking 𝑟 = 𝑙 ◼

17

goes to ∞ , as 𝑟 → 𝑙

Our ur Resu esult lts: upper bou

• unds

s estimates for large values of 𝑟

Proo

• of

f The same as before, estimation of expectation conditioning on the event ∀ 𝑣 ≠ 𝑤 ∈ 𝑌, 𝑑𝑝𝑜𝑢𝑠

𝑔 𝑣, 𝑤 ≤ 𝑜

2 𝑙 (holds with const prob).

Normalize by an appropriate factor (depends on 𝑟 and 𝑙)

18

Theorem (Moment analysis of JL transform) There is a map (normalized JL) 𝑔: 𝑌 → ℓ2

𝑙 s.t. for a given 𝑟 ≥ 1 with const. prob.

ℓ𝒓-dist(f) =

1 ≤ 𝑟 < 𝑙 𝑙 ≤ 𝑟 ≤ 𝑙/4 𝑙/4 ≤ 𝑟 ≤ 𝑙 𝑟 = 𝑙 𝑙 ≤ 𝑟 ≤ ∞ 1 + 𝑃 1 𝑙 1 + 𝑃 𝑟 𝑙 − 𝑟 𝑙 𝑙 − 𝑟

𝑃(1/𝑟)

𝑃 log𝑜

1/𝑙

𝑜

𝑃 1 𝑙−1 𝑟

◼

Our ur Resu sult lts: : upper boun

• unds

s REM and additive measures

▪ All the additive measures ≤ 𝑆𝐹𝑁 ▪ As before, estimate the appropriate integral to get the bound ▪ Tight for 𝑟 ≥ 2 (equilateral space on 𝑜 points, 𝐹𝑜) Based on [Alon90]: lower bound for embedding with 1 + 𝜗 w.c. dist For 1 ≤ 𝑟 < 2 the lower bound is Ω 1/𝑙1/𝑟

20

Theorem (REM and additive measures analysis of JL) There is a map (JL) 𝑔: 𝑌 → 𝑆𝑙, for 𝑙 ≥ 2, s.t. with const. prob. for all 1 ≤ 𝑟 ≤ 𝑙 − 1 (simultaneously) 𝜏-dist, 𝑇𝑢𝑠𝑓𝑡𝑡𝑟, 𝑇𝑢𝑠𝑓𝑡𝑡∗, 𝐹𝑜𝑓𝑠𝑓𝑕𝑧𝑟 𝑔 ≤ 𝑆𝐹𝑁𝑟 𝑔 = 𝑃( Τ 𝑟 𝑙) 𝑆𝐹𝑁𝑟 = 𝐹Π[ 𝑒𝑗𝑡𝑢𝑔 𝑣, 𝑤 − 1

𝑟]

Our ur Resu sult lts: : opti timal l emb mbedding for for 𝑭𝒐 small q

• There is a gap with respect to the upper bounds provided by the JL

embedding

• This is the best we can do for 𝐹𝑜 space:

24

Theorem ➢ Any embedding 𝑔: 𝐹𝑜 → ℓ2

𝑙 must have:

ℓ𝑟- dist 𝑔 = 1 + Ω(𝑟/𝑙) for 1 ≤ 𝑟 ≤ 𝑙 𝐹𝑜𝑓𝑠𝑕𝑧𝑟 𝑔 = Ω Τ (1 𝑙)1/𝑟 for 1 ≤ 𝑟 < 2 For every 𝑙 ≥ 3, for all 1 ≤ 𝑟 ≤ 𝑙, for every distr. Π over pairs of 𝐹𝑜, there is a random map 𝑔: 𝐹𝑜 → ℓ2

𝑙 s.t. with const. prob.

ℓ𝑟 - dist 𝑔 = 1 + 𝑃(𝑟/𝑙)

Our ur Resu sult lts: : opti timal l emb mbedding for for 𝑭𝒐 small 𝑟 < 𝑙

Intuition

⦁ ⦁ ⦁ ⦁ ⦁ ⦁

25

Theorem

For every 𝑙 ≥ 3, for all 1 ≤ 𝑟 ≤ 𝑙, for every distr. Π over pairs of 𝐹𝑜, there is a random 𝑔: 𝐹𝑜 → ℓ2

𝑙 s.t. with const. prob. ℓ𝑟 - dist 𝑔 = 1 + 𝑃(𝑟/𝑙)

𝑓1 𝑓2 𝑓3 … 𝑓𝑙 uniform metric in 𝒐 dim uniform metric in 𝐥 dim 𝑤𝑗 → uniformly chosen 𝑓

𝑘

𝐹𝑜 = {𝑤1, 𝑤2, … , 𝑤𝑜} 𝑤1 𝑤2 𝑤3 𝑤4

𝐹(ℓ𝑟− dist(𝑔))𝑟 = 𝐹 [(𝑒𝑗𝑢𝑡𝑔 𝑣, 𝑤 )𝑟] = Pr 𝑔 𝑣 = 𝑔 𝑤 ⋅ 𝐹 [ 𝑒𝑗𝑢𝑡𝑔 𝑣, 𝑤 )𝑟 𝑔 𝑣 = 𝑔(𝑤)] + Pr 𝑔 𝑣 ≠ 𝑔 𝑤 ⋅ 1 ∞

Our ur Resu sult lts: opti timal l emb mbedding for for 𝑭𝒐 small 𝑟 < 𝑙

26

• each 𝑇[𝑘] has radius 𝑠

𝑘 = 1/ 2

• Each sphere is of dim. 𝑒, 𝑇 𝑘 ⊂ ℓ2

𝑟2

• ⧣ spheres 𝑙/𝑟2

… 𝑇[1] 𝑇[2]

𝑇[𝑙/𝑟2]

Algorithm: map 𝐺: 𝐹𝑜 → ℓ2

𝑙

∀ 𝒘 ∈ 𝑭𝒐 ⦁ ind. & uniformly choose a sphere 𝑇 𝑘 ⦁ place 𝑤 ind. & uniformly on 𝑇[𝑘]; 𝐺

𝑘 (𝑤) → ℓ2 𝑟2

⦁ 𝐺 𝑤 →

00000 00000 𝐺

𝑘(𝑤)𝐺

00000

▪ 𝐹(ℓ𝑟− dist(𝑔))𝑟 = 𝐹 [(𝑒𝑗𝑢𝑡𝑔 𝑣, 𝑤 )𝑟] = Pr 𝑔 𝑣 = 𝑔 𝑤 ⋅ 𝐹 [ 𝑒𝑗𝑢𝑡𝑔 𝑣, 𝑤 )𝑟 𝑔 𝑣 = 𝑔(𝑤)] + Pr 𝑔 𝑣 ≠ 𝑔 𝑤 ⋅ 1 ▪ If 𝑤 ∈ 𝑇 𝑘 and 𝑣 ∈ 𝑇 𝑗 ⟹ 𝐺 𝑤 − 𝐺 𝑣

2 =

𝐺

𝑘 𝑤 ԡ2 − ԡ𝐺 𝑘 𝑣 2 = 1/2 +1/2 = 1

2 ( Τ 𝑒 (𝑒 − 𝑟))𝑟/2 Τ 𝑟2 𝑙 ≤ [𝑒=𝑟2] 1 + ( Τ 𝑟2 𝑙) ⋅ 𝑑𝑝𝑜𝑡𝑢 ▪ Taking 1/𝑟 on both sides, ℓ𝑟 - dist 𝑔 = 1 + O(q/k).

Our ur Resu sult lts: low

• wer boun
• unds

phase transition at 𝑟 = 𝑙

Proof

• It is enough to show that for any non-expansive 𝑔: 𝐹𝑜 → ℓ2

𝑙, ℓ𝑙-dist 𝑔 ≥ Ω log 𝑜

1 𝑙

𝑙

• Claim: if for any non-expansive 𝐺: 𝐹𝑜 → 𝑍, ℓ𝑙-dist 𝐺 ≥ 𝐸 𝑙, 𝑜 , then

for any 𝑔: 𝐹𝑜 → 𝑍, ℓ𝑙-dist 𝑔 ≥ 𝑑𝑝𝑜𝑡𝑢 ⋅ 𝐸 𝑙, 𝑜 .

• Since ℓ2

𝑙 ∼ ℓ∞ 𝑙 with distortion 𝑙, it is enough to prove for any non-expansive

𝑔: 𝐹𝑜 → ℓ∞

𝑙 has ℓ𝑙-dist 𝑔 ≥ Ω log 𝑜 1/𝑙

28

Theorem

• For any 𝑙 ≥ 1, any embedding 𝑔: 𝐹𝑜 → ℓ2

𝑙 has ℓ𝑙-dist 𝑔 = Ω log 𝑜

1 𝑙

𝑙1/4

.

• For any q > 𝑙 ≥ 1, any embedding 𝑔: 𝐹𝑜 → ℓ2

𝑙 has ℓ𝑟-dist 𝑔 = Ω(𝑜

1 2𝑙− 1 2𝑟) .

Our ur Resu sult lts: (alm lmost) t) opti timal l low

• wer boun
• und for 𝑟 > 𝑙

Claim: For every non-expansive 𝑔: 𝐹𝑜 → ℓ∞

𝑙 , ℓ𝑙-dist 𝑔 ≥ Ω log 𝑜 1/𝑙 .

Proof

• Basically, embedding (non-expansively) 𝐹𝑜 into ℓ∞

𝑙 is

as embedding it (non-expansively) into a family of certain tree metrics 𝟑-HST metrics of degree 𝒍 – a family of all rooted trees on 𝑜 leaves, with each node having at most 2𝑙 children. The nodes have labels, decreasing by a factor of 2 along the paths from the root to each leaf. The root’s label is 1. Each tree defines a metric over the set of its leaves: 𝑒𝑗𝑡𝑢 𝑣, 𝑤 = 𝑚𝑏𝑐𝑓𝑚(𝑚𝑑𝑏(𝑣, 𝑤))

29

1 Τ 1 2 Τ 1 2

Τ 1 2 Τ 1 4

Our ur Resu sult lts: (alm lmost) t) opti timal l low

• wer boun
• und for 𝑟 > 𝑙
• Every non-expansive embedding 𝑔: 𝐹𝑜 → ℓ∞

𝑙 can be modified to the one that

embeds 𝐹𝑜 into a 2-HST tree from the family of degree 𝑙, with better ℓ𝑙- distortion For 𝑙 = 2, r ecursively construct 2-HST tree, of degree 4

30

1 1

Τ 1 2 Τ 1 2

Τ 1 4

1

Our ur Resu esult lts: (alm almost) t) opti timal l low

• wer bou
• und for 𝑟 = 𝑙

So, it is enough to prove that: Proof By induction on 𝑜, showing that the best tree is the perfectly balanced (each node has exactly 2𝑙 children). Computing its weight completes the proof. ◼

31

Claim Any non-expansive embedding 𝑔 of 𝐹𝑜 into a family of 2-HST’s

• f degree 𝑙, has (ℓ𝑙−𝑒𝑗𝑡𝑢(𝑔))𝑙 ≥ Ω(log 𝑜).

Our ur Resu sult lts: approxi ximati ting g opti timal emb mbedding

[ABN06] Every finite X embeds into ℓ𝑞

𝑃𝑞(log 𝑜), with ℓ𝑟-distortion 𝑃(𝑟/𝑞).

Gives 𝑃(𝑟) approximation to the optimum under ℓ𝑟-distortion [HIL03] 2-approx. to 𝑇𝑢𝑠𝑓𝑡𝑡∞, for embedding into 𝑙 = 1 dim [Bado03] 𝑃(1)-approx. to 𝑇𝑢𝑠𝑓𝑡𝑡∞, for embedding into 𝑙 = 2 dim, under 𝑚1norm [Dham04] 𝑃(log1/𝑟 𝑜)-approx. to 𝑇𝑢𝑠𝑓𝑡𝑡𝑟, for embedding into 𝑙 = 1 dim

32

General Metrics: Approximating the Optimal Embedding For a given finite 𝑌 and for 𝑙 ≥ 1, compute an embedding of X into k-dim Euclidean space that approximates the best possible embedding, for a given 𝑁𝑓𝑏𝑡𝑣𝑠𝑓𝑟.

Our ur Resu sult lts: : approxi ximati ting g opti timal emb mbedding

For a given 𝑌 and q, 𝑙 ≥ 1, for an objective measure 𝑃𝑐𝑘𝑟 ∶ 𝑃𝑄𝑈: 𝑌 → ℓ2

𝑙 is an optimal embedding for the measure 𝑃𝑐𝑘𝑟

Proo

• of

f Outl tlin ine: (For Stress)

• Find an optimal embedding 𝑔 without constraining the dimension
• Reduce dimension with JL (IM implementation) and show this works

33

Theorem For a given finite 𝑌, for a given 𝑙 ≥ 3 and 2 ≤ 𝑟 ≤ 𝑙 − 1, there is a randomized polytime algorithm that computes an embedding F: 𝑌 → ℓ2

𝑙 s.t. with const. prob.

• 𝑚𝑟-𝑒𝑗𝑡𝑢 𝐺 =

1 + 𝑃

𝟐 𝒍 + 𝒓 𝒍−𝒓

⋅ 𝑷𝑸𝑼

• 𝑃𝑐𝑘𝑟

𝛲 𝐺 = 𝑃 𝑃𝑐𝑘𝑟 𝑷𝑸𝑼

+ 𝑃 𝑟/𝑙 , for all the rest objective measures

Our ur Resu sult lts: approxi ximati ting g opti timal emb mbedding proof outline

Find an optimal embedding 𝑔 without constraining the dimension

• [LLR95]: SDP computes a map with optimal worst case distortion

For 𝑌 = {𝑦0, … , 𝑦𝑜−1}, for each pair, variable 𝑨𝑗𝑘 = 𝑔 𝑦𝑗 − 𝑔 𝑦𝑘

2,

𝑕𝑗𝑘 = 1/2(𝑨0𝑗 + 𝑨0𝑘 − 𝑨𝑗𝑘) represents < 𝑔 𝑦𝑗 , 𝑔(𝑦𝑘) >, 𝑔 𝑦0 = 0 𝑨𝑗𝑘 are Euclidean distances iff matrix 𝐻 𝑗, 𝑘 ≔ 𝑕𝑗𝑘 is PSD

• Convex objective function for 𝑇𝑢𝑠𝑓𝑡𝑡𝑟: for 𝑟 ≥ 2

min σ0<𝑗<𝑜

𝑨𝑗𝑘 𝑒𝑗𝑘 − 1 2 𝑟/2

, s.t. 𝑨𝑗𝑘 ≥ 0, 𝐻 is PSD

34

Our ur Resu sult lts: approxi ximati ting g opti timal emb mbedding proof outline

• Apply JL 𝑕: 𝑔 𝑌 → ℓ2

𝑙

• 𝐺 ≔ 𝑕 ∘ 𝑔
• Jl embedding is as in the claim, thus 𝐹 𝐹𝑜𝑓𝑠𝑕𝑧𝑟 𝐺

≤ 𝑑 ⋅ 𝑷𝑸𝑼 + 𝑃

𝑟 𝑙

◼

35

Claim If 𝑔: 𝑌 → 𝑍 is some embedding and 𝑕: 𝑍 → 𝑎 is a random map that has 𝐹 𝑓𝑦𝑞𝑏𝑜𝑡𝑕 𝑣, 𝑤 = 𝐵, and 𝐹 𝑑𝑝𝑜𝑢𝑠

𝑕 𝑣, 𝑤

= 𝐶, for all pairs in 𝑌, then 𝐹 𝐹𝑜𝑓𝑠𝑕𝑧𝑟 𝑕 ∘ 𝑔 ≤ 4 ⋅ 𝐹𝑜𝑓𝑠𝑕𝑧𝑟 𝑔 ⋅ 𝐹 𝐹𝑜𝑓𝑠𝑕𝑧𝑟 𝑕

1 𝑟 + 4 ⋅ 𝐹 𝐹𝑜𝑓𝑠𝑕𝑧𝑟 𝑕 1 𝑟

Conc nclu lusions / Thank k you

• u slide

➢ We initiate theoretical study of measurement criteria widely used in practical applications ➢ We give theoretical bounds for these criteria, by showing that the JL random projection is, essentially, the optimal tool for dimensionality reduction ➢ Our bounds result in approximate algorithm for embedding any finite metric into 𝑙-dim space, with proven approximation ratio guarantees ➢ One of the central open questions arising from our work : close the gap for values of 𝑟 ≤ 𝑙, and 𝑟 < 2 ➢ Thank you!

36