Tracking the Norm with ` 2 Constant Update Time Chi-Ning Chou - - PowerPoint PPT Presentation

Tracking the Norm with Constant Update Time

Chi-Ning Chou Zhixian Lei Preetum Nakkiran Harvard University APPROX 2019

`2

• 1

Streaming Algorithms

2

Streaming Algorithms

• Input: A stream of inputs from the alphabet set.

2

Streaming Algorithms

• Input: A stream of inputs from the alphabet set.

Example: . a1, a2, . . . , am ∈ [n]

2

Streaming Algorithms

• Input: A stream of inputs from the alphabet set.

Example: .

• Output: Some statistics of the inputs.

a1, a2, . . . , am ∈ [n]

2

Streaming Algorithms

• Input: A stream of inputs from the alphabet set.

Example: .

• Output: Some statistics of the inputs.

Example: # of distinct elements in the input stream. a1, a2, . . . , am ∈ [n]

2

Streaming Algorithms

• Input: A stream of inputs from the alphabet set.

Example: .

• Output: Some statistics of the inputs.

Example: # of distinct elements in the input stream.

• Frequency Vector: For each , define

t ∈ [m], i ∈ [n]

f (t)

i

= | {t0 ∈ [t] : at0 = i} | .

a1, a2, . . . , am ∈ [n]

2

Streaming Algorithms

• Input: A stream of inputs from the alphabet set.

Example: .

• Output: Some statistics of the inputs.

Example: # of distinct elements in the input stream.

• Frequency Vector: For each , define

Example: norm = # of distinct elements; norm = t. t ∈ [m], i ∈ [n]

f (t)

i

= | {t0 ∈ [t] : at0 = i} | .

a1, a2, . . . , am ∈ [n]

`0

`1

2

Streaming Algorithms

• Input: A stream of inputs from the alphabet set.

Example: .

• Output: Some statistics of the inputs.

Example: # of distinct elements in the input stream.

• Frequency Vector: For each , define

Example: norm = # of distinct elements; norm = t.

• Example: and .

t ∈ [m], i ∈ [n]

f (t)

i

= | {t0 ∈ [t] : at0 = i} | .

a1, a2, . . . , am ∈ [n]

`0

`1

2

n = 5

m = 10

Streaming Algorithms

• Input: A stream of inputs from the alphabet set.

Example: .

• Output: Some statistics of the inputs.

Example: # of distinct elements in the input stream.

• Frequency Vector: For each , define

Example: norm = # of distinct elements; norm = t.

• Example: and .

t ∈ [m], i ∈ [n]

f (t)

i

= | {t0 ∈ [t] : at0 = i} | .

a1, a2, . . . , am ∈ [n]

`0

`1

2

n = 5

m = 10

1

1

f (1) =

Streaming Algorithms

• Input: A stream of inputs from the alphabet set.

Example: .

• Output: Some statistics of the inputs.

Example: # of distinct elements in the input stream.

• Frequency Vector: For each , define

Example: norm = # of distinct elements; norm = t.

• Example: and .

t ∈ [m], i ∈ [n]

f (t)

i

= | {t0 ∈ [t] : at0 = i} | .

a1, a2, . . . , am ∈ [n]

`0

`1

2

n = 5

m = 10

1 2

f (2) =

1 1

Streaming Algorithms

• Input: A stream of inputs from the alphabet set.

Example: .

• Output: Some statistics of the inputs.

Example: # of distinct elements in the input stream.

• Frequency Vector: For each , define

Example: norm = # of distinct elements; norm = t.

• Example: and .

t ∈ [m], i ∈ [n]

f (t)

i

= | {t0 ∈ [t] : at0 = i} | .

a1, a2, . . . , am ∈ [n]

`0

`1

2

n = 5

m = 10

1 2 4

f (3) =

1 1 1

Streaming Algorithms

• Input: A stream of inputs from the alphabet set.

Example: .

• Output: Some statistics of the inputs.

Example: # of distinct elements in the input stream.

• Frequency Vector: For each , define

Example: norm = # of distinct elements; norm = t.

• Example: and .

t ∈ [m], i ∈ [n]

f (t)

i

= | {t0 ∈ [t] : at0 = i} | .

a1, a2, . . . , am ∈ [n]

`0

`1

2

n = 5

m = 10

1 2 4 2

f (4) =

1 2 1

Streaming Algorithms

• Input: A stream of inputs from the alphabet set.

Example: .

• Output: Some statistics of the inputs.

Example: # of distinct elements in the input stream.

• Frequency Vector: For each , define

Example: norm = # of distinct elements; norm = t.

• Example: and .

t ∈ [m], i ∈ [n]

f (t)

i

= | {t0 ∈ [t] : at0 = i} | .

a1, a2, . . . , am ∈ [n]

`0

`1

2

n = 5

m = 10

1 2 4 2 5

f (5) =

1 2 1 1

Streaming Algorithms

• Input: A stream of inputs from the alphabet set.

Example: .

• Output: Some statistics of the inputs.

Example: # of distinct elements in the input stream.

• Frequency Vector: For each , define

Example: norm = # of distinct elements; norm = t.

• Example: and .

t ∈ [m], i ∈ [n]

f (t)

i

= | {t0 ∈ [t] : at0 = i} | .

a1, a2, . . . , am ∈ [n]

`0

`1

2

n = 5

m = 10

1 2 4 2 5 2

f (6) =

1 3 1 1

Streaming Algorithms

• Input: A stream of inputs from the alphabet set.

Example: .

• Output: Some statistics of the inputs.

Example: # of distinct elements in the input stream.

• Frequency Vector: For each , define

Example: norm = # of distinct elements; norm = t.

• Example: and .

t ∈ [m], i ∈ [n]

f (t)

i

= | {t0 ∈ [t] : at0 = i} | .

a1, a2, . . . , am ∈ [n]

`0

`1

2

n = 5

m = 10

1 2 4 2 5 2 1

f (7) =

2 3 1 1

Streaming Algorithms

• Input: A stream of inputs from the alphabet set.

Example: .

• Output: Some statistics of the inputs.

Example: # of distinct elements in the input stream.

• Frequency Vector: For each , define

Example: norm = # of distinct elements; norm = t.

• Example: and .

t ∈ [m], i ∈ [n]

f (t)

i

= | {t0 ∈ [t] : at0 = i} | .

a1, a2, . . . , am ∈ [n]

`0

`1

2

n = 5

m = 10

1 2 4 2 5 2 1 1

f (8) =

3 3 1 1

Streaming Algorithms

• Input: A stream of inputs from the alphabet set.

Example: .

• Output: Some statistics of the inputs.

Example: # of distinct elements in the input stream.

• Frequency Vector: For each , define

Example: norm = # of distinct elements; norm = t.

• Example: and .

t ∈ [m], i ∈ [n]

f (t)

i

= | {t0 ∈ [t] : at0 = i} | .

a1, a2, . . . , am ∈ [n]

`0

`1

2

n = 5

m = 10

1 2 4 2 5 2 1 1 5

f (9) =

3 3 1 2

Streaming Algorithms

• Input: A stream of inputs from the alphabet set.

Example: .

• Output: Some statistics of the inputs.

Example: # of distinct elements in the input stream.

• Frequency Vector: For each , define

Example: norm = # of distinct elements; norm = t.

• Example: and .

t ∈ [m], i ∈ [n]

f (t)

i

= | {t0 ∈ [t] : at0 = i} | .

a1, a2, . . . , am ∈ [n]

`0

`1

2

n = 5

m = 10

1 2 4 2 5 2 1 1 5 2

f (10) =

3 4 1 2

Streaming Algorithms

• Input: A stream of inputs from the alphabet set.

Example: .

• Output: Some statistics of the inputs.

Example: # of distinct elements in the input stream.

• Frequency Vector: For each , define

Example: norm = # of distinct elements; norm = t.

• Applications: Database optimization, network traffic etc.

t ∈ [m], i ∈ [n]

f (t)

i

= | {t0 ∈ [t] : at0 = i} | .

a1, a2, . . . , am ∈ [n]

`0

`1

3

Streaming Algorithms

• Input: A stream of inputs from the alphabet set.

Example: .

• Output: Some statistics of the inputs.

Example: # of distinct elements in the input stream.

• Frequency Vector: For each , define

Example: norm = # of distinct elements; norm = t.

• Applications: Database optimization, network traffic etc.
• Goal: Randomized algorithms using sublinear space.

t ∈ [m], i ∈ [n]

f (t)

i

= | {t0 ∈ [t] : at0 = i} | .

a1, a2, . . . , am ∈ [n]

`0

`1

3

Streaming Algorithms

• Input: A stream of inputs from the alphabet set.

Example: .

• Output: Some statistics of the inputs.

Example: # of distinct elements in the input stream.

• Frequency Vector: For each , define

Example: norm = # of distinct elements; norm = t.

• Applications: Database optimization, network traffic etc.
• Goal: Randomized algorithms using sublinear space.

t ∈ [m], i ∈ [n]

f (t)

i

= | {t0 ∈ [t] : at0 = i} | .

a1, a2, . . . , am ∈ [n]

`0

`1

3

Deterministic algorithm: space. Θ(min{m, n})

Streaming Algorithms

• Input: A stream of inputs from the alphabet set.

Example: .

• Output: Some statistics of the inputs.

Example: # of distinct elements in the input stream.

• Frequency Vector: For each , define

Example: norm = # of distinct elements; norm = t.

• Applications: Database optimization, network traffic etc.
• Goal: Randomized algorithms using sublinear space.

t ∈ [m], i ∈ [n]

f (t)

i

= | {t0 ∈ [t] : at0 = i} | .

a1, a2, . . . , am ∈ [n]

`0

`1

3

Faster time!? Deterministic algorithm: space. Θ(min{m, n})

Estimation

4

`2

Estimation

• Goal: Estimating the norm of the frequency vector in

sublinear space.

4

`2

`2

Estimation

• Goal: Estimating the norm of the frequency vector in

sublinear space.

• -One-shot estimation: Output s.t.

4

`2

`2

σm

Pr h

• m kf (m)k2

2

• > ✏kf (m)k2

2

i  .

(✏, )

Estimation

• Goal: Estimating the norm of the frequency vector in

sublinear space.

• -One-shot estimation: Output s.t.
• -Weak tracking: Output s.t.

4

`2

`2

σm

Pr h

• m kf (m)k2

2

• > ✏kf (m)k2

2

i  .

(✏, )

Pr h 9t∈[m]

• t kf (t)k2

2

• > ✏kf (m)k2

2

i  .

σ1, σ2, . . . , σm

(✏, )

Estimation

• Goal: Estimating the norm of the frequency vector in

sublinear space.

• -One-shot estimation: Output s.t.
• -Weak tracking: Output s.t.
• -Strong tracking: Output s.t.

4

`2

`2

σm

Pr h

• m kf (m)k2

2

• > ✏kf (m)k2

2

i  .

(✏, )

Pr h 9t∈[m]

• t kf (t)k2

2

• > ✏kf (m)k2

2

i  .

σ1, σ2, . . . , σm

(✏, )

Pr h 9t∈[m]

• t kf (t)k2

2

• > ✏kf (t)k2

2

i  .

σ1, σ2, . . . , σm

(✏, )

Estimation

• Goal: Estimating the norm of the frequency vector in

sublinear space.

• -One-shot estimation: Output s.t.
• -Weak tracking: Output s.t.
• -Strong tracking: Output s.t.

4

`2

`2

Strong tracking => Weak tracking => One-shot σm

Pr h

• m kf (m)k2

2

• > ✏kf (m)k2

2

i  .

(✏, )

Pr h 9t∈[m]

• t kf (t)k2

2

• > ✏kf (m)k2

2

i  .

σ1, σ2, . . . , σm

(✏, )

Pr h 9t∈[m]

• t kf (t)k2

2

• > ✏kf (t)k2

2

i  .

σ1, σ2, . . . , σm

(✏, )

Estimation

• Goal: Estimating the norm of the frequency vector in

sublinear space.

• -One-shot estimation: Output s.t.
• -Weak tracking: Output s.t.
• -Strong tracking: Output s.t.

4

`2

`2

Strong tracking => Weak tracking => One-shot σm

Pr h

• m kf (m)k2

2

• > ✏kf (m)k2

2

i  .

(✏, )

Pr h 9t∈[m]

• t kf (t)k2

2

• > ✏kf (m)k2

2

i  .

σ1, σ2, . . . , σm

(✏, )

Pr h 9t∈[m]

• t kf (t)k2

2

• > ✏kf (t)k2

2

i  .

σ1, σ2, . . . , σm

(✏, )

5

Linear Sketch

5

Linear Sketch

• Linear sketch is a special class of streaming algorithms.


5

Linear Sketch

• Linear sketch is a special class of streaming algorithms.


Sketching matrix Π

}

k ⌧ n

}

n

5

Linear Sketch

• Linear sketch is a special class of streaming algorithms.


Sketching matrix Π

Sketching vector Πf (t)

}

k ⌧ n

}

n

5

Linear Sketch

• Linear sketch is a special class of streaming algorithms.


Sketching matrix Π

Sketching vector Πf (t)

1 -1 1 -1 1 1 -1 -1 -1 1 1 -1 -1 1 1

• 1 -1 1

1 -1 1 -1 -1 1 -1 -1 -1 -1 -1 1 1 1 -1 1 1 1 -1 -1 -1 -1 1 -1 1 1 1

• 1 1 -1 1

1 1 -1 -1 -1 -1 1 -1 -1 -1 -1 1 -1 -1 1 -1 1 1 -1 1 -1 -1 1 1 -1 -1

1 √ 5

AMS Sketch [Alon-Matias-Szegedy 96]

}

k ⌧ n

}

n

5

Linear Sketch

• Linear sketch is a special class of streaming algorithms.


Sketching matrix Π

Sketching vector Πf (t)

• 1
• 1

1 1

• 1

1 -1 1 -1 1 1 -1 -1 -1 1 1 -1 -1 1 1

• 1 -1 1

1 -1 1 -1 -1 1 -1 -1 -1 -1 -1 1 1 1 -1 1 1 1 -1 -1 -1 -1 1 -1 1 1 1

• 1 1 -1 1

1 1 -1 -1 -1 -1 1 -1 -1 -1 -1 1 -1 -1 1 -1 1 1 -1 1 -1 -1 1 1 -1 -1

1 √ 5

AMS Sketch [Alon-Matias-Szegedy 96]

}

k ⌧ n

}

n

2

5

Linear Sketch

• Linear sketch is a special class of streaming algorithms.


Sketching matrix Π

Sketching vector Πf (t)

• 1
• 1

1 1

• 1
• 2
• 2
• 2

1 -1 1 -1 1 1 -1 -1 -1 1 1 -1 -1 1 1

• 1 -1 1

1 -1 1 -1 -1 1 -1 -1 -1 -1 -1 1 1 1 -1 1 1 1 -1 -1 -1 -1 1 -1 1 1 1

• 1 1 -1 1

1 1 -1 -1 -1 -1 1 -1 -1 -1 -1 1 -1 -1 1 -1 1 1 -1 1 -1 -1 1 1 -1 -1

1 √ 5

AMS Sketch [Alon-Matias-Szegedy 96]

}

k ⌧ n

}

n

2 8

5

Linear Sketch

• Linear sketch is a special class of streaming algorithms.


Sketching matrix Π

Sketching vector Πf (t)

• 1
• 1

1 1

• 1
• 2
• 2
• 2
• 3
• 1

1 1

• 1

1 -1 1 -1 1 1 -1 -1 -1 1 1 -1 -1 1 1

• 1 -1 1

1 -1 1 -1 -1 1 -1 -1 -1 -1 -1 1 1 1 -1 1 1 1 -1 -1 -1 -1 1 -1 1 1 1

• 1 1 -1 1

1 1 -1 -1 -1 -1 1 -1 -1 -1 -1 1 -1 -1 1 -1 1 1 -1 1 -1 -1 1 1 -1 -1

1 √ 5

AMS Sketch [Alon-Matias-Szegedy 96]

}

k ⌧ n

}

n

2 8 4

5

Linear Sketch

• Linear sketch is a special class of streaming algorithms.


Sketching matrix Π

Sketching vector Πf (t)

• 1
• 1

1 1

• 1
• 2
• 2
• 2
• 3
• 1

1 1

• 1
• 4
• 2

2 2

• 2

1 -1 1 -1 1 1 -1 -1 -1 1 1 -1 -1 1 1

• 1 -1 1

1 -1 1 -1 -1 1 -1 -1 -1 -1 -1 1 1 1 -1 1 1 1 -1 -1 -1 -1 1 -1 1 1 1

• 1 1 -1 1

1 1 -1 -1 -1 -1 1 -1 -1 -1 -1 1 -1 -1 1 -1 1 1 -1 1 -1 -1 1 1 -1 -1

1 √ 5

AMS Sketch [Alon-Matias-Szegedy 96]

}

k ⌧ n

}

n

2 8 4 2

5

Linear Sketch

• Linear sketch is a special class of streaming algorithms.


Sketching matrix Π

Sketching vector Πf (t)

• 1
• 1

1 1

• 1
• 2
• 2
• 2
• 3
• 1

1 1

• 1
• 4
• 2

2 2

• 2
• 3
• 3

3 3

• 3

1 -1 1 -1 1 1 -1 -1 -1 1 1 -1 -1 1 1

• 1 -1 1

1 -1 1 -1 -1 1 -1 -1 -1 -1 -1 1 1 1 -1 1 1 1 -1 -1 -1 -1 1 -1 1 1 1

• 1 1 -1 1

1 1 -1 -1 -1 -1 1 -1 -1 -1 -1 1 -1 -1 1 -1 1 1 -1 1 -1 -1 1 1 -1 -1

1 √ 5

AMS Sketch [Alon-Matias-Szegedy 96]

}

k ⌧ n

}

n

2 8 4 2 5

6

Linear Sketch

• Linear sketch is a special class of streaming algorithms.


Sketching matrix Π

Sketching vector Πf (t)

1 -1 1 -1 1 1 -1 -1 -1 1 1 -1 -1 1 1

• 1 -1 1

1 -1 1 -1 -1 1 -1 -1 -1 -1 -1 1 1 1 -1 1 1 1 -1 -1 -1 -1 1 -1 1 1 1

• 1 1 -1 1

1 1 -1 -1 -1 -1 1 -1 -1 -1 -1 1 -1 -1 1 -1 1 1 -1 1 -1 -1 1 1 -1 -1

1 √ 5

AMS Sketch [Alon-Matias-Szegedy 96]

}

k ⌧ n

}

n

6

Linear Sketch

• Linear sketch is a special class of streaming algorithms.


• Space complexity: , truly random


, pseudo-random

Sketching matrix Π

Sketching vector Πf (t)

1 -1 1 -1 1 1 -1 -1 -1 1 1 -1 -1 1 1

• 1 -1 1

1 -1 1 -1 -1 1 -1 -1 -1 -1 -1 1 1 1 -1 1 1 1 -1 -1 -1 -1 1 -1 1 1 1

• 1 1 -1 1

1 1 -1 -1 -1 -1 1 -1 -1 -1 -1 1 -1 -1 1 -1 1 1 -1 1 -1 -1 1 1 -1 -1

1 √ 5

O(kn)

O(k log n)

{

AMS Sketch [Alon-Matias-Szegedy 96]

}

k ⌧ n

}

n

6

Linear Sketch

• Linear sketch is a special class of streaming algorithms.


• Space complexity: , truly random


, pseudo-random

Sketching matrix Π

Sketching vector Πf (t)

1 -1 1 -1 1 1 -1 -1 -1 1 1 -1 -1 1 1

• 1 -1 1

1 -1 1 -1 -1 1 -1 -1 -1 -1 -1 1 1 1 -1 1 1 1 -1 -1 -1 -1 1 -1 1 1 1

• 1 1 -1 1

1 1 -1 -1 -1 -1 1 -1 -1 -1 -1 1 -1 -1 1 -1 1 1 -1 1 -1 -1 1 1 -1 -1

1 √ 5

O(kn)

O(k log n)

{

AMS Sketch [Alon-Matias-Szegedy 96]

}

k ⌧ n

}

n

Can be even better

6

Linear Sketch

• Linear sketch is a special class of streaming algorithms.


• Space complexity: , truly random


, pseudo-random

• AMS sketch: for one-shot [Alon-Matias-Szegedy 96]

and for weak tracking [Braverman-Chestnut-Ivkin-Nelson-Wang-

Woodruff 17]. Sketching matrix Π

Sketching vector Πf (t)

1 -1 1 -1 1 1 -1 -1 -1 1 1 -1 -1 1 1

• 1 -1 1

1 -1 1 -1 -1 1 -1 -1 -1 -1 -1 1 1 1 -1 1 1 1 -1 -1 -1 -1 1 -1 1 1 1

• 1 1 -1 1

1 1 -1 -1 -1 -1 1 -1 -1 -1 -1 1 -1 -1 1 -1 1 1 -1 1 -1 -1 1 1 -1 -1

1 √ 5

O(kn)

O(k log n)

{

AMS Sketch [Alon-Matias-Szegedy 96]

}

k ⌧ n

}

n

k = O(✏−2)

Can be even better

7

Update Time

7

Update Time

• Update time complexity for a linear sketch algorithm is the

number of field operations needed in each update.

7

Update Time

• Update time complexity for a linear sketch algorithm is the

number of field operations needed in each update.

• E.g., AMS sketch has update time complexity.



 
 Θ(k) = Θ(✏−2)

1 -1 1 -1 1 1 -1 -1 -1 1 1 -1 -1 1 1

• 1 -1 1

1 -1 1 -1 -1 1 -1 -1 -1 -1 -1 1 1 1 -1 1 1 1 -1 -1 -1 -1 1 -1 1 1 1

• 1 1 -1 1

1 1 -1 -1 -1 -1 1 -1 -1 -1 -1 1 -1 -1 1 -1 1 1 -1 1 -1 -1 1 1 -1 -1

Π =

}

k ⌧ n

}

n

7

Update Time

• Update time complexity for a linear sketch algorithm is the

number of field operations needed in each update.

• E.g., AMS sketch has update time complexity.


• Application: Packet passing problem [Krishnamurthy-Sen-

Zhang-Chen 03]

Θ(k) = Θ(✏−2)

1 -1 1 -1 1 1 -1 -1 -1 1 1 -1 -1 1 1

• 1 -1 1

1 -1 1 -1 -1 1 -1 -1 -1 -1 -1 1 1 1 -1 1 1 1 -1 -1 -1 -1 1 -1 1 1 1

• 1 1 -1 1

1 1 -1 -1 -1 -1 1 -1 -1 -1 -1 1 -1 -1 1 -1 1 1 -1 1 -1 -1 1 1 -1 -1

Π =

}

k ⌧ n

}

n

7

Update Time

• Update time complexity for a linear sketch algorithm is the

number of field operations needed in each update.

• E.g., AMS sketch has update time complexity.


• Application: Packet passing problem [Krishnamurthy-Sen-

Zhang-Chen 03]

Θ(k) = Θ(✏−2)

1 -1 1 -1 1 1 -1 -1 -1 1 1 -1 -1 1 1

• 1 -1 1

1 -1 1 -1 -1 1 -1 -1 -1 -1 -1 1 1 1 -1 1 1 1 -1 -1 -1 -1 1 -1 1 1 1

• 1 1 -1 1

1 1 -1 -1 -1 -1 1 -1 -1 -1 -1 1 -1 -1 1 -1 1 1 -1 1 -1 -1 1 1 -1 -1

Π =

}

k ⌧ n

}

n

7

Update Time

• Update time complexity for a linear sketch algorithm is the

number of field operations needed in each update.

• E.g., AMS sketch has update time complexity.


• Application: Packet passing problem [Krishnamurthy-Sen-

Zhang-Chen 03]

Θ(k) = Θ(✏−2)

1 -1 1 -1 1 1 -1 -1 -1 1 1 -1 -1 1 1

• 1 -1 1

1 -1 1 -1 -1 1 -1 -1 -1 -1 -1 1 1 1 -1 1 1 1 -1 -1 -1 -1 1 -1 1 1 1

• 1 1 -1 1

1 1 -1 -1 -1 -1 1 -1 -1 -1 -1 1 -1 -1 1 -1 1 1 -1 1 -1 -1 1 1 -1 -1

Π =

}

k ⌧ n

}

n

(<130 nanoseconds per packet) Rate: 7.75 × 106

8

Update Time

• Update time complexity for a linear sketch algorithm is the

number of field operations needed in each update.

• E.g., AMS sketch has update time complexity.


• When is small, AMS sketch is slow. 🐍

✏

Θ(k) = Θ(✏−2)

1 -1 1 -1 1 1 -1 -1 -1 1 1 -1 -1 1 1

• 1 -1 1

1 -1 1 -1 -1 1 -1 -1 -1 -1 -1 1 1 1 -1 1 1 1 -1 -1 -1 -1 1 -1 1 1 1

• 1 1 -1 1

1 1 -1 -1 -1 -1 1 -1 -1 -1 -1 1 -1 -1 1 -1 1 1 -1 1 -1 -1 1 1 -1 -1

Π =

}

k ⌧ n

}

n

8

Update Time

• Update time complexity for a linear sketch algorithm is the

number of field operations needed in each update.

• E.g., AMS sketch has update time complexity.


• When is small, AMS sketch is slow. 🐍

Q: Is AMS Sketch optimal in update time complexity? ✏

Θ(k) = Θ(✏−2)

1 -1 1 -1 1 1 -1 -1 -1 1 1 -1 -1 1 1

• 1 -1 1

1 -1 1 -1 -1 1 -1 -1 -1 -1 -1 1 1 1 -1 1 1 1 -1 -1 -1 -1 1 -1 1 1 1

• 1 1 -1 1

1 1 -1 -1 -1 -1 1 -1 -1 -1 -1 1 -1 -1 1 -1 1 1 -1 1 -1 -1 1 1 -1 -1

Π =

}

k ⌧ n

}

n

9

Faster One-Shot Estimation

9

Faster One-Shot Estimation

• [Dasgupta-Kumar-Sarlós 10] and [Kane-Nelson 14] showed that

sparse JL achieves one-shot with update time. O(✏−1)

9

Faster One-Shot Estimation

• [Dasgupta-Kumar-Sarlós 10] and [Kane-Nelson 14] showed that

sparse JL achieves one-shot with update time.

• [Thorup-Zhang 12] showed that CountSketch (proposed by

[Charikar-Chen-Farach-Colton 02]) achieves one-shot with

update time. O(1)

O(✏−1)

9

Faster One-Shot Estimation

• [Dasgupta-Kumar-Sarlós 10] and [Kane-Nelson 14] showed that

sparse JL achieves one-shot with update time.

• [Thorup-Zhang 12] showed that CountSketch (proposed by

[Charikar-Chen-Farach-Colton 02]) achieves one-shot with

update time.

• Application: Packet passing problem [Krishnamurthy-Sen-

Zhang-Chen 03]
 
 


O(1)

O(✏−1)

Rate: 7.75 × 106

(<130 nanoseconds per packet)

9

Faster One-Shot Estimation

• [Dasgupta-Kumar-Sarlós 10] and [Kane-Nelson 14] showed that

sparse JL achieves one-shot with update time.

• [Thorup-Zhang 12] showed that CountSketch (proposed by

[Charikar-Chen-Farach-Colton 02]) achieves one-shot with

update time.

• Application: Packet passing problem [Krishnamurthy-Sen-

Zhang-Chen 03]
 
 
 [Thorup-Zhang 12] showed that CountSketch improves AMS

sketch from 182 nanoseconds to 30 nanoseconds! O(1)

O(✏−1)

Rate: 7.75 × 106

(<130 nanoseconds per packet)

9

Faster One-Shot Estimation

• [Dasgupta-Kumar-Sarlós 10] and [Kane-Nelson 14] showed that

sparse JL achieves one-shot with update time.

• [Thorup-Zhang 12] showed that CountSketch (proposed by

[Charikar-Chen-Farach-Colton 02]) achieves one-shot with

update time.

• Application: Packet passing problem [Krishnamurthy-Sen-

Zhang-Chen 03]
 
 
 [Thorup-Zhang 12] showed that CountSketch improves AMS

sketch from 182 nanoseconds to 30 nanoseconds! O(1)

O(✏−1)

Rate: 7.75 × 106

(<130 nanoseconds per packet)

Only for one-shot

10

What About Faster Linear Sketch for Weak Tracking?

10

What About Faster Linear Sketch for Weak Tracking?

Known

• time for one-shot
• time for weak tracking

O(✏−2)

O(1)

10

What About Faster Linear Sketch for Weak Tracking?

Known

• time for one-shot
• time for weak tracking

O(✏−2)

O(1)

Unknown

• time for weak tracking

O(1)

11

CountSketch Provides Weak Tracking

11

CountSketch Provides Weak Tracking

Theorem (informal) CountSketch with rows provides -weak tracking. O(✏−2)

(✏, 0.1)

• Weak tracking: Output s.t.

(✏, )

Pr h 9t∈[m]

• kΠf (t)k2

2 kf (t)k2 2 > ✏kf (m)k2 2

• i



kΠf (1)k2

2, . . . , kΠf (m)k2 2

11

CountSketch Provides Weak Tracking

Theorem (informal) CountSketch with rows provides -weak tracking. O(✏−2)

(✏, 0.1)

• Weak tracking: Output s.t.

(✏, )

Pr h 9t∈[m]

• kΠf (t)k2

2 kf (t)k2 2 > ✏kf (m)k2 2

• i



kΠf (1)k2

2, . . . , kΠf (m)k2 2

Corollary (informal) There is an time algorithm provides -weak tracking. (✏, 0.1)

O(1)

11

CountSketch Provides Weak Tracking

Theorem (informal) CountSketch with rows provides -weak tracking. O(✏−2)

(✏, 0.1)

• The first analysis for weak tracking with constant update time.

11

CountSketch Provides Weak Tracking

Theorem (informal) CountSketch with rows provides -weak tracking. O(✏−2)

(✏, 0.1)

• The first analysis for weak tracking with constant update time.
• Using the median trick, there is a streaming algorithm

provides -weak tracking with update time. (✏, )

O(log δ−1)

11

CountSketch Provides Weak Tracking

Theorem (informal) CountSketch with rows provides -weak tracking. O(✏−2)

(✏, 0.1)

• The first analysis for weak tracking with constant update time.
• Using the median trick, there is a streaming algorithm

provides -weak tracking with update time.

• The packet passing problem now has tracking guarantee.

(✏, )

O(log δ−1)

11

CountSketch Provides Weak Tracking

Theorem (informal) CountSketch with rows provides -weak tracking. O(✏−2)

(✏, 0.1)

• The first analysis for weak tracking with constant update time.
• Using the median trick, there is a streaming algorithm

provides -weak tracking with update time.

• The packet passing problem now has tracking guarantee.

(✏, )

O(log δ−1)

The rest of the talk will focus on the proof sketch.

11

CountSketch Provides Weak Tracking

Theorem (informal) CountSketch with rows provides -weak tracking. O(✏−2)

(✏, 0.1)

• The first analysis for weak tracking with constant update time.
• Using the median trick, there is a streaming algorithm

provides -weak tracking with update time.

• The packet passing problem now has tracking guarantee.

(✏, )

O(log δ−1)

The rest of the talk will focus on the proof sketch.

12

CountSketch [Charikar-Chen-Farach-Colton 02]

12

CountSketch [Charikar-Chen-Farach-Colton 02]

• Idea: Exactly one non-zero entry in each column.


1 1 -1 0 0 -1 1 0 -1 0 0 -1 0 0 -1 -1 -1 0 1

• 1 0

1 0 -1 0 0 -1

Π =

12

CountSketch [Charikar-Chen-Farach-Colton 02]

• Idea: Exactly one non-zero entry in each column.


[Thorup-Zhang 12] showed that CountSketch with rows

achieve one-shot estimation. O(✏−2)

1 1 -1 0 0 -1 1 0 -1 0 0 -1 0 0 -1 -1 -1 0 1

• 1 0

1 0 -1 0 0 -1

Π =

12

CountSketch [Charikar-Chen-Farach-Colton 02]

• Idea: Exactly one non-zero entry in each column.


[Thorup-Zhang 12] showed that CountSketch with rows

achieve one-shot estimation.

• Analysis:

O(✏−2)

1 1 -1 0 0 -1 1 0 -1 0 0 -1 0 0 -1 -1 -1 0 1

• 1 0

1 0 -1 0 0 -1

Π =

12

CountSketch [Charikar-Chen-Farach-Colton 02]

• Idea: Exactly one non-zero entry in each column.


[Thorup-Zhang 12] showed that CountSketch with rows

achieve one-shot estimation.

• Analysis:
• Obs:

O(✏−2)

1 1 -1 0 0 -1 1 0 -1 0 0 -1 0 0 -1 -1 -1 0 1

• 1 0

1 0 -1 0 0 -1

Π =

E ⇥ (Π>Π)ij ⇤ = 1i=j .

12

CountSketch [Charikar-Chen-Farach-Colton 02]

• Idea: Exactly one non-zero entry in each column.


[Thorup-Zhang 12] showed that CountSketch with rows

achieve one-shot estimation.

• Analysis:
• Obs:
• Expectation:

O(✏−2)

1 1 -1 0 0 -1 1 0 -1 0 0 -1 0 0 -1 -1 -1 0 1

• 1 0

1 0 -1 0 0 -1

Π =

E h kΠf (m)k2

2

i = E 2 4 X

i,j2[n]

(Π>Π)ijf (m)

i

f (m)

j

3 5 = kf (m)k2

2 .

E ⇥ (Π>Π)ij ⇤ = 1i=j .

12

CountSketch [Charikar-Chen-Farach-Colton 02]

• Idea: Exactly one non-zero entry in each column.


[Thorup-Zhang 12] showed that CountSketch with rows

achieve one-shot estimation.

• Analysis:
• Obs:
• Expectation:
• Apply Chebyshev’s inequality.

O(✏−2)

1 1 -1 0 0 -1 1 0 -1 0 0 -1 0 0 -1 -1 -1 0 1

• 1 0

1 0 -1 0 0 -1

Π =

E h kΠf (m)k2

2

i = E 2 4 X

i,j2[n]

(Π>Π)ijf (m)

i

f (m)

j

3 5 = kf (m)k2

2 .

E ⇥ (Π>Π)ij ⇤ = 1i=j .

13

Intuition for Weak Tracking

13

Intuition for Weak Tracking

• First attempt: Apply union bound on one-shot analysis.


13

Intuition for Weak Tracking

• First attempt: Apply union bound on one-shot analysis.



 
 ✓ ✏, m ◆

• one-shot

13

Intuition for Weak Tracking

• First attempt: Apply union bound on one-shot analysis.



 
 ✓ ✏, m ◆

• one-shot

(✏, )

• weak tracking

13

Intuition for Weak Tracking

• First attempt: Apply union bound on one-shot analysis.


• Using rows (or rows after

median trick). O

• ✏−2−1m
• <latexit sha1_base64="r7GVqf+Zy12af+Hmex90QVb+5vk=">ACVXicdVDdbtMwGHXCGNv4WQeX3FhUSOCKglFHXeVuOGOIdFtUlMqx/nSWvNPZH9hRFZehKfhFl4A8TBIuFmRGIjWTo+5/t07FPUjhMkh9RfGvn9u6dvf2Du/fuPzgcHD08c6axHGbcSGMvCuZACg0zFCjhorbAVCHhvLh8vfHP4J1wuj32NawUGylRSU4wyAtB+O3uYQKj3OonZBGf/DPsy4vQSILNO183md4C2WnutyK1RqfLQfDZJT0oIFk2Tck5eTV5OMpltrSLY4XR5F+3lpeKNAI5fMuXma1LjwzKLgErqDvHFQM37JVjAPVDMFbuH76I4+DUpJK2PD0Uh79c8Nz5RzrSrCpGK4dn97G/Ff3rzB6mTha4bBM2vg6pGUjR0xUthQWOsg2EcSvCWylfM8s4hkZvpLSm0StkRfiJhitulGK69HltZBsqRPiErvL9rQvl/W6I/p+cZaP0xSh7Nx5Op9sa98hj8oQck5RMyJS8IadkRj5TL6Qr+Rb9D36Ge/Eu9ejcbTdeURuID78BflRt7g=</latexit>

O

• ✏−2−1log m
• <latexit sha1_base64="vuiumjrqXRQdtKkSR75/ZCQ7huU=">ACWnicdVBNb9QwFPSGr24LdAvcuFiskMqBVRKlt5W4sKNIrFtpc2ycpyXrFV/RPYLbWTlt/BruMIZiR+DN10kimAkS+OZ9zT25LUDuP4xyC6dfvO3Xs7w929+w8e7o8OHp0601gOc26ksec5cyCFhjkKlHBeW2Aql3CWX7zd+GefwTph9Edsa1gqVmlRCs4wSKvR8ftMQomHGdROSKM/+ZdplxUgkQWadD7rM7yFosukqajqMiuqNb5YjcbxJO5BA0mn8VFPXk+PpylNtaYbHGyOhgMs8LwRoFGLplziySucemZRcEldLtZ46Bm/IJVsAhUMwVu6fv8j4PSkFLY8PRSHv1zw3PlHOtysOkYrh2f3sb8V/eosHyzdILXTcIml8HlY2kaOimMFoICxlGwjVoS3Ur5mlnEMtd5IaU2jK2R5+ImGS26UYrwW1kG3pEuEJX+v7WhfJ+N0T/T07TSfJqkn4Gs9m2xp3yFPyjByShEzJjLwjJ2ROPlCvpJv5PvgZxRFw2jvejQabHcekxuInvwC/he4qQ=</latexit>

✓ ✏, m ◆

• one-shot

(✏, )

• weak tracking

13

Intuition for Weak Tracking

• First attempt: Apply union bound on one-shot analysis.


• Using rows (or rows after

median trick).

• Idea: Using chaining argument [Braverman-Chestnut-Ivkin-Nelson-

Wang-Woodruff 17] to get a fancier (and tighter) union bound.

O

• ✏−2−1m
• <latexit sha1_base64="r7GVqf+Zy12af+Hmex90QVb+5vk=">ACVXicdVDdbtMwGHXCGNv4WQeX3FhUSOCKglFHXeVuOGOIdFtUlMqx/nSWvNPZH9hRFZehKfhFl4A8TBIuFmRGIjWTo+5/t07FPUjhMkh9RfGvn9u6dvf2Du/fuPzgcHD08c6axHGbcSGMvCuZACg0zFCjhorbAVCHhvLh8vfHP4J1wuj32NawUGylRSU4wyAtB+O3uYQKj3OonZBGf/DPsy4vQSILNO183md4C2WnutyK1RqfLQfDZJT0oIFk2Tck5eTV5OMpltrSLY4XR5F+3lpeKNAI5fMuXma1LjwzKLgErqDvHFQM37JVjAPVDMFbuH76I4+DUpJK2PD0Uh79c8Nz5RzrSrCpGK4dn97G/Ff3rzB6mTha4bBM2vg6pGUjR0xUthQWOsg2EcSvCWylfM8s4hkZvpLSm0StkRfiJhitulGK69HltZBsqRPiErvL9rQvl/W6I/p+cZaP0xSh7Nx5Op9sa98hj8oQck5RMyJS8IadkRj5TL6Qr+Rb9D36Ge/Eu9ejcbTdeURuID78BflRt7g=</latexit>

O

• ✏−2−1log m
• <latexit sha1_base64="vuiumjrqXRQdtKkSR75/ZCQ7huU=">ACWnicdVBNb9QwFPSGr24LdAvcuFiskMqBVRKlt5W4sKNIrFtpc2ycpyXrFV/RPYLbWTlt/BruMIZiR+DN10kimAkS+OZ9zT25LUDuP4xyC6dfvO3Xs7w929+w8e7o8OHp0601gOc26ksec5cyCFhjkKlHBeW2Aql3CWX7zd+GefwTph9Edsa1gqVmlRCs4wSKvR8ftMQomHGdROSKM/+ZdplxUgkQWadD7rM7yFosukqajqMiuqNb5YjcbxJO5BA0mn8VFPXk+PpylNtaYbHGyOhgMs8LwRoFGLplziySucemZRcEldLtZ46Bm/IJVsAhUMwVu6fv8j4PSkFLY8PRSHv1zw3PlHOtysOkYrh2f3sb8V/eosHyzdILXTcIml8HlY2kaOimMFoICxlGwjVoS3Ur5mlnEMtd5IaU2jK2R5+ImGS26UYrwW1kG3pEuEJX+v7WhfJ+N0T/T07TSfJqkn4Gs9m2xp3yFPyjByShEzJjLwjJ2ROPlCvpJv5PvgZxRFw2jvejQabHcekxuInvwC/he4qQ=</latexit>

✓ ✏, m ◆

• one-shot

(✏, )

• weak tracking

13

Intuition for Weak Tracking

• First attempt: Apply union bound on one-shot analysis.


• Using rows (or rows after

median trick).

• Idea: Using chaining argument [Braverman-Chestnut-Ivkin-Nelson-

Wang-Woodruff 17] to get a fancier (and tighter) union bound.

O

• ✏−2−1m
• <latexit sha1_base64="r7GVqf+Zy12af+Hmex90QVb+5vk=">ACVXicdVDdbtMwGHXCGNv4WQeX3FhUSOCKglFHXeVuOGOIdFtUlMqx/nSWvNPZH9hRFZehKfhFl4A8TBIuFmRGIjWTo+5/t07FPUjhMkh9RfGvn9u6dvf2Du/fuPzgcHD08c6axHGbcSGMvCuZACg0zFCjhorbAVCHhvLh8vfHP4J1wuj32NawUGylRSU4wyAtB+O3uYQKj3OonZBGf/DPsy4vQSILNO183md4C2WnutyK1RqfLQfDZJT0oIFk2Tck5eTV5OMpltrSLY4XR5F+3lpeKNAI5fMuXma1LjwzKLgErqDvHFQM37JVjAPVDMFbuH76I4+DUpJK2PD0Uh79c8Nz5RzrSrCpGK4dn97G/Ff3rzB6mTha4bBM2vg6pGUjR0xUthQWOsg2EcSvCWylfM8s4hkZvpLSm0StkRfiJhitulGK69HltZBsqRPiErvL9rQvl/W6I/p+cZaP0xSh7Nx5Op9sa98hj8oQck5RMyJS8IadkRj5TL6Qr+Rb9D36Ge/Eu9ejcbTdeURuID78BflRt7g=</latexit>

O

• ✏−2−1log m
• <latexit sha1_base64="vuiumjrqXRQdtKkSR75/ZCQ7huU=">ACWnicdVBNb9QwFPSGr24LdAvcuFiskMqBVRKlt5W4sKNIrFtpc2ycpyXrFV/RPYLbWTlt/BruMIZiR+DN10kimAkS+OZ9zT25LUDuP4xyC6dfvO3Xs7w929+w8e7o8OHp0601gOc26ksec5cyCFhjkKlHBeW2Aql3CWX7zd+GefwTph9Edsa1gqVmlRCs4wSKvR8ftMQomHGdROSKM/+ZdplxUgkQWadD7rM7yFosukqajqMiuqNb5YjcbxJO5BA0mn8VFPXk+PpylNtaYbHGyOhgMs8LwRoFGLplziySucemZRcEldLtZ46Bm/IJVsAhUMwVu6fv8j4PSkFLY8PRSHv1zw3PlHOtysOkYrh2f3sb8V/eosHyzdILXTcIml8HlY2kaOimMFoICxlGwjVoS3Ur5mlnEMtd5IaU2jK2R5+ImGS26UYrwW1kG3pEuEJX+v7WhfJ+N0T/T07TSfJqkn4Gs9m2xp3yFPyjByShEzJjLwjJ2ROPlCvpJv5PvgZxRFw2jvejQabHcekxuInvwC/he4qQ=</latexit>

✓ ✏, m ◆

• one-shot

(✏, )

• weak tracking

We can get rid of the m dependency!

14

Step 1: Extracting the Correlation

14

Step 1: Extracting the Correlation

• Weak tracking: Output s.t.

(✏, )

Pr h 9t∈[m]

• kΠf (t)k2

2 kf (t)k2 2 > ✏kf (m)k2 2

• i



kΠf (1)k2

2, . . . , kΠf (m)k2 2

14

Step 1: Extracting the Correlation

• Rewrite the error as:


where kΠf (t)k2

2 kf (t)k2 2 = σ> ˜

Bη,f (t)σ

• Weak tracking: Output s.t.

(✏, )

Pr h 9t∈[m]

• kΠf (t)k2

2 kf (t)k2 2 > ✏kf (m)k2 2

• i



kΠf (1)k2

2, . . . , kΠf (m)k2 2

14

Step 1: Extracting the Correlation

• Rewrite the error as:


where

• and

kΠf (t)k2

2 kf (t)k2 2 = σ> ˜

Bη,f (t)σ

σ ∈ {−1, 1}n

• Weak tracking: Output s.t.

(✏, )

Pr h 9t∈[m]

• kΠf (t)k2

2 kf (t)k2 2 > ✏kf (m)k2 2

• i



kΠf (1)k2

2, . . . , kΠf (m)k2 2

14

Step 1: Extracting the Correlation

• Rewrite the error as:


where

• and
• depends on and .

kΠf (t)k2

2 kf (t)k2 2 = σ> ˜

Bη,f (t)σ

σ ∈ {−1, 1}n

˜ Bη,f (t)

Π

f (t)

• Weak tracking: Output s.t.

(✏, )

Pr h 9t∈[m]

• kΠf (t)k2

2 kf (t)k2 2 > ✏kf (m)k2 2

• i



kΠf (1)k2

2, . . . , kΠf (m)k2 2

14

Step 1: Extracting the Correlation

• Rewrite the error as:


where

• and
• depends on and .

kΠf (t)k2

2 kf (t)k2 2 = σ> ˜

Bη,f (t)σ

σ ∈ {−1, 1}n

˜ Bη,f (t)

Π

f (t)

• Weak tracking: Output s.t.

(✏, )

Pr h 9t∈[m]

• kΠf (t)k2

2 kf (t)k2 2 > ✏kf (m)k2 2

• i



kΠf (1)k2

2, . . . , kΠf (m)k2 2

Highly correlated

14

Step 1: Extracting the Correlation

• Rewrite the error as:


where

• and
• depends on and .
• The bad event becomes:

kΠf (t)k2

2 kf (t)k2 2 = σ> ˜

Bη,f (t)σ

σ ∈ {−1, 1}n

˜ Bη,f (t)

Π

f (t)

• Weak tracking: Output s.t.

(✏, )

Pr h 9t∈[m]

• kΠf (t)k2

2 kf (t)k2 2 > ✏kf (m)k2 2

• i



kΠf (1)k2

2, . . . , kΠf (m)k2 2

Highly correlated

14

Step 1: Extracting the Correlation

• Rewrite the error as:


where

• and
• depends on and .
• The bad event becomes:

sup

t2[m]

• > ˜

Bη,f (t)

• > ✏kf (m)k2

2

kΠf (t)k2

2 kf (t)k2 2 = σ> ˜

Bη,f (t)σ

σ ∈ {−1, 1}n

˜ Bη,f (t)

Π

f (t)

• Weak tracking: Output s.t.

(✏, )

Pr h 9t∈[m]

• kΠf (t)k2

2 kf (t)k2 2 > ✏kf (m)k2 2

• i



kΠf (1)k2

2, . . . , kΠf (m)k2 2

Highly correlated

14

Step 1: Extracting the Correlation

• Rewrite the error as:


where

• and
• depends on and .
• The bad event becomes:

sup

t2[m]

• > ˜

Bη,f (t)

• > ✏kf (m)k2

2

kΠf (t)k2

2 kf (t)k2 2 = σ> ˜

Bη,f (t)σ

σ ∈ {−1, 1}n

˜ Bη,f (t)

Π

f (t)

Different from [BCINWW17]

• Weak tracking: Output s.t.

(✏, )

Pr h 9t∈[m]

• kΠf (t)k2

2 kf (t)k2 2 > ✏kf (m)k2 2

• i



kΠf (1)k2

2, . . . , kΠf (m)k2 2

Highly correlated

15

Step 2: -Net ✏

15

Step 2: -Net ✏

Goal: Pr " sup

t2[m]

• > ˜

Bη,f (t)

• > ✏kf (m)k2

2

#  0.1 .

15

Step 2: -Net ✏

Goal: Pr " sup

t2[m]

• > ˜

Bη,f (t)

• > ✏kf (m)k2

2

#  0.1 .

…

˜ B1

˜ B2

˜ B3

˜ B4

˜ B5

˜ B6

˜ B7

˜ Bm

˜ Bm−1

˜ Bm−2

15

Step 2: -Net ✏

Goal: Pr " sup

t2[m]

• > ˜

Bη,f (t)

• > ✏kf (m)k2

2

#  0.1 .

…

˜ B1

˜ B2

˜ B3

˜ B4

˜ B5

˜ B6

˜ B7

˜ Bm

˜ Bm−1

˜ Bm−2

• A sequence of nets such that

T0, T1, . . .

15

Step 2: -Net ✏

Goal: Pr " sup

t2[m]

• > ˜

Bη,f (t)

• > ✏kf (m)k2

2

#  0.1 .

…

˜ B1

˜ B2

˜ B3

˜ B4

˜ B5

˜ B6

˜ B7

˜ Bm

˜ Bm−1

˜ Bm−2

• A sequence of nets such that

T0, T1, . . .

15

Step 2: -Net ✏

Goal: Pr " sup

t2[m]

• > ˜

Bη,f (t)

• > ✏kf (m)k2

2

#  0.1 .

…

˜ B1

˜ B2

˜ B3

˜ B4

˜ B5

˜ B6

˜ B7

˜ Bm

˜ Bm−1

˜ Bm−2

• A sequence of nets such that

T0, T1, . . .

15

Step 2: -Net ✏

Goal: Pr " sup

t2[m]

• > ˜

Bη,f (t)

• > ✏kf (m)k2

2

#  0.1 .

…

˜ B1

˜ B2

˜ B3

˜ B4

˜ B5

˜ B6

˜ B7

˜ Bm

˜ Bm−1

˜ Bm−2

• A sequence of nets such that

T0, T1, . . .

15

Step 2: -Net ✏

Goal: Pr " sup

t2[m]

• > ˜

Bη,f (t)

• > ✏kf (m)k2

2

#  0.1 .

…

˜ B1

˜ B2

˜ B3

˜ B4

˜ B5

˜ B6

˜ B7

˜ Bm

˜ Bm−1

˜ Bm−2

• A sequence of nets such that
• The coarser the net is, the smaller it is.

T0, T1, . . .

15

Step 2: -Net ✏

Goal: Pr " sup

t2[m]

• > ˜

Bη,f (t)

• > ✏kf (m)k2

2

#  0.1 .

…

˜ B1

˜ B2

˜ B3

˜ B4

˜ B5

˜ B6

˜ B7

˜ Bm

˜ Bm−1

˜ Bm−2

• A sequence of nets such that
• The coarser the net is, the smaller it is.
• Telescoping using these nets

˜ Bη,f (t)

T0, T1, . . .

15

Step 2: -Net ✏

Goal: Pr " sup

t2[m]

• > ˜

Bη,f (t)

• > ✏kf (m)k2

2

#  0.1 .

…

˜ B1

˜ B2

˜ B3

˜ B4

˜ B5

˜ B6

˜ B7

˜ Bm

˜ Bm−1

˜ Bm−2

• A sequence of nets such that
• The coarser the net is, the smaller it is.
• Telescoping using these nets
• ˜

Bη,f (t)

sup

t∈[m]

γ ⇣ ˜ B⌘,f (t) ⌘ ≤ sup

t∈[m]

γ ⇣ B(t)

⌘,0

⌘ +

∞

X

`=1

sup

t∈[m]

γ ⇣ B(t)

⌘,` − B(t) ⌘,`−1

⌘

T0, T1, . . .

15

Step 2: -Net ✏

Goal: Pr " sup

t2[m]

• > ˜

Bη,f (t)

• > ✏kf (m)k2

2

#  0.1 .

…

˜ B1

˜ B2

˜ B3

˜ B4

˜ B5

˜ B6

˜ B7

˜ Bm

˜ Bm−1

˜ Bm−2

• A sequence of nets such that
• The coarser the net is, the smaller it is.
• Telescoping using these nets
• ˜

Bη,f (t)

sup

t∈[m]

γ ⇣ ˜ B⌘,f (t) ⌘ ≤ sup

t∈[m]

γ ⇣ B(t)

⌘,0

⌘ +

∞

X

`=1

sup

t∈[m]

γ ⇣ B(t)

⌘,` − B(t) ⌘,`−1

⌘

T0, T1, . . .

T0

T`

15

Step 2: -Net ✏

Goal: Pr " sup

t2[m]

• > ˜

Bη,f (t)

• > ✏kf (m)k2

2

#  0.1 .

…

˜ B1

˜ B2

˜ B3

˜ B4

˜ B5

˜ B6

˜ B7

˜ Bm

˜ Bm−1

˜ Bm−2

• A sequence of nets such that
• The coarser the net is, the smaller it is.
• Telescoping using these nets
• ˜

Bη,f (t)

sup

t∈[m]

γ ⇣ ˜ B⌘,f (t) ⌘ ≤ sup

t∈[m]

γ ⇣ B(t)

⌘,0

⌘ +

∞

X

`=1

sup

t∈[m]

γ ⇣ B(t)

⌘,` − B(t) ⌘,`−1

⌘

Different from [BCINWW17]

T0, T1, . . .

T0

T`

16

What’s Missing…

16

What’s Missing…

• Dudley’s inequality.

16

What’s Missing…

• Dudley’s inequality.
• How to bound the size of -net for ?

✏

{ ˜ Bη,f (t)}t∈[m]

16

What’s Missing…

• Dudley’s inequality.
• How to bound the size of -net for ?
• Greedily pick from and analyze by the

insertion-only structure of the input stream. ✏

{ ˜ Bη,f (t)}t∈[m]

{ ˜ Bη,f (t)}t∈[m]

16

What’s Missing…

• Dudley’s inequality.
• How to bound the size of -net for ?
• Greedily pick from and analyze by the

insertion-only structure of the input stream.

• How to bound the error magnitude?

✏

{ ˜ Bη,f (t)}t∈[m]

{ ˜ Bη,f (t)}t∈[m]

16

What’s Missing…

• Dudley’s inequality.
• How to bound the size of -net for ?
• Greedily pick from and analyze by the

insertion-only structure of the input stream.

• How to bound the error magnitude?
• Using the Hansen-Wright inequality for the moments of


. ✏

{ ˜ Bη,f (t)}t∈[m]

{ ˜ Bη,f (t)}t∈[m]

σ>Bσ − E[σ>Bσ]

16

What’s Missing…

• Dudley’s inequality.
• How to bound the size of -net for ?
• Greedily pick from and analyze by the

insertion-only structure of the input stream.

• How to bound the error magnitude?
• Using the Hansen-Wright inequality for the moments of


.

• High probability regime?

✏

{ ˜ Bη,f (t)}t∈[m]

{ ˜ Bη,f (t)}t∈[m]

σ>Bσ − E[σ>Bσ]

16

What’s Missing…

• Dudley’s inequality.
• How to bound the size of -net for ?
• Greedily pick from and analyze by the

insertion-only structure of the input stream.

• How to bound the error magnitude?
• Using the Hansen-Wright inequality for the moments of


.

• High probability regime?
• Median trick.

✏

{ ˜ Bη,f (t)}t∈[m]

{ ˜ Bη,f (t)}t∈[m]

σ>Bσ − E[σ>Bσ]

16

What’s Missing…

• Dudley’s inequality.
• How to bound the size of -net for ?
• Greedily pick from and analyze by the

insertion-only structure of the input stream.

• How to bound the error magnitude?
• Using the Hansen-Wright inequality for the moments of


.

• High probability regime?
• Median trick.

✏

{ ˜ Bη,f (t)}t∈[m]

{ ˜ Bη,f (t)}t∈[m]

σ>Bσ − E[σ>Bσ]

Ask me offline for more details!

17

Conclusion

17

Conclusion

• We show the first streaming algorithm achieving weak tracking

for estimation with constant update time.

`2

17

Conclusion

• We show the first streaming algorithm achieving weak tracking

for estimation with constant update time.

• CountSketch and packet passing problem now have

tracking guarantee!

`2

17

Conclusion

• We show the first streaming algorithm achieving weak tracking

for estimation with constant update time.

• CountSketch and packet passing problem now have

tracking guarantee!

• Future directions:

`2

17

Conclusion

• We show the first streaming algorithm achieving weak tracking

for estimation with constant update time.

• CountSketch and packet passing problem now have

tracking guarantee!

• Future directions:
• Empirical performance of CountSketch?

`2

17

Conclusion

• We show the first streaming algorithm achieving weak tracking

for estimation with constant update time.

• CountSketch and packet passing problem now have

tracking guarantee!

• Future directions:
• Empirical performance of CountSketch?
• Weak tracking for norm with faster update time?

`2

`p

17

Conclusion

• We show the first streaming algorithm achieving weak tracking

for estimation with constant update time.

• CountSketch and packet passing problem now have

tracking guarantee!

• Future directions:
• Empirical performance of CountSketch?
• Weak tracking for norm with faster update time?
• Other applications of charing technique?

`2

`p

17

Conclusion

• We show the first streaming algorithm achieving weak tracking

for estimation with constant update time.

• CountSketch and packet passing problem now have

tracking guarantee!

• Future directions:
• Empirical performance of CountSketch?
• Weak tracking for norm with faster update time?
• Other applications of charing technique?

`2

`p

Thanks for your attention, questions?