L ECTURE 7 Last time Communication complexity Other models of - - PowerPoint PPT Presentation

9/24/2020

Sublinear Algorithms

LECTURE 7

Last time

• Communication complexity
• Other models of computation

Today

• Streaming

Sofya Raskhodnikova;Boston University

Data Stream Model [Alon Matias Szegedy 96]

Motivation: internet traffic analysis Model the stream as 𝑛 elements from [𝑜], e.g., 𝑏1, 𝑏2, … , 𝑏𝑛 = 3, 5, 3, 7, 5, 4, … Goal: Compute a function of the stream, e.g., median, number of distinct elements, longest increasing sequence.

2

B L A - B L A - B L A - B L A - B L A - B L A - B L A -

(2) Limited working memory (3) Quickly produce output (1) Quickly process each element

Streaming Algorithm

Based on Andrew McGregor’s slides: http://www.cs.umass.edu/~mcgregor/slides/10-jhu1.pdf

Streaming Puzzle

A stream contains 𝑜 − 1 distinct elements from 𝑜 in arbitrary order. Problem: Find the missing element, using 𝑃(log 𝑜) space.

3

Sampling from a Stream of Unknown Length

Warm-up: Find a uniform sample 𝑡 from a stream 𝑏1, 𝑏2, … , 𝑏𝑛

• f known length 𝑛.

4

Sampling from a Stream of Unknown Length

Problem: Find a uniform sample 𝑡 from a stream 𝑏1, 𝑏2, … , 𝑏𝑛

• f unknown length 𝑛

Analysis: What is the probability that 𝑡 = 𝑏𝑗 at some time 𝑢 ≥ 𝑗? Pr 𝑡 = 𝑏𝑗 = 1 𝑗 ⋅ 1 − 1 𝑗 + 1 ⋅ … ⋅ 1 − 1 𝑢 = 1 𝑗 ⋅ 𝑗 𝑗 + 1 ⋅ … ⋅ 𝑢 − 1 𝑢 = 1 𝑢 Space: 𝑃(𝑙 log 𝑜 + log 𝑛) bits to get 𝑙 samples.

5

Algorithm (Reservoir Sampling)

• 1. Initially, 𝑡 ← 𝑏1
• 2. On seeing the 𝑢th element, 𝑡 ← 𝑏𝑢 with probability 1/𝑢

Counting Distinct Elements

Input: a stream 𝑏1, 𝑏2, … , 𝑏𝑛 ∈ 𝑜 𝑛 Warm-up: Output the number of distinct elements in the stream. Exact solutions:

• Store 𝑜 bits, indicating whether each domain element has appeared.
• Store the stream: O(𝑛 log 𝑜) bits.

Known lower bounds:

• Every deterministic algorithm requires Ω(𝑛) bits

(even for a constant-factor approximation).

• Every exact algorithm (even randomized) requires Ω 𝑜 bits.

Need to use both randomization and approximation to get polylog(𝑛, 𝑜) space

6

Counting Distinct Elements

Input: a stream 𝑏1, 𝑏2, … , 𝑏𝑛 ∈ 𝑜 𝑛 Goal: Estimate the number of distinct elements in the stream up to a multiplicative factor (1 + 𝜁) with probability ≥ 2/3

• Studied by [Flajolet Martin 83, Alon Matias Szegedy 96,...]
• Today: 𝑃(𝜁−2 log 𝑜) space algorithm

[Bar−Yossef Jayram Kumar Sivakuar Trevisan 02]

• Optimal: 𝑃(𝜁−2 + log 𝑜) space algorithm [Kane Nelson Woodruff 10]

7

Counting Distinct Elements

Input: a stream 𝑏1, 𝑏2, … , 𝑏𝑛 ∈ 𝑜 𝑛 Goal: Estimate the number of distinct elements in the stream up to a multiplicative factor (1 + 𝜁) with probability ≥ 2/3 Analysis:

• Algorithm uses 𝑃(𝜁−2 log 𝑜) bits of space (not accounting for storing ℎ)
• We'll show: estimate ǁ

𝑠 has good accuracy with reasonable probability.

8

Algorithm 1. Apply a random hash function ℎ ∶ 𝑜 → [𝑜] to each element 2. Compute 𝑌, the 𝑢-th smallest value of the hash seen where 𝑢 = 10 ⁄ 𝜁2 3. Return ǁ 𝑠 = 𝑢 ⋅ 𝑜/𝑌 as estimate for 𝑠, the number of distinct elements.

Claim.

Pr ǁ 𝑠 − 𝑠 ≤ 𝜁𝑠 ≥ 2/3

Counting Distinct Elements: Analysis

Proof: Suppose the distinct elements are 𝑓1, … , 𝑓𝑠

• Overestimation:

Pr ǁ 𝑠 ≥ 1 + 𝜁 𝑠 = Pr 𝑢 ⋅ 𝑜 𝑌 ≥ 1 + 𝜁 𝑠 = Pr 𝑌 ≤ 𝑢 ⋅ 𝑜 𝑠 1 + 𝜁

• Let 𝑍

𝑗 = 𝟚 ℎ(𝑓𝑗) ≤ 𝑢⋅𝑜 𝑠 1+𝜁

and 𝑍 = σ𝑗=1

𝑠

𝑍

𝑗

E 𝑍 = 𝑠 ⋅ 𝐹 𝑍

1 = 𝑠 ⋅

𝑢 𝑠 1 + 𝜁 = 𝑢 1 + 𝜁 Var 𝑍 = Var ෍

𝑗=1 𝑠

𝑍

𝑗 = ෍ 𝑗=1 𝑠

Var 𝑍

𝑗

≤ ෍

𝑗=1 𝑠

E 𝑍

𝑗 2 = ෍ 𝑗=1 𝑠

E 𝑍

𝑗 = E 𝑍

9

Claim.

Pr ǁ 𝑠 − 𝑠 ≤ 𝜁𝑠 ≥ 2/3

𝑌: 𝑢-th smallest hashed value 𝑢 = 10 ⁄ 𝜁2 ǁ 𝑠 = 𝑢 ⋅ 𝑜/𝑌 𝐹[𝑍] = 𝑢 1 + 𝜁 Var 𝑍 ≤ 𝐹[𝑍]

Counting Distinct Elements: Analysis

Proof: Suppose the distinct elements are 𝑓1, … , 𝑓𝑠

• Overestimation:

Pr ǁ 𝑠 ≥ 1 + 𝜁 𝑠 = Pr 𝑢 ⋅ 𝑜 𝑌 ≥ 1 + 𝜁 𝑠 = Pr 𝑌 ≤ 𝑢 ⋅ 𝑜 𝑠 1 + 𝜁

• Let 𝑍

𝑗 = 𝟚 ℎ(𝑓𝑗) ≤ 𝑢⋅𝑜 𝑠 1+𝜁

and 𝑍 = σ𝑗=1

𝑠

𝑍

𝑗

Pr 𝑌 ≤ 𝑢 ⋅ 𝑜 𝑠 1 + 𝜁 = Pr 𝑍 ≥ 𝑢 = Pr 𝑍 ≥ 1 + 𝜁 E 𝑍

• By the Chebyshev’s inequality, for 𝜁 ≤ 2/3,

Pr 𝑍 ≥ 1 + 𝜁 E 𝑍 ≤ Var[𝑍] 𝜁 ⋅ E 𝑍

2 ≤

1 𝜁2E 𝑍 = 1 + 𝜁 𝜁2 ⋅ 𝑢 = 1 + 𝜁 10 ≤ 1 6

• Underestimation: A similar analysis shows Pr ǁ

𝑠 ≤ 1 − 𝜁 𝑠 ≤

1 6

10

Claim.

Pr ǁ 𝑠 − 𝑠 ≤ 𝜁𝑠 ≥ 2/3

𝑌: 𝑢-th smallest hashed value 𝑢 = 10 ⁄ 𝜁2 ǁ 𝑠 = 𝑢 ⋅ 𝑜/𝑌 𝐹[𝑍] = 𝑢 1 + 𝜁 Var 𝑍 ≤ 𝐹[𝑍]

Removing the Random Hashing Assumption

Idea: Use limited independence

• A family ℋ = {ℎ: 𝑏 → 𝑐 } of hash functions is 𝑙-wise independent if for

all distinct 𝑦1, … , 𝑦𝑙 ∈ [𝑏] and all 𝑧1, … , 𝑧𝑙 ∈ 𝑐 , Pr

ℎ∈ℋ ℎ 𝑦1 = 𝑧1, … , ℎ 𝑦𝑙 = 𝑧𝑙 = 1

𝑐𝑙 Note: a uniformly random family is 𝑙-wise independent for all 𝑙

• Observations: For 𝑦1, … , 𝑦𝑙 as above,

1. ℎ(𝑦1) is uniform over 𝑐 2. ℎ 𝑦1 , … , ℎ(𝑦𝑙) are mutually independent.

11

Based on Sepehr Assadi’s lecture notes for CS 514 (Lecture 7, 03/20/20) at Rutgers

Construction of 𝒍-wise Independent Family

Idea: Use limited independence

• A family ℋ = {ℎ: 𝑏 → 𝑐 } of hash functions is 𝑙-wise independent if for

all distinct 𝑦1, … , 𝑦𝑙 ∈ [𝑏] and all 𝑧1, … , 𝑧𝑙 ∈ 𝑐 , Pr

ℎ∈ℋ ℎ 𝑦1 = 𝑧1, … , ℎ 𝑦𝑙 = 𝑧𝑙 = 1

𝑐𝑙

• Space to store ℎ is 𝑃 𝑙 log 𝑞
• For arbitrary 𝑏, 𝑐, need 𝑃 𝑙 ⋅ log 𝑏 + log 𝑐

space.

12

Based on Sepehr Assadi’s lecture notes for CS 514 (Lecture 7, 03/20/20) at Rutgers

Construction of 𝑙-wise Independent Family of Hash Functions 1. Let 𝑞 be a prime. 2. Condider the set of polynomials of degree 𝑙 − 1 over 𝔾𝑞 ℋ = {ℎ: {0, … , 𝑞 − 1} → {0, … , 𝑞 − 1} ∣ ℎ 𝑦 = 𝑑𝑙−1𝑦𝑙−1 + ⋯ + 𝑑1𝑦 + 𝑑0, with 𝑑0, … , 𝑑𝑙−1 ∈ 𝔾𝑞} 3. To sample ℎ ∈ ℋ, sample 𝑑0, … , 𝑑𝑙−1 ∈ 𝔾𝑞 u.i.r.

Counting Distinct Elements: Final Algorithm

Input: a stream 𝑏1, 𝑏2, … , 𝑏𝑛 ∈ 𝑜 𝑛 Goal: Estimate the number of distinct elements in the stream up to a multiplicative factor (1 + 𝜁) with probability ≥ 2/3 Analysis:

• Algorithm uses 𝑃(𝜁−2 log 𝑜) bits of space
• Our correctness analysis applies.

13

Algorithm 1. Sample a hash function ℎ ∶ 𝑜 → [𝑜] from a 2-wise independent family and apply ℎ to each element 2. Compute 𝑌, the 𝑢-th smallest value of the hash seen where 𝑢 = 10 ⁄ 𝜁2 3. Return ǁ 𝑠 = 𝑢 ⋅ 𝑜/𝑌 as estimate for 𝑠, the number of distinct elements.

Frequency Moments Estimation

Input: a stream 𝑏1, 𝑏2, … , 𝑏𝑛 ∈ 𝑜 𝑛

• The frequency vector of the stream is 𝑔 = (𝑔

1, … , 𝑔 𝑛),

where 𝑔

𝑗 is the number of times 𝑗 appears in the stream

• The 𝑞-th frequency moment is 𝐺

𝑞 =

𝑔

𝑞 𝑞 = σ𝑗=1 𝑜

𝑔

𝑗 𝑞

𝐺0 is the number of nonzero entries of 𝑔 (# of distinct elements) 𝐺

1 = 𝑛 (# of elements in the stream)

𝐺2 = 𝑔

2 2 is a measure of non-uniformity

used e.g. for anomaly detection in network analysis 𝐺

∞ = max 𝑗

𝑔

𝑗 is the most frequent element

Goal: Estimate 𝐺

𝑞 up to a multiplicative factor (1 + 𝜁) with probability ≥ 2/3

14

Summary

Streaming Model

• Reservoir sampling
• Distinct Elements (approximating 𝐺

0)

• 𝑙-wise independent hashing

20