not exactly approximate algorithms for big data
play

NOT EXACTLY! APPROXIMATE ALGORITHMS FOR BIG DATA FANGJIN YANG - PowerPoint PPT Presentation

Sep 24, 2022 •454 likes •1.11k views

NOT EXACTLY! APPROXIMATE ALGORITHMS FOR BIG DATA FANGJIN YANG DRUID COMMITTER METAMARKETS NELSON RAY QUANTITATIVE ANALYST GOOGLE OVERVIEW THE PROBLEM MANAGE DATA COST EFFICIENTLY THE DATA DEALING WITH EVENT STREAMS SIMPLIFYING

  1. NOT EXACTLY! APPROXIMATE ALGORITHMS FOR BIG DATA FANGJIN YANG · DRUID COMMITTER · METAMARKETS NELSON RAY · QUANTITATIVE ANALYST · GOOGLE

  2. OVERVIEW THE PROBLEM MANAGE DATA COST EFFICIENTLY THE DATA DEALING WITH EVENT STREAMS SIMPLIFYING STORAGE DATA SUMMARIZATION FINDING UNIQUES HYPERLOGLOG ESTIMATING DISTRIBUTION APPROXIMATE HISTOGRAMS

  3. THE PROBLEM

  4. Real-time Bidding Fangjin Yang & Nelson Ray 2014

  5. PROBLEMS ‣ Storing/processing billions of rows is expensive ‣ Reduce storage, improve performance ‣ Reduce storage by throwing away information ‣ Throwing away information reduces accuracy Fangjin Yang & Nelson Ray 2014

  6. THE DATA

  7. THE DATA Timestamp Bid Price 2013-10-28T02:13:43Z 1.19 2013-10-28T02:14:21Z 0.05 2013-10-28T02:55:32Z 1.04 2013-10-28T03:07:28Z 0.16 2013-10-28T03:13:43Z 1.03 2013-10-28T04:18:19Z 0.15 2013-10-28T05:36:34Z 0.01 2013-10-28T05:37:59Z 1.03 Fangjin Yang & Nelson Ray 2014

  8. DATA SUMMARIZATION Timestamp Bid Price Timestamp Revenue Number of Prices 2013-10-28T02:13:43Z 1.19 2013-10-28T02 2013-10-28T02:14:21Z 0.05 2.28 3 2013-10-28T02:55:32Z 1.04 2013-10-28T03:07:28Z 0.16 2013-10-28T03 1.19 2 2013-10-28T03:13:43Z 1.03 2013-10-28T04 0.15 1 2013-10-28T04:18:19Z 0.15 2013-10-28T05:36:34Z 0.01 2013-10-28T05 1.04 2 2013-10-28T05:37:59Z 1.03 Fangjin Yang & Nelson Ray 2014

  9. COMBINING SUMMARIZATIONS Timestamp Revenue Number of Prices Timestamp Revenue Number of Prices 2013-10-28T02 2.28 3 2013-10-28 2013-10-28T03 1.19 2 4.66 8 2013-10-28T04 0.15 1 2013-10-28T05 1.04 2 Fangjin Yang & Nelson Ray 2014

  10. Fangjin Yang & Nelson Ray 2014

  11. SUMMARIZATION SUMMARY ‣ Throw away information about individual events ‣ Drastically reduce storage and improve query speed • On average, 40x reduction in storage on with our own data ‣ We’ve lost info about individual prices ‣ Data summarization is not always trivial Fangjin Yang & Nelson Ray 2014

  12. CASE STUDY 1

  13. CASE STUDY 1 ‣ Problem: determine unique number of elements in a set ‣ Use case: measuring number of unique users DATA BIG DATA Fangjin Yang & Nelson Ray 2014

  14. EXACT SOLUTION ‣ Store every single username (in a Java HashSet) ‣ No loss of information, no accuracy tradeoff Fangjin Yang & Nelson Ray 2014

  15. HASHSET Timestamp Username Timestamp Usernames 2013-10-28T02:13:43Z user1 2013-10-28T02 2013-10-28T02:14:21Z user2 {user1, user2} 2013-10-28T02:55:32Z user1 2013-10-28T03:07:28Z user4 2013-10-28T03 {user4, user97} 2013-10-28T03:13:43Z user97 2013-10-28T04 {user2} 2013-10-28T04:18:19Z user2 2013-10-28T05:36:34Z user9834 {user9834, 2013-10-28T05 user97} 2013-10-28T05:37:59Z user97 Fangjin Yang & Nelson Ray 2014

  16. HASHSET Usernames Timestamp Usernames Timestamp 2013-10-28T02 {user1, user2} {user1, user2, 2013-10-28 2013-10-28T03 {user4, user97} user4, user97, user9834} 2013-10-28T04 {user2} {user9834, 2013-10-28T05 user97} Fangjin Yang & Nelson Ray 2014

  17. EXACT SOLUTION ‣ Storage/Computation: O(# uniques) ‣ We’re not throwing away any information about usernames ‣ Accuracy: 100% Fangjin Yang & Nelson Ray 2014

  18. INFEASIBLE STORAGE ‣ High cardinality user dimensions == infeasible storage • Storage cost for 10^9 unique elements == ~48GB of storage Fangjin Yang & Nelson Ray 2014

  19. CARDINALITY ESTIMATION ‣ Plenty of literature • Linear Counting • Count-Min Sketch • LogLog Fangjin Yang & Nelson Ray 2014

  20. HYPERLOGLOG ‣ Storage: 1.5 KB ( for cardinalities 10^9 and above) • 99.999997% decrease in storage size ‣ Computation: O(1) ( for cardinalities < ~10^10) ‣ Accuracy: 97% Fangjin Yang & Nelson Ray 2014

  21. HASH FUNCTIONS ‣ Maps value in one space (generally larger) to another value in another space (generally smaller) String 0001 HashFn Fangjin Yang & Nelson Ray 2014

  22. WHAT MAKES A GOOD HASH FUNCTION? ‣ Bits of output value are independent and have an equal probability of occurring (50%) String 50% Probability 0xxx HashFn String 50% Probability 1xxx HashFn Fangjin Yang & Nelson Ray 2014

  23. HASHING TWO STRINGS user1 0xxx HashFn user2 1xxx HashFn Fangjin Yang & Nelson Ray 2013

  24. THE NEXT BIT String 00xx 25% Probability HashFn String 10xx 25% Probability HashFn String 25% Probability 01xx HashFn String 25% Probability 11xx HashFn Fangjin Yang & Nelson Ray 2013

  25. HASHING 4 STRINGS user1 00xx HashFn user2 10xx HashFn user3 01xx HashFn user4 11xx HashFn Fangjin Yang & Nelson Ray 2013

  26. HYPERLOGLOG ‣ What about 001x? • If we hashed one string, 12.5% chance this could occur • If we hashed 8 strings, one of them should be this value ‣ What about 000001…x? • Extremely unlikely to occur if we only hashed one string Fangjin Yang & Nelson Ray 2013

  27. HYPERLOGLOG ‣ Looks at distribution of bits of hashed values ‣ Cares about the position of the left most ‘1’ bit ‣ 1000 -> position == 1 ‣ 0100 -> position == 2 ‣ 0011 -> position == 3 Fangjin Yang & Nelson Ray 2014

  28. HYPERLOGLOG ‣ Stores the max position of the left-most ‘1’ bit of hashed values ‣ User1 —> hash —> 1000 (position == 1) ‣ User2 —> hash —> 0100 (position == 2) ‣ User3 —> hash —> 0011 (position == 3) ‣ HLL will store position == 3 Fangjin Yang & Nelson Ray 2014

  29. HYPERLOGLOG Fangjin Yang & Nelson Ray 2014

  30. HYPERLOGLOG ACCURACY String 00xx HashFn String 10xx HashFn String 25% Probability 01xx HashFn String 11xx HashFn Fangjin Yang & Nelson Ray 2013

  31. HYPERLOGLOG ‣ If we fed the stream through a second hash function, we’d have a second independent estimate ‣ Adding more hash functions gives us more independent estimates that we can combine together for a lower variance estimate ‣ This is expensive because we have to hash the same data n times Fangjin Yang & Nelson Ray 2014

  32. HYPERLOGLOG ‣ Instead we can split the stream ‣ Estimate the cardinality of each sub-stream ‣ For each sub-stream ‣ Store the maximum over the positions of the leftmost '1' bit for hashed values of the sub-stream Fangjin Yang & Nelson Ray 2014

  33. HYPERLOGLOG Buckets -INF -INF -INF -INF Fangjin Yang & Nelson Ray 2014

  34. HYPERLOGLOG Buckets user1 01xxx...x 2 HashFn -INF -INF -INF Fangjin Yang & Nelson Ray 2014

  35. HYPERLOGLOG Buckets user1 01xxx...x 2 HashFn user4 01xxx...x 2 HashFn user12 01xxx...x 2 HashFn user7 1xxxx...x 1 HashFn Fangjin Yang & Nelson Ray 2014

  36. HYPERLOGLOG Buckets user6 001xx...x 2 -> 3 HashFn 2 2 1 Fangjin Yang & Nelson Ray 2014

  37. DETERMINING FINAL CARDINALITY Buckets 3 11.00 2 MATH 2 1 Fangjin Yang & Nelson Ray 2014

  38. HYPERLOGLOG Timestamp Buckets 2013-10-28T02 [3, 2, 2, 1] 2013-10-28T03 [1, 2, 1, 2] 2013-10-28T04 [2, 1, 4, 1] 2013-10-28T05 [2, 2, 3, 1] Fangjin Yang & Nelson Ray 2014

  39. HYPERLOGLOG Timestamp HLL Object 2013-10-28 [3, 2, 4, 2] Fangjin Yang & Nelson Ray 2014

  40. Fangjin Yang & Nelson Ray 2014

  41. RESULTS Fangjin Yang & Nelson Ray 2014

  42. CASE STUDY 2

  43. CASE STUDY 2 ‣ Problem: determine distribution of values ‣ Use case: quantiles and histograms ‣ Hourly truncation Fangjin Yang & Nelson Ray 2014

  44. THE DATA Timestamp Bid Price 2013-10-28T02:13:43Z 1.19 2013-10-28T02:14:21Z 0.05 2013-10-28T02:55:32Z 1.04 2013-10-28T03:07:28Z 0.16 2013-10-28T03:13:43Z 1.03 2013-10-28T04:18:19Z 0.15 2013-10-28T05:36:34Z 0.01 2013-10-28T05:37:59Z 1.03 Fangjin Yang & Nelson Ray 2014

  45. EXACT SOLUTION Bid Price Timestamp Timestamp Bid Prices 2013-10-28T02:13:43Z 1.19 2013-10-28T02 2013-10-28T02:14:21Z 0.05 [1.19, 0.05, 1.04] 2013-10-28T02:55:32Z 1.04 2013-10-28T03:07:28Z 0.16 2013-10-28T03 [0.16, 1.03] 2013-10-28T03:13:43Z 1.03 2013-10-28T04:18:19Z 0.15 2013-10-28T04 [0.15] 2013-10-28T05:36:34Z 0.01 2013-10-28T05 [0.01, 1.03] 2013-10-28T05:37:59Z 1.03 Fangjin Yang & Nelson Ray 2014

  46. EXACT SOLUTION Timestamp Bid Prices Timestamp Bid Prices 2013-10-28T02 [1.19, 0.05, 1.04] 2013-10-28 [1.19, 0.05, 1.04, 0.16, 2013-10-28T03 [0.16, 1.03] 1.03, 0.15, 0.01, 1.03] 2013-10-28T04 [0.15] 2013-10-28T05 [0.01, 1.03] Fangjin Yang & Nelson Ray 2014

  47. EXACT SOLUTION ‣ Arrays of values ‣ Storage: Linear ‣ Computation: Linear ‣ Accuracy: 100% ‣ Problem: Storing raw values can often be more expensive than storing the rest of the row. ‣ Solution: Store an approximate representation! Fangjin Yang & Nelson Ray 2014

  48. APPROXIMATE HISTOGRAMS ‣ “A Streaming Parallel Decision Tree Algorithm” ‣ Yael Ben-Haim & Elad Tom-Tov ‣ Storage: Sublinear/Linear ‣ Computation: Sublinear/Linear ‣ Accuracy: pretty good Fangjin Yang & Nelson Ray 2014

  49. RAW DATA • 40 Prices: 3.46, 5.37, 5.62, 5.87, 6.21, 6.79, 7.11, 7.36, 7.55, 7.64, 7.89, 7.9, 8.07, 8.44, 8.62, 8.78, 8.87, 9.03, 9.24, 9.36, 9.58, 9.59, 9.81, 10.31, 10.35, 10.39, 10.47, 10.77, 10.93, 11.04, 11.1, 13.1, 13.27, 13.29, 13.87, 14.29, 14.51, 14.9, 15.75, 17.07 Fangjin Yang & Nelson Ray 2013

  50. RAW DATA Fangjin Yang & Nelson Ray 2013

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Empirical Comparison of Approximate Inference Algorithms for Networked Data Prithviraj Sen Lise

Empirical Comparison of Approximate Inference Algorithms for Networked Data Prithviraj Sen Lise

Empirical Comparison of Approximate Inference Algorithms for Networked Data Prithviraj Sen Lise Getoor Department of Computer Science University of Maryland, College Park. Workshop on Open Problems in Statistical Relational Learning, 2006

326 views • 16 slides
Algorithms for Big Data (X) Chihao Zhang Shanghai Jiao Tong University Nov. 22, 2019 Algorithms

Algorithms for Big Data (X) Chihao Zhang Shanghai Jiao Tong University Nov. 22, 2019 Algorithms

Algorithms for Big Data (X) Chihao Zhang Shanghai Jiao Tong University Nov. 22, 2019 Algorithms for Big Data (X) 1/10 Today we will introduce a Monte-Carlo algorithm to approximate Matrix Multiplication . Algorithms for Big Data (X) . .

528 views • 32 slides
Approximate Search and Data Reduction Algorithms Research Questions Kyle Porter NTNU Gjvik

Approximate Search and Data Reduction Algorithms Research Questions Kyle Porter NTNU Gjvik

Approximate Search and Data Reduction Algorithms Research Questions Kyle Porter NTNU Gjvik Outline of Presentation Introduction: Problems General Goals Research Questions Brief theoretical/practical background

243 views • 21 slides
New Algorithms for Approximate Minimization of the Difference Between Submodular Functions, with

New Algorithms for Approximate Minimization of the Difference Between Submodular Functions, with

Background v = f g Procedures for min f g Additional Theoretical Results Experiments Summary New Algorithms for Approximate Minimization of the Difference Between Submodular Functions, with Applications Rishabh Iyer Jeff Bilmes

644 views • 37 slides
Quantum and Classical Algorithms for Approximate Submodular Function Minimization Yassine

Quantum and Classical Algorithms for Approximate Submodular Function Minimization Yassine

Quantum and Classical Algorithms for Approximate Submodular Function Minimization Yassine Hamoudi, Patrick Rebentrost, Ansis Rosmanis, Miklos Santha arXiv: 1907.05378 1. Approximate Submodular Function Minimization 2. Quantum speed-up for

994 views • 98 slides
Quantum and Classical Algorithms for Approximate Submodular Function Minimization Yassine

Quantum and Classical Algorithms for Approximate Submodular Function Minimization Yassine

Quantum and Classical Algorithms for Approximate Submodular Function Minimization Yassine Hamoudi, Patrick Rebentrost, Ansis Rosmanis, Miklos Santha arXiv: 1907.05378 1. Approximate Submodular Function Minimization 2. Quantum speed-up for

1.54k views • 113 slides
Counting and sampling algorithms at low temperature New frontiers in approximate counting STOC

Counting and sampling algorithms at low temperature New frontiers in approximate counting STOC

Counting and sampling algorithms at low temperature New frontiers in approximate counting STOC 2020 Will Perkins (UIC) Algorithms and phase transitions When are phase transitions barriers to e ffi cient algorithms? What algorithmic

497 views • 30 slides
Approximate Implicitization and CAD- type intersection algorithms Tor Dokken and Vibeke Skytt

Approximate Implicitization and CAD- type intersection algorithms Tor Dokken and Vibeke Skytt

Approximate Implicitization and CAD- type intersection algorithms Tor Dokken and Vibeke Skytt SINTEF ICT, Department of Applied Mathematics This work is partly funded by the European Union through the projects: GAIA II, IST-2001-35512,

437 views • 28 slides
Approximate methods for scalable data mining Andrew Clegg Data Analytics &amp; Visualization

Approximate methods for scalable data mining Andrew Clegg Data Analytics &amp; Visualization

Approximate methods for scalable data mining Andrew Clegg Data Analytics &amp; Visualization Team Pearson Technology Twitter: @andrew_clegg Outline 1. Intro 2. What are approximate methods and why are they cool? 3. Set membership (finding

683 views • 41 slides
Approximate Search of Regular Expressions Using Bit-Parallel Algorithms Kristo Tammeoja Jaak Vilo

Approximate Search of Regular Expressions Using Bit-Parallel Algorithms Kristo Tammeoja Jaak Vilo

Approximate Search of Regular Expressions Using Bit-Parallel Algorithms Kristo Tammeoja Jaak Vilo Teooriapevad Ruges, 2007 Contents Regular expression (RE) syntax Glushkovs automaton Existing bit-parallel algorithms Exact

817 views • 52 slides
Approximate inference: Sampling methods Probabilistic Graphical Models Sharif University of

Approximate inference: Sampling methods Probabilistic Graphical Models Sharif University of

Approximate inference: Sampling methods Probabilistic Graphical Models Sharif University of Technology Spring 2018 Soleymani Approximate inference Approximate inference techniques Deterministic approximation Variational algorithms

1.95k views • 97 slides
Dynamic Inverse Problems: Schmitt Efficient Algorithms and Approximate Inverse Problems

Dynamic Inverse Problems: Schmitt Efficient Algorithms and Approximate Inverse Problems

Inverse Problems A.K. Louis, U. Dynamic Inverse Problems: Schmitt Efficient Algorithms and Approximate Inverse Problems Definitions Inverse Two Constraints Semi - discrete Case Examples A.K. Louis 1 U. Schmitt 2 Approximate Inverse 1

541 views • 29 slides
Probabilistic &amp; Unsupervised Learning Convex Algorithms in Approximate Inference Maneesh

Probabilistic &amp; Unsupervised Learning Convex Algorithms in Approximate Inference Maneesh

Probabilistic &amp; Unsupervised Learning Convex Algorithms in Approximate Inference Maneesh Sahani maneesh@gatsby.ucl.ac.uk Gatsby Computational Neuroscience Unit, and MSc ML/CSML, Dept Computer Science University College London Term 1,

861 views • 41 slides
Projections for Approximate Policy Iteration Algorithms Riad Akrour , Joni Pajarinen, Gerhard

Projections for Approximate Policy Iteration Algorithms Riad Akrour , Joni Pajarinen, Gerhard

Projections for Approximate Policy Iteration Algorithms Riad Akrour , Joni Pajarinen, Gerhard Neumann, Jan Peters IAS, TU Darmstadt, Germany ICML19 Entropy Regularization in RL Widespread with actor-critic methods ICML19 Hard vs Soft

374 views • 6 slides
Statistical Algorithmic Profiling for Randomized Approximate Programs Keyur Joshi , Vimuth

Statistical Algorithmic Profiling for Randomized Approximate Programs Keyur Joshi , Vimuth

CCF-1629431 CCF-1703637 Statistical Algorithmic Profiling for Randomized Approximate Programs Keyur Joshi , Vimuth Fernando, Sasa Misailovic University of Illinois at Urbana-Champaign ICSE 2019 Randomized Approximate Algorithms Modern

609 views • 42 slides
Chapter 3: Top-k Query Processing and Indexing 3.1 Top-k Algorithms 3.2 Approximate Top-k Query

Chapter 3: Top-k Query Processing and Indexing 3.1 Top-k Algorithms 3.2 Approximate Top-k Query

Chapter 3: Top-k Query Processing and Indexing 3.1 Top-k Algorithms 3.2 Approximate Top-k Query Processing 3.3 Index Access Scheduling 3.4 Index Organization and Advanced Query Types 3-1 IRDM WS 2005 3.1 Top-k Query Processing with Scoring

592 views • 36 slides
Autoplacer : Scalable Self-Tuning Data Placement in Distributed Key-value Stores ICAC13 Jo

Autoplacer : Scalable Self-Tuning Data Placement in Distributed Key-value Stores ICAC13 Jo

Autoplacer : Scalable Self-Tuning Data Placement in Distributed Key-value Stores ICAC13 Jo ao Paiva , Pedro Ruivo, Paolo Romano, Lu s Rodrigues Instituto Superior T ecnico / Inesc-ID, Lisboa, Portugal June 27, 2013 Outline

605 views • 48 slides
Using OpenACC to parallelize irregular computation (Session:S7478) Sunita Chandrasekaran Arnov

Using OpenACC to parallelize irregular computation (Session:S7478) Sunita Chandrasekaran Arnov

Using OpenACC to parallelize irregular computation (Session:S7478) Sunita Chandrasekaran Arnov Sinha (arnov@udel.edu) (schandra@udel.edu) M.S. (Graduating Summer17) Assistant Professor University of Delaware, DE, USA May 10, GTC 2017,

436 views • 22 slides
Scalable Content- Addressable Network Eireann Leverett How Torus We use a Torus because it is

Scalable Content- Addressable Network Eireann Leverett How Torus We use a Torus because it is

Scalable Content- Addressable Network Eireann Leverett How Torus We use a Torus because it is un- Dimensions ending in each Nodes dimension. It is a Hashing circle where the last Realities address neighbours Zone takeover the first,

922 views • 10 slides
How to Construct State Registries Matching State registry Na ve solution Undeniability with

How to Construct State Registries Matching State registry Na ve solution Undeniability with

Secure registries M. Kutyowski How to Construct State Registries Matching State registry Na ve solution Undeniability with Public Security Our solution Mirosaw Kutyowski joint work with Przemysaw Kubiak and Jun Shao

330 views • 18 slides
Detecting Hidden Anomalies in DNS Communication CZ.NIC Ondrej Mikle-Barat / ondrej.mikle@nic.cz

Detecting Hidden Anomalies in DNS Communication CZ.NIC Ondrej Mikle-Barat / ondrej.mikle@nic.cz

Detecting Hidden Anomalies in DNS Communication CZ.NIC Ondrej Mikle-Barat / ondrej.mikle@nic.cz Karel Slan / karel.slany@nic.cz 18. 10. 2011 1 Outline Motivation Method description original work algorithm DNS specifics

366 views • 18 slides
Ahoy: A Proximity-Based Discovery Protocol Robbert Haarman Contents 1. Introduction to Ahoy 2.

Ahoy: A Proximity-Based Discovery Protocol Robbert Haarman Contents 1. Introduction to Ahoy 2.

Ahoy: A Proximity-Based Discovery Protocol Robbert Haarman Contents 1. Introduction to Ahoy 2. Protocol Overview 3. Message Types 4. Attenuated Bloom Filters 5. My Contributions 6. Summary 7. Questions Part 1 Introduction to Ahoy What is

1.23k views • 71 slides
New Curves in DNSSEC Ond ej Sur, CZ.NIC SafeCurves(.cr.yp.to) Work by Daniel J. Bernstein

New Curves in DNSSEC Ond ej Sur, CZ.NIC SafeCurves(.cr.yp.to) Work by Daniel J. Bernstein

New Curves in DNSSEC Ond ej Sur, CZ.NIC SafeCurves(.cr.yp.to) Work by Daniel J. Bernstein and Tanja Lange Difference between security of: Elliptic-Curve Cryptography (ECC) Elliptic-Curve Discrete-Logarithm Problem (ECDLP)

384 views • 6 slides
Message-locked Encryption with Deduplication Consistency Sbastien Canard 1 , Fabien Laguillaumie

Message-locked Encryption with Deduplication Consistency Sbastien Canard 1 , Fabien Laguillaumie

Message-locked Encryption with Deduplication Consistency Sbastien Canard 1 , Fabien Laguillaumie 2 and Marie Paindavoine 1,2 1 Orange Labs, Applied Crypto Group, France. 2 Universit Claude Bernard Lyon 1, LIP (CNRS/ENSL/INRIA/UCBL), France.

256 views • 11 slides

More recommend