big data architectures facebook
play

Big Data Architectures@ Facebook QCon London 2012 Ashish Thusoo - PowerPoint PPT Presentation

Aug 02, 2023 •32 likes •572 views

Big Data Architectures@ Facebook QCon London 2012 Ashish Thusoo Thursday, March 8, 12 Outline Big Data @ Facebook - Scope & Scale Evolution of Big Data Architectures @ FB Past, Present and Future Questions Thursday, March

  1. Big Data Architectures@ Facebook QCon London 2012 Ashish Thusoo Thursday, March 8, 12

  2. Outline • Big Data @ Facebook - Scope & Scale • Evolution of Big Data Architectures @ FB • Past, Present and Future • Questions Thursday, March 8, 12

  3. Big Data @ FB: Scale • 25 PB of compressed data • equivalent to 300 years of HD-TV video Thursday, March 8, 12

  4. Big Data @ FB: Scale • 150 PB of uncompressed data • equivalent to 3 x the entire written works of mankind from the beginning of recorded history in all languages Thursday, March 8, 12

  5. Big Data @ FB: Scale • 400 TB/day (uncompressed) of new data • That is a lot of disks Thursday, March 8, 12

  6. Big Data @ FB: Scope • Simple reporting • Model generation • Adhoc analysis + data science • Index generation • Many many others... Thursday, March 8, 12

  7. A/B Testing Email #1 Thursday, March 8, 12

  8. A/B Testing Email #2 Thursday, March 8, 12

  9. A/B Testing Email #2 is 3x Better Thursday, March 8, 12

  10. Friend Map By Paul Butler - https://www.facebook.com/notes/facebook-engineering/visualizing-friendships/ 469716398919 Thursday, March 8, 12

  11. Big Data @ FB: Scope • one new job every second • ~ 15% of the company uses the clusters Thursday, March 8, 12

  12. Evolution: 2007-2011 DW Size in TB 30000 25000 22500 15000 8000 7500 800 250 15 0 2007 2008 2009 2010 2011 Thursday, March 8, 12

  13. 2007: Traditional EDW Thursday, March 8, 12

  14. 2007: Traditional EDW Web Clusters MySQL Clusters Thursday, March 8, 12

  15. 2007: Traditional EDW Web Clusters RDBMS Data Warehouse MySQL Clusters Thursday, March 8, 12

  16. 2007: Traditional EDW Scribe Mid-Tier Web Clusters RDBMS Data Warehouse MySQL Clusters Thursday, March 8, 12

  17. 2007: Traditional EDW Scribe Mid-Tier Web Clusters NAS Filers RDBMS Data Warehouse MySQL Clusters Thursday, March 8, 12

  18. 2007: Traditional EDW Scribe Mid-Tier Summarization Cluster Web Clusters NAS Filers RDBMS Data Warehouse MySQL Clusters Thursday, March 8, 12

  19. 2007: Pain Points Scribe Mid-Tier Web Clusters Summarization Cluster NAS Filers MySQL Clusters RDBMS Data Warehouse Thursday, March 8, 12

  20. 2007: Pain Points Scribe Mid-Tier Web Clusters Summarization Cluster NAS Filers MySQL Clusters - daily ETL > 24 hours RDBMS Data Warehouse Thursday, March 8, 12

  21. 2007: Pain Points Scribe Mid-Tier Web Clusters Summarization Cluster NAS Filers MySQL Clusters - daily ETL > 24 hours - Lots of tuning/indexes etc. RDBMS Data Warehouse Thursday, March 8, 12

  22. 2007: Pain Points Scribe Mid-Tier Web Clusters Summarization Cluster NAS Filers MySQL Clusters - daily ETL > 24 hours - Lots of tuning/indexes etc. - Lots of hardware planning RDBMS Data Warehouse Thursday, March 8, 12

  23. 2007: Pain Points - compute close to storage (early map/reduce) Scribe Mid-Tier Web Clusters Summarization Cluster NAS Filers MySQL Clusters - daily ETL > 24 hours - Lots of tuning/indexes etc. - Lots of hardware planning RDBMS Data Warehouse Thursday, March 8, 12

  24. 2007: Limitations • Most use cases were in business metrics - data science, model building etc. not possible • Only summary data was stored online - details archived away Thursday, March 8, 12

  25. 2008: Move to Hadoop Scribe Mid-Tier Summarization Cluster Web Clusters NAS Filers RDBMS Data Warehouse MySQL Clusters Thursday, March 8, 12

  26. 2008: Move to Hadoop Batch Scribe Mid-Tier copier/ Web Clusters loaders Hadoop/Hive Data Warehouse NAS Filers MySQL Clusters RDBMS Data Mart Thursday, March 8, 12

  27. 2008: Immediate Pros • Data science at scale became possible • For the first time all of the instrumented data could be held online • Use cases expanded Thursday, March 8, 12

  28. 2009: Democratizing Data Scribe Mid-Tier Web Clusters Hadoop/Hive Data Warehouse NAS Filers MySQL Clusters RDBMS Data Mart Thursday, March 8, 12

  29. 2009: Democratizing Data Databee & Nectar: Chronos: Data instrumentation & Pipeline schema aware Framework data collection HiPal: Adhoc Scrapes: Hadoop/Hive Data Queries + Data Configuration Warehouse Discovery Driven Thursday, March 8, 12

  30. 2009: Democratizing Data(Nectar) • Typical Nectar Pipeline • Simple schema evolution built in • json encoded short term data • decomposing json for long term storage Thursday, March 8, 12

  31. 2009: Democratizing Data (Tools) • HiPal - data discovery and query authoring • Charting and dashboard generation tools Thursday, March 8, 12

  32. 2009: Democratizing Data (Tools) • Databee: Workflow language • Chronos: Scheduling tool Thursday, March 8, 12

  33. 2009: Cons of Democratization • Isolation to protect against Bad Jobs • Fair sharing of the cluster - what is a high priority job and how to enforce it Thursday, March 8, 12

  34. 2010: Controlling Chaos • Isolation • Reducing operational overhead • Better resource utilization • Measurement, ownership, accountability Thursday, March 8, 12

  35. 2010: Isolation Scribe Mid-Tier Web Clusters Hadoop/Hive Data Warehouse NAS Filers MySQL Clusters Thursday, March 8, 12

  36. 2010: Isolation Scribe Mid-Tier Web Clusters Platinum Warehouse Hive Replication NAS Filers MySQL Clusters Silver Warehouse Thursday, March 8, 12

  37. 2010: Ops Efficiency Scribe HDFS Web Clusters Platinum Warehouse Hive Replication MySQL Clusters Silver Warehouse Thursday, March 8, 12

  38. 2010: Ops Efficiency Scribe HDFS Web Clusters Platinum Warehouse Hive Replication near real time data consumers MySQL Clusters Silver Warehouse Thursday, March 8, 12

  39. 2010: Ops Efficiency Scribe HDFS Web Clusters ptail: Platinum Warehouse parallel tail Hive Replication on hdfs near real time data consumers MySQL Clusters Silver Warehouse Thursday, March 8, 12

  40. 2010: Resource Utilization (Disk) • HDFS-RAID: from 3 replicas to 2.2 replicas • RCFile: Row columnar format for compressing Hive tables Thursday, March 8, 12

  41. 2010: Resource Utilization (CPU) • Continuous copier/ loaders • Incremental scrapes • Hive optimizations to save CPU Thursday, March 8, 12

  42. 2010: Monitoring(SLAs) • Per job statistics rolled up to owner/group/team • Expected time of arrival vs Actual time of arrival of data • Simple data quality metrics Thursday, March 8, 12

  43. 2011: New Requirements • More real time requirements for aggregations • Optimizing resource utilization Thursday, March 8, 12

  44. 2011: Beyond Hadoop • Puma for real time analytics • Peregrine for simple and fast queries Thursday, March 8, 12

  45. 2010: Puma Scribe HDFS Web Clusters ptail: Platinum Warehouse parallel tail Hive Replication on hdfs near real time data consumers MySQL Clusters Silver Warehouse Thursday, March 8, 12

  46. 2010: Puma Scribe HDFS Web Clusters ptail: Platinum Warehouse parallel tail Hive Replication on hdfs near real time data consumers MySQL Clusters Silver Warehouse Thursday, March 8, 12

  47. 2010: Puma Thursday, March 8, 12

  48. 2010: Puma Scribe HDFS ptail: parallel tail on hdfs Thursday, March 8, 12

  49. 2010: Puma Scribe HDFS ptail: parallel tail on hdfs Puma Clusters Thursday, March 8, 12

  50. 2010: Puma Scribe HDFS ptail: parallel tail on hdfs Puma Clusters Hbase Cluster Thursday, March 8, 12

  51. Other Challenges Of HyperGrowth • Moving data centers • Moving sustainably fast Thursday, March 8, 12

  52. HyperGrowth - Moving Data Centers DW Size in TB 30000 25000 22500 15000 8000 7500 800 250 15 0 2007 2008 2009 2010 2011 Thursday, March 8, 12

  53. HyperGrowth - Moving Data Centers • Moved 20 PB of data • Leverage replication with fast switch • 2-3 months to accomplish the entire move Blog Post on FB by Paul Yang: http://www.facebook.com/notes/paul-yang/moving-an-elephant-large- scale-hadoop-data-migration-at-facebook/10150246275318920 Thursday, March 8, 12

  54. Questions Contact Information: ashish.thusoo@gmail.com http://www.linkedin.com/pub/ashish-thusoo/0/5a8/50 https://www.facebook.com/athusoo https://twitter.com/ashishthusoo Thursday, March 8, 12

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Facebook Exchange Facebook Exchange (FBX) (FBX) Facebook Exchange The Facebook Exchange allows

Facebook Exchange Facebook Exchange (FBX) (FBX) Facebook Exchange The Facebook Exchange allows

Facebook Exchange Facebook Exchange (FBX) (FBX) Facebook Exchange The Facebook Exchange allows programmatic buying of Facebook ad inventory through real-time bidding. Agencies, trading desks and direct marketers can buy Facebook ads using

343 views • 7 slides
Facebook and Cambridge Analytica $1M to buy the data 50 million profiles Analyzing

Facebook and Cambridge Analytica $1M to buy the data 50 million profiles Analyzing

Big data - the large volume of data both structured and unstructured that inundates a business on a day-to-day basis. Big data can be analyzed for insights that lead to better decisions and strategic business moves Facebook and Cambridge

680 views • 38 slides
Facebook Strategies Facebook www.facebook.com Facebook TIPS Idea #1: Share the School Calendar.

Facebook Strategies Facebook www.facebook.com Facebook TIPS Idea #1: Share the School Calendar.

Facebook Strategies Facebook www.facebook.com Facebook TIPS Idea #1: Share the School Calendar. Idea #2: Link to Positive Stories. Idea #3: Post Photos of Events. Idea #4: Congratulate Students and Staff Who Achieve. Facebook TIPS Going

427 views • 39 slides
Facebook 101 FACEBOOK 101 What is Facebook? Facebooks Mission is to give people the

Facebook 101 FACEBOOK 101 What is Facebook? Facebooks Mission is to give people the

Facebook 101 FACEBOOK 101 What is Facebook? Facebooks Mission is to give people the power to share and make the world more open and connected. Mission Statement Facts: More than 500 million active users 50% of our

202 views • 17 slides
Device I/O I/O architectures: busses 10A I/O Architectures 10B I/O Mechanisms CPU

Device I/O I/O architectures: busses 10A I/O Architectures 10B I/O Mechanisms CPU

Device I/O I/O architectures: busses 10A I/O Architectures 10B I/O Mechanisms CPU controller 10C Disks control address main bus 10D Low Level I/O Techniques data interrupts 10E Higher Level I/O Techniques memory

657 views • 16 slides
Device I/O I/O architectures: busses 10A I/O Architectures 10B I/O Mechanisms CPU

Device I/O I/O architectures: busses 10A I/O Architectures 10B I/O Mechanisms CPU

Device I/O I/O architectures: busses 10A I/O Architectures 10B I/O Mechanisms CPU controller 10C Disks control address main bus 10D Low-Level I/O Techniques data interrupts 10E Higher Level I/O Techniques memory

493 views • 17 slides
Hive* A Petabyte Scale Data Warehouse Using Hadoop Authors Facebook Data Infrastructure

Hive* A Petabyte Scale Data Warehouse Using Hadoop Authors Facebook Data Infrastructure

Hive* A Petabyte Scale Data Warehouse Using Hadoop Authors Facebook Data Infrastructure Team CS 743, Fall 2014 Conference Data Engineering (ICDE), 2010 IEEE UNIVERSITY OF WATERLOO Presenter Malek NAOUACH, Nets&Dist Sys November

317 views • 13 slides
A D A C C O U N T SET-UP PROCESS Facebook: 3 STEP SET-UP 1. Facebook Ad Account 2.

A D A C C O U N T SET-UP PROCESS Facebook: 3 STEP SET-UP 1. Facebook Ad Account 2.

A D A C C O U N T SET-UP PROCESS Facebook: 3 STEP SET-UP 1. Facebook Ad Account 2. Facebook Business Page 3. Instagram FACEBOOK & INSTAGRAM SET-UP: Without Business Manager To create, manage and optimize your campaigns, SYVO

563 views • 20 slides
FACEBOOK July 12, 2009 JGS of the Conejo Valley and Ventura County Who or What Is Facebook?

FACEBOOK July 12, 2009 JGS of the Conejo Valley and Ventura County Who or What Is Facebook?

FACEBOOK July 12, 2009 JGS of the Conejo Valley and Ventura County Who or What Is Facebook? Facebook was started at Harvard in 2004 for students, by student Mark Zuckerberg Facebook was so popular that many tried to purchase the

867 views • 23 slides
One System To Fit Them All: Shared MySQL Hosting At Facebook Andrew Regner Production

One System To Fit Them All: Shared MySQL Hosting At Facebook Andrew Regner Production

One System To Fit Them All: Shared MySQL Hosting At Facebook Andrew Regner Production Engineer | MySQL Infrastructure Data choices @Facebook Everyone has data to persist Also have: ZippyDB, ODS, Scuba, HBase, Scribe, RocksDB,

590 views • 45 slides
MY FACEBOOK STORY How I got started marketing on Facebook How I built an overseas

MY FACEBOOK STORY How I got started marketing on Facebook How I built an overseas

HOW TO BUILD THE ULTIMATE FACEBOOK LEAD MACHINE MCGRAW-HILL INTERNATIONAL BEST-SELLING AUTHOR JAMES E. DICKS JR. CEO, DNA PULSE, LLC MY FACEBOOK STORY How I got started marketing on Facebook How I built an overseas business with

1.06k views • 67 slides
Scuba: Diving into Data at Facebook Presenter: Lavanya Subramanian 1 Need for Data Analysis

Scuba: Diving into Data at Facebook Presenter: Lavanya Subramanian 1 Need for Data Analysis

Scuba: Diving into Data at Facebook Presenter: Lavanya Subramanian 1 Need for Data Analysis Performance monitoring Detect unexpected performance drops/rises Pattern mining Understand user response to new features Ad revenue

605 views • 17 slides
Realtime Data Processing at Facebook Abhay Venkatesh Actionable reports Why e.g. Chorus:

Realtime Data Processing at Facebook Abhay Venkatesh Actionable reports Why e.g. Chorus:

Realtime Data Processing at Facebook Abhay Venkatesh Actionable reports Why e.g. Chorus: what is trending right now? Realtime monitoring Streaming at e.g. dashboard queries Facebook? Hybrid realtime-batch pipelines e.g.

301 views • 18 slides
Architectures Architectural styles Software architectures Architectures versus middleware

Architectures Architectural styles Software architectures Architectures versus middleware

Architectures Architectures Architectural styles Software architectures Architectures versus middleware Self-management in distributed systems 1 / 26 Architectures Architectural styles Architectural styles Basic idea Organize into

571 views • 33 slides
WHY FACEBOOK? AND HOW TO USE IT TO TAKE OVER THE WORLD 27 July 2015 WHO IS JAMIE WONG? IN THE

WHY FACEBOOK? AND HOW TO USE IT TO TAKE OVER THE WORLD 27 July 2015 WHO IS JAMIE WONG? IN THE

WHY FACEBOOK? AND HOW TO USE IT TO TAKE OVER THE WORLD 27 July 2015 WHO IS JAMIE WONG? IN THE NEXT 15 MINUTES Wi y Facebook? Wi y Facebook isnt really free Types of Paid Advertising Wi at makes good content Useful sites WHY FACEBOOK ?

459 views • 14 slides
Facebook Basics Hannah Digital Literacy Specialist Skokie Public Library What is Facebook?

Facebook Basics Hannah Digital Literacy Specialist Skokie Public Library What is Facebook?

Facebook Basics Hannah Digital Literacy Specialist Skokie Public Library What is Facebook? Facebook is a popular free social networking website launched in 2004 by founder Mark Zuckerberg. Facebook is reported to have 2 billion

653 views • 15 slides
ATTACK: Learning the Distributions of Adversarial Examples for an Improved Black-Box Attack on

ATTACK: Learning the Distributions of Adversarial Examples for an Improved Black-Box Attack on

ATTACK: Learning the Distributions of Adversarial Examples for an Improved Black-Box Attack on Deep Neural Networks Yandong Li* 1 Lijun Li* 1 Liqiang Wang 1 Tong Zhang 2 Boqing Gong 3 *Equal Contribution 1 University of Central Florida 2 Hong

440 views • 11 slides
The Aspect-Oriented Design of the ++ Parser Framework Puma C/C Matthias Urban Daniel Lohmann

The Aspect-Oriented Design of the ++ Parser Framework Puma C/C Matthias Urban Daniel Lohmann

The Aspect-Oriented Design of the ++ Parser Framework Puma C/C Matthias Urban Daniel Lohmann Olaf Spinczyk pure systems GmbH Friedrich-Alexander University Technical University Magdeburg Erlangen-Nuremberg Dortmund 9th International

595 views • 56 slides
Introduction to Robotics Jianwei Zhang zhang@informatik.uni-hamburg.de Universit at Hamburg

Introduction to Robotics Jianwei Zhang zhang@informatik.uni-hamburg.de Universit at Hamburg

MIN-Fakult at Department Informatik Universit at Hamburg - Introduction to Robotics Introduction to Robotics Jianwei Zhang zhang@informatik.uni-hamburg.de Universit at Hamburg Fakult at f ur Mathematik, Informatik und

511 views • 19 slides
Taking the PUMA E P&L to the next level Stefan Seidel Teamhead PUMA.Safe Ecology

Taking the PUMA E P&L to the next level Stefan Seidel Teamhead PUMA.Safe Ecology

Taking the PUMA E P&L to the next level Stefan Seidel Teamhead PUMA.Safe Ecology Sustainable Innovation 2012 October 30th 2012 PUMA PUMA PUMA is the leading Sportlifestyle company with products which start in Sport and end in

324 views • 14 slides
On Stability theory for C 0 -Semigroups and applications Francis Flix Crdova Puma

On Stability theory for C 0 -Semigroups and applications Francis Flix Crdova Puma

C 0 -Semigroup Stability Theory Applications Stabilization Results - Ideas of demonstrations On Stability theory for C 0 -Semigroups and applications Francis Flix Crdova Puma Departamento de Cincias Exatas e Educao / UFSC 23 06

1.01k views • 83 slides
Formal Methods for Interactive Systems Part 8 Cognitive Architectures Antonio Cerone

Formal Methods for Interactive Systems Part 8 Cognitive Architectures Antonio Cerone

Formal Methods for Interactive Systems Part 8 Cognitive Architectures Antonio Cerone United Nations University International Institute for Software Technology Macau SAR China email: antonio@iist.unu.edu web: www.iist.unu.edu A. Cerone,

1.42k views • 137 slides
A Generic Framework for Interprocedural Analysis of Numerical Properties + Markus Mller-Olm

A Generic Framework for Interprocedural Analysis of Numerical Properties + Markus Mller-Olm

A Generic Framework for Interprocedural Analysis of Numerical Properties + Markus Mller-Olm Helmut Seidl + Mnster Mnchen PUMA Ringvorlesung, 2009 1 Outline: Background The framework for affine relations Grangers

1.53k views • 113 slides
CCDSC 2016 10/4/2016 Equivalent platforms for unmodified application c o r e Application

CCDSC 2016 10/4/2016 Equivalent platforms for unmodified application c o r e Application

Tiziano Passerini, Jaroslaw Slawinski, Umberto Villa, Sofia Guzzetti Alessandro Veneziani, Vaidy Sunderam Mathematics & Computer Science Emory University, Atlanta, USA CCDSC 2016 10/4/2016 Equivalent platforms for unmodified application c

447 views • 18 slides

More recommend