Hadoop: Scalable Infrastructure for Big Data QCon London 2012 - PowerPoint PPT Presentation

Nov 06, 2022 •239 likes •598 views

Hadoop: Scalable Infrastructure for Big Data QCon London 2012 Parand Tony Darugar Founder and CEO, Xpenser parand@xpenser.com QCon London 2012 What is Hadoop? QCon London 2012 Hadoop is the Linux of Big Data Processing QCon London 2012

Hadoop: Scalable Infrastructure for Big Data QCon London 2012 Parand Tony Darugar Founder and CEO, Xpenser parand@xpenser.com QCon London 2012
What is Hadoop? QCon London 2012
Hadoop is the Linux of Big Data Processing QCon London 2012
Infrastructure for Large Scale Computation & Data Processing on a network of Commodity Hardware. QCon London 2012
Why Hadoop? QCon London 2012
Scale QCon London 2012
Cost QCon London 2012
Freedom QCon London 2012
Does Anyone Use Hadoop? QCon London 2012
eHarmony IBM Zion's bank VISA NY Times Microsoft Twitter Facebook eBay Yahoo LinkedIn AOL ... ... QCon London 2012
Alternatives Build your own Get creative with RDBMS architecture QCon London 2012
What's the idea? QCon London 2012
Commodity Hardware Distributed Operation QCon London 2012
Wisdom: Embrace Failure (hardware) Be Resilient (software) QCon London 2012
What's in the box? QCon London 2012
Hadoop Distributed File System QCon London 2012
Distributed Computation Framework QCon London 2012
Map-Reduce Programming Model QCon London 2012
HDFS ● Your data in triplicate ● Built-in resiliency to large scale failures ● Intelligent Data Distribution ● Very large data sizes QCon London 2012
Distributed Computation ● Built-in resiliency to large scale failures ● Distribute work to workers, collect results from fastest ● Move computation to data (not data to computation) QCon London 2012
Map Reduce Very simple programming model: Map(anything)->key, value Sort, partition on key Reduce(key,value)->key, value No parallel processing or message passing semantics Programmable in Java or any other language (streaming) QCon London 2012
Ecosystem HBase: NoSQL BigTable clone Hive: Somewhat-SQL data store Pig: SQL-like programming model Chukwa, Scribe, Mahoot, Cassandra, Oozie, Sqoop, ... QCon London 2012
Commercial Support Cloudera HortonWorks IBM ... QCon London 2012
How? Try it in non-distributed mode Try it on a few spare machines Try it on EC2 Try it! http://hadoop.apache.org/ QCon London 2012
Case Studies QCon London 2012
eHarmony QCon London 2012
Biz360 (Attensity) QCon London 2012
Yahoo! QCon London 2012
You! QCon London 2012
Start with ETL QCon London 2012
Start with batch, non time-critical tasks QCon London 2012
Start with storing your large data on HDFS QCon London 2012
Move batch processing to Hadoop Serve from RDBMS QCon London 2012
Embrace. Be One With The Hadoop. QCon London 2012
Questions? Parand Tony Darugar parand@xpenser.com QCon London 2012

Download Presentation

Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

SAS Data Loader for Hadoop Agenda Intro What is Hadoop? What do I get from Hadoop?

SAS Data Loader for Hadoop Agenda Intro What is Hadoop? What do I get from Hadoop? Hadoop components Why SAS Data Loader for Hadoop? SAS Data Loader for Hadoop overview Demo Introduction Doug Cutting, creator of Hadoop

285 views • 11 slides

Systems Infrastructure for Data Science Web Science Group Uni Freiburg WS 2014/15 Hadoop

Systems Infrastructure for Data Science Web Science Group Uni Freiburg WS 2014/15 Hadoop Evolution and Ecosystem Hadoop Map/Reduce has been an incredible success, but not everybody is happy with it 3 DB Community: Criticisms of

685 views • 57 slides

Hive* A Petabyte Scale Data Warehouse Using Hadoop Authors Facebook Data Infrastructure

Hive* A Petabyte Scale Data Warehouse Using Hadoop Authors Facebook Data Infrastructure Team CS 743, Fall 2014 Conference Data Engineering (ICDE), 2010 IEEE UNIVERSITY OF WATERLOO Presenter Malek NAOUACH, Nets&Dist Sys November

317 views • 13 slides

Scalable Data Science with Hadoop, Spark and R Mario Inchiosa, PhD Principal Software Engineer

Scalable Data Science with Hadoop, Spark and R Mario Inchiosa, PhD Principal Software Engineer Microsoft Data Group DSC 2016 July 2, 2016 Microsoft R Server Cloud Hadoop & Spark R Server portfolio R Server Technology EDW RDBMS

475 views • 23 slides

BinaryPig: Scalable Binary Data Extraction in Hadoop Created By: Jason Trost, Telvis Calhoun,

BinaryPig: Scalable Binary Data Extraction in Hadoop Created By: Jason Trost, Telvis Calhoun, Zach Hanif Bringing data science to cyber security, allowing you to sense, analyze and act in

804 views • 44 slides

COMP9313: Big Data Management Hadoop and HDFS Hadoop Apache Hadoop is an open-source

COMP9313: Big Data Management Hadoop and HDFS Hadoop Apache Hadoop is an open-source software framework that Stores big data in a distributed manner Processes big data parallelly Builds on large clusters of commodity hardware.

2.93k views • 60 slides

Hadoop Infrastructure @Uber Past , Present and Future Mayank Bansal U B E R | Data Ubers

Hadoop Infrastructure @Uber Past , Present and Future Mayank Bansal U B E R | Data Ubers Mission Transporta=on as reliable as running water , everywhere, for everyone 75+ Countries 500+ Ci=es And growing U B E R | Data How

858 views • 31 slides

Rule Based Classification on a Multi Node Scalable Hadoop Cluster Shashank Gugnani Devavrat

Rule Based Classification on a Multi Node Scalable Hadoop Cluster Shashank Gugnani Devavrat Khanolkar BITS Pilani Tushar Bihany K K Birla Goa Campus Nikhil Khadilkar Data Hypergrowth Reuters-21578: about 10K docs (ModApte) Bekkerman

752 views • 15 slides

The Evolution of Hadoop at Spotify Rafal Wojdyla (rav@spotify.com) Josh Baer (jbx@spotify.com)

The Evolution of Hadoop at Spotify Rafal Wojdyla (rav@spotify.com) Josh Baer (jbx@spotify.com) @l_phant @ravwojdyla Technical Product Owner Data Engineer Data Infrastructure Hadoop Team Overview Growing Pains Gaining Focus The

945 views • 62 slides

CLUSTER CONTINUOUS DELIVERY WITH OOZIE Clay Baenziger Bloomberg Hadoop Infrastructure

CLUSTER CONTINUOUS DELIVERY WITH OOZIE Clay Baenziger Bloomberg Hadoop Infrastructure ApacheCon Big Data 5/18/2017 ABOUT BLOOMBERG 2 BIG DATA AT BLOOMBERG Bloomberg quickly and accurately delivers business and financial information,

713 views • 30 slides

Big Data with R and Hadoop Jamie F Olson June 11, 2015 ; R and Hadoop Review various tools

Big Data with R and Hadoop Jamie F Olson June 11, 2015 ; R and Hadoop Review various tools for leveraging Hadoop from R. MapReduce Spark Hive/Impala Revolution R . . . . . . . . . . . . . . . . . . . . . . . . . .

565 views • 52 slides

Apache Flume Getting data into Hadoop Problem Getting data into HDFS is not difficult: %

Apache Flume Getting data into Hadoop Problem Getting data into HDFS is not difficult: % hadoop fs --put data.csv . works great when data is neatly packaged and ready to upload Unfortunately, e.g. a webserver is creating data

495 views • 24 slides

Large-Scale Data Engineering Hadoop MapReduce in more detail event.cwi.nl/lsde2015 How will I

Large-Scale Data Engineering Hadoop MapReduce in more detail event.cwi.nl/lsde2015 How will I actually learn Hadoop? This class session Hadoop: The Definitive Guide RTFM There is a lot of material out there There is also a lot

265 views • 26 slides

Stratosphere for Hadoop Users Potsdam, January 03, 2012 Arvid Heise Outline 2 1 Overview over

Stratosphere for Hadoop Users Potsdam, January 03, 2012 Arvid Heise Outline 2 1 Overview over Stratosphere 2 Dataflow Orientation 3 Tuple-based Data Model 4 Other Differences 5 Seminar Organization Arvid Heise | Scalable Data Analysis

482 views • 45 slides

S9334: Building And Managing Scalable AI Infrastructure With NVIDIA DGX Pod And DGX Pod

S9334: Building And Managing Scalable AI Infrastructure With NVIDIA DGX Pod And DGX Pod Management Software Building your AI Data Center with DGX Reference Architectures Agenda Creating Network Topologies DGX POD Management

1.17k views • 82 slides

Building Scalable Big Data Pipelines NOSQL SEARCH ROADSHOW ZURICH Christian Ggi, Solution

Building Scalable Big Data Pipelines NOSQL SEARCH ROADSHOW ZURICH Christian Ggi, Solution Architect 19.09.2013 AGENDA Opportunities & Challenges Integrating Hadoop Lambda Architecture Lambda in Practice Recommendations

420 views • 28 slides

CS520 Data Integration, Warehousing, and Provenance 7. Big Data Systems and Integration IIT

CS520 Data Integration, Warehousing, and Provenance 7. Big Data Systems and Integration IIT DBGroup Boris Glavic http://www.cs.iit.edu/~glavic/ http://www.cs.iit.edu/~cs520/ http://www.cs.iit.edu/~dbgroup/ Outline 0) Course Info 1)

781 views • 54 slides

Algorithms for Big Data (III) Chihao Zhang Shanghai Jiao Tong University Sept. 29, 2019

Algorithms for Big Data (III) Chihao Zhang Shanghai Jiao Tong University Sept. 29, 2019 Algorithms for Big Data (III) 1/16 We introduced the notion of universal families of Hash functions. Review of the Last Lecture Last time, we proved a

666 views • 62 slides

Partnerships across industry, academia, nonprofits, and government Meredith M. Lee Executive

Partnerships across industry, academia, nonprofits, and government Meredith M. Lee Executive Director | West Big Data Innovation Hub NIWR Annual Meeting | Feb 27, 2017 2012: US Federal Big Data R&D Initiative $200M 2014: 2015: 2016:

1.1k views • 14 slides

Data-Intensive Distributed Computing CS 431/631 451/651 (Fall 2019) Part 2: From MapReduce to

Data-Intensive Distributed Computing CS 431/631 451/651 (Fall 2019) Part 2: From MapReduce to Spark (1/2) September 19, 2019 Ali Abedi These slides are available at http://roegiest.com/bigdata-2019w/ This work is licensed under a Creative

437 views • 27 slides

Compact Multi-Signatures for Smaller Blockchains Dan Boneh 1 , Manu Drijvers 2 , Gregory Neven 2 1

Compact Multi-Signatures for Smaller Blockchains Dan Boneh 1 , Manu Drijvers 2 , Gregory Neven 2 1 Stanford University 2 DFINITY Bitcoin Blockchain and transactions Input 1 Output 1 Witness Input 2 Output 2 Witness Pointer to previous

372 views • 16 slides

Bitcoin Anonymity Mike Fleder Mike Kester Sudeep Pillai "Voodah"

Bitcoin Anonymity Mike Fleder Mike Kester Sudeep Pillai "Voodah" "darkskypoet" 1G6EQwiAfTVyTpK4j3XZ65CvonjDGrPsQ 1QEZohXPbh4ywbzPJATjMBDnSjJsZrZtQ1 Goal & Results Attacker Goal Tie real name to

331 views • 13 slides

The Changing Nature of Cryptocurrencies: Bitcoin and Its Copies During Their Cloning Thomas H. A.

Introduction Tested hypothesis Econometric approach Results Robustness Conclusion The Changing Nature of Cryptocurrencies: Bitcoin and Its Copies During Their Cloning Thomas H. A. Joubert Panth eon Assas University Departement of

394 views • 21 slides

Bitcoins Chester Rebeiro Assistant Professor Department of Computer Science and Engineering IIT

Bitcoins Chester Rebeiro Assistant Professor Department of Computer Science and Engineering IIT Madras Traditional Currencies Alice gives bill to Bob, Bob gives coffee to Alice CR 2 Characteristics of Paper Money No double spending

961 views • 54 slides