SLIDE 1
QCon London 2012
QCon London 2012
Hadoop: Scalable Infrastructure for Big Data QCon London 2012 - - PowerPoint PPT Presentation
Hadoop: Scalable Infrastructure for Big Data QCon London 2012 - - PowerPoint PPT Presentation
Hadoop: Scalable Infrastructure for Big Data QCon London 2012 Parand Tony Darugar Founder and CEO, Xpenser parand@xpenser.com QCon London 2012 What is Hadoop? QCon London 2012 Hadoop is the Linux of Big Data Processing QCon London 2012
SLIDE 2
SLIDE 3
QCon London 2012
Hadoop is the Linux
- f Big Data Processing
SLIDE 4
QCon London 2012
Infrastructure
for
Large Scale Computation & Data Processing
- n a network of
Commodity Hardware.
SLIDE 5
QCon London 2012
Why Hadoop?
SLIDE 6
QCon London 2012
Scale
SLIDE 7
QCon London 2012
Cost
SLIDE 8
QCon London 2012
Freedom
SLIDE 9
QCon London 2012
Does Anyone Use Hadoop?
SLIDE 10
QCon London 2012
IBM VISA Microsoft Facebook Yahoo AOL ... eHarmony Zion's bank NY Times Twitter eBay LinkedIn ...
SLIDE 11
QCon London 2012
Alternatives
Build your own Get creative with RDBMS architecture
SLIDE 12
QCon London 2012
What's the idea?
SLIDE 13
QCon London 2012
Commodity Hardware Distributed Operation
SLIDE 14
QCon London 2012
Wisdom: Embrace Failure (hardware) Be Resilient (software)
SLIDE 15
QCon London 2012
What's in the box?
SLIDE 16
QCon London 2012
Hadoop Distributed File System
SLIDE 17
QCon London 2012
Distributed Computation Framework
SLIDE 18
QCon London 2012
Map-Reduce Programming Model
SLIDE 19
QCon London 2012
HDFS
- Your data in triplicate
- Built-in resiliency to
large scale failures
- Intelligent Data Distribution
- Very large data sizes
SLIDE 20
QCon London 2012
Distributed Computation
- Built-in resiliency to
large scale failures
- Distribute work to workers,
collect results from fastest
- Move computation to data
(not data to computation)
SLIDE 21
QCon London 2012
Map Reduce
Very simple programming model: Map(anything)->key, value Sort, partition on key Reduce(key,value)->key, value No parallel processing or message passing semantics Programmable in Java or any other language (streaming)
SLIDE 22
QCon London 2012
Ecosystem
HBase: NoSQL BigTable clone Hive: Somewhat-SQL data store Pig: SQL-like programming model Chukwa, Scribe, Mahoot, Cassandra, Oozie, Sqoop, ...
SLIDE 23
QCon London 2012
Commercial Support
Cloudera HortonWorks IBM ...
SLIDE 24
QCon London 2012
How?
Try it in non-distributed mode Try it on a few spare machines Try it on EC2 Try it! http://hadoop.apache.org/
SLIDE 25
QCon London 2012
Case Studies
SLIDE 26
QCon London 2012
eHarmony
SLIDE 27
QCon London 2012
Biz360 (Attensity)
SLIDE 28
QCon London 2012
Yahoo!
SLIDE 29
QCon London 2012
You!
SLIDE 30
QCon London 2012
Start with ETL
SLIDE 31
QCon London 2012
Start with batch, non time-critical tasks
SLIDE 32
QCon London 2012
Start with storing your large data
- n HDFS
SLIDE 33
QCon London 2012
Move batch processing to Hadoop Serve from RDBMS
SLIDE 34
QCon London 2012
- Embrace. Be One
With The Hadoop.
SLIDE 35
QCon London 2012