Big Data Processing Techniques Chentao Wu Associate Professor - - PowerPoint PPT Presentation

Big Data Processing Techniques

Chentao Wu Associate Professor

• Dept. of Computer Science and Engineering

wuct@cs.sjtu.edu.cn

Schedule

• lec1: Introduction on big data and cloud

computing

• Iec2: Introduction on data storage
• lec3: Data reliability (Replication/Archive/EC)
• lec4: Data consistency problem
• lec5: Block level storage and file storage
• lec6: Object-based storage
• lec7: Distributed file system
• lec8: Metadata management

Final Grade

• Attendance 20%
• Projects 80%
• Projects will be given in the following classes.
• Place: Room 317, SEIEE-4th Building
• Time: 8:00-11:40
• Date: Friday of 1st, 2nd, 3rd, 5th week

Collaborators

Contents

Introduction to Big Data

1

Big Data Definition

• No single standard definition…

“Big Data” is data whose scale, diversity, and complexity require new architecture, techniques, algorithms, and analytics to manage it and extract value and hidden knowledge from it…

Types of Data

• Structured
• Semi-Structured/Quasi-Structured/Unstructured

Unstructured Quasi-Structured Semi-Structured Structured

• Data that has no inherent structure and is usually stored as

different types of files.

• E.g. Text documents, PDFs, images, and videos
• Textual data with erratic formats that can be formatted with

effort and software tools

• E.g. Clickstream data
• Textual data files with an apparent pattern, enabling

analysis

• E.g. Spreadsheets and XML files
• Data having a defined data model, format, structure
• E.g. Database

Increasing Growth

Characteristics of big data

(1-Scale: Volume)

• Data Volume
• 44x increase from 2009 2020
• From 0.8 ZettaBytes to 44ZB
• Data volume is increasing

exponentially

Exponential increase in collected/generated data

Characteristics of big data

(2-Complexity: Varity)

• Various formats, types, and

structures

• Text, numerical, images, audio,

video, sequences, time series, social media data, multi-dim arrays, etc…

• Static data vs. streaming data
• A single application can be

generating/collecting many types

• f data

To extract knowledge all these types of data need to linked together

Characteristics of big data

(3-Speed: Velocity)

• Data is begin generated fast and need to be processed

fast

• Online Data Analytics
• Late decisions  missing opportunities
• Examples
• E-Promotions: Based on your current location, your purchase history, what

you like  send promotions right now for store next to you

• Healthcare monitoring: sensors monitoring your activities and body 

any abnormal measurements require immediate reaction

Big Data (3Vs)

Big Data (4Vs)

Big Data (5Vs/6Vs)

Volume

• Massive volumes
• f data
• Challenges in

storage and analysis

Velocity

• Rapidly changing

data

• Challenges in

real-time analysis

Variety

• Diverse data

from numerous sources

• Challenges in

integration, and analysis

Variability

• Constantly

changing meaning of data

• Challenges in

gathering and interpretation

Veracity

• Varying quality

and reliability of data

• Challenges in

transforming and trusting data

Value

• Cost-

effectiveness and business value

Harnessing Big Data

• OLTP: Online Transaction Processing (DBMSs)
• OLAP: Online Analytical Processing (Data Warehousing)
• RTAP: Real-Time Analytics Processing (Big Data Architecture & technology)

Who’s Generating Big Data

Social media and networks (all of us are generating data) Scientific instruments (collecting all sorts of data) Mobile devices (tracking all objects all the time) Sensor technology and networks (measuring all kinds of data)

• The progress and innovation is no longer hindered by the ability to collect data
• But, by the ability to manage, analyze, summarize, visualize, and discover

knowledge from the collected data in a timely manner and in a scalable fashion

The Model Has Changed…

• The Model of Generating/Consuming Data has Changed

Old Model: Few companies are generating data, all others are consuming data New Model: all of us are generating data, and all of us are consuming data

What’s driving Big Data

• Ad-hoc querying and reporting
• Data mining techniques
• Structured data, typical sources
• Small to mid-size datasets
• Optimizations and predictive analytics
• Complex statistical analysis
• All types of data, and many sources
• Very large datasets
• More of a real-time

Value of Big Data Analytics

• Big data is more real-time in nature

than traditional DW applications

• Traditional DW architectures (e.g.

Exadata, Teradata) are not well-suited for big data apps

• Shared nothing, massively parallel

processing, scale out architectures are well-suited for big data apps

Challenges in Handling Big Data

• The Bottleneck is in technology
• New architecture, algorithms, techniques are needed
• Also in technical skills
• Experts in using the new technology and dealing with big

data

Big Data Landscape

Big Data Technology

Contents

Introduction to Cloud Computing

2

What is Cloud Computing?

• A cloud is a collection of network-accessible hardware

and software resources

• Consists of IT resource pools deployed in data centers
• Cloud model enables consumers to hire IT resources as

services

A model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources, (e.g., servers, storage, networks, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction.

– U.S. National Institute of Standards and Technology, Special Publication 800-145

Cloud Computing

What is Cloud Computing? (Cont'd)

Cloud Infrastructure

Applications Platform Software Network Compute Storage

LAN/WAN

Laptop Tablet and Mobile Desktop

Essential Cloud Characteristics

Resource Pooling

3

Measured Service

5

Rapid Elasticity

4

Broad Network Access

2

On-demand self- service

1

Cloud Infrastructure

Cloud Service Models

Software as a Service (SaaS)

3

Platform as a Service (PaaS)

2

Infrastructure as a Service (IaaS)

1

Cloud Infrastructure

Infrastructure as a Service

The capability provided to the consumer is to provision processing, storage, networks, and other fundamental computing resources where the consumer is able to deploy and run arbitrary software, which can include operating systems and

• applications. The consumer does not manage or

control the underlying cloud infrastructure but has control over operating systems, storage, and deployed applications; and possibly limited control of select networking components, (e.g., host firewalls).

– U.S. National Institute of Standards and Technology, Special Publication 800-145

Infrastructure as a Service

Cloud Infrastructure

Provider’s Resources Consumer’s Resources

Platform as a Service

Cloud Infrastructure

Provider’s Resources Consumer’s Resources

The capability provided to the consumer is to deploy

• nto the cloud infrastructure consumer-created or

acquired applications created using programming languages, libraries, services, and tools supported by the provider. The consumer does not manage or control the underlying cloud infrastructure including network, servers, operating systems, or storage, but has control over the deployed applications and possibly configuration settings for the application- hosting environment.

– U.S. National Institute of Standards and Technology, Special Publication 800-145

Platform as a Service

Software as a Service

Cloud Infrastructure

Provider’s Resources

The capability provided to the consumer is to use the provider’s applications running on a cloud

• infrastructure. The applications are accessible from

various client devices through either a thin client interface, such as a web browser, (e.g., web-based email, or a program interface. The consumer does not manage or control the underlying cloud infrastructure including network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.

– U.S. National Institute of Standards and Technology, Special Publication 800-145

Software as a Service

Cloud Deployment Models

Private Cloud

2

Hybrid Cloud

4

Community Cloud

3

Public Cloud

1

Cloud Infrastructure

Public Cloud

Enterprise P

Cloud Provider’s Resources

Enterprise Q Individual R

Private Cloud

Enterprise P

Resources of Enterprise P

1) On-premise Private Cloud

Cloud Provider’s Resources

Dedicated for Enterprise P

Enterprise P

2) Externally-hosted Private Cloud

Community Cloud

• On-premise Community Cloud

Resources of Enterprise P

Enterprise P

Resources of Enterprise Q

Enterprise Q Enterprise R

Community Cloud

• Externally-hosted Community Cloud

Cloud Provider’s Resources

Dedicated for Community Enterprise P Enterprise Q Enterprise R Community Users

Hybrid Cloud

Enterprise P

Resources of Enterprise P

Individual R

Cloud Provider’s Resources

Enterprise Q

Contents

Industrial Solutions

3

Hadoop

• Apache top level project, open-source implementation of

frameworks for reliable, scalable, distributed computing and data storage.

• It is a flexible and highly-available architecture for large scale

computation and data processing on a network of commodity hardware.

• Designed to answer the question: “How to process big

data with reasonable cost and time?”

Origin of Hadoop (1)

• Search Engine in 1990’s

Origin of Hadoop (2)

• Search Engine in 1998 and 2010’s

Origin of Hadoop (3)

2005: Doug Cutting and Michael J. Cafarella developed Hadoop to support distribution for the Nutch search engine project. The project was funded by Yahoo. 2006: Yahoo gave the project to Apache Software Foundation.

Origin of Hadoop (4)

2003 2004 2006

Hadoop Framework

Google

Compute

Storage

Amazon

 AWS is Amazon’s umbrella description of all of their web- based technology services.  Mainly infrastructure services:

• Amazon Elastic Compute Cloud (EC2)
• Amazon Simple Storage Service (S3)
• Amazon Simple Queue Service (SQS)
• Amazon CloudFront
• Amazon SimpleDB

Amazon

AWS Management Console

Microsoft Azure (1)

Microsoft Azure (2)

Microsoft Azure (3)

Aliyun Framework(1)

Aliyun Framework (2)

Thank you!