Big Data Processing Techniques Chentao Wu Associate Professor - PowerPoint PPT Presentation

Big Data Processing Techniques Chentao Wu Associate Professor Dept. of Computer Science and Engineering wuct@cs.sjtu.edu.cn

Schedule • lec1: Introduction on big data and cloud computing • Iec2: Introduction on data storage • lec3: Data reliability (Replication/Archive/EC) • lec4: Data consistency problem • lec5: Block level storage and file storage • lec6: Object-based storage • lec7: Distributed file system • lec8: Metadata management

Final Grade • Attendance 20% • Projects 80% • Projects will be given in the following classes. • Place: Room 317, SEIEE-4th Building • Time: 8:00-11:40 • Date: Friday of 1st, 2nd, 3rd, 5th week

Collaborators

Contents Introduction to Big Data 1

Big Data Definition • No single standard definition… “ Big Data ” is data whose scale, diversity, and complexity require new architecture, techniques, algorithms, and analytics to manage it and extract value and hidden knowledge from it…

Types of Data • Structured • Semi-Structured/Quasi-Structured/Unstructured • Data that has no inherent structure and is usually stored as Unstructured different types of files. • E.g. Text documents, PDFs, images, and videos • Textual data with erratic formats that can be formatted with Quasi-Structured effort and software tools Increasing Growth • E.g. Clickstream data • Textual data files with an apparent pattern, enabling Semi-Structured analysis • E.g. Spreadsheets and XML files • Data having a defined data model, format, structure • E.g. Database Structured

Characteristics of big data (1-Scale: Volume) • Data Volume • 44x increase from 2009 2020 • From 0.8 ZettaBytes to 44ZB • Data volume is increasing exponentially Exponential increase in collected/generated data

Characteristics of big data (2-Complexity: Varity) • Various formats, types, and structures • Text, numerical, images, audio, video, sequences, time series, social media data, multi-dim arrays, etc… • Static data vs. streaming data • A single application can be generating/collecting many types of data To extract knowledge  all these types of data need to linked together

Characteristics of big data (3-Speed: Velocity) • Data is begin generated fast and need to be processed fast • Online Data Analytics • Late decisions  missing opportunities • Examples • E-Promotions: Based on your current location, your purchase history, what you like  send promotions right now for store next to you • Healthcare monitoring: sensors monitoring your activities and body  any abnormal measurements require immediate reaction

Big Data (3Vs)

Big Data (4Vs)

Big Data (5Vs/6Vs) Volume Velocity Variety Variability Veracity Value • Massive volumes • Rapidly changing • Diverse data • Constantly • Varying quality • Cost- of data data from numerous changing and reliability of effectiveness sources meaning of data data and business value • Challenges in • Challenges in storage and real-time • Challenges in • Challenges in • Challenges in analysis analysis integration, and gathering and transforming analysis interpretation and trusting data

Harnessing Big Data • OLTP: Online Transaction Processing (DBMSs) • OLAP: Online Analytical Processing (Data Warehousing) • RTAP: Real-Time Analytics Processing (Big Data Architecture & technology)

Who’s Generating Big Data Mobile devices (tracking all objects all the time) Social media and networks Scientific instruments (all of us are generating data) (collecting all sorts of data) Sensor technology and networks (measuring all kinds of data) • The progress and innovation is no longer hindered by the ability to collect data • But, by the ability to manage, analyze, summarize, visualize, and discover knowledge from the collected data in a timely manner and in a scalable fashion

The Model Has Changed… • The Model of Generating/Consuming Data has Changed Old Model: Few companies are generating data, all others are consuming data New Model: all of us are generating data, and all of us are consuming data

What’s driving Big Data - Optimizations and predictive analytics - Complex statistical analysis - All types of data, and many sources - Very large datasets - More of a real-time - Ad-hoc querying and reporting - Data mining techniques - Structured data, typical sources - Small to mid-size datasets

Value of Big Data Analytics • Big data is more real-time in nature than traditional DW applications • Traditional DW architectures (e.g. Exadata, Teradata) are not well-suited for big data apps • Shared nothing, massively parallel processing, scale out architectures are well-suited for big data apps

Challenges in Handling Big Data • The Bottleneck is in technology • New architecture, algorithms, techniques are needed • Also in technical skills • Experts in using the new technology and dealing with big data

Big Data Landscape

Big Data Technology

Contents 2 Introduction to Cloud Computing

What is Cloud Computing? Cloud Computing A model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources, (e.g., servers, storage, networks, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction. – U.S. National Institute of Standards and Technology, Special Publication 800-145 • A cloud is a collection of network-accessible hardware and software resources • Consists of IT resource pools deployed in data centers • Cloud model enables consumers to hire IT resources as services

What is Cloud Computing? (Cont'd) Cloud Infrastructure Desktop LAN/WAN Laptop Compute Network Storage Applications Platform Software Tablet and Mobile

Essential Cloud Characteristics 1 On-demand self- service 3 2 Broad Network Resource Pooling Access Cloud Infrastructure 4 5 Rapid Elasticity Measured Service

Cloud Service Models 1 Infrastructure as a Service (IaaS) Cloud Infrastructure 2 3 Platform as a Service Software as a Service (PaaS) (SaaS)

Infrastructure as a Service Infrastructure as a Service The capability provided to the consumer is to provision processing, storage, networks, and other fundamental computing resources where the consumer is able to deploy and run arbitrary Consumer’s Resources software, which can include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure but has Cloud control over operating systems, storage, and Infrastructure deployed applications; and possibly limited control of Provider’s Resources select networking components, (e.g., host firewalls). – U.S. National Institute of Standards and Technology, Special Publication 800-145

Platform as a Service Platform as a Service The capability provided to the consumer is to deploy onto the cloud infrastructure consumer-created or Consumer’s Resources acquired applications created using programming languages, libraries, services, and tools supported by the provider. The consumer does not manage or control the underlying cloud infrastructure including Provider’s network, servers, operating systems, or storage, but Resources Cloud has control over the deployed applications and Infrastructure possibly configuration settings for the application- hosting environment. – U.S. National Institute of Standards and Technology, Special Publication 800-145

Software as a Service Software as a Service The capability provided to the consumer is to use the provider’s applications running on a cloud infrastructure. The applications are accessible from various client devices through either a thin client interface, such as a web browser, (e.g., web-based email, or a program interface. The consumer does not Provider’s Resources manage or control the underlying cloud Cloud infrastructure including network, servers, operating Infrastructure systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings. – U.S. National Institute of Standards and Technology, Special Publication 800-145

Cloud Deployment Models 1 2 Public Cloud Private Cloud Cloud Infrastructure 3 4 Community Cloud Hybrid Cloud

Public Cloud Enterprise P Enterprise Q Cloud Provider’s Resources Individual R

Private Cloud Enterprise P Cloud Provider’s Resources Enterprise P Resources of Dedicated for Enterprise P Enterprise P 2) Externally-hosted Private Cloud 1) On-premise Private Cloud

Community Cloud • On-premise Community Cloud Enterprise Q Enterprise P Resources of Resources of Enterprise Q Enterprise P Enterprise R

Community Cloud • Externally-hosted Community Cloud Enterprise P Enterprise Q Enterprise R Community Users Cloud Provider’s Resources Dedicated for Community

Hybrid Cloud Enterprise Q Cloud Provider’s Resources Enterprise P Resources of Enterprise P Individual R

Contents 3 Industrial Solutions