csc 369 distributed computing
play

CSC 369: Distributed Computing Alex Dekhtyar May 6 Day 14: Java - PowerPoint PPT Presentation

Feb 23, 2024 •289 likes •611 views

CSC 369: Distributed Computing Alex Dekhtyar May 6 Day 14: Java Hadoop API CSC 369: Distributed Computing Alex Dekhtyar May 6 Day 14: Java Hadoop API HAPPY EQUATOR DAY! Housekeeping Lab 4 (mini-project): due Sunday night Lab 5: due

  1. CSC 369: Distributed Computing Alex Dekhtyar May 6 Day 14: Java Hadoop API

  2. CSC 369: Distributed Computing Alex Dekhtyar May 6 Day 14: Java Hadoop API HAPPY EQUATOR DAY!

  6. Housekeeping Lab 4 (mini-project): due Sunday night Lab 5: due tonight (grace period tomorrow) Lab 6: full lab coming out Friday Grading: slowly happening…

  7. Hadoop Java API

  8. Hadoop API Current Version is 3.2.1. hadoop Command-line tools hdfs We limit ourselves to hadoop jar yarn

  9. Hadoop Java API org.apache.hadoop Let’s concentrate on things we absolutely need

  10. Hadoop Java API org.apache.hadoop Core MapReduce classes org.apache.hadoop.mapreduce Inuput/Output org.apache.hadoop.mapreduce.lib.input parsing org.apache.hadoop.mapreduce.lib.output atomic type wrappers org.apache.hadoop.io Job configuration org.apache.hadoop.conf File system classes org.apache.hadoop.fs

  11. org.apache.hadoop.mapreduce MapReduce Job org.apache.hadoop.mapreduce.Job org.apache.hadoop.mapreduce.Mapper Extensible Mapper org.apache.hadoop.mapreduce.Reducer Extensible Reducer Parent class for org.apache.hadoop.mapreduce.Partitioner Partitioning tasks org.apache.hadoop.mapreduce.InputFormat Parent classes for org.apache.hadoop.mapreduce.OutputFormat Input/Output Formats Parent class for org.apache.hadoop.mapreduce.InputSplit Input Split

  12. How it works Input File

  13. How it works InputSplit InputSplit InputSplit Input File

  14. How it works Job InputSplit Mapper Combiner (Reducer) InputSplit Reducer InputSplit Input File

  15. How it works Compute Node1 Job InputSplit Mapper Combiner (Reducer) Compute Node2 InputSplit Reducer InputSplit Compute Node3 Input File

  16. How it works Compute Node1 Job InputSplit Mapper Combiner (Reducer) Compute Node2 InputSplit Reducer InputSplit Compute Node3 Input File

  17. How it works Compute Node1 InputSplit Combiner (Reducer) Mapper Compute Node2 InputSplit Combiner (Reducer) Mapper InputSplit Compute Node3 Combiner (Reducer) Mapper Mapper Input File

  18. Compute Node1 InputSplit Mapper Combiner (Reducer)

  19. Compute Node1 InputSplit Mapper Combiner (Reducer)

  20. time MAP STAGE Reduce STAGE Compute Node1 Compute Node1 Mapper Reducer Combiner Compute Node2 Compute Node2 Mapper Reducer Combiner Compute Node3 Compute Node3 Reducer Mapper Combiner

  21. time MAP STAGE Reduce STAGE Compute Node1 Compute Node1 Partitioner Mapper Reducer Combiner Compute Node2 Compute Node2 Partitioner Mapper Reducer Combiner Compute Node3 Compute Node3 Partitioner Reducer Mapper Combiner

  22. time MAP STAGE Reduce STAGE Compute Node1 Compute Node1 Partitioner Mapper Reducer Combiner Compute Node2 Compute Node2 Partitioner Mapper Reducer Combiner Compute Node3 Compute Node3 Partitioner Reducer Mapper Shuffle STAGE Combiner

  23. time MAP STAGE Reduce STAGE Compute Node1 Compute Node1 Partitioner Mapper Reducer Combiner Compute Node2 Compute Node2 Partitioner Mapper Reducer Combiner Compute Node3 Compute Node3 Partitioner Reducer Mapper Shuffle STAGE Combiner

  24. Mapper in a nutshell protected void setup(org.apache.hadoop.mapreduce.Mapper.Context context) protected void map(KEYIN key, VALUEIN value, org.apache.hadoop.mapreduce.Mapper.Context context) protected void cleanup(org.apache.hadoop.mapreduce.Mapper.Context context) void run(org.apache.hadoop.mapreduce.Mapper.Cont ext context)

  25. run(InputSplit s, Context c): Run setup() once setup(s,c); for each record in s do: Run map() for each record map(record, c); end for; cleanup(s,c) Run cleaunp() once

  26. Reducer in a nutshell protected void setup(org.apache.hadoop.mapreduce.Mapper.Context context) protected void reduce(KEYIN key, Iterable<VALUEIN> value, org.apache.hadoop.mapreduce.Mapper.Context context) protected void cleanup(org.apache.hadoop.mapreduce.Mapper.Context context) void run(org.apache.hadoop.mapreduce.Mapper.Cont ext context)

  27. Shuffle Sort SecondarySort run(InputSplit s, Context c): Run setup() once setup(s,c); Run map() for each record for each record in s do: map(record, c); end for; Run cleaunp() once cleanup(s,c)

  28. Hadoop Java API org.apache.hadoop Core MapReduce classes org.apache.hadoop.mapreduce Inuput/Output org.apache.hadoop.mapreduce.lib.input parsing org.apache.hadoop.mapreduce.lib.output atomic type wrappers org.apache.hadoop.io Job configuration org.apache.hadoop.conf File system classes org.apache.hadoop.fs

  29. Hadoop Java API org.apache.hadoop Core MapReduce classes org.apache.hadoop.mapreduce Inuput/Output org.apache.hadoop.mapreduce.lib.input parsing org.apache.hadoop.mapreduce.lib.output atomic type wrappers org.apache.hadoop.io Job configuration org.apache.hadoop.conf File system classes org.apache.hadoop.fs

  30. org.apache.hadoop.mapreduce.lib.input Single File Input Format Generic Input File format (others extend it) FileInputFormat Text Input TextInputFormat User-defined Key-Value Pairs KeyValueInputFormat Fixed Length Records in input FixedLengthInputFormat NLineInputFormat Controls the size of split (in terms of #lines)

  31. org.apache.hadoop.mapreduce.lib.input Single File Input Format Generic Input File format (others extend it) FileInputFormat Text Input TextInputFormat User-defined Key-Value Pairs KeyValueInputFormat Fixed Length Records in input FixedLengthInputFormat Controls the size of split (in terms of #lines) NLineInputFormat Other Important Classes Multiple Files as inputs to a single Mapper MultipleInputs File Partitions FileSplits

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Advanced Architectures Goals of Distributed Computing better services 15A. Distributed

Advanced Architectures Goals of Distributed Computing better services 15A. Distributed

6/7/2018 Advanced Architectures Goals of Distributed Computing better services 15A. Distributed Computing scalability 15B. Multi-Processor Systems apps too big to run on a single computer 15C. Tightly Coupled Distributed Systems

347 views • 9 slides
Distributed Computing Environments 1 Distributed computing environment consists of entities

Distributed Computing Environments 1 Distributed computing environment consists of entities

Distributed Computing Environments 1 Distributed computing environment consists of entities entities communicate with each other the goal is to find a solution to a common problem 2 Entities entity: a computational unit of a

564 views • 13 slides
61A Lecture 35 Characteristics of distributed computing: Computers are independent they do

61A Lecture 35 Characteristics of distributed computing: Computers are independent they do

Distributed Computing A distributed computing application consists of multiple programs running on multiple computers that together coordinate to perform some task. Computation is performed in parallel by many computers. Information can be

478 views • 3 slides
Advanced Architectures Goals of Distributed Computing better services 15A. Distributed

Advanced Architectures Goals of Distributed Computing better services 15A. Distributed

5/27/2017 Advanced Architectures Goals of Distributed Computing better services 15A. Distributed Computing scalability 15B. Multi-Processor Systems apps too big to run on a single computer 15C. Tightly Coupled Systems grow

479 views • 8 slides
Algorithms in Nature Distributed computing Example Distributed systems Internet ATM

Algorithms in Nature Distributed computing Example Distributed systems Internet ATM

Algorithms in Nature Distributed computing Example Distributed systems Internet ATM (bank) machines Intranets/Workgroups Computing landscape will soon consist of ubiquitous network-connected devices The network is the

447 views • 16 slides
On safety in distributed computing Srivatsan Ravi On safety in distributed computing Safety in

On safety in distributed computing Srivatsan Ravi On safety in distributed computing Safety in

On safety in distributed computing Srivatsan Ravi On safety in distributed computing Safety in distributed computing 1 Something bad never happens 2 Some invariant holds at every step in the execution 3 If something bad happens in an

951 views • 42 slides
Collaborative Human Computing Zack Zhu March 31, 2010 Seminar for Distributed Computing 1

Collaborative Human Computing Zack Zhu March 31, 2010 Seminar for Distributed Computing 1

Collaborative Human Computing Zack Zhu March 31, 2010 Seminar for Distributed Computing 1 Distributed Computing... 2 ...redefined: Distributed Thinking 3 Crowdsourcing + Human Resource = $$$$$!! Internet + Web 2.0 $ $ 4

787 views • 36 slides
CSC 369: Distributed Computing Alex Dekhtyar Day 1: Welcome Syllabus Teaching and

CSC 369: Distributed Computing Alex Dekhtyar Day 1: Welcome Syllabus Teaching and

CSC 369: Distributed Computing Alex Dekhtyar Day 1: Welcome Syllabus Teaching and Communication Textbook(s) Grading Exams Labs Late Policies Course What is distributed computing Why study it?

997 views • 42 slides
Distributed Computing Programs that cooperate and communicate over a network E-mail

Distributed Computing Programs that cooperate and communicate over a network E-mail

CMSC 433 Programming Language Technologies and Paradigms Java RMI Distributed Computing Programs that cooperate and communicate over a network E-mail Web server and web client SETI @Home 2 1 Distributed Computing

565 views • 17 slides
Historical perspective From sequential computing to distributed computing and ...

Historical perspective From sequential computing to distributed computing and ...

Never forget .... Distributed Universal Constructions a guided tour Michel R AYNAL Institut Universitaire de France Academia Europaea IRISA, Universit e de Rennes, France Polytechnic University (PolyU), Hong Kong Distributed Universal

531 views • 27 slides
Coordinating distributed systems Marko Vukoli Distributed Systems and Cloud Computing Previous

Coordinating distributed systems Marko Vukoli Distributed Systems and Cloud Computing Previous

Coordinating distributed systems Marko Vukoli Distributed Systems and Cloud Computing Previous lectures Distributed Storage Systems CAP Theorem Amazon Dynamo Cassandra 2 Today Distributed systems coordination Apache

1.57k views • 62 slides
Self-Stabilization in Distributed Systems Course: Distributed Computing Faculty: Dr. Rajendra

Self-Stabilization in Distributed Systems Course: Distributed Computing Faculty: Dr. Rajendra

Self-Stabilization in Distributed Systems Course: Distributed Computing Faculty: Dr. Rajendra Prasath Spring 2019 About this topic This course covers various concepts in Self- Stabilization in Distributed Systems. We will also focus on the

330 views • 31 slides
DISTRIBUTED SYSTEMS Department of Computing Science Umea University Distributed Systems - D N

DISTRIBUTED SYSTEMS Department of Computing Science Umea University Distributed Systems - D N

DISTRIBUTED SYSTEMS Department of Computing Science Umea University Distributed Systems - D N Ranasinghe 1 Fundamental Concepts Distributed Systems - D N Ranasinghe 2 About Distributed Computing devising algorithms for a set of

1.41k views • 105 slides
Distributed Quantum Computing Using the most precise description of our world to solve some of

Distributed Quantum Computing Using the most precise description of our world to solve some of

Distributed Quantum Computing Using the most precise description of our world to solve some of the hardest problems by Matthias Egli Based on the paper Distributed Quantum Computing by Harry Buhrman and Hein Rhrig These slides are just

969 views • 31 slides
ATLAS Worldwide Distributed Computing ATLAS Worldwide Distributed Computing Zhongliang Ren 03

ATLAS Worldwide Distributed Computing ATLAS Worldwide Distributed Computing Zhongliang Ren 03

ATLAS Worldwide Distributed Computing ATLAS Worldwide Distributed Computing Zhongliang Ren 03 May 2006 ISGC 2006, Academia Sinica, Taipei, Taiwan Agenda of My Talk Agenda of My Talk Why the LHC and ATLAS ? The Standard Model The

427 views • 39 slides
Distributed Algorithms Tutorial Roger Wattenhofer ETH Zurich Distributed Computing

Distributed Algorithms Tutorial Roger Wattenhofer ETH Zurich Distributed Computing

Distributed Algorithms Tutorial Roger Wattenhofer ETH Zurich Distributed Computing www.disco.ethz.ch Distributed Algorithms Message Shared Passing Memory Example: Maximal Independent Set (MIS) Given a network with n nodes, nodes

1.15k views • 65 slides
New Ideas Track: Testing MapReduce-Style Programs Christoph Csallner, Leonidas Fegaras, Chengkai

New Ideas Track: Testing MapReduce-Style Programs Christoph Csallner, Leonidas Fegaras, Chengkai

New Ideas Track: Testing MapReduce-Style Programs Christoph Csallner, Leonidas Fegaras, Chengkai Li Computer Science and Engineering Department University of Texas at Arlington (UTA) European Software Engineering Conference / ACM SIGSOFT

727 views • 19 slides
Apache Spark Dr. Mihail Content derived from: Ankam, Venkat. Big Data Analytics. Packt

Apache Spark Dr. Mihail Content derived from: Ankam, Venkat. Big Data Analytics. Packt

Apache Spark Dr. Mihail Content derived from: Ankam, Venkat. Big Data Analytics. Packt Publishing, 2016. July 9, 2019 (Dr. Mihail ) Intro Big Data July 9, 2019 1 / 8 Apache Spark Why Hadoop and MapReduce have been around for 10 years and

443 views • 9 slides
Apache Solr Injection Michael Stepankin @artsploit DEF CON 27 @whoami Michael Stepankin

Apache Solr Injection Michael Stepankin @artsploit DEF CON 27 @whoami Michael Stepankin

Apache Solr Injection Michael Stepankin @artsploit DEF CON 27 @whoami Michael Stepankin Security Researcher @ Veracode Web app breaker Works on making Dynamic and Static Code Analysis smarter Penetration tester in the

518 views • 41 slides
Kernel HTTPS/TCP/IP stack for HTTP DDoS mitigation Alexander Krizhanovsky Tempesta Technologies,

Kernel HTTPS/TCP/IP stack for HTTP DDoS mitigation Alexander Krizhanovsky Tempesta Technologies,

Kernel HTTPS/TCP/IP stack for HTTP DDoS mitigation Alexander Krizhanovsky Tempesta Technologies, Inc. ak@tempesta-tech.com Who am I? CEO &amp; CTO at Tempesta Technologies (Seattle, WA) Developing Tempesta FW open source Linux Application

740 views • 48 slides
DONT OPTIMIZE MY QUERIES, ORGANIZE MY DATA! Julian Hyde (Apache Calcite) TELUQ, Montral,

DONT OPTIMIZE MY QUERIES, ORGANIZE MY DATA! Julian Hyde (Apache Calcite) TELUQ, Montral,

DONT OPTIMIZE MY QUERIES, ORGANIZE MY DATA! Julian Hyde (Apache Calcite) TELUQ, Montral, 2018/09/24 A simple query Data Query SELECT SUM (householdSize) 2010 U.S. census FROM CensusHouseholds; 100 million records

675 views • 46 slides
ROOT4J / SPARK-ROOT: ROOT I/O for JVM and Applications for Apache Spark V. Khristenko 1 J.

ROOT4J / SPARK-ROOT: ROOT I/O for JVM and Applications for Apache Spark V. Khristenko 1 J.

Introduction Functionality Examples Summary ROOT4J / SPARK-ROOT: ROOT I/O for JVM and Applications for Apache Spark V. Khristenko 1 J. Pivarski 2 1 Department of Physics The University of Iowa 2 Princeton University - DIANA ROOT I/O Workshop,

544 views • 19 slides
Apache Spark Corso di Sistemi e Architetture per Big Data A.A. 2017/18 Valeria Cardellini The

Apache Spark Corso di Sistemi e Architetture per Big Data A.A. 2017/18 Valeria Cardellini The

Universit degli Studi di Roma Tor Vergata Dipartimento di Ingegneria Civile e Ingegneria Informatica Apache Spark Corso di Sistemi e Architetture per Big Data A.A. 2017/18 Valeria Cardellini The reference Big Data stack High-level

692 views • 37 slides
An Introduction to Apache Spark Amir H. Payberah amir@sics.se SICS Swedish ICT Amir H. Payberah

An Introduction to Apache Spark Amir H. Payberah amir@sics.se SICS Swedish ICT Amir H. Payberah

An Introduction to Apache Spark Amir H. Payberah amir@sics.se SICS Swedish ICT Amir H. Payberah (SICS) Apache Spark Feb. 2, 2016 1 / 67 Big Data small data big data Amir H. Payberah (SICS) Apache Spark Feb. 2, 2016 2 / 67 Big Data

1.09k views • 86 slides

More recommend