applied research group
play

Applied research group Systems+database people building prototypes, - PowerPoint PPT Presentation

Aug 03, 2023 •29 likes •879 views

Applied research group Systems+database people building prototypes, publishing papers Applied research group Systems+database people building prototypes, publishing papers Collaborating with Big Data product group at MS Shipping our code to

  3. Applied research group Systems+database people building prototypes, publishing papers

  4. Applied research group Systems+database people building prototypes, publishing papers Collaborating with Big Data product group at MS Shipping our code to production

  5. Applied research group Systems+database people building prototypes, publishing papers Collaborating with Big Data product group at MS Shipping our code to production Open-sourcing our code Apache Hadoop, REEF, Heron

  6. Resource Distributed Query management tiered storage optimization Stream Log analytics processing

  7. Resource Distributed Query management tiered storage optimization Stream Log analytics processing

  8. Node Node Node Manager Manager Manager

  9. • Node Node Node Manager Manager Manager

  10. • • Node Node Node Manager Manager Manager

  11. • • Node Node Node Manager Manager Manager

  12. • • 1. Request Node Node Node Manager Manager Manager

  13. • • 1. Request 2. Allocation Node Node Node Manager Manager Manager

  14. • • 1. Request 2. Allocation 3. Start task Node Node Node Manager Manager Manager

  15. • • 1. Request 2. Allocation • 3. Start task Node Node Node Manager Manager Manager

  16. • • 1. Request 2. Allocation • • 3. Start task Node Node Node Manager Manager Manager

  17. • • 1. Request Do we really need a Resource Manager? 2. Allocation • • 3. Start task Node Node Node Manager Manager Manager

  18. Hadoop 1 World Hadoop 2 World • monolithic Users Application Frameworks Hive / Pig Hive / Pig Ad-hoc Ad-hoc app Ad-hoc Apps Ad-hoc Ad-hoc Ad-hoc Scope app Programming app app MR v1 app Model(s) on MR ... YARN Tez Giraph Storm Spark Dryad v2 Heron REEF Hadoop 1.x (MapReduce) Cluster OS (Resource YARN Management) File System HDFS 2 HDFS 1 Hardware

  19. Hadoop 1 World Hadoop 2 World • monolithic Users Application Frameworks • Reuse of RM Hive / Pig Hive / Pig Ad-hoc Ad-hoc app Ad-hoc Apps Ad-hoc component Ad-hoc Ad-hoc Scope app Programming app app MR v1 app Model(s) on MR ... YARN Tez Giraph Storm Spark Dryad v2 Heron REEF Hadoop 1.x (MapReduce) Cluster OS (Resource YARN Management) File System HDFS 2 HDFS 1 Hardware

  20. Hadoop 1 World Hadoop 2 World • monolithic Users Application Frameworks • Reuse of RM Hive / Pig Hive / Pig Ad-hoc Ad-hoc app Ad-hoc Apps Ad-hoc component Ad-hoc Ad-hoc Scope app Programming app app MR v1 app Model(s) on MR ... YARN Tez Giraph Storm Spark Dryad v2 Heron REEF Hadoop 1.x (MapReduce) YARN Cluster OS • (Resource YARN Management) layering abstractions File System HDFS 2 HDFS 1 Hardware

  22. But is all this good enough for the Microsoft clusters?

  23. High resource Scalability utilization Production jobs Workload and heterogeneity predictability

  24. 100% Utilization

  26. 0

  27. • Wide variety

  28. • Wide variety

  29. • Wide variety • •

  30. deadlines recurring >60% • Predictability over-provisioned

  31. 4 Hadoop committers in CISL 404 patches as of last night • Rayon/Morpheus: • Mercury/Yaq: • YARN Federation: • Medea:

  32. 4 Hadoop committers in CISL 404 patches as of last night • Rayon/Morpheus: • Mercury/Yaq: • YARN Federation: • Medea:

  33. [Hadoop 3.0; ATC 2015, EuroSys 2016]

  34. RM N1 N2

  35. j1 RM N1 N2

  36. j1 RM N1 N2

  37. j2 RM N1 N2

  38. j2 RM N1 N2

  39. j2 RM N1 N2

  40. j2 RM N1 N2

  41. j2 RM N1 N2

  42. j2 RM • Feedback delays idle between allocations N1 N2

  43. j2 RM • Feedback delays idle between allocations N1 N2 5 sec 10 sec 50 sec Mixed-5-50 Cosmos-gm 60.59% 78.35% 92.38% 78.54% 83.38%

  44. j2 RM • Feedback delays idle between allocations N1 N2 5 sec 10 sec 50 sec Mixed-5-50 Cosmos-gm 60.59% 78.35% 92.38% 78.54% 83.38% • Actual

  45. • Introduce task queuing at nodes • Mask feedback delays • Improve cluster utilization • Improve task throughput (by up to 40%) • Container types • GUARANTEED and OPPORTUNISTIC • Keep guarantees for important jobs • Use opportunistic execution to improve utilization

  46. RM N1 N2

  47. RM N1 N2

  48. j1 RM N1 N2

  49. j1 RM N1 N2

  50. j2 RM N1 N2

  51. j2 RM N1 N2

  52. j2 RM N1 N2

  53. j2 RM N1 N2

  54. • j2 RM N1 N2

  55. • j2 RM • N1 N2

  56. • j2 RM • N1 N2

  58. •

  59. • •

  60. • So all we need to do is use long queues? •

  62. can be detrimental for job completion times • Despite the utilization gains

  63. can be detrimental for job completion times • Despite the utilization gains Proper queue management techniques are required

  64. N1 N2 N3

  65. N1 N2 N3

  66. N1 N2 N3

  67. N1 N2 N3

  68. Prioritize task Place tasks to execution node queues (queue reordering) Bound queue lengths

  69. Prioritize task Place tasks to execution node queues (queue reordering) Bound queue lengths Yaq improves median job completion time by 1.7x over YARN

  70. RM N1 N2 N3

  71. queue length RM N1 N2 N3

  72. queue length RM N1 N2 N3

  73. queue length RM N1 N2 N3

  74. queue length RM queue wait time N1 N2 N3

  75. queue length RM queue wait time N1 N2 N3

  76. • Shortest Remaining Job First (SRJF) • Least Remaining Tasks First (LRTF)

  77. RM j2: 5 tasks j3: 9 tasks j1: 21 tasks • Shortest Remaining Job First (SRJF) • Least Remaining Tasks First (LRTF) N1 N2 N3

  78. RM j2: 5 tasks j3: 9 tasks j1: 21 tasks • Shortest Remaining Job First (SRJF) • Least Remaining Tasks First (LRTF) N1 N2 N3

  79. RM j2: 5 tasks j3: 9 tasks j1: 21 tasks • Shortest Remaining Job First (SRJF) • Least Remaining Tasks First (LRTF) N1 N2 N3

  80. RM j2: 5 tasks j3: 9 tasks j1: 21 tasks • Shortest Remaining Job First (SRJF) • Least Remaining Tasks First (LRTF) N1 N2 N3 job-aware

  81. lower throughput longer job completion times

  82. • 1.7x improvement in median JCT over YARN

  83. • Container types distributed scheduling any distributed scheduler over-commitment multi-tenancy • Pricing

  84. cluster utilization queue management techniques job completion time

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

The Applied Research Institute Overview The University of Virginia Applied Research Institute

The Applied Research Institute Overview The University of Virginia Applied Research Institute

The Applied Research Institute Overview The University of Virginia Applied Research Institute (ARI) was created to enlarge the scope of projects and opportunities available to faculty and students. ARI leverages the Universitys human and

405 views • 11 slides
work? Dr Richard Fitton (ABERG Salford) Applied Buildings and Energy Research Group Why are we

work? Dr Richard Fitton (ABERG Salford) Applied Buildings and Energy Research Group Why are we

Energy Savings Measures, do they work? Dr Richard Fitton (ABERG Salford) Applied Buildings and Energy Research Group Why are we researching retrofit (UK)? Solid wall properties such as those represented by the Energy house currently number

710 views • 53 slides
Agenda Introduction to ARP ARP functionality Proxy ARP RARP Applied

Agenda Introduction to ARP ARP functionality Proxy ARP RARP Applied

1/13 ARP: Address Resolution Protocol Surasak Sanguanpong nguan@ku.ac.th http://www.cpe.ku.ac.th/~nguan Last updated: July 3, 2002 Applied Network Research Group Department of Computer Engineering, Kasetsart

510 views • 7 slides
ASURE an applied-research organization affiliated with the nation s fastest growing research

ASURE an applied-research organization affiliated with the nation s fastest growing research

The Aerospace & Defense Forum Arizona Chapter May 12, 2015 ASU RESEARCH ENTERPRISE Introductory Briefing Security | Innovation | Service We conceive | We develop | We implement | We deliver ASURE ASURE an applied-research

488 views • 16 slides
Big Data for Health Policy in Early Childhood: The TARGet Kids! Network The Applied Research

Big Data for Health Policy in Early Childhood: The TARGet Kids! Network The Applied Research

Big Data for Health Policy in Early Childhood: The TARGet Kids! Network The Applied Research Group for Kids! Cory Borkhoff PhD Research Associate, Department of Pediatric Medicine, The Hospital for Sick Children Assistant Professor, iHPME,

622 views • 9 slides
Archiving Computational Research in Virtual Machines Sorin Mitran Applied Mathematics

Archiving Computational Research in Virtual Machines Sorin Mitran Applied Mathematics

UNCvm Prologue Teaching Research Epilogue Archiving Computational Research in Virtual Machines Sorin Mitran Applied Mathematics University of North Carolina Chapel Hill Applied Mathematics Perspectives Reproducible Research Workshop

812 views • 55 slides
The Evolution of Research on Doctoral Student Success at UM Higher Education Applied Research

The Evolution of Research on Doctoral Student Success at UM Higher Education Applied Research

The Evolution of Research on Doctoral Student Success at UM Higher Education Applied Research (HEAR) seminar November 28, 2017 John A. Gonzalez, Ph.D. Director of Institutional Research Merle Feldbaum Research Area Specialist Senior Allyson

329 views • 16 slides
Riskcenter is a research group that specializes in the field of risk management , insurance and

Riskcenter is a research group that specializes in the field of risk management , insurance and

Riskcenter is a research group that specializes in the field of risk management , insurance and finance based at the University of Barcelona . The research group on Risk in Insurance and Finance is attached to the Institute of Applied Economics

358 views • 10 slides
applied in Egypt? P. Michael Link 1,2,3 Jrgen Scheffran 1,2 1 University of Hamburg, Research

applied in Egypt? P. Michael Link 1,2,3 Jrgen Scheffran 1,2 1 University of Hamburg, Research

Adaptation to sea level rise in the European Wadden Sea: can promising approaches be applied in Egypt? P. Michael Link 1,2,3 Jrgen Scheffran 1,2 1 University of Hamburg, Research Group Climate Change and Security 2 University of Hamburg,

462 views • 29 slides
Development of a Common Research Model for Applied CFD Validation Studies John Vassberg &

Development of a Common Research Model for Applied CFD Validation Studies John Vassberg &

BOEING PROPRIETARY Development of a Common Research Model for Applied CFD Validation Studies John Vassberg & Mark DeHaan The Boeing Company Phantom Works Melissa Rivers & Rich Wahls NASA Langley Research Center AIAA Applied

690 views • 20 slides
Applied Geology APPLIED GEOLOGY SUBJECT AREA GROUP Tuning Africa Phase II Third General Meeting

Applied Geology APPLIED GEOLOGY SUBJECT AREA GROUP Tuning Africa Phase II Third General Meeting

Applied Geology APPLIED GEOLOGY SUBJECT AREA GROUP Tuning Africa Phase II Third General Meeting Accra, Ghana, 17th to 19th October, 2016 By: Dr. Digne Edmond Rwabuhungu Rwatangabo Group Coordinator Coordinator: Digne Edmond RWABUHUNGU R.

198 views • 16 slides
Applied Physics Day 2017 Welcome! Department of Applied Physics Present AlbaNova Electrum

Applied Physics Day 2017 Welcome! Department of Applied Physics Present AlbaNova Electrum

Applied Physics Day 2017 Welcome! Department of Applied Physics Present AlbaNova Electrum SciLifeLab Applied Physics: 185 people 88 % Research 12 % Basic edu 220 MSEK turnover KTH Kista KTH Main campus KTH Flemingsberg KTH

344 views • 12 slides
Access to EU Research Funding for Universities of Applied Sciences 19-20 November, Spanish

Access to EU Research Funding for Universities of Applied Sciences 19-20 November, Spanish

Welcome to the international AESIS course on Access to EU Research Funding for Universities of Applied Sciences 19-20 November, Spanish National Research Council, Brussels, Belgium Access to EU Research Funding for Universities of Applied

1.04k views • 88 slides
Center for Applied Energy Research at University of Kentucky www.caer.uky.edu Catalysis Research

Center for Applied Energy Research at University of Kentucky www.caer.uky.edu Catalysis Research

Center for Applied Energy Research at University of Kentucky www.caer.uky.edu Catalysis Research and Testing Center Research and Testing Center 2540 Research Park Dr. Lexington, KY 45011 Anderson-Schulz-Flory Product Distribution Can it

576 views • 39 slides
Cornell University Program on Applied Demographics Applied Demographics CaRDI Spring Research

Cornell University Program on Applied Demographics Applied Demographics CaRDI Spring Research

Cornell University Program on Applied Demographics Applied Demographics CaRDI Spring Research Roundtable April 6, 2009 April 6 2009 Program on Applied Demographics Web: http://pad.human.cornell.edu Email: PADInfo@Cornell.edu Mission

336 views • 18 slides
The$Canadian$Centre$for$Applied$ Research$in$Cancer$Control$(ARCC)$

The$Canadian$Centre$for$Applied$ Research$in$Cancer$Control$(ARCC)$

The$Canadian$Centre$for$Applied$ Research$in$Cancer$Control$(ARCC)$ Dr.$Stuart$Peacock,$$Dr.$Jeffrey$Hoch,$Kimberly$van$der$Hoek$and$Sarah$Benn ! ! ARCC!Research! 5$themaGc$research$program$areas$to$address$cancer$control$ I

194 views • 3 slides
Optimization Models EECS 127 / EECS 227AT Laurent El Ghaoui EECS department UC Berkeley Spring

Optimization Models EECS 127 / EECS 227AT Laurent El Ghaoui EECS department UC Berkeley Spring

Optimization Models EECS 127 / EECS 227AT Laurent El Ghaoui EECS department UC Berkeley Spring 2020 Sp20 1 / 30 LECTURE 26 Implicit Deep Learning The Matrix is everywhere. It is all around us. Morpheus Sp20 2 / 30 Outline 1 Implicit

762 views • 32 slides
1 Wednesday, October 6, 2010 Image credit: http://browsertoolkit.com/fault-tolerance.png 2

1 Wednesday, October 6, 2010 Image credit: http://browsertoolkit.com/fault-tolerance.png 2

1 Wednesday, October 6, 2010 Image credit: http://browsertoolkit.com/fault-tolerance.png 2 Wednesday, October 6, 2010 Image credit: http://browsertoolkit.com/fault-tolerance.png 3 Wednesday, October 6, 2010 Image credit:

959 views • 71 slides
Non-Transitive Connectivity and DHTs Mike Freedman Karthik Lakshminarayanan Sean Rhea Ion

Non-Transitive Connectivity and DHTs Mike Freedman Karthik Lakshminarayanan Sean Rhea Ion

X Non-Transitive Connectivity and DHTs Mike Freedman Karthik Lakshminarayanan Sean Rhea Ion Stoica WORLDS 2005 X Distributed Hash Tables k R System assigns keys to nodes All nodes agree on assignment Chord assigns keys as

516 views • 25 slides
z Towards Plan-aware Resource Allocation in Serverless Query Processing Malay Bag Alekh Jindal

z Towards Plan-aware Resource Allocation in Serverless Query Processing Malay Bag Alekh Jindal

z Towards Plan-aware Resource Allocation in Serverless Query Processing Malay Bag Alekh Jindal z Hiren Patel z Resour ource Alloc ocati tion Issue ue in Serverless Query Processing Hard to estimate resource requirement at compile

673 views • 11 slides
programmers? Frances Buontempo @fbuontempo 1 Yes But @fbuontempo 2 What is AI?

programmers? Frances Buontempo @fbuontempo 1 Yes But @fbuontempo 2 What is AI?

Can AI replace programmers? Frances Buontempo @fbuontempo 1 Yes But @fbuontempo 2 What is AI? @fbuontempo 3 What is AI? Coined by John McCarthy in 1955 Reproduce human intelligence electronically What is real?

1.07k views • 63 slides
Programming language shapes Programming thought programming languages are not merely

Programming language shapes Programming thought programming languages are not merely

CMPS 112: Spring 2019 Comparative Programming Languages Lecture 1: Course Overview. Owen Arden UC Santa Cruz A Programming Language Two variables L1: x++; x , y y--; Three operations (y=0)?L2:L1 x++ L2:

476 views • 18 slides
CE419 Session 1: Fundamentals Web Programming Let's begin with the premise that everything

CE419 Session 1: Fundamentals Web Programming Let's begin with the premise that everything

CE419 Session 1: Fundamentals Web Programming Let's begin with the premise that everything you've known up to this point is wrong. 4 Story of the Web The World Wide Web has gone from never heard of it to cant live without

906 views • 53 slides
1Q18 Earnings Call Presentation April 25, 2018 Sands Macao Sands Bethlehem Four Seasons Macao The

1Q18 Earnings Call Presentation April 25, 2018 Sands Macao Sands Bethlehem Four Seasons Macao The

The Venetian Macao Marina Bay Sands, Singapore Sands Cotai Central, Macao The Parisian Macao 1Q18 Earnings Call Presentation April 25, 2018 Sands Macao Sands Bethlehem Four Seasons Macao The Venetian Las Vegas The Palazzo, Las Vegas Forward Looking

1.01k views • 63 slides

More recommend