how i learned to
play

How I Learned to Rohit Zambre,* Stop Aparna - PowerPoint PPT Presentation

Jan 09, 2024 •624 likes •1.16k views

How I Learned to Rohit Zambre,* Stop Aparna Chandramowlishwaran,* Worrying Pavan Balaji About *University of California, Irvine User-Visible Argonne National Laboratory Endpoints and Love MPI 2 HOW I LEARNED TO STOP WORRYING

  1. How I Learned to Rohit Zambre,* Stop Aparna Chandramowlishwaran,* Worrying Pavan Balaji ⌃ About *University of California, Irvine User-Visible ⌃ Argonne National Laboratory Endpoints and Love MPI

  2. 2 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI MPI everywhere Node Core Process

  3. 3 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI MPI everywhere ▸ Model artifact : high memory requirements that worsen with increase domain- dimensionality and number of ranks. ▸ Hardware usage : resource wastage with static split of limited resources on processor Node Core Process Increasing number of cores Decreasing memory per core

  4. 4 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI MPI+threads ▸ Model artifact : reduces duplicated data by a factor of number of threads. ▸ Hardware usage : able to use the many cores while sharing all of processor’s resources. Node Core Process Thread Increasing number of cores Decreasing memory per core

  5. 5 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI 7 Category Computation 6 Corresponding Allgatherv AlltoAllv MPI everywhere 5 Time (seconds) MPI+threads 4 OOM! 3 runs 2 1 0 1x440x110 1x220x220 1x110x440 6x180x45 6x90x90 6x45x180 Processor Grid (threads x processor rows x processor columns) Buluc et al. Distributed BFS (https://arxiv.org/abs/1705.04590)

  6. 6 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI 7 Category Computation 6 Corresponding Allgatherv AlltoAllv MPI everywhere 5 Time (seconds) MPI+threads 4 OOM! 3 runs 2 1 0 1x440x110 1x220x220 1x110x440 6x180x45 6x90x90 6x45x180 Processor Grid (threads x processor rows x processor columns) Buluc et al. Distributed BFS (https://arxiv.org/abs/1705.04590) MPI_Isend (8 B) 60 Communication Million Messages/s performance of 40 Model MPI everywhere MPI+threads is MPI+threads (MPI_THREAD_FUNNELED) MPI+threads (MPI_THREAD_MULTIPLE) 20 dismal 0 1 2 4 8 16 Number of cores

  7. 7 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI Node Outdated view : Network is a single device Modern reality : Network features parallelism Network Interface Card Network Interface Card Network hardware context

  8. 8 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI MPI everywhere MPI+threads P0 P1 P2 P3 P0 Application MPI library Network Interface Card Network Interface Card Software communication channel Network hardware context

  9. 9 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI MPI everywhere MPI+threads P0 P1 P2 P3 P0 Application MPI library Network Interface Card Network Interface Card Global critical section + 1 communication channel per process Software communication channel Network hardware context

  10. 10 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI No logical MPI everywhere MPI+threads parallelism expressed P0 P1 P2 P3 P0 Application MPI library Network Interface Card Network Interface Card Global critical section + 1 communication channel per process Software communication channel Network hardware context

  11. 11 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI MPI_Comm_create_endpoints(…,num_ep,…,comm_eps[]); MPI_Isend/Irecv(…,comm_eps[tid],ep_rank,…); MPI process EP0 EP2 EP3 EP4 MPI library Network Interface Card MPI Communicator MPI Endpoint Software communication channel Network hardware context

  12. 12 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI MPI_Comm_create_endpoints(…,num_ep,…,comm_eps[]); MPI_Isend/Irecv(…,comm_eps[tid],ep_rank,…); MPI process Pros ▸ Explicit control over network contexts EP0 EP2 EP3 EP4 Cons MPI library ▸ Intrusive extension of the MPI standard ▸ Onus of managing network contexts on user Network Interface Card MPI Communicator MPI Endpoint Software communication channel Network hardware context

  13. 13 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI Logical parallelism MPI everywhere MPI+threads expressed P0 P0 P1 P2 P3 Application C0 C1 C2 C3 MPI library Network Interface Card Network Interface Card Fine-grained critical sections + Multiple communication channel per process MPI Communicator Software communication channel Network hardware context

  14. 14 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI Do we need user-visible endpoints? Logical parallelism MPI everywhere MPI+threads expressed P0 P0 P1 P2 P3 Application C0 C1 C2 C3 MPI library Network Interface Card Network Interface Card Fine-grained critical sections + Multiple communication channel per process MPI Communicator Software communication channel Network hardware context

  15. 15 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI CONTRIBUTIONS AS DEVIL’S ADVOCATE ▸ In-depth comparison between MPI-3.1 and user-visible endpoints ▸ A fast MPI+threads library that adheres to MPI-3.1’s constraints ▸ Optimized parallel communication streams applicable to all MPI libraries ▸ Recommendations for the MPI user to express logical parallelism with MPI-3.1 MPI library Interconnects Evaluation ▸ Based on ▸ Intel Omni-Path (OPA) with OFI:PSM2 platforms MPICH:CH4 ▸ Mellanox InfiniBand (IB) with UCX:Verbs

  16. 16 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI OUTLINE ▸ Introduction ▸ For MPI users: Parallelism in the MPI standard ▸ For MPI developers: Fast MPI+threads ▸ Fine-grained critical sections for thread safety ▸ Virtual Communication Interfaces (VCIs) for parallel communication streams ▸ Microbenchmark and Application analysis

  17. 17 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI POINT-TO-POINT COMMUNICATION ▸ <comm,rank,tag> decides matching ▸ Non-overtaking order ▸ Receive wildcards Two or more operations on a Can be issued on parallel process with communication streams? Comm Rank Tag Send Recv

  18. 18 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI POINT-TO-POINT COMMUNICATION ▸ <comm,rank,tag> decides matching Rank 0 (sender) ▸ Non-overtaking order ▸ Receive wildcards Two or more operations on a Can be issued on parallel <CA,R1,T1> <CB,R1,T1> process with communication streams? Comm Rank Tag Send Recv <CA,R0,T1> <CB,R0,T1> Different or Different or Different Yes Yes Same Same Rank 1 (receiver)

  19. 19 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI POINT-TO-POINT COMMUNICATION ▸ <comm,rank,tag> decides matching Rank 0 (sender) ▸ Non-overtaking order ▸ Receive wildcards Two or more operations on a Can be issued on parallel <CA,R1,T1> <CA,R2,T1> process with communication streams? Comm Rank Tag Send Recv <CA,R0,T1> <CA,ANY,T1> Different or Different or Different Yes Yes Same Same Different or Same Different Yes No Rank 1 (receiver) Same Wildcards

  20. 20 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI POINT-TO-POINT COMMUNICATION Rank 0 (sender) ▸ <comm,rank,tag> decides matching ▸ Non-overtaking order ▸ Receive wildcards <CA,R1,T1> Two or more operations on a Can be issued on parallel process with communication streams? <CA,R1,T2> Comm Rank Tag Send Recv Different or Different or Different Yes Yes <CA,R0,T3> <CA,R0,ANY> Same Same Different or Same Different Yes No Same Wildcards Different or Rank 1 (receiver) Same Same No No Same Non-overtaking order

  21. 21 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI RMA COMMUNICATION Two or more operations Can be issued on parallel communication streams? on a process with Window Rank Put Get Accumulate Different or Different Yes Yes Yes Same Same Different Yes Yes Yes Same Same Yes Yes No

  22. 22 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI RMA COMMUNICATION Two or more operations Can be issued on parallel communication streams? on a process with Explicitly expressing Window Rank Put Get Accumulate parallelism Different or Different Yes Yes Yes Same Same Different Yes Yes Yes Implicit parallelism Same Same Yes Yes No No order between multiple Gets and Puts

  23. 23 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI RMA COMMUNICATION Two or more operations Can be issued on parallel communication streams? on a process with Explicitly expressing Window Rank Put Get Accumulate parallelism Different or Different Yes Yes Yes Same Same Different Yes Yes Yes Implicit parallelism Same Same Yes Yes No Ordering of accumulate No order between operations to the same multiple Gets and Puts memory location

  24. 24 HOW I LEARNED TO STOP WORRYING ABOUT USER-VISIBLE ENDPOINTS AND LOVE MPI OUTLINE ▸ Introduction ▸ For MPI users: Parallelism in the MPI standard ▸ For MPI developers: Fast MPI+threads ▸ Fine-grained critical sections for thread safety ▸ Virtual Communication Interfaces (VCIs) for parallel communication streams ▸ Microbenchmark and Application analysis

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Lessons Learned Lessons Learned From From Lessons Learned Lessons Learned From From

Lessons Learned Lessons Learned From From Lessons Learned Lessons Learned From From

Protg 2006, July 26th, Stanford I R I S A C 1 R E C Saint-Cyr Lessons Learned Lessons Learned From From Lessons Learned Lessons Learned From From Ontology Ontology Design Design Ontology Ontology Design Design Jean Andr B

371 views • 12 slides
Wh t h What have we learned from What have we learned from Wh t h l l d f d f building the

Wh t h What have we learned from What have we learned from Wh t h l l d f d f building the

Wh t h What have we learned from What have we learned from Wh t h l l d f d f building the LHC (CMS) DAQ building the LHC (CMS) DAQ systems. systems. S. Cittolin PH-CMD. CERN Openlab meeting. 3-4 March 2009 DAQ at LHC overview CMS

352 views • 33 slides
4 5 6 CSE 142 vs CSE 143 CSE 142 / AP CS A CSE 143 You learned how to write Return of

4 5 6 CSE 142 vs CSE 143 CSE 142 / AP CS A CSE 143 You learned how to write Return of

4 5 6 CSE 142 vs CSE 143 CSE 142 / AP CS A CSE 143 You learned how to write Return of the objects programs and You learned to solve decompose large more complex tasks problems with: efficiently Print statements Data

705 views • 22 slides
WHAT HAVE WHAT HAVE I LEARNED I LEARNED AT THE AT THE DISC DISC SUMMER SCHOOL? SUMMER

WHAT HAVE WHAT HAVE I LEARNED I LEARNED AT THE AT THE DISC DISC SUMMER SCHOOL? SUMMER

WHAT HAVE WHAT HAVE I LEARNED I LEARNED AT THE AT THE DISC DISC SUMMER SCHOOL? SUMMER SCHOOL? Christos G. Cassandras CODES Lab. - Boston University WHY HYBRID SYSTEMS ? COMPLEXITY ( PHYSICAL , BUT MOSTLY OPERATIONAL/ORGANIZATIONAL )

691 views • 11 slides
What We Have Learned from the Pandemic NURSING SIMULATION: What We Have Learned from the

What We Have Learned from the Pandemic NURSING SIMULATION: What We Have Learned from the

July 17, 2020 Nursing Simulation: What We Have Learned from the Pandemic NURSING SIMULATION: What We Have Learned from the Pandemic PAMELA R. JEFFRIES CRYSTEL L. FARINA PAUL LUND PAMELA SLAVEN-LEE George Washington University George

790 views • 21 slides
Object-Oriented Design What Do We Mean by OO Design? Remember how we learned about functions?

Object-Oriented Design What Do We Mean by OO Design? Remember how we learned about functions?

Module 21 Object-Oriented Design What Do We Mean by OO Design? Remember how we learned about functions? First learned to call functions made for us Then learned to define our functions Finally learned to properly design functions

471 views • 30 slides
has learned Col Rakesh Jetly OMM,CD,MD,FRCPC. Lots of Ground to Cover!! Why am I here?

has learned Col Rakesh Jetly OMM,CD,MD,FRCPC. Lots of Ground to Cover!! Why am I here?

Mental Health In the Workplace: What CAF has learned Col Rakesh Jetly OMM,CD,MD,FRCPC. Lots of Ground to Cover!! Why am I here? Lessons learned from Military medicine CAF in Kandahar MH Lessons Learned MH in The

1.18k views • 80 slides
Electrical Team Marina Beshai, Leslie Kim, and Sara Sacks Overview of the Semester: Learned

Electrical Team Marina Beshai, Leslie Kim, and Sara Sacks Overview of the Semester: Learned

Electrical Team Marina Beshai, Leslie Kim, and Sara Sacks Overview of the Semester: Learned how to test the point system. Learned how to use the oscilloscope. Started the portable power generator. Solder: Learned how to

252 views • 14 slides
What have we learned after 10 years and 120,000 cryoablations? Richard Schilling conflicts -

What have we learned after 10 years and 120,000 cryoablations? Richard Schilling conflicts -

What have we learned after 10 years and 120,000 cryoablations? Richard Schilling conflicts - speaker fees and research grants Medtronic, Biosense Webster Talk outline Brief procedure description What have I learned anecdote, personal

575 views • 23 slides
Institutionalizing Lessons Learned October 25, 2006 Loren Plisco Region II Background

Institutionalizing Lessons Learned October 25, 2006 Loren Plisco Region II Background

Institutionalizing Lessons Learned October 25, 2006 Loren Plisco Region II Background SEP 2002 Davis-Besse Lessons Learned Task Force AUG 2004 - Effectiveness Review Lessons Learned Task Force JAN 2005 - EDO Charters

269 views • 22 slides
OSHA Lessons Learned Adam Fries OSHA Compliance Officer February 13, 2018 OSHA Lessons Learned

OSHA Lessons Learned Adam Fries OSHA Compliance Officer February 13, 2018 OSHA Lessons Learned

OSHA Lessons Learned Adam Fries OSHA Compliance Officer February 13, 2018 OSHA Lessons Learned Jobsite Safety and Health Inspections/Audits better known as; PAPER WORK You hate it, Your safety director requires it, OSHA loves it, Does it

926 views • 35 slides
3/8/2019 Epidemiology, Risk Factors, and Outcomes of Pediatric PVD: LESSONS learned from the

3/8/2019 Epidemiology, Risk Factors, and Outcomes of Pediatric PVD: LESSONS learned from the

3/8/2019 Epidemiology, Risk Factors, and Outcomes of Pediatric PVD: LESSONS learned from the Spanish Registry: Lessons Learned from the Registries o Is It possible (and worthy) to do a national registry? Lessons learned from the o

1.16k views • 43 slides
DEBUGGING LESSONS LEARNED WHILE DEBUGGING LESSONS LEARNED WHILE FIXING NETBSD FIXING NETBSD

DEBUGGING LESSONS LEARNED WHILE DEBUGGING LESSONS LEARNED WHILE FIXING NETBSD FIXING NETBSD

DEBUGGING LESSONS LEARNED WHILE DEBUGGING LESSONS LEARNED WHILE FIXING NETBSD FIXING NETBSD ABOUT ME ABOUT ME maya@NetBSD.org coypu@sdf.org NetBSD/pkgsrc for the last 3 years THIS TALK THIS TALK Mix of a bunch of bugs Not solo work

695 views • 31 slides
Lessons Learned A Value Added Product of the Project Life Cycle R Gilman April 19, 2006 Agenda

Lessons Learned A Value Added Product of the Project Life Cycle R Gilman April 19, 2006 Agenda

Lessons Learned A Value Added Product of the Project Life Cycle R Gilman April 19, 2006 Agenda Introduction What are Lessons Learned? Value to the Project Process and Organization Keanes Approach to Lessons Learned Keys to

261 views • 24 slides
LEARNED SO FAR C A R M E N &amp; N I C O L E &amp; A N I TA D D C I N T E R N S 2 0 1 7

LEARNED SO FAR C A R M E N &amp; N I C O L E &amp; A N I TA D D C I N T E R N S 2 0 1 7

WHAT WEVE LEARNED SO FAR C A R M E N &amp; N I C O L E &amp; A N I TA D D C I N T E R N S 2 0 1 7 THE HISTORY OF DISABILITY Di Disability sability Hi Hist story ory st start rted ed in n 40 400 0 B.C. Disability

439 views • 15 slides
{ We are at the end of our first cycle of Learning Outcomes Assessment What has COA learned:

{ We are at the end of our first cycle of Learning Outcomes Assessment What has COA learned:

{ We are at the end of our first cycle of Learning Outcomes Assessment What has COA learned: Weve learned that Learning Assessment is not going away. Weve learned that Learning Assessment can provide valuable information.

313 views • 15 slides
Update Parallelism April 30, 2018 1 HW 3 Posted 2 Parallelism Models Option 4: Shared

Update Parallelism April 30, 2018 1 HW 3 Posted 2 Parallelism Models Option 4: Shared

Update Parallelism April 30, 2018 1 HW 3 Posted 2 Parallelism Models Option 4: Shared Nothing in which all communication is explicit. CPU Memory Disk Well be talking about shared nothing today. Other models are easier

1.56k views • 106 slides
C S P , C a s p e r a n d S e c u r i t y V e r i f i c a t i o n

C S P , C a s p e r a n d S e c u r i t y V e r i f i c a t i o n

C S P , C a s p e r a n d S e c u r i t y V e r i f i c a t i o n 2 0 0 3 . 8 . 2 0 S e c u r i t y S e c u r i t y : C o n f i d

497 views • 31 slides
Verifying Business Process Compatibility Peter Wong, University of Oxford, UK (Joint work with

Verifying Business Process Compatibility Peter Wong, University of Oxford, UK (Joint work with

01 Verifying Business Process Compatibility Peter Wong, University of Oxford, UK (Joint work with Jeremy Gibbons) June 2007 02 Introduction Business processes may be described as a Business Process Modelling Notation (BPMN) diagram;

517 views • 37 slides
Chapter 4: Processes Process Concept Process Scheduling Operations on Processes

Chapter 4: Processes Process Concept Process Scheduling Operations on Processes

Chapter 4: Processes Process Concept Process Scheduling Operations on Processes Cooperating Processes Interprocess Communication Communication in Client-Server Systems Operating System Concepts Silberschatz, Galvin and

672 views • 19 slides
DISTRIBUTING INFORMATION AGENDA What does information distribution mean? Delegation Closures

DISTRIBUTING INFORMATION AGENDA What does information distribution mean? Delegation Closures

DISTRIBUTING INFORMATION AGENDA What does information distribution mean? Delegation Closures Notifications Property Observers Bindings / Key Value Observation INFORMATION DISTRIBUTION Mobile apps are highly interactive Multiple interfaces

642 views • 39 slides
LRTS: A Portable High Performance Low-level Communication Interface Yanhua Sun 1 ale 1

LRTS: A Portable High Performance Low-level Communication Interface Yanhua Sun 1 ale 1

LRTS: A Portable High Performance Low-level Communication Interface Yanhua Sun 1 ale 1 Laxmikant(Sanjay) V. K 1 University of Illinois at Urbana-Champaign sun51@illinois.edu April 15, 2013 Yanhua Sun U of Illinois at Urbana-Champaign 1/24

347 views • 32 slides
Distributed Systems Secure Communication Paul Krzyzanowski pxk@cs.rutgers.edu Except as

Distributed Systems Secure Communication Paul Krzyzanowski pxk@cs.rutgers.edu Except as

Distributed Systems Secure Communication Paul Krzyzanowski pxk@cs.rutgers.edu Except as otherwise noted, the content of this presentation is licensed under the Creative Commons Attribution 2.5 License. Symmetric cryptography Both parties

860 views • 60 slides
System-on-Chip Design HW/SW Interfaces and Communica;ons Hao Zheng Comp Sci &amp; Eng U of

System-on-Chip Design HW/SW Interfaces and Communica;ons Hao Zheng Comp Sci &amp; Eng U of

System-on-Chip Design HW/SW Interfaces and Communica;ons Hao Zheng Comp Sci &amp; Eng U of South Florida 1 System Structural Model Mem CPU P1 P2 Arbiter Bridge P3 P5 P4 HW HW 2 Basic Elements of HW/SW Interfaces 1. On-chip

235 views • 22 slides

More recommend