Lecture 19: Load Balance Abhinav Bhatele, Department of Computer - - PowerPoint PPT Presentation

▶
lecture 19 load balance
SMART_READER_LITE
LIVE PREVIEW

Lecture 19: Load Balance Abhinav Bhatele, Department of Computer - - PowerPoint PPT Presentation

Jan 07, 2024 694 likes •916 views

Introduction to Parallel Computing (CMSC498X / CMSC818X) Lecture 19: Load Balance Abhinav Bhatele, Department of Computer Science Announcements Quiz 2: Nov 12 Quiz 3: Dec 3 Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING 2

slide-1
SLIDE 1

Lecture 19: Load Balance

Abhinav Bhatele, Department of Computer Science

Introduction to Parallel Computing (CMSC498X / CMSC818X)

slide-2
SLIDE 2

Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING

Announcements

  • Quiz 2: Nov 12
  • Quiz 3: Dec 3

2

slide-3
SLIDE 3

Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING

Performance issues

  • Algorithmic overhead
  • Speculative loss
  • Critical paths
  • Bottlenecks
  • Sequential performance issues
  • Load imbalance
  • Communication performance

3

slide-4
SLIDE 4

Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING

Load imbalance

4

slide-5
SLIDE 5

Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING

Load imbalance

  • Definition: unequal amounts of “work” assigned to different processes
  • Work could be computation or communication or both

4

slide-6
SLIDE 6

Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING

Load imbalance

  • Definition: unequal amounts of “work” assigned to different processes
  • Work could be computation or communication or both
  • Why is load imbalance bad?
  • Overloaded processes can slow down everyone

4

slide-7
SLIDE 7

Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING

Load imbalance

  • Definition: unequal amounts of “work” assigned to different processes
  • Work could be computation or communication or both
  • Why is load imbalance bad?
  • Overloaded processes can slow down everyone

4

Load imbalance = max_load avg_load

slide-8
SLIDE 8

Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING

Load balancing

  • The process of balancing load across threads, processes etc.
  • Goal: to bring the maximum load close to average as much as possible
  • Determine if load balancing is needed
  • Determine when to load balance
  • Determine what information to gather/use for load balancing

5

slide-9
SLIDE 9

Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING

Is load balancing needed?

  • Need the distribution of load (“work”) across processes
  • Collect empirical information using performance tools
  • Developer knowledge
  • Analytical models of load distribution

6

slide-10
SLIDE 10

Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING

When to load balance?

  • Initial work distribution or static load balancing
  • At program startup
  • Or sometimes in a separate run to determine load distribution
  • Dynamic load balancing: does load distribution evolve over time?
  • During program execution

7

slide-11
SLIDE 11

Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING

Information gathering for load balancing

  • Centralized load balancing
  • Gather all load information at one process — global view of data
  • Distributed load balancing
  • Every process only knows the load of a constant number of “neighbors”
  • Hybrid or hierarchical load balancing

8

slide-12
SLIDE 12

Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING

What information is used for load balancing

  • Computational load
  • Possibly, communication load (number/sizes of messages)
  • Communication graph

9

slide-13
SLIDE 13

Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING

Load balancing algorithms

  • Input: Amount of work (ni) assigned to each process pi
  • Output: New assignments of work units to different processes
  • Goals:
  • Bring maximum load close to average
  • Minimize the amount of load migration
  • Secondary goals:
  • Balance (possibly reduce) communication load
  • Keep the time for doing load balancing short

10

slide-14
SLIDE 14

Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING

Examples of static load balancing

  • Decomposition of 2D Stencil
  • Using orthogonal recursive bisection (ORB)

11

http://datagenetics.com/blog/march22013/ https://en.wikipedia.org/wiki/Z-order_curve

slide-15
SLIDE 15

Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING

Examples of static load balancing

  • Decomposition of 2D Stencil
  • Using orthogonal recursive bisection (ORB)

11

http://datagenetics.com/blog/march22013/ https://en.wikipedia.org/wiki/Z-order_curve

slide-16
SLIDE 16

Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING

Examples of static load balancing

  • Decomposition of 2D Stencil
  • Using orthogonal recursive bisection (ORB)

11

http://datagenetics.com/blog/march22013/ https://en.wikipedia.org/wiki/Z-order_curve http://charm.cs.uiuc.edu/workshops/charmWorkshop2011/slides/CharmWorkshop2011_apps_ChaNGa.pdf

Space- filling curves ORB

slide-17
SLIDE 17

Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING

Simple greedy strategy

12

slide-18
SLIDE 18

Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING

Other considerations

  • Communication-aware load balancing
  • Network topology-aware load balancing

13

slide-19
SLIDE 19

Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING

Hierarchical load balancing

14

1024 63488 64512 1023 1024 2047 63488 64511 64512 65535 Token Object Load Data Load Data Greedy load balancing Refinement load balancing

slide-20
SLIDE 20

Abhinav Bhatele 5218 Brendan Iribe Center (IRB) / College Park, MD 20742 phone: 301.405.4507 / e-mail: bhatele@cs.umd.edu