Using Sampling to Understand Parallel Program Performance Nathan - - PowerPoint PPT Presentation
Using Sampling to Understand Parallel Program Performance Nathan Tallent John Mellor-Crummey M. Krentel, L. Adhianto, M. Fagan, X. Liu Dept. of Computer Science, Rice University Parallel Tools Workshop TU Dresden Sept 26-27, 2011
hpctoolkit.org
Parallel Tools Workshop • TU Dresden • Sept 26-27, 2011
Nathan Tallent John Mellor-Crummey
Monday, September 26, 2011
High-performance Infrastructure for Scalable Tools
— in conjunction with PPoPP 2012 — February 25-29, 2012 — New Orleans, LA, USA
— special emphasis on performance and scalability — special emphasis on infrastructure
2
Monday, September 26, 2011
— monitors every instance of something — must be used with care
– may preclude measurement of production runs – encourages selective instrumentation ➟ blind spots
procedures (systematic error)
— note: specialized techniques may reduce overheads
— provides representative and detailed measurements
3
Instrumentation types
Use sampling when possible; use instrumentation when necessary.
Monday, September 26, 2011
— attribution to outer loops trivial with source-code instrumentation
— stack unwinding can be hard: try using libunwind or GLIBC backtrace() in a SEGV handler for optimized code
— perfect stack unwinding not sufficient
— it is possible to miss the first cause in a chain of events
— with instrumentation, it is easy to track lock acquire/release
4
Monday, September 26, 2011
— parallel idleness and overhead in work stealing — lock contention — load imbalance
5
Monday, September 26, 2011
Measure & attribute costs in calling context
Sampling-based Call Path Profiling is Easy...*
6
Call path sample Calling Context Tree (CCT)
Overhead proportional to sampling frequency... ...not call frequency
instruction pointer return address return address return address
“main” sample point
CCT size scales well
Monday, September 26, 2011
— ✓ overhead proportional to sampling frequency
— ✓ overhead effects are not systematic — ✗ must be able to unwind from every point in an executable
— frame pointers (linking every frame on the stack)
— complete (and correct) unwind information
7
Dynamically analyze binary to compute unwind information
Monday, September 26, 2011
— for dynamically linked code, do at runtime — for statically linked code, do at compile time
— scan the procedure’s object code, tracking the locations of
— create unwind recipes between pairs of frame-relevant instructions
— accurate call path profiles — overheads of < 2% for sampling frequencies of 200/s
8
PLDI 2009. Distinguished Paper Award.
Monday, September 26, 2011
— call paths differ from user-level
— loops differ from user-level
— to be useful, a tool should
— cannot use instrumentation to measure user-level loops
9
Statically analyze binary to compute an object to source-code mapping
Monday, September 26, 2011
1-2% overhead
costs for
PLDI 2009. Distinguished Paper Award.
Monday, September 26, 2011
— parallel idleness and overhead in work stealing — lock contention — load imbalance
11
Monday, September 26, 2011
— measure at scale
– record data on a process with probability p – simplification of Gamblin et al., IPDPS ’08
— analyze measurements
— present performance data (without requiring an Altix)
— identify scaling bottlenecks in context…
12
simple and effective
Monday, September 26, 2011
13
Pinpointing & Quantifying Scalability Bottlenecks
200 s. 400 s. 600 s.
P × − P Q Q ×
Weak scaling : no coefficients Strong scaling : red coefficients Average CCTP Average CCTQ
Coarfa, Mellor-Crummey, Froyd, Dotsenko. ICS’07 SC 2009
Monday, September 26, 2011
14
Weak scaling on BG/P 256 to 8192 cores
21% of scaling loss is due to looping
build adaptive mesh
Monday, September 26, 2011
15 Graph courtesy of Anshu Dubey, U Chicago Note: lower is better
Monday, September 26, 2011
— parallel idleness and overhead in work stealing — lock contention — load imbalance
16
Monday, September 26, 2011
losses rather than causes
— worker threads waiting for work — threads waiting for a lock — MPI process waiting for peers in a collective communication
— active measurement — post-mortem analysis only
17
Monday, September 26, 2011
18
cilk int fib(n) { if (n < 2) return n; else { int x, y; x = spawn fib(n-1); y = spawn fib(n-2); sync; return (x + y); } }
f(n-2) f(n) f(n-1) f(n-3) f(n-2) f(n-4) f(n-3)
... ... ... ... ... ... ... ...
spawn: asynchronous call; creates a logical task that
spawn + recursion (fib): quickly creates significant logical parallelism
To map logical tasks to compute cores: lazy thread creation + work-stealing scheduler
Monday, September 26, 2011
19
thread 1 thread 2 thread 3 f(n-1) f(n-3)
...
f(n) f(n-1) f(n-3)
...
Logical call path profiling: Recover full relationship between physical and source-level execution Problem: Work stealing separates source-level calling contexts in space and time
f(n-2) f(n) f(n-1) f(n-3) f(n-2) f(n-4) f(n-3)
... ... ... ... ... ... ... ...
Consider thread 3:
Monday, September 26, 2011
— parallel idleness: parallelism is too coarse-grained — parallel overhead: parallelism is too fine-grained — idleness and overhead: wrong algorithm
— measuring idleness
in the scheduler!
— measuring overhead
machine instructions
20
while (noWork) { t = random thread; try stealing from t; } save-live-vars();
user-work(); retract-task(); You spent time here!
Goal: Insightfully attribute idleness and overhead to logical contexts
PPoPP 2009 IEEE Computer 12/09
Monday, September 26, 2011
— parallel idleness: when a thread (core) is idle or blocked
— before: blame idleness on self (victim) — now: blame idleness on working threads (perpetrators)
— maintain a small amount of global state: # of workers, # idlers
— on sampling a working thread t:
— on sampling an idle thread t:
21 apportion blame for current idleness
Monday, September 26, 2011
— overhead: when a thread works on non-user code
— instrument to distinguish between useful-work & overhead
— post-mortem analysis:
22 no measurement overhead!
Monday, September 26, 2011
23
Pinpoints serial initialization/finalization routines.
We can pinpoint and quantify the effect of serialization.
percent percent
Monday, September 26, 2011
— parallel idleness and overhead in work stealing — lock contention — load imbalance
24
Monday, September 26, 2011
— fine-grain locking is the gold-standard for performance — explicit threading (e.g., Pthreads) — implementing higher-level models
— when a thread (and therefore core) is idling waiting for a lock
— chief form of parallel idleness in many programs
25
Monday, September 26, 2011
— But: tool must not itself become a bottleneck!
— imprecise: blames all workers (rather than one) for idleness
— on sampling a working thread t:
— on sampling an idle thread t waiting for lock L:
— when working thread t releases lock L:
26 Associate idleness with lock, not thread. exactly represents contention while t held lock L
Monday, September 26, 2011
27
lock contention accounts for 23.5%
Adding futures to shared global work queue. µs 16 cores; 1 thread/core (4 x Barcelona) Quantum chemistry; MPI + pthreads
1-5% overhead
PPoPP 2010
Monday, September 26, 2011
— parallel idleness and overhead in work stealing — lock contention — load imbalance
28
Monday, September 26, 2011
29
— example: SCALASCA
scale executions using call path profiling?
— ✓ sample-based call path profiles scale and are very detailed — ✗ but they collapse potential causal chains
Monday, September 26, 2011
30
manifested in idleness (e.g., MPI_Cray_Progress_Wait)
(procedures or loops): A balance point cannot contribute to imbalance
point (nearer to cause)
scatter plots... CCT
Precise! Post mortem! 0 measurement overhead!
Monday, September 26, 2011
31
call sites...
sensitive cycle values: Early finishers... ...become early arrivers at Allreduce
a balance point. Imbalance: blame-shifted idleness. 8K cores, Cray XT5 SC 2010
Monday, September 26, 2011
— parallel idleness and overhead in work stealing — lock contention — load imbalance
32
Monday, September 26, 2011
— but tracing is difficult to scale to long or large executions
— on each sample, record call path of each thread — organize the samples for each thread along a time line — view how the execution hierarchically evolves
— use sampling to scalably render large-scale traces
33
Processes Time Call stack
slice
“main”
ICS 2011
Monday, September 26, 2011
34
PFLOTRAN, 8184 processes, Cray XT5
MPI rank time
Process-time view at selected depth Depth-time view for selected rank
Call path depth
Monday, September 26, 2011
— we have a display window of dimensions h × w — typically many more processes (or threads) than h — typically many more samples (trace records) than w
35
Trace with n processes
time process p1 pi pn
h w
process time
each sample defines a pixel
samples (of samples)
On a laptop!
Monday, September 26, 2011
36
— but, important anomalies will have (local/non-local) effects
Using sampling for both measurement and presentation clearly exposed the problem.
In an unusual execution, 8184 processes took 190 s to complete MPI_Init! (FLASH, JaguarPF, Cray XT5) a lagging process...
Monday, September 26, 2011
— parallel idleness and overhead in work stealing — lock contention — load imbalance
37
Monday, September 26, 2011
call path profiling, binary analysis, and blame-shifting
attribution for fully optimized code (1-3% overhead)
— statements in their full static and dynamic context — project low-level measurements to much higher levels
— scalability bottlenecks — parallel inefficiency in work-stealing — causes of lock contention — load imbalance — temporal patterns — problematic data structures (work by Xu Liu)
38
Monday, September 26, 2011
39 Attribute Costs to Code Analyze Behavior
Assess Imbalance and Variability Associate Costs with Data Shift Blame from Symptoms to Causes Pinpoint & Quantify Scaling Bottlenecks
hpctoolkit.org
Monday, September 26, 2011