A Topology-Aware Performance Monitoring Tool for Shared Resource - - PowerPoint PPT Presentation

▶
a topology aware performance monitoring tool for shared
SMART_READER_LITE
LIVE PREVIEW

A Topology-Aware Performance Monitoring Tool for Shared Resource - - PowerPoint PPT Presentation

Mar 30, 2023 145 likes •450 views

A Topology-Aware Performance Monitoring Tool for Shared Resource Management in Multicore Systems TADaaM Team - Nicolas Denoyelle - Brice Goglin - Emmanuel Jeannot August 24, 2015 1. Context/Motivations 2. Fast presentation of the tool 3.

slide-1
SLIDE 1

TADaaM Team - Nicolas Denoyelle - Brice Goglin - Emmanuel Jeannot August 24, 2015

A Topology-Aware Performance Monitoring Tool for Shared Resource Management in Multicore Systems

slide-2
SLIDE 2
  • 1. Context/Motivations
  • 2. Fast presentation of the tool
  • 3. Demonstration
  • 4. How does it works ?
  • 5. How is it made ?
  • 6. Features & Future Works

TADaaM Team - Nicolas Denoyelle - Brice Goglin - Emmanuel Jeannot August 24, 2015

slide-3
SLIDE 3

Topology Aware Performance Monitoring August 24, 2015- 3

MOTIVATIONS

Memory hierarchy is growing deeper and larger. No performance without a fair usage of the system topology Batch schedulers, runtimes, applications themeselves . . . are getting topology aware.

IO Network Machine NUMA Shared Memory Shared Cache Private Cache Core Processing Unit PU Private Cache Core PU PU NUMA Shared Memory Shared Cache Private Cache Core PU PU Private Cache Core PU PU

slide-4
SLIDE 4

Topology Aware Performance Monitoring August 24, 2015- 4

MOTIVATIONS

Memory hierarchy is growing deeper and larger. Hence, data management gives

  • pportunities for

performance improvements.

IO Network Machine NUMA Shared Memory Shared Cache Private Cache Core Processing Unit PU Private Cache Core PU PU NUMA Shared Memory Shared Cache Private Cache Core PU PU Private Cache Core PU PU

slide-5
SLIDE 5

Topology Aware Performance Monitoring August 24, 2015- 4

MOTIVATIONS

Memory hierarchy is growing deeper and larger. Hence, data management gives

  • pportunities for

performance improvements.

IO Network Machine NUMA Shared Memory Shared Cache Private Cache Core Processing Unit PU Private Cache Core PU PU NUMA Shared Memory Shared Cache Private Cache Core PU PU Private Cache Core PU PU

slide-6
SLIDE 6

Topology Aware Performance Monitoring August 24, 2015- 4

MOTIVATIONS

Memory hierarchy is growing deeper and larger. Hence, data management gives

  • pportunities for

performance improvements. It is a multi-level and a multi-criteria problem.

IO Network Machine NUMA Shared Memory Shared Cache Private Cache Core Processing Unit PU Private Cache Core PU PU NUMA Shared Memory Shared Cache Private Cache Core PU PU Private Cache Core PU PU

slide-7
SLIDE 7

Topology Aware Performance Monitoring August 24, 2015- 5

MOTIVATIONS

  • Need to match use cases, and relevant performance

metrics for each level.

  • Need to match performance and topology.
  • Requires topology modeling skills.
  • Requires adaptable performance monitoring.
slide-8
SLIDE 8

Yet Another Tool to Monitor Applications Performance

Topology Aware Performance Monitoring August 24, 2015- 6

  • Focus on data presentation to link the results with topology

informations.

  • Relies on two cornerstones of topology modeling (hwloc)

and performance counter abstraction (PAPI) to map the latter on the former

  • Minimal configuration and software requirements.
  • Can help finding and caracterizing localized bottlenecks.
slide-9
SLIDE 9

Hardware Locality (hwloc)

Portable abstraction of hierarchical architectures for high-performance computing

  • Performs topology discovery and extracts hardware

component information.

  • Provides tools for memory and process binding.
  • Many operating systems supported
  • ...
  • lstopo utility to display the topology:

Developped at Inria Bordeaux.

Topology Aware Performance Monitoring August 24, 2015- 7

slide-10
SLIDE 10

Hardware Locality (hwloc)

Topology Aware Performance Monitoring August 24, 2015- 7

Machine (31GB total) NUMANode P#0 (31GB) Package P#0 L3 (20MB) L2 (256KB) L1d (32KB) L1i (32KB) Core P#0 PU P#0 PU P#16 L2 (256KB) L1d (32KB) L1i (32KB) Core P#1 PU P#2 PU P#18 L2 (256KB) L1d (32KB) L1i (32KB) Core P#2 PU P#4 PU P#20 L2 (256KB) L1d (32KB) L1i (32KB) Core P#3 PU P#6 PU P#22 L2 (256KB) L1d (32KB) L1i (32KB) Core P#4 PU P#8 PU P#24 L2 (256KB) L1d (32KB) L1i (32KB) Core P#5 PU P#10 PU P#26 L2 (256KB) L1d (32KB) L1i (32KB) Core P#6 PU P#12 PU P#28 L2 (256KB) L1d (32KB) L1i (32KB) Core P#7 PU P#14 PU P#30

slide-11
SLIDE 11

Performance Application Programming Interface (PAPI)

Consistent interface and methodology for use of the performance counter hardware.

  • Real time relation between software performance and

processor events.

  • Many operating systems supported too.
  • Reliable and actively supported.
  • Used in a wide range of performance analysis applications.

An abstraction layer to plug some other performance library is under development.

Topology Aware Performance Monitoring August 24, 2015- 8

slide-12
SLIDE 12

Dynamic Lstopo (example)

Topology Aware Performance Monitoring August 24, 2015- 9

Machine (16GB total) NUMANode P#0 (16GB) Package P#0 L3 (20MB) L2 (256KB) L1d (32KB) L1i (32KB) Core P#0 PU P#0 PU P#16 L2 (256KB) L1d (32KB) L1i (32KB) Core P#1 PU P#2 PU P#18 L2 (256KB) L1d (32KB) L1i (32KB) Core P#2 PU P#4 PU P#20 L2 (256KB) L1d (32KB) L1i (32KB) Core P#3 PU P#6 PU P#22 L2 (256KB) L1d (32KB) L1i (32KB) Core P#4 PU P#8 PU P#24 L2 (256KB) L1d (32KB) L1i (32KB) Core P#5 PU P#10 PU P#26 L2 (256KB) L1d (32KB) L1i (32KB) Core P#6 PU P#12 PU P#28 L2 (256KB) L1d (32KB) L1i (32KB) Core P#7 PU P#14 PU P#30 9,8000000000e+01 3,0100000e+02 2,1900000e+02 5,0700000e+02 2,4000000e+02 2,4000000e+02 2,7600000e+02 2,2400000e+02 2,5900000e+02 6,8200000e+02 6,4200000e+02 1,0560000e+03 7,2600000e+02 6,8700000e+02 6,8700000e+02 6,8200000e+02 6,5800000e+02 1,7130000e+03 1,7440000e+03 2,3410000e+03 1,7670000e+03 1,7380000e+03 1,7220000e+03 1,7350000e+03 1,7140000e+03 2,49344e+00 1,51514e+00 2,58947e+00 1,47808e+00 1,47165e+00 3,47249e+00 2,82472e+00 1,63417e+00 2,64271e+00 1,51689e+00 1,52142e+00 2,68772e+00 1,47441e+00 2,45916e+00 1,40868e+00 2,82176e+00

Sample of hardware performance counters mapped on a single socket of an Intel Xeon E5-2650 CPU.

slide-13
SLIDE 13

A Demonstration Worth Thousand Words

Topology Aware Performance Monitoring August 24, 2015- 10 PU0 PU1 PU2 PU3 L1 L2 L3 ∗4 . . .

Accesses to a linked list of variable size.

slide-14
SLIDE 14

A Demonstration Worth Thousand Words

Topology Aware Performance Monitoring August 24, 2015- 10 PU0 PU1 PU2 PU3 L1 L2 L3 ∗4 . . .

Accesses to a linked list of variable size.

slide-15
SLIDE 15

A Demonstration Worth Thousand Words

Topology Aware Performance Monitoring August 24, 2015- 10 PU0 PU1 PU2 PU3 L1 L2 L3 ∗4 . . .

Accesses to a linked list of variable size.

slide-16
SLIDE 16

A Demonstration Worth Thousand Words

Topology Aware Performance Monitoring August 24, 2015- 11

L3_MISS{ OBJ = L3; CTR = PAPI_L3_TCM; LOGSCALE = 1; } L2_MISS{ OBJ = L2; CTR = PAPI_L2_TCM; LOGSCALE = 1; } L1_MISS{ OBJ = L1d; CTR = PAPI_L1_DCM; LOGSCALE = 1; } SINGLE_L3_MISS{ OBJ = PU; CTR = PAPI_L3_TCM; LOGSCALE = 1; }

slide-17
SLIDE 17

Dynamic Lstopo (Usage)

Topology Aware Performance Monitoring August 24, 2015- 12

Machine (16GB total) NUMANode P#0 (16GB) Package P#0 L3 (4096KB) L2 (256KB) L1d (32KB) L1i (32KB) Core P#0 PU P#0 PU P#1 L2 (256KB) L1d (32KB) L1i (32KB) Core P#1 PU P#2 PU P#3 2,5120000000e+03 L2 (256KB) L1d (32KB) L1i (32KB) Core P#1 PU P#2 PU P#3 L2 (256KB) L1d (32KB) L1i (32KB) Core P#0 PU P#0 PU P#1 L1d (32KB) L1i (32KB) Core P#1 PU P#2 PU P#3 L1d (32KB) L1i (32KB) Core P#0 PU P#0 PU P#1 L1i (32KB) Core P#1 PU P#2 PU P#3 L1i (32KB) Core P#0 PU P#0 PU P#1 Core P#1 PU P#2 PU P#3 Core P#0 PU P#0 PU P#1 PU P#3 PU P#2 PU P#1 PU P#0

Counters input:

SINGLE L3 MISS{ OBJ = L3 ; CTR = PAPI L2 TCM/PAPI L2 TCA ; LOGSCALE = 1; MAX=1000000; MIN=0; }

Command line: lstopo –perf-input counters input

slide-18
SLIDE 18

Dynamic Lstopo (Theory)

Topology Aware Performance Monitoring August 24, 2015- 13 L2 L1 PU + PU + L1 PU + PU

  • 1. Spawn one pthread per

hardware thread (PU#0, . . . , PU#3).

slide-19
SLIDE 19

Dynamic Lstopo (Theory)

Topology Aware Performance Monitoring August 24, 2015- 13 L2 L1 PU + PU + L1 PU + PU

  • 1. Spawn one pthread per

hardware thread (PU#0, . . . , PU#3).

  • 2. For each timestamp, each

thread collects a local set of performance counters.

slide-20
SLIDE 20

Dynamic Lstopo (Theory)

Topology Aware Performance Monitoring August 24, 2015- 13 L2 L1 PU + PU + L1 PU + PU

  • 1. Spawn one pthread per

hardware thread (PU#0, . . . , PU#3).

  • 2. For each timestamp, each

thread collects a local set of performance counters.

  • 3. Counters are accumulated in

each upper level.

slide-21
SLIDE 21

Dynamic Lstopo (Theory)

Topology Aware Performance Monitoring August 24, 2015- 13 L2 L1 PU + PU + L1 PU + PU % % % % % % %

  • 1. Spawn one pthread per

hardware thread (PU#0, . . . , PU#3).

  • 2. For each timestamp, each

thread collects a local set of performance counters.

  • 3. Counters are accumulated in

each upper level.

  • 4. For each level, a leaf computes

an arithmetic expression of the performance counters in the set.

slide-22
SLIDE 22

Dynamic Lstopo Software Architecture in Brief

Topology Aware Performance Monitoring August 24, 2015- 14

Machine static topology Machine dynamic performance counters hwloc library PAPI library monitors library Application Monitors output lstopo utility

slide-23
SLIDE 23

Dynamic Lstopo Software Architecture in Brief

Topology Aware Performance Monitoring August 24, 2015- 14

Machine static topology Machine dynamic performance counters hwloc library PAPI library monitors library Application Monitors output lstopo utility

slide-24
SLIDE 24

API

monitors = l o a d M o n i t o r s f r o m c o n f i g (NULL, ” m y p e r f f i l e ” , ” m y o u t p u t f i l e ” , 0) ; Monitors watch pid ( monitors , getpid () ) ; M o n i t o r s s t a r t ( monitors ) ; /∗ . . . ∗/ Monitors update counters ( monitors ) ; d e l e t e M o n i t o r s ( monitors ) ;

Topology Aware Performance Monitoring August 24, 2015- 15

slide-25
SLIDE 25

Dynamic Lstopo (Output paje trace)

Topology Aware Performance Monitoring August 24, 2015- 16

%EventDef val 0 % Id int % Phase int % Time_us date % Value double %EndEventDef %EventDef container 1 % Id int % Level string % Sibling int % Name string % Logscale int %EndEventDef 1 3 PU 0 SINGLE_L3_MISS 1 1 2 PU 1 SINGLE_L3_MISS 1 1 1 PU 2 SINGLE_L3_MISS 1 1 0 PU 3 SINGLE_L3_MISS 1 0 2 0 962832762224 67 ,00 0 1 0 962832762224 58 ,00 0 3 0 962832762225 77 ,00 0 0 0 962832762236 64 ,00 0 3 0 962832860676 94514 ,00 0 0 0 962832860676 121746 ,00 0 2 0 962832860676 205170 ,00 0 1 0 962832860717 200931 ,00

slide-26
SLIDE 26

Features

  • Record and/or Display live machine performance counters and

match them with topology.

  • Several settings: counters accumulation, sampling rate, attach

to a process. . .

  • Replay any trace with a topology file (for external display)..
  • Sample specific parts of an application with the monitor

library.

  • Support legacy lstopo options (restrict topology, change

display format. . . ).

Topology Aware Performance Monitoring August 24, 2015- 17

slide-27
SLIDE 27

Future works

  • Match code and performance informations
  • Accept user defined aggregation operator.
  • Provide performance abstraction layer.
  • Be able to delimit phases during execution.
  • Find and give explicit hints on bottlenecks.

Topology Aware Performance Monitoring August 24, 2015- 18

slide-28
SLIDE 28

Conclusion

Data locality becomes a main criterion for high performance. We built a tool based on a topology model and a performance library to help taking up the challenge. It maps performance values to machine objects. It is a visual tool, fast and easy to use. It is lightweight and causes less than 1% CPU overhead. Let you build topology aware performance models. Dynamic lstopo is into the process of beeing merged with hwloc project.

Topology Aware Performance Monitoring August 24, 2015- 19

slide-29
SLIDE 29

Thank you

Topology Aware Performance Monitoring August 24, 2015- 20

Now available from https://github.com/NicolasDenoyelle/dynamic lstopo