Score-P A Joint Perform ance Measurem ent Run-Tim e I nfrastructure - - PowerPoint PPT Presentation

▶
score p a joint perform ance measurem ent run tim e i
SMART_READER_LITE
LIVE PREVIEW

Score-P A Joint Perform ance Measurem ent Run-Tim e I nfrastructure - - PowerPoint PPT Presentation

Oct 17, 2023 270 likes •494 views

VIRTUAL INSTITUTE HIGH PRODUCTIVITY SUPERCOMPUTING Score-P A Joint Perform ance Measurem ent Run-Tim e I nfrastructure VI-HPS Team VIRTUAL INSTITUTE HIGH PRODUCTIVITY SUPERCOMPUTING Score-P Infrastructure for instrumentation and

slide-1
SLIDE 1

VIRTUAL INSTITUTE – HIGH PRODUCTIVITY SUPERCOMPUTING

Score-P – A Joint Perform ance Measurem ent Run-Tim e I nfrastructure

VI-HPS Team

slide-2
SLIDE 2

VIRTUAL INSTITUTE – HIGH PRODUCTIVITY SUPERCOMPUTING

Score-P

  • Infrastructure for instrumentation and performance measurements
  • Instrumented application can be used to produce several results:
  • Call-path profiling:

CUBE4 data format used for data exchange

  • Event-based tracing:

OTF2 data format used for data exchange

  • Online profiling:

In conjunction with the Periscope Tuning Framework

  • Supported parallel paradigms:
  • Multi-process:

MPI, SHMEM

  • Thread-parallel:

OpenMP , Pthreads

  • Accelerator-based:

CUDA, OpenCL

  • Open Source; portable and scalable to all major HPC systems
  • Initial project funded by BMBF
  • Close collaboration with PRIMA project funded by DOE

PERFORMANCE ENGINEERING WITH SCORE-P AND VAMPIR, PASSAU, SEPTEMBER 15, 2015 2

slide-3
SLIDE 3

VIRTUAL INSTITUTE – HIGH PRODUCTIVITY SUPERCOMPUTING

Architecture overview

Application

Vampir Scalasca Periscope TAU

Accelerator-based parallelism (CUDA, OpenCL)

Score-P measurement infrastructure

Event traces (OTF2)

User instrumentation

Call-path profiles (CUBE4, TAU) Online interface Hardware counter (PAPI, rusage)

Process-level parallelism (MPI, SHMEM) Thread-level parallelism (OpenMP, Pthreads)

Instrumentation wrapper

Source code instrumentation

CUBE TAUdb

3 PERFORMANCE ENGINEERING WITH SCORE-P AND VAMPIR, PASSAU, SEPTEMBER 15, 2015

slide-4
SLIDE 4

VIRTUAL INSTITUTE – HIGH PRODUCTIVITY SUPERCOMPUTING

Partners

  • Forschungszentrum Jülich, Germany
  • German Research School for Simulation Sciences, Aachen, Germany
  • Gesellschaft für numerische Simulation mbH Braunschweig, Germany
  • RWTH Aachen, Germany
  • Technische Universität Darmstadt, Germany
  • Technische Universität Dresden, Germany
  • Technische Universität München, Germany
  • University of Oregon, Eugene, USA

4 PERFORMANCE ENGINEERING WITH SCORE-P AND VAMPIR, PASSAU, SEPTEMBER 15, 2015

slide-5
SLIDE 5

VIRTUAL INSTITUTE – HIGH PRODUCTIVITY SUPERCOMPUTING

Hands-on: NPB-MZ-MPI / BT

slide-6
SLIDE 6

VIRTUAL INSTITUTE – HIGH PRODUCTIVITY SUPERCOMPUTING

Perform ance analysis steps

  • Reference preparation for validation
  • Program instrumentation
  • Summary measurement collection
  • Summary experiment scoring
  • Summary measurement collection with filtering
  • Summary analysis report examination
  • Event trace collection
  • Event trace examination & analysis

PERFORMANCE ENGINEERING WITH SCORE-P AND VAMPIR, PASSAU, SEPTEMBER 15, 2015 6

slide-7
SLIDE 7

VIRTUAL INSTITUTE – HIGH PRODUCTIVITY SUPERCOMPUTING

NPB-MZ-MPI / BT instrum entation

  • Start in the tutorial

directory again and clean- up the build

7

% cd .. % make clean

PERFORMANCE ENGINEERING WITH SCORE-P AND VAMPIR, PASSAU, SEPTEMBER 15, 2015

slide-8
SLIDE 8

VIRTUAL INSTITUTE – HIGH PRODUCTIVITY SUPERCOMPUTING

NPB-MZ-MPI / BT instrum entation

  • Edit config/ make.def to

adjust build configuration

  • Modify specification of

compiler/ linker: MPIF77

8

# SITE- AND/OR PLATFORM-SPECIFIC DEFINITIONS #--------------------------------------------------------------- # Items in this file may need to be changed for each platform. #--------------------------------------------------------------- COMPFLAGS = -fopenmp ... #--------------------------------------------------------------- # The Fortran compiler used for MPI programs #--------------------------------------------------------------- #MPIF77 = mpif77 # Score-P variant to perform instrumentation ... MPIF77 = scorep mpif77 # This links MPI Fortran programs; usually the same as ${MPIF77} FLINK = $(MPIF77) ...

Uncomment the Score-P compiler wrapper specification

PERFORMANCE ENGINEERING WITH SCORE-P AND VAMPIR, PASSAU, SEPTEMBER 15, 2015

slide-9
SLIDE 9

VIRTUAL INSTITUTE – HIGH PRODUCTIVITY SUPERCOMPUTING

NPB-MZ-MPI / BT instrum ented build

  • Return to root directory

and clean-up

  • Re-build executable using

Score-P compiler wrapper

9 PERFORMANCE ENGINEERING WITH SCORE-P AND VAMPIR, PASSAU, SEPTEMBER 15, 2015

% make bt-mz CLASS=W NPROCS=4 cd BT-MZ; make CLASS=W NPROCS=4 VERSION= make: Entering directory 'BT-MZ' cd ../sys; cc -o setparams setparams.c -lm ../sys/setparams bt-mz 4 W mpif77 -c -O3 -fopenmp bt.f [...] cd ../common; scorep mpif77 -c -O3 -fopenmp timers.f scorep mpif77 –O3 -fopenmp -o ../bin.scorep/bt-mz_W.4 \ bt.o initialize.o exact_solution.o exact_rhs.o set_constants.o \ adi.o rhs.o zone_setup.o x_solve.o y_solve.o exch_qbc.o \ solve_subs.o z_solve.o add.o error.o verify.o mpi_setup.o \ ../common/print_results.o ../common/timers.o Built executable ../bin.scorep/bt-mz_W.4 make: Leaving directory 'BT-MZ‘

slide-10
SLIDE 10

VIRTUAL INSTITUTE – HIGH PRODUCTIVITY SUPERCOMPUTING

Measurem ent configuration: scorep-info

  • Score-P measurements

are configured via environmental variables

10

% scorep-info config-vars --full SCOREP_ENABLE_PROFILING Description: Enable profiling [...] SCOREP_ENABLE_TRACING Description: Enable tracing [...] SCOREP_TOTAL_MEMORY Description: Total memory in bytes for the measurement system [...] SCOREP_EXPERIMENT_DIRECTORY Description: Name of the experiment directory [...] SCOREP_FILTERING_FILE Description: A file name which contain the filter rules [...] SCOREP_METRIC_PAPI Description: PAPI metric names to measure [...] SCOREP_METRIC_RUSAGE Description: Resource usage metric names to measure [... More configuration variables ...]

PERFORMANCE ENGINEERING WITH SCORE-P AND VAMPIR, PASSAU, SEPTEMBER 15, 2015

slide-11
SLIDE 11

VIRTUAL INSTITUTE – HIGH PRODUCTIVITY SUPERCOMPUTING

NPB-MZ-MPI / BT sum m ary m easurem ent collection

  • Change to the directory

containing the new executable before running it with the desired configuration

  • Run instrumented

application

11 PERFORMANCE ENGINEERING WITH SCORE-P AND VAMPIR, PASSAU, SEPTEMBER 15, 2015

% cd bin.scorep % export SCOREP_EXPERIMENT_DIRECTORY=scorep_bt-mz_W_4x4_sum % OMP_NUM_THREADS=4 mpirun -np 4 ./bt-mz_W.4 NAS Parallel Benchmarks (NPB3.3-MZ-MPI) - BT-MZ MPI+OpenMP \ >Benchmark Number of zones: 4 x 4 Iterations: 200 dt: 0.000800 Number of active processes: 4 Use the default load factors with threads Total number of threads: 16 ( 4.0 threads/process) Calculated speedup = 15.78 Time step 1 [... More application output ...] BT-MZ Benchmark Completed. Time in seconds = 100.41

slide-12
SLIDE 12

VIRTUAL INSTITUTE – HIGH PRODUCTIVITY SUPERCOMPUTING

NPB-MZ-MPI / BT sum m ary analysis report exam ination

  • Creates experiment

directory including

  • A record of the measurement

configuration (scorep.cfg)

  • The analysis report that was

collated after measurement (profile.cubex)

12

% ls bt-mz_W.4 scorep_bt-mz_W_4x4_sum % ls scorep_bt-mz_W_4x4_sum profile.cubex scorep.cfg

PERFORMANCE ENGINEERING WITH SCORE-P AND VAMPIR, PASSAU, SEPTEMBER 15, 2015

slide-13
SLIDE 13

VIRTUAL INSTITUTE – HIGH PRODUCTIVITY SUPERCOMPUTING

Congratulations!?

  • If you made it this far, you successfully used Score-P to
  • instrument the application
  • analyze its execution with a summary measurement, and
  • examine it with one the interactive analysis report explorer GUIs
  • ... revealing the call-path profile annotated with
  • the “Time” metric
  • Visit counts
  • MPI message statistics (bytes sent/ received)
  • ... but how good was the measurement?
  • The measured execution produced the desired valid result
  • however, the execution took rather longer than expected!
  • even when ignoring measurement start-up/ completion, therefore
  • it was probably dilated by instrumentation/ measurement overhead

13 PERFORMANCE ENGINEERING WITH SCORE-P AND VAMPIR, PASSAU, SEPTEMBER 15, 2015

slide-14
SLIDE 14

VIRTUAL INSTITUTE – HIGH PRODUCTIVITY SUPERCOMPUTING

Perform ance analysis steps

  • Reference preparation for validation
  • Program instrumentation
  • Summary measurement collection
  • Summary experiment scoring
  • Summary measurement collection with filtering
  • Summary analysis report examination
  • Event trace collection
  • Event trace examination & analysis

PERFORMANCE ENGINEERING WITH SCORE-P AND VAMPIR, PASSAU, SEPTEMBER 15, 2015 14

slide-15
SLIDE 15

VIRTUAL INSTITUTE – HIGH PRODUCTIVITY SUPERCOMPUTING

NPB-MZ-MPI / BT sum m ary analysis result scoring

  • Report scoring as textual
  • utput
  • Region/ callpath classification
  • MPI pure MPI functions
  • OMP pure OpenMP regions
  • USR user-level computation
  • COM “combined” USR+ OpenMP/ MPI
  • ANY/ ALL aggregate of all region

types

% scorep-score scorep_bt-mz_W_4x4_sum/profile.cubex

Estimated aggregate size of event trace: 1025MB Estimated requirements for largest trace buffer (max_buf): 265MB Estimated memory requirements (SCOREP_TOTAL_MEMORY): 273MB (hint: When tracing set SCOREP_TOTAL_MEMORY=273MB to avoid intermediate flushes

  • r reduce requirements using USR regions filters.)

flt type max_buf[B] visits time[s] time[%] time/visit[us] region ALL 277,799,918 41,157,533 1284.51 100.0 31.21 ALL USR 274,792,492 40,418,321 286.86 22.3 7.10 USR OMP 6,882,860 685,952 862.00 67.1 1256.64 OMP COM 371,956 45,944 112.21 8.7 2442.29 COM MPI 102,286 7,316 23.44 1.8 3204.09 MPI

1 GB total memory 265 MB per rank!

15

USR USR COM COM USR OMP MPI

PERFORMANCE ENGINEERING WITH SCORE-P AND VAMPIR, PASSAU, SEPTEMBER 15, 2015

slide-16
SLIDE 16

VIRTUAL INSTITUTE – HIGH PRODUCTIVITY SUPERCOMPUTING

NPB-MZ-MPI / BT sum m ary analysis report breakdow n

16

% scorep-score –r scorep_bt-mz_W_4x4_sum/profile.cubex

[...] [...] flt type max_buf[B] visits time[s] time[%] time/visit[us] region ALL 277,799,918 41,157,533 1284.51 100.0 31.21 ALL USR 274,792,492 40,418,321 286.86 22.3 7.10 USR OMP 6,882,860 685,952 862.00 67.1 1256.64 OMP COM 371,956 45,944 112.21 8.7 2442.29 COM MPI 102,286 7,316 23.44 1.8 3204.09 MPI USR 85,774,338 12,516,672 88.69 6.9 7.09 matmul_sub USR 85,774,338 12,516,672 91.14 7.1 7.28 binvcrhs USR 85,774,338 12,516,672 86.03 6.7 6.87 matvec_sub USR 7,974,876 1,170,624 7.58 0.6 6.48 lhsinit USR 7,974,876 1,170,624 7.76 0.6 6.63 binvrhs USR 3,473,912 526,848 5.65 0.4 10.73 exact_solution [...]

USR USR COM COM USR OMP MPI More than 270 MB just for these 6 regions

PERFORMANCE ENGINEERING WITH SCORE-P AND VAMPIR, PASSAU, SEPTEMBER 15, 2015

slide-17
SLIDE 17

VIRTUAL INSTITUTE – HIGH PRODUCTIVITY SUPERCOMPUTING

NPB-MZ-MPI / BT sum m ary analysis score

  • Summary measurement analysis score reveals
  • Total size of event trace would be ~ 1025 MB
  • Maximum trace buffer size would be ~ 265 MB per rank
  • smaller buffer would require flushes to disk during measurement resulting in substantial perturbation
  • 99.8% of the trace requirements are for USR regions
  • purely computational routines never found on COM call-paths common to communication routines or OpenMP parallel

regions

  • These USR regions contribute around 22% of total time
  • however, much of that is very likely to be measurement overhead for frequently-executed small routines
  • Advisable to tune measurement configuration
  • Specify an adequate trace buffer size
  • Specify a filter file listing (USR) regions not to be measured

17 PERFORMANCE ENGINEERING WITH SCORE-P AND VAMPIR, PASSAU, SEPTEMBER 15, 2015

slide-18
SLIDE 18

VIRTUAL INSTITUTE – HIGH PRODUCTIVITY SUPERCOMPUTING

NPB-MZ-MPI / BT sum m ary analysis report filtering

  • Report scoring with

prospective filter listing 6 USR regions

18

% cat ../config/scorep.filt SCOREP_REGION_NAMES_BEGIN EXCLUDE binvcrhs* matmul_sub* matvec_sub* exact_solution* binvrhs* lhs*init* timer_* % scorep-score -f ../config/scorep.filt \ >scorep_bt-mz_W_4x4_sum/profile.cubex

Estimated aggregate size of event trace: 23MB Estimated requirements for largest trace buffer (max_buf): 8MB Estimated memory requirements (SCOREP_TOTAL_MEMORY): 16MB (hint: When tracing set SCOREP_TOTAL_MEMORY=16MB to avoid intermediate flushes

  • r reduce requirements using USR regions filters.)

23 MB of memory in total, 8 MB per rank!

PERFORMANCE ENGINEERING WITH SCORE-P AND VAMPIR, PASSAU, SEPTEMBER 15, 2015

slide-19
SLIDE 19

VIRTUAL INSTITUTE – HIGH PRODUCTIVITY SUPERCOMPUTING

NPB-MZ-MPI / BT sum m ary analysis report filtering

  • Score report breakdown

by region

19

% scorep-score -r –f ../config/scorep.filt \ > scorep_bt-mz_W_8x4_sum/profile.cubex

flt type max_buf[B] visits time[s] time[%] time/visit[us] region

  • ALL 277,799,918 41,157,533 1284.51 100.0 31.21 ALL
  • USR 274,792,492 40,418,321 286.86 22.3 7.10 USR
  • OMP 6,882,860 685,952 862.00 67.1 1256.64 OMP
  • COM 371,956 45,944 112.21 8.7 2442.29 COM
  • MPI 102,286 7,316 23.44 1.8 3204.09 MPI

* ALL 7,357,804 739,321 1284.51 100.0 31.21 ALL-FLT + FLT 274,791,764 40,418,212 286.86 22.3 7.10 FLT

  • OMP 6,882,860 685,952 862.00 67.1 1256.64 OMP-FLT

* COM 371,956 45,944 112.21 8.7 2442.29 COM-FLT

  • MPI 102,286 7,316 23.44 1.8 3204.09 MPI-FLT

* USR 728 109 0.00 0.0 18.68 MPI-FLT [...]

Filtered routines marked with ‘+ ’

PERFORMANCE ENGINEERING WITH SCORE-P AND VAMPIR, PASSAU, SEPTEMBER 15, 2015

slide-20
SLIDE 20

VIRTUAL INSTITUTE – HIGH PRODUCTIVITY SUPERCOMPUTING

NPB-MZ-MPI / BT filtered sum m ary m easurem ent collection

  • Set new experiment

directory and re-run measurement with new filter configuration

20 PERFORMANCE ENGINEERING WITH SCORE-P AND VAMPIR, PASSAU, SEPTEMBER 15, 2015

% export SCOREP_EXPERIMENT_DIRECTORY=\ >scorep_bt-mz_W_4x4_sum_filtered % export SCOREP_FILTERING_FILE=../config/scorep.filt % OMP_NUM_THREADS=4 mpirun -np 4 ./bt-mz_W.4 NAS Parallel Benchmarks (NPB3.3-MZ-MPI) - BT-MZ MPI+OpenMP \ >Benchmark Number of zones: 4 x 4 Iterations: 200 dt: 0.000800 Number of active processes: 4 [... More application output ...] BT-MZ Benchmark Completed. Time in seconds = 6.90