Debugger Usage at HRSK 2009-03-19 Zellescher Weg 12 Willers-Bau - - PowerPoint PPT Presentation

Zellescher Weg 12 Willers-Bau A106

• Tel. +49 351 - 463 - 31945

Matthias Lieber (Matthias.Lieber@tu-dresden.de)

Center for Information Services and High Performance Computing (ZIH)

Debugger Usage at HRSK

2009-03-19

Why using a Debugger?

Your program… – terminates abnormally – produces wrong results – shows incomprehensible behavior You want to know what your program is (really) doing Typical example: your program crashes with a segmentation fault % gcc myprog.c –o myprog % ./myprog Segmentation fault % What’s going wrong?

What can a Debugger do?

Observe a running program: – Print variables (scalars, arrays, structures / derived types, classes) – Inform about current source code line and function (function call stack) Control the program execution: – Stop the program at a specific source code line (breakpoint) – Stop the program by evaluating variable expressions (watchpoint) – Stop the program before terminating abnormally – Execute the program line-by-line (stepping)

Function call stack

program main ... call print_date ... end program main subroutine print_date ... call date_and_time ... end subroutine print_date #0: main #1: print_date #2: date_and_time subroutine date_and_time ...

calls calls program is here Function call stack: information of all active functions of a process which have been called but not yet finished. Stack:

Typical Usage of a Debugger

Compile the program with the –g compiler flag – gcc –g myprog.c –o myprog Run the program under control of the debugger: – ddt ./myprog – Locate the position of the problem and examine variables – Find out why the program shows unexpected behavior Edit the source code, modify parameters, etc. Repeat until problem is solved

Debugging Tools at HRSK

vendor / name command graphical user interface parallel support Mars Deimos GNU Debugger gdb Threads

• GNU Data Display

Debugger ddd

• Threads
• Allinea

Distributed Debugging Tool ddt

• Threads,

MPI

• Totalview

totalview

• Threads,

MPI

• Valgrind

valgrind Threads

Example: Allinea DDT (Distributed Debugging Tool)

Process and thread selection Source file browser Variables pane Evaluation window Process control: run, stop, stepping Output, Breakpoints, Watchpoint, Call stack Help Button Source view

Example: Totalview

Example: GNU Debugger (gdb)

Basic GDB commands

GDB Reference Sheet: http://users.ece.utexas.edu/~adnan/gdb-refcard.pdf h, help (<command>) Help (on <command>) r, run <options> Run program with options bt, backtrace Print program call stack l, list (<line>) Print source code (at <line>) p, print <expression> Display value of expression (variables) s, step / n, next Step to next line (n ... over func. calls) up / down Navigate in the program call stack b, break <file:line> Set breakpoint info break List breakpoints

Example: GNU DDD (GUI for gdb)

Allinea DDT

Commercial debugger from Allinea DDT - Distributed Debugging Tool Support for parallel applications (Threads, OpenMP, MPI, and hybrid) Supports C, C++, Fortran Internally utilizes the GNU debugger (or other) Memory debugging and other advanced features Intuitive graphical user interface More Information: http://www.allinea.com

DDT Usage at HRSK – 3 methods to start DDT

On login CPUs (not recommended) – works for short runs only (10 minutes runtime limit) As batch job – start your program like normal batch job, but under control of DDT – closing DDT finishes batch job – new job necessary for next DDT run From interactive batch session (my recommendation) – start a shell on compute CPUs and start program from there – very useful if you plan to do multiple debugging sessions: consecutive program runs in one batch job – in the interactive batch session you can work like on the login CPUs – but with your own private CPUs

DDT Usage at HRSK – serial programs

Compile your code with the –g compiler flag (common across all compilers) icc –g –O0 myprog.c –o myprog Set the environment for DDT module load ddt Start the program under control of DDT – on the login CPUs (short runs only): ddt ./myprog – as batch job: bsub –W 2:00 –n 1 ddt ./myprog – from interactive batch session: bsub –W 2:00 –n 1 –Is bash ddt ./myprog

Example: Debugging with DDT on Mars

Example: Debugging with DDT on Mars

Compiling myproc.c Loading module for DDT Starting interactive batch session Starting DDT to debug myprog

Example: Debugging with DDT on Mars

Run and Debug a Program

Example: Debugging with DDT on Mars

Run

Example: Debugging with DDT on Mars

Hit “Play” Button to start program

Example: Debugging with DDT on Mars

Segmentation Fault! Hit “Pause” to stop the program

Example: Debugging with DDT on Mars

This is the line where the program crashed

Example: Debugging with DDT on Mars

i = 291016 No the expected value!

Example: Debugging with DDT on Mars

Restart the debugging session to find out what happens to “i”

Example: Debugging with DDT on Mars

Example: Debugging with DDT on Mars

Adding a Breakpoint just before the while loop

Example: Debugging with DDT on Mars

run the program Breakpoint is shown by red dot

Example: Debugging with DDT on Mars

Hit “Pause” to examine variables

Example: Debugging with DDT on Mars

Open the “Locals” tab to see all local variables

Example: Debugging with DDT on Mars

i = 291016 before the loop starts! Forgot initialization?! Use the “step” Button to step line-by- line through the program

Example: Debugging with DDT on Mars

Use the “step” Button to step line-by- line through the program

Example: Debugging with DDT on Mars

Use the “step” Button to step line-by- line through the program

Example: Debugging with DDT on Mars

DDT Usage at HRSK – MPI parallel programs

Mars (MPI Implementation: altix-mpi) Deimos (MPI Implementation: OpenMPI)

module load ddt ddt ./myprog

• module load ddt

bsub –W 2:00 –n8 ddt ./myprog module load ddt bsub –W 2:00 –n8 –Is –aopenmpi ddt ./myprog module load ddt bsub –W 2:00 –n8 –Is bash ddt ./myprog module load ddt bsub –W 2:00 –n8 –Is –aopenmpi bash ddt ./myprog

Login CPUs Batch Job Interactive batch session

Other Debugging Options (1)

Attach to a running program Useful if program shows incomprehensive behavior – Hangs – Wrong results But program was not started under debugger control Core file debugging Core dumps are memory and register state of a crashed program written to disk These files can be analyzed after program run with a debugger

Other Debugging Options (2)

Memory debugging with Valgrind Detects memory management bugs – Use of uninitialized

• r free’d

memory – Access memory out off allocated bounds – etc. Deimos

• nly

Core file debugging

Core dumps are memory and register state of a crashed program written to disk These files can be analyzed after program run with a debugger Compile with –g compiler flag: ifort -g myprog.f90 -o myprog (Fortran) icc -g myprog.c -o myprog (C) Check current core dump limit (reports kB): ulimit –c If necessary, set the limit for core dumps: ulimit –c 100000 (set limit to 100MB) Intel Fortran requires this environment variable to be set before running program: export decfort_dump_flag=yes (Intel Fortran only) Start the program without debugger: ./myprog When the program crashes, core dump(s) [HOSTNAME].[PID].core will be created Analyze the core dump with DDT: – Run ddt ./myprog and select the core dump via “Core file…” button

Core file debugging

Core dump of myprog

Core file debugging

Open a Core File

Core file debugging

Choose core file

Core file debugging

Memory Debugging with Valgrind (Deimos only)

Detects memory management bugs: – Use of uninitialized memory – Access free’d memory – Access memory out off allocated bounds – Access inappropriate areas on the stack – Memory leaks – when pointers to allocated areas get lost forever – Mismatched use of malloc and free (C, Fortran), new and delete (C++) – Wrong use of memcpy() and related functions Available on Deimos via module load valgrind Simply run program under valgrind: valgrind ./myprog More Information: http://www.valgrind.org

Memory Debugging with Valgrind (Deimos only)

The buggy program from the DDT example works on Deimos with gcc Let’s see if valgrind finds the bug…

Memory Debugging with Valgrind (Deimos only)

Loading valgrind module and starting program under valgrind Valgrind reports the lines where we access uninitialized i

How to avoid Debuggers

Write portable programs, i.e. don’t make assumptions on the compiler, MPI library, or any other software and hardware used In other words: Adhere to standards! Test your code on different platforms to find out if you have portability problems Use Valgrind to verify memory usage Comment your code and write readable code Also check prerequisites in the code, e.g.: assert(i>=0 && i<N); a[i] = x; Use compiler’s check capabilities (see next slide) – Read compiler’s manual

Getting more information from the Intel Compiler

Common flags: – -g (produce debug information) – -O0 (disable optimization) Intel C Compiler: – Compile-time information:

• Wall (enable almost all warnings)
• Wp64 (diagnostics for 64-bit porting)
• Wuninitialized

(check for uninitialized variables – not reliable!)

• strict-ansi

(strict ANSI C/C++) Intel Fortran Compiler: – Compile-time information:

• warn all (enable all warnings)
• std90 / -std95 (strict Fortran 90/95)

– Run-time information:

• traceback

(call stack traceback when severe error occurs)

• C (run-time checking, e.g. array bounds, uninitialized

variables)

• fpe0 (abort on floating point exceptions, e.g. division by zero,
• verflow)

Remarks on Compiler Flags

Use compiler’s check capabilities like -Wall etc. – Read compiler’s manual: man {gcc|ifort|pathcc|pgf90|...} – Intel C Compiler: -Wall -Wp64 -Wuninitialized -strict-ansi – Intel Fortran Compiler: -warn all -std95 -C -fpe0 -traceback Always compile your application with the –g flag, especially during developing and testing Optimizations often interfere with debugging (e.g. functions or variables of interest are “optimized away”) – If necessary, compile with –O0 to disable optimizations

Links

Slides of the user training (with more details!): http://www.tu-dresden.de/zih/hpc/support/usertraining Allinea DDT User Guide: http://www.allinea.com/downloads/userguide.pdf Totalview: http://www.totalviewtech.com/products/totalview.html GDB Website: http://www.gnu.org/software/gdb/ DDD Website: http://www.gnu.org/software/ddd/ Valgrind: http://www.valgrind.org