malt malloc tracker
play

MALT : MALloc Tracker A memory profiling tool 3/02/2019 MALT, - PowerPoint PPT Presentation

Sep 10, 2023 •355 likes •762 views

MALT : MALloc Tracker A memory profiling tool 3/02/2019 MALT, Sbastien Valat 1 Questions We have good profiling tool for timings (eg. Valgrind or vtune) But for what memory profiling ? Memory can be an issue : Availability of

  1. MALT : MALloc Tracker A memory profiling tool 3/02/2019 MALT, Sébastien Valat 1

  2. Questions • We have good profiling tool for timings (eg. Valgrind or vtune) • But for what memory profiling ? • Memory can be an issue : – Availability of the resource – Performance • Three main questions : – How to reduce memory footprint ? – How to improve overhead of memory management ? – How to improve memory usage ? 3/02/2019 MALT, Sébastien Valat 2

  3. Some issue examples • I wanted to point : – Where memory is allocated. – Properties of allocated chunks. – Bad allocation patterns for performance. Global variables and TLS Indirect allocations __thread Int gblVar[SIZE]; int * func(int size) { Leak child_func_with_allocs(); void * ptr = new char[size]; double* ret = new double[size*size*size]; for (auto it : iter_Items) Might lead to swap for large size { double* buffer = new double[size]; C++11 auto induced allocs //short and quick do stuff delete [] buffer; } Short life allocations return ret; 3/02/2019 3 } MALT, Sébastien Valat

  4. What I want to provide • Same approach than valgrind/kcachgind • Mapped allocations on sources lines and call stacks • Using a web-based GUI – I started with kcachgrind – But wanted more flexibility and time charts 3/02/2019 MALT, Sébastien Valat 4

  5. How it works • Use LD_PRELOAD to intercept malloc /free/… as Google heap profiler • Map allocations on call stacks • Build & consolidate summary metrics • Generate JSON output file 3/02/2019 MALT, Sébastien Valat 5

  6. Source annotations Web technology ( NodeJS , D3JS , Jquery , AngularS ) Inclusive/Exclusive Metric selector Per line annotation Call stacks reaching the selected Symbols Details of symbol or line site. 3/02/2019 MALT, Sébastien Valat 6

  7. Call tree view 3/02/2019 MALT, Sébastien Valat 7

  8. Per thread statistics 3/02/2019 MALT, Sébastien Valat 8

  9. Fragmentation issue • Memory consumption over time – Physical – Virtual – Requested (malloced) 3/02/2019 MALT, Sébastien Valat 9

  10. Dynamics 3/02/2019 MALT, Sébastien Valat 10

  11. Example on AVBP init phase • Issue with reallocation on init • Detected with allocation rate & cumulated allocatated mem. Time 3/02/2019 MALT, Sébastien Valat 11

  12. Usage • Optionally recompile with debug flags : gcc -g … • Run malt [--config=file.ini] YOUR_PRGM [OPTIONS] • Use the web view && http://localhost:8080: malt-webview -i malt-{YOUR_PRGM}-{PID}.json • In case there is a QT wrapper embedding NodeJS + Webkit malt-qt -i malt-{YOUR_PRGM}-{PID}.json 3/02/2019 MALT, Sébastien Valat 12

  13. Status • Open sourced since one year on https://github.com/memtt • Co-hosted with a similar tool : NUMAPROF for Non Uniform Memory Access profiling. • My research on memory management for HPC : http://svalat.github.io/ 3/02/2019 MALT, Sébastien Valat 13

  14. Thank you. QUESTIONS ? 3/02/2019 MALT, Sébastien Valat 14

  15. BACKUP 3/02/2019 MALT, Sébastien Valat 15

  16. Possibly huge impact Execution time (s) 500 • Memory management 450 can have huge impact on 4x 400 performance 350 300 • Extreme case on a 1.5 250 million C++ lines HPC 200 150 simulation app. on a 16 100 processors server 50 0 • Can see 10-15% improvement on MySQL by changing allocator User System Idle 3/02/2019 MALT, Sébastien Valat 16

  17. Output, first idea, kcachegrind Callgrind compatibiltiy • Can use kcachgrind • Might be usefull for some users, cannot provide all metrics. 3/02/2019 MALT, Sébastien Valat 17

  18. What is missing to kcachegrind • Started with kcacegrind GUI…. But … • Display human readable units – You prefer 15728640 or 15 MB ? – I want to compare to what I expect . • Cannot handle non sum cumulative metrics – Inclusive costs only rely on + operator – Some mem. metrics requires max/min (eg. lifetime) • No way to express time charts • No way to express parameter distributions (eg. sizes). 3/02/2019 MALT, Sébastien Valat 18

  19. Ideas of improvement • Add NUMA statistics • Provide virtual/physical ratio • Estimate page fault costs • Exploit traces in GUI for deeper analysis – Alive allocations at a certain time – Fragmentation analysis – Time charts from call sites – Usage over threads for call sites 3/02/2019 MALT, Sébastien Valat 19

  20. Global summary • Show global program statistics 3/02/2019 MALT, Sébastien Valat 20

  21. Temporal metrics Profile over time :  ▪ Allocation rate ▪ Physical / Virtual / Requested memory ▪ Stack size for each thread (require function instrumentation) Example on YALES2 with gfortran :  3/02/2019 MALT, Sébastien Valat 21

  22. Chunk size distribution Example from YALES2 with gfortran issue Many really small allocations 3/02/2019 MALT, Sébastien Valat 22

  23. EXISTING TOOLS 3/02/2019 MALT, Sébastien Valat 23

  24. Existing tools • Valgrind (massif) – Memory over time (snapshots) & functions – Memory per function at peak – Has a simple GUI • Valgrind (memchek) – Leaks – No real GUI • Google heap profiler (tcmalloc) – Memory over time (snapshots) – Faster then valgrind – No GUI 3/02/2019 MALT, Sébastien Valat 24

  25. Existing tools / Google heap profiler • Google heap profiler (tcmalloc): – Small overhead. – Similar metric than massif – Only provide snapshots of allocated memory per stacks . – Peak might not be captured. – Lack of a real GUI to use it. % pprof gfs_master profile.0100.heap 255.6 24.7% 24.7% 255.6 24.7% GFS_MasterChunk::AddServer 184.6 17.8% 42.5% 298.8 28.8% GFS_MasterChunkTable::Create 176.2 17.0% 59.5% 729.9 70.5% GFS_MasterChunkTable::UpdateState 169.8 16.4% 75.9% 169.8 16.4% PendingClone::PendingClone 76.3 7.4% 83.3% 76.3 7.4% __default_alloc_template::_S_chunk_alloc 49.5 4.8% 88.0% 49.5 4.8% hashtable::resize 3/02/2019 MALT, Sébastien Valat 25

  26. Existing tools • TAU memory profiler – Provide profiles – Follow stacks – Track leaks – Parallel, done for HPC/MPI – Lack easy matching with sources • FOM 3/02/2019 MALT, Sébastien Valat 26

  27. Existing tools / Commercials • IBM Purify++ / Parasoft Insure++ – Commercial – Leak detection, access checking, memory debugging tools. – Use binary or source instrumentation. – Windows / Redhat • Visual Studio Ultimate Edition Memory profiler – Nice but windows only and commercial 3/02/2019 MALT, Sébastien Valat 27

  28. Stack tracking • Two approach implemented : backtrace and instrumentation • Backtrace (default) : – Work out of the box – Manage all dynamic libraries – Slow for large number of calls (~>10M) • Instrumentation : – Need source recompilation (available) : -finstrument-function – Or tools for binary instrumentation : MAQAO / Pintool (experimental) – Faster for really large number of calls to malloc – Only provide stacks for the instrumented binaries 3/02/2019 MALT, Sébastien Valat 28

  29. What is good in kcachgrind • List of functions with exclusive/inclusive costs • Nice call tree • Annotated sources 3/02/2019 MALT, Sébastien Valat 29

  30. SOME VIEWS 3/02/2019 MALT, Sébastien Valat 30

  31. Global summary • Provide a small summary • Provide some warnings 3/02/2019 MALT, Sébastien Valat 31

  32. Global summary : top 5 functions • Summarize top functions for some metrics • Points to check • Examples on YALES2 3/02/2019 MALT, Sébastien Valat 32

  33. Tracking stack memory Display largest stack for thread ID Stack space used by functions on peak Thread ID Stack size over time 3/02/2019 MALT, Sébastien Valat 33

  34. Chunk size distribution Example from YALES2 Many really small allocations 3/02/2019 MALT, Sébastien Valat 34

  35. Global variables 3/02/2019 MALT, Sébastien Valat 35

  36. REAL CASES 3/02/2019 MALT, Sébastien Valat 36

  37. Performance 100 90 80 70 valgrind-memcheck 60 50 valgrind-massif 40 gperf 30 igprof 20 malt 10 malt-finstr 0 3/02/2019 MALT, Sébastien Valat 37

  38. Allocatable arrays on YALES2 • Issue only occur with gfortran , ifort uses stack arrays. Search intensive alloc functions Huge number of allocation for a line programmer think it doesn’t do any ! And mostly really small allocations ! 3/02/2019 MALT, Sébastien Valat 38

  39. We can found allocs of 1B ! • Examples on YALES 2, small allocations : Search for the minimal chunk size. Many codes produce allocations of 1B. OK with moderation. 3/02/2019 MALT, Sébastien Valat 39

  40. Fragmentation issue • Example of fragmentation detection • Using the time chart with physical , virtual and requested memory • Solution : avoid interleaved allocation of chunks with different lifetime . • Looking on source annotation : most of them can be avoided . 3/02/2019 MALT, Sébastien Valat 40

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

1 2 3 4 5 6 7 $ sed -n 22,27p glibc/malloc/malloc.c This is a version (aka ptmalloc2) of

1 2 3 4 5 6 7 $ sed -n 22,27p glibc/malloc/malloc.c This is a version (aka ptmalloc2) of

1 2 3 4 5 6 7 $ sed -n 22,27p glibc/malloc/malloc.c This is a version (aka ptmalloc2) of malloc/free/realloc written by Doug Lea and adapted to multiple threads/arenas by Wolfram Gloger. There have been substantial changes made after the

734 views • 47 slides
CS 241 Data Organization malloc and free March 20, 2018 <stdlib.h>: malloc() void

CS 241 Data Organization malloc and free March 20, 2018 <stdlib.h>: malloc() void

CS 241 Data Organization malloc and free March 20, 2018 <stdlib.h>: malloc() void *malloc(size_t byteSize) The malloc function allocates unused space for an object whose size in bytes is specified by byteSize . The order and

377 views • 15 slides
Malt COA Bucket Analysis Approach Presenters Mike Heinrich Tyler Schoales Craft Malt Specialist

Malt COA Bucket Analysis Approach Presenters Mike Heinrich Tyler Schoales Craft Malt Specialist

Breakdown of a Malt COA Bucket Analysis Approach Presenters Mike Heinrich Tyler Schoales Craft Malt Specialist NA Craft Sales Manager Country Malt Group Great Western Malting Breakdown of a Malt COA Agenda Overview of Malting and

453 views • 22 slides
Malloc & VM Malloc & VM By sseshadr Agenda Agenda Administration d st at o

Malloc & VM Malloc & VM By sseshadr Agenda Agenda Administration d st at o

Malloc & VM Malloc & VM By sseshadr Agenda Agenda Administration d st at o Process lab code will be inked by Thursday (pick up in ECE hub) Malloc due soon (Thursday, November 4 th ) Exam 2 in a week (Tuesday, November 9

559 views • 18 slides
Caitlin Tegels Malt-O-Meal Improving Utility Efficiency Malt-O-Meal Caitlin Tegels Advisors:

Caitlin Tegels Malt-O-Meal Improving Utility Efficiency Malt-O-Meal Caitlin Tegels Advisors:

Caitlin Tegels Malt-O-Meal Improving Utility Efficiency Malt-O-Meal Caitlin Tegels Advisors: Paul Pagel, John Polanski Campbell Mill in Northfield, MN Ready-to-eat cereals ~800 employees at that location Motivations for Change

388 views • 18 slides
Crop 2018 Viking Malt Countries All Viking Malt countries have been together with other countries

Crop 2018 Viking Malt Countries All Viking Malt countries have been together with other countries

Malting Barley Crop 2018 Viking Malt Countries All Viking Malt countries have been together with other countries hidden av a severe drought under the summer 2018. The malting barley did the right thing and was setting fewer and shorter ears

501 views • 9 slides
Gif++ CR tracker setup G. Aielli for the RPC collaboration Overall Layout of the CR tracker

Gif++ CR tracker setup G. Aielli for the RPC collaboration Overall Layout of the CR tracker

Gif++ CR tracker setup G. Aielli for the RPC collaboration Overall Layout of the CR tracker Detailed layout: Top Tracker Trigger and high time resolution. 4 independent detectors area 1x0.5 m^2; Gap structure to be finalized strips 3

390 views • 8 slides
Tips of malloc & free Making your own malloc library for troubleshooting 2013.2.22 Embedded

Tips of malloc & free Making your own malloc library for troubleshooting 2013.2.22 Embedded

Tips of malloc & free Making your own malloc library for troubleshooting 2013.2.22 Embedded Linux Conference Tetsuyuki Kobayashi 1 The latest version of this slide will be available from here

648 views • 51 slides
Crop 2018 Viking Malt Countries All Viking Malt countries together with many other countries have

Crop 2018 Viking Malt Countries All Viking Malt countries together with many other countries have

Malting Barley Crop 2018 Viking Malt Countries All Viking Malt countries together with many other countries have been struggling with a severe drought under the summer 2018. The malting barley did the right thing and was setting fewer and

346 views • 9 slides
Hardrock GPS Satellite Tracker Introduction 1. Camp Hardrock Tracker Orientation Schedule 2.

Hardrock GPS Satellite Tracker Introduction 1. Camp Hardrock Tracker Orientation Schedule 2.

Hardrock GPS Satellite Tracker Introduction 1. Camp Hardrock Tracker Orientation Schedule 2. Tracker Diagram 1 - Buttons 3. Tracker Diagram 2 - Dimensions 4. Rescue Initiation Instructions 5. Tracker Fun Facts and Reminders Hardrock 100 GPS

402 views • 6 slides
Home Brew Con 2018 LOW-OXYGEN EN BREWING Preserves the fresh malt/grain flavor that exists in

Home Brew Con 2018 LOW-OXYGEN EN BREWING Preserves the fresh malt/grain flavor that exists in

Low Oxygen Brewing Notes: Home Brew Con 2018 LOW-OXYGEN EN BREWING Preserves the fresh malt/grain flavor that exists in your malt before you even begin brewing. Theo Th eory Ascorbic Acid Oxidase (AAO) is a malt antioxidant that, when

315 views • 10 slides
Society of Barley Engineers 2oth Anniversary 20 th Anniversary Barley Champagne Malt: Belgian

Society of Barley Engineers 2oth Anniversary 20 th Anniversary Barley Champagne Malt: Belgian

Society of Barley Engineers 2oth Anniversary 20 th Anniversary Barley Champagne Malt: Belgian Pilsner Malt 1.1L +15% Cane Sugar Hops: 83 IBU with Nelson Sauvin late hops Yeast: Cal Ale grown on Session IPA Mash: 2 step @146F &

261 views • 23 slides
MALT & NUMAPROF , Memory Profiling for HPC Applications SBASTIEN VALAT FOSDEM 2019

MALT & NUMAPROF , Memory Profiling for HPC Applications SBASTIEN VALAT FOSDEM 2019

1 MALT & NUMAPROF , Memory Profiling for HPC Applications SBASTIEN VALAT FOSDEM 2019 TRACK HPC Origin of the tools 2 PhD. on memory management for HPC (at CEA/UVSQ) MALT , post-doc at Versailles : NUMAPROF , side

532 views • 16 slides
A Scalable Concurrent malloc(3) Implementation for FreeBSD Jason Evans <jasone@FreeBSD.org>

A Scalable Concurrent malloc(3) Implementation for FreeBSD Jason Evans <jasone@FreeBSD.org>

A Scalable Concurrent malloc(3) Implementation for FreeBSD Jason Evans <jasone@FreeBSD.org> Overview What is malloc(3) ? Previous allocators jemalloc algorithms and data structures Benchmarks Fragmentation

873 views • 52 slides
SDR-BASED APRS TRACKER SDR-BASED APRS TRACKER INTRODUCTION What APRS application to aid

SDR-BASED APRS TRACKER SDR-BASED APRS TRACKER INTRODUCTION What APRS application to aid

EOSS PRESENTATION SDR-BASED APRS TRACKER SDR-BASED APRS TRACKER INTRODUCTION What APRS application to aid trackers in payload recovery Primary Features Software based (no traditional radios) Simultaneous reception of multiple

649 views • 5 slides
The LHCb Silicon Tracker Frank Lehner University of Zurich representing the Silicon Tracker

The LHCb Silicon Tracker Frank Lehner University of Zurich representing the Silicon Tracker

The LHCb Silicon Tracker Frank Lehner University of Zurich representing the Silicon Tracker group of LHCb 11 th International Workshop on Vertex Detectors, Nov. 03-08, Hawaii 2002 The LHCb Silicon Tracker LHCb dedicated b-physics

481 views • 18 slides
Using Hardware Features for Increased Debugging Transparency Fengwei Zhang, Kevin Leach, Angelos

Using Hardware Features for Increased Debugging Transparency Fengwei Zhang, Kevin Leach, Angelos

Using Hardware Features for Increased Debugging Transparency Fengwei Zhang, Kevin Leach, Angelos Stavrou, Haining Wang, and Kun Sun. In S&P'15. Fengwei Zhang Wayne State University CSC 6991 Topics in Computer Security 1 Overview

451 views • 28 slides
MALT: Distributed Data Parallelism for Existing ML Applications Hao Li*, Asim Kadav, Erik Kruus,

MALT: Distributed Data Parallelism for Existing ML Applications Hao Li*, Asim Kadav, Erik Kruus,

MALT: Distributed Data Parallelism for Existing ML Applications Hao Li*, Asim Kadav, Erik Kruus, Cristian Ungureanu * University of Maryland-College Park NEC Laboratories, Princeton Data, data everywhere User-generated Software Hardware

439 views • 42 slides
Session Objectives 1) Describe the purpose and key aspects of incorporating standards of

Session Objectives 1) Describe the purpose and key aspects of incorporating standards of

3/15/2017 Sherri Jones, MS, MBA, RDN, LDN, FAND Improvement Specialist UPMC Shadyside Session Objectives 1) Describe the purpose and key aspects of incorporating standards of excellence into practice 2) Utilize the Academys Standards of

268 views • 14 slides
Nutritional Considerations: Nutrient Content & Variety Living (Well!) with Gastroparesis

Nutritional Considerations: Nutrient Content & Variety Living (Well!) with Gastroparesis

Nutritional Considerations: Nutrient Content & Variety Living (Well!) with Gastroparesis Program Class 7 Nutritional Considerations in the GP-friendly Diet Nutrient Content Variety Balance (class 8) Quality (class 8) Things

354 views • 10 slides
Ensemble Models for Dependency Parsing: Cheap and Good? Mihai Surdeanu and Christopher D. Manning

Ensemble Models for Dependency Parsing: Cheap and Good? Mihai Surdeanu and Christopher D. Manning

Ensemble Models for Dependency Parsing: Cheap and Good? Mihai Surdeanu and Christopher D. Manning Stanford University June 3, 2010 Ensemble Parsing Parser 2 Parser 1 Parser 3 Ensemble Parser Parser 4

195 views • 18 slides
Transparent System Introspection in Support of Analyzing Stealthy Malware Kevin Leach PhD

Transparent System Introspection in Support of Analyzing Stealthy Malware Kevin Leach PhD

Transparent System Introspection in Support of Analyzing Stealthy Malware Kevin Leach PhD Dissertation kjl2y@virginia.edu November 30, 2016 Analogy: Volkswagen Scandal Volkswagen cheated on emissions test (over 10x EPA requirements) 2

1.56k views • 78 slides
Algorithms R OBERT S EDGEWICK | K EVIN W AYNE L INEAR P ROGRAMMING brewers problem

Algorithms R OBERT S EDGEWICK | K EVIN W AYNE L INEAR P ROGRAMMING brewers problem

Algorithms R OBERT S EDGEWICK | K EVIN W AYNE L INEAR P ROGRAMMING brewers problem simplex algorithm implementations Algorithms reductions F O U R T H E D I T I O N R OBERT S EDGEWICK | K EVIN W AYNE

633 views • 50 slides
Inspecting the Structural Biases of Dependency Parsing Algorithms Yoav Goldberg and Michael

Inspecting the Structural Biases of Dependency Parsing Algorithms Yoav Goldberg and Michael

Inspecting the Structural Biases of Dependency Parsing Algorithms Yoav Goldberg and Michael Elhadad Ben Gurion University CoNLL 2010, Sweden There are many ways to parse a sentence There are many ways to parse a sentence Transition Based

1.06k views • 81 slides

More recommend