A Support Engineer Walkthrough on pt-stalk Marcos Albe Marcelo - PowerPoint PPT Presentation

A Support Engineer Walkthrough on pt-stalk Marcos Albe Marcelo Altmann Principal Support Engineer - Percona Senior Support Engineer - Percona

Thank You Sponsors! 2

April 23-25, 2018 SAVE THE DATE! Santa Clara Convention Center CALL FOR PAPERS OPENING SOON! www.perconalive.com 3

Who is speaking ?

Who is Speaking ? • Marcelo Altmann - Senior Support Engineer @ Percona - Resident specialist on Surf & Brazil leisure • Marcos Albé - Principal Support Engineer @ Percona - Chief Barbeque Officer & Designated Beer Analyst 5

Agenda

Agenda • What do we USE • Who’s stalkin’? • How pt-stalk can be USEd • Walkthrough 7

What do we USE

We use one of these... 9

U tilization 10

U tilization S aturation 11

U tilization S aturation E rrors 12

13

Who’s stalkin’?

check out those resources! 15

Who’s stalkin’? • Aspersa stalk + Aspersa collect in one! (thanks Daniel & Baron) • # The name 'stalk' is because 'watch' is already taken, and 'stalk' is fun. • Now part of pt-toolkit • Monitors variable or a script’s single-valued output • Triggers capture when set threshold is reached - f iles from /proc: diskstats, interrupts, meminfo, slabinfo, procstat, procvmstat. - vmstat, iostat, top, netstat, pmap, mpstat, sysctl, df, hostname, mysqladmin ext, mysqladmin debug, lsof). - SHOW ENGINE INNODB STATUS, SHOW ENGINE INNODB MUTEX, SHOW GLOBAL VARIABLES, SHOW FULL PROCESSLIST and SHOW OPEN TABLES. • Production safe! • Custom triggers and custom captures • Companion tools (pt-sift, pt-mext, pt-diskstats, pt-summary); Bash dissection 16

Who’s stalkin’: common parameters • Stalking part - --daemonize | --no-stalk - --function=status - --interval=1 - --variable=Threads_running - --threshold=25 - --cycles=5 • Collecting part - --run-time=30 - --sleep=300 - --iterations= - --plugin= - --dest=/var/lib/pt-stalk • Values shown are the defaults for each (where given) 17

Who’s stalkin’: work flow 18

Who’s stalkin’: working examples I Collect data during 60 seconds for 3 consecutive times, starting immediately. pt-stalk --no-stalk \ --dest=$HOME/percona \ --user=root \ --ask-pass \ --run-time=60 --sleep=60 --iterations=3 19

Who’s stalkin’: working examples II Run as a daemon and trigger collection if InnoDB has more than 5 pending fsyncs. pt-stalk --daemonize \ --function=status \ --variable=Innodb_data_pending_fsyncs \ --threshold=5 \ --cycles=2 \ --interval=1 \ --iterations=5 \ --plugin=/usr/local/bin/raid_card_status.sh 20

Who’s stalkin’: custom triggers #!/bin/bash # Save this bash snippet as /root/delayed_slave_trg.sh; trg_plugin() { mysql $EXT_ARGV -Ee "SHOW SLAVE STATUS" | \ grep "Seconds_Behind_Master" | \ awk "{ print $2 }"; } # use like pt-stalk --function=/root/delayed_slave_trg.sh --etc --etc 21

Who’s stalkin’: custom capture #!/bin/bash # save as /root/collect_perf.sh before_collect() { # run perf stat; match sleep length to --run-time /usr/bin/perf stat --all-cpus \ --output=${d}/${p}-stat_a \ -- sleep ${OPT_RUN_TIME}; } # use like pt-stalk --plugin= /root/collect_perf.sh --etc --etc 22

How pt-stalk can be USE’d

How pt-stalk can be USE’d Applications System libraries System Call Interface CPU Interconnect VFS Sockets Scheduler CPU File Systems TCP/UDP Linux Kernel Volume Manager IP Virtual Memory Memory Bus Block Device Interface Ethernet DRAM Device Drivers I/O Bus I/O Bridge Expander Interconnect I/O Controller Network Controller Interface Transports Disk Disk Swap Port Port 24

SHOW ENGINE INNODB STATUS SHOW ENGINE INNODB MUTEX oprofile SHOW OPEN TABLES /proc/interrupts Applications (MySQL!) SHOW FULL PROCESSLIST /proc/procstat System libraries SHOW ENGINE TOKUDB STATUS mpstat System Call Interface SHOW GLOBAL VARIABLES CPU Interconnect VFS Sockets SHOW SLAVE STATUS Scheduler CPU mysqladmin ext ps File Systems TCP/UDP Linux Kernel gdb (backtraces) top strace Volume Manager IP Virtual Memory Memory Bus vmstat Block Device Interface Ethernet lsof /proc/meminfo df /proc/slabinfo DRAM Device Drivers iostat /proc/vmstat I/O Bus pmap /proc/diskstats netstat sysctl tcpdump I/O Bridge Expander Interconnect I/O Controller Network Controller Interface Transports Disk Disk Swap Port Port 25

How pt-stalk can be USE’d: Here Be Dragons! gdb (backtraces) oprofile strace tcpdump 26

A walkthrough

A walkthrough Using pt-stalk to solve problems with... ● ...High IO ● ...High CPU ● ...Networking 28

Useful links ● grep commands - https://gist.github.com/altmannmarcelo/eec40a54568299c2cc11b6cf5859 203d ● Check List - http://www.brendangregg.com/USEmethod/use-linux.html 29

Questions ? Marcelo Altmann Marcos Albe @altmannmarcelo marcos.albe@percona.com marcelo.altmann@percona.com