A Support Engineer Walkthrough on pt-stalk Marcos Albe Marcelo - - PowerPoint PPT Presentation

▶
a support engineer walkthrough on pt stalk
SMART_READER_LITE
LIVE PREVIEW

A Support Engineer Walkthrough on pt-stalk Marcos Albe Marcelo - - PowerPoint PPT Presentation

Feb 10, 2024 554 likes •873 views

A Support Engineer Walkthrough on pt-stalk Marcos Albe Marcelo Altmann Principal Support Engineer - Percona Senior Support Engineer - Percona Thank You Sponsors! 2 April 23-25, 2018 SAVE THE DATE! Santa Clara Convention Center CALL FOR

slide-1
SLIDE 1

A Support Engineer Walkthrough on pt-stalk

Marcos Albe Principal Support Engineer - Percona Marcelo Altmann Senior Support Engineer - Percona

slide-2
SLIDE 2

2

Thank You Sponsors!

slide-3
SLIDE 3

3

SAVE THE DATE!

CALL FOR PAPERS OPENING SOON!

www.perconalive.com

April 23-25, 2018

Santa Clara Convention Center

slide-4
SLIDE 4

Who is speaking ?

slide-5
SLIDE 5

5

Who is Speaking ?

  • Marcelo Altmann
  • Senior Support Engineer @ Percona
  • Resident specialist on Surf & Brazil leisure
  • Marcos Albé
  • Principal Support Engineer @ Percona
  • Chief Barbeque Officer & Designated Beer Analyst
slide-6
SLIDE 6

Agenda

slide-7
SLIDE 7

7

Agenda

  • What do we USE
  • Who’s stalkin’?
  • How pt-stalk can be USEd
  • Walkthrough
slide-8
SLIDE 8

What do we USE

slide-9
SLIDE 9

9

We use one

  • f these...
slide-10
SLIDE 10

10

Utilization

slide-11
SLIDE 11

11

Utilization Saturation

slide-12
SLIDE 12

12

Utilization Saturation Errors

slide-13
SLIDE 13

13

slide-14
SLIDE 14

Who’s stalkin’?

slide-15
SLIDE 15

15

check out those resources!

slide-16
SLIDE 16

16

Who’s stalkin’?

  • Aspersa stalk + Aspersa collect in one! (thanks Daniel & Baron)
  • # The name 'stalk' is because 'watch' is already taken, and 'stalk' is fun.
  • Now part of pt-toolkit
  • Monitors variable or a script’s single-valued output
  • Triggers capture when set threshold is reached
  • files from /proc: diskstats, interrupts, meminfo, slabinfo, procstat, procvmstat.
  • vmstat, iostat, top, netstat, pmap, mpstat, sysctl, df, hostname, mysqladmin ext, mysqladmin debug, lsof).
  • SHOW ENGINE INNODB STATUS, SHOW ENGINE INNODB MUTEX, SHOW GLOBAL VARIABLES, SHOW FULL

PROCESSLIST and SHOW OPEN TABLES.

  • Production safe!
  • Custom triggers and custom captures
  • Companion tools (pt-sift, pt-mext, pt-diskstats, pt-summary); Bash dissection
slide-17
SLIDE 17

17

Who’s stalkin’: common parameters

  • Stalking part
  • --daemonize | --no-stalk
  • --function=status
  • --interval=1
  • --variable=Threads_running
  • --threshold=25
  • --cycles=5
  • Collecting part
  • --run-time=30
  • --sleep=300
  • --iterations=
  • --plugin=
  • --dest=/var/lib/pt-stalk
  • Values shown are the defaults for each (where given)
slide-18
SLIDE 18

18

Who’s stalkin’: work flow

slide-19
SLIDE 19

19

Who’s stalkin’: working examples I

Collect data during 60 seconds for 3 consecutive times, starting immediately.

pt-stalk --no-stalk \

  • -dest=$HOME/percona \
  • -user=root \
  • -ask-pass \
  • -run-time=60
  • -sleep=60
  • -iterations=3
slide-20
SLIDE 20

20

Who’s stalkin’: working examples II

Run as a daemon and trigger collection if InnoDB has more than 5 pending fsyncs.

pt-stalk --daemonize \

  • -function=status \
  • -variable=Innodb_data_pending_fsyncs \
  • -threshold=5 \
  • -cycles=2 \
  • -interval=1 \
  • -iterations=5 \
  • -plugin=/usr/local/bin/raid_card_status.sh
slide-21
SLIDE 21

21

Who’s stalkin’: custom triggers

#!/bin/bash # Save this bash snippet as /root/delayed_slave_trg.sh; trg_plugin() { mysql $EXT_ARGV -Ee "SHOW SLAVE STATUS" | \ grep "Seconds_Behind_Master" | \ awk "{ print $2 }"; } # use like

pt-stalk --function=/root/delayed_slave_trg.sh --etc --etc

slide-22
SLIDE 22

22

Who’s stalkin’: custom capture

#!/bin/bash # save as /root/collect_perf.sh before_collect() { # run perf stat; match sleep length to --run-time /usr/bin/perf stat --all-cpus \

  • -output=${d}/${p}-stat_a \
  • - sleep ${OPT_RUN_TIME};

} # use like

pt-stalk --plugin=/root/collect_perf.sh --etc --etc

slide-23
SLIDE 23

How pt-stalk can be USE’d

slide-24
SLIDE 24

24

How pt-stalk can be USE’d

Linux Kernel

CPU

CPU Interconnect

DRAM I/O Bridge I/O Controller Network Controller

I/O Bus Expander Interconnect

Disk Disk Swap Port Port

Applications System libraries System Call Interface VFS Sockets Scheduler File Systems TCP/UDP Volume Manager IP Virtual Memory Block Device Interface Ethernet Device Drivers Memory Bus Interface Transports

slide-25
SLIDE 25

25

Linux Kernel

CPU

CPU Interconnect

DRAM I/O Bridge I/O Controller Network Controller

I/O Bus Expander Interconnect

Disk Disk Swap Port Port

Applications (MySQL!) System libraries System Call Interface VFS Sockets Scheduler File Systems TCP/UDP Volume Manager IP Virtual Memory Block Device Interface Ethernet Device Drivers Memory Bus Interface Transports ps df /proc/diskstats SHOW ENGINE INNODB STATUS /proc/interrupts iostat lsof /proc/meminfo mpstat SHOW ENGINE INNODB MUTEX mysqladmin ext SHOW OPEN TABLES pmap /proc/procstat SHOW FULL PROCESSLIST /proc/vmstat /proc/slabinfo sysctl SHOW ENGINE TOKUDB STATUS top SHOW GLOBAL VARIABLES vmstat SHOW SLAVE STATUS netstat strace tcpdump gdb (backtraces)

  • profile
slide-26
SLIDE 26

26

strace tcpdump gdb (backtraces)

  • profile

How pt-stalk can be USE’d: Here Be Dragons!

slide-27
SLIDE 27

A walkthrough

slide-28
SLIDE 28

28

A walkthrough

Using pt-stalk to solve problems with...

  • ...High IO
  • ...High CPU
  • ...Networking
slide-29
SLIDE 29

29

Useful links

  • grep commands -

https://gist.github.com/altmannmarcelo/eec40a54568299c2cc11b6cf5859 203d

  • Check List - http://www.brendangregg.com/USEmethod/use-linux.html
slide-30
SLIDE 30

Questions ?

Marcos Albe marcos.albe@percona.com Marcelo Altmann @altmannmarcelo marcelo.altmann@percona.com