D e b c o n f 1 6 LTTng: Kernel and userspace tracing in - - PowerPoint PPT Presentation

LTTng: Kernel and userspace tracing in Debian

mjeanson@effjcios.com 

D e b c

• n

f 1 6

w h

• a

mi

 Michael Jeanson, Software developer @ EffjciOS

• Debian Maintainer
• Ubuntu Member
• Fedora Packager
• Offjcial and unoffjcial packager for other distros

C

• n

t e n t

• What is tracing?
• A description of the difgerent tools involved
• LTTng compared to other tracing tools like strace
• The state of LTTng in Debian
• Basic use cases and workfmows
• Analysis of kernel traces

Wh a t i s t r a c i n g ?

• Like a black box / fmight recorder for your system
• Record runtime information

– Syscalls – Function entry/exit

• Enable/Disable event(s) at runtime
• Low overhead

Wh y u s e t r a c i n g ?

• Problems that are not easily diagnosed with

traditional tools or debugging

– Narrow down bug causes – Identify performance hogs

• Very low performance impact

– Can be used on production systems

T

• l

s

• Tracers
• Control utilities
• Viewers
• Post-processing / analysis

T r a c e r s

• lttng-modules: OOT kernel tracer modules

– compatible with kernels 2.6.38 to latest rc – you don’t need to recompile your kernel – lttng-modules-dkms in Debian

• lttng-ust: user-space tracer, in-process library

– Java JUL and log4j agent – Python logging agent

C

• n

t r

• l

u t i l i t i e s

• lttng-tools: cli utilities and daemons for trace

control

– lttng: main cli command – lttng-ctl: tracing control library – lttng-sessiond: tracing registry daemon – lttng-consumerd: extract trace data – lttng-relayd: network streaming daemon

V i e w e r s

• babeltrace: cli text viewer and trace converter
• tracecompass:

– GUI front-end for lttng – Collect, visualize and analyze traces – Eclipse plugin or standalone version

• lttngtop: ncurse top-like viewer

P

• s

t

• p

r

• c

e s s i n g / A n a l y s i s

• Lttng-analyses

– Record your system's activity – Do whatever it takes for your problem to occur. – Diagnose your problem's cause offmine (when

tracing is stopped).

C

• mp

a r e

• strace: syscall and signal tracer
• ftrace: in-kernel function and event tracer
• perf: in-kernel profjler and tracer

L T T n g i n D e b i a n

• All the tools are packaged
• 2 maintainers
• Testing/unstable: latest 2.8 stack
• Stable: 2.5 stack, unsupported :(
• Stable-backports: 2.8 stack coming soon
• Oldstable: Ancient stufg, don’t use it

L T T n g i n U b u n t u

• Xenial: 2.7 stack, supported
• Trusty: 2.4 stack, unsupported :(
• PPAs

– Daily builds – Stable branch builds – Release builds for latest LTS

U s e c a s e s

• Debugging complex and hard to reproduce

problems

• Embedded development with remote tracing
• Use snapshot mode for diffjcult to reproduce bugs
• Low-level metric collection with network streaming
• f traces
• Low-overhead top-like monitoring with lttngtop

Wo r k f l

• w

s

• Given a reproducible problem
• Gather trace (high level at fjrst)
• Analyze (narrow down the problem source)
• Add instrumentation if needed
• Rince, repeat

L T T n g a n a l y s e s

• Demo!

– lttng create – lttng enable-event -k -a – lttng start – ...wait for the problem to appear... – lttng stop – lttng destroy

D e mo : I O u s a g e

$ lttng-iousagetop demo-trace/

D e mo : I O l a t e n c y

$ lttng-iolatencystats demo-trace/ --minsize 2

D e mo : I O l a t e n c y

$ lttng-iolog demo-trace --timerange [12:18:50.162776739,12:18:51.157522361]

• Wouldn’t be a demo if everything worked
• From the previous step, we know when the

latency happened, look at the log

• Find the root cause

Q u e s t i

• n

s

 https://{git | www}.lttng.org  lttng-dev@lists.lttng.org  @lttng_project  #lttng on irc.oftc.net

?

LTTng Project