Low-Overhead System Tracing With eBPF Akshay Kapoor DevOps - - PowerPoint PPT Presentation

▶

Jul 11, 2023 364 likes •636 views

Low-Overhead System Tracing With eBPF Akshay Kapoor DevOps Engineer @ SAP Labs May 2018 Low-Overhead System Tracing With eBPF Low-Overhead System Tracing With eBPF Low-Overhead System Tracing With eBPF You don't need to know how to operate

SLIDE 1

Low-Overhead System Tracing With eBPF

May 2018

Akshay Kapoor DevOps Engineer @ SAP Labs

SLIDE 2

Low-Overhead System Tracing With eBPF

SLIDE 3

Low-Overhead System Tracing With eBPF

SLIDE 4

Low-Overhead System Tracing With eBPF

SLIDE 5

You don't need to know how to operate an X-ray machine, but you do need to know that if you swallow a penny, an X-ray is an option!

~ www.bredangregg.com

SLIDE 6

CLASSIC PKT. FILTERING

BPF VIRTUAL MACHINE EXTENDED BPF BPF SYSTEM CALL

ADDTL. PROBES

1993 Before 1992 2013 2014 Today

EVOLUTION OF BPF

STACK BASED KERNEL -> USPACE COPIES REGISTER BASED (2) LESSER COPIES IMPROVED ISA & eBPF MAPS MORE REGISTERS (10) EXPOSED TO USER SPACE KERNEL FILTERS UPROBES, KPROBES USDT, TRACEPOINTS

SLIDE 7

tcpdump -n "dst host 192.168.1.1 and dst port 23"

SLIDE 8

# Credits : https://suchakra.wordpress.com/

HOW BPF WORKS ?

SLIDE 9

BCC (BPF COMPILER COLLECTION)

https://github.com/iovisor/bcc
Lead Developer – Brenden Blanco

#Credits : Sasha Goldshtein

SLIDE 10

SLIDE 11

BCC Tools [examples…]

SLIDE 12

BCC Tools [examples…]

SLIDE 13

BCC Tools [examples…]

SLIDE 14

BCC Tools [examples…]

SLIDE 15

Flamegraphs [ BCC/BPF Visualizations ]

SLIDE 16

Flamegraphs [ BCC/BPF Visualizations] (Source : https://blog.cloudflare.com/tracing-system-cpu-on-debian-stretch/)

Call Stacks

No. of Samples

SLIDE 17

BPF and Containers

SLIDE 18

Namespaces

BPF and containers…

PID Y PID X

CONTAINER HOST

Restricts Visibility

mnt
pid
net
. . .

SLIDE 19

Namespaces
CGroups

BPF and containers…

CONTAINER 1

HOST

Restricts Quota/Usage

cpu
mem
blkio
. . .

CONTAINER 2

CPU SHARES

SLIDE 20

Namespaces
CGroups
Analysis from the host
PID Mappings (/sys/fs/cgroup/docker/*)
Symbol file locations

BPF and containers…

0x7f82b510ddda 0x7f82b510999d 0x7f82b510f665 0x7f82b510t546

SLIDE 21

Namespaces
CGroups
Analysis from the host
PID Mappings (/sys/fs/cgroup*)
Symbol file locations
Deployment Methodologies

BPF and containers…

APP CONTAINER 1 APP CONTAINER 2 BCC TOOLS ON HOST

KERNEL EVENTS

SLIDE 22

Namespaces
CGroups
Analysis from the host
PID Mappings (/sys/fs/cgroup*)
Symbol file locations
Deployment Methodologies

BPF and containers…

APP CONTAINER 1 APP CONTAINER 2

KERNEL EVENTS

BCC CONTAINER 3

SLIDE 23

DEMO

OBSERVABILITY NETWORKING SECURITY

SLIDE 24

BPF Implementations…

Seccomp
Control system calls made by a process
Cilium
Controls Networking, Security and Load Balancing for containers
Weavescope
Observability into containerized application stacks like Docker and Kubernetes
Iptables
Bpfilter implementations to optimize ingress/outgress security rules
Systemtap
BPF backend for optimizations

SLIDE 25

References

Sasha Goldshtein (goldshtn) Brendan Gregg (brendangregg) Suchakra (tuxology) Julia Evans (b0rk)

@Follow

https://github.com/iovisor/bcc http://man7.org/linux/man-pages/man2/bpf.2.html http://brendangregg.com/ebpf.html https://github.com/goldshtn/linux-tracing-workshop https://suchakra.wordpress.com/ - eBPF https://blog.yadutaf.fr/ - Networking & eBPF https://jvns.ca/blog/2017/07/05/linux-tracing-systems/ https://www.youtube.com/watch?v=aaTQM7wcmfk – Kernel Meetup | eBPF https://cilium.io/blog/2018/04/17/why-is-the-kernel-community-replacing-iptables/ https://blog.yadutaf.fr/2016/03/30/turn-any-syscall-into-event-introducing-ebpf-kernel-probes/ https://lwn.net/Articles/740157/ - Thorough eBPF intro https://developers.redhat.com/blog/2017/12/13/introducing-stapbpf-systemtaps-new-bpf-backend/ https://lwn.net/Articles/747551/ - BPF comes to firewalls

SLIDE 26

Thank You !

akshay.kapoor@sap.com akskap akskap akskap