Low-Overhead System Tracing With eBPF Akshay Kapoor DevOps - PowerPoint PPT Presentation

Low-Overhead System Tracing With eBPF Akshay Kapoor DevOps Engineer @ SAP Labs May 2018

Low-Overhead System Tracing With eBPF

Low-Overhead System Tracing With eBPF

Low-Overhead System Tracing With eBPF

You don't need to know how to operate an X-ray machine, but you do need to know that if you swallow a penny, an X-ray is an option! ~ www.bredangregg.com

EVOLUTION OF BPF REGISTER BASED (2) EXPOSED TO USER SPACE LESSER COPIES KERNEL FILTERS BPF VIRTUAL BPF SYSTEM CALL MACHINE Before 1992 2013 Today ----------------------------------------------------------------------- 1993 2014 CLASSIC PKT. FILTERING EXTENDED BPF ADDTL. PROBES UPROBES, KPROBES STACK BASED IMPROVED ISA & eBPF MAPS USDT, TRACEPOINTS KERNEL -> USPACE COPIES MORE REGISTERS (10)

tcpdump -n "dst host 192.168.1.1 and dst port 23"

HOW BPF WORKS ? # Credits : https://suchakra.wordpress.com/

BCC (BPF COMPILER COLLECTION) https://github.com/iovisor/bcc • Lead Developer – Brenden Blanco • #Credits : Sasha Goldshtein

BCC Tools [examples…]

BCC Tools [examples…]

BCC Tools [examples…]

BCC Tools [examples…]

Flamegraphs [ BCC/BPF Visualizations ]

Call Stacks No. of Samples Flamegraphs [ BCC/BPF Visualizations] (Source : https://blog.cloudflare.com/tracing-system-cpu-on-debian-stretch/)

BPF and Containers

BPF and containers… • Namespaces Restricts Visibility • mnt CONTAINER • pid PID X net • . . . • PID Y HOST

BPF and containers… • Namespaces Restricts Quota/Usage • CGroups • cpu CONTAINER 1 CONTAINER 2 • mem blkio • . . . • HOST CPU SHARES

BPF and containers… • Namespaces • CGroups • Analysis from the host 0x7f82b510ddda • PID Mappings (/sys/fs/cgroup/docker/*) 0x7f82b510999d • Symbol file locations 0x7f82b510f665 0x7f82b510t546

BPF and containers… • Namespaces BCC APP APP TOOLS ON CONTAINER 1 CONTAINER 2 HOST • CGroups • Analysis from the host • PID Mappings (/sys/fs/cgroup*) • Symbol file locations KERNEL EVENTS • Deployment Methodologies

BPF and containers… • Namespaces APP APP BCC CONTAINER 1 CONTAINER 2 CONTAINER 3 • CGroups • Analysis from the host • PID Mappings (/sys/fs/cgroup*) • Symbol file locations KERNEL EVENTS • Deployment Methodologies

DEMO NETWORKING OBSERVABILITY SECURITY

BPF Implementations… - Seccomp Control system calls made by a process • - Cilium Controls Networking, Security and Load Balancing for containers • - Weavescope Observability into containerized application stacks like Docker and Kubernetes • - Iptables Bpfilter implementations to optimize ingress/outgress security rules • - Systemtap BPF backend for optimizations •

References @Follow Sasha Goldshtein (goldshtn) https://github.com/iovisor/bcc http://man7.org/linux/man-pages/man2/bpf.2.html Brendan Gregg (brendangregg) http://brendangregg.com/ebpf.html https://github.com/goldshtn/linux-tracing-workshop Suchakra (tuxology) https://suchakra.wordpress.com/ - eBPF Julia Evans (b0rk) https://blog.yadutaf.fr/ - Networking & eBPF https://jvns.ca/blog/2017/07/05/linux-tracing-systems/ https://www.youtube.com/watch?v=aaTQM7wcmfk – Kernel Meetup | eBPF https://cilium.io/blog/2018/04/17/why-is-the-kernel-community-replacing-iptables/ https://blog.yadutaf.fr/2016/03/30/turn-any-syscall-into-event-introducing-ebpf-kernel-probes/ https://lwn.net/Articles/740157/ - Thorough eBPF intro https://developers.redhat.com/blog/2017/12/13/introducing-stapbpf-systemtaps-new-bpf-backend/ https://lwn.net/Articles/747551/ - BPF comes to firewalls

Thank You ! akshay.kapoor@sap.com akskap akskap akskap