Kernel Debugging and Virtualization John Baldwin January 15, 2015 - PowerPoint PPT Presentation

Kernel Debugging and Virtualization John Baldwin January 15, 2015

What is Kernel Debugging ● Step in Kernel Development ● Post-Mortem Crash Analysis ● Live Inspection ● Performance Analysis ● Userland Debugging

Wait, Userland Debugging? ● Userland Debugging == User Process Debugging ● Processes Have In-Kernel State ● Not Exposed via gdb/lldb ● Other Tools like lsof/fstat/procstat ● Not All State Exposed via Tools – “Which FDs is a thread waiting on in select() / poll() / kqueue()?” – “Which directory is a thread waiting on a lock for?”

Similarities to User Process Debugging ● Kernel is Like a Multithreaded Process ● Shared Address Space ● Threads With Stacks ● Memory Dumps

Differences from User Process Debugging ● Longer Turnaround (Rebooting is Slow) ● More Setup Required for Full Debug ● Can Debug Itself (but Limited)

Kernel Debugging Tools on FreeBSD ● printf() ● Often tool of “first resort” ● Recompile / reboot to log “new” data ● Run Time Checks ● Assertions ● WITNESS ● DEADLKRES ● malloc() Debugging

Kernel Debugging Tools on FreeBSD ● Crash Dumps ● Memory Dumps – ps(1), netstat(1), etc. ● Text Dumps ● In-kernel Debugger (DDB) ● Console Access Required ● Extendable ● Variable Examination is Primitive ● Stops System

Kernel Debugging Tools on FreeBSD ● kgdb ● Different Targets – Crash Dump – Live (Does Not Stop System) – Remote (OOB Connection) ● Mature Variable Expansion ● Extendable via Scripts ● dtrace

Virtualization Makes It Better ● Isolation on a Single Machine ● Quicker Turn Around (Reboot) ● Console Access and OOB are Simple ● /dev/nmdm<foo> vs Physical Cables ● Device Pass Through

Demo