Kernel Debugging and Virtualization John Baldwin January 15, 2015 - - PowerPoint PPT Presentation

Kernel Debugging and Virtualization

John Baldwin January 15, 2015

What is Kernel Debugging

• Step in Kernel Development
• Post-Mortem Crash Analysis
• Live Inspection
• Performance Analysis
• Userland Debugging

Wait, Userland Debugging?

• Userland Debugging == User Process

Debugging

• Processes Have In-Kernel State
• Not Exposed via gdb/lldb
• Other Tools like lsof/fstat/procstat
• Not All State Exposed via Tools

– “Which FDs is a thread waiting on in select() / poll() /

kqueue()?”

– “Which directory is a thread waiting on a lock for?”

Similarities to User Process Debugging

• Kernel is Like a Multithreaded Process
• Shared Address Space
• Threads With Stacks
• Memory Dumps

Differences from User Process Debugging

• Longer Turnaround (Rebooting is Slow)
• More Setup Required for Full Debug
• Can Debug Itself (but Limited)

Kernel Debugging Tools on FreeBSD

• printf()
• Often tool of “first resort”
• Recompile / reboot to log “new” data
• Run Time Checks
• Assertions
• WITNESS
• DEADLKRES
• malloc() Debugging

Kernel Debugging Tools on FreeBSD

• Crash Dumps
• Memory Dumps

– ps(1), netstat(1), etc.

• Text Dumps
• In-kernel Debugger (DDB)
• Console Access Required
• Extendable
• Variable Examination is Primitive
• Stops System

Kernel Debugging Tools on FreeBSD

• kgdb
• Different Targets

– Crash Dump – Live (Does Not Stop System) – Remote (OOB Connection)

• Mature Variable Expansion
• Extendable via Scripts
• dtrace

Virtualization Makes It Better

• Isolation on a Single Machine
• Quicker Turn Around (Reboot)
• Console Access and OOB are Simple
• /dev/nmdm<foo> vs Physical Cables
• Device Pass Through

Demo