Eliminating bugs in BPF JITs using automated formal verification Luke Nelson with Jacob van Ge ff en, Emina Torlak, and Xi Wang
BPF is used throughout the kernel • Many uses for BPF: tracing, networking, security, etc. • In-kernel JIT compilers for better performance BPF program BPF verifier Linux BPF JIT compiler kernel Native code
BPF JITs are hard to get right • Developers have to think about code at multiple levels • The JIT itself and the machine code produced by the JIT • Kernel selftests + fuzzing are e ff ective at preventing many bugs • But the search space is too large to exhaust all possibilities • Many corner cases in the input to JIT and input to the BPF program • Compiled code runs in kernel; JIT bugs can become kernel vulnerabilities
BPF JITs are hard to get right case BPF_LDX | BPF_MEM | BPF_W: ... switch (BPF_SIZE(code)) { case BPF_W: Control flow in the JIT if (!bpf_prog->aux->verifier_zext) Control flow in the JIT or even compiled code break ; if (dstk) { EMIT3( 0xC7 , add_1reg( 0x40 , IA32_EBP), STACK_VAR(dst_hi)); Emitting instructions as raw bytes EMIT( 0x0 , 4 ); } else { EMIT3( 0xC7 , add_1reg( 0xC0 , dst_hi), 0 ); } There’s a bug in this code: can you break ; spot it? 🐝
Eliminating bugs with formal verification • Formally prove the absence of bugs • Specification: abstract description of intended behavior • Prove that implementation satisfies the specification ≈ specification proof implementation
Developer burden using formal verification • Formal verification requires more manual e ff ort compared to testing • Requires writing down a specification • Specification must prevent bugs and cover existing implementations • Requires proving the implementation meets that specification • Manual proofs are time-consuming, can be >10 ✕ LOC proof to implementation • Existing automated techniques will not scale well
Main results • Jitterbug is a tool for automated formal verification of the BPF JITs in Linux. • JIT specification + automated proof strategy • Implementation in a domain-specific language (DSL) • Found and fixed 30+ new bugs in existing BPF JITs across 11 patches. • Manual translation of JITs to DSL for verification, several weeks per JIT • Developed a new BPF JIT for 32-bit RISC-V (riscv32 / RV32). • Written in DSL; automated extraction to C code • Developed 12 new optimization patches for existing JITs.
Outline • Overview of how the BPF JITs in Linux work • Case study of bugs in BPF JITs • Overview of Jitterbug’s JIT specification • How to use Jitterbug • Verification e ff ort • Demonstration • Future directions for JIT verification
Outline • Overview of how the BPF JITs in Linux work • Case study of bugs in BPF JITs • Overview of Jitterbug’s JIT specification • How to use Jitterbug • Verification e ff ort • Demonstration • Future directions for JIT verification
BPF JIT overview (1/2) • Verifier checks if BPF program is safe to execute. • JIT compiles program if verifier deems it safe • Jitterbug focuses on BPF JIT — Assumes BPF verifier to be correct BPF program BPF verifier Linux BPF JIT compiler kernel Native code
BPF JIT overview (2/2) Function prologue emit_prologue • Static register allocation BPF program Function body emit_insn BPF_ADD_X R0, R1 • Emits prologue / epilogue to set up addq %rax, %rdi … stack, etc. … • Compiles one BPF instruction at a time • Repeats JIT until code converges Function epilogue emit_epilogue
Outline • Overview of how the BPF JITs in Linux work • Case study of bugs in BPF JITs • Overview of Jitterbug’s JIT specification • How to use Jitterbug • Verification e ff ort • Demonstration • Future directions for JIT verification
eBPF JIT History 2020 2014 2015 2016 2017 2018 2019 x86-64 arm32 x86-32 riscv64 riscv32 s390 ppc64 arm64 mips64 sparc64 • JIT support for eBPF added over past ~7 years • We looked at x86, Arm, & RISC-V (32- and 64-bit)
Bugs in BPF JITs • We manually reviewed bug-fixing commits in existing BPF JITs • 82 JIT correctness bugs across 41 commits from from May 2014— Apr. 2020 • Correctness bug: JIT produces wrong native code for a BPF instruction Prologue / Epilogue / Tail call 15 CALL ALU 3 33 JMP 13 MEM 18
Bugs found using Jitterbug ◦ bpf, riscv: clear high 32 bits for ALU32 add/sub/neg/lsh/rsh/arsh ◦ bpf, x86_32: Fix incorrect encoding in BPF_LDX zero-extension ◦ arm, bpf: Fix bugs with ALU64 {RSH, ARSH} BPF_K shift by 0 ◦ arm, bpf: Fix offset overflow for BPF_MEM BPF_DW ◦ arm64: insn: Fix two bugs in encoding 32-bit logical immediates ◦ riscv, bpf: Fix offset range checking for auipc+jalr on RV64 ◦ bpf, x32: Fix bug with ALU64 {LSH, RSH, ARSH} BPF_K shift by 0 ◦ bpf, x32: Fix bug with ALU64 {LSH, RSH, ARSH} BPF_X shift by 0 ◦ bpf, x32: Fix bug with JMP32 JSET BPF_X checking upper bits ◦ bpf, x86_32: Fix clobbering of dst for BPF_JSET ◦ bpf, x86: Fix encoding for lower 8-bit registers in BPF_STX BPF_B
Bugs found using Jitterbug ◦ bpf, riscv: clear high 32 bits for ALU32 add/sub/neg/lsh/rsh/arsh ◦ bpf, x86_32: Fix incorrect encoding in BPF_LDX zero-extension ◦ arm, bpf: Fix bugs with ALU64 {RSH, ARSH} BPF_K shift by 0 ◦ arm, bpf: Fix offset overflow for BPF_MEM BPF_DW ◦ arm64: insn: Fix two bugs in encoding 32-bit logical immediates ◦ riscv, bpf: Fix offset range checking for auipc+jalr on RV64 ◦ bpf, x32: Fix bug with ALU64 {LSH, RSH, ARSH} BPF_K shift by 0 ◦ bpf, x32: Fix bug with ALU64 {LSH, RSH, ARSH} BPF_X shift by 0 ◦ bpf, x32: Fix bug with JMP32 JSET BPF_X checking upper bits ◦ bpf, x86_32: Fix clobbering of dst for BPF_JSET ◦ bpf, x86: Fix encoding for lower 8-bit registers in BPF_STX BPF_B
Example bug (1/2) Zero-extension of 32-bit ALU instructions on riscv64 • BPF 32-bit ALU instructions ( BPF_ALU ) zero-extend to 64 bits • riscv64 32-bit ALU instructions (e.g., subw ) sign-extend to 64 bits • Bug: Mismatch between BPF and RISC-V semantics • Fix: Emit additional instructions to zero-extend result case BPF_ALU | BPF_SUB | BPF_X: case BPF_ALU64 | BPF_SUB | BPF_X: emit(is64 ? rv_sub(rd, rd, rs) : rv_subw(rd, rd, rs), ctx); + if (!is64) + emit_zext_32(rd, ctx); break;
Example bug (2/2) mov encoding in LDX on x86-32 • 4-byte BPF memory load zero- case BPF_LDX | BPF_MEM | BPF_W: extends upper 32 bits ... • 2 x86 registers per 1 BPF register switch (BPF_SIZE(code)) { case BPF_W: • zero-extending is setting reg ... holding the high bits to 0 if (dstk) { EMIT3( 0xC7 , add_1reg( 0x40 , IA32_EBP), • JIT uses the following instruction: STACK_VAR(dst_hi)); EMIT( 0x0 , 4 ); } else { movl $0, %dst_hi EMIT3( 0xC7 , add_1reg( 0xC0 , dst_hi), 0 ); } break ;
Example bug (2/2) mov encoding in LDX on x86-32 movl $0, %dst_hi EMIT3( 0xC7 , add_1reg( 0xC0 , dst_hi), 0 ); • EMIT3 : Emit 3 bytes of instruction • 0xC7 : Opcode for “ mov r/m32, imm32 ” • add_1reg( 0xC0 , dst_hi) : Encodes destination register • 0 : one byte of immediate • Bug: “ mov ” expects imm32 , missing 3 bytes of the immediate! • Fix: Use “ xor ” instead: correct encoding, fewer bytes xorl %dst_hi, %dst_hi EMIT2( 0x33 , add_2reg( 0xC0 , dst_hi, dst_hi));
Outline • Overview of how the BPF JITs in Linux work • Case study of bugs in BPF JITs • Overview of Jitterbug’s JIT specification • How to use Jitterbug • Verification e ff ort • Demonstration • Future directions for JIT verification
How to systematically rule out bugs? • Need a specification that rules out classes of bugs in BPF JITs • Encoding bugs, semantics bugs, etc. • ALU, JMP , MEM, CALL, etc. • What does JIT correctness even mean? • How to prove implementation meets specification?
Specification: End-to-end correctness For all BPF programs, for all inputs, compiled code should produce same output and trace of events as BPF program BPF program Output + trace of events = JIT Input (packet, etc.) Output + trace of events Compiled program Trace: sequence of memory loads / stores + function calls
Specification: End-to-end correctness For all BPF programs, for all inputs, compiled code should produce same output and trace of events as BPF program BPF program Output + trace of events Hard to prove: cannot enumerate all BPF programs = JIT Input (packet, etc.) Output + trace of events Compiled program Trace: sequence of memory loads / stores + function calls
Specification: Breaking down to three parts • Prologue correctness • The JIT prologue sets up BPF state (e.g., the stack) correctly • Per-instruction correctness • The JIT produces correct machine code for each individual BPF instruction • Epilogue correctness • The JIT epilogue tears down BPF state correctly and returns correct value
Specification: Breaking down to three parts • Prologue correctness • The JIT prologue sets up BPF state (e.g., the stack) correctly • Per-instruction correctness • The JIT produces correct machine code for each individual BPF instruction • Epilogue correctness • The JIT epilogue tears down BPF state correctly and returns correct value
Recommend
More recommend
