Static Analysis versus Software Model Checking for bug finding - - PowerPoint PPT Presentation

Static Analysis versus Software Model Checking for bug finding

Dawson Englers, Madanlal Musuvathi Stanford University, CA, USA Presented By: Prateek Agarwal ETH, Zürich

Agenda

• Goal
• Aim
• Scope
• Methodologies used

– Meta Compilation – C Model Checker

• Case Studies
• Conclusions

Goal

Checking validity of general perception

Static Analysis is easy to apply, finds shallow bugs Model Checking is harder but strictly better

Aim

• Documentation of experience
• Set of case studies
• Comparison of two approaches, based on

– Bugs/Lines of Code – False positives vs Actual Bugs – Time & Effort required – Types of Bugs: generic vs protocol specific

• Find as many bugs as possible

Scope

• Verification of system/protocol software

– Critical – Hard to test/inspect/reason about manually – Highly optimized

• Protocol codes follow event driven model

Meta Compilation (MC)

• Extension of compiler
• Used for Static Analysis & Model Checking
• Metal: Language for

– Slice specification by pattern matching – Code translation – State machines

• xg++ over g++ compiler

Meta Compilation: Key Features

• Code should be highly structured

– e.g. special MACRO/functions for read/write

• No source code annotation needed
• Emphasis on reducing false positives
• No Simulation of code

– No model of heap, tracking variable values – Simulation tends to become model checking

Static Analysis using MC

• Patterns & Actions
• Small State machines for checking

Example Rule: Sync before read ... start: {WAIT_FOR_DB_FULL(addr);} ==> stop | {MISCBUS_READ_DB(addr);} ==> {err(“Buffer read not sync");}; ....

Start Error Stop misc_bus_read_db wait_for_db_full

Engler & Masuvathi [1]

Model Checking using MC

• Extraction & Translation

– protocol code Mur

→ Φ Model

• Eliminates handcrafted system models
• MurΦ

– Explicit state enumeration model checker – Each reachable state visited once

Model Checking using MC

State Variable (Metal Slicer) Translation Pattern (Metal Printer) Correctness Properties Protocol Model Protocol Code (Implementation) Start State Hardware Model Error List xg++ MurØ

Automatically Generated Hand Crafted by User Program

Lie, Chou, Engler, Dill [2]

Legend:

Unedited source code

Example: Model Checking using MC

Metal Slicer: ... /* Patterns for n/w and proc mesgs, which use length field */ pat sends = { NI_SEND (type, data, keep,swp, wait,nl);} | { P1_SEND(type, data, keep,swp, wait,nl);} ... Metal Printer ... /* Automatically insert length assertion before send */ {N1_SEND(type, data, keep, swap, wait, null);} ==> { if(mgk_int_cst(data) != 0)a mgk_e(“assert(nh.len = len_data);”); else mgk_e(“assert (nh.len = len_nodata);”); mgk_e(“ni_send(%t, %t, procNum,nh”,type,swap); ...

Lie, Chou, Engler, Dill [2]

Model Checking : C Model Checker

• Motivation

– xg++ user needs to know system intricacies – MurΦ lacks many C constructs – Eliminate need of higher level system model

• Use the implementation code itself
• Process scheduling & execution by CMC

CMC Key Features

• Searches for all transitions
• Complete system state saved and restored
• Heuristics & Optimizations for state space

explosion

• Correctness properties

– Assertions from implementation code – Memory leaks etc. in built – User specified invariants

• Environment Model still handcrafted

Case Study 1: FLASH

• Cache coherence protocol for FLASH

– Multiprocessor architecture – Code runs on each cache miss

• 5 FLASH protocols
• Code Size: 10K-18K

John Hennessy [3]

Results

• Static Analysis

– Bugs Found: 33, False Positives: 28 – Most rules on buffer management – e.g. allocation before use, deallocation after use,

not used after deallocation

• Model Checking

– 8 bugs – e.g. queue overflows, no invalidation of an

exclusive line

– Handwritten model lead to delays

Summary

• Static Analysis Advantage:

– Better for same set of properties – Works best for code visible rules

• Model Checking Advantage:

– Can cover complex properties, invariants – e.g. counter overflow missed by SA

• Model Checking Disadvantage:

– Simple models Less Coverage

→

– Detailed model Increased Time & Effort

→

– Erroneous model False Positives

→

Case Study 2: AODV Protocol

• Ad-hoc On-demand Distance Vector Protocol
• Routing protocol
• Simplified environment

– Input: route request, timer interrupt, packet receipt

• 3 publicly available implementations used
• Average 6 KLOC
• Model Checking using CMC

Results

• Model Checking

– 42 unique bugs, 1 bug in AODV specification

• Static Analysis: 34 bugs

Both Model Checking Static Analysis Generic Properties 21 1 13 Protocol Specific 20 Total 21 21 13

comparison of bug count classified by properties

Comparison

• Generic Properties:

– CMC found only one bug not found by SA – SA found 13 bugs not found by CMC

• Protocol Specific Properties:

– Difficult to check statically 0 bugs

→

• SA hit more code but CMC hit more properties!

Summary

• Static Analysis Advantage:

– Wins when checking same set of properties (again)

• Model Checking Advantage:

– Simulates execution of 'entire system' – Looks for actual errors (not 'causes' of error) – Gives actual execution trace

• More checks more bugs

→

– CMC executed the code

Case Study 3: TCP

• TCP Code size ~ 10 * (AODV code size)
• Frequently audited, Heavily Tested code
• Only Model Checking done (using CMC)
• System tightly coupled with Environment

– System: TCP stack – Environment: Linux Kernel, Kernel Modules

• Incremental refinement of model

– Start simple, keep adding complexity

Case Study 3: TCP

• External Functions can be included in System
• Advantage

– Model Checker will find errors in them – No effort to create stubs

• Disadvantage

– Increase in state space – Dependencies on other functions

• Essential Question

– Where to draw the line between system and

environment?

System Separation

• Cut on narrowest interface
• TCP – Kernel module

– Poorly documented – Effectively simulating kernel – Months of effort! – Difficult to determine false positives

• New approach: Use well defined interface

– System calls interface – Hardware abstraction layer – Effectively running entire kernel in CMC!

Results

• 4 bugs found
• Metric to measure coverage:

– Line Coverage: Lines of code covered – Protocol Coverage: %age of abstract protocol

behavior covered

Description Line Coverage Protocol Coverage Bugs Standard server, client 47.4 % 64.7 % 2 + Simultaneous connect 51.0 % 66.7% + Partial Close 52.7 % 79.5% 2 + Message corruption 50.6% 84.3% Combined Coverage 55.4% 92.1%

Conclusions

• No model is as good as implementation itself

– Simplification , omission missing critical errors

→

• Manual work scale badly for complex systems
• Use well defined interfaces
• More code coverage More bugs found

→

My Opinion

• Usage of live software project is a big plus
• Meta-Compilation & CMC give huge advantage
• Better comparison if same tools used across

case studies

• Time & Effort not measured accurately
• No comparison with other existing tools

Future Scope

• How to increase code coverage in meta

compilation and CMC?

• Analysis of class of properties checked

References

1)Experiences using static analysis & model checking for bug finding – Dawson Engler and Madanlal Masuvathi 2)A simple method for extracting models from protocol code – David Lie, Andy Chou, Dawson Engler, David L. Dill 3)Stanford FLASH Multiprocessor: Status, Some Lessons and Plans – John Hennessy

Q & A