T9 May 15, 2003 11:30 AM T HE T ESTING H OLODECK : A FREE T OOL FOR - PDF document

BIO PRESENTATION T9 May 15, 2003 11:30 AM T HE T ESTING H OLODECK : A FREE T OOL FOR R UNTIME F AULT I NJECTION James Whittaker Florida Institute of Technology International Conference On Software Testing Analysis & Review May 12-16, 2003 Orlando, FL USA

James Whittaker James A. Whittaker is a professor of computer science at the Florida Institute of Technology. He earned his Ph.D. in computer science from the University of Tennessee in 1992. His research interests are software testing, software security, software vulnerability testing and anti cyber warfare technology. He is the author of How to Break Software (Addison-Wesley, 2002) and over 50 peer-reviewed papers on software development and computer security. He holds patents on various inventions in software testing and defensive security applications and has attracted millions in funding, sponsorship and license agreements while a professor at Florida Tech. He also has served as a testing and security consultant for Microsoft, IBM and many more US companies. His research team at Florida Tech is known for its testing technologies and tools, which include the highly acclaimed runtime fault injection tool Holodeck . His research group is also well known for their development of exploits against software security, including cracking encryption, passwords and infiltrating protected networks via novel attacks against software defenses.

Holodeck: A Tool for Runtime Fault Injection James A. Whittaker

Outline of Presentation • Show some cool bugs • Understand what causes these bugs • Talk about using this understanding to become better testers • Demo a new tool that allows you to develop this understanding and put it into practice

First, Let’s Look at Some Bugs… • Excel crash demo…can you figure out why? • PowerPoint crash demo…can you figure out why? • Excel turns off the keyboard…can you figure out why? • Digital rights management can be bypassed…can you figure out why?

Now, Let’s Think About… • What testing are you doing now that you could do better? – Are you missing important bugs? • What testing aren’t you doing because you can’t? – What functionality are you not getting to?

We Need More Information • We need to understand the environment in which our applications execute • We want to maximize code and data coverage during testing • We want to make failures easier to reproduce and faults easier to debug • We need to understand how security (and other important features) are architected

So how do we accomplish all this? Enter: the software Holodeck!

We Need to Understand Our Application’s Environment • What are our app’s dependencies and how does it interact with its local resources? • Can other app’s break our app? • How fragile is our application with respect to environmental variation and stress?

We Want to Maximize Coverage • Are we executing all the exceptions? • Are we seeing all the error dialogs? • How can we (easily) test invalid input and unexpected return values?

We Want to Make Failures Easier to Reproduce • What behaviors lead up to a failure? • How can we reproduce stress/fault based bugs? • Can we collect information that will make software easier to debug?

We Need to Understand How Security is Implemented • What parts of the environment is our application trusting explicitly? • How are secrets being stored and manipulated? • How do we detect security problems even when they have invisible behavior?

Demo of the Ultimate Testing Tool: Holodeck