T9 May 15, 2003 11:30 AM T HE T ESTING H OLODECK : A FREE T OOL FOR - - PDF document

BIO PRESENTATION

International Conference On Software Testing Analysis & Review May 12-16, 2003 Orlando, FL USA

T9

May 15, 2003 11:30 AM

THE TESTING HOLODECK: A FREE T OOL FOR RUNTIME FAULT INJECTION

James Whittaker Florida Institute of Technology

James Whittaker

James A. Whittaker is a professor of computer science at the Florida Institute of

• Technology. He earned his Ph.D. in computer science from the University of Tennessee

in 1992. His research interests are software testing, software security, software vulnerability testing and anti cyber warfare technology. He is the author of How to Break Software (Addison-Wesley, 2002) and over 50 peer-reviewed papers on software development and computer security. He holds patents on various inventions in software testing and defensive security applications and has attracted millions in funding, sponsorship and license agreements while a professor at Florida Tech. He also has served as a testing and security consultant for Microsoft, IBM and many more US

• companies. His research team at Florida Tech is known for its testing technologies and

tools, which include the highly acclaimed runtime fault injection tool Holodeck. His research group is also well known for their development of exploits against software security, including cracking encryption, passwords and infiltrating protected networks via novel attacks against software defenses.

Holodeck: A Tool for Runtime Fault Injection

James A. Whittaker

Outline of Presentation

• Show some cool bugs
• Understand what causes these bugs
• Talk about using this understanding to

become better testers

• Demo a new tool that allows you to develop

this understanding and put it into practice

First, Let’s Look at Some Bugs…

• Excel crash demo…can you figure out

why?

• PowerPoint crash demo…can you figure out

why?

• Excel turns off the keyboard…can you

figure out why?

• Digital rights management can be

bypassed…can you figure out why?

Now, Let’s Think About…

• What testing are you doing now that you

could do better?

– Are you missing important bugs?

• What testing aren’t you doing because you

can’t?

– What functionality are you not getting to?

We Need More Information

• We need to understand the environment in

which our applications execute

• We want to maximize code and data

coverage during testing

• We want to make failures easier to

reproduce and faults easier to debug

• We need to understand how security (and
• ther important features) are architected

So how do we accomplish all this?

Enter: the software Holodeck!

We Need to Understand Our Application’s Environment

• What are our app’s dependencies and how

does it interact with its local resources?

• Can other app’s break our app?
• How fragile is our application with respect

to environmental variation and stress?

We Want to Maximize Coverage

• Are we executing all the exceptions?
• Are we seeing all the error dialogs?
• How can we (easily) test invalid input and

unexpected return values?

We Want to Make Failures Easier to Reproduce

• What behaviors lead up to a failure?
• How can we reproduce stress/fault based

bugs?

• Can we collect information that will make

software easier to debug?

We Need to Understand How Security is Implemented

• What parts of the environment is our

application trusting explicitly?

• How are secrets being stored and

manipulated?

• How do we detect security problems even

when they have invisible behavior?

Demo of the Ultimate Testing Tool: Holodeck