T6 August 22, 2003 2:00PM R EDUCE R ISK U SING S ECURITY QA A - - PDF document

BIO PRESENTATION

International Conference On Software Test Automation August 19-22, 2003 Boston, MA USA

T6

August 22, 2003 2:00PM

REDUCE RISK USING SECURITY QA AUTOMATION TECHNIQUES

Alexander Mouldovan Cenzic Inc

Alexander Mouldovan

As Director of Product Marketing for Cenzic, Alexander Mouldovan is responsible for the definition of Cenzic’s award-winning security QA platform, Hailstorm. He has developed and tested innovative software and hardware for trusted email, digital rights management, security, scanner, and pen computing applications.

Reduce Risk Using Security QA Automation

Alexander Mouldovan Cenzic Inc. Presented to Test Automation March, 2003 Update Available at: http://www.cenzic.com/CenzicTASlides.ppt

• 12+ years in software

– 4 years product management – 5 years as developer – 4 years testing and evaluating software and hardware products

• Security Background
• The Problem
• Vulnerabilities (some examples)
• Some solutions
• Security QA
• Opportunities

Outline

Introduction

The Problem

• Evolution of security

–Security by obscurity

• Obscurity doesn’t last

–Hardened perimeters

• “Crunchy on the outside, chewy on the

inside”

Evolution

–Perimeter security – firewall, IDS, …

• Have to let data through

–Web services

• Built to tunnel through firewalls!
• Functionality over security

– Security is the #1 reason for not adopting WS

–Applications: the final frontier

• WIDE OPEN!!!

Security Technology

• Firewall, IDS, Intrusion Prevention
• Access management, encryption
• Scanners (Nessus)
• Anti-Virus (Symantec)
• Integrated frameworks (Symantec, CA)

The Weakest Link

• A system is only as secure as the least

secure link

• Now that perimeter security has matured,

Application logic is the weakest link

• Gartner Group, 2002

– "However, close to 75% of today's attacks are tunneling through applications.” – John Pescatore, Security Analyst, Gartner Group

Vulnerabilities

Common Vulnerabilities

• Buffer Overflow
• Buffer Underflow
• SQL Parser
• SQL Disclosure
• Command Insertion
• Path traversal
• Format String Tests
• Data Input Validation
• Random input
• Session ID Hijacking
• Parameter tampering
• Cross site scripting
• Privilege escalation
• Alternate Encodings

SQL Parser Attack

SQL Disclosure Attack

Where are these Vulnerabilities Found?

• Vendors’ platforms

– Databases – Appservers – Network equipment – Messaging platforms – Operating systems

• Your custom code

– ASP – JSP – Java beans – CGI, PHP – ISAPI – .Net – Java – C/C++

Severity of a Defect

• Not all bugs are created equal
• Compare:

– Cosmetic bug – Bug that prevents access to functionality – Bug that crashes the server – Bug that reveals customer data

• Credit cards numbers
• Passwords

• Unmanned carts on a track
• Bad failure recovery/detection

– Piles of fallen bags would not stop the unloaders

• Carts got out of sync

– Full carts continue to get loaded – Empty carts get unloaded

• Delayed airport opening for 11 months

– $1 million dollars a day in cost due to interest bond issues

Denver Airport Baggage

What are Companies Doing?

Finding Security Vulnerability Early Saves Engineering Time and Money

Source: National Institute of Standards and Technology

2 4 6 8 10 12 14 16

Requirements Code/Unit Test Integration Beta Test Post Prod'n

Hours per bug fix

0% 10% 20% 30% 40% 50%

Requirements Code/Unit Test Integration Beta Test Post Prod'n

Where bugs are found

Outsourced Testing

Penetration tests:

• Costly (ex. small engagements run ~$500,000)
• Time consuming
• Black box
• Human driven, not repeatable
• Run against production applications

Stretching Scarce Resources

• Small internal security groups

– Don’t scale, skill levels vary widely – Delay shipment, pressure to ship with holes – Estimated cost to fix a security problem after deployment: $100,000

Nothing

– Live with risk and hacks – Rely on “Damage Control” – Over 90% of companies surveyed detected security breaches with over 80% incurring financial losses as a result… * – 44% were willing/able to quantify their losses… averaging over $2M/yr per respondent*

*Computer Crime and Security Survey, Computer Security Institute 4/02

Relative Costs

• Hacked System

– $1,000,000/hour of downtime*

• Fix bugs after deployment

– $100,000 per bug in finished product

• Outsourced testing

– $10,000 per function point

• Automated Security QA

– < $1,000 per defect

Cost per Incident Phase

Downtime Due to Hacks Scanning at Production Outsource Testing Automation Design Implement Integration Production

* Financial Services

Can We Do Better?

Testing Approaches

• Black Box

– Deep coverage of possible vulnerabilities using broad input set

• White Box

– Analyze source code, designs for flaws

• Grey Box

– Use black box tools with monitoring technology

Testing For Security

• Destructive testing – QA with attitude!

– Try to find weaknesses to exploit

• Need standardized methods, metrics

– Hunt and peck security testing doesn’t reduce risk reliably

• Need to leverages security expertise

within organization

– Most QA teams do not have security backgrounds

Fault Injection

• Critical systems must not fail

– BART tested with fault injection

• Create failures in a controlled environment

– Launching frozen chickens into jet engines to

• bserve failure

– Crash test dummies for safety testing

• Don’t trust your input

– UAL ticket scam for Europe airfares (unchecked text entry)

Fault Injection (2)

• Fault Injection is the ultimate black box

tool – use the most malicious input possible

• Stress the system every way possible prior

to deployment

> This is how you reduce risk.

Fault Detection

• When input was perturbed, did the

application behave correctly?

– Did it give an error page?

• How does the application fail?

– Did it reveal stored data or information about the infrastructure? – How long did it take to respond?

Security QA

Integrating Security and QA

• Security expertise

– Precious resource – Knowledge of vulnerabilities – Can assess severity of vulnerabilities – Can teach secure coding

• Quality Assurance

– Close to the developer – Use repeatable processes – Critical for reliability of applications – Automation skill – Track and manage defects

Security Test Automation

• Create a baseline
• Insert inappropriate data, change timings,

alter state transitions, break assumptions

• Watch for unexpected behavior
• Identify, track, remove issues before

deployment

How Security and QA Collaborate

• Security designs

policies that define security testing

• QA applies policies to

applications in development

– Manually run Fault Injection – Embed Fault Injection in build validation suite

Security Personnel and QA collaboration is critical

• Policy documentation is a

dead-end

– Policies must be codified and implemented

• QA and Security experts

can define appropriate tests for application components

• QA team learns about

Security, Security team gets visibility into compliance

Benefits

Visibility into Risk

Vulnerability Breakdown

Conclusions

Reliability

• Reliability requires:

– Functionality – Performance … and …

Security!

Pain is High

• Companies are ill prepared to deal with

Security Testing

– Spending lots of time on vulnerabilities – Wasting gobs of money

• Chance to impact the bottom line

– Reduce the cost of fixing critical bugs – Get applications to market faster

Hottest Sector in QA: Security!

• Security: the fastest growing segment in

Quality Assurance

• Great opportunity:

– Risk of being hacked is high

• (probability of hack * loss due to hack)

– Not many experts in the field