"No Silver Bullet? Silver Buckshot May Work" Presented - - PDF document

KT3

Keynote 11/10/2011 12:45:00 PM

"No Silver Bullet? Silver Buckshot May Work"

Presented by: Gregory Pope Law rence Livermore National Laboratory

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073 888‐268‐8770 ∙ 904‐278‐0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Gregory Pope Law rence Livermore National Laboratory

With more than forty years of experience developing software in the commercial and government sectors, Gregory Pope currently works for the Lawrence Livermore National Laboratory as a software quality engineering group leader, and verification and validation project leader for advanced simulation. Previously, Greg founded and ran a software testing company, patented automated software testing tools, and held management and technical positions involving mission-critical testing of military systems and development of software code for avionics and aerospace uses. Greg has given industry keynote addresses, written technical papers, taught on software quality internationally, and been a consultant.

9/28/2011 1

• Gregory Pope

SQE Group Leader

• This work performed under the auspices of the U.S. Department of Energy by

Lawrence Livermore National Laboratory under Contract DE$AC52$07NA27344

• LLNL$PRES$493892 $ Draft
• The proverbial Silver Bullet

Most common request Definition of Better Good ideas reincarnate Some modern challenges Buckshot – Common problems and solutions

9/28/2011 2

• LLNL$PRES$493892 $ Draft
• The phrase typically

appears with an expectation that some new technological development or practice will easily cure a major prevailing problem.

• LLNL$PRES$493892 $ Draft
• High Level Language $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ c. 1968

Top Down Structured Programming $$$$$$$$$$$$$$$$$$$$c. 1974 Waterfall Lifecycle $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ c. 1976 DoD Standard 2167A $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ c. 1988 Computer Aided Software Engineering (CASE)$$$$$ c. 1990 Object Oriented Design and Programming $$$$$$$$$$ c. 1992 CMM/CMMI $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ c. 1993 Rational Unified Process $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ c. 1996 Automated Testing Tools $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ c. 1995 Continuous Integration $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ c. 1999

9/28/2011 3

• LLNL$PRES$493892 $ Draft
• Increased productivity

Less spaghetti code In process error detection, iterative, incremental spins More documentation is not the answer $ RUP, artifacts Frameworks and plug ins $ Eclipse, Team Forge Reusability $ Java, Python, Ruby, C# Regulated vendor minimum competencies, Outsource Expect requirements to change $ Agile Nightly regression testing – Junit, Subversion, QTP Defects detected earlier – CMAKE, BuildBot, Jenkins

• LLNL$PRES$493892 $ Draft
• !"

What can we do to make our process better?

9/28/2011 4

• LLNL$PRES$493892 $ Draft
• #
• A. Make the software better $$$$$$$$$$$$$$$$$$$$ c. 1970
• B. Make the user experience better$$$$$$$$$$$$c. 1990
• C. Make the developer experience better$$$$$c. 2005

Better = A & B & C

• LLNL$PRES$493892 $ Draft
• If the improvements do not make the developers life

better, they will probably not be easily adopted. Dah Examples:

Integrated Development Environments (IDEs) Networking Static analysis Continuous integration Distributed Code repositories

9/28/2011 5

• LLNL$PRES$493892 $ Draft
• !$%&'(

Good idea, but we are busy, we will do it later.

• LLNL$PRES$493892 $ Draft
• )*+

The “land of later” is a mythical place where nothing ever happens. Why? Because if you are successful you will be whisked

• ff to a new project.

If you unsuccessful you will be whisked off.

9/28/2011 6

• LLNL$PRES$493892 $ Draft
• ,-

Debug and try it Write unit test drivers and stubs manually Stand alone framework to write unit tests Automated tool (Automated Regression Tests to run periodically) Automated the Automated tool, run unit tests when a code change is detected

• LLNL$PRES$493892 $ Draft
• !

Leave coding styles up to individuals Ask team to follow a coding standard Have a plug in tool that formats the code to the correct style (i.e. indents) in the IDE Have a static analyzer that checks for style violations

9/28/2011 7

• LLNL$PRES$493892 $ Draft
• $&

No peer review of code Meeting to peer review code Meeting to discuss findings in code Collaborative tool to allow code review Collaborative tool with built$in diff and tracker interface (to requirements and bugs) and CM tool

• LLNL$PRES$493892 $ Draft
• )
• Compiler, debugging, and testing to find bugs
• Static analysis on integrated code
• New rules added to reduce false alarms
• Static analysis on code as it is built
• Automated static analysis on check in and nightly
• Automate the automated static analysis emailing only new issues found

9/28/2011 8

• LLNL$PRES$493892 $ Draft
• %!.

Cost to manually test – 4 hours per bug* Cost of automated test – 1 hour per bug* Cost of static analysis – 10 minutes per bug** * Cost to design tests (scripts) and execute ** Cost to build and triage code

Source: William Oliver LLNL “Quantifying the Value of Static Analysis”, Starwest 2011

• LLNL$PRES$493892 $ Draft
• Source: Boris Beizer Software System Testing and Quality Assurance, page 34

All bugs are not structuralLL But some structural bugs cause system, functional, and data bugs.

)

Structural Bugs 27% Functional Bugs 27%

15%

System 16% Data, Code, Other 30%

15% 15%

9/28/2011 9

• LLNL$PRES$493892 $ Draft
• ##

Study from 1927 to 1932 at a Western Electric Company Plant in Cicero, Illinois by Harvard Researcher Elton Mayo. One reasonable conclusion is that the workers were pleased to receive attention from the researchers who expressed an interest in them. Any new tool or process can cause process improvement.

• LLNL$PRES$493892 $ Draft
• % &

Built$in$test for avionics $$$$$$ c. 1972 Design by Contract Assertions

9/28/2011 10

• LLNL$PRES$493892 $ Draft
• % &

Mission critical software $$$ c. 1972 Test Driven Design Pairs Programming

Missile Simulator Software New Software RWR Software

• LLNL$PRES$493892 $ Draft
• % &

ITT 1978 “stand up” meetings Accomplished, Plan to Accomplish, Obstacles Five minutes in length, can not sit down All decision makers present Scrum meeting “Main purpose of SCRUM is to find problems early”. Assumption is if a problem is identified early it will get fixed early. Not for all however LLL

Source: Lunch with Ken Schwaber 2007

9/28/2011 11

• LLNL$PRES$493892 $ Draft
• !&,

The XYZ project is ahead of schedule and under budget. What do you mean the software isn't ready for test yet?

• OK. You've got to
• finish. Give it to test

for a couple of days. We have got to add a month to this schedule. Admits there is a problem. We should have time to test if we get the software on schedule. You can't use our test schedule to keep writing new code. We can't even begin to test XYZ, let alone finish, in two days. You can't deliver XYZ without a minimum of testing. Admits the need to automate testing. Let's plan a picnic to celebrate the success$ ful XYZ delivery. Perhaps we should delay the picnic by a week or so. Perhaps we should delay the picnic by a month. Refers Test Manager to Employee Assistance Program. Finds job for old program manager. Admits self to Employee Assistance Program. Trust me. We just have a few minor functions to add. That last function required changing 50%

• f our design. Sorry.

Tomorrow, for certain, will be the day we give it to test. We didn't know you wanted all functions in the first version. Admits the need to plan based on prior experiences. XYZ will increase our sales by 50% next quarter. XYZ will be even better than we first thought. XYZ is so good, test hasn't even found any bugs in it. What do you mean it's late and doesn't work? We'll lose market share! Admits the need to keep in closer contact with engineering. Denial Anger Depression Bargaining Acceptance XYZ PROGRAM MANAGER TEST$QA MANAGER HR MANAGER ENGINEERING MANAGER MARKETING MANAGER DEPT STAGE

Get to acceptance faster LL

• LLNL$PRES$493892 $ Draft
• !!

Prototypes become products, options: Start over Refactor Wrappers − Disable features − Input checking − Disable inputs Testing in Domain of Interest

9/28/2011 12

• LLNL$PRES$493892 $ Draft
• !!
• Code Security

Static Analysis Security Testing C, C++ written before there was an internet * Security not a function or test, it is an intrinsic value*

• Optimistic Schedules

Estimation Tools Acceleration Release Prediction Tools

* Source: James Gosling, inventor Java, speaking at Sandia Labs 8/18/2011

• LLNL$PRES$493892 $ Draft
• !!

More Code Writers

Automated workflow

Text Books

Some Examples

Understanding Requirements

Some thoughts

Good Design

Some thoughts

9/28/2011 13

• LLNL$PRES$493892 $ Draft
• !!

Buggy Text Book Code

Type first integer : 5 Type second integer : 6 Type third integer : 7 5 is in the range 30 to 199 : FALSE at least two integers you typed are equal : FALSE 5 != 6 : TRUE NOT (5 < 6) : FALSE 5 <= 6 : TRUE 7 > 6 : FALSE (7 = 5) AND (6 != 7) : FALSE (7 <= 5) XOR (6 >= 7) : FALSE (7 > 5) AND (6 <= 7) : TRUE Polarity

• LLNL$PRES$493892 $ Draft
• !

Please enter January 1's starting day; A 0 indicates January 1 is on a Monday, A 1 indicates January 1 is on a Tuesday, etc: 5 Enter the year you want the calender generated: 2011 The calendar for the year 2011 January Sun Mon Tue Wed Thu Fri Sat $$$ $$$ $$$ $$$ $$$ $$$ $$$ 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 November Sun Mon Tue Wed Thu Fri Sat $$$ $$$ $$$ $$$ $$$ $$$ $$$ 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

Change string to Sunday and Monday Spelling

9/28/2011 14

• LLNL$PRES$493892 $ Draft
• Tracking Tool: $ Collabnet
• LLNL$PRES$493892 $ Draft
• &/

The real answers are 6 levels down Who What Where Why How Who Dynamic tracing to code and tests

9/28/2011 15

• LLNL$PRES$493892 $ Draft
• 0.&/

Usability Maintainability Scalability Availability Extensibility Security Portability Safety

• LLNL$PRES$493892 $ Draft
• &/

Prototypes help elicit requirements Good question asking skills Experience Can be too many or too few Requirement writers make good test designers.

9/28/2011 16

• LLNL$PRES$493892 $ Draft
• 1 &/

Never and Always LLLLLLLLL... Talk to the help desk folks Testing not needed for reuse (fail) Sit in the user’s seat

• LLNL$PRES$493892 $ Draft
• Emerging

Chaos Chaos Perceived Order Emerging Problem Solving Chaos Emerging Order Order Deliberate Problem Seeking

Instead of trying to make order exist initially, chaos is allowed to exist so all stakeholders can problem seek together. Emerging chaos closely resembles the waterfall

• r V$model, an attempt to capture all (or most)
• f the software requirements at the outset

(create apparent order).

Source: Moshe F. Rubinstein and Iris R. Firstenberg, University of California, Los Angeles, John Wiley and Sons, pg. 91

&/!#!

9/28/2011 17

• LLNL$PRES$493892 $ Draft
• #2

Hard to get developers to do this Peer Pressure Manager Pressure Director Pressure Owner Pressure Even when done the design is seldom updated

• LLNL$PRES$493892 $ Draft
• !2

The source code is an abstraction of the executing binaries. It is a detailed design Source Code is like a blueprint is to a structure Architecture Design is important, like a scale model is to a structure.

9/28/2011 18

• LLNL$PRES$493892 $ Draft
• #

Most safety problems with software intensive systems did not involve the software failing. Most safety problems were the interaction of the software components with other system components. Reliability is not the same as safety in software. Software does not fail the same as hardware.

Source: Nancy Levenson,

• LLNL$PRES$493892 $ Draft
• !+

Weak coding practices allow hackers to exploit code Examples: Array bounds overflow Memory Leaks Tainted Inputs Null Pointer Dereference Uninitialized Variables Use tools to find this stuff So Better supports more Secure

9/28/2011 19

• LLNL$PRES$493892 $ Draft
• #

Graded approach to rigor In general: The amount of rigor used for software process is proportional to the severity of the consequence of failure of the software being built. Managed, Documented, Understood

• LLNL$PRES$493892 $ Draft
• 23 4)*)

Running out of memory again: Power consumption for an exascale (1018 flops) system in 2018 even under optimistic assumptions would be 100–200 MW, $100 Million annual electric bill. Memory reliability again Cosmic radiation has a much higher probability of disturbing RAM memory.

Title Source: Yogi Berra, New York Yankees http://www.lbl.gov/CS/html/SC08ExascalePowerWorkshop/gara.pdf Yuan Chen, Jet Propulsion Laboratory, Pasadena, California

9/28/2011 20

• LLNL$PRES$493892 $ Draft
• )#2

%2##

Generational Cultural Industrial Geographical Language

• LLNL$PRES$493892 $ Draft
• 1.

Matures 1933$1945 Boomers 1946$1964 Gen X 1965$1976 Gen Y 1977$1998 34 Million 76 Million 41 Million 75 Million

Source: Love’Em or Lose’Em, Kay and Jordan$Evans, page 236

9/28/2011 21

• LLNL$PRES$493892 $ Draft
• 1!

Matures Boomers Generation X Generation Y Work Ethic

Work 'til you drop. Work long hours and tell your manager about it. Personal life first, work is important. Lifestyle comes first.

Loyality

Loyal to employer. Loyal to employer, with reservations. Career and professional loyalty. Career options.

Technology

Technology fascination. Technology challanged. Technology proficient. Technology immersed.

Reporting Relationships

Strong chain of command. Chain of command. What is the purpose of a chain of command? Be respectful but move ahead.

Source: !" !", Kay and Jordan$Evans, page 245

• LLNL$PRES$493892 $ Draft
• !2##

Globalization makes this more critical Teaching in Asia Questions written Prescriptive approach Teaching in Israel Highly Interactive Principle based approach Teaching in California

9/28/2011 22

• LLNL$PRES$493892 $ Draft
• !

"!#""!#! $!%& #' (# ( ("%&) (* !&&$#! (%% *"! +!%!$!# !( (, (* !+#($!%) (* %-!"#

• !.!

#*&&!/$!% (%%#% "!##+(!$!%0#" %(#+(# (&#+( !% # ( ) %"-%"+ !/%#+-%"+ ## ("&$ %&!$" $!%*#" +#+#"!""!!"

• %+

'!!(/!#"#&$#&" ! $!%+!&!+#((!+) & ($&#+ !(& (# (+&) &-%#&"!(!# ('$!%! (#+(#% (!"!$!%&-!%() &0 +! % !(# ! (' !-#+#+(# !-&!$#! ##&$%" "($$!%&( $ (* & $ %-!! ! #"$!% !% & ((# #($!% #(/!#"#$!% ! %#/!!#! (# ) (/* "$ ! ("%& ! &#! ! %&#+!# ##! "!"# #(# &#! (# &"&$# %#""% #!'

• LLNL$PRES$493892 $ Draft
• %2##

Commercial System Military MIS Outsourced

Source: Capers Jones, #$%$$ Source: Gregory Pope, &$$$"

Many SQA practices are not portable across industries, i.e. Military to Commercial

9/28/2011 23

• LLNL$PRES$493892 $ Draft
• #2$)$

Know who our customers are and meet or exceed their expectations with valuable software systems that are supportable. Understand the context in which our software systems will operate as well as the technical constraints and interfaces. Assemble the right team of individuals, give them the environment and support they need, and trust them to get the job done. Support the customer’s expectations throughout the software project life cycle. Pursue technical excellence by continuously preventing, detecting, and removing defects from the software system. Collaborate and coordinate with partners frequently throughout the software project. Create, integrate, and manage the software codes focusing on continuous improvement of the process and products. Demonstrate frequently during development that the software system supports the customer’s expectations and is fully supportable.

Source: Gregory Pope, &$$$"

• LLNL$PRES$493892 $ Draft
• !2-

But you can run all the experiments you want.

9/28/2011 24

• LLNL$PRES$493892 $ Draft
• %+

.-56-7

• LLNL$PRES$493892 $ Draft
• !

"# " "$ %&' %(') %* +! +, +- .$' . .- /0 /, /$ 10 12 3(') 3$ , 4

28 #1

9/28/2011 25

• LLNL$PRES$493892 $ Draft
• $

, !) 4)' "- ", %) %,) +' +, .0 ., /' /, 1 1, 3&55 3 4

• 28!% -
• LLNL$PRES$493892 $ Draft
• 6!78877

)'87' 9:78): 2 - '0,) 44)'!7 87'7)) 7 :8777)8' :87 7 0 (')$;

&+

Source: Gregory Pope, &

9/28/2011 26

• LLNL$PRES$493892 $ Draft
• Harley Davidson

Cold Stone Mongolian Barbeque Oakley Risk Based Graded Approach Crystal Let Developers Pick Solutions

• LLNL$PRES$493892 $ Draft

9/28/2011 27

• LLNL$PRES$493892 $ Draft
• )
• 2
• "

345 67

• 82
• 000 Buck

.36 6 00 Buck .33 8 0 Buck .32 9 1 Buck .30 11 2 Buck .27 15 3 Buck .25 19 4 Buck .24 21 BB .18 50 2 Bird .148 90 4 Bird .129 135 5 Bird .12 170 6 Bird .109 225

• LLNL$PRES$493892 $ Draft
• ) %#-

Scrum Feature List Acceleration Sprints User Stories Pairs Programming Test Driven Design Automated Unit / System Testing Continuous Integration

9/28/2011 28

• LLNL$PRES$493892 $ Draft
• Determine problems, communicate with:

Developers Testers Stakeholders, Users Marketing Help Desk Determine problems Look up problems to determine solutions Load your “shells” with these solutions

• LLNL$PRES$493892 $ Draft
• #

!

9/28/2011 29

• LLNL$PRES$493892 $ Draft
• 2
• LLNL$PRES$493892 $ Draft
• 29 $

9/28/2011 30

• LLNL$PRES$493892 $ Draft
• 29
• LLNL$PRES$493892 $ Draft
• $

9/28/2011 31

• LLNL$PRES$493892 $ Draft
• $2#

51 52 64 28 11 17 22 64 13 28 15 30 16 10 20 30 40 50 60 70

94:;

Prevention

• LLNL$PRES$493892 $ Draft
• 22#

7 41 9 33 41 19 18 13 31 41 13 6 7 6 7 20 24 7 41 4 33 17 7 5 10 15 20 25 30 35 40 45

"44:;

Detection

9/28/2011 32

• LLNL$PRES$493892 $ Draft
• !

The proverbial Silver Bullet

No cure alls, but each has contributed

Most common request

How to make things better, cheaper, faster

Definition of Better

Better for everyone involved

Good ideas reincarnate

• New names, new groupings

Some modern challenges

Security, Complexity, Scale, Safety

Silver Buckshot – Common problems and solutions

Buckshot Wiki, expert system on a Wiki