Level Up Quality, Security, & Safety Todd L. Montgomery - - PowerPoint PPT Presentation

▶
level up
SMART_READER_LITE
LIVE PREVIEW

Level Up Quality, Security, & Safety Todd L. Montgomery - - PowerPoint PPT Presentation

Aug 27, 2023 375 likes •1.44k views

StoneTor Level Up Quality, Security, & Safety Todd L. Montgomery @toddlmontgomery 1986 https://www.nap.edu/catalog/10616/post-challenger-evaluation-of-space-shuttle-risk-assessment-and-management IV&V Independent Verification

slide-1
SLIDE 1

Level Up

Quality, Security, & Safety

Todd L. Montgomery @toddlmontgomery

StoneTor

slide-2
SLIDE 2

1986…

slide-3
SLIDE 3
slide-4
SLIDE 4
slide-5
SLIDE 5

https://www.nap.edu/catalog/10616/post-challenger-evaluation-of-space-shuttle-risk-assessment-and-management

slide-6
SLIDE 6
slide-7
SLIDE 7

IV&V

Independent Verification & Validation

slide-8
SLIDE 8
slide-9
SLIDE 9

“Safety is a system property, not a component property, and must be controlled at the system level, not the component level.”

― Nancy G. Leveson, Engineering a Safer World: Systems Thinking Applied to Safety

slide-10
SLIDE 10

Software Safety + Mission Safety

slide-11
SLIDE 11
slide-12
SLIDE 12
slide-13
SLIDE 13
slide-14
SLIDE 14

Small Teams - 1-2 or 5-6 Small Budgets - if any Typically…

slide-15
SLIDE 15

Not just when people could die…

slide-16
SLIDE 16
slide-17
SLIDE 17
slide-18
SLIDE 18
slide-19
SLIDE 19

More relevant every day

slide-20
SLIDE 20

Tax Day 2018

https://www.vox.com/2018/4/17/17247614/irs-glitch-2018-delays

slide-21
SLIDE 21

Security == Quality Quality == Security

slide-22
SLIDE 22
slide-23
SLIDE 23
slide-24
SLIDE 24

https://en.wikipedia.org/wiki/List_of_data_breaches

slide-25
SLIDE 25

https://en.wikipedia.org/wiki/List_of_data_breaches

slide-26
SLIDE 26

Don’t Worry … The Cloud

slide-27
SLIDE 27

Don’t Worry … The Cloud

slide-28
SLIDE 28

Don’t Worry … The Cloud

slide-29
SLIDE 29

Software Quality

slide-30
SLIDE 30

Most people don’t expect software to work*

* - Seriously, ask others about this…

slide-31
SLIDE 31

Large Projects

slide-32
SLIDE 32

Software Project Success Rates

  • Standish Group Chaos Report 2010

Successful: 32% Challenged: 44% Failure: 24%

slide-33
SLIDE 33

EULAs

slide-34
SLIDE 34

Software Quality

slide-35
SLIDE 35

Systems meeting Functional Requirements

slide-36
SLIDE 36
slide-37
SLIDE 37
slide-38
SLIDE 38

“Non”-Functional Requirements?

slide-39
SLIDE 39

Security Performance Quality Robustness Safety Stability Usability

https://en.wikipedia.org/wiki/Non-functional_requirement

slide-40
SLIDE 40

When not met is the system not “Non-Functional”?

slide-41
SLIDE 41

https://en.wikipedia.org/wiki/Non-functional_requirement

slide-42
SLIDE 42

“Non”-Functional Requirements Are Unspoken / Incomplete Functional Requirements

slide-43
SLIDE 43

Quality, Security, Safety At best, afterthoughts!

slide-44
SLIDE 44

Quality isn’t an Issue … Until it (suddenly) is

s/Quality/Security… s/Quality/Scalability…

slide-45
SLIDE 45

“What could possibly go wrong?”

slide-46
SLIDE 46

“Oh… It went wrong… what now?!”

slide-47
SLIDE 47

“Throw testers at it!” “Patch/Hot Fix!” “Source Code Reviews” “Bug Triage Meetings”

slide-48
SLIDE 48
slide-49
SLIDE 49

It can help… But, by itself, NOTHING is magic

slide-50
SLIDE 50

More “What Quality ISN’T”

* - Based on ROI from 100s of NASA projects & personal experiences

slide-51
SLIDE 51

Agile/Waterfall/Scrum/etc. Language Choice Framework/Library Choice Formal Methods Functional vs. Imperative/OOP Reference Enterprise Architecture Vendor Stack

slide-52
SLIDE 52
slide-53
SLIDE 53
slide-54
SLIDE 54

… “never” / “always” Recruiting Process Web Scale Architecture Distributed Algorithm AI / Big Data / IoT / Reactive / OSS / …

slide-55
SLIDE 55

… Source Code Reviews Bug Triage Meetings Code Coverage Comprehensive Test Plan

slide-56
SLIDE 56

Dogma is Anathema to Quality

slide-57
SLIDE 57
slide-58
SLIDE 58

Languages are Inspirational Java, C/C++, C#, Python, Erlang, OCaml, Ada, etc. Even JavaScript…

slide-59
SLIDE 59

Technologies are Inspirational Microservices, Reactive, SOA, CRDTs, Blockchain, etc. Even ORM, DI, RPC, …

slide-60
SLIDE 60

Case Studies

* - Even when you do everything else well…OK…

slide-61
SLIDE 61

100% code coverage

slide-62
SLIDE 62
slide-63
SLIDE 63

Error Handling (is hard)

slide-64
SLIDE 64

Create File File Sender Process Process File Directory File Receiver Process Bus/Network File Processor

slide-65
SLIDE 65

// create empty file // check params // fill file return 0; error: unlink(file); return -1;

In C

slide-66
SLIDE 66

try { // create empty file // check params // fill file } catch (final Exception ex) { file.delete(); }

And Java

slide-67
SLIDE 67

Keeping up-to-date

slide-68
SLIDE 68

@toddlmontgomery

Request Response Service

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-------------------------------+ | Version |R|S| Flags | ... +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-------------------------------+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-------------------------------+ | Version |X|0| Flags | ... +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-------------------------------+

slide-69
SLIDE 69

@toddlmontgomery

Request(New) Response Service (Old)

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-------------------------------+ | Version |0|1| Flags | ... +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-------------------------------+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-------------------------------+ | Version | Flags | ... +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-------------------------------+

“I know nothing of your … logic” “Service must have Q=1 now!”

slide-70
SLIDE 70

Async all the things!

slide-71
SLIDE 71

@toddlmontgomery

Request Response Block La-La … La-La Request Block Service …? Service Request Timeout!!! Service Timeout!!!

slide-72
SLIDE 72

@toddlmontgomery

Request Response La-La … La-La Request Service …? Service Request Timeout!!! Service Timeout!!!

slide-73
SLIDE 73

@toddlmontgomery

Request Service …? Request Timeout!!! Service Timeout!!! Request Response La-La … La-La Level 2!!!

slide-74
SLIDE 74

Root Cause Matters

slide-75
SLIDE 75

Bet I can Exploit That

slide-76
SLIDE 76

Don’t keep things around!

slide-77
SLIDE 77

https://en.wikipedia.org/wiki/Row_hammer

slide-78
SLIDE 78

https://arstechnica.com/information-technology/2019/12/scientists-pluck-crypto-keys-from-intels-sgx-by-tweaking-cpu-voltage/

slide-79
SLIDE 79

“But it doesn’t have to be fast”

slide-80
SLIDE 80

“But it doesn’t have to be fast” Doesn’t have to be SLOW

slide-81
SLIDE 81

https://www.forbes.com/sites/forbestechcouncil/2017/12/15/why-energy-is-a-big-and-rapidly-growing-problem-for-data-centers/#344456665a30 https://www.datacenterdynamics.com/opinions/power-consumption-data-centers-global-problem/ https://www.nature.com/articles/d41586-018-06610-y

slide-82
SLIDE 82

https://en.wikipedia.org/wiki/Wirth%27s_law

Software is getting slower more rapidly than hardware is becoming faster

— Niklaus Wirth, "A Plea for Lean Software”, 1995

slide-83
SLIDE 83

What Quality IS

slide-84
SLIDE 84

One uncomfortable truth…

slide-85
SLIDE 85

Pride Ownership Responsibility One (un)comfortable truth…

slide-86
SLIDE 86
slide-87
SLIDE 87

“It does this…” “I do this…” We do this…

slide-88
SLIDE 88
slide-89
SLIDE 89

Putt's Law: "Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand.” Putt's Corollary: "Every technical hierarchy, in time, develops a competence inversion." with incompetence being "flushed out of the lower levels" of a technocratic hierarchy, ensuring that technically competent people remain directly in charge of the actual technology while those without technical competence move into management.

https://en.wikipedia.org/wiki/Putt's_Law_and_the_Successful_Technocrat

slide-90
SLIDE 90

We take responsibility, or…

slide-91
SLIDE 91

https://www.nytimes.com/2017/08/25/business/volkswagen-engineer-prison-diesel-cheating.html

slide-92
SLIDE 92
slide-93
SLIDE 93

Pointy-Haired Boss Takeaway

slide-94
SLIDE 94

ROI Return on Investment

slide-95
SLIDE 95
  • https://www.nasa.gov/sites/default/files/1-4a-ivv_conference_bob_hunt_dulos_kalman.pdf
  • https://www.nasa.gov/centers/ivv/dynamic_analysis_roi.html
  • https://www.nasa.gov/sites/default/files/166681main_NASA_Annual_Report_2005.pdf
  • https://www.nasa.gov/sites/default/files/1-1a-nasa_workshop_-_measuring_ivv_roi_-_greendart.pdf
  • http://www.iceaaonline.com/ready/wp-content/uploads/2014/03/Software-Test-Cost-and-ROI-Galorath-

Feb-14-Hunt.pdf

  • https://www.researchgate.net/publication/

220845858_Estimating_direct_return_on_investment_of_independent_verification_and_validation

  • http://dau.dodlive.mil/2014/12/19/the-path-to-software-cost-control/
  • https://books.google.com/books?

id=6LcpBgAAQBAJ&pg=PA114&lpg=PA114&dq=return+on+investment+nasa+ivv&source=bl&ots=nwVR Bx47aO&sig=1YEozo_huDJL4QfGl_5BgkFQou4&hl=en&sa=X&ved=0ahUKEwj6- Lanx5PWAhWGjlQKHTqCAcUQ6AEITzAI#v=onepage&q=return%20on%20investment%20nasa%20ivv &f=false

  • http://media.govtech.net/Events/2006Events/2006Pennsylvania/

6_230_IndependentValidationandVerificationIVV_HURLEY.ppt

  • http://catal0g.info/downloads/nasa-iv-v-metrics.pdf

ROI

slide-96
SLIDE 96

Allen Nikora https://scholar.google.com/citations?user=K-c0IEUAAAAJ&hl=en Leslie Lamport https://lamport.azurewebsites.net/pubs/pubs.html Martin Feather https://trs.jpl.nasa.gov/browse?value=Feather%2C+Martin+S.&type=author http://www.sciencedirect.com/science/article/pii/S1877050914001124

slide-97
SLIDE 97

Specifications as Communication Early Requirements Analysis Early Domain Expertise Culture of Accountability

slide-98
SLIDE 98

Some Suggestions

slide-99
SLIDE 99

TDD/BDD - The Spec? Agile - Intent! Language - Rise Above! Functional/OOP - Rise Above! Dogma - holds you back

slide-100
SLIDE 100
slide-101
SLIDE 101

The Spec is Nothing The writing of the Spec is EVERYTHING

slide-102
SLIDE 102
slide-103
SLIDE 103

Develop Good Taste Taste Matters

slide-104
SLIDE 104
slide-105
SLIDE 105

Twitter: @toddlmontgomery

Thank You!

Questions?

StoneTor