Testing Red Hat Enterprise Linux the Microsoft way Alexander - - PowerPoint PPT Presentation

▶
testing red hat enterprise linux the microsoft way
SMART_READER_LITE
LIVE PREVIEW

Testing Red Hat Enterprise Linux the Microsoft way Alexander - - PowerPoint PPT Presentation

Jun 01, 2023 112 likes •554 views

Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 S What is pairwise testing ISTACON.ORG 16 17 November, Sofja Wheels 19 or 21 Battery (if you remember) 60 kWh, 75 kWh, 85 kWh or 100

slide-1
SLIDE 1

Testing Red Hat Enterprise Linux the Microsoft way

Alexander Todorov

FOSDEM 2018

slide-2
SLIDE 2

S

slide-3
SLIDE 3
slide-4
SLIDE 4

What is pairwise testing

slide-5
SLIDE 5

16 – 17 November, Sofja ISTACON.ORG

slide-6
SLIDE 6
  • Wheels

– 19” or 21”

  • Battery (if you remember)

– 60 kWh, 75 kWh, 85 kWh or 100 kWh

  • Engine

– Single or Dual

  • Performance mode

– Yes or No

slide-7
SLIDE 7

2 x 2 x 2 x 4 == 32 combinations

slide-8
SLIDE 8

Wheels Battery Engine Performance mode 19 “ 60 kWh Single Yes 19 “ 75 kWh Single Yes 19 “ 85 kWh Single Yes 19 “ 100 kWh Single Yes 21 “ 60 kWh Dual No 21 “ 75 kWh Dual No 21 “ 85 kWh Dual No 21 “ 100 kWh Dual No

slide-9
SLIDE 9
slide-10
SLIDE 10

I've pairwise tested Red Hat Enterprise Linux install during the entire test campaign! across all product variants!

slide-11
SLIDE 11

Installation testing 101

slide-12
SLIDE 12

16 – 17 November, Sofja ISTACON.ORG

slide-13
SLIDE 13

16 – 17 November, Sofja ISTACON.ORG

slide-14
SLIDE 14

9 different product variants

I consider them platform independent

slide-15
SLIDE 15

3 test groups: Tier #1, #2 and #3

slide-16
SLIDE 16

6000 test case executions

slide-17
SLIDE 17

“Insanity - doing the same thing

  • ver and over

and expecting different results.”

Albert Einstein

slide-18
SLIDE 18

1) Take all platform dependent tests (pairwise where possible)

slide-19
SLIDE 19

2) Pairwise all tests with parameters

slide-20
SLIDE 20

storage / iSCSI / No authentication / Network init script storage / iSCSI / CHAP authentication / Network Manager storage / iSCSI / Reverse CHAP authentication / Network

  • Authentication type: None, CHAP

, reverse CHAP (3)

  • Networking system: NetworkManager or SysVinit (2)
slide-21
SLIDE 21
  • 3 x 2 == 6
  • Pairwise: 3 x 2 == 6
  • Across all variants: 9 x 3 x 2 == 54
  • Pairwise across all variants: 9 x 3 == 27
slide-22
SLIDE 22

3) Randomize tests without parameters

slide-23
SLIDE 23

Partitioning / swap on LVM

  • No parameters!
  • Pairwise can't reduce variant as parameter

– 9 x 1 == 9

  • Execute on random product variant each time!
slide-24
SLIDE 24

Acceptance criteria

slide-25
SLIDE 25

Less test case executions

slide-26
SLIDE 26

Don't miss existing bugs *

how does pairwise compare to full test suite wrt defect fjnding abilities ?

slide-27
SLIDE 27

Don't increase product risk *

how many critical defects would I miss if I don't execute the full test suite ?

slide-28
SLIDE 28

Experiment results

slide-29
SLIDE 29

65 % less test case executions !

2119 test cases in pairwise test plan

slide-30
SLIDE 30

76 % execution completion rate

previous releases are around 85%

slide-31
SLIDE 31

3 x 30 % bug discovery rate

slide-32
SLIDE 32

30 % of bugs found by Tier #1

good job, test cases not included in experiment

slide-33
SLIDE 33

30 % of bugs found by Pairwise

same were detected by following regular test plan

slide-34
SLIDE 34

30 % of bugs found by ET

we don't have test cases for them! Ouch !

slide-35
SLIDE 35

Pairwise missed 4 critical bugs

3 were regressions

slide-36
SLIDE 36
  • #1396949 - After installation with ibft the default

route is missing

– gPXE, fjrmware dependent

  • #1421039 - Anaconda fails to get kickstart from nfs
  • n s390x

– Corner case on s390x – IPv6 != IPv4

slide-37
SLIDE 37
  • #1400844 - Interface binding makes iscsi connection

fail

– Waived due to bad infrastructure setup – Waived again b/c ComputeNode doesn't support

Specialized Storage

  • #1420300 - Certmonger scriptlet outputs errors to

update.log during anaconda upgrade

– tested and not being re-tested

slide-38
SLIDE 38

Lessons learned

slide-39
SLIDE 39
slide-40
SLIDE 40

Perform test review regularly

found hidden parameters in tests found (sort of) duplicate test cases

slide-41
SLIDE 41

Observed optimization patterns

combine or pipeline independent TCs common set-up for multiple TCs across variants ... and pairwise, pairwise, pairwise

slide-42
SLIDE 42

Risk of not detecting regressions

risk is signifjcant in Snapshots phase due to historical aggregation of results

slide-43
SLIDE 43

Ask me anything !

@atodorov_ http://atodorov.org atodorov@redhat.com