[PPT] - Standalone applications testing and automation Desktop QA team Red PowerPoint Presentation

SLIDE 1

Standalone applications testing and automation

Desktop QA team Red Hat, Inc.

2014-02-02

SLIDE 2

2 2 02 February 2014 02 February 2014 FOSDEM 2014 FOSDEM 2014

Who we are Who we are

R e d H a t ' s D e s k t

p

Q E t e a m Q u a l i t y a s s u r a n c e f

r

: D e s k t

p

h a r d w a r e Wi r e l e s s , g r a p h i c s , a u d i

.

. . D e s k t

p

a p p l i c a t i

n

s t a c k D e s k t

p

e n v i r

n

m e n t s G U I m a n a g e m e n t t

l

s O ffj c e a p p l i c a t i

n

s

SLIDE 3

3 3 02 February 2014 02 February 2014 FOSDEM 2014 FOSDEM 2014

Distributions under test Distributions under test

R H E L 7 . D e s k t

p

R H E L 5

6

u p d a t e s F e d

r

a R a w h i d e / R H E L n e x t L a t e s t v e r s i

n

s t e s t i n g : N e t w

r

k M a n a g e r E v

l

u t i

n

SLIDE 4

4 4 02 February 2014 02 February 2014 FOSDEM 2014 FOSDEM 2014

Distributions under test Distributions under test

R H E L 7 . D e s k t

p

R H E L 5

6

u p d a t e s F e d

r

a R a w h i d e / R H E L n e x t L a t e s t v e r s i

n

s t e s t i n g : N e t w

r

k M a n a g e r E v

l

u t i

n

SLIDE 5

5 5 02 February 2014 02 February 2014 FOSDEM 2014 FOSDEM 2014

Distributions under test Distributions under test

R H E L 7 . D e s k t

p

R H E L 5

6

u p d a t e s F e d

r

a R a w h i d e / R H E L n e x t L a t e s t v e r s i

n

s t e s t i n g : N e t w

r

k M a n a g e r E v

l

u t i

n

SLIDE 6

6 6 02 February 2014 02 February 2014 FOSDEM 2014 FOSDEM 2014

Distributions under test Distributions under test

R H E L 7 . D e s k t

p

R H E L 5

6

u p d a t e s F e d

r

a R a w h i d e / R H E L n e x t L a t e s t v e r s i

n

s t e s t i n g : N e t w

r

k M a n a g e r E v

l

u t i

n

SLIDE 7

7 7 02 February 2014 02 February 2014 FOSDEM 2014 FOSDEM 2014

Infrastructure bits Infrastructure bits

B e a k e r m a n a g e s a v a i l a b l e m a c h i n e s a n d d i s t r i b u t i

n

s S i mp l e T e s t H a r n e s s t a s k f e t c h e s t e s t a u t

m

a t i

n

c

d

e a n d r u n s r e q u i r e d t e s t s B e h a v e c

n

t r

l

s a u t

m

a t i

n

e x e c u t i

n

D

g

t a i l / p e x p e c t e x e c u t e s a c t i

n

s

n

U I / C L I C

d

e r e v i e w i n G e r r i t T e s t c a s e m a n a g e m e n t

N

i t r a t e

SLIDE 8

8 8 02 February 2014 02 February 2014 FOSDEM 2014 FOSDEM 2014

Infrastructure bits Infrastructure bits

B e a k e r m a n a g e s a v a i l a b l e m a c h i n e s f r

m

t h e p

l

a n d d i s t r i b u t i

n

s S i mp l e T e s t H a r n e s s t a s k f e t c h e s t e s t a u t

m

a t i

n

c

d

e a n d r u n s r e q u i r e d t e s t s B e h a v e c

n

t r

l

s a u t

m

a t i

n

e x e c u t i

n

D

g

t a i l / p e x p e c t e x e c u t e s a c t i

n

s

n

U I / C L I C

d

e r e v i e w i n G e r r i t T e s t c a s e m a n a g e m e n t

N

i t r a t e

SLIDE 9

9 9 02 February 2014 02 February 2014 FOSDEM 2014 FOSDEM 2014

Infrastructure bits Infrastructure bits

B e a k e r m a n a g e s a v a i l a b l e m a c h i n e s f r

m

t h e p

l

a n d d i s t r i b u t i

n

s S i mp l e T e s t H a r n e s s t a s k f e t c h e s t e s t a u t

m

a t i

n

c

d

e a n d r u n s r e q u i r e d t e s t s B e h a v e c

n

t r

l

s a u t

m

a t i

n

e x e c u t i

n

D

g

t a i l / p e x p e c t e x e c u t e s a c t i

n

s

n

U I / C L I C

d

e r e v i e w i n G e r r i t T e s t c a s e m a n a g e m e n t

N

i t r a t e

SLIDE 10

10 10 02 February 2014 02 February 2014 FOSDEM 2014 FOSDEM 2014

Infrastructure bits Infrastructure bits

B e a k e r m a n a g e s a v a i l a b l e m a c h i n e s f r

m

t h e p

l

a n d d i s t r i b u t i

n

s S i mp l e T e s t H a r n e s s t a s k f e t c h e s t e s t a u t

m

a t i

n

c

d

e a n d r u n s r e q u i r e d t e s t s B e h a v e c

n

t r

l

s a u t

m

a t i

n

e x e c u t i

n

D

g

t a i l / p e x p e c t e x e c u t e s a c t i

n

s

n

U I / C L I C

d

e r e v i e w i n G e r r i t T e s t c a s e m a n a g e m e n t

N

i t r a t e

SLIDE 11

11 11 02 February 2014 02 February 2014 FOSDEM 2014 FOSDEM 2014

Infrastructure bits Infrastructure bits

B e a k e r m a n a g e s a v a i l a b l e m a c h i n e s f r

m

t h e p

l

a n d d i s t r i b u t i

n

s S i mp l e T e s t H a r n e s s t a s k f e t c h e s t e s t a u t

m

a t i

n

c

d

e a n d r u n s r e q u i r e d t e s t s B e h a v e c

n

t r

l

s a u t

m

a t i

n

e x e c u t i

n

D

g

t a i l / p e x p e c t e x e c u t e s a c t i

n

s

n

U I / C L I C

d

e r e v i e w i n G e r r i t T e s t c a s e m a n a g e m e n t

N

i t r a t e

SLIDE 12

12 12 02 February 2014 02 February 2014 FOSDEM 2014 FOSDEM 2014

Beaker Beaker

h t t p : / / b e a k e r

p

r

j

e c t .

r

g A u t

m

a t i

n

a n d t a s k e x e c u t i

n

s y s t e m f

r

l a b s

f

t e s t c

m

p u t e r s V M / b a r e

m

e t a l m a c h i n e s s u p p

r

t M u l t i a r c h i t e c u r e s u p p

r

t F l e x i b l e p e r m i s s i

n

s s y s t e m

SLIDE 13

13 13 02 February 2014 02 February 2014 FOSDEM 2014 FOSDEM 2014

Beaker Beaker

M

s

t n

t

a b l e t a s k s u s e d :

/ d e s k t

p

/ r h e l 7 / i n s t a l l

I n s t a l l s t h e d e s k t

p

c

m

p

n

e n t s ( G N O M E / K D E )

/ d e s k t

p

/ s i m p l e t e s t h a r n e s s

f e t c h e s a u t

m

a t i

n

c

d

e f r

m

g i t / g z i p s

u

r c e s s t a r t s s p e c i fj e d t e s t s t

b

e e x e c u t e d s t

r

e s t e s t

u

t p u t , r e p

r

t s a n d a r t i f a c t s

SLIDE 14

14 14 02 February 2014 02 February 2014 FOSDEM 2014 FOSDEM 2014

Beaker Beaker

M

s

t n

t

a b l e t a s k s u s e d :

/ d e s k t

p

/ r h e l 7 / i n s t a l l

I n s t a l l s t h e d e s k t

p

c

m

p

n

e n t s ( G N O M E / K D E )

/ d e s k t

p

/ s i m p l e t e s t h a r n e s s

f e t c h e s a u t

m

a t i

n

c

d

e f r

m

g i t / g z i p s

u

r c e s s t a r t s s p e c i fj e d t e s t s t

b

e e x e c u t e d s t

r

e s t e s t

u

t p u t , r e p

r

t s a n d a r t i f a c t s

SLIDE 15

15 15 02 February 2014 02 February 2014 FOSDEM 2014 FOSDEM 2014

Dogtail Dogtail

P y t h

n

G U I a u t

m

a t i

n

f r a m e w

r

k t a k i n g a d v a n t a g e

f

A c c e s s i b i l i t y t e c h n

l
g

i e s B a s e d

n

A T

S

P I – t

l

k i t

n

e u t r a l t e c h n

l
g

y , u s e d b y G T K , Q t , M

z

i l l a , L i b r e O ffj c e

SLIDE 16

16 16 02 February 2014 02 February 2014 FOSDEM 2014 FOSDEM 2014

Dogtail Dogtail

D

g

t a i l

b

a s e d u p s t r e a m t e s t s u i t e s P i T i V i G N O M E S

f

t w a r e E v i n c e

SLIDE 17

17 17 02 February 2014 02 February 2014 FOSDEM 2014 FOSDEM 2014

Sniff Sniff

S n i fg i s a G U I f

r

A T

S

P I s t r u c t u r e

SLIDE 18

18 18 02 February 2014 02 February 2014 FOSDEM 2014 FOSDEM 2014

Scripts and unittests Scripts and unittests

S c r i p t s

>

U n i t t e s t s

>

B D D B D D – B e h a v i

u

r D r i v e n D e v e l

p

m e n t

SLIDE 19

19 19 02 February 2014 02 February 2014 FOSDEM 2014 FOSDEM 2014

Gherkin Gherkin

G h e r k i n – B u s i n e s s R e a d a b l e D

m

a i n S p e c i fj c L a n g u a g e T e s t s c e n a r i

s

a r e h u m a n

r

e a d a b l e l i s t

f

s t e p s t

b

e p e r f

r

m e d S t e p s a r e m a t c h e d t

p

y t h

n

p r

c

e d u r e s , c a l l e d s t e p d e fj n i t i

n

s

SLIDE 20

20 20 02 February 2014 02 February 2014 FOSDEM 2014 FOSDEM 2014

Gherkin Gherkin

C

r

e k e y w

r

d s : F e a t u r e , B a c k g r

u

n d , S c e n a r i

S

c e n a r i

s

c a n b e g r

u

p e d b y f e a t u r e

r

u s i n g t a g s

SLIDE 21

21 21 02 February 2014 02 February 2014 FOSDEM 2014 FOSDEM 2014

Behave Behave

h t t p s : / / p y p i . p y t h

n

.

r

g / p y p i / b e h a v e P y t h

n

i m p l e m e n t a t i

n
f

B D D a p p r

a

c h F l e x i b l e

u

t p u t f

r

m a t t e r s ( H T M L , p l a i n e t c . ) S e t u p / b r e a k d

w

n i s i m p l e m e n t e d v i a b e f

r

e _ * / a f t e r _ * p r

c

e d u r e s

SLIDE 22

22 22 02 February 2014 02 February 2014 FOSDEM 2014 FOSDEM 2014

Behave Behave

SLIDE 23

23 23 02 February 2014 02 February 2014 FOSDEM 2014 FOSDEM 2014

Tips and Tricks Tips and Tricks

H T M L r e p

r

t w i t h s c r e e n s h

t

s a f t e r e a c h s t e p a n d l

g

s f r

m

j

u

r n a l c t l S c r e e n c a s t r e c

r

d i n g D e t e c t a p p c r a s h e s v i a a b r t

SLIDE 24

24 24 02 February 2014 02 February 2014 FOSDEM 2014 FOSDEM 2014

Benefits of BDD approach Benefits of BDD approach

A u t

m

a t e d t e s t s c e n a r i

s

a r e h u m a n

r

e a d a b l e C a n b e u s e d a s a i n s t r u c t i

n

s f

r

m a n u a l t e s t s E a s y t

u

p d a t e / e n h a n c e S c e n a r i

s

c a n b e w r i t t e n b y d e s i g n e r s d r a f t n e w f e a t u r e s d

c

u m e n t a p p b e h a v i

u

r S t e p s c a n b e r e u s e d a c r

s

s s e v e r a l p r

j

e c t s : G n

m

e O n l i n e A c c

u

n t s h a n d l i n g Wo r k i n g w i t h G N O M E

p

e n / s a v e fj l e d i a l

g

s

SLIDE 25

25 25 02 February 2014 02 February 2014 FOSDEM 2014 FOSDEM 2014

Benefits of BDD approach Benefits of BDD approach

A u t

m

a t e d t e s t s c e n a r i

s

a r e h u m a n

r

e a d a b l e C a n b e u s e d a s a i n s t r u c t i

n

s f

r

m a n u a l t e s t s E a s y t

u

p d a t e / e n h a n c e S c e n a r i

s

c a n b e w r i t t e n b y d e s i g n e r s d r a f t n e w f e a t u r e s d

c

u m e n t a p p b e h a v i

u

r S t e p s c a n b e r e u s e d a c r

s

s s e v e r a l p r

j

e c t s : G n

m

e O n l i n e A c c

u

n t s h a n d l i n g Wo r k i n g w i t h G N O M E

p

e n / s a v e fj l e d i a l

g

s

SLIDE 26

26 26 02 February 2014 02 February 2014 FOSDEM 2014 FOSDEM 2014

Benefits of BDD approach Benefits of BDD approach

A u t

m

a t e d t e s t s c e n a r i

s

a r e h u m a n

r

e a d a b l e C a n b e u s e d a s a i n s t r u c t i

n

s f

r

m a n u a l t e s t s E a s y t

u

p d a t e / e n h a n c e S c e n a r i

s

c a n b e w r i t t e n b y d e s i g n e r s d r a f t n e w f e a t u r e s d

c

u m e n t a p p b e h a v i

u

r S t e p s c a n b e r e u s e d a c r

s

s s e v e r a l p r

j

e c t s : G n

m

e O n l i n e A c c

u

n t s h a n d l i n g Wo r k i n g w i t h G N O M E

p

e n / s a v e fj l e d i a l

g

s

SLIDE 27

27 27 02 February 2014 02 February 2014 FOSDEM 2014 FOSDEM 2014

Benefits of BDD approach Benefits of BDD approach

U I a b s t r a c t i

n

i n s c e n a r i

s

:

S a m e s c e n a r i

s

c a n b e u s e d t

t

e s t v a r i

u

s f r

n

t e n d s : e . g . N e t w

r

k M a n a g e r ' s n m c l i / n m t u i E v

l

u t i

n

t e s t s w i t h m i n i m a l a d j u s t m e n t s c a n b e u s e d f

r

R H E L 6 , R H E L 7 , F e d

r

a 2 a n d F e d

r

a R a w h i d e

D u e t

g

r

u

p i n g s c e n a r i

s

b y f e a t u r e w e c a n e a s i l y r u n r e g r e s s i

n

c h e c k f

r

a fg e c t e d f e a t u r e

SLIDE 28

28 28 02 February 2014 02 February 2014 FOSDEM 2014 FOSDEM 2014

Benefits of BDD approach Benefits of BDD approach

U I a b s t r a c t i

n

i n s c e n a r i

s

:

S a m e s c e n a r i

s

c a n b e u s e d t

t

e s t v a r i

u

s f r

n

t e n d s : e . g . N e t w

r

k M a n a g e r ' s n m c l i / n m t u i E v

l

u t i

n

t e s t s w i t h m i n i m a l a d j u s t m e n t s c a n b e u s e d f

r

R H E L 6 , R H E L 7 , F e d

r

a 2 a n d F e d

r

a R a w h i d e

D u e t

g

r

u

p i n g s c e n a r i

s

b y f e a t u r e w e c a n e a s i l y r u n r e g r e s s i

n

c h e c k f

r

a fg e c t e d f e a t u r e

SLIDE 29

29 29 02 February 2014 02 February 2014 FOSDEM 2014 FOSDEM 2014

Success stories: NM, Evolution Success stories: NM, Evolution

E v

l

u t i

n

t e s t i n g a c r

s

s a v a i l a b l e d i s t r i b u t i

n

s P r

p
s

e d p a t c h t e s t i n g N e t w

r

k M a n a g e r

U s e s p e x p e c t i n s t e a d

f

d

g

t a i l S c e n a r i

s

c a n b e r e

u

s e d t

t

e s t v a r i

u

s N M f r

n

t e n d s : n m c l i , n m t u i ( n c u r s e s

b

a s e d ) S h a r i n g s t e p s w i t h G N O M E C

n

t r

l

C e n t e r

SLIDE 30

30 30 02 February 2014 02 February 2014 FOSDEM 2014 FOSDEM 2014

Success stories: NM, Evolution Success stories: NM, Evolution

E v

l

u t i

n

t e s t i n g a c r

s

s a v a i l a b l e d i s t r i b u t i

n

s P r

p
s

e d p a t c h t e s t i n g N e t w

r

k M a n a g e r

U s e s p e x p e c t i n s t e a d

f

d

g

t a i l S c e n a r i

s

c a n b e r e

u

s e d t

t

e s t v a r i

u

s N M f r

n

t e n d s : n m c l i , n m t u i ( n c u r s e s

b

a s e d ) S h a r i n g s t e p s w i t h G N O M E C

n

t r

l

C e n t e r

SLIDE 31

31 31 02 February 2014 02 February 2014 FOSDEM 2014 FOSDEM 2014

Future plans Future plans

Wo r k i n g w i t h F e d

r

a Q A t

h

a v e a s i m i l a r p r

c

e s s f

r

F e d

r

a E x e c u t e t e s t s d i r e c t l y i n u p s t r e a m C I :

G N O M E

c
n

t i n u

u

s B u i l d . k d e .

r

g

SLIDE 32

32 32 02 February 2014 02 February 2014 FOSDEM 2014 FOSDEM 2014

Future plans Future plans

Wo r k i n g w i t h F e d

r

a Q A t

h

a v e a s i m i l a r p r

c

e s s f

r

F e d

r

a E x e c u t e t e s t s d i r e c t l y i n u p s t r e a m C I :

G N O M E

c
n

t i n u

u

s B u i l d . k d e .

r

g

SLIDE 33

2014-02-02 33