Automated Test Oracles Automated Test Oracles for GUIs for GUIs - - PDF document

▶
automated test oracles automated test oracles for guis
SMART_READER_LITE
LIVE PREVIEW

Automated Test Oracles Automated Test Oracles for GUIs for GUIs - - PDF document

Jul 05, 2023 300 likes •421 views

1 Automated Test Oracles Automated Test Oracles for GUIs for GUIs Eighth International Symposium Eighth International Symposium on the Foundations of Software on the Foundations of Software Engineering, San Diego, CA, Engineering, San

slide-1
SLIDE 1

1

1

Automated Test Oracles Automated Test Oracles for GUIs for GUIs

Eighth International Symposium Eighth International Symposium

  • n the Foundations of Software
  • n the Foundations of Software

Engineering, San Diego, CA, Engineering, San Diego, CA,

  • Nov. 6
  • Nov. 6-
  • 10, 2000.

10, 2000.

2

A Test Case for WordPad A Test Case for WordPad

SelectText (“This”) Format Font 18 OK SelectText (“text”) Format Font Underline OK This is the text.

This is the text.

slide-2
SLIDE 2

2

3

What Is Correct Behavior What Is Correct Behavior

This is the text.

This is the text.

SelectText (“This”) Format Font 18 OK SelectText (“text”) Format Font Underline OK

Check State, not only Output !!

This is the text. ?!

4

Research Focus Research Focus

  • Goal

Goal

– – To check the To check the GUI’s state GUI’s state after each event after each event

  • Approaches

Approaches

– – Manual Manual – – Automated Automated

Actual State Expected State Automated Verifier Verdict

  • Challenges

Challenges

– – Generating expected state Generating expected state – – Extracting actual state Extracting actual state – – Comparing expected & actual states Comparing expected & actual states

slide-3
SLIDE 3

3

5

Outline Outline

  • Overview of GUI Oracle

Overview of GUI Oracle

  • Generating Expected State

Generating Expected State

– – Modeling the GUI’s State Modeling the GUI’s State

  • Objects

Objects

  • Properties

Properties

– – Modeling the Events Modeling the Events

  • Obtaining Actual GUI’s State

Obtaining Actual GUI’s State

  • Comparing Actual & Expected States

Comparing Actual & Expected States

  • Case Study: MS WordPad

Case Study: MS WordPad

  • Concluding Remarks

Concluding Remarks

6

Overview of GUI Oracle Overview of GUI Oracle

Test Case

Expected-state Generator Verifier

Expected State

Execution Monitor

Oracle Oracle Oracle Oracle

Actual State Run-time information from executing GUI Verdict Formal GUI Model

slide-4
SLIDE 4

4

7

Modeling the GUI Modeling the GUI

A GUI consists of Objects Button Form Label

Window State Width AutoScroll Align Caption Color Font Caption Enabled Visible Height wsNormal 1088 TRUE Cancel TRUE TRUE 65 alNone Files of type: clBtnFace (tFont)

8

All Properties of All Properties of Cancel Cancel

slide-5
SLIDE 5

5

9

Determining Properties Determining Properties

  • Manual Examination of GUI

Manual Examination of GUI

  • Specifications (Reduced Set)

Specifications (Reduced Set)

– – GUI being tested GUI being tested

  • Toolkit/Language (Complete Set)

Toolkit/Language (Complete Set)

– – All available properties All available properties

Now we know how to represent the GUI’s state

10

Modeling Events Modeling Events

  • Events are State Transducers

Events are State Transducers

This is the text.

SelectText (“This”)

This is the text. This

State: Si State: Sj Event: e Notation: Sj = [Si, e]

slide-6
SLIDE 6

6

11

Representing Events Representing Events

  • We define an event as:

We define an event as: Statej = [Statei, event]

  • For example:

For example: Statej = [Statei, cut]

  • Need a compact representation

Need a compact representation

12

Operator :: CUT Preconditions: isCurrent(Menu2). Effects: FORALL Obj in Objects Selected(Obj) ⇒ ⇒ ⇒ ⇒ ADD inClipboard(Obj) DEL onScreen(Obj) DEL Selected(Obj) ADD isCurrent(Menu1) DEL isCurrent(Menu2). Menu2 Menu1 File Edit View Ins

Cut

Operators Operators

Obtaining next state

slide-7
SLIDE 7

7

13

Deriving Expected State Deriving Expected State

  • Given

Given S S0

0, the initial state,

, the initial state,

  • A sequence of events

A sequence of events

  • Obtain

Obtain S S1

1 = [S

= [S0

0, e

, e1

1]

]

  • And

And S Si

i = [

= [S Si

i-

  • 1

1,

, e ei

i]

] e1 e2 e3 en e1 e2 e3 en S0 S0 S1 S1 S2 S2 S3 S3 Sn Sn

14

Obtaining Actual GUI’s State Obtaining Actual GUI’s State

  • Execution Monitor

Execution Monitor

– – Screen Scraping Screen Scraping – – Queries Queries – – Compatible with Expected State Compatible with Expected State – – Returns <Object, Property, Value> Returns <Object, Property, Value> <Button1, “Caption”, “Cancel”> <Button1, “Caption”, “Cancel”>

slide-8
SLIDE 8

8

15

Automated Execution Automated Execution

Expected State Expected State Test Executor GUI Under Test Execution Monitor

ACTUAL STATE: (isCurrent ROOT) (Contains ROOT D (Contains ROOT D

Verifier Test Cases Test Cases

16

Comparing Comparing Actual and Expected States Actual and Expected States

  • Verifier

Verifier

  • Three Levels of Testing

Three Levels of Testing

– – Changed Property Set Changed Property Set (Operators) (Operators) – – GUI Relevant Property Set GUI Relevant Property Set (Specifications) (Specifications) – – Complete Property Set Complete Property Set (Toolkit/Language) (Toolkit/Language)

  • Hybrid Approach

Hybrid Approach

– – Use all 3 Use all 3

slide-9
SLIDE 9

9

17

Case Study Case Study

  • Purpose: Determine

Purpose: Determine

– – Time to Derive Expected State Time to Derive Expected State – – Time to Execute Monitor and Verifier Time to Execute Monitor and Verifier

  • Experimental Design

Experimental Design

– – GUI GUI: : Our Version of MS WordPad (36 Modal Our Version of MS WordPad (36 Modal Windows, 362 events) Windows, 362 events) – – Test Cases Test Cases: : Generated 290 Test Cases (6 Generated 290 Test Cases (6-

  • 56

56 events) using an AI Planner events) using an AI Planner – – Hardware Platform Hardware Platform: : 350 MHz Pentium based 350 MHz Pentium based Machine, 256 MB RAM Machine, 256 MB RAM – – Properties Properties: : Reduced Set Reduced Set – – Level of Testing Level of Testing: : GUI Relevant Property Set GUI Relevant Property Set

18

Deriving Expected State Deriving Expected State

Generating Test Cases and Deriving Expected State 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 6 11 16 21 26 31 36 41 46 51 56 Test-Case Length Time (sec.)

Expected State Test Case Test Case + Expected State

Total CPU time (test case and expected state) 75.84 sec.

slide-10
SLIDE 10

10

19

Execution Execution

Executing Test Cases, Verifier and Execution Monitor 1 2 3 4 5 1 6 11 16 21 26 31 36 41 46 51 56 Test-Case Length Time (sec.)

Test Case Verifier + Execution Monitor Test Case + Verifier + Execution Monitor

Relevant-properties verification Total running time < 10 minutes