Presentation Failures Using Computer Vision-Based Techniques Sonal - - PowerPoint PPT Presentation

▶
presentation failures using computer
SMART_READER_LITE
LIVE PREVIEW

Presentation Failures Using Computer Vision-Based Techniques Sonal - - PowerPoint PPT Presentation

Oct 01, 2022 910 likes •1.44k views

Detection and Localization of HTML Presentation Failures Using Computer Vision-Based Techniques Sonal Mahajan and William G. J. Halfond Department of Computer Science University of Southern California Presentation of a Website What do we

slide-1
SLIDE 1

Detection and Localization of HTML Presentation Failures Using Computer Vision-Based Techniques

Sonal Mahajan and William G. J. Halfond Department of Computer Science University of Southern California

slide-2
SLIDE 2

Presentation of a Website

  • What do we mean by presentation?

– “Look and feel” of the website in a browser

  • What is a presentation failure?

– Web page rendering ≠ expected appearance

  • Why is it important?

– It takes users only 50 ms to form opinion about your website (Google research – 2012) – Affects impressions of trustworthiness, usability, company branding, and perceived quality

2

End user – no penalty to move to another website Business – loses out on valuable customers

slide-3
SLIDE 3

Motivation

  • Manual detection is difficult

– Complex interaction between HTML, CSS, and Javascript – Hundreds of HTML elements + CSS properties – Labor intensive and error-prone

  • Our approach – Automate debugging of

presentation failures

3

slide-4
SLIDE 4

Two Key Insights

  • 1. Detect presentation failures

4

Oracle image Test web page Presentation failures Visual comparison

Use computer vision techniques

slide-5
SLIDE 5

Two Key Insights

  • 2. Localize to faulty HTML elements

5

Test web page Faulty HTML elements Layout tree

Use rendering maps

slide-6
SLIDE 6
  • Regression Debugging

– Current version of the web app is modified

  • Correct bug
  • Refactor HTML (e.g., convert <table> layout to <div> layout)

– DOM comparison techniques (XBT) not useful, if DOM has changed significantly

  • Mockup-driven development

– Front-end developers convert high-fidelity mockups to HTML pages – DOM comparison techniques cannot be used, since there is no existing DOM – Invariants specification techniques (Selenium, Cucumber, Sikuli) not practical, since all correctness properties need to be specified – Fighting layout bugs: app independent correctness checker

6

Limitations of Existing Techniques

slide-7
SLIDE 7

Running Example

Web page rendering Expected appearance (oracle)

≠

7

slide-8
SLIDE 8

Our Approach

8

  • P1. Detection
  • P2. Localization

Oracle image Test web page Visual differences Pixel-HTML mapping Report

Goal – Automatically detect and localize presentation failures in web pages

slide-9
SLIDE 9
  • P1. Detection

9

  • Find visual differences (presentation failures)
  • Compare oracle image and test page

screenshot

  • Simple approach: strict pixel-to-pixel

equivalence comparison

– Drawbacks

  • Spurious differences due to difference in platform
  • Small differences may be “OK”
slide-10
SLIDE 10

Perceptual Image Differencing (PID)

  • Uses models of the human visual system

– Spatial sensitivity – Luminance sensitivity – Color sensitivity

  • Configurable parameters

– Δ : Threshold value for perceptible difference – F : Field of view of the observer – L : Brightness of the display – C : Sensitivity to colors

10

Shows only human perceptible differences

slide-11
SLIDE 11

Filter differences belonging to dynamic areas

  • P1. Detection – Example

11

Test web page screenshot Oracle Visual comparison using PID Apply clustering (DBSCAN)

A B C

slide-12
SLIDE 12
  • P2. Localization
  • Identify the faulty HTML element
  • Use R-tree to map pixel visual differences

to HTML elements

  • “R”ectangle-tree: height-balanced tree,

popular to store multidimensional data

Use rendering maps to find faulty HTML elements corresponding to visual differences

12

slide-13
SLIDE 13

13

  • P2. Localization - Example
slide-14
SLIDE 14

R1

Sub-tree of R-tree

14

  • P2. Localization - Example

R2 R3 R4 R5

slide-15
SLIDE 15

(100, 400)

Map pixel visual differences to HTML elements

15

  • P2. Localization - Example

R 2 R 1

tr[2 ]

R 3

t d tabl e t r t d

Result Set: /html/body/…/tr[2] /html/body/…/tr[2]/td[1] /html/body/…/tr[2]/td[1]/table[1] /html/body/…/tr[2]/td[1]/table[1]/tr[1] /html/body/…/tr[2]/td[1]/table[1]/td[1]

slide-16
SLIDE 16

Special Regions Handling

  • Special regions = Dynamic portions (actual content

not known)

16

  • 1. Exclusion Region
  • 2. Dynamic Text Region
slide-17
SLIDE 17
  • 1. Exclusion Regions
  • Only apply size bounding property

17

Difference pixels filtered in detection Advertisement box <element> reported as faulty

Test web page Oracle

slide-18
SLIDE 18
  • 2. Dynamic Text Regions
  • Style properties of text known

18

Text color: red Font-size: 12px Font-weight: bold

Test web page Modified test web page (Oracle) Run P1, P2 News box <element> reported as faulty

slide-19
SLIDE 19
  • P3. Result Set Processing
  • Rank the HTML elements in the order of

likelihood of being faulty

  • Weighted prioritization score
  • Lower the score, higher the likelihood of

being faulty

Use heuristics based on element relationships

19

slide-20
SLIDE 20

3.1 Contained Elements (C)

20

parent child1 child2 child1 child2 parent

Expected appearance Actual appearance

✖ ✖ ✖

slide-21
SLIDE 21

3.2 Overlapped Elements (O)

21

parent child1 child2 child1 child2 parent

Expected appearance Actual appearance

✖ ✖

slide-22
SLIDE 22

3.3 Cascading (D)

22

element 1

Expected appearance Actual appearance

element 2 element 3 element 1 element 2 element 3

✖ ✖

slide-23
SLIDE 23

3.4 Pixels Ratio (P)

23

parent child

✖ ✖

Child pixels ratio = 100% Parent pixels ratio = 20%

slide-24
SLIDE 24

/html /html/body /html/body/table . . . /html/body/table/…/img

  • 1. /html/body/table/…/img

. . .

  • 5. /html/body/table
  • 6. /html/body
  • 7. /html
  • P3. Result Set Processing - Example

24

A B C D E

Report

Cluster B Cluster C Cluster D Cluster E

/html /html/body /html/body/table . . . /html/body/table/…/img

Cluster A

slide-25
SLIDE 25

Empirical Evaluation

  • RQ1: What is the accuracy of our approach

for detecting and localizing presentation failures?

  • RQ2: What is the quality of the localization

results?

  • RQ3: How long does it take to detect and

localize presentation failures with our approach?

25

slide-26
SLIDE 26

Experimental Protocol

  • Approach implemented in “WebSee”
  • Five real-world subject applications
  • For each subject application

– Download page and take screenshot, use as the oracle – Seed a unique presentation failure to create a variant – Run WebSee on oracle and variant

26

slide-27
SLIDE 27

Subject Applications

27

Subject Application Size (Total HTML Elements) Generated # test cases

Gmail

72 52

USC CS Research

322 59

Craigslist

1,100 53

Virgin America

998 39

Java Tutorial

159 50

slide-28
SLIDE 28

RQ1: What is the accuracy?

28

93%

92% 92% 90% 97% 94% Gmail USC CS Research Craigslist Virgin America Java Tutorial

Localization accuracy

  • Detection accuracy: Sanity check for PID
  • Localization accuracy: % of test cases in

which the expected faulty element was reported in the result set

slide-29
SLIDE 29

RQ2: What is the quality of localization?

29

12 (16%) 17 (5%) 32 (3%) 49 (5%) 8 (5%)

Gmail USC CS… Craigslist Virgin America Java Tutorial Result Set Size

23 (10%)

  • 1. <>…</>
  • 2. <>…</>

. . . . . .

  • 23. <>…</>

Rank = 4.8 (2%)

✖ ✔ Distance = 6

faulty element not present

slide-30
SLIDE 30

RQ3: What is the running time?

21% 25% 54%

P1: Detection P2: Localization P3: Result Set Processing

30

7 sec 3 min

87 sec Sub-image search for cascading heuristic

slide-31
SLIDE 31

Comparison with User Study

76% 36% 100% 93% Detection Localization

Accuracy

Students WebSee

  • Graduate-level students
  • Manual detection and

localization using Firebug

  • Time

– Students: 7 min – WebSee: 87 sec

31

slide-32
SLIDE 32

Case Study with Real Mockups

  • Three subject applications
  • 45% of the faulty elements reported in

top five

  • 70% reported in top 10
  • Analysis time similar

32

slide-33
SLIDE 33

Summary

  • Technique for automatically detecting and

localizing presentation failures

  • Use computer vision techniques for detection
  • Use rendering maps for localization
  • Empirical evaluation shows positive results

33

slide-34
SLIDE 34

Thank you

34

Detection and Localization of HTML Presentation Failures Using Computer Vision-Based Techniques

Sonal Mahajan and William G. J. Halfond spmahaja@usc.edu halfond@usc.edu

slide-35
SLIDE 35

Normalization Process

  • Pre-processing step before detection
  • 1. Browser window size is adjusted based
  • n the oracle
  • 2. Zoom level is adjusted
  • 3. Scrolling is taken care of

35

slide-36
SLIDE 36

Difference with XBT

  • XBT use DOM comparison

– Find matched nodes, compare them

  • Regression debugging

– Correct bug, refactor HTML (e.g. <table> to <div> layout) – DOM significantly changed

  • XBT cannot find matching DOM nodes, not accurate

comparison

  • Mockup Driven Development

– No “golden” version of page (DOM) exists – XBT techniques cannot be used

  • Our approach

– Uses computer vision techniques for detection – Applies to both scenarios

36

slide-37
SLIDE 37

Pixel-to-pixel Comparison

37

Oracle Test Webpage Screenshot

slide-38
SLIDE 38

Pixel-to-pixel Comparison

38

Difference image

98% of the entire image is shown in difference!

Difference pixel Matched pixel

slide-39
SLIDE 39
  • P1. Detection

39

Analyze using computer vision techniques

  • Our approach: Perceptual image

differencing (PID)

  • Find visual differences (presentation failures)
  • Simple approach: strict pixel-to-pixel

equivalence comparison

slide-40
SLIDE 40

Perceptual Image Differencing

40

Difference image

Difference pixel Matched pixel

slide-41
SLIDE 41

41

High fidelity mockups… reasonable?

slide-42
SLIDE 42

42

High fidelity mockups… reasonable?

slide-43
SLIDE 43

43

High fidelity mockups… reasonable?

slide-44
SLIDE 44

44

High fidelity mockups… reasonable?

slide-45
SLIDE 45

45

High fidelity mockups… reasonable?

slide-46
SLIDE 46

46

High fidelity mockups… reasonable?

slide-47
SLIDE 47

47

High fidelity mockups… reasonable?

slide-48
SLIDE 48

48

High fidelity mockups… reasonable?

slide-49
SLIDE 49

49

High fidelity mockups… reasonable?

slide-50
SLIDE 50

50

High fidelity mockups… reasonable?

slide-51
SLIDE 51

51

High fidelity mockups… reasonable?

slide-52
SLIDE 52

52

High fidelity mockups… reasonable?