SLIDE 1
1
Explaining Visual Changes in Web Interfaces
Brian Burg, Andrew J Ko, Michael Ernst
University of Washington
2
Explaining Visual Changes in Web Interfaces Brian Burg , Andrew J - - PowerPoint PPT Presentation
Explaining Visual Changes in Web Interfaces Brian Burg , Andrew J - - PowerPoint PPT Presentation
Explaining Visual Changes in Web Interfaces Brian Burg , Andrew J Ko, Michael Ernst University of Washington 1 2 Feature Location Feature location is the activity of identifying an initial location in the source code that
SLIDE 2
SLIDE 3
Feature Location
3
Feature location is the activity of identifying an initial location in the source code that implements functionality in a software system.
Dit, B., Revelle, M., Gethers, M., and Poshyvanyk, D., "Feature Location in Source Code: A Taxonomy and Survey", Journal of Software: Evolution and Process (JSEP), vol. 25, no. 1, January 2013, pp. 53–95
SLIDE 4
Feature Location for Interactive Web Content
4
SLIDE 5
5
Visual Output
😅
How can I interact with it?
SLIDE 6
6
Visual Output DOM & CSS
🤕 😅
How is the visual effect achieved?
SLIDE 7
7
Visual Output DOM & CSS JavaScript
🤕 😨 😅
What code is ultimately responsible?
SLIDE 8
8
No Output History Scattered State No Links Between Source and State Visual Output DOM & CSS JavaScript
SLIDE 9
9
Scry
Staged, Interactive Feature Location
SLIDE 10
10
Output Examples
Staged, Interactive Feature Location
SLIDE 11
11
State Differences Output Examples
Δ(pre, post)
Staged, Interactive Feature Location
SLIDE 12
12
JavaScript Mutations State Differences Output Examples
Δ(pre, post)
Staged, Interactive Feature Location
SLIDE 13
SLIDE 14
SLIDE 15
15
JavaScript Mutations State Differences Output Examples
Δ(pre, post)
SLIDE 16
What determines visual appearance?
16
DOM Tree Structure CSS Style Properties
Color, layout mode, visual styling, text rendering, handling of children
Rendering Engine
Visual State
SLIDE 17
17
Target Element Snapshots
DOM Subtree Computed Styles & Related Rules
Element Screenshots Mutation Operations
SLIDE 18
Why does visual appearance change?
18
DOM Tree Mutations
Insertion Deletion Ordinal Change
class=“active” textContent = … :hover, :focus
Attributes & States
Style Property Changes
Rule Changes Inline Styles Animations
SLIDE 19
19
Detecting changes in appearance
Visual Diff >1%?
NO YES
Commit New Snapshot
YES
Painted Rects
Do they intersect?
NO
Snapshot of Target Element
SLIDE 20
20
Comparing State Snapshots
Per-element change summaries
Structure: Insertion, Deletion, Attributes, … Styles: Added, Removed, Value Change
Relies on stable DOM element identity
Doesn’t work well when view state is split from DOM
Δ(pre, post)
SLIDE 21
Change-Relevant Operation Slicing
- 1. Instrument and record mutation operations.
- 2. Build a dependency graph for operations between
the pre-state and post-state.
- 3. Based on change summary, find an equivalent
mutation operation to explain the change.
- 4. Return equivalent operation + dependencies
SLIDE 22
22
Technical Challenges
Visual containment, stacking Software vs hardware rendering Unstable DOM element identities Megamorphic call sites in library code Pruning ineffective styles and attributes
SLIDE 23
Summary
Feature Location via Visual States
States can be automatically captured when drawing.
Juxtapose Captured Inputs and Outputs
State and output snapshots help explain each other.
Diff Markers Filter Relevant Operations
Slicing algorithms can show responsible operations.
23
SLIDE 24
24
Explaining Visual Changes in Web Interfaces
Brian Burg, Andrew J Ko, Michael Ernst
University of Washington