Chapter 5 Analysis: Four Level for Validation Vis/Visual Analytics, - - PowerPoint PPT Presentation

Chapter 5 Analysis: Four Level for Validation

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 1

Contents

• Why Validate?
• Why is Validation

difficult?

• Four Levels of Design

– Domain Situation – Task & Data Abstraction – Visual Encoding &

I nteraction

– Algorithm

• Angles of Attack
• Threats to Validity

– Different for what, why,

how

• Validation Approaches

– Domain – Abstraction – I diom – Algorithm – Mismatches

• Examples

– Genealogical Graphs – MatrixExplorer – … and 4 more

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 2

Why Validate?

• Why?

– The vis design space is huge, and most

designs are ineffective

– Think about how you might validate your

choices from the very beginning of the design space, rather than leaving it at the end

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 3

Four Nested Levels of Design

• Why four nested

levels?

– Splitting the

complex vis design into four cascading levels provides an analysis framework that lets you to address different concerns separately

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 4

Four Nested Levels of Design

• Four nested levels

– Consider the details

• f a particular

application domain

– The what-why

abstraction

• Map domain-

specification problems and data into forms that are independent of the domain

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 5

Four Nested Levels of Design

• Four nested levels

– How level: design of

idioms that specify the visual encoding and interaction

– Design of algorithms

to instantiate those idioms computationally

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 6

Four Nested Levels of Design

• Four nested levels

– The output from an

upstream level is input to the downstream below

– Choosing a poor

choice at an upstream level inevitably cascades to all downstream levels

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 7

Four Nested Levels of Design

• Domain situation

– A domain situation

includes

• a group of target users

–

e.g. computational biologists

• domain of interest

–

comparative genomics

• Questions

–

e.g. genetic source of adaptivity in a species

• Data

–

e.g. genomic sequence data

– Methods to identify

domain

• I nterviews
• bservations
• careful research

User introspection is insufficient!

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 8

Four Nested Levels of Design

• Domain situation

– Output

• A detailed set of

questions asked or actions carried out by the target uses, about a possibly heterogeneous collection of data

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 9

Two of the questions asked by the computational biologist working

• n the comparative genomics:
• 1. What’s the difference between individual nucleotides (核甘酸) of feature pairs
• 2. Where are the gaps across a chromosome (染色體)?

Other: What is the genetic basis of a disease? (X, not specific enough)

Four Nested Levels of Design

• Data/ Task Abstraction

– Domain-specific into

generic representation

• I dentify abstract tasks

– Ex. Browsing, comparing,

summarizing

• Design abstract data

forms

– Are designed, a creative

design step

» Often choose to

transform the original data to something quite different

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 10

Four Nested Levels of Design

• Data/ Task Abstraction

– Explicitly considering the

choices made in abstracting tasks and data can be very useful in design process.

– Bad alternative: to do

this implicitly and w/ o justification

» Solving the “lost in

hyperspace” problem done by showing the searcher a website hyperlink connectivity graph?

» Wrong, too much

cognitive load.

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 11

Four Nested Levels of Design

• Visual encoding and

interaction idiom

– Decide on the specific

way to create and manipulate the visual

• rep. of the abstract

data block, guided by the abstract tasks

– I diom: each distinct

possible approach

• Visual encoding

– How to represent data

visually (what users see)

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 12

Four Nested Levels of Design

– I diom

• Visual encoding

– How to represent data

visually (what users see)

• I nteraction

– How to manipulate that

representation dynamically (how users change what they see)

• Possible to analyze

encoding and interaction as separate decision. I n some cases, need to be considered as a combined idiom

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 13

Four Nested Levels of Design

– I diom blocks are

designed

• A big design space…
• Abstracting data and task

can be used to rule out many bad options

• Should make decisions

about good and bad matches based on understanding human abilities, especially

– Visual perception – memory

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 14

Four Nested Levels of Design

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 15

Word Tree combines the visual encoding idiom of a hierarchical tree of keywords laid out horizontally and the interaction idiom of navigation based on keyword selection.

Four Nested Levels of Design

• Algorithm

– A detailed procedure

that allows a computer to automatically carry

• ut the desired goal
• To efficiently handle visual

encoding and interaction

– Are designed

• Could have many

algorithms for the same

• idiom. Ex. Many algorithms

for direct volume rendering

– Primary concerns:

computational issues

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 16

Visual/interaction: primary concerns are human perceptual issues

Four Levels of Design

• Dependency:

– Wrong block upstream

cascades downstream choices

• poor t as

ask, perfect id idio iom = > X

• I terative process:

– Consider each level

separately

– A better

understanding of one block will refine other levels

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 17

Angles of Attack for Vis Design

• Two angles of attack for vis design

– Top down (problem-driven) or bottom up

(technique driven)

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 18

Angles of Attack for Vis Design

• Problem-driven (Top-down)

– Start with the problems of real-world user and

attempt to design solution that that helps them work more effectively

– Often the problem can be solved using existing

visual coding and interaction idioms

• Much of the challenge lies at the abstraction level

– Sometimes the problem motivates the design of

new idioms, if no existing ones will adequately solve the abstracted design problem

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 19

Angles of Attack for Vis Design

• Problem-driven (Top-down)

– Considering the four levels of nested model

explicitly can help you avoid the pitfall of skipping important steps in problem-driven approach

• Some designers skip over the domain situation level

completely, short-circuit the abstraction level by assuming that the first abstraction is right and jump immediately into the third level

• THE ABSTRACTI ON LEVEL I S OFTEN THE HARDEST TO

GET RI GHT!!

• The design process for problem-driven work involves

iterative refinement at all levels.

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 20

Angles of Attack for Vis Design

• Technique driven (Bottom-Up)

– Start with idiom or algorithm design

• Goal is to invent new idioms that better support

existing abstractions, or new algorithms that better support existing idioms

– Considering the four nested model can help you

articulate your assumptions at the level just above your focus

• Articulate the abstraction requirement for new idiom,
• r articulate the idiom requirement for new algorithm

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 21

Treats to Validity

• Validating the effectiveness of a vis design

is difficult because there are so many possible questions

– Considering the validity questions at each level

separately helps

– Each level has a different set of treats to validity

• Different fundamental reasons why you might have

made wrong choices

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 22

Threats to Validity

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 23

Validation Approaches

• Different treats require very different

approaches

• Difference between immediate and

downstream validation

– Most kinds of validation for the outer levels are

not immediate because they require results from the level nested within them

– A poor showing of a test may misdirect

attention upstream, when in fact the problem results from a poor choice at the current level

– Downstream validation is necessary!! – The immediate validation only offer partial

evidence of success

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 24

Validation Approaches

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 25

Validation Approaches

• Domain Validation

– The primary treat is that the problem is

mischaracterized

• The target users do not in fact have these problems

– An immediate validation is to interview and

• bserve the target users to verify the

characterization

• Field study

– Observe how people act in real-world scenarios – Ask questions when clarification is needed – I nterview people about their needs

– Downstream validation: Report adoption rates

– What the target users do of their own accord

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 26

Validation Approaches

• Abstraction Validation (downstream)

– The treat: the identified task abstraction and

the designed data abstraction do not solve the characterized problems

– Key validation: the system must be tested by

target users doing their own work, rather than doing an abstracted task specified by the designer

• Collect anecdotal evidence that the tool is in fact useful

– insights found or hypothesis confirmed

– A more rigorous validation: to conduct a field

study to observe and document how the target user uses the deployed system

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 27

Validation Approaches

• I diom Validation

– Treat: the chosen idioms are not effective at

communicating the desired abstraction to the users

– One immediate validation:

carefully justify the idiom w.r.t. known perceptual and cognitive principles

• Heuristic evaluation
• Expert review

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 28

Validation Approaches

• I diom Validation

– A downstream validation

• Lab study

– A controlled experiment in a lab setting – Tease out the impact of specific idiom choice by measuring

human performance on abstract tasks

– Quantitative measurement » Time spent, errors made » Performance » Logging actions such as mouse moves and clicks » Tracking eye movements – Qualitative measurement » Reflect about the strategies through questionnaires » The number of participants needs to be sufficient for

statistical significance

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 29

Validation Approaches

• I diom Validation

– Another downstream validation

• Presentation of and qualitative discussion of results in

the form of still images or video

• Sometimes occur as usage scenarios, supporting that

the tool is useful for a particular task-data abstraction

– Third downstream validation

• The quantitative measures of result images by using

quality metrics

– e.g. Measure # of edge crossings and edge bends for

graphs

– I nformal usability study

• Lead to better and more usable systems, but neither
• ffer validation nor provide evidence of the superiority
• f an approach for a particular context

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 30

Validation Approaches

• Algorithm Validation

– Threat: the algorithm is suboptimal

• Either to a theoretical minimum or in comparison with

previous methods

• I s the algorithm correct?

Does it have good performance?

– I mmediate validation

• Analyze complexity and memory

– Downstream validation

• Measure wall-clock time & memory usage
• Primary consideration: scalability, how data size affect

the speed

• One trickier question: what data used?

– Use standard benchmarks

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 31

Validation Approaches

• Algorithm Validation

– Another threat: incorrectness at the algorithm

• Poor algorithm design, or
• The implementation

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 32

Validation Approaches

• Mismatches

– Mismatch between the level at which the

benefit is claimed and the validation methodology

– Example

• The benefit of a visual encoding cannot be validated by

wall-clock timings of the algorithm I n practice, not all validation tests are adopted

– The nested model explicitly separates the

design problem into levels in order to guide validation according to the unique threats at each level

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 33

Validation Examples

• Genealogical (宗譜) Graphs

– Proposed

• Multiple new visual encoding

– Classical dual-tree, classicial left-to-right, new indented

graph

• I nteraction idioms

– Automatic camera framing, animated transitions, and a

new widget for ballistically dragging out subtrees to arbitrary depth

– Explicitly cover all four levels

• Domain situation

– Domain is genealogy, discuss the need and current tools

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 34

Validation Examples

• Abstraction level

– Point out the term fam ily t ree is highly misleading because

the data type is a more general graph with specialized constraints on its structure

– Discuss conditions for which the data type is a tree, a

multi-tree, or a directed acyclic graph

– Map the domain problem of recognizing nuclear family

structure into an abstract task of determining subgraph structure

• Visual encoding and interactions

– Discuss the strengths and weakness of several visual

encoding design choices

» Connection, containment, adjacency and alignment,

indentation

– Address interaction idiom design

• Algorithm design

– Dual-tree layout

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 35

Validation Examples Genealogical Graphs

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 36

Genealogical graphs. (a) Three layouts for the dual-tree: classical node–link top-to-bottom at the top, classical left-to-right on the left, and the new indented outline algorithm on the right. (b) Widget for subtree collapsing and expanding with ballistic drags.

Validation Examples

– Validation

• Abstract level

– Downstream informal testing of a system prototype with a

target user to collect anecdotal evidence

• Visual encoding and interaction design

– I mmediate justification of Established principles – Downstream method of a qualitative discussion of result

images and video

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 37

Validation Examples

• MatrixExplorer for social network analysis

– Proposed

• Matrix representation to minimize clutter for large and

dense graph

• More intuitive node-link graph for smaller network

– Explicitly cover all four levels

• Domain situation

– Explicit characterization of social network analysis domain – Validated with qualitative interviews and exploratory

study with social scientists

• Abstraction level

– I nclude a detail list of the requirements in abstract form

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 38

Validation Examples

• MatrixExplorer for social network analysis
• Visual encoding and interaction

– A thorough discussion of the primary encoding idiom – Discussion of basic and more complex interaction » I nteractive reordering and clustering – Validation » Use the immediate validation » An extensive downstream validation using qualitative

discussion of result images

• Algorithm design

– Focus on reordering algorithm – Validation » Downstream benchmark timing

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 39

Validation Examples (MatrixExplorer)

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 40

MatrixExplorer features both node–link and matrix representations in an interface designed for sociologists and historians to explore social Networks (Munzner 84)

Validation Examples (MatrixExplorer)

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 41

MatrixExplorer validation methods

Validation Examples (Flow Maps)

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 42

Flow maps showing migration patterns from 1995 to 2000 US Census

• data. (a) Migration from California. (b) The top ten states that sent

migrants to California shown in green, and to New York in blue (Munzner 86)

Validation Examples (Flow Maps)

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 43

Validation Examples (LiveRAC)

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 44

LiveRAC supports exploration of system management time-series data with a reorderable matrix and semantic zooming. (a) The first several dozen rows have been stretched out to show sparklines for the devices. (b) The top three rows have been enlarged more, so the charts appear in full detail. (Munzner 87)

Validation Examples (LiveRAC)

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 45

Validation Examples (LinLog)

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 46

Validation Examples (Sizing the Horizon)

Vis/Visual Analytics, Chap 5 Validation CGGM Lab., CS Dept., NCTU Jung Hong Chuang 47