Comp/Phys/APSc 715 Evaluation of Visualization Vector Visualization - - PDF document

▶
comp phys apsc 715
SMART_READER_LITE
LIVE PREVIEW

Comp/Phys/APSc 715 Evaluation of Visualization Vector Visualization - - PDF document

Aug 11, 2023 326 likes •480 views

3/3/2014 Comp/Phys/APSc 715 Evaluation of Visualization Vector Visualization Redesign 3/6/2014 Evaluation Comp/Phys/APSc 715 Taylor 1 Example Videos Vis 2008, Wang: vis-1013_final_video.mp4 Focus + context display in 3D Vis 2008,

slide-1
SLIDE 1

3/3/2014 1

Comp/Phys/APSc 715

Evaluation of Visualization Vector Visualization Redesign

3/6/2014 Evaluation 1 Comp/Phys/APSc 715 Taylor

Example Videos

  • Vis 2008, Wang: vis-1013_final_video.mp4

– Focus + context display in 3D

  • Vis 2008, Wangchao: idtvdv.avi

– Importance-driven rendering

  • Vis 2008, Zhou: 2008 Vis. Visibility Based

Mesh Analysis.submission.mov

– Importance-driven rendering from CAD model

3/6/2014 Evaluation 2 Comp/Phys/APSc 715 Taylor

Administrative

  • There will be more data and more questions

for all of the projects than was in the HW.

– To determine, meet with scientist

  • Let me know your project preferences

– Total of 100 points to allocate to all 4 – More points means more preference – Trade-offs to being both client and on team… – Email me by tomorrow (Friday)

3/6/2014 Evaluation Comp/Phys/APSc 715 Taylor 3

slide-2
SLIDE 2

3/3/2014 2

3/6/2014 Evaluation Comp/Phys/APSc 715 Taylor 4

Evaluation

  • “…we often design and evaluate methods by

presenting results informally to potential users.” [Kosara et al 2003]

– We will be doing this in this course – We’ll also add a more formal task but only for a single person doing one task: see instructions – Note that even this will be a nontrivial effort – start planning for it now

3/6/2014 Evaluation 5 Comp/Phys/APSc 715 Taylor

Potential Types of Evaluation

  • Re-use existing designs (art, cartography)
  • Hire an expert visual designer to leapfrog into

known “best-practice” space

  • Videotaping one or more users working with

the system

  • User Studies: evaluating performance

3/6/2014 Evaluation 6 Comp/Phys/APSc 715 Taylor

slide-3
SLIDE 3

3/3/2014 3

Why Conduct User Studies?

3/6/2014 Evaluation 7 Comp/Phys/APSc 715 Taylor

Why Conduct User Studies?

  • Offer scientifically sound method to measure a

visualization’s performance

– Accuracy and speed

  • Provide insight into why a technique is effective

– By varying conditions and parameters to see effect

  • Determine if theoretical principles derived from

psychophysics apply to visualization design

– Taking the study up one level of complexity

3/6/2014 Evaluation 8 Comp/Phys/APSc 715 Taylor

Types of Studies

  • Perceptual studies

– Very simple tasks and stimuli

  • “Which types of texture enhance surface perception,

and which camouflage it?”

  • “What is the best color map to display ratio scalar fields

with high spatial frequency data?”

  • Usability studies

– User performs a (perhaps complex) task

3/6/2014 Evaluation 9 Comp/Phys/APSc 715 Taylor

slide-4
SLIDE 4

3/3/2014 4

What to collect?

  • Careful statistical data about performance

– time and error measures

  • Close observation of user behavior

– when did they get frustrated? – when did they make errors?

  • Free-form comments from the users

3/6/2014 Evaluation 10 Comp/Phys/APSc 715 Taylor

Doing Experiments on People is Serious Business

  • Requires serious commitment of time and effort

– Planning the experiment (seek help from psych!) – Evaluating the results (seek help from stat during plan!) – Iterating 2-3 times (uncompelling results)

  • Requires approval of Institutional Review Board on

campus

– Seeks to preserve respect for and rights of subjects – Seeks to prevent new occurrences of egregious past acts of misconduct

  • Kosara, et al, report that it is usually worth the effort

3/6/2014 Evaluation 11 Comp/Phys/APSc 715 Taylor 3/6/2014 Evaluation 12 Comp/Phys/APSc 715 Taylor

slide-5
SLIDE 5

3/3/2014 5

Laidlaw Vector Field Study

IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 11, NO. 1, JANUARY/FEBRUARY 2005

David H. Laidlaw, Robert M. Kirby, Cullen D. Jackson, J. Scott Davidson, Timothy S. Miller, Marco da Silva, William H. Warren, and Michael J. Tarr

3/6/2014 Evaluation 13 Comp/Phys/APSc 715 Taylor

Laidlaw Vector Field Study

  • Question 1: Where are the critical points?

IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 11, NO. 1, JANUARY/FEBRUARY 2005

David H. Laidlaw, Robert M. Kirby, Cullen D. Jackson, J. Scott Davidson, Timothy S. Miller, Marco da Silva, William H. Warren, and Michael J. Tarr

3/6/2014 Evaluation 14 Comp/Phys/APSc 715 Taylor

Laidlaw Vector Field Study

  • Question 2: What type of critical point?

IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 11, NO. 1, JANUARY/FEBRUARY 2005

David H. Laidlaw, Robert M. Kirby, Cullen D. Jackson, J. Scott Davidson, Timothy S. Miller, Marco da Silva, William H. Warren, and Michael J. Tarr

3/6/2014 Evaluation 15 Comp/Phys/APSc 715 Taylor

slide-6
SLIDE 6

3/3/2014 6

Laidlaw Vector Field Study

  • Question 3: Where would the point go?

IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 11, NO. 1, JANUARY/FEBRUARY 2005

David H. Laidlaw, Robert M. Kirby, Cullen D. Jackson, J. Scott Davidson, Timothy S. Miller, Marco da Silva, William H. Warren, and Michael J. Tarr

3/6/2014 Evaluation 16 Comp/Phys/APSc 715 Taylor

Laidlaw Vector Field Study

  • Results: Which was the best?

– It depends on the task – GSTR better than average on all metrics

IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 11, NO. 1, JANUARY/FEBRUARY 2005

David H. Laidlaw, Robert M. Kirby, Cullen D. Jackson, J. Scott Davidson, Timothy S. Miller, Marco da Silva, William H. Warren, and Michael J. Tarr

3/6/2014 Evaluation 17 Comp/Phys/APSc 715 Taylor

Laidlaw Vector Field Study

  • Of note: Experts and non-experts similar!

Brief training sufficient

IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 11, NO. 1, JANUARY/FEBRUARY 2005

David H. Laidlaw, Robert M. Kirby, Cullen D. Jackson, J. Scott Davidson, Timothy S. Miller, Marco da Silva, William H. Warren, and Michael J. Tarr

3/6/2014 Evaluation 18 Comp/Phys/APSc 715 Taylor

N= non- E= expert

slide-7
SLIDE 7

3/3/2014 7

Laidlaw Vector Field Study

  • Of note: Advection was always pretty good!

< ~5 degrees of error

IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 11, NO. 1, JANUARY/FEBRUARY 2005

David H. Laidlaw, Robert M. Kirby, Cullen D. Jackson, J. Scott Davidson, Timothy S. Miller, Marco da Silva, William H. Warren, and Michael J. Tarr

3/6/2014 Evaluation 19 Comp/Phys/APSc 715 Taylor

Laidlaw Vector Field Study

  • Of note: Not significantly better, but faster

– for critical point type

IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 11, NO. 1, JANUARY/FEBRUARY 2005

David H. Laidlaw, Robert M. Kirby, Cullen D. Jackson, J. Scott Davidson, Timothy S. Miller, Marco da Silva, William H. Warren, and Michael J. Tarr

3/6/2014 Evaluation 20 Comp/Phys/APSc 715 Taylor 3/6/2014 Evaluation Comp/Phys/APSc 715 Taylor

Compared to Class Guesses…

Par Li Gl1 2 3 4 5

Tex

Sfc Clr

L L+

Ok/4

5 3 2 1

Ok/6

X X

Sources and sinks (Identify critical points)

L L+

Ok/4

5 3 2 1

Ok/1

X X

Sources and sinks (Locate critical points)

Ok Tr

Ok/4

5 1 3 2

Ok/6

X Ok

Fast/slow/still (Find zero-flow locations)

Ok L Ok

Ok

X X

Center of rotation

Ok + Ok

Ok

X X

Shape of flow

Ok ++ ?

Ok

X X

Where is flow laminar vs. turbulent?

++ T

L/4

4 3 1 2

Dye/ 4

L X

Where would a pushed object end up?

T T L T

L

X

Where does a concentration come from?

Ok Ok Ok ? + X

Where does stress cause strain?

T T Ok T T ++

Positive vs. negative field (scalar)?

21

slide-8
SLIDE 8

3/3/2014 8

Evaluation in this class

  • Formal

– Asks primary goal of the scientist – On a data set truth is known for (often synthetic) – Non-team-member who has not seen the data

  • Informal client feedback

– What new things did the client learn? – How is it better/worse than existing tools? – How do they like it?

3/6/2014 Evaluation 22 Comp/Phys/APSc 715 Taylor 3/6/2014 Evaluation 23 Comp/Phys/APSc 715 Taylor

3D DDS User Study

  • How well does 3DS work?

– At what? – Compared to what?

  • More specific

– At identifying relationships and extracting values. – Compared to other glyph-based technique.

Feng D., Lee, Y., Kwock L., and Taylor, R., “Evaluation of Glyph-based Scalar Multivariate Volume Visualization Techniques,” in Proceedings of the 6th Symposium on Applied Perception in Graphics and Visualization 2009. ACM Press, New York, NY, pp. 61-68.

3/6/2014 Evaluation 24 Comp/Phys/APSc 715 Taylor

slide-9
SLIDE 9

3/3/2014 9

Relationships

  • What kinds of relationships?

– Linear – Overlap/Intersection – Multivariate – Etc.

  • Data

– Real? No. goal is to discover relationship – Fake? What kind?

3/6/2014 Evaluation 25 Comp/Phys/APSc 715 Taylor

Data

  • Application driven

– Controlled, but resembles original data – 3D randomly oriented Gaussian splats

  • What resolution?

– Again, application driven – 15x15x15

3/6/2014 Evaluation 26 Comp/Phys/APSc 715 Taylor

Compare to SQ Glyphs

  • Superquadric glyphs
  • Recently published for use in multivariate 3D scalar

vis.

  • 4 parameters

– 2 roundnesses – thickness – color

3/6/2014 Evaluation 27 Comp/Phys/APSc 715 Taylor

slide-10
SLIDE 10

3/3/2014 10

Legend?

  • 2D Legend?
  • 3DS

– It’s a 3D glyph, pointless for size-varying

  • SQ

– 4D parameter space. – Can’t show it all – Four examples: full range in 1 var, middle in others.

3/6/2014 Evaluation 28 Comp/Phys/APSc 715 Taylor 3/6/2014 Evaluation Comp/Phys/APSc 715 Taylor 29

Other Controls

  • Control skill level

– No mouse, keyboard – Spacebar for camera rotate, keypad for value selection

  • Control environment

– Dark room – 3D stereo glasses, Eye-separation corrected

3/6/2014 Evaluation 30 Comp/Phys/APSc 715 Taylor

slide-11
SLIDE 11

3/3/2014 11

Value Extraction

  • Easy in 2D, how to label a spot in 3D?

– Dot, sphere, cube…

  • Wireframe cube

– What color? – White probably a bad choice in-band for color). Oops.

  • Average value? Interpolated value?

– Confused users…

3/6/2014 Evaluation 31 Comp/Phys/APSc 715 Taylor 3/6/2014 Evaluation 32 Comp/Phys/APSc 715 Taylor

Between Sub vs. In Sub

  • Between subject

– Compare absolute performance of different participants between conditions – Compare Sphere avg to SQ average

  • In subject

– Compare relative performance of different participants between conditions. – Average improvement

3/6/2014 Evaluation 33 Comp/Phys/APSc 715 Taylor

slide-12
SLIDE 12

3/3/2014 12

How to pick?

  • Variability between subjects

– First-person-shooter-playing students might be better? – Experts vs. non-experts

  • Fewer controls Variability More subjects

– Oh boy…

  • Lead David to pick In-Subjects design

3/6/2014 Evaluation 34 Comp/Phys/APSc 715 Taylor

What to capture?

  • Ideally: everything

– System Interaction

  • Mouse events, keyboard events, etc

– Interviews – Timing – Performance

3/6/2014 Evaluation 35 Comp/Phys/APSc 715 Taylor

Details details…

  • How many subjects do we need?

– Run a pilot, ask stats person

  • How many trials should each participant do?

– Run a pilot, ask stats person

  • How much training to I need to do?

– Run a pilot

  • What age range to we sample?
  • Do you offer compensation? How much?
  • How much help to give?
  • What do you do with outliers?
  • How do I know this applies to my real data?

– Uh…

  • Help, I don’t know statistics!

– Me neither…

3/6/2014 Evaluation 36 Comp/Phys/APSc 715 Taylor

slide-13
SLIDE 13

3/3/2014 13

3/6/2014 Evaluation Comp/Phys/APSc 715 Taylor 37

Vector Visualization Redesign

  • Keller & Keller
  • How does wind velocity correlate with

temperature?

– Magnitude – Direction – Critical Points

3/6/2014 Evaluation 38 Comp/Phys/APSc 715 Taylor 3/6/2014 Evaluation Comp/Phys/APSc 715 Taylor 39