Vis tools and case studies Vis tools and case studies Thomas - - PowerPoint PPT Presentation

▶
vis tools and case studies vis tools and case studies
SMART_READER_LITE
LIVE PREVIEW

Vis tools and case studies Vis tools and case studies Thomas - - PowerPoint PPT Presentation

Jun 21, 2023 119 likes •1.4k views

9/3/2018 Visualization of machine learning algorithms Vis tools and case studies Vis tools and case studies Thomas Torsney-Weir VDA research group, University of Vienna http://localhost:8080/slides.html#/ 1/127 9/3/2018 Visualization of

slide-1
SLIDE 1

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 1/127

Vis tools and case studies Vis tools and case studies

Thomas Torsney-Weir VDA research group, University of Vienna

slide-2
SLIDE 2

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 2/127

Outline Outline

Introduction to Machine learning Vis helping machine learning Machine learning helping vis

slide-3
SLIDE 3

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 3/127

What is machine learning? What is machine learning?

Russell, Stuart, and Peter Norvig. Artificial intelligence: A modern approach, 2009.

Algorithms that can improve their performance based on training data "A rational agent is one that acts so as to achieve the best outcome or, when there is uncertainty, the best expected outcome"

slide-4
SLIDE 4

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 4/127

What is machine learning? What is machine learning?

You

slide-5
SLIDE 5

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 5/127

What is machine learning? What is machine learning?

You

INPUT x FUNCTION f: OUTPUT f(x)

slide-6
SLIDE 6

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 6/127

What is machine learning? What is machine learning?

You

INPUT x FUNCTION f: OUTPUT f(x)

X

slide-7
SLIDE 7

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 7/127

What is machine learning? What is machine learning?

INPUT x FUNCTION f: OUTPUT f(x)

slide-8
SLIDE 8

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 8/127

Types of algorithms Types of algorithms

Regression Classification Clustering

slide-9
SLIDE 9

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 9/127

What to use? What to use?

Regression: Predict continuous values Classification: Predict discrete values Clustering: Find distributions

slide-10
SLIDE 10

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 10/127

Regression Regression

Predict continuous values

Bishop, Christopher M. Pattern recognition and machine learning (information science and statistics), 2006.

slide-11
SLIDE 11

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 11/127

slide-12
SLIDE 12

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 12/127

Regression Regression

Predict continuous values

Bishop, Christopher M. Pattern recognition and machine learning (information science and statistics), 2006.

slide-13
SLIDE 13

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 13/127

slide-14
SLIDE 14

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 14/127

Regression Regression

Predict continuous values

Bishop, Christopher M. Pattern recognition and machine learning (information science and statistics), 2006.

slide-15
SLIDE 15

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 15/127

slide-16
SLIDE 16

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 16/127

Regression Regression

Predict continuous values

Bishop, Christopher M. Pattern recognition and machine learning (information science and statistics), 2006.

slide-17
SLIDE 17

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 17/127

slide-18
SLIDE 18

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 18/127

Regression Regression

Predict continuous values

Bishop, Christopher M. Pattern recognition and machine learning (information science and statistics), 2006.

slide-19
SLIDE 19

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 19/127

slide-20
SLIDE 20

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 20/127

Regression Regression

Predict continuous values

Bishop, Christopher M. Pattern recognition and machine learning (information science and statistics), 2006.

slide-21
SLIDE 21

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 21/127

slide-22
SLIDE 22

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 22/127

Regression Regression

Predict continuous values

Bishop, Christopher M. Pattern recognition and machine learning (information science and statistics), 2006.

slide-23
SLIDE 23

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 23/127

slide-24
SLIDE 24

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 24/127

Classification Classification

Predict discrete values

Bishop, Christopher M. Pattern recognition and machine learning (information science and statistics), 2006.

slide-25
SLIDE 25

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 25/127

slide-26
SLIDE 26

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 26/127

Clustering Clustering

Find distributions

Bishop, Christopher M. Pattern recognition and machine learning (information science and statistics), 2006.

slide-27
SLIDE 27

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 27/127

slide-28
SLIDE 28

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 28/127

Clustering Clustering

Find distributions

Bishop, Christopher M. Pattern recognition and machine learning (information science and statistics), 2006.

slide-29
SLIDE 29

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 29/127

slide-30
SLIDE 30

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 30/127

Clustering Clustering

Find distributions

Bishop, Christopher M. Pattern recognition and machine learning (information science and statistics), 2006.

slide-31
SLIDE 31

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 31/127

slide-32
SLIDE 32

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 32/127

Clustering Clustering

Find distributions

Bishop, Christopher M. Pattern recognition and machine learning (information science and statistics), 2006.

slide-33
SLIDE 33

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 33/127

slide-34
SLIDE 34

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 34/127

Clustering Clustering

Find distributions

Bishop, Christopher M. Pattern recognition and machine learning (information science and statistics), 2006.

slide-35
SLIDE 35

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 35/127

slide-36
SLIDE 36

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 36/127

Clustering Clustering

Find distributions

Bishop, Christopher M. Pattern recognition and machine learning (information science and statistics), 2006.

slide-37
SLIDE 37

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 37/127

slide-38
SLIDE 38

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 38/127

Types of ML algorithms Types of ML algorithms

Regression: Predict continuous values Classification: Predict discrete values Clustering: Find distributions

slide-39
SLIDE 39

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 39/127

Uses Uses

naive Bayes: spam filtering classification: recommender systems neural networks: handwriting recognition HMM: speech recognition

slide-40
SLIDE 40

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 40/127

Email filtering Email filtering

is email spam or not? use words as features

Sahami, Mehran, Susan Dumais, David Heckerman, and Eric Horvitz. “A bayesian approach to filtering junk e-mail,” 1998.

P(C=ck|X=x)= P(X=x|C=ck)P(C=ck) P(X=x)

slide-41
SLIDE 41

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 41/127

Recommender systems Recommender systems

http://blog.soton.ac.uk/hive/2012/05/10/recommendation-system-of-hive/

slide-42
SLIDE 42

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 42/127

Handwriting recognition Handwriting recognition

slide-43
SLIDE 43

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 43/127

Speech recognition Speech recognition

http://recognize-speech.com/images/LanguageModel/left_to_right_HMM.png

slide-44
SLIDE 44

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 44/127

Vis and ML Vis and ML

both vis and ML seem to have similar goals: make sense of complex data

Machine learning Machine learning

INPUT x FUNCTION f: OUTPUT f(x)

Visualization Visualization

Morton, Kristi, Ross Bunker, Jock Mackinlay, Robert Morton, and Chris Stolte. “Dynamic workload driven data integration in Tableau,” 2012.

slide-45
SLIDE 45

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 45/127

Who helps whom? Who helps whom?

both! both!

Vis helps ML: evaluating models ML helps vis: ML for embedded analysis

slide-46
SLIDE 46

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 46/127

Why is this hard? Why is this hard?

Machine learning Machine learning

Fast algorithms Sufficient data Automatic learning

Visualization Visualization

Multi-dimensional spaces Comparing complex data Showing uncertainty

slide-47
SLIDE 47

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 47/127

Vis helping ML Vis helping ML

slide-48
SLIDE 48

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 48/127

Vis helping ML Vis helping ML

How do they work together? Building models Validating models Understanding models

slide-49
SLIDE 49

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 49/127

Building models Building models

slide-50
SLIDE 50

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 50/127

Building models Building models

Meta parameters Model selection

slide-51
SLIDE 51

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 51/127

What are meta parameters? What are meta parameters?

Meta parameters control how learning takes place Learning rate Number and size of network layers Slack variables Stopping conditions

slide-52
SLIDE 52

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 52/127

Why study meta-parameters? Why study meta-parameters?

slide-53
SLIDE 53

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 53/127

Why study meta-parameters? Why study meta-parameters?

slide-54
SLIDE 54

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 54/127

Manual method Manual method

1.0 0.0 0.5 0.5 1.0

slide-55
SLIDE 55

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 55/127

Manual method Manual method

1.0 0.0 0.5 0.5 1.0

slide-56
SLIDE 56

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 56/127

How to study them? How to study them?

run a bunch of models and examine outputs paramorama design galleries

slide-57
SLIDE 57

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 57/127

Paramorama Paramorama

Manual sampling Automated sampling

Pretorius, A. Johannes, Mark-Anthony P. Bray, Anne E. Carpenter, and Roy A. Ruddle. “Visualization of parameter space for image analysis,” 2011.

slide-58
SLIDE 58

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 58/127

Paramorama Paramorama

Pretorius, A. Johannes, Mark-Anthony P. Bray, Anne E. Carpenter, and Roy A. Ruddle. “Visualization of parameter space for image analysis,” 2011.

slide-59
SLIDE 59

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 59/127

Paramorama Paramorama

Pretorius, A. Johannes, Mark-Anthony P. Bray, Anne E. Carpenter, and Roy A. Ruddle. “Visualization of parameter space for image analysis,” 2011.

slide-60
SLIDE 60

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 60/127

Design galleries Design galleries

Marks, Joe, Brad Andalman, Paul A. Beardsley, William Freeman, Sarah Gibson, Jessica Hodgins, Thomas Kang, et al. “Design Galleries: A general approach to setting parameters for computer graphics and animation,” 1997.

slide-61
SLIDE 61

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 61/127

slide-62
SLIDE 62

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 62/127

How to study them? How to study them?

use a more principled approach

slide-63
SLIDE 63

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 63/127

Objective measures Objective measures

Image Ground truth Dice: 0.85

  • Error: 0.25
  • ...
slide-64
SLIDE 64

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 64/127

Visual parameter space exploration Visual parameter space exploration

conceptual pipeline

Michael Sedlmair, Christoph Heinzl, Stefan Bruckner, Harald Piringer, and Torsten Möller "Visual parameter space analysis: A conceptual framework" IEEE Transactions on Visualization and Computer Graphics. 20(12) 2014.

slide-65
SLIDE 65

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 65/127

Tuner Tuner

  • Torsney-Weir, Thomas, Ahmed Saad, Torsten Möller, Britta Weber, Hans-Christian Hege, Jean-Marc Verbavatz, and Steven Bergner. “Tuner: Principled

parameter finding for image segmentation algorithms using visual response surface exploration,” 2011.

slide-66
SLIDE 66

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 66/127

slide-67
SLIDE 67

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 67/127

Tuner Tuner

Torsney-Weir, Thomas, Ahmed Saad, Torsten Möller, Britta Weber, Hans-Christian Hege, Jean-Marc Verbavatz, and Steven Bergner. “Tuner: Principled parameter finding for image segmentation algorithms using visual response surface exploration,” 2011.

slide-68
SLIDE 68

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 68/127

slide-69
SLIDE 69

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 69/127

Tuner Tuner

Torsney-Weir, Thomas, Ahmed Saad, Torsten Möller, Britta Weber, Hans-Christian Hege, Jean-Marc Verbavatz, and Steven Bergner. “Tuner: Principled parameter finding for image segmentation algorithms using visual response surface exploration,” 2011.

slide-70
SLIDE 70

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 70/127

slide-71
SLIDE 71

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 71/127

Tuner Tuner

Torsney-Weir, Thomas, Ahmed Saad, Torsten Möller, Britta Weber, Hans-Christian Hege, Jean-Marc Verbavatz, and Steven Bergner. “Tuner: Principled parameter finding for image segmentation algorithms using visual response surface exploration,” 2011.

slide-72
SLIDE 72

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 72/127

Building models Building models

Meta parameters can have a large influence on performance Almost all ML algorithms require tuning Manual tuning is time consuming and error prone

slide-73
SLIDE 73

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 73/127

Validating and verifying models Validating and verifying models

slide-74
SLIDE 74

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 74/127

What do we mean? What do we mean?

How do we know our models are working? model selection

Committee on Mathematical Foundations of Verification, Validation, and Uncertainty Quantification; Board on Mathematical Sciences and Their Applications, Division on Engineering and Physical Sciences, National Research Council. Assessing the reliability of complex models: Mathematical and statistical foundations of verification, validation, and uncertainty quantification, 2012. . http://www.nap.edu/openbook.php?record_id=13395

slide-75
SLIDE 75

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 75/127

slide-76
SLIDE 76

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 76/127

Validating and verifying models Validating and verifying models

Summary statistics are not always enough Balancing multiple objectives is difficult Certain training points might be very important

slide-77
SLIDE 77

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 77/127

Examples Examples

HyperMoVal - local inspection Sliceplorer - global inspection Tuner - error inspection

slide-78
SLIDE 78

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 78/127

HyperMoVal HyperMoVal

Piringer, Harald, Wolfgang Berger, and Jurgen Krasser. “HyperMoVal: Interactive visual validation of regression models for real-time simulation,” 2010.

slide-79
SLIDE 79

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 79/127

Sliceplorer views Sliceplorer views

Single layer NN (26 nodes) SVM (polynomial kernel) Dual layer NN (5 and 3 nodes) SVM (RBF kernel)

Torsney-Weir, Thomas, Michael Sedlmair, and Torsten Möller. “Sliceplorer,” 2017.

slide-80
SLIDE 80

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 80/127

Tuner error views Tuner error views

Examining multi-dimensional functions error view shows where model is unsure can visually verify the model

Prediction Prediction

Error view

Optimization Optimization

Error view

slide-81
SLIDE 81

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 81/127

Validating and verifying models Validating and verifying models

Understand fit for individual samples Visual inspection to understand extrapolation Uncertainty can help to understand quality of prediction

slide-82
SLIDE 82

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 82/127

Understanding models Understanding models

slide-83
SLIDE 83

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 83/127

Who needs this? Who needs this?

models are complex the business world likes spreadsheets because they can walk through the calculations

slide-84
SLIDE 84

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 84/127

Simple vs complex models Simple vs complex models

Simple Simple

few factors small integer factors low-depth trees

Complex Complex

multi-layer neural network Gaussian process model non-linear many decisions

slide-85
SLIDE 85

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 85/127

What does complexity buy us? What does complexity buy us?

Global vs local models Deep-learning networks can deal with feature selection Can deal with edge cases

slide-86
SLIDE 86

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 86/127

Understanding models Understanding models

Just an answer is not enough (show your work) Humans have trouble understanding complex models Interactivity can bring people into the model

slide-87
SLIDE 87

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 87/127

Methods Methods

interaction walkthroughs simpler models ala LIME (Ribeiro et al. 2016) direct inspection

slide-88
SLIDE 88

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 88/127

Examples Examples

regression: Muhlbacher and Piringer clustering: Dis-function text processing: TagRefinery smaller models: Explanation explorer

slide-89
SLIDE 89

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 89/127

Mühlbacher and Piringer Mühlbacher and Piringer

Directly interact with the model building process

Mühlbacher, Thomas, and Harald Piringer. “A partition-based framework for building and validating regression models,” 2013. Best Paper Award.

slide-90
SLIDE 90

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 90/127

slide-91
SLIDE 91

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 91/127

Dis-function Dis-function

Build a distance function interactively

Brown, Eli T, Jingjing Liu, Carla E Brodley, and Remco Chang. “Dis-Function: Learning Distance Functions Interactively,” 2012.

slide-92
SLIDE 92

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 92/127

slide-93
SLIDE 93

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 93/127

TagRefinery TagRefinery

Tutorial/walkthrough system Text processing pipeline

Kralj, Christoph, Mohsen Kamalzadeh, and Torsten Möller. “TagRefinery: A visual tool for tag wrangling,” 2017.

slide-94
SLIDE 94

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 94/127

TagRefinery TagRefinery

Kralj, Christoph, Mohsen Kamalzadeh, and Torsten Möller. “TagRefinery: A visual tool for tag wrangling,” 2017.

slide-95
SLIDE 95

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 95/127

TagRefinery TagRefinery

Kralj, Christoph, Mohsen Kamalzadeh, and Torsten Möller. “TagRefinery: A visual tool for tag wrangling,” 2017.

slide-96
SLIDE 96

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 96/127

LIME method LIME method

Ribeiro, Marco Tulio, Sameer Singh, and Carlos Guestrin. “‘Why should I trust you?’: Explaining the predictions of any classifier,” 2016.

slide-97
SLIDE 97

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 97/127

slide-98
SLIDE 98

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 98/127

Explanation explorer Explanation explorer

Krause, Josua, Aritra Dasgupta, Jordan Swartz, Yindalon Aphinyanaphongs, and Enrico Bertini. “A workflow for visual diagnostics of binary classifiers using instance-level explanations,” 2017.

slide-99
SLIDE 99

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 99/127

slide-100
SLIDE 100

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 100/127

Direct inspection Direct inspection

e.g. hidden states in a neural network

slide-101
SLIDE 101

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 101/127

DeepEyes DeepEyes

How many layers are needed?

Pezzotti, Nicola, Thomas Höllt, Jan van Gemert, Boudewijn Lelieveldt, Elmar Eisemann, and Anna Vilanova. “DeepEyes: Progressive visual analytics for designing deep neural networks,” 2018.

slide-102
SLIDE 102

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 102/127

DeepEyes DeepEyes

How stable are the layers?

Pezzotti, Nicola, Thomas Höllt, Jan van Gemert, Boudewijn Lelieveldt, Elmar Eisemann, and Anna Vilanova. “DeepEyes: Progressive visual analytics for designing deep neural networks,” 2018.

slide-103
SLIDE 103

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 103/127

DeepEyes DeepEyes

How discriminating are the layers?

Pezzotti, Nicola, Thomas Höllt, Jan van Gemert, Boudewijn Lelieveldt, Elmar Eisemann, and Anna Vilanova. “DeepEyes: Progressive visual analytics for designing deep neural networks,” 2018.

slide-104
SLIDE 104

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 104/127

Vis helping ML Vis helping ML

How do they work together? Building models Validating models Understanding models What about the other way?

slide-105
SLIDE 105

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 105/127

Machine learning helping vis Machine learning helping vis

slide-106
SLIDE 106

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 106/127

How? How?

Use the strengths of ML and vis together machines are good at calculating humans are good at intuition Vis assisted by ML algorithms

Sacha, D., A. Stoffel, F. Stoffel, Bum Chul Kwon, G. Ellis, and Daniel A Keim. “Knowledge generation model for visual analytics,” 2014.

slide-107
SLIDE 107

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 107/127

Book ad! Book ad!

slide-108
SLIDE 108

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 108/127

How? How?

ML provides data aggregation/filtering/selection User can steer algo to produce desired results

slide-109
SLIDE 109

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 109/127

Examples Examples

Cluster and calendar view - clustering KeyVis - clustering FluidExplorer - clustering Cell Cognition Explorer - active learning

slide-110
SLIDE 110

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 110/127

Cluster and calendar view Cluster and calendar view

Understanding power consumption When do people use the most power? What are seaonal patterns? How does energy use change at different scales Days Weeks Months

slide-111
SLIDE 111

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 111/127

Cluster and calendar view Cluster and calendar view

Van Wijk, J.J., and E.R. Van Selow. “Cluster and calendar based visualization of time series data,” 1999. . http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=801851

slide-112
SLIDE 112

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 112/127

slide-113
SLIDE 113

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 113/127

Cluster and calendar view Cluster and calendar view

Van Wijk, J.J., and E.R. Van Selow. “Cluster and calendar based visualization of time series data,” 1999. . http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=801851

slide-114
SLIDE 114

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 114/127

slide-115
SLIDE 115

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 115/127

Cluster and calendar view Cluster and calendar view

Van Wijk, J.J., and E.R. Van Selow. “Cluster and calendar based visualization of time series data,” 1999. . http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=801851

slide-116
SLIDE 116

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 116/127

Cluster and calendar view Cluster and calendar view

Van Wijk, J.J., and E.R. Van Selow. “Cluster and calendar based visualization of time series data,” 1999. . http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=801851

slide-117
SLIDE 117

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 117/127

slide-118
SLIDE 118

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 118/127

KeyVis KeyVis

How to find related work? How helpful are keywords? Do keywords relate to each

  • ther?

Isenberg, Petra, Tobias Isenberg, Michael Sedlmair, Jian Chen, and Torsten Möller. “Visualization as seen through its research paper keywords,” 2017. . https://tobias.isenberg.cc/VideosAndDemos/Isenberg2017VST

slide-119
SLIDE 119

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 119/127

KeyVis KeyVis

step 1: cluster the papers based on keywords

Isenberg, Petra, Tobias Isenberg, Michael Sedlmair, Jian Chen, and Torsten Möller. “Visualization as seen through its research paper keywords,” 2017. . https://tobias.isenberg.cc/VideosAndDemos/Isenberg2017VST

slide-120
SLIDE 120

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 120/127

KeyVis KeyVis

step 2: give an interface to this clustering

Isenberg, Petra, Tobias Isenberg, Michael Sedlmair, Jian Chen, and Torsten Möller. “Visualization as seen through its research paper keywords,” 2017. . https://tobias.isenberg.cc/VideosAndDemos/Isenberg2017VST

slide-121
SLIDE 121

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 121/127

FluidExplorer FluidExplorer

Bruckner, Stefan, and Torsten Möller. “Result-driven exploration of simulation parameter spaces for visual effects design,” 2010. . http://www.ncbi.nlm.nih.gov/pubmed/20975188

slide-122
SLIDE 122

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 122/127

FluidExplorer FluidExplorer

Bruckner, Stefan, and Torsten Möller. “Result-driven exploration of simulation parameter spaces for visual effects design,” 2010. . http://www.ncbi.nlm.nih.gov/pubmed/20975188

slide-123
SLIDE 123

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 123/127

Cell Cognition Explorer Cell Cognition Explorer

Sommer, Christoph, Rudolf Hoefler, Matthias Samwer, and Daniel W. Gerlich. “A Deep Learning And Novelty Detection Framework For Rapid Phenotyping In High-Content Screening,” 2017. . http://dx.doi.org/10.1091/mbc.E17-05-0333

slide-124
SLIDE 124

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 124/127

slide-125
SLIDE 125

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 125/127

The future! The future!

slide-126
SLIDE 126

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 126/127

Interesting projects Interesting projects

More using ML to build models for vis tools More generalized tools Understand what "understandability" means

slide-127
SLIDE 127

9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 127/127

Thanks! Thanks!

thomas.torsney-weir@univie.ac.at http://www.tomtorsneyweir.com