Past, Present, and Future of Multidimensional Scaling Patrick J. F. - - PowerPoint PPT Presentation
Past, Present, and Future of Multidimensional Scaling Patrick J. F. Groenen *Econometric Institute, Erasmus University Rotterdam, The Netherlands, groenen@few.eur.nl, http://people.few.eur.nl/groenen/ Summary: 1 What is MDS? 2 Some Historical
Patrick J. F. Groenen
*Econometric Institute, Erasmus University Rotterdam, The Netherlands, groenen@few.eur.nl, http://people.few.eur.nl/groenen/
Summary: 1 What is MDS? 2 Some Historical Milestones 3 Present 4 Future 5 Summary of highlights in MDS
Past, Present, and Future of MDS – 2 –
Bor- deaux Brest Lille Lyon Mar- seille Nice Parijs Strassb
Tou- louse Tours Bordeaux Brest 9h58 Lille 6h39 7h11 Lyon 8h05 7h11 4h52 Marseille 5h47 8h49 6h12 1h35 Nice 8h30 13h36 8h20 4h33 2h26 Parijs 2h59 4h17 1h04 2h01 3h00 5h52 Strassbourg 8h08 10h16 6h54 4h36 7h04 11h15 4h01 Toulouse 2h02 13h52 9h42 4h25 3h26 6h29 5h14 10h56 Tours 2h36 5h38 4h17 4h21 5h13 9h04 1h13 6h03 6h06
Marseille Toulouse Bordeaux Lyon Nice Paris Tours Lille Strassbourg Brest Marseille Toulouse Bordeaux Lyon Nice Paris Tours Lille Strassbourg Brest
MDS map of travel time by train. Geographic map of France.
Past, Present, and Future of MDS – 3 –
dissimilarity matrix
O1 O2 O3
On O1 O2 δ12 O3 δ13 δ23
δ1,n-1 δ2,n-1 δ3,n-1 On δ1n δ2n δ3n
dim 1 dim 2 O1 x11 x12 O2 x21 x22 O3 x31 x32
xn-1,1 xn-1,2 On xn1 xn1 O
1
O
2
O
3
O
n
O
n-1
Past, Present, and Future of MDS – 4 –
Multidimensional scaling (MDS) is a method that represents measurements of similarity (or dissimilarity) among pairs of objects as distances between points of a low- dimensional space.
– psychology, – medicine, – sociology, – chemistry, – archaeology, – network analysis – biology, – economists, etc.
– Large similarity approximated by small distance in MDS. – Large dissimilarity (δij) approximated by large distance in MDS. – General term: proximity.
Past, Present, and Future of MDS – 5 –
Map of Durham county – Cartographer: Jacob van Langren – Date 1635 Newcastle Durham
Past, Present, and Future of MDS – 6 –
Provides a solution for classical MDS based on eigendecomposition
Provides independently the same solution for classical MDS and gives connection to principal components analysis. Classical MDS: minimize Strain(X) = 1/4||J(∆ ∆ ∆ ∆(2)–D(2)(X))J||2 with J centering matrix by eigendecomposition of –½ J∆ ∆ ∆ ∆(2)J
Past, Present, and Future of MDS – 7 –
Provides a solution for classical MDS based on eigendecomposition
Provides independently the same solution for classical MDS and gives connection to principal components analysis.
Provides a heuristic for MDS.
Past, Present, and Future of MDS – 8 –
Provides a solution for classical MDS based on eigendecomposition
Provides independently the same solution for classical MDS and gives connection to principal components analysis.
Provides a heuristic for MDS.
Establishes least-squares MDS. Provides a minimization algorithm. Proposes ordinal MDS plus optimization Minimize Stress-I: σI(X,d ˆ ) =
<
j i ij j i ij ij
2 2
ij
d ˆ disparity satisfying a monotone relation with proximities.
Past, Present, and Future of MDS – 9 –
– Classic example: Rothkopf (1957) Morse code confusion data + Is there some systematic way in which people confuse Morse codes? + 36 Morse code (26 for alphabet, 10 for numbers) + Subjects task: judge whether two Morse codes are the same or not. For example: + Is .- (N) the same as .-. (R)? Yes (1), or no (2) + Stimulus pair presented in two orders: pair NR and RN. + Each subject judges many combinations of Morse codes. + N = 598. + Morse code confusion table: proportion confused. + Data are similarities A B C D
A 92 4 6 13
B 5 84 37 31
C 4 38 87 17
D 8 62 17 88
3 11 2
Past, Present, and Future of MDS – 10 –
– Classic example: Rothkopf (1957) Morse code confusion data .-
Past, Present, and Future of MDS – 11 –
Provides a solution for classical MDS based on eigendecomposition
Provides independently the same solution for classical MDS and gives connection to principal components analysis.
Provides a heuristic for MDS.
Establishes least-squares MDS. Provides a minimization algorithm. Proposes ordinal MDS plus optimization
Facet theory and regional interpretation in MDS. – In facet theory, extra information (external variables) is available on the objects according to the facet design by which the objects are generated:
Past, Present, and Future of MDS – 12 –
Facet theory and regional interpretation in MDS. + Every object i belongs to a category on one or more facets. + See, e.g., Guttman (1959), Borg & Shye (1995), Borg & Groenen (1997, 1998) Dissimilarity matrix : Facet design
O1 O2 O3
On 1 2 3 O1 O1 1 1 3 O2 δ12 O2 1 2 3 O3 δ13 δ23 O3 2 1 3
On-1 3 1 1 On δ1n δ2n δ3n
On 3 2 1
– The extra facet information is used to partition the objects in the MDS space in regions. – Facets are used for regional hypotheses about the empirical structure of the data.
a a a a a b b b b b c c c c a a a a b b b b b b c c c c c c c a a a a a a a b b b c c c c b b c c axial modular polar
Past, Present, and Future of MDS – 13 –
Facet theory and regional interpretation in MDS. – For the Morse code data, we have additional information available:
Letter Morse code Length Signal type Letter Morse code Length Signal type A .- 25 1=2 S ... 25 1 B
45 1>2 T
2 C
55 1=2 U ..- 35 1>2 D
35 1>2 V ...- 45 1>2 E . 05 1 W .-- 45 1<2 F ..-. 45 1>2 X
55 1=2 G
45 1<2 Y
65 1<2 H .... 35 1 Z
55 1=2 I .. 15 1 1 .---- 85 1<2 J .--- 65 1<2 2 ..--- 75 1<2 K
45 1<2 3 ...-- 65 1>2 L .-.. 45 1>2 4 ....- 55 1>2 M
2 5 ..... 45 1 N
25 1=2 6
55 1>2 O
2 7
65 1>2 P .--. 55 1=2 8
75 1<2 Q
65 1<2 9
85 1<2 R .-. 35 1>2
1 S ... 25 1
Past, Present, and Future of MDS – 14 –
Facet theory and regional interpretation in MDS. – Borg and Groenen (2005): Regional restrictions through Proxscal, by specifying: + two dimensions + two external variables, + each variable is transformed ordinally using the primary approach ties.
1 1>2 1=2 2>1 2 95 85 75 65 55 45 35 05 25 15 1111 112 121 211 111 12 21 11 122 212 2121 2211 2122 2212 1222 22221 222 1221 221 22 2 1 11222 22211 11122 22111 21111 11112 11111 1112 1121 1211 2111 2112 12222 22222 1 1>2 1=2 2>1 2 95 85 75 65 55 45 35 25 15 05 12 2111 2121 211 1 1121 221 1111 11 1222 212 1211 22 21 222 1221 2212 121 111 2 112 1112 122 2112 2122 2211 12222 11222 11122 11112 11111 21111 22111 22211 22221 22222
Unconstrained Regionally constrained
Past, Present, and Future of MDS – 15 –
Provides a solution for classical MDS based on eigendecomposition
Provides independently the same solution for classical MDS and gives connection to principal components analysis.
Provides a heuristic for MDS.
Establishes least-squares MDS. Provides a minimization algorithm. Proposes ordinal MDS plus optimization
Facet theory and regional interpretation in MDS.
Dimension weighting models in 3-way MDS 1970: Carroll and Chang: Introduction (INDSCAL, IDIOSCAL)
Past, Present, and Future of MDS – 16 –
Dimension weighting models in 3-way MDS 1970: Carroll and Chang: Introduction (INDSCAL, IDIOSCAL) – 3-way MDS: more than one dissimilarity matrix: – In the weighted Euclidean model or each source, the common space G may be stretched or shrunk along the axes. – Model δijk ≈ dij(GSk) with + G a single common space and + Sk is a diagonal matrix of dimension weights and – INDSCAL uses STRAIN loss.
J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J
∆ ∆ ∆ ∆ 1 ∆ ∆ ∆ ∆ 2 ∆ ∆ ∆ ∆ 3
Common space s 11 = 1.5 s 12 = .5 s 21 = .8 s 22 = 1.5 s 31 = 1 s 32 = .3
Past, Present, and Future of MDS – 17 –
Dimension weighting models in 3-way MDS 1970: Carroll and Chang: Introduction (INDSCAL, IDIOSCAL) – 3-way MDS: more than one dissimilarity matrix: – In the weighted Euclidean model or each source, the common space G may be stretched or shrunk along the axes. – In the generalized Euclidean model, the common space G may rotated, then stretched
– Model δijk ≈ dij(GSk) with + G a single common space and + Sk is any matrix of dimension weights – IDIOSCAL uses STRAIN loss.
J J J J J J J J J J J J J J J J
∆ ∆ ∆ ∆ 1 ∆ ∆ ∆ ∆ 2 ∆ ∆ ∆ ∆ 3
Common space α
3
= -10
= 1 s 32 = .3
J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J
α
2
= 45
= .8 s 22 = .5 α
1
= 30
= 1.2 s 12 = .5
Past, Present, and Future of MDS – 18 –
Provides a solution for classical MDS based on eigendecomposition
Provides independently the same solution for classical MDS and gives connection to principal components analysis.
Provides a heuristic for MDS.
Establishes least-squares MDS. Provides a minimization algorithm. Proposes ordinal MDS plus optimization
Facet theory and regional interpretation in MDS.
Dimension weighting models in 3-way MDS 1970: Carroll and Chang: Introduction (INDSCAL, IDIOSCAL)
Past, Present, and Future of MDS – 19 –
2.1 Milestones in MDS algorithms
Torgerson & Gower: solutions for classical MDS.
Introduction Stress-I loss function plus minimization and ordinal MDS.
Introduction SMACOF (Scaling by MAjorizing a COomplicated function) algorithm for MDS.
MDS algorithm allowing transformations of the dissimilarities, constraints on the configuration, and three-way dimension weighting extensions.
Convergence results derived of the SMACOF algorithm.
distances. De Leeuw Heiser Mathar
Past, Present, and Future of MDS – 20 –
σr(X, d ˆ ) = w ij
i <j
d
ij − d ij (X)
2
with wij ≥ 0 and δij ≥ 0 where ˆ d
ij
disparity, d–hat, pseudo-distance: optimal transformation of dissimilarities subject to (ordinal) restrictions and
i ij ijd
w
2
ˆ = n(n–1)/2 to avoid the trivial solution d ˆ =0 and X=0 dij(X) Euclidean distance between rows i and j of X X n×p matrix of coordinates of n objects by p dimensions wij nonnegative weights (for example, to code missings)
Past, Present, and Future of MDS – 21 –
– Easy to combine majorization with constraints. – The majorizing function ˆ σ (X,Y) can be conveniently expressed as ˆ σ (X,Y) =
2 δ
η + tr XV'X – 2 tr X'B(Y)Y =
2 δ
η + tr XV'X – 2 tr X'VX =
2 δ
η + (tr XV'X + tr X 'VX – 2 tr X'VX ) – tr X 'VX =
2 δ
η + tr(X – X )'V(X – X ) – tr X 'VX with Y the previous configuration X the unconstrained update
2 δ
η the sum of squared dissimilarities V a fixed (positive semi-definite) matrix depending on the weights. Quadratic in X Constant Constant
Past, Present, and Future of MDS – 22 –
∆ ∆ ∆ ∆1 ∆ ∆ ∆ ∆2 ∆ ∆ ∆ ∆3
– Any constraint on X that is solved easily by minimizing least squares error, e.g., the linear constraints X = ZC (for given Z) – Three-way MDS through constrained MDS(De Leeuw & Heiser, 1980): – Minimize σr(G,S1,S2,...,Sk)=
= < K k j i ijk
w
1
2
where + G is the n×p matrix of coordinates (the common space) + Sk is the p×p matrix of weights – Consider the block matrices: * =
3 2 1
3 2 1
3 2 1
Past, Present, and Future of MDS – 23 –
Meulman: integration of (nonlinear) multivariate analysis and MDS. – Much emphasis on the representation of objects, less on the variables. – Fitting by MDS through Stress as a dimension reduction technique. – Including a wide variety of MVA techniques: + (Nonlinear) PCA + Multiple Correspondence Analysis + Correspondence Analysis + Generalized Canonical Correlation Analysis + Discriminant Analysis.
Past, Present, and Future of MDS – 24 –
Meulman: integration of (nonlinear) multivariate analysis and MDS.
Buja: Constant dissimilarities – Data with all δij = 1 can be seen as maximum noninformative in MDS, since all pairs of objects are equally dissimilar. – Suppose = c
1 1 1 1 1 1 1 1 1 1 1 with c > 0 – What configuration does MDS yield with constant data?
Past, Present, and Future of MDS – 25 –
Meulman: integration of (nonlinear) multivariate analysis and MDS.
Buja: Constant dissimilarities – Data with all δij = 1 can be seen as maximum noninformative in MDS, since all pairs of objects are equally dissimilar. – Suppose = c
1 1 1 1 1 1 1 1 1 1 1 with c > 0 – What configuration does MDS yield with constant data? – Buja, Logan, Reeds, & Shepp (1994) proved: 1 dimensional 2 dimensional 3 dimensional or higher points equally spaced points on points on a sphere
concentric circles
Past, Present, and Future of MDS – 26 –
Meulman: integration of (nonlinear) multivariate analysis and MDS.
Buja: Constant dissimilarities
Various authors: local minima in MDS:
Daniel Defays Larry Hubert & Mathew Hesson-McInnes Phipps Arabie Jose Fernando Vera
Past, Present, and Future of MDS – 27 –
– When do local minima occur? + Unidimensional scaling (De Leeuw & Heiser, 1977; Defays, 1978; Hubert and Arabie 1986; Pliner, 1996). + City-block MDS (Hubert, Arabie & Hesson-McInnes, 1992). + Depends on data: with increasing dimensionality fewer local minima and error structure.
Past, Present, and Future of MDS – 28 –
– When do local minima occur? + Unidimensional scaling (De Leeuw & Heiser, 1977; Defays, 1978; Hubert and Arabie 1986; Pliner, 1996). + City-block MDS (Hubert, Arabie & Hesson-McInnes, 1992). + Depends on data: with increasing dimensionality fewer local minima and error structure. – What can you do about local minima? + Multiple random starts. + Tunneling (Groenen & Heiser, 1996) + Distance smoothing: unidimensional scaling (Pliner, 1996), city-block MDS, general MDS (Groenen et al. 1999). + Meta heuristics + simulated annealing: De Soete, Hubert, Arabie (1988), Brusco (2001), Vera & Heiser (2005, 2007) + genetic algorithm, …..
Past, Present, and Future of MDS – 29 –
Meulman: integration of (nonlinear) multivariate analysis and MDS.
Constant dissimilarities
Various authors: local minima in MDS:
Applying weights in Stress to mimic loss functions + Choose wij =
2 − ij
− δ
j i ij ij ij
w
2
) ( d X =
−
−
j i ij ij ij 2 2
) ( d X δ δ = <
j i ij ij 2
ij
=
− =
ij ij ij
e X small
ij
=
− =
ij ij ij
e X + This is a very good idea to attach equal importance to small and large errors.
Past, Present, and Future of MDS – 30 –
Meulman: integration of (nonlinear) multivariate analysis and MDS.
Constant dissimilarities
Various authors: local minima in MDS:
Applying weights in Stress to mimic loss functions (after Buja, 1998).
Past, Present, and Future of MDS – 31 –
SMACOF in R.
Past, Present, and Future of MDS – 32 –
SMACOF in R.
Large scale MDS ISOMAP heuristic for.
– Problems: + Computationally too demanding. + Storage is a problem (n2). + Uninformative solutions.
10 100 1000 10000 .01 .1 1 10 100 1000 n CPU seconds
Past, Present, and Future of MDS – 33 –
– Solution Groenen, Trosset, Kagie: + Use only a fraction of the data. + Make use of smart designs. + Use sparseness of the data efficiently to obtain a fast majorization algorithm. – Comparison large scale majorization versus SMACOF – n = 1,000 – Proportion nonmissing: .05 (Nnonmis = 23,000 out of 499,500)
0.5 1 1.5 2 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 CPU seconds Stress Large scale majorization SMACOF
– n = 10,000 – Proportion nonmissing: .005 (Nnonmis = 250,000 out of 49,995,000)
50 100 150 200 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 CPU seconds Stress Large scale majorization SMACOF
Past, Present, and Future of MDS – 34 –
– Avoiding uninformative solutions: + Edinburgh Associative Thesaurus (EAT) data set (1968, 1971): words associated with stimulus + cij contains the number of associations between words i and j. + n = 23,219 terms. + 325,060 nonzero association counts between terms (sparseness = 0.1%). + Solution when choosing: δij = 1/cij
Past, Present, and Future of MDS – 35 –
+ Solution when choosing the gravity model δij =
ij j i
5 ij
Past, Present, and Future of MDS – 36 –
Past, Present, and Future of MDS – 37 –
SMACOF in R.
Large scale MDS ISOMAP heuristic.
Dynamic MDS visualization in the G-GVis software.
Dynamic MDS visualization through iMDS. Andreas Buja, Deborah Swayne, Di Cook
Past, Present, and Future of MDS – 38 –
SMACOF in R.
Large scale MDS ISOMAP heuristic.
Dynamic MDS visualization in the G-GVis software.
Dynamic MDS visualization through iMDS.
Symbolic MDS of intervals
Symbolic MDS of histograms
Past, Present, and Future of MDS – 39 –
0.2 0.4 0.6
0.2 0.4 0.6 1 2 3 4 5 6 7 8 9 10 d28
(L)
d28
(U)
Symbolic MDS of interval data – Instead of δij, its interval is given: δij ∈ [
) ( ) ( , U ij L ij
δ δ ] – Find MDS with coordinates xis also in an interval. – Minimize I-Stress: ) , (
2
R X
I
σ =
−
j i L ij L ij ij
d w
2 ) ( ) (
) , ( R X δ +
−
j i U ij U ij ij
d w
2 ) ( ) (
) , ( R X δ with ) , (
) (
R X
L ij
d longmallest distance between rectangles ) , (
) (
R X
U ij
d longest distance between rectangles
Past, Present, and Future of MDS – 40 –
Symbolic MDS of histograms – Instead of δij, its empirical distribution (percentiles) are given: α α α α = [.20, .30, .40] Lower bound Upper bound k α α α α percentile
) (L ijk
δ percentile
) (U ijk
δ 1 .20 20
) ( 1 L ij
δ 80
) ( 1 U ij
δ 2 .30 30
) ( 2 L ij
δ 70
) ( 2 U ij
δ 3 .40 40
) ( 3 L ij
δ 60
) ( 3 U ij
δ – Minimize ) ,..., , (
1 2 K HistI
R R X σ =
−
k j i k L ij L ijk ij
d w
2 ) ( ) (
) , ( R X δ +
−
k j i k U ij U ijk ij
d w
2 ) ( ) (
) , ( R X δ subject to 0 ≤ ris1 ≤ ris2 ≤ … ≤ risK.
Past, Present, and Future of MDS – 41 –
SMACOF in R.
Large scale MDS ISOMAP heuristic.
Dynamic MDS visualization in the G-GVis software.
Dynamic MDS visualization through iMDS.
Symbolic MDS of intervals
Symbolic MDS of histograms
Dynamic MDS of Dutch political parties
Past, Present, and Future of MDS – 42 –
to rate 30 politcal statements (www.stemwijzer.nl), e.g.,
disappear at the latest in 2015. Agree Don’t know Disagree
Agree Don’t know Disagree – 11 political parties also rated these 30 items. – What is the political landscape in the Dutch elections of 2010? – Do iMDS on the distances between the 11 parties in 30 dimensional space.
Past, Present, and Future of MDS – 43 –
Past Main author(s) Topic 1958, 1966 Torgerson,Gower Classical MDS 1964 Kruskal Least-squares MDS through Stress with transformations 1964 Guttman Facet theory and regional interpretations in MDS 1969, 1970 Horan, Carroll Three-way MDS models (INDSCAL, IDIOSCAL) 1977- De Leeuw and others The majorization algorithm for MDS Present 1986-1998 Meulman Distance-based MVA through MDS 1994 Buja Constant dissimilarities 1978, 1995- Various Local minimum problem 1998 Buja Smart use of weights in MDS
Past, Present, and Future of MDS – 44 –
Future 1999, Heiser, Meulman, Busing Modern MDS software: Proxscal in SPSS (PASW) 2000 Tenenbaum, et al. Large scale MDS ISOMAP heuristic 2002 Buja, Swayne, Cook Dynamic MDS in GGvis (part of GGobi) 2003 Groenen Dynamic MDS visualization through iMDS 2005- Groenen, Trosset, Kagie Large scale MDS through Stress 2002 Denœux, Masson, Groenen, Winsberg, Diday Symbolic MDS of interval dissimilarities 2006 Groenen, Winsberg Symbolic MDS of histograms 2009 De Leeuw, Mair Smacof package in R