FlowCAP-I: Results Ryan Brinkman Senior Scientist, Terry Fox - - PowerPoint PPT Presentation

FlowCAP-I: Results

Ryan Brinkman

Senior Scientist, Terry Fox Laboratory, BC Cancer Agency Associate Professor, Medical Genetics, UBC

Sept 22, 2010

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

Outline

Sections What it means to be better (F-measure, ranking) Challenge 1 results Challenge 2 results Challenge 3 results Challenge 4 results So, which method should you use?

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

What it means to be better - Part I

Some comparisons are easy to quantify and understand intuitively

Does Raphael have more hair/cm2 of skull than Richard?

Some aren’t

Is Richard better looking than Raphael?

In which case you can use a gold standard

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

Problems with gold standards

1 It is possible they are flawed

You are unaware of intrinsic problems of your standard You start over-optimizing for some qualities of the standard

Rogain vs. Steroids - remember this now

2 Can never be better (looking) than the standard Ryan Brinkman – British Columbia Cancer Agency FlowCAP

How to evaluate gating vs. the gold standard?

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

How to evaluate gating vs. the gold standard?

Several categories of clustering comparison metrics Pair counting

Measures likelihood of grouping pair of data points together

Set-matching

Measures overlap between gold standard “classes” and hypothesized “clusters”

Entropy-based

Measures how well clusters only contain data points from a single class (i.e., homogeneity & completeness)

Several examples within each category MCR, V-measure, VI, Rand Index, F-measure F-measure has the minimum overall error for flow data

• C. J. van Rijsbergen. Information Retrieval. Butterworths, London,

1979

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

F-measure

Everything you need to know about the F-measure* Mis-classification rate is generally used for evaluating classifiers For FlowCAP, we performed cluster matching to label the clusters & calculate the misclassifications But its very time consuming to find the best cluster matching F-measure uses heuristic cluster matching algorithm

Does not guarantee best answer but is significantly faster

Mis-classification rate is then normalized by the size of the cluster. *Andrew Rosenberg and Julia Hirschberg. V-Measure: A conditional entropy-based external cluster evaluation measure.

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

What it means to be better - Part II

Some differences are easy to test for significance Null hypothesis: Raphael has significantly different hair thickness than Richard

Count # hairs in 30 random 1 cm2 patches on Raphael’s head Count # hairs in 30 matched 1 cm2 locations on Richard’s Do a paired t-test & check significance table

Some aren’t

H0: flowMeans’ results are significantly different than SamSpectral n = 5 (datasets) is too small Gold standard is manual gating Is an F-measure of .72 significantly different than .73?

What does such a difference even mean?

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

Scoring: fractional ranking and Borda count

Reducing complex data by evaluating it using certain criteria Evaluate match to human gating per sample using F-measure Rank F-measures high to low Score “best” algorithm N = # algorithms points Rank second highest algorithm N-1 points Group algorithms with overlapping F-measure 95% CI Give grouped algorithms average score of the group Sum scores across datasets

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

Challenge 1: Automated algorithms

Unsupervised clustering The “We really don’t know what we are looking for challenge” Given FCS files, markers (sometimes), general biology No tweaking of algorithms across datasets Compare to manual gates

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

F-Measure Distributions: Challenge 1: GvHD

Figure 1: Distributions of F-Measures for the GvHD dataset, challenge 1.

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

Example

Boxplots of F-measure values of different algorithms for Challenge 4: GvHD. There is a general agreement between the algorithms and the manual analysis. Sample 2: A sharp change in the F-measure values: The algorithms don’t agree with the human expert. Ryan Brinkman – British Columbia Cancer Agency FlowCAP

F-Measure CIs: Challenge 1: GvHD

Figure 2: Confidence Intervals of F-Measures for the GvHD dataset, challenge 1.

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

Challenge 1: GvHD

GvHD Rank Score FlowVB 0.85 (0.78, 0.90) 8.0 FLOCK 0.84 (0.77, 0.90) 8.0 flowMeans 0.88 (0.82, 0.93) 8.0 FLAME 0.85 (0.76, 0.92) 8.0 MM&PCA 0.84 (0.74, 0.93) 8.0 MM 0.83 (0.74, 0.91) 8.0 SamSPECTRAL 0.87 (0.82, 0.93) 8.0 CDP 0.52 (0.46, 0.57) 2.5 FEK 0.64 (0.57, 0.71) 2.5 flowClust/Merge 0.69 (0.56, 0.79) 2.5 SWIFT 0.63 (0.56, 0.69) 2.5

Table 1: Mean and 95 percent CIs for the F-Measures and rank scores for challenge 1 dataset GvHD

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

Challenge 1: DLBCL

DLBCL Rank Score FLOCK 0.88 (0.85, 0.91) 8.80 flowMeans 0.92 (0.90, 0.95) 8.80 FLAME 0.91 (0.88, 0.93) 8.80 MM 0.90 (0.86, 0.92) 8.80 SamSPECTRAL 0.86 (0.83, 0.90) 8.80 FlowVB 0.87 (0.85, 0.90) 4.75 CDP 0.85 (0.81, 0.88) 4.75 flowClust/Merge 0.84 (0.81, 0.86) 4.75 MM&PCA 0.85 (0.82, 0.88) 4.75 FEK 0.79 (0.74, 0.83) 2.00 SWIFT 0.67 (0.63, 0.71) 1.00

Table 2: Mean and 95 percent CIs for the F-Measures and rank scores for challenge 1 dataset DLBCL

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

Challenge 1: HSCT

HSCT Rank Score flowMeans 0.92 (0.90, 0.94) 10 FLAME 0.94 (0.92, 0.95) 10 MM&PCA 0.91 (0.88, 0.94) 10 FLOCK 0.86 (0.83, 0.89) 7 flowClust/Merge 0.81 (0.77, 0.85) 7 SamSPECTRAL 0.85 (0.82, 0.88) 7 FlowVB 0.75 (0.70, 0.79) 4 FEK 0.70 (0.65, 0.74) 4 MM 0.73 (0.66, 0.80) 4 SWIFT 0.59 (0.55, 0.63) 2 CDP 0.50 (0.48, 0.52) 1

Table 3: Mean and 95 percent CIs for the F-Measures and rank scores for challenge 1 dataset HSCT

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

Challenge 1: WNV

WNV Rank Score FLOCK 0.83 (0.80, 0.86) 10.5 flowMeans 0.88 (0.86, 0.90) 10.5 FlowVB 0.81 (0.78, 0.83) 7.0 FEK 0.78 (0.75, 0.81) 7.0 flowClust/Merge 0.77 (0.74, 0.79) 7.0 FLAME 0.80 (0.76, 0.84) 7.0 SamSPECTRAL 0.75 (0.61, 0.85) 7.0 CDP 0.71 (0.67, 0.74) 2.5 MM&PCA 0.64 (0.52, 0.72) 2.5 MM 0.69 (0.60, 0.75) 2.5 SWIFT 0.69 (0.64, 0.74) 2.5

Table 4: Mean and 95 percent CIs for the F-Measures and rank scores for challenge 1 dataset WNV

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

Challenge 1: ND

ND Rank Score SamSPECTRAL 0.92 (0.92, 0.93) 11.00 FLOCK 0.91 (0.89, 0.92) 8.33 flowMeans 0.85 (0.76, 0.92) 8.33 FLAME 0.90 (0.89, 0.91) 8.33 CDP 0.86 (0.81, 0.89) 7.50 SWIFT 0.87 (0.86, 0.88) 7.50 FEK 0.81 (0.80, 0.82) 4.00 FlowVB 0.85 (0.84, 0.86) 3.00 flowClust/Merge 0.73 (0.58, 0.85) 3.00 MM&PCA 0.76 (0.75, 0.77) 2.50 MM 0.75 (0.74, 0.76) 2.50

Table 5: Mean and 95 percent CIs for the F-Measures and rank scores for challenge 1 dataset ND

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

Challenge 1: Overall (lots of choice for automate analysis)

Rank Score Total Runtime flowMeans 45.6 00:04:23:27 FLOCK 42.6 00:00:37:38 FLAME 42.1 00:05:31:12 SamSPECTRAL 41.8 00:07:21:44 MM&PCA 27.8 00:00:04:35 FlowVB 26.8 03:02:23:09 MM 25.8 00:00:13:00 (sorry) flowClust/Merge 24.2 10:13:00:00 FEK 19.5 00:15:25:00 CDP 18.2 00:01:48:06 SWIFT 15.5 05:23:24:30

Table 6: Total runtimes (dd:hh:mm:ss) and rank scores for challenge 1

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

Challenge 1: Overall (lots of choice for automate analysis)

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

Challenge 2: Tuned Algorithms (in the Absence of Example Human-Provided Gates)

Add in the number of clusters Same as challenge 1, and ... You can tweak algorithm parameters to get a better “fit” to the data

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

Challenge 2: GvHD

GvHD Rank Score NMF-curvHDR 0.76 (0.69, 0.82) 5.0 FLOCK 0.84 (0.76, 0.90) 5.0 FLAME 0.81 (0.75, 0.87) 5.0 SamSPECTRAL 0.87 (0.79, 0.93) 5.0 SamSPECTRAL-Fixed-K 0.87 (0.80, 0.93) 5.0 CDP 0.59 (0.52, 0.64) 1.5 flowClust/Merge 0.69 (0.54, 0.79) 1.5

Table 7: Mean and 95 percent CIs for the F-Measures and rank scores for challenge 2 dataset GvHD

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

Challenge 2: DLBCL

DLBCL Rank Score FLOCK 0.88 (0.85, 0.91) 5.5 flowClust/Merge 0.87 (0.85, 0.90) 5.5 FLAME 0.87 (0.84, 0.90) 5.5 SamSPECTRAL 0.92 (0.89, 0.94) 5.5 NMF-curvHDR 0.84 (0.82, 0.86) 2.5 SamSPECTRAL-Fixed-K 0.85 (0.81, 0.89) 2.5 CDP 0.75 (0.69, 0.81) 1.0

Table 8: Mean and 95 percent CIs for the F-Measures and rank scores for challenge 2 dataset DLBCL

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

Challenge 2: HSCT

HSCT Rank Score flowClust/Merge 0.96 (0.94, 0.97) 7 CDP 0.84 (0.78, 0.89) 4 FLOCK 0.86 (0.83, 0.89) 4 FLAME 0.87 (0.83, 0.90) 4 SamSPECTRAL 0.90 (0.87, 0.93) 4 SamSPECTRAL-Fixed-K 0.90 (0.87, 0.92) 4 NMF-curvHDR 0.71 (0.67, 0.74) 1

Table 9: Mean and 95 percent CIs for the F-Measures and rank scores for challenge 2 dataset HSCT

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

Challenge 2: WNV

WNV Rank Score NMF-curvHDR 0.81 (0.77, 0.84) 5.5 FLOCK 0.84 (0.82, 0.86) 5.5 FLAME 0.84 (0.83, 0.85) 5.5 SamSPECTRAL 0.85 (0.83, 0.88) 5.5 CDP 0.75 (0.71, 0.78) 2.0 flowClust/Merge 0.77 (0.75, 0.79) 2.0 SamSPECTRAL-Fixed-K 0.76 (0.71, 0.81) 2.0

Table 10: Mean and 95 percent CIs for the F-Measures and rank scores for challenge 2 dataset WNV

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

Challenge 2: ND

ND Rank Score flowClust/Merge 0.88 (0.81, 0.91) 5.67 SamSPECTRAL 0.91 (0.91, 0.92) 5.67 SamSPECTRAL-Fixed-K 0.92 (0.91, 0.93) 5.67 CDP 0.86 (0.84, 0.88) 3.50 FLOCK 0.89 (0.87, 0.91) 3.50 FLAME 0.87 (0.86, 0.87) 3.00 NMF-curvHDR 0.83 (0.82, 0.84) 1.00

Table 11: Mean and 95 percent CIs for the F-Measures and rank scores for challenge 2 dataset ND

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

Challenge 2: Overall (does tweaking always help?

Rank Score Total Runtime SamSPECTRAL 25.7 00:13:00:00 FLOCK 23.5 00:00:28:31 FLAME 23.0

• flowClust/Merge

21.7 10:13:00:00 SamSPECTRAL-Fixed-K 19.2 00:08:26:10 NMF-curvHDR 15.0 07:23:04:00 CDP 12.0 00:00:33:30

Table 12: Total runtimes (dd:hh:mm:ss) and rank scores for challenge 2

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

Challenge 2: Overall (does tweaking always help?)

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

Challenge 3: Assignment of Cells to Populations with Pre- defined Number of Populations

Pre-defined number of clusters Same as challenge 1, and ... We’ll give you “k” No tweaking

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

Challenge 3: GvHD

GvHD Rank Score FLOCK 0.86 (0.79, 0.93) 7.0 flowMeans 0.91 (0.85, 0.96) 7.0 flowClust/Merge 0.88 (0.83, 0.93) 7.0 FLAME 0.85 (0.79, 0.91) 7.0 SamSPECTRAL 0.85 (0.76, 0.93) 7.0 SWIFT 0.90 (0.84, 0.95) 7.0 TCLUST 0.93 (0.90, 0.95) 7.0 flowKoh 0.85 (0.80, 0.90) 3.0 CDP 0.72 (0.63, 0.80) 1.5 curvHDR-NMF 0.74 (0.69, 0.78) 1.5

Table 13: Mean and 95 percent CIs for the F-Measures and rank scores for challenge 3 dataset GvHD

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

Challenge 3: DLBCL

DLBCL Rank Score FLOCK 0.92 (0.89, 0.94) 7.5 flowMeans 0.94 (0.92, 0.96) 7.5 flowClust/Merge 0.90 (0.86, 0.94) 7.5 FLAME 0.90 (0.86, 0.93) 7.5 SamSPECTRAL 0.93 (0.91, 0.95) 7.5 TCLUST 0.93 (0.91, 0.95) 7.5 CDP 0.78 (0.72, 0.83) 3.0 flowKoh 0.85 (0.82, 0.88) 3.0 curvHDR-NMF 0.84 (0.80, 0.89) 3.0 SWIFT 0.00 (0.00, 0.00) 1.0

Table 14: Mean and 95 percent CIs for the F-Measures and rank scores for challenge 3 dataset DLBCL

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

Challenge 3: HSCT

HSCT Rank Score FLOCK 0.97 (0.95, 0.98) 9.0 flowMeans 0.95 (0.93, 0.96) 9.0 SamSPECTRAL 0.97 (0.96, 0.98) 9.0 flowKoh 0.87 (0.84, 0.91) 5.5 FLAME 0.86 (0.81, 0.90) 5.5 SWIFT 0.88 (0.84, 0.92) 5.5 TCLUST 0.93 (0.90, 0.95) 5.5 flowClust/Merge 0.83 (0.79, 0.88) 2.5 curvHDR-NMF 0.80 (0.76, 0.84) 2.5 CDP 0.72 (0.68, 0.77) 1.0

Table 15: Mean and 95 percent CIs for the F-Measures and rank scores for challenge 3 dataset HSCT

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

Challenge 3: Overall (everything goes better with k)

Rank Score Total Runtime FLOCK 23.5 00:00:01:40 flowMeans 23.5 00:00:01:03 SamSPECTRAL 23.5 00:03:00:00 FLAME 20.0 00:03:59:02 TCLUST 20.0 00:00:47:00 flowClust/Merge 17.0 02:11:16:48 SWIFT 13.5 00:01:55:36 flowKoh 11.5 00:00:49:50 curvHDR-NMF 7.0

• CDP

5.5 00:00:25:00

Table 16: Total runtimes (dd:hh:mm:ss) and rank scores for challenge 3

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

Challenge 3: Overall (everything goes better with k)

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

Challenge 4: Supervised Clustering Approaches Trained using Human-Provided Gates

Supervised clustering Same as challenge 1, and ... We’ll give you some manual gates to train your algorithm

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

Challenge 4: GvHD

GvHD Rank Score randomForests 0.85 (0.79, 0.91) 4.0 flowClust/Merge 0.92 (0.88, 0.94) 4.0 Radial SVM 0.89 (0.84, 0.94) 4.0 CDP 0.66 (0.56, 0.77) 1.5 FLOCK 0.82 (0.77, 0.87) 1.5

Table 17: Mean and 95 percent CIs for the F-Measures and rank scores for challenge 4 dataset GvHD

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

Challenge 4: DLBCL

DLBCL Rank Score FLOCK 0.91 (0.89, 0.94) 4.5 flowClust/Merge 0.92 (0.90, 0.95) 4.5 randomForests 0.78 (0.74, 0.83) 2.0 CDP 0.76 (0.72, 0.81) 2.0 Radial SVM 0.84 (0.80, 0.87) 2.0

Table 18: Mean and 95 percent CIs for the F-Measures and rank scores for challenge 4 dataset DLBCL

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

Challenge 4: HSCT

HSCT Rank Score flowClust/Merge 0.95 (0.92, 0.97) 4.5 Radial SVM 0.98 (0.96, 0.99) 4.5 randomForests 0.81 (0.79, 0.83) 2.5 FLOCK 0.86 (0.76, 0.93) 2.5 CDP 0.70 (0.67, 0.73) 1.0

Table 19: Mean and 95 percent CIs for the F-Measures and rank scores for challenge 4 dataset HSCT

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

Challenge 4: WNV

WNV Rank Score Radial SVM 0.96 (0.94, 0.97) 5.0 randomForests 0.87 (0.84, 0.90) 2.5 CDP 0.85 (0.82, 0.87) 2.5 FLOCK 0.86 (0.82, 0.89) 2.5 flowClust/Merge 0.84 (0.82, 0.86) 2.5

Table 20: Mean and 95 percent CIs for the F-Measures and rank scores for challenge 4 dataset WNV

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

Challenge 4: ND

ND Rank Score randomForests 0.94 (0.92, 0.96) 4.5 Radial SVM 0.93 (0.92, 0.94) 4.5 FLOCK 0.86 (0.76, 0.92) 2.5 flowClust/Merge 0.89 (0.88, 0.90) 2.5 CDP 0.73 (0.65, 0.81) 1.0

Table 21: Mean and 95 percent CIs for the F-Measures and rank scores for challenge 4 dataset ND

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

Challenge 4: Overall (how good is good enough?)

Rank Score Total Runtime Radial SVM 20.0 00:00:32:49 flowClust/Merge 18.0 26:12:00:00 randomForests 15.5 00:04:00:00 FLOCK 13.5 00:00:07:57 CDP 8.0 00:00:27:18

Table 22: Total runtimes and rank scores for challenge 4

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

Challenge 4: Overall (how good is good enough?)

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

So what method should you use?

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

So what method should you use?

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

(over)Optimization of algorithms

Not every approach is perfect for every dataset Programmers make assumptions about data Algorithms are optimized to match these assumptions When applied to data, modelled to fit to these assumptions Difficult to model many assumptions

Taking Rogain and steroids together isn’t as effective as taking each separately

If only we could merge assumptions ...

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

Ensemble clustering

Not every approach is perfect for every dataset, but thats OK Two heads are better than one Ensemble approach seems to work generally across biological datasets

Gene prediction, gene function , RNA structure prediction

Several computing environments are readily available

These are completely abstracted from flow data We used R package clue by Kurt Hornik

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

Ensemble Clustering: Challenge 1

GvHD DLBCL HSCT WNV ND Mean FlowVB 0.85 0.87 0.75 0.81 0.85 0.82 CDP 0.52 0.85 0.50 0.71 0.86 0.69 FEK 0.64 0.79 0.70 0.78 0.81 0.74 FLOCK 0.84 0.88 0.86 0.83 0.91 0.86 flowMeans 0.88 0.92 0.92 0.88 0.85 0.89 flowClust/Merge 0.69 0.84 0.81 0.77 0.73 0.77 FLAME 0.85 0.91 0.94 0.80 0.90 0.88 MM&PCA 0.84 0.85 0.91 0.64 0.76 0.80 MM 0.83 0.90 0.73 0.69 0.75 0.78 SamSPECTRAL 0.87 0.86 0.85 0.75 0.92 0.85 SWIFT 0.63 0.67 0.59 0.69 0.87 0.69 EC 0.91 0.94 0.96 0.86 0.93 0.92

Table 23: Mean F-Measure values for challenge 1.

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

Ensemble Clustering: Challenge 2

GvHD DLBCL HSCT WNV ND Mean NMF-curvHDR 0.76 0.84 0.71 0.81 0.83 0.79 CDP 0.59 0.75 0.84 0.75 0.86 0.76 FLOCK 0.84 0.88 0.86 0.84 0.89 0.86 flowClust/Merge 0.69 0.87 0.96 0.77 0.88 0.83 FLAME 0.81 0.87 0.87 0.84 0.87 0.85 SamSPECTRAL 0.87 0.92 0.90 0.85 0.91 0.89 SamSPECTRAL-Fixed-K 0.87 0.85 0.90 0.76 0.92 0.86 EC 0.87 0.93 0.97 0.86 0.91 0.91

Table 24: Mean F-Measure values for challenge 2.

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

Ensemble Clustering: Challenge 3

GvHD DLBCL HSCT Mean CDP 0.72 0.78 0.72 0.74 FLOCK 0.86 0.92 0.97 0.92 flowKoh 0.85 0.85 0.87 0.86 flowMeans 0.91 0.94 0.95 0.93 flowClust/Merge 0.88 0.90 0.83 0.87 FLAME 0.85 0.90 0.86 0.87 curvHDR-NMF 0.74 0.84 0.80 0.79 SamSPECTRAL 0.85 0.93 0.97 0.92 SWIFT 0.90 0.00 0.88 0.59 TCLUST 0.93 0.93 0.93 0.93 EC 0.95 0.96 0.98 0.96

Table 25: Mean F-Measure values for challenge 3.

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

Ensemble Clustering: Challenge 4

GvHD DLBCL HSCT WNV ND Mean randomForests 0.85 0.78 0.81 0.87 0.94 0.85 CDP 0.66 0.76 0.70 0.85 0.73 0.74 FLOCK 0.82 0.91 0.86 0.86 0.86 0.86 flowClust/Merge 0.92 0.92 0.95 0.84 0.89 0.90 Radial SVM 0.89 0.84 0.98 0.96 0.93 0.92 EC 0.92 0.92 0.93 0.93 0.93 0.93

Table 26: Mean F-Measure values for challenge 4.

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

Ensembl clustering: The reality

Some caveats Open source methods? Common cluster output (Classification results (CLR)) file format?

Distinct from Gating-ML, which provides analytical process

Runtime?

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

FlowCAP-I: Future work

To do list Aberration analysis

Iterate through the algorithms by iteratively removing lowest scoring approach Repeat until one algorithm is left. This tells us how much each algorithm is contributing to ensemble

Other algorithms? Examples where manual analysis “fails” Manuscript

Ryan Brinkman – British Columbia Cancer Agency FlowCAP

FlowCAP acknowledgements: Coordination, data, participants, analysis & funding

BCCA Nima Aghaeepour, Philip Mah, Alireza Khodabakhshi. Josef ˘ Spidlen UBC Holger Hoos UTSW Richard Scheuermann FHCRC Raphaël Gottardo Tree Star Jill Schoenfeld Participants Cliburn Chan, Greg Finak, Faysal El Khettabi, Jose M. Maisog Iftekhar Naim, Radina Nikolic, Yu Qian, John Quinn, Parisa Shooshtari, Istavan Sugar, Kui Wang, Habil Zare Data Providers Connie Eaves (BCCA), Andrew P. Weng (BCCA), Hugh Wand (Amgen), Jonathan Bramson (McMaster) Summit funding NIH/NIAID Ryan Brinkman – British Columbia Cancer Agency FlowCAP