FLOW CYTOMETRY DATA COMPRESSION A.E. Bras PhD Student Erasmus - - PowerPoint PPT Presentation

▶
flow cytometry data compression
SMART_READER_LITE
LIVE PREVIEW

FLOW CYTOMETRY DATA COMPRESSION A.E. Bras PhD Student Erasmus - - PowerPoint PPT Presentation

Dec 07, 2022 254 likes •953 views

FLOW CYTOMETRY DATA COMPRESSION A.E. Bras PhD Student Erasmus University, Rotterdam, the Netherlands Flow Cytometry Flow Cytometry Flow Cytometry Flow Cytometry Flow Cytometry Flow Cytometry Older Systems Older Systems Newer Systems

slide-1
SLIDE 1

A.E. Bras

PhD Student Erasmus University, Rotterdam, the Netherlands

FLOW CYTOMETRY – DATA COMPRESSION

slide-2
SLIDE 2

Flow Cytometry

slide-3
SLIDE 3

Flow Cytometry

slide-4
SLIDE 4

Flow Cytometry

slide-5
SLIDE 5

Flow Cytometry

slide-6
SLIDE 6

Flow Cytometry

slide-7
SLIDE 7

Flow Cytometry

slide-8
SLIDE 8

Older Systems

slide-9
SLIDE 9

Older Systems

slide-10
SLIDE 10

Newer Systems

slide-11
SLIDE 11

Newer Systems

slide-12
SLIDE 12

Newer Systems

slide-13
SLIDE 13

FLOATING POINT DATA

  • Newer Systems
slide-14
SLIDE 14

Newer Systems

FLOATING POINT DATA

  • MILLIONS
slide-15
SLIDE 15

Storage

FLOATING POINT DATA

  • MILLIONS
slide-16
SLIDE 16

Storage .FCS

FLOATING POINT DATA

  • MILLIONS
slide-17
SLIDE 17

Storage .FCS FLOW CYTOMETRY STRANDARD

FLOATING POINT DATA

  • MILLIONS
slide-18
SLIDE 18

Storage .FCS FLOW CYTOMETRY STRANDARD PRO WIDELEY USED

FLOATING POINT DATA

  • MILLIONS
slide-19
SLIDE 19

Storage .FCS

FLOATING POINT DATA

  • MILLIONS

FLOW CYTOMETRY STRANDARD PRO WIDELEY USED CON NO COMPRESSION

slide-20
SLIDE 20

Lossless Compression .FCS

slide-21
SLIDE 21

Lossless Compression .FCS .ZIP

≈ 70 %

slide-22
SLIDE 22

Lossless Compression .FCS .ZIP

≈ 70 %

Better alternative?

slide-23
SLIDE 23

Lossless Compression - Benchmark

slide-24
SLIDE 24

Lossless Compression - Benchmark

167.131

.FCS

slide-25
SLIDE 25

RANDOM

Lossless Compression - Benchmark

167.131

.FCS

slide-26
SLIDE 26

RANDOM

Lossless Compression - Benchmark

167.131

.FCS

slide-27
SLIDE 27

RANDOM

Lossless Compression - Benchmark

167.131

CODEC RATIO LZIP 0.533 LZMA 0.533 XZ 0.533 GLZA 0.558 LZHAM 0.568 CSC 0.590 BROTLI 0.598 TORNADO 0.622 ZSTD 0.626 XPACK 0.637 ZLING 0.665 LIBDEFLATE 0.676 LZFSE 0.690 CRUSH 0.692 ZLIB 0.695 UCL_NRV2D 0.725 UCL_NRV2E 0.725 UCL_NRV2B 0.734 LZO1X 0.755 LZO1Z 0.757 LZSSE8 0.758 LZO1Y 0.768 LZSSE2 0.768 LIZARD 0.770 DENSITY 0.771 LZO2A 0.774 LZO1B 0.775 LZ4HC 0.777 ... ... CODEC RATIO ZPAQ 0.460 BCM 0.510 LZIP 0.533 LZMA 0.533 FLZMA2 0.539 LZHAM 0.569 BROTLI 0.571 BZIP2 0.580 CSC 0.590 BALZ 0.598 XPACK 0.612 ZSTD 0.625 ZOPFLI 0.674 LIBDEFLATE 0.676 LZFSE 0.690 CRUSH 0.692 ZLIB 0.695 BRIEFLZ 0.727 DOBOZ 0.750 LZSSE8 0.758 BSCQLFC 0.763 LZSSE2 0.768 LZ4 0.777 LZSSE4 0.781 LZG 0.803 SUBOTIN 0.819 FASTAC 0.819 ZLIBH 0.823 ... ...

.FCS

slide-28
SLIDE 28

RANDOM

Lossless Compression - Benchmark

167.131

CODEC RATIO LZIP 0.533 LZMA 0.533 XZ 0.533 GLZA 0.558 LZHAM 0.568 CSC 0.590 BROTLI 0.598 TORNADO 0.622 ZSTD 0.626 XPACK 0.637 ZLING 0.665 LIBDEFLATE 0.676 LZFSE 0.690 CRUSH 0.692 ZLIB 0.695 UCL_NRV2D 0.725 UCL_NRV2E 0.725 UCL_NRV2B 0.734 LZO1X 0.755 LZO1Z 0.757 LZSSE8 0.758 LZO1Y 0.768 LZSSE2 0.768 LIZARD 0.770 DENSITY 0.771 LZO2A 0.774 LZO1B 0.775 LZ4HC 0.777 ... ... CODEC RATIO ZPAQ 0.460 BCM 0.510 LZIP 0.533 LZMA 0.533 FLZMA2 0.539 LZHAM 0.569 BROTLI 0.571 BZIP2 0.580 CSC 0.590 BALZ 0.598 XPACK 0.612 ZSTD 0.625 ZOPFLI 0.674 LIBDEFLATE 0.676 LZFSE 0.690 CRUSH 0.692 ZLIB 0.695 BRIEFLZ 0.727 DOBOZ 0.750 LZSSE8 0.758 BSCQLFC 0.763 LZSSE2 0.768 LZ4 0.777 LZSSE4 0.781 LZG 0.803 SUBOTIN 0.819 FASTAC 0.819 ZLIBH 0.823 ... ...

.FCS

slide-29
SLIDE 29

Lossless Compression - Benchmark

167.131

CODEC RATIO LZIP 0.533 LZMA 0.533 XZ 0.533 GLZA 0.558 LZHAM 0.568 CSC 0.590 BROTLI 0.598 TORNADO 0.622 ZSTD 0.626 XPACK 0.637 ZLING 0.665 LIBDEFLATE 0.676 LZFSE 0.690 CRUSH 0.692 ZLIB 0.695 UCL_NRV2D 0.725 UCL_NRV2E 0.725 UCL_NRV2B 0.734 LZO1X 0.755 LZO1Z 0.757 LZSSE8 0.758 LZO1Y 0.768 LZSSE2 0.768 LIZARD 0.770 DENSITY 0.771 LZO2A 0.774 LZO1B 0.775 LZ4HC 0.777 ... ...

.FCS

CODEC RATIO ZPAQ 0.460 BCM 0.510 LZIP 0.533 LZMA 0.533 FLZMA2 0.539 LZHAM 0.569 BROTLI 0.571 BZIP2 0.580 CSC 0.590 BALZ 0.598 XPACK 0.612 ZSTD 0.625 ZOPFLI 0.674 LIBDEFLATE 0.676 LZFSE 0.690 CRUSH 0.692 ZLIB 0.695 BRIEFLZ 0.727 DOBOZ 0.750 LZSSE8 0.758 BSCQLFC 0.763 LZSSE2 0.768 LZ4 0.777 LZSSE4 0.781 LZG 0.803 SUBOTIN 0.819 FASTAC 0.819 ZLIBH 0.823 ... ...

slide-30
SLIDE 30

Lossless Compression - Benchmark

slide-31
SLIDE 31

Lossless Compression - Benchmark

≈ 0.45

slide-32
SLIDE 32

Implementation in R

slide-33
SLIDE 33

Implementation in R base::memCompress(type="XZ")

slide-34
SLIDE 34

Implementation in R base::memCompress(type="XZ")

slide-35
SLIDE 35

Bioconductor - flowCore

slide-36
SLIDE 36

Bioconductor - flowCore

slide-37
SLIDE 37

C:\input.fcs

flowCore::read.FCS flowSom::FlowSOM flowCore::write.FCS

C:\output.fcs

Bioconductor - flowCore

slide-38
SLIDE 38

C:\input.fcs

flowCore::read.FCS flowSom::FlowSOM flowCore::write.FCS

C:\output.fcs

Bioconductor - flowCore base::readBin

slide-39
SLIDE 39

C:\input.fcs

flowCore::read.FCS flowSom::FlowSOM flowCore::write.FCS

C:\output.fcs

Bioconductor - flowCore base::readBin base::writeBin

slide-40
SLIDE 40

C:\input.fcs

flowCore::read.FCS flowSom::FlowSOM flowCore::write.FCS

C:\output.fcs

Bioconductor - flowCore base::readBin base::memCompress base::writeBin

slide-41
SLIDE 41

C:\input.fcs

flowCore::read.FCS flowSom::FlowSOM flowCore::write.FCS

C:\output.fcs

Bioconductor - flowCore base::readBin base::memDecompress base::memCompress base::writeBin

slide-42
SLIDE 42

C:\input.fcs

flowCore::read.FCS flowSom::FlowSOM flowCore::write.FCS

C:\output.fcs

Bioconductor - flowCore .TAR.GZ

slide-43
SLIDE 43

C:\input.fcs

flowCore::read.FCS flowSom::FlowSOM flowCore::write.FCS

C:\output.fcs

Bioconductor - flowCore .TAR.GZ

slide-44
SLIDE 44

C:\input.fcs

flowCore::read.FCS flowSom::FlowSOM flowCore::write.FCS

C:\output.fcs

Bioconductor - flowCore .TAR.GZ

slide-45
SLIDE 45

C:\input.fcs

flowCore::read.FCS flowSom::FlowSOM flowCore::write.FCS

C:\output.fcs

Bioconductor - flowCore .FCS.XZ

slide-46
SLIDE 46

C:\input.fcs

flowCore::read.FCS flowSom::FlowSOM flowCore::write.FCS

C:\output.fcs

Bioconductor - flowCore .FCS.XZ

slide-47
SLIDE 47

C:\input.fcs

flowCore::read.FCS flowSom::FlowSOM flowCore::write.FCS

C:\output.fcs

Bioconductor - flowCore .FCS.XZ

slide-48
SLIDE 48

C:\input.fcs

flowCore::read.FCS flowSom::FlowSOM flowCore::write.FCS

C:\output.fcs

Bioconductor - flowCore .FCS.XZ

slide-49
SLIDE 49

Bioconductor – flowCore - Pipelines

slide-50
SLIDE 50

Bioconductor – flowCore - Pipeline

C:\input\01.fcs C:\input\02.fcs C:\output\01.fcs C:\output\02.fcs

flowCore::read.flowSet flowCore::write.flowSet

slide-51
SLIDE 51

Bioconductor – flowCore - Pipeline

C:\input\01.fcs C:\input\02.fcs C:\output\01.fcs C:\output\02.fcs

flowCore::read.flowSet flowCore::write.flowSet

C:\input.zip

base::unzip

slide-52
SLIDE 52

Bioconductor – flowCore - Pipeline

C:\input\01.fcs C:\input\02.fcs C:\output\01.fcs C:\output\02.fcs

flowCore::read.flowSet flowCore::write.flowSet

C:\input.zip C:\output.zip

base::unzip base::zip

slide-53
SLIDE 53

Bioconductor – flowCore - Pipeline

C:\input\01.fcs C:\input\02.fcs C:\output\01.fcs C:\output\02.fcs

flowCore::read.flowSet flowCore::write.flowSet

C:\input.zip C:\output.zip

base::unzip base::zip

slide-54
SLIDE 54

Bioconductor – flowCore - Pipeline

C:\input\01.fcs C:\input\02.fcs C:\output\01.fcs C:\output\02.fcs

flowCore::read.flowSet flowCore::write.flowSet

C:\input.zip C:\output.zip

base::unzip base::zip

.ACS

ARCHIVAL CYTOMETRY STANDARD

slide-55
SLIDE 55

Bioconductor – flowCore - Pipeline

.ZIP .ACS

ARCHIVAL CYTOMETRY STANDARD

=

slide-56
SLIDE 56

Bioconductor – flowCore - Pipeline

.ZIP

.FCS .FCS .FCS

.ACS

ARCHIVAL CYTOMETRY STANDARD

=

slide-57
SLIDE 57

Bioconductor – flowCore - Pipeline

.ZIP

.FCS .FCS .FCS .TXT

.ACS

ARCHIVAL CYTOMETRY STANDARD

=

slide-58
SLIDE 58

Bioconductor – flowCore - Pipeline

C:\input\01.fcs C:\input\01.fcs C:\output\01.fcs C:\output\01.fcs

flowCore::read.flowSet flowCore::write.flowSet

C:\input.zip C:\output.zip

base::unzip base::zip

slide-59
SLIDE 59

Bioconductor – flowCore - Pipeline

C:\input.acs C:\output.acs

flowCore::read.flowSet flowCore::write.flowSet

slide-60
SLIDE 60

Bioconductor – flowCore - Pipeline

C:\input.acs C:\output.acs

flowCore::read.flowSet flowCore::write.flowSet base::unzip

slide-61
SLIDE 61

Bioconductor – flowCore - Pipeline

C:\input.acs C:\output.acs

flowCore::read.flowSet flowCore::write.flowSet base::unzip base::zip

slide-62
SLIDE 62

C:\input.acs C:\output.acs

flowCore::read.flowSet flowCore::write.flowSet

C:\input.fcs.xz C:\output.fcs.xz

flowCore::read.FCS flowCore::write.FCS

slide-63
SLIDE 63

C:\input.acs C:\output.acs

flowCore::read.flowSet flowCore::write.flowSet

C:\input.fcs.xz C:\output.fcs.xz

flowCore::read.FCS flowCore::write.FCS PRO PERFORMANCE

slide-64
SLIDE 64

C:\input.acs C:\output.acs

flowCore::read.flowSet flowCore::write.flowSet

C:\input.fcs.xz C:\output.fcs.xz

flowCore::read.FCS flowCore::write.FCS PRO PERFORMANCE CON COMPATIBILITY

slide-65
SLIDE 65

C:\input.acs C:\output.acs

flowCore::read.flowSet flowCore::write.flowSet

C:\input.fcs.xz C:\output.fcs.xz

flowCore::read.FCS flowCore::write.FCS PRO PERFORMANCE CON COMPATIBILITY PRO COMPATIBILITY

slide-66
SLIDE 66

C:\input.acs C:\output.acs

flowCore::read.flowSet flowCore::write.flowSet

C:\input.fcs.xz C:\output.fcs.xz

flowCore::read.FCS flowCore::write.FCS PRO PERFORMANCE CON COMPATIBILITY PRO COMPATIBILITY CON PERFORMANCE

slide-67
SLIDE 67

C:\input.acs C:\output.acs

flowCore::read.flowSet flowCore::write.flowSet

C:\input.fcs.xz C:\output.fcs.xz

flowCore::read.FCS flowCore::write.FCS PRO PERFORMANCE EASY CON COMPATIBILITY PRO COMPATIBILITY EASY CON PERFORMANCE

slide-68
SLIDE 68

C:\input.acs C:\output.acs

flowCore::read.flowSet flowCore::write.flowSet

C:\input.fcs.xz C:\output.fcs.xz

flowCore::read.FCS flowCore::write.FCS PRO PERFORMANCE EASY BENIFIT CON COMPATIBILITY PRO COMPATIBILITY EASY BENIFIT CON PERFORMANCE

slide-69
SLIDE 69

Questions?

Lossless Compression of Cytometric Data

Anne E. Bras & Vincent H. J. van der Velden