Assessing Migration Risk for Scientific Formats Chris Frisz, Sam - - PowerPoint PPT Presentation

▶

Jul 26, 2023 341 likes •1.82k views

Assessing Migration Risk for Scientific Formats Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Presented 7 December 2011 Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing

SLIDE 1

Assessing Migration Risk for Scientific Formats

Chris Frisz, Sam Waggoner, and Geoffrey Brown

Indiana University Bloomington

Presented 7 December 2011

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 2

Overview

Introduction

Motivation Hypothesis Approach

Background

Data set used Formats studied Conversion issues encountered

Tools written Results and discussion Conclusions

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 3

Motivation

Many migration tools exist for converting from obsolete to standard data formats.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 4

Motivation

Many migration tools exist for converting from obsolete to standard data formats. Mismatches in source and target formats introduce risk for migration.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 5

Motivation

Many migration tools exist for converting from obsolete to standard data formats. Mismatches in source and target formats introduce risk for migration. Automatic tools often fail silently when converting inconsistent features.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 6

Motivation (cont.)

Creating migration tools is hard.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 7

Motivation (cont.)

Creating migration tools is hard. Development often requires large programs written over a long time.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 8

Motivation (cont.)

Creating migration tools is hard. Development often requires large programs written over a long time. Migration is easier using existing tools.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 9

Hypothesis

Where migration tools already exist, they work well on the majority of data files despite differences in formats.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 10

Hypothesis

Where migration tools already exist, they work well on the majority of data files despite differences in formats. The remainder of the files can be identified for rarely-used, risky features.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 11

Hypothesis

Where migration tools already exist, they work well on the majority of data files despite differences in formats. The remainder of the files can be identified for rarely-used, risky features. Data files are separated into many that are “safe” to migrate versus a few that are “risky.”

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 12

Hypothesis (in visual form)

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 13

Approach

Wrote simple and fast analysis tools to categorize files by migration risk through deep inspection.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 14

Approach

Wrote simple and fast analysis tools to categorize files by migration risk through deep inspection. Identified 4 scientific formats with migration risks from a data set of U.S. Government documents.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 15

Approach

Wrote simple and fast analysis tools to categorize files by migration risk through deep inspection. Identified 4 scientific formats with migration risks from a data set of U.S. Government documents.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 16

Approach

Wrote simple and fast analysis tools to categorize files by migration risk through deep inspection. Identified 4 scientific formats with migration risks from a data set of U.S. Government documents. Found that the vast majority of files show few to no migration risks. This comes with some caveats.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 17

Format Overview

Lotus 1-2-3

A formerly popular spreadsheet program migratable to Excel with some calculation differences.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 18

Format Overview

Lotus 1-2-3

A formerly popular spreadsheet program migratable to Excel with some calculation differences.

CDF and netCDF

Array-based data formats with common roots but evolved with some different data representation and encoding features.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 19

Format Overview

Lotus 1-2-3

A formerly popular spreadsheet program migratable to Excel with some calculation differences.

CDF and netCDF

Array-based data formats with common roots but evolved with some different data representation and encoding features.

HDF

Hierarchical format for relating data artifacts that underwent significant changes from version 4 to 5.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 20

Data Set

Set of 2747 CD-ROM images from the United States Government Printing Office.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 21

Data Set

Set of 2747 CD-ROM images from the United States Government Printing Office. Thirty-six (36) images contained 14,022 Lotus 1-2-3, version 1 files. Sixty-eight (68) images contained 61,247 CDF files. Four (4) images contained 3,162 netCDF files. Two (2) images contained 2,213 HDF files.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 22

Data Set (cont.)

Lotus 1-2-3 files published from many different U.S. agencies:

CDC Census Bureau

Dept. of Education

Office of Business and Management

CDF and HDF files primarily from NASA. NetCDF files came from University of Maine, Dept. of Climatology.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 23

Formats – Lotus 1-2-3

Primary spreadsheet application used in the 1980s and early 1990s, but was supplanted by Microsoft Excel.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 24

Formats – Lotus 1-2-3

Primary spreadsheet application used in the 1980s and early 1990s, but was supplanted by Microsoft Excel. Microsoft provided conversion from 1-2-3 to Excel through 2003.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 25

Formats – Lotus 1-2-3

Primary spreadsheet application used in the 1980s and early 1990s, but was supplanted by Microsoft Excel. Microsoft provided conversion from 1-2-3 to Excel through 2003. Differences between the formats were documented by Microsoft and retrieved from knowledge base articles.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 26

Formats – Lotus 1-2-3 – Conversion issues

Operations calculated differently

@MOD @VLOOKUP @HLOOKUP

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 27

Formats – Lotus 1-2-3 – Conversion issues (cont.)

Exponentiation ( ˆ ) and unary negation ( - ) differ in order of

perations.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 28

Formats – Lotus 1-2-3 – Conversion issues (cont.)

Exponentiation ( ˆ ) and unary negation ( - ) differ in order of

perations.

Exponentiation was evaluated first in Lotus 1-2-3.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 29

Formats – Lotus 1-2-3 – Conversion issues (cont.)

Exponentiation ( ˆ ) and unary negation ( - ) differ in order of

perations.

Exponentiation was evaluated first in Lotus 1-2-3. Negation was evaluated first in Excel.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 30

Formats – Lotus 1-2-3 – Example

In Lotus 1-2-3: −42

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 31

Formats – Lotus 1-2-3 – Example

In Lotus 1-2-3: −42

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 32

Formats – Lotus 1-2-3 – Example

In Lotus 1-2-3: −42 = −16

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 33

Formats – Lotus 1-2-3 – Example

In Lotus 1-2-3: −42 = −16 In Excel: −42

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 34

Formats – Lotus 1-2-3 – Example

In Lotus 1-2-3: −42 = −16 In Excel: −42

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 35

Formats – Lotus 1-2-3 – Example

In Lotus 1-2-3: −42 = −16 In Excel: −42 = 16

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 36

Formats – Lotus 1-2-3 – Example

In Lotus 1-2-3: −42 = −16 In Excel: −42 = 16 Traditional mathematical order of operations favors Lotus.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 37

Formats – Lotus 1-2-3 – Conversion issues (cont.)

Comparison/logical operators (i.e. = or #and#) and string concatenation (&) also differ in order of operations.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 38

Formats – Lotus 1-2-3 – Conversion issues (cont.)

Comparison/logical operators (i.e. = or #and#) and string concatenation (&) also differ in order of operations.

Comparison and logical operators were evaluated first in Lotus 1-2-3.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 39

Formats – Lotus 1-2-3 – Conversion issues (cont.)

Comparison/logical operators (i.e. = or #and#) and string concatenation (&) also differ in order of operations.

Comparison and logical operators were evaluated first in Lotus 1-2-3. Concatenation was evaluated first in Excel.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 40

Formats – Lotus 1-2-3 – Conversion Issues – Example

In Lotus 1-2-3: “Fo”&“o” = “Foo”

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 41

Formats – Lotus 1-2-3 – Conversion Issues – Example

In Lotus 1-2-3: “Fo”&“o” = “Foo”

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 42

Formats – Lotus 1-2-3 – Conversion Issues – Example

In Lotus 1-2-3: “Fo”&“o” = “Foo” → False

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 43

Formats – Lotus 1-2-3 – Conversion Issues – Example

In Lotus 1-2-3: “Fo”&“o” = “Foo” → False In Excel: “Fo”&“o” = “Foo”

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 44

Formats – Lotus 1-2-3 – Conversion Issues – Example

In Lotus 1-2-3: “Fo”&“o” = “Foo” → False In Excel: “Fo”&“o” = “Foo”

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 45

Formats – Lotus 1-2-3 – Conversion Issues – Example

In Lotus 1-2-3: “Fo”&“o” = “Foo” → False In Excel: “Fo”&“o” = “Foo” → True

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 46

Formats – CDF and netCDF

CDF and netCDF are both file formats utilized for multidimensional data.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 47

Formats – CDF and netCDF

CDF and netCDF are both file formats utilized for multidimensional data. Often used to represent image, climate, and elevation data.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 48

Formats – CDF/netCDF Layout

Record rVariable rVariable . . . rVariable Number 1 2 n 1 !!!!! !!!!! !!!!! !!!!! !!!!! !!!!! !!!!! !!!!! . . . !!!!! !!!!! !!!!! !!!!! !!!!! !!!!! !!!!! 2 !!!!! !!!!! !!!!! !!!!! !!!!! !!!!! !!!!! !!!!! . . . !!!!! !!!!! !!!!! !!!!! !!!!! !!!!! !!!!! 3 !!!!! !!!!! !!!!! !!!!! !!!!! !!!!! !!!!! !!!!! . . . !!!!! !!!!! !!!!! !!!!! !!!!! !!!!! !!!!!

Image courtesy of NASA/Goddard Space Flight Center Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 49

Formats – CDF/netCDF Layout

Image courtesy of NASA/Goddard Space Flight Center Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 50

Formats – CDF/netCDF Layout

Image courtesy of NASA/Goddard Space Flight Center Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 51

Formats – CDF/netCDF Layout

Image courtesy of NASA/Goddard Space Flight Center Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 52

Formats – CDF/netCDF – Background

CDF originally developed by NASA.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 53

Formats – CDF/netCDF – Background

CDF originally developed by NASA. NetCDF developed later by NCAR based on the CDF.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 54

Formats – CDF/netCDF – Background

CDF originally developed by NASA. NetCDF developed later by NCAR based on the CDF. Both formats still currently supported.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 55

Formats – CDF/netCDF – Background (cont.)

Separate development allowed for evolution of different features.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 56

Formats – CDF/netCDF – Background (cont.)

Separate development allowed for evolution of different features. Overall functionality remained similar.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 57

Formats – CDF/netCDF – Background (cont.)

Separate development allowed for evolution of different features. Overall functionality remained similar. Primary conversion path between CDF and netCDF was through NASA’s Data Translation Web Service (DTWS).

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 58

Formats – CDF – Conversion Issues

Features present in CDF, not in netCDF:

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 59

Formats – CDF – Conversion Issues

Features present in CDF, not in netCDF:

Multi-file format for organizing variables into different files.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 60

Formats – CDF – Conversion Issues

Features present in CDF, not in netCDF:

Multi-file format for organizing variables into different files. Native-mode encoding for faster data access on particular system architectures.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 61

Formats – CDF – Conversion Issues

Features present in CDF, not in netCDF:

Multi-file format for organizing variables into different files. Native-mode encoding for faster data access on particular system architectures. Epoch data type for high-resolution time data.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 62

Formats – CDF – Conversion Issues

Features present in CDF, not in netCDF:

Multi-file format for organizing variables into different files. Native-mode encoding for faster data access on particular system architectures. Epoch data type for high-resolution time data.

Multi-file and native-mode differences were identified in CDF documentation.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 63

Formats – CDF – Conversion Issues

Features present in CDF, not in netCDF:

Multi-file format for organizing variables into different files. Native-mode encoding for faster data access on particular system architectures. Epoch data type for high-resolution time data.

Multi-file and native-mode differences were identified in CDF documentation. Epoch data type mismatch was discovered through DTWS source code review.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 64

Formats – netCDF – Conversion Issues

Features present in netCDF, not in CDF:

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 65

Formats – netCDF – Conversion Issues

Features present in netCDF, not in CDF:

Descriptive named dimensions usable for data access

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 66

Formats – netCDF – Conversion Issues

Features present in netCDF, not in CDF:

Descriptive named dimensions usable for data access Support for up 32 dimensions per variable (versus CDF’s 10)

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 67

Formats – netCDF – Conversion Issues

Features present in netCDF, not in CDF:

Descriptive named dimensions usable for data access Support for up 32 dimensions per variable (versus CDF’s 10)

Named dimensions mismatch was documented in NASA’s CDF FAQ.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 68

Formats – netCDF – Conversion Issues

Features present in netCDF, not in CDF:

Descriptive named dimensions usable for data access Support for up 32 dimensions per variable (versus CDF’s 10)

Named dimensions mismatch was documented in NASA’s CDF FAQ. Maximum dimension mismatch was discovered through netCDF API code review.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 69

Formats – HDF

Hierarchical data format for relating and interacting with hetergenous data sets.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 70

Formats – HDF

Hierarchical data format for relating and interacting with hetergenous data sets. Organized similarly to Unix file system with Vgroups like directories and Vdata like files.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 71

Formats – HDF layout

345&4"+"&<+.01+0.%,

T UVW IX WRYZ VXR U1Z [ S=1VXW W=YU1S U=Y1XU S=ZY1I I=XV1Z \ 2?5?3?9 9?3?5?2 A?C?"?7 ]?G?;?B B?;?G?]

=",+%.&->":% 9**'+"+2'* ?"$%++% <12%*+2(21&4"+"&<%+ ^P+;<797B#4:7*42;$2))2@_ @/"+"

!"7:$%&'$A7;#$3*4<274:$*4# #L2B,;#$*A$#23"$%&'$92<2

^!25;#_ ^XF57<?$USF57<$249$(#4#)2;$ /2:<#)_

T UVW IX WRYZ VXR U1Z [ S=1VXW W=YU1S U=Y1XU S=ZY1I I=XV1Z \ 2539 9352 AC"7 ]G;B B;G] !"7:$%&'$A7;#$3*4<274:$*4# #L2B,;#$*A$#23"$%&'$92<2$<@,#=$

@:.'0A ^()*+,$*A$%&'$92<2$:<)+3<+)#:_

<@,#=$ $

Image courtesy of the HDF Group. Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 72

Formats – HDF – Background

Developed by the National Center for Supercomputing Applications.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 73

Formats – HDF – Background

Developed by the National Center for Supercomputing Applications. Support provided by the HDF Group.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 74

Formats – HDF – Background

Developed by the National Center for Supercomputing Applications. Support provided by the HDF Group. Most recent version was HDF5.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 75

Formats – HDF – Background (cont.)

Previous versions were backwards compatible.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 76

Formats – HDF – Background (cont.)

Previous versions were backwards compatible. HDF5 drastically changed data model and broke backwards compatibility.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 77

Formats – HDF – Background (cont.)

Previous versions were backwards compatible. HDF5 drastically changed data model and broke backwards compatibility. HDF Group provided both conversion API and automatic tool.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 78

Formats – HDF – Conversion Issues

Merging Vgroups with elements sharing the same name resulted in renaming of one element.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 79

Formats – HDF – Conversion Issues

Merging Vgroups with elements sharing the same name resulted in renaming of one element.

This was only relevant for manual conversion.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 80

Formats – HDF – Conversion Issues

Merging Vgroups with elements sharing the same name resulted in renaming of one element.

This was only relevant for manual conversion.

Data object shared between Vgroups were copied on conversion.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 81

Formats – HDF – Conversion Issues – Example

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 82

Formats – HDF – Conversion Issues – Example

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 83

Formats – HDF – Conversion Issues – Example

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 84

Formats – HDF – Conversion Issues – Example

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 85

Formats – HDF – Conversion Issues

Merging Vgroups with elements sharing the same name resulted in renaming of one element.

This was only relevant for manual conversion.

Data object shared between Vgroups were copied on conversion. Unnamed data objects were given default names

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 86

Formats – HDF – Conversion Issues

Merging Vgroups with elements sharing the same name resulted in renaming of one element.

This was only relevant for manual conversion.

Data object shared between Vgroups were copied on conversion. Unnamed data objects were given default names The HDF Group documented all of these issues for the HDF4-to-HDF5 conversion API and automated tool.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 87

Tools – Lotus 1-2-3

We wrote a C program to traverse 1-2-3 files and parse formulas.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 88

Tools – Lotus 1-2-3

We wrote a C program to traverse 1-2-3 files and parse formulas. It identified presence of @MOD, @VLOOKUP, or @HLOOKUP in formulas.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 89

Tools – Lotus 1-2-3

We wrote a C program to traverse 1-2-3 files and parse formulas. It identified presence of @MOD, @VLOOKUP, or @HLOOKUP in formulas. The program also conservatively reported presence of both exponentiation and negation or logical/comparison operators and string concatenation.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 90

Tools – Lotus 1-2-3 (cont.)

Tool consisted of approximately 500 lines.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 91

Tools – Lotus 1-2-3 (cont.)

Tool consisted of approximately 500 lines. Processed our entire data set in less than 15 mintues.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 92

Tools – CDF and netCDF

We wrote C programs for each CDF and netCDF.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 93

Tools – CDF and netCDF

We wrote C programs for each CDF and netCDF. CDF program consisted of 300 lines using the version 3.3.0 API from NASA.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 94

Tools – CDF and netCDF

We wrote C programs for each CDF and netCDF. CDF program consisted of 300 lines using the version 3.3.0 API from NASA. NetCDF program was 150 lines using the version 4.1.3 API from Unidata.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 95

Tools – CDF and netCDF

We wrote C programs for each CDF and netCDF. CDF program consisted of 300 lines using the version 3.3.0 API from NASA. NetCDF program was 150 lines using the version 4.1.3 API from Unidata. Processed entire 61,000-file data set in 55 minutes.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 96

Tools – CDF and netCDF

We wrote C programs for each CDF and netCDF. CDF program consisted of 300 lines using the version 3.3.0 API from NASA. NetCDF program was 150 lines using the version 4.1.3 API from Unidata. Processed entire 61,000-file data set in 55 minutes. NetCDF tool exhibited similar performance.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 97

Tools – HDF

Yet again, wrote a C program.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 98

Tools – HDF

Yet again, wrote a C program. Written in 900 lines using the 4.2.6 API from the HDF Group.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 99

Tools – HDF

Yet again, wrote a C program. Written in 900 lines using the 4.2.6 API from the HDF Group. This tool was longer because of large number of interfaces.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 100

Tools – HDF

Yet again, wrote a C program. Written in 900 lines using the 4.2.6 API from the HDF Group. This tool was longer because of large number of interfaces. Processed all HDF files in our data set within 1.5 minutes.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 101

Results – Lotus 1-2-3

We ran our analysis tool on 14,022 version 1 files.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 102

Results – Lotus 1-2-3

We ran our analysis tool on 14,022 version 1 files. It detected a single file containing 7 formulas with possible

rder of operations mismatches between 1-2-3 and Excel.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 103

Results – Lotus 1-2-3 (cont.)

Example formula from the file: @IF($EJ$85=“NA”, +“ ”&$EJ$85,+$EJ$85)

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 104

Results – Lotus 1-2-3 (cont.)

Example formula from the file: @IF($EJ$85=“NA”, +“ ”&$EJ$85,+$EJ$85) The other six also followed this form.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 105

Results – Lotus 1-2-3 (cont.)

Example formula from the file: @IF($EJ$85=“NA”, +“ ”&$EJ$85,+$EJ$85) The other six also followed this form. Logical comparison and string concatenation appeared in the same formula, but would not conflict if converted to Excel.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 106

Discussion – Lotus 1-2-3

The vast majority of files can be converted conventially without risk.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 107

Discussion – Lotus 1-2-3

The vast majority of files can be converted conventially without risk. Only a few files may require a more robust conversion process

r by-hand translation.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 108

Discussion – Lotus 1-2-3

The vast majority of files can be converted conventially without risk. Only a few files may require a more robust conversion process

r by-hand translation.

All 14,022 files in our data set could have been converted without risk after manually verifying a single file.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 109

Results – CDF

Our tool ran on 61,247 CDF version 2 files.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 110

Results – CDF

Our tool ran on 61,247 CDF version 2 files. 14,574 (23.8%) files with no potential conversion risk to netCDF.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 111

Results – CDF

Our tool ran on 61,247 CDF version 2 files. 14,574 (23.8%) files with no potential conversion risk to netCDF. 46,669 (76.2%) utilized the Epoch data type.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 112

Results – CDF

Our tool ran on 61,247 CDF version 2 files. 14,574 (23.8%) files with no potential conversion risk to netCDF. 46,669 (76.2%) utilized the Epoch data type. 4 files used multi-file format.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 113

Results – CDF

Our tool ran on 61,247 CDF version 2 files. 14,574 (23.8%) files with no potential conversion risk to netCDF. 46,669 (76.2%) utilized the Epoch data type. 4 files used multi-file format. There were no files which used native encoding.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 114

Discussion – CDF

Use of Epoch data type was prevalent (76.2%).

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 115

Discussion – CDF

Use of Epoch data type was prevalent (76.2%). CDF API included functions to convert Epochs to strings.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 116

Discussion – CDF

Use of Epoch data type was prevalent (76.2%). CDF API included functions to convert Epochs to strings.

DTWS tool used this method during conversion.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 117

Discussion – CDF

Use of Epoch data type was prevalent (76.2%). CDF API included functions to convert Epochs to strings.

DTWS tool used this method during conversion. Tools for converting date string formats are widely available (i.e. Unix).

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 118

Discussion – CDF

Use of Epoch data type was prevalent (76.2%). CDF API included functions to convert Epochs to strings.

DTWS tool used this method during conversion. Tools for converting date string formats are widely available (i.e. Unix).

Multi-file format was handled by DTWS tools, despite its rare appearance.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 119

Discussion – CDF (cont.)

The results indicated a minimal migration risk for converting CDF to netCDF, which supported our hypothesis.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 120

Results – netCDF

We ran our tool on 3,162 netCDF files.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 121

Results – netCDF

We ran our tool on 3,162 netCDF files. All files included named dimensions.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 122

Results – netCDF

We ran our tool on 3,162 netCDF files. All files included named dimensions.

We expected this result.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 123

Results – netCDF

We ran our tool on 3,162 netCDF files. All files included named dimensions.

We expected this result.

No files included variables with more than CDF’s maximum 10 dimensions.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 124

Results – netCDF

We ran our tool on 3,162 netCDF files. All files included named dimensions.

We expected this result.

No files included variables with more than CDF’s maximum 10 dimensions.

This indicated it was a rare feature.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 125

Discussion – netCDF

Dimensions names (present in all netCDF datasets) were not saved in conversion.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 126

Discussion – netCDF

Dimensions names (present in all netCDF datasets) were not saved in conversion. This represented actual metadata loss.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 127

Discussion – netCDF

Dimensions names (present in all netCDF datasets) were not saved in conversion. This represented actual metadata loss. Though raw data was preserved in conversion, this conflicted with our hypothesis.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 128

Discussion – netCDF (cont.)

One possible solution was to save names in a separate metadata file.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 129

Discussion – netCDF (cont.)

One possible solution was to save names in a separate metadata file. We were not aware of an existing tool to do this.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 130

Results – HDF

Tool ran on 352 HDF3 and 1,861 HDF4 (2,213 total) files.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 131

Results – HDF

Tool ran on 352 HDF3 and 1,861 HDF4 (2,213 total) files. 324 (14.6%) files with no conversion risks.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 132

Results – HDF

Tool ran on 352 HDF3 and 1,861 HDF4 (2,213 total) files. 324 (14.6%) files with no conversion risks. 1,891 (85.4%) with multiple Vgroups containing objects with the same name.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 133

Results – HDF

Tool ran on 352 HDF3 and 1,861 HDF4 (2,213 total) files. 324 (14.6%) files with no conversion risks. 1,891 (85.4%) with multiple Vgroups containing objects with the same name. 1,889 (85.4%) with data objects shared between Vgroups.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 134

Results – HDF

Tool ran on 352 HDF3 and 1,861 HDF4 (2,213 total) files. 324 (14.6%) files with no conversion risks. 1,891 (85.4%) with multiple Vgroups containing objects with the same name. 1,889 (85.4%) with data objects shared between Vgroups. No unnamed data objects.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 135

Discussion – HDF

Duplicate Vdata object names were irrelevant for automatic conversion.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 136

Discussion – HDF

Duplicate Vdata object names were irrelevant for automatic conversion. Shared object copying broke data relationships from the source files.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 137

Discussion – HDF (cont.)

Issues would not manifest when converting for purely archival reasons.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 138

Discussion – HDF (cont.)

Issues would not manifest when converting for purely archival reasons. This overall supported our hypothesis with a caveat.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 139

Conclusions

Existing conversion tools could safely convert the vast majority of files in general.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 140

Conclusions

Existing conversion tools could safely convert the vast majority of files in general. Caveats:

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 141

Conclusions

Existing conversion tools could safely convert the vast majority of files in general. Caveats:

NetCDF-to-CDF conversion loses metadata and requires a separate solution.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 142

Conclusions

Existing conversion tools could safely convert the vast majority of files in general. Caveats:

NetCDF-to-CDF conversion loses metadata and requires a separate solution. HDF4-to-HDF5 conversion breaks data relationships and is

nly completely safe for archival purposes.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 143

Conclusions (cont.)

The results for our data set overall supported our hypothesis.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 144

Conclusions (cont.)

The results for our data set overall supported our hypothesis. Our findings supported use of simple and fast tools for migration risk analysis

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 145

Conclusions (cont.)

The results for our data set overall supported our hypothesis. Our findings supported use of simple and fast tools for migration risk analysis Open formats (e.g. CDF, netCDF, HDF) are easier to analyze than proprietary ones (i.e. Lotus 1-2-3).

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 146

Acknowledgements

The authors would like to gratefully acknowledge the support of the Data to Insight Center, a partnership of the School of Informatics and Computing, Digital Libraries and Pervasive Technology Institute at Indiana University. This research funded in part by a grant provided by the Lilly Endowment Inc.

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats

SLIDE 147

Time for questions and comments

Chris Frisz, Sam Waggoner, and Geoffrey Brown Indiana University Bloomington Assessing Migration Risk for Scientific Formats