Data Preservation lesperienza di ESA e il caso studio in EVER-EST - - PowerPoint PPT Presentation

ESA UNCLASSIFIED – For Official Use

Data Preservation l’esperienza di ESA e il caso studio in EVER-EST ISMAR 05 July 2016

Data lifecycle: sensed data needs to be acquired…

001 4100 002 4102 003 4102 004 6144 005 6150 006 7168 007 6146

Level 0

...to be combined and processed to get this

3

Level 2 Level 0 Level 1 Processing Processing/c

• mbining

...or even further to get this

Scientific Publications

...through complex processing schemes

Algorithm Manual

Earth Medical History

2

Since when are you feeling like this ?

3

I don’t remember but luckily someone has preserved and updated my health records; here they are

4

Great!! We have new instruments today but knowing your past is crucial. Let me consult with my colleagues …

5

We identified the cause and have the right treatment for you !!!

• 1. EO Heritage Data :

2.

• the health record of our planet

Heritage Data Programme (LTDP+)

• preserve and valorize the past to better shape the future

Heritage Data Programme (LTDP+)

• preserve and valorize the past to better shape the future

1

I don’t feel very well

6

Thank you !!! I feel much better now !!!

Heritage Data Long Term Preservation Programme Objectives

• 1. ESA EO heritage missions data & knowledge holdings preservation

Secure state-of-art preservation of ESA EO heritage data assets and associated information holdings relying on state of art technologies, systems and standards, and ensuring data authenticity and understandability in the long term (Authority Entity) with the assumption/awareness that one can never anticipate the exploitation value

• f the data asset in the future
• 2. Coordination and cooperation with MS on

data preservation

establish an harmonised approach in cooperation with data owners in the MS to ensure European EO data preservation coordination with non-European actors

• 3. European interest heritage data assets

preservation

prevent loss of non-ESA EO data at risk and judged of European long-term interest

ESA UNCLASSIFIED – For Official Use

Heritage Data: Valorise the past to better understand the future

ESA UNCLASSIFIED – For Official Use

Spot 5 map

ESA Long Time Data Series (excerpt)

Covered by LTDP Covered by LTDP+

ESA UNCLASSIFIED – For Official Use

Data Management Plan

Discoverability

DMP-1: METADATA FOR DISCOVERY

Accessibility

DMP-2: ONLINE ACCESS

Usability

DMP-3: DATA ENCODING DMP-4: DATA DOCUMENTATION DMP-5: DATA TRACEABILITY DMP-6: DATA QUALITY-CONTROL

Preservation

DMP-7: DATA PRESERVATION DMP-8: DATA AND METADATA VERIFICATION

Curation

DMP-9: DATA REVIEW AND REPROCESSING DMP-10: PERSISTENT AND RESOLVABLE IDENTIFIERS

ESA UNCLASSIFIED – For Official Use

Digital Objects Preservation

Generic definition:

Digital Preservation is the management and maintenance of digital objects so they can be accessed and used by future users. In Earth Observation Context

Digital Objects:

Data Records: these include raw data and/or Level-0 data, higher-level products, browse images, auxiliary and ancillary data, calibration and validation data sets, and descriptive metadata; Associated Knowledge: this includes all the Tools used in the Data Records generation, quality control, visualization and value adding, and all the Information needed to make the Data Records understandable and usable by the Designated Community.

ESA UNCLASSIFIED – For Official Use

Data Records Preservation

DATA Records Preservation

Data in different formats, quality, accuracy, etc… Heritage Data Programme:

• Ensures preservation of data and knowledge
• Ensure data discoverability and accessibility
• Valorise and aligns old data to new data to generate time series

Data Preservation Heritage Missions

SEASAT ESA DATA HOLDINGS RECOVERY

• Data recovered from an LTO Tape in 2012 (received from DLR), information needed to understand and open

data retrieved from paper documents.

• Processor created starting from JERS-1 SAR one: alignment of SEASAT, ALOS and JERS-1 SAR L-band data. Full

reprocessing campaign, metadata catalogue, web page creation with all information for users. Data archiving and dissemination.

• JERS-1 (1992-1998) SAR and OPS will follow in Q2 2016.

Comparison of images from Seasat, ERS-2 and Sentinel-1 to map etreat

• f two large glaciers in

southeast Greenland

• ver a 36-year period.

SEASAT SAR data recovery:

• Greenland glaciers as seen by 3 generations of radar missions

Comparison of images from Seasat on 16 August 1978, ERS-2 on August 1996 and Sentinel-1 on 20 August 2014 to map the retreat of two large glaciers in southeast Greenland over a 36-year period. Retreat has been estimated at a rate of about 180 m/yr (top glacier) and 61.5 m/yr (bottom glacier)

Seasat L-band SAR ESA data holdings (European coverage Jul-Oct 1978) fully recovered and accessible online for the first time ever. SEASAT, ALOS and JERS-1 data aligned to generate a long time data series in L-band.

NOAA AVHRR

• 1987-2011

Nimbus-7

• Coastal Zone Color Scanner (CZCS)
• Nov 1978 – Jun 1986

Marine Observation Satellites MOS-1 & MOS-1b

• MESSR (Multi-Spectral Electronic Self-Scanning Radiometer)
• VTIR (Visible and Thermal Infrared Radiometer)
• Feb 1987 – Nov 1995 & Feb 1990 – April 1996

Adeos-1

• OCTS (Ocean Colour and Temperature Scanner)
• Aug 1996 – Jun 1997

SPOT-1 & SPOT-2

• Feb 1986 – Dec 1990 & Jan 1990 – Jun 2009
• HRV (High Resolution Visible) & HRVIR (High Resolution Visible IR)

Priority order being confirmed with CCI representatives

Heritage datasets

Preserving Bits: Technology Timeline

Archives evolution Robotic Libraries

Preserving Bits: Technology Timeline

Archive Media evolution CCT 140MB --------- T10000 8TB

Discoverability: Outputs Timeline

Discoverability evolution

On-request access (via phone or fax). Paper catalogue quicklook to check availability, followed by generation of image on paper and physical delivery (e.g. courier)

Web Catalogues Data Retrieval Time: from days to seconds

Accessibility: Outputs Timeline

Meteosat 1984

Paper / Photographic Paper Data Delivery Format evolution

http://esamultimedia.esa.int/docs/corporate/ESA50_slideshow.pdf

Digital Images

First ERS image Sentinel -1

Film / Printouts

Accessibility: Outputs Timeline

Media dissemination evolution HDD/CD/DVD Exabyte DLT CCT CCT 140MB --------- Unlimited

From Selected Scientist to Crowd Science Outputs Timeline

Usability evolution Selected and expert users (restricted access and usability)

Medium and Large registered Designated community Open & Free on TEP- Thematic Exploitation Platform Open & Free on App & Scientific Cloud

Any user can access the data Selected scientist Restricted Communities

ESA UNCLASSIFIED – For Official Use

Associated Knowledge Preservation

ESA UNCLASSIFIED – For Official Use

Associated Knowledge elements

Software/Tools:

• Software Applications:
• Data Product generation
• Quality control
• Product visualization
• Value adding

Information:

• Documentation
• Images
• Metadata file (information on creation,

access rights, restrictions, preservation history, and rights management)

• Multimedia (Video/Audio)
• SW related “IT Infrastructure”:
• Compiler
• Programming language
• Storage system
• Operative System
• Libraries
• Databases
• Workflows
• Bi directional links
• Schemas
• Email

ESA UNCLASSIFIED – For Official Use

Information Preservation

Information Format for Digital Preservation

• Text documents (often MS Word, Excel Files, txt, etc.) can

be preserved as:

• PostScript, PDF, DSSSL, RTF, ASCII, SGML, TIFF, CGM
• PostScript, PDF, RTF are proprietary
• DSSSL, SGML not (yet?) widely used
• CGM has multiple variants in use
• Images can be preserved as:
• Loss of Quality JPEG, JPEG2000
• Lossless compression TIFF, PBM, PNG, FITS.
• Metadata can be preserved as:
• ASCII, the most durable format for metadata because it is

widespread, backwards compatible when used with Unicode (superset of ASCII), and utilizes human- readable characters, not numeric codes.

• For higher functionality, SGML or XML should be used.
• Multimedia can be preserved as:
• AVI, QuickTime, MPEG, WMV, MJ2.

ESA UNCLASSIFIED – For Official Use

PDF , PDF/A, FITS TIFF , FITS ASCII, XML

MJ2

From various Standards and Publications Sources

Preservation metadata: PREMIS

• 1. Descriptive metadata (domain specific EAD for documents, FITs,

SAFE, etc, ISAR, ISAD)

• 2. Administrative (including rights and permissions)
• 3. Technical (physical needed for implementing most static preservation

functions)

• 4. Structural
• 5. Documenting digital provenance (history)
• 6. Documenting relationship in the preservation repository
• 7. Collaborative
• 8. Generated during the life cycle of the asset to be preserved

ESA UNCLASSIFIED – For Official Use

Data Stewardship Best Practices/Guidelines

Data Stewardship Best Practices

CEOS-WGISS DSIG ESA LTDP Team and GSCB LTDP WG OGC CCSDS CEOS

USA Japan Brazil Other Europe

GEO

? ?

CEOS CEOS CEOS

Spaceborne Earth Observation Spaceborne Earth Observation Spaceborne, airborne, and in- situ Earth Observation Spaceborne, airborne, and in- situ Earth Observation NEEDS

Technical Documents Policy Documents

CEOS Data Stewardship Best Practices Document Tree

Preservation Workflow EO Data Set Consolidation Process

EO Data Preservation Guidelines

Glossary of Acronyms and Terms CEOS EO Space Data Sets EO Data Stewardship Cooperative Framework

Preserved Data Set Content Persistent Identifiers Best Practice

Data Purge Alert Data Purge Alert Procedure & White Paper Individual

• rganizations'

policies (stewardship, access, ...)

Applied to

…

Support Technical implementation procedures Guidelines and best practices on specific topics General guidelines and best practices

…

High level framework documents Applied to

Associated Knowledge Preservation

EVER-EST

Thanks for your attention

ESA UNCLASSIFIED – For Official Use

Backup Slides

ESA UNCLASSIFIED – For Official Use

Heritage Mission Preservation Preserved Data Set Content

EO Missions/Sensors Dataset is defined as:

• 1. Data Records: these include raw data and/or Level-0

data, higher-level products, browse images, auxiliary and ancillary data, calibration and validation data sets, and descriptive metadata;

• 2. Associated Knowledge: this includes all the Tools

used in the Data Records generation, quality control, visualization and value adding, and all the Information needed to make the Data Records understandable and usable by the Designated Community

ESA UNCLASSIFIED – For Official Use

Data Records

• Raw data
• Level 0 data (L0)
• Level 1 (L1) to higher levels mission data products when generated

as part of the mission requirements and/or reprocessed

• Browses whenever generated
• Ancillary data (spacecraft ephemeris information, attitude, etc.)
• Auxiliary data (required to process the telemetry payload data to

generate the nominal mission products)

• Calibration and validation datasets (needed to calibrate the satellite

instruments and monitor data quality)

• Metadata

ESA UNCLASSIFIED – For Official Use

Tools

• 1. L0 consolidation software
• 2. Data processing software (for products generation from

Level 0 to higher levels according to mission requirements)

• 3. Quality control software
• 4. Data/products visualization tools
• 5. Value adding tools

ESA UNCLASSIFIED – For Official Use

Information

• 1. Mission architecture documents describing purpose, scope and

performances of the mission and of the on-board instruments,

• 2. Data and product format specifications.
• 3. Measurement requirements and/or measurement performances

(theoretical models).

• 4. Instruments characteristics, performances and instrument

description (physical implementations).

• 5. Reports concerned with measurement trends, failures, changes of

performances, un-availabilities

• 6. Reports and outcomes from events such as: congresses, studies,

communities and investigators concerned with models’ review, algorithm changes, and Cal/Val changes affecting data processing chains.

ESA UNCLASSIFIED – For Official Use

Information

• 1. Documents related to the process of data qualification: precision,

numerical representations, formats, uncertainties, errors, adjustment/correction methods (e.g. Cal/Val procedures and documents).

• 2. Document related to workflows, work procedure, documentation

three and bi-directional link

• 3. Scientific publications based on the data exploitation or relevant to

them (properly linked to the data) and outreach material.

• 4. Administrative (Memorandum, Intellectual Property Rights, etc.)
• 5. Mission Data Records and Documentation Tree

ESA UNCLASSIFIED – For Official Use

• Data records and knowledge consolidation, maturity assessment and improvements

implementation, based on new user requirements and value of the data set for societal application domains

• Metadata: for accessibility and interoperability with family of sensors, data

management, citation (persistent identifiers)

• Documentation as need to fill in gaps of broadened target community knowledge

base and maximize exploitation

• Alignment to output format of latest sensor family product to facilitate massive long

term data series processing

• Reprocessing baseline alignment and characterisation to latest family sensors

algorithm, product model, and/or improved ancillary and auxiliary data (navigation/calibration data)

• Quality information generation and extraction in accordance to guidelines

Heritage Mission Curation