Data-Intensive Workfmows A journey to a Holistjc Framework for - - PowerPoint PPT Presentation
Data-Intensive Workfmows A journey to a Holistjc Framework for Data-Intensive Workfmows Ian Corner Design and Implementatjon Lead May 2016 INFORMATION MANAGEMENT AND TECHNOLOGY (IMT) CSIRO Data-Intensive Workfmows Holistjc
A journey to a Holistjc Framework for Data-Intensive Workfmows
INFORMATION MANAGEMENT AND TECHNOLOGY (IMT)
Ian Corner – Design and Implementatjon Lead – May 2016
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
Commonwealth Scientjfjc and Industrial Research Organisatjon
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
Strategy 2020 – Australia’s Innovatjon Catalyst
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
Australian Natjonal Insect Collectjon 12,000,000 specimens (+100,000 per year) Australian Natjonal Fish collectjon 5,000 species Australian Natjonal Algae Culture Collectjon 1,000 strains of more than 300 micro-algae species Australian Natjonal Herbarium 1,000,000 herbarium (Captain Cook’s 1770 expeditjon to Australia) Australian Natjonal Wildlife Collectjon 200,000 irreplaceable specimens of wildlife http://www.csiro.au/en/Research/Collections
Commonwealth Scientjfjc and Industrial Research Organisatjon
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
http://www.csiro.au/en/Research/Collections/ANIC
Physical collectjons, Captured and Preserved
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
http://data.csiro.au/
We need collectjons digitjsed, discoverable, consumable
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
Commonwealth Scientjfjc and Industrial Research Organisatjon
http://www.csiro.au/en/Research/Facilities/Marine-National-Facility/RV-Investigator
RV Investigator is our state-
vessel, supporting Australia’s atmospheric,
and geosciences research from the tropical north to the Antarctic ice-edge.
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
CSIRO started by asking what good is our data if it: is unable to be found? can not speak?
can not repeat the same story twice? speaks so slowly the message is lost?
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
As data growth and proliferation continued to outpace research grade infrastructure, we considered a new approach?
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
Lets revisit the “monolithic approach”
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
We split the monolithic file systems into named and discoverable 'datasets.'
The 'dataset' approach delineated the 'responsibility' between infrastructure owners and dataset managers.
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
Within the dataset we developed 'categories‘ as a tool for data management.
Categories enabled mapping of the workflow to technology of best fit.
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
Categories “kick” started the discussion about workflows.
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
We established the ‘relationships’ between owners, domain specialists, users, consumers, and infrastructure.
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
As workflows matured, “science apps” evolved enabling domain specific datasets to be usable by non-domain consumers.
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
The Pyrotron - CSIRO National Bushfire Research Facility
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
http://www.csiro.au/en/Do-business/Services/Testing-and-technical-services/Enviro/Pyrotron
CSIRO – Workspace - Intuitive Workflow Development Tool
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
https://research.csiro.au/workspace/
CSIRO – SPARK – A wild fire simulation tool
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
https://research.csiro.au/spark/
SPARK – A wildfire simulation framework for researchers and experts in the disaster resilience field.
Our leading edge researchers combined domain specific workflows to produce higher value layered products.
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
Our leading edge researchers combined domain specific workflows to produce higher value layered products.
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
Below the line 'technology' is a consumable, replaceable, discardable commodity.
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
Below the line - the “fit for purpose” pool of generic infrastructure
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
CSIRO's value proposition is the “Workflow.”
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
Crossing the line we deliver to the 'current' profile of the researchers workflow.
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
Layers of abstraction enabled us to “scale up.”
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
Layers of abstraction enabled us to “scale up.”
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
Layers of abstraction enabled us to “scale out.”
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
We came from a position where data, code and compute were isolated by the approach to HPC infrastructure.
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
We engineered a solution where data, code and compute are all now directly connected.
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
CSIRO’s data-intensive workflows are a valuable source of information. How do we discover them, trust them and consume them?
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
CSIRO’s data-intensive workflows are a valuable source of information. How do we discover them, trust them and consume them?
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
CSIRO’s data-intensive workflows are a valuable source of information. How do we discover them, trust them and consume them?
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
CSIRO’s data-intensive workflows are a valuable source of information. How do we discover them, trust them and consume them?
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
Metadata is a pathway to making data and workfmows discoverable.
Lets look at Wikipedia: htups://en.wikipedia.org/wiki/Metadata
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
Metadata is "data that provides informatjon about other data”. Two types of metadata exist: structural metadata and descriptjve metadata. Structural metadata is data about the containers of data. Descriptjve metadata uses individual instances of applicatjon data or the data content.
Dr Victoria Stodden at the CSIRO Computatjon Simulatjon Sciences and eResearch Annual Conference in Melbourne 2014
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
Pragmatjc use of the web.
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
Lets look at Wikipedia: htups://en.wikipedia.org/wiki/Semantjc_Web
The Semantjc Web is an extension of the Web through standards by the World Wide Web Consortjum (W3C). The standards promote common data formats and exchange protocols on the Web, most fundamentally the Resource Descriptjon Framework (RDF).
We need to link metadata, provenance and semantics in an automatic and extensible manner to increase our value.
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
What are we currently loosing
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
Preserving the context of discrete events
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
Create a blank dataset
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
Create a blank dataset – Preserving metadata, establishing provenance, ...
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
Ingestjng sensor data – Preserving metadata, establishing provenance, ...
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
Lets consider a real world applicatjon - PlantScan
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
http://www.plantphenomics.org.au/services/plantscan/ PlantScan provides non-invasive analyses of plant structure (topology, surface orientation, number of leaves), morphology (leaf size, shape, colour, area, volume) and function by utilising cutting edge information technology including high resolution cameras and three-dimensional (3D) reconstruction software.
Plant surface mesh reconstruction Morphological mesh segmentation Accurate phenotypic data extraction Longitudinal matching
Plant Scan: Workfmow with integrated metadata
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
E X A M P L E O N L Y
Benefjt 1 – Repeatable Analysis
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
Now there is no ‘simple’ fjx to this issue. But if we issue the ‘birth certjfjcate’ before the baby leaves the hospital then we have at least improved our positjon. We reduce the size
Benefjt 2 – Quality Assurance
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
File_1 became File_2 using version 56 of Fred_1. Now if File_1 had a calibratjon issue. Or Fred_1 had an analysis bug. Guest what? We reduce our problem more.
Other benefjts
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
Benefjt 3: Immediate Consumptjon
Other benefjts
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
Benefjt 3: Immediate Consumptjon Benefjt 4: Benchmarks
Other benefjts
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
Benefjt 3: Immediate Consumptjon Benefjt 4: Benchmarks Benefjt 5: Infrastructure Management
Other benefjts
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
Benefjt 3: Immediate Consumptjon Benefjt 4: Benchmarks Benefjt 5: Infrastructure Management Benefjt 6: Failures
Summary
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
In Unix everything is a fjle (pretuy much) there are a set of well writuen simple tools which you can tje together in an ad-hoc way to produce high value outcomes in a dynamic yet robust manner.
Everything is a dataset, there are a set of published, well proven and tested set of ‘research’ workfmows which you can tje together in an ad-hoc way to produce high value outcomes in a dynamic yet robust manner.
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
Summary
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016
INFORMATION MANAGEMENT AND TECHNOLOGY (IMT)
CSIRO – Data-Intensive Workfmows – Holistjc Framework for Data-Intensive Workfmows – Ian Corner 2016