Active Data A Data-Centric Approach to Data Life-Cycle Management - - PowerPoint PPT Presentation
Introduction Active Data Discussion Conclusion Active Data A Data-Centric Approach to Data Life-Cycle Management Anthony Simonet 1 Gilles Fedak 1 Matei Ripeanu 2 Samer Al-Kiswany 2 1 Inria, ENS Lyon, University of Lyon 2 University of British
Introduction Active Data Discussion Conclusion
1Inria, ENS Lyon, University of Lyon 2University of British Columbia
Active Data (PDSW’13) November 18th, 2013 1/20
Introduction Active Data Discussion Conclusion
Active Data (PDSW’13) November 18th, 2013 2/20
Introduction Active Data Discussion Conclusion
Active Data (PDSW’13) November 18th, 2013 3/20
◮ Science and Industry have become data-intensive
◮ Volume of data produced by science and industry grows exponentially ◮ How to store this deluge of data? ◮ How to extract knowledge and sense? ◮ How to make data valuable?
◮ Some examples
◮ CERN’s Large Hadron Collider: 1.5PB/week ◮ Large Synoptic Survey Telescope, Chile: 30 TB/night ◮ Billion edge social network graphs ◮ Searching and mining the Web
Introduction Active Data Discussion Conclusion
◮ Creation/Acquisition ◮ Transfer ◮ Replication ◮ Disposal/Archiving
Active Data (PDSW’13) November 18th, 2013 4/20
Introduction Active Data Discussion Conclusion
◮ Execution of workflow ◮ Complex interactions between software ◮ Need to quickly react to operational events
◮ Hard to program, maintain and debug ◮ No formal specification ◮ Complicates interactions between systems
Active Data (PDSW’13) November 18th, 2013 5/20
Introduction Active Data Discussion Conclusion
◮ 100TB of raw data per day ◮ Raw data are preprocessed and registered in a Globus dataset
◮ Data are analyzed by various applications ◮ Results are stored in the dataset catalog and shared
Instrument (Beamline) Local Storage
Transfer
Metadata Catalog
Extract & Register Metadata
Remote Data Center
T r a n s f e r
Academic Cluster
Analysis More analysis Upload result R e g i s t e r r e s u l t m e t a d a t a
Active Data (PDSW’13) November 18th, 2013 6/20
Introduction Active Data Discussion Conclusion
◮ Independent scripts ◮ Express data-dependancies ◮ Hard to program, maintain, verify ◮ Cross data-center coordination ◮ Coarse granularity ◮ User-level fault-tolerance ◮ Incremental processing
Active Data (PDSW’13) November 18th, 2013 7/20
Introduction Active Data Discussion Conclusion
◮ Simply represent the life cycle of data distributed across
◮ Simplify DLM modeling and reasoning ◮ Hide the complexity resulting from using different
◮ Be easy to integrate with existing systems
Active Data (PDSW’13) November 18th, 2013 8/20
Introduction Active Data Discussion Conclusion
t1 Written t2 Read t3 t4 Terminated
Active Data (PDSW’13) November 18th, 2013 9/20
Introduction Active Data Discussion Conclusion
Created t1
t2 Read t3 t4 Terminated
Active Data (PDSW’13) November 18th, 2013 9/20
Introduction Active Data Discussion Conclusion
Created t1
t2 Read t3 t4 Terminated public void handler () { computeMD5 (); }
Active Data (PDSW’13) November 18th, 2013 9/20
Introduction Active Data Discussion Conclusion
◮ Allows to react to life cycle progression ◮ Exposes transparently distributed data sets ◮ Can be integrated with existing systems ◮ Has scalable performance and minimum overhead over
Active Data (PDSW’13) November 18th, 2013 10/20
Introduction Active Data Discussion Conclusion
◮ Prototype implemented in Java (≃ 2,800 LOC) ◮ Client/Service communication is Publish/Subscribe ◮ 2 types of subscription:
◮ Every transitions for a given data item ◮ Every data items for a given transition
Client Client
Client
Active Data (PDSW’13) November 18th, 2013 11/20
Introduction Active Data Discussion Conclusion
◮ Several ways to publish transitions
◮ Instrument the code ◮ Read the logs ◮ Rely on an existing notification system
◮ The service orders transitions by time of arrival
Client
Client
Client
Active Data (PDSW’13) November 18th, 2013 11/20
Introduction Active Data Discussion Conclusion
◮ Clients run transition handler code locally ◮ Transition handlers are executed
◮ Serially ◮ In a blocking way ◮ In the order transitions were published
Client
Client
Client
Active Data (PDSW’13) November 18th, 2013 11/20
Introduction Active Data Discussion Conclusion
10 50 100 200 300 400 450 500 550 # clients 5,000 10,000 15,000 20,000 25,000 30,000 35,000 Transitions per second
Active Data (PDSW’13) November 18th, 2013 12/20
Introduction Active Data Discussion Conclusion
10 50 100 200 300 400 450 500 550 # clients 5,000 10,000 15,000 20,000 25,000 30,000 35,000 Transitions per second
Active Data (PDSW’13) November 18th, 2013 12/20
Introduction Active Data Discussion Conclusion
◮ Assess the quality of data ◮ Keep track of the origin of data over time ◮ Specialized Provenance Aware Storage Systems
Active Data (PDSW’13) November 18th, 2013 13/20
Introduction Active Data Discussion Conclusion
◮ Assess the quality of data ◮ Keep track of the origin of data over time ◮ Specialized Provenance Aware Storage Systems
Active Data (PDSW’13) November 18th, 2013 13/20
Introduction Active Data Discussion Conclusion
◮ Assess the quality of data ◮ Keep track of the origin of data over time ◮ Specialized Provenance Aware Storage Systems
Active Data (PDSW’13) November 18th, 2013 13/20
Introduction Active Data Discussion Conclusion
Active Data (PDSW’13) November 18th, 2013 14/20
Created Get Put Terminated t5 t9 t6 t7 t8 t10
iRODS
t1 t2 Succeeded Failed t3 t4 Terminated
Globus Online
Id: {GO: 7b9e02c4-925d-11e2}
Introduction Active Data Discussion Conclusion
Active Data (PDSW’13) November 18th, 2013 14/20
Created Get Put Terminated t5 t9 t6 t7 t8 t10
iRODS
Created t1 t2
Failed t3 t4 Terminated
Globus Online
public void handler () { iput (...); }
Introduction Active Data Discussion Conclusion
Active Data (PDSW’13) November 18th, 2013 14/20
Get Put Terminated t5 t9 t6 t7 t8 t10
iRODS
Id: {GO: 7b9e02c4-925d-11e2, iRODS: 10032}
Created t1 t2 Succeeded Failed t3 t4 Terminated
Globus Online
Introduction Active Data Discussion Conclusion
Active Data (PDSW’13) November 18th, 2013 14/20
Created Get
Terminated t5 t9 t6 t7 t8 t10
iRODS
public void handler () { annotate (); } Created t1 t2 Succeeded Failed t3 t4 Terminated
Globus Online
Introduction Active Data Discussion Conclusion
$ imeta ls -d test/ out_test_4628 AVUs defined for dataObj test/ out_test_4628 : attribute: GO_FAULTS value: 0
value: 2013 -03 -21 19:28:41Z
value: 2013 -03 -21 19:28:17Z
value: 7b9e02c4 -925d -11e2 -97ce -123139404 f2e
value: go#ep1 /~/ test
value: asimonet#fraise /~/ out_test_4628
Active Data (PDSW’13) November 18th, 2013 15/20
Introduction Active Data Discussion Conclusion
◮ Simple and graphical way to program DLM operations ◮ Allows to formally verify some properties of data life cycles ◮ Easy coordination between systems ◮ Easy to scale ◮ Easy to debug ◮ Easy fault tolerance ◮ Fine-grain interaction with data life cycle
Active Data (PDSW’13) November 18th, 2013 16/20
Introduction Active Data Discussion Conclusion
◮ Complexity to reason in terms of life cycle events ◮ Lack of standard
Active Data (PDSW’13) November 18th, 2013 17/20
Introduction Active Data Discussion Conclusion
◮ Programing models:
◮ MapReduce and higher level abstractions: PigLatin, Twister ◮ Incremental systems: MapReduce-Online, Percolator, Chimera,
◮ Other models with implicit parallelism: Swift, Dryad, Allpairs
◮ Storage systems
◮ BitDew ◮ MosaStore ◮ Provenance Aware Storage Systems ◮ Active Storage
Active Data (PDSW’13) November 18th, 2013 18/20
Introduction Active Data Discussion Conclusion
◮ Data-centric & Event-driven ◮ System-level data integration
◮ Advanced representation of operations that consume and
◮ Data collection abilities ◮ Distributed implementation of the Publish/Subscribe layer
Active Data (PDSW’13) November 18th, 2013 19/20
Introduction Active Data Discussion Conclusion
Active Data (PDSW’13) November 18th, 2013 20/20