named data networking in scientific applications
play

NAMED DATA NETWORKING IN SCIENTIFIC APPLICATIONS Susmit - PowerPoint PPT Presentation

Sep 28, 2022 •569 likes •937 views

NAMED DATA NETWORKING IN SCIENTIFIC APPLICATIONS Susmit Shannigrahi, Chengyu Fan and Christos Papadopoulos Colorado State University March 23, 2017 Work supported by NSF #1345236 and #13410999 CMIP5 Servers 2 2 3 Years of CMIP5 Data Access

  1. NAMED DATA NETWORKING IN SCIENTIFIC APPLICATIONS Susmit Shannigrahi, Chengyu Fan and Christos Papadopoulos Colorado State University March 23, 2017 Work supported by NSF #1345236 and #13410999

  2. CMIP5 Servers 2 2

  3. 3 Years of CMIP5 Data Access  CMIP5 is a 3.3PB archive of climate data, made available to the community through ESGF (~25 nodes) (CMIP6 estimated into the exabytes)  We look at one server log collected at the LLNL ESGF node  Approximately 3 years of requests (2013 to 2016)  18.5 million total requests (many duplicate)  1.5M Unique datasets requested  Total size Requests (with dups) = 1,844TB 3

  4. Client Locations 4 4

  5. ASN Map • Done using reverse traceroute • Little path overlap, but view from only one ESGF node 5

  6. User/Clients Statistics Unique Users 5692 Unique Clients (IP 9266 addresses) Unique ASNs 911 6

  7. User Distribution per ASN 7

  8. Dataset Size Distribution 95% percentile: 1.34GB 8

  9. Data Popularity (98% of the datasets was requested by 10 users or less) 9

  10. Successful vs Failed Requests 10

  11. Summary: Data Statistics CMIP5 Archive Size 3.3PB Total Data Requested Equivalent of 1.8PB (18.5M requests) Total Data Successfully 234 TB (1.9M requests) Retrieved Total Data Successfully 113 TB (415K requests) Retrieved (Excluding Duplicates) Number of unique datasets 1.5 million requested 11

  12. A Closer Look at Failures Number of requests 18.5 million Successful Requests 1,935,256 Failed Requests 16,673,815 12

  13. Client Request Failures 13

  14. Duplicate Requests by Failure Group 14

  15. Failure Heatmap 15

  16. CMIP5 Data Retrieval Today  HTTP://someESGFnode:/CMIP5/output/MOHC/HadCM3/dec adal1990/day/atmos/tas/r3i2p1/tas_Amon_HADCM3_ historical_r1i1p1_185001-200512.nc 16

  17. CMIP5 Retrieval with NDN  HTTP://someESGFnode:/CMIP5/output/MOHC/HadCM3/dec adal1990/day/atmos/tas/r3i2p1/tas_Amon_HADCM3_ historical_r1i1p1_185001-200512.nc 17 17

  18. Why make the change?  Does it improve performance ?  Does it improve publishing ?  Does it improve discovery ?  Does it improve resilience/availability ?  Does it improve security/integrity ?  We begin to answer these questions by analyzing a real CMIP5 log 18 18

  19. NDN Catalog and Retrieval Catalog node 1 Data storage Catalog node 3 Publisher NDN Data storage Consumer Catalog node 2 19

  20. NDN Catalog and Retrieval Catalog node 1 Data storage Catalog node 3 (1)Publish Dataset names Publisher NDN Data storage Consumer Catalog node 2 20

  21. NDN Catalog and Retrieval Catalog node 1 Data storage Catalog node 3 Publisher NDN Data storage Consumer Catalog node 2 21

  22. NDN Catalog and Retrieval Catalog node 1 Data storage Catalog node 3 (2) Sync changes Publisher NDN Data storage Consumer Catalog node 2 22

  23. NDN Catalog and Retrieval Catalog node 1 Data storage Catalog node 3 Publisher NDN Data storage Consumer Catalog node 2 23

  24. NDN Catalog and Retrieval Catalog node 1 Data storage Catalog node 3 Publisher NDN Data storage (3) Query for Dataset names Consumer Catalog node 2 24

  25. NDN Catalog and Retrieval Catalog node 1 Data storage Catalog node 3 Publisher NDN Data storage Consumer Catalog node 2 25

  26. NDN Catalog and Retrieval Catalog node 1 Data storage Catalog node 3 Publisher NDN Data storage Consumer Catalog node 2 26

  27. NDN Catalog and Retrieval Catalog node 1 Data storage Catalog node 3 (1)Publish Dataset names (2) Sync changes Publisher NDN Data storage (3) Query for Dataset names Consumer Catalog node 2 27

  28. NDN Catalog and Retrieval Catalog node 1 Data storage Catalog node 3 (1)Publish Dataset names (2) Sync changes Publisher NDN Data storage (3) Query for Dataset names (4) Retrieve data Consumer Catalog node 2 28

  29. NDN Catalog and Retrieval Catalog node 1 Data storage Catalog node 3 (1)Publish Dataset names (2) Sync changes Publisher NDN Data storage (3) Query for Dataset names (4) Retrieve data Consumer Catalog node 2 29

  30. NDN Catalog and Retrieval Catalog node 1 Data storage Catalog node 3 (1)Publish Dataset names (2) Sync changes Publisher NDN Data storage (3) Query for Dataset names (4) Retrieve data Consumer Catalog node 2 30

  31. Improvements with NDN  Performance – seamless retrieval from the best performing locations  Publishing – authenticated, only owner can publish  Discovery – distributed catalog, anycast-style discovery  Resilience/availability - seamless retrieval from multiple locations  Security/integrity – enabled by signed data 31 31

  32. Science NDN Testbed  NSF CC-NIE campus infrastructure award  10G testbed (courtesy of ESnet, UCAR, and CSU Research LAN)  Currently ~50TB of CMIP5, ~20TB of HEP data 32

  33. Vision: Integration with OS and FS With Alex Afanasyev and Lixia Zhang 33 33

  34. Conclusions  NDN encourages common data access methods where IP encourages common host access methods  NDN encourages interoperability at the content level  NDN unifies scientific data access methods  Eliminates repetition of functionality  Adds significant security leverage  Rewards structured naming 34

  35. For More Info christos@colostate.edu susmit.shannigrahi@gmail.com http://named-data.net http://github.com/named-data 35

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

NAMED DATA NETWORKING (NDN) Named Data Networking NDN BRIEF HISTORY When the Networking was

NAMED DATA NETWORKING (NDN) Named Data Networking NDN BRIEF HISTORY When the Networking was

NAMED DATA NETWORKING (NDN) Named Data Networking NDN BRIEF HISTORY When the Networking was developed in the 60s and 70s Networking was mainly used for resource sharing. IP was the effective communication protocol in place.

484 views • 29 slides
Advancing Applications from the Edge In with Named Data Networking Lixia Zhang July, 2017

Advancing Applications from the Edge In with Named Data Networking Lixia Zhang July, 2017

Advancing Applications from the Edge In with Named Data Networking Lixia Zhang July, 2017 ice-ar.named-data.net 1 NDN-Enabled Secure Edge Networking with Augmented Reality Integration of heterogeneous low-level wireless tech. with

540 views • 13 slides
Support Mobile and Distributed Applications with Named Data Networking Zhenkai Zhu Computer

Support Mobile and Distributed Applications with Named Data Networking Zhenkai Zhu Computer

Support Mobile and Distributed Applications with Named Data Networking Zhenkai Zhu Computer Science Department University of California, Los Angeles CA, 90095 zhenkai@cs.ucla.edu May 22, 2013 Zhenkai Zhu (UCLA) Ph.D Final Defense May 22,

807 views • 57 slides
Distributed Dataset Synchronization in Named Data Networking Wentao Shang Final defense

Distributed Dataset Synchronization in Named Data Networking Wentao Shang Final defense

Distributed Dataset Synchronization in Named Data Networking Wentao Shang Final defense 06/01/2017 1 Research Problem Distributed applications require efficient support for multi-party communication Multiple nodes publish and share

1.31k views • 71 slides
Named Data Networking 1. V Jacobson, D.K. Smetters, J.D. Thornton, M.F. Plass, N.H. Briggs, R.L.

Named Data Networking 1. V Jacobson, D.K. Smetters, J.D. Thornton, M.F. Plass, N.H. Briggs, R.L.

Named Data Networking 1. V Jacobson, D.K. Smetters, J.D. Thornton, M.F. Plass, N.H. Briggs, R.L. Braynard: Networking Named Content, CoNEXT, 2009. 2. L. Zhang, D. Estrin, J. Burke, V. Jacobson, J.D. Thornton, D.K. Smetters, B. Zhang, G. Tsudik,

762 views • 15 slides
beyond network selection: exploiting access network hetero- geneity with named data networking

beyond network selection: exploiting access network hetero- geneity with named data networking

2nd ACM Conference on Information-Centric Networking . Klaus M. Schneider, Udo R. Krieger October 2, 2015 University of Bamberg, Germany 0 beyond network selection: exploiting access network hetero- geneity with named data networking 1

725 views • 25 slides
Sh Shari ring mH mHealth Da Data a via via Na Named Da Data Ne Networking Haitao Zhang 1

Sh Shari ring mH mHealth Da Data a via via Na Named Da Data Ne Networking Haitao Zhang 1

Sh Shari ring mH mHealth Da Data a via via Na Named Da Data Ne Networking Haitao Zhang 1 , Zhehao Wang 2 , Christopher Scherb 3 , Claudio Marxer 3 , Jeff Burke 2 , Lixia Zhang 1 , Christian Tschudin 3 1. UCLA IRL 2. UCLA REMAP 3.

752 views • 20 slides
Application aware access and distribution of digital objects using Named Data Networking (NDN)

Application aware access and distribution of digital objects using Named Data Networking (NDN)

Application aware access and distribution of digital objects using Named Data Networking (NDN) July 4, 2017 University of Amsterdam System and Network Rahaf Mousa Engineering Supervisor: dr.Zhiming Zhao 1 Motivation In big data

586 views • 21 slides
Towards Evaluating Named Data Networking for the IoT: A Framework for OMNeT++ Amar Abane 1 2 ,

Towards Evaluating Named Data Networking for the IoT: A Framework for OMNeT++ Amar Abane 1 2 ,

Towards Evaluating Named Data Networking for the IoT: A Framework for OMNeT++ Amar Abane 1 2 , Paul Muhlethaler 3 , Samia Bouzefrane 1 , Mehammed Daoui 2 , and Abdella Battou 4 1 Conservatoire National des Arts et Mtiers - Paris, France 2

558 views • 23 slides
NDN: an Orchestrated Microservice Architecture for Named Data Networking Xavier MARCHAL,

NDN: an Orchestrated Microservice Architecture for Named Data Networking Xavier MARCHAL,

NDN: an Orchestrated Microservice Architecture for Named Data Networking Xavier MARCHAL, Thibault CHOLEZ, Olivier FESTOR LORIA, UMR 7503 (University of Lorraine, CNRS, INRIA) Vandoeuvre-les-Nancy, F-54506, France September 22, 2018

786 views • 21 slides
From Crowd to Cloud: Apply Named Data Networking Principles to Manage Air Sensor Data Yifang

From Crowd to Cloud: Apply Named Data Networking Principles to Manage Air Sensor Data Yifang

From Crowd to Cloud: Apply Named Data Networking Principles to Manage Air Sensor Data Yifang Zhu, UCLA School of Public Health Alex Afanasyev & Lixia Zhang, UCLA Computer Science 1 New opportunities Technology advances in low-cost and

384 views • 12 slides
NDN DeLorean: An Authentication System for Data Archives in Named Data Networking Yingdi Yu

NDN DeLorean: An Authentication System for Data Archives in Named Data Networking Yingdi Yu

NDN DeLorean: An Authentication System for Data Archives in Named Data Networking Yingdi Yu (UCLA), Alexander Afanasyev (Florida International University), Jan Seedorf (HFT Stuttgart), Zhiyi Zhang (UCLA), Lixia Zhang (UCLA), ACM Information

455 views • 18 slides
Realizing a Virtual Private Network using Named Data Networking September 28, 2017 Craig

Realizing a Virtual Private Network using Named Data Networking September 28, 2017 Craig

Realizing a Virtual Private Network using Named Data Networking September 28, 2017 Craig Partridge Samuel Nelson Derrick Kong Th This document does not contain technology or te technical data controlled under either the U.S. In Inter

431 views • 13 slides
A World on NDN Affordances and Implications of the Named Data Networking Future Internet

A World on NDN Affordances and Implications of the Named Data Networking Future Internet

A World on NDN Affordances and Implications of the Named Data Networking Future Internet Architecture Shilton, Burke, claffy, Duan, Zhang Examines the social impacts of NDN generated by key aspects of the architecture and critical

277 views • 6 slides
Pending Interest Table Sizing in Named Data Networking Luca Muscariello Orange Labs Networks /

Pending Interest Table Sizing in Named Data Networking Luca Muscariello Orange Labs Networks /

Pending Interest Table Sizing in Named Data Networking Luca Muscariello Orange Labs Networks / IRT SystemX G. Carofiglio (Cisco), M. Gallo, D. Perino (Bell Labs) 2 nd ACM Conference on Information-Centric Networking San Francisco 1st of

412 views • 19 slides
Schematizing Trust in Named Data Networking Yingdi Yu 1 , Alex Afanasyev 1 , David Clark 2 , kc

Schematizing Trust in Named Data Networking Yingdi Yu 1 , Alex Afanasyev 1 , David Clark 2 , kc

Schematizing Trust in Named Data Networking Yingdi Yu 1 , Alex Afanasyev 1 , David Clark 2 , kc claffy 3 , Van Jacobson 1 , Lixia Zhang 1 1. UCLA 2. MIT 3. CAIDA 1 Motivation Usability is critical to security solutions

259 views • 22 slides
Care Transitions Network Data Jam October 28, 2016 National Council for Behavioral Health

Care Transitions Network Data Jam October 28, 2016 National Council for Behavioral Health

Care Transitions Network Data Jam October 28, 2016 National Council for Behavioral Health Montefiore Medical Center Northwell Health New York State Office of Mental Health Netsmart Technologies Erica Van De Wal Anni Kramer Medical

709 views • 26 slides
GDPR Update 3 October 2019 Phil Tompkins and Dean Murray Newcastle | Leeds | Manchester 2 What

GDPR Update 3 October 2019 Phil Tompkins and Dean Murray Newcastle | Leeds | Manchester 2 What

GDPR Update 3 October 2019 Phil Tompkins and Dean Murray Newcastle | Leeds | Manchester 2 What we will look at today GDPR to date How to handle data subject access requests Data security and handling data breaches Whats

808 views • 49 slides
1. The potential of data sharing 2. An ideal professed but not practiced * 1. Researchers 2.

1. The potential of data sharing 2. An ideal professed but not practiced * 1. Researchers 2.

Perspectives on Academic Data Sharing Benedikt Fecher, German Institute for Economic Research Berlin, 19th April 2016 8,65 Data Sharing in Academia Agenda 1. The potential of data sharing 2. An ideal professed but not practiced * 1. Researchers

329 views • 22 slides
Internet Indirection Infrastructure Karthik Lakshminarayanan UC Berkeley Contrasting LNA, HIP,

Internet Indirection Infrastructure Karthik Lakshminarayanan UC Berkeley Contrasting LNA, HIP,

Internet Indirection Infrastructure Karthik Lakshminarayanan UC Berkeley Contrasting LNA, HIP, and i3 LNA = Layered Naming Architecture LNA, HIP, i3: All network architecture proposals Separate location and identity

257 views • 15 slides
Learning to Hash with its Application to Big Data Retrieval and Mining o Department of

Learning to Hash with its Application to Big Data Retrieval and Mining o Department of

Learning to Hash with its Application to Big Data Retrieval and Mining o Department of Computer Science and Engineering Shanghai Jiao Tong University Shanghai, China Joint work with h , , L Dec 21, 2013 Li (

760 views • 58 slides
CYBERSECURITY STRATEGIES TO MANAGE BUSINESS RISKS A C O N V E R S A T I O N W I T H H O R N E

CYBERSECURITY STRATEGIES TO MANAGE BUSINESS RISKS A C O N V E R S A T I O N W I T H H O R N E

CYBERSECURITY STRATEGIES TO MANAGE BUSINESS RISKS A C O N V E R S A T I O N W I T H H O R N E I T S P E C I A L I S T S P A N E L I S T S M I K E S K I N N E R , C PA , C I S A , C I T P S e n i o r M a n a g e r I T A s s u r a n

634 views • 18 slides
ATLAS I/O Overview Peter van Gemmeren (ANL) gemmeren@anl.gov for many in ATLAS 8/23/2018 Peter

ATLAS I/O Overview Peter van Gemmeren (ANL) gemmeren@anl.gov for many in ATLAS 8/23/2018 Peter

ATLAS I/O Overview Peter van Gemmeren (ANL) gemmeren@anl.gov for many in ATLAS 8/23/2018 Peter van Gemmeren (ANL): ATLAS I/O Overview 1 High level overview of ATLAS Input/Output framework and data persistence. Athena: The ATLAS event

690 views • 28 slides
Stream Algorithmics Albert Bifet March 2012 Data Streams Big Data & Real Time Data Streams

Stream Algorithmics Albert Bifet March 2012 Data Streams Big Data & Real Time Data Streams

Stream Algorithmics Albert Bifet March 2012 Data Streams Big Data & Real Time Data Streams Data Streams Sequence is potentially infinite High amount of data: sublinear space High speed of arrival: sublinear time per example

969 views • 50 slides

More recommend