Enterprise Data Management (EDM) and Enterprise Product Generation (EPG) Proving Ground in the AWS Cloud 2019 AMS Annual Meeting Rich Baker Solers, Inc. ESPDS Development Chief Architect Peter MacHarrie, Solers , Inc. John Sobanski, Solers, Inc. Solers Email: richard.baker@solers.com Jakku Reddy, Solers, Inc. Steve Walsh, Solers, Inc. NOAA Email: richard.baker@noaa.gov Hieu Phung, Solers, Inc. Ron Niemann, Solers, Inc. Phone: (240) 790-3338 Steve Causey, Solers, Inc. Dan Beall, Solers, Inc.
Solers created a Proving Ground for Enterprise Data Management (EDM) and Enterprise Product Generation (EPG) services in a FedRAMP-approved Amazon Web Services (AWS) cloud environment, leveraging native AWS cloud services and NESDIS product generation algorithms. Developed under the Environmental Satellite EDM and EPG Proving Ground in the AWS Cloud Processing and Distribution System (ESPDS) contract. EDM service provides data storage, a flexible and searchable inventory/catalog of product Cloud Watch metadata, and science data manipulation through RESTful interfaces. Leverages native SNS/SQS Data Transport EDM EPG AWS cloud services including: Elasticsearch, RDS, S3, Lambda, and API Gateway. ESPDS PDA @ DynamoDB ElasticSearch RDS NSOF I&T EPG is capable of generating NESDIS level 1+ sensor, science, EDM SNS SQS RDS EDM Client and tailored product types. Client Leverages native AWS cloud Job Factory services including: EC2 with Auto- S3 Scaling, RDS, SNS, and SQS . NCEI GOES 16 EDM Client Data currently being ingested: EDM Lambdas Computer GOES-16 data from the NOAA/NCEI o Nodes Big Data Project (AWS S3 bucket). S3 EC2 Auto Scale API Gateway S-NPP, JPSS-1, and GCOM-W data o (Orbit-Based) from ESPDS PDA at NSOF I&T. VPC/IAM SNS 2
Solers created a Proving Ground for Enterprise Data Management (EDM) and Enterprise Product Generation (EPG) services in a FedRAMP-approved Amazon Web Services (AWS) cloud environment, leveraging native AWS cloud services and NESDIS product generation algorithms. Developed under the Environmental Satellite EDM and EPG Proving Ground in the AWS Cloud Processing and Distribution System (ESPDS) contract. EDM service provides data storage, a flexible and searchable inventory/catalog of product Cloud Watch metadata, and science data manipulation through RESTful interfaces. Leverages native SNS/SQS Data Transport EDM EPG AWS cloud services including: Elasticsearch, RDS, S3, Lambda, and API Gateway. ESPDS PDA @ DynamoDB ElasticSearch RDS NSOF I&T EPG is capable of generating NESDIS level 1+ sensor, science, EDM SNS SQS RDS EDM Client and tailored product types. Client Leverages native AWS cloud Job Factory services including: EC2 with Auto- S3 Scaling, RDS, SNS, and SQS . NCEI GOES 16 EDM Client Data currently being ingested: EDM Lambdas Computer GOES-16 data from the NOAA/NCEI o Nodes Big Data Project (AWS S3 bucket). S3 EC2 Auto Scale API Gateway S-NPP, JPSS-1, and GCOM-W data o (Orbit-Based) from ESPDS PDA at NSOF I&T. VPC/IAM SNS 3
Solers created a Proving Ground for Enterprise Data Management (EDM) and Enterprise Product Generation (EPG) services in a FedRAMP-approved Amazon Web Services (AWS) cloud environment, leveraging native AWS cloud services and NESDIS product generation algorithms. Developed under the Environmental Satellite EDM and EPG Proving Ground in the AWS Cloud Processing and Distribution System (ESPDS) contract. EDM service provides data storage, a flexible and searchable inventory/catalog of product Cloud Watch metadata, and science data manipulation through RESTful interfaces. Leverages native SNS/SQS Data Transport EDM EPG AWS cloud services including: Elasticsearch, RDS, S3, Lambda, and API Gateway. ESPDS PDA @ DynamoDB ElasticSearch RDS NSOF I&T EPG is capable of generating NESDIS level 1+ sensor, science, EDM SNS SQS RDS EDM Client and tailored product types. Client Leverages native AWS cloud Job Factory services including: EC2 with Auto- S3 Scaling, RDS, SNS, and SQS . NCEI GOES 16 EDM Client Data currently being ingested: EDM Lambdas Computer GOES-16 data from the NOAA/NCEI o Nodes Big Data Project (AWS S3 bucket). S3 EC2 Auto Scale API Gateway S-NPP, JPSS-1, and GCOM-W data o (Orbit-Based) from ESPDS PDA at NSOF I&T. VPC/IAM SNS 4
Solers created a Proving Ground for Enterprise Data Management (EDM) and Enterprise Product Generation (EPG) services in a FedRAMP-approved Amazon Web Services (AWS) cloud environment, leveraging native AWS cloud services and NESDIS product generation algorithms. Developed under the Environmental Satellite EDM and EPG Proving Ground in the AWS Cloud Processing and Distribution System (ESPDS) contract. EDM service provides data storage, a flexible and searchable inventory/catalog of product Cloud Watch metadata, and science data manipulation through RESTful interfaces. Leverages native SNS/SQS Data Transport EDM EPG AWS cloud services including: Elasticsearch, RDS, S3, Lambda, and API Gateway. ESPDS PDA @ DynamoDB ElasticSearch RDS NSOF I&T EPG is capable of generating NESDIS level 1+ sensor, science, EDM SNS SQS RDS EDM Client and tailored product types. Client Leverages native AWS cloud Job Factory services including: EC2 with Auto- S3 Scaling, RDS, SNS, and SQS . NCEI GOES 16 EDM Client Data currently being ingested: EDM Lambdas Computer GOES-16 data from the NOAA/NCEI o Nodes Big Data Project (AWS S3 bucket). S3 EC2 Auto Scale API Gateway S-NPP, JPSS-1, and GCOM-W data o (Orbit-Based) from ESPDS PDA at NSOF I&T. VPC/IAM SNS 5
Solers created a Proving Ground for Enterprise Data Management (EDM) and Enterprise Product Generation (EPG) services in a FedRAMP-approved Amazon Web Services (AWS) cloud environment, leveraging native AWS cloud services and NESDIS product generation algorithms. Developed under the Environmental Satellite EDM and EPG Proving Ground in the AWS Cloud Processing and Distribution System (ESPDS) contract. EDM service provides data storage, a flexible and searchable inventory/catalog of product Cloud Watch metadata, and science data manipulation through RESTful interfaces. Leverages native SNS/SQS Data Transport EDM EPG AWS cloud services including: Elasticsearch, RDS, S3, Lambda, and API Gateway. ESPDS PDA @ DynamoDB ElasticSearch RDS NSOF I&T EPG is capable of generating NESDIS level 1+ sensor, science, EDM SNS SQS RDS EDM Client and tailored product types. Client Leverages native AWS cloud Job Factory services including: EC2 with Auto- S3 Scaling, RDS, SNS, and SQS . NCEI GOES 16 EDM Client Data currently being ingested: EDM Lambdas Computer GOES-16 data from the NOAA/NCEI o Nodes Big Data Project (AWS S3 bucket). S3 EC2 Auto Scale API Gateway S-NPP, JPSS-1, and GCOM-W data o (Orbit-Based) from ESPDS PDA at NSOF I&T. VPC/IAM SNS 6
Primary Objectives: To leverage the flexibility and agility provided by a cloud environment to prototype candidate architectures and implementations for EDM and EPG services, and evaluate them for efficacy, performance, scalability, and maintainability. To demonstrate the flexibility of the proposed EPG service to execute multiple types of algorithms, such as existing ESPDS NDE 2.0 product algorithms, JPSS Risk Reduction algorithms, NESDIS/STAR Enterprise Algorithm implementations of legacy products, and GOES-R L2+ product algorithms. To assess the cost of running these algorithms in a cloud environment. Secondary Objectives: To consider how cloud-hosted EDM and EPG services could be used for collaboration and integration of future product generation algorithms, both within NOAA/NESDIS and with collaborative research organizations. To identify cost breakpoints for technology, ingress & egress, performance, etc. 7
EDM and EPG environments are established in the in the NOAA OCIO FedRAMP-approved AWS Cloud environment Utilizes AWS Cloud services and existing science algorithms o Data feeds from ESPDS PDA at NSOF I&T (GCOM-W, JPSS-1, S-NPP) and NOAA/NCEI Big o Data Project S3 Bucket (GOES-16) Products are being generated from Polar and Geo Missions, including: GCOM-W: AMSR2-L1, GAASP o JPSS-1: Active Fire, JRR(Alpha), NUCAPs, OMPS, Tailoring, True-Color o S-NPP: ACSPO, Active Fire, GVF, JRR, MiRS, NUCAPS, OMPS, OMPS V8 TOS, Tropical o Cyclone, SR, VH, VI, Polar Winds, Tailoring, True-Color GOES-16: GOES-R L2 Products (~ half) via U-Wisconsin CSPP Package, DMW Algorithm o (STAR), DMW BUFR, Tailoring In the process of coordinating with OSPO/STAR for cursory product quality analysis 8
Recommend
More recommend
