Data-centric Computing for Earth Observation . . Comments and - - PowerPoint PPT Presentation

. . . . . .

Data-centric Computing for EO: General Discussion Current Work Comments and Discussion

. .

Data-centric Computing for Earth Observation

Lizhe Wang

Center for Earth Observation and Digital Earth (CEODE) Chinese Academy of Sciences

June 22nd, 2012

Lizhe Wang Data-centric Computing for Earth Observation

. . . . . .

Data-centric Computing for EO: General Discussion Current Work Comments and Discussion

. . Contents

.

1 Data-centric Computing for EO: General Discussion

.

2 Current Work

Multi-datacenter computing for Earth Observation Data-intensive computing for RS image processing . .

3 Comments and Discussion

Lizhe Wang Data-centric Computing for Earth Observation

. . . . . .

Data-centric Computing for EO: General Discussion Current Work Comments and Discussion

. . About CEODE

.

Satellite Remote Sensing Center

receiving, archiving and processing RS data from satellites (home & abroad)

Airborne Remote Sensing Center

acquisition, processing and storage of airborne RS data

Center for Spatial Data

processing, distribution and archiving of airborne & spaceborne RS data

Laboratory of Digital Earth Sciences

research on geo-spatial information science, remote sensing, and scientific platform for Digital Earth

. . . . . . . .

Lizhe Wang Data-centric Computing for Earth Observation

. . . . . .

Data-centric Computing for EO: General Discussion Current Work Comments and Discussion

. . Remote sensing methodology: data-centric system

. . Remote sensing

use of aerial sensor technologies to detect objects on Earth by means of propagated signals (e.g. electromagnetic radiation emitted from aircraft or satellites)

Data flow in remote sensing engineering

acquisition → transmission → processing → archiving → distribution → visualization

. .

Lizhe Wang Data-centric Computing for Earth Observation

. . . . . .

Data-centric Computing for EO: General Discussion Current Work Comments and Discussion

. . How big?

. . . Sources Data generation

• No. Sources

High-resolution satellite 10TB/Day 100 Airborne RS 5TB/flight 10,000 Ground sensors 1MB/day 106 . .

EO Resolution EO Platform Pixels/km2 Data Set 1000m Meteorological satellite 1 0.5GB 100m EO-1 102 50GB 10m SPOT 104 5TB 1m Quickbird 106 0.5PB 0.1m GeoEye 108 50PB

Lizhe Wang Data-centric Computing for Earth Observation

. . . . . .

Data-centric Computing for EO: General Discussion Current Work Comments and Discussion

. . How intensive?

. . .

Processing Process No. Data size Throughput (MB/S) NDVI 32 305 (MB) 896.9981 Radiometric correction 80 700 (MB) 80.6173 Geometric correction 80 700 (MB) 36.08247 Gridding of AIRS Data 5 500 (GB) 1.36 AWI from MODIS Data 1 500 (GB) 2.5

. .

Lizhe Wang Data-centric Computing for Earth Observation

. . . . . .

Data-centric Computing for EO: General Discussion Current Work Comments and Discussion

. . How complex?

. .

Multiple sources: various sensors, various resolutions, various regions Multiple dimensions: time, space, multiple spectrum, geographical information, social information Highly unstructured with various metadata information Complex processing algorithms: pixel based, region based, and global based Parameter assessment and model development

. .

Lizhe Wang Data-centric Computing for Earth Observation

. . . . . .

Data-centric Computing for EO: General Discussion Current Work Comments and Discussion Multi-datacenter computing for Earth Observation Data-intensive computing for RS image processing

. . Current work

Multi-datacenter computing for Earth Observation Data-intensive computing for RS image processing

Lizhe Wang Data-centric Computing for Earth Observation

. . . . . .

Data-centric Computing for EO: General Discussion Current Work Comments and Discussion Multi-datacenter computing for Earth Observation Data-intensive computing for RS image processing

. . Project background and objective

Integrated earth observation (satellite, airborne and ground-borne) and quantitative RS production system Sub-project: integrated EO from multiple satellite datacenters and common-featured RS production system Objective

Production system: industry standards, event driven, workflow management, service oriented 6 satellite data centers PB-level data processing: data fusion, production, sharing and publish 40 types of comment-featured remote sensing products: time continues and space continues

Lizhe Wang Data-centric Computing for Earth Observation

. . . . . .

Data-centric Computing for EO: General Discussion Current Work Comments and Discussion Multi-datacenter computing for Earth Observation Data-intensive computing for RS image processing

. . Software architecture

Product search Product download Product publish Product

• rder

Metadata aggregation Data scheduling/query/search Algorithm management/query/publish Order management Workflow management Metadata management Data movement/storage/download Monitoring & security Task scheduling Functionalities Multi-datacenter level Datacenter level Algorithm deployment & task execution Lizhe Wang Data-centric Computing for Earth Observation

. . . . . .

Data-centric Computing for EO: General Discussion Current Work Comments and Discussion Multi-datacenter computing for Earth Observation Data-intensive computing for RS image processing

. . Key challenges

Metadata management: extraction, aggregation, query, publish Data management: move, index, search, download Algorithm/executables management: standardized development, metadata management, automatic deployment Production system

Lizhe Wang Data-centric Computing for Earth Observation

. . . . . .

Data-centric Computing for EO: General Discussion Current Work Comments and Discussion Multi-datacenter computing for Earth Observation Data-intensive computing for RS image processing

. . Project background and objective

Research project funded by Chinese Academy of Sciences: 2011 – 2014 Collaboration with German DFG project Engineering project: Parallel Image Process System (PIPS) Objective: high-performance data-intensive RS image processing

RS-GPPS: Generic Parallel Programming Skeletons from Remote Sensing Optimized parallel file system Rumtime framework

Lizhe Wang Data-centric Computing for Earth Observation

. . . . . .

Data-centric Computing for EO: General Discussion Current Work Comments and Discussion Multi-datacenter computing for Earth Observation Data-intensive computing for RS image processing

. . RS image processing algorithms

Lizhe Wang Data-centric Computing for Earth Observation

. . . . . .

Data-centric Computing for EO: General Discussion Current Work Comments and Discussion Multi-datacenter computing for Earth Observation Data-intensive computing for RS image processing

. . Parallel programming with RS-GPPS

Lizhe Wang Data-centric Computing for Earth Observation

. . . . . .

Data-centric Computing for EO: General Discussion Current Work Comments and Discussion Multi-datacenter computing for Earth Observation Data-intensive computing for RS image processing

. . The Architecture of RS-GPPS

Lizhe Wang Data-centric Computing for Earth Observation

. . . . . .

Data-centric Computing for EO: General Discussion Current Work Comments and Discussion

. . Experiences and comments for RS applications

Data set:

10 GB: 1 RS image 10-100 TB: a global EO problem Big? It depends! Intensive? Yes!

Programming:

Fine-grained parallelism: MPI/OpenMP/PGAS Hadoop/HDFS: No Workflow, dataflow: Yes

Database:

RDBMS: Yes NoSQL: No

Lizhe Wang Data-centric Computing for Earth Observation

. . . . . .

Data-centric Computing for EO: General Discussion Current Work Comments and Discussion

. . Discussion

Current problem:

Performance Runtime system

Road map:

Shared memory vs. distributed memory Fine-grained vs. coarse-grained Programming model

Lizhe Wang Data-centric Computing for Earth Observation

. . . . . .

Data-centric Computing for EO: General Discussion Current Work Comments and Discussion

Thank you! Happy Duanwu Festival! Contact: Lizhe.Wang@gmail.com

Lizhe Wang Data-centric Computing for Earth Observation