DeepSAT: A Deep Learning Framework for Satellite Image - - PowerPoint PPT Presentation

National Aeronautics and Space Administration www.nasa.gov

DeepSAT: A Deep Learning Framework for Satellite Image Classification

Sangram Ganguly

NASA Ames Research Center/ Bay Area Environmental Research Institute Contributions from: Gayaka Shreekant, Subodh Kalia, Ramakrishna Nemani, Andrew Michaelis, Thomas Vandal. Supratik Mukhopadhyay

DeepSAT

2

NASA EARTH EXCHANGE (NEX).

+ NEX is virtual collaborative that brings scientists together in a knowledge-based social network and provides the necessary tools, computing power, and access to bigdata to accelerate research, innovation and provide transparency. OVERVIEW To provide “science as a service” to the Earth science community addressing global environmental challenges

VISION

To improve efficiency and expand the scope of NASA Earth science technology, research and applications programs

GOAL

NEX Provides a Complete Work Environment –

“Science As A Service”

COLLABORATION

Over 400 Members

CENTRALIZED DATA REPOSITORY

Over 2.3 PB of Data & growing..

COMPUTING Scalable Diverse Secure/Reliable KNOWLEDGE Workflows Machine Images Model codes Re-useable software

DeepSAT

DeepSAT

4

Science @ NEX

Global Vegetation Biomass at 100m resolution by blending data from 4 different satellites High resolution climate projections for climate impact studies High resolution monthly global data for monitoring forests, crops and water resources Mapping fallowed area in California during drought Machine learning and Data mining  moving towards more data-driven approaches

DeepSAT

5

.

Models

Physics based

Data Volume Monitoring

Models

Machine Learning

Models

Physics based

DeepSAT

6

NEX AI Lab.

DeepSAT

7

Multiple Conditions Shadows/no-shadows, BRDF effects, cloud cover, aerosols, mixed-pixel effects, presence/absence of atmospheric correction, view angle effects, sensor altitude, etc. Multiple Classes Roads, impervious, grasses/shrubs, tree, water, rooftops, etc. Multiple Sensors Worldview, Ikonos, high-res airborne, Landsat, Sentinel-1/2 etc.

High Resolution Satellite/Airborne Image Classification.

Multiple Applications Tree cover, carbon sequestration, water extent mapping, solar efficiency, road network monitoring, construction monitoring, habitat monitoring, climate modeling, etc.

NASA Carbon Monitoring System (CMS)- funded activity

DeepSAT

8

High Resolution Tree Cover Classification.

Tree cover delineation is a hard problem Quality of data affected by data acquisition, pre-processing and filtering. Significant inter-class overlaps and

• ften hard to distinguish between

classes. Accuracy of present algorithms is low and there is a pressing need to create high resolution land cover maps. Need to harness strong discriminative features and efficient learning algorithm.

DeepSAT

9

Lots of big images!

Landsat Thematic Mapper 1984-2012 Monthly composites of Biophysical products such as LAI Focus on: Land cover changes Migration of ecosystems High altitude ecosystems Forest mortality

330,000 NAIP Scenes 65 Terabytes of Images 7000x7000 Image Matrix Big Data Need for Big Computation Images fed in parallel to cores in HPC

Current End-to-end Processing Time (California with 11,000 scenes) -> 48 hours Total Wall time for processing Continental U.S. -> 2.64 million hours Total size of Feature Vectors Extracted  2.8 PetaBytes

One Epoch (2010 – 2012)

DeepSAT

10

NAIP Processing Architecture.

NEX

NASA Earth Exchange Storage

HPC M1

HPC Module 1

HPC M2

HPC Module 2

HPC M3

HPC Module 3 NASA Earth Exchange High Performance Computing (HPC) NEX HPC NEX HPC

NEX NEX HPC M3 HPC M1 HPC M2

INPUT IMAGE VOTING UPDATE TRAINING DATASET OUTPUT IMAGE

DeepSAT

11

NAIP Processing Architecture on AWS.

• Configure a base set of

AWS services to build the processing pipeline

• Process ~15,000 Scenes
• ~5000 x 5000 pixels /

scene

• Leveraged Spot Instances
• 70% savings
• Managed services
• Spinup, process,

tear down in 1 week.

• More that just computing…

DeepSAT

12

Learning Module – Training Phase.

Training data EXTRACT FEATURE VECTORS CCM DCT NDVI EVI INITIALIZE WEIGHTS OF NEURAL NETWORK USING DEEP BELIEF NETWORK TRAINING CLASS LABELS TAKE SUB-SAMPLE OF THE FEATURE VECTORS APPEND CLASS LABEL AND FEED TO ANN TRAIN ANN WITH BACKPROPAGATION AND STOCHASTIC GRADIENT DECENT Trained Neural Network EACH LAYER IN DBN IS A RBM AND TRAINED USING CONTRASTIVE DIVERGENCE WITH REPEATED GIBBS SAMPLING

From Unsupervised Pre-training to Supervised Learning

Learning Module - Testing/ Prediction Phase.

NAIP Tile EXTRACT FEATURE VECTORS CCM DCT NDVI EVI Trained Neural Network CLASS MASK PREDICT CLASS AND GENERATE LABELS NORMALIZE DATA AND FEED TO ANN Basu, S.; Ganguly, S.; Nemani, R.R.; Mukhopadhyay, et al., A Semiautomated Probabilistic Framework for Tree-Cover Delineation From 1-m NAIP Imagery Using a High-Performance Computing Architecture, in Geoscience and Remote Sensing, IEEE Transactions on , vol.53, no.10, pp.5690- 5708, Oct. 2015 doi: 10.1109/TGRS.2015.2428197.

DeepSAT

Experimental Results.

Densely Forested Fragmented forests Urban areas Overall Total samples 12000 12000 12000 36000 Tree samples 6000 6000 6000 18000 Non-tree samples 6000 6000 6000 18000 True Positive Rate (%) 85.87 88.26 73.65 82.59 False positive Rate (%) 2.21 0.99 1.98 1.73

Total scenes processed = 11095 for the whole of California

DeepSAT

California Tree Cover Mosaic.

San Francisco Bay Area

DeepSAT

DeepSAT

DeepSAT A Learning Framework for Satellite Imagery

Feature- enhanced DBN CNN Stacked Autoencoder MODELS OUR DATA SAT-4 SAT-6

500,000 Image Patches 4 Land Cover Types (Barren, Tree, Grass, All Other) 405,000 Image Patches 6 Land Cover Types (Barren, Tree, Grass, Road, Building, Water Bodies)

RESULTS

SAT4 Classifier Accuracy: 97.946 SAT4 Classifier Accuracy: 86.827 SAT4 Classifier Accuracy: 79.978 SAT6 Classifier Accuracy: 93.916 SAT6 Classifier Accuracy: 79.063 SAT6 Classifier Accuracy: 78.430

Saikat Basu, Sangram Ganguly, Supratik Mukhopadhyay, Robert Dibiano, Manohar Karki and Ramakrishna Nemani, DeepSat - A Learning framework for Satellite Imagery, ACM SIGSPATIAL 2015 Saikat Basu, Manohar Karki, Sangram Ganguly, Robert DiBiano, Supratik Mukhopadhyay, Ramakrishna Nemani, Learning Sparse Feature Representations using Probabilistic Quadtrees and Deep Belief Nets, European Symposium on Artificial Neural Networks, ESANN 2015

CNN: Convolutional Neural Network

SATNet – The satellite imagery training database & model zoo.

Model Zoo (pre-trained models for different satellites e.g. Landsat, Sentinel-1/2, Worldview, etc. Inspired by Imagenet, we are building a huge database of labeled satellite/aerial imagery dataset Labeled data generated by experts using a GUI interface. Covers different landcover classes – trees, barren lands, shrubs, rooftops, water bodies and much more. The goal is to create a dataset with 5,000,000 labeled samples.

Current State of Art – more deep architectures (SegNet).

DeepSAT

Original NAIP data (RGB)

Forest mask

NAIP forest scene

Training Data collection

Training data generation.

DeepSAT

17 US states – AL, AR, AZ, CO, CT, DE, FL, GA, IA, ID, IL, IN, KS, KY, LA, MA, MD

Selection of 10 random NAIP forest scenes for each US state Dividing each scene into 200 patches (dimension: 600 x 600) Random selection of 60 patches Forest mask generation from each patch Total number of forest mask generated = 1020 Phase – 1 Phase – 2

8 US states – CA, ME, MI, MN, MO, MS, MT, NC (Under progress)

Sample training data – forest/ non-forest.

DeepSAT

Roadmap.

DeepSAT

Phase II (started & almost in completion):

• Generate training data – 60 patches of 600 x 600 for 50 states
• Established baseline land cover classification accuracy using CNN based segmentation

architecture such as SegNet and FCN

• Scale results to all 50 states ~ 330,000 tiles of 7000 x 8000 pixels

Phase II (in progress):

• Extend SegNet & FCN architecture to fuse information from multiple bands and other data

Phase III:

• Develop techniques to combine large amount of unlabeled data with supervised techniques

to further improve classification (e.g. model learnt from DBN as a replacement to the fully connected layer in CNN)

SegNet Sample Result.

DeepSAT

Input image Ground truth 100,000 iterations 6000x6000 NAIP Tile 6000x6000 Prediction

THE WHOLE TILE Statis istic ics Ove vera rall ll Total samples 14,565 True Positive Rate (%) 83.29 False Positive Rate (%) 4.61 Accuracy (%) 92.87 Time to predict a tile ~2 minutes

Super Resolution CNNs to Downscale Climate Models.

DeepSAT

Global climate models (GCMs) exist at low resolutions (> 100km) but climate change effects are local Downscaling Problem: Learn a mapping from low resolution GCMs precipitation (and high resolution topography) to high resolution observed precipitation. Proposed Method: Apply a Super-Resolution CNN [Dong 2014] to downscaling

1. Dong, Chao, et al. "Learning a deep convolutional network for image super-resolution." European Conference on Computer

• Vision. Springer International Publishing, 2014.

SRCNN Architecture.

DeepSAT

SRCNN Training.

DeepSAT

Data:

• Precipitation from Prism 4km

upscaled to 16km and 50km interpolated to 16km grid

• Training years: 1981-2000
• Testing years: 2001-2015

Mapping: 50km -> 16km “Sub-images”:

• Crop 51x51 patches with stride 30
• Count > 10 million in training set

Computing: Tensorflow + 4 GPUs from the Nvidia Devbox

Figure: https://www.tensorflow.org/versions/r0.11/tutorials/deep_cnn/index.html

Daily RMSE over the Continental United States .

DeepSAT

RMSE: BCSD = 2.23, SRCNN = 1.57

* BCSD is a widely used statistical downscaling method for global climate models

Massively Leveraging Nvidia .

DeepSAT

All our CNN models are trained on Nvidia devbox cluster Pleiades GPU cluster is used to train models with multiple sets of train/test datasets – both DBN and CNN architectures DGX-1 like systems are important !

• Model size: It has been recently shown deep neural networks – such as 152 layer ResNet architecture outperforms

shallower networks such as VGG-16. The increase in layers leads to a tremendous increase in number of parameters. Hence, the mathematical operations needed such as gradient and nonlinear function of inputs, also increase. These networks require ~15 billion FLOPS and this number will keep increasing as the networks grow in size and complexity.

• The model complexity as well as input data size is limited by the GPU memory. With 128 GB vram, more complicated

models can be used for experimentation. Increasing input size has also many desirable effects e.g., the gradient descent will have less noise, in the context of CNN, bigger images could provide more context for classification and hence, improve classification/segmentation accuracy.

• Training time: As training time is reduced due to P100 improvements over last generation (Maxwell architecture), as well

significant increase in number of cores, the training time will be much shorter. Faster training implies more experimentation and faster innovation. Pascal architecture has been shown to perform 5-7x faster than the last generation using some Deep Learning benchmarks.

Going Forward.

DeepSAT

• NEX-AI’s core is to focus on blending physical models with state-of-art machine

learning frameworks to address NASA’s mission objectives

• NEX-AI currently has focus on a number of problems related to satellite image

classification, climate downscaling and large scale anomaly detection

• DeepSAT will provide the current modeling frameworks along with access to

training data for NEX users

• NEX-AI will collaborate with industry leading experts in testing newer

frameworks in AI and in defining hard problems in the land-climate-atmosphere continuum that can be “possibly” be solved by clever ensemble learning models

Contact Email: sangram.ganguly@nasa.gov sangramganguly@gmail.com