DeepSAT: A Deep Learning Framework for Satellite Image - PowerPoint PPT Presentation

National Aeronautics and Space Administration DeepSAT: A Deep Learning Framework for Satellite Image Classification Sangram Ganguly NASA Ames Research Center/ Bay Area Environmental Research Institute Contributions from: Gayaka Shreekant, Subodh Kalia, Ramakrishna Nemani, Andrew Michaelis, Thomas Vandal. Supratik Mukhopadhyay www.nasa.gov

NASA EARTH EXCHANGE (NEX) . OVERVIEW + NEX is virtual collaborative that brings scientists together in a knowledge-based social network and provides the necessary tools, computing power, and access to bigdata to accelerate research, innovation and provide transparency. VISION To provide “ science as a service ” to the Earth science community addressing global environmental challenges GOAL To improve efficiency and expand the scope of NASA Earth science technology, research and applications programs DeepSAT 2

NEX Provides a Complete Work Environment – “Science As A Service” CENTRALIZED DATA REPOSITORY COLLABORATION Over 400 Members Over 2.3 PB of Data & growing.. COMPUTING KNOWLEDGE Scalable Workflows Diverse Machine Images Secure/Reliable Model codes Re-useable software DeepSAT

Science @ NEX Global Vegetation Biomass at 100m resolution High resolution climate projections High resolution monthly global data for monitoring by blending data from 4 different satellites for climate impact studies forests, crops and water resources Mapping fallowed area in California Machine learning and Data mining  moving towards more data-driven approaches during drought DeepSAT 4

. Models Machine Learning Monitoring Models Physics Models based Physics based Data Volume DeepSAT 5

NEX AI Lab. DeepSAT 6

High Resolution Satellite/Airborne Image Classification . Multiple Sensors Multiple Classes Worldview, Ikonos, high-res airborne, Roads, impervious, grasses/shrubs, Landsat, Sentinel-1/2 etc. tree, water, rooftops, etc. Multiple Conditions Multiple Applications Shadows/no-shadows, BRDF effects, Tree cover, carbon sequestration, cloud cover, aerosols, mixed-pixel water extent mapping, solar efficiency, effects, presence/absence of road network monitoring, construction atmospheric correction, view angle monitoring, habitat monitoring, climate effects, sensor altitude, etc. modeling, etc. NASA Carbon Monitoring System (CMS)- funded activity DeepSAT 7

High Resolution Tree Cover Classification . Significant inter-class overlaps and Quality of data affected by data acquisition, often hard to distinguish between pre-processing and filtering. classes. Tree cover delineation is a hard problem Need to harness strong Accuracy of present algorithms is low and there discriminative features and is a pressing need to create high resolution land efficient learning algorithm. cover maps. DeepSAT 8

Lots of big images! Landsat Thematic Mapper 330,000 Need for Big NAIP 1984-2012 Computation Scenes Monthly composites of 65 Terabytes of Big Data Biophysical products such Images as LAI 7000x7000 One Epoch (2010 – 2012) Image Images fed in Matrix Focus on: parallel to cores in HPC Land cover changes Migration of ecosystems High altitude Current End-to-end Processing Time (California with 11,000 scenes) -> 48 hours ecosystems Forest mortality Total Wall time for processing Continental U.S. -> 2.64 million hours Total size of Feature Vectors Extracted  2.8 PetaBytes DeepSAT 9

NAIP Processing Architecture . NEX NEX HPC UPDATE TRAINING DATASET HPC M3 OUTPUT VOTING IMAGE INPUT IMAGE HPC M2 HPC M1 NEX HPC Module 1 HPC M1 NASA Earth Exchange NEX Storage HPC Module 2 HPC M2 NASA Earth Exchange High Performance Computing (HPC) NEX HPC HPC Module 3 HPC M3 DeepSAT 10

NAIP Processing Architecture on AWS . • Configure a base set of AWS services to build the processing pipeline • Process ~15,000 Scenes • ~5000 x 5000 pixels / scene • Leveraged Spot Instances • 70% savings • Managed services • Spinup, process, tear down in 1 week. • More that just computing… DeepSAT 11

Learning Module – Training Phase . From Unsupervised Pre-training to Supervised Learning TRAINING CLASS LABELS APPEND CLASS TAKE SUB-SAMPLE OF THE FEATURE VECTORS LABEL AND FEED TO ANN CCM INITIALIZE WEIGHTS DCT OF NEURAL EXTRACT NDVI NETWORK USING FEATURE EVI DEEP BELIEF VECTORS TRAIN ANN WITH NETWORK BACKPROPAGATION AND Training data STOCHASTIC GRADIENT DECENT EACH LAYER IN DBN IS A RBM AND TRAINED USING CONTRASTIVE DIVERGENCE WITH REPEATED GIBBS SAMPLING Trained Neural Network DeepSAT 12

Learning Module - Testing/ Prediction Phase . CCM DCT PREDICT CLASS EXTRACT NORMALIZE NDVI FEATURE DATA AND EVI AND GENERATE VECTORS FEED TO ANN LABELS CLASS MASK Trained Neural Network NAIP Tile Basu, S.; Ganguly, S.; Nemani, R.R.; Mukhopadhyay, et al., A Semiautomated Probabilistic Framework for Tree-Cover Delineation From 1-m NAIP Imagery Using a High-Performance Computing Architecture, in Geoscience and Remote Sensing, IEEE Transactions on , vol.53, no.10, pp.5690- 5708, Oct. 2015 doi: 10.1109/TGRS.2015.2428197. DeepSAT

Experimental Results . Total scenes processed = 11095 for the whole of California Densely Fragmented Urban areas Overall Forested forests Total samples 12000 12000 12000 36000 Tree samples 6000 6000 6000 18000 Non-tree 6000 6000 6000 18000 samples True Positive 85.87 88.26 73.65 82.59 Rate (%) False positive 2.21 0.99 1.98 1.73 Rate (%) DeepSAT

California Tree Cover Mosaic . San Francisco Bay Area DeepSAT

DeepSAT A Learning Framework for Satellite Imagery Saikat Basu, Sangram Ganguly, Supratik Mukhopadhyay, Robert Dibiano, Manohar Karki and Ramakrishna Nemani, DeepSat - A Learning framework for Satellite Imagery, ACM SIGSPATIAL 2015 Saikat Basu, Manohar Karki, Sangram Ganguly, Robert DiBiano, Supratik Mukhopadhyay, Ramakrishna Nemani, Learning Sparse Feature Representations using Probabilistic Quadtrees and Deep Belief Nets, European Symposium on Artificial Neural Networks, ESANN 2015 SAT-4 SAT-6 OUR DATA 405,000 Image Patches 500,000 Image Patches 6 Land Cover Types 4 Land Cover Types (Barren, Tree, Grass, Road, (Barren, Tree, Grass, All Other) Building, Water Bodies) Feature- Stacked enhanced CNN Autoencoder MODELS DBN SAT4 Classifier Accuracy: SAT4 Classifier Accuracy: SAT4 Classifier Accuracy: RESULTS 97.946 79.978 86.827 SAT6 Classifier Accuracy: SAT6 Classifier Accuracy: SAT6 Classifier Accuracy: 93.916 78.430 79.063 CNN: Convolutional Neural Network DeepSAT

SATNet – The satellite imagery training database & model zoo . Inspired by Imagenet, we are building a huge database of labeled satellite/aerial imagery dataset Model Zoo (pre-trained models for Labeled data generated by experts using a GUI interface. different satellites e.g. Landsat, Sentinel-1/2, Covers different landcover classes – trees, barren lands, shrubs, rooftops, water bodies and much Worldview, etc. more. The goal is to create a dataset with 5,000,000 labeled samples.

Current State of Art – more deep architectures (SegNet) . Forest mask NAIP forest scene Original NAIP data (RGB) Training Data collection DeepSAT

Training data generation . Phase – 1 17 US states – AL, AR, AZ, CO, CT, DE, FL, GA, IA, ID, IL, IN, KS, KY, LA, MA, MD Selection of 10 random NAIP forest scenes for each US state Dividing each scene into 200 patches (dimension: 600 x 600) Random selection of 60 patches Forest mask generation from each patch Total number of forest mask generated = 1020 Phase – 2 8 US states – CA, ME, MI, MN, MO, MS, MT, NC (Under progress) DeepSAT

Sample training data – forest/ non-forest . DeepSAT

Roadmap. Phase II (started & almost in completion): • Generate training data – 60 patches of 600 x 600 for 50 states • Established baseline land cover classification accuracy using CNN based segmentation architecture such as SegNet and FCN • Scale results to all 50 states ~ 330,000 tiles of 7000 x 8000 pixels Phase II (in progress): • Extend SegNet & FCN architecture to fuse information from multiple bands and other data Phase III: • Develop techniques to combine large amount of unlabeled data with supervised techniques to further improve classification (e.g. model learnt from DBN as a replacement to the fully connected layer in CNN) DeepSAT

SegNet Sample Result . THE WHOLE TILE 6000x6000 NAIP Tile Input image Ground truth 100,000 iterations Statis istic ics Ove vera rall ll Total samples 14,565 True Positive Rate (%) 83.29 6000x6000 Prediction False Positive Rate (%) 4.61 Accuracy (%) 92.87 Time to predict a tile ~2 minutes DeepSAT

Super Resolution CNNs to Downscale Climate Models . Global climate models (GCMs) exist at low resolutions (> 100km) but climate change effects are local Downscaling Problem: Learn a mapping from low resolution GCMs precipitation (and high resolution topography) to high resolution observed precipitation. Proposed Method: Apply a Super-Resolution CNN [Dong 2014] to downscaling 1. Dong, Chao, et al. "Learning a deep convolutional network for image super-resolution." European Conference on Computer Vision . Springer International Publishing, 2014. DeepSAT

SRCNN Architecture . DeepSAT