Todd Stavish, In-Q-Tel CosmiQ Works SpaceNet Overview Inspiration - - PowerPoint PPT Presentation

GTC 2017 Todd Bacastow, DigitalGlobe Radiant Todd Stavish, In-Q-Tel CosmiQ Works

SpaceNet Overview

Inspiration Components Datasets Competitions

Inspired by ImageNet

1. Datasets Publicly available satellite imagery & labeled data 2. Competition Public challenges against remote sensing problems

• 1st Release (8/16)

50cm 8-band over Rio de Janeiro

• 2nd Release (1/17)

Points of Interest (POI)

• ver Rio
• 3rd Release (2/17)

30cm 8-band over Las Vegas, Paris, Shanghai & Khartoum

• 1st Competition

Completed 12/16

• 2nd Competition

Launched on 3/20

Source: https://cdn.pixabay.com/photo/2016/04/10/19/20/colored-pencils-1320548_1280.jpg

Source: DigitalGlobe, Inc.

Source: https://commons.wikimedia.org/wiki/File:Openstreetmap_logo.svg

The data management challenge

10

11

Home broadband: 300 years DirectConnect: 6-18 months ($$$) X 1,400 How do we get 100 PB into the cloud?

… or a bigger “snowball”

12

– a Snowmobile

13

14

SpaceNet on AWS is an open repository of 5,700+ km2 of satellite imagery and 520,000+ vectors made available to developers to enable geospatial machine learning.

Rio de Janeiro Buildings

Released August 2016 Imagery: 50 cm WV-2 mosaic and 8-band MSI covering 1900 km2 Building Footprints: 220,594 covering 252 km2

Las Vegas, Paris, Khartoum, and Shanghai Buildings

Released February 2017 Imagery: 30 cm WV-3 image strips and 8-band MSI covering 3,880 km² Building Footprints: 221,376

Rio de Janeiro Points of Interest (POIs)

Released January 2017 Imagery: 50 cm WV-2 mosaic POIs: 120,155 individual POIs from 460 feature classes Released with NGA support

SpaceNet Datasets

15

Rio Public Data Set

13 14

Rio de Janeiro, Brazil

Imagery: 50cm WV-2 mosaic + 8-band MSI covering 1900 km2 Building Footprints: 220,594 covering a 252 km2 AOI https://aws.amazon.com/public-data-sets/spacenet/

Rio Points of Interest Dataset

Utilities Transportation Public Facilities

POI Datast Includes

• 12 datasets with 35 unique layers containing more

than 120,000 individual points of interest

• Subset of 11,114 points across 139 features that

have been identified as discernable in the provided satellite imagery

• Released in GIS (geodatabase) and machine

learning friendly formats (parsable JSON)

• Provides quality estimation attributes (e.g.

confirmation and resolution)

• Introduces the concept of an object hierarchy akin

to ImageNet’s use of WordNet (e.g. infrastructure- >buildings->apartments)

Rio POI Dataset

18

Las Vegas Paris Shanghai Khartoum

Newly Released Public Data Sets

270 km2 1,560 km2 1,170 km2 800 km2 109,807 Footprints 16,663 Footprints 69,433 Footprints 25,463 Footprints 69GB Raster Data 402GB Raster Data 302GB Raster Data 373GB Raster Data

Imagery: 30cm WV-3 single strip images + 8-band MSI Total Building Footprints: 221,376 covering a 3,880 km2 AOI across for 4 additional cities: Las Vegas, Paris, Shanghai, and Khartoum

SpaceNet | March 2017

Lowering the Barrier of Entry for SpaceNet

• SpaceNet contains a massive amount of labeled data in GeoJSON files, an

unfamiliar format for most data scientists.

• We released code to transform these labels into a multitude of other formats

(NumPy arrays, image masks, etc.) more conducive to machine learning.

19

Imagery Courtesy of DigitalGlobe

Imagery Courtesy of DigitalGlobe Imagery Courtesy of DigitalGlobe

* Naïve approach yields F1=0.57

20

crowdsourcing.topcoder.com/spacenet

Automated Mapping Challenge - Round 1

• Nov. – Dec. 2016

Rio de Janeiro Building extraction $35,000 in prizes

High Revisit Activity Detection Challenge

Mid-2017 Imagery will show places with economic indicators and focus

• n activity-based analytics

Automated Mapping Challenge - Round 2

March – May 2017 Las Vegas, Paris, Khartoum, Shanghai Building extraction w/ 2x performance $15,500 in prizes

The SpaceNet Challenge is a series of coding competitions with cash prizes that make use of SpaceNet on AWS datasets to accelerate geospatial machine learning.

SpaceNet Challenges

SpaceNet Challenge Metric and Scoring

• Metric was an IoU comparison with a threshold
• IoU(A,B) = area(A intersection B) / area(A union B)
• Top public leaderboard F1 score was 0.255
• precision = TP / (TP + FP)
• recall = TP / (TP + FN)
• F1= 2 * precision * recall / (precision + recall)

22

Source: Walber (Own work) [CC BY-SA 4.0 (http://creativecommons.org/licenses/by-sa/4.0), via Wikimedia Commons. https://commons.wikimedia.org/wiki/File%3APrecisionrecall.svg

23

SpaceNet Challenge - Round 1

The relatively low F1 scores of the winning submissions indicate that automated building footprint extraction remains a challenging problem that warrants further research

Competitors 42 competitors worldwide Submissions 242 submissions Winning Result F1 Score of 0.255 from Brazil International Top 5 submissions were international Challenge Competition focused on automated feature extraction Evaluation Results were evaluated with scientifically grounded metrics (F1 Score) Cash Prizes $35,000 in prizes were paid to the top performing teams

10/24 10/31 11/7 11/14 12/8 12/22

Pre- Registration Training Data + Visualizer Released Google OnAir Hangout w/ SpaceNet Experts Match Began (3-Week Competition) Competition Ends Winners Announced

Competition Timeline

SpaceNet Challenge – Round 1: Winning Solution

• The winning implementation was

developed by a Brazilian Topcoder

• Implementation was custom and used

random forests with brute force polygon search

• Results of the first challenge were

promising given limited time and use of an early training dataset

• More information

CosmiQ Works blog “SpaceNet: Winning Implementations and New Imagery Release” Summary of approach:

1. Classify pixels into 3 categories: border, inside a building, and other. 2. Based on individual pixel classification, generate candidate polygons that may contain buildings 3. Evaluate polygon candidates to select those with a confidence above a given threshold; discard remaining polygons

24

Round 1 Winning Solution

26

SpaceNet Challenge - Round 2

Cash Prizes

Up to $15,500 in prizes to be paid to the top performing teams

Challenge

Competition on footprint extraction over four diverse cities

Evaluation

Highest F1 per city and averaged across all cities

2/17 4/1 5/23 5/31

Training Data Released

3/20

Match Began (9-Week Competition) Early Incentive Awarded Competition Ends Winners Announced

Competition Timeline (Estimated)

SpaceNet Challenge Round 2 Early Results

• F-score: ~0.6, average of all

four cities

• Improvements in imagery

resolution and vector labels

• Higher F-scores in Round 2

initially seems to be directly related to better training data - imagery and labels

• 1. Utilize SpaceNet on AWS data for research
• Use the data to train models for research or commercial uses
• Publish open source code, blog posts, and research papers
• 2. Participate in current/future SpaceNet Challenges
• SpaceNet Challenge Round 2 is live
• Tell your friends
• 3. Contribute/sponsor future open data releases
• Looking for new participants to contribute to the release of

additional data sets

• The data must have an ‘open’ license and come ‘prepared’

28

How to Get Involved

Thank You