Using deep learning and Using deep learning and Google Street View - - PowerPoint PPT Presentation
Using deep learning and Using deep learning and Google Street View to estimate Google Street View to estimate the demographic makeup of the demographic makeup of neighbourhoods across the US neighbourhoods across the US Gebru , T. et al.
Gebru , T. et al. (2017) DOI: 10.1073/pnas.1700035114
1
The United States spends more than $250 million each year on the American Community Survey (ACS), a labor-intensive door-to-doors tudy that measures statistics relating to race, gender, education, occupation, unemployment, and other demographic factors. Although a comprehensive source of data, the lag between demographic changes and their appearance in the ACS can exceed several years. As digital imagery becomes ubiquitous and machine vision techniques improve, automated data analysis may become an increasingly practical supplement to the ACS. Here, we present a method that estimates socioeconomic characteristics of regions spanning 200 US cities by using 50 million images of street scenes gathered with Google Street View cars. Using deep learning-based computer vision techniques, we determined the make, model, and year of all motor vehicles encountered in particular neighborhoods. Data from this census of motor vehicles, which enumerated 22 million automobiles in total (8% of all automobiles in the United States), were used to accurately estimate income, race, education, and voting patterns at the zip code and precinct level. (The average US precinct contains 1,000 people.) The resulting associations are surprisingly simple and powerful. For instance, if the number of sedans encountered during a drive through a city is higher than the number of pickup trucks, the city is likely to vote for a Democrat during the next presidential election (88% chance); otherwise, it is likely to vote Republican (82%). Our results suggest that automated systems for monitoring demographics may effectively complement labor-intensive approaches, with the potential to measure demographics with fine spatial resolution, in close to real time.
2
The United States spends more than $250 million each year on the American Community Survey (ACS), a labor-intensive door-to-doors tudy that measures statistics relating to race, gender, education, occupation, unemployment, and other demographic factors. Although a comprehensive source of data, the lag between demographic changes and their appearance in the ACS can exceed several years. As digital imagery becomes ubiquitous and machine vision techniques improve, automated data analysis may become an increasingly practical supplement to the ACS. Here, we present a method that estimates socioeconomic characteristics of regions spanning 200 US cities by using 50 million images of street scenes gathered with Google Street View cars. Using deep learning-based computer vision techniques, we determined the make, model, and year of all motor vehicles encountered in particular neighborhoods. Data from this census of motor vehicles, which enumerated 22 million automobiles in total (8% of all automobiles in the United States), were used to accurately estimate income, race, education, and voting patterns at the zip code and precinct level. (The average US precinct contains 1,000 people.) The resulting associations are surprisingly simple and powerful. For instance, if the number of sedans encountered during a drive through a city is higher than the number of pickup trucks, the city is likely to vote for a Democrat during the next presidential election (88% chance); otherwise, it is likely to vote Republican (82%). Our results suggest that automated systems for monitoring demographics may effectively complement labor-intensive approaches, with the potential to measure demographics with fine spatial resolution, in close to real time.
3
Demographic statistics for all US cities/counties with population ≥ 65,000 Comprehensive, but demographic changes are typically reported with a lag of several years Expensive and labour-intensive
4
Demographic statistics for all US cities/counties with population ≥ 65,000 Comprehensive, but demographic changes are typically reported with a lag of several years Expensive and labour-intensive
5
6
7
taken from Google Street View Exterior of houses, landscaping, vehicles parked
neural networks (CNN) to classify vehicles Recognise vehicles and determine their characteristics (e.g. make, model etc.)
demographic statistics and socioeconomic attributes
8
50 million Google street view images from 3068 ZIP codes and 39286 voting precincts across 200 cities Product shot images: 2657 visually distinct car categories Amazon Mechanical Turk to gather labeled car images from specific websites (cars.com, craigslist.org etc.) Expert annotation of a subset
9
Car detection Key challenge: balance accuracy and efficiency Deformable Part Model (DPM) as
Detection of 22 million distinct vehicles Car classification CNN trained to distinguish different types of vehicle Classify each vehicle into one of 2657 categories
10
Based on 88 car-related attributes: average #cars per image, average car price, %hybrid cars, %foreign cars, %cars of a specific make, etc. Socioeconomic data obtained from ACS between 2008-2012 2008 election data from another study
Experimental procedure Dataset partitioned based on county into training (A-C; 35/200) and test (D-Z) sets Ridge regression model for income and voter affiliation estimation Logistic regression for race and education prediction Five models trained in each case using fivefold cross-validation to select regularisation parameter (average)
11
Strong associations between vehicle distribution and different demographic factors
12
Strong associations between vehicle distribution and different demographic factors
Toyotas/Hondas → Asian Sedans → Democrat Pickup trucks → Caucausian/Republican Chrysler/Buick → African American
13
Strong correlation between test results and ACS values for all demographic variables at city resolution; 0.54 ≤ r ≤ 0.87 Test results at ZIP code resolution also exhibited close correspondence with ACS values
14
Strong correlation between estimates and actual voting results of 2008 elections at city resolution; r =0.73 Precinct-level estimates also closely matched ground truth data; r =0.57
15
Neighbourhood images can be used to accurately estimate demographic statistics and voter preferences in the US via automated machine learning algorithms. Only data from a few cities are required to provide up-to-date statistics for many cities and ZIP codes. Model could be improved by expanding object recognition incorporate global image features integrating other data types.
16
17