Faades of Interest in Street View Panoramic Sequences Andr A. - - PowerPoint PPT Presentation

This work was sponsored by

Accurate Location of Façades of Interest in Street View Panoramic Sequences

André A. Araújo, Jonas C. Sampaio, Raphael S. Evangelista, Altobelli B. Mantuan, Leandro A. F. Fernandes

{andrealvarado,revangelista}@id.uff.br, {jsampaio,amantuan,laffernandes}@ic.uff.br

What are those places?

SIBGRAPI 2015 2

Overview

• A real example

3

Query image

Actual user’s location Assigned user’s location Selected vertex Estimated building’s location Visited vertex Candidate vertex

Contributions

SIBGRAPI 2015 4

• An algorithm to calculate the location of buildings in

a completely automatic way

• A strategy for computing gnomonic projections

restricted to the sidewalks

• A heuristic to select which are the best Street View

environments facing the building of interest

• A derivative of how to estimate the uncertainty

associated with the computed locations

Outline

• Proposed approach
• Results
• Concluding remarks

SIBGRAPI 2015 5

Pipeline

SIBGRAPI 2015 6

Breadth-First Search Select Best Panoramic Views Triangulate the Target Building Estimate the Uncertainty Query Image + Inaccurate GPS Coordinates Building Location + Uncertainty

Breadth-First Search

The First Stage

SIBGRAPI 2015 7 Actual user’s location Assigned user’s location Selected vertex Estimated building’s location Visited vertex Candidate vertex

Stopping criteria: ≥ 𝑙; or 𝑒𝑗𝑡𝑢( , ) ≥ 𝑠

𝑛𝑏𝑦

Query image

Breadth-First Search

Image Rectification

SIBGRAPI 2015 8

• Equirectangular projection (native format)

Actual user’s location Assigned user’s location Selected vertex Estimated building’s location Visited vertex Candidate vertex

Breadth-First Search

Image Rectification

9

• Partial cubic projection (Sampaio et al., 2015)

Sampaio et al. “Determining the location of buildings given a single picture, environment maps and inaccurate GPS coordinates,” in Proc. of the ACM SAC, 2015.

Breadth-First Search

Image Rectification

SIBGRAPI 2015 10

• Two gnomonic projections (our approach)

Actual user’s location Assigned user’s location Selected vertex Estimated building’s location Visited vertex Candidate vertex

Breadth-First Search

Image Comparison

11

• Partial cubic X gnomonic projection

18 matching ASIFT features 190 matching ASIFT features

Sampaio et al. (2015) Our Approach

Sampaio et al. “Determining the location of buildings given a single picture, environment maps and inaccurate GPS coordinates,” in Proc. of the ACM SAC, 2015.

Query image

Selecting Best Views

The Second Stage

SIBGRAPI 2015 12 Left projections Right projections Query image Actual user’s location Assigned user’s location Selected vertex Estimated building’s location Visited vertex Candidate vertex The endpoint of white segments indicate detected features.

Selecting Best Views

The Proposed Algorithm

SIBGRAPI 2015 13 Left projections Right projections Query image Actual user’s location Assigned user’s location Selected vertex Estimated building’s location Visited vertex Candidate vertex

Selecting Best Views

Histogram of Detected Features

SIBGRAPI 2015 14 Actual user’s location Assigned user’s location Selected vertex Estimated building’s location Visited vertex Candidate vertex

Triangulating the Target Building

The Third Stage

SIBGRAPI 2015 15 Actual user’s location Assigned user’s location Selected vertex Estimated building’s location Visited vertex Candidate vertex

Error Propagation

The Last Stage

• Errors in input variables propagate to the results
• Uncertain input data

▪ Location of selected vertices, 𝑞𝑗 = 𝑦𝑗, 𝑧𝑗 𝑈, with independent bivariate normal distributions ▪ Angle of preferred directions, 𝜄𝑗, with independent normal distributions

• We use first order error propagation

SIBGRAPI 2015 16

Location + Errors Data Errors

+

Data Errors

+

Transformations Transformations

Error Propagation

Resulting Uncertain Location

SIBGRAPI 2015 17 Actual user’s location Assigned user’s location Selected vertex Estimated building’s location Visited vertex Candidate vertex

Results

• Implementation

▪ Java language ▪ ImageJ, EJML, Google Maps API and Affine-SIFT libraries

• Experiments

▪ 30 pictures ▪ 27 places

SIBGRAPI 2015 18

Results

Successfully Accomplished Detections

SIBGRAPI 2015 19

Detection result for case 6 Uncertain ellipse includes the target building. Detection result for case 13 Street View provides

• nly one panoramic view
• f the target,

and it was found.

Results

Partially Successful Detections

SIBGRAPI 2015 20

Detection result for case 18 𝑒𝑗𝑡𝑢 , exceeded the searching radius threshold 𝑠

𝑛𝑏𝑦 = 200 meters. By increasing 𝑠 𝑛𝑏𝑦,

the target would be detected.

Results

Failure due to Feature Matching Issues

SIBGRAPI 2015 21

Detection result for case 27 Selected vertices pointed to different parts of the building, leading to divergent preferential directions.

Results

Cases that Cannot be Handled by Our System

SIBGRAPI 2015 22

Case 30 The query image was taken at night. Case 28 The user took the picture of a tree. It is not a façade! Case 29 The only panorama that records the façade is blocked. Street View Query Image

Concluding Remarks

• An automatic image-based approach to estimate the

location of target buildings

▪ Single query image and inaccurate GPS coordinates ▪ Breadth-first search in Google Street View ▪ Gnomonic projections of the sidewalks ▪ Algorithm for selecting the best views of the building ▪ Analytic derivation of uncertainty in computed locations

• Future work

▪ Replace Affine-SIFT by a faster feature extractor ▪ Implement it as a mobile application

SIBGRAPI 2015 23

Thank you!

SIBGRAPI 2015 24