FP7 Project Grant Number 288238 Brief introduction of WP4 10min - - PowerPoint PPT Presentation
FP7 Project Grant Number 288238 Brief introduction of WP4 10min - - PowerPoint PPT Presentation
FP7 Project Grant Number 288238 Brief introduction of WP4 10min General objectives and role in VENTURI Partners, tasks, deliverables, milestones Timeline Main results (first year) 25min Plans for second year 15min Conclusions
» Brief introduction of WP4
˃ General objectives and role in VENTURI ˃ Partners, tasks, deliverables, milestones ˃ Timeline
» Main results (first year) » Plans for second year » Conclusions and questions
2 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
10min 25min 15min
3 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
WP5: Adaptive Content Harvesting, Creation and Delivery WP4: Context Sensing and Interpretation WP3: User Interface and Interaction Design WP1: Project Management WP6 Technology Integration, Evaluation and Test-Cases WP7: Dissemination and Exploitation WP2: Architecture Definition and Development
4 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
5 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
On-board sensors:
- vision: camera(s)
- audio: microphone(s)
- location: GPS, …
- relative orientation / motion:
accelerometers, gyroscopes
- absolute orientation: compass
- ther: pressure, proximity
Data sources:
- maps
- DEMs
- geo-tagged data: images,
3D models, etc.
6 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
7 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
8 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
9 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
10 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
Absolute orientation (yaw, pitch, roll, field of view, etc.) Geographical localization (lat/lon/alt) (also indoor)
11 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
- Scene reconstruction
- Model the immediate
surroundings of the mobile device
12 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
- User behavior understanding
- What’s happening here?
- Noise level / scene classification
- Infer what the user is doing and
what he could experience with his senses
13 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
Enable and explore innovative ways to interact with the environment
Example: SixthSense (Fluid Interface Group at MIT Media Lab)
14 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
FBK 28MM ST-Italy 21MM ST-France Fraunhofer HHI 19MM metaio 16MM INRIA 18MM Sony 6MM STE 3MM
Total WP4 MMs: 111
15 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
T4.1 Hardware sensor interpretation T4.2 Auditory scene analysis T4.3 Visual analysis of the environment and geo-position refining T4.4 Physical surroundings modeling T4.5 What is, or could be, going on around me?
16 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4 M1 = oct 2011 M12 = sep 2012 M24 = sep 2013 M36 = sep 2014
T4.1 T4.2 T4.3 T4.4 T4.5
M18 = mar 2013 M6 = mar 2012 M15 = dec 2012 M30 = mar 2014 M9 = jun 2012
first year
17 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
» D4.1 Synchronized dataset containing a dump of all on-board
sensors of a real device, while being used in a real environment (STE, Mar 2012, M6, O-CO)
» D4.2 Report on expected platform requirements of WP4
algorithms (FBK, Jun 2012, M9, R-CO)
» D4.3 WP4 outcome definitions and API specification for inter-
task / inter-WP communications (ST-Italy, Dec 2012, M15, R-CO)
» D4.4 Prototype of and Report on multi-modal estimation of geo-
location and orientation (FBK, Mar 2013, M18, P-CO / R-PU)
» D4.5 In-the-lab prototype and Report on visual analysis and
reconstruction of the scene (Fraunhofer, Sep 2013, M24, P-CO / R- PU)
» D4.6 Final prototype integrating all WP4 outcomes (FBK, Mar
2014, M30, P-CO)
first year
» MS9 First version of chain for geo-localization / orientation estimation, based
- n non-audio / non-video (FBK, Sep 2012, M12)
» MS10 Final version of chain for geo-localization / orientation estimation
(FBK, Mar 2013, M18)
» MS11 WPC version of visual analysis and reconstruction of the scene, ready
for integration (FBK, Sep 2013, M24)
18 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
first year
19 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4 M1 = oct 2011 M12 = sep 2012 M24 = sep 2013 M36 = sep 2014
T4.1 T4.2 T4.3 T4.4 T4.5
M18 = mar 2013 M6 = mar 2012 M15 = dec 2012 M30 = mar 2014 M9 = jun 2012
D4.1 D4.2 D4.3 D4.4 D4.5 D4.6
first year
MS9 MS10 MS11
» Brief introduction of WP4
˃ General objectives and role in VENTURI ˃ Partners, tasks, deliverables, milestones ˃ Timeline
» Main results (first year) » Plans for second year » Conclusions and questions
20 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
10min 25min 15min
21 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
» Cross-partner / Cross-task results
˃ Shared tools and Android OS improvements ˃ Deliverables
» Results showcased in Y1 demo
˃ Algorithms and their integration
» Other results (relevant for Y2 and Y3 Use Cases)
˃ Analysis of state of the art ˃ Design, development, implementation of algorithms ˃ Evaluation and testing
22 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
23 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
» One of the main requirement elicited (in T2.1.2): accurate and common timestamping between camera and sensor domain
ST9500-based platform: done ST9540-based platform: done, under debugging Next (8680) platform: taken into account from the beginning
24 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
Video: timestamping
Android porting layer Android libraries
Kernel space User space
Android application Access services Android application framework Core Multimedia Connectivity Android runtime
Android
- rigin
ST-Ericsson proprietary Non-Android FOSS
Legend:
Sensor HAL Sensor Service (server) Sensor Manager library (client) Sensor Manager Class Sensor Manager JNI I2C Core I2C driver Application Sensor data are Time stamped here
Third- party/other Ambient Light driver Proximity driver Gyro-scope driver Magneto- meter driver Accelero- meter driver
Sensors
I2C
Hardware
Sensor HAL provides Timestamp here Sensor drivers (Linux) not providing Timestamps Imaging Signal Processor
Android ICS Vanilla
Android porting layer Android libraries
Kernel space User space
Android application Access services Android application framework Core Multimedia Connectivity Android runtime
Android
- rigin
ST-Ericsson proprietary Non-Android FOSS
Legend:
Sensor HAL Sensor Service (server) Sensor Manager library (client) Sensor Manager Class Sensor Manager JNI I2C Core I2C driver Application Sensor data are Time stamped here
Third- party/other Ambient Light driver Proximity driver Gyro-scope driver Magneto- meter driver Accelero- meter driver
Sensors
I2C
Hardware
Sensor HAL provides Timestamp here Time comes from here SIA IIO Framework
VENTURI modification for 9500/9540 platforms
» Ongoing experiments to quantitatively evaluate the impact of accurate timestamping on actual WP4 algorithms.
27 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
» FBKSequenceRecorder tool
» It can capture and dump to disk timestamped data from all on-board sensors (raw video, audio, other sensors). » On STE boards, it exploits the new timestamping APIs (still debugging on STE9540). However, it works on any Android device. » Source code and APK shared with VENTURI partners. » Possibility to share it with general public.
28 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
29 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
» D4.1 Synchronized dataset containing a dump of all on-board
sensors of a real device, while being used in a real environment (STE, Mar 2012, M6, O-CO)
» D4.2 Report on expected platform requirements of WP4
algorithms (FBK, Jun 2012, M9, R-CO)
» D4.3 WP4 outcome definitions and API specification for inter-
task / inter-WP communications (ST-Italy, Dec 2012, M15, R-CO)
» D4.4 Prototype of and Report on multi-modal estimation of geo-
location and orientation (FBK, Mar 2013, M18, P-CO / R-PU)
» D4.5 In-the-lab prototype and Report on visual analysis and
reconstruction of the scene (Fraunhofer, Sep 2013, M24, P-CO / R- PU)
» D4.6 Final prototype integrating all WP4 outcomes (FBK, Mar
2014, M30, P-CO)
D4.1
» Use Case datasets and algorithm specific
- datasets. GT provided when possible /
relevant. » Immediately shared with VENTURI partners through mailing list. » Integral version on restricted section of VENTURI website:
https://venturi.fbk.eu/reserved/datasets
» General interest datasets shared with the public: https://venturi.fbk.eu/results/public-
datasets
» Submitted document D4.1 needs to be updated with current status. » Website and D4.1 document will periodically be updated with new data during the entire life of VENTURI.
30 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
Website
31 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
» D4.1 Synchronized dataset containing a dump of all on-board
sensors of a real device, while being used in a real environment (STE, Mar 2012, M6, O-CO)
» D4.2 Report on expected platform requirements of WP4
algorithms (FBK, Jun 2012, M9, R-CO)
» D4.3 WP4 outcome definitions and API specification for inter-
task / inter-WP communications (ST-Italy, Dec 2012, M15, R-CO)
» D4.4 Prototype of and Report on multi-modal estimation of geo-
location and orientation (FBK, Mar 2013, M18, P-CO / R-PU)
» D4.5 In-the-lab prototype and Report on visual analysis and
reconstruction of the scene (Fraunhofer, Sep 2013, M24, P-CO / R- PU)
» D4.6 Final prototype integrating all WP4 outcomes (FBK, Mar
2014, M30, P-CO)
D4.2
» Report on expected platform requirements of WP4 algorithms (FBK, Jun 2012, M9, R-CO) » Incremental wrt existing deliverables:
˃ D2.1.1 Use cases, application definition and system requirements for STE U8500-based platform ˃ D2.2.1 Early Detailed Design Specifications for STE U8500-based platform
» Submitted on July, 11th, 2012
˃ Initial identification of algorithmic modules (input/output, impact on requirements). ˃ List of requirements updated wrt D2.1.1 and D2.2.1
32 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
» Cross-partner / Cross-task results
˃ Shared tools and Android OS improvements ˃ Deliverables
» Results showcased in Y1 demo
˃ Algorithms and their integration
» Other results (relevant for Y2 and Y3 Use Cases)
˃ Analysis of state of the art ˃ Design, development, implementation of algorithms ˃ Evaluation and testing
33 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
35 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
36 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
» Map reconstruction -> .3dmap file » Indoor and small-scale environments (table-top scenarios) with roughly one meter cube volume. » Development and testing also for larger scale environments for UCY2 and UCY3.
37 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
» .3dmap file -> 3D markerless tracking
38 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
39 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
» Cross-partner / Cross-task results
˃ Shared tools and Android OS improvements ˃ Deliverables
» Results showcased in Y1 demo
˃ Algorithms and their integration
» Other results (relevant for Y2 and Y3 Use Cases)
˃ Analysis of state of the art ˃ Design, development, implementation of algorithms ˃ Evaluation and testing
40 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
41 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4 M1 = oct 2011 M12 = sep 2012 M24 = sep 2013 M36 = sep 2014
T4.1 T4.2 T4.3 T4.4 T4.5
M18 = mar 2013 M6 = mar 2012 M15 = dec 2012 M30 = mar 2014 M9 = jun 2012
D4.1 D4.2 D4.3 D4.4 D4.5 D4.6
first year
MS9 MS10 MS11
42 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
Audio Patterns Database FFT Transform Ambiant audio Classifier OSM Data Analysis Results Feature Extractor
Filter possible patterns using OSM context Office Restaurant Parking … other places Car Phone ringing … other objects
43 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
Name Type (scene or object)
OSM Tags OSM Tags
OSM Tags Feature Vector Feature Vector Feature Vector
…
Features
Feature vectors based
- n
mean frequency power repartition over time
Fingerprints
Classifier uses objects and scene fingerprints stored in XML format
44 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
Example: Airbus A319
An Airbus A319 when seating near the aisle
Example: restaurant
Busy restaurant
» Source code released to partners » Cross-platform C++ API shared with partners » Ready for integration with Android NDK » Fully documented (also in D4.3) » Can perform real-time and offline classification
- f environment areas and objects
45 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
» implementation of a pedometer based on walking models and vertical acceleration patterns » estimation of orientation based on gyro data and an OpenStreetMap network » estimation of position based on estimated stride length, orientation and OpenStreetMap network
46 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
47 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
Input
HEADING DRIFT COMPENSATION
Pedestrian Dead-Reckoning
Navigation network Output
PEDOMETER
Position Heading
GYROMETRY DISTANCE COMPENSATION
heading position
INTERNAL VARIABLES Map-Matching
48 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
Distance errors Map Matching needed Heading errors due to drift HDC needed Distance and Heading errors Map Matching + HDC needed
49 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
Video: PDR
» Scene classification (also known as Visual
categorization) aims to automatically classify images into a set of different semantic classes, based on information extracted from the images. » holistical technique, based on features extracted from the YUV images (global features, SVM).
50 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
Target classes: {Building, Open Country, Forest} Building
Image representation Classification model
51 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
» Downloaded a publicly available dataset, in order to have a benchmark vs state of the art technique (“Modeling the shape of the scene: a holistic representation of the spatial envelope” - Aude Oliva, Antonio Torralba); » 2668 images in 8 classes;
coast forest highway inside city mountain
- pen country
street tall building
52 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
Accuracy: 83.3%; Torralba-Oliva obtain 83.7% of accuracy with a much more complex approach (using FFT);
First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
Video: SceneClass
54 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
55 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
Video: alignFBK
56 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4 Camera image 3D model
?
Goal: register known 3D model with 2D camera image
57 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
» 2D-3D matching for pose estimation
˃ Exploitation of architectural properties of man-made buildings, e.g. by extraction of vanishing lines / points ˃ Use of inertial sensor data and camera intrinsic calibration to constrain the calculation of vanishing points
[Buhrig, Eisert: “Pose estimation for urban environments”, CVMP, Dec. 2012]
58 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
» 2D-3D matching for pose estimation
˃ Example: register 3D model of tax office in image ˃ Result: exact position and orientation of 3D model
Estimated orientation with vanishing cues Position and
- rientation refinement
59 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
Camera pose estimation Map creation and update Features detector Temporal tracking Video Analytics Camera position and Sparse 3d map GEOMETRY PIPELINE
Structure from Motion
Goal: estimating the trajectory of a moving camera and reconstruct the shape of the surrounding environment
60 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
» Android app developed to easy handle camera properties and processing. » Incremental algorithm porting and testing using JNI API to integrate c/cpp code in Android. » Developed an unique build system solution to target X86/ARM/Android platform. » Tested application in indoor/outdoor.
Features Detection: indoor
- utdoor
Optical flow indoor
Demo
61 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
» User-generated 3D content
˃ Take “3D snapshots” of the world with mobile camera ˃ Dense reconstruction from high-resolution stereo-images, allows recovering object surfaces at high detail ˃ Example: reconstruction from 2 freehand-shots in a museum (uncalibrated setup, uncontrolled illumination)
62 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
Indoor example
63 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
Outdoor example:
reconstruction from 2 free-hand shots (uncalibrated setup, uncontrolled illumination)
» Text-in-scene detection and recognition (T4.3.3). » Robust detection of interest points / regions (T4.3.1). » Real-time terrain model rendering on a mobile device (T4.3.2). » Gesture recognition (T4.4). » Generation of a sparse 3D map, single camera (T4.4). » Image match with google StreetView
64 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
» Cross-partner / Cross-task results
˃ Timestamping (T4.1) ˃ Acquisition tool ˃ Deliverables D4.1, D4.2
» Results showcased in Y1 demo
˃ Map reconstruction (T4.4) ˃ 3D markerless tracking (T4.3)
» Other results (Y2 and Y3 Use Cases)
˃ Auditory scene analysis (T4.2) ˃ PDR-based localization (T4.1) ˃ Scene Classification (T4.3.1) ˃ Terrain model alignment (T4.3.2) ˃ 2D-3D model building matching (T4.3.2) ˃ Sparse 3D map – SfM (T4.4) ˃ Dense 3D reconstruction (T4.4) ˃ SoTA analysis, other preliminary works
65 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
» Brief introduction of WP4
˃ General objectives and role in VENTURI ˃ Partners, tasks, deliverables, milestones ˃ Timeline
» Main results (first year) » Plans for second year » Conclusions and questions
66 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
10min 25min 15min
67 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
T4.1 T4.2 T4.3 T4.4 T4.5 progress
68 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
T4.1 T4.2 T4.3 T4.4 T4.5
M12 M12 M12 M12 M12
progress
» Complete D4.3: WP4 outcome definitions and API specification for inter-task/inter-WP communications (ST-I, M15 Dec2012, R-CO)
˃ Define all algorithmic modules: brief description, input/output, APIs ˃ Following expected UCY2 and UCY3, identify interactions, bottlenecks, role of T4.5 (integration)
» Complete T4.1, T4.3, T4.4, following specifications defined in D4.3, leading to D4.5. » Implement UCY2, integrating and fusing appropriate WP4 outcomes (T4.5)
69 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4
70 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18th of December 2012 WP4 M1 = oct 2011 M12 = sep 2012 M24 = sep 2013 M36 = sep 2014
T4.1 T4.2 T4.3 T4.4 T4.5
M18 = mar 2013 M6 = mar 2012 M15 = dec 2012 M30 = mar 2014 M9 = jun 2012
D4.1 D4.2 D4.3 D4.4 D4.5 D4.6
first year
MS9 MS10 MS11