Amateur: Augmented Reality Based Vehicle Navigation System
Chu Cao 1, Zhenjiang Li 2, Pengfei Zhou 1, Mo Li 1
Nanyang Technological University 1, Singapore City University of Hong Kong 2, Hong Kong, China
UbiComp’19, London, UK 12 September 2019
Amateur: Augmented Reality Based Vehicle Navigation System Chu Cao 1 - - PowerPoint PPT Presentation
Amateur: Augmented Reality Based Vehicle Navigation System Chu Cao 1 , Zhenjiang Li 2 , Pengfei Zhou 1 , Mo Li 1 Nanyang Technological University 1 , Singapore City University of Hong Kong 2 , Hong Kong, China 12 September 2019 UbiComp19,
Chu Cao 1, Zhenjiang Li 2, Pengfei Zhou 1, Mo Li 1
Nanyang Technological University 1, Singapore City University of Hong Kong 2, Hong Kong, China
UbiComp’19, London, UK 12 September 2019
Vehicular navigation system Mobile navigation service
2
Problem: display digital map Gap: real v.s. virtual
3
✦ Display front-view road condition ✦ Instructions on live world ✦ Comparable navigation ✦ Easy to deploy 4
❖ Challenges
✦ Host-lane identifying ✦ Annotations placement
Lane-level localisation accuracy Depth information in video
Determine the correct instruction at proper position on screen.
5
❖ Related work
CarLoc: Precisely Tracking Automobile Position [SenSys’15] Real time Detection of Lane Markers in Urban Streets [IVS’08]
✦
20 built-in sensors
✦
Dead-reckoning
✦
Complicated image processing
✦
Too heavy to be affordable
6
❖ Related work
Tesla autopilot 2.0 Google automobile
✦
Maintain on hostlane
✦
Rich sensor embedded
7
❖ Related work
Towards Unified Depth and Semantic Prediction from a Single Image [CVPR’15]
✦
Three complicated neural networks
✦
Large volume of training data
8
Instructional Sign Placement Lane Identification Intersection Inference
Intersection Pin-hole Model Traffic Light Detection Subtracter Peak Detection Host Lane Particle Filter Frame Slice Instruction Type Instruction Position
❖ System architecture
System
Navigation GPS User Input Lane-Road Information Origin and Destination
ce
Result AR GPS & Motion sensor Live video
Frames
9
❖ System architecture
Navigation
Instructional Sign Placement
GPS User Input
Lane Identification Intersection Inference
Result AR GPS & Motion sensor Live video Intersection Pin-hole Model Traffic Light Detection Subtracter Peak Detection Host Lane Particle Filter Frame Slice Instruction Type Instruction Position
Frames
Lane-Road Information Origin and Destination
10
❖ System architecture — lane identification
Lane detection task. Based on pure videos. Avoid collisions for automobiles. Lane identification task. Based not only on videos… IMU sensors on mobile phone & Extra lane number information Assistant for drivers.
11
❖ System architecture — lane identification
One frame in video Image slicing of 60 frames
12
❖ System architecture — lane identification
200 400 600 800 1000 1200 1400 1600 1800 50 100 150 200
Grey value Pixel indicator
h p w
1 2 3 4 5 6 7
h ≥ 1 2max{gi}n
i=1
<latexit sha1_base64="9RDv2MYFcOXvS2/hoKTrS5Q6S04=">ACDHicdVDLSgMxFM34rPVdekmWARBKDNtoXYhFNy4rGAf0Kklk2ba0MzDJCOWMB+gC5f+gB/gxoUibv0Ad36A/2FmWkFDwQO5zLzT1OyKiQpvluzMzOzS8sZpayura+u5jc2mCKOSQMHLOBtBwnCqE8akpG2iEnyHMYaTmjo8RvXRAuaOCfynFIuh4a+NSlGEkt9XL5oT0g59B2OcLKilUx9tClrQY9asc9RQ+t+CxJmYVqCjghlfKUVC1oFcwU+dr+XfP247pU7+Xe7H6AI4/4EjMkRMcyQ9lViEuKGYmzdiRIiPAIDUhHUx95RHRVekwMd7XSh27A9fMlTNXvEwp5Qow9Ryc9JIfit5eIf3mdSLoHXUX9MJLEx5NFbsSgDGDSDOxTrBkY0Q5lT/FeIh0r1I3V9Wl/B1KfyfNIsFq1Qonug2ymCDNgGO2APWKACauAY1EDYHAF7sEjeDJujAfj2XiZRGeM6cwW+AHj9ROtdJ+1</latexit>Brightness
p = h − min{gi}
gk+ w
2
gk− w
2
<latexit sha1_base64="Rkf7FqIuQSgMdOn9OZY+TVWMXbI=">ACG3icdZDLSgMxFIYz9VbrerSTVBEQVpmxkLtQi40GUF2yqdWjJpg3NZIYko5Rh3sONj6IbF4q4ElwIfRjTi2BFfwj8fOcTs7vhoxKZqfRmpmdm5+Ib2YWVpeWV3Lrm/UZBAJTKo4YIG4dJEkjHJSVQxchkKgnyXkbrbOxnW6zdESBrwC9UPSdNHU49ipHSqJW1w+NuzqfciTst6iTXce6d+B4AuH4NontJGkNSW6KZHfMfGkODbFwsSULGjlzZF2yvtXp3uDh0GlX132gGOfMIVZkjKhmWGqhkjoShmJMk4kSQhwj3UIQ1tOfKJbMaj2xK4q0kbeoHQjys4oj8nYuRL2fd3ekj1ZW/a0P4V60RKe+oGVMeRopwPF7kRQyqA6Dgm0qCFasrw3Cguq/QtxFOgal48zoEL4vhf+bmp23DvP2uU6jAMZKgy2wDfaBYqgDM5ABVQBnfgETyDF+PeDJejbdxa8qYzGyCKRkfX4oprQ=</latexit>1 3p ≤ p − G10th
<latexit sha1_base64="SZ0J2yGuG58XxklAN9wP4PnaU=">ACB3icdVDLSsNAFJ34rPUVdSnIYBEsSRtoXZXcKHLCvYBbQiT6aQdOnk4MxFKyM6NfoA/4M6NC0Xc+gvu/AD/w2lSQUPXDicy/3uOEjApGO/azOzc/MJibim/vLK6tq5vbLZEHFMmjhgAe84SBGfdKUVDLSCTlBnsNI2xkdT/z2JeGCBv65HIfE8tDApy7FSCrJ1nd6Lkc4NpO4nISwx8gFjMPDEzs2DTlMElsvGMVaCpiRamVKaiY0i0aKQv3grnX7cVNu2Ppbrx/gyCO+xAwJ0TWNUFox4pJiRpJ8LxIkRHiEBqSrqI8Iqw4/SOBe0rpQzfgqnwJU/X7RIw8Icaeozo9JIfitzcR/K6kXSPrJj6YSJj7NFbsSgDOAkFNinGDJxogzKm6FeIhUsFIFV1ehfD1KfyftEpFs1wsnak0KiBDmyDXbAPTFAFdXAKGqAJMLgC9+ARPGnX2oP2rL1krTPadGYL/ID2+gnkFp0C</latexit>Sharpness
13
Lane Number Lane Marker Number Lane Direction [-
❖ System architecture — lane identification
A road segment with 4 lanes Templates of a 4-lane road
[474] [790] [1106] [474] [790] [-474]
[474] [790] [-474]
[474] [-474] [-790]
[-474] [-790] [-1106]
14
❖ System architecture — lane identification
Ideally detect the peaks
Matching Xobs X3
[474] [-474] [-790] [474] [-474] [-790]
1. Blockage of frontal vehicles
15
❖ System architecture — lane identification
✦ Particle filter design
[474] [-474] [-790] [790]
Xobs
[474] [790] [1106] [474] [790] [-474]
[474] [790] [-474]
[474] [-474] [-790]
[-474] [-790] [-1106]
initialisation
Dynamic time wrapping Weight updating
16
❖ System architecture — lane identification
b = e−db
<latexit sha1_base64="eXOYNe48FvqEsizy0aM4VtsE/i0=">AB9XicdVDLSsNAFJ3UV62vqhvBzWAR3BiSWqhFhIblxXsA9o0TCaTdujkwczEUkL3foIbF4q4danf4c6/cZpUNEDFw7n3Mu9zgRo0IaxoeW1hcWl7JrxbW1jc2t4rbOy0RxhyTJg5ZyDsOEoTRgDQlYx0Ik6Q7zDSdkYXM79Q7igYXAtJxGxfDQIqEcxkrqj2nH52TfnLs2s7ULpYMvZYCZqRamZOaCU3dSFGq791ar6Ozt4ZdfO+5IY59EkjMkBd04iklSAuKWZkWujFgkQIj9CAdBUNkE+ElaRXT+GhUlzohVxVIGqfp9IkC/ExHdUp4/kUPz2ZuJfXjeW3qmV0CKJQlwtsiLGZQhnEUAXcoJlmyiCMKcqlshHiKOsFRBFVQIX5/C/0mrJsnevlKpVEBGfJgHxyAI2CKqiDS9ATYAB3fgATxqY+1e9Kes9acNp/ZBT+gvXwCiCWLQ=</latexit>[474] [-474] [-790] [790]
Matching Xobs X3
[474] [-474] [-790]
✦ Particle filter design
Dynamic time wrapping under Euclidean distance
17
❖ System architecture — lane identification
Movement: lane switching
✦ Particle filter design
Resampling based on importance
[474] [-474] [-790] [790]
Xobs
18
❖ System architecture — lane identification
✦ Particle filter design
Resampling based on importance
19
❖ System architecture — lane identification
✦ Particle filter design
Resampling based on importance Lane marker of host-lane has a traversing phenomenon during lane switching.
20
❖ System architecture — lane identification P(Ls|O) = P(O|Ls) · P(Ls) P(O)
<latexit sha1_base64="OilGbpP5ivGFm41K08aA2r/hG9I=">ACdnicdVHLagIxFM1MX9a+bLsqhRIUqW6GStYFwWhmy4KWqgPcEQyMaPBzIMkU5BxPqE/12/o5suGx+VWvRA4OTcR27OdUJGhTNT03f2d3bP0gdpo+OT07PMucXLRFEHJMmDljAOw4ShFGfNCWVjHRCTpDnMNJ2xo+zePuNcED/1VOQtLz0NCnLsVIKqmfebc9JEcYsbiRFb8OemL6epWT4oPtsRjcm15PpWmHRxoNAbu1bTLa1KSb9TM40qnPABamUl6RqQcsw58iBJRr9zIc9CHDkEV9ihoToWmYoezHikmJGkrQdCRIiPEZD0lXURx4RvXhuWwLzShlAN+Dq+BLO1b8VMfKEmHiOypwNKf7HZuKmWDeS7n0vpn4YSeLjxUNuxKAM4GwHcEA5wZJNFEGYUzUrxCOkDJZqU2lwu9P4XbSKhnWnVF6KedqxtKOFLgGWVAFqiAGngCDdAEGHxpV1pWy2nf+o2e128Xqbq2rLkEa9DNH9d/w3E=</latexit>P(Ls|O) = P(O|Ls) · β
<latexit sha1_base64="F+5Sx7dI6tf6U3W8FKAEb7P86qI=">ACQHicdVBLSwMxGMz6tr6qHr0Ei9Bel0t1B6EghcPghWsFbqlZNsG5p9kHwrlO3+NC/+BG+evXhQxKsn03YVlToQmMx8Q76MGwmuwLIejbn5hcWl5ZXV3Nr6xuZWfnvnWoWxpKxBQxHKG5coJnjAGsBsJtIMuK7gjXdwenYb94yqXgYXMEwYm2f9ALucUpAS5180/EJ9CkRST0tfvPztKNG37eLtHQyc+wiHf2KlBzaDcFxGZBOvmCZ1QnwlFTKGana2DatCQoQ72Tf3C6IY19FgAVRKmWbUXQTogETgVLc06sWETogPRYS9OA+Ey1k0kBKT7QShd7odQnADxRfyYS4is19F09Od5X/fXG4iyvFYN3E54EMXAjp9yIsFhCP28RdLhkFMdSEUMn1rpj2iSQUdOc5XcLXT/H/5PrQtI/Mw8tyoWZmdaygPbSPishGFVRDZ6iOGoiO/SEXtCrcW8G2/G+3R0zsgyu+gXjI9P65iy9w=</latexit>P(O|Ls) = e−d0
s
<latexit sha1_base64="2WczuQIuT4KbxgclCcSKZ6L9I+0=">ACHXicdVDLSgMxFM3UV62v+ti5CRaxLhxmaqF2IRTcuBCsYGuhHYdMJtOGZh4kGaGM8yNu/BU3LhRx4Ub8G9OHRUPBE7OuZd73EiRoU0jA8tMzM7N7+QXcwtLa+sruXN5oijDkmDRykLcJAijAWlIKhlpRZwg32HkyumfDP2rG8IFDYNLOYiI5aNuQD2KkVSnS93fCR7GLGknhan/DyFt3D6O0tsQ+PIblODlxb7KV2vmDo1RHgmFTKE1I1oakbIxTABHU7/9ZxQxz7JCYISHaphFJK0FcUsxImuvEgkQI91GXtBUNkE+ElYyuS+GuUlzohVy9QMKR+r0jQb4QA9RlcONxW9vKP7ltWPpHVkJDaJYkgCPB3kxgzKEw6igSznBkg0UQZhTtSvEPcQRlirQnArh61L4P2mWdPNQL12UC7WtSRxZsA12QBGYoAJq4BTUQNgcAcewBN41u61R+1Fex2XZrRJzyb4Ae39E8DzojI=</latexit>✦ Particle filter design
Resampling based on importance
21
❖ System architecture — lane identification
✦ Particle filter design
Resampling based on importance
ˆ wl = X
s+i=l
P(Ls|O) · wi = X
s+i=l
β · P(O|Ls) · wi
<latexit sha1_base64="F35vbOycUXeIgJTW5UQYFwL5VQ=">AChXicdVHbatAEF2pzaXOzWn61pchJuAQYiTH5PIQGuhLHwpxIE4ClhGj9TpesrqwO0owiv6kX9W3/k3XthqS4hxYOHtm5uzsTJQpacjz/juh49Lyurn2pr6xubW/XtzcmzTUXPZ6qVN9FaISieiRJCXuMi0wjpS4jR6+T+O3j0IbmSbXNMnEIMb7RI4kR7JSWP8VjJGKpzJUcA6ByeOwMAfSclVCECONOaqiWzZf+M8yNM8vt8tyHwI+TAmeQgkLCJBWGUstLs4RnemL8yDOsNr3U2A8zJSaciZz74LW+GBqvQDeu/g2HK81gkxBUa0/e9jAYFapJcibIW5EZkyB/wXvQtTAWZlDMpljCnlWGMEq1PQnBTH1dUWBszCSObOa0X/N/bCouivVzGp0OCplkOYmEzx8a5QohelKYCi14KQmliDX0vYKfIwaOdnF1ewQ/v0U3ic37Z/1GpfdRoX6pxrLKvbJc1mc9O2AX7wbqsx7jOk3Hd9ruinvodtzjearVDU7A3cb38BSmPDAg=</latexit>22
Navigation
Instructional Sign Placement
GPS User Input
Lane Identification Intersection Inference
Result AR GPS & Motion sensor Live video Intersection Pin-hole Model Traffic Light Detection Subtracter Peak Detection Host Lane Particle Filter Frame Slice Instruction Type Instruction Position
Frames
Lane-Road Information Origin and Destination
❖ System architecture
Instructional S Lane Identification
Peak Detection Host Lane Particle Filter Frame Slice Instruction Type
23
Navigation
Instructional Sign Placement
GPS User Input
Lane Identification Intersection Inference
Result AR GPS & Motion sensor Live video Intersection Pin-hole Model Traffic Light Detection Subtracter Peak Detection Host Lane Particle Filter Frame Slice Instruction Type Instruction Position
Frames
Lane-Road Information Origin and Destination
❖ System architecture
Instructional S Lane Identification
Peak Detection Host Lane Particle Filter Frame Slice Instruction Type
✓
24
Navigation
Instructional Sign Placement
GPS User Input
Lane Identification Intersection Inference
Result AR GPS & Motion sensor Live video Intersection Pin-hole Model Traffic Light Detection Subtracter Peak Detection Host Lane Particle Filter Frame Slice Instruction Type Instruction Position
Frames
Lane-Road Information Origin and Destination
❖ System architecture
Instructional S Lane Identification
Peak Detection Host Lane Particle Filter Frame Slice Instruction Type
✓
25
Navigation
Instructional Sign Placement
GPS User Input
Lane Identification Intersection Inference
Result AR GPS & Motion sensor Live video Intersection Pin-hole Model Traffic Light Detection Subtracter Peak Detection Host Lane Particle Filter Frame Slice Instruction Type Instruction Position
Frames
Lane-Road Information Origin and Destination
❖ System architecture
al Sign Placement Intersection Inference
Intersection Pin-hole Model Traffic Light Detection Subtracter Instruction Position
26
Please refer to our paper for more details.
Navigation
Instructional Sign Placement
GPS User Input
Lane Identification Intersection Inference
Result AR GPS & Motion sensor Live video Intersection Pin-hole Model Traffic Light Detection Subtracter Peak Detection Host Lane Particle Filter Frame Slice Instruction Type Instruction Position
Frames
Lane-Road Information Origin and Destination
❖ System architecture
Instructional Sign Placement Lane Identification Intersection Inference
Intersection Pin-hole Model Traffic Light Detection Subtracter Peak Detection Host Lane Particle Filter Frame Slice Instruction Type Instruction Position
27
❖ Examples
1 2 2 2
Cloudy Daytime Nightfall
28
❖ Demo
29
❖ Evaluation
30
❖ Evaluation
✦ Running on rental taxis ✦ Levenshtein distance ✦ Ground truth: manually checking the video ✦ Implemented on Nexus 5X
Around 300 km travel distance Vgen = {LC2, L, RC1, R}, Vgt = {LC1, L, RC1, R}
31
❖ Evaluation
acc = Levenshtein |Vgt|
<latexit sha1_base64="PHUkX7ZaVIUey43QnpEritsaU=">ACXicdVA9SwNBEN2LXzF+RS1tFoNgFe5iQC2EgI2FhYJhCSEvc1csmRv79idE8J5rY1/xcZCEVv/gZ3/xs2HoKIPBh7vzTAz4+lMOi6H05ubn5hcSm/XFhZXVvfKG5uNUyUaA51HslIX/vMgBQK6ihQwnWsgYW+hKY/PB37zRvQRkTqCkcxdELWVyIQnKGVukXKOD9pB5rx9BxuQJkBglBZetvopn3MbrNuseSWjyegU3JYnZFj3pld4ISmeGiW3xv9yKehKCQS2ZMy3Nj7KRMo+ASskI7MRAzPmR9aFmqWAimk04+yeieVXo0iLQthXSifp9IWjMKPRtZ8hwYH57Y/Evr5VgcNRJhYoTBMWni4JEUozoOBbaExo4ypEljGthb6V8wGwsaMr2BC+PqX/k0al7B2UK5fVUq06iyNPdsgu2SceOSQ1ckYuSJ1wckceyBN5du6dR+fFeZ25pzZzDb5AeftE+0Um8U=</latexit>43 74 101 67 86 # of instructions
20 40 60 80 100 A B C D E Morning Peak Evening Peak Off-peak
Accuracy (%) Route
32
❖ Evaluation
33
❖ Evaluation
✦ Around 27m error (52m) ✦ 105 pixels offset on screen (127 pixels)
CDF 0.2 0.4 0.6 0.8 1 Turning Arrow Placement Offset on Screen (pixels) 20 40 60 80 100 120 140
(90, 0.9) (109, 0.9) Our system GPS readings
34
❖ System architecture — intersection inference
1. Flickering feature of LED bulbs 2. Rolling shutter effect on CMOS
RGB to Gray Subtracter Binarization Color filtering y of lights
Frames Frames Frames
37
❖ System architecture — intersection inference
ia = h h − H it
<latexit sha1_base64="5f7cJ9qrJCfMc3dVDs2dBiKBF48=">AB/XicdZBLS8NAFIUn9VXrKz52bgaL4MaQ1ELtQi46bKCfUAbwmQ6aYZOJmFmItRQ/CtuXCji1v/hzn/jtI1QRQ8MHM65l7l8fsKoVLb9aRWVtfWN4qbpa3tnd09c/+gI+NUYNLGMYtFz0eSMpJW1HFSC8RBEU+I1/fD3ru3dESBrzWzVJiBuhEacBxUjpyDOPqIeuBoFAOAunWXjenFJPeWbZsey5oG3VtWrV3NQd+F2VQa6WZ34MhjFOI8IVZkjKvmMnys2QUBQzMi0NUkShMdoRPrachQR6Wbz6fwVCdDGMRCP67gPF3eyFAk5STy9WSEVCh/d7Pwr6fquDSzShPUkU4XnwUpAyqGM5QwCEVBCs20QZhQfWtEIdIo1AaWGkZwv+mU7GcC6tyUy03qjmOIjgGJ+AMOKAGqAJWqANMLgHj+AZvBgPxpPxarwtRgtGvnMIfsh4/wIKapW</latexit>The position of traffic light on the screen is known.
Image Plane Traffic Light D H h f it ia Focal Point A Road Surface
38
❖ Evaluation
✦ Until converging ✦ Accurately identifying in 5s ✦ Force output in 2s ✦ More observations ->
better performance
25 50 75 100 2 3 4 5 Accuracy Time
Accuracy (%) Latency (s)
2 1.5 1 0.5
Number of lanes Number of lanes
5 4 3 2
Latency (s)
1 2 3 4 5
Mean Standard deviation
39
❖ Evaluation
Likert scale rating questions
✦
Ease of use (Q1 & Q5)
✦
Perceived distraction (Q2 & Q6)
✦
Navigational experience (Q4 & Q8)
✦
User-friendliness (Q3 & Q7)
No. Rating Question Statement Q1 It was easy to navigate using this navigation service. Q2 I need to pay extra attention on this navigation service when driving. Q3 This navigation service provided user-friendly guidance . Q4 This navigation service was useful in helping me navigate properly. Q5 It was easy for me to learn how to use this navigation service. Q6 I paid most of my attention on driving using this navigation service. Q7 The guidance was user-friendly to interact with. Q8 This navigation service provided me with effective guidance.
40
❖ Evaluation
✦
With-in subject user study
✦
Wilcoxon Signed Rank Tests
✦
Easier to use
✦
Less distracted
✦
More user-friendly
No. Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 p-value 0.0016 <0.0001 <0.0001 0.0003 <0.0001 <0.0001 <0.0001 <0.0001
Scale Rating
1 2 3 4 5 6 7 8
Question Indicator
A 1 G 1 A 2 G 2 A 3 G 3 A 4 G 4 A 5 G 5 A 6 G 6 A 7 G 7 A 8 G 8
41
Route A B C D E Length (km) 12.7 17.4 36.8 12.5 16.8 Length of Expressway (km) 8.73 15.35 36 4.46 14.28 Length of Highway (km) 3.97 2.05 0.8 8.04 2.52 Number of Traffic Lights 15 19 5 18 17 Average Velocity (km/h) 49.7 51.2 56.1 45.4 53.7
❖ Evaluation — details of routes
42
Property Description of Group A Description of Group B Age (year) 32 - 61 (mean: 40.3) 31 - 57 (mean: 42.8) Driving Experience (year) 2 - 27 (mean: 15.8) 1 - 30 (mean: 17.6) Gender 21 Male (84%), 4 Female (16%) 19 Male (76%), 6 Female (24%)
❖ Evaluation — details of drivers
43