Wrap Up Lecture Instructor - Simon Lucey 16-423 - Designing Computer Vision Apps
Today • Review - Project Presentations • Emerging Trends in Mobile Vision • Project Discussions
Reminder - Project Presentation • Each team will be given approximately 2.5 minutes per member to present (for example a 2 member team will have 5 minutes allotted). • Each team will fill out the following form, providing a short (must be shorter than your allotted time) YouTube clip describing your App in action. • Teams can submit their YouTube clips through the form http://goo.gl/forms/YoeQt0c1Hf. • 16423 staff will select the best presentations, with the winner receiving the the best project prize.
Today • Review - Project Presentations • Emerging Trends in Mobile Vision • Project Discussions
Why is Mobile CV Different? 5
Why is Mobile CV Different? 5
Why is Mobile CV Different? 6
Why is Mobile CV Different? 6
Why is Mobile CV Different? 6
Balancing Power versus Perception 7
Algorithm Software Architecture SOC Hardware
Correlation Filters with Limited Boundaries Hamed Kiani Galoogahi Terence Sim Simon Lucey Algorithm Istituto Italiano di Tecnologia National University of Singapore Carnegie Mellon University Genova, Italy Singapore Pittsburgh, USA hamed.kiani@iit.it tsim@comp.nus.edu.sg slucey@cs.cmu.edu Abstract Correlation filters take advantage of specific proper- ties in the Fourier domain allowing them to be estimated efficiently: O ( ND log D ) in the frequency domain, ver- sus O ( D 3 + ND 2 ) spatially where D is signal length, and N is the number of signals. Recent extensions to cor- Software (a) (b) relation filters, such as MOSSE, have reignited interest of their use in the vision community due to their robustness and attractive computational properties. In this paper we demonstrate, however, that this computational efficiency 1 comes at a cost. Specifically, we demonstrate that only D proportion of shifted examples are unaffected by boundary effects which has a dramatic effect on detection/tracking � � (c) (d) performance. In this paper, we propose a novel approach to correlation filter estimation that: (i) takes advantage of Figure 1. (a) Defines the example of fixed spatial support within inherent computational redundancies in the frequency do- the image from which the peak correlation output should occur. main, (ii) dramatically reduces boundary effects, and (iii) (b) The desired output response, based on (a), of the correlation is able to implicitly exploit all possible patches densely ex- filter when applied to the entire image. (c) A subset of patch ex- tracted from training examples during learning process. Im- amples used in a canonical correlation filter where green denotes pressive object tracking and detection results are presented a non-zero correlation output, and red denotes a zero correlation in terms of both accuracy and computational efficiency. output in direct accordance with (b). (d) A subset of patch ex- Architecture amples used in our proposed correlation filter. Note that our pro- posed approach uses all possible patches stemming from different 1. Introduction parts of the image, whereas the canonical correlation filter simply employs circular shifted versions of the same single patch. The Correlation between two signals is a standard approach central dilemma in this paper is how to perform (d) efficiently in to feature detection/matching. Correlation touches nearly the Fourier domain. The two last patches of (d) show that D − 1 T every facet of computer vision from pattern detection to ob- patches near the image border are affected by circular shift in our ject tracking. Correlation is rarely performed naively in the method which can be greatly diminished by choosing D << T , where D and T indicate the length of the vectorized face patch in spatial domain. Instead, the fast Fourier transform (FFT) (a) and the whole image in (a), respectively. affords the efficient application of correlating a desired tem- plate/filter with a signal. Correlation filters, developed initially in the seminal proach is that it attempts to learn the filter in the frequency work of Hester and Casasent [15], are a method for learning domain due to the efficiency of correlation in that domain. a template/filter in the frequency domain that rose to some prominence in the 80s and 90s. Although many variants Interest in correlation filters has been reignited in the vi- have been proposed [15, 18, 20, 19], the approach’s central sion world through the recent work of Bolme et al. [5] on SOC Hardware tenet is to learn a filter, that when correlated with a set of Minimum Output Sum of Squared Error (MOSSE) correla- training signals, gives a desired response, e.g. Figure 1 (b). tion filters for object detection and tracking. Bolme et al.’s Like correlation, one of the central advantages of the ap- work was able to circumvent some of the classical problems
Algorithm Software Ax = b Architecture SOC Hardware
Algorithm Software Architecture Hardware
Algorithm Software Architecture Hardware
Algorithm Software SIMD (Single Instruction, Multiple Data) Architecture SOC Hardware
Algorithm 4-way x + Software SIMD (Single Instruction, Multiple Data) � � Architecture � (length 2, 4, 8, …) vectors of integers or floats � Names: MMX, SSE, SSE2, … � SOC Hardware � �
Algorithm Software Architecture SOC Hardware
Algorithm Software Architecture SOC Hardware APIs in the current versions of OpenGL ES do not have the “scatter”
Algorithm Software Architecture SOC Hardware APIs in the current versions of OpenGL ES do not have the “scatter”
Algorithm Software Architecture SOC Hardware
Algorithm Optimize Software Architecture SOC Hardware
Algorithm Software Architecture SOC Hardware Optimize
Algorithm Optimize Software Architecture SOC Hardware
MATLAB OpenCV
MATLAB OpenCV
MATLAB OpenCV
Some Insights for Mobile CV • Very difficult to write the fastest code. • When you are prototyping an idea you should not worry about this, but • You have to be aware of where bottle necks can occur. • This is what you will learn in this course. • Highest performance in general is non-portable. • If you want to get the most out of your system it is good to go deep. • However, options like OpenCV are good when you need to build something quickly that works. • To build good computer vision apps you need to know them algorithmically. • Simply knowing how to write fast code is not enough. • You need to also understand computer vision algorithmically. • OpenCV can be dangerous here. Some insights taken from Markus Püschel’s lectures on “How to Write fast Numerical Code”.
Source: http://www.slashgear.com/iphone-7-potential-wanes-as-android-n-starts-to-tango-20440932/
Source: http://www.slashgear.com/iphone-7-potential-wanes-as-android-n-starts-to-tango-20440932/
Better Selfies Ohad Fried, Eli Shechtman, Dan B Goldman, and Adam Finkelstein. Perspective-aware Manipulation of Portrait Photos. ACM Transactions on Graphics (Proc. SIGGRAPH), July 2016.
Better Selfies Ohad Fried, Eli Shechtman, Dan B Goldman, and Adam Finkelstein. Perspective-aware Manipulation of Portrait Photos. ACM Transactions on Graphics (Proc. SIGGRAPH), July 2016.
Emerging Trends - Low Power 18 Taken from: http://lpirc.net/
Emerging Trends - High Speed Camera iPhone 6 Samsung Galaxy S5 19
Emerging Trends - High Speed Camera iPhone 6 Samsung Galaxy S5 19
Depth From Shake H. Alismail, B. Browning, S. Lucey. “Photometric Bundle Adjustment" ACCV 2016.
Depth From Shake H. Alismail, B. Browning, S. Lucey. “Photometric Bundle Adjustment" ACCV 2016.
Results
Results
Results
Results
Emerging Trends - Depth Cameras 23
Kinect 2 Sensor Standard Basketball Hoop 24
Ball Tracking
Ball Tracking
Ball Tracking
26
26
Limitations - Range 27
Limitations - Ambient Light • A sunny day on Earth can reach up to 1120Wm -2 • Tabletop projector releases on average 10W of light. Spectral Irradiance (in Wm − 2 nm − 1 ) 2.5 Extraterrestrial Radiation Direct + Circumsolar Irradiance 2 1.5 1 0.5 0 0 500 1000 1500 2000 2500 3000 3500 4000 Wavelength (in nm) 28
The Future
The Future
Emerging Trends - Augmented Reality 30
Emerging Trends - Mobile SLAM 33 P. Tanskanen, K. Kolev, L. Meier, F. Camposeco, O. Saurer, M. Pollefeys : Live metric 3d reconstruction on mobile phones. (ICCV 2013)
Emerging Trends - Mobile SLAM 33 P. Tanskanen, K. Kolev, L. Meier, F. Camposeco, O. Saurer, M. Pollefeys : Live metric 3d reconstruction on mobile phones. (ICCV 2013)
Emerging Trends - Mobile SLAM 33 P. Tanskanen, K. Kolev, L. Meier, F. Camposeco, O. Saurer, M. Pollefeys : Live metric 3d reconstruction on mobile phones. (ICCV 2013)
Recommend
More recommend
