Having Fun with OpenCV Instructor - Simon Lucey 16-423 - Designing Computer Vision Apps
?
2010
2015
Ideal Von Neumann Processor • each cycle, CPU takes data from registers, does an operation, and puts the result back • load/store operations (memory ←→ registers) also take one cycle • CPU can do different operations each cycle output of one operation can be input to next ✲ time op1 ✲ ✲ ✲ op2 ✲ ✲ ✲ op3 ✲ ✲ ✲ • CPU’s haven’t been this simple for a long time! 4 Taken from http://people.maths.ox.ac.uk/gilesm/cuda/lecs/lec0.pdf
CPU clock is stuck!!!! • CPU clock stuck at about 3GHz since 2006 due to high power consumption (up to 130W per chip) • chip circuitry still doubling every 18-24 months • ⇒ more on-chip memory and MMU (memory management units) • ⇒ specialised hardware (e.g. multimedia, encryption) ⇒ multi-core (multiple CPU’s on one chip) • peak performance of chip still doubling every 18-24 months 5 Taken from http://people.maths.ox.ac.uk/gilesm/cuda/lecs/lec0.pdf
ASICs for Low Energy • Application Specific Integrated Circuits (ASIC) • ASICs are perfect for targeting a specific application domain. • Inherently low-power as they are “frozen in silicon” for a specific application domain (e.g. graphics cards, ethernet cards, DSPs, etc.). • Drawbacks, • incredibly expensive to develop. • time consuming and resource-intensive to develop. • Positives, • Extremely energy efficient. 6
Example: Adding Numbers 1 - 10 1 + 1 = 2 1 + 2 = 3 ……… 10 + 10 = 20 7
System on a Chip (SoC) • SoCs attempt to find balance between energy and programmability. • Designed with emphasis on low power consumption. • SOC shares the same system bus with CPU, GPU and DSP. • Therefore has much lower memory bandwidth. • Useful for computer vision algorithm design as one can switch between CPU and GPU with little memory overhead. • Not possible on conventional architecture. • More on this later….. (Taken from K. Cheng, Y. Wang “Using Mobile GPU for General-Purpose Computing – A Case Study of Face Recognition on Smartphones”) 8
Battle of Two Platforms 9
Battle of Two Platforms 10
Battle of Two Platforms 11
When it comes to Computer Vision R&D • Class is not about which platform is better. • Instead, we want to choose ONE mobile platform and explore it deeply. • For the purposes of this class this is Apple’s iOS. • Nearly all concepts are easily transferable to Android. 12
Today • Philosophy to Mobile Computer Vision R&D • Getting started with OpenCV.
Applications of Computer Vision “Face Recognition” “Pose Estimation” “Body Tracking” “Speech Reading” “Palm Recognition” “Car Tracking”
Applications of Computer Vision “Face Recognition” “Pose Estimation” “Body Tracking” “Speech Reading” “Palm Recognition” “Car Tracking”
Balancing Power versus Perception 15
Algorithm Software Architecture SOC Hardware
Correlation Filters with Limited Boundaries Hamed Kiani Galoogahi Terence Sim Simon Lucey Algorithm Istituto Italiano di Tecnologia National University of Singapore Carnegie Mellon University Genova, Italy Singapore Pittsburgh, USA hamed.kiani@iit.it tsim@comp.nus.edu.sg slucey@cs.cmu.edu Abstract Correlation filters take advantage of specific proper- ties in the Fourier domain allowing them to be estimated efficiently: O ( ND log D ) in the frequency domain, ver- sus O ( D 3 + ND 2 ) spatially where D is signal length, and N is the number of signals. Recent extensions to cor- Software (a) (b) relation filters, such as MOSSE, have reignited interest of their use in the vision community due to their robustness and attractive computational properties. In this paper we demonstrate, however, that this computational efficiency 1 comes at a cost. Specifically, we demonstrate that only D proportion of shifted examples are unaffected by boundary effects which has a dramatic effect on detection/tracking � � (c) (d) performance. In this paper, we propose a novel approach to correlation filter estimation that: (i) takes advantage of Figure 1. (a) Defines the example of fixed spatial support within inherent computational redundancies in the frequency do- the image from which the peak correlation output should occur. main, (ii) dramatically reduces boundary effects, and (iii) (b) The desired output response, based on (a), of the correlation is able to implicitly exploit all possible patches densely ex- filter when applied to the entire image. (c) A subset of patch ex- tracted from training examples during learning process. Im- amples used in a canonical correlation filter where green denotes pressive object tracking and detection results are presented a non-zero correlation output, and red denotes a zero correlation in terms of both accuracy and computational efficiency. output in direct accordance with (b). (d) A subset of patch ex- Architecture amples used in our proposed correlation filter. Note that our pro- posed approach uses all possible patches stemming from different 1. Introduction parts of the image, whereas the canonical correlation filter simply employs circular shifted versions of the same single patch. The Correlation between two signals is a standard approach central dilemma in this paper is how to perform (d) efficiently in to feature detection/matching. Correlation touches nearly the Fourier domain. The two last patches of (d) show that D − 1 T every facet of computer vision from pattern detection to ob- patches near the image border are affected by circular shift in our ject tracking. Correlation is rarely performed naively in the method which can be greatly diminished by choosing D << T , where D and T indicate the length of the vectorized face patch in spatial domain. Instead, the fast Fourier transform (FFT) (a) and the whole image in (a), respectively. affords the efficient application of correlating a desired tem- plate/filter with a signal. Correlation filters, developed initially in the seminal proach is that it attempts to learn the filter in the frequency work of Hester and Casasent [15], are a method for learning domain due to the efficiency of correlation in that domain. a template/filter in the frequency domain that rose to some prominence in the 80s and 90s. Although many variants Interest in correlation filters has been reignited in the vi- have been proposed [15, 18, 20, 19], the approach’s central sion world through the recent work of Bolme et al. [5] on SOC Hardware tenet is to learn a filter, that when correlated with a set of Minimum Output Sum of Squared Error (MOSSE) correla- training signals, gives a desired response, e.g. Figure 1 (b). tion filters for object detection and tracking. Bolme et al.’s Like correlation, one of the central advantages of the ap- work was able to circumvent some of the classical problems
Algorithm Software Ax = b Architecture SOC Hardware
Algorithm Software Architecture Hardware
Algorithm Software Architecture Hardware
Algorithm Software SIMD (Single Instruction, Multiple Data) Architecture SOC Hardware
Algorithm 4-way x + Software SIMD (Single Instruction, Multiple Data) � � Architecture � (length 2, 4, 8, …) vectors of integers or floats � Names: MMX, SSE, SSE2, … � SOC Hardware � �
Algorithm Software Architecture SOC Hardware
Algorithm Software Architecture SOC Hardware APIs in the current versions of OpenGL ES do not have the “scatter”
Algorithm Software Architecture SOC Hardware APIs in the current versions of OpenGL ES do not have the “scatter”
Algorithm Software Architecture SOC Hardware
Algorithm Optimize Software Architecture SOC Hardware
Algorithm Software Architecture SOC Hardware Optimize
Algorithm Optimize Software Architecture SOC Hardware
MATLAB OpenCV
MATLAB OpenCV
MATLAB OpenCV
Some Insights for Mobile CV • Very difficult to write the fastest code. • When you are prototyping an idea you should not worry about this, but • You have to be aware of where bottle necks can occur. • This is what you will learn in this course. • Highest performance in general is non-portable. • If you want to get the most out of your system it is good to go deep. • However, options like OpenCV are good when you need to build something quickly that works. • To build good computer vision apps you need to know them algorithmically. • Simply knowing how to write fast code is not enough. • You need to also understand computer vision algorithmically. • OpenCV can be dangerous here. Some insights taken from Markus Püschel’s lectures on “How to Write fast Numerical Code”.
Today • Philosophy to Mobile Computer Vision R&D • Getting started with OpenCV.
Recommend
More recommend
