EVC: Image Processing & Computer Vision http:// - - PDF document

▶
evc image processing computer vision
SMART_READER_LITE
LIVE PREVIEW

EVC: Image Processing & Computer Vision http:// - - PDF document

Feb 09, 2023 226 likes •634 views

04.03.2013 Einfhrung in Visual Computing (EVC) Image Processing & Computer Vision 186.822 VU 5.0 6 ECTS Robert Sablatnig 1 Robert Sablatnig, Computer Vision Lab, EVC 2: Introduction EVC: Image Processing & Computer Vision http://

slide-1
SLIDE 1

04.03.2013 1 Einführung in Visual Computing (EVC) Image Processing & Computer Vision

186.822 VU 5.0 6 ECTS Robert Sablatnig

1 Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction

EVC: Image Processing & Computer Vision

  • Content:
  • What are the basic concepts of Image Processing and Computer

http://www.caa.tuwien.ac.at/cvl/teaching/sommersemester/evc

Vision and how are they used in applications? The course answers these questions by describing the creation of digital images using digital cameras and the subsequent steps in order to derive information kept in digital images automatically.

  • A closer look is taken into classical image processing techniques like

image enhancement and compression.

  • The next step consists in the development of digital filters and

segmentation techniques in order to be able to extract specific information.

  • Interest Points Computational Photography, 3D and motion are

further topics.

  • Application of Algebra and Analysis in reality

2 Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction

slide-2
SLIDE 2

04.03.2013 2

Logistics

  • Lectures: 13:00 ‐ 15:00
  • Instructors: Robert Sablatnig (VO) and

g ( ) Sebastian Zambanini (UE)

  • Textbook: 4 A4 pages available at Lectures and Website
  • Further Reading:
  • Richard Szeliski, Computer Vision: A Modern Approach

http://szeliski.org/Book/

  • Sonka Hlavac Boyle: Image Processing Analysis and

Sonka, Hlavac, Boyle: Image Processing, Analysis, and Machine Vision, 2nd Edition

  • Webpage:

http://www.caa.tuwien.ac.at/cvl/teaching/sommersemester/evc

Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction 3

Readings

4 Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction

slide-3
SLIDE 3

04.03.2013 3

Introduction: What is Image Processing? Computer Graphics vs. Computer Vision

6 Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction

slide-4
SLIDE 4

04.03.2013 4

Categorization

  • Image Processing
  • Manipulation of Image Data,

p f g ,

  • Like removal of Noise, Correction of Sharpness on digital

images.

  • Computer Vision
  • Generation of non‐graphical Data from images,
  • Like Character‐and Text Recognition, Segmentation of images

into „interesting“ parts, Detection of lines and corners. „ g p ,

  • Computer Graphics
  • Generation of Images from non‐graphical data,
  • like bar charts, 3d graphics „VR“ in real time, graphical outputs

7 Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction

Categorization

  • Image Editing: Manipulation of Images (e.g. Photoshop)
  • Visually
  • Visually
  • Interactive
  • User‐defined Parameters
  • Image Processing: Mathematical algorithmic processes
  • Image enhancement

g

  • Image transformation (geometric)
  • Image compression
  • Image segmentation

8 Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction

slide-5
SLIDE 5

04.03.2013 5

Kategorisierung

9 Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction

Example Image Processing: Filter (Noise Removal)

10 Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction

slide-6
SLIDE 6

04.03.2013 6

Example Image Processing: Image Enhancement

Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction 11

Example: Image Restoration

12 Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction

slide-7
SLIDE 7

04.03.2013 7

Example: Special Effects

13 Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction

Difference: Pattern Recognition – Image Processing?

  • Pattern Recognition:
  • Classification of Patterns into a (finite) number of pre‐defined

( ) p classes

  • like 2‐dimensional patterns, OCR
  • Standard book: Duda and Hart 1973, "Pattern Classification and

Scene Analysis"

  • Image Processing:
  • Processing of an image to get a new image that is better suited

g g g g for a specific task.

  • Image enhancement, image transformation, image

compression, image segmentation, image restauration…

  • Standard book: Rosenfeld and Kak 1982, "Digital Picture

Processing", 2nd Edition

14 Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction

slide-8
SLIDE 8

04.03.2013 8

Example Pattern Recognition

15 Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction

Example Pattern Recognition

16 Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction

slide-9
SLIDE 9

04.03.2013 9

Examples for Pattern Recognition

17 Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction

Example: Computer Vision

  • Face Detection

18 Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction

slide-10
SLIDE 10

04.03.2013 10

Google Street View

Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction 19

Google Street View

Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction 20

slide-11
SLIDE 11

04.03.2013 11

Introduction: What is Computer Vision? Computer Vision

  • Vision is derived from Human

Vision (Human Visual System) ( y )

  • Humans „see“ in 3 Dimensions

=> Computer Vision has 3d components

  • Evolution millions of years: Human

visual system not faultless

=> if human visual system is not faultless how can we expect from a machine that it is?

Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction 22

slide-12
SLIDE 12

04.03.2013 12

What is Computer Vision ?

"Computer Vision describes the automatic deduction of the structure and the properties of a (possible dynamic) three‐ p p f (p y ) dimensional world from either a single or multiple two‐ dimensional images of the world" ‐ Vishvjit S. Nalwa: A guided tour of computer vision. Addison‐Wesley 1993

  • Images:

Color or Grayscale C Fi d bl

  • Camera:

Fixed or movable

Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction 23

Computer Vision – Industry Related

  • Computer Vision is an exciting new

research area that studies how to make computers efficiently perceive, process, and understand visual data such as images and videos. The ultimate goal is for computers to emulate the striking perceptual capability of human eyes and brains, or even to surpass and assist the human in certain ways. – Microsoft Research

Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction 24

slide-13
SLIDE 13

04.03.2013 13

Computer Vision

  • At least three goals:

1

Understand biological visual systems

1.

Understand biological visual systems

2.

Build machines that see

3.

Understand fundamental processes of seeing

Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction 25

Computer Vision

We still do not know

  • Is vision a well organized process with fundamental principles or
  • a bag of tricks

Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction 26

slide-14
SLIDE 14

04.03.2013 14

Goals and Applications of Computer Vision

  • It is not the goal of Computer Vision to develop a robot that is

similar to humans [Whitney86] [ y ]

  • Goal is to surpass and assist humans
  • Applications:
  • Automation (Assembly line)
  • Inspection (Measuring of Parts)
  • Remote Sensing (Maps)
  • Human ‐ Computer Interfaces
  • Systems for Disabled
  • Many more……

Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction 27

Computer Vision vs. Human Vision

  • Why not simply copy human vision researched by

neurophysiologists, psychologists, and psychophysics? [Levine91] p y g , p y g , p y p y [ ]

  • Eye research is finished – Human Vision research is not!
  • Seeing is not only a process within the eye – eye is only

producing images formed to “impressions” by the brain

  • => Beginning of Computer Vision in the area of Artificial

I lli Intelligence

Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction 28

slide-15
SLIDE 15

04.03.2013 15

Computer Vision vs. Seeing

  • Seeing has adopted itself to environment und therefore not

faultless! Is Seeing an integral part of intelligence?

  • Is Seeing an integral part of intelligence?
  • Do we see reality – or what we want to see?
  • Is Seeing and Thinking separable?

Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction 29

It’s Just An Illusion: Visual Illusions

  • Classical optical illusions

Zöllner Illusion (1860) Poggendorf Illusion (1860)

Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction 30

slide-16
SLIDE 16

04.03.2013 16

Visual Illusions

  • Classical optical illusions

Helmholtz Squares (1866) Müller-Lyer Illusion (1860)

Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction 31

Visual Illusions

  • Non existing 3D objects:

Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction 32

slide-17
SLIDE 17

04.03.2013 17

Perspective Illusions by Julian Beever

Make Poverty History Babyfood...

Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction 33 Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction 34

slide-18
SLIDE 18

04.03.2013 18

M.C. Escher Ambigious Interpretations

Indian vs. Inuit Young/Old Lady

Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction 36

slide-19
SLIDE 19

04.03.2013 19

Rotary Effects

Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction 37

Are these phenomena caused by manipulation of the visual system by unreal images?

  • After all, if we cannot believe what we see, what are we to

believe?

  • Absolute faith in human visual system is not justified for 2‐

dimensional images!

 Either: 3d images ‐> real world  or: right limitations of scene features (perspective  or: right limitations of scene features (perspective,

lighting, direction, etc.)

Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction 38

slide-20
SLIDE 20

04.03.2013 20

Are these phenomena caused by manipulation of the visual system by unreal images?

Every image is an image of an object, which is understandable

  • nly to those who know about its origins and are able to create a

y g corresponding image in their imagination (Helmholtz, 1910)

Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction 39

History

  • Serious attempts to create computer vision systems have now a

history of 40‐50 years. y y

  • First digital image 1964 (Mariner 4)
  • Focus on Sensors  Digital Image Processing
  • Analysis Focus  Computer Vision

Mariner 4: First the first close‐up image ever taken

  • f Mars 1964

Source: NASA Source: NASA

40 Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction

slide-21
SLIDE 21

04.03.2013 21

Computer Vision

The systems today are still exceedingly limited in their performance  considerable room for improvement p p

41 Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction

Future Challenges of Computer Vision

  • Where do the innovations come from?
  • 1. Hardware

Wavi Xtion Kinect

  • 2. Algorithms/Software

Kinect

http://www.gigapixel.com

42 Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction

slide-22
SLIDE 22

04.03.2013 22

Hardware

  • First time that HW is no longer a real limitation !!
  • Processing
  • Image Resolution
  • Storage
  • Internet
  • Mobile Devices
  • Networks of cameras

43 Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction

Processing

  • Moore’s Law still holds!
  • Multi‐core CPUs
  • Highly Parallel  GPUs (+ Software eg. Cuda)

44 Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction

slide-23
SLIDE 23

04.03.2013 23

Processing

  • What to expect:
  • Image Processing (Feature Extraction) will be instantaneous
  • Image Processing (Feature Extraction) will be instantaneous
  • Real‐time Libs: Basic algorithms (IPP,Cuda …)
  • Real time vision (cf. Real time Rendering)
  • Parallelization (GPU implementation) is a feature of an algorithm

45 Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction

Resolution

Ever growing resolution:

  • 1975: 100 x 100 = 0 01 MP
  • 1975: 100 x 100 = 0.01 MP
  • 2009: 13.280 x 9.184 Pixel = 120 MP
  • UltraCamx: 216 MP

Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction 46

slide-24
SLIDE 24

04.03.2013 24

What can we do with that?

Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction 47

Some Questions to Tackle

  • How/What shall we sample in space, time,

wavelengths, polarization, ….? g , p ,

48 Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction

slide-25
SLIDE 25

04.03.2013 25

Some Questions to Tackle

Optimal sampling strategies in 3D/4D …

  • No constraints of view‐points
  • No constraints of view points
  • Multiple Images  Redundancy
  • Control the illumination of each pixel
  • How real are images?

49 Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction

Storage

  • We have huge disks and we fill them
  • A color VGA image ~1 MB
  • A color VGA image 1 MB
  • Every 10 second 1 image

 8.4 GB a day  240 TB a Life

  • Can we index that?  Algorithms

50 Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction

slide-26
SLIDE 26

04.03.2013 26

Internet

Huge repository of images

  • Flickr:

Aug 2011 ~ 6 Billion Photos On line

  • Aug. 2011 ~ 6 Billion Photos On‐line
  • 4.5 million photo added per day
  • YouTube:
  • 65.000 new Videos a day
  • 1 trillion video playbacks
  • 20% of Internet Traffic
  • Facebook:
  • 600 Million users
  • 3 Billion photos added per month

What can we do with these images?

Source: Internet 2011 in numbers

51 Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction

What is Vision?

  • What does it mean, to see? “to know what is where by looking”.
  • How to discover from images what is present in the world where
  • How to discover from images what is present in the world, where

things are, what actions are taking place.

from Marr, 1982

52 Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction

slide-27
SLIDE 27

04.03.2013 27

The Importance of Images

  • Some images are more important

than others

  • 100 million $

“Dora Maar au Chat” Pablo Picasso, 1941

53 Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction

Where is now Computer Vision? Where is now Computer Vision? (only a few examples)

slide-28
SLIDE 28

04.03.2013 28

Assisted Driving

Pedestrian and car detection

meters meters Ped Ped Car

Lane detection

Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction 55

  • Collision warning systems

with adaptive cruise control,

  • Lane departure warning

systems,

  • Rear object detection

systems,

Iris Recognition

Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction 56

JOHN DAUGMAN

http://www.cl.cam.ac.uk/~jgd1000/iriscollage.jpg

slide-29
SLIDE 29

04.03.2013 29

Image Stitching

Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction 57

Image Stitching

Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction 58

Brown, Lowe, 2007

slide-30
SLIDE 30

04.03.2013 30

Photo Turism

PhotoSynth

Snavely et al. 2006

Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction 59

(Goesele et al. 2007).

Finding Paths through the World's Photos

Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction 60

slide-31
SLIDE 31

04.03.2013 31

Why is Vision hard – The Plenoptic Function The Structure of Ambient Light

62 Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction

slide-32
SLIDE 32

04.03.2013 32

The Structure of Ambient Light

63 Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction

The Plenoptic Function

Adelson & Bergen, 91 The intensity P can be parameterized as:

P ( t,  Vx, Vy, Vz)

“The complete set of all convergence points constitutes the permanent possibilities of vision.” Gibson

Image coordinates (sperical) Color Time 3D space

64 Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction

slide-33
SLIDE 33

04.03.2013 33

Measuring the Plenoptic Function

  • “The significance of the plenoptic function is this: The

world is made of 3D objects but these objects do not world is made of 3D objects, but these objects do not communicate their properties directly to an observer. Rather, the objects fill the space around them with the pattern of light rays that constitutes the plenoptic function, and the observer takes samples from this function.” Adelson & Bergen 91.

  • function. Adelson & Bergen 91.

65 Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction

Measuring the Plenoptic Function

Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction 66

Why is there no picture appearing on the paper?

slide-34
SLIDE 34

04.03.2013 34

Measuring the Plenoptic Function

  • Light rays from many different parts of the scene strike the same

point on the paper. p p p

Forsyth & Ponce

Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction 67

Camera Obscura

68

slide-35
SLIDE 35

04.03.2013 35

Measuring the Plenoptic Function

The camera obscura The pinhole camera

Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction 69

Camera Obscura

  • Latin:
  • Camera for "vaulted

chamber/room"

  • obscura for "dark"
  • together "darkened

chamber/room“ "When images of illuminated objects penetrate through a small

  • When images of illuminated objects ... penetrate through a small

hole into a very dark room ... you will see [on the opposite wall] these objects in their proper form and color, reduced in size ... in a reversed position, owing to the intersection of the rays". ‐ Da Vinci

Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction 70

http://www.acmi.net.au/AIC/CAMERA_OBSCURA.html (Russell Naughton)

slide-36
SLIDE 36

04.03.2013 36

Camera Obscura

  • Used to observe eclipses (eg., Bacon, 1214‐1294)
  • By artists (eg Vermeer)
  • By artists (eg., Vermeer).

71 Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction

Camera Obscura

Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction 72

slide-37
SLIDE 37

04.03.2013 37

Camera Obscura

Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction 73

Jetty at Margate England, 1898.

Pinhole Camera

  • Simple Model of Camera Obscura: Pinhole camera
  • Very small hole (aperture ~ 0), Light passes through the hole

and forms image on back (upside down and flipped)

74 Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction

slide-38
SLIDE 38

04.03.2013 38

Pinhole Camera

  • Abstract camera model ‐ box with a small hole in it
  • Pinhole cameras work in practice
  • Pinhole cameras work in practice

75 Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction

Commercial Pinhole Cameras

Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction 76

slide-39
SLIDE 39

04.03.2013 39

Playing with Pinholes

Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction 77

Effect of Pinhole Size

Wandell, Foundations of Vision, Sinauer, 1995

78 Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction

slide-40
SLIDE 40

04.03.2013 40

Limits of Pinhole Cameras

  • A picture of a filament taken with a pinhole camera. In the image
  • n the left, the hole was too big (blurring), and in the image on

, g ( g), g the right, the hole was too small (diffraction).

Ruechardt, 1958

79 Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction

Pinhole Camera Images with Variable Aperture

  • Why not making the aperture

as small as possible? l h h h

2 mm 1 mm 0.6 mm 0.35 mm

  • Less light gets through
  • Diffraction effect

80 Robert Sablatnig, Computer Vision Lab, EVC‐2: Introduction

0.15 mm 0.07 mm