Signals, Information and Sampling Steve McLaughlin University of - PowerPoint PPT Presentation

“Signals, Information and Sampling” Steve McLaughlin University of Edinburgh, 16 th January, 2008 Page 1 of 49

“Signals, Information and Sampling” Steve McLaughlin “Signals, Information and Sampling” or Some new directions in signal processing and communications Steve McLaughlin IDCOM, School of Engineering & Electronics with thanks to Mike Davies, Bernie Mulgrew, John Thompson University of Edinburgh, 16 th January, 2008 Page 2 of 49

“Signals, Information and Sampling” Steve McLaughlin What are signals? A signal is a time (or space) varying quantity that can carry information. The concept is broad, and hard to define precisely. ( Wikipedia ) University of Edinburgh, 16 th January, 2008 Page 3 of 49

“Signals, Information and Sampling” Steve McLaughlin What is signal processing? Signal processing is the analysis, interpretation, and manipulation of signals. Signals of interest include sound, images, biological signals such as ECG, radar signals, and many others. Processing of such signals includes storage and reconstruction, separation of information from noise (e.g., aircraft identification by radar), compression (e.g., image compression), and feature extraction (e.g., speech-to-text conversion). ( Wikipedia ) University of Edinburgh, 16 th January, 2008 Page 4 of 49

“Signals, Information and Sampling” Steve McLaughlin What is information? I nformation theory is a branch of applied mathematics and engineering involving the quantification of information. Historically, information theory developed to find fundamental limits on compressing and reliably communicating data. Since its inception it has broadened to find applications in statistical inference, networks other than communication networks, biology, quantum information theory, data analysis , and other areas, although it is still widely used in the study of communication. ( Wikipedia ) University of Edinburgh, 16 th January, 2008 Page 5 of 49

“Signals, Information and Sampling” Steve McLaughlin What is informatics? I nformatics includes the science of information, the practice of information processing, and the engineering of information systems. Informatics studies the structure, behaviour, and interactions of natural and artificial systems that store, process and communicate information. I nformatics is broader in scope than information theory. ( Wikipedia ) University of Edinburgh, 16 th January, 2008 Page 6 of 49

“Signals, Information and Sampling” Steve McLaughlin Part I: Sparse Representations and Coding University of Edinburgh, 16 th January, 2008 Page 7 of 49

“Signals, Information and Sampling” Steve McLaughlin What are signals made of? The Frequency viewpoint (Fourier): Signals can be built from the sum of harmonic functions (sine waves) 1.5 1.5 1.5 1.5 1.5 1.5 Joseph Fourier 1 1 1 1 1 1 ∑ = φ ( ) ( ) x t c t 0.5 0.5 0.5 0.5 0.5 0.5 k k k 0 0 0 0 0 0 signal −0.5 −0.5 −0.5 −0.5 −0.5 −0.5 Harmonic functions −1 −1 −1 −1 −1 −1 Fourier coefficients −1.5 −1.5 −1.5 −1.5 −1.5 −1.5 50 50 50 50 50 50 100 100 100 100 100 100 150 150 150 150 150 150 200 200 200 200 200 200 250 250 250 250 250 250 University of Edinburgh, 16 th January, 2008 Page 8 of 49

“Signals, Information and Sampling” Steve McLaughlin Sampling and the digital revolution Today we are more familiar with discrete signals (e.g. audio files, digital images). This is thanks to: Whittaker–Kotelnikov–Shannon Sampling Theorem: “Exact reconstruction of a continuous-time signal from discrete samples is possible if the signal is bandlimited and the sampling frequency is greater than twice the signal bandwidth.” Sampling below this rate introduces aliasing University of Edinburgh, 16 th January, 2008 Page 9 of 49

“Signals, Information and Sampling” Steve McLaughlin Audio representations Which representation is best: time or frequency? 180 0.6 160 140 0.4 120 0.2 100 80 0 60 −0.2 40 20 −0.4 0 0.2 2 0.4 4 0.6 6 0.8 8 1 10 1.2 12 1.4 14 1.6 16 1.8 18 2 20 2.2 frequency time University of Edinburgh, 16 th January, 2008 Page 10 of 49

“Signals, Information and Sampling” Steve McLaughlin Audio representations Time and Frequency (Gabor) “Theory of Communication,” J. I EE (London) , 1946 “… a new method of analysing signals is presented in which time and frequency play symmetrical parts…” Frequency (Hz) a Gabor ‘atom’ Time (s) University of Edinburgh, 16 th January, 2008 Page 11 of 49

“Signals, Information and Sampling” Steve McLaughlin Gabor and audio coding Time and Frequency (Gabor) “Theory of Communication,” J. I EE (London) , 1946 “…In Part 3, suggestions are discussed for compressed transmission and reproduction of speech or music…” Modern audio coders owe as much to Gabor’s notion of time- frequency analysis as it does to Shannon’s paper of a similar title, two years later, that heralded the birth of information and coding theory. “A Mathematical Theory of Communication,” Bell System Technical Journal, 1948. C. E. Shannon University of Edinburgh, 16 th January, 2008 Page 12 of 49

“Signals, Information and Sampling” Steve McLaughlin Image representations … Space and Scale: the wavelet viewpoint: “Daubechies, Ten Lectures on Wavelets ,” SI AM 1992 Images can be built of sums of wavelets . These are multi- resolution edge-like (image) functions. University of Edinburgh, 16 th January, 2008 Page 13 of 49

“Signals, Information and Sampling” Steve McLaughlin Transform Sparsity What makes a good transform? 50 100 150 200 250 50 100 150 200 250 “TOM” image Wavelet Domain Good representations are efficient - Sparse! University of Edinburgh, 16 th January, 2008 Page 14 of 49

“Signals, Information and Sampling” Steve McLaughlin Coding signals of interest What is the difference between quantizing a signal/image in the transform domain rather than the signal domain? Compressed to 0.5 bits per pixel Compressed to 3 bits per pixel Compressed to 2 bits per pixel Compressed to 0.1 bits per pixel Compressed to 2 bits per pixel Compressed to 1 bits per pixel 50 50 50 50 50 50 50 50 50 50 50 50 50 50 50 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 250 250 250 250 250 250 250 250 250 250 250 250 250 250 250 50 50 50 50 50 100 100 100 100 100 150 150 150 150 150 200 200 200 200 200 250 250 250 250 250 50 50 50 50 50 100 100 100 100 100 150 150 150 150 150 200 200 200 200 200 250 250 250 250 250 50 50 50 50 50 100 100 100 100 100 150 150 150 150 150 200 200 200 200 200 250 250 250 250 250 Quantization in Tom’s nonzero Quantization in wavelet domain wavelet coefficients pixel domain University of Edinburgh, 16 th January, 2008 Page 15 of 49

“Signals, Information and Sampling” Steve McLaughlin Coding signals of interest An important question is: what are the signals of interest? If we digitize (via sampling) each signal is a point in a high dimensional vector space. e.g. a 5 Mega pixel camera image lives in a 5,000,000 dimensional space. What is a good signal model? Geometric Model I Consider the set of finite energy signals: the L 2 ball (an n-sphere ) . Coding can be done by covering the set with ε -balls. The L 2 ball is NOT a good signal model! Almost all signals look like this… University of Edinburgh, 16 th January, 2008 Page 16 of 49

“Signals, Information and Sampling” Steve McLaughlin Coding signals of interest Efficient transform domain representations implies that our signals of interest live in a much smaller set. These sets can be covered with much fewer ε -balls and require much fewer ‘bits’ to approximate. Sparse signal model L 2 ball (not sparse) University of Edinburgh, 16 th January, 2008 Page 17 of 49

“Signals, Information and Sampling” Steve McLaughlin Learning better representations Recent efforts have been targeted at learn better representations for a given set of signals, x(t): ∑ = ϕ x(t) c ( t ) k k k ϕ That is, learn dictionaries of functions that represent signals of ( t ) k interest with only a small number of significant coefficients, c k . For Audio (Abdallah & Plumbley, Proc. ICA 2001): For images (Olshausen and Field, Nature , 1996): University of Edinburgh, 16 th January, 2008 Page 18 of 49

“Signals, Information and Sampling” Steve McLaughlin Build bigger dictionaries Another approach is to try to build bigger dictionaries to provide more flexible descriptions. Consider the following test signal: Heisenberg’s uncertainty principle implies that a Time-Frequency analysis has: 0 0 either good time resolution and poor 0 frequency resolution 0 0 0 or good frequency resolution and 0 0 poor time resolution 0 0 0 0 0 0 0 100 20 200 300 40 400 60 500 600 80 700 100 800 900 120 1000 University of Edinburgh, 16 th January, 2008 Page 19 of 49