Graph Signal Processing for Image Coding & Restoration
.
Gene Cheung National Institute of Informatics 17th November, 2016
1 Dagstuhl 11/17/2016
Image Coding & Restoration . Dagstuhl 11/17/2016 1 - - PowerPoint PPT Presentation
Gene Cheung National Institute of Informatics 17 th November, 2016 Graph Signal Processing for Image Coding & Restoration . Dagstuhl 11/17/2016 1 Acknowledgement Collaborators: Y. Mao, Y. Ji (NII, Japan) W. Hu, P. Wan, W. Dai, J.
.
1 Dagstuhl 11/17/2016
2 Dagstuhl 11/17/2016
3 Dagstuhl 11/17/2016
4 Dagstuhl 11/17/2016
5 Dagstuhl 11/17/2016
2D DCT basis is set of outer-product of 1D DCT basis in x- and y-dimension.
1
N n n k
1
6
desired signal transform transform coeff. Compact signal representation
Dagstuhl 11/17/2016
7
[1] D. I. Shuman et al.,”The Emerging Field of Signal Processing on Graphs: Extending High-dimensional Data Analysis to Networks and other Irregular Domains,” IEEE Signal Processing Magazine, vol.30, no.3, pp.83-98, 2013.
example graph-signal
8
* Graph Signal Processing Workshop, Philadelphia, US, May 25-27, 2016. https://alliance.seas.upenn.edu/~gsp16/wiki/index.php?n=Main.Program
9
j j i i i
, ,
1 2 3 4
2 , 1
1 1 1 1 A
2 , 1 2 , 1
w w
1 2 1 D
2 , 1 2 , 1
w w
1 1
1 1 1 2 1 1 1 L
2 , 1 2 , 1 2 , 1 2 , 1
w w w w
*https://en.wikipedia.org/wiki/Second_derivative
2
h
4 3 2 : , 3
undirected graph
10
i i i
eigenvalue eigenvector 1st AC eigenvector
1 2 3 4 8
2 , 1
1 1
Dagstuhl 11/17/2016
11
eigenvalue eigenvector
2 / 1 2 / 1 2 / 1 2 / 1
1 1
*
[1] Wei Hu, Gene Cheung, Antonio Ortega, "Intra-Prediction and Generalized Graph Fourier Transform for Image Coding," IEEE Signal Processing Letters, vol.22, no.11, pp. 1913-1917, November 2015.
spectral clustering eigen-analysis of graph Laplacian, adjacency matrices graphical model, manifold learning, classifier learning Laplace- Beltrami
Laplace equation
Max cut, graph transformation
12 Dagstuhl 11/17/2016
13 Dagstuhl 11/17/2016
14
GFT
[1] G. Shen et al., “Edge-adaptive Transforms for Efficient Depth Map Coding,” IEEE Picture Coding Symposium, Nagoya, Japan, December 2010. [2] M. Maitre et al., “Depth and depth-color Coding using Shape-adaptive Wavelets,” Journal of Visual Communication and Image Representation, vol.21, July 2010, pp.513-522.
Shape-adaptive wavelets can also be done.
Transform Representation Transform Description
Karhunen-Loeve Transform (KLT) “Sparsest” signal representation given available data / statistical model Can be expensive (if poorly structured) Discrete Cosine Transform (DCT) non-sparse signal representation across sharp boundaries little (fixed transform) Graph Fourier Transform (GFT) minimizes the total rate of signal’s transform representation & transform description
15
[1] Wei Hu, Gene Cheung, Antonio Ortega, Oscar Au, "Multiresolution Graph Fourier Transform for Compression of Piecewise Smooth Images," IEEE Transactions on Image Processing, vol.24, no.1, pp.419-433, January 2015.
16
5 10 15 20 25 30 35 40 45 50 500 1000 1500 2000 2500 3000 3500
16
Rate of transform coefficient vector Rate of transform description T
where,
k-th
smooth jump non-zero mean random var.
17
1 2
k-1
k N
k k
, 1
1
non-zero mean RV kth row mean vector
Dagstuhl 11/17/2016
k-th row
1 1 1 1 1
2 2 2 1
g g T
m E bb
18
(k-1)-th row k-th row
1 1 1
19
Dagstuhl 11/17/2016
HR-DCT: 6.8dB HR-SGFT: 5.9dB SAW: 2.5dB MR-SGFT: 1.2dB
20
HR-DCT HR-SGFT MR-GFT
21 Dagstuhl 11/17/2016
red: WGFT blue: UGFT
22 Dagstuhl 11/17/2016
chain code with symbols {l, s, r}.
compute symbol probabilities for arithmetic coding.
[2] Amin Zheng, Gene Cheung, Dinei Florencio, "Context Tree based Image Contour Coding using A Geometric Prior," accepted to IEEE Transactions on Image Processing, November 2016. [1] I. Daribo, G. Cheung, D. Florencio, “Arbitrarily Shaped Sub-block Motion Prediction in Depth Video Compression using Arithmetic Edge Coding," IEEE Trans on Image Processing, Nov 2014.
23
24 Dagstuhl 11/17/2016
: prediction residuals
Boundary pixel (predictor) Predicted pixels 𝑦𝑗
[1] W. Hu et al., “Intra-Prediction and Generalized Graph Fourier Transform for Image Coding,” IEEE Signal Processing Letters, vol.22, no.11, pp. 1913-1917, November, 2015.
25
Class k Class l
26
class1 class3 class0 class0 class2 bin average approximation error
Dagstuhl 11/17/2016
27
[1] J. Han et al., “Jointly Optimized Spatial Prediction and Block Transform for video and Image Coding,” IEEE Transactions on Image Processing, vol.21, no.4, April 2012, pp.1874-1884.
inaccuracy of intra-prediction discontinuities within signal Variance of approx. error
28 Dagstuhl 11/17/2016
29
analysis filter synthesis filter
30
sub-matrix for 2 partitions KL divergence bipartite graph Laplacian
b b KL L
b
2 , 1
[3] Jin Zeng, Gene Cheung, Antonio Ortega, "Bipartite Subgraph Decomposition for Critically Sampled Wavelet Filterbanks on Arbitrary Graphs," IEEE International Conference on Acoustics, Speech and Signal Processing, Shanghai, China, March, 2016. [2] S. Narang and A. Ortega, “Compact support biorthogonal wavelet filterbanks for arbitrary undirected graphs,” IEEE Transactions on Signal Processing, vol. 61, no. 19, pp. 4673–4685, Oct 2013. [1] S. Narang and A. Ortega, “Perfect reconstruction two-channel wavelet filter banks for graph structured data,” IEEE Transactions on Signal Processing, vol. 60, no. 6,pp. 2786–2799, June 2012.
31 Dagstuhl 11/17/2016
32
k k k j i j i j i T
2 , 2 ,
noise desired signal
2 2 T x
smoothness prior fidelity term signal smooth in nodal domain signal contains mostly low graph freq.
1 2 3 4
2 , 1
1 1
[1] P. Milanfar, “A Tour of Modern Image Filtering: New Insights and Methods, Both Practical and Theoretical,” IEEE Signal Processing Magazine, vol.30, no.1, pp.106-128, January 2013.
33
[1] W. Hu, G. Cheung, M. Kazui, "Graph-based Dequantization of Block-Compressed Piecewise Smooth Images," IEEE Signal Processing Letters, vol.23, no.2, pp.242-246, February 2016.
2 2 T x
pixel intensity difference pixel location difference
2 2 2 2 2 1 2 2 ,
j i j i j i
34
T
G
T T 1 2 1
T
Optimal regularizer Non-local self-similarity and MMSE formulation
T
G
T 1
·
n N n n
f f
G
T 1 2
i N
ij i j ij
w d
2 2 2 ij i j
d v v
feature function vector distance edge weight metric space
[1] J. Pang, G. Cheung, "Graph Laplacian Regularization for Inverse Imaging: Analysis in the Continuous Domain," submitted to IEEE Transactions on Image Processing, April 2016.
35
I
Original Noisy, 16.48 dB K-SVD, 26.84 dB BM3D, 27.99 dB PLOW, 28.11 dB OGLR, 28.35 dB
Dagstuhl 11/17/2016
36
I
Original Noisy, 18.66 dB BM3D, 33.26 dB NLGBT, 33.41dB OGLR, 34.32 dB
[1] W. Hu et al., "Depth Map Denoising using Graph-based Transform and Group Sparsity," IEEE International Workshop on Multimedia Signal Processing, Pula (Sardinia), Italy, October, 2013.
37 Dagstuhl 11/17/2016
38
i i i
DCT Coefficients 8x8 pixel block quantization parameter DCT
i i i i i
[1] A. Zakhor, “Iterative procedures for reduction of blocking effects in transform image coding,” IEEE Transactions on Circuits and Systems for Video Technology,, vol. 2, no. 1, pp. 91–95, Mar 1992. [2] K. Bredies and M. Holler, “A total variation-based JPEG decompression model,” SIAM J. Img. Sci., vol. 5, no. 1, pp. 366–393, Mar. 2012. [3] H. Chang, M. Ng, and T. Zeng, “Reducing artifacts in jpeg decompression via a learned dictionary,” IEEE Transactions on Signal Processing,,vol. 62, no. 3, pp. 718–728, Feb 2014.
39
[1] W. Hu, G. Cheung, M. Kazui, "Graph-based Dequantization of Block-Compressed Piecewise Smooth Images," IEEE Signal Processing Letters, vol.23, no.2, pp.242-246, February 2016.
2 2 T x
pixel intensity difference pixel location difference
Comments: 1. L is NOT normalized. 2. Why works well for PWS signals?
2 2 2 2 2 1 2 2 ,
j i j i j i
40
B A
,
B j A i j i
, ,
A i i i
,
i T T
f
f
T T T
min cut min normalized cut
[1] J. Shi and J. Malik, “Normalized cuts and image segmentation,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 22, no. 8, pp. 888–905,
41
1 n v *
T T T
2 / 1 1 2 / 1
Rayleigh quotient
2
k k k
n 2 2 T x
candidate objective
Dagstuhl 11/17/2016
42
[1] Xianming Liu, Gene Cheung, Xiaolin Wu, Debin Zhao, "Random Walk Graph Laplacian based Smoothness Prior for Soft Decoding
projection of signal x to D1/2, then Ln
43
Graph Laplacian Symmetric Normalized DC e-vector Combinatorial Yes No Yes Symmetrically Normalized Yes Yes No Random Walk No Yes Yes Doubly Stochastic [1] Yes Yes Yes
[1] A. Kheradmand and P. Milanfar, “A general framework for regularized, similarity-based image restoration,” IEEE Transactions on Image Processing, vol. 23, no. 12, pp. 5136–5151, Dec 2014.
44
~
2 2 1 min
k k k T
d
[1] Xianming Liu, Gene Cheung, Xiaolin Wu, Debin Zhao, "Random Walk Graph Laplacian based Smoothness Prior for Soft Decoding
45
sparsity prior graph-signal smoothness prior quantization bin constraint fidelity term
graph-signal, code vector
Dagstuhl 11/17/2016
46 Dagstuhl 11/17/2016
47
48
49
50
51
52 Dagstuhl 11/17/2016
53 Dagstuhl 11/17/2016
[2] B. Renoust et al., "Estimation of Political Leanings via Graph-Signal Restoration," submitted to IEEE International Conference on Acoustics, Speech and Signal Processing, New Orleans, USA, March, 2017 [1] G. Cheung et al., "Robust Semi-Supervised Graph Classifier Learning with Negative Edge Weights," submitted to special issue on “Graph Signal Processing” in IEEE Journal on Selected Topics on Signal Processing, Nov, 2016.
55 Dagstuhl 11/17/2016