Image Reconstruction with Predictive Filter Flow Shu Kong, Charless Fowlkes Dept. of Computer Science, University of California, Irvine { skong2, fowlkes } @ics.uci.edu [Project Page], [Github], [Slides], [Poster] Abstract ill-posed and massively under-constrained. Many contem- porary techniques to inverse problems have focused on reg- ularization techniques which are amenable to computational We propose a simple, interpretable framework for solv- optimization. While such approaches are interpretable as ing a wide range of image reconstruction problems such as Bayesian estimators with particular choice of priors, they denoising and deconvolution. Given a corrupted input im- are often computationally expensive in practice [13, 45, 2]. age, the model synthesizes a spatially varying linear filter Alternately, data-driven methods based on training deep which, when applied to the input image, reconstructs the convolutional neural networks yield fast inference but lack desired output. The model parameters are learned using interpretability and guarantees of robustness [48, 61]. In supervised or self-supervised training. We test this model this paper, we propose a new framework called Predictive on three tasks: non-uniform motion blur removal, lossy- Filter Flow that retains interpretability and control over the compression artifact reduction and single image super res- olution. We demonstrate that our model substantially out- resulting reconstruction while allowing fast inference. The performs state-of-the-art methods on all these tasks and is proposed framework is directly applicable to a variety of significantly faster than optimization-based approaches to low-level computer vision problems involving local pixel deconvolution. Unlike models that directly predict output transformations. pixel values, the predicted filter flow is controllable and in- As the name suggests, our approach is built on the no- terpretable, which we demonstrate by visualizing the space tion of filter flow introduced by Seitz and Baker [44]. In of predicted filters for different tasks. 1 filter flow pixels in a local neighborhood of the input im- age are linearly combined to reconstruct the pixel centered at the same location in the output image. However, unlike 1. Introduction convolution, the filter weights are allowed to vary from one spatial location to the next. Filter flows are a flexible class Real-world images are seldom perfect. Practical en- of image transformations that can model a wide range of gineering trade-offs entail that consumer photos are often imaging effects (including optical flow, lighting changes, blurry due to low-light, camera shake or object motion, lim- non-uniform blur, non-parametric distortion). The original ited in resolution and further degraded by image compres- work on filter flow [44] focused on the problem of estimat- sion artifacts introduced for the sake of affordable transmis- ing an appropriately regularized/constrained flow between a sion and storage. Scientific applications such as microscopy given pair of images. This yielded convex but impractically or astronomy, which push the fundamental physical limita- large optimization problems (e.g., hours of computation to tions of light, lenses and sensors, face similar challenges. compute a single flow). Instead of solving for an optimal Recovering high-quality images from degraded measure- filter flow, we propose to directly predict a filter flow given ments has been a long-standing problem for image analysis an input image using a convolutional neural net (CNN) to and spans a range of tasks such as blind-image deblurring regress the filter weights. Using a CNN to directly predict a [4, 30, 13, 45], compression artifact reduction [46, 35], and well regularized solution is orders of magnitude faster than single image super-resolution [41, 59]. expensive iterative optimization. Such image reconstruction tasks can be viewed mathe- Fig. 1 provides an illustration of our overall framework. matically as inverse problems [50, 22], which are typically Instead of estimating the flow between a pair of input im- ages, we focus on applications where the model predicts 1 Due to that arxiv limits the size of files, we put high-resolution figures, both the flow and the transformed image. This can be as well as a manuscript with them, in the project page. 1
Figure 1: Overview of our proposed framework for Predictive Filter Flow which is readily applicable to various low-level vision prob- lems, yielding state-of-the-art performance for non-uniform motion blur removal, compression artifact reduction and single image super- resolution. Given a corrupted input image, a two-stream CNN analyzes the image and synthesizes the weights of a spatially-varying linear filter. This filter is then applied to the input to produce a deblurred/denoised prediction. The whole framework is end-to-end trainable in a self-supervised way for tasks such as super-resolution where corrupted images can be generated automatically. The predicted filters are easily constrained for different tasks and interpretable (here visualized in the center column by the mean flow displacement, see Fig. 6). viewed as “blind” filter flow estimation, in analogy with To summarize our contribution: (1) we propose a novel, blind deconvolution. During training, we use a loss defined end-to-end trainable, learning framework for solving vari- over the transformed image (rather than the predicted flow). ous low-level image reconstruction tasks; (2) we show this This is closely related to so-called self-supervised tech- framework is highly interpretable and controllable, enabling niques that learn to predict optical flow and depth from un- direct post-hoc analysis of how the reconstructed image is labeled video data [15, 16, 21]. Specifically, for the recon- generated from the degraded input; (3) we show experimen- struction tasks we consider such as image super-resolution, tally that predictive filter flow outperforms the state-of-the- the forward degradation process can be easily simulated to art methods remarkably on the three different tasks, non- generate a large quantity of training data without manual uniform motion blur removal, compression artifact reduc- collection or annotation. tion and single image super-resolution. The lack of interpretability in deep image-to-image re- 2. Related Work gression models makes it hard to provide guarantees of ro- bustness in the presence of adversarial input [31], and con- Our work is inspired by filter flow [44], which is an op- fer reliability needed for researchers in biology and medical timization based method for finding a linear transformation science [36]. Predictive filter flow differs from other CNN- relating nearby pixel values in a pair of images. By im- based approaches in this regard since the intermediate filter posing additional constraints on certain structural properties flows are interpretable and transparent [52, 12, 34], provid- of these filters, it serves as a general framework for under- ing an explicit description of how the input is transformed standing a wide variety of low-level vision problems. How- into output. It is also straightforward to inject constraints ever, filter flow as originally formulated has some obvious on the reconstruction (e.g., local brightness conservation) shortcomings. First, it requires prior knowledge to specify which would be nearly impossible to guarantee for deep a set of constraints needed to produce good results. It is not image-to-image regression models. always straightforward to model or even come up with such To evaluate our model, we carry out extensive experi- knowledge-based constraints. Second, solving for an opti- ments on three different low-level vision tasks, non-uniform mal filter flow is compute intensive; it may take up to 20 hours to compute over a pair of 500 × 500 images [44]. We motion blur removal, JPEG compression artifact reduction and single image super-resolution. We show that our model address these by directly predicting flows from image data. surpasses all the state-of-the-art methods on all the three We leverage predictive filter flow for targeting three specific tasks. We also visualize the predicted filters which reveals image reconstruction tasks which can be framed as perform- filtering operators reminiscent of classic unsharp masking ing spatially variant filtering over local image patches. filters and anisotropic diffusion along boundaries. Non-Uniform Blind Motion Blur Removal is an ex-
Recommend
More recommend
