Integrated Deep and Shallow Networks for Salient Object Detection - PowerPoint PPT Presentation

Integrated Deep and Shallow Networks for Salient Object Detection Jing Zhang, Bo Li, Yuchao Dai, Fatih Porikli, Mingyi He Jing Zhang 1 , 2 , Bo Li 1 , Yuchao Dai 2 , Fatih Porikli 2 , Mingyi He 1 1 Northwestern Polytechnical University 2 Australian National University zjnwpu@gmail.com robert libo@qq.com yuchao.dai@anu.edu.au fatih.porikli@anu.edu.au myhe@nwpu.edu.cn 2017-08-17 Jing Zhang, Bo Li, Yuchao Dai, Fatih Porikli, Mingyi He ANU, NPU Integrated Deep and Shallow Networks for Salient Object Detection

What is salient object detection? Salient object detection aims at identifying the visually interesting objects regions that stand out relative to their neighbors and are consistent with human perception . Sample images and their corresponding saliency maps. Jing Zhang, Bo Li, Yuchao Dai, Fatih Porikli, Mingyi He ANU, NPU Integrated Deep and Shallow Networks for Salient Object Detection

Deep features vs handcrafted features ◮ Deep features can efficiently capture semantic information. ◮ Handcrafted features, which is summarized and described with human knowledge, are pivotal for simple scenarios. ◮ Deep features based salient object detection achieves the state-of-the-art performance; ◮ There exist situations where handcrafted saliency methods would outperform deep saliency methods. Jing Zhang, Bo Li, Yuchao Dai, Fatih Porikli, Mingyi He ANU, NPU Integrated Deep and Shallow Networks for Salient Object Detection

Deep features and handcrafted features together Image GT OURS DC[5] MDF[3] RBD[6] ◮ Whether data-driven (e.g. deep learning) based saliency detection methods sufficiently exploit statistical information? ◮ Whether unsupervised saliency and data-driven saliency can be combined to achieve even better performance? Jing Zhang, Bo Li, Yuchao Dai, Fatih Porikli, Mingyi He ANU, NPU Integrated Deep and Shallow Networks for Salient Object Detection

Motivation ◮ Deep features can be a double-edged sword: ◮ Deep features provide high-level semantic cues critical for saliency detection, however ◮ Structure information may be neglected in high-level deep features, ◮ Existing FCNN based deep saliency methods cannot incorporate handcrafted prior knowledge, ◮ Feature maps from FCNN are usually blurred around edges. Jing Zhang, Bo Li, Yuchao Dai, Fatih Porikli, Mingyi He ANU, NPU Integrated Deep and Shallow Networks for Salient Object Detection

Integrating deep features and handcrafted features Given an input image, our deep model produces a coarse saliency map. Then a shallow model integrates deep saliency and handcrafted saliency. Finally, a multi-scale superpixel level fusion (MSSF) obtains a spatially coherent saliency map. Jing Zhang, Bo Li, Yuchao Dai, Fatih Porikli, Mingyi He ANU, NPU Integrated Deep and Shallow Networks for Salient Object Detection

Fully convolutional neural networks for saliency detection ◮ Finetune an FCNN [Chen, 2016] [He, 2016] with dilated convolutional layers for semantic segmentation to adapt it to salient object detection. ◮ 3,000 images from the MSRA10K for training. Jing Zhang, Bo Li, Yuchao Dai, Fatih Porikli, Mingyi He ANU, NPU Integrated Deep and Shallow Networks for Salient Object Detection

Multi-scale superpixel level fusion Steps for multi-scale superpixellevel fusion: ◮ SLIC for image over-segmentation X = { X 1 , X 2 , · · · , X N } , where N = 100 , 200 , 300 , 400 to achieve multi-scale image over-segmentation; ◮ Per-superpixel saliency map S k , k = 1 , 2 , 3 , 4 where saliency value of each superpixel is defined as median saliency prediction score of saliency map from our deep-shallow model S DS ; ◮ Saliency fusion: S DSM = � S k Jing Zhang, Bo Li, Yuchao Dai, Fatih Porikli, Mingyi He ANU, NPU Integrated Deep and Shallow Networks for Salient Object Detection

Experimental results Image GT MDF[3] RFCN[26] DC[5] DeepMC[4] DMT[8] OURS Salient object detection results on challenging images by different methods Jing Zhang, Bo Li, Yuchao Dai, Fatih Porikli, Mingyi He ANU, NPU Integrated Deep and Shallow Networks for Salient Object Detection

Experimental results DUT THUR 0.9 0.9 0.8 0.8 0.7 0.7 DMT RFCN 0.6 0.6 DMT DeepMC Precision Precision RFCN LEGS DeepMC 0.5 MDF 0.5 LEGS DC MDF DRFI 0.4 0.4 DC RBD DRFI DSR 0.3 0.3 RBD MC DSR DISC MC 0.2 0.2 DS DS DSM DSM 0.1 0.1 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 Recall Recall ECSSD HKU-IS 1 1 0.9 0.9 0.8 0.8 0.7 0.7 DMT Precision RFCN Precision RFCN 0.6 DeepMC DeepMC 0.6 MDF LEGS 0.5 DC DC 0.5 DRFI DRFI 0.4 RBD RBD DSR DSR 0.4 0.3 MC MC DISC DISC 0.3 0.2 DS DS DSM DSM 0.2 0.1 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 Recall Recall Figure: Comparison of Precision-Recall curves on four datasets. Jing Zhang, Bo Li, Yuchao Dai, Fatih Porikli, Mingyi He ANU, NPU Integrated Deep and Shallow Networks for Salient Object Detection

Model Analysis MAE on eight benchmark datasets. Jing Zhang, Bo Li, Yuchao Dai, Fatih Porikli, Mingyi He ANU, NPU Integrated Deep and Shallow Networks for Salient Object Detection

Conclusion ◮ An end-to-end FCNN based approach for saliency detection ◮ Multi-level superpixel level saliency fusion to enhance saliency maps ◮ Small and relatively simple training dataset with state-of-the-art performance ◮ Efficient for saliency prediction in testing stage, 0.4 sec per image with 0.2 sec for image over-segmentation. Jing Zhang, Bo Li, Yuchao Dai, Fatih Porikli, Mingyi He ANU, NPU Integrated Deep and Shallow Networks for Salient Object Detection

Key references L.C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, “DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs,” arXiv, 2016 K. He, X. Zhang, S. Ren, J. Sun, “Deep residual learning for image recognition,” CVPR 2016 Jing Zhang, Bo Li, Yuchao Dai, Fatih Porikli, Mingyi He ANU, NPU Integrated Deep and Shallow Networks for Salient Object Detection

Thanks! Jing Zhang, Bo Li, Yuchao Dai, Fatih Porikli, Mingyi He ANU, NPU Integrated Deep and Shallow Networks for Salient Object Detection