Integrated Deep and Shallow Networks for Salient Object Detection - - PowerPoint PPT Presentation

Integrated Deep and Shallow Networks for Salient Object Detection

Jing Zhang, Bo Li, Yuchao Dai, Fatih Porikli, Mingyi He

Jing Zhang1,2, Bo Li1, Yuchao Dai2, Fatih Porikli2, Mingyi He1

1Northwestern Polytechnical University 2Australian National University

zjnwpu@gmail.com robert libo@qq.com yuchao.dai@anu.edu.au fatih.porikli@anu.edu.au myhe@nwpu.edu.cn

2017-08-17

Jing Zhang, Bo Li, Yuchao Dai, Fatih Porikli, Mingyi He ANU, NPU Integrated Deep and Shallow Networks for Salient Object Detection

What is salient object detection?

Salient object detection aims at identifying the visually interesting objects regions that stand out relative to their neighbors and are consistent with human perception. Sample images and their corresponding saliency maps.

Jing Zhang, Bo Li, Yuchao Dai, Fatih Porikli, Mingyi He ANU, NPU Integrated Deep and Shallow Networks for Salient Object Detection

Deep features vs handcrafted features

◮ Deep features can efficiently capture semantic information. ◮ Handcrafted features, which is summarized and described with

human knowledge, are pivotal for simple scenarios.

◮ Deep features based salient object detection achieves the

state-of-the-art performance;

◮ There exist situations where handcrafted saliency methods

would outperform deep saliency methods.

Jing Zhang, Bo Li, Yuchao Dai, Fatih Porikli, Mingyi He ANU, NPU Integrated Deep and Shallow Networks for Salient Object Detection

Deep features and handcrafted features together

Image GT OURS DC[5] MDF[3] RBD[6]

◮ Whether data-driven (e.g. deep learning) based saliency

detection methods sufficiently exploit statistical information?

◮ Whether unsupervised saliency and data-driven saliency can

be combined to achieve even better performance?

Jing Zhang, Bo Li, Yuchao Dai, Fatih Porikli, Mingyi He ANU, NPU Integrated Deep and Shallow Networks for Salient Object Detection

Motivation

◮ Deep features can be a double-edged sword:

◮ Deep features provide high-level semantic cues critical for

saliency detection, however

◮ Structure information may be neglected in high-level deep

features,

◮ Existing FCNN based deep saliency methods cannot

incorporate handcrafted prior knowledge,

◮ Feature maps from FCNN are usually blurred around edges. Jing Zhang, Bo Li, Yuchao Dai, Fatih Porikli, Mingyi He ANU, NPU Integrated Deep and Shallow Networks for Salient Object Detection

Integrating deep features and handcrafted features

Given an input image, our deep model produces a coarse saliency

• map. Then a shallow model integrates deep saliency and

handcrafted saliency. Finally, a multi-scale superpixel level fusion (MSSF) obtains a spatially coherent saliency map.

Jing Zhang, Bo Li, Yuchao Dai, Fatih Porikli, Mingyi He ANU, NPU Integrated Deep and Shallow Networks for Salient Object Detection

Fully convolutional neural networks for saliency detection

◮ Finetune an FCNN [Chen, 2016] [He, 2016] with dilated

convolutional layers for semantic segmentation to adapt it to salient object detection.

◮ 3,000 images from the MSRA10K for training.

Jing Zhang, Bo Li, Yuchao Dai, Fatih Porikli, Mingyi He ANU, NPU Integrated Deep and Shallow Networks for Salient Object Detection

Multi-scale superpixel level fusion

Steps for multi-scale superpixellevel fusion:

◮ SLIC for image over-segmentation X = {X1, X2, · · · , XN},

where N = 100, 200, 300, 400 to achieve multi-scale image

• ver-segmentation;

◮ Per-superpixel saliency map Sk, k = 1, 2, 3, 4 where saliency

value of each superpixel is defined as median saliency prediction score of saliency map from our deep-shallow model SDS;

◮ Saliency fusion: SDSM = Sk

Jing Zhang, Bo Li, Yuchao Dai, Fatih Porikli, Mingyi He ANU, NPU Integrated Deep and Shallow Networks for Salient Object Detection

Experimental results

Image GT MDF[3] RFCN[26] DC[5] DeepMC[4] DMT[8] OURS

Salient object detection results on challenging images by different methods

Jing Zhang, Bo Li, Yuchao Dai, Fatih Porikli, Mingyi He ANU, NPU Integrated Deep and Shallow Networks for Salient Object Detection

Experimental results

0.2 0.4 0.6 0.8 1 Recall 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Precision DUT DMT RFCN DeepMC LEGS MDF DC DRFI RBD DSR MC DISC DS DSM 0.2 0.4 0.6 0.8 1 Recall 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Precision THUR DMT RFCN DeepMC LEGS MDF DC DRFI RBD DSR MC DS DSM 0.2 0.4 0.6 0.8 1 Recall 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Precision ECSSD DMT RFCN DeepMC MDF DC DRFI RBD DSR MC DISC DS DSM 0.2 0.4 0.6 0.8 1 Recall 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Precision HKU-IS RFCN DeepMC LEGS DC DRFI RBD DSR MC DISC DS DSM

Figure: Comparison of Precision-Recall curves on four datasets.

Jing Zhang, Bo Li, Yuchao Dai, Fatih Porikli, Mingyi He ANU, NPU Integrated Deep and Shallow Networks for Salient Object Detection

Model Analysis

MAE on eight benchmark datasets.

Jing Zhang, Bo Li, Yuchao Dai, Fatih Porikli, Mingyi He ANU, NPU Integrated Deep and Shallow Networks for Salient Object Detection

Conclusion

◮ An end-to-end FCNN based approach for saliency detection ◮ Multi-level superpixel level saliency fusion to enhance saliency

maps

◮ Small and relatively simple training dataset with

state-of-the-art performance

◮ Efficient for saliency prediction in testing stage, 0.4 sec per

image with 0.2 sec for image over-segmentation.

Jing Zhang, Bo Li, Yuchao Dai, Fatih Porikli, Mingyi He ANU, NPU Integrated Deep and Shallow Networks for Salient Object Detection

Key references

L.C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, “DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs,” arXiv, 2016

• K. He, X. Zhang, S. Ren, J. Sun, “Deep residual learning for image

recognition,” CVPR 2016

Jing Zhang, Bo Li, Yuchao Dai, Fatih Porikli, Mingyi He ANU, NPU Integrated Deep and Shallow Networks for Salient Object Detection

Thanks!

Jing Zhang, Bo Li, Yuchao Dai, Fatih Porikli, Mingyi He ANU, NPU Integrated Deep and Shallow Networks for Salient Object Detection