Saliency Detection Feiyang Chen July 27, 2019 Background - - PowerPoint PPT Presentation

▶

Oct 30, 2023 330 likes •562 views

Saliency Detection Feiyang Chen July 27, 2019 Background Humans are able to detect visually distinctive, so-called salient , scene regions effortlessly and rapidly in a pre- attentive stage It helps to find the objects or regions

SLIDE 1

Saliency Detection

Feiyang Chen July 27, 2019

SLIDE 2

Background

Humans are able to detect visually distinctive, so-called

salient, scene regions effortlessly and rapidly in a pre- attentive stage

It helps to find the objects or regions that efficiently

represent a scene, a useful step in complex vision problems such as scene understanding

Some topics that are closely or remotely related to visual

saliency include: salient object detection, fixation prediction, image quality assessment, scene attributes…

SLIDE 3

Definition

A process of two stages:

1) detecting the most salient object
2) segmenting the accurate region of that object.

Three criteria:

1) good detection: the probability of missing real salient regions and falsely marking the

background as a salient region should be low,

2) high resolution: saliency maps should have a high or full resolution to accurately

locate salient objects and retain original image information,

3) computational efficiency: as front-ends to other complex processes, these models

should detect salient regions quickly.

SLIDE 4

History

The First Wave:

One of the earliest saliency models, proposed by Itti et
al. in 1998, generated the first wave of interest across

multiple disciplines including cognitive psychology, neuroscience, and computer vision.

This model is an implementation of earlier general

computational frameworks and psychological theories of bottom-up attention based on center-surround mechanisms.

SLIDE 5

History

The Second Wave:

The second wave of interest surged with

several works who defined saliency detection as a binary segmentation problem.

These authors were inspired by some earlier

models striving to detect salient regions or proto-objects.

SLIDE 6

History

The Third Wave:

A third wave of interest has appeared recently with the surge in popularity
f CNNs, and in particular with the FCNs.
Unlike the majority of classic methods based on contrast cues, CNN-based

methods both eliminate the need for hand-crafted features, and alleviate the dependency on center bias knowledge, and hence have been adopted by many researchers.

Neurons with large receptive fields provide global information that can

help better identify the most salient region in an image, while neurons with small receptive fields provide local information that can be leveraged to refine saliency maps produced by the higher layers. This allows highlighting salient regions and refining their boundaries.

SLIDE 7

Methods

Supervised or Unsupervised
Traditional algorithms or Deep Learning-based methods

SLIDE 8

Classic Method: ITTI

Itti L, Koch C, Niebur E. A model of saliency-based visual attention for rapid scene analysis[J].

SLIDE 9

Classic Method: SR

Hou X, Zhang L. Saliency detection: A spectral residual approach[C]//2007 IEEE Conference on Computer Vision and Pattern Recognition. Ieee, 2007: 1-8.

SLIDE 10

Classic Method: LC

Zhai Y, Shah M. Visual attention detection in video sequences using spatiotemporal cues[C]

SLIDE 11

DL-based Methods

Deep Unsupervised Saliency Detection: A Multiple Noisy Labeling Perspective

SLIDE 12

DL-based Methods

An Unsupervised Game-Theoretic Approach to Saliency Detection

SLIDE 13

DL-based Methods

SalGAN: visual saliency prediction with adversarial networks

SLIDE 14

Experiments

SLIDE 15

Future Work

Co-Saliency
Improved SalGAN
Unsupervised Pre-Train
Combine other methods

SLIDE 16

Saliency Survey

Traditional Methods

1.[ITTI][1998][PAMI] A Model of Saliency-Based Visual Attention for Rapid Scene Analysis, 2.[AIM][2005][NIPS]Saliency Based on Information Maximization, 3.[SR] [2007][CVPR]Saliency Detection A Spectral Residual Approach, 4.[GB][2007][NIPS]Graph-Based Visual Saliency, 5.[SUN][2008][JOV]SUN: A bayesian framework for saliency using natural statistics, 6.[FT][2009][CVPR]Frequency-tuned Salient Region Detection, 7.[CA][2010][CVPR]Context-Aware Saliency Detection, 8.[SEG][2010][ECCV]Segmenting Salient Objects from Images and Videos, 9.[MSSS][2010][ICIP]Saliency Detection using Maximum Symmetric Surround, 10.[HC,RC][2011][CVPR]Global Contrast based Salient Region Detection, 11.[CB][2011][BMVC]Automatic Salient Object Segmentation Based on Context and Shape Prior, 12.[SF][2012][CVPR]Saliency Filters Contrast Based Filtering for Salient Region Detection, 13.[LR][2012][CVPR]A Unified Approach to Salient Object Detection via Low Rank Matrix Recovery, 14.[BSF][2012][ICIP]Saliency Detection Based on Integration of Boundary and Soft-Segmentation, 15.[GC][2013][ICCV]Efficient Salient Region Detection with Soft Image Abstraction, 16.[MR][2013][CVPR]Saliency Detection via Graph-Based Manifold Ranking, 17.[MC][2013][ICCV]Saliency Detection via Absorbing Markov Chain, 18.[DRFI][2013][CVPR]Salient Object Detection A Discriminative Regional Feature Integration Region

SLIDE 17

Saliency Survey

Traditional Methods

19.[DSR][2013][ICCV]Saliency Detection via Dense and Sparse Reconstruction, 20.[HS][2013][CVPR]Hierarchical Saliency Detection, 21.[PCA][2013][CVPR]What Makes a Patch Distinct, 22.[CRF][2013][CVPR]Saliency Aggregation A Data-driven Approach, 23.[UFO][2013][ICCV]Salient Region Detection by UFO Uniqueness, Focusness and Objectness, 24.[COV][2013][JOV]Visual saliency estimation by nonlinearly integrating features using region covariances, 25.[GR][2013][SPL]Graph-Regularized Saliency Detection With Convex-Hull-Based Center Prior, 26.[LSSC][2013][TIP]Bayesian Saliency via Low and Mid Level Cues, 27.[HDCT][2014][CVPR]Salient Region Detection via High-Dimensional Color Transform, 28.[RBD][2014][CVPR]Saliency Optimization from Robust Background Detection, 29.[MSS][2014][SPL]Saliency Detection with Multi-Scale Superpixels, 30.[GP][2015][ICCV]Generic Promotion of Diffusion-Based Salient Object Detection, 31.[MBS][2015][ICCV]Minimum Barrier Salient Object Detection at 80 FPS, 32.[WSC][2015][CVPR]A Weighted Sparse Coding Framework for Saliency Detection, 33.[RRW][2015][CVPR]Robust saliency detection via regularized random walks ranking, 34.[TLLT][2015][CVPR]Saliency Propagation from Simple to Difficult, 35.[BL][2015][CVPR]Salient Object Detection via Bootstrap Learning

SLIDE 18

Saliency Survey

Traditional Methods

36.[BSCA][2015][CVPR]Saliency Detection via Cellular Automata, 37.[GLC][2015][PR]Salient Object Detection via Global and Local Cues, 38.[LPS][2015][TIP]Inner and Inter Label Propagation Salient Object Detection in the Wild, 39.[MAP][2015][TIP]Saliency Region Detection based on Markov Absorption Probabilities, 40.[BFS][2015][NC]Saliency Detection via Background and Foreground Seed Selection, 41.[MST][2016][CVPR]Real-Time Salient Object Detection with a Minimum Spanning Tree, 42.[PM][2016][ECCV]Pattern Mining Saliency, 43.[DSP][2016][PR]Discriminative saliency propagation with sink points, 44.[WLRR][2017][SPL]Salient Object Detection via Weighted Low Rank Matrix Recovery, 45.[MIL][2017][TIP]Salient Object Detection via Multiple Instance Learning, 46.[SMD][2017][PAMI]Salient Object Detection via Structured Matrix Decomposition, 47.[MDC][2017][TIP]300-FPS Salient Object Detection via Minimum Directional Contrast, 48.[WMR][2018][NC]Saliency detection via affinity graph learning and weighted manifold ranking, 49.[RCRR][2018][TIP]Reversion correction and regularized random walk ranking for saliency detection, 50.[WFD][2018][PR]Water flow driven salient object detection at 180 fps

SLIDE 19

Saliency Survey

Deep-Learning Methods

1.LEGS: Deep networks for saliency detection via local estimation and global search, Wang, L. et al, CVPR, 2015. 2.MC: Saliency detection by multi-context deep learning, Zhao, R., et al, CVPR, 2015. 3.MDF: Visual saliency based on multiscale deep features, Li, G., et al, CVPR, 2015. 4.DCL: Deep Contrast Learning for Salient Object Detection, Li, G.,et al, CVPR, 2016. 5.ELD: Deep Saliency with Encoded Low level Distance Map and High Level Features, Gayoung, L., et al, CVPR, 2016. 6.DHS: DHSNet: Deep Hierarchical Saliency Network for Salient Object Detection, Liu, N., et al, CVPR, 2016. 7.RFCN: Saliency detection with recurrent fully convolutional networks, Wang, L., et al, ECCV, 2016. 8.CRPSD: Saliency Detection via Combining Region-Level and Pixel-Level Predictions with CNNs, ECCV, 2016. 9.DISC: DISC: Deep image saliency computing via progressive representation learning, Chen, T., et al, TNNLS, 2016 10.DS: DeepSaliency: Multi-task deep neural network model for salient object detection, Li, X., et al, TIP, 2016. 11.IMC: Deep Salient Object Detection by Integrating Multi-level Cues, Zhang, J., et al, WACV 2017. 12.DSS: Deeply supervised salient object detection with short connections, Hou, Q., et al, CVPR, 2017/ TPAMI, 2018. 13.NLDF: Non-local deep features for salient object detection, Luo, Z., et al, CVPR, 2017. 14.AMU: Amulet: Aggregating multi-level convolutional features for salient object detection, Zhang, P., et al, ICCV, 2017. 15.UCF: Learning Uncertain Convolutional Features for Accurate Saliency Detection, Zhang, P., et al, ICCV, 2017.

SLIDE 20

Saliency Survey

Deep-Learning Methods

16.MSRNet: Instance-Level Salient Object Segmentation, Li, G., et al, CVPR, 2017. 17.WSS: Learning to Detect Salient Objects with Image-level Supervision, CVPR, 2017. 18.MDC: 300-FPS Salient Object Detection via Minimum Directional Contrast, Huang, X., et al, TIP,

2017. [code]

19.SRM: A stagewise refinement model for detecting salient objects in images, CVPR, 2017. 20.CARNet: Salient Object Detection using a Context-Aware Refinement Network, BMVC,2017. 21.DLS: Deep Level Sets for Salient Object Detection, ECCV, 2017. 22.R3Net: Recurrent Residual Refinement Network for Saliency Detection, IJCAI, 2018. 23.EARNet: Embedding Attention and Residual Network for Accurate Salient Object Detection, Cybernetics, 2018 24.PiCANet: Learning pixel-wise contextual attention for saliency detection, CVPR, 2018. 25.BDMPM: A Bi-directional Message Passing Model for Salient Object Detection, CVPR, 2018. 26.PAGRN: Progressive attention guided recurrent network for salient object detection, CVPR, 2018. 27.DGRL: Detect globally, refine locally: A novel approach to saliency detection, CVPR, 2018. 28.ASNet: Salient Object Detection Driven by Fixation Prediction, CVPR, 2018. 29.LPS: Learning to Promote Saliency Detectors, CVPR, 2018. 30.RAS: Reverse Attention for Salient Object Detection, ECCV, 2018.

SLIDE 21

Saliency Survey

Deep-Learning Methods

31.C2S: Contour Knowledge Transfer for Salient Object Detection, ECCV, 2018. 32.RADF: Recurrently aggregating deep features for salient object detection, AAAI, 2018. 33.SalGAN: visual saliency prediction with adversarial networks, CVIU, 2018. 34.SD: Super Diffusion for Salient Object Detection, arXiv, 2018. 35.UGA: An Unsupervised Game-Theoretic Approach to Saliency Detection, TIP, 2018. 36.DEF: Deep Embedding Features for Salient Object Detection, AAAI, 2019. 37.LFRWS: Salient Object Detection with Lossless Feature Reflection and Weighted Structural Loss, TIP, 2019. 38.SIBA: Selectivity or Invariance Boundary-aware Salient Object Detection, arXiv, 2019. 39.DRMC: Deep Reasoning with Multi-scale Context for Salient Object Detection, arXiv, 2019. 40.RDSNet: Richer and Deeper Supervision Network for Salient Object Detection, arXiv, 2019. 41.DNA: Deeply-supervised Nonlinear Aggregation for Salient Object Detection, arXiv, 2019. 42.DSAL-GAN: Denoising Based Saliency Prediction with Generative Adversarial Networks, arXiv, 2019. 43.SAC-Net: Spatial Attenuation Context for Salient Object Detection, arXiv, 2019. 44.SE2Net: Siamese Edge-Enhancement Network for Salient Object Detection, arXiv, 2019. 45.RRNet: Region Refinement Network for Salient Object Detection, arXiv, 2019.

SLIDE 22

Saliency Survey

Deep-Learning Methods

36.ROSA: Robust Salient Object Detection against Adversarial Attacks, arXiv, 2019. 37.HyperFusion-Net: Densely Reflective Fusion for Salient Object Detection, Pingping Zhang et al, PR, 2019. 38.LFRWS: Salient Object Detection with Lossless Feature Reflection and Weighted Structural Loss, TIP, 2019. 39.AFNet: Attentive Feedback Network for Boundary-Aware Salient Object Detection, Mengyang Feng et al, CVPR, 2019. 40.CapSal: Leveraging Captioning to Boost Semantics for Salient Object Detection, Lu Zhang et al, CVPR, 2019. 41.BASNet: Boundary-Aware Salient Object Detection, CVPR, 2019. 42.CPD: Cascaded Partial Decoder for Fast and Accurate Salient Object Detection, CVPR, 2019. 43.MWS: Multi-source weak supervision for saliency detection, Yu Zeng et al, CVPR, 2019. 44.MLMSNet: A Mutual Learning Method for Salient Object Detection with intertwined Multi-Supervision, Runming Wu et al, CVPR, 2019. 45.PoolNet: A Simple Pooling-Based Design for Real-Time Salient Object Detection, Jian-Jiang Liu et al, CVPR, 2019. 46.ICNet: An Iterative and Cooperative Top-down and Bottom-up Inference Network for Salient Object Detection, W. Wang et al, CVPR, 2019. 47.PAGE-Net: Salient Object Detection with Pyramid Attention and Salient Edge, Wenguan Wang et al, CVPR, 2019. 48.HCA: Salient Object Detection via High-to-Low Hierarchical Context Aggregation, CVPR, 2019. 49.GCBR: Salient Object Detection in Low Contrast Images via Global Convolution and Boundary Refinement, CVPR, 2019. 50.EGNet: Edge Guidance Network for Salient Object Detection, xxx, 2019.