Online Video SEEDS Dr. Michael Van den Bergh Superpixels - - PowerPoint PPT Presentation

Online Video SEEDS

• Dr. Michael

Van den Bergh

SEEDS Superpixels Extracted via Energy-
 Driven Sampling ECCV 2012

What are superpixels?

• grouping pixels based on

similarity (color)

• speeds up segmentation
• objects are made up of a

small number of superpixels

Existing superpixel methods

gradual addition of cuts

• high accuracy
• very slow (contradictory)
• e.g. Entropy Rate Superpixels (Liu et al.)

Existing superpixel methods

growing from centers

• faster
• reduced accuracy (local minima + stray labels)
• still not fast enough
• e.g. SLIC Superpixels (Achanta et al.)

new approach: SEEDS Superpixels

• initialize with rectangular boundaries
• gradually refine boundaries
• SEEDS: Superpixels Extracted via Energy-driven

Sampling - ECCV 2012

initalization largest block update medium block update smallest block update pixel-level update

Advantages of SEEDS

• faster than growing centers
• nly needs to evaluate at the boundaries
• highly efficient evaluation using color histograms (1 memory lookup)

initalization largest block update medium block update smallest block update pixel-level update

Advantages of SEEDS

• faster than growing centers
• nly needs to evaluate at the boundaries
• highly efficient evaluation using color histograms
• accuracy matches or exceeds state-of-the-art
• avoids local minima
• ptimization only evaluates valid partitionings


initalization largest block update medium block update smallest block update pixel-level update

Advantages of SEEDS

0.75 1.5 2.25 3 50 100 200 400

Undersegmentation Error

number of superpixels

SEEDS (15Hz) SLIC (5Hz) Entropy Rate (1Hz) Felzenszwalb and Huttenlocher

0.2 0.4 0.6 0.8 1 50 100 200 400

Boundary Recall

number of superpixels

SEEDS (15Hz) SLIC (5Hz) Entropy Rate (1Hz) Felzenszwalb and Huttenlocher

0.84 0.88 0.91 0.95 0.98 50 100 200 400

Achievable Segmentation Accuracy

number of superpixels

SEEDS (15Hz) SLIC (5Hz) Entropy Rate (1Hz) Felzenszwalb and Huttenlocher

Advantages of SEEDS

• faster than state-of-the-art

• accuracy matches or exceeds state-of-the-art


initalization largest block update medium block update smallest block update pixel-level update

Advantages of SEEDS

• faster than state-of-the-art

• accuracy matches or exceeds state-of-the-art

• control over run-time
• whenever the algorithm is stopped, a valid partitioning is available
• state-of-the-art accuracy at 30 Hz (single core)


initalization largest block update medium block update smallest block update pixel-level update

Advantages of SEEDS

• faster than state-of-the-art

• accuracy matches or exceeds state-of-the-art

• control over run-time
• whenever the algorithm is stopped, a valid partitioning is available
• state-of-the-art accuracy at 30 Hz (single core)

• control over superpixel shape
• ne or more priors can be applied during boundary updating

initalization largest block update medium block update smallest block update pixel-level update

(b) SEEDS with 3 × 3 smoothing prior (b) SEEDS with compactness prior (b) SEEDS with edge prior (snap to edges) (b) SEEDS with combined prior (3 × 3 smoothing + compactness + snap to edges)

Advantages of SEEDS

• faster
• more accurate
• control over run-time
• control over shape
• temporal

Advantages of SEEDS

initalization largest block update medium block update smallest block update pixel-level update

Online Video SEEDS

frame 0 frame 1 frame 2 initialization pixel-updates

• frame

block-updates propagation

Video SEEDS

Video SEEDS

initialization layer 3 (blocks) layer 2 (blocks) layer 1 (pixels) initialization layer 2 (blocks) layer 1 (pixels) t = 0 t = 1

Figure 4. Efficient updating at different block sizes.

Video SEEDS

Termination

, cAt

m

t n

, cAt

m

t:0 p

, cAt

m

t-1:0 n

time current frame

Creation

t m

, cAt

m

t:0 m

time current frame

|, |Bt

n

Figure 5. Termination and creation of superpixels.

• superpixels per frame
• superpixel rate (time)

Video SEEDS

Number of Supervoxels 3D Undersegmentation Error

StreamGBH (t=1) Video SEEDS (t=1) StreamGB (t=1) GBH (t= ) Meanshift StreamGBH (t=10)

Number of Supervoxels 3D Boundary Recall

StreamGBH (t=1) Video SEEDS (t=1) StreamGB (t=1) GBH (t= ) Meanshift StreamGBH (t=10)

Number of Supervoxels Explained Variation

StreamGBH (t=1) Video SEEDS (t=1) StreamGB (t=1) GBH (t= ) Meanshift StreamGBH (t=10)

• Chen Xiph.org benchmark
• t=∞ means the entire video is analyzed
• t=1 means it is online (not streaming)
• we are at 30Hz, they are at 0.25 Hz

SEEDS in OpenCV

Randomized SEEDS

(b) SEEDS with 3 × 3 smoothing prior (b) SEEDS with compactness prior (b) SEEDS with edge prior (snap to edges) (b) SEEDS with combined prior (3 × 3 smoothing + compactness + snap to edges)

Randomness Injection

multiple SEEDS samples labels

Randomized SEEDS

multiple SEEDS samples Randomized SEEDS labels

Temporal Video Objectness

multiple SEEDS samples Randomized SEEDS

• bjectness score

labels

Temporal Video Objectness

Tubes of Bounding Boxes

Tubes of Bounding Boxes

Temporal Video Objectness (SEEDS)

10 10

10

10

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

gPb (auc: 0.473) Canny (auc: 0.408) Randomized SEEDS - 1 sample (auc: 0.428) Randomized SEEDS - 5 samples (auc: 0.475) # windows Detection Rate Objectness on still images: baselines

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Objectness [1] (auc: 0.490) van de Sande [16] Feng et al. [7] (auc: 0.475) Rahtu et al. [12] (auc: 0.3680)

10 10

10

10

# windows Detection Rate Randomized SEEDS (auc: 0.475) Objectness on still images: s-o-a

0.09 0.18 0.27 0.36 0.45 0.54 0.63 0.72 0.81 0.9

3D edge - 5 samples (auc: 0.652) 3D edge - 1 sample (auc: 0.523)

• nly propagation - 5 samples (auc: 0.628)
• nly propagation - 1 sample (auc: 0.309)

Video Objectness: temporal window performance Detection Rate # temporal windows (tubes)

10 10

10

10

10

Thank You.