Spatially Accurate and Temporally Dense Extraction of Primary Object - PowerPoint PPT Presentation

Video Object Segmentation through Spatially Accurate and Temporally Dense Extraction of Primary Object Regions Dong Zhang 1 , Omar Javed 2 , Mubarak Shah 1 presented by Sehyun Joo based on Dong’s slide

Review : Semantic Based Action Retrievals

Review : NETVLAD

Video Object Segmentation through Spatially Accurate and Temporally Dense Extraction of Primary Object Regions Dong Zhang 1 , Omar Javed 2 , Mubarak Shah 1 presented by Sehyun Joo based on Dong’s slide

Motivation • Interesting, but ‘easy -to- understand’ approach to solve the whole problem • Might have insight of solving Image Search problems also.

Video Object Segmentation: Outline • Problem Statement • Related Work • Problem Analysis • Solution – Method Framework – Object Proposal Expansion – Layered DAG based Primary Object Selection – GMMs and MRF Optimization • Results

Problem Statement Segmentation of the primary moving object in videos

Video Object Segmentation • D. Tsai, M. Flagg and J. Rehg , “Motion Coherent Tracking with Multi - label MRF optimization”, BMVC, 2010: – SegTrack dataset – Assume manual annotations • Y . Lee, J. Kim and K. Grauman, ” Key-segments for video object segmentation ”, ICCV, 2011: – Object proposals – Extract “key - segments” • T. Ma, M. Yi and L. Latecki , “Maximum weight cliques with mutex constraints for video object segmentation ”, CVPR, 2012: – Object proposals – NP-hard problem, approximate optimization, computation cost

Object Proposal • Object proposal methods [1,2] output regions as potential object regions … … [1] Ian Endres and Derek Hoiem, “ Category Independent Object Proposals”, ECCV, 2010 [2] Alexe, B., Deselares, T. and Ferrari, V., “What is an object ?”, CVPR, 2010

Segtrack (monkeydog) Frame Ranked object proposals index # 3 1 2 4 18 1 2 100 3 4 … … 30 Object proposals are not reliable 1 2 3 4 100 … … 40 100 1 2 3 4 … … Sampling a lot of proposals and selecting the right ones is important! 51 1 2 17 3 4 … … 60 1 2 21 3 4 … …

Segtrack (parachute) Frame index Ranked object proposals expansion # 3 1 2 4 33 1 2 96 3 4 … … 38 1 2 3 4 98 … … 40 100 1 2 3 4 … … 43 True object region may break 1 2 100 3 4 … … into multiple object proposals 49 1 2 100 3 4 … …

Challenges • HOW to select correct object regions consistently ? • HOW to formulate the problem for an efficient solution.

Yes! Solutions? (1) Object proposal expansion • HOW to select correct (2) Flow warped shape and color object regions consistently ? similarity measure  connect frames (3) Optical flow gradient  scoring function • HOW to formulate the (1) A novel DAG formulation problem for an efficient (2) Dynamic programming solution solution.

Framework Input Videos Object Proposal Generation & Expansion Layered DAG Optimization for Primary Object Selection GMMs and MRF based Optimization Object Segmentation

Graph Structure for Object Proposal Selection: Unary Edge An object proposal Beginning node Ending node Unary edge Represents object-ness

Unary Edge: Score 𝑻 𝒗𝒐𝒃𝒔𝒛 = 𝑵 𝒔 + 𝑩(𝒔) 𝑵(𝒔) : average Frobenius norm for optical flow gradient 𝒗 𝒚 𝒗 𝒛 𝟑 + 𝒗 𝒛 𝟑 + 𝒘 𝒚 𝟑 + 𝒘 𝒛 𝟑 𝑽 𝒚 = = 𝒗 𝒚 𝒘 𝒚 𝒘 𝒛 𝑮 𝑩 𝒔 : appearance score from [1] [1] Ian Endres and Derek Hoiem, “ Category Independent Object Proposals”, ECCV, 2010

Unary Edge: Motion Score Boundary region Original video frame Object region Optical flow Optical flow gradient OF gradient around boundary

[1] I. Endres and D. Hoiem “ Category Independent Object Proposals ” ECCV, 2010

Graph Structure for Object Proposal Selection: Binary Edges Frame i+1 … … Frame i Frame i+2 … … … … … … Binary edge … … … … … … … … … … … … … … … … … … … … … … … …

Binary Edge Score 𝑻 𝒄𝒋𝒐𝒃𝒔𝒛 𝒔 𝒏 , 𝒔 𝒐 = 𝝁 ∙ 𝑻 𝒑𝒘𝒇𝒔𝒎𝒃𝒒 𝒔 𝒏 , 𝒔 𝒐 ∙ 𝑻 𝒅𝒑𝒎𝒑𝒔 (𝒔 𝒏 , 𝒔 𝒐 ) 𝑻 𝒑𝒘𝒇𝒔𝒎𝒃𝒒 (𝒔 𝒏 , 𝒔 𝒐 ) = 𝒔 𝒏 ∩ 𝒙𝒃𝒔𝒒 𝒏𝒐 (𝒔 𝒐 ) 𝒔 𝒏 ∪ 𝒙𝒃𝒔𝒒 𝒏𝒐 (𝒔 𝒐 ) 𝑻 𝒅𝒑𝒎𝒑𝒔 (𝒔 𝒏 , 𝒔 𝒐 ) = 𝒊𝒋𝒕𝒖(𝒔 𝒏 ) ∙ 𝒊𝒋𝒕𝒖(𝒔 𝒐 ) 𝑼

Graph Structure for Object Proposal Selection Frame i-1 Frame i Frame i+1 …… …… …… …… s t …… …… Layer Layer Layer Layer Layer Layer 2i-3 2i-2 2i-1 2i 2i+1 2i+2 Goal: Find only one object proposal from each frame, such that all of them have high object-ness and high similarity across frames.  Find the highest weighted path in the DAG.  Longest Path Problem of DAG  Dynamic Programming Solution.

SegTrack Dataset Original video Ground truth Selected object proposals Segmentation results Region within the red boundary is the object region

SegTrack Dataset Original video Ground truth Selected object proposals Segmentation results Region within the red boundary is the object region

SegTrack Dataset Original video Ground truth Segmentation results Region within the red boundary is the object region

SegTrack Dataset Original video Ground truth Segmentation results Region within the red boundary is the object region

SegTrack: Quantitative Results* Ours [14] [13] [20] [6] Supervised? N N N Y Y Birdfall 155 189 288 252 454 Cheetah 633 806 905 1142 1217 Girl 1488 1698 1785 1304 1755 Monkeydog 365 472 521 533 683 Parachute 220 221 201 235 502 Avg. 452 542 592 594 791 * Average per-frame pixel error rate. The smaller, the better.

GaTech Dataset Original video Segmentation results Region within the red boundary is the object region

GaTech Dataset Original video Segmentation results Region within the red boundary is the object region

Thank you!

Quiz 1. What kind of graph this paper is trying to build? (a) Directed Acyclic Graph (b) Directed Cyclic Graph (c) Undirected Acyclic Graph (d) Undirected Cyclic Graph 2. Unary edge is weight of how much object-ness. What is criteria of weight of binary edge? (a) how much non-moving (b) how much similar between object across frames (c) how much the difference of color histogram (d) how much the object in frame has precise shape