Q&A of GrabCut Philipp Krhenbhl 1 6-May-13 - PowerPoint PPT Presentation

Q&A ¡of ¡GrabCut ¡ Philipp ¡Krähenbühl ¡ 1 ¡ 6-­‑May-­‑13 ¡ Lecture 1 - � Philipp Krähenbühl � �

Goal ¡ • Bounding ¡Box ¡ – provided ¡ 2 ¡ 6-­‑May-­‑13 ¡ Philipp Krähenbühl � Lecture 1 - � �

Goal ¡ • SegmentaFon ¡of ¡object ¡within ¡bounding ¡box ¡ 3 ¡ 6-­‑May-­‑13 ¡ Philipp Krähenbühl � Lecture 1 - � �

Overview ¡ • Boykov ¡& ¡Jolly ¡segmentaFon ¡model ¡ ∑ ∑ E ( α , z ) = D ( α n , z n ) w n , m [ α n ≠ α m ] + γ n n , m – SegmentaFon ¡using ¡Graph ¡Cuts ¡ • GMM ¡foreground ¡and ¡background ¡model ¡ D ( α n , θ , z n ) – IteraFve ¡opFmizaFon ¡ • Graph ¡Cuts ¡ • GMM ¡esFmaFon ¡ 4 ¡ 6-­‑May-­‑13 ¡ Lecture 1 - � Philipp Krähenbühl � �

Initialisation • User initialises trimap T by supplying only T B . The fore- ground is set to T F = / 0; T U = T B , complement of the back- ground. • Initialise α n = 0 for n ∈ T B and α n = 1 for n ∈ T U . • Background and foreground GMMs initialised from sets α n = 0 and α n = 1 respectively. Iterative minimisation Project ¡2 ¡ 1. Assign GMM components to pixels: for each n in T U , k n : = argmin D n ( α n , k n , θ , z n ) . k n 2. Learn GMM parameters from data z : θ : = argmin θ U ( α , k , θ , z ) 3. Estimate segmentation: use min cut to solve: min U } min k E ( α , k , θ , z ) . { α n : n ∈ T 4. Repeat from step 1, until convergence. Not ¡required ¡ 5. Apply border matting (section 4). User editing • Edit: fix some pixels either to α n = 0 (background brush) or α n = 1 (foreground brush); update trimap T accord- OpFonal ¡ ingly. Perform step 3 above, just once. • Refine operation: [optional] perform entire iterative min- imisation algorithm. 5 ¡ 6-­‑May-­‑13 ¡ Philipp Krähenbühl � Lecture 1 - � �

Initialisation NotaFon ¡ • User initialises trimap T by supplying only T B . The fore- ground is set to T F = / 0; T U = T B , complement of the back- ground. • Initialise α n = 0 for n ∈ T B and α n = 1 for n ∈ T U . • Background and foreground GMMs initialised from sets • Trimaps ¡ α n = 0 and α n = 1 respectively. Iterative minimisation – Sets ¡of ¡pixels ¡ 1. Assign GMM components to pixels: for each n in T U , k n : = argmin D n ( α n , k n , θ , z n ) . – T B : ¡background ¡ k n 2. Learn GMM parameters from data z : – T F : ¡foreground ¡ θ : = argmin θ U ( α , k , θ , z ) 3. Estimate segmentation: use min cut to solve: – T U : ¡undecided ¡ min U } min k E ( α , k , θ , z ) . { α n : n ∈ T 4. Repeat from step 1, until convergence. • SegmentaFon ¡ 5. Apply border matting (section 4). – α i ¡for ¡each ¡pixel ¡i ¡ User editing • Edit: fix some pixels either to α n = 0 (background brush) or α n = 1 (foreground brush); update trimap T accord- ingly. Perform step 3 above, just once. • Refine operation: [optional] perform entire iterative min- imisation algorithm. 6 ¡ 6-­‑May-­‑13 ¡ Philipp Krähenbühl � Lecture 1 - � �

Initialisation IniFalizaFon ¡ • User initialises trimap T by supplying only T B . The fore- ground is set to T F = / 0; T U = T B , complement of the back- ground. • Initialise α n = 0 for n ∈ T B and α n = 1 for n ∈ T U . • Background and foreground GMMs initialised from sets α n = 0 and α n = 1 respectively. Iterative minimisation 1. Assign GMM components to pixels: for each n in T U , k n : = argmin D n ( α n , k n , θ , z n ) . k n 2. Learn GMM parameters from data z : θ : = argmin θ U ( α , k , θ , z ) • Outside ¡ 3. Estimate segmentation: use min cut to solve: – Background ¡( fixed ) ¡ min U } min k E ( α , k , θ , z ) . { α n : n ∈ T – α n =0 ¡ 4. Repeat from step 1, until convergence. • Inside ¡ 5. Apply border matting (section 4). – IniFal ¡foreground ¡ User editing • Edit: fix some pixels either to α n = 0 (background brush) – α n =1 ¡ or α n = 1 (foreground brush); update trimap T accord- ingly. Perform step 3 above, just once. – updated ¡ • Refine operation: [optional] perform entire iterative min- imisation algorithm. 7 ¡ 6-­‑May-­‑13 ¡ Philipp Krähenbühl � Lecture 1 - � �

Initialisation GMM ¡IniFalizaFon ¡ • User initialises trimap T by supplying only T B . The fore- ground is set to T F = / 0; T U = T B , complement of the back- ground. • Initialise α n = 0 for n ∈ T B and α n = 1 for n ∈ T U . • Find ¡parameters ¡θ ¡ • Background and foreground GMMs initialised from sets α n = 0 and α n = 1 respectively. • Standard ¡method ¡ Iterative minimisation 1. Assign GMM components to pixels: for each n in T U , – random ¡+ ¡EM ¡ k n : = argmin D n ( α n , k n , θ , z n ) . k n 2. Learn GMM parameters from data z : • Re-­‑esFmate ¡θ ¡in ¡ θ : = argmin θ U ( α , k , θ , z ) 3. Estimate segmentation: use min cut to solve: step ¡2 ¡ min U } min k E ( α , k , θ , z ) . { α n : n ∈ T – IniFalizaFon ¡only ¡ 4. Repeat from step 1, until convergence. 5. Apply border matting (section 4). needed ¡for ¡step ¡1 ¡ User editing – Trick: ¡IniFalize ¡k ¡ • Edit: fix some pixels either to α n = 0 (background brush) or α n = 1 (foreground brush); update trimap T accord- ingly. Perform step 3 above, just once. instead ¡ • Refine operation: [optional] perform entire iterative min- imisation algorithm. 8 ¡ 6-­‑May-­‑13 ¡ Philipp Krähenbühl � Lecture 1 - � �

Initialisation GMM ¡IniFalizaFon ¡ • User initialises trimap T by supplying only T B . The fore- ground is set to T F = / 0; T U = T B , complement of the back- ground. • Initialise α n = 0 for n ∈ T B and α n = 1 for n ∈ T U . • Trick: ¡IniFalize ¡k ¡ • Background and foreground GMMs initialised from sets α n = 0 and α n = 1 respectively. – k-­‑means ¡clustering ¡ Iterative minimisation 1. Assign GMM components to pixels: for each n in T U , – skip ¡step ¡1 ¡in ¡first ¡ k n : = argmin D n ( α n , k n , θ , z n ) . k n iteraFon ¡ 2. Learn GMM parameters from data z : θ : = argmin θ U ( α , k , θ , z ) 3. Estimate segmentation: use min cut to solve: min U } min k E ( α , k , θ , z ) . { α n : n ∈ T 4. Repeat from step 1, until convergence. 5. Apply border matting (section 4). User editing • Edit: fix some pixels either to α n = 0 (background brush) or α n = 1 (foreground brush); update trimap T accord- ingly. Perform step 3 above, just once. • Refine operation: [optional] perform entire iterative min- imisation algorithm. 9 ¡ 6-­‑May-­‑13 ¡ Philipp Krähenbühl � Lecture 1 - � �

Initialisation GMM ¡assignment ¡ • User initialises trimap T by supplying only T B . The fore- ground is set to T F = / 0; T U = T B , complement of the back- ground. • Initialise α n = 0 for n ∈ T B and α n = 1 for n ∈ T U . • Gaussian ¡log ¡prob. ¡ • Background and foreground GMMs initialised from sets α n = 0 and α n = 1 respectively. D n ( α n , k n , θ , z n ) = Iterative minimisation 1. Assign GMM components to pixels: for each n in T U , − log π ( α n , k n ) + 1 2 logdet Σ ( α n , k n ) k n : = argmin D n ( α n , k n , θ , z n ) . k n 2. Learn GMM parameters from data z : + 1 T Σ ( α n , k n ) − 1 z n − µ ( α n , k n ) 2 z n − µ ( α n , k n ) [ ] [ ] θ : = argmin θ U ( α , k , θ , z ) 3. Estimate segmentation: use min cut to solve: • Enumerate ¡all ¡k n ¡ min U } min k E ( α , k , θ , z ) . { α n : n ∈ T 4. Repeat from step 1, until convergence. • Each ¡pixel ¡already ¡ 5. Apply border matting (section 4). assigned ¡to ¡FG ¡or ¡ User editing • Edit: fix some pixels either to α n = 0 (background brush) or α n = 1 (foreground brush); update trimap T accord- BG ¡( α n ¡fixed ) ¡ ingly. Perform step 3 above, just once. • Refine operation: [optional] perform entire iterative min- imisation algorithm. 10 ¡ 6-­‑May-­‑13 ¡ Philipp Krähenbühl � Lecture 1 - � �

Initialisation GMM ¡leaning ¡ • User initialises trimap T by supplying only T B . The fore- ground is set to T F = / 0; T U = T B , complement of the back- ground. • Initialise α n = 0 for n ∈ T B and α n = 1 for n ∈ T U . • Mixture ¡param. ¡ • Background and foreground GMMs initialised from sets α n = 0 and α n = 1 respectively. Iterative minimisation F ( k ) π ( α = 1, k ) = 1. Assign GMM components to pixels: for each n in T U , ∑ F ( k ) k n : = argmin D n ( α n , k n , θ , z n ) . k n 2. Learn GMM parameters from data z : k θ : = argmin θ U ( α , k , θ , z ) µ ( α = 1, k ) = mean n ∈ F ( k ) ( z n ) 3. Estimate segmentation: use min cut to solve: min U } min k E ( α , k , θ , z ) . Σ ( α = 1, k ) = cov n ∈ F ( k ) ( z n ) { α n : n ∈ T 4. Repeat from step 1, until convergence. 5. Apply border matting (section 4). • F(k) ¡set ¡of ¡FG ¡ User editing pixels ¡assigned ¡to ¡ • Edit: fix some pixels either to α n = 0 (background brush) or α n = 1 (foreground brush); update trimap T accord- comp. ¡k ¡ ingly. Perform step 3 above, just once. • Refine operation: [optional] perform entire iterative min- imisation algorithm. 11 ¡ 6-­‑May-­‑13 ¡ Philipp Krähenbühl � Lecture 1 - � �