UFO 2 : A Unified Framework towards Omni-supervised Object Detection - - PowerPoint PPT Presentation

UFO2: A Unified Framework towards Omni-supervised Object Detection

Zhongzheng Ren, Zhiding Yu, Xiaodong Yang, Ming-Yu Liu, Alexander G. Schwing, Jan Kautz ECCV 2020

Omni-supervised Object Detection

unlabeled boxes tags TV cat zebra elephant weakly-supervised strongly-supervised

• mni-supervised

commonly used in e.g., segmentation points scribbles

Omni-supervised Object Detection

(Gao et al., 2018) (Uijlings et al., 2018) (Radosavovic et al., 2019)

Prior work

• Stage-wise training; pipelined
• Require some strong labels

Ours

• Unified
• Strong labels are not necessary
• More labels supported

Semi-supervised Object Detection

UFO2: a Unified Framework

Proposals R Image feat. ROI feat. Proposal Refinement

Unified

𝝔 𝒕 𝒕𝒅 𝒕𝒆 𝒕𝒕 𝒕𝒔 Classification Regression Strong supervision Weak Supervision

𝒕𝒕 𝒕𝒔

Classification Regression 𝝔 𝒕 𝒕𝒅 𝒕𝒆 𝒕𝒕 Classification

Partial Labels (tags, points, scribbles)

Pseudo GT 𝓜𝑼𝒃𝒉𝒕 Partial Labels Points & Scribbles

No GT boxes available Teacher heads:

• image-level multi-label classification (𝓜𝑼𝒃𝒉𝒕)

Teacher Student

Generate pseudo-GT online Student heads:

• RoI classification
• RoI regression

Image feat. ROI feat. Proposal Refinement

Unified

𝝔 𝒕 𝒕𝒅 𝒕𝒆 𝒕𝒕 𝒕𝒔 Proposals R

Strong Labels (boxes)

Naïve solution: directly supervise student heads using GT boxes Issue: Weak Teacher & Strong Students

Boxes Image feat. ROI feat. Proposal Refinement

Unified

𝝔 𝒕 𝒕𝒅 𝒕𝒆 𝒕𝒕 𝒕𝒔 Proposals R Teacher Student Strong Labels

Strong Labels (boxes)

𝓜𝑼𝟑 𝓜𝑼𝟐 𝓜𝑼𝒃𝒉𝒕

Make Teacher Great Again!

• Image-level multi-label classification (𝓜𝑼𝒃𝒉𝒕)
• RoI classification (𝓜𝑼𝟐)
• RoI objectness regularization (𝓜𝑼𝟑)

Boxes Image feat. ROI feat. Proposal Refinement

Unified

𝝔 𝒕 𝒕𝒅 𝒕𝒆 𝒕𝒕 𝒕𝒔 Proposals R Teacher Student Strong Labels

Unlabeled Data

• Take confident classes from teacher's prediction
• Then follow “Tags” setting

dog book Image feat. ROI feat.

Unified

𝝔 𝒕 𝒕𝒅 𝒕𝒆 𝒕𝒕 𝒕𝒔 Teacher Student Proposals R Pseudo GT

Inference

• Use only the student heads
• As efficient as standard supervised detectors

Image feat. ROI feat.

Unified

𝒕𝒕 𝒕𝒔 Classification Regression Proposals R Student

Dataset: Partial Labels Simulation

distance transform * gaussian

Region Point Distance Transform Gaussian

distance transform * gaussian

Region Scribble Mask Skeleton

Dataset: Partial Labels Simulation

COCO images & boxes points scribbles

22.7 41.5 24.3 27 29.8 46.4 10 20 30 40 50 tags points scribbles boxes

COCO val-2014 results (AP-50) prior work

• urs

22.7 41.5 10 20 30 40 50 tags points scribbles boxes

COCO val-2014 results (AP-50) prior work

• urs

Experiments: Train from Scratch

Training from scratch (single label)

Experiments: Improve Pre-trained Models

29.1 29.4 30.1 30.9 28 28.5 29 29.5 30 30.5 31 31.5 pre-trained tags points scribbles

COCO minival (AP) Improving pre-trained models (Mixed labels)

32.7 33.9 32 32.2 32.4 32.6 32.8 33 33.2 33.4 33.6 33.8 34 pre-trained unlabeled

COCO minival (AP) COCO-35 COCO-115

Experiments

Tags Points Scribbles Boxes

Budget-aware Omni-supervised Detection

Policy Labels # Labeled Images AP STRONG B + U 2312 + 7688 13.97 ± 0.98 80%B P + B + U 1804 + 1850 + 6346 14.11 ± 1.01 Given a fixed annotation budget (time), the most common strategy:

• STRONG: annotate using only boxes
• Approx. per-img budget on COCO:
• tags: 80s, points: 88.7s, scribbles: 160.4s, boxes: 346s

UFO2 allows a promising new policy:

• N%B: use N% budget for boxes and (1-N%) for points

Thanks for watching!