A Good Box is not a Guarantee of a Good Mask Pantone / 185C C 75 / - - PowerPoint PPT Presentation
LVIS Challenge 2020 A Good Box is not a Guarantee of a Good Mask Pantone / 185C C 75 / M 59 / Y 37 / K 0 C 71 / M 63 / Y 60 / K 14 C 0 / M 95 / Y 85 / K 0 R 88 / G 106 / B 135 R 89 / G 89 / B 89 R231 / G 36 / B 39 #586a87 #595959 #e72427
1 Pantone / 185C C 0 / M 95 / Y 85 / K 0 R231 / G 36 / B 39 #e72427 Pantone / 129C C 0 / M 20 / Y 100 / K 0 R253 / G 208 / B 0 #fdd000 C 75 / M 59 / Y 37 / K 0 R 88 / G 106 / B 135 #586a87 C 71 / M 63 / Y 60 / K 14 R 89 / G 89 / B 89 #595959
Jingru Tan1 Gang Zhang2 Hanming Deng3 Changbao Wang3 Lewei Lu3 Quanquan Li3 LVIS Challenge 2020
1Tongji University 2Tsinghua University 3Sensetime Research
2
Long tail distribution High quality mask annotations
Representation learning stage Fine-tuning stage
Inconsistent annotations Objects that are hard to represent with boxes
Improvements & tricks
3
Long tail distribution High quality mask annotations
4
Classifier is heavily biased towards head categories. Tail categories are hard to classify
5
6
Long tail distribution High quality mask annotations
Representation learning stage Fine-tuning stage
7
8
Equalization Loss for Long-tailed Object Recognition, CVPR 2020
YOLOv4: Optimal Speed and Accuracy of Object Detection, Arxiv preprint
LVIS: A Dataset for Large Vocabulary Instance Segmentation , CVPR 2019
9
To further enhance the model performance, pseudo label is inferenced on LVIS and external datasets like Open Images for self-training. For self-training, we ignore all proposals matched with the pseudo boxes.
AP@Seg m AP@r AP@ c AP@f AP@BBox Baseline
26.2 17.1 26.2 30.2 27.0
Open Images 26.8 17.5 27.2 30.5 28.1 Ignore LVIS pseudo 26.8 17.0 27.1 30.9 27.8
Pseudo Label
10
Classifier is heavily biased towards head categories. Balanced Group Softmax
Overcoming Classifier Imbalance for Long-tail Object Detection with Balanced Group Softmax. CVPR 2020
Category Frequency BBox AP Mask AP Area Mask/BBox Mask AP - BBox AP coatrack common 73.6 10.1 0.29
tripod frequent 40.9 3.8 0.22
necklace frequent 32.8 3.0 0.17
ski pole frequent 36.2 7.5 0.15
fork frequent 47.2 21.9 0.26
windshield wiper frequent 29.6 5.2 0.26
giraffe frequent 79.2 60.7 0.33
Mask AP - BBox AP < -5.0: 270 of 1203 categories (~25%) Mask AP - BBox AP < -5.0 and Area Mask/BBox < 0.5: 168 of 1203 categories
Large BBox, Small Mask
11
12
The smaller the mask/bbox ratio, the larger the gap between mask and bbox AP.
13
Tripod Ski pole Giraffe
14
New Strategy for Mask Proposal Assignment
AP@Segm AP@r AP@c AP@f AP@BBox Baseline
34.7 26.1 34.9 38.1 37.6
Ratio Assign 35.0 26.4 35.2 38.6 37.6
Feature Pyramid Networks for Object Detection, CVPR 2017
15 AP@Segm AP@r AP@c AP@f AP@BBox Baseline
34.7 26.1 34.9 38.1 37.6
+BML 35.0 26.1 35.3 38.5 37.6
Balanced Mask Loss: foreground/background imbalance
16 Category Frequency Area Mask/BBox AP Gap AP Gap (Ours) Improvemen t coatrack common 0.29
+3.4 tripod frequent 0.22
+4.0 necklace frequent 0.17
+2.1 ski pole frequent 0.15
+4.2 fork frequent 0.26
+4.0 windshield wiper frequent 0.26
+3.3 giraffe frequent 0.33
+4.1
However, it is still an open problem. Leave the further research in the future.
17
AP@Segm AP@r AP@c AP@f AP@BBox Baseline 34.7 26.1 34.9 38.1 37.6 + Ratio Assign 35.0 26.2 35.2 38.5 37.6 + Balanced Mask Loss 35.2 26.0 35.4 38.9 37.6 + Boundary Supervision* 35.6 26.9 35.6 39.3 37.6 + 7 Convs for Mask Head 35.8 26.8 35.9 39.6 37.6 + Deformable RoI Pooling 36.1 28.8 35.8 39.8 38.3
Boundary-preserving Mask R-CNN, ECCV 2020
18
Long tail distribution High quality mask annotations
Representation learning stage Fine-tuning stage
Improvements & tricks
19
19.2 baseline
20
19.2 baseline 20.3 data augmentation
21
baseline 19.2 20.3 data augmentation 22.4 EQL baseline
22
19.2 baseline 20.3 data augmentation 22.4 EQL 26.2 RFS
23
baseline 19.2 20.3 data augmentation 22.4 EQL 26.2 RFS 28.8 HTC baseline
24
19.2 baseline 20.3 data augmentation 22.4 EQL 26.2 RFS 28.8 HTC 32.0 S101 & DCN
25
19.2 baseline 20.3 data augmentation 22.4 EQL 26.2 RFS 28.8 HTC 32.0 S101 & DCN 33.2 tricks
Make sampling probability in mosaic align with RFS Make rotated boxes align with rotated masks
26
19.2 baseline 20.3 data augmentation 22.4 EQL 26.2 RFS 28.8 HTC 32.0 S101 & DCN 33.2 tricks 33.7 self training
27
19.2 baseline 20.3 data augmentation 22.4 EQL 26.2 RFS 28.8 HTC 32.0 S101 & DCN 33.2 tricks 33.7 self training 36.5 mask scoring + pseudo ignore + S269
28
19.2 baseline 20.3 data augmentation 22.4 EQL 26.2 RFS 28.8 HTC 32.0 S101 & DCN 33.2 tricks 33.7 self training 36.5 mask scoring + pseudo ignore + S269 37.6 balanced group softmax
29
19.2 baseline 20.3 data augmentation 22.4 EQL 26.2 RFS 28.8 HTC 32.0 S101 & DCN 33.2 tricks 33.7 self training 36.5 misc 37.6 balanced group softmax 38.8 high quality mask
misc: mask scoring, pseudo ignore, ResNeSt269
30
19.2 baseline 20.3 data augmentation 22.4 EQL 26.2 RFS 28.8 HTC 32.0 S101 & DCN 33.2 tricks 33.7 self training 36.5 misc 37.6 balanced group softmax 38.8 high quality mask 41.5 TTA
TTA: (1) multi-scale testing (2) scale-aware inference (3) revised Softnms
31
Long tail distribution High quality mask annotations
Representation learning stage Fine-tuning stage
Inconsistent annotations Objects that are hard to represent with boxes
Improvements & tricks
32
Fire Hose (Mask AP 3.9) Hose (Mask AP 6.5)
33
Stirrup (Mask AP 1.2) Hook (Mask AP 7.3)
34
Crib (Mask AP - BBox AP = -51.6)
35 Pantone / 185C C 0 / M 95 / Y 85 / K 0 R231 / G 36 / B 39 #e72427 Pantone / 129C C 0 / M 20 / Y 100 / K 0 R253 / G 208 / B 0 #fdd000 C 75 / M 59 / Y 37 / K 0 R 88 / G 106 / B 135 #586a87 C 71 / M 63 / Y 60 / K 14 R 89 / G 89 / B 89 #595959
LVIS Challenge 2020