Deep learning 8.4. Networks for semantic segmentation Fran cois - - PowerPoint PPT Presentation

Deep learning 8.4. Networks for semantic segmentation

Fran¸ cois Fleuret https://fleuret.org/ee559/ Nov 2, 2020

The historical approach to image segmentation was to define a measure of similarity between pixels, and to cluster groups of similar pixels.

Fran¸ cois Fleuret Deep learning / 8.4. Networks for semantic segmentation 1 / 9

The historical approach to image segmentation was to define a measure of similarity between pixels, and to cluster groups of similar pixels. Such approaches account poorly for semantic content.

Fran¸ cois Fleuret Deep learning / 8.4. Networks for semantic segmentation 1 / 9

The historical approach to image segmentation was to define a measure of similarity between pixels, and to cluster groups of similar pixels. Such approaches account poorly for semantic content. The deep-learning approach re-casts semantic segmentation as pixel classification, and re-uses networks trained for image classification by making them fully convolutional.

Fran¸ cois Fleuret Deep learning / 8.4. Networks for semantic segmentation 1 / 9

Shelhamer et al. (2016) proposed the FCN (“Fully Convolutional Network”) that uses a pre-trained classification network (e.g. VGG 16 layers). The fully connected layers are converted to 1 × 1 convolutional filters, and the final one retrained for 21 output channels (VOC 20 classes + “background”).

Fran¸ cois Fleuret Deep learning / 8.4. Networks for semantic segmentation 2 / 9

Shelhamer et al. (2016) proposed the FCN (“Fully Convolutional Network”) that uses a pre-trained classification network (e.g. VGG 16 layers). The fully connected layers are converted to 1 × 1 convolutional filters, and the final one retrained for 21 output channels (VOC 20 classes + “background”). Since VGG16 has 5 max-pooling with 2 × 2 kernels, with proper padding, the

• utput is 1/25 = 1/32 the size of the input.

This map is then up-scaled with a de-convolution layer with kernel 64 × 64 and stride 32 × 32 to get a final map of same size as the input image.

Fran¸ cois Fleuret Deep learning / 8.4. Networks for semantic segmentation 2 / 9

Shelhamer et al. (2016) proposed the FCN (“Fully Convolutional Network”) that uses a pre-trained classification network (e.g. VGG 16 layers). The fully connected layers are converted to 1 × 1 convolutional filters, and the final one retrained for 21 output channels (VOC 20 classes + “background”). Since VGG16 has 5 max-pooling with 2 × 2 kernels, with proper padding, the

• utput is 1/25 = 1/32 the size of the input.

This map is then up-scaled with a de-convolution layer with kernel 64 × 64 and stride 32 × 32 to get a final map of same size as the input image. Training is achieved with full images and pixel-wise cross-entropy, starting with a pre-trained VGG16. All layers are fine-tuned, although fixing the up-scaling de-convolution to bilinear does as well.

Fran¸ cois Fleuret Deep learning / 8.4. Networks for semantic segmentation 2 / 9

3d 1 2 , 64d 1 4 , 128d 1 8 , 256d 1 16 , 512d 1 32 , 512d 1 32 , 4096d 2× conv/relu + maxpool 2× conv/relu + maxpool 3× conv/relu + maxpool 3× conv/relu + maxpool 3× conv/relu + maxpool 2× fc-conv/relu VGG without its last layer Fran¸ cois Fleuret Deep learning / 8.4. Networks for semantic segmentation 3 / 9

3d 1 2 , 64d 1 4 , 128d 1 8 , 256d 1 16 , 512d 1 32 , 512d 1 32 , 4096d 2× conv/relu + maxpool 2× conv/relu + maxpool 3× conv/relu + maxpool 3× conv/relu + maxpool 3× conv/relu + maxpool 2× fc-conv/relu 1 32 , 21d 21d fc-conv deconv

×32

Fran¸ cois Fleuret Deep learning / 8.4. Networks for semantic segmentation 3 / 9

Although the FCN achieved almost state-of-the-art results when published, its main weakness is the coarseness of the signal from which the final output is produced (1/32 of the original resolution). Shelhamer et al. proposed an additional element, that consists of using the same prediction/up-scaling from intermediate layers of the VGG network.

Fran¸ cois Fleuret Deep learning / 8.4. Networks for semantic segmentation 4 / 9

3d 1 2 , 64d 1 4 , 128d 1 8 , 256d 1 16 , 512d 1 32 , 512d 1 32 , 4096d 2× conv/relu + maxpool 2× conv/relu + maxpool 3× conv/relu + maxpool 3× conv/relu + maxpool 3× conv/relu + maxpool 2× fc-conv/relu 1 32 , 21d 1 16 , 21d 1 16 , 21d 1 16 , 21d 1 8 , 21d 1 8 , 21d 21d 1 8 , 21d fc-conv deconv

×2

fc-conv fc-conv + deconv

×2

+ deconv

×8

Fran¸ cois Fleuret Deep learning / 8.4. Networks for semantic segmentation 4 / 9

FCN-8s SDS [14] Ground Truth Image

Left column is the best network from Shelhamer et al. (2016).

Fran¸ cois Fleuret Deep learning / 8.4. Networks for semantic segmentation 5 / 9

Image Ground Truth Output Input learning. and 6.3 FCNs tation tion. this upper r images r The P achieve

Results with a network trained from mask only (Shelhamer et al., 2016).

Fran¸ cois Fleuret Deep learning / 8.4. Networks for semantic segmentation 6 / 9

The most sophisticated object detection methods achieve instance segmentation and estimate a segmentation mask per detected object. Mask R-CNN (He et al., 2017) adds a branch to the Faster R-CNN model to estimate a mask for each detected region of interest.

RoIAlign RoIAlign class box conv conv conv conv

Figure 1. The MaskR-CNN framework for instance segmentation.

(He et al., 2017)

Fran¸ cois Fleuret Deep learning / 8.4. Networks for semantic segmentation 7 / 9

horse1.00 horse1.00 horse1.00 bus1.00 bus1.00 car.98 truck.88 car.93 car.78 car.98 car.91 car.96 car.99 car.94 car.99 car.98 truck.86 car.99 car.95 car1.00 car.93 car.98 car.95 car.97 car.87 car.99 car.82 car.78 car.93 car.95 car.97 person.99 traffic light.73 person1.00 person.99 person.95 person.93 person.93 person1.00 person.98 skateboard.82 suitcase1.00 suitcase.99 suitcase.96 suitcase1.00 suitcase.93 suitcase.98 suitcase.88 suitcase.72 stop sign.88 person1.00 person1.00 person1.00 person1.00 person.99 person.99 bench.76 skateboard.91 skateboard.83 handbag.81 surfboard1.00 person1.00 person1.00 surfboard1.00 person1.00 person.98 surfboard1.00 person1.00 surfboard.98 surfboard1.00 person.91 person.74 person1.00 person1.00 person1.00 person1.00 person1.00 person1.00 person.98 person.99 person1.00 person.99 umbrella1.00 person.95 umbrella.99 umbrella.97 umbrella.97 umbrella.96 umbrella1.00 backpack.96 umbrella.98 backpack.95 person.80 backpack.98 bicycle.93 umbrella.89 person.89 handbag.97 handbag.85 person1.00 person1.00 person1.00 person1.00 person1.00 person1.00 motorcycle.72 kite.89 person.99 kite.95 person.99 person1.00 person.81 person.72 kite.93 person.89 kite1.00 person.98 person1.00 kite.84 kite.97 person.80 handbag.80 person.99 kite.82 person.98 person.96 kite.98 person.99 person.82 kite.81 person.95 person.84 kite.98 kite.72 kite.99 kite.84 kite.99 person.94 person.72 person.98 kite.95 person.98 person.77 kite.73 person.78 person.71 person.87 kite.88 kite.88 person.94 kite.86 kite.89 zebra.99 zebra1.00 zebra1.00 zebra.99 zebra1.00 zebra.96 zebra.74 zebra.96 zebra.99 zebra.90 zebra.88 zebra.76 dining table.91 dining table.78 chair.97 person.99 person.86 chair.94 chair.98 person.95 chair.95 person.97 chair.92 chair.99 person.97 person.99 person.94 person.99 person.87 person.99 chair.83 person.94 person.99 person.98 chair.87 chair.95 person.97 person.96 chair.99 person.86 person.89 chair.89 wine glass.93 person.98 person.88 person.97 person.88 person.88 person.91 chair.96 person.95 person.77 person.92 wine glass.94 cup.83 wine glass.94 wine glass.83 cup.91 chair.85 dining table.96 wine glass.91 person.96 cup.98 person.83 dining table.75 cup.96 person.72 wine glass.80 chair.98 person.81 person.82 dining table.81 chair.85 chair.78 cup.75 person.77 cup.71 wine glass.80 cup.79 cup.93 cup.71 person.99 person.99 person1.00 person1.00 frisbee1.00 person.80 person.82 elephant1.00 elephant1.00 elephant1.00 elephant.97 elephant.99 person1.00 person1.00 dining table.95 person1.00 person.88 wine glass1.00 bottle.97 wine glass1.00 wine glass.99 tv.98 tv.84 person1.00 bench.97 person.98 person1.00 person1.00 handbag.73 person.86 potted plant.92 bird.93 person.76 person.98 person.78 person.78 backpack.88 handbag.91 cell phone.77 clock.73 person.99 person1.00 person.98 person1.00 person1.00 person1.00 person.99 person.99 person.99 person1.00 person1.00 person.98 person.99 handbag.88 person1.00 person.98 person.92 handbag.99 person.97 person.95 handbag.88 traffic light.99 person.95 person.87 person.95 traffic light.87 traffic light.71 person.80 person.95 person.95 person.73 person.74 tie.85 car.99 car.86 car.97 car1.00 car.95 car.97 traffic light1.00 traffic light.99 car.99 person.99 car.95 car.97 car.98 car.98 car.91 car1.00 car.96 car.96 bicycle.86 car.97 car.97 car.97 car.94 car.95 car.94 car.81 person.87 parking meter.98 car.89 donut1.00 donut.90 donut.88 donut.81 donut.95 donut.96 donut1.00 donut.98 donut.99 donut.94 donut.97 donut.99 donut.98 donut1.00 donut.95 donut1.00 donut.98 donut.98 donut.99 donut.96 donut.89 donut.96 donut.95 donut.98 donut.89 donut.93 donut.95 donut.90 donut.89 donut.89 donut.89 donut.86 donut.86 person1.00 person1.00 person1.00 person1.00 person1.00 person1.00 person1.00 dog1.00 baseball bat.99 baseball bat.85 baseball bat.98 truck.92 truck.99 truck.96 truck.99 truck.97 bus.99 truck.93 bus.90 person1.00 person1.00 horse.77 horse.99 cow.93 person.96 person1.00 person.99 horse.97 person.98 person.97 person.98 person.96 person1.00 tennis racket1.00 chair.73 person.90 person.77 person.97 person.81 person.87 person.71 person.96 person.99 person.98 person.94 chair.97 chair.80 chair.71 chair.94 chair.92 chair.99 chair.93 chair.99 chair.91 chair.81 chair.98 chair.83 chair.81 chair.81 chair.93 sports ball.99 person1.00 couch.82 person1.00 person.99 person1.00 person1.00 person1.00 person.99 skateboard.99 person.90 person.98 person.99 person.91 person.99 person1.00 person.80 skateboard.98

Figure 5. More results of Mask R-CNN on COCO test images, using ResNet-101-FPN and running at 5 fps, with 35.7 mask AP (Table 1).

(He et al., 2017)

Fran¸ cois Fleuret Deep learning / 8.4. Networks for semantic segmentation 8 / 9

It is noteworthy that for detection and semantic segmentation, there is an heavy re-use of large networks trained for classification. The models themselves, as much as the source code of the algorithm that produced them, or the training data, are generic and re-usable assets.

Fran¸ cois Fleuret Deep learning / 8.4. Networks for semantic segmentation 9 / 9

The end

References

• K. He, G. Gkioxari, P. Doll´

ar, and R. Girshick. Mask R-CNN. In International Conference

• n Computer Vision, pages 2980–2988, 2017.
• E. Shelhamer, J. Long, and T. Darrell. Fully convolutional networks for semantic
• segmentation. CoRR, abs/1605.06211, 2016.