Toward Large-Scale Image Segmentation On Summit Sudip K. Seal , - - PowerPoint PPT Presentation

β–Ά
toward large scale image segmentation on summit
SMART_READER_LITE
LIVE PREVIEW

Toward Large-Scale Image Segmentation On Summit Sudip K. Seal , - - PowerPoint PPT Presentation

Jan 17, 2023 217 likes β€’350 views

Toward Large-Scale Image Segmentation On Summit Sudip K. Seal , Seung-Hwan Lim, Dali Wang, Jacob Hinkle, Dalton Lunga, Aristeidis Tsaris Oak Ridge National Laboratory, USA August 19, 2020 International Conference on Parallel Processing Alberta,

slide-1
SLIDE 1

ORNL is managed by UT-Battelle, LLC for the US Department of Energy

Toward Large-Scale Image Segmentation On Summit

Sudip K. Seal, Seung-Hwan Lim, Dali Wang, Jacob Hinkle, Dalton Lunga, Aristeidis Tsaris Oak Ridge National Laboratory, USA August 19, 2020 International Conference on Parallel Processing Alberta, Canada

slide-2
SLIDE 2

2 2

Introduction

Semantic Segmentation of Images

Ø Given an image with 𝑂×𝑂 pixels and a set of 𝑙 distinct classes, label each of the 𝑂! pixels with one of the 𝑙 distinct classes. Ø For example, given a 256 Γ—256 image of a car, road, buildings and people, a semantic segmentation of the image classifies each of the 256Γ—256 = 2"# pixels into one of 𝑙 = 4 classes {car, road, building, people}.

Input Image Segmented Image

Image credit: https://mc.ai/how-to-do-semantic-segmentation-using-deep-learning/ Semantic Segmentation

conv 3 x 3, ReLU max pool, 2 x 2 up-conv 2 x 2 conv 1 x 1 copy and crop

Input image Segmented image

slide-3
SLIDE 3

3 3

The U-Net Model

𝑂 𝑂 𝑂 βˆ’ πœ— 𝑂 βˆ’ πœ—

U-Net Architecture

Input Image Output Image U-Net: Convolutional Networks for Biomedical Image Segmentation Olaf Ronneberger, Philipp Fischer, Thomas Brox Medical Image Computing and Computer- Assisted Intervention (MICCAI), Springer, LNCS, Vol.9351: 234--241, 2015.

  • Refer to this as the πœ— βˆ’ region (halo).
  • Halo width (πœ—) is a function of U-Net

architecture (depth, channel width, filter sizes, etc.).

  • Halo width (πœ—) determines the

receptive field of the model.

  • Larger the receptive field, wider the

length-scales of identifiable objects.

πœ— = 3 β‹… 2$%! βˆ’ 1 π‘’π‘œ&

slide-4
SLIDE 4

4 4

Why Is It A Summit-scale Problem?

Ø Satellite images collected at high-resolutions (30-50 cm) yield very large 10,000 x 10,000 images. Ø Most computer vision workloads deal with images of 𝑃(10!Γ— 10!) resolution (for example, ImageNet). Ø This work targets ultra-wide extent images with 𝑃(10'Γ—10') resolution β‡’ 10,000-fold larger data samples! Ø At present, requires many days to train a single model (even

  • n special-purpose DL platforms like DGX boxes).

Ø Hyperparameter tuning of these models take much longer. Ø Need accurate scalable high-speed training framework. Ø Large U-Net models are needed to resolve multi-scale objects (buildings, solar panels, land cover details). Ø Advanced DAQ systems generate vast amounts of high- resolution images β‡’ large data volume.

Sample size Model size Data size 10000-fold larger image size Larger receptive fields require larger models Multi-TB of data from DAQ systems.

slide-5
SLIDE 5

5 5

Sample Parallelism - Taming Large Image Size

Leveraging Summit’s Vast GPU Farm

1 , 10,000

Tile size chosen such that appended tile plus model parameters fit on a single Summit GPU.

Ø Given a 𝑂×𝑂 image, U-Net segments a 𝑂 βˆ’ πœ— Γ—(𝑂 βˆ’ πœ—) inset square. Ø Partition each 𝑂×𝑂 = 10000Γ—10000 image sample into non-overlapping tiles. Ø Append an extra halo region of width πœ— along each side of each tile. Ø Assign each appended tile to a Summit

  • GPU. Use standard U-Net to segment

appended tile. Ø Each GPU segments an area equal to that

  • f the original non-overlapping tile.

𝑂 𝑂 𝑂 βˆ’ πœ— 𝑂 βˆ’ πœ—

Blue dashed square is segmented for each appended tile.

slide-6
SLIDE 6

6 6

Ø Optimal tiling for each 10000Γ—10000 sample image was found to be 8Γ—8. Ø Each 1250Γ—1250 tile was appended with a halo of width πœ— = 92 and assigned to a single Summit GPU. Ø 10 – 11 Summit nodes to train each 10000Γ— 10000 image sample. Ø A U-Net model was trained on a data set of 100 10000Γ—10000Γ—4 satellite images, collected at 30- 50 cm resolution. Ø The training time per epoch was shown to be ∼12 seconds using 1200 Summit GPUs compared to ∼1,740 seconds on a DGX-1. Ø Initial testing revealed no appreciable loss of training/validation accuracy using the new parallel framework.

Performance of Sample-Parallel U-Net Training

+100X Faster U-Net Training

slide-7
SLIDE 7

7 7

Β§ 𝐿 β†’ πΊπ‘—π‘šπ‘’π‘“π‘  𝑑𝑗𝑨𝑓 Β§ 𝑇 β†’ 𝑇𝑒𝑠𝑗𝑒𝑓 π‘šπ‘“π‘œπ‘•π‘’β„Ž Β§ 𝑄 β†’ π‘„π‘π‘’π‘’π‘—π‘œπ‘• π‘šπ‘—π‘¨π‘“ Β§ π‘œ! β†’ 𝑂𝑝. 𝑝𝑔 π‘‘π‘π‘œπ‘€π‘‘ π‘žπ‘“π‘  π‘šπ‘“π‘€π‘“π‘š Β§ 𝑀 β†’ π‘œπ‘. 𝑝𝑔 𝑉𝑂𝑓𝑒 π‘šπ‘“π‘€π‘“π‘šπ‘‘ Β§ 𝑂×𝑂 = π‘Ÿ"(π‘ˆ#Γ—π‘ˆβ€²)

Limitations of Sample Parallelism

N = 1 , 𝑂 = 10,000 π‘ˆ β€² Γ— π‘ˆ β€² π‘ˆΓ—π‘ˆ π‘ˆΓ—π‘ˆ π‘ˆβ€²Γ—π‘ˆβ€²

Ø An image of size 𝑂× 𝑂 is partitioned into a π‘ŸΓ—π‘Ÿ array of π‘ˆ(Γ—π‘ˆ( tiles. Ø 𝐹 ∼

)*+,- .*-/01 *2 &*03/+,+4*56 317 +4-1 )*+,- .*-/01 *2 /612/- &*03/+,+4*56 317 +4-1 = 𝑃 )! )"! ∼ 𝑃 1 + π‘Ÿ '8 9

Ø Ideally, 𝐹 = 1. Ø Decreasing π‘Ÿ (increasing tile sizes) increases the memory requirement and quickly overtakes memory available per GPU. Ø Decreasing πœ— decreases the receptive field of the model. Ø On the other hand, the goal is to decrease π‘Ÿ and increase πœ—. Ø Decrease π‘Ÿ β‡’ increasing tile size π‘ˆβ€² and decreasing πœ— steers away from target receptive fields. Ø To satisfy both, larger U-Net models than can fit on a GPU needed. Ø Need model-parallel execution. 𝑂 = π‘Ÿπ‘ˆ( πœ— = 3 β‹… 2$%! βˆ’ 1 π‘’π‘œ& 𝑒 = 9 :%" ;<%!=

:

βˆ’ 1

slide-8
SLIDE 8

8 8

TorchGPipe: PyTorch implementation of Gpipe* Framework

Model-Parallelism - Taming Large Model Size

Node-level Pipeline-Parallel Execution

GPU 1 GPU 2 GPU 3 GPU 4 GPU 5 GPU 6 No load balance

  • - skip connections omitted for ease of presentation --

!

!!

!

!"

!

!#

!

!$

!

"!

!

""

!

"#

!

"$

!

#!

!

#"

!

##

!

#$

!

$!

!

$"

!

$#

!

$$

"$$ "$# "$" "$! "#$ "## "#" "#! ""$ ""# """ ""! "!$ "!# "!" "!! #! #" ## #$ $! $" $# $$ $% $& $' $( $) $* $"! $"" $"# $"$ $"%

Update Update Update Update

$"&

Single Summit Node

Ø Number of consecutive layers mapped to a GPU is called partition. Ø Number of layers in each partition is called balance. Ø Subdivide each mini-batch of tiles into smaller micro- batches that are assigned to each partition. Ø Micro-batches per partition ≑ π‘›π‘π‘žπ‘ž Memory needed/GPU = size(micro-batch) + size(partition)

* Huang, Yanping, Yonglong Cheng, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le and Zhifeng Chen. β€œGPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism.” NeurIPS (2019).

slide-9
SLIDE 9

9 9

Model Parallel Experiments

Single Node Execution

SAMPLE PARALLEL

!Γ—! Image Samples #Γ—# Padded Tiles

!Γ—! !Γ—! !Γ—! !β€²Γ—!β€² !β€²Γ—!β€² !β€²Γ—!β€²

MODEL PARALLEL U-NET 96 GB SUMMIT NODE

!Γ—!

Model

  • No. of

Levels

  • Conv. Layers

Per Level

  • No. of Trainable

Parameters 𝝑 Small (Standard) 5 2 72,301,856 92 Medium-1 5 5 232,687,904 230 Medium-2 6 2 289,357,088 188 Large 7 2 1,157,578,016 380

  • 10Γ— larger number of trainable parameters.
  • 4Γ— fold larger receptive field.

Benchmark U-Net Models

Medium -1 πŸ‘. πŸ—Γ— (192), πŸ‘. πŸ”Γ— (512) and πŸ‘Γ— (1024) speedup using 6 pipeline stages. Speedup doubles (small: 1.97; medium-2: 2.01) as no. of pipeline stages increases from 1 to 6.

slide-10
SLIDE 10

10 10

Need for Performance Improvement

Single Node Execution

Ø Small, Medium-2 and Large Models: Ø Layers: 109, 129 and 149. Ø Balances: small {14, 24, 30, 22, 12, 7}; medium-2 {16, 26, 38, 26, 12, 11}; large {18, 30, 44, 30, 14, 13}. Ø Need load balanced pipelined execution. Ø Encoder memory: Ø Decoder memory: Ø Memory profile: 𝐹ℓ + 𝐸ℓ( vs. β„“ , β„“( = 𝑀 βˆ’ β„“

E` = O I2

` + 2`nf nc

X

i=1

(I` βˆ’ i Β· d)2 ! D`0 = O 2`0nf 2I2

`0 + nc

X

i=1

(I`0 βˆ’ i Β· d)2 !!

slide-11
SLIDE 11

11 11

Wrapping Up

SAMPLE PARALLEL

!Γ—! Image Samples #Γ—# Padded Tiles

MODEL PARALLEL U-NET MODEL PARALLEL U-NET MODEL PARALLEL U-NET MODEL PARALLEL U-NET 96 GB SUMMIT NODE

DATA PARALLEL

!Γ—! !Γ—! !Γ—! !Γ—! !Γ—! !Γ—! !Γ—! !β€²Γ—!β€² !β€²Γ—!β€² !β€²Γ—!β€²

Ongoing Work: Sample + Model + Data Parallel Framework Load Balance Heuristics Data Parallelism

SAMPLE PARALLEL

!Γ—! Image Samples #Γ—# Padded Tiles

!Γ—! !Γ—! !Γ—! !β€²Γ—!β€² !β€²Γ—!β€² !β€²Γ—!β€²

MODEL PARALLEL U-NET 96 GB SUMMIT NODE

!Γ—!

This Paper: Prototype Sample + Model Parallel Framework

  • πŸπŸΓ— larger number of trainable parameters.
  • πŸ“Γ— fold larger receptive field.
  • πŸπŸπŸπŸπŸΓ— larger image size.

Ø Training image segmentation neural network models become extremely challenging when: Ø Image sizes are very large Ø Desired receptive fields are large Ø Volume of training data is large. Ø Fast training/inference needed for geo-sensing applications –satellite imagery, disaster assessment, precision agriculture, etc. Ø This work is a first step – can train 10Γ— larger U-Net models with 4Γ— larger receptive field on 10000Γ— larger images. Ø Ongoing efforts are underway to integrate load balancing heuristics and data-parallel execution to handle large volumes of training data efficiently.

slide-12
SLIDE 12

12 12

THANK YOU