Temporally Distributed Networks for Fast Video Semantic Segmentation - - PowerPoint PPT Presentation

Temporally Distributed Networks for Fast Video Semantic Segmentation

Ping Hu1 Fabian Caba Heilbron2 Oliver Wang2 Zhe Lin2 Stan Sclaroff1 Federico Perazzi2

1Boston University 2Adobe Research

❏ Video Semantic Segmentation

❏ High data volume ❏ Content redundancy ❏ Spatial-temporal variations between frames ❏ Requiring: (1) High Accuracy; (2) High Speed; (3) Low-latency;

Challenge

frame {..., T-1, T, T+1, …} frame {..., T-1, T, T+1, …}

Main Contribution & Novelty

❏ Temporally distributed network ⇨ Low-latency video processing. ❏ Attention propagation mechanism ⇨ Robust feature aggregation. ❏ Grouped knowledge distillation ⇨ Effective model training. TDNet - SOTA in accuracy and speed.

Temporally Distributed Networks

Temporally Distributed Networks

Temporally Distributed Networks

Temporally Distributed Networks

❏ Challenge: Pixelwise tasks are sensitive to the spatial misalignment caused by motion between frames.

❏ Attention propagation

Temporally Distributed Networks

❏ Challenge: Pixelwise tasks are sensitive to the spatial misalignment caused by motion between frames.

❏ Attention propagation

Temporally Distributed Networks

❏ Challenge: Pixelwise tasks are sensitive to the spatial misalignment caused by motion between frames. ❏ Attention Propagation: ❏ Attention Downsampling: Saving computation by downsample the reference data in attention.

Temporally Distributed Network

❏ Grouped Knowledge Distillation

❏ Transfer knowledge at the subspace level. ❏ Enhance the complementarity of sub-feature maps in the full feature space.

Approaches & Challenge

Previous Methods Our TDNet Key-frame Temporal-context Overall-accuracy × √ √ Overall-speed √ × √ Low-latency × × √