PartNet: A Large-Scale Benchmark for Fine-Grained and Hierarchical - - PowerPoint PPT Presentation

PartNet: A Large-Scale Benchmark for Fine-Grained and Hierarchical Part-Level 3D Object Understanding

Kaichun Mo1, Shilin Zhu2, Angel Chang3, Li Yi1, Subarna Tripathi4, Leonidas Guibas1, Hao Su2 Presenter: Shilin Zhu2

1Stanford University, 2UC San Diego, 3Eloquent Labs, 4Intel AI Lab

We are Living in a 3D World

3D Understanding Enables Rich Applications

Geometry and Shape Classification Correspondence Generative Model Segmentation This is a Bunny Interaction

3D Deep Learning

“Part” is the Key to Structured 3D Object Understanding

Challenges of 3D Understanding with Parts

Part Definition has Ambiguities

Geometric Functional Manufactural More Challenging for fine-grained parts!

Lacking high-quality 3D part data has been the major bottleneck

Ours outperforms all the existing part datasets in terms of amount of part instances, and is the only

• ne that has hierarchical structures

PartNet Overview

• Total (27,904) out of ShapeNetCore (51,300) covering 24 most common indoor object categories.
• Chair (6,778), Table (8,436), Storage Furniture (2,366), Lamp (2,318), Knife (424), Faucet (744), Display (1,093), Bottle (638),

Microwave (217), Vase (1,198), Clock (651), Bed (233), Bag (149), Bowl (210), Hat (254), Dishwasher (203), Door (247), Earphone (248), Keyboard (213), Laptop (460), Mug (214), Refrigerator (209), Scissors (88), Trashcan (343)

• Num of Diff Types of Parts: keyboard (2), bowls (6), knife (21), Chairs (203), Tables(965), Lamps (1364)
• We leave the outdoor categories for future work.

Hierarchical 3D Part Template

chair seat base back arm

• ther

apron cushion pillow seat surface pedestal_base leg_base rocker_base swivel_base caster_base foot_base

• ther

gas_lift swivel_control star_swivel_base footrest_rig

• ther

leg

• ther

armrest arm support

• ther
• ther

simple back complex back cushion spindle stile splat rail other AND Node: Sub-components Or Node: Sub-categories

……. ……. …….

Give annotators freedom to annotate part/ type “other” since our template cannot cover everything

• ther

Hierarchical 3D Part Template

Rich Categories

Video for Illustration of PartNet Annotation

• Created by: Shilin Zhu, Angel Chang

PartNet can Support Many 3D Tasks

• Segmentation is the key to all part based applications

Fine-Grained Semantic Segmentation on Point Clouds

• Understand locally through discriminative features
• Understand globally in the context of the whole shape

PartNet has the most Fine-Grained Part Annotation Ever!

Benchmark Algorithms

• PointNet
• PointNet++

Benchmark Algorithms

• SpiderCNN
• PointCNN

Encode local Geodesic info Transform points Respond to local shape Instead of arbitrary ordering

Results

PointCNN performs best

Hierarchical Semantic Segmentation on Point Clouds

Ensemble Top-Down Bottom-Up

• Predict semantic part labels in the entire shape hierarchy
• Both coarse and fine-grained parts are covered

PartNet is the only one That has hierarchical structure!

Instance Segmentation on Point Clouds

• Detect every individual part instance and segment it out from shape

PartNet has instance-level annotations for every coarse and fine-grained part!

Benchmark Algorithms

• SGPN
• Ours

Results

Comparison Consistency

Potential Applications Supported

• Computer Vision

3D Object and Scene Understanding

Potential Applications Supported

Potential Applications Supported

• Robotics

Interaction and Task Planning

Potential Applications Supported

• Graphics and Design

Generating Novel Objects with Layouts Shape Modification

Summary

• PartNet: large-scale, fine-grained, hierarchical, instance-level 3D shape

segmentation

• Efficient And-Or-Graph (AOG) to guide annotation workflow
• Propose three shape segmentation benchmarks
• Future directions on annotating mobility to animate objects and

investigating shape grammars for synthesis

Acknowledgement

Grant Fellowships Gifts Annotation Data++