PartNet: A Large-Scale Benchmark for Fine-Grained and Hierarchical - PowerPoint PPT Presentation

PartNet: A Large-Scale Benchmark for Fine-Grained and Hierarchical Part-Level 3D Object Understanding Kaichun Mo 1 , Shilin Zhu 2 , Angel Chang 3 , Li Yi 1 , Subarna Tripathi 4 , Leonidas Guibas 1 , Hao Su 2 Presenter: Shilin Zhu 2 1 Stanford University, 2 UC San Diego, 3 Eloquent Labs, 4 Intel AI Lab

We are Living in a 3D World

3D Understanding Enables Rich Applications This is a Bunny Geometry and Shape Classification Segmentation 3D Deep Learning Correspondence Generative Model Interaction

“Part” is the Key to Structured 3D Object Understanding

Challenges of 3D Understanding with Parts

Part Definition has Ambiguities Geometric Functional Manufactural More Challenging for fine-grained parts!

Lacking high-quality 3D part data has been the major bottleneck Ours outperforms all the existing part datasets in terms of amount of part instances, and is the only one that has hierarchical structures

PartNet Overview - Total (27,904) out of ShapeNetCore (51,300) covering 24 most common indoor object categories. - Chair (6,778), Table (8,436), Storage Furniture (2,366), Lamp (2,318), Knife (424), Faucet (744), Display (1,093), Bottle (638), Microwave (217), Vase (1,198), Clock (651), Bed (233), Bag (149), Bowl (210), Hat (254), Dishwasher (203), Door (247), Earphone (248), Keyboard (213), Laptop (460), Mug (214), Refrigerator (209), Scissors (88), Trashcan (343) - Num of Diff Types of Parts : keyboard (2), bowls (6), knife (21), Chairs (203), Tables(965), Lamps (1364) - We leave the outdoor categories for future work.

Hierarchical 3D Part Template Give annotators freedom to annotate part/ type “other” since our template cannot cover everything chair arm other seat base back other seat cushion pillow apron arm other armrest surface support foot_base ……. ……. simple rocker_base complex other pedestal_base back back other caster_base swivel_base leg_base cushion spindle stile splat rail other ……. gas_lift footrest_rig other star_swivel_base swivel_control AND Node: Sub-components Or Node: Sub-categories leg other

Hierarchical 3D Part Template

Rich Categories

Video for Illustration of PartNet Annotation • Created by: Shilin Zhu, Angel Chang

PartNet can Support Many 3D Tasks • Segmentation is the key to all part based applications

Fine-Grained Semantic Segmentation on Point Clouds PartNet has the • Understand locally through discriminative features most Fine-Grained Part Annotation • Understand globally in the context of the whole shape Ever!

Benchmark Algorithms • PointNet • PointNet++

Benchmark Algorithms Encode local Geodesic info • SpiderCNN • PointCNN Transform points Respond to local shape Instead of arbitrary ordering

Results PointCNN performs best

Hierarchical Semantic Segmentation on Point Clouds • Predict semantic part labels in the entire shape hierarchy PartNet is the only one That has hierarchical • Both coarse and fine-grained parts are covered structure! Ensemble Top-Down Bottom-Up

Instance Segmentation on Point Clouds • Detect every individual part instance and segment it out from shape PartNet has instance-level annotations for every coarse and fine-grained part!

Benchmark Algorithms • SGPN • Ours

Results Comparison Consistency

Potential Applications Supported • Computer Vision 3D Object and Scene Understanding

Potential Applications Supported

Potential Applications Supported • Robotics Interaction and Task Planning

Potential Applications Supported • Graphics and Design Generating Novel Objects with Layouts Shape Modification

Summary • PartNet: large-scale, fine-grained, hierarchical, instance-level 3D shape segmentation • E ffi cient And-Or-Graph (AOG) to guide annotation workflow • Propose three shape segmentation benchmarks • Future directions on annotating mobility to animate objects and investigating shape grammars for synthesis

Acknowledgement Grant Gifts Annotation Fellowships Data++