GENERATION USING 3D PIPELINES Stefan Schoenefeld, GTC 2017 - PowerPoint PPT Presentation

SYNTHETIC TRAINING IMAGE GENERATION USING 3D PIPELINES Stefan Schoenefeld, GTC 2017 sschoenefeld@nvidia.com

TRAINING WITH GENERATED IMAGES GENERATE TRAIN DETECT 2

…BUT WHY? ACCELERATE ANNOTATE SIMULATE Rendering is fast Annotation is trivial Environmental effects Unlimited amount of Bounding boxes Create new scenarios combinations of cameras, lights and objects Image segmentation 3 SYNTHIA Dataset

CASE STUDY IMAGE SEGMENTATION 4

WORKFLOW 3D OBJECT DATASET NVIDIA DIGITS 5.0 3D OBJECT OBJECT DATASET SYNTH FCN PLUGIN TRAIN CREATE IMAGE DATASET NVPRO-PIPELINE SHUFFLE 5

SIMPLE 3D OBJECTS PROOF OF CONCEPT 6

RESULTS 7

CASE STUDY IMAGE CATEGORISATION 8

NVIDIA BOXES BOX DETECTION Create 3D dataset from artwork Textures instead of shapes Render using IRAY/UE4 Train network to categorise 9

NVIDIA BOXES ...they all look the same QUADRO GP100 QUADRO M6000 24GB QUADRO P600 QUADRO M4000 10

SETUP - Iterative, start simple and increase complexity - Raytracing on an iray cluster - UE4 for „traditional“ rendering - 1024x10124 images, ~2500 images per category, 15 categories - Using a googlenet DNN with a random crop of 768x768 11

GOOD RESULTS 12

BAD RESULTS 13

REAL WORLD ISSUES SHRINK WRAP 14

RESULTS CTD AFTER ADDING SHRINK WRAP TO THE RENDERING 15

NEXT STEPS 16

NEXT STEPS FULLY INTEGRATED MULTIPLE OBJECTS Everything in the same framework Combining and creating natural scenes Advanced positioning ON DEMAND Real time generation while training No more „Out -of- space“ Scene overhead 17

THANK YOU AND ENJOY THE PARTY!

REFERENCES Stefan Schoenefeld sschoenefeld@nvidia.com NVIDIA Digits https://github.com/NVIDIA/DIGITS NVPro-Pipeline https://github.com/nvpro-pipeline/pipeline Synthia Dataset http://synthia-dataset.net/ 19