53023 - EGLSTREAMS: INTEROPERABILITY FOR CAMERA, CUDA AND OPENGL - PowerPoint PPT Presentation

53023 - EGLSTREAMS: INTEROPERABILITY FOR CAMERA, CUDA AND OPENGL Debalina Bhattacharjee Sharan Ashwathnarayan

Tegra SOC and typical use-cases • Why Interops • EGLStream and Its Key Features • Examples on EGLStream • Agenda Connect EGLStream to NvMedia and CUDA • Perform CUDA processing on Camera inputs • Connect EGLStream to NvMedia and OpenGL • Display with OpenGL. • Future Scope • 2

TRY IT OUT!!! Download: scp – r nvidia_1b@10.30.25.175:GTCEGL12OCT /home/nvidia Open the pdf: Go to /home/nvidia/GTCEGL12OCT/ GTCEGL12OCT.pdf Run: cd /home/nvidia/GTCEGL12OCT/ export DISPLAY=:0 chmod +x ./x11/egl_stream_demo IF Monitor has A on it: ./x11/egl_stream_demo A IF Monitor has B on it: ./x11/egl_stream_demo B 3

TEGRA SOC Tegra SOC engines CPU COMPLEX GEFORCE GPU Armv8 CPU Geforce GPU ISP Video Encode SECURITY VIDEO VIDEO AUDIO 2D ENGINE ENGINE ENCODER DECODER ENGINE (APE) (VIC) Video Decode And More.. SAFETY SAFETY ENGINE MANAGER BOOT PROC (BPMP) CAN PROC (SPE) IMAGE PROC (ISP) (SCE) (HSM) I/O 4

WHY IS INTEROP NEEDED? Different API and libraries for different engines Processor API GPU CUDA OpenGL ISP Argus NvMedia 5

INTEROP Same physical memory shared between different API and no copy involved • Avoid No Interop With Interop Memcpy = Perf Gain memcpy buffer buffer buffer Memory MAP Memory Single Buffer = ISP GPU Less ISP GPU Memory FootPrint NvMedia NvMedia Synchronize 6

AUTOMOTIVE USE CASES Typical use cases utilize all resources on SOC ISP GPU GPU Capture Display compute Video GPU GPU Decode compute Display ISP GPU Video Capture compute Encode 7

INTEROP BETWEEN APIS Engines Need To Talk With Each Other CUDA (GPU) NvMedia ? (ISP) OpenGL (GPU) EGLDisplay Argus (Display) (ISP) 8

EGLSTREAM Unified interface to communicate between multiple APIs CUDA (GPU) NvMedia (ISP) OpenGL (GPU) EGLStream Argus EGLDisplay (ISP) (Display) 9

EGLSTREAM – HOW IT WORKS Transfer a sequence of image frames from one API to another. Producer-Consumer Model Enables and Accepts Produces hides details of frame Frames buffer transport Buffer Buffer EGLStream Consumer Producer 10

EGLSTREAMS Usage Semantics: • PresentFrame() B Acquire() Producer Consumer EGL stream Release() ReturnFrame() Buffer/Allocation ownership B 11

NVMEDIA_CUDA API SEQUENCE Consumer Producer CUDAConsumerConnect Consumer NvMediaProducerConnect connects first NVMEDIA BLIT() Consumer waits for a frame to presented Implicit Synchronization NvMediaProducerPostImage CUDAConsumerAcquireFrame Run CUDA A returned frame kernel on CUDA Kernel is safe to be acquired presented again frame Release the frame CUDAConsumerReleaseFrame NvMedia ProducerGetImage Released frame is Returned to producer Implicit Synchronization CUDAEGLStreamConsumerDisconnect NvMediaProducerDisconnect 12

CUDA_EGLSTREAM INTEROP Key Advantages Performance improvement – no memcpy needed with iGPU • • Less Memory footprint – single buffer is shared with mapping Ease of use - Support for implicit synchronization • Cross – Process support - Producer/Consumer can be in different processes (IPC) • No need of individual interop – unified interface • Portable across cameras - Support for both Interleaved and Multi-planar format • Supported Platforms • 13

EGLSTREAMS CUDA 9.0 support Support for Discrete GPU on DrivePX 2 • Transfer buffers from camera to iGPU or dGPU efficiently • Support on x86/x86_64 Linux • Support added for easier development • Additional YUV multiplanar color formats • 14

DEMO APPLICATION Built on Vibrante 4.1.8.0 • • Uses NvMedia for Producer CUDA used for compute processing • • OpenGL used for Display One EGLStream per Camera • 15

IMPORTANT Don’t pull out the camera! • • If you are confused about which file to edit, call TA. Call for TA’s help if something is wrong. • • Refer to README file for details. (Especially for killing the app) 16

DEMO APPLICATION WorkFlow: NVMEDIA  CUDA Nvmedia APIs capture Image & Presents it to CUDA • • CUDA acquires the image, runs fisheye correction & YUV to RGB conversion Hands over the image to googleNet inference Engine. • • Inference on the image done & result reported. CUDA releases the acquired Image. • NvMedia accepts the returned image. • • Cycle continues. 17

NVMEDIA_CUDA API SEQUENCE Consumer cuda_consumer.c Producer img_producer.c cuEGLStreamConsumerConnect Line: 256 NvMediaEglStreamProducerCreate Line: 309 Line: 152 NvMediaEglStreamProducerPostImage cuEGLStreamConsumerAcquireFrame Line: 133 inferSingleFrame Line: 163 NvMediaEglStreamProducerGetImage cuEGLStreamConsumerReleaseFrame Line: 180 Line: 178 Line: 319 cuEGLStreamConsumerDisconnect Line: 352 NvMediaEglStreamProducerDestroy 18

NVMEDIA – CUDA Build: cd /home/nvidia/GTCEGL12OCT/ make Run: export DISPLAY=:0 IF Monitor has A on it: ./x11/egl_stream_demo A IF Monitor has B on it: ./x11/egl_stream_demo B 19

OTHER CONSUMERS 20

DEMO APPLICATION WorkFlow: NVMEDIA  GL Nvmedia APIs capture Image & Presents it to CUDA • • GL acquires the image & renders it to the DISPLAY. GL releases the acquired Image. • • NvMedia accepts the returned image. Cycle continues. • 21

APPLICATION Main/Camera Producer Thread GL Consumer Thread 1. Initialize Camera resources, create an 1. Initialize helper GL resources EglStream 2. Create GLConsumer 2. Launch OpenGL Consumer thread and 3. Connect GLConsumer to the EglStream pass the EglStream to it 4. Loop 3. Connect NvMediaProducer to EGL stream 1. Acquire frame from the EGLStream. Wait if frame is not available 4. Loop 2. Render the acquired frame 1. Post frame on NvMediaProducer 3. Release the acquired frame 22

REPLACE CUDA CONSUMER WITH GL CONSUMER cd /home/nvidia/GTCEGL12OCT/ Open interop.c with an Editor Comment out cuda_consumer.h & uncomment gl_consumer.h Search and Replace CudaConsumer with GlConsumer 23

NVMEDIA - GL Build: cd /home/nvidia/GTCEGL12OCT/ make Run: export DISPLAY=:0 IF Monitor has A on it: ./x11/egl_stream_demo A IF Monitor has B on it: ./x11/egl_stream_demo B 24

THINGS TO TRY More complex pipelines Integrated GPU EGL Stream Camera CUDA CUDA CUDA Producer Consumer Processing Producer EGL Stream discrete GPU EGL Stream OpenGL CUDA CUDA CUDA Consumer Producer inference Consumer 25

REFERENCE EGL_KHR_stream: https://www.khronos.org/registry/EGL/extensions/KHR/EGL_KHR_stream.txt • • EGL_KHR_stream_consumer_gltexture: https://www.khronos.org/registry/EGL/extensions/KHR/EGL_KHR_stream_consumer_gltexture.txt • CUDA: http://docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__EGL.html#group__CUDA__EGL • NvMedia : https://developer.nvidia.com/embedded 26

THANK YOU