AI Argus A Unique Insight Into Logistics cs Neo Song SF Technology - PowerPoint PPT Presentation

AI Argus A Unique Insight Into Logistics cs Neo Song SF Technology · Department of Computer Vision 2019.02

CONTENT AI Argus Introduction • Scenario Analysis and Algorithm Design • Acceleration with NVIDIA • Future Planning •

AI Argus A Unique Insight Into Logistics cs Argus Introduction

LPSS Loading Procedure Structuring System Vehicle License Plate Analysis Vehicle Trajectory Analysis Loading Rate Detection Staff Efficiency Analysis

VAPD Violated Action Pattern Detection 6s Regularization Detection Violent Operation Detection Safety Production Business Management

Argus Cloud Service Push h Notifi fications ns LPSS recorded data in 2 nd , April,2018  Web 74% 80%  APP 70% 60% 44% 50% 40% Trend nd Graph 28% 30% 16% 20% 5% 5% 10% 0% 0% 0% 粤 BDB566 粤 BFH239 粤 BBT853 粤 BBY411 Loading Rate at Arrival Time Loading Rate at Departure Time Rank nking ng List. Condition Moni nitoring ng Config Computing camera Node

AI Argus A Unique Insight Into Logistics cs Scenario Analysis and Algorithm Design

Scenario Analysis 01 02 01 02 Standardization Behavior Loading Gate Behavior management Monitoring Operation Monitoring Analysis Distribution Cen enter er Sorting Cen enter er 03 04 03 04 Tool Positioning Safety Equipment Specific Area Detection Production Monitoring Monitoring

LPSS Active Data Collection for Unfamiliar Scenes and Transfer Learning 32bit binary VGG16 yes codes Image Database update Database calculate >thres hamming hold distance VGG16 discard no 32bit binary code New image Unseen Sample

LPSS Loading Gate Working Status and Staff Efficiency Analysis 01 01 Optical Flow Calculator Arrival or departure y-axis x-axis arrive 02 02 Action Detection flow_x+flowy+gray merge Pelee-net Classification result arrive arrive 03 03 State Machine arrival arrive arrive arrive Image sequence flow+gray merge sequence Pelee net Classification result State machine result

Vehicle License Plate Analysis LPSS What you expect to see VS What Argus actually sees Asymmetric Illumination Partial Covered Image Blur Deformation/Soiling

Vehicle License Plate Analysis LPSS Object Detection Image Quality Assessment Attention OCR Video （ Image ） SSD+tracker GoogleNet CNN+LSTM+CTC end 粤 BCG570 SoftMax SSD GoogleNet CNN+LSTM+CTC

The Instant Loading Rate Detection LPSS true =D ？ discard true =E ？ ResNet50 true =A/B/C ？ Network1 Decision-making tree predict1 predict2 & ResNet50 predict6 Network 2

LPSS The Process Loading Rate Detection Variable Length Sequence Feature Learning

LPSS Structured Data List ⚫ Vehicle License Plate Loading Rat ate The Ve Vehicle Vehicle Gate The Ve Vehicle Load ading Rat ate at at at Arrival al Star art Time End Time Depar arture Stat ate License Plat ate Number Arrival al Time Depar arture Time ⚫ Gate Number Time Time ⚫ The Vehicle Arrival Time No.1 2018-04-12 2018-04-12 2018-04-12 2018-04-12 粤 BDB566 uploading 74% 5% uploading 22:07:53 22:07:59 22:23:24 22:23:34 ⚫ Loading Rate at Arrival Time gate ⚫ Working Start Time No.1 2018-04-12 2018-04-12 2018-04-12 2018-04-12 粤 BFH239 44% uploading uploading 0% ⚫ Working End Time 22:24:16 22:24:20 22:44:29 22:44:39 gate ⚫ Loading Rate at Departure Time No.2 2018-04-12 2018-04-12 2018-04-12 2018-04-12 uploading uploading 粤 BGZ502 95% 0% ⚫ The Vehicle Departure Time 22:45:41 22:45:45 23:13:40 23:13:45 gate ⚫ Working State judgment No.1 2018-04-12 2018-04-12 2018-04-13 2018-04-13 loading 粤 BV8026 loading 5% 49% 22:13:54 22:13:59 01:21:34 01:21:40 gate No.2 2018-04-13 2018-04-13 2018-04-13 2018-04-13 loading loading 粤 B3G15U 5% 79% 03:34:48 03:34:54 04:07:05 04:07:12 gate No.12 loading 2018-04-12 2018-04-12 2018-04-13 2018-04-13 粤 BZ5717 loading 0% 85% 22:21:15 22:21:21 02:18:49 02:18:56 gate No.13 2018-04-12 2018-04-12 2018-04-13 2018-04-13 loading loading 粤 BBT853 0% 16% gate 22:10:06 22:10:11 01:00:58 01:01:04 No.16 loading 2018-04-12 2018-04-12 2018-04-13 2018-04-13 loading 粤 BBY411 5% 28% gate 22:08:10 22:08:14 03:51:56 03:52:15

VAPD The Illegal Throwing Behavior Detection challenges A short distance throwing is Pushing a box is not an Throwing a file is not an not an Illegal Throwing Illegal Throwing Behavior. Illegal Throwing Behavior. Behavior. ACTION RATING PARCEL TYPE SPATIAL DISTANCE

VAPD The Illegal Throwing Behavior Detection

VAPD Fine Grained Illegal Throwing Behavior Detection via ROI Extraction and 3D Space Recovery

VAPD Structured Data List ⚫ Warning Start Time Number er of Time of Warning Start Ti Time Lates est Warning Ti Time Continuous Duration Precision ⚫ Latest Warning Time violation Warning ⚫ Number of Continuous Warning 1 1 2018/11/29 15:33:50 2018/11/29 15:33:50 15.0s 67% ⚫ Duration ⚫ Time of violation 2018/11/29 15:13:35 2018/11/29 15:13:35 1 15.0s 1 91% ⚫ Precision 1 2 2018/11/29 14:41:08 2018/11/29 14:41:08 2.0min 50% 2018/11/29 14:20:41 2018/11/29 14:20:41 1 15.0s 1 86% 1 1 2018/11/29 13:48:43 2018/11/29 13:48:43 15.0s 52% 2018/11/29 13:47:51 2018/11/29 13:47:51 1 15.0s 1 84% 1 1 2018/11/29 13:43:39 2018/11/29 13:43:39 15.0s 79% 2018/11/29 12:53:50 2018/11/29 12:53:50 1 25.5s 1 72%

AI Argus A Unique Insight Into Logistics cs Acceleration with NVIDIA

Technology Stack - Edge User Applications Loading Monitoring Action Recognition Retail Analytics Smart City 3D Perception Argus Edge Framework Payload Daemon Frame Pool Frame Stack Pre-Processing Primitives Node Status Report System Failure Parallel Processing Queue Inter-Process Recovery Communication In system upgrade Initial Setup Flow HTTPS Token Encryption Asynchronous I/O Interface Video Quality Assessment Heterogeneous Computing Memory Model Multi-Payload Management TensorRT DeepStream CUDA x86 / ARM 64bit Linux Distribution Nvidia Tesla P4 / Xavier

Technology Stack - Cloud SAAS API Gateway WEB APP TCP API HTTP API Payload Msg Auth Manage Alarm Handler Monitor Notification Param Device System Setup Video File Image File Config Management Management User Statistics APP Model Payload Node Management Report Upgrade Upgrade Msg Status Token Command Distribution VUE ElementUI SFIM Msg Spring SpringBoot JDK Netty MyBatis PAAS Kafka Redis ZooKeeper RocketMQ Jetty IAAS MySQL Docker NAS

Argus System Architecture: Edge Computing Edge Network Argus Edge Edge Network Deamon ……. 1 INTERNET (HTTPS/TCP) ……. Edge Network LPSS LPSS VAPD VAPD 2 INTERNET (HTTPS/TCP) Argus Cloud NVR NVR ……. ……. SWITCH INTERNET (HTTPS/TCP) Edge Network N ……. Camera 1 Camera 2 Camera 3 Camera N

Argus System Architecture: Cloud Computing GPU1 Mobile Per Model scheduling Q ueues Phone ... GPU2 ... Inference Request ... GPU3 Bar Code Req/Res HTTP/gRPC Client Scanner Handling Framework Backends TensorRT Inference Request TensorFlow+T LPSS RT NVR Caffe2 VAPD Model Management Model Repo Camera Camera

Mapping Deep Stream into Argus Software Architecture ……. NVR Compute Node Camera 1 Camera 2 Camera N ……. Business Node User/Client ……. NVR Compute Node Camera 1 Camera 2 Camera N Input Processing Output DeepStream

Flexible Streaming Pipeline Design Deep Stream Recognization Discrimination Network network Detect Optical Network Flow Discrimination network 1. A plugin Model based pipeline architecture 2. Graph-based pipeline interface to allow high-level component interconnect • On-Demand Computing 3. Heterogenous processing on GPU and CPU • Reuse Calculation 4. Hides parallelization and synchronization under the hood 5. Inherently multi-threaded

Optical Flow Speed-up with CUDA Runtime ： 8ms cv::cuda:OpticalFlowDual_TVL1

Optical Flow Speed-up with CUDA Motion compensation on non-stationary camera Security camera is fixed Runtime: 3.6ms Assume camera is fixed in cv::cuda:OpticalFlowDual_TVL1 Runtime: 2.8ms Using CUDA float array instead of cv:GpuMat

Optical Flow Speed-up with CUDA 2.8ms 1.3ms 224*224 current 224*224 current 224*224 current = 32*32* 49 1. Global Memory 2. Block = (224*224)/(32*8) 1. Share Memory 3. each Step Sync all Block 2. Block = (224*224)/(32*32)=49 3. Finally Sync all Block 224*224 pre 224*224 pre = 32*32* 49 224*224 pre In deployment, the GPU server adopts Tesla P4 GPU.

Concurrent Asynchronous Producer Consumer Producer Consumer Producer Consumer Buffer Producer Consumer Producer Consumer Queue Producer Consumer Producer Consumer Consumer Producer

Concurrent Asynchronous With Mutil-Stream Producer HD1 K1.1 K1.2 K1.3 K1.4 K1.5 K1.6 DH1 Stream1 Producer HD2 K2.1 K2.2 K2.3 K2.4 K2.5 K2.6 DH2 Stream2 Producer Buffer HD3 K3.1 K3.2 K3.3 K3.4 K3.5 K3.6 DH3 Producer Stream3 Producer Queue HD4 K4.1 K4.2 K4.3 K4.4 K4.5 K4.6 DH4 Stream4 Producer HD5 K5.1 K5.2 K5.3 K5.4 K5.5 K5.6 DH5 Stream5 Producer HD6 K6.1 K6.2 K6.3 K6.4 K6.5 K6.6 DH6 Stream6 Producer