P I X E V I A : A I B A S E D , R E A L - T I M E C O M P U T E R V I S I O N S Y S T E M F O R D R O N E S Mindaugas Eglinskas, CEO at PIXEVIA www.pixevia.com
Origins in R&D projects for Lithuanian MoD. Autonomous systems research at Vilnius University (2004 – 2017)
4
5
P I X E V I A FUSION Information / sensor fusion NAV OBJECTS Navigation AI based object recognition CORE Real-time imaging CORE X1 Hardware interfaces
Integrates AI technology in daily commercial drone operations. Uses NVIDIA Jetson TX1 and PIXEVIA for: First responder - Person detection & tracking - Vehicle detection & tracking - License plate recognition - Privacy masking Inspection - Object detection / Classification - Recognizing condition deviations - Determine fault location - Distance measurement Real-time onboard mapping
Drones with AI for security applications. Uses NVIDIA Jetson TX1 and PIXEVIA system: carrierboard, pipelines and object classification.
Fully autonomous control Main driving forces powered by Fully autonomous information processing machine learning Fusion of information from different machines and data sources
U S E C A S E S F O R A I P O W E R E D D R O N E S
Surveillance: defence / law enforcement / private security cars (with number plate recognition) trucks boats people heavy machinery
Automated infrastructure inspections Automated inspection: power lines utility poles insulators foreign objects
Inventory management Containers Wagons Locomotives Cars Materials Packages
Smart city Parkings Car flows Persons Security
S O F T W A R E A R C H I T E C T U R E / D E C I S I O N S / L E S S O N S L E A R N E D
CORE Intelligent real-time imaging: collection, transformation, communication Interfaces with sensors and file formats (input) On-board image processing. Image processing pipelines Image processing modules Geographical metadata Distributed processing Industry standards
OBJECTS Object detection, properties of objects (size, speed, coordinates) Cars / Trucks License plate recognition People … Face detection Other objects: boats, environment, etc.
NAVIGATION Visual position estimation Visual odometry Image-map matching Foreign object detection on the landing site
INFORMATION FUSION Fusion of information in real-time Information fusion from different sources (drones, cameras) … Vizualization on 3D map
1. Interfaces USB3 / USB2 SD-card UART HDMI GPIO CAN miniPCIe CSI I2C SPI PWM Accel, gyro, compass, barometer
2. Sensors Long range Industrial digital datalink block camera (Microhard) Thermal GPRS datalink imaging camera Gimbal control Wifi , local communcations Ultra fast camera for visual odometry
3. Modular architecture Spaghetti type of integration will kill any bigger project OpenVX / Gstreamer NVIDIA provides accelerated Gstreamer modules for encoding DDS for real time communication Processing can be changed before the mission or during the flight
4. Simple description of image processing pipeline and tools
5. Geographical metadata Every video frame contains geographical information: - image corners with coordinates, position of drone/camera, angles of sensors, camera geometry. Allows integration with GIS systems, provides data for later learning.
6. Hardware and software frameworks used HARDWARE NVIDIA Frameworks Other frameworks
7. GPU based moving object detection VisionWorks (OpenVX graph) CUDA OpenCV with CUDA optimization Caffe cuDNN
8. Region detection with convolutional neural networks cuDNN Caffe Fully convolutional network Single shot detection Filtering after detection
9. Self-adaptation mode - “sleep” Real-time Sleep
Self-adaptation during the sleep Slow models Information from (big neural nets, the fast other computer vision neural models algorithms, physics)
10. Simulation and learning
31 MobilEye 600 people doing labeling MobilEye photo
AI (neural networks / SGD) limitations - It can win all games (if can play more than human plays through all life) - Recognize images (if can see more than human sees through all life) Terrible performance with small datasets
Simulated data from unity3d
Simulation for data fusion
11. 3D reconstructions during the flight 210 000 points, 90 seconds on Jetson TX1, 4 images
12. Visual position estimation Terrain segmenation Visual odometry Deep neural networks 500 000 training item dataset. Convolutional neural network, cuDNN Multiple hyphothesis tracking Image - map matching
13. User interfaces in the embedded system Web / Qt via HDMI and datalinks
Current status PIXEVIA version 0.5 Technology preview
AI for autonomous systems FUSION Information / sensor fusion NAV OBJECTS Navigation AI based object recognition CORE Real-time imaging CORE X1 Hardware interfaces
P I X E V I A : A I B A S E D , R E A L - T I M E C O M P U T E R V I S I O N S Y S T E M F O R D R O N E S Mindaugas Eglinskas, CEO at PIXEVIA www.pixevia.com
Recommend
More recommend
