P I X E V I A : A I B A S E D , R E A L - T I M E C O M P U T E R - - PowerPoint PPT Presentation

▶

Mar 14, 2023 121 likes •536 views

P I X E V I A : A I B A S E D , R E A L - T I M E C O M P U T E R V I S I O N S Y S T E M F O R D R O N E S Mindaugas Eglinskas, CEO at PIXEVIA www.pixevia.com Origins in R&D projects for Lithuanian MoD. Autonomous systems research

SLIDE 1

P I X E V I A : A I B A S E D , R E A L - T I M E C O M P U T E R V I S I O N S Y S T E M F O R D R O N E S Mindaugas Eglinskas, CEO at PIXEVIA www.pixevia.com

SLIDE 2

Origins in R&D projects for Lithuanian MoD. Autonomous systems research at Vilnius University (2004 – 2017)

SLIDE 3

SLIDE 4

SLIDE 5

SLIDE 6

P I X E V I A

CORE

Real-time imaging

CORE X1

Hardware interfaces

OBJECTS

AI based object recognition

NAV

Navigation

FUSION

Information / sensor fusion

SLIDE 7

Integrates AI technology in daily commercial drone operations. Uses NVIDIA Jetson TX1 and PIXEVIA for: First responder

Person detection & tracking
Vehicle detection & tracking
License plate recognition
Privacy masking

Inspection

Object detection / Classification
Recognizing condition deviations
Determine fault location
Distance measurement

Real-time onboard mapping

SLIDE 8

Drones with AI for security applications. Uses NVIDIA Jetson TX1 and PIXEVIA system: carrierboard, pipelines and

bject classification.

SLIDE 9

Fully autonomous control Fully autonomous information processing Fusion of information from different machines and data sources

Main driving forces powered by machine learning

SLIDE 10

U S E C A S E S F O R A I P O W E R E D D R O N E S

SLIDE 11

Surveillance: defence / law enforcement / private security

cars (with number plate recognition) trucks boats people heavy machinery

SLIDE 12

Automated infrastructure inspections

Automated inspection: power lines utility poles insulators foreign objects

SLIDE 13

Inventory management

Containers Wagons Locomotives Cars Materials Packages

SLIDE 14

Smart city

Parkings Car flows Persons Security

SLIDE 15

S O F T W A R E A R C H I T E C T U R E / D E C I S I O N S / L E S S O N S L E A R N E D

SLIDE 16

CORE

Intelligent real-time imaging: collection, transformation, communication

Interfaces with sensors and file formats (input) On-board image processing. Image processing pipelines Image processing modules Geographical metadata Distributed processing Industry standards

SLIDE 17

OBJECTS

…

Object detection, properties of objects (size, speed, coordinates)

Cars / Trucks License plate recognition People Face detection Other objects: boats, environment, etc.

SLIDE 18

NAVIGATION

Visual position estimation

Visual odometry Image-map matching Foreign object detection on the landing site

SLIDE 19

INFORMATION FUSION

…

Fusion of information in real-time

Information fusion from different sources (drones, cameras) Vizualization on 3D map

SLIDE 20

USB3 / USB2 SD-card UART HDMI GPIO CAN miniPCIe CSI I2C SPI PWM Accel, gyro, compass, barometer

1. Interfaces

SLIDE 21

Long range digital datalink (Microhard) Industrial block camera GPRS datalink Thermal imaging camera Gimbal control Ultra fast camera for visual

dometry

Wifi , local communcations

2. Sensors

SLIDE 22

Spaghetti type of integration will kill any bigger project OpenVX / Gstreamer NVIDIA provides accelerated Gstreamer modules for encoding DDS for real time communication Processing can be changed before the mission or during the flight

3. Modular architecture

SLIDE 23

4. Simple description of image processing pipeline and tools

SLIDE 24

5. Geographical metadata

Every video frame contains geographical information:

image corners with coordinates, position of

drone/camera, angles of sensors, camera geometry. Allows integration with GIS systems, provides data for later learning.

SLIDE 25

HARDWARE NVIDIA Frameworks Other frameworks

6. Hardware and software frameworks used

SLIDE 26

7. GPU based moving object detection

VisionWorks (OpenVX graph) CUDA OpenCV with CUDA optimization Caffe cuDNN

SLIDE 27

8. Region detection with convolutional neural networks

cuDNN Caffe Fully convolutional network Single shot detection Filtering after detection

SLIDE 28

9. Self-adaptation mode - “sleep”

Real-time Sleep

SLIDE 29

Self-adaptation during the sleep

Information from the fast neural models Slow models (big neural nets,

ther computer vision

algorithms, physics)

SLIDE 30

10. Simulation and learning

SLIDE 31

MobilEye

600 people doing labeling

MobilEye photo

SLIDE 32

AI (neural networks / SGD) limitations

It can win all games

(if can play more than human plays through all life)

Recognize images

(if can see more than human sees through all life) Terrible performance with small datasets

SLIDE 33

Simulated data from unity3d

SLIDE 34

Simulation for data fusion

SLIDE 35

11. 3D reconstructions during the flight

210 000 points, 90 seconds on Jetson TX1, 4 images

SLIDE 36

12. Visual position estimation

Visual odometry Image - map matching Terrain segmenation

Deep neural networks 500 000 training item dataset. Convolutional neural network, cuDNN

Multiple hyphothesis tracking

SLIDE 37

13. User interfaces in the embedded system

Web / Qt via HDMI and datalinks

SLIDE 38

PIXEVIA version 0.5 Technology preview

Current status

SLIDE 39

AI for autonomous systems

CORE

Real-time imaging

CORE X1

Hardware interfaces

OBJECTS

AI based object recognition

NAV

Navigation

FUSION

Information / sensor fusion

SLIDE 40

P I X E V I A : A I B A S E D , R E A L - T I M E C O M P U T E R V I S I O N S Y S T E M F O R D R O N E S Mindaugas Eglinskas, CEO at PIXEVIA www.pixevia.com