Aquatic Identification and Sorting System SENIOR PROJECT PROPOSAL - - PowerPoint PPT Presentation

▶
aquatic identification and sorting system
SMART_READER_LITE
LIVE PREVIEW

Aquatic Identification and Sorting System SENIOR PROJECT PROPOSAL - - PowerPoint PPT Presentation

Sep 13, 2022 116 likes •571 views

Aquatic Identification and Sorting System SENIOR PROJECT PROPOSAL PRESENTATION JOHN SCHULZ AND REINER LINTAG ADVISOR: DR. IMTIAZ DECEMBER 4 TH , 2018 Agenda Background Project Objectives System Functions Design Results

slide-1
SLIDE 1

Aquatic Identification and Sorting System

SENIOR PROJECT PROPOSAL PRESENTATION JOHN SCHULZ AND REINER LINTAG ADVISOR: DR. IMTIAZ DECEMBER 4 TH, 2018

slide-2
SLIDE 2

Agenda

  • Background
  • Project Objectives
  • System Functions
  • Design
  • Results
  • Parts
  • Division of Labor
  • Schedule
  • Literature Search
  • References

AQUATIC IDENTIFICATION AND SORTING SYSTEM

2

slide-3
SLIDE 3

Problem Statement

  • Modular fish sorting system
  • Invasive species
  • Avoid premature fish harvesting.

AQUATIC IDENTIFICATION AND SORTING SYSTEM

3

slide-4
SLIDE 4

Background

TWO MAJOR COMMERCIAL USES:

Aquaculture

  • $50 billion dollars a year loss from inefficient

harvesting practices [1].

  • Aquaculture in 2012 was a $128 billion dollar

industry [1]. Governments

  • Inspired by Asian Carp in the Illinois River.
  • Army Corps of Engineering estimate $15 to $18

billion to be spent to stop Asian Carp [2].

  • DNA of Asian Carps already found in the Great

Lakes [3].

AQUATIC IDENTIFICATION AND SORTING SYSTEM

4

slide-5
SLIDE 5

Objectives

  • Scalable modular system
  • Fishing lures used for humane practice
  • Ensure system safety electrically
  • Transportable and operate off grid operation
  • AI specie classification
  • Aid in invasive specie detection and removal efforts.
  • Harvest fish based on size vs. bulk harvesting.

AQUATIC IDENTIFICATION AND SORTING SYSTEM

What is it? What size is it? Will a sort be possible?

5

slide-6
SLIDE 6

System Functions

  • Distinguish between different fish/creatures (lures)
  • Sorting based on AI decision from camera
  • Off grid operation
  • Data collection system

AQUATIC IDENTIFICATION AND SORTING SYSTEM

6

slide-7
SLIDE 7

Hardware Design

Camera FPGA: PYNQ IOT Sorting Servo Data Cache

AQUATIC IDENTIFICATION AND SORTING SYSTEM

7

Battery Powered

slide-8
SLIDE 8

Prototype System Design

AQUATIC IDENTIFICATION AND SORTING SYSTEM

8

Images Training DNN DNN FPGA Camera Sort Power Region

System Prototype

Here

slide-9
SLIDE 9

System Prototype

  • Pump controls water flow from left to right
  • Open output pipe for gushing design
  • Modern lures swim when water passes over

them.

  • 3 different lengths of output pipe to suite

different lures’ swimming types.

AQUATIC IDENTIFICATION AND SORTING SYSTEM

9

Figure 1 “Current Simulation”

slide-10
SLIDE 10

Simulation Tank

AQUATIC IDENTIFICATION AND SORTING SYSTEM

Figure 2 “Tank Parts” Figure 3 “Assembled Tank”

10

Output Pipe Pump Pump Controller

slide-11
SLIDE 11

Tank Build

  • Size: 18” x 26” x 15”
  • Capacity: 4 ¾ Gallons
  • Material: food grade polycarbonate
  • Crystal-clear
  • Sealable lid
  • Custom Additions
  • External reflection shielding
  • Aquarium grade led lighting strip

AQUATIC IDENTIFICATION AND SORTING SYSTEM

11

Figure 4 “Tank Design”

slide-12
SLIDE 12

Lure Mount Points

AQUATIC IDENTIFICATION AND SORTING SYSTEM

12

Horizontal Mount Angle Mount Top Mount

Output Pipe

  • 14”/17”/20”

Figure 5 “Mount Points”

Figure 6 “3/4” Piping”

slide-13
SLIDE 13

Lure Design

AQUATIC IDENTIFICATION AND SORTING SYSTEM

13

Weight Chamber Ball Bearing Leveling Fin Attachment Point Treble Hooks Figure 7 “Lure Design”

slide-14
SLIDE 14

Imaging

AQUATIC IDENTIFICATION AND SORTING SYSTEM

14

Images Training DNN DNN FPGA Camera Sort Power Region

System Prototype

Here

slide-15
SLIDE 15

Imaging Angle

  • Subject shifted left or right to remove pipe

reflections.

  • Subject imaged at ±15° horizontally and

vertically

  • Pseudo 3-D mapping of subject
  • Assuming subject is symmetrical if flipped

AQUATIC IDENTIFICATION AND SORTING SYSTEM

15

0° +15°

  • 15°

Center

Figure 8 “Imaging Angle”

Output Pipe Intake Pipe

slide-16
SLIDE 16

Database

  • 20 Different fish classes
  • Minimum 14 minutes recorded per class
  • 4 Water clarity levels
  • 3 Imaging angles captured
  • 4 min. -15°
  • 6 min. 0°
  • 4 min. +15°
  • 3 Database duplication techniques
  • Gaussian noise
  • Flip left right
  • Flip up down

Database size:

1.6 million

Images total, created in house.

AQUATIC IDENTIFICATION AND SORTING SYSTEM

16

slide-17
SLIDE 17
slide-18
SLIDE 18

Image Intensity

  • Image Intensity – the number of levels of

color accuracy provided for each color

  • False Contouring – the appearance of lines of

color that exist due to a lack of image intensity.

AQUATIC IDENTIFICATION AND SORTING SYSTEM

18

Figure 10 “2-Bit Color” Figure 11 “3-Bit Color” Figure 9 “8-Bit Color” Figure 12 “4-Bit Color”

slide-19
SLIDE 19

Imaging

  • Video captured at 720p 60 fps
  • Image processing:
  • 12-bit color depth, 4-bits per RGB pixel
  • Video processing and bit slicing done in

Matlab

  • Neural network trained on 12-bit images
  • Advantages
  • Minimize resource usage in FPGA
  • Clear enough
  • Disadvantages
  • Minor false contouring

AQUATIC IDENTIFICATION AND SORTING SYSTEM

19

Figure 13 “Processing Example”

slide-20
SLIDE 20

Neural Network

AQUATIC IDENTIFICATION AND SORTING SYSTEM

20

Images Training DNN DNN FPGA Camera Sort Power Region

System Prototype

Here

slide-21
SLIDE 21

AI Classes

  • 2 different species in different sizes
  • Similar species of the same size
  • Species design to test dynamic range
  • Classes Include:
  • Fish: Trout, Blue Gill, Minnow, Bass
  • Worms
  • Mice
  • Frogs

AQUATIC IDENTIFICATION AND SORTING SYSTEM

21

Figure 14 “Test Classes of Color Difference ” A B C D Figure 15 “Test Classes of Different Sizes”

slide-22
SLIDE 22

Neural Net Layout

AQUATIC IDENTIFICATION AND SORTING SYSTEM

22

Figure 16 “AI Hierarchy”

  • AlexNet structure [4][6]
  • Space efficient on FPGAs
  • Designed for quick response time
  • Initially tested on 4 classes
slide-23
SLIDE 23

Training to FPGA

Imaging

  • Building

Database of Classes GPU Training

  • Matlab ML

Toolbox Export Model

  • Weights
  • Structure
  • .onnx file

standard VHDL Code Gen

  • Generate

Neural Net code for FPGA in Matlab

AQUATIC IDENTIFICATION AND SORTING SYSTEM

23

Figure 17 “Training Flowchart”

slide-24
SLIDE 24

Why FPGA?

Latency!

  • GPU
  • High throughput or low latency
  • Widely supported
  • Power hungry
  • Large Size
  • FPGA
  • High throughput and low latency
  • Best for high frame rate inferencing
  • 16x Performance/Watt , vs GPU

5,000 10,000 15,000 20,000 25,000 30,000 35,000 GPU 2016 GPU 2017 GPU 2018 FPGA 2018 FPGA 2019 Images/Second

Inference Performance

AQUATIC IDENTIFICATION AND SORTING SYSTEM

24

Source: Xilinx Developers Conference 2018[4]

slide-25
SLIDE 25

FPGA

  • PYNQ Development Board
  • Xilinx ZYNQ XC7Z020 FPGA
  • Logic Cells: 101,440
  • DSP Blocks: 220
  • Ex. Ram: 512 Mb DDR3
  • Block Ram: 4.9 Mb
  • 100 MHz FPGA Clock
  • Video: HDMI In and Out
  • Dual-core 650 MHz Cortex-A9
  • Max: 36 W

AQUATIC IDENTIFICATION AND SORTING SYSTEM

25

HDMI Field Programmable Gate Array (FPGA) – Array of reprogrammable logic gates, used for high throughput tasks.

slide-26
SLIDE 26

AQUATIC IDENTIFICATION AND SORTING SYSTEM

26

Results

slide-27
SLIDE 27

Results: Simulation Tank

  • 3 Lure Mount Points
  • Magnetically mounted from top
  • Angle mounted to the output pipe
  • Horizontal mounted to the output pipe
  • Output Pipe
  • 14”, 17”, 20” lengths
  • Reflection Shielding
  • Hefty trash bags taped externally to reduce

light bleed and reflections

  • Electronics
  • All electronics are external of the sealed

container

AQUATIC IDENTIFICATION AND SORTING SYSTEM

27

Figure 18 “Working Simulation Tank”

slide-28
SLIDE 28

Video

AQUATIC IDENTIFICATION AND SORTING SYSTEM

28

slide-29
SLIDE 29

Database Size

  • All 20 Classes

5 ½ Hours Recorded

  • Current size of processed database

(4 Classes):

103k Images

> 16k per training class 200x150 Res.

AQUATIC IDENTIFICATION AND SORTING SYSTEM

29

Figure 19 “Processed Classes” Class: F1_XRAP Class: F2_FROG Class: F3_MICE Class: F4_XRAP

slide-30
SLIDE 30

Training Precision

  • Video Card: GTX 1060 6GB
  • Training Accuracy: FP16
  • FPGA Accuracy: FP12 (“FIX_12_8”)

AQUATIC IDENTIFICATION AND SORTING SYSTEM

30

Figure 20 “Tested FPGA Accuracy”

slide-31
SLIDE 31

Neural Network

AQUATIC IDENTIFICATION AND SORTING SYSTEM

31

  • Optimizer: Stochastic Gradient Descent (SGD) Method

Figure 21 “Training Accuracy”

slide-32
SLIDE 32

Neural Network

  • Loss Function: Mean Square Error (MSE)

AQUATIC IDENTIFICATION AND SORTING SYSTEM

32

Figure 22 “Training Loss”

slide-33
SLIDE 33

Prediction

AQUATIC IDENTIFICATION AND SORTING SYSTEM

33

Figure 23 “Prediction Accuracy”

slide-34
SLIDE 34

Power Diagram

AQUATIC IDENTIFICATION AND SORTING SYSTEM

34

Images Training DNN DNN FPGA Camera Sort Power Region

System Prototype

Here

slide-35
SLIDE 35

Power

Distribution

  • Currently using grid power for the system
  • Implement a maneuverable power distribution system
  • Portable Power Station
  • Car Battery
  • Solar Panel

AQUATIC IDENTIFICATION AND SORTING SYSTEM

35

Figure 25 “Solar Panel” Figure 24 “Inverter”

slide-36
SLIDE 36

Power

System Power Consumption

  • Xilinx ZYNQ XC7Z020 FPGA – 36 [W]
  • Jebao DCP Water Pump – 22 [W] to 23 [W]
  • Mingdak LED Aquarium Light – 4 [W]

System Total Wattage = 62 [W] to 63 [W] Calculation: Low = FPGA + Water Pump (Low) + LED High = FPGA + Water Pump (High) + LED

AQUATIC IDENTIFICATION AND SORTING SYSTEM

36

slide-37
SLIDE 37

Parts

Tank components

  • Jebao DCP – 2500 Water Pump
  • Cambro – Lid
  • Cambro – Container
  • PVC Pipe ¾”
  • Mingdak – B00X84LQ5S Top Light
  • Hefty – Black Garbage Bags

AQUATIC IDENTIFICATION AND SORTING SYSTEM

37

Figure 26 “Pump” Figure 27 “Tank Light”

slide-38
SLIDE 38

Parts

Safety

  • Watt’s Wire – WW-G12T003Y GFCI
  • AmazonBasics - MW-A1/B3-1650 Extension Cord

The team is working with and in possible water spill areas. These parts were bought and inspected before making the fully assembled tank “Live”. Safety is the number one priority for this project.

AQUATIC IDENTIFICATION AND SORTING SYSTEM

38

Figure 28 “GFCI”

slide-39
SLIDE 39

Parts

Data Acquisition

  • Canon 70D DSLR camera using a 35-50mm
  • Nikon D3500 DSLR camera using 18-55mm

Data Storage

  • 1TB Samsung 970 EVO

AI Implementation

  • PYNQ Dev. Board w/ Xilinx ZYNQ FPGA

AI Training

  • Alienware 13, 32GB Ram, GTX1060 6GB

AQUATIC IDENTIFICATION AND SORTING SYSTEM

39

slide-40
SLIDE 40

Parts: Software

  • Matlab 2018b
  • Parallel Computing Toolbox
  • Statistics and Machine Learning Toolbox
  • Text Analytics Toolbox
  • Communications Toolbox
  • Image Processing Toolbox
  • Xilinx Vivado 2018.1
  • BitQuick

AQUATIC IDENTIFICATION AND SORTING SYSTEM

40

slide-41
SLIDE 41

Division of Labor

JOHN SCHULZ REINER LINTAG

  • Build simulation tank
  • Testing image processing methods
  • Power system
  • Creating website
  • Log all work
  • Update Bill of Material (BOM)

AQUATIC IDENTIFICATION AND SORTING SYSTEM

41

  • Build simulation tank
  • Testing image processing methods
  • Build fish database
  • AI training and implementation

Shared Labor

slide-42
SLIDE 42

Schedule

August

  • Begin VHDL AI Code Gen
  • Project Approved

September

  • Finalize Simulation Tank Design
  • Power System Design
  • Finding Parts

October

  • Image Intensity Testing
  • Parts Ordered
  • Power System

November

  • Built Simulation Tank
  • Start Database
  • Train Test NN Model

December

  • Building Database
  • Demonstrate NN Accuracy & Structure
  • Demonstrate Power Analytics

January

  • Begin Embedded Analytic System

February

  • Complete VHDL AI Code Gen
  • Load Testing of Power System

March

  • Demonstrate functioning NN
  • Demonstrate power support
  • Setup NN Data Transmission

April

  • Demonstrate Embedded Analytics
  • Complete NN and Power System

May

  • Present Final Project

AQUATIC IDENTIFICATION AND SORTING SYSTEM

42 Key: In Progress

Done To Be Done

slide-43
SLIDE 43

Literature Search

  • New University of Alberta
  • Managing invasive aquatic species using machine
  • Algorithms only: decision trees
  • Targeting: intercoastal species of plants and shellfish
  • Oregon State University
  • Algorithms only: decision trees
  • Targeting: Lion Fish only
  • Carnegie Mellon University and Microsoft
  • Algorithms: machine learning and game theory
  • Targeting: patrol routes of poachers

AQUATIC IDENTIFICATION AND SORTING SYSTEM

43

slide-44
SLIDE 44

References

[1] World bank, “FISH TO 2030 Prospects for Fisheries and Aquaculture,” Rep. no. 83177-GLB, Dec. 2013. [Online]. Available: http://www.fao.org/docrep/019/i3640e/i3640e.pdf. Accessed on: April 23, 2018. [2] News Desk, “Keeping Asian carp out of the Great Lakes will cost billions and take decades,” PBS NewserHour Productions LLC., Jan. 6, 2014. [Online]. Available: https://www.pbs.org/newshour/science/keeping-asian-carp-out-of-the-great-lakes-will-costbillions-and-take-decades. Accessed on: April 23, 2018. [3] Vice News, “The Worst Fish in America: Asian Capocalypse,” Sep 24, 2014. [Online]. Available: https://www.youtube.com/watch?v=YnZp1jtOhR0. Accessed on: April 23, 2018. [4] Raje, Salil, “AI Acceleration”, presented at Xilinx XDF Silicon Valley 2018 Conf., Xilinx Inc., Oct. 10, 2018, San Jose, CA. [Online]. Available: https://www.youtube.com/watch?v=qWFREOeRiqo. Accessed on: Oct. 25, 2018. [5] Gao, Hao. “A Walk-Through of AlexNet”, Medium Corp., Aug. 7, 2017. [Online]. Available: https://medium.com/@smallfishbigsea/a-walk- through-of-alexnet-6cbd137a5637. Accessed on: Oct. 25, 2018. [6] Ayster, Yusuf. “Lecture: Deep Learning for Vision”, MIT Computer Vision (Course 6.869: Advances in Computer Vision). Cambridge, MA., USA. [Online]. Available: http://6.869.csail.mit.edu/fa16/lecture/6.869-Lecture19-DeepLearning.pdf. Accessed on: Oct. 29, 2018.

AQUATIC IDENTIFICATION AND SORTING SYSTEM

44

slide-45
SLIDE 45

Questions?

AQUATIC IDENTIFICATION AND SORTING SYSTEM

45