Detailed Design Review MSD 18047 - VIRTUAL CANE Agenda Project and - - PowerPoint PPT Presentation

Detailed Design Review

MSD 18047 - VIRTUAL CANE

Agenda

Project and Concept Breakdown Picture Matching SIFT Obstacle Avoidance OpenAL IMU Motion Estimation Housing

Team

Team Member Role Major Demo/Focus Suhail Prasathong Team Lead Computer Engineering

Obs Av., Pic Mat, Obs Av ML

Deepti Chintalapudi Project Manager Industrial Engineering

Housing

E J Team Member Electrical Engineering

Research

Josh Drezner Purchasing and BOM Electrical Engineering

IMU/Gyro

Aziz Alorifi Communications Computer Engineering

Documentation & Housing

Stuart Burtner Team Member Computer Engineering

OpenAL, Pic Mat

Project Background

• To build a hands-free device to assist Visually Impaired Individuals regain some degree of

independence

• Primary key challenges to overcome include orientation and localization
• A secondary key challenge is obstacle avoidance

Task & Update Tracking - POA

• IMU position tracking Feasibility Joshua
• Picture Matching Demo Suhail
• SIFT Demo Stuart
• Obstacle Avoidance Demo Suhail
• Machine learning demo Suhail
• OpenAL Audio Demo Stuart
• Triangulation Demo EJ
• Housing Demo Deepti
• Concept Breakdown VIdeo Aziz
• Edge Maintenance Aziz

Concept Breakdown Review

Picture Matching Demo

Picture Matching Demo

• Pre-SIFT implementation to explore picture matching possibilities

○ Uses opencv technology ○ Maps pixels but does not account for angle, just rotation

• Outcomes:

○ Matches pictures well ○ Does not handle angles unless multiple pre-defined angles are provided ○ This is a possible solution but definitely not an optimal solution

SIFT Algorithm Demo

SIFT Algorithm - Overview

Match an input image to one existing in a database:

• Necessary component of the design - solves localization & orientation problem
• Must handle varying degrees of distance and angular skew from reference point

When this system produces a match between an Input picture and an existing picture - sound will be played at the location of the match

SIFT Algorithm - Process

Three ‘reference points’ taken across household:

• Two “2-Dimensional” reference points
• One “3-dimensional” reference point

Fourteen test pictures taken:

• Many slightly skewed & scaled images
• Some false-positives (Should match no image)
• Some severely skewed images

SIFT Algorithm - Process

Reference Point 1

SIFT Algorithm - Process

Reference Point 2

SIFT Algorithm - Process

Reference Point 3

SIFT Algorithm - Process

Score = 65.2%

SIFT Algorithm - Process

Score = 31.25% Score = 36.36%

SIFT Algorithm - Process

No Match

SIFT Algorithm - Process

Score = 84.06%

SIFT Algorithm - Process

Score = 19.23% Score = 23.53%

SIFT Algorithm - Process

Score = 0% Score = 45.45%

SIFT Algorithm - Process

Score = 38.46%

SIFT Algorithm - Process

Score = 0%

SIFT Algorithm - Process

No Match

SIFT Algorithm - Process

No Match

SIFT Algorithm - Process

Score = 0%

SIFT Algorithm - Process

No Match

SIFT Algorithm - Process

No Match

SIFT Algorithm - Process

Score = 13.79% Score = 15.0%

SIFT Algorithm - Process

Score = 30.76%

SIFT Algorithm - Outcomes

Overall: Sift does not fit our needs

• Does not work with high skew
• Only produces reasonable results in highly

similar circumstances Moving forward: Try ASIFT

• Capable of matching at heavy skew

○ Image to the right produced 51 ASIFT matches ○ Originally produced 0 SIFT matches

Position Tracking/IMU Feasibility

IMU Position Tracking Feasibility - Process

• Integration:
• Drift was handled by creating a threshold of 0.15m/s2 where any measurement below would be

rounded to 0. Thus integration was only performed during assumed periods of movement

• Tests were done in which the IMU was moved a set distance at varying “speeds” to measure the

accuracy and precision of the integration and error handling.

IMU Position Tracking Feasibility - Outcome

• Overall: Not Feasible.
• No repeatable test was produced

○ Threshold varied from 0.1 up to 0.5 (m/s2) ○ Varied Time Delay from 75ms to 250ms ○ Equations changed to instantaneous readings:

• Next Steps:

○ Possibly look into gravity cancellation ○ Assist with Camera Triangulation Feasibility ○ Adjust scope of device so that it only works while standing still, and after movement the RP will need to be reestablished

Obstacle Avoidance

Obstacle Avoidance - Overview

• Create handsfree system to help user avoid obstacle
• Explore three major facets:

○ AI Poly API ○ Ultrasonic ○ IR Sensor

• Future potential - 3D Depth Sensor

Obstacle Avoidance - Process

• Step 1: Discussed and evaluated AI Poly Potential with Dr. Hochgraf and Machine Intelligence Lab
• Step 2: Considered Ultrasonic option

Obstacle Avoidance - Process

• Step 3: Tested 2 IR sensors in conjunction

○ Gathered IR information independently from left and right side ○ Goal is to expand this so it covers the user from all sides

• Step 4: Explored 3D Depth Sensor

Obstacle Avoidance - Outcome

• Outcome 1 - Ultrasonic is not an aesthetically sound option
• Outcome 2 - AI Poly does not provide any value for obstacle avoidance
• Outcome 3 - IR Sensors are difficult in terms of utility and are not as accurate as initially hoped

Obstacle Avoidance - Conclusion

• Final Conclusion:

○ Shelf obstacle avoidance till February ○ Accomplish primary goals of orientation and localization ○ If accomplished in allotted timeline, work towards obstacle avoidance

Obstacle Avoidance - Alternative/Simplification

• Demo using two main functions:

○ Machine learning of object and image recognition ○ Getting depth through triangulation

Obstacle Avoidance - Afterthought

• For the future, it was determined that RaspberryPi’s depth sensing camera is probably the best path

forward

• Pi 3D sensing offers:

○ Built in obstacle avoidance library ○ Stationary and motion obstacle avoidance ○ Plug and play capabilities ○ $179.99

OpenAL

OpenAL - Overview

Given an input X, Y, Z Coordinate & a rotation, play a sound

• Critical component for generating a reasonable sound output
• Must produce localizable sound - capable of distinguishing between sounds from the left, right,

forward, and behind

• Must be capable of responding to rotational changes

OpenAL - Process

A program was created to repeatedly play one sound:

• A .wav file is parsed and read into a format understandable by OpenAL
• At the command-line, X & Y location of the sound can be changed dynamically while the sound

continues playing

• Similarly, the listener can be rotated such that it simulates the user ‘turning’ within an environment

OpenAL - Process

OpenAL - Outcome

Results:

• Sound is localizable in front of and to either side of a listener
• Difficult to distinguish between forward and behind
• Rotational vector is not correctly implemented

Next steps:

• Add reverb FX to sound (available within library)
• Search for methods to improve “behind-the-head” phenomenon
• Determine how to implement 360º rotation

Motion Estimation

Motion Estimation - Overview

This serves as a means to track the velocity of an object in an image as well as a method of compression The technique to investigate is a Motion estimation method called spatio-temporal gradient, where by we estimate the motion in an image using Optical Flow

Motion Estimation - Process

Motion Estimation - Process

Motion Estimation - Demo

References

Slides and Demo reference obtained from Image and Video Processing on Cousera - https://www.coursera.org/learn/digital

Housing

Housing - Overview

Design & develop housing for both prototypes

1.

Housing for IMU Prototype

2.

Check viability of Product Design for final concept Determined that :

• Housing HAS to be compact - especially prototype to be worn on the wrist
• PLA can be used for proof-of-concept (melts at relatively high temp.)
• ABS should be used for final housing (Stronger, holds more weight)

Housing - Process

Plastic Casing for Ultrasonic Sensor+IMU Prototype Components : Speaker, Arduino, Battery Module, IMU Board, Circuit Board Potential Component Replacements : Battery Module, Smaller Circuit Board Design needs to be improved to fit all components in a more compact casing (currently quite big = 3.7 x 2.9 x 2 inches)

Housing - Process

2 Pi Cam Casings for Glasses Safety Ergonomic Glasses Earphones Casing for External Piping for Electrical Wiring Design for PI Cam Concept

Housing - Outcome

• Issues to mitigate : Three failed prints due to scaling issues on Cura-ABS
• Work on alternative rapid prototyping processes for casings & testing design
• Change latching mechanism orientation for pi cam casing
• Test out functionality of Glasses+Earphones concept with working components
• Make sure that pi cam is steady
• Take into consideration components that require to be pointing in certain directions - component placing

BOM Update

Total Spent So Far: $175.25 Total Remaining: $324.75