Environment Awareness for Low-Vision Patients Environment Awareness - - PowerPoint PPT Presentation

▶

Apr 10, 2023 26 likes •337 views

Environment Awareness for Low-Vision Patients Environment Awareness for Low-Vision Patients IntelliSight Team Xin Xinyua uan Z Zhang Rafael Ca Carranza Vanessa M Mejia ia IMU Software Camera Software PCB and Mount Integration

SLIDE 1

Environment Awareness for Low-Vision Patients

SLIDE 2

IntelliSight Team

Rafael Ca Carranza

Camera Software Integration

Vanessa M Mejia ia

PCB and Mount Design

Xin Xinyua uan Z Zhang

IMU Software Integration

SLIDE 3

Overview

When we are out in the world, we are able to understand our surroundings by using both our global and local context.

Global Context → Where we are in the world
Local Context → What objects are in our surroundings

Global Context Local Context

SLIDE 4

The Problem

According to the World Health Organization there are 285 million low vision individuals in

the world.

Rely on their senses and on the people around them to understand their local context.
Although technology has come a long way, it is still unable to help them understand what is

in their surroundings in a quick and easy way.

SLIDE 5

The Solution

IntelliSight solves this problem by developing a pair of smart sunglasses that uses:

Visual information from a camera
Orientation information from an IMU
Location information using GPS

SLIDE 6

Hardware

SLIDE 7

Block Diagram

SLIDE 8

ESP32:

Interfaces our camera and IMU sensor
Data → Android phone via Bluetooth
Onboard USB-to-Serial converter
Operating voltage: 3.7 V

Microcontroller

SLIDE 9

IMU: BNO055

Collects orientation data
Capture gestures
Operating voltage: 3.3 V
ESP32 → I2C

Data Collection

Camera: ArduCam Mini 2MP

Takes pictures of the user’s surrounding environment
Operating voltage: 5 V
ESP32 → SPI

SLIDE 10

LiPo Battery:

Output Voltage: 3.7 V
Powers the PCB

PowerBoost 500C

Takes 3.7 V as input and outputs 5 V
Powers the camera

Power Supply

SLIDE 11

Printed Circuit Board

SLIDE 12

PCB Schematic

SLIDE 13

PCB Layout

SLIDE 14

Final PCB

SLIDE 15

Software

SLIDE 16

Camera Mode

Software

IMU Mode

Gesture detection Building detection Text-to-speech Object capture Object detection Text-to-speech

SLIDE 17

Camera Mode

SLIDE 18

Camera Mode

Capture objects in pictures taken of the user’s

surrounding environment

Our phone application can identify the objects in the

pictures using TensorFlow Lite

Relay information using Text-to-Speech

SLIDE 19

IMU Mode

SLIDE 20

IMU Mode

Detects gesture Transmits bearing data to app via bluetooth Scans along the bearing to detect landmark Collects bearing

SLIDE 21

IMU Node: Nodding

z - yaw y - pitch x - roll Pitch Time

Difference Threshold

SLIDE 22

IMU Mode: Gesture Detection

Azimuth [0,360) degree Select NDOF mode

SLIDE 23

Bearing Data

IMU Mode: Building Detection

Building Search User’s GPS Location | 2m |

SLIDE 24

IMU Mode: Building Detection

Bearing Data User’s GPS Location Building Search Failed | 2 m | Next Building Search | 6 m |

SLIDE 25

IMU Mode: Value Returned

Return the final result in voice

Distance Output < 30m “ X is in front of you” 30 - 80m “X is close to you” > 80m “No building nearby”

SLIDE 26

IMU Mode: Further Development

Higher accuracy to determine the landmark

○ Compass accuracy ○ Better state estimation

SLIDE 27

Final Product

SLIDE 28

Final Prototype

SLIDE 29

SLIDE 30

A special thanks to:

Yogananda Isukapalli Aditya Wadaskar Kyle Douglas

SLIDE 31

Environment Awareness for Low-Vision Patients