An Intelligent Procedure Tracker Problem Specification Development - - PowerPoint PPT Presentation

▶
an intelligent procedure tracker
SMART_READER_LITE
LIVE PREVIEW

An Intelligent Procedure Tracker Problem Specification Development - - PowerPoint PPT Presentation

Sep 14, 2023 394 likes •857 views

An Intelligent Procedure Tracker Problem Specification Development Team Ryan Lorica Lead | Computer Vision Jiacheng Liu PCB | Peripheral Interfacing Leo Mei PCB | Peripheral Interfacing Jingzhe Chen UI | Algorithm Design Anzhe Ye

slide-1
SLIDE 1

An Intelligent Procedure Tracker

slide-2
SLIDE 2

Problem Specification

Development

Team

slide-3
SLIDE 3

Ryan Lorica Lead | Computer Vision Jiacheng Liu PCB | Peripheral Interfacing Leo Mei PCB | Peripheral Interfacing Jingzhe Chen UI | Algorithm Design Anzhe Ye UI | Algorithm Design

Members

slide-4
SLIDE 4

Problem Specification

Introduction

Motivation

slide-5
SLIDE 5

Problem Specification

slide-6
SLIDE 6

Problem Specification

slide-7
SLIDE 7

Motivation

slide-8
SLIDE 8

Problem Specification

Motivation

slide-9
SLIDE 9

Problem Specification

Motivation

slide-10
SLIDE 10

Problem Specification

Up to 40 minutes... Motivation

slide-11
SLIDE 11

Problem Specification

Goal

slide-12
SLIDE 12

Problem Specification

Goal

To verify an astronaut’s accuracy to standard

  • perating procedure
slide-13
SLIDE 13

Problem Specification

Introduction

Solution

slide-14
SLIDE 14

Problem Specification

PCB embedded glove(Motion/Tool detection) LCD display with 3D-printed case Computer Vision powered Body-mounted camera TX2 and power supply inside the backpack NFC tags embedded tool

Solution

slide-15
SLIDE 15

Problem Specification

Hardware

Overview

slide-16
SLIDE 16

Problem Specification

Hardware Block Diagram

slide-17
SLIDE 17

Problem Specification

Hardware Processor Nvidia Jetson TX2

  • ARMv8 (64-bit) Multiprocessor CPU Complex
  • 256 core Nvidia Pascal GPU
  • Power Requirement: 5.5 V - 19.6 V
  • Interfaces: I2C, HDMI, USB, Uart, SPI
  • Connectivity: Bluetooth, WiFi
  • Powerful On-board Computer
slide-18
SLIDE 18

Problem Specification

Hardware Sensors Bluetooth Beacon (Transmitter)

  • One - way Bluetooth Low Energy signal transmitter
  • Range of transmission: 1 m (expected)
  • Localization
  • Embedded in marker

On-Board Bluetooth Unit (Receiver)

  • On - board Bluetooth Version 4.1
  • Custom function to translate Beacon signal to distance
  • Onboard Bluetooth Receiver
  • Beacon
slide-19
SLIDE 19

Problem Specification

Hardware Sensors Camera

  • Logitech C270 HD Webcam
  • Interface: USB
  • Record videos of objects
  • Neural Network powered Computer Vision
slide-20
SLIDE 20

Problem Specification

Hardware Sensors LCD Touch Screen

  • 5 inch LCD touch screen
  • Power Requirement: 5 V via Micro - USB
  • Interface: HDMI
  • Display tasks and errors
  • Wrist Mounted Touch Screen LCD
slide-21
SLIDE 21

Problem Specification

Hardware Sensors Inertial Measurement Unit

  • BNO055
  • 9 Degree of Freedom
  • Memory - mapped addressing to specify sensor

Allowing for reading specific data

  • Interface: I2C
  • Accelerometer & Gyroscope

○ Analyze data to determine hand movements such as hammering and etc.

slide-22
SLIDE 22

Problem Specification

Hardware Sensors Near Field Communication

  • PN532
  • Range of transmission: 10 cm
  • Interface: I2C
  • Recognize a unique tag when it is closed to the chip
  • Detect current using tool

○ NFC chip embedded in glove, ○ tags embedded in tools

slide-23
SLIDE 23

Problem Specification

Hardware Sensors

  • Hand Orientation and Motion Tracking Glove
  • Tool Detection and Identification Glove
slide-24
SLIDE 24

Problem Specification

Hardware PCB

Printed Circuit Board

  • A combination of NFC and IMU
  • Challenges

○ Minimize the size ○ Design and tune the antenna

  • 4 layers PCB

○ 1st layer: SMT components ○ 2nd & 3rd layer: internal connections ○ 4th layer: NFC antenna

slide-25
SLIDE 25

Problem Specification

Hardware PCB

Top Layer Plane Inertial Measurement Unit Near Field Communication Bottom Layer Plane Antenna

slide-26
SLIDE 26

Problem Specification

Software

Overview

slide-27
SLIDE 27

Problem Specification

Software Block Diagram

slide-28
SLIDE 28

Problem Specification

Software Task Manager

slide-29
SLIDE 29

Problem Specification

Software

Action Recognition Algorithm

slide-30
SLIDE 30

Problem Specification

Software Recognizer

Task 1 Task 2

slide-31
SLIDE 31

Problem Specification

Software Recognizer

slide-32
SLIDE 32

Problem Specification

Software Recognizer

slide-33
SLIDE 33

Problem Specification

Software Recognizer

Task 1 G G G

slide-34
SLIDE 34

Problem Specification

Software Recognizer

Task 1 G G G

slide-35
SLIDE 35

Problem Specification

Software Recognizer

Task 1 G G G

slide-36
SLIDE 36

Problem Specification

Software Recognizer

Task 1 G G G

slide-37
SLIDE 37

Problem Specification

Software Recognizer

Task 1 G G G

slide-38
SLIDE 38

Problem Specification

Software Recognizer

Task 1 Task 2

slide-39
SLIDE 39

Problem Specification

Software User Interface

Green -- “Done” Grey -- “Waiting” Yellow -- “In-progress” Red -- “Warning”

Go to another interface with task details To switch task:

  • Click “previous” or “next”

button to the adjacent task.

  • Click “task list” to another

interface so that the user can select one task and go to that task.

slide-40
SLIDE 40

Problem Specification

Software User Interface

  • Task detail interface (upper left)
  • Main interface, when working status is warning,

indicates user’s action is out of accord with that described in task.(upper right)

  • The final interface (lower left) allows user to review

each task and check the progress.

slide-41
SLIDE 41

Problem Specification

Demonstration

Video

slide-42
SLIDE 42

Sample Tasks

slide-43
SLIDE 43

Problem Specification

Demo Video

slide-44
SLIDE 44

Special Thanks to...

  • Dr. Yogananda Isukapalli

Carrie Segal Brandon Pon

  • Dr. Jessica Marquez
  • Dr. Richard Joyce

and

Acknowledgements