Real-time Tracking of Human Arm Movements Jacob Phillips, Erik - - PowerPoint PPT Presentation

Real-time Tracking of Human Arm Movements

Jacob Phillips, Erik Guetz, Dr. Mohammad Imtiaz

Project Overview

Problem

• Be able to track, diplay, and predict human arm movements
• Provide a way to access the data for various applications

Problem Overview

Project Goals

• Accurate arm motion tracking

– Filtering and prediction

• Long battery life

– Efficient embedded system

• Minimal human interference

– Easy to set up and run

• Easy to read and understandable display

– Mobile phone app with graphs and data readouts

Problem Solution

• Inertial Measurement Units (IMU)

– Three sensors on arm

• Embedded system

– Custom PCB with RTOS

• Mobile phone app

– Android application on smartphone

Problem Solution System Diagram

Functional Requirements

• Three IMU’s placed on arm
• Data streaming wirelessly
• Data storage facility
• Stream data for downstream systems

System Design Process

Embedded Systems Specifications

• 32-bit Atmel SAM4S32B Microcontroller

– Up to 120MHz, 2MB FLASH, 160KB SRAM

• 4Gbit NAND FLASH
• STC3100 battery “gas gauge”/Coulomb counter
• Sparkfun Bluetooth Mate Silver

– RN-42 Bluetooth module

• LSM6DS3 Inertial Measurement Unit

– Up to 6.6KHz and 8Kb on board FIFO

Custom PCBs

• Two boards

– IMU and main board

• Created in Eagle PCB

– Custom made libraries and parts

• Main board contains 32-bit ARM microcontroller

– FLASH memory, Bluetooth, battery charging

• IMU board designed as small as possible

– Final dimensions: 18.5x15.5 mm

IMUs

IMU

• Small, simple device
• Low power
• Fast communications

IMU PCB

• Kept as simple and small as possible
• Only four components total

Sensor Controller

Sensor Controller

• Low power
• Multi-day storage capability
• Wireless streaming
• On-board battery charging

Sensor Controller PCB

• Optimized for low power consumption
• Simple operation with only a single

hardware switch

• All control besides power-on handled

through app

Power and Charging Schematic

Connector Schematic

FLASH Memory Schematic

Embedded System Initialization

Application Interface System

Application Interface System

• Data port
• Application Program Interface (API)
• Sample Applications

– Predictive model

• LSTM Neural Network

– Visualization

Data Acquisition

Data Filtering and Estimation

Data Prediction

Data Visualization

Accomplishments to Date

Sensor Reading

• Only read data if sensor

gives proper “Who am I”

• Data is collected from
• n-board FIFO
• Sent to host over UART

using CMOS to RS232

Initial Sensor Measurement

• Used Realterm to save text files containing measurement data
• Used Python to interpret saved measurement files and convert them to a

MATLAB readable format

• Used MATLAB to visualize the data and estimate position

Work In Progress

Revised Sensor Measurements

• Used Python to read serial data from

COM port

• Used Python to estimate velocity and

position data from raw acceleration data

• Used Matplotlib to plot acceleration,

velocity, and position

Revised Sensor Measurements (Cont.)

Estimated Velocities Estimated Displacement

Kalman Filtering

• Also called Linear Quadratic Estimation
• Uses series of measurements containing errors and

statistical noise

• Produces estimates of unknown variables
• More accurate than filtering using one measurement by

using a joint probability distribution

Our Use of the Kalman Filter

• To estimate velocity and position of the IMU from the

acceleration given

• To remove any noise and error attached to the incoming

measurement data (i.e. sensor drift)

Future

RTOS

• Used to keep precision timing on IMU samples
• Will feature priority system and task scheduling

– IMU sampling is high priority, FLASH writes are medium priority, and battery status reads are low priority

• Allows better use of system resources
• Avoids wasting processor time in delays
• Also will feature a “diagnostic system” to alert host device
• f errors

Embedded Memory Controller

• FLASH memory requires 8-bit parallel writing
• SRAM FIFO used in combination with FLASH

– Allows bulk writes to FLASH – Minimize time writing to FLASH

• No hardware memory controller, must be created through

software

Position Prediction Neural Network

• Used to predict the next arm position based on past

positions

• A Recurrent Neural Network (RNN) with built-in long term

memory also known as a Long Short Term Memory network (LSTM)

Smartphone Application

• Receives streams of data
• Used to visualize data
• Used to send commands to the sensor controller

Jacob Erik

• Design PCBs
• Develop RTOS
• Develop embedded

subsystems (IMU, memory controller, etc)

• Adding components to the

IMU PCBs

• Developed Application

Interface

• Worked on Android app

development

Division of Labor

Timeline

Questions?