EEC284 Course Project Richie Rehal December 11, 2014 Outline - - PowerPoint PPT Presentation

Richie Rehal December 11, 2014

Strawberry Picking Action Recognition

EEC284 Course Project

• Background
• Experimental Setup
• Results
• Conclusions

Outline

FRAIL-bots: Fragile cRop hArvest-aIding mobiLe robots

• Goal: develop teams of inexpensive,

relatively small, harvest-aiding mobile robots

• Will increase worker efficiency while

consciously minimizing worker strain

• Depending on the commodity, labor

contributes up to 60% of the variable production cost and recent labor shortages have led to significant loss

• f production
• Co-bots will support human pickers

by supplying them with empty containers and by transporting containers filled with harvested crops to unloading stations

• Framework developed in this project

is applicable to multiple different manually harvested crops

Source: NRI: Small: FRAIL-bots: Fragile cRop hArvest- aIding mobiLe robots, courtesy Stavros Vougioukas

FRAIL-bots: Applications in Strawberry Harvesting

• Strawberries used initially as the

focus crop

– High economic importance – Frail nature – Specific plant bearing characteristics – So far robotic and mechanical harvesting has proven commercially impractical for this crop

• Inherent delay time due to filling

crates with produce and returning crates to centralized location

• Strawberry fields have long, low,

and narrow furrows

• Strawberry farms widely present

in California

Photos courtesy Stavros Vougioukas

FRAIL-bots: Wearable Devices

• Small, wearable electronics

devices worn by the workers to monitor their state

– Picking rate – Rate of advancement – Ergonomics data

• Captures motion data from

wrist and lower back

• Battery life extremely

important as work day is 8+ hours

• Device footprint cannot

interfere on normal worker function

– Needs to be low footprint and accepted by the workers

Photos courtesy Stavros Vougioukas

• Gather real world data for training the learning

algorithms

• Analyze features for classification
• Implement action recognition system with good

degree of accuracy

• Gather knowledge to be applied to future action

recognition endeavors

Project Goals

• Data Gathering
• Feature Extraction
• Classification
• Training
• Prediction
• Results!

General Work Flow

• Searching for strawberries
• Extract / detach strawberry
• Throwing bad strawberries away
• Placing strawberries into tray/cart

– and arranging them in the tray

• Pushing (advancing) the cart
• Walking to deliver tray
• Resting

Action Selection

• Searching for strawberries
• Extract / detach strawberry
• Throwing bad strawberries away
• Placing strawberries into tray/cart

– and arranging them in the tray

• Pushing (advancing) the cart
• Walking to deliver tray
• Resting

Action Selection

• Candidates:

– Mean – Median – Standard Deviation – Variance – Root Mean Square – Percentile Ranges (ex: Interquartile Range) – Frequency Distribution – Covariance/Correlation – Jerk?

Feature Selection

Example Data

• 1.5
• 1
• 0.5

0.5 1 1.5 2 4 6 8 10 12 14

Extract

Ax Ay Az

Example Data

• Frequency Distribution as a candidate

Feature Selection

• Average of Jerk (x)
• Standard Deviation of Jerk (x)
• 75th Percentile of Acceleration (xy)
• Root Mean Square of Acceleration (xyz)
• Correlation of Ax and Ay

Feature Selection

• Options:

– Naïve Bayes (Bayesian Network) – Decision Tree – Support Vector Machines (SVMs) – Neural Networks – Hidden Markov Models (HMMs) – http://en.wikipedia.org/wiki/List_of_machine_learning_co ncepts

• Choice: Naïve Bayes to start

Classification

System Block Diagram

IMU 3-Space Sensor Laptop Matlab Acceleration Data (xyz) Serial connection (wireless) Real time data stream

Classification Label

Output of Naïve Bayes Model

• IMU Device (9-axis)

– Accelerometer, Gyroscope, Compass – Built in Kalman filtering – Serial interface

• Can be connected wirelessly in a sensor network
• Targeted for use in motion capture

YEI 3-Space Sensors

3-Space Sensor Orientation

Video Demo

https://www.dropbox.com/s/ho9oauk1yio0e02/rsrehal%20EEC284%20Course%20Project%20Demo.wmv?dl=0

• Successful implementation of Naïve Bayes classifier
• Successful identification of key features of gathered

data

• Successful framework implementation built
• Consistent detection of 4 out of 5 actions (5th is not

as consistently detected)

Results

• Small training data set, so much more room for

improvement

• Limited to larger window size currently, untested

with decreasing in width window sizes

– Non-ideal for real world use, since discrete actions are not likely to be in large groups

• Actions are somewhat exaggerated, and are likely

not representative of real world picking

Limitations

• Feature extraction not a trivial problem!
• Training data set highly influential on successful

action prediction

• Can do a lot with just accelerometer data

Conclusions

• Increase data set and perform more precise accuracy

measurements

• Implement with Neural Networks

– Requires gathering much more data

• Explore unsupervised learning algorithms
• Apply to real world data from real strawberry pickers
• Extend to other fruit picking procedures?

Future Work

• [1] Mi Zhang and Alexander A. Sawchuk. 2011. “A feature

selection-based framework for human activity recognition using wearable multimodal sensors”. In Proceedings of the 6th International Conference on Body Area Networks (BodyNets '11). ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering), ICST, Brussels, Belgium, Belgium, 92-98.

• [2] Maurer, U.; Smailagic, A.; Siewiorek, D.P.; Deisher, M.,

"Activity recognition and monitoring using multiple sensors

• n different body positions," Wearable and Implantable Body

Sensor Networks, 2006. BSN 2006. International Workshop

• n , vol., no., pp.4 pp.,116, 3-5 April 2006

References

Questions?