EEC284 Course Project Richie Rehal December 11, 2014 Outline - PowerPoint PPT Presentation

Strawberry Picking Action Recognition EEC284 Course Project Richie Rehal December 11, 2014

Outline • Background • Experimental Setup • Results • Conclusions

FRAIL-bots: F ragile c R op h A rvest-a I ding mobi L e ro bots • Goal: develop teams of inexpensive, relatively small, harvest-aiding mobile robots • Will increase worker efficiency while consciously minimizing worker strain • Depending on the commodity, labor contributes up to 60% of the variable production cost and recent labor shortages have led to significant loss of production • Co-bots will support human pickers by supplying them with empty containers and by transporting containers filled with harvested crops to unloading stations • Framework developed in this project is applicable to multiple different manually harvested crops Source: NRI: Small: FRAIL-bots: Fragile cRop hArvest- aIding mobiLe robots , courtesy Stavros Vougioukas

FRAIL-bots: Applications in Strawberry Harvesting • Strawberries used initially as the focus crop – High economic importance – Frail nature – Specific plant bearing characteristics – So far robotic and mechanical harvesting has proven commercially impractical for this crop • Inherent delay time due to filling crates with produce and returning crates to centralized location • Strawberry fields have long, low, and narrow furrows • Strawberry farms widely present in California Photos courtesy Stavros Vougioukas

FRAIL-bots: Wearable Devices • Small, wearable electronics devices worn by the workers to monitor their state – Picking rate – Rate of advancement – Ergonomics data • Captures motion data from wrist and lower back • Battery life extremely important as work day is 8+ hours • Device footprint cannot interfere on normal worker function – Needs to be low footprint and accepted by the workers Photos courtesy Stavros Vougioukas

Project Goals • Gather real world data for training the learning algorithms • Analyze features for classification • Implement action recognition system with good degree of accuracy • Gather knowledge to be applied to future action recognition endeavors

General Work Flow • Data Gathering • Feature Extraction • Classification • Training • Prediction • Results!

Action Selection • Searching for strawberries • Extract / detach strawberry • Throwing bad strawberries away • Placing strawberries into tray/cart – and arranging them in the tray • Pushing (advancing) the cart • Walking to deliver tray • Resting

Action Selection • Searching for strawberries • Extract / detach strawberry • Throwing bad strawberries away • Placing strawberries into tray/cart – and arranging them in the tray • Pushing (advancing) the cart • Walking to deliver tray • Resting

Feature Selection • Candidates: – Mean – Median – Standard Deviation – Variance – Root Mean Square – Percentile Ranges (ex: Interquartile Range) – Frequency Distribution – Covariance/Correlation – Jerk?

Example Data 1.5 Extract 1 0.5 Ax 0 Ay 0 2 4 6 8 10 12 14 Az -0.5 -1 -1.5

Example Data

Feature Selection • Frequency Distribution as a candidate

Feature Selection • Average of Jerk (x) • Standard Deviation of Jerk (x) • 75 th Percentile of Acceleration (xy) • Root Mean Square of Acceleration (xyz) • Correlation of Ax and Ay

Classification • Options: – Naïve Bayes (Bayesian Network) – Decision Tree – Support Vector Machines (SVMs) – Neural Networks – Hidden Markov Models (HMMs) – http://en.wikipedia.org/wiki/List_of_machine_learning_co ncepts • Choice: Naïve Bayes to start

System Block Diagram Acceleration Data (xyz) IMU Laptop 3-Space Sensor Serial connection (wireless) Real time data stream Classification Matlab Label Output of Naïve Bayes Model

YEI 3-Space Sensors • IMU Device (9-axis) – Accelerometer, Gyroscope, Compass – Built in Kalman filtering – Serial interface • Can be connected wirelessly in a sensor network • Targeted for use in motion capture

3-Space Sensor Orientation

Video Demo https://www.dropbox.com/s/ho9oauk1yio0e02/rsrehal%20EEC284%20Course%20Project%20Demo.wmv?dl=0

Results • Successful implementation of Naïve Bayes classifier • Successful identification of key features of gathered data • Successful framework implementation built • Consistent detection of 4 out of 5 actions (5 th is not as consistently detected)

Limitations • Small training data set, so much more room for improvement • Limited to larger window size currently, untested with decreasing in width window sizes – Non-ideal for real world use, since discrete actions are not likely to be in large groups • Actions are somewhat exaggerated, and are likely not representative of real world picking

Conclusions • Feature extraction not a trivial problem! • Training data set highly influential on successful action prediction • Can do a lot with just accelerometer data

Future Work • Increase data set and perform more precise accuracy measurements • Implement with Neural Networks – Requires gathering much more data • Explore unsupervised learning algorithms • Apply to real world data from real strawberry pickers • Extend to other fruit picking procedures?

References • [1] Mi Zhang and Alexander A. Sawchuk. 2011. “ A feature selection-based framework for human activity recognition using wearable multimodal sensors ”. In Proceedings of the 6th International Conference on Body Area Networks (BodyNets '11). ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering), ICST, Brussels, Belgium, Belgium, 92-98. • [2] Maurer, U.; Smailagic, A.; Siewiorek, D.P.; Deisher, M., " Activity recognition and monitoring using multiple sensors on different body positions ," Wearable and Implantable Body Sensor Networks, 2006. BSN 2006. International Workshop on , vol., no., pp.4 pp.,116, 3-5 April 2006

Questions?