Tomorrows Food Production AgRA Webinar: October 29 th 2014 Satoshi - PowerPoint PPT Presentation

Automation Technology for Tomorrow’s Food Production AgRA Webinar: October 29 th 2014 Satoshi Yamamoto Visiting Faculty, CPAAS, WSU Senior Researcher, BRAIN, NARO 1

Motivation for the automation How to keep the current level? 140,000 130,000 Food production 120,000 Population in Japan  Sustainable 110,000 100,000  Reliability & Safety Peak: 2008 90,000 80,000 70,000  Efficient work 60,000  Information management 50,000 1920 1940 1960 1980 2000 2020 2040 2060 Year *1920 – 2010: Statistics Bureau, Japan *2010 – 2060: National Institute of Population and Automation Technology Social Security Research, Japan 2

TOPICS 1. Back ground 2. Components of plant factory for strawberries in BRAIN, NARO 3. 3D modeling of apple fruit in CPAAS, WSU 3

National Agriculture and Food Research Organization Researcher: 1,542 (April, 2013) The fiscal 2013budget: 529M US$ (1US$ = ¥109) Research institute under MAFF Largest research organization addressing “agriculture, food and rural communities” 4 http://www.naro.affrc.go.jp/english/index.html

Outline of strawberry production in Japan MAFF Planted Production Wholesale Value (2011) a) Area (2012) Quantity (2012) Fruit (10 6 USD) (ha) (t) Tomato 12,000 722,400 1,522 5,720 163,200 1,573 Strawberry Cucumber 11,600 586,600 1,444 Egg plant 9,860 327,400 805 Sweet Peppers 3,420 145,000 602 “Unshu”, Mandarins 43,700 895,900 1,496 37,400 793,800 1,199 Apple 20000 a) Calculated as 1 USD = 100 JPY. Area Harvested (ha) 15000 10000 California Japan 5000 0 5 1970 1975 1980 1985 1990 1995 2000 2005 2010

* MAFF, 2007 Annual working hours (h/0.1ha) 2,000 Harvest season (months) 6 (December to May) Average of planted area per producer (ha) 0.3 7,000 – 8,000 Planting density (plants/0.1ha) 3 – 5 Production (t/0.1ha) Labor Percentage of working hours management 1% Seedling 10% Sorting, Planting Packing 4% 27% Fertilization 3% Pest control 4% Harvesting Cultivation management 23% 6 28%

Plant Factory for Strawberry Production 3. Sorting & packing robot 1. Movable bench system 2. Stationary harvesting robot 7

Movable bench system Movie  Space saving  Automated spraying  Saving energy cost  Increasing yield per area  Improvement labor condition 8

Measurement growth information Movable Bench System Kinect Growth information of all plants every day 9

Measurement growth information Color Image Depth Image Easy to extract leaf area using depth info 10

Measurement growth information 42 beds 4 m Color Feb. 23 Depth Color Mar. 29 Depth Color Apr. 26 Depth 11

Measurement growth information Health diagnosis Lack of Iron High EC or Water stress Basic info of plants: height & width 12

Strawberry harvesting robot Prototype 1 <Development Target> Basic type (no storing function) • 1. More than 60% success rate Cylindrical manipulator (3 DOF) • Three camera 2. 10s to pick & place a fruit • Four halogen lamp • 3. 0.1ha / night (8-12h) Finger for cutting & holding stem • 4. No bruise Suction tube to cancel the depth • error 13

Cylindrical manipulator Prototype 2 Five LED • Through type photo sensor • Tilting motion of robot hand • logistic function for fruit • containers 14

Picking motion a) Approach to a fruit with suction tube Suction tube Finger b) Move finger forward Through type c) Move finger & tube backward photo sensor 15

Tiling motion before picking a) Right direction b) Left direction Two independent air cylinders 16

Prototype 3 No suction tube • Movie Movable platform • 17

Prototype 4 One LED • Movie Diffused photo sensor • Shibuya Seiki Co., Ltd. Bending motion for placement • 18

Stationary type with movable bench Movie Commercialized by Shibuya Seiki Co., Ltd. 19

Change of robot’s faces Simplicity • 20 Compactness •

Image processing 1. Binarization 3. Maturity assessment 21 2. Occlusion assessment 4. Stem detection

Difference of coloring among varieties 100 100 80 80 Estimation (%) Estimation (%) 60 60 40 40 20 20 y = 1.0633x - 1.3707 y = 0.991x - 2.7616 R² = 0.8207 R² = 0.9557 0 0 0 20 40 60 80 100 0 20 40 60 80 100 Human eye (%) Human eye (%) Amaotome Beni-hoppe 22

Fruit condition 100 90 A B C 80 Fruit condition (%) 70 60 50 D E 40 30 20 A B C D E 10 0 Aisle Aisle Bed Bed (Feb.) (May) (Feb.) (May) Aisle Side Bed Side 23 ‘Beni hoppe’ cultivar

Harvesting from bed side Mayekawa mfg. Co., Ltd. Waseda University Prototype 3 Prototype 2 24

Harvesting from bed side Movie Stereo vision for position detection Hand-eye-camera for stem detection 25

Reduction of influence of fruit condition Movie Separate from Approach Pick Place adjoining fruits 26

Reduction of influence of fruit condition Movie Coloration End-effector Measurement Unit Position a) Vacuum b) Grip Detection Unit Mobile bench Unit 7 DOF Manipulator 27

Reduction of influence of fruit condition Movie Picking Hand Coloration Measurement Unit Vacuum Hand 7DOF Manipulator Position Detection Unit Mobile 3DOF Bench Unit Manipulator 28

Mini-summary for harvesting robot 2003 Cylindrical manipulator (3 DOF) • Three camera, Four halogen lamp • Finger for cutting & holding stem • Suction tube • 2006 2010 Machine vision & software: Maturity, Occlusion… • 2013 Finger shape • Tilt function of robot hand • Diffused photo sensor • Don’t move Stationary harvesting robot expensive robot! Harvesting success rate: 40 – 70 % 29

Strawberry sorting & packing robot From harvesting box to shipping tray 30

Shipping types Single layer 1 Double layer Small pack Hart shape Single layer2 Single layer 3 31

Strawberry sorting & packing robot Movie Sorting & Packing unit Returnable tray Supply unit 32 Single-layer tray

Supplying unit Camera Manipulator Suction hand (3 DOF) Harvesting container 33

Sorting & packing unit Camera Manipulator (4 DOF) Suction hand Collision Returnable tray Safe Single-layer tray 34

Strawberry sorting & packing robot (2) Machine vision: Kinect End-effector Machine vision: Color camera Manipulator Conveyer for harvesting containers Conveyers for shipping trays Fruit conveyer 35

Start Kinect Supply fruits and shipping tray Detect the suction point of target fruit in harvesting container Pick up fruit, move to digital camera Weight and orientation of the held fruit Place on shipping tray Continue? Digital camera 36 Stop

Segmentation of fruits in harvesting container Segmentation of fruits using color & depth info 37

Fruit orientation Movie V of HSV R – G image Maximum error: 25.1 ˚ MEAN : 0.3˚ Size & Orientation SD: 5.1 ˚ 38

Strawberry packing robot in grading line Movie Packing Robot Color camera IR sensor Weight scale 39 Yanmar Green System Co., Ltd.

Strawberry packing robot (Basic) 40

Mini-summary for sorting & packing robot 2007 Robot hand Supplying unit 7 s / fruit 2011 4 s / fruit More than human ability! 2013 Packing robot Sorting & Packing robot (Basic) Packing robot in grading line using Kinect < 1 s / fruit 1.5 s / fruit 41

3D modeling of apple fruit in CPAAS, WSU Density: important factor for evaluation of a fruit inner  quality. Volume: not a common technique in a fruit sorting system.  Appearance: check using surface color information.  3D reconstruction for fruit sorting system using Kinect 42

Measurement setting LED Kinect Apple 43

3D models using Kinect CAD data can be download from website of GrabCAD. Automated grading system • Inner quality: density • Appearance assessment How should we use them? 44

Summery Grading based on 3D model Packing robot Movable bench in grading line system Growth measurement Stationary harvesting robot 45

For the tomorrow’s food production…  Is it time to get out of the plant factory?  Construction the preferred environment for automation will be important.  Consumer 3D sensor has changed the accessibility to 3D info.. Simple hardware & smart software Over the Row Sensor Platform (left), Detection of apple fruits (right) Stem detection will be a key 46 for a robotic harvester… CPAAS, WSU (Prof. Karkee)

Thank you for your attention! 47