Developing Robust Systems for Lettuce Thinning and Phenotyping Jim - - PowerPoint PPT Presentation

Developing Robust Systems for Lettuce Thinning and Phenotyping

Jim Ostrowski 2015-04-28

1

What is Blue River? What is Lettuce Thinning? Why Lettuce Thinning? Blue River Automated Thinning Key lessons What’s next for lettuce? Other ventures: High throughput Phenotyping Working with breeders Weeding for other crops Plant-by-plant care

Outline

2

• “Advanced Technology for Better Agriculture”
• Focus on computer vision and robotics in agriculture
• Co-founders Jorge Heraud and Lee Redden
• Met at Stanford and developed idea in Steve Blank’s Entrepreneurship class
• Researched a variety of ideas, including automated mowing (e.g., golf courses) and

carrot weeding

• Based in Sunnyvale, CA:
• Started in 2011
• 31 employees
• Initial funding from Khosla Ventures
• NSF SBIR Grant
• First commercial revenue in May, 2013
• Goal of reducing chemical usage and improving

existing agricultural practices

• No one wants a Green River…

Blue River…?

3

31 world-class employees Diversity of backgrounds & education

4

Lettuce thinning

• Farmers overplant lettuce and then thin
• E.g., plant seed every 2” and thin to

plant every 10”

• Driven by poor germination in lettuce

plants

• Thinning traditionally a dull, slow manual

process using a hoe

• ~40 people / 25 acre field / day
• About 20-30,000 plants per acre
• Lettuce grown locally in Salinas Valley
• Existing, but very slow, manual process
• Labor shortages
• Grown year-round
• Operate as a service
• Stay close to customers
• Don’t have to perfect a robotic system

1 2 3 4 5 6 7 8

5

• Camera used to identify plants
• Determine plants versus weeds and best plants to keep
• Spray fertilizer on early stage plants
• This is toxic to young plants and kills them
• Provides some residual fertilizer to remaining plants
• AVOID spraying on plants you want to keep…!

Our approach

6

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Target spacing (inches):

10.0

Minimum spacing (inches):

9.0

Target spacing

Decision-making overview

7

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Target spacing (inches):

10.0

Minimum spacing (inches):

9.0

Never keep these plants Target spacing

Decision-making overview

8

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Target spacing

Blue River’s ¼” precision minimizes doubles left in the field

9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Target spacing Killed plants Killed plants

Blue River’s ¼” precision minimizes doubles left in the field

10

Optimize spacing Eliminate all doubles

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Keep the next plant Target spacing

11

Optimize spacing Eliminate all doubles

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Keep the next plant Target spacing Killed plants Killed plants

12

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Optimize spacing Eliminate all doubles

Target spacing Balance spacing and doubles in each decision

13

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Optimize spacing

Target spacing Balance spacing and doubles in each decision

14

15

Main vision system

• Shrouded system to control lighting (mostly)
• RGB GigE camera
• Processes plants at roughly 5-10 Hz
• Required at speeds of 2.5 – 3.0 MPH
• Use color, gradients, shape, size to identify plants
• Classifies plants: identifies plant centers and boundaries
• SVM used in classification; exploring deep learning
• Filters out most weeds
• Better than 98% detection rates
• Less than 5% false positives (weeds as plants)

16

• In Ag parlance, our system is called an “Implement”
• High-speed vision system
• Takes action on ~1,000,000 plants per day per machine
• NOT fully autonomous
• Implement is pulled by a tractor (which is manually driven)
• Autonomy would be cool (challenging to navigate turns), but not economical
• 100-fold increase in speed and width since first prototype

About our machine

17

• Modular system
• Complete processing and spray system for each row of lettuce
• Reconfigurable in the field to different planting formats

− From 40” beds with 2 seedlines per bed to 80” beds with 6 seedlines per bed

About our machine

18

About our fleet

Yield increases of ~10% Six fully capable machines

• Each is capable of thinning 15-30 acres /

day − Roughly 40 times faster than manual thinning

• Centralized, remote management software
• People, trucks, trailers, etc. operate as a

daily service to the growers

• More precise spacing
• No damage to root structure by hoe-

strikes on neighboring plants

19

• Engineering challenges
• Rugged outdoor system with a LOT of computing power
• System simple enough to be fully operated by tractor driver
• Fertilizer destroys everything that’s not stainless steel (or

plastic/rubber)

• Sprayer’s solenoids get tens of millions of cycles per year and

get sticky/slow

• Anything that can be walked on or hit with a hammer will be…
• Economic risk
• At 20,000+ plants per acre, each acre is worth about $10K
• At 4-5 acres per hour, machines can do a LOT of damage

very quickly

• How to safeguard against this?

We employ a secondary vision system to self-monitor and self- calibrate our systems

Challenges

“Collinear” sprays

20

Image pipeline walk-through

21

Image pipeline walk-through

22

Image pipeline walk-through

23

Image pipeline walk-through

24

Image pipeline walk-through

25

• Natural next step is to do weeding in lettuce and other vegetable crops
• Requires several small changes to system, e.g., spray accuracy, material, vision
• Approx. $25B spent annually in US on weeding (including corn and other commodity

crops)

Next opportunity: Weeding

Spray weeds without harming stand

26

Other work: High-throughput phenotyping

“Phenotyping limits the ability to derive full value from the DNA” – N. America Field Breeding Leader, Dow Agro Science, @ corn breeding school 2014

Most phenotypic info gathered by hand today

Breeding is the key to higher productivity & phenotyping is the key to breeding

27

Phenotyping as a service

Multiple sensors collect robust plant-by-plant data High-throughput, in-field data collection

PASSIVE Visual + NIR + Thermal

Reflected wavelengths Incoming wide spectrum radiation (LEDs) Absorbed wavelengths

ACTIVE

Scanning laser (LiDAR)

28

Multiple metrics can be gathered from every plant 28

CONFIDENTIAL

Visible light Leaf Area Index

#174 LAI = 1.2 #175 LAI = 1.4 #176 LAI = 1.3 #177 LAI = 1.6

Nitrogen

#174 N = 1.7% #175 N = 1.8% #176 N = 1.6% #177 N = 2.2% #178 LAI = 1.7 #178 N = 2.3%

Planned metrics

Time of measurement Geospatial position Distance to nearest plant Planting density Height of top leaf Number of leaves Leaf angle Leaf width Projected leaf area LAI Lodging (angle) Stalk diameter Tassel size # of ears Ear length Ear diameter Water potential Nitrogen content Stomatal conductance NDVI TCARI/OSAVI CCCI CWSI PRI … open to others

29

Create 3D plant reconstruction and extract useful metrics

Metric extraction for use in breeding program 3D reconstruction & analysis from sensor data

Plot Plant Time stamp Lat Lon Nearest (cm) Nearest5 (cm) HighLeaf (cm) ProjLA (-) LAI (-) LeafAngle (deg) LeafWidth (cm) LodgeAngle (deg) 17 32 4/30/14 10:15 41.71385

• 94.06316

13.3 29.5 120.5 1.5 1.9 21.4 4.5 0.8 17 33 4/30/14 10:15 41.71383

• 94.06323

11.6 28.3 75.7 1.4 2.2 31.0 4.4 37.4 17 34 4/30/14 10:15 41.71382

• 94.06330

15.3 39.7 98.2 1.6 2.0 24.6 4.2 0.0 17 35 4/30/14 10:15 41.71380

• 94.06337

16.0 39.0 121.1 1.6 2.0 31.0 4.4 2.2 17 36 4/30/14 10:15 41.71378

• 94.06343

18.7 39.0 122.4 1.3 2.0 25.8 4.6 0.0 17 37 4/30/14 10:15 41.71377

• 94.06350

15.1 29.4 122.6 1.3 2.0 26.7 4.1 0.0 17 38 4/30/14 10:15 41.71375

• 94.06357

14.0 30.2 92.2 1.5 1.8 25.5 5.1 0.0 17 39 4/30/14 10:15 41.71374

• 94.06364

17.9 30.7 109.5 1.4 1.8 26.4 5.4 0.4 17 40 4/30/14 10:15 41.71372

• 94.06371

17.8 34.5 78.2 1.6 1.9 23.6 5.0 0.0 17 41 4/30/14 10:15 41.71370

• 94.06377

16.8 39.0 114.1 1.5 1.9 22.6 4.8 24.4 17 42 4/30/14 10:15 41.71369

• 94.06384

17.2 27.6 126.9 1.5 2.3 34.5 4.8 18.0 17 43 4/30/14 10:15 41.71367

• 94.06391

11.6 35.9 87.8 1.3 2.1 38.9 4.7 2.4 17 44 4/30/14 10:16 41.71366

• 94.06398

15.1 39.0 99.7 1.3 1.9 25.7 5.1 36.9 17 45 4/30/14 10:16 41.71364

• 94.06405

19.4 30.3 77.5 1.4 1.9 33.1 4.8 31.0 17 46 4/30/14 10:16 41.71362

• 94.06411

18.1 27.4 122.2 1.4 1.9 23.6 5.1 15.6 17 47 4/30/14 10:16 41.71361

• 94.06418

19.8 31.6 81.1 1.7 2.0 26.7 5.1 1.5 17 48 4/30/14 10:16 41.71359

• 94.06425

11.6 29.5 131.0 1.7 1.7 22.3 3.7 0.0 17 49 4/30/14 10:16 41.71358

• 94.06432

17.6 33.3 126.7 1.6 1.8 39.2 4.5 0.0 17 50 4/30/14 10:16 41.71356

• 94.06439

18.3 38.7 118.9 1.6 2.2 22.3 5.4 22.8 17 51 4/30/14 10:16 41.71354

• 94.06445

13.5 34.7 128.4 1.4 2.1 31.1 3.8 7.0 17 52 4/30/14 10:16 41.71353

• 94.06452

19.7 31.5 96.0 1.7 2.2 22.7 4.5 0.0 17 53 4/30/14 10:16 41.71351

• 94.06459

19.9 34.7 75.3 1.3 1.8 32.6 4.0 0.0 17 54 4/30/14 10:16 41.71350

• 94.06466

16.2 39.5 76.6 1.5 1.8 39.9 3.8 33.9 Plot ave 4/30/14 10:15 41.71367

• 94.06391

16.3 33.6 104.5 1.5 2.0 28.3 4.6 10.2 Plot stdev 2.6 4.3 20.0 0.1 0.2 5.6 0.5 13.7

30

Current Status

Deep learning algorithms for plant segmentation Prototype hardware and sensor platform in testing

31

We believe there is great potential to accelerate the breeding process with plant-by-plant phenotypic data 31

CONFIDENTIAL

Full set of phenotypic metrics combined with environmental, management, and genotypic data “Machine Learning” algorithms to identify patterns, predictors, and new questions p1,1 pN,1 p1,M pN,M

Height

mN mN-1

• Weather

QTL Yield

tM tM-1

Drought Salt NPK Pests

“ ”

Target traits Parameters Metrics

32

Plant-by-plant phenotyping could break down barriers in experimental breeding 32

Accelerate breeding progress by selecting plants early at higher throughput, for wider range of traits

• Higher density / yield
• Drought and salt tolerance
• Lower input requirements
• Efficient biomass production
• Disease and pest resistance
• Geographical and management

customization

• Climate adaptation

If US maize yield doubled in 20 yrs …

= f G, E, = f G, E,

Germplasm, breeding techniques Soil, climate, management

Blue River adds a new tool Plant-by-plant phenotypic information

• ver time

Traits, breeding efficiency Traits, breeding efficiency

• Source: http://blog.lib.umn.edu/ione/eyeonearth/2010/01/us-

corn-yields-to-double-in-the-next-twenty-years.html

33

Weed control result of foundational tech & businesses

Tools for sustainable agriculture that feed the world with fewer chemicals – “Make Every Plant Count” Weed control Computer vision & machine learning Reliable, rugged robotic platform High- throughput, real-time data processing Closed-loop robotic actuation Phenotyping Vegetable Services

34

Questions?