Plant-Background Image Segmentation

Marianne Bjorner and Carter Sifferman

Task - Plant Segmentation

• The goal is to separate plant from background
• Useful for monitoring plant growth and health

Segmentation Algorithm

Input RGB image

Output binary image

Existing Research

• Recent research focuses on the more difficult problems of leaf counting and individual leaf segmentation
• Kuznichov et al. (2019) achieve high accuracy on the more difficult problem of individual leaf segmentation using a convolutional neural network
• Yin et al. (2018) are able to segment and keep track of individual leaves over the course of a plant’s growth

Kuznichov (2019) result

Ground truth

RGB input

Gaps in Existing Research

• Nearly all papers from the last few years use the CVPPP2017 dataset or the older LSC2014
• Kuznichov (2019) also tests on a dataset of ~200 avocado leaf images
• The CVPPP2017 dataset is not very diverse, and we don’t know how well algorithms will generalize to more diverse datasets

Avocado leaf segmentation result from Kuznichov (2019)

Our Goal

• Create a more challenging and diverse dataset for testing plant-background segmentation algorithms
• Create novel plant segmentation algorithms of our own
• Test our own algorithms and an existing algorithm on both an existing dataset and our own more challenging dataset

CVPPP 2017 Dataset

• 810 overhead images of plants in four different settings, along with labels
• Large dataset
• Entirely indoors, small leafy plants
• Not very diverse

RGB image

Individual leaves

Plant-background

Leaf centers (dots)

CVPPP: Example of Four Settings

RGB image

Ground truth

Our Dataset

• 68 overhead images of plants
• Diverse settings
• Leafy plants, grasses, succulents
• Indoors and outdoors
• Bright and dim light
• Challenging backgrounds
• Non-green plants
• Better suited for evaluation than for training
• Also includes tags for each image for easily testing against a subset

Portion of the tags.json file for our dataset

Grass

Leafy

Purple Leaves

Challenging Background

Indoors

Succulent

Measuring Accuracy - Jaccard Score

• Also known as Intersection Over Union (IoU)
• Standard metric for measuring the similarity of two sets
• Jaccard score of 1 corresponds to a perfect match
• State-of-the-art image segmentation techniques often score above 0.95

Image source: Data Science Bootcamp

Our Segmentation Approaches

• Green channel thresholding
• Per-pixel logistic regression
• Smoothed and denoised per-pixel regression
• K-means clustering

Green Channel Thresholding

• Mark each pixel with a green value above some threshold as plant - otherwise mark as background

208 > 100

105 > 100

13 < 100

plant

plant

background

Green Channel Thresholding

• Predictably, this does not perform very well - bright colors give false positives

Input RGB image

Ground truth

Green channel > 100

Per-Pixel Logistic Regression

• What if we use some formula to determine whether a pixel is the color of a plant?
• We can use logistic regression to find such a formula if we have a labeled dataset
• Training on CVPPP2017 we get the following:

Per-Pixel Logistic Regression

• Greatly improved performance with much fewer false positives

Input RGB image

Ground truth

Per-pixel regression result

Per-Pixel Logistic Regression

• A few images still have false positives

Input RGB image

Ground truth

Per-pixel regression result

Smoothed and Denoised Per-Pixel Regression

• We remove small clusters of pixels and apply a gaussian blur to smooth edges
• Improved performance overall

Input RGB image

Ground truth

Per-pixel regression result

Smoothed and denoised

Smoothed and Denoised Per-Pixel Regression

• In a few cases, smoothing led to worse performance

Input RGB image

Ground truth

Per-pixel regression result

Smoothed and denoised

K-Means Clustering

• First, segment the image with k-means clustering
• Label clusters with a certain proportion of “plant colored” pixels as plant

Input RGB image

Final Binary Mask

Intermediate clustering and logistic regression result

Results on Our Dataset

Takeaways / Future Work

• Our logistic regression based approaches are very fast but somewhat inaccurate
• Using more sophisticated features than RGB values could yield better results
• K-means and logistic regression promising for individual leaf segmentation task

Many other ways to approach the problem:

• Convolutional Neural Networks
• Mean Shift
• Object Recognition

Thank You