Aplicaciones Industiales de Aprendizaje Automatico para el escaneo rapido de Frutas y Cultivos

​

​

Javier Sánchez Galán, PhD

​

Universidad Tecnológica de Panamá

Grupo de Investigación en Biotecnología, Bioinformática y Biología de Sistemas – GIBBS

6 de noviembre de 2023

​

+

Agenda

• Industria 4.0
• Analisis de Imagenes RGB/Vis-NIR para frutos (sandia)
• Analisis hiperespectrales para cultivos (arroz)
• Conclusiones

+

La Situación Agricola Panameña

+

Agribusiness is an open opportunity for Industry 4.0

+

Industry 4.0

+

�

Caceres-Hernandez, D.,... & Sanchez-Galan, J. E. (2023). Recent Advances in Automatic Feature Detection and Classification of Fruits including with a special emphasis on Watermelon (Citrillus lanatus): a Review. Neurocomputing.

​

+

�

Caceres-Hernandez, D.,... & Sanchez-Galan, J. E. (2023). Recent Advances in Automatic Feature Detection and Classification of Fruits including with a special emphasis on Watermelon (Citrillus lanatus): a Review. Neurocomputing.

​

+

Industry 4.0

AI & Machine Learning

Edge Computing

Internet of Things

Cloud Computing

Cyber Security

Digital Twin

https://www.ibm.com/topics/industry-4-0

+

Industry 4.0

AI & Machine Learning

Edge Computing

Internet of Things

Cloud Computing

Cyber Security

Digital Twin

https://www.ibm.com/topics/industry-4-0

+

Métodos de Clasificación

+

Análisis de imágenes RGB y medidas espectrales de reflectancia para determinar la calidad de sandías para exportación

+

Objetivo: Desarrollar un Prototipo de Sistema Experto para la Clasificación de la Sandia como apoyo a mejorar al productor agrícola.

Objetivos específicos: Determinar los parámetros o características que describen al fruto para exportación, consumo nacional o descarte. Diseñar un prototipo para la clasificación del fruto de la sandia empleando el uso de sistemas expertos

​

Parte A: Tratamiento de Imagenes

Parte B: Espectroscopia

UTP-FIE

Dr. Danilo Caceres

Est. Fatima Rangel

Est. Emmy Saez

Est. Kenji Contreras

Colaboradores:

UTP-FISC

Dr. Javier Sanchez Galan

​

“PROYECTO FID18-060 SISTEMA INTELIGENTE PARA CLASIFICACIÓN DE LA SANDIA PARA EXPORTACIÓN” (2019-2023)

+

The Team

Principal Investigator

Ph. D. Danilo Cáceres Hernández Facultad de Ingeniería Eléctrica Universidad Tecnológica de Panamá

Co-Investigator 1

Ph. D.Javier Sánchez Galán

Facultad de Ingeniería de Sistemas Computacionales

Universidad Tecnológica de Panamá

Co-Investigator 2 Licdo. Anel Henry Royo

Centro Regional de Azuero Universidad Tecnológica de Panamá

Student Fatima Rangel

Campus Victor Levi Sasso

Facultad de Ingeniería Eléctrica

Universidad Tecnológica de Panamá

Student Emmy Saez Centro Regional de Azuero

Facultad de Ingeniería Eléctrica

Universidad Tecnológica de Panamá

Student Kenji Contreras

Campus Victor Levi Sasso

Facultad de Ingeniería Eléctrica

Universidad Tecnológica de Panamá

+

​

Parte B: Espectroscopia

Parte A: Tratamiento de Imagenes

​

+

Javier Sanchez Galan, PhD

Universidad Tecnológica de Panamá

​

June 2021

2021 IEEE 30th International Symposium on Industrial Electronics (ISIE)

​

SS: Machine Vision, Control and Navigation

​

KD-001406

​

Using Visible/Near Infrared Reflectance Spectroscopy and Chemometrics for the Rapid Evaluation of Watermelon (Citrullus lanatus) in an Integrated Machine Vision System

​

​

​

Introduction

​

Panama is positioned as one of the Central American countries with most watermelons exports. Profits over 5 million have been reported for the first quarter of 2018. This represents a net growth of 20% when compared to the first quarter of 2017.

The export quality of watermelons must comply with specific external and internal characteristics.

Among the external characteristics are symmetrical, uniform and have a waxy and shiny surface, moreover, they must not have any type of surface defect.

Also, internally they must comply with having a live red color and having pH between 5.18 to 5.60, which can be very informative about growing conditions, processing methods and overall maturity of the product, and required a total sugar content measured as Degrees Brix of 10°Brix.

Motivation

​

​

The externals characteristics can be determined by a simple visual inspection, but the same cannot be possible with the internals characteristic.

The techniques used to determinate the Brix content or pH of a product require a destructive test, and therefore this product must be discarded.

So, many non-destructive techniques have been studied over the years, in order not to lose samples at the time of the evaluation, on of the most used techniques is optical spectroscopy, which studied the light transmitted, absorbed or reflected on a unit of time in a specific range of the electromagnetic spectrum.

Objective

​

The objective of this study can be stated as follows: to assess which Chemometrics methods are the most suitable to predict the internal watermelon parameters of Brix content (representing the SSC) and the pH of two local watermelon varieties, only from its NIR reflectance signature.

Materials (Watermelons)

Samples Characteristics

• Fourteen (14) local market watermelons, eight (8) were of the Quetzaly (A) and six (6) of the Mickylee (B) cultivars ,
• They were chosen because they present different external parameters (textures on the rind, sizes/weights) and internal parameters (pH and Brix content).
• In both cases are commercial cultivars with short harvest cycles (64 and 62 days, respectively). Quetazaly fruits can reach a Brix of ◦12.5 and an average weight of 6.5 kg/unit, while Mickylee fruits can reach a Brix of ◦13, which is considered excellent, and the average weight is 4.5 kg/unit

Materials (Collection Equipment)

Collection of Internal and External Parameters

• For the collection of the raw reflectance spectra, we used a Lumini T portable spectrometry camera, it has a wavelength rage of 315 to 890 nm with optical resolutions less than 8 nm and wavelength precision of 0.5 nm. Vis/Nir spectral sample were taken in triplicate from each measurement side: two on the watermelon body, stem (tail) and posterior (flied spot). The distance between sample and spectral camera was 15 cm and camera was placed at a height of 7 cm with respect to the support.

​

Materials (Collection Equipment)

Collection of Internal and External Parameters

• Once we have all the spectral signature, we proceeded with the collect of the internal parameters, Brix and pH, for taking this parameter we use the Brix meter and pH meter of Milwaukee Instruments.
• The first step was cut in half and three small scoops were taken as samples in the direction of the higher sugar concentration gradient (from the middle moving outwards), then samples were liquefied with a blender, then samples were ready for measurement.

​

Chemometrics Methods

• How to relate spectral signals to measured values?

​

​

Source:Biancolillo Alessandra, Marini Federico. "Chemometric Methods for Spectroscopy-Based Pharmaceutical Analysis". Frontiers in Chemistry 6, 576-, 2018

Data vector Y

Chemometrics

Chemometrics Methods: Preprocessing

• To conditioning the signature for the chemometrics methods, the first step is to handling with the additive and multiplicative effects in the spectrum, so we used derivative methods because they can eliminate this problems. Savitzky and Golay popularized a method for numerical derivation of a vector that includes a smoothing step.

​

• We have the problem to work with limited spectral signature, to overcome the limitation of working with only the 14 spectral signatures, data augmentation (DA) techniques were used. The basic idea of DA is to expand the number of samples used at the time of training by simulating various types of expected changes in a database. In the case of spectral data, random compensations, changes in the slope and multiplications to the existing spectra are used to expand the data set, this technique was performed using the methods already described in a study by Bjerrum et al.

​

Multiplication = 1 ± 0.10 * STD

Offset = 1 ± 0.10 * STD

Slope = Random(0.95,1.05)

​

​

Where:

h: the standardization factors.

ai: coefficients of the polynomial.

np: the number of data points used for the smoothing.

​

​

Non-linear regression method

• Principal Component Regression
• Partial Least Squares

Non-linear regression method

• Multilayer Perceptron Regression (MLPR)

Chemometrics Methods

• R2: this metric determinate the ability of a model to predict future values, the best possible result is 1.0 and it occurs when the prediction coincides with the exact values of the target variable, this metric can be negative too, and it’s when the prediction it´s bad.
• RMSE: is the square root of the average distance between the real value and the predicted value, it indicates the fit of the model to the data, how close the real data are to the predicted ones, so the lower values of RMSE represent a best fit.
• MAE (Mean average error): It’s calculated as the average of the absolute differences between the target values and the predictions, this is a linear score, which means that all individual differences are weighted equally in the average.

​

Chemometrics Methods (Error Metrics)

Results: Externals and Internals Parameters

• Spectral Signature Characteristics

In the spectral signatures can be observed that between the wavelengths 500-600 nm (which corresponds to the green color of the visible spectrum) it can be found the first peak of the spectral signature, with the Quetzaly variety has its maximum at a reflectance value of approximately 20,000 r.u (reflectance units), while the Mickylee variety is above 25,000. A second distinctive peak occurs between 670-800 nm (in the near-infrared band, which can be attributed to a chlorophyll peak), in this case the maximum for both varieties is approximately 27,500 r.u., and no distinction can be made of either watermelon variety.

• Internals Parameters Characteristics

The mean Brix content for variety 1 was 10.35, while for variety 2 was 9, this difference was found to be significant (P=0.03), thus Quetzaly, were found to have a higher Brix content in our experiments. As for pH, the means per variety were 5.23 and 5.14, there was no significant difference found.

Experimental Models

Before any regression model was made, the spectral signatures were normalized using the Standard Scaler function, then smoothed using the Savitzky-Golay (SG) first derivative filter. Subsequent tests were performed with the net reflectance values and with the values resulting from the application of the filter.

For all Chemometric methods, four data configurations were applied as shown in Table. Where X, represent the internal parameters (spectral signatures or inputs) and y, is the internal parameters (target variables, chemical properties, calibration targets, outputs or data treatment).

PCR and PLS Results

The best results in terms of R2 are obtained when applying methods to predict the Brix degrees of the watermelon fruit. For Brix content, regardless of being PLS or PCR, shows a higher correlation coefficient of 0.98 and a standard error of prediction of 0.14/0.16, with MAE of 0.09/0.09, respectively. This result is closely followed by PCR with and R2 of 0.96, on both its raw and smoothed forms for Brix prediction. However, the raw reflectance has a lower RMSE value with 0.007.

MLPR Results

For the MLPR, 1000 hidden layers were used, ADAM was set as the solver for weight optimization, learning rate α=0.001 and a tolerance for the optimization of 1×10−4 .

The best resulting predictions, in terms of R2 , were for prediction of Brix, just like as the previous methods. To obtain the best result with the MLPR method, 181 iterations were required, almost the double that for other Experiments. Although, as it can be seen in the figure small improvements are made after 60 iterations, so fewer loops are required to obtain a similar result.

Conclusions

• Besides the physical difference found between cultivars in terms of rind coloration and size, results suggest that Quetzaly watermelons had a higher Brix content, with no clear distinction in the pH measurements. However, since the sample is small this pH result cannot be generalized, a larger sample size is needed.
• In terms of Chemometrics, all three methods PLS, PCR and MLPR are suitable to predict the Brix content of the watermelon fruit, and to a lesser degree the pH. These results are found in the be positive since the Brix content is the most relevant internal characteristic for watermelon exports.
• MLPR worked best to predict Brix and very poorly for pH. Something that will need to be verified in detail.

​

​

​

I

IEEE International Symposium on Industrial Electronics

Alaska, June 2022

SS Machine Vision, Control and Navigation

Comparing Convolutional Neural Networks and�Deep Metric Learning Methods for Classification�of Export Watermelon (Citrullus lanatus) Varieties

Kenji Contreras, Anel Henry, Danilo Cáceres-Hernández and Javier E. Sanchez-Galan

I

Objectives

• Improve the watermelon exportation process over traditional methods using modern feature extraction and machine learning techniques:
• Developing a better computer vision strategy for the task of watermelon recognition.
• Determine the best approach that allows the recognition of watermelon between varieties with two distinct Deep Learning methods: 1) Transfer Learning and 2) Deep Metric Learning based classification.�

I

Materials

• A dataset comprised of 3 local export varieties.
• Watermelon RGB images were originally captured at a resolution of 360x270 pixels (with a white background).
• Images were sub-sampled, cropped and resized to 40x40 pixels

• Data distribution for model training and testing:
• Transfer Learning: 80% / 20%
• Deep Metric Learning: 70% / 30%

I

Methods

Feature separation

Feature discrimination

I

Transfer Learning

• Pre-trained convolutional models:
• Different convolutional architectures were tested (VGG-19, ResNet50 and EfficientNetB0).
• Regularization methods such as Dropout layers and Global Average Pooling 2D were included.
• Hyperparameter optimization with Grid search.

I

Deep Metric Learning

Triplet Neural Networks and Sampling Strategy

F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embedding for face recognition and clustering,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.

I

Deep Metric Learning

Applying Triplet Neural Networks and Sampling Strategy for Watermelon

Triplets are decided online between the three classes.

The final latent embedding is then reduced via PCA, and fed to a KNN Classifier thus providing a final clustered representation of the points and clases.

I

Transfer Learning Results

• Best pretrained model: EfficientNetB0
• Models with 5-Fold Cross Validation.
• 20% of training data for validation.
• 70 epochs
• Batch size of 16.
• Adam optimizer with a learning rate of 0.001.

I

Deep Metric Learning Results

• Identical Transfer Learning architecture with EfficientNetB0.
• 30% of training data for validation.
• Embedding size of 16
• 8 epochs
• Batch size of 8.
• Adam optimizer with a learning rate of 0.001.

​

• Batch Normalization instead of Dropout.
• Semi-Hard Triplet loss function with a Margin (⍺) of 0.3.

​

I

Performance Comparison

Test dataset evaluation:

I

41 / 14

• Both methods present acceptable results despite presenting signs of overftting:
• Transfer Learning architectures showed heavier signs of overfitting during training in comparison to Deep Metric Learning.
• Deep Metric Learning showed very similar results which could improve classic Deep Learning methods with more data and additional watermelon varieties.
• Compensate the low performance with additional non-invasive methods (Ensemble Learning or Voting Scheme).
• Data Augmentation did not yield good results, feature patterns are complex.

​

• Dificulties to correctly identify the Joya class:
• Higher degree of similarity to other varieties.
• Lower availability of training samples.

• Future works:
• Test more complex methods such as Zero-shot Learning – if we have more watermelon varieties.
• Experiment with Generative Adversarial Neural Networks (GANs) – generate artificial samples to account data imbalances.

Conclusions and future work

+

I

Works

• A. H. Royo, K. Kung, K. Jo, and D. C. Hernández, “Design and implementation of a smart system for watermelon recognition,” in 2019 12th International Conference on Human System Interaction (HSI), pp. 82–86, 2019.

​

• J. E. Sánchez-Galán, A. Henry, F. Rangel, E. Sáez, K.-H. Jo, and D. Cáceres-Hernández, “Recognition of multiple panamanian watermelon varieties based on feature extraction analysis,” in Intelligent Computing Theories and Application (D.-S. Huang, K.-H. Jo, J. Li, V. Gribova, and A. Hussain, eds.), (Cham), pp. 65–75, Springer International Publishing, 2021.

​

• F. Rangel, E. Sáez, A. Henry, D. Cáceres-Hernández, and J. S. Galán, “Using visible/near-infrared reflectance spectroscopy and chemometrics for the rapid evaluation of two panamanian watermelon (Citrullus lanatus) varieties,” in 2021 IEEE 30th International Symposium on Industrial Electronics (ISIE), pp. 1–6, IEEE, 2021.

​

• Contreras, Kenji; Henry, Anel; Cáceres-Hernández, Danilo; Sanchez-Galan, Javier E; ",Comparing Convolutional Neural Networks and Deep Metric Learning Methods for Classification of Export Watermelon (Citrullus lanatus) Varieties,2022 IEEE 31st International Symposium on Industrial Electronics (ISIE),1141-1146,2022,IEEE

​

• Sánchez-Galán, Javier E; Henry-Royo, Anel; Jo, Kang-Hyun; Cáceres-Hernández, Danilo; ",Automatic Feature Detection and Classification for Watermelon (Citrillus lanatus),2022 International Workshop on Intelligent Systems (IWIS),1-7,2022,IEEE

​

• Caceres-Hernandez, Danilo; Gutierrez, Ricardo; Kung, Kelvin; Rodriguez, Juan; Lao, Oscar; Contreras, Kenji; Jo, Kang-Hyun; Sanchez-Galan, Javier E; ",Recent Advances in Automatic Feature Detection and Classification of Fruits including with a special emphasis on Watermelon (Citrillus lanatus): a Review,Neurocomputing,2023,Elsevier

​

​

I

This work was supported by the Secretaría Nacional de Ciencia, Tecnología e Innovación de Panamá (SENACYT) Grant funded by the Panamanian Government (Project 165-2019-FID18-060).

Acknowlegments

• Funding:

• Administrative support:

Analisis de Imagenes Hiperespectrales de cultivo de Arroz

+

Objetivo: El proyecto IDDS15-184 y el Proyecto APY-NI-2017-16, tienen como objetivo estudiar herramientas tecnológicas que sirvan para el monitoreo y manejo de áreas agrícolas en Panamá, entre los casos de estudio está el cultivo de arroz (Oriza sativa L.) en Coclé, Panamá; caracterizando las firmas espectrales (reflectancia, UV-NIR) por tipo y estado del cultivo.

UTP-CIHH

Dr. Jose Fabrega

Mgter. Ulises Jiimenez

Colaboradores:

UTP-CEPIA

Dr. Javier Sanchez Galan

Ing. Jorge Serrano

“Diseño de un sistema experto basado en firmas

espectrales de coberturas agropecuarias en Panamá” y “Teledetección de índice de área foliar para diversos estados fenológicos del Cultivo de Arroz (Oryza Sativa)” (2017-2019)

​

IDIAP

Dra. Evelyn Quiroz

+

The Team

Principal Investigator

PhD. Jose Fabrega

CIHH - Universidad Tecnológica de Panamá

Co-Investigator 1

Ph D.Javier Sánchez Galán

CEPIA / Facultad de Ingeniería de Sistemas Computacionales

Universidad Tecnológica de Panamá

Co-Investigator 2

Dra Evelyn Quiros

IDIAP

Msc. Jorge Serrano

CEPIA / Facultad de Ingeniería Mecanica

Universidad Tecnológica de Panamá

Msc. Jose Ulises Jimenez

CIHH - Universidad Tecnológica de Panamá

+

+

Firmas Espectrales Obtenidas en Campo

+

Base de Datos Espectral de

Cultivos Nacionales

Serrano Reyes, J., Jiménez, J. U., Quirós-McIntire, E. I., Sanchez-Galan, J. E., & Fábrega, J. R. (2023). Comparing Two Methods of Leaf Area Index Estimation for Rice (Oryza sativa L.) Using In-Field Spectroradiometric Measurements and Multispectral Satellite Images. AgriEngineering, 5(2), 965-981.

​

+

Base de Datos Espectral de

Cultivos Nacionales

+

Coberturas Existentes en la Base de Datos

+

Flujo de Trabajo del Estudio

NDVI

El índice de vegetación de diferencia normalizada (NDVI) es una métrica ampliamente utilizada para cuantificar la salud y la densidad de la vegetación utilizando datos de sensores.

+

Flujo de Trabajo del Estudio

MTVI2

El método del Índice de Vegetación Triangular Modificado (MTVI2) es un índice de vegetación para detectar el contenido de clorofila de las hojas a escala del dosel, siendo relativamente insensible al índice de área foliar. Utiliza reflectancia en las bandas verde, roja y NIR.

+

Firmas Satelitales

+

Integración de la Base de Datos de Medidas Espectrales y de Imagenes Satelitales

Se seleccionarion las imagenes satelitales con fechas cercanas a las fechas de medicion

+

Tratamiento de Imágenes Satelitales

Bandas Espectrales y valores con las cuales se conforman las Imagenes del Satelite PlanetScope

​

​

​

​

​

Acotación de los puntos medidos con espectrorradiometro en las imagenes satelitales

​

​

​

​

​

​

​

​

​

​

​

+

¿Como Comparar las Firmas?

+

Métodos de Clasificación

+

Calcular distancia para clasificación angular

​

Donde (n) es el número de bandas

-Obtener el ángulo de separación entre la firma de referencia de una fase fenológica y la firma en estudio.

+

Calcular distancia para clasificación angular

=

+

Flujo de Trabajo del Estudio

+

=

+

¿Como Comparar las Firmas (mas eficiente)?

+

Clasificación de Imagenes

+

Clasificación de Imagenes

+

Transformar Firmas Espectrales a la Forma de Firmas Satelitales

+

Procedimiento de Recuperación de Imágenes Satelitales

+

Procedimiento de Recuperación de Imágenes Satelitales

Satelital

Espectral

+

Procedimiento de Recuperación de Imágenes Satelitales

Satelital

Espectral

+

Criterios de Recuperación de Imágenes Satelitales

Lo que se busca es que la imagen satelital:

​

• Que haya sido capturada lo mas cercano a las horas en que se realizo una medicion de campo.

​

• Que cubriera el area de estudio (fincas y parcelas) en la cual se realizan las mediciones en campo.

​

• Que no tuviera nubosidad.

+

Flujo de Trabajo del Estudio (Campo)

Determinacion de LAI (real)

+

Flujo de Trabajo del Estudio (Satelitales)

Determinacion de LAI (por satelite)

+

Resultados del Estudio

+

Agradecimientos

+

Works

​

• Serrano Reyes, J., Jiménez, J. U., Quirós-McIntire, E. I., Sanchez-Galan, J. E., & Fábrega, J. R. (2023). Comparing Two Methods of Leaf Area Index Estimation for Rice (Oryza sativa L.) Using In-Field Spectroradiometric Measurements and Multispectral Satellite Images. AgriEngineering, 5(2), 965-981.
• Sánchez-Galán, J. E., Barranco, F. R., Reyes, J. S., Quirós-McIntire, E. I., Jiménez, J. U., & Fábrega, J. R. (2019). Using Supervised Classification Methods for the Analysis of Multi-spectral Signatures of Rice Varieties in Panama. Advances in Science, Technology and Engineering Systems Journal, 6(2), 552-558.
• Serrano Reyes, J., Jiménez, J. U., Quirós-McIntire, E. I., Sanchez-Galan, J. E., & Fábrega, J. R. (2023). Comparing Two Methods of Leaf Area Index Estimation for Rice (Oryza sativa L.) Using In-Field Spectroradiometric Measurements and Multispectral Satellite Images. AgriEngineering, 5(2), 965-981.
• Serrano, J., Fábrega, J., Quirós, E., Sánchez-Galán, J., & Jiménez, J. U. (2018). Análisis prospectivo de la detección hiperespectral de cultivos de arroz (Oryza sativa L.). KnE Engineering, 69-79.
• Jiménez, J. U., Quirós-McIntire, E. I., Camargo-García, V., Serrano, J., Sánchez-Galán, J., & Fábrega, J. (2018). Caracterización morfológica y espectral de 6 variedades criollas de arroz (Oryza sativa L.) en Panamá.

+

Conclusiones

• El agro y las tecnologias de información son buenos amigos!
• Para lograr crear Agro-Industria 4.0 debemos
• Tener claro la tecnologia de sensado (y como se representan esas medidas)
• Ubicar como podemos usar estadistica o ML para crear modelos utiles en el agro (redes neuronales, quimiometria, clustering, otros modelos?)
• El Como interpretar los resultados, es decir como hacerle la vida mas facil al productor.
• Como convertir nuestras implementaciones en reduccion de costos!

+

Javier Sánchez Galán, PhD

javier.sanchezgalan@utp.ac.pa

http://biotecnologia.utp.ac.pa/

@j_sgalan

@utppanama @utpfisc�

Grupo de Investigación en Biotecnología, Bioinformática y Biología de Sistemas – GIBBS

​

+

+