Source: FLORIDA INSTITUTE OF TECHNOLOGY, INC. submitted to
AI-ENABLED DROPLET TRACKING FOR CROP SPRAYING SYSTEMS
Sponsoring Institution
National Institute of Food and Agriculture
Project Status
EXTENDED
Funding Source
Reporting Frequency
Annual
Accession No.
1029667
Grant No.
2021-67022-38910
Project No.
FLAW-2022-09599
Proposal No.
2022-09599
Multistate No.
(N/A)
Program Code
A1541
Project Start Date
Sep 15, 2022
Project End Date
Sep 14, 2024
Grant Year
2023
Project Director
Nguyen, K.
Recipient Organization
FLORIDA INSTITUTE OF TECHNOLOGY, INC.
150 W UNIVERSITY BLVD OFC
MELBOURNE,FL 329016975
Performing Department
(N/A)
Non Technical Summary
This project proposes to develop an artificial intelligence (AI) technique capable of processing images, detecting and tracking all droplets appearing across the image frames, and measuring the droplet size and motion. Our central hypothesis is that the integration of deep-learning techniques into the image processing algorithm will enable precise and reliable detection, tracking, and measuring of droplet motion and size. In addition, the framework will produce consistent results under a variety of uncertain imagery conditions. The rationale is that deep learning extracts meaning from imagery data and human-labeled data to train a learning scheme that increases the reliability and accuracy of droplet tracking.We plan to test this central hypothesis by pursuing the following three specific aims: 1) Develop a deep-learning framework for droplet detection with a fast processing rate, 2) Integrate a filtering algorithm into the deep-learning framework for droplet tracking, and 3) Design and implement the metrics to assess the success rate of droplet detection and tracking.The project outcomes will provide an AI solution to a challenging precision agriculture problem, namely measuring the dynamic property of droplets from a crop spraying system. The tool will lead to the next generation of agricultural nozzles with significantly improved performance, which will improve producer efficiencies and minimize chemical runoff that pollutes the environment.
Animal Health Component
0%
Research Effort Categories
Basic
20%
Applied
60%
Developmental
20%
Classification

Knowledge Area (KA)Subject of Investigation (SOI)Field of Science (FOS)Percent
40472992020100%
Goals / Objectives
The overarching goal of this research is to innovate computer vision and deep-learning techniques to track and measure the motion of droplets sprayed from an agricultural nozzle and captured by a high-speed camera. There is a direct relationship between the size and velocity of sprayed droplets and treatment effectiveness, therefore, the efficacy of a spraying system. Thus, droplet motion data, a measure of the droplets' size and velocity, is crucial in optimizing nozzle design. No method currently exists to track multiple dynamic droplets. Accordingly, there is an urgent need for this tracking and measurement technology.Our central hypothesis is that the integration of deep-learning techniques into the image processing algorithm will enable precise and reliable detection, tracking, and measurement of droplet properties. In addition, the framework will produce consistent results under a variety of uncertain imagery conditions. We plan to test this central hypothesis by pursuing the following three specific objectives:Objective 1. Develop a deep-learning framework for droplet detection with a fast processing rateObjective 2. Integrate a Kalman filter into the deep-learning framework for droplet trackingObjective 3. Design and implement the metrics to assess the success rate of droplet detection and trackingThe proposed research is original and transformative because it will create an advanced tool for a crucial and challenging precision agriculture problem, namely tracking and measuring droplet motion. This tool is currently missing and it will enable a significant improvement in spray nozzle design. It will lay a foundation for computer vision and AI technologies to solve agricultural problems. For example, we could extend the framework to track farm animals for extracting biometric data, for early detection of diseases, and for predicting reproductive behaviors to increase breeding success.
Project Methods
The outcome of this project will be a methodology for analyzing high-speed videos of droplets and extracting geometric and dynamic data of each droplet detected. The algorithm must produce reliable and precise results within a reasonable amount of time. To achieve this, we divide the project into three strategic research objectives as follows.Objective 1. Develop a deep-learning framework for droplet detection with a fast processing rateThe current experimental testbed with Raven's nozzles allows for closely capturing droplets as they move across the camera frame. To overcome the challenges described previously, we will implement a data-driven approach by designing a deep neural network (DNN) that will be trained by small data sets extracted from the droplet video.In particular, the training process will result in square boxes placed at possible regions of interest (ROIs) where droplets might be with high probability. Then, boxes are re-scaled and passed back to the DNN for classification of whether a certain box contains a droplet. Since the classification is performed by comparing the probability with a user-labeled dataset, we expect consistent results.One problem we anticipate is the process may take excessive time to be completed. There might be hundreds of boxes detected due to the similarity between the background and droplets' color and lighting conditions. We plan to address this bottleneck by first sharpening and filtering individual images, defining boundaries, then connecting similar pixels along the boundary by closed contours to identify ROIs. Since closed contours are used instead of generic square boxes, there will be significantly fewer ROIs to be processed, hence less processing time.Objective 2. Integrate a Kalman filter into the deep-learning framework for droplet trackingIn objective 1, we discussed the methodology for solving the detection problem, in which an algorithm processes individual images and tries to put a contour that best fits a detected droplet. However, the detection algorithm considers images as independent ones and neglects the information implied between images, namely the dynamics of each droplet governed by physical laws. The deep-learning droplet detection algorithm is crucial in measuring droplet size.In objective 2, we aim to solve the tracking problem, in which the algorithm not only detects, but also tracks each droplet. This is done by inferring dynamics information of each droplet, labeling it, and then monitoring its motion throughout a video. The tracking algorithm is required when measuring the velocities of droplets. To accomplish these objectives, we will develop and implement an AI-enabled framework. In particular, a set of human-labeled images of droplets is used to train a DNN, which was specifically designed for droplets. The trained DNN is then used for both droplet detection and tracking. AI-enabled droplet detection is highly reliable and precise since only one image is processed, but it takes a long processing time per image.For this reason, the AI-enabled detection should not be performed every frame, but only once in 100 frames as a self-correction mechanism to minimize drifting errors during tracking. In between these frames, we perform a fast and simple detection by analyzing pixels and placing an ellipse at an ROI that is most different from the background in terms of color and contrast properties. As a result, this fast detection is noisy and inaccurate. In addition to the noisy measurements, another key challenge with the droplet tracking problem is droplet identification, i.e. how to tell which droplet is which among multiple images. Solving these two fundamental problems will allow the algorithm to record the dynamics of each droplet which would achieve the goal.We will address both challenges, namely noisy measurement and droplet identification, by developing a tracking algorithm grounded in the Kalman filter. This statistical technique is widely used in radar tracking and robotics navigation, and we are implementing the filter in a separate robotics project. Specifically, we will first establish a second-order dynamical model for the droplets, which is often used to describe the motion of a ball flying through space. More complicated and precise models can be found in fluid mechanics literature, but we will start with the simple second-order physical model. The model will enable the predictor to predict the state of a droplet (position and velocity) after each image. The scheme compares the predicted states with the noisy measurements of states from fast droplet detection. It then uses the comparison to update the covariance model and reject the noise. The entire process is iterated for the next image. The outcome of each iteration will be an ellipse with an identification label placed at a more precise ROI that contains a droplet with high probability.Objective 3. Design and implement the metrics to assess the success rate of droplet detection and tracking A critical component of this project is the reliable and accurate metrics to assess the success rate of droplet detection and tracking methods developed in objectives 1 and 2. The performance metrics will be established by the number of droplets successfully detected by the algorithm versus a ground truth.To guarantee the independent and objective ground truth that can be related to the measurements extracted from the video sequence, we will use data manually and carefully gathered by a human operator who uses a point-and-clickuser interface to select droplets that s/he can see. There is a probability of ROI being a droplet coming out of each deep-learning classification calculation. This probability will be used as the confidence level. If the confidence level exceeds 95%, a droplet is considered successfully detected by the algorithm. We will compute the performance metrics of selected videos as the number of detected droplets over the ground truth (droplets detected by a human). If the performance metric is below 90%, we will tune the DNN, add training data, and iterate the process until the performance metrics rise beyond 90%.

Progress 09/15/22 to 09/14/23

Outputs
Target Audience:Target audiences include the precision agriculture industry and the research community. The project also illustrates a research pipeline from theory to implementation and practice for those who are seeking agriculture applications for their AI and computer vision techniques. Education-based audiences, such as undergraduate research students and graduate students, are also targeted. Changes/Problems: There are several major changes that affect the progress of the project in the last 15months: The PI moved from South Dakota State University (SDSU) to the Florida Institute of Technology (FIT) in 2022: The PI's contract with SDSU ended in May 2022 and the PI's position at FIT started in August 2022. The grant transfer process was completed in December 2022 when FIT received the award notification. Therefore, the PI cannot use the funds to progress, and the project was essentially halted from May 2022 to December 2022. The Ph.D. student who worked on the project with the PI at SDSU, Praneel Acharya, decided not to follow the PI to FIT because it was the final year of his graduate program (He is now an assistant professor at Minnesota State University, Mankato). Additionally, FIT officially received the grant in December 2022, and the timing left us with very limited options for new graduate student recruitment who can start in January 2023. As a result, we could not hire students who have substantial relevant skill sets and experience to immediately contribute toward the project objectives. Most of Spring 2023 was dedicated to training the new Ph.D. students in data science, machine learning, and crop spraying systems. This challenge caused further delays in executing the research tasks and achieving the project objectives. Despite these challenges, we still made significant progress in this project with 85% of Objective 1, 30% of Objective 2, and 70% of Objective 3 completed. We have published two papers in Computers and Electronics in Agricultureand Scientific Reports, which aretop journals in the field of precision agriculture.Furthermore, we filed a patent on the technique developed in this project with a licensing agreement with our industry partner, Raven Industries Inc, who has provided a lot of support and resources that alleviate the spending burden of this grant. We presented our work at the USDA NIFA AI in Agriculture: Innovation and Discovery to Equitably Meet Producer Needs and Perceptions Conference in Orlando, April 2023. We are requestinga no-cost extension to accomplish the proposed research objectives. What opportunities for training and professional development has the project provided?Three Ph.D. students have been working on this project under full or partial support: Praneel Acharya was a Ph.D. student working on this project at South Dakota State University (SDSU). In Fall 2022, the PI moved from SDSU to the Florida Institute of Technology (FIT). Praneel decided to stay at SDSU to finish his Ph.D. program since that was his final year. He is now an assistant professor atMinnesota State University, Mankato Nhut Huynh and Hanh Pham are the two newly recruited Ph.D. students working on the project at FIT. The training and professional development activities for thesestudents includelearning knowledge and skills related to machine learning and computer vision, collecting droplet data, innovating and applying machine learning and computer vision techniques on the data to accomplish the goals, and communicating with engineers from Raven Industries Inc to coordinate the collaborative research tasks. In addition, during the project period, the students havealso been trained in transferring the machine-learning and computer vision knowledge and skills to successfully obtain a full-time internship position in Summer 2022 and Spring 2023. How have the results been disseminated to communities of interest?We have been collaborating closely with a team of engineers from Raven Industries, Inc, a well-known manufacturer of precision agricultural equipment, especially agricultural nozzles and sprayer systems. Indeed, we have filed a patent together as a result of this collaboration. Additionally, we have published two manuscripts, one inComputers and Electronics in Agricultureand one inScientific Reports. We also presented our project at the USDA NIFA AI in Agriculture: Innovation and Discovery to Equitably Meet Producer Needs and Perceptions Conference in Orlando, April 2023. What do you plan to do during the next reporting period to accomplish the goals?Objective 1: Develop a deep-learning framework for droplet detection with a fast-processing rate. The goal for the rest of the project period in this objective is to wrap up the training protocol for the developed deep-learning pipeline. We will use the performance metrics to guide the neural-network training process, collect and analyze the experimental results, and disseminate the findings via peer-review publications. In addition, we will finish the algorithm to segment the spray pattern and estimate the spray angle. We will produce innovative modifications to our current framework to enable it to be deployed and operate on mobile platforms. Additionally, we will look into classifying droplets based on their size and look for innovative ways to further improve droplet detection performance. We will also explore the feasibility of using generative machine learning to produce artificial data for the training of machine-learning models. If successful, this will lay the foundation for a transformative framework that helps reduce the need for collecting a large amount of data, which is usually expensive and time-consuming. Objective 2: Integrate a Kalman filter into the deep-learning framework for droplet tracking. In the next project period, we will complete the tracking task, in which the algorithm not only detects but also tracks each droplet. We are halfway there but still need to fine-tune the algorithm through testing with real data. This will be done by inferring the dynamics information of each droplet, labeling it, and then monitoring its motion throughout a video. The tracking algorithm is required when measuring the velocities of droplets. To accomplish these objectives, we will develop and implement an AI-enabled framework with a set of human-labeled images of droplets used in the training process, which will be specifically designed for droplets. The trained neural network will be then employed for both droplet detection and tracking. The AI-enabled droplet detection in Objective 1 is highly reliable and precise since only one image is processed, but takes a long processing time per image. For this reason, the detection will not be carried out every frame, but only once in 100 frames as a self-correction mechanism to minimize drifting errors during tracking. In between these frames, we perform a fast and simple detection by analyzing pixels and placing an ellipse at a region of interest that is most different from the background in terms of color and contrast properties. As a result, this fast detection is noisy and inaccurate. In addition to the noisy measurements, another key challenge with droplet tracking is droplet identification, i.e., how to tell which droplet is which among multiple images. Solving these two fundamental problems will allow the algorithm to record the dynamics of each droplet which would achieve the objective. We will address both challenges, namely noisy measurement and droplet identification, by developing a tracking algorithm grounded in the Kalman filter. This statistical technique is widely used in radar tracking and robotics navigation. Specifically, we will first establish a second-order dynamical model for the droplets, which is often used to describe the motion of a ball flying through space. More complicated and precise models can be found in fluid mechanics literature, but we will start with the simple second-order physical model. The model will enable the predictor to predict the state of a droplet (position and velocity) after each image. The scheme compares the predicted states with the noisy measurements of states from fast droplet detection. It then uses the comparison to update the covariance model and reject the noise. The entire process is iterated for the next image. The outcome of each iteration will be an ellipse with an identification label placed at a more precise region of interest that contains a droplet with high probability. Objective 3: Design and implement the metrics to assess the success rate of droplet detection and tracking. A critical component of this project is reliable and accurate metrics to assess the success rate of droplet detection and tracking methods developed in objectives 1 and 2. In this reporting period, the performance metrics have been established by the number of droplets successfully detected by the algorithm versus a ground truth as described above. In the rest of the project period, we will further refine the performance metric for droplet detection. In particular, to guarantee the independent and objective ground truth that can be related to the measurements extracted from the video sequence, we will use data manually and carefully gathered by a human operator who uses a 'point and click' user interface to select droplets that they can see. There is a probability of a region of interest containing a droplet coming out of each deep-learning classification calculation. This probability will be used as the confidence level. If the confidence level exceeds 95%, a droplet is considered successfully detected by the algorithm. We will compute the performance metrics of selected videos as the number of detected droplets over the ground truth (droplets detected by a human). If the performance metric is below 90%, we will tune the neural network, add training data, and iterate the process until the performance metrics rise beyond 90%. In addition, we will design and implement a separate set of performance metrics for tracking droplets as described in objective 2. This set of performance metrics will be used to guide the design of the droplet tracking algorithm.

Impacts
What was accomplished under these goals? Objective 1: Develop a deep-learning framework for droplet detection with a fast-processing rate. (85% Accomplished) We have focused on the development of a deep-learning framework capable of processing videos of sprayed droplets, detecting all droplets appearing across the image frames, and measuring droplet geometric and dynamic data.We have been closely collaborating witha team of engineers from Raven Industries, an international precision agriculture company headquartered in Sioux Falls, SD. They provided the droplet data and technical discussions regarding spray nozzles and droplet dynamics. The droplet detection deep-learning framework that we have developed in this research task is composed of four modules: Module 1 - Feature extraction: During the feature extraction process, a deep neural network was designed to extract important characteristics of a droplet and produce feature maps.These feature mapswill be the input to the region proposal module. Module 2 - Region proposal: The goal of using the region proposal network is to extract regions from the input image that might contain a droplet. To do so, the region proposal module takes a feature map as input andplaces a fixed number of rectangles called anchor boxes of different dimensions at each pixel of a feature map. For each anchor box in the feature map, we can get a corresponding rectangle in the input image. Each of these rectangles covers a section in the input image and each such section might have droplets in them. Module 3 - Region-of-Interest (RoI) pooling: From the region proposal module, we get numerous regions with their parameters, including the size, location, and probability of having an object in the region covered by a bounding box. Not all regions are of the same size (height and width). Thus, the main goal of ROI pooling is to resize each region obtained from the region proposal to have a fixed regular size. The outputs of this RoI pooling process are regions of the same size that may contain droplets or other objects in the background. Module 4 - Inference: This module is a refinement step, in addition to the above processes. Here, we formulate another neural network that takes the above regions as inputs and output regions with a high probability of containing only droplets (not other objects as before). In particular, the outputs from this module are the probability of a region containing a droplet and the location and size of the respective region. The goal of this neural network is to minimize a loss function specially formulated for refinement purposes. In addition to detecting individual droplets, we have created a tool forspray pattern segmentation and spray cone angle estimation.These two characteristics are important to understanding the sprayer system such as nozzles used in agriculture. The core of this work includes three deep-learning convolution-based models. These models are trained and their performances are compared. After the best model is selected based on its performance, it is used for spray region segmentation and spray cone angle estimation. The output from the selected model provides a binary image representing the spray region. This binary image is further processed using image processing to estimate the spray cone angle. The validation process is designed to compare results obtained from this work with manual measurements. Objective 2: Integrate a Kalman filter into the deep-learning framework for droplet tracking. (30% Accomplished) The deep-learning algorithm developed in Objective 1 detects droplets in a given image frame. The detection algorithm considers images as independent ones and neglects the information implied between images, namely the dynamics of each droplet governed by physical laws. The deep-learning droplet detection algorithm is crucial in measuring droplet size. In this research task, we aim to solve the droplet tracking problem, in which the algorithm not only detects but also tracks each droplet. This is achieved by relating the position of each droplet between frames, inferring its dynamics information, labeling it, and then monitoring its motion throughout a video. Up to this point in the project, we have designed and implemented a deep-learning-based tracking techniqueto achieve the desirable precision and accuracy of droplet tracking. Moreover, these methods enable the framework to efficiently operate in realtime on a mobile platform. The general tracking procedure is presented as follows: Droplets' Presence in the Initial Video Frame: The tracking process begins with the identification of droplets in the first frame. This step involves analyzing the image or video frame to detect regions corresponding to droplets. Various object detection techniques, such as deep learning-based approaches or feature-based methods, can be employed to accurately detect and localize droplets within the frame. Assigning Identities to Detected Droplets: To establish identity continuity, each detected droplet is assigned a unique identity or label. This step ensures that the same droplet in subsequent frames can be correctly associated, despite changes in appearance or occlusions. Identity assignment can be achieved by associating a unique identifier, such as a numerical value or a unique tag, with each detected droplet. Tracking and Temporal Data Association: The core of the droplet tracking framework lies in tracking the detected droplets across frames and associating them temporally. By employing techniques such as Kalman filtering, particle filtering, or data association algorithms, the framework establishes correspondences between droplets in the current frame and those in the subsequent frames. This temporal data association allows for the tracking of individual droplets over time, enabling the analysis of their motion, interactions, and behavior. Real-time Operation on Mobile Platform: The developed deep-learning-based tracking technique enhances the precision and accuracy of droplet tracking, enabling the framework to operate efficiently in real-time on a mobile platform. This objective is about 30% completed and will be the focus of the next project period. Objective 3: Design and implement the metrics to assess the success rate of droplet detection and tracking. (70% Accomplished) The ground truth represents the ideal performance achieved through meticulous human inspection. Thedroplet detections are compared to the corresponding ground truth annotations. The performance metrics for droplet detection include: Intersection over Union (IoU): IoU measures the overlap between the predicted bounding boxes and the ground truth bounding boxes. It quantifies how well the model's detections align with the actual droplet locations. A higher IoU indicates a better match. Precision-Recall Curve: For the droplet class, we calculate the precision and recall at different IoU thresholds. Precision represents the ratio of correctly detected droplets to the total number of detections, while recall measures the proportion of true droplets that are correctly detected. By varying the IoU threshold, we generate a precision-recall curve. Average Precision (AP):AP is calculated by computing the area under the precision-recall curve. It captures the overall detection performance across various levels of precision and recall. A higher AP indicates better accuracy in droplet detection. Mean Average Precision (mAP):To obtain the mAP, we average the AP values across all droplet classes. This accounts for variations in detection performance for different droplet sizes, shapes, or orientations. The mAP provides an overall assessment of the algorithm's ability to detect droplets accurately across multiple classes. This task is 70% completed. the remaining goal is to establish a set of performance metrics to evaluate the efficacy of the droplet tracking tool when it is fully developed.

Publications

  • Type: Journal Articles Status: Published Year Published: 2022 Citation: Acharya, P., Burgers, T., & Nguyen, K. D. (2022). AI-enabled droplet detection and tracking for agricultural spraying systems. Computers and Electronics in Agriculture, 202, 107325.
  • Type: Journal Articles Status: Published Year Published: 2023 Citation: Acharya, P., Burgers, T., & Nguyen, K. D. (2023). A deep-learning framework for spray pattern segmentation and estimation in agricultural spraying systems. Scientific Reports, 13(1), 1-14.
  • Type: Conference Papers and Presentations Status: Published Year Published: 2023 Citation: Huynh, N, Acharya, P., Burgers, T., & Nguyen, K. D. (2023). Droplet detection and tracking for agricultural spraying systems: A deep-learning approach. USDA NIFA AI in Agriculture: Innovation and Discovery to Equitably Meet Producer Needs and Perceptions Conference, Orlando, April 2023.