Progress 03/01/21 to 09/07/22
Outputs
Target Audience:Target audiences include the precision agriculture industry and the research community. The project also illustrates a research pipeline from theory to implementation and practice for those who are seeking agriculture applications for their AI and computer vision techniques. Education-based audiences, such as undergraduate research students and graduate students, are also targeted. Changes/Problems:We have been very economical in the use of the funds from the project by taking advantage of the resources available to us. In Spring 2023, we received an industrial grant to work on a project that is closely related to the research in this project, involving machine learning, image processing, and computer vision. We asked the Ph.D. student to share his time between the industry project and this NIFA project. The industry project covered most of his stipend and tuition. In addition, the industry grant also led to a paid summer internship for the Ph.D. student in the summer of 2022 to do work related to this NIFA project. During this time, the PI accepted the offer to move from South Dakota State University to the Florida Institute of Technology. The PI did not spend any funds from this grant during this transition period. Despite spending so little, we still made significant progress in this NIFA project with 80% of Objective 1, 25% of Objective 2, and 70% of Objective 3 completed. We have written two papers submitted to the "Computers and Electronics in Agriculture" journal, which is a top journal in the field of precision agriculture. One of these was accepted for publication while the other is under review. Furthermore, we filed a patent on the technique developed in this project with a licensing agreement with our industry partner, Raven Industries Inc, who has provided a lot of support and resources that alleviate the spending burden of this grant. Our plan after the grant is transferred to the new institution is to request a no-cost extension to accomplish the proposed research objectives. What opportunities for training and professional development has the project provided?Ph.D. student Praneel Acharya has been working on this project. The training and professional development activities for this student included learning knowledge and skills related to machine learning and computer vision, collecting droplet data, innovating and applying machine learning and computer vision techniques on the data to accomplish the goals, and communicating with engineers from Raven Industries Inc to coordinate the collaborative research tasks. In addition, during the project period, the student has also been trained in transferring the machine-learning and computer vision knowledge and skills to successfully obtain a full-time internship position in Summer 2022. How have the results been disseminated to communities of interest?We have been collaborating closely with a team of engineers from Raven Industries, Inc, a well-known manufacturer of precision agricultural equipment, especially agricultural nozzles and sprayer systems. Indeed, we have filed a patent together as a result of this collaboration. Additionally, we have written and submitted two manuscripts to the Computers and Electronics in Agriculture journal. This is a top journal in the field and is widely popular in the precision agriculture community. One paper was accepted for publication, while the other is currently under review. What do you plan to do during the next reporting period to accomplish the goals?The PI has moved from South Dakota State University to the Florida Institute of Technology. The PI is requesting to transfer the grant to his new institution to continue the proposed research. This Final Report is a required component of the grant transfer process. Below are the remaining objectives to be accomplished in the remaining of the project period: Objective 1: Develop a deep-learning framework for droplet detection with a fast-processing rate. The goal for the rest of the project period in this objective is to wrap up the training protocol for the developed deep-learning pipeline. We will use the performance metrics to guide the neural-network training process, collect and analyze the experimental results, and disseminate the findings via peer-review publications. We will also explore the feasibility of using generative machine learning to produce artificial data for the training of machine-learning models. If successful, this will lay the foundation for a transformative framework that helps reduce the need for collecting a large amount of data, which is usually expensive and time-consuming. Objective 2: Integrate a Kalman filter into the deep-learning framework for droplet tracking. In the next project period, we will complete the tracking task, in which the algorithm not only detects but also tracks each droplet. We are halfway there but still need to fine-tune the algorithm through testing with real data. This will be done by inferring the dynamics information of each droplet, labeling it, and then monitoring its motion throughout a video. The tracking algorithm is required when measuring the velocities of droplets. To accomplish these objectives, we will develop and implement an AI-enabled framework with a set of human-labeled images of droplets used in the training process, which will be specifically designed for droplets. The trained neural network will be then employed for both droplet detection and tracking. The AI-enabled droplet detection in Objective 1 is highly reliable and precise since only one image is processed, but takes a long processing time per image. For this reason, the detection will not be carried out every frame, but only once in 100 frames as a self-correction mechanism to minimize drifting errors during tracking. In between these frames, we perform a fast and simple detection by analyzing pixels and placing an ellipse at a region of interest that is most different from the background in terms of color and contrast properties. As a result, this fast detection is noisy and inaccurate. In addition to the noisy measurements, another key challenge with droplet tracking is droplet identification, i.e., how to tell which droplet is which among multiple images. Solving these two fundamental problems will allow the algorithm to record the dynamics of each droplet which would achieve the objective. We will address both challenges, namely noisy measurement and droplet identification, by developing a tracking algorithm grounded in the Kalman filter. This statistical technique is widely used in radar tracking and robotics navigation. Specifically, we will first establish a second-order dynamical model for the droplets, which is often used to describe the motion of a ball flying through space. More complicated and precise models can be found in fluid mechanics literature, but we will start with the simple second-order physical model. The model will enable the predictor to predict the state of a droplet (position and velocity) after each image. The scheme compares the predicted states with the noisy measurements of states from fast droplet detection. It then uses the comparison to update the covariance model and reject the noise. The entire process is iterated for the next image. The outcome of each iteration will be an ellipse with an identification label placed at a more precise region of interest that contains a droplet with high probability. Objective 3: Design and implement the metrics to assess the success rate of droplet detection and tracking. A critical component of this project is reliable and accurate metrics to assess the success rate of droplet detection and tracking methods developed in objectives 1 and 2. In this reporting period, the performance metrics have been established by the number of droplets successfully detected by the algorithm versus a ground truth as described above. In the rest of the project period, we will further refine the performance metric for droplet detection. In particular, to guarantee the independent and objective ground truth that can be related to the measurements extracted from the video sequence, we will use data manually and carefully gathered by a human operator who uses a 'point and click' user interface to select droplets that they can see. There is a probability of a region of interest containing a droplet coming out of each deep-learning classification calculation. This probability will be used as the confidence level. If the confidence level exceeds 95%, a droplet is considered successfully detected by the algorithm. We will compute the performance metrics of selected videos as the number of detected droplets over the ground truth (droplets detected by a human). If the performance metric is below 90%, we will tune the neural network, add training data, and iterate the process until the performance metrics rise beyond 90%. In addition, we will design and implement a separate set of performance metrics for tracking droplets as described in objective 2. This set of performance metrics will be used to guide the design of the droplet tracking algorithm.
Impacts
What was accomplished under these goals?
Objective 1: Develop a deep-learning framework for droplet detection with a fast-processing rate. (80% Accomplished) In the first year of the project we focused on research to develop an artificial intelligence (AI)-enabled framework capable of processing videos of sprayed droplets, detecting all droplets appearing across the image frames, and measuring droplet geometric and dynamic data. Our central hypothesis is that the integration of deep-learning techniques into the image processing algorithm will enable precise and reliable detection, tracking, and measuring of droplet properties. In addition, the framework will produce consistent results under a variety of uncertain imagery conditions. We formulated this hypothesis on the basis of our pilot study with a team of engineers from Raven Industries, an international precision agriculture company headquartered in Sioux Falls, SD. They provided the droplet data and technical discussions regarding spray nozzles and droplet dynamics. The rationale is that deep learning extracts meanings from imagery data, and that human-labeled data can be used to train a multi-layer neural network to increase the reliability and accuracy of droplet tracking. The droplet detection deep-learning framework that we have developed in this research task is composed of four modules: Module 1 - Feature extraction: During the feature extraction process, a deep neural network was designed to extract important characteristics of a droplet that can be used to uniquely identify it from another droplet present in the image. Additionally, this module refined the imagery data, leading to a decrease in image size by a reduction factor. For example, an image of 512 pixels by 512 pixels would be reduced to a final feature map of size 32 pixels by 32 pixels, with a reduction factor of 16. Hence, instead of directly dealing with the original image size, the droplet detection algorithm works with feature maps that have reduced size compared to the original image. Therefore, much less information needs to be processed, and this reduces processing time and computational power. Also, our algorithm achieved this imagery reduction while preserving unique characteristics ("features") of each droplet present in the image. This module is fully completed. These features will be the input to the region proposal module. Module 2 - Region proposal: The goal of using the region proposal network is to extract regions from the input image that might contain a droplet. To do so, the region proposal module takes a feature map (which is an output from the feature extraction process) as an input. For any given pixel coordinates in a feature map, we can compute the corresponding pixel location in the input image. At each pixel of a feature map, we place a fixed number of rectangles called anchor boxes of different dimensions. For each anchor box in the feature map, we can get a corresponding rectangle in the input image. Each of these rectangles covers a section in the input image and each such section might have droplets in them. Therefore, a region proposal neural network needs to be formulated and well trained with labeled ground truth data to effectively find sections ("regions") that contain droplets with high probability. This module is fully completed. Module 3 - Region-of-Interest (RoI) pooling: From the region proposal module, we get numerous regions with their parameters, including the size, location, and the probability of having an object in the region covered by a bounding box. Not all regions are of the same size (height and width). Thus, the main goal of ROI pooling is to resize each region obtained from the region proposal to have a fixed regular size. The outputs of this RoI pooling process are regions of the same size that may contain droplets or other objects in the background. This module is fully completed. Module 4 - Inference: This module is a refinement step, in addition to the above processes. Here, we formulate another neural network that takes the above regions as inputs and output regions with a high probability of containing only droplets (not other objects as before). In particular, the outputs from this module are the probability of a region containing a droplet and the location and size of the respective region. Similar to before, the goal of this neural network is to minimize a loss function specially formulated for refinement purposes. This module is fully completed. Objective 2: Integrate a Kalman filter into the deep-learning framework for droplet tracking. (25% Accomplished) The deep-learning algorithm developed in Objective 1 detects droplets in a given image frame. The detection algorithm considers images as independent ones and neglects the information implied between images, namely the dynamics of each droplet governed by physical laws. The deep-learning droplet detection algorithm is crucial in measuring droplet size. In this research task, we aim to solve the droplet tracking problem, in which the algorithm not only detects but also tracks each droplet. This is achieved by relating the position of each droplet between frames, inferring its dynamics information, labeling it, and then monitoring its motion throughout a video. In the first year of the project, we developed a correspondence matching algorithm to relate droplets between frames. We have also explored the idea of formulating a Kalman filter to predict the state of each droplet to maintain the continuation of droplet tracking. We are in the initial phase of this research task and will have more accomplishments to discuss in the next report. Objective 3: Design and implement the metrics to assess the success rate of droplet detection and tracking. (70% Accomplished) To compute the performance of the developed algorithm, we run the trained network through a set of evaluation images. Each evaluation image has a ground truth, in which every droplet in each evaluation image is manually labeled with a bounding box. Hence, the ground truth is the best possible performance resulted from careful inspection by a human operator. We pass each image through a trained model and compare the results with the ground truth. In doing so we store the following information for each image: 1) Total number of droplets missed by the trained model MD for a given image when compared to its ground truth; 2) Total number of incorrect detections FD made by the trained model for a given image when compared to its ground truth GT. These incorrect detections can occur when the trained model detects a droplet, but the ground truth does not consider it as a droplet; 3) Total number of detections by the trained model for the given image DT. Note that not all detections contain droplets. For example, FD consists of detections counted in DT that are not droplets when compared to the ground truth; 4) True detection TD which is computed by subtracting the total number of droplets detected by the trained model for the given image by the total number of incorrect detections for that image; and 5) Accuracy which is computed as TD/(TD+FD+MD). With this set of metrics, we penalize for every missed detection and any incorrect detection. As described, the value of accuracy will be 1 if the droplet detector result exactly matches the ground truth GT. With multiple images, overall accuracy is the average accuracy over all the experimental images. The accuracy defined here will be used as a performance metric to quantify the ability of the model to detect droplets.
Publications
- Type:
Journal Articles
Status:
Accepted
Year Published:
2022
Citation:
Acharya, P., T. Burgers, K.-D. Nguyen. 2022. AI-enabled droplet detection and tracking for agricultural spraying systems. Computers and Electronics in Agriculture. Accepted for publication.
- Type:
Journal Articles
Status:
Under Review
Year Published:
2022
Citation:
Acharya, P., K.D. Nguyen. 2022. A deep-learning framework for spray pattern segmentation and estimation in agricultural spraying systems. Computers and Electronics in Agriculture. Under review.
|
Progress 03/01/21 to 02/28/22
Outputs
Target Audience:Target audiences include the precision agriculture industry and the research community. The project also illustrates a research pipeline from theory to implementation and practice for those who are seeking agriculture applications for their AI and computer vision techniques. Education-based audiences, such as undergraduate research students and graduate students, are also targeted. Changes/Problems:
Nothing Reported
What opportunities for training and professional development has the project provided?PhD student Praneel Acharya working on this project. The training and professional development activities for this student include learning knowledge and skills related to machine learning and computer vision, collecting droplet data, innovating and applying machine learning and computer vision techniques on the data to accomplish the goals, and communicating with engineers from Raven Industries Inc to coordinate the collaborative research tasks. In addition, during the project period, the student has also been trained in transferring the machine-learning and computer vision knowledge and skills to successfully obtain a full-time internship position in Summer 2022. How have the results been disseminated to communities of interest?We have been collaborating closely with a team of engineers from Raven Industries, Inc, a well know manufacturer of precision agricultural equipment, especially agricultural nozzles and sprayer systems. Indeed, we have filed a patent together as a result of this collaboration. Additionally, we have written and submitted a manuscript to the Computers and Electronics in Agriculture journal. This is a top journal in the field and is widely popular in the precision agriculture community. What do you plan to do during the next reporting period to accomplish the goals?Objective 1: Develop a deep-learning framework for droplet detection with a fast-processing rate. The goal in the second year will be to design and implement a training protocol for the developed deep-learning pipeline. We will use the performance metrics to guide the neural-network training process, collect and analyze the experimental results, and disseminate the findings via peer-review publications. Objective 2: Integrate a Kalman filter into the deep-learning framework for droplet tracking. In the next project period, we aim to solve the tracking problem, in which the algorithm not only detects but also tracks each droplet. This will be done by inferring the dynamics information of each droplet, labeling it, and then monitoring its motion throughout a video. The tracking algorithm is required when measuring the velocities of droplets. To accomplish these objectives, we will develop and implement an AI-enabled framework with a set of human-labeled images of droplets used in the training process, which will be specifically designed for droplets. The trained neural network will be then employed for both droplet detection and tracking. The AI-enabled droplet detection in the objective 1 is highly reliable and precise since only one image is processed, but takes a long processing time per image. For this reason, the detection will not be carried out every frame, but only once in 100 frames as a self-correction mechanism to minimize drifting errors during tracking. In between these frames, we perform a fast and simple detection by analyzing pixels and placing an ellipse at an ROI that is most different from the background in terms of color and contrast properties. As a result, this fast detection is noisy and inaccurate. In addition to the noisy measurements, another key challenge with the droplet tracking problem is droplet identification, i.e., how to tell which droplet is which among multiple images. Solving these two fundamental problems will allow the algorithm to record the dynamics of each droplet which would achieve the objective. We will address both challenges, namely noisy measurement and droplet identification, by developing a tracking algorithm grounded in the Kalman filter. This statistical technique is widely used in radar tracking and robotics navigation. Specifically, we will first establish a second-order dynamical model for the droplets, which is often used to describe the motion of a ball flying through space. More complicated and precise models can be found in fluid mechanics literature, but we will start with the simple second-order physical model. The model will enable the predictor to predict the state of a droplet (position and velocity) after each image. The scheme compares the predicted states with the noisy measurements of states from fast droplet detection. It then uses the comparison to update the covariance model and reject the noise. The entire process is iterated for the next image. The outcome of each iteration will be an ellipse with an identification label placed at a more precise region of interest that contains a droplet with high probability. Objective 3: Design and implement the metrics to assess the success rate of droplet detection and tracking. A critical component of this project is reliable and accurate metrics to assess the success rate of droplet detection and tracking methods developed in objectives 1 and 2. In this reporting period, the performance metrics have been established by the number of droplets successfully detected by the algorithm versus a ground truth as described above. In Year 2 of the project, we will further refine the performance metric for droplet detection. In particular, to guarantee the independent and objective ground truth that can be related to the measurements extracted from the video sequence, we will use data manually and carefully gathered by a human operator who uses a 'point and click' user interface to select droplets that they can see. There is a probability of a region of interest containing a droplet coming out of each deep-learning classification calculation. This probability will be used as the confidence level. If the confidence level exceeds 95%, a droplet is considered successfully detected by the algorithm. We will compute the performance metrics of selected videos as the number of detected droplets over the ground truth (droplets detected by a human). If the performance metric is below 90%, we will tune the neural network, add training data, and iterate the process until the performance metrics rise beyond 90%. In addition, we will design and implement a separate set of performance metrics for tracking droplets as described in objective 2. This set of performance metrics will be used to guide the design of the droplet tracking algorithm.
Impacts
What was accomplished under these goals?
Objective 1: Develop a deep-learning framework for droplet detection with a fast-processing rate. (80% Accomplished) In the first year of the project, we focused on research to develop an artificial intelligence (AI)-enabled framework capable of processing videos of sprayed droplets, detecting all droplets appearing across the image frames, and measuring droplet geometric and dynamic data. Our central hypothesis is that the integration of deep-learning techniques into the image processing algorithm will enable precise and reliable detection, tracking, and measuring of droplet properties. In addition, the framework will produce consistent results under a variety of uncertain imagery conditions. We formulated this hypothesis on the basis of our pilot study with a team of engineers from Raven Industries, an international precision agriculture company headquartered in Sioux Falls, SD. They provided the droplet data and technical discussions regarding spray nozzles and droplet dynamics. The rationale is that deep learning extracts meanings from imagery data, and that human-labeled data can be used to train a multi-layer neural network to increase the reliability and accuracy of droplet tracking. The droplet detection deep-learning framework that we have developed in this research task is composed of four modules: Module 1 - Feature extraction: During the feature extraction process, a deep neural network was designed to extract important characteristics of a droplet that can be used to uniquely identify it from another droplet present in the image. Additionally, this module refined the imagery data, leading to a decrease in image size by a reduction factor. For example, an image of 512 pixels by 512 pixels would be reduced to a final feature map of size 32 pixels by 32 pixels, with a reduction factor of 16. Hence, instead of directly dealing with the original image size, the droplet detection algorithm works with feature maps that have reduced size compared to the original image. Therefore, much less information needs to be processed, and this reduces processing time and computational power. Also, our algorithm achieved this imagery reduction while preserving unique characteristics ("features") of each droplet present in the image. This module is fully completed. These features will be the input to the region proposal module. Module 2 - Region proposal: The goal of using the region proposal network is to extract regions from the input image that might contain a droplet. To do so, the region proposal module takes a feature map (which is an output from the feature extraction process) as an input. For any given pixel coordinates in a feature map, we can compute the corresponding pixel location in the input image. At each pixel of a feature map, we place a fixed number of rectangles called anchor boxes of different dimensions. For each anchor box in the feature map, we can get a corresponding rectangle in the input image. Each of these rectangles covers a section in the input image and each such section might have droplets in them. Therefore, a region proposal neural network needs to be formulated and well trained with labeled ground truth data to effectively find sections ("regions") that contain droplets with high probability. This module is fully completed. Module 3 - Region-of-Interest (RoI) pooling: From the region proposal module, we get numerous regions with their parameters, including the size, location, and the probability of having an object in the region covered by a bounding box. Not all regions are of the same size (height and width). Thus, the main goal of ROI pooling is to resize each region obtained from the region proposal to have a fixed regular size. The outputs of this RoI pooling process are regions of the same size that may contain droplets or other objects in the background. This module is fully completed. Module 4 - Inference: This module is a refinement step, in addition to the above processes. Here, we formulate another neural network that takes the above regions as inputs and output regions with a high probability of containing only droplets (not other objects as before). In particular, the outputs from this module are the probability of a region containing a droplet and the location and size of the respective region. Similar to before, the goal of this neural network is to minimize a loss function specially formulated for refinement purposes. This module is fully completed. Objective 2: Integrate a Kalman filter into the deep-learning framework for droplet tracking. (20% Accomplished) The deep-learning algorithm in the previous research task detects droplets in a given image frame. The detection algorithm considers images as independent ones and neglects the information implied between images, namely the dynamics of each droplet governed by physical laws. The deep-learning droplet detection algorithm is crucial in measuring droplet size. In this research task, we aim to solve the droplet tracking problem, in which the algorithm not only detects but also tracks each droplet. This is achieved by relating the position of each droplet between frames, inferring its dynamics information, labeling it, and then monitoring its motion throughout a video. In the first year of the project, we developed a correspondence matching algorithm to relate droplets between frames. We have also explored the idea of formulating a Kalman filter to predict the state of each droplet to maintain the continuation of droplet tracking. We are in the initial phase of this research task and will have more accomplishments to discuss in the next report. Objective 3: Design and implement the metrics to assess the success rate of droplet detection and tracking. (70% Accomplished) To compute the performance of the developed algorithm, we run the trained network through a set of evaluation images. Each evaluation image has a ground truth, in which every droplet in each evaluation image is manually labeled with a bounding box. Hence, the ground truth is the best possible performance resulted from careful inspection by a human operator. We pass each image through a trained model and compare the results with the ground truth. In doing so we store the following information for each image: 1) Total number of droplets missed by the trained model MD for a given image when compared to its ground truth. 2) Total number of incorrect detections FD made by the trained model for a given image when compared to its ground truth GT. These incorrect detections can occur when the trained model detects a droplet, but the ground truth does not consider it as a droplet. 3) Total number of detections by the trained model for the given image DT. Note that not all detections contain droplets. For example, FD consists of detections counted in DT that are not droplets when compared to the ground truth. 4) True detection TD which is computed by subtracting the total number of droplets detected by the trained model for the given image by the total number of incorrect detections for that image. 5) Accuracy which is computed as TD/(TD+FD+MD). With this set of metrics, we penalize for every missed detection and any incorrect detection. As described, the value of accuracy will be 1 if the droplet detector result exactly matches the ground truth GT. With multiple images, overall accuracy is the average accuracy over all the experimental images. The accuracy defined here will be used as a performance metric to quantify the ability of the model to detect droplets. With the performance metrics defined, the next section will discuss the training process and droplet detection results. Work on this objective will begin after progress in the prior objectives is made.
Publications
- Type:
Journal Articles
Status:
Under Review
Year Published:
2022
Citation:
Acharya, P., T. Burgers, K.-D. Nguyen. 2022. AI-enabled droplet detection and tracking for agricultural spraying systems. Computers and Electronics in Agriculture. Under Review.
|