Progress 09/15/22 to 09/14/24
Outputs
Target Audience:Target audiences include the precision agriculture industry and the research community. In fact, we have been collaborating closely with a team of engineers from Raven Industries (now acquired by CNH Industrial), a large manufacturer of precision agricultural equipment, especially agricultural nozzles and sprayer systems. We have filed a patent together as a result of this industry collaboration. The project also illustrates a research pipeline from theory to implementation and practice for those who are seeking agriculture applications for their AI and computer vision techniques. Educationalaudiences, such as undergraduate research students and graduate students, will also benefit from this research. Changes/Problems:
Nothing Reported
What opportunities for training and professional development has the project provided?Three Ph.D. students have been working on this project under full or partial support: Praneel Acharyawas a Ph.D. student working on this project at South Dakota State University (SDSU). In Fall 2022, the PI moved from SDSU to the Florida Institute of Technology (FIT). Praneel decided to stay at SDSU to finish his Ph.D. program since that was his final year. He is now an assistant professor atMinnesota State University, Mankato Nhut HuynhandHanh Phamare the two newly recruited Ph.D. students working on the project at FIT since 01/2023. The training and professional development activities for thesestudents includelearning knowledge and skills related to machine learning and computer vision, collecting droplet data, innovating and applying machine learning and computer vision techniques on the data to accomplish the goals, and communicating with engineers from Raven Industries Inc to coordinate the collaborative research tasks. In addition, during the project period, the students havealso been trained in transferring the machine-learning and computer vision knowledge and skills to successfully obtain a full-time internship position at Raven Industries in Summer 2022 and Spring 2023. How have the results been disseminated to communities of interest?We have been collaborating closely with a team of engineers from Raven Industries (now acquired by CNH Industrial), a well-known manufacturer of precision agricultural equipment, especially agricultural nozzles and sprayer systems. Indeed, we have filed a patent together as a result of this collaboration. Additionally, we have published 4 journal papers and 2other journal articles under peer review. We also presented our project at the USDA NIFA AI in Agriculture in Orlando FL in 2023, and the NIFA PD meeting in Manhattan KS in 2024. What do you plan to do during the next reporting period to accomplish the goals?
Nothing Reported
Impacts
What was accomplished under these goals?
Objective 1: We have developed a deep-learning framework capable of processing videos of sprayed droplets, detecting all droplets appearing across the image frames, and measuring droplet geometric and dynamic data.We have been closely collaborating witha team of engineers from Raven Industries, a precision agriculture company headquartered in Sioux Falls, SD. They provided the droplet data and technical discussions regarding spray nozzles and droplet dynamics. The droplet detection deep-learning framework that we have developed in this research task is composed of four modules: Feature extraction: A deep neural network was designed to extract important characteristics of a droplet and produce feature maps. Region proposal: The region proposal network narrows down regions from the input image that might contain a droplet. Region-of-Interest (RoI) pooling: The main goal of ROI pooling is to resize each region obtained from the region proposal to have a fixed regular size. Inference: This refinement step is another neural network that takes the above regions as inputs and output regions with a high probability of containing only droplets. In particular, the outputs from this module are the probability of a region containing a droplet and the location and size of the respective region. We have finalized the training protocol for the developed deep-learning pipeline. We used the performance metrics to guide the neural network training process, collect and analyze the experimental results, and disseminate the findings to Raven Industries (now acquired by CNH industrial) and via peer-reviewed publications. In addition, we have finished the algorithm to segment the spray pattern and estimate the spray angle. We produced innovative modifications to our current framework to enable it to be deployed and operate on mobile platforms. Additionally, we have investigated a method to classify droplets based on their size and further improved droplet detection performance. We have also explored the feasibility of using generative machine learning to produce artificial data for the training of machine-learning models. This lays the foundation for a transformative framework that helps reduce the need to collect a large amount of data, which is usually expensive and time-consuming. In addition to detecting individual droplets, we have created a tool forspray pattern segmentation and spray cone angle estimation.These two characteristics are important to understanding the sprayer system such as nozzles used in agriculture. The core of this work includes three deep-learning convolution-based models. These models are trained and their performances are compared. After the best model is selected based on its performance, it is used for spray region segmentation and spray cone angle estimation. The output from the selected model provides a binary image representing the spray region. This binary image is further processed using image processing to estimate the spray cone angle. The validation process is designed to compare results obtained from this work with manual measurements. Objective 2: The deep-learning algorithm developed in Objective 1 detects droplets in a given image frame. The detection algorithm considers images as independent ones and neglects the information implied between images, namely the dynamics of each droplet governed by physical laws. The deep-learning droplet detection algorithm is crucial in measuring droplet size. In this research task, we aim to solve the droplet tracking problem, in which the algorithm not only detects but also tracks each droplet. This is achieved by relating the position of each droplet between frames, inferring its dynamics information, labeling it, and then monitoring its motion throughout a video. We have designed and implemented a deep-learning-based tracking techniqueto achieve the desirable precision and accuracy of droplet tracking. Moreover, these methods enable the framework to efficiently operate in real time on a mobile platform. The general tracking procedure is presented as follows: Droplets' Presence in the Initial Video Frame:The tracking process begins with the identification of droplets in the first frame. This step involves analyzing the image or video frame to detect regions corresponding to droplets. Assigning Identities to Detected Droplets:To establish identity continuity, each detected droplet is assigned a unique identity or label. This step ensures that the same droplet in subsequent frames can be correctly associated, despite changes in appearance or occlusions. Tracking and Temporal Data Association:By leveraging Kalman filtering, the framework establishes correspondences between droplets in the current frame and those in the subsequent frames. This temporal data association allows for the tracking of individual droplets over time, enabling the analysis of their motion, interactions, and behavior. Real-time Operation on Mobile Platform:The developed deep-learning-based tracking technique enhances the precision and accuracy of droplet tracking, enabling the framework to operate efficiently in real-time on a mobile platform. We have fine-tuned the algorithm through testing with real data. This was done by inferring the dynamics information of each droplet, labeling it, and then monitoring its motion throughout a video. The tracking algorithm is required when measuring the velocities of droplets. We addressed the challenges associated with noisy measurement and droplet identification by developing a tracking algorithm grounded in the Kalman filter. Specifically, we first established a second-order dynamical model for the droplets. The model enables the predictor to predict the state of a droplet (position and velocity) after each image. The scheme compares the predicted states with the noisy measurements of states from fast droplet detection. It then uses the comparison to update the covariance model and reject the noise. The entire process is iterated for the next image. Objective 3: A critical component of this project is reliable and accurate metrics to assess the success rate of droplet detection and tracking methods developed in objectives 1 and 2. The performance metrics have been established by the number of droplets successfully detected by the algorithm versus the ground truth, including Intersection over Union,Precision-Recall Curve, Average Precision (AP), and Average Precision (mAP). To guarantee the independent and objective ground truth that can be related to the measurements extracted from the video sequence, we used data manually and carefully gathered by a human operator who uses a 'point and click' user interface to select droplets that they can see. There is a probability of a region of interest containing a droplet coming out of each deep-learning classification calculation. This probability is used as the confidence level. If the confidence level exceeds 95%, a droplet is considered successfully detected by the algorithm. The AI model computes the performance metrics of selected videos as the number of detected droplets over the ground truth (droplets detected by a human). If the performance metric is below 90%, we tunethe neural network, add training data, and iterate the process until the performance metrics rise above 90%. In addition, we designed and implemented a separate set of performance metrics for tracking droplets as described in objective 2. This set of performance metrics was used to guide the design of the droplet tracking algorithm. We have accomplished significant scientific contributions to both precision agriculture and machine learning fields. The project has resulted in 6 journal papers and a patent. We have disseminated our results to our industry partner, as well as presented our research at the USDA NIFA AI in Agriculture conference in Orlando FL in 2023, and the NIFA PD meeting in Manhattan KS in 2024.
Publications
- Type:
Journal Articles
Status:
Published
Year Published:
2024
Citation:
Huynh, N. and Nguyen, K.D., 2024. Real-time droplet detection for agricultural spraying systems: A deep learning approach. Machine Learning and Knowledge Extraction, 6(1), pp.259-282.
- Type:
Journal Articles
Status:
Published
Year Published:
2023
Citation:
Acharya, P., Burgers, T. and Nguyen, K.D., 2023. A deep-learning framework for spray pattern segmentation and estimation in agricultural spraying systems. Scientific Reports, 13(1), p.7545.
- Type:
Journal Articles
Status:
Published
Year Published:
2022
Citation:
Acharya, P., Burgers, T. and Nguyen, K.D., 2022. AI-enabled droplet detection and tracking for agricultural spraying systems. Computers and Electronics in Agriculture, 202, p.107325.
- Type:
Journal Articles
Status:
Under Review
Year Published:
2024
Citation:
Huynh, T.N. and Nguyen, K.D., 2024. Deep-Learning Methods for Efficient Real-Time Droplet Tracking in Crop-Spraying Systems. Under review.
- Type:
Journal Articles
Status:
Under Review
Year Published:
2024
Citation:
Pham, T.H. and Nguyen, K.D., 2024. Enhanced Droplet Analysis Using Generative Adversarial Networks. arXiv preprint arXiv:2402.15909. Under review.
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2023
Citation:
Nhut Huynh, Praneel Acharya, Travis Burgers, Kim-Doang Nguyen, Droplet detection and tracking for agricultural spraying systems: A deep-learning approach , USDA NIFA AI in Agriculture Conference, April 17-19, 2023, Orlando, FL
- Type:
Journal Articles
Status:
Published
Year Published:
2023
Citation:
Nguyen, K.D., 2023. An adaptive control framework for a class of nonlinear time-delay systems. Transactions of the Institute of Measurement and Control, 45(7), pp.1271-1281.
|
Progress 09/15/22 to 09/14/23
Outputs
Target Audience:Target audiences include the precision agriculture industry and the research community. The project also illustrates a research pipeline from theory to implementation and practice for those who are seeking agriculture applications for their AI and computer vision techniques. Education-based audiences, such as undergraduate research students and graduate students, are also targeted. Changes/Problems: There are several major changes that affect the progress of the project in the last 15months: The PI moved from South Dakota State University (SDSU) to the Florida Institute of Technology (FIT) in 2022: The PI's contract with SDSU ended in May 2022 and the PI's position at FIT started in August 2022. The grant transfer process was completed in December 2022 when FIT received the award notification. Therefore, the PI cannot use the funds to progress, and the project was essentially halted from May 2022 to December 2022. The Ph.D. student who worked on the project with the PI at SDSU, Praneel Acharya, decided not to follow the PI to FIT because it was the final year of his graduate program (He is now an assistant professor at Minnesota State University, Mankato). Additionally, FIT officially received the grant in December 2022, and the timing left us with very limited options for new graduate student recruitment who can start in January 2023. As a result, we could not hire students who have substantial relevant skill sets and experience to immediately contribute toward the project objectives. Most of Spring 2023 was dedicated to training the new Ph.D. students in data science, machine learning, and crop spraying systems. This challenge caused further delays in executing the research tasks and achieving the project objectives. Despite these challenges, we still made significant progress in this project with 85% of Objective 1, 30% of Objective 2, and 70% of Objective 3 completed. We have published two papers in Computers and Electronics in Agricultureand Scientific Reports, which aretop journals in the field of precision agriculture.Furthermore, we filed a patent on the technique developed in this project with a licensing agreement with our industry partner, Raven Industries Inc, who has provided a lot of support and resources that alleviate the spending burden of this grant. We presented our work at the USDA NIFA AI in Agriculture: Innovation and Discovery to Equitably Meet Producer Needs and Perceptions Conference in Orlando, April 2023. We are requestinga no-cost extension to accomplish the proposed research objectives. What opportunities for training and professional development has the project provided?Three Ph.D. students have been working on this project under full or partial support: Praneel Acharya was a Ph.D. student working on this project at South Dakota State University (SDSU). In Fall 2022, the PI moved from SDSU to the Florida Institute of Technology (FIT). Praneel decided to stay at SDSU to finish his Ph.D. program since that was his final year. He is now an assistant professor atMinnesota State University, Mankato Nhut Huynh and Hanh Pham are the two newly recruited Ph.D. students working on the project at FIT. The training and professional development activities for thesestudents includelearning knowledge and skills related to machine learning and computer vision, collecting droplet data, innovating and applying machine learning and computer vision techniques on the data to accomplish the goals, and communicating with engineers from Raven Industries Inc to coordinate the collaborative research tasks. In addition, during the project period, the students havealso been trained in transferring the machine-learning and computer vision knowledge and skills to successfully obtain a full-time internship position in Summer 2022 and Spring 2023. How have the results been disseminated to communities of interest?We have been collaborating closely with a team of engineers from Raven Industries, Inc, a well-known manufacturer of precision agricultural equipment, especially agricultural nozzles and sprayer systems. Indeed, we have filed a patent together as a result of this collaboration. Additionally, we have published two manuscripts, one inComputers and Electronics in Agricultureand one inScientific Reports. We also presented our project at the USDA NIFA AI in Agriculture: Innovation and Discovery to Equitably Meet Producer Needs and Perceptions Conference in Orlando, April 2023. What do you plan to do during the next reporting period to accomplish the goals?Objective 1: Develop a deep-learning framework for droplet detection with a fast-processing rate. The goal for the rest of the project period in this objective is to wrap up the training protocol for the developed deep-learning pipeline. We will use the performance metrics to guide the neural-network training process, collect and analyze the experimental results, and disseminate the findings via peer-review publications. In addition, we will finish the algorithm to segment the spray pattern and estimate the spray angle. We will produce innovative modifications to our current framework to enable it to be deployed and operate on mobile platforms. Additionally, we will look into classifying droplets based on their size and look for innovative ways to further improve droplet detection performance. We will also explore the feasibility of using generative machine learning to produce artificial data for the training of machine-learning models. If successful, this will lay the foundation for a transformative framework that helps reduce the need for collecting a large amount of data, which is usually expensive and time-consuming. Objective 2: Integrate a Kalman filter into the deep-learning framework for droplet tracking. In the next project period, we will complete the tracking task, in which the algorithm not only detects but also tracks each droplet. We are halfway there but still need to fine-tune the algorithm through testing with real data. This will be done by inferring the dynamics information of each droplet, labeling it, and then monitoring its motion throughout a video. The tracking algorithm is required when measuring the velocities of droplets. To accomplish these objectives, we will develop and implement an AI-enabled framework with a set of human-labeled images of droplets used in the training process, which will be specifically designed for droplets. The trained neural network will be then employed for both droplet detection and tracking. The AI-enabled droplet detection in Objective 1 is highly reliable and precise since only one image is processed, but takes a long processing time per image. For this reason, the detection will not be carried out every frame, but only once in 100 frames as a self-correction mechanism to minimize drifting errors during tracking. In between these frames, we perform a fast and simple detection by analyzing pixels and placing an ellipse at a region of interest that is most different from the background in terms of color and contrast properties. As a result, this fast detection is noisy and inaccurate. In addition to the noisy measurements, another key challenge with droplet tracking is droplet identification, i.e., how to tell which droplet is which among multiple images. Solving these two fundamental problems will allow the algorithm to record the dynamics of each droplet which would achieve the objective. We will address both challenges, namely noisy measurement and droplet identification, by developing a tracking algorithm grounded in the Kalman filter. This statistical technique is widely used in radar tracking and robotics navigation. Specifically, we will first establish a second-order dynamical model for the droplets, which is often used to describe the motion of a ball flying through space. More complicated and precise models can be found in fluid mechanics literature, but we will start with the simple second-order physical model. The model will enable the predictor to predict the state of a droplet (position and velocity) after each image. The scheme compares the predicted states with the noisy measurements of states from fast droplet detection. It then uses the comparison to update the covariance model and reject the noise. The entire process is iterated for the next image. The outcome of each iteration will be an ellipse with an identification label placed at a more precise region of interest that contains a droplet with high probability. Objective 3: Design and implement the metrics to assess the success rate of droplet detection and tracking. A critical component of this project is reliable and accurate metrics to assess the success rate of droplet detection and tracking methods developed in objectives 1 and 2. In this reporting period, the performance metrics have been established by the number of droplets successfully detected by the algorithm versus a ground truth as described above. In the rest of the project period, we will further refine the performance metric for droplet detection. In particular, to guarantee the independent and objective ground truth that can be related to the measurements extracted from the video sequence, we will use data manually and carefully gathered by a human operator who uses a 'point and click' user interface to select droplets that they can see. There is a probability of a region of interest containing a droplet coming out of each deep-learning classification calculation. This probability will be used as the confidence level. If the confidence level exceeds 95%, a droplet is considered successfully detected by the algorithm. We will compute the performance metrics of selected videos as the number of detected droplets over the ground truth (droplets detected by a human). If the performance metric is below 90%, we will tune the neural network, add training data, and iterate the process until the performance metrics rise beyond 90%. In addition, we will design and implement a separate set of performance metrics for tracking droplets as described in objective 2. This set of performance metrics will be used to guide the design of the droplet tracking algorithm.
Impacts
What was accomplished under these goals?
Objective 1: Develop a deep-learning framework for droplet detection with a fast-processing rate. (85% Accomplished) We have focused on the development of a deep-learning framework capable of processing videos of sprayed droplets, detecting all droplets appearing across the image frames, and measuring droplet geometric and dynamic data.We have been closely collaborating witha team of engineers from Raven Industries, an international precision agriculture company headquartered in Sioux Falls, SD. They provided the droplet data and technical discussions regarding spray nozzles and droplet dynamics. The droplet detection deep-learning framework that we have developed in this research task is composed of four modules: Module 1 - Feature extraction: During the feature extraction process, a deep neural network was designed to extract important characteristics of a droplet and produce feature maps.These feature mapswill be the input to the region proposal module. Module 2 - Region proposal: The goal of using the region proposal network is to extract regions from the input image that might contain a droplet. To do so, the region proposal module takes a feature map as input andplaces a fixed number of rectangles called anchor boxes of different dimensions at each pixel of a feature map. For each anchor box in the feature map, we can get a corresponding rectangle in the input image. Each of these rectangles covers a section in the input image and each such section might have droplets in them. Module 3 - Region-of-Interest (RoI) pooling: From the region proposal module, we get numerous regions with their parameters, including the size, location, and probability of having an object in the region covered by a bounding box. Not all regions are of the same size (height and width). Thus, the main goal of ROI pooling is to resize each region obtained from the region proposal to have a fixed regular size. The outputs of this RoI pooling process are regions of the same size that may contain droplets or other objects in the background. Module 4 - Inference: This module is a refinement step, in addition to the above processes. Here, we formulate another neural network that takes the above regions as inputs and output regions with a high probability of containing only droplets (not other objects as before). In particular, the outputs from this module are the probability of a region containing a droplet and the location and size of the respective region. The goal of this neural network is to minimize a loss function specially formulated for refinement purposes. In addition to detecting individual droplets, we have created a tool forspray pattern segmentation and spray cone angle estimation.These two characteristics are important to understanding the sprayer system such as nozzles used in agriculture. The core of this work includes three deep-learning convolution-based models. These models are trained and their performances are compared. After the best model is selected based on its performance, it is used for spray region segmentation and spray cone angle estimation. The output from the selected model provides a binary image representing the spray region. This binary image is further processed using image processing to estimate the spray cone angle. The validation process is designed to compare results obtained from this work with manual measurements. Objective 2: Integrate a Kalman filter into the deep-learning framework for droplet tracking. (30% Accomplished) The deep-learning algorithm developed in Objective 1 detects droplets in a given image frame. The detection algorithm considers images as independent ones and neglects the information implied between images, namely the dynamics of each droplet governed by physical laws. The deep-learning droplet detection algorithm is crucial in measuring droplet size. In this research task, we aim to solve the droplet tracking problem, in which the algorithm not only detects but also tracks each droplet. This is achieved by relating the position of each droplet between frames, inferring its dynamics information, labeling it, and then monitoring its motion throughout a video. Up to this point in the project, we have designed and implemented a deep-learning-based tracking techniqueto achieve the desirable precision and accuracy of droplet tracking. Moreover, these methods enable the framework to efficiently operate in realtime on a mobile platform. The general tracking procedure is presented as follows: Droplets' Presence in the Initial Video Frame: The tracking process begins with the identification of droplets in the first frame. This step involves analyzing the image or video frame to detect regions corresponding to droplets. Various object detection techniques, such as deep learning-based approaches or feature-based methods, can be employed to accurately detect and localize droplets within the frame. Assigning Identities to Detected Droplets: To establish identity continuity, each detected droplet is assigned a unique identity or label. This step ensures that the same droplet in subsequent frames can be correctly associated, despite changes in appearance or occlusions. Identity assignment can be achieved by associating a unique identifier, such as a numerical value or a unique tag, with each detected droplet. Tracking and Temporal Data Association: The core of the droplet tracking framework lies in tracking the detected droplets across frames and associating them temporally. By employing techniques such as Kalman filtering, particle filtering, or data association algorithms, the framework establishes correspondences between droplets in the current frame and those in the subsequent frames. This temporal data association allows for the tracking of individual droplets over time, enabling the analysis of their motion, interactions, and behavior. Real-time Operation on Mobile Platform: The developed deep-learning-based tracking technique enhances the precision and accuracy of droplet tracking, enabling the framework to operate efficiently in real-time on a mobile platform. This objective is about 30% completed and will be the focus of the next project period. Objective 3: Design and implement the metrics to assess the success rate of droplet detection and tracking. (70% Accomplished) The ground truth represents the ideal performance achieved through meticulous human inspection. Thedroplet detections are compared to the corresponding ground truth annotations. The performance metrics for droplet detection include: Intersection over Union (IoU): IoU measures the overlap between the predicted bounding boxes and the ground truth bounding boxes. It quantifies how well the model's detections align with the actual droplet locations. A higher IoU indicates a better match. Precision-Recall Curve: For the droplet class, we calculate the precision and recall at different IoU thresholds. Precision represents the ratio of correctly detected droplets to the total number of detections, while recall measures the proportion of true droplets that are correctly detected. By varying the IoU threshold, we generate a precision-recall curve. Average Precision (AP):AP is calculated by computing the area under the precision-recall curve. It captures the overall detection performance across various levels of precision and recall. A higher AP indicates better accuracy in droplet detection. Mean Average Precision (mAP):To obtain the mAP, we average the AP values across all droplet classes. This accounts for variations in detection performance for different droplet sizes, shapes, or orientations. The mAP provides an overall assessment of the algorithm's ability to detect droplets accurately across multiple classes. This task is 70% completed. the remaining goal is to establish a set of performance metrics to evaluate the efficacy of the droplet tracking tool when it is fully developed.
Publications
- Type:
Journal Articles
Status:
Published
Year Published:
2022
Citation:
Acharya, P., Burgers, T., & Nguyen, K. D. (2022). AI-enabled droplet detection and tracking for agricultural spraying systems. Computers and Electronics in Agriculture, 202, 107325.
- Type:
Journal Articles
Status:
Published
Year Published:
2023
Citation:
Acharya, P., Burgers, T., & Nguyen, K. D. (2023). A deep-learning framework for spray pattern segmentation and estimation in agricultural spraying systems. Scientific Reports, 13(1), 1-14.
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2023
Citation:
Huynh, N, Acharya, P., Burgers, T., & Nguyen, K. D. (2023). Droplet detection and tracking for agricultural spraying systems: A deep-learning approach. USDA NIFA AI in Agriculture: Innovation and Discovery to Equitably Meet Producer Needs and Perceptions Conference, Orlando, April 2023.
|