Progress 07/01/23 to 06/30/24
Outputs
Target Audience: Researchers and students in academia who are in agriculture, engineering, and technology learning orconducting research in precision and digital agricultural technologies, site-specific crop management, remote sensing, andautonomous systems. The precision and digital agriculture industry, which develops drones, aerial applications, and crop management technologies, stands to benefit significantly from our project. We aim to develop the intelligent site-specific application technology which can be eventually integrated in the current commercial spraying drone systems, which could lead to significant advancements in the field. Public agencies, such as the Nebraska Weed Association and Nebraska NRD Environmental Services, are crucial in regulating agricultural aerial applications and agricultural and natural resources management. While we may not directly work with growers in this project, they are the ultimate beneficiary of our study, and their needs are the driving force behind ourproject direction. Changes/Problems:We requested the no-cost-extension to finish up the project objectives and publish the results. What opportunities for training and professional development has the project provided? Graduate and undergraduate student training: we have twoMaster students graduated whose thesis projects focused on orclosely related to this project. We havea PhD student focused on this project is graduating this year and will continue research in similar area (intelligent system development) as a postdoc researcher in academia. We had an undergadaute stduent worked closely with us in the past growing season. Nebraska Extension: we have been working with the lead educator and specialist inNebraska Extension to deliver some workshop talks with the project progress and results to farmers and other stakeholders. Invasive species group:our work has received attention by the invasive species management group. We were invited to host a field trip with a presentation and demonstration at the North American Invasive Species Management Association 31st Annual Conference held in Lincoln, NE, October 16-19, 2023 (https://conference.naisma.org/). The conference committee told us afterwards that they received "a ton of positive comments" from the attendees about our presentation and demonstration. And we later received an invitation from one attendee to give a presentation at the Great Lakes Phragmites Collaborative (https://www.greatlakesphragmites.net/) in their next research webinar series. Industry: we have been interactingwith private industry (drone and spray drone companies) to discuss project idea and seek their interests and inputs. How have the results been disseminated to communities of interest?The project results have been disseminated through various channels includingacademic conference presentations, posters, papers, extension and public talks, interactions with industry. What do you plan to do during the next reporting period to accomplish the goals?Objectives 1 and 3 are the key focuses according to the project goal; hence, lots of efforts have been made on those two objectives and we even accomplished more than what we proposed in the original proposal. However, the objective 2 was relatively behind and what we will catch up and complete in the NCE period, as well as keep progressing beyond objectives 1 and 3and wrapping up manuscript publications.
Impacts
What was accomplished under these goals?
Objective 1 is to devise an edge-computing component integrated into a sensing drone that maps the spatial pattern of weed pressure and generates rough prescription maps in real-time for time-sensitive applications. This objective has been well implemented as planned with about 70% accomplished and some additional achievements that were not included in the original proposal. We have already developed the software and hardware prototype of the edge-computing component that does real-time weed recognition with a compact imaging system to acquire images and a mini-computer running the deep learning algorithms onboard the drone. Various state-of-the-art CNN and vision-transformer-based models weredeveloped and deployed on the NVIDIA Jetson Nano and Orin companion computers and realized accurate andreal-time inference. The results were obtained using data collected from both row crop fields with weeds (Palmer amaranth) and the invasive species in riparian areas (Phragmites). A couple of conference presentations have been made in international conferences and two manuscripts are under preparation for this objective. Objective 2 is to develop a cloud-computing component for data storage that facilitates deriving more accurate prescription maps and long-term, user-contributed refinement of the decision-making model. This objective is a little behind plans with about 60% accomplished. Other than organizing the data asset for publishing later, we have been making theoretical progress ondemonstrating the federated learning based joint edge-cloud computing system in agricultural management applications. It has been tested on a formulated an energy-aware device scheduling problem to assign communication resources to the optimal edge node subset for minimizing the global loss function. This objective turns out to be less of a focus for this project as we've added much more relevant tasks; however, the finding so far can beimportant for both the theoretical and implementation of federated learning in edge-cloud networking in generic agricultural applications. Objective 3 is to customize and test a prototype of a site-specific sprayer drone that implements flight and application plans derived from the prescription map. This objective has also been well implemented as planned with about 70% accomplished and some key additional achievements that were not included in the original proposal. We are now having a working prototype of sprayer drone that can follow customized flight path controlled by Jetson mini-computer onboard with a customized spraying system. This system also features with its integrated sensing, computing, and actuation capability to do real-time weed identification (linked to Objective 1) and make decision on spraying which was not included in the proposal. A conference presentation will be delivered this summer in ASABE AIM and a manuscript will be written this fall.
Publications
- Type:
Conference Papers and Presentations
Status:
Accepted
Year Published:
2024
Citation:
Islam, M.D., Steele, K., Shi, Y., Pitla, S., Luck, J., Ge, Y., Zhang, K., Riggan, B., Jhala, A. & Knezevic, S. (2024) An update on developing a prototype of intelligent unmanned aerial application system. The American Society of Agricultural and Biological Engineers (ASABE), 2024 Annual meeting, 28 - 31 July 2024, Anaheim, CA.
- Type:
Other
Status:
Other
Year Published:
2024
Citation:
Steele, K., & Shi, Y. (February 2024). Machine Learning Detection of Phragmites Using Drone Based Imagery. 77th Annual Nebraska Weed Control Association (NWCA) Conference, Norfolk, Nebraska.
- Type:
Other
Status:
Published
Year Published:
2024
Citation:
(White Paper) Steele, K., & Shi, Y. (April 2024). UNL Team Using Drones to Detect and Spray Phragmites. Weed Awareness newsletter. Nebraska Lancaster County Weed Authority.
- Type:
Conference Papers and Presentations
Status:
Accepted
Year Published:
2024
Citation:
TBA - Kevin's ASABE presentation
- Type:
Conference Papers and Presentations
Status:
Accepted
Year Published:
2024
Citation:
Shi, Y., Islam, M.D., Steele, K., Luck, J.D., Pitla, S., Ge., Y., Jhala, A., Knezevic, S., Riggan, B. & Zhang, K. (2024) Onboard weed identification and application test with spraying drone systems. 16th International Conference on Precision Agriculture, 21 - 24 July 2024, Manhattan, KS.
- Type:
Other
Status:
Other
Year Published:
2024
Citation:
Steele, K., & Shi, Y. (October 2023). University of Nebraska Drone Research. North American Invasive Species Management Association (NAISMA) 31st Annual Conference, Field Trip Presentation and Demo, Lincoln, Nebraska.
|
Progress 07/01/22 to 06/30/23
Outputs
Target Audience: Researchers and students in academia who are in the areas of agriculture, engineering, and technology learning or conducting research in precision and digital agricultural technologies, site-specific crop management, remote sensing, and autonomous systems. Precision and digital agriculture industry that develop technologies in areas of drones, aerial applications, and crop management. Public agencies that regulate agricultural aerial applications or agricultural and natural resources management such as the Nebraska Weed Association and Nebraska NRD Environmental Services. We do not directly work with growers in this project, but they are this study's ultimate target, and their needs drive the project direction. Changes/Problems:
Nothing Reported
What opportunities for training and professional development has the project provided?We have been creating these opportunities for training and professional development: Graduate and undergraduate student training - the PI and Co-PIs have been working with a few PhD, MS, and undergraduate students on this project either as their main thesis or dissertation projects, side projects, or undergraduate senior design projects. PI Shi has been working on integrating the project idea and findings in her digital agriculture course as a featured lecture on drone technology, hands-on experiential learning with drone demonstration, and drone data processing as project assignments. Students were very interested in the latest spraying and sensing drone technology, and we made them aware of the site-specific application concept, the drift issue, and the potential of the sense and spray technology. PI Shi has been working with the University of Nebraska Extension on a drone course open to the public and University of Nebraska students and staff. How have the results been disseminated to communities of interest?We delivered a couple of presentations and talks to various groups: Two scientific presentations by graduate students in the 2023 ASABE annual international meeting. Presentations and demonstrations about "University of Nebraska Drone Research" at the North American Invasive Species Management Association 31st Annual Conference, October 16-19, 2023. Lincoln, Nebraska. The audience was a group (40) of invasive species management leaders nationwide. What do you plan to do during the next reporting period to accomplish the goals? devise an edge-computing component integrated into a sensing UAS that maps the spatial pattern of weed pressure and generates rough prescription maps in real-time for time-sensitive applications; Improve the current weed identification models and test a couple more model structures. Select one or two for converting to real-time models. Convert the selected model currently running on desktop computers to the real-time model running on the companion computer that will be carried on the drone. develop a cloud-computing component for data storage that facilitates deriving more accurate prescription maps and long-term, user-contributed refinement of the decision-making model; Test the edge-cloud computing and federated learning algorithms using the data collected in field. Develop the edge to cloud data transmission framework that can display the map and the identified weed areas. customize and test a prototype of a site-specific UAAS that implements flight and application plans derived from the prescription map. A refined prototype sense and spray drone system with custom flight control system with the ability to do coordinate following, an integrated camera and real-time weed identification capability, and a spray module controlled by the onboard computer.
Impacts
What was accomplished under these goals?
devise an edge-computing component integrated into a sensing UAS that maps the spatial pattern of weed pressure and generates rough prescription maps in real-time for time-sensitive applications; In this funding period, we focused on collecting data of Palmer Amaranth in corn and soybean fields and developed an initial weed detection algorithm using deep learning techniques that can locate weed infestations using low-altitude drone imagery with acceptable application accuracy and speed. Two different encoder-decoder based semantic segmentation models, LinkNet and UNet, were trained with transfer learning techniques. We performed various measures such as backpropagation optimization and refining of the dataset used for training to address the class-imbalance problem which is a common issue in developing weed detection models. It was found that LinkNet model with ResNet18 as the encoder section and the use of the 'Focal loss' loss function was able to achieve the highest mean and class-wise Intersection over Union scores for different class categories while performing predictions on an unseen dataset. We currently achieved a mean IoU score of 0.801 and an IoU score for the weed class category of 0.691. While the initial model developed needs to improve, we simultaneously started working on developing the edge-computing component that will be integrated into the drone for real-time imaging and model implementation. The student has been working on getting familiar with the Jetson nano companion computer and we are able to run the initial weed identification models on it, although the inference time is slow which we plan to improve in the next period. develop a cloud-computing component for data storage that facilitates deriving more accurate prescription maps and long-term, user-contributed refinement of the decision-making model; This will be the focus for the next funding period. We have been utilizing the university cloud storage for the data that has been collected and processed data. At the same time, algorithms have been developed for model refinement based on federated learning. Co-PI Zhang's group has developed a joint federated learning framework to handle data that are vertically, horizontally, and non-identically distributed to save communication costs. customize and test a prototype of a site-specific UAAS that implements flight and application plans derived from the prescription map. We selected the DJI Matice series drones to start with in this project due to the availability and cost. With a lot of effort on the drone flight control system, we achieved a customized flight control utilizing the drone flight SDK, the Jetson system, and the ROS. Outdoor testing was conducted and after trials and errors the drone can now follow basic customized waypoints and speed control. We are working on flight control for coordinates following. At the same time, we tested the positioning accuracy of drones with different GPS systems. This is important for us to understand how precise and accurate a system can follow the desired flight path in the field. Details are reported in a MS thesis (Izere, P., 2023) and we found that the drone-based RTK system can achieve inch level positioning accuracy which is promising to use for precision spraying. We also started working on a prototype drone with a spraying system. We do not plan to start with modifying an expensive commercial sprayer drone for this but have been building our own prototype by integrating the spraying system with a big payload general purpose drone. We've already purchased DJI M600 drone. Calculations were made for the tank size, payload, and spraying time, which guided the drone model selection.
Publications
- Type:
Conference Papers and Presentations
Status:
Accepted
Year Published:
2023
Citation:
Islam, M.D., Izere, P., Singh, P., Yu, C., Riggan, B., Zhang, K., Luck, J., Jhala, A., Knezevic, S. & Shi, Y. (2023) Developing, deploying, and evaluating real-time weed detection models on an edge device. The American Society of Agricultural and Biological Engineers (ASABE), 2023 Annual meeting, 9 - 12 July 2021, Omaha, NE.
- Type:
Conference Papers and Presentations
Status:
Accepted
Year Published:
2023
Citation:
Yu, C., Meng, Z., Zhang, K. & Shi, Y. (2023) Efficient Multi-Layer Stochastic Gradient Descent Algorithm for Federated Learning in Cyber Physical Agriculture Systems. The American Society of Agricultural and Biological Engineers (ASABE), 2023 Annual meeting, 9 - 12 July 2021, Omaha, NE.
- Type:
Theses/Dissertations
Status:
Published
Year Published:
2022
Citation:
Singh, Puranjit, "Semantic Segmentation based deep learning approaches for weed detection" (2022). Biological Systems Engineering--Dissertations, Theses, and Student Research. 137. https://digitalcommons.unl.edu/biosysengdiss/137
- Type:
Theses/Dissertations
Status:
Published
Year Published:
2023
Citation:
Izere, Pascal, "Plant Height Estimation Using RTK-GNSS Enabled Unmanned Aerial Vehicle (UAV) Photogrammetry" (2023). Biological Systems Engineering--Dissertations, Theses, and Student Research. 145. https://digitalcommons.unl.edu/biosysengdiss/145
|
Progress 07/01/21 to 06/30/22
Outputs
Target Audience: Researchers and students (also including international students / Indian undergraduate visiting students). Growers and public agencies including NebraskaWeed Association andNebraska NRD environmental services. Lincoln public school teachers and students. General public and citizens. Changes/Problems: Personnel change One of our original Co-PI Dr. Psota left academia at the beginning of the project. Hence, we replaced his role with a new Co-PI Dr. Riggan who is in a very similar area. Dr. Riggan has been important and very helpful to the project. A worldwide shortage of embedded system electronics This project relies on onboard computer/embedded systems which is the NVIDIA Jetson system. With the current worldwide shortage of electronics/embedded system, weso far have not been able to purchase any Jetson system. The Jetson nano we reported earlier in the accomplishment section is actually not an authentic product but even that system tookus quite some effort to get. Itsarchitecture and performance are different than the authentic Jetson ones.If this situation continues, it would indeed havea negative impact on this project. What opportunities for training and professional development has the project provided?We have been collaborating with Nebraska Natural Resource Department (NRD)for a service program for weed mapping along streams in eastern Nebraska counties. We also participated in the professional development program of the Nebraska Weed Association. For example, a talk titled "Drone Technology for Weed Management" was delivered at Nebraska Weed Control Association Fall Conference/Training on November 3, 2021, in Kearney, Nebraska. This was one of the professional developments forall weed superintendents across the state. We have been working with the city of Lincoln Public School system to provide education and hands-on drone flight training. Two events had been held on December 9, 2021, at Northeast High School, Lincoln, Nebraska, and on April 28, 2022, at East Campus, University of Nebraska-Lincoln. How have the results been disseminated to communities of interest?Since we just finished year 1 of the project, there are not too many outcomes to deliver. However, as mentioned above, we have been participating in the professional development program of theNebraska Weed Association, and deliver the general drone applications and the current state-of-the-art for weed management. Other random events were the interactions and displayingof our drone systems and current research projects on the university's open-to-public events,and to a group of undergraduate visiting students from India. What do you plan to do during the next reporting period to accomplish the goals?Objective 1. Devise an edge-computing component integrated in a sensing UAS that maps the spatial pattern of weed pressure and generates rough prescription maps in a real-time for time-sensitive applications. Task 1.1. Use a sensing UAS to collect a rich set of imagery under various environmental conditions and crop/weed phenotypes for the development of decision-making model. We will continue collecting/adding new data in Year 2 to add tothe current dataset. We will focus on adding data that we find missing in the CNN model training. Also, we try to collect additional types of noxious weeds if possible. Task 1.2. Train and evaluate CNN-based decision-making models for frame-based real-time weed detection and mapping. In the next project year, we will be working on improving the model accuracy with more data, testing and tuning the model on different backgrounds and types of weeds, comparing it with other models in terms of accuracy and inference time, and simplifying the model for real-time applications. Task 1.3. Develop and test the edge-computing component integrated in the sensing UAS for real-time imaging and model implementation. This will be one of the major focuses in Year 2. Given the challenges we've met, we will spend more effort onfiguring out those bottleneck issues related with the onboard computer and improving the inference time and performance to meet the ultimate real-time purpose. Objective 2. Develop a cloud-computing component for data storage that facilitates deriving more accurate prescription maps and long-term, user-contributed refinement of the decision-making model. The theoretical analysis will be extended to prove the convergence with weaker assumptions which is different from the strong assumption used in the conference paper. After theoretical analysis, we aim to extend the experiments in the field based on the preliminary theoretical results and simulations. The implementation requires practical settings from device to edge nodes in the agriculture field. In Year 2 we will start testing the model with real datasets collected in Year 1 (Objective 1) tofurther improve the FL framework in model training. Objective 3. Customize and test a prototype of a site-specific UAAS that implements flight and application plans derived from the prescription map. We will identify a model for the controllable and customizable application/spraying drone, and develop an initial control system (e.g., based on PixHawk system) for it. If we have the capacity, we will also work on calibrating and fine-tuning flight control parameters for controlled and safe flights.
Impacts
What was accomplished under these goals?
Objective 1. Devise an edge-computing component integrated in a sensing UAS that maps the spatial pattern of weed pressure and generates rough prescription maps in a real-time for time-sensitive applications. Objective 1 was the focus in our Year 1's effort. Task 1.1. Use a sensing UAS to collect a rich set of imagery under various environmental conditions and crop/weed phenotypes for the development of decision-making model. Quite some efforts were made to collect a large set of weed imagery using a sensing UAS. This includes Palmer Amaranth in soybean and maize in experiment and commercial fields near Clay Center, NE, and leafy spurge (which is a common noxious weed species in pasture and rangelands) in Lincoln, NE. For each weed, data were collected along the season at different growth stages of the weed itself and different growth stages of the background crop or vegetation. As we stated in the proposal, we also intentionally collected the data under different environment illumination conditions, i.e., sunny, cloudy or overcast days. In addition, they were also collected at different altitudes, different flying modes (path following and hovering), and different modalities (primarily RGB but also some multispectral data). We will continue collecting/adding new data in the rest of the project time, but we believe we have created a fairly good dataset of low-altitude aerial imagery to start with, which is also valuable to later share with the research community. Task 1.2. Train and evaluate CNN-based decision-making models for frame-based real-time weed detection and mapping. We have developed an initial version of CNN-based decision-making model for frame-based weed detection and mapping. Eventually it will be a real time model, however, we first wanted to develop a model that is accurate enough before we simplify that model or replace it with other lighter models for a real time application. We have identified U-Net model as the starting segmentation modelto have boundaries of the weed patches more precise delineated. We started training the model using images collected over palmer in mid-growth stage maize which were the first set of data we collected. Several different pre-trained architectures were evaluated and'ResNet34' performed best taking into consideration of the number of parameters and execution time. Several loss functions were used to optimize the model and so far that 'Focal loss' performed best out of all. Precision, Recall, F1 score, Intersection over union (IoU), and loss valueswere obtained.The current U-Net modelwas tested on some images which the model was not trained with, and the preliminary results showed that the current model performs well. In the next project year, we will be working on improving the model accuracy with more data, testing and tuning the model on different background and weeds, comparing it with other models in terms of accuracy and inference time, and simplifying the model for the real-time applications. Task 1.3. Develop and test the edge-computing component integrated in the sensing UAS for real-time imaging and model implementation. This project involves two UAVs (drones), a sensing drone and an application/spraying drone. Each of thedrones will have an edge-computing component (onboard/companion computer) integratedwith them for real time decision makings: (i) The task of the edge-computingcomponent integrated with the sensing drone is to collect in-field images, recognize the weed patches and map their locations, andgenerate aroughprescription map, so that larger area can be covered and management decisions can be made with less time delay. The derived prescription map will be transferred to the spraying drone. (ii) The task of the edge-computing component integrated with the spraying drone is to control the spraying drone follow the customize flight path to each weed patch at a much lower flying altitude, perform the real-time more precise and accurate weed detection, derive the precise locations of the weed patches, then perform spraying. It is known that the GPS onboard of the drone has certain positioning error even with the state-of-the-art RTK GPS. Hence, the real-time weed detection on the spraying drone is critical. The CNN-based weed detection model developed in Task 2 is not directly deployable in edge-computingcomponents due to the memorylimitation of the onboard/companion computer. So, the original U-Net model was converted to a lighter version(tensorflowlite format). We haveselected the Jetson nano (4 GB)developer kit as the edge-computing component (we wanted to have more powerful model but there has been a severe out of stock for Jetson products). However, even with the lighter model, the inference time on the Jetson nano remainslonger than what we need. Hence, wehave been trying to figure out the bottleneck issues and improve the inference time to meet the near real time purpose. It is noteworthy to mention that, afterconverting the original model to a lighter version,the performance is not same as the original model. So we also need to figure out that issue in the next phase of the project. Meanwhile, we have identified and been working on an open architecture drone based on PixHawk control. This is very challenging too and we have met issues on the drone flight control and calibration so far. Objective 2. Develop a cloud-computing component for data storage that facilitates deriving more accurate prescription maps and long-term, user-contributed refinement of the decision-making model. Task 2.1. Establish a cloud-edge assisted heterogeneous network for optimal learning. As planned, we started working on this task half way in Year 1. The project team designs a novel computing framework between edge and cloud server for UAV data collection in the field. We observe that the field data are cooperatively collected by sensor devices and UAV, i.e., vertically distributed data. The data on various sensor devices share the same feature set but are different in sample spaces, i.e., horizontally partitioned data. Meanwhile, hospitals target various field types/groups resulting in high data diversity, i.e., non-identically distributed data. These three characteristics cause that existing federated learning frameworks cannot efficiently train models on agriculture data. We address the problem of how to efficiently and rapidly train global models on agriculture field data. Specifically, we propose a multi-layer federated learning framework to cope with data that are vertically, horizontally, and non-identically distributed. Moreover, we design a Multi-Layer Stochastic Gradient Descent (MLSGD) algorithm towards the proposed framework to learn the optimal global model. To improve training efficiency, partial models learned by devices are aggregated on edge nodes before exchanging intermediate results with cloud server. The weight of local models is proportional to local data size when performing global aggregation to balance the impact of local models on the global model. We also prove the convergence of the MLSGD algorithm from a theoretical perspective. We conduct simulations by using public dataset in e-health to validate that the proposed algorithm converges fast and achieves desired accuracy. Task 2.2. Develop model refinement algorithms on cloud with users' contribution. This taskwas planned to start in Year 2. Objective 3. Customize and test a prototype of a site-specific UAAS that implements flight and application plans derived from the prescription map. This objective is related to the application/spraying drone and we wouldnot start working on this objective until the latter half of Year 2 as planned. However, we have been mindful of this objective when we are working on the previous objectives.
Publications
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2022
Citation:
Yu, C., Shen, S., Zhang, K., Zhao, H., & Shi, Y. (2022, April). Energy-Aware Device Scheduling for Joint Federated Learning in Edge-assisted Internet of Agriculture Things. In�2022 IEEE Wireless Communications and Networking Conference (WCNC)�(pp. 1140-1145). IEEE.
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2022
Citation:
Izere, P., Zhao, B., Ge, Y., & Shi, Y. (2022, June). Estimation of plant height using UAS with RTK GNSS technology. In Autonomous Air and Ground Sensing Systems for Agricultural Optimization and Phenotyping VII (Vol. 12114, pp. 195-205). SPIE.
- Type:
Conference Papers and Presentations
Status:
Accepted
Year Published:
2022
Citation:
Singh, P., Izere, P., Nikahal, K., Riggan, B., Jhala, A., Knezevic, S., Zhang, K., Luck, J., & Shi, Y. (2022, July) Weed detection from UAV imagery using UNet. 2022 ASABE Annual International Meeting, in Houston, TX.
|
|