Progress 02/01/24 to 01/31/25
Outputs
Target Audience:The main target audiences for our project include researchers, plant breeders, industry stakeholders, and growers specializing in row crops such as flaxseed, canola, tomato, and strawberry. We intend to make the design and methodologies openly accessible in the public domain. After thorough testing of the technology in research fields, our goal is to facilitate the transfer of this technology to growers' fields for on-farm demonstrations. The aim is to encourage growers to adopt and benefit from the innovative technology in their crop cultivation practices. Changes/Problems:In 2024, field experiments were conducted at two locations: the NDSU campus and NW22 in Fargo, ND. Significant weed pressure was observed in the NDSU campus plots which is an ideal condition for this study, whereas the NW22 site had minimal weed presence. As a result, we decided to exclude data from the NW22 site from our analysis. This variability is common in agricultural field experiments, which is why multiple locations and years are selected to ensure repeatable and reliable results. What opportunities for training and professional development has the project provided?Afrina Rahman a graduate student at North Dakota State University (NDSU), was employed on this project since May 2021 to March 2023. Later Md. Fahad Hasan, a graduate student at NDSU joined this project and continue the research. In 2024, he conducted field experiments at two locations in North Dakota using the protocol described in the project methodology. He is also working in the greenhouse and lab for characterizing nanomaterials and testing those in the greenhouse. Currently, he is working on the data generated in 2024. Raj Shanker Hazra and Ms. Tanha Tabassum were former graduate students at North Dakota State University and partially worked for this purpose. Hazra defended his graduate thesis in 2023. The graduate students worked on two major aspects of the proposal - (A) Determining the composition of nanoformulations, and (B) Understanding the physico-chemical behavior of polymeric surfactants that will be used in the final, robot-dispensable formulations. Andrew Choi, a PhD student who was partially funded by this project. He obtained his PhD after defending his thesis titled "Simulation of Deformable Objects for Sim2Real Applications in Robotics" in 2023. He is currently a research scientist at Horizon Robotics. Shivam Panda and Yongkyu Lee conducted the field experiments in Fargo during the summer of 2022 and 2023, working closely with our collaborators, PI Rahman in Plant Science and his students Afrina Rahman, Md. Golam Robbani, Md Jony and Fahad Hassan. The performance and outcome of robotic weed control are maintained by PI Rahman's group. The image data collected through this project also allowed for work on autonomous navigation. This work is currently in progress and is expected to be presented at one of the top robotics conferences. The graduate students (Shivam Kumar Panda and Yongkyu Lee) contributed to preparing this project report. Undergraduate students: UCLA has a research program called the "Undergraduate Research Center" (http://sciences.ugresearch.ucla.edu/courses/srp/) for undergraduate students that enables qualified students to work on research projects in exchange for academic credit. Opportunities for some of the undergraduate students were provided through this program. (1) Howard Zhu: Howard has started the project in January 2023. He has participated in research through the REU program. His main contributions were developing the software for the new version of the robot. He also participated in field experiments at Fargo, North Dakota during the Summer of 2023. (2) Patrick Dai: Patrick has joined the project in Summer 2023. He has worked on integrating the RTK-GNSS system into the new version of the robot and is currently working on the implementation of our proposed framework on a real physical system. He is participating in the REU program. (3) Mehul Jain: Mehul has joined the project in Fall 2023. He is mostly involved with the analysis of data collected from the robot. He is currently working on developing algorithms to calculate stem diameter from stereo-pair images. (4) Dylan Santora: Dylan started working on the project with Shivam and Yongkyu in Fall 2022, under the Research Experience for Undergraduates (REU) program of the National Science Foundation. His stipend is not included in the USDA budget. He has mainly worked on the CAD design of the new version of the robot and graduated in the Summer of 2023. He is now a Masters student at UCLA. (5) Manya Agrawal: Manya contributed to the project in the Fall of 2022 under the SRP 99 (Student Research Program) course at UCLA. She has been tasked with implementing the push-push mechanism for the charging station to enable safe and easy contact. Her work for the next quarter will focus on designing a PCB to test out charging through this new mechanism. (3) Other students: The project opened new opportunities for four other students (Nathan Gong, Nemi Desai, Oliver Simpson) starting from January 2023. All students have participated in the SRP 99 program. Nathan and Nemi are Computer Science majors who are responsible for developing the software for different modules of the robot. Oliver is a Mechanical Engineering student who focuses on design and hardware. How have the results been disseminated to communities of interest?The results have been disseminated via publication in peer-reviewed journal (ACS Appl Bio Mater. 2023 Jul 17;6(7):2698-2711. doi: 10.1021/acsabm.3c00171. Epub 2023 Jul 5. PMID: 37405899). One journal article has already been published, and the manuscript is under preparation for the second manuscript. A cover-page was also received for this manuscript. A webpage (https://structures.computer/weed-management) has been created under Prof. Jawed's lab website to record and share the findings on this research project. Latest videos and news are shared through this webpage. It will be continually updated to reflect current progress. Two articles have been published in International Conference on Robotics and Automation (ICRA), which is one of the top conferences in Robotics, and IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. (1) Du, Yayun, Guofeng Zhang, Darren Tsang, and M. Khalid Jawed. "Deep-CNN based Robotic Multi-Class Under-Canopy Weed Control in Precision Farming." International Conference on Robotics and Automation (ICRA) (2022) https://doi.org/10.1109/ICRA46639.2022.9812240 (2) Du, Y., Saha, S.S., Sandha, S.S., Lovekin, A., Wu, J., Siddharth, S., Chowdhary, M., Jawed, M.K., Srivastava, M., "Neural-Kalman GNSS/INS Navigation for Precision Agriculture" in press with IEEE International Conference on Robotics and Automation (ICRA) (2023) A video description of our work on Neural-Kalman GNSS/INS Navigation for Precision Agriculture is available on youtube(https://youtu.be/9e3Q_9aTCQ4) The dataset and code for this work is publicly available on Github (https://github.com/nesl/agrobot) The ICRA presentation on "Deep-CNN based Robotic Multi-Class Under-Canopy Weed Control in Precision Farming" has been published on Youtube (https://www.youtube.com/watch?v=mLXtLS94m38). (3) Shivam Kumar Panda, Yongkyu Lee, Mohammad Khalid Jawsed, "Agronav: Autonomous Navigation Framework for Agricultural Robots and Vehicles Using Semantic Segmentation and Semantic Line Detection." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (pp. 6271-6280) (2023). The dataset and code for this work is publicly available on Github (https://github.com/StructuresComp/agronav/). An educational YOUTUBE video (https://www.youtube.com/watch?v=v0DWQGlzyKQ And https://www.youtube.com/watch?v=7B1g0MbuKtc) of efficiently used drones and robots for crop phenotyping and weed control was created by Dr. Rahman's lab. The dataset and code for this work are publicly available on GitHub (https://github.com/StructuresComp/agronav/). What do you plan to do during the next reporting period to accomplish the goals? We have generated sufficient data from field experiments and will analyze all data from 2021 to 2024 for the final report. We will also continue in writing a manuscript from this project to publish in a peer-reviewed journal. Additionally, my graduate student will continue studying the efficiency and deposition of nano-herbicides on weeds and crops under lab and greenhouse conditions. If necessary, we will extend the experiment to field conditions Analysis of residence time of modified herbicides on the surface of various herbs. Last years, we have optimized the formulation of polyelectrolytes that can increase residence time of aqueous solution on weed leaves. Polyelectrolyte solutions will be sprayed on the plants in the presence or absence of commercial herbicide preparation. The growth profile of weeds and their survival will be evaluated following the treatment. The effect of polyelectrolyte nanoparticle-assisted application of commercial herbicide will be tested using contact angle, TEM and brightfield microscopic measurements. Data generated from the greenhouse studies will be analyzed for correlating weed type, herbicide efficacy, and nanoparticle deposition. Any potential side-effects of nanoparticle treatment will also be evaluated to find out the exact dosage range. Histological evaluations of weeds will be carried out to identify the possible translocation of polyelectrolyte complexes inside the plant. Based on the data generated in 7-8, evaluation of the efficiency of herbicides in the greenhouse and field, with or without the assistance of the robot. Enhancements and Field Testing of the Upgraded Robot: Responding to last year's feedback, we are progressing with enhancements to our larger robot. The updated design, already finalized in CAD, includes new wheels and motors that have been recently acquired. We are poised to reassemble the robot with these improvements, targeting field trials in the summer of 2024. In a move towards open innovation, we plan to share the design and specifications of our robot with interested communities, making them publicly accessible. Development of an overall navigation algorithm framework: Our robot's navigation system is being refined to incorporate both local and global navigation capabilities. The local navigation is powered by Agronav, which processes live imagery to identify traversable areas and determine a centerline for movement. These navigable paths are integrated into our visual SLAM system. Concurrently, a global direction is provided by the RTK (Real-Time Kinematic) system. The robot's action module will synthesize inputs from both the global RTK direction and the local Agronav centerline to generate precise movement commands. Essentially, the perception system will guide the robot during intra-row navigation, while the RTK system will take precedence for inter-row navigation. Coordinates are continuously updated in both the RTK system and SLAM, ensuring seamless operation throughout the robot's activity. This dual-system approach promises to enhance the accuracy and efficiency of the robot's navigational capabilities. The charging station: We are in process of developing a novel charging station for mobile robots that significantly reduces the time required for charging batteries from the overall operation time. The major contribution of this charging station is its autonomous interface for the exchange of batteries. This allows mobile robots to continue their operations while their batteries are being charged, thus increasing the efficiency and productivity of the entire system. The charging station is designed with an advanced battery exchange mechanism that enables quick and seamless battery swap-out, reducing the downtime of the mobile robots. This innovative charging station is expected to be a game-changer for industries that rely on mobile robots, such as logistics, warehousing, and agriculture, by increasing their operational efficiency and reducing downtime.
Impacts
What was accomplished under these goals?
TASK I: Two field experiments were conducted during the summer of 2024 at the NDSU campus in Fargo, and the NW22 site in Fargo, ND, utilizing a randomized complete block design with four replications. The herbicides included Basagran®, MCPA (2-methyl-4-chlorophenoxyacetic acid), and Select Max®. The application rates were 0.75 pint, 0.5 pint, and 1.0 pint, respectively, mixed with 10 gallons of water for a rate of one acre of land. The study involved a total of thirteen treatments such as (1) no weed control, (2) weed control by hand, (3) using traditional equipment by mixed-herbicides (3rd leaf and bud stage), (4) using robot simulation by mixed-herbicides (non-NANO) (3rd leaf and bud stage), (5) using robot simulation by mixed-herbicides (non-NANO) (spray every 15 days), (6) using robot simulation by NANO-Organic-mixed-herbicides (3rd leaf and bud stage), (7) using robot simulation by NANO-mixed-Organic-herbicides (spray every 15 days), (8) using robot simulation by NANO-Synthetic-mixed-herbicides (3rd leaf and bud stage), (9) using robot simulation by NANO-mixed-Synthetic-herbicides (spray every 15 days), (10) using robot simulation by NANO-Duel-Organic mixed herbicides (3rd leaf and bud stage), (11) using robot simulation by NANO-Duel-Organic mixed herbicides (spray every 15 days), (12) using robot simulation by NANO-Duel-Synthetic mixed herbicides (3rd leaf and bud stage), and (13) using robot simulation by NANO-Duel-Synthetic mixed herbicides (spray every 15 days). Among the 13 treatments, treatment #2 (hand weeding) exhibited the significantly highest seed yield (868 lb/a) of flax, while treatment #1 (no weeding) showed the lowest yield (320 lb/a). No significant differences were observed among the other treatments. Various types and combinations of nano-materials did not have a significant impact on seed yield. However, the percentage of weed suppression significantly increased with all treatments over the initial weed count at 25 days to final weed count at 71 days. In control, weed pressure was increased by 180%. We have observed a significant crop injury by application of synthetic nano-herbicides over organic nano-herbicides or traditional herbicides. This is a message that synthetic nano-herbicides are not a good option for flax. The percentage of weed suppression significantly improved with all treatments from the initial weed count at 25 days to the final count at 71 days. In contrast, weed pressure in the control group increased by 180%. It is interesting to note that the application of synthetic nano-herbicides caused considerable crop injury compared to organic nano-herbicides and traditional herbicides. These findings indicate that synthetic nano-herbicides are not a suitable option for flax cultivation. However, herbicides containing synthetic nano-materials demonstrated superior weed suppression compared to those with organic nano-materials. TASK II: This year marks a significant milestone in our project with the design, prototyping, and testing of a completely new robot. Designed for increased operational efficiency, this robot can cover 4-5 flax rows simultaneously. It features advanced edge computing capabilities and provides more space for instruments, efficient sprayers, and herbicides. Size: Based on the inputs from the previous field trials the aim was to increase efficiency and spray for longer distances and larger area. Hence we designed a larger robot of size 75 in (L) x 63 in (W) x 40 in (H). Here the width is adjustable, so it can be varied from 58 in to 70 in based on different sizes of the plot. Onboard electronics: The robot's main controller has been upgraded to the Jetson Orin, an edge AI computing platform with a processing capacity of 70 TOPS (Tera Operations per Second). ESP32s are being used to control the motors/motor drivers, the pump & valves, and communicate with the RC Controller. Navigation algorithm: We have developed a proprietary vision-based navigation algorithm named Agronav. It utilizes transformer-based semantic segmentation to differentiate traversable and non-traversable regions in flax rows. Additionally, the Deep-Hough transform is employed for lane detection within rows. The implementation of Agronav on the robot is ongoing. So the new iteration of the robot designed this year had the following specifications: Size: 6.25 ft long, 4.4 feet wide, and 3.3 feet tall. Payload: 100 gallons of high-efficiency herbicide. Speed: Top speed is 15 cm/sec. Range: 5 miles on a single charge Onboard intelligence: The partial autonomous robot drives on vision-based information for inter-row navigation. The weed detection algorithm is the same as before. Field trials were conducted in three phases across two flax fields in North Dakota. Analysis of these trials led to several key decisions for enhancing the larger robot's performance. The need for higher torque motors was identified to ensure smooth traversal over uneven terrain. Additionally, it was determined that larger wheels with increased grip are essential for the driving mechanism. These upgrades are planned for implementation within the year, setting the stage for subsequent retrials in the field. This approach underscores our commitment to continuous improvement based on practical field experience, aiming to optimize the robot's functionality in real-world agricultural settings. The paper for the vision-based navigation system has been published. The main contributions of this work include presenting a simple navigation framework based on a single monocular camera. The task of detecting the centerline, which will potentially serve as a reference trajectory the robot should follow has been accomplished through two downstream tasks. Semantic segmentation provides a pixel-level annotation of the key features of the image. A transfer-learning based approach was used to train this model. Semantic line detection outputs a pair of boundary lines between the traversable ground and crops. The centerline is obtained from this pair of boundary lines. In addition, the key contribution of this work is the creation of the largest open-source annotated dataset, "AgroNav," specifically designed for agricultural navigation tasks for agricultural robots, vehicles, and drones, which will serve as a valuable resource for developing advanced agricultural navigation systems. Currently the team is involved with the implementation of the framework proposed in this work on a mobile robot, which is minimally equipped with an on-board computer (Jetson Xavier) and a monocular camera. TASK III: One of the graduate students currently supported from this project, Mr. Fahad Hasan, has developed polyelectrolyte-type nanoparticles, which can increase the adhesion of commercially-available fungicide solution on the leaf surface. Two types of polyelectrolyte complexes have been derived - One based on biobased polyelectrolytes, such as chitosan and alginic acid. The other types of polyelectrolytes were of chemical origin, namely, poly (ethylene imine) and poly(acrylic acid). Important to mention that in the following years we will focus more on biobased polyelectrolyte complex. This is because our data show that synthetic polyelectrolyte complexes produce off-target toxicity to other plants. The microstructure of different types of herbs have been investigated using Transmission Electron Microscopy (TEM). Afterwards, the surface roughness of these leaves has been correlated with their water-repellency properties. This is an important correlation because the more hydrophobic the herbs are, the more difficult it is to suppress their growth via foliar spray of fungicides. Eventually, we showed that, incorporation of polyelectrolyte complexes of chitosan/alginate can decrease the contact angle of water on these leaf surfaces, which will provide higher residence time of fungicide formulation on the leaf surface favoring their absorption.
Publications
|
Progress 02/01/23 to 01/31/24
Outputs
Target Audience:The main target audiences include researchers, plant breeders, industry stakeholders (if applicable), and growers of row crops such as flaxseed, canola, tomato, and strawberry. We will release the design and methodologies to the public domain. Following successful testing in research fields, the technology may be implemented in growers' fields for on-farm demonstrations. AmeriFlax, (which represents the North Dakota Flax Producers Association) will collaborate to assess the technology in farmers' fields, aiming to promote its adoption among growers. Changes/Problems:In 2023, field experiments were carried out at two locations: the NDSU campus and NW22 in Fargo, ND. Substantial weed pressure was noticed at the NDSU campus experiment plots, while no significant weed pressure was observed at the NW22 site. Consequently, we decided to exclude the NW22 site in Fargo, ND, from data collection. Unexpected situations like this are common in field research programs, and that is why we conduct experiments at multiple locations to mitigate any unforeseen challenges that may arise during the experiments. We might experience agglomeration of commercial herbicides when engineered with polymeric surfactants. Our earlier studies showed indications that such agglomeration might take place. To counter this problem, we will use a modification of the addition processes of the polymeric surfactants to the herbicide solution. In this modified process, polymeric surfactants (additives) will be added to the commercial herbicides using an overhead stirrer or ultrasonication. This will form fine, stabilized droplets of additives within the herbicide formulation to augment its various properties, such as leaf adhesion. What opportunities for training and professional development has the project provided?Graduate students: Fahad Hasan, a graduate student at North Dakota State University, employed on this project since May 2021. She has been conducting experiments in the greenhouse and in the field to evaluate the performance of small robots to spray herbicides and weed killing. In 2022, she conducted field experiments at three locations in North Dakota using the protocol described in the project methodology. Currently, she is working on the data generated in 2023. Raj Shanker Hazra and Ms. Tanha Tabassum were former graduate students at North Dakota State University and partially worked for this purpose. Hazra defended his graduate thesis in 2023. The graduate students worked on two major aspects of the proposal - (A) Determining the composition of nanoformulations, and (B) Understanding the physico-chemical behavior of polymeric surfactants that will be used in the final, robot-dispensable formulations. Andrew Choi, a PhD student who was partially funded by this project. He obtained his PhD after defending his thesis titled "Simulation of Deformable Objects for Sim2Real Applications in Robotics" in 2023. He is currently a research scientist at Horizon Robotics. Shivam Panda and Yongkyu Lee conducted the field experiments in Fargo during the summer of 2022 and 2023, working closely with our collaborators, PI Rahman in Plant Science and his students Afrina Rahman, Md. Golam Robbani, Md Jony and Fahad Hassan. The performance and outcome of robotic weed control are maintained by PI Rahman's group. The image data collected through this project also allowed for work on autonomous navigation. This work is currently in progress and is expected to be presented at one of the top robotics conferences. The graduate students (Shivam Kumar Panda and Yongkyu Lee) contributed to preparing this project report. Undergraduate students: UCLA has a research program called the "Undergraduate Research Center" (http://sciences.ugresearch.ucla.edu/courses/srp/) for undergraduate students that enables qualified students to work on research projects in exchange for academic credit. Opportunities for some of the undergraduate students were provided through this program. (1) Howard Zhu: Howard has started the project in January 2023. He has participated in research through the REU program. His main contributions were developing the software for the new version of the robot. He also participated in field experiments at Fargo, North Dakota during the Summer of 2023. (2) Patrick Dai: Patrick has joined the project in Summer 2023. He has worked on integrating the RTK-GNSS system into the new version of the robot and is currently working on the implementation of our proposed framework on a real physical system. He is participating in the REU program. (3) Mehul Jain: Mehul has joined the project in Fall 2023. He is mostly involved with the analysis of data collected from the robot. He is currently working on developing algorithms to calculate stem diameter from stereo-pair images. (4) Dylan Santora: Dylan started working on the project with Shivam and Yongkyu in Fall 2022, under the Research Experience for Undergraduates (REU) program of the National Science Foundation. His stipend is not included in the USDA budget. He has mainly worked on the CAD design of the new version of the robot and graduated in the Summer of 2023. He is now a Masters student at UCLA. (5) Manya Agrawal: Manya contributed to the project in the Fall of 2022 under the SRP 99 (Student Research Program) course at UCLA. She has been tasked with implementing the push-push mechanism for the charging station to enable safe and easy contact. Her work for the next quarter will focus on designing a PCB to test out charging through this new mechanism. (3) Other students: The project opened new opportunities for four other students (Nathan Gong, Nemi Desai, Oliver Simpson) starting from January 2023. All students have participated in the SRP 99 program. Nathan and Nemi are Computer Science majors who are responsible for developing the software for different modules of the robot. Oliver is a Mechanical Engineering student who focuses on design and hardware. How have the results been disseminated to communities of interest?The results have been disseminated via publication in a peer-reviewed journal (ACS Appl Bio Mater. 2023 Jul 17;6(7):2698-2711. doi: 10.1021/acsabm.3c00171. Epub 2023 Jul 5. PMID: 37405899). One journal article has already been published, and the manuscript is under preparation for the second manuscript. A cover page was also received for this manuscript which is displayed below: https://www.youtube.com/watch?v=v0DWQGlzyKQ https://www.youtube.com/watch?v=7B1g0MbuKtchttps://structures.computer/weed-management IEEE International Conference on Robotics and Automation (ICRA) (2023) https://youtu.be/9e3Q_9aTCQ4https://github.com/nesl/agrobothttps://www.youtube.com/watch?v=mLXtLS94m38https://github.com/StructuresComp/agronav/). What do you plan to do during the next reporting period to accomplish the goals? The field experiments will be conducted again in the three locations in North Dakota. We will continue with the new thirteen treatments for in-field weed control and will take the data on weeds and flax crops. Finally, we will harvest the crop and weeds from each plot. Next year, we will use the newly developed plot-size robot developed in Dr. Jawad's lab in the field. Enhancements and Field Testing of the Upgraded Robot: Responding to last year's feedback, we are progressing with enhancements to our larger robot. The updated design, already finalized in CAD, includes new wheels and motors that have been recently acquired. We are poised to reassemble the robot with these improvements, targeting field trials in the summer of 2024. In a move towards open innovation, we plan to share the design and specifications of our robot with interested communities, making them publicly accessible. Development of an overall navigation algorithm framework: Our robot's navigation system is being refined to incorporate both local and global navigation capabilities. The local navigation is powered by Agronav, which processes live imagery to identify traversable areas and determine a centerline for movement. These navigable paths are integrated into our visual SLAM system. Concurrently, a global direction is provided by the RTK (Real-Time Kinematic) system. The robot's action module will synthesize inputs from both the global RTK direction and the local Agronav centerline to generate precise movement commands. Essentially, the perception system will guide the robot during intra-row navigation, while the RTK system will take precedence for inter-row navigation. Coordinates are continuously updated in both the RTK system and SLAM, ensuring seamless operation throughout the robot's activity. This dual-system approach promises to enhance the accuracy and efficiency of the robot's navigational capabilities. The charging station: We are in the process of developing a novel charging station for mobile robots that significantly reduces the time required for charging batteries from the overall operation time. The major contribution of this charging station is its autonomous interface for the exchange of batteries. This allows mobile robots to continue their operations while their batteries are being charged, thus increasing the efficiency and productivity of the entire system. The charging station is designed with an advanced battery exchange mechanism that enables quick and seamless battery swap-out, reducing the downtime of the mobile robots. This innovative charging station is expected to be a game-changer for industries that rely on mobile robots, such as logistics, warehousing, and agriculture, by increasing their operational efficiency and reducing downtime. Analysis of residence time of modified herbicides on the surface of various herbs. The analysis will be conducted using microscopy and surface measurements. Analysis of the efficiency of herbicides in the greenhouse and field, with or without the assistance of the robot.
Impacts
What was accomplished under these goals?
TASK I: Two field experiments were conducted during the summer of 2023 at the NDSU campus in Fargo, and the NW22 site in Fargo, ND, utilizing a randomized complete block design with four replications. The herbicides included Basagran®, MCPA (2-methyl-4-chlorophenoxyacetic acid), and Select Max®. The application rates were 0.75 pint, 0.5 pint, and 1.0 pint, respectively, mixed with 10 gallons of water for a rate of one acre of land. The study involved a total of thirteen treatments such as (1) natural plot i.e. no weed control, (2) weed free plot i.e. all the weeds were cleaned by hand, (3) weed control using traditional equipment by mixed-herbicides (spray at 3rd leaf and bud stage), (4) weed control using robot simulation by mixed-herbicides (non-NANO) (spray at 3rd leaf and bud stage), (5) weed control using robot simulation by mixed-herbicides (non-NANO) (spray every 15 days), (6) weed control using robot simulation by NANO-Organic-mixed-herbicides (spray at 3rd leaf and bud stage), (7) weed control using robot simulation by NANO-mixed-Organic-herbicides (spray every 15 days), (8) weed control using robot simulation by NANO-Synthetic-mixed-herbicides (spray at 3rd leaf and bud stage), (9) weed control using robot simulation by NANO-mixed-Synthetic-herbicides (spray every 15 days), (10) weed control using robot simulation by NANO-Duel-Organic (+ve & -ve separately) mixed herbicides (spray at 3rd leaf and bud stage), (11) weed control using robot simulation by NANO-Duel-Organic (+ve & -ve separately) mixed herbicides (spray every 15 days), (12) weed control using robot simulation by NANO-Duel-Synthetic (+ve & -ve separately) mixed herbicides (spray at 3rd leaf and bud stage), and (13) weed control using robot simulation by NANO-Duel-Synthetic (+ve & -ve separately) mixed herbicides (spray every 15 days). Substantial weed pressure was noted in the experimental plots at the NDSU campus, Fargo. The small plot robot faced challenges navigating through plots with high weed pressure, affecting efficient in-plot weed control. Consequently, we decided to use a backpack sprayer to simulate the application of robotic herbicides. Among the 13 treatments, treatment #2 exhibited the significantly highest seed yield of flax, while treatment #1 showed the lowest yield. No significant differences were observed among the other treatments. Various types and combinations of nano-materials did not have a significant impact on seed yield. However, weed suppression significantly increased with simulated robotic herbicide spray compared to traditional weed control, aligning with our primary objective of weed control using robots. This year, we incorporated two types of nano-materials: organic and synthetic. The herbicides with synthetic nano-materials exhibited a significantly higher level of crop injury compared to those based on organic nano-materials. However, herbicides containing synthetic nano-materials demonstrated superior weed suppression compared to those with organic nano-materials. TASK II: This year marks a significant milestone in our project with the design, prototyping, and testing of a completely new robot. Designed for increased operational efficiency, this robot can cover 4-5 flax rows simultaneously. It features advanced edge computing capabilities and provides more space for instruments, efficient sprayers, and herbicides. (a) Size: Based on the inputs from the previous field trials the aim was to increase efficiency and spray for longer distances and larger areas. Hence, we designed a larger robot of size 75 in (L) x 63 in (W) x 40 in (H). Here the width is adjustable, so it can be varied from 58 in to 70 in based on different sizes of the plot. (b) Onboard electronics: The robot's main controller has been upgraded to the Jetson Orin, an edge AI computing platform with a processing capacity of 70 TOPS (Tera Operations per Second). ESP32s are being used to control the motors/motor drivers, and the pump & valves, and communicate with the RC Controller. (c) Navigation algorithm: We have developed a proprietary vision-based navigation algorithm named AgroNav. It utilizes transformer-based semantic segmentation to differentiate traversable and non-traversable regions in flax rows. Additionally, the Deep-Hough transform is employed for lane detection within rows. The implementation of AgroNav on the robot is ongoing. Field trials were conducted in three phases across two flax fields in North Dakota. Analysis of these trials led to several key decisions for enhancing the larger robot's performance. The need for higher torque motors was identified to ensure smooth traversal over uneven terrain. Additionally, it was determined that larger wheels with increased grip are essential for the driving mechanism. These upgrades are planned for implementation within the year, setting the stage for subsequent retrials in the field. This approach underscores our commitment to continuous improvement based on practical field experience, aiming to optimize the robot's functionality in real-world agricultural settings. The paper for the vision-based navigation system has been published. The main contributions of this work include presenting a simple navigation framework based on a single monocular camera. The task of detecting the centerline, which will potentially serve as a reference trajectory the robot should follow has been accomplished through two downstream tasks. Semantic segmentation provides a pixel-level annotation of the key features of the image. A transfer-learning-based approach was used to train this model. Semantic line detection outputs a pair of boundary lines between the traversable ground and crops. The centerline is obtained from this pair of boundary lines. In addition, the key contribution of this work is the creation of the largest open-source annotated dataset, "AgroNav," specifically designed for agricultural navigation tasks for agricultural robots, vehicles, and drones, which will serve as a valuable resource for developing advanced agricultural navigation systems. Currently, the team is involved with the implementation of the framework proposed in this work on a mobile robot, which is minimally equipped with an onboard computer (Jetson Xavier) and a monocular camera. TASK III: Over the last year, we extensively studied the surface-active properties of the additives that we will incorporate in robot-assisted spray formulation for increased herbicidal action. The hydrodynamic diameter (Z-average size) and zeta potential of the polyelectrolytes were determined using dynamic light scattering (DLS). Additive solutions of sodium alginate and chitosan were prepared in water. Mixing these polymeric surfactants resulted in the formation of nanoscale aggregates with zeta potential (surface charge) ranging from +60 to -60 mV. The formation of such nanoscale aggregates is important since these colloidal aggregates will provide nanoscale defects on the otherwise hydrophobic leaf surface. These defects will promote the adhesion of herbicide droplets (in water) on the leaf surface. We analyzed the surface of different types of weeds that are grown in flax fields in ND. This was achieved by measuring the contact angle of water on the leaf surfaces of these weeds. We observed that the contact angle of water varied dramatically over different leaves. Therefore, we also observed significant variation in adhesion energy of different commercially available herbicide formulations, indicating why some herbicides do not show adequate effects against selective weeds. The observation was also validated by scanning electron microscopy. This data collectively formed the basis of our next step of the experiment, where we will measure how the incorporation of additives will augment the residence time of robot-assisted herbicide formulation on herb surfaces.
Publications
- Type:
Journal Articles
Status:
Published
Year Published:
2023
Citation:
Hazra RS, Roy J, Jiang L, Webster DC, Rahman MM, Quadir M. Biobased, Macro-, and Nanoscale Fungicide Delivery Approaches for Plant Fungi Control. ACS Appl Bio Mater. 2023 Jul 17;6(7):2698-2711. doi: 10.1021/acsabm.3c00171. Epub 2023 Jul 5. PMID: 37405899.
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2022
Citation:
Du, Yayun, et al. "Deep-cnn based robotic multi-class under-canopy weed control in precision farming." 2022 International Conference on Robotics and Automation (ICRA). IEEE, 2022.
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2023
Citation:
Shivam Kumar Panda, Yongkyu Lee, Mohammad Khalid Jawsed, AgroNav: Autonomous Navigation Framework for Agricultural Robots and Vehicles Using Semantic Segmentation and Semantic Line Detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (pp. 6271-6280) (2023)
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2023
Citation:
Yayun Du, Swapnil Sayan Saha, Sandeep Singh Sandha, Arthur Lovekin, Jason Wu, S. Siddharth, Mahesh Chowdhary, Mohammad Khalid Jawed, and Mani Srivastava, Neural-Kalman GNSS/INS Navigation for Precision Agriculture, 2023 IEEE International Conference on Robotics and Automation (ICRA 2023). DOI: 10.1109/ICRA48891.2023.10161351
|
Progress 02/01/22 to 01/31/23
Outputs
Target Audience:The primary audiences are the researchers, plant breeders, industry (if any), and growers of row crops, e.g. flaxseed, canola, tomato, and strawberry. We will release the design and methodologies to the public domain. Once the technology is tested in the research field, it could be transferred into the growers' field for on-farm demonstration. AmeriFlax (represents North Dakota Flax Producers Association) will cooperate to test the technology in farmer's fields to encourage growers to adopt the technology. Changes/Problems:In 2022, the field experiments were conducted in three locations. A contrasting weed pressure was observed in two locations. In Fargo near the NDSU-GH location, a huge weed pressure was observed, whereas at Fargo Flax breeding nursery the weed germination was almost zero. Therefore, we dropped the Fargo Flax breeding nursery for data taking. In the field research program, this type of unexpected situation is common and therefore we conduct field experiments at multiple locations to mitigate the unexpected situation may seen by any experiment. What opportunities for training and professional development has the project provided?Graduate students: Afrina Rahman, a graduate student at North Dakota State University, employed on this project from May 2021. She has been conducting experiments in the greenhouse and in the field to evaluate the performance of small robots to spray herbicides and weed killing. In 2022, she conducted the field experiments at three locations of North Dakota using the protocol described in the project methodology. Currently, she is working on the data generated in 2022. Md Rakib Hasan Khan, a graduate student at North Dakota State University, employed for six months on this project in 2022. The graduate student worked with PI to generate robot-actuable, in-field herbicide delivery reagents and mechanisms for sustained and automated management of weed growth and proliferation. Yayun Du, a female PhD student, worked on this project since its inception. She successfully completed her doctoral program in 2022 and went on to join Northwestern University as a postdoctoral researcher. Her work has been published in several journals; see our response to the query, "How have the results been disseminated to communities of interest?" Shivam Panda and Yongkyu Lee conducted the field experiments in Fargo during the summer of 2022, working closely with our collaborators, PI Rahman in Plant Science and his student Afrina Rahman. The robot conducted three successful runs of spraying across multiple flax fields. The performance and outcome of robotic weed control are maintained by PI Rahman's group. The image data collected through this project also allowed for work on autonomous navigation. This work is currently in progress and is expected to be presented at one of the top robotics conferences. The graduate students (Shivam Kumar Panda and Yongkyu Lee) contributed to preparing this project report. Undergraduate students: UCLA has a research program called the "Undergraduate Research Center" (http://sciences.ugresearch.ucla.edu/courses/srp/) for undergraduate students that enables qualified students to work on research projects in exchange for academic credit. Opportunities for some of the undergraduate students were provided through this program. (1) Dylan Santora: Dylan started working on the project with Shivam and Yongkyu in Fall 2022, under the Research Experience for Undergraduates (REU) program of the National Science Foundation. His stipend is not included in the USDA budget. He has worked on the CAD design of the new version of the robot and will devote the next quarters to the manufacturing and testing of the robot. (2) Manya Agrawal: Manya contributed to the project from Fall 2022 under the SRP 99 (Student Research Program) course of UCLA. She has been tasked with implementing the push-push mechanism for the charging station to enable safe and easy contact. Her work for the next quarter will focus on designing a PCB to test out charging through this new mechanism. (3) Other students: The project opened new opportunities for four other students (Howard Zhu, Nathan Gong, Nemi Desai, Oliver Simpson) starting from January 2023. Howard is participating in research through the REU program whilst the others are under the SRP 99 program. Howard, Nathan and Nemi are Computer Science majors who will be responsible for developing the software for different modules of the robot. Oliver is a Mechanical Engineering student who will focus on design and hardware. How have the results been disseminated to communities of interest?An educational YOUTUBE video (https://www.youtube.com/watch?v=7B1g0MbuKtc) of efficient used drone and robot for crop phenotyping and weed control was created by Dr. Rahman lab. A webpage (https://structures.computer/weed-management) has been created under Prof. Jawed's lab website to record and share the findings on this research project. Latest videos and news are shared through this webpage. It will be continually updated to reflect current progress. Two articles have been published in International Conference on Robotics and Automation (ICRA), which is one of the top conferences in Robotics. (1) Du, Yayun, Guofeng Zhang, Darren Tsang, and M. Khalid Jawed. "Deep-CNN based Robotic Multi-Class Under-Canopy Weed Control in Precision Farming." International Conference on Robotics and Automation (ICRA) (2022) https://doi.org/10.1109/ICRA46639.2022.9812240 (2) Du, Y., Saha, S.S., Sandha, S.S., Lovekin, A., Wu, J., Siddharth, S., Chowdhary, M., Jawed, M.K., Srivastava, M., "Neural-Kalman GNSS/INS Navigation for Precision Agriculture" in press with IEEE International Conference on Robotics and Automation (ICRA) (2023) A video description of our work on Neural-Kalman GNSS/INS Navigation for Precision Agriculture is available on youtube(https://youtu.be/9e3Q_9aTCQ4) The dataset and code for this work is publicly available on Github (https://github.com/nesl/agrobot) The ICRA presentation on "Deep-CNN based Robotic Multi-Class Under-Canopy Weed Control in Precision Farming" has been published on Youtube (https://www.youtube.com/watch?v=mLXtLS94m38). We submitted the manuscript describing our work of azoxystrobin encapsulation into nanoparticles in the Journal of Agricultural and Food Chemistry. However, the manuscript has been recommended for another journal by the editors. Currently, we are addressing the reviewers' comments for this manuscript. What do you plan to do during the next reporting period to accomplish the goals? We will continue the evaluation of the mode of action and performance of herbicide alone and herbicide loaded in the nano material in the greenhouse. We will also optimize the doses of the herbicides and herbicides loaded nano compounds for effective killing of weeds in the greenhouse. The field experiments will be conducted again in the three locations in North Dakota. We will continue with the seven treatments for in-field weed control and will take the data on weeds and flax crops. Next year, we will use the newly developed plot-size robot developed in Dr. Jawad's lab in the field. Finally, we will harvest the crop and weeds from each plot. We will Prepare at least one new class of nanoparticles-reinforced herbicides. We will investigate the mechanism by which nanoparticles create nano-defects on the leaf surfaces of herbs for augmented herbicidal action. Manufacturing and field trials of the larger robot: A larger design that spans over an entire plot (4-5 rows) will be developed to match the efficiency of industry standards. The dimensions of the larger platform are: 75 in (L) x 63 in (W) x 40 in (H). Manufacturing of the new platform will begin starting February 2023 and is expected to take approximately three months. Software for weed detection and spraying, as well as autonomous navigation of the robot will be refined and implemented on the new robot simultaneously. We aim to develop a fully functioning robot by June 2023 to be prepared for field trials during the summer. The newer design will also house a pressure controlled spraying system to generate fine mist for effective spraying. Development of an overall navigation algorithm framework: The robot will have a perception system for local navigation and an RTK-system for global navigation. The perception system takes in the live images as input and our ML model does image segmentation to find the traversable areas within it. This traversable area is updated in our visual SLAM. The segmented image is used as an input to the centerline prediction model which predicts the centerline the robot should follow. And we already have a global direction from the RTK system. Now the global direction and local centerline are taken as input to the action block to take the final action command. The coordinates are accordingly updated on the RTK system as well as the SLAM. And this process loops on for the time of operation. The charging station: We are in the process of developing a novel charging station for mobile robots that significantly reduces the time required for charging batteries from the overall operation time. The major contribution of this charging station is its autonomous interface for the exchange of batteries. This allows mobile robots to continue their operations while their batteries are being charged, thus increasing the efficiency and productivity of the entire system. The charging station is designed with an advanced battery exchange mechanism that enables quick and seamless battery swap-out, reducing the downtime of the mobile robots. This innovative charging station is expected to be a game-changer for industries that rely on mobile robots, such as logistics, warehousing, and agriculture, by increasing their operational efficiency and reducing downtime.
Impacts
What was accomplished under these goals?
We have three TASKs under this project. TASK I: We have conducted three field experiments using a randomized complete block design with four replications during summer of 2022 at Fargo, North Dakota (Near NDSU Greenhouse, Flax breeding nursery, and NW22). We planted the flax cultivar "ND Hammond" for this study. We used a mixture of three herbicides commonly used in flax breeding nursery in robots to control in-plot broadleaf and narrow-leafed weeds. The herbicides were Basagran®, MCPA (2-methyl-4-chlorophenoxyacetic acid), Select Max®, and the rate of application were 0.75 pint, 0.5 pint and 1.0 pint, respectively, mixed with 10 gallon water for one acre land. The treatments of the experiments were (1) natural plot i.e. no weed control (negative-control), (2) weed free plot, all the weeds were cleaned by hand, (3) weed control using traditional equipment by mixed-herbicides (spray at 3rd leaf and bud visible stage), (4) weed control using robot by mixed-herbicides (non-NANO) (spray at 3rd leaf and bud visible stage), (5) weed control using robot by mixed-herbicides (non-NANO) (spray at about every 15 days), (6) weed control using robot by NANO-mixed-herbicides (spray at 3rd leaf and bud visible stage), and (7) weed control using robot by NANO-mixed-herbicides (spray at about every 15 days). The seeding rate was 35 lb/acre and maintained about 65 plants per square foot. Weather is always a problem for field crop production. We did not see weed pressure at Flax breeding nursery site and therefore could not get effective data from there. An unusual huge weed pressure was observed at Near NDSU Greenhouse and NW22 sites. The small plot Robot had difficulty efficiently controlling in-plot weeds. Therefore, treatment 2 (weed control by hand) and treatment 3 (weed control by traditional big equipment) showed significantly higher seed yield over other treatments. However, the two robotic weed control treatments (4, 5) showed statistically similar seed yield compared to hand weeding (treatment 2). The seed yield of all robotic weed control treatments was statistically similar; however, they were statistically lower than the control treatment (no weed control). In addition to seed yield, we also obtained data on (i) weed density/sq.m at 25 days after germination, (ii) weed density/ sq.m at 40 days after germination, (iii) weed density/ sq.m at 50 days after germination, (iv) weed biomass, (v) plant height, (vi) number of bolls per plant, (vii) number of tillers per plant, and (viii) 1000 seed weight. We are in progress to analyze all data and publish in peer-reviewed journal. TASK II: A new iteration of the robot was completed this year. The major changes from the previous version were the aspect ratio of the robot, the onboard electronics, and the navigation algorithm. Aspect Ratio: Based on the inputs from the previous field trials aspect ratio of the new robot was changed from 1:0.67:1 to 1:0.67:0.75, providing a more stable traversing over uneven terrain. Onboard electronics: Based on the requirements of the robot, the main controller of the robot was switched from Jetson Xavier to Jetson Nano which consumes less power. The Arduino was removed from the robot connecting all motors and sensors to the Jetson. All the wirings, motor and pump drivers, and other electronic components were changed, reorganized, and made more robust for user application. This reduced the susceptibility to circuit breakdowns on the field and, at the same time, increased the range from 5.8 miles to 7.2 miles on a single charge. Navigation algorithm: This year, we switched to complete vision-based navigation for inter-row traversing. The algorithm uses state-of-the-art transformers-based semantic segmentation for segmenting the rows (or traversable regions) from the crops (or non-traversable regions). The implementation of this algorithm is ongoing. So the new iteration of the robot designed this year had the following specifications: Size: 1.1 feet long, 0.75 feet wide, and 0.82 feet tall. Payload: 2.65 pounds or 40.6 oz of high-efficiency herbicide. Speed: Top speed is 20 cm/sec. Range: 7.2 miles on a single charge Onboard intelligence: The partial autonomous robot drives on vision-based information for inter-row navigation. The weed detection algorithm is the same as before. Field operations were executed in three cycles over 3 flax fields in North Dakota. Based on the observations, a decision was made to scale up the operation by using a larger robot that can go over multiple rows altogether. This would improve the robot's stability, increase its time efficiency by 12 times, provide housing for an air-mist sprayer system and increase the payload to over 2 gallons. The CAD design of the larger robot has been completed, and the manufacturing is ongoing. With the objective of building a large open-source dataset in agriculture for vision-based navigation, over 1000 images were collected from 6 different crops at different heights of 20 cm, 140 cm, 3 m, 10 m and 30 m from the ground. This covers the position of cameras for different robots, vehicles and drones in an agricultural setting. Over 100 images have been annotated and the target is to reach 500 annotated images. The paper for the vision-based navigation system is in the writing process. A tentative abstract for the paper is as follows. This paper explores the use of semantic segmentation-based machine learning for autonomous navigation in agriculture. The key contribution of this work is the creation of the largest open-source annotated dataset, "AgroNav," specifically designed for agricultural navigation tasks for agricultural robots, vehicles, and drones. Additionally, the paper presents a hierarchical strategy for accurate transfer-learning with the limited dataset for a downstream task such as agriculture navigation by utilizing the checkpoints of architectures trained on the cityscape dataset. The performance of multiple semantic segmentation models is evaluated and compared based on both accuracy and real-time performance. This research provides a valuable resource for developing advanced agricultural navigation systems. The CAD model for the new charging station system was completed. The major contribution of The charging station is designed with an advanced battery exchange mechanism that enables quick and seamless battery swap-out, reducing the downtime of the mobile robots. In the following, we report on the findings that have been published or are under review. (1) Du, Yayun, et al. "Deep-cnn based robotic multi-class under-canopy weed control in precision farming." 2022 International Conference on Robotics and Automation (ICRA). IEEE, 2022. [https://ieeexplore.ieee.org/document/9812240] (2) Yayun Du, Swapnil Sayan Saha, Sandeep Singh Sandha, Arthur Lovekin, Jason Wu, S. Siddharth, Mahesh Chowdhary, Mohammad Khalid Jawed, and Mani Srivastava, "Neural-Kalman GNSS/INS Navigation for Precision Agriculture," IEEE International Conference on Robotics and Automation (ICRA) 2023 (accepted) TASK III: Liquid nano complexes/particles have been manufactured, and their chemical characterization is ongoing. Variability of herbicide function as a function of herb types is currently under investigation.
Publications
|
Progress 02/01/21 to 01/31/22
Outputs
Target Audience:The targeted audiences are researchers, plant breeders, industry (if any), and growers. Changes/Problems:In 2021, the field experiments were conducted in two locations. A contrasting weed pressure was observed in the two locations. In Casselton, the weed germination was almost zero. However, weed germination and growth was unexpectedly severe in Fargo. Therefore, we are planning to conduct the experiments in three locations in 2022. What opportunities for training and professional development has the project provided?Graduate students: Yayun Du, a female graduate student, led the project at UCLA. She was named one of the 2019 ``Top Ten Students" at Harbin Institute of Technology, Weihai (http://hitxg.hitwh.edu.cn/_t35/2015/0608/c1208a45843/page.htm) and awarded the Mazuguang scholarship. After joining our Structures-Computer Interaction Lab, she published or submitted six first-authored articles and two co-authored articles within 2.5 years in top journals (e.g. Soft Robotics, IEEE Robotics, and Automation Letters) and conference proceedings (International Conference on Intelligent Robots and Systems (IROS)) in robotics. Our research supported by this funding was also published in IROS 2021 and was a finalist for the Best Paper Award in two categories: (1) Agri-Robotics and (2) Robot Mechanisms and Design (Only 4 out of 1261 papers were selected as finalists in each category, https://www.iros2021.org/awards). She was also named a Rising Star by MIT Civil and Environmental Engineering (https://www.mae.ucla.edu/mae-ph-d-student-yayun-du-selected-as-a-rising-star-by-mit-cee/) and received a four-year UCLA Graduate Division Fellowship. She is slated to graduate in 2022. Two new graduate students, Shivam Panda and Yongkyu Lee have been recruited to continue the work on this project. Yayun visited and stayed in Fargo during the summers of 2019 and 2021, working closely with our collaborators, PI Rahman in Plant Science and PI Qadir in Chemical Engineering, as well as their Master's and doctoral students at North Dakota State University (NDSU). One female Ph.D. student in Professor Rahman's group, Afrina Rahman is also supported by this funding. She will mainly work on robotic control in her coming Ph.D. career. She is mainly working on robotic control in her coming Ph.D. career. She set the planned experiments at three locations using appropriate experimental design. She harvested the experimental plots and conducted a statistical analysis. Our robot ran in flaxseed fields in the summer of 2021 and sprayed weeds three times, on 07/07, 07/22, and 08/02. The performance and outcome of robotic weed control are decent. The flaxseeds were harvested and PI Rahman's group recorded the details of seeds and harvests, such as seed yield, number of bolls per plant, seed weight, and so on, as described in the proposal. The graduate students (Yayun Du, Shivam Kumar Panda, Yongkyu Lee) contributed to preparing this project report. One graduate student worked full time over the three (03) summer months of 2021 under Dr. Mohi Quadir's lab for spray formulation of nanocomplexes/particles that were deployable via robotic devices. Undergraduate students: (1) Arthur Lovekin: Under the Research Experience for Undergraduates (REU) program of the National Science Foundation, Arthur Lovekin worked with Yayun Du in the Winter of 2020 and stayed with Yayun in Fargo throughout the summer of 2021. He is applying to graduate schools. UCLA has a research program called the "Undergraduate Research Center" (http://sciences.ugresearch.ucla.edu/courses/srp/) for undergraduate students that enables qualified students to work on research projects in exchange for academic credit. This does not incur any cost to the grant. Three students (Bhrugu Mallajosyula https://www.linkedin.com/in/bhrugum/, Arthur Lovekin (arthurlovekin.com), Jingyi Chen https://www.linkedin.com/in/jingyi-yazmine-chen/) worked on this project through the Undergraduate Research Center. Bhrugu is now an AV Controls Integration Engineer at General Motors, while Jingyi enrolled in graduate school at Cornell. Arthur worked in Fall 2021 under the course SRP 99. Among these students, Bhrugu Mallajosyula and Jingyi Chen were co-authors of the manuscript on "Low-cost Robot with Autonomous Recharge and Navigation for Weed Control in Fields with Narrow Row Spacing". Yayun is preparing two manuscripts for this project and Arthur will also be one of the authors. How have the results been disseminated to communities of interest?A webpage (https://structures.computer/weed-management) has been created under Prof. Jawed's lab website to record and share the findings on this research project. Latest videos and news are shared through this webpage. It will be continually updated to reflect current progress. Two articles have been published in International Conference on Intelligent Robots and Systems (IROS) and International Conference on Robotics and Automation (ICRA), both being the top conferences in Robotics. (1) Yayun Du, Guofeng Zhang, Darren Tsang, Mohammad Khalid Jawed. "A Low-cost Robot with Autonomous Recharge and Navigation for Weed Control in Fields with Narrow Row Spacing." In the 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021 [https://ieeexplore.ieee.org/abstract/document/9636267]. (2) Du, Yayun, Guofeng Zhang, Darren Tsang, and M. Khalid Jawed. "Deep-CNN based Robotic Multi-Class Under-Canopy Weed Control in Precision Farming." arXiv preprint arXiv:2112.13986 (2021). [https://arxiv.org/abs/2112.02162] The dataset produced in the above article along with its multi-class classification between plants and weeds has been made public through Github (https://github.com/StructuresComp/Multi-class-Weed-Classification) The ICRA presentation on "Deep-CNN based Robotic Multi-Class Under-Canopy Weed Control in Precision Farming" has been published on Youtube (https://www.youtube.com/watch?v=mLXtLS94m38). Multiple videos of the Robot in operation and on the implementation of the above literature have been made public on YouTube. Robot with autonomous recharge & navigation for weed control in fields with narrow row spacing: https://www.youtube.com/watch?v=UBDXrMeFY8U Deep learning-based real-Time robotic multi-Class weed identification: https://www.youtube.com/watch?v=AOYxt7r6_vU Robot spraying through three continuous rows of flax: https://www.youtube.com/watch?v=opfVQwYf9Z0 Robot navigating back to the charging station: https://youtu.be/_lGMyo9wS3Q Video on practical use of Robot in NDSU Oilseed Breeding program for precision weed control have been made public on YouTube: Drone and Robot in NDSU Oilseed Breeding program https://www.youtube.com/watch?v=48V2Y_Hpqxo What do you plan to do during the next reporting period to accomplish the goals?(1) New version of design: We are planning to improve the design of the agricultural robot. In the new design, the space would be optimized with a new herbicide tank, a new gimbal and gimbal adapter for the front top camera, compact electronics, and organized wiring. The robot will be more compact and robust in terms of center-of-mass balance. (2) Implementation of SLAM algorithms in the field: SLAM algorithms to be developed during the spring quarter to implement and experiment during the summers for navigation. (3) Redesigning the charging station: We have a concept of developing a new system for the charging station. It would possess an interface (consisting of a manipulator and battery adapter) where the batteries from the agricultural robot can be exchanged with newly charged batteries, and the used batteries can be put back to charging at the station. This would make the charging process more efficient in terms of time and energy loss. We can expect a publication based on the design of this charging station (as its application can also be extended to UAVs in the future). (4) Data collection using the agricultural robot: Quantification of several morphological traits of flax crops and hyperspectral imaging for oil content analysis requires the robot to collect videos of the crops. The UGV will be equipped with multiple cameras for stereo vision so that it can survey both the crops and the weeds. (5) The experiment will be repeated in 2022 at three locations in North Dakota using the same experimental design proposed in the project proposal. We will also test the efficiency of herbicide and nano-coded herbicide in the greenhouse. (6) Completion of the manuscript, completing characterization of fungicide-loaded nanoparticles. Field trials of nanosuspensions and evaluation of nanoparticle efficiency against weeds.
Impacts
What was accomplished under these goals?
TASK I: We have conducted two experiments using a randomized complete block design with four replications during summer of 2021 at Fargo and Casselton of North Dakota. We planted the flax cultivar "ND Hammond" for this study. We used a mixture of three herbicides commonly used in flax breeding nursery in robots to control in-plot broadleaf and narrow-leafed weeds. The herbicides were Basagran®, MCPA (2-methyl-4-chlorophenoxyacetic acid), Select Max®, and the rate of application were 0.75 pint, 0.5 pint and 1.0 pint, respectively, mixed with 10 gallon water for one acre land. The treatments of the experiments were (1) natural plot i.e. no weed control (negative-control), (2) weed free plot, all the weeds were cleaned by hand, (3) weed control using traditional equipment by mixed-herbicides at around 3rd pair true leaves stage and bud visible growth stage, (4) weed control using robot by mixed-herbicides (non-NANO) on July 07, July 22, and August 02 at bud visible growth stage, (5) weed control using robot by NANO-mixed-herbicides on July 07, July 22, and August 02 at around bud visible growth stage. The seeding rate was 35 lb/acre and maintained about 65 plants per square foot. Weather is always a problem for field crop production. We did not see any weed pressure at Casselton site and therefore could not get effective data from there. An unusual huge weed pressure was observed at the Fargo location. The small plot Robot had difficulty efficiently controlling in-plot weeds. Therefore, treatment 2 (weed control by hand) and treatment 3 (weed control by traditional big equipment) showed significantly higher seed yield. However, the two robotic weed control treatments (4, 5) showed higher seed yield over control treatment (no weed control). TASK II: Our project progressed beyond expectations. The original technical features of the robot were as follows: (1) Size: 1 foot (or less) in length, width, and height. (2) Payload: 1 pound of robot-actuable high-efficiency herbicides (equivalent to approximately 10 pounds of conventional herbicides). (3) Speed: 10 cm/sec on the uneven terrain of flax fields. (4) Range: 1 mile on a single charge. (5) Onboard intelligence: Processing power for navigation and weed detection. The actual technical specifications for the robot design that we have now are as follows: (1) Size: 0.98 feet long, 0.66 feet wide, and 0.98 feet tall. (2) Payload: 1.1 pounds of robot-actuable high-efficiency herbicide. (3) Speed: About 21 cm/sec on the uneven terrain of flax fields on average. The actual robot speed depends on the density of weeds in the fields. If the weeds are denser, the robot should move slower to detect and spray the herbicide onto the detected weeds. (4) Range: 5.8 miles on a single charge. (5) Onboard intelligence: We have cheap off-the-shelf cameras, LiDAR, embedded board, and inertial measurement unit (IMU) for inter and intra row navigation, weed detection, and autonomous recharging using algorithms relying on the input from the vision (i.e. cameras). All original objectives stated in the proposal have been achieved. In the following, we report on the findings that have been published or are under review. (1) Yayun Du, Guofeng Zhang, Darren Tsang, Mohammad Khalid Jawed. "A Low-cost Robot with Autonomous Recharge and Navigation for Weed Control in Fields with Narrow Row Spacing." In the 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021 [https://ieeexplore.ieee.org/abstract/document/9636267]. In this literature, we have presented an autonomous robot for weed management for narrow row crops such as flax and canola. According to the Subtask II. As of the proposal, the design of the robot presented here meets the size, payload, range, and speed requirements of the project (also discussed above). As proposed in Subtask II.B in the proposal, a vision-based navigation system on a color-based contour algorithm has been developed which accomplishes navigation between straight crop lines as well as curved and irregular crop lines. Further, the robot also achieves inter-row navigation using monocular camera vision and the Inertial Measurement Unit (IMU). A novel design of an angle-adjustable spraying system has been developed for Subtask II.C. Also, an autonomous recharging system has been established where computer vision has been used to align the charging arm of the robot with the socket of the charging station. Finally, we present our test results achieved from our experiments in the real flaxseed fields in North Dakota. (2) Du, Yayun, Guofeng Zhang, Darren Tsang, and M. Khalid Jawed. "Deep-CNN based Robotic Multi-Class Under-Canopy Weed Control in Precision Farming." arXiv preprint arXiv:2112.13986 (2021). [https://arxiv.org/abs/2112.02162] This work pursues the software development mentioned in Subtask II.C of the proposal on weed detection. In this paper, we present AIWeeds, a large image dataset of 10,000 labeled images of flax and 14 of the most common weeds found in agricultural fields of North Dakota, Central China, and California. The images have diversification in terms of illumination, weather conditions, perspective, and plant growth. A multi-class weed classification pipeline has been accomplished for model deployment even on low-end GPU boards e.g. Jetson Nano, Jetson Xavier Nx. Further, we studied the classification performance using five benchmark CNN models. Finally, we deployed MobileNetV2 on the robot for real-time weed detection and achieved 90% test accuracy. This can be considered a milestone for real-time autonomous precision weed control. TASK III: The major goals of this project were (i) preparation and characterization of sprayable nanocomplexes of commercially available fungicide, (ii) development of spray formulation of these nanocomplexes/particles that can be deployable via robotic devices, and (iii) preparation of ground-deployable, micro-structured beads loaded with herbicides. The accomplished goals are: Microbead development has been achieved. Functional and structural characterization have been completed. Liquid nanocomplexes/particles have been manufactured, and their chemical characterization is ongoing.
Publications
- Type:
Journal Articles
Status:
Published
Year Published:
2021
Citation:
Yayun Du, Guofeng Zhang, Darren Tsang, Mohammad Khalid Jawed. "A Low-cost Robot with Autonomous Recharge and Navigation for Weed Control in Fields with Narrow Row Spacing." In the 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021 [https://ieeexplore.ieee.org/abstract/document/9636267].
- Type:
Journal Articles
Status:
Published
Year Published:
2021
Citation:
Du, Yayun, Guofeng Zhang, Darren Tsang, and M. Khalid Jawed. "Deep-CNN based Robotic Multi-Class Under-Canopy Weed Control in Precision Farming." arXiv preprint arXiv:2112.13986 (2021).
|
|