Source: UNIVERSITY OF NEBRASKA submitted to NRP
UGV AND UAV COLLABORATION FOR AUTOMATED SEED REFILLING IN ROW CROPS (U2AGV-REFILL)
Sponsoring Institution
National Institute of Food and Agriculture
Project Status
ACTIVE
Funding Source
AFRI COMPETITIVE GRANT
Reporting Frequency
Annual
Accession No.
1025550
Grant No.
2021-67021-34411
Cumulative Award Amt.
$452,783.00
Proposal No.
2020-08908
Multistate No.
(N/A)
Project Start Date
Mar 1, 2021
Project End Date
Feb 28, 2026
Grant Year
2021
Program Code
[A1521]- Agricultural Engineering
Recipient Organization
UNIVERSITY OF NEBRASKA
(N/A)
LINCOLN,NE 68583
Performing Department
Biological Systems Engineering
Non Technical Summary
he notion of replacing one large piece of agricultural equipment with multiple small autonomous Unmanned Ground Vehicles (UGVs) is gaining momentum in discussions across research and industry groups. This notion is promising in small acreages, which are characteristic of specialty crops and orchards, however, for Midwestern row-crop production settings, where field sizes are large, and time windows for finishing the tasks are short, many challenges exist to fully realize this concept. One of the important challenges in using multiple small UGVs that require innovation and attention is the ability to automatically handle crop input material such as seeds, chemicals, and fertilizers within the field. The full potential of a multi-robot systems consisting of multiple small UGVs can only be realized if a practical and robust material refilling strategy is devised. A more efficient approach could involve the ability to refill the tanks of the small robotic planters without the need for the planting robot to leave the planting locations in the field. We propose a novel approach for material refilling of UGV seed tanks with UAVs by developing an automatic material refill system framework called U2AGV-Refill. We hypothesize that the ability to refill the tanks of small UGVs in the field as opposed to refilling at the field edge, will offer significant advantages towards improved field efficiencies and capacities of robotic planting. U2AGV-Refill system will address important knowledge gaps in the area of UGV-UAV collaborative strategies for automatic material refill in row-crop production systems.
Animal Health Component
40%
Research Effort Categories
Basic
30%
Applied
40%
Developmental
30%
Classification

Knowledge Area (KA)Subject of Investigation (SOI)Field of Science (FOS)Percent
40253102020100%
Knowledge Area
402 - Engineering Systems and Equipment;

Subject Of Investigation
5310 - Machinery and equipment;

Field Of Science
2020 - Engineering;
Goals / Objectives
The long-term goal of our research is to deploy a system consisting of both UGVs and UAVs working collaboratively to accomplish field tasks autonomously in Midwestern row crops. The goal of this project is develop and test U2AGV-Refill system, which consists of logistic models, field usable automatic material refill mechanisms, control systems, and algorithms for enabling UGV and UAV collaboration.To accomplish our project goal of developing U2AGV-Refill system, our specific objectives are to:Develop logistic models, algorithms, and control systems for UGV and UAV collaboration for automated material refilling.Design field usable automated refill mechanisms at the interface of UGV and UAV for refilling of crop materials (e.g., seeds).Simulate and field test U2AGV-Refill system for refilling of corn seed using Flex-Ro robotic platform (UGV) and UAVs.
Project Methods
Overall logistic models and the collaboration framework of the U2AGV-Refill system will be developed as part of objective 1. Once the framework is developed, algorithms will be deployed on two test beds. The first test bed is a 56 HP multi-purpose UGV called Flex-Ro, capable of pulling two corn row planter units at typical ground speeds. The robotic UGV planter is a hydrostatic machine with 4WD and independent four wheel steering with footprint of 10 ft x 10 ft. It is equipped with a controller area network (CAN) bus system with multiple electronic control units (ECUs), where each ECU is dedicated to perform a unique vehicle function. All the ECUs exchange CAN message on the bus using SAEJ1939 standard protocols. Additional ECUs can be easily added to the CAN bus system for controlling additional vehicle functions. The Flex-Ro UGV has a mission planning software primarily for AB line navigation. The mission planner was developed using MATLAB and the application resides on the task computer of the UGV. Additional modules will be added to the mission planner to enable UGV-UAV collaboration. The second test-bed is a multi-rotor UAV platform (Model: DJI MATRICE 600 PRO) which will used as a seed tender UAV. Both these platforms are available in the PI and Co-PI labs and will be used during the lab and field evaluation project phase of the U2AGV-Refill framework.Novel UGV and UAV mechanisms and mechatronic systems will be developed to automate the process of refilling of the seed hopper on the UGV with a UAV seed tender tank.RTK GPS coordinates of the UGV will be used for aligning the UAV on top of the seed hopper of the UGV. LiDAR sensor and other camera feedback from the UAV will utilized to compensate for any potential misalignments. In addition to sensor feedback based alignment strategies, novel design elements will be included to ensure mechanical alignment. A tank with a conical bottom will be fitted to the UAV that will nest in a similarly shaped cradle on the UGV platform. This will facilitate automatic alignment of the UAV seed tender tank with the seed hopper on the UGV. The docking interface and seed transfer mechanism will also have radial symmetry so that specific orientation between the vehicles is not required.As the UAV docks on the UGV, momentary and limit switches will be utilized to indicate that the UAV is in position signaling that the system is ready for seed transfer. While docking in the UGV cradle, the weight of the UAV will actuate a mechanical linkage/mechanism to open the UGV seed tank before lowering into final position. The nozzle of the UAV seed tender tank will then be opened with an electronic actuator to transfer the payload to the UGV. When the transfer is complete, the UAV nozzle will close and as the UAV lifts off the mechanically actuated gate/cap on the UGV tank will close. A paddle switch and other proximity sensors will be used in the UGV seed hopper that can detect when the tank is full, signaling the UAV to stop transferring seed.Full-scale commercial implementation of this concept would require custom designed seed tanks and supporting structures to fit the needs of the UAVs and UGV in use. However, to complete the proof-of-concept as part of this research objective, commercially available components will be used where possible although they may not be fully integrated or optimized for the available vehicles. Seed tanks will be procured of an appropriate size for each vehicle and mounting structures designed and will be fabricated locally. Resources at UNL (see facilities and resources document for more details) and local industry will be utilized for manufacturing and fabrication, machining, and 3D printing of components of various designs.The simulation tests at the NTTL test track will focus on the ability of the U2AGV-Refill system to enable exchange of machine states and material information between the UAV and UGV, implement leader-follower behavior, docking of the UAV on the UGVs, and finally demonstrate the automatic transfer of seed from the UAV's seed tender tank into the UGV's seed hopper. Many operations need to happen in sequence to achieve the goal of seed transfer from UAV to the UGV. Actual planting of the seed will not take place during the simulation tests on the NTTL track, rather the focus will be primarily on the performance testing of the automated refill mechanism and the leader-follower behavior.Evaluation of the U2AGV-Refill system will be conducted in actual field conditions at the research field plots located in the Rogers's memorial farm (RMF) which is located approximately 10 miles east of the Department of Biological Systems Engineering (PI's department). The RMF is a no-till research farm, which is approximately 300 acres in size. Approximately 24 acres of field plot will utilized to demonstrate robotic planting and automatic refilling with the U2AGV system. The robotic planter UGV will be used to plant corn in 30 in. rows at an average ground speed of 6.4 Kmh-1 (4 mph). In this field evaluation, the seed hopper on the UGV will be filled in increments of 10 kg (approx. 22 lbs) as that payload is the maximum capacity of the UAV test-bed we are using in the study. Approximately 160 lbs of seed (3 bags) of seed corn is required for every 8 acres, so a total of 480 lbs (approx. 220 kg) of seed corn is required to plant a 24 acre field plot. This robotic planting operation will result in approximately 22 trips by the UAV to refill the seed hopper of the UGV at a rate of 10 kg seed drop per trip. Again, the performance of the U2AGV-Refill system will be evaluated based on three important criteria 1) the ability of the UGV and UAV to exchange wireless machine and material statuses, 2) the ability of the UAV to follow and hover above the UGV, and 3) the ability to dock and transfer material into the UGV's seed hopper successfully. The number of successful trips out of 22 trips made by the UAV will determine the overall efficacy of the U2AGV-Refill system. The results obtained will be extrapolated to actual field production system settings, where the seed tender tank capacity of the UAV could go up to 100 kgs, in which case, approximately a little more than two trips are needed for refilling the tank of the robotic UGV planter to finish planting 24 acres of land. Field efficiencies and field capacities of the system will be extrapolated for a quarter-section field area based on the results obtained in the field plot evaluation.

Progress 03/01/23 to 02/29/24

Outputs
Target Audience:Undergraduate and graduate communities at UNL were trained during this year. Project methodologies and research outputs were presented to the scientiific community at conferences, to equipment manufactures and collaborating producers. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided?-Three graduate students ( 3 MS) and two undergraduate student were mentored by the PD and Co-PDs. -As part of professional development activities, 5 lecture type presenations and one poster presenation was succesfully presented by the graduate students at international conferences -Students gained knowledge, skills and abilities (KSAs) in the areas of Robotic Operating Systems (ROS), machine vision, robotic simulators, drone opertion, algorithm development,CAN bus technologies, and UAV - UGV collobaration dynamics. How have the results been disseminated to communities of interest?- Results have been disseminated at ASABE AIM meetings, IEEE ROS workshops, and Ag equipment technology conferences. - Demonstrations of the technology to producer community and k-12 students were done at Farm Progress Show in Iowa, Discover days at UNL campus, and farm shows in Nebraska What do you plan to do during the next reporting period to accomplish the goals?- Continue to imporve the U2AGV refill system by testing it in field conditions - Submit journal articles to disseminate developed technologies and results

Impacts
What was accomplished under these goals? Objective 1:Develop logistic models, algorithms, and control systems for UGV and UAV collaboration for automated material refilling - Logistic and mission planning modelsfor multiple UGV and UAV collaboration for planting operation and material refeilling is finished - Hardware and software for autonomous UGV and UAV navigation is developed and field tested Objecitve 2:Design field usable automated refill mechanisms at the interface of UGV and UAV for refilling of crop materials (e.g., seeds). - Automated seed refill mechanisms of both UGV and UAV are designed and tested in controlled field conditions with planting as the target application - Hardware, instrumetnation, and the vision algorithms are developed and tested Objective 3:Simulate and field test U2AGV-Refill system for refilling of corn seed using Flex-Ro robotic platform (UGV) and UAVs. - Simulation of U2AGV refill system for refilling corn seed on Flex-Ro platform isfinished in controlled field conditions

Publications

  • Type: Theses/Dissertations Status: Published Year Published: 2023 Citation: MS Thesis 1:Ground and Aerial Robot Collaboration: Machine Vision and Autonomy Framework Development for Material Refilling in Row-Crop Agriculture , Herve Mwunguzi
  • Type: Theses/Dissertations Status: Published Year Published: 2023 Citation: MS Thesis 2: Ground and Aerial Robot Collaboration Architecture: Material Refilling Models for Supporting Collaborative Heterogeneous Field Agricultural Robots, Terence Irumva
  • Type: Theses/Dissertations Status: Published Year Published: 2023 Citation: MS Thesis 3: Tempelmeyer, Ian J. (2023). Flex-Ro: Development and Testing of a Semi-Autonomous Robotic Planter (Master's thesis). University of Nebraska-Lincoln, Lincoln, NE
  • Type: Conference Papers and Presentations Status: Published Year Published: 2023 Citation: T. Irumva, H. Mwunguzi, S.K. Pitla. Modeling the UAV-UGV Collaboration Architecture for Material Refilling in Support of Robotic-Autonomous Planting. ASABE Conference Presentations during 2023 AIM, Omaha, Nebraska
  • Type: Conference Papers and Presentations Status: Published Year Published: 2023 Citation: T. Irumva, H. Mwunguzi, S.K. Pitla.Validating Multi-Robot Collaboration Physics-Informed Models for Agricultural Applications Using Robotics Simulators . ASABE Conference Presentations during 2023 AIM, Omaha, Nebraska
  • Type: Conference Papers and Presentations Status: Published Year Published: 2023 Citation: T.Irumva, C.T. Liew, I. Tempelmeyer. Assessing the Feasibility of Replacing Big Agricultural Machinery with Smaller Agricultural Robots for Planting Operations . ASABE Conference Presentations during 2023 AIM, Omaha, Nebraska
  • Type: Conference Papers and Presentations Status: Published Year Published: 2023 Citation: H. Mwunguzi, S.K.Pitla, T. Irumva. Development and testing of a custom UAV testbed for autonomous agricultural operations.ASABE Conference Presentations during 2023 AIM, Omaha, Nebraska
  • Type: Conference Papers and Presentations Status: Published Year Published: 2023 Citation: H. Mwunguzi, S.K.Pitla, C.Liew, T. Irumva. Design and development of UAV-UGV coordination mechanism (U2AGV) for material refilling in autonomous agricultural operations. ASABE Conference Presentations during 2023 AIM, Omaha, Nebraska
  • Type: Conference Papers and Presentations Status: Published Year Published: 2023 Citation: H. Mwunguzi, S.K.Pitla, C.Liew, T. Irumva. Vision-based autonomous UAV landing for material refilling for agricultural operations. ASABE Conference Presentations during 2023 AIM, Omaha, Nebraska


Progress 03/01/22 to 02/28/23

Outputs
Target Audience:Undergraduate and graduate communities at UNL, Scientific Community working in the area of agricultural robotics, producers, equipment manufactures and start-up companies. Changes/Problems:One major challenge has been the non-availability of UAVs that can be fully programmable and open-source. We have addressed this challenge by building our own custom drones with flight controllers and other computing hardware which can be fully programmableto deploy our custom algorithms. What opportunities for training and professional development has the project provided? Three gradaute students and two undergraduate students are trained in the area of ground and aerial robots for row-crop production use cases Three graduate students and two undergraduate students are trained in conducting field level agricultural robotic research Professional development activities include presentations and demos at conferences and workshops (e.g., ASABE conference, Farm Progress Show (Boone, Iowa), Nitrogen Use Efficiency Workshop (Lincoln, Nebraska) How have the results been disseminated to communities of interest? The results of the project were disseminated with presentations and posters at conferences The project restults were disseminated using robotic demos to K-12 and producers of Nebraska What do you plan to do during the next reporting period to accomplish the goals? Test the UAV-UGV Refill mechanism Test the Autonomous features of the UAV-UGV refill system in field conditions Demostrate UAV-UGV coordiantion in simulation environment

Impacts
What was accomplished under these goals? Develop logistic models, algorithms, and control systems for UGV and UAV collaboration for automated material refilling. UAV and UGV logistics and refill algorithms for corn planting are developed and are under validation Models for coordination of multiple UAVs as tender vehicles for multiple UGVs are developed UAV models for optimizing battery power, flight time, and payload capacity are develope. 2. Design field usable automated refill mechanisms at the interface of UGV and UAV for refilling of crop materials (e.g., seeds). Hopper mechanism on the UGV is designed and installed UAV hopper mechanism is designed and 3D printed 3. Simulate and field test U2AGV-Refill system for refilling of corn seed using Flex-Ro robotic platform (UGV) and UAVs. Robotic planting with Flex-Ro (UGV) is successfully finished UAV and UGV coordination algorithms are developed Base simulation environment for UAV-UGV collaboration is finished

Publications


    Progress 03/01/21 to 02/28/22

    Outputs
    Target Audience:Graduate and undergraduate student communities at UNL were trained during this projec period. Project initiation discussions with Nebraska producers, collaborating research sites, start-up companies, and equipment manufacturers were conducted Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided?PI and Co-PIs are providing training onUGVs and UAVs to three graduate and one undergraduate student How have the results been disseminated to communities of interest?- Submitted and Intellectural Property application on "robotic planter mechanisms"to UNL's commercialization office - Project concept was communicated to collaborating producers What do you plan to do during the next reporting period to accomplish the goals?- Finish robotic planting prototype development - Finish robotic reflilling mechanisms - Make progress on UAV-UGV coordiantion modeling - Field test UGV robotic planting - Install custom software on UAVs for path planning and machine vision based landing

    Impacts
    What was accomplished under these goals? Thelong-termgoal of our research is to deploy a system consisting of both UGVs and UAVs working collaboratively to accomplish field tasks autonomously in Midwestern row crops. Thegoal of this projectis develop and test U2AGV-Refill system, which consists of logistic models, field usable automatic material refill mechanisms, control systems, and algorithms for enabling UGV and UAV collaboration.To accomplish our project goal of developing U2AGV-Refill system, our specific objectives are to: 1. Develop logistic models, algorithms, and control systems for UGV and UAV collaboration for automated material refilling. -Systemic literature review of UGV-UAV collaboration was completed - Model for material refilling in field conditions is under development 2. Design field usable automated refill mechanisms at the interface of UGV and UAV for refilling of crop materials (e.g., seeds). - Novel robotic planting system is designed, which will be tested in controlled conditions - Multiple designs are critically discussed to identify the best design before developing a prototype 3.Simulate and field test U2AGV-Refill system for refilling of corn seed using Flex-Ro robotic platform (UGV) and UAVs. - Work with respect to simulation and field evnironments planning is underway

    Publications