NRI: INT: COLLAB: AN AUTONOMOUS INSECT SENSE, IDENTIFY, AND MANAGE PLATFORM (SIMPL) TO ADVANCE CROP PROTECTION STRATEGIES

• Sponsoring Institution: National Institute of Food and Agriculture
• Project Status: COMPLETE
• Funding Source: AFRI COMPETITIVE GRANT
• Reporting Frequency: Annual
• Accession No.: 1017750
• Grant No.: 2019-67021-28995
• Cumulative Award Amt.: $882,920.00
• Proposal No.: 2018-06105
• Project Start Date: Dec 1, 2018
• Project End Date: Nov 30, 2024
• Grant Year: 2019
• Program Code: [A7301]- National Robotics Initiative

Recipient Organization
KANSAS STATE UNIV
(N/A)
MANHATTAN,KS 66506

Performing Department
Biological & Ag Engineering

Non Technical Summary
Annually 37% of crops are lost to pest damage (13% of which are due to insects), which is a major threat to incomes of rural families and overall global food security. Chemical pest control is a necessary component of crop production for maximizing yields; so much so that $15.2 billion worth of pesticides were applied in 2016 [51]. As we move to sustainably intensify production to feed a growing population, can we limit the environmental impact while increasing farmer profitability by making these applications more efficient? Typically, pest incidence and severity are only fractionally present in a field, spread spatially, but prudence, time and labor constraints warrant whole field spraying and treating every plant the same way regardless of injury level. In case of insect pests, producers wait for the insect infestation to reach a treatment threshold and then spray the entire field using a linear array of nozzles to apply an excessive, uniform, and continuous spray pattern. The decision is based on a field "average"--some plants have reached an injurious level, while others are pest-free. Under several management scenarios, only a small fraction of areas receives a justified amount of pesticide, while other areas lose yield due to delayed timing and damage by pests, and remaining areas receive a superfluous spray application when there is no pest presence. The decision to treat an infestation is largely based on an "entire field mentality," which is mostly out of necessity and a lack of automated resources or technologies to streamline decision making with application needs. Automation provides a viable and marketable solution; however, the development of robotic technology for insecticide application has not been explored primarily due to unavailable camera vision capabilities to locate pest incidence and severity within a complicated crop canopy. There is a demand for an intelligent application system designed to accurately spray on infested canopy but only where infestations are present. Lack of automation research in pesticide application is a crucial problem for the long-term food production goals and environmental stewardship goals and sustainably intensifying production to meet a growing population. Quite simply, current manual labor costs are too high and modern spray technologies are too inaccurate. It is imperative that we seek ways to use to technology to address the grand challenge of global food production.Our long-term goal is to reduce the high-value pesticide usage on crops using an intelligent autonomous spray system to conduct site-specific application based on critical pest populations on select plants. The overall objective of this proposal, which is key to achieving our long-term goal, is to develop a state-of-the-art autonomous system to sense, identify, and manage pest for all crop growth stages. This objective is based on the hypothesis and preliminary results (Sec 4.) that computer vision can be used to detect pest incidence and severity on a highly spatial basis and be managed using a machine-vision system to site-specific spray insecticides. Successful achievement of this objective will help: 1) apply pesticide only on critically infested plants by eliminating pesticide application on soil and plants not requiring application; 2) reduce overall input cost (less insecticide and labor costs) but increase yield potential due a more timely decision to spray only infested plants; and 3) reduce pesticide effects on the environment by reducing exposure of these management tactics on beneficial organisms, thus promoting a more integrated and balanced approach to pest management. Our proposed system will be fully scalable to train computer vision system to sense, identify and geo-locate insect pests. We envision other functions like sensing of plant health status (disease, nutrient deficiencies, etc.,) being integrated into future interactions. The proposed research creates a foundation for more accelerated evolution in technology adoption, including future development and deployment of multi-robot mission architecture to execute whole field operation to harness economic and agronomic advantages. Our current proposal is foundational to the deployment of such architectures.

Animal Health Component

25%

Research Effort Categories

Basic

50%

Applied

25%

Developmental

25%

Classification

Knowledge Area (KA)	Subject of Investigation (SOI)	Field of Science (FOS)	Percent
402	1520	2020	80%
211	1599	1130	20%

Knowledge Area
211 - Insects, Mites, and Other Arthropods Affecting Plants; 402 - Engineering Systems and Equipment;

Subject Of Investigation
1520 - Grain sorghum; 1599 - Grain crops, general/other;

Field Of Science
2020 - Engineering; 1130 - Entomology and acarology;

Keywords

computer vision

smart chemical spraying

insect scouting

control

multi-robot: field robotics

agriculture

Goals / Objectives
Our long-term goal is to reduce the high-value pesticide usage on crops using an intelligent autonomous spray system to conduct site-specific application based on critical pest populations on select plants. The overall objective of this proposal, which is key to achieving our long-term goal, is to develop a state-of-the-art autonomous system to sense, identify, and manage pest for all crop growth stages. This objective is based on the hypothesis and preliminary results (Sec 4.) that computer vision can be used to detect pest incidence and severity on a highly spatial basis and be managed using a machine-vision system to site-specific spray insecticides. Successful achievement of this objective will help: 1) apply pesticide only on critically infested plants by eliminating pesticide application on soil and plants not requiring application; 2) reduce overall input cost (less insecticide and labor costs) but increase yield potential due a more timely decision to spray only infested plants; and 3) reduce pesticide effects on the environment by reducing exposure of these management tactics on beneficial organisms, thus promoting a more integrated and balanced approach to pest management. Our proposed system will be fully scalable to train computer vision system to sense, identify and geo-locate insect pests. We envision other functions like sensing of plant health status (disease, nutrient deficiencies, etc.,) being integrated into future interactions. The proposed research creates a foundation for more accelerated evolution in technology adoption, including future development and deployment of multi-robot mission architecture to execute whole field operation to harness economic and agronomic advantages. Our current proposal is foundational to the deployment of such architectures.To accomplish our overall objective, we will pursue the following four aims with associated sub-tasks:Develop an imaging and analysis system for rapid pest identification to provide quantitative pest severity and incidence at high spatial resolution. Sub-tasks: 1) efficient algorithms for pest identification and quantification; 2) On-the-go pest severity and incidence maps; and 3) decision support to develop site-specific, on-target prescription application.Develop a site-specific on-target liquid application based on decision derived from imaging system. Sub-tasks: 1) Develop and systematically evaluate pulse width modulated individual nozzle control liquid application system with scouting decision in the feedback loop; and 2) install an accepted system on the ground autonomous platform to conduct experimental validation of system response, accuracy and locational accuracy of spray on infested plantDevelop optimal autonomous vehicle field navigation for real-time sensing, identification and management of insect pests. Sub-tasks: 1) refine current preliminary vehicle designs for efficiency and application; and 2) develop routing algorithms to optimize autonomous coverage of crops as well as to mark insect boundaries to collaborate with larger spray robotic vehicles.Quantify the effect of altering pest population dynamics through targeted insecticide applications: sugarcane aphid in sorghum as a case study. Sub-tasks: 1) model the spatial and temporal impact of source colony recruitment of winged aphids and plant-to-plant dispersal under controlled (greenhouse) and natural (replicated, small plot trials) conditions; and 2) evaluate field-level impacts of simulated and project prototypes on aphid population dynamics using commercial production fields.

Project Methods
Develop an imaging and analysis system for rapid pest identification to provide quantitative pest severity and incidence at high spatial resolution: Data generation and preprocessing. The success of deep learning heavily depends on large annotated datasets; however, there is currently no available labelled dataset for sugarcane aphid detection and recognition to our knowledge. In our previous study, the team has already collected and labelled around 200 images. The annotation is tedious and time consuming, and requires the understanding of domain knowledge and skills. For this aim, we will collect and annotate thousands more images. We propose to preprocess and augment existing and new datasets to generate more meaningful training data for the system. End-to-end learning framework for pest detection and recognition. We are proposing a framework that combines appearance and statistical features of objects to solve the actual challenge presented by various clusters with multiple scales and anomaly distribution of small objects.Low-resolution and small object recognition via transfer learning. In practice, an aphid colony may only occupy a small area in the original captured images; thus, we propose to sample the image into small patches.The basic intuition is to exploit the discriminative feature-representations of already learned from one task, then fine tune the network to adapt to the new problem via knowledge transfer or transfer learning.Develop a site-specific, on-target liquid application based on decision derived from imaging system: Site-specific spray application system design. Two solid-tube booms with three electronically actuated nozzles will be fabricated for left and right side of the autonomous vehicle. The boom height will be 1.5 m and nozzle will be placed at 0.5 m spacing, starting from 0.2 m of lower end of the boom. A real-time kinematic global positioning system (RTK-GPS) and a color camera mounted on the leading boom will be used to swiftly spray chemical on target plants. A look ahead feature will be built within the control system to initiate spray and timing of the spray will be derived from vehicle speed. The nozzle spray rate will be controlled using a PWM signal, and data collected will be real-time shared using a CAN controller with the master station using a miniature mesh 900 MHz spread spectrum wireless modem. Laboratory and field validation of spray system. A setup of five artificial plants representing sorghum features will be setup in the lab. The application sub-system will include: 1) vision system to accurate align plants; 2) vehicle travel system to derive spraying time of the left and right boom; 3) spray system response time of the nozzle actuation; and 4) control system to accurately send control commands, record data and relay to the master. The system operation will be considered successful if the vision response, spray actuation of the left and right boom time, time for which booms stay actuated and no spray occurs outside the bounds of the plant canopy spread. Additional tests similar to the laboratory will be conducted in actual field conditions.Develop optimal autonomous vehicle field navigation for real-time sense, identify and manage insect pests: The first sub-task will be to refine the scout design. The first refinement will be to develop a sensor suite that will adequately allow the vehicle to navigate the lane without contacting any plants. Past work has used imaging systems, or laser rangefinders.The second sub-task focuses on the development of a full system control scheme. The initial routing of the scout vehicle will be a simple "mission planner" approach where the scouts follow a simple full field coverage plan. This plan will be executed using waypoints transmitted from the main computer to the scouts under local control to stay in the crop row. These scouts will be imaging plants as they traverse and will be in constant contact with the central computer sending telemetry and any insect-sighting event. The SPRAY vehicle also has the pest image analysis capability and will be used as a means to find further insect incidences.Altering pest population dynamics through targeted insecticide applications using sugarcane aphid in sorghum: Understanding effects of source colony recruitment of winged aphids and plant-to-plant dispersal. Population dynamics is a result of quantifying the birth and death rates for a given population and understanding the level of immigration (adding outside individuals to a population) and those leaving to find more suitable host plants. We propose conducting a series of greenhouse and field experiments to quantify the effects of founding aphid colonies on population growth. We will use protein marker studies to release marked aphids and assess landing rates and patterns of colonization under controlled conditions (greenhouse) and "real-world" (small sorghum field plots, 4 rows x 12 m with a 72 cm row-spacing) scenarios. Field-level effects of removing plants at damaging aphid levels. Larger field studies (Fig. 10) will build on results from greenhouse studies and will measure the effect of removing sugarcane aphid founders from established field plots using a simulated SIMPL approach during the first two field seasons.

Progress 12/01/22 to 11/30/23

Outputs
Target Audience:The target audience was the industry, including robotics platforms, GPS, communication, system integration, startups, industry professionals, service providers, graduate students, high school students, teachers, and faculty. Other target audiences include policymakers and university administrators. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided?The following training and professional development opportunities Techniques to select cameras to develop field-ready imaging systems for consistent crop images The team members learned about identifying aphid colonies and how to label the images for image analysis approaches. Use of ROS 2 and micro ROS to develop control and navigation, including integration of different sub-systems Strategies to integrate sub-systems in order to realize a end-to-end sense identify and manage platform. How have the results been disseminated to communities of interest?Results were disseminated through participation in departmental events, international conferences, and one-to-one interaction with stakeholders. What do you plan to do during the next reporting period to accomplish the goals? Testing and validation of computer vision system Quantify aphid detection accuracy using the develop models Testing integrated system accuracy to identify and real-time spray the plant with aphids Develop and test algorithms for autonomous row-navigation

Impacts
What was accomplished under these goals? Develop a computer vision system and integrate the image analysis algorithm with the computer vision system; conduct test for system validation The images of aphids in sorghum plants were obtained from Co-PI Dr. Guanghui Wang for training a real-time pest detection model for aphids. Various segmentation and detection models were tested to optimize aphid identification and quantification. Among them, U-Net segmentation model demonstrated superior performance in detecting the pest incidence and delineating aphid regions compared to other models. NVIDIA Orin (8-core, ARM processor, 32-GB RAM, GPU) is used as the computational backbone for the robotic platform to satisfy real-time processing needs. At the same time, the Robotic Operating System (ROS) is the middleware to integrate and ensure seamless communication among subsystems. Two different cameras: Basler acA 1920-40 gc (high-resolution imaging) and Intel Real Sense D 457 (providing depth information, enabling 3D perception) were mounted on the front side of the rover with the help of 3D printed camera mounts and extrusion. The trained U-Net Segmentation model was deployed on the computing platform using the NVIDIA Isaac ROS hardware-accelerated pipeline. The initial run using the dataset yielded promising results, setting the stage for real-world deployment. Additionally, the plant employed April tags as aphid proxies to fine-tune camera angles and achieve high spatial resolution. This will help guide our cameras to capture aphids with utmost precision. Moreover, an algorithm for targeted spraying is being developed to support decision-making based on aphid segmentation. Initial results showed more than 65% accuracy. Develop a plan to integrate with spray system using ROS-2 The goal of developing a plan to integrate the spray system with ROS 2 was accomplished. This was achieved by creating ROS nodes within the micro-ROS system that publish data from the sprayer system's sensors, which include liquid pressure, pesticide tank level, and liquid flow. Additionally, a node was developed within the same micro-ROS system that controls the sprayer solenoids. These ROS nodes, residing in the micro-ROS system, are crucial for integrating the entire system. Given the physical distance between the sprayer sensors and the rover's main computer, an NVIDIA Jetson AGX Orin, a microcontroller was decided to place near the sensors. This microcontroller, specifically an STM32, interfaces with the sensors and the solenoid drivers. It uses the micro-ROS framework, which houses the ROS nodes, to interface with Orin, allowing for seamless integration with the ROS 2 system running on the main computer. CAN FD was chosen as the transport layer between the microcontroller running micro-ROS and the Jetson Orin. Thanks to the micro-ROS framework, it was possible to develop ROS nodes on the microcontroller for sensor data publication and solenoid control are now fully integrated into the ROS2 system running on the Jetson Orin. This integration allows the main computer to have full control over the sprayer system. As a result, now it is possible to develop a ROS 2 controller node that operates the sprayer based on feedback provided by the camera system, which is also implemented with ROS 2. This accomplishment marks a significant step forward in the project. Integrate a CAN-based spray system with the developed autonomous platform and validate the functioning of the system for desired application accuracy The control system of the sprayer consists of an Electronic Control Unit (ECU), Drivers, and STM-32 as a microcontroller. The micro-Robotic Operating System (Micro-ROS) is the agent between the computing platform and the microcontroller. All sensors related to the spraying system, such as pressure transducer, flowmeter, and water level, are integrated with the help of the microcontroller. The segmentation algorithm running on the computing platform will detect the aphid in the plant and, in turn, send a signal to the microcontroller to activate the solenoid valve attached to the nozzle via ECU. The entire system is based on CAN communication protocol. Pulse Width Modulation (PWM) technology is utilized in the spraying system to ensure consistent system and nozzle pressure, even when operating under variable conditions. In the initial experiment, the duty cycle response was analyzed in relation to the command and response time for the ON/OFF valve operation. A pressure sensor (M3234-000005-100PG) was positioned in the delivery line and the nozzle body, linked to an analog-to-digital converter and connected to an STM32 microcontroller. Analog Discovery was employed to measure solenoid voltage and clearly visualize the signal and response. Results indicated application pressure within 5% of desired for operating conditions. Develop strategies to integrate all sub-systems using ROS-2 platform The goal of developing strategies to integrate all subsystems using the ROS-2 platform was successfully achieved. The sprayer system, computer vision system, and navigation system were all implemented with ROS 2, resulting in a comprehensive integration of all subsystems. This integration allows all subsystems to access data published by other subsystems and request actions from them. A key outcome of this integration is the development of a spray controller node, as discussed before. This node requires real-time aphid detection data from the computer vision system, as well as the latency from the camera image sample to DNN model inference results and data from the navigation system, such as velocity. It also requires control of the sprayer solenoids. Thus, integrating all subsystems using the ROS-2 platform has enabled a more efficient and coordinated operation of the entire system. This accomplishment represents a significant advancement in the project. Implement additional design modification in the robotic platform based on learning from 2022, and calibrate all systems Our robotic platform's development was guided by a strategic focus on enhancing its operational efficiency and adaptability to complex environments. This comprehensive overhaul encompassed significant upgrades to the motor system, power supply, and motor controllers and the structural integration of these components, culminating in the creation of a sophisticated digital twin model to facilitate advanced calibration and simulation activities. Motor System Enhancement The core of the modifications involved the replacement of the original brushed DC motor with four high-efficiency brushless DC motors. Each of these new motors boasts a power rating of 450 W and operates at 24 V, featuring a brushless configuration that inherently reduces maintenance requirements and increases lifespan. The motors are equipped with a gear ratio of 20:1, optimizing torque delivery and overall performance. Custom fittings were designed and installed to integrate these motors, with the wheelbase's center distance extended to 18 inches to accommodate the new setup, enhancing the robot's stability and maneuverability. Controller Upgrades Concurrently, the transition from roboclaw motor controllers to odrive motor controllers marked a significant upgrade in the control system's capability. Odrive controllers, specifically engineered for brushless DC motors, support a wide range of input voltages (12 to 50 V) and can handle up to 2000 W of peak regenerated power. These controllers are distinguished by their inclusion of built-in encoders and compatibility with CAN communication protocols, enabling precise and independent control of each motor, thereby significantly enhancing the robot's operational precision and flexibility.

Publications

• Type: Journal Articles Status: Published Year Published: 2023 Citation: Cheppally, R. H., A. Sharda, and G. Wang. (2023). Seed Localization System Suite with CNNs for Seed Spacing Estimation, Population Estimation, and Doubles Identification. Submitted to Smart Agricultural Technology, 4 (2023) 100182. https://doi.org/10.1016/j.atech.2023.100182.
• Type: Conference Papers and Presentations Status: Accepted Year Published: 2023 Citation: Shende, K., A. Sharda. 2023. Wireless data communication system for multiagent autonomous farming applications. Presentation no. 2301265. ASABE-AIM, July 09-12th, 2023. Omaha, Nebraska.
• Type: Conference Papers and Presentations Status: Accepted Year Published: 2023 Citation: Cheppally, R. H., and A. Sharda. (2023). Framing multi-Robot Pesticide Spraying As Reinforcement Learning. Presentation no. 2301546. ASABE-AIM, July 09-12th, 2023. Omaha, Nebraska.

Progress 12/01/21 to 11/30/22

Outputs
Target Audience:The target audience was the industry, including robotics platforms, GPS, communication, system integration, startups, industry professionals, service providers, graduate students, high school students, teachers, and faculty. Other target audiences include policymakers and university administrators. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided?The following training and professional development opportunities Techniques to select cameras to develop field-ready imaging system for consistent crop images Introductory learning of ROS-2 for robotic system navigation and sub-system integration CAN systems for integration with robotic systems The team members learned about identifying aphid colonies and how to label the images for image analysis approaches. How have the results been disseminated to communities of interest?Results were disseminated through participation in departmental events, international conferences and one-to-one interaction with stake holders.? What do you plan to do during the next reporting period to accomplish the goals? Develop a computer vision system and integrate the image analysis algorithm with the computer vision; conduct test and validation and develop a plan to integrate with spray system using ROS-2 Integrate a CAN based spray system with thedeveloped autonomous platform, and validate the functioning of the system for desired application accuracy Develop strategies to integrate all sub-systems using ROS-2 platform Implement additional design modification in the robotis platform based on learning from 2022, and calibrate all systems Develop basis navigation and control for robotic vehicle.

Impacts
What was accomplished under these goals? For year 2022, major accomplishements were achieved in the Aim - 2 and 3 Aim-2 A pulse width modulation (PWM) based robotic liquid application system developed in 2021 has the potential to apply the correct amount of pesticides, while providing maximum coverage and sustainably managing insect pests in row crops. Utilizing the developed robotic spraying system a spray coverage study was conducted using water-sensitive cards under different emitter orientations, application rates, and platform movement strategies. The water-sensitive cards were scanned and processed using MATLAB to quantify the percentage coverage in many sampling locations. The configuration with 15 GPA and regular pass in both 0° (0BCT) and 45° (4BCT) provided the highest mean deposition of 17.33%; both provided the maximum coverage in the high canopy height than the medium and lower. In configuration 0BCT, the spray coverage on the top, medium, and bottom zone was 30%, 22%, and 13% in the inner and 22%, 9%, and 8% in the middle canopy region. 4BCT provided 25%, 22%, and 17% in the inner and 16%, 15%, and 9% in the middle canopy region on the top, medium, and lower heights. It shows that the 4BCT configuration provided more consistent coverage across the sampling locations with minor variance. In all test case scenarios, the spray coverage was higher in the inner canopy region near the corn stalk than in the middle of the plant leaves. In both 0BCT and 4BCT configurations, there was around 62% of total coverage in the inner and 38% of the total in the outer. Also, the spray deposition of up to 56% of the total on the other side of the crop row shows a significant penetration capability of the spraying system. This study quantified the spray coverage at different plant locations under two separate emitter mounting configurations, two application rates, and two sprayer movement strategies. This research aimed to evaluate the performance of a robotic spraying system developed especially for row crops. The test was conducted in a greenhouse environment simulating the actual crop canopy structure with the minimum effect of external factors such as wind speed and temperature. Water-sensitive papers were analyzed using the image processing toolbox in MATLAB to quantify the spray coverage. The spray coverage was higher in the top canopy height followed by medium and low under 0° nozzle orientation. However, it was higher in the medium canopy region followed by top and bottom in 45° emitter orientation. The overall coverage was better while using 45°configurations as compared to 0°. As expected, the double pass strategy provided significantly higher mean coverage at all plant heights and all eight configurations. In the dual-pass method, both 0° and 45° provided higher spray coverage in the 15 GPA application rate compared to 12 GPA. In the lateral canopy regions, the spray deposition was much higher near the stalk compared to the middle of the plant leaves in all test case scenarios. The penetration data shows up to 56% of the total coverage on the leaves present at the other side of the stalk, suggesting the capability of the spraying system to penetrate the canopies and deposit comparable spray droplets on both sides. Aim - 3 Different camera and lens combinations available from Basler were studied to develop a prelimnary computer vision system. Various use case scenario criteria were considered when shorliting the desired system which included distance from the robotic vehicle, area of interest, resolution, image data, and easy integration with overall system. Since the overall robotic system was still under development, a separate study to localize seeds was conducted to develop computer vision techniques required for implementation. Currently, seed placement information for evaluating planters is obtained by tedious manual methods like using a pogostick or a ruler. To obtain more information and reduce the time required, there was a need for an automated system. Therefore a study was conducted with a goal to design such a system with the use of GPS and seed detection. A 12-row planter was instrumented with cameras and a GPS. A Basler (acA1920-40gc, Basler AG, Ahrensburg, Germany) color camera was selected for image acquisition of seeds in the seed trench. The camera pro- vided 1920x1200 pixel resolution (2.3 MP), 42 frames per second (FPS), and gigabit ethernet (GigE) connectivity. A 10 m GigE cable (2000028341, Basler AG, Ahrensburg, Germany) was selected to connect the camera (RJ45) to the data acquisition system (RJ45). The camera body was paired with a ruggedi- zed 12.5 mm fixed focal length Kowa (LM12HC-V, Basler AG, Ahrensburg, Germany) lens. A laser line scanner (OXH7-Z0500.HI0660.VI, Baumer Ltd., Southington, USA) with a measurement range of 100-500mm from the sensor surface; measuring frequency of 1520 Hz, measurement resolution of 4 to 25 µm; and measuring width range of 13 to 66 mm to provide surface profile characteristics was selected for seeding (trench) depth measurement. The distance estimation process can be separated into two steps 1) Seed Detection: Seeds are localized using deep learning based object detection algorithms, and 2) Distance Estimation: Distance estimation is done by filtering seen seeds and by using gps data. The planter was operated at planting speeds of 9.66 kmph and 12.87 kmph with a seed population of 74,131 and 86,4868 seeds per hectare respectively and seed spacing was measured manually using a ruler. YOLOR-P6, YOLOR-CSPX, YOLOX-S, YOLOX-M, YOLOX-L, YOLOX-TINY, and YOLOV4 were trained to detect seeds on a different dataset. An algorithm was designed to estimate the distance between two consecutive seeds by filtering out old detections from the detector with the use of the IOU metric and GPS stream. The seed spacing estimator was evaluated on Jensen-Shannon Divergence (JSD), mean, standard deviation, ?Count, and RMS metrics. Results indicate that the developed system along with the algorithm was able to perform better at 9.66 kmph with JSD distances of 0.223 and 0.244 (74,131 and 86,4868 seeds per hectare respectively) compared to 0.298 and 0.258 (74,131 and 86,4868 seeds per hectare respectively) at 12.87 kmph. Moreover, this algorithm was able to reduce the total time taken to detect seed and estimate seed spacing information from 2hrs by manual method to 1 min 14 seconds using YOLOR-CSPX.

Publications

• Type: Journal Articles Status: Awaiting Publication Year Published: 2023 Citation: Cheppally, R. H., A. Sharda, and G. Wang. (2023). Seed Localization System Suite with CNNs for Seed Spacing Estimation, Population Estimation, and Doubles Identification. Submitted to Smart Agricultural Technology.
• Type: Conference Papers and Presentations Status: Accepted Year Published: 2023 Citation: Cheppally, R. H., and A. Sharda. (2023). Framing multi Robot Pesticide Spraying As Reinforcement Learning. Presentation no. 2301546. ASABE-AIM, July 09-12th, 2023. Omaha, Nebraska.
• Type: Conference Papers and Presentations Status: Accepted Year Published: 2023 Citation: Shende, K., A. Sharda. 2023. Wireless data communication system for multiagent autonomous farming applications. Presentation no. 2301265. ASABE-AIM, July 09-12th, 2023. Omaha, Nebraska.

Progress 12/01/20 to 11/30/21

Outputs
Target Audience:The target audience was specifically industry including robotics platform, GPS, communication, system integration; startups; industry professionals; service providers; graduate students; high school students; teachers; and faculty. Other target audiences include policymakers and university administrators Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided? Nothing Reported How have the results been disseminated to communities of interest?Results were disseminated through participation in departmental events, international conferences and one-to-one interaction with stakeholders. What do you plan to do during the next reporting period to accomplish the goals?For the next reporting cycle, the team anticipates: Develop a computer vision system and integrate the image analysis algorithm with the computer vision; conduct test and validation and develop a plan to integrate with spray system The developed autonomous platform to maneuver within planted sorghum crop rows would be integrated with a basic control system for manual maneuverability of the platform in the lab and in the field to test liquid application and computer vision system Conduct field tests to evaluate the uniformity of spray coverage with selected nozzle orientation, nozzle types, and duty cycles. These results will be utilized to develop autonomous platform travel routines to maximize efficiencies.

Impacts
What was accomplished under these goals? Lab and field-scale testing was conducted to evaluate vehicle one testing a robotic vehicle design in field conditions. The results were used in the next design iteration (Figure 1) that will better fit the sorghum environment. The new design is 20 inches wide and 44 inches long. It also uses two 12V 54Ah Li-ion batteries that will run in series to make 24V. Extruded metal ends will be used for better sensor and mast mounting. The wheel and tire combination, of the first iteration, proved to work well in the field. New DC motors will be incorporated in this iteration that are more compact, lighter, and more efficient. The new platform will be used to integrate the liquid application system. Figure 1. The re-designed autonomous platform for greater ease to maneuver within the sorghum crop row 2. The intelligent liquid application system was designed utilizing Teejet PWM nozzle bodies with solenoids, flow control components, and a 20-gallon liquid tank (Figure 2). Numerous structural designs were considered while developing the liquid application system, integrating it with the rover, and providing different functionalities. The chemical tank was mounted with an aluminum cradle (72.39cm L × 39.37cm W × 19.05cm H) and three bands and fixed on the top of the robotic platform to protect the tank. The booms were attached on a rectangular frame and fixed at the rear side of the platform, where the booms can slide along the frame width. This functionality was critical to adjust the spray overlap according to the row spacing. The vertical positioning of the boom was made adjustable to use the sprayer in different crops and throughout a crop cycle. Additionally, the rectangular frame was designed to be forward-folded to protect the booms from possible damages during transportation. Figure 2. Hydraulic circuit design for the liquid application system, One of the significant advantages of implementing PWM spray controllers is maintaining a constant system and nozzle pressure under varying operating conditions. Hence, it was essential to measure the pressure at few separate locations and understand the functionality and capability of the developed system during operation. For all tests, a pressure sensor with a less than 1 ms response time (PTC-2, PCB PIZOTRONICS, Depew, NY) was installed at the sprayer manifold, and four similar pressure sensors were mounted right before the nozzle tips at four nozzle bodies. During all tests, the solenoids were operated at 10 Hz, and pressure data were recorded at 1000 Hz using NI cRIO, NI 9221 I/O module, and LabVIEW 2019. The pressure data was recorded in the text (.txt) format for about 20 seconds and was imported into the excel sheet for further processing. The sprayer was developed for a vertical row crop environment with a small effective spray width (Figure 3). So, it was necessary to operate the solenoids at a lower duty cycle (< 40%) to maintain a target application rate for an available range of commercial nozzle tips. Therefore, 40% duty cycle was chosen to test and analyze the sprayer performances even though the sprayer was developed for site-specific spot spraying, there might be the case during actual field operation when all six nozzles will get activated based on the infestation level. Despite the number of energized nozzles, the sprayer should be able to maintain uniform system pressure to maintain a constant application rate and droplet size. So, it was essential to understand the pressure consistency despite varying the duty cycle and number of active nozzles. Hence, the system pressure data were collected at 1000 Hz and averaged to get 100 ms (10 Hz) data and plotted for 20 seconds to see the pressure consistency at two duty cycles (20% and 40%) and three different numbers of active nozzles (1, 3, and 6). Figure 3. Structural Design of Robotic System Another unique feature of the PWM technology is its low response time. The solenoid should hold the product at the target pressure during the OFF state and apply the chemical at the desired pressure when it gets energized. For example, at 40 PSI, the pressure should sharply increase and reach the targeted pressure (40 PSI), remain constant for the duration of 40 ms, and sharply drop down to zero PSI. So, the signal input to the solenoid was measured at the 1000 Hz frequency and plotted for one solenoid cycle (100 ms) together with the nozzle pressures to evaluate the functionality of the PWM system. Measurement was done at the bottom nozzle of the right boom by turning ON one, three, and all six nozzles at 40% duty cycle to compare the nozzle pressure stability with the target pressure under three different active nozzles (1, 3, and 6). 3. The images of sorghum plants with aphids were labeled using a custom-designed toolbox. The labeled images are being used by collaborating faculty Dr. Guanghui Wang to develop image analysis approaches, and develop software to be integrated with the intelligent sprayers,

Publications

• Type: Conference Papers and Presentations Status: Published Year Published: 2021 Citation: Pokharel, P., A. Sharda, D. Flippo, G. Wang, and B. McCornack. 2021. Design, Development and Control System Evaluation of a Site-Specific On-Target Liquid Application System for an In-Row Autonomous Platform 2101053. ASABE-AIM, July 12-16th, 2021.
• Type: Journal Articles Status: Submitted Year Published: 2022 Citation: Pokharel, P., A. Sharda, D. Flippo, and B. McCornack. 2022. Design, development, and evaluation of a site-specific liquid application system for a robotic platform. Submitted to Computers and Electronics in Agriculture.

Progress 12/01/19 to 11/30/20

Outputs
Target Audience:The target audience was specifically industry including robotics platform, GPS, communication, system integration; startups; industry professionals; service providers; graduate students; high school students; teachers; and faculty. Other target audiences include policymakers and university administrators. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided?The following training and professional development opportunities Techniques to select cameras to develop field-ready imaging system for consistent crop images The team members learned about identifying aphid colonies and how to label the images for image analysis approaches. How have the results been disseminated to communities of interest?Results were disseminated through participation in departmental events, international conferences, and one-to-one interaction with stakeholders. What do you plan to do during the next reporting period to accomplish the goals?For the next reporting cycle, the team anticipates: Images collected from the sorghum crop containing aphids will be labeled and quality images will be provided for training image analysis techniques The developed autonomous platform to maneuver within planted sorghum crop rows would be integrated with a basic control system for manual maneuverability of the platform in the lab and in the field to test liquid application and computer vision system The liquid application system will be developed with the capability to actuate nozzles based on image analysis algorithm feedback to selectively spray on;y on targeted areas.

Impacts
What was accomplished under these goals? An autonomous platform developed was a 4-wheel drive, differential steering platform capable of bidirectional movement between the crop rows. The payload capacity was sufficient to carry 75.7-liter pesticides and other required sub-systems. To integrate the spray booms, T-slotted framings were mounted at the back of the platform. Platform Control: One of the goals of the proposed study was to develop a control system to operate the vehicle at different speeds and directions. An incremental encoder (model 260 - incremental encoder, Encoder Products Company, Sagle, ID) that outputs 1024 cycles per revolution of the motor shaft having measurement accuracy within 0.01° mechanical was integrated to measure the vehicle speed. A 2×60 dual motor driver (Sabertooth 2×60, Dimension Engineering, Hudson, OH) was deemed sufficient and was chosen to power the DC brushed motors of the vehicle. The selected motor driver could continuously supply up to 60A per channel, and it came with thermal and overcurrent protection. It had two terminals (B+ and B-) to supply the power from the battery, motor terminals (M1A/M1B, M2A/M2B) to supply power to the motors and signal input terminals (SP1 and SP2) to control the motor driver. A closed-loop proportional-integral-derivative (PID) control system was designed to sense the speed of the motor and adjust it based on the desired or the reference signal. To implement PID, national instruments myRIO 1900 and encoders were programmed with LabVIEW software. A gravity MOSFET power controller was used to start or cut off the power supply to the motor driver battery terminal. The developed software read the encoder counts and computed the error value from any difference between the actual motor speed measured by the encoder and the set desired speed. Finally, the PID program converted that error value into a command (PWM) signal that adjusted the voltage supplied by the motor terminals to run the motors at the set speed. A liquid application system was designed and developed to conduct accurate, site-specific spray on either side of the autonomous platform. Two main tank selection criteria were the need to frequently fill the tank and the capability of the rover to carry the liquid. Besides the above two, corrosion-resistant, easy to fill and clean, tank profile, adequate openings for hydraulic lines, and fluid level markings were considered. Two solid booms were designed for the left and right sides of the robotic platform. The boom height was estimated using the approximate height of the main row crops such as corn and sorghum. Each boom had a height of 1.2 m, where each nozzle was placed at 0.4 m spacing, starting from 0.2 m of the lower end of the tube. When kept at the center of the 30-inch row-row spacing crop, those booms could provide 50% vertical spray overlap during broadcast application. However, the percentage overlap varies depending on the sideways boom placement and the crop row-row spacing. The most important part of a spraying system design was the pump selection. The pump should supply to fulfill the spray requirements, agitation requirements, and account for the pump wear (typically 20% greater capacity). It was critical to consider the construction, composition, sizes, and ability to withstand the chemical chemistries at the operating pressure while selecting hydraulic components. Therefore, the chemical-resistant polypropylene materials were used on the suction and discharge side of the pump to eliminate the chances of collapsing. A study conducted by Sharda et al., 2013 showed that the nozzle pressure varied between 7-20% of the target while operating a flow-based sprayer. It affects the droplet size distribution, drift potential, and ultimately the spray coverage and penetration in the crop canopies. However, the PWM technology markedly reduces the above drawbacks of a flow-based spray control system by maintaining the pressure within ± 5% of the target regardless of section control (Mangus et al., 2015). This technology allows individual nozzle shut-off control and turn compensation to maintain the desired application rate at constant pressure by changing the actuation pattern based on the duty cycle input. Therefore, solenoid valves from Teejet (115880-1-12-05) were mounted on the nozzle bodies. Numerous structural designs were considered while developing the liquid application system, integrating it with the rover, and providing different functionalities. The sprayer control and data acquisition system had to manage four primary components; a liquid pump, the PWM solenoids, servomotors, and the pressure sensors. Sprayer Control and Data Acquisition Setup. A LabVIEW program was developed to run on the cRIO to control the sprayer and collect the pressure data. The host VI was programmed to input the operating parameters (solenoid frequency, duty cycle, and data loop rate), compute the PWM signal to the servomotor, write the pressure data with the time stamp in a text file, and store the text file at a specified location. On the other hand, the FPGA VI was developed to switch the six solenoids, two servo motors, and a pump based on the host VI input. Also, the FPGA VI was programmed to continuously read the pressure data based on the defined sampling rate. The overall system (Hardware, cRIO, and LabVIEW) was tested by connecting to the laptop using a c type USB cable. Each component was switched ON and OFF to see the functionality and observed the incoming data stream on the LabVIEW front panel. Three GoPro cameras were mounted on a custom-made structure to simultaneously capture images of a plant from multiple angles. Three such systems were used to capture a large image dataset of sorghum field with aphid infestation of varying intensities.

Publications

• Type: Conference Papers and Presentations Status: Submitted Year Published: 2020 Citation: Pokharel, P., A. Sharda, D. Flippo, G. Wang, and B. McCornack. 2020. Design, Development and Control System Evaluation of a Site-Specific On-Target Liquid Application System for an In-Row Autonomous Platform
• Type: Other Status: Published Year Published: 2020 Citation: https://content.govdelivery.com/accounts/USDANIFA/bulletins/29819bf
• Type: Other Status: Published Year Published: 2020 Citation: https://content.govdelivery.com/accounts/USDANIFA/bulletins/29392ed

Progress 12/01/18 to 11/30/19

Outputs
Target Audience:The target audience was specifically industry including robotics platform, GPS, communication, system integration; startups; industry professionals; service providers; graduate students; high school students; teachers; and faculty. Other target audiences include policymakers and university administrators. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided? Nothing Reported How have the results been disseminated to communities of interest? Nothing Reported What do you plan to do during the next reporting period to accomplish the goals?For the next reporting cycle, the team anticipates: Cameras and mechanisms will be developed to capture images of sorghum crops with aphid infestation. Multiple camera systems will be developed for multiple student teams Collect a large number of images of aphids on the sorghum crop for labeling, and image processing algorithms Develop an autonomous platform that could maneuver with planted sorghum crop rows with computer vision and smart spraying integrated. Develop a basic architecture of an intelligent liquid application system

Impacts
What was accomplished under these goals? In 2019, graduate students were hired to start working on their respective objectives. Graduate students for all the goals were hired but hiring was staggered primarily driven by the right student availability and timing to join the program.

Publications