Source: UNIVERSITY OF ILLINOIS submitted to
Sponsoring Institution
National Institute of Food and Agriculture
Project Status
Funding Source
Reporting Frequency
Accession No.
Grant No.
Project No.
Proposal No.
Multistate No.
Program Code
Project Start Date
Mar 15, 2021
Project End Date
Mar 14, 2025
Grant Year
Project Director
Chowdhary, G.
Recipient Organization
2001 S. Lincoln Ave.
Performing Department
Ag and Bio Engineering
Non Technical Summary
This project seeks to transform agriculture by developing a novel, context-aware cyber-agricultural system that encompasses sensing, modeling, and actuation to enable farmers to respond to crop stressors with lower cost, greater agility, and significantly lower environmental impact than current practices. The technical objectives are to create better AI for individualized plant health estimation,to implement data-driven, multi-scale planning and reasoning, and to develop individualized sensing and actuation via autonomous robots with dexterous manipulators.
Animal Health Component
Research Effort Categories

Knowledge Area (KA)Subject of Investigation (SOI)Field of Science (FOS)Percent
Knowledge Area
402 - Engineering Systems and Equipment;

Subject Of Investigation
5310 - Machinery and equipment;

Field Of Science
2020 - Engineering;
Goals / Objectives
The COALESCE project seeks to transform CPS capabilities in agriculture by developing a novel, context-aware cyber-agricultural system that encompasses sensing, modeling, and actuation to enable farmers to respond to crop stressors with lower cost, greater agility, and significantly lower environmental impact than current practices. The technical objectives are to embed biophysics in machine learning for individualized crop modeling, to apply multi-modal information fusion and robust learning for individualized sensing, to implement data-driven, multi-scale planning and reasoning, and to develop individualized sensing and actuation via autonomous robots with dexterous manipulators. Modeling of biophysical conditions in reasoning models contributes to CPS research areas in data analytics, information management, and real-time systems. The individualized sensing thrust contributes to data analytics, information management, and autonomy research areas. Multi-scale reasoning and farm management based on individualized actuation contributes to the areas of control, data analytics, autonomy, networking, safety, and real-time systems research areas.This is a collaborative project between UIUC and Iowa State.The UIUC portion of this project will focus on the robotics part. Our primary goals are:- Data driven planning and control for multi-scale reasoning- Indivizualized sensing and actuation via autonomous robots with dexterous manipulators
Project Methods
The key methods used in this project are from the realm of Cyber Physical Systems (CPS). The project will advance keyCPS science, technology and engineering. Our algorithms and tools will address foundational questions such as:Will machine learning models embedded with domain knowledge, physical, laws, invariances, and conservation principles lead to more data-efficient predictions? Can data fusion strategies combine robust and uniform abstraction of diverse data with minimum information loss? Will distributed robotic systems capable of treating biotic and abiotic stressors on an individual plant basis lead to fundamentally more sustainable and robust farming systems? Will learning based control and decision making lead to robust distributed autonomy and coordination in harsh, changing, and uncertain field environments? Will soft arms and manipulators with reinforcement learning based controllers lead to new mechanisms for controlling biotic and abiotic stressors requiring little or no chemicals?In summary, oureffort will contribute to the following core CPS research areas: Control, data analytics, autonomy, information management, networking, real-time systems, and safety. We note that while the notion of real-time may be slightly relaxed for Ag applications (compared to other CPS sectors such as aerospace), the large spatial scale of agricultureoperations still makes it time-critical, i.e., it demands fast sensing, information processing and actuation for timely mitigation of crop stresses over large fields.

Progress 03/15/21 to 03/14/22

Target Audience: The following target audiences were reached: 1. Researchers: This includes graduate students, undergraduate students, faculty, and postdoctoral associates in the performing team. It also includes other researchers who are not a part of this project but in the community, with whom we communicated through papers and presentations. These researchers were from diverse fields of agriculture, computer sciecne, and engineering. 2. Farmers: We interacted with a number of farmers to discuss with them technologies and potentials of mechanical weeding. 3. Press: We interacted with several journalists to help them communicate the potential of mechanical weeding with small robots 4. Stakeholders: We communicated with government and industry stakeholders about the potential of mechanical weeding agbots. This includes startup companies, more established companies, program managers in goverment funding organizations, and politicians. 5. Students: We incorporated material from this course into various undergraduate and graduate courses. We also invited and communicated with K-12 students who were interested in learning more about agriculture and the role of technology in agriculture. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided?Graduate students: Graduate students worked on key aspects of the research. They were provided opportunity and mentoringto lead papers and research activities. Undergraduate students: Undergraduate students helped with key research aspects and were provided research mentorship and opportunity to co-author papers. Postdocs: Postdocs were provided with training and mentoring to lead teams of researchers. In particular, Dr. Andres Baqueros is being trained to lead his own lab as he is interested in an academic position. Faculty: Faculty directed students and postdocs to drive research program objectives, and worked together to collaboratively solve problems that lie at the intersection of disciplines. The ASA/CSSA/SSSA International Annual Meeting, Salt Lake City, UT (ISU) American Society of Plant Biologists (ASPB) Biology Summit (Virtual) (ISU) Neural Information Processing Systems (NeurIPS) Conference (Virtual) (ISU, UIUC) North American Plant Phenotyping Network Annual Conference (Hybrid) (ISU) Association for the Advancement of Artificial Intelligence (AAAI) Conference (Virtual)(ISU, UIUC) RF Baker Plant Breeding Symposium (Hybrid) (ISU) American Geophysical Union Fall Meeting (OSU) Our project team members took part in demonstrations/talks including: Plant Phenomics and Artificial Intelligence to Glean Information from Plant Sensing Technologies, Genetics, Genomics and Bioinformatics Symposium, ASAS Midwest Section Meeting, March 2022. (ISU) Role of Interpretable Machine Learning in Cyber-Agricultural Systems, International Conference on Digital Technologies for Sustainable Crop Production (DIGICROP), March 2022. (ISU) A Cyber-Physical Systems Approach to Agricultural Sustainability, Japanese Society of Agricultural Informatics (JSAI) Annual Conference, May 2021. (ISU) Application of Machine Learning and Artificial Intelligence in Plant Breeding, Tri-Societies Meetings, Salt Lake, City, UT (ISU) Soybean Breeding at ISU, Iowa Soybean Association, Ankeny, IA (ISU) Artificial Intelligence (AI), Human Cognition, and Ethics,CRAI-CIS Seminar Series, Department of Computer Science, Aalto University, Helsinki, Finland (GMU) Building an Infrastructure to Help Scientists Handle Big Data, Arizona Science (National Public Radio - Episode 297) Tucson, AZ (UofA) Coupling Vegetation Biophysics and Ecohydraulics for Improved Simulation of Land-Atmosphere Carbon-Water-Energy Exchange, American Geophysical Union Fall Meeting (OSU) Our project team members provided Education and Outreach opportunities including: Fundamentals of Plant Physiology- Online (ISU) Uni-High FarmBot Educational Outreach, AGORA Day, Champaign, IL (UIUC) COntext Aware LEarning for Sustainable CybEragricultural SystemsPoster presentation at ISU Day, Des Moines, IA(ISU) Additionally, students (including multiple women and underrepresented minority students) associated with the COALESCE project had the opportunity to participate in bi-weekly meetings to provide project update presentations to share their research findings. These opportunities provide valuable feedback on their progress and allow them to receive direction for further research and collaboration. How have the results been disseminated to communities of interest?The key results have been published in prestigious journals such as Plant Phenomics, IEEE Robotics and Automation Letters and presented at AI-Ag workshops at NeurIPS 2021 and AAAI 2022. All the papers are listed in products. PI Sarkar and Co-PIs have delivered multiple invited talks that included results from this project. In order to help the nascent Cyber-Ag research community grow and thrive, the COALESCE team has organized international workshops such as the Third International Workshop on Machine Learning for Cyber-Agricultural Systems (MLCAS) and the AI for Agriculture and Food Systems (AIAFS) workshop under the umbrella of THE AAAI 2022 conference. What do you plan to do during the next reporting period to accomplish the goals?Debuggin the Field Problem: Robotic insect monitoring and mitigation that includes the following objectives - Learn areas frequented by pests, Detect and distinguish harmful pests and Spray pests with pesticide locally on plants with robots. Visual Servoing: Our main goal will be to utilize cameras on the tip of the robot and on the base of the robot to navigate through obstacles. Networking for Field - CPS We expect to perform field experiments to answer the key questions raised through last year's analysis. Under-Canopy Cover-Crop Planting We expect to continue to work with industry to advance under-canopy cover-crop robot planting in the 2022 season.

What was accomplished under these goals? Formulation of the Debuggin the FieldProblem On average 20% of crops are lost due to pests. Farmers are unable to systematically identify, localize, and mitigate pests in a timely and cost-effective manner. A fleet of mobile ground robots, equipped with dexterous sensors and manipulators, are uniquely equipped to manage insect infestations under the canopy and increase plant yield. Currently, soft robotic arms lack precision and robustness for applications. A soft arm controller, based on a non-linear autoregressive network with exogenous inputs (NARX) architecture, is being designed and tested to solve this problem. Precision within tree cm has currently been shown in simulation and tested on the physical system. Improvements to improve this precision and add orientation tracking will soon be available. Chowdhary, Krishnan, and Arti Singh will work together on this problem. Visual Servoing for Agricultural Robots For soft continuum arms, visual servoing is a popular control strategy that relies on visual feedback to close the control loop. However, robust visual servoing is challenging as it requires reliable feature extraction from the image as well as accurate control models and sensors to perceive the shape of the arm, both of which can be hard to implement in a soft robot. We circumventthese challenges by presenting a deep neural network-based method to perform smooth and robust 3D positioning tasks on a soft arm by visual servoing using a camera mounted at the distal end of the arm. A convolutional neural network is trained to predict the actuations required to achieve the desired pose in a structured environment. Integrated and modular approaches for estimating the actuations from the image are proposed and are experimentally compared. A proportional control law is implemented to reduce the error between the desired and current image as seen by the camera. The model together with the proportional feedback control makes the described approach robust to several variations such as new targets, lighting, loads, and diminution of the soft arm. Furthermore, the model lends itself to be transferred to a new environment with minimal effort. Krishnan and Chowdhary are working together on this problem Control of Soft Continum Arms Soft continuum arms (SCAs) that are controlled by visual servoing (VS) present trade-offs between the camera range and tracking accuracy. Cameras placed at a distance (eye-to-hand) can observe a larger workspace area and the SCA tip, while a camera at the end effector (eye-in-hand) can more accurately survey the target. In this project, we present a hybrid eye-to-hand and eye-in-hand VS scheme to track a desired object in the SCA's worksapce. When the target is not in the field-of-view of the tip camera, hand-to-eye VS is implemented using a wide field-of-view camera on the soft robot's base, to servo the soft robot's tip to a feasible region where the target is expected to be seen by the tip camera. This region is estimated by solving an optimization problem that finds the best region to place the SCA assuming a constant curvature model for the SCA. When the target is seen by the tip camera, the system switches to a hand-in-eye controller that keeps the target in the desired image position of the tip camera. Experimental results on the popular BR^2SCA demonstrates the effectiveness of the hybrid VS scheme under practical settings that include external disturbances. Krishnan and Chowdhary are working together on this problem Networking for Field CPS Emerging autonomous farm applications have diverse requirements e.g. teleoperation and virtual walkthrough have low latency requirement, whereas data collection for precision agriculture requires high throughput. Unfortunately, the IoT network in autonomous farms has limited connectivity options and power constraints. Every task or data transfer utilizes limited resources and depletes power. It is useful to have a centralized traffic controller or scheduling system that schedules tasks such that it satisfies requirements of tasks while managing limited resources. We will start off by studying different applications and understanding workload requirements of these applications. Then we understand different connectivity options that are optimal for different applications. Given these connectivity options, a centralized traffic controller can schedule jobs from different applications over different options, as per the application requirements and power constraints. Mittal, Arti Singh and Chowdhary are working together on this problem Foundational Advances in Planning for High Degree of Freedom Robotic Manipulators Hauser and students continued to advance the foundations of planning and reasoning AI for control of high degree of freedom manipulator arms. These advances are designed to enable arms to plan more complex trajectories faster and more efficiently. The problems are inspired by difficult challenges in manipulation afforded by agricultural robotic CPS. Hauser is working on this problem Under-Canopy Cover-Crop Planting We developed algorithms and software for under-canopy cover-crop planting robots. We showed through numerical simulations that a team of fivesuch robots can cover an eightyacre field planting cover-crops in under fivehours when operating fully autonomously. We evaluated our algorithms for autonomy on robots created by EarthSense Inc., and demonstrated the feasibility of vision and LiDAR based autonomy for field operation. This has inspired further development at EarthSense for bringing this technology to farmers as a lower-cost option towards the farms of the future. These activities were conducted at the Illinois Autonomous Farm, where we also demonstrated the feasibility of autonomous control of under-canopy robots using vision sensors, by extending our work from Narenthiran et al. 2020 RSS (CropFollow). Chowdhary is working on this problem Feasibility of Mechanical Weeding In our latest paper published in the prestigious IEEE Transactions of Robotics, we proposed AgBots 3.0, an algorithmic framework that leverages prediction of weed emergence patterns to optimize robot paths so that fields can be covered optimally with the least number of robots. Our results demontrate through extensive numerical simulation that mechanical weeding robots can feasibly keep fields weed-free, and highlight ways in which cost can be reduced by minimizing the number of robots through smart planning. This work was continuation from Chowdhary and Adam Davis' earlier NSF-CPS project funded through NIFA.


  • Type: Journal Articles Status: Published Year Published: 2022 Citation: W. McAllister, J. Whitman, J. Varghese, A. Davis and G. Chowdhary. 2022. Agbots 3.0: Adaptive Weed Growth Prediction for Mechanical Weeding Agbots. In: IEEE Transactions on Robotics, vol. 38, no. 1, pp. 556-568, Feb. 2022, doi: 10.1109/TRO.2021.3083204.
  • Type: Journal Articles Status: Published Year Published: 2022 Citation: S. Kamtikar, S. Marri, B. Walt, N.K. Uppalapati, G. Krishnan and G. Chowdhary. 2022. Visual Servoing for Pose Control of Soft Continuum Arm in a Structured Environment. In: IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 5504-5511, April 2022, doi: 10.1109/LRA.2022.3155821.
  • Type: Conference Papers and Presentations Status: Published Year Published: 2022 Citation: S. Kamtikar, S. Marri, B. Walt, N. K. Uppalapati, G. Krishnan and G. Chowdhary. 2022. Visual Servoing for Pose Control of Soft Continuum Arm in a Structured Environment. In: RoboSoft Conference, and jointly in IEEE Robotics and Automation Letters. Edinburgh, Scotland, April 2022.