SALIENCY-DRIVEN ROBOTIC NETWORK FOR SPATIO-TEMPORAL PLANT PHENOTYPING

• Sponsoring Institution: National Institute of Food and Agriculture
• Project Status: TERMINATED
• Funding Source: AFRI COMPETITIVE GRANT
• Reporting Frequency: Annual
• Accession No.: 1010607
• Grant No.: 2017-67021-25965
• Project No.: IOWW-2016-07876
• Proposal No.: 2016-07876
• Program Code: A7301
• Project Start Date: Dec 15, 2016
• Project End Date: Dec 14, 2022
• Grant Year: 2017

Project Director
Bhattacharya, S.

Recipient Organization
IOWA STATE UNIVERSITY
2229 Lincoln Way
AMES,IA 50011

Performing Department
Mechanical Engineering

Non Technical Summary
The objective of this project is to build a network of ground robots that can collect multi-modal data in research farms for high throughput modular plant phenotyping. The robotic network will have the following capabilities (i) Navigate in a farm to collect data with minimal human intervention during operation (ii) Autonomous decision making i.e, it can take its own decisions for maximizing the value of information of the acquired data (iii) Scalable in terms of the size of the farmland (iv) Work in collaboration with humans to improve their situational awareness in multi-dimensional genome wide studies. Our approach will leverage opportunistic sensing, task partitioning and scout-task allocation, machine learning, and spatio-temporal importance map building, to enable resolution of the above science questions that cannot be addressed without the use of robotic systems.The outcomes of this research would benefit a broad spectrum of the agricultural community, from plant scientists to small scale farmers in developing countries to domestic large scale farming operations. Outcomes of this research are expected to be adopted by the agriculture industry (seed, chemical). Our broader outcomes extend to training the next generation of roboticists that understand and contribute to the societally critical problem of agriculture improvement. ). Additionally, it powers genetic studies by identifying the expression pattern of quantitative trait loci (QTL), understanding Genetic and Environment (GxE) interactions, and interactions across the temporal axis. This allows biologists and geneticists to focus efforts and resources on the identification of meaningful loci constantly across development stages and environments, and speed the discovery process while providing valuable insights into the gene functions. Further identification of the casual genetic variants for the QTLs gives the potential of improving cultivar through genome editing that only modifies the target gene and shortens the breeding cycle.

Animal Health Component

0%

Research Effort Categories

Basic

50%

Applied

40%

Developmental

10%

Classification

Knowledge Area (KA)	Subject of Investigation (SOI)	Field of Science (FOS)	Percent
201	1820	2020	80%
201	1820	1081	10%
201	1820	1080	10%

Knowledge Area
201 - Plant Genome, Genetics, and Genetic Mechanisms;

Subject Of Investigation
1820 - Soybean;

Field Of Science
2020 - Engineering; 1081 - Breeding; 1080 - Genetics;

Keywords

machine learning

multi-robot networks

plant phenotyping

Goals / Objectives
Motivation: Currently, the highest priorities for agriculture production are to sustainably meet the food/feed/fuel/fiber requirements of a growing population worldwide. Changes in consumption patterns -coupled with the rise in population growth and disposable income levels in emerging economies - require the development of sustainable agricultural production practices. Towards this goal, plant scientists, engineers, and statisticians are increasingly working to improve efficiency and reduce costs of production. Seed companies are exploring modern tools of genomics and phenomics for their cultivar development and chemical (pesticides) programs. Agricultural equipment companies, for their part, are deploying state-of-the-art sensor technology to develop precision agriculture and farm-specific solutions to improve disease, nutrient and weed management.Goals: Our long term vision is to foster close collaboration among roboticists, computational scientists and plant scientists to meet the demands of a growing population on a profoundly changing planet. Our team recognizes a transcending opportunity through the use of high throughput, robotic, precision phenotyping at the farm scale, integrated with image analytics using autonomous robotic networks to address the issue of increased crop production and sustainability. The tangible deliverables of the proposed work are: (i) An autonomous robotic network capable of collecting and processing data for current and futuristic plant research needs to utilize plant genetic diversity and increase genetic resistance to biotic and abiotic stresses (ii) Associated real-time image processing tools for improving the situational awareness of human researchers engaged in trait data collection including disease detection and scouting, (iii) Data-driven decision-making framework in the context of the high-dimensional (hyper spectral) trait collection, (iv) and a deployment of this framework to identify the genetic basis of four diseases that affect soybean production. These outcomes would benefit a broad spectrum of the agricultural community. The science outcomes of this research are expected to be widely adopted by the agriculture industry (Co-PI A.K. Singh is a plant breeder who has an extensive history of developing high performance cultivars). Our broader outcomes extend to training the next generation of roboticists that understand and contribute to the societally critical problem of agriculture improvement.Objectives: We intend to further build upon our promising preliminary work that leverages high throughput image data and machine learning to identify, characterize and predict disease type and spread in soybean.?Based on our initial efforts, we will develop a coherent and integrated multi-disciplinary platform for other important traits with the following objectives:(a) Development of an autonomous robotic platform fordata acquisition: Integrating disparate, heterogeneous image streams taken from unmanned ground system (UGS) and manned systems (MSS) using RBG, hyperspectral and multispectral imaging techniques.(b) Data curation and feature extraction: Leveraging recent advances in data storage and indexing to curate very large data-sets, and develop machine learning methodologies to extract features enabling disease identification and prediction.(c) Deployment and impact assessment: Develop and disseminate easy-to-use apps for disease detection for broad use by farmers, breeders, pathologists and agricultural and sensor technology manufacturing companies. Assess the social and economic impact and diffusion of these scientific advances among the intended stakeholders across the globe. This feedback process will be critical to educate and inform the predominately non-technical stakeholders (policy makers, subsistence farmers, other key agricultural value chain actors) of the transformative impact of this big-data driven approach for efficient as well as sustainable agricultural practices.(d) Expanding the scope of inference: We will integrate phenotyping with changing environment (the GxE interaction). This approach will lead to disease predictions in farmers and experimental plots.This project will eventually lead to application in other major agricultural crops of economic and social importance.

Project Methods
The central objective of this project is to build a network of ground robots that can collect multi-modal data in research farms for high throughput modular plant phenotyping.Challenges: The proposed robotic network faces several challenges associated with the data deluge due to the inherent nature of its applications. Unlike engineered systems, event (developmental stages or observations) in biological species are prone to significant spatio-temporal uncertainty. Any autonomous sensor platform deployed to capture the salient features in the spatio-temporal evolution of the system has to acquire frequent measurements to ensure that important events are not missed. Moreover, lack of proper models for the evolutions of botanical systems often leads to a data-driven approach to discover the unknown relationships between the genotype of the species and its phenotype. In such cases, a large training set helps to build a better prediction model of the underlying system which in turn requires more measurements. Therefore, the problem of data deluge is indispensable in such studies.Proposed Solution:In order to alleviate the issues associated with sampling at ahigh frequencyand resolution, we envision a hierarchical deployment of a robotsto capture the spatio-temporal variations among a set of genetic entities. At the lower level, several mobile platforms (called scouts) will be deployed to persistently acquire data regarding the health/state of the genetic entities. These platforms will be equipped with low fidelity, low resolution sensors that acquire frequent measurements, and infer changes that take place at small-time scales. This information is sent to the robots at the upper level (called Modular Autonomous Rover for Sensing (MARS)). The rovers build animportance mapwhich prioritizes the plant specimens based on the severity of plant stress related traits. The rovers are high end mobile platforms equipped with high resolution multi-modal sensors and advanced computational resources. The rovers have two primary functions: (i) Capture high fidelity measurements for in-depth diagnosis and long term prognosis for critical plant stress related traits (ii) Build animportance mapwhich prioritizes the plant specimens based on the severity of plant stress related traits.The proposed research has been divided into the 3 researchthruststo ensure joint collaboration and student mentoring among investigators sharing common research interests.Thrust 1: Sensor activation and motion planning techniques for mobile platformsIn this thrust, we will design efficient task allocation strategies, path planning techniques and communication schemes for the scouts. The main objective of the design is to minimize the energy spent by the scouts for the aforementioned activities. As a part of this thrust,the following research tasks will be ensued:Research Task 1: Opportunistic communication:PI Bhattacharyawill investigatecompressive sensing techniques to propose optimal transmission policies from the scouts to the rovers in order to exploit the inherent sparsity in the data collected by the scouts. This leads to energy efficient communication strategies.Research Task 2: Task partitioning and scout-task allocation:PI Bhattacharyawill develop novel partitioning algorithms for task partitioning and scout-specimen allocation using the framework of coalition game theory. Furthermore, task allocation in decentralized as well as centralized scenarios will be investigated.Thrust 2: Importance map and data analysis of high dimensional dataThe data for constructing the importance map, and subsequent traits will be very high dimensional (e.g., hyperspectral images) and heterogeneous (e.g., soil condition, weather, genotype). We will deploy hierarchical feature extraction algorithms that can extract meaningful features at different scales from data as well as enable multi-modal information fusion at the abstract feature level.As a part of this effort,the following research tasks will be ensued:Research Task3:Spatio-temporal Importance map building:In this task,Co-PIs Ganapathysubramanian and A.K. Singhwill build RGB image processing techniques to generate a spatio-temporal importance map that can lead the robotic platform to arrive at the 'right place at the right time' to gather critical information regarding individual alleles planted in the field.Research Task 4:Deep learning on hyperspectral images:In this task,Co-PIs Sarkar, Ganapathysubramanian and A. Singhwill explore deep learning architectures for learning hierarchical features from the large number of information channels of hyperspectral data in a scalable manner for disease trait detection. These features will then be used for subsequent GWAS studies to identify genetic loci that putatively affect said features. Research Task 5:Spatio-temporal graphical modeling:In this task,Co-PIs Sarkar and A. K. Singhwill develop spatio-temporal graphical modeling and information fusion techniques to answer key plant science questions related to the genome and connectome using ultra high volume of spatio-temporal data collected by the robotic platform.Thrust 3: Field Deployment: Data collection,analysis and evaluation to resolve plant science questionsIn this thrust, we will deploy the computational and robotic platform in research farms to collect and analyze data regardingdisease spread signatures in soybean.ResearchTask 5: Robotic platform design and assembly:In this task,the team of investigators will advise a group of undergraduate and graduate students to build MARS through a capstone project in the first year of the project.Additionally, the communication on-board MARS would be integrated with Cy-Eye, a networked swarm roboticplatform developed at PI Bhattacharya's lab,to enable MARS to communicate with the sensor network.ResearchTask 6: Field Tests:We have identified several economically important diseases and stresses to target in this proposal: Soybean Cyst Nematode (SCN), Sudden Death Syndrome (SDS), Iron Deficiency Chlorosis (IDC), frogeye leaf spot (FLS). We have chosen these diseases as they are important to the soybean growing regions as well as having similar and confounding symptoms expressions on the canopy. Furthermore, they are a good mix of biotic and abiotic stresses that will emphasize the credibility of the proof-of-concept products. We will initially deploy MARS along with Cy-Eye within the university farms network and the plant diagnostics lab at ISU. While our initial focus is on soybean, we will carefully retrofit these platforms to enable a straightforward progression to other economically impactful crops like maize and sorghum (for which we have expertise at ISU.)We anticipate using ultra-high dimension genome wide association (GWA) and epistatic (GWE) studies, which are possible due to the large RBG and hyperspectral datasets produced. The ML tool chain will be used to extract a vast amount of precision data points and features that is impossible via current manual measurements. This itself is a major advance in the area of high throughput phenotyping. While digital imaging is now starting to be used in a few labs, the computational and field robotics advances are new in agriculture. This research will result in best practices in data collection that will be invaluable for future work.

Progress 12/15/16 to 12/14/22

Outputs
Target Audience:1) Researchers in Field Robotics 2) Plant science researchers 3) Graduate and undergraduate students in mechanical engineering, computer science and agronomy. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided?1) 4 PhD students, 2 postdocs and 5 undergraduate students were supported by the project. They were from Mechanical Engineering, Computer Science and Agronomy. The team trained them in several facets of robotics, machine learning and agronomy. 2) The team demonstrated the robotic technology to farmers in farm progress show in 2019. 3) The robotic technology was demonstrated to public and university officials at several occasions. How have the results been disseminated to communities of interest?1) The results were published and presented in topconferences (IROS and ICRA)in the area of Robotics. 2) The team held several workshops over the period of 6 years at the university to engage with the stake holders and demonstrate the technology. 3) The PI and the team have given several talks and presentations in severaluniversities regarding the robotics research and devlopment related to the project. 4) The PI has also given tutorials related to agricultural robotics at international venues. What do you plan to do during the next reporting period to accomplish the goals? Nothing Reported

Impacts
What was accomplished under these goals? The accomplishments of the team on an annual basis are provided below: Year 1: 1) The research team designed and built a ground robot from scratch that can image canopies of individual soybean plants. They developed an algorithm to process the image to predict the extent of iron deficiency chlorosis among soy plants. The algorithm was coded, and incorporated into the robot's onboard computinf module. The robot was deployed in the fields to image the plant canopies, and compute IDC on-board. 2) The research team built a autonomous rover from scratch that will have the capability of hyperspectral imaging. An autosteer mechanism for the rover was programmed on-board for trajectory planning. 3) The research tem developed and published algorithms for importance sampling and salient data exploration to guide the ground robots. Year 2: 1) The research team developed path collision-free path planning algorithms for multi-robot systems in "communication denied" environments. 2) The research team developed importance sampling strategies based on information-theoretic metrics to aid robot navigation. 3) The research team deployed robots in the field to collect data on flowering initiation and pod-counting in soybean crops. Year 3: The accomplishment in this year was primarily in the area of field robotics. The multirobot system built by PI bhattacharya and his graduate students was test run in the field. A robotcollected thedata for pod-counting in soybeans which was provided to the ML experts in the group to process. Year 4: During the first no-cost extension year, we achieved the following: 1) The team developed deployment strategies for a group of aerial vehicles to work with the ground vehicles deployed for phenotyping. Two important tasks for the aerial teams were chosen. The first task was to gain information about human beings working in the fields alongside robots. This information can be used by the ground robot teams to work in regions that maximize "working distance" from the humans. The second task for the aerial vehicles is to deliver batteries to the ground robots for long-term persistent deployment. 2) The ground robots were deployed in the field to image soybeans. The team developed an automated system to predict the yield from the images. The PI's in thrust 2 developed a multi-view image-based yield estimation framework utilizing deep learning architectures. The results demonstrate the promise of using robotic platforms along with ML models in making breeding decisions with significant reduction of time and human effort, and opening new breeding methods avenues to develop cultivars. Year 5: The PI and a graduate student were able to build an auonomous aerial recharging system to refuel the battery onboard the ground robots. They had a publication in IROS 2021 related to the aforementioned effort. The PI and a second graduate student designed an algorithm to for a multi-UAV platform to navigate and loiter around humans in the field. Graduate student Tianshuang Gao (supervisedbythe PI) defended his PhD thesis successfully in the area path planing for multi-robot systems in aisle environments. Year 6: Two graduate students worked on the problem of deploying multiple-UAVs for delivery of battery to ground robots. The main accomplishmentof the team was to developa battery operating system that can 1) monitor the level of battery within multiple ground robots deployed in the field 2) Enablea remote UAV to supply a battery when needed 3) Plan/Route a team of UAVs to simultaneously supply batteries to multiple ground robots.One MS student defended his thesis successfully on UAV-enabled battery charging system.

Publications

• Type: Conference Papers and Presentations Status: Published Year Published: 2018 Citation: H. Emadi and S. Bhattacharya, On Myopic Strategies For Resource Constrained Informative Sampling, European Control Conference
• Type: Journal Articles Status: Published Year Published: 2018 Citation: T. Gao, H. Emadi, H. Saha, J. Zhang, A. Lofquist, A. Singh, B. Ganapathysubramanian, S. Sarkar, A. Singh, and S. Bhattacharya, A multi-robot platform for distributed phenotyping, Robotics (4), 2018.
• Type: Conference Papers and Presentations Status: Accepted Year Published: 2022 Citation: Shashwata Mandal and Sourabh Bhattacharya, Planning for Aerial Robot Teams for Wide-Area Biometric and Phenotypic Data Collection, Proceedings of ICRA workshop on Intelligent Aerial Robotics: From Autonomous Micro Aerial Vehicles to Sustainable Urban Air Mobility and Operations, 2022.
• Type: Conference Papers and Presentations Status: Accepted Year Published: 2022 Citation: David Adegbesan and Sourabh Bhattacharya, UAGV-Enabled Battery Swapping Recharging System, Proceedings of ICRA workshop on Energy Storage and Delivery in Robotic Systems, 2022.
• Type: Theses/Dissertations Status: Submitted Year Published: 2022 Citation: Drone enabled autonomous charging-on-demand, David Adegbesan, MS Thesis, Department of Mechanical Engineering, Iowa State University
• Type: Conference Papers and Presentations Status: Published Year Published: 2017 Citation: Z. Jiang, A. Balu, C. Hegde, S. Sarkar, Collaborative Deep Learning in Fixed Topology Networks, Proceedings of Advances in Neural Information Processing Systems (NIPS), (Long Beach, CA), 2017.
• Type: Conference Papers and Presentations Status: Published Year Published: 2017 Citation: H. Saha, T. Gao, H. Emadi, Z. Jiang, A. Singh, B. Ganapathysubramanian, S. Sarkar, A. Singh, S. Bhattacharya, Autonomous mobile sensing platform for spatiotemporal plant phenotyping, Proceedings of ASME 2017 Dynamic Systems and Control Conference (DSCC) (Tysons, VA), 2017.
• Type: Conference Papers and Presentations Status: Published Year Published: 2018 Citation: T. Gao, H. Emadi, H. Saha, J. Zhang, A. Lofquist, A. Singh, B. Ganapathysubramanian, S. Sarkar, A. Singh, and S. Bhattacharya, Navigation Strategies for a Multi-Robot Ground-Based Row Crop Phenotyping Platform, In Proceedings of the ASME Dynamic Systems and Control Conference, V003T32A009-V003T32A009, 2018.
• Type: Other Status: Published Year Published: 2018 Citation: T. Gao, H. Emadi, A. Lofquist, J. Zhang, H. Saha, A. Singh, S. Sarkar, B. Ganapathysubramanian, A. Singh, S. Bhattacharya, A Multi-Robot Ground-Based Row Crop Phenotyping System, Poster Presentation, IEEE International Conference on Robotics and Automation, 2018.
• Type: Conference Papers and Presentations Status: Published Year Published: 2019 Citation: T. Gao, S. Bhattacharya, Multirobot Charging Strategies: A Game-Theoretic Approach. IEEE International Conference on Robotics and Automation, 2019.
• Type: Journal Articles Status: Published Year Published: 2019 Citation: "Multirobot Charging Strategies: A Game-Theoretic Approach", Sourabh Bhattacharya and Tianshuang Gao, Robotics and Automation Letters, 4(3), 2823-2830.
• Type: Conference Papers and Presentations Status: Published Year Published: 2019 Citation: "Multirobot Charging Strategies: A Game-Theoretic Approach", Sourabh Bhattacharya and Tianshuang Gao, International conference on Intelligent Robots and Systems, 2019
• Type: Other Status: Published Year Published: 2019 Citation: "A Deep Vision based approach to Real-time Detection and Counting of Soybean Pods", Zhisheng Zhang, Sambuddha Ghosal, Johnathon Shook, Matthew Carroll, Luis Riera, Tianshuang Gao, Arti Singh, Sourabh Bhattacharya, Baskar Ganapathysubramanian, Asheesh Singh and Soumik Sarkar, MLCAS 2019
• Type: Conference Papers and Presentations Status: Published Year Published: 2021 Citation: Shashwata Mandal, Tianshuang Gao, Sourabh Bhattacharya: Planning for Aerial Robot Teams for Wide-Area Biometric and Phenotypic Data Collection, International Conference on Intelligent Robots and systems, 2021
• Type: Journal Articles Status: Published Year Published: 2022 Citation: "Deep Multi-view Image Fusion for Soybean Yield Estimation in Breeding Applications", Luis G Riera, Matthew E. Carroll, Zhisheng Zhang, Johnathon M. Shook, Sambuddha Ghosal, Tianshuang Gao, Arti Singh, Sourabh Bhattacharya, Baskar Ganapathysubramanian, Asheesh K. Singh, and Soumik Sarkar, Plant Phenomics
• Type: Conference Papers and Presentations Status: Published Year Published: 2021 Citation: Tianshuang Gao, Yan Tian, Sourabh Bhattacharya: Refuel Scheduling for Multirobot Charging-on-Demand, International Conference on Intelligent Robots and systems, 2021

Progress 12/15/20 to 12/14/21

Outputs
Target Audience:Roboticists in the area of field robotics. Plant scientists. Changes/Problems:The PI is now investigating the possibility of incorporating cooperation between ground and aerial vehicles in the field. What opportunities for training and professional development has the project provided?1 graduate studentin computer science wastrained during summer in deploying drones that could collaboratively work with the ground robots. How have the results been disseminated to communities of interest?2 papers related to the project have been published and presented in this year's IROS conference. What do you plan to do during the next reporting period to accomplish the goals?The PI plans to train a graduate student to work on the multi-robotplatforms and design algorithms for planning and routing efficiently. The remaining funds can support a graduate student for a semester.

Impacts
What was accomplished under these goals? The PI and a graduate student were able to build an auonomousaerial recharging system to refuel the battery onboard the ground robots. They had a publication in IROS 2021 related to the aforementioned effort. The PI and a second graduate student designed an algorithm to fora multi-UAV platform to navigate and loiter aroundhumans in the field.

Publications

• Type: Conference Papers and Presentations Status: Published Year Published: 2021 Citation: Shashwata Mandal, Tianshuang Gao, Sourabh Bhattacharya: Planning for Aerial Robot Teams for Wide-Area Biometric and Phenotypic Data Collection, International Conference on Intelligent Robots and systems, 2021
• Type: Conference Papers and Presentations Status: Published Year Published: 2021 Citation: Tianshuang Gao, Yan Tian, Sourabh Bhattacharya: Refuel Scheduling for Multirobot Charging-on-Demand, International Conference on Intelligent Robots and systems, 2021

Progress 12/15/19 to 12/14/20

Outputs
Target Audience:1) Researchers in robotics and plant science. 2) Practitioners in industry in robotics and plant science. 3) graduate and undergraduate students in robotics and agronomy. Changes/Problems:Some changes in the goals have been made by including aerial robots in addition to ground robots in the research. The availability of graduate students and the ability to perform field work has been significantly affected due to the ongoing pandemic. What opportunities for training and professional development has the project provided?1) Graduate and undergraduate students were trained during the no-cost extension year. How have the results been disseminated to communities of interest?The results have been submitted to international conferences (International conference on robotics and automation) and journals (Robotics and automation letters, Plant Phenomics). What do you plan to do during the next reporting period to accomplish the goals?Currently, the robotic platforms face significant challenges in working in fullyautonomous mode in the fields alongside humans for long hours. In the next reporting period, we plan to work on improving the persistence and autonomyof the ground robotsby including a team of aerial vehicles. To be specific, we plan to acheive the following: 1) Improve the guidance and navigation of the robots. We plan to investigate and implement minimum violation planning for the ground robots to maximize the distance from the soybean plants between two rows, and maximize the distance from humans working in the field. 2) Develop a mechanism on the ground robots toreceive and plug-in batteries deliveredbyaerial platforms. We also plan to develop pick-up and delivery mechanisms for drones to transport batteries across the fields to robots in need.

Impacts
What was accomplished under these goals? During the first no-cost extension year, we achieved the following: 1) The team developed deployment strategies for a groupof aerial vehicles to work with the ground vehicles deployed for phenotyping. Two important tasks for the aerial teams were chosen. The first task was to gain information about human beings working in the fields alongside robots. This information can be used by the ground robotteams to workin regions that maximize"working distance" from the humans. The second task for the aerial vehicles isto deliverbatteries tothe ground robots for long-term persistent deployment. 2) The ground robots were deployed in the field to image soybeans. The team developed an automated system to predict the yield from the images.The PI's in thrust 2developed a multi-view image-based yield estimation framework utilizing deep learning architectures.Theresults demonstrate the promise of using robotic platforms along with ML models in making breeding decisions with significant reduction of time and human effort, and opening new breeding methods avenues to develop cultivars.

Publications

• Type: Conference Papers and Presentations Status: Under Review Year Published: 2021 Citation: "Planning for Aerial Robot Teams for Wide-Area Biometric and Phenotypic Data Collection", Tianshuang Gao, Shashwata Mandal, Sourabh Bhattacharya
• Type: Journal Articles Status: Under Review Year Published: 2021 Citation: "Refuel scheduling for a team of aerial vehicles serving a team of ground robots deployed in multi-aisle environments", Tianshuang Gao, Yan Tian, Sourabh Bhattacharya
• Type: Journal Articles Status: Under Review Year Published: 2021 Citation: "Deep Multi-view Image Fusion for Soybean Yield Estimation in Breeding Applications", Luis G Riera, Matthew E. Carroll, Zhisheng Zhang, Johnathon M. Shook, Sambuddha Ghosal, Tianshuang Gao, Arti Singh, Sourabh Bhattacharya, Baskar Ganapathysubramanian, Asheesh K. Singh, and Soumik Sarkar

Progress 12/15/18 to 12/14/19

Outputs
Target Audience:1) PI Bhattacharya and graduate student Tianshuang gao provided a 3 hour demo and tutorial on agricultural robots in controls systems course to ISU undergrads. 2) Tianshuang Gao presented posters at NAPB annual meeting and Machine Learning for Cyber-Agricultural Systems. 3) PI Bhattacharya gave an invited talk to faculty and students at Indian Institute of Sciences part of which covered agricultural robots. 4) Tianshuang Gao gave a demo of the robots to departmental seminar speakers, Dean of College of Engineering and Dean of LAS. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided?Several undergraduates were trained as part of the college level honors program as well as a part of the project in hardware, design and development of robotic systems. How have the results been disseminated to communities of interest?The results have been published in premier journals and conference in robotics. Additionally, posters have been presented at the annual NAPB meeting and MLCAS workshop. What do you plan to do during the next reporting period to accomplish the goals?The project involves building, testing and deploying multiple ground robot platforms for data collection in soybean farms. Due to inclement weather (severe rainfall in the midwest) during the past summers and fall, testing the robotic platforms outdoor (in the fields) has been a challenge. This has delayed the deployment plans. Per our current plan, the robotic platforms havebeen deployed in late fall this year for data collection. We are requesting a one year no-cost extension for processing the data collected to evaluate the performance of the robotic platforms, and improve their navigation capabilities for future deployments.

Impacts
What was accomplished under these goals? A multirobot system was built and deployed for data collection and pod-counting.

Publications

• Type: Journal Articles Status: Published Year Published: 2019 Citation: "Multirobot Charging Strategies: A Game-Theoretic Approach", Sourabh Bhattacharya and Tianshuang Gao, Robotics and Automation Letters, 4(3), 2823-2830.
• Type: Journal Articles Status: Submitted Year Published: 2019 Citation: "Refill Scheduling for Harvesting-on-Demand", Yan Tian and Sourabh Bhattacharya, Transactions on Automation Science and Engineering.
• Type: Conference Papers and Presentations Status: Published Year Published: 2019 Citation: "Multirobot Charging Strategies: A Game-Theoretic Approach", Sourabh Bhattacharya and Tianshuang Gao, International conference on Intelligent Robots and Systems, 2019
• Type: Other Status: Published Year Published: 2019 Citation: "A Deep Vision based approach to Real-time Detection and Counting of Soybean Pods", Zhisheng Zhang, Sambuddha Ghosal, Johnathon Shook, Matthew Carroll, Luis Riera, Tianshuang Gao, Arti Singh, Sourabh Bhattacharya, Baskar Ganapathysubramanian, Asheesh Singh and Soumik Sarkar, MLCAS 2019

Progress 12/15/17 to 12/14/18

Outputs
Target Audience:1) PI Bhattacharya and graduate student Tianshuang Gao gave a 3 hour tutorial and demo on agricultural robotics in the courseBCB/GDCB/ME 585 - Fundamentals of Predictive Plant Phenomics 2) PI Bhattacharya gave an invited talk on agricultural robotics at the ISU retirees association symposium. 3) PI Bhattacharya spent two lectures in ME 411: Automatic controls in Fall 2018 on the topic of agricultural robotics. 4) Graduate student Tianshuang Gao and PI Bhattacharya demonstrated the robots to visitors fro UIUC and Tokyo university, Japan. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided?Graduate as well as undergraduates students have been trained through several research and classroom activities on campus. How have the results been disseminated to communities of interest?Through conference publications, journal publications and tutorials. What do you plan to do during the next reporting period to accomplish the goals?We plan to develop motion strategiesforrobots that can be deployed for extended periods of time to collect data on the field keeping into account their energy and communication needs.

Impacts
What was accomplished under these goals? 1) The research team developed path collision-free path planning algorithms for multi-robot systems in "communication-denied" environments. 2) The research team developed importance sampling strategies based on information-theoretic metrics to aid robot navigation. 3) The research team deployed robots in thje field to collect data on flowering initiation and pod-counting in soybean crops.

Publications

• Type: Conference Papers and Presentations Status: Published Year Published: 2018 Citation: T. Gao, H. Emadi, A. Lofquist, J. Zhang, H. Saha, A. Singh, S. Sarkar, B. Ganapathysubramanian, A. Singh, S. Bhattacharya, A Multi-Robot Ground-Based Row Crop Phenotyping System, Poster Presentation, IEEE International Conference on Robotics and Automation, 2018.
• Type: Conference Papers and Presentations Status: Published Year Published: 2018 Citation: H. Emadi, S. Bhattacharya, On Myopic Strategies for Resource Constrained Informative Sampling, In Proceedings of IEEE/EUCA European Control Conference, pp. 1998 - 2003, 2018.
• Type: Conference Papers and Presentations Status: Submitted Year Published: 2019 Citation: T. Gao, S. Bhattacharya, Multirobot Charging Strategies: A Game-Theoretic Approach. IEEE International Conference on Robotics and Automation, 2019.
• Type: Conference Papers and Presentations Status: Published Year Published: 2018 Citation: T. Gao, H. Emadi, H. Saha, J. Zhang, A. Lofquist, A. Singh, B. Ganapathysubramanian, S. Sarkar, A. Singh, and S. Bhattacharya, Navigation Strategies for a Multi-Robot Ground-Based Row Crop Phenotyping Platform, In Proceedings of the ASME Dynamic Systems and Control Conference, V003T32A009-V003T32A009, 2018.
• Type: Journal Articles Status: Published Year Published: 2018 Citation: T. Gao, H. Emadi, H. Saha, J. Zhang, A. Lofquist, A. Singh, B. Ganapathysubramanian, S. Sarkar, A. Singh, and S. Bhattacharya, A multi-robot platform for distributed phenotyping, Robotics (4), 2018.

Progress 12/15/16 to 12/14/17

Outputs
Target Audience:1) PI Bhattacharya developed and taughta threelecture module on agricultural robotic path planning in his course on Automatic Control taught in Fall 2017. 2) PI Bhattacharya mentored 8 honors undergraduate students in Spring 2017 to design and develop an autonomous charging station for robots deployed on field. 3) PI Bhattacharya mentored Brian Welch, an honors student,in Summer 2017 to develop algorithms for autonomous robot docking to charging stations. Brian was supported in partby ISU from summer honors fellowship program. 4) PI Bhattacharya , graduate students Tianshuang Gao and Homagni Saha,demonstrated the autonomous phenotyping platform developed in his lab to Dr. Ranveer Chandra from Microsoft Research hosted by PI Ganapathysubramanian in summer 2017. 5) PI Bhattacharya, graduate students Hamid Emadi and Tianshuang Gao presentedthe progress regarding importance sampling and autonomous phenotyping platform to a Japanese delegation of faculty members hosted by PI Sarkar in Fall 2017. Changes/Problems:NA What opportunities for training and professional development has the project provided?1) We have trained 6graduate students for different periodsduring the project. The background of the graduate students was a mix of Mechanical, Electrical, Computer Engineering and Agronomy. 2) We trained >20 undergraduate students in robotics and agronomy. How have the results been disseminated to communities of interest?We have published 3 conference papers that have been listed in the REEport. Additionally, PIs have given invited talks regarding the work at academic institutions, and technical meetings. What do you plan to do during the next reporting period to accomplish the goals?We will deploy and test the multirobot system in the field for gathering images of the canopy. We are also working on energy-efficient communications schemes for recharging of the autonomous platforms.

Impacts
What was accomplished under these goals? 1) The research team designed and built a ground robot from scratch that can image canopies of individual soybean plants. They developed an algorithm to process the image to predict the extent of iron deficiency chlorosis among soy plants. The algorithm was coded, and incorporated into the robot's onboard computinf module. The robot was deployed in the fields to image the plant canopies, and compute IDC on-board. 2) The research team built a autonomous rover from scratch that will have the capability of hyperspectral imaging. An auto-steer mechanism for the rover was programmed on-board for trajectory planning. 3) The research tem developed and published algorithms for importance sampling and salient data exploration to guide the ground robots.

Publications

• Type: Conference Papers and Presentations Status: Accepted Year Published: 2017 Citation: Z. Jiang, A. Balu, C. Hegde, S. Sarkar, Collaborative Deep Learning in Fixed Topology Networks, Proceedings of Advances in Neural Information Processing Systems (NIPS), (Long Beach, CA), 2017.
• Type: Conference Papers and Presentations Status: Published Year Published: 2017 Citation: H. Saha, T. Gao, H. Emadi, Z. Jiang, A. Singh, B. Ganapathysubramanian, S. Sarkar, A. Singh, S. Bhattacharya, Autonomous mobile sensing platform for spatiotemporal plant phenotyping, Proceedings of ASME 2017 Dynamic Systems and Control Conference (DSCC) (Tysons, VA), 2017.
• Type: Conference Papers and Presentations Status: Awaiting Publication Year Published: 2018 Citation: H. Emadi and S. Bhattacharya, On Myopic Strategies For Resource Constrained Informative Sampling, European Control Conference