HIGH INTENSITY PHENOTYPING SITES: A MULTI-SCALE, MULTI-MODAL SENSING AND SENSE-MAKING CYBER-ECOSYSTEM FOR GENOMES TO FIELDS

• Sponsoring Institution: National Institute of Food and Agriculture
• Project Status: EXTENDED
• Funding Source: AFRI COMPETITIVE GRANT
• Reporting Frequency: Annual
• Accession No.: 1022368
• Grant No.: 2020-68013-30934
• Project No.: IOW05612
• Proposal No.: 2019-05478
• Program Code: A1141
• Project Start Date: Jun 1, 2020
• Project End Date: May 31, 2024
• Grant Year: 2020

Project Director
Schnable, P. S.

Recipient Organization
IOWA STATE UNIVERSITY
2229 Lincoln Way
AMES,IA 50011

Performing Department
Agronomy

Non Technical Summary
To date much of the focus of agricultural research has been on increasing yield rather than ensuring the stability of yields within and across regions and years. It is of course important to develop higher yielding crop varieties. However, increasingly variable weather patterns have already begun to negatively impact agriculture. We currently lack the knowledge and tools necessary to efficiently develop resilient crop varieties that will provide stable and economically viable yields across increasingly variable environments. This problem is exacerbated by the fact that breeding new crop varieties takes 7-10 years, and at many locations today's weather may not be an accurate representation of the spectrum of weather new varieties will experience at that same locations 10 years from now. To address the challenge of breeding next generation resilient crop varieties we require accurate and mechanistically based models that can predict phenotypic outcomes based on genetic, environmental, and crop management data. Fortunately, advances in the plant sciences, computational and data sciences, and engineering offer the potential to help us address this challenge and thereby create a more sustainable, resilient and profitable US agricultural system.Developing accurate predictive crop models requires an enhanced understanding of the combined effect of crop variety (G) and environment (9), GxE. This in turn requires large collections of plant traits and environmental data gathered from common sets of crop varieties grown in diverse environments. With support from state and national Corn Growers, the Genomes to Fields (G2F) initiative has been conducting community-based experiments to do just that. Since 2014, G2F participants have been generating and analyzing genotypic, environmental, and crop management data from commercially relevant maize germplasm to learn how GxE interactions influence plant traits.The proposed project, G2F-HIPS, will support and intensify G2F by deploying, evaluating and validating a combination of established, image-based sensing technologies and promising new field-based agricultural sensors, generating and sharing reference data to foster community innovation, developing and democratizing analysis pipelines for phenotypic data, conducting proof-of-principle research projects to identify genes responsible for crop responses to environmental variation, and contributing in a substantial manner to the training of current and future agricultural researchers to make use of these innovations. As such, G2F-HIPS will promote the widespread adoption of new sensing technologies, methods of data analysis and thinking across the many G2F sites. In combination, these activities have the potential to facilitate a more mechanistic understanding of how phenotypes respond to genotypic and environmental variation, thereby facilitating the development of more resilient crop varieties that make more efficient use of agricultural inputs such as nitrogen and water, with corresponding environmental benefits.

Animal Health Component

0%

Research Effort Categories

Basic

70%

Applied

15%

Developmental

15%

Classification

Knowledge Area (KA)	Subject of Investigation (SOI)	Field of Science (FOS)	Percent
201	1510	1081	80%
201	1510	1080	20%

Knowledge Area
201 - Plant Genome, Genetics, and Genetic Mechanisms;

Subject Of Investigation
1510 - Corn;

Field Of Science
1081 - Breeding; 1080 - Genetics;

Keywords

quantitative genetics

sensors

community research

data science

gxe interactions

phenotyping

Goals / Objectives
Increasingly variable weather patterns have already begun to negatively impact agriculture. We currently lack the knowledge and tools necessary to efficiently develop resilient crop varieties that will provide stable and economically viable yields across increasingly variable environments. To address the challenge of breeding next generation resilient crop varieties we require accurate and mechanistically based models that can predict phenotypic outcomes based on genetic, environmental, and crop management data. Developing accurate predictive crop models requires an enhanced understanding of GxE (Genotype X Environment). This in turn requires large collections of phenotypic and environmental data gathered from common sets of crop varieties grown in diverse environments. With support from state and national Corn Growers, the Genomes to Fields (G2F) initiative has been conducting community-based experiments to do just that. Since 2014, G2F participants have been generating and analyzing genotypic, environmental, and crop management data from commercially relevant maize germplasm to learn how GxE interactions influence phenotypeThe proposed project, G2F-HIPS, will support and intensify G2F by deploying, evaluating and validating a combination of established, image-based sensing technologies and promising new field-based agricultural sensors, generating and sharing reference data to foster community innovation, developing and democratizing analysis pipelines for phenotypic data, conducting proof-of-principle research projects to identify genes responsible for crop responses to environmental variation, and contributing in a substantial manner to the training of current and future agricultural researchers to make use of these innovations.The objectives of the proposed project are to:Objective 1. Deploy and evaluate for use by G2F, a combination of established, image-based sensing technologies and highly promising new field-based agricultural sensors (for nitrate and water) and generating and sharing reference phenomic data to foster community innovation.Objective 2. Develop and democratize analysis pipelines for phenotypic data.Objective 3. Conduct proof-of-principle research projects to identify agronomically relevant genes from phenomic data.Objective 4. Contribute in a substantial manner to the training of current and future agricultural researchers to make use of these innovations.

Project Methods
During years 1 & 2 replicated yield plots will be grown in 10 unique environments at 6 locations across a 700 mile west to east transect that varies in elevation from 4,100-600' and that experiences between 18-37" of rainfall annually. At one site we will apply zero, partial or full irrigation to create three distinct water environments. At another, we will apply three levels of N fertilizer to create three distinct N environments. In year 1, two replications of ~45 core exPVP hybrids that have been included annually in all G2F field trials since 2014 will be grown in each of the 10 G2F-HIPS environments. In Year 2, in addition to the exPVP hybrids we will grow two replications of an association population (the "SAM panel") that consists of approximately 380 inbreds, which has been previously genotyped with a set of 1.2M high quality segregating SNPs and that has provided single-gene association mapping resolution. To provide baseline data for comparison to new phenotyping methods, in years 1 & 2, 12 traits will be manually collected at maturity from the 10 G2F-HIPS environments using G2F's SOP. Additional data will be collected using robots, UAVs and sensors. All data will be released to the community to accelerate development of improved statistical approaches for use of high throughput phenotyping data gathered from diverse environments to analyze GxE and promote gene discovery.Objective 1G2F-HIPS will provide the first comprehensive multimodal crop-sensing infrastructure. We will deploy imaging-based sensors using both ground-based (e.g., field robots) and aerial (e.g., UAV/drone) platforms, as well as next-generation plant sensors specifically designed to directly measure key traits (e.g. stalk and soil N, leaf transpiration and soil moisture). G2F-HIPS will collect hierarchical phenotypic data of structural, physiological, and performance-related traits ranging from molecular to plant to plot levels, and will conduct validations of sensor measurements to ensure data integrity and biological interpretations, and perform proof-of-principle experiments to demonstrate the value of coupled direct soil and plant measurements of N concentrations and water/transpiration.Objective 2G2F requires scalable software tools that can assimilate data and provide actionable feedback on data collected. This will be particularly true as increasingly complex, diverse, and multi-scale sensor data are collected. We envision a generalizable, Lego-like modular framework that provides end-users the ability to extract physiologically meaningful traits, fuse diverse data, and obtain actionable feedback on amount, and quality of data collected for specific analysis outcomes. Such a framework will only be truly impactful when done at scale and made accessible to a wide community. This will result in a sustainable community enabling consistent and reproducible data analysis. Our vision is supported by the following sub-objectives: Sustainable tool deployment to democratize data analytics pipelines, and tool development for G2F-HIPS data: annotation tool chains, turking on CyVerse.Objective 3G2F-HIPS will deploy conventional, high throughput, and high intensity phenotyping techniques across a diverse set of natural environments, management practices, and maize germplasm. The resulting dataset will be released to the community to enable both a wide range of future biological analyses and the development of new quantitative genetics and image processing analysis methods. G2F-HIPS will also conduct three "showcase" analyses using these data.Objective 4We will use lessons learned from ISU's Predictive Plant Phenomics (P3) paradigm graduate training program to offer traditional courses that are foundational for helping new graduate students to think differently. Courses will be offered in-person at ISU and virtually at UA; course materials will be disseminated broadly. We also will deliver just-in-time workshops, bootcamps, and challenges for students, researchers, and agriculture learners at national and international venues so that they, too, can develop the skills and expertise required to effectively participate in high-intensity phenotyping. Although personnel related to G2F will have priority, these activities will be open to the entire plant phenomics community on a space-available basis.Meeting project goals will require the collaborative efforts of researcher/educators with expertise in diverse fields. Towards this end, a diverse, highly collaborative group of investigators has been assembled and individual co-PIs will have sole or shared responsibilities for specific project objectives as outlined.

Progress 06/01/22 to 05/31/23

Outputs
Target Audience:The plant genetics community and breeders. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided? Two engineering graduate students, along with one undergraduate specializing in mechanical engineering and another in electrical engineering, actively contributed to the project. Their roles encompassed design, fabrication, assembly, and deployment of various sensor types. Their efforts hold the potential to enhance agricultural efficiency and profitability, while advancing soil and water conservation. This project provided students with practical experience in manufacturing, electronic design, and applying sensor technologies to real-world agricultural challenges. Dr. Talukdar Zaki Jubery, a staff scientist, pursued learning and adaptation, integrating cutting-edge machine vision advancements into the PhieldCam experiment analytics pipeline. In addition, he mentored multiple graduate students and collaborated with partners from CyVerse and AIIRA to democratize software products and workflows. A research technicianand anundergraduate student learned annotation techniques and conducted preliminary analyses on these datasets. Upcoming plans include the public release of datasets, R codes, functions, and a package. These resources will be valuable for training researchers in analyzing high-dimensional datasets. A biology graduate student received specialized training involving the execution of the PhenoBot imaging project within a subset of the HIPS population in 2022. They imaged around 120 genotypes in two key data collection stages: the knee-high growth phase and the full maturity phase, enabling precise identification of leaf angle variations within genotypes. This hands-on experience equipped the student with advanced phenotyping skills, sophisticated data collection methods, and expertise in utilizing state-of-the-art robotic systems for plant-centric research. Another biology graduate student gained hands-on experience in sensor deployment, involving tasks like organizing battery boxes and customizing solar panel cables for field sensors. Their successful deployment of various sensors across six sites resulted in the collection of essential data and GPS coordinates. They developed proficiency in image capture using PhenoBot and contributed to the removal of field sensors, as well as the collection of soil samples from all sites for subsequent analysis. Additionally, the student actively participated in maintaining and preparing battery boxes and cables, in anticipation of the sensor deployment scheduled for 2023. This comprehensive training equipped them with valuable skills encompassing sensor deployment, data collection, and maintenance. A series of workshops, as listed in this report, provided training and professional development opportunities, contributing to a skilled community of agricultural researchers. How have the results been disseminated to communities of interest? The dissemination strategies encompassed various channels, including: The publication of research outcomes through seven peer-reviewed journal articles and one Ph.D. thesis. Presentations at the 2022 international annual meeting of the American Society of Agricultural, Biological Engineers, addressing the robotic plant phenotyping field and PhenoBot 3.0 to influential entities such as the Board of National Corn Growers Association, the National Association of Plant Breeders, and Iowa Legislators. A series of presentations which covered computational tools for field phenotyping, building digital twins of plants, enabling climate-smart agriculture through smart computing, discussions on AI for sustainable agriculture at the Corteva Sustainability Forum, insights on large interdisciplinary teams for agriculture at the 2023 CUAHSI Biennial meeting, and addressing challenges and opportunities in AI for Agriculture to state legislatures at the NCSL in Chicago. Additionally, workshops were documented and recorded, made accessible via YouTube, and supplemented by materials hosted on CyVerse and AG2PI's website. Software and data artifacts were made available through platforms like CyVerse, Google Colaboratory, and Binder, linked with workshop materials. Moreover, code resources were made publicly available on GitHub. Some examples are: Introduction to Scientific Computing: Workshop Day 1: https://www.youtube.com/watch?v=Ogl9l6QJqJ8 Workshop Day 2: https://www.youtube.com/watch?v=onLL4Tk-8lI Workshop Day 3: https://www.youtube.com/watch?v=8iB-1rVodjU Google Colab Notebook: https://colab.research.google.com/drive/1hYEwdV4y54x7jhtKxtQjVfR6OzhUKDBK Homework: https://colab.research.google.com/drive/1c30bFwwcPJ86W6x6RiwmpElbbysQTawH Solutions: https://colab.research.google.com/drive/1Sx3ujwBkzXg0EpfH78vtaCaEl47LP6Zy#scrollTo=RptAD8wx6Ny9 GitHub Code: https://github.com/phytooracle/AG2PI_Introduction_to_Scientific_Computing An upcoming initiative is focused on making datasets, along with R codes, functions, and an encompassing package, accessible to the public. These resources are poised to serve as essential tools for researchers, enabling them to adeptly analyze intricate high-dimensional datasets. This step signifies a substantial move towards broadening engagement and disseminating knowledge within the relevant communities. What do you plan to do during the next reporting period to accomplish the goals?Objective 1: Our ongoing focus revolves around sensor optimization, installation, and the seamless acquisition of data. Objective 2: We persist in advancing workflow automation and refining feature extraction, giving particular attention to the implementation of self-supervised learning algorithms. Objective 3: We plan to generate parallel datasets akin to those obtained in 2022. These datasets will be integrated into our analyses, while the enhancement of our R package/functions will streamline the analyses. Objective 4: As the CyVerse phase nears completion, our forthcoming objectives involve aiding other HIPS project team members in effectively integrating tools and data into CyVerse.

Impacts
What was accomplished under these goals? Overall impact statement: We innovated agricultural sensing by deploying a diverse range of sensors across six HIPS sites, powering them with solar panels and battery boxes. Our efforts spanned data collection, from ear and plant height data to tens of thousands of harvested ears, soil samples, and valuable partnerships with local providers. Data analysis pipelines were refined, incorporating self-supervised learning and large vision models for precise trait extraction, while advanced agronomic gene discovery through diverse data aggregation and satellite imagery integration. We focused on cultivating proficient researchers via collaborative workshops with CyVerse and AG2PI, fostering a skilled community poised to harness emerging agricultural research advancements. The collaborative efforts of a diverse team of graduate and undergraduate students have far-reaching impacts. Their roles in sensor design, fabrication, and deployment hold potential for transformative effects on agriculture, improving efficiency and sustainability. The integration of cutting-edge machine vision and mentoring activities further enriched the project. Training in data management, image analysis, and sensor deployment equipped students with practical skills for their careers. The project's resources, including datasets and codes, are set to train future researchers. Hands-on experiences in PhenoBot imaging and sensor technology advanced students' expertise, while workshops contributed to a broader community of skilled researchers, ensuring sustained impact. **Objective 1: Advancing Agricultural Sensing and Data Sharing** Our mission under Objective 1 was to deploy image-based sensing technologies and emerging agricultural sensors for nitrate and water assessment. During the summer of 2022, we prepared 350 battery boxes and customized over 100 solar panel cables for sensor power. Across six HIPS sites, we deployed 450 moisture/transpiration sensors, 110 soil nitrate sensors, 110 plant nitrate sensors, 9 soil water sensors, and 39 commercial soil moisture sensors. Weather stations and ground control points were installed for accurate satellite imaging. In fall 2022, we collected ear and plant height data, harvested tens of thousands of ears, and gathered soil samples. Our team invested approximately 6,000 hours in trait collection and data analysis. Collaborations with local HIPs providers enriched our datasets. In spring 2023, we prepared hybrid and inbred seeds for planting across six HIPS sites in IA and NE, optimizing field maps and seed organization. This year marked a pilot PhenoBot experiment on a subset of the HIPS population including G2F hybrids, commercial hybrids, and select genotypes used in previous N rate trials, capturing 120 genotypes at two time-points with over 6,000 plants imaged per hour to study leaf angle variations. We focused on enhancing PhenoBot 3.0's navigation and image capabilities, aiming for autonomous traversal using 2D and 3D LiDAR sensors and refining the PhenoStereo camera and neural network models for precise leaf angle detection. **Objective 2: Empowering Phenotypic Data Analysis Pipelines** Advancing Objective 2, we refined the analytics pipeline developed during Y1-Y2, extracting crucial plant traits from PhieldCam images. Our exploration covered models reliant on abundant annotated data, semi-supervised, self-supervised, and zero-shot models. Self-supervised learning and Zero-shot methods showed promise in pretraining on unlabeled data followed by fine-tuning with labeled data, demonstrating exemplary transfer learning performance. We explored Large Vision Models (LVM) for zero-shot maize kernel segmentation, highlighting the Segment Anything Model (SAM) for individual kernel segmentation in RGB images. Objective 2 enabled HIPS to integrate data and computational workflows into CyVerse, supported by comprehensive training resources. We harnessed CyVerse for Open Data Science, CyVerse Usage, Data Management, and Tool Integration training. **Objective 3: Pioneering Agronomically Relevant Gene Discovery** Objective 3 focused on uncovering genes of agronomic significance through sequential proof-of-principle projects. In 2022, we harvested representative ears from hybrid and inbred plots, gathering diverse cob-related traits and seed composition data using NIR technology. Aggregating data from morphological, cob-related, yield, and seed composition traits was a critical step, ensuring data integrity via exploratory analysis and verification of genotype IDs. As inbred ear and seed phenotyping concludes, field measurements from the 2023 season will integrate into a comprehensive framework, involving spatial corrections and investigating genotype-environment interactions. We acquired satellite and UAV images, automatically extracting plot-level data for public access, serving AIIRA collaborators globally. Leveraging these images, we anticipate predicting yield-contributing traits and forecasting yield under varied nitrogen levels. Moreover, we enhanced inbred marker density to 46 million markers from 1.2 million, facilitating finer genomic region mapping. Our curated data analytics code, slated for conversion into an R package, will support multi-location, multi-trait dataset cleaning and analysis. These adaptable functions hold potential for diverse datasets. **Objective 4: Empowering Agricultural Research Training** In collaboration with CyVerse and AG2PI, we hosted workshops spanning diverse domains to equip agricultural researchers with essential skills. Workshops ranged from Open Science, Web Development, and BioViz to Single-Cell RNA-Seq, Containerization, Data and Workflow Management, and Image Analysis. AG2PI workshops encompassed software platforms, phenotyping data sharing, segmentation techniques, water dynamics, remote sensing, and virtual fence technology. Our commitment to enabling proficient researchers fostered a skilled community equipped to harness emerging advancements.

Publications

• Type: Journal Articles Status: Published Year Published: 2023 Citation: Grzybowski, Marcin W., Ravi V. Mural, Gen Xu, Jonathan Turkus, Jinliang Yang, and James C. Schnable. "A common resequencing?based genetic marker data set for global maize diversity." The Plant Journal 113, no. 6 (2023): 1109-1121.
• Type: Conference Papers and Presentations Status: Published Year Published: 2022 Citation: Xiang, L., X. Liu, A. Raj, J. Yu, P. Schnable, L. Tang. 2022., Robotic Field-based Plant Architectural Traits Characterization Using Stereo Vision and Deep Neural Networks. Fourth International Workshop on Machine Learning for Cyber-Agricultural Systems (MLCAS2022).
• Type: Conference Papers and Presentations Status: Published Year Published: 2022 Citation: Xiang, L., Liu, X., Raj, A., and L. Tang. 2022. Detection and characterization of maize plant architectural traits in the field using stereo vision and deep convolutional neural networks. 2022 ASABE Annual International Meeting. Houston, TX, July 17-20, 2022. Paper No. 2200972.
• Type: Journal Articles Status: Published Year Published: 2023 Citation: Xiang, L., J. Gai, Y. Bao, J. Yu, P. S. Schnable, L. Tang. 2023. Field-based robotic leaf angle detection and characterization of maize plants using stereo vision and deep convolutional neural networks. Journal of Field Robotics. (doi.org/10.1002/rob.22166)
• Type: Journal Articles Status: Published Year Published: 2022 Citation: H. Ibrahim, S. Moru, P.S. Schnable, and L. Dong, "Wearable plant sensor for in-situ monitoring of volatile organic compound emissions from crops", ACS Sensors, 7, 2293-2302 (2022).
• Type: Journal Articles Status: Published Year Published: 2022 Citation: H. Ibrahim, S. Yin, S. Moru, Y. Zhu, M. J. Castellano, and L. Dong, "in planta nitrate sensor using a photosensitive epoxy bio-resin," ACS Applied Materials & Interfaces, 14 (22), pp. 2594925961 (2022).
• Type: Journal Articles Status: Published Year Published: 2023 Citation: Dong, D., Nagasubramanian, K., Wang, R., Frei, U.K., Jubery, T.Z., L�bberstedt, T. and Ganapathysubramanian, B., 2023. Self-supervised maize kernel classification and segmentation for embryo identification. Frontiers in Plant Science, 14, p.1108355.
• Type: Conference Papers and Presentations Status: Published Year Published: 2022 Citation: Liu, X., Xiang, L., L. Tang. 2022. In-field soybean seed pod phenotyping on harvest stocks using 3D imaging and deep learning. 2022 ASABE Annual International Meeting. Houston, TX, July 17-20, 2022. Paper No. 2201222.
• Type: Conference Papers and Presentations Status: Published Year Published: 2022 Citation: Raj, A., D. Maier, L. Johnson, J. Orrey, J. Compton, and L. Tang. 2022. Detection of Sweet Potato using Airborne Imaging Platform to estimate Post-Harvest Losses. 2022 ASABE Annual International Meeting. Houston, TX, July 17-20, 2022. Paper No. 2200973.
• Type: Conference Papers and Presentations Status: Published Year Published: 2023 Citation: Saleem, N., Ganapathysubramanian, B., Balu, A., Jubery, T.Z., Sarkar, S., Singh, A. and Singh, A.K., 2023, January. Optimized Class-specific Data Augmentation for Plant Stress Classification. In 2nd AAAI Workshop on AI for Agriculture and Food Systems.
• Type: Conference Papers and Presentations Status: Published Year Published: 2023 Citation: Jignasu, A., Herron, E., Jubery, T.Z., Afful, J., Balu, A., Ganapathysubramanian, B., Sarkar, S. and Krishnamurthy, A., 2023, January. Plant Geometry Reconstruction From Field Data Using Neural Radiance Fields. In 2nd AAAI Workshop on AI for Agriculture and Food Systems.
• Type: Conference Papers and Presentations Status: Published Year Published: 2023 Citation: Prasad, A.D., Jignasu, A., Jubery, Z., Sarkar, S., Ganapathysubramanian, B., Balu, A. and Krishnamurthy, A., 2022, February. Deep implicit surface reconstruction of 3D plant geometry from point cloud. In AI for Agriculture and Food Systems.

Progress 06/01/21 to 05/31/22

Outputs
Target Audience:The plant genetics community and breeders. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided?A genetic graduate student was involved and in charge of using PhieldCam to collect phenotype data. Not only learning the genetics and phenomics for this project, the student was trained in the overall project management skills that were needed for the planning, assembling and disassembling of the system, maintenance and trouble-shooting of the equipment. Two engineering graduate students were involved in this project who work on the development of PhenoBot. These students were exposed to the experience of optimizing automated image capture that will meet the need of the project while adapting to the external elements. A staff scientist, Dr. Talukdar Zaki Jubery continued to learn, translate and adapt cutting edge developments in machine vision to the PhieldCam experiment analytics pipeline. He has also been given the opportunity to mentor and guide several graduate students this past year. Finally, he has been working closely with collaborators from CyVerse to ensure that all software products and workflows are democratized and eventually available on CyVerse. Three engineering graduate and two undergraduate students were involved in this project developing and deploying sensors. These students learned and developed new measurement technologies that can improve the productivity and profitability of the agricultural industry, and advance soil and water conservation. The project provided the students with experience in advanced manufacturing, electronics design, and sensor applications for solving real-life challenges. Postdocs Dr. Ryan Bartelme and Dr. Michael Culshaw-Mauer from CyVerse were co-funded through this project. Both came from a background of microbial biology and worked with CyVerse to further develop their computational and data science expertise, with a strong desire to help other scientists through computational resources. Their involvement with HIPS permitted them to support agriculture researchers and learn more about agricultural sciences. Dr. Bartelme even took a strong interest in developing home gardening/hydroponics systems for indoor agriculture driven by raspberry pi micro-computers and leveraged people he met to create his home system. Dr. Bartelme was hired recently to work for Accelerate Diagnostics. How have the results been disseminated to communities of interest?We disseminated the research results through conference presentation, peer-reviewed journal publications and a Ph.D. thesis. Presentation by Baskar Ganapathysubramanian: - PSI faculty presentation, "Computational tools for field phenotyping", November 2021 - "Reducing annotation workload for plant phenotyping" Plant Phenomics Phridays, Iowa State University, Aug 6, 2021. - Digital twin approaches in agriculture", Department of Animal Science, Iowa State University, Nov 2021. - AG2PI field day, "AI and Agriculture: AIIRA", virtual presentation Aug 18, 2021 - DIGICROP 2022, "Plant phenotyping with limited annotation: Doing more with less", March 28, 2022 - USDA NIFA SAS PI meeting, "Collaborative opportunities for AI in Ag", April, 2022 Presentation by Lie Tang: - Conference presentations at the 2021 international annual meeting of the American Society of Agricultural, Biological Engineers (virtual). - Delivered an online workshop on "Developing Enabling Robotics Systems for Plant Phenotyping" under Agricultural Genome to Phenome Initiative (AG2PI). Training workshops were recorded with permission of the trainer and made available on YouTube; materials and documentation for workshops are hosted through CyVerse or AG2PI's website; software and data artifacts are made available through CyVerse or other cloud providers (e.g., Google Colaboratory, Binder) and linked through the workshop materials. Code is available via GitHub. Example: Introduction to Scientific Computing: Workshop Day 1: https://www.youtube.com/watch?v=Ogl9l6QJqJ8 Workshop Day 2: https://www.youtube.com/watch?v=onLL4Tk-8lI Workshop Day 3: https://www.youtube.com/watch?v=8iB-1rVodjU Google Colab Notebook: https://colab.research.google.com/drive/1hYEwdV4y54x7jhtKxtQjVfR6OzhUKDBK Homework: https://colab.research.google.com/drive/1c30bFwwcPJ86W6x6RiwmpElbbysQTawH Solutions: https://colab.research.google.com/drive/1Sx3ujwBkzXg0EpfH78vtaCaEl47LP6Zy#scrollTo=RptAD8wx6Ny9 GitHub Code: https://github.com/phytooracle/AG2PI_Introduction_to_Scientific_Computing What do you plan to do during the next reporting period to accomplish the goals?• Continue yield trial data collection at various nitrogen levels at sites across IA, NE and MO. Manually collect ground-truthing data for yield and plant traits. • Continue working on sensor optimization, installation, and data acquisition, with a focus on reducing power consumption and increasing the number of sensors of each type. • Complete the auto-navigation function and instrument the entire fleet of the PhenoBots. • Collect image data via UAV, PhenoBot and Satellite. Deploy tattoo, in-plant and in-soil sensors along with commercial soil sensors to collect data. • Continue developing workflow automation and feature extraction, with a focus on self-supervised learning algorithms • Continue to provide instructor and workshop support, and additional migration of training materials to be hosted in the CyVerse Learning Center. Two core scientific areas that are in demand for agricultural applications are remote sensing technologies (e.g., drones) and machine learning techniques (e.g., phenotyping from remote sensing data).?

Impacts
What was accomplished under these goals? Overall impact statement: We continued to make progress toward accomplishing project objectives. We collected image data by using ground- (PhieldCam and PhenoBot), aerial- (UAV) and field-based (agricultural sensors) platforms from our field sites in IA and NE. In addition to the development of this infrastructure, these data are also available for ground-truthing purpose for the community. Based on the data analytic pipeline developed in Year 1, we explored using domain adaptation and self-supervised learning to improve efficiency and extensibility of this pipeline. We successfully conducted the field trials and collected data which serves as a pilot for the more ambitious Year 2 yield trial. During the pandemic, we continued to provide resources and conducted trainings on these high-throughput methods for the communing through making available a class recording, seminars and workshops. In combination, these activities have the potential to facilitate a more mechanistic understanding of how phenotypes respond to genotypic and environmental variation, thereby facilitating the development of more resilient crop varieties that make more efficient use of agricultural inputs such as nitrogen and water, with corresponding environmental benefits. Accomplishments: Objective 1. The 45 ex-PVP hybrids and their 39 parents were imaged at the Ames site using the PhieldCam. ~2,000 images were collected per genotype. The 45 ex-PVP hybrids were imaged at our three IA sites using a DJI Phantom 4 Pro V2.0 UAV for both RGB & NDVI. All three sites were imaged in early June & then again mid-June At the North Platte, NE site we deployed two sensor types, including 24 soil water potential sensors and 24 wearable transpiration sensors. At Ames IA sites, we deployed three sensor types, including 24 soil nitrate sensors, 26 plant nitrate sensors, and 24 wearable transpiration sensors. Solar panels were installed. Each solar panel supported up to eight sensors at two plots. Eight hybrids were monitored with sensors. At North Platte and Ames, we were able to put sensors on one replicate of each treatment due to limited number of sensors available. On each site, there were 8 hybrids, 3 treatments and one replicate. More than 60% of the installed sensors provided continuous phenotypic data during the growing season. These sensors generated data that would help to understand the value of coupled direct soil and plant measurements of N concentrations and water status. We have continued to develop robotic solutions for field-based plant phenotyping to meet our goal of deploying new methods of sensing and data analytics to advance phenomics research. Improvements have been made or are underway in the PhenoBot's sensor mast design, instrumentation, navigation control, and the redesigned 3D stereo vision module - PhenoStereo. On top of the algorithms for maize plant stalk size sensing and leaf angle measurement, new algorithms for maize plant leaf area and leaf length sensing and soybean plant seed pod characterization are being developed. A large number of stereoscopic images for maize plant leaf angle measurement was collected using PhenoBot 3.0 and PhenoStereo in Ames and Boone, IA. Objective 2. Continued to improve the analytics pipeline developed in Year 1, based on python to enable broad usage and to ensure extensibility. This analytics pipeline can extract plant height, leaf angle, number of leaves, and growth rate from PhieldCam images. Usually, training such model typically requires the availability of copious amounts of annotated data; however, creating large-scale annotated dataset requires non-trivial efforts, time, and resources. This has become a major bottleneck in deploying such analytics tools in practice. Self-supervised learning (SSL) methods give exciting solution to this problem, as these methods use unlabeled data to produce pretrained models for subsequent fine-tuning on labeled data, and have demonstrated superior transfer learning performance on down-stream classification tasks. We explored domain adaptation and self-supervised learning for extracting the individual plants from PhieldCam images for subsequent statistical analysis. Objective 3. In the past year we successfully completed our Year 1 field trials, collecting grain yield and trait data from a set of ~80 replicated exPVP maize hybrids grown in ten environments across six locations in two states (IA and NE). We collected UAV data from a number of sites and have begun to process these data to enable us to extract images from individual plots at individual points in time. While the Year 1 trials did not yet contain sufficient numbers of genotypes to conduct GWAS and identify genes, these trials served as a pilot for the analyses, pipelines, and meta-data tracking that will be necessary to process the similar UAV based datasets we aim to collect in Year 2 (see below) which should enable proof of concept gene discovery. During the Spring 2022 we focused on establishing the much more ambitious Year 2 trials which will include both inbred and hybrid yield plots. It is the data from the inbred plots planned for Year 2, including more than 350 maize inbreds genotyped with high density markers, which should enable us to successfully complete Objective 3: identifying specific genes controlling phenomically measured traits. It was possible to expand the number of evaluation environments for the hybrids trials in Year 2 from 10 to 18, incorporating variable nitrogen rate applications at a larger number of locations. Objective 4. We have uploaded the video recordings of the ISU graduate course entitled, "GDCB 585: Foundations in Predictive Plant" to Vimeo. They are now available for dissemination to other institutions via this link: Vimeo 'Show Case' URL: https://vimeo.com/showcase/8796225. We held a total of 11 presentations in the Plant Phenomics Phridays seminar and discussion series. Recording of these presentations are available here: https://phenomics.iastate.edu/events/past In partnership with CyVerse and the NIFA-funded Agricultural Genomes to Phenomes Initiative (AG2PI), several virtual workshops were hosted over the past year. Most events were free and open for anyone to register, with workshops recorded and subsequently made available for general viewing on YouTube. Live attendance ranged from 15-150 people. For workshops with nominal registration fees (Foundation of Open Science Skills and Container Camps), HIPS participants received waivers. These training included: AG2PI Collaboration: Intermediate Omics Data-Enabled Genomics Prediction (July 26, 2022) AG2PI Collaboration: Bison-Fly: A UAV Pipeline Applied to Plant Breeding (July 12, 2022) AG2PI Collaboration: Crop Modeling as a tool for understanding GxE interaction (June 23, 2022) CyVerse Collaboration: Foundation of Open Science Skills (June 6-10, 2022) CyVerse Collaboration: Advanced Container Camp (May 17-19, 2022) CyVerse Collaboration: Basic Container Camp (May 12-13, 2022) AG2PI Collaboration: Introduction to Scientific Computing (April 8, 15, 22, 2022) AG2PI Collaboration: Hands-On Machine Learning with Agricultural Applications (Feb 18, 2022) AG2PI Collaboration: Developing Enabling Robotic Systems for Plant Phenotyping (Nov 12, 2021) CyVerse VICE for cloud-based interactive analyses (October 8, 2021) CyVerse Collaboration: Foundational Open Science SKills (Sept - Nov, 2021) AG2PI and CyVerse Collaboration: Unix Skills (Sept 8, 2021) AG2PI and CyVerse Collaboration: Git Skills (Sept 9, 2021) CyVerse Collaboration: Advanced Container Camp (Aug 2-4, 2021) CyVerse Collaboration: Basic Container Camp (July 26-28, 2021) AG2PI and CyVerse Collaboration: Phenomic Data Processing using Machine Learning, Distributed Computing, and Container Technology (July 22, 2021) AG2PI and CyVerse Collaboration: A practical guide to GWAS (June 24, 2021) AG2PI and CyVerse Collaboration: An Introduction to SNP analysis (May 20, 2021)

Publications

• Type: Journal Articles Status: Published Year Published: 2021 Citation: Xiang, L., L. Tang, J. Gai, L. Wang. 2021. Measuring stem diameter of sorghum plants in the field using a high-throughput stereo vision system. Transactions of the ASABE 64(6): 1999-2010. (doi: 10.13031/trans.14156)
• Type: Journal Articles Status: Published Year Published: 2021 Citation: Y. Chen, Z. Tang, Y. Zhu, M. J. Castellano, and L. Dong, "Miniature multi-ion sensor integrated with artificial neural network," IEEE Sensors Journal. 21(22), pp.25606-25615 (2021). DOI: 10.1109/JSEN.2021.3117573
• Type: Theses/Dissertations Status: Submitted Year Published: 2021 Citation: Chen, Yuncong. "In-situ soil water potential sensor and nutrient sensor." Iowa State University. Ph.D. Disseratation (2021). https://doi.org/10.31274/etd-20210609-36
• Type: Journal Articles Status: Under Review Year Published: 2022 Citation: Hussam Ibrahim, Shihao Yin, Moru Satyanarayana, Yunjiao Zhu, Michael J. Castellano, and Liang Dong, An in planta nitrate sensor using a photosensitive epoxy bio-resin (under review).
• Type: Journal Articles Status: Under Review Year Published: 2022 Citation: Ibrahim, Hussam; Moru, Satyanarayana; Schnable, Patrick; Dong, Liang, A wearable plant sensor for in-situ monitoring of volatile organic compound emissions from crops (under review)
• Type: Conference Papers and Presentations Status: Published Year Published: 2021 Citation: Esfandiari, Y., Nagasubramanian, K., Fotouhi, F., Schnable, P.S., Ganapathysubramanian, B. and Sarkar, S., 2021, September. Distributed Deep Learning for Persistent Monitoring of agricultural Fields. In NeurIPS 2021 AI for Science Workshop.
• Type: Conference Papers and Presentations Status: Published Year Published: 2021 Citation: Chiranjeevi, S., Young, T., Jubery, T.Z., Nagasubramanian, K., Sarkar, S., Singh, A.K., Singh, A., Ganapathysubramanian, B. Exploring the use of 3D point cloud data for improved plant stress rating. InAI for Agriculture and Food Systems 2021 Nov 20.
• Type: Conference Papers and Presentations Status: Published Year Published: 2021 Citation: Moothedath, S., Lee, X.Y., Jubery, T., Ganapathysubramanian, B. and Sarkar, S., 2021, November. A Conservative Stochastic Contextual Bandit Based Framework for Farming Recommender Systems. In AI for Agriculture and Food Systems.
• Type: Conference Papers and Presentations Status: Published Year Published: 2021 Citation: Prasad, A.D., Jignasu, A., Jubery, Z., Sarkar S., Ganapathysubramanian, B., Balu, A., Krishnamurthy, A. Deep implicit surface reconstruction of 3D plant geometry from point cloud. InAI for Agriculture and Food Systems 2021 Nov 20.

Progress 06/01/20 to 05/31/21

Outputs
Target Audience:The plant genetics community and breeders. Changes/Problems:As discussed above, challenges associated with the initial COVD lockdown requires the delay of our first year of field trials from summer 2020 to summer 2021. These trials have now been planted and, as of the end of this reporting period, exhibited good emergence and stand. What opportunities for training and professional development has the project provided?The project provided training opportunities to six engineering Ph.D. students (Yuncong Chen, Hussam Ibrahim, Shihao Yin, Raufur Khan, Yang Tian, and Qinming Zhang) in Co-PD Dong's lab at ISU. These students designed, manufactured, tested and installed sensor systems. In the sensor research community, generally, most graduate students are exposed to the development of individual sensing elements in the laboratory; there are very few opportunities of performing experiments with the sensors at the system level in the field. This project helped students to develop skills and competencies in sensor R&D through field research. A staff scientist, Dr. Talukdar Zaki Jubery continued to learn, translate and adapt cutting edge developments in machine vision to the PhieldCam experiment analytics pipeline. In response to the global pandemic, we adapted an online format for the course "GDCB 585: Foundations in Predictive Plant Phenomics" and offered to students. Although the course was not delivered due to insufficient enrollment, it is ready and scheduled to be deliver in Fall 2021. We also offered the Summer Field Phenomics Course both 2020 and 2021 albeit cancelled due to the pandemic. We will offer the course again in the summer of 2022. Although many planned activities for teaching advanced quantitative skills and data science were cancelled, we delivered the HIPS-focused FOSS and Container camp online and offered to all HIPS participants. How have the results been disseminated to communities of interest?The results have been disseminated to communities of interest through PhD theses, publications, the CyVerse website, and presentations at workshop and symposiums. What do you plan to do during the next reporting period to accomplish the goals?Objective 1: We had to delay the summer 2020 planting to summer 2021 due to the global pandemic. At the time of reporting, we have completed these plantings and are collecting phenotype data. We will continue to improve the performance of the field-based agricultural sensors and will increase the number of sensors installed at the two major sites. Additionally, we will complete the upgrades of the PhenoBots, implement and test its auto-steer function, equip the robot with the 2nd generation PhenoStereo cameras, and collect field scale images. Finally, we will conduct a workshop on "things to consider when selecting a plant phenotyping robot. Objective 2: We will continue developing workflow automation and feature extraction, with a focus on self-supervised learning algorithms. Objective 3: In 2021-2022, we will collect both manual measurements and high throughput phenotyping sensor data (UAVs, nitrogen and water sensors) from the research plots. We will conduct analyses to evaluate the proportion of variance in each individual traits which are explained by genetic or environmental factors as well as which traits exhibit significant genotype by environment interactions. Subsequent to the conclusion of the 2021 field season, we will improve standard operating procedures, troubleshoot any errors or omissions in data collection and scale up hiring in preparation for the larger scale field experiments planned for the same sites with the same data collection methods in 2022. Towards the end of Spring of 2022, we will begin planting these larger-scale field experiments at all three locations and collect data on emergence and stand from each. Objective 4: We will offer the course "GDCB 585: Foundations in Predictive Plant Phenomics" for enrollment in Fall 2021. Course evaluation and student experience will be use to adapt the course for Coursera. The trainings in quantitative skills in engineering and data science will resume as the pandemic situation allows. The FOSS and HIPS-focused Container Camp will remain available via CyVerse.

Impacts
What was accomplished under these goals? Overall impact statement: Much effort in plant breeding has been focused on identifying and developing cultivars of high yield traits in the past decades. With the increasingly unstable weather patterns, it is important to incorporate environmental factors such as weather, location, and crop management practices to improve breeding strategies. For example, instead of focusing on yields only, plant cultivars that will perform across various locations and weather patterns will be of value in improving food security. Creating such cultivars is a daunting task with current plant breeding strategies. It takes 7-10 years to generate new varieties and the resulting cultivar may not adapt to the environment 10 years later. This year, despite the limitations imposed by the global pandemic we have made progress in the proposed objectives. We are moving towards developing high-throughput data acquisition and analysis methods to collect data from the plants and their environment. These technologies will enable the optimization of accurate predictive models which incorporate the combined effects of crop variety and environmental factors. These models will not only greatly shorten the time needed for developing new cultivars, they may help to create a more sustainable, resilient and profitable US agricultural system. Some of the image capture and analyses have been published for the research community to access. Several training courses were made available to the research community via an online platform, which will expose students and researchers to these high-throughput methods. Accomplishment under each relevant objective: Objective 1. Deploy and evaluate for use by G2F, a combination of established, image-based sensing technologies and highly promising new field-based agricultural sensors (for nitrate and water) and generating and sharing reference phenomic data to foster community innovation. During 2020-2021, we worked with HIPS location site collaborators & service providers to plan out HIPS locations. Working with site collaborators, we determined where the sites would be, gained soil type information about the sites, and then determined experimental designs to allow for optimal sensor deployment. We prepared seeds for all HIPS sites and shipped out to HIPS collaborators and service providers for planting in the summer of 2021. Weather stations were deployed at both IA and NE sites. All sensors were prepared and tested for the deployment during the summer of 2021. We field tested the prototype of PhenoBot 3.0 and PhenoStereo during the summer 2020. Important lessons were learned in how to improve the robot's reliability and useability. On the trait extraction side, we also developed and tested a new deep learning algorithm - AngleNet - for leaf angle detection and characterization by using PhenoStereo cameras onboard of the PhenoBot. Navigation algorithms were tested with a fully equipped PhenoBot. However, we realized that the design of the sensor mast has some flaws which seriously affected the stability and safe operation of the robot. Since then, we have designed, assembled and tested the new sensor mast, which exhibits significant improvements in stability and controllability. We also found that PhenoStereo cameras could overheat on hot summer days in the field, and have thus developed the 2nd generation of the PhenoStereo. When pairing the PhenoStereo with a series of traits characterization algorithms such as our "StalkNet" and "AngleNet", an automated 3D morphological trait acquisition tool can be developed for maize plants. Objective 2. Develop and democratize analysis pipelines for phenotypic data. We created an analytics pipeline, based on Python, to enable broad usage and to ensure extensibility. This analytics pipeline can extract plant height, leaf angle, number of leaves, and growth rate from PhieldCam images. We use the popular segmentation framework UNet with ResNet as a backbone during pipeline creation, intelligent sampling for choosing and annotating training data, and port algorithms from a previously created software ARIA (https://bitbucket.org/baskargroup/aria2.1/src/master/) Objective 3. Conduct proof-of-principle research projects to identify agronomically relevant genes from phenomic data. As a result of delays associated with the global pandemic in summer 2020, our year one field research was initiated in May of 2021. In May of 2021 we completed planting of the year 1 research field plots of exPVP maize hybrids as planned. Initial emergence and stand count data look promising and we anticipate being able to successfully complete our year 1 data collection and analysis objectives during 2021. Objective 4. Contribute in a substantial manner to the training of current and future agricultural researchers to make use of these innovations. We planned, prepared and adapted courses for the training of graduate students in conducting transdisciplinary research. In late summer of 2020, we adapted remote delivery method for the course "GDCB 585: Foundations in Predictive Plant Phenomics". This was not hard for the lecture portion of the course given that lectures could be recorded and delivered to students, but it posed real challenges for the lab portion of the course. To address the need for students to grow and interact with live plants, we reworked the lab portion of the curriculum to enable students to grow and work with plants remotely using commercially-available hydroponics kits. Coupled with this controlled-environment system, students would use an array of sensors and other data collection devices to measure plant phentoypesphenotypes over time and growth stages, and would deliver a seminar on what they measured toward the end of the semester. Unfortunately, enrollment was not sufficient to deliver the course, but these preparations remain excellent changes to the curriculum that enable broad dissemination of the course. This is good for our next step, which is to adapt the course for delivery broadly via Coursera. The course is now scheduled for Fall 2021 delivery, with enrolled students trying out the adapted curriculum and offering feedback prior to Coursera remote course deployment. Likewise, we planned the Summer Field Phenomics Course (for both Summer 2020 and Summer 2021) but had to cancel them due to the global pandemic. 2020 course organization efforts are foundational for Summer 2022 deployment, and the impact of effort from this grant will be kept by expanding the number of students and their support to expand engagement at that time. During the reporting period, many planned activities we proposed under the subobjective to teach advanced quantitative skills in engineering and data science were delayed due to the pandemic. However, some were adapted for remote delivery with great success. We delivered one HIPS-focused Foundational Open Science Skills (FOSS) course and one HIPS-focused Container Camp via CyVerse which are available to all HIPS participants. For the remaining trainings, the originally scheduled number of offerings will be made up by increasing the number of trainings in years two and three.

Publications

• Type: Journal Articles Status: Published Year Published: 2021 Citation: Mirnezami SV, Srinivasan S, Zhou Y, Schnable PS, Ganapathysubramanian B. Detection of the Progression of Anthesis in Field-Grown Maize Tassels: A Case Study. Plant Phenomics. 2021 Mar 3;2021.
• Type: Journal Articles Status: Published Year Published: 2021 Citation: Kusmec A, Zheng Z, Archontoulis S, Ganapathysubramanian B, Hu G, Wang L, Yu J, Schnable PS. Interdisciplinary strategies to enable data-driven plant breeding in a changing climate. One Earth. 2021 Mar 19;4(3):372-83.
• Type: Journal Articles Status: Published Year Published: 2021 Citation: Yin, S., Ibrahim, H., Schnable, P.S., Castellano, M.J. and Dong, L., 2021. A Field?Deployable, Wearable Leaf Sensor for Continuous Monitoring of Vapor?Pressure Deficit. Advanced Materials Technologies, 6(6), p.2001246. https://doi.org/10.1002/admt.202001246.
• Type: Journal Articles Status: Published Year Published: 2021 Citation: Wang, L., L. Xiang, L. Tang, H. Jiang. 2021. A convolutional neural network-based method for corn stand counting in the field. Sensors 21, 507. DOI: 10.3390/s21020507
• Type: Journal Articles Status: Published Year Published: 2021 Citation: Gai, J., L. Xiang, L. Tang. 2021. Using a depth camera for crop row detection and mapping for under-canopy navigation of agricultural robotic vehicle, Computers and Electronics in Agriculture 188, DOI: 10.1016/j.compag.2021.106301.
• Type: Conference Papers and Presentations Status: Other Year Published: 2021 Citation: PSI faculty presentation, Computational tools for field phenotyping, April 2021 by Baskar Ganapathysubramanian.
• Type: Journal Articles Status: Published Year Published: 2020 Citation: Kenchanmane Raju, S.K., Adkins, M., Enersen, A., Santana de Carvalho, D., Studer, A.J., Ganapathysubramanian, B., Schnable, P.S. and Schnable, J.C., 2020. Leaf Angle eXtractor: A high-throughput image processing framework for leaf angle measurements in maize and sorghum. Applications in plant sciences, 8(8), p.e11385.
• Type: Theses/Dissertations Status: Submitted Year Published: 2021 Citation: PhD dissertation: In-situ soil water potential sensor and nutrient sensor, Yuncong Chen, Iowa State University, 2021. https://lib.dr.iastate.edu/etd/18475/