Non Technical Summary
Information on cotton crop development such as maturity, fruit retention patterns, and plant architecture is limited to non-existent for most breeding and official variety trial programs in the U.S. due to the infeasibility of time-consuming manual measurements. The automated phenotyping technologies and resulting big data would empower cotton breeders to make greater genetic gains and shorten selection cycles. To address the challenges for researchers to effectively manage and efficiently process phenomics data, the overall goal of this project is to develop data science tools and AI-based models to facilitate breeding cotton germplasm with improved ideotypes. Specific objectives of the proposed project are to: 1) manage heterogeneous field data by developing data curation, storage, and sharing workflows; 2) develop real-time video tracking and 3D plant part segmentation deep learning models to measure cotton organ-level high resolution phenotypes; and 3) validate and leverage phenomics data for ideotype breeding to maximize cotton yield. Significant improvement on breeding efficiency would make U.S. cotton industry more sustainable and competitive in the global market. The methodologies developed in this project will also benefit data driven decision support systems for other crops and make the AI-based tools more broadly available to crop breeders and stakeholders. This standard research project is related to the Program Area "Data Science for Food and Agricultural Systems (DSFAS)" and addresses the AFRI priority area of "Plant health and production and plant products". In particular, this project focuses on AI and machine learning for monitoring, analytics, and automation in crop development.

Classification

Knowledge Area (KA)	Subject of Investigation (SOI)	Field of Science (FOS)	Percent
404	1719	1081	100%

Knowledge Area
404 - Instrumentation and Control Systems;

Subject Of Investigation
1719 - Cotton, other;

Field Of Science
1081 - Breeding;

Goals / Objectives
Our long-term goal is to develop data science tools and AI-based models to facilitate breeding cotton germplasm with improved ideotypes. We will develop a workflow and data management framework to collect, curate, store, and share the large volume, multi-source heterogeneous data following the FAIR (Findable, Accessible, Interoperable, and Reusable) principle. Focusing on measuring crop growth and yield-related traits with high-resolution imagery, we will develop video tracking and 3D deep learning models for cotton plant organ mapping and architectural traits relating to yield components, thereby providing an alternative path to increasing yield to complement traditional breeding methods. Specifically, in this proposed project we seek to: 1) Manage heterogeneous field data by developing data collection, curation, storage, and sharing workflows. 2) Develop real-time video tracking and 3D plant part segmentation deep learning models for plant organ-level high-resolution phenotyping. 3) Apply phenomics data for timely decision support on genotype selection to evaluate segregating populations for ideotype breeding.

Project Methods
Objective 1. Manage multi-scale heterogeneous field data by developing data collection, curation, storage, and sharing workflows. The proposed system is based on a three-tier computing system connecting field, local, and remote computing resources. The system will implement edge computing and FAIR data management principles to handle big data challenges facing modern plant phenomics. In the field, phenotyping systems with edge computing devices can perform not only data collection, but also data reduction, pre-processing, and online inference (Objective 2) using light-weight deep learning models. Pre-processed data along with the meta-data will be uploaded via a wireless network to on-farm server computers that automatically perform data decompression and integrity checks and organize and transfer retrieved raw data to cloud-based storage services. Ultimately, the collected raw data and meta-data will be transferred to cloud-based services for storage, processing, visualization, and sharing. We will leverage cloud computing and deep learning hardware (GPU-enabled computing nodes) from both the HiPerGator super computer and CyVerse for tasks such as CNN and 3D point cloud deep learning model training. The model training will be mainly done using the data collected in prior years, but new data will be collected to fine tune the model on the cloud using continual learning. The trained model will be deployed on the edge computing devices for inference.Objective 2: Plant organ-level high resolution phenotyping through real-time video tracking and 3D deep learning models. Objective 2.1: Cotton flowering habits and yield estimation using online multi-object tracking with inference on the edge. A deep learning based multi-object detection and tracking method from video frames will be deployed on the edge device to count stand, flowers, and mature bolls in real-time (except the detection model training on the cloud). We will take the tracking by detection paradigm which consists of three main steps: detection, motion estimation, and data association. We will develop a deep learning-based optical flow motion estimation model Recurrent All-Pairs Field Transforms (RAFT) to estimate the nonlinear motions of the camera due to disturbances from the uneven terrain. We will adopt the light weight version RAFT-S for online processing. Three video cameras will be mounted on a ground robotic phenotyping platform to record multiple-view (top and two side-view) videos of cotton flowers and bolls, whereas one top-view video camera will be adequate for seedling counting. The three-view videos can be registered based on camera intrinsic and extrinsic parameters to remove duplicated counts from overlapping areas. Tracking performance will be evaluated by the widely used CLEAR MOT Metrics. In addition, the efficiency (in seconds) of each stage of the method, including data preprocessing, detection, motion estimation and data association will be recorded. Objective 2.2: 3D plant mapping through point cloud segmentation using deep learning. Boll mapping and plant architecture (such as node of first fruiting branch, number of nodes to the first fruiting branch, and branching pattern) in cotton can provide detailed insights into the genetic basis of crop maturity, a critical juncture in which the plants transition from its vegetative growth phase to reproductive growth. In this objective, we will acquire high resolution 3D point cloud data for boll detection and plant architecture assessment. The 3D point cloud data will be collected by a terrestrial LiDAR sensor mounted on the ground robot with an RTK GPS to scan the plant from different perspectives. Given that this is an end-of-the-season measurement, three plants from each plot will be cut and moved to the warehouse near the field to be scanned in a batch mode (10 plants without occlusions will be scanned at a time). The computing performs at the three levels: sensing platform, farm gateway computer, and the cloud. We will evaluate the segmentation performance of the deep learning model in terms of IoU, Recall, and Precision. The overall segmentation performance is evaluated through mean IoU (mIOU) and Accuracy. The efficiency of the method is evaluated in terms of average inference time per point cloud. Novel architectural traits will be defined and evaluated by co-PI Chee's lab.Objective 3: Applying phenomics data in a phenotypic selection system towards ideotype breeding. The goal of this objective is to assess the genetic complexity of the traits obtained from plant organ-level high resolution and 3D plant mapping phenotyping described in Obj.2, which will guide cotton breeders to identify specific target traits that are heritable, and therefore amenable to selective breeding to design a preferred cotton ideotype for their production environments. We will leverage the genetic resources and analytical tools developed from a previously funded NIFA project to defray the costs for population development, field evaluation, and SNPs genotyping of the genetic population, while focusing on novel phenotypic traits such as flowering habits, boll positions, and plant architecture. In year 1, we will plant the 500 breeder's selections and 150 "genetic control" lines in the F4 nursery in a single row, repeating block design so that an estimate of plot yield can be obtained, corrected for nearby field variation. We will narrow down the 500 breeder's selection down to 120 lines (on the basis of yield and fiber quality) to be planted for the following two years of field trials. In the third year of the project, the initial F4 population will have made it to the end of the F6 trial (end of year 2 of the funding period), for each genotype we will have obtained image-derived data observed from a total of 6 plots in two location-years of field trials, including 1,440 plots with phenotypes assessed, giving us an accurate picture of trait stability and the distribution of phenotypes in each environment. The broad sense heritability (H2) of all measured traits, calculated from the ratio of total genetic variance to total phenotypic variance, will be estimated by parent-offspring regression (F5 to F6). We will leverage the expertise from Hulse-Kemp's Lab at North Carolina State Univ. to run genomic selection models across generations and different data subsets. Our long-term goal here is to develop a genomic selection model that can predict a genotype's performance (breeding values) with the highest accuracy, thereby allowing breeders to identify genotypes with superior plant ideotypes based on genotype data with a better balance of accuracy and cost/time than phenotyping data alone.