ALFALFA YIELD PREDICTION USING UAV-BASED HYPERSPECTRAL IMAGERY AND MACHINE LEARNING

• Sponsoring Institution: National Institute of Food and Agriculture
• Project Status: COMPLETE
• Funding Source: HATCH
• Reporting Frequency: Annual
• Accession No.: 1020412
• Project Start Date: Oct 1, 2019
• Project End Date: Sep 30, 2021
• Program Code: [(N/A)]- (N/A)

Recipient Organization
UNIV OF WISCONSIN
21 N PARK ST STE 6401
MADISON,WI 53715-1218

Performing Department
Biological Systems Engineering

Non Technical Summary
Alfalfa is a valuable and intensively produced perennial forage crop in the United States, and it plays an important role in the food supply chain as a feedstock for animals, specifically dairy and beef cattle and the equine market.Pre-harvest insight to yield can enable real-time management responses to fluctuating market conditions or feed needs. Therefore, methods capable of rapid and accurate predictions at the field-scale are needed to improve the economic efficiency and resilience of alfalfa production.Traditional yield assessment methods are time consuming and difficult or even impossible to practice in large areas. This project aims to use UAV-based remote sensing technology for field data collection, and develop a statistical model for alfalfa yield prediction. Specifically, a hyperspectral camera will be integrated with the UAV platform for weekly data acquisition. Then, data fusion strategies will be applied for processing the time series data to characterize the crop growth dynamics. Finally, a statistical model will be developed based on the times series hyperspectral imagery and the yields obtained from prior harvests.

Animal Health Component

(N/A)

Research Effort Categories

Basic

60%

Applied

(N/A)

Developmental

40%

Classification

Knowledge Area (KA)	Subject of Investigation (SOI)	Field of Science (FOS)	Percent
402	7210	2020	100%

Knowledge Area
402 - Engineering Systems and Equipment;

Subject Of Investigation
7210 - Remote sensing equipment and technology;

Field Of Science
2020 - Engineering;

Keywords

alfalfa yield

unmanned aerial vehicle

hyperspectral remote sensing

machine learning

Goals / Objectives
Alfalfa yield is determined by a complex interplay of processes and varies across time and locations. Hyperspectral data, that contain fruitful spectral information representing the internal relations between radiation energy and biophysical characteristics of agricultural crops, and therefore can be used as a promising approach for yield prediction. Recent progress in unmanned aerial vehicles (UAVs) have advanced traditional satellite-based capabilities, providing a capacity for high-spatial, spectral and temporal response. This study aims to build a machine learning model for alfalfa yield prediction by using the UAV-based hyperspectral imagery, and three specific objectives for the proposed research are as follows:1.Design an UAV-based hyperspectral imaging platform for agricultural data collection.2.Develop feature extraction and feature fusion approaches to extract the informative features from the time series hyperspectral images.3.Build a machine learning model for alfalfa yield prediction by using the hyperspectral image features and yields obtained in prior harvests, and compare its performance with the traditional model using a simple vegetation index.

Project Methods
This proposed research encompasses the development of the UAV-based hyperspectral sensing system and demonstrate its applicability for alfalfa yield prediction. To accomplish this, the following methods will be used:1. System Integration and Data Collection: The Headwall Nano-Hyperspec pushbroom scanner will be integrated with the DJI M600 Pro UAV platform. A DJI Ronin gimbal stabilizer will also be used to stabilize the hyperspectral camera during the flight. With the integrated system, the hyperspectral data will be collected weekly at two alfalfa fields during the summer and fall seasons over two years. Ground control targets will be deployed in the fieldand their ground coordinates will be surveyed by an RTK GPS receiver. Spectral calibration panels will also be deployed in the field, and their images will be captured to provide calibration values for the hyperspectral camera. Yield data will be collected from the study field at each harvest.2. Time Series Data Processing: To alleviate the high dimensionality issue of the hyperspectral data, feature extraction approaches will be employed to extract and preserve essential information in a much lower dimensional feature space, prior to perform the prediction step. Two types of feature extraction methods will be used in this study, including the non-linear manifold learning methods and the supervised metric learning methods.Additionally, an important novel direction that will be pursued is to learn unified features for multiple modalities with each modality devoting to the data collected at a specific time point. The single manifold and metric learning strategies discussed above will be extended into multimodal feature setting, which aims to learn multiple manifolds or metrics jointly in order to exploit the relationship among different features.3. Build a Machine Learning Model for Alfalfa Yield Prediction:After the feature extraction and feature fusion, the learned features along with the yields obtained from prior harvests will be used as input of the prediction models.Specifically, the following five machine learning models will be explored and compared, including Support Vector Regression (SVR), Stochastic Gradient Boosting (SGB), Multi-layer perception (MLP), Convolutional neural network (CNN), and Long-short term memory (LSTM).4. Results Evaluation: A cross-validation strategy will be used for model evaluation. Specifically, the available data points will be randomly partitioned into four subsets of equal size. In each round, one of the four subsets was retained as the independent test set, whereas the remaining three subsets were used as training data. We evaluated the model performance, by quantifying the following metrics based on the testing data for each round, including 1) the root mean squared error (RMSE), 2) the coefficient of determination (R2), and also 3) the ratio of performance to interquartile distance (RPIQ), which is defined as interquartile range of the observed values divided by the RMSE. The RPIQ takes account of both the prediction error and variation of observed values, and therefore it is more objective than the RMSE and more easily to compare among models. The final accuracy was summarized by averaging these metrics from the four rounds and the corresponding standard deviations were also recorded.

Progress 10/01/19 to 09/30/21

Outputs
Target Audience:The target audiences will include: 1) students in agriculture science and engineering: The UAV-based remote sensing knowledge and data analysis approaches will be taught through formal classroom instruction. The software used for processing the hyperspectral data will be demonstrated through laboratory instruction. 2) alfalfa growers: The developed yield prediction model will help growers make more efficient use of farm input resources, and it can also help them make plans for the harvest, processing, and transport of the crop. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided?A PhD student has been supported and trained through this project. A few prestigious journal articles have been published by the PhD student and he has made successful presentations at academic conferences with the support of this project. Overall, this student has been well-trained and impressive progress has been made over the past two years. How have the results been disseminated to communities of interest?Results have been disseminated through multiple channels. Specifically, the results have been (1) published in several peer-reviewed journal articles; (2) presented at academic conferences (e.g., ASABE, AGU, etc.) where stakeholders are typically present; (3) presented at invited seminars hosted by different organizations, such as Weston Roundtable at Nelson Institute, Forage Genetics International; (4) communicated to alfalfa breeders at USDA-ARS. What do you plan to do during the next reporting period to accomplish the goals? Nothing Reported

Impacts
What was accomplished under these goals? The outcomes of the project produced a change in knowledge in helping students and alfalfa stakeholders gain experience in applying hyperspectral sensing and machine learning technologies for alfalfa yield and quality estimation. Also, it helped produce a change in action for assessing alfalfa and other forage crop yield and quality, such that rapid estimations can be achieved for improved farm management decisions. For each specific objective, we have accomplished the follows: Objective 1: Design a UAV-based hyperspectral imaging platform for agricultural data collection. A UAV-based hyperspectral imaging system was established through the project. Specifically, the hyperspectral camera was integrated with a high-performance GNSS/INS navigation system for direct data geo-referencing. The imaging system has been used for collecting data over not only alfalfa fields but also other crop fields, including grain corn, corn silage, and potato, over the past 2 years. The system is continuously being used for other research work. Objective 2:Develop feature extraction and feature fusion approaches to extract the informative features from the time series hyperspectral images. A feature reduction method was successfully developed for hyperspectral feature extraction and fusion. The method was used in conjunction with an ensemble learning model for alfalfa yield prediction. The results showed the reduced features selected using the proposed method could achieve better yield modeling performance than using all the features without feature reduction. The results were published in a peer-reviewed journal article. Objective 3: Build a machine learning model for alfalfa yield prediction by using the hyperspectral image features and yields obtained in prior harvests and compare its performance with the traditional model using a simple vegetation index. An ensemble learning model that combines multiple individual single learning models was developed for alfalfa yield prediction. The model was validated on alfalfa yield data and the results demonstrated that the developed ensemble model outperformed the other state-of-the-art single learning models. Besides yield, we also developed a multi-task learning model for the prediction of multiple alfalfa quality traits including crude protein (CP), acid detergent fiber (ADF), and neutral detergent fiber (NDF) concentrations. This multi-task learning model was able to estimate the multiple quality traits in one framework and its performance outperformed the traditional single-task learning methods, and the results were validated on alfalfa quality data.

Publications

• Type: Journal Articles Status: Published Year Published: 2021 Citation: Feng, L., Wang, Y., Zhang, Z., & Du, Q. (2021). Geographically and temporally weighted neural network for winter wheat yield prediction. Remote Sensing of Environment, 262, 112514.
• Type: Journal Articles Status: Published Year Published: 2021 Citation: Ma, Y., Zhang, Z., Yang, H. L., & Yang, Z. (2021). An adaptive adversarial domain adaptation approach for corn yield prediction. Computers and Electronics in Agriculture, 187, 106314.
• Type: Journal Articles Status: Published Year Published: 2021 Citation: Ma, Y., Zhang, Z., Kang, Y., & �zdo?an, M. (2021). Corn yield prediction and uncertainty analysis based on remotely sensed variables using a Bayesian neural network approach. Remote Sensing of Environment, 259, 112408.
• Type: Journal Articles Status: Published Year Published: 2022 Citation: Feng, L., Zhang, Z., Ma, Y., Sun, Y., Du, Q., Williams, P., ... & Luck, B. (2021). Multitask Learning of Alfalfa Nutritive Value From UAV-Based Hyperspectral Images. IEEE Geoscience and Remote Sensing Letters.
• Type: Journal Articles Status: Published Year Published: 2019 Citation: Zhang, Z., Pasolli, E., & Crawford, M. M. (2019). An adaptive multiview active learning approach for spectralspatial classification of hyperspectral images. IEEE Transactions on Geoscience and Remote Sensing, 58(4), 2557-2570.
• Type: Conference Papers and Presentations Status: Published Year Published: 2019 Citation: Ma, Y., Kang, Y., Ozdogan, M., & Zhang, Z. (2019, December). County-level corn yield prediction using deep transfer learning. In AGU Fall Meeting Abstracts (Vol. 2019, pp. B54D-02).

Progress 10/01/19 to 09/30/20

Outputs
Target Audience:The research outcomes have been published in peer-reviewed journal articles, presented at conference meetings and research seminars. Therefore, a large number of researchers in this area could benefit from the outcomes of this project. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided?A graduate student is being trained under this project. The student also presented the rebsearch outcomes at an international conference meeting. How have the results been disseminated to communities of interest?The research outcomes have been disseminated through peer-reviewed journal publications, conference presentations, and invited research seminars. What do you plan to do during the next reporting period to accomplish the goals?1. Develop novel machine learning models to integrate the time series hyperspectral data to further enhance the prediction accuracy. 2. Develop machine learning models to estimate alfalfa nutritive values.

Impacts
What was accomplished under these goals? Alfalfa yield is determined by a complex interplay of processes and varies across time and locations. Hyperspectral data, that contain fruitful spectral information representing the internal relations between radiation energy and biophysical characteristics of agricultural crops, and therefore can be used as a promising approach for yield prediction. Recent progress in unmanned aerial vehicles (UAVs) have advanced traditional satellite-based capabilities, providing a capacity for high-spatial, spectral, and temporal response.This project aims to build a machine learning model for alfalfa yield prediction by using the UAV-based hyperspectral imagery. Specifically, we have accomplished: 1. System Integration and Data Collection: The hyperspectral data were acquired by a Headwall nano-hyperspec push-broom scanner.This sensor covers 273 spectral bands ranging between 400-1000 nm with a bandwidth of 2.2 nm.Each scan line contains 640 pixels with a pixel pitch of 7.4 um. A VectorNav (VN)300 GNSS/INSnavigation system was integrated with the hyperspectral camera to directly provide the position andorientation of the camera for data geo-referencing. The DJI Matrice 600 Pro (M600) was used as theUAV platform. Furthermore, a DJI Ronin MX three-axis gimbal stabilizer was used on the M600airframe. By using the gimbal, the sensor can maintain a nadir view regardless of airframe orientation.This capability can help stabilize the hyperspectral camera during the flight, leading to improveddata geometry quality. Multiple UAV flights were conducted in 2020.The UAV was flying at a speed of 5 m/s from an altitude of 40 m, andthe corresponding ground sampling distance (GSD) was 2.5 cm. 2. Feature extraction: The acquired hyperspectral data contain hundreds of spectral bands and many adjacent bands are highly correlated. To reduce the data dependency, instead of using all the original bands, we extracted the narrow-band indices as spectral features and used them for modeling the alfalfa yield in this study. Specifically, we explored 80 published indices, with each derived from two or more spectral bands. The indices included simple ratio index (SRI), NDVI, REPI, chlorophyll absorption ratio index, modified versions of these indices such as mND705 and combination of them such as TCARI/OSAVI1. Although the calculations are varied, most wavebands used were in the red and NIR ranges. Feature selection was then performed to reduce the data dimensionality. Specifically, a recursive feature elimination approach was used for feature selection. 3. Model development: To enhance the prediction performance, an ensemble model was proposed based on a stacking strategy including the following two steps: (1) train and apply multiple-machine learning models independently; and (2) combine multiple prediction results via a linear regressor. It is important to select appropriate base learners to develop a successful ensemble model, and diversity is a critical condition, as the similarity between different models must be minimized for providing complementary information. SVR, K-nearest neighbors (KNN), and RF are commonly used machine learning approaches with distinct principles, and their predictive ability for crop yield prediction has been assessed by many studies. Therefore, we used these three models as the base learners.

Publications

• Type: Journal Articles Status: Published Year Published: 2020 Citation: Sun, C., Feng, L., Zhang, Z., Ma, Y., Crosby, T., Naber, M., Wang, Y., Prediction of end-of-season tuber yield and tuber set in potatoes using in-season UAV-based hyperspectral imagery and machine learning. Sensors, 2020, 20(18), 5293.
• Type: Journal Articles Status: Published Year Published: 2020 Citation: Feng, L., Zhang, Z., Ma, Y., Du, Q., Williams, P., Drewry, J., and Luck, B., Alfalfa Yield Prediction Using UAV-Based Hyperspectral Imagery and Ensemble Learning. Remote Sensing, 2020, 12, 2028.
• Type: Journal Articles Status: Published Year Published: 2020 Citation: Wang, Y., Zhang, Z., Feng, L., Du, Q., and Runge, T., Combining multi-source data and machine learning approaches to predict winter wheat yield in the conterminous united states. Remote Sensing, 2020, 12, pp.275-295.
• Type: Conference Papers and Presentations Status: Published Year Published: 2019 Citation: Ma, Y., Kang, Y., Ozdogan, M. and Zhang, Z., County-level corn yield prediction using deep transfer learning. American Geophysical Union Fall Meeting, Dec. 2019, San Francisco, USA.