AN AI-BASED GROUND ROBOTIC VISION SYSTEM FOR AUTOMATED INVENTORY AND QUALITY ASSESSMENT OF BAREROOT SEEDLINGS IN FOREST TREE NURSERY PRODUCTION

• Sponsoring Institution: National Institute of Food and Agriculture
• Project Status: COMPLETE
• Funding Source: AFRI COMPETITIVE GRANT
• Reporting Frequency: Annual
• Accession No.: 1028791
• Grant No.: 2022-67021-37860
• Cumulative Award Amt.: $548,897.00
• Proposal No.: 2021-11086
• Project Start Date: Aug 1, 2022
• Project End Date: Jul 31, 2025
• Grant Year: 2022
• Program Code: [A1521]- Agricultural Engineering

Recipient Organization
AUBURN UNIVERSITY
108 M. WHITE SMITH HALL
AUBURN,AL 36849

Performing Department
College of Agriculture

Non Technical Summary
Accurate inventory of seedling stock is crucial to commercial forest tree nurseries for planning of stock shipments and outplantings. The U.S. forestry tree nursery industry currently relies on manual labor for sampling-based seedling inventories on a large scale at every nursery. The process is labor-intensive, time-consuming, error-prone, and ergonomically poor for workers. As the U.S. farm labor supply is expected to continue to decline in the long term, an automated seedling inventory technology is needed to meet the national and global goals for sustainable practice. The main objective of this project is to develop and evaluate an AI-based ground robotic vision system for automated inventory and quality assessment (i.e., stem diameter, shoot height, and health status) of bareroot pine seedlings at stand level. Deep convolutional neural networks, actively-illuminated 3D stereo imaging, and field robotics will be integrated into an autonomous ground-based seedling detection and measurement system. The system performance will be extensively evaluated against multiple factors, i.e., year, location, pine species, seedlot and growth stage. The secondary objective is to develop and evaluate a GIS decision support system that enables management, visualization, and spatial analysis of the automated seedling inventory and quality data.The proposed precision sensing and information management technologies will provide near-term and long-term solutions for nursery managers to improve efficiency, profitability and sustainability of forestry tree seedling production.

Animal Health Component

40%

Research Effort Categories

Basic

30%

Applied

40%

Developmental

30%

Classification

Knowledge Area (KA)	Subject of Investigation (SOI)	Field of Science (FOS)	Percent
404	7210	2020	45%
123	0611	3100	10%
402	5310	2020	45%

Knowledge Area
404 - Instrumentation and Control Systems; 123 - Management and Sustainability of Forest Resources; 402 - Engineering Systems and Equipment;

Subject Of Investigation
0611 - Conifer forests of the South; 5310 - Machinery and equipment; 7210 - Remote sensing equipment and technology;

Field Of Science
2020 - Engineering; 3100 - Management;

Keywords

artificial intelligence

automation

forest nursery management

seedling inventory

Goals / Objectives
The long-term goal of this project is to automate seedling inventory and quality assessment for the U.S. forest tree nursery industry.The four specific objectives of this project are to 1) develop a ground robotic stereo imaging platform for bareroot pine seedling productionfields,2)develop a deep learning-based stereo video processing pipeline that can performdetection, tracking, root collar diameter estimation, shoot height estimation and healthstatus quantification for individual seedlings,3)evaluate the system performance against year, location, pine species, seedlot, and growthstage,and4) develop and evaluate a GIS decision support system that enables management,visualization and spatial analysis of the automated seedling inventory and quality data.

Project Methods
A ground robotic stereo imaging platform will be developed to autonomously map seedling density and quality for bare-root pine seedling production fields. An electric UGV will be built as the mobility platform with four servo hub motors to realize a 4-wheel-drive skid-steering mechanism. The navigation system will integrate a suite of sensors including a RTK-GNSS module, an IMU, a magnetometer, wheel encoders, and a stereo camera. The start and end points of each seedling bed will be surveyed with a RTK-GNSS unit to create way points along the beds for global path planning of the UGV. Local path tracking will rely on track detection based on the stereo camera. A real-time RANSAC-based 3D plane segmentation algorithm will be developed to detect tracks and estimate UGV heading and lateral position errors leveraging the height difference between seedling beds and tracks. The extended Kalman filter will be used to fuse all navigation sensor data for robot localization and pose estimation, and the pure pursuit algorithm will be used for path tracking. Robot Operating System (ROS) and GAZEBO will be employed to simulate the UGV and field conditions and develop the navigation system. Four synchronized stereo cameras with high-intensity LED strobe lights will be carried by the UGV to acquire top-view stereo videos of eight drill rows of seedlings at 30 frames per second. Each stereo camera will be stabilized by a 3-axis gimbal and connect to an edge computer for control, monitoring, and data storage. Spatial mapping of pine seedlings in videos will be investigated using state-of-the-art deep convolutional neural networks (CNNs) for instance segmentation (e.g., Mask R-CNN, YOLACT++) and tracking (e.g., DeepSort, MaskTrack R-CNN). For late-stage seedlings that fully cover the entire bed, a push bar will be installed on the UGV to mechanically expose the seedling stems for imaging. In case of heavy occlusion of seedling stems, end-to-end seedling density estimation from videos will also be explored using long short-term memory networks. Root collar diameter (RCD) and shoot height will be measured based on the instance segmentation result in conjunction with stereo 3D reconstruction and point cloud analysis. Seedling health will be assessed using RGB-based vegetation indices and image color analysis. The above-mentioned algorithms will be integrated into an efficient video processing pipeline by sharing the same CNN feature extractor for multiple tasks (instance segmentation, tracking, stereo matching). The feasibility of real-time stereo video processing on an edge computer will be assessed by using neural network pruning and quantization techniques. To reduce annotation and accelerate dataset curation, active learning and interactive image annotation tools such as Curve-GCN will be employed. For evaluation of the robotic system, field experiments will be conducted at two commercial nurseries (one in Alabama and one in Georgia) through the Southern Forest Nursery Management Cooperative (SFNMC). Four visits per nursery are planned throughout each growing season during the project duration. For each visit, manual measurements of stand count, shoot height, and RCD (late stage) will be collected in 24 1 foot x 4 foot plots distributed across seedling beds. Two major southern pine species (loblolly and slash) and two seedlots/genotypes per species will be included. Areas of substandard seedlings will be identified through scouting or communication with the nursery managers. Additional imagery data will be collected in these areas using the robotic system along with visual ratings. These manually measured seedling traits will be used to evaluate the machine-derived counterparts using statistical analyses (Pearson correlation, linear regression, and Bland-Altman plot) and error metrics (RMSE, MAE, and MAPE). The performance of the autonomous navigation system will be quantified using mean time between human interventions. In Year 1, a tractor-based platform will be retrofitted with the proposed stereo imaging subsystem for data collection. Meanwhile, the UGV will be developed and tested at the nursery near the end of Year 1. In Year 2, the UGV and the stereo imaging subsystem will be integrated to collect stereo video data. The UGV will be controlled manually while the autonomous navigation is being developed and evaluated. In Year 3, the fully autonomous system will be further tested and used to collect data at nursery level. After data collection in each year, statistical analyses will be performed to evaluate the system performance and identify areas that need to be improved. Statistical analysis will be conducted in the statistical program R. A GIS decision support system will be developed in collaboration with the nursery managers to enable management, visualization and spatial analysis of the machine-derived seedling inventory and quality data. The GIS decision support software will be developed using QGIS and the PyQGIS library. Annual remote/in-person meetings with collaborating nursery managers will be organized to provide training on using the decision support system and collect their feedback on the utility and usability of the software for continual optimization for the project duration.

Progress 08/01/22 to 02/07/24

Outputs
Target Audience:The main target audience that our team reached was the US forest nursery industry (e.g., major private companies such as IFCO, ArborGen, and Weyerhaeuser) through our field tests of the robotic system and the annual advisory board meeting hosted by the Auburn University Southern Forest Nursery Cooperative. These companies are the direct stakeholders as they operate the majority of the forest nurseries in the Southeastern US. Researchers, faculty, and agricultural and biological engineering graduate students were also reached through invited talks and oral and poster presentations at professional conferences such as the ASABE Annual International Meeting. Changes/Problems:There is no major changes/problems in appraoch. But the PD moved from Auburn Universityto University of Delaware one month after year 1 of the project. The PD has acquired approval from NIFA for grant transfer to continue the project as planned. What opportunities for training and professional development has the project provided?Three graduate students were trained to acquire computer vision, robotics, and artificial intelligence knowledge and skills. They also acquired general knowledge about the US forest nursery industry through literature review, field tests, and interaction with nursery managers and scientists. Two students presented their research at departmental seminars, the Auburn Student Research Symposium, and the 2023 ASABE Annual International Meeting. Southern Forest Nursery Cooperative members were provided basic education on how artificial intelligence (AI) works and how AI was used to perform seedling inventory. How have the results been disseminated to communities of interest?The PI presented "AI-driven robotic forest nursery inventory" at the 2023 annual contact meeting of the Southern Forest Nursery Cooperative in Auburn, AL, on July 18. The PI was also invited to present to managers and scientists of Weyerhaeuser nurseries in the Southeast and West to facilitate further collaboration. The research progress was presented at the 2023 ASABE Annual International Meeting in Omaha, NE, on July 9-12 via one poster and two oral presentations. What do you plan to do during the next reporting period to accomplish the goals? Nothing Reported

Impacts
What was accomplished under these goals? Our team has collected stereo RGB video data of bare-root pine seedlings for spring and fall inventories at three commercial forest nurseries in Alabama, Georgia, and South Carolina. Our initial research on seedling counting using computer vision and deep learning techniques resulted in an accuracy of 92.5% for fall inventory compared to 89% of the current manual, sampling-based practice at the 2-foot linear bed resolution. Our study showed the feasibility of mapping spatial variability in stand count for bare-root forest nursery production at a ground speed of 1.5 MPH. For spring inventory, our research showed that the state-of-the-art deep learning-based object detection and tracking method could produce a counting accuracy between 85% and 91%, depending on model complexity. A ground-based robotic system was built, and a stereo vision-based navigation algorithm was successfully developed and evaluated in simulation as well as on-farm field tests with ground speeds up to 1.5 MPH for straight seedling beds. We have received positive feedback and strong interest from the nursery managers regarding the potential of reducing labor costs and improving harvesting precision.

Publications

• Type: Journal Articles Status: Published Year Published: 2023 Citation: Bidese-Puhl, R., Bao, Y., Payne, N., Stokes, T., Nadel, R. & Enebak, S. A. (2023). In-field pine seedling counting using end-to-end deep learning for inventory management. Journal of the ASABE, 66(2), 469-477. https://doi.org/10.13031/ja.15383
• Type: Conference Papers and Presentations Status: Other Year Published: 2023 Citation: Bao, Y., Bidese-Puhl, R., Shabani, S., Abrahams, A., McDonald, T., Tang, L. (2023). An AI-based ground robotic vision system for automated forest nursery inventory. 2023 ASABE Annual International Meeting, July 9-12, Omaha, Nebraska, USA. (Poster ID 2301487)
• Type: Conference Papers and Presentations Status: Other Year Published: 2023 Citation: Bidese-Puhl, R., Bao, Y., Payne, N., Stokes, T., Enebak, S. (2023). Real-time infield counting of early-stage pine seedlings using efficient object detection and multiple object tracking. 2023 ASABE Annual International Meeting, July 9-12, Omaha, Nebraska, USA.
• Type: Conference Papers and Presentations Status: Other Year Published: 2023 Citation: Shabani, S., Bidese-Puhl, R., Bao, Y., McDonald, T., Tang, L. (2023). Development of a ground-based robotic platform towards automated inventory for precision forest nursery management. 2023 ASABE Annual International Meeting, July 9-12, Omaha, Nebraska, USA.

Progress 08/01/22 to 07/31/23

Outputs
Target Audience:The main target audiencethat our teamreached was theUSforest nursery industry (e.g.,major private companies such as IFCO, ArborGen, and Weyerhaeuser)through our field tests of the robotic system andthe annual advisory board meeting hosted by the Auburn UniversitySouthern Forest Nursery Cooperative. These companies are the direct stakeholders as they operate the majority of the forest nurseries in the Southeastern US. Researchers, faculty, and agricultural and biological engineering graduate students were also reached through invited talks andoral and posterpresentations at professional conferences such as the ASABE Annual International Meeting. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided?Three graduate students were trained to acquire computer vision, robotics, and artificial intelligence knowledge and skills. They also acquired general knowledge about the US forest nursery industry through literature review, field tests, andinteraction with nursery managers and scientists. Two students presented their research at departmentalseminars, the Auburn Student Research Symposium, and the 2023 ASABE Annual International Meeting. Southern Forest Nursery Cooperative members were provided basic education on how artificial intelligence (AI) works and how AI was used to perform seedling inventory. How have the results been disseminated to communities of interest?The PI presented"AI-driven robotic forest nursery inventory" at the 2023 annual contact meeting of the Southern Forest Nursery Cooperative in Auburn, AL, on July 18. The PI was also invited to present to managers and scientists ofWeyerhaeuser nurseries in the Southeast and West to facilitate further collaboration.The research progress was presentedat the 2023 ASABE Annual International Meeting in Omaha, NE, on July 9-12 via one poster and two oral presentations. What do you plan to do during the next reporting period to accomplish the goals?We plan to perform large-scaleon-farm testsof the developed forest nursery inventory robot during the next reporting period. The robot-derived inventory data will be compared to the inventory data collected by the nurseries to assess if the automated inventory would reduce crop loss during lifting. Meanwhile, we will continue to improve the accuraciesof our seedling counting algorithms and investigate how an increase in ground speed affectscounting accuracy. We will also develop algorithms to assess seedling quality (stem diameter, shoot height, and needle coloration).An actively-illuminated stereo camera has been under development to improve image quality. The requirements forhigh instantaneous current draw and high frame rate (30FPS) have caused overheating. A new heat sink will be redesigned to overcome the challenge. The robot-collected inventory data will be used to develop the GIS decision support tool.

Impacts
What was accomplished under these goals? Our team has collected stereo RGB video data of bare-root pine seedlings for spring and fall inventories at three commercial forest nurseries in Alamaba, Georgia, and South Carolina.Our initial research onseedling counting using computer vision and deep learning techniquesresulted in an accuracy of 92.5% for fall inventory compared to 89% ofthe current manual, sampling-based practice at the 2-foot linear bed resolution.Our study showed the feasibility of mapping spatial variability in stand count for bare-root forest nursery production at a ground speed of 1.5 MPH. For spring inventory, our research showed that the state-of-the-art deep learning-basedobject detection and tracking method could produce a counting accuracy between 85% and 91%, depending on model complexity.A ground-based robotic system was built, and a stereo vision-basednavigation algorithm wassuccessfullydeveloped and evaluatedin simulation as well as on-farm field tests with ground speeds up to 1.5 MPH for straight seedling beds. We have received positive feedback and strong interest from thenursery managersregarding the potential of reducinglabor costs and improving precision of harvesting.

Publications

• Type: Journal Articles Status: Published Year Published: 2023 Citation: Bidese-Puhl, R., Bao, Y., Payne, N., Stokes, T., Nadel, R. & Enebak, S. A. (2023). In-field pine seedling counting using end-to-end deep learning for inventory management. Journal of the ASABE, 66(2), 469-477. https://doi.org/10.13031/ja.15383
• Type: Conference Papers and Presentations Status: Other Year Published: 2023 Citation: Bao, Y., Bidese-Puhl, R., Shabani, S., Abrahams, A., McDonald, T., Tang, L. (2023). An AI-based ground robotic vision system for automated forest nursery inventory. 2023 ASABE Annual International Meeting, July 9-12, Omaha, Nebraska, USA. (Poster ID 2301487)
• Type: Conference Papers and Presentations Status: Other Year Published: 2023 Citation: Bidese-Puhl, R., Bao, Y., Payne, N., Stokes, T., Enebak, S. (2023). Real-time infield counting of early-stage pine seedlings using efficient object detection and multiple object tracking. 2023 ASABE Annual International Meeting, July 9-12, Omaha, Nebraska, USA.
• Type: Conference Papers and Presentations Status: Other Year Published: 2023 Citation: Shabani, S., Bidese-Puhl, R., Bao, Y., McDonald, T., Tang, L. (2023). Development of a ground-based robotic platform towards automated inventory for precision forest nursery management. 2023 ASABE Annual International Meeting, July 9-12, Omaha, Nebraska, USA.