NRI: INT: COLLAB: TREE FRUIT HARVESTING WITH ARRAYS OF VISION-GUIDED LINEAR ROBOT ARMS

• Sponsoring Institution: National Institute of Food and Agriculture
• Project Status: COMPLETE
• Funding Source: AFRI COMPETITIVE GRANT
• Reporting Frequency: Annual
• Accession No.: 1020591
• Grant No.: 2020-67021-30759
• Cumulative Award Amt.: $663,460.00
• Proposal No.: 2019-04793
• Project Start Date: Apr 1, 2020
• Project End Date: Mar 31, 2025
• Grant Year: 2020
• Program Code: [A7301]- National Robotics Initiative

Recipient Organization
UNIVERSITY OF CALIFORNIA, DAVIS
410 MRAK HALL
DAVIS,CA 95616-8671

Performing Department
Bio & Ag Engineering

Non Technical Summary
Harvesting is one of the most labor-intensive operations in fresh-market fruit production, incurring high cost and dependence on a large seasonal semi-skilled workforce, which is becoming less available. Existing mechanical harvesting methods - such as trunk or canopy shaking - result in unacceptable fruit damage and cannot be used to harvest fruits at a given ripening stage or size selectively. Robotic harvesting is still at a pre-commercial stage, despite active research for at least 30 years. State-of-the-art robots can harvest fruits from trees with thin, almost two-dimensional canopies that have been meticulously pruned and thinned to offer very high fruit visibility and reachability. Still, robotic fruit picking efficiency and speed are inadequate for cost-effective operation. The primary goal of this project is to significantly improve the fruit picking efficiency and picking cycle of robotic harvesters and expand their operation to broader classes of orchards, with trellised and hedged trees with deeper canopies and less stringent pruning and thinning requirements. Our approach deviates significantly from the established paradigm in robotic fruit harvesting in two significant ways. First, instead of relying on a few, very fast arms with many degrees-of-freedom that are too expensive to deploy in large numbers, Robot-arm Arrays of many inexpensive linear arms will be used to reach fruits. Second, it is recognized that fruit visibility is key to fruit detection, and it is proposed that tree foliage be agitated via airflow in a targeted and controlled fashion, and that image sequences from cameras at multiple viewpoints be utilized, in conjunction with deep learning, to drastically increase fruit visibility and detection. A prototype robot that embodies the above approaches will be built and evaluated in high-density apple and pear orchards.Intellectual merit. Existing approaches for scheduling and planning coordinated motions for large numbers of robot arms that need to optimally reach a large set of points assume that all points are fixed and accurately known in advance. In robotic fruit harvesting, changes in fruit visibility and robot-canopy interactions cause a constant change in the set of fruits to be picked (in terms of the number of fruits, their positions, system confidence in their presence). Additionally, arms have limited time to reach fruits because the harvester moves. Finally, inter-robot geometric constraints must be respected. Therefore a new approach is needed for scheduling in real time the picking motions of multi-arm fruit harvesters. We plan to address this intellectual challenge by establishing correspondences between this problem and Stochastic Dynamic Vehicle Routing Problems and by exploiting recent advances in their solution that utilize parallel hardware (Graphics Processor Units - GPUs) to compute near-optimal robust solutions that incorporate uncertainty. Also, resolving fruit presence and position ambiguities in deep and vigorous tree canopies necessitates significantly improved fruit detection in individual images and integration of a large number of viewpoints. Our combined approach of foliage agitation and deep learning multi-view detection addresses the first challenge. Placing several cameras on the harvester creates an imaging array; processing and integrating the resulting images - in the presence of moving arms - constitutes an intellectual and technical challenge that will be addressed via probabilistic merging of multiple fruit projections from neighboring cameras and utilization of parallel graphics processors.Broader impact. Cost-efficient robotic fruit harvesters will increase the competitiveness and sustainability of the fruit industry and benefit growers and their families directly. Automation enables increased production of low-cost, high-quality fruits, leading to more year-round jobs for farm workers; higher-paid operator jobs and increased labor demand at the postharvest stage, and increased health and nutrition for consumers, especially for low-income families. Location data accompanying picked fruit can lead to yield maps for better management of water and nutrients leading to economic and environmental benefits, and to increased traceability for consumer safety. Furthermore, the project's educational agenda spans the graduate, undergraduate, and K-12 levels and utilizes project-based learning and fieldwork to cross-pollinate among disciplines. The researchers will leverage the themes of the project and the increasing awareness and concern for sustainable and healthy food production to engage K-12 students in STEM-promoting activities.

Animal Health Component

60%

Research Effort Categories

Basic

(N/A)

Applied

60%

Developmental

40%

Classification

Knowledge Area (KA)	Subject of Investigation (SOI)	Field of Science (FOS)	Percent
402	1110	2020	20%
402	1115	2020	10%
402	7299	2020	70%

Knowledge Area
402 - Engineering Systems and Equipment;

Subject Of Investigation
1110 - Apple; 1115 - Pear; 7299 - Research equipment and methods, general/other;

Field Of Science
2020 - Engineering;

Keywords

robotics

harvesting

computer vision

robot scheduling

harvest efficiency

Goals / Objectives
Research Goal 1: Development of Linear Robot Arm ArraysThis goal includes the development of a simulator for the mult-arm harvester using digitized trees and fruits and of algorithms to compute the optimal coordinated motion trajectories for the robot arms. An economic model for robotic harvesting will also be developed.Objective 1.1: Modeling and Simulation of LRA HarvesterThis technical objective aims at developing: a model for LRAs with several arms in each cell; an integrated model for an LRA on a carrier platform, and a harvesting model of the system operating in a digitized orchard. The model will capture the essential kinematics and dynamics of the system.Objective 1.2: Fruit-Picking Order Assignments for LRAsThis objective aims at developing algorithms to compute the motion trajectories of the multiple arms, which are dynamic; incorporate uncertainty about fruit location and presence; minimize picking cycle times, and execute in real-time.Objective 1.3: Economic analysisThe goal of this objective is to develop an economic model of robotic harvesting that will be used to guidethe design and functional parameters of the machine but will also be available for other researchers, startup companies and investors to use when exploring commercial robotic harvest technologies.Objective 1.4: Engineering Design and Development of LRAIn addition to significant design issues, this technical objective covers the detailed CAD design of the LRA system (telescopic arm, mobile arm base module, air nozzles - with CMU, frame); fabrication of needed components; integration of machine, actuators, pneumatic, electrical and electronic systems (hardware); development and tuning of the low-level motion control software; integration of perception system, fruit- picking assignment (high-level control) and motion control system. As the last step, the LRA will be installed on the orchard platform. Given our budgetary constraints, the goal is to build 15 arms and compare their performance in orchards against the predicted PCT vs. number-of-arms curve.Research Goal 2: Increase Fruit Visibility and DetectionA combination of novel approaches that include active foliage agitation via controlled air streams, deep learning-based partially occluded fruits detection, and multi-view imaging system are formulated under this goal to significantly increase fruit visibility in canopies of trellised and hedged trees.Objective 2.1: Fruit detection systemRobust detection and localization of fruits is an essential component for an automated fruit harvester. We plan to use the current state- of-the-art, Mask R-CNN as a multi-class detector to instantly detect and segment not only clearly visible and partially occluded fruits but also the foliage in the scene.Objective 2.2: Improved Fruit Visibility via Multi-View GeometryThis objective aims at detecting occluded fruits using a multi-view approach that accounts for occlusion and multiple fruit count. Additionally, dynamic changes in the set of fruits to be picked will be tracked.Objective 2.3: Increased Fruit Visibility via Foliage AgitationThis objective aims at designing, building and testing a novel tree foliage agitation system that increases fruit visibility in a sequence of images or a video stream. More specifically, by causing leaves to move it is expected that different parts of fruit surface will become visible over a series of successive image frames.Research Goal 3: System Integration and EvaluationThis goal covers the integration of hardware and software used for the perception and actuation systems, as well as the evaluation of the functional system in laboratory and real-world conditions in commercial orchards.Objective 3.1: Integration of actuation and perception systemsThe information flows between the perception and actuation subsystems will be defined and implemented to achieve their integration. The computational pipeline for feeding target 3D points (fruit centers) - with corresponding detection probabilities - from the cameras to the LRA's picking order assignment algorithm will be established.Objective 3.2: Evaluation of LRA actuation systemEvaluations of the LRA actuation system (without perception) will be performed in laboratory conditions.Objective 3.3: Evaluation of LRA perception and actuation in commercial orchardsThe LRA with integrated actuation and perception systems will be tested in an orchard with V-trellised Fuji apple trees, and V-trellised and hedged high-density Bartlett pear trees.

Project Methods
Research Goal 1: Development of Linear Robot Arm ArraysObjective 1.1: Modeling and Simulation of LRA HarvesterModels of pear trees and cling peach trees are available, but V-trellised apple trees and their fruits will be digitized. The perception and detachment steps of the pick cycle will be assumed constant; however, value ranges for them will be used. Different ways to harvest, where the LRA picks simultaneously from opposite sides of different trees, from both sides of each tree, and from only one tree side will be implemented to compare performance. All simulation code will be developed in Python and C++ and will utilize the Robot Operating System (ROS) and the Open Source Bullet Physics Engine. Results will be processed and visualized with Matlab.Objective 1.2: Fruit-Picking Order Assignments for LRAsThe Multiple Scenario Approach (MSA) for SDVRP will be used, which incorporates uncertainty about future "customers" (fruits) to improve routing. MSA uses the pool of candidate routes and a 'consensus function' to compute a 'distinguished' tour, which serves the existing customers only. The insight is that this distinguished tour can better accommodate any future customers that are close to the predicted ones. In the context of fruit harvesting, existing customers would correspond to fruits with detection scores over a threshold (e.g., 0.9). Sampling 'future customers' would correspond to including in scenarios fruits with probability equal to their detection scores. Scenarios will be generated in parallel on cores of Nvidia Graphics Processing Units running Cuda.Objective 1.3: Economic analysisOur study will develop an improved, updated economic model for robotic harvesting. It will calculate the up-front and ongoing costs of adopting a robotic fruit harvester. We will simulate the performance of the harvester under different potential yields and canopy maintenance practices. We will compare these findings to the costs and returns associated with a hand-harvested crop. We will repeat the economic analysis under differing labor supply conditions. We will determine under what farm labor supply conditions robotic harvesters have higher economic returns than hand-harvest.Objective 1.4: Engineering Design and Development of LRAIn addition to significant design issues, this technical objective covers the detailed CAD design of the LRA system (telescopic arm, mobile arm base module, air nozzles - with CMU, frame); fabrication of needed components; integration of machine, actuators, pneumatic, electrical and electronic systems (hardware); development and tuning of the low-level motion control software; integration of perception system, fruit- picking assignment (high-level control) and motion control system. As the last step, the LRA will be installed on the orchard platform.Research Goal 2: Increase Fruit Visibility and DetectionObjective 2.1: Fruit detection systemWe will use the current state- of-the-art, Mask R-CNN as a multi-class detector to instantly detect and segment not only clearly visible and partially occluded fruits but also the foliage in the scene. This algorithm extends Faster-RCNN by adding a layer in the network architecture that predicts the object mask in parallel to bounding box recognition. Under this objective, we will not only generate a hand-labeled data-set and utilize it for our task, but also publicly share it to motivate future research activities.Objective 2.2: Improved Fruit Visibility via Multi-View GeometryCost-effective active-lighting camera systems will be used. In these cameras, bright pulses of light from high- power Xenon flashes are synchronized with the camera's trigger mechanism, thus reducing the adverse effects of natural lighting significantly. We will mount a number of stereo pairs on the frame of the LRA at fixed locations, which combined cover the entire canopy with overlapping regions. The global flash system will also be mounted at selected fixed positions of the LRA frame to generate consistently controlled lighting.Objective 2.3: Increased Fruit Visibility via Foliage AgitationWe hypothesize that multi-view imaging combined with vision-guided, properly-selected agitation patterns (e.g., scan-line, cyclic, diagonal) can improve fruit visibility. Pre-programmed airflow patterns may have negligible or even adverse effects on fruit detection. For example, blowing air at fruits that are already well exposed may cause unwanted occlusion. For this reason, vision-guided agitation will be explored too. Canopy regions where the numbers of detected fruits are "too low" or fruits have too low detection scores will be classified as severely occluded areas. Additionally, the vegetative region segmented by Mask-RCNN will be added as an additional layer of data in the classifier. Airflow from nozzle groups can be targeted at such canopy regions.Research Goal 3: System Integration and EvaluationObjective 3.1: Integration of actuation and perception systemsThe information flows between the perception and actuation subsystems will be defined and implemented to achieve their integration. First, each camera in a triggered stereo pair will detect fruits using using Mask-RCNN and the 3D points corresponding to fruit centers will be computed using using stereo correspondence. Next, the point clouds from the multiple views that are triggered will be registered in pairs using Point to Plane Iterative Closest Point (ICP) algorithm. ICP algorithms can be computationally expensive. Finally, the coordinates of the resulting point cloud will be transformed to the coordinate frame. Once the coordinates are transformed, the system will assign picking order for LRAs as described under objective 1.2.Objective 3.2: Evaluation of LRA actuation systemEvaluations of the LRA actuation system (without perception) will be performed in the lab, at UC Davis. After single-arm control has been tuned and optimized, experiments of fully replicated, randomized complete block statistical design will take place. Two types sets of points (blocking factor) will be generated: (B1) random points within the LRA's workspace that match fruit density histograms from digitized trees, and (B2) random points with a uniform distribution of similar mean spatial density with B1. In one set of experiments, static harvesting will be evaluated, and the response variable of interest (evaluation metric) will be PCT. The goal is to assess the PCT of the overall system. In the second set of experiments the points in Fi will be moving linearly to simulate dynamic harvesting with the LRA platform moving at speed, Vi. Response variables are PCT and the reachability component of FPE. The moving speed and the number of arms of the LRA platform will be the two primary factors of the experiments. Nonlinear regression will be used to identify the dependence of the response variables on these factors.Objective 3.3: Evaluation of LRA perception and actuation in commercial orchardsThe LRA with integrated actuation and perception systems will be tested in orchards with V-trellised apple trees and hedged pear trees. One set of experiments will assess the precision of deep learning-based fruit detection by comparing the algorithm's prediction to manual count at image-to-image and image-to-absolute fruit count levels. The performance will be analyzed using different performance characterization metrics including F1/F2 scores and MSE. Another set of experiments will assess the effectiveness of the agitation systems using Analysis of variance (ANOVA) on the visibilities. The third set of experiments will assess the effectiveness of the multi-view perception system integrated with the most effective leaf agitation system and the fruit reachability of the LRA. The fourth set of experiments will assess the LRA's PCT, with platform speed and number of arms acting again as primary factors.

Progress 04/01/23 to 03/31/24

Outputs
Target Audience:UC Davis Two graduate students and one undergraduate student were engaged in research and development and were mentored during the project's fourth year. The PI gave presentations on robotic harvesting technologies at national and international conferences, workshops, growers' conferences, and academic lectures. The audiences included growers, entrepreneurs, researchers, faculties, students, extension agents, visiting scholars, and the general public. CMU Co-PI Kantor and scientist staff Silwal and Yandun have included findings from this research at several national (local growers conference, invited speakers), international conferences, and academic lectures. The audiences included growers, industry, entrepreneurs, researchers, faculties, students, visiting scholars, guest lecturers, and the public. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided?UC Davis Two Ph.D. students and one undergraduate student were mentored during the fourth year of this project at UC Davis. The graduate students worked closely with PI Vougioukas and gained experience in modeling and solving task assignments and scheduling problems using mathematical programming and numerical optimization tools, such as the Gurobi open-source optimized. The undergraduate student developed and documented real-time motion control software to control the motions of the motors of the multi-arm prototype. He also incorporated limit switches to safely restrict the motion of each arm. All students gained more programming experience in python, C++, and ROS 2. CMU Under objective 2, a graduate student completed his Master's degree, building the perception system to detect and track apples in 3D space. Two other Masters students were also involved in improving and testing the system through internships. A high school graduate designed and fabricated the agitation mechanism as a part of her summer internship. Summer internships in robotics for two undergraduates were also made available as leadership opportunities for staff scientists at CMU. How have the results been disseminated to communities of interest?UC Davis PI-Vougioukas gave the following presentations as an invited speaker. April 27, 2023. HortForum Webinar: "The present and future of the use of autonomous equipment and robotic harvesters in field-based fruit production". ISHS (International Society of Horticultural Science). June 2, 2023. "TIG-IV: From Farm To Fork" workshop at the 2023 IEEE International Conference on Robotics and Automation (ICRA2023). "Fairness metrics for scheduling harvest-assist robots" May 11, 2023. UC Davis Western Center for Agricultural Health and Safety workshop "Emerging Technology in Agriculture: Keeping Health & Safety at the Forefront". "Robotics Harvest-aids and Worker Well-being". February 14, 2024. Wallace Heuser Presidential Lecture. 2024 IFTA Annual Conference (International Fruit Tree Association). Yakima, Washington. "Labor in 2023 and the Turn Towards Automation." February 6, 2024. 2024 California Plant and Soil Conference. Fresno, CA. "Advances in automation technologies and their effect on farm workers". What do you plan to do during the next reporting period to accomplish the goals?UC Davis We are close to finishing a small-scale physical prototype (summer 2024), which we plan to deploy and test in the lab and outside in the fall of 2024. We will also continue to develop advanced heuristics for dynamic fruit assignment and scheduling in real-time. FInally, we will perform data analysis and reporting. CMU Under each objective (2.1 - 2.3), we have listed the limitations of our existing baseline algorithms. We plan to improve the performance of these baseline algorithms by incorporating the latest developments in 3D shape completion and Reinforcement learning and report the comparison.

Impacts
What was accomplished under these goals? Research Goal 1: Development of Linear Robot Arm Arrays Objective 1.1: Modeling and Simulation of LRA Harvester This objective was completed. Objective 1.2: Fruit-Picking Order Assignments for LRAs In the previous year, the robot-to-fruit assignment and scheduling problem was modeled as a time-dependent team orienteering problem with time windows (TDTOPTW) mixed integer programming (MIP). This problem is NP-hard, so it could not be solved for large numbers of fruits. A First-Come-First-Serve dynamic policy was programmed to run the assignment in real time. Furthermore, the problem of jointly optimizing the assignment and schedule and the motions of the arms inside the same cell was modeled as a large MIP. The correctness of the mode was established by running it for two arms and ten fruits. However, larger problems were impossible to solve due to the large number of variables and model complexity. Hence, a heuristic algorithm was developed to harvest single orchard segments, which can execute in under ten seconds for up to thirty arms and thousands of fruits. Objective 1.3: Economic analysis In the previous year, Co-PI Charlton created a cost-benefit model for a representative apple grower's decision to adopt robotic harvest technology. Using the cost-benefit model as a framework, Charlton and co-authors performed data analysis to solve for the break-even cost of a robot such that profits from manual harvesting are equal to profits from robotic harvesting. This year, the model was expanded to include the cost of manual harvesting for the fruits that a robot cannot pick due to limited picking efficiency. The updated model and analyses were submitted as a peer-reviewed paper. Objective 1.4: Engineering Design and Development of LRA A small-scale prototype of the LRA was built in the lab. A novel multi-lead-screw mechanism was designed and fabricated to drive the vertical motions of the arms inside a cell frame. Lead screws and rack-pinion mechanisms were developed to drive the horizontal and extension degrees of freedom. The frame of the system was designed and built using aluminum extrusions. Motors were attached and electrical wiring was routed to the motors from the batteries. Electrical and mechanical safety are being incorporated into the design. Position motion control was developed, and visual serving is under development. The prototype is expected to be functional in the summer of 2024. Research Goal 2: Increase Fruit Visibility and Detection Objective 2.1: Fruit detection system In the previous reporting, using the latest YOLO network, we reported our trained models to achieve high fruit detection accuracy of 0.9 mean average precision (mAP) for Intersection of Union (IoU) of 0.5 and an average mAP of 0.75 for IoU from 0.5 to 0.95 using images from our active lighting camera system. These results are adequate for our testing and currently we are considering this objective to be completed. Objective 2.2: Improved Fruit Visibility via Multi-View Geometry We have successfully designed and tested two baseline algorithms to track apples in images. At broad, these algorithms are white box and black box in nature and have different contrasting features for comparison. The first algorithm is a 3D Kalman filter using Mahalanobis distance as a measure to keep track of the same apples in consecutive images. Using a linear and constant velocity model, the tracker initiates and terminates each track of an apple as a single Kalman filter thread for tracking the apples in images. The tracker was implemented in 3D using stereo cameras on images captured in commercial apple orchards. The second algorithm uses deep SORT (Simple Online Real Time Tracking), a combination of deep learning and Kalman filter-based approach to tracking objects in images. Like the naive Kalman filter, the position estimates of the fruits assume a constant velocity model. Both algorithms show similar accuracy in testing while lacking robustness in highly occluded apples from foliage, branches, and nearby fruits. Objective 2.3: Increased Fruit Visibility via Foliage Agitation We have completed the design and fabrication of a canopy agitation system using a modified leaf blower, digital servo motors, and control systems to actuate them. More specifically, the control system allows the canopy agitation system to target any location in the canopy as well as the volume of air. We have incorporated the tracking algorithm with the agitation system and completed systems integration. Our current baseline algorithm actuates the agitation system using density and fruit detection scores from obj. 2.1. To improve performance, we are also investigating a deep Reinforcement Learning (RL) algorithm to intelligently agitate the canopy that learns from trials and errors. A simulation environment is currently being developed to train the RL model. Research Goal 3: System Integration and Evaluation This goal will be addressed in the final year, starting late summer-fall of 2024.

Publications

• Type: Journal Articles Status: Published Year Published: 2023 Citation: Arikapudi R., Vougioukas, S.G. (2023). Robotic Tree-fruit Harvesting with Arrays of Cartesian Arms: A Study of Fruit Pick Cycle Times. Computers and Electronics in Agriculture (211), 108023. https://doi.org/10.1016/j.compag.2023.108023
• Type: Journal Articles Status: Accepted Year Published: 2024 Citation: Villacr�s, Vougioukas, S. G. (2024). Assessing a Multi-Camera System for Enhanced Fruit Visibility. Computers and Electronics in Agriculture - Special Issue on " AI-driven Agriculture"
• Type: Journal Articles Status: Submitted Year Published: 2024 Citation: Charlton, D., Devadoss, S., Gallardo, R.K., Luckstead, J., Vougioukas, S. (2024) Robotic Apple Harvesters: A Cost-Benefit Analysis. Applied Economic Perspectives and Policy

Progress 04/01/22 to 03/31/23

Outputs
Target Audience:UC Davis Four graduate students and one undergraduate student were engaged in research and development and were mentored during the project's third year. The PIs and scientist staff Silwal gave presentations on robotic harvesting technologies at national and international conferences, workshops, growers' conferences, and academic lectures. The audiences included growers, entrepreneurs, researchers, faculties, students, extension agents, visiting scholars, and the general public. CMU Co-PI Kantor and including scientist staff Silwal and Yandun have included findings from this research at several national (local growers conference, invited speakers), international conferences, and academic lectures. The audiences included growers, industry, entrepreneurs, researchers, faculties, students, visiting scholars, guest lecturers, and the public. MSU Findings from economic analysis will be disseminated through peer-reviewed publications and blogposts. The target audiences include engineers, growers, and policymakers. Findings can help engineers determine the tradeoffs in harvesting profitability between improvements in various robot parameters. Growers can use the analysis to better assess progress in robotic efficiency and determine whether a robot might be profitable for adoption given various speeds, share of apples picked from the orchard, and bruising/cull rates. They can also see how the profitability of robot adoption varies as farm worker wages rise. Finally, discussion of economic findings highlights the need for ongoing research to investigate the potential consequences of robotic adoption on employment, wages, and required skills up and down the agricultural supply chain. This and related research has important policy implications for farm labor employment, immigration, international trade, and agricultural research and development. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided?UC Davis Two Ph.D. students and one undergraduate student were mentored during the third year of this project at UC Davis. The graduate students worked closely with PI Vougioukas and gained experience in modeling and solving task assignments and scheduling problems using mathematical programming and numerical optimization tools, such as the Gurobi open-source optimized. One undergraduate student presented her work at the ASABE meeting. The undergraduate student worked during the summer and developed real-time motion control software to control the motions of the motors of the multi-arm prototype. All students gained more programming experience in python, C++, and ROS and had some exposure to ROS 2. The undergraduate student also got hands-on experience with building the electronics of the robot's motion control system. CMU Summer internship in robotics for two undergraduates, professional training for one graduate student, and leadership opportunities to staff scientists at CMU. How have the results been disseminated to communities of interest?UC Davis PI-Vougioukas gave the following presentations as an invited speaker. May 20, 2022. UC Merced Multicampus Research Program Initiative for Labor & Automation in California Agriculture: Equity, Productivity, & Sustainability. "Worker and Robot Collaboration for Fruit Harvesting" May 24, 2022. UC Davis Emerging Technologies Postharvest Technologies Workshop. "Theory and Research in Mechanized & Robotic Harvesting" September 16, 2022. Pomology Cross-Commodity Discussion Meeting - Labor. Online. "Theory and Research in Mechanized & Robotic Harvesting" October 18, 2022. FIRA USA, Fresno CA. "Harvest-Assist Collaborative Robots" November 14, 2022. Sustainable Ag Expo & Winegrowing Summit, San Luis Obispo, CA. "Agricultural Robots: Technology, Applications, and Adoption" March 20, 2023. UC Davis. Farm Labor in the 2020s Demand, Supply, and Markets. "Robotic and Robot-Assisted Fruit Harvesting" What do you plan to do during the next reporting period to accomplish the goals? Nothing Reported

Impacts
What was accomplished under these goals? Research Goal 1: Development of Linear Robot Arm Arrays Objective 1.1: Modeling and Simulation of LRA Harvester The distributed simulator was developed and tested in the project's first year, as planned in the proposed timeline. This year, some effort was invested in debugging to increase the robustness of the model and simulator. Objective 1.2: Fruit-Picking Order Assignments for LRAs An existing time-dependent team orienteering problem with time windows (TDTOPTW) mixed integer programming (MIP) was extended and modified to provide better results than previously achieved. Two performance metrics are being used in these tests, Fruit Picking Efficiency (FPE, the percent of marketable fruits harvested) and Fruit Picking Throughput (FPT, the average number of fruits picked per unit of time). The original TDTOPTW formulation was only optimized to obtain the maximum FPE, but harvesting time (FPT) is also very important to growers. Three different strategies were pursued to include optimization for FPT: (a) the original TDTOPTW formulation was run in a loop which tested changes in resulting FPE and FPT as vehicle speeds increased. The chosen schedule was the one with the highest FPT value, given a minimum FPE value was met. Because this is a greedy algorithm, the results were reasonably good but very slow. For strategies (b) and (c), the MIP formulation was changed to include vehicle speed, with changes to constraints and the objective function. Strategy (b) had an objective function to maximize FPE*FPT with soft minimum FPE and FPT constraints, while strategy (c) had the objective to minimize the makespan, or the schedule's length of time from start to finish, with soft constraints for a minimum FPE. Lastly, a first-come-first-serve (FCFS) naive schedule was used within the greedy velocity loop to find an initial lower-bound vehicle speed that still produces a minimum FPE. This lower bound was then used by the MIP formulations when solving for the schedule. This reduced the problem space, increasing solving time and improving the results. All MIP formulations had problems finding solutions when solving for the length of an entire orchard row. A single side can have thousands of apples. Thus, the orchard row was broken into smaller, overlapping sections that would be easier to solve and allow for more "dynamic" scheduling. Not only could the near-optimal vehicle speed be found for each section, allowing the system to speed up or slow down according to incoming fruits, but the arms' vertical workspace within a column could also be optimized to provide basic load balancing. This was done by allowing the arms to share the whole column as a workspace but limiting their vertical movement through software controls to avoid the need for collision avoidance. The vertical columns were divided into software-imposed horizontal rows that contain the same number of fruits. Furthermore, by overlapping the sections, changes to fruit locations, such as branches moving as apples are removed, could be acted upon. Lastly, a strategy inspired by model predictive control (MPC) was used to strengthen the minimum FPE and FPT requirements. This was needed because scheduling for small, overlapping sections resulted in not being able to predict the minimum FPE needed to harvest 95% of total fruits in the entire orchard row. First, a schedule was obtained for a section of the orchard row in front of, and equal in length to, the robot's workspace, or the volume of space the arms are able to harvest from that point in time, plus a horizon, or the section immediately in front of the workspace. However, the robot only follows through with the initial parts of the schedule (e.g., the schedule is solved for 2.4 m, but only the first 0.6 m of the schedule is acted upon) before creating a new schedule that takes into account any changes. Testing is being performed to see how this addition affects the different MIP formulations and what parameters can be changed to further maximize the FPE and FPT combination, as well as decrease solving speeds. Objective 1.3: Economic analysis Co-PI Charlton created a cost-benefit model for a representative apple grower's decision to adopt robotic harvest technology. Interviews with apple growers in Washington State helped inform the model. Using the cost-benefit model as a framework, Charlton and co-authors performed data analysis to solve for the break-even cost of a robot such that profits from manual harvesting are equal to profits from robotic harvesting. Parameters for the data analysis came from the "2019 Cost Estimates of Establishing, Producing and Packing Gala Apples in Washington".[1] Co-PI Vougioukas additionally provided ranges for robotic parameters, including picking speed, the share of apples in the orchard that the robot picks, and bruising rates. The Co-PIs conducted 5 break-even analyses using this model: First, they examined the maximum robot cost such that profits are equal for robotic and traditional manual harvesting. Second, they analyzed how the break-even cost of the robot changes for different picking speeds, harvest-induced damage rates, and percentages harvested. Third, for a given upfront cost of a robot, they computed the tradeoff between the wage rate and changes in the three robotic parameters. Fourth, because revenues, costs, harvest-induced damage rates, etc., differ across apple varieties, they computed the break-even cost of the robot for Honeycrisp apples, which have a higher farmgate price than Gala and bruise more easily. [1] Gallardo, R.K., and S.P. Galinato. 2021. "2019 Cost Estimates of Establishing, Producing and Packing Gala Apples in Washington." Washington State University Extension Bulletin TB18E. Objective 1.4: Engineering Design and Development of LRA Research Goal 2: Increase Fruit Visibility and Detection Objective 2.1: Fruit detection system In the previous reporting, using the latest YOLO network, we reported our trained models to achieve high fruit detection accuracy of 0.9 mean average precision (mAP) for Intersection of Union (IoU) of 0.5 and an average mAP of 0.75 for IoU from 0.5 to 0.95 using images from our active lighting camera system. These results are adequate for our testing, and currently we are considering this objective to be completed Objective 2.2: Improved Fruit Visibility via Multi-View Geometry We have successfully designed and tested two baseline algorithms to track apples in images. The first algorithm includes a 3D Kalman filter using Mahalanobis distance as a measure to keep track of the same apples in consecutive images. The second algorithm uses deep SORT (Simple Online Real Time Tracking), a deep learning-based approach to tracking objects in images. Both algorithms show similar accuracy in testing while lacking robustness in highly occluded apples from foliage, branches, and nearby fruits. Our current efforts are to use 3D shape completion on partially visible fruits to improve apple tracking. Objective 2.3: Increased Fruit Visibility via Foliage Agitation We have completed the design and fabrication of a canopy agitation system using a modified leaf blower and digital servo motors to actuate them. More specifically, the control system allows the canopy agitation system to target any location in the canopy as well as the volume of air. Currently, we are investigating a baseline algorithm to actuate the system using density and fruit detection scores from obj. 2.1. We are also investigating a deep Reinforcement Learning (RL) algorithm to intelligently agitate the canopy that learns from trials and errors. A simulation environment is currently being developed to train the RL model. Research Goal 3: System Integration and Evaluation This goal will be addressed in the coming year.

Publications

• Type: Journal Articles Status: Submitted Year Published: 2023 Citation: Arikapudi R., Vougioukas, S.G. (2023). Robotic Tree-fruit Harvesting with Arrays of Cartesian Arms: A Study of Fruit Pick Cycle Times. Computers and Electronics in Agriculture
• Type: Journal Articles Status: Submitted Year Published: 2023 Citation: Charlton, D., Devadoss, S., Gallardo, R.K., Luckstead, J., Vougioukas, S. (2023) Economic Viability of Robotic Fruit Harvesters to Reduce Large Seasonal Labor Demands. Journal of Agricultural and Resource Economics
• Type: Journal Articles Status: Published Year Published: 2023 Citation: Peng, C., Fei, Z., Vougioukas, SG. (2023). GNSS-Free End-of-Row Detection and Headland Maneuvering for Orchard Navigation Using a Depth Camera. Machines 11(1), 84.
• Type: Journal Articles Status: Published Year Published: 2022 Citation: Fei, Z., Vougioukas, S.G. (2022). Row-sensing Templates: A Generic 3D Sensor-based Approach to Robot Localization with Respect to Orchard Row Centerlines. Journal of Field Robotics, 39(6):712-738
• Type: Conference Papers and Presentations Status: Published Year Published: 2022 Citation: Pueyo Svoboda, N., Vougioukas, S.G. (2022). Testing a combinatorial?based melon harvesting scheduling algorithm on fruit orchard harvesting and the effects of varying the robots geometric configuration. ASABE Annual International Meeting. Houston, Texas. Presentation # 2200427.

Progress 04/01/21 to 03/31/22

Outputs
Target Audience:Two post-doctoral researchers, two graduate students, and one undergraduate student were engaged in research and development and were mentored in the second year of the project. Also, the PI and Co-PIs and scientist staff Silwal and Yandun gave several presentations related to robotic harvesting technologies at several national, and international conferences, workshops, growers' conferences, and academic lectures. The audiences included growers and allied industry, entrepreneurs, researchers, faculties, students, extension agents, visiting scholars, guest lecturers, and the general public. Due to the pandemic, field visits to farms to calibrate a model for cost-benefit analysis were delayed. However, co-PI Charlton managed to visit apple orchards in California and Washington State, and she has created a preliminary cost-benefit table to assess the break-even cost of a robot at which robotic harvesting would be equivalent to hiring workers to hand-pick apples. The table can be adjusted for a range of aspects, including bruising and culling, and yield. Charlton also published a paper in the Applied Economics Perspectives and Policy journal on the relationship between monthly variation in historical fruit, vegetable, and horticultural employment and new cases of COVID-19 within U.S. counties in 2020. The findings help illuminate the possible relationship between COVID-19 and fruit harvest employment, which could be of interest in anticipating changes in demand for robotic harvesters. Changes/Problems:Due to the issue caused in the global supply chain by the COVID-19 pandemic, the acquisition of actuation and imaging sensors and other necessary supplies have been severely affected. The significant increase in the lead time to acquire such necessary hardware has delayed the design and fabrication of the LRA and the camera system and delayed field visits. Also, field visits to farms to calibrate a model for cost-benefit analysis were delayed. The PIs anticipate that a no-cost extension of the project may prove to be necessary. What opportunities for training and professional development has the project provided?One Ph.D. student and two post-doctoral researchers were mentored during the second year of this project, at UC Davis. They worked closely with PI Vougioukas, and gained experience in modeling, simulation, task assignment and scheduling algorithms, and ROS 2, which is used to program the simulator and the real-time platform. The Ph.D. student also got hands-on experience with building the electronic hardware of the robot's motion control system. One graduate student and one undergraduate student were mentored by PI Kantor in the area of high-speed imaging, machine learning, modeling, and simulation. How have the results been disseminated to communities of interest?PI-Vougioukas gave the following presentations, as an invited speaker. "The Smart Shift to Advanced Agriculture in the U.S. and Sweden". Webinar organized by the Swedish Trade & Invest Council. Sept. 9, 2021. "Collaborative-Robotic Harvest-aid platforms can increase harvesting speed". Webinar during the 2021 World Ag Expo. Aug. 19, 2021. "Human-robot Collaboration for Fruit Harvesting". CITRIS Research Webinar. Sept. 15, 2021. "Agricultural Robotics and labor". Digital Transformation Workshop, UC Davis Graduate School of Management. Nov. 10, 2021. "Robotic harvesting and precision yield mapping". UC ANR Precision Ag Workgroup Meeting. Mar. 9, 2022. "Agricultural robotics, robotic-aided harvesting and automation". Webinar organized by the Phenomics and Plant Robotics Center (P2RC) at the University of Georgia. Mar. 17, 2022. PI Kantor and scientist staff Silwal and Yandun gave several presentations related to robotic harvesting technologies at several national, and international conferences, workshops, growers' conferences, and academic lectures. What do you plan to do during the next reporting period to accomplish the goals? Nothing Reported

Impacts
What was accomplished under these goals? Research Goal 1: Development of Linear Robot Arm Arrays Objective 1.1: Modeling and Simulation of LRA Harvester The distributed simulator was developed and tested in the first year of the project, as planned in the proposed timeline. This year, some effort was invested in testing and debugging to increase the robustness of the model and simulator. Objective 1.2: Fruit-Picking Order Assignments for LRAs Arm-to-fruit assignment strategies were tested to produce better results than the previously used First-Come-First-Serve, greedy strategy. Two performance metrics are being used in these tests, Fruit Picking Efficiency (FPE, the percent of marketable fruits harvested) and Fruit Picking Throughput (FPT, the average number of fruits picked per unit of time). The first algorithm attempted was an existing combinatorial heuristic, originally developed for a melon harvesting robot with multiple cartesian arms set in a row. Although very fast and optimal when maximizing FPE (and only FPE), this algorithm proved less useful for our system. It assumes that it takes the same, constant picking time for every fruit, that the fruit distribution is two-dimensional, and that arms cannot revisit locations even if fruits may have been missed. This works well for melons, which grow on the ground (2D) and in low densities (their examples went up to 4 fruits/m^2). However, in orchard fruits like apples, which can grow inside or outside of the canopy and have much higher densities such as 46 fruits/m^3 (from digitized fruit data), would be badly served by this combinatorial heuristic. As our second assignment strategy, we developed a Mixed Integer Programming model expressed as a time-dependent team orienteering problem with time windows (TDTOPTW). The combinatorial heuristic is based on this particular formulation, so it also has a constant picking time and does not revisit locations; however, it is easily expandable (e.g. the addition of columns of arms through a single mathematically-described constraint). The model proved useful for testing design parameters such as the effects of the vehicle's velocity, dividing the rows by different parameters such as the number of fruits or the overall height of the robot, or different fruit distributions. An important accomplishment was an 'extension' of the MIP formulation to minimize the overall picking time, also known as the makespan. This new formulation computes the optimal harvester travel speed and the optimal overall arms-to-fruits schedule that maximizes the FPT for a given minimum FPE (say 95% picked minimum). This problem becomes intractable for even low densities because it is one of the harder NP-hard and NP-complete problems. Hence, heuristics to solve it are being developed. Objective 1.3: Economic analysis Co-PI Charlton visited apple orchards in California and Washington State, and she has created a preliminary cost-benefit table to assess the break-even cost of a robot at which robotic harvesting would be equivalent to hiring workers to hand-pick apples. The table can be adjusted for a range of aspects, including bruising and culling, and yield. Charlton also published a paper in the Applied Economics Perspectives and Policy journal on the relationship between monthly variation in historical fruit, vegetable, and horticultural employment and new cases of COVID-19 within U.S. counties in 2020. The findings help illuminate the possible relationship between COVID-19 and fruit harvest employment, which could be of interest in anticipating changes in demand for robotic harvesters. Objective 1.4: Engineering Design and Development of LRA Numerous design options were investigated and modeled in CAD with an emphasis on low cost, robustness, and ease of fabrication and repair. The selected design for the cell consists of a rigid aluminum frame that can accommodate up to three robotic arms. Each arm is mounted on a carriage and moves in all three linear axes. Vertical motion of the carriage is accomplished through a rack and pinion system. The horizontal side-to-side motion of the arm is via a fast-travel Acme screw drive. Extension and retraction of the arm use a lightweight rack and pinion-type system. A prototype cell has been built with a single carriage though it can accommodate up to three carriages. A low-power electromagnetic brake prevents the carriage from moving in the vertical direction when its vertical drive motor is not engaged. Motion in all three axes can occur simultaneously in order to move the arm to the desired location as quickly as possible. Research Goal 2: Increase Fruit Visibility and Detection Objective 2.1: Fruit detection system Fruit Detection: Robust detection and localization of fruits is an essential component for an automated fruit harvester. To detect apples in color 2D images, we have trained several variants of the Yolo V5 models. To train and validate these deep object detectors, we hand-curated several hundred training and testing datasets. The training samples included clearly visible as well partial and heavily occluded apples in images. Currently, our trained models have achieved high fruit detection accuracy of 0.9 mean average precision (mAP) for Intersection of Union (IoU) of 0.5 and an average mAP of 0.75 for IoU from 0.5 to 0.95. Camera system: Deep learning models, whether to localize objects with bounding boxes or semantically label pixels just belonging to each apple, work accurately when data is abundant. Acquiring and curating datasets in an Ag setting is a resource extensive process. To minimize the size of data, we have designed and developed low-cost versions of the active light stereo camera that generate consistent images in the outdoors and are shown to reduce data size to train deep object detectors. Furthermore, deep learning models also require powerful Graphics Processing Unit (GPUs) both for training and deployment. Although these models can be trained offline using dedicated IT facilities, the requirement for high compute and could easily overwhelm computes that can be deployed in the field. The active light cameras also include a small compute that can run lightweight deep learning pipelines onboard in real-time. Our current approach is to use this camera feature as a mechanism to control image data traffic by only processing images from the camera that contain apples in them. This feature of on-the-edge fruit detection and multiple copies of these cameras also become necessary to fulfill our research thrust under the second sub-objective, which is described below. The deliverable under this sub-objective is the detected fruit bounding boxes in each image, along with the detection confidence score for each bounding box. Objective 2.2: Improved Fruit Visibility via Multi-View Geometry To this aim, we are currently in the finalizing stage of implementing an Extended Kalman Filter (EKF) based Simultaneous Localization and Mapping (SLAM) pipeline. This computer vision-based approach essentially takes detected apples from objective 2.1 as landmarks and accounts for multiple detections by tracking the centroids of each apple in the world reference frame. The deliverable under this sub-objective is the three-dimensional (3D) coordinates of all detected apples, along with the measurement uncertainties for each fruit location. Objective 2.3: Increased Fruit Visibility via Foliage Agitation To achieve this goal, we are currently designing and fabricating a low-cost forced air agitation system. This unit mainly consists of a battery-operated leaf blower as a source of forced air and articulated vents with two degrees of freedom to vector the thrust. We are currently testing parts of this design with mockup artificial trees.

Publications

• Type: Conference Papers and Presentations Status: Published Year Published: 2021 Citation: Pueyo Svoboda, N., Vougioukas, S.G. (2021) Evaluation of a First-Come-First-Serve, Arm-to-Fruit Assignment Policy for a Multi-Armed Harvesting Robot in a Simulated Orchard Environment. Presentation 2100077.
• Type: Journal Articles Status: Under Review Year Published: 2022 Citation: Arikapudi R., Vougioukas, S.G. (2022). Robotic Tree-fruit Harvesting with Arrays of Cartesian Arms: A Study of Fruit Pick Cycle Times. UNDER REVISION. Computers and Electronics in Agriculture.
• Type: Journal Articles Status: Published Year Published: 2021 Citation: Charlton, Diane (2021) "Seasonal Farm Labor and COVID-19 Spread" Applied Economic Perspectives & Policy.

Progress 04/01/20 to 03/31/21

Outputs
Target Audience:One post-doctoral researcher, four graduate students, and one undergraduate student were engaged in research and development and were mentored in the first year of the project. Also, the PI and Co-PIs gave several presentations related to robotic harvesting technologies. The audiences included growers and allied industry, entrepreneurs, researchers, faculties, students, extension agents, visiting scholars, guest lecturers, and the general public. Changes/Problems:Due to the COVID-19 pandemic, fieldwork to collect data from individual farms could not be carried out in 2020. Instead, CMU and UCD researchers used existing recorded data from prior apple harvesting experiments to test their perception and simulation systems, respectively. MSU researchers completed a review of the economic literature of technology adoption in agriculture and conducted research to assess health-related risks correlated with agricultural employment during the COVID-19 pandemic. These findings inform whether the pandemic possibly affected perceived risks associated with labor inputs. If producers perceive higher production risks associated with labor inputs, they will likely seek out additional labor-saving technologies, including robotic harvesters. Fieldwork to collect farm-level data on inputs and production will begin in 2021. What opportunities for training and professional development has the project provided?UC Davis Two Ph.D. students and one post-doctoral researcher were mentored during the first year of this project.They worked closely with PI Vougioukas and gained experience in modeling, stochastic simulation, scheduling algorithms, and ROS 2, which is used to program the simulator and the real-time platform. CMU In the first year of this project, one scientific staff (Dr. Silwal), one postdoc (Dr. Francisco Yandun), and one graduate intern (Tanvir Parhar) were hired to work on this project. This team workedwith Co-PI Kantor in the design and fabrication of the activelight camera system, with special attention paid to training the postdoc and the student in the area of custom imager design and AI-enabled object tracking. Kantor, Silwal, and Yandun also collaborated on the development and execution of anew CMU course 16889 Robotics and AI in Agriculture, and design and AI concepts from thisproject were used as a case study in the class. MSU Co-PI Charlton worked on her own during the first year of the project. How have the results been disseminated to communities of interest?PI-Vougioukas gave the following presentations. - "Harvest-support technology development at UC Davis", Stavros Vougioukas, invited speaker. UCANR Small Fruit and Vegetable Meeting, Aug. 18, 2020. - "Human-robot collaboration for fruit harvesting". Stavros Vougioukas, panelist. CITRIS-CPAR, Open Problems for Robots in Food Supply Chain, Aug. 24, 2020. -"Co-Robotic harvest-aid platforms can increase harvesting speed". Stavros Vougioukas, invited speaker. Washington State Tree Fruit Extension and Oregon State University Extension Service, Oct. 15, 2020. - "Human and ag-robot collaboration for fruit harvesting", Stavros Vougioukas, invited speaker. Ag Expo 2021: The Future of Farming. Un. of Florida, May 10-11. 2021. Co-PI Kantor in collaboration with other technical staff presented the recent progress in several national and international level workshops and conferences. - "Semantically-Assisted SLAM in Tree Canopies", George Kantor, keynote speaker, IROS 2020 Workshop on Perception, Planning and Mobility in Forestry Robotics (WPPMFR 2020) Oct 29. - "Robotics, Sensing, and AI for Agriculture". George Kantor, invited speaker, NASEM Board on Agriculture and Natural Resources. Artificial Intelligence: The Potential for Food, Agriculture, and Natural Resources, Online Meeting, July 6, 2020. - "Visual 3D Reconstruction and Dynamic Simulation of Fruit Trees for Robotic Manipulation." Francisco Yandunm IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshop: Agriculture-Vision: Challenges & Opportunities for Computer Vision in Agriculture, June 2020. Co-PI Charlton reached agricultural producers, extension agents, and others in the agricultural field about changes in the agricultural workforce and interactions between labor market dynamics and the demand for labor-saving technologies through various online seminars and media interviews: - "Sustaining Agriculture in the Presence of Farm Labor Uncertainty" Virginia Sustainable Farms and Agribusiness Education, Virginia Tech webinar series (April 29, 2021) - "COVID-19 and Labor in the Food Supply Chain" Fall Economics Conference: COVID-19's Impact on Montana Agriculture. Montana State University, Department of Agricultural Economics & Economics (November 13, 2020) -"Farm Labor and the Coronavirus Pandemic" Montana State University, Snowmester guest lecture (December 15, 2020) -"Farm Labor in Uncertain Times" Montana State University, Department of Agricultural Economics & Economics, Extension Webinar (November 25, 2020) - "The Farm Labor Problem." Interview/Webinar with K. Aleks Schaefer, JD, PhD and Trey Malone, PhD for Closing Bell webinar at Michigan State University. (August 21, 2020) -"Food and Farmworkers" Interview with Sarah Gonzalez at NPR Planet Money. (March 25, 2020) What do you plan to do during the next reporting period to accomplish the goals? Nothing Reported

Impacts
What was accomplished under these goals? Research Goal 1: Development of Linear Robot Arm Arrays Objective 1.1: Modeling and Simulation of LRA Harvester A decentralized simulator has been built in ROS2 and Python3 with four main nodes: a vehicle node to travel within the orchard, a computer vision system node to simulate fruit localization, low-level arm-movement control node(s) for the individual arms, and a scheduling node. The robot is modeled with a constant forward velocity along an orchard row (no stopping at individual trees), with over 10 m of distributed fruit. Testing has been performed on uniformly random distributed fruit with fruit density ranging between 1 fruit/m^3 and 32 fruit/m^3, as well as 'extreme' fruit distributions (such as lines and sinusoidal geometries) to test the model's reactions and various parameter effects. The arms are Cartesian robots with three independent linear axes of freedom that share their workspace with other arms in one of three horizontally stacked workspace 'cells.' The movement of each arm is based on a trapezoidal velocity profile, bounded by maximum acceleration and speed. Objective 1.2: Fruit-Picking Order Assignments for LRAs The simulator has been tested using an FCFS (First-Come-First-Serve) algorithm, implemented on non-informed, uniform randomly-distributed fruit, to observe how two performance metrics - Fruit Picking Efficiency (FPE, the percent of marketable fruits harvested) and Fruit Picking Throughput (FPT, the average number of fruits picked per unit of time) - were affected by changes in design parameters such as: arm maximum speed, number of arms per cell, and vehicle speed. Changes in maximum arm velocity or the number of arms per cell caused FPE and FPT to increase, following a sigmoid growth curve, as the parameter values increased, reaching a plateau when FPE reached 100% (FPT was around 1 fruit/s). Increases in vehicle speed, in contrast, had FPE holding steady at 100%, until speed reached 0.1 m/s when FPE began to drop linearly. FPT followed a reverse parabola, starting close to 0 fruit/s, rising to 0.8 fruit/s at around 50% FPE, then dropping. By studying the percent of the time the individual arms were idle versus picking, it was possible to see that FPT and FPE are affected by workload imbalances; the front arms are doing all the work unless the vehicle moves so fast that the front arms miss fruit that can be picked by the back arms. Specific result values are highly dependent on the harvester's set design parameters. However, by using the FCFS algorithm, none of the tests went over 1 fruit/s, an unacceptable value for a system with many arms. The largest effect on the FPT value is due to load imbalances between the arms. A harvester using FCFS acts closer to a one-armed harvester and varying the design parameters such as arm maximum velocity or the number of arms only affects the final FPE value so much. This shows the need for more targeted scheduling strategies that incorporate workload balancing to better distribute the work between the arms and increase picking speeds. Potential targeted scheduling strategies being investigated include dynamic pick up and delivery, multi-scenario approach, model-based adaptive control, as well as combinatorial optimization. Objective 1.3: Economic analysis We started a literature review of agricultural technology adoption. We organized a comprehensive list of parameters needed to model the decision to adopt the proposed robotic harvester. We created a cost-benefit model for technology adoption on farms. We are currently finalizing plans for additional data collection from fruit farms to generate input price elasticities of demand. These elasticities will enable the researchers to simulate robotic technology adoption under various scenarios, including inward labor supply shifts and possible shifts in the supply of other necessary inputs to production. Research Goal 2: Increase Fruit Visibility and Detection Objective 2.1: Fruit detection system Robust detection and localization of fruits is an essential component for an automated fruit harvester. We plan to use the current state-of-the-art,Mask R-CNN, as a multi-class detector to instantly detect and segment not only clearly visible and partially occluded fruits but also the foliage in thescene. In the reporting period, we evaluated several deep learning-based object detection algorithms to detect apples incolor images. Our current datasets consist of a sequence of images of apple canopies manually agitated with a stream of air from a consumer-grade leaf blower to expose hidden fruits. Further, the datasets also include image sequences in stationary positions as well as images with a lateral motion to capture the entire row.The list of algorithms to detect apples in images included Faster-RCNN VGG-16, Faster-RCNN RASNET-50SSD, and YoLo V3. The fruit detection accuracies were 0.89, 0.81, 0.62, and 0.81 average precisionrespectively. Objective 2.2:Improved Fruit Visibility via Multi-View Geometry This objective aims at detecting occluded fruits using a multi-view approach that accounts for occlusion and multiple fruit count. Additionally, dynamicchanges in the set of fruits to be picked will be tracked. We developed a computer vision pipeline to detect apples from multiple viewpoints, and to track the locations of apples across a sequence of images. This system maintains a 3D position estimate of every apple it has detected, which allows it to track apples across a sequence of images, including in frames where the apples are not detectable due to occlusion.This pipeline has been tested in data sets the represent the following scenarios: moving camera with no canopy agitation; non-moving camera with canopy agitation; and moving camera with canopy agitation. The results areyet to be quantified, but qualitatively they look promising: the system is able to track more apples than our previous methods in all of the test cases. Objective 2.3:Increased Fruit Visibility via Foliage Agitation This objective aims at designing, building, and testing a novel tree foliage agitation system that increases fruit visibility in a sequence of images or avideo stream. More specifically, by causing leaves to move it is expected that different parts of the fruit surface will become visible over a series ofsuccessive image frames. We developed a new low-cost active perception system that will be more easily integrated with the canopy agitation system. The system is composed of a stereo pair with active flash lighting, which is less susceptible to natural lightingconditions. The primary advance of the new system compared to our earlier imagers is its small footprint, which will make it possible to integrate an array of imagers with the canopy harvesting and leaf agitation platform. The design consists of small baseline stereocolor cameras housed in a 3D printed body with carbon fiber composites. This design weighs 260 grams and thedimensions are 73 x 38 x 108 mm (W x H x D). This imager was used to collect an initial data set of images of apple canopies under forced-air agitation. The agitation, in this case, was provided by a COTS leaf blower. The resulting data set was used to support the detection and tracking work described under Objectives 2.1 and 2.2. It will also be used to inform the design of a custom canopy agitator, which will be developed in the next reporting period.

Publications

• Type: Journal Articles Status: Under Review Year Published: 2021 Citation: Arikapudi, R., Vougioukas, S.G. (2021) A Study of Fruit Picking Cycle Times of Arrays of Linear Arms. Computers and Electronics in Agriculture.
• Type: Conference Papers and Presentations Status: Accepted Year Published: 2021 Citation: Pueyo Svoboda, Natalie, Vougioukas, S.G. (2021) Evaluation of a first-come-first-serve, arm-to-fruit assignment policy for a multi-armed harvesting robot in a simulated orchard environment. ASABE Annual Intl. Meeting, July 12, Annaheim, CA.
• Type: Journal Articles Status: Under Review Year Published: 2021 Citation: Diane, C. (2021) Seasonal Farm Labor and Risk of COVID-19 Spread. Applied Economic Perspectives and Policies.