Progress 04/01/22 to 03/31/23
Outputs
Target Audience:UC Davis Four graduate students and one undergraduate student were engaged in research and development and were mentored during the project's third year. The PIs and scientist staff Silwal gave presentations on robotic harvesting technologies at national and international conferences, workshops, growers' conferences, and academic lectures. The audiences included growers, entrepreneurs, researchers, faculties, students, extension agents, visiting scholars, and the general public. CMU Co-PI Kantor and including scientist staff Silwal and Yandun have included findings from this research at several national (local growers conference, invited speakers), international conferences, and academic lectures. The audiences included growers, industry, entrepreneurs, researchers, faculties, students, visiting scholars, guest lecturers, and the public. MSU Findings from economic analysis will be disseminated through peer-reviewed publications and blogposts. The target audiences include engineers, growers, and policymakers. Findings can help engineers determine the tradeoffs in harvesting profitability between improvements in various robot parameters. Growers can use the analysis to better assess progress in robotic efficiency and determine whether a robot might be profitable for adoption given various speeds, share of apples picked from the orchard, and bruising/cull rates. They can also see how the profitability of robot adoption varies as farm worker wages rise. Finally, discussion of economic findings highlights the need for ongoing research to investigate the potential consequences of robotic adoption on employment, wages, and required skills up and down the agricultural supply chain. This and related research has important policy implications for farm labor employment, immigration, international trade, and agricultural research and development. Changes/Problems:
Nothing Reported
What opportunities for training and professional development has the project provided?UC Davis Two Ph.D. students and one undergraduate student were mentored during the third year of this project at UC Davis. The graduate students worked closely with PI Vougioukas and gained experience in modeling and solving task assignments and scheduling problems using mathematical programming and numerical optimization tools, such as the Gurobi open-source optimized. One undergraduate student presented her work at the ASABE meeting. The undergraduate student worked during the summer and developed real-time motion control software to control the motions of the motors of the multi-arm prototype. All students gained more programming experience in python, C++, and ROS and had some exposure to ROS 2. The undergraduate student also got hands-on experience with building the electronics of the robot's motion control system. CMU Summer internship in robotics for two undergraduates, professional training for one graduate student, and leadership opportunities to staff scientists at CMU. How have the results been disseminated to communities of interest?UC Davis PI-Vougioukas gave the following presentations as an invited speaker. May 20, 2022. UC Merced Multicampus Research Program Initiative for Labor & Automation in California Agriculture: Equity, Productivity, & Sustainability. "Worker and Robot Collaboration for Fruit Harvesting" May 24, 2022. UC Davis Emerging Technologies Postharvest Technologies Workshop. "Theory and Research in Mechanized & Robotic Harvesting" September 16, 2022. Pomology Cross-Commodity Discussion Meeting - Labor. Online. "Theory and Research in Mechanized & Robotic Harvesting" October 18, 2022. FIRA USA, Fresno CA. "Harvest-Assist Collaborative Robots" November 14, 2022. Sustainable Ag Expo & Winegrowing Summit, San Luis Obispo, CA. "Agricultural Robots: Technology, Applications, and Adoption" March 20, 2023. UC Davis. Farm Labor in the 2020s Demand, Supply, and Markets. "Robotic and Robot-Assisted Fruit Harvesting" What do you plan to do during the next reporting period to accomplish the goals?
Nothing Reported
Impacts
What was accomplished under these goals?
Research Goal 1: Development of Linear Robot Arm Arrays Objective 1.1: Modeling and Simulation of LRA Harvester The distributed simulator was developed and tested in the project's first year, as planned in the proposed timeline. This year, some effort was invested in debugging to increase the robustness of the model and simulator. Objective 1.2: Fruit-Picking Order Assignments for LRAs An existing time-dependent team orienteering problem with time windows (TDTOPTW) mixed integer programming (MIP) was extended and modified to provide better results than previously achieved. Two performance metrics are being used in these tests, Fruit Picking Efficiency (FPE, the percent of marketable fruits harvested) and Fruit Picking Throughput (FPT, the average number of fruits picked per unit of time). The original TDTOPTW formulation was only optimized to obtain the maximum FPE, but harvesting time (FPT) is also very important to growers. Three different strategies were pursued to include optimization for FPT: (a) the original TDTOPTW formulation was run in a loop which tested changes in resulting FPE and FPT as vehicle speeds increased. The chosen schedule was the one with the highest FPT value, given a minimum FPE value was met. Because this is a greedy algorithm, the results were reasonably good but very slow. For strategies (b) and (c), the MIP formulation was changed to include vehicle speed, with changes to constraints and the objective function. Strategy (b) had an objective function to maximize FPE*FPT with soft minimum FPE and FPT constraints, while strategy (c) had the objective to minimize the makespan, or the schedule's length of time from start to finish, with soft constraints for a minimum FPE. Lastly, a first-come-first-serve (FCFS) naive schedule was used within the greedy velocity loop to find an initial lower-bound vehicle speed that still produces a minimum FPE. This lower bound was then used by the MIP formulations when solving for the schedule. This reduced the problem space, increasing solving time and improving the results. All MIP formulations had problems finding solutions when solving for the length of an entire orchard row. A single side can have thousands of apples. Thus, the orchard row was broken into smaller, overlapping sections that would be easier to solve and allow for more "dynamic" scheduling. Not only could the near-optimal vehicle speed be found for each section, allowing the system to speed up or slow down according to incoming fruits, but the arms' vertical workspace within a column could also be optimized to provide basic load balancing. This was done by allowing the arms to share the whole column as a workspace but limiting their vertical movement through software controls to avoid the need for collision avoidance. The vertical columns were divided into software-imposed horizontal rows that contain the same number of fruits. Furthermore, by overlapping the sections, changes to fruit locations, such as branches moving as apples are removed, could be acted upon. Lastly, a strategy inspired by model predictive control (MPC) was used to strengthen the minimum FPE and FPT requirements. This was needed because scheduling for small, overlapping sections resulted in not being able to predict the minimum FPE needed to harvest 95% of total fruits in the entire orchard row. First, a schedule was obtained for a section of the orchard row in front of, and equal in length to, the robot's workspace, or the volume of space the arms are able to harvest from that point in time, plus a horizon, or the section immediately in front of the workspace. However, the robot only follows through with the initial parts of the schedule (e.g., the schedule is solved for 2.4 m, but only the first 0.6 m of the schedule is acted upon) before creating a new schedule that takes into account any changes. Testing is being performed to see how this addition affects the different MIP formulations and what parameters can be changed to further maximize the FPE and FPT combination, as well as decrease solving speeds. Objective 1.3: Economic analysis Co-PI Charlton created a cost-benefit model for a representative apple grower's decision to adopt robotic harvest technology. Interviews with apple growers in Washington State helped inform the model. Using the cost-benefit model as a framework, Charlton and co-authors performed data analysis to solve for the break-even cost of a robot such that profits from manual harvesting are equal to profits from robotic harvesting. Parameters for the data analysis came from the "2019 Cost Estimates of Establishing, Producing and Packing Gala Apples in Washington".[1] Co-PI Vougioukas additionally provided ranges for robotic parameters, including picking speed, the share of apples in the orchard that the robot picks, and bruising rates. The Co-PIs conducted 5 break-even analyses using this model: First, they examined the maximum robot cost such that profits are equal for robotic and traditional manual harvesting. Second, they analyzed how the break-even cost of the robot changes for different picking speeds, harvest-induced damage rates, and percentages harvested. Third, for a given upfront cost of a robot, they computed the tradeoff between the wage rate and changes in the three robotic parameters. Fourth, because revenues, costs, harvest-induced damage rates, etc., differ across apple varieties, they computed the break-even cost of the robot for Honeycrisp apples, which have a higher farmgate price than Gala and bruise more easily. [1] Gallardo, R.K., and S.P. Galinato. 2021. "2019 Cost Estimates of Establishing, Producing and Packing Gala Apples in Washington." Washington State University Extension Bulletin TB18E. Objective 1.4: Engineering Design and Development of LRA Research Goal 2: Increase Fruit Visibility and Detection Objective 2.1: Fruit detection system In the previous reporting, using the latest YOLO network, we reported our trained models to achieve high fruit detection accuracy of 0.9 mean average precision (mAP) for Intersection of Union (IoU) of 0.5 and an average mAP of 0.75 for IoU from 0.5 to 0.95 using images from our active lighting camera system. These results are adequate for our testing, and currently we are considering this objective to be completed Objective 2.2: Improved Fruit Visibility via Multi-View Geometry We have successfully designed and tested two baseline algorithms to track apples in images. The first algorithm includes a 3D Kalman filter using Mahalanobis distance as a measure to keep track of the same apples in consecutive images. The second algorithm uses deep SORT (Simple Online Real Time Tracking), a deep learning-based approach to tracking objects in images. Both algorithms show similar accuracy in testing while lacking robustness in highly occluded apples from foliage, branches, and nearby fruits. Our current efforts are to use 3D shape completion on partially visible fruits to improve apple tracking. Objective 2.3: Increased Fruit Visibility via Foliage Agitation We have completed the design and fabrication of a canopy agitation system using a modified leaf blower and digital servo motors to actuate them. More specifically, the control system allows the canopy agitation system to target any location in the canopy as well as the volume of air. Currently, we are investigating a baseline algorithm to actuate the system using density and fruit detection scores from obj. 2.1. We are also investigating a deep Reinforcement Learning (RL) algorithm to intelligently agitate the canopy that learns from trials and errors. A simulation environment is currently being developed to train the RL model. Research Goal 3: System Integration and Evaluation This goal will be addressed in the coming year.
Publications
- Type:
Journal Articles
Status:
Submitted
Year Published:
2023
Citation:
Arikapudi R., Vougioukas, S.G. (2023). Robotic Tree-fruit Harvesting with Arrays of Cartesian Arms: A Study of Fruit Pick Cycle Times. Computers and Electronics in Agriculture
- Type:
Journal Articles
Status:
Submitted
Year Published:
2023
Citation:
Charlton, D., Devadoss, S., Gallardo, R.K., Luckstead, J., Vougioukas, S. (2023) Economic Viability of Robotic Fruit Harvesters to Reduce Large Seasonal Labor Demands. Journal of Agricultural and Resource Economics
- Type:
Journal Articles
Status:
Published
Year Published:
2023
Citation:
Peng, C., Fei, Z., Vougioukas, SG. (2023). GNSS-Free End-of-Row Detection and Headland Maneuvering for Orchard Navigation Using a Depth Camera. Machines 11(1), 84.
- Type:
Journal Articles
Status:
Published
Year Published:
2022
Citation:
Fei, Z., Vougioukas, S.G. (2022). Row-sensing Templates: A Generic 3D Sensor-based Approach to Robot Localization with Respect to Orchard Row Centerlines. Journal of Field Robotics, 39(6):712-738
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2022
Citation:
Pueyo Svoboda, N., Vougioukas, S.G. (2022). Testing a combinatorial?based melon harvesting scheduling algorithm on fruit orchard harvesting and the effects of varying the robots geometric configuration. ASABE Annual International Meeting. Houston, Texas. Presentation # 2200427.
|
Progress 04/01/21 to 03/31/22
Outputs
Target Audience:Two post-doctoral researchers, two graduate students, and one undergraduate student were engaged in research and development and were mentored in the second year of the project. Also, the PI and Co-PIs and scientist staff Silwal and Yandun gave several presentations related to robotic harvesting technologies at several national, and international conferences, workshops, growers' conferences, and academic lectures. The audiences included growers and allied industry, entrepreneurs, researchers, faculties, students, extension agents, visiting scholars, guest lecturers, and the general public. Due to the pandemic, field visits to farms to calibrate a model for cost-benefit analysis were delayed. However, co-PI Charlton managed to visit apple orchards in California and Washington State, and she has created a preliminary cost-benefit table to assess the break-even cost of a robot at which robotic harvesting would be equivalent to hiring workers to hand-pick apples. The table can be adjusted for a range of aspects, including bruising and culling, and yield. Charlton also published a paper in the Applied Economics Perspectives and Policy journal on the relationship between monthly variation in historical fruit, vegetable, and horticultural employment and new cases of COVID-19 within U.S. counties in 2020. The findings help illuminate the possible relationship between COVID-19 and fruit harvest employment, which could be of interest in anticipating changes in demand for robotic harvesters. Changes/Problems:Due to the issue caused in the global supply chain by the COVID-19 pandemic, the acquisition of actuation and imaging sensors and other necessary supplies have been severely affected. The significant increase in the lead time to acquire such necessary hardware has delayed the design and fabrication of the LRA and the camera system and delayed field visits. Also, field visits to farms to calibrate a model for cost-benefit analysis were delayed. The PIs anticipate that a no-cost extension of the project may prove to be necessary. What opportunities for training and professional development has the project provided?One Ph.D. student and two post-doctoral researchers were mentored during the second year of this project, at UC Davis. They worked closely with PI Vougioukas, and gained experience in modeling, simulation, task assignment and scheduling algorithms, and ROS 2, which is used to program the simulator and the real-time platform. The Ph.D. student also got hands-on experience with building the electronic hardware of the robot's motion control system. One graduate student and one undergraduate student were mentored by PI Kantor in the area of high-speed imaging, machine learning, modeling, and simulation. How have the results been disseminated to communities of interest?PI-Vougioukas gave the following presentations, as an invited speaker. "The Smart Shift to Advanced Agriculture in the U.S. and Sweden". Webinar organized by the Swedish Trade & Invest Council. Sept. 9, 2021. "Collaborative-Robotic Harvest-aid platforms can increase harvesting speed". Webinar during the 2021 World Ag Expo. Aug. 19, 2021. "Human-robot Collaboration for Fruit Harvesting". CITRIS Research Webinar. Sept. 15, 2021. "Agricultural Robotics and labor". Digital Transformation Workshop, UC Davis Graduate School of Management. Nov. 10, 2021. "Robotic harvesting and precision yield mapping". UC ANR Precision Ag Workgroup Meeting. Mar. 9, 2022. "Agricultural robotics, robotic-aided harvesting and automation". Webinar organized by the Phenomics and Plant Robotics Center (P2RC) at the University of Georgia. Mar. 17, 2022. PI Kantor and scientist staff Silwal and Yandun gave several presentations related to robotic harvesting technologies at several national, and international conferences, workshops, growers' conferences, and academic lectures. What do you plan to do during the next reporting period to accomplish the goals?
Nothing Reported
Impacts
What was accomplished under these goals?
Research Goal 1: Development of Linear Robot Arm Arrays Objective 1.1: Modeling and Simulation of LRA Harvester The distributed simulator was developed and tested in the first year of the project, as planned in the proposed timeline. This year, some effort was invested in testing and debugging to increase the robustness of the model and simulator. Objective 1.2: Fruit-Picking Order Assignments for LRAs Arm-to-fruit assignment strategies were tested to produce better results than the previously used First-Come-First-Serve, greedy strategy. Two performance metrics are being used in these tests, Fruit Picking Efficiency (FPE, the percent of marketable fruits harvested) and Fruit Picking Throughput (FPT, the average number of fruits picked per unit of time). The first algorithm attempted was an existing combinatorial heuristic, originally developed for a melon harvesting robot with multiple cartesian arms set in a row. Although very fast and optimal when maximizing FPE (and only FPE), this algorithm proved less useful for our system. It assumes that it takes the same, constant picking time for every fruit, that the fruit distribution is two-dimensional, and that arms cannot revisit locations even if fruits may have been missed. This works well for melons, which grow on the ground (2D) and in low densities (their examples went up to 4 fruits/m^2). However, in orchard fruits like apples, which can grow inside or outside of the canopy and have much higher densities such as 46 fruits/m^3 (from digitized fruit data), would be badly served by this combinatorial heuristic. As our second assignment strategy, we developed a Mixed Integer Programming model expressed as a time-dependent team orienteering problem with time windows (TDTOPTW). The combinatorial heuristic is based on this particular formulation, so it also has a constant picking time and does not revisit locations; however, it is easily expandable (e.g. the addition of columns of arms through a single mathematically-described constraint). The model proved useful for testing design parameters such as the effects of the vehicle's velocity, dividing the rows by different parameters such as the number of fruits or the overall height of the robot, or different fruit distributions. An important accomplishment was an 'extension' of the MIP formulation to minimize the overall picking time, also known as the makespan. This new formulation computes the optimal harvester travel speed and the optimal overall arms-to-fruits schedule that maximizes the FPT for a given minimum FPE (say 95% picked minimum). This problem becomes intractable for even low densities because it is one of the harder NP-hard and NP-complete problems. Hence, heuristics to solve it are being developed. Objective 1.3: Economic analysis Co-PI Charlton visited apple orchards in California and Washington State, and she has created a preliminary cost-benefit table to assess the break-even cost of a robot at which robotic harvesting would be equivalent to hiring workers to hand-pick apples. The table can be adjusted for a range of aspects, including bruising and culling, and yield. Charlton also published a paper in the Applied Economics Perspectives and Policy journal on the relationship between monthly variation in historical fruit, vegetable, and horticultural employment and new cases of COVID-19 within U.S. counties in 2020. The findings help illuminate the possible relationship between COVID-19 and fruit harvest employment, which could be of interest in anticipating changes in demand for robotic harvesters. Objective 1.4: Engineering Design and Development of LRA Numerous design options were investigated and modeled in CAD with an emphasis on low cost, robustness, and ease of fabrication and repair. The selected design for the cell consists of a rigid aluminum frame that can accommodate up to three robotic arms. Each arm is mounted on a carriage and moves in all three linear axes. Vertical motion of the carriage is accomplished through a rack and pinion system. The horizontal side-to-side motion of the arm is via a fast-travel Acme screw drive. Extension and retraction of the arm use a lightweight rack and pinion-type system. A prototype cell has been built with a single carriage though it can accommodate up to three carriages. A low-power electromagnetic brake prevents the carriage from moving in the vertical direction when its vertical drive motor is not engaged. Motion in all three axes can occur simultaneously in order to move the arm to the desired location as quickly as possible. Research Goal 2: Increase Fruit Visibility and Detection Objective 2.1: Fruit detection system Fruit Detection: Robust detection and localization of fruits is an essential component for an automated fruit harvester. To detect apples in color 2D images, we have trained several variants of the Yolo V5 models. To train and validate these deep object detectors, we hand-curated several hundred training and testing datasets. The training samples included clearly visible as well partial and heavily occluded apples in images. Currently, our trained models have achieved high fruit detection accuracy of 0.9 mean average precision (mAP) for Intersection of Union (IoU) of 0.5 and an average mAP of 0.75 for IoU from 0.5 to 0.95. Camera system: Deep learning models, whether to localize objects with bounding boxes or semantically label pixels just belonging to each apple, work accurately when data is abundant. Acquiring and curating datasets in an Ag setting is a resource extensive process. To minimize the size of data, we have designed and developed low-cost versions of the active light stereo camera that generate consistent images in the outdoors and are shown to reduce data size to train deep object detectors. Furthermore, deep learning models also require powerful Graphics Processing Unit (GPUs) both for training and deployment. Although these models can be trained offline using dedicated IT facilities, the requirement for high compute and could easily overwhelm computes that can be deployed in the field. The active light cameras also include a small compute that can run lightweight deep learning pipelines onboard in real-time. Our current approach is to use this camera feature as a mechanism to control image data traffic by only processing images from the camera that contain apples in them. This feature of on-the-edge fruit detection and multiple copies of these cameras also become necessary to fulfill our research thrust under the second sub-objective, which is described below. The deliverable under this sub-objective is the detected fruit bounding boxes in each image, along with the detection confidence score for each bounding box. Objective 2.2: Improved Fruit Visibility via Multi-View Geometry To this aim, we are currently in the finalizing stage of implementing an Extended Kalman Filter (EKF) based Simultaneous Localization and Mapping (SLAM) pipeline. This computer vision-based approach essentially takes detected apples from objective 2.1 as landmarks and accounts for multiple detections by tracking the centroids of each apple in the world reference frame. The deliverable under this sub-objective is the three-dimensional (3D) coordinates of all detected apples, along with the measurement uncertainties for each fruit location. Objective 2.3: Increased Fruit Visibility via Foliage Agitation To achieve this goal, we are currently designing and fabricating a low-cost forced air agitation system. This unit mainly consists of a battery-operated leaf blower as a source of forced air and articulated vents with two degrees of freedom to vector the thrust. We are currently testing parts of this design with mockup artificial trees.
Publications
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2021
Citation:
Pueyo Svoboda, N., Vougioukas, S.G. (2021) Evaluation of a First-Come-First-Serve, Arm-to-Fruit Assignment Policy for a Multi-Armed Harvesting Robot in a Simulated Orchard Environment. Presentation 2100077.
- Type:
Journal Articles
Status:
Under Review
Year Published:
2022
Citation:
Arikapudi R., Vougioukas, S.G. (2022). Robotic Tree-fruit Harvesting with Arrays of Cartesian Arms: A Study of Fruit Pick Cycle Times. UNDER REVISION. Computers and Electronics in Agriculture.
- Type:
Journal Articles
Status:
Published
Year Published:
2021
Citation:
Charlton, Diane (2021) "Seasonal Farm Labor and COVID-19 Spread" Applied Economic Perspectives & Policy.
|
Progress 04/01/20 to 03/31/21
Outputs
Target Audience:One post-doctoral researcher, four graduate students, and one undergraduate student were engaged in research and development and were mentored in the first year of the project. Also, the PI and Co-PIs gave several presentations related to robotic harvesting technologies. The audiences included growers and allied industry, entrepreneurs, researchers, faculties, students, extension agents, visiting scholars, guest lecturers, and the general public. Changes/Problems:Due to the COVID-19 pandemic, fieldwork to collect data from individual farms could not be carried out in 2020. Instead, CMU and UCD researchers used existing recorded data from prior apple harvesting experiments to test their perception and simulation systems, respectively. MSU researchers completed a review of the economic literature of technology adoption in agriculture and conducted research to assess health-related risks correlated with agricultural employment during the COVID-19 pandemic. These findings inform whether the pandemic possibly affected perceived risks associated with labor inputs. If producers perceive higher production risks associated with labor inputs, they will likely seek out additional labor-saving technologies, including robotic harvesters. Fieldwork to collect farm-level data on inputs and production will begin in 2021. What opportunities for training and professional development has the project provided?UC Davis Two Ph.D. students and one post-doctoral researcher were mentored during the first year of this project.They worked closely with PI Vougioukas and gained experience in modeling, stochastic simulation, scheduling algorithms, and ROS 2, which is used to program the simulator and the real-time platform. CMU In the first year of this project, one scientific staff (Dr. Silwal), one postdoc (Dr. Francisco Yandun), and one graduate intern (Tanvir Parhar) were hired to work on this project. This team workedwith Co-PI Kantor in the design and fabrication of the activelight camera system, with special attention paid to training the postdoc and the student in the area of custom imager design and AI-enabled object tracking. Kantor, Silwal, and Yandun also collaborated on the development and execution of anew CMU course 16889 Robotics and AI in Agriculture, and design and AI concepts from thisproject were used as a case study in the class. MSU Co-PI Charlton worked on her own during the first year of the project. How have the results been disseminated to communities of interest?PI-Vougioukas gave the following presentations. - "Harvest-support technology development at UC Davis", Stavros Vougioukas, invited speaker. UCANR Small Fruit and Vegetable Meeting, Aug. 18, 2020. - "Human-robot collaboration for fruit harvesting". Stavros Vougioukas, panelist. CITRIS-CPAR, Open Problems for Robots in Food Supply Chain, Aug. 24, 2020. -"Co-Robotic harvest-aid platforms can increase harvesting speed". Stavros Vougioukas, invited speaker. Washington State Tree Fruit Extension and Oregon State University Extension Service, Oct. 15, 2020. - "Human and ag-robot collaboration for fruit harvesting", Stavros Vougioukas, invited speaker. Ag Expo 2021: The Future of Farming. Un. of Florida, May 10-11. 2021. Co-PI Kantor in collaboration with other technical staff presented the recent progress in several national and international level workshops and conferences. - "Semantically-Assisted SLAM in Tree Canopies", George Kantor, keynote speaker, IROS 2020 Workshop on Perception, Planning and Mobility in Forestry Robotics (WPPMFR 2020) Oct 29. - "Robotics, Sensing, and AI for Agriculture". George Kantor, invited speaker, NASEM Board on Agriculture and Natural Resources. Artificial Intelligence: The Potential for Food, Agriculture, and Natural Resources, Online Meeting, July 6, 2020. - "Visual 3D Reconstruction and Dynamic Simulation of Fruit Trees for Robotic Manipulation." Francisco Yandunm IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshop: Agriculture-Vision: Challenges & Opportunities for Computer Vision in Agriculture, June 2020. Co-PI Charlton reached agricultural producers, extension agents, and others in the agricultural field about changes in the agricultural workforce and interactions between labor market dynamics and the demand for labor-saving technologies through various online seminars and media interviews: - "Sustaining Agriculture in the Presence of Farm Labor Uncertainty" Virginia Sustainable Farms and Agribusiness Education, Virginia Tech webinar series (April 29, 2021) - "COVID-19 and Labor in the Food Supply Chain" Fall Economics Conference: COVID-19's Impact on Montana Agriculture. Montana State University, Department of Agricultural Economics & Economics (November 13, 2020) -"Farm Labor and the Coronavirus Pandemic" Montana State University, Snowmester guest lecture (December 15, 2020) -"Farm Labor in Uncertain Times" Montana State University, Department of Agricultural Economics & Economics, Extension Webinar (November 25, 2020) - "The Farm Labor Problem." Interview/Webinar with K. Aleks Schaefer, JD, PhD and Trey Malone, PhD for Closing Bell webinar at Michigan State University. (August 21, 2020) -"Food and Farmworkers" Interview with Sarah Gonzalez at NPR Planet Money. (March 25, 2020) What do you plan to do during the next reporting period to accomplish the goals?
Nothing Reported
Impacts
What was accomplished under these goals?
Research Goal 1: Development of Linear Robot Arm Arrays Objective 1.1: Modeling and Simulation of LRA Harvester A decentralized simulator has been built in ROS2 and Python3 with four main nodes: a vehicle node to travel within the orchard, a computer vision system node to simulate fruit localization, low-level arm-movement control node(s) for the individual arms, and a scheduling node. The robot is modeled with a constant forward velocity along an orchard row (no stopping at individual trees), with over 10 m of distributed fruit. Testing has been performed on uniformly random distributed fruit with fruit density ranging between 1 fruit/m^3 and 32 fruit/m^3, as well as 'extreme' fruit distributions (such as lines and sinusoidal geometries) to test the model's reactions and various parameter effects. The arms are Cartesian robots with three independent linear axes of freedom that share their workspace with other arms in one of three horizontally stacked workspace 'cells.' The movement of each arm is based on a trapezoidal velocity profile, bounded by maximum acceleration and speed. Objective 1.2: Fruit-Picking Order Assignments for LRAs The simulator has been tested using an FCFS (First-Come-First-Serve) algorithm, implemented on non-informed, uniform randomly-distributed fruit, to observe how two performance metrics - Fruit Picking Efficiency (FPE, the percent of marketable fruits harvested) and Fruit Picking Throughput (FPT, the average number of fruits picked per unit of time) - were affected by changes in design parameters such as: arm maximum speed, number of arms per cell, and vehicle speed. Changes in maximum arm velocity or the number of arms per cell caused FPE and FPT to increase, following a sigmoid growth curve, as the parameter values increased, reaching a plateau when FPE reached 100% (FPT was around 1 fruit/s). Increases in vehicle speed, in contrast, had FPE holding steady at 100%, until speed reached 0.1 m/s when FPE began to drop linearly. FPT followed a reverse parabola, starting close to 0 fruit/s, rising to 0.8 fruit/s at around 50% FPE, then dropping. By studying the percent of the time the individual arms were idle versus picking, it was possible to see that FPT and FPE are affected by workload imbalances; the front arms are doing all the work unless the vehicle moves so fast that the front arms miss fruit that can be picked by the back arms. Specific result values are highly dependent on the harvester's set design parameters. However, by using the FCFS algorithm, none of the tests went over 1 fruit/s, an unacceptable value for a system with many arms. The largest effect on the FPT value is due to load imbalances between the arms. A harvester using FCFS acts closer to a one-armed harvester and varying the design parameters such as arm maximum velocity or the number of arms only affects the final FPE value so much. This shows the need for more targeted scheduling strategies that incorporate workload balancing to better distribute the work between the arms and increase picking speeds. Potential targeted scheduling strategies being investigated include dynamic pick up and delivery, multi-scenario approach, model-based adaptive control, as well as combinatorial optimization. Objective 1.3: Economic analysis We started a literature review of agricultural technology adoption. We organized a comprehensive list of parameters needed to model the decision to adopt the proposed robotic harvester. We created a cost-benefit model for technology adoption on farms. We are currently finalizing plans for additional data collection from fruit farms to generate input price elasticities of demand. These elasticities will enable the researchers to simulate robotic technology adoption under various scenarios, including inward labor supply shifts and possible shifts in the supply of other necessary inputs to production. Research Goal 2: Increase Fruit Visibility and Detection Objective 2.1: Fruit detection system Robust detection and localization of fruits is an essential component for an automated fruit harvester. We plan to use the current state-of-the-art,Mask R-CNN, as a multi-class detector to instantly detect and segment not only clearly visible and partially occluded fruits but also the foliage in thescene. In the reporting period, we evaluated several deep learning-based object detection algorithms to detect apples incolor images. Our current datasets consist of a sequence of images of apple canopies manually agitated with a stream of air from a consumer-grade leaf blower to expose hidden fruits. Further, the datasets also include image sequences in stationary positions as well as images with a lateral motion to capture the entire row.The list of algorithms to detect apples in images included Faster-RCNN VGG-16, Faster-RCNN RASNET-50SSD, and YoLo V3. The fruit detection accuracies were 0.89, 0.81, 0.62, and 0.81 average precisionrespectively. Objective 2.2:Improved Fruit Visibility via Multi-View Geometry This objective aims at detecting occluded fruits using a multi-view approach that accounts for occlusion and multiple fruit count. Additionally, dynamicchanges in the set of fruits to be picked will be tracked. We developed a computer vision pipeline to detect apples from multiple viewpoints, and to track the locations of apples across a sequence of images. This system maintains a 3D position estimate of every apple it has detected, which allows it to track apples across a sequence of images, including in frames where the apples are not detectable due to occlusion.This pipeline has been tested in data sets the represent the following scenarios: moving camera with no canopy agitation; non-moving camera with canopy agitation; and moving camera with canopy agitation. The results areyet to be quantified, but qualitatively they look promising: the system is able to track more apples than our previous methods in all of the test cases. Objective 2.3:Increased Fruit Visibility via Foliage Agitation This objective aims at designing, building, and testing a novel tree foliage agitation system that increases fruit visibility in a sequence of images or avideo stream. More specifically, by causing leaves to move it is expected that different parts of the fruit surface will become visible over a series ofsuccessive image frames. We developed a new low-cost active perception system that will be more easily integrated with the canopy agitation system. The system is composed of a stereo pair with active flash lighting, which is less susceptible to natural lightingconditions. The primary advance of the new system compared to our earlier imagers is its small footprint, which will make it possible to integrate an array of imagers with the canopy harvesting and leaf agitation platform. The design consists of small baseline stereocolor cameras housed in a 3D printed body with carbon fiber composites. This design weighs 260 grams and thedimensions are 73 x 38 x 108 mm (W x H x D). This imager was used to collect an initial data set of images of apple canopies under forced-air agitation. The agitation, in this case, was provided by a COTS leaf blower. The resulting data set was used to support the detection and tracking work described under Objectives 2.1 and 2.2. It will also be used to inform the design of a custom canopy agitator, which will be developed in the next reporting period.
Publications
- Type:
Journal Articles
Status:
Under Review
Year Published:
2021
Citation:
Arikapudi, R., Vougioukas, S.G. (2021) A Study of Fruit Picking Cycle Times of Arrays of Linear Arms. Computers and Electronics in Agriculture.
- Type:
Conference Papers and Presentations
Status:
Accepted
Year Published:
2021
Citation:
Pueyo Svoboda, Natalie, Vougioukas, S.G. (2021) Evaluation of a first-come-first-serve, arm-to-fruit assignment policy for a multi-armed harvesting robot in a simulated orchard environment. ASABE Annual Intl. Meeting, July 12, Annaheim, CA.
- Type:
Journal Articles
Status:
Under Review
Year Published:
2021
Citation:
Diane, C. (2021) Seasonal Farm Labor and Risk of COVID-19 Spread. Applied Economic Perspectives and Policies.
|
|