Source: UNIVERSITY OF CALIFORNIA, DAVIS submitted to NRP
MODEL-BASED DESIGN OF FRUIT TREE MECHANIZED HARVESTING SYSTEMS
Sponsoring Institution
National Institute of Food and Agriculture
Project Status
COMPLETE
Funding Source
Reporting Frequency
Annual
Accession No.
1008489
Grant No.
2016-67021-24532
Cumulative Award Amt.
$370,486.00
Proposal No.
2015-06149
Multistate No.
(N/A)
Project Start Date
Jan 1, 2016
Project End Date
Dec 31, 2019
Grant Year
2016
Program Code
[A1521]- Agricultural Engineering
Recipient Organization
UNIVERSITY OF CALIFORNIA, DAVIS
410 MRAK HALL
DAVIS,CA 95616-8671
Performing Department
Biological & Agricultural Eng.
Non Technical Summary
Engineers working in fruit harvest mechanization lack the modeling tools that would enable them to investigate the coupled effects of orchard layout, tree structure, fruit distribution and harvester design on fruit picking productivity. Instead, engineers rely on a prototype-building and field-testing cycle, which is slow, costly, and cannot explore the above-mentioned interactions.Our goal is to build and validate an open-source design tool that integrates mechanical simulation with validated models of fruit-bearing orchard trees, in order to evaluate the fruit picking productivity of selective fruit picking systems. In support of this goal, a library of agronomically correct geometric models of ready-to-harvest fruit-bearing trees will be generated, based on data from extensive digitization of tree shapes and fruit positions in commercial orchards. This library will be used to create virtual orchards within a customized open-source robotics simulator. The predictive power of the developed tool will be evaluated by simulating and executing harvesting using a two-arm robot on a mobile platform, and by comparing the predicted and achieved productivities under controlled conditions. The simulator will also be used to calculate the optimal design parameters that maximize the productivity of a conceptual robotic harvester with several actuated arms, operating on alternative tree architectures.The proposed open-source design tool is expected to accelerate the development of novel orchard mechanized harvesting systems. It will contribute towards the sustainability of the US fruit production industry by improving fruit quality, hygiene, and safety, through the reduction of the industry's exposure to labor shortage related risks.
Animal Health Component
20%
Research Effort Categories
Basic
30%
Applied
20%
Developmental
50%
Classification

Knowledge Area (KA)Subject of Investigation (SOI)Field of Science (FOS)Percent
40253102020100%
Knowledge Area
402 - Engineering Systems and Equipment;

Subject Of Investigation
5310 - Machinery and equipment;

Field Of Science
2020 - Engineering;
Goals / Objectives
Our long-term goals are to build and experimentally validate an open-source software design tool for orchard harvest mechanization, and use this tool to design novel tree fruit harvesting machines. This tool will be able to predict fruit harvesting productivity metrics of a specific selective (i.e., not shake-and-catch) fruit harvester design in a given type of orchards by simulating the harvesting process. The productivity metrics that will be computed will be fruit picking efficiency, throughput, and size. The first two metrics have been identified as the most important variables (along with purchase price) that define harvest cost. The third metric relates to quality and expected economic return. All metrics are strongly coupled to machine design and can be used to guide it. Actually, quantitative evaluation of machine productivity could be also used to evaluate alternative orchard layouts and tree training systems.Objective 1. Orchard modelingThis objective aims at removing the main obstacle in model-based design of tree-fruit harvesting systems. Its goal is to create a library of statistical models of fruit-bearing trees that can be used/sampled to generate instances of agronomically and structurally correct geometric representations of ready-to-harvest trees. The models will be created using data collected from real trees in commercial orchards. We don't anticipate that the models will have any predictive power for problems like yield-prediction. Instead, these models are intended to be representative of trees in large blocks of the orchard they were collected from, on that particular day, so that they can used to populate the harvesting simulator.Objective 2. Harvest simulationThe goal is to develop an open source tree-fruit harvesting simulator that calculates fruit picking productivity (FPE, PFT, FPS). This simulator should calculate machine kinematics and dynamics, and detect obstacle collisions. A software module will enable the generation of orchards with trees and fruits using models from the library developed in Objective 1.Objective 3. Experimental evaluationThis objective aims at evaluating the utility of the developed tool for design by simulating and executing the harvesting procedure using a two-arm robot on a mobile platform, and by comparing the predicted and achieved productivities under controlled conditions.Objective 4. DisseminationA main objective of this project is to enable the creation of a community that will use the data and the design tool to develop and evaluate novel fruit harvesting machine concepts. Dissemination activities will target researchers, academics, and students, engineers from agricultural equipment manufacturers and robotics start-ups.
Project Methods
1 Orchard modelingA1. Collect fruit position and size data in orchards.A2. Digitize orchard trees to extract structural-geometrical data.Fruit positions and tree geometries will be digitized using a Power TRAK 360TM digitizer. A frame structure is being designed to hold six digitizer sources that collectively establish a large-area magnetic field around a tree.A3. Develop and validate fruit spatial distribution models.We shall use the fruit coordinates to estimate the nonparametric spatial and size probability density functions (s-pdf) of fruits in canopies. The estimated fruit distribution of a given orchard row will be validated by measuring fruit locations from test trees (in the same orchard block) that were not included in the calculation of the histogram and use a standard two-sample Kolmogorov-Smirnov test to check if the marginal distributions (heights, radii and angle) of the test tree-fruits come from the corresponding cdf's.A4. Develop and validate tree structural-geometrical models.We propose to use frustums as geometric primitives, and encode virtual trees as stochastic Markov chains that produce minimum-length sequences of frustums and their corresponding optimal parameters (e.g., position, rotation, size), which give the best match with measured trees. It is expected that the pdf's of same-order branches will be the same, thus reducing further the number of parameters to be modeled. Possibly, different Markov models can be used for different branch orders. The choice of Markov models, their structures and inter-coupling are open research questions; overall feasibility is given since similar approaches have been used in the literature. The validation procedure will be identical to the one described in activity (A3).A5. Develop integrated tree and fruit models.The tree geometries and the fruit locations will be integrated. A possible approach to accomplish this is to generate the tree first, and then set rules for accepting and rejecting fruit locations as they are sampled from the s-pdf. Another alternative is to shift the fruit location in a direction that makes it a feasible fruit location under the existing rules. Of course, such actions change the effective spatial distribution of the fruits. The two-sample Kolmogorov-Smirnov (K-S) test between the original s-pdf and the altered distribution (after rejecting and resampling) will be used as validation for this procedure.2 Harvest simulationA6. Customize robotics simulator.The Gazebo open source robotics simulator will be used. Additionally, the 'MoveIt!' open source motion-planning library will be used to develop a fruit reaching and branch-avoidance motion-planning module, and a module for computing near-optimal fruit picking itineraries under the assumption that the harvester proceeds along an orchard row.A7. Develop virtual orchard module.This software module creates a virtual orchard in the simulator workspace. Essentially it accepts as inputs the orchard layout (tree spacing), the number of trees, and the fruit tree variety, training and age. Then it uses an appropriate stochastic model (from Activity A5) to place instances of the model on predefined locations in the orchard. All tree geometries and fruits are converted to the Simulator Description Format (SDF), which is an XML file format used to describe all the elements in a Gazebo simulation environment.3 Experimental validationA8. Develop robotic platform.A commercial scissor lift will be placed on a drive-by-wire utility vehicle and later on a harvest-aid platform. A Baxter robot will be placed on the lift and its arms will provide two picking actuators. Simplified CAD drawings for the vehicle and lift will be created, kinematics will be computed, and all information will be converted to Gazebo SDF format. The unknown vehicle and lift dynamics will be identified using the Matlab Identification Toolbox. Since Baxter is running the Robot Operating System (ROS), the same algorithms that control virtual Baxter in Gazebo will be used for the real robot. A ROS layer will need to be implemented to control the Workman vehicle and the lift. The minimization of RMS positional accuracy when the robot arm tooltip is commanded to reach georeferenced locations (x, y, z) will be used as an evaluation criterion to calibrate the harvester model. A9. Compare predicted and achieved productivities.The model of the robotic platform will be programmed to reach a set of points in open space that correspond to fruit positions from a fruit distribution without tree branches. Then the actual robot will be asked to reach the same points in real execution. The same experiment will be repeated with virtual branches incorporated into the actual robot's motion planner and the productivities will be compared. The minimum, maximum, average and RMS errors between predicted and achieved FPE and FPT will be used as evaluation criteria.4 DisseminationA10. Open-source digital contentAll data files, software and documentation will be made available under GNU General Public License, version 3.0 (GPLv3) on a website that will be created for this purpose. Also, permanent Digital Object Identifiers will be assigned to the digital content through the UC California Digital Library EZID service.A11. Mechanization stakeholdersResults from this work will be presented at scientific conferences and published in peer-reviewed journals. Webinars will also be organized and advertised through the IEEE Technical Committee on Agricultural Robotics (PI is a member). Results and the availability of the data, simulator and documentation will be communicated to Commodity Boards, interested parties in agricultural machinery manufacturing companies, robotics start-ups.Product evaluationP1: Files containing: orchard reference location (UTM), tree ID and location, raw fruit location data for each tree, and data acquisition metadata (from Activity A1).P2: The product will be evaluated by the number of downloads and visits.P3: The product will be evaluated by the number of downloads and visits.P4: The product will be evaluated by the number of downloads and visits.P5: The product will be evaluated by the number of downloads and visits.P6: The product will be evaluated by the number of downloads and visits.P7: The product will be evaluated by the number of downloads and visits.P9: The product will be evaluated by the number of downloads and visits.P10: The product will be evaluated by the number of website visits.P11: The product will be evaluated by the number of papers and presentations.P12: The product will be evaluated by the number of graduate, undergraduate and high-school students trained.EffortsGraduate and undergraduate students will acquire theoretical knowledge and practical skills on tree architectures, spatial modeling and robotic and automation technologies for machine-aided tree-fruit harvesting through thesis research, senior design projects, research internships and attendance of the EBS289K graduate course "Topics in Agricultural Robotics".Researchers (US and international) will gain a better understanding of the effects of tree architecture on robotic harvesting efficiency and throughput through journal publications, presentations in conferences and access to the project's website and the data and code. Platform manufacturers will also have access to this information through the website, targeted correspondence and attendance of field day trials.Growers will gain better understanding of: the spatial variability of fruits in commercial orchard trees; the effects of tree architecture on robotic harvesting efficiency and throughput. These will be achieved through presentations at growers meetings and field days at selected orchards as well as specialized publications, like the "Good Fruit Grower" that reach growers at a national level.

Progress 01/01/16 to 12/31/19

Outputs
Target Audience:One Ph.D. student was mentored in the context of this project. Informal laboratory instruction was the key instrument. Our research team worked closely with growers in Modesto, Stockton, and Lodi CA and conducted field experiments in their fields. The PI and Ph.D. student gave several presentations related to tree modeling and mechanized harvesting. The audiences included growers, entrepreneurs, researchers, students, academic staff and the general public. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided?One Ph.D. student, one M.Sc. student, two undergraduate students, and one post-doctoral researcher were mentored in the context of this project. Informal laboratory instruction was the key instrument. The post-doctoral researcher got a faculty position in Brazil. The Ph.D. student finished his Ph.D. in Fall 2019 and is continuing work on agricultural automation at UC Davis, as a post-doctoral researcher. The M.Sc. student is working at a local robotics start-up company that is developing a mushroom harvesting robot. How have the results been disseminated to communities of interest?The PI gave several presentations related to the project and to mechanical harvesting in general. The audiences were growers, entrepreneurs, researchers, students, academic staff and the general public. January 30, 2019. Cling Peach Board, Sacramento, CA. February 5, 2019. California Pear Board, Davis, CA. February 13, 2019. Morning Star Company, Davis, CA. March 19, 2019. World Bank Headquarters, Washington, D.C. April 1, 2019. Western Center for Agricultural Health and Safety, Davis, CA. April 3, 2019. California Strawberry Commission, Cal Poly at Saint Luis Obispo. August 27, 2019. Taylor Farms, Salinas, CA. What do you plan to do during the next reporting period to accomplish the goals? Nothing Reported

Impacts
What was accomplished under these goals? In the context of this project, a set of trees and their fruits were digitized and software was developed to simulate two main mechanical fruit harvesting methods: shake-catch harvesting, and robotic harvesting. Progress in mechanical harvesting is crucial for national food security, especially given the farm labor shortages we are facing. Regarding shake-catch harvesting, simulations indicated that it is possible to harvest fruits by using novel multi-level fruit catching systems. As a result, an SCRI proposal was developed and submitted to develop such systems. Regarding robotic harvesting, simulations showed that novel multi-arm robots are needed to harvest fruits efficiently. The results from this project led to a successful proposal under the NSF National Robotics Initiative - funded by NIFA - to develop a physical multi-arm robot for fruit harvesting. The project's major accomplishments are listed next. A large data set of tree geometries and attached fruits was created. Twenty V-shaped clingstone peach trees were digitized at Escalon, CA and twenty high-density Bartlett pear trees were digitized at Ukiah, CA. Matlab code was written to process, filter data and calculate fruit spatial distributions and tree branching parameters. Spatial accuracy was experimentally verified to be better than 1 cm. Also, code was written to isolate parts (branches with fruits) of the existing tree models and combine them together in order to create new instances of trees that were used to create larger virtual orchards. The data were uploaded onto an open data repository (https://datadryad.org) with permanent DOI: https://doi.org/10.25338/B8WW41. Software was developed to compute fruit motions (falling, bouncing, rolling) during their interaction with machine surfaces, and layers of parallel tubes were pushed horizontally inside ten pear and ten cling peach digitized trees. Each fruit fell vertically and the fruit catching efficiency was calculated as the percentage of dropped fruits that landed on a tube or the ground (where they are collected without damage) without colliding with a branch. Results showed that without tubes, 36% of pears and 21% of peaches hit a branch, whereas with four layers these numbers dropped to 12% and 8% respectively. So, it is possible to harvest fruits by using novel multi-level fruit catching systems. As a result, an SCRI proposal was developed and submitted to develop such systems. Also, software was written to evaluate the reachability of fruits when linear robot arms are used to pick, rather than arms with revolute joints. This is referred to as Linear Fruit Reachability (LFR) and is essentially the Fruit Picking Efficiency (FPE) metric calculated for linear arms. Simulations were performed with different arm parameters (maximum acceleration, travel speed, arm extension length, gripper size). The simulation results showed that about 81% of pears and 87% of peaches were reachable. This implies that for some trees of SNAP-type architectures fruit reachability may not require complex and expensive arms with many degrees of freedom. Also, software was written to assess the Fruit Picking throughput (FPT) for linear arms harvesting in virtual orchards. The simulator modeled the kinematics and dynamics of a harvester comprising identical, independent, non-interfering arms. Each arm moved inside a 'cell' in an array configuration and picked fruits only from the canopy volume allocated to it, in a sequence that minimized arm travel time. Overall, a ten-fold decrease in PCT (from 4.6 s to 0.46 s) was only possible with optimally sized cells and at least 14 arms. These results indicate that achieving low PCTs for cost-effective fruit tree harvesting requires many arms that must be carefully load-balanced in the presence of changing, partially known fruit distributions. Based on these results, a proposal was funded by NIFA under the NSF National Robotics Initiative to develop a physical multi-arm robot for fruit harvesting. A two-arm robot (Baxter) was programmed to detect fruit using a camera and reach fruit using its arms and suction grippers. In lab settings, picking times of 4 s per fruit were recorded, in good correspondence with simulated fruit picking using a model of the robot. Major dissemination activities included conference presentations, journal publications, presentations at grower meetings, Commodity Boards and industry, demonstrations to the public during UC Davis's picnic day, and the release of publicly open data for the digitized trees and fruits: https://doi.org/10.25338/B8WW41.

Publications

  • Type: Journal Articles Status: Published Year Published: 2019 Citation: Vougioukas, S.G. (2019). Agricultural Robotics. Annual Review of Control, Robotics, and Autonomous Systems. 2:365-392. https://doi.org/10.1146/annurev-control-053018-023617
  • Type: Journal Articles Status: Published Year Published: 2019 Citation: Charlton, D., Edward Taylor, J.E., Vougioukas, S.G., Rutledge, Z. (2019). Innovations for a Shrinking Agricultural Workforce. Choices, 2nd Quarter 34(2). http://www.choicesmagazine.org/choices-magazine/submitted-articles/estimating-value-damages-and-remedies-when-farm-data-are-misappropriated/innovations-for-a-shrinking-agricultural-workforce
  • Type: Journal Articles Status: Published Year Published: 2019 Citation: Charlton, D., Edward Taylor, J.E., Vougioukas, S.G., Rutledge, Z. (2019). Can Wages Rise Quickly Enough to Keep Workers in the Fields? Choices, 2nd Quarter 34(2). http://www.choicesmagazine.org/choices-magazine/submitted-articles/can-wages-rise-quickly-enough-to-keep-workers-in-the-fields


Progress 01/01/18 to 12/31/18

Outputs
Target Audience:Students: One Ph.D. student and one post-doctoral researcher were mentored in the context of this project. Informal laboratory instruction was the key instrument. Growers: PI Vougioukas presented results from this project in grower and industry stakeholder meetings such as: Cling Peach Day on January 4, 2018; Lake County North Coast Pear Growers meeting on February 7, 2018; Sacramento River District Growers meeting, February 6, 2018; Tree Fruit Mechanization Day, Walnut Grove and Linden, CA, June 26, 2018. General public: UC Davis picnic Day, April 21, 2018; Innovation Summit, UC Davis, April 23, 2018. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided?Students: One Ph.D. student and one post-doctoral researcher were mentored in the context of this project. Informal laboratory instruction was the key instrument. The post-doctoral researcher got a faculty position in Brazil. The Ph.D. student will submit his dissertation in May 2019. How have the results been disseminated to communities of interest?Growers: PI Vougioukas presented results from this project in grower and industry stakeholder meetings such as: Cling Peach Day on January 4, 2018; Lake County North Coast Pear Growers meeting on February 7, 2018; Sacramento River District Growers meeting, February 6, 2018; Tree Fruit Mechanization Day, Walnut Grove and Linden, CA, June 26, 2018. General public: A demo of a fruit grasping robot was set up during UC Davis picnic day on April 21, 2018, and hundreds of visitors - including young children - interacted with it and learned about robotic fruit harvesting research. The PI presented at the UC Davis Innovation Summit, April 23, 2018, to an audience of entrepreneurs, investors, growers, students, faculty and general public. What do you plan to do during the next reporting period to accomplish the goals?In the summer of 2018, we could not digitize trees in Lake County orchards in late August-early September because of wildfires in the area. With the no-cost extension provided to us, we'll be able to gather some more data in the next picking season. We'll make the data available in the UC Davis open-access repository. We will also finalize the harvest simulation analysis and auto-guidance experiments and complete the manuscripts that report our findings.

Impacts
What was accomplished under these goals? Accomplishments Objective 1. Orchard modeling Data was not gathered during this reporting period because of wildfires in CA. However, the existing orchard models had a large number of trees (40) that enabled performing harvesting simulations (Objective 2). More data will be gathered in the summer of 2019. Objective 2. Harvest simulation In the previous report period, software was written to evaluate the reachability of fruits when linear arms - rather than arms with revolute joints - are used to pick fruits. This is referred to as Linear Fruit Reachability (LFR) and corresponds to the Fruit Picking Efficiency (FPE) metric calculated for linear arms. In this reporting period, our work extended the concept of LFR by introducing physical constraints for the harvesting actuators, such as maximum reach, gripper cross-section, minimum distance constraint of the machine from the canopy due to branches extending in the row, and possible directions of approach given the arm length. High density trellised pear and v-trellised cling peaches were the training systems chosen for the current research. Arm lengths of 100 to 200 cm with an increment of 25 cm for pears and 200 to 300 cm with an increment of 25 cm for peaches, and gripper sizes of 8 to 12 cm for pears and 7 to 11 cm for peaches were tested. Two gripper designs were considered. 'Adaptive' grippers, whose cross-section can increase (by opening) from a minimum size to the size of the fruit to be harvested, and fixed-size grippers (e.g., vacuum ones) whose cross-section must be larger than the largest fruits to be picked. One significant result was that adaptive grippers picked 7% more fruit compared to vacuum grippers. Another main finding was that the cumulative LFR after three passes was estimated to be 93.5% for pears and 94.2% for peaches. This means that linear arms are a practical way of harvesting these fruit trees. In this reporting period, the fruit picking cycle times (PCTs) of linear multi-arm harvester configurations were also investigated for all reachable fruits under static harvesting scenario. A methodology was developed to estimate the fruit PCT of linear multi-arm harvesters using models of digitized trees. Simulation analyses showed the dependence of single-arm PCT on maximum arm acceleration and maximum velocity. A range of accelerations and velocities were tested for pear and peach trees, and results showed that PCT dependence on maximum acceleration and maximum speed is described by a power law. The dependence of linear arm array PCT on the number of arms was investigated, and results indicated that PCT follows a decaying power law as the number of arms increases for both peaches and pears. Hence increasing the number of arms provides diminishing returns in terms of per-arm. The PCTs of linear arm arrays were estimated under two different scenarios for the size and shape of each arm's work cell: a) the work cells of all arms were equal in size (square), and b) the work cell of each arm had the same amount of fruit in it, resulting in rectangular but unequal work cells. Results indicated that harvesting pear and peach fruits using arms allocated under the same fruit load work cells is 25% to 50% faster than harvesting with arms operating in equal-sized work cells. Finally, the dependence of PCT on three different harvest scenarios were tested: harvesting both sides of a row, straddling the trees and harvesting both sides of each tree, and independent harvesting of each side of a row. PCTs achieved for the 2nd harvesting technique are higher than the PCTs achieved by 1st and 3rd techniques for the case of equal fruit load allocation to arms. The PCTs achieved for the 2nd harvesting technique are lower than the PCTs achieved by 1st and 3rd techniques. Also, the PCTs were low for 1st technique compared to the 3rd technique initially as the arms increased the 3rd technique has lower PCTs than the 1st technique for the case of equal cell size allocation to arms. Simulations results on PCT of the multi-arm harvester harvesting one side with an unequal number of arms with equal cell size allocation indicates that for a given number of arms several combinations of arms mounted on each side attain similar or better PCT's. Objective 3. Experimental evaluation During this reporting period, our efforts focused on the autonomy of the vehicle that will be used, and in particular on software for navigation in orchards using a single 3D camera, without continuous GPS signal availability. The approach developed is innovative in that it uses the 3D point cloud at places in the orchard where the presence of landmarks provides ground truth for the robot pose, to develop automatically a sensor model for the vehicle to use in a particle filtering approach. Results in simulation and orchard experiments are excellent (cm-level accuracy in lateral offset and less than 5o yaw error) and a manuscript is under preparation. Objective 4. Dissemination Our dissemination activities are described in the section "How have the results been disseminated to communities of interest?"

Publications

  • Type: Conference Papers and Presentations Status: Published Year Published: 2018 Citation: Arikapudi, R., Vougioukas, S. (2018). A Study on Pick-Cycle-Times of Robotic Multi-Arm Tree Fruit Harvesters. Intl. Conference on Agricultural Engineering (AgEng 2018)
  • Type: Journal Articles Status: Accepted Year Published: 2019 Citation: Vougioukas, S.G. (2019). Agricultural Robotics. Annual Review of Control, Robotics, and Autonomous Systems. (2):xx-xx. https://doi.org/10.1146/annurev-control-053018-023617
  • Type: Journal Articles Status: Published Year Published: 2018 Citation: Durand-Petiteville, A., Le Flecher, E., Cadenat, V., Sentenac, T., Vougioukas, S.G. (2018). Tree detection with low-cost 3D sensors for autonomous navigation in orchards. IEEE Robotics and Automation Letters. 3(4): 3876-3883. https://doi.org/10.1109/LRA.2018.2857005


Progress 01/01/17 to 12/31/17

Outputs
Target Audience:Students: One Ph.D. student and two undergraduate students were mentored in the context of this project. Informal laboratory instruction was the key instrument. The Ph.D student also presented at the 2017 ASABE Inlt. meeting at Spokane, WA. Growers: Our team visited one grower in Escalon, CA and another one in Ukiah, CA and informed them about the project and also performed a lot of digitization work in their orchards. Growers were also reached at annual grower meetings (see section on dissemination). General public: Precision Ag conference series, organized by the World Food Center at UC Davis, April 27, 2017; CITRIS Agricultural Technology Fair, UC Merced, March 8, 2017; UC Davis picknic Day, April 22, 2017. Researchers: The international researcher community was reached vat the ASABE 2017 international meeting at Spokane, WA. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided?One Ph.D., two undergraduate students and one staff member were mentored and trained in tree and fruit digitization and geometric modeling. Training was performed via collaboration, mentoring and supervision in the lab and in field experiments. The Ph.D. student, Mr. Arikapudi, attended and presented his work at the 2017 ASABE Intl. meeting, at Spokane, WA. How have the results been disseminated to communities of interest?The PI presented results from this project in grower and industry stakeholder meetings such as: sweet cherries grower meeting on Jan. 5, 2017 at Cabral Agricultural Center, Stockton, CA; Washington Tree Fruit Commission, at UC Davis, March 1, 2017; Almond Board of California, Harvest Technology Roundtable at UC Davis, March 15, 2017; Silicon Valley Forum at UC Davis, 4 April 4, 2017; Interpera conference, June 15, 2017; A demo of a fruit grasping robot was set up during UC Davis picknic day on April 22, 2017 and hundreds of visitors - including young children - interacted with it and learned about robotic fruit harvesting research. Presentations were also made to wide audiences (growers, students, academics, industry) at the Precision Ag conference series, organized by the World Food Center at UC Davis, April 27, 2017; at the CITRIS Agricultural Technology Fair, UC Merced, March 8, 2017. What do you plan to do during the next reporting period to accomplish the goals? Nothing Reported

Impacts
What was accomplished under these goals? Objective 1. Orchard modeling A new, large data set of tree geometries and attached fruits was created in summer 2017. Twenty V-shaped clingstone peach trees were digitized at Escalon, CA and twenty high-density Bartlett pear trees were digitized at Ukiah, CA. Matlab code was written to process, filter data and calculate fruit spatial distributions and tree branching parameters. Also, code was written to isolate parts (branches with fruits) of the existing tree models and combine them together in order to create new instances of trees that were used to create larger virtual orchards. Objective 2. Harvest simulation Software was written to evaluate the reachability of fruits when linear arms are used to pick, rather than arms with revolute joints. This is referred to as Linear Fruit Reachability (LFR) and is essentially the Fruit Picking Efficiency (FPE) metric calculated for linear arms. Simulations were performed with different arm parameters (maximum acceleration, travel speed, arm extension length, gripper size). The simulation results based on the digitized geometric tree models and fruit locations showed that a highest percentile of the fruit i.e., about 81% were reachable using an arm extension of 150 cm and the gripper size being 13 cm for pears. For peaches 87% fruits were reachable using an arm extension of 250 cm and the gripper size of 12 cm. This implies that for some trees of SNAP-type architectures fruit reachability may not require complex and expensive arms with many degrees of freedom. Of course, thin, flexible branches were not included in this study and their effects would need to be evaluated. The LFR's for 1st pass was 78%, 2nd pass was 2% and 1% for the 3rd pass respectively considering the gripper size of 13 cm and an arm extension of 150 cm. The LFR's for 1st pass was 80%, 2nd pass was 5% and 2% for the 3rd pass respectively considering the gripper size of 12 cm and an arm extension of 250 cm for peaches. This knowledge about the percentile fruit reachability for individual passes was used to determine the type of robot structure that could be used to harvest the fruit on these training systems. This helps in making economic decisions in the harvester design as to see the harvest cost per fruit for individual passes. Also, software was written to assess the Fruit Picking throughput (FPT) for linear arms harvesting in virtual orchards. The simulator modeled the kinematics and dynamics of a harvester comprising identical, independent, non-interfering arms. Each arm moved inside a 'cell' in an array configuration and picked fruits only from the canopy volume allocated to it, in a sequence that minimized arm travel time. Each tree was harvested independently, while the harvester was static in front of it. Digitized models of fruit-bearing, high-density trellised pear trees were used. Individual fruit distributions were non-uniform and varied significantly among trees. Such yield variation can introduce significant inefficiencies in multi-arm robot harvesting. Two scenarios were considered: A) all arm cells had the same size; B) cells were unequal and optimally sized to contain approximately the same number of fruits. Scenario (B) serves as an 'upper bound' on performance, since all arms harvest the same load and finish almost together. Results showed that in both scenarios Picking Cycle Time (PCT), which is the inverse of FPT, followed a power law; it did not drop linearly as the number of arms increased. Overall, the PCTs of scenario A were three times worse than the corresponding PCTs of scenario B; this is the result of non-uniform fruit distribution. A ten-fold decrease in PCT (from 4.6 s to 0.46 s) was only possible with optimally sized cells and at least 14 arms. Achieving such low PCT with uniform cells required more than 40 arms. These results indicate that achieving low PCTs for cost-effective fruit tree harvesting requires many arms that must be carefully load-balanced in the presence of changing, partially known fruit distributions. Objective 3. Experimental evaluation During this reporting period our efforts focused on the autonomy of the mobile vehicle that will be used, and in particular on software for navigation in orchards using 3D cameras, without continuous GPS signal availability. A low cost, efficient vision-based system was developed to detect accurately and robustly the trees in orchard rows so that the line to be followed can be extracted. It is made of four stereo cameras which provide a point cloud characterizing the environment. The key idea is to find the tree trunks by detecting their shadows which are materialized by concavities in the obtained point cloud. In this way, branches and leaves are not taken into account, improving the detection robustness which is an important feature to obtain an efficient navigation. A complete, real-time point cloud processing pipeline was developed and implemented using ROS and validated using data sequences taken in our orchards. A manuscript is under preparation. Objective 4. Dissemination Our dissemination activities are described in the section "How have the results been disseminated to communities of interest?"

Publications

  • Type: Conference Papers and Presentations Status: Published Year Published: 2017 Citation: Arikapudi R., Vougioukas, S.G., (2017). Linear Reachability of Fruits in Orchard Trees with Actuator Constraints. ASABE Annual International Meeting. Paper Number 1701421, Spokane, Washington


Progress 01/01/16 to 12/31/16

Outputs
Target Audience:Students: The PI mentored one Ph.D. student, one MSc. student, one post-doctoral researcher and two undergraduate students (internships) in the context of this project. Informal laboratory instruction was the key instrument. Also, undergraduate and graduate students were reached via seminars: Mechanical and Aerospace Dept., UC Davis (4.28.16); UC Merced (4.22.16), and UC Davis Biological and Agricultural Eng. Dept. (6.9.16). Researchers: The international researcher community was reached via one journal publication in one of the best journals in our field (Biosystems Engineering) and in the context of two international conferences: AgriControl, in Seattle, USA, and CIGR-Eurageng, in Aarhus, Denmark. Growers: On 5.24.2016 the PI presented results from cling-peach digitization to growers from the Canning Peach Mechanization Research Fund. Co-PI Elkins presented at the Hood River Winter Horticulture Meeting, Oregon, USA, (2.5.16). Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided?One Ph.D. student (Rajkishan Arikapudi) was mentored and trained in tree and fruit digitization and in modeling. One post-doctoral researcher (Adrien Durand Petitvile) was mentored and trained in autonomous vehicle navigation control and robot control. One MSc (Mathew Lefort) student was mentored and trained in lidar-based path-tracking control for autonomous vehicles. Two undergraduate students (Joshua Munic, Emile Le Flecher) were mentored and trained in simulation development and in robot programming. How have the results been disseminated to communities of interest?The international researcher community was reached via one journal publication in one of the best journals in our field (Biosystems Engineering) and in the context of two international conferences: AgriControl, in Seattle, USA, and CIGR-Eurageng, in Aarhus, Denmark. On 5.24.2016 the PI presented results from cling-peach digitization to growers from the Canning Peach Mechanization Research Fund. co_PI Elkins presented at the Hood River Winter Horticulture Meeting, Oregon, USA, (2.5.16). What do you plan to do during the next reporting period to accomplish the goals? Nothing Reported

Impacts
What was accomplished under these goals? The tree and fruit models generated during the first year of the project were used to explore - initialy in software - alternative ways to harvest mechanically fresh-market fruits using either novel shake-and-catch systems or novel robot designs. Preliminary results are very encouraging. Progress in harvest mechanization will offer significant relief to the labor shortage proglems that the Nation faces. Objective 1. Orchard modeling A1. Collect fruit position and size data in orchards. A2. Digitize orchard trees to extract structural-geometrical data. Activities A1 and A2 were conducted together in cling-peach (Martini Ranch, Echelon, CA) and pear (Ruddick Ranch, Ukiah) high-density orchards. Fruits positions and sizes were digitized by recording three points on each fruit. Also, the parent branch of each fruit was recorded. A3. Develop and validate fruit spatial distribution models. Matlab code was written to calculate fruit spatial distributions. Marginal one and two-dimensional distributions were calculated to characterize fruit density as functin of height, length along-row and combined. A4. Develop and validate tree structural-geometrical models. Matlab code was written to approximate tree geometries and structure. Branches were represented as sequences of frustums and tree structure was represented using an encoding scheme that captures tree branch topology. A5. Develop integrated tree and fruit models. Matlab code was developed to place the fruits on tree models developed in activity A4 and to visualize the fruits. Objective 2. Harvest simulation A6. Customize robotics simulator. The open-source Gazebo simulator was downloaded and installed and a model of the autonomous vehicle was created from scratch in Simulator Description Format (SDF) and imported to the simulator. The geometric and kinematic SDF model of the Baxter robot was also imported to Gazebo. Integration of vehicle and robot started but is under development. A7. Develop virtual orchard module. Geometric models of the digitized trees and the fruits were created in SDF format and imported into the Gazebo simulator. Objective 3. Experimental evaluation A8. Develop robotic platform. A ROS layer was developed to control the autonomous vehicle; its kinematics were identified. Autonomous navigation software is being developed using a commercial stereo camera. Software was written to communicate with the Baxter robot and perform basic control of its arms and read images from its camera. Closed-loop camera-based fruit reaching was programmed. A9. Compare predicted and achieved productivities No activity. Objective 4. Dissemination A10. Open-source digital content No activity. This will start when datasets have been completed. A11. Mechanization stakeholders Results were disseminated to growers and researchers, as described in the dissemination section (next).

Publications

  • Type: Journal Articles Status: Published Year Published: 2016 Citation: Arikapudi, R., Vougioukas, S.G., Jim�nez- Jim�nez, F., Farangis Khosro Anjom, F. (2016). Estimation of Fruit Locations in Orchard Tree Canopies Using Radio Signal Ranging and Trilateration. Computers and Electronics in Agriculture (125):160-172. http://dx.doi.org/10.1016/j.compag.2016.05.004
  • Type: Conference Papers and Presentations Status: Published Year Published: 2016 Citation: Vougioukas, S., Arikapudi R., Munic, J. (2016). A Study of Fruit Reachability in Orchard Trees by Linear-Only Motion. 5th Agricontrol Conference, Seattle, USA. IFAC-PapersOnLine, 49(16), pp.277-280
  • Type: Conference Papers and Presentations Status: Published Year Published: 2016 Citation: Vougioukas, S., Arikapudi R., Munic, J. (2016). Upper Bound Estimates of Fruit Reachability in Orchard Trees using Linear Motion. Proceedings of the Intl. Conference on Agricultural Engineering (AgEng  CIGR 2016), Paper 529, Aarhus, Denmark.
  • Type: Conference Papers and Presentations Status: Published Year Published: 2016 Citation: Munic, J.P., Vougioukas, S.G., Arikapudi, R. (2016). A Study on Intercepting Falling Fruits with Canopy Penetrating Rods. 2016 ASABE Annual International Meeting. Paper Number 162456923, Orlando, Florida