AUTOMATION OF DORMANT PRUNING OF SPECIALTY CROPS

• Sponsoring Institution: National Institute of Food and Agriculture
• Project Status: COMPLETE
• Funding Source: OTHER GRANTS
• Reporting Frequency: Annual
• Accession No.: 0230318
• Grant No.: 2012-51181-19878
• Cumulative Award Amt.: $3,027,747.00
• Proposal No.: 2012-01537
• Project Start Date: Sep 1, 2012
• Project End Date: Aug 31, 2017
• Grant Year: 2012
• Program Code: [SCRI]- Specialty Crop Research Initiative

Recipient Organization
PURDUE UNIVERSITY
(N/A)
WEST LAFAYETTE,IN 47907

Performing Department
Horticulture and Landscape Architecture

Non Technical Summary
To maintain its viability in the highly competitive global market, the U.S. specialty crop industry must reduce the need for a large seasonal workforce by developing more mechanized and automated techniques. This is not only due to the expense of seasonal labor but also the uncertainty surrounding the availability of labor. This is especially true for labor-intensive crops such as deciduous fruits, for which pruning is among the most labor-intensive practices. Using a systems approach, this Standard Research and Extension Project will develop innovative technologies for automating the dormant pruning of grape and apple, by utilizing modern sensors, computers, and robotic manipulators. We will conduct socio-economic assessments in commercial pilot orchards and vineyards in Pennsylvania, California, Washington and Indiana. We will also develop outreach tools and extension activities to communicate results and applications directly to growers and industry personnel and will involve students in both research and extension activities. This trans-disciplinary project will result in reduced labor requirements for dormant pruning and more precise management of tree architecture for improved adaptation to future technologies such as automated harvesting. The proposed robotic pruner will enhance the long-term profitability and sustainability of fruit production through improvements in production economy, quality, consistency and dependability.

Animal Health Component

50%

Research Effort Categories

Basic

(N/A)

Applied

50%

Developmental

50%

Classification

Knowledge Area (KA)	Subject of Investigation (SOI)	Field of Science (FOS)	Percent
205	1110	1060	10%
205	1110	2020	10%
205	1131	1060	10%
205	1131	2020	10%
404	1110	1060	10%
404	1110	2020	10%
404	1131	1060	10%
404	1131	2020	10%
601	1110	1060	10%
601	1131	1060	10%

Knowledge Area
601 - Economics of Agricultural Production and Farm Management; 205 - Plant Management Systems; 404 - Instrumentation and Control Systems;

Subject Of Investigation
1110 - Apple; 1131 - Wine grapes;

Field Of Science
2020 - Engineering; 1060 - Biology (whole systems);

Keywords

apple

grape

grapevine

labor efficiency

tree canopy

3d imaging

decision system

social impact

economic impact

pruning rules

canopy structure

Goals / Objectives
Goals. 1. Formulate and evaluate rules that describe optimal pruning in terms of measurable physical attributes of canopy structure. 2. Develop 3D imaging, decision system, and robot control technologies for automating dormant pruning operations in order to construct autonomous pruning systems. 3. Determine social and economic impacts of the proposed autonomous pruning system. 4. Communicate results and involve growers, industry groups, academia and students so they can adopt these technologies and incorporate the knowledge gained into their orchards, vineyards, businesses, classrooms and laboratories. Outputs. 1. Collect sensed data from apple and grape canopies to ensure that pruning cuts result in favorable outcomes 2. Formulate crop-specific pruning rules to determine which specific branches to remove 3. Use decision system to form action steps that result in appropriate pruning cuts 4. Determine social impacts of autonomous pruning systems

Project Methods
Methods for each of the goals articulated above: 1. Formulate standardized crop-dependent pruning rules to ensure that pruning cuts result in favorable outcomes (since improper pruning rules will result in negative outcomes). For each crop, one representative canopy training system was chosen - the bilateral cordon system for grapes and the tall spindle for apples. We will develop and refine pruning rules that describe optimal pruning of grape and apple. We will implement these rules manually and evaluate how the rules encapsulate current best pruning practices. We will use these evaluations to refine the pruning rules, leading to another iteration of implementation and evaluation of the rules in the following year. 2. 3D sensing and image analysis will be used to determine which specific branches to remove to obtain optimal canopy dimensions. We will create a decision system for the application of the pruning rules to the sensed data to form action steps that result in appropriate pruning cuts. Once we identify branch junction points, the rest of the reasoning is driven by placing coordinate frames at each such junction point to identify the limbs emanating from the junction point. We will develop a graphical user interface (GUI) to allow a user to visualize the 3D reconstructions of trees and vines as well as their physical attributes (e.g., the diameter of a branch, the vertical angle of a shoot, total light intercepted value, light distribution information, etc.) and the final calculated pruning points determined by the decision system. The ability to visualize the 3D reconstructions and the relevant physical attributes will play a critical role in verifying the accuracy of our 3D sensing and reconstruction algorithms. During this project, we will bring a proof-of-concept autonomous grape pruner to a production level. 3. To determine social and economic impacts of the proposed autonomous pruning system we will develop a technology commercialization roadmap, perform 15-20 case studies of apple and grape growers and analyze 800 completed grower surveys. The technology commercialization roadmap will quantify the socio-economic impact and adoption feasibility of the proposed autonomous pruning system by developing AgProfitTM economic models, assessing technology adoption feasibility by determining the Grower's Financial Indifference Value (gFIV) and using AgFinanceTM and key financial ratios to determine minimum farm productivity to adoption, and by conducting an industry survey of technology adoption. 4. We will develop outreach materials in the form of multi-media presentations with facilities for direct and immediate user feedback. We will establish on-farm applied research trials demonstrating new labor savings and environmental stewardship in orchards and vineyards in participating states. The on-farm plantings will also serve as field laboratories for student engineering interns. Commercial pilot studies allow unique opportunities for grower and community input and successful technology transfer. To increase technology transfer in non-participating states, we will conduct train-the-trainer programs for extension professionals.

Progress 09/01/12 to 08/31/13

Outputs
Target Audience: The target audiece reached directly during this reporting period (09/12-08/13) includes apple growers in Pennsylvania, attendees of the International Fruit Tree Association Summer Tour, winegrape growers in California, and attendees at the American Society for Horticultural Science annual meeting in Palm Desert, CA. Additionally, the broader fruit growing community was reached via publications in research (1), extension (3) and trade industry (11) journals and those viewing the project website (www.pruningautomation.org). Changes/Problems: Due to a change in employment, Dr. Clark Seavert left the project. To accomplish the economic analyses in this work, Dr. Jayson Harper, economist at Pennsylvania State University, has recently joined the project. Due to this change in personnel, the development of the technology commercialization roadmap has been delayed. Additionally, a graduate student on the project (rural sociology) left Pennsylvania State University’s graduate program. This delayed some of the interviews that were intended to be completed during the fall of 2013. To make up for the lost time, one graduate student was added for the 2013-2014 academic year. Another was added for the spring semester of 2014. The Purdue engineering team had several issues finding graduate students to work on the project. The two students that were working on the project initially decided to leave after less than six months. One of them graduated and the other had health issues. We found two additional students to replace them, but one of them decided to move to a different graduate program after spending several weeks learning the technology. We are now in the process of training a new student to take the lead on the design of the algorithms. There have been usual engineering technical issues, but our team is addressing these and working through the challenges. Due to the development status of the automated pruner in 2013, the pruning treatments and vine measurements were not conducted in the commercial vineyard in Washington. Instead, simulations were conducted. What opportunities for training and professional development has the project provided? Engineer teammates were educated about apple tree pruning, the severity and types of cuts that need to be made. This will enhance the outcomes of their efforts to develop appropriate sensor and pruning technology. The project allowed 5 graduate students and 2 undergraduate student volunteers to become familiar with the methods and tools used in the construction of models using laser scanners and processing 3D data using range images and point clouds. A grad student from Purdue completed a week-long training at VRC in December 2013 and brought the complete software package back to Purdue.This should foster closer collaboration between thetechnical groups.It should also give Purdue a head start onparts of the apple pruning system and enables them to offer concrete system enhancements for the pruner. Three graduate students were trained in horticulture and the integration of pomology and engineering considerations. One student was trained in the development and application of survey instruments to measure grower attitudes regarding the adoption of new technological methods in their orchards and vineyards. one post-doc scholar gained experience using a Laser Measurement System and using gathered point clouds to develop tree skeletons. How have the results been disseminated to communities of interest? Members of our team are well integrated with cooperative extension at Pennsylvania State University and Purdue University. As such, our team has been very active in communicating the goals, activities, results and implications to growers across the United States. The project has been described in articles in some of the major industry publications such as The Good Fruit Grower (2102) and The Fruit Growers News (2013). Presented preliminary results of pruning rule experiments to Mid-Atlantic fruit growers and participants in the International Fruit Tree Summer Tour at Penn State Fruit Research and Extension Center Grower Field Day, July 17, 2013. Field day demonstration of robotic grape pruner to commercial vineyard managers and allied industry personnel, Lodi, CA. March 2013 Presented project overview at annual conference of the American Society for Horticultural Science. July 24, 2013. Project team meeting with advisory committee, Biglerville, PA. July 2013. A technology transfer workshop was conducted December, 2012, for 125 fruit industry participants. A technology transfer tour was conducted July 2013 for 325 growers from 25 states and seven countries. During these educational programs for specialty crop growers, the science of pruning based on preliminary horticultural investigations was presented along with pruning rules. Orchard employees attended the pruning workshop, which was presented in both Spanish and English. What do you plan to do during the next reporting period to accomplish the goals? Conduct a second season of the four experiments conducted in 2013 on pruning rules to further validate outcomes and increase knowledge about the cumulative effects of the pruning rules. Conduct one or more additional experiments to refine favorable outcomes of the pruning rules. Test the broad applicability of pruning rules across a wider range of grower farms and apple cultivars. Collaborate with project engineers to conduct imaging of apple trees and verify imaging with ground truth as regards the size and location of branches. In a research vineyard, three types of pruning will be conducted and the vines measured throughout the season for the effect of pruning on canopy and crop load. The automated pruning method will be simulated because of the geographic distance between the machine (California) and the research vineyard (Washington). The experimental protocols and measurement matrix developed for the research vineyard will be replicated in a commercial vineyard in California, with actual automated pruning as one of the three treatments. The vines will be evaluated for canopy and crop load metrics throughout the season. In Washington State, a non-selective precision mechanical pruner will be studied alongside manual pruning by "grower best practices." After the algorithms are robust enough to work on real data, we will start measuring the relevant parameters of the apple trees. In the discussions with the horticulturist ream, we decided that the initial parameters of interest are the diameters and angles of the primary branches with respect to the tree trunk. We will use the criteria provided by the horticulturists to automatically locate the pruning points. The accuracy of the automatic detection will be tested against the opinion of the experts (in view of the conclusions from the pruning severity studies carried out by the PSU team). A complete system that scans the trees, generates the models and locates the pruning points in real time will then be tested in the field. Interviews with wine-grape growers in California are being scheduled for March of 2014 Graduate students will conduct 10-15 interviews with apple growers at the New York fruit grower expo in Syracuse in January of 2014. A workshop on the art and science of pruning is planned for the Indiana Horticultural Congress in Indianapolis, IN in January 2014. This features work conducted within this project. Two abstracts describing work from the project have been accepted for oral presentations at the International Horticultural Congress in Australia in August 2014. A paper “Automated system to measure tree branches” will be submitted to the International Conference on Intelligent Robots and Systems, Chicago, IL, 2014. A field day to demonstrate the robotic grapevine pruner to growers and allied industry personnel is planned for March 11 this year. We will also use this opportunity for both project members and the advisory panel to meet.

Impacts
What was accomplished under these goals? Pruning of both apples and grapes, once thought to be a mysterious blend of art and science, has been described by a set of rules. These rules have been evaluated and compare favorably with human-pruned trees and vines. We are using these rules to develop automated decision systems, and to educate and train human pruners to prune their crops more effectively and efficiently. With grapevines, we have improved the integration of hardware to be able to interact with the vines to improve the speed and precision of the robotic pruner. We have developed software to better recreate 3D representations of tree canopies and determine canopy structure. These representations are necessary so that an automated system can decide on the optimal pruning points. We have developed survey instruments to determine grower attitudes towards technology and its adoption. Goal 1. Formulate and evaluate rules that describe optimal pruning in terms of measurable physical attributes of canopy structure. Activity 1-1: Formulate a set of rules that describe optimal pruning for grape and apple We conducted four experiments at the Penn State field laboratory and commercial partner’s orchard to develop and refine pruning rules. Data were used to refine the preliminary pruning rules. Specific grapevine pruning rules for hand pruning cordon-trained, spur-pruned wine grapes were written. Activity 1-2: Evaluate the formulated rules for grape and apple Pruning rules for apples were evaluated in 8 experiments. They appear to encapsulate optimal pruning with similar effects as human pruning. These rules will also prove valuable as an educational toolto help growers make pruning cuts that result in favorable outcomes. GOAL 2: Develop 3D imaging, decision system, and robot control technologies for automating pruning operations Activity 2-1: Develop and test various 3D sensing methodologies for creating accurate 3D reconstructions of leafless canopies under various outdoor conditions An apple tree measuring system is being developed that leverages the grape pruner. Alternative technologies are also being explored. A SICK Laser Measurement System is being used to build 3D tree models. It measures distance and signal strength, has a fast scanning rate and high resolution, is weather resistant and is relatively inexpensive. These studies using laser sensors in a laboratory setting indicated apple branches could be detected using clustering algorithms to form tree skeletons. We have coordinated with grower cooperator orchards in Pennsylvania and Indiana to collect ground truth data of physical attributes of canopy structure. Activity 2-2: Develop a novel decision system for determining optimal pruning points on the 3D reconstructions of the canopies We generated simulated trees to evaluate the algorithms we designed. We developed software to collect 3D images from apple trees from multiple perspectives and integrate them into a single image. Tests with the automated grape pruner were conducted in Lodi, CA, in early 2013. The trials demonstrated consistent and accurate "bud cut" pruning and autonomous steering of the pruner/tractor. It was determined for "spur cuts" that slightly more accurate modeling of vines would be helpful. Activity 2-3: Develop a multifunctional graphical user interface (GUI) for allowing a user to visualize and interact with the 3D reconstructions and the pruning decisions We designed an algorithm to identify the trunk and primary branches of apple trees and tested it using the simulated data. Activity 2-4: Develop sensor-based robot control techniques for guiding a robot arm to precise locations of the calculated pruning points on the canopies This work is scheduled to begin in the second year of the project, however significant progress was made during the current year. Refinements made to the robotic grapevine pruner include modeling, choosing accurate pruning points, defining the best ways for robots to prune, autonomous tractor steering, arm planning and control, synchronization of sensor and actuator subsystems, and eliminating noise and grounding issues. Activity 2-5: Field test grapevine pruner and identify technical areas that need further improvement. Develop proof-of-concept autonomous pruner for apple trees The pruning rules for grapevines were not changed, but we increased our understanding of how the hardware needs to interact with the vines to prune according to the rules previously developed. The pruning rules cannot be met if the system is unable to reach the necessary pruning points. Alternative sources of robot manipulators are being evaluated for weight, speed, weather resistance, payload and cost. Goal 3: Determine social and economic impacts of the proposed autonomous pruning system Activity 3-1: Develop technology commercialization roadmap While work has begun on some aspects of the development of the technology commercialization roadmap, progress has been limited due to a change in project personnel (see further explanation under Changes and Problems). Activity 3-2: Perform 20-25 case studies of growers of different socio-economic status in different regions This work is scheduled to begin in the second year of the project. Activity 3-3: Develop a comprehensive survey and analyze 1000 completed surveys Although this work was scheduled to begin in the 4th year of the project, significant progress has already been made. We developed a survey instrument and have begun surveying apple growers concerning their attitudes toward technological change. We have started interviewing wine-grape and apple growers in Pennsylvania. A survey was conducted of Washington State winegrape growers for current pruning practices, both hand and mechanical. One third of the wine grape acreage in Washington State was represented. Key members of the California wine grape industry were surveyed about pruning practices, both hand and mechanical. Goal 4: Communicate results and involve stakeholders and students so they can adopt these technologies and incorporate knowledge gained into their orchards, vineyards, businesses, classrooms, and laboratories Activity 4-1: Develop and maintain project communication tools; conduct technology transfer tours and workshops and develop educational resources for stakeholders The principle project communication tool is the project website located at http://www.pruningautomation.com. The project team has met via telephone conference and also in person with the advisory committee, in July 2103 at Biglerville, PA. Agenda items included finding a replacement economist for Clark Seavert, website, budget, annual report, research and outreach presentations, and future meetings. Advice from the advisory panel (and other invited industry personnel) were recorded and used in future planning. A technology transfer workshop was conducted Dec. 2012, for 125 participants, and a technology transfer tour was conducted July 2013 for 325 growers from 25 states. Orchard employees attended the pruning workshop, which was presented in both Spanish and English. Technology transfer tour participants who completed an exit survey (n=67) indicated that the three main obstacles to grower adoption of automation were cost, reliability of equipment and lack of equipment flexibility The robotic grapevine pruner was demonstrated at grower field days in CA. Activity 4.2: Conduct trials in whole-farm sustainable demonstration plantings; train student interns; conduct train-the-trainer programs for extension professionals Whole-farm sustainable demonstration plantings have been established in two Indiana commercial orchards and one Pennsylvania commercial orchard site. Following the pruning workshops, twelve orchard employees agreed to participate in a trial to assess ease of following the pruning rules. They quickly adopted the rules, and the trees they pruned could not be visually distinguished from trees pruned by horticulturists.

Publications

• Type: Conference Papers and Presentations Status: Published Year Published: 2013 Citation: Hirst, P., T. Baugher, L. Glenna, A. Kak, T. Koselka, J. Park, A. Patel-Campillo, J. Schupp, C. Seavert, J. Tarara and B. Wallach. 2013. Developing autonomous pruning for specialty crops. Hort Science 46(9) (supplement). S87.
• Type: Journal Articles Status: Accepted Year Published: 2013 Citation: Caplan, S., B. Tilt, G. Hoheisel, T. Auxt Baugher. 2013. Specialty crop growers perspectives on adopting new technologies. HortTechnology
• Type: Websites Status: Published Year Published: 2012 Citation: Anon. 2012. Robotic pruning for grapes, apples. http://www.agriview.com/briefs/crop/robotic-pruning-for-grapes-apples/article_a7e1ea6c-2a93-11e2-b144-0019bb2963f4.html
• Type: Websites Status: Published Year Published: 2013 Citation: Anon. 2013. Robotic vineyard pruning under development. Western Farm Press. http://westernfarmpress.com/grapes/robotic-vineyard-pruning-under-development
• Type: Other Status: Published Year Published: 2013 Citation: Schupp, J., T. Auxt Baugher, E. Winzeler, T. Kon and M. Schupp. 2013. Labor efficient apple and peach production. PA Fruit News 93(1):35-37.
• Type: Conference Papers and Presentations Status: Published Year Published: 2013 Citation: Schupp, J. 2013. Mechanizationimproving efficiency on your farm. Proceedings Empire State Producers Expo, Syracuse, N.Y. www.hort.cornell.edu/expo/2013 proceedings.
• Type: Other Status: Published Year Published: 2013 Citation: Schupp, James R. 2013. Renewal pruning for high density apple plantings. Penn State Fruit Times. http://extension.psu.edu/plants/tree-fruit/news/2013/renewal-pruning-for-high-density-apple-plantings
• Type: Websites Status: Published Year Published: 2013 Citation: Anon. 2013. Robotic pruning system in works for vineyards. Growing Produce. http://www.growingproduce.com/fruits-nuts/grapes/robotic-pruning-system-in-works-for-vineyards/
• Type: Websites Status: Published Year Published: 2013 Citation: Brown, D. 2013. Robotic pruning system under development. Ag Annex. http://www.agannex.com/production/robotic-pruning-system-under-development
• Type: Other Status: Published Year Published: 2013 Citation: Colby, Sally. 2013. Automated pruning for orchards. Growing 8:42-43.
• Type: Websites Status: Published Year Published: 2013 Citation: Colby, Sally. 2013. International fruit tree association 2013 study tour. Country Folks Grower. www.cfgrower.com
• Type: Websites Status: Published Year Published: 2013 Citation: Granchiff, M. 2013. Edward Scissorhands replacing hand pruning. Midwest Wine Press http://midwestwinepress.com/2013/01/11/mechanical-grape-pruning/
• Type: Other Status: Published Year Published: 2013 Citation: Lehnert, Richard. 2012. Robotic pruning: SCRI project uses $6 million to develop system for dormant pruning apples and grapes. Good Fruit Grower.
• Type: Other Status: Published Year Published: 2013 Citation: Nelson, A. 2012. The robots are coming. The Packer. http://www.thepacker.com/opinion/The-robots-are-coming-178161251.html
• Type: Other Status: Published Year Published: 2013 Citation: Torres, Chris. 2013. Pruning workshop focuses on high-density fruit. Lancaster Farming. www.lancasterfarming.com
• Type: Other Status: Published Year Published: 2013 Citation: Wallheimer, Brian. 2013. Purdue developing robotic pruning for grapes, apples. Fruit Growers News. http://fruitgrowersnews.com