Recipient Organization
Carnegie Mellon University
5000 Forbes Avenue
Pittsburgh,PA 15213-3815
Performing Department
Robotics Institute
Non Technical Summary
Our long-term goal is to develop a commercially viable fully autonomous pruning system to reduce dependency on seasonal semi-skilled workers while excelling productivity. The overall objective is to investigate the state-of-the-art in robotics technology to significantly improve and stabilize the balance between vegetative and reproductive growth that would yield better fruit quality and predictable crop load. Our approach deviates significantly from the established paradigm in robotic grapevine pruning in two major ways. Firstly, it is recognized that a grapevine training system that facilitates robotic technology in vineyards is the key to the successful implementation of autonomous and selective pruning of vines. Second, the design of the proposed robot is general-purpose and multi-functional that makes it compatible with different varieties and canopy architectures and is more novel compared to existing systems. Furthermore, the concept of balanced pruning (balancing plant vegetative and reproductive growth) is common among most woody perennial cropping systems (apples, cherries, and other tree fruits, and nut trees). The technology and concepts developed here for juice and wine grapes would translate to other systems as well.A first iteration of the prototype robot that embodies a simpler and commonly practiced spur pruning has already been built and recently evaluated in a commercial vineyard. This early stage system evaluation played a crucial role in understanding the practical requirements in the field. The objectives in this proposal are significant improvements to the existing system and are based on the \textbf{lessons learned} from using the prototype in real field deployments. We also use diverse vanguard learning methodologies in synergy with classical approaches to prune real vines in commercial fields. Thus, we can overcome their individual limitations and push research in the right direction and leverages the benefits of both approaches. We believe that continuity of this research could lead to a practical and economical solution for automated pruning within a reasonable time frame. The adoption of this technology will have significant impacts in the U.S. grape industry both in the mid and long terms.Intellectual Merits: Robot systems to selectively prune grape vines do not exist while the industry has clear needs for it in today's economy. Pruning a vine without any modification and in its natural form poses multiple interesting challenges that requires advanced research in multiple branches of robotics including perception, manipulation, and AI. In this proposal we investigate fundamental research advances in robotics that will have broader impact, ranging from automation in more general tree canopies to a range of everyday tasks that require intelligent interaction with flexible materials in cluttered spaces. From perception perspective, we propose illumination-invariant imaging capabilities to generate reliable and consistent pixel information in the outdoor environment. Dormant season vines contain dense, criss-crossing branches that effectively fill a 3D volume while also leaving many small unoccupied spaces. The resulting highly occluded complex geometry is difficult to model, and existing modeling methods such as SLAM are not capable of generating complete maps. Our approach under the perception goal addresses this complex problem with novel approach that systematically and optimally identify region of interest(s) and recover missing information to complete vine models. Similarly, deciding where to make a pruning cut requires intelligence to understand the canopy at multiple levels, including its geometry, its topology (what is connected to what), and its semantic meaning (what parts are canes, buds, etc.). The ability to automatically generate this level of understanding does not currently exist. Thus, from manipulation standpoint, we are pushing research boundaries to operate robot arms in cluttered and full of flexible objects and AI modules that learns to avoid or push away objects in order to reach deeper into the canopy. Currently existing standard manipulation planning approaches are not equipped to handle these cases.Broader Impacts: (i) This research while pushing the current bounds of robotics and AI research, also has a real potential to deliver more productive and sustainable agriculture, especially in a scenario of farming labor shortage and climate change. (ii) The approaches presented here would increase the economic competitiveness of the U.S. Grape industry and establish a partnership between academia, industry, and stakeholders. (iii) The research team will train underrepresented groups, a graduate student, and expose local high school students through established programs at CMU.
Animal Health Component
25%
Research Effort Categories
Basic
75%
Applied
25%
Developmental
(N/A)
Goals / Objectives
The overall goal of this project is to develop and evaluate an autonomous robotic platform to execute dormant season grapevine pruning in the service of promoting vine balance.There are three technical objectives and one viticultural objectiveResearch Goal 1: Active perception for vine modelingHaving comprehensive 3D information of plants is a basic requirement for any robotic system that interacts with them. This goal is to use robotic perception systems to create complete model of a dormant grapevine, i.e. a model without significant gaps or missing functional relationship between parts of the vine. The research will overcome shortcomings present in current state-of-art mapping and modeling systems when applied in agricultural settings. The primary such shortcoming the large model gaps that result from the significant level of occlusion in grapevines. To find a solution to this problem, we pose two questions: 1) where are the discontinuities in the input pointcloud and 2) how to fill-in the missing information to complete the model.Research Goal 2: Robust vine vigor measurementIt is well known that there is a strong correlation between cane weight and length. This information could be good enough itself to estimate vine size/ pruning weight by using simple linear regression models. However, such process would require multiple intermediate and heuristic-based approach which would be harder to generalize in complex structures. Here, for end-to-end solution we leverage on the robustness of deep networks to estimate vine size (i.e., pruning weight) directly from the pointcloud.Research Goal 3: Learning-based manipulationThe task of pruning dormant trees using a robotic manipulator can be interpreted as a sequence comprising four steps: i) scout the canopy to locate the desired pruning points, ii) find a path that (without damaging the tree) places the cutting tool in a region with a direct line of sight (i.e., free of obstacles) to the desired point, iii) move towards the cutting point and iv) make the cut. This goal is to develop necessary fundamental and systems advances necessary to implement these four steps autonomously. This goal is broken into sub-goals that include data driven approaches to learn pruning rules, and reinforcement learning driven policies to safely reach a manipulator into desired cut points within a vine canopy.Research Goal 4: Further design and develop grapevine training systems for robotic applicationsThis is to deploy, validate, and understand the impact of using the automated pruning system that results from Research Goals 1-3 in a viticultural setting. There are two subgoals: to test the robot in three commonly used vine architectures; and to understand the best way to use an autonomous pruning robot in combination with currently existing mechanized pruning machines.
Project Methods
The overall approach to the program is to develop fundamental advances necessary to achieve research goals 1-3, then integrate the results into a fieldable robotic system for research goal 4. A description of the methods for each of the research goals follows.Research Goal 1: Active perception for vine modeling:To deal with point cloud discontinuities in the model, we start by voxelizing the incoming (incomplete) pointcloud from the global stereo cameras using methods such as octree data structure. At high level, the next logic here is then to track the topological changes in the graph as a function of the voxel size. As the voxel size iteratively decreases, strongly connected components start to form clusters as smaller voxels are unable to maintain connection to the neighbouring points and to the initially connected larger voxels. The next step in the pipeline is then to simply keep track of all new disjoints over all iterations. Thus, at the end of the cycle the pipeline outputs a list of missing edges in the graph that can be traced to its actual 3D location(s) in the digitized model. A custom built in-hand camera attached to the robot arm positions itself optimally in the vicinity of the canopy indicated by the algorithm.Once discontinuities have been identified and the in-hand camera has been positioned, we will find ways to connect the disconnected regions of the model using the in-hand camera pointcloud and fusing it back to the input pointcloud. Traditional approaches for registering pointclouds such as Iterative Point Cloud (ICP) by could also work here. However, it requires very accurate initialization and usually do not converge well for sparse or noisy inputs. For robust pointcloud infill without any initialization and extrinsic calibration of eye-in-hand camera, we propose a deep learning-based approach. Motivated MaskNet, we present "Dual MaskNet" that learns an additional binary vector which is the difference between the inliers and the input in-hand pointcloud that essentially learns to capture missing links.Research Goal 2: Robust vine vigor measurementWe approach the problem of translating vine pointcoulds to a vigor measurement by building upon new and more generalized form of deep neural network called Graph Neural Network (GNN) that has received much attention in recent history. This type of network can directly take graphs as input and learn complex patterns from them. Here, the input to the GNN network would be a graph of segmented cane pointclouds (the nodes are the cane segments, the edges are the physical connections between them).As multiple layer Graph Convolution layers extract high dimensional features, the end of the network features fully connected layers for regression that outputs the vine size. Under this objective, we will further advance this approach by investigating into the latest development in GNN with attention networks for various tasks. We hypothesize that the graph attention network could dynamically adjust its kernel to adapt to the structure of vines. Recent work has shown similar capabilities in pointcloud segmentation with GNNs. Such features could allow us to process pointclouds without pre-processing for more robust end-to-end capabilities.Research Goal 3: Learning-based manipulationResearch goal 3 has two objectives:Learning the pruning rule, andreaching the desired pruning point.Tolearn the pruning rule, we will explore methods to estimate the best set of cut points based on the vine sensing and modeling described in earlier sections. The selection of the pruning points is crucial to maintain the balance of the vines though all seasons and has to be carefully planned. This planning involves all the decision process such as tree structure, its geometric characteristics, the localization of the buds, and the vine vigor measurements in the selection of points pruning points.In this project we will study two different data-driven approaches in an effort to mimic the decision making of expert human pruners: supervised and self-supervised learning. We will first use a binary supervised learning method, which fits well to address this problem. The training dataset can be composed of the tree model (possibly including its geometric features) and the 3D location of all the buds. Each bud will be labeled as cut or no cut. These labels will be provided by human experts, who will mark the buds selected for pruning in real vines. We can subsequently identify those points in their virtual counterparts.Toreaching the pruning points,we will explore the trade-offs between classic approaches and research in reinforcement learning (RL). Here, we use the term "classical approach" to refer traditional trajectory and path planners such as rapid exploring random trees or optimization-based. Once the tool is in direct line of sight, driving it to the cutting point using the classical approach is a pragmatic solution. However, the step of moving the tool from the home position to the place of direct line of sight requires something more advanced to deal with the presence of complex occlusions. Here we will design an approach mainly based on RL, as it naturally fits in the exploration-exploitation behavior required for globally or locally scouting the tree and locating new potential pruning points. We plan to maintain the research direction in on-policy methods such as Proximal Policy Optimization (PPO) as it provides a theoretical learning framework to improve over known issues in other methods such as sample efficiency, implementation complexity or hyper-parameter tuning.Research Goal 4: Further design and develop grapevine training systems for robotic applicationsIn this research goal, we will integrate the above capabilities into a fieldable demonstration platform for autonomous grapevine pruning that includes a mobile base, necessary perception systems, and a robotic arm equipped with a vine cutting tool. It will deployed at the Cornell Lake Erie Research and Extension Laboratory (CLEREL) in a variety of viticultural settings to validate performance and investigate the potential impact of using this technology in commercial production systems. All of the varieties, training systems, mechanization field comparisons, and vineyard equipment have already been established at the Cornell Lake Erie Research and Extension Laboratory (CLEREL). Dr. Bates and his team have extensive experience in vineyard mechanization and precision viticulture management.The pruning robot will be tested in V. vinifera, V. labrusca, and inter-specific hybrid research vineyard plots. 'White Riesling' will be low-wire cordon trained with vertical shoot positioning, 'Concord' will be high-wire trained with sprawling canes, and 'Vignoles' will be high-wire cordons trained as a semi-sprawl system. For each variety, the robot will be used in an unpruned system to test the ability of the robot to complete 100% of the pruning job and in a mechanically pre-pruned system to test the ability of the robot to refine fruiting bud quantity and quality. These two treatments (robotic, mechanical + robotic) will be compared to the control of 100% manual balanced pruning. Observations will be collected on vine vegetative growth (pruning weight and retained nodes), reproductive growth (yield and yield components), and fruit quality (juice soluble solids, titratable acidity, pH, and color).To explore the relationship between trellis architecture and automation efficiency, a traditional grape trellis with wood posts at every third vine will be compared to a vineyard with a metal post at each vine. The wood trellis system is less expensive to install but will sag over time and creates challenges for machine operation. The metal trellis design costs more to install but maintains uniformity and could potentially improve efficiency. A side by side comparison will be conducted with the pruning robot at CLEREL.