Progress 06/01/20 to 05/31/24
Outputs
Target Audience:The target audience for this project was the fresh market tree fruit industry and the robotics research community. Changes/Problems:
Nothing Reported
What opportunities for training and professional development has the project provided?Over its4-year duration, the project provided research experiences forthree PhD students and one MS student (all in Robotics). Additionally, selected portions of the project provided topics for one undergraduate Honors thesis, one undergraduate mechanical engineeringcapstone project, foursummer REU students (NSF REU: Robots in the Real World), and one high school summer intern. How have the results been disseminated to communities of interest?Our research results were presented at three international robotics conferences and three workshops on agricultural robotics. Additionally, the project was discussed in various media products accessible to the general public, including the following: "And on that farm, he had a robot," Engineering Out Loud podcast, Oregon State University, 2021. https://engineering.oregonstate.edu/season-9-robotics-and-ai/and-farm-he-had-robot-s9e5 "Will this fruit-picking robot transform agriculture," The Guardian, May 2022. https://www.theguardian.com/us-news/2022/may/28/robot-agriculture-farming-artificial-intelligence "Where are the orchard bots? OSU shares the latest research on tree fruit robotics," Capital Press, November 3, 2022. https://www.capitalpress.com/ag_sectors/research/where-are-the-orchard-bots-osu-shares-the-latest-research-on-tree-fruit-robotics/article_70a19070-5b0c-11ed-b668-0f73806bdcfa.html "A helping (robotic) hand for harvest," Capital Press, December 28, 2023. https://www.capitalpress.com/ag_sectors/orchards_nuts_vines/a-helping-robotic-hand-for-harvest/article_3324175a-9fa2-11ee-a13c-8f001982d4b0.html What do you plan to do during the next reporting period to accomplish the goals?
Nothing Reported
Impacts
What was accomplished under these goals?
In the following sections, we describe our accomplishments over the duration of the project in the areas ofnovel hardware design, learning, and data collection. Our initial robotic hand design, presented at the 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), featured three passively compliant sweeping fingers with tactile sensors in the finger tips. The fingers initially move in an arc, then begin to bend once the bases of the fingers contact an object in the environment. The fingers stop bending when a pre-determined tension is achieved in the actuating cables, generally when the tip of the finger makes firm contact. This shape-conforming behavior allowed the hand to securely grasp the fruits without the need to calculate exact contact forces and contact locations for the grasp. However, when the path of a finger was obstructed by a branch, the hand could not fully close around the fruit. This was the most common grasp failure scenario we encountered during our 2020field trials, motivating us to create a gripper with a more compact and task-suitable finger path. In late 2020, we createda physical twin to replicate the dynamics of apple picks. We had two goals with the phsyical twin: 1) train robotic controllers for fruit harvesting in the lab;and 2) use the platform to design a new prototype gripper. Our physical twin (i.e. apple proxy) consists of a simple set of springs and magnets which mimic key dynamic traits from apple picks: detachment force, branch stiffness, and sudden detachment. We tuned the proxy by using off-the-shelf springs with spring constants that match branch stiffnesses measured duringfield trials. We also used magnets with forces that match stem detachment forces observed during field trials. These components can easily be replaced to replicate apple varieties with different properties. In 2021 we evaluated the realismof our proxy by comparinga LSTM-based grasp classifier trained with data obtained with robotic picks on the proxy with onetrained using real world data. We published theresults of this work at the 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Using thephysical proxy mentioned previously, from 2022-2023 wedesigned a task-oriented gripper with a small form-factor, robust to error in fruit perception and localization. Our new prototype design consists of a novel robotic dual-actuation gripper with in-hand perception (camera and time-of-flight). For actuation, the gripper has three multi-bellow suction cups on the palm that provide an initial compliant engagement with the fruit. The suction cups together with in-line air-pressure sensors provide tactile feedback that can be usedtoadjustthe gripper's pose during the grasping stage. The second actuation mode consists of a cam-based mechanism that deploys telescoping fingers from underneath the gripper's palm. The fingers are deployed after the suction cups engage, and provide the forces required for further manipulation. We tested the gripper with our proxyand validated it with field trials in a commercial orchard (Prosser, WA) in Fall 2023. We published our suction-based results at the 2024 IEEEInternational Conference of Robotics and Automation (ICRA) and are currently preparing a second manuscript on thedual-actuation concept and field results. As mentioned above, in the Fall of 2023 we performed robotic apple picking trials on three unique orchard rows: a trellised row of Gala apples in Randwijk, Gelderland, Netherlands, an untrellised row of Gala apples in Randwijk, Gelderland, Netherlands, and a row of trellised Envy apples in Prosser, Washington, USA. In addition to collecting robotic sensor data during these trials, we attached a set of three sensor modules to the tree branches from which we were harvesting. We plan to release the data as a publicly available dataset alongside a report on our findings from the trials. The manuscript of this report is currently in progress.
Publications
- Type:
Conference Papers and Presentations
Status:
Awaiting Publication
Year Published:
2024
Citation:
A. Velasquez, C. Grimm, and J.R. Davidson, Dynamic evaluation of a suction based gripper for fruit picking using a physical twin, In Proc. IEEE Intl Conf. on Robotics and Automation (ICRA), Yokohama, Japan, May 2024.
|
Progress 06/01/22 to 05/31/23
Outputs
Target Audience:The target audience for this reporting period was the robotics research community. During the prior year, we focused on disseminating our research results through technical workshops and conferences. Changes/Problems:
Nothing Reported
What opportunities for training and professional development has the project provided?In addition to the three PhDstudents who made research contributions to this project over the prior year, the development of the in-hand perception system served as the capstoneproject for three mechanical engineering undergraduate students. A summer 2022 REU student (NSF REU: Robots in the Real World) and high school intern also actively worked on the project over the prior year. How have the results been disseminated to communities of interest?The results have been disseminated through a presentation (1x) and submission (1x) to international robotics conferences, and one poster presentation at an agricultural robotics workshop. What do you plan to do during the next reporting period to accomplish the goals?Our focus forthe upcoming year will be a second round of harvesting field trials in a commercial orchard(Fall 2023). The goals of the field trials will be to evaluate the performance of the new gripper prototype as wellasvalidate our learning method for estimatingimportant pick features during precision fruit manipulation.
Impacts
What was accomplished under these goals?
Learning By examining the sensor data from our previous harvesting field experiments, we were able to identify characteristic patterns in the fingertip inertial measurements before and during slip events. We used this observation, along with our understanding of the underlying mechanics, to develop custom sensor features for slip detection and prediction. Using laboratory experiments, we developed a machine learning approach which uses only six sensor channels from a soft hand and a random forest classifier to assign grasps as static or slipping. The performance of our classifier was comparable to the state of the art, and achieved with a lower computational complexity and lower training overhead. This work is currently under review for the IEEE/RSJ Int'l Conference on Intelligent Robots and Systems (IROS). Additionally, we have begun efforts to capture the deformation mechanics of apple trees for the purpose of informing our robotic manipulation of apples. We have created a model of the apple tree as an articulated object with stiffnesses at some articulations. Using this model, we have created a simulation that generates sets of sensor data paired with model parameters that let us reconstruct key features of the tree. We plan to leverage this simulation to design exploratory procedures that allow us to localize important tree features such as the location of the abscission layer for a grasped fruit. This will in turn allow us to execute mechanically ideal picking patterns with the robot during manipulation of the fruit. Gripper redesign One of the main findings from our earlier field experiments was the importance of having a gripper with a small form factor to minimize unintended collisions with surrounding vegetation during fruit picking. With this design requirement in mind, over the past year we developed a new gripper that comprises two actuation modes (suction cups and cam-following fingers) and several perception modes. In the redesigned gripper, a set of three fixed, multi-bellow suction cups are placed in the palm of the gripper; three telescoping 'fingers' are located underneath the palm and along the body of the gripper. The suction cups are fixed and offset from the center of the palm to provide space for an in-hand camera (IHC) and a time-of-flight (ToF) sensor; the cups are also tilted at an angle, which optimizes the performance under angular and cartesian noise in gripper localization. The optimal angle was found empirically by performing 640 proxy experiments with different sets of tilt angles, offsets, and sphere diameters. The results of this work were recently accepted to a workshop at the 2023 IEEE Int'l Conference on Robotics and Automation. The fingers consist of a cam-following mechanism that results in the tip of the fingers following a curved path that sweeps the shape of the fruit. We hypothesize that this type of finger trajectory will be advantageous for pushing away leaves, branches and neighboring apples that could interfere with successful contact between the gripper and target fruit (thereby improving the pick success rate). We will evaluate the performance of the new gripper during field trials in Fall 2023. Over the past year we also developed a closed-loop apple approach method that utilizes a compact camera and time of flight sensor that can fit in the palm of the gripper. This method uses a deep neural network that has been additionally trained using images of apples taken from nearby trees. We have evaluated this method on the apple proxy and found that it is capable of bringing the hand to the apple in 90% of our trials, regardless of hand starting position. We are currently evaluating potential next steps to improve speed and accuracy as well as how to generalize the method to handle occlusions from leaves and other fruit.
Publications
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2022
Citation:
A. Velasquez-Lopez, N. Swenson, M. Cravetz, C. Grimm, and J.R. Davidson, Predicting fruit-pick success using a grasp classifier trained on a physical proxy, in Proc. IEEE/RSJ Intl Conf. on Intelligent Robots and Systems (IROS), Kyoto, Japan, Oct. 2022, pp. 9225-9231.
- Type:
Conference Papers and Presentations
Status:
Under Review
Year Published:
2023
Citation:
M. Cravetz, C. Grimm, and J.R. Davidson, Detecting and predicting slip during robotic harvesting with inertial signals: A simple learning-based approach, submitted to IEEE/RSJ Intl Conf. on Intelligent Robots and Systems (under review)
|
Progress 06/01/21 to 05/31/22
Outputs
Target Audience:The target audience for this reporting period was the robotics research community. During the prior year, we focused on disseminating our research results through technical workshops and conferences. Changes/Problems:
Nothing Reported
What opportunities for training and professional development has the project provided?For this performance period, the project has provided research experience for three PhD students, two undergraduate Honors College students, and one capstone team of undergraduate computer science students. One REU student (NSF REU: Robots in the Real World) also worked on the project during the summer of 2021. How have the results been disseminated to communities of interest?The results have been disseminated through two submissions to international robotics conferences, one undergraduate thesis, and one poster presentation at an agricultural robotics workshop. What do you plan to do during the next reporting period to accomplish the goals?Additional proxy data collection is planned to provide a dataset that is labeled with measured locations of the proxy abscission joint. This dataset will be used to further develop our model estimation software and will also account for unusual picking configurations. Once we collect this data, our model identification software can be evaluated and integrated with the ROS interface. Given the success of our preliminary grasp adjustment detection technique, we plan to expand the principles on which it is based to more sophisticated techniques. This includes both more accurate techniques for grasp adjustment detection as well as classification methods for determining whether an adjustment will result in grasp failure. For the upcoming year we also plan to redesign the robotic gripper. Our objectives will be to i) minimize the geometric span of the design to reduce the chance for unintended collisions with obstacles during a pick; and ii) incorporate additional soft components for increased robustness. We plan to evaluate our system in a commercial orchard during the summer of 2023.
Impacts
What was accomplished under these goals?
Data collection A major accomplishment over the past year was the collection of three datasets. Our data collection process consisted of collecting measurements from real world apple picks (from an un-trellised honeycrisp apple tree) as well as emulated apple picks using the apple proxy that we developed during the first year of the project, the latter being split into grasps that emulated the real-world ones and random ones. In total we completed 70 real apple pick experiments and 419 proxy experiments. For each pick attempt we collected the robot's wrench (i.e. torques/forces from the sensor located on the wrist of our Universal Robots UR5e manipulator), inertial measurements (angular velocity and linear acceleration) from each of the three fingers on our robotic gripper, and motor states (position, velocity and effort) from each finger's servo motor. Altogether, for each pick we collected temporal measurements of 33 variables. After collecting the data, we then post-process it as a prerequisite step prior to the implementation of machine learning models. Post-processing mainly included noise filtering, downsampling, and data augmentation. Fine Control We have developed a physical model of the apple picking task which shows good agreement when the model parameters are fit by software. Work has begun which uses this model to infer critical system parameters for robot decision making (finger location, stem location and orientation wrt the palm). A preliminary algorithm for this purpose has been developed with positive results on simulated data. We have also created a Robot Operating System (ROS) package which connects the relevant datastreams to processing programs, which will be used for online operation of the algorithm once it is completed. In parallel, we have developed a system for grasp adjustment detection which monitors for both failure and non-failure slip events during the picking task. To measure the true time of slip, we had a research assistant manually label the time of slip during the first 260 of the 419 proxy experiments. The time of slip was determined visually from webcam data. Using a windowed variance across multiple sensor channels, we were able to create a simple slip-detection heuristic with ~83% accuracy. In addition to detecting slip, we are also studying the prediction of successful or failed apple picks before they occur. To do this, we performed a myriad of experiments using a Long Short-Term Memory network. This allowed us to test both the viability of the apple proxy as a stand-in for the real world and to perform an ablation study to understand which sensors were most useful for grasp classification. Based on our current sensors, we can achieve 87% accuracy at grasp classification when trained on the proxy and tested on the real world. This finding validated that our proxy is useful as a tool for training machine learning agents. We were also able to determine which sensors were most important for classification and found that certain sensors are more important during different phases of the grasp.
Publications
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2021
Citation:
L.M. Dischinger, M. Cravetz, J. Dawes, C. Votzke, C. VanAtter, M.L. Johnston, C.M. Grimm, and J.R. Davidson, Towards intelligent fruit picking with in-hand sensing, in Proc. IEEE/RSJ Intl Conf. on Intelligent Robots and Systems (IROS), virtual, September 2021, pp. 3285-3291. https://doi.org/10.1109/IROS51168.2021.9636341
- Type:
Conference Papers and Presentations
Status:
Under Review
Year Published:
2022
Citation:
A. Velasquez, N. Swenson, M. Cravetz, C.M. Grimm, and J.R. Davidson, Predicting fruit pick success using a grasp classifier trained on a physical proxy, Submitted to the IEEE/RSJ Intl Conf. on Intelligent Robots and Systems (IROS), 2022
- Type:
Theses/Dissertations
Status:
Published
Year Published:
2022
Citation:
Lissette J.H. Wilhelm, Design of 3d-printed soft pneunet actuator for robotic fruit harvesting, Honors College thesis, Oregon State University, 2022.
|
Progress 06/01/20 to 05/31/21
Outputs
Target Audience:The target audience for this reporting period was the robotics research community. During the prior year, we focused on disseminating our research results through technical workshops and conferences. Changes/Problems:Our efforts to collect data with the sensorized end-effector in field conditions were negatively impacted by extensive wildfires in Oregon during the 2020 harvest season. What opportunities for training and professional development has the project provided?For this performance period, this project provided research experiences for four graduate students and fourundergraduate students. How have the results been disseminated to communities of interest?The results have been disseminated through one conference submission, two workshop submissions (workshop on manipulation in agriculture), and one poster presentation (workshop on soft robotics in agriculture). What do you plan to do during the next reporting period to accomplish the goals?For the upcoming reporting period, we will focus on the following activities: Completing extensive picking trials with the instrumented orchard proxy to collect sensor datasets. The data will be used to train machine learning algorithms that can detect the following three events: i) initial grasp complete; ii) the apple slipped from the hand during picking; and iii) apple successfully picked. Additionally, machine learning plus kinematics/dynamics models will be used to i) estimate the pose of the end effector; ii) estimate the size, center, and stem orientation of the apple. This year we will attempt to collect more extensive data on gripper finger pressure and wrist force during picking with controlled trials in outdoor conditions, using the modified end effector. This will allow us to evaluate the performance of the in-hand sensors and determine whether modifications will be required to the first prototype design.
Impacts
What was accomplished under these goals?
During this reporting period, we completed activities in support of objectives 1 and 2: 1. Create an extended manipulation feedback loop incorporating a rich set of in-hand sensors with the objective of providing fine control over how the fruit is separated from the tree. The dataset of measurements from the instrumented end-effector have been analyzed to (i) detect contact between the end effector and the environment; (ii) classify attempted apple picking sequences as successful or failed; (iii) identify the most informative sensor measurements for grasp quality assessment; and (iv) identify needed improvements for future generations of the end-effector. A second end-effector and testbed have been developed to study a potential state-estimation method for localizing the proximal and distal links in underactuated robotic grippers. We intend to use the estimates generated by this method to inform our grasping and manipulation strategies. Work has begun on generating a dataset using this testbed. 2. To overcome the constraints of data collection in the field, we will develop a learning framework for compartmentalizing the tasks and design an instrumented proxy to serve as a training environment. An artificial apple proxy has been designed in order to (i) enable robotic apple picking regardless of the season; and (ii) provide a testbed where we can generate the large sets of sensor data required to implement machine learning techniques (by executing thousands of picks). To replicate the mechanics of the apple picking process as realistically as possible, we specified the following initial design guidelines for the proxy: The stem of the apple should have compliance along three degrees of freedom. The apple should be able to rotate with respect to the stem. The system should reflect the 1~2 inches of deflection that is perceived while picking an apple. The system should permit multiple apples, including clustered fruit, to be installed with adjustable positioning. The apple should separate from the mechanism after the application of approximately 45 N of force. The fabricated proxy consists of three subsystems: Apple: A hollow 3d-printed apple with internal electronics (i.e. microcontroller and inertial measurement unit) to keep track of the apple's position and velocity. Mechanical stem: This is the component that keeps the apple attached to the branch and allows rotation of the apple. Additionally, it has an internal telescopic component that resembles the deflection obtained when an apple is manually picked. The tip of the stem consists of an off-the-shelf magnet properly selected to release the apple upon reaching the desired deflection. Branch: This component consists of a shaft that holds a series of bushings which represent the interface to the stems. These bushings are polyhedrons with multiple faces such that clusters of apples can be installed with variable orientations. Additionally, the shaft is connected to an electric motor that can adjust the angle of the branch. To execute initial apple picks with the proxy, we have integrated a commercial robotic gripper (Robotiq 2f85) with the UR5e industrial robot (Universal Robots). We are using the Robotic Operating System (ROS) and MoveIt! software package to communicate with, and control, the manipulator. Some preliminary picking experiments with the proxy indicate that the system is performing as desired (i.e. replicating the mechanics of apple picking reasonably well).
Publications
- Type:
Conference Papers and Presentations
Status:
Under Review
Year Published:
2021
Citation:
L. Dischinger, M. Cravetz, J. Dawes, C. Votzke, C. VanAtter, M.L. Johnston, C. Grimm and J.R. Davidson, Towards intelligent fruit picking with in-hand sensing, 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (under review).
|
|