Progress 06/01/21 to 05/31/22
Outputs
Target Audience:The target audience for this reporting period was the robotics research community. During the prior year, we focused on disseminating our research results through technical workshops and conferences. Changes/Problems:
Nothing Reported
What opportunities for training and professional development has the project provided?For this performance period, the project has provided research experience for three PhD students, two undergraduate Honors College students, and one capstone team of undergraduate computer science students. One REU student (NSF REU: Robots in the Real World) also worked on the project during the summer of 2021. How have the results been disseminated to communities of interest?The results have been disseminated through two submissions to international robotics conferences, one undergraduate thesis, and one poster presentation at an agricultural robotics workshop. What do you plan to do during the next reporting period to accomplish the goals?Additional proxy data collection is planned to provide a dataset that is labeled with measured locations of the proxy abscission joint. This dataset will be used to further develop our model estimation software and will also account for unusual picking configurations. Once we collect this data, our model identification software can be evaluated and integrated with the ROS interface. Given the success of our preliminary grasp adjustment detection technique, we plan to expand the principles on which it is based to more sophisticated techniques. This includes both more accurate techniques for grasp adjustment detection as well as classification methods for determining whether an adjustment will result in grasp failure. For the upcoming year we also plan to redesign the robotic gripper. Our objectives will be to i) minimize the geometric span of the design to reduce the chance for unintended collisions with obstacles during a pick; and ii) incorporate additional soft components for increased robustness. We plan to evaluate our system in a commercial orchard during the summer of 2023.
Impacts
What was accomplished under these goals?
Data collection A major accomplishment over the past year was the collection of three datasets. Our data collection process consisted of collecting measurements from real world apple picks (from an un-trellised honeycrisp apple tree) as well as emulated apple picks using the apple proxy that we developed during the first year of the project, the latter being split into grasps that emulated the real-world ones and random ones. In total we completed 70 real apple pick experiments and 419 proxy experiments. For each pick attempt we collected the robot's wrench (i.e. torques/forces from the sensor located on the wrist of our Universal Robots UR5e manipulator), inertial measurements (angular velocity and linear acceleration) from each of the three fingers on our robotic gripper, and motor states (position, velocity and effort) from each finger's servo motor. Altogether, for each pick we collected temporal measurements of 33 variables. After collecting the data, we then post-process it as a prerequisite step prior to the implementation of machine learning models. Post-processing mainly included noise filtering, downsampling, and data augmentation. Fine Control We have developed a physical model of the apple picking task which shows good agreement when the model parameters are fit by software. Work has begun which uses this model to infer critical system parameters for robot decision making (finger location, stem location and orientation wrt the palm). A preliminary algorithm for this purpose has been developed with positive results on simulated data. We have also created a Robot Operating System (ROS) package which connects the relevant datastreams to processing programs, which will be used for online operation of the algorithm once it is completed. In parallel, we have developed a system for grasp adjustment detection which monitors for both failure and non-failure slip events during the picking task. To measure the true time of slip, we had a research assistant manually label the time of slip during the first 260 of the 419 proxy experiments. The time of slip was determined visually from webcam data. Using a windowed variance across multiple sensor channels, we were able to create a simple slip-detection heuristic with ~83% accuracy. In addition to detecting slip, we are also studying the prediction of successful or failed apple picks before they occur. To do this, we performed a myriad of experiments using a Long Short-Term Memory network. This allowed us to test both the viability of the apple proxy as a stand-in for the real world and to perform an ablation study to understand which sensors were most useful for grasp classification. Based on our current sensors, we can achieve 87% accuracy at grasp classification when trained on the proxy and tested on the real world. This finding validated that our proxy is useful as a tool for training machine learning agents. We were also able to determine which sensors were most important for classification and found that certain sensors are more important during different phases of the grasp.
Publications
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2021
Citation:
L.M. Dischinger, M. Cravetz, J. Dawes, C. Votzke, C. VanAtter, M.L. Johnston, C.M. Grimm, and J.R. Davidson, Towards intelligent fruit picking with in-hand sensing, in Proc. IEEE/RSJ Intl Conf. on Intelligent Robots and Systems (IROS), virtual, September 2021, pp. 3285-3291. https://doi.org/10.1109/IROS51168.2021.9636341
- Type:
Conference Papers and Presentations
Status:
Under Review
Year Published:
2022
Citation:
A. Velasquez, N. Swenson, M. Cravetz, C.M. Grimm, and J.R. Davidson, Predicting fruit pick success using a grasp classifier trained on a physical proxy, Submitted to the IEEE/RSJ Intl Conf. on Intelligent Robots and Systems (IROS), 2022
- Type:
Theses/Dissertations
Status:
Published
Year Published:
2022
Citation:
Lissette J.H. Wilhelm, Design of 3d-printed soft pneunet actuator for robotic fruit harvesting, Honors College thesis, Oregon State University, 2022.
|
Progress 06/01/20 to 05/31/21
Outputs
Target Audience:The target audience for this reporting period was the robotics research community. During the prior year, we focused on disseminating our research results through technical workshops and conferences. Changes/Problems:Our efforts to collect data with the sensorized end-effector in field conditions were negatively impacted by extensive wildfires in Oregon during the 2020 harvest season. What opportunities for training and professional development has the project provided?For this performance period, this project provided research experiences for four graduate students and fourundergraduate students. How have the results been disseminated to communities of interest?The results have been disseminated through one conference submission, two workshop submissions (workshop on manipulation in agriculture), and one poster presentation (workshop on soft robotics in agriculture). What do you plan to do during the next reporting period to accomplish the goals?For the upcoming reporting period, we will focus on the following activities: Completing extensive picking trials with the instrumented orchard proxy to collect sensor datasets. The data will be used to train machine learning algorithms that can detect the following three events: i) initial grasp complete; ii) the apple slipped from the hand during picking; and iii) apple successfully picked. Additionally, machine learning plus kinematics/dynamics models will be used to i) estimate the pose of the end effector; ii) estimate the size, center, and stem orientation of the apple. This year we will attempt to collect more extensive data on gripper finger pressure and wrist force during picking with controlled trials in outdoor conditions, using the modified end effector. This will allow us to evaluate the performance of the in-hand sensors and determine whether modifications will be required to the first prototype design.
Impacts
What was accomplished under these goals?
During this reporting period, we completed activities in support of objectives 1 and 2: 1. Create an extended manipulation feedback loop incorporating a rich set of in-hand sensors with the objective of providing fine control over how the fruit is separated from the tree. The dataset of measurements from the instrumented end-effector have been analyzed to (i) detect contact between the end effector and the environment; (ii) classify attempted apple picking sequences as successful or failed; (iii) identify the most informative sensor measurements for grasp quality assessment; and (iv) identify needed improvements for future generations of the end-effector. A second end-effector and testbed have been developed to study a potential state-estimation method for localizing the proximal and distal links in underactuated robotic grippers. We intend to use the estimates generated by this method to inform our grasping and manipulation strategies. Work has begun on generating a dataset using this testbed. 2. To overcome the constraints of data collection in the field, we will develop a learning framework for compartmentalizing the tasks and design an instrumented proxy to serve as a training environment. An artificial apple proxy has been designed in order to (i) enable robotic apple picking regardless of the season; and (ii) provide a testbed where we can generate the large sets of sensor data required to implement machine learning techniques (by executing thousands of picks). To replicate the mechanics of the apple picking process as realistically as possible, we specified the following initial design guidelines for the proxy: The stem of the apple should have compliance along three degrees of freedom. The apple should be able to rotate with respect to the stem. The system should reflect the 1~2 inches of deflection that is perceived while picking an apple. The system should permit multiple apples, including clustered fruit, to be installed with adjustable positioning. The apple should separate from the mechanism after the application of approximately 45 N of force. The fabricated proxy consists of three subsystems: Apple: A hollow 3d-printed apple with internal electronics (i.e. microcontroller and inertial measurement unit) to keep track of the apple's position and velocity. Mechanical stem: This is the component that keeps the apple attached to the branch and allows rotation of the apple. Additionally, it has an internal telescopic component that resembles the deflection obtained when an apple is manually picked. The tip of the stem consists of an off-the-shelf magnet properly selected to release the apple upon reaching the desired deflection. Branch: This component consists of a shaft that holds a series of bushings which represent the interface to the stems. These bushings are polyhedrons with multiple faces such that clusters of apples can be installed with variable orientations. Additionally, the shaft is connected to an electric motor that can adjust the angle of the branch. To execute initial apple picks with the proxy, we have integrated a commercial robotic gripper (Robotiq 2f85) with the UR5e industrial robot (Universal Robots). We are using the Robotic Operating System (ROS) and MoveIt! software package to communicate with, and control, the manipulator. Some preliminary picking experiments with the proxy indicate that the system is performing as desired (i.e. replicating the mechanics of apple picking reasonably well).
Publications
- Type:
Conference Papers and Presentations
Status:
Under Review
Year Published:
2021
Citation:
L. Dischinger, M. Cravetz, J. Dawes, C. Votzke, C. VanAtter, M.L. Johnston, C. Grimm and J.R. Davidson, Towards intelligent fruit picking with in-hand sensing, 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (under review).
|
|