COLLABORATIVE RESEARCH: NRI: PERCEPTION-AWARE SOFT ROBOT MANIPULATION AND BIPEDAL LOCOMOTION FOR FRESH MARKET CANEBERRY HARVESTING

• Sponsoring Institution: National Institute of Food and Agriculture
• Project Status: ACTIVE
• Funding Source: AFRI COMPETITIVE GRANT
• Reporting Frequency: Annual
• Accession No.: 1031258
• Grant No.: 2023-67021-41397
• Cumulative Award Amt.: $763,948.00
• Proposal No.: 2022-11065
• Project Start Date: Oct 1, 2023
• Project End Date: Sep 30, 2027
• Grant Year: 2024
• Program Code: [A7301]- National Robotics Initiative

Recipient Organization
GEORGIA INSTITUTE OF TECHNOLOGY
(N/A)
ATLANTA,GA 30332

Performing Department
(N/A)

Non Technical Summary
The U.S. berry industry has experienced rapid growth in the past several decades and accounted for 22.1% of the fruit market with a total value of $7.5 billion in 2019. Caneberries, sometimes called brambles, are parts of the Rubus family and refer to a number of berries with delicate drupelets on the outer surface. Among them, blackberries and raspberries account for $697 million and $1.1 billion of the market value, respectively. What is more, the caneberry industry is not stagnant, rather, it is growing, especially the fresh-market industry. With the increased demand and new cultivars, the total blackberry market value has grown by 7.3% between 2019 and 2020, and the total raspberry value has grown by 12.3% in the same period. The global market value for the fresh-market caneberry industry specifically is projected to grow 11.3% between 2019 and 2025.There is a growing interest to choose robotics as the alternatives for agriculture harvesting tasks. Compared to conventional manual approach, robot-assisted harvesting has been shown to reduce the cost of production, improve the supply chain, increase labor efficiency, and reduce wastage, etc. Despite these compelling advantages, robot-assisted harvesting methods for fresh-market caneberries have not yet been explored with the following technical challenges. First, conventional robotic harvesting methods rely on rough handling of fruit by either (1) cutting the stem, (2) shaking the fruit off of the plant, or (3) picking the fruit with various grippers. However, these conventional handling methods will inevitably damage the surface of delicate fruits, such as caneberries in this project, limiting the post-harvesting quality for fresh market. Second, from the agricultural field accessibility perspective, deploying wheeled or tracked robotic systems manifest an evident disadvantage in terms of mobility and versatility compared to human labor. In addition, the berries grow at different locations within and on the outside of dense woody and leafy canopies. These facts pose significant constraints on the reachability space of the robot gripper (e.g., being positioned too far from the plant) if integrated with a wheeled or tracked mobile platform. Third, conventional DNN-based perception methods are able to provide accurate feedback for large fruits (e.g., apple, citrus, kiwifruit), but unfortunately cannot provide sufficient information for precise robotic caneberry harvesting. The perception approach for detecting and localizing small size fruit like caneberries needs to be further investigated using a dual-camera system for an improved localization accuracy for optimal selective harvesting in orchards.The ideal caneberry harvesting robotic platform is expected to have the following core features: 1) Harvesting module. It should dexterously manipulate the grippers around the canopies and enable compliant contact between berry-gripper to achieve premium post-harvesting quality; 2) Locomotion platform. It should be adaptive to difficult terrain conditions and drive the harvesting module to different locations that are close to the trellis or within the canopies; and 3) Robot perception. It will provide accurate perceptions to guide the harvesting module and locomotion platform in complicated agriculture scenarios. The existing robotic units can only provide part of these core features, but not all of them. For example, our soft robotic gripper is able to harvest the blackberries with minimal to no RDR, but lacks the capability to manipulate the soft gripper towards the target locations with real-time perception feedback.In this proposal, we aim to integrate soft robot, bipedal robot locomotion, and robotic perception to create a novel "Ostrich-like" robotic platform and its system integration for fresh market caneberry harvesting. Our proposed work consists of four research tasks: (1) Soft robot design, modeling, and control, where we will first develop a multi-objective, data-driven optimal design strategy to create the soft robotic hardware. We will then investigate soft robot modeling, and develop control algorithm for accurate motion. (2) Enabling versatile bipedal locomotion over highly unstructured agricultural fields and accomplishing fruit harvesting, where we will design real-time, terrain-aware trajectory optimization for bipedal robot versatile locomotion to cooperate with the soft robotic manipulation and robot perception to achieve agricultural tasks. (3) Deep neural networks (DNNs)-driven robotic perception, where we will adopt, adapt, and optimize DNNs architecture to process the imaging feedback to obtain the berry location and ripeness perceptions for closed-loop control. (4) Integrating soft robot, bipedal locomotion planning and decision-making, and perception to create an innovative agricultural robotic system, and perform evaluations in both laboratory settings and commercial orchards.

Animal Health Component

80%

Research Effort Categories

Basic

(N/A)

Applied

80%

Developmental

20%

Classification

Knowledge Area (KA)	Subject of Investigation (SOI)	Field of Science (FOS)	Percent
404	1129	2020	100%

Knowledge Area
404 - Instrumentation and Control Systems;

Subject Of Investigation
1129 - Berries and cane fruits, general/other;

Field Of Science
2020 - Engineering;

Keywords

agriculture

locomotion

perception

planning

soft robotics

Goals / Objectives
The major goal of this proposal is to design an integrated "Ostrich-like" agricultural robotic system composed of dexterous soft robot, versatile bipedal locomotion planning, and deep-learning-enabled robot perception to achieve fresh market caneberry harvesting in complex agriculture environments. The integrated robotic system will address a series of key technical challenges and create innovative robotic system integration via the following specific objectives:Objective 1: Soft Harvesting Module. Soft robot design, modeling, and control. In this objective, we will first develop a multi-objective optimization-based design strategy to create the soft robotic hardware. We will then investigate soft robot kinematic/dynamic modeling, and propose the robust control algorithm to achieve accurate manipulation for harvesting tasks.Objective 2: Legged Locomotion Planning and Decision-making. Enabling versatile, integrated locomotion and manipulation (i.e., loco-manipulation hereafter) planning over austere agricultural terrain. In particular, we will design a real-time task and motion planning framework for our integrated full-body robotic system to achieve versatile mobility tasks in unstructured agricultural environments.Objective 3: Robot Perception. Deep neural networks (DNNs)-driven robotic perception: adopt, adapt, and optimize. We will utilize and optimize DNNs to process the feedback from imaging sensor(s) for closed-loop robot control. Specifically, the architectures of the DNNs will be redesigned to tackle with the small objects of caneberries at different locations and ripeness levels.Objective 4: Integration and Experimental Validation. We will integrate soft robotic system, bipedal locomotion platform, and perception to create the complete system, and perform evaluations in both laboratory settings and commercial orchards.

Project Methods
The proposed goals can be accomplished via the four research tasks. Specifically, we will perform the following research efforts and evaluation plan during the course of this project:1. Soft Harvesting Module. Efforts: We will perform the multi-objective optimization-based design to find the optimal soft robot. These will involve soft robot simulations, evaluations in the virtual environment, and fitness value calculations. We will then investigate soft robot kinematic/dynamic modeling, and propose the robust control algorithm to achieve accurate manipulation for harvesting tasks. Evaluation: Once the soft robot hardware is obtained, we will perform soft robot control in the benchtop environment. This experiment will be evaluated by the grasping success rate (percent), efficiency, and power consumption. We will also quantify the robot dynamic control performance. When the robot tracks a desired trajectory, we will record the robot tip using an electromagnetic tracker (Aurora, NDI Inc. Tracking accuracy: 0.7 mm) to analyze the control stability, dynamic tracking error, and transient state performance.2. Locomotion unit experiment. Efforts: We will propose a contact-aware task and motion planning (TAMP) with embedded symbolic decisions to form a bilevel optimization for versatile locomotion. In Particular, we will devise a distributed trajectory optimization approach as the motion planner while proposing a scalable decision-making method for long-horizon tasks using the Planning Domain Definition Language method with causal graph decomposition. Evaluation: We will evaluate this planning framework in harvesting tasks such as maneuvering over uneven terrain with weedy and muddy lands, and avoiding trampling over areas with agricultural landscape fabrics. The TAMP algorithm performance will be benchmarked with conventional planning methods. We expect the proposed method to outperform the conventional ones in computational efficiency and numerical convergence rate. We will measure the algorithm performance via three metrics -- locomotion success rate (percent) to evaluate the robustness to real-world terrain variations and uncertainties, real-time planning performance (Hertz) to evaluate the efficacy of the TO method, and locomotion energy efficiency (cost of transport). 3. Perception system development. Efforts: Computer vision models will be studied and developed to detect and localize the target berry fruit for robotic harvesting. Specifically, 1) to tackle with the small size of berries, an eye-to-hand and eye-in-hand dual-camera system will be investigated to coordinate and pass the berry location to the end-effector accurately. 2) Various deep learning models and algorithms will be researched and modified to detect and differentiate the ripeness levels of the target berries, such as ripe, ripening, and unripe berries, for selective harvesting. Evaluation: The performance of the robot perception will be primarily validated in the field conditions, while some experiments will also be conducted in indoor settings to assist the robotic arm/gripper and bipedal locomotion (and integration of full body robot) during the early development phase. Major metrics, such as overall detection accuracy (percent), Precision (P), Recall (R), and mean Average Precision (mAP) under various thresholds (e.g., 0.25-0.75) of Intersection over Union (IoU), will be recorded and utilized to evaluate the performance of robot perception. Additionally, detection speed will also be further improved and recorded based upon the current metric of approximately 17 ms per image.4. Integrated System Evaluation. Efforts: we will create an integrated system that combines all the technical tasks together, and then perform system-level evaluations. Evaluation: We will first characterize the complete system in laboratory settings by harvesting the berries on artificial plants. The integrated robotic system will perform the harvesting task according to the experimental steps detailed in our proposal. We aim to quantify harvesting accuracy (error of gripper position with respect to the berry locations), efficiency (total operation time to harvest all the berries, the time to harvest each berry, and the average number of trials to harvest each berry), and power consumption to complete the harvesting task. Then we will perform end-to-end evaluations in orchards at the local farm within UGA Tifton campus and commercial growers (Ritter Production Farms, Inc.) in Arkansas. Evaluations of the in-field performance will be done on 50 linear row meters (m) of plants to include speed of harvest (number of minutes to complete harvest of ripe fruit), percent accuracy of harvest (number of ripe berries remaining/total number of ripe berries), percent plant damage at harvest (number of damaged canes/total number of canes), and percent of acceptable berries (number of ripe berries/total number of berries in the 1-gallon bucket).

Progress 10/01/23 to 09/30/24

Outputs
Target Audience:Berry industry Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided? There have been graduate and undergraduate students training conducted during this project. How have the results been disseminated to communities of interest?The research results were disseminated to the communities via publications,presentations, andnews report. ? What do you plan to do during the next reporting period to accomplish the goals?The team is in good progress and we will follow the timeline detailed in the proposal to achieve the proposed goals.

Impacts
What was accomplished under these goals? Objective 1 (Lead: PI Yue Chen) Soft robot design, modeling, and control. In Year 1, we have developed a novel tendon-driven soft gripper integrated with ripeness sensing camera for effective harvesting (published paper 1). We also developed the soft gripper mechanics model that can provide reliable berry handling and grasping (under review paper 1). In terms of soft robot, we explored the modular soft robot design that can provide enhanced manipulation (under review paper 2), and soft robotic arm dynamic control (under review paper 3), both contribute to the final field validation of the proposed soft robotic harvesting system. Objective 2 (Lead: co-PI Ye Zhao) Enabling versatile bipedal locomotion over highly unstructured agricultural fields and accomplish fruit harvesting. We designed a locomotion and manipulation planning framework that combines model-based trajectory optimization with reinforcement learning to achieve robust whole-body loco-manipulation. We generated optimal reference motions for the Digit humanoid robot using differential dynamic programming (DDP) and trained reinforcement learning (RL) policies to track these trajectories in dynamic simulations using Mujoco. Domain randomization was employed to reduce the sim-to-real gap. Our results demonstrated that the proposed framework outperforms pure RL methods in both training efficiency and task performance, and we successfully transferred our approach to real-world loco-manipulation tasks. These results lead to an in-preparation paper. Objective 3 (Lead: co-PI Xin Zhang) Deep neural networks (DNNs)-driven robotic perception.During this reporting period, we developed an in-field blackberry detection system using AI-based computer vision technology. We aimed to assess and compare the feasibility, accuracy, and efficiency of a series of state-of-the-art YOLO models in detecting multi-ripeness blackberries in the farm conditions. A total of 1,086 images containing three different ripeness levels of blackberries were collected during the two-year harvesting season, including ripe berries (in black color), berries in the ripening stage (in pink color), and unripe berries (in green color). Eight YOLO models were trained, validated, and tested using randomly selected 809 (74%), 193 (18%), and 84 (8%) images of datasets, respectively. Among all, YOLOv7-x achieved the optimal mean Average Precision (mAP) of 92.6%, F1-score of 86.4%, and inference speed of 12.6 ms per image with 1,024 × 1,024 pixels across all classes of ripeness (published paper 2). Objective 4 (Lead: all PIs) Integration and Experimental Validation. The team has started the system integration work in year 1. We have developed the model that combines the bipedal robot and soft robotic arm. We aim to validate the model in the laboratory settings in year 2.

Publications

• Type: Conference Papers and Presentations Status: Published Year Published: 2023 Citation: Qiu, A., Young, C., Gunderman, A. L., Azizkhani, M., Chen, Y., & Hu, A. P. (2023, May). Tendon-driven soft robotic gripper with integrated ripeness sensing for blackberry harvesting. In 2023 IEEE International Conference on Robotics and Automation (ICRA) (pp. 11831-11837). IEEE.
• Type: Conference Papers and Presentations Status: Published Year Published: 2023 Citation: Zhang, X., Thayananthan, T., Usman, M., Liu, W., & Chen, Y. (2023, June). Multi-ripeness level blackberry detection using YOLOv7 for soft robotic harvesting. In Autonomous Air and Ground Sensing Systems for Agricultural Optimization and Phenotyping VIII (Vol. 12539, pp. 85-96). SPIE. (https://doi.org/10.1117/12.2663367)
• Type: Journal Articles Status: Submitted Year Published: 2024 Citation: Gunderman, A., Wang, Y., Gunderman, B., Qiu, A., Azikhani, M., Sommer, J., and Chen, Y., Kinetostatics and Retention Force Analysis of Soft Robot Grippers with External Tendon Routing, IEEE Robotics and Automation Letters, under review (submitted on Jul 24, 2024).
• Type: Journal Articles Status: Submitted Year Published: 2024 Citation: Cai, Y., Xu, H., Wang, Y., Chen, D., Wojciech, M., Shou, W., and Chen, Y., Modular Self-Reconfigurable Continuum Robot for General Purpose Loco-Manipulation, IEEE Robotics and Automation Letters, under review (submitted on Aug 31, 2024).
• Type: Journal Articles Status: Submitted Year Published: 2024 Citation: Azizkhani, M., Ha, J., Gunderman, A. L., & Chen, Y. Soft Robot Kinematic Control via Manipulability-Aware Redundancy Resolution. ASME Journal of Mechanisms and Robotics, under review (submitted on July 26, 2024).
• Type: Journal Articles Status: Under Review Year Published: 2024 Citation: Thayananthan, T., Zhang, X., McWhirt, A. L., Threlfall, R. T., Liu, W., Huang, Y., Zhao, Y., Gunderman, A. L., & Chen, Y. In-field multi-ripeness blackberry detection for soft robotic harvesting. Journal of the ASABE