Progress 09/01/23 to 08/31/24
Outputs
Target Audience:Specialty crop producers, agricultural equipment manufacturers, technology providers, scientists and scholars, who are seeking new solutions to improve specialty crop production and efficiency, are the target audience of this project. Because of the user-centered nature of this research project, the end-users of developed technologies, as well as fellow researchers and the general public in the Pacific Northwest (PNW) region, are included in our research outcome disseminating group, with frequent communications to get feedback from them. Changes/Problems:
Nothing Reported
What opportunities for training and professional development has the project provided?Three Ph.D. students were actively involved in this project in 2022/2023. In addition, two PhD students who received funding from an NSF Graduate Fellowship were involved in this project. PIs and students interacted frequently to discuss the progress, address challenges, and plan future tasks and activities. Students carried out most of the day-to-day research activities, including data collection and analysis. Students were also supervised for research paper writing, presentations, and publications. In addition, students were provided opportunities to share the results with tree fruit growers, learn from them, and present in extension/outreach events. How have the results been disseminated to communities of interest?The results were presented at local, national, and international meetings and conferences. Project activities and outcomes were also discussed with news outlets (e.g., Good Fruit Growers), which disseminated the information to growers, researchers, and the public in the Pacific Northwest region as well as nationally. One such conference attended and presented at was "The 2024 Annual International Meeting of the American Society of Agricultural and Biological Engineers, Anaheim, CA; July 28-31, 2024". One manuscript has been submitted to a journal for peer review. The soft-robotic manipulator and end-effectors being developed in this project were demonstrated to farmers and technology company representatives in both orchard and laboratory settings on multiple occasions. One such event was the 2024 Technology Open House organized by the Washington State University Center for Precision and Automated Agricultural Systems on July 12, 2024. In addition, our project is featured in an article titled "Robotic gripper for automated apple picking developed" published by WSU Insider What do you plan to do during the next reporting period to accomplish the goals?Obj # 1: Machine vision system More images for flower clusters will be collected and vision system models will be further trained, and evaluated for accurately delineating target flowers Distance estimation module for cluster polygon boundary will be developed and tested to localize the target flowers and obstacles Statistical approaches will be used for making thinning decisions for king flower A global and local camera-based vision system with visual servoing capability will be integrated to minimize the uncertainty of the soft manipulator path planning during robotic thinning of the target flower clusters Overall vision pipeline will be tested in the field Obj # 2: Low-cost, soft growing manipulation, and end-effector System integration will be completed and experiments will be conducted in the lab environment Obj # 3: System integration and evaluation A multi-camera vision system will be integrated with soft arm controller A sensor fusion method will be developed and tested to combine position data from multiple sensors (cameras and encoders) The integrated system will be tested in the orchard environment
Impacts
What was accomplished under these goals?
Obj.1# Develop a fast and precise machine vision system for blossom/flower detection and localization in tree fruit crops 1) Major Activities Data collection and analysis: A set of images of tree canopies were collected using color and RGB-D sensors. Specifically, Intel Real Sense D415 as a Global Camera and D405 as a local camera (to be placed at the end-effector) have been used to collect the images. More than 1000 images have been captured from the field with these sensors with varying distances from the tree canopies. In addition, a dataset was collected in the laboratory with varying distances between the camera and target objects. Point cloud from the Real Sense camera was configured to estimate the accuracy of depth estimation for objects with varying distances from the camera. This experiment will be essential to evaluate the model performance in detecting and localizing objects for visual servoing. Deep learning model development: Acquired images were manually annotated to create the dataset for training and testing deep learning models for object detection and localization. Multiple deep learning models were trained and evaluated to detect target objects such as flowers for robotic blossom thinning application. Specifically, YOLOv7, YOLOv8 and Mask R-CNN models were trained and tested. Evaluating impact of soft manipulator on machine vision system: An experiment was performed to evaluate if the images acquired with a camera mounted at the tip of a soft manipulator will rotate around the access of extension with increasing extension length. Distance Estimation: Using a stereo vision-based depth map, the region of interest encapsulated by the bounding boxes of detected flowers and other objects was used to estimate the distance (Z value) to the center of the target objects, which was then used to estimate 3D coordinates of the target objects. This information was then given to the controller of the soft manipulator to perform thinning. Multi-Camera Visual Servoing: After obtaining the 3D position of the target objects, two vision systems are used to determine the 3D position of the tip mount and its relative position to the targeted objects. Currently, the tip positioning system uses a global camera that locates a QR code identifier attached to the tip mount while the relative object positioning system utilizes both local and global cameras. Using the global and local vision systems, visual servoing was designed and tested in a lab environment to improve the accuracy of reaching the target objects despite uncertainty in the manipulation accuracy of the soft manipulator. 2) Key Outcomes ? Object detection accuracy with YOLOv7 was good when the target object was between 30 cm to 1.2 m, which decreased as the object of interest was drawn closer to the camera. When objects such as flowers and apples were at a distance < 20cm from the cameras, the objects covered most of the image space, and the bounding boxes detected did not encapsulate the entire objects. The detection accuracy was significantly improved with the YOLOv8 model compared to the same with YOLOv7. ? The latest object detection model with YOLOv8 achieved an accuracy of 98% in detecting apples in commercial orchards, which will be expanded to flower detection in the future. ? It was found that depth estimation with the Real Sense camera was accurate for objects farther than 0.9 m from the camera. This range is good for our application since the canopy objects from the global camera (which is used for depth estimation) will be around 1.0 to 1.2 m. ? Two camera-based system for real-time positioning and control of soft manipulator showed promise for desired accuracy and applicability in orchard environments. Obj #2: Design and implement low-cost, soft growing manipulation, and an end-effector for flower thinning. 1) Major activities completed / experiments conducted. A functional prototype of the soft-growing manipulator designed for orchard operations has been developed and tested. The current platform is low-cost but is robust enough to handle the orchard environment. The robotic platform is composed of four main components: the fabric arm, pressurized enclosure, steering system, and end-effector mount. The fabric arm is made of a lightweight thermoplastic polyurethane-coated fabric that is heat-sealed into a tube-like shape. The arm features a 1.2 m length and 3.2 in diameter, which are sufficient for reaching trees in a commercial orchard and supporting an end-effector tool. The fabric arms can reliably operate at pressures below 10 psi and have a maximum pressure of 18 psi, which has been experimentally verified. A model reference adaptive controller (MRAC) was implemented for soft manipulator control, which significantly improved the system's performance by making all system parameters follow a desired behavior, resulting in minimal distance covered and all parameters converging simultaneously. The MRAC allowed for consistent control behavior regardless of additional payloads. 2) Summary Statistics and Discussion of Results. ? The manipulator reached a point near the edge of its workspace from its default position with a target rise time of 1.28 seconds and a settling time of 3.30 seconds with less than 0.04 inch of steady-state error. ? The pressurized enclosure was designed to withstand 20 psi, which has been experimentally verified. The enclosure also housed a central pulley and motor that controlled the arm's length. This motor allowed for a linear maximum retraction speed of 0.86 ft/s at 3 psi and an extension speed of 1.23 ft/s at 8 psi. ? The system control reliably compensated for additional payloads up to 1.21 lbs. without significant impact to the system's behavior. ? The end-effector mount was composed of two shells, one interior, and one exterior, that interact via roller magnets through the fabric arm. The magnets provided a strong mounting force that took 6 lbf to separate. The end-effector mount was also lightweight due to being made from 3D-printed plastic at 9.7 oz. ? A steering system was developed for the manipulator consisting of three DC motors connected to pulleys that pull on the base of the fabric arm in order to orient the fabric in the desired direction. The motors were arranged 120 degrees apart on the front of the pressurized enclosure. 3) Key Outcomes ? The soft manipulator was capable of automatically reaching any target or following a defined trajectory within its workspace with decent performance in terms of accuracy, payload, and speed. ? Our soft manipulator design resulted in a maximum payload of 0.99 lbf at 10 psi while at a 4 ft arm length, and an approximated cost of $4,230 (cost of materials and fabrication in a shop). These results indicated that the manipulator will be practically applicable and commercially viable for orchard applications. ? The lightweight flower-thinner tool was composed of a 3D-printed plastic frame, a small high-power brushless DC motor, and a plastic weed-whacking tool. This design made the tool very lightweight at only 3.1 oz in total. But the high-power motor also allowed for a maximum rotational speed of 21,000 RPM. Thus, the thinning tool was expected to reach speeds desired for thinning out selected flowers. Obj # 3: Integrate thinning end-effector and vision system with soft manipulator, and evaluate the thinning robot in commercial orchards 1) Major activities completed / experiments conducted. Integration of the soft robot, vision system, and end effector and testing for flower thinning is ongoing in a lab environment. We plan to submit the relevant publication by next March. We have also set up the soft robot system in a commercial orchard and conducted initial testing through teleoperation. The system will be ready for the next flower thinning season. 3) Key outcomes Our system is ready for the flower-thinning experiments in a commercial orchard for the upcoming season.
Publications
- Type:
Peer Reviewed Journal Articles
Status:
Published
Year Published:
2024
Citation:
Bhattarai, U., Bhusal, S., Zhang, Q., & Karkee, M. (2024). AgRegNet: A deep regression network for flower and fruit density estimation, localization, and counting in orchards. Computers and Electronics in Agriculture, 227, 109534.
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2024
Citation:
Ninatanta, C., Cole, R., Wells, I., Ramos, A., Pilgrim, J., Benedict, J., Taylor, R., Dorosh, R., Yoshida, K., Karkee, M. & Luo, M., 2024, April. Design and evaluation of a lightweight soft electrical apple harvesting gripper. In 2024 IEEE 7th International Conference on Soft Robotics (RoboSoft) (pp. 479-484). IEEE.
|
Progress 09/01/22 to 08/31/23
Outputs
Target Audience:Specialty crop producers, agricultural equipment manufacturers, technology providers, scientists and scholars, who are seeking new solutions to improve specialty crop production and efficiency, are the target audience of this project. Because of the user-centered nature of this research project, the end-users of developed technologies, as well as fellow researchers and general public in Pacific Northwest (PNW) region, are included in our research outcome disseminating group, with frequent communications to get feedback from them. Changes/Problems:
Nothing Reported
What opportunities for training and professional development has the project provided?Three Ph.D. students were actively involved in this project in 2022/2023. PIs and students interacted frequently to discuss the progress, address challenges, and plan future tasks and activities. Students carried out most of the day-to-day research activities, including data collection and analysis. Students were also supervised for research paper writing, presentation, and publications. In addition, students were provided opportunity to share the results with tree fruit growers, learn from them and present in extension/outreach events. How have the results been disseminated to communities of interest?The results were presented at local, national, and international meetings and conferences. Project activities and outcomes were also discussed with news outlets (e.g., Good Fruit Growers), which disseminated the information to growers, researchers, and the public in the Pacific Northwest region as well as nationally. One such conference attended and presented at was "The IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS): DETROIT, MI; Oct 5-9, 2023". One manuscript has been submitted to a journal for peer review. The soft-robotic manipulator and end-effectors being developed in this project were demonstrated to farmers and technology company representatives in both orchard and laboratory (Washington State University Center for Precision and Automated Agricultural Systems) settings on multiple occasions. What do you plan to do during the next reporting period to accomplish the goals?Obj #1: Machine vision system · More images will be collected and vision system models will be further trained, and evaluated for accurately delineating target flowers. · Distance estimation module will be developed and tested to localize the target flowers and obstacles. · A global and local camera-based vision system will be integrated to minimize the uncertainty of the soft manipulator in reaching target flowers for thinning. · Overall vision pipeline will be tested in the field. Obj # 2: Low-cost, soft growing manipulation, and end-effector · Improve the maximum pressure of the fabric arm, and design an optimal control system to manipulate the arm to reach targets. · Finalize an updated design of the manipulator arm to address issues with manufacturing and reliability. · Improve the design of the flower thinning end-effector by adding a lightweight servo to allow the system to approach selected flowers from ideal angles. Obj # 3: System integration and evaluation · Integrate the vision system and end-effector mount with the soft manipulator. · Minimize and compensate for the spatial positioning error of the integrated system. · Evaluate the integrated system in the lab and the field in varying lighting conditions.
Impacts
What was accomplished under these goals?
Key issues being addressed: Labor availability and cost is a critical challenge tree fruit growers are facing in Washington State and around the country as production operations of these crops such as pruning, flower, and fruitlet thinning and pollination are highly labor intensive and laborious. These operations also pose worker health and safety issues such as exposure to extreme heat and cold weather conditions, ladder fall and repetitive motion injury, and pesticide exposure. Robotic technologies have been sought as an alternative to minimize the dependence on manual labor and improve worker health and safety. However, the current generation of robotic solutions to these labor-intensive operations is limited by high complexity and cost, and slow speed. This research focused on building a simple, reliable, robust yet cost-effective system for automated blossom thinning for tree fruit crops using soft robotics-based manipulation. Obj #1: Develop a fast and precise machine vision system for blossom/flower detection andlocalization in tree fruit crops 1) Major Activities Data collection and analysis: A set of images of tree canopies were collected using color and RGB-D sensors. Specifically, Intel Real Sense D415 as a Global Camera and D405 as a local camera (to be placed at the end-effector) have been used to collect the images. To integrate these cameras and facilitate image acquisition with the overall robotic system, their respective dependencies and Robot Operating System (ROS) packages have been configured. More than 1000 images have been captured from the field with these sensors with varying distances from the tree canopies. In addition, a dataset was collected in the laboratory with varying distances between the camera and a target object. Point cloud from the Real Sense camera was configured to estimate the accuracy of depth estimation for objects with varying distances from the camera. This analysis will be essential to evaluate the model performance in detecting and localizing objects for visual servoing. Deep learning model development: Acquired images were manually annotated to create the dataset for training and testing deep learning models for object detection and localization. Multiple deep learning models were trained and evaluated to detect target objects such as flowers for robotic blossom thinning application. Specifically, YOLOv7, YOLOv8 and Mask R-CNN models were trained and tested. Evaluating impact of soft manipulator on machine vision system: An experiment was performed to evaluate if the images will rotate around the access of extension of the soft manipulator with increasing extension length. 2) Key outcomes When using YOLOv7, the accuracy of the model decreased as the object of interest was drawn closer to the camera as shown below. When objects at a distance < 20cm from the cameras, the objects such as an apple covered most of the area in these images, and the bounding boxes detected did not encapsulate the entire object. In such a situation, the centroid of object could not be determined accurately. The detection accuracy was significantly improved with the use of YOLOv8 compared to the same with YOLOv7. It was found that depth estimation with the Real Sense camera was accurate for objects farther than 0.9 m from the camera. This range is good for our application since the canopy object from the global camera (which is used for depth estimation) will be around 1.0 to 1.2 m. The unintended image rotation with increasing length of the soft manipulator during extension was found to be negligible over the shorter extension. It reached up to 10 degrees with the full extension of the manipulator (which was around 4 ft), which may have to be accounted for to track a target object during the visual servoing. Obj #2: Design and implement low-cost, soft growing manipulation, and an end-effector forflower thinning. 1) Major activities completed / experiments conducted. A functional prototype of the soft-growing manipulator designed for orchard operations has been developed and tested. The current platform is low-cost but is robust enough to handle the orchard environment. The robotic platform is composed of four main components: the fabric arm, pressurized enclosure, steering system, and end-effector mount. The fabric arm is made of a lightweight thermoplastic polyurethane-coated fabric that is heat-sealed into a tube-like shape. The arm features a 4 ft length and 3.2 in diameter, which are sufficient for reaching trees in a commercial orchard and supporting an end-effector tool. The fabric arms can reliably operate at pressures below 10 psi and have a maximum pressure of 18 psi, which has been experimentally verified. 2) Summary statistics and discussion of results. The pressurized enclosure is designed to withstand 20 psi, which has been experimentally verified. The enclosure also houses a central pulley and motor that control the arm's length. This motor allows for a linear maximum retraction speed of 0.82 ft/s at 3 psi and an extension speed of 0.89 ft/s at 8 psi. The end-effector mount is composed of two shells, one interior, and one exterior, that interact via roller magnets through the fabric arm. The magnets provide a strong mounting force that takes 6 lbf to separate. The end-effector mount is also lightweight due to being made from 3D-printed plastic at 9.7 oz. A steering system was developed for the manipulator consisting of three DC motors connected to pulleys that pull on the base of the fabric arm in order to orient the fabric in the desired direction. The motors are arranged 120 degrees apart on the front of the pressurized enclosure. 3) Key outcomes Our soft manipulator design resulted in a maximum payload of 0.99 lbf at 10 psi while at a 4 ft arm length, and an approximated cost of $4,230 (cost of materials and fabrication in a shop). These results indicated that the manipulator will be practically applicable and commercially viable for orchard applications. The lightweight flower-thinner tool is composed of a 3D-printed plastic frame, a small high-power brushless DC motor, and a plastic weed-whacking tool. This design makes the tool very lightweight at only 3.1 oz in total. But the high-power motor also allows for a maximum rotational speed of 21,000 RPM. Thus, the thinning tool is expected to reach speeds desired for thinning out selected flowers. It was found that it is efficient to use only two of the steering motors to actuate at a time while the remaining motor holding its initial position. With this system, the fabric arm could steer up to 60 degrees in a single direction. This configuration resulted in a spherical sector-shaped workspace with a radius of 4 ft and 60 degrees of actuation in the 2D plane, which is crucial in improving the applicability of this system for orchard operation. Obj # 3: Integrate thinning end-effector and vision system with soft manipulator, and evaluate the thinning robot in commercial orchards Since the machine vision system and mechatronic component of the project are still under development, no research activities were carried out this year on overall system integration and evaluation.
Publications
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2023
Citation:
Dorosh, R., Allen J., He, Z., Ninatanta, C., Coleman, J., Spieker, J., Tuck, E., Kurtz, J., Zhang, Q., Whiting, M., Luo, J., Karkee, M., & Luo, M (2023). Design, Modeling, and Control of a Low-Cost and Rapid Response Soft-Growing Manipulator for Orchard Operations, in The IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS): DETROIT, MI.
|