Progress 02/01/21 to 01/31/24
Outputs
Target Audience:Overall, the targeted audiences for the project duration were 1) chile pepper stakeholders, including chile pepper growers and distributors, local farmers, agriculturalresearchers, agricultural industries/companies, 2) researchers in robotics and intelligent systems, and 3) agricultural sciences and engineering students. 1) Although we are still in the early stages of developing chili cultivars for robotics, breeding efforts were conveyed to the NM Chile Commission during Dr. Walker's annual researchupdates. Dr. Walker is the co-PI on this project. 2) Dr. Haghshenas (PI) and Luke Garcia (graduate student) presented the research findingsrelated to Objective 1 and Objective 2, respectively, at the 2023 Chile Pepper Conference. Umme Kawsar Alam (graduate student) also presented the latest research findingsrelated to Objectives 1-4at the 2024 Chile Pepper Conference. The audiences were chile pepper growers anddistributors, local farmers, agricultural researchers, agricultural industries/companies, and students. 3) Jordan Linford (graduatestudent) presented the research on mobile manipulator path planning and the study of the workspace of a floating-basedrobotic arm at the 2023 IEEE/SICE International Symposium on System Integration (SII). Moreover, Dr. Haghshenas-Jaryani (PI) presented his research on softrobotic arms for harvesting tasks at the 2023 IEEE International Conference on Robotics and Biomimetics (ROBIO2023), where the audiences were researchers in robotics,AI, and intelligent machines. Changes/Problems:
Nothing Reported
What opportunities for training and professional development has the project provided?Overall, sixgraduate and two undergraduate students hadthe opportunity to work on this project and get the training for learning new skills related to robotics, design and prototyping, machine learning and computer vision, testing, data collection, statistical analyses, andfarming Chile peppers. Last project year, fourgraduate students and one undergraduate student were involved in the research project, learning about robotics, visualtracking robot arm motion planning, designing and developing a human-like harvesting gripper, and deep-learning fruit detection.The students also had an opportunity toreceive professional development for doing research, data collection, scientific manuscript preparation, thesispreparation,attending conferences, and presenting at the conference. How have the results been disseminated to communities of interest?Our research findings on robotic systems, motion planning,and automated fruit detection were published and presented in multiple conferencesand journal papers (please see the products) related to agricultural robotics and automation andthe Chile pepper conference. The PI, co-PIs, and students presented the findings and outcomes at different seminars. Videos of harvesting trials have been posted on the PI's research website and the publicly accessible YouTube channel. What do you plan to do during the next reporting period to accomplish the goals?
Nothing Reported
Impacts
What was accomplished under these goals?
This project accomplished the developmentand a feasibility study oftherobotic harvesting of chile peppers in a lab setting, employinga 6-DOF robotic arm with a scissor-type cutting end-effector.The system utilizes a machine learning-based computer visionand a depth camera to detect and localize chile peppers inthe camera frame. Green chile peppers are detected through a deeplearning method and localized by utilizing a camera's depth and RGBcolor images. The locations are then transformed into therobot's operational frame. A motion planning algorithm wasdeveloped to minimize the robot's travel time for harvesting.Themotion planning algorithm in Cartesian space has been developed to guide the cutter in transitioning from its startingpoint to each detected chile's location for the harvesting task.The algorithm is based on finding the minimum path betweenthe starting point of the cutter and all detected/localizedchile peppers in the plant. A correction equation is derived to address inaccuracies incamera-based localization while eliminating unreachable chiles for the robot. From a dataset of 86 chile peppers, thestudy reports key harvesting metrics: a detection success rateof 62.8%, a localization success rate of 90.74%, a detachmentsuccess rate of 55.10%, a harvest success rate of 31.39%, anda damage rate of 6.97%. We also designed and prototyped a seriesof human-like harvesting grippers for performing fruit detachment and destemming. The preliminary results show promise for utilizing the customized gripper for integration intotherobotic arm for fully automated harvesting tasks. The research project's outcomes haveshownthe potential of using robotic arms with integrated visual fruit detection for harvesting green chile peppers. Successful implementation of these developed technologies in actual farms would help green chile pepper farmers and growers address the significantchallenge of labor shortage, reduce the cost of production, andincrease the plantation acreage and final yield.?
Publications
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2024
Citation:
M. Haghshenas-Jaryani, "Timoshenko Beam-based Analytical Formulation and Numerical Simulation of Continuum Soft-bodied Robotic Arms," 2023 IEEE International Conference on Robotics and Biomimetics (ROBIO), Koh Samui, Thailand, 2023, pp. 1-6, doi: 10.1109/ROBIO58561.2023.10354597.
- Type:
Conference Papers and Presentations
Status:
Accepted
Year Published:
2024
Citation:
UK. Alam, L. Garcia, J. Grajeda, M. Haghshenas-Jaryani, and L.E. Boucheron, Automated Harvesting of Green Chile Peppers with a Deep Learning-based Vision-enabled Robotic Arm", the 2024 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM), July 15-18, 2024, Boston, USA
- Type:
Conference Papers and Presentations
Status:
Other
Year Published:
2024
Citation:
U.K. Alam and M. Haghshenas-Jaryani "Using a Robotics Arm with an Integrated Deep Learning?based Computer Vision for Detection and Harvesting Green Chile Pepper , Mechanization Technology Session, 2024 NM Chile Pepper Conference, NMSU, February 6th, 2024, Las Cruces, NM
|
Progress 02/01/22 to 01/31/23
Outputs
Target Audience:Audiences were chile pepper stakeholders, including chile pepper growers and distributors, local farmers, agricultural researchers, agricultural industries/companies, and students. Although we are in the early stages of chile cultivar development for robotics, breeding efforts were conveyed to the NM Chile Commission during Walker's annual research update, January 31, 2022. Dr. Haghshenas (PI) and Luke Garcia (graduate student) presented the new research findings related to Objective 1 and Objective 2 at Chile Pepper Conference. The audiences were chile pepper growers and distributors, local farmers, agricultural researchers, agricultural industries/companies, and students. Jordan Linford (graduate student) presented his research on mobile manipulator path planning and the study of the workspace of a floating-based robotic arm at the 2023 IEEE/SICE International Symposium on System Integration (SII), where the audiences were robotics, AI, and intelligent machines, researchers. Changes/Problems:
Nothing Reported
What opportunities for training and professional development has the project provided?Under objective 1, one graduate student has been involved in the robotic arm objectives/tasks over the previous year. He received training to develop robot motion planning and control for agricultural tasks. The student used this work as his MSME thesis research and successfully defended his MSME (Robotics) in November 2022. The student also had an opportunity to receive professional development for doing research, data collection, scientific manuscript preparation, thesis preparation, attending conferences, and presenting at the conference. Objective 2 has supported one graduate student in electrical engineering. That student used this work as his MSEE (Robotics) thesis research and successfully defended his MSEE in April 2022. That student has accepted a job at Sandia National Labs with very positive feedback on his training from the hiring manager. For objective 3, three undergraduate students have been trained in managing chile plants in the greenhouse and performing controlled cross-pollination to initiate breeding lines. How have the results been disseminated to communities of interest?Our research findings on robotic systems and automated fruit detection were published in conference proceedings, journal papers, and master theses. Additionally, Dr. Haghshenas (PI) and Luke Garcia (graduate student) presented the new research findings related to Objectives 1 and 2 at Chile Pepper Conference. Although we are in the early stages of chile cultivar development for robotics, breeding efforts were conveyed to the NM Chile Commission during Walker's annual research update in January 2023. Jordan Linford (graduate student) presented his research on mobile manipulator path planning and the study of the workspace of a floating-based robotic arm at the 2023 IEEE/SICE International Symposium on System Integration (SII). What do you plan to do during the next reporting period to accomplish the goals?Finalizing the project by 1) the outcomes of the objective 1 and objective 2 will be integrated; therefore, we will do on-line fruit detection, motion planning, and harvesting. 2) human-like harvesting using a soft robotic gripper will be investigated to simultaneously achieve the harvesting and destemming. A series of soft grippers will be designed and prototyped while evaluated for desired harvesting performance, 3) evaluation of the integrated fruit detection/robotic harvesting system;a comprehensive experimental study will be carried out to determine the effectiveness of the fruit detection algorithm, robot manipulation, human-like harvesting attempt, 4)we will investigate the harvesting of the efficient mechanized harvesting breed investigated under (Objective 3) to evaluate the effect of this harvesting efficiency using robotic arms.
Impacts
What was accomplished under these goals?
Objective 1: 1) A preliminary study of the 6-DoF robotic arm for harvesting chile pepper in the lab with the cutter end-effector has been carried out; we developed an early-stage motion planning code for moving the robot using predefined locations of the chile peppers; 2) A feasibility study of using mobile manipulators, including a wheeled rover with an attached 6 DOF arm and a tracked-wheeled rover with an attached flipper and 4 DOF robotic arm, was carried out. Additionally, their workspace was initially analyzed, resulting in a path-planning approach based on integrating a floating base; the proposed approach was developed and studied to achieve the highest dexterity of the mobile manipulator on the field for agricultural tasks such as harvesting. Objective 2: Objective 2 was focused on transitioning the automated detection of green chile to a form that can be integrated with the robotic arm in Objective 1 and validating performance in that new scenario. Detection of green chile in images can output an (x,y,z) location of the estimated stem location of the chile as input to the motion planning algorithm from Objective 1. Objective 3: Chile paper pods were planted and cultivated for image capturing and experimental testing in the lab. Objective 4: A series of testing and validation of the automated green chile pepper detection has been carried out. A success rate of about 75% in the detection of fruit and stem has been achieved. The results show that the developed fruit-detection machine learning algorithm can also distinguish and detect stems from fruits. Experiments have been conducted using the 6-DOF robotic arm and its associated motion planning in harvesting chile pepper in the lab for predefined locations. We achieved a 70% success rate in harvesting chile pepper without any fruit damage.
Publications
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2023
Citation:
Linford, J., & Haghshenas-Jaryani, M. (2023, January). Workspace Study of Floating-Base Ground Mobile Manipulator for Soil Moisture Monitoring in NM-Type Green Chile Pepper Farming. In 2023 IEEE/SICE International Symposium on System Integration (SII) (pp. 1-6). IEEE.
- Type:
Journal Articles
Status:
Published
Year Published:
2022
Citation:
Lozada, D. N., Bosland, P. W., Barchenger, D. W., Haghshenas-Jaryani, M., Sanogo, S., & Walker, S. (2022). Chile pepper (Capsicum) breeding and improvement in the multi-omics era. Frontiers in Plant Science, 13.
- Type:
Theses/Dissertations
Status:
Published
Year Published:
2022
Citation:
Garcia, L. (2022). Automated Detection of Green Chile from RGB Images Using Deep Learning (Master Thesis, New Mexico State University).
- Type:
Theses/Dissertations
Status:
Published
Year Published:
2022
Citation:
Linford, J. (2022). Use of Robotic Manipulator for Monitoring Crop Health and Soil Moisture (Master Thesis, New Mexico State University).
|
Progress 02/01/21 to 01/31/22
Outputs
Target Audience:Audiences were chile pepper stakeholders including, chile pepper growers and distributers, local farmers, agricultural researchers, agricultural industries/companies, and students. Although we are in the early stages of chile cultivar development for robotics, breeding efforts were conveyed to the NM Chile Commission during Walker's annual research update, January 31, 2022. Changes/Problems:We have decided to break down the robotic harvesting task into more simple steps, including, harvesting using a cutter end-effector which helped us to focused on the motion planning, workspace analysis of the robotic arms with a variety of degrees-of-freedom as well as studying the fruit geometrical aspects (shape, size, etc.) and plant fruit distributions. The outcomes of these studies would help us in the continuation of the project to achieve the human-like harvesting of chile peppers in the current year. What opportunities for training and professional development has the project provided?Under objective 1, two graduate students and one undergraduate student have been involved in the robotic arm objectives/tasks over the previous year. They received training in the design and development of robotic components for the harvesting tasks. The Ph.D. student had an opportunity to receive professional development for doing research, data collection, and scientific manuscript preparation. Objective 2 has supported one graduate student in electrical engineering. That student used this work as his MSEE thesis research and successfully defended his MSEE in April 2022. That student has accepted a job at Sandia National Labs with very positive feedback on his training from the hiring manager. For objective 3, three undergraduate students have been trained in the management of chile plants in the greenhouse and how to perform controlled cross-pollination to initiate breeding lines. How have the results been disseminated to communities of interest?Our feasibility studies on the use of robotic arms for harvesting were published in the journal of Robotics (Masood and Haghshenas-Jaryani, Robotics 2021, 10(3), 94. https://doi.org/10.3390/robotics10030094, Special issue: Advances in Agriculture and Forest Robotics). Additionally, the outcomes of objectives were presented by the research teams (PI and co-PIs) in the NMSU Chile Pepper Conference 2022 with audiences were chile pepper stakeholders including, chile pepper growers and distributers, local farmers, agricultural researchers, agricultural industries/companies, and students. Although we are in the early stages of chile cultivar development for robotics, breeding efforts were conveyed to the NM Chile Commission during Walker's annual research update, January 31, 2022. What do you plan to do during the next reporting period to accomplish the goals?Objective 1: 1) We will focus on the study of a 6-DoF robotic arm for harvesting chile pepper in the lab with the cutter end-effector, using two cameras (one fixed to the worktable and one attached to the wrist of the robotic arm), research on visual-tracking motion planningand time/energy optimized task-planning. 2) human-like harvesting using a soft robotic gripper will be investigated to simultaneously achieve the harvesting and destemming. A series of soft grippers will be designed and prototyped while evaluated for desired harvesting performance. Objective 2: Efforts in the next reporting period for Objective 2 will focus on transitioning the automated detection of green chile to a form that can be integrated with the robotic arm in Objective 1 and validating performance in that new scenario. Detection of green chile in images can output an (x,y,z) location of the estimated stem location of the chile as input to the motion planning algorithm from Objective 1. Objective 3: The F1 population from the NM type green chile lines and 'Yatsufusa' crosses will be increased in the greenhouse, and the highly segregating F2 and F3 populations will be grown in the field. From the segregating population, single plant selections will be made of the best plants exhibiting both NM type fruit characteristics and determinate habit with minimal leaf cover over the fruit. Inbreeding and backcrossing will be initiated as needed with the long-term goal of developing new chile cultivars with idea architecture for visioning and harvesting with robotic technology. Objective 4: We will integrate the machine learning-based fruit identification algorithms and localization (Objective 2) into the robotic arm system (Objective 1) to enable the robot autonomously detects the fruit, plans for harvesting, and finally performs the task in the laboratory setting. We will perform a series of testing for evaluation of the integrated fruit detection/robotic harvesting system. Additionally, we will investigate the harvesting of the efficient mechanized harvesting breed investigated under (Objective 3) to evaluate the effect of this harvesting efficiency using robotic arms.
Impacts
What was accomplished under these goals?
Objective 1: A series of feasibility studies using of robotic systems for chile pepper farming are being implemented. For example, in preliminary work on the robotic harvesting for chile pepper, a 5-degrees-of-freedom (DoF) and a 6-DoF robotic arms with a customized cutter end-effector have been investigated in a laboratory setting (Masood and Haghshenas-Jaryani, Robotics 2021, 10(3), 94. https://doi.org/10.3390/robotics10030094). The end-effector of the manipulator, a scissor-type cutting mechanism, was designed, prototyped, and experimentally tested in a lab setup that cuts the chile stem to detach the fruit. A simple and easy to develop 3-D location estimation system was developed with a considerably minimal mean error which can be efficiently used for various research application which involve data logging. Through a MATLAB-based program, the location of the chile pepper is estimated in the robot's reference frame using Intel RealSense Depth Camera. The accuracy of the 3D location estimation system matches the maximum accuracy claimed by the manufacturer of the hardware. The forward and inverse kinematics are developed, and the reachable and dexterous workspaces of the robot are studied. An application-based path planning algorithm is developed to minimize the travel for a specified harvesting task. The robotic harvesting system was able to cut the chile pepper from the plant based on 3D location estimated by the MATLAB program. The harvesting robot showed promising results with high localization, detachment, and harvest success rates, low damage rate, and a cycle time comparable to the performance of other harvesting robots and human harvesters. The overall results have demonstrated the feasibility of using robotics approaches in harvesting chile pepper. Additionally, a mobile robotic arm was developed by assembling a 6-DOF robotic arm to a rover robot for navigation and harvesting in the field. An R&D payload was designed for the the mobile manipulator which encompasses all of the equipment required for navigation and control of mobile manipulator in the field including, a computer, a robotic arm DC controller, batteries, an Arduino board, Lidar, stereo cameras, and cables. The prototype was tested for navigation and general operation (no harvesting at this stage) in the field which was successful. Objective 2: A dataset of green chile images has been acquired in environmental conditions expected to mimic the ultimate harvesting conditions, namely images taken outdoors in natural sunlight. Those images were acquired at varying times of day, and at varying distances and angles to the plant. This image acquisition setup allows for acquisition of images that span a range of expected appearances. A subset of those images (285 images) has been annotated manually by delineating regions corresponding to chile and chile stems (a total of some 2500 annotations). These annotated images are critical in training a network to recognize green chile in images. The annotated images were augmented via various transformations (e.g., illumination changes, rotation, scaling) to create a larger dataset to span an even wider range of appearances, resulting in 3990 augmented images available to train. This augmentation is necessary since deep learning networks require large datasets to train since they have such a large number of parameters. The augmented dataset was randomly split 80%/20% into training and testing data. The training data were used to transfer learn a Mask-RCNN network with both the ResNet-50 and ResNet-101 backbones to find and delineate (via an object mask) chile and chile stems in images. The Mask-RCNN with the ResNet-101 backbone outperformed the ResNet50 backbone, indicating that the deeper features in ResNet-101 are more useful for green chile detection. The network was initialized to weights trained on the very large MSCOCO dataset and then allowed to adapt to the green chile dataset. Additionally, the anchor scales and ratios of the bounding boxes (hyperparameters that determine the expected size and shape of objects in an image in the Mask-RCNN formulation) were modified for the green chile dataset to improve detection performance. The best performing network yielded a precision of 75%, a recall of 56%, a mAP score of 79% and an F1 score of 65%. These results are on-par or slightly lower than other automated methods for fruit detection, although the detection of green chile is more challenging than other fruit detection tasks due to the similarity in color. It is the size of the training dataset that is believed to be the limiting factor in improving performance. The method, as currently implemented, can operate at a frame rate of approximately 1.5 frames per second, but could be potentially sped up with code efficiencies or implementation on more powerful hardware. The method has also been demonstrated on a completely separate video acquired from the robotic arm from Objective 1 with encouraging results, despite significant differences in lighting and image resolution. Objective 3: Objective 3 involves investigates NM type green chile (Capsicum annuum) cultivars that will work in a robotic harvesting system. Abundant past research has been conducted towards the mechanical harvest of NM type green chile peppers. These efforts identified a system that includes use of an inclined double helix picking head and resulted in the release of a new cultivar, 'NuMex Odyssey,' that provided the best harvest performance in this purely mechanical system. However, a serious deficiency noted with this system was the lack of pedicel removal from the fruit which must be accomplished for NM type green chile destined for commercial processing. Robotic harvesting provides the potential for gentle harvest of the green chile fruit while leaving the pedicel on the plant, similar to hand harvest by human laborers. A challenge for robotic harvesting of NM type green chile is the abundant leaf cover over the fruit that blocks visioning and harvest of individual fruit. While all NM type green chile cultivars exhibit this leafy cover, other Capsicum annuum cultivars exhibit highly determinate plant architecture with erect fruit that are devoid of leaf cover. In the initial stages of this project, we've made controlled cross-pollinations in the greenhouse between several NM type chile cultivars and 'Yatsufusa,' a highly determinate cultivar with minimal leaf cover over the fruit. Objective 4: Nothing has been done under this objective in the first year.
Publications
- Type:
Journal Articles
Status:
Published
Year Published:
2021
Citation:
Masood, M.U.; Haghshenas-Jaryani, M. "A Study on the Feasibility of Robotic Harvesting for Chile Pepper". Robotics 2021, 10(3), 94. https://doi.org/10.3390/robotics10030094
|
|