Progress 07/01/20 to 06/30/24
Outputs
Target Audience:Over the project, the target audiences mainly include professional society, grower society, and other audiences: 1. Professional society: 1) The research results on green fruit thinning end-effector development as well as the machine vision system for green fruit detection were presented at 2021 ASABE annual meeting (July 12-16, 2021); 2) The research results on robotic green fruit thinning system were presented by PIs and graduate students at 2022 ASABE annual meeting (July 12-16, 2022) and 2022 NABEC annual meeting (July 31 - August 3, 2022); 3) The research outcomes from this project were presented as partial results of specialty crop production in 2022 FIRA robotics conference (October 18-22, 2022), and AI for Ag conference (April 17-19, 2023); 4) Invited seminars to present work related to apple crop load management including green fruit thinning at University of Pennsylvania (4/19/2023) and Tennessee State University (June 13, 2023); 5) Project PIs also presented the outcomes from the project at W3009 multi-state project annual meetings (2021- 2023); 6) The research results on green fruit sizing was presented in 2024 NABEC meeting (July 15-17, 2024). 2. Grower society: 1) The research results on green fruit detection and orientation were presented at 2022 Mid-Atlantic Fruit and Vegetable Convention (~150 participants); 2) The introduction of robotic system for crop load management was presented at 2021 Penn State Extension Winter Fruit School (~100 participants); 3) The robotic green fruit thinning demonstration was presented to State Horticultural Association of Pennsylvania (SHAP) research committee (12 participants); 4) Project results and system demonstration were presented to 2022 International Fruit Tree Association Tour (~120 participants); 5) Present the research outcomes at Penn State Winter Fruit Schools in six different locations (February - March 2023, in total > 200 participants); 6) Demonstrate the robotic green fruit thinning system at 2022 FREC field day (July 2022, ~ 120 participants); 7) Demonstrate and present at 2022 Penn State Ag Progress Day; 8) Presented and demonstrated the system at 2023 FREC Precision Ag Field Day (June, 2023); 9) Presented the research outcome of robotic green fruit thinning system in 2024 Mid-Atlantic Convention (January 31, 2024). 3. Other audiences: 1) The developed green fruit thinning system was presented to the 2021 Legislative tour (~70 participants); 2) Presented green fruit thinning project outcomes at 2022 June College Connection webinar (~80 participants); 3) Presented the related research outcomes at the 2022 Penn State Ag Council Delegate Meeting (~30 participants); 4) Presented the outcomes from this project as partial materials in 2023 Emerging and Advanced Technology Initiative (~25 participants); 5) Presented the outcomes from this project as partial materials in 2024 Penn State Technologies for Agriculture and Living System Symposium (~50 participants). Changes/Problems:
Nothing Reported
What opportunities for training and professional development has the project provided?1. Two graduate students in this project had the opportunities to get training on agricultural robotic system development, including machine vision system, deep learning based model for agricultural object detection, and control system. 2. Two undergraduate students worked on the project during the summer time, mainly assisted the graduate students on image data lableling, field experiemnt setup, and data processing. 3.The project supported the graduate students attending professional conferences, grower meetings, and other outreach activities. How have the results been disseminated to communities of interest?The results from this project have been widely disseminated to various communities, including professional society, grower community, and general public. We have published a few journal articles and conference proceedings based on the outcomes from this project. Meanwhile, the PIs and the graduate students presented this project at various conferences, which attracted lots of interest from the society. We have actively participated some grower meetings and many other extension events to present the results from this project to the grower community. We also demonstrated the developed robotic green fruit thinning system to growers and other stakeholders a few times in the past. We also presented and demonstrated the system to general public during different events, such as Penn State Ag Progress Day andScouting BSA AgSci-focused Camporee. What do you plan to do during the next reporting period to accomplish the goals?
Nothing Reported
Impacts
What was accomplished under these goals?
Objective 1: Investigation of green fruit removal dynamics and fruit thinning end-effector development. 1)A series of field tests were conducted to measure the dynamics of green fruit removal. The measurements included fruit removal forces, orientation, as well as the specifications of the fruits, which are the critical parameters for robotic systemdesign. 2)A mechanical green fruit removal end-effector prototype was designed, developed, and tested in the field. The developed end-effector successfully removed majority of the targeted green fruits in the test. 3)A new type of green fruit thinning end-effector was developed to singulate and cut unwanted green fruits in a cluster. The field tests in 2022 spring showed the mechanism can successfully singulate the king fruits from other lateral fruits, which can be further developed to be integrated with a robotic arm. Objective 2: Development of sensing systems to detect green fruits in orchard environment, as well as the decision making for fruit removal. 1)A stereo vision imaging system was built to take the images in the orchard to detect green fruit clusters at the early stage.A set of images were acquired during the green fruit stage for three apple cultivars at different fruit sizes. 2)A deep learning based algorithm was developed to detect green fruits and fruit stems in the images. The trained model provided more than 80% detection accuracy. 3)An image processing algorithm was developed to determine the orientation of green fruit, which can provide information for manipulating the end-effector to remove the targeted fruits. Meanwhile, various parameters were evaluated during the fruit orientation measurement. 4)Algorithms were developed for green fruit and stem pairing, as well as fruit clustering, which are used for integrated system to conduct robotic green fruit thinning. Objective 3:Integration and evaluation of robotic green fruit thinning system by combining the sensing system and thinning end-effector. 1)An integrated robotic system was developed with a collision free path algorithm to engage the targeted fruits. Experiments were conducted to engage both single fruit at a time and a fruit cluster in a sequence. 2)An integrated robotic system was developed with integrating robotic end-effector, manipulator, machine vision system, and a collision free path algorithm. A series of experiments were conducted to evaluate the performance of the system. Results indicated that the developed robotic system can effectively removal targeted green fruit in a cluster. While the efficiency needs to be improved.
Publications
- Type:
Journal Articles
Status:
Under Review
Year Published:
2024
Citation:
Hussain, M., He, L., Schupp, J. and Heinemann, P., 2024. Green fruit-stem pairing and clustering for machine vision system in robotic thinning of apples. Journal of Field Robotics, 2024.
- Type:
Theses/Dissertations
Status:
Published
Year Published:
2023
Citation:
Hussain, M. 2023. Robotic green fruit thinning for apple production. PhD Dissertation, The Pennsylvania State University.
- Type:
Theses/Dissertations
Status:
Published
Year Published:
2024
Citation:
Arguijo, J. 2024. Vision system for the detection of apples in the green fruit stage. MS Thesis, The Pennsylvania State University.
- Type:
Journal Articles
Status:
Published
Year Published:
2023
Citation:
Mirbod, O., Choi, D., Heinemann, P.H., Marini, R.P. and He, L. 2023. On-tree apple fruit size estimation using stereo vision with deep learning-based occlusion handling. Biosystems Engineering, 226, pp.27-42.
- Type:
Conference Papers and Presentations
Status:
Other
Year Published:
2022
Citation:
Presentation: He, L. (October 19, 2022). Robotic crop load management for apples. 2022 FIRA USA, Fresno, CA.
- Type:
Conference Papers and Presentations
Status:
Other
Year Published:
2023
Citation:
Presentation: He, L. (April 18, 2023). Artificial intelligence (AI) for robotic apple crop load management. AI for Ag Conference, Orlando, FL.
- Type:
Conference Papers and Presentations
Status:
Other
Year Published:
2024
Citation:
Presentation: Arguijo, J., He, L., & Heinemann, P. (July 16, 2024). Vision System for the detection and sizing of apples in green fruit stage. NABEC, Stage College, PA.
- Type:
Conference Papers and Presentations
Status:
Other
Year Published:
2022
Citation:
Presentation: Hussain, M., & He, L. (August 1, 2022). Green fruit and stem pairing for robotic green fruit thinning vision system. 2022 Northeast Agricultural and Biological Engineering Conference, Edgewood, MD.
- Type:
Conference Papers and Presentations
Status:
Other
Year Published:
2021
Citation:
Presentation: Hussain, M., & He, L. (July 14, 2021). Fruit removal dynamics for robotic green fruit thinning. 2021 American Agricultural and Biological Engineers Annual International Meeting, Virtual.
- Type:
Conference Papers and Presentations
Status:
Other
Year Published:
2023
Citation:
Presentation: Hussain, M., He, L., & Heinemann. (July, 11, 2023). Robotic Green Fruit Thinning for Apple Production. ASABE Annual International Meeting, Omaha, NE.
- Type:
Conference Papers and Presentations
Status:
Other
Year Published:
2023
Citation:
Presentation: M. Hussain, L. He, P. Heinemann. (August 1, 2023). 3D Orientation Estimation of Green Fruit for Robotic Thinning. NABEC, Guelph, ON.
|
Progress 07/01/22 to 06/30/23
Outputs
Target Audience:1. Professional society: 1) The research results on robotic green fruit thinning system were presented by PIs and graduate students at 2022 ASABE annual meeting (July 12-16, virtual) and 2022 NABEC annual meeting (July 31 - August 3); 2) Project PIs also presented the outcomes from the project at various professional meetings, including W3009 multi-state project annual meeting (June 2023), 2022 FIRA robotics conference (October 2022), and AI for Ag conference (April 2023). 2. Grower society: 1) Present the research outcomes at Penn State Winter Fruit Schools in six different locations (February - March 2023, in total > 200 participants); 2) Demonstrate the robotic green fruit thinning system at 2022 FREC field day (July 2022, ~ 120 participants); 3) Demonstrate and present at 2022 Penn State Ag Progress Day; 4) Presented and demonstrated the system at 2023 FREC Precision Ag Field Day (June, 2023). Changes/Problems:We have requested a one year no-cost extension for the project, and the new end date is 6/30/2024. What opportunities for training and professional development has the project provided?1. Two graduate students in this project had the opportunities to get training on agricultural robotic system development, including machine vision system, deep learning based model for agricultural object detection, and control system. 2. The project supported the graduate students attending professional conferences, grower meetings, and other outreach activities. How have the results been disseminated to communities of interest?With these targeted audiences, we have disseminated our project results to multiple communities of interest via presentations, demonstrations, and articles. These communities include professional society, grower society, and some other groups. What do you plan to do during the next reporting period to accomplish the goals?During the next reporting period, we will focus more on the system integration and extension activities. Objective 2: 1. Will further improve the algorithm to increase the accuracy of fruit detection and counting at tree level. 2. Will further investigate green fruit detection and localization in 3D environment. Objective 3: 1. Further improve the integrated robotic green fruit thinning system.
Impacts
What was accomplished under these goals?
Objective 1: No activities under this objective during this period. Objective 2: 1. A series of images were acquired throughout the green fruit stage, and AI based algorithms were developed to detect green fruit at tree level. 2. Algorithms were developed for green fruit and stem pairing, as well as fruit clustering, which are used for integrated system to conduct robotic green fruit thinning. Objective 3: 1. An integrated robotic system was developed with integrating robotic end-effector, manipulator, machine vision system, and a collision free path algorithm. A series of experiments were conducted to evaluate the performance of the system. Results indicated that the developed robotic system can effectively removal targeted green fruit in a cluster. While the efficiency needs to be improved.
Publications
- Type:
Journal Articles
Status:
Published
Year Published:
2023
Citation:
Hussain, M., He, L., Schupp, J., Lyons, D., & Heinemann, P. (2023). Green fruit segmentation and orientation estimation for robotic green fruit thinning of apples. Computers and Electronics in Agriculture 207, 107734.
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2023
Citation:
Pawikhum, K., Heinemann, P., He, L., Sommer, H., & Bock, R. (2023). Design of end-effectors for apple robotic thinning in green fruit stage. Paper No. 202300491. 2023 ASABE Annual International Meeting. (pp. 7)
- Type:
Theses/Dissertations
Status:
Published
Year Published:
2022
Citation:
Kittiphum Pawikhum. 2022. Design of end-effectors for thinning apple in the green fruit stage. MS Thesis. The Pennsylvania State University.
|
Progress 07/01/21 to 06/30/22
Outputs
Target Audience:1. Professional society: 1) The research results on green fruit thinning end-effector developmentwere presented at 2021 ASABE annual meeting (July 12-16, virtual); 2) The research results on green fruit thinning end-effector and fruit detection were presented at the W3009 multi-state project annual meeting. 2. Grower society: 1) The research results on gree fruit detection and orientation were presented at 2022 Mid-Atlantic Fruit and Vegetable Convention (~150 participants); 2) The introduction of robotic system for crop load manangement was presented at 2021 Penn State Extension Winter Fruit School (~100 participants); 3) The robotic green fruit thinning demomstraion was presented to State Horticultural Association of Pennsylvania (SHAP) research committee (12 participants); 3) Project results and system demonstration were presented to 2022 International Fruit Tree Association Tour (~120 participants). 3. Other audiences: 1) The developed green fruit thinning system was presented to the 2021Legislative tour (~70 participants); 2) Presented green fruit thinning project outcomes at 2022 June College Connection webinar (~80 participants); 3) Presented the related research outcomes at the 2022 Penn State Ag Councile Delegate Meeting (~30 participants). Changes/Problems:
Nothing Reported
What opportunities for training and professional development has the project provided?1. Two graduate students in this project had the opportunities to get training on agricultural robotic system development, including robotic end-effector, machine vision system, and deep learning based model for agricultrual object detection. 2. An undergraduate student was hired in the summer of 2021 and 2022 (partially) to help the graduate students in the field data collection, image labelling, and other related works. 3. The project supported the graduate students attending professionalconferences, grower meetings, and other outreach activities. How have the results been disseminated to communities of interest?With these targeted audiences, we have disseminated our project results to multiple communities of interest via presentations, demonstrations, and articles. These communities include professional society, grower society, and some other groups. What do you plan to do during the next reporting period to accomplish the goals?Objective 1: 1. Contiue to improve and test the green fruit thinning end-effector in 2023 spring. Objective 2: 1. Continue to improve the green fruit/stem detection model, espeically we will consider the fruits in a 3D environment. 2. Further investigate the collision free path devleopment by considering the 3D environment and using Robotic Operation System to improve the efficiency. Objective 3: 1. Integrate the robotic system with the fruit removal end-effector, the fruit/stem detection machine vision system anda path planning algorithm to engage fruits. 2. Field tests will be conducted to test the accuracy and efficiency of the integrated system.
Impacts
What was accomplished under these goals?
Objective 1: 1. A new type of green fruit thinning end-effector was developed to singulate and cut unwanted green fruits in a cluster. The field tests in 2022 spring showed the mechanism can successfully singulate the king fruits from other lateral fruits, which can be further developed to be integrated with a robotic arm. Objective 2: 1. A set of images were acquired during the green fruit stage for three apple cultivars at different fruit sizes. 2. A deep learning based algorithm was developed to detect green fruits and fruit stems in the images. The trained model provided more than 80% detection accuracy. 3. An image processing algorithm was developed to determine the orientation of green fruit, which can provide information for manipulating the end-effector to remove the targeted fruits.Meanwhile, various parameters were evaluated during the fruit orientation measurement. Objective 3: 1. An integrated robotic system was developed with a collision free path algorithm to engage the targeted fruits. Experiments were conducted to engage both single fruit at a time and a fruit cluster in a sequence.
Publications
- Type:
Journal Articles
Status:
Published
Year Published:
2022
Citation:
Hussain, M., He, L., Schupp, J. and Heinemann, P., 2022. Green Fruit Removal Dynamics for Robotic Green Fruit Thinning End-Effector Development. Journal of the ASABE, 65(4).
|