Progress 09/15/23 to 09/14/24

Outputs
Target Audience:Researchers, academic administrators and policy makers, K-12, graduate and undergraduate students, ag technology companies, tree fruit growers, extension educators, specialty crop industry members (including but not limited to fruit processors, insurance providers, equipment manufacturers, farm workers), news medium, and the public. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided?The project team has recruited and trained 5 Ph.D. students and 3 undergraduate students in engineering and agricultural economics disciplines, who have participated in research under supervision and mentoring of PIs/co-PIs. The project provided an opportunity for five graduate students to attend a field day on orchard automation technology in Yakima, Washington in July 2024 and tour packinghouses in Michigan in March 2024 and Washington in July 2024, to enable them to learn about the state-of-the-art technologies in orchard and packinghouse automation. One Ph.D. student also attended the first USDA Agri-food Innovation Symposium in Washington, DC in June 2024. Four graduate students were provided opportunities to attend 2024 ASABE annual international meeting in Anaheim, CA in July 2024 to present their research and learn about the latest progress in robotics, automation, and AI in agriculture. Additional opportunities were also provided to two graduate students, one junior faculty in Mechanical Engineering and one PI to visit commercial orchards in the state of Washington and observe firsthand a commercial robotic harvester prototype. All PIs/co-PIs were able to keep up with the latest developments in their areas of expertise by attending various conferences, grower meetings, workshops and seminars. How have the results been disseminated to communities of interest?A public demonstration of the new robotic harvesting system was held at a commercial orchard in Sparta, MI on September 10, 2024. The demonstration was attended by growers, extension educators, researchers and ARS national program leader, equipment manufacturers, graduate and undergraduate students, news media and the public. A new website (www.aimsforapples.org) was created, providing comprehensive, updated information about the project to growers, researchers, extension educators, and the public. In addition, more than 100 copies of the project brochure have been distributed at various grower and public events. Project information was provided to growers, researchers, and fruit industry members and stakeholders through emails sent to all regional Michigan State University tree fruit extension listservs. Emails reached over 1,345 individuals across the state of Michigan. Three social media posts provided pictures, videos, and information about the robotic harvester during the fall. Posts were on the Facebook page of a project PI, which has 1,499 friends. The project's robotic harvesting team participated in Michigan State University Science Festival, enabling high school students to learn about the robotic harvesting research. Five oral presentations were given about the project's research progress at 2024 Northeast Agricultural and Biological Engineer Conference at State College, PA on July 16, 2024 and at 2024 American Society of Agricultural and Biological Engineers (ASABE) annual international meeting in Anaheim, CA on July 28-31, 2024. Our robotic harvesting research was selected for exhibitionat the first USDA Agri-Food Innovation Forum in Washington, DC on June 11, 2024. Three PIs/co-PIs helped organize and/or participated in the pre-conference technology workshop of 2024 International Fruit Tree Association annual meeting in Yakima, WA on February 11, 2024, and a pre-field day technology workshop, hosted by Washington State University and Washington Tree Fruit Research Commission in Richland, WA on August 25, 2024. PIs gave an exhibition and presentations about the project and robotic harvesting research at the two workshops and participated in discussion with growers and technology developers on challenges and opportunities for enhancing technology adoption and collaboration among/between researchers and commercial technology developers. With the arrangement by an extension co-PI at Washington State University, four members of the project's robotic harvesting team travelled to Yakima, Washington on September 18-19, 2024 and observed a robotic harvester picking apples in commercial orchards, developed by a technology company in California. The team members shared with the company's engineers about the project's goals and progress and discussed the areas of collaboration in robotic harvesting. What do you plan to do during the next reporting period to accomplish the goals? Improve the robot's perception system for robust fruit and branch detection and localization; build a new version of the robotic harvesting module with improved picking performance. Construct a two-lane in-field sorting system for fruit defect detection and grading. Develop a computer vision algorithm pipeline for fruit defect detection and grading and fully integrate it with the sorting system for online evaluation. Construct an autonomous mobile platform for integration with a robotic harvest module and an in-field sorter. Develop automated bin filling and handling functions for integration with the mobile platform. Write extension articles and present updated project information at grower meetings and field days and through emails, project website, social media posts, etc. Conduct surveys to get grower feedback about the robotic harvester and in-field pre-sorter, additional needs or challenges for harvesting and sorting fruit in the field as well as potential barriers for adoption. Evaluate deep learning-based algorithms for apple fruit detection and surface quality assessment in orchard environment. Develop an open-access framework for robotic apple harvesting and infield sorting with apple fruit images and benchmark AI models. Update the cost of production to 2024, and to the parameters recommended by the team members at different states and conduct sensitivity analysis with respect to robot speed and efficiency parameters and worker wages. Develop and implement a survey of potential H-2A workers to find their reservation wage to work in U.S. agriculture on the H-2A visa and determine whether and to what extent proposed policies in the Farm Workforce Modernization Act, including proposals to include a pathway to permanent residency through the H-2A visa, impact workers' reservation wage and whether and to what extent the opportunity to work with robotic harvesters impacts reservation wages under the H-2A visa program.

Impacts
What was accomplished under these goals? 1. A new dual-arm robotic system was developed, where the two arms are coordinated to effectively harvest fruits. Optimum planning and control algorithms were developed to coordinate the vacuum supply and multiple arm movement. New image segmentation algorithms were developed to segment fruits, branches, and foliage from the background, which is critical to robotic harvesting of apples. Further efforts have also been made on the development of computer algorithms to identify tree branches and their positions.An initial collision-free multi-arm planning algorithm was developed and demonstrated, which showed promising results. A sun filtering system to efficiently control the natural sunlight to enhance the performance of the perception or computer vision system has been developed and tested. A preliminary selective harvesting algorithm for estimating fruit size and ripeness has been evaluated on an open-sourced dataset as well as our own collected dataset. In addition, preliminary laboratory and orchard studies were carried out to evaluate the feasibility of using air blowing to mitigate fruit occlusions by leaves. Factors that were considered in the air blowing study included type of air nozzles, air flow pattern and pressure, direction of air blowing relative to the leave/branch positions, and the distance between the air nozzle and target fruit. Field evaluations showed that the new dual-arm robotic system resulted in 9% to 34% harvest time improvements, compared to the single-arm robotic system and achieved 60% successful picking rate. Air blowing demonstrated potential to mitigate the leave occlusion effect on fruit detection. 2. To develop a new AI-based in-field sorting system, images were collected for over 200 apples, using an existing in-field sorting system. These apples were imaged using three different color-depth sensors at different conveyor speeds. The acquired images are being analyzed to evaluate the performance of these sensors for online quality detection of apples. In addition, more than 1,200 apples of three varieties with different quality conditions were collected from orchards in Michigan, and these apples will be imaged by a new computer vision system for further development of defects detection algorithms. 3. An improved fruit transporting and bin filling system was assembled for transporting apples harvested by the robot to a fruit bin for postharvest storage. Preliminary evaluation of the fruit handling system showed that it can meet the need for handling apples harvested by the robot with minimum bruising. Furthermore, the first phase of designing an autonomous mobile platform has been completed; major parts and components are being acquired; and construction of the system is expected to be completed in the second year. 4. The new robotic harvesting module was evaluated in a high-density tree system in Michigan apple orchard during 2024 harvest season. The robot performance was evaluated using multiple metrics, including picking speed (seconds per fruit), picking rate (percentage of fruits in the workspace that were picked by the robot), bruise rate for harvested apples, presence or absence of stems and/or spurs. Analyses of failed picks by the harvesting robot are being conducted to determine the factors causing the failed picks (i.e., inaccurate localization due to leave occlusion, branch occlusion, inappropriate action by the vacuum system, etc.), which will then be used for further improving the robot's perception and manipulation performance. 5. A deep learning model was developed to detect individual objects on trees in the orchard environment. The model was first tested on apple leaf infection detection. A web-based segmentation tool was integrated into the model for precise infection measurements and disease severity analysis. This model will be used for on-tree fruit quality identification for selective harvesting. Color-depth images of at least five apple varieties were collected from orchards in Michigan and Washington for training and testing image segmentation algorithms for robotic harvesting. More than 2,000 color images were collected of apples with different types of surface defects caused by different diseases and insects. In addition, 600 images of normal fruits for three varieties were collected during the harvest season, which will be used for establishing a database for selective robotic harvesting of apples. All these datasets have been or are being made available publicly, to allow other researchers and technology developers to train, test, and evaluate different AI-based detection models and algorithms. 6. The cost of production studies was updated for apple varieties Gala, Honeycrisp, Granny Smith, and WA38 for 2024. In addition, an Excel spreadsheet model was built to compare the costs and benefits of the robot and manual harvesting.

Publications

• Type: Peer Reviewed Journal Articles Status: Published Year Published: 2024 Citation: Lammers, K., Zhang, K., Zhu, K., Chu, P., Li, Z., Lu, R. (2024). Development and evaluation of a dual-arm robotic apple harvesting system. Computers and Electronics in Agriculture. 227, 109586. https://doi.org/10.1016/j.compag.2024.1