Source: WASHINGTON STATE UNIVERSITY submitted to NRP
HUMAN-MACHINE COLLABORATION FOR AUTOMATED HARVESTING OF TREE FRUIT
Sponsoring Institution
National Institute of Food and Agriculture
Project Status
COMPLETE
Funding Source
AFRI COMPETITIVE GRANT
Reporting Frequency
Annual
Accession No.
1000339
Grant No.
2013-67021-20942
Cumulative Award Amt.
$548,735.00
Proposal No.
2013-04559
Multistate No.
(N/A)
Project Start Date
Sep 1, 2013
Project End Date
Aug 31, 2017
Grant Year
2013
Program Code
[A7301]- National Robotics Initiative
Recipient Organization
WASHINGTON STATE UNIVERSITY
240 FRENCH ADMINISTRATION BLDG
PULLMAN,WA 99164-0001
Performing Department
Agricultural Research Center
Non Technical Summary
Harvest is the most labor-intensive operation in apple and pear orchards, requiring heavy utilization of seasonal labor. The development of robotic technology for harvesting tree fruit has achieved only limited success due to insufficient speed and accuracy of fruit recognition and removal. Lack of such technology is a crucial problem for the long-term sustainability of the domestic tree fruit industry because the cost of labor continues to increase and the availability of a semi-skilled labor force is becoming increasingly uncertain. The long-term goal of this work is to reduce dependency on human labor through mechanization and human-machine collaboration while increasing yields of premium quality fruit. The overall objective is to develop a framework for knowledge transfer and collaboration between human and machine. This objective will be achieved through the understanding of the dynamics of the hand picking of fruit, development of an effective end-effector based on the knowledge of hand picking, and a framework of hardware and software for optimal collaboration between human and machine for fruit identification. A trans-disciplinary team of experts is envolved in this project, which is crucial for the successful completion of these activities. It is anticipated that the outcomes of this innovative project will be a knowledge-base including fruit growth habits and the dynamics of hand picking, an effective fruit picking end-effector based on the knowledge of hand picking, and a framework for human-machine collaboration for improved fruit identification. Attainment of these technologies will lead to improved systems for tree fruit harvesting, which will have a significant impact on sustaining the global competitiveness of the U.S. tree fruit industry by reducing dependency on semi-skilled seasonal labor. In addition, the new approaches developed for object recognition and manipulation will have applications in the wider areas of robotics including the military, manufacturing and medicine.
Animal Health Component
10%
Research Effort Categories
Basic
75%
Applied
10%
Developmental
15%
Classification

Knowledge Area (KA)Subject of Investigation (SOI)Field of Science (FOS)Percent
4021110202085%
4041110106015%
Knowledge Area
404 - Instrumentation and Control Systems; 402 - Engineering Systems and Equipment;

Subject Of Investigation
1110 - Apple;

Field Of Science
2020 - Engineering; 1060 - Biology (whole systems);
Goals / Objectives
Our long-term goal is to reduce the requirement for large numbers of seasonal semi-skilled employees through mechanization and human-machine collaboration in the production of high quality fruit for the fresh market. The overall objective of this project is to create a framework for knowledge transfer and collaboration between human and machine for effective harvesting of tree fruit. Successful achievement of this objective will help improve the accuracy, speed, and robustness of robotic tree fruit harvesting systems, which will reduce labor demand for harvesting and remove ladders from the operation. To accomplish our overall objective, we will pursue the following two specific aims: 1) Study fruit growth and fruit hand picking to design effective fruit picking end-effector i. Study phenotype and fruit detachment of major apple varieties under WA conditions ii. Study biomechanical dynamics of hand picking of fruits iii. Develop and evaluate an end-effector prototype 2) Investigate human-machine collaboration for improved fruit identification
Project Methods
Specific Aim #1: Study manual fruit picking process to improve end-effector design

In this aim, natural-growth patterns and fruit picking dynamics will be studied and the knowledge gained through this study will be used to design, fabricate and test a proof-of-concept fruit picking end-effector capable of mimicking human hand-picking. The study will focus on apples but the results obtained from this research will be applicable to other tree fruit crops, as well as other biological objects/systems with similar features. This research aim will be achieved by (1) defining the basic functionalities for an automated apple picking end-effector; (2) investigating human hand picking process, measured by motions and force/torque applied; 3) studying fruit natural growth patterns; (4) creating a conceptual design of an apple picking end-effector; and (5) fabricating a research prototype to validate its feasibility, functionality and practicability both in a laboratory setup and in commercial apple orchards. Fruit Growth Habit of Apples: The growth characteristics (phenotype) of apples on a tree will provide critical information that is essential for designing an effective robotic apple-picking end-effector. Stem length, stem stiffness, fruit shape, size and orientation will impact design and function of the end-effector. These characteristics differ between varieties of apple and can even vary from year to year and site to site. Fruit characteristics and industry production standards will be obtained and recorded for common strains of three major apple varieties - Gala, WA38 and Honeycrisp. Observation and test blocks in three distinct growing regions will be identified in the spring and sample trees and fruit will be tagged when green fruit thinning is completed in June. Fruit growth will be observed and monitored. Final fruit size, shape, orientation, stem length and stiffness will be measured on fruit when internal indices indicate that the fruit is mature and ready to pick. This data will be used to determine the expected range for size, shape, orientation, number of fruit per spur and stem stiffness and length. Dynamics of Fruit Picking: In this task, we will perform field experiments to track and quantify forces, and twist torque in the fingers and fruit-hand motion during apple detaching processes. The gripping force and twisting torque applied to apple will be measured using a force sensing system. Hand-picking motion and process will also be recorded using high-speed cameras from two different perspectives. The data will be analyzed to develop a dynamic model of apple picking capable of analyzing the forces and motions of the most commonly seen human apple-picking patterns. Such analytical tool (and knowledge) will provide the essential information for the design of an effective apple-picking end-effector. End-effector Design, Development and Evaluation: The fruit growth- and human hand-inspired end-effector will be accomplished through; 1) designing mechanism of an end-effector based on observed hand motion in harvesting apples; and 2) fabricating an end-effector prototype, and testing it with a commercial robot arm. In this task, we are aiming at an end-effector that is cost-effective, simple and reliable, uses minimal sensors and actuators, and is light-weight to reduce total payload for the robotic arm. As such, optimal number of fingers, hand joints, and contacts, and sensors and actuators will be determined through analysis and simulation. These results will be used to design mechanisms of an end-effector along with comprehensive human hand picking data. An end-effector prototype will be fabricated and tested by itself and with a commercial robot arm in the laboratory setup. The machine vision system and the end-effector will be integrated with the manipulator. A test bed will be built in the engineering laboratory at Washington State University (WSU) -Tri-Cities in which the integrated systemwill be tested under a replicated apple tree environment to validate its feasibility, functionality and practicability. Once the laboratory evaluation is completed, field evaluation for the integrated system will also be performed. Specific Aim #2: Human-machine collaboration for improved fruit identification

Unpredictable and varying natural lighting conditions, occlusion by leaves, branches and other fruits, and clustering of fruit are three important challenges a machine-vision-based apple identification system faces. In this aim, we will address this issue through an innovative approach of human-machine collaboration. A machine vision system (a system of cameras and image processing techniques) can identify individual fruits with good visibility very quickly and accurately. But the machine requires complex algorithms and long computational time to identify fruits when they are in clusters, are under varying lighting condition or are partially or heavily occluded. Humans, on the other hand, can very quickly identify fruits in these complex situations where a machine takes a long time even while failing to achieve a desired accuracy. These complementary capabilities of human and machine will be exploited through a collaborative fruit identification environment. A framework for collaboration between a machine and a human for improved fruit identification will be developed. The system will have a set of color cameras to look at the canopy from multiple perspectives. These images will be presented to both an image processing system as well as to a human for fruit identification. The image processing system will be tailored to start from quick identification of fruits that are easily visible and are in a good lighting condition. Humans will be trained to start from the most difficult part of the identification, such as fruits that are heavily occluded. To communicate with the human operator, fruit identified by either of human or machine will be circled with a distinct color. At the same time, the machine will keep identifying and updating the database with newly identified fruit. If the human finds any errors in machine identification, s/he will correct it by moving the center of the fruit to a desired location or by removing incorrectly identified fruit. The process will be continued until all the fruits are identified in the images captured from two different perspectives. The system will be mobile in the field, so the identification will be continuously performed in real time. If the human lags behind, the machine will continue to identify fruits in the more complex environment. If the machine lags, the human can continue to identify apples in relatively simpler conditions until all the fruit in the scene are identified.

Progress 09/01/13 to 08/31/17

Outputs
Target Audience:The outcome of this project was communicated with researchers and industry representatives from around the world through publication and presentation in international professional conferences. Demonstration and display were conducted with WA industry and communities, including local growers and high school students and research organizations like Pacific Northwest National Lab. Project outcomes were also demonstrated and discussed with local news outlets, which disseminated the information to general public. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided?Three PhD students and two MS students were actively involved in this project. PIs and students interacted frequently to provide guidance to the students in carrying out various research activities while also helping them address challenges faced during the project implementation. Students carried out day-to-day research activities including data collection and analysis. Students were also supervised for research paper writing, presentation and publications. How have the results been disseminated to communities of interest?Research results have been disseminated through peer-reviewed publications, theses, conference papers, and technical reports; presented to fellow research agencies (e.g. Pacific Northwest National Lab); presentations at national and international meetings (e.g. American Society of Agricultural and Biological Engineering Annual Meetings, Automation Technologies for Off-Road Vehicle Conferences; IEEE International Conference on Robotics and Automation; IEEE/RSJ International Conference on Intelligent Robots and Systems); student symposia (Webinar to the IEEE Technical Society on Agricultural Robotics); community and high-school and local college students (WA precision agricultural Expo, Tri-cities area students during multi Science, Technology, Engineering and Match Event, and Prosser high school outreach, Prosser, WA; seminar at Heritage College, Toppenish, WA); and collaborators (e.g. Allan Brothers, Inc.). What do you plan to do during the next reporting period to accomplish the goals? Nothing Reported

Impacts
What was accomplished under these goals? Task 1. Study fruit growth and fruit hand picking to design effective fruit picking end-effector Study of the basic dynamics of apple hand picking provided fundamental knowledge on force analysis needed in designing effective and efficient end-effectors for apple picking. Our analysis suggested that grasping force varies from one apple to the next, which might be related to the firmness of the apple. The grasping force differed substantially between different types of grasping pattern from 9.6 N for cutting to 41.4 N for pulling, on average. Based on this knowledge, several prototype end-effectors (2 to 4 finger mechanical hand, and a three figure pneumatic hand) were designed, fabricated, and analyzed; and were integrated with a serial link manipulator for system level evaluation. It was found that grasping with three fingers would be more effective to detach apples compared to two-finger grasping. Detection of fruit orientation and stem location, and development of an end-effector that can hinge one finger against fruit stem would be helpful to improve effectiveness of fruit detachment end-effector. Also, experimental analysis of device's robustness revealed that a successful grasp can be completed for a maximum fruit position error of 3 cm. The end-effector developed in this work will meet specific design requirements for apple harvesting in modern orchards while decreasing the cost compared to more complex robotic hands. Field data and controlled laboratory experiments also showed that fruit separation can be detected with sensors. Accelerometer measurements were used to calculate the average distance to fruit separation from the onset of picking, which varied from 3 to 7 centimeters. In addition, a catching robot was developed and fabricated for fruit catching. Laboratory study was conducted to assess the effect of pick-and-catch approach versus a pick-and-place approach on overall cycle time. The pick-and-catch technique resulted in an approximately 50 percent reduction in overall cycle time compared to the conventional pick-and-place method of fruit storage. During integrated system testing in the lab, the catching robotic system caught all harvested fruit. Using a point-to-point motion control with the collection system resulted in multiple unintended collisions between the catching end-effector and tree canopy. Considering joint limits and singularities, motion planning that maintains the catching end-effector away from the tree during all movement is constrained by the two-link planar design with revolute joints. Task 2. Human-machine collaboration for improved fruit identification A machine vision system including a sensor platform and an image processing algorithm was developed. The sensing system provided an ability to operate in the night time and consisted of static cameras with a global view of the canopy that required imaging only once at the beginning of each harvesting cycle. The image processing algorithm based on a deep learning network was successful in identifying partially blocked apples and most of the individual apples in clusters. A hierarchical apple identification approach was used to improve the performance of machine vision system during robotic harvesting. Results showed that a fruit identification accuracy of 98% could be achieved for robotic apple harvesting; 91% accuracy was achieved by the machine independently, and 7% improvement was achieved through human collaboration. Hierarchical apple identification method was simple yet provided unique insight into vision system for robotic harvesting. This work showed what kind of harvesting accuracy could be achieved with a robotic harvester. Additionally, the work also showed a promise that involving human in the loop or human machine collaboration would have a high impact/significance on robotic harvesting. Machine vision system was further improved through exposure fusion technique that helped remove hard shadows and saturated regions of images acquired in the outdoor environment in bright daylight conditions. In addition, two different training models using Haar and local binary pattern (LBP) features were used for fruit identification, which improved the system performance in terms of speed as well as accuracy in detecting fruit. During the integrated harvesting system tests, the vision system was able to identify apples with up to 100% accuracy. Optimal fruit prioritization rule using traveling salesman problem (TSP) was used for efficient harvesting sequence. Field testing was completed for three harvesting seasons (2015 to 2017) in commercial apple orchards with the integrated robotic harvesting system that included the machine vision system, different types of end-effectors, a manipulator with varying degrees of freedom and a catching hand. For the latest machine vision system used in this work, the average localization time was 0.2 s per fruit. The harvesting system successfully picked about 85% of the fruit in a commercial orchard with an average picking time around 5 to 6.0 s per fruit. The most frequent cause of missed fruit (34.8% of missed ones) was a missed grasp due to the accumulation of position error during fruit localization. It was observed that even with a planar canopy structure where access to individual fruit was better than traditional architectures, improved manipulation is essential. More sophisticated manipulation, which depends on additional visual sensing of the environment, will improve system robustness.

Publications

  • Type: Conference Papers and Presentations Status: Other Year Published: 2016 Citation: Silwal, A., Davidson, J.R., Karkee, M., Mo, C., and Zhang, Q. 2016. Robotic Apple Harvesting. Poster presented at: 2016 Precision Farming Expo. Kennewick, WA. January 7-8.
  • Type: Conference Papers and Presentations Status: Other Year Published: 2016 Citation: Davidson, J.R., Silwal, A., Hohimer, C.J., Karkee, M., Mo, C., and Zhang, Q. 2016. Proof-of-Concept of a Robotic Apple Harvester. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Daejon, Korea, October 9-14, 2016.
  • Type: Journal Articles Status: Published Year Published: 2017 Citation: Silwal, A., J. Davidson, M. Karkee, C. Mo, Q. Zhang, K. Lewis. 2017. Design, Integration and Field Evaluation of a Robotic Apple Harvester. Journal of Field Robotics, 34(6): 1140-1159.
  • Type: Journal Articles Status: Published Year Published: 2016 Citation: Silwal, A., Karkee, M. andZhang, Q. 2016. A Hierarchical approach of apple identification for robotic harvesting. Transactions of ASABE, 59(5): 1079-1086.
  • Type: Journal Articles Status: Published Year Published: 2016 Citation: Davidson, J.R., Silwal, A., Karkee, M., Mo, C., and Zhang, Q. 2016. Hand Picking Dynamic Analysis for Undersensed Robotic Apple Harvesting. Transactions of the ASABE, 59(4): 745-758.
  • Type: Journal Articles Status: Published Year Published: 2016 Citation: Li, J., Karkee, M., Zhang, Q., Xiao, K. and Feng, T. 2016. Characterizing apple picking patterns for robotic harvesting. Computers and Electronics in Agriculture, 127:633-640.
  • Type: Journal Articles Status: Published Year Published: 2015 Citation: Gongal, A., Amatya, S., Karkee, M., Zhang, Q., and Lewis, K. 2015. Sensors and systems for fruit detection and localization: A review. Computers and Electronics in Agriculture, 116:8-19.
  • Type: Journal Articles Status: Published Year Published: 2015 Citation: Silwal, A., Gongal, A., and Karkee, M. 2014. Apple Identification in Field Environment with Over-The- Row Machine Vision System. Agricultural Engineering International: Agric Eng Intl (CIGR Journal), 16(4): 66-75.
  • Type: Conference Papers and Presentations Status: Other Year Published: 2017 Citation: Davidson, J.R., Hohimer, C.J., Mo, C., and Karkee, M. 2017. Dual Robot Coordination for Apple Harvesting. In ASABE Annual International Meeting, Spokane, WA, July 16-19, 2017.
  • Type: Conference Papers and Presentations Status: Published Year Published: 2016 Citation: Silwal, A., Davidson, J., Karkee, M., Mo, C., Zhang, Q., Lewis, K. 2016. Effort towards robotic apple harvesting in Washington State. ASABE 2016 Annual International Meeting; 17-21 July 2016; Orlando, Florida, USA. Paper ID: 162460869.
  • Type: Conference Papers and Presentations Status: Other Year Published: 2016 Citation: Davidson, J.R., Hohimer, C.J., and Mo, C. 2016. Preliminary Design of a Robotic System for Catching and Storing Fresh Market Apples. Agricontrol 2016: 5th IFAC Conference on Sensing, Control, and Automation for Agriculture. Seattle, WA.
  • Type: Conference Papers and Presentations Status: Published Year Published: 2015 Citation: Silwal, A., Karkee, M., and Zhang, Q. 2015. A hierarchical approach of apple identification for robotic harvesting. American Society of Agricultural and Biological Engineers (ASABE) annual international meeting, Paper #: 152167504; 26-29 July 2015; New Orleans, USA.
  • Type: Conference Papers and Presentations Status: Other Year Published: 2015 Citation: Davidson, J.R., Mo, C., Silwal, A., Karkee, M., Li, J., Xiao, K., Zhang, Q., Lewis, K. 2015. Human-Machine Collaboration for the Robotic Harvesting of Fresh Market Apples. IEEE International Conference on Robotics and Automation (ICRA) Workshop on Robotics in Agriculture. Seattle, WA.
  • Type: Conference Papers and Presentations Status: Other Year Published: 2015 Citation: Davidson, J.R. and Mo, C. 2015. Mechanical Design and Initial Performance Testing of an Apple-Picking End-Effector. ASME International Mechanical Engineering Congress and Exposition. Houston, TX.
  • Type: Conference Papers and Presentations Status: Other Year Published: 2014 Citation: Davidson, J. and C. Mo. 2014. Conceptual Design of an End-Effector for an Apple Harvesting Robot. 6th Automation Technology for Off-road Equipment Conference (ATOE 2014), Sept 16-19, 2014, Beijing, China.
  • Type: Conference Papers and Presentations Status: Other Year Published: 2014 Citation: Silwal, A. and M. Karkee. 2014. Apple Identification in Field Environment with Over-The-Row Machine Vision System. 6th Automation Technology for Off-road Equipment Conference (ATOE 2014), Sept 16-19, 2014, Beijing, China.
  • Type: Conference Papers and Presentations Status: Published Year Published: 2014 Citation: Tong, J., Q. Zhang, M. Karkee, H. Jiang, J. Zhou. 2014. Understanding the Dynamics of Hand Picking Patterns of Fresh Market Apples. 2014 ASABE and CSBE/SCGAB Annual International Meeting, July 13-16, 2014, Montreal, Quebec, Canada; ASABE Paper Number: 14-1898024.


Progress 09/01/15 to 08/31/16

Outputs
Target Audience:The outcome of this project was communicated with researchers and industry representatives from around the world through publications and presentations in regional, national and international conferences. Project outcomes were also demonstrated and discussed with local news outlets, which disseminated the information to general public. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided?Two PhD students and one post-doc were actively involved in this project. PIs and students interacted frequently to provide guidance to the students. Students carried out day-to-day research activities including data collection and analysis. Students were also supervised for research paper writing, presentation and publications. How have the results been disseminated to communities of interest?The results were disseminated to the research community during a webinar to the IEEE Technical Society on Agricultural Robotics. Also, results were presented to Tri-Cities area students during multiple Science, Technology, Engineering, & Math (STEM) events, including science fairs and robotics/programming workshops. Our work was also presented in a number ofconferences and we have been in several interviews and news articles that desseminated the findings to variousstakeholders and general public. What do you plan to do during the next reporting period to accomplish the goals? Improvement of image segmentation and development of object detection algorithms to improve robustness of machine vision system. Improved integrated system will be tested in a laboratory and field environment. The catching robot along with second prototype of robotic harvester will be tested in a field environment to study the potential benefits of multi-robot collaboration for harvesting cycle time.

Impacts
What was accomplished under these goals? Summary of Impacts Robotic arm being developed in this work will be around 25% cheaper than currently available industrial robotic arms while meeting necessary design specifications to harvest apples and other similar fruit crops. Human-machine collaboration in apple identification has led to identification accuracy of >98% in both day and night time operation, which is an acceptablelevel for commercial robotic harvesting. Integration of robotic arm, end-effector and vision system has been completed and the first prototype was evaluated in field environment in 2015. Based on findings from last year,an integrated system is being improved, which will be further evaluated in the field in September and October of 2016. It is expected that the completion of these activities will lead to a successful development of automated apple harvesting system. When commercialized, the technology can reduce the labor use (and risk associated with uncertain labor force) in apple (and similar fruit) harvesting by 80% while reducing or maintaining the harvest time and cost around the same level. Reduced labor use also proportionally reduces the hazards to worker and insurance claims for the industry. Obj.# 1: Study fruit growth and fruit hand picking to design effective fruit picking end-effector 1) Major activities completed / experiments conducted; Dynamic analysis of four different hand picking methods in commercial orchards Integrated system with a seven degrees of freedom (DOF) harvesting manipulator, end-effector, and machine vision system evaluated in a laboratory environment Preliminary design and fabrication of an apple catching robot for fruit collection and storage 2) Data collected; Four picking techniques that do not require knowledge of fruit orientation were applied to five apple varieties growing in several different cultivation systems. Data consisted of 120 samples of apples for each variety of fruit. The sensors used during hand picking included force sensors and an inertial measurement unit. Experimental results were obtained for normal contact forces during a three-fingered power grasp as well as the angle of rotation around the axis of the forearm. 3) Summary statistics and discussion of results Controlled laboratory experiments andfield data show that fruit separation can be detected with sensors. Accelerometer measurements were also used to calculate the average distance to fruit separation from the onset of picking, which varied from three to seven centimeters. During integrated system testing in the lab, the secondary robotic system caught all harvested fruit. Likewise, the pick-and-catch technique resulted in an approximately 50 percent reduction in overall cycle time compared to the conventional pick-and-place method of fruit storage. 4) Key outcomes or other accomplishments realized. The passive compliance incorporated in the end-effector design enhanced grasping robustness. Obj.# 2: Machine vision and field evaluation of integrated system 1) Major activities completed / experiments conducted; Image acquisition system using exposure fusion to acquire well-exposed images of apple tree canopies at natural daylight outdoor environment. Vision system consisting of static cameras with a global view of the canopy that required imaging only once at the beginning of each harvesting cycle. Optimal fruit prioritization rule using traveling salesman problem (TSP) for efficient harvesting sequence. Field testing in a commercial apple orchard with the integrated robotic harvesting system 2) Data collected More than 150 Images of apple tree canopies were acquired at different lighting conditions. Data collected during field studies with the integrated system in a commercial orchard included the following: percentage of fruit successfully picked, average localization time, average picking time, and stem retention percentage. The causes formissed fruit were also documented. 3) Summary statistics and discussion of results Images acquired using exposure fusion technique had uniform intensities that greatly facilitated machine vision system and accurately identified all fruits in the scene. The average localization time was 1.5 s per fruit. For this experiment, a black curtain was mounted behind the canopy to block view of adjacent canopy and other unwanted objects in the background. The seven degrees of freedom harvesting system successfully picked 127 of the 150 fruit attempted in a commercial orchard achieving an overall success rate of 84% with an average picking time of 6.0 s per fruit. Of the 127 fruit, 86 successfully harvested had an intact stem. The most frequent cause of missed fruit (34.8% of missed ones) was a missed grasp due to the accumulation of position error during fruit localization. 4) Key outcomes or other accomplishments realized. The key lessons learned from integrated field studies included the following A global camera set-up can robustly detect and localize the apple center in 3D space. Additional imaging is not required between successive fruit picks. It was observed that even with a planar canopy structure where access to individual fruit was better than traditional architectures, improved manipulation is essential. More sophisticated manipulation, which depends on additional visual sensing of the environment, will improve system robustness.

Publications

  • Type: Journal Articles Status: Accepted Year Published: 2016 Citation: Silwal, A., M. Karkee, Q. Zhang, et al. (2016). "A Hierarchical approach of apple identification for robotic harvesting." Transactions of ASABE. Accepted.
  • Type: Journal Articles Status: Awaiting Publication Year Published: 2016 Citation: Davidson J.R., Silwal A., Karkee M., Mo C., & Zhang Q. (2016). Hand Picking Dynamic Analysis for Undersensed Robotic Apple Harvesting. Transactions of the ASABE. In press.
  • Type: Journal Articles Status: Under Review Year Published: 2016 Citation: Silwal, A., J. Davidson, M. Karkee, C. Mo, Q. Zhang, K. Lewis, et al.(2016). "Design, Integration and Field Evaluation of a Robotic Apple Harvester." Journal of Field Robotics. Under Review.
  • Type: Conference Papers and Presentations Status: Published Year Published: 2016 Citation: Silwal, A., Davidson, J., Karkee, M., Mo, C., Zhang, Q., Lewis, K. Effort towards robotic apple harvesting in Washington State. ASABE 2016 Annual International Meeting; 17-21 July 2016; Orlando, Florida, USA. Paper ID: 162460869.
  • Type: Conference Papers and Presentations Status: Published Year Published: 2016 Citation: Davidson J.R., Hohimer C.J., & Mo C. (2016). Preliminary Design of a Robotic System for and Storing Fresh Market Apples. Agricontrol 2016: 5th IFAC Conference on Sensing, Control, and Automation for Agriculture. Seattle, WA.
  • Type: Conference Papers and Presentations Status: Published Year Published: 2016 Citation: Silwal A., Davidson J.R., Karkee M., Mo C., & Zhang Q. (2016). Robotic Apple Harvesting. Poster presented at: 2016 Precision Farming Expo. Kennewick, WA. January 7-8.


Progress 09/01/14 to 08/31/15

Outputs
Target Audience:The outcome of this project was communicated with researchers and industry representatives from around the world through publication and presentation in local, national and international meetings and conferences. Project outcomes were also demonstrated and discussed with local news outlets, which disseminated the information to general public. Changes/Problems:It was initially planned to procure an off-the-shelf, industrial manipulator for integration with the complete harvesting system. However, after contact with multiple vendors it was determined that the cost of an industrial manipulator would be prohibitively expensive for farmers, in part because the systems provided some state-of-the-art capabilities not required for apple harvesting. Therefore, the project team decided to design and construct its own manipulator. What opportunities for training and professional development has the project provided?Two PhD students and three visiting scholars from China were actively involved in this project. PIs and students interacted frequently to provide guidance to the students. Students and scholars carried out day to day research activities including data collection and analysis. Students were also supervised for research paper writing, presentation and publications. How have the results been disseminated to communities of interest?Three conference proceedings were submitted and presented based on the results from this work. One proceeding was presented in ASABE Annual International Meeting in New Orleans, LA in July, 2015. Another proceedings was presented in Sep, 2015 in Automaton Technologies for Off-road Vehicle Conference in Beijing, China. We also presented in IEEE International Conference on Robotics and Automation (ICRA) Workshop on Robotics in Agriculture, Seattle, WA; May 30, 2015. What do you plan to do during the next reporting period to accomplish the goals? Lab and field tests will be continued to validate the findings in force and pattern necessary for effective apple detachment. Methods to detect apple orientation and stem location will also be studied. The vision system and manipulator will be integrated and tested in a laboratory mock-up with a replica apple tree. Field testing of the complete, integrated system will be conducted in a commercial apple orchard. The vision system and manipulator will be integrated and tested in a laboratory mock-up with a replica apple tree. Field testing of the complete, integrated system will be conducted in a commercial apple orchard.

Impacts
What was accomplished under these goals? Summary of Impacts Robotic arm being developed in this work will be around 25% cheaper than currently available industrial robotic arms while meeting necessary design specifications to harvest apples and other similar fruit crops.Human-machine collaboration in apple identification have led to identification accuracy of >95% in both day and night time operation, which is an acceptable level for commercial robotic harvesting.Integration of robotic arm, end-effector and vision system is underway. It is expected that the completion of these activities will lead to a successful development of automated apple harvesting system. When commercialized, the technology can reduce the labor use (and risk associated with uncertain labor force) in apple (and similar fruit) harvesting by 80% while reducing or maintaining the harvest time and cost around the same level. Reduced labor use also proportionally reduces the hazards to worker and insurance claims for the industry. Obj.# 1: Study fruit growth and fruit hand picking to design effective fruit picking end-effector 1) Major activities completed / experiments conducted; Basic physics of apple hand picking was studied to provide fundamental knowledge on force analysis needed in designing effective end-effector for apple picking A six Degree of Freedom (DOF), serial link manipulator was constructed An inverse kinematics algorithm was developed and implemented in the manipulator's controller Two prototype end-effectors were designed, fabricated, and analyzed 2) Data collected; Field and lab tests to acquire new set of data on grasping force and pattern for manual apple picking. Lab tests were also conducted to identify the pressure threshold that would cause bruising on various apple cultivars. Experimental analysis of the prototype end-effector was completed. Experiments included the measurement of finger normal forces as well as the amount of force required to pull an object from the end-effector's grasp. Also, extensive analysis of the system's robustness to position error was conducted in order to determine the acceptable range of error during fruit localization. 3) Summary statistics and discussion of results It was found that grasping with three fingers would be more effective to detach apples compared to two-finger grasping. Detection of fruit orientation and stem location, and development of an end-effector that can hinge one finger against fruit stem would be helpful to improve effectiveness of fruit detachment end-effector. Results also showed that approximately 10% of the actuator's current is required to produce the desired finger normal forces. Also, experimental analysis of device robustness revealed that a successful grasp can be completed for a maximum fruit position error of 3 cm. 4) Key outcomes or other accomplishments realized. Knowledge on fruit grasping patterns and forces is useful for scientists and engineers to design and develop effective end-effectors to achieve desired grasping patterns, but more studies will be needed to draw conclusions with confidence. The end-effector being developed in this work will meet specific design requirements for apple harvesting in modern orchards while decreasing the cost compared to more complex robotic hands. Obj.# 2: Investigate human-machine collaboration for improved fruit identification 1) Major activities completed / experiments conducted; A hierarchical apple identification approach was developed to improve the performance of machine vision system A unique method was investigated to locate fruit stem in the images, which will facilitate optimal grasping position 2) Data collected Experimental analysis of hierarchical approach in field environment with more than 800 images over 20 trees with 1844 apples More than 100 close-up images of apples with clearly visible stems were acquired in orchard environment for experimental analysis 3) Summary statistics and discussion of results Results showed that a fruit identification accuracy of 98% could be achieved for robotic apple harvesting; 91% accuracy was achieved by the machine independently, and 7% improvement was achieved with human involvement. Preliminary results showed a promise for identification of fruit stem independent of orientation of fruit in the images. However, quantitative results are yet to be obtained. 4) Key outcomes or other accomplishments realized. Hierarchical apple identification method was simple yet provided unique insight into vision system for robotic harvesting. This work showed what kind of harvesting accuracy could be achieved with a robotic harvester. Additionally, the work also showed a promise that involving human in the loop or human machine collaboration would have a high impact/significance on robotic harvesting. Stem detection method will provide a robust estimation of fruit stem in images, which would help optimally position the harvesting end-effector for grasping and detachment.

Publications

  • Type: Conference Papers and Presentations Status: Published Year Published: 2014 Citation: Silwal, A., Gongal, A., Karkee, M., 2014. Apple Identification in Field Environment with Over-The- Row Machine Vision System. Proceedings of the 6th Automation Technology for Off-road Equipment Conference (ATOE); 15-19 September 2014; Beijing, China.
  • Type: Journal Articles Status: Published Year Published: 2014 Citation: Silwal, A., Gongal, A., Karkee, M., 2014. Apple Identification in Field Environment with Over-The- Row Machine Vision System. Agricultural Engineering International: Agric Eng Intl (CIGR Journal), 16(4): 66-75.
  • Type: Conference Papers and Presentations Status: Published Year Published: 2015 Citation: Davidson, J.R., Mo, C., Silwal, A., Karkee, M., Li, J., Xiao, K., Zhang, Q., Lewis, K., 2015. Human-Machine Collaboration for the Robotic Harvesting of Fresh Market Apples. IEEE International Conference on Robotics and Automation (ICRA) Workshop on Robotics in Agriculture. Seattle, WA.
  • Type: Conference Papers and Presentations Status: Accepted Year Published: 2015 Citation: Davidson, J.R., Mo, C., 2015. Mechanical Design and Initial Performance Testing of an Apple-Picking End-Effector. ASME International Mechanical Engineering Congress and Exposition. Houston, TX. Accepted for publication.


Progress 09/01/13 to 08/31/14

Outputs
Target Audience: The outcome of this project was communicated with researchers and industry representatives from around the world through publication and presentation in international professional conferences. Changes/Problems: It was initially planned to procure an off-the-shelf, industrial manipulator for integration with the complete harvesting system. However, after contact with multiple vendors it was determined that the cost of an industrial manipulator would be prohibitively expensive for farmers, in part because the systems provided some state-of-the-art capabilities not required for apple harvesting. Therefore, the project team decided to design and construct its own manipulator. What opportunities for training and professional development has the project provided? Two PhD students and one visiting scholar from China were actively involved in this project. PIs and students interacted frequently to provide guidance to the students. Students carried out day to day research activities including data collection and analysis. Students were also supervised for research paper writing, presentation and publications. How have the results been disseminated to communities of interest? Three conference proceedings were submitted this from this work. One proceeding was presented in ASABE Annual International Meeting in Montreal, Canada in July, 2014. Two more proceedings will be presented in Sep, 2014 in Automatoon Technologies for Off-road Vehicle Conference in Beijing, China. What do you plan to do during the next reporting period to accomplish the goals? +The highest force in hand picking process needs to be analyzed more thoroughly for gaining a better understanding of the dynamics of the picking process. A sensing system with three-figure configuration will be developed and used to collect more data in 2014 harvest season, which will then be analyzed to compare the signature of force exerted during various ways of apple picking. The forces in the peduncle-apple junction and peduncle-branch junction will also be studied. The goal will be to understand detailed movements during hand picking that could be used to improve end-effector designs. +A conceptual design of the harvesting end-effector will be completed and a prototype will be fabricated. +Apple identification accuracy will be analyzed through an iterative identification and removal method to mimic robotic harvesting environment. Collaboration with human and multiple-camera machine vision system will also be investigated. High resolution 3D imaging systems will be investigated and fused with color images for accurate apple fruit identification and localization.

Impacts
What was accomplished under these goals? Summary of Impacts Robotic arm being developed in this work will be around 25% cheaper than currently available industrial robotic arms while meeting necessary design specifications to harvest apples and other similar fruit crops.Apple identification method we have developed works in day and night time with an accuracy of 90%. When we add human and machine collaboration, it is expected that the apple identification accuracy will be more than 95% in the latest apple orchards, which is an acceptable level for commercial robotic harvesting.It is expected that the completion of these activities will lead to a successful development of automated apple harvesting system. When commercialized, the technology can reduce the labor use (and risk associated with uncertain labor force) in apple (and similar fruit) harvesting by 80% while reducing or maintaining the harvest time and cost around the same level. Reduced labor use also proportionally reduces the hazards to worker and insurance claims for the industry. Obj.# 1: Study fruit growth and fruit hand picking to design effective fruit picking end-effector 1) Major activities completed / experiments conducted; ·Study basic physics of apple hand picking for providing fundamental knowledge on force analysis needed in designing effective and damage-less end-effector of apple picking ·Procure an off-the-shelf, industrial manipulator from a robotics manufacturer ·Develop an initial conceptual design of a harvesting end-effector 2) Data collected; ·A grasping force measurement system was developed and used to collect field data with various ways of apple picking including pulling, cutting, and twisting. Apple physical parameters including shape, weight and firmness were also collected and correlated with grasping forces. Three different varieties tested were ‘Granny Smith’, ‘Gala’, and ‘Fuji’. Preliminary analysis of the relationship between grasping forces and apple physical parameters was conducted. ·A custom design for a 6 Degree of Freedom (DOF), serial link manipulator was completed. Required actuators and brackets for prototype development have been acquired. Conceptual design of an end-effector is in progress. 3) Summary statistics and discussion of results ·Preliminary analysis suggested that grasping force varies from one apple to the next, which might be related to the firmness of the apple. The grasping force differed substantially between different types of grasping pattern from 9.6 N for cutting to 41.4 N for pulling, on average. No strong correlation of grasping force with the variety, shape, size, and firmness of apples was observed. 4) Key outcomes or other accomplishments realized. ·Knowledge on fruit grasping patterns and forces is useful for scientists and engineers to design and develop effective end-effectors to achieve desired grasping patterns, but more studies will be needed to draw conclusions with confidence. ·Robotic arm being developed in this work will meet specific design requirements for apple harvesting in modern orchards while decreasing the cost compared to currently available industrial robotic arms. Obj.# 2: Investigate human-machine collaboration for improved fruit identification 1) Major activities completed / experiments conducted; Develop a machine vision system for apple identification in orchard environment 2) Data collected A machine vision system including an over-the-row sensor platform and an image processing algorithm was developed. The sensing system provided a controlled lighting environment and an ability to operate in the night time. 3) Summary statistics and discussion of results The image processing algorithm was successful in identifying partially blocked apples and individual apples in clusters. Preliminary results showed an accuracy of 90% in apple identification. 4) Key outcomes or other accomplishments realized. Machine vision system has shown promise to be highly accurate in identification of apples in fruiting wall orchards both in day and night time operations, which is crucial for the success of robotic apple harvesting. When we human and machine collaboration is incorporated, it is expected that the apple identification accuracy will be more than 95% in the latest apple orchards, which is an acceptable level for commercial robotic harvesting.

Publications

  • Type: Conference Papers and Presentations Status: Accepted Year Published: 2014 Citation: Davidson, J. and C. Mo. 2014. Conceptual Design of an End-Effector for an Apple Harvesting Robot. 6th Automation Technology for Off-road Equipment Conference (ATOE 2014), Sept 16-19, 2014, Beijing, China. Accepted for Presentation.
  • Type: Conference Papers and Presentations Status: Accepted Year Published: 2014 Citation: Silwal, A., and M. Karkee. 2014. Apple Identification in Field Environment with Over-The-Row Machine Vision System. 6th Automation Technology for Off-road Equipment Conference (ATOE 2014), Sept 16-19, 2014, Beijing, China. Accepted for Presentation.
  • Type: Conference Papers and Presentations Status: Published Year Published: 2014 Citation: Tong, J., Q. Zhang, M. Karkee, H. Jiang, J. Zhou. 2014. Understanding the Dynamics of Hand Picking Patterns of Fresh Market Apples. 2014 ASABE and CSBE/SCGAB Annual International Meeting, July 13-16, 2014, Montreal, Quebec, Canada; ASABE Paper Number: 14-1898024.
  • Type: Journal Articles Status: Submitted Year Published: 2014 Citation: Davidson, J., C. Mo, M. Karkee. 2014. Recent Developments in Robotic Manipulator and End-Effector Technologies for the Automated Harvesting of Agricultural Products. Computers and Electronics in Agriculture.