Progress 10/01/23 to 09/30/24
Outputs
PROGRESS REPORT Objectives (from AD-416): 1. Enable a new, efficient, and cost-effective robotic technology, coupled with automated infield sorting and quality tracking technologies, for commercial harvesting of apples. 2. Develop a new imaging technology, based on structured illumination integrated with artificial intelligence and advanced data analytics, with substantially improved capabilities for commercial quality inspection of fruits and vegetables. Approach (from AD-416): Development of enabling technologies for automated fruit harvesting and nondestructive quality inspection during postharvest handling can provide an effective solution to the labor availability and cost issues, and enhance production efficiency, product quality, and thus profitability and sustainability for the specialty crop industries. In recent years, much research has been focused on fruit robotic harvesting, but the progress has been slow and unsatisfactory in meeting industry needs, mainly due to the several key technical hurdles encountered in robotic perception (identifying and localizing fruit), manipulation (reaching out for and picking fruit), and systems integration and coordination. While machine vision technology is widely used for postharvest quality inspection of horticultural products, it still is short of meeting industry expectations in detection of quality-degrading defects and symptoms. This research is therefore aimed at developing a new, cost effective robotic technology for automated harvesting of apples and a new generation imaging technology with substantially enhanced capabilities for quality inspection of fruits and vegetables (e.g., pickling cucumber and tomato) during postharvest handling. Innovative concepts and designs, coupled with artificial intelligence, will be used in the development of the new robotic harvesting system for fruit imaging, detection, localization, and picking. The new robotic system will be integrated with the recently developed apple harvest and infield sorting machine, to enable automated harvesting, sorting, grading and tracking of apples in the orchard. Moreover, a new imaging system, using our newly developed technique on improved reflectance for imaging structures, will be assembled to enable rapid, real-time inspection of harvested horticultural products for quality-degrading defects caused by bruising, physiological disorders, and disease infection. The new knowledge and technologies generated from the research will enable growers and packers/ processors to achieve significant labor and cost savings in harvesting, enhance product marketability, and reduce postharvest product loss. OBJECTIVE 1: Robotic harvesting needs to be cost effective and efficient in fruit harvesting to meet grower expectations. Modular designs are advantageous in reducing overall system costs and enabling better, or easier, adaption to, or integration with, different harvesting platforms. Towards meeting this goal and built on the promising performance of the single-arm harvesting robot developed earlier, we have designed and built a new dual-arm robotic harvesting system for enhancing fruit harvest efficiency. The dual-arm harvesting robot shares one common perception system and one central vacuum system, thus making the system more compact, more energy efficient and cost effective, compared to the single-arm robotic system. A new version of the perception module with two line lasers, based on our innovative Active LAser-Camera Scanning (ALACS) technology, was designed and assembled to provide fruit localization information for the two robot arms to pick target fruits. New planning/ coordination algorithms were developed for control and coordination of the two arms for fruit harvesting. Furthermore, a fruit handling system has been assembled and integrated with the harvesting robot for receiving and transporting picked fruits to the bin. Field tests were conducted of the dual-arm harvesting robot in orchards with complex tree canopies during 2023 harvesting season. The harvesting robot achieved 60% successful picking rate and improved harvesting efficiency by 9% to 34%, compared to the single-arm configuration. A non-provisional patent application has been filed for the new dual-arm robotic harvesting system. Based on the promising performance of the dual-arm robot in the field harvesting test, we are redesigning the control of the vacuum system to further improve the harvesting efficiency of the dual-arm robot. New controls and control logic are being incorporated with the vacuum system for more efficient coordination of the two robot arms for faster apple picking. In addition, efforts have been made to develop new, improved algorithms for more accurate detection and localization of fruits. Different image segmentation techniques were developed and compared, and improved fruit segmentation results were obtained, compared to the current segmentation algorithm. Furthermore, new algorithms for detecting and localizing branches, based on image segmentation and deep learning techniques, are being developed, which are important to the development of an effective obstacle avoidance algorithm for robotic picking of fruits that are occluded or blocked by branches. OBJECTIVE 2: Structured-light imaging (SLI), as a promising emerging technique, enables more effective detection of many types of subsurface and surface defects of horticultural and food products, which may be difficult to detect using conventional imaging techniques. The conventional SLI technique, however, can only be implemented when objects or products are in stationary status. We have made progress on implementing the SLI technique in online settings, where samples are moving. An online SLI prototype has been constructed, which enables collection of pattern images from moving samples at different conveyor speeds. Theoretical and experimental analyses were conducted to select appropriate parameters related to the acquisition of pattern images from moving samples. The parameters studied include conveyor speed, spatial frequency of the structured light, orientation of structured light patterns (parallel and perpendicular to the moving direction), frequency of the structured-light pattern shift, and imaging frame rate by the camera and its synchronization with the light projector. Experiments were performed with reference and apple samples to determine the optimum combinations of the parameters for implementing the SLI technique for online applications. Results showed that the new SLI prototype was able to obtain good-quality pattern images that are needed for image demodulation at a conveyor speed of up to 400 mm/s when samples were illuminated at lower spatial frequencies (e.g., 0.05 cycles/mm). This represents a significant step in the implementation of the SLI technique for online detection of surface and/or subsurface defects from moving samples. However, at higher spatial frequencies, some abnormal stripes were observed around the edge of samples for both direct and alternate component images. A further investigation of an effective method to correct these abnormal stripes is being conducted. A manuscript on the design, calibration and optimization of the online SLI system prototype is being prepared for journal publication. Experiments are being conducted on the effectiveness of the SLI for detecting defects of apples and other fruits in moving status. Artificial Intelligence (AI)/Machine Learning (ML) Deep learning algorithms (i.e., convolutional neural networks) and foundation models have been developed for detection of fruits and tree branches in high-density apple orchards. The deep learning/foundation models were implemented using stand-alone GPU computers. The development and use of AI or deep learning/foundation models and algorithms is critical to achieving superior fruit detection results for robotic harvesting of apples. ACCOMPLISHMENTS 01 A new, more efficient dual-arm apple harvesting robot with great promise to save labor in the apple industry. Harvesting labor is the single largest cost in production of apples and other tree fruits. Harvest automation is thus urgently needed to address the rising costs and growing shortage of labor for fruit production. Based on the previous single-arm robotic harvesting technology, ARS researchers in East Lansing, Michigan, in collaboration with Michigan State University, developed a new dual-arm harvesting robot to enhance fruit harvest efficiency and cost effectiveness. The new robot demonstrated up to 34% improvements in harvesting efficiency, compared to the single-arm robot, with great potential for further performance enhancement. This new robot design provides a commercially viable solution to automated harvesting of apples, which is critical to the long-term sustainability and global competitiveness of the U.S. apple industry.
Impacts
(N/A)
Publications
- Chu, P., Li, Z., Zhang, K., Lammers, K., Lu, R. 2024. High-precision fruit localization using active laser-camera scanning: Robust laser line extraction for 2D-3D transformation. Smart Agricultural Technology. 2024(7) . Article 100391. https://doi.org/10.1016/j.atech.2023.100391.
- Zhang, K., Chu, P., Lammers, K., Li, Z., Lu, R. 2023. Active laser-camera scanning for high-precision fruit localization in robotic harvesting: system design and calibration. Horticulturae. 10(1):40. https://doi.org/10. 3390/horticulturae10010040.
- Zhang, K., Lammers, K., Chu, P., Li, Z., Lu, R. 2023. An automated apple harvesting robot⿿from system design to field evaluation. Journal of Field Robotics. 1-17. https://doi.org/10.1002/rob.22268.
- Li, J., Lu, Y., Lu, R. 2023. Identification of early decayed oranges using structured-illumination reflectance imaging coupled with fast demodulation and improved image processing algorithms. Postharvest Biology and Technology. 207. Article 112627. https://doi.org/10.1016/j.postharvbio. 2023.112627.
- Chu, P., Li, Z., Zhang, K., Chen, D., Lammers, K., Lu, R. 2023. O2RNet: Occluder-occludee relational network for robust apple detection of clustered orchard environments. Smart Agricultural Technology. 5. Article 100284. https://doi.org/10.1016/j.atech.2023.100284.
- Burks, T., Watson, A., Frederick, Q., Migliaccio, K., Lu, R. 2023. Frontier: Creating parallel SmartAg systems certificate programs for engineering and applied science graduate students. Journal of the ASABE. 66(5):1187-1203. https://doi.org/10.13031/ja.15358.
|
Progress 10/01/22 to 09/30/23
Outputs
PROGRESS REPORT Objectives (from AD-416): 1. Enable a new, efficient, and cost-effective robotic technology, coupled with automated infield sorting and quality tracking technologies, for commercial harvesting of apples. 2. Develop a new imaging technology, based on structured illumination integrated with artificial intelligence and advanced data analytics, with substantially improved capabilities for commercial quality inspection of fruits and vegetables. Approach (from AD-416): Development of enabling technologies for automated fruit harvesting and nondestructive quality inspection during postharvest handling can provide an effective solution to the labor availability and cost issues, and enhance production efficiency, product quality, and thus profitability and sustainability for the specialty crop industries. In recent years, much research has been focused on fruit robotic harvesting, but the progress has been slow and unsatisfactory in meeting industry needs, mainly due to the several key technical hurdles encountered in robotic perception (identifying and localizing fruit), manipulation (reaching out for and picking fruit), and systems integration and coordination. While machine vision technology is widely used for postharvest quality inspection of horticultural products, it still is short of meeting industry expectations in detection of quality-degrading defects and symptoms. This research is therefore aimed at developing a new, cost effective robotic technology for automated harvesting of apples and a new generation imaging technology with substantially enhanced capabilities for quality inspection of fruits and vegetables (e.g., pickling cucumber and tomato) during postharvest handling. Innovative concepts and designs, coupled with artificial intelligence, will be used in the development of the new robotic harvesting system for fruit imaging, detection, localization, and picking. The new robotic system will be integrated with the recently developed apple harvest and infield sorting machine, to enable automated harvesting, sorting, grading and tracking of apples in the orchard. Moreover, a new imaging system, using our newly developed technique on improved reflectance for imaging structures, will be assembled to enable rapid, real-time inspection of harvested horticultural products for quality-degrading defects caused by bruising, physiological disorders, and disease infection. The new knowledge and technologies generated from the research will enable growers and packers/ processors to achieve significant labor and cost savings in harvesting, enhance product marketability, and reduce postharvest product loss. Objective 1: Detection of target fruit on trees is the first step in robotic harvesting of apples. Accurate and robust detection of apples on trees is challenging, due to complex orchard environments which involve varying light conditions, fruit clustering and foliage/branch occlusions. Apples in clusters often overlap with each other, which presents a significant challenge for their individual identification. A comprehensive dataset of color images was collected from two varieties of apples from commercial orchards under different natural lighting conditions (i.e., direct lighting, back lighting, and overcast) with varying degrees of apple occlusions. A novel deep learning-based apple detection algorithm using convolutional neural networks was developed for detection of apples in clustering and occlusion situations. The new algorithm was evaluated for its performance using the collected images, and it achieved 94% overall detection accuracy, which outperformed 12 other state-of-the-art deep learning models. A manuscript has been submitted to journal for publication consideration. Accurate localization of apples on trees is a key step in robotic harvesting. Different techniques, including stereo vision, light detection and ranging (LiDAR) and time of flight imaging, have been used for fruit localization. However, these techniques have been found unsatisfactory when target fruits are occluded by leaves, and their performance also suffers under the varying natural lighting conditions in an orchard. Hence, a new, more robust and accurate perception system is needed for improved localization of apples. We have designed and built a novel perception system based on a laser triangulation principle. Calibration procedures were developed for the perception system, which achieved the maximum localization error of less than 4 mm and the average error of less than 1 mm. A patent application is being filed for the new perception system and a manuscript has been submitted to journal for publication consideration. A new version of the apple harvesting robot was developed. This version of the harvesting robot is mainly composed of the new perception system, a four-degree-of-freedom manipulator, an improved soft end-effector, and a dropping/catching module to receive and transport harvested fruits. A new perception strategy integrating the new perception system and deep learning fruit detection algorithm, was able to achieve more than 90% apple detection accuracy and precise localization of target apples. The harvesting robot was evaluated in two apple orchards in 2022. In the orchard where trees were young and well pruned, the robot achieved a 82. 4% successful harvesting rate, whereas in a second, older orchard with dense and cluttered branches and foliage, the robot had a 65.2% successful harvesting rate. The robot was also able to detach nearly 100% of the apples once they were gripped by the robot⿿s end effector. Overall, the harvesting robot has demonstrated superior performance, compared to the previous versions and other reported studies. Objective 2: Structured-light imaging (SLI) is an emerging technique for enhanced detection of surface and subsurface defects of horticultural and food products, which may otherwise be difficult to detect using conventional imaging techniques. With the conventional SLI technique, three phase-shifted pattern images are needed, thus making it difficult to implement the technique for online inspection applications. A study was conducted to explore a faster methodology for identification of defects in horticultural products. Pattern images were acquired, using three phase-shifted illumination patterns, from orange fruits that were infected with penicillium digitatum fungus, the most serious and devastating pathogen for orange fruit. An efficient image demodulation algorithm developed by our team, was used to obtain alternative component (AC) images from one or two original pattern images. The obtained AC images were then processed using two brightness adjustment and correction techniques, coupled with a machine learning algorithm and image segmentation technique, to identify pathogen infected areas from the orange fruit samples. Three image processing strategies were proposed and evaluated; all of them achieved high identification rates of greater than 95%. It was found that accurate detection of early fungus-infected symptoms on the orange fruit, which were not visible at the surface, can be achieved by using one or two phase shifted pattern images. This finding provides a basis for real-time implementation of the SLI technique for quality inspection of horticultural products. One paper based on this study has been published in a peer-reviewed journal, with a second paper under review for journal publication. Improvements to the current imaging system have been made for real-time implementation of the SLI technique for online quality inspection of horticultural products. A new sample conveying system was built to allow samples to be imaged by the SLI system at different conveying speeds, in synchronization with the digital light projector and high-speed camera. Preliminary tests on the digital light projector and imaging system have been carried out. Final integration of the software and hardware of the SLI system is being performed for real-time acquisition and processing of patten images for detection and classification of defective horticultural products. Artificial Intelligence (AI)/Machine Learning (ML) Deep learning algorithms for fruit detection and tree branch identification. Deep learning algorithms (i.e., convolutional neural networks) have been developed for detection of fruits and tree branches in high-density apple orchards. The deep learning algorithms were implemented using a stand-alone GPU computer. The development and use of AI or deep learning algorithms is critical to achieving superior fruit detection results for robotic harvesting of apples. ACCOMPLISHMENTS 01 A new perception system for robotic harvesting of apples achieves superior performance. Robotic harvesting is urgently needed to alleviate the growing shortage and rising cost of labor for the apple and other specialty crop industries. Accurate localization of target fruits on trees is critical to robotic harvesting of apples. Currently, stereo vision and light detection and ranging (LiDAR) systems are commonly used for fruit localization, but their performance is unsatisfactory and susceptible to fruit occlusions by leaves and/or branches and natural light variation. ARS researchers in East Lansing, Michigan, in collaboration with Michigan State University, designed a new perception system for improved localization of apples. The new system has demonstrated superior performance with the maximum localization error of no more than 4 mm, and it still provides accurate localization information when target fruits are occluded by leaves. A new version of the robotic harvester integrated with the perception system was able to pick 82.4% of apples on trees in a high-density orchard. A patent application is being filed for the new perception system. This new system represents a significant step towards the development of a commercially viable robotic apple harvesting technology to help the U.S. apple industry reduce the reliance on manual labor and achieve long-term sustainability and profitability.
Impacts
(N/A)
Publications
- Pothula, A., Zhang, Z., Lu, R. 2023. Evaluation of a new apple in-field sorting system for fruit singulation, rotation and imaging. Computers and Electronics in Agriculture. 208. Article 107789. https://doi.org/10.1016/j. compag.2023.107789.
- Li, J., Lu, Y., Lu, R. 2022. Detection of early decay in navel oranges by structured-illumination reflectance imaging combined with image enhancement and segmentation. Postharvest Biology and Technology. 196. Article 112162. https://doi.org/10.1016/j.postharvbio.2022.112162.
- Lu, R., Dickinson, N., Lammers, K., Zhang, K., Chu, P., Li, Z. 2022. Design and evaluation of end effectors for a vacuum-based robotic apple harvester. Journal of the ASABE. 65(5):963-974. https://doi.org/10.13031/ ja.14970.
|
Progress 10/01/21 to 09/30/22
Outputs
PROGRESS REPORT Objectives (from AD-416): 1. Enable a new, efficient, and cost-effective robotic technology, coupled with automated infield sorting and quality tracking technologies, for commercial harvesting of apples. 2. Develop a new imaging technology, based on structured illumination integrated with artificial intelligence and advanced data analytics, with substantially improved capabilities for commercial quality inspection of fruits and vegetables. Approach (from AD-416): Development of enabling technologies for automated fruit harvesting and nondestructive quality inspection during postharvest handling can provide an effective solution to the labor availability and cost issues, and enhance production efficiency, product quality, and thus profitability and sustainability for the specialty crop industries. In recent years, much research has been focused on fruit robotic harvesting, but the progress has been slow and unsatisfactory in meeting industry needs, mainly due to the several key technical hurdles encountered in robotic perception (identifying and localizing fruit), manipulation (reaching out for and picking fruit), and systems integration and coordination. While machine vision technology is widely used for postharvest quality inspection of horticultural products, it still is short of meeting industry expectations in detection of quality-degrading defects and symptoms. This research is therefore aimed at developing a new, cost effective robotic technology for automated harvesting of apples and a new generation imaging technology with substantially enhanced capabilities for quality inspection of fruits and vegetables (e.g., pickling cucumber and tomato) during postharvest handling. Innovative concepts and designs, coupled with artificial intelligence, will be used in the development of the new robotic harvesting system for fruit imaging, detection, localization, and picking. The new robotic system will be integrated with the recently developed apple harvest and infield sorting machine, to enable automated harvesting, sorting, grading and tracking of apples in the orchard. Moreover, a new imaging system, using our newly developed technique on improved reflectance for imaging structures, will be assembled to enable rapid, real-time inspection of harvested horticultural products for quality-degrading defects caused by bruising, physiological disorders, and disease infection. The new knowledge and technologies generated from the research will enable growers and packers/ processors to achieve significant labor and cost savings in harvesting, enhance product marketability, and reduce postharvest product loss. Objective 1: Major improvements have been made to enhance the overall performance of the harvesting robot for picking apples. Specifically, the robot⿿s manipulation system, including the pan/tilt and rotation mechanisms, was upgraded or redesigned for faster and more accurate movement of the robot⿿s arm to target fruit. A new, fully automated fruit dropping mechanism was constructed and incorporated into the robot, which allows quick dropping of harvested apples to the fruit catching module. An optimization scheme was implemented in the robot⿿s planning and control algorithm to determine the best picking and dropping strategies to reduce the fruit picking cycle. In addition, a modified camera configuration was used in the perception system to improve fruit detection caused by leave occlusion and hence fruit localization. A new catching module was constructed, which allows the robot to release picked apples in the shortest distance and time possible. Field testing and evaluation of the improved version of the robot was conducted in the research orchards of Michigan State University⿿s Horticultural Teaching and Research Center at Holt, Michigan during 2021 harvest season. Results showed that the robot was able to perform continuous picking at an average speed of 3.6 s per fruit, which represents a significant improvement over the previous version of the robot and is also much faster than that reported by other research groups. Field tests also showed that a modified camera configuration has resulted in about a 3 percent fruit detection rate improvement, but their performance for fruit localization was still short of meeting our expectations. To improve fruit localization by the robot⿿s perception system, a sensor fusion approach was proposed by integrating a Lidar sensor with our existing perception system. Calibration and evaluation of the lidar sensor was conducted in the laboratory to determine its ability to locate target fruit. Results showed that the lidar sensor was unable to meet the fruit localization accuracy requirement. Hence, research is being conducted to design and construct a new perception system for more accurate fruit localization. Although the end effector tested in 2021 field tests performed much better than the original version, it was still short of meeting our requirement of picking 99% fruit after the end effector engages with target fruit. Hence, further effort was made to design a new version of the end effector along with the selection of a more efficient vacuum system. After several rounds of trial, a new end effector was designed. In laboratory evaluation, this new effector has exceeded our expectation in generating suction forces needed for gripping and detaching fruit from trees. An invention disclosure was submitted for the new end effector in May 2022. Objective 2: Two versions of the structured-light imaging (SLI) system, one for laboratory use and the other for future online real-time inspection of food products, have been designed. The laboratory version system integrated with an inhouse developed computer program has been tested and evaluated for taking pattern images from moving samples, while the online version system with a different optical configuration is being assembled for simultaneous acquisition of multiple pattern images from moving samples. Preliminary evaluation of the two system designs indicated that they would meet the imaging speed requirements for laboratory and future online inspection applications. Major components needed for the online SLI system have been acquired and ready for testing and integration with an existing laboratory platform. ACCOMPLISHMENTS 01 An integrated robotic system for efficient apple picking. Automated harvesting technology is urgently needed to reduce the fruit industry⿿s reliance on manual labor for fruit harvesting and overall production cost. However, there still exist major technological hurdles in developing cost effective and fast harvesting robots. A team of researchers from ARS and Michigan State University in East Lansing, Michigan has developed an integrated robotic system with new, improved fruit picking, dropping and catching mechanisms and by optimizing the robot⿿s planning and control strategies, to enhance its overall harvest efficiency. The improved fruit picking mechanism has had superior performance, while the optimized planning and control algorithms coupled with the fruit dropping and catching mechanisms have greatly improved the robot⿿s harvesting speed. The optimized robotic system was able to pick apples at an average rate of 3.6 s per fruit during 2021 field harvest testing, which is significantly better than the previous version of the robotic system and is also much faster than that reported by other research groups. With further improvements, the technology will help U.S. apple growers mitigate harvesting labor shortage and achieve overall production cost savings
Impacts
(N/A)
Publications
- Lu, Y., Zhang, Z., Lu, R. 2022. Development and preliminary evaluation of a new apple harvest assist and in-field sorting machine. Applied Engineering in Agriculture. 38(1):23-35. https://doi.org/10.13031/aea. 14522.
- Zhang, K., Lammers, K., Chu, P., Li, Z., Lu, R. 2021. System design and control of an apple harvesting robot. Mechatronics. 79. Article 102644. https://doi.org/10.1016/j.mechatronics.2021.102644.
|
Progress 10/01/20 to 09/30/21
Outputs
Progress Report Objectives (from AD-416): 1. Enable a new, efficient, and cost-effective robotic technology, coupled with automated infield sorting and quality tracking technologies, for commercial harvesting of apples. 2. Develop a new imaging technology, based on structured illumination integrated with artificial intelligence and advanced data analytics, with substantially improved capabilities for commercial quality inspection of fruits and vegetables. Approach (from AD-416): Development of enabling technologies for automated fruit harvesting and nondestructive quality inspection during postharvest handling can provide an effective solution to the labor availability and cost issues, and enhance production efficiency, product quality, and thus profitability and sustainability for the specialty crop industries. In recent years, much research has been focused on fruit robotic harvesting, but the progress has been slow and unsatisfactory in meeting industry needs, mainly due to the several key technical hurdles encountered in robotic perception (identifying and localizing fruit), manipulation (reaching out for and picking fruit), and systems integration and coordination. While machine vision technology is widely used for postharvest quality inspection of horticultural products, it still is short of meeting industry expectations in detection of quality-degrading defects and symptoms. This research is therefore aimed at developing a new, cost effective robotic technology for automated harvesting of apples and a new generation imaging technology with substantially enhanced capabilities for quality inspection of fruits and vegetables (e.g., pickling cucumber and tomato) during postharvest handling. Innovative concepts and designs, coupled with artificial intelligence, will be used in the development of the new robotic harvesting system for fruit imaging, detection, localization, and picking. The new robotic system will be integrated with the recently developed apple harvest and infield sorting machine, to enable automated harvesting, sorting, grading and tracking of apples in the orchard. Moreover, a new imaging system, using our newly developed technique on improved reflectance for imaging structures, will be assembled to enable rapid, real-time inspection of harvested horticultural products for quality-degrading defects caused by bruising, physiological disorders, and disease infection. The new knowledge and technologies generated from the research will enable growers and packers/ processors to achieve significant labor and cost savings in harvesting, enhance product marketability, and reduce postharvest product loss. Objective 1: Based on the preliminary work in a prior year, a new version of robotic apple harvester prototype was built. Improvements were made to the design of the tilting and panning mechanisms for more accurate and precise movements of the robot arm in the vertical and horizontal directions. In addition, the rotation mechanism was redesigned, which resulted in a smoother detachment of fruit from trees. Three control strategies for the robot arm movement, including open-loop velocity control, position control, and close-loop velocity control, were evaluated in laboratory with fruit hanging at different positions within the robot�s working space. Results showed that among the three control strategies, the close-loop velocity control had the best performance with the average error of less than 20 mm in reaching the target fruit, which has met the initial expectations for the robot. In addition, preliminary research was conducted to optimize robot path planning for fruit picking and obstacle (tree branches) avoidance. To improve the fruit gripping capability of the robot, three new end effectors were designed and fabricated using a silicon material. The new end effectors were tested for generating fruit-sucking pressure or force. Further, field tests were also conducted for three varieties of apple to determine the forces needed to detach fruit from trees by pulling or twisting movement. The collected picking force information was then used as a guidance in the design of the new end effectors. During the 2020 harvest season, the three new end effectors were evaluated for detaching fruit from trees, and the �straight� end-effector design had overall better performance with 87% picking success rate. New computer algorithms, based on deep learning technique, were developed for detecting fruit from trees. The algorithms were trained and tested with more than 1,500 images collected using a color and depth camera from a commercial orchard in Sparta, Michigan during 2019 harvest season under different natural light conditions (i.e., direct, back light, and overcast). The new algorithm, called suppression mask region-based convolutional neural network (or suppression Mask R-CNN), achieved superior performance in detecting fruit of �Gala� and �Blondee� cultivars, with the overall detection rate of over 90%. The algorithm was integrated with the robot hardware, and it showed promising performance for picking apples in orchard. However, field tests showed that the new algorithm had difficulty detecting those fruit that were heavily occluded by leaves. Moreover, it was found that heavy leaf occlusions could also cause large errors in determining the position of target fruit on trees, thus affecting the successful picking rate. Hence, further efforts are being made to improve the accuracy and efficiency of detecting and localizing target fruit. To facilitate the development, testing and evaluation of a new robotic apple harvester, an artificial orchard environment was constructed to simulate the actual apple orchard. The new artificial orchard consists of artificial trees installed with different sizes of branches coupled with specially-designed artificial fruit stems for hanging real apples. A light system has been installed in the artificial orchard that simulates different sky conditions (i.e., clear sky, overcast and cloudy) for different times of day with adjustable light incident angles. The new artificial orchard environment is also installed with a commercial object localization system composed of eight cameras, which provides quick and accurate information on the three-dimensional position of objects within the working space. This new artificial orchard environment provides a useful platform to test and evaluate how a robotic harvester would operate under the real orchard environment. Objective 2: Data analyses were completed for detection of subsurface bruising in fresh pickling cucumbers, using structured-light imaging (SLI) technique. The original SLI images for 240 �Vlaspick� pickling cucumbers were first processed to obtain two sets of new images, i.e., direct component (DC) which are equivalent to the images that would be acquired by a conventional imaging system under uniform, diffuse illumination, and alternating component (AC) which are unique to SLI and may reveal some hidden features that would otherwise be difficult to detect by DC images. Image enhancement was then performed for the DC and AC images, using a newly developed technique, called fast bi-dimensional empirical mode decomposition (BEMD). Image features were extracted from the enhanced DC and AC images, and top 50 features were selected for establishing classification models based on a machine learning algorithm, called support vector machines. Superior classification results for bruised and normal cucumbers were obtained with the overall classification accuracy of 91%, when DC and AC images were combined. This study demonstrated that SLI can provide an effective means for detecting subsurface bruising of pickling cucumbers. A manuscript from this study was submitted for journal publication. Preliminary tests were conducted to evaluate the performance of a high- speed imaging sensor for real-time acquisition of images from samples under structured lighting with different frame rates (i.e., images per second). These tests showed that this imaging setup would meet our need for building a laboratory SLI-based system for online inspection of horticultural products, such as tomato and cucumber. Record of Any Impact of Maximized Teleworking Requirement: While we have managed to accomplish the research tasks as much as we could during the pandemic, the maximized telework posture has hindered our capability of making full progress for all research objectives. In particular, some of the laboratory activities that needed be done onsite, such as the assembly, testing, and evaluation of our harvesting robot and structured-light imaging system, have not been performed at full capacity due to the in-person restrictions at the worksite imposed under COVID-19. ACCOMPLISHMENTS 01 Development of a new robotic apple harvesting technology. Automated harvesting technology is urgently needed to address labor shortage and increasing labor cost issues facing the multi-billion dollar U.S. apple industry and other tree fruits. While considerable research in robotic harvesting has been reported in recent years, there still exist critical technical challenges of picking fruit from clusters and/or occluded by leaves and branches. In collaboration with researchers at Michigan State University, an ARS researcher at East Lansing, Michigan has developed a new robotic apple harvesting technology, which utilizes an innovative concept of vacuum sucking and rotation, coupled with a simple and effective robot arm movement mechanism, for picking fruit from trees. The new harvesting robot has demonstrated its effectiveness and dexterity in picking fruit from clusters and/or deep in canopy during field testing in 2020, and a patent application for the technology has been filed. With further research, the technology has the potential to change the way of how apples are being harvested and reduce the apple industry�s reliance on manual labor for fruit harvesting.
Impacts
(N/A)
Publications
- Lu, Y., Saeys, W., Kim, M.S., Peng, Y., Lu, R. 2020. Hyperspectral imaging technology for quality and safety evaluation of horticultural products: A review and celebration of the past 20-year progress. Postharvest Biology and Technology. 170. Article 111318. https://doi.org/10.1016/j.postharvbio. 2020.111318.
- Sun, Y., Lu, R., Pan, L., Tu, K., Wang, X. 2020. Assessment of the optical properties of peaches with fungal infection using spatially-resolved diffuse reflectance technique and their relationships with tissue structural and biochemical properties. Food Chemistry. 321. Article 126704. https://doi.org/10.1016/j.foodchem.2020.126704.
- Zhang, Z., Lu, R., Cannayen, I. 2020. A time and motion study for evaluation of apple harvest processes with different harvest methods. Transactions of the ASABE. 63(6):1957-1967. https://doi.org/10.13031/trans. 14144.
- Chu, P., Li, Z., Lammers, K., Lu, R., Liu, X. 2021. Deep learning-based apple detection using a suppression mask R-CNN. Pattern Recognition Letters. 147:206-211. https://doi.org/10.1016/j.patrec.2021.04.022.
- Zhang, Z., Lu, Y., Lu, R. 2021. Development and evaluation of an apple infield grading and sorting system. Postharvest Biology and Technology. 180. Article 111588. https://doi.org/10.1016/j.postharvbio.2021.111588.
- Lu, Y., Lu, R., Zhang, Z. 2021. Detection of subsurface bruising in fresh pickling cucumbers using structured-illumination reflectance imaging. Postharvest Biology and Technology. 180. Article 111624. https://doi.org/ 10.1016/j.postharvbio.2021.111624.
- Lu, Y., Lu, R. 2021. Detection of chilling injury in pickling cucumbers using dual-band chlorophyll fluorescence imaging. Foods. 10(5). Article 1094. https://doi.org/10.3390/foods10051094.
|
|