Source: WASHINGTON STATE UNIVERSITY submitted to NRP
INTEGRATED SYSTEMS RESEARCH AND DEVELOPMENT IN AUTOMATION AND SENSORS FOR SUSTAINABILITY OF SPECIALTY CROPS
Sponsoring Institution
National Institute of Food and Agriculture
Project Status
COMPLETE
Funding Source
Reporting Frequency
Annual
Accession No.
1016172
Grant No.
(N/A)
Cumulative Award Amt.
(N/A)
Proposal No.
(N/A)
Multistate No.
W-3009
Project Start Date
Oct 1, 2018
Project End Date
Sep 30, 2023
Grant Year
(N/A)
Program Code
[(N/A)]- (N/A)
Recipient Organization
WASHINGTON STATE UNIVERSITY
240 FRENCH ADMINISTRATION BLDG
PULLMAN,WA 99164-0001
Performing Department
Center for Precision & Automation Ag Systems (CPAAS)
Non Technical Summary
Increased demand for safe and high-quality food, pressure from global competition, and the need to preserve sustainability in natural resources and the environment, represent formidable challenges for specialty crop production in the U.S. Additionally, the decline in availability of skilled and unskilled farm labor is a continuing trend with very significant negative impact on specialty crop production. Producers and processors are urgently seeking new advancements in tools, methodologies, and systems that will aid them during production, harvesting, sorting, storing, processing, packaging, marketing, and transportation while maintaining competitiveness in production costs. There is continuing need for sufficient and effective sensors, optimized assistive mechanization, and automation systems for specialty crops (fruits, vegetables, tree nuts, dried fruits and nursery). This is because many of the underlying biological processes related to quality and condition of fruits and vegetables are not readily transferred into engineering concepts. Biological variability, coupled with variable environmental factors, creates challenges and opportunities for developing sensors and automation systems for effective implementation at various stages of the production, harvest, and postharvest handling chain. Additionally, obtaining measurements of internal biological factors by using external, nondestructive sensors is not simple. Such devices or processes must adapt to a wide variation in shape, size, maturity, and other parameters of the commodity being processed. Although modern crop architectures, systems, and growing practices are slowly being upgraded to accommodate mechanized and automated production, a large portion of the production system, such as orchards, are still based on conventional, labor-intensive production practices. In contrast to grain crop production, it is a challenge for any single specialty crop sector to afford the cost of research, development, and commercialization of this complex level of automation, due to its relative small scale. It is thus important for public entities to assist this economically vital agricultural sector with integrated research and development in sensing, mechanization, and automation using an integrated approach.
Animal Health Component
70%
Research Effort Categories
Basic
30%
Applied
70%
Developmental
(N/A)
Classification

Knowledge Area (KA)Subject of Investigation (SOI)Field of Science (FOS)Percent
4021119202040%
4041119202050%
2052410106010%
Goals / Objectives
Develop specialty crop architectures and systems that are more amenable to mechanized production Study interactions between machinery and crop to provide basis for creating optimal mechanical and/or automated solutions for specialty crop production Develop sensors and sensing systems which can phenotype and measure and interpret quality parameters Design and evaluate automation systems which incorporate varying degrees of mechanization and sensors to assist specialty crop industries with labor, management decisions, and reduction of production costs Develop collaboration and work in partnership with equipment and technology manufacturers to commercialize and implement the outcomes of this project Adapt biological concepts associated with specialty crop production, harvest, and postharvest handling into quantifiable parameters which can be sensed
Project Methods
This multi-state project aims at coordinating the participants to assist an economically vital agricultural sector of specialty crop production with integrated research, education, and development in crop, sensing, mechanization, and automation in a system approach. A key and continuing theme in this project is a multi-dimensional integration of research activities; integration from the adaptation of biological concepts to measurable parameters through the commercialization of new products; and integration within each objective among different specialty crops. Although the objectives seem sequential in execution, we anticipate a swirl of concurrent activities centered about and driven by the needs of specialty crop growers for automation of growing, harvesting and postharvest operations. A major task for the project leadership is to facilitate communication and collaboration among the members of this project, and between project members and other stakeholders in specialty crop agriculture. A technical committee will be formed by project leaders for the contributing states, an administrative advisor, and CSREES representatives. Voting membership includes all persons with contributing projects. A Chairperson, Vice Chairperson and Secretary will be elected from the voting membership at the first authorized committee meeting after the project has been approved. The Chair, Vice-Chair and Secretary will serve two years, if so desired by the membership. They will be responsible for meeting arrangements, annual reports and implementation of the Outreach Plan. A set of milestones for this project period has been defined as follows:Year 1 (2018-19): Identify emerging technologies in specialty crop and forecast future research needsYear 2 (2019-20): Organize systems-driven research and extension teams on each focus area and develop technology elements; obtaining industrial and federal grants to support team researchYear 3 (2020-21): Create testbeds to transfer new technologies developed by the teams; filing patent application for developed technologiesYear 4 (2021-22): Organizing workshops and helping industry to commercialize and apply the technologiesYear 5 (2022-23): Continue supporting industry to commercialize and apply the technologies; project ends (renewal in place)

Progress 10/01/19 to 09/30/20

Outputs
Target Audience:Scientists, specialty crop producers, agricultural equipment manufacturers and technology providers who are seeking new solutions to improve specialty crop production and efficiency. It also includes the end-users of developed technologies and fellow researchers and general public. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided?This project provided a large group of (4) post-doctoral research associates, (14) graduate students, (2) exchange graduate students and (3) visiting scholars from foreign institutes an opportunity to be trained by working in a collaborative working environment to develop their wide-spectrum of research skills and professionalism. How have the results been disseminated to communities of interest?Research outcomes have been presented at international professional conferences and regional industry meetings, including the 6th IFAC AgriControl Conference (December 2019, Sydney, Australia), the Annual International Meeting of ASABE (virtual, July 2020), the 21st IFAC World Congress (virtual, July 2020). In addition, we have hosted more than 3 groups of stakeholders (e.g. growers, scientists, and policy makers) from Northwest Pacific Region of US for site visits, and multiple (virtual) seminars to scientists and students in the other parts of the world, including an Online International Training Program on Agriculture 4.0 organized by MPKV of India. Various research outcomes have been demonstrated or introduced to these visitors or seminar participants. What do you plan to do during the next reporting period to accomplish the goals?Continue the integrated system research focusing on developing mechanization and automation technologies for specialty crop production and precise management.

Impacts
What was accomplished under these goals? WA team has worked on multiple aspects of sensing and automation technologies, including robotic devices, crop sensing systems, advanced spraying technologies, for supporting more effective specialty cropproductions. The accomplishments in the past year including (but not limited to): (1) optimized a vision-based fruit orientation estimation and obstacle avoidance system for a 12-armed robotic apple harvester; (2) developed a machine-vision system for detecting flower clusters (with 86% accuracy) and estimating flower density (with 84% accuracy) in apple and cherry orchards, which will be used in developing an automated thinning and pollination systems; (3) developed a machine-vision and robotic system for pruning apple trees, which was tested in the laboratory environment with more than 90% accuracy in reaching and cutting target branches; (4) developed and tested an automated green shoot thinning mechanism for grapevines (with 1.4 cm cordon following accuracy) that used a deep-learning-based machine vision system for detecting vine cordon and shoots and for estimating trajectory fitting model which reached a 80% cordon trajectory estimation accuracy;; (5) further optimized a fixed spray system (i.e. solid set chemical delivery system configurations and pertinent field validation for tree fruit orchards; (6) improved and conducted field testing of an Internet-of-Things (IoT) enabled Crop Physiology Sensing System (CPSS) for apple fruit surface temperature monitoring and real-time actuation of modified evaporative cooling system for heat stress management in fresh market apple cultivars; and (7) developed hyperspectral imaging based spectral sensing for grapevine leafroll disease detection at asymptomatic stages and pertinent data mining for identification of salient spectral bands.

Publications

  • Type: Journal Articles Status: Published Year Published: 2020 Citation: Bahlol, H., A. Chandel, G.-A. Hoheisel and L. R. Khot*. 2020. Developing understanding on orchard sprayer air-assists and volume output patterns using smart spray analytical system. Crop Protection, 127: 104977.
  • Type: Journal Articles Status: Published Year Published: 2020 Citation: Bahlol, H. Y., A. Chandel, G.-A. Hoheisel, and L. R. Khot*. 2020. Smart spray analytical system for orchard sprayer calibration: a-proof-of-concept and preliminary results. Transactions of the ASABE, 62(6): 29-35.
  • Type: Journal Articles Status: Published Year Published: 2020 Citation: Davidson, J., S. Bhusal, C. Mo, M. Karkee, and Q. Zhang. 2020. Robotic Manipulation for Specialty Crop Harvesting: A Review of Manipulator and End-Effector Technologies. Global Journal of Agriculture and Allied Sciences, 2(1), 25-41.
  • Type: Journal Articles Status: Published Year Published: 2020 Citation: Fu, L., Y. Majeed, X. Zhang, M. Karkee and Q. Zhang, 2020. Faster R-CNN-based apple detection in dense-foliage fruiting-wall trees using RGB and depth features for robotic harvesting. Biosystems Engineering, 197: 245-256.
  • Type: Journal Articles Status: Published Year Published: 2020 Citation: Gao, F., L. Fu, X. Zhang, Y. Majeed, R. Li, M. Karkee and Q. Zhang. 2020. Multi-class fruit-on-plant detection for apple in SNAP system using Faster R-CNN. Computers and Electronics in Agriculture, 176, 105634.
  • Type: Journal Articles Status: Published Year Published: 2020 Citation: Gao, Z., R. A. Naidu, Q. Zhang* and L. R. Khot*. 2020. Early detection of grapevine leafroll disease in a red-berried wine grape cultivar using hyperspectral imaging. Computers and Electronics in Agriculture, 179, 105807
  • Type: Journal Articles Status: Published Year Published: 2020 Citation: Khot, L. R.. 2020. Transitioning from precision to decision horticulture: technology landscape. ISHS Acta Horticulturae 1279, XXX International Horticultural Congress IHC2018: VII Conference on Landscape and Urban Horticulture, IV Conference on Turfgrass Management and Science for Sports Fields and II Symposium on Mechanization, Precision Horticulture, and Robotics, 43:1
  • Type: Journal Articles Status: Published Year Published: 2020 Citation: Majeed, Y., J. Zhang, X. Zhang, L. Fu, M. Karkee, Q. Zhang, and M.D. Whiting, 2020. Deep learning based segmentation for automated training of apple trees on trellis wires. Computers and Electronics in Agriculture, 170. Article 105277.
  • Type: Journal Articles Status: Published Year Published: 2020 Citation: Majeed, Y., M. Karkee, Q. Zhang, L. Fu, and M.D. Whiting, 2020. Determining grapevine cordon shape for automated green shoot thinning using semantic segmentation-based deep learning networks. Computers and Electronics in Agriculture, 171. Article 105308.
  • Type: Journal Articles Status: Published Year Published: 2020 Citation: Majeed, Y., M. Karkee, and Q. Zhang, 2020. Estimating the trajectories of vine cordons in full foliage canopies for automated green shoot thinning in vineyards. Computers and Electronics in Agriculture, 176. Article 105671.
  • Type: Journal Articles Status: Published Year Published: 2020 Citation: Zhang, J., M. Karkee, Q. Zhang, X. Zhang, Y. Majeed, L. FU, and S. Wang, 2020. Multi-class object detection using faster R-CNN and estimation of shaking locations for automated shake-and-catch apple harvesting. Computers and Electronics in Agriculture, 173. Article 105384.
  • Type: Journal Articles Status: Published Year Published: 2020 Citation: Zhang, X., L. He, J. Zhang, M.D. Whiting, M. Karkee and Q. Zhang, 2020. Determination of key canopy parameters for mass mechanical apple harvesting using supervised machine learning and principal component analysis (PCA). Biosystems Engineering, 193: 247-263.
  • Type: Journal Articles Status: Published Year Published: 2020 Citation: Zhang, X., He, L., Karkee#, M., Whiting, M. D., and Zhang, Q. 2020. Field Evaluation of Targeted Shake-and-Catch Harvesting Technologies for Fresh Market Apple. Transactions of the ASABE, 2020.
  • Type: Journal Articles Status: Published Year Published: 2020 Citation: Ranjan, R., L. R. Khot, R. Troy Peters, M. R. Salazar-Gutierrez and G. Shi?. 2020. In-field crop physiology sensing aided real-time apple fruit surface temperature monitoring for sunburn prediction. Computers and Electronics in Agriculture, 157: 105558.
  • Type: Journal Articles Status: Published Year Published: 2020 Citation: Santiago, W. E., N. J. Leite, B. J. Teruel, M. Karkee, and C. A.M. Azania. 2019. Evaluation of bag-of-features (BoF) technique for weed management in sugarcane production. Australian Journal Crop of Science, 13(11):1819-1825.
  • Type: Journal Articles Status: Published Year Published: 2020 Citation: Sinha, R., R. Ranjan, H. Y. Bahlol, L. R. Khot, G.A. Hoheisel and M. Grieshop. 2020. Development and performance evaluation of a pneumatic spray delivery based solid set canopy delivery system for high-density apple orchard. Transactions of the ASABE, 62(6): 37-48.
  • Type: Journal Articles Status: Published Year Published: 2020 Citation: Sinha, R., R. Ranjan, G. Shi?, G.-A. Hoheisel, M. Grieshop and L. R. Khot. 2020. Solid set canopy delivery system for efficient agrochemical delivery in modern architecture apple and grapevine canopies. Acta Horticulturae 1269: II International Symposium on Innovative Plant Protection in Horticulture, 277-286.
  • Type: Journal Articles Status: Published Year Published: 2020 Citation: Wang, B., R. Ranjan, L. R. Khot and R. Troy Peters. 2020. Smartphone application?enabled apple fruit surface temperature monitoring tool for in?field and real?time sunburn susceptibility prediction. Sensors, 20, 608.
  • Type: Theses/Dissertations Status: Published Year Published: 2020 Citation: Gao, Zongmei (2020). Spectral imaging based non-contact detection of biotic and abiotic stress in berry crops. Ph.D. Dissertation. April 2020, Washington State University.
  • Type: Theses/Dissertations Status: Published Year Published: 2020 Citation: Majeed, Yaqoob (2020). Machine Vision System for the Automated Green Shoot Thinning in Vineyards. Ph.D. Dissertation. April 2020, Washington State University.
  • Type: Theses/Dissertations Status: Published Year Published: 2020 Citation: Zhang, Xin (2020). Study of Canopy-Machine Interaction in Mass Mechanical Harvest of Fresh Market Apples. Ph.D. Dissertation. March 2020, Washington State University.


Progress 10/01/18 to 09/30/19

Outputs
Target Audience:Specialty crop producers, agricultural equipment manufacturers and technology providers who are seeking new solutions to improve specialty crop production and efficiency. It also includes the end-users of developed technologies and fellow researchers and general public. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided?Graduate students (22 total), post-doctoral research associates (3 total), exchange graduate students from foreign institutes (2 total) and visiting scholars (6 total) working on the project took advantages of opportunities to develop wide-spectrum of research skills and professionalism in a collaborative working environment. How have the results been disseminated to communities of interest?Research outcomes have been presented at international professional conferences, regional industry meetings and field days including Annual International Meeting of American Society of Agricultural and Biological Engineers and International Conference on Precision Agriculture. We have organized a CPAAS Agricultural Technology Expo in July 2019 with about 100 attendees from agri-business, local growers, commodity group representatives, researchers and media. Event theme was Automation in Specialty Crop Production. Besides plenary session talks (3 industry experts and 3 WSU faculties), CPAAS affiliates had 27 poster presentations and 7 technology demonstrations. We have also organized two workshops on UAS applications in agriculture to local Agri-business industry, growers, researchers and policy makers in 2019, with 27 and 16 attendees, respectively. In addition, we have hosted more than 10 groups of stakeholders (e.g. growers, scientists, engineers, students and policy makers) from Washington and other parts of the country as well as from around the world. Various technologies being developed with the project have been demonstrated to these visitors/stakeholders. What do you plan to do during the next reporting period to accomplish the goals?Continue the integrated system research focusing on developing automation and sensing technologies for specialty crop production and precise management.

Impacts
What was accomplished under these goals? WA team has worked on multiple aspects of sensing and automation technologies, including robotic devices, crop sensing systems, advanced spraying technologies, for supporting more effective specialty crop, from vegetables, grapevines, to fruit tree, productions. The accomplishments in the past year including (but not limited to): (1) developed and integrated a vision-based fruit orientation estimation and obstacle avoidance system to a 12-armed robotic apple harvester, and evaluated in commercial orchards environment; (2) improved and further evaluated targeted shake-and-catch fresh-market harvesting system and reached a 90% or higher fruit removal efficiency with 10% or less fruit damage in formally trained orchards for selected apple varieties; (3) modified the self-leveling mechanism and improved the vision weed detection system of a self-propelling weeding robot and further evaluated it on commercial carrot and onion fields. This improved machine system achieved a 99% or higher weed detection accuracy and 2 mm or less spraying accuracy while travelling on uneven natural terrain; (4) conceptualized an automated green shoot thinning mechanism for grapevines using a deep-learning-based machine vision system for detecting vine cordon and shoots and developed a trajectory fitting model to represent cordon position and orientation which reached a 80% cordon trajectory estimation; (5) further optimized a solid set chemical delivery system (SSCDS) for vineyards and orchards applications through creating a reservoir sub-system to maintain the pressure for achieving more precise application control and reducing drift losses to the air; (6) developed an Internet-of-Things (IoT) enabled Crop Physiology Sensing System (CPSS) through encompassing a thermal-RGB imager integrated with a single board computer for monitoring apple fruit surface temperature and controlling SSCDS performing automated evaporative cooling; and (7) conducted a preliminary field trial of spraying cellulose nanocrystal (CNC) for frost protection at two commercial orchards in 2019 Spring, and found it could increase bud cold hardness by up to 5 °C and did not observe any adverse impact on trees nor fruits.

Publications

  • Type: Book Chapters Status: Published Year Published: 2019 Citation: Zhang, Q., M. Karkee, and A. Tabb, 2019. The Use of Agricultural Robots in Orchard Management. In Robotics and Automation for a More Sustainable Agriculture (Editor: John Billingsley); rXiv preprint arXiv:1907.13114.
  • Type: Book Chapters Status: Published Year Published: 2019 Citation: Karkee, M., J. Gord�n, B. Sallto and M. Whiting, Optimizing fruit production efficiencies via mechanization. 2019. In Achieving sustainable cultivation of temperate zone tree fruits and berries, Volume 1 - Physiology, genetics and cultivation (Editor: Dr Greg Lang); Burleigh Dodds Science Publishing.
  • Type: Books Status: Published Year Published: 2019 Citation: Zhang, Q. 2019. Basics of Hydraulic Systems (2nd Edition). CRC Press, (324 pp).
  • Type: Conference Papers and Presentations Status: Published Year Published: 2019 Citation: Karkee, M. 2019. Agricultural Robotics: Global Perspective and Local Context (Invited Speech), Annual Convention of Indian Society of Agricultural Engineers, Jan 28-30, 2019.
  • Type: Conference Papers and Presentations Status: Published Year Published: 2019 Citation: Karkee, M. 2019. Automation and Robotics in Agriculture (Key-note Speech), Asian-Australian Conference on Precision Agriculture, Ludhiana, India, October 14-17, 2019.
  • Type: Conference Papers and Presentations Status: Published Year Published: 2018 Citation: Karkee, M. 2018. Agricultural Automation and Robotics - Challenges and Opportunities (Key-note Speech), Australasian Conference on Robotics and Automation, Christchurch, New Zealand, Dec 3-6, 2018.
  • Type: Journal Articles Status: Published Year Published: 2019 Citation: Bhusal, S., K. Khanal, S. Goel, M. Karkee, and M. Taylor. 2019. Bird deterrence in a vineyard using an unmanned aerial system (UAS). Transactions of the ASABE; 62(2): 561-569 (doi: 10.13031/trans.12923).
  • Type: Journal Articles Status: Published Year Published: 2019 Citation: Chakraborty, M., L.R. Khot, S. Sankaran, and P. Jacoby. 2019. Evaluation of mobile 3D light detection and ranging based canopy mapping system for tree fruit crops. Computers and Electronics in Agriculture,158: 284-293 (doi: 10.1016/j.compag.2019.02.012).
  • Type: Journal Articles Status: Published Year Published: 2019 Citation: Chandel, A., L.R. Khot, Y. Osroosh, and R.T. Peters. 2018. Thermal-RGB imager derived in-field apple surface temperature estimates for sunburn management. Agricultural and Forest Meteorology, 253-254: 132-140 (doi: 10.1016/j.agrformet.2018.02.013).
  • Type: Journal Articles Status: Published Year Published: 2019 Citation: He, L., X. Zhang, Y. Ye, M. Karkee, and Q. Zhang. 2019. Effect of shaking location and duration on mechanical harvesting of fresh market apples. Applied Engineering in Agriculture; 35(2): 175-183 (doi: 10.13031/aea.12974).
  • Type: Journal Articles Status: Published Year Published: 2019 Citation: Hohimer, C.J., H. Wang, S. Bhusal, J. Miller, C. Mo, and M. Karkee. 2019. Design and field evaluation of a robot apple harvesting system with 3D printed soft-robotic end-effector. Transactions of the ASABE; 62(2): 405-414 (doi: 10.13031/trans.12986).
  • Type: Journal Articles Status: Published Year Published: 2019 Citation: Khanal, K., S. Bhusal, M. Karkee, P. Scharf, and Q. Zhang. 2019. Design of improved and semi-automated red raspberry cane bundling and tying machine based on the field evaluation results. Transactions of the ASABE. 62(3): 821-829 (doi: 10.13031/trans.12973).
  • Type: Journal Articles Status: Published Year Published: 2019 Citation: Osroosh, Y., L.R. Khot, and R.T. Peters. 2019. Detecting fruit surface wetness using a custom-built low-resolution thermal-RGB imager. Computers and Electronics in Agriculture, 157: 509517 (doi: 10.1016/j.compag.2019.01.023).
  • Type: Journal Articles Status: Published Year Published: 2019 Citation: Pena Quinones, A.J., M. Keller, M.R. Salazar-Gutierrez, L.R. Khot, and G. Hoogenboom. 2019. Comparison between grapevine tissue temperature and air temperature. Scientia Horticulturae, 247: 407420 (doi: 10.1016/j.scienta.2018.12.032).
  • Type: Journal Articles Status: Published Year Published: 2019 Citation: Ranjan, R., G. Shi, R. Sinha, L. R. Khot, G. Hoheisel, and M. Grieshop. 2019. Automated solid set canopy delivery system for large scale spray applications in perennial treefruit crops. Transactions of the ASABE, 62(3): 585-592 (doi: 10.13031/trans.13258).
  • Type: Journal Articles Status: Published Year Published: 2019 Citation: Sharda, A., M. Karkee, G. Hoheisel, and Q. Zhang. 2019. Design and evaluation of solid set canopy delivery system for spray application in high-density apple orchards. Applied Engineering in Agriculture 35(5): 751-757 (doi: 10.13031/aea.12512).
  • Type: Journal Articles Status: Published Year Published: 2019 Citation: Sinha, R., L.R. Khot, A. Rathnayake, Z. Gao, and N. Rayapati. 2019. Visible?near infrared spectroradiometry-based detection of grapevine leafroll-associated virus in a red?fruited wine grape cultivar. Computers and Electronics in Agriculture, 162: 165-173 (doi: 10.1016/j.compag.2019.04.008).
  • Type: Journal Articles Status: Published Year Published: 2019 Citation: Sinha, R., L.R. Khot, GA. Hoheisel, M. Grieshop, and H.Y. Bahlol. 2019. Feasibility of a solid set canopy delivery system for efficient agrochemical delivery in vertical shoot positioning trained vineyards. Biosystems Engineering, 179: 59-70 (doi: 10.1016/j.biosystemseng.2018.12.011).
  • Type: Journal Articles Status: Published Year Published: 2018 Citation: Zhang, J., L. He, M. Karkee, Q. Zhang, X. Zhang, and Z. Gao, 2018. Branch detection for apple trees trained in fruiting wall architecture using depth features and Regions-Convolutional Neural Network (R-CNN). Computers and Electronics in Agriculture. 155: 386-393 (doi: 10.1016/j.compag.2018.10.029).