Recipient Organization
MICHIGAN STATE UNIV
(N/A)
EAST LANSING,MI 48824
Performing Department
BIOSYSTEMS AG EGR
Non Technical Summary
Rising labor costs are threntening the profitability of the United States apple industry. It is imperative to develop cost- and labor-saving technology to maximize the net profit of apple industry stakeholders and ensure the continued competitiveness of the fruit industry. Automated, in-field sorting of harvested fruit, prior to postharvest operations, can achieve economic benefits of hundreds of millions of dollars for apple growers and packers. Currently there is no commerical automation technology for in-field sorting of apples. Although machine vision technology has been adopted for automated fruit sorting at packinghouses, critical challenges exist in developing machine vision systems for orchard application. To fail the gap, building upon previous research, our team propose to develop an AI (artificial intelligence) -enhanced multispectral vision-based in-field apple sorting system. Our long-range goal is to develop machine vision technology for commercial in-field grading and sorting of tree fruits. Specific objectives of the proposed research are to: 1) develop a multispectral imaging system integrated with a screw conveyor for full-surface fruit inspection, 2) develop deep learning models for grading apples according to size, color and surface defects, 3) develop a sorter to efficiently sort apples into three grades , 4) develop user-friendly, open-sourced software programs for system operation and real-time visualization, and 5) perform system integration and conduct field tests and demonstration of the innovative in-field apple sorting machine prototype in commercial orchards.
Animal Health Component
25%
Research Effort Categories
Basic
50%
Applied
25%
Developmental
25%
Goals / Objectives
Theoverall objectiveof this project is to develop an innovative, artificial intelligence-enhanced multispectral vision-based in-field sorting system for apples, with the capabilities of sorting harvested fruit for size, color, and surface defects into three grades (i.e., "Fresh", "Processing", and "Cull"). Our specific objectives are to: 1) develop a multispectral imaging system integrated with a screw conveyor for acquiring images from traveling apples, 2)develop deep learning models for grading apples according to quality conditions, with emphasis on defects, 3) develop a new sorter integrated with the proposed multispectral vision system to sort apples into three grades, 4)develop user-friendly, open-sourced software programs for system operation and real time visualization, and 5)perform system integration to build an in-field apple sorting machine prototype and conduct field tests and demonstration in commercial orchards.
Project Methods
Objective #1: Develop a multispectral imaging system integrated with a screw conveyor for acquiring images from traveling apples.We will design and build a new screw conveyor for fruit transportation that is to be used for online appleinspection by a multispectral vision system.A custom-built multispectral camera will be mounted above the screw conveyor for real-time fruit inspection and operated inside an enclosed chamber to prevent the interference of ambient light. Artificial, uniform lighting will be provided for the imaging scene using strips of LEDs.In-depth experiments will be conducted to optimize imaging settings (e.g., camera-conveyor distance, conveyor speed, framerates) to ensure full-surface fruit inspection within compact working space (short camera-conveyor and fruit traveling distances).Objective #2: Develop deep learning models for grading apples according to quality conditions, with emphasis on defects. Multispectral images will acquired from apple samples of two popular varieties ("Gala" and "Golden Delicious") from local apple packinghouses and orchards, and apples in images will be manually labeled into three quality grades ("Fresh", "Processing", and "Cull") according to quality conditions. Modeling experiments will be conducted on a suite of state-of-the-art deep learning model architectures, with emphasis on convolutional neural networks (CNNs) for image classification, especially those efficient in model training and inference and suited for real-time application.To facilitate creating large-scale datasets to harness the power of deep learning models, we will investigate semi-/non-supervised learning techniques for image labeling and generation. Performance metrics in terms of applegrading accuracy and inference times will be obtained, and models achieving the best tradeoff will be deployed for further in-field fruit grading. Both datasets and models will be released in open data repositories.Objective #3: Develop a new sorter integrated with the multispectral vision system for sorting apples into three quality grades. In the sorter design,we will prioritize the use of DC-powered rotary or linear solenoids that efficiently and gently push the product towards the right exit, due to the robustness and simplicity in control.We will prefer single 3-position solenoids over two regular 2-position solenoids to best save space needed for the sorting area. In addition to using electric solenoids,pneumatic sorting mechanismswill also be experimented forfast responeses with short cycle times, and 3D printing techniques will be explored to rapid prototyping of sorter designs.Laboratory tests will be conducted to assess the performance of different sorters in terms of sorting repeatability and accuracy.Objective #4: Develop user-friendly, open-sourced software programs for system operation and real time visualization. Qt will be utilized as the main framework to develop the software interface. The interface willthe major functionalities including interfacing with the custom-built multispectral camera,other data acquisition devices and the sorter, the selection and deployment of deep learning models trained in PyTorch (or TensorFlow), and performing real-time fruit trackingand visualization of fruit segmentation and grading as well as saving image data and grading results on demand. The software will be eventually made open-accessible to the research community.Objective #5: Perform system integration to build an in-field sorting machine prototype and conduct field tests and demonstration in commercial apple orchards.We will assemble all modular hardware components and integrate hardware and software programs into amachine prototype, which expects to a standalone mobile system and will permit further integration onto a harvest-aid platform in future efforts. Systematic testing of the machine prototype will be initially conducted in laboratory conditions. Thereafter, in-field tests and demonstration will be conducted n commercial orchards in the Fruit Ridge region of Michigan. Sortedapples will be collectedand visually examined to determine the sorting accuracy and the percentages of bruised apples.