Non Technical Summary
In the US alone, 3 billion pounds of consumed fresh tomatoes are grown in greenhouse farms. These farms improve food security, require less land, and consume less resources; however, for these farms to operate, large labor forces are needed to maintain, harvest, and package the tomatoes. These jobs, though paying well above minimum wage, do not attract local workers and force farmers to use immigrant labor. This lack of local labor availability and unsupportive governmental regulations has led to high turnover rates and high training costs. This prevents growers from growing their full potential and limits their ability to expand their operations. With new governmental regulations increasing labor costs by 35% in Ontario (one of the largest greenhouse growing regions in North America) and major labor availability issues in the US, growers are in desperate need of solutions. Many are actively seeking out automation to solve this issue, but there are currently no solutions for them to use. This high rising need, with a current absence of a solution, makes the development of greenhouse automation crucial for the success of this industry.With harvesting as the largest expenditure for greenhouse labor, a tomato harvesting robot is proposed. Previous researchers have attempted to develop a tomato harvesting robot for this environment, but none has yet to enter the market. The main reason for this has been a lack of accuracy and speed in computer vision algorithms; however, in the last five years, convolutional neural networks (CNN) have enabled this once impossible task to become possible. By applying CNNs to tomato harvesting, a higher level of accuracy will be reached, thereby making the creation of a tomato harvesting robot feasible.

Classification

Knowledge Area (KA)	Subject of Investigation (SOI)	Field of Science (FOS)	Percent
402	1460	2020	100%

Knowledge Area
402 - Engineering Systems and Equipment;

Subject Of Investigation
1460 - Tomato;

Field Of Science
2020 - Engineering;

Goals / Objectives
The proposed research aims to create an effective way for greenhouses to grow more produce at a lower cost through development of a tomato harvesting robot. In Phase I, a vision system will be created through research and utilization of CNNs and applying them to the greenhouse industry. A deep CNN will be designed and trained to identify ripe grape tomatoes and determine the harvesting locations.1) A properly labelled dataset of over 1000 images for machine learning training of grape tomato determination.2) An object detection system that can output the proper grape tomato locations with an mAP of over 803) An object detection vision system that can output the proper three-dimensional coordinates for harveting location with a mAP of 80 or above.4) A fully functional vision control algorithm that is fed a stereo camera input image and outputs a 3D coordinate for the cut location with an mAP > 80

Project Methods
Image Dataset Creation Since machine learning based computer vision is highly dependent on the quality of its training data, the first critical task is to create the necessary image datasets for proper training, validation, and testing.Collecting these images will require the acquisition of thousands of images of tomato plants in varying light under all types of greenhouse conditions. To perform such a task, an image gathering rig will be created that includes a stereo camera and a lighting source. Following the acquisition of the images, the team will need to carefully label every image by hand with bounding boxes and labels. For optimal results, these images will be collected at commercial greenhouses and clusters will be labeled based on grower feedback for ripe and unripe clusters.Because YOLOv2 utilizes "context clues" of pixels outside the bounding box, the dataset will include negative training images to ensure that the harvesting locations are determined only for ripe and nearly ripe tomatoes.For this dataset, 20% of the images in each category will not be used in training the neural network and will be used for testing the algorithms. When selecting the 20% of each category, a random sampling will be taken. This will allow for the test images to have variation in the time of day, and greenhouse location, which will ensure the generalization of the algorithms for proper operation in all greenhouses.CNN Training With a properly labelled dataset, training and testing for object detection will begin. We will start by training the YOLOv2 detector with the Darknet-19 classifier because the literature review revealed this to be one of the fastest, most accurate systems. Transfer learning will be used because to train a CNN from scratch takes millions of images. Since there is an architecture that could provide promising results with some alteration, it was deemed more efficient to build off work that had already been performed, than to rebuild everything from scratch.During this task, the YOLO9000 CNN will be retrained on the YOLOv2 architecture as detailed in (Redmon and Farhadi 2016) to identify ripe clusters and cutting ROIs. This will occur through transfer learning and will occur using the Darknet open source neural network framework. During the retraining process, parameters such as batch size, subdivisions, learning rate, decay rate, will be varied, to bring the retrained network to the highest accuracy possible.Data Analysis and Interpretation To evaluate its performance, the trained algorithm will be run on the test images, and the output bounding boxes that label and identify the objects in the image will be compared with the ground truth labeled images. The test images will have not have been used for training and therefore will simulate the algorithm operating on a new environment. After running the algorithm on these images, a precision recall (PR) curve will be calculated. This curve will then be utilized to evaluate the effectiveness of the algorithm, including the determination of the mAP. By comparing the PR curves of different CNNs trained using different hyperparameters, patterns between output results and hyperparameters will be identified and these insights will improve hyper parameter tuning. For each new CNN, a different 20% of the images will be segmented for the test images. This ensures that the algorithm is not being accidentally overfitted to the test data by the human evaluators.In addition to this quantitative evaluation of the algorithm, a qualitative evaluation of the algorithm will also occur. This analysis will primarily be focused on the error images, and will be focused on understanding why these errors occurred, with a strong emphasis on false positive errors. This is because a human supervisor can always harvest missed tomatoes, but cutting an unripe cluster is lost revenue. From these evaluations, insights will be gained to determine if changing the architecture design or changing the dataset labeling is necessary.Wrapping SystemIn order to wrap the system, four main pieces must be performed: Integrating the camera input to CNN1, Parallelizing the depth mapping with the CNN analysis, and outputting the proper 3D coordinates. In order to perform these four tasks, various libraries like OpenCV will be used for image management and interfacing.The primary evaluation of this task is whether it will be capable of accepting a stereo camera feed and output a 3D harvesting location. Upon successful completion of this task, the speed of the wrapping will be evaluated. This will occur by timing each of the individual steps previously performed in above tasks. By using the time for CNN1, and Depth mapping and the total time, we will be able to determine the amount of time the wrapping portions take. The wrapping system will continually be improved, until it reaches the criterion for acceptance. In addition, the full system will be evaluated for its mAP on test images, but since every previous task will have reached its mAP metric, the integrated system should have no issues.