Progress 09/01/19 to 10/31/20

Outputs
Target Audience:The target audience for this project are commercial pineapple growers. Initially we're targeting growers in Hawaii and Latin America. Later we would focus on growers in Southeast Asia. Pineapple is an important and symbolic crop for Hawaii. While U.S.-grown pineapple has a reputation for superior quality and flavor, pineapple production has declined significantly due to foreign competition, and most U.S.-consumed pineapple (especially on the mainland) is supplied from Central America. Global production of pineapple was 29.9M metric tons in 2018, a 9% increase vs. 2017. Intellinair is engaged with producers in several countries: Costa Rica, the number one pineapple producer (3.4M metric tons); Brazil, the number three producer (2.7M metric tons); the U.S., the number 28 producer (158M metric tons); and Honduras, the number 36 producer (81M metric tons).[i] Most production practices used worldwide were developed in Hawaii. The University of Hawaii at Manoa's College of Tropical Agriculture and Human Resources (UH) is a key center for research and technology development for this crop. For instance, in 2015, Prof. Robert Paull (part of the project team) and Nancy Chen were part of a team that sequenced the genes in the pineapple genome for the first time.[ii] Prof. Paull has also authored and edited books on pineapple production and post-harvest handling.[iii],[iv] Improving grower efficiency through the capture and analysis of aerial data will reduce production costs and in-field losses and incent growers to increase acreage. Additionally, the analytics developed would potentially apply to other specialty crops. One of Intellinair's collaborators, Agrícola Famosa (a large producer of fruit in Brazil), believes that this technology could apply to melons. In Hawaii, as on all pineapple plantations around the world, growers are confronted with spatial and temporal flowering variations. Natural flowering of pineapple occurs as the days become shorter and the nights cooler. While up to 100% of the plants may flower naturally, such flowering occurs over a period of several weeks. This uneven flowering causes fruit to mature across the field at different times, requiring several harvests to maximize yield. To "force" pineapple plants to flower in a field uniformly, growers spray a compound that produces ethylene (Ethephon). This spraying is carried out at night when the plants are mature enough to produce a high-quality fruit of the desired size. Different fields are forced at two-week intervals to achieve fruit production year-round. However, if significant natural flowering occurs before the field is forced, fruit will mature over a number of weeks necessitating multiple harvest passes at increased cost. Alternatively, the grower may not conduct a successive harvest if there is insufficient fruit at the desired stage to cover costs, which leads to losses as good fruit rots in the field. Today, growers are only able to survey the progress of pineapple flowering from the edges of the field - walking through the field is difficult as the plants grow very close together at a rate of ~30,000 plants per acre. As a result, spatial and temporal variation of fruit maturity in a field due to natural flowering is subjectively estimated, making it difficult to ensure proper harvest scheduling and frequency. Over the last 20 years, the shift of consumer demand towards fresh fruit has resulted in a dramatic change in the varieties of pineapple grown worldwide. The newer, low acid MD1 (PRI 73-50) and MD2 (PRI-114) varieties are more susceptible to natural flowering ahead of planned chemical flower forcing. This inability to control harvest timing and frequency caused by natural flowering results in significant fruit losses and higher harvesting costs. Pineapple growers want to know the spatial and temporal distribution of flowering and uniformity of fruit maturation in a field. With this knowledge, they can plan a harvest schedule that achieves the highest yield with the lowest number of harvest passes while minimizing in-field losses. If only 10% of the plants in a field show natural flowering ahead of forcing (3,000 potential fruit) they may not harvest as the cost to do so is greater than the profit obtained. However, if 30% of the plants naturally flower before forcing, then another harvest cycle may be economically justified. A grower with good data on the temporal and spatial distribution of flowering and stage of fruit maturity will schedule harvests to achieve maximum yield with the least field loss from fruit that is too green or too ripe. The proposed capture and analysis of field-wide aerial imagery can help growers solve the uneven flowering problem. Other problems can also be improved through future extensions of the technology. Harvesting, preparation, and marketing represent about 30% of production costs. Eliminating a second or third harvest by synchronizing flowering and fruit maturation reduces production and shipping costs by 7% to 8%. Precision application of nutrients and pesticides could further reduce production costs by 5% and improve yields by 5% to 8%. Crop management through aerial imagery analysis can improve delivered costs by 15%, significantly closing the price gap between imported and domestic pineapple in the U.S. These data and estimates were developed through discussions with Dole Hawaii staff. Pineapple production on large farms is carried out in fields that are divided into ~3 acre blocks with 30,000 plants per acre; about 10 acres are forced to flower at any one time. Pineapple flowering is distinct, starting when the center of the plant turns red. This can only be seen when viewed directly from above the plant. Figure 1 Left: Redness of a newly flowering plant. Right: Fruit developed from naturally induced flowering and a young flower bud from chemical-induced flowering. [i] FAOSTAT. Food and Agriculture Organization of the United Nations. http://www.fao.org/faostat/en/?#data [ii] "UH Manoa researchers help unlock pineapple genome," University of Hawaii at Manoa website. https://manoa.hawaii.edu/news/article.php?aId=7518 [iii] Rohrbach, K. G.., Bartholomew, D. P.., Paull, Robert E.. The Pineapple: Botany, Production, and Uses. United Kingdom: CAB International, 2002. [iv] Handbook of Pineapple Technology: Production, Postharvest Science, Processing and Nutrition. United Kingdom: Wiley, 2017. Changes/Problems:There were no significant changes to the proposed project plan. Some minor changes included testing different cameras and conducting an airplane flight to explore opportunities to identify weeds, nitrogen deficiencies, and water stress. Regarding problems we did find identifying fruit size challenging. We did conduct a flight to capture oblique views of the pineapple to see if this would allow us to identify size. However this brought up further complications as perspective is difficult to determine exactly even with the IMU data from the gimbal. This is an advanced task outside the scope of the Phase I that will require future research. What opportunities for training and professional development has the project provided?This project provided numerous opportunities for training and professional development. This included opportunities for Intelinair employees, University of Hawaii technicians and students, and Dole Hawaii staff. How have the results been disseminated to communities of interest?Intellinair has filed for a provisional patent on our algorithm, and we have published a paper that we presented to the AI for Social Good workshop at the 2020 International Joint Conference on Artificial Intelligence. What do you plan to do during the next reporting period to accomplish the goals? Nothing Reported

Impacts
What was accomplished under these goals? (confidential) Data Collection and Mosaicking Intelinair partnered with Drone Services Hawaii (DSH) to capture imagery over pineapple fields on the Dole plantation. While we had originally intended to capture imagery with Intelinair's RD-100 drone, after discussing with DSH, we decided to use their DJI Matrice 210 drone instead as they were more familiar with flying this system. We flew several different camera systems: DJI X3: This is a three band camera (Blue, Green, Red) with a 12MP Bayer sensor; we captured imagery at a ground sample distance (GSD) of 1.3 to 5.3 cm over various flights. Micasense Red Edge 3: This is a five band (Blue, Green, Red, Red Edge, NIR) camera with each band captured by a 1.3MP monosensor. We captured imagery at 4 to 8 cm GSD over various flights. MAPIR Kernel: This is a six band (Blue, Green, Yellow, Red, Red Edge, NIR) camera with each band captured by a 3.2MP monosensor. We captured imagery at 2.2 to 4.3 cm GSD. Autel EVO II 8k: This is a three band camera (Blue, Green, Red) with a 48MP Bayer sensor; we captured imagery at a GSD of 1 cm. Seehttps://bit.ly/3sRe2GN for table (not able to paste in graphics) Table 1 Drone flights conducted. These drone flights were used to collect imagery to identify flowers. The images also captured weeds of different types and severities, herbicide damage, nitrogen deficiency, and areas of poor growth. The oblique flight flown on April 30, 2020 was an experiment to identify the crown and fruit, in addition to the flower, to see if fruit size could be estimated. We mosaicked the X3, Micasense, and EVO II imagery in-house using a mapping pipeline that we developed using Pix4D mosaicking software. The MAPIR camera produces images in a proprietary RAW format; because of this, for efficiency, we contracted the mosaicking for this camera to Skyline Software. We also partnered with Resource Mapping Hawaii (RMH) to capture imagery over the entire 3,000 acre Dole plantation using an airplane. The airplane was outfitted with two Phase One P65+ medium format cameras, one for Blue, Green, Red and another for NIR (10 cm GSD). Additionally, the plane had an ICI 8640P LWIR thermal camera (1 m GSD). This flight was conducted on January 20, 2020. RMH delivered this imagery already mosaicked. See below for examples of images capture; notice the flowers represented by the small "dots." Seehttps://bit.ly/3sRe2GN for figure (not able to paste in graphics) Figure 3 X3 imagery at 2.1 cm GSD, left. Micasense imagery at 4 cm GSD, right. Seehttps://bit.ly/3sRe2GN for figure (not able to paste in graphics) Figure 4 MAPIR RGB at 2.2 cm GSD, left. MAPIR CIR at 2.2. cm GSD, right. Additionally, Intelinair: Scouted the fields that we imaged taking ground-level photos, identifying areas of variable flowering, weeds, nitrogen deficiencies, and other issues; and Obtained radiometry values at different stages of the crop cycle and from different weed types to identify signature wavelengths for future work. Seehttps://bit.ly/3sRe2GN for figure (not able to paste in graphics) Figure 5 Radiometry data for plants of varying maturities. Image Analysis Intelinair focused on the X3 RGB imagery for analysis. This camera consistently produced excellent images where flowers were detectable. Our specific task was to count the flowers so as to produce a map of flowering density across the field or block. We considered several methods, and settled on "density-estimation-based counting." Density-estimation-based counting is appropriate when the entities being counted are small and/or overlapping.[1] The density map is predicted and the count determined by integrating over a region. Requirements for annotation are a single point per instance - appropriate for pineapple flowers due to their small size. It is a fully convolutional method, amenable to any image size or shape. Another benefit is that run complexity does not scale with the number of entities and models tend to be lighter-weight and faster. We used Labelbox to perform annotations. Images were cut into 256 x 256 windows. We then used data augmentation techniques during model training. Annotations were "blurred" to generate a ground truth density map. The model used a U-Net model architecture; this has been used successfully for cell-density prediction in biomedical applications.[2] Seehttps://bit.ly/3sRe2GN for figure (not able to paste in graphics) Figure 6 Labelbox annotation tool, left. U-Net model architecture, right. After tuning, the model produced very good results - sometimes even better than the human annotator - with a root mean square error (RMSE) of 0.004 for both training and validation. There were no systematic residuals in the error profile, and the model had an inference speed of 1.6 seconds/image on the CPU and 0.04 seconds/image on the GPU. This performance suggests high potential for running the algorithm on an edge device, such as a drone or tablet. Seehttps://bit.ly/3sRe2GN for figure (not able to paste in graphics) Figure 7 RMSE for Training Dataset, left. RMSE for Validation Dataset, right. Below are examples of model performance for 256 x 256 cut areas: Seehttps://bit.ly/3sRe2GN for figure (not able to paste in graphics) Figure 8 Image vs. Ground-Truth vs. Predicted Seehttps://bit.ly/3sRe2GN for figure (not able to paste in graphics) Figure 9 Image vs. Ground-Truth vs. Predicted Below is an example of the model performance across an entire field: Seehttps://bit.ly/3sRe2GN for figure (not able to paste in graphics) Figure 10 Overhead image of Waialua 03A field (displayed in Intelinair web interface) Seehttps://bit.ly/3sRe2GN for figure (not able to paste in graphics) Figure 11 Model results for November 15, 2019 image showing different blocks with different flowering densities. Pineapple goes through various growth stages as it develops through the course of a season, and Dole tracks the various actions taken on each block, from transplanting to forcing to harvest. Seehttps://bit.ly/3sRe2GN for figure (not able to paste in graphics) Figure 13 Dole document tracking field data From these dates and field observations, it can be determined the pineapple's growth stage. We then plotted these growth stages to the algorithm results. Seehttps://bit.ly/3sRe2GN for figure (not able to paste in graphics) Figure 14 Growth stage by block for Waialua 03A vs. algorithm results Variability in flowering is an evident problem for growers. This can be seen in the inconsistent density for the blocks with a common forcing date (e.g. consider the blocks in Early Flower stage, 64 days post-forcing, where the blocks labeled "64 EF" have dissimilar flowering densities). Second, the algorithm performs well for the flowering stage but less so once the fruit begins to develop. This performance difference is expected given the datasets on which the algorithm was trained. [1] Wang, Chuan; Zhang, Hua; Yang, Liang; Liu, Si; and Cao, Xioachun. Deep People Counting in Extremely Dense Crowds. Association for Computing Machinery Multimedia 2015, Brisbane, Australia. https://yangliang.github.io/pdf/sp055u.pdf [2] Ronneberger, Olaf; Fischer, Philipp; and Brox, Thomas. U-Net: Convolutional Networks for Biomedical Image Segmentation. Medical Image Computing and Computer-Assisted Intervention 2015, Munich, Germany. https://arxiv.org/abs/1505.04597

Publications

• Type: Conference Papers and Presentations Status: Accepted Year Published: 2020 Citation: Hobbs, J., Paull, R., Markowicz, B., and Rose, G. (2020) Flowering Density Estimation from Aerial Imagery for Automated Pineapple Flower Counting. AI for Social Good workshop, International Joint Conferences on Artificial Intelligence Organization. Yokohama, Japan, 2020. https://bit.ly/2zz86eT