FACT: DEEP LEARNING FOR IMAGE-BASED AGRICULTURE EVALUATION

• Sponsoring Institution: National Institute of Food and Agriculture
• Project Status: COMPLETE
• Funding Source: AFRI COMPETITIVE GRANT
• Reporting Frequency: Annual
• Accession No.: 1019881
• Grant No.: 2019-67022-29922
• Cumulative Award Amt.: $499,989.00
• Proposal No.: 2018-09253
• Project Start Date: Aug 1, 2019
• Project End Date: Jul 31, 2024
• Grant Year: 2019
• Program Code: [A1541]- Food and Agriculture Cyberinformatics and Tools

Recipient Organization
RUTGERS, THE STATE UNIVERSITY OF NEW JERSEY
3 RUTGERS PLZA
NEW BRUNSWICK,NJ 08901-8559

Performing Department
Electrical and Computer Engine

Non Technical Summary
Recent advances in machine learning, drone imaging and have created exciting new opportunities for improving agriculture. This project is a public and private partnership integrating multi-disciplinary research at Rutgers University and research at Siemens Corporation. We combine our expertise in computer vision and plant biology to create new, paradigm-shifting approaches for evaluating plant health using drone imagery data and state-of-the-art algorithms for analyzing drone imagery. Our work will focus on cranberry crops at Philip E. Marucci Blueberry and Cranberry Research and Extension Center at Rutgers University. The long-term goal is to enable and support real-time crop assessment to facilitate management and to optimize crop yields. We will develop novel algorithms and develop a practical and efficient drone data collection system to collect video imagery for use in crop health analysis and yield estimation.

Animal Health Component

(N/A)

Research Effort Categories

Basic

100%

Applied

(N/A)

Developmental

(N/A)

Classification

Knowledge Area (KA)	Subject of Investigation (SOI)	Field of Science (FOS)	Percent
204	1121	2020	100%

Knowledge Area
204 - Plant Product Quality and Utility (Preharvest);

Subject Of Investigation
1121 - Cranberry;

Field Of Science
2020 - Engineering;

Keywords

computational agriculture

computer vision

cranberry

deep learning

drones

machine learning

Goals / Objectives
Recent advances in machine learning, imaging and data science have created exciting new opportunities for applications in computational agriculture. In this work, we combinecomputer visionwith plant biology to create new, paradigm-shifting approaches for quantitatively evaluating plant health using imagery data. We will apply and develop cutting edge algorithms usingmachine learning methods ondatasets collected using multi-spectral drones. Ourwork will focus on cranberry crops at Philip E. Marucci Blueberry and Cranberry Research and Extension Center at Rutgers University.The long-term goal is to enable and supportreal time crop assessment to facilitate management and to optimize crop yields. Toward this goal, the current work has four specific objectives:1) recognitionof cranberry fruit and segmentation to delineate the exact fruit region among image pixels,2) characterization of temporal change in albedo in order to classify the current and predicted state of the fruit,3) measurement and classification of daily cloud coverage conditions to estimate the interplay of solar exposure with fruit health and4) the development and implementation of a drone data collection system to collect video imagery at multiple time points for use in crop health analysis and yield estimation.The video imagery and analysis tools will be used to perform automated segmentation tocount the fruit per unit area and estimate yield of the entire cultivated bed.Using a stratified random sampling approach, we will estimate the amount ofcrop exposed to solar radiation to determine future steps such as irrigation planning. We will make the resulting software toolset image database publicly available to researchers.

Project Methods
We will develop a drone capture system and conduct two measurement sessions on the fields of the PE Marucci Center for Blueberry and Cranberry Research and Extension. We will build machine learning algorithms and an accompanying software toolkit for cranberry segmentation, and classification of fruit albedo variation and classification of cloud coverage. Our initial algorithm implementation will be tested and evaluated on the images obtained in Year 1. We will assess our results both within our research groups and with our advisory group and plan for the second capture set in Year 2. The second capture set will provide a larger dataset for our machine learning algorithms that rely on large data and will provide an opportunity to modify collection parameters such as spatial image sampling rate, temporal sampling rate, and image resolution. We will further develop the algorithm set and create the second version of the toolset. We will disseminate results and solicit feedback from the advisory group.Recognition and Segmentation of CranberriesRecognition and segmentation in computer vision has been impacted greatly by the development of deep learning. Semantic segmentation assigns per-pixel predictions of object categories for the given image, combining recognition and segmentation into a single algorithm. State-of-the-art semantic segmentation approaches are typically based on the Fully Convolutional Network (FCN) framework. This approach has evolved into several variations and we leverage our recent work of context-encoding for semantic segmentation} that learns a class-dependent feature map so that local scene context can be implicitly included. This approach provides significant improvement over prior state of the art with small computational cost over FCN. We have developed a method of context-aware semantic segmentation using end-to-end deep learning with high performance in segmentation. Our expertise in building deep networks for segmentation in complex scenes will be utilized in this cranberry segmentation problem. Our prior results in cranberry segmentationhave an accuracy of approximately 80%. We expect to significantly improve that rate with big datasets from our measurements as part of the scope of this project. In order to assess accuracy, we will create a training set of manually segmented data. Software for efficient manual segmentation will be created for ease-of-use. As is standard practice in machine learning, the labelled set will be divided into a training set, validation set and test set. N-fold cross validation will be used to accurately assess the performance.We will develop a new approach to cranberry segmentation that is tuned to the specific dot-like shape of cranberries. We propose to use ground truth that generates realistic cranberries under all types of illumination using a generative adversarial network GAN that can generate large volumes of training data from synthetic data of varying parameters such as scale and albedo. The GAN generates realistic data by using large amounts of real image data that is unaligned with synthetic data. The generative component of the network creates realistic versions of the synthetic data; our innovation will be to use this approach but tune to the known circular shape so that the training times will be shorter, and we expect the performance to be more robust.Temporal Change of AlbedoAlbedo or reflectance variation over time provides a signature to assess the crop health. There are two main challenges in creating meaningful temporal signatures for assessment. The first challenge is to obtain albedo information from pixel values. Pixel measurements are related to reflected light intensity, but the measurements depend on the sensor characteristics and input illumination. To convert pixel values to reflectance measurements, we needphotometric/radiometric calibration. The calibration will include measurement of incident light, measurement of reflectance standards, and the conversion of pixel values to reflectance. With reflectance measurements, and an estimate of the bidirectional reflectance distribution function, the albedo parameter can be estimated for each measured wavelength.The second challenge is linkingtemporal measurements that are not from the same spatial location due to drone sampling. We will approach this by mapping distributions over time so that the change in albedo distribution over time can be mapped to the change in individual albedo measurement over time to obtain the temporal signature (albedo as a function of time). We will cluster temporal signatures and then use expert labelling to assign semantic meaning to the clustered signatures. In this manner, a dictionary of meaningful temporal signatures will be developed. The classification algorithm will determine the dictionary label for each fruit and the distribution of labels for the crop is the overall health metric. Expert-evaluation of temporal signatures via the domain expertise of co-PI Oudemansto assign a level of crop vulnerability to each group of canonical albedo variation.Measurement and Classification of Cloud CoverageThe most important factor that determines the short-term risk related to berry cooking is solar radiation. Cloud coverage variation of the sunlight results in unstable irradiation. Unfortunately, the cloud dynamics in a local area and within a short time horizon, say, 30 minutes, cannot be accurately predicted by any state-of-the-art computational model. This short-term prediction is essential for decision-making of short-term irrigation strategies. A fisheye camera based system has the potential to fulfill the need by taking the sky pictures continuously every few seconds. From the time series of the images and potentially with other weather information (e.g., the wind direction and speed), it is possible to obtain a good estimate of the cloud trajectory and thus predict when and how much the sunlight would be occluded.Drone Data Collection SystemWe will develop a drone imaging method based on random sampling large fields to obtainmanageable size, high resolution imagery from multispectral imaging drones. We will develop and employ methods of photometric calibration with reflectance standards. The collection period will be approximately 6 weeks in length (August to mid-September) and the imaging will be done once per week. The expected uncompressed image size is approximately 12 MB (megabyte) ($1280 \times 960$ 16bit, 5 spectral bands). With a spatial sampling rate of 10 images per acre over 128 acres, the weekly data collection will be approximately 15 GB (gigabyte). The resulting dataset for two seasons of measurements and for two drone-to-crop distances will be approximately 375 GB. Two drone heights will be used so that the cranberry counting and cranberry appearance evaluation can be done with the appropriate scale. The complete dataset will provide temporal sequence images to support the above specific aims.Stakeholder involvementThe advisory groupconsists of business leaders and growers from the largest growing regions in the USA. We will interact with the advisory group by soliciting feedback on the project using meetings or surveys in each of the project years. Year 1 assembly interactions will assist in the project planning. Year 2 meeting will discuss the project progress. The software generated from this project for cranberry segmentation, temporal signature classification, and cloud cover classification will be made publicly available for researchers. Year 3 survey will solicit feedback on the software tools from the advisory group. This feedback will be considered and be used for modifications in Year 3. Invitations to present results to growing regions and industry leaders at the end of the grant are anticipated. Letters of support have been provided for all members.

Progress 08/01/19 to 07/31/24

Outputs
Target Audience:The stakeholders in this project are the cranberry growers and business leaders who make decisions on agricultural practices and utilization of irrigation resources. Our advisory board listed in this report under Dissemination of Results is representative of this target audience. Representatives are from the following organizations Cape Cod Cranberry Growers Association, Ocean Spray Cranberries, Wisconsin State Growers Association, Cranberry Institute, American Cranberry Growers Association. Additionally, researchers in computational agriculture are part of the target audience. Our algorithm development in this project has impact for other crops and precision agriculture methods. Advisory Board: • Mike Haines, Managar at Pine Island Cranberry Co. [e-mail], • Rodney Serres, Director of Research Ocean Spray Cranberries, [e-mail], • Tom Lochner, Executive Director at Wisconsin State Cranberry Gowers Association [e-mail], • John Wilson; Director, Cranberry Institute, Carver, MA [e-mail], • Brian Wick, Executive Director at Cape Cod Cranberry Growers' Association, [e-mail], • Shawn Cutts President, American Cranberry Growers Association, fifth generation grower, Cutts Brothers Inc, [e-mail] • Stephen Lee, III, Board member, Cranberry Marketing Commission, President Lee Brothers Inc, [e-mail], • Michael Little, Lead Developer for commercial development Vineland-Downstown Airport, [e-mail] • Lindsay Wells-Hansen: Sr.Sr. AgriculturalScientist, Ocean Spray ([e-mail]) Stakeholdersin this project are also researchers interested in our dataset and software. We have on-going discussions with Dr. Giverson Mupambi from U Mass and Dr. Jeffrey Neyhart. Additional stakeholdersin this project are collaborating researchers with whom we have developed new proposals from ideas emerged from our workin this project. In particular, wehave written a new USDA proposal on AI-on-the-bog for plant breeding collaborating with PI Dana and co-PI Oudemans as well as Dr. Gina Sideli andDr. Jeffrey Neyhart. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided? The following undergraduate students have participated in research internships on this project: Ronan John, Chris Dib and Luke Mackara. The following graduate students have been trained in research roles for this project: Ronan John (after his graduation). Faith Johnson, Ryan Meegan, Aditiya Kesari. In addition to research students have learned project management, writing, presentation,and website development skills. How have the results been disseminated to communities of interest?Results have been disseminated to communities of interest via Publication: Faith Johnson, Ryan Meegan, Peter Oudemans, Kristin Dana "Agtech Framework for Cranberry-Ripening Analysis using Vision Foundation Models", to appear IEEE Winter Conference on Computer Vision WACV 2025 Poster Presentation:Quantifying Crop Quality Using Vision Foundation Models, Ryan Meegan, Ronan John, Peter Oudemans, Kristin Dana, presented at Rutgers Robotics Workshop September 27, 2024 Talks: Invited Plenary Speaker, AI-on-the-Bog, 2024 Annual Summer Meeting of the American Cranberry Growers Associationg, Marucci Cranberry Blueberry Research Center, Chatsworth, NJ August 22, 2024 Invited Plenary Speaker, Foundations of AI: Computer Vision, NJ AI Summit, Princeton University, Princeton NJ April 11, 2024 Invited Plenary Speaker, Socially Cognizant Robotics, Lecture Series: AI Through a Rutgers Lens: A three-part series in NYC exploring Artificial Intelligence, Microsoft UN office New York, NY February 2024 Website:https://computervisionrutgers.github.io/finding_berries/ Lecture: "Robotics and Humanity" Byrne Seminar, Lecture for Rutgers For Rutgers University Freshman, incorporated results from this grant into that lecture. What do you plan to do during the next reporting period to accomplish the goals? Nothing Reported

Impacts
What was accomplished under these goals? Agricultural domains are being transformed by recent advances in AI and computer vision that support quantitative visual evaluation. Using aerial and ground imaging over a time series, we develop a framework for characterizing the ripening process of cranberry crops, a crucial component for precision agriculture tasks such as comparing crop breeds (high-throughput phenotyping) and detecting disease. Using drone imaging, we capture images from 20 waypoints across multiple bogs, and using ground-based imaging (hand-held camera), we image same bog patch using fixed fiducial markers. Both imaging methods are repeated to gather a multi-week time series spanning the entire growing season. Aerial imaging provides multiple samples to compute a distribution of albedo values. Ground imaging enables tracking of individual berries for a detailed view of berry appearance changes. Using state-of-the-art AI methods, vision transformers (ViT), for feature detection after segmentation, we extract a high dimensional feature descriptor of berry appearance. Interpretability of appearance is critical for plant biologists and cranberry growers to support crop breeding decisions (e.g. comparison of berry varieties from breeding programs). For interpretability, we create a 2D manifold of cranberry appearance by using a UMAP dimensionality reduction on ViT features. This projection enables quantification of ripening paths and a useful metric of ripening rate. We demonstrate the comparison of four cranberry varieties based on our ripening assessments. This work is the first of its kind and has future impact for cranberries and for other crops including wine grapes, olives, blueberries, and maize. Aerial and ground datasets are made publicly available. In prior years we accomplished a framework for segmenting and counting cranberries, and predicting conditions for cranberry overheating..Computer vision and AI for smart agriculture have exciting potential in optimizing crop yield while reducing resource use for better environmental and commercial outcomes. The goal of this work is to develop state-of-the-art computer vision algorithms for image-based crop evaluation and weather-related risk assessment to support real-time decision-making for growers. We develop a cranberry bog monitoring system that maps cranberry density and also predicts short-term cranberry internal temperatures. We have two important algorithm contributions. First, we develop a method for cranberry instance segmentation that provides the number of sun-exposed cranberries (not covered by the crop canopy) that are at risk of overheating. The algorithm is based on a novel weakly supervised framework using inexpensive point-click annotations, avoiding time-consuming annotations of fully-supervised methods. The second algorithmic contribution is an in-field joint solar irradiation and berry temperature prediction in an end-to-end differentiable network. The combined system enables over-heating risk assessment to inform irrigation decisions. To support these algorithms, we employ drone-based crop imaging and ground-based sky imaging systems to obtain a large-scale dataset at multiple time points. Through extensive experimental evaluation, we demonstrate high accuracy in cranberry segmentation, irradiance prediction and internal berry temperature prediction. This work is a pioneering step in using computer vision and machine learning for rapid, short-term decision-making that can assist growers n irrigation decisions in response to complex time-sensitive risk factors.

Publications

• Type: Conference Papers and Presentations Status: Accepted Year Published: 2025 Citation: Faith Johnson, Ryan Meegan, Peter Oudemans, Kristin Dana "Agtech Framework for Cranberry-Ripening Analysis using Vision Foundation Models", to appear IEEE Winter Conference on Computer Vision WACV 2025

Progress 08/01/22 to 07/31/23

Outputs
Target Audience:The stakeholders in this project are the cranberry growers and business leaders who make decisions on agricultural practices and utilization of irrigation resources. Our advisory board listed in this report under Dissemination of Results is representative of this target audience. Representatives are from the following organizations Cape Cod Cranberry Growers Association, Ocean Spray Cranberries, Wisconsin State Growers Association, Cranberry Institute, American Cranberry Growers Association. Additionally, researchers in computational agriculture are part of the target audience. Our algorithm development in this project has impact for other crops and precision agriculture methods. Advisory Board: Serres, Rodney, Director of Research Ocean Spray Cranberries, [e-mail], Tom Lochner, Executive Director at Wisconsin State Cranberry Gowers Association [e-mail], Brian Wick, Executive Director at Cape Cod Cranberry Growers' Association, [e-mail], Shawn Cutts President, American Cranberry Growers Association, fifth generation grower, Cutts Brothers Inc, [e-mail],Stephen Lee, III, Boardmember, Cranberry Marketing Commission, President Lee Brothers Inc, [e-mail], Michael Little, Lead Developer for commercial development Downstown Airport, [e-mail] Changes/Problems:The project has been approved for a one year extension. What opportunities for training and professional development has the project provided?Peri Akiva is a PhD graduate student in the Electrical and Computer Engineering (ECE) Dept. of Rutgers has been supported in as a Graduate Research Assistant in this project He completed his Phd in January 2023. His research was instrumental in our accepted publications (Computers and Electronics in Agriculture) that has been accepted with minor revisions. (Fall 2022) Aayush Patel, an undergraduate intern at PE Marucci Cranberry and Blueberry Research Center, participated in this project throughout Summer and part of Fall 2022. Aayush is an undergraduate student at Atlantic Cape Community College (ACCC). (Summer 2022) Faith Johnson, a graduate student at Rutgers ECE dept. and Jack Lowry, an ECE undergraduate at Rutgers were instrumental in doing the albedo analysis of this project. (Spring 2023) Another ECE graduate student Zachary Taylor has started participated in data collection and algorithm development. (end of Summer 2022) How have the results been disseminated to communities of interest? Our project website (includes code, papers, and additional project information) has been updated at: https://computervisionrutgers.github.io/finding_berries/ Advisory Board Meeting and Presentation for Stakeholders: We had our third advisory board meeting forDecember 8th 2022. Attendees included Kristin Dana (PI Rutgers), Peter Oudemans (co-PI Rutgers), Aditi Roy (co-PI Siemens), Peri Akiva (Rutgers graduate student), Jack Lowry (Rutgers undergraduate student), Mike Haines ( Executive Director, Cape Cod Cranberry Growers' Association), Rodney Serres (Director of Research Ocean Spray Cranberries), Lindsay Wells-Hansen (Senior Agricultural Scientist, Ocean Spray Cranberries), Tom Lochner (Executive Director at Wisconsin State Cranberry Gowers Association), Brian Wick (Executive Director at Cape Cod Cranberry Growers' Association), Shawn Cutts (President, American Cranberry Growers Association, fifth generation grower, Cutts Brothers Inc.), Stephen Lee, III (Boardmember, Cranberry Marketing Commission, President Lee Brothers Inc.), Michael Little (Lead Developer for commercial development Downstown Airport). Our paper was published in thejournal "Computers and Electronics in Agriculture" (journal impact score: 6.757). PI Dana gave a keynote for the 4th International Workshop on Agriculture Vision help in conjunction with IEEE/CVPR 2023 with a talk entitled: AI on the Bog: Agriculture Evaluation with Time-Series Drone Imaging Co-PI Oudemans gave the following talks and disseminated the results of this project: 08/13/2022 "Strategies for improving cranberry fruit quality" American Cranberry Growers Association Summer Meeting New Jersey Cranberry Industry (~60) 12/14/2022, Managing Cranberry Fruit Quality, New Jersey Cranberry Research Council, Research Council Members (10) 12/14/2022 Approaches for Blueberry Disease Management, New Jersey Blueberry Research Council, Research Council Members (10) 01/19/2023Research Addressing Current and Future Solutions for Cranberry Fruit Rot Management, American Cranberry Growers Association Winter Meeting, New Jersey Cranberry Industry (~40) 02/23/2023, Research Addressing Current and Future Solutions for Cranberry Fruit Rot Management, 2023 Pacific Northwest Cranberry Conference, Online conference (>200) What do you plan to do during the next reporting period to accomplish the goals? New measurements combining with high temporal frequency possibly combining ground measurements and drone images Heat Event Before/After: A heat event is not predictable, so each season results in limited data. High temporal frequency imaging iwill help us obtain more before/after imaging of heating events. New algorithms for human-in-the-loop annotations so that fruit segmentations can be done without annotations or with minimal annotations. Networks for albedo sequence modeling with machine learning to predict ripening Testing of new algorithms and networks on most recent dataset Meeting with stakeholders. Our advisory board meets annually and members have given us important feedback. Our most recent meeting was December 2022. We will plan another Winter 2023-2024 Finalizing experiments, make visualizations and disseminating results to advisory board and other stakeholders.

Impacts
What was accomplished under these goals? Specific Objective One:recognitionof cranberry fruit and segmentation to delineate the exact fruit region among image pixels. We develop a cranberry bog monitoring system that maps cranberry density and also predicts short-term cranberry internal temperatures. We e developed a method for cranberry instance segmentation that provides the number of sun-exposed cranberries (not covered by the crop canopy) that are at risk of overheating. The algorithm is based on a novel weakly supervised framework using inexpensive point-click annotations, avoiding time-consuming annotations of fully-supervised methods. With our framework, the approximate volume of fruit at-risk in the canopy can be estimated by using a combination of drone imagery and machine learning. Risk assessment methods provide commercial growers a strategic approach to implement evaporative cooling methods, e.g., irrigation in areas of high risk (with a high density of exposed cranberries) at a time with no expected cloud cover. Optimal irrigation management is a challenge for cranberry growers because too little allows cranberries to overheat, whereas too much leads to fungal disease and causes both irreversible damage to the fruit and crop loss. The decision to irrigate is location specific: exposed cranberries on a hot day require cooling irrigation, while cranberries in dense vegetation do not. The lack of intelligent irrigation system has caused increasin losses of fruit in the upper part of cranberry canopies. Specific Objective 2: Characterization of temporal change in albedo in order to classify the current and predicted state of the fruit Using our drone imaging dataset CRAID 2022 we develop a framework for characterizing the ripening process of cranberry crops. Our method consists of drone-based time series collection over a cranberry growing season, photometric calibration for albedo recover from pixels, and berry segmentation with semi-supervised deep learning networks using point click annotations. By extracting time-series berry albedo measurements, we evaluate four different varieties of cranberries and provide a quantification of their ripening rates. Such quantification has practical implications for 1) assessing real-time overheating risks for cranberry bogs; 2) large scale comparisons of progeny in crop breeding; 3) detecting disease by looking for ripening pattern outliers. This work is the first of its kind in quantitative evaluation of ripening using computer vision methods and has impact beyond cranberry crops including wine grapes, olives, blueberries, and maize. Screening for the heritability of novel genotypes requires high through-put phenotyping (HTP) methods to discover desirable genetic traits.In crop breeding, there may be hundreds to thousands of progeny/offspring to evaluate, and high throughput methods make this evaluation practical. Computer vision algorithms for segmentation and calibrated albedo measurements enable quantitative comparisons. The methodology we develop in this research fits those requirements well, and HTP is an application domain for this work. The rate of color development is a crop trait that can affect the quality of cranberries at harvest. For consumer appeal, the timing and uniformity of ripening is critical, i.e. asynchronous ripening is a problem. For breeding, uniformity is desirable so HTP is used to look at multiple genotypes. Specific Objective 3: measurement and classification of daily cloud coverage conditions to estimate the interplay of solar exposure with fruit health We tested our systems for solar irradiance prediction based cloud analysis on deep learning approach and compared to other methods.Siemens has further refined their irradiance prediction results. A new sky and weather dataset was collected in 2021. A comparison of irradiance prediction results to state-of-the-art and baseline was done and included in the 2022 publication.Rutgers and Siemens used acustom sky imaging station consisting of a fish-eye lens camera mounted on a roof-height structure at PE Marucci Cranberry and Blueberry Research. We use Mobotix fisheye camera to capture 180-degree sky image and the pyranometer sensor to measure the solar irradiance in unit of watt/m2. The pipeline involves sun position estimation, cloud segmentation, velocity estimation, and solar irradiance prediction. The pipeline also includes predict berry internal temperature as well within a time horizon of 15-20 minutes to provides farmers a window of opportunity to mitigate over-heating events that may be irreversibly damaging to the fruit. Specific Aim 4: Data Collection For data collection, we introduce CRAID-2022, obtained from our bog monitoring system at PE Marucci Center for Blueberry and Cranberry Research, a substation of the Rutgers New Jersey Agricultural Experiment Station (Chatsworth, NJ). CRAID 2022 is an augmentation of the existing CRAID 2019 and CRAID 2021. Images were collected in 2022 using a Phantom 4 drone from a small range of altitudes with manually fixed camera settings.The imaging protocol includes imaging calibration devices (Macbeth color chart and silvered sphere) to assist photometric calibration. In the 2022 season, seven bogs were imaged, over the late July - September growing season. In total, there were four varieties (Mullica Queen, Stevens, Crimson Queen, Haines) imaged over six dates roughly a week apart with 20 waypoints per bog, once or twice a week with the following measurement dates: 8/2, 8/16, 8/25, 8/31, 9/9, and 9/14 in 2022. We are able to characterize berry appearance variation over the course of the season providing a rich dataset. This dataset enabled a ripening assessment to determine which variety ripens fastest and therefore may be more vulnerable to overheating.

Publications

• Type: Journal Articles Status: Published Year Published: 2022 Citation: Akiva, Peri, Benjamin Planche, Aditi Roy, Peter Oudemans, and Kristin Dana. "Vision on the bog: Cranberry crop risk evaluation with deep learning." Computers and Electronics in Agriculture 203 (2022): 107444.
• Type: Theses/Dissertations Status: Accepted Year Published: 2022 Citation: COMPLEX SCENE UNDERSTANDING WITH MINIMAL SUPERVISION, PhD Dissertation, Rutgers University, Peri Akiva, 2023
• Type: Websites Status: Published Year Published: 2019 Citation: Deep Learning for Image-based Agriculture Evaluation, https://computervisionrutgers.github.io/finding_berries/
• Type: Journal Articles Status: Other Year Published: 2023 Citation: In preparation, Faith Johnson, Jack Lowry, Kristin Dana, and Peter Oudemans, Vision-Based Cranberry Crop Ripening Assessment, to be submitted Computers and Electronics in Agriculture 2023

Progress 08/01/21 to 07/31/22

Outputs
Target Audience:The stakeholders in this project are the cranberry growers and business leaders who make decisions on agricultural practices and utilization of irrigation resources. Our advisory board listed in this report under Dissemination of Results is representative of this target audience. Representatives are from the following organizations Cape Cod Cranberry Growers Association, Ocean Spray Cranberries, Wisconsin State Growers Association, Cranberry Institute, American Cranberry Growers Association. Additionally, researchers in computational agriculture are part of the target audience. Our algorithm development in this project has impact for other crops and precision agriculture methods. Advisory Board: Mike Haines, Managar at Pine Island Cranberry Co. [e-mail], Rodney Serres, Director of Research Ocean Spray Cranberries, [e-mail], Tom Lochner, Executive Director at Wisconsin State Cranberry Gowers Association [e-mail], John Wilson; Director, Cranberry Institute, Carver, MA [e-mail], Brian Wick, Executive Director at Cape Cod Cranberry Growers' Association, [e-mail], Shawn Cutts President, American Cranberry Growers Association, fifth generation grower, Cutts Brothers Inc, [e-mail] Stephen Lee, III, Board member, Cranberry Marketing Commission, President Lee Brothers Inc, [e-mail], Michael Little, Lead Developer for commercial development Vineland-Downstown Airport, [e-mail] Additional stakeholders in this project are researchers interested in our dataset and software. We have on-going discussions with Dr. Giverson Mupambi from U Mass and Dr. Jeffrey Neyhart. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided?Peri Akiva is a PhD graduate student has been supported in as a Rutgers Graduate Research Assistant in this project He completed his Phd proposal this semester and intends to defend his thesis next semester. He worked on this project as part of his PhD thesis research. His research was instrumental in our accepted publications (Computers and Electronics in Agriculture) that has been accepted with minor revisions. Michael King, an agricultural workers at PE Marucci Cranberry and Blueberry Research Center, participated in this project throughout Summer and part of Fall 2021. Michael is a graduate of the Altantic Cape Community College ACCC Drone Pilot and Technician program in 2022. http://www.atlantic.edu/drones/ Hailey Benson, was a summer Intern from Ryder University. For 2022, Hailey has applied to Rutgers Graduate Program in Plant Biology. Hailey assisted in data collection for blueberries, helping to acquire firmtech measurements, photographs of appearance both RGB and hyperspectral. Aayush Patel is a summer Intern from Atlantic Cape Community College. Aayush is enrolled in the Air Traffic Control Program and has a completed the drone operator program offered at ACCC. He will graduate in the Fall 2022. Rutgers graduate student Zachary Taylor who recently completed his undergraduate degree at Rutgers (May 2022)has a summer internship under this program fordata collection and algorithm development. We have continued to develop our public-commercial partnership through our ongoing collaboration between Rutgers University and Siemens Corporation. This has given the student the opportunity to collaborated with researchers from industry. An undergraduate intern from Rutgers University, Jack Lowry, participated in Spring 2022. Jack implemented photometric calibration and support vector machine classification to test SVM-based counting. He also generated visualization data and histogram visualization data to observe cranberry bog appearance over time for all four measurement bogs. How have the results been disseminated to communities of interest? Our project website (includes code, papers, and additional project information) has been established at: https://computervisionrutgers.github.io/finding_berries/ We are planning out third advisory board meeting for Fall 2022. We are continuing our conversations with Dr. Giverson Mupambi from U. Mass. We have developed our cranberry counting software for him to use. We have tested that software and are pleased that the results appear visually correct when segmenting images that are similar from our training set, but not obtained with our sensor. The upcoming plan is for Dr. Mupambi to provide us with his images that we will run through our segmentation procedure. We are also conversing with Dr. Jeffrey Neyhart to use this software. Toward this end, we ported all the software to a new stand-alone computer, compiled software to run the segmentation work so that it can be run remotely. We plan to test this Fall 2022. Our expected publication in thejournal "Computers and Electronics in Agriculture" (impact score: 6.757). What do you plan to do during the next reporting period to accomplish the goals? Collection of the CRAID 2022 Dataset. The new dataset will have the following major differences from the year 2021 dataset RGB high-resolution (the pixel size in CRAID 2021 was significantly smaller than CRAID 2020, so the berry size in pixels was smaller). Collection of a column of data so that we can collect images at several drone heights. In 2021, wenoticed that drone height variation made challenging processing since the image scale is variable. For the next data collection we will aim for a dataset that is effectively collected from the same-height by post processing the image "column" with a scale-matching method. Evaluation of effect of heating events that may occur in 2022. We are interested in assessing the visual cues of overheating. These overheating events are rare and each season gives another opportunity for observing the appearance of fruit rot. Inclusion of sky and weather measurements at the same bog that is being imaged. 2022 Advisory Board Meeting intended for Fall/Winter 2022. Collaboration with other research groups to use our berry segmentation code and collect feedback. Algorithm testing for estimating fruit firmness from photographic appearance. Annotations needed for CRAID 2021 and CRAID 2022 so that ground truth counting numbers are available. Evaluation of new and existing counting algorithms on CRAID 2021 and 2022.

Impacts
What was accomplished under these goals? Specific Objective 1:recognitionof cranberry fruit and segmentation to delineate the exact fruit region among image pixels. Finished Collection and Processing of CRAID 2021 (see Table 1) Quantitative Evaluation of Segmentation Performance of our approach compared to other state-of-the-art segmentation for the cranberry segmentation problem (see Table 2) We Tested and refinement of our cranberry segmentation and counting methods on the new dataset. (Both tables will appear in the publication in Computers and Electronics in Agriculture). Transfer learning evaluation. We visually tested the performance of the segmentation algorithm for images other than our own imagery and saw that the performance was reasonable for other cranberry images. This result exceeded our expectation. Since the algorithm is tuned for the training data, imagery with a new sensor obtained from a different crop region is challenging. Specific Objective 2: characterization of temporal change in albedo in order to classify the current and predicted state of the fruit Last reporting season multispectral signatures of cranberries were explored from clicked points (since automated segmentation methods had been developed for CRAID 2019 (which was higher resolution that CRAID 2021) . This reporting period we used SVM (support vector machine) on the multispectral values for segmentation with classical machine learning SVM. We observed plots ofthe multispectral values over time for CRAID 2021 data over time. We are able to characterize the progression of the multispectral signature over the course of the growing season. We were hoping to be able to spot the effect of the heating events showing that the damage could be spotted in the multispectral signature of the berries, however this was not clearly observable. Specific Objective 3: measurement and classification of daily cloud coverage conditions to estimate the interplay of solar exposure with fruit health We tested our systems for solar irradiance prediction based cloud analysis on deep learning approach and compared to other methods. Siemens has further refined their irradiance prediction results. A new sky and weather dataset was collected in 2021. A comparison of irradiance prediction results to state-of-the-art and baseline was done and included in the 2022 publication. Rutgers and Siemens used a custom sky imaging station consisting of a fish-eye lens camera mounted on a roof-height structure at PE Marucci Cranberry and Blueberry Research. We use Mobotix fisheye camera to capture 180-degree sky image and the pyranometer sensor to measure the solar irradiance in unit of watt/m2. The pipeline involves sun position estimation, cloud segmentation, velocity estimation, and solar irradiance prediction. The pipeline also includes predict berry internal temperature as well within a time horizon of 15-20 minutes to provides farmers a window of opportunity to mitigate over-heating events that may be irreversibly damaging to the fruit. Specific Objective 4: Data Collection: In the 2021 season, we completed CRAID 2021 with multispectral sensor over a longer time period compared to 2020. The new imaging protocol includes imaging calibration devices (Macbeth color chart and silvered sphere) as well as taking video segments for potential 3D reconstruction of canopy structure at a later date. Settings will be adjusted to manual so that intensities over the course of the imaging session so that pixel values can be interpreted photometrically to reveal appearance (automated settings create unwanted variations). Four bogs were imaged, with 20 waypoints per bog, once or twice a week with the following measurement dates: 8/6, 8/11, 8/13, 8/16, 8/18, 8/24, 8/30, 9/3, 9/8/, 9/13, 9/20 and 9/24 (post-harvest). We are able to characterize berry appearance variation over the course of the season providing a comprehensivedataset.

Publications

• Type: Journal Articles Status: Other Year Published: 2022 Citation: â¿¿Vision on the Bog: Cranberry Crop Risk Evaluation with Deep Learning⿝, Peri Akiva, Benjamin Planche, Aditi Roy, Peter Oudemans, and Kristin Dana, accepted with minor revisions to Computers and Electronic in Agriculture

Progress 08/01/20 to 07/31/21

Outputs
Target Audience:The stakeholders in this project are the cranberry growers and business leaders who make decisions on agricultural practices and utilization of irrigation resources. Our advisory board listed in this report under Dissemination of Results is representative of this target audience. Representatives are from the following organizations Cape Cod Cranberry Growers Association, Ocean Spray Cranberries, Wisconsin State Growers Association, Cranberry Institute, American Cranberry Growers Association. Additionally, researchers in computational agriculture are part of the target audience. Our algorithm development in this project has impact for other crops and precision agriculture methods. This year our annual advisory board meeting for this project was Friday December 11th at 10AM. Attendees: Attendees: Executive [e-mail], Brian Wick, Executive Director at Cape Cod Cranberry Growers' Association, [e-mail], Shawn Cutts President, American Cranberry Growers Association, fifth generation grower, Cutts Brothers Inc, [e-mail] Stephen Lee, III, Boardmember, Cranberry Marketing Commission, President Lee Brothers Inc, [e-mail], Michael Little, Lead Developer for commercial development Downstown Airport, [e-mail] Changes/Problems:Covid shutdown has delayed our data collection last season and the overall imaging weeks were shortened. Michael Mars became a remote worker and could not assist in drone flying. The affect was small though and we did complete a large CRAID 2 multispectral dataset in 2020. The 2021 season imaging plan so far is not affected by Covid shutdowns. What opportunities for training and professional development has the project provided?Peri Akiva is a PhD graduate student supported full time on this project for Spring 2020 as a Rutgers Graduate Research Assistant. He developed a new algorithm for cranberry instance-based segmentation to support cranberry counting. He worked on this project as part of his PhD thesis research. His research was instrumental in our accepted publications (WACV 2021) and two submitted publications (under review) (ICCV 2021 and BMVC 2021). He presented this research (virtually due to Covid) at IEEE Winter Conference on Applications of Computer Vision (WACV) 2021. Michael Mars, agricultural workers at PE Marucci Cranberry and Blueberry Research Center, participated in this project throughout Fall 2021 and Spring 2020. Michael Mars provided annotations for the cranberry image data collection. Mr. Mars is a graduates of Stockton University of Galloway Township and obtained drone piloting licenses via training from Atlantic Cape Community College (ACCC). ACCC was awarded an NSF grant for ourUnmanned Aircraft Systems Operation and Maintenance Education and Training Project. ACCC offers drone training in preparation for Federal Aviation Administration Remote Pilot Certificate with a Small Unmanned Aircraft Systems Rating. Michaels Mars is a co-author on the WACV 2021 paper. Two graduate students participated in the data collection for a short academic visit in the 2020 season. Katie Vorsa is a graduate student at the University of Pennsylvania. Ethan Aizikovitz completed his masters from the Iowa State University and is now learning to be a farmer at Whelan Farms, Shamong NJ. In Summer 2021, Michael King has joined our program for 2021 season data collection. Michael is a graduate of the Altantic Cape Community College ACCC Drone Pilot and Technician program. http://www.atlantic.edu/drones/ We have continued to develop our public-commercial partnership through our ongoing collaboration between Rutgers University and Siemens Corporation. How have the results been disseminated to communities of interest?Our publication "Finding Berries: Segmentation and Counting of Cranberries using Point Supervision and Shape Priors" was presented by Peri Akiva (virtually due to Covid) at the First International Workshop on for Vision Agriculture held with IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2020. For this reporting period, we have a new publication "AI on the Bog: Monitoring and Evaluating Cranberry Crop Risk.WACV2021 (Winter Conference on Applications of Computer Vision).Peri Akiva, Benjamin Planche, Aditi Roy, Kristin Dana, Peter Oudemans, Michael Mars. This was presented at the WACV 2021 conference (virtually) by Peri Akiva. An additional related publication is Akiva, P. and Dana, K., 2021. Towards Single Stage Weakly Supervised Semantic Segmentation.arXiv preprint arXiv:2106.10309. Arxiv publications are unreviewed technical reports. Our project website (includes code, papers, and additional project information) has been established at: https://computervisionrutgers.github.io/finding_berries/ Our first advisory board meeting was held April 17, 2020 (virtual due to COVID). The attendees were very enthusiastic about our goals and progress and said they would be happy to join in future meetings. Our second advisory board meeting was held December 10, 2020 (virtual due to covid). The following stakeholders comprising our advisory group were in attendance: · Brian Wick, Executive Director, Cape Cod Cranberry Growers Association, [e-mail] · Serres, Rodney; [[e-mail]], Director of Research Ocean Spray Cranberries · Tom Lochner, [e-mail]; Executive Director, Wisconsin State Growers Association, · John Wilson; [[e-mail]], director Cranberry Institute · Shawn Cutts [[e-mail]], President, American Cranberry Growers Association · Stephen Lee, III; [[e-mail]], Boardmember, Cranberry Marketing Commission · Michael Little, [[e-mail]] , Lead Developer for commercial development Downstown Airport The advisory board meeting was a two hour event with presentations from Rutgers and Siemens and a discussion from the entire board. Topics that were discussed were the need for the growers to have information to inform them on when to irrigate. Also, the possibility of collaboration through internships was raised. The advisory board was informed of the project progress as well as the tools that have been developed. We have begun a collaborative exchange with Giverson Mupambi from (Extention Assistant Professor) at the UMass Cranberry Station. We intend to assist Prof. Mupambi by providing our imaging protocol so that he can repeat the data acquisition experiments at UMass and therefore directly use our software tools. In collaboration with Dr. Mupambi, we have submitted an application for internship funding via the 2021 Cranberry Institute Horticultural and Environmental Research Program ( https://www.cranberryinstitute.org) What do you plan to do during the next reporting period to accomplish the goals?1) Collection of the CRAID3 Dataset. The new dataset will have the following major differences from the year 1 dataset a. Both RGB high-res and Multispectral b. Additional calibration data including both color charts, silvered sphere and checkerboard patterns. This will assist in determining physical pixel size and drone height deviations c. Longer time frame to evaluate albedo variation. We hope to investigage the effect of sun scalding if applicable to the particular measurement site in 2021. d. Inclusion of sky and weather measurements at the same bog that is being imaged 2) Testing of our systems for solar irradiance prediction based cloud analysis on deep learning approach. 3) Testing and refinement of our cranberry segmentation and counting methods on the new dataset 4) Collaboration with out-of-state growers so they can replicate our data collection and analysis at their site.

Impacts
What was accomplished under these goals? In this work, we develop a three part computer-vision based approach to 1) automatically segment and count exposed cranberries from down-facing drone imagery to provide near real-time estimates of crops with potential high risk of sun exposure and over-heating; 2) analyze up-facing sky imagery to determine cloud cover and cloud motion for future irradiance prediction and 3) predict internal berry temperature to assess crop risk of overheating. Segmenting and counting the cranberries enables estimation of the monetary crop value at a given time. By segmenting and counting visible cranberries, the method essentially detects cranberries with direct exposure to sunlight. By predicting future irradiance with a time horizon of 15-20 minutes through sky monitoring, our method can provide farmers a window of opportunity to mitigate over-heating events that may be irreversibly damaging to the fruit. Proposed pipeline consists of two branches: the berry temperature prediction module, and the berry segmentation and counting module. The sequences of up-facing (sky) images taken near the cranberry bog are used as input to the berry temperature prediction branch to compute future solar irradiance over the field. This information together with other environmental data (eg humidity, wind speed) are used to forecast internal berry temperature. Internal berry temperature is crucial in determining berry spoilage, as they become permanently damaged beyond certain temperatures. Specific Objective One:recognitionof cranberry fruit and segmentation to delineate the exact fruit region among image pixels. We use cranberry fruit segmentation to delineate the exact fruit regions in the cranberry field image that are exposed to sun. We accomplish this with cranberry segmentation algorithms using modern deep learning networks. Specifically we have developed two segmentation networks, Triple-S Network and Pseudo-Masks from Points (PMP) , that use point annotations that are relatively easy to obtain by point-clicking o cranberry images. The main contribution of Triple-S network is the novel loss function that optimizes towards segmentation of circular shaped objects using point annotations. The method proposes two losses, shape and split losses. The shape loss optimizes predicted blobs in output prediction towards convex shapes, producing better fitting predictions for cranberry segmentation. The split loss utilizes the selective watershed algorithm to penalize the network for blobs predicted outside the feasible object region determined by the selective watershed algorithm. We replace the use of point annotations with pseudo-masks obtained by Pseudo-Masks from Points PMP method The PMP method as described in the project publications utilizes a pixel-adaptive convolutional refinement network. By combining the Triple-S and PMP methods, our most recent method aims to use the best of both algorithms, and results in high accuracy cranberry segmentation. Specific Objective 2: characterization of temporal change in albedo in order to classify the current and predicted state of the fruit. Using multispectral data collection we measured and observed time-varying multispectral signatures of cranberry appearance. The multispectral signatures of cranberries reveal the appearance change in cranberries over time as shown in the figure which shows average multispectral image intensity over cranberry pixels for each band (red, green, blue, NIR,RE) over 4 dates (8/20/20,8/27/20,9/4/20 and 9/8/20). Specific Objective 3: measurement and classification of daily cloud coverage conditions to estimate the interplay of solar exposure with fruit health. Siemens have developed cloud segmentation and motion estimation code based on their existing sky database. We use a deep learning method for joint estimation of motion and segmentation from cloud image sequences. The baseline network consists of two branches: a cloud motion estimation branch which is built on an unsupervised Siamese style recurrent spatial transformer network, and a cloud segmentation branch that is based on a fully convolutional network. The network takes as input consecutive sky images. Each pair of images (new and preceding) is passed to the motion estimation branch which evaluates the optical flow tying the two frames. In parallel, each distinct frame in the sequence is also fed to the semantic segmentation branch to estimate the cloud pixel map. We utilized both the Siemens Sky Dataset and a new sky and weather dataset collected onsite at the PE Marucci Cranberry Center at Rutgers. Rutgers and Siemens have constructed a sky imaging station consisting of a fish-eye lens camera mounted on a roof-height structure at PE Marucci Cranberry and Blueberry Research. We use Mobotix fisheye camera to capture 180-degree sky image and the pyranometer sensor to measure the solar irradiance in unit of watt/m2. The proposed pipeline involves sun position estimation, cloud segmentation, velocity estimation, and solar irradiance prediction. Though it was not originally included in the proposal, we found the need to predict berry internal temperature as well within a time horizon of 15-20 minutes to provides farmers a window of opportunity to mitigate over-heating events that may be irreversibly damaging to the fruit. In the first phase of the project, we developed separate modules for cloud estimation and solar irradiance prediction. First, we employed a deep learning framework for joint cloud segmentation and motion estimation. Given the cloud detection and motion maps, plus the clear sky solar radiation as basic information, in the second stage we estimate solar irradiance. In this phase of the project, we focused on building an end-to-end deep learning network for cloud segmentation, velocity estimation, and solar irradiance prediction. Next, we developed another machine learning module for berry temperature prediction from estimated irradiance and other climate data. We estimate future solar irradiation based on extracted features from the prediction zone and leveraging other non-image-based features, e.g. local weather data, time of day, etc. We employ long short-term memory (LSTM) , followed by multiple dense layers for feature sequence analysis. We explore a feature selection and regression framework to infer the berry internal temperature from predicted and measured data samples. The main drivers of berry internal temperature are environmental condition such as solar irradiance, humidity, wind speed and wetness. It is therefore required to record weather data corresponding to the berry internal temperature synchronously. We collected weather data with a one-minute sampling frequency that contains information about the ambient temperature (degree Fahrenheit), wind speed (mile/hr), gust speed (mile/hr), wind direction (0-360 degree), relative humidity, dew point (degree Fahrenheit), rain (in), wetness (%), solar irradiance (w/m2). In order to first rank these features w.r.t. their impact on the berry temperature, we train a random forest classifier and configure it to return feature importance scores. We observe that the top-2 features are solar irradiance and humidity. This shows the importance of predicting irradiance, since it varies frequently due to cloud movement, unlike the relatively constant humidity feature. Specific Objective 4: data collection. The 2020 season imaging was done with a DJI Phantom 4 Multispectral with Enterprise Shield Basic (with batteries and base station). This CRAID 2 dataset contains 1,906 cranberry multispectral images of size 512x512. Images were collected using a Phanthon 4 drone from a small range of altitudes. The dataset is available at the project website. We have planned the imaging protocol for the 2021 season and will image with both RGB and multispectral sensors over a longer time period.

Publications

• Type: Conference Papers and Presentations Status: Published Year Published: 2021 Citation: Akiva, P., Planche, B., Roy, A., Dana, K., Oudemans, P. and Mars, M., 2021. AI on the Bog: Monitoring and Evaluating Cranberry Crop Risk. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 2493-2502).
• Type: Websites Status: Published Year Published: 2020 Citation: https://computervisionrutgers.github.io/finding_berries/
• Type: Conference Papers and Presentations Status: Submitted Year Published: 2021 Citation: Vision on the Bog: Cranberry Overheating Risk Evaluation, Peri Akiva, Benjamin Planche, Aditi Roy, Kristin Dana, Peter Oudemans, submitted to British Machine Vision Conference BMVC 2021
• Type: Other Status: Other Year Published: 2021 Citation: Akiva, Peri, and Kristin Dana. "Towards Single Stage Weakly Supervised Semantic Segmentation." arXiv preprint arXiv:2106.10309 (2021).

Progress 08/01/19 to 07/31/20

Outputs
Target Audience:The stakeholders in this project are the cranberry growers and business leaders who make decisions on agricultural practices and utilization of irrigation resources. Our advisory board listed in this report under Dissemination of Results is representative of this target audience. Representatives are from the following organizations Cape Cod Cranberry Growers Association, Ocean Spray Cranberries, Wisconsin State Growers Association, Cranberry Institute, American Cranberry Growers Association, Commercial Developer Downstown Airport. Changes/Problems:Covid shutdown has delayed our data collection by a few weeks but at this time we expect the delay to be small we expect that the data collection not to be affected significantly. What opportunities for training and professional development has the project provided?Peri Akiva is a PhD graduate student supported full time on this project for Spring 2020 as a Rutgers Graduate Research Assistant. He developed a new algorithm for cranberry instance-based segmentation to support cranberry counting. He worked on this project as part of his PhD thesis research and was the lead author of the publication "Finding Berries: Segmentation and Counting of Cranberries using Point Supervision and Shape Priors". He presented this research (virtually due to Covid) at the First International Workshop of Vision Agriculture held with IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2020. Additionally, David Nuhn and Michael Mars, agricultural workers at PE Marucci Cranberry and Blueberry Research Center, participated in this project. Dave Nuhn and Michael Mars helped set up the drone imaging protocol for the cranberry image data collection. Michael Mars continued in labelling the data for training machine learning models for the CVPR workshop publication. Mr. Nuhn and Mr. Mars are graduates of Stockton University of Galloway Township and obtained drone piloting licenses via training from Atlantic Cape Community College (ACCC). (ACCC was awarded an NSF grant for ourUnmanned Aircraft Systems Operation and Maintenance Education and Training Project. ACCC offers drone training in preparation for Federal Aviation Administration Remote Pilot Certificate with a Small Unmanned Aircraft Systems Rating.) Dave Nuhn is now a GIS technician with National Aviation Facilities Experimental Center. How have the results been disseminated to communities of interest?Our publication "Finding Berries: Segmentation and Counting of Cranberries using Point Supervision and Shape Priors" was presented by Peri Akiva (virtually due to Covid) at the First International Workshop on for Vision Agriculture held with IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2020. Our project website has been established at: https://periakiva.github.io/finding_berries/index.html This website accessible at the PI's website: www.ece.rutgers.edu/~kdana/research/ The dataset and code are available on the website. Our first advisory board meeting was held April 17, 2020 (virtual due to COVID). The following stakeholders comprising our advisory group were in attendance. The attendees were very enthusiastic about our goals and progress and said they would be happy to join in future meetings. Brian Wick, Executive Director, Cape Cod Cranberry Growers Association, [e-mail] Serres, Rodney; [[e-mail]], Director of Research Ocean Spray Cranberries Tom Lochner, [e-mail]; Executive Director, Wisconsin State Growers Association, John Wilson; [[e-mail]], director Cranberry Institute Shawn Cutts [[e-mail]], President, American Cranberry Growers Association Stephen Lee, III; [[e-mail]], Boardmember, Cranberry Marketing Commission Michael Little, [[e-mail]] , Lead Developer for commercial development Downstown Airport What do you plan to do during the next reporting period to accomplish the goals?1) Collection of New Extended Dataset. The new dataset will have the following major differences from the year 1 dataset a. Multispectral Collection. We have purchased a DGI multispectral drone for the data collection b. Larger time frame to evaluate albedo variation 2) Develop temporal albedo models. We expect our 2020 dataset of cranberry images to include images of fruit rot and we will use this data to train machine learning algorithms to predict cranberry health based on albedo variation and rate of color change over time. 3) We will develop a short-term system for solar irradiance prediction based cloud analysis on deep learning approach. 4) Improve cranberry and cloud segmentation accuracy and improvement of our software toolset.

Impacts
What was accomplished under these goals? Specific Objective One: Our initial approach seeks to count and segment cranberries in RGB images. Recent segmentation methods require training algorithms using pixel-wise ground truth obtained manually, but such ground truth is expensive to obtain (requiring many hours for clicking to delineate the precise borders of every individual cranberry). We develop a novel method using only point-wise annotations which are an order magnitude cheaper than full pixelwise supervision. Our approach pairs the point-click annotations with additional shape and convexity cues to produce instance segmentation results. Our approach is built upon U-Net and consists of three branches: segmentation, split, and shape, constructing our proposed Triple-S Network. The segmentation branch aims to provide overall segmentation loss against point ground truth. The split and shape branches separate and refine individual blobs in segmentation outputs in accordance to boundaries and shape priors. The network consists of an encoder with eight blocks, each consists of two convolution layers followed by batch normalization and rectified linear unit (ReLU) layers. After each block we apply a 2 × 2 max pooling layer with a stride of 2. The encoder captures 3 channel inputs, and yields 1024 channel output. The decoder is also formed with eight blocks, each consists of feature map upsampling, two up convolution layers which halve the number of channels followed by batch normalization and ReLU layers. At the final layer, we use a 1×1 convolution layer to map 64 channel output to the number of classes. Our publication presents comparisons between baselines and our method for counting and segmentation metrics. As can be observed, our method outperforms state-of-the-art. Specific Objective 2: Our project began towards the end of growing season last year so the dataset does not exhibit the full range of albedo variation. This year we are collecting data that will encompass the full albedo change and characterization of the temporal change is an upcoming goal. Specific Objective 3: Siemens have developed cloud segmentation and motion estimation code based on their existing sky database. We use a deep learning method for joint estimation of motion and segmentation from cloud image sequences.The baseline network consists of two branches: a cloud motion estimation branch which is built on an unsupervised Siamese style recurrent spatial transformer network, and a cloud segmentation branch that is based on a fully convolutional network. The network takes as input consecutive sky images. Each pair of images (new and preceding) is passed to the motion estimation branch which evaluates the optical flow tying the two frames. In parallel, each distinct frame in the sequence is also fed to the semantic segmentation branch to estimate the cloud pixel map. The segmentation branch is trained via a supervised or weakly-supervised loss comparing the predicted semantic masks and the target ones (the training is considered weakly-supervised if the available target labels are noisy or coarse, as it is the case in this application). To further guide this branch and compensate for noisy labels, a self-supervised loss is added to ensure the consistency of consecutive semantic masks, comparing them after warping one using the predicted optical flow. Siemens Sky Dataset: Siemens had collected a dataset of hemispheric sky images (180o/360o) at an interval of 5 seconds spread over ~12 hrs. per day with Mobotix MX-Q24 fish eye camera from three locations namely Princeton-USA, Erlangen-Germany, and Isabela Island-Galapagos. Corresponding irradiation measurement was collected using Kipp & Zonen SMP11 pyranometer in unit of watt/m2. The camera is calibrated using the OcamCalib Toolbox with 11 checkerboard images. We tested our framework on datasets featuring two locations (Princeton-USA and Isabela-Galapagos) over a variety of time of the day and month of the year. The training set was composed of image sequences and their noisy labels (~10,590 pairs) captured in Isabela over the months of June and September; whereas the test set was made of sequences from Princeton in September and Isabela in May and August (therefore training and testing on distinct datasets). Observation of the semantic segmentation task on Isabela location data show how, despite being trained on noisy target labels, the pipeline achieves more consistent results thanks to the additional supervisions (through motion estimation and sequence consistency). Results Pyrometers were used to measure the irradiance near the camera positions for the SCCD dataset, so these timestamped ground-truth labels can be used to evaluate the end prediction of the proposed pipeline. Here we discuss performance of the proposed system for downstream irradiation prediction task in several scenarios. Scenario 1: Irradiation prediction results for clear sky without any cloud coverage of the sun. The proposed framework matches ground truth irradiation in our initial experiments. . Scenario 2: For irradiation prediction results while the sun occlusion occurs in distant future, we observe that our proposed framework closely follows the overall ground truth irradiation trend. However, minor variation in irradiation is not predicted by the correct method. Scenario 3: For irradiation prediction results for sudden pop-up cloud events in otherwise clear sky situation, our initital results show that although our framework correctly identified presence of such cloud, it overestimated its affect. It might be caused due to over-segmentation of the cloud pixels. Scenario 4: Irradiation prediction results while the sun is completely occluded by light cloud is a challenging scenario. The framework correctly detects presence of cloud covering the sun. However, since the cloud is not dense enough, even after occlusion the irradiance value is quite high. This scenario shows that although binary classification of cloud pixels is sufficient for irradiation prediction in most of the cases, it might not be enough for certain situation. Cloud type detection might be one way to handle this problem. Sky Imaging at Cranberry Bog Rutgers and Siemens have constructed a sky imaging station consisting of a fish-eye lens camera mounted on a roof-height structure at PE Marucci Cranberry and Blueberry Research. We use Mobotix fisheye camera to capture 180-degree sky image and the pyranometer sensor to measure the solar irradiance in unit of watt/m2. We have completed the sky imaging protocol and the sky imaging station is functioning to record continual imagery. Additionally we are using cranberry-probe temperature sensors, an artificial cranberry with an embedded temperature sensor for a wireless real-time temperature reading to approximate internal berry temperature. Specific Aim 4: Data Collection:We have collected 21,436 cranberry images of size 456 × 608 to create the largest repository of aerial RGB imagery of cranberry fields which we name CRAID. Images were collected in 2019 using a Phantom 4 drone from a small range of altitudes with manually fixed camera settings. Data was acquired at weekly intervals to capture albedo variations in cranberries, starting at early berry to post harvest. Drone trajectory is fixed throughout the collection season using initial randomly sampled path points at each cranberry bed. Before each recording session, a set of images of a checkerboard from different angels is captured for camera calibration purposes. Average point-wise annotation time for a single image is 4.32 minutes, while berry-wise annotations take 22.13 minutes. We have purchased a DJI Phantom 4 Multispectral with Enterprise Shield Basic (with batteries and base station). We have developed the imaging protocol for this multispectral drone and have begun cranberry bog imaging for the 2020 season.

Publications

• Type: Conference Papers and Presentations Status: Published Year Published: 2020 Citation: Akiva, P., Dana, K., Oudemans, P., & Mars, M. (2020). Finding Berries: Segmentation and Counting of Cranberries using Point Supervision and Shape Priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (pp. 50-51).