Source: UNIV OF PENNSYLVANIA submitted to
ROBOT SWARMS AND HUMAN SCOUTS FOR PERSISTENT MONITORING OF SPECIALTY CROPS
Sponsoring Institution
National Institute of Food and Agriculture
Project Status
NEW
Funding Source
Reporting Frequency
Annual
Accession No.
1007026
Grant No.
2015-67021-23857
Project No.
PENW-2015-08504
Proposal No.
2015-08504
Multistate No.
(N/A)
Program Code
A7301
Project Start Date
Sep 1, 2015
Project End Date
Aug 31, 2018
Grant Year
2015
Project Director
KUMAR, R.
Recipient Organization
UNIV OF PENNSYLVANIA
(N/A)
PHILADELPHIA,PA 19104
Performing Department
1306 - Mechanical Engineering
Non Technical Summary
Unmanned Aerial Vehicles provide outstanding opportunities to have a transformative impact on farming technology. This proposal outlines research designed to assist farmers in improving the productivity and profitability of their farms thus strengthening rural economies and increasing the economic competitiveness of U.S agriculture.The motivation for this project comes from the challenges faced by the $60B (US only) specialty crop industry (over $600B worldwide). While the three target crops in this proposal account for a small fraction of this industry ($6B in the US), they are representative crops. Solutions that are developed for these three crops can be extended and applied to the entire industry. Florida had an estimated production value of $76.4 million on 4,300 acres of blueberries and $1.27 billion on 476,300 acres of citrus in 2014; the tomato production value was $455.9 million on 34,000 acres in 2013. The three problems addressed in this proposal‚ - estimating yield, detecting crop stress and monitoring pests- are also representative of the problems faced by this industry.Estimating yield prior to harvest is crucial for growers to better manage their resources. The yield estimation of specialty crops is especially important because of the high economic value of these crops. Early yield prediction gives growers more opportunities to set a market price and plan post-harvest logistics. Providing accurate counts of immature (green) fruit on all trees across the orchard will indicate the quantity of fruitlets on individual trees and increase the efficiency in the cataloging and marketing of product. In addition, this information will guide advanced manual thinning of the initial fruit set, permitting the grower to optimize the size and quantity of fruit.Detecting crop stress will enable corrective action that includes irrigation and application of fertilizer. Abiotic stresses such as water stress or nutrient deficiency are common problems in crop production, and sometimes have symptoms similar to biotic stresses. One of the objectives of this study is to investigate methods of differentiating the abiotic stresses from the biotic ones for more accurate decision-making.Diseases and pest infestations are a common threat to crop production. In blueberry production, major diseases include mummy berry, phytophthora root rot, botrytis blight, and septoria leaf spot. They are also common in Florida and Pennsylvania. The onset of many of these diseases is rapid and the time from infection to symptom appearance and to death of the plant is short. In addition, early detection is essential because these diseases spread rapidly if treatment is not timely. Finally, some diseases and pests are not exclusive to blueberries, tomatoes, or citrus. Because they have many hosts and are very mobile, they can become a problem at any time during the season, so persistent scouting is necessaryIt is common practice for human scouts to inspect farms periodically to estimate yield, detect crop stress, and monitor pest density. Although the expert knowledge of a human scout is invaluable, it is impractical for humans to conduct thorough inspections and collect the quantitative data that is necessary for precision farming. Indeed a swarm of low-flying co-robots working synergistically with one or more human scouts can enable efficient information gathering to optimize the use of resources such as water, labor, and fertilizers.A swarm can collect information either opportunistically or deliberately over several fields in a small time interval. This ability to conduct operations in a small, prescribed time window is particularly useful for gathering consistent stress data. A swarm can alert the human scout and direct her to possible "trouble spots" in a field with low vigor or yield. Typically, a human scout uses a truck to conduct inspections. The swarm can leverage automated charging stations on the truck to recharge allowing for optimal coverage, while adapting to the paths chosen by the scout as well as the intelligence gathered by the scout. Thus a human scout assisted by a swarm of co-robots and a decision support system can monitor large areas and gather actionable intelligence at an unprecedented scale.UAVs are game changers! They will have a transformative impact on farming technology. Beyond the broader impacts of the research, we will work closely with national agricultural groups to disseminate the results of the research conducted. The technology described in this proposal also provides a unique opportunity to connect STEM education with agriculture. As described above we will do this by developing courseware at all levels ranging from middle school students to graduate students. While our work in Philadelphia will help us target a primarily low-income and underrepresented minority population, the work in Florida will help us reach out to growers, farmers, and crop consultants to implement these units. We will also reach out to the general public through demonstrations, public lectures and websites
Animal Health Component
20%
Research Effort Categories
Basic
70%
Applied
20%
Developmental
10%
Classification

Knowledge Area (KA)Subject of Investigation (SOI)Field of Science (FOS)Percent
2162499202030%
4027310202040%
2031120102010%
2031460102010%
2030999102010%
Goals / Objectives
In an effort to reduce the amount of food lost annually between production and consumption by rendering the agricultural systems that generate food, fiber, and feed smarter and more efficient, this project brings together research groups from the University of Pennsylvania and the University of Florida to address the problems of:(a) obtaining timely estimates of yield;(b) diagnosis of crop stress; and(c) detection of pests to enable efficient utilization of resources such as labor, water, fertilizers, and pesticides, as well as risk management and financial planning for specialty crops.Our focus is on three representative crops -blueberries, tomatoes, and citrus- each with a different approach to farming (trellis-based, close-to-ground rows, and trees respectively), but representative of other crops such as apples, peaches, and strawberries.We propose the synergistic use of a swarm of Unmanned Aerial Vehicles (UAVs) that operate with human scouts. Customized sensors carried by low-flying co-robots working synergistically with human scouts will enable production of high resolution, multidimensional maps, which can be used to monitor large farms at unprecedented spatio-temporal resolutions, greatly improving the efficiency and yield of farm operations. Although UAVs have been successfully used before for precision agriculture, we will, for the first time, investigate autonomous flight at low altitudes that will enable detailed, multidimensional maps from side views.We will develop the framework and algorithms to deploy multiple UAVs that can collaborate with and be controlled by a single human scout.We will develop a decision support system that will enable a human scout and the swarm of co-robots to operate in concert over extended periods while accommodating constraints on sensing, navigation speeds, and power consumption.In addition to exploring new frontiers in the use of UAV swarms and co-robots, the proposed research will contribute to agriculture science along three directions:(1) developing automatic stress monitoring for agriculture using inter-row sensing with a swarm of UAVs;(2) investigating the potential for differentiating water-stressed trees from disease-infected plants or trees by comparing and combining the discrimination results from both thermal sensing and hyperspectral sensing; and(3) investigating the feasibility of pest density monitoring and mapping using RGB images and physical samples obtained from the UAV swarm.Most importantly, we will develop a novel approach to precision agriculture that will provide growers with a data-driven deployment strategy that makes synergistic use of a networked robotic system working interactively with a human scout.
Project Methods
We propose a swarm of co-robots (UAVs) that assist a human scout (farmer) by providing them with the information and actionable intelligence required to manage the farm. Specifically, providing reliable yield estimates will allow the farmer to plan the logistics of harvesting and distribution. Monitoring crop stress will allow optimized irrigation and scheduling of nutrient delivery. Detecting pests will permit the farmer to target diseased plants for treatment before pests overrun the farm. The swarms are co-robots - they coordinate with the scout, extending their reach to the entire farm. They collaborate with the scout, providing information that is complementary and synergistic with the information that can be collected manually.The scout uses a truck to drive around the farm for inspection. The truck will be instrumented with GPS and equipped with charging stations for the UAVs. As shown in our previous work, each UAV will be capable of landing autonomously. It will use GPS to provide an initial estimate of the truck and onboard sensors to precisely locate its landing/charging station. Landing positions can differ from the launch positions, thereby allowing the scout to move independently of the UAVs. We will develop a Decision Support System that will advise the scout of routes likely to yield maximal information and warn them of areas out of reach of the robots due to battery limitations.A semantic map is a metric map composed of regions with such labels as "high crop-yield", "nutrient-stressed", "water-stressed"‚ "high pest-pressure", and "unknown". The map is designed to be easy to interpret. The proposed research will use principled methods from machine learning, and planning algorithms to generate an optimal deployment plan for the scout and the swarm to generate the map."Efforts": We will work closely with national agricultural groups to disseminate the results of the research conducted. The technology described in this proposal also provides a unique opportunity to connect STEM education with agriculture. We propose the creation of middle school science units that ask students to study food security issues, challenges faced by farmers, and how robots can help solve those problems. We will work with the Philadelphia School District, a primarily low-income and underrepresented minority population, to implement these units. We will pursue the development of UAV swarms and instructional material for use in undergraduate courses at the University of Pennsylvania. We will also engage the general public through an extensive scheduling of demonstrations that include lab tours given to 1,200 K-12 students per year, Philadelphia Science Festival activities that reach thousands of children and adults, and Science After Hours presentations at museums in Philadelphia and Citrus Show and various field days in Florida."Evaluation": The experiments will be carried out in three stages over three years.Through year one, we will develop six UAVs, test the automated recharging system, develop AgDSS, test multi-spectral sensor package using potted trees, and test UAV control and navigation algorithms, part of which will be carried out at the netted outdoor flight facility at the University of Pennsylvania.In Florida, during year one, we will investigate the spectral reflectance characteristics of major diseases affecting tomatoes and blueberries and create a library of spectral signatures. We will conduct a band selection on the spectral data to detect key bands for detection of different diseases. These bands will be used later to select filters for multi-band cameras onboard the UAVs.During year two, we will carry out field testing of our system at blueberry, tomato, and citrus farms in Florida, working in close collaboration with growers.Year three will be used to refine the system and explore opportunities for technology transfer with industry. The specific evaluation plan for the three major goals contributing to agriculture science is given next.Yield estimation - Appropriate sampling strategies for each fruit will be evaluated. High resolution images acquired will be used for automated fruit counting using machine vision algorithms. For citrus, UAVs will carry out close-range flights between trees. For blueberry vines, UAV flights will be carried out between trellises, at an altitude of about five feet. Since some blueberry fruits may be hidden due to canopy structure, we will evaluate sampling strategies to observe yield from close-range images when only a subset of fruits on the canopy is visible. Tomatoes grow in row structures at ground level, presenting a challenge, and acquisition of ground-truth data of yield may involve more human supervision. We will correlate the fruit count obtained from select locations with multispectral imagery and canopy characteristics. For each case, estimation accuracy will be evaluated by comparing predicted yield with ground-truth yield from a random set of locations in the farm.Measuring crop stress and disease - The evaluation will consist of two phases. For the first phase of experiments, we will develop a model that detects and differentiates stress and diseases using samples of stressed, diseased, and healthy trees as ground-truth. The second phase will use this model to test data-driven sampling strategies to generate maps of stress and disease hotspots.Analysis of variance (ANOVA) will be performed on the spectral data of water-stressed, diseased and healthy plants at the 0.01 level of significance by a Tukey's studentized range test. Several data mining techniques will be applied in order to select the best hyperspectral wavelengths to discriminate between hyperspectral data of the studied classes. The "one data out" approach for cross-validation will be used in order to assess the classification accuracy of each model (multilayer perceptron, and radial basis function). We will compare and evaluate the discrimination results on disease detection for the three categories: (i) with only the thermal data, (ii) with only the hyperspectral data, and (iii) with the combination of both data. The combination of both data will allow us to reduce misclassified plants obtained with thermal data or spectral measurements performed independently. Classification accuracy will be evaluated using ROC analysis, following which, we will determine the necessity of using both a thermal camera and a spectral camera, or a single system alone to properly classify water-stressed, diseased or healthy plants. Following the training and evaluation of the above models, we will carry out data-driven experiments to generate maps of stress and disease for the whole farm during phase two. Evaluation will be carried out by ground-truth data collection by human scouts.Pest density estimation - We will evaluate two approaches for pest density estimation: (i) special pest sticky-tapes at specific elevated posts in the farm that are photographed by UAVs; (ii) sticky-tapes directly on the UAVs and use data-driven sampling methodology to guide them to locations with high expected pest density. For the first approach, the relationship between the flight altitudes and counting and detection accuracy will be determined by carrying out multiple flights over the same site by varying the altitude from 10 ft up to 100 ft at increments of 10 ft. For the second approach, the UAV tours will be planned to obtain covariates of pest density, followed by landing at a location where they stay for a pre-defined dwell time. On automatic and manual retrieval of the UAVs, ex-situ pest-counting will be carried out in a lab setting to obtain ground-truth estimates of pest density. Image processing techniques used in the first approach will also be investigated for automatic pest density count onboard the UAVs. The two approaches will be evaluated by comparing the estimated pest density map with manually collected ground truth data.

Progress 09/01/15 to 08/31/16

Outputs
Target Audience: Growers Washington Tree Fruit Research Commission (apple) Driscoll's (blueberry, strawberry) Lipman Produce (tomato) Agriculture Researchers University of Florida, Quincy and Lake Alfred extensions Penn State University, Fruit Research and Extension Center (FREC), Biglerville, PA Industry AS&E Inc. (backscatter X-ray imaging for improved fruit counting) Glades Crop Care Nouka Technologies Changes/Problems:A key concern has been fast changing FAA regulations, and lack of clarity on autonomy for swarms (infeasible to have one pilot per robot for field deployments). This impacts the scope of the project and restricts flight planning. Our mitigation plan is the use of the outdoor 100'x50'x50' netted test facility at UPenn that allows emulating field conditions (with GPS) for realistic tests on a sample set of plants. Ground-truth position information is available from an outdoor motion capture system attached to the structure. What opportunities for training and professional development has the project provided? High-school and undergraduate researchers worked on AgDSS development, design and fabrication of phytobiopsy system and pest-trap system, as well as development of a smart sensor suite with active illumination. DaVonne Henry, an undergraduate researcher under the NSF REU program carried out extensive spectroscopy studies, demonstrating potential efficacy of hyper-spectral imaging for improved fruit counting and disease detection. Delaney Kaufman, an undergraduate rising sophomore, and Steven Chen, a masters student, got the opportunity to attend the 5th anniversary of the National Robotics Initiative held in the Capitol complex in Washington DC on June 9, 2016. At the event, they were able to interact with leading robotics researchers and congresspersons, and demonstrated their research on this project. Xin Wang, a masters student in Environmental Studies, and Environmental Advocacy and Education at UPenn did her Capstone video project and report titled "Potentials and limitations of unmanned aerial vehicle (UAV) applications in environmental research" in part based on interviews during a field trip to Lipman Produce, and Driscoll's in Florida in March 2016 along with the project team. Edidiong Okon, an undergraduate junior funded by the NSF Louis Stokes Alliances for Minority Participation (LSAMP) Program interned on this project during Summer 2016, investigating design-based fruit counting methods for comparison with data-driven deep learning approaches. How have the results been disseminated to communities of interest? Yield estimation in citrus with SUAVs (Citrus Industry article, published 04/01/2016) http://www.crec.ifas.ufl.edu/extension/trade_journals/2016/2016_April_suavs.pdf http://label.ag is open-source, full sources available at https://github.com/dcunhas/agdss/ Summary video of research (made March 2016) https://www.youtube.com/watch?v=X6W8VVQwCIg Press: NSF article on National Robotics Initiative (NRI) 5th-anniversary event, Living in the Robotic Age, that features photo and a few lines on this project. Article also redirected to by official White House OSTP Twitter page tweet. Conference papers in preparation for IROS and CASE 2017 submissions (Virginia Tech) The paper describing the offline solution to the scouts+UAV deployment problem is planned to be submitted to IEEE Transactions on Robotics in September 2016. The ROS package implementing our algorithm will be released on github. What do you plan to do during the next reporting period to accomplish the goals? Field trials of autonomous aerial phytobiopsy, and autonomous pest trap deployment and recovery. Current tests are being conducted in a space with a motion capture system. For the next cycle we will scale-up to field conditions with natural lighting, and visual-servoing. We are designing an experiment in collaboration with Prof. Ehsani at UFL (Co-PI) for Spring 2017 wherein we will compare aerial pest-trap based pest-pressure monitoring with current best-practices. AgDSS development - we will continue extending http://label.ag (AgDSS) for visualization of model outputs (trained on labeled data). We are also extending the user-interface for use on mobile devices (e.g., iPad, smart phones with multi-touch screens). Finally, we are working on an API that will allow fast search and use of vector labels and images from AgDSS for machine learning. (Virginia Tech) In year 2, there are two main research thrusts on the algorithmic side. (1) We will extend our offline UAV+scout deployment algorithm to the online case, where the points that must be sensed appear/disappear over time. We have already started working on the online model using Gaussian Processes as the underlying framework. (2) We will incorporate additional real-world constraints such as limited energy, unequal speeds, and sensing constraints. Our plan is to extend the GTSP algorithm using the Constrained Markov Decision Processes framework, which allows for multi-objective optimization. (University of Florida) In year 2, we plan to (i) to repeat the yield estimation using low cost camera under different light conditions and using controllable light sources; (ii) to design and built an over- the-row platform equipped with light source and camera sensors array for field trial in tomato fields; (iii) to study the feasibility of combining a low-cost x-ray system with camera based yield estimation sensor.

Impacts
What was accomplished under these goals? Crop data collection A joint field experiment in October 2015 with Dr. Jim Schupp at Penn State University's Fruit Research and Extension Center (FREC), Biglerville, and AS&E Inc., demonstrated for the first time the use of backscatter X-ray imaging for detection of apples occluded by canopy. The experiment was carried out at FREC's apple orchards. Two trips were carried out to Florida in collaboration with Prof. Ehsani to Driscoll's and Lipman Produce in December 2015, and March 2016 for collection of visual and backscatter X-ray data. The March 2016 backscatter X-ray scans of strawberries and tomatoes demonstrated possible application for fruit counting of fruits that are significantly occluded by canopy. Two data collection trips were carried out to Washington state with Washington tree fruit research commission in August and October 2015. Nighttime data with active illumination on both green and red apples are collected, as well as human counted ground truth. Our counting algorithm was applied to those data and achieved 95% accuracy compared to ground truth. University of Florida carried out a series of data collection experiments with a spectrometer and controlled illumination from a halogen lamp (broad spectrum), in order to study ripeness of tomatoes, for comparison of leaves and fruits. Based on these results, the median NDVI value for tomato leaves were determined to be 0.15. NDVI increased almost linearly to 0.17 for raw green, 0.19 for matured green, and 0.22 for red tomato samples. Thus, calculating NDVI simplifies the detection and classification process for the tomato samples. Agricultural Decision Support System (AgDSS) We prototyped and evaluated http://label.ag, an agricultural decision support system for labeling and annotation of fruits and visual symptoms of biotic stresses The AgDSS has been used to acquire labels for orange and apple data acquired using our sensor suite Compute resources for deep learning on agricultural big-data A 16-core CPU based compute node, with two latest graphics cards (Nvidia TITAN X Pascal) has been set up for large-scale training on datasets labeled on http://label.ag Vehicles: Aerial robot team being developed with heterogeneous capabilities. Autonomy under development, manual tests being carried out Aerial 'phytobiopsy' or leaf sample collection for ex-situ analysis (e.g., genomics), enabling precision phytopathology A smart pest-trap for monitoring of pest density. Designed for deployment and recovery using autonomous aerial robots Swarm and Scout Planning and Coordination (Virginia Tech subcontract) (Virginia Tech) In year 1, progress was made in formulating and solving the UAV+scout deployment problem. We have developed an offline algorithm that solves for deploying teams of UAVs and scouts to visit and sense around a given set of points while minimizing the total travel time. Our model incorporates scenarios where there are three types of points: (i) that can be sensed by UAV alone; (ii) that can be sensed by scouts alone; and (iii) points that can be sensed by both scouts and UAVs. It also allows for multiple scouts and possibly heterogeneous teams of UAVs. Our solution is based on a Generalized Traveling Salesperson (GTSP) Algorithm, and finds the optimal tour for each scout and UAV. A journal paper describing these results will be submitted in September. The code implementing our algorithm will be released along with the paper. Our current work is on extending this algorithm to the online case where the points appear/disappear due to spatio-temporal Gaussian Processes.

Publications

  • Type: Conference Papers and Presentations Status: Accepted Year Published: 2016 Citation: S. Sarkar, J. Das, R. Ehsani, V. Kumar, Towards autonomous phytopathology: Outcomes and challenges of citrus greening disease detection through close-range remote sensing, presented at the 2016 IEEE International Conference on Robotics and Automation (ICRA), Stockholm, Sweden, 2016, pp. 51435148.
  • Type: Other Status: Published Year Published: 2016 Citation: R. Ehsani and J. Das, Yield estimation in citrus with SUAVs, Citrus Extension Trade Journals, pp. 16-18, 2016.
  • Type: Other Status: Published Year Published: 2016 Citation: Reza Ehsani, Arash Toudeshki, Peng Wan, New sensor technology for yield estimation and disease detection, in Tomato Proceedings, September 2016, pp. 15-16.
  • Type: Conference Papers and Presentations Status: Submitted Year Published: 2017 Citation: D. Orol, J. Das, L. Vacek, I. Orr, M. Paret, C. J. Taylor, V. Kumar, "An Aerial Phytobiopsy System: Design, Evaluation, and Lessons Learned," in review, 2017 IEEE International Conference on Robotics and Automation (ICRA).
  • Type: Other Status: Published Year Published: 2016 Citation: Reza Ehsani, Dvoralai Wulfsohn, Jnaneshwar Das, Ines Zamora Lagos, "Yield Estimation: A Low-Hanging Fruit for Application of Small UAS," in ASABE Resource: Engineering & Technology for a Sustainable World, July 2016, pp. 16-18.
  • Type: Journal Articles Status: Submitted Year Published: 2017 Citation: "Counting Apples and Oranges with Deep Learning: A Data-Driven Approach", Steven Chen, Shreyas Skandan, Sandeep Dcunha, Jnaneshwar Das, Chao Qu, Camillo J. Taylor, Vijay Kumar, in review (revise and resubmit) 2017 Robotics and Automation Letters.