Progress 06/15/18 to 06/14/23
Outputs
Target Audience:
Nothing Reported
Changes/Problems:This project took place during the COVID-19 pandemic, and consequently, we encountered problems completing project objectives. The work plan during summer 2020 was changed. Data collection was still available in the USDA-ARS-AFRS location, with one person working alone, so we sent data from AFRS to Marquette University and communicated about best practices for operating the UASs. At Marquette University, data collection efforts during summer 2020 had to be postponed. Instead, we focused on further improving our detection and projection algorithms using the datasets that had already been collected. One of the Co-PIs (Medeiros) changed positions from Marquette University to the University of Florida. We were not able to transfer Medeiros' subaward as planned, which meant that another graduate student to continue the project was not available. Finally, we determined that another UAV with a heavier payload is needed for long-term work on the project. At AFRS, there were various delays with the company and delivery of a working unit was delayed by more than a year. What opportunities for training and professional development has the project provided?Two graduate students completed their master's degrees in computer engineering at Marquette University and their research concerned Obj. 1 and Obj. 2 of this project. The first graduate student had an international robotics conference paper accepted and was able to present his work at the IEEE International Conference on Robotics and Automation 2019. The second computer engineering graduate student worked on extending the capabilities of the software and hardware systems for insect detection developed by the first student, and joined a healthcare corporation as a senior software engineer. An undergraduate computer engineering student also worked on the project in the first year. Both graduate students gained familiarity with operating UASs as well as the regulatory requirements for operating them. They both became FAA 107A drone operator's license holders over their time on the project. The engineering technician working on the project re-certified his FAA 107A drone operator's license, and has continued to refine skills on image acquisition, camera selection and settings, and troubleshooting for night flights. We also received a waiver from the FAA that allows for flying at night, which was key to being able to work on project objectives as we required images acquired at night from the UAS. Three undergraduate interns participated in setting up trials associated with tracking marked insects in the environment with the entomology group at AFRS. One intern designed the protocol for creating the static insect markers to evaluate the UAV detection efficiency. All interns also collected data on the detection efficiency by reviewing the video from the UAV flights. How have the results been disseminated to communities of interest?1. Presentation at a scientific meeting: Rice, K.B., Hernandez M., Tooker, J.F., Medeiros, H., Tabb, A. and Leskey, T.C. 2018. Lights lasers and drones: New techniques for tracking insects in the field. Vancouver, Canada. Entomological Society of America. 2. Presentation at a scientific meeting: B. Stumph, M. Hernandez Virto, H. Medeiros, A. Tabb, S. Wolford, K. Rice, T. Leskey, ``Detecting Invasive Insects with Unmanned Aerial Vehicles," at 2019 IEEE International Conference on Robotics and Automation. 3. Poster presentation at a university colloquium by a graduate student -- 2018 Marquette University Forward Thinking Colloquium: B. Stumph. 2018 "Quantification of Dispersal Patterns of Invasive Insect Species with Unmanned Aerial Vehicles." 4. Public presentations at USDA-ARS-AFRS: we presented aspects of the project to visitors of the location, ranging from PreK students, middle school robotics team members, home school students, focus group / stakeholders' group, and the general public. 5. Presentation at scientific meeting. IEEE International Conference of Automation and Robotics 2019, "Detecting Invasive Insects with Unmanned Aerial Vehicles", B. Stumph. What do you plan to do during the next reporting period to accomplish the goals?
Nothing Reported
Impacts
What was accomplished under these goals?
[Objective 1] Our proposal suggested use of a UAS, UAS-mounted camera, and focusable laser to capture insects in images. We modified the original concept to use a UV LED (ultraviolet light emitting diode) lighting matrix as a light source. The UV lighting matrix provided more stable light than the focusable laser, and a duplicate system was created for part of the team in a different location. We evaluated camera-lens combinations and focus and aperture settings, combined with dye colors, to determine which combinations produced images with visibile and high contrast insects are in the images. Near the end of the project, we concluded that a UAS with a larger payload, as well as the ability to mount the lights on a gimbal, would provide images where a higher proportion of the image region was illuminated as compared to our existing systems. We evaluated fluorescent powders to identify optimized powder type and color to maximize survival and visualization of marked insects; it was found that the different fluorescent coatings we evaluated did not affect insect survival. [Objective 2] Early in the project, we developed a simple method to aggregate multiple insect detections projected onto the global coordinate frame into a common detection representing a single insect. These multiple detections help alleviate missed detections at some video frames and are used as input to our tracking algorithms. We incorporated a multiple hypothesis tracking algorithm into our insect detection framework to associate unique identifiers to each insect before its detections are projected to the global coordinate system. We improved the estimation of the heading angle of the unmanned aerial vehicle using an angular Kalman filter. We manually labeled four datasets collected at different locations containing nighttime UAS videos of dead insects coated with fluorescent powder. The videos contain a total of 33,626 frames and 269 insects, where insects were labeled with unique identifiers to allow the evaluation of the global tracking performance of our algorithms. Our method shows a global insect association performance improvement of approximately 52% with respect to a baseline approach that associates detections by minimizing the total distance among insect detections in a pair of video frames using the Hungarian assignment algorithm. On average, our method can successfully associate 70% of the insect detections in at least 80% of the video frames. [Objective 3] Performed experiments to determine an UAS' efficacy in detecting insects at different locations in the canopy by using dead insects. The UAS was flown in a pre-programmed flight and the subsequent video was manually examined. We were able to visualize about half the insects in the peach system, only about one-third in apples. Height within the canopy did not seem to impact detection, but depth was important. Insects in the front third of the canopy (closest to the UAV) were easily seen, while insects in the farthest third from the camera/light were the least detectable. Efficiency was close to 100% when no leaves were present - which is not biologically meaningful, but shows that the leaves are the main factor that prevents detection. The result of these experiments was to purchase a different UAS with more flexibility in where the camera is situated on the unit to view more areas of the canopy, as well as improve stability during flight.
Publications
|
Progress 06/15/21 to 06/14/22
Outputs
Target Audience:
Nothing Reported
Changes/Problems:Co-PI Medeiros accepted a new position as an Associate Professor with tenure at the Department of Agricultural and Biological Engineering at the University of Florida. The process to transfer the subaward to the University of Florida is currently underway. A new graduate student is being recruited to join the project in January of 2023. What opportunities for training and professional development has the project provided?The second computer engineering master's degree student working on the project successfully defended his thesis in the summer of 2021 and joined a healthcare corporation as a senior software engineer. Three undergraduate interns participated in setting up trials associated with tracking marked insects in the environment with the entomology group at AFRS. One intern designed the protocol for creating the static insect markers to evaluate the UAV detection efficiency. All interns also collected data on the detection efficiency by reviewing the video from the UAV flights. How have the results been disseminated to communities of interest?
Nothing Reported
What do you plan to do during the next reporting period to accomplish the goals?Two locations are working on hardware; AFRS with getting their new UAS up and running, and UF, the same. Getting the UASes improved for data acquisition is a goal for the winter. [Objective 1] Initially, we had planned to use harmonic radar to assess the insect response to the UAS. The experimental plan was changed in favor of using video and live, marked BMSB to assess insect response and that work is planned for the next reporting period. [Objective 2] A manuscript is currently being prepared on global insect association, or counting, from UAS video. [Objective 3] Our peach tree block has been fully evaluated for insect detection efficiency by the UAS (insect height and depth within the tree canopy). The apple block has been evaluated for lower and mid canopy, but not high canopy, so that evaluation is planned for the next reporting period.
Impacts
What was accomplished under these goals?
[Objective 2] We manually labeled four datasets collected at different locations containing nighttime UAS videos of dead insects coated with fluorescent powder. The videos contain a total of 33,626 frames and 269 insects, where insects were labeled with unique identifiers to allow the evaluation of the global tracking performance of our algorithms. Our method shows a global insect association performance improvement of approximately 52% with respect to a baseline approach that associates detections by minimizing the total distance among insect detections in a pair of video frames using the Hungarian assignment algorithm. On average, our method can successfully associate 70% of the insect detections in at least 80% of the video frames. A manuscript reporting these findings is being prepared. [Objective 3] Performed experiments to determine an UAS' efficacy in detecting insects at different locations in the canopy by using dead insects. The UAS was flown in a pre-programmed flight and the subsequent video was manually examined. We were able to visualize about half the insects in the peach system, only about one-third in apples. Height within the canopy did not seem to impact detection, but depth was important. Insects in the front third of the canopy (closest to the UAV) with easily seen, while insects in the farthest third from the camera/light were the least detectable. Efficiency was close to 100% when no leaves were present - no biologically meaningful, but shows that the leaves are the main factor that prevents detection. The result of these experiments was to purchase a different UAS with more flexibility in where the camera is situated on the drone to view more areas of the canopy, as well as improve stability during flight.
Publications
|
Progress 06/15/20 to 06/14/21
Outputs
Target Audience:
Nothing Reported
Changes/Problems:Due to the COVID-19 pandemic, the work plan during summer 2020 was changed. Data collection was still available in the USDA-ARS-AFRS location, with one person working alone, so we sent data from there to Marquette University and communicated about best practices for operating the sUAS's. At Marquette University, data collection efforts during the summer had to be postponed. Instead, we focused on further improving our detection and projection algorithms using the datasets that were collected during the previous performance period. What opportunities for training and professional development has the project provided?The second master's degree student working on the project has completed most of the requirements for graduation and is expected to successfully defend his thesis in the summer of 2021. How have the results been disseminated to communities of interest?
Nothing Reported
What do you plan to do during the next reporting period to accomplish the goals?We plan to conclude our work on Objective 1, specifically 1.4 on quantifying and minimizing the influence of the UAS on target-insect behavior using harmonic radar technology by performing field experiments with live BMSB in August 2021.
Impacts
What was accomplished under these goals?
We incorporated a multiple hypothesis tracking algorithm into our insect detection framework to associate unique identifiers to each insect before its detections are projected to the global coordinate system. We improved the estimation of the heading angle of the unmanned aerial vehicle using an angular Kalman filter. [Objective 2]
Publications
|
Progress 06/15/19 to 06/14/20
Outputs
Target Audience:
Nothing Reported
Changes/Problems:Due to the COVID-19 pandemic, the work plan during summer 2020 was changed. Data collection was still available in the Kearneysville, WV, with one person working alone, so we sent data from West Virginia to Wisconsin and communicated about best practices for operating the sUAS's. At Wisconsin, data collection efforts during the summer had to be postponed. Instead, we focused on further improving our detection and projection algorithms using the datasets that were collected during the previous performance period. What opportunities for training and professional development has the project provided?After the first master's degree student graduated, a new master's student joined the project. The student had the opportunity to learn how to operate and program an UAS. The student also obtained a part FAA 107A drone operator's license. The student also developed substantial programming skills and is becoming increasingly familiar with state-of-the-art machine learning and computer vision techniques. The engineering technician working on the project re-certified his FAA 107A drone operator's license, and has continued to refine skills on image acquisition, camera selection and settings, and troubleshooting for night flights. We also received a waiver from the FAA that allows for flying at night. How have the results been disseminated to communities of interest?Publication submitted and accepted to journal Environmental Entomology. What do you plan to do during the next reporting period to accomplish the goals?We plan to conclude our work on Objective 1 and continue our activities on Objective 2. Specifically, we will finalize the evaluation of our insect detection and mosaic creation algorithms and will start incorporating a tracking algorithm to localize the insects over time.
Impacts
What was accomplished under these goals?
1. Continued troubleshooting on camera combinations for the low-light, small field-of-view needed in our experiments. Found and camera-lens combination with interchangeable lenses for manual F-stop and focus settings to achieve the best image. Also evaluated different cameras and lenses with respect to different dye colors. We have developed an improved insect detection algorithm based on deep convolutional autoencoders that performs better in the presence of background clutter (e.g., leaves) or under varying illumination conditions. We have also developed algorithms to project the insect locations obtained in each video frame to a global coordinate system using information obtained from the UAS's motion sensors and the GPS unit. We developed trajectory planning and motion control tools based on the open source Robot Operating System to facilitate the future dissemination and reproducibility of our results. 2. We developed a simple method to aggregate multiple insect detections projections onto the global coordinate frame into a common detection representing a single insect. These multiple detections help alleviate missed detections at some video frames and will be used as input to our tracking algorithms.
Publications
- Type:
Journal Articles
Status:
Published
Year Published:
2020
Citation:
Danielle M Kirkpatrick, Kevin B Rice, Aya Ibrahim, Shelby J Fleischer, John F Tooker, Amy Tabb, Henry Medeiros, William R Morrison, III, Tracy C Leskey, The Influence of Marking Methods on Mobility, Survivorship, and Field Recovery of Halyomorpha halys (Hemiptera: Pentatomidae) Adults and Nymphs, Environmental Entomology, , nvaa095, https://doi.org/10.1093/ee/nvaa095
|
Progress 06/15/18 to 06/14/19
Outputs
Target Audience:We did outreach within the entomology and robotics scientific communities this year, with presentations at the Entomological Society of America 2018and a large robotics conference (IEEE ICRA 2019). Changes/Problems:
Nothing Reported
What opportunities for training and professional development has the project provided?One graduate student and one undergraduate student at Marquette University worked on the project. The graduate student finished his master's degree research, and learned how to operate UAS's as well as gaining his part FAA 107A drone operator's license. The graduate student was also able to attend the IEEE International Conference on Robotics and Automation to present his work. How have the results been disseminated to communities of interest?1. Presentation at a scientific meeting: Rice, K.B., Hernandez M., Tooker, J.F., Medeiros, H., Tabb, A. and Leskey, T.C. 2018. Lights lasers and drones: New techniques for tracking insects in the field. Vancouver, Canada. Entomological Society of America. 2. Presentation at a scientific meeting: B. Stumph, M. Hernandez Virto, H. Medeiros, A. Tabb, S. Wolford, K. Rice, T. Leskey, ``Detecting Invasive Insects with Unmanned Aerial Vehicles,"at 2019 IEEE International Conference on Robotics and Automation. 3. Poster presentation at a university colloquium by a graduate student--2018 Marquette University Forward Thinking Colloquium: B. Stumph. 2018"Quantification of Dispersal Patterns of Invasive Insect Species with Unmanned Aerial Vehicles." 4. Public presentations at USDA-ARS-AFRS: we presented aspects of the project to visitors of the location, ranging from PreK students, middle school robotics team members, home school students, and the general public. What do you plan to do during the next reporting period to accomplish the goals?We plan to continue work on Objective 1, specifically: 1.2, activity 2: Evaluation of fluorescent coatings' detection distance using digital cameras. Initiate tests with the UAS at different distances, using different fluorescent powders. 1.3: Develop algorithms for detecting and localizing target insects relative to the UAS in images from a range of heights above, and distances from, the plant canopy and in a range of environments such as forest and agricultural fields.: some of this work has been done, but additional tests will be done to determine the best and worst-case heights. 1.4: Quantify and minimize influence of UAS on target-insect behavior using harmonic radar technology.: experiments are planned for late August, early September in WV, when the insects are predicted to be active.
Impacts
What was accomplished under these goals?
For this project year, work focussed on Objective 1. 1.1 Create and evaluate mounting system for focusable laser(s) and a camera on a UAS.Activity 1 (Mount focusable laser (s) and camera on a UAS) and 2 (Evaluate laser-camera system on a UAS) were completed. Furthermore, we modified the lighting system by replacing the focusable laser with a LED system, which provided more stable light. These results are detailed in our publication, Stumph et al. 2019, presented at IEEE ICRA 2019, and avaliable on arXiv: https://arxiv.org/abs/1903.00815 . 1.2 Evaluate fluorescent powders to identify optimized powder type and color to maximize survival and visulization of marked insects. Activity 1, Quantify behavioral effects of the fluorescent coatings, was completed and a manuscript is close to submission by the entomology group. It was found that the different fluorescent coatings did not affect insect survival. We had an annual meeting on June 11, 2019.
Publications
- Type:
Conference Papers and Presentations
Status:
Awaiting Publication
Year Published:
2019
Citation:
B. Stumph, M. Hernandez Virto, H. Medeiros, A. Tabb, S. Wolford, K. Rice, T. Leskey, ``Detecting Invasive Insects with Unmanned Aerial Vehicles," in 2019 IEEE International Conference on Robotics and Automation (ICRA), 2019. [Presented, to appear.] Available at https://arxiv.org/pdf/1903.00815.pdf
|
|