Progress 08/01/23 to 07/31/24
Outputs
Target Audience:The training and involvement of students targeted - undergraduate and graduate students in Computer Science - undergraduate and graduate students in Biology and Environmental Science The dissemination efforts during this reporting period targeted: - Dissemination to scientific community - Outreach to undergraduate students and general public Changes/Problems:Due to the difficulty in hiring a machine learning specialist in year 1, a part of the Machine Learning work was performed by graduate students, and another part was delayed. Two new hirings performed in November 2022 and March 2023 helped us bring the progress back to expected stage at end of year 1, with usable Re-Identification models, a real-time video analysis prototype and models and tools to scale the annotations in the next year. The leave of one of our hire at the end of July 2023 reduced our capacity, which was partially compensated by graduate and undergraduate students contributions. Much progress has been made, with the user interface of the system to be used for Objective 3 ready to be tested experimentally in summer 2024, and extensive data annotation and model training for Objective 4 was performed during the year. An extension of an additional year will enable the finalization of the application of the new models to Objective 4, and publish the experimental evaluation in Objective 3 and 4 in peer-reviewed journals. What opportunities for training and professional development has the project provided?The Computational team subproject (Activity 1 and 2) provided training and professional development opportunities to 7 Computer Science undergraduate students. The training included for the undergraduates: software development in python, web development, database development, machine learning. All students applied their training to biological and ecological data within the project. Two of these students were part of the local Research Experience for Undergraduates program supported by CAHSI (Computing Alliance of Hispanic-Serving Institutions). The Managed Pollinator's team (Activity 3) subproject provided training and professional development opportunities to 4 students. The training included Digital Design and use of 3D printer and Laser-cutter; electronics, arduino programing, management of honey bee colonies and general Honey bee biology and neuroscience concepts, application of Computer Vision software from Activity 2 to the biology application. Students were also trained in reporting, and the scientific publication process. The Wild Pollinator team subproject (Activity 4) provided training and professional development opportunities to 4 students. They received training in the use of digital annotation platforms using the Labelbee system and the use of Machine Learning models for image analysis. How have the results been disseminated to communities of interest?Outreach activities to high-school and undergraduates: Presentation at the research symposium for highschool students at the University of Puerto Rico, Cayey campus.On April 12, 2024, Co-PI Dr. Tugrul Giray presented our research as the plenary talk in the first research symposium targeting high school students in a rural, undergraduate campus of the University of Puerto Rico in the town of Cayey. The topic for the symposium was "Global warming and Agriculture" and Dr. Giray presented his talk titled "Abejas y su comportamiento" (Bees and their behavior). Presentation at Puerto Rico Brown Exploration workshop to a group of undergraduate students in Natural Sciences. On January 17, 2024, PI Dr. Remi Megret presented the research as part of a general presentation about the power of Artificial Intelligence to contribute to Experimental Sciences. What do you plan to do during the next reporting period to accomplish the goals?Activity 1: Integrate tracking for the mango pollinator videos to finalize the automatized event annotation in Activity 4 Activity 3: Publish results on experimental validation of the real-time system and its validation for behavioral assays performed during summer 2024. Activity 4: - Finalize large-scale annotation of mango dataset using developed ML tools. - Publish results related to the validation of the approach on pollinator visits analsyis.
Impacts
What was accomplished under these goals?
During this third year, work was performed on all 4 activities matching the 4 objectives stated in the goals. Activity 1: Machine Learning Models Group: Remi Megret, PhD (PI); Josue Rodriguez (Research Scientist); Luke Meyers (BS student); Gabriel Santiago (BS/MS Student); Luis Aviles (BS student); Jose Plaud (BS student) T1.1 Detection and pose estimation Based on extensive annotations of insects in mango videos collected in Activity 4, detection models were trained using YOLO models showing promising results. Tiling was used to be able to process the large resolution images without reducing pixel resolution during detection, using the SAHI (Slicing Aided Hyper Inference) library from (Akyon, 2022). T1.2 Individual Tracking and flower visits As annotation efforts in Activity 4 uncovered the high movement of flowers due to large prevalence of windy conditions, tracking of the flowers was explored using tracking algorithms from the OpenCV library. Different approaches were compared to manually annotated data and with respect to computational performance. T1.3 Payload, individual, and species identification. Based on more extensive Re-ID datasets collected and annotated during this reporting period, improved performance and more thorough evaluation was achieved compared to last year's preliminary results. The new 64 IDs dataset built in Activity 3 was used to perform an extensive comparison of convolutional network re-identification approaches. The first approach uses supervised learning to estimate the paint code directly; the second approach uses contrastive learning to learn an identity feature vector that is then used to query a database of known identities. Best performance reached 85% correct identification for all 64 identities, and up to 97.6% for 8 identities, showing the potential of the technique. Ablation studies with variation in training data and selection of IDs provide guidance for future use of this technique in the field. Occlusion maps showed that the models focus on the paint codes first and neighboring region second and tend to ignore the background, thus confirming that the models were able to learn the importance of the paint for identification without explicit guidance. This work was presented at VISAPP conference (Santiago 2024). Activity 2: Video collection and analysis platforms Group: Remi Megret, PhD (PI); Carlos Corrada, PhD (Co-PI); Josue Rodriguez (Research Scientist); Gabriel Santiago (BS/MS student); Eduardo Figueroa (BS student); Adriana Hernandez (BS student) T2.1 Cloud Platform for collaborative annotation and visualization. We extended and improved the Labelbee video annotation system further with usability and efficiency features including: a) Improved interface for navigating the video and annotation database b) Development of routing mechanisms to open a specific video and associated annotations at a specific frame for more efficient integration with external tools. c) Automatic visualization of annotation statistics in the interface itself for more efficient tracking of annotation efforts. T2.2 Real-time Edge Computing Platform. Based on the real-time processing prototype developed last year for continuous recording on the NVIDIA Xavier platform, a more performant prototype was developed by porting to a NVIDIA Jetson Orin platform. The system now includes automatic detection of the visits in real-time, which are stored in the MongoDB database. Such events can be visualized in a custom Web application that receives in real-time the detections for the experimenter to review and annotate the identity. The system was presented at conference ISICN 2024 [Rodriguez, 2024]. Activity 3: Managed Pollinators Monitoring Group: Tugrul Giray, PhD(Co-PI); José L. Agosto-Rivera, PhD (Co-PI); Noel Fanfan, MS (PhD student); Yilmaz Berk (PhD student); Luke Meyers (BS student); Edwin Lara (MS student); Edwin Florez (Collaborating Faculty at UPR-Mayaguez) T3.1 Collection of video data for honey bee forager visit to the artificial flower patch. In the current reporting period, a series of experiments confirmed the current based detection of flower visits does not alter honey bee behavior. This work is prepared as a publication to be submitted to the Journal of Insect Science in 2024. In addition, a prototype, and simulation model of the artificial flower patch of 36 flowers with current-based nectar and visit detectors were prepared. T3.2 Annotation of video data. An additional larger-scale dataset captured from marked bees was annotated with bees identity with paint-markings of single and dual colors. Out of the collected data, we built a mostly balanced dataset of 8062 images of honeybees marked with one or two paint dots from 8 different colors, generating 64 distinct codes, repeated twice on distinct individual bees. Last year's dataset was limited to 27 identities. The new dataset includes 64 distinct codes, and the repetition into 2 independent batches reaches 128 distinct identities, which ensures suitability to properly evaluate the performance on a large number of distinct colorIDs. The data was used to train improved Re-ID models in Activity 1. Activity 4: Wild Pollinators Monitoring Group: Elvia J. Meléndez Ackerman, PhD (Co-PI), Irma Cabrera Asencio, PhD (Senior Personnel), Skyler Williams (BS student), Kitana Anderson (BS student), Kenjiro Garcia (BS student), Alexander Galarza (BS student). T4.1 Annotation of wild pollinators video data. Extensive annotation of the videos took place in Fall 2023 and Spring 2024. A total of 1603 frames were annotated with 5381 annotations. Because such annotation is very time consuming, especially due to the really small scale of insects in the videos, it requires very careful visualization. For this reason, the methodology focused on sparse annotation of a large diversity of videos to gather a large variety of conditions and insects. Following this methodology, around 70 videos with medium to high activity (more than 5 insects) were annotated, covering 360 minutes of video, or 6h of total recording spread over multiple days and weeks. In addition, 53 videos with low activity were surveyed and annotated which contained less than 5 insects. This effort generated a 10x increase in available annotated data for the training of models in Activity 1.
Publications
- Type:
Journal Articles
Status:
Published
Year Published:
2024
Citation:
Giannoni-Guzm�n, M.A., Claudio, E.P., Aleman-Rios, J., Hernandez, G.D., Torres, M.P., Moreno, A.M., Loubriel, D., Moore, D., Giray, T. and Agosto-Rivera, J.L., 2024. The role of temperature on the development of circadian rhythms in honey bee workers. PeerJ, 12, p.e17086.
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2024
Citation:
Gabriel Santiago, Luke Meyers, Andrea Gomez Jaime, Rafael Melendez Rios, Fanfan Noel, Jose Agosto, Tugrul Giray, Josue Rodriguez Cordero and Remi Megret. "Identification of honeybees with paint codes using Convolutional Neural Networks", 19th International Conference on Computer Vision Theory and Applications (VISAPP), 2024. DOI: 10.5220/0012460600003660
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2024
Citation:
Josu� A. Rodr�guez-Cordero, Gabriel A. Santiago-Plaza, Luke Meyers, Fanfan Noel, Eduardo J Figueroa-Santiago, R�mi M�gret, Carlos Corrada Bravo, Jos� L. Agosto-Rivera and Tugrul Giray. "A Real-Time Edge System for Honeybee Flower Patch Assays", International Symposium on Intelligent Computing and Networking (ISICN 2024).
- Type:
Other
Status:
Published
Year Published:
2024
Citation:
Eduardo J Figueroa-Santiago et al.. Web application for Real-Time Monitoring and Analysis of Bee Behavior
from Video , poster presentation at Seminario Interuniversitario de Investigaci�n en Ciencias Matem�ticas, Humacao, February 2024.
- Type:
Other
Status:
Published
Year Published:
2024
Citation:
Y. Ortiz-Alvarado, M. Doke, T. Giray, N. Fanfan. DNA repair and long-term memory consolidation in honey bees. Poster presentation at Biology and Genomics of Social Insects Meeting, Cold Spring Harbor Laboratory, NY, March 25-28, 2024.
- Type:
Other
Status:
Published
Year Published:
2023
Citation:
Rodriguez-Alemany, C., Giray, T. The effects of dibutyl phthalate developmental exposure on worker honey bee behavior and circadian rhythms. Poster presentation at SfN, Neuroscience 2023, Washington, DC, November 2023.
- Type:
Other
Status:
Published
Year Published:
2023
Citation:
Ortiz-Ortiz, J.M., Rosa-Colon, F., Rodriguez, I., Agosto-Rivera, J. L., Giray, T. SMURF Bees: Development of an intestinal integrity assay for aging Apis mellifera. Poster presentation at Molecular Cellular Cognition Society Satellite Meeting at the SfN, Neuroscience 2023, Washington, DC, November 2023.
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2024
Citation:
R. Megret. "How Artificial Intelligence is reshaping animal studies". Oral presentation at Seminario Interuniversitario de Investigaci�n en Ciencias Matem�ticas conference, Humacao, February 2024.
|
Progress 08/01/22 to 07/31/23
Outputs
Target Audience:The dissemination efforts during this reporting period targeted: - Dissemination to scientific community - Involvement of computer science and biology students - Outreach to general public - Outreach to high-school and elementary school students in Puerto Rico Changes/Problems:Due to the difficulty in hiring a machine learning specialist in year 1, a part of the Machine Learning work was performed by graduate students, and another part was delayed. Two new hirings performed in November 2022 and March 2023 helped us bring the progress back to expected stage at end of year 1, with usable Re-Identification models, a real-time video analysis prototype and models and tools to scale the annotations in the next year. We will use the no-cost extension year to perform the work planned for Year 2. What opportunities for training and professional development has the project provided?The Computational team subproject (Activity 1 and 2) provided training and professional development opportunities to 4 Computer Science undergraduate students: 1) Joel Gonzalez, 2) Alejandro Soledad, 3) Jose Hernandez, 4) Gabriel Santiago. The training included for the undergraduates: software development in python, web development, database development. All students applied their training to biological and ecological data within the project. The Managed Pollinator's team (Activity 3) subproject provided training and professional development opportunities to 5 students: 1) Noel Fanfan, Ph.D student, 2) Edwin Lara, MS student; 3) Evelyn J. Aviles Rios (undergraduate), 4) Luke Meyers (undergraduate), 5) Valentina Aviles (undergraduate), 6) Andrea Gonzalez (undergraduate). The training included Digital Design and use of 3D printer and Laser-cutter; electronics, arduino programing, management of honey bee colonies and general Honey bee biology and neuroscience concepts, as well as application of Computer Vision software from Activity 2 to the biology application. Students were also trained in reporting, and the scientific publication process. The Wild Pollinator team subproject (Activity 4) provided training and professional development opportunities to 2 Biology students: 1) Solimar Marrero, Ph.D student, 2) Skyler Williams (undergraduate). They received training on the visual identification of insects of four different orders (Coleoptera, Diptera, Hymenoptera and Lepidoptera) and in the use of digital annotation platforms using the Labelbee system. How have the results been disseminated to communities of interest?Presentation at the Science Week in the University Gardens High School (UGHS): On April 17, 2023. Dr. Agosto-Rivera made a presentation and a Honey Bee Observation hive demonstration at a local high school specialized in science. The students prepared an youtube video documenting the activity which can be found in the following link: https://youtu.be/CMhIYqG58KU Presentation to public elementary school students: On March 30, 2023, Tugrul Giray made two presentations, to 6-8th grade and later to 9-12th grade students, in the STEAM FAIR at the Monserrate León de Irizarry High School, Cabo Rojo, PR. This activity included two video demonstrations, and a power point presentation where students learned about live and activities of honey bees in their colony and at flowers. The power point included the following topics: 1) Challenges pollinators face in the environment; 2) how bees handle flowers resulting in effective pollination, 3) Significance of behavioral differences for colony function. On March 30, 2023, Jose Luis Agosto made two presentations, to 6-8th grade and then to 9-12th grade students, in the STEAM FAIR at the Monserrate León de Irizarry High School, Cabo Rojo, PR. The main topic of the presentation was importance of daily activity patterns for honey bee and other bee pollinators. Press articles: Co-PI Dr. Tugrul Giray's participation in the Forward Research summit organized by the Puerto Rico Science Technology and Research trust was highlighted in an article in NIMB online forum.(see link "a"below). Dr. Giray also updated the status of honey bees in Puerto Rico, after the 2017 hurricane Maria to the National Public Radio;s "Radio Ambulante". This update and the earlier report was transmitted on NPR on the 25th of April 2023 (see link "b" below) https://newsismybusiness.com/forward-research-summit-pushes-to-accelerate-scientific-activity/ https://radioambulante.org/transcripcion/la-colonia-perdida-repeticion-transcripcion? What do you plan to do during the next reporting period to accomplish the goals?Activity 1: The various models will be adapted to their integration on to the system in Activity 2. The good results obtained in Re-ID will be used to define the new guidelines to collect more extended dataset. More extensive validation using larger datasets will be done to support the publication of the approach in peer-reviewed publication. Activity 2: Both cloud and edge systems have a working improved version. For the cloud system, the Labelbee annotation system will be improved further with better integration of the ML models. For the edge system, the working real-time detection will be extended with the Re-ID model and dashboards for deployment during the year. Activity 3: We will continue to collect additional videos of the flower patch, with collection of extended experimental data for validation of the approach planned for pollination season in Summer 2024. Activity 4: Trained models will be used for larger scale behavior analysis for validation of the approach, with the extensive annotation of the visit and species in the existing video dataset.
Impacts
What was accomplished under these goals?
During this second year, work was performed on all 4 activities matching the 4 objectives stated in the goals. Activity 1: Machine Learning Models Group: Remi Megret, PhD (PI); Rafael Melendez Rios (Data Scientist); Luke Meyers (BS student); Alejandro Soledad (BS student); Jose Hernandez (BS Student); T1.1 Detection and pose estimation. We trained new deep convolutional networks to detect individual honeybees in the videos. The new models leverage the YOLOv5 model, to be deployed for the real-time Edge Platform. T1.2 Individual tracking and flower visits. Existing approaches were used for tracking and visit detection in the cloud server. For Edge detection, a new approach using NVIDIA NvDCF tracker was tested, which provided improved speed and performance, and will be used for future experimentations. T1.3 Payload, individual, and species identification. Individual Re-ID was evaluated in new data from Activity 3 using paint codes. Preliminary results were presented at (Melendez-Rios, 2023), and improvements were presented in (Meyers, 2023). We showed that paint markings are a feasible approach to automatize the analysis of behavioral assays involving honey bees in the field where marking has to be as lightweight as possible. Using the new Re-ID dataset from Activity 3, we performed contrastive learning with a ResNet backbone and triplet loss led to identity representation features. Almost perfect recognition is achieved in closed setting where the 27 identities are known in advance (99.3% Top-1 accuracy with a gallery of 10 images with 9 distractors using the complete aligned image. The capability to generalize to separate IDs was also evaluated, reaching 87% Top-1 accuracy with a gallery of 10, and 98% accuracy when using nearest neighbor reidentification with the complete training set used as reference keys. The impact of using different body parts for identification also showed new insights, with re-ID from thorax only obtaining best Top-1 accuracy in open-set setting, compared to 87% for the complete image. Activity 2: Video collection and analysis platforms Group: Remi Megret, PhD (PI); Carlos Corrada, PhD (Co-PI); Josue Rodriguez (Research Scientist); Gabriel Santiago (BS Student). T2.2 Real-time Edge Computing Platform. Based on the video collection system implemented last year for continuous recording on the NVIDIA Xavier platform, a prototype for real-time processing was developed. The system uses NVIDIA DeepStream technology to implement a complete detection and tracking pipeline that includes: source video (direct camera capture or prerecorded video file), insect detection, tracking, reporting to a database. The detection was performed using Yolov5 object detector, specifically trained on the flowerpatch videos. Tracking uses NVIDIA NvDCF tracker, which allows increase in framerate by requiring detection only one in two frames, the rest of the trajectory being extended by pure visual tracking. The trajectories are then exported frame by frame in JSON format through a Kakfa protocol to be distributed to event subscribers, to allow real-time monitoring of the detections, or their storage into a database for later analysis. The database used is MongoDB, as NoSQL database that is very performant for appending such sequences of events and querying them later through different criterions. At the moment, the system can run at 59 fps on 1184x1184 pixels image data, which is significantly higher than the target 20 fps. This allows for a significant margin to add the ReID models developed in Activity 1 and automatic visit detection as a next step, as well as its visualization through real-time dashboard. Activity 3: Managed Pollinators Monitoring Group: Tugrul Giray, PhD(Co-PI); José L. Agosto-Rivera, PhD (Co-PI); Noel Fanfan, MS (PhD student); Juan R. Leon (Institutional Staff); Evelyn J. Aviles Rios (BS student); Luke Meyers (BS student); Valentina Avile (BS student); Edwin Lara (MS student); Edwin Florez (Collaborating Faculty at UPR-Mayaguez) T3.1 Collection of video data for honey bee forager visit to the artificial flower patch. In our previous report, we had successfully developed automated nectar delivery system: A) Syringe Pump; B) sucrose "nectar" sensor and artificial flower and; C) electronic control system. We now improved the nectar sensor to detect partial consumption of nectar or evaporation in artificial flower based on addition of an oscillator in line with the nectar droplet. This also required printing of artificial flower based on circuit boards. The artificial flower design, including sensor, nectar delivery system, control circuit and software are described in a publication in preparation (Edwin Lara et al. in preparation). Since the sensor was modified, we performed additional experiments to see if the small electric current passing through the nectar that will be consumed by the visiting bees alter behavior. The experiment has 2 flowers: one "with current" using the electric sensor described above, and another flower "without current" that would be refilled by pressing a button after a bee visit. We determined that either flower is visited at a rate of 20 visits per bee per foraging bout. We will further test color association of flower visits on these artificial flowers with full sensor and automatization assembly. T3.2 Annotation of video data. We developed a standard protocol to examine foraging strategy of Puerto Rico honey bees on a manual flower patch. We also developed specific neuropharmacological treatments to modify honey bee strategy in specific directions. Dopamine and its antagonist, flupentixol resulted in greater or lesser color constancy in flower visits (Noel et al. in preparation). Control and treatment bees will be recorded in the field on the automated flowers in the next field season. A new dataset captured from marked bees was annotated with bees identity with paint-markings of single and dual colors. The dataset contains 4392 images split into 27 identities, all extracted from videos collected outside, at Gurabo agricultural experimental station of the UPR, in light conditions similar to the flower patch assay. The data was used to train Re-ID models in Activity 1 Activity 4: Wild Pollinators Monitoring Group: Elvia J. Meléndez Ackerman, PhD (Co-PI), Irma Cabrera Asencio, PhD (Senior Personnel), Skyler Williams (Undergraduate student) T4.1 Annotation of wild pollinators video data. Training was provided to new students and faculty in terms of detecting and classifying the typical mango pollinators from the videos, using the guide developed last year. A field trip to the Juana Diaz agricultural experimental station was organized in June 2023 to see the insect collection, the data collection field and discuss the specific setup for next collection. Hundred of new annotations using Labelbee system were done to train a detection model, to be finalized during summer 2023.
Publications
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2022
Citation:
E. Courtney, Y. B. Koru, E. Aviles-Rios, Y. Ortiz-Alvarado, Doke, Mehmet A., A. A. Ruggieri, E. J. Aviles, N. Rodriguez, R. Giordano, R. K. Donthu, J. R. Leon, T. Giray, J. L.Agosto-Rivera. Gut microbiota affects the ontogeny of circadian rhythms in Apis mellifera. ABRCMS, Annaheim, CA, November 2022. Poster presentation.
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2022
Citation:
E. Aviles Rios, E. Courtney, Y. B. Koru, Y. Ortiz-Alvarado, M. Doke, A. Montes, A. A. Ruggieri, N. Rodriguez, R. Giordano, D. Ravi, J.Leon, A. Ghezzi, T. Giray, J. L. Agosto-Rivera. The development of circadian behavior is associated with changes in the expression of IGFBP-ASL in the brain of honey bees Apis mellifera. SfN, Neuroscience 2022, San Diego, CA, November 2022. Poster presentation.
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2022
Citation:
Y. B. Koru, E.C. Courtney, E. Aviles-Rios, Y. Ortiz-Alvarado, Doke, Mehmet A., A. A. Ruggieri, E. J. Aviles, N. Rodriguez, R. Giordano, R. K. Donthu, J. R. Leon, T. Giray, J. L.Agosto-Rivera. Gut microbes affect the onset of circadian rhythms and regulate gene expression in honey bee (Apis mellifera). SfN, Neuroscience 2022, San Diego, CA, November 2022. Poster presentation.
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2022
Citation:
Y. Ortiz-Alvarado, M. Doke, T. Giray, N. Fanfan. Role of DNA repair in long-term memory consolidation and decision-making on honey bees. SfN, Neuroscience 2022, San Diego, CA, November 2022. Poster presentation.
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2023
Citation:
Alejandro Soledad-Mendez, Jose Hernandez-Campos, Rémi Mégret, Robert A. Espaillat Perez, Jose L. Agosto-Rivera. Video Monitoring of Behavior Assays of Honeybees. XXXVIII Interuniversity Seminar on Research in the Mathematical Sciences (SIDIM) on February 24 and 25, 2023. Poster presentation.
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2022
Citation:
Rafael Meléndez-Rios, Luke Meyers, Jeffrey Chan, Rémi Mégret, Fanfan Noel, Jose L. Agosto-Rivera, Tugrul Giray. Towards automatic bee re-identification with paint markings. Oral presentation at XXXVIII Interuniversity Seminar on Research in the Mathematical Sciences (SIDIM) on February 24 and 25, 2023. Oral presentation.
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2022
Citation:
Luke Meyers, Rémi Megret. Automating Data Collection of Pollinator Studies using Deep Learning and Computer Tracking. The 2022 SACNAS National Diversity in STEM Conference. Oct 27-29, 2022. Poster presentation.
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2022
Citation:
Giray, T. ⿿BeeID: A molecular tool that identifies honey bee subspecies and populations from different geographic locations. Forward Research Symposium, San Juan, PR November, 2022. Invited talk.
- Type:
Journal Articles
Status:
Published
Year Published:
2023
Citation:
H Siviter, A Fisher, B Baer, MJF Brown, IF Camargo, J Cole, Y Le Conte, B Dorin, JD Evans, W Farina, J Fine, LR Fischer, MPD Garratt, TC Giannini, T Giray, H Li-Byarlay, MM López-Uribe, JC Nieh, K Przybyla, NE Raine, AM Ray, G Singh, M Spivak, K Traynor, KM Kapheim, JF Harrison 2023. Protecting pollinators and our food supply: understanding and managing threats to pollinator health. Insectes Sociaux (2023) 70:5⿿16 doi.org/10.1007/s00040-022-00897-x
- Type:
Journal Articles
Status:
Published
Year Published:
2022
Citation:
J. Marcelino, C. Braese, K. Christmon, J. D. Evans, T. Gilligan, T. Giray, A. Nearman, E. L. Niño, R. Rose, W. S. Sheppard, D. vanEngelsdorp, J. D. Ellis. 2022. The movement of western honey bees (Apis mellifera L.) among U.S. states and territories: history, benefits, risks, and mitigation strategies. Frontiers in Ecology and Evolution doi: 10.3389/fevo.2022.850600
- Type:
Journal Articles
Status:
Published
Year Published:
2022
Citation:
Giordano, R.; Galindo-Cardona, A.; Melendez-Ackerman, E.; Chen, S.-C.; Giray, T. 2022. Adaptation of invasive species to islands, focus on the honey bee of Puerto Rico. Frontiers in Ecology and Evolution doi: 10.3389/fevo.2022.946737
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2023
Citation:
Luke Meyers, Rafael Meléndez-Rios, Josue Rodriguez Cordero, Carlos Corrada Bravo, Fanfan Noel, Jose Agosto-Rivera, Tugrul Giray, Remi Megret. Towards Automatic Honey Bee Flower-Patch Assays with Paint Marking Re-Identification. CV4Animals workshop at the IEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR), Vancouver, June 2023. Oral and poster presentations.
|
Progress 08/01/21 to 07/31/22
Outputs
Target Audience:The dissemination efforts during this reporting period targeted: - Dissemination to scientific community - Involvement of computer science and biology students - Outreach to general public - Outreach to high-school and elementary school studentsin Puerto Rico Changes/Problems:The hiring of the postdoc in Computer Vision could not be performed during this first year. Applications received did not match the need for this project. The hiring of postdocs in Artificial Intelligence is particularly challenging, due to the high competition from companies offering much higher salaries for similar profiles. The position has been advertised throughout the year, and we continue to disseminate through our academic network. The tasks that were planned to be performed by the postdoc specialist in Activity 1were partly performed by using synergies with external resources, in particular graduate student Jeffrey Chan, that was mentored by PI Megret on Machine Learning tasks: as we already have software for baseline processing of tasks T1.1 (detection) and T1.2 (tracking), we focused on task T1.3b, to create the foundation for automatic individual identification and Deep Appearance Descriptors, that led to a publication. The integration of these tools is currently done this summer through the work of a team of interns to collect and apply the existing approaches to the data from the project. We postponed improvement of the models in T1.1 and T1.2 to the time the postdoc can be hired, as it requires deeper knowledge. If a suitable postdoc hire is not found by this summer, we consider hiring a specialist in Computer Vision or Machine Learning with a Master degree instead, and complement the work needed with additional students, as we have done this year. The scope of the work is not changed, but this resulted in a significantly lower expense that initially planned. We may request a no cost extension to be able to complete the tasks with more time to accomodate for the delayed availability of this specialist. What opportunities for training and professional development has the project provided?The Computational team subproject (Activity 1 and 2) provided training and professional development opportunities to 3 Computer Science undergraduate students: 1) Daniel Suazo, 2) Joel Gonzalez, 3) Alejandro Soledad and 1 Mathematics/Computer Science students: 1) Jeffrey Chan, MSc student. The training included for the undergraduates: software development in python, web development, database development; and for the graduate student: advanced machine learning design, software development and high performance computing. All students applied their training to biological and ecological data within the project. The Managed Pollinator's team (Activity 3) subproject provided training and professional development opportunities to 4 Biology students: 1) Noel Fanfan, Ph.D student; and 2) Evelyn J. Aviles Rios (undergraduate), 3) Luke Meyers (undergraduate), 4) Valentina Aviles (undergraduate). The training included Digital Design and use of 3D printer and Laser-cutter; electronics, arduino programing, management of honey bee colonies and general Honey bee biology and neuroscience concepts. The Wild Pollinator team subproject (Activity 4) provided training and professional development opportunities to 1 Biology student: 1) Solimar Marrero, Ph.D student. She received training on the visual identification of insects of four different orders (Coleoptera, Diptera, Hymenoptera and Lepidoptera) and in the use of digital annotation platforms using the Labelbee system How have the results been disseminated to communities of interest? Presentation at the Technology Week 2021: In October 27, 2021, one of our Biology undergraduate students, Evelyn Jun Avilés Rios made a presentation titled "The use of technology to understand decision-making processes in the honey bee". This was a virtual presentation that impacted more than 200 students of 9th to12th grade from 2 different public schools: 1) Escuela Especializada en Ciencias y Matemáticas Papá Juan XXIII en Bayamón; 2) Escuela Secundaria de la Universidad de Puerto Rico. The presentation can be found at the following link: https://drive.google.com/file/d/1iFCLQdrQLx-jl7ir8wc4c1Dstww9VasX/view?ts=61799c4c Presentation to public elementary school students: In April 25, 2022, Evelyn Jun Avilés Rios made a presentation to a section of 2nd grade students in an elementary school named Escuela Rafael Colón Salgado in the city of Bayamón, PR where 23 students were impacted. This activity had a power point presentation portion but also included hands-on experience using a real observation hive where student learn to distinguish workers from the queen and the drones. The power point included the following topics: 1) Importance of bees in pollination of our food; 2) How is like to be a scientist; 3) Basic bee biology concepts and; 4) Decision making in honey bees. Newspaper articles: One of the mainstream newspapers in Puerto Rico named Primera Hora published an article on the 18 of May 2022 where Co-PI Dr. Tugrul Giray is interviewed about his scientific discoveries of regarding the honey bees of Puerto Rico and its potential commercial value (see link "a"below). The same article was also published in Ciencia Puerto Rico, which is a scientific community network that includes puerto rican scientist around the globe (see link "b"below) https://www.primerahora.com/noticias/gobierno-politica/notas/sin-despegar-la-industria-apicola-en-la-isla/ https://www.cienciapr.org/es/external-news/sin-despegar-la-industria-apicola-en-la-isla What do you plan to do during the next reporting period to accomplish the goals?Activity 1: The various models have shown very good preliminary results in terms of accuracy. With the hiring of the postdoc or an alternate person, we plan to integrate these models better, train them on the new data from Activity 3 and 4, and work on the computational performance aspect in term of speed and integration in the real-time system. Activity 2: Both cloud and edge system have a working preliminary version. For the cloud system, we plan to connect the Labelbee annotation system to the models in Activity 1 to improve integration of the platform for end-user. For the edge system, we will integrate the models from Activity 1 on the platform to validate the feasibility of the real-time processing in the field. Activity 3: The preliminary system for the flower patch data collection works on 2 flowers. During the summer, we plan to finish the experiments required to publish the automated nectar delivery system during the summer and prepare the publication during the fall 2022. We will also collect videos of the flower patch as described in T3.1a-c. Activity 4: Now that the methodology and tools for annotation have been co-designed between ecology and computing teams, larger scale annotation will be performed, and used in Activity 1 for model training. Integration of automatic detection models will help in this annotation, to extract the data for the analysis of pollinator population and behavior on yield. Dissemination: We plan to continue with our outreach activities and present in the Entomology Society Meeting.
Impacts
What was accomplished under these goals?
During this year, work was performed on all 4 activities matching the 4 objectives stated in the goals. Activity 1: Machine Learning Models Group: Remi Megret, PhD (PI); Jeffrey Chan, BS (MS student); Luke Meyers (BS student) T1.1 Detection and pose estimation. Using video data from T3.1/T3.2, we trained deep convolutional networks to detect individual honeybees in the videos. The pose estimation accuracy was excellent and directly usable for T1.2. T1.2 Individual tracking and flower visits. Detections from T1.1 were used to generate identity-based tracks. We developed a script to count the total number of visits using the proximity of tracks to known flower locations. An evaluation on 3 validation videos collected from Activity 3 showed good preliminary performance: the pipeline could recall all 98 human annotated visits, with a 9% loss of precision due to double detections. T1.3 Payload, individual, and species identification. The honeybee video dataset gurabo10 from a previous project was used to create a new large-scale image dataset for honeybee re-identification. The dataset contains 118,616 images of aligned individual honeybees organized into short-term tracks, where 8,962 images also have long-term tag identity. It was found that the model trained with only short-term tracks could outperform the models trained using tag information. We developed re-identification at track level, reaching 84.3% Top-1 re-identification accuracy for re-identification on the same day amongst 10 distractors (Chan, 2022). A major outcome of this study is to show the possibility of recognizing honeybees without any markers. The same setup will be re-used for re-identification of the complete body, which we expect to have better performance when paint markings is available. Activity 2: Video collection and analysis platforms Group: Remi Megret, PhD (PI); Carlos Corrada, PhD (Co-PI); Joel Gonzalez (BS student); Alejandro Soledad (BS student) T2.1 Cloud Platform for collaborative annotation and visualization. We extended and improved the Labelbee video annotation system: a) A major refactoring was performed to connect to a database that stores and manages all the entities manipulated in the interface: users, videos with their metadata, annotations, as well as improve efficiency and accessibility. It is deployed at the High Performance Computing facility of the UPR. b) Performance improvements now provides users with fast display even with high numbers of flowers and skimming through the video interactively to help in the determination of precise visit timing and species. c) The labeling system was extended to allow different types of annotations/events and their associated labels (species, flower type...). The new "flower visit" complex event can receive optional labels and temporal intervals. The design was iterated multiple times with the group from Activity 4 to enable flexible and efficient annotation even in difficult cases where an insect pollinates multiple flowers at once. Python code was developed to extract visitation statistics (e.g. number of visits and their average duration). T2.2 Real-time Edge Computing Platform. The proof-of-concept of the video collection system T2.2a was implemented. It provides continuous recording on the NVIDIA Xavier platform and was deployed in the field in T3.1. Activity 3: Managed Pollinators Monitoring Group: Tugrul Giray, PhD(Co-PI); José L. Agosto-Rivera, PhD (Co-PI); Noel Fanfan, MS (PhD student); Juan R. Leon (Institutional Staff); Evelyn J. Aviles Rios (BS student); Luke Meyers (BS student); Valentina Avile (BS student). T3.1 Collection of video data for honey bee forager visit to the artificial flower patch. The flower patch automation requires 2 main components: 1) an automated nectar delivery system to eliminate the need of researchers refilling the flowers after each visit. 2) a video data analysis set up to eliminate the need of 2 researchers / flower patch observing which bee goes to which flower. We have successfully developed the following components of the automated nectar delivery system: A) Syringe Pump; B) sucrose "nectar" sensor and artificial flower and; C) electronic control system. Since the sensor developed involved a small electric current passing through the nectar that will be consumed by the visiting bees, an experiment to examine whether this current was aversive to the bees was performed. The experiment has 2 flowers: one "with current" using the electric sensor described above, and another flower "without current" that would be refilled by pressing a button after a bee visit. Although we need more experiments to be able to draw a firm conclusion, we observed that both flowers were highly visited during our experiment. The T2.2 Edge platform was deployed in the field on this 2 flower setup. We collected data for training and evaluation of Activity 1: 3 videos of bees visiting the flowers and 11 videos of bees with diverse color spot markings for re-identification. T3.2 Annotation of video data. We manually annotated 170 frames from the T3.1 videos for pose detection (body parts: head, thorax, abdomen...), as well as flower position and 98 individual visits to the flowers. These annotations were used in T1.1 and T1.2. Activity 4: Wild Pollinators Monitoring Group: Elvia J. Meléndez Ackerman, PhD (Co-PI), Irma Cabrera Asencio, PhD (Senior Personnel), Solimar Marrero (PhD student) T4.1 Annotation of wild pollinators video data. The objective is to annotate existing videos of insect flower visitors of four cultivars of Mangifera indica (mango: Tommy Atkins, Kent, Keitt and Julie) to develop reference data sets to facilitate insect recognition and complex analyses of floral visitation. The Wild Pollinator group met weekly with the computing team to receive training on the use of the LabelBee platform and co-design the new features. Meetings resulted in an iterative process where weekly decisions on the metadata of the videos and the type of variables to be generated and labeling needed. We decided on a metadata scheme to organize all videos and integrate them in the Labelbee platform in a way that will allow us to easily track back the data to individual trees. We co-designed annotations to evaluate the duration of inflorescence visits. To that effect we annotated three events: the region of the inflorescence visited, the entry and exit points of each insect. point of entry and the point of exit. Five videos have been observed to obtain a visual estimation of the number of flowers visited per inflorescence. We annotated all distinguishable flowers observed on 25 input videos of the cultivar Tommy Atkins (~648 hrs of annotation for 12.5 hrs of video), and labeled them as "male", "female", or "sex-unknown". In each video, more than one hundred and fifty flowers were identified and labeled. Annotations allowed to id each individual flower in such a way that the insect flight trajectory from T1.2 can be used in the future to obtain a finer scale analysis of insect behavior (ex. Handling time) within an inflorescence and their variations among species. Five of these videos were annotated for both flowers and visits and the remaining videos were annotated for flower visits. A list of the insects that can appear in the videos was prepared and an abbreviation was made to make it easier to locate on the Labelbee platform. We began training the annotator to identify the insect species and to provide and ID in the video. An insect guide was developed to facilitate further annotation and help other annotators to identify the insects. It currently contains 18 types of mango pollinators. This guide will be expanded upon, as ongoing annotation work uncovers additional species and their appearance in typical monitoring videos
Publications
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2022
Citation:
Jeffrey Chan, Hector Carri�n, R�mi M�gret, Jose L. Agosto-Rivera, Tugrul Giray:
Honeybee Re-identification in Video: New Datasets and Impact of Self-supervision. VISIGRAPP (5: VISAPP) 2022: 517-525
|