Progress 07/01/20 to 04/29/22
Outputs
Target Audience:Content from this project has been incorporated into formal classwork at the undergraduate and graduate levels at UIUC and Stanford University. This content covered various aspects of the project, from image processing and registration techniques for shadow detection to using neural networks for modeling the dynamic evolution of cloud shadows and safe control and coordination of multiple drones. Developments from this project were disseminated to the robotics, control, and machine learning communities through severalpeer-reviewed publications in the areas of collision free control of multiple unmanned vehicles, recurrent neural networks, and delay robust cyber-physical systems. We have also reached out to the precision agriculture community through informal dissemination about project activities and outcomes to individuals at the University of Minnesota Research Outreach Centers and the Minnesota Department of Agriculture. Changes/Problems:Staff transition and replacement may impact our research schedule. Due to administrative delay, the subcontract at UC Berkeley was not initiated until Feb 17, 2021, leading to a slower ramp up of the activities on this grant. However,afterrampingup to full activity, Berkeley has completedthe project sooner than the rest of the team, thatare all applying for a one year no-cost extension. What opportunities for training and professional development has the project provided?Each team member on this project has participated in the practical application of robotics, machine learning, guidance navigation and control, image processing, and Structure from Motion theory to an important problem in precision agriculture. The training and professional development obtained by active participation on a collaborative project from concept development through implementation, testing, and evaluation, is a unique educational and professional experience. A Stanford undergraduate has had one-on-one mentoring with faculty and graduate students at Stanford as well as with individuals from industry at Sentek Systems. By developing the shadow detection module and much of the drone interface software, he developed new technical skills pertaining to image processing, network programming, mobile App development, and working with new application programming interfaces. Through presentations to the full team, he has also improved his communication and presentation skills. A graduate student at UIUC has been mentored and guided by the project personnel at UIUC, UC Berkeley and Sentek Systems. By developing and implementing vehicle guidance algorithms in a real system, with access to diverse expertise, he has developed his skills at bridging the gap between engineering theory and practice. He also developed a new estimation scheme that has been successfully applied to stochastic approximations and learning. Some of this work has been completed and submitted to the 2021 NIPS conference, which is one of the leading conferences in machine learning. A graduate student at Stanford University was mentored by Stanford faculty and individuals at Sentek Systems. By contributing to the shadow detection algorithms and software and integrating it into the Recon ground control station, she gained valuable real-time systems integration and development skills. Also, by mentoring and supporting one of the undergraduates on the project and keeping those activities progressing and on schedule she developed technical management skills. A postdoctoral research associate at UC Berkeley has been working closely with all project personnel on developing an estimation scheme to model cloud dynamics. The current approach relies on deep neural network based machine learning algorithms, although other more traditional approaches have also been explored. He is working most closely with the faculty PI and a Master's student at UC Berkeley on the UAV path planning problem, plus an undergraduatesophomore student on the shadow propagation esimation problem. By providing research mentorship to the students over the course of one year, he has gained professional management skills. With the UC Berkeley and UIUC PIs he has also been working on modeling and controlling adversarial attacks on recurrent neural networks. This work has resulted in oneconference publication and one in anappliedcontrol journal. This work will lead to safer and more reliable recurrent neural network dynamical systems. Motivated by this project, arelated work on communication delay robust cyber-physical system has resulted in a publication in another applied controls journal.The sophomore undergraduate student has gained valueable experience in training and deploying deep neural network based prediction systems to real world robotics problems, as well as gainedsoftware development skills through this project. The Master's student has been working closely with the posctoctoral research associate and individuals at Sentek Systems to explore several different approaches to coverge path planning, and has developed expertise in reinforcement learning based approaches to robot path planning. He has developed important reseach skills, and is currently implementing/testing his algorithms and documenting themcarefully as part of his Master's thesis. Undergraduates, graduate students, and post-docs involved in this project have benefited from participating in the systems level engineering on a complex problem. Working through requirement specifications, complex problem decomposition, and precisely defining interfaces between components, all with the guidance of faculty and industrial professionals, is excellent preparation for working on future large-scale engineering projects in team environments in both industry and academia. Many of the students and the postdoc involved in this project who are benefiting from the mentorship and professional development opportunities it has created are from under-represented groups, including women and persons of color. How have the results been disseminated to communities of interest?Dissemination of project developments has been achieved through three publications in peer-reviewed journals and refereed conferences from UC Berkeley with one morepublicationunder-review. Material from this project has been incorporated into coursework at Berkeley, Stanford, and UIUC in support of our educational objectives. Outreach to the agricultural community in this reporting period has been through informal discussions with individuals at the University of Minnesota Research Outreach Centers and the Minnesota Department of Agriculture. As the project matures, outreach activities will be more extensive. The outcomes are also disseminated and made publicly available through the project website https://publish.illinois.edu/mvscsfipa/sample-page/, which is maintained by the UIUC personnel and the help of UIUC College of Engineering IT department.Demonstrations will also be incorporated into community outreach activities conducted by UIUC, Stanford, and UC Berkeley, including through the UIUC Engineering Open House and the UIUC Engineering Outreach Society. What do you plan to do during the next reporting period to accomplish the goals?
Nothing Reported
Impacts
What was accomplished under these goals?
In this project, we aim to develop a system of multiple cooperating UAVs that will track cloud shadows and collaborate to collect shadow-free imagery of an agricultural field. This will significantly extend the range of lighting and cloud conditions under which drone-based imaging can be conducted, potentially tripling the amount of usable imagery that can be collected in a given amount of time in many parts of the country. Additionally, by enabling the simultaneous use of multiple, collaborating drones in agricultural settings, we aim to make it possible to scale up the use of drones in agriculture to realistic, industrial scale fields. This will increase the viability of incorporating drones into farm management practices, enabling more modern, efficient, and adaptive nutrient management methodologies, thereby reducing excess nutrients left in the environment and reducing fertilization costs to farmers. The expected audience interested in this project's outcomes will be from the precision agriculture, robotics, control, and machine learning communities. Outreach activities will target underrepresented groups as presented in the broader impacts section of the proposal. Two datasets were created to support goal #1 of this project. The first is a large-scale dataset derived from publicly available sources. It consists of 5,720 sequences that are 20 frames in length, of size 64x64, showing cloud evolution in various sky landscapes and lighting conditions. The second is a collection of high-resolution videos that we collected using drones with fisheye cameras showing dynamic cloud shadows cast on agricultural landscapes. This dataset consists of 102 minutes of 4K imagery from 7 different sites and includes ground control points (with known GPS locations) for each site. Several neural network models have been explored and tested using these datasets to model cloud shadow evolution. Additional techniques (e.g. autoencoders) have been explored for compressing the representation of cloud shadow imagery to reduce dimensionality and complexity of the models. From UC Berkeley, this work led to threepeer-reviewed publications: one in the 2020 IEEE Conference on Decision and Control, one in IEEE Control Systems Letters, and one in the IEEE Transactions on Control Systems Technology. Two student project has been started and arestill ongoing to support goal #2. The project focuses on combining and blending sometimes conflicting objectives to create coveragealgorithms for tasking drones to collaboratively image a given area based on a diverse set of available information, including predicted shadow locations, environmental constraints such as obstacles and no-fly zones, and drone limitations (e.g. battery remaining, and anti-collision constraints).A sub-project focused on biologically inspired collision avoidance directly supported goal #3. This work led to a publication that is published in the Proceedings of the 2021 American Control Conference. Several activities supported the validation of theory and algorithms being developed under this project (goal #4). Three DJI Inspire 2 drones were outfitted with multi-spectral camera systems and another was outfitted with a calibrated fisheye camera to support data collection and experimentation. An open-source, multi-vehicle ground control station is under development with major components being completed within this reporting period. This software project includes modules with precisely defined interfaces for the different key components of this project. It will enable us to bring together the work on shadow detection, shadow evolution modeling with neural networks, and the guidance and control components into an integrated, fully functional system that will be used for testing, validation, performance evaluation, demonstration, and outreach. In support of goal #4, dissemination of project developments has been achieved through three publications in peer-reviewed journals and refereed conferences from UC Berkeley and a few additonal ones from UIUC. Material from this project has been incorporated into coursework at UIUC, UC Berkeley, and Stanford in support of our educational objectives. Outreach to the agricultural community in this reporting period has been through informal discussions with individuals at the University of Minnesota Research Outreach Centers and the Minnesota Department of Agriculture. As the project matures outreach activities will be more extensive. Goal #5 is being supported at Berkeley through the Girls in Engineering program. This year, the annual summer camp was moved online due to COVID-19. 111 campers are attending the online camp, which runs 8:30-12:30 from June 15 through July 17. The curriculum includes maker projects, virtual tours of research labs, Q\&As with researchers, outdoor experiences, and Fabulous Fridays, which focus on fabrication skills, prototyping, building and making. Parts kits were provided to all campers, as well as Chromebooks and internet hotspots for campers who needed those resources.
Publications
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2021
Citation:
Shankar A. Deka, Duaan M Stipanovic, and Claire J Tomlin. "Feedback-Control Based Adversarial Attacks on Recurrent Neural Network." In 59th IEEE Conference on Decision and Control (CDC), pages 46774682, 2020. https://doi.org/10.1109/CDC42340.2020.9303949.
- Type:
Journal Articles
Status:
Awaiting Publication
Year Published:
2022
Citation:
Shankar A. Deka, Duaan M Stipanovic, and Claire J Tomlin. "Dynamically Computing Adversarial Perturbations for Recurrent Neural Networks." In press, IEEE Transactions on Control Systems Technology (2022). https://doi.org/10.1109/TCST.2022.3165980
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2022
Citation:
Shankar A. Deka, Donggun Lee, and Claire J. Tomlin. "Towards CyberPhysical Systems Robust to Communication Delays: A Differential Game Approach." IEEE Control Systems Letters 6 (2021): 2042-2047.
- Type:
Journal Articles
Status:
Under Review
Year Published:
2022
Citation:
Shankar A. Deka, Alonso M. Valle, and Claire J. Tomlin. "Koopman-based Neural Lyapunov functions for general attractors." Under review, IEEE Control Systems Letters (2022).
|
Progress 07/01/20 to 06/30/21
Outputs
Target Audience:Content from this project has been incorporated into formal classwork at the undergraduate and graduate levels at UIUC and Stanford University. This content covered various aspects of the project, from image processing and registration techniques for shadow detection to using neural networks for modeling the dynamic evolution of cloud shadows and safe control and coordination of multiple drones. Developments from this project were disseminated to the robotics, control, and machine learning communities through three peer-reviewed publications during this reporting period in the areas of collision free control of multiple unmanned vehicles and recurrent neural networks. We have also reached out to the precision agriculture community through informal dissemination about project activities and outcomes to individuals at the University of Minnesota Research Outreach Centers and the Minnesota Department of Agriculture. Changes/Problems:The current staff on this project is in transition. The graduate student at Stanford University, who is currently leading the shadow detection effort, will be completing her PhD and most likely accepting a position in industry. The postdoctoral fellow at Berkeley leading the shadow propagation effort may be accepting a faculty position at another institution. The undergraduate at Stanford University currently working on the drone interface component of the ground control station has finished his REU and now has another internship. Staff transition and replacement may impact our research schedule. Due to admisntrative delay, the subcontract at UC Berkeley was not initiated untilFeb 17, 2021, leading to a slower ramp up of the activities on this grant. However, Berkeley has now ramped up to full activity, and we do notanticipate that this will anyproblem from now on. What opportunities for training and professional development has the project provided?Each team member on this project has participated in the practical application of robotics,machine learning, guidance navigation and control, image processing, and Structure from Motion theory to an important problem in precision agriculture. The training and professional development obtained by active participation on a collaborative project from concept development through implementation, testing, and evaluation, is a unique educational and professional experience. A Stanford undergraduate has had one-on-one mentoring with faculty and graduate students at Stanford as well as with individuals from industry at Sentek Systems. By developing the shadow detection module and much of the drone interface software, he developed new technical skills pertaining to image processing, network programming, mobile App development, and working with new application programming interfaces. Through presentations to the full team, he has also improved his communication and presentation skills. A graduate student at UIUC has been mentored and guided by the project personnel at UIUC, UC Berkeley and Sentek Systems. By developing and implementing vehicle guidance algorithms in a real system, with access to diverse expertise, he has developed his skills at bridging the gap between engineering theory and practice. He also developed a new estimation scheme that has been successfully applied to stochastic approximations and learning. Some of this work has been completed and submitted to the 2021 NIPS conference, which is one of the leading conferences in machine learning. A graduate student at Stanford University was mentored by Stanford faculty and individuals at Sentek Systems. By contributing to the shadow detection algorithms and software and integrating it into the Recon ground control station, she gained valuable real-time systems integration and development skills. Also, by mentoring and supporting one of the undergraduates on the project and keeping those activities progressing and on schedule she developed technical management skills. A postdoctoral research associate at UC Berkeley has been working closely with all project personnel on developing an estimation scheme to model cloud dynamics. The current approach relies on deep neural network based machine learning algorithms, although other more traditional approaches have also been explored. He is working most closely with the faculty PI and an incoming Master's student at UC Berkeley on the shadow estimation problem, plus an undergraduate rising sophomore student on the UAV path planningproblem.By providing research mentorship to the students over the course of one year, he has gained professional management skills. With the UC Berkeley and UIUC PIs he has also been working on modeling and controlling adversarial attacks on recurrent neural networks. This work has resulted in one conference publication and an upcoming submission to a top applied control journal. This work will lead to safer and more reliable recurrent neural network dynamical systems. Undergraduates, graduate students, and post-docs involved in this project have benefited from participating in the systems-level engineering on a complex problem. Working through requirement specifications, complex problem decomposition, and precisely defining interfaces between components, all with the guidance of faculty and industrial professionals, is excellent preparation for working on future large-scale engineering projects in team environments in both industry and academia. Many of the students and the postdoc involved in this project who are benefiting from the mentorship and professional development opportunities it has created are from under-represented groups, including women and persons of color. How have the results been disseminated to communities of interest?Dissemination of project developments has been achieved through three publications in peer-reviewed journals and refereed conferences. Material from this project has been incorporated into coursework at Berkeley, Stanford, and UIUC in support of our educational objectives. Outreach to the agricultural community in this reporting period has been through informal discussions with individuals at the University of Minnesota Research Outreach Centers and the Minnesota Department of Agriculture. As the project matures, outreach activities will be more extensive. The outcomes are also disseminated and made publicly available through the project website https://publish.illinois.edu/mvscsfipa/sample-page/, which is maintained by the UIUC personnel and the help of UIUC College of Engineering IT department. What do you plan to do during the next reporting period to accomplish the goals?In the next reporting period, there will be several activities to finish the development of the multi-vehicle system testbed. We will finish the remaining components and modules of the Recon ground control station. Specifically, the student project relating to shadow propagation modeling with neural networks and the student project on guidance and control will continue into the next period. Additionally, new students are being brought on board to continue work on the drone interface component and to work on testing and debugging that portion of the ground control station. After this, multi-drone testing and full system testing will be conducted. Sentek will make necessary changes to their Structure from Motion pipeline to support processing datasets collected by multiple collaborating drones. This will enable us to collect datasets with the multi-vehicle testbed and evaluate overall system effectiveness and performance by generating shadow-free reconstructions and evaluating their quality. The team will continue to publish results that have been supported by this project to disseminate advancements to the interested academic and industrial communities. Additional results will also be incorporated into course curricula at UIUC, Stanford, and UC Berkeley to support ongoing educational objectives. As the testbed matures, outreach activities to the precision agriculture communities will accelerate and we expect to give presentations and either live or recorded demonstrations of the system to interested groups through events at University of Minnesota Outreach Centers and Minnesota Department of Agriculture field days. Demonstrations will also be incorporated into community outreach activities conducted by UIUC, Stanford, and UC Berkeley, including through the UIUC Engineering Open House and the UIUC Engineering Outreach Society.
Impacts
What was accomplished under these goals?
In this project, we aim to develop a system of multiple cooperating UAVs that will track cloud shadows and collaborate to collect shadow-free imagery of an agricultural field. This will significantly extend the range of lighting and cloud conditions under which drone-based imaging can be conducted, potentially tripling the amount of usable imagery that can be collected in a given amount of time in many parts of the country. Additionally, by enabling the simultaneous use of multiple, collaborating drones in agricultural settings, we aim to make it possible to scale up the use of drones in agriculture to realistic, industrial-scale fields. This will increase the viability of incorporating drones into farm management practices, enabling more modern, efficient, and adaptive nutrient management methodologies, thereby reducing excess nutrients left in the environment and reducing fertilization costs to farmers. The expected audience interested in this project's outcomes will be from the precision agriculture, robotics, control, and machine learning communities. Outreach activities will target underrepresented groups as presented in the broader impacts section of the proposal. Two datasets were created to support goal #1 of this project. The first is a large-scale dataset derived from publicly available sources. It consists of 5,720 sequences that are 20 frames in length, of size 64x64, showing cloud evolution in various sky landscapes and lighting conditions. The second is a collection of high-resolution videos that we collected using drones with fisheye cameras showing dynamic cloud shadows cast on agricultural landscapes. This dataset consists of 102 minutes of 4K imagery from 7 different sites and includes ground control points (with known GPS locations) for each site. Several neural network models have been explored and tested using these datasets to model cloud shadow evolution. Additional techniques (e.g. autoencoders) have been explored for compressing the representation of cloud shadow imagery to reduce dimensionality and complexity of the models. This work led to two peer-reviewed publications: one in the 2020 IEEE Conference on Decision and Control, and one in the Journal of Optimization Theory and Applications. A student project has been started and is still ongoing to support goal #2. The project focuses on combining and blending sometimes conflicting objectives to create guidance algorithms for tasking drones to collaboratively image a given area based on a diverse set of available information, including predicted shadow locations, environmental constraints such as obstacles and no-fly zones, and drone limitations (e.g. battery remaining, and anti-collision constraints). A sub-project focused on biologically inspired collision avoidance directly supported goal #3. This work led to a publication that is published in the Proceedings of the 2021 American Control Conference. Several activities supported the validation of theory and algorithms being developed under this project (goal #4). Three DJI Inspire 2 drones were outfitted with multi-spectral camera systems and another was outfitted with a calibrated fisheye camera to support data collection and experimentation. An open-source, multi-vehicle ground control station is under development with major components being completed within this reporting period. This software project includes modules with precisely defined interfaces for the different key components of this project. It will enable us to bring together the work on shadow detection, shadow evolution modeling with neural networks, and the guidance and control components into an integrated, fully functional system that will be used for testing, validation, performance evaluation, demonstration, and outreach. In support of goal #4, dissemination of project developments has been achieved through three publications in peer-reviewed journals and refereed conferences. Material from this project has been incorporated into coursework at UIUC, UC Berkeley, and Stanford in support of our educational objectives. Outreach to the agricultural community in this reporting period has been through informal discussions with individuals at the University of Minnesota Research Outreach Centers and the Minnesota Department of Agriculture. As the project matures outreach activities will be more extensive. Goal #5 is being supported at Berkeley through the Girls in Engineering program. This year, the annual summer camp was moved online due to COVID-19. 111 campers are attending the online camp, which runs 8:30-12:30 from June 15 through July 17. The curriculum includes maker projects, virtual tours of research labs, Q\&As with researchers, outdoor experiences, and Fabulous Fridays, which focus on fabrication skills, prototyping, building and making. Parts kits were provided to all campers, as well as Chromebooks and internet hotspots for campers who needed those resources.
Publications
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2021
Citation:
Shankar A Deka, Duaan M Stipanovic, and Claire J Tomlin. Feedback-Control Based Adversarial Attacks on Recurrent Neural Network. In 59th IEEE Conference on Decision and Control (CDC), pages 46774682, 2020. https://doi.org/10.1109/CDC42340.2020.9303949.
- Type:
Journal Articles
Status:
Submitted
Year Published:
2021
Citation:
Shankar A Deka, Duaan M Stipanovi?, and Claire J Tomlin. Dynamically Computing Adversarial Perturbations for Recurrent Neural Networks. Submitted to IEEE Transactions on Control Systems Technology.
- Type:
Conference Papers and Presentations
Status:
Under Review
Year Published:
2021
Citation:
Shankar A Deka, Jason J Choi, and Claire J Tomlin. Koopman Based Sum-of-Squares Construction of Lyapunov-like Functions. In 60th IEEE Conference on Decision and Control (CDC), 2021, Austin, Texas, USA.
|