Recipient Organization
SKYWARD, LTD.
5717 HUBERVILLE AVE STE 300
DAYTON,OH 45431
Performing Department
UAS & Geospatial Solutions
Non Technical Summary
Fire location, direction, speed and intensity are critical pieces of information when planning for and engaging in forest wildland firefighting operations. Real-time and post mission imagery are available to Incident Management Teams, but are generally not disseminated to front line firefighters. Image-derived fire maps are used to plan and guide operations; however, the imagery is generally collected at night, hours before missions and the resulting maps are not widely distributed. The lack of broadly distributed data is likely attributable to the sizes and volumes of available data and a deficiency in commonly used field deployable Common Operating Picture (COP) visualization applications.Skyward will address these challenges by developing a small edge-computing device to interface with common optical sensors and communication equipment used on aircraft. Custom Machine Learning (ML) algorithms and software will locate and characterize fire lines and hotspots on collected imagery and convert these detections into small files in common Geographic Information System (GIS) formats. These files will be useable by a variety of COP visualization tools such as CivTAK, Enterprise Global Portal (EGP), and Skyward's Pursuer application. The device will meet the requirements for use on Unmanned Aerial Vehicles (UAVs) and manned aircraft to ensure broad platform suitability. The result will be a widely usable tool which produces small, real-time, image-derived fire detection GIS files for dissemination to a variety of COP applications.During Phase I, Skyward will benchmark and evaluate hardware, software, and machine learning approaches to identify optimal solutions for Phase II.
Animal Health Component
20%
Research Effort Categories
Basic
10%
Applied
20%
Developmental
70%
Goals / Objectives
1 Phase I Work PlanDuring Phase I, Skyward will conduct research to ensure selection of the best and most appropriate hardware and software solutions, develop testable prototypes, and run successful benchmarks. All work will be carried out at Skyward's offices in Dayton, Ohio. To maximize available budget and schedule, Skyward will use established resources, such as fire community contacts, university and government research databases, and internal hardware devices and software applications. Skyward will develop a final technical report and marketing strategy in tandem with research and development activities. This approach has been tested through numerous past SBIR projects for similar solutions, which have concluded successfully. This project is divided into 9 primary Tasks:1.1 Task 1 - Fire Characteristics ResearchThe primary objective of this task is to define important fire characteristics to determine which sensor types are most appropriate for visualizing fire and hotspots effectively. Research will focus on fire location, direction, speed and intensity, as these will not only affect sensor selection, but also help define data collection parameters like the periodicity of collection and the minimum geospatial accuracy required for reporting. Skyward is currently reviewing data available through sources such as the Missoula Fire Sciences Laboratory in an attempt to accelerate research. Successful completion of this task will lay the groundwork for the remaining tasks.1.2 Task 2 - Sensor Risk Reduction ResearchTwo major goals of this project are to ensure that our data processing solution is capable of connecting to a broad variety of optical sensors and processing most common imagery data formats. As such, the primary objectives of this task are to identify the most common optical sensors currently in use by the forest wildland fire community and down-select to specific sensors for this project based upon those findings and the results of Task 1. Additionally, Skyward will conduct research to learn which imagery data formats are most commonly used in general and which are available via sensors currently in operation, like the Phoenix Infrared line-scanner sensor used by the USDA Forest Service National Infrared Operations (NIROPS) Unit. Successfully acquiring this information will support follow-on tasks and ensure that our hardware solution can physically interface with commonly used optical sensors, and that our algorithms can process the most commonly used image data formats.1.3 Task 3 - Communication Systems Risk Reduction ResearchAircraft and ground-based personnel transmit and receive data through a variety of means. The goal of this task is to learn the methods and devices most commonly used by manned aircraft, UAS, Incident Management Teams and handcrew personnel. This information will help define requirements for hardware interfaces and specifications for data throughput. For instance, based upon previous research, we anticipate the integration of an RJ45 ethernet coupling into the FireEdge system for the passage of imagery into the device. We also assess the likelihood of serial port integration into the device as most radios support data output through serial and ethernet interfaces. Once we've confirmed which interfaces are best suited for data receipt and transmission, we will integrate these into our Phase I working prototype. As a result, Skyward's solution will be capable of interfacing with communication devices most often used during forest wildland fire operations, ensure that data outputs do not burden available bandwidths, and establish a solid baseline from which to develop a demonstration ready system in a follow Phase II project.1.4 Task 4 - Data Mining and CollectionIn order to train the ML algorithms discussed in Section 7.5 representative data from the sensors chosen in Section 7.2 is required. In the Phase I effort Skyward will obtain data from sources that are representative of the sensor data to be used in Phase II.1.5 Task 5 - Machine Learning (ML) Risk Reduction ResearchDuring the Phase Ieffort, Skyward will research the best methodology for creating a Neural Network (NN) for detection fire lines and hotspots in various sensor data. The detected fire lines and hotspots will be used to draw fire maps. During Phase I, NNs will be built and trained for the various types of sensor data. Skyward will benchmark the developed networks to evaluate the feasibility of fire mapping on edge.1.6 Task 6 - Software System DevelopmentThe primary objective of this task is to create the software system prototype. The core functionality will be to collect data from the hardware, process the data using the machine learning engine, send the resulting information to downstream systems for presentation to users. In order to meet this objective, we will perform several subtasks:• Requirements Identification and documentation• Solution approach decision and architectural design• Build software system environment and develop software• Test software systemSuccessfully completing this task will result in a foundational piece of software upon which further capabilities can be added and finalized during a Phase II project.1.7 Task 7 - Edge Detector Risk Reduction ResearchSkyward will prototype an edge computing device to receive and process imagery and deliver fire line and hotspot detections. Skyward has developed an ML-accelerated edge device for anomaly detection in Automatic Dependent Surveillance - Broadcast (ADS-B) data that is UAS-mountable, as shown in Figure 2.(NOT SHOWN / Figure 2: Skyward's UAS ML-accelerated ADS-B Anomaly Detector)Skyward will adapt the existing ADS-B edge computing device for the wildfire challenge. Modifications to this existing system, discovered through research and work during the previous tasks, will be implemented to develop an edge device suitable to fire mapping via aerial data collection. Additional environmental hardening will be considered to address issues such as weather, vibration and smoke. A working prototype will be completed in the Phase I effort.1.8 Task 8 - Common Operating Picture Risk Reduction ResearchThe primary objectives of this task are to perform research into integrating the system's data output with downstream systems, prototype connection interfaces, and integrate Wide Area Motion Imagery (WAMI) into the Pursuer visualization environment. Successful completion of this task will enable wildfire related information to be displayed on the various COP systems discussed in section 4.2.1.9 Task 9 - ReportingThe reporting task will occur over the course of the entire project. Two progress reports will be prepared at 3 and 6 months, and a final report will document the technical efforts of Phase I, establishing a technically feasible baseline system design and representative cost model, which will serve as the basis for entering Phase 2. The final report will be delivered to the customer at the end of the project. Included in the final report will be a cost analysis for projected commercialization. This will be based on the established baseline design and estimates for further development. Projected costs will account for currently known elements, and unknown elements discovered during the Phase I project, and will consider all the major cost areas like development, capital, tooling, labor, ILS, preproduction test, required maintenance/repairs/consumables costs, warranty, etc. This model can be run parametrically for varying manufactured quantities and systems usage rates (i.e., varied service life).
Project Methods
Technical ObjectivesTo lay the groundwork for the successful development of a persistent, real-time, automated fire line mapping capability, we will in this Phase I perform risk reduction research and prototype development. The Phase I effort will have four areas of risk reduction research: sensor, communication systems, edge detector, and common operating picture.Sensor Risk Reduction ResearchSome sensors and sensor types are better suited to locating and characterizing fires than others. Choosing sensors from which to build fire maps requires an understanding of the fire characteristics needed to successfully locate and characterize the properties of fires. Building fire maps requires the use of appropriate data types for visualization within COP environments. To do this, it is necessary to understand the characteristics of fire. Using those characteristics, sensor hardware requirements can be established to identify fire and hotspots in real time and produce data suitable for visualization. Skyward will research the capabilities of various sensors, including types used by the firefighting community, and the spectral characteristics of wildfires (both fire line area and discrete hot spots) to better inform our tool and algorithm development.Communication Systems Risk Reduction ResearchSkyward will research communication methods and devices most commonly used by manned aircraft and UAS employed by the forest wildland fire community. Incident Management Teams and handcrew personnel will be consulted regarding the common types of equipment used to ensure integration of the developed technology will be successful. This information will help define requirements for hardware interfaces and specifications for data throughputs, so Skyward's solution will be capable of interfacing with communication devices most often used during firefighting operations and data outputs do not burden available bandwidths.Edge Detector Risk Reduction ResearchCurrent wildfire detection is mostly based on mathematical and statistical methods with predefined thresholds[1]. Object detection in images is a well-established area in ML, however, most existing ML research on wildfires is focused on simple classification, and does not locate the fire or smoke in an image. Object detection can be used to locate fire within an image in combination with image geotags to build a fire map. It has been shown that the Faster Region-based Convolutional Neural Network (Faster R-CNN) can be used to locate fire[2] and smoke[3] in images. The data being considered in this effort will be spatial data, but is not limited to electro-optical (EO) images. Since EO imagery is limited in capturing valuable fire data the previous work using Faster R-CNN is not sufficient for the firefighting community. The convolutional layers in Faster R-CNN learn the spatial relationships in data, unlike fully-connected layers used in traditional ML problems. Convolutional layers will be utilized in the developed model, but the input layers to the neural network will be developed for each sensor of interest. Skyward will develop a custom ML algorithm, advancing on previous work that creates a fire map by identifying fire lines and hotspots in sensor data. The developed ML algorithms will provide the ability to utilize existing sensor feeds to autonomously create fire maps in semi-real time. The developed algorithms will be trained on acquired and simulated sensor data and benchmarked to ensure edge computing capability.Data for fire maps is generally collected using manned aircraft. However, trained aircrew are in limited supply and cannot operate indefinitely without rest, which may be a factor limiting the ability to collect map data persistently. As such, Skyward wants to ensure our solution can be deployed on UAS to provide a more flexible, persistent, cost-effective, and safer option. Despite the devices small size, the solution will rely on onboard processing to distill large quantities of low value sensor data (e.g., images) into small quantities of high value information (e.g., fire line and hotspot location points) that is ready for use without further processing. This will aid in quickly providing the most useful sensor data to firefighters in bandwidth limited environments. The edge detector device will also be environmentally hardened in order to address potential issues such as weather, platform vibration and smoke.The edge detector will require various software components to ingest sensor feeds, process the data, run the Neural Networks (NNs), and post-process the data into desired file types. Skyward will provide a prototype of the software for receiving, processing, and delivering wildfire related information. The prototypes will be tested on the edge detector hardware to establish the feasibility of performing wildfire detection on a UAS platform.Common Operating Picture (COP) Risk Reduction ResearchTo provide situation awareness for wildfires, it is necessary to interface with a COP to make information readily available and digestible to those in the field. In the Phase I effort, Skyward will develop prototypes of the interface and display capabilities between the edge device and various COPs to ensure users will have timely and actionable wildfire data. This will include CivTAK, which has been tested within the fire community and is presently in use with the Colorado Center of Excellence.[1] (Gabbert, Andalibi, & Jacob, 2015)[2] (Barmpoutis, Dimitropoulos, Kaza, & Grammalidis, 2019)[3] (Zhang, Lin, Zhang, Xu, & Wang, 2017)