Progress 07/01/21 to 08/31/22

Outputs
Target Audience: Nothing Reported Changes/Problems:Plan Problems Encountered The Underlined text is considered proprietary. Within SA1 we had a learning curve on how to integrate a remote web socket to be a subscriber to our webrtc based radio traffic. Because of the integration of four cloud based endpoints (.NET Core app, our node app, twilio api and google speech to text) these endpoints in some form needed to communicate with other endpoints in order to be operational. Desktop based development using Docker virtualization to host more than one app was marginally acceptable because it would require proper firewall mapping and open ports which are specific to each programmer's desktop (or Covid work at home options) working on the research. We evolved into having development servers in our data center with shared ssh or rdp methods to iterate testing with a known public ip/port set. We learned the telephony subscription was limited to a 8khz rate and this fact had to be propagated to properly handle recordings, mp3 conversion and the Google speech streams. To influence the speech recognition and have it adapt to the phraseology used by incident commanders we iterated through methods which post the metadata on each recognition request and eventually mastered the Google API so that we could store static JSON representation of the phraseology and weighting we wanted the speech recognition to use. What opportunities for training and professional development has the project provided? Nothing Reported How have the results been disseminated to communities of interest?Trials: Nick Brunacini with bshifter.com facilitated 2 focus groups on HazPrep over 12 months in conjunction with Train the Trainer and MayDay seminars held in Phoenix Arizona with a total of 33 officer level firefighters. Subsequent to the project the IAFF also put together 2 focus groups with their key hazmat instructors for a total of 14 participants. Feedback included methods to utilize the platform for after incident debriefs, the benefit of the enhanced Phase II search tools for risk/injury/exposure types and how the platform may be a companion to the HazMat curriculum. There was also significant interest in using the platform as a pre planning tool. What do you plan to do during the next reporting period to accomplish the goals? Nothing Reported

Impacts
What was accomplished under these goals? Progress for Specific Aim 1 (SA1) "Refine Functional Prototypes of Virtual Radio Technologies" The highlighted text is considered proprietary. inXsol developed a node-based application which is hosted in the cloud. This specialty server added in SA1 simplified the demands on the individual trainee participants, broadening accessibility to users with a modern web browser, modest internet connectivity and a speaker/microphone system. In SA1 we needed to address two design challenges of the prototype, the first being a reliance on the Chrome web browser because of its support for voice recognition, and secondarily a need for bandwidth optimization. Browser based voice communications occur over a protocol known as WebRTC. We coded a node application to become a "subscriber" to the voice (aka simulated radio traffic) channels being used in the training scenarios. In the Node application, since we now had the audio stream, we integrated into an enterprise class Google speech to text API. This removed the platform dependence on Chrome and also reduced client bandwidth needs. We advanced our real time messaging code using the SignalR web sockets technology. The web browser clients participating in the training, our .NET/SQl server where PracticeCom exists and the NODE based server (subscribing to audio streams) are kept in sync by this web socket protocol. The integration from .NET Core to the NODE server provides messaging containing metadata of who is speaking, their role and other attributes are passed into node. The NODE websocket manages to "peel" the relevant audio stream packets for a speaker and establishes a Google cloud speech to text session. The interim and final recognition is messaged back to the .NET Core application tagged with speaker info. Concurrently, the relevant audio stream for the radio transmission is assembled into a raw audio file and on conclusion of the transmission the raw audio is converted to a mp3 file for session logging and after training reviews. On the .NET Core side, the user interface displaying transcripts offers an editor to repair any misinterpreted phrases or words. These are stored in a database tagged with the user role and the correction. On subsequent transmissions - this weighted preference for "text word or phrase -> should be __" is being sent to node where it can be shipped to Google Speech to text to influence subsequent machine learning based recognition. During user testing we iterated the instructor experience to allow intuitive edit of transcripts and behind the scenes thai became new JSON formatted data passed to the Google cloud to better train for first responder unique phraseology. Progress for Specific Aim 2 (SA2) "Create Instructor Scenario Control for participants and Feature Set." The highlighted text is considered proprietary. inXsol developed a data storage structure and authoring user interface in the .NET Core to offer the instructor an ability to identify the planned scenario participant roles and work with "swimlanes" to load up scenario evolution visualization for each participant. During the pilot tests we adapted instructor led scenarios into the environment and iterated support for videos (looping or 1 time or on demand play), JPG/PNG stills, PDF and support for iframe which enabled the embedding of support tools like the first responder Emergency Response Guidebook or other web sites into the scenario. We iterated and optimized the instructor's ability to preview or drill into cells and mechanisms to push scenario evolution to participating students. We also iterated how the transcript would store what the student was seeing at the time they composed a radio transmission. Progress for Specific Aim 3 (SA3) "Conduct Pilot Tests and Refinements." inXsol utilized the platform in pilot tests alongside three "Mayday" training symposiums held in Phoenix Arizona. The Mayday events include role play radio exercises. For the in person events we paralleled the in building radio traffic into the system. The transmissions were also recorded at full fidelity for reuse as needed to accelerate iterations of the speech adaptation in SA1. We then conducted a series of test sessions with the International Association of Fire Fighters (IAFF) senior instructor cadre.

Publications