Progress 07/01/21 to 02/28/22
Outputs
Target Audience:The study investigated how the implementation of a suite of assistive-reading technologies bundled with on-demand reading instruction could impact the accessibility and comprehension of employer training and policy materials for adults with limited literacy or English proficiency (LLEP). The primary audiences were LLEP adults and their employers. Trial participants included an all-immigrant set of employees at a fabricated steel products plant in eastern Pennsylvania and a diverse group of frontline employees at a window and door manufacturer in Minnesota whose preferred languages, i.e. first languages, were English, Hmong, Karen, and Vietnamese. We also held 67 interviews with employers in various industries and geographies: manufacturing (fabricated steel products, injection molding, custom enclosures, window and door), dairy and protein processing, and healthcare in Illinois, Michigan, Minnesota, Missouri, Oklahoma, Pennsylvania, Wisconsin and Texas. The employers ranged from a multinational food and protein processor with thousands of employees and locations in 50 countries, a steel products manufacturer with 900 employees spread across 10 locations in eight states to a small manufacturer of metal enclosures with just 50 employees. These interviews identified three specific areas in which implementing GogyUp's assistive-reading technologies in employee training and communications could play a significant role for employers and employees: Safety Employee Hiring and Retention Employee Productivity A secondary audience were learners enrolled in adult basic education or adult English as a second language (ESL) classes. Just prior to the project's start date, GogyUp was approved as a distance learning platform by Minnesota Adult Education for adult education. This enabled us to deploy GogyUp to adult education classes in three locations for both remote and hybrid (mixed remote and in-person, on-site) instruction. This implementation expanded both each adult education program's ability to train their learners and their learners' opportunity to receive instruction. A third, unintended audience were the 60 individual end-users who discovered the GogyUp Reader app on their own during the project's timeline, presumably through their mobile device's app store. These independent end-users averaged 5.3 sessions and 80 minutes during the trial period although one exceptional end-user read for over 22 sessions and a total of 5.5 tracked hours. Changes/Problems: 1) Goal 3 became unattainable due to the impact of the COVID-19 pandemic on a pre-existing labor market crunch. The additive follow-on labor shortage understandably severely affected employer receptiveness to outreach from the team or our partners at Michigan State University. This was the primary barrier to establishing a greater number of trials with a larger number of participants and the random control trial initially described in our application. Weather also played an unanticipated role in limiting the number of trials we were able to launch. Hurricane Nicholas disrupted one facility's power and operations in September 2021 and the ensuing backlogs scuttled a planned pilot. Another facility was impacted by the devastating Kentucky tornadoes in early December 2021 and it seemed inappropriate to pursue a trial. These setbacks along with the timing and effort involved in establishing trials will be factored into our effort analysis for future studies and commercialization plan. 2) The team's assumption that a suite of assistive-reading technologies could impact the accessibility and understandability of workplace documents was partially validated by the trial data and consistent themes emerging from employer interviews and employee surveys. A critical discovery was that these technologies must be flexibly packaged to meet the demands of different use cases - instead of the "all-in-one" app we prototype we developed. This key finding has ramifications for both the design of GogyUp's products but also the commercialization plan. 3) An employer'spolicies and security architecture can present a significant barrier to successful implementation and may require additional partnerships or the development of additional ancillary services. What opportunities for training and professional development has the project provided?
Nothing Reported
How have the results been disseminated to communities of interest?GogyUp's Co-Founder and CEO, Ned Zimmerman-Bence, presented findings from the employee trials to the Board of Directors of the Wire Reinforcement Institute (WRI) - the leading association of manufacturers, allied industries, professionals, and educators engaged in the production and application of structural welded wire reinforcement. That presentation led to an invitation to present our findings as WRI's guests at their World of Concrete booth - a trade event attended by 37,000 steel and concrete professionals - over a three day period in late January, 2022. GogyUp also presented during virtual "lunch and learn" hosted by Northeast Missouri economic development centers to the manufacturing and healthcare employer groups as well as the Community Venture Group's September gathering in Minnesota - attended by economic development centers from across the MidWest. Findings will be incorporated into a presentation to be shared with the adult education and workforce development communities through conference proposals similar to the presentation GogyUp gave at the national Coalition of Adult Basic Education conference in 2021. What do you plan to do during the next reporting period to accomplish the goals?
Nothing Reported
Impacts
What was accomplished under these goals?
Goal 1: Our primary goal was to evaluate whether implementing GogyUp would make documents more accessible. Surveys were administeredduring each product trial hosted by employers and adult learning centers. Results were compiled and analyzed by Dr. William Knudson, Product Marketing Economist at Michigan State University. Averages on questions such as "This app made it easier to understand the information" and "this app improved my ability to understand and comply with job requirements" were both 4.2 with individual responses ranging three to five. Lower scores came from the respondents who noted occasional difficulty understanding work documents (and therefore have a reduced need for the reading assistance). Respondents who noted consistent difficulty gave the app higher scores. The surveys also documented GogyUp's impact on understanding. Responses to the statement "this app improved my ability to understand and comply with job requirements" averaged 4.0 and ranged from three to five on a scale of 1 to 5.Given that the data was limited to a sample of 43 without pre- and post- testing, no definitive claim can be based on these results. But the resultsraise the possibility that GogyUp improves understanding - a possibility we are eager to study further. Focus groups comments also indicateda positive effect on employees'experience with training documents. For instance, employees found that GogyUp's EZ Text formatting conveyed the sense that they, "had more time - like less stressful since there's so much space".1 In-app usage data further indicated GogyUp's ability to increase content accessibility and how well employees understood the content. Metrics such as time on prompt, number of mistakes per attempt, and number of attempts on comprehension puzzles indicated how end-users engaged with the content andtheirabilityto understand the content. An anonymized usage and comprehension reportfrom the first Hazle Township trial is available here: https://bit.ly/gupreport1? Goal 2: Trials at the Hazle Township, PA and the Cottage Grove, MN facilities and the 67 customer interviews we held during the project indicate several potential gains or value points for implementing GogyUp in a workplace setting, especially for facilities with limited resources for training employees and LLEP adults in particular. A key value for employers was the "new window" into what and how well employees understood training and company information (e.g., policies) without the overhead and implementation costs of a new software system (such as a Learning Management System), new content, or new hardware. As the longtime Corporate Manager of Safety and the Environment at Jennie-O Turkey Store stated: "the game changing value I see, as a Safety and Environment professional, is the unprecedented depth of real time feedback and historical data regarding each employee's level of comprehension. Many learners fear acknowledging failure to comprehend and will feign understanding that does not exist. We have had serious accidents result from "properly trained" employees who clearly lacked understanding. GogyUp goes beyond to show HOW each employee understood the material and the activity they undertook to achieve mastery."2 Goal 3: Progress on identifying metrics to measure GogyUp's broader impact on the employer and community was minimal beyond a literature search and preliminary planning. In hindsight, the goal itself was most likely unachievable for the time frame. The obstacles we noted in the risk section of our application were unfortunately prescient and will be further described in the "Changes / Problems" section. Objective 1: The usage data generated by the workplace and education trials will provide the raw information for several measures to be developed in the future. These include the level and type of assistive-technology used on a per-document and per-individual basis, measures of knowledge acquisition different then the pre- and post-test paradigm, and measuring the impact of employee feedback (through anonymized usage data) to employers on current employer materials and subsequent revisions made based on that feedback. Objective 2: While several design changes and additional functionality needs were identified, the GogyUp Reader prototype's core product offering successfully met the technical, customer, and end-user design and function requirements. Key lessons spurred additional product development is continuing past this project. Key lessons stemmed from engagement with larger corporations with established information technology infrastructure and procurement requirements - all of which became valuable experience in reviewing the GogyUp codebase to ensure GogyUp met those requirements. Importantly, those experiences expanded our understanding of key constituencies within our customers and when to include them in sales and implementation discussions. It became clear at onboarding trials that the in the moment need to understand an employer's documentmight be a role for GogyUp's direct-to-consumer app, Snap Reader. Snap Reader app matches Google Translate functionalitywith GogyUp's additional features: saving documents for later reference, GogyUp's EZ Text formatting to facilitate comprehension, original document layout and media elements. However, additional engineering will be required to include the value employers and employees identified in GogyUp Reader: the comprehension feedback loop, metrics on engagement, etc.. It should be noted that while onboarding is only one use case it is a significant one as most orientations contain significant amounts of information that must be understood in order to work safely and efficiently. Importantly, this issue occurs not only with immigrant or non-English speaking employees but also with employees who had marginal education and never acquired the ability to read technical documentation but are otherwise English proficient - even native speakers.3 Objective 2, Item 1b, assessing network access, was not thoroughly achieved. Internally, the team decided that developing assistive technologies that could operate independent of network functionality (an assumed requirement for different rural locations) was not technically feasible in the study's timeline. Surprisingly, network independence did not seem to be a concern for the employers interviewed nor was it an issue in the more remotely located trials we conducted.. The team did uncover an unanticipated barrier to "frictionless" adoption - the company's IT department. Atrial was almost scuttled due to a central IT office changing the WiFi password without providing access to the facility's staff. This prompted a change in our approach and allowed us to experiment with providing and supporting cellular enabled tablets. Provisioning tablets that operate independently of an employer's infrastructure may turn into a value-added service, especially if GogyUp partners with a mobile device management organization. Objective 3: As with Goal 3, Objective 3 was overly ambitious for the project's timeline and was not achieved. The limited number and length of the employee trials the team was able to secure prevented measuring impact on training outcomes, let alone local labor force participation. Item 3b, quantifying impact on labor force participation, will most likely require a longitudinal study that was far outside the possible scope for this project. While benefits to training outcomes were also not quantified, employer interviews did identify benefits and metrics that could be used to enumerate GogyUp's value proposition. Further research and a longer project will be required to test these findings. 1. Post trial focus group, Hazle Township, PA. September 3, 2021. 2. Letter of support for Phase II NIFA SBIR Application - Submitting in 2023 3. Interview with Director of Safety, St. Petersburg, FL., July 15, 2021.
Publications
- Type:
Other
Status:
Other
Year Published:
2021
Citation:
We presented our findings through a slide deck and product demonstrations at Wire Reinforcement Institutes booth during the World of Concrete conference, January 20 to 23 2022, and the Northeast Missouri Manufacturing and Northeast Missouri Healthcare employers round tables on October 5th and November 15th, 2021.
|