Source: The Julia Group submitted to NRP
CROSSROADS: A GAME OF CHOICES: COMPUTER-BASED INTERVENTION FOR RURAL YOUTH IMPACTED BY SUBSTANCE ABUSE
Sponsoring Institution
National Institute of Food and Agriculture
Project Status
COMPLETE
Funding Source
SMALL BUSINESS GRANT
Reporting Frequency
Annual
Accession No.
1016786
Grant No.
2018-33610-28593
Cumulative Award Amt.
$599,187.00
Proposal No.
2018-03198
Multistate No.
(N/A)
Project Start Date
Sep 1, 2018
Project End Date
Aug 31, 2020
Grant Year
2018
Program Code
[8.6]- Rural & Community Development
Recipient Organization
The Julia Group
2111 7th St Number 8
Santa Monica,CA 90405
Performing Department
(N/A)
Non Technical Summary
Alcohol and drug use is epidemic in rural America, as geographic barriers impede contact between youth and service providers, there is a shortage of mental health professionals and low literacy precludes extensive written assessment and education.Drawing on research from substance abuse prevention and treatment, effective game design and adaptation of software for rural conditions, Crossroads develops and evaluates a mobile game to augment mental health services to youth in rural communities. The game engages high-risk youth, teaches and models healthy decision-making. Counselors receive screening test results and user responses to incorporate in services offered to game users. Users access Crossroads on devices they are comfortable using, remotely and at their convenience.By successfully meeting Phase I objectives, The Julia Group established the necessary foundation to bring to commercialization a product that addresses the needs of drug-endangered rural youth. Phase II expands our prototype with development of 12 commercial-quality game levels, addition of augmented reality features, creation of text, audio and video components for input to maximize youth engagement. The project has expanded to include five sites across two reservations and two regional centers in rural counties, adding as partners an in-patient treatment program, vocational rehabilitation program, school district and a university where we will conduct a multi-method usability analysis followed by an impact assessment.Games will be sold at $15 per user, a fraction of the cost of therapy. Long-term commercialization plans are three-pronged: expansion to more reservations communities; to other areas of risk and outside of tribal communities.
Animal Health Component
25%
Research Effort Categories
Basic
15%
Applied
25%
Developmental
60%
Classification

Knowledge Area (KA)Subject of Investigation (SOI)Field of Science (FOS)Percent
8066010307040%
8056099307040%
7246020302020%
Knowledge Area
724 - Healthy Lifestyle; 805 - Community Institutions, Health, and Social Services; 806 - Youth Development;

Subject Of Investigation
6010 - Individuals; 6020 - The family and its members; 6099 - People and communities, general/other;

Field Of Science
3070 - Psychology; 3020 - Education;
Goals / Objectives
Phase II extends and refines the Phase I prototype, expanding to more levels, more measures, more features and a much larger sample size. The technical questions needed to establish the commercial viability are:Can a game-based instructional system addressing substance abuse and mental health:Increase recruitment and retention of tribal youth in substance abuse programs?Increase the persistence of youth in completing interactive journals assigned in therapy?Increase the quantity and quality of assessment data collected on measures of substance abuse, depression, anxiety, trauma and PTSD?Can mobile video, text, audio, geographic and tactile capabilities be effectively integrated in a game format for assessment and treatment?How can hardware and network speed requirements be minimized to allow a mobile game to be playable over the average rural telecommunications network?Can a software application reduce substance abuse and related risks?Technical Objectives:Complete 12 commercial-quality levels in a virtual world that integrate video scenarios, decisions, gameplay, augmented reality and instructional videos or web pages. Twelve levels were selected as both large enough to have an impact and the maximum number that could be completed within the budget.Expand interactive journals modeled on motivational interviewing using audio, video and text messages, including alerts sent to staff in case of emergency.Develop in-game assessment for substance abuse and related risk factors.Conduct usability testing with youth in families affected by alcohol and drug abuse.Optimize software for performance in rural and remote locations.Create support website for program staff, with videos on game usage and installation, supplemental materials including presentations, videos and worksheets.Conduct a one-year, multi-site analysis of impact on tribal youth recruitment and retention in programs serving rural communities.
Project Methods
Usability testingUsability testing will be conducted with 20 subjects testing off-site in small towns and farm communities on and adjacent to reservations. While the previous task tests usability under optimal conditions, with a trained staff member present to provide support, in-building wi-fi - off-site usability testing tests the limits of bandwidth (via wi-fi and local cellular networks) and user experience requirements under conditions during which much of Phase II testing will occur. For the off-site condition, users will be provided refurbished phones with a data plan, at a cost of $40 per month. This hardware reproduces the level generally available to the target population. Data will be collected electronically on frequency and duration of gameplay sessions, modules completed and accesses of help screens. Project staff will record number, content, duration and resolution of phone and email contacts for customer supportImpact AssessmentA cohort of 125 youth will be recruitedfrom five sites in North Dakota.Baseline data will be collectedon average hours of intervention (excluding assessment), number and retention of youth served, substance abuse and number of positive goals in written change plans. Demographic data and visits on youth served by the programs who elect not to play the games will also be recorded, with game usage entered as zeroes. At the end of 12 weeks of gameplay, participants will be requested to replay the sections recording substance abuse and depression. Youth who don't play the game will complete the standard written assessment.Descriptive statistics will be computed for all demographic, usage and outcome variables. Correlations will be computed of software engagement, as measured by levels completed and total time played, with number of therapy sessions completed and pre- and post-test scores on measures of depression and substance abuse. Poisson regression analyses will be performed with number of visits as the dependent variable and software engagement, age, gender and substance abuse as the predictor variables. Z-tests will be used to compare average number of visits, average number of positive goals and average score on substance abuse and depressive measures to means for the prior three years, used as a historical baseline. Repeated measures ANOVA will be used to test for reductions in substance abuse and depressive symptoms, controlling for adverse events and age.

Progress 09/01/18 to 08/31/20

Outputs
Target Audience:The primary target audience is rural Native American youth who have a family member addicted to alcohol or drugs. Due to the location of American Indian reservations in rural communities, Native American youth are far more likely to reside in rural areas than the national average. This primary target audience is also predominantly socio-economically disadvantaged. Secondary audiences are: researchers and policy makers providing services to rural Native American youth who have a family member addicted to alcohol or drugs, and researchers and policy makers providing services to other rural and nonrural youth in families where a family member is addicted to alcohol or drugs. Changes/Problems:In June, 2020 the Center for Disease Control found that prevalence of symptoms of anxiety disorder was three times higher and depressive symptoms four times higher in adults 18 and older compared to in the second quarter of 2019 (Czeisler et al., 2020). Among youth 18-24 the situation was even worse, 75% experienced one or more adverse mental or behavioral health symptoms, compared to 41% of all respondents. The current pandemic fits the textbook definition of the history effect as a threat to internal validity of a study. That is, the environment changed the conditions such that the scores of the independent or dependent variables are changed and the scores of one group would change more than the other. Clearly, the pandemic had no impact on baseline data, as it occurred after these data were collected. For youth in the software testing group, however, it could be expected that increases in depression, anxiety and substance abuse would occur, just as happened for the country at large. One hypothesis to be tested was whether youth participating in the study, using Crossroads software, would attend more visits for intervention and complete more hours of intervention. In early March, 2020, several of the cooperating programs closed their doors temporarily and were providing no direct services. On-site services were prohibited by state mandate and they had not yet arranged for counseling via telemedicine or implemented social distancing protocols. Further complicating the situation, some of the sites were located on tribal lands and thus under the jurisdiction of tribal governments which could, and did, enforce their own work from home policies. On site observation and data collection by the investigator was not possible after February, 2020. Impact testing in the age of Coronavirus In response to the challenges to data collection presented by closed programs, programs closed to non-staff members, youth learning from home and recommendations against non-essential travel, the project conducted two studies of youth engagement. The first study included a sample of 1,370 users who were monitored solely electronically, collecting data on app usage. The second study recruited 47 middle school and high school youth to participate in an online game design course using the Crossroads application for beta testing. What opportunities for training and professional development has the project provided?Webinar - Level Up Your Lessons with 7 Generation Games with 40 synchronous attendees and 364 views to date of video presentations, August , 2020 Workshop - Game design as project-based learning for decision-making skills, 24 attendees, Spirit Lake Nation - February, 2020 In addition, 22conference presentations were given by project staff over the grant period, including 16in the current reporting period. How have the results been disseminated to communities of interest?Results have been disseminated primarily through conference papers and workshops. As noted above, a total 22 conference papers were presented, 16 in the current year. Two workshops were presented in the current reporting period and three in the prior period. In addition, the company has a mailing list of 1,915subscribers which was sent regular project updates and a summary of the interim and final reports. The project also maintains a website with general information and supplemental resources on The Julia Group site. In addition, the company has partnered with a non-profit foundation, Strong Mind Strong Body Foundation to distribute software and disseminate information with a quickstart guide for software usage, videos, frequently asked questions and technical support. The project also maintains a YouTube channel with videos promoting the software and supplemental resources. What do you plan to do during the next reporting period to accomplish the goals? Nothing Reported

Impacts
What was accomplished under these goals? Twenty-one tasks were proposed to address these seven goals. Nineteen tasks were either completed as proposed or exceeded. Two tasks related to subject recruitment and onsite data collection were modified due to accommodation to local conditions, and in response to the Coronavirus pandemic challenges. Major accomplishments include the completion of three mobile applications that comprised 21 levels (exceeding the objective), completing 28 interactive journal prompts, including the option for voice or text input. In-app automated data collection and scoring programs have been completed for five mental health measures. Modifications were designed in throughout the application to produce the highest possible quality game with the minimal download and device requirements. Creative design decisions delivered an application that works offline, on low-end hardware, downloads over low-speed networks and has a small 'footprint' on the device. The project also produced a multimedia online orientation for the software including detailed download instructions, gameplay instructions and explanations of all facets of application from access to scored measures to use of results in counseling sessions. The site also includes a quick start guide for new users, descriptions of each application and links for downloads. It also includes reports on data from individual users that can be accessed through three levels of security to protect user privacy. Research findings and results Four products were completed - three video games that include instructional resources, game play and assessment; and a website for technical support and resources. As a result of social distancing measures taken to reduce the spread of Coronavirus, the interest in educational software and telemedicine, including mental health applications, rose dramatically in 2020. Project staff presented at 22 conferences, and conducted five workshops on software design and use. Of these, 17 conference presentations and two workshops occurred in the current reporting period. Most of the conference presentations and webinars were conducted virtually in 2020. Initial usability testing revealed different priorities between staff and youth users with the former more focused on data collection and educational activities while the latter emphasized graphics and game play in recommended improvements. Both groups agreed on the need for an application that included positive as well as negative choices, teaching youth what to do in situations such as budgeting and job search as well as what not to do in high-risk situations. Usability was not a barrier in terms of understanding how to use the application or being able to play except in a small number of cases where users had significant learning or cognitive disabilities that resulted in reading below the third-grade level. Levels of usage by 1,365 participants was more than double the average for educational apps, but this is a modest accomplishment with only about one-third of educational licenses ever used. Initial results found 86% of software downloaded was played past the introductory level, however, only one-third of users played two or more levels. Modifications to the design of the second application were made based on these results, implementing the most-used aspects of the first application and adding social media simulations which did not require Internet connectivity, e.g., mock "Instagram-type" polls and text messages. The second application, with 112 users in the initial wave of data collection, had three times as many users completing at least two levels, five times the average number of levels completed and five times the number of users completing all modules. A study of retention and attendance of 47 participants in grades five through twelve had a course completion rate of 96% with average daily attendance over 90%. These figures included a class at a tribal school, which, according to administrators, outperformed the school average in daily attendance over a five-week period.

Publications

  • Type: Conference Papers and Presentations Status: Published Year Published: 2020 Citation: Burns Ortiz, M. (2020). Going Global: Adapting High-Quality EdTech Games for Low-Resource Settings. mEducation Alliance.
  • Type: Conference Papers and Presentations Status: Published Year Published: 2020 Citation: Burns Ortiz, M. (2020). You Are Here: Using Game Design to Empower Underserved Youth. Games for Change Summit.
  • Type: Conference Papers and Presentations Status: Published Year Published: 2020 Citation: Burns Ortiz, M. (2020). Learning Without Hovering: How to Excite, Empower and Educate National Social Studies UnConference.
  • Type: Conference Papers and Presentations Status: Published Year Published: 2019 Citation: Burns Ortiz, M. , De Mars, A. & Mennem, A. (2019). Tradition Meets Tech: Teaching Native Youth through Video Game Design Principles. National Indian Education Association Annual Conference, Minneapolis, MN
  • Type: Conference Papers and Presentations Status: Published Year Published: 2019 Citation: Burns Ortiz, M. (2019). When Solving a Small Problem Can Have a Big Impact: How Developing Games for U.S. Tribal Communities Set Us Up for Global Expansion" Games for Change Summit, New York, NY
  • Type: Conference Papers and Presentations Status: Published Year Published: 2019 Citation: Burns Ortiz, M. (2019). Learning Language, Culture and Math Through Video Games. Standing Rock Education Consortium, Fort Yates, ND
  • Type: Conference Papers and Presentations Status: Published Year Published: 2020 Citation: De Mars, A. (2020). Motivating and educating with educational software in the new normal. Presentation at the South Dakota Indian Education Summit.
  • Type: Conference Papers and Presentations Status: Published Year Published: 2020 Citation: Burns Ortiz, M. (2020). Parental Un-involvement: Designing Lessons that Students Can Do with Minimal Home Help,21st Century Math: Engaging Online Students in Multi-Sensory Learning. National Mathematics Unconference
  • Type: Conference Papers and Presentations Status: Published Year Published: 2019 Citation: Burns Ortiz, M. (2019). Helping Students Become Producers of Technology. Ignite Learning Summit; August 13; Fergus Falls, MN
  • Type: Conference Papers and Presentations Status: Published Year Published: 2019 Citation: Burns Ortiz, M. , De Mars, A. & Mennem, A. (2019). Tradition Meets Tech: Teaching Native Youth Using Educational Video Games and through Game Design Principles. General Assembly Session. Native American Grant School Association Annual Conference, Las Vegas,NV
  • Type: Conference Papers and Presentations Status: Published Year Published: 2020 Citation: De Mars, A., Burns Ortiz, M., Mennem, A. , Taken Alive, J. & Jensen, J. (2020). Reducing, not introducing, barriers to success in STEM education. Presentation for the National Indian Education Association
  • Type: Conference Papers and Presentations Status: Published Year Published: 2019 Citation: De Mars, A. (2019). Cultural games  math and life choices. Presentation at the Higher Education Resources conference. Bismarck, ND.
  • Type: Conference Papers and Presentations Status: Published Year Published: 2019 Citation: De Mars, A. (2019). Using culture to teach mathematics and life skills. Presentation at the South Dakota Indian Education Summit. Pierre, SD.
  • Type: Conference Papers and Presentations Status: Published Year Published: 2019 Citation: Mennem, A. & De Mars, A. (2019). Developing Educational Technology with Native Youth. Presentation at the annual meeting of the National Indian Education Association, Minneapolis,MN.
  • Type: Conference Papers and Presentations Status: Published Year Published: 2020 Citation: Burns Ortiz, M. (2020). Game Design as a Framework for Indigenous Education. W�okiye Project Teacher Summit. Virtual. August 2020.
  • Type: Conference Papers and Presentations Status: Published Year Published: 2020 Citation: Burns Ortiz, M. (2020) Level Up: Engaging Students Using Video Games and Game Design Principles. South Dakota Indian Education Summit. Virtual. August 2020.
  • Type: Websites Status: Published Year Published: 2020 Citation: Demars, A. (2020). Crossroads: Games of Choices https://strongmindstrongbody.org/crossroads-games-of-choices/
  • Type: Websites Status: Published Year Published: 2019 Citation: Demars, A. (2019). Crossroads: Games of Choices https://www.thejuliagroup.com/crossroads/


Progress 09/01/18 to 08/31/19

Outputs
Target Audience:The primary target audience is rural Native American youth who have a family member addicted to alcohol or drugs. Due to the location of American Indian reservations in rural communities, Native American youth are far more likely to reside in rural areas than the national average. This primary target audience is also predominantly socio-economically disadvantaged. Secondary audiences are: researchers and policy makers providing services to rural Native American youth who have a family member addicted to alcohol or drugs, and researchers and policy makers providing services to other rural and non-rural youth in families where a family member is addicted to alcohol or drugs. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided?A two-week game design workshop was presented at the Spirit Lake Dakota Nation in North Dakota, in conjunction with usability testing for the application. A half-day workshop was presented at the Standing Rock Reservation, with staff from North and South Dakota, on using games to teach at-risk youth. A half-day workshop was presented in Fergus Falls, Minnesota on educational software for rural communities. A presentation on transition from school to work for youth on the Spirit Lake Nation was given in Albuquerque, New Mexico. A presentation on transition from school to work for youth on the Spirit Lake Nation was given at the National Indian Education Association conference in Hartford, Connecticut. A presentation on technology for social impact was presented at the UnidosUS conference in San Diego, CA. How have the results been disseminated to communities of interest?Results in year one have been disseminated primarily through sixconference papersand three workshops. In addition, the company has a mailing list of 845 subscribers which was sent highlights from the interim report. What do you plan to do during the next reporting period to accomplish the goals?In year two, we will complete the support website and continue to expand interactive journal activities. We will conduct a one-year multi-site analysis of impact on tribal youth recruitment and retention in programs serving rural communities. We will present results at four conferences, conduct three workshops in rural communities and submit two articles to peer-reviewed journals.

Impacts
What was accomplished under these goals? Nineteen tasks were proposed to address these six objectives. All tasks are either completed or in progress. Major accomplishments include the completion of 17 levels (exceeding the objective), completing 18 interactive journal prompts, including the option for voice or text input. In-app automated data collection and scoring programs have been completed for five mental health measures. Modifications are designed in throughout the application to produce the highest possible quality game with the minimal download and device requirements. Creative design decisions delivered an application that works offline, on low-end hardware, downloads over low-speed networks and has a small 'footprint' on the device. Three products were completed - three video games that include instructional resources, game play and assessment. One product, a website for technical support and resources is under development and will be completed in year two. Project staff presented at six papers at five conferences and conducted three workshops on software design and use. Six additional presentations on research results have been accepted for presentation at three conferences, to occur in the first quarter of year two.

Publications

  • Type: Journal Articles Status: Published Year Published: 2018 Citation: De Mars, A. & Longie, E. S. (2018). The value of perseverance: Using Dakota culture to teach mathematics. Transmotion, 4 (1), 113-131.
  • Type: Conference Papers and Presentations Status: Other Year Published: 2018 Citation: De Mars, A. & Longie, M. J. (2018). Transition from school to work: How tribal vocational rehabilitation programs can help youth with disabilities succeed in their post-secondary goals. Paper presented at the EPICS Native American Special Educational conference, Albuquerque, New Mexico.
  • Type: Conference Papers and Presentations Status: Published Year Published: 2018 Citation: De Mars, A. & Longie, M. J. (2018). Transition from School to Work on the Spirit Lake Nation. Paper presented at the annual meeting of the National Indian Education Association, Hartford, CT.
  • Type: Conference Papers and Presentations Status: Published Year Published: 2019 Citation: De Mars, A. (2019). Thirteen Statistics Every Biostatistician Should Know. Proceedings of SAS Global Forum. https://www.sas.com/content/dam/SAS/support/en/sas-global-forum-proceedings/2019/3561-2019.pdf
  • Type: Conference Papers and Presentations Status: Published Year Published: 2018 Citation: Digital Natives: Helping Tribal Youth Become Producers (Not Just Consumers) of Technology. Minnesota Indian Education Association annual conference. Hinkley, MN.
  • Type: Conference Papers and Presentations Status: Published Year Published: 2019 Citation: Burns Ortiz, M. (2019). Latinx entrepreneurs developing tech for good. UnidosUS annual conference, San Diego, CA.
  • Type: Conference Papers and Presentations Status: Published Year Published: 2018 Citation: Burns Ortiz, M. (2018). Tradition Meets Tech: Teaching Native Youth Using Educational Video Games and through Game Design Principles. Minnesota Indian Education Association annual conference. Hinkley, MN.