Source: The Julia Group submitted to
Sponsoring Institution
National Institute of Food and Agriculture
Project Status
Funding Source
Reporting Frequency
Accession No.
Grant No.
Project No.
Proposal No.
Multistate No.
Program Code
Project Start Date
Aug 15, 2016
Project End Date
Apr 14, 2017
Grant Year
Project Director
De Mars, A.
Recipient Organization
The Julia Group
2111 7th St Number 8
Santa Monica,CA 90405
Performing Department
Non Technical Summary
Methamphetamine use is a scourge of rural communities - and the problem is getting worse. The combination of declining local economies, secluded sites for concealing illegal activities and use of chemicals common in farming result in disproportionately more meth labs located in rural than in urban communities. Not only is meth use greater in rural communities, but barriers to effective treatment and prevention are greater as well. Two barriers exist to reaching youth in rural communities: geographic barriers impede frequent contact between youth and service providers and it is difficult to maintain the interest of youth.Drawing on research from two distinct areas: substance abuse intervention and educational video games, Crossroads: A Game of Choices is an interactive video game that teaches knowledge and skills to improve decision-making by Native American youth affected by methamphetamine use in rural communities, intervening in the intergenerational cycle of addiction.Phase I is focused on prototype development, usability testing, creation of customized assessments and educational resources. Crossroads combines cutting-edge game design and graphics with effective therapeutic methods -- Community Reinforcement and Family Therapy and Motivational Interviewing (MI) - that are adaptable to computer-assisted learning. The result is a product that will engage youth at risk of meth use/abuse and teach/model healthy decision-making. Data collection will be within the context of the game, integrated into the storyline. Positive choices will be rewarded with gameplay, while negative choices will route users to software that allows users to participate in MI remotely, in a media format that accommodates for any below grade-level communication skills. Data is collected continuously throughout the game and used for game improvement, e.g., to raise the rate of module completion. Counselors receive MI feedback from users to incorporate in support and services offered to game users. Users access Crossroads on the devices they are most comfortable using, remotely and at their convenience.In Phase I, we will focus on game and educational design with the emphasis on product creation and refinement based on usability testing feedback. We will conduct two rounds of testing with tribal youth ages 11-18 on the Fort Berthold reservation in North Dakota; an initial 15-person on-site usability testing, followed by a 10-person off-site usability study.The project offers significant short-term benefit applied to methamphetamine abuse, and vastly greater long-term benefit as a successful Phase I project can be extended to other types of substance abuse and other risk behaviors under a Phase II.
Animal Health Component
Research Effort Categories

Knowledge Area (KA)Subject of Investigation (SOI)Field of Science (FOS)Percent
Goals / Objectives
The major goal of this project is to draw on research from two distinct areas: substance abuse intervention and educational video games, to create a prototype of an interactive video game that teaches knowledge and skills to improve decision-making by Native American youthaffected by methamphetamine use in rural communities, intervening in the intergenerational cycle of addiction.The project has three technical objectives for Phase I.Develop three playable levels in a virtual world that integrate video scenarios, decisions, game play and instructional videos or web pagesCreate interactive journals modeled on motivational interviewing using audio and video,Develop in-game assessment for substance abuse, risk factors and knowledge.Conduct usability testing on the game developed with adolescents in families affected by methamphetamine abuse
Project Methods
1.1 Create final game design A Game Design Document will be finalized, integrating all sound and graphic assets required, character development, storyline, scenarios, assessment points, choices, educational resources and game play. Data collection will be incorporated in the game in a natural fashion. For example, on Level 1, characters within the game meet the players outside at night. After introducing themselves, since they cannot 'see' the player well, they ask basic demographic questions about age, gender.1.2 Create three playable game levels: In an adventure game format, users select an avatar that will be posed questions throughout the game. Navigating within the game virtual world, a player triggers a video that shows a common scenario in homes affected by methamphetamine addiction, e.g., arriving home alone to a house with no food and inadequate heat. The player is guided through possible choices, with the assistance of motivational interviewing (MI) techniques. After a decision is made, an instructional resource plays. If the answer is correct, the choice is reinforced through the opportunity to play the game. For example, choosing to call grandmother results in an activity driving a tractor across fields strewn with obstacles to arrive at her house before time runs out.Task 1.3 Using local Native American actors, COL staff will create three videos of common scenarios: arriving home alone and finding drug paraphernalia, witnessing domestic violence and drug parties occurring in homes with minor children. Research has shown that student engagement is enhanced when users perceive features of a learning environment to be visually realistic . Likewise, learning in a virtual world, such as in Crossroads, that resembles the player's own reality of belonging to a Native American culture may better sustain his or her interest and persevere with the topic.Task 1.4 COL and 7G staff will create 3 in-game decision points, each with one correct and two incorrect choices. Each decision will route to one of three educational resources, explaining either why the decision was correct or incorrect. If a dysfunctional choice is made, the player is returned to the MI activity to reinforce what was learned, and then can select a different decision.Task 1.5 Create 9 in-game educational resources As data have repeatedly shown the difficulty methamphetamine users have with taking accountability for the consequences of their actions , both correct and incorrect decisions will be followed by educational resources emphasizing the connection between decision and results. For example, going to the home of a neighbor known to have a meth lab could result in violence, as the player's appearance is unexpected.Task 2.1 Create motivational interview prompts/guides - To prevent the 'homework' look of many training applications, prompts will be written by COL staff and recorded in brief videos to give the application a 'YouTube' or 'Snap Chat' feel. Characters will be a mix of COL staff, youth and animation. Prior research has found games with variation in format to have high engagement and completion rate by youth, in contrast to the low completion rates of single format games (Neil et al. 2009).Task 2.2 An audio input feature will allow users the option of voice recordings of their answers to MI prompts to be shared with counselors. Given that 37% of Native American students are below basic level in academics by the eighth grade (NCES, 2015) an intervention that relies primarily on written correspondence between counselor and patient will have limited access. Audio input will be incorporated both to accommodate user needs and to assess impact on persistence and hours of use.Task 3.1 Data collection forms will be written in PHP and data stored in a MySQL database. A reliable assessment requires complete data on a relatively large number of items. Data collection forms appear throughout the game in the form of 'conversations' with an animated avatar chosen by the player. Data collection forms appear at logical points so as to not disrupt the flow of the game.Task 3.2 Write program for data analysis Writing the programs before usability testing insures all necessary variables are included in data collection forms. Game usage data is collected continuously. Analyses include descriptive statistics for frequency and duration of game play session and number of modules completed and inferential statistical analysis of relationships between player demographics and persistence. Diagnostic data are collected via assessment data collected through interactions with the avatar. Descriptive statistics will be computed and psychometric analyses performed. Given the small sample size, analyses must be viewed as preliminary and a basis for replication and extension in Phase II.Task 4.1 Recruit Sample The target population is youth aged 11-17 with a household member who is abusing alcohol or drugs, with a particular emphasis on methamphetamine. In Phase I, 30 subjects will be recruited from families of participants in the COL substance abuse treatment and referrals from social service agencies.Task 4.2 On-site usability testing will first be conducted initially with 5 youth participants observed individually for a 45-minute session to identify major issues, e.g., incompatible with older operating system, RAM available results in low performance, Internet connection is inadequate. Testing on-site in a remote community is a key activity. For example, previous research found unacceptable downtime for users; this issue was only regularly observable in remote locations. Second, we will conduct 3 weeks of on-site tests. Youth will play two 40-minute sessions per week. A maximum variation sample of 15 participants will include five participants each in age groups 11-12,13-14 and 15-17, with both genders in each age group. Researchers will observe two selected participants per session for twenty minutes, followed by individual interviews. Observations will be recorded electronically using a usability survey developed by the researchers. The third week, the monitor will select 4 players to individually "think-aloud" as they work through the game. With 9 hours of observations, 15 structured interviews, 4 think-aloud sessions plus quantitative data collected electronically for 60 hours, adequate data will be gathered to identify performance issues under optimal settings. Nielsen and Landauer (1993) found that 85% of usability problems can be found with 5 users and recommended breaking samples into multiple, small, iterative studies of 5 users. With 3 weeks of observation across age groups, observers will be able to assess usability issues early and intermediate term.Task 4.3 After a 3-week break, during which game levels are revised, conduct off-site testing for another 3-week period. An additional 10 users will be recruited both to assess usability for naïve users in remote off-site locations, and to account for probable attrition. For the off-site condition, users will be provided refurbished iPhones with a limited data plan. Data will be collected electronically on frequency and duration of game play sessions, modules completed and accesses of help screens. Project staff will record number, content, duration and resolution of phone and email contacts for customer support. COL staff members will visit 5 new users and 5 experienced users at home to observe usage and individually 'think aloud' as they play the game.Task 4.4 Conduct data analysis Data analysis programs will run weekly to address any problems with data early with minimal loss of data quality. Qualitative data will be summarized in at least 5 user case studies, identifying typical and 'edge' users, e.g., users with zero or extremely high level of experience playing games, users with special needs.

Progress 08/15/16 to 04/14/17

Target Audience:The primary target audience is children and youth in rural communities who have a family member abusing methamphetamines. Another primary audience is staff members of education, treatment and social service organizations providing treatment and prevention services to these families. With Native Americans both more likely to reside in remote communities and at higher risk of methamphetamine abuse, the target audience in Phase I has been primarily Native Americans and those providing services to Native Americans. Secondary audiences are: researchers and policy makers providing services for children, youth and families affected by methamphetamine addiction and other substance abuse, particularly those conducting research or making policy in rural communities. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided?The researchers worked with 11 staff members, from cooperating programs on two reservations and one university. Three were part of a design team developing in-game activities, scenarios and assessment. All eleven staff members participated in software testing. How have the results been disseminated to communities of interest?Research design and preliminary data were presented at the 2016 National Indian Education Association conference and also on the 7 Generation Games website and to 750 educators and social service professionals on the company mailing list. These methods were chosen over peer-reviewed journals both because of the small sample size of pilot data and also because these media are used consumed more by the communities of interest. What do you plan to do during the next reporting period to accomplish the goals? Nothing Reported

What was accomplished under these goals? All four of the project objectives were met. Prior to starting the project Institutional Review Board approval was obtained through National University where the Principal Investigator is an adjunct faculty member both in the Departments of Public Health and Applied Engineering. IRB was key to this project since it deals with two potentially vulnerable populations - minors and individuals in treatment for substance abuse. 1. Develop three playable levels in a virtual world that integrate video scenarios, decisions, game play and instructional videos or web pages This objective was exceeded. Three playable levels were created. In addition, as a result of tester feedback conducted under objective 4, an additional activity was added of a virtual store, where players can purchase items with currency earned in the game. Also in response to user feedback, customization options were added to the levels, a feature that will be expanded in Phase II. 2. Create interactive journals modeled on motivational interviewing using audio and video This objective was met. Motivational interview prompts and guides were completed in consultation with addiction counselors and a psychologist from the Circle of Life program on the Fort Berthold Reservation. Their participation in this project was funded as an in-kind match from the Three Affiliated Tribes. An audio input feature was created and added to the interactive journal activity. After the initial screens of clicking and drag-and-drop activities, the player is presented with a series of prompts via "text messages" which can be answered via text message or by holding a button to record and a second button to send. Video input was tested but not included in the final prototype because the lower bandwidth in the remote areas where the application was tested made video upload impractical as connections were dropped, sounds was garbled or data size was so large as to be unacceptable for users on minimum data plans. 3. Develop in-game assessment for substance abuse, risk factors and knowledge. To maintain player interest, multiple types of forms were used. Choices presented within the three game scenarios assessed players' knowledge of appropriate decisions in risk situations. As a prototype for standard assessment forms to be included in the final commercial product, the Center for Epidemiologic Studies Depression Scale for Children (CESD-C) by Weissman et al. (1980) was integrated into the game in an activity where the player, now employed as a junior counselor, completes the form in an effort to convince a client , Jessie, that it's not too difficult. Upon completion, the player receives a 'salary' which can then be spent in the virtual store. The CES-DC is a twenty-item scale that has been used extensively for assessment of depressive symptoms in children and adolescents. Game usage data was collected continuously throughout game play. Programs were written for analyses to include descriptive statistics for frequency and duration of game play session and number of modules completed and inferential statistical analysis of relationships between player demographics and persistence. Diagnostic data were collected via assessment data collected through interactions with the avatar. Descriptive statistics were computed for assessment and demographic items. Psychometric properties were computed (internal consistency reliability, item variance, inter-item correlations) for the measures. Given the small sample size, these analyses must be viewed as preliminary and a basis for replication and extension in Phase II. Writing the programs before usability testing insured all necessary variables were included in data collection forms. 4. Conduct usability testing on the game developed with adolescents in families affected by methamphetamine abuse The target population was youth aged 11-19 with a household member who is abusing alcohol or drugs, with a particular emphasis on methamphetamine. During Phase I it was decided to extend the age maximum for youth recruited from the original plan of 17 years to 19 given the substantial number of youth aged 18 and 19 still in high school and living at home. Through contacts in the Spirit Lake Dakota Nation and Fort Berthold Reservation, it was possible to expand the recruitment from the initial plan of a single site on one reservation to include two stages of sampling. First, five testers at one site played the earliest prototype to identify any major problems with connectivity, instructions or game play. Subsequently, subjects were recruited on two reservations, in two small rural communities (population: 75 and 341) adjacent to the reservations and from youth attending school in a university town near a reservation. After the planned 30 youth had been tested, based on comments recommending inclusion of younger players, 5 additional subjects aged 9 and 10 were recruited. Subjects were recruited from families of participants in substance abuse treatment and referrals from social service agencies. Staff members posted flyers in their offices and distributed personally to family members of clients. The game was also tested with 11 staff members from rural substance abuse and social services programs serving youth who have families with methamphetamine addiction. Thus, while the original task targeted data collection from 30 subjects on one reservation, through extensive cooperation with programs it was possible to collect data from 51 subjects - 5 participants in an initial usability study, 11 staff members and 35 youth playing the completed game. Age data was collected during game play and is shown below for the 35 participants. One individual, age 21, with special needs and still in the school system, was also included in the sample. Usability testing with youth was conducted first on-site in a substance abuse treatment program with five subjects to identify any major problems. With each tester, the researcher observed game play and recorded field notes. At the end of user testing, a usability survey was completed in a semi-structured interview. As usability testing overlapped with development, it was possible to implement feedback from earlier testing and revise the software design. Following the first stage, usability testing was conducting on-site with the completed prototype with 19 subjects. These subjects were tested in three schools with after-school programs and at a university student support center in a larger town (population 46,000). Note that subjects were not university students but rather, children from the reservation living in the area temporarily. Usability testing was also conducted with 16 subjects testing off-site in small towns where youth resided, ranging in population from 75 to 2,363, on and adjacent to the reservation. Participants played the games at home, observed by a staff member from one of the cooperating programs on the reservation. Only one of the participants had a wi-fi connection at home. Two played on the wi-fi at a relative's home. The remaining 13 subjects played the game over the cellular network. Analysis of prototype data found no usability issues with youth testers. Some, but not all, of the older staff member testers had difficulty with game play, in particular, dragging smaller items in the game and in moving their avatar across the basketball court.


  • Type: Conference Papers and Presentations Status: Published Year Published: 2016 Citation: De Mars, A. & Gillette, B. (2016). Customized Video Games as Effective Education for Tribal Youth. Paper presented at the annual meeting of the National Indian Education Association, Reno, NV.