Source: The Julia Group submitted to
DAKOTA LEARNING: COMPUTER-ASSISTED MATHEMATICS INSTRUCTION IN A CULTURAL CONTEXT
Sponsoring Institution
National Institute of Food and Agriculture
Project Status
TERMINATED
Funding Source
Reporting Frequency
Annual
Accession No.
0228732
Grant No.
2012-33610-19471
Project No.
CALW-2012-00085
Proposal No.
2012-00085
Multistate No.
(N/A)
Program Code
8.6
Project Start Date
May 15, 2012
Project End Date
Jan 14, 2013
Grant Year
2012
Project Director
DEMARS, A.
Recipient Organization
The Julia Group
2111 7th St Number 8
Santa Monica,CA 90405
Performing Department
(N/A)
Non Technical Summary
Mathematics achievement of Native American students is the lowest of all racial and ethnic groups, a disadvantage that is evident by the fourth grade. Students living on American Indian reservations, which are located in rural persistent poverty counties, perform even below the national average for Native Americans. Dakota Learning Project (DLP) applies research in mathematics education and computer gaming to bring to commercialization a product that addresses this performance gap. In Phase I we develop and test an educational program that integrates mathematics with the teaching of Dakota culture for students in fourth, fifth and sixth grades. DLP is centered on a computer game interface where a Native American avatar guides the student through collection of assessment data and completion of increasingly difficult problems in computation, data analysis and algebra concepts. Immediate feedback is provided to students, with reinforcement in the form of prizes earned for correct answers. The program analyzes incorrect answers and routes the student to appropriate instructional content based on the type of error identified. Instructional methods offer options of on-line games, quizzes, videos, animation and virtual manipulatives. Teachers receive daily emailed reports on individual student and class performance, with links to recommended on-line examples, class handouts and PowerPoint presentations. In Phase I we compare the pre- and post-test scores on the North Dakota State Assessment Exam for mathematics of students from two reservation schools using the same mathematics curriculum, with one supplemented with DLP. The two schools are located less than 25 miles apart and are closely matched on ethnicity, class size, average achievement and income.Phase II will extend this research to modules covering all state mathematics standards for third through sixth grades, and track student progress longitudinally to document a long-term effect.
Animal Health Component
(N/A)
Research Effort Categories
Basic
10%
Applied
40%
Developmental
50%
Classification

Knowledge Area (KA)Subject of Investigation (SOI)Field of Science (FOS)Percent
9017310209010%
9036010208060%
9036010302030%
Goals / Objectives
The goal of Dakota Learning is to develop and test a pilot program that applies educational technology to reduce the deficits of rural Native American children in mathematics. Technical Objectives: 1. Write a program to collect on-line assessment data for evaluation of students' progress in mathematics and usage of DLP resources. 2. Create prototype of an on-line application that integrates instruction in Dakota culture with instruction in mathematics standards for grades four, five and six. 3. Create supplemental materials for instructors. 4. Write a program to collect on-line data on teachers' usage of DLP resources. 5. Write programs that will serve as a basis for extension of the modules for additional lessons in Phase II and to additional tribal groups in Phase III. 6. Conduct an analysis of impact on student performance that will serve as a guide for Phase II design. Outputs include an assessment of students pre- and post-interventions, a teacher's guide and a web-based application written in javascript.
Project Methods
In Phase I we will develop and test an educational program that integrates mathematics with the teaching of Dakota culture for students in fourth, fifth and sixth grades. DLP is centered on a computer game interface where a Native American avatar guides the student through collection of assessment data and completion of increasingly difficult problems in computation, data analysis and algebra concepts. Immediate feedback is provided to students, with reinforcement in the form of prizes earned for correct answers. The program analyzes incorrect answers and routes the student to appropriate instructional content based on the type of error identified. Instructional methods offer options of on-line games, quizzes, videos, animation and virtual manipulatives. Teachers receive daily emailed reports on individual student and class performance, with links to recommended on-line examples, class handouts and PowerPoint presentations. In Phase I we compare the pre- and post-test scores on the North Dakota State Assessment Exam for mathematics of students from two reservation schools using the same mathematics curriculum, with one supplemented with DLP. The two schools are located less than 25 miles apart and are closely matched on ethnicity, class size, average achievement and income. Any differences that occur in teacher experience, homework hours or other school or classroom variables will be controlled statistically through Analysis of Covariance.

Progress 05/15/12 to 01/14/13

Outputs
Target Audience: Target Audience The primary target audience is communities with a high proportion of Native American students in the population. Due to the location of American Indian reservations in rural communities, this population is both far more likely to reside in rural areas than the national average. This primary target audience is also predominantly socio-economically disadvantaged. Secondary audiences are: researchers and policy makers providing services for Native Americans, and other rural and non-rural school districts, particularly those serving students who are socioeconomically disadvantaged or limited in English proficiency. Efforts Twenty-two supplemental resources for teachers were developed Pre-test and post-test data were collected and analyzed for two games. A workshop was held for teachers at two schools, to demonstrate the prototype and solicit feedback from subject matter experts. Changes/Problems: One unexpected problem identified was that due to the need to run a particular mandated assessment program, the Bureau of Indian Education schools in the area used Windows XP. Rather than develop for an operating system quickly becoming obsolete, the project purchased with company funds, 15 laptop computers to be used in development. While use of a vendor's testing software expedited development, it also limited customization. User testing, discussed under objective one, showed that students made fewer errors when links were large and highlighted. While manuals on best practices in web design decry flashing animation with comments such as, “Don’t use animated gifs unless your target market is ten-year-olds”, it so happens that our target market is ten-year-olds and large, flashing buttons that say “Click here to return” are effective. As the quiz questions, multiple choice options and program logic have all been written rewriting these in HTML and JavaScript can be accomplished quickly in Phase II. What opportunities for training and professional development has the project provided? One workshop was offered, attended by teachers from two reservation schools. One undergraduate student intern worked on this project in a technical role, funded with matching funds from The Julia Group. How have the results been disseminated to communities of interest? The primary methods of dissemination in Phase I have been through a workshop for teachers on the reservation and through the company blog, which received over 100,000 visitors per month during the grant period. Most dissemination efforts for this project are planned after the end of the grant period, as the data collection and analysis occurred, as scheduled, near the end of the grant. What do you plan to do during the next reporting period to accomplish the goals? Nothing Reported

Impacts
What was accomplished under these goals? All six of the project objectives were met. Technical progress and accomplishments are discussed by objective below. Objective 1: Write a program to collect on-line assessment data Data to address this objective were collected from three sources; in-game input forms, Surveymonkey quizzes and a browser cache file. Student answers to challenges were saved in an SQL database on the corporate server. Pre-test and post-test data and all quizzes to which students were routed following incorrect answers were created and stored in Surveymonkey. Student usage data were stored in a file on the laptop for monitoring use of local files and on the corporate server to record usage of on-line resources. Three separate types ofprograms were written to read each data type. Each program reads the data into separate variable fields, checks for out-of-range values, computes frequency distributions and produces tables of items by difficulty level. Programs for quizzes and tests also compute internal consistency reliability statistics. All programs have been documented and can easily be converted into macros to be called as needed for analyses of Phase II game challenges, quizzes and usage data analyses. Objective 2: Create prototype of an on-line application that integrates instruction in Dakota culture with instruction in mathematics Six game levels were created, centered on a 3-D virtual world created with the Unity 3-D Integrated Development Environment (IDE). Each level includes mathematics challenges, e.g., how many pits must Hoksinato jump over to get to the lake. Students provide input using an HTML form. JavaScript is used to provide prompts and hints, determine whether the answer is correct and the number of attempts. Student answers and the number of attempts are passed via PHP to an SQL server at the corporate site. STAGE ONE: Three levels were created, tested and revised. During development, two days of user-testing on-site were held to identify issues with integration with the school system and other usability problems. User input was collected both electronically and by our site coordinator. Based on this feedback, the levels were revised. STAGE TWO: After three weeks of user testing, all four levels were revised again. To maximize the proportion of time spent on the challenge problems and related mathematics, barriers were added. It became apparent that many students required instruction in basic keyboarding and some students were further behind in grade level than had been anticipated. Thus, the introductory level was expanded to include both second-grade level material and detailed instructions on using the keyboard and browser, e.g., to click on tabs at the top of a page to see a new page. STAGE THREE: After this second revision, an additional three weeks of testing of the prototype occurred using all six levels. The game levels are organized hierarchically, by grade level, with problems from standards for second grade at level 1, third- and fourth-grade at levels 2 and 3, problems from fourth-grade standards at level 4, and fifth-grade at levels 5 and 6.As North Dakota, along with 44 other states, will soon be making a change to the Common Core Standards, with a heavier emphasis on number sense and operations in elementary grades, this is the area of the standards where the greater emphasis was placed in Phase I. Objective 3: Create supplemental materials for instructors including PowerPoint presentations and printable study guides. A draft version of a printable teacher’s manual was written and two on-line resource directories created, comprising of proprietary materials in teacher resource directory, and public resources, respectively. The teacher resource directory includes five types of resources; Practice problem presentations that show how to solve sample problems similar to those used in the game challenges, Instructional presentations that teach topics tested in the game, Game instructional resources, including one PowerPoint and several printable handouts, Copies of the pre-test and post-test used, Printable certificates to give to students as reinforcements for passing a level. Teacher resources were maintained in a password-protected site, enabling us to monitor which teachers accessed these resources for classroom use and which did not. All three of the fourth grade teachers downloaded the supplemental materials, while none of the three fifth-grade teachers did. Our on-site coordinator made extensive use of the public resources that we had curated, using at least one of these sites per day for every day of the last three weeks of testing. While the fourth-grade teachers were in the classroom when this usage occurred, and commented positively on the sites and the students’ reactions, they did not initiate usage of these resources outside of the planned intervention time. Objective 4: Write a program to collect on-line data on teachers’ usage of DLP resources. This objective was met. The program to collect data was written, however, the program was not used in Phase I. None of the fifth-grade teachers accessed the DLP resources. The students used the program outside of the regular classrooms, and although all of the teachers were offered a username and password for site access, none of the fifth-grade teachers requested one. Thus, for fifth-grade teachers, there was no usage to analyze. In contrast, all three of the fourth-grade teachers requested, and received, usernames and passwords. In this situation, on-line usage data did not prove to be an indicator of teacher usage of DLP resources. The site coordinator noted that the fourth-grade teachers worked collaboratively. One teacher would, for example, download a “cheat sheet” for the game that gave keyboarding instructions, and then print copies for all of the classes. Objective 5: Write programs that will serve as a basis for extension of the levels for additional lessons in Phase II and to additional tribal groups in Phase III. Extensibility was a consideration throughout the program design. All input forms for answers to the games math challenges call the same JavaScript file where the problem, correct answer, prompts and hints are all variables. All game input uses the same JSON object and PHP script. Thus, a new HTML file can be written, with artwork to reflect another reservation’s geography and a problem on a different mathematical topic, re-using the same scripts from Phase I for checking the correct answer, providing prompts and hints for student guidance. Game customization is simplified by having relative links to artwork, sound and video files in separate folders. To change the game from Dakota to another language, the sound directory can be replaced. Hundreds of game links to files that speak Dakota words will automatically link to files with the new language. Similarly, to change artwork, the appropriate directory can be replaced. Automatically, the links in the game to characters with dress and features representing one tribe will be replaced with characters for a new tribe. The programs written to analyze the data can be re-run to produce reports by school, grade or classroom for any site by simply adding one statement to filter the relevant criteria. Objective 6: Conduct an analysis of impact on student performance that will serve as a guide for Phase II design. A sample of fourth- and fifth-grade students was selected from two reservation schools. Pre-test and post-test data were collected to measure impact on mathematics achievement. Usage data were collected to enable comparison of students’ time on task with improvement.

Publications