Progress 05/15/12 to 01/14/13
Outputs
Target Audience: Target Audience The primary target audience is communities with a high proportion of Native American students in the population. Due to the location of American Indian reservations in rural communities, this population is both far more likely to reside in rural areas than the national average. This primary target audience is also predominantly socio-economically disadvantaged. Secondary audiences are: researchers and policy makers providing services for Native Americans, and other rural and non-rural school districts, particularly those serving students who are socioeconomically disadvantaged or limited in English proficiency. Efforts Twenty-two supplemental resources for teachers were developed Pre-test and post-test data were collected and analyzed for two games. A workshop was held for teachers at two schools, to demonstrate the prototype and solicit feedback from subject matter experts. Changes/Problems: One unexpected problem identified was that due to the need to run a particular mandated assessment program, the Bureau of Indian Education schools in the area used Windows XP. Rather than develop for an operating system quickly becoming obsolete, the project purchased with company funds, 15 laptop computers to be used in development. While use of a vendor's testing software expedited development, it also limited customization. User testing, discussed under objective one, showed that students made fewer errors when links were large and highlighted. While manuals on best practices in web design decry flashing animation with comments such as, “Don’t use animated gifs unless your target market is ten-year-olds”, it so happens that our target market is ten-year-olds and large, flashing buttons that say “Click here to return” are effective. As the quiz questions, multiple choice options and program logic have all been written rewriting these in HTML and JavaScript can be accomplished quickly in Phase II. What opportunities for training and professional development has the project provided? One workshop was offered, attended by teachers from two reservation schools. One undergraduate student intern worked on this project in a technical role, funded with matching funds from The Julia Group. How have the results been disseminated to communities of interest? The primary methods of dissemination in Phase I have been through a workshop for teachers on the reservation and through the company blog, which received over 100,000 visitors per month during the grant period. Most dissemination efforts for this project are planned after the end of the grant period, as the data collection and analysis occurred, as scheduled, near the end of the grant. What do you plan to do during the next reporting period to accomplish the goals?
Nothing Reported
Impacts
What was accomplished under these goals?
All six of the project objectives were met. Technical progress and accomplishments are discussed by objective below. Objective 1: Write a program to collect on-line assessment data Data to address this objective were collected from three sources; in-game input forms, Surveymonkey quizzes and a browser cache file. Student answers to challenges were saved in an SQL database on the corporate server. Pre-test and post-test data and all quizzes to which students were routed following incorrect answers were created and stored in Surveymonkey. Student usage data were stored in a file on the laptop for monitoring use of local files and on the corporate server to record usage of on-line resources. Three separate types ofprograms were written to read each data type. Each program reads the data into separate variable fields, checks for out-of-range values, computes frequency distributions and produces tables of items by difficulty level. Programs for quizzes and tests also compute internal consistency reliability statistics. All programs have been documented and can easily be converted into macros to be called as needed for analyses of Phase II game challenges, quizzes and usage data analyses. Objective 2: Create prototype of an on-line application that integrates instruction in Dakota culture with instruction in mathematics Six game levels were created, centered on a 3-D virtual world created with the Unity 3-D Integrated Development Environment (IDE). Each level includes mathematics challenges, e.g., how many pits must Hoksinato jump over to get to the lake. Students provide input using an HTML form. JavaScript is used to provide prompts and hints, determine whether the answer is correct and the number of attempts. Student answers and the number of attempts are passed via PHP to an SQL server at the corporate site. STAGE ONE: Three levels were created, tested and revised. During development, two days of user-testing on-site were held to identify issues with integration with the school system and other usability problems. User input was collected both electronically and by our site coordinator. Based on this feedback, the levels were revised. STAGE TWO: After three weeks of user testing, all four levels were revised again. To maximize the proportion of time spent on the challenge problems and related mathematics, barriers were added. It became apparent that many students required instruction in basic keyboarding and some students were further behind in grade level than had been anticipated. Thus, the introductory level was expanded to include both second-grade level material and detailed instructions on using the keyboard and browser, e.g., to click on tabs at the top of a page to see a new page. STAGE THREE: After this second revision, an additional three weeks of testing of the prototype occurred using all six levels. The game levels are organized hierarchically, by grade level, with problems from standards for second grade at level 1, third- and fourth-grade at levels 2 and 3, problems from fourth-grade standards at level 4, and fifth-grade at levels 5 and 6.As North Dakota, along with 44 other states, will soon be making a change to the Common Core Standards, with a heavier emphasis on number sense and operations in elementary grades, this is the area of the standards where the greater emphasis was placed in Phase I. Objective 3: Create supplemental materials for instructors including PowerPoint presentations and printable study guides. A draft version of a printable teacher’s manual was written and two on-line resource directories created, comprising of proprietary materials in teacher resource directory, and public resources, respectively. The teacher resource directory includes five types of resources; Practice problem presentations that show how to solve sample problems similar to those used in the game challenges, Instructional presentations that teach topics tested in the game, Game instructional resources, including one PowerPoint and several printable handouts, Copies of the pre-test and post-test used, Printable certificates to give to students as reinforcements for passing a level. Teacher resources were maintained in a password-protected site, enabling us to monitor which teachers accessed these resources for classroom use and which did not. All three of the fourth grade teachers downloaded the supplemental materials, while none of the three fifth-grade teachers did. Our on-site coordinator made extensive use of the public resources that we had curated, using at least one of these sites per day for every day of the last three weeks of testing. While the fourth-grade teachers were in the classroom when this usage occurred, and commented positively on the sites and the students’ reactions, they did not initiate usage of these resources outside of the planned intervention time. Objective 4: Write a program to collect on-line data on teachers’ usage of DLP resources. This objective was met. The program to collect data was written, however, the program was not used in Phase I. None of the fifth-grade teachers accessed the DLP resources. The students used the program outside of the regular classrooms, and although all of the teachers were offered a username and password for site access, none of the fifth-grade teachers requested one. Thus, for fifth-grade teachers, there was no usage to analyze. In contrast, all three of the fourth-grade teachers requested, and received, usernames and passwords. In this situation, on-line usage data did not prove to be an indicator of teacher usage of DLP resources. The site coordinator noted that the fourth-grade teachers worked collaboratively. One teacher would, for example, download a “cheat sheet” for the game that gave keyboarding instructions, and then print copies for all of the classes. Objective 5: Write programs that will serve as a basis for extension of the levels for additional lessons in Phase II and to additional tribal groups in Phase III. Extensibility was a consideration throughout the program design. All input forms for answers to the games math challenges call the same JavaScript file where the problem, correct answer, prompts and hints are all variables. All game input uses the same JSON object and PHP script. Thus, a new HTML file can be written, with artwork to reflect another reservation’s geography and a problem on a different mathematical topic, re-using the same scripts from Phase I for checking the correct answer, providing prompts and hints for student guidance. Game customization is simplified by having relative links to artwork, sound and video files in separate folders. To change the game from Dakota to another language, the sound directory can be replaced. Hundreds of game links to files that speak Dakota words will automatically link to files with the new language. Similarly, to change artwork, the appropriate directory can be replaced. Automatically, the links in the game to characters with dress and features representing one tribe will be replaced with characters for a new tribe. The programs written to analyze the data can be re-run to produce reports by school, grade or classroom for any site by simply adding one statement to filter the relevant criteria. Objective 6: Conduct an analysis of impact on student performance that will serve as a guide for Phase II design. A sample of fourth- and fifth-grade students was selected from two reservation schools. Pre-test and post-test data were collected to measure impact on mathematics achievement. Usage data were collected to enable comparison of students’ time on task with improvement.
Publications
|