Performing Department
(N/A)
Non Technical Summary
Preparing well-rounded veterinarians equipped with the required skill sets is essential for our society as they play a critical role in the health of animals, environmental protection, food safety, and public health. Veterinary medicine curriculum requires a great harmony across courses for students to be able to skillfully create mental models of the knowledge and skills they gain during their education and graduate as practice-ready professionals. However, faculty rarely know in adequate detail what students have encountered in other courses. This results in lost opportunities to build on prior knowledge, and can also lead to student disengagement when there are excessive gaps or redundancies. The overall objective of this proposal is to bring innovation to veterinary curriculum analysis by utilizing artificial intelligence (AI) approaches, which will not only provide a much-needed platform for analyzing the actual curriculum as taught, but will do so in an efficient and cost-effective manner. To meet our overall objective, we will pursue two specific objectives: 1) develop an online searchable database of the veterinary curriculum, and 2) develop an online mapping tool for curriculum analysis. The deliverable for Objective 1 will be an online database of the veterinary medicine curriculum where users can search for any phrases and identify which course(s) mentions that specific phrase in all the relevant course materials. The deliverable for Objective 2 will be an online mapping tool which will utilize artificial intelligence approaches to categorize the content of the curriculum and map it against competencies and program objectives.
Animal Health Component
50%
Research Effort Categories
Basic
0%
Applied
50%
Developmental
50%
Goals / Objectives
The overall objectiveof this projectis to bring innovation to curriculum mapping by utilizing artificial intelligence (AI) approaches including machine learning and text mining, which will not only provide a much-needed platform for analyzing the actual curriculum as taught, but will do so in an efficient and cost-effective manner.Ourlong-term goalis to create an automated sustainable system that hosts an online searchable database of the veterinary medicine curriculum that uses real classroom data. Users will be able to search any phrases to check if that phrase is covered in any of the courses in the curriculum.To meet our overall objective, we will pursue two specific objectives:Develop an online searchable database of the veterinary curriculum.Develop an online mapping tool for curriculum analysis.
Project Methods
Methodology to reach Specific Objective # 1: Develop an online searchable database of the veterinary curriculum. The goal for the research under this specific objective is to utilize real classroom data to develop a searchable online database of the veterinary curriculum available for faculty, students, and curriculum administrators. Our working hypothesis, based on previous literature and our preliminary work, is that we can use the transcripts of the lecture recordings to create a database of lecture content that can be tied back to actual recordings. Users, then, can search for a keyword and see in which courses that keyword was mentioned, and access the related materials (e.g. lecture video, course syllabus). We will follow a human-centered design approach and involve our stakeholders in testing and refining the searchable database to develop a user-friendly and functional website. The rationale for this objective is that an online searchable database will make the curriculum more transparent and will enable students to be able to search information for concepts covered in courses across the curriculum, and refresh their knowledge base. Faculty and curriculum administrators, on the other hand, will be able to identify which courses cover similar content and how that content is presented, so they can make evidence-based assumptions about the prerequisite knowledge students bring to specific courses rather than trying to guess what they have learned previously.To achieve the first objective of this project, we will follow a human-centered design approach to design and development of the online searchable database of the veterinary medicine curriculum by focusing on the potential users and create an innovative solution that is tailor-made to suit their needs (Abras et al., 2004). This will entail an iterative process as displayed in Figure 2. We will start by retrieving all course-related materials, including lecture recordings, transcripts, course syllabi, and course information already available in digital form on the course management site. Next, we will create a quick prototype using only a limited number of course materials to test if the system works. Once we have a working prototype, we will ask for feedback from a small number of users (students and faculty) to test the website. Based on the feedback, we will keep iterating, testing, and refining until we have fine-tuned our searchable database. Finally, we will make the database available for use. To assess if the searchable database reaches its objective, we will collect user traffic data to determine the number of clicks, downloads, and access to resources. Additionally, we will survey students and faculty members to examine how they used the database, and whether it was helpful in enhancing their understanding of how various components of the curriculum fit together.The expected result of this specific objective is to bring in a complete four-year veterinary medicine curriculum to the fingertips of faculty members and students. Students will be able to visualize how concepts and principles build upon each other across courses, and use the tool to refresh their knowledge base in a time-efficient manner. Faculty members will be able to locate where the prerequisite knowledge is covered in the curriculum, and make informed decisions about improvements at the course and curriculum level. In turn, this will improve the communication between instructors and help them create a cohesive curriculum that successfully prepares students for an ever-changing veterinary practice. Additionally, because the curriculum will be easily accessible, curriculum assessment will be a continuous effort rather than a one-time exercise.Methodology to reach Specific Objective # 2: Develop an online mapping tool for curriculum analysis. The goal of the research under this specific objective is to develop, implement, and evaluate a natural language processing (NLP) system that will automatically map curricular materials onto program objectives and outcomes. A corpus--collection of texts--of the curriculum will be created using all the available digital curricular materials (lecture transcripts, course syllabi, and course information documents), which will be extracted from the searchable database developed in Specific Objective #1. In collaboration with three faculty members from ISU CVM (see support letters) as content experts, a taxonomy of domain concepts representing program objectives and outcomes will be developed, based on AMVA COE Standard 9 (Curriculum) and Standard 11 (Outcomes Assessment), as well as the Program Objectives adopted by ISU CVM. A representative sample of curricular corpus will be built using standard corpus-linguistic methods to ensure that the sample is balanced and representative of the general population. Portions of the corpus that are representative of disciplines which are related to a single domain concept can be used for model training directly. However, most disciplines represent an overlap of multiple domain concepts. Therefore, the corpus will be manually annotated for the identified domain concepts, and the inter-annotator reliability will be controlled to ensure high quality of annotation.From the annotated corpus, two types of representations of the texts will be derived: (1) lexical representations, with raw and stemmed n-grams and skip-grams as features; and (2) deep-learning-based semantic representations (e.g. using Bidirectional Encoder Representations from Transformers [Devlin et al., 2018]). These representations will be combined into an engineered set of features to be used as input to a feature-based, machine-learning classifier (e.g. a Support Vector Machine [Zhang et al., 2008]). Following a supervised machine-learning approach with the manually annotated corpus as a gold standard, the classifier will be trained to recognize, based on the engineered features set, segments within the text of curricular materials that correspond to particular domain concepts. Due to the inherent ambiguity of natural language, classification uncertainties (i.e. instances when a text segment can be interpreted as corresponding to several domain concepts at once) are expected. In these cases, the classifier will yield a confidence metric associated with each of the predicted concepts. Standard cross-validation procedures will be employed to assess the performance of the classifier on portions of the manually annotated gold-standard corpus that have not been employed in the training of the classifier. Performance metrics of precision, recall, and F-score will be used to guide the refinement of the classifier.It is expected that the baseline classifier may not deliver acceptable performance. If that is the case, to improve the classifier, we will experiment with several machine-learning algorithms, fine-tune the engineered feature sets and learning algorithm parameters, identify confusing domain concepts, if any, and clarify their definitions with content experts. This experimental work will continue until acceptable performance of the classifier is attained. We will then follow a user-focused participatory design approach to develop a way of visualizing the output of the classifier for end users of the final system (instructors and students of veterinary medicine) both in terms of individual lecture-level output, and in terms of the horizontal and vertical relationships across courses.