ETIPS

Abstract: This paper presents the first year of findings on an alternative delivery model for case methods, which has been designed to provide learners with a highly authentic, online learning environment. The cases develop and scaffold the cognitive processes needed by future leaders to diagnose and interpret problems, and make decisions about leading a school. Findings indicate a correlation between the contributions of the case experience to both decision making self efficacy and confidence. Students reported that cases were authentic and fostered learning of the decision making process. The cases offer opportunities to develop and practice decision making skills as part of a leadership preparation program that better bridges theory and practice.

Abstract: Case methods of instruction have been advocated as a signature pedagogy for the preparation of school leaders that would provide more authentic learning environments for developing future leaders' ability to apply theory into practice. Our field testing during the past year with a test-bed of faculty members using newly created cases shows (1) that case-based methods of instruction are a demanding pedagogy, and (2) that positive student experiences with cases mainly stem from their instructor's during- and after-case methods of instruction. These data suggest that discussion and feedback strategies are both key, but are not always utilized. Implications for design of case-based learning environments discussed.

Abstract: In this paper we investigate students' perspectives on the classroom implementation of online cases designed to provide pre-service administrators multiple opportunities to practice applying theory in their decision making within virtual yet realistic school settings. Data from this study indicate that several characteristics of effective learning environments were explicit in the case implementation and contributed to student learning and knowledge transfer. However, the case implementation could be improved by closer attention to the application of certain aspects of effective learning environments through the implementation of more value-added features of new technology. These findings offer insights for how the learning sciences can guide successful implementation of technology-based learning environments.

Abstract: Cases can not only provide opportunities to learn, but can also work as an effective assessment tool by providing a window into the critical thinking of the learners. The approach and findings to determine the construct validity of newly designed instructional cases as a performance assessment of administrators' decision-making expertise revealed expert in-service principals positive response evidence about the validity of the cases as an assessment of decision making skill and knowledge and that their experience in completing a case was an accurate simulation of their on-the-job decision making. The implications are discussed for instructional materials and professional development.

Abstract:The use of online cases as part of a traditional administration course were found to produce significant positive affective and cognitive outcomes for leadership preparation students. The findings from this study support the use of cases to develop students' decision making skill, and more generalized self-efficacy, confidence and certainty about the decision making process. Not only can decision making skill be taught but the resulting student learning can be measured. Results from this research have implications for the assessment of student decision making skills within preparation programs and future program development to support student understanding of this key leadership task.

Abstract: This paper presentation will report on the findings of pre-service administrators' ability to frame problems by assessing actual case responses in an online environment. Findings suggest that if programs are to equip administrators for the work they will do in schools, then it is essential that the programs address the work that is core to school leadership — that of problem-framing. Statistically significant improvement was found between the pre-intervention and post-intervention case with regard to students' problem-framing ability. The findings have implications for the assessment of student skills within leadership preparatory programs and utilizing case methods of instruction to develop students' ability to frame problems in an intentionally integrative and holistic manner.

Abstract: Case methods of instruction have been advocated as a signature pedagogy for leadership preparation. These nine cases studies of professors' implementation of online cases over multiple semesters illustrate instructional change and growth. These data suggest that while professional development and practice are key, more powerful levers like program assessment and national standards are needed to trump time constraints and other conditions constraining the utilization of best practices that support optimal student learning from cases.

Abstract: Survey data and focus group responses provide insights into students' experiences with and attitudes toward automated essay scoring software. This formative assessment tool was available for their use while completing short answer essays in online cases. Students' reactions to the scorer were both positive and negative, but were strong. They reported that a numeric score, even in conjunction with a rubric, was not sufficient information to know how to revise their work. The accuracy of the scoring software was low, which led some students to lose interest in revising their work after receiving a score prediction. The implication of a reliable automated essay scoring tool in use within network-based online learning environments is that it makes possible formative feedback, and presumably improved performances, on complex sets of skills.

Abstract: This study tests the impact of an automated essay scorer (AES) which provides formative feedback on essay drafts written as part of a series of online teacher education case studies. A total of 70 preservice teachers in four teacher education courses were assigned to complete two cases. Each student was randomly assigned to either a condition where the scorer was available (experimental condition) or a condition where the scorer was unavailable (control condition). Those students in the experimental condition submitted higher-quality final essays and conducted more efficient searches of the case than students in the control condition. Essay scores were positively associated with the number of drafts submitted to the scorer for formative feedback.

Abstract: A cognitive task analysis of twenty experienced in-service teachers to report the key knowledge, concepts and skills they draw upon to plan technology-integrated instruction. Data, collected through focus groups, show expert technology-integrating teachers' focus on curriculum goals, consider their students' technology skills and interests, and find models of the technology product expected of students important for guiding their pupils' understanding of their assignment. How much teachers plan for logistics and trouble-shooting depends on the level of access to technology in their school.

Abstract: The purpose of this research is to analyze preservice teachers' use of and reactions to an automated essay scorer as a means of formative feedback in an online, case-based learning environment. Data analyzed include pre- and post-semester surveys, a user log of students' actions within the cases, instructor-assigned scores on final essays, and in-depth interviews with four selected students. Analysis illustrates that extensive use of the scorer was associated with improved essays as measured by their final human-assigned scores. Study findings also suggest factors that must be considered in using technology to provide formative feedback including the design, nature, accuracy, and specificity of feedback, all of which were found to impact student performances and experiences. These outcomes boost confidence in the potential utility of instructional technology to aid in formative assessment within the ETIPS cases learning environment.

Abstract: A mix of faculty interviews, site observations, and student pre and post-surveys were used to examine the implementation of eTIP cases during the 2002-2003 academic year in teacher education courses. Case implementation varied in the extent to which the cases were discussed in class, instructors integrated the cases into other course topics, and instructors offered formative feedback on student work with cases. Hierarchical linear modeling was used with data from 243 students nested in 18 different course sections. Individual student ratings of case usefulness in learning about technology integration were predictive of gains in self-assessed technology integration skill. The impact of case ratings varied by class although the models employed did not correctly specify class effects.

Abstract: The work of 449 preservice teachers using the ETIPS cases in 2002-2003 is examined with a focus on how their information searches within the cases resemble those of experts (relevancy index scores) and how this is related to the extensiveness of their search (number of steps). In general, the relevancy index scores increased over multiple cases and were negatively related to number of steps taken in a case. Users made an average of 31 separate steps to access information in a case, an average which decreased over multiple cases. Individual search strategies for achieving high relevancy index scores included broad, sweeping searches of available information and shorter, focused searches of relevant information only.

Abstract: Characteristics of scores assigned to ETIPS case essays were explored. Two essay scoring rubrics were tested in 2002-2003 with 133 preservice teachers in 12 courses using the cases. Instructors made greater use of the full range of values (1-3) for each criterion with a seven-score rubric employed in fall 2002 than a three-score rubric employed in spring 2003. Overall, individual scores were strongly associated with one another for a student on a given case. Scores were only moderately correlated across cases for the same individual. There was no evidence of systematic growth in essay scores over time with either rubric.

Abstract: The statistical relationship between scores assigned to ETIPS case essays and measures of the relevancy of the user's search to the case question are explored. The sample includes data from 117 student users in ten course sections over two semesters. There was no significant relationship between essay scores and the relevancy index (sum of relevancy weights divided by the number of steps taken in the case). There was a weak relationship between the number of steps taken in a case and essay scores such that highly rated essays reflected at least a minimal search of the case. An alternative measure of relevancy, the number of unique items relevant to the case question accessed, did have a statistically significant relationship to essay scores. This relationship was strongest in later rather earlier cases completed by the user.

Abstract: The appropriateness of a 2 x 2 typology of case users was explored using data from the first year of field testing. The typology was based on the interactions between two measures: quality of case essays and degree of expertise in case search (relevancy). The typology includes four types of users: (1) those having a high quality essay and relevant search; (2) those having a low quality essay but relevant search; (3) those having a high quality essay and irrelevant search; and (4) those having a low quality essay and irrelevant search. The validity of the typology was confirmed through an examination of the case search characteristics of each of the four types of users. Those having an irrelevant search typically accessed much less information than those having a relevant search although they tended to focus heavily on information about available technology (without hitting relevant items). There were users, however, who could offer a thoughtful essay response without relying on relevant case information. Nevertheless, access to relevant case information does not guarantee that users will translate such information into a high quality essay response about the case.

Abstract: A series of statistical tests were carried out to examine the relationship between individual user characteristics (age, gender, student teaching experience, technology skill) and student performance in the ETIP cases. Measures of case outcomes include instructor-assigned essay scores, relevancy of case search, extent of case search, and proportion of search devoted to different categories of case information. Overall, individual student characteristics have a weak and inconsistent relationship to how students work with the cases. By comparison, between class variance accounted from 9 to 39 percent of the variance in case outcomes.