NOVEMBER 8, 1996
To: Members of the Evaluation Subcommittee
From: Parker A. Small, Jr., M.D., Chair, Evaluation Subcommittee
A. REMEMBER NEXT MEETING, WEDNESDAY, 7:30 A.M. IN CG-78. The agenda is to review and modify the "strengths and weaknesses" document.
B. At the last meeting of our evaluation committee, members requested copies of the College's mission statement, goals and objectives. I have therefore distributed by mail:
1. Statements from our LCME database. Items IIA,D on pages 3-5 are particularly relevant.
2. End-of-Third-Year Competencies. (There is a minimum of basic science relevant competencies in this list, but page 3, Problem Definition, item 3 covers a great deal of information).
C. The following is a very rough draft of "strengths and weaknesses," in which I have attempted to summarize our first two meetings. I hope you will each rewrite and/or add to those sections in which you have expertise.
The College of Medicine is responsible for educating medical students to be able perform capably in any residency program of their choice by providing them with a general medical education. The Curriculum Committee is responsible for the design, implementation and monitoring of the education program. As an important step in this process the Curriculum Committee has identified the "end of third year competencies." The success and continued improvement of the education program is dependent upon appropriate evaluation including the feedback of the evaluation data to the necessary people. Authentic summative evaluation of students validates all aspects of our education program and should be the primary driving force. Such evaluation should identify strengths and weaknesses of the education program. Appropriate feedback of the data to students, faculty and administration should lead to improvement.
Formative evaluation of students is required to assure student progress toward the identified competencies. Evaluation of faculty, courses, curriculum and the educational environment should identify strengths and weaknesses that can account for student success and failure. The goal of our subcommittee is to design a system that collects the appropriate data and shares it with the necessary people so as to assure continued improvement of student accomplishment and our educational program.
A. Student Evaluation 1. Strengths - Two methods make our student evaluation system innovative in the nation. a. The end of second and third year OSCE exams, which utilize standardized patients, are authentic exams, which are evolving into a powerful system for providing feedback to students and faculty. There is anecdotal evidence that feedback of student performance on the 2nd year OSCE is improving faculty instruction in physical diagnosis.
b. Peer evaluation: The formative peer evaluation system used in (Rarey's course) and immunology improves students ability to teach each other. According to Dr. Hill, the summative peer evaluation, at the beginning of year 4, enables some students to get better residencies than they would have, based just on standard class ranking.
2. Weaknesses: a. One of the major weaknesses, especially in the basic science years, is that our evaluation system relies heavily on non-authentic exams. This forces students to choose between studying to be good doctors and studying to get good grades. There is excessive use of MCQ's, few of which are problem solving questions. Another problem is that the evaluation system sustains the competitive behaviors promoted by admissions policies and premed programs.
b. Feedback to students relative to professional development is minimal in the basic science years and unpredictable in the clinical years. We instead tend to focus primarily on knowledge development, however even in the knowledge sphere there is inadequate feedback to students and faculty. Performance on exams, reported as grades, is the primary feedback to students and faculty. For example, 4th year students do not get feedback on station by station performance unless they fail that station. Few basic science faculty know about student retention of the knowledge those faculty taught them, much less the students' ability to use that knowledge in a clinical setting.
c. During the clinical years, student evaluations are based on a more diverse set of observations made by faculty and residents plus a variety of cognitive exams. This lack of structure leads to variability in both reliability and validity of evaluations within and between rotations.
d. The feedback about student cognitive and non-cognitive development is not shared with the admissions committee.
B. Faculty Evaluations 1. Strengths The portfolio approach has been introduced for use by promotion and "Teacher Improvement Program" decisions. Student evaluation of teachers is collected in most courses and given to faculty. 2. Weaknesses Feedback is often collected and provided to faculty long after the teaching activity, thereby decreasing the specificity and effectiveness. Peer evaluation of faculty has been introduced, but is not yet widely used.
C. Course Evaluations 1. Strengths The debriefing sessions following each course have been of great help in improving the quality of the courses. This informal feedback to faculty is usually of more help than the summary of the evaluation data provided by the OME, but the latter including individual comments is also valuable.
2. Weaknesses Immediate feedback to faculty, e.g. relative to confusion about lecture content, is often inadequate.
D. Curriculum Evaluation 1. Strengths Graduation and post graduation alumni feedback is helpful even though it is of necessity "out of date" given the rapidly evolving curriculum.
NBME scores and LCME evaluations are shared with some faculty. A comprehensive database is being developed and collected for each student.
2. Weaknesses Feedback from residency programs relative to the strengths and weaknesses of our graduates is not shared with the faculty.