The charge to the Task Force on Student Evaluation was:
Susan Case and David Swanson, of the National Board of Medical Examiners, cited 5 purposes for evaluating students:
The task force takes a strong position that a "grade" is not synonymous with evaluation. Rather, assignment of a grade is only one of the purposes of the evaluation system.
The task force reviewed the concept map. Changes recommended: 1)Add a communication box to clinical skills section; 2) include family in box with patient education to read family and patient education; 3) consider moving the box reading criterion-based directly under the box for student evaluations at the top of the concept map. (See Attachment 1)
Attention then turned to the Student Evaluation Matrix. The matrix was not consistent with the concept map in the description of the clinical skills. The column for Pt. Rapport was deleted. Columns were added for communication(written and oral), professional behavior, and procedures. The word "family" was added to the column for patient education to read "Pt. & Family Educ." Evaluation methods appearing in the rows of the matrix were reordered to place similar methods in proximity. (See Attachment 2)
Definitions of the evaluation methods were developed so that the task force could use the same criteria for evaluation of the methods. In some of the definitions, a brief statement on the reliability and validity of the method was included. (See Attachment 3)
Effectiveness of a method was judged by the ability of the method to produce data on the competency being considered, the reliability of the data and validity of the method. Cost is judged by resources required. Human resources were judged to be the most expensive and faculty resources topped the list of human resources.
The student evaluation task force identified five areas of emphasis where omissions or deficiencies in the evaluation system have been noted: 1) formative and summative feedback to students; 2) written course and clerkship objectives related to graduation competencies; 3) criterion-referenced evaluation; 4) evaluation methods effective for testing all of the required competencies; 5) assessment of professional behavior; and 6) strategies for periodic evaluation of skills and abilities which are developmental.
Students complain that feedback is neither timely nor specific enough to determine their progress and take corrective action. Often the only feedback received is a letter grade e.g. A, B, C, at the end of the course. On the other hand, faculty believe that they provide frequent feedback to the students but the students donÕt perceive it as feedback because it is not written or labeled as such.
The task force recommends that at a minimum, feedback be provided in written form at the mid point of the course/clerkship and at the end. Students who are experiencing difficulty or who's performance is unsatisfactory must be notified in writing as quickly and as frequently as necessary to assist the student to take corrective action. Formative and summative evaluations must provide comments which are specific and provide guidance to the learner for future learning endeavors. Evaluation should reward appropriate behavior and provide counsel on behavior that must be modified.
The task force recommends the use of praise cards/early concern notes. A stack of cards and envelops should be sent from COMEC at the beginning of each course. Cards are not used on all students - only those deserving high praise or are demonstrating behavior of concern. (See attachment 4)
Course and Clerkship Objectives
Students often report that the first opportunity they have to find out the course objectives is when they see the final exam.
The task force recommends that learner objectives be provided to the students at the beginning of each course. The objectives should be placed in a database and on the WWW so that all students and faculty have access to the objectives of all courses. Faculty should review the learner objectives with the class at the beginning of the course and periodically throughout the course, calling attention to the course activities which accomplish the course objectives as well as the graduation competencies. Course objectives should be directly related to outcome measures of the course and also to the graduation competencies. Test content and emphasis must be consistent with the objectives. The tests should assess major points and not isolated, obscure facts. All courses should incorporate evaluation of higher cognitive level functioning e.g. problem-solving.
Criterion-referenced Evaluation
Criterion-referenced or content-referenced tests are concerned with mastery of defined skills or content areas. The focus is upon reaching an established standard of performance. On the other hand, with norm referenced tests, acceptable performance is set by group comparison.
The task force recommends that standards of performance for mastery in each course be established and appropriate evaluation methods be used to determine mastery. Students must be advised of the standards of performance required for mastery.
Employ Effective Evaluation Methods for Assessing Competencies
A variety of evaluation methods must be employed to assess the complex array of clinical skills. Multiple choice question examinations are used extensively in medical schools even though they primarily test simple recall of factual information. Proponents argue that MCQs are objective, reliable, and inexpensive to administer. While this is accurate when the objective of the exam is to assess simple recall, the results from MCQ exams should not be extended to decisions beyond the realm of the test. To assess skills and knowledge beyond recall of facts other evaluation methods must be employed.
The task force recommends that all courses and clerkships review their objectives in relation to testing methods currently being used and make a written report to the curriculum committee and/or the evaluation subcommittee using a specified format. The reports will be reviewed and feedback given to the course directors on the validity and reliability of the evaluation methods for assessing the course objectives with recommendations for changes if indicated. (See Attachment 5)
Strategies for Periodic Evaluation of Developmental Skills and Abilities
Many clinical skills require development over time to achieve higher levels of performance. For example, interviewing expertise evolves with multiple opportunities for instruction, practice and critique. The skills and abilities which mature over time and therefore occur in multiple courses must be evaluated at defined points along the educational continuum. Many of these skills are introduced during the first and second year Essentials of Patient Care and continue through the clerkship rotations. The task force reviewed all required clerkship student evaluation forms. All of these forms could be classified as global forms. Some forms simply ask for a letter grade with little direction to the evaluator as to what level of performance is required for the grade increments. Other forms have multiple skills listed with descriptors at anchor points on a rating scale.
The task force recommends that BCSI and II and 2nd year physical/clinical diagnosis courses coordinate the evaluation of basic clinical skills at increasing levels of performance on OSCE 1 and OSCE 2. The second recommendation is that required clerkships develop an evaluation form with standard items across all clerkships. For example, items which all clerkships should have in common are history taking, physical exam, communication (oral and written), patient and family education, problem-solving, diagnosis and treatment, and professional behavior. Items which are unique to a clerkship may be added to the list of standard items. The standard evaluation document should be on a scannable form and processed by the Office of Medical Education (COMEC). Reports generated from the evaluation of items on the standard form will provide the students and institution with information on progress toward the accomplishment of these developmental abilities. A third recommendation is that a review of the 4th year course evaluation forms be reviewed. This task force had time constraints which did not allow sufficient attention to these forms.
Assessment of Professional Behavior
The task force spent considerable effort discussing the attitudes and values that constitute professional behavior. Definitions were solicited from task force members. Possible approaches to evaluate these attitudes and values were explored. All agreed that professionalism should be fostered throughout the educational process. The task force reviewed the publication from the American Board of Internal Medicine Committee on Evaluation of Clinical Competence, Project Professionalism, 1992-94.
The task force recommends that professional behavior be a defined and formal component of the graduation competencies. Each course/clerkship should have written objectives specifying the professional behaviors that are being addressed and evaluated in the course.
The task force recommends for consideration the following definition of professionalism and elements of professionalism taken from the ABIM:
The task force recommends that appropriate elements of professionalism be incorporated into all courses/clerkships evaluations of the students.
As with all evaluation, multiple sources and methods should be tapped for a complete assessment. When considering professional behaviors, peers have a different perspective from faculty. Residents will have a unique viewpoint. Patients and their families will have a distinct outlook. Nurses and other staff should also be surveyed.
Examples of evaluation strategies for assessing professionalism are included as an addendum. These examples are from ABIM Project Professionalism and demonstrate that attitudes and values can be defined and assessed. (See Attachment 6 and 7)
The task force reviewed and accepted the report of the Subcommittee on Evaluation/Competencies Report of September 7, l994. The questions asked by the younger members of the task force were "why wasn't this implemented in 1994 when it is substantively what we are dealing with now?" and "what is going to ultimately be done with the work of this task force?" These questions should be discussed by the larger committee.