Curriculum Evaluation Plan (Exhibit 6)

Back to exhibits

I. Evaluation of Student Achievement
II. Evaluation of Courses/Clerkships
III. Evaluation of Faculty Instruction
IV. Evaluation of the Medical Curriculum

Approved by the Curriculum Committee March 9, 1999

In 1997, the Evaluation Subcommittee was charged by the Curriculum Committee to design and implement a comprehensive evaluation system that included:

• a means to more effectively evaluate students' competencies and for students to have frequent, specific feedback about their proficiency of mastering the knowledge, behavior and skills required for graduation.
• a means to evaluate courses and clerkships that provide the directors of courses/clerkships feedback as to strengths and concerns, and integration with the comprehensive curriculum plan.
• a means to evaluate the teaching and provide teaching faculty timely feedback about their teaching effectiveness
• a means to evaluate the overall medical curriculum

Accordingly, the evaluation subcommittee has crafted four respective evaluation formats to be implemented July 1, 1999. The formats are to evaluate:

• student achievement
• courses/clerkships
• faculty instruction
• the medical curriculum

The formats are based upon the concepts presented in the document, "University of Florida College of Medicine Curriculum Renewal Plan, B. An Evaluation System: Evaluation of Student Achievement, Faculty Instruction, Courses and Clerkships" approved by the Curriculum Committee in June 1997.

I. Student Evaluation

[Originally approved by the Curriculum Committee, June 30, 1998; Revised 1/27/99, 12/11/02]

The goals of addressing student evaluation are threefold:

• to measure a student's academic achievement and competency development
• to promote timely and specific feedback to students so that they can evaluate their progress
• to identify students with academic problems in order to support and provide remediation as needed, to identify outstanding students and provide appropriate recognition.

These goals can be achieved successfully by accomplishing the following seven objectives. Such objectives are not totally new; some objectives were proposed in reports of past task forces of the curriculum committee. **

**Subcommittee on evaluation/competencies report, M. Duerson, chair, 9/7/94
Final report of the committee for review of student evaluation, O. Harris, chair, 4/87
Report of Task Force on Student Evaluation, M. Duerson, chair, 3/25/97
Revised Report of task force on student evaluation, M. Duerson, chair, 4/9/97
Subcommittee on competencies and evaluation, M. Duerson, chair, 10/1/93


Objective 1: Student evaluations must be based upon learning objectives that are competency-based and criterion-referenced when possible.

1. Measurable, learning course/clerkship objectives must be given to students at the outset of each course/clerkship, and must relate to graduation competencies. The objectives will serve as explicit criteria to measure mastery of student's knowledge, behaviors and skills.
2. Faculty development will be provided selectively to assist faculty in writing measurable learning objectives and to encourage faculty to use innovative criterion-referenced methods of assessment.

Objective 2: Performanced-based assessment should be a component of all course and clerkship evaluation plans.

1. Examinations must be based upon measurable learning objectives.
2. The use of performance-based formats, other than multiple choice questions (MCQs) must be increased. Use of performance-based evaluations is strongly encouraged in all courses, but is required in all clerkships.
• observed performances, e.g., observations of patient interaction in the clinical setting or PBEs
• small group learning activities
• oral, practical or essay examinations
• student thesis
3. Faculty development will be provided selectively to faculty to use innovative methods in assessment.

Objective 3: Formative evaluations and feedback must be incorporated into the evaluation process.

1. Written competency-based formative feedback must be given to a student by course faculty at least once during the course or clerkship. Ideally this should be accomplished mid-way or earlier during accrues/clerkship.
2. The inclusion of student self-assessment is strongly encouraged during the formative feedback process.
3. Faculty development will be provided selectively to enhance the faculty's ability to provide effective feedback.

Objective 4: Summative evaluations of students' learning must be based upon competency-based objectives.

1. There will be two summative feedback systems utilized for evaluation of medical students:
a. Course and Clerkship Evaluations: Students level of competencies will be evaluated for each course and clerkship.
b. Performance-Based Examinations Series (PBE): Four PBEs (I-IV) will be administered during the four years of medical curriculum to assess the progress of students development.


PBE 1	PBE 2	PBE 3A	PBE 3B
Diagnostic Skills	Diagnostic Skills	Diagnostic Skills	Diagnostic Skills
History	History	History	History
Communication	Communication	Communication	Communication
Professionalism	Professionalism	Professionalism	Professionalism
		Problem-solving

1. PBEs will be criterion-referenced for judging the standard of performance.
2. Students not meeting standards will be required to remediate.
3. PBE remediation will be competency specific and based on predetermined criteria.
4. Students' Competency Profiles will be maintained in the students' academic records.
2. Faculty development will be provided selectively to faculty to use a variety of summative evaluation techniques.

Objective 5: Evaluation by individuals, other than teaching faculty [e.g., students (self and peer), patients, nurses], when possible, must be incorporated into the evaluation process.

1. Self-assessment: Students are encouraged to utilize a self-assessment in monitoring their progress. Student self-assessment should be encouraged by faculty teaching specific courses/clerkships. A journal may serve as a means for students to document their assessment of learning.
2. Peer assessment: Evaluation by peers should be included in the evaluation process when possible.
3. Patient assessment: Evaluations of students by patients (standardized or actual) should be incorporated in assessing students. The use of standardized patients will decrease variability and increase reliability. In some circumstances, standardized patients should be introduced into clinical settings without the students' knowledge.
4. Other health care professionals assessment: Evaluations by individuals of the health care team should be considered in determining the competency of a student.

Objective 6: Assessment of professional behavior and human relationship competencies must be incorporated in the evaluation process for all clerkships, and is strongly encouraged in preclinical courses.

Evaluation of students' professional behavior and human relationship competencies must be conducted during all clerkships, and is strongly encouraged for basic science courses.

Objective 7: A uniform competency-based evaluation system must be established.

1. A standard student evaluation will be created for all first and second year courses, required clerkships, and electives that have common performance criteria to assess student achievement of competencies.
2. Course and Clerkship Directors must identify levels of competency upon which formative and summative evaluations are based, and identify minimal standards for satisfactory performance.
3. The evaluation of competencies will be reported on a four-interval (descriptive scale)* evaluation system. This scale will provide formative and summative feedback and assessment on evaluation of clinical skills and application of scientific knowledge.
4. An electronic database will be created to track student competency development through all four years of the curriculum.
*The recommended descriptive labeling system is based upon the following select references:
*1. Downing, S. Improving Measures of Student Performance, Acad Med 65(9):S57-58, 1990.
2. Magarian, GJ, Majur, DJ. A National Survey of Grading Systems Used in Medicine Clerkships, Acad Med 65:636-639, 1990.
3. McLeod, PJ. So Few Medical Schools, So Many Clerk Rating Systems! Can Med Assoc J 146(12):21612-2164, 1992.

II. Evaluation of Courses and Clerkships

Introduction

Mission-based budgeting seeks to allocate educational funds equitably, support desired educational outcomes, promote innovation, and reward outstanding teachers, courses and clerkships. To accomplish this, instruction must be evaluated by mission outcome, as well as by activity. Courses and clerkships are evaluated quantitatively through the Faculty Effort Reports you submit. The qualitative component is by evaluation of your annual report, composed of the syllabus and your written report, and numerical data from the College of Medicine education Center (COMEC). Course and clerkship evaluation is performed by the Evaluation Subcommittee of the Curriculum Committee chaired by Kyle E. Rarey, Ph.D. Recommendations of the committee are forwarded to the Curriculum Committee and to the Senior Associate Dean of Educational Affairs. Recommendations for providing resources to courses and clerkships are then forwarded to them by COMEC.

The annual report consists of three parts:

I. The course or clerkship syllabus given to the students
II. The written report prepared by the director with accompanying appendixes
III. Data supplied by the Office of Medical Education (OME)

In order to facilitate evaluation, course/clerkship directors should be sure to include all material requested in the exact form and organization requested. Each section of the syllabus and report should be clearly delineated and identified to correspond with these instructions.  

Each section of the syllabus and report are evaluated longitudinally by members of the subcommittee. Four parameters are used to assess quality of a course or clerkship syllabus and four parameters to assess the written report. A reviewer will assess the same section in all reports, i.e. like an NIH grant evaluation. Therefore, each section should “stand alone.” For example: if a deficiency is identified in the section on student evaluation of instruction and a strategy for improvement implemented, the improvement should appear in that section in addition to the separate section on plans for improvement. Do not refer a reviewer to another section.

An example of an exemplary submission(course report/syllabus, clerkship report/syllabus) and the evaluation form used by the subcommittee is provided. It is strongly recommended that these documents be reviewed during preparation of materials.

I. COURSE/CLERKSHIP SYLLABUS

Given to the students. In order, it must contain:

A. Overview

This section should contain the four following elements at a minimum:

1. A brief description of the course/clerkship
2. The administrative structure

   a. Department chair with contact information
   b. Course/clerkship director with contact information
   c. Course/clerkship administrative assistant/secretary with contact information

3. An overview of the clerkship content

   a. Length, hours, units of study as appropriate
   b. Sites for instruction
   c. Schedule for a typical week

4. The educational philosophy

   a. Course/clerkship
   b. Department - if there is one

B. Learning Objectives

These must be in the specific format requested (see representative course and clerkship syllabus template)

1. The competency categories to be addressed
2. The measurable learning objectives for each competency category
3. The learning/instructional activity used to achieve each learning objective
4. The evaluation methods used to assess student achievement of each learning objective.

C. Student Performance Criteria

1. Identify the internal and external measures used to assess student's knowledge and skills. Representative forms should be included.
2. Describe the system for giving formative feedback. Representative forms should included, but alone are not sufficient.
3. Describe the system for determining summative feedback. Representative forms should be included.

II. DIRECTOR'S WRITTEN REPORT

Representative course and clerkship written report templates.

A. Identification of the Teaching Faculty for the Course or Clerkship

1. Director(s): Total FTE
2. Administrative support: Total FTE
3. Departmental Teaching Faculty: Total FTE
4. Non-departmental Teaching Faculty/Staff/Students: Total FTE

B. Specific Changes in the Course or Clerkship

1. Compare to last year and the rationale or basis for any changes.
2. Address remediation of areas of weakness or deficiency cited in the previous report.

C. Evaluation of Student Performance

1. Identify types of evaluations used to assess student levels of knowledge and/or skills. Comment on the appropriateness of them.
2. Describe how students performed on internal or external (e.g., NBME subject examinations) instruments relative to the course/clerkship learning objectives.
3. Based upon the data from the Office of Medical Education, comment on the distribution of competency scoring and final grade distribution of student performances. Do you believe that the course evaluation system identified students who were truly exceptional and those that may need additional assistance?

Number of student performances scored in the following competency and grade categories.

Competency Category	Needs Remediation	Acceptable	Very Good	Exemplary
1. Professionalism
2. Patient Care
3. Medical knowledge
4. Practice-based learning
5. Interpersonal and communication skills
6. Systems-based practice

 

Year	D or F	C	C+	B	B+	A
Current Year




D. Feedback to Students

1. Identify the student rating to the on-line course question about formative mid-course feedback.
2. Describe your interpretation of such scoring in reference to what is written in your course syllabus about formative feedback, and what actually occurs.
3. Discuss the legitimacy of student ratings.

E. Student Evaluation of the Course/Clerkship and Director

1. Provide a synopsis/summary of student on-line evaluations of the course/clerkship. This should include analysis and comment by the Director of strengths and areas needing improvement. Numerical data alone is not sufficient. Discuss the legitimacy of the student ratings.
2. Provide a synopsis/summary of the student debriefings. This should include analysis and comment by the director of student concerns. Minutes alone are not sufficient. Discuss the legitimacy of the student ratings.
3. Provide a synopsis/summary of student evaluation of the Director, and administrative support, if applicable. Discuss the legitimacy of the student ratings.

F. Evaluation of Instruction

1. Provide a synopsis/summary of student evaluation (on-line and debriefings) of the overall instruction by faculty, residents and staff, as applicable.
2. Critique the overall teaching performance of faculty, residents, and staff from the Director's perspective. Identify and comment on strengths and areas needing improvement.
3. Identify any plans for faculty development.

G. Evaluation of Course/Clerkship

1. Discuss the timing and appropriateness of the course/clerkship to the curriculum's general professional education program including content and instructional methods.
2. Discuss the strengths and weaknesses of the course/clerkship and describe plans for improvement. Include a description of present and/or future innovative programs.
3. Discuss the long-term plans for continued development and new directions for the course/clerkship.
4. Describe the resources needed from the Office of Medical Education to improve the course/clerkship.

III. COURSE/CLERKSHIP AND STUDENT NUMERICAL DATA SUPPLIED BY THE OFFICE OF MEDICAL EDUCATION (OME)

Note: Course and Clerkship directors are not required to submit data for section III in their annual reports.


A. Course/Clerkship Evaluation - Student year end evaluations
B. Course/Clerkship Instruction - Student year-end evaluations
C. Course/Clerkship Director - Student year-end evaluations
D. Student performance

1. NBME subject examinations
2. USMLE Steps 2 and Steps 2, if applicable.

III. Evaluation of Faculty Instruction

Evaluation of faculty instruction is based upon following premises:

a. Effective teaching skills can be defined. Characteristics of good teaching that can be evaluated are reasonably consistent in the literature. These include:

1. being well prepared for class
2. motivating students
3. effective communication skills
4. demonstrating comprehensive knowledge
5. treating students with respect
6. organized presentation
b. Sources to assess teaching effectiveness can be identified. The most commonly identified evaluators of teaching effectiveness are students, peers, self, and college administration, with each group having its unique role. Faculty self-evaluation, when used with student and peer evaluation, is helpful in faculty growth and development. This self assessment must be consistent with assignments and the goals and objectives for faculty development agreed upon with the departmental chair.
c . Some, but not all, faculty evaluation systems change behavior. Faculty development and evaluation are related processes. The former aspires to improve faculty performance and the latter aims to make judgment regarding worth. The processes can be integrated,and can be a powerful technique in changing behavior. On the other hand, there is little evidence that faculty evaluation alone improves instruction. As with any feedback system, faculty evaluation conducted early in the course of instruction favored instructional improvement because it allows faculty members the time and opportunity to make modifications.


Over the past five years, there have evolved standard methods and forms for student and peer evaluation of the faculty through College- and University-Wide initiatives. All teaching units should conform with the standards and use the required forms. Compliance will allow course directors to perform their responsibilities more efficiently, department chairs to have more meaningful assessments of faculty achievement, and the administration to more accurately assess the effectiveness of the overall teaching programs.

For the Curriculum Committee to assess the quality of instruction by the teaching faculty within the courses/clerkships, course directors are responsible that the following objective is addressed.

Objective: Student Evaluation of Teaching Faculty

Standard forms for evaluating teaching faculty by students (Exhibits 2-5) should be used for student assessment (they are available in hard copy and online in Internet-based versions). The forms are available in four formats: Basic Science Teaching, Clinical Teaching, Graduate Medical Education, and a short form for Lectures/Discussion Groups. The latter form should be used only when a faculty member has a minor role in teaching activities. These data will be included in the annual course report.

FOOTNOTE*
1 In preparing this report, the recommendations of two subcommittees that prepared reports on Faculty Evaluation in 1997 were reviewed. A subcommittee chaired by Pat Alguire submitted their report on April 9, 1997. A second report, prepared by Patrick Duff, Jeaninne Webb, and Tim VanSusteren, was submitted to the Curriculum Committee on September 10, 1997. In general, we confirm and support the recommendations of the Alguire Subcommittee. The major recommendations of the other subcommittee dealt with modifying the questions and format of the evaluations forms used in the College of Medicine, as part of the SUS standardization of the evaluation of the faculty. Since the COM is required to conform to the SUS standards, it is not productive to propose such modification.

IV. Evaluation of the Medical Curriculum

The primary mission of the COM is to educate medical students in humanistic, scientific and practical principles of medicine in keeping with the emerging needs of society. Therefore, an effective, dynamic curriculum that serves its students and supports its teaching faculty must be continually reviewed to determine its quality and how students perform within it.

The following are the general goals of the curriculum evaluation system:

1. To ensure that the curriculum is "appropriate" for the education and development of competent physicians.
2. To assess whether the measurement tools are "appropriate" to gather data about the effectiveness of the programs.
3. To update deans, teaching faculty and students as to the effectiveness of the educational program.

Curriculum planning at the UF COM is guided by a set of principles, which were adopted by the Curriculum and Executive Committees. These 12 principles describe which characteristics that the overall curriculum design should possess. The principles, therefore, should be the yardsticks to measure the success of the overall program. A specific evaluation plan is designed to assess the degree of accomplishment of each principle.

These principles are listed below:

1. A general professional education is the goal of the curriculum.

The curriculum is intended to provide a set of knowledge, skills and behaviors that encompass a broad overview of the practice of medicine and is relevant to becoming a competent caring physician regardless of subsequent specialty choice.

Evaluation Issue:

• Is there "appropriate" content in the curriculum for a general professional education?
• Determine whether the general professional education program is having the desired impact on students

Evaluation plan:

• Solicit opinions of graduates as to how well the curriculum prepared them for the general practice of medicine.
• Review periodically the college's-adopted list of competencies to ensure a general professional education of each student.
• Annual survey of UFCOM students performance in residency programs.
2. The educational program and evaluations are competency based.

The general professional education is defined by a set of competencies expected to be "mastered" by the students.

Evaluation Issues:

• Does the curriculum content and structure, and the student evaluation system, adequately include the use of competencies?

Evaluation Plan:

• Determine whether all competencies are addressed in the curriculum (i.e., that at least one course teaches it and evaluates students to be sure they are competent in the area).
• Review competency categories to determine where each objective fits into the curriculum .
• Course and clerkship reports should be analyzed for the extent to which education and evaluation is competency based.
• Review descriptors to be sure appropriate descriptors are being used for competencies.
• Assess competency list every 3-5 years to update/modify it.
3. A conceptual framework for defining knowledge promotes learning and effective utilization of that knowledge and serves as a basis for curriculum integration.

The curriculum committee has adopted the clinical presentation model as the primary mechanism for integrating basic and clinical science content.

Evaluation plan:

• An annual narrative report should be provided by the evaluation subcommittee to describe progress in implementing the clinical presentation model.
• Course and clerkship reports should be analyzed for the extent to which they use the clinical presentation model.
4. The ability to learn independently is essential for the physician to provide quality health care.

Evaluation Issue:

• Does the curriculum require students to learn independently (and if so, specify where)?

Evaluation plan:

• Annually evaluate student's perception of their academic environment as a promoter of independent learning. Specific items in the annual class survey should assess student independent learning.
• Course and clerkship reports should include information about activities to promote independent learning.
5. Incorporation of the characteristics of outstanding physicians in the educational program is essential for complete professional development of students.

Evaluation plan:

• Utilize student self and peer evaluation.
6. Effective health care delivery requires the attention to family and community context.

Evaluation plan:

• Assess how and where in the curriculum this topic is being formally addressed.
7. A balanced variety of clinical settings are essential for students acquiring the mastery of competencies.

Evaluation plan:

• Decide upon and monitor the amount of inpatient versus outpatient clinical training site.
• Decide upon and monitor the amount of primary care versus specialty care training.
8. Informatics is essential for effective acquisition and utilization of information by students.

Evaluation plan:

• Provide a narrative description of the progress made toward implementing the Informatics plan adopted by the curriculum committee.
• Evaluate students use of informatics.
• Determine what course and clerkships utilize formal informatics activities.
• Urban vs rural; underserved vs others
9. The evaluation program is competency-based.
• Is the competency system having the desired outcome in terms of student performance?
• Have (for each clinical rotation/basic science course) an additional test, besides the shelf exams, that includes questions designed to see if students are competent in OUR competency areas, e.g., domestic violence, etc.
• Using the data from the competency tracking system, each class should be annually assessed to ensure that the students as a group are making adequate progress in each of the competency categories.
• Using data from our Performance Based Exam Series, each class should be annually assessed to ensure that the students as a group are making adequate progress in the competency categories assessed by those exams.
10. Discovery of new knowledge and solutions are part of the medical profession.

Evaluation plan:

• Assess the level of student educational scholarship, e.g., medical student research and publications.
11. Health care delivery requires individual and team efforts.

Evaluation plan:

• Determine the quality of activities which span colleges, departments, specialties, and provider types.
12. Learning and professional development requires a humane environment, which fosters respect, personal integrity, service orientation and a sense of personal well being.

Evaluation plan:

• Insert items, addressing these issues, in the annual class surveys.
• Institute a critical incident report program

Updated: 3/29/04, 6/9/04

Back to exhibits

Office of Educational Affairs

Curriculum Committee

Membership, Minutes, Exhibits, Curriculum Meeting Schedule, Documents

Evaluation Subcommittee

Schedule, Members, Minutes, Evaluation Form, Program Evaluation Data, Templates and Guides

College of Medicine Links

Office of Admissions
Office of Continuing Medical Education
Office of Financial Aid
Office of Graduate Medical Education
Office of Medical Education
Office of Minority Affairs
Office for Program Evaluation & Faculty Development
Office of Student Affairs & Registration

Search

Other UF Resources

COM Directory
COM Departments
UF Directory
MedInfo Server
HSC Library
HSC Website
UF Web Sites
myUFL (login)
COM Jacksonville

Updated: Sep 30, 2005 at 09:34AM :: Contact: Webmaster
Location: http://www.med.ufl/oea/cc/exhibits/exhibit6.shtml
Disclaimer and Permitted Use :: © 2005 University of Florida