Report of Task Force On Curriculum Evaluation

Special Issues or Questions to be Raised in Annual Probes
Task Force Members:
Richard Reynolds, Larry Rooks, Emanuel Suter

Introduction

The charge to this task force was to: (1) identify the information the Curriculum Committee (and the Sr. Associate Dean for Educational Affairs) needs to carry out its mission of planning and evaluating the curriculum; (2) assess the adequacy of the currently available data and their use by the Curriculum Committee and (3) recommend measures for possible changes in the nature of the data collected, their analysis, and use.

1. Data Needs

The concept map (see attachment 1) and the matrix (attachment 2) show that the Curriculum Committee requires data or anecdotal information from a wide range of sources and activities covering nearly all aspects of the medical school, as they all may impact the quality of the educational program leading to the M.D. Degree. Every seven years, on the occasion of the institutional self-study for LCME reaccreditation, a wide array of data and information is brought together in the LCME database and analyzed in the LCME report. Although the Curriculum Committee participates in and is a major contributor to the assembly of the data base, it takes no formal action on its findings. Nevertheless, the LCME self-study process and site visit and the LCME accreditation report significantly influence some of the Curriculum Committee's actions.

  1. Effective planning by the Curriculum Committee requires a wide range of information regarding resources in support of education (see concept map). The Curriculum Committee must be aware of resource opportunities and restraints when either changing programs or creating new ones. For example, the creation of the Harrell Development and Assessment Center (a facility for the use of standardized patients) provided an opportunity for better instruction of history taking and physical examination, as well as for implementing the principle of competency-based evaluation. Likewise, full implementation of mission-based budgeting may have a significant impact on the Curriculum Committee's ability to engage the faculty in educational innovation.
  2. The greatest information and data needs are in the area of instruction (process) and student accomplishments (outcome). Such data are the indicators of the quality of program planning and implementation. They must address performance of courses, clerkships, faculty, and students.
  3. Considering the significant changes in the organization and financing of health and medical care, the scientific and technological foundations of medical practice, and the information management revolution, there is a need for information about the graduates' preparation for successfully adapting to change.

2. Adequacy of Currently Available Data

COMEC develops an enormous wealth of data which are distributed to various parties: Sr. Associate Dean for Educational Affairs; course/clerkship directors; teaching faculty; department chairs; Curriculum Committee; and others. Recipients, depending on inclination, reach their own conclusions and may or may not take corrective action if deficiencies are found. However, rarely is a comprehensive analysis done that may result in specific recommendations to the Curriculum Committee. If such recommendations are made, they may be generated more by chance than by deliberate planning.

Data currently available to the Curriculum Committee fall into the following categories:

  1. Student performance on internal and external examinations: For example, the progressive decline of performance by PIMS students on the NBME examinations part I resulted in significant changes of the PIMS program initiated jointly by the Sr. Associate Dean for Educational Affairs at the College of Medicine of the University of Florida and the Provost of Florida State University.
  2. Results of course and clerkship evaluations by students may reach the Curriculum Committee and result in remedial action. Since the discontinuation of an organized course/clerkship review process, this is less likely to happen.
  3. Graduate surveys of a retrospective nature have been annually carried out at the time of graduation and one, three, and six years after graduation. They have yielded reams of data which have never been analyzed in a comprehensive fashion. However, it is doubtful that several years after graduation and after a minimum of three years of housestaff training any specific information about courses or clerkships can be expected. Nevertheless, it is interesting that two courses, human anatomy and physiology, consistently have received higher ratings than the others. Overall, the surveys together with the AAMC graduation questionnaire indicate a high degree of student satisfaction with the curriculum.
  4. Performance of graduates: (1) An annual survey is sent to the first year residency program directors to give a global assessment of the graduates' performance. Not much more information can be gleaned from the returns than the fact that up to 80% of our graduates are placed in the upper third of their 1st year housestaff program. (2) A special task force (Longitudinal Survey Committee) explored various options--that could in theory yield useful information about our graduates' performance when in practice. It suggested alternative questions that could be used in future longitudinal surveys as well as more sophisticated approaches for assessing quality of care rendered. The Curriculum Committee has not followed up on any of the recommendations.

3. Recommendations

Without any doubt, the Curriculum Committee must have a system of surveillance that furnishes it timely and useful data on the performance of the educational program. Achieving and maintaining high quality require vigilance. The Curriculum Committee cannot delegate this task to the NBME or the Liaison Committee on Medical Education, as important and well qualified these organizations are in regard to their own respective missions.

For the Curriculum Committee to be able to monitor the "health" of the curriculum, we recommend the following:

  1. The four class survey should be discontinued on an annual basis. Information, based on experiences in courses which have changed significantly in the interval, varies little from year to year and therefore has lost validity. Administration could be continuedd on two consecutive years preceding the LCME self-study to obtain quantitative information for the purpose of data display.
  2. In lieu of this survey, the Curriculum Committee, or its standing Evaluation Committee (see item e. below), should initiate a more issue oriented annual survey that may provide needed information not only to the Curriculum Committee but also to those in charge of GME and CME. Potential questions are suggested in item d below and in attachment 3.
  3. Improve the survey form sent annually to residency directors to obtain more specific performance data. For instance, include questions about:
  4. Another potential probe is an exploration of the extent to which the curriculum adheres to the principles which the Curriculum Committee approved in 1994 and is in the process of reaffirming in 1997 (if it ever does). Questions, the same in content directed to students and faculty but formulated differently, could address the following:
  5. from the graduate's perspective, did we create an enduring foundation in the sciences basic to medicine?

  6. There should be a standing "Curriculum Oversight Committee" charged to present, at a special session of the Curriculum Committee (if not a retreat), a report which will:

    1. summarize data obtained from all relevant evaluations, including students, faculty, and courses and clerkships
    2. identify weak aspects of the program, if any, and suggest interventions
    3. recommend specific survey questions or "probes" for the forthcoming academic year to obtain data on areas of special interest to the Curriculum Committee (as recommended in item b)

    When approved by the Curriculum Committee, this report should be distributed to the faculty. Final comment: In this Subcommittee's opinion, this last recommendation sould have highest priority for implementation as it would have a beneficial impact on all evaluations covered by the Evaluation Subcommittee's activities. The major issues in our evaluation system are not the instruments but the lack of mechanisms to close the feedback loop.

    Special Issues or Questions to be Raised in Annual Probes

    (These items were taken and slightly modified from a January 20, 1993 report of the Longitudinal Survey Subcommittee and from a letter of March 10 by Dr. Richard Reynolds to Emanuel Suter).

    1. How do you maintain your professional competence and currency (no. hours per week)?

2. When confronted with a patient with a puzzling condition, what do you do to obtain information that might be helpful in the management of the patient?

Note: one could ask for the number of hours per month

3. How do you accomplish your CME requirements (give options)?

4. Do you have a systematic plan for CME?

5. Is your CME based on recurring patient problems seen in practice or on other issues (e.g. genetics, managed care)?

6. Have any learning activities or habits developed in medical school and residency been of help to you in the process of information retrieval (about patients or other specific subjects) or continuing learning?

7. Do you have any specific CME needs that the College of Medicine could help you with? (List areas)

8. Have we succeeded or failed in preparing you to adapt to the many changes in medical practice?

  Updated: April 6, 1997