Curriculum Committee Minutes - December 10, 2002

Present: Davidson, Bottom, Wallace, B. Wright, Watson, Gold, Gulig, Allen, T. Wright, Rarey, Lawson, Sumners, Hatch, Duerson, Karle, Butson, Normann, Meuleman, Genuardi, Harrell, Lind, Rathe

No announcements to report. The updated list of subcommittees is now posted on the Curriculum Committee website. Dr. Wright recommended Vonda Douglas to be a member of the research group. Dr. Tisher is appointing faculty to the committees and Dr. Davidson will pass this information on to Dr. Tisher.

Dr. Davidson introduced Paul Duncan, who is Professor in Health Services Administration.

Tentative agendas for the next few meetings include:
January: Interdisciplinary Health Professions Education
February: Retreat at the Sheraton Hotel
March: Combining ACGME and COM competencies

How do we measure the outcome of our students? Are we happy with our end-product?

 

Measuring Quality of Care
Donabedian
(Davidson's Corollary: Measuring Quality of
Undergraduate Medical Education)

Structure
Measures pertaining to the capacities, technologies, and infrastructure that make up the structure of the educational process (e.g., clinical information system, number and types of staff, types of facilities, access to treatment modalities.

• Quality of facilities, adequacy of faculty, appropriate resources (library, IT, number of rooms?)
• Not used as frequently as process and outcome measures
• May be used for licensing purposes

Process

• Measures typically measure activities that are actually done in educating students, AND that are believed to lead to positive outcomes. The use of process measures is based on a belief that improving the process of education will yield improved outcomes. Also called performance measures and proxy measures
• Primary issue: Does process relate to outcome, or even have any correlation
• Quality of courses, including learning objectives, student and faculty feedback, and written materials?

  Student knowledge, including:

  • Grades (process)
  • Board performance (process and outcome)
  • Quality of residency placement? (process and outcome)

Outcome

• Outcome measures represent the final effects of the intervention on the student's success
• Measures obviously must be done either at the conclusion or after training
• Measures more difficult to obtain (timing), but likely more accurate
  • Student practice as senior students?
  • Student performance as residents?
  • Student performance as physicians?

Limitations: Measuring Student Knowledge

• Knowledge may not be associated with actual practice
• The knowledge you assess may not be the knowledge that leads to high quality practice

Limitations: Measuring Student Practice

• Accuracy depends on validity of measurement (i.e. PBE, which essentially measures Hx and PE skills)
• Timing is important: because of the location of the 4th year externship, the best evaluation of practice would be at the end of medical school.

Limitations: Measuring Residency Performance

• Measurements are variable
• Problem with response rate from program directors
• validity of the assessment? Is the assessment of the PD a true measure of performance? Peer evaluation, patient evaluation, staff evaluation

Limitations: Measuring Physician Practice

• Attractive because it is the most important final result, but logistically challenging...and which skills relate to UME and which to GME?

Multiple Measures

• Structure and process well evaluated
• Outcomes: Capstone evaluation, after 4th year, including clinical decision-making and clinical reasoning, currently not evaluated.
• 360 degree evaluation as extern
• Program Director assessment: quantitative and qualitative
• Peer and patient assessment in residency program

Problem

• You have just started your new job as Chair of the Curri culum Committee. You believe that assessment is the basis for change. How would you determine the quality of the education provided to your students?

Specifics

What will you measure? How do you define a quality product?

Dr. Davidson explains that this is a baseline quality we need to have to support the educational effort.
Process measures - activities that are actually done; underlying theory that we will get a desired outcome; also called proxy meaasure or performance measures. Does it relate to outcome or is it related to outcome?

Outcome represents the final effects of the intervention on the student's success. As it is appropriate to determine the outcome of the educational process. He askes, what will you measure? How will you define a quality product?

Answers from some members were: satisfactory completion of a PBE; a good doctor; can they pass licensure; how are they performing during internship year; clinical skills exam should be a measure.

Dr. Meuleman thought we should be studying residency education more. Some comments were:

• The best interns are not always the best MD's 10 years after training.
• Cannot rely on program director's opinions about residents
• Dr. Harrell asked abut measuring life-long learning
• Dr. Lind indicated that maybe need to change the internship and residency

Maybe we should ask what bad outcomes are common in practicing physicians; poor interrelations with colleagues and patients; disruptive physician; lifelong learning is another area physicians have problems.

The curriculum committee is not responsible for admitting students, but they are responsibile for medical student education. Dr. Rathe suggested that we have markers or themes to evaluate (same as competencies). The question is: When do we evaluate our students, and how do we do it? It was suggested that there be some kind of evaluation at the end of the 4th year besides the questionnaire. Dr. Lind asked: what about self-assessment? How can we measure self-assessment?

Dr. Hatch stated that Capstone should be required, not voluntary. Dr. Davidson stated that unless we determine what the deficiencies are we should not change with the curriculum. The reason for our interest in these concepts is that the evaluation of outcomes will provide information about the most appropriate changes needed for the curriculum.

Summary: Whitcombe's editorial

• The most important issue in assessing the quality of education is selecting the most appropriate realistic quality measures, whether structure, process or outcome.
• We BELIEVE that improved structure increases the likelihood of improved process which increases the likelihood of improved outcomes.
• BUT the amount of information that clearly links these three measures is extremely scanty.
• THEREFORE we need to move beyond the measuring process and determine what outcomes we think are important and how to measure them.

Paul Duncan suggested that the committee needs to measure quality both at the beginning and at the end of training. He also suggested doing something with T1 measure when students come into medical school. Dr. Genuardi pointed out that we don't do assessment in year 1, and everyone doesn't come in with the same baseline level of knowledge or attitude. Dr. Normann suggests that we give feedback to the people on admissions committee about medical students. Also, give better feedback to interviewers in the process.

Next meeting will be January 14. We will talk about interdisciplinary education. Meeting adjourned at 8:53 a.m.