Dr. Rarey welcomed the course and clerkship directors, and medical students. He stated that we (faculty) wanted a positive dialogue to learn how the evaluation of student performance is working, as well as areas of improvement. We (faculty) want to be proactive.

In addition to designing and implementing, the curriculum committee is required by the LCME to have an evaluation process in place. Therefore, we want to focus on 3^rd year as well as 2^nd year curriculum, and examine the evaluative tools. Look at instruments we have in place, identify areas of strength and areas that need improvement. He also stated that the course and clerkship directors take the data generated from this session, review it, and make suggestions and/or recommendations in June at a summary retreat. We will learn what improvements will be needed to be incorporated to evaluate students, whether at mid-clerkship or at the end of a clerkship.

Why the retreat? We want to be able to evaluate student performances, i.e. student acquisition of knowledge and skills, as well as demonstration of behavior in our competency-based curriculum.

Section B is the UFCOM student evaluation system that lists seven objectives that deal with goals to measure academic achievement and competency development. In reference to the retreat handout, Section C represents the competency-based evaluation systems that are in place in the 1^st and 2^nd year. Section D is a collection of the evaluation forms used in the 3^rd-year clerkships.

It was noted that the clerkships have moved forward over this past academic year in recording student performance for selected competencies. For each student, they (clerkships) have a record of academic performance and competency performance.

Dr. Rarey asked for any questions about the origin or how we evolved at this point in time.

Dr. Rooks: I want to make a practical statement in reality what this translates into. In reality, what we have are three evaluation modalities and three methods that apply to that. We still have grades, but they do not accurately access student development. Therefore, we superimposed a system to track competencies independent of grades. This was just in place this year. We're saying if you do poorly, for example, in unprofessional behavior, we will know that's a problem area for you. Competency-based tracking, if you will, is independent of grades. The third system is passing the boards (another way of evaluating student performance), which is not part of this discussion today. (Central issues that we will get into when we develop scale of competent, or non-competent.) It was really intended to be a competency statement, not a grading scale. We are aware that problems developed.

Aaron Wohl: In third year, if it's a competency-based evaluation, do attendings give me a letter grade and a competency evaluation?

Dr. Rooks: Yes, which is the way it was intended to work. We want attendings to make a competency statement, nothing to do with a grade per se.

Student: In all fairness, medical students look at the competency scale as a five-point grading scale.

Dr. Kellner: Part of this issue is growing pains, anticipated with a changing system and it will take time to filter down with reinforcement. The idea is to get away from letter grades; they carry emotional baggage with them. We want to get to a description of competency. The original system that we developed did away with grades entirely. Students have to understand that in all courses and clerkships, the course and clerkship directors assign the final grade based on feedback from the people who have worked with you on a daily basis on practicals, orals, etc. It needs a maturation process.

Dr. Rooks stated the ultimate goal is to provide real feedback.

Kelly Shimp: Prior to this year, how were the 3^rd-year evaluations just a letter grade or were they broken down at all?

Dr. Nadeau: Broken down for more than 10 years, history and physical, professionalism, clinical competence, writeups. Although they are not the same with every clerkship, but they have the same fundamental things. What is new is to try to lend meaning to each rating. It's been traditional 1-5 on every category.

Dr. Kellner: Every clerkship had different parameters factored into a final grade. Some clerkships were more detailed than others. You didn't get this breakdown.

Dr. Rooks: The old system was not uniform. We thought we needed a standard method of evaluation that everyone uses. The competencies came from those forms, and what has been looked at for years.

Dr. Rarey: [In reference to the retreat notebook] This is the first time we have had one document that has all clerkship evaluation forms to review collectively.

Dr. Kellner: I think we need to identify whether we are looking at forms that faculty and evaluators use to evaluate students and then hand in to the clerkship director, or forms which the students receive from the clerkship director.

Dr. Ledbetter: No, they don't get the form, but a summary of the form.

Acquinonette: The form that surgery gives out has been pretty helpful. They send it to the different attendings, make notes through the first three weeks of the clerkship, and then midway they call you in for pertinent points and things you can improve on. They went through each category and what you can improve on. I thought it was excellent feedback. I appreciated it as a student and the comments I received from students in my group said that it was helpful to round out the clerkship in a positive manner.

John Lovejoy: This brings up an important point of communication. My group says we don't get enough written feedback. Students need direct written comments back on how did they improve. Mid-term is simply if you are doing satisfactory or if you are competent. Unless you make a concerted effort to actually get involved, you don't get any real assessment or foundation to improve upon.

Dr. Lind: An educational responsibility we have is that faculty do not know how to give good constructive feedback, which is a problem as well.

Dr. Rarey: We may need appropriate faculty development about human relations and communications.

Erin Dunbar: During my rotation, I gave my mid-term evaluator my perspective of what I'm doing. Am I accurate in my weaknesses and strengths. That way I was guaranteed to have some kind of feedback. This also takes the pressure off of the evaluator.

Dr. Lind: It also emphasizes self-assessment on the part of the student. Great idea, we're going to steal it from you.

Dr. Kellner: Scott hit the nail on the head. Feedback is the number one toughest thing everyone has to deal with. Such objectives are listed in section B, and none of it has happened.

Student: Some give feedback, some don't do it.

Dr. Watson: We've had more than one faculty development session, bringing in outside experts. The issue: Giving positive feedback is easy; giving negative feedback is tough.

Student: Students need mid-term evaluations listed for the student to see. We need to know three things: How is the student doing? Where are strong points? How can the student improve? Sort of know what they (attendings) want to address.

Dr. Stair: [Emergency Medicine clerkship director] One thing the competency form, as it stands, lends itself to, is a grading system. I can envision someone doing well on the competency-based evaluation and doing poorly on a letter grade, which will determine their class ranking. It is important to stay out of the pitfall of these being translated into letter grades.

Kelly Shimp: In some clerkships, you only get comments. I had several attendings sit down with me and fill out the evaluation with me sitting there. Sometimes verbal feedback is easier to give. Encourage faculty to sit down and give verbal feedback in a formal way. That was done at the half-way point of psychiatry rotation.

Dr. Rooks: That's the ideal scenario you described. The problem we're getting into is other attendings want to give the student a B and then want to know what box competency to check which is meaningless. We have to shift from that.

Ankit: If these are the competencies we are trying to master, why not call them the letter grades. Why not ABC's as feedback? That's what I'm supposed to master.

Dr. Kellner: Again, it gets back to the "emotional baggage." A considerable amount of effort went into this form and we need to get away from all the connotations of letter grades. Time will slowly help us get away from letter grades.

Student: Pediatrics, in particular, is important at the end in that we get something back that has these different levels on it as to how we did in these different areas.

Kelly Shimp: I know we are trying to get away from "ABC." No matter how you word it, if you are not at the top, there's always something better. I agree, it's better stated if what you are trying to develop are better physicians.

Dr. Moseley: There seems a fair amount of confusion comes from the five-point competency scale because it is the same as the grading scale. I would argue that you're either competent or not competent, maybe marginally competent, but that's it.

Dr. Rarey: It looks like a five-point scale, your point is well taken.

Dr. Small: Another possible scale is characteristics of a 2^nd-year student, 3^rd-year student, etc.

John Lovejoy: The most important thing that comes up with a final grade scheme is that we are given fair chance to achieve that level. One of the biggest factors is knowing how we are doing as we go along; students are not getting enough feedback to improve in a rotation. Six weeks after our rotation, we were supposed to get grades; some didn't get them because faculty didn't fill them out. We got one or two sentences. Students would feel like the descriptors were very clear and throughout the rotation, the faculty go through these with students to discuss and share how we are doing in the rotation.

Dr. Rarey: It makes no difference what the header is, but how the description for a selected competency reads.

Gina: Comparison of level of 1^st year: The danger is they didn't get exceptional at a resident level of performance. How can we be compared to the level of a resident (not fair)? Some descriptions sound unbelievable to achieve.

Aaron Wohl: We shouldn't be expected to act as interns.

Scott Howard: In the military we do officer reports all the time. We also have a five-point scale. They (evaluators) have a histogram next to the scale; exemplary means different things to different people.

Dr. Kellner: This is a recommendation by AAMC and done here that all transcripts be accompanied by a histogram that shows distribution of grades; the histogram idea is a great idea.

Nadeau: This is not working (grading scale); some of the students had anxiety that top categories were basically equivalent of walking on water; by standards of neurology faculty, a lot of you can walk on water. This is our 10th month in use (used as "ABCD"). We tried a histogram approach; middle category was explicitly the average UF COM student. We used that system for three years.

Chris Hurt: Suggestion: we evaluate our professors, and we rank him/her on competency; it could be turned around and applied to us. We do it online, it's easy. One advantage is that it pitches it off the backs of the course directors back to residents and attendings; the people that have the most contact with yourate rates you. Each clerkship have their own descriptors of criteria.

Dr. Rarey: Regarding Dr. Nadeau's comments, we are trying to find a system that evaluates the performance in a uniform way.

Dr. Meuleman maintains a four-point scale.

Acquinonette: Overall they (Medicine evaluators) did an excellent job. They used descriptors, percentages. They also listed the competencies and what they were looking for; what you should exemplify as a 3^rd-year student. It put my mind at ease knowing what they are looking for. Then they got together as a group with the clerkship director and talked about it. They give you a breakdown how they divided the grade. They gave a great presentation.

Kelly: At the beginning of a clerkship, knowing your expectations make you goal oriented; then you go through the six weeks not knowing where you stand.

Aaron Wohl: Attendings see very little of us. Is this fact taken into consideration with these forms? Three separate evaluations allow the course director to see the skew of opinions. How can an attending evaluate me when they have never seen me?

Dr. Small: Dr. Meuleman has tried to even out the evaluation.

Dr. Meuleman: It's a great idea to have a third party meet with the whole team; having facilitators even out a student's performance for those students who are not with us. The feedback to the student is coming from the group.

Dr. Kellner: This system works from our (faculty) side. We get negative feedback.

Sonya Walker: You can be the type of student getting A's; and then once a clerkship rates you honestly, and you don't get that A, you get upset. I like the way Medicine has their evaluation done and the third party is responsible for putting information on the sheet. Having that third party there and people giving comments is helpful.

Rodney: I liked what Dr. Meuleman has done with the form; putting anchors in. Competence becomes a four-letter word. You see it, and you see a C.

Dr. Rarey moves on to section C of the retreat notebook.

Dr. Koroly's course. She identified and weighted the competencies for the biochemistry course.

Dr. Rooks chose three competencies for his course, and rated students on a five-point scale. Dr. Rooks stated his course is pass/fail.

Scott Howard: My first reaction was not to look at the competency (maybe my military background). For me, the comments were the most important.

Dr. Ritz: There's a marked difference in terms of motivation.

Dr. Kellner: We got rid of the personality stuff when we made the competencies. They are not what make a good student; they are what make a good physician.

Dr. Rarey to Dr. Kellner: Do the comments you submitted have a mix of competency and personality? Dr. Kellner's response was "yes."

Chris Hurt: The OB/GYN evaluation is meant to be a parallel system to the grades. These sheets are not parallel, but integrated. Is it parallel or isn't it, and why? Do you need to have two different systems?

Dr. Ledbetter: They are not separate at all. The ideal way is to separate them as Dr. Rooks stated.

Aaron Wohl: Dr. Nadeau said this isn't working. If it has been integrated into the system. I think the students are actually hurting. In an objective course, you get 90-100 and you can get an A.

Dr. Rooks: Both systems highlight a real important need to evaluate.

Dr. Nadeau: Let me clarify what I meant by "the system's not working." Do I think students are getting the grades they deserve, yes. We give serious consideration to it. In terms of fairness and accuracy, the current system is doing fine. What I mean by not working is--are faculty looking at things (descriptors that most fit this student)? No, they are not. In another sense, at the end of this year about 50% will receive A's and 50% B's. There's a constant tendency to grade inflation and if you're not always watching, 2/3 or 3/4 of the students are given A's, the rest B's. We have not succeeded in the bugaboo of grade inflation.

Dr. Rarey: We're sitting in the room wrestling with an issue that's been around for some time so that we can have a better evaluation system next year. It's supposed to be dynamic and evolve so we give the student better feedback.

Dr. Lind: I have a sense we put too much emphasis on evaluation and the process. I'm from Canada, and this country (US) is test, test, test, evaluate, evaluate, evaluate. It takes the fun out of learning.

[As session ends, Dr. Rarey asked for the group to please give feedback on sheets in their booklet.]

Dr. Rarey acknowledged Dr. Goldfeder who wanted to thank two students who showed emotional maturity: Scott Howard and Erin Dunbar. He explained that most comments [from students] focus on the fact that they're not getting enough feedback. Students are telling us what they need: dedicated time by the staff to spend with them. This is a great medical school and can only be better if we actually define to the chairpersons that it is of the highest value that the appropriate time and resources be given to the clerkship director to meet frequently and often with the students.

Dr. Romrell: Every student here has given great insight on how to improve the system and we appreciate all faculty coming as well.

1. The UF/PIMS retreat is June 2 here at UF and on the final day of

exams.

2. Clerkship Directors retreat is June 1.

3. The White Coat Ceremony is Saturday, July 8 and all faculty are welcome to attend.

Dr. Rooks: I'd like to ask the students:

What do you really want out of an evaluation system?

What are your priorities?

What information do you need, and what is it for?

What do grades do for you?

If grades can't relate to the attributes that we see and want in a physician, then there's something wrong with the way we give grades. I think they do relate.

Dr. Rooks: You've defined a real dilemma. What motivation is there for you to be here to train to be a physician? We still measure our success by the grades we have.

Dr. Small: You're not going to get grades in your residency. They don't mean anything then.

Dr. Stair: The reality still is from Emergency Medicine specialty. Students are always going to be focused on that final grade. As directors, we must be clear in terms of delineation. What makes up my final grade? When you look at residency spots, it matters.

Dr. Small: What enables people to get residencies who are in the middle-third of the class is the peer evaluation, which is given at the end of 3^rd year or beginning of 4^th year. Peers evaluate their colleagues. The top 15-20 students are at the top. Smiley Hill has been pleased with this system for years. The top 2-3 students as judged by their peers carry more weight in their eyes. There is no concept of why we are the only school doing it. It is something that makes the tyranny of grades a little less severe.

Dr. Rarey thanked the students and the course/clerkship directors for their input. The Retreat adjourned 7:00 p.m.