Curriculum Committee Meeting
December 13, 2005

Members present: Dr. Robert Watson, Wayne Bottom, Dr. Richard Davidson, Dr. Lynn Romrell, Dr. Beverly Vidauretta, Doug Arnold, Cynthia Karle, Dr. David Caro, Dr. Kyle Rarey, Dr. Sigurd Normann, Dr. Heather Harrell, Dr. Peggy Wallace, Dr. Louis Ritz, Dr. Mike Chen, Dr. Colin Sumners, Linda Butson, Dr. Robert Averbuch, Dr. John Meuleman, and Dr. Maureen Novak

Student announcements: 3rd year student, Doug Arnold reports things are going well. He reports the students are enjoying school and thinking about graduation events.

Announcements: Dr. Caro reported receiving an unexpected grant for the Simulation Center in Jacksonville. Dr. Davidson mentioned that the Curriculum Committee will visit Jacksonville in the spring. Dr. Romrell put the enormous size of the Simulation Center in Jacksonville in perspective, stating that it encompasses three times the dedicated space that UF has for med students on this campus.

Dr. Watson stated that the Carnegie Visit was a great success and states the visitors were very impressed with our program. He also announced that Paulette Hahn, who worked with Dr. Heather Harrell to develop the 4th year elective in Advanced Clinical Diagnosis, Physical Diagnosis and Clinical Decision-making, put in a grant request to a national rheumatology organization and just learned that they are giving her $50,000 per year in salary support for each of three years to develop this learning opportunity for residents and medical students. He said to “give her a pat on the back when you see her” and it was stated that she is a very dependable, very valuable educator. Dr. Watson announced that Daryl Hirsh has been chosen as the next president for the AAMC. He stated that he feels he will be a terrific president and stated that he is very academic, education oriented and is a super-nice person.

Dr. Davidson handed out the revised edition of the presentation entitled ‘Evaluating Student Performances in the College of Medicine Competency-Based Curriculum’ that was voted on and approved last month. This revised edition will be presented to the Course Directors today at noon and the Clerkship Directors when they have their next meeting. He stated that this document will provide general guidelines and noted that the changes included removing the mandated pass/fail for courses that give 50% A’s for two years. The revised edition states that the director would present the case to the course or clerkship director for feedback and to the Evaluation Subcommittee for their approval. The Evaluation Subcommittee would then make recommendations to the Curriculum Committee about altering the course if indicated. Another addition stated that if a course anticipates giving more than 50% A’s, in other words if the grades have been done and there are more than 50% A’s and that is what will be turned in, then it will be necessary to get in touch with members of the Evaluation Subcommittee before the grades are released to the students. The Evaluation Subcommittee felt that this should be a part of this document. Dr. Harrell questioned whether the grades would be deferred until the Evaluation Subcommittee met again. Dr. Rarey confirmed that it would need to be presented to the Evaluation Subcommittee as long as it was done in a timely manner. Dr. Romrell stated that there are deadlines in submitting the grades to campus; therefore the grades would have to be submitted on time to campus. He stated that for the clerkships in particular, if a grade is submitted by the deadline, there should be at least 2-3 weeks to take this action. He stresses that this cannot be a long arduous process and states that if the deadline is missed for the submission of grades, every grade has to go in as a whole grade and then it is a one by one process of putting them into a registrar’s system.

Dr. Watson stated that he received an e-mail from Dr. Duff who stated that he was unaware that our medical students participated in the Family Medicine program in Jacksonville. This issue arose because Jacksonville has discontinued its Family Medicine program. Dr. Bob Nuss, Dr. Ann Harwood and Dr. Genuardi assured Dr. Watson that if anything, it will improve the medical students experience there because they will not have to go through the residency. Dr. Watson’s big concern was if doing away with their residency program would hurt their medical student education experience, but it was promised that it wouldn’t.

Dr. Watson also stated that he formed a committee chaired by Dr. Pat Duff (Mr. Wayne Bottom is on this committee) for us to develop some policies, procedures and guidelines about criminal background checks on medical students. It was a topic at the AAMC meeting that will be further discussed and acted-upon once there is a national database, but Dr. Watson realized that actions needed to be taken now so that we would know what to do with the data when we get it and reports they are in the process of doing that. Mr. Bottom’s students participate in programs all over the state and he has already had difficulties due to each hospital having different requirements for criminal background checks. Dr. Watson feels that we are going to end up testing for substance abuse and other kinds of things as well and welcomed any input in this matter. He stated that he doesn’t think that students or residents should be asked to do something that faculty won’t, so he hopes that any policy that comes out will apply to everyone. Mr. Bottom stated that on December 20th there will be a Health Center-wide meeting of all interested parties that will deal with the contractual level of just exactly what we are going to agree to legally in varied contracts with hospitals that we deal with externally. Mr. Bottom agreed to discuss the results of that meeting at next month’s Curriculum Committee meeting.

1.Dr. Lynn Romrell presents results from NBME and subject examination scores.

Dr. Romrell distributed handouts detailing the highlights of the scores and showed the trends on what is happening. He started with the subject exam in the pre-clinical years and went through the class of 2008-2009. He stated that you can see in general that the trend for our school is up, especially the average for the subject exams over each of the years. He states that our percentile rank overall is right about the 75th percentile on those subject exams, so our students are doing really well and most of the clerkships now have scores above the 60th percentile. He stated that even though the subject exam is used as an outside measure, the teaching is not specifically towards the subject exams; the students are simply doing well on those exams. He states that the subject exams do predict well how well students will do on the USMLE. He thinks our students have an advantage when they go into that exam, having seen questions that come from basically the same question-writers. He states that when predictions are made about how well they will do, MCAT is a poor predictor, but better than overall GPA which really doesn’t predict anything. Subject exams are better than any other thing that can be looked at, although how well students do in medical school correlates well with how they are going to do on those exams, and that is reassuring that if the students work hard all along they can expect to do well on those subject exams. The correlation for the subject exams and the USMLE score is about .9 over the last three years, so when Dr. Romrell meets with a class and starts talking about registering to take the exam and what to expect, he can tell them to expect to do well. The handout is a graphic of how well the scores plot to the predictions. He took data based on how well the students would do based on the two previous years by plotting the subject exam average against the scores that are predicted they would get on the USMLE and states therefore he can tell a class that there is no reason they can’t pass that exam, it’s just a matter of how well they’ll do. The class actually did much better than was even predicted; they did extraordinarily well and for the second year in a row no one failed. He stated that if you look at the data over all of the years since the USLME was introduced, the trend nationally is that the scores go up over a number of years and now they are flat. He stated that the data shows that UF students are doing even better than predicted and he attributes that to the students working hard and doing well as they are taking the exam and that they are learning in an environment where they can excel. He shares this data with students who are applying to medical school and tells them this is a rich environment where they will learn the things that will really help them excel. He also shows the fail rates and states that our trends are wonderful. Over the last 15 years since the USMLE was introduced, UF has a 2% fail rate, which is about a quarter of the national rate, and that has lessened over the past couple of years to .8%. UF has had a 99+% pass rate over the past four years. He states that our students are really doing superb.

The data concerning subject areas, which comes out about a year after the exam is taken, shows a trend over the last four years that shows how strongly our students are doing on these exams. This data can also be used to show any weaknesses there may be in certain subject areas.

He stated that the trend for the Step 2 scores has been steady upward. It was actually predicted that the students should have done a little better, but his feeling is that when the Step 2 scores are looked at, the students are actually performing better than the Subject Exams say. He thinks that it may benefit the students to have more dedicated study-time. He states the performance is solid, but not as well as may be predicted. He pointed-out that over half of the schools now require students to pass Step 2 in order to graduate. He stated that our school had this requirement in the past, but it was discontinued due to non-compliance. He is unsure what to do about this requirement. He showed data that indicated there are 2-3 students who fail this exam each year and states that performance on this exam is not as good as Step 1. He states that some of the scores have been extremely high and the scores are good overall. It was stated that they may not studied enough for the exam or may have expected their chosen elective to be lighter than it turned out to be. Dr. Vidauretta asked if there was a process that identified high-risk students to the people that signed-off on senior electives. Dr. Romrell said there is a high correlation between a student’s scores on Step 1 and Step 2 exams. Dr. Harrell suggested that this data could be used to identify students that may need to be required to take the test earlier in case they need remediation and explain to the student that it would be best to get it behind them. Dr. Romrell agreed with Dr. Harrell and said we have many tools that will help her deal with each student individually in the 4th year oversight.

Dr. Romrell reports that the Clinical Skills Examination scores are reported only as pass or fail. Only 2 of 99 students failed this exam, which is a higher pass rate than the national average. The students felt the activities in the Harrell Center prepared them well for this exam. Dr. Vidauretta asked whether there was going to be a deadline for the students to take the exam. Dr. Romrell said there are a limited number of slots, so the students were simply required to take it some time before they graduated. It was stated that the National Board has given the students lots of flexibility in scheduling/canceling the exam. Dr. Romrell felt the costs and travel involved should motivate the student to pass this exam the first time. Dr. Harrell and Dr. Averbuch stated that the Step 2 exam has become more difficult since they took it while in medical school. Dr. Romrell stated that he and Dr. Duff have emphasized to the students over the past two years the importance of these exams. He says that the scores on Step 1 and Step 2 exams are looked at hard by residency programs directors as the equalizer or medical I.Q., which they really are not, and the students would benefit from good scores on these exams. He re-emphasizes that every subject area is above the national average and states that our trends are moving in the right direction.

Dr. Davidson stated that external measures are very important. He hopes that Dr. Romrell is right that this class’s Step 2 results are as high as he’s predicted they are, because he does not feel like the scores in the past on Step 2 and Subject Exam have been reflective of how students have performed on Step 1. He says there are only a few options to explain this-either the test is bad and is not reflecting what is taught on the clerkship, or that the students are not given enough time to study, or that the clerkships are not properly preparing the students for the exam, which is something he doesn’t believe is happening.

Doug Arnold feels like a lot of questions on the test are very vague and can go both ways. He thinks maybe the exam may not be that good. He agrees that the dramatic build-up to include the month off to study for the Step 1 exam is a factor in motivating and driving the students to do well. He feels that Step 2 is done more on an independent basis and suggests that the scores may improve if the students are given more time to study. He felt that the questions for Step 1 were more on target. Dr. Davidson stated that some subjects will obviously have more uncertainty than others. Doug stated that it is difficult trying to answer multiple choice questions that involve dealing with patients. Dr. Novak states that they don’t teach to the subject exam on pediatrics, they use it as a tool to evaluate their performance. She has been told by several students that they don’t have the time to study and that it is definitely affecting how they do, but she doesn’t feel they should need extra time off to study. Dr. Harrell stated that this is part of the transition of 3rd year and that they won’t get time off from here on out.

Dr. Normann questioned whether it would be worth considering for those that are at high risk offering an elective in which the primary goal is preparation for these exams. It was stated that that has been done before for some high-risk students. Dr. Davidson agrees with Dr. Novak and Dr. Harrell and said there may be a need to worry about how students do on the Subject Exam because if it is not a good test then it shouldn’t be made as the discriminator for their grades, which is what it ends up being in a lot of clerkships and he’s not so sure he agrees with that. Dr. Caro stated that we should not be expected to teach every student every single topic and then require training and that there is some room for them to go out and get information on their own and learning it on their own. Dr. Davidson stated that what bothers him about the whole arrangement is if the job is to develop a curriculum and the endpoint is to have the best prepared PGY-1, how is it ascertained whether that’s being done? He stated that the Step and Clinical Skills exams could be looked at, for instance, which test the history, physical and communication skills. He feels that what is never assessed independently, and what he considers to be the most important aspect of training, is clinical decision-making, the things that Dr. Hahn will hopefully develop in her 4th year course. He stated that the last time the students had any formal examination is at the end of their 3rd year. He stated students are never measured after they’ve done their required SubI during which they probably learn as much as they do in any one month the entire four years before that. He stated that there is no idea other than the feedback that is received from program directors about how really well prepared our students are when they walk out the door. Several months were spent last year developing a test that could be given to the 4th year students, not graded, when they walked out the door. The test was put aside due to the inability to determine consistent outcomes. He agreed that external measures are great, that LCME will look at them and they’re very impressive, but he is not sure that how a student performed on Step 1 or Step 2 should be a final determinant when they walk out the door.

Dr. Harrell stated there is a discussion about having another standardized exam at the end of the medicine SubI, which she thinks is a huge mistake because their knowledge doesn’t need to be tested yet again. She states that her department also questions how they are assessing the students. She states that institutions are at different levels of how their curriculum is developed for the SubI, while trying to maintain learning by doing, yet there are probably key things that need to be taught specifically in the SubI that could be made more explicit and there needs to be a balance. She thinks that is the time those issues need to be addressed.

Dr. Davidson feels that oral examinations, which have been discarded by almost everyone, are actually a pretty good way to do this. He states that our students are tested four times on history and physical and they know how to do them, but what is not known is that if they are given a history and physical how well they would put that together into diagnosis and treatment. He stated that the only way that could be done is to figure out what a student is thinking, which can’t be done during a PBE. He stated the only way to be able to tell what they’re thinking is for them to have the opportunity to say what they’re thinking and that is the strength of an oral examination. A student would be provided with information and would then respond to it. Dr. Davidson mentioned that there was going to be a study where students took their oral exams in pediatrics and then retook the oral exam in their 4th year with the same examiner to see what changed about their performance on the same questions. He agrees that there are all sorts of problems with oral exams, but relates that it would be the best way to determine what students are thinking.

Dr. Averbuch stated that they still administer oral exams and that the way they preserved validity and reliability is that he administers all of the exams. He says it is difficult sitting for 6-7 hours watching students interview, but it works. He does it with the training director of his program and it works pretty well. He agreed that it is a great way of assessing someone’s decision-making. Doug Arnold stated that he had taken the exam with Dr. Averbuch and that he took a lot away with him from that exam and felt it was worth a lot. He feels that an oral exam gives him the chance to fairly demonstrate what he knows and how he thinks, particularly when at the end there can be a discussion that goes back and forth where there is a chance to explain your thought process. He says he feels that he’s always taken something back with him after taking an oral examination and has even learned new things in the discussions afterwards.

Dr. Davidson said the reason oral exams have been dropped from certification from many areas is because of inconsistencies in how they are graded. He says if it is not graded and the purpose of the oral exam is to find out how somebody thinks and there is not a grade associated with it, that that would be a different kind of setting. He thinks that holds great promise if someone could do that. Dr. Harrell said there is a lot of data out there that is already known that might provide insight without having to set up another program. The information is already there and could simply be thought about it in a different way. For example, observation of SP cases seen during the PBE’s could provide some insight into how students are thinking.

Dr. Romrell mentioned that in Canada there is also a physician observer who is present when their exams are administered. Dr. Davidson pointed out that there are a lot of research opportunities available relating to this discussion.

The meeting was adjourned at 8:45am.