Curriculum Committee Meeting Minutes 2/28/06
Present: Dr. Andrew Kaunitz, Dr. Richard Davidson, Mr. Wayne Bottom, Dr. Robert Watson, Marie Coleman, Dr. Kyle Rarey, Doug Arnold, Beth Layton, Dr. Margaret Duerson, Omayra Marrero, Dr. Lynn Romrell, Cynthia Karle, Dr. Margaret Wallace, Dr. Colin Sumners, Dr. Heather Harrell, Dr. Sigurd Normann, Dr. Juan Cendan, Dr. Rebecca Pauly, Dr. Maureen Novak, Dr. Louis Ritz, Dr. Mike Chen, Dr. Robert Averbuch, Dr. Robert Hatch, Dr. John Meuleman, Dr. William Winter, Dr. Cari Hernandez
Student Announcements: Omayra Marrero, MSI, stated that first year students had a great Family and Friends Day. Doug Arnold, MSIII, stated that more of the third year students have made up their minds as to what they want to do. He spoke for the second year students and said they are happy to be done with pathology. He stated that the LCME survey is going to be up by the end of the week and that it looks good and is moving right along.
Announcements:
Dr. Romrell stated that he knows the students are looking forward to Spring Break, which is March 11th. He stated they are working with second year class and are getting ready for the Step Exam. He stated that the data for that class indicates that they will do very well on the Step Exam. They are moving along and are anxious to get ready for the clerkships.
Dr. Rarey stated that part of the AAMC is the RIME section (Research in Medical Education). There is a call for nominations for individuals who would like to be considered becoming a member of the RIME Planning Committee and the deadline for nominations is this week. There are three openings available and Dr. Rarey suggested that those interested should talk to Dr. Davidson who is a member of RIME. He stated that the University of Florida actually operates the RIME website. Cynthia Karle has been involved with RIME for a couple of years and the nominations process, etc. goes through Cynthia. Dr. Davidson stated he will be glad to talk to anyone about it and that it has been a great experience. He stated if you are interested in research in medical education it is a great way to learn a lot. There are two non-M.D. positions and one M.D. position available. Dr. Davidson stated that committee is comprised of four medical doctors and four non-medical doctors.
Dr. Watson announced a one-man play called Ball, which will be part of the Chapman Society Induction on this Thursday evening and he invited everyone to come. He also announced the all-day 50th Anniversary event recognizing alumni on March 11th and would like as many faculty to attend as possible.
Mr. Wayne Bottom was very pleased to announce that in the PA program there was 100% pass grade in the class of 2006 and for the first time in the history of the school we were in the 87th percentile on scores and 16 students were in the 99th percentile, which is the best performance we have ever had.
Dr. Kaunitz stated that in Jacksonville there are a lot of active searches for chairs of departments; they are transitioning from associates to actual chairs. The new chairs for radiology and anesthesiology just started recently and in terms of internal medicine, a gentleman from St. Louis has accepted the position and will be starting soon. The physical plant housing the proton beam accelerator is looking complete and he thinks they are scheduled to do the first procedure sometime this summer. He stated that when you talk to radiation oncologists who are not affiliated one way or another with UF or Jacksonville, you get very mixed perspectives on the value of protons.
Dr. Romrell stated that June 1st is an important day because the course and clerkship directors and program directors are going to Jacksonville to have the annual retreat of clerkship directors, and a major emphasis will be placed on simulation. We will talk about simulation and how it will be used in our educational programs and also get to visit the proton beams. Dr. Watson stated that Dr. Burns, when he was the dean, stated that the Curriculum Committee is responsible for methods of teaching and since standardized patients and simulators are going to be increasingly important, it will be good to go see the simulators in Jacksonville. An announcement will be sent out and spots are being reserved.
Dr. Winter stated that the Course Directors Committee had an enthusiastic discussion about the recommendations put forward regarding some guidelines as to the percentage of A’s that should usually be given in a course that is designed to differentiate students. The committee did not come to consensus, but they certainly came to discussion. Dr. Hatch stated that things are being tuned up for the upcoming LCME visit and planning the Jacksonville joint retreat.
Dr. Davidson and Dr. Romrell have submitted the educational standards to all the course directors, which will be discussed at upcoming meetings. Dr. Meuleman will present what will be going on in the new Geriatrics Clerkship at next month’s meeting. We will also have a discussion about the competency forms.
1. Dr. Rarey will present the annual course and clerkship evaluations by the Evaluation Subcommittee.
The members of the Evaluation Subcommittee evaluated 35 courses and clerkships and 3 SubIs in our 4-year training program. A handout was distributed during the meeting that included highlights of the document. The entire document can be found online at: http://www.med.ufl.edu/oea/eval/minutes/webpage/webpage/2005_toc_members.pdf.
Dr. Rarey gave an overview and foundation of the role of the Evaluation Subcommittee, which is comprised of 16 members. The membership is composed of course directors, clerkship directors, student representatives, as well as members of the staff of the Office of Medical Education and the Office of Program Evaluation and Faculty Development. The idea of the Evaluation Subcommittee was that it would be composed of members of directors from both courses and clerkships so there would be somewhat of a peer review process involved. The entire educational program is under the office of Dean Tisher who charged our Senior Associate Dean, Dr. Bob Watson, with the management of that program. He oversees the Curriculum Committee which is led by Dr. Rick Davidson who is empowered to plan or design the four-year curriculum that is implemented through the Office of Medical Education, which is Dr. Lynn Romrell’s area. Until 1997 there was not a way for the Curriculum Committee to be involved in evaluating the curriculum at periodic intervals, therefore Dr. Watson created the Office of Program Evaluation and the Evaluation Subcommittee. The Curriculum Committee designs and implements the curriculum through the Office of Medical Education and has the curriculum assessed by the Evaluation Subcommittee. The feedback from the Evaluation Subcommittee comes right back to the Curriculum Committee, so they are essentially the assessment arm of the Curriculum Committee. The members of the Evaluation Subcommittee assess the curriculum, give feedback to the Curriculum Committee and deans, and provide needs assessments for the Course and Clerkships Directors on their written reports that identify resource needs for the upcoming academic year to enhance their particular course or clerkship. The needs assessments are then forwarded to Dr. Romrell. The Evaluation Subcommittee works in concert with the chair of the Curriculum Committee based upon several aspects of the program. Dr. Rarey feels it is important for the members to understand the format in which the members of this Evaluation Subcommittee have worked. Included in the online document is a brief introduction of why they look at the evaluations of the courses and clerkships; they want to look at the quality of components of aspects of the courses and clerkships in terms of looking at the instruments of the syllabus and the written report. The annual report is composed of a review of the course and clerkship syllabi as well as the director’s reports and also the data from the Office of Medical Education. There is a format or guideline for the syllabi by which we want continuity across all four years of our educational program. Each course and clerkship director is to put their syllabi online and it is to be formatted with regards to an overview of measurable learning objectives, as well as identify student performance criteria. The system for giving formative feedback is emphasized several times. The Director’s Report should identify specific changes that may have occurred within a course or clerkship during an academic year. Once the data with regards to student performances is obtained, there is a need to identify the types of evaluations used to assess student’s knowledge and skills, and identify how students perform either by internal or external instruments towards learning the course objectives, etc. Dr. Rarey encourages the members to review the online information individually. The second aspect of the written report is the feedback of the students. The students identify the quality of the formative feedback. They describe their interpretation in reference to what was written in the course syllabus about formative feedback and what actually occurred. He wanted to show the various parameters and pointed-out that this is a Curriculum Committee policy. Dr. Rarey also discussed the internal policies with regards to the LCME guidelines and stated there are 48 standards that relate to our educational program. He directed the members to ED30 Standard, which stated the directors of all courses and clerkships must design and implement a system of formative and summative evaluation of student achievement in each course or clerkship, which is why the policy is related towards that.
ED33 Standard states there must be an integrated institutional responsibility of the overall design, management and evaluation of a coherent and coordinated curriculum, which is the Curriculum Committee. We are empowered as members of the Curriculum Committee and charged with this responsibility. The Curriculum Committee is to be involved in the design and implementation of the curriculum through the Evaluation Subcommittee’s assessments, as well as being involved in the evaluation and quality of our different units and overall program leading to the MD degree. One particular area involves the evaluation of program effectiveness by looking at outcome analysis using national norms, marking the content and workload, the identifications of omissions and unwanted redundancies, reviewing the stated objectives of individual courses and clerkships, the methods of pedagogy and student evaluations to assure congruence with our institutional educational objectives.
ED35 states the objectives, content and pedagogy of each segment of the curriculum for courses and clerkships, as well as the curriculum as a whole, must be subject to periodic review and revision by the faculty. The Evaluation Subcommittee is providing data this morning of their assessments and recommendations and the role of the Curriculum Committee members is to evaluate that data and put some value in it.
ED46 states that in order to guide program improvement, medical schools must evaluate the effectiveness of the educational program by documenting to the extent which the objectives have been met.
ED47 states that in assessing the program quality, schools must consider student evaluations of their courses and teachers, and an appropriate variety of outcome measures. Our Curriculum Committee policy is to look at the infrastructure of courses and clerkships, look at how students evaluate the overall courses and clerkships, the overall instruction, the overall management and organization of the course or clerkship and then look at their performance.
He pointed out that one year from now we will have passed our site visit, which is February 18th -21st, 2007. The document containing the summative assessment of each course and clerkship is divided into required courses from the first through the fourth year and for the first time also includes the three Sub Internships that are in the fourth year. Electives have not been factored in. Included in the handouts distributed today are composites of the first through fourth year courses based on certain parameters, composite student evaluations and workload reports. It was determined through Dr. Kellner’s office that we would determine whether there were any questions on the Graduation Questionnaire that relate to the courses and clerkships about the quality of learning the students had within those and use that as some way to incorporate with our overall assessments and these results are listed as well. Dr. Romrell’s office provided the data relating to student performances on external measures. The Subcommittee’s assessments are displayed in graph form with gold representing the highest performance, silver representing good performance, bronze representing satisfactory performance, and red representing unsatisfactory performance. This chart is used as a summary as to how the Evaluation Subcommittee came to some recommendations. Dr. Rarey acknowledged that his Program Assistant, Marie Coleman, is very adept at managing the data and findings that the Evaluation Subcommittee has come up with and presenting it in some kind of format that we can have a discussion about.
Dr. Rarey stated that there are 12 courses in the first year and there are three sets of data that go into creating the composites to include scoring of the course syllabi, student evaluations of the course, as well as assessment of the director’s analysis on either written report. There are a number of courses who have good infrastructure that are evaluated highly by students. There are a number of courses over 5 academic years that are above 75th percentile in terms of their infrastructure and student evaluation and there are several courses that are below that line for one reason or another. He stated that this Evaluation Subcommittee is probably the best, hardest working group that he’s aware of. The committee meets twice monthly and works to ensure the scoring is as valid and as reliable as possible. They have met with various course directors to discuss the results as well.
Presentations of the Subcommittee’s work were presented to the committee. These presentations included genetics, in which the student evaluations were slightly lower than in the past as was the evaluation of the course report, and Interdisciplinary Family Health, where the student evaluations were lower than would have been expected by the concurrent evaluations by the faculty.
Dr. Meuleman inquired whether Genetics had changed the syllabus, but Dr. Wallace stated it has been the same over the last four years. She stated that if anything has changed it may be that the students come in with different expectations than they did from the year before. She stated that one example is the midterm. They didn’t traditionally have a session after the midterm to review the answers. There was an opportunity for the students to come back in and go over the answers if they chose to do so, but only about 5-8 students showed up. The online system of reviewing was not put in place because there wasn’t a demand and she thinks some of the students may have been expecting that.
Dr. Meuleman supported Dr. Harrell’s statement that qualitative evaluation is really where you’re going to get improvement where you give people specifics, not numerical scores. Dr. Rarey stated that each director receives specific comments/feedback on their syllabus. Dr. Davidson said he has heard several people state that the Evaluation Subcommittee is evaluating the syllabus and not the course. He states that the Evaluation Subcommittee does what they can do given their requirements and the one thing that he disagrees with is that the Evaluation Subcommittee does not evaluate content. Content is evaluated by external measures only. The Evaluation Subcommittee doesn’t get into a course and say “Are they teaching this microbiological technique or about this new bacteria?” He says they do not do that. He doesn’t know of any way they can do that, but until there is some way to assess the content as well as these other measures, he doesn’t think you can really say you’re evaluating the entire course. He thinks what they are evaluating is what they have the ability to evaluate. He stated that for years they have talked about wanting to hire somebody, an experienced very smart clinician that can go through med school and sit in the courses and comment on the relevance of the content because that is a really important question and yet it is beyond the scope of the Evaluation Subcommittee to do that.
Dr. Rarey stated that when the student outcome data is assessed, the Evaluation Subcommittee looks at how the students performed on the Subject Exams and NBME as an indicator, as well as Step 1. He states that it is interesting because, for instance, on Step 1 in the first and second years, all of our courses are above the national mean in terms of standard deviation. One of the things you see in Genetics, for example, is that the students performance overall is .3 SD above average. These are scores from June 2003 and our best score ever on the national boards are not reflected in that report. Dr. Romrell reports that the trend will continue to go up because it takes about a year for the national board to analyze it and we’ll see last year’s scores in about two months. He stated that we are doing well in all subject areas and that is an excellent external measure. Dr. Davidson says that it works pretty well for those courses that do have those external measures, but courses such as EPC or IFH and other courses that don’t, it’s much more difficult to measure.
Dr. Rarey stated that there were several first year courses that in terms of their infrastructure, for example EPC1 and EPC2 that, as seen by the trends, that have had a positive change in terms of the infrastructure looking at compliance within the parameters. He stated that there are courses in the first and second years in which the directors have done an admirable job in terms of addressing the various parameters. He states there are 12 courses in first year and that by and large all the courses are in compliance. It is the Evaluation Subcommittee’s recommendation that there are several there that will need to monitored and interacted with to enhance the program. It was stated that courses such as EPC, which only have three measures, are only averaged with those three measures, the fourth one is dropped. Dr. Romrell stated that maybe we should find a way to use the Clerkship Directors as an external measure for something like EPC, although Dr. Pauly stated that because of overlap it would be hard to allocate that for a particular course.
Dr. Davidson stated that they have gone through educational standards and the number of educational standards for which the only responses are EPC, Primary Care Preceptorship or IFH in terms of its content deemed educational standards by the LCME is remarkable. There are, for instance, no standards relating to the quality of the anatomy assessment, but there are many related to content dealing with those courses and possibly the LCME report could be considered an external measure for those courses that don’t have other measures.
Dr. Rarey discussed a question from the 2005 Graduation Questionnaire related to how well the students felt the courses prepared them for the clinical clerkships and electives. Genetics, for example, was rated higher than the national mean of the other medical schools in the United States. Dr. Kaunitz stated that it was fascinating that on the one hand compared to other courses in the first year, Genetics may be somewhat of an outlier. On the other hand when you look at performance on the boards they received a “gold medal”, but then when you look nationwide there seems to be this widespread negative perception of Genetics looking at many medical schools, although it is less so here. Dr. Watson stated that when he presents this to the Executive Committee over all courses and all clerkships, he specifically makes that point. For example, he remembers that here out of all our clerkships, Pediatrics was rated near the top and nationwide it was rated near the bottom because Pediatrics everywhere is very well done. Dr. Kaunitz stated that when you first look at it there may have been real concerns about Genetics, but when you give it a broader perspective you realize Genetics is really doing a lot of things in a stellar fashion. Dr. Rarey emphasized that the Evaluation Subcommittee makes sure all of the data is seen and it is the role of the Curriculum Committee to evaluate what the data means. Dr. Romrell stated that the seniors may be reflecting back and may have a very different perception of how valuable things were. He thinks that seniors are also seeing what is going to be the knowledge-base that you have and he thinks they recognize that Genetics is unequivocal as we move toward a new way to treat and identify disease.
Dr. Normann stated that there is another way to interpret that data, which is to not look at the national norms, but to look at where they scored at your individual school. He states that you need to ask why that particular course is the lowest of them all and focus your attention on where a weakness might potentially be by looking at that data and then going into more detail about what is in that course that could be enhanced to make it more relevant for the clinical years.
Dr. Rarey stated that he would discuss second year’s results at next month’s meeting. He states that there is less noise on the graph in the second year and even less noise in the third year. He thinks we should look at individuals, not the overview. He discussed the color coded performance scale (gold, silver, bronze and red). To receive gold a course had to have 90%; if you are non-compliant you had below 66-67%. They tried to make it a reasonable range and wanted to distinguish the truly exemplary courses and some way to identify the courses that will need to be helped. He asked the members to contact either he or Dr. Ritz with any comments in regards to the formatted assessments.
Dr. Davidson invited the members to share any thoughts or comments during the remainder of the meeting.
Beth Layton announced that Linda Butson is on extended medical leave. Therefore, she stated that anyone who needs assistance in the library can contact Library Administration and ask for a supervisor. They are doing weeding in the library collection. She states the library has 15 year old review books and that they’re getting rid of things they should have gotten rid of many years ago. She said there is no cause for alarm and that librarians are very conservative and are erring on the side of caution in the information they will dispose of. She states that the library is part of 50th Anniversary and there will be tours of the library on that Saturday.
Dr. Watson stated that he knew when he created the Office of Program Evaluation and Faculty Development that he was setting up whoever led this office to be in an unusual position of somehow or other being seen as an enemy to course and clerkship directors. He thinks Dr. Rarey and the Evaluation Subcommittee have done a fantastic job in providing what he perceives to be an enormous missing link. The Curriculum Committee in the past would make all kinds of decisions and then two or three years later find out the decision hadn’t even been implemented, much less the Curriculum Committee knowing whether it was going well or not. He thanked them for filling a tremendous gap.
Dr. Harrell stated that she does agree that there is absolute value in reflecting back on your course, your objectives and how you are evaluated and enforcing doing that. The fact that in general the courses are doing so well makes it seem like we are now at a point where there is some way we can enhance this to make it seem more relevant to the course directors. There is another sense that somehow how well your course is doing really is being evaluated, but then there are these inconsistencies that we are picking up that create frustration. She states that it doesn’t seem like there is a mechanism built-in that before we come to this situation and receive your course report that you can then discuss “Well I don’t understand where you coming from with this or here’s what we’re doing or what standard is this that we’re using” and provide some justification. She feels there are times where the reports have not changed from year to year but the evaluations received are quite different. She agrees that we should get insight from the students, but states that we have to be very careful that the students are not driving this evaluation process because, similar to the FCAT discussion that is going on, we can all teach to the Shelf Exam, there is no doubt about that. Our external measures are very content oriented, in particular EPC and IFH courses and in the clerkships that may not be what our primary goal is and it seems like we are at a point now where we can take it to the next level H of where there may be a little less frustration with the course and clerkship directors if they feel like there is a little more internal validity.
Dr. Watson questioned how you go from the beginning of some form of accountability and some way to measure and then really start providing useful formative feedback whether you’re talking about a student or whether you’re talking about faculty and that is the challenge to take it to the next level. He states that in a broader sense he actually thinks that this committee, which has been together and has discussed many issues, is in the position and has the data to start focusing on the developmental process of medical student education instead of looking at individual courses and clerkships as much, which is vitally important. He feels we should look at the broader idea starting on day one until the day they walk across the stage and states we have the right developmental process not only in gaining a knowledge base, but understanding the art of medicine and understanding the obligation to society because he thinks that is where it is going to be. He states that is why it’s so important to go look at the Simulator Center and know more about this and standardized patients. He questions where in the first two years you can use simulation to help things become more clinically relevant and easier to remember and use. He thinks that if this committee doesn’t do it then there is no other committee that actually can do it. He would love to see us start in that direction over time because it is a very effective committee of carefully chosen people. He stated that this is just a feedback session and you don’t want the Curriculum Committee to start thinking that this is what it’s all about. We want to look down the longer road. He doesn’t know how the Curriculum Committee deals with not only the developmental process, but the environment because we are stuck in very difficult environment as far as he is concerned in terms of developing the humane skills of our students and our residents because the healthcare system is not friendliest place in the world.
Dr. Watson mentioned that two of our alumni have gotten into legal troubles and he wonders about environmental things and if we have a nurturing supportive system where someone, whether student, faculty or resident is concerned about a friend that they can have a way to approach that so its not judgmental or punitive where they get arrested or worse. Dr. Davidson stated that one of things that he has been working on and will eventually present to the committee is a longitudinal curriculum in clinical decision-making and that will hopefully be coming-up sometime before the summer, which is a small part of what Dr. Watson is talking about.
Dr. Meuleman said that several years ago when he was Medicine Clerkship Director they used to have meetings twice a year where all the clerkship directors would get together with the academic representatives. They would give them oral feedback on how things were going and part of it was developmental. He stated that now that we have gotten more quantitatively-based and much slicker, he thinks we need to balance the qualitative feedback, which we may have lost a little bit of, with all of the good quantitative feedback. He thought that was a very useful session to hear the third year class representatives from each rotation sit in their rooms and get into some specifics about courses. It was mainly about the transition from second year to third year and making sure the students felt they were prepared and he thinks it might be useful to bring that back. Dr. Hatch stated that one of the things that they found was that there was a huge discrepancy between what was said there and what was said at their clerkship retreat. He stated that it was anecdotal and the few students that were there drove the discussion in whatever their priority was, whereas when you had the whole clerkship group there that didn’t happen and you had balance. Dr. Meuleman said in terms of the integration of third year or the transition between the second and third year, he doesn’t think we are capturing that as well. Dr. Davidson said they had a spontaneous meeting with a number of third year students about halfway through the year with Dr. Romrell, Dr. Watson, Dr. Rarey and Dr. Harrell one evening by satellite to Jacksonville as well as with a number of third year students that were here. He felt that was very valuable session even though the student’s complaints missed the mark in several ways. Dr. Romrell stated that was a focal discussion on a specific concern in preparation for the clerkship, which he thought was very beneficial. He stated that what Dr. Hatch had pointed out is that when you do a review of clerkships mid-year there is a chance for bias, but that there is some value in it. He does agree that with the bias of a sample of two or three students it may not represent things as well.
It was announced that the College of Medicine was invited by the AAMC as one of six institutions to participate in a conference with a goal of developing new ways of graduate medical education.
Omayra Marrero suggested that maybe students should meet with Dr. Vidauretta at least once in the first year of medical school to become acquainted with her, therefore they would have an outlet in the second and third years should something occur. Dr. Watson expressed how valuable Dr. Vidauretta is and stated that she is a terrific resource.
The meeting was adjourned at 9am.