OUR LITERATURE SEARCH

We wanted to provide you with the mother of all simulation bibliographies. So we began the search with references from 1969 when the seminal article about simulation in medical education was published by Abrahamson and then proceed all the way to June 2005. We searched five literature databases (ERIC, MEDLINE, PsychINFO, Web of Science, and Timelit) and employed a total of 91 single search terms and concepts and their Boolean combinations (Table 9-1). Because we know that electronic databases are not perfect and often miss important references, we also manually searched key publications that focused on medical education or were known to contain articles on the use of simulation in medical education. These journals included Academic Medicine, Medical Education, Medical Teacher, Teaching and Learning in Medicine, Surgical Endoscopy, Anesthesia and Analgesia, and Anesthesiology.

Table 9-1 Search Terms and Phrases.

In addition, we also manually searched the annual Proceedings of the Medicine Meets Virtual Reality Conference, the annual meeting of the Society for Technology in Anesthesia, now the International Meeting on Simulation in Healthcare and the biannual Ottawa Conference on Medical Education and Assessment. These Proceedings include “gray literature” (e.g., papers presented at professional meetings, doctoral dissertations) that we thought contained the most relevant references related to our review.

We also performed several basic Internet searches using the Google search engine—an invaluable resource to locate those articles you cannot find anywhere else (it reviews every CV on the web—so you are bound to find even the most obscure reference). Our aim in doing all this was to perform the most thorough literature search possible of peer-reviewed publications and reports in the unpublished “gray literature” that have been judged at some level for academic quality.

All of the 91 search terms could not be used within each of the five databases because the databases do not have a consistent vocabulary. Although each database also has unique coverage and emphasis, we did attempt to use similar text word or keyword/phrase combinations in the searches. Thus the essential pattern was the same for each search, but adjustments were made for databases that enabled controlled vocabulary searching in addition to text word or keyword/phrase searching. This approach acknowledges the role of “art” in information science, recognizing that information retrieval requires professional judgment coupled with high-technology informatics—and a whole lot of extra time on your hands. [Ojala M. Information professionals as technologists. Online 2002;26(4)5.]

GENERAL AREAS OF SIMULATION RESEARCH

For the past 36 years, the primary focus of medical simulation research has been to justify its use as a training and assessment method. Nearly all of the articles begin with the obvious comparison of medicine to aviation and clinicians to pilots. Then they spend the rest of the time in a defensive tone justifying simulation as a valid training to the point that you think simulators are the ugly stepsister of books, lectures, small group discussions, and patient rounds. We believe it is time to stop all of this defensive research and start moving forward—let’s end the meaningless studies comparing simulators to other unproven methods and begin determining the most effective ways to use simulation for training and assessment. We have an important responsibility, as the current generation of trainers who have seen simulation develop and become integrated with traditional training (we are in a sense simulation immigrants). We need to start planning on training the next generations of clinicians who have grown up with simulation (simulation natives) and not worry about previous generations of clinicians (simulation Luddites) who have looked at simulation as some threat to their unproven, outdated, and unsafe “see one, do one, teach one” philosophy. Let us heed the words of Eric Hoffer: “In a time of drastic change, it is the learners who inherit the future. The learned usually find themselves equipped to live in a world that no longer exists.”

Simulators

SIM ONE

See below.

CASE

See below.

GAS

✓ BUCK GH. Development of simulators in medical education. Gesnerus 1991;48:7–28.

✓ GOOD ML, LAMPOTANG S, GIBBY GI, GRAVENSTEIN JS.Critical events simulation for training in anesthesiology. J Clin Monit Comput 1988;4:140.

While there is evidence of using simulators for military training in ancient Rome, their use in medicine did not occur until the mid-sixteenth century. Although it can be argued that Italian physicians such as Mondino de’Luzzi (1275–1326) used “simulators” when he employed cadavers to complement lectures, the idea to use simulation methods to demonstrate rare conditions or a difficult procedure did not occur until the 1540s.

Why then? At the time, many institutions starting to become concerned regarding the safety of women during childbirth. Although physicians (all men) had the knowledge to deliver babies, it was considered a social taboo for a man to perform a task that was the responsibility of the midwives. However, midwives had no formal training and were graduates of the famous “see one, do one, teach one” university. Initial attempts at formal instruction consisted of lectures with illustrations. This did not affect the infant and mother mortality rates; and more than 100 years later, a father and son physician team from France did something about it—they developed an obstetric simulator.

The Gregoires’ Simulator—it was crude by today’s standards—human skeletal pelvis contained in a wire basket with oil skin to simulate the genitalia and coarse cloth to simulate the reaming skill. “Real fetuses, likely preserved by some means, were used in conjunction with the manikin.” The simulator could reproduce the birth of a child and some complications that would require a trained person to fix.

And yes—there were complaints regarding its validity and transfer to real patients, but for the first time someone said, “it stands to reason we can do a better job and not allow these poor women and children to die.”

Over the next two centuries, there were additional obstetric simulators developed in England and the United States—and they appeared to have enjoyed support from lay people and some other physicians. However, some very familiar factors limited their widespread adoption.

• Cost

• Resistance to adopt new methods of instruction

• Skepticism that what was learned from a Simulator could not be transferred to actual practice

You think after 400 years we would have adequately addressed these issues! Even when the majority of students in the late nineteenth century graduated medical school (there was no such thing as residency) without any direct experience with childbirth, available simulators were not adopted, even though “the use of the simulator would provide medical students with at least some experience with birthing techniques and with some of the related complications.” But no—we would have to wait 80 years before another attempt at simulation for training.

✓ Denson JS, Abrahamson S. A computer-controlled patient simulator. JAMA 1969;208:504–8. [Simulator: SIM-One]

Do not be fooled by the title, the journal, or the year—this article showed the potential of simulators 20 years before the rest of the world would again approach the subject in anesthesia. This simulator was developed under the guidance of Denson, an anesthesiologist, and Abrahamson, a medical educator, both at the University of Southern California; and it was manufactured by Sierra Engineering Company.

Here’s what this simulator could do.

• Life-size—6 feet tall, 195 lb on an operating table

• Able to “breath” in normal manner, with carotid and temporal pulses

• Normal blood pressure taken by auscultation

• Eyelid could blink, eyes could dilate and constrict

• Could respond physiologically to four drugs and two gases

• A computer controlled several conditions

Cardiac dysrhythmias, “bucking” during intubation

Changing blood pressure, pulses, respiratory rate, jaw tension, vomiting

Laryngospasm and obstruction of a mainstem bronchus

So what was the purpose of this Simulator, built before Neil Armstrong took his famous walk?

• “a student could learn necessary manual skills before his first examination of a patient”

• “could learn skills in a planned, systematic way”

• “could learn skills in hours or days rather than in months”

• “more time to study of the patients’ problems and diseases”

• “saving in instructor time and mental anxiety reduced”

• “greatly reduced hazard or discomfort for many patients”

So we had a Simulator that could do many of the things modern simulators can do, and Denson and Abrahamson had identified all of the potential benefits for simulators that we are talking about now! They performed one formal study involving 10 anesthesia residents for endotracheal intubation (the study is described later, in the Halfway to the Station section).

Over the years, Denson and Abrahamsom went on to train many more health care providers, including medical students, interns, inhalation therapists, nurses, nursing students, and ward attendants. In addition to intubation, they trained in ventilator application, induction of anesthesia, intramuscular injection, recovery room care and pulse and respiration measurement (HOFFMAN KI, ABRAHAMSON S. The “cost-effectiveness” of Sim One. J Med Educ 1975;50:1127–8).

Although additional simulators were planned, funding dried up and the culture was not ready for this type of training—the old guard was skeptical of technology, and there was no appreciation of the need to reduce medical errors and improve patient safety, although Denson and Abrahamson clearly made a case for it. In the words of Abrahamson, the factors that led to Sim-One’s demise was “internal administrative problems,” which means a lack of university support. As a result “the funding agencies were no longer interested” and there was growing “low esteem the academic world was developing for education.” Ouch! (ABRAHAMSON S. Sim One: a patient simulator ahead of its time. Caduceus 1997;13(2):29–41).

What is the legacy of Sim One? As Abrahamson states, “the effectiveness of simulation depends on the instructional method with which the simulation is being compared … if there is no alternative training method available (limited patient availability or restrictions on the use of patients), the effectiveness of a simulation device probably depends on the simple fact that the device provides some kind of learning experience as opposed to none.” Thus, Abrahamson was saying 30 years ago that it stands to reason we should be using these devices if nothing else exists or if traditional training is too dangerous.

What did they think about this Simulator at the time?

“From an anesthesiologist’s point of view, SIM 1 might represent man’s most impressive attempt, thus far, to manufacture himself from something other than sperm and ovum.”

“The appropriateness of the anesthetist’s response to each stress is automatically recorded for his later bemusement and education.”

“The next phase, Sim II, would appear to be an automated trainer to eliminate the need for a flesh-and-blood instructor, and the obvious finale is to simulate the learner as well.”

This is not a community-based practitioner reminiscing about the good-old-days of ether and a biting stick; this was the official response of the Association of University Anesthesiologists! [HORNBEIN TF. Reports of scientific meetings. Anesthesiology 1968;29:1071–7.]

We would have to wait until the late 1980s to pick up from where these pioneers left off.

✓ Gaba DM, DeAnda A. A comprehensive anaesthesia simulation environment: re-creating the operating room for research and training. Anaesthesiology 1988;69:387–94.

This article describes the rediscovery of full-body simulators for anesthesia training and introduced Gaba as a player in the wild, wooly world of simulation. You will see his name again and again in this bibliography. Based out of Stanford, home of lots of smart people, it comes as no surprise that Gaba, too, is smart and on a mission to see simulators reach their potential.

Way back in 1988, Gaba laid out how to do a simulation, and he made clear the argument that it just plain “stands to reason” that simulation is a good way to train. He described their setup and how they went through simulations. He argues that a “total simulation” requires the complete capabilities for noninvasive and invasive monitoring. Also, other tasks are performed using standard operating room equipment so the scenario recreates the anesthesiologist’s physical as well as mental task environment.

Gaba and DeAnda described a script, actors in the field, “on the fly” decisions by the simulator director, a high-fidelity mannequin—basically all the stuff we do now in the Simulator. He ran 21 people through the Simulator and they all judged the experience as highly realistic. This article did not actually do any kind of study, it just laid out how simulations are done and how much the participants liked it. Finally, Gaba proposed that simulation has “major potential for research, training, evaluation, and certification.” Amen to that, Dr. Gaba.

✓ SCHWID HA, O’DONNELL D. The anesthesia simulator-recorder: a device to train and evaluate anesthesiolgists’ responses to critical incidents. Anesthesiology 1990;72:191–7.

Dr. Schwid has shown us that simulators come in all shapes, sizes, types, costs, range of feasibility. This multicenter study evaluated the acceptance of a computer-based anesthesia simulator that uses sophisticated mathematical models to respond to user-controlled interventions, including airway management, ventilation control, and fluid and drug administration (53 different agents).

The Simulator also provided detailed feedback that tracked all of the user’s and Simulator’s responses—this could be used for formative feedback during training or summative evaluation to determine if the learner has mastered the key critical events. The Simulator was evaluated by 44 residents and attendings at seven institutions. Feedback was very positive, as nearly all participants found the patient’s responses to management interventions as realistic and determined it was a good device to test anesthesiologists’ responses to critical events. A significant and important finding was that there were no differences in response among any of the institutions—demonstrating the practical transferability of this training device.

It is always tempting to compare this Simulator with the full-body, comprehensive simulator environment developed by Gaba and Good and Gravensein. To do so misses the point! A comprehensive training environment is as much dependent on the faculty facilitator, the debriefing feedback sessions, and the involvement of the “team” as it is on the Simulator.

Schwid’s computer-based Simulator and others similar to it have several advantages.

• Greater accessibility to users at any time and place.

• Does not require additional human resources to use (the instructional design and feedback take the place of having an on-site facilitator).

• Allows greater numbers to be trained in fundamental problems-solving skills in a greater number of cases that just is not feasible or possible with a human patient Simulator—numerous studies have demonstrated that problem-solving skills are disease-specific, meaning that a learner’s ability to treat hypotension due to tension pneumothorax does not translate into the ability to treat hypotension as a result of an acute myocardial infarction—up to 20 cases per condition may be needed here, which is possible only with computer-based models.

Finally, the two following extreme cases illustrate the use of these devices.

• A resident who consistently fails to treat “patients” correctly on a computer-based Simulator is very likely to have significant problems in the real environment.

• There are always some trainees who perform well on computer simulation but panic or are ineffective in the realistic setting. At least, you know the learner’s failure is not the result of a cognitive deficiency—this enables the instructor to focus on communication or team leadership skills.

Anesthesia has consistently looked to aviation as its “model” for training. Well, aviation manufacturers, including Boeing and Airbus, are now “equipping” pilots with computer-based simulators to master prior to attending the full-scale simulator. Rather than compare one simulator type with another, we should focus on the most effective methods in the best mix for training.

✓ GABA DM. Improving anesthesiologists’ performance by simulating reality [editorial]. Anesthesiology 1992;76:491–4.

Gaba starts out by discussing a screen-based Simulator study by Schwid. Schwid discovered that residents made errors.

• Missed esophageal intubations.

• Fouled up ACLS protocols.

• Couldn’t manage myocardial ischemia, anaphylaxis, or cardiac arrest.

Although Gaba never draws the analogy between the simulation and the aforementioned canaries in the mineshaft, we can see how they fulfill this crucial function. If deadly methane gas had seeped out of the coal deposits, the canaries would suffer a severe case of death, alerting miners to the danger. Maybe simulators should be our “canaries.” Instead of waiting for a methane explosion in the mine (a patient catastrophe in the operating room), we should see how the canary’s doing (run residents through the Simulator and uncover their weaknesses).

Usually, we analyze cases retrospectively, after disaster has befallen. This analysis is clouded by incomplete records, failed memories, and, who knows, perhaps a little defensiveness? “I have no idea what went wrong!” So, looking at stuff after the fact isn’t too good.

We could videotape cases as they occur and, in effect, see disasters during the fact. Only problem with that is that most of the time nothing happens. We’d be looking at millions of hours of people sitting on a chair. It would be like watching the medical equivalent of C-SPAN. We might save a few patients that way, but we’ll kill scores of people with boredom. So, looking at stuff during the fact is no good.

How about looking at stuff before the fact? Time travel. Back to the Future instead of C-SPAN. Only the Simulator can provide that kind of time travel. “It stands to reason” that the Simulator is a good idea. You don’t have to wait until a patient is hurt (the retrospective way); you don’t have to wade through miles of stultifying tape (the real-time way); you can “create the problems” without patient harm. You do it ahead of time (the prospective way).

Gaba also reviewed the limits of Simulators, including that, despite their sophistication, they can never create a patient with all of the inherent variables seen in clinical medicine—but so long as they are “reasonable” representations of real patients they could be considered valid by experienced anesthesiologists.

Another limitation is that the trainee is never convinced the simulation is 100% real—leading to hypervigilance in which the poor resident is always worried that something bad is going to happen. This would be okay, except that many errors may result, in reality, from the very boredom and fatigue that occur in real practice. At the other end of the spectrum are the smart alecks who believe that they can do whatever they want because no real patient is at risk.

However, this is true in other industries, and they have made successful use of simulation. In medicine, the validation of simulation will be even more difficult than aviation because no two patients are alike (unlike a 747); the effects of training should be measured over years of training and remediation not after a single training session. Gaba summarized his editorial by making the important point: “No industry in which human lives depend in the skilled performance of responsible operators has waited for unequivocal proof of the benefits of simulation before embracing it.” I say we embrace it too.

✓ Gaba DM. The future vision of simulation in health care. Qual Saf Health Care 2004;13(Suppl 1):i2–10.

In this article, Gaba shows why he is the maven of high-fidelity simulation in health care. He describes a comprehensive framework for future applications of simulations as the key enabling technique for a revolution in health care—one that optimizes safety, quality, and efficiency.

Gaba begins by making the case that simulation addresses current deficiencies of the health care system.

• Places premium on basic science education while leaving clinical training to an unsystematic apprenticeship model

• Emphasizes individual knowledge and skill rather than clinical teams

• Unstructured and minimal continuing education

To address these problems, Gaba proposes that Simulators must be integrated into the fabric of health care delivery at all levels, which is much more complex than piling it on top of the existing system. To do so, he outlines 11 dimensions (and gradients within each) that can take us there. Next, Gaba outlines the various social entities, driving forces, and implementation mechanisms that could forward the agenda of simulation in medicine. Finally, he paints two possible scenarios (he has had lots of practice at developing scenarios) for the fate of simulation in health care.

Optimistic scenario

• Merging of various driving forces

• Emerging proof of the benefits of specific applications of simulation

• Major institutions with dedicated programs to multiprofessional team training

• Public demand for safety in medicine on par to other high-risk industries—liability insurance tied to organizations that embrace simulation as a means to respond to the public demand

• Government support and later demand for simulation training

Pessimistic scenario

• Public becomes more interested in access to care and cost than the quality or safety of care

• Failure reforming the systems of clinical work to match what is being taught in the simulation centers

• Large multicentered trials never materialize owing to lack of funding—thus long-term proof of benefit of simulation never occurs

• Professional organizations focus more on cost of medical care at the expense of increased investment in training

• Simulation centers become liable for trainees who later commit medical errors

Although we certainly take the optimistic view, we know it stands to reason that Simulators will have a significant future in medical training because of the dedication and hard work of individuals who will ensure that it happens.

✓ HELMREICH RL, DAVIES JM. Anaesthetic simulation and lessons to be learned from aviation [editorial]. Can J Anaesth 1997;44:907–12.

This editorial points out that simulators have a lot of potential for serving as tests. All the usual arguments hold—you don’t put a patient at risk, you can reproduce the scene. But this editorial goes on to point out a crucial problem with using a Simulator as a “test vehicle.” A key problem is the idea of “equifinality”—that is, different techniques can give you the same end result. (The article does not mention the following example, we made it up just to illustrate the point.) For example, one anesthesiologist may use epinephrine to achieve a goal, whereas another may use dobutamine to achieve a goal. Both achieve the same goal—better cardiac output. So, in the Simulator, what do you do? Grade someone wrong who uses epinephrine because the “simulator grade book” says you should use dobutamine? The editorial finishes by saying “there is a need to provide opportunities for practice and assessment until the culture supports the fairness of the assessment process.” In other words, it “stands to reason” that a Simulator is a good way to test, but we haven’t quite gotten there yet.

✓ MURRAY WB, SCHNEIDER AJ, ROBBINS, R. The first three days of residency: an efficient introduction to clinical medicine. Acad Med 1998;73:595–6.

Dr. Murray and the fine folks at Penn State (you can almost smell the chocolate from the Hershey factory) describe the first 3 days of their anesthesia residency. Rather than just shoveling a ton of stuff at their residents, they make the learning more active, using (what else) the Simulator. Result—a questionnaire showed “improvement in the residents’ confidence in their ability to carry out clinical tasks.”

So, it “stands to reason” that if a Simulator increases the confidence of a resident, a Simulator must be a good thing. A hard-nosed scientific drudge could look at this and say, “This is not rigorous proof.” A skeptic could look at it and say, “So what, what difference does that make, a little more confidence?” But I’ll bet that to those Penn State residents the added confidence made all the difference in the world when they walked into the OR the first day.

✓ MURRAY DJ. Clinical simulation: technical novelty or innovation in education [editorial]. Anesthesiology 1998;89:1–2.

Dr. Murray is the big cheese in Simulation at Washington University in St. Louis. This is a “do we really need Simulators?” editorial. What did we do in the “B.S. (before simulator) era”? We did residency and did a lot of cases with supervision. We did lectures, one-on-ones with attendings. But why use the past tense? That’s what we are doing right now!

So, do we need to throw Simulators into the mix? Yes. You can use Simulators to teach.

• Physiology to medical students

• Crisis management to a “mixed crew”

• Conscious sedation to nurses, techs, and therapists

Murray goes on to say that a lot of different groups need to work in the Simulator. Anesthesiologists alone can’t keep the thing humming all the time. A Simulator is a Lamborghini—you bought it, now drive it! Don’t let it sit in the garage all day collecting dust. Get that thing on the road.

✓ ISSENBERG SB, MCGAGHIE WC, HART IR. Simulation technology for health care professional skills training and assessment. JAMA 1999;282:861–6.

Dr. Issenberg, who is one of the authors of this book, oversees the development “Harvey,” the Cardiology Patient Simulator at the University of Miami. In this Special Communication, Issenberg et al. touch on all the simulation technologies that were available in 1999, laparoscopy simulators to train surgeons, their own mannequin Harvey to train students about 27 cardiac conditions, flat screen computer simulators, and finally anesthesia simulators.

What does Dr. Issenberg have to say about the anesthesia simulators? “The high cost and requirements for accompanying equipment, space, and personnel have resulted in research to justify the installation of such devices.” (Hence so many “justification of simulators” articles in this bibliography.) If you look at “intermediate” benefits of simulators, Issenberg points out the following.

Simulators are highly realistic.

Training on a simulator can improve the acquisition and retention of knowledge compared with sitting in a lecture hall.

If ever used as a certification tool, “They allow the examinee to demonstrate clinical skills in a controlled clinical environment while still exhibiting cognitive and language skills.”

So, as study after study comes out hinting that simulators can make us better practitioners, do we have to wait for proof positive? No.

✓ GORDON JA, WILKERSON WM, SHAFFER DW, ARMSTRONG EG. “Practicing” medicine without risk: students’ and educators’ responses to high-fidelity patient simulation. Acad Med 2001;76:469–72.

This is a “feel good” qualitative paper about simulators, pure and simple. Altogether, 27 clinical medical students and clerks and 33 educators went through the Simulator and were asked how they feel about it. The medical students were instructed to evaluate and treat two patients: (1) a trauma patient with hypovolemic shock and a tension pneumothorax and (2) a cardiac patient with marginally stable ventricular tachycardia. The educators, on the other hand, were instructed to care for a patient with anaphylaxis. All participants were debriefed in a case discussion afterward and then completed several evaluations to determine who liked the experience.

To get back to the “theme” of this group of articles—It “stands to reason” that an educational method that everyone likes should be an educational method we should use. Everyone likes Simulators. Even better than the statistics (85% loving the Simulator) were the “raw comments” that hammer home just how cool Simulators are.

“I think everyone could benefit from this.” “Every medical student should have the opportunity to learn using this Simulator several times each year.”

How can you argue with that?

This study also demonstrates the benefit of relatively small sample sizes—you can collect more qualitative data so you know not only what they liked but, more importantly, why they liked it.

✓ Gordon JA, Oriol NE, Cooper JB. Bringing good teaching cases “to life”: a simulator-based medical education service. Acad Med 2004;79:23–7.

Based on their successful pilot studies of positive learner reactions to simulation-based education, Dr. Gordon and his colleagues set out to develop a comprehensive on-campus simulation program at Harvard Medical School. They provide a descriptive case study of how to develop a simulator program in an undergraduate medical curriculum. And when the folks at Harvard give free advice—we listen.

The authors outline several initial steps that are critical to get a simulation program off the ground and make sure it lasts.

• Step 1: Interdisciplinary oversight—make sure you have input from all possible stakeholders and include them in the process. This includes education deans, faculty physicians, administrators, educators, and bioengineers.

• Step 2: Capital equipment and training—you need money to buy these simulators, and more importantly you need to make sure they maintain close contact with the technical staff of manufacturers to avoid equipment that remains “in the box.”

• Step 3: Dedicated space allocation—a centralized location available to all students and faculty is important. After that you need no more than 400 square feet to get started.

• Step 4: Administration and partnership—give the new program a fancy name to distinguish it as a dedicated on-campus resource: “The MEC Program in Medical Simulation at Harvard Medical School.”

The authors provide practical tips on integrating simulation into the existing medical school curriculum by using existing material rather than “reinventing the wheel.” Students in every year of medical school can have meaningful education and training using simulation—you don’t need to restrict this to junior and seniors in medical school.

However, what separates this program from all others is the development and implementation of a “medical education service” dedicated to providing “education on demand” for any student who wants to use the Simulators. Faculty members and residents provide the instruction so students can use whatever “down time” they have to hone their skills.

This has become very successful, as evidenced by a group of 15 graduating students who wrote to the dean, “the Simulator stands out as the most important educational adventure at Harvard Medical School.”

What can be better than that?

✓ GREENBERG R, LOYD G, WESLEY G. Integrated simulation experiences to enhance clinical education. Med Educ 2002;36: 1109–10.

Dr. Greenberg and her faithful minions from the University of Louisville Patient Simulation Center at the Alumni Center for Medical Education (see? what did we tell you about the importance of having an impressive name for your simulation center) combined a high-fidelity Simulator with a standardized patient. The ultimate simulatory experience—first you talk with an actor pretending to have a condition, then you go to the Simulator as if the actor has now “become” the mannequin. Great idea!

First, students meet a patient (SP—standardized patient, the actor) about to have an appendectomy. Next, the student follows the patient into the OR and participates in anesthetizing the patient (Simulator) throughout the procedure. Then the student returns to the waiting room to discuss the procedure with the patient’s spouse (SP). Finally, the student examines the patient (SP) 2 weeks later when she presents with a fever. Whew! Faculty like exploring new clinical teaching and testing methods, and the students are more engaged in their education.

This is an educational twist—that it “stands to reason” is a great way to teach. You combine the best of both worlds and give the student a hell of an experience.

✓ EPSTEIN RM, HUNDERT EM. Defining and assessing professional competence. JAMA 2002;287:226–35.

When you think of “medical science” you think of hard data: blood levels of propofol, heart rates that say “beta-blockade achieved,” or gastric emptying times. And even in the “softer” realm of medical education, you still look for “hard data”: test scores, percentage pass rate of a program, and (in our Simulator world) checklists.

This JAMA article takes us even farther into the “soft.” What is competence? How do you assess it? Look at their definition of competence and ask yourself, “Just how could I assess competence?” and, not to be too cagey about it, “Could I use the Simulator to assess competence?”

Competence is “the habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individual and the community.”

OK, genius, just how in blue blazes do you assess that? (For our nefarious purposes, can a couple of hours in the Simulator fill that tall order?) JAMA tells us that the three biggies for assessing competence are:

1. Subjective assessments by supervising clinicians.

2. Multiple-choice exams.

3. Standardized patient assessments, that is, the “pretend” patients in the objective structured clinical exam.

Note: Simulators are not mentioned. The million dollar question—Should Simulators be included?

OK, our goal is to assess competence, and we currently have three ways of doing it. Are they any good? (By extension, does a budding Simulationologist see any defects in the current system that the Simulator could fill?)

1. Subjective assessments by supervising clinicians. Any problems here? Evaluators often don’t see the resident in action—think of the call night, when the attending is not around much. Evaluators have different standards and are subject to bias.

2. Multiple-choice exams. Any problems here? Test scores have been inversely correlated with empathy, responsibility, and tolerance—think of a high-scoring resident who is a creep and treats patients like dirt.

3. Standardized patient assessments. Any problems here? Yes. Defining pass/fail is difficult. Assessing interpersonal skills may take a lot of exams.

So here we have the current three methods of assessing competence. Look again at the definition of competence and ask yourself if any of these three really hit the nail on the head. Competence is “the habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individual and the community.”

Does an attending physician’s evaluation of a resident assess “the habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individual and the community.” Not really.

Does a multiple choice exam assess “the habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individual and the community.” Not really.

Does a standardized patient assessment evaluate “the habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individual and the community.” Um, closer. I think.

This whole world is murky and quasi-scientific. Go ahead, try to make a bold and sure statement about assessing competence. “The best method for assessing competence is the standardized patient assessment!” Someone asks you, “Prove it.” You say, uh, you say … what do you say?

So wouldn’t it be great if the JAMA then said, “So the current methods of assessing competence aren’t any good. But putting people through the Simulator fits the bill perfectly!” Well, they didn’t. Too bad. But they did say that we need to develop innovative ways to assess professional competence. And, who are we kidding, that is exactly what we’re trying to do with our Simulators.

✓ DILLON GF, BOULET JR, HAWKINS RE, SWANSON DB. Simulations in the United States Medical Licensing Examination (USMLE). Qual Saf Health Care 2004;13(Suppl 1):i41–5).

This is the article we have been waiting for—the people in charge of providing the assessment requirement for a medical license in the United States predicting the inevitable use of simulators in high-stakes examinations.

They provide a current description of the US medical licensing system and explain how all of them use some form of simulation.

• Step 1—Focuses on concepts of science basic to the practice of medicine, a computer-delivered examination made up of multiple-choice questions. The simulation is in the form of brief descriptions of patient care situations.

• Step 2—Two components: Clinical knowledge (CK)—one-day computerized multiple-choice examination to assess whether an individual possesses the medical knowledge and understanding of clinical science considered essential for the provision of patient care under supervision. Like step 1, the simulation is in the form of brief descriptions of patient care situations. Clinical skills (CS)—one-day 12-station standardized patient-based examination intended to assess the examinee’s data gathering and communication skills directly. The simulations are in the form of actors portraying 12 common, important clinical problems.

• Step 3—Two-day examination combining computer-based MCQs and computer-based case simulations intended to assess whether the individual can apply medical knowledge and understanding of biomedical and clinical science essential for the unsupervised practice of medicine. The simulations are in the form of computer-simulated case presentations that unfold according to the responses of the examinee.

The authors, all affiliated with the National Board of Medical Examiners or the Educational Commission of Foreign Medical Graduates acknowledge the use of Simulators, both task trainers and full patient simulators for assessment. Their use in high-stakes testing (for a license) has been limited by their high cost and lack of reliable, valid scoring mechanisms

However, the authors acknowledge that “as the cost of these mannequins declines, and additional … studies are completed, they could have a unique role within the licensure process….” Why?

• Simulators can model rare events prone to medical errors with no risks to patients, especially skills that cannot be measured with real patients.

• Real-time responses to therapeutic interventions can be modeled, and thus the management of patient conditions can be assessed.

• It is possible to develop scoring systems based on measurable patient outcomes rather than the judgment of the examiner.

• It is possible to assess joint patient care efforts of a team, including multidisciplinary communication skills.

There—the folks in charge of testing and therefore education and training (testing drives learning) have just stated what we knew all along. Want to go for a ride?

✓ Seropian MA. General concepts in full scale simulation: getting started. Anesth Analg 2003;97:1695–705.

This article is cited later in this book, where we mention, “If you are thinking of starting a simulation center, and you’re looking for a good ‘how-to’ article, this is the one.” Dr. Seropian pays most attention to the person running the Simulator, not so much the Simulator mannequin itself. It’s the live

Only gold members can continue reading. Log In or Register to continue

Share this:

• Click to share on Twitter (Opens in new window)
• Click to share on Facebook (Opens in new window)

Related

Related posts:

Attendance and Scheduling Issues With What Other Disciplines Should We Work? How Anesthesia Simulation Is Done What Is a Simulator–a Clinical Checklist or a Theater? The Great Debate Simulation Equipment