Patient Simulators Breathe Life into Product Testing

Medical Device & Diagnostic Industry Magazine
MDDI Article Index

An MD&DI June 1999 Column

Patient simulators enable device manufacturers to conduct realistic usability tests early in the development process, without placing patients at risk.

Late one evening in Chattanooga, a control-room operator scrambles to reestablish coolant flow to a nuclear reactor that threatens to overheat. On final approach to Atlanta's Hartsfield International Airport, a Boeing 757 pilot urgently advances the engines to full power to escape severe wind shear. And in Boston, an anesthesiologist works frenetically to lower the temperature of a patient experiencing malignant hyperthermia, a potentially fatal reaction to anesthesia.

How were these crises resolved? In all three cases, fast and effective action prevented disaster. But in reality, no one was at risk—because these emergencies occurred in advanced, computer-based simulators that could simply be reset, allowing simulation participants another chance to avert disaster.

Advanced simulators have been an integral part of control-room operator and pilot training programs for over two decades. In fact, today's airline pilots are often certified to fly new types of aircraft based solely on their simulator training experience. By comparison, the use of simulators for medical training purposes is a more recent development, but one that is gaining in popularity.

The patient mannequin moves, breathes, and reacts to stimuli, heightening the realism of the simulation. Photo courtesy of MEDSIM (Ft. Lauderdale, FL).

Advanced simulation can overcome the limitations of the traditional training methods—such as classroom instruction and observation—used in medicine. It allows learning by means of trial and error, in a risk-free manner. Adding to the credibility of this new training method is the fact that several teaching hospitals now include time spent working in a simulation facility as a residency rotation.

Medical device manufacturers who seek feedback on their product designs should regard the medical establishment's shift toward using simulators for training purposes as good news. Simulators give manufacturers the opportunity to see how prototypes perform in realistic use scenarios, without placing patients at risk. For example, a simulator makes it possible to test a prototype infusion pump's ease of operation without fear of overdosing a real patient. Such testing may confirm the pump's operation as logical and intuitive, or it may expose design shortcomings that frustrate clinicians and cause them to make mistakes. Discovering and fixing such design shortcomings early in the development process—rather than after the product is already in clinical trials or in widespread use—has several payoffs for manufacturers:

REALISTIC TESTING

Typically, manufacturers use a combination of testing methods to validate their product designs. During the early stages of product development, a manufacturer may conduct a series of usability tests, which typically take place in a controlled laboratory setting. Testing is carried out in one room while unobtrusive observation takes place in an adjacent room via a one-way mirror and video equipment. Such tests focus solely on user-device interactions, albeit in a manner that is isolated from certain factors influencing human performance, such as the working environment, which could be congested and noisy, or interactions with other people who may present a distraction.

During a usability test, participants may freely explore how a device works as well as perform specific tasks, including those critical to patient safety. While performing tasks, test participants normally talk aloud while they work so that the test administrators can follow their thoughts, decisions, and actions and understand how these relate to the product design under evaluation. Such tests, which may include just a few days of hands-on product use by test participants, can be performed well ahead of the clinical trial while it is still relatively easy and inexpensive to make meaningful design changes.

Later in the development cycle, once operational devices are available, manufacturers typically conduct clinical trials over a period of several weeks or months. The focus of such trials is to evaluate a device's safety and efficacy in a real-world scenario with the obvious regulatory overtones. Clinical trials focus on usability in a less-systematic way than in a usability test. Inevitably, it is difficult to conduct a complete usability evaluation of a device in clinical trials because some user scenarios rarely arise. Also, usability problems are reported anecdotally, which may result in inaccuracies, instead of being documented by trained observers as in the case of usability testing. Conducting usability tests in simulators provides the realism associated with clinical trials as well as greater control and freedom to introduce a wide range of user scenarios.

THE TECHNOLOGY

One of the first modern simulators was constructed in 1986 at Stanford University. Presently, there are about 100 patient simulators in use worldwide, but the number of installations is increasing rapidly and may double within a few years, according to Dan Raemer, PhD, program coordinator at the Center for Medical Simulation (CMS) in Boston.

The majority of simulation facilities are concentrated in the United States on the East and West coasts, with several others scattered across the country and abroad. Most of the facilities are operated by teaching hospitals affiliated with academic institutions. CMS, for example, is a resource shared by four teaching hospitals in Boston, including the renowned Massachusetts General Hospital and Boston Children's Hospital.

Simulators comprise several core elements. The most visible is the patient mannequin (Figure 1), which resembles the mannequins used for cardiopulmonary resuscitation training, although much more advanced. For instance, patient mannequins used for simulation purposes produce breathing sounds as the electro-mechanical lungs inhale and exhale according to computer-based instructions. They have anatomically correct airways as well as a palpable pulse and heart rhythm that can be monitored on an electrocardiograph. Some mannequins have arms and legs that are capable of moving and swelling, and computer-controlled eyes that respond appropriately to various stimuli. Some even have gas analyzers that recognize the makeup of inhaled medications and cause the mannequin to respond accordingly.

The patient mannequin and its simulated responses are controlled via keyboard and mouse commands.

The Eagle Patient Simulator, developed by David Gaba, MD, and others, at Stanford University, and marketed by MedSim (Ft. Lauderdale, FL), connects to an interface cart that drives the mannequin's electromechanical functions. The cart also serves as the interface for conventional monitoring equipment found in the operating room. For example, it provides a flow of physiological data to off-the-shelf pulse oximeters and invasive blood pressure monitors, further heightening the realism of the simulation.

Stanford's simulator has a "split brain," which consists of two computers that operate simultaneously to control all aspects of the simulation. One computer runs programs designed to simulate the human body, including its cardiovascular, pulmonary, metabolic, fluid and electrolyte balance, and thermal-regulation characteristics. The computer program's sophistication makes it capable of accurately modeling the body's reaction to myriad physiological inputs, such as intravenous drug administration.

The second computer runs the software that allows the test administrator to control the simulation. For example, it enables the technicians to determine the patient's cardiovascular health and procedural events such as hypoxemia, anaphylaxis, or myocardial ischemia.

Medical Education Technologies Inc. (Sarasota, FL) also produces a patient simulator similar to the Eagle unit. The Human Patient Simulator (HPS) is used at more than 70 institutions worldwide, including universities, hospitals, community colleges, technical schools, and military sites.

RUNNING A SIMULATION

Technically, it only takes one person to control a simulator via keyboard and mouse commands (Figure 2). However, another staff member is generally needed to orchestrate the simulation and coach the participants so the simulation proceeds productively.

The number of participants on the clinician side depends on the nature of the simulated activity. For a simulation of a surgical procedure, for example, there is often a multidisciplinary team that includes one or more surgeons, an anesthesia delivery team, and nurses. At the opposite extreme, a single clinician may work alone, which allows for environmental realism but lacks the realism of people working together as a team. Sometimes, if the simulation exercise has a technological research and development focus, one or more clinical engineers and manufacturer's representatives observe or participate in the simulation.

In keeping with a high-fidelity simulation, participants may follow a normal, preoperative routine, including scrubbing and donning gowns. To further increase the realism of the simulation, someone may play the role of the patient, conversing with clinicians prior to being wheeled into the operating room. A live conversation can still take place in the operating room between the patient mannequin and the clinician (presumably the anesthesiologist) up until induction and then again during recovery, by means of a speaker and microphone built into the mannequin.

THE ENVIRONMENT

To achieve maximum realism, unused operating rooms are an excellent place to set up a patient simulator. However, the hospital setting presents logistical challenges—such as obtaining clearances for test participants and observers—that ultimately make it less tenable. For this reason, CMS is set up as a dedicated training facility that mimics an operating room in a conventional office building.

CMS's operating room is equipped with an operating table, appropriate lighting, a cadre of patient-monitoring and therapy devices, and attendant supplies, so that clinicians can work naturally. The simulated operating room is also equipped with one-way mirrors and video cameras for unobtrusive observation and documentation of simulator activities, including usability tests of medical devices. Rooms are set aside for technicians to run the simulator, for participants to prepare for mock surgery, and for observers and participants to discuss the proceedings.

While the majority of simulators are set up to mimic operating rooms, they are also used to simulate general and emergency medicine, and intensive-care settings and situations. CMS reports that more than half of its simulation exercises are focused on acute-care situations, as opposed to anesthesia delivery alone.

THE COSTS

A state-of-the-art simulator costs about $200,000. Facility modifications and support equipment, including audio and visual recording equipment, can cost an additional $200,000 or so, although more austere solutions are possible. Still, costs approaching half a million dollars are modest in comparison to the cost of medical facilities in general, particularly if the simulator costs are borne by large organizations that have an extensive, continuing need to train medical staff.

Such costs do place patient simulators out of reach of many medical device manufacturers. In fact, cost justification for building a simulator may continue to hinge on the training benefits to medical students, since many medical device companies are just beginning to embrace the need for user testing prior to clinical trials and have not earmarked the funds necessary for more advanced approaches. However, FDA's push for companies to take a more user-centered design approach could lead to increased investments in such testing facilities.

While facilities such as CMS may appear to be set up primarily for training purposes, the staff is anxious to work with device manufacturers as well. "We have run a few usability studies for manufacturers at our facility, but we could and should be doing a lot more," says Raemer. "Considering how productive and comparatively economical the tests have been, I'm frankly surprised by the limited demand that we have experienced to date." CMS charges manufacturers $2200 a day to use the facility, including the services of a technician—slightly more than the daily rental cost of some focus-group facilities.

THE PARTICIPANT EXPERIENCE

People who have participated in operating room simulations—occupying the so-called "hot seat"—regard the experience as highly realistic. "People get geared up pretty quickly," Raemer says. "Within a few minutes, participants are taking things seriously. They get totally absorbed in the experience, to the point that it becomes nearly indistinguishable from reality." Although participating clinicians may initially tend to act somewhat reserved, this attitude typically changes quite quickly to a more relaxed behavior.

Raemer observes that a high degree of realism is essential to the thorough evaluation of medical devices: "People are going to behave differently under stressful and high-task-loading conditions. Whereas clinicians may be able to focus all of their attention on a device in a laboratory setting, they may be able to give the device only a fraction of that attention in an OR—real or simulated—when the patient is desaturating [experiencing a decline in the blood's oxygen level]."

Raemer compares these differing levels of attention to the experience of using a cellular phone in a car. "It's a lot easier trying to program numbers into a cellular phone when you are sitting in a parking lot, as opposed to while driving at high speed in heavy traffic. Analogously, the medical device that seemed so intuitive to operate in the laboratory may actually be confusing to operate in real-world conditions, for reasons you might not guess."

As an example, Raemer says he has watched innumerable test participants struggle to set up and operate a blood-warming device that seemed quite easy to use in an isolated environment, such as a test laboratory. He has also observed many simulator test participants experience difficulty locating a wrench used for adjusting the valves on gas bottles mounted to a particular anesthesia workstation, even though it is placed in a seemingly accessible position.

Raemer believes that not only is the hot-seat experience great for training medical personnel, but it can be equally beneficial for designer and engineer training. He allows that designers and engineers face limitations in terms of their clinical knowledge to the extent that they may require coaching; however, after their simulator experience, they are able to understand the user's perspective. For this reason, CMS offers an "Anesthesia for Amateurs" course, which is quite popular.

A CLINICIAN'S VIEWPOINT

Matthew Weinger, MD, professor of anesthesiology at the University of California, San Diego, practices at the Veterans Administration Medical Center (San Diego) and has a strong, professional interest in user-interface research and development. Weinger is co-chair of AAMI's human engineering committee, and has spent approximately a year working with Gaba at Stanford's simulator. In Weinger's view, "Simulators are an effective compromise or middle ground between laboratory testing and testing in the actual operating room or acute clinical-care environment."

Drawing a comparison with laboratory testing, he states that "the simulator enables significantly greater consideration of the context of performing tasks and interacting with other people. This may be especially important in the evaluation of devices that are complex, resource-intensive, and highly integrated with other devices and data sources. It is also important for devices that will replace well-established devices, or require tasks to be performed in a new or novel manner."

Regarding the selection of test participants, Weinger believes that simulators are too sophisticated to use non-domain experts as test subjects. As such, he advocates using surgeons in tests involving surgical devices, for example, as opposed to using people who may have extensive knowledge of surgery but have never used a scalpel. Nonetheless, Weinger acknowledges the value of involving nonclinicians in simulations to orient them to task demands and make them more empathetic designers.

Comparing it to a real clinical setting, Weinger says, "The simulator potentially may have greater availability, making scheduling easier; however, you still need experienced users who may be reluctant to participate without receiving renumeration, especially if they must forego regular clinical duties. This may result in an additional cost. The simulator can run the same scenario over and over—a real advantage over the clinical setting—thus it offers a more controlled environment. The simulator can also be used for worst-case scenario testing (e.g., crises or critical events). Generalization to the real clinical-use setting is much easier with the simulator than with laboratory tests, but it is still a bit of a leap—albeit a surmountable one."

What role should simulators play in medical device development efforts? Weinger thinks they can and should play a role, especially in devices that meet the criteria previously mentioned (i.e., complex, resource-intensive devices that are highly integrated with other devices and data sources, as well as devices that replace other well-established devices or require tasks to be performed in a new manner). However, he does not think that simulators can wholly replace testing in actual clinical-use environments. "There are too many complex, unanticipated factors," he says, "especially interpersonal interactions. So, I see simulators as a complementary, intermediate tool with specific and valuable uses, especially worst-case scenario testing. Their use may permit postponement of appreciable actual-use-environment testing to later in the development cycle; however, it will not reduce the need for extensive and continuous end-user input and involvement in the development process."

Weinger adds a cautionary note: "The cost [of a simulator for device testing] may be a big limitation for a manufacturer, unless the company already has its own simulator and trained personnel to run it. It may be more expensive than actual operating room testing, although a formal cost-effectiveness comparison has not been done. Another cost-related limitation is the need for clinical personnel to design and run the simulations."

Thus, Weinger views simulation-based product testing as clinician driven, whereas others may see clinicians as marginally involved, except when it comes to actual task performance in the simulator. The costs and benefits of both approaches are likely to be case dependent.

CONCLUSION

In view of FDA regulations requiring the application of human factors engineering in design, the medical industry is just now embracing usability testing as a means to evaluate medical devices before they are brought to market. The change will be good for an industry noted for producing complex devices that can pose usability problems even for some of the most intelligent and highly trained individuals.

Some companies will approach usability testing in a minimalist manner, such as setting up a prototype device in an office and having prospective users run through a handful of tasks. This approach may very well produce useful findings, considering that a lot can be learned about a product when it's used intensively, in an isolated manner. However, many human factors experts would agree that test results can be distorted by such isolation. Accordingly, companies that are driven toward design excellence through usability testing should carefully consider the advantages of using a simulator.

The costs of a simulator are almost certain to be higher compared with laboratory-based or more minimalist testing approaches, but the benefits may be worth it, considering the potential competitive advantage arising from the resulting design improvements and associated marketing claims.

Companies should also consider the potential savings arising from finding subtle usability problems that manifest themselves only in realistic use, at a stage when it is still economical to correct them. Fixing such problems can be very expensive if detection is forestalled until clinical trials begin or—even worse—when the product comes to market.

Michael E. Wiklund is director of the American Institutes for Research (Concord, MA), a consulting firm specializing in user-interface design and testing.

Copyright ©1999 Medical Device & Diagnostic Industry