Respected Medical Device Tester Is Facing Questions

B. Braun Medical officials are unhappy over what they say is a lack of protocol at the ECRI Institute, which has thousands of members and clients around the world.

Nancy Crotti

Medical device review organization ECRI Institute found a lot to like about the Infusomat Space pump, which it recently compared with a handful of others.

The institute gave the large-volume infusion pump made by B. Braun Medical Inc. good marks for performance, safety, interoperability, and maintenance. For ease of use, or workability, it gave the updated pump two out of five stars, a "fair" rating. In the same document, ECRI gave B. Braun's Outlook 400ES a two-star rating, saying "it can be  frustrating and confusing to use."

B. Braun is unhappy with both ratings, but is more upset with what it considers the 50-year-old ratings organization's testing standards.

"We  have had meetings with ECRI before about their lack of protocol" and how they reach their conclusions, said B. Braun spokesperson Constance Walker. "They do not have experience with workflow; they do not do human factor testing; they do not do usability testing."

Those are tough accusations against a nonprofit medical device tester that boasts more than 5000 members and clients around the world, including hospitals, health systems, public and private payers, U.S. federal and state government agencies, health clinics, patients, policymakers, ministries of health, associations, and accrediting agencies.

For the Infusomat evaluation, ECRI's "experienced team of clinical engineers performed hands-on testing in our onsite testing laboratory with additional input from our problem reporting system, our user experience network, and our accident investigation program, as well as input from clinical reviewers," Michael Argentieri, vice president of market development at the institute, said in an email.

The head of another independent testing organization, Health Human Factors (HHF) in Toronto, questioned whether ECRI was using FDA's latest safety guidance, issued in February. HHF worked with B. Braun on Infusomat for two years, "further refining their product through iterative testing of versions of their pump and leading to the submission to the FDA and eventually onto the market," said Joseph Cafazzo, executive director of HHF.

"We're kind of past the point where safety and things like workflow can be tested at the bench," Cafazzo said. "I think that's sort of reflected (in) the FDA requirements as well. The level of rigor is significantly more than it was even a decade ago."

FDA tells whether a particular device meets design requirements, but does not compare devices' performance, Argentieri said.

"ECRI Institute's testing compares product performance, which is beyond what the FDA does," Argentieri added. "The FDA requires testing that validates if the user interface meets its design objectives."

HHF is part of University Health Network, which Cafazzo said is a subscriber to ECFI. He said that the hospital network values ECRI's alerting systems and has purchased its products for decades.

"I think they need to evolve their testing methods as well if they're going to make claims of user safety in particular, and workflow," Cafazzo added. "From my knowledge of what ECRI does, they don't do field or simulation testing or testing with real users. Really, that's the only way of determining user safety and work safety, or efficiency, for that matter."

In ECRI's defense, Cafazzo said the organization may not be able to do the rigorous kinds of testing expected today. He believes FDA's new human factor testing guidelines set a very high standard.

"It's very time consuming, resource intensive and expensive to do this type of testing," he said. "The way they've been doing their testing for decades, they may not have the resources to do it... I knew that this was going to catch up with the industry. It's catching up with the entire industry, manufacturers, and ECRI, which evaluate safety and workflow."

FDA defines the focus of human factors and usability engineering as "understanding how people interact with technology and studying how user interface design affects the interactions people have with technology."

"Our testing covers pump capabilities, performance, ease of use, and service," the ECRI website says. "But the primary focus is on safety features, and primarily the dose error reduction system (DERS). Smart pumps should have features that support hospitals' existing clinical practice while also encouraging safe practices where possible."