Scrutinizing the Scrutinizers

Q & A

Erik Swain

October 1, 2006

22 Min Read
Scrutinizing the Scrutinizers

Former FDA-insider Timothy Wells has a new perspective on the state of QSIT.

Few people have shaped today's FDA medical device inspection climate as much as Timothy Wells. As a leader (with fellow agency officials Georgia Layloff and Robert Ruff) of the Quality System Inspection Reengineering Team during his tenure at FDA, he helped author the Quality System Inspection Technique (QSIT) that modernized CDRH's inspection approach. He co-wrote the QSIT Handbook and coordinated QSIT training of FDA's field inspectors and compliance officers. That was the highlight of his 24 years at FDA, which began in 1976 as an investigator in the Chicago District and led to a post as a branch chief in CDRH's Office of Compliance. In 2000 he left the agency to found QualityHub Inc./Wells & Associates (Rockville, MD), a firm that provides consulting and training to device and pharmaceutical clients around the world.

FDA created QSIT as a means to conduct inspections more efficiently and effectively in a climate of scarce resources. Wells believes that while QSIT has achieved most of its goals, especially in terms of providing more useful information about compliance deficiencies, it has been sometimes been misused by the agency to justify inspections lacking in rigor. As a result, he has some ideas for how it can be tweaked. He spoke to MD&DI Editor-in-Chief Erik Swain in July.

Q: What prompted CDRH to craft a new inspection technique?

A: In the late 1990s, CDRH Director Bruce Burlington took the opportunity to look at processes within the device segment of FDA and reengineer them. That was a time when the highest levels of government were encouraging improvement in performance of the agencies and the agency programs. Numerous areas within devices were looked at for improvement. One of those areas was the inspection process.

The inspection process was chosen for reengineering because the Quality System Regulation (QSR) had recently been issued, but the inspection technique was not upgraded to incorporate quality system principles. The agency was using the bottom-up approach for device inspections, which was to start the inspection with complaints and often to document how bad things were based on those complaints. We believed there was a need for a top-down inspection that would look at the system as opposed to the data that resides within that quality system. Two initial purposes of quality system inspection reengineering were that inspections were taking a long time, and that inspections were not always focused. In other words, the inspector spent a lot of time looking at individual documents but did not always yield usable evidence on the compliance end of things. A third purpose, although not a project design input, was inspection inconsistency.

Q: How did the team come up with the QSIT scheme?

A: Early work included identification of stakeholders. Interviews were conducted across stakeholders. A large public meeting was held in Rockville, where we learned that industry actually wanted a thorough FDA inspection. In fact, their preference was for comprehensive inspections that occurred less frequently rather than more frequent partial inspections . They felt they needed to know the state of their quality system so they could fix it, as opposed to piecemeal efforts. It's like when you go to the doctor to get a physical, you want a total physical, and not a partial one, so you can determine the true state of your health. This was a very healthy attitude, in my opinion. Industry was a major player in the design and development of QSIT. A working group consisting of FDA and industry experts was formed. Meetings (hosted by the Food and Drug Law Institute) were held over several months. QSIT was sketched out by the group. The final details were written by a small group within FDA, the names of whom are
listed in the QSIT Handbook.

Q: How did industry react when QSIT was announced? How well has industry become familiar with it?

A: Since they helped design it, there was no adverse reaction. We took the time to hold frequent industry workshops. And we held the workshops nationwide, with industry sponsors, for months after QSIT was launched. Buy-in with industry was actually easier than buy-in from the FDA investigators. There was some resistance. But back to the industry buy-in, it's interesting to note that a large group from industry and FDA was given the Vice President's Hammer Award for reinventing government by improving the quality system inspection program. The effort resulted in cost savings because it reduced inspection time but produced more effective and focused inspections. Quite honestly, my FDA and industry colleagues and I were proud of the outcome. You don't always get an opportunity in your professional career to do something that makes a difference. We believe that QSIT made a difference.

Q: How easy or difficult was it to train FDA inspectors in the new technique? How easily was it accepted culturally within FDA?

A: We had huge support by CDRH and ORA management. Support from the ground level troops was another matter. We knew it would not be easy to change someone's methodology that they had developed over 20 years beforehand. We wanted field buy-in. I think we came up with some good methods to achieve that. One way we achieved this was the development of a training program for the device investigators and CDRH compliance officers. We developed a fairly comprehensive training course for QSIT. We also created an exam for the QSIT training. We got field management to require a passing grade before allowing any investigator to do QSIT inspections. This was a relatively new concept, in my opinion. In the end, hundreds of investigators were approved to do the QSIT inspections. You could say the users were calibrated, more or less, by the QSIT exam.

In addition to the training, we rewrote the Device Inspection Compliance Program. We developed a three-tiered process for executing QSIT in the field and wrote criteria for FDA action. The second tier was the primary QSIT inspection of four subsystems: Management Controls, Design Controls, CAPA and Production and Process Controls (PPC). We then trained the field compliance officers on the compliance program because they were the people who would be writing the warning letters and taking legal actions from the QSIT inspections. Obviously, we also had FDA Chief Counsel's buy-in that QSIT inspections were strong enough to support legal actions.

With all that stakeholder work we had no real problems getting QSIT accepted within FDA or industry. We had some investigators who did not wish to follow it—they preferred the old bottom-up approach and continued to use it. I can't say how widespread this is today, however, because I left FDA in 2000, shortly after we got QSIT off the ground. With all the things that have gone on at FDA since that time, it may not be known who is and who isn't following QSIT. It wouldn't be hard to do a follow-up study. It took two years of work to make this, but we didn't calibrate it in terms of inspectors following it after rolling it out.

Q: Has QSIT accomplished the goals you were hoping it would? What if anything would you change today?

A: Yes, we clearly achieved our goals. A full validation was done to provethat point. The QSIT inspection was more focused and saved time over the previous inspection process. This validation report is still available on FDA's QSIT Web site ( Regarding the actual QSIT inspection process, I would change it a bit. Here is what I would do.

I would not change the Management Controls portion.

I would change the CAPA portion to spend less time on preventive action and more time on trending, risk analysis, and root cause investigations. I would look at the methods the firm used to perform the root cause investigation. I would emphasize more on reviewing complaints. I would review the trend reports both before and after corrective actions were taken. This would show whether the corrective actions work. I would also focus more on whether the CAPA had effectiveness checks rather than on whether the CAPA was validated before implementation. I'm finding that some firms are doing validations of the corrective actions in advance while others are doing corrective actions in real-time. In either case, effectiveness checks are important because they show follow-through. During CAPA review I would check several feeder systems (such as complaints and non-conforming materials) as well as the main CAPA program. Both systems should be robust. Finally, I would check the overall CAPA system in terms of how well it is being managed. Timeframes should be established and work should be tracked against those timeframes.

Another change would be to redo the PPC section. I would favor more rigors here. I would change from reviewing a process to reviewing a product. I would pick a product known to have problems and check all aspects of its production controls. The most important aspect to check is its verification and validation. Does the device meet its specifications, and how can the company prove it? If it is not doing destructive testing, such as for sterility, it would need to demonstrate how it can have confidence in the process being used and the tests being done. This would involve validation reviews. I also would check training of the people involved in the process, the acceptance activities on the components, the calibration and maintenance issues on the equipment for that product, and validation of the equipment being used. I also would check the testing done and associated method validations, calibrations, etc. All records associated with the product should include master records, history records, change control, and other related documents. I think PPC can be beefed up. The new inspection would add hours onto PPC.

Q: That sounds more like the old bottom-up approach.

A: Now that I'm on the outside, I can see that you need a certain amount of bottom-up inspection as well as a certain amount of top-down. An all-top-down approach may not be the most productive. To really understand a firm's quality system, you've got to get to the raw data, so you can see what actually happens. But one reason we came up with QSIT is that all the inspectors were doing was looking at the raw data. They never looked at things like the firm's quality manual, so it was hard to put it in context. And QSIT does allow you to drill down. What I'm saying is that with production and process controls, the important things aren't so much theoretical as they are in the application. And if you start with a problematic product, you can find a wealth of information about how things are really being done.

Q: What about the Design Controls portion?

A: I would reduce that part of the inspection. By focusing more on high-risk areas within design controls, hours could be shaved off the inspection of this system. I would focus more on four key items: risk management (design FMEAS, etc.), design verification, design validation, and design transfer. I believe these are the key areas that are weak in the industry. I would spend less time on design planning, design inputs, design outputs, design review, etc. I would probably do CAPA ahead of design controls so I could look at the design issues related to complaints. In other words, I would check if there were linkage between device problems and the design control system. I would check if design FMEAs are updated after new risks are identified via complaints, and so on.

Overall I suspect the time should be the same with my proposed QSIT process as the original. What is important is that FDA has very limited time with manufacturers. It must use that limited time to do the most comprehensive and useful inspection of the quality system. But FDA has made great strides with QSIT. Prior to it, inspections were about 9 ? days and outputs were irregular and inconsistent, and not always useful to headquarters. QSIT has cut that time to about 5 days and produced much more useful documentation of violations. QSIT is a good tool to weed out problems in the quality system if you follow it. I still get questions about it to this day.

Q: How else would you change the FDA inspection program today if you could?

A: I was disappointed to see that FDA changed the way it managed the inspection program a few years back. It started to allow Level One inspections on firms that had never had an inspection. Level One inspections (CAPA plus one other system) were only supposed to be used on firms who had passed a Level Two QSIT inspection. It is my understanding that this was done to save resources. The problem is that many companies who are getting CAPA-plus-one inspections are getting quick two-day inspections where FDA never really gets into the key weaknesses of their quality systems. Companies are getting a false sense of security from the Level One inspection because FDA is not finding much in the way of quality system issues. Also, the public is not being served with this less-rigorous inspection. Level Two was supposed to be a baseline inspection. Every company was supposed to have one before being considered for CAPA-plus-one. To use the doctor analogy again, you need a full physical at least once before you should be comfortable with getting a partial physical. Taking a brand-new company and giving it a partial audit is risky. The agency has pretty much done that to save money.

In fact, I would greatly change or eliminate Level One inspections. In retrospect, CAPA-plus-one is probably not getting what we want. The thing about CAPA-plus-one is that if there is any problem, it should show up in the CAPA system. But what if the firm doesn't have CAPA or doesn't use it? Then the inspector walks out the door without a good assessment of what's going on. But it seemed like a good idea at the time.

All inspections should cover the four systems mentioned in QSIT. I might encourage FDA to develop a mini-QSIT tool for Level One inspections with four or five main questions from each of the four QSIT systems. FDA could use this on firms with at least several years of Voluntary Action Indicated or No Action Indicated inspections.

FDA's current policy of doing Level One inspections on new firms or firms with a checkered history is dangerous. Since recalls and Medical Device Reports (MDRs) are occurring on both Class II and Class III devices, basing the level of inspection on device class is not wise, in my opinion. Both need FDA attention. Between the risk-based assignment of firms (inspecting Class III companies more often) and the policy of prioritizing Level Two inspections on Class III device companies, making it easier for Class II companies to get Level One inspections, FDA might be missing the mark. FDA is overemphasizing Class III devices and underemphasizing Class II devices. Both need the full rigor of the QSIT inspections. Just look at the recent history of recalls and MDRs and you will see these are occurring for both Class II and Class III devices.

Q: How would you assess the third-party inspection program so far? What will it take for it to catch on?

A: Being on the outside for nearly six years now, I can see why companies do not show much interest in the third-party inspection program. I think part of this is the stigma associated with the ISO audits. While device companies regularly get ISO audits, and they are extremely important, the companies know the rigor is not the same compared with FDA inspections. Another reason may be that FDA is FDA. You can't really replace it. The investigators have years of training and FDA experience, which is so with the third party auditors. Maybe respect is a better word. The third party auditors, even though trained by FDA, are not getting the respect that FDA investigators get. I believe companies feel that if they are about to get an FDA inspection, they should get it from the real deal.

I can't really say how it will catch on. Coupling it with the other audits, such as ISO, may actually hurt more than it helps. Since the companies pay these auditors and FDA is a public health agency, there will always be a question of pleasing the client versus addressing public health concerns. It leaves a cloud of doubt that may never be removed, regardless of how hard FDA pushes the third-party program. Companies do not want an easy audit, only to be clobbered by the real FDA. They want the real FDA audit with the resulting findings. This was a point made at the QSIT industry meetings. Companies want FDA to do a thorough job so they know what they are doing well and where they need to make improvements. All variations on the theme, including using non-FDA auditors, may add up to less than the full FDA audit.

Q: Industry often complains that there is too much variation from inspector to inspector. Is this complaint fair? If so, what can the agency do?

A: As I mentioned earlier, inconsistency has always been industry's number-one complaint about inspections. We tried to fix that somewhat with QSIT. It provided a standard approach. What did not happen was FDA management enforcing the consistency on their investigators. Some investigators simply do not follow QSIT. FDA should study this matter and decide how to handle those investigators who perform device inspections outside of the QSIT method.

That being said, there is some benefit to having inconsistency. Variation of depth is needed and serves a purpose in giving FDA some flexibility. Firms have to plan for the worst. I tell companies to plan for the toughest and deepest inspection.

Extreme variation, however, is another matter. Some investigators are on the fringe in terms of being too adversarial, accusatory, and inflexible. Other investigators are too collegial, timid, and non-aggressive. FDA should always seek a balance with its investigators. Personalities play into this. I would only say that FDA should use those tougher investigators with companies that need that tough approach and use the less-aggressive investigators on companies that have shown great cooperation over the years.

Q: But that would entail FDA admitting that all investigators are not the same. Could it bring itself to do that?

A: They do it now, in terms of sending in certain people when faced with repeat offenders. I'm just saying that maybe there's a place for the harshest investigators. There are some companies that need a tough investigator. Who would you want inspecting a company that falsifies records? That's a criminal act.

Q: What are the most important things a firm should keep in mind when preparing for an inspection?

A: Planning and preparation is key. Poor inspection management is a problem worth noting. Two things that can cause a problem during inspection, and can be avoided, are not being ready for FDA's questions, and not being familiar with procedures and outputs from a given area. I suggest the following approach:

• Decide now on the inspection team members—leads, scribes, runners, and back room staff. Make sure the inspection leaders are capable of managing the event.
• Develop a list of Subject Matter Experts (SMEs) and backups.
• Develop talking points and answers to FDA's questions even before FDA arrives. You can start planning this now. Try to predict both the common questions, such as the QSIT questions, as well as those difficult questions you hope won't be asked. Develop answers to those questions now. Develop briefing booklets for each area within the quality system as well as for each recall and each FDA-483 item from the last inspection. Make sure you have an assigned SME for each item and a back-up SME.
• Rehearse the inspection. Have each SME present his material to a mock auditor. Have the SME bring the appropriate SOPs and example of work outputs to the exercise. In the case of 483 and recall topics bring the corresponding documentation (objective evidence) showing the problem has been handled. Bring evidence to show it was handled both locally and systemically. Have the leads attend these sessions so they are familiar with the quality system as well as all product and prior inspection issues. Make sure the lead can help out the SME if necessary during the interviews.
• Practice the mock-interviews again and again. Weed out inappropriate SMEs and fix up the presentations.
• Develop the starting line-up for the real inspection.

Practice makes perfect. Preparation can make the inspection run smoother and convey that the firm has it together. By being efficient and effective, you are showing respect for the limited time the investigator has in your site. You are eliminating confusion that can arise with unprepared SMEs and inspection leads. And you are leaving a positive impression of your company with the investigator.

Q: What are the most common mistakes firms make when it comes to inspections?

A: Aside from having unprepared experts, another mistake is to fail to anticipate the direction the inspections take. The inspection leads and the back room personnel need to almost predict each question before it is being asked. Experience with FDA inspections helps here. The firm should be following the path the investigators are on and predict where they are going next. An example is having records in the room even before they are asked for. If you know the path FDA is on you can predict where they are going. SOPs for related systems and processes should be at the ready. Related documents, such as investigations related to CAPAs, should be brought in the room before being asked for. I would not provide them to FDA until asked, but I would expect the investigator to ask for any related documents and files.

Good planning and good SME presentations can neutralize a bad opinion that an investigator has. But the firm must be able to convey its points. Solid evidence is needed to convince the FDA. Poor planning prior to the inspection and poor inspection management can result in unnecessary FDA-483s. There are often 483s that show up because the company couldn't speak properly to convey their state of control. If you put the wrong person or an unprepared person in front of the investigator, you will get 483's that you otherwise might not have gotten.

Q: How might inspections evolve in the future? What are the biggest challenges FDA and firms face?

A: At one time I believed that the future would hold more accountability by the companies and less inspections by FDA. I'm not sure I still believe this. I was hoping executive management would step up to the plate and become the FDA surrogate by having tough internal audits along with great management reviews and robust quality systems. I can't say that I see this happening. What I see is that executive management at many companies are too preoccupied with sales, growth, and other business matters to become those surrogates. I still see reactive management more than proactive management. I don't know if this is simply human nature or whether ten years is not enough time for the quality system mentality to permeate the industry adequately.

I don't see FDA inspections ever going away. In fact I see more need for inspections now than in the past. As more and more products enter the market, with greater complexity in technology, continued vigilance through inspections is needed. I suspect that there may soon be fewer inspections— mostly because of ever-dwindling FDA resources—but at the same time the inspections should be more robust. FDA will need to retain highly trained and motivated investigators. Their training program should be looked at to ensure the skills are in place. Retention is a problem at FDA that needs to be addressed in the inspection area immediately. The old guard is retiring and the new guard is coming online. I have seen many great new guards. Let's hope FDA can continue to hire and adequately train investigators.

Team inspections seem to accomplish much more than solo inspections. I suspect team inspections will continue to be used. An FDA challenge is deciding whom to inspect, and where to use those few FDA resources in order to get the biggest bang for the buck. Class II companies should be in the picture along with Class III companies. Historically, every time FDA ignores a segment of industry, their compliance starts slipping. FDA has to pay attention to the entire device industry for inspections, covering all classes of devices.

FDA also needs to look into and address the increase in device recalls. At one time FDA thought that design controls would reduce the number of recalls. Yet with design controls the number of recalls is still increasing. FDA should seek out the reasons for more recalls. They should subsequently devise a program to drive that number down. Industry can work with FDA to make improvement but they need FDA to lead the way. The agency does keep databases on FDA-483 trends and recalls. Sharing this information with industry through cooperative workshops could help in both parties developing solutions to many of the problems faced by FDA and industry today. I am sure with cooperation we can get there.

Finally, I would say that overall, devices today are as safe as they ever have been. Improved complaint and CAPA programs have given visibility to issues that may have gone unnoticed in the past. That is not an excuse, however, to ignore those problems. Constant work is needed to continue to make better products and to continue to inspect the industry. As some quality gurus have stated, the work never ends. You are never there because there is a need for continuous improvement.

Copyright ©2006 Medical Device & Diagnostic Industry

Sign up for the QMED & MD+DI Daily newsletter.

You May Also Like