Medical Device & Diagnostic Industry Magazine [5]
MDDI Article Index [6]
Originally Published January 2000
An extensive validation project includes a range of tests and analyses designed to determine whether or not QSIT in fact meets user needs and intended uses.
In today's competitive global market, it has become increasingly more important for FDA to review how it accomplishes its mission of protecting and promoting public health at a minimum burden to the industry as well as to itself. One of the better ways of realizing that mission is to ensure that "good" products get into the market as quickly as possible and that "bad" products are kept out of the market. A key means by which FDA achieves this goal is by challenging the adequacy of the quality systems of medical device manufacturers through inspections.
Shrinking FDA resources, a growing inventory of medical device manufacturers, and a corresponding increase in the complexity of process and product technology were among the reasons the Center for Medical Devices and Radiological Health (CDRH) decided to reengineer the process used for quality system inspections.
The product of that reengineering effort is the quality system inspection technique (QSIT) [for an overview of QSIT, see QSIT: The New Quality System Inspection Technique, [7] MD&DI October 1999]. QSIT is a top-down inspection approach that focuses on four of the seven subsystems within a quality system—management controls, design controls, corrective and preventive actions (CAPA), and production and process controls (PAPC). The three remaining subsystems (records, documents, and change controls; equipment and facility controls; and material controls) are inspected during a QSIT inspection via linkages or pathways from the four targeted subsystems.
The development of QSIT was unique because it was the product of a team comprising staff members from CDRH as well as from the Office of Regulatory Affairs (ORA), both headquarters and field units. The team also made a concerted effort to obtain input from industry. The successful coordination and execution of the project demonstrated that FDA could conduct a major development effort with both internal and external stakeholders.
Another unique aspect was that the QSIT team attempted to apply design controls to its own design and development efforts. This included validating the finished QSIT work product.
Any new design—whether a new medical device, a novel style of car, or an innovative inspection technique—is created when users or customers who need a product for certain intended uses realize that what they have available is inadequate. Innovative companies or organizations then work to create a product that their designers believe will meet those user needs and intended uses. In design validation the finished design or work product is evaluated to see if it indeed meets the necessary criteria. In other words, the product is evaluated to see if it is "fit for use."
In validating QSIT, the needs of FDA's internal and external stakeholders and the intended uses of the work product were expressed in terms of goals. The goals were to decrease inspectional time, increase inspectional focus, more closely harmonize the inspection technique with that used in the international community, and provide inspectional coverage of the quality system regulation.1 These goals were to be achieved without compromising the agency's mission of consumer protection. The results or consequences of achieving the goals were also projected and expressed as outcomes. It was believed that use of QSIT would increase inspectional consistency, manufacturer compliance, product quality, and the efficiency of the enforcement action review process.
Validation requires confirmation by examination and the provision of objective evidence or "proof." To determine whether or not the QSIT work product met user needs and intended uses, several validation activities in the forms of tests and analyses were planned and conducted during the QSIT predeployment phase. Each activity had a "champion" who volunteered from the ranks of the QSIT design team or from other CDRH and ORA offices. Representatives from the regulated industry also formed an ad hoc group that championed a validation activity.
A total of 25 validation activities were carried out in accordance with specified protocols. Because there were many ideas being put forth on how goal and outcome accomplishment should be measured, each proposed protocol was required to identify the strengths and weaknesses of the particular activity. The proposal was also required to state the reasons why the activity represented one of the best possible approaches for measuring the accomplishment of the goal or outcome.
QSIT STUDY
Design validation requires testing under actual or simulated use conditions. As part of the QSIT validation, a study was conducted in which inspections were actually performed using QSIT.
During the period from October 1, 1998, through February 18, 1999, a total of 42 study inspections were conducted in FDA's Denver, Minneapolis, and Los Angeles districts. A cadre of 12 QSIT-trained investigators, who represented a cross section of experience, performed the study inspections. There were four investigators from each of the study districts. Three QSIT-trained compliance officers—one from each study district—evaluated the inspectional findings. Throughout the study, data were captured through routine reporting and evaluations by the investigators and compliance officers and through surveys of the compliance officers and the inspected firms.
The selection of the firms to be inspected during the study was left up to the districts. However, each district was asked to select a variety of firms, both in terms of size and types of products produced. A range of firms was inspected during the study, from small companies with annual sales of less than $25,000 to manufacturers with sales in excess of $50 million. While the majority of those inspected had 75 or fewer employees, the inspected companies also included medium- (76–200 employees) and large-sized firms (201–500 employees).
A variety of products was covered during the study inspections, including in vitro diagnostic products. During some of the inspections, more than one product type was covered. For example, an OB/GYN product may have been selected to evaluate a company's production and process controls while a general and plastic surgery product may have been selected to challenge the firm's design control system.
GOALS
Decrease Time. One of the major goals of QSIT was to decrease the in-plant time as well as the total time for conducting a comprehensive quality system inspection.
To assist investigators in conducting inspections more efficiently, the QSIT Inspection Handbook—later renamed the Guide to Inspections of Quality Systems—was developed.2,3 It was written to be user-friendly and contains guidance tools such as lists of inspectional objectives, flowcharts, sampling tables, and narrative sections explaining how to accomplish each objective. The investigators evaluated the tools contained within the Handbook during each inspection and stated that they found them useful.
The target was to conduct a comprehensive quality system inspection within an average in-plant time of 1 week. The average in-plant times required to conduct assessments of the four main subsystems, as well as the total time, are shown in Figure 1. The average in-plant time for conducting an inspection using QSIT was found to be 28.2 hours. Because investigators are not typically on-site at a firm for a full 8-hour day, an in-plant day was defined as 6 hours. This equated to just under 5 days of in-plant time to conduct an inspection using QSIT. Although FDA's normal time-reporting systems do not capture in-plant time, the calculated in-plant time using the non-QSIT approach was 67.1 hours. This meant that use of QSIT resulted in a direct in-plant time reduction of 58%.
Figure 1. QSIT in-plant time. (Non-QSIT in-plant time = 67.1 hours.)
One aspect of QSIT that affects in-plant time is the preinspection record review. Under QSIT, if the inspection is preannounced, the investigator will ask the firm to voluntarily provide certain high-level quality documents prior to starting the inspection. During the study, this preinspection record review was conducted 66.7% of the time and, when conducted, took 4 hours. Since this is time that under the non-QSIT approach would have been expended in-plant, it was necessary to factor that time back into the hours of the QSIT in-plant time in order not to bias the QSIT approach. Even after factoring the preinspection record review time into the QSIT in-plant time, the time reduction using QSIT was still a very significant 54%.
Each time the preinspection record review was used, the investigators were asked whether it increased the efficiency of the inspection. A full 96% of the responses indicated that the review did increase inspectional efficiency. In addition, 94% of the firms responding to the survey stated that providing such records facilitated the inspection process.
The total inspectional time reported by investigators is a summation of three figures: the time required to prepare for the inspection, to conduct the inspection, and to write the inspectional report with the associated documentation. Using the non-QSIT approach, it took an average of 98.6 hours to conduct a comprehensive quality system inspection, including an assessment of design controls. The QSIT inspections took an average of 55.2 hours—a time reduction of 44%.
It was the opinion of 84% of the industry survey respondents that the use of QSIT resulted in a more-efficient inspection by FDA. The responses noted benefits such as the following: "Did not require excessive amount of time away from day to day activities"; "In just a few days I knew what work I had to do"; or "We spent less time in the audit procedure by light reviews of areas we had strengths in and emphasizing our weaknesses." Similarly, 81% of investigators also thought that QSIT was more efficient.
The investigators filled out evaluation forms following each of the QSIT inspections they performed. A shift in their responses was noted as they conducted more inspections. For instance, after conducting his or her first inspection, an investigator would sometimes either not respond to a question about the efficiency of QSIT, or would respond with an answer of "not sure" or an ambivalent "yes/no" type of comment. After the investigators had conducted their second or third inspection, however, they tended to move to a definite "yes" response.
Increase Focus. Increasing the focus of the inspectional approach was another major goal of QSIT. The intent was to bring about a "zeroing in" or "honing in" on the adequacy of a company's compliance with the key elements of the quality system regulation.
Accomplishment of this goal was measured in several ways. One way was through an industry survey question. The inspected firms were asked, "Did QSIT focus on the key elements of your quality system?" Fully 100% of the companies answered "yes." One respondent stated that QSIT "provided clear focus for the investigation and helped provide insight in areas of improvement for the firm." Another said that "it immediately directed us to areas we need to improve," while a third noted that QSIT "allowed us to pull the appropriate documents quicker, with less confusion on the direction the audit was moving."
Overall, 88% of the investigator evaluations concluded that use of QSIT resulted in a more-focused inspection. As with the question regarding overall efficiency, this was another example of the responses shifting toward a definite "yes" as the investigators became more familiar with the technique.
Besides increasing the focus of the overall inspection, another QSIT goal was to increase the focus of inspectional observations on FDA-483 items if they were noted. In other words, the goal was to make any FDA-483s that were issued more meaningful or substantive.
Flowchart steps in the QSIT Inspection Handbook were designed to correspond to the key elements of the quality system that were to be evaluated during the inspection. An analysis was performed of the FDA-483 items that resulted from the QSIT study inspections to see how well they matched the flowchart steps: it was found that 89% of the 200 FDA-483 items matched the steps directly. The remaining items were primarily associated with document deficiencies and were found through linkages while inspecting subsystems such as CAPA and PAPC.
FDA-483s were issued to 67% of the firms that were inspected during the study. This means that 28 of the 42 firms inspected using QSIT had deficiencies that were significant enough to warrant placement on an FDA-483. A breakdown of the FDA-483 items and the subsystems with which they are associated is shown in Figure 2. Of the four major subsystems inspected, management controls had the most deficiencies (57 of 200). This was a significant finding, given the importance of the management controls subsystem.
Figure 2. FDA-483 items per subsystem (200 total items).
Management is the "glue" that holds the subsystems together, providing the infrastructure for the quality system and serving as the engine that drives the entire quality system process. Management provides direction and "nourishment" to ensure that the quality system remains healthy and effective. In other words, management is responsible for establishing—that is, for defining, documenting, and implementing—the quality system. Under QSIT, the FDA-483 items show that management is being held accountable for this responsibility.
The subsystems associated with the 10 most frequently cited FDA-483 items for non-QSIT inspections are depicted in Figure 3. Half of the items are associated with deficiencies within the CAPA subsystem; there are no management deficiencies among the top 10. By contrast, QSIT study inspections presented a different picture. As shown in Figure 4, management deficiencies appear at a 40% level. CAPA deficiencies are still frequent, appearing at a 30% level.
Figure 3. Top FDA-483 subsystem areas for non-QSIT inspections.
Figure 4. Top FDA-483 subsystem areas for QSIT inspections.
We have seen that QSIT was shown to take less time and to be more focused than the non-QSIT approach. But was QSIT as effective in finding deficiencies if they existed? Figure 5 compares the inspectional classifications for QSIT versus non-QSIT inspections. It was found that 21% of the QSIT study inspections were determined to be "official action indicated" (OAI). The comparable violative rate for non-QSIT inspections for FY 98 was 19%. In addition, a slightly higher percentage of QSIT inspections were classified as "voluntary action indicated" (VAI) compared with the VAI percentage for non-QSIT inspections. Correspondingly, the "no action indicated" (NAI) rate for QSIT inspections was less than for non-QSIT inspections.
Figure 5. Classification of district decisions for QSIT and non-QSIT (FY 98) inspections.
More Closely Harmonize. Two important features of QSIT that more closely harmonize this FDA inspection technique with that employed by the international community are the use of a top-down as opposed to a bottom-up approach, and the use of the preinspection record review.
A survey of the inspected companies found that 74% of respondents agreed that the QSIT approach was similar to that used by auditing organizations utilized by their firms (e.g., notified bodies, third-party assessors, etc.). Respondents indicated that the similarity was useful; as one noted, "We are ISO 9001 certified, and QSIT allows us to standardize our approach to all processes and achieve full compliance for both ISO and the QSR." Another respondent commented that it is "easier to explain our quality system to the auditors/inspectors when there is a common focus."
Coverage of the Quality System Regulation. One of the basic goals of the reengineering project was to design an inspection approach that would provide broad coverage of the quality system regulation when conducting a comprehensive quality system inspection. As part of the validation effort, a detailed gap analysis was performed in which the QSIT approach was compared with the quality system regulation. The analysis revealed that minor gaps were found. Sections of the quality system regulation were not assessed in QSIT directly via inspectional objectives or narrative discussions, or indirectly through linkages. As a result, a revision of the QSIT Inspection Handbook was undertaken to fill in the gaps.
An industry ad hoc group also compared the QSIT approach to the regulation. The group evaluated whether or not QSIT adequately addressed the requirements of the regulation and assessed the quality system. It was the consensus of the group that QSIT did in fact achieve these objectives.
OUTCOMES
The QSIT validation subteam also took on the task of attempting to measure the accomplishment of the predefined QSIT outcomes. While it was more difficult to objectively measure such accomplishments, particularly during the predeployment phase, data were generated that indicate that the outcomes will be met.
Increase Consistency. It was projected that use of the QSIT approach would increase the consistency of quality system inspections across investigators and thereby among districts. This means that an inspection done in one district would be consistent with an inspection conducted in any other district.
The structure of the QSIT approach was compared with that of the non-QSIT comprehensive inspection technique. It was found that QSIT provided for a more well-defined, succinct, and prescriptive methodology for inspection. For example, QSIT can accomplish a comprehensive inspection with approximately 26% fewer tasks while using 67% fewer references. QSIT also promotes consistency by reflecting contemporary requirements and providing end points for the review of records.
Another important question was whether the investigators actually followed the QSIT approach during the study. Activities designed to measure this involved a review of the FDA-483s issued during the study as well as the establishment inspection reports (EIRs) associated with the study inspections. Review of the FDA-483s found that they matched the QSIT inspectional areas; review of the EIRs found that they documented coverage of the four major subsystems. Thus, the acceptance criteria for these activities were met.
Increase Compliance. Another projected outcome was that use of QSIT would result in an increase in compliance of medical device manufacturers with the quality system regulation. In the Blair House Papers, issued by President Clinton and Vice President Gore, there is a discussion of the relationship between regulators and those they regulate. It states: "Not everyone is going to play by the rules. But experience shows that most businesses and communities do want to comply and will, if they can figure out what it is they're supposed to do."4
The draft QSIT Inspection Handbook was made available to industry. (The final Guide to Inspections of Quality Systems [8] is available on the FDA Web site, in PDF format.) The QSIT design team believed that promoting the information contained within that reference was one way to increase the industry's knowledge and understanding of the requirements of the regulation, and that an increase in knowledge and understanding would in turn lead to an increase in compliance. As part of the industry survey performed during the QSIT study, the inspected firms were asked if they thought that using QSIT would increase the medical device industry's knowledge and understanding of the requirements of the quality system regulation. More than 94% of the respondents stated that they thought it would. One respondent stated that "This was the most straightforward and understandable inspection I have ever had." Another found that "The QSIT Inspection Handbook provides insight into FDA's expectations with respect to the quality system regulation, and therefore gives the industry detailed guidance."
The inspected firms were also asked directly whether they thought that use of QSIT would result in improved industry compliance with the quality system regulation. The majority of respondents, 67%, stated that they thought it would. Specific comments ranged from "The better the understanding of the requirements, the better the compliance with the quality system regulation," to "Our experience with QSIT did help our compliance with [the] quality system regulation."
Improve Product Quality. An ultimate outcome of the QSIT process is to improve the quality of medical devices. A majority of firms responding to the survey (63%) stated that use of QSIT would result in an improvement of the quality of products manufactured by the medical device industry. "Constantly improving quality systems yield improved products," commented one respondent. Another stated that "Focus on the quality system subsystems and improvement in those should lead to improved quality, much more reliably than the 'bottom up' approach in correcting defects."
Improve Review Efficiency. The final projected outcome deals with improving the efficiency of the enforcement action review process. QSIT EIRs and FDA-483s were reviewed by the compliance officers participating in the study, who evaluated them and made mock compliance decisions using the Draft Compliance Program Part V criteria. Following each of their reviews, they were asked if the investigators' focus on key areas helped make their review easier; 95% answered "yes." They were also asked if the QSIT tools were helpful during their reviews. Their responses indicated that the majority of the time they found the tools to be helpful.
At the conclusion of the study, the compliance officers were surveyed. It was their opinion that overall the QSIT EIRs were better organized, easier to review, easier to evaluate, and generally of higher quality than non-QSIT EIRs. They especially noted that the documentation of the violations was more thorough, and that they expended less time in reviewing the QSIT EIRs.
CONCLUSION
Through the conduct of these various tests and analyses and the results generated, it was concluded that the QSIT work product was validated to meet the project goals and user needs and intended uses. The evidence required to tip the scales in favor of moving from the predeployment phase to full-scale launch of this new inspection technique had been obtained and documented. The predeployment data also indicated that the projected outcomes will be met.
As regulators, we must continuously strive to be more effective in promoting and protecting the public health. Sometimes trying to fix or merely improve an existing process or approach isn't enough, and we need to have the courage to start over from scratch. It took wisdom and vision on the part of senior FDA managers in CDRH and ORA to initiate reengineering of the medical device inspection process, and it took teamwork, creativity, diligence, drive, and determination on the part of the numerous individuals both inside and outside FDA to make it happen. The success of this project—along with the ongoing commitment by senior FDA managers to increase effectiveness—makes us confident such efforts will continue not only in the inspection area but in others as well.
ACKNOWLEDGMENTS
We would like to thank the QSIT team members, FDA and industry validation activity champions, QSIT study investigators, compliance officers and district managers, validation evaluation group members, industry survey respondents, and all our enthusiastic supporters who contributed their efforts to make this validation project a success.
REFERENCES
1. "Quality System Regulation," Code of Federal Regulations, 21 CFR 820.
2. QSIT Inspection Handbook (Washington, DC: Food and Drug Administration, October 1998 draft).
3. Guide to Inspections of Quality Systems (Washington, DC: Food and Drug Adminstration, August 1999).
4. W Clinton and A Gore, Blair House Papers, National Performance Review (Washington, DC: U.S. Government Printing Office, January 1997).
Georgia A. Layloff (glayloff@ora.fda.gov [9]) and Robert G. Ruff ( rruff1@ora.fda.gov [10]) are core members of the QSIT team and participated in and contributed to all aspects of the development of QSIT. As FDA investigators, they are stationed in the Kansas City district (St. Louis branch) and New Jersey district offices, respectively. Timothy R. Wells (trw@cdrh.fda.gov [11]) is the leader of FDA's quality system inspections reengineering team. He is a branch chief in the CDRH Office of Compliance in Rockville, MD.
Return to the MDDI January table of contents [12] | Return to the MDDI home page [5]