Understanding FDA Thinking: Risk-Benefit Determinations in Design Control

On August 15, 2011, FDA published a draft guidance, "Factors to Consider when Making risk-benefit determinations in Medical Device Premarket Review"1 for FDA reviewers and industry. The guidance states that the purpose is "to provide greater clarity for FDA reviewers and industry regarding the factors the agency considers when making risk-benefit determinations during the premarket review process for certain medical devices." A recent FDA publication has identified particular categories of products as subject to greater risks.2 In another recent publication, FDA has stated that the draft guidance “describes for the first time the criteria CDRH uses to make such determinations….”3 This means that the practices in the draft guidance are already in place for FDA product approvals. Therefore, each medical device company should use this new draft guidance as a basis to reexamine risk management aspects of its design control system for fine-tuning to help ensure the smoothest possible pathway to product approval. 21 CFR §820.30 provides a useful framework for the reexamination.

General

Perhaps the most important message of the guidance is implied, rather than stated. By explicitly emphasizing the use of risk-benefit determinations in the product approval process (even if for a limited subset of products), FDA is drawing further attention to the need for a full integration of risk management principles into all aspects of design control.

Risk management principles are clearly stated in ISO 14971:2007.4 Further advice on implementation can be found in PLUS 14971, a publication of the Canadian Standards Association.5 Guidance on incorporation of these principles into a quality management system can be found in a Global Harmonization Task Force document.6

In reexamining a design control system for fine-tuning, particular attention should be given to the requirement in ISO 14971 (3.2) that top management must define policy for determining criteria for risk acceptability. Although the FDA risk-benefit guidance does not and cannot specify exactly a boundary between acceptability/approvability and unacceptability, a company may wish to review its policy on criteria for risk acceptability to ensure consistency with FDA expectations.

Design and Development Planning

At a minimum, the most current version of this guidance document should be made part of the "required reading" for each product development team as it starts planning. To make it easier for engineers and scientists on product development teams to use the advice in the guidance, it would be worthwhile for each company to recast the material into a more succinct version using terms in common use within the company.

The Reviewer Worksheet attached to the guidance should provide a framework for one aspect of the planning. Recognizing that reviewers will expect to find certain information related to product risk-benefit, the team should ensure that this information is available in a clear, straightforward manner in the submission. In instances where reviewer worksheet information is available for prior submissions for related products, this information should inform the planning process. Similarly, to the extent that information is available in the form of publicly available summaries of decisions on similar products from other companies, this information should also be used. With regard to these prior submissions the team should also keep in mind the FDA advice (section 3.3) that risk-benefit weightings can change with time.

If a device is first of a kind, planning is particularly important to ensure that the risk management documentation fully addresses possible risks. In such cases, the development team should look beyond design control and ensure that plans for collecting post-production information (ISO 14971:2007, section 9) are active and not just passive.

Design Input

As often emphasized, time spent early in the project, during creation of the design input, is time that can save large amounts of money by increasing significantly the chances of success both in the approval process and in the market. In this draft guidance the FDA implicitly reminds each development team that in addition to knowing the product under development and carrying out a risk analysis for that product, the team needs to:

One of the examples based on actual FDA risk-benefit determinations in the past provides a cautionary note best heeded at the beginning of the development process. We learn of a permanently implanted cardiovascular monitoring device intended to diagnose heart failure. Its use reduced the number of hospitalization days by three, but the implantation required two days of hospitalization, and other devices with similar benefits did not require implantation. FDA determined the benefits did not outweigh the risks. Without knowing further details, one must assume that this product did not receive adequate attention in early design stages and that a more complete assessment of the downside potential for the product might well have led to cancellation of the project with considerable savings to the company.

Similarly, the guidance describes a hypothetical example for a diagnostic test where the clinical studies were not conducted on the true target population for the product and therefore "FDA is not likely to approve." The predictive value of a positive result depends heavily on the incidence of the disease in the population tested. In this case, the population tested was largely a diseased population, and therefore the results appear much more favorable than they would in the general population if the test is to be used for screening as intended. This is another (in this case hypothetical) case where early analysis of research data and disease incidence should dictate not carrying the product forward into development.

Design Output

The Risk Management File (RMF) is a component of the design output critical to success in the approval process and ultimately critical to the technical success of the product in the market. In addition to the documentation collected for risk-benefit analysis related to design input as noted above, the guidance serves as a reminder that the RMF should include documentation that:

Although the Reviewer Worksheet appears to indicate that the FDA reviewer should be thinking of "ways to mitigate the risks," it should be the goal of the development team to have already considered all appropriate mitigations prior to the drafting of the submission.

Design Review

If necessary, company procedures or guidelines should be modified to ensure that company design reviews address potential FDA concerns prior to any submission to the FDA. This, of course, has always been a goal of design reviews, but now the FDA guidance document provides improved insight into those potential concerns.

Note the question at the end of the reviewer worksheet: "Do the probable benefits outweigh the probable risks?" This is essentially an abbreviated version of ISO 14971:2007 section 7 "Evaluation of overall residual risk acceptability." Provided that the principles of risk management have been incorporated throughout the design control process, a late stage design review will address the required review leading to the Risk Management Report, and there should be sufficient documentation to ensure that the FDA reviewer will be able to give a positive answer on the Reviewer Worksheet.

Design Verification

Recognizing that most clinical studies for medical devices last less than a year, the FDA guidance emphasizes the importance of use simulation studies to prove claims for long-term functioning of the device. These studies deserve prominence among the verification studies.

The guidance provides an example where apparently the simulation studies for an implanted birth control device did not test all of the possible situations that should have been tested. The x-rays for some patients in the clinical studies showed that the implanted device had fractured—something that had not occurred in any bench or animal studies. With more complete simulation studies the device might have been redesigned to avoid fracture and gain product approval.

Design Validation

As with other guidance information about design validation, this guidance identifies a variety of clinical and non-clinical scientific evidence that the agency may accept. The list of types of clinical studies has the appearance of being in a rank order, with studies more readily able to provide reliable scientific evidence listed first. It is logical to assume that FDA reviewers prefer studies capable of stronger scientific evidence; however, this is not stated explicitly in the guidance.

Much of the guidance document addresses the benefits and risks as actually observed during the clinical evaluation(s) for validating the design. It is clear that, in many situations, products have received FDA approval even in cases where the clinical data were not entirely straightforward, provided the agency was convinced that the benefits for the product outweigh the risks. Nevertheless, it is logical to assume that the approval process was more time consuming and difficult in such instances. To facilitate the approval process, there are some lessons to be learned. Companies need to ensure that:

Design Transfer

One aspect of design control not addressed in the guidance document is design transfer. Despite the silence in the guidance, ongoing attention to the risks associated with design transfer, particularly for outsourced manufacturing, is critical to an effective risk management process. Mitigation of risks related to outsourced activities should be given careful attention in the risk management file. In the case of a PMA application, the premarket inspection can be expected to address the subject, and the routine inspection process can address questions related to other products at any time.

Design Changes

Design change for risk mitigation is of course an integral part of the product development process. Normally, design change is undesirable once the product approval submission has been made, although the guidance does mention design change as an option for mitigation of harms.

It is apparent from the guidance that FDA wants to approve new technology when it will bring real benefit to patients with manageable risks. The agency notes that “With subsequent iterations of the device its risk-benefit profile may improve, the expected level of safety and effectiveness may increase, and later versions may offer significant advantages over the initial device.” This suggests that design change for further risk mitigation will be an important consideration even after initial approval of a device incorporating new technology.

Design History File

Although the Design History File (DHF) is not mentioned in the guidance explicitly, it is clear that its role as primary documentation for product approval information is central to success. The information in the DHF must be organized so that the persons responsible to prepare the product submission can readily find the required information. It is probably worthwhile for each company to define a required DHF structure (relating to FDA expectations as shown in the draft Reviewer Worksheet) in sufficient detail so that the needed information can be efficiently pulled together by those drafting submissions. In this way the risk management principles incorporated into the design control process can be best communicated to the agency for their risk-benefit assessment during product approval.

References

1. FDA Draft Guidance, “Factors to Consider when Making Benefit-Risk Determinations in Medical Device Premarket Review,” issued August 15, 2011. 

2. FDA CDRH, “Understanding Barriers to Medical Device Quality,” October 31, 2011.

3. FDA CDRH, “Medical Device Pre-Market Programs: An Overview of FDA Actions,” October 19, 2011. 

4. ANSI/AAMI/ISO 14971:2007, “Medical devices – Application of risk management to medical devices”

5. Canadian Standards Association, “PLUS 14971, The ISO 14971:2007 essentials – A practical handbook for implementing the ISO 14971 Standard for medical devices,” November 2007. 

6. Global Harmonization Task Force, Study Group 3, “Implementation of risk management principles and activities within a Quality Management System,” May 20, 2005. 

William I. White, PhD is the founder of Quality System Strategies LLC (Elkhart, IN), where he serves as a senior consultant.