A Systems Engineering Approach to Requirements Validation

Originally Published MDDI January 2002

VALIDATION

A Systems Engineering Approach to Requirements Validation

Product development risks can be reduced by validating project requirements before the design process begins.

Trace Baker

Measured in money, time, safety, and reputation, the errors that are most costly to manufacturers of medical devices are often those made early in the product development cycle. One such error—committing resources in an attempt to meet product requirements that later prove to be invalid—can be prevented by conducting a requirements validation.

Although design input is a focus of the design control sections of the quality system regulation (QSR), most of the literature on medical device development treats requirements review only superficially. There are, however, techniques developed by systems engineers for other industries that device companies can adopt. Modeling, in particular, has proven extremely effective for ensuring the validity of product requirements.

FDA CRITERIA

Unlike design validation, which is performed after a design has been implemented, requirements validation is performed before the design process starts. The relationship of requirements validation to a typical medical product development process is shown in Figure 1, which was adapted from FDA's design control guidance.¹ FDA's guidance also offers five criteria for "assessing design input requirements for adequacy"; specifically, requirements must be unambiguous and verifiable by an objective method of analysis, inspection, or testing; include quantitative limits; be self-consistent and nonconflicting; characterize the use environment; and comply with relevant industry standards. The first three items concern the requirements' validity, the last two their completeness.

It is easy to create examples of requirements that are obviously invalid under the first three criteria. Consider the statement "The deionized-water container shall have a capacity of 1 L and the weight of the filled container shall not exceed 1 kg." A liter of water weighs about 1 kg, so there is no budget left for the weight of the container. Thus, the requirements clearly conflict. Similarly, the requirement that "the instrument shall be capable of fitting through a door" is clearly ambiguous. Adding the dimensions of the smallest door through which the instrument must fit would remove the ambiguity.

Like the FDA guidance, the literature on systems engineering also emphasizes the need for eliminating inconsistency and ambiguity from requirements specifications.^2–4 Medical device manufacturers may find an article in the Handbook of Systems Engineering and Management particularly helpful.⁵ It defines the requirements validation process as ascertaining the answers to three questions:

A CASE STUDY

The systems engineering approach to requirements validation is also exemplified in the following case study, which involves the development of a fluid-delivery subsystem for a diagnostic device. To use the device, a laboratory technician places each patient sample in a separate well of an eight-well block, which is then handled as a unit. Four requirements, R021–R024, were suggested to specify the delivery to the wells of the reagent needed to process the samples:

These requirements use the common definitions for the sample meanand standard deviation (s) for a sample size of 30:

The need for these requirements is understandable. Because variation in performance is inevitable in a physical system, limits to that variation have to be defined by what the chemistry can tolerate and still produce correct results.

Common medical industry practice since the implementation of design controls is to collect requirements, write a requirements document, and then conduct a requirements review. Would this set of hypothetical requirements pass a review using the first three items listed in the FDA guidance? All four requirements can be tested by simple gravimetric measurement or by inspection, and all four include quantitative limits. They also appear consistent; each addresses a separate aspect of fluid delivery, and they are related to one another through a series of equations. Finally, if there is conflict among the requirements, it is not immediately evident. It seems logical that if the subsystem can meet error requirement R022 for a series of 30 consecutive dispenses, it will be capable of doing so for the eight consecutive dispenses demanded by R024. However, using a systems engineering approach to requirements validation reveals that this assumption is not necessarily true.

A systems engineer might begin validating this set of requirements by posing the question, Is the proposed fluid-delivery subsystem capable of meeting requirement R024 given that it can meet R022 and R023? He or she would then attempt to determine the answer by constructing the simple mathematical models of the three requirements described below. (These analyses are based on two assumptions: that the dispenses are statistically independent, and that dispense error is distributed normally.)

The first model considers a limited case where the subsystem is operating with a consistent level of dispense error and coefficient of variation that fall within the R022 and R023 requirements: specifically, an error of +1.5% and a coefficient of variation of 0.8%. With the constant error of +1.5%, the average dispense volume will be 60.9 µl, which is high but still within specification. This case is shown graphically in Figure 2, with the height of the curve representing the probability that any single dispense will have the volume given on the abscissa. The shaded area under the curve is the probability that the volume dispensed in one well will meet the error requirement in R022. For the combination in this example, the probability of eight consecutive wells meeting R022 is 0.97⁸, or approximately 0.78, meaning that as many as 22 of 100 tests attempted may fail to meet this requirement because of the random variation that is allowed by R023. This level of performance is clearly unacceptable for a diagnostic device.

The general case that covers the entire specified range of parameters allowed under R022 for error and R023 for coefficient of variation can also be modeled. In Figure 3, this range is enclosed by the box, and the probability that the subsystem will meet requirement R024 at any allowed combination of error and coefficient of variation is shown by the colored surface within the box. If the subsystem were capable of meeting R024 under all specified conditions, the probability would be 1 over the entire boxed-in area and the top surface of the box would be a uniform color. As it is, the surface lies below the required probability over much of the area.

The models reveal that, in this set of requirements, R024 is not valid, and there is a simple explanation for this result. Requirement R024 states that "all wells in a single block must meet the error requirement R022." Since every well is part of a block, the unintended requirement is that every well in the device must meet the ±2% error requirement. This is inconsistent with R022, which was formulated to apply to a sample of 30 wells, not to individual wells. Can a practical system be built that operates only in the colored area of the response surface? Perhaps, but at what price?

IMPLEMENTING REQUIREMENTS VALIDATION

If the requirements validation modeling described in the case study had not been performed, the consequences of accepting the set of four requirements for the diagnostic device's fluid-delivery subsystem probably would not have been discovered until design validation or production testing. The cost would have been extensive rework and revenue lost to delays in introducing the product to the market. Although requirements validation itself may seem like a time-consuming exercise, device manufacturers can add it to the product life cycle without sacrificing rapid development as a project goal by heeding the following advice.

A checklist for preparing requirements documents appropriate to medical device development can be compiled for a project by consulting systems engineering books and articles.^2–9 In the sample checklist that follows, the questions apply to both the entire set of requirements and to individual requirements.

CONCLUSION

Following a systems engineering approach to requirements validation compels developers to address more issues than the five criteria given by FDA for assessing design input adequacy. Yet the benefits of such increased scrutiny make the expenditure of time and effort worthwhile. By validating requirements before design begins, manufacturers can guard against the need for unexpected rework late in the development cycle or after the product enters the marketplace. The resulting savings in time and money—and even reputation—can be significant.

REFERENCES

1. Design Control Guidance for Medical Device Manufacturers (Rockville, MD: Food and Drug Administration, 1997).

2. I Sommerville and P Sawyer, Requirements Engineering (Chichester, England: Wiley, 1997).

3. BP Douglass, Doing Hard Time (Reading, MA: Addison-Wesley, 1999).

4. RS Pressman, Software Engineering, a Practitioner's Approach (New York: McGraw-Hill, 2001).

5. AT Bahill and FF Dean, "Discovering Systems Requirements," in Handbook of Systems Engineering and Management, eds. AP Sage and WB Rouse (New York: Wiley, 1999).

6. Standard for Application and Management of the Systems Engineering Process, IEEE Std 1220-1998 (New York: Institute of Electrical and Electronics Engineers, 1999).

7. Systems Engineering Handbook, release 1.0 (Seattle: International Council on Systems Engineering, January 1998).

8. [Sub]system Requirements Analysis Phase, JPL D-4003, version 3.0 (Pasadena, CA: Jet Propulsion Laboratory, 1988).

9. Systems Engineering Handbook, vol. 1, MSFC-HDBK-1912 (Huntsville, AL: Marshall Space Flight Center, 1991).

Trace Baker works for Colorado MEDtech in Boulder, CO, where he is a senior principal systems engineer.

Copyright ©2002 Medical Device & Diagnostic Industry