Originally published January 1996
John K. Suzuki
The use of computer-controlled equipment and processes in the creation of finished medical devices has been increasing since the early 1980s. Even small companies are using software to automate and control manufacturing processes in order to increase output, improve productivity, and reduce labor costs. Although many of these systems are developed by outside vendors and contractors, it is the device manufacturers that are ultimately responsible for ensuring their safety and adequacy.
Manufacturers cannot simply assume that purchased systems have been properly documented and validated, since the vendors and contractors may be unaware of the FDA guidelines for computer-controlled devices and software used in a good manufacturing practices (GMP) environment. These guidelines specify that there must be documented evidence to support the various assumptions of the systems and manufacturing operations.
FDA can take adverse steps against a company if during a GMP inspection it finds the software validation documentation incomplete or inadequate. Severe violations may lead to the temporary closure of production facilities while the manufacturer makes corrections and updates documentation. Proper software validation, therefore, should lead to less-troublesome inspections. The primary benefit of proper validation, however, is that bugs and errors are identified early in the software life cycle, which in turn reduces development and maintenance costs. This article provides small manufacturers of medical devices with practical information on the software validation activities and documentation necessary to meet current GMP requirements.
WHICH SYSTEMS REQUIRE SOFTWARE VALIDATION?
Despite the availability of reference documentation and standards on developing high-quality software, many small device manufacturers are using systems that are not properly validated or have incomplete documentation. The questions that these companies might ask are, Which computer-controlled systems require software validation? and, What documentation is appropriate for software system validation? The main FDA document that provides guidance on validation is the Medical Device Good Manufacturing Practices Manual.1 The agency also has produced a document titled "Application of the Medical Device GMPs to Computerized Devices and Manufacturing Processes" to provide guidance on validating processes controlled by computers.2 This document states that the GMP requirements apply to computerized devices, the control of computerized manufacturing, quality assurance systems, and automated systems for recordkeeping. The guidance itself does not provide a specific matrix of required documentation.3 It has also been suggested that validation may be necessary for the following:
* Special processes that cannot be evaluated in the final product.
* Systems or processes that may affect product safety.
* Processing events that require monitoring to ensure efficacy.
* Equipment, processes, or software that are custom designed.
* Equipment, processes, or software with unknown reliability.4
The level of evaluation and documentation required for validation depends on the criticality and level of concern for the device, process, or system in question, which must be determined by the manufacturer using it. One approach is to group the various systems, devices, and processes into the three categories shown in Table I.5 Category I systems are those that do not directly affect the safety, uniformity, or performance of the product or manufacturing process and those processes whose reliability is easily confirmed with standards and calibration. Category II systems are those that do affect the performance or uniformity of the product but do not require full software validation; installation qualification or operational qualification may be all that is necessary. Category III systems directly affect product safety, or are custom designed, or have a reliability level that is unknown or suspect. These systems require a comprehensive software validation effort as well as extensive safety-related documentation. It must be remembered, however, that the emphasis of software validation should not be on producing elegant documentation, but on the content of that documentation.6
Many small manufacturers are understandably concerned with the costs and resources required to validate the software of a manufacturing process or device. One way they can minimize such expenses and efforts is to determine if any portion of the software does not affect product performance, product uniformity, or human safety. For instance, a service program that tracks spare-part inventories for a critical medical device may also contain additional software modules that track service personnel response time, part costs, discounts, and other financial data. In this case, since these modules do not affect product performance, product uniformity, or human safety, the manufacturers may want to exclude them from software validation and supply written justification for doing so. The validation plan should include written justification for the exclusion of any software modules.
The first step of the software validation process is to document the validation plan and procedures in baseline documents before validation begins. These baseline documents will serve as the basis for further development; they must be formally reviewed and accepted, after which they can be changed only through a formal change control process.
FDA's guidance "Application of the Medical Device GMPs to Computerized Devices and Manufacturing Processes" states that "manufacturers who receive software from vendors must establish a program for ensuring that the vendor has demonstrated a capability to produce quality software." There are several ways to determine if the vendor has such capability. One is to formally review the vendor's software quality assurance (SQA) program to determine if a defined and documented software quality process is being followed. An SQA plan should list the type of activities performed and documents generated during development, verification, validation, use, and maintenance of the software. It should also list any standards, practices, and conventions used on the software as well as report, track, and resolve any problems discovered during software verification and validation. Inspection of software documentation, formal audits, and interviews with the vendor's software developers may be neces-sary to verify that the company has a well-defined SQA process. Small manufac- turers can perform these assessments cost-effectively by using simple checklists and questionnaires.
Another way to assess a vendor's capability is a software capability evaluation (SCE) that uses the Software Engineering Institute's Capability Maturity Model as a standard.7 One benefit of this technique is that a small manufacturer can tailor its SCEs to each vendor or contractor for the areas of software development that it feels present the greatest risks of preventing the project from being completed on time and within budget.
The validation planning documentation can be generated by the manufacturer, the contractor or vendor, or both parties jointly. As described above, the level of detail and the type of documents that are needed are based on the risk to humans and level of concern for the process or device, which, in many cases, keeps the documentation at a manageable level. For instance, a low-risk computer-controlled process may only need a one-page software development plan and a two-page validation plan or protocol, whereas a Category III device or process may need a 10-page software development plan and a 15-page verification and validation plan because of the size of the project and the associated risk. For new Category III systems, the planning documentation might include the following components:
* A software development plan that dis- cusses the organization and responsibil- ities of the software development group, the scope of the project, the methods used to develop the software or system, the documentation generated at each phase of development, and the tools used to develop the software. This document often also includes a description of the methods or processes used to control the development activities and to ensure the quality of the final product.
* A software verification and validation plan that discusses all the steps taken and documentation created to ensure that the product is adequate for its intended use. Such steps usually include testing at the component, integration, and system levels of the product or system. Reviews, inspections, walk-throughs, and audits may also be part of the verification and valida- tion efforts.
* A software configuration management plan that describes the processes used to manage different versions of the software and documentation and changes to the software and those documents. A history of why changes were made is necessary for GMP compliance.
* Standards and guidelines that describe standard operating procedures or in-house rules that are followed to develop the software or system. Examples of such documents include coding standards, naming conventions, procedures for duplicating the software, and templates for software documentation.
Small manufacturers that use Category III systems or processes (or the vendors that provide them) can use a number of strate-gies to create these documents in a cost- effective manner. Standard templates or existing documents from other sources are a great way to jump-start the documenta- tion process, and their reuse can save considerable time and money. Such sample documents are available from a number of sources. In addition to asking their business colleagues for sample documents that are not confidential, manufacturers can obtain templates from local Software Process Improvement Networks, publications of the Institute of Electrical and Electronics Engineers Computer Society and the Association for Computing Machinery, various texts on software engineering, computer bulletin boards, the Internet, the Software Engineering Institute, and consultants. Once these documents have been generated electronically they can be reused for future development projects, allowing manufacturers and software developers to benefit from the effort and time put in previously.
FDA's "Application of the Medical Device GMPs to Computerized Devices and Manufacturing Processes" states that there needs to be "assurance that the requirements for software are clearly defined, communicated, and completely understood by the vendor." Perhaps the most difficult aspect of software development, this task is typically accomplished via a written contract or proposal from the vendor or contractor and a written software requirement specification (SRS). The SRS records what the system or software does, lists critical operations and parameters, and is used as an input to the design and testing process.
Small manufacturers or their vendors can use standard or customized templates to facilitate the requirements-generation process. Prototyping is also helpful to create, clar-ify, or reduce software requirements for a computer-controlled device or manufacturing process. A simple word processor and flowcharting tool can be used to manage the documented requirements. Another approach to simplifying the SRS documentation process is to limit the functionality of the application. This is sometimes the only way to proceed if schedules are tight and resources limited.
SAFETY AND HAZARD ANALYSIS
If a computerized system or process is critical to product performance or will have an impact on human safety, a hazard and safety assessment needs to be performed as part of the software validation. The assessment document should describe the potential hazards of the device or process, the meth-od of control, and the corrective action or processes to be used during software development.8 Some of the techniques that can be used by small manufacturers to reduce risks and hazards in software include adding error-handling capability, using checklists, performing safety inspections, and conducting informal reviews of the software.
Fault tree analysis and failure mode and effects analysis (FMEA) are among the techniques used to assess system safety.911 Several commercially available software tools can help manufacturers in creating fault tree diagrams; however, a simpler approach is to use commercial flowcharting tools with custom-designed fault tree symbols. FMEA tables can be created easily on most word processors using a table-generation command. Templates for safety and hazard analyses are also available; these can be used directly or customized to make the evaluation process more cost- and time-efficient.11
DESIGN AND ARCHITECTURE INFORMATION
The guidance document "Application of the Medical Device GMPs to Computerized Devices and Manufacturing Processes" further states that each device master record (DMR) must "include a description of how the software will interact with the hardware to accomplish various functions of the device's design . . . [and that] additional documentation that describes the design of the program [should be] maintained."(The DMR is the compilation of records containing the design, formulation, specifications, complete manufacturing procedures, quality assurance requirements, and labeling of a finished device, manufacturing process, or QA program.) The software design description is usually contained in an architecture document and a detailed design document. Again, the level of detail and description in these documents is dependent on the criticality of the device or system and its effect on human safety. Small manufacturers or vendors can use templates and sample design documents in order to minimize documentation time and costs.
The term software architecture refers to the overall structure of the system as well as the interrelationships of its various components. In other words, the architecture represents the implementation of the well- defined software requirements. Typically, the software architecture document describes the software down to the module level. A data object model is often used to list all the data requirements for the application. A diagram may be the best way to represent the architecture--Figure 1 provides an example. High-level diagrams such as that shown in the figure can be created using standard graphics or drawing programs and do not require costly computer-aided software engineering (CASE) tools. (Although a few CASE tools are relatively inexpensive, many packages cost considerably more than small manufacturers may want to spend.)
The detailed design document serves as the primary medium for communicating the system's software design and is used to facilitate analysis, planning, implementation, and decision making. Its goal is to provide a complete procedural description of each software routine or module used. Various techniques such as structure charts, box diagrams, decision tables, PDL (program design language), and flowcharts can be used to document how the software system will be structured to satisfy the requirements listed in the SRS. An example of a template for a detailed design document is given in Figure 2. This sample template can be used as a starting point to the design documentation process, and a word processor and commercially available flowcharting tool can then be used to document the design details. Small manufacturers or vendors can often create a detailed software design for one module and reuse that document or portions of it for other modules or applications, thereby saving considerable time and money.
From a practical standpoint, many small manufacturers can reduce the burden of detailed design documentation by focusing on the high-level functionality of the software and the interfaces between key software modules. The safety and hazard analysis serves as a guide to documenting the detailed design: Documentation for the safety-critical modules, functions, or interfaces may contain detail down to the actual function or code level, while the documents covering modules, functions, and interfaces that are not safety related will contain correspondingly less detail.
CODING AND IMPLEMENTATION
According to the FDA guidance "Application of the Medical Device GMPs to Computerized Devices and Manufacturing Processes," each DMR must also include "the computer source code as either hard copy or on magnetic medium." This documentation is necessary for maintaining the program and evaluating the impact of any change on other program functions. To enhance readability, a standard template or prologue is usually used for the source code, and standard wording is used to make comments useful and easy to read. Enough documentation should be included with the code to describe the subroutines or modules for the language used. A rule of thumb is that the source code documentation should contain detail and depth compatible with the complexity of the system or device involved. An example of a software module template or prologue is given in Figure 3. Reusing code and standard functions such as generic error-handling routines can improve the productivity of the programmer and make each software module more robust.
TESTING AND ERROR TRACKING
To ensure that the software is adequate for its intended use, there must be documented evidence that it complies with the specified requirements. As "Application of the Medical Device GMPs to Computerized Devices and Manufacturing Processes" states, "The preparation and use of an adequate test plan, based on knowledge of software logic and the hardware environment in which the software will run, will assure the software is adequately tested or evaluated and thereby establish confidence that it does meet specifications." Small manufacturers and vendors can use templates to aid in creating or tailoring a verification and validation plan that includes formal testing, reviews, audits, and a final certification. Once the plan is created electronically it can be easily updated for use on other projects.
The level of testing needed for validation should be tied to the risk level of the software. For example, Category III systems will usually require unit, integration, system, and acceptance testing. The results of each test during these testing phases need to be documented. A traceability matrix is usually used to show that the design meets the requirements. Reviews and code inspections are useful in finding defects and bugs before testing begins, and audits can be used to determine if the specified test plan is being followed and required documentation is being created. Such activities need to be documented as well.
"Application of the Medical Device GMPs to Computerized Devices and Manufacturing Processes" also states that "retests are necessary when the software is revised or a software failure is encountered," which means that regression tests need to be performed and documented if changes are made to the source code before shipping. The GMPs also require that manufacturers have a procedure for handling complaints, including the review, investigation, and evaluation of both hardware and software failures and a corrective action process. The software should be evaluated before an actual correction is made; after changes are implemented, regression tests should be used to determine if any other module or routine has been affected.
For Category III systems, the suggested documentation related to software testing and error tracking includes the following:
* Verification and validation documents, such as unit, integration, systems, and acceptance test procedures and results; reviews (phase and code); audit reports; and certification.
* Traceability matrices.
* Software problem and corrective action reports.
* Regression testing procedures and results.
Specialized software engineering tools are not required to have a strong validation program. Word processors can be used to create test reports, checklists, review sheets for audits, and traceability documents, and simple relational databases can be used to track and report the status of software problems or bugs. Test templates from other projects can be reused to speed the documentation of the individual test cases. For many modular products, the same test cases, or parts of them, can be used for several modules or applications, saving both money and time. Small manufacturers or vendors can use other approaches to reducing the cost of testing as well. Script-recording or screen-replay tools can be very cost-effective over time if regression testing is performed multiple times, but users may need time to learn the scripting language, and the tools are expensive. Therefore, the best approach to reducing the cost of testing is to have clear and unambiguous software requirements. It is also important to estimate the size of the application before software development actually begins so that sufficient resources can be allocated to the testing effort to ensure that it can be completed in the scheduled time.
Table II summarizes the techniques, strategies, and tools that small manufacturers can use to reduce the documentation burden and cost of software validation. Although achieving full compliance with the GMP requirements may still be expensive, it offers several benefits. For example, the company will then have a solid base of documentation and procedures that will help it satisfy such international standards as ISO 9001 and ISO 9000-3. Achieving certification to these standards can be a competitive marketing advantage, both domestically and in the European Union.
From a practical standpoint, the validation documents, procedures, and plans, if correctly created and applied, can actually save manufacturers from many costly mistakes and protect them against product recalls and plant closures after FDA inspections. Proper software validation can ease excessive testing costs, prevent costly software updates, reduce software mainte-nance efforts, and identify potential software errors before the product is released. Most importantly, a good software valida- tion program ensures that a system or process is safe for use in medical device manufacturing.
1. Medical Device Good Manufacturing Practices Manual, fifth ed, HHS Publica- tion FDA-4179, Rockville, MD, FDA, Cen- ter for Devices and Radiological Health (CDRH), 1991.
2. "Application of the Medical Device GMPs to Computerized Devices and Manufac- turing Processes," Rockville, MD, FDA, CDRH, 1990.
3. Olivier DP, "Required Documentation for Software Validation," Med Dev Diag Indust, 15(7):9498, 1993.
4. DeSain C, and Vercimak Sutton C, Valida- tion for Medical Device and Diagnostic Manufacturers, Buffalo Grove, IL, Inter-pharm Press, p 9, 1994.
5. DeSain C, and Vercimak Sutton C, Valida- tion for Medical Device and Diagnostic Manufacturers, Buffalo Grove, IL, Inter- pharm Press, pp 4243, 1994.
6. Johnson R, "Validation: Proof of Perfor- mance or Document Elegance?," Med Dev Diag Indust, 17(6):2022, 1995.
7. "Software Capability Evaluation Method Description Document," version 2.0, CMU/SEI-94-TR-06, Pittsburgh, Software Engineering Institute, Carnegie Mellon University, June 1994.
8. Murthy S, "Retrospective Validation of Medical Device Software," Med Dev Diag Indust, 17(5):186196, 1995.
9. Wood BJ, and Ermas JW, "Applying Haz- ard Analysis to Medical Devices, Part I: Initial Hazard Analysis," Med Dev Diag In- dust, 15(1):7983, 1993.
10. Wood BJ, and Ermas JW, "Applying Hazard Analysis to Medical Devices, Part II: Detailed Hazard Analysis," Med Dev Diag Indust, 15(3):5864, 1993.
11. Leveson NG, Safeware: System Safety and Computers, Reading, MA, Addison-Wesley, pp 287393, 1995.
John K. Suzuki is principal of JKS & Associates, a consulting firm that specializes in software requirements analysis and design and software verification.