SOFTWARE

In the past decade, there has been a sea change in the business software domain. Many companies are no longer expending significant internal resources to develop software from scratch. Instead, they are opting for software that meets most or all of the business requirements as delivered “off the shelf” by a third party. Commercial off-the-shelf (COTS) software is an extremely broad category that encompasses software that can be purchased and used with minimal or no configuration. There are virtually unlimited types of COTS software. Some examples include resource planning applications, customer relationship management tools, and quality system databases for CAPA, complaint handling, auditing, and document control. It also comprises laboratory information systems, accounting software, and software embedded in medical devices.

There are many benefits to using COTS software. Foremost, it is the vendor that expends the resources to design, develop, test, and support the software. Often, the software vendor also has extensive expertise in the target market for the software and thus is able to incorporate functionality into the software to support best-practice methodologies. For certain types of COTS software, the vendor will also provide hosted software, which eliminates the need for the buyer to purchase and maintain servers and supporting hardware. Altogether, this can allow a company to implement tailored software more quickly and cost-effectively. But there are potential downsides to using COTS software as well.

One of the fundamental limitations of implementing COTS software is that a company typically does not have direct control over the software's feature set, including what functionality is added, changed, or removed with each release. As a result, the company may not be able to dictate the schedule for the incorporation of business- or compliance-critical functionality in the system. In addition, the release schedule of the software is also usually determined by the software vendor. The timing could lead to a forced upgrade of the software, either because the prior version is no longer supported or because the latest version will be pushed to all clients simultaneously, such as in a multitenant, hosted environment. Lastly, the company is reliant on the vendor for the software system's technical details, which can be critical to successful application integration and interfacing. Such details are also important for the troubleshooting and resolution of software issues that arise.

Although COTS software can virtually eliminate internal software development activities for a business, it presents a unique set of challenges and does not obviate the need to ensure compliance through software validation and procedural controls. One of the more common compliance challenges for businesses in the medical device industry is in the validation of COTS software. This article focuses on the validation of business and quality system COTS software. It also discusses other aspects of implementing such COTS systems that can have a significant effect on a company's business.
COTS Validation Decision
The first step in deciding whether to validate a COTS software system is to understand its intended use. Per FDA's General Principles of Software Validation guidance, software must be validated if it is used to achieve compliance with predicate rules (e.g., 21 CFR 820, 21 CFR 1271). The guidance states:
When computers or automated data processing systems are used as part of production or the quality system, the [device] manufacturer shall validate computer software for its intended use according to an established protocol.
FDA further states that “off-the-shelf software may have many capabilities, only a few of which are needed by the device manufacturer…When device manufacturers purchase ‘off-the-shelf' software, they must ensure that it will perform as intended in their chosen application.” This means that regardless of how the software vendor defines the intended utilization of the system, each business should independently define its specific use and ensure that this use is validated.
If a software system or functionality within the system is not used for compliance with FDA regulations, a company may not need to perform validation of the whole or parts of the system. In other words, the breadth of the validation is driven by the functionality used to comply with regulatory requirements. An example is an accounting package, which is used only to capture financial business transactions. Although the system is important to the business, the tracking of finances is not required by FDA regulations; consequently, FDA does not require it to be validated. Even so, some companies may choose to validate all software systems because it makes good business sense to ensure that systems operate per specifications and to set a consistent company expectation regarding software validation activities.
COTS Validation Approach
Once the high-level decision has been made regarding validation, there are two specific elements that need to be addressed: the system's actual compliance with applicable predicate rules, standards, and general regulations as well as its formal validation.
Although the software vendor may strongly assert that its application is compliant with FDA regulations and other standards, it is prudent for the device company to verify compliance. Ideally, this verification is performed before purchasing the system, but it absolutely must happen before moving the system to production use. Again, the system should be compliant with applicable elements of the predicate rule and should also adhere to 21 CFR Part 11 requirements.
In addition, a company may decide to demonstrate adherence to other industry standards such as ISO 13485. A compliance assessment should be performed for each applicable regulation and standard. Relevant elements of the applicable regulations and standards will also be incorporated into the system requirements so that they are validated. Upon moving the system to production use, it would render an entire validation effort meaningless if the system is not also compliant with applicable regulations and standards. Stated another way, validation establishes that the intended use of the system is satisfied, and the intended use inherently must include compliance considerations.
As an example, a complaint system used by a medical device company must be compliant with 21 CFR Part 820.198, which covers complaint files. The use of the system to approve and store official electronic complaint records would also necessitate compliance with 21 CFR Part 11. The specific subsections of Part 820.198 that are related to functions performed within the software are captured as requirements against which validation is performed. To limit downstream risk, a compliance assessment is performed at the time of software selection. If gaps are found during the compliance assessment, the company can use the findings to direct additional vendor development or configuration activities, or it can seek a different, more compliant product.
The basic methodology for validating COTS software involves the following activities, which are performed in concert with implementation tasks such as system configuration:
•Define company's use of the system, ideally including use cases and explicit clarification of in-scope and out-of-scope functionality.•Determine validation deliverables set based on system type, system risk, project scope, and degree of system modification.•Review the existing vendor system and validation documentation.•Devise strategy for validation that leverages vendor documentation and systems as applicable.•Create applicable system requirements specification and design documentation.•Generate requirements-traceable validation protocol and execute validation.•Establish system use, administration, and maintenance procedures to ensure that the system is used as intended and remains in a validated state.
If the vendor has documented system requirements, the company needs to assess them to ensure that its needs are met. If there are requirements beyond those documented by the vendor, the company needs to create a separate requirements document. The validation testing will then need to cover and trace to these requirements. If this scenario applies, the firm should determine whether any of the vendor requirements and testing can be leveraged toward its own validation. Remember that even if all the requirements are covered in the vendor and company documentation, it must be easy to trace and explain. Otherwise, it can be difficult to demonstrate compliance.
Vendor As Critical Business Partner
After covering the top-level aspects of COTS software selection and validation, it is almost equally important to assess the software vendor's supporting processes to determine the vendor's capabilities. This step can help ensure long-term satisfaction with the system. Unless a company purchases a single version of the software and never updates it, it must rely on the vendor's operations on a continual basis. Much of the total cost of ownership of a software system can be associated with production use and updates of the system after its initial implementation, so knowledge of a vendor's overall support capabilities and software update practices is critical. For this reason, companies must thoroughly evaluate a vendor both qualitatively and quantitatively, and preferably before any agreement has been reached. Use the selection process to thoroughly qualify the vendor. In addition to following standard supplier qualification processes, which take into account the risk and criticality of the software for a company, the following areas should also be considered.
General Responsiveness. Submit several requests for data during the selection process and meet with as many people as possible at the vendor to ascertain its level of engagement, commitment, and expertise. Does the vendor do what it says it will do? Be wary of vendors that don't admit to any deficiencies in their systems or operations. Be on the lookout for instances during the sales process where a prospective vendor incorrectly communicates the capabilities of the software or the support functions or doesn't follow through on commitments to provide more information. Communication problems during the sales and qualification process are unlikely to go away after a contract is signed; they often serve as a warning for future challenges.
COTS Degree. A company's relationship with the vendor will be somewhat dependent on the degree to which the software is off-the-shelf. For instance, word-processing software purchased at the local computer store has a different level of vendor support than an COTS quality system software product. The target consumer of the system dictates the level of personalized customer service provided, and recognizing the “degree of COTS” allows the implementation, validation, and maintenance paths to be charted appropriately.
Client Mix. Consider the vendor's mix of software clients (e.g., how many clients, whether they are large or small, which markets they serve, the percentage of clients in each market, etc.). This information can help determine the uniqueness of your company's use of the system. For financial reasons, the vendor logically shapes its software to satisfy as much of its client base as possible. The mix of clients and markets served dictates the future path of the software and support processes. Ideally, a company is similar to the majority of the vendor's clients in terms of functional and regulatory requirements that are satisfied by the system. Companies should understand the vendor's commitment to serve their market segment. If, for example, a company is small and has unique needs in a market segment to which the vendor is not committed, the company should assess competitive offerings from other vendors that may better match its size and specific needs.
Hosted Application Sharing. If the software system is Web-based and hosted, it is important to understand how many clients access the same hosted deployment of the application. The number of clients on a hosted deployment influences the flexibility the vendor has regarding software release dates and times. If it is important to have control over the production release of each new software version, a company should make arrangements to have a dedicated instance of the hosted application so that no other companies can influence the release schedule.
Software Update Frequency. Try to determine how often the vendor releases new software versions as well as whether each new release requires an upgrade. The typical release schedule dictates ongoing validation activities, which can significantly contribute to the total cost of ownership. This schedule should be taken into account for internal resource planning purposes. The best-case scenario is that a company can control whether and when system updates occur. Also, vendors will often apply bug fixes and hot patches to the software behind the scenes with little or no notice. These are changes to the system to fix glitches and should not change its intended use. Ideally, the vendor alerts the client to bug fixes and hot patches so that the company can assess how these elements affect software functionality. What the vendor deems an insignificant bug fix may be more than that to a client.
Software Update Previews. Companies should see whether the vendor is willing to provide a preview of each new release before it is moved to production. This is a key concern if the software is a hosted, Web-based product because the client often has little control over the timing of the production release of the software. Preproduction availability of the software allows a company to become familiar with new software functionality, evaluate its effect on the established intended use, and perform any necessary validation activities before the production release.
Software Release Notes. Detailed software release notes help a company assess whether and how a functional change to the software affects its use of the software. If there are multiple modules in the software and a company does not use or validate all of them, it is particularly important that the release notes break out changes by module. The vendor should also provide user and administrative documentation (i.e., manuals or help files) that walks through all aspects of system use as well as administrative functions to be performed by users.
Key Software Controls. In addition to the basic quality system elements that allow the software vendor to ensure the ongoing quality of its products and services, pay particular attention to the following processes: change control, configuration management, and environment deployment.
Change control ensures that software application updates are assessed and integrated in a systematic fashion. The vendor should be able to demonstrate how system change requests are logged, reviewed, and linked with a release, as well as how decisions are relayed to the requestor. Firms should ensure that the vendor understands the technical, regulatory, and business effects of each proposed change.
Configuration management controls the elements of the software system and its documentation so that they are identifiable and synchronized, and so that only approved versions are moved to production. If the vendor is performing technical configuration or customization to the system on a company's behalf, the vendor must be able to tightly control and deploy updates to the company's system. Besides reviewing procedures, there are other practical ways to assess the effectiveness of change control and configuration management. One method is to review the system change requests submitted by internal and external resources. The requests should be thoroughly documented and there should be evidence of review, disposition, and feedback for each request. The implemented changes should be linked to a version of the software.
Environment deployments are often the most difficult aspect of implementing and updating a database application. Discrete environments should be used for development, test, and production use. Configuration changes are typically made in development and promoted to the test and production environments. The promotion process needs to take into account, either automatically or manually, that there may be existing master data that should not be altered when updating the production environment. Using the test environment solely for acceptance and testing ensures that test or sample data are not included in an environment promotion.
Technical Troubleshooting and Root Cause Analysis. Inevitably, there will be problems with the software system that occur over time. As noted previously, a company is reliant on the COTS software vendor for the software system technical details. During the assessment, review previously reported issues and how they were resolved. The current software technology requires the coordinated performance of hardware, operating systems, vendor software, Web browsers, printer drivers, etc., so it is essential that the vendor not only demonstrate technical expertise across these interconnected domains but also a commitment to truly resolving technical issues that affect its customer base.
Part 11 Knowledge. Is the vendor aware of and well versed in 21 CFR Part 11? A vendor that is unfamiliar with Part 11, which covers electronic records and signatures, may make changes to the software over time that unintentionally affect its Part 11 compliance. (Note that not all COTS software is subject to Part 11 requirements.)
Software Configuration versus Customization
COTS software is typically configured by the client company. Configuration often involves activities such as adding system users, populating field drop-down lists, adding entities, etc. But it can also involve renaming data fields to allow the process to better match an existing client process. Software customizations, which are sometimes required to ensure compliance or to match a client's defined business process, involve branching the code from the base set. As a result, customizations frequently require additional development coding each time a new release of the software is installed, which can result in the expenditure of development resource time with each release and increase the time required to perform validation. Limiting client software changes to configuration allows the client to more easily upgrade the software with each new release.
Conclusion
Companies can use COTS software to tap into established industry best practices while avoiding the need to design and develop custom software. The upfront costs of implementation can also be reduced by selecting hosted, Web-based software package. However, using an outside software vendor includes giving up some level of control. To mitigate this risk, companies should perform upfront vendor assessment activities, including both qualitative and quantitative reviews.
In concert with a standard supplier quality assessment, a company should also understand key parameters such as the degree to which the software is off the shelf, the vendor's client mix and demographics, the vendor's approach to hosted software as applicable, the frequency of and communications related to software updates by the vendor, the vendor's key software controls, and its knowledge of 21 CFR Part 11. The software buyer is still responsible for ensuring compliance through software validation and procedural controls, so all of these factors should be taken into account when calculating the true cost of software ownership.
References

1. General Principles of Software Validation; Final Guidance for Industry and FDA Staff (Rockville, MD; FDA, January 11, 2007).

Bryan Chojnowski is a director of quality for Reglera LLC (Lakewood, CO).
Copyright ©2009 Medical Device & Diagnostic Industry