Software Risk Management: Not Just for the Big Manufacturers?

Medical Device & Diagnostic Industry Magazine
MDDI Article Index

An MD&DI June 1997 Column

Cover Story

Risk management is not as complex or expensive as many small device manufacturers may think. In fact, it can actually save money.

The use of software to control both the manufacture and operation of medical devices is rapidly increasing. As the need for such specialized medical software grows, so too does the importance of following rigorous and thorough software development methods. Whether the developer is a large firm with deep pockets or a small one with very little to spend on research and development, medical software must be safe for its intended use.

Manufacturers can ensure that software will be safe by systematically identifying and eliminating software risks before they become either threats to successful operation or major sources of software rework.¹ Regulatory officials recognize the need for such deliberate risk management. For instance, FDA's September, 1996, 510(k) draft guidance contains a section that spe-cifically addresses software risk management and requires that it be integrated into the software development life cycle. The International Electrotechnical Commission's 601-1-4 standard, released in 1996, also addresses software risk management for programmable medical devices. The standard requires the preparation of a documented risk management plan and the application of risk analysis and risk control throughout development.

Risk management can not only help ensure that the final product is safe, but can also be used to determine whether a development project will be finished on time and on budget. Cost overruns and scheduling delays are two of the largest risks for any software project, and can be especially devastating for a small manufacturer.

Despite the seeming consensus on the merits of risk management, however, many small medical manufacturers do not include it in their software development projects. There are several possible reasons why they don't. Manufacturers may have an exaggerated idea of the complexity of the math involved in risk management. They may believe that software risk management should only be performed on large software systems, or feel that formal software risk management will add too much to an already full development schedule. They may lack discipline in applying the methods, or doubt whether the techniques will have value, or simply lack knowledge of or training in software risk management.

And it is not just misunderstanding or misinformation that keeps many small manufacturers from instituting formal risk management. Most available risk management methods are too complex or expensive to be practical for any but the largest corporations. For small manufacturers, a risk management method would have to be simple and cost-effective to be practical. In order to be not only practical but also ideal for small manufacturers, it would have to meet several criteria: It would have to be customizable. It would have to be mostly qualitative in nature, and quantitative if needed. It would have to apply to the full life cycle, including the maintenance phase. It would have to be easy to update as the development project progresses. And it would have to be compatible with any development model, such as spiral, waterfall, joint application design, recent application design, incremental, or prototyping.

Figure 1. The relationship of software risk elements to risk factors.

One risk management technique that meets these criteria is the Software Engineering Risk (SERIM), Model, which can be applied by anyone who is familiar with spreadsheet applications and who reads this article.² SERIM uses risk questions to derive numerical probabilities for a set of risk factors. These numerical values are then analyzed using spreadsheets that have been programmed with particular statistical equations. To explain this quantitative analysis process clearly, some basic concepts, such as risk management activities, risk elements and factors, and measuring risk are addressed in the following sections.

RISK MANAGEMENT ACTIVITIES

Risk management can be organized in six general steps.

1. Identification. Finding and recording project-specific risks. The identified risks can be generated by a number of techniques such as checklists, comparison with experience, common sense assessments, and analogies to well-known cases.

2. Risk Strategy and Planning. Creating plans and alternatives for addressing each identified software risk and coordinating these plans with other documents, such as the software development plan.

3. Risk Assessment. Categorizing each risk and determining its relative magnitude. This step includes deciding whether the risk will be managed or ignored during the development process.

4. Risk Mitigation/Avoidance. Taking the steps that have been planned either to reduce each risk or to avoid it.

5. Risk Reporting. Formally reporting the status of the risks that were identified by the first four steps.

6. Risk Prediction. Using historical data and estimates to forecast risk. The information gathered in the previous five steps is used to derive the new risk predictions.

RISK ELEMENTS AND RISK FACTORS

Risk Elements. The three major elements of software risk are technical, cost, and scheduling problems. Technical risks, such as maintainability or reusability, are associated with the overall performance of the software system. Cost risks, such as budget or fixed costs, are associated with the cost of the software system. Scheduling risks, such as overlapping projects with shared resources, are associated with completion of the software system during development.

Risk Factors. Software risk factors can be thought of as more specific subcategories of the three general risk elements. Risk factors are more closely relat-ed to software issues.³ A software risk factor can be a subcategory of more than one risk element (see Figure 1). Each software risk factor can have an influence on each risk element. That influence can be categorized as low, medium, or high.

For the SERIM model, Karolak has identified 10 important risk factors.⁴

1. Organization. The maturity of the organization's structure, communications, functions, and leadership.

2. Estimation. The accuracy of the estimations of resources and schedules needed during software development, and their costs.

3. Monitoring. The ability to identify problems.

4. Development Methodology. The methods by which the software is developed.

5. Tools. The software tools used when software is developed.

6. Risk Culture. The management decision-making process in which risks are considered.

7. Usability. The functionality of the software product once it is delivered to the end-user or customer.

8. Correctness. Whether the product suits the customer's needs.

9. Reliability. How long the software performs for the customer without bugs.

10. Personnel. The number of people used in development and their abilities.

Figure 2. The relationship of risk questions to software risk elements and risk factors.

Table I shows the influence of the 10 software risk factors on each of the software risk elements. Each factor can have a different influence on each element. For instance, using a poor estimation technique may have little technical impact on a software project but lead to faulty prediction of the size and cost of a project, and, therefore, greatly extend the development schedule. From a risk element perspective, the influence of the estimation technique's risk factor on a project can be characterized as high on cost and schedule, but low on technical risk. Software risk factors can also be classified by their importance, either major or minor, to product or process risk (see Table II).

Table I. The effect of software risk factors on risk elements.

Table II. The effect of software risk factors on software categories.

MEASURING RISK

Risk Questions. One approach to assessing risk is to use risk questions to measure risk factors. Answers to the questions can be recorded in a yes-or-no format (0 or 1) or as a numerical range of possible responses. Response ranges can use any numerical value from 0 through 1. For example, a response range could be defined as none = 0, little = 0.2, some = 0.5, most = 0.8, and all = 1.0.

Figure 2 shows the relationship of risk questions to risk elements and risk factors under the SERIM model. Note that each risk question can have a relationship with multiple risk factors, and there can be multiple questions for a given risk factor. The number of questions for each risk factor is determined by its relative importance in predicting success or failure in a software development project. The number of questions can be increased or decreased based on the characteristics of a project.

The types of questions asked for each risk factor are usually based on industry trends, data, publications, and observations of both successful and unsuccessful software development projects. For each risk factor, a range of questions can be developed. They can be identified by the first initial of the corresponding risk factor and a number. For the organization software risk factor, for example, an O is used to identify each question.⁵

O1. Are you using or do you plan to use experienced software managers?

O2. Has your company been producing software similar to this in the past?

O3. Is there a documented organizational structure either in place or planned?

O4. Is the organizational structure stable?

O5. What is the confidence level of your management team?

O6. Does good communication exist between different organizations supporting the development of the software project?

O7. Are software configuration management functions being performed?

O8. Are software quality functions being performed?

To devise a numerical range of responses to question O1, for example, a 0 could indicate that only managers who have little or no software engineering experience will be used on the project. A rating of 0.5 could indicate that a management staff of both experienced and inexperienced software engineering managers will be used. A rating of 1.0 could indicate that only managers who are experienced in software engineering will be used on the project. The lowest end of the scale actually represents the highest risk for a risk factor.

Additional questions to cover the risk factors of estimation (E1­E7), monitoring (M1­M7), development methodology (DM1­DM7), tools (T1­T9), risk culture (RC1­RC11), usability (U1­U6), correctness (C1­C9), reliability (R1­R12), and personnel (P1­P5) are listed in Karolak.⁶ The SERIM model includes 81 questions for the 10 software risk factors.

Table III. Correlation of risk management activities and organization risk factor questions.

Mapping Risk Questions to Risk Management Activities. One way to ensure that questions are comprehensive and cover the scope of the software risk management activities is to correlate their risk factors to those activities. For example, Table III shows the organization risk questions related to the six risk management activities. The table shows that none of the eight organization questions covers the reporting activity. This information may indicate that additional questions to cover risk reporting should be formulated for this factor.

Table IV. Integrated risk management activities correlated with software development life cycle phases.

Mapping Risk Management Activities to the Software Development Life Cycle. An integrated risk management approach to software development should be able to predict risks across any phase of development. The six risk management activities should be present throughout the software development life cycle (see Table IV). Typical development phases include prerequirements, requirements, designing, coding, testing, delivery, and maintenance.

Within each phase of development, the six risk management activities are also evaluated using the risk questions. Tables can be generated for each of the software development phases to show the relationship of those six activities and the relevant risk questions.

ANALYZING RISK DATA: THE SERIM MODEL

As the data on risk factors are gathered using the risk questions, they can be entered into any commercial spreadsheet application. The SERIM equations and parameters can be easily programmed into spreadsheets to create templates for data analysis. When the data are entered into the templates, factors can be analyzed during or before each development phase, or at any time needed. The model can be updated, expanded with more questions, or otherwise modified as more information is gathered during development. This flexibility makes SERIM well suited for small as well as large development projects.

Model Parameters. The SERIM model uses simple probabilities to assess potential risks. The basic parameters of the calculations in this model include:

1. P(A) is the probability of event A.

2. The probability of event A ranges from 0 to 1.

3. The probability of the sample space = 1, and the probability of no outcomes = 0.

4. If A₁, A₂, ... A_n are a sequence of mutually exclusive events, then P(A₁ * A₂ * ... A_n) = P(A₁) + P(A₂) + ... P(A_n). Restated, the probability of a sequence of mutually exclusive events is equal to the sum of the individual probabilities.

SERIM assumes that the probabilities are assigned by previous experience or by analogy to past events. Because this process is subjective, the probability assigned to a specific event may vary at different times in the software life cycle. It may also vary depending on the individuals who come up with the assignment. The numeric values used in the SERIM model are set by the responses to the risk questions. Simple probability trees are then used to calculate a risk statistic for each risk factor, which is a weighted average of all the responses to all the risk questions associated with that factor. This statistic is expressed mathematically as P(A) = w₁P(A₁) + w₂P(A₂) + ... + w_nP(A_n) where w_n is the weight for each probability.

An Integrated Software Model. To describe SERIM as an integrated model, the six software risk management activities are related to specific questions that span a particular development phase for a project. In turn, each software development phase is connected to a set of software risk questions. These questions are related to specific software risk factors, which relate to the three software risk elements. These risk elements then constitute the total risk for the project.

In this model, risk is represented as a probability tree. P(A) represents the overall probability of a successful development project. P(A₁), P(A₂), and P(A₃) denote the individual probabilities for success in each of the three software risk elements. P(A₄) to P(A_n) represent the probabilities for the software risk factors. P(B) through P(M) represent the probability of success of the project based on the software life cycle phase and the six software risk management activities.

Figure 3. Model relationships between software risk categories and risk factors.

Figure 3 illustrates the relationships between the software risk categories and risk factors for both process and product. P(N) is the probability of project success based on a specific software process. P(O) is the probability of project success based on product quality.

The Analysis Equations. The basic equations used for the SERIM model are a series of probability trees for each parameter. Each equation is dependent on the number of questions for each software risk factor and the relative weights placed on each question. The main sets of equations that can be easily entered into a spreadsheet are given below. For instance, the probability of event A occurring is derived using Equation 1.

3P(A) = [  P(A_n)]/3n = 1

This equation assumes that all risk elements are equal in weight. P(A) is the probability of a successful project. If the weight of each element differs between them, then P(A) =w₁P(A₁) + w₂P(A₂) + w₃P(A₃), where w₁ is a positive number and w₁ +w₂ + w₃ = 1.

The probability of a risk element 1 (technical risk) is given by Equation 2.

13P(A₁) = [  w_nP(A_n)]n = 4

Where w₄ = 0.043, w₅ = 0.043, w₆ = 0.087, w₇ = 0.087, w₈ = 0.087, w₉ = 0.13, w₁₀ = 0.13, w₁₁ = 0.13, w₁₂ = 0.13, and w₁₃ = 0.13. This equation assumes that a weight of 0.043 is assigned for a low value, 0.087 for a medium value, and 0.13 for a high value. The probability of risk elements 2 and 3 are calculated with the same formula, but with different weights.

The probability of P(A₄), the organization risk factor, is given by Equation 3.

8P(A₄) = [  P(O_n)]/8n = 1

where O_n is the numeric value of risk question n for the category of organization. The probabilities of P(A₅) through P(A₁₃) are calculated with the same formula, but the upper limit of the summation and divisors for the number of questions will vary for each risk factor category.

The probability of P(B), the prerequirements phase, is given by Equation 4.

P(B) =    (O1, O2, O3, O4, O5, E1, E2, E3, E4, E6, E7, M1, M2, M3, M4, M6, M7, DM1, DM2, DM6, T1, T6, T9, RC1, RC2, RC3, RC4, RC5, RC6, RC7, RC8, RC9, RC10, RC11, C5, P1, P2, P3, P4, P5)/40

where the elements of the sum are the values for the corresponding risk questions. The probability values listed in this formula all address the relationship between the prerequirements phase and the six risk management activities. The probabilities of P(C) to P(M) are calculated with the same formula, but the particular questions and the divisor will vary for each phase of development.

The probability of P(N), the process, is given by Equation 5.

13P(N) =[  w_nP(A_n)]n = 4

where w₄ = 0.125, w₅ = 0.125, w₆ = 0.125, w₇ = 0.125, w₈ = 0.125, w₉ = 0.125, w₁₀ = 0.04, w₁₁ = 0.04, w₁₂ = 0.04, and w₁₃ = 0.125. The formula assumes that a weight of 0.04 is assigned for a minor influence and 0.125 for a major influence. The value derived from this equation represents the probability of project success using the current software process.

The probability of P(O), the product, is given by Equation 6.

13P(O) = [  w_nP(A_n)]n = 4

where w₄ = 0.045, w₅ = 0.045, w₆ = 0.045, w₇ = 0.045, w₈ = 0.14, w₉ = 0.14, w₁₀ = 0.14, w₁₁ = 0.14, w₁₂ = 0.14, and w₁₃ = 0.14. The equation assumes that a weight of 0.14 is assigned for a major influence, 0.045 for a minor one. The value derived represents the probability of product success.

SAMPLE APPLICATION OF THE SERIM METHOD

A real-world example will best illustrate how SERIM can be used. The method was used to determine the risk of developing a custom software package to control the assembly and testing of an invasive cardiovascular device. The software, which was written in C, contained about 110,000 lines of code, and the application resided on five modular PC-controlled stations linked with a moving conveyor line. Each station controlled various assembly and testing steps. Data collection and acceptance testing of the device were also controlled by the application.

The original schedule projected that development of the software would take about 13 months, including testing and complete FDA validation. Commercially available development tools were used for compiling and debugging.

Table V. SERIM probability assessment for the real-world software development example.

The probability assessments based on the responses to the SERIM risk questions are shown in Table V. The results represent the risks that were determined at the beginning of the proj-ect based on the original contract and the estimated schedule of delivery to the customer.

Interpretation of the Probability Results. The probability of successful delivery P(A) for this example is 0.60. This low value of success is attributed to a number of software risk factors. The low value of 0.16 for the risk factor of estimation, P(A₅), was returned because no software cost or schedule estimation data from similar projects were used for this project.

The highest responses, 0.96 and 0.97, were for the software risk factors of correctness,P(A₁₁), and reliability, P(A₁₂). The correctness result related to requirements, design, code, test traceability, and the low expectation of new or changing requirements. There were very few changes or additions made to the software requirements during the development effort. The reliability result was associated with error and fault handling in the code, the use of software reliability modeling, proper types of testing, and defect tracking. All these activities were performed on this project.

The model also identified a major risk that could occur during the prerequirements phase, P(B). This risk was related to the use of very few data from similar and previous projects in the development of the schedule and budget. The lowest probability from the six risk management activities (0.28) was in the area of risk strategy and planning, P(I). This suggests that few plans and alternatives were created for schedule and cost risks. The project had few schedule or cost alternatives other than reducing functional requirements and adding people to the development and testing effort.

As it turned out, the risks of budget and schedule overruns were realized. The project actually took about 20 months to complete and was over budget. The model also predicted that the testing would be laborious because no automated tools or regression tools were used. The actual project data showed that testing, verification, and validation took about twice the time the initial schedule had predicted.

CONCLUSION

The SERIM method is a simple and flexible way to perform software risk management. It is particularly well suited for small manufacturers that may not be able to use more expensive and complex processes. Besides determining the risks of current projects, the method can predict risks for future projects by using benchmark data. It also allows updates throughout the development cycle. Numeric response values can be changed easily, and probabilities automatically calculated with an off-the-shelf spreadsheet application. The number and type of risk questions can also be customized to reflect the type and size of a project as well as any other specific project concerns. SERIM integrates well with other conventional project management tools, and it uses the development phases most software developers are already accustomed to using.

SERIM can also help users satisfy FDA and IEC requirements for risk management. It can be part of the risk management file system and risk management plan required for IEC 601-1-4. It can be used in the risk analysis, risk control, and risk estimation stages of development. IEC 601-1-4 is currently being harmonized with the FDA 510(k) requirements, so compliance to the international standard will also ensure compliance with the 510(k) process. SERIM can help manufacturers meet the quality system regulation's design control requirements for software validation and risk analysis.⁷

Risk management need not be prohibitively expensive or complicated for small medical device manufacturers. In fact, using it early in the process can actually reduce costs by identifying a project's vulnerabilities before they become disasters. Using a simple and cost-effective risk analysis method such as SERIM will help small manufacturers incorporate this important activity into every software development project.

REFERENCES

1. Boehm B (ed), Software Risk Management, Los Alamitos, CA, IEEE Computer Society Press, 1989.

2. Karolak DW, Software Engineering Risk Management, Los Alamitos, CA, IEEE Computer Society Press, 1996.

3. Karolak DW, Software Engineering Risk Management, Los Alamitos, CA, IEEE Computer Society Press, p 44, 1996.

4. Karolak DW, Software Engineering Risk Management, Los Alamitos, CA, IEEE Computer Society Press, pp 44­49, 1996.

5. Karolak DW, Software Engineering Risk Management, Los Alamitos, CA, IEEE Computer Society Press, pp 52­54, 1996.

6. Karolak DW, Software Engineering Risk Management, Los Alamitos, CA, IEEE Computer Society Press, pp 52­75, 1996.

7. 21 CFR 808, 812, and 820, "Current Good Manufacturing Practice (CGMP); Final Rule," Federal Register, October 7, 1996.

John Suzuki is owner of JKS & Associates (Laguna Niguel, CA), a software consulting firm, and Dale W. Karolak, PhD, is a consultant (Brighton, MI).

Copyright ©1997 Medical Device & Diagnostic Industry