How To Generate Usability Requirements and Conduct Usability Testing

What is the best way to approach the vital step of usability testing? An expert shows how device makers can understand user needs, generate usability requirements, and perform the necessary assessments.

Dean Hooper

In many organizations, iterative usability testing and sound product design has become a widespread concept to the point of being considered a tautology. The consumer product industry views it as a competitive advantage and regulated product makers are feeling pressure to comply with human factors principles during product development. Indeed, FDA, for one, is requiring many medical device producers to demonstrate product safety and usability characteristics in order to market their products. This is done through simulated use and traditional usability testing.

However, product development teams and managers remain concerned about the resources, both time and people, needed to include user-centered activities, as well as the best approach to assess error and its impact. Product managers want to know what to test, how many iterations to perform, and how to know when formative studies are finished. A concrete, agreed-upon definition of "finished" would allow for planning while making the results of usability testing understood and owned by the entire team.

The generation and testing of usability requirements provide the team with usability success (and failure) criteria for making a determination as to when the design is "good enough" and to ensure with confidence that potential errors have been investigated and evaluated.

Figure 1 summarizes the life cycle of usability requirements from initial user needs to final design inputs. The activities outlined in red are iterative in nature. As empirical usability data is collected, requirements may be refined and additional errors exposed for further evaluation. The mitigations resulting from the uFMEA are then translated to design inputs.

Figure 1. Flow of usability requirements through product development.

Through the use of a case study, this paper will begin with the definition and examples of usability needs and goals, description of activities used for the formation of use functions and testable requirements, how requirements drive testing scenarios, and, ultimately, how they define design controls for iterative testing and design decisions.

Case

A conceptual blood pressure monitoring system was used as an example. A main component of the system is a smart phone app that allows the patient to monitor levels real time via a blood pressure gauge, detect hypo- or hypertensive conditions, record activity and related information, and send information to the clinic. 

User Needs

Just a (very) few words about user needs. In any one industry all products are designed to meet a (very) finite set of needs. These needs must be solution free and describe the intended goals of the end user. Given this definition I propose the following set of needs for all medical devices.

User Needs 1 through 4 relate to the clinical outcome itself.

     1. Don't hurt yourself, another, or the patient.

     2. Monitor and diagnose accurately.

     3. Maintain or improve the intended patient's condition.

     4. Determine efficacy and/or compliance of previously prescribed therapy.

User Needs 5 and 6 relate to personal needs given the specific malady.

     5. The treatment shall be discrete and not impose on patient privacy.

     6. The treatment shall be administered in a timely fashion to maximize the efficacy of the therapy.

Functional Analysis

The next step after solution-free needs are formalized is to determine what the user must do to meet those needs. These functions are usually defined with a solution in mind. Indeed, and as illustrated in Figure 1, the technology chosen may drive the formation of functions. For example, the decision to use a cellphone app will add a user responsibility to maintain and charge the cell phone battery as needed. In the case of a blood pressure monitoring smart phone app, the following functions were identified:

1) Battery Monitoring and Charging. In order for the system to function properly, battery power must be at an operating level. The battery must be charged from time to time depending on usage.

2) Application Set-up. The system software must be tailored for the needs of the individual patient. This is accomplished via software settings.

3) Log Events. While not crucial for the use of the system, the patient user may log significant events into the application.

Usability Requirements

Based on the functional analysis and the resulting set of user functions, a set of system usability requirements or goals are generated. These requirements must be stated so as to make them testable. For more on this, explore the US standard on usability requirement generation. In order to reach established usability goals, each requirement must be met by all adequately trained users on the first attempt with the aid of instructions for use, if necessary. Any observed error shall be subject to an analysis to determine the nature of design mitigations, if needed. This subsection contains the usability requirements subjected to verification. They are organized by their respective function. The appropriate user must be able to:

1) Battery Monitoring and Charging.

2) Application Set-up. The system software must be tailored for the needs of the individual patient. 

3) Log Events. The patient user must be able to enter relevant information concerning a specific pressure reading into the application. These events may be:

Task and Human Failure Analysis and Testing Scenarios

The identified usability requirements may then be used as use goals in a goal oriented hierarchical task analysis. Each goal is evaluated on the discrete tasks and sub-tasks required to achieve the goal. This exhaustive task analysis is then used as the foundation for a use error analysis. Figure 2 illustrates part of this analysis. Again this is an incomplete analysis for illustrative purposes. Here is a detailed description of a human failure assessment process. 

Figure 2. Partial Human Failure Analysis

Once task and human failure analyses are complete, the design team can formulate usability testing scenarios to verify the requirements. The tasks and sub-tasks may be used as a guide to evaluate testing performance if use difficulties do occur.

Usability vs Safety

Most industries such as mass consumer electronics would conduct a more formal requirements verification test as formative studies. Each requirement is transformed to a usability goal by adding success criteria. For example, such a criterion could be that 90 percent of test users successfully perform the task related the requirement. If this level is achieved, then the usability goal is reached and the design is formalized. If the goal is not met, the design is iterated and retested, if needed, until the usability goal is met.

In the case of regulated medical device development, the only criterion can be a 100% success rate. While requirements can drive scenario development and subsequent testing, any errors must be evaluated for cause and potential risk. That is, the requirement becomes less usability goal and more a guide for testing scenario generation and subsequent evaluation of user performance. However, both usability goal verification and use error testing can be conducted simultaneously.

Dean Hooper, principal at HE Consulting, provides user-centered design expertise to all phases of product development, from initial needs gathering to final product validation. He can be reached at [email protected]

[Image courtesy of STUART MILES/FREEDIGITALPHOTOS.NET]