Breaking Through the Bias: Debunking Common Misconceptions About Usability Studies

Carefully crafted user studies can give you valuable insights into the market as well as on your products.

Image source Pixabay

Usability studies—do they have purpose? For healthcare, it’s all too common to think of usability studies as a box to check to satisfy regulatory requirements. But by doing this, companies leave good information on the table, and they don’t explore the full value of user studies. Thorough, structured usability testing can be a key tool to ensuring a device or product will be valued and adopted by users.

Here we review some of the misconceptions, mistakes, and even excuses companies might encounter when it comes to usability.

I showed my network the product, and they loved it. This is a sufficient usability study.

This happens all too often. While this may get you some good information, you may not be getting unbiased, controlled data. Good user studies provide transparency regarding your product’s strengths and weaknesses. This means their results can be uncomfortable to hear. This might interfere with an entrepreneur’s or even a seasoned veteran’s ego. It often takes admitting that we don’t know what we don’t know, and this can be hard for anyone, especially if they’ve been in the industry for a long time.

There is an enormous amount of personal bias to overcome. It can be in the form of enthusiastic friends who want to support your efforts. It can be the result of unspoken pressure felt by employees or the amount of work an employee has already put in. Even if the person giving feedback is an expert in the field, there are subtleties of use that most of us aren’t aware of—humans perform tasks through a series of unconscious gestures, especially experts.

Some informal user studies are necessary and can have valuable results in the early stages of development. However, these are not validated and therefore might not yield the true potential of a product. Informal studies, when used to replace formal ones, can impact the decision process to the point of failure.

Professional usability experts aim to combat bias and provide both consistency and documentation of decision making, which is valuable for regulatory requirements, as well as uncovering more-nuanced user habits. Even if the outcome is similar to that gained through informal processes, the differences can be subtle, but they are critically important.

I won’t get real scientific data from a usability study.

There is no guarantee that a usability study will yield significant learnings. In some cases, the data only reveal patterns that may or may not be useful. Results from studies can be subtle and aren’t always what we normally think of as hard scientific data.

Companies might be tempted to skip or ignore the outcomes of a study because they don’t see how it can improve the product. But again, they are doing themselves a disservice. The science of user studies is often rooted in psychology and human behavior. And in some ways, the art of these studies is as important as the science.

Make no mistake, user studies are a science. They rely on a specific structure and involve rigorous documentation of the results to provide unbiased insight.

A good user study considers all the factors that could influence design. It is scientific in its selection of participants, recruiting users who are diverse enough to yield a wide understanding of the product but still fit a specific profile.

Further, tools are being developed to improve data collection so that the results really are better at capturing use conditions. In addition to videos, questionnaires, and environmental observation, sensors can be employed to capture use data that might help illuminate how users engage in with a product. This is an emerging science that will require effort and change on the part of usability experts.

The value in user studies is in being able to tie information back into insight that can inform design or operation of the final product.

It is never (and yet it is always) a good time to do a usability study.

This has truth. There is a real problem of knowing the precise point at which a user study will yield the best results. It might be early in the design phase, when key design decisions are being made, or it might be later, when those tests begin to resemble or coincide with marketing efforts. Too soon and you won’t get the full benefit; too late, same problem.

This is why understanding the various tools of usability is so critical. They can help a company determine the best time as well as the best tools to conduct user studies.

Some well-worn words of wisdom are “test early and often,” but various methods are best deployed at different times. Depending on the phase, you might be strategizing, executing, or assessing.

Strategizing is a beginning phase in which a company typically considers new ideas and opportunities for the future. Research methods should be both qualitative and quantitative. Tools might include field studies, diary studies, surveys, data mining, or analytics.

Eventually, a company will reach a "go/no-go" decision point when it transitions into a period of continually improving the design direction that’s been chosen. This is the execution phase. Research in this phase is mainly qualitative and includes card sorting, field studies, participatory design, paper prototype, usability studies, desirability studies, and customer emails.

Finally, a company is ready to engage in assessment, where it seeks to measure how well the product is received. This is typically quantitative in nature and might be done against the product’s own historical data or against its competitors. Usability benchmarking, online assessments, surveys, and A/B testing might all be employed at this point.

Use the testing and research until you are confident that you understand all angles of your users and the problem your product is trying to solve. It takes work, understanding, and compromise to create a marriage between your products and the world people live in.

A good indication is to consider that a large part of the value of a user study is its process of documenting and testing to gain credibility. Conducting user studies just before you need that credibility will help you move forward in a positive manner.

Conclusion

When you are lifting weights, you hire a coach to make sure you are using the proper form. The concept is easy to understand, but if you don’t know what you are doing, it’s easy to mess up. In the same way, use-trained UX researchers or usability professionals help you map out an ideal path to extract the most amount of information in the shortest amount of time. They are well versed in understanding what questions should be asked and how to achieve rich results.

It’s easy to listen to the echo chamber, but when conducted with the right attitude, user studies can give you more valuable insight into the market. The key is to create a study using scientific methods and then let go of expectations. Mindset is critical. Approaching user studies without personal bias and the appreciation for learning something new will yield the best results.

Greg Martin

Greg Martin

Greg Martin serves as the director of design for Product Creation Studio, an integrated medical product design and engineering consultancy in Seattle. There he leads a team of industrial design and user research experts to produce new and innovative hardware solutions.  His work has achieved XPRIZE accolades, design industry awards, and millions of dollars in development funding. He has worked with and led designs for start-ups and brand names to the likes of L’Oréal, Olympus, Intellectual Ventures, P&G, and beyond. Greg helps organizations approach design in a data-driven way and leverages the human-centered design process to drive innovation and make decisions.

Filed Under
500 characters remaining