Ethnographic Research and the Problem of Validity

DESIGN RESEARCH

In 2001, William Reese and I wrote an article about the use of ethnographic research for the development of new medical devices.¹ In that article, we described what ethnographic research is and why it is a useful tool for identifying user wants and needs for medical devices. The article looked at the limitations of device users' descriptions of what they do and what they need, and it explored how ethnographic research can transcend those limitations. It also discussed how ethnographic research differs from conventional market research in employing direct, real-world observation of device use.

That article examined how ethnographic interviews, which take place in the environment of device use, can yield richer and more-accurate information than interviews conducted at a neutral site (and usually conducted long after the behavior of interest has taken place). Finally, it described some of the characteristics of ethnographic research, such as spending significant amounts of time in the environment of use, working to develop rapport with informants, and carefully considering the context in which procedures take place.

The use of ethnographic research has since become much more common in medical device development. However, its relative ubiquity raises an important issue: How can ethnographic research achieve validity? By validity, I mean the degree to which the research findings accurately describe the real-world facts that they purport to describe.

The issue of validity does not always apply to ethnographic research, at least not to all research to which the term ethnographic is applied. Much so-called ethnographic research—perhaps most of it—is designed simply to generate ideas, that is, to stimulate creativity. Inevitably, when members of device-design teams go into the field and see directly how their devices and other devices are used, it generates insight and stimulates new ideas. This is certainly a reasonable and productive purpose for field research.

However, there is another, perhaps more ambitious, purpose to which ethnographic research can be applied—to guide business decisions regarding new product development, e.g., to determine what new devices are needed, what characteristics new devices should have, and so on. What “guiding business decisions” amounts to, of course, is providing information to determine how millions, or tens of millions, or even hundreds of millions of R&D dollars can be most productively spent.

Much of what people do is largely unconscious, and, ironically, this is more true as one becomes more skilled. For example, novice surgeons may be conscious of the position of their hands at a particular point in tying sutures. Experienced surgeons are more likely to be thinking about the next step of the procedure or even where they are going to have lunch. Also, it is well known that people tend to answer questions in ways that reflect positively on themselves or that tell the questioner what he or she appears to want to hear, often at the expense of strict accuracy.

There is plenty of room for observational research of device users that does not purport to yield a true picture of reality, but rather simply provides interesting and useful fodder for ideation. However, when the research is intended to be used to guide device-development decision making, particularly in the medical area, it is crucial that the validity of the research be carefully addressed.

Achieving Validity

In certain fields, including the medical device industry, ethnography is see as qualitative research, whereas validity associated with medical devices is usually based on quantitative research (clinical trials, etc.). If we take clinical trials or even usability testing for validation research (e.g., to address FDA's design control requirements), then there are conventions for how to achieve validity. For example, based on prototype-testing data and sound statistical principles, it is possible to predict with some accuracy how many incidents or errors can be expected once the relevant device is introduced. It is also possible to estimate the number of uses can be estimated.

However, this formal, statistical approach to validity is not typically, if ever, possible with ethnographic research. Sample sizes tend to be modest and, by definition, ethnographic research does not entail systematic variation of independent variables (i.e., experimentation in the formal sense). This does not mean, though, that validity cannot be achieved. Ethnographic researchers use several methods to address the validity of their research findings. A sampling of these methods is discussed below.

Carefully Choosing the Sample, Based on Prior Knowledge. As with any research, validity is enhanced to the extent that the research sample accurately reflects the population of interest. Achieving this goal is particularly challenging with small sample sizes (10–20 sites is a typical sample size for ethnographic research). However, sample selection can become much more sophisticated when prior research can provide guidance for which variables make a difference. For example, for in-hospital research within the United States, the type of hospital (e.g., large teaching versus small community) often makes a big difference, but geographic region (e.g., East Coast versus West Coast) makes little difference. On the other hand, the differences from one adjacent country to the next in Europe can be vast. It follows, then, that in constructing the sample, geographic region can be ignored within the United States, but not in Europe.

A common mistake that can yield disastrous results is to stack the sample with so-called thought leaders who are dramatically more sophisticated than typical users of a given device. In fact, it is often more important to understand the needs of less-sophisticated device users, because a device that they can use easily will also be usable by more-sophisticated users.

Using Measurement Whenever Possible. While in the field, a researcher can go beyond his or her judgment and intuition by taking careful measurements. Measurement of sound frequency and amplitude, ambient light levels, surface heights, shapes and sizes of capital equipment, and so on can provide a solid, factual basis for making certain types of device decisions.

Engaging in Cyclical Hypothesis Testing. Well-trained ethnographic researchers do not simply take what they see at face value, but also engage in a cyclical form of hypothesis testing to squeeze out error in the data. This involves forming hypotheses about motivations, relationships, patterns, etc., and testing those hypotheses by additional observations or by saying or doing certain things and determining what results from the test. In other words, true ethnographic research requires a skeptical stance toward the data that goes beyond simple observation. This is one of the most important factors to achieving validity and one of the ways that ethnographic research, as conducted by trained researchers, differs from much of what goes by the same name.

Using Video as a Tool to Obtain Objective Data. It is difficult to obtain accurate quantitative data while in the field. Taking the development of a new surgical instrument as an example, it is logically useful to have good data about how instruments are actually used in surgery. Data would include the specific instruments that are used, temporal patterns of use, personnel who use the instruments, the number of times instruments are exchanged, which instruments are used simultaneously, etc. Such data are difficult or impossible to obtain in real time in the field. However, if procedures are videotaped, objective data can be obtained about such issues by careful evaluation at a later time.

Also, events that happen too quickly or too slowly to be easily seen with the naked eye can be carefully studied with slow- and fast-motion video playback. Therefore, video can be used to uncover the details of multistep procedures that take place in very short periods of time and to make detailed measurements of timing. It can also help researchers obtain precise counts of particular events (e.g., the number of sutures that are tied, the number of times instruments are picked up or reinserted, etc.).

Using Converging Methods. Confidence in findings can be increased by looking for consistency between the answers to questions asked in different ways, between verbal and behavioral data, or between different types of behavioral data. At the very least, a researcher should be able to segregate those findings that are known with confidence from those that are not by finding different ways to elicit answers to the same questions.

Looking for Consistency between Sites. Even with small sample sizes, a high level of confidence can be obtained when consistent patterns are found. For example, suppose that there is a question about whether a particular procedure is performed with the lights on or off. It seems reasonable to take the a priori probability of each alternative to be 50%. If so, we can use the binomial theorem to obtain the following probabilities that one finding or the other (i.e., lights on or off) is due to chance (i.e., not real) when the findings are consistent from site to site:

This is the same simple probability logic that allows us to conclude that the chances of flipping heads five times in a row are 0.016. Therefore, for those observations that yield consistent results—if we can assume that the sample is reasonably representative—then even as few as five sites can yield statistical significance (conventionally, when the chances of a given finding being a random artifact are less than 5%).

Finding Face Validity. Sometimes, a finding, although surprising at first, makes so much intuitive sense upon reflection that it provides confidence that it is representative of the typical site. This is what is meant by face validity. Face validity can sometimes lead one astray, but often a highly experienced ethnographic researcher can accurately predict after the first observation which findings will be virtually universal. For example, it is common to find time-saving shortcuts or workarounds that a device's design team did not predict, but that, once seen, make a great deal of sense. It is logical that other device users would have developed similar practices.

For experienced medical device ethnographers, then, ethnography is more than just field research. Anyone can ask a question. Anyone can write down the answer. Anyone can be in the use environment. However, if important design decisions are to be made based on the data, mere experience and observation are never enough. Such decisions require research methods, such as those described above, that increase confidence in the validity of the findings.

Conclusion

In effect, the term ethnographic research is used to describe two very different forms of research. The first type, by far the most common, is really just an alternative term for field research, any research that entails real-world observation as opposed to (or in addition to) various forms of testing, on the one hand, or interviews or surveys, on the other. This type of research certainly has its place. It can give the device-design team a general feel for what takes place in the real world, and it can be a very useful tool for generating ideas, with the expectation that some other form of research will be necessary to determine whether the ideas, thus generated, have merit vis-à-vis users.

The problem, though, is that the widespread application of the term ethnographic to such research has left the impression in many circles that ethnographic research can only be used to generate ideas, as opposed to being used as a real decision-making tool.

This article makes the case that there is another type of ethnographic research (not to be confused with the first type) that can be used to guide business decisions. However, conducting such research is difficult, time-consuming, and, frankly, expensive, in comparison with the idea-
generation type of ethnographic research. It is important to note that when it comes to decisions that involve users, ethnographic research, despite its difficulties, has the great advantage of not being exclusively based upon what people say.

Quantitative survey research tends to be the preferred tool when decision makers want objective data about what device users want and need. Such research yields good, hard numbers to support an objective approach to decision making. However, the drawback with such quantitative market research is that the data are only as good as the accuracy of answers to survey questions. There is enormous evidence that the answers to questions are not always accurate, for a variety of reasons.

In other words, the apparent precision and accuracy of quantitative survey research may be misleading, depending on the specific questions that are asked. For some questions, it is better to observe than to ask the user for the answer. The central point of this article is that observational research does not have to be exclusively qualitative. It can be done in such a way that it yields valid data, although validity is achieved through methods that are different from typical quantitative market research. Ethnographic research is unlikely to supplant quantitative market research or even qualitative market research any time soon, if ever, in medical device research. However, it can be an important supplement to other forms of research and can provide a deeper understanding of the wants and needs of device users.

Stephen Wilcox is the founder and a principal of Design Science (Philadelphia). He can be reached at [email protected].

Reference

1. Stephen B Wilcox and William J Reese, “Ethnographic Methods for New Product Development,” Medical Device & Diagnostic Industry 23, no. 9 (2001): 68–76.

Copyright ©2008 Medical Device & Diagnostic Industry