Here are the four critical questions to ask before taking on the literature search and review process.

Scott Ahlberg

Whether your organization is about to bring a new medical device to market or preparing to submit a periodic safety update report, you must demonstrate that the device complies with all the relevant regulatory requirements. Additionally, more than establishing that the device delivers its intended benefits, you must also identify all known risks, capture any adverse events, and report the findings along with all the appropriate evidence.

It's a big and difficult job, and one that relies upon efficient and effective processes--as well as experts to run those processes. And where auditors are concerned, process is the name of the game. Consequently, the methodologies you employ when conducting a comprehensive medical device literature search must be clear and based upon established best practices. The results must bear up to scrutiny as auditors look not only for supporting evidence, but also for tell-tale indications of potential bias in the reporting. As such, it's a high-stakes game. Failure to support safety claims evidenced by literature can mean going back to the expensive and time-consuming drawing board of clinical trials.

With this in mind, the literature search and review process is a critical component of the overall and continuing clinical evaluation. The only remaining question is how to approach these challenges. If you choose to "go it alone" and produce compliance reporting using internal resources, ask yourself the following four questions before getting started:

1. Should I go it alone?

To insource or outsource--that is the question. While there may be some advantages to producing the compliance reporting inhouse, particularly if your organization possesses the deep competencies that the job requires, the pitfalls could be quite perilous. One of these pitfalls is the inherent prejudice you'll face by regulatory bodies if you choose this path. The four-letter word that looms large in their minds? Bias.

In fact, this prejudice has actually been codified in such guidelines as MEDDEV 2.7.1--the standard for clinical evaluation in the EU for medical devices. The document clearly encourages manufacturers to employ objective third parties to conduct independent evaluations, stating, "The appraisal should be thorough and objective, i.e., it should identify and attribute adequate weighting both to favorable and unfavorable contents of each document."

So where, exactly, can bias raise its ugly head? It turns out that it creeps in across many areas. To narrow this down a bit, let's look at the standard's definition of the word: "Bias is a systematic deviation of an outcome measure from its true value, leading to either an overestimation or underestimation of a treatment's effect. It can originate from, for example, the way patients are allocated to treatment, the way treatment outcomes are measured and interpreted, and the way data are recorded and reported."

There are three big and non-exhaustive categories found here: control (selection bias), measurement (measurement bias), and reporting (analysis bias)--all of which an internal operation may be prone to exhibiting. Let's take a closer look at each.

Selection bias occurs when subjects are allocated to a treatment group in such a way that they do not accurately represent the population, or in such a way that treatment groups are systematically different. A whole host of events can contribute to bias in reporting. For example, basing results on subjective endpoint assessments like pain, or when the endpoints or symptoms assessed are subject to natural fluctuations. Selection bias can also arise when effectiveness studies are conducted with subjects who are likely to take effective cointerventions, including over-the-counter medication and other therapies, or any number of other influencing factors including the variability of the patient population, of the disease, and even of subjects' skill in using the device.

Measurement bias surfaces with respect to responses to treatment, as well as when the outcome is inaccurate due to instrument bias (have the measurement tools been properly calibrated?). Measurement bias can also be the result of subjective expectations (or predispositions) of study participants, researchers, or care staff--each of whom can impact the determination of the actual performance of the device.

Analysis bias occurs when relevant information is omitted or miscalculated. It can also happen when the risks identified in the risk management documentation and literature have not been adequately addressed. The regulatory bodies expect that the clinical evaluation demonstrates that any risks that may be associated with the intended purpose are minimized and balanced against the benefits to the patient. Regulators also expect that the device is safe and accompanied by sufficient information to reduce the risk of use error. The fulfillment of these expectations must be supported by relevant clinical evidence.

While auditors have a sensitive trip wire on these issues, they're on even higher alert when clinical trial investigations are sponsored by companies with established physician relationships, particularly when the sponsors are codeveloping a device. Consequently, both the EU and FDA reviewers will continue to be vigilant when it comes to accuracy and bias in data reporting. As such, if you are performing this work inhouse, it is likely that you will be held to a higher standard.

2. Do I have the right search strategy?

One essential element of the submission documentation is a description of the search strategy protocol that will be used to determine how articles will be discovered, selected, and excluded. The protocol must include a description of your search resources, a list of search terms used, the specific inclusion and exclusion criteria, and an explanation about how the clinical literature will be weighted.

This can be a daunting task, as many devices may either not have been evaluated clinically, or may have been around for so long that a plethora of information exists. In the latter case, a properly constructed search can return thousands of hits. The selection of appropriate articles, is therefore a key component for a successful evaluation. This is especially true if the clinical literature evaluation will serve as the bulk of your clinical evidence.

The purpose of the literature search protocol is to provide a plan for the search phase. It should be developed and executed by professionals with specific expertise in information sciences, as their involvement will help to optimize literature retrieval and identify all relevant published literature.

A rigorous and demonstrable search methodology, as outlined by the excellent protocol template MEDDEV 2.7.1, thoroughly addresses the following points:

Now here's the rub and another reason auditors ratchet up their scrutiny of processes performed inhouse: many smaller organizations simply lack staff with the relevant information services and knowledge management expertise to properly perform this function. But even when they possess the talent, it is not always efficient or economical for lay searchers to perform comprehensive searches.

The key is in ensuring that the search strategy is broad enough to capture all relevant literature, but specific enough that it doesn't yield a barrage of false hits. It is not uncommon for an exhaustive literature search to turn up as many as 2,000 abstracts to sift through before safety assurances can be satisfied. Conversely, if the search criteria is too narrow, you run the risk of missing important papers.

Finally, one search does not fit all. Search syntaxes vary according to database. For example, when you're searching PubMed, you may well apply different search terms and operators than when searching EMBASE. A proper comprehensive search will also use the indexing in the database--the "tags" that have been applied to all the articles. Sometimes the terminology that you're looking for is in the abstract or article itself (often referred to as free-text searching), but sometimes it's in the metadata that's been applied to the article. When attempting to perform a comprehensive search it's important to apply all tactics.

3. Will my search turn up the best evidence?

Once the search phase is completed, the evaluation of the results begins. The goal of the clinical literature evaluation is to provide evidence of the safety and efficacy of the medical device in question. When relying on the literature, the evaluation must unequivocally demonstrate equivalence in design and performance in the specified clinical indication.

To this end, the evaluation process requires a thorough and critical assessment of the nature and quality of the evidence revealed in the search process. This is vitally important because it may be possible to draw upon the literature alone to establish the requisite clinical evidence, thereby eliminating the need to generate new clinical data. Suddenly the search protocol and execution takes on new significance! Taking the clinical literature evaluation route can actually be used to justify the decision not to conduct clinical trials.

So what constitutes sufficient evidence? In short, a critical mass of high-quality clinical evidence to guarantee the validity of the conclusions, and to demonstrate conformity with the requirements covering clinical performance and safety. If you conclude there is not sufficient clinical evidence to satisfy these requirements, then you'll have to recall the devices and suspend marketing activities until conformity is restored. From the perspective of reviewers, it's not hard to see that this costly possibility only amplifies a manufacturers' tendency to skew findings, which further amplifies the regulatory bodies' marked preference for objective third party involvement.

Because all papers are not created equal, the evaluation should also describe the methods of weighting the literature. It should include a market analysis of the same or similar devices, as well as the results of post-market studies and any published adverse events--including for those of similar/competing devices. Finally, the preparer should offer a conclusion that justifies the assessment of the device's safety and efficacy.

4. Do I have the right tools to do this job?

In addition to any inhouse expertise your organization may possess, research retrieval tools and third-party service providers should also be counted among your key assets.

Research Retrieval

Obtaining research content is a process that can be fraught with difficulties, not the least of which is the overwhelming volume and complexity of that content and the myriad ways of accessing and managing it. Content comes in many shapes and sizes--whether embodied in journal articles, book chapters, conference proceedings, theses, posters, standards, pre-press articles, supplementary materials, pharmacopeias, or other formats. Identifying the papers you need is difficult enough. Actually getting your hands on them in an efficient and cost-effective manner is another matter.

So now what? The first step is to link your discovery process directly to the delivery of content. Removing the friction between discovery and delivery accelerates the process, particularly when users don't have to leave their preferred search and discovery platforms. Other ways of removing this friction include automating routine processes, providing instantaneous article access, resolving copyright issues, and enhancing search results with the information needed to zero in on the best content to acquire.

Additionally, a search result from PubMed or Google Scholar saved in a bibliographic management widget streamlines the document retrieval process. A good research retrieval solution will also deliver research support services, especially when you need to go the extra mile in locating and securing more elusive documents.

Taken together, a services-augmented research retrieval platform comprises a workflow management solution that enables the most efficient and cost-effective use of literature resources. For organizations that consume tens of thousands of medical journal articles every year--typical of medical device companies--the terms "efficiency" and "cost-effective" take on significant meaning.

Conclusions

Because literature search and evaluation in support of regulatory compliance is a continuous process, the last thing you want to do is reinvent the wheel each time. As such, a well-oiled and repeatable solution that integrates all the moving parts and partners seamlessly will provide noticeable returns.  

Whether or not you opt to perform the function inhouse, a proper evaluation of workflow models will require taking a holistic view in order to account for the true and total cost of ownership. As you chart out the options, a useful exercise is to include "current state" and potential "future state" workflow comparisons. For example, in your present workflow, are you able to rent or preview versus buy? Do you have options for single article order/delivery versus batch/bulk? Are you able to accommodate invoicing and billing requirements, including departmental chargebacks? Can you check re-use rights when buying to avoid unnecessary spending? Such an exercise will reveal the potentials for improved processes and what could add up to significant cost savings.

Indeed, the logistical hoops that regulatory submissions require are not to be underestimated. Navigating them effectively and efficiently will have a direct bearing on your outcomes. The good news is that help is available to optimize workflows and user experiences to ensure all parties and stakeholders are more productive and successful.

Scott Ahlberg, chief operations officer at Reprints Desk, has decades of experience in various content businesses, mostly serving the pharmaceutical and medical device industries. He oversees a wide range of content delivery professionals in multiple business lines. 

[Image courtesy of NESE/PIXABAY]