Medical Device & Diagnostic Industry Magazine
MDDI Article Index

An MD&DI May 1997 Column

CLINICAL TRIALS

For medical device manufacturers who must conduct clinical trials, extra effort early in the process, in the form of voluntary auditing, can prevent costly errors.

Clinical trials for establishing the safety and efficacy of a medical device can be among the most costly and time-consuming elements of product development. If procedural errors render unusable the information gathered in a series of clinical trials, the time and expense required to conduct the studies again can easily terminate the entire development effort. Thus, for the many medical device manufacturers who need to sponsor clinical trials to gain premarket approvals (PMAs) or submit 510(k) notifications, it is essential to maintain confidence throughout the clinical trial process that the data will be usable.

One way for sponsors to ensure that there will be no unpleasant surprises after clinical trials are finished is to plan for one or more independent investigator audits during the trials. An audit of a clinical trial provides the research sponsor with independent appraisal of the quality and completeness of the data generated by the trial. Although auditing alone cannot transform a poorly planned, executed, monitored, or analyzed trial into a credible one, an active clinical trial audit program will point out potential problem areas early, so solutions can be found before it is too late. Used effectively, therefore, audits can reduce costs, maintain project schedules, and ensure regulatory compliance.

FDA already requires that clinical trials be subject to monitoring. Current procedures at the Center for Devices and Radiological Health (CDRH) make it highly likely that at least two investigational sites of each major PMA clinical study will be inspected by the CDRH bioresearch monitoring program. Studies undertaken for 510(k) submissions are now also reviewed by FDA bioresearch monitoring staff.^1,2 By choosing to perform independent auditing of investigational sites, sponsors can increase their chances of performing well during an FDA inspection.

Auditing, which is typically conducted no more than once or twice during the course of a clinical trial, encompasses many aspects of the study. Auditors review a sample of data taken from a representative group of study sites. They also check regulatory documents such as the study protocol, institutional review board (IRB) communications and approvals, informed consent documentation, investigator agreements and curricula vitae, clinical laboratory certifications and normal values (if necessary), randomization documentation, and device accountability records to determine whether the documents conform to sponsor requirements. Auditors compare the data in the source documentation--most commonly, the medical record--against data entered on the case report form (CRF). Because of the vast amount of material involved, this comparison must often be limited to just a sample of patients and selected data fields. Auditors also review source documentation to ensure protocol compliance and full evaluation and reporting of adverse device effects (ADEs).

The most common reason that sponsors choose to perform independent auditing is to prepare for an FDA inspection. However, many device manufacturers are beginning to follow the lead of the pharmaceutical industry and incorporate this activity into all phases of major research programs. Sponsors are finding that clinical trials audits serve a variety of purposes.

Independent review of a clinical trial can bring a fresh perspective and new insights into the project. For example, the longstanding relationship between a clinical research associate (CRA) and the investigative site that he or she oversees can be a potential source of bias. A CRA typically visits a study site at regular intervals to make sure that procedures are being followed. An auditor, who does not have such ongoing contact with a site, can be more forceful in calling for any needed changes.

Audits provide the chance to review one clinical research system, such as control of investigational devices, across several studies. They can also determine how data generated outside the United States can be used in an FDA submission.

In rare instances, an audit can be used to investigate possible fraud. In such cases, data audits can show whether data in source documentation, such as medical records and laboratory reports, are consistent with entries on CRFs and data listings.

So, the benefits of auditing can be well worth the extra effort required. To make sure that these benefits will be realized, however, sponsors must develop a comprehensive auditing strategy. Sponsors must decide not only which sites and data will be audited and how often the audits will be performed, but also how the information generated from the audits will be used to improve procedures.

CHOOSING AN AUDITOR

One of the first, and most important, decisions the manufacturer faces is choosing an auditor. Because they are sometimes not part of the sponsors' organizations, contract auditors can promote a spirit of objectivity and encourage investigators to communicate problems openly. Auditors can also be chosen from the sponsor's quality assurance organization or from any group not associated with direct management of the clinical trial.

Auditors must be knowledgeable about the applicable regulations, the technical aspects of device operation, and the medical condition indicated for the device. This can represent a rather unusual combination of knowledge, which is acquired only by working in related fields in industry. Experience in good manufacturing or laboratory practices is certainly a good qualification for auditors, but a strong background in the relevant clinical area and experience reviewing medical records are also essential. Many potential problems can only be recognized by someone with clinical experience. For example, if a CRF indicated that a study subject was discharged from a hospital halfway through a course of antibiotics given to treat a postoperative infection, an auditor with clinical experience would be able to recognize that such a patient would be unlikely to be discharged at that point, and question the discharge date on the CRF.

Auditors must also be able to organize and write a report that accurately describes the audit observations. For the report to be useful, descriptions of deviations must be detailed enough that solutions can be found and implemented.

In addition to the well-defined qualifications mentioned above, there are some equally important, but less tangible, characteristics that are important in an auditor. The auditor must have an aptitude for details and logical thinking. A successful auditor is able to notice a minor inconsistency, deduce possible causes, and then investigate, uncovering significant issues. An auditor should have the patience to search for details and follow these leads, often under tight deadlines.

Auditors must also conduct themselves in a professional manner during all interactions with the study site personnel and corporate coworkers.

PLANNING THE AUDIT

Once the auditor is chosen, the manufacturer must plan where and when the audit will be performed as well as what data will be audited.

Confusion and schedule conflicts can often be avoided by preparing an audit agenda and sharing it first with internal management and then with site personnel. The agenda will tell all parties what information will be audited, who must be present, and when each portion of the audit will occur.

Audits are most frequently conducted when studies are at least half completed, so that the auditors can review a meaningful number of patients. Audits performed after a study site has been closed can present some special challenges when corrective actions are necessary. Some site personnel can become reluctant to invest time in the correction process once all research fees have been paid. Also, corrections become more difficult as time elapses. Patients are more difficult to track down, and medical records can become more difficult to access.

Neither research sponsors nor FDA have the resources to audit every site of every clinical trial. Site-selection criteria include the greatest number of patients enrolled, or the largest number of ADEs or protocol violations. The number of patients is often the most important factor in selecting audit sites because it is more cost-effective to review many patient records at a small number of sites than a smaller number of patient records at many sites. Geographic considerations can also be a factor in determining where to conduct an audit.

Of course, one of the most important aspects of planning the audit is determining what data to investigate. To decide this, auditors should become thoroughly familiar with the sponsor's study protocol and all amendments to it. In some cases, the sponsor may wish to add additional agenda items for an audit, and the auditor should become familiar with these items as well.

Auditors should also meet with the responsible CRAs and project managers, and review data listings, if available, for unusual patterns. The meetings should cover all relevant issues, such as ADE reporting, IRB and informed consent issues, investigational device accountability and failure issues, and patient-assessment questions. The review of the data listings also provides an opportunity for key safety and effectiveness CRFs and specific data fields to be identified.

CONDUCTING THE AUDIT

Good communication is essential to a good audit. Auditors who arrive at a site unprepared waste time and diminish goodwill between the sponsor and the researcher. Researchers who either do not understand or do not prepare for an audit can also adversely affect the process.

The first step in promoting this communication is for the auditor to contact the site to announce that the audit will be conducted and plan the details. In most cases in the United States, three or four weeks' notice is adequate. International audits often require longer notification periods.

The auditor must confirm that all necessary site personnel and the documentation that will be reviewed will be available during the audit. To make the initial communication as unambiguous as possible, any telephone agreements should be followed up by a fax or E-mail. Written documentation is useful for U.S. audits and essential for international ones.

During the audit, every opportunity to gather information should be taken, and auditors should be alert for communication nuances that may signal hidden problems. For example, an investigator who claims to have only a vague idea where the study protocol is stored may not be familiar enough with the protocol.

There is no single method that is appropriate for all medical device clinical research audits. Auditors should adapt their techniques to the clinical trial under review. For example, an audit of an in vitro diagnostic study will concentrate on sample handling and a data comparison between analytical methods, whereas an audit of a study for a life-sustaining device will include detailed review of medical records.

However, there are documentation tools that auditors can use with minor modifications for nearly any study.

Audit Checklists. Checklists are invaluable when conducting multisite audits, especially if more than one auditor is involved or if the study is international. Checklists can capture statistical summary data, such as the total number of informed consent forms reviewed, the number acceptable, and the most common deviations. They can be customized for different studies to list the key data points that must be reviewed during the audit. Although they are not substitutes for good observation and analysis, checklists can help ensure that each subject at each site during a complex audit is reviewed in a uniform manner.

Chronology Lists. A chronology of key events can simplify regulatory document review. To make such lists, the auditor constructs a table with columns for the date and for IRB, sponsor, and FDA actions. Each action, as described in the study site's documentation file, is listed in sequence on its own line. Once completed, chronologies can clearly show how various actions are interrelated and point out any conflicts between actions and exceeded time frames.

AREAS TO REVIEW

Although audits vary depending on the study, there are several aspects of clinical studies that should be routinely reviewed during an audit because they are potential sources of difficulty for many studies.

Regulatory Documentation. A variety of documents must be present before an investigator can begin a medical device clinical trial, and an auditor should make sure they are in order.

If the device poses a significant risk (SR) to study subjects, an investigational device exemption (IDE) must be filed and approved by FDA. If the device poses a nonsignificant risk (NSR), no formal IDE filing is necessary; however, the record-keeping requirements as stated in 21 CFR 812 continue to apply. In an NSR study, the reviewing IRB must make two determinations: first, that the study is NSR, and second, that the study protocol is approved. Some IRBs become focused on the second objective and fail to document the first decision.

Randomization. One item becoming more common as medical device clinical trials increase in complexity is randomization. In order to reduce bias, patients are randomly assigned to either the experimental treatment or to treatment with a comparison, or control, device. There are several valid methods for randomizing treatments. The chosen method must be carefully reviewed to ensure that it is functioning as intended. Clinical investigators should not be able to influence the treatment group assignment for any patient.

Clinical Documentation. Comparison of CRFs to source documentation is almost always the most time-consuming on-site task of the audit. CRFs are rarely arranged in the same order or format as the medical records. Also, the organization of medical records varies greatly from one site to another. An effective auditor can easily adapt to changing conditions, however.

Unlike a CRA, who reviews virtually all data recorded on each CRF, the auditor commonly reviews only a carefully chosen sample of data fields. Often, only a subset of patients at the site are reviewed. Selecting these patients for review is a key aspect of ensuring the validity of the entire audit process. In situations where all the patients appear to have been treated in the same way and with the same result, a simple technique, such as, can be employed to determine sample size, where n = number of patients enrolled. When using this method, care should be taken to ensure that patients are then selected from the beginning, middle, and end of the study enrollment period.

If there is greater variability among the study subjects, those with large numbers of ADEs and several protocol violations, early terminators, and those lost to followup often involve the greatest number of functional areas in the clinical research system and will demonstrate the degree of coordination between these systems.

CRF and Source Documentation Consistency. Comparing CRF entries with source documentation is a major component of all clinical trials audits. Source documents, or medical records, may have a wide variety of formats. For example, orthopedic implant studies will use x-ray films, while cardiology studies will use electrocardiographic traces, angiography films, and other testing data. Clinical laboratory data, pathology data, and even patient self-assessments and diaries may be the source documentation.

Outpatient studies can often present documentation challenges. An investigator may enter only a few lines of progress notes, and most of the study data may be entered directly on the CRFs. To ensure that there will be adequate backup data for the CRFs, some research sponsors will work with investigators to develop worksheets for the investigator's records where the data can be recorded.

No matter which form the source documentation takes, the auditor must assess it for credibility and consistency with the data recorded on the CRF.

Assessment of data for credibility is usually straightforward, unless fraud is suspected. In rare instances, investigators have manufactured data and even entire patients by repeatedly photocopying a laboratory report and adding a new name for each new study record generated.

One indicator of potential fraud is vastly different appearance of the signatures of either the study subject or the study site personnel on different documents. If a study subject signed the informed consent form the day before the use of an investigational device and the signature appears significantly different from the signature on the hospital's surgical consent form, then further investigation is justified.

Auditors need to review several aspects of the medical records. For example, the appropriate personnel must operate the device. Many studies of therapeutic devices require that the device be operated by the investigator or a subinvestigator. It is often hard to show that the device was operated according to the protocol, but this should be done whenever possible.

Another area that an auditor should review is whether all measurements generated consistent results. Inconsistencies in assays may be caused by technical faults in the test methods, or by how the methods are applied by investigators. If a site is performing a nonroutine assay in conjunction with the study, sufficient validation data should be available to demonstrate that the assay is capable of producing reproducible results.

CRF Corrections. It is inevitable that improper entries in CRFs will be identified during the data entry process. When data entry personnel are unsure of an entry or the data are clearly out of the expected range, a query is generated. Large, complex trials require special systems to document and track queries. Data entry errors may be handled in a less formal manner for most smaller medical device studies. However such problems are resolved, the copy of the CRF at the study site must be made to accurately reflect the sponsor's database. Audits conducted at the close of a study often review a sample of data queries to ensure that site documentation conforms with the sponsor database. Here, auditors need to distinguish between random lapses and persistent system failures.

ADE Documentation and Reporting. ADE reporting is one of the most difficult tasks in medical device clinical research. All members of the research team can become involved in reporting an ADE, and all activities are governed by time limits in the regulations. Chronology lists can make this part of an audit easier by showing when reports were made.

Investigational Device Accountability Records. The FDA IDE regulations require that supplies of investigational devices be controlled. When dealing with single-use devices, this requirement is usually simple to audit. Sponsor shipping records and study site receiving records show what devices should have arrived at the site. Use records can be found in the medical records, CRFs, or dispensing logs. It is essential that all these sources agree. Disposals, returns, and repairs of devices should also be recorded in site documentation and a final check made to determine the number of devices shipped to the site and the fate of each one.

FDA concerns in this area involve possible protocol violations stemming from using different quantities of the device per subject, diverting investigational supplies to nonstudy patients, or transferring investigational supplies to investigators who are unaffiliated with the study. Accountability for durable medical devices, both therapeutic and diagnostic, can become more complicated, especially if there are no consumables necessary for the use of the device. Auditors may have to gain access to data stored in device controller memory or use other methods to confirm accountability.

SPECIAL CHALLENGES FOR AUDITORS

One of the most difficult tasks the auditor will encounter is to ask the hard questions. For example, if it becomes clear that a study did not have IRB approval for a three-month period after the initial IRB approval expired and before the next approval took effect, the investigator or study coordinator must be asked about this situation. This could be a significant observation, or simply a misunderstanding. The auditor must resolve such situations so that the audit report will be accurate.

Although outright fraud is rarely encountered in medical device clinical research, its consequences are so severe that one must always be on the alert for it. If it appears that the site is involved in fraudulent activities such as the creation of study subjects or falsification of test data, the auditor needs to gather conclusive evidence, if possible, and quickly inform the sponsor. Frequently, the sponsor's response is to terminate the audit and conduct further site interactions through regulatory or legal representatives.

USING THE FINDINGS

When the audit is complete, a wrap-up meeting should be held to discuss the findings. This meeting is the most important part of the audit process for study site personnel and often one of the most difficult tasks for the auditor. The auditor's goal for the meeting is to summarize key findings and identify any misunderstandings before they become part of the audit report. The investigator's goal is to get an answer to the question: "Did we pass?"

A scheduled date for the wrap-up meeting should be agreed on at the beginning of the audit to make sure that as many of the key people as possible will be present. Rushed meetings in the hallway with the auditor's bags packed and the airport shuttle van waiting outside should be avoided. A little thought and even 10 or 15 minutes of organization on the part of the auditor can make a big difference in the quality of the presentation.

A good way to start the wrap-up meeting is to thank site personnel for their time and assistance. Next, the (hopefully) brief list of important audit findings should be reviewed. In some cases, auditors can be reluctant to share negative observations with the study staff. If there is a possibility of fraud, this concern may be justified, but in other cases, it helps everyone if findings are frankly discussed. Discussions should center on facts, not personalities. Photocopies of documents demonstrating the observations are very useful.

A somewhat controversial item that may also be included at wrap-up meetings is a review of corrective actions. Some firms encourage auditors to work with the sites to solve problems, while others feel that this type of interaction jeopardizes the objectivity of the auditor. Auditors who choose to suggest corrective actions need to be aware of the conduct of the entire study, not just one site, and should be careful not to commit their organization to promises that it cannot keep.

Many sponsors of clinical research also ask auditors to perform some form of brief training of study site personnel as a part of the audit. Distributing a package of relevant documentation including copies of 21 CFR 50, 56, and 812, the FDA compliance guidance manual for clinical investigators, and some of the FDA information sheets can be helpful for training.³ This information package can be especially useful if the audit is conducted after the last monitoring visit, because the next time the site study documentation is reviewed may be with an FDA investigator during a bioresearch monitoring inspection. Most clinical investigators realize that this type of training and information is generally applicable to all the studies they conduct and welcome the opportunity to gain a better understanding of the regulations.

The Audit Report. The observations discussed during the wrap-up meeting need to be documented in a report so that effective corrective actions can be taken. In addition, appropriate levels of management at the sponsor's firm must be made aware of the results of the audit. One advantage that auditors of medical device clinical trials have over auditors of manufacturing facilities is that nearly all the observations made during clinical study audits relate to documents, which can be copied and included in the report to support the report's conclusions.

There are many possible formats for audit reports. Reports for data audits generally contain less narrative and more tables that list individual deviations along with error rates for specific CRFs or even for specific data fields. Audit reports of the more conventional midstudy, or retrospective, audits usually include most of the following items.

Scope. This section delineates what was reviewed (and not reviewed) during the audit. It is important to inform management of any limitations in the scope of the audit. This section could include phrases such as, "Only study X was reviewed during this audit," or, "Time permitted review of only seven out of the nine patient records that were initially designated for review."

Introduction. This section gives the background for the audit, listing all the personnel involved and their roles, the reason for the audit, the audit methods, and a very brief description of the study that was audited.

Executive Summary. This section contains one or two paragraphs explaining in general terms the audit results and their significance.

Major Findings. The most significant findings are listed in this section as bulleted items, with the most significant ones first. References to more-detailed descriptions in the body of the report can also be included.

Regulatory Documents. Observations relating to informed consent, IRB activities, or any other nonmedical study issues should be described in this section. In most cases, it should be possible to photo- copy documents related to these items, which may be cross-referenced and included as an appendix to the report.

Clinical Data Review. Results of the comparison of source documentation against CRFs or data listings are described in this section, as well as protocol violations such as difficulties reporting ADEs, improper use of the device, failure to perform specified testing on patients, or failure to follow study schedules.

Conclusion. Opinions and impressions can be important results of the audit. They need to be documented, but clearly identified as opinions.

Follow-Up Actions. Completed audit reports must be carefully protected. A very limited number of copies should be circulated within the company, and review of the report should be completed according to a predetermined schedule.

Corrective actions must be agreed on, implemented, and documented. Once an organization invests time and money in an audit program, the follow-up activities become as important as the audit and should not be forgotten.

All corrective actions, including minutes of postaudit meetings and revised documents, should be reported to the auditor for inclusion in the audit file. Once all issues have been resolved, all copies of the report should be recalled and the report, along with supporting documentation, filed in secure storage. An audit log should be maintained listing studies and sites audited, dates of audits, and the identities of the auditors. Ordinarily, FDA investigators are entitled to review the audit log but not the audit reports.

Many medical device firms can address a specific list of problems. Ensuring they do not arise at other sites and with other studies is more difficult.

Quality assurance tools used in the manufacturing part of the organization, such as trend analysis and examination of the root causes of problems, are very useful in the clinical research environment as well. If issues arise because of lack of knowledge on the part of the responsible CRA or the study site staff, training can be modified to address them. One of the most important functions of the clinical audit is to look at systems and ask whether they continue to be adequate for the clinical research that is being conducted.