Double-Redundant and Fail-Safe Design for Packaging Machinery

Originally Published MDDI August 2002

COVER STORY

When properly applied, double-redundant and fail-safe design methods focus validation practices on machine enhancement and reliability, reducing dependence on operator vigilance and intervention.

Ray Johnson

Validation has been an FDA-mandated requirement in the medical device industry for many years. At first, the regulation caused immense confusion, and the initial focus by many was on generating documentation. As the practice of validation has advanced, however, a greater emphasis has been placed on the process of validating machinery and equipment.

The art of validation continues to develop and expand. As the validation testing and qualification processes mature, a greater understanding of manufacturing risks emerges. One aspect, positive, fail-safe machine design, has always been a feature of good manufacturing practices (GMPs). But as the technology of processing and packaging machinery advances, there is now a greater risk of failure and a correspondingly greater challenge to design fail-safe systems.

The regulatory bodies continue to be vigilant where customer's lives may be threatened. This vigilance is particularly important where the customer has to place absolute trust in the product. For example, ensuring product and package integrity is critical in cases of packaging where information essential to identifying the product is included on the package—or in the case of sterile products, where the packaging is part of the product.

In the medical manufacturing industry, the consumer has no alternative but to place absolute faith in the package and the product. Errors such as packaging mix-ups or faulty packs that lead to nonsterile products can easily go undetected by the consumer. Such a high level of consumer trust in the product places a high responsibility upon the manufacturer—a reliance that can only be met through GMPs, process and product controls, and process validation.

Validation has been an evolutionary process for most companies in the medical device industry. Firms have generally taken a responsible stance in implementing validation, thereby ensuring the robustness of their processes, packs, and products. A large number of companies have followed guidelines on testing and qualifying packaging machines, and there is no doubt that the validation process has substantially improved the design, quality, and performance of such machines and processes.

In the drive for near-zero defects and the need for absolute control of the packaging process, however, questions are being raised regarding faults that occur at a very low and unpredictable frequency. Such undetected faults are aberrations that are more likely to occur today as a result of technology advances. This is an issue of grave concern, because such faults potentially impact the integrity of the package—and therefore, the product.

DOUBLE-REDUNDANT AND FAIL-SAFE DESIGN

To address these concerns about undetected faults, double-redundant and fail-safe design is becoming a requirement in the area of medical device processing machinery. In this process, manufacturers carefully examine the risks of undetected errors occurring to ensure that systems are modified so that either the mistakes cannot occur or there is zero risk of them going undetected. But many medical manufacturers and their machinery suppliers have thus far failed to realize the need for or consequences of not embracing double-redundant and fail-safe design in their packaging machinery and processes.

The need for these design criteria in medical manufacturing emerged as a result of two basic factors in industry as a whole: the limitations of statistical analysis and machine complexity.

STATISTICAL ANALYSIS

Sample inspection or statistical controls cannot detect spurious machine events or aberrations in a process. A well-recognized and legitimate method of controlling a machine or process is to apply statistical tools that will predict the standard deviation of a batch. Using such tools on a reasonable sample, it is possible to predict to a high probability whether the entire batch is going to be within the specification. Of course, these statistical tools must be applied with care—the incorrect use of statistical tools in applications where there are multiple variables can be dangerous and should only be approached with absolute care. But statistical analysis is essential and valuable in the field of validation, and a sound mathematical basis for ensuring process stability and capability.

Statistical analysis assumes stable conditions; however, the potential aberrations in modern packaging machines render a statistical analysis inadequate for predicting all errors. Furthermore, statistical tools rely on the sample being representative of the population, which may or may not be the case. Therefore, any spurious events or aberrations in the packaging process will not be predicted with the measurement of a supposed representative sample batch—unless the person taking the sample just happens to include the fault in one of the sample batches. The problem, of course, is that in validation you cannot rely upon luck.

MACHINE COMPLEXITY

The second factor affecting the potential for undetected faults is the rise in machine complexity. Thirty years ago, packaging machines were primarily mechanically driven and most processes were linked from a single drive motor through drive shafts, cams, and belts. If there were any failures on these machines, they were almost all mechanical in nature. Mechanical failures are unlikely to be spasmodic; indeed, nearly all would be permanent such that the machine could not run without intervention and repair. Such failures simply do not repair themselves, in other words, and the fault would be certain to be detected through normal GMP batch inspections.

But today, with the need for flexibility and controls that allow rapid pack changeover, machines are designed in a much more complex manner. Instead of the central drive system being composed of cams and pulleys, packaging machines now have servomotors, electronically controlled pneumatic systems, and other devices that are linked through a computer network to provide the synchronization necessary to harmonize each element of machine operation. Such machines have the essential benefits of greater flexibility (almost all changeovers are electronic), higher speed, lower maintenance, and better overall reliability. They also have the added benefit of improved GMPs, since there is a significant reduction of wear parts and parts requiring lubrication.

The flexibility and speed inherent in modern machines rely upon the control system to ensure all independent elements act in concert. The coordinating element is typically a programmable logic controller (PLC). Properly applied, the PLC ensures that commands are given to each element in the correct sequence and at the correct time.

If the PLC fails to detect the correct performance of any element to which it has sent a command and the fault remains in place, however, the fault may go undetected. The process of batch quarantine prior to sample inspection, and subsequent release after inspection, may miss the intermittent failures, thereby leading to the release of bad product into the marketplace.

APPLYING DOUBLE-REDUNDANT AND FAIL-SAFE DESIGN

In the world of machine manufacturing, there is no such thing as perfection. It is possible with new packaging systems for machine faults to be transient—in other words, the faults are only present for a small number of machine cycles and are then corrected. There is little chance that such transient faults will be detected and effected products eliminated before they are released to market without a more sophisticated level of design and controls.

There are examples in other industries where double-redundant and fail-safe design has been introduced as a regular manufacturing component. Examples include dual-brake circuitry on cars, dual-control systems on aircraft, and perhaps the simplest case of all—the backup parachute.

There has long been a need for dual circuitry in machine safety systems where there is a high potential risk to operator life. There is also a widely accepted belief, however, that no matter how careful the design, quality, and assembly procedures, errors may still occur, albeit infrequently. Thus, where lives are directly at risk, double-redundant or fail-safe design is mandatory. Since errors in the medical packaging industry also threaten lives, double-redundant and fail-safe design is no less vital.

Figure 1 illustrates a well-proven system for minimizing the risk of aberrations or transient errors. The first requirement is that validation be specified as a primary requirement, not an afterthought that is performed after the packaging process is in place and FDA is knocking on the door. In this way the machine or process is designed to be validated from the outset. This simple stipulation immediately improves the robustness and repeatability of the machine. The second step is intended to remove weaknesses in the design that may lead to transient faults or aberrations.

Risk analysis, also known as failure mode effects analysis (FMEA), enables manufacturers to identify whether transient mistakes can lead to a reduction in product integrity. The process of designing a machine where all risks are eliminated is called fail-safe design. The act of designing out a potential mistake usually requires the addition of a double-redundant system.

Process design for the medical packaging industry must include double-redundant and fail-safe design to eliminate all potential faults. The FMEA risk analysis of the process involves detailed analysis of each machine element and the potential consequences of its failure. If failure is possible, and if it will give rise to an undetected faulty package, then either the machine must be redesigned or a double-redundant system added to ensure the system is checked for correct operation each machine cycle. Machine elements with the potential for failure are termed system defects. The double-redundant system-defect subsystems do not contribute to the overall machine performance, but are put in place solely to ensure that what is expected to happen actually does happen, in every case.

Any single event that is likely to lead to an undetected reduction in pack integrity should be designed out. Single-event possibilities should be monitored by independent systems so that two independent errors have to occur in a way that produces a plausible result before undetected errors can occur. If this concept is applied along with fail-safe philosophy, the chances of two such failures arising in this manner are as close to impossible as can realistically be achieved.

In considering fail-safe design, the objective should first be to make it impossible for the error to occur. If this cannot be achieved, then it is necessary to ensure that the error does not go undetected. If the error is detected, the only way to ensure it has not induced a product or package failure is to stop the machine or to remove potentially affected items from the machine with an automatic ejection system.

POSITIVE FAIL-SAFE DESIGN PHILOSOPHY

Fail-safe design and FMEA analysis can be applied to software as well as to mechanical systems. Such analysis dictates the use of a "positive" fail-safe philosophy throughout the software design—especially for elements such as inspection systems or shift registers that sequence faults through the packaging machine. The positive philosophy is a method of processing good signals only. Any type of reject or failure would result in an absence of signals, indicating failure or "bad" product. In this way, an overall system failure or aberration that would result in an absence of signals would lead the machine to treat the occurrence as a failure by default.

A classic example is a shift register that tracks bad product through a machine to the eject station for automatic rejection. If the shift register is designed to process negative or bad signals (as many systems are—one failure leads to one signal) and the shift register loses the contents of its memory, all good and bad product will pass by the reject station. If the shift register is designed to process only positive or "good" signals (i.e., positive philosophy) and loses the contents of its memory, all good and bad product will be automatically ejected at the reject station, and then presumably after a number of consecutive faults, the machine will be programmed to stop.

How does it work in practice? The machine should be designed with a sensor to detect any product present just prior to the reject station. If the control system sends a positive "all clear" signal associated with the product, then the reject station is signaled to hold back and allow the product to pass. But if the reject station receives no positive signal as a result of a problem (i.e., heat was out of limit, pressure was too low, etc.), then it automatically ejects the product.

If there is a system failure such as a power loss or the control system becomes locked in some undefined state (like the blue screen that sometimes appears on desktop PCs), then no signals will be sent to the reject station and all product will be automatically rejected. Even in the case of a well-meaning employee placing a stray product back into the process, the reject station will sense the product's presence but the control system will not send an associated positive signal. As a result, the product will be automatically ejected.

The reject station would be fail-safe using positive-signal processing. But what happens if the air line or pneumatic cylinder that controls the reject station fails? In that case, when the reject station senses the presence of a bad product and the control system does not send a positive "hold back" signal, the product that should be ejected in the next step instead may go out the door with the good product. If this could occur, an eject verification sensor must be added in the reject chute as a double-redundant system. If the reject station does not take bad product out of the process for any reason, the eject verification sensor will not see the bad product pass through the reject chute and therefore will signal a system defect and stop the machine. The addition of the eject verification sensor makes the reject station fail-safe and double redundant.

VALIDATION TESTS, MANDATORY PLANNED MAINTENANCE, AND SOPS

The risk analysis and fail-safe design process readily identify the required validation tests. Maintenance issues that can impact validation are also identified as a result of the process, as are the standard operating procedures necessary to maintain package integrity.

Part of the operation qualification of a packaging machine involves challenging system-defect devices. An error is artificially induced for each element that has been designated as a risk, and the machine response observed. For example, with the reject station, the air to the reject mechanism is turned off and a bad product is introduced into the process. The expected response is that the eject verification sensor will not detect the ejected product and signal that a system failure has occurred.

Under all circumstances, the machine or process is expected to detect the fault and either stop the process immediately or reject the offending product. These tests are an essential part of the validation process and should always be performed to demonstrate that fail-safe design methods are working. Typically, if fail-safe design methods are used, a significant proportion of the validation testing procedures are created to ensure that the double-redundant systems are working.

In endeavoring to determine whether or not mistakes can be made, it is best to assume that if something can happen, it probably will. No matter how elaborate or complex or expensive the system is, failure can—and often does—occur. With this in mind, consider the following simple examples of components included in most packaging machines.

Pneumatics. Pneumatics are widely used and are often critical to the integrity of the pack or process. This especially applies in the case of sealing machines. Pneumatics are prone to wear, however, and can often work intermittently. It is possible for solenoid valves to stick for one cycle or for a cylinder to spasmodically fail to reach its desired position. The result could be a single faulty pack, or a single missed reject. If such events occur infrequently, they are most likely to go undetected, and as a result even an ostensibly validated process will fail.

This risk can be designed out, however, by ensuring that the correct positions of the pneumatic components are independently monitored and the positional information is sent back to the control system. Such a system would require the failure of both components (the primary device and the secondary sensing device) before product integrity could be impacted. In designing the system, the secondary positional sensing element must be positive fail-safe. The controlling system should monitor the status of the secondary element such that it cannot be overridden. The validation test should challenge both the operation of the secondary element and its positive fail-safe nature.

Servomotors. Servomotors operate by receiving instructions from a motion controller to move to a predetermined position. The controller observes the motor's position from feedback, usually in the form of pulses from an independent motion detector, known as a resolver or shaft encoder. Information is constantly updated so the controller always knows the position of the motor. The controller sends constant commands to the motor to move to the correct position at the correct time. If not robustly designed, however, the return signal could degrade or be inaccurate, in which case the servomotor will move to the wrong position. Depending on the nature of the motor's design, it is possible for such a fault to rectify itself during the next machine cycle, leaving just one potentially faulty product.

This potential risk can be addressed in one of two ways: either by choosing an inherently fail-safe servomotor and control system, or by adding secondary monitoring systems to separately detect the correct position of the machine on every cycle. Again, the testing procedure should challenge the robustness of such systems under the fault conditions that have been identified in the risk analysis.

Temperature Controllers. Most temperature-control systems are considered robust. It is only when risk analysis or FMEA is applied that the scope for potential errors is realized. For example, a heat-sealing-type packaging machine must hold its sealing temperature steady within a predetermined operating range or otherwise signal a system defect. All such machines should have double-redundant and fail-safe design to ensure that no product is packaged outside of the operating range.

In the case of a heating system with multiple heater cartridges but only one feedback temperature probe, however, there is a risk that a single element could fail without being detected by the temperature probe. By ensuring that all heaters are connected in series—so that if one heater fails, all will fail—the temperature probe is then sure to detect the failure and the risk can be eliminated. Alternatively, current-monitoring devices for each cartridge can be used to eliminate this potential risk.

CONCLUSION

Double-redundant and fail-safe design methods are becoming essential features of packaging machines in a validated environment. Properly applied and embraced, they will further increase the reliability of the machines and the quality of the products emerging from the medical device industry.

More importantly, double-redundant and fail-safe design methods will help focus validation on machine enhancement and reliability, and reduce dependence on operator vigilance and intervention. This approach is the only certain way to eliminate the transient faults and aberrations that have arisen from the increased complexity of modern machine design.

Ray Johnson is president of Doyen Medipharm Inc., Lakeland, FL.

Copyright ©2002 Medical Device & Diagnostic Industry