Everybody hates an unplanned machine failure. Your customers hate them because any unscheduled shutdown means loss of production, loss of revenue, and maybe even the need to call in the cavalry to fix the machine immediately.
Machine builders hate unplanned failures because they're the cavalry. When your phone rings, it's the unhappy customer on the other end of the line, and he wants your troops to arrive right now and get things running again.
Maintenance and repair people at both the user and machine builder hate unscheduled shutdowns because of all the pressure exerted to get the machine running again.
And, because of the old rule about what runs downhill, it seems like control system designers always take the heat. Everyone understands when a motor or gearbox fails--it's mechanical, and failures are to be expected. But no one understands when a control system sputters, the HMI screen goes blue screen, or the network stops networking.
Fortunately, modern machine control technology is coming to rescue, making it possible for control engineers to build exceptional reliability into their machines, increase uptime, reduce the number of unexpected failures, and become heroes instead of villains.
Here are a few design ideas to discuss with your customers as a means for them to achieve competitive advantage through uptime and reliable operation.
A way to boost uptime is to minimize downtime. Sounds obvious, but many companies discount the time wasted to refixture a tool and reset the machine to run a new part.
"We have seen our customers move away from dedicated machinery that only allows one particular part to be manufactured," notes Jason Munk, controls engineer and project estimator at machine builder QSI Automation. "Now, the majority of our customers have a family of products that are required to run on one automated system."
It's the same everywhere, it seems. "Job changes are frequent in die casting, especially now that many plants are using Lean Manufacturing and holding less inventory," says Dave Woods, product development manager at Rimrock, a builder of machines that make die cast parts.
Running smaller lot sizes puts pressure on a machine and also on the company's ability to process orders. "Historically, manufacturers would regularly receive big orders for 10,000 products," says John Warner, president of Accenture, an asset management software company. "Now, orders require smaller quantities and speedier turnarounds."
Advances in hardware fixturing can cut downtime during changeovers. QSI, for example, uses "family-oriented fixturing" that allows an entire group of products to run using identical tooling. "Another method we use is simply making the fixturing rotational," explains Munk. "When a product is changed, just remove a couple of bolts and rotate the fixture 90 or 180[degrees] to accommodate the new product."
Michigan Custom Machines builds special machinery for assembly and test of diesel fuel systems, such as injectors, pumps, nozzles, and subassemblies. "We have designed our machines to be universal," says Michael Schena, president of MCM. "Injector and nozzle machines accommodate several different products with a 5-min. fixture and component change."
Hardware fixturing is not the only way to accommodate rapid part changes. "We are doing a lot of work in the automotive industry here and in Europe to convert robotic assembly stations from traditional hardtooling part location to visual fixturing," claims Jordan Merhib, vice president of Isra Vision Systems.
Merhib explains that robots are programmed to perform a function along a predefined path. With traditional mechanical fixturing, a vehicle or part must be precisely located at the exact same point every time. For a vehicle, the required accuracy is 1.0 mm. "The tooling to do this is expensive, subject to wear, and model specific," says Merhib.
By using cameras for visual fixturing, the vehicle can be stopped anywhere within 25 mm of the target, a feat easily accomplished with a typical conveyor. "We then use the vision system to provide the required precision," he says. Essentially, position data from four cameras determine the exact X, Y, Z coordinates of the vehicle, and this information is downloaded to robots. "They are then able to perform their function to the 1 mm tolerance required."
Typical time for this process is 2-3 secs., and it can accommodate multiple products coming down the line. Merhib reminds that visual fixturing has been used in circuit board assembly machines for years.
Completely automatic changeovers are possible, too. "We design machines to handle various parts using servos and PLCs to change tooling position," says Edwin Zimmerman, director of technical operations at Apex Automation. "The factory automation industry is using more servos and electronics where there are stringent requirements for frequent product changeovers." But not everywhere, it seems. "There is still a niche market for low-cost, single-product machines," he adds.
"We try to work with components that allow for automatic downloading of software parameters for part changeovers," says Jeff Bell, manager of automated assembly systems at Taylor-Winfield. "The operator pushes one button and the system automatically changes over the weld controller, robot program controllers, laser and impact marking systems, electrical testing systems, and so on."
Having the control system in charge of product changeovers certainly makes it easier on the operator and ensures the machine, robots, and test equipment all know what product is coming down the line. "By giving the operator the ability to perform changeovers with the push of a button, we have been able to dramatically reduce changeover time," says Steve Stegora, manager of material handling systems at PaR Systems.
Simplifying With Servos
Another cause of downtime is the need to maintain, adjust, and repair the many mechanical devices that make up a machine. To reduce complicated mechanical systems, many machine builders are turning to versatile servo motors instead of fixed automation.
As Zimmerman noted, parts setup with servos eliminates the need for some fixtures. "I'm using more encoders on main drives and servo systems slaved off these encoders for speed, direction, and positioning," he says.
Servos are also replacing the lineshafts in the filling, packaging, and converting industries, a approach that builds complex machines driven by one main drive/motor. Servos can replace the lineshafts, takeoffs, pulleys, and gears in these machines.
"We are starting to get more questions from our machine builder customers about building servo-based machines," says Mike Nager, marketing manager at Phoenix Contact. "They are interested in the machines that have been coined 'Generation 3' by the Open Modular Architecture Control (OMAC) Packaging Group. The idea is that, by eliminating much of the mechanical components, both the machine OEM and end user will benefit from reduced costs and faster startup." Nager says several machine builders showed Generation 3 machines at InterPack and PackExpo last year, all designed around PCs and servo motors.
"With the cost of servos and stepper-motor packages coming down, the opportunity to use them will increase because of the flexibility the system will have for different operations," adds Bell.
Preventive Maintenance Palls
Simplifying mechanical parts is an excellent step toward increasing uptime. Alas, this is not where unexpected failures occur. It turns out that most failures are random, and not in the mechanicals.
According to a white paper by Aladon, a company that specializes in reliability-centered maintenance (RCM), studies in the commercial airline industry completely upset the conventional theories of maintenance and reliability.
A Pattern of Failures
Thirty years ago, devices failed when they wore out (top). Today, devices fail according to these six patterns of performance.
A. High incidence of infant mortality, followed by a constant probability of failure, then by a wear-out zone.
B. Traditional pattern: Constant probability of failure, ending in a wear-out zone.
C. Slowly increasing probability of failure, but no wear-out zone.
D. Low probability of failure when new, then rapid increase to constant probability of failure.
E. Constant probability of failure at all ages (random failures).
F. High infant mortality, then constant or slowly decreasing probability of failure.
"Many people still believe that the best way to optimize plant availability is to do some kind of proactive maintenance on a routine basis," says Aladon. This assumes a traditional view of failure, where devices fail as they enter a wear-out zone after a certain period of time.
This was true 30 years ago, but equipment is much more complex these days. Today, there are six identifiable patterns of failure to deal with (Figure 2).
And here's a startling development: According to Aladon, studies on commercial aircraft showed that a whopping 68% of failures conformed to "pattern F"--that is, high infant mortality followed by random failures.
"These findings contradict the belief that there is always a connection between reliability and operating age," says Aladon. "Nowadays, this is seldom true. Unless there is a dominant age-related failure mode, age limits do little or nothing to improve the reliability of complex items. In fact, scheduled overhauls increase overall failure rates by introducing infant mortality back into otherwise stable systems."
Adam Brown runs tech support for the Odd Form and Final Assembly Div. of circuit assembly equipment maker Universal Instruments, and he agrees with these findings, especially when comparing today's equipment with the cam-driven and relay-timer circuit control systems of the past. "There were many moving parts involved, hence a more predictable rate of failure," he says. "Today, software and programming is what makes or breaks a machine."
The equipment used in aviation and machine control systems is similar. Both use pneumatics, electro-hydraulics, servomotors, networks, control valves, pumps, miles of wire and cables, networks, computers, electronic controls, and flow, temperature, level, and pressure sensors.
This does not apply, of course, to the mechanical parts of a machine, such as pumps, motors, valves, and so on, which do suffer age-related failures. "Most of the mechanical items will begin to show wear before the component completely fails," says Bell. "The electrical components that are involved in complex, high-tech systems usually show no wear or failure pattern when they fail."
Zimmerman echoes this sentiment. "Factory automation is becoming increasingly more controls-intensive, and it is difficult to predict the useful life of electrical components," he says. "But the work is still done through power transmission devices, manipulators, end effectors, and tooling that contacts the product. These items can still be inspected for wear and replaced based on life predictions."
Most mechanical failures are relatively predictable, and when the components start to wear they don't necessarily stop the machine. Machine builders are more concerned with failures that stop the machine and cause unscheduled downtime.
"QSI agrees that the majority of failures we see are random, and the items that fail have no particular maintenance schedule anyway," says Munk. "Generally, the failures our customers see are not related to the mechanical items of the machine."
As noted, everybody hates surprises. But with the increasingly technology-rich nature of today's machines, and with failures of high-tech components being random, what's a machine builder to do?
Building in diagnostics can help. "We have historically used sensors to detect problems such as parts not in place, stroke not completed, part not ejected, and the overload of an actuator or tool," says Zimmerman.
"Every component in our design has a status indicator," says Woods. "Older systems may have had 40 LEDs or component indicators within the enclosure; a typical new design has more than 150 indicators. We even display the temperature of the control box."
Having too many alarms can cause problems, too. Damon Clinch, controls engineer at Perceptron-USNR, a global supplier of sawmill machinery, says his company is using smarter alarms in its control system diagnostics because lots of information didn't always identify the problem. "Any time the outfeed stops on a board edger, a board could come to rest on the parking ramps," he explains. "Since the board blocks all the photoelectrics, all eight table alarms come on, clouding the real reason the machine stopped. We now monitor the photoelectrics only after we think a piece should have been fed into the machine, or if we are waiting to load a new piece on the table."
End users are getting into the picture. "We are starting to see more customers request remote diagnostic and service capabilities," says Bell. "Most of our systems now incorporate diagnostic capabilities."
A machine's control system can identify problems if a sensor sounds off, but it can also predict problems by looking for suspicious data. "We take special care to address failures in software, so that if a component fails or is beginning to fail, we can identify the problem quickly," says Schena. "We actually have our machines track the standard deviation of its measurements and set an alarm if a threshold is exceeded."
This sounds like asset management (AM) being put to work in the machine business. Over on the process control side of the business, engineers have learned to capture data from fieldbus equipment, HART-enabled instrumentation, and SCADA systems and DCSes. They feed this data into AM and computerized maintenance management system (CMMS) software, which analyze all the data and look for indications that certain assets, equipment, or instrumentation require tending to. HART and fieldbus-enabled instrumentation and control equipment perform internal diagnostics, monitor critical parameters, and provide status and condition information to a control system.
AM and CMMS for maintenance and diagnostics are starting to take hold, both at the machine builder level and with the control hardware and software vendors.
Bob Nelson, product manager for engineering and PC-based control software at Siemens Energy and Automation, says even the highest quality hardware and software cannot completely eliminate faults. Therefore, consideration should be given to the abilities of the control system itself to provide complete and informative system diagnostics (module failure, channel faults, etc.) and machine diagnostics (interlock not fulfilled, limit switch fault, etc.).
"An engineer can design the control strategy in ways that also increase uptime," says Nelson. "Universal programs can be designed that can be developed, tested, and applied to a variety of configurations--thus complementing the increasingly flexible electromechanical design of a machine."
Accenture's Warner agrees that advanced control schemes can replace preventive maintenance. "We're helping our clients move into predictive maintenance. This means that real-time data is feeding up into the MES applications and forming a body of information that provides mean time to failure and repair among various automation components," he says.
Some of this AM and CMMS capability is already built into the many HMI packages. "Companies with a limited support staff can benefit from using advanced software tools, such as the productivity analysis features in Entivity PC control software," explains Jim Allison, product manager at AutomationDirect. "Entivity provides easy to develop and understand data and graphs that analyze downtime by cause and duration, and compare production yields by cycle, across shifts or lies, or as compared to an ideal yield."
Just about all the major SCADA/HMI software vendors offer the same or similar capability. Therefore, when considering the addition of new HMI or OI software for a machine, it might be a good idea to investigate the software's ability to help out on the maintenance side.
"Some of our larger customers are purchasing SCADA systems, such as Wonderware and Intellution, for HMI, data collection, and communications," says PaR's Stegura.
Networking for Reliability
Ethernet-based networks are starting to take over the machine building business. At first, machine builders added networks when end users wanted their machines to participate in factory networks. Then, builders found running a network through the machine proved to be much less expensive than running wires for every sensor, motor starter, and switch all the way back to the controller. The final kick came when builders discovered networks were actually more reliable and easier to debug than hardwired systems.
"Networking is more reliable than hardwiring," says Woods. "Transferring signals electronically uses a lot fewer components and can be configured very quickly without pulling wire."
Woods uses PLCs and Ethernet networks from Schneider Electric, but not every machine is networked. "Most of our systems are hardwired between machines," he laments. "At least we now have the option to network them using Ethernet-to-anything-else converters."
You can buy networks with diagnostic capabilities built in, says Dave Quebbemann, industrial automation marketing manager at Omron Electronics. "More reliable, intelligent diagnostics like snooper devices are being built into network systems," he says, "With networking, you have a lot fewer wires to track down. This lends itself much easier to modular subsystems."
Networking modular subsystems is definitely a trend. Apex Automation says it is doing much more networking of I/O and modules, Michigan Custom Machines links all its assembly and test system modules via a single communication line, and PaR Systems uses a distributed I/O and networking structure when it builds packaging and palletizing machines.
"In today's control systems, Ethernet has proven to be a very popular and reliable networking interface," says Clinch. "Most sawmills are networking all control and optimization systems to their main computer network to gain access to the production information. We have standardized on Ethernet for PLC remote racks, motion controllers, HMI, optimizer, and variable frequency drives."
One way to increase reliability is to buy better components. Theoretically, it should be easy to convince end users that high-reliability components lower overall lifecycle costs. But, in some potential customer minds, such an argument violates the three most important aspects of any machine design: cost, cost, and cost.
"High-reliability products come with a higher prices, which ultimately drives up the cost of automated machinery," laments QSI's Munk. "It seems so many companies are now buying on cost alone, they've forced us to use less-costly and less-reliable components. We have not yet seen the trend return to the quality of the equipment outweighing the cost."
Brown at Universal Instruments agrees: "The focus has lately turned to minimizing cost so people have lost the long-term perspective on automation. We usually back up our products with a one-year warranty, so we are concerned with selecting components that can meet this lifecycle for the lowest possible cost."
Fortunately, many components for machine builders are getting more reliable all the time. Companies that make high-reliability components are being forced by the market to lower prices to stay competitive, and the increasing tendency of machine builders to monitor parts has led to better quality. "We monitor our warranty cost and component replacement costs on a regular basis and remove any suppliers if they start having problems," says Bell.
Word gets around, so many companies have learned which suppliers to avoid, and which ones to insist upon. When it comes to PLCs, sensors, switches, motor drives, and similar parts, machine builders often run up against customers who demand that certain parts be used in their machines. When machine builders have to use certain components that are noted for reliability, it levels the playing field.
Another way to level the playing field is to factor in the cost of installation, maintenance, and warranties when specifying components. "Machine builders are concluding that, when factoring in the maintenance costs to the whole lifecycle of the machinery, the total cost can be less when a more expensive but durable component is installed," says Susanne Mettberg, product specialist at Lumberg.
Sometimes it's not that complicated. One customer, she explains, had a crankshaft grinding machine that used syntactic cooling fluid. The chemical agents in this fluid are extremely harsh and wreak havoc on any components inside the enclosure. "Previously, the manufacturer was using the cheapest I/O distribution box it could find, but they were only lasting a few weeks before breakdown," she says. "They were even wrapping the I/O boxes in plastic trash bags. Our solution was to install an IP-67-rated modular bus station, which is much better suited for such a harsh environment. This means less downtime."
For similar reasons, machine builders may want to specify terminal blocks with nickel plating to prevent corrosion; quick disconnect cables to minimize installation and maintenance costs; devices that protect themselves electronically instead of by fuses or circuit breakers; cord sets with exceptionally long flex life and high bending cycles (Figure 3); and wires and cables that are protected against lubricants, oil, and chemicals. Although all these cost more, the savings they provide in reduced maintenance and repair costs can offset their higher costs.
Taking Downtime Under Control
Some techniques for increasing uptime are mechanical in nature, as we've seen. But diagnostics, condition monitoring, managing networks, acquiring data, and coordinating predictive maintenance, asset management, and CMMS are definitely control functions, and they are coming your way. Your life may never be the same.
By designing control systems that support reliability and increase uptime, plus diagnose problems quickly to decrease downtime, a good control system designer at a machine builder can make or break the entire company. Today, you may be the goat when a machine stops for no apparent reason; tomorrow, you may be the reason why your company wins orders on its reputation for having reliable equipment.