Benefits of Remote Diagnostics and Monitoring

In 2005, We Saw How the Benefits Gained By Remotely Troubleshooting Your Customers' Machine Controls Can Far Outweigh Costs Involved

By Rich Merritt

 

August 2005, cover imageRemote diagnostics and troubleshooting capability is a hot button for machine builders and customers today. As the machine builder, you'll rightly ask, "what's in it for me?" Well, it's becoming a bit clearer that you can drastically cut your costs of warranty service, routine service calls, and maintenance of customer machines. 

With remote diagnostics, your support engineers and service techs can determine what's wrong with a machine, tell the customer how to fix it, or dispatch a tech with the correct parts. They can do this from the comfort of your central office, 24 hours a day, seven days a week. Your techs even can do it on vacation, with a cell phone from that beach in Hawaii.

SEE ALSO: Three Ways to Remote Diagnostics

But there is a price to pay. You have to buy "smart devices" with built-in diagnostics. You have to integrate the diagnostic data into your machine's control system and HMI. In addition, you have to set up a central office to handle maintenance and diagnostics.

The payoff can be astounding, because you can minimize warranty and service costs, while you give your customers the best service in the industry.

Better Service Sells
"The simplicity of remote condition monitoring over the Internet gives us an edge over a standalone machine without connectivity," says Francois Broche, chief controls engineer at specialty machine builder Southern Engineering & Automation. "It lets us offer an affordable service contract because we can proactively keep the machine operating without flying to the customer location." Beryl Lawrence, Southern's sales manager, adds, "It's better marketing through science."

Mike Krolak, senior director of controls technology at Asyst Technologies, agrees. Asyst builds Spartan EFEM (equipment front end module) machines for semiconductor manufacturing, and it also uses web-based monitoring. "The idea behind this web functionality is to take advantage of remote diagnostics from machine level to enterprise," says Krolak. "The impact in predictive maintenance and field service gives the Spartan EFEM a clear advantage in customer service, machine availability and reliability."

It's not just machine builders who are discovering the benefits of remote diagnostics. Control equipment vendors are there already. Siemens is enabling as many of its products as possible for remote services. According to its magazine, Pictures of the Future, Siemens has 40 specialists in Karlsruhe, Germany, who are responsible for equipment at 140 power plants worldwide. Siemens also operates a Power Diagnostics Center in Orlando, Fla., that monitors the condition of 180 gas turbines around the world. 

GE Fanuc started putting advanced diagnostics in its Genius I/O more than 10 years ago, says Paul Scanlon, manager of machine automation solutions. He says GE Fanuc even set up a remote monitoring and diagnostics (RM&D) program for OEMs. RM&D collects data from remote assets, and provides the software and hardware tools for monitoring equipment performance and performing diagnostics. "This enables enhanced collaboration between manufacturing plants globally, and between an OEM and end user, within a secure environment," he says. Advantages of a remote diagnostics program are shown in the tables at the end of this article. 

Start with Smart Stuff
For this article, we asked machine builders and vendors alike if it was possible or practical for a machine builder to design its own machine diagnostic system from scratch. Most said it was too much of a challenge, required too much deep hardware and software knowledge, and it was a waste of time, because you already can buy everything you need in the form of smart products with built-in diagnostics.

"While it might be possible to build your own system, it certainly is not advisable," cautions William Parker, control systems engineer at American Axle & Manufacturing. "Smart products make diagnostics less costly to implement and help speed your product to market. The initial cost to implement usually is regained in the first downtime incident."

Parker has numbers to back this up. "I added smart power monitors to high-speed welded radiator tube mills headed for China," he notes. "They saved our service people and the customer innumerable days of troubleshooting when power fluctuations were the root cause. The smart power monitors cost about $700 each. My service engineer's time on site was booked at $993 per day, and he was averaging three power alerts per week. He didn't waste time making useless weld adjustments when power stability was the real issue."

Motors might be your biggest source of problems, so smart motors could be a smart investment. Fortunately, most of the major motor manufacturers already offer motors with built-in diagnostics. 

Roger Bigler, CEO of Animatics says his company's motors make such information available. "With a SmartMotor, the internal program, a PLC, or an HMI can read the amount of current flowing through the motor, the precise voltage level applied, plus dozens of error flags relating to virtually every aspect of the motor's communications and internal operation," he explains.

Baldor has a similar system. Jeff Lovelace, product manager of drives, says the H2 drive has a diagnostic screen that monitors many variables while the motor is running. "These include all digital and analog I/O, line voltage, bus voltage, encoder feedback, motor voltage, current and speed," he says. "This allows a technician to view several voltages and currents without the need for multimeters or scopes, not to mention the benefit of avoiding live wires."

SEE ALSO: Why Go Wireless: Benefits and Challenges of Wireless Remote Monitoring

Siemens makes more than operating 4,000 parameters available for its MicroMaster 440 drive, probably more than you could use. 

Combining smart motors with suitable diagnostic software solves many problems. "We started having trouble with our drives and after conducting a cost/benefit analysis, we discovered that replacing all of our controls and drives was the most prudent and logical option," says Brad Kennison, national sales manager at Nova Verta, builder of automotive and industrial spray paint booths (See Figure 1).

FIGURE 1: SPRAY BOOTH GETS SMART

Nova Verta builds paint booths equipped with remote diagnostics, so technicians in a central service center can help customers find problems over the phone. Source: Nova Verta

Switching suppliers, installing new Rockwell Automation technology and making sure it integrates with the existing architecture are very difficult changes to make, adds Kennison, but he says it has paid off, especially in reduced service costs. Kennision says Nova Verta eliminated 98% of its technical service calls and now is better able to clearly identify the source of customer problems. If a problem arises, Nova Verta's staff can troubleshoot over the phone. This allows technicians to diagnose the problem remotely rather than having to send someone out to the customer site. Also, operators are better able to monitor the drives through the touchscreen display that allows more effective troubleshooting.

Nearly every motor and drive manufacturer has or is developing similar capabilities. That means, with a little research, you could replace every motor in your machine with a smart motor that will provide diagnostic information to your controller. That could help identify half of the potential problems in your machine.

Smart and Complex
Complex devices such as vision systems, motion controllers, vibration analyzers and communications systems can be a bear to troubleshoot. The problem gets even tougher when a motion control system works with a vision system over a communications link. When it fails, which system is at fault? Fortunately, because those manufacturers also realize the sales value of built-in diagnostics, you can buy smart devices that will simplify maintenance.

For example, Danaher/Motion Engineering's motion controllers have built-in testing capabilities that Asyst used in its Spartan EFEM machine. "With the Windows-based interface and a SynqNet motion controller, I could excite any axis or two axes simultaneously, and see the frequency response of the system like never before," says Asyst's Krolak. "And, since the Spartan EFEM host runs VxWorks, I could perform all analysis testing in client-mode from a remote host."

Communications problems plague everyone, even when using a tried and true network such as Modbus. Kepware ran into this when solving customer problems. "In the past, when a customer encountered an issue with one of our communication drivers, we would be flying blind unless we went to the site," says Mark Hensley, president of Kepware. "This was a real problem with devices that use Modbus." It seems that some hardware developers take a little liberty with the protocol and create problems. "In every case, he adds, "we found ourselves saying, 'if only we could get a protocol analyzer on the link, we could figure this out.' After saying that enough times, we decided to build one into the product."

Seeing the benefit of its Modbus analyzer, Kepware put an OPC analyzer into its drivers. Because of the importance of OPC in linking allegedly compatible equipment from different vendors, such a device could be of great help to a machine builder. "Unlike a communications link that is serial or Ethernet, the OPC connection is essentially hidden within the inter-process communications realm of the Microsoft COM API," explains Hensley. "We added special diagnostic codes to every OPC function in the server and display this data so that it would be useful to us and the customer." 

Wago has a similar system. "With any network product, it's crucial that devices be able to detect faulty network conditions and either maintain or shutdown a process if it loses communications with a host device," says Mark DeCramer, product manager, advanced electronics. "Wago's intelligent bus couplers detect the status of the network -- Ethernet, DeviceNet, Profibus, etc.--and react in a predetermined manner. Network errors, internal or external, can be difficult to troubleshoot, so having these diagnostic and reaction capabilities is a tremendous benefit in a control system."

Machine vision devices also have diagnostics. Cognex, for example, has a CountError function built into its InSight vision sensors. The tool can be used to monitor and track certain functions, and report when they are not working. "The technique can troubleshoot and provide a diagnosis of why a measurement or application is failing," says Mark Sippel, product marketing manager. 

Some vibration analyzers diagnose the problem themselves and simply report their findings. For example, a vibration analyzer from Emerson Process Management automates rotating machine diagnostics. It plugs into a motor/pump unit, checks for problems, and reports anything it finds via bus or serial link. Such a system costs $4,000-$7,000, but if it can protect an expensive rotating machinery system, it might be well worth the cost.

For other machine sensors, you probably don't need anything more than standard proximity sensors, limit switches, and other inputs you already use for control. You do need, however, some means of determining when an input has failed. "The first line of built-in diagnostics is supported by extra sensors, such as a prox at both ends of motion," advises AAM's Parker. "This kind of simple diagnostics easily can be included in PLC logic and displayed on HMIs. This is commonplace in most automotive manufacturing systems. All the machinery in my plant has at least this level of basic diagnostics. Most have significant portions of their logic devoted to fault isolation and troubleshooting."

Dealing with Diagnostic Data
Now that you have streams of different diagnostic data coming into the control system from smart motors, drives, networks, motion control systems, vision systems, vibration systems and all the regular I/O on your machine, what do you do with it? 

Fortunately, almost every vendor that puts diagnostics into its hardware has a little software program that inputs, formats and diagnoses the data. Wago has Wago-IO-Check software; Motion Engineering has a suite of design and diagnostic tools; Baldor has a setup and troubleshooting tool; and Cognex has In-Sight Explorer software. You can bet if a vendor has built diagnostics into a device, it also has a software tool so its own techs can extract and analyze that data. 

"I hadn't really thought about how many new diagnostic software packages I had acquired in recent months, until I considered the questions you posed in researching this article," says Parker. "This is due to the sophisticated network-based controls in our latest products. I need these packages to provide my front-line first responders with the best tools I can."

Even if you don't set up a remote diagnostics system over the web, smart devices will make troubleshooting much easier for your service techs when they arrive on site.

To achieve remote diagnostics, first you have to set up a system to collect all the data, put it some recognizable common database, diagnose the system in real time at the machine, report problems to the customer and to your remote service center, respond to inquiries from your service center, and carry out on-line diagnostic procedures from afar. That's a long list to ponder.

Getting the data in might not be as difficult as it seems. Most of the diagnostic software tools from equipment suppliers are Windows-based, so they already are more or less compatible with your HMI software. Further, you can put data compatibility requirements into your purchase order to ensure that the data arrives in a desired form, say, in a flat file, ASCII text or something suitable for an SQL database. This puts the compatibility monkey onto the equipment suppliers' backs.

Magnificent Obsessions
Once you have the data in your control system, you have to write a few software programs or configure a few HMI screens to figure out what's wrong with the machine when problems occur. "I tell our customers that when they have the motors doing precisely what they want them to do in the machine, they are 20% complete with their programming," says Bigler of Animatics. "They now need to obsess about what might go wrong, how that would be seen by the many sensing and recording functions within the motor or system, and program the unit to offer this information upstream for delivery to the operator. For example, if the software is programmed to check that the motor is operating 10º F hotter today than it normally does, a message can be sent to lubricate the mechanism before failure. Motor current or position error can be read to detect jams. There is no limit to how clairvoyant the machine can be with some sensing capability and a little programming."

Playing "what if?" is a common technique for most software programmers. Essentially, you work down a decision tree that goes through a logical sequence of questions. Let's say a digital input indicates the machine is not feeding parts to the assembly module. The diagnostic logic might ask if the parts feeder is on? Are there parts available for the feeder? Is there a jam? These essentially are the same questions that your service tech now asks when a customer calls in, complaining that its machine isn't feeding parts.

"Data is the key," says Matt Miller, OEM sales and marketing manager for OSIsoft. "I have seen two approaches to diagnostic solutions used successfully, namely model-based and rules-based. The model-based solution generally is more comprehensive, requires more time to implement, and is suitable for complex applications. Rules-based solutions tend to be easier to embed in a standard product, can be replicated widely and tend to be simpler to use by field personnel."

A model-based solution gathers reams of operational information for the machine builder over time, and develops algorithms to determine if the machine is running properly. A rules-based system is what was described above.

The trick to automating the diagnosis is to have all the inputs you need to answer the questions, so, in the above example, you'd need sensors that indicate the feeder actually is on and parts are available, the motor current on the parts conveyor indicates a jam, and so on. As Parker says, much of this diagnostics can be done in PLC logic.

Going Remote
The huge benefit of remote diagnosis is the elimination of service calls, because you can do all the diagnostics you'd do on site in your office. To accomplish this, you need to set up a system so the server HMI in your office sees the same data as the client HMI in your machine. 

The technical details of setting up such a system are beyond the scope of this article, however, most HMI software from major vendors can operate in client/server-mode, where a remote server HMI can communicate with multiple local machine HMIs via a private network, Internet, intranet, modem, cell phone, microwave, satellite, etc. 

Schneider Electric offers such a service. "We have web-enabled PLCs that allow the customer to read and write data to the PLC over the Internet/intranet," says Val Tur, senior engineer at Schneider Electric. "He can read/write data via a table format or he can customize graphics via the Java Enabled Graphics Editor to his own layout preferences. Also, the web-enabled PLCs include a 'Rack Viewer' that allow the user to see the diagnostic LEDs on the PLC via a web browser in real time."

FIGURE 2: PLCS TALK TO THE WEB
This robotic tray packager uses a PLC that has web access, so the machine builder can see live data from the customer's PLCs, drives and a camera over the Internet.
Source: Southern Engineering

Southern Engineering uses Schneider PLCs (See Figure 2). "We used a modem, but that's antiquated technology that gave us limited access," says Broch. "We can do and see a lot more through Ethernet, because the critical control components -- PLCs, drives, HMI and a camera on the machine -- are connected, giving us a lot of accessibility." 

Joe Moore, director of information technology, Radiology Consultants of Iowa, says setting up an Internet link is really simple. "Set up a virtual private network at your server, and use the Internet to link HMIs at individual machines to your central server," he advises. "This gives you a secure link, because the VPN acts like a private network within the public network. Then, the server can treat all the client systems as if they were on a network within a plant."

With a VPN or a similar system, your central server can contain one copy of the historian, asset management, CMMS, vibration analysis and other ultra-expensive software, and use that software to process data from hundreds of individual machines, no matter where they are in the world. All the remote machines need is access to the Internet.

With the advent of MCoIP (Mobile Control over IP), your techs can call up measurement data and diagnostic information over their cell phones. Therefore, even if they really are on vacation in Hawaii, they will still be able to service their customers from the beach. 

No rest for the weary, thanks to modern technology.


Like this content? Get it delivered straight to your inbox! Click here to sign up for free e-news alerts.