It should be no surprise that the three most commonly referenced issues regarding how we obtain reliable networks are: 1) design, or getting it right in the first place; 2) installation, meaning proper equipment and construction methods, which are not the same as for analog systems; and 3) training, so the people responsible for providing the design, installation and ongoing maintenance have the skills to perform the work.
SEE ALSO: Designing a Cost Effective, High Performance and Highly Reliable Industrial Ethernet Network
Proper design needs to include the capability to incorporate appropriate diagnostic abilities. Once in use, just like with any other piece of equipment, maintenance practices must include periodic check-ups and baseline measurements to support proactive repairs that ensure uninterrupted network operations. Baseline measurements need to include successful implementation of network maintenance practices that will minimize post-failure repairs, which is when most traditional diagnostics are performed.
Dive Into It
Derrick Stableford, electrical technologist, Associated Engineering, believes that to be effective, you need to be able to "get into the weeds" at times, and adds that, "Probably one of the big skills is the ability to read/interpret an oscilloscope. A scope is the multimeter of networks, and being able to determine noise levels and square wave voltage levels are biggies."
Personal experience has shown it can be difficult to identify each waveform with the associated source of that waveform and be able to associate it with the anomaly, so having a way to trigger the scope on the offending or a particular node is useful.
Merril Harriman, Ethernet systems architect for Industry Business at Schneider Electric, says you also need the right person, an analytical thinker, and he reaffirms the need for the right tools. "There are many simple tools (ping, tracert) that can be employed very easily on a PC, but often are of limited help in an industrial network," Harriman says. "Knowledge of other tools such as SNMP MIB browsers, web pages and native diagnostics capabilities of the protocol in use at the site can be very beneficial as well. Wireshark can be an indispensable tool, but it is surgical in nature, offering a detailed look at a small piece of the network. If you don't know exactly where to look, it could take a long time to find the problem. And for transient problems, it might be hard to catch the event. A good network management tool that will always monitor your entire network can really expedite the process by telling you exactly what the problem is, or by pointing you in the right direction to find the root cause. Such a tool can pay for itself very quickly as the complexity of these networks far outreaches the capabilities of their maintainers."
Jonas Berge, director for applied technology at Emerson Process Management, disagrees somewhat with Stableford on the use of oscilloscopes. "Oscilloscopes might be too difficult for many to use," Berge says. "It has to be set correctly and you need to interpret the waveform. You can of course learn this if you have an instruction with samples of different good and bad waveforms. Therefore, I prefer and recommend use of dedicated testers, which are easier to use and give you pass/fail criteria."
Stableford adds that a strong set of "soft skills" will help you get the support where and when you need it. He lists those as:
- Be able to "translate geek" to suitable language that can be understood by non-technical staff and managers, without it becoming a weapon of mass sleepiness.
- Think outside the box, because gremlins certainly will.
- Maintain a sense of humor, and because this is such a fast moving industry, learn something new every day.
Jonas' Ladder
| Rung | Wired Network | Wireless Network |
| Seven | Mode - Input status "Bad" - Configuration error |
|
| Six | Device disconnected (stopped communicating) See "device not powering on" and "device not found" |
Device disconnected (stopped communicating) See "device not powering on" and "device not found" |
| Five | PV "Bad" status/alarm - Device configuration - Sensor failure - Device failure |
PV "Bad" status/alarm - Device configuration - Sensor failure - Device failure |
| Four | PV burst/publication not received - Device configuration |
PV burst/publication not received - Device configuration |
| Three | Displays in generic mode - HART protocol version - EDDL files missing - Other system files missing |
Displays in generic mode - EDDL files missing - Other system files missing |
| Two | Device not found Wrong protocol device type Device configuration Interface Network configuration (host) Connections Noise level Moisture in device compartment or field junction box Loose connections Component selection (power supply, line conditioner, input/output filter, barrier) Signal level Device failure |
Device not found - Device configuration - Network configuration (host) - Device failure |
| One | Device not powering on - Power supply failure - Safety barrier failure - Excessive loop/bus resistance - Open loop/bus - Insufficient supply voltage - Short circuit - Connections - Device failure |
Device not powering on - Battery flat - Device failure |
The Simple Way
A strong proponent of the KISS principle is Bernd Kremer, general manager, Trebing+Himstedt. "We see a huge demand for providing easy-to-access information about network reliability," Kremer states. "Even for cross-protocol-based and Ethernet TCP/IP-related protocols." Towards that end, Trebing+Himstedt identified a Formula for Success for Efficient Network Diagnostics:
- Permanent network monitoring instead of reactive troubleshooting.
- Automatic alerts in case of failures. The network notifies the user in case of problems.
- Use of intelligent devices (managed components such as switches). Experience has shown that using intelligent components brings substantial diagnostic benefits that more than make up for higher purchase costs.
- Traffic-light principle for diagnostics, i.e., users are presented information that is already processed, so they don't have to troubleshoot at the protocol, telegram or signal level.
- Permanent documentation and inventory control of networks and devices.
The good news about helping to hide the intricacies of a protocol from the average user, while still allowing them to support and maintain their networks, is that practically all industrial protocols have one or more permanent, network-monitoring tools on the market. However, the ideal network monitoring tool does not affect or alter the network. Of course, this is never 100% achievable, but we can get close.
Andrew Verwer, Verwer Training and Consultancy, agrees with the need to be non-intrusive. "To make the network analyzer useful in a real situation in a factory or plant, it is essential that the analyzer is portable and can run without a mains supply, such as your laptop battery," he explains. "I've seen the need to run without being connected to the ac network when using a USB-based oscilloscope. Though the oscilloscope gave acceptable measurements, we observed that the baseline seemed rather ‘noisy.' However, as soon as we unplugged the laptop — so that it was running on its battery — the signal cleaned right up." Just shows that the ac/dc isolation on your computer is no better quality than your battery requires to charge.
SEE ALSO: Five Things You Need-to-Know About Industrial Ethernet
Learn From example
Verwer also emphasizes the need for training with examples of how training helps identify what might not be intuitive solutions to problems related to reflections, and has a warning about the use of repeaters. "Reflections can be caused by a wide range of faults or errors that cause the signal transmitted by a device to bounce back along the cable," Verwer explains. "The reflection, just like an echo, causes multiple signals to appear on the cable resulting in corrupted telegrams. The problem is that the devices most-affected are those farthest from the fault that is causing the reflection because the delay is greatest the farther you get from the source. Devices on the other side of a repeater will show a waveform on the oscilloscope, but this is actually the signal transmitted by the repeater and not the original signal transmitted by the device."
Sally Offner, president at Levelese, says that, "Since the mid-1980s, introduction of ‘smart' field devices with their on-board diagnostics, auto-zero and ambient-temperature compensation — features that were largely driven by users' economic need to minimize scheduled maintenance — field troubleshooting frequency has been falling, but not eliminated. There remains the need to detect and repair signal wiring, ground loops and the occasional screwdriver adjustment. For those who have smart instruments, controllers and final control elements in the field, a field communicator for the specific products and networking system is a must."
Interestingly, many of the causes of network problems that Fluke Networks identifies in its literature are very similar to the ones that affect the devices themselves and, in fact, differentiate an industrial device from its peers. These factors are temperature and humidity extremes; heavy-duty, machinery-induced power disturbances, noise and electrical interferences; unpredictable installation environments, long cable runs, abuse from heavy-duty machinery, vibration; and other common factors.
Based on his many years of helping customers with networking problems, particularly for field-level networks, Berge also developed a hierarchy of troubleshooting for wired networks. It's called "Jonas' Ladder" (see table) because you start at the lowest rung, Level 1, and work your way up to the top.
Relcom's president, Maris Graube, says in almost all instances when they get a call for assistance from a customer, they find that, like Jonas' ladder and as noted at the start of the article, "There are several skill sets and tools needed to set up a fieldbus network. The basic one is to get the wiring right. The more complex one is to get the system configured correctly. One of the common problems associated with ‘getting the wiring right' is to be sure the screw terminals are tight, and the best way to do that is with a torque-limiting screwdriver."
Dave Brown, EPC senior engineer at Bechtel Corp., who has been involved in commissioning and start-up of fieldbus systems around the world, agrees and adds, "On the prior major project I worked on, we kept seeing intermittent issues on different segments in different areas of the plant. We ran a test in the water treatment plant, and found every screw on the coupler needed from one half to two turns or more before it was sufficiently tight. After all the terminations were corrected, all intermittent issues cleared and the system operated just fine."
Khalid Ansari, production information and control systems engineer at Qatar Aluminium, also feels that today's automation issues are primarily related to networking. "One has to have good troubleshooting skills and tools for first ensuring the connectivity, and then protocol-specific diagnoses," he says. "Further, though it's not strictly a protocol, OPC is frequently used to connect disparate systems, so understanding how OPC works should also be considered part of the requirements for today's well equipped network troubleshooter."
Then, There's Wireless
This leads us to the "new" industrial network — wireless. It should not come as a surprise that many of the rules and concepts that work for wired networks are true for wireless installations as well. As Prabhu Soundarrajan, global director of applications at RAE Systems indicates once again, getting it right starts with design. "We recommend a complete field survey to map and troubleshoot any new network we're installing in the field," Soundarrajan says. "We recommend fixed access points (routers) in clear zones to ensure trouble-free connectivity. And lastly, when using wireless mobile devices, we recommend a spectrum analyzer to survey the installation area to ensure real-time data and trouble-free connectivity."
When it comes to wireless, Berge adds that the key to reliability is the placement of routers. For mesh topology, strengthening is done by placing an extra wireless transmitter (router) to form a redundant path. "A star topology would have to be done by wiring up an extra backbone router in the plant, or by going for mesh topology," Soundarrajan says.
Not yet consistently addressed with wireless is how to non-intrusively monitor a wireless network. Dick Caro, CMC Associates, says, "The ISA100.8 User Working Group identified that networked instruments usually need a hardware configuration/troubleshooting tool. Unfortunately, the problem is that a different tool will be required if they install a network not supported by the communicator already strapped to their field tech's repair belt. They envision the poor technicians with a dozen communicators strapped to their belts. Not a good picture."
Caro adds, "They want to reduce the configuration and maintenance tool to exactly one — a smartphone with near field communications (NFC), which requires the device to be within millimeters of the actual instrument, but not necessarily touching it or requiring opening of the instrument enclosure. Most new smartphones already are equipped with NFC, so the potential exists for configuration and maintenance functions available as smartphone apps, and conveniently not using the same frequencies as the device itself for communications."
Even though it appears today's automation systems are becoming more complex, the good news is that tools to support them are keeping pace. The basics of design, installation and training of people with the right attitude are excellent precursors to success. Fortunately, young people entering the workforce are comfortable with bits-and-bytes technology and its continuing evolution, and the excitement of seeing these technologies in the manufacturing sector could be one of the lures to secure the next generation of automation and control professionals.
