Monitor and Mend Network Health

Just as fruits, vegetables, whole grains, lean protein, regular exercise and routine physical checkups can keep your heart and cardiovascular system up and running, there are many ways to maintain the health of your industrial networks.

Unfortunately, just as with our physical bodies, a lack of awareness and initiative means many solutions for network wellness aren't used as often as they should be — and so they're overwhelming when real problems show up. Likewise, many control and automation engineers are so focused on the uptime and performance of their process applications and production lines that they don't pay enough attention to the switches, wires and other components bringing them the critical data for those processes.

"We recently helped with a proprietary Ethernet ring configuration that includes two distributed control systems (DCSs), three PLCs and seven or eight SCADA nodes running the steam plant at the University of Virginia at Charlottesville," says Reid Garst, vice president at Sterling Engineering Solutions, an applications engineering and distribution firm in Salem, Va. "This network already had Ethernet switches for about 10 years, but more PLCs and other equipment from multiple vendors were added since then, and these devices were prone to flooding the network with multicast traffic. Unfortunately, multicast has sort of a shotgun approach that tries to put 10 lb of data in a 5 lb bag. However, because the existing Ethernet switches were not fully managed, they couldn't handle all the traffic, and the network was overwhelmed and started generating a lot of communication errors."

Consequently, Sterling used managed Ethernet switches from Moxa that could filter multicast communications to help diagnose the steam plant's network, and found that about 80% of the network had been taken over by multicast. "As a result, we preconfigured and put in four of Moxa's EDS 518A managed Ethernet switches in a simple ring topology, and they filtered the multicast communications, decreased traffic back to an acceptable 20%, and reduced the errors and alarms."

Jim Toepper, product marketing manager for Moxa's industrial Ethernet infrastructure division, adds, "Our efforts on network monitoring, maintenance and health are sort of following on what IT was doing about 10 years ago with HP OpenView and SolarWinds software. Now, managed Ethernet switches have become ubiquitous in control and automation, our networks are much bigger, and they need to be monitored regularly and sometimes controlled with network management software (NMS) tweaked for our industries, such as our MXView, Hirschmann's HiVision or Network Vision's IntraVue, which can tell users at a glance if bandwidth utilization is over 60%, and if they should increase network capacity by adding more Layer 3 switches or establishing another virtual local area network (VLAN) segment."

Let's Have a Look 
Likewise, many other tools and methods for network health and optimization are getting easier to use, and more closely integrated with traditional controls equipment and software. For instance, aluminum smelter Qatalum is a joint venture between Qatar Petroleum and Hydro Aluminum of Norway, and its two 1.2 km pot lines, carbon plant, cast house, 1,350 MW power plant, port and storage at its plant in Mesaieed, Qatar, can produce 585,000 tons of aluminum extrusion ingots, foundry alloys and other products per year (Figure 1). Naturally, these applications include thousands of sensors and other components reporting in via hundreds of Ethernet switches and a variety of networks, and so Qatalum recently decided to coordinate all this traffic with Industrial HiVision NMS, including MultiConfig tool for multiple, simultaneous device configuration, from Belden's Hirschmann Automation and Controls division.

"The key elements in our network demand fast, redundant switching, data communications and monitoring, and controls based on fast Ethernet," says Graham Patton, Qatalum's senior network engineer. "Gigabit Ethernet was required in the backbone ring and in the hot-swappable media modules with high port density. These devices were also required to have high predictive hardware lifetimes and lengthy mean time between failures (MTBF). The structure of our network also needed to ensure that no single point of failure can interrupt plant communications, and so all nodes are monitored from a centralized HiVision NMS server. NMS simplifies our daily activities, and makes  resources available for other networking tasks." Other benefits that Qatalum gained from NMS include maximized uptime from its redundant Ethernet ring topology, improved network availability from frequent status updates and threshold-setting functions, auto-topology and discovery functions that show what's connected where in the network, and more efficient plant operations from fault-prediction detection and diagnoses.

"Network management has been a bit of a black hole for many control engineers because they often don't really know where the network is or what's in it," says Mark Cooksley, Hirschmann's product manager for software tools. "Network health requires going back to the beginning, checking the original network design, and making sure it follows recognized design characteristics. You need to ask, 'What's on our network, what applications are using it, what availability do they require, and what do we need to add?' Next, you need to correct redundant links, deploy managed switches where appropriate, and implement NMS to handle switch data. This can include bandwidth utilization, device temperature, power status, link states and errors, and then you need to set thresholds for each. For example, you might want an alert when bandwidth goes above 30%, so you can do some preemptive problem solving. You also need to observe your network over time, and establish baselines for normal performance, which will help identify anomalies when they show up."