Keep Your Network Healthy

By Jim Montague, Executive Editor

How healthy is your network? No, I don't mean is your refining process filling its tanks, or are your machines spitting out products? What I want to know this time is how healthy are the managed Ethernet switches and gateways moving all of the critical operating information?

This is a crucial difference. It's important because many network devices can pass along important production values and alerts, but users sometimes forget to check on the switches themselves.

Jim Montague is the executive editor for Control, Control Design and Industrial Networking. Email him at [email protected] or check out his Google+ profile. Some switches can pass along lots of process data, but they can't deliver information on their own status and well-being. Unfortunately, if a network switch goes down or is overwhelmed, then vital process data won't get through—just as if the application had unexpectedly shut down.

Evaluating a network once meant plugging in a handheld device and checking voltages and analog signal levels, but these tools evolved into network sniffing capabilities and network software that uses the simple network management protocol (SNMP) and other protocols to check pre-defined parameters and thresholds on switches and network segments, collect and view these indicators in on-screen displays and manage their networks from more central or remote locations. So while you can never take your eyes off essential process values, you must also find a way to check on the switches that support them.

Of course, many network management tools originated on the information technology (IT) side, but now they're employed and adapted for the plant floor. This is necessary because many manufacturers are realizing they have hundreds if not thousands of Ethernet switches, signal converters and other devices to monitor. Fortunately, there are lots of software tools for examining and managing industrial Ethernet networks. As always, you just have to choose and use the right one.

Orange County Overhaul 

Orange County Utilities (www.ocfl.net) in Orlando, Florida., pumps 35-40 million gallons per day (mgd) of drinking water to more than 140,000 users in the county's mostly unincorporated areas. The utility operates 15 water production sites, re-pumping stations, elevated tanks, several 600-hp pumps and other facilities (Figure 1). These sites are coordinated by a central control facility, which is usually manned by just one or two operators.

Figure 1: Orange County Utilities uses managed Ethernet switches and network software to help manage 15 water production facilities that provide 35-40 million gal/day of drinking water in unincorporated Orlando, Florida, and find and troubleshoot slower serial-based components much faster. Orange County Utilities

"We monitor pressure, fluoride, residual chlorine, pump status and many other values, and test for lead, copper, bacteria and other contaminants," says Robert Doyon, SCADA supervisor at Orange County Utilities. The operators monitor water production and seek  out bad equipment trends by keeping manual and computer logs and also using HMI/SCADA software from GE Intelligent Platforms.

Besides being supplied almost 100% by groundwater, Doyon reports that Orange County Utilities also is almost 100% Ethernet in its well houses, processing areas, generators and other areas. The utility began building its Ethernet network about five years ago and installed a fiberoptic backbone, consisting of 12 pairs of cables, which always has four loops active and serves about 7,000 I/O tags and three databases at the main control facility. Where it used to have PLCs linked by Modbus Plus, it now uses Schneider Electric's PLCs and GarrettCom's managed Ethernet switches, and all of these devices run on Modbus TCP/IP Ethernet protocol.

"Once this whole network was out there and the PLCs were talking to each other, we needed to identify process areas where the equipment wasn't talking," says Donyon. "We wanted software that could look at the whole network. Also, when we converted from Modbus Plus to Modbus TCP/IP, we had some bandwidth problems. As a result, with all the processes we have going on, if they all tried talking at once, we'd just get more traffic from all the repeats and retries, and this would really use up our bandwidth." The utility had slowdowns in PCs polling PLCs for data in each process area, including its wells, chlorine and fluoride feeds and CO2 injection station for reducing pH. "Some of the older modules on the serial side were slow because they ran at 1.5 Mbps on Modbus Plus," explains Donyon. "Now we're scanning at 100 Mbps. Whenever you go from one communication system to another, you're going to have issues like jitter, lost data packets and some failed communications. But we still need to know what's going on at our wells, so we know how much chlorine to feed and have the right concentration at the furthest point in the system."

Simply Seeing Better

To handle these types of challenges, Doyon implemented Network Vision Software's (www.intravue.net) IntraVue network diagnostic and troubleshooting software. "IntraVue lets us look at the bandwidth that individual devices suck up, and then we can put in threshold settings and alarms for when they use too much," says Doyon. "IntraVue also maps the whole network, which we never had before, and then helps find any Internet protocol (IP) devices on it. We can see the network, each switch and whatever is hanging off of that switch. If there's a bandwidth or loop problem at a particular device, then it will be highlighted on-screen with a yellow or red circle. This saves a huge amount of time in finding and troubleshooting offending devices."

In the past, if there was a duplicate IP address on their network, the utility's staff would have to ping and go to many devices in a suspect process area, and this could take hours or days. "Now, that device and its location show up immediately, even though it takes 5-7 minutes to build the initial network scan," says Doyon. "We can now poll all the PLCs at every plant in the county at 1-sec intervals for their flow and pressure data."

Mark Fondl, president of Network Vision Software, says use of network software has swelled because users can no longer plug into a network at any point. "Before Ethernet, all communications were shared on the same physical wire, either serial or daisy-chained connections, and so diagnostics were easy because you could plug in and listen," he explains. "However, there were a lot of collisions, especially when many devices were on that one wire. Ethernet switches isolate network wires and communications into sections, which avoids collisions and increases availability and reliability, but they add a lot of complexity. Because the switches are now part of the network's internal infrastructure, it's also harder to diagnose problems like communication faults. So, network software is needed to monitor all sections of the network, ask components for their locations and paths and find devices that are confused, sending messages to the wrong places, have the wrong IP or MAC addresses or are oscillating because of these problems."

SNMP and Friends

So, who or what is the big shot in network management software and methods? Well, that would be SNMP, which basically monitors devices attached to networks to find those in need of attention, and so is supported by most devices with an Ethernet port. SNMP is a 22-year-old part of the Internet protocol suite (IPS) defined by the Internet Engineering Task Force and is likely the most widely used and imitated method of monitoring and managing the health of network switches and related components. SNMP works with virtual databases called management information bases (MIBs) to manage participants in a network. Also, SNMP operates in the application layer of IPS, which is Layer 7 of the open systems interconnection (OSI) model—the layer cake already well-known to many fieldbus and Ethernet users on the plant floor. More recently, SNMP's second version let users create names and passwords, and its third version helps encrypt data for added security.

"If you're going to be adding hardware all day, such as switches, media converters, serial-to-Ethernet devices, wireless components and embedded PCs, then you're going to need some way to view it all, such as seeing switches on our MXView software," says Jim Toepper, marketing manager for industrial networking and video solutions for Moxa (www.moxa.com). "And whenever you're implementing network management software, it's going to need the ability to speak SNMP and the other protocols."

Dan Schaffer, specialist for Phoenix Contact's (www.phoenixcon.com) networks and security products, explains, "SNMP is so powerful because it's open, non-proprietary and scalable, and it can be encrypted. You can set it up to collect data every 10 minutes, see what devices are being used, check traffic over a port, find errors, check broadcast or multi-cast communications and other functions. Its MIBs can push or pull configuration values, turn redundancy functions on or off, enable or disable spanning tree capabilities or multi-cast filters or even do a soft reboot. Network software on the business side has more of IT's bells and whistles, while plant-floor network software like our Factory Manager is more concerned with connecting to machines. But they're coming together now because they both want alerts, trending and to know what their network traffic is doing." 

Find Faults, Add a viewpoint

Once users begin to employ network software to get an overall picture of their layout and individual devices, they find other uses for these capabilities. For example, Huhtamaki (www2.huhtamaki.com) in Lurgan, Northern Ireland, uses an automated, paper-forming process to make molded-fiber egg packaging and cup carriers. It recently adopted IntraVue to monitor the Ethernet network transmitting overall equipment effectiveness (OEE) data it uses to ensure productivity, efficiency and constant renewal (Figure 2). This information is captured from its packaging-production PLCs by Wonderware historian software and analyzed on template spreadsheets.

Figure 2: Molded-fiber packaging manufacturer, Huhtamaki, deployed network software to manage its upgraded network and then found some older equipment that was hampering the network's performance with old 10-Mbps connections.
Network Vision Software

Similar to other networking tales, Huhtamaki's challenge began when it tried to alleviate problems with missing data, sluggish performance and too many nodes by overhauling its 48-node network with a gigabit-Ethernet backbone, managed switches and network monitoring and management tools. The upgrade lessened problems, but Huhtamaki's users found they needed more network insight to find lost data. Consequently, they installed IntraVue with help from HardwarePT; its threshold-throughput display and dynamic-connection, color-change display quickly showed that some older equipment wasn't as reliable as believed and was strangling network performance with old 10-Mbps connections. With IntraVue's direct and measured evidence, Huhtamaki's engineers were quickly able to justify an upgrade of their controllers.

"It would be impossible to keep track of our network's development and its real-time performance without IntraVue," says Andy McKinney, Huhtamaki's process maintenance engineer. "We also customized the dynamic interface, so each node has a thumbnail image of the actual equipment connected, which enables fast recognition of problems when alarms are generated. We also know that IntraVue's inherent security allows us to immediately detect new connections and disable them if they're not authorized."

In addition, once these initial gains were made, McKinney and his colleagues at Huhtamaki decided to use IntraVue's interface with IP CCTV cameras, which are set up to monitor robotic and other unmanned processes. These can be scanned and viewed via IntraVue to spot any improper operations without having to physically visit them or remember IP addresses. IntraVue can access any device that serves its own Web pages, such as printers, PLCs and inverter drives to monitor status and prioritize maintenance.

Newer Tools

Though they're still mostly circling the plant floor and not really staging a full invasion yet, several of the newest network software tools are definitely making some inroads into industrial applications.

For instance, after an outside contractor neglected to deliver alerts on production network failures, Solvay Pharmaceuticals (www.solvaypharmaceuticals-us.com), in Marietta, Georgia., decided to do the job itself with help from IpSwitch's WhatsUp Gold software with SNMP and Windows management instrumentation (WMI) monitoring to make sure production environments at each manufacturing site are constantly monitored for device, network and temperature issues. 

"Visibility into the network is big because I like to see the heartbeat of what's going on," says Bruce McMillan, Solvay's emerging technologies manager. "Previously, we were only getting quarterly reports, and we couldn't pull them actively. WhatsUp Gold let me set up by company, sites and buildings. We monitor routers, switches and servers, and when there are no dependencies, we still get a page from every device at that site. Now, with dependencies set up, an alarm announcing a lost switch leaves everything connected to other switches alone. Also, if I lose a site, I don't get paged from every device. I get only the top-level device—the default gateway."

In addition, McMillan adds that WhatsUp Gold allows Solvay's engineers to customize network monitoring functions using standard scripting syntax software, such as Visual Basic Script or Java Script, to create an Active Script, many others of which are listed on IpSwitch's website. "We now have scripts in place that help us get a feel for when problems might occur," he says. "We're in a much more proactive mode now than we were before." Also, a recent move to WhatsUp Gold's Web-based version allows McMillan and his colleagues to monitor their applications and sites from anywhere with an Internet connection.

Finally, WhatsUp Gold even helped fill in when another environmental monitor went down. "We have environmental monitoring by the facilities group, but there was a recent problem with their reporting mechanism," he explains. "I'd installed an SNMP-compliant probe, allowing me to chart historical temperature and humidity patterns. Not only does it show me when the temperature is out of compliance, but I can create a chart that reports the history."

Stay on Top

Following its gains so far, Orange County Utilities is using its networking experience to build the control facility for an additional new 30 million gal/day water production plant that likely will have another 7,000 I/O tags, no doubt managed by Ethernet switches appearing on IntraVue. It will back up the utility's existing plant and have a large ozone-based water treatment system, too.

"We learned that, when you go to an IP-based network and use Ethernet, you have to keep on top of it," says Doyon. "You have to think about what's talking to whom and how fast you need your data to go. Software tools like IntraVue can help a lot because they're automated, but this also all depends on the switches being set up right."

Doyon adds that he's intrigued by open-source freeware, such as some Wireshark and SolarWinds solutions, and he even uses some of them in his applications.

However, Donyon cautions that they require extra work to install and operate. "We use every kind of network management tool, but if we use SolarWinds' protocol analyzer, then we have to go in and set up the analyzer ourselves, and we also have to set up a way to receive and interpret the data," he explains.