Monitor and Mend Network Health

ACC investigated and found a 7 s, heartbeat-timer timeout was causing the shutdowns. This reset signal for the timer came from a 1.25 s, square-wave pulse from another PLC when it was cued, and so ACC's engineers decided to check the communication network for the cue using FTS4Control from Frontline Test Equipment. The pulse and timeout were supposed to occur only when a task failed to execute, but ACC's engineers used FTS4Control's filter and frame display to view the network's data at varying granularity, and learned that increased load on the network created a longer token rotation, less samples taken off the pulse output, and a mismatch between the square-wave and network data transfer timing. In short, the pulses were generated normally, but the sampling was infrequent enough to increase the possibility that only low-pulse samplings were transmitted during a given 7 s period. So ACC introduced a PLC heartbeat code with an incrementing number, which removed the dependency on asynchronous timing, and allowed the network and automation system to perform smoothly.

Likewise, thanks to high gold prices, Newmont Mining in Denver says its Asia-Pacific operations are growing quickly, and especially fast in Indonesia, which means the networks supporting its 1,500 production applications, equipment and servers and 5,000 employees and contractors in Sumbawa and Jakarta have high-bandwidth problems. They needed better real-time monitoring, reporting and troubleshooting of multiple sites and applications, so Newmont's IT technicians tested and implemented Orion Network Performance Monitor (NPM), Application Performance Monitor and Engineer's Toolset from SolarWinds.

"Our team needed to adopt a solution that could monitor status of our vendor devices, routers and switches," says Samuel Tirayoh, Newmont's IT manager for Indonesia. "Previous solutions limited us to monitoring support on just a small range of products. Adopting NPM helped resolve our high-bandwidth utilization issues, so now we can track and locate devices across locations, which saves time on solving problems and saves money on replacing broken devices."

Despite the healthful benefits of collaborating with IT, Shawn Gold, marketing manager for the Industrial IT Solutions Service at Honeywell Process Solutions, reports there's still a big skills gap around networking in process control settings. "Many control engineers still think of networking as a black box they can ignore, set and forget, or assume IT will take care of, even if the controls side hasn't allowed IT access to it," he explains. "Some process control users are proactive and work closely with their IT departments, but others don't let IT go into or below a certain demilitarized zone (DMZ) on their network, so IT ignores that area and controls might not know what to do to maintain it."

Gold adds it's more productive for engineers to coordinate with IT, jointly check loading on managed Ethernet switches, upgrade those that need it, and set parameters, thresholds and alerts — just as they do with DCSs. "We've used Wireshark and SolarWinds software to troubleshoot network problems, but we found they can have an adverse effect on DCSs, and so we built our Service Node NMS from third-party tools," Gold says. "Service Node works with our DCS without causing loading problems, and it provides a secure remote connection for network troubleshooting, log analyzing and patching, and next year we're going to add a cybersecurity dashboard."