How (not) to troubleshoot industrial Ethernet failures
Industrial Ethernet is the same as commercial Ethernet, except for the hardware, which has been hardened for the factory floor. TCP/IP is widely used, along with specific protocols such as Ethernet/IP and Modbus/TCP.
It has become a de facto standard in plants and machines for interconnecting systems and in fact individual devices. So, what happens when it fails? What would make it fail?
I read a post on LinkedIn that went something like this:
The plant warehouse management system (WMS) stopped communicating with the programmable logic controller (PLC) that had all the connected sensors attached to it. The first step in troubleshooting this issue was to check the PLC code, wrote the author.
How many times has the PLC code been responsible for system failures? It worked five minutes ago, so one wonders. But I digress.
Evidently the PLC code checked out. Next up, the sensors. Check to make sure they are being scanned and reacting properly. It’s the WMS that isn’t communicating, which no sensor has had any effect, everyone commented.
The next frame on the graphic shows the Ethernet connection, and funnily enough it shows a failed diagnostic light. The resulting reboot of the WMS fixed the problem. What caused it to go offline, one may ask.
It probably is a Windows-based computer running the WMS, which leads us to a probable option of the PLC driver going off-line. The diagnostic failure light on the switch, router or PLC indicates that there was a network failure of some sort. It wasn’t clear as to where it was.
So, a review of the systemic information should have taken minutes and not the 13 hours the author suggests it took to find and fix the issue.
What if there was no failure light? How do we troubleshoot network issues, and what tools do we have to help?
There may be helpful displays on the PLC itself, or maybe not. A network mapping tool would be helpful which would show the user what devices are successfully connected to the network and in the same subnet. The IP address range for areas of the plant and/or machine typically will be the same and typically connected to the same group of switches.
The mapping tool will tell you if a node is off-line and not responding. If the link is alive and well, then the issue would typically be a software or message-framing issue.
Step one is to confirm cabling on a network computer with the ping command using the Internet control message protocol (ICMP). This tests the physical cable, as well as the integrity of both ends of the message route. A positive ping response suggests that the link is alive and responsive at both ends.
Get your subscription to Control Design’s daily newsletter.
If the ping command fails, it suggests that one of the nodes is disconnected from the network. Most if not all Ethernet ports have two indicating lights—one that blinks with activity and a second LED that indicates the link connectivity and the speed at which it is connected. Note that, for 10 MBPS, that light will be off, so it’s important to know the configuration.
Ethernet modules check for situations such as jitter, which is, in one word, inconsistent. This can be due to many things such as network congestion, which can happen when an Ethernet module or card goes rogue. This typically suggests that a piece of hardware is intermittently failing.
This is not an easy find however and may require a deep dive into the network.
There have been times when a port on a switch drops the connection, and a power cycle was required to bring it back online. With a Windows computer and application, a restart is successful some of the time.
In the above case, that is what restarted the communication between the WMS and the PLC, and the user had the benefit of a diagnostic LED to aid in the troubleshooting.
In the event of not having that periscope into the network, tools such as protocol monitoring software may be needed to find a rogue node.
If the systems are connected to a managed switch, there are various interfaces that a user can use to monitor each node and view the configuration.
There are various ways one can troubleshoot a communication issue within a subnet, and activity lights and link connectivity lights would be the first step. Specific hardware device status lights may be present.
In the above example, is the computer a commercial off-the-shelf computer with a standard Ethernet module, or is it an industrial PC with a hardened network module?
So much needs to be known about the network configuration to be successful. Map it out, now that you have the time. It will pay dividends.
