How to troubleshoot OT networks

Troubleshooting industrial Ethernet networks can be a daunting task. Most often you or we do not have the cable routing diagrams and the switches/routers that devices are connected to. Rarely does the OT/plant floor Ethernet network have its own segmentation, which is the way it should be.

When I created the maintenance network at a 1.7 million sq ft distribution center, we had our own switch system creating our own network and cabling layouts. However, we needed to have some of the devices on the IT network, as well, which was handled by connecting those devices to the IT switches and routers, of which we had no knowledge. VLANs were set up by the IT group to allow communication between the two networks.

When a problem occurred, however, it was important to be able to ascertain where the possible issue was and within which network.

On the OT network, we had 14 PCs running an HMI application which was configured to be able to reach the 28 mobile AS/RS cranes in the building. The 28 PLCs located on the AS/RS cranes talked to the OT and IT networks wirelessly. There were three main PLCs on the IT network that controlled the movement of product throughout the building. There were multiple Ethernet-enabled PLCs to transition product between buildings, and we had to track it all on a server to be sure the product ended up at the right distribution point.

If we lost the pallet data at any point, product would end up at a place that would take an intervention to get it back on track or a manual extraction. Not desirable.

If we lost one piece of product data, the error could cascade throughout the system and cause all other product data to be incorrect resulting in pallets showing up where they weren’t supposed to.

So, the integrity of the network was paramount. It is a pain point with multiple points of failure. So, what happens when it fails?

We had cabling issues due to self-made cable ends, which on occasion came loose. Switch node failure, Ethernet card port jitter (noise), power supply issues and EMF issues caused an intermittent problem. You hope it is a fixed problem, such as a blown fuse. It was a closed system and not connected to the internet.

The server needs to communicate with a PLC more than 600 ft away, and where does the cable go? We are missing product data. Network timing was important, so the messaging had to get to Point B in a certain time frame. The custom-designed messaging protocol created multiple messages to move product around the buildings.

There was a UNIX server upwards of a mile away that read and wrote data to move product, just to further complicate to issue. In all fairness, it was pretty reliable.

Get your subscription to Control Design’s daily newsletter.

When it wasn’t, we had pain points.

Troubleshooting tools now come into play. Can the server see the PLC? This is where we have to know IP addresses, which dictates that certain devices on the OT network have static IP addresses. An Excel spreadsheet should document this.

The first line of defense is the Ping ICMP command. Ping followed by the IP address of the target will respond with an error or a reply with a timestamp. If any packets get lost, we may have a noise issue. If there is no response, we have a problem somewhere on the path from the source to the target. The PLC may in fact be off-line. Also, check Ethernet port activity lights for data transfer.

If the device is present on the network, but communication is spotty, then a PathPing command may be in order. It will trace the pathway from your network location to the target device with a response to indicate if any node/device on the path is losing packets. The node where this is happening needs to be identified and checked. This can be a replacement for the ICMP tracert command.

If you are working from a laptop, you may have to be on the same subnet as the devices you are trying to access if the address is a non-routable address. If the PLC IP address is 192.168.10.100, your laptop has to be on the same Class C address, which means it must have an IP address of 192.168.10.x. Some laptops and computers get their IP addresses from a dynamic host configuration protocol (DHCP) server, which may give you an IP in a different address range. The ipconfig/all command will give you your IP address. If it doesn’t match, then you have to reset your IP using the network configuration tool.

Network latency causes many issues in industrial networks. Having a network diagram with an IP listing is paramount when you have to figure things out. If you don’t have one, create one or use a network configuration tool. You will thank me later.