En Route to a Safe Network

Ian Verhappen is an ISA Fellow, Certified Automation Professional, and a recognized authority on industrial communications technologies with 25+ years experience predominantly in the process control field.Corporate-wide enterprise networks generally are protected and monitored for new and evolving threats. Ensuring security of real-time process control systems comes with another set of challenges, and linking business applications with data generated from real-time systems creates a greater need for real-time security.

Not only must modern control systems integrate with today's business systems, in many cases, they also have to share much of the same technology: Windows-based HMIs, Ethernet-based backbones, RFID-tagged components and handheld PDA maintenance tools.

Standards-based communications protocols have accompanied the growth of these trends. Although these protocols are relatively new to real-time environments, they are widely accepted in the corporate IT environment.

It also is widely accepted that much of the equipment used in the corporate environment is not robust or fast enough for real-time control. With all the advances in technology and processing speed, is this still true?

This also begs the question: How real must real time be? Our criteria for real time is that information be delivered and available where it is required, when it is required. A typical continuous process such as a refinery may have updates of seconds or even minutes, while many manufacturing processes such as filling operations require scan rates in milliseconds.

So how real is real time once the communications protocols are introduced into the real-time process environment? The first three layers of the OSI model, physical, data and network layers, require the most attention since they control the actual transmission of data/signals.

Hardware and Layers
Switches and routers are critical elements to the network infrastructure. These connectivity elements coupled with a solid design will deliver security, reliability and quality of service. Switches represent Layer 2 of the OSI whereas routers represent Layer 3. There also are Layer 3 switches that are defined as multilayer/routing switches.

Figure 1: In the lowest two layers of the OSI model, applied to many commonly used networking protocols, Ethernet, and therefore "industrial Ethernet" is defined.Figure 1 indicates how the lowest two layers of the OSI model are applied to many commonly used networking protocols. Note that Layers 1 and 2 define Ethernet and therefore "industrial ethernet" is defined only by these layers. Hence, despite the fact that these protocols use Ethernet as the basis for their transmission, the protocols themselves all are different and incompatible.

The most basic Layer 1 devices are network hubs or repeaters, which were predecessors to local area network (LAN) switches. Hubs operate at Layer 1 of the OSI model, physically connecting nodes where network traffic is broadcast through the entire flat network hierarchy. In a small workgroup environment having a hub traditionally was acceptable. However, most network environments are growing, dynamic, and based on the types of data traffic. This increasing level and complexity of data traffic is what drove the requirement for switches.

Switches are a key LAN component because the devices enable multiple users to directly communicate with each other. Relative to routers, switches improve throughput for users by virtually defining users within logical workgroups. Switches recognize only other locally connected nodes/devices. Unlike basic network hubs, data broadcasts are restricted to the defined domains or user groups. For hubs, data that arrives at one port is retransmitted or repeated to all the other ports, and consequently devices connected to the hub with a resultant risk of high data traffic, but little information transfer. As a result of Layer 2 protocols, the switch reviews and learns media access control (MAC) layer addressing from the header of every packet. The switch then compares the MAC address against its look-up table to confirm that the destination is connected to one of its ports and then forwards the packet to that port only. If there are no matches, the packet is discarded, reducing the bandwidth requirements on all ports. This translates into faster throughput because there is little overhead to review between source and destination.

LAN Formations
The collection of remote devices (data sources/sinks), network devices (routers, switches, hubs) and data devices (historians, PLCs, servers, HMIs) combine to form a LAN. Since each device on a digital network theoretically can communicate with every other device, the number of potential connections can grow exponentially and become difficult to manage even with switches to help manage the load. Figure 2 shows how each layer in the OSI model requires the addition of information and overhead to a message.

Figure 2: Each layer in the OSI model requires additional information and overhead to a message.Virtual LANs (VLANs)commonly are created to organize logical workgroups or workgroup domains independent of physical locations within a defined network. Participation within a VLAN is defined by policies or business requirements. VLANs communicating with other VLAN groups must do so through a router. Common VLAN nodes hear traffic from other VLAN members on the same VLAN. To a degree, this serves security since certain user groups might have confidential data that must be separated from other workgroups. Subdividing Layer 2 networks through broadcast domains will reduce segment use and increase throughput.

Multilayer or routing switches are referred to as Layer 3 switching. Practically speaking, a Layer 3 switch is a high-speed router having only one network protocol, primarily an IP. With newer backplane interconnect technology developments, wire speed capabilities now exist with Layer 3 technology. Market availability includes 10, 100 and 1,000-Mbps interfaces. This throughput is enabled further because it does not carry the overhead of supporting multiprotocol functionality. A Layer 3 switch routes IP traffic and switches Layer 2 traffic via MAC addressing.

Routers are very important as part of a network fabric when there are multiple network protocols and or multiple network locations. Routers select the optimal route between networks. Routers also support multiple protocols or connect different types of network interfaces. This is enabled by the design of routers, which take a packet, open it, review the algorithms, and then determine the best route for the destination address based on pre-defined parameters set in the routing tables by the network administrator. This may represent lowest-cost routes, least-hop counts, etc.

Choose Your Weapon Key network elements include switches and routers. The evaluation factors in serving time-sensitive protocols require a number of considerations when determining whether to use switches or routers. When planning a network design, identify the real-time protocol's latency parameters and match these to the Layer 2 and 3 parameters. The key metrics, including reliability, routing delay and latency should be considered. Reliability, in the context of routing algorithms, refers to bit-error rate on each network link. There are some network links that go down more often than others. If a network fails, certain network links might be remedied more quickly than other links. Any reliability factor can be taken into account with the design considerations of the reliability ratings. Routing delay refers to the length of time required to move a packet from source to destination through the internetwork. Delay depends on many factors including the bandwidth of intermediate network links, the port queues at each router along the way, network congestion on all intermediate network links, and the physical distance to be traveled. Because delay is a conglomeration of several important variables, it is a common and useful metric.

Routers also can interface a variety of different network-based protocols. Essentially routers represent overhead when considering throughput, or time-sensitive protocols. Routers offer additional security features such as VPN support, intrusion detection or access control lists. Some vendors also position integrated firewall features in their routers.

Integrated firewall functions include packet and session inspection, intrusion detection and authentication. The primary value for integrated firewall solutions is for remote offices or departmental deployments where switches or routers are used. Of course, each of these "transactions" adds a little more time to the network communications overhead.

MPLS Listings
Another consideration is the development of Multi-Protocol Label Switching (MPLS), a newer standards-based development led by the Internet Engineering Task Force (IETF). This protocol will help prioritize alerts to switches or routers to direct traffic. In other words, the most important data traffic receives special delivery service. As stated, switches are simpler and faster as a means to send data to a destination. Routers have intelligence that determines how and where to send multi-protocol data. MPLS can tell a network what the data type is, with the ability to prioritize data with quality of service. If certain network traffic is sensitive to latency delays, MPLS can help by assigning each message or packet a priority rating.

Multi-protocol label switching was designed for mesh-based networks providing a secure Virtual Private Network (VPN) connection at either Layer 2 or 3, and to allow networks to operate more efficiently over a variety of access methods. Traffic engineering is a key feature associated with MPLS-based VPNs. Based on this architecture, MPLS is designed for larger enterprise systems where a customer might use a common network with a carrier.

However, this technology is young and MPLS connectivity might not be available at all sites. The potential and/or desire to use MPLS should be identified early in the planning phase of an infrastructure project so appropriate equipment and service providers can be included in the design and procurement process.

Delicate Balance
The bottom line is that your network infrastructure should provide quality service in a secure, efficient manner to ensure continuous and seamless communications operations.

Network design is like most other engineering activities,it's a balancing act. In this case, the need for speed and associated real-time control has to be balanced with the need to provide a secure environment, safe from internal faults that could propagate and cripple a system, and external threats such as hackers, worms, trojans and other malicious code from the corporate environment now shared with the real-time world.

In many cases, a critical first step is to understand the present situation, so a well-engineered long-term, secure solution can result. Process control security consultants can help set that benchmark and identify a path forward, while incorporating the latest standards and infrastructure to provide an overall cost-efficient solution uniquely tailored to the individual facility.