Jim Montague is the executive editor for Control. Email him at [email protected].
Industrial network security is one of these endless and often thankless chores. You must assess applications and facilities, turn on passwords, and set up firewalls. However, you can't assume you're now safe to frolic behind your many layers of impregnable barriers. Although preliminary security is essential, there are increasing examples of probes, intrusions and hacks, such as Stuxnet in 2010, that do an end-run on most security devices and software, make it appear that nothing's wrong, and cause potentially huge amounts of damage. Yes, Stuxnet was a narrowly targeted attack, but most experts maintain that other viruses and malware that use similar methods almost certainly are coming soon, if they haven't arrived already.
Routine Creates Consistency
The good news is that useful tools to protect and secure industrial networks are multiplying in variety and sophistication. These include more-capable Ethernet switches, better encryption, precisely targeted data transmission and reception, more-thorough network monitoring and data packet inspection, quicker identification of unusual traffic, and faster responses to probes and hacks (Figure 1). However, it remains that two of the most important tricks are to get staff trained and committed to help with network security, and to routinely and consistently update security tools, policies and capabilities.
"It's enlightening when we get to see what's really happening in our network," says Charles Harper, director of National Supply and Pipeline Operations at Air Liquide Large Industries U.S. "Gaining visibility into this world of previously undetected cyber-threats helped reassure our team that we were doing the right thing by adding intrusion prevention technology across our industrial network."
Mark Heard, control system cybersecurity lead at Eastman Chemical, says his company views process security as a routine business activity. "Cybersecurity must be taken as just another task that needs to be done to be a grownup and stay in business," says Heard, who reported on Eastman's efforts at the recent Invensys OpsManage meeting in Nashville. "Cybersecurity is a necessary layer in overall plant security — and safety. In fact, much traditional safety thinking can and should be applied directly to cybersecurity, too. As a result, safety and security are simply good business. This is because undesirable incidents of any sort detract from the value of a business. Safety and security incidents have negative impacts on all stakeholders, including employees, shareholders, customers and the communities in which each plant operates. No one wants to have downtime and deal with cleanup, regardless of whether it was caused by a design problem or a security issue."
The key to cybersecurity is collaboration, adds Ernie Rakaczky, program director for control system cybersecurity at Invensys Operations Management. "It all comes down to controls, IT and everyone else being responsible for mitigating security problems and balancing the risks in individual processes," he argues. "As a result, we're focusing on vulnerability mitigation because there are a lot of opinions on how to do cybersecurity, and some researchers are launching vulnerability programs that are irresponsible and unprofessional. Some are even blindly posting controls information to the outside world."
Protection Part of Every Day
A useful way to make network security more consistent and effective is to shift away from thinking of network security as some exotic add-on to the regular network, and begin to accept security as a truly integral part of that network and how it's applied to its individual application and facility. So, while it's still crucial to secure industrial networks by dividing them into segments separated by firewalls, it's also essential to monitor what happens next and be alert for unusual behavior.
For instance, to maintain its water production, distribution and treatment facilities, and minimize and shorten any plant disruptions, Espoon Vesi water treatment plant in Espoo, Finland, recently implemented network maintenance and remote services. Besides treating about 100,000 m3 of wastewater per day, Espoon Vesi pumps about 70,000 m3 of fresh water per day from the nearby Damman water treatment plant and from Helsinki to its system (Figure 2).
This project updated the plant's automation network with Honeywell's Uniformance PHD data historian with reporting functions, and installed its Service Node hardware server with software configuration in the plant network's demilitarized zone (DMZ) or Level 3 gateway. This server runs and manages the network's antivirus software, patch distribution, and related data gathering and monitoring tasks. Espoon Vesi uses this approach to gain an overall view of its automation system, receive information on damaged devices for immediate action, report on resource deficiencies, and send remote alerts and alarms to onsite personnel.
In addition, Espoon Vesi's automation network was upgraded to comply with Honeywell's cybersecurity standards to enable remote monitoring and reporting. The change was accomplished by transferring Honeywell's PHD and AWR reporting servers and the MySQL server of another supplier to the DMZ area. The distribution of Microsoft Software Update Services (MS SUS) batches and antivirus software batches was made automatic, and the remote connections were based on the virtual private network (VPN) gateway. The remote connection for monitoring is established as a VPN tunnel that allows secure network monitoring, and enables Honeywell's IT staff to monitor, analyze and report back to Espoon Vesi on its network, network devices and servers.
"Now, our plant alarms are relayed to the phone of the person on duty, who has a closed VPN connection to the plant's automation network server," says Jari Alvasto, Espoon Vesi's automation engineer. "We're able to quickly get the information on a damaged device and take immediate action if necessary."
Likewise, Schneider Electric reports that its EcoStruxure energy management platform uses open standards and an Ethernet-based backbone to tie together safety, reliability, efficiency and sustainability functions, and this unification enables it to bring an IT-style, defense-in-depth security strategy to the plant floor. "Control engineers are happier because IT realizes that process applications have different requirements, especially for network security," says David Doggett, Schneider's cybersecurity program director. "More companies learn both sides of this story, and understand that network security also is knowing exactly what components they have, how they're configured, and how to limit any changes to them."
Read Sysmantec Security Response's Network Security To-Do List to find out how you can minimize existing threats and keep ahead of new ones.
Secure Structure, Intelligent intervention
Eastman Chemical's Heard explains that successful mitigation for network security begins with doing site and application inventories, risk assessments and other cybersecurity-related homework before it's due. "Winning without fighting is best in cybersecurity, so planning and preparation are vital because there always will be faults and other items that need to be patched," Heard explains.
"However, cybersecurity is costly, so while you might end up with more resilient code, it can be hard to show that benefit on the bottom line. Fortunately, the operations side can learn from IT's five- to 10-year head start, and adopt many of its methods for patching software, and learning about the real costs of legacy systems. Running equipment until it rusts adds risk to commercial, off-the-shelf technologies."
Though it might be a relief to have new firewalls and a complete network security solution in place, this is when the real monitoring and detection chores begin. For instance, British Columbia Transmission (BCTC) supplies bulk electricity to the province, and it also must comply with North American Electric Reliability's (NERC) Critical Infrastructure Protection (CIP) standards.
"We have hard rules in effect to protect our critical infrastructure from inside and outside threats," says Tony Dodge, BCTC's IT planner and coordinator. "One key aspect of this is promoting a need-to-know management strategy to ensure only those who need to access our critical assets for their work-related duties are permitted to do so. But making sure all of the devices in our network are configured appropriately for the different levels of access can be challenging."
Consequently, BCTC uses Cisco's Intrusion Detection/Prevention System (IDS/IPS). Because it already used some of Cisco's security appliances, the utility was able to add IPS capabilities by installing Cisco's Advanced Inspection and Prevention Security Services Modules (AIPSSMs). Now, besides providing firewall and VPN services, ASA can monitor all network traffic to identify and lock down abnormal activity. To protect other network segments, BCTC uses standalone Cisco IPS sensors that provide the same defense and can be segmented into multiple "virtual" sensors. This enables BCTC to extend IPS protection across logically separated corporate and transmission networks without having to invest in separate hardware. To coordinate and enforce the diverse security policies required by these different networks, BCTC also implemented Cisco Security Manager (CSM) to manage different network policies, configure and tune security devices from one interface, and help BCTC demonstrate compliance with CIP regulations.
"Under CIP requirements, we have to track anyone making changes or updates to any of our firewall or IPS configurations," Dodge says. "We have to ensure that they're logged in properly and that we have a history of all changes made, and CSM can manage all of those policies centrally."
Packet Inspection, Patching at Many Plants
Although handling security in one application or plant might seem challenging enough, some users have to manage security in dozens if not hundreds of facilities.
For example, American Air Liquide operates about 200 plants nationwide, including 130 industrial gas plants. Many of these products have to be pharmaceutical-grade, and comply with the U.S. Food and Drug Administration's production regulations. To protect its PLCs, supervisory control and data acquisition (SCADA) systems and distributed control systems (DCSs), Air Liquide says it recognized early that it had to protect its plants from cyber-threats, and so it implemented sophisticated firewalls and embedded supervision of existing equipment, but an assessment showed it needed more protection of its SCADA and industrial networks.
"We examined a couple of ways to achieve our industrial network security objectives, beginning with simple TCP/UDP port blocking approaches in Layer 3 switches, but the resulting protection wasn't what we envisioned," Harper says. "We then evaluated intrusion prevention solutions (IPSs) from several vendors, and chose Top Layer Networks' IPS 5500." (Top Layer is now Corero Network Security) Layer 3 port blocking was inadequate because it couldn't inspect traffic allowed through its open ports, and its device configuration and management was manual.
In each of its plants, Air Liquide runs a small data center that processes terabytes of data for real-time, SCADA-based command and control of thousands of data feeds from its pipeline controls and production lines. IPS 5500 protects these data centers from outside threats by performing deep packet inspection of incoming information. IPS 5500's bypass mode allowed Air Liquide to plug it into its network and observe security events, including many posing real security risks to the network. In fact, IPS 5500 immediately identified oversized ping packets and nefarious DNS protocol violations. In addition, the bypass mode identified many active attacks originating from multiple sources and unexpected locations. For example, several threats were initiated by compromised computers that hadn't been patched with the latest Microsoft security updates, which caused Air Liquide to revise its patching process. So far, the company has installed this intrusion-detection solution at more than 100 plants across the U.S.
