Do Not Slip Up On Security

When It Comes to Protecting Your Network Against Intruders, You Can't Afford to Fall Down on the Job

By Jim Montague, Executive Editor

Pay attention. That's the first thing.

Icy pavement, dark alleys, newly mopped floors, biker bars, Wall Street, banana peels and industrial control networks all demand caution—or expect bruised backsides or worse. Your choice.

Likewise, anyone can kick in your front door or break a window, but that doesn't mean locking them is a waste of time or that you never can leave the house. Common-sense barriers prevent most residential break-ins, and keeping cash and other valuables in a bank lessens the losses from the few intrusions that statistics show must happen eventually. Both deterrence and mitigation mean never having to say you made it easy for the bad guys. Your choice again.

These physical analogies hold equally true for industrial network security. It also demands attention and focus because some intrusions might occur, but ongoing and evolving deterrence and mitigation can prevent most break-ins and limit those that do happen.

The Need to Prepare

"Our end users aren't making drastic changes because of viruses like Stuxnet, but they're certainly aware of it," says Steve Goldberg, industrial systems division director at Matrix Technologies (www.matrixti.com), a system integrator in Maumee, Ohio. "Most of our larger end users in oil and gas and food products feel like they're on top of their network security because they know they have to learn as they go. They already have their own security policies, but now they're telling their independent contractors that they must use the end user's laptops to access their system."

We still want basic antivirus software on each PC, so it helps that it's been learning to play nicer with our HMI/SCADA and controls software over the past five years.

Tom Lycans, Matrix's senior engineer, adds, "One adverse incident would show some end users how they could prepare their network security and the worth of doing it, but it's still hard for many of the smaller ones to justify it ahead of time. For instance, one client has its whole plant and corporate office on the same network, but just doesn't have the manpower to divide it yet. That's why we're helping them to prevent exposing the control layer to the business layer by only allowing communications across a firewall, which defines what data can go to the control layer, or by only letting data go one way from the PLC to business level. For example, we typically deploy two separate IP ranges on one PC with two NIC cards."

Matrix helps its users define and understand their specific network security needs, such as how much outside contact their users and applications require and how much they can do without, Goldberg and Lycans add. Next, they review what type and level of security is needed in each area of the organization, such as how much access is allowed to the corporate VPN, or if machines on the plant floor need to be locked down. Based on these studies, Matrix helps users pick and apply equipment that satisfies the security level required for each area.

"We still want basic antivirus software on each PC, so it helps that it's been learning to play nicer with our HMI/SCADA and controls software over the past five years," Lycans says. "I think Microsoft, Norton and McAfee got enough complaints about antivirus software identifying HMIs as viruses, so they added more-intelligent software engines, and stopped putting on the broad and indiscriminate threat masks they used to put on. We also have a couple of users that maintain ideal images of just the operating system software they need to run their PCs, and then ghost them to each new PC at installation or when needed. They also back up their data and software, so they can restore it during a recovery. These are tasks that can't be ignored. We and our users have to be as vigilant as we can, and these efforts have to be ongoing, and then evolve as needed."

How Badly Stuxnet bites

Despite these logical security steps, some of the always-simmering panic over network security boiled over last summer when Stuxnet officially emerged on July 14. The violently destructive Trojan-style virus used a security breach to exploit all versions of Microsoft Windows and infect Siemens Industry's Simatic WinCC SCADA software, PCS 7 distributed control system (DCS) and S7 controllers.

Symantec (www.symantec.com) reported that Stuxnet can steal code and design projects, hide using a Windows rootkit, use programming software to upload its code to PLCs monitored by SCADA systems, and then hide these code blocks, too. As a result, Stuxnet isn't just a rootkit that hides itself on Windows, but is the first publicly known rootkit that is able to hide injected code located on a PLC, according to Symantec. It adds that Stuxnet contains 70 encrypted code blocks that appear to replace some "foundation routines," which take care of simple yet common tasks such as comparing file times, and others that are custom code and data blocks. By writing code to a PLC, Stuxnet potentially can control or alter how a system operates.

Although Stuxnet seems to have been coded specifically for Siemens products, other products could be just as vulnerable to similar viruses or attempted intrusions, according to John Cusimano, security services director at exida (www.exida.com). "WinCC is by far the largest SCADA HMI package. It's embedded into everything. Whether you know you're buying it or not, it might be embedded in your system. That's probably why it was the target."

After Stuxnet was created and initial versions started circulating in June 2009, its developers created a second, more powerful iteration, which allowed it to spread among USB devices with virtually no intervention by victims. They also used encryption keys belonging to chipmakers Realtek and JMicron, and digitally signed the malware, so antivirus scanners would have a harder time detecting it. This allowed Stuxnet to defeat multiple-factor authentication.

One of the worst aspects of Stuxnet is its ability to use a man-in-the-middle attack. This means its software lets it appear to be just another PLC on a network, and deliver indications that everything is fine on a network, while doing damage behind the scenes.

Some early reports indicated that Stuxnet was developed by a U.S. intelligence organization to disrupt or damage nuclear fuel centrifuges in Iran. More recent evidence indicates that it was tested in Israel.

Run Away or Be Resilient?

How do you handle viruses such as Stuxnet and other network threats? Microsoft issued four patches to prevent further infections, and Siemens developed fixes for its devices. Most experts warn users not to plug unexamined USB data sticks or laptops into their networks, which is Stuxnet's preferred entry method, even though it also can get in via file transfers and other pathways (Figure 1). However, now that Stuxnet's software and method have been revealed, other similar viruses are likely just a matter of time.

{sideba1}

Your first impulse in response to Stuxnet might be to make like a turtle and pull your head in—sever your application and plant from all Ethernet-based and Internet-enabled connections to the outside world, tell your CEO and accountants to come get their production data manually, and tell any remote troubleshooters to get used to flying and visiting in person again. A reasonable impulse, especially if you haven't yet linked your plant to any Ethernet that connects to the business level and beyond.

Unfortunately, going standalone isn't exactly practical these days. Too many basic production and even critical systems now depend on Ethernet-based links to other systems, often because there are few staff left to tend many processes.

"It's not possible to put the genie back in the bottle at this point," says Eric Cosman, engineering consultant at Dow Chemical (www.dow.com) and ISA-99 committee co-chair. "Machines and processes can be infected by USB sticks or other data file transfers, so there's no way to guarantee that an industrial network won't be impacted by Stuxnet or another virus. We just have to try and prepare our networks to be as resilient as possible, just like when we back up data files to be ready for hard disk crashes. This is what Dow and everyone else is doing. We're developing contingency plans for when an incident happens, such as adopting zone-prudent practices to reduce or mitigate our risk and reduce any damage that does occur. For instance, we only allow process control to be done on PLCs for process control, and we typically don't allow unfettered access to the Internet from the plant floor. Now, this doesn't eliminate all risk, but it does reduce it a lot. It helps to think of security as a risk-management exercise."

Find and Fix Vulnerabilities

Fortunately, there are many ways to avoid and blunt intrusions, and dilute and mitigate them when they happen. This begins by inventorying and characterizing all the devices and applications on a given network, and then identifying all the security vulnerabilities on them. Many experts report that most plant-floor networks have many more paths to the outside world than they realize.

"Windows and Ethernet-based networking gave us standardized, open systems, and allowed companies to optimize their processes," says Eric Byres, CTO of Byres Security (www.tofinosecurity.com). "Even 10 years ago, you could order a shipment of paper in Japan, and 15 minutes later, a pulp and paper application in Vancouver would start setting up to produce it. All kinds of just-in-time manufacturing depend on these networks, so I don't think it's possible to go back to standalone systems. This is why it's so important to know where and how data flows in and out of your plant."

For example, Byres says, a couple of years ago he was asked to examine the network security of an oil and gas application. "They thought they had one Ethernet pathway with a firewall, but we found they had 17 pathways between the plant and the corporate level," he recalls. "There also was no accounting for serial links between the plant and the business. They didn't know that several of their PCs had networking capabilities via dual-homed cards. And there were pathways via laptops and USB sticks, too." Typically, a dual-homed PC, server or other device has two network interface cards (NICs), which can be located between two different networks. "Many people think dual-homed PCs can act like a firewall and help provide more secure access, but this is incorrect—hackers typically see them as juicy targets for attack," adds Byres.

So, users must document data flow and interface connections in their operations and facilities, build a threat model and risk assessment for each data source, and separate their network into isolated zones with firewalls that limit communications to essential data only, Byres says.

Cosman reports that users can begin to assess network security on their own, but adds it's often worthwhile to hire an expert to help find network vulnerabilities that need to be fixed. "Once a network's vulnerabilities are identified, there usually are many simple ways to help mitigate those risks," he says. "These include keeping software up to date and installing patches, and making sure your controls vendors certify those patches before you implement them. Many suppliers such as Honeywell, ABB and Invensys have been working hard to reduce this time from months to days, and Dow is benefiting from it. Also, it helps to design control systems with elements that can be taken down for patching without taking down the whole system. For example, if your application needs three HMIs, then it might be good to install a fourth, so they can be patched one at a time without disabling the system."

Common-Sense Segments and Firewalls

How do you decide how to segment your networks, where to locate your firewalls, and how much to restrict the data allowed to pass through them? Or do you establish white lists that define permitted data sources and recipients ahead of time?

"Critical areas must be managed differently than regular or support areas," Cosman explains. "You've got to ask, ‘If we lose one computer, what will happen? But, if we lose another computer, then what will happen to it?' If what happens is different on each, then they probably need different levels of security. This is the fundamental basis for the zone concept as described in the ISA-99.01.01 standard."

For example, a large, multi-unit petrochemical plant in South Africa discovered in December 2009 that the Win32/Sality virus had infected its DCS. This virus deletes many executable files, copies itself to removable files, ends processes and lowers security settings by modifying the registry. In this case, it shut down two OPC servers that were critical to the front end of the plant's process, and which could have halted the entire plant's operations, according to exida's Cusimano, who presented its case study at the Industrial Control Systems Joint Working Group (ICSJWG) conference this past October.

One of the OPC servers handled control-to-safety system communications, while the other handled control-to-electrical automation traffic. It's believed the virus came from an infected USB stick or from some software in new equipment that had been added to the servers recently or from the plant's business level. This uncertainty drove the plant's engineers to seek both immediate protection and a long-term evaluation of how to make their network stronger.

Initially, because the servers were inoperable, the petrochemical plant operators ran the unit partially blind for about eight hours using manual controls and radios, while their engineers wiped and reinstalled the two servers' software from scratch, and updated their access control list. The plant recovered without losing production, but the event was still an unnerving wake-up call.

"The engineers immediately updated the lockdown process for their safety instrumented system (SIS) and deployed new antivirus software, but they also called us in a few months later to assist their root-cause analysis and help re-architect their system to protect it better," Cusimano says. "They also implemented several policy and procedural changes, including a configuration management policy for IT switches, new third-party software policy, antivirus management policy, a prohibition on remote access, and an updated portable-media policy. We found network connections were not well-documented, insufficient separation between the business LAN and control system VLANs and ACLs, boundaries were unclear and had no boundary devices, several computers had hundreds of established network connections, and there were several dual-homed servers to manage."

Consequently, exida drafted nine critical recommendations and 48 other recommendations for the plant, including segmenting its network with firewall devices between its business zone, process information and electrical information subzones in its demilitarized zone, and the DCS, electrical, vibration and safety system subzones in its process control zone (Figure 2). To further harden its network, the plant was advised to remove all unnecessary applications and services, apply the vendor-recommended or NIST hardening settings to all workstations and servers, remove any unnecessarily shared devices, disable or lock any unused ports, and use physical devices to lock cables into used ports and block access to unused ports.

"The client learned that network segmentation is critical, antivirus software must be used per supplier recommendations, portable media can be dangerous, awareness and training is important, and that systems should be hardened and patched per supplier recommendations," Cusimano says. "However, we also learned that, while the ANSI/ISA-99.02.01 standard provides a good security structure, it can't be used as a checklist, and that zone and conduit modeling works, suppliers' reference architectures need to be adjusted for real applications, and that data collection must be performed very carefully on a live control system."

Besides employing ANSI/ISA-99 and other standards to help them segment their networks and prioritize their communications, some international end users also work with the International Instrument Users' Assn. (WIB, www.wib.nl) in the Netherlands and its 72 members to develop security requirements for its suppliers. The three-part organization includes Evaluation International, WIB and Exera.

"End users want a guarantee that vendors supply secure systems and services at all stages of the lifecycle," stated Ted Angevaare, Shell's global DACA manager and chair of WIB Plant Security's working group, during its seminar in March 2010. "We need fit-for-purpose security based on best practices of the WIB members, an affordable way for both large and small vendors to gain security certification, and a minimum standard freely available for everyone."

Good Security = Evolving Security

Not only is eternal vigilance the price of freedom, but it's also good for network security. "There's no free lunch on security," Cosman says. "You might want to just complete a checklist to know you're secure, but that's not possible. You can take steps to lower the probability of an infection, but you have to assume that you'll be impacted eventually, and so you have to back up files and data, create a disaster recovery plan, and perform drills based on it. Network security is a lot like an arms race because better solutions will evolve, but the threats evolve too. So, instead of just trying to prevent incidents, we also have to measure and improve how quickly we can recover."

In fact, several developers say that firewalls and other network gateways soon will perform data authentication at the switch level, which should make it easier to weed out and prevent potentially destructive traffic.  Ben Orchard, applications engineer at Opto 22 (www.opto22.com), reports that its PAC controller is adding SSL encryption to some of its communication methods in the near future, and is looking at extending similar functionality to other areas of the controller. Similarly, Rockwell Automation (www.rockwellautomation.com) uses digital signature checking in its ControlLogix L7x to authenticate its firmware, and this method likely will be used in the future to check software-based communications, network connections and individual data packets.

"People want to put security systems in place that will protect them, but what we haven't come to grips with yet is that our new security reality needs a cultural change, too," says Ernie Rakaczky, portfolio program manager for control systems and cybersecurity at Invensys Operations Management (www.invensys.com). "This will have to be similar to the change by which everyone came together on process safety, took self-ownership, and helped each other be responsible. Likewise, besides looking at security's tactical side of surrounding and protecting code, we also have to look at security's strategic side by learning to write code that is itself stronger and less vulnerable in the first place."