The two main questions in network security are: How closed does your network need to be? And, how open can you afford it to be?
Industrial network security is a delicate balancing act. In this case, the balance is between keeping equipment and processes protected—but typically isolated as they were in the past — and carefully allowing them to touch larger computing realms via Ethernet protocols and the Internet to gain new connections and capabilities — but exposed to potential viruses and attacks.
If you install too much security, little access is available and no productive work can get done, especially when users want to employ new devices and software tools that rely on external links. If you have too little security, your machines, application and plant floor are vulnerable to viruses, malicious software and even outright attacks. Several engineers report the biggest threat is someone inadvertently plugging an infected flash drive into the network.
Jim Montague is the executive editor for Control. Email him at [email protected].
"Whenever you try to move control system data to the IT network, you're going to need some kind of server, but we see security problems more as a result of conflicts with IT than from viruses trying to come in," says Francis Lauryssens, software specialist at Sun Chemical's pigments plant in Muskegon, Mich. "All the PCs, HMIs and other devices used to have hard-coded IP addresses, and I think a lot of the security problems we have now started when IT wanted to change these back to DHCP, which automatically assigns IP addresses. This allowed IT to change settings that we might not want to change and also means we're no longer sure where many devices we're communicating with are actually located. This is why we need firewalls and segmented networks and why we negotiated and collaborated with IT and agreed that there are some PCs that they can't touch."
Security Through Safety
Eric Cosman, engineering consultant at Dow Chemical and ISA 99's committee co-chair, adds, "We have a small staff group that is responsible for industrial control system cybersecurity, and we follow a multi-year plan that takes into account all the available standards and guidance. When we ask ourselves why we need to implement cybersecurity, we remember that our first concern as manufacturers is to protect people and processes. Whether an unsafe condition is caused by an equipment failure, mistake or cybersecurity incident doesn't matter. We are about safety first, and so we follow a similar approach with cybersecurity because the goal is the same." ISA 99 is the International Society of Automation's Manufacturing and Control Systems Security standard.
Likewise, security and safety both require inventorying related equipment and applications, dividing processes into manageable segments, conducting risk assessments, prioritizing potential hazards, implementing appropriate protections and then reevaluating on a regular schedule. Thinking about network security's similarities to safety could make it easier for many engineers to embrace.
"Though we in the controls and automation community usually know what we're talking about with safety, we still don't know what we're talking about when we talk about security," says Joe Weiss, PE, CISM, of Applied Control Solutions and author of Control's "Unfettered" blog. "Most security discussions still focus on IT issues, and so our controls folks need to get much more involved. We're the only ones who can develop security for our variable-speed drives and field devices. The major controls suppliers and SCADA vendors got religion on the need for security, but they still don't have a vision or plan for accomplishing it, and the many proprietary monitoring and controls systems usually haven't addressed any security issues."
Weiss claims there have been more than 170 cybersecurity incidents since 1998, including two cases in which people were killed, three that caused large-scale electrical outages and others that resulted in large equipment damage and significant spills to the environment. "Many of these were unintentional, some were intentional, and some were the result of unintended consequences such as software worm propagation," adds Weiss. "However, the bottom line is that people still talk about cybersecurity incidents as if they were hypothetical and blow it off. The other problem is that, even though we have some cyber-forensics tools for Windows, there are none for proprietary systems, and so many users wouldn't even know if they did have an incident."
Weiss explains that, while IT staffs believe that cybersecurity vulnerabilities require either a connection to the Internet, running Windows or using IP addresses, many security incidents in control systems have none of these three red flags. "In a test and demonstration of a real-world cyber-attack at Idaho National Laboratories about two years ago, the staff was able to physically destroy the couplings of a large diesel generator, and they did it via a dial-up modem," says Weiss. "Unfortunately, the related industries generally have done nothing on cybersecurity since then."
To help solve these problems, Weiss adds that both controls engineers and their IT counterparts must first convince their senior management to buy in with budgetary and organizational support. "Next, they need to find out what they have in their networks by walking it down, checking their equipment piece by piece, determining what each device is and how it's configured, and then doing risk assessments to help decide what security solutions each one needs," he says. "You have to understand which devices you have in the field that are cyber-sensitive, such as modems, and then understand what they're doing and what the ramifications are. This will help determine the risk for each and what should be done to mitigate it. Finally, your team needs to draft a cybersecurity policy, regularly reevaluate and redo it and update the cybersecurity for any equipment that needs it."
Bradford Hegrat, CISSP, Rockwell Automation's principal security consultant, adds, "Safety tends to start with a localized approach, and so it was driven from the bottom up. On the other hand, security is more systemwide, and so it's been driven from the top down. The trick now is to get them to meet and balance in the middle."
Better Tools
Thankfully, many of the basic network security tools are more capable and easier for new users to implement. These tools include intelligent network switches and routers, firewalls and hardware and software devices for managing network connections, authenticating users, encrypting data and establishing demilitarized zones (DMZs) and other segregated areas.
"The cybersecurity field has gotten more mature in the past two years, but it needs another five to 10 years to get standard methods, processes and engineering in place," says Eric Byres, CTO of Byres Security, which develops firewalls and other security devices, such as its Tofino firewall-configuration appliance. "There is good consulting advice available on cybersecurity, of course, but the average plant doesn't yet have a definitive code for cybersecurity. Still, the standards are coalescing, and this has tangible results on the plant floor because engineering managers are getting a better idea of what they need to do for cybersecurity."
Francis Lauryssens, software specialist at Sun Chemical's pigments plant in Muskegon, Mich., says there are several main tasks users must perform to increase network security.
- Turn on virus protection software, and be vigilant about installing patches. However, realize that security is about preventing unwanted events, and so it also involves keeping processes running properly.
- Use complex passwords that include numbers and mixed characters. In fact, uncommon passwords can be easier to remember than simple ones, if the brain is trained to do it. Also, change passwords every three to six months.
- Install firewalls, but then also monitor them to check on who is accessing the network and what software they're using.
- Turn off unnecessary ports and devices, so people and devices use only the applications appropriate to what they need to do their jobs.
- Turn down and lock down PCs as much as possible. For example, an HMI should only run the programs it needs to operate and only interact with appropriate devices, so delete programs like Internet Explorer that it shouldn't be using anyway.
- Train staff to follow security policies and not work around them.
Byres also agrees that network security and safety are two sides of the same coin, and that engineers can use how they think about and address safety to accomplish cybersecurity. "There are many safety techniques that apply to security, and so I think it's easier to turn a safety engineer into a security expert that the other way around," says Byres. "For example, the safety world can teach security people to be consistent, evaluate all points attached to their network and be more diligent about accounting for everything."
Similarly, advises Hegrat, to improve the cybersecurity of their own networks, users should first know their applications and determine who and which devices are supposed to talk to whom on the plant floor, so they can create functional zones insde their control networks. Next, they need to secure these subzones using Layer 3 access control lists (ACLs) and follow the Principle of Least Route to give the network more resilience to recover if a failure or security event occurs. Also, users need to inventory and prioritize their equipment and areas by criticality, rating the impact of the loss of those assets against the potential for occurrence and then use the results to select and implement appropriate protections, such as firewalls and DMZs."
In addition, while any network that uses TCP/IP needs to have firewalls and other security measures, some Ethernet-based versions of the fieldbuses have additional specifications, rules and requirements that can help discourage intrusions. For example, Chuck Lucasik, director of the CC-Link Partner Assn. Americas, reports that CC-Link Industrial Ethernet (IE) doesn't use the typical IP addresses in its devices, and so it's less susceptible to potential hacking because typical TCP/IP or UDP/IP messages can't reach them as easily. "CC-Link IE also uses an Ethernet Adapater switch to constrain the amount of traffic allowed onto its network, and this offers even more protection," says Lucasik.
Standards Coming Together
Dozens, if not hundreds, of cybersecurity efforts have been undertaken by governments, trade organization, standards bodies and other corporate groups in recent years, and a few are emerging as clear leaders:
- ANSI/ISA 99, "Security for Industrial Automation and Control Systems," covers all of the major process industries. Part 2 of its three parts was approved in January 2009, and it's scheduled to be delivered to ISO/IEC in February for eventual adoption as an international standard. ISA 99 provides guidelines to establish an industrial cybersecurity program for almost any control and automation system, such as DCSs, batch, SCADA and even discrete applications.
- North American Electric Reliability Council-Critical Infrastructure Protection (NERC-CIP) standard took effect Dec. 31 and governs the power generating and distributing industries. It requires users to have one year of auditable compliance by the end of 2010. However, critics say it allows users to define too many of their operations and assets as non-critical.
- American Chemistry Council's Chemical Sector Cybersecurity Program has been in place since 2002.
- National Petroleum & Refiners Assn. and its Cybersecurity Subcommittee work with other groups on cybersecurity issues, and have developed a variety of resources and recommendations on cybersesecurity procedures.
"I think the overall network security picture is getting clearer and better defined, and so users will be able to learn about and begin to implement them more easily," says Kevin Staggs, engineering fellow for cybersecurity at Honeywell Process Solutions, who also is one of ISA 99's working group leaders. "For example, ISA 99 is a lot more visible now. Over the next 18 months, we'll also see certification of many users' process application by vendors certified by the ISA Security Compliance Institute (ISCI) to evaluate compliance with ISA 99. Likewise, Honeywell and others can audit customers for compliance with NERC-CIP and other standards and provide recommendations and services for correcting problems, performing security updates, patch management and monitoring operations."
Here is a list of the major government and related organizations that deal with cybersecurity issues, coordinate responses across many industries and are helping to draft network security standards. Each has many resources and guides that can help users improve the security of their individual networks.
- U.S. Dept. of Homeland Security (http://www.dhs.gov/)
- National Institute of Standards and Technology (www.nist.gov)
- Idaho National Laboratories and Sandia National Laboratories' National SCADA Test Bed (www.inl.gov/scada)
- DHS and NIST's Process Control Systems Forum (www.pcsforum.org)
- Process Control Security Requirements Forum (PCSRF) (www.isd.mel.nist.gov/projects/processcontrol)
- U.S. Computer Emergency Readiness Team (US-CERT) and its Control System Security Program (CSSP) (www.us-cert.gov)
- North American Electric Reliability Council (www.nerc.com)
- SANS Institute (www.sans.org)
- Chemical Sector Cyber Security Program (www.chemicalsybersecurity.com)
- The Repository of Industrial Security Incidents (www.securityincidents.org)
Byres adds that much of ISA 99 was developed from two or three main sources. "A lot of the guts came from the ISO 27001 standard for establishing an IT security program, and were modified for the process and discrete manufacturing industries by James Gilsinn at NIST's Intelligent Systems Division," he says. "Much of ISA 99 was influenced by best practices for network security provided by Eric Cosman at Dow Chemical, Tom Good at DuPont and Johan Nye at Exxon Mobil. They deserve a lot of thanks because they and many other people busted chops and made a huge effort to make ISA 99 a reality."
Cosman adds that the best news right now is that there's a lot of cross-pollination between people involved in the major cybersecurity efforts. "We have a lot of work on NERC-CIP and smart grid security going through NIST documents and then on to be included in the ISA 99 work products," says Cosman. "The other good news is that we've made good progress in establishing a liaison relationship between ISA and IEC and that ISA 99 standards are immediately submitted to become international standards."
In addition, paralleling the recent cooperation on standards is an equally large gathering of users, suppliers and government agencies formed and launched as the Industrial Control Systems Joint Working Group (ICSJWG) in March-April 2009 under the jurisdiction of the U.S. Department of Homeland Security and its Control Systems Security Program (CSSP).
"In the past year or two, we've seen a much stronger understanding by the user community of the need for cybersecurity, but many folks still struggle to finance it," says Ernie Rakaczky, program manager for Invensys Operations Management's (IOM) control systems cybersecurity portfolio. "This is happening because traditional control system environments and processes, such as measurement, levels and alarms, are beginning to be thought of as something that can drive business models, too. This creates new risks and the need for cybersecurity over the life of the plant. This means setting up firewalls and then updating them as needed. This can be done by setting up two redundant firewalls, updating one while the other runs, and then going back and updating the first one."
Proper Patching
While generally stated standards and recommendations might be helpful, one of the most difficult, confusing and often unresolved security problems that controls engineers face is how to manage the frequent software patches coming into their networks. How are you supposed to apply patches to a production line or process application that must run continuously and can't be shut down and restarted whenever some bit of software shows up? Many of these updates are delivered by software suppliers on a pre-determined day each month, such as Microsoft's well-known second-Tuesday schedule, or more often if needed.
Consequently, while many IT staffs accept and push out patches to stop newfound viruses, control engineers must try to test patches to make sure they won't hinder or damage their application before distributing them to equipment on the plant floor or out in the field. In fact, many IT security departments still are settling on communication policies and procedures for identifying vulnerabilities and managing patches, but even more control security departments don't have the know-how or tools for deploying the protection that the patches are supposed to provide.
"We too have a pretty robust patching policy that continues to evolve," adds Cosman. "We're a big company with many systems to patch, but we manage to get it done. So, we tell others facing the same task that it's doable and to just go do it. Think of patch management the same way you think of doing other kinds of preventive maintenance you should have been doing all along. If we don't do preventive maintenance, we pay a price. It's the same with patching.
Cybersecurity isn't a job that can be done by engineers or IT alone, argues Cosman. "They must work together," he says. "It's tough to build and maintain these relationships, and it's even harder and more challenging to do on an ongoing basis, but it simply must be done."
Basically, the IT side has its firewalls, intrusion detection systems (IDSs), intrusion protection systems (IPSs) and other protection gear, but these devices don't have enough intelligence or sets of rules to function in process environments. For process control to gain the same benefits as IT security, its firewalls, IDSs, IPSs or security-configured switches need the right control-specific information embedded in them, such as what kind of network traffic can cause a loss of control.
As a result, the main strategy is to send patches to a segregated area, test them to make sure they don't adversely affect the control system, equipment and application and then download them later. "For our Windows software-based systems, we quarantine patches in a live environment, install the lead patch on a sample machine and see how that machine reacts before we let the patch onto the plant floor," says Marty Jansons, network consultant at Siemens Industry.
Similarly, Wurldtech Security Technologies creates resilience profiles for each device it tests, and these include a list of vulnerabilities, safe operating parameters and intrusion-detection signatures and firewall rule sets. This allows users to take the firewall, load it with these rule sets and give that firewall the ability to operate in a process control application by blocking the traffic that would otherwise trigger vulnerabilities in the control device.
To help increase the time between patches, Rockwell's Hegrat suggests that controls engineers, who know their applications and equipment well, and IT technicians, who know the IP source and destination addresses and TCP/UDP ports for those devices, can cooperate to draft an application white list for the network. "At some point, software patches will require a system to be rebooted, and this can hinder production," explains Hegrat. "However, a white list of known addresses and ports can help users design and engineer the system and network more appropriately, lessen the time needed to do real-time patches of devices when necessary and allow more lengthy periods between the times when patches and anti-malware are installed."
Cosman adds that the security-by-design concept also is beginning to emerge and that this will be the future of network security. "The big control system vendors are stepping up to the table, contributing to ISA 99's meetings and committees and designing security in from the ground up. They're starting to catch on that cybersecurity can be a market differentiator and competitive advantage," he says. "This is where we need to go, but it's still going to take awhile. The key is for users to demand security capabilities as part of the acquisition process."
