Who should provide your network support?

By Wayne Labs, Contributing EditorCONGRATULATIONS. The factory floor networks are installed and running pretty much as designed. The important role you played in the project went well. Enjoy a moment of satisfaction for a job well done. OK, that moment is over. Now it’s time to consider, or perhaps reconsider, the decisions made about who will provide the network support to keep the networks reliable and secure. Do you keep it in-house, or do you find some hired guns?Outsourcing plant IT services—which include maintenance, upgrades, tech support and education—can be a cost-effective way for manufacturers to manage their plant networks, especially when they lack the manpower to go it alone. Both control vendors and system integrators provide all these services, including 24-hour a day on-site maintenance and operations.What issues are critical in deciding whether to outsource your network maintenance, upgrades, tech support and training? What about insourcing (doing the work in house) with the help of your IT department? Ironically, many manufacturers are using outsourcing, especially network training, to get started with insourcing. What ever path you choose, you need to carefully consider all the security issues entwined with today’s open TCP/IP-based networks.Just Call the Repair Man?
Most system integrators and control vendors discourage “point” service calls and contracts because it’s a little like calling the repair man to fix your refrigerator. You hope he shows up in a reasonable time, has the right parts and knows how to fix your appliance. Most manufacturers find they just can’t wait for service and need to take matters into their own hands.Doug Jackson, process control and SCADA specialist at Parker Water & Sanitation in suburban Denver, says, “When there’s a problem, we need to get it fixed right away, and we have the talent in house to do it. It makes it a lot easier when you don’t have to try and find vendors at 3:00 a.m.”

Companies that insource may still need to outsource their security needs—at least until they get familiar with all the nuances of security.

A former control engineer who is currently the director of information for a pulp and paper company based in Washington State said it was easier to fix a problem in-house, especially when it involved, for example, overnight network administration, virus scanning and user IDs. As an IT person formerly involved with the control side, he has a thorough understanding of what is needed in plant network.“When your network’s down, everybody’s scrambling, so you’re not going to wait for someone to show up,” adds George Thomas, president of Contemporary Control Systems, about his own operation. “Manufacturers need to build up some level of expertise in house to protect themselves against these crises.”Educating customers to become self-sufficient is a goal of Mike Wehrenberg, automation and control sales manager for Kendall Group, and a member of the EtherNet/IP Infrastructure Task Group. Wehrenberg conducts seminars on Rockwell Automation, Cisco, and Hirschmann networks for his customers’ control engineers and finds, that as a distributor/SI, these three companies will go in and help a customer when a problem arises that’s too big to handle alone.Outsource for the Long TermFor Evan Rademeyer, information executive for Sasol Synfuels in Secunda, South Africa, long-term outsourcing has been a requisite in keeping his company’s 4,500 workstation, 450-node network running. This network has 186 miles of fiberoptic cable and 112 miles of copper, and the site occupies more than six square miles of territory. The network is comprised of an old, proprietary token-ring system and an Ethernet backbone. Rademeyer’s internal staff maintains the token-ring system, and a local IT company, Business Connexion, maintains the Ethernet backbone. Rademeyer, who worked closely with a group of control engineers to set up a manufacturing execution system (MES), says that outsourcing has been very cost-effective. His staff consists of upwards of 10 people, and works with his outsource partners who maintain the interface equipment between the plant and business systems.John Eva, vice president of customer service at Invensys, agrees that outsourcing should mean more to a customer than just fixing a problem when it occurs. “We have support people located near our customers who can deal with all three networks [business IT, control, and field], but our strength comes from the field and control networks,” he states.“The issue is to take more of the responsibility for maintaining the availability of a network as opposed to correcting the network when it fails. If you look at it from a transactional point-of-view, the tendency among users is to wait until something breaks and then fix it. We’d rather maintain a network—not on a transaction basis—but on an annual agreement basis.”Another important aspect of an ongoing relationship is that a vendor’s support program can help a company look at its networks in ways it previously hadn’t considered. Rockwell Automation says it can help customers take a holistic approach to security. “We start with a standards-based approach, in other words, ISA SP 99,” says, Gary Slivka, product manager of Industrial Network Services for Rockwell Industrial Automation Network Services. “We want to help the customer look at security issues from a risk-management perspective and consider all the parameters that might be responsible for loss or damage should someone get into the system. We look at safety and regulatory compliance as drivers. We look at the network infrastructure, the firewalls, and the separation between business systems and process control. We ask the customer about its maintenance practices, and whether they allow anyone to come in with a laptop or if they restrict people.”            FIGURE 1: TOUGH ENVIRONMENTS TO SUPPORT 

            Many networks fail because of basic plant environmental conditions. The right physical connection choices such as those shown here can eliminate much of the worry. Source: Woodhead

---

Rockwell Automation sees a trend toward customers with service contracts choosing long-term maintenance contracts, rather than point service, which yields limited benefits when technicians tend to come in cold on a problem. Slivka points out that many network failures occur because of good old-fashioned plant environmental problems, i.e., intermittent connections due to dirt, nasty chemicals, vibration or outright broken ones from accidents such as forklifts cutting cables or smashing into equipment (See Figure 1). So before you call in the specialist, make sure you’ve checked the obvious things first such as device LEDs at both ends or confirm that you at least have signal.Connie Chick, controller & I/O business manager for GE Fanuc Automation, says its customers often enlist outside services to design and build network infrastructure, but once the network is up and running, they generally like to take care of their own networks.Remotely Helpful
Another source of help are the remote monitoring services many vendors offer. It still, however, has its detractors. “We’ve offered remote monitoring for some time on our machines, but it has yet to take off in the U.S.,” says Luis De La Mora, marketing, international sales, for Rovema Packaging. “Remote diagnostics are very popular in Europe where there are more solutions in the field, and more customers have reached a comfort level with remote diagnostics.”When does remote monitoring work well? For G. James Australia, an integrated glass and aluminum manufacturer and contractor, remote monitoring can be used safely from thousands of miles away to fix a PLC problem. Headquartered in Brisbane, Australia, G. James uses Cisco networking equipment and software, and recently integrated its factory floor systems with its business systems, linking factories across five major sites, manufacturing in Australia and Malaysia, and outlets in four countries. “When I was in the U.S., I was notified of a problem in one of our Sydney plants,” says David Moy, technical services manager. “Instead of explaining the nature of the problem over the phone, I was able to VPN connect back to Brisbane and use the internal network to access the relevant PLC in Sydney and fix it.”Why Not Insource?
Many manufacturers realize their business systems must be in sync with plant systems, and this means getting plant floor engineers to work with IT, the primary answer for insourcing. G. James wanted to integrate administration and manufacturing systems to reduce costs. Administrative and machine control networks were autonomous and did not communicate. Orders on the manufacturing floor were completed using paper and pen causing long delays in collating information. “Orders were commonly filled before all the paperwork was collated,” says Moy. “The process was complex and time-consuming, and customers could not be kept up-to-date. So we began looking into extending our network to the manufacturing floor. We knew that by connecting manufacturing processes to order management systems and automating them, we could streamline our operations, improve visibility, increase efficiencies, and enable our customers to remotely inquire as to the status of their orders in real time.”The most significant challenges were integrating administration and manufacturing systems so staff could gain access to real-time information, and enabling the automated equipment to be managed by the network. This meant crossing boundaries between administration and operational control on the factory floor and redesigning process control systems so they could be interrogated remotely. “This vision has been from the top down, with systems being built from the ground up,” explains Moy, “to ensure communications of end devices with upper applications, because it’s easier to build the upper applications knowing the boundaries of system end points.”How can successful insourcing work? Manufacturers working on this issue have achieved varying results depending on their internal resources. A controls engineer for a global coatings and materials company notes that he hopes to get to the point where corporate IT and the control group understand each other’s needs. He says that they’re not there yet, and the biggest issue is on-time reliability of the network—where the business systems can afford blips, but the control equipment can’t. Right now, he says, his company maintains distance between the plant floor systems and the business systems, although they are investigating pulling data upward into their SAP system. According to him, many security issues still have to be worked out.Mark Buettner, director, electrical & controls engineering, and Terry Ledoux, director, information systems—product supply, Nestle Purina PetCare—North America, presented “Plant I.S. & Controls: Shared Responsibilities Operations Guide” at the recent ARC Performance Driven Manufacturing Forum. They concluded that such “turf wars” had arisen for several reasons including open network topologies and PCs becoming a shared resource with different needs between controls and the IT department, and that there was no defined ownership or cooperation between the two groups.Neither organization took the time to understand the complexities of its counterparts’ operating environment and end goals. There was too much focus on controlling and maintaining the “kingdom” of knowledge.In addition, misunderstanding and lack of cooperation led to reluctance between organizations to provide timely and reliable services to one another.These issues and more were covered in “A Delicate Balance,” (Fall, ‘04 p12), which offers a thorough discussion of how the controls group and IT department can find common purpose.

Security: A Problem Regardless
Security issues affect networks whether work is outsourced or insourced. “We outsource, and that’s why I know outsourcing is a bad, dirty, untouchable subject,” a senior plant engineer from a major vitamin company adamantly proclaims. He cites examples of contractors who indiscriminately lock down systems, making them difficult for operators to use. He accuses contractors of having a double standard. “While they preach about how to avoid viruses and worms by staying away from certain web sites and not opening e-mail attachments from unknown senders, the same contractors bring in infected notebook computers, tie them into the control system and infect machines on the plant network,” he laments.

His company now takes care of all its DCS equipment, the process information computer, etc. “Before, we had e-mails show up on the root directory of our process information system,” he states. “And that’s because we let these boys maintain it.” He says he’s seen it all—including contractors who wrongly think they can back up open files. “They can’t maintain files that are open. They can’t back up an open file,” he fumes. “So I ask myself what the hell are they doing in there?”

Of course, not all contractors fall into the category just described. But before selecting a third party to maintain your network, consider an incident that occurred September 2, 2003, during an outage at the Davis-Besse nuclear power plant in Ohio. The U.S. Nuclear Regulatory Commission (NRC) alerted plant operators to a potential network vulnerability caused by the Microsoft SQL Server worm. This warning described a worm infection that increased data traffic to the site’s network, resulting in the plant’s safety parameter display system and process computer being unavailable for several hours. Though safe operation of the plant was not affected, NRC regulations require that safety-related systems be isolated or have send-only communication with other systems. The cause of the infection was a contractor that made an unprotected computer connection to its corporate network, through which the worm reached the plant network. NRC also found that plant computer engineering personnel were unaware of a security patch that prevented the worm from working.

Insourcing: Security Assured
With insourcing and the merging of groups and networks, security issues become highly relevant responsibilities. “My world changed dramatically when the blaster worm hit,” says Dave Jones, automation engineer at Ash Grove Cement, Overland Park, Kan. who has sole responsibility for this company’s networks. “That was when we started taking these issues very seriously.” Jones notes that Ash Grove has been fortunate so far, and with the right precautions, it’s possible to head off disaster. Concerns remain, however; for example, numerous OS patches need to be tested with control software before they’re placed on a computer. One of Jones’ pet peeves is cleaning up spyware and keeping it off computers in the first place. “Educating users is not enough,” he says. “Putting strict policies on computers, clamping them down, and providing specific Internet-only machines can help clean up problems.”

When IT groups and control engineers don’t communicate, bad assumptions from each group can spell security problems. Holly Beum, president and consultant, Interface Technologies, works with both Fortune 100 and smaller companies alike. “In the majority of cases, nobody is doing much of anything, which is very scary,” she says. “Some control engineers think that the IT department is protecting them through an upstream firewall, and they don’t realize that the firewall is not sufficiently secure—primarily because the access control rules are set wrong (See Figure 2 below). The IT group thinks that if the control engineers needed security, they’d let IT know. So security falls between the cracks. It’s not a well-defined responsibility. The biggest problem is the lack of security awareness on the part of control engineers. If they realized their systems were insecure, they’d do something about it.”

FIGURE 2: FIREWALLS LEAK

A properly installed and configure firewall protects and separates. The control engineers might think the IT department is protecting them through that upstream firewall, but who makes sure the access control rules are set properly? Source: Cisco

Companies that insource may still need to outsource their security needs—at least until they get familiar with all the nuances of security. In this instance, security experts, SIs, and vendor companies stand ready to help. Says Invensys’ Eva, “We remind users that while they’re learning how to protect their systems, others are learning how to attack their systems. What you put in is obsolete the day it’s installed. The unfortunate part of security is that the weaknesses are always changing.”Work Together
No matter the size of the manufacturer, insourcing has many benefits including quick response time and keeping the knowledge of the network in house, which many companies perceive to be an important security benefit in itself. A potential downside, namely giving engineers more challenge and responsibility, making them more marketable to others, can be a plus as well. Engineers who feel challenged and rewarded in their jobs will stick around as long as they are adequately compensated. If you insource, you may still need to outsource education and training. Outsourcing education on networks may be free from some vendors, or it may cost money, but in the long run, it will be worth every penny.ALL TRAFIC IS NOT CREATED EQUALLY

	Control	Video	Data (Best Effort)	Voice
Bandwidth	Low to Moderate	Moderate to High	Moderate to High	Low
Random Drop Sensitivity	High	Low	High	Moderate
Delay Sensitivity	High	High	Low	Moderate to High
Jitter Sensitivity	High	High	Low	High

The care and handling needs of control network data, although relatively low in bandwidth, are very sensitive to random drops, delays and jitter. Source: Cisco