CONGRATULATIONS. The factory floor networks are installed and running pretty much as designed. The important role you played in the project went well. Enjoy a moment of satisfaction for a job well done. OK, that moment is over. Now its time to consider, or perhaps reconsider, the decisions made about who will provide the network support to keep the networks reliable and secure. Do you keep it in-house, or do you find some hired guns?
Outsourcing plant IT serviceswhich include maintenance, upgrades, tech support and educationcan be a cost-effective way for manufacturers to manage their plant networks, especially when they lack the manpower to go it alone. Both control vendors and system integrators provide all these services, including 24-hour a day on-site maintenance and operations.
What issues are critical in deciding whether to outsource your network maintenance, upgrades, tech support and training? What about insourcing (doing the work in house) with the help of your IT department? Ironically, many manufacturers are using outsourcing, especially network training, to get started with insourcing. What ever path you choose, you need to carefully consider all the security issues entwined with todays open TCP/IP-based networks.
Just Call the Repair Man?
Most system integrators and control vendors discourage point service calls and contracts because its a little like calling the repair man to fix your refrigerator. You hope he shows up in a reasonable time, has the right parts and knows how to fix your appliance. Most manufacturers find they just cant wait for service and need to take matters into their own hands.
Doug Jackson, process control and SCADA specialist at Parker Water & Sanitation in suburban Denver, says, When theres a problem, we need to get it fixed right away, and we have the talent in house to do it. It makes it a lot easier when you dont have to try and find vendors at 3:00 a.m.
Companies that insource may still need to outsource their security needsat least until they get familiar with all the nuances of security.
Companies that insource may still need to outsource their security needsat least until they get familiar with all the nuances of security.
When your networks down, everybodys scrambling, so youre not going to wait for someone to show up, adds George Thomas, president of Contemporary Control Systems, about his own operation. Manufacturers need to build up some level of expertise in house to protect themselves against these crises.
Educating customers to become self-sufficient is a goal of Mike Wehrenberg, automation and control sales manager for Kendall Group, and a member of the EtherNet/IP Infrastructure Task Group. Wehrenberg conducts seminars on Rockwell Automation, Cisco, and Hirschmann networks for his customers control engineers and finds, that as a distributor/SI, these three companies will go in and help a customer when a problem arises thats too big to handle alone.
Outsource for the Long Term
For Evan Rademeyer, information executive for Sasol Synfuels in Secunda, South Africa, long-term outsourcing has been a requisite in keeping his companys 4,500 workstation, 450-node network running. This network has 186 miles of fiberoptic cable and 112 miles of copper, and the site occupies more than six square miles of territory. The network is comprised of an old, proprietary token-ring system and an Ethernet backbone. Rademeyers internal staff maintains the token-ring system, and a local IT company, Business Connexion, maintains the Ethernet backbone. Rademeyer, who worked closely with a group of control engineers to set up a manufacturing execution system (MES), says that outsourcing has been very cost-effective. His staff consists of upwards of 10 people, and works with his outsource partners who maintain the interface equipment between the plant and business systems.
John Eva, vice president of customer service at Invensys, agrees that outsourcing should mean more to a customer than just fixing a problem when it occurs. We have support people located near our customers who can deal with all three networks [business IT, control, and field], but our strength comes from the field and control networks, he states.
The issue is to take more of the responsibility for maintaining the availability of a network as opposed to correcting the network when it fails. If you look at it from a transactional point-of-view, the tendency among users is to wait until something breaks and then fix it. Wed rather maintain a networknot on a transaction basisbut on an annual agreement basis.
Another important aspect of an ongoing relationship is that a vendors support program can help a company look at its networks in ways it previously hadnt considered. Rockwell Automation says it can help customers take a holistic approach to security. We start with a standards-based approach, in other words, ISA SP 99, says, Gary Slivka, product manager of Industrial Network Services for Rockwell Industrial Automation Network Services. We want to help the customer look at security issues from a risk-management perspective and consider all the parameters that might be responsible for loss or damage should someone get into the system. We look at safety and regulatory compliance as drivers. We look at the network infrastructure, the firewalls, and the separation between business systems and process control. We ask the customer about its maintenance practices, and whether they allow anyone to come in with a laptop or if they restrict people.
|FIGURE 1: TOUGH ENVIRONMENTS TO SUPPORT|
|Many networks fail because of basic plant environmental conditions. The right physical connection choices such as those shown here can eliminate much of the worry. Source: Woodhead|
Connie Chick, controller & I/O business manager for GE Fanuc Automation, says its customers often enlist outside services to design and build network infrastructure, but once the network is up and running, they generally like to take care of their own networks.
Another source of help are the remote monitoring services many vendors offer. It still, however, has its detractors. Weve offered remote monitoring for some time on our machines, but it has yet to take off in the U.S., says Luis De La Mora, marketing, international sales, for Rovema Packaging. Remote diagnostics are very popular in Europe where there are more solutions in the field, and more customers have reached a comfort level with remote diagnostics.
When does remote monitoring work well? For G. James Australia, an integrated glass and aluminum manufacturer and contractor, remote monitoring can be used safely from thousands of miles away to fix a PLC problem. Headquartered in Brisbane, Australia, G. James uses Cisco networking equipment and software, and recently integrated its factory floor systems with its business systems, linking factories across five major sites, manufacturing in Australia and Malaysia, and outlets in four countries. When I was in the U.S., I was notified of a problem in one of our Sydney plants, says David Moy, technical services manager. Instead of explaining the nature of the problem over the phone, I was able to VPN connect back to Brisbane and use the internal network to access the relevant PLC in Sydney and fix it.
Why Not Insource?
Many manufacturers realize their business systems must be in sync with plant systems, and this means getting plant floor engineers to work with IT, the primary answer for insourcing. G. James wanted to integrate administration and manufacturing systems to reduce costs. Administrative and machine control networks were autonomous and did not communicate.
Orders on the manufacturing floor were completed using paper and pen causing long delays in collating information. Orders were commonly filled before all the paperwork was collated, says Moy. The process was complex and time-consuming, and customers could not be kept up-to-date. So we began looking into extending our network to the manufacturing floor. We knew that by connecting manufacturing processes to order management systems and automating them, we could streamline our operations, improve visibility, increase efficiencies, and enable our customers to remotely inquire as to the status of their orders in real time.
The most significant challenges were integrating administration and manufacturing systems so staff could gain access to real-time information, and enabling the automated equipment to be managed by the network. This meant crossing boundaries between administration and operational control on the factory floor and redesigning process control systems so they could be interrogated remotely.
This vision has been from the top down, with systems being built from the ground up, explains Moy, to ensure communications of end devices with upper applications, because its easier to build the upper applications knowing the boundaries of system end points.
How can successful insourcing work? Manufacturers working on this issue have achieved varying results depending on their internal resources. A controls engineer for a global coatings and materials company notes that he hopes to get to the point where corporate IT and the control group understand each others needs. He says that theyre not there yet, and the biggest issue is on-time reliability of the networkwhere the business systems can afford blips, but the control equipment cant. Right now, he says, his company maintains distance between the plant floor systems and the business systems, although they are investigating pulling data upward into their SAP system. According to him, many security issues still have to be worked out.
Mark Buettner, director, electrical & controls engineering, and Terry Ledoux, director, information systemsproduct supply, Nestle Purina PetCareNorth America, presented Plant I.S. & Controls: Shared Responsibilities Operations Guide at the recent ARC Performance Driven Manufacturing Forum. They concluded that such turf wars had arisen for several reasons including open network topologies and PCs becoming a shared resource with different needs between controls and the IT department, and that there was no defined ownership or cooperation between the two groups.
Neither organization took the time to understand the complexities of its counterparts operating environment and end goals. There was too much focus on controlling and maintaining the kingdom of knowledge.
In addition, misunderstanding and lack of cooperation led to reluctance between organizations to provide timely and reliable services to one another.
These issues and more were covered in A Delicate Balance, (Fall, â04 p12), which offers a thorough discussion of how the controls group and IT department can find common purpose.
Security: A Problem Regardless
Security issues affect networks whether work is outsourced or insourced. We outsource, and thats why I know outsourcing is a bad, dirty, untouchable subject, a senior plant engineer from a major vitamin company adamantly proclaims. He cites examples of contractors who indiscriminately lock down systems, making them difficult for operators to use. He accuses contractors of having a double standard. While they preach about how to avoid viruses and worms by staying away from certain web sites and not opening e-mail attachments from unknown senders, the same contractors bring in infected notebook computers, tie them into the control system and infect machines on the plant network, he laments.
His company now takes care of all its DCS equipment, the process information computer, etc. Before, we had e-mails show up on the root directory of our process information system, he states. And thats because we let these boys maintain it. He says hes seen it allincluding contractors who wrongly think they can back up open files. They cant maintain files that are open. They cant back up an open file, he fumes. So I ask myself what the hell are they doing in there?
Of course, not all contractors fall into the category just described. But before selecting a third party to maintain your network, consider an incident that occurred September 2, 2003, during an outage at the Davis-Besse nuclear power plant in Ohio. The U.S. Nuclear Regulatory Commission (NRC) alerted plant operators to a potential network vulnerability caused by the Microsoft SQL Server worm. This warning described a worm infection that increased data traffic to the sites network, resulting in the plants safety parameter display system and process computer being unavailable for several hours. Though safe operation of the plant was not affected, NRC regulations require that safety-related systems be isolated or have send-only communication with other systems. The cause of the infection was a contractor that made an unprotected computer connection to its corporate network, through which the worm reached the plant network. NRC also found that plant computer engineering personnel were unaware of a security patch that prevented the worm from working.
Insourcing: Security Assured
With insourcing and the merging of groups and networks, security issues become highly relevant responsibilities. My world changed dramatically when the blaster worm hit, says Dave Jones, automation engineer at Ash Grove Cement, Overland Park, Kan. who has sole responsibility for this companys networks. That was when we started taking these issues very seriously. Jones notes that Ash Grove has been fortunate so far, and with the right precautions, its possible to head off disaster. Concerns remain, however; for example, numerous OS patches need to be tested with control software before theyre placed on a computer. One of Jones pet peeves is cleaning up spyware and keeping it off computers in the first place. Educating users is not enough, he says. Putting strict policies on computers, clamping them down, and providing specific Internet-only machines can help clean up problems.
When IT groups and control engineers dont communicate, bad assumptions from each group can spell security problems. Holly Beum, president and consultant, Interface Technologies, works with both Fortune 100 and smaller companies alike. In the majority of cases, nobody is doing much of anything, which is very scary, she says. Some control engineers think that the IT department is protecting them through an upstream firewall, and they dont realize that the firewall is not sufficiently secureprimarily because the access control rules are set wrong (See Figure 2 below). The IT group thinks that if the control engineers needed security, theyd let IT know. So security falls between the cracks. Its not a well-defined responsibility. The biggest problem is the lack of security awareness on the part of control engineers. If they realized their systems were insecure, theyd do something about it.
|FIGURE 2: FIREWALLS LEAK|
A properly installed and configure firewall protects and separates. The control engineers might think the IT department is protecting them through that upstream firewall, but who makes sure the access control rules are set properly? Source: Cisco
Companies that insource may still need to outsource their security needsat least until they get familiar with all the nuances of security. In this instance, security experts, SIs, and vendor companies stand ready to help. Says Invensys Eva, We remind users that while theyre learning how to protect their systems, others are learning how to attack their systems. What you put in is obsolete the day its installed. The unfortunate part of security is that the weaknesses are always changing.
No matter the size of the manufacturer, insourcing has many benefits including quick response time and keeping the knowledge of the network in house, which many companies perceive to be an important security benefit in itself. A potential downside, namely giving engineers more challenge and responsibility, making them more marketable to others, can be a plus as well. Engineers who feel challenged and rewarded in their jobs will stick around as long as they are adequately compensated. If you insource, you may still need to outsource education and training. Outsourcing education on networks may be free from some vendors, or it may cost money, but in the long run, it will be worth every penny.
ALL TRAFIC IS NOT CREATED EQUALLY
|Control||Video||Data (Best Effort)||Voice|
|Bandwidth||Low to Moderate||Moderate to High||Moderate to High||Low|
|Random Drop Sensitivity||High||Low||High||Moderate|
|Delay Sensitivity||High||High||Low||Moderate to High|
The care and handling needs of control network data, although relatively low in bandwidth, are very sensitive to random drops, delays and jitter. Source: Cisco