IN0410_CvrStry_Fig2

A delicate balance

Oct. 14, 2004
Factory networks and enterprise networks seem like opposing forces in the industrial universe. There are ways, however, to find the Chi that reveals the harmony of a common business purpose.
 By Dan Hebert, Senior Technical Editor, and Joe Feeley, Editor in Chief

D

OGS AND CATS, LIBERALS AND CONSERVATIVES, Red Sox fans and Yankee fans. Some groups share a common space but just won&rsquot get along. Ever. A common view in our industrial networking space seems to be that factory and plant floor controls engineers and arch-rival IT professionals are polar opposites, too.

Until recently, these groups passed like ships in the night and the systems they captained did too. Plant and factory control systems running on hardy, hard-wired I/O networks went one way, mostly trafficking in isolated machine data and occasionally communicating machine to machine. Meanwhile office, enterprise data and telecommunications systems went another, running on fast, but delicate data networks that managed the flow of information across the organization. If the front office needed operating data from the factory, it came through the door on a clipboard via SneakerNet or was keyed into an office network workstation at the end of the day.

Then open-standards computing, Windows-based architectures, OPC, and Ethernet began to overrun the factory control network engineer&rsquos domain. In an ever-growing number of companies, much of the data in the machine and process system networks now needs to flow into the office, often in real time. 

The factory people didn&rsquot like outsiders messing with their networks. The IT group viewed factory and plant floor networks as archaic undisciplined balls of data-carrying wire.

But the mandate to make everything work in harmony is unavoidable. This is a true work-in-progress. After all, it&rsquos likely you&rsquove overheard some of the frustrations bubble up from the ranks of each group:

  • &ldquoThe IT folks don&rsquot appreciate that this is where our company makes money. They impose arbitrary standards that delay installation and support, then go home at 5:00 and we are left trying to solve the problem.&rdquo
  • &ldquoThe plant staff act like cowboys and don&rsquot appreciate that we have procedures that must be followed to protect the integrity of our computer systems.&rdquo
  • &ldquoThe IT staff is so paranoid about hackers that they won&rsquot make any changes that violate the configuration standards established for the office systems.&rdquo
  • &ldquoThey only were concerned about the plant floor when they installed that system. The databases are a mess and it is almost impossible to translate it to our accounting system.&rdquo
  • &ldquoThey just don&rsquot understand our business.&rdquo
In this article we&rsquoll explore what plant people view as the big issues and challenges, how that matches up with the perspectives of IT counterparts, and exactly where the differences are and, most important, where common ground can be found. This isn&rsquot easy, but there&rsquos reason to be optimistic.Vive la DifferenceIT departments constantly battle the ever-increasing need to provide more computing power and software applications, and the resulting demand for more bandwidth. A poorly designed enterprise systems easily can cripple heavily burdened communications channels. The IT department must be convinced that a new enterprise automation system will not adversely affect the rest of the communications traffic.The plant floor operates in real time, often on a millisecond-level scan cycle. PLCs are always talking, and when polling is used to process this chatter—as is usually the case—too much traffic is created on the network, at least from an IT perspective. The traffic can slow the entire network, clogging it with a continuous flow of information.When dealing with mission-critical ops, the factory floor needs much more reliability than IT, and &ldquonot only fewer outages, but faster recovery, along with more visibility of entire network,&rdquo says Gary Workmann, principal engineer of plant floor networks for the controls, conveyors, robotics and welding division of General Motors (www.gm.com). &ldquoDifferent system architectures are sometimes needed to satisfy IT vs. control needs.&rdquoControl system downtime cannot be tolerated, stresses David Glanzer, director of technology development at the Fieldbus Foundation (www.fieldbus.org). &ldquoEnterprise computing is mostly supervisory or accounting-related and can tolerate some downtime without affecting production or safety,&rdquo says Glanzer. &ldquoBecause of this difference the plant floor personnel will want to control access to network equipment and software changes. IT personnel are not used to asking anyone for permission to make changes.&rdquoIt is imperative that plant control systems be protected. &ldquoFrom a network perspective the technical issues for the plant floor are fairly straight forward,&rdquo says Chris Rogers, manager of electrical and process controls at Boise Cascade&rsquos (www.bc.com) Boise Paper Solutions in Boise, Idaho. &ldquoSome time is required working with the IT folks to develop a network standard that includes the plant floor network and the business network. The biggest issue is system security.&rdquoSecurity is indeed the big issue, agrees Russ Ropken, vice president of RSI Co. (www.rsicompany.com). RSI provides PLC and HMI development services and its specialty is OPC-based HMI Systems. &ldquoWhen the plant floor has its own, private network, security is not an issue,&rdquo he argues. &ldquoOnce connected to the enterprise, your system is open to viruses, hackers, etc. The last thing you want to deal with is a factory shutdown due to a virus that's running around the Internet. On the other hand, you don't want to lock down your plant machines to the point where your operators can't do their job.&rdquo
Get Set Up
Network setup issues are challenging for both sides, as PLCs use specific TCP/IP port numbers and getting those ports opened in firewalls and routers between the parts of the enterprise can be challenging. &ldquoIT folks are paid to keep things secure, every open port is a risk point,&rdquo says John Weber, president of SoftWare Toolbox (www.softwaretoolbox.com). &ldquoThe controls engineer can't go force the PLCs to use a specific port all the time. Sometimes they can. Other times they cannot.&rdquoOnce a compatible physical setup is achieved, the next issue for the plant is to identify what data is important to process, what format it is in naturally, and what format it might need to be converted to for use by other processes, says Michael Gurney, one of the principals of Concept Systems, an industrial system integrator that recently installed a new PC-based process control system running at industrial packaging manufacturer SpaceKraft Inc., Salem, Ore. &ldquoOne of the challenges that Concept had to overcome in the SpaceKraft application is that they needed to do some data manipulation in the controlling PLC in order to provide it in the correct format at the PC. Specifically, they needed to generate a special ‘motor runtime&rsquo parameter at the PLC by programming a counter.&rdquoThat thought gets plenty of seconds. &ldquoThe biggest challenge we faced with connecting to the enterprise was displaying the data in a manner that made sense to everyone from the plant floor to the front office,&rdquo says Rob Barnett, maintenance manager for Nursery Supplies Inc. (www.nurserysupplies.com), Chambersburg, Pa., a leading manufacturer of plastic containers for the wholesale nursery industry.Training Daze
When operators get more involved with operating data, they need a strong understanding of what they&rsquore doing and why. &ldquoTraining is probably the biggest plant-floor challenge,&rdquo says Steve Keller, an independent system integrator who works on projects that include small automated assemblies, and embedded programming and design. &ldquoPreviously information was written down and collected at the end of the shift and entered in the computer. Now, most of the information is entered via touchscreen, which updates the ERP system in real time.&rdquoHe says a lot of thought was put into designing the operator interface screens to be easy to use. Many of the operators are temporary workers who have very little experience.Misunderstood
&ldquoControls engineers we work with often feel that IT people don't understand their world,&rdquo says Weber. &ldquoBoth parties are very concerned about security, however, controls engineers tell me they have a hard time getting the IT people to understand that PLCs and control systems do things differently than traditional IT systems. Sure, it&rsquos all Ethernet, but on the shop floor, time can be much more critical in terms of responsiveness of networks, than on the office side.&rdquoBack to security for a moment. Ropken reminds us that the PCs in the automation system have a specific purpose: to provide a view into the plant floor system for the operators. &ldquoOnce connected to the enterprise, those operators might have access to uses that are not intended,&rdquo he warns. &ldquoYou don't want your operators to make bids on e-Bay or visit their favorite internet gambling site. These are new issues for automation engineers. They are not used to dealing with these outside challenges and it takes a real mindset change to insure these issues are dealt with. Then there&rsquos the often misguided idea that the plant loses control of how it presents data. &ldquoI think that the main concern on the plant floor in making the connection to the office systems is that the manager&rsquos life is one of change,&rdquo says David Stupar, Director, Information Services for the Ocean County Utilities Authority (www.ocua.com), Bayville, N.J. &ldquoIn many cases, plant floor personnel live in a world where the raw data is obscured from their bosses, so they can present the data in the best possible light. With transparent network connectivity, supervisors of plant floor personnel have complete access to the raw data and can form their own opinions on what is happening.&rdquoAnd in This Corner The IT group sometimes has a right to feel put upon. &ldquoThe plant floor wants to use 2-3% of the bandwidth of the existing IT fiber Ethernet network instead of putting in a new Ethernet network,&rdquo says Workmann. &ldquoSo the plant floor requires IT to upgrade its network, then tests the upgraded network for Quality of Service and other features needed for suitable performance in a control network.&rdquoPart of the problem, says Workmann, is that IT feels that all networks must be designed, installed, and supported by IT professionals. &ldquoThe plant floor concedes the need for professional design and installation,&rdquo says Workmann, &ldquobut feels that plant floor networks must be supportable by plant floor technicians and other plant floor personnel.&rdquoFor an IT person, says Weber, it is a challenge to understand the myriad of controls system hardware that is potentially on the plant floor. &ldquoThe terminology varies, the brands are many, the setup tools often look nothing like what they find in the IT world,&rdquo says Weber. &ldquoThe IT person needs the controls person to help assimilate and communicate what is needed, but in terms that the IT person can understand.&rdquoTo maintain consistency, IT will emphasize the need to install common router and firewall equipment at the site and will promote configuration standards established for the office environments, says says Tom Black, president of CygNet Software (www.cygnetetm.com). &ldquoThis is often problematic for establishing enterprise automation systems because of the real-time nature of the data communications that the IT staff may not initially appreciate,&rdquo he says. &ldquoFor instance, IT may not understand the need to open TCP/IP UDP ports for faster communications socket connections needed for data updates.&rdquoGetting the data into the proper places in the ERP system is one of the bigger challenges, says Keller. He works on larger projects--such as for Plaspros (www.plaspros.com), a medium-sized custom plastic injection molder. They have a main plant in McHenry, Ill., and a second plant in Batesville, Miss. The McHenry operation has 21 molding machines and Batesville has 15.&ldquoThe ERP software, Enterprise IQ by IQMS (www.iqms.com), is specifically designed for injection molders and other cycle-based manufacturing,&rdquo he says. &ldquoEnterprise IQ does have a system just for counting cycles, but we wanted to move more information to the ERP system, so we created a customized solution.&rdquoGurney agrees with Keller, saying the most important first IT step for merging the networks is defining the data that processing at the enterprise level wants to see. &ldquoBefore any programming is begun, we develop an interface design spec that lays out all of the data that will be required by a process and how it will be accessed, for example, which tag within a PLC will be referenced, and how the data will be routed to the server.&rdquoConnectivity Is Easy
For the IT folks, developing connectivity standards is fairly straight forward, says Rogers. &ldquoAddressing interfaces between multiple proprietary plant floor networks must be managed. But, managing domains on two sides of a firewall can be problematic. Not all plant floor networks are managed by the same group of resources, so multiple efforts to address an overall site architecture are needed. Eliminating multiple HMIs in the same operating space becomes a challenge.&rdquoWorkmann hits on a major difference. &ldquoIT optimizes its network for bandwidth, not for determinism,&rdquo he says. In addition, he says, IT feels that plant personnel are network novices not to be trusted, especially when connecting to IT networks.Clearly, security is a big deal for the IT folks, says Ropken. &ldquoWhen an automation system is added to the IT department's network, they want to ensure that the system will not pose a threat to the rest of the network. The IT department typically has a procedure to follow and strict rules for how a PC will be structured on their network. Details such as operating system, virus scanners, log-in rights, and password requirements are all going to apply to the automation system.&rdquoThese items, although important, can cause problems on an automation PC. &ldquoAs a quick example, virus scanners are very important for security, but these scanners can be very resource-intensive,&rdquo cautions Ropken. &ldquoYou don't want the virus scanner to negatively impact the performance of your automation system.&rdquoThen there&rsquos the &ldquocheck out the big brain on Brad&rdquo syndrome. IT resists plant floor use of Ethernet because they think they are the experts, believes Workmann. &ldquoAt GM, plant floor personnel submit Ethernet hardware to IT for interoperability testing, and this must be done before the hardware can be used in plant floor Ethernet networks,&rdquo he adds. &ldquoThe GM IT group found that hardware from different vendors—that is supposed to adhere to standards and be compatible—may not be compatible.&rdquoSome of the issues are quite specific. &ldquoAs far as IT challenges, the biggest challenge was justifying an additional web server to individuals not familiar with the data and its uses,&rdquo adds Barnett. A system that Barnett implemented had OPC interfaces but he needed a visualization front end for the information they could provide. A key design objective in building the system was the ability to access the system information anytime, anywhere, without having to load any software on a client PC (Figure 1). FIGURE 1: INFO ON DEMAND

Nursery Supplies Inc., a manufacturer of plastic containers, needed a visualization front end that could access the system information anytime, anywhere, without having to load any software on a client.
For automation systems that require local database servers, the IT department often will adapt its policies to accommodate sharing server management tasks with the plant floor staff, says Black, managing the database servers remotely, or establishing a local IT presence. &ldquoEstablishing on-site IT staff often leads to a problem down the road in that the IT department begins to feel that the local staff become ‘converted&rsquo by the plant floor and become advocates of the plant in times of conflict with the IT department,&rdquo he adds.Find That Common Ground
Rogers says coming to agreement on roles and responsibilities is tough. &ldquoPlant floor folks don't trust the critical nature of the plant floor network to IT folks, who have not always had stellar performance in the past,&rdquo he states. &ldquoIT folks don't really understand the real-time environment well. Their thinking is still batch.&rdquoThis is a much bigger issue than IT folks realize, says Rogers. Security issues are a real concern for the plant floor folks. All agree there is little value in redundant organizational roles. &ldquoMost of the issues are cultural and require a cultural solution, but neither IT nor plant floor folks are very good on people issues,&rdquo observes Rogers.Often there's a dispute over who owns the data and the systems. &ldquoProgram changes at the plant floor can affect the IT department's data, while IT's demand for data can affect how it is generated on the process floor,&rdquo says Gurney.&ldquoIt is fair to say that the two worlds are rapidly coming together, largely driven by the common standards and challenges that exist in both worlds,&rdquo says Marc Leroux, product marketing--collaborative production management, ABB Inc. (www.abb.com). &ldquoBoth, for example, rely on Ethernet and TCP/IP as the backbone for all activities, and both are subject to the threat imposed by hackers and PC viruses, and to some extent, desktop applications may be similar as well.&rdquoThat said, Leroux sees significant remaining differences. &ldquoThe plant floor is the world of OPC and single-value, time-series information,&rdquo he says. &ldquoThe IT world lives with SQL and transactional data. Transactional data can potentially be rolled back, or undone if it fails during any of the steps it must take. On the plant floor, information is normally related to manufacturing steps, and once a step has occurred, the data must be recorded as a historical event. There is no possibility to undo it later.&rdquoThen there are the separatist solutions. &ldquoI like the idea of two networks,&rdquo states Scott Hoffman of Warnecke Design Services, Ottoville, Ohio. &ldquoOne for the office and administration and one for the machine or plant-floor level. The network on the plant floor is more likely to have troubles such as tow motors hitting or running over cable and machines being moved. When machines are being moved, a lot of times the network is not properly disconnected from the machine. Now if the entire facility was connected together, the entire network could go down.&rdquo Keller client Plaspros is a relatively small company without a dedicated IT department. IT functions are contracted out, so &ldquothere were some disagreements with some of the implementation,&rdquo says Keller. &ldquoSome of the IT consultants pushed for VLANS, managed switches and tying the plant into the same switches the office was using. We opted for a separate system for simplicity. Because there is no dedicated IT staff we wanted to keep it as easy as possible to troubleshoot and repair.&rdquoStupar thinks differences arise from the fact that the two networks serve very different purposes and neither group wants to have their network relegated to a position of lesser importance than the other. Both sides fear compromise.One of the involved trade groups recognizes that threat of having to compromise. &ldquoThe great promise of open systems often has gone unfulfilled because the devices, programs and processes used at the various layers of the seven-layer Open System Interconnect (OSI) model have different options, capabilities and standards—or lack of,&rdquo believes Katherine Voss, executive director of the Open DeviceNet Vendors Assn. (ODVA, www.odva.org). &ldquoIntegrating these networks requires extra resources and programming. Even then, gaps between the systems often cannot be fully and seamlessly bridged. Consequently, users compromise their investments and rarely achieve all of the productivity and quality benefits promised by open network technology.&rdquo She argues that common application layers such as CIP (Figure 2), the Common Industrial Protocol that ODVA and member companies tout, are the key to advanced communication and true network integration, allowing complete integration of control with information, multiple CIP networks and Internet technologies.FIGURE 2: SHARED PARTS
ODVA claims common application layers such as the Common Industrial Protocol (CIP) are the key to advanced communication and true network integration, allowing complete integration of control with information, multiple CIP networks and Internet technologies.Success Breeds Familiarity
The manner of data reporting can go a long way toward satisfying both plant and office needs. &ldquoOur DataWorx software packages uses report-by-exception data logging, rather than the traditional polling method,&rdquo says Bill Glover, product  development manager, Inteworx.Net (www.inteworx.com), a software provider to AutomationDirect. &ldquoTraditional SCADA or HMI polling software packages act as an intermediary between PLCs and the computing system. SCADA or HMI applications have a tendency to tie up a company&rsquos network because they are continually asking for unnecessary data updates. Report-by-exception logging, however, allows direct communication between PLCs and the computing system, eliminating the need for an intermediary. The PLC is placed as a user on the network, which allows it to send information only when needed.&rdquoA report-by-exception logging method can ease IT&rsquos network traffic concerns, while still satisfying the plant floor&rsquos need for optimum performance and maximum speed. &ldquoThis minimizes some of the system challenges and communication barriers that traditionally have been seen in attempts to connect shop floor systems with corporate computing systems,&rdquo states Glover. &ldquoThe software allows the two systems--which run on different timetables, speak in different languages and have notably different goals with respect to data collection--to work together more efficiently.&rdquo Since Nursery Supplies installed a browser-based facilities-monitoring application built in-house using OPC application tools, Barnett says he&rsquos seen efficiency gains of nearly 15% and the annual maintenance budget is at an all-time low. He attributes the improvements to knowing what&rsquos going on in his critical facilities systems at anytime from anywhere. &ldquoWith this system, we literally feel we can be in two places at one time,&rdquo says Barnett. &ldquoThe knowledge we obtain from our application allows us to make intelligent, informed decisions about what systems need maintenance and when.&rdquoAfter considering a variety of alternatives, Barnett settled on OPC Web Client from Software Toolbox. &ldquoIt&rsquos very cost effective, provided data in my browser that would update without refreshing the page, did not require any software to be loaded on client PCs and was easy to implement. I&rsquom a PLC controls engineer and maintenance manager and I built this system myself.&rdquoBarnett says the software allowed him to access equipment with the click of a button anywhere on site, corporate wide, or even from home. I no longer have to wait on reports or track them down, I can see what&rsquos going on with critical equipment by clicking on our maintenance web site,&rdquo he says. &ldquoThis solution has already paid for itself. Our IT people love it—all of our connectivity issues were handled at the maintenance and plant level.&rdquo Gurney&rsquos SpaceKraft solution united multiple network protocols. &ldquoThe factory had three different incumbent networking technologies on the plant floor. We provided a PC with interfaces to DH+, DH485, and Ethernet/IP. Rockwell Software&rsquos PlantMetrics was set up on this PC, which was located in the plant's server room.&rdquo In addition to being able to talk to all the machines on the floor using their native network interfaces, executives at SpaceKraft can tap into the PlantMetrics server to get reports at their desks.&rdquoKeller says DataWorx allowed the Plaspros data from PLCs in the molding machines to be &ldquopushed&rdquo up to the servers instead of the servers polling each machine (Figure 3). &ldquoThis reduces the amount of network traffic greatly and makes the system more responsive and scaleable,&rdquo he says. &ldquoVB.NET is used to format the data before it is placed into the ERP database.&rdquo

Injection molders Plaspros pushed data from the machine PLCs to the servers instead of the servers polling each molding machine. This reduces the amount of network traffic greatly and makes the system more responsive and scaleable. VB.NET is used to format the data before it is placed into the ERP database.

Two Shall Become One
&ldquoIn our experience, we find that it is just as likely that the plant floor will champion the installation of enterprise software for the immediate benefits to plant operations as it is for the general office to desire enterprise connectivity and uniformity,&rdquo claims CygNet&rsquos Black. &ldquoThe best results are achieved when both departments are championing the effort together.&rdquoHe says that over time, the front office develops a greater understanding and appreciation for the operation of the plant floor. &ldquoFor instance, engineers who only visit the facility occasionally will be able to observe the operations of the plant in real time and provide feedback and guidance that would otherwise not be possible,&rdquo adds Black. &ldquoIn turn, request for facility and procedure changes are more easily sold to front office management.&rdquoAfter all, everyone really is on the same side here. &ldquoEnterprise systems, by nature, encourage uniformity throughout all of the plant floor sites,&rdquo reminds Black. &ldquoBest practices become apparent more quickly due to increased visibility and spread naturally to other sites. The result is better overall production and efficiency.&rdquoFinally, there&rsquos the need for essential upper management support. &ldquoI think the most import aspect of connecting the floor to the enterprise is the upper management&rsquos dedication to the process,&rdquo concludes Keller. &ldquoWe can collect every piece of data on the shop floor and present it many colorful ways, but if management does not bother to look at the data or require action when necessary then ultimately it has been a fun exercise in system integration, but the company will never benefit and the whole project will be looked at as a waste of money and time.&rdquo