The right hand has to know what the left hand is doing, especially when they're working on top of each other, or close to it. In these lean-and-mean days, operations and performance knowledge needs to move faster than ever. As a result, this information must be allowed to be exchanged between plant floors and business levels despite the risks posed by computer viruses and unauthorized intruders. Here's how some experienced users and their networks are doing it.
Jim Montague is the executive editor for Control. Email him at [email protected].
Bleeding-Edge Integration
Manufacturing Kogenate Factor VIII blood-clotting agent for hemophilia patients is serious and complicated business, so it demands some equally complex and well-coordinated networking. The biotech production process that Bayer Health Care uses to make it at its Berkeley, Calif., facility involves the usual two-stage method of growing cells in fermentation, and then purifying them to extract active ingredients. However, the specific process for building this coagulation drug is more complicated because it's difficult to assemble what is one of the world's largest complex-protein molecules. This complexity is essential for Kogenate to properly mimic in its patients the regular clotting factor found in people without hemophilia.
"When Kogenate Factor VIII was invented 20 years ago, it changed how this illness was managed," says David Kavanaugh, Bayer's principal process control system (PCS) engineer. "It enabled hemophilia to be treated with preventive therapy instead of having to be reactive."
The operational controls at Bayer's Kogenate production and related facilities include three primary systems located in multiple buildings at the site. The first monitors and controls flows, temperatures, levels, weights, dissolved oxygen and other variables in Bayer's fermentation vessels and related equipment using a variety of I/O modules and controllers managed by ABB's System 800xA distributed control system (DCS). There are about 8,000 I/O points spread among multiple fermenting units in Bayer's main production building.
The second system controls an advanced HVAC application that provides Bayer with a sterile manufacturing atmosphere for its cleanrooms. The motors, dampers, filters and other equipment are controlled by six building automation systems (BASs) from Siemens Industry, while their sensors and I/O components are controlled by Siemens' Apogee automation system.
The third system is Bayer's central utilities, which provide steam, ultra-pure water for injection (WFI) into the vessels, coolant, lights and power, and are monitored and controlled by GE Intelligent Platforms' Proficy iFix HMI/SCADA system. This system also includes several of Rees Scientific's Centron environmental monitoring systems (EMSs) for lab temperature monitoring.
Although each of the three systems worked well enough on its own island, Bayer's engineers and manager craved better data access, so they could make better and faster process decisions, and be more proactive and productive by throttling back downstream when an upstream problem occurs, or ramping up upstream when downstream indicates that it's idle and needs more to do.
One for All, All for One
"It's always been part of our vision to make process data available to everyone that needs it, and to do it in a way that's easy to use," Kavanaugh says. "Previously, all three systems were standalone, each with their own PCs, HMIs and historians, and the only way to get information was to be physically in front of each one. So, our staff had to do a lot of running between buildings to get information that wasn't accessible otherwise and couldn't be shared. As a result, our main job for the past five years has been to tie these three systems together with an overall process control system. We began by creating a standalone fiberoptic and copper PCS network that reached all our buildings, and piggybacked it on top of the corporate IT infrastructure that already had fiber, switches and telecom running to all the buildings. However, we then installed our own dedicated Ethernet switches that were under our control, so we wouldn't disrupt corporate IT and they wouldn't disrupt us."
The PCS network uses TCP/IP communications via Cisco Systems Ethernet switches that enable intelligent data routing. On the plant floor, the 800xA DCS also converts data into a form the PCS above it can use and send up to the enterprise level.
To make the PCS network secure, Kavanaugh follows Bayer's existing corporate IT security standards, which include firewalls for physical access, passwords and other standard IT policies and procedures (Figure 1). "Our PCS network is physically separate from the corporate network, except for one connection through a firewall, which is Cisco's Adaptive Security Appliance switch."
Figure 1: Bayer Health Care's biotech production process relies on a process control system that combines its local DCS devices controlling fermentation and purification, a highly automated HVAC system for cleanrooms, and a central utilities system. This three-part PCS piggybacks on Bayer's corporate IT network, but is physically separated from it, except for one firewalled connection.
Bayer Health Care
Marc Leroux, ABB's marketing director for collaboration and productivity, adds, "The only totally secure network is one that has no connections, but you can't run control systems and applications that way. So, security requirements get a lot more stringent with dedicated ports on their firewalls that only allow communications between specific MAC addresses, require authentication and security certificates to prevent man-in-the-middle attacks, maintain only one point of contact between the plant and the enterprise, and examine all traffic at that point, such as making sure all its communications are OPC messages."
More Needed
Unfortunately, although the PCS helped improve some operations, there were still too many times during the past two or three years when Bayer's users had to visit the standalone systems to secure data, Kavanaugh says. So, in 2009-2010, Kavanaugh and his colleagues added an ABB Smart Client web server to the 800xA system on the PCS. "Now, instead of running between buildings to check on tanks and other equipment twice per shift on the old DCS, central utilities can monitor performance and do troubleshooting and maintenance from its own plant, and we can check process data from our offices," he adds. "Our engineers really began to embrace the new PCS when they realized how much time and labor they could save and how much they could improve productivity. Though there are some people who haven't embraced it yet, we still know the web server is being highly used because now we get a lot of calls saying, ‘I can't get to my machine' whenever the Internet is down momentarily."
For example, process engineers can explore the DCS via the Smart Client web server for process and analytics data to help decide if a particular loop needs tuning, or check if a suspicious reading means that a filter may be plugged. Likewise, managers can check throughput, learn if they're getting the right percentage yield, and make sure that quality is staying within Bayer's pharmaceutical licensing guidelines.
Coaxing Human Cooperation
Although there are many hardware and software issues involved in improving plant-to-enterprise interaction, most require technical fixes that are relatively easy to implement. However, it can be much harder to get control and automation engineers, business administrators and IT technicians to recognize and cooperate on common goals.
"Even though we have all kinds of network connectivity and smart devices that can generate reports and send them back to users, other machines and the corporate level, many departments still don't understand each other's requirements and the needs of their customers," explains Bryan Newman, THINC product specialist and IT director for two years at Okuma America in Charlotte, N.C. The company develops computer numerical controls (CNCs), machining centers and The Intelligent Numerical Control (THINC). "As a result, IT's traditional answer on requests to connect factory floor to corporate was ‘no,' but it was because the business side didn't explain the value of the business intelligence they would get and why they needed it. There is no magic wand for this. It isn't a technical issue. Getting these parties to talk is a political issue."
Consequently, Okuma works with its staff and end users to help them understand the value of real-time data integration. "Once all sides are aware of the other's needs, improvements can be made fast," Newman says. "You just need to get up the ladder to the director of operations, CIO or IT director, help them understand the business value of enterprise integration, get them onboard to adjust staff priorities, and solve integration problems quickly. Small and mid-sized manufacturers can do it more easily. It's harder to climb up the chain in larger organizations."
To traverse different network protocols like Profibus and Modbus and access their data for other users, Okuma incorporates Ethernet gateways from HMS Anybus in many of its machines, including those using its OSP-P200 and THINC controllers (Figure 2). "In the past, if a robot talked Profinet, it would need 24-gauge, discrete I/O points, which meant a lot more wire and problems," Newman says. "So, six years ago, we began using Ethernet gateways, and the connectivity is much simpler, more robust and easier to troubleshoot, and users can see the I/O points and the machine's performance at a distance. These gateways also make it easier to transmit data items, setup information and programming changes, back into the machine via its interface."
Likewise, THINC's application program interface (API) can expose controller information to its own Windows-based operating system, and then the API can handle integration between the end user's machine tools and business applications. "For example, a bar-feeding machine can know it's time to make a part and which one to make because the information on the part's order document has been routed to the machine," Newman says. The business value of this integration is obvious because orders no longer have to be manually written and routed to the machine, and it can instead make the part and generate a real-time report. Once operations and corporate really begin working jointly, it's only logical to group the factory within the corporate IT network, so it can segregate and control data access, and improve security, too."
Several key groups need to cooperate to bring together plant-floor and business-level networks successfully. Bayer Health Care combined its process operations, HVAC and utilities systems into one process control system (PCS), and its principal PCS engineer, David Kavanaugh, advises:
- Partner with all parties involved, and especially with corporate IT on security.
- Recruit an upper-management sponsor and secure corporate commitment.
- Confer with all end users on what capabilities they truly need and want from a combined network.
- Make sure that data delivered to all parties is in formats they can accept and will use.
In addition, by using the thousands of software objects in its standard API, Okuma no longer needs to make the custom-programmed interfaces that used to be required to make machine data available to remote users and the enterprise level.
Safe Conversations
Despite the benefits, many potential users remain reluctant to do integration because of security concerns. Veteran users and other experts say this is where corporate IT can really help out its colleagues on the plant floor.
"The most practical networking methods to tie together and coordinate different organizational areas are well known to IT professionals," explains Roy Kok, evangelist at the OPC Foundation. "These include maintaining separate automation networks and enterprise networks, and then using firewalls to connect them. What was difficult in the past is that automation products didn't expose their internal requirements for communications, so networks were difficult to close down for security purposes. This has largely changed, with vendors being more open about what resources they need, and making them configurable. In the case of OPC-UA, this has been a core requirement from the start."
Many industrial networking protocols add software to enable more secure plant-to-business communications. For example, CC-Link IE has an embedded, two-tier, direct-to-IT, enterprise-connectivity appliance, MES Interface IT, which isolates user networks from viruses because it doesn't use TCP/IP. "The only information transported from the plant to the business through this gateway appliance is data specified to be consumed by that particular business system," explains Chuck Lukasik, president of the CC-Link Partner Assn. "All communications go through the appliance, rather than directly connecting Ethernet to a PLC."
Ernie Rakaczky, program director for process control system cybersecurity at Invensys Process Systems, adds, "A lot of security efforts focus on technical issues, but the single most needed security activity is good management. There is too much value in the plant and business sides working together, and so continuous security management must be adopted as a culture."
Near Future
Once a useful and secure plant-floor/business-level network is established, other users and technologies will be attracted to getting in on the new enterprise integration act.
For example, wireless is poking its nose into Bayer's PCS. The company already had been using wireless-enabled laptops with Remote Desktop Protocol (RDP) that links directly to the DCS to enable preventive maintenance on pumps and other process equipment in places where the DCS wasn't available before. Just a few weeks ago, Kavanaugh also demonstrated how Bayer's staff could use iPads and other tablet devices to access the DCS via RDP.
For the future, Kavanaugh is examining a "Virtualization of Computers" concept that Bayer's corporate IT department has used for about six years. This virtualization involves taking five or six physical web servers, and running them as virtual entities on one to three larger servers. This process breaks PC-based data-processing functions of the original servers into software-based pieces, so formerly separate and inefficient activities like idle time can be taken up and used more efficiently by the other virtual servers. Kavanaugh says he plans to convert about 10 physical servers to virtual, and load them onto a cluster of three vSphere servers from VMware. Servers to be virtualized could include the PCS's antivirus server, Windows patch server, two domain controllers and a network monitoring server.
"Virtualization can reduce a lot of overhead operations and maintenance costs by saving energy, allowing built-in redundancy, and making backup and recovery about 50-70% faster," Kavanaugh says. "Also, if one of the three new servers dies, the other two can keep the 10 virtual servers running. This means we'll have a much higher level of fault tolerance than with 10 regular servers running separately."
