by Stefan Winzinger, groninger USA
One of the characteristics of a world-class machine builder is taking care of its customers after the sale. This means service engineers often have to travel to customer sites to repair what turn out to be relatively minor problems. At other times, the engineers spend countless hours on the phone explaining technical fixes to customers.
When access to machines at customer sites is limited, providing excellent technical support is a challenge. The technical aspects of providing customer support are typically not at issue since no one knows the machines better than the builder.
Taking care of its customers is a key imperative for groninger USA. A wholly owned subsidiary of groninger & co. GmbH in Germany, groninger USA, calls Charlotte, N.C., home for its primary North American manufacturing and service facility. The company designs and manufactures fill and finish processing lines for the pharmaceutical and cosmetic industries (Figure 1). Since its formation in 1980, groninger has installed more than 3,000 machines globally, and more than 500 machines in North America.
Figure 1: This groninger packaging machine installed at a customer’s facility is connected back to groninger’s regional service office via a VPN router.
Spread Too Thin
A virtual private network (VPN) is a method to set up a secure connection between networks or end devices, regardless of where they’re physically located. A VPN is one way to create a private network over an otherwise public network, such as the Internet. Devices can communicate within a VPN as if they were directly connected. In some cases, a VPN is created between two devices already located on the same network simply to provide additional security and privacy.
Devices set up VPN communication by first authenticating their partner, generally with certificates or a pre-shared key, which functions as a password. After authentication, the VPN sets up encryption policies, and decides which methods to use to keep the data secure. The VPN also negotiates how often to refresh security information.
VPNs exchange encryption keys. Traffic comes from a node on one side of the VPN tunnel, is encrypted, and then is sent to the other end of the tunnel. The tunnel can span any physical distance, and go over numerous Internet routers to get to the other side of the tunnel. Once the other side of the tunnel receives the data, it decrypts it and sends it along to the node for which it’s intended.
VPNs replace slow and costly “point-to-point” dial-up connections by using the Internet as a way to get from point A to point B. They excel at securely and quickly exchanging data with remote sites, customer locations and branch offices.
In short, we wanted modern communications technologies that would allow us to securely and quickly perform remote diagnosis and troubleshooting over the Internet.
A VPN Solution
Our company developed Remote Video Service (RVS), which provides a customer-initiated secure virtual private network (VPN) connection between a machine at the customer’s facility and groninger’s secure internal service network. We use a VPN network address translation (NAT) router on both the customer and groninger service network sides to initiate and maintain a secure encrypted VPN connection. See the sidebar titled “What’s a VPN?” for further definition and explanation.
Remote Video Service is offered as an option when purchasing a new machine, or an upgrade for existing machines with Ethernet capability. The remote service function allows groninger to provide better support to more of our customers.
Remote Video Service Operation
“If a customer has a technical problem with a machine, he or she can call the regional groninger service office in Charlotte,” Klaus explains. “The groninger engineer on duty will ask the customer to initiate a VPN connection by operating a key switch. A VPN tunnel then connects the customer to that regional groninger service office through the Internet. At the service office, the VPN tunnel connects with the groninger internal service network (Figure 2). This is a secure network that connects all groninger service offices. This secure configuration allows the engineer on duty in Charlotte as well as groninger specialists in Germany to access the VPN tunnel to the customer.”
Figure 2: A VPN tunnel connects the customer to the regional groninger service office through the Internet. This secure configuration allows the engineer on duty in Charlotte as well as groninger specialists in Germany to access the VPN tunnel to the customer.”
The machine network—which can include PLCs, HMI, servo controllers, and other Ethernet devices—is at the customer’s end of the VPN tunnel. The services engineers and specialists at groninger can access these devices to see live program status, make changes if necessary, backup and/or restore programs, create new recipes, and provide machine or software updates or revisions. Here are the tasks that can be performed with the remote service function:
- Upgrade PLC software and/or programs
- Upgrade HMI software and/or programs
- Perform machine diagnostics
- Perform condition monitoring
- Perform machine troubleshooting
- Perform long-tem machine monitoring
- Create new recipes
- Backup PLC code and data
- Restore PLC software
- Restore HMI software
- Perform operator training
- Perform preventive maintenance visit preparation
- Monitor machine for preventive maintenance signals or flags
- Identify part sizes for spares
And the remote service function benefits:
- Eliminates unnecessary travel
- Lowers service and maintenance costs
- Allows service engineers to accommodate more customers in less time
- Reduces end user downtime
- Increases machine reliability
- Provides immediate troubleshooting
- Performs remote program and recipe updates quickly and securely
- Performs preventive maintenance tasks remotely
- Allows service engineers to visualize machine problems from the operators’ perspective via remote-controlled video cameras
“The remote service function also provides the capability for the customer to connect a remote-control camera to the machine network,” Klaus adds. “In addition to actually seeing live PLC and I/O status, groninger engineers can see the machine from an operator’s perspective by panning, tilting and zooming the remote-controllable camera to the problem area (Figure 3). After the problem is resolved or the machine PLC or program is updated, the customer can switch the VPN key to disconnect the machine network from the groninger service network.”
Figure 3: The operator interface terminal and other machine points of interest can be viewed at groninger’s home office via a remotely controlled video camera.
Typically, it’s our customers who control their VPN key switch and initiate the connection to the groninger service network. However, this isn’t the only scenario. In some cases, customers could choose an always-on VPN connection. Machine access for ongoing remote preventive maintenance is a good reason to have an always-on connection. To enable the always-on VPN connection, the customer would leave the key switch in the “on” position. In this mode, basically all analyzing, recording and monitoring items can be done automatically if the customer chooses.
[pullquote]The secure groninger service network is set up to handle more than one VPN tunnel from its customers, and we manage the IP addresses to make this possible. Firewall settings prevent customers from accessing the groninger service network outside of service engineer control. This is how groninger prevents customers from accessing other customers’ VPN tunnels that could be open at the same time.
“Every groninger machine also has an analog phone modem,” Klaus says. “However, while Internet speed continues to increase, using analog modems and phone lines is becoming increasingly problematic. In many facilities, analog phone lines are few and far between. If one can actually be found, it might not be in convenient proximity to the machine. The baud rate of an analog phone line is also very slow, especially as compared to a modern high-speed broadband Internet connection.”
There are, however, some customers who want a phone line modem because they still don’t trust a VPN enough and are afraid of compromising their own network security.
How It Works
Designing a system that is easy to set up and maintain in many different IT environments at various customer sites was one of groninger’s primary goals. We wanted to develop a system that would allow us to provide faster help with lower travel costs, resulting in less downtime for our customers. We also wanted to free up our service engineers so they could help more customers in a shorter period of time.
Network address translation (NAT) is the translation of an Internet protocol (IP) address used within one network to a different IP address within another network. One network is designated the inside network, and the other is the outside network.
Typically, a company maps its local inside network addresses to one or more global outside IP addresses. It also un-maps the global IP addresses on incoming packets back into local IP addresses. This helps ensure security since each outgoing or incoming request must go through a translation process that also offers the opportunity to qualify or authenticate the request or match it to a previous request.
NAT also conserves the number of global IP addresses that a company needs, and lets the company use a single IP address when communicating with the world. NAT routing can be used to join devices with identical IP addresses into a single, functioning facility network. NAT can be further broken down to 1:1 NAT, where one IP address is mapped directly to another IP address, and IP masquerading, where all “inside” addresses are translated to the router WAN IP address.
Many companies supply similar solutions, but we believed that mGuard had some advantages, such as the possibility to connect a key switch to open a VPN tunnel. Furthermore, through local supplier Cross Automation, it was possible to get all components for the RVS (remote camera, VPN Initiator, Power over Ethernet device, etc.) from one source and develop the system together with one contact.
“The router provides stateful firewall protection, network routing, NAT address translation, and support for IT networking protocols such as DHCP, DNS, QoS and VLANs,” says John Finta, Phoenix Contact automation sales engineer. “The router’s VPN feature supports all the necessary certificates, authentications and encryptions.”
A router moves packets of data through a series of networks from source to destination. Routers are often confused with bridges or switches, which perform packet forwarding, but only on a local network by using MAC addresses. Routers enable messages to travel via the Internet, and they can connect multiple networks together.
“A stateful inspection firewall is important because it keeps track of the state of network connections such as TCP streams or UDP communications as this information travels through the firewall,” Finta adds. “The router algorithm can distinguish legitimate packets for different types of connections. For example, a TCP packet that has the FIN flag set will not be accepted if a TCP packet with the SYN set hasn’t been seen in that stream. Only packets matching a known connection state will be allowed by the firewall; others will be dropped or rejected.”
Setting explicit rules for inbound communication is time-consuming when using a non-stateful firewall or a simple access-control list. Sometimes, this rule-setting step is skipped or not performed completely, allowing unwanted traffic to enter the network and rendering the firewall virtually useless.
With a stateful firewall, the intelligent connection tracking algorithm works on its own and allows users to only define rules for permitted unsolicited traffic such as a PLC that initiates a connection.
Easy remote camera setup was another major requirement for our company. “Camera setup should be as easy as plugging in one connector,” Klaus says. “The customer shouldn’t have to make multiple connections such as power, data and network. To meet this requirement, we use Phoenix Contact’s Power over Ethernet module to supply the remote camera with power and data in one cable.”
Wireless is another setup option for groninger’s RVS. With this option, there’s one mGuard, one key switch, and one wireless access point at each production floor. Each groninger machine using the wireless option has an antenna installed that allows it to connect to the wireless access point. “Wireless is especially effective for many of our cosmetic customers who must reconfigure their production lines regularly to accommodate changes in packaging size, shape and types,” Klaus explains. “We expected more hesitation towards wireless, but that didn’t turn out to be an issue.”
Phoenix Contact was closely involved in the RVS development process. For example, the NAT routing required an mGuard firmware version, which Phoenix Contact provided in just a few days. It also had a superior solution for initiating the VPN tunnel. This solution was to employ an integrated input to the mGuard to connect the key switch for initiating the VPN tunnel. Other companies proposed to just power off the VPN initiator with a dedicated contactor.
Deployment Progress
We formally introduced Remote Video Service at PackExpo 2010 in Chicago, and it was well-received by attendees, and subsequently by our customers. Because of RVS, our customers now receive immediate troubleshooting assistance and ongoing production support.
At the moment, a low percentage of customers use this service. Our pharmaceutical customers are hesitant because of revalidation and change control reasons. Adding it to an existing production line means a lot of paper work. Further, our existing customers are used to and happy with groninger’s good and quick field service; they would rather pay more for a person on site—even for small issues—than have to revalidate their equipment.
We now include RVS with the new machines we sell, so it gets validated right from the start.
Most of our cosmetic customers are huge corporations. So even if you have the operations people or the entire facility on your side, they still need approval from their headquarters to tap RVS into their LAN. In smaller companies, these decisions aren’t issues, and that’s where we have the RVS system installed. Smaller companies often are much more interested in cost- and time-efficient solutions, and it’s easier to jump the bureaucratic hurdles.
Mutual Benefits
The one time list price of an RVS system is so low that it pays for itself if it avoids a customer having to get about eight days of field engineer service. This doesn’t even include the production downtime losses they would suffer.
For groninger, the advantages are less about money, and more about introducing a much more flexible method of field service. A field service engineer who was bound to one customer can now serve more customers in the same time.
One lesson we’ve learned is to avoid going through customer LANs if we can. We are working on a project to add 4G cell phone capabilities to Phoenix Contact’s future mGuard devices.
Stefan Winzinger is an electrical and programming engineer at groninger USA (www.groningerusa.com). After completing a four-year apprenticeship at groninger in Crailsheim, Germany, in 1997, he worked there as an electronic service engineer until 2005, when he became part of a newly founded support team (called Technology Team) as engineer of electronic support and programming at groninger USA. In addition to expertise with servo systems and robotics, Winzinger is also proficient in programming systems and products.
