Modernization project upgrades automation in extreme environment
Steel components used in products such as cars, trucks, farm machinery, construction vehicles and military equipment have to tolerate high friction, force and temperature over a long lifetime. Design engineers specify various treatment processes for these components to improve properties such as wear resistance, extending their lives and reducing cracking and fatigue.
Heat treatment is one of the most commonly requested processes for this purpose, and American Metal Processing (AMP) in Warren, Michigan, is one of the nation’s foremost providers.
Established in 1945, AMP specializes in rotary heat treatment for carburizing, carbonitriding and neutral hardening (“quench and temper”) processes. With 18 employees and seven furnace lines operating 24 hours a day, AMP is able to process millions of parts, typically no larger than 5 inches, subject to some of the most extreme operating requirements.
Compared to competing methods, rotary heat treating yields tighter physical tolerances and greater uniformity through precise process control.
So, when AMP saw that its automation systems were reaching the ends of their lives, it undertook the difficult task of modernization.
Problems and opportunities
The first rotary furnaces introduced to AMP in the 1970s were controlled through precisely calibrated weights and scales. Computerized control was introduced in 1993 with a DOS-based control system and upgraded to PLC control in 2000, but these systems were aging poorly and had limited capabilities (Figure 1):
- Programmable logic controllers (PLCs) were experiencing regular failures and were limited to controlling only the feed system.
- The operating system (OS) of the feeder system’s human-machine interface (HMI) was approaching its end of life, and AMP had no access to the HMI program source.
- The majority of system components were not integrated or automated beyond direct motor control.
Figure 1: Computerized control was introduced in 1993 with a DOS-based control system and upgraded to PLC control in 2000, but these systems were aging poorly.
The company was on borrowed time, but we saw the opportunity to use this modernization project to make the system better than before. The system was primitive by today’s standards. It needed to be more than a control system. It needed to be intelligent.
Modernization would address the need for automated interlocking and alarm notification, as well as reduce operator error through a rich HMI.
In addition to these improvements, AMP also wanted to address pressing needs around data integrity and connectivity. We wanted a back end that would be on our network and which could be fully connected to our SQL database.
AMP’s customers are typically Tier 1 and 2 suppliers to original equipment manufacturers (OEMs), and a majority of those are in the automotive industry. These customers require multi-year part traceability in case they need to investigate a particular lot.
Twenty years ago, AMP invested in a work order tracking database for this purpose, which now contained billions of records and had become important to process metrics and quality control. Intelligently and seamlessly integrating operational data into this database became an important goal of modernization as well.
While purpose-built data logging systems are available for the heat treatment industry, they are usually sold as stand-alone systems, and AMP was uncertain of their ability to communicate with the work order tracking database. AMP decided to focus on upgrade designs that included integrated data logging as a feature of the system.
Pre-upgrade system description
AMP’s hardener furnace lines consist of five primary components: feeder, prewash, retort furnace, quench tank and conveyor.
Feeder: The heating process begins with a two-stage vibratory metering system with the feed rate controlled via an RS-232 connection from an Allen-Bradley programmable logic controller (PLC). Parts to be treated are fed through the prewash system and then the furnace at the rate specified by the operator via a Windows PC-based HMI. The HMI provides visualization of the current feed rate and the weight of parts loaded on the scale.
Prewash: Next, parts pass through a fixed-speed rotating barrel driven by an ac motor and speed reducer. Parts are cleaned in a high-temperature water spray on their way into the furnace.
Retort furnace: A rotating cylindrical retort conveys parts from one end to the other by slowly conveying parts forward across a series of internal vanes. Internal temperature is maintained by PID temperature controllers, and a two-position selector allows the operator to drive the furnace continuously in one direction or to select alternating forward-and-reverse motion when longer dwell times are needed.
The period of the alternating cycle is controlled by a dual-setpoint timer, and the rate of rotation is controlled by a variable frequency drive (VFD) and a 1-hp motor through a gear reducer. Operators manually set the output frequency of the drive via its integrated control panel.
Quench tank: Parts fall from the end of the furnace through a chute into a 7,000-gallon tank where they are rapidly cooled in either water or oil. The quenchant is circulated constantly by a fixed-speed pump driven by a large ac motor.
Conveyor: Treated parts are removed from the quench tank by a conveyor belt driven by an ac motor.
Control platform evaluation
Given the scope of its controls upgrade—including computerized weighing, burner control, and drive control—AMP knew it would require a custom solution. After receiving quotes from several industry system integrators, it became clear that we had two options: either pay someone more than $60,000 and give them the keys to the car or consider a homegrown solution.
Most of the companies we contacted wanted to completely rebuild the feeder system’s electrical panels, including custom programming for new PLCs and accompanying HMIs, leaving AMP with no access to the source code unless we also paid for the required design software licenses. AMP wondered whether it could take on the job itself, potentially reducing costs and giving it the option of retaining full ownership of its control logic.
For a business like AMP with fewer than 20 employees, developing a solution in-house is inherently more attractive than paying an outside company, since it allows complete control over the product and the company can modify, adapt or expand it as needs or equipment change. AMP’S general process is something that can be sketched out on a whiteboard, and we believed the solution should be simple to implement and understand.
AMP began researching what a homegrown solution would require. Opto 22 was identified as a potential solution vendor while exploring a feeder system I/O panel dating back to 1993 (Figure 2). Opto 22 I/O modules were some of the few parts still working. We got a quote for a groov EPIC system and were very impressed with the value and versatility offered.
Figure 2: AMP began researching what a homegrown solution would require. It identified a potential solution vendor while exploring a feeder system I/O panel dating back to 1993.
With EPIC as a potential foundation for its new control system, the company began mapping out a path to the goals it had identified using the content available through Opto 22’s developer forum, training videos and support channels. As a chemical engineer with no programming experience, I began learning Opto 22’s free PAC Control programming software with guidance from sales and support staff.
The experience gave AMP the direction it needed. Opto 22 does an amazing job of teaching people who have no background in programming. When I realized we could do data logging with the groov EPIC, that checked another box for me. Once we found a way for our lot tracking database to use REST (representational state transfer) calls to extract data from the groov EPIC, then we had a complete solution. EPIC made it possible to create a homegrown system.
System upgrade
AMP began developing its first groov EPIC system with a phased plan for upgrading each of its seven furnace lines. A month later, however, AMP suffered a critical PLC failure on one of its lines and decided to put an early version of the new system into service.
Built on the groov EPIC processor (GRV-EPIC-PR1), the first system incorporated the following I/O modules, each with channel-to-channel isolation, which allowed AMP to establish a basic control system (Figure 3):
- GRV-CSERI-4: four-channel serial communication module
- GRV-OVMAILP-8: eight-channel V/mA analog output module with chassis-supplied loop power
- GRV-ODCIS-12: 12-channel simple discrete dc output module
- GRV-ITMI-8: eight-channel thermocouple/mV analog input module.
Figure 3: The first system incorporated I/O modules, each with channel-to-channel isolation, which allowed AMP to establish a basic control system.
The feeder system scale was monitored through one of the system’s serial inputs, with both vibratory stages controlled through 0–10 Vdc outputs.
The retort furnace controls were integrated into the control program using one input channel to measure the rotational speed of the furnace; one analog output for setpoint control to the VFD; one discrete dc output to the VFD to control the direction of rotation; and four temperature readings of the furnace’s internal atmosphere and the quench tank.
The conveyor was integrated into the control program using an additional input to monitor the belt speed.
Finally, AMP’s groov EPIC was connected to the company network, completing a basic design that laid the foundation for continuing enhancements.
Because groov EPIC uses a quad-core ARM processor capable of running multiple applications in parallel and includes additional software applications, control logic was just the beginning of AMP’s modernization program.
Operator interface
One of the software applications included with groov EPIC is groov View, an embedded visualization server for creating onboard, external, mobile and desktop HMI displays. Along with duplicating the functions of the previous PC-based feeder HMI in groov View, AMP took the opportunity to integrate other control elements and create a unified operator interface (Figure 4).
Figure 4: Along with duplicating the functions of the previous PC-based feeder HMI, AMP took the opportunity to integrate other control elements and create a unified operator interface.
Furnace temperature, retort speed, which corresponds to dwell time, and quench tank temperature were added to the existing feed system readings, while the manual furnace timer and rotational direction selector components were replaced with HMI control gadgets. The result was a logically arranged operator interface that anyone could understand.
To enhance the interface even further, AMP utilized previously installed IP cameras directly above the feeder hoppers so operators could continuously observe the level of parts in each. AMP used the groov View camera gadget to add these camera feeds directly to the main HMI display, allowing operators to verify the hopper level at a glance and manage equipment from one location.
All HMI screens and camera feeds are available to operators from an external display connected to the groov EPIC’s HDMI port (Figure 5). And since groov View also supports mobile displays, our managers are able to supervise operations from their smartphones.
The new system is easy to understand and even easier to visualize, which builds confidence with existing customers and helps attract new ones.
Figure 5: All HMI screens and camera feeds are available to operators from an external display connected to the HDMI port.
Alarming
A controlled, multi-stage, high-temperature process like AMP’s involves various safety and productivity risks, so AMP also used groov EPIC’s toolset to add layers of protection.
In PAC Control, AMP implemented overfeed and underfeed monitoring along with process logic to interlock the upstream feeder and prewash systems should any of the downstream systems stop. It also designed an emergency oscillation routine for the furnace in the case that either the quench pump or conveyor motor stopped unexpectedly.
For operator safety and responsiveness, AMP added lights and sound beacons around the furnace perimeter. They added a discrete ac output module to the groov EPIC controller and wrote additional logic to drive the alert system whenever an alarm occurs. AMP configured event monitoring in groov View so any of these alarm occurrences would also be recorded in the HMI’s operator log.
As a final measure, AMP used another of groov EPIC’s embedded applications to interact with the company PA system. Node-RED is a visual event-driven programming language from IBM for connecting data from across the Internet. It includes a library of functions for designing transactions with databases, web services, and networked devices.
AMP used Node-RED to query the company’s operations database for alarm occurrences, generate a string representation of each (for example, “Temper furnace 204 overtemp alarm”), and send the string to a text-to-speech (TTS) function. Each EPIC stored a copy of this program, which executes in a browser on the company’s PA-management computer, creating unique audio announcements any time an alarm is triggered. Instead of an operator having to follow his ears to identify the location of an alarm, announcements direct him to the specific piece of equipment, so he can deal with the situation promptly.
Lot tracking and data logging
In addition to providing traceability of all the parts that AMP processes, the company’s SQL database stores heat-treating recipes and essentially serves as an ERP system. Previously, operators had to look up the appropriate recipe parameters in the database, input those into the feeder system HMI, as well as other furnace controls, and record work order information in the database prior to each run.
To reduce the possibility of human error, AMP integrated all of those tasks into a common interface. Now, the controller queries the company database via Node-RED and transmits recipe parameters—feed rate, retort speed, temperature, oscillation period and more—to variables in the PAC Control strategy. Work order information is recorded and sent to the database from the Work Order Data section at the top of the main HMI display. Operators can also use that interface to query the database for information on previous runs.
AMP then enhanced its basic work-order tracking by configuring periodic data logging of multiple process variables and setpoints for each run, including feed rate, VFD output and current draw, rotational direction, conveyor status and various temperature readings. These were stored in a one-week buffer on each EPIC with key data displayed in a 24-hour trend on the main HMI display. For long term storage, a time-series database (TSDB) was created and hosted on a secure cloud-based platform.
Now, each groov EPIC sends continuous process data through Node-RED directly to the TSDB. The level of redundancy offered by cloud-hosting is better than what AMP could have built on-premises, and the TSDB’s automatic compaction, compression, and downsampling algorithms intelligently balance data precision across time with ongoing storage costs.
The TSDB also made it easy to connect our process data to a visualization platform, so we created live dashboards of our process data that are displayed throughout the plant. In addition to combining work order and process data to calculate operational equipment effectiveness (OEE) and other key performance indicators (KPIs), AMP now has large screens displaying a beautiful bingo-board-style Andon system fed by live production floor data (Figure 6).
Figure 6: Live production dashboards like this one are fed by connecting groov EPIC data directly to a TSDB and visualization platform
The components of the groov EPIC, Node-RED and TSDB technology stack all play nicely with each other. Having a data logger is essential, but usually it's a separate system with its own set of logic and alarms built on technology that is far behind what we have now.
If you would have asked me two years ago if we could do this, I would have told you you were crazy, but now the pace of advancement is quite good. Operators and supervisors get instant feedback from the process. We monitor throughput and quality in real-time and can continuously improve our process based on what we learn. Our OEE will likely continue to improve over time (Figure 7).
Figure 7: AMP can monitor throughput and quality in real-time and can go back and change things throughout the week based on what’s learned. It expects OEE to improve over time.
What’s next?
AMP’s control system modernization project with groov EPIC has yielded tighter process control, multiple levels of error-proofing, better traceability, and historical data retention and auditing. But that was just the beginning.
AMP is now using an additional groov EPIC to monitor plant-wide systems. It started with cooling water temperature and ammonia supply level monitoring but was recently expanded to include dewpoint monitoring for endothermic gas production. AMP has five of these gas generators, which are critical auxiliary systems and require frequent sampling. We are now designing an automated valve manifold and PAC Control strategy to control the sampling process across all generators.
Beyond this, there are still many opportunities for improving and expanding basic control system functions. For instance, quench tank level management, like the feeder hopper, relies on operator observation. AMP would like to automate this and many other functions around the plant. Fans, motors, and temperature systems being controlled via manual controls are all candidates for modernization. If we can just whittle away 2%, which shoulld be achievable, the investment would pay for itself. It's like LED lighting. Why wouldn't you do it?
AMP is also in the early phases of planning furnace control and automation for a recently acquired sibling company, Detroit Steel Treating, which offers heat treating using batch furnaces. It intends to base these upgrades on groov EPIC, as well Opto 22’s groov RIO edge I/O module.
ALSO READ: PLC vs. Arduino for industrial control

Leaders relevant to this article: