In most industrial applications, real-time control seems to break down into a large population of users who have relatively little hard real-time requirements, and a somewhat smaller group with significant real-time requirements.
Microsoft is doing a brisk business in providing Windows NT/CE/XP operating systems to meet the needs of the relatively low-hanging fruit in the control and automation orchard, and for ample reason. The development environment is well-known by software engineers, connectivity to higher corporate systems is a relative no-brainer, and the operating systems require little, if any, in-house development and maintenance costs.
But for that last segment of the control community--the roughly 10% of players who have hard real-time operating requirements--the choice of operating system becomes much more murky.
Although the end user almost certainly will want connectivity to higher systems and easy operating controls, the machine OEM first and foremost is concerned that the equipment meets its real-time mission responsibilities in a wholly reliable and repeatable fashion. Microsoft products also compete in this arena, but the field is much more open than in higher-level systems.
When in Doubt, Compare
Asyst Technologies (http://www.asyst.com) has a long history of working with Windows products for motion-intensive wafer handling systems that service the semiconductor industry (Figure 1). Karl Shieh, Asyst senior director of software technology, studied the real-time performance capabilities of both Windows CE and VxWorks, and opted to switch to VxWorks from Wind River Systems.
"VxWorks looks a little better in terms of performance," he says. "We do a lot of servo control and need to know that those motors are moving very smoothly with no delays, and that they have all the information they need in order to move with reliability and repeatability. With VxWorks, for the same code base, we get a 20-25% performance increase."
Shieh quickly adds that the performance boost is due to VxWorks being a smaller operating system--it's not as feature-rich as its Windows counterpart. "That's a double-edged sword for us," he notes. "We realize it doesn't have the same level of protection, inherently. But we trade that off for performance."
Sheih says that Asyst software developers have found the Wind River Tornado development environment to be very similar to Microsoft's Developer's Studio, and therefore the migration to VxWorks was relatively painless for the engineers.
Motion Control as a Necessary Evil
Machine builders that make equipment for use in industries such as the manufacture of semiconductors know their customers, with expensive semiconductor fab lines, consider robotics and other machinery to be something of a necessary evil.
Fab line managers want motion control operations to be completely transparent to the overall operation by being entirely reliable and clean, never causing damage to in-process or finished wafers. Those engineers often chose a control system not for its operating software, but for its overall performance capability.
"The operating system itself was not a key factor in the controller decision for our application," notes the automation engineering manager for a semiconductor tool manufacturer. "The primary factors in the decision were the performance and functionality offered by the controller for an application of this type."
As a result, development ease is not always a pivotal issue. Some engineers believe the up-front investment required to learn a proprietary programming language will be recovered in the end product. Operator ease, however, is almost always a key concern, and therefore the design usually includes a Windows application as a sub-system graphical user interface for set-up, configuration, and troubleshooting, as well as data-management purposes.
Indeed, many industrial OEMs include Windows at some level of their real-time systems, even if they opt to perform real-time control on another OS. To achieve the real-time control on a Windows OS, some machine builders find third-party extensions can provide adequate performance for their control needs. Thus, the end user could maintain a Windows-centric platform corporate-wide.
"Most intelligent equipment today is part of a larger plant information system that the equipment connects with," explains Myron Zimmerman, chief technology officer with real-time extension developer VenturCom (http://www.venturcom.com). "It's easier for the control manufacturers if they tell their customers a Windows operating system is being used on their equipment. That makes everybody feel it's going to be easier to integrate that equipment with other Windows-based plant information equipment."
The Joy of RTOS
Some OEMs may opt for an RTOS even if their real-time requirements are less stringent than the sub-millisecond range simply because the system provides a level of assurance to their users. Officials at Gem City Engineering Co. (http://www.gemcity.com), a manufacturer of automation equipment used in the semiconductor, microelectronics, and biotechnology industries (Figure 2), have found that, in such cases, a non-Windows RTOS can be more economical than the Windows systems they install on most of their equipment.
"There are times when I use a real-time system, not because it's a real-time system, but because I can get a single-board computer for a very low price when I have multiple builds of the job," says David Burton, Gem City engineering manager, controls and software. "I don't want to have to put a whole PC or even Windows CE on it. So I go with something that has the capability of being a real-time system, but not necessarily use it at its highest speed."
In such cases, Gem City may use a VME rack with a third-party proprietary RTOS embedded on the chip. "Typically, we find that DOS is very capable when it comes to real-time management."
Gem City also uses single-board microcomputers from Z-World (http://www.zworld.com) programmed in a Dynamic C environment for cost-effective real-time control.
Opting for Others
The maturation of the PC-based control arena has cut into Microsoft's market share because other companies have had time to become established in given market segments.
"Four years or so ago, there was a lot of appeal to Windows because of its compatibility with other systems, and the software was easy to understand," says Todd Hansell, chief software engineer with Woodward Industrial Controls (http://www.woodward.com). "That isn't the case as much anymore."
Woodward's Atlas PC industrial controller for turbine and engine control currently uses Windows NT 4.0 and VenturCom real-time extensions, but later this year the system will be offered with VxWorks. "There has been a leveling of the playing field with VxWorks," notes Hansell. "The availability of drivers and I/O for the VxWorks RTOS has taken away much of the advantage of the Windows OS. We've found that VxWorks gives better performance in many cases."
Like many industrial automation suppliers do when choosing a real-time operating system for a new control platform, engineers at GE Fanuc (http://www.gefanuc.com), took a good look at the Microsoft products as well as other solutions in the marketplace. In the end, GE Fanuc opted to split the difference and will offer customers both Windows and VxWorks versions of its next control platform.
"The choice of an operating system is based on the flexibility that we can give the customer by choosing it," says GE Fanuc control manager Connie Chick. "The customer is concerned that the machine runs reliably when they want it to run, that it won't fail, and has certain features."
Those features may include a view functionality to combine the machine operations with an operator interface. Customers with that requirement, such as a high-end turbine control application, may be likely to choose the Windows NT or CE OS. Other customers who would just do upward communications from a machine without viewing needs might opt for the VxWorks version.
"As we move forward with this new platform, we'll be able to load essentially the same firmware for each control system," notes Chick. "Someone programming our system will be programming for one system and then putting on the hardware of choice, depending on cost or features."
As a PLC manufacturer, GE Fanuc has a long track record of building its own proprietary control systems and even processors and chips as a way to ensure system reliability. Advances in off-the-shelf operating systems and processors have largely obviated the need for such rigid vertical manufacturing requirements, and thus control manufacturers can focus on their core competencies to improve control functionality.
"Control companies had to worry about the OS and the CPUs and the chips, in addition to what functionality to bring to the customer," says Chick. "Now some of that is settling down so that we can all focus on what we want to focus on, which is to make the control better." She adds that standard OS and processor usage also reduces time-to-market for control manufacturers.
Hand-in-Hand Control
In many control environments, Windows-based PCs can be employed to manage non-real-time controls while an ancillary system meets hard real-time requirements.
"There are different economic models between the PC world and the controls world," explains Mario Lento, product specialist with Berkeley Process Control (http://www.berkeleyprocess.com). "We feel that machine builders need to take advantage of ongoing improvements to PC hardware and software, which make those machine systems faster, better and cheaper. Windows allows them to do this, and also enables the tools to work in a manufacturing execution systems environment with a factory interface for remote diagnostics, data logging, recipe control, and other factory communications. Windows, indeed, has a place on a machine tool, however, it's just not well-suited within the machine-critical boundary."
Lento argues that the marriage of machine control hardware to a unique set of PC hardware requires the creation of custom software to define the uniqueness of the set of specific components attached to the specific PC hardware. After defining the hardware, a unique control system has been created. Expanding this system, however, can be a daunting software challenge that will affect the integrity of the custom control system. In addition, it's conceivable, if not likely, that before the custom control system is completed it will be made obsolete by processor developments.
"Typically, when hardware is plugged into the PC's back-plane, every piece of hardware needs to be programmed in its own proprietary language," says Lento. "Then it is up to the application program, which is typically programmed in C++, to schedule all of the devices to execute the programs. Windows is generally not privy to the real-time state of these components plugged into the PC's back-plane, and therefore cannot effectively manage real-time control of mechanisms connected to the controls components."
Therefore, he argues, a real-time control system is needed at the heart of the mechanisms being controlled. "A practical real-time control system should make crucial state information available to Windows, regardless of the specific hardware set," adds Lento. "When machine designers effectively define the machine control boundary, they can take advantage of advances in PC technology."
Linux: Promises to Keep
You keep hearing that advances in Linux for real-time have some machine builders at least testing the waters surrounding that open-standards operating system. We hear it, too, but find precious little industrial evidence to support the claim.
Cost reduction is a major driver behind the Linux movement in general, and that may be fueling interest in the operating system for real-time control, as well.
"Eventually, the cost of running on a highly expensive, customized system is going to become so great that people will switch over," opines Victor Yodaiken, CEO of Finite State Machine Labs (http://www.fsmlabs.com), developer of RTLinux OS for real-time control.
But while potential cost reduction may get customers to kick the tires on the lot, performance remains king of the road in the hard real-time environment.
Yodaiken further argues that the use of Linux for machine control also protects plants from downtime caused by Microsoft-based virus attacks. "All it's going to take is one company with a very expensive factory that's brought down by a virus to make people conscious of how important security is," he cautions.
Linux evangelists say a Linux RTOS is at least as good as any Windows competitor in every regard. "It's interesting that the people who are choosing Linux are probably choosing it for some of the same reasons, conceptually, as those choosing Windows: that is, for interoperability and connectivity--with the exception that Linux is more reliable," says Bill Weinberg of MontaVista Software.
"It's being used because the shop floor is just another set of nodes in a network. Whereas in previous times standard networking was too expensive or considered a luxury or the networking in the front office was highly proprietary compared to the networking on the shop floor, low-cost Ethernet and other bridges to IT-based networks have leveled that out," he says. "And in terms of being able to install IT-based networking, it's a lot easier and more flexible on a Linux-based system because Linux, as did UNIX, grew up in IT networking."
But machine builders wonder whether Linux is appropriate for real-time machine control applications. "We've always been a little shy of Linux because we want a commercial-grade product," notes Asyst's Shieh. "From a development standpoint, I've always been worried that someone's going to get into the source code and then we have yet another branch off of the standard Linux version. And then I have a situation where I can't support it because I've got to bring in someone to understand our particular variety of the OS. I'd rather buy a commercial product and depend on them to fix the bugs and give me the upgrades."
Weinberg counters that MontaVista and others in the Embedded Linux Consortium have made large investments in making the Linux kernel pre-emptible, which permits use of a secondary scheduler that is hard with regard to prioritization. This brings scheduling latencies below a millisecond, he says. Substitute kernels--somewhat analogous to VenturCom Windows extensions--also can improve on stock Linux performance.
The conversation around industrial-strength Linux will continue until either evidence of successful industrial application reaches critical mass, or until developers get tired of the wait and decide to move on.
Jeff Hoch is a freelance journalist specializing in science, environment, and technology issues. He can be reached at
[email protected].
