Upgrading Control: Migration or Evolution?
At a recent automation supplier user group, a speaker cited an interesting statistic. He said that 50% of the DCS (distributed control system) platforms running process plants today are at least 20 years old. Knowing that, it isn’t hard to understand why control system upgrades are on many peoples’ minds as companies face growing cost and competitive pressures.
Should you be looking at replacing your control system simply because it’s in the half that’s more than 20 years old? Not necessarily, any more than you might want to think about trading in your 1987 car. Your aging DCS may regulate your process perfectly well, but like an old car, it can become maintenance-intensive and certainly lacks some of the capabilities of newer designs. Even if you decide those new capabilities aren’t all that important, eventually parts become harder to find, and keeping the system going gets costly.
While age-related problems are certainly a compelling argument for upgrading a system, functionality issues can also enter into the discussion. Even newer systems may not have some of the functions that could create value for your business.
There are two main drivers for control system upgrades: obsolescence and functionality needs.
The first and most powerful driver that pushes change is obsolescence. When a platform is failing and the original provider no longer supports it, the risks of leaving it in place become too high. Obtaining replacement hardware eventually becomes problematic, leaving users to depend on parts recyclers and eBay. If part of a system goes down and there are no replacements, a whole process unit can grind to a halt.
Still, some plant managers have to learn the hard way. “It’s very difficult to build a business case based only on obsolescence,” says Mike Vernak, manager for Rockwell Automation’s legacy DCS migration program. “Plant managers tell their plant engineers all the time, ‘If your system’s working and we’re not experiencing any downtime, I can’t justify a capital expenditure based on obsolescence.’ Of course, if you are experiencing downtime, the ROI is easy to calculate. But it’s hard to calculate it based on no downtime.”
While unreliable performance is the most obvious problem, moving toward obsolescence can have more subtle effects, including knowledge erosion. “Companies are finding it difficult to maintain technical expertise for these 20- or 25-year-old legacy systems,” Vernak adds. “The OEMs especially don’t have expertise, either because of attrition or retirements. Kids coming out of colleges today that are 22 or 24 years old have absolutely no idea about that technology. I hear things like, ‘That system is older than I am. They didn’t teach me this in college.’ They don’t have a clue how to work on it. It’s also expensive to maintain for training. I’ve seen companies that have multiple disparate DCS and PLC systems in a single plant. I can’t imagine how much cost that must be every year for the plant manager to maintain multiple folks for multiple control systems.”
The second driver that pulls change is functionality. A platform does not have to be all that old to lack specific functions that might be very helpful. For example, a system that does not have the I/O capability to handle HART diagnostics will probably not be able to support an asset management program. Others may not have the connectivity to support growing integration with enterprise level systems.
John Murray, ABB global business development manager for control systems evolution, says his group did a survey on what motivates companies to upgrade control platforms. “It surprised us that a very significant number, over 50%, said improving operator performance or maintenance practices were reasons that they were considering upgrades. We see that a lot today, as people recognize that’s where they have to get the step improvement in their business. The term is overused but it’s really about operational excellence. Solutions like asset management, information management, and data mining can really give you a competitive advantage if used properly. What we call a traditional control system has to be enhanced with these capabilities,” he says.
Vernak agrees with that assessment and adds: “Most customers are migrating not because of obsolescence, but because of lack of performance from their systems. It doesn’t necessarily mean it isn’t controlling the process. It means that they aren’t able to get data that they need out of the system to be able to make better decisions.
“Because of the proprietary nature of the system, it’s hard to get data out or to be selective with the data they can get. These systems do not interface well with modern IT systems, with modern historians, etc. Many aren’t capable of connecting to expert systems that allow them to do advanced process control easily or cost effectively. It’s a lack of functionality.”
Older systems were designed to function in greater isolation, so cyber security is often rudimentary or nonexistent. “A lot of people think, wrongly, that legacy systems are safe because they’re legacy systems,” says Ken Keiser, Siemens migration marketing manager. “When it was first installed, that system might have been completely isolated. But as time goes on, maybe someone put in a modem line somewhere, or a link to an intermediate historian and that historian happens to be on the Internet where they didn’t think about the intermediate connection. You never know how something is connected after a 20-year period of time.”
Justifying your needs
The first step in any upgrade project is figuring out what you need, in as specific terms as possible. Some answers may be very obvious but others that are equally important may require some probing. One of the downsides of remaining with an old system is that you don’t realize how technology has progressed. The result is that you may not think to ask for some types of functions because you don’t know that they are even available.
“Everybody wants to improve profit,” says Murray. “But you have to work with customers to determine how they want to achieve that. The better a customer understands what problem he wants to solve, very specifically, we will come up with a much better solution. Whether it’s a new feature, new capability, or just adding controllers to provide better availability, once they understand that, they get a real sense of what those improvements are going to be. Then they can quantify that financially in terms of benefits they’ll derive or cost reductions, and that helps the cost justification process.
“If you have some grandiose target or scheme, then you’re really scratching for ‘how do I justify these expenditures?’ It’s important to drill down into the details for everybody’s sake to understand the problem and get the best solution.”
No plan is complete without a financial justification. Some situations will be very clear and direct. Replacing an old system that is waiting to fail and cannot be repaired should not be difficult to sell to management, particularly if its failure will halt production.
The difficult part may be convincing those responsible that the failure could be imminent. More subtle changes can be challenging. Your ability to place a value on an upgrade that adds some new functionality could depend on the extent to which you are personally convinced.
Marjorie Ochsner, senior product manager, DCS platform migrations, for Honeywell Process Solutions, offers some useful questions to ask yourself early in the process:
What’s your cost of doing nothing, and what are the expected benefits of doing the migration?
How much is that unplanned shutdown costing you?
What is your actual cost of maintenance for this older system?
How much are you paying for that replacement hardware at the parts recycler or eBay?
What is the cost of inadequate response?
Then there’s the positive side: What are the added benefits of a new system? What’s the value of renewed operator effectiveness? Are there advanced control strategies that you can bring in? What is the value of bringing in those HART signals from the field? “We look at the answers to those questions and have a very detailed analysis that we help our customers with,” says Ochsner.
How drastic a change?
Control platform upgrades can range from small incremental changes to full rip-and-replace projects. Typically, the older the system, the more drastic the change. Older platforms were more integrated and not designed with open architecture which makes changing one part of the system more difficult. More modern approaches tend to be modular, allowing for evolutionary improvements.
Frequently companies begin with the HMI, and it’s more than just about graphics. The HMI is the main connection point for the data pipeline, providing an interface to higher level systems and a mechanism for more advanced control strategies.
“The HMI is the component in a DCS that is made obsolete first, due to internal or external forces,” says Siemens’ Keiser. “When customers approach us, they are coming because they have a specific problem with the control system. It may be a problem in the controller, but more likely it is a problem viewing the data in the controller, so it’s really an HMI problem.”
Keiser says an HMI is not all that difficult or expensive to replace, so it’s a relatively easy migration to make for a first step. “You have a new look and feel, but the same exact equipment beneath it. As far as the process is concerned, it has not changed, the process control has not changed, so all of that equipment, knowledge and know-how is the same,” he says.
In some cases, a migration may involve a newer system from the same supplier, assuming that company still exists in a recognizable form. Other times it may be a more drastic change, moving to a completely different platform.
“When you get to the point that a system can’t meet your business needs, a more radical solution may be the only answer,” says Mark Bitto, ABB global business development manager, control systems evolution. “But even there, you should still look at what kind of investments you can protect. Can I leave the wiring in place? Can I leave the terminations in place? What costs can I eliminate best through whatever solution I have?”
A full change is traumatic in many respects, and companies should not take it lightly.
“There has to be a very critical reason why customers don’t want to proceed down an incremental migration path,” says Keiser. “They’re saying, ‘We’re done with this vendor. We need to see what’s out there.’ In one case I can think of, the customer was considering a stepwise migration, but the more they looked at, they realized they had to take the big step. Smaller companies are more likely to do that. Some huge chemical company might not, preferring a lower-risk approach.”
No time to shut down
Still, the extent of a change may be limited by production requirements for a plant. Full rip-and-replace projects invariably involve some downtime, whether it’s planned or not. A more incremental change can reduce this risk.
“Shutting down a process is often more expensive than keeping it going while you’re doing the upgrade,” Honeywell’s Ochsner suggests. “So most of the time owners do it incrementally on a unit-by-unit or controller-by-controller basis, laying the foundation with the networking and everything else so that they make a fairly smooth cut-over.
“It helps to do things from the top of the architecture on down, replacing the HMI first, then the networking, then the controllers,” she adds.
Planning, executing changes
Once you have decided to proceed down a specific path, it’s time to lay out the details. “The first step is always planning,” advises Oschsner. “You’ve got to know what you have, upfront.”
The first thing Honeywell looks at is the state of documentation, “which, sadly, is not usually in the greatest shape,” says Oschsner. “We want accurate and complete documentation upfront, even to the point of knowing what model numbers of controllers and I/O you have in the field.”
Graham Bennett, migration consultant for Invensys Process Systems has been involved with many projects and offers his suggestion for sharing responsibilities with the customer.
“We compile a work activities breakdown as a standard procedure,” Bennett says. “It lists pre-migration, migration, and post-migration items. We have action items that leave no stone unturned. Everybody has items and we follow it to the letter. One of the things is to confirm functionality of any third-party protocol.”
The ultimate benefits of a well-thought-out and well-executed upgrade project can be huge. Improvements of control capabilities, data mining, operator interface, feedstock utilization, product quality, and many other areas are all possible. But all of them depend on an intimate knowledge of your process, combined with effective analysis and planning.
|Peter Welander is process industries editor. Reach him at PWelander@cfemedia.com .|
Mining/chemicals case history: Uniting disparate systems
Teck Cominco is a large mining, metals, and chemical company based in Vancouver, Canada. Its Trail facility in central British Columbia, includes one of the world’s largest fully integrated zinc and lead smelting and refining complexes, including the Waneta hydroelectric dam and transmission system. Trail’s metallurgical operations produce refined zinc and lead along with a variety of specialty metals, chemicals, and fertilizer products. The Waneta dam provides power for the facility, local customers, and the U.S.
Rob Zwick is superintendent of process control, and he has been involved in an effort to move the entire facility to one common platform. As he describes it, “We have five major plants. We ended up with five platforms: ABB, Fisher Provox, Honeywell TDC 2000, Foxboro, and a PLC/Wonderware. One of the things we’ve been working toward is consolidating on Foxboro [I/A Series control system from Invensys Process Systems] as a common platform as best we can.
“We have migrated away from the Honeywell. We’re migrating away from the Fisher Provox, and by the end of this year, the Foxboro system will handle about 75% of our I/O,” Zwick says. “All our plants are coming to an identical platform with Foxboro hardware and software.”
Maintenance and training problems were a major driver toward convergence. Parts availability for the oldest platforms had been pushing a change, as the company cannot risk unscheduled downtime. Of course the pressure to maintain uninterrupted production has not made the migration efforts any easier.
“We’re on our fourth migration now,” Zwick notes. “Our operations certainly run 24/7, and all the plants are inextricably linked, so we have to coordinate shutdowns between plants very carefully. We do have regular maintenance shutdowns for a couple hours here or there on a monthly basis, and we look for those to do the migration.”
Making the decision
During the evaluation phase, Zwick and his colleagues considered many possibilities and came to some interesting conclusions. “Thinking about the platforms that I’ve been exposed to, the functionality is really becoming hard to distinguish,” he reflects. “So, to justify a choice on technical advantage of one DCS over another is actually getting pretty difficult.”
All companies have been working at improving and modernizing their hardware and software, Zwick says. “Foxboro has a good installed base with mining and metals in Canada, whereas others don’t. Others have done better in oil and gas. It strikes me that differences are more historical than any advantages of features in a particular industry sector,” he says.
One of the things that has helped facilitate the changeovers has been special Foxboro I/O cards that provide an interface between the old and new system. Graham Bennett, migration consultant for Invensys Process Systems describes the approach:
“We retain all the field wiring up to and including the marshalling cabinets, and then the interface cabling between the marshalling cabinets and the actual I/O of the legacy vendor. We take the connection off and replace the card one-for-one using a new card made to the form factor of the legacy vendor,” he says.
“Having the one-for-one lineup, there isn’t any configuration change needed, and every point on the I/O from the field lines up with the original card layout,” Bennett adds.
Zwick is getting ready for his next major move. “It’s a big one,” he says. “We’ll have a 12-hour window, but it’s 4,000 I/O points, so we’ll need all of it. Downtime on this plant is $100,000 per hour,” he says.
Utilities case history: Choosing the right tool
Colorado Springs Utilities is a municipally owned utility group providing electric power, gas, water, and wastewater services to residences and businesses. The electric power division operates a small fleet of coal-fired units, several hydroelectric plants, and is a co-owner of a gas-fired combined cycle turbine plant. Additionally it operates the 54 MW George Birdsall plant, with a gas-fired conventional boiler for 12-hour emergency and peaking capacity.
The Birdsall plant was built in the mid-1950s and still has its original boiler and turbine. Consequently, it lacks the efficiency of newer plants and no longer operates as base-loaded capacity for cost reasons. However, with growing power demand in the area, its operating time is increasing each year. Brent Richardson is plant manager, and recently brought the facility through a control system migration.
The plant had been running with a Rosemount RS3 DCS since about 1991, but Richardson reported that they decided to look for an alternative when he received word that the platform was into its final decade of OEM support. Given that Birdsall is not a very large plant with fewer than 150 I/O points and straightforward control strategy, Richardson and his colleagues decided to explore a variety of options.
“We were familiar and comfortable with DCS, for its power and speed,” says Richardson. “We first set out to look at DCS platforms, including Delta V. Our Ray Nixon generating plant has an ABB system, so we considered that. We also looked at our water department. It is doing all its process control with Allen-Bradley ControLogix PLCs. Once we looked at the power and speed of the new PLC based system, we found that it was actually about 10 times faster than what we were controlling the units with right now.”
The idea of sharing the same system as the water department had considerable appeal, and serves as a prototype for retrofits that are anticipated at the hydroelectric plants within the next four years.
“There were numerous systems that could have done the job,” Richardson adds. “We want a system that can control our entire fleet, including the hydro plants. The PLC system gives us what we need here, and will integrate with all our other facilities. As soon as I put in my system, I had the whole intelligence of the water department on the ControLogix platform, since they were standardized on it. I can use their resources, including five or six trained I&C technicians and even spare parts. I’m no longer stuck on a stand-alone platform. Having those resources available brought huge value to us.”