Assisted living for your aging DCS
Has your control system moved past the point where you can depend on it? Understand the signs that it’s time to retire your platform before it’s too late.
A process control system is supposed to help a plant or process unit operate with a minimum of human intervention. After all, that's the whole idea behind automation. The problem these days is there are many old systems still running after 20 or even 30+ years, and sooner or later these can become troublesome to the point that the level of attention they demand is disproportionate to the service they perform. Let's compare it to owning a car. Maybe you really like your 1992 Chevrolet Impala, but after 22 years and 250,000 miles, it requires a lot of attention. Brakes, front end, shocks, transmission, and maybe even a whole new engine. At some point it's time to trade it in.
That's fine if all you're talking about is a standard passenger car. But let's say your vehicle is a van that has been outfitted to carry lots of specialized equipment for your business. Buying a new van means transferring all that shelving and racks, and making many other modifications. In that case, the decision to trade in isn't so easy because of other costs involved. It's a different situation with different economic considerations.
In either scenario, the last thing you want is the vehicle to break down 400 miles from your home. Given your vehicle's age and performance history, you know that anytime you go on the road there could be a failure, so you calculate the risk before you drive it on the trip. If it does break down, you could be stuck having to make decisions under pressure and may have to consider a solution that is not optimal simply for the sake of expedience.
The risks of keeping an old system
Every day that an obsolete DCS platform controls a plant, the risk increases that it will suffer a failure and interrupt production. Of course that could also happen with a brand-new platform, but risk grows with age.
For purposes of this discussion, let's restrict the scope to digital control systems. No panel boards, pneumatics, electronic analog, or other subsets of those. Most of what we're talking about would be considered second-generation DCS platforms installed from the late 1970s through the advent of MS Windows-based systems and their broad deployment in the late 1990s. Few if any of these systems are truly "original" since hardware fails in a variety of ways depending on what it is. The devices with the shortest lifespan are generally in the operator interface. No control system that has been used every day for 20+ years still has all its original keyboards, displays, and hard-disk drives.
The parts that seem to go on forever are basic controllers, I/O, and field wiring, but even these reach an end. Operators may see individual field devices go dark because an I/O card has lost a channel or perhaps all the devices connected through a specific card. Hopefully, there is a replacement, but one of these days all the spare-part inventory will be gone, and the boards sent in for repair will be returned as they were with the regrets of the OEM. Maintenance then will be forced to work with operators to perform system triage to make sure the most critical elements are still functioning while others fall into disrepair. Eventually, users have to start combing the Internet for hoarders, parts recyclers, and maybe even try eBay. (A recent eBay search on "Bailey Infi90" yielded 495 items. A search on "Moore APACS" yielded 186 items. Both turned up various boards, modules, and accessories.)
Is system age a relevant issue?
The age of a control system is not necessarily a problem in itself. An old DCS that is matched well with the process and has been carefully maintained over its life can do an excellent job. Like the specialized van, the user may have spent a great deal of time and effort getting it configured just right and optimizing the control methodology with specialized programming. Old platforms are capable of very sophisticated functionality, although there are fewer programming tools compared to more modern versions. Operator interfaces can be updated, at least to some extent, so new capabilities can be added.
Intellectual property tied to the process and developed over many years may be difficult to move to a new platform, so a company may decide to keep the old system and maintain it scrupulously. It will stockpile critical parts to keep it operational, but it will also have an exit strategy for when the time comes that a change is necessary. This sense of intentionality is what differentiates it from a company that simply gives in to inertia and doesn't want to make an effort to upgrade.
Other users make improvements incrementally to reduce their risk profile and minimize a technology gap with more current systems. "A significant number of our customers are making continuous investments in their DCS architectures," says Sean Sims, vice president of lifecycle services for process systems and solutions at Emerson Process Management. "They see, and have realized, the operational value and overall lifecycle cost benefits of maintaining current levels of technology within their facilities. Some choose to invest in small, incremental steps whilst the facility is fully operational, whilst some choose to align their investment timing with major maintenance events such as a plant turnaround. Whatever the timing strategy, deliberate incremental investment results in reduced exposure to component and technology obsolescence."
Some older systems may have reached their capability limits. If a plant has reached the maximum number of I/O points a platform can handle, there aren't many options for adding some new piece of equipment to the process that has another dozen or so field devices.
The Windows invasion
Control system architecture took a much different direction about 15 years ago as control system vendors began to shift from proprietary hardware and software to MS Windows-based platforms using more off-the-shelf equipment. Some vendors considered that an invasion where they lost control of their own destiny and suddenly found themselves having to follow Microsoft. Over subsequent years, owners of these proprietary systems often found the easiest way to upgrade was to "bolt on" a new system that took advantage of Windows' flexibility. Control systems that have been untouched in this way still exist, but there aren't many left.
In more extreme situations where a company is trying to preserve a legacy system that moved into the no-longer-supported category, users resort to putting on what is, in effect, a third-party SCADA system on top of the DCS to support a user interface, extend the system's capability to communicate with other equipment using Windows or OPC, and move data throughout the system. This approach requires much integration effort, but in a situation where a company insists on running a platform well beyond its useful life, it may be the only alternative.
Migration vs. emergency retrofit
Ultimately, any control system will have to be phased out. It will get to the point where failures overpower the ability to repair or replace critical parts. Migrations can be easy or challenging to deal with depending on the level of planning and preparation, but when systems are failing and much of the plant is being run manually, moving to a new platform is truly nightmarish.
As Rich Clark, principal applications consultant for Honeywell Process Solutions, points out, it is possible to reach a point where it's effectively too late to migrate because the human resources needed to carry out that program are totally engaged simply keeping the plant running. His advice is to plan while you still have options. "There are periods where you look at your risk curve and your opportunity curve, and both are pretty flat. You have plenty of options at that point. But eventually you reach a tipping point, and it's hard to know where that is precisely. That's when your options become limited and you have to do things in a hurry. If you're the person managing that system, you have to set some red lines and some yellow lines, and look at your system at least once a year and see where you are. If you don't, you can pass the tipping point and suddenly you're dealing with a cascade of events that forces you to do a migration project in a way that has a higher cost and greater risk. You're trying to do it on a compressed time schedule using people that have other jobs, and that pressure creates more risk.
"In some refining and petrochemical environments, you may have an outage or an opportunity to migrate once every five years. So when you're doing your planning, you have to ask yourself, is my system maintainable right now? You might say yes, but there is a non-zero chance that five years from now it might not be. So there is an opportunity to migrate two years from now or wait seven years. If you're planning, you have to take those kinds of things into account. We try to help our customers understand their real risk tolerance and help them identify opportunities."
In addition to human constraints, companies trying to make a migration into a crash program may find other unpleasant surprises:
- Documentation might be largely non-existent or has not been kept up-to-date.
- Critical information and programming may be on old memory media (e.g., floppy disks, tape drives) that aren't readable anymore.
- Key individuals who were involved in system design are long gone.
- Companies that allow a control system to continue to run past its end-of-life often let many parts of the plant deteriorate similarly, so launching one upgrade program may open a complex can of worms extending into other functional areas.