Effective process control system migration, Part 2: Open standards help

System integrators often help with effective process control system migrations and open systems make the process easier. The Control Engineering webcast, “Effective Process Control Migration,” provided advice as many distributed control systems (DCS) and supervisory control and data acquisition (SCADA) systems have reached the end of useful life. Replacement process control systems need to be secure, reliable, and safe. Unsupported or outdated systems need to be migrated to newer, supported, more capable process control system technologies. What are desirable traits and advantages of a modern process control system? What differs about migration projects compared to a few years ago? Before starting a DCS to PCS migration examine these lessons learned.

Real-time artificial intelligence, linear and nonlinear model predictive control and real-time optimization

Don Bartusiak is president of Collaborative Systems Integration and co-chair of the Open Process Automation Forum. In October 2020, he retired as chief engineer, process control for Exxon Mobile Research and Engineering with 33 years of experience. Previously, he was a research engineer for Bethlehem Steel. At Exxon Mobile, he implemented real-time artificial intelligence, linear and nonlinear model predictive control and real-time optimization applications. Bartusiak held supervisory or senior technical positions responsible for instrumentation, process analyzers, control systems and control applications. From 2000 to 2002, he was an adjunct professor at Rice University. Bartusiak received a bachelor of science from University of Pennsylvania and a master of science and PhD degree from Lehigh University. He is co-inventor on five patents.

In-the-trenches view of DCS migration

From an engineering and operations perspective, Bartusiak offers an “in-the-trenches view on what happens during DCS migrations.” His comments and advice follow covering the current state of the art of DCS and migrations, and what is possible for DCS and process control systems in the future. That is, “What do migrations look like now?” and “What is motivating us to do better than what we can do now?”

DCS migrations are typically acts of necessity motivated by obsolescence. We always try to justify DCS migration projects with a return-on-investment (ROI) angle. It’s often tough to do that when with an in-kind replacement that doesn’t enable innovations developed since the last DCS design.

In-kind replacement of a DCS basically means taking currently available state of the art and either sticking with the same supplier and upgrading to a new version of that supplier’s product, or, in some cases, switching from one supplier to another. The typical phrase used for that scenario is competitive replacement.

Company procurement strategy may determine if a company continues with the current supplier or switch to another. Those involved may have preferences, a preferred-supplier short list, be required to competitively bid, or single-source the same supplier. Procurement strategies can affect the tactics of a DCS migration. Considering principles of reliability engineering, “Do I replace in-kind, or do I go after an improvement?” Improvements in DCS migration include open architectures.

DCS migration: On process or a cold cut over during a shutdown?

A DCS migration can happen during operation, on process, or cut over done during a shutdown, also known as a cold cutover. Industry type often determines the most-common approach.

If a manufacturing unit, for whatever reasons, is shut down from time to time, then there’s an opportunity for a cold cutover. Some companies, especially Japanese companies, shut down for routine safety inspections every year, even in units like refineries and big petrochemical plants, allowing for a cold cut over.

For continuous manufacturing operations that go years without a planned shutdown, a hot-cut-over strategy is more likely.

The typical argument is that, with a hot-cutover scenario, risks are divided into smaller pieces. With smaller parts, errors are smaller, such as a botched a loop transfer, or an initial error in the new control strategy. Those are small risks compared to the risk of having to extend the downtime or risking a big unplanned production loss because of delays in startups after a massive replacement project during a turnaround.

Figure 1: Typical DCS migration costs for a hot cutover are 20% for new hardware and software licenses, about 40% for engineering and construction services and about 40% for project management, according to Don Bartusiak, president of Collaborative Systems Integration and co-chair of the Open Process Automation Forum. Courtesy: Control Engineering webcast: Effective process control system migration

Today’s DCS designs are reliable, tightly integrated

Currently available industrial control systems have been honed over decades and are reliable. They deliver high availability to customers. Today’s DCSs also are very tightly integrated, and one benefit is relative ease of use of the system. When engineers build a tag, whether they want to use that tag in the control strategy or put data into the human-machine interface (HMI) or historize data, it’s relatively easy to do. Those are among benefits of tight integration.

Negative attributes of today’s DCS designs

Today’s DCS designs do have some negative attributes.

These include big costs. Often a DCS has proprietary interfaces and communications mechanisms, with limited interoperability between systems sourced from different suppliers. DCS design limitations impede the business value captured by end users. There are barriers to entry for innovations from third-party suppliers, in hardware or software. DCS designs make it difficult to implement, integrate, and update those solutions within the boundary of the DCS design.

When a component piece becomes obsolete, the downside of the tight integration often means the whole system [or large portions] must be replaced. In the case of computing nodes, the advantages of Moore’s Law are lost, because new computing technology cannot be easily integrated inside the boundaries of the DCS without the supplier doing it. The other unintended consequence of tight DCS integration, because DCS platforms differ among suppliers, any upgrades within one supplier’s generation of products may require that the end user rewrite control strategies, reprogram and reconfigure, simply because proprietary language has changed between suppliers or even from one supplier’s generation of products to another.

Another attribute of the current state of the art is that most designs of today originated before cybersecurity was the threat that it is today. The cybersecurity risk gap must be closed, requiring improve ability to introduce cybersecurity technologies into the industrial control system.

See inside the real life of a DCS migration project

A DCS migration requires planning and project management techniques with a DCS migration. Many in-the-trenches experiences show how a DCS migration can stray from agreed-upon project management plans. Watch out for these areas during DCS migration.

1. Migration gateway details: During a hot-cutover scenario, where the old system and the new system are going to co-exist for an extended period of time, one of the things that can be done is to introduce a temporary migration gateway device. A temporary migration gateway device enables the old and the new system to inter-operate for a period of time allowing for a staged movement of controls from the old system to the new system. Many devils are in the details about what goes into these temporary migration gateway types of devices, including cooperation of the intellectual property owner.
2. Take time to plan ahead for innovation: During the design of the applications in the target system, be aware that if project management talent is brought into the project here, the frame of reference will likely be to minimize changes and control changes [with timing and cost among considerations]. The tendency is to adopt a venture philosophy that says the new applications are going to look exactly the same as the old applications. The downside of that is that only using capabilities of the prior DCS can forego a lot of value to do the job better, faster or cheaper by taking advantage of the target system’s capabilities. I’ve seen projects, head-to-head, apples-to-apples comparisons of projects that have taken what I call a transliterate approach. That’s where the venture philosophy is that, the new applications should be exactly the same as the old, versus a translate approach, which takes advantage of new system capabilities. Watch out for this trap, which can be worse depending on the staffing and resources on the migration team.
3. Modern HMI design: HMI graphics need to be designed in the target system. Operators have a lot of ownership on the HMI schematics, so it’s important to get them involved during the planning stage as new schematics are proposed. High-performance HMIs have advantages that help all involved. HMIs also are what management sees. Management might not always understand all the arcane details that control system experts do, but graphics catch management’s attention for a quick reaction. A clean modern HMI is an outward representation of the unseen math, computer science and other wizardry that goes into control system design. Operators take great ownership of HMI design, so it’s an important matter.
4. Operator console migration: Also design the logistics for operator console migration. This is probably less important now that in the old days when the operator console was a masterwork of stainless-steel fabrication and a really big deal. New style operator consoles tend to be almost a self-contained stand-up system, with sound curtains and wider visibility compared to the old days.
5. Input/output (I/O) system cutover logistics: I/O system part of a DCS migration is perhaps the most difficult and most costly part of a migration project. Often the constraint is the availability of space in the equipment room. I’ve been on projects that have had to build new equipment rooms just to enable the cutover. Luckier projects have enough space in the equipment room to establish an initial footprint for the new I/O, since I/O density has gotten higher over the years, a gain in real estate is likely during cut over and progressive demolition of the old I/O kit during installation of new I/O. I/O transition can be some of the most-costly stuff in the project. [Most I/O connections do not have updated documentation.]
6. Migration execution resource plan: DCS migration will have instrumentation teams and systems teams to do the computer work. Applications engineers are probably doing most of the work, and there’s good chance that operator consoles will require double staffing during hot-cutover periods. Execution discussion as part of the planning exercise, can be pretty quick, but needs details on installing the new operator consoles stations, and day-to-day work, such as building new tags, building applications, new HMI graphics in the target system, and a loop-by-loop plan for a hot-cutover scenario, transferring I/O from the old to the new system and connecting the I/O to the tags and applications built in the target system.
7. Commissioning and testing: Commissioning new applications, frequently, includes tuning and vetting to ensure they work properly.
8. Decommissioning: After cut-over and a period of reliable operation, the old system needs to be decommissioned.

DCS migration can take 4 or more years, tens of millions of dollars

Depending on system size, DCS hot cutovers can take years. In Figure 1, the pie graph shows an accurate, but not precise, semi-quantitative breakdown of the cost distribution for a hot cutover DCS migration. In the neighborhood of 20% of the cost will be for new hardware and software licenses. About 40% will be for engineering and construction services and about 40% will be for project management.

A company’s policies and practices can move the line between what is called project management and what is called engineering. The basic takeaway is that, the majority of DCS migration costs are going to be in the work hours of the operating company or for contractors to do this work.

Based on personal experiences in many hot cutover projects, ranging in size from 1,000 to 10,000 I/O, from the planning and project justification stage through decommissioning, DCS migration can take two to four or so years. DCS migration cost can be millions to tens of millions of dollars. Hardware and software costs are a minor portion. Instrumentation systems and applications engineering costs are a function of the I/O count, and the amount of application rewriting required. Project management costs are largely a function of project duration.

Future DCS migrations can be significantly easier

Re-imagining DCS migration requires moving past incremental changes of in-kind replacements. A step-change DCS migration will bring significantly more benefits by moving toward open process automation, an initiative underway more than four years by bringing open architecture principles to the industrial control marketplace. In contrast to the Purdue model that we use that reflects the reference architecture or design principles of today’s DCS designs, the second figure depicts the reference architecture of open process automation.

Three basic elements differentiate an open process automation system from a traditional DCS.

1. Edge devices, which are called distributed control nodes, or DCNs are I/O connections exist. This is where you do the first touch of computing in a DCS migration, so to map it to the Purdue model. These devices for Purdue model level 1 and level 2 types of applications.
2. Open Process Automation Forum’s OPA connectivity framework, in contrast to currently available products, provides an industry standard networking technology for communications among nodes and with other parts of the enterprise.
3. The advanced compute platform is the third aspect of the open process automation reference architecture. Think of it as the platform for Purdue model level 2 and 3 types of applications to use the types of innovations developed in information technology and data center technology, scaled down to the low latency, high availability requirements of a control system for on-premise location.

Open control design to resolve pain points, reduce cost, address cybersecurity

Open process automation architecture resolves pain points and provides incentives for new business value generation. With open architecture, it’s easier to introduce and adopt innovations in software or hardware technologies much more rapidly than with traditional DCS design. Interfaces in the open architecture will be defined in an industry standard manner, for more frequent introductions of new value-adding technologies than with today’s DCS designs.

Open process automation architectures reduce total cost of ownership because open systems more readily allow upgrades or replacements at the component level, rather than the whole system level. This will radically change how migrations and upgrades get done.

Open process automation architectures build in mechanisms for adaptable cybersecurity to be added and incorporated into component products that comprise an OPA standard-based system.

To clarify, these comments and efforts are not bashing the suppliers in the DCS space. Control systems have gotten incrementally better since the 1980s. There was a step change in innovation that marked the transition from single loop electronic controls in the 1970s to the DCS designs of the 1980s. Since then, it’s been largely incremental progress.

DCS migrations haven’t changed much in a material way over the last three to four decades. Operating companies have generated the most new value by using applications delivered in the Purdue model level 3, really above the control system. Comparing the rate of value creation and cost reduction experienced by the tech industry and what we experience as consumers and users of consumer electronics, the contrast is quite stark.

Those using industrial automation and controls cannot be doing the same things over and over and expect different results. To change the game for control system upgrades, the time has come to step back and take a different approach to allow a large step up.

Open process automation doesn’t only exist in slides. Many smart people and companies have been working on the open process automation standard for years. Open process automation is being demonstrated now. Soon operating companies start writing bid specifications to and using OPA types of technologies.

Figure 2: Open architecture design from Open Process Automation reference architecture adds to value creation, reduces total cost of ownership and presents a safe secure design, among other benefits, explained Don Bartusiak, president of Collaborative Systems Integration and co-chair of the Open Process Automation Forum. Courtesy: Control Engineering webcast: Effective process control system migration

More information on the OPAS Standard

The OPAS Standard, Version 2.1 Preliminary Standard published May 17, 2021, “once fully defined, will allow for construction of safe, reliable, secure process automation systems that are scalable from very small to very large, which do not require system shutdown to perform updates and extensions, and which can be applied to existing systems and to new construction,” according to The Open Group Open Process Automation Forum.

Don Bartusiak is president of Collaborative Systems Integration and co-chair of the Open Process Automation Forum. Edited by Mark T. Hoske, content manager, Control Engineering, CFE Media and Technology, mhoske@cfemedia.com.

KEYWORDS: Process control system migration, open process automation

CONSIDER THIS

Will your next process control system migration be to an open system?

ONLINE extra

RCEP/PDH webcast, available until March 23, 2022: Effective process control system migration https://www.controleng.com/webcasts/effective-process-control-system-migration/

Effective process control system migration, Part 1: Planning advice

Effective process control system migration, Part 2: Open standards help

Effective process control system migration, Part 3: Poll results, answers

Effective process control system migration: More answers https://www.controleng.com/articles/effective-process-control-system-migration-more-answers

Do you have experience and expertise with the topics mentioned in this content? You should consider contributing to our CFE Media editorial team and getting the recognition you and your company deserve. Click here to start this process.