Implementing a DCS

Most discussions examining distributed control system (DCS) migration projects concentrate on the early planning stages or the potentially contentious issues of vendor selection. Those are important topics, but if a project runs its full course, at some point, the shiny new automation system will arrive at the plant, waiting to be installed and started up. The risk during the entire installation process, particularly the cutover from the old automation system to the new one, is enormous, with the potential for production to be affected either for good or ill. This is where the rubber meets the road.

A well-planned project should have a detailed roadmap for implementation. If your company already is working with a system integrator or an automation system vendor, the steps of the cutover should be thoroughly outlined and scheduled. If they are not, or if you are still planning, let’s look at what must be included in the cutover.

The cutover is the point where the parts of the larger automation system designated to remain in place are moved to the new platform. These typically include field instrumentation, valves, motor controllers, and so on—with all the supporting networks and wiring. These components interface with the system via input/output (I/O) cards, and every connection must be moved from the old platform to the new one.

Steps prior to DCS delivery

Cutover is the final chapter of the project, so, by the time the new DCS is delivered to your facility, this work should have been done: 

• Documentation for the facility or unit reviewed and updated
• Field devices, supporting wiring, and all labeling should be examined, evaluated, and repaired where necessary
• Factory acceptance test (FAT) of the new automation system performed and passed
• Supporting infrastructure for the new automation system in place and working.

These four points cover a lot of ground, so let’s consider their impact individually. 

1. Documentation can make or break a migration project. A successful implementation depends on having a detailed, complete, and accurate picture of the existing facility. When a DCS is replaced, the facility will have gone through many changes and various updates since the older automation system was installed. If the documentation does not reflect all those changes, problems will emerge at many stages of the migration project. If gaps remain after the earlier stages of the project, they will certainly emerge during the cutover when it becomes necessary to find individual connections, one-by-one, and execute connection changes flawlessly.

During the project, the documentation should be reviewed in detail. This includes:

• Piping and instrumentation diagrams (P&IDs)
• Loop sheets
• Panel drawings
• Rack room drawings
• Cable and conduit schedules.

If your company is bringing in a system integrator or other outside help for the cutover, it is critical that the team leader performs an audit of the documentation at some level. The degree of detail will vary according to the situation, but much depends on the documentation, so its accuracy should not be taken for granted. [subhead2]

2. Field devices and their supporting wiring should be verified: 

• All devices are in the location shown on the P&ID
• The devices are working correctly
• They are wired as shown
• The wires are terminated in the marshaling cabinets and I/O card where indicated with correct labels (see Figure). 

3. The FAT will have been completed before the system is shipped to the field, and what happens during this step merits its own article. Suffice it to say, by the time the cabinets and racks reach your facility, the control software and hardware should have undergone extensive examination by the vendor and the project team. If the testing was done correctly, there should be no surprises, such as bad I/O cards or hardware glitches. 

4. Any infrastructure necessary to support the new system should be installed and working by the time the automation system arrives. Any changes in electrical power distribution, HVAC and lighting improvements, holes for wiring and conduit, and even paint on the walls should have been done. These elements can be overlooked as superficial, but in the real world, they turn out to be problematic if they must be done at the last minute. Cutting an extra hole through a cinder-block wall can create a lot of gritty dust, and risks damage to the cabinet, so it should be done before the new equipment is in place. 

After the DCS is delivered

After the new racks and cabinets have been installed and wired to power, it’s time for the site acceptance test (SAT). Some of the same routines checked during the FAT are repeated to make sure there was no damage during shipping, but probably run as spot checks rather than every point. Mostly, it includes basic verification of the hardware and software.

Those who will be involved with the cutover should make sure the I/O cards are in the correct positions, network connections are working, and so forth. Technicians will have to spend time with meters verifying terminations and communications before the cutover begins.

Cutover: Cold versus hot

There are two basic cutover approaches, cold and hot. A cold cutover happens during a shutdown when nothing is running. The old system is disconnected, and all the field devices and other components are then attached to the new system and tested during the outage. When the process restarts, it runs entirely on the new platform.

A hot cutover brings the new platform in while the old one is still operating. Both systems have to run simultaneously and be coordinated so they perform the same control functions. The process continues running while technicians methodically move each device and loop to the new platform, one by one, hence the need for coordinated parallel operation. If done correctly, the unit does not experience any loss of production.

Those sound like clear-enough choices, but how do they work in the real world? Why would a company choose one over the other? In most respects, a cold cutover is the better choice, but naturally there are qualifications.

First, the company must be willing to tolerate a shutdown and schedule it at the appropriate time, or the cutover has to be scheduled to coincide with a shutdown. This means all the elements have to come together at the appointed time. With good planning, this is certainly possible, but few companies are willing to give an open-ended time window. Just as the outage will have a start date, it will also have a finish date and problems could ensue if these dates are missed. Production that will be lost during the shutdown and subsequent startup should be built into the larger financial calculations.

Because cold shutdowns must be completed in a given time window, they become resource intensive. Most companies bring in external resources, which presents an interesting cost calculation. How does the profit from one day’s production compare with the cost to bring in the additional people to get the project done one, two, or however many days sooner? These kinds of hard dollars-and-cents calculations need to enter into the evaluation. 

Some like it hot

When a process can’t be interrupted easily, or where demand makes outages intolerably costly, a hot cutover is attractive since production does not stop. The new DCS is put in place and started up. It is linked to the old system so the two can run in parallel, executing the same commands simultaneously. Technicians then move devices and supported loops from one system to the other while the two systems share the control duties. During the time a device is disconnected for moving, the control room loses view to the instrument, so any function that depends on it must be handled manually. This sounds tricky, and it is for a variety of reasons.

First, the technicians must know exactly what’s happening. The connection for the specific device should be identified positively, de-terminated, and moved to the correct terminals on the new I/O card. In cases where marshaling panels exist, the DCS-side of the terminations can be de-terminated, and new multiconductor cable connected to I/O in the new DCS can be connected.

This is where documentation is extremely important. Technicians holding a disconnected cable don’t want to find the connection to the new terminals aren’t what they thought. There are various ways to make such a move depending on the installation. In easy situations, it can be done in a marshaling cabinet very quickly. The opposite extreme may involve pulling the cable out of one cabinet and moving it so it can be inserted into a new one.

Second, the technicians need to plan exactly what order the moves have to happen. This requires a thorough understanding of the different devices and loops, and how they interact within the process. Again, documentation plays a major role. Detailed ISA-style loop sheets can help make the determinations easier, but the selection requires know-how on the part of the planners involved to make the final determinations. This can’t be executed randomly or hurriedly.

Third, during the time the cable to a field instrument is being relocated, the DCS and control room will lose contact and its data will not be received. Missing the variable from a level instrument on a large tank can probably be tolerated for some minutes because nothing is likely to change quickly. The same can be said for many field instruments in monitoring applications.

The situation gets more complex for devices acting as final control elements performing real-time functions. If the process is generally stable and tends to run in a steady-state condition, a company might be willing to try a hot cutover. If the DCS is constantly making adjustments, an operator may have to go sit next to the disconnected device and relay information to the control room via walkie-talkie: "Open the valve a little more…too much. Go back about half…"

Fourth, while the process may not shut down completely, it is possible that manual control efforts performed by the operators and technicians could result in off-spec product. Naturally the likelihood of this happening and its effects will vary by situation, but users should not assume production will not be affected.

These are cautionary considerations, but there are important positive aspects to a hot cutover in addition to remaining in operation. Since it doesn’t have to happen during a shutdown, scheduling is far easier. The time pressure on making the cutover also may be less, making it less resource intensive than a cold cutover. A smaller group of technicians can carry out the re-terminations following a logical sequence, although these technicians must be very skilled as any mistakes made during a hot cutover can have serious consequences. 

Hybrid cutover: Balancing risk factors

Often, the ultimate factor driving the choice between a cold and hot cutover becomes risk in multiple forms, which include: 

• Risk of missing production goals because an outage has to be extended when the cold cutover takes longer than expected
• For a hot cutover, risk of a safety incident in a hazardous plant during the time control has to be put into manual
• Risk of making off-spec product during the hot cutover if process control is partially lost.

Fortunately, the decision between cold and hot cutover is not necessarily one or the other. Many facilities choose cold cutovers for the reasons discussed. When some system integrators work with clients, this is the assumption unless the migration team asks for a hot cutover. Some clients do want a true hot cutover, but relatively few.

Many companies opt for a hybrid project: the most critical loops and functions are moved during a brief shutdown, accompanied by a period when less critical loops and monitoring functions can be moved hot. This approach still requires a shutdown, so for some companies it isn’t practical. Still, most processing plants can plan an outage in the context of a project of this magnitude.

Naturally, this calls for precise planning and resource flexibility. All the critical functions have to be examined, identified, and staged in the best order. The right people have to be ready to go when the shutdown begins to ensure everything can be moved in the appointed time.

A hybrid cutover offers a best-of-both-worlds approach in many situations, mitigating the risks of a hot cutover, while reducing the costs and resource demands of a cold cutover. Working with the right project partner can help make this type of hybrid cutover run smoothly and ensure all advantages are realized.

Having enough skilled people available when needed for the intense activity during the shutdown has to be balanced with a smaller force doing the preparation and follow-up work. Few companies have the internal resources able to deliver the kind of flexibility to minimize the time of the shutdown and maintain momentum during the follow-up efforts. Bringing in an effective automation partner can make all the difference. 

Brian Batts is director of consulting and solutions at Maverick Technologies, a Rockwell Automation company. Charles Toth is business development manager for Maverick Technologies. 

This article appears in the Applied Automation supplement for Control Engineering and Plant Engineering.

– See other articles from the supplement below.

Do you have experience and expertise with the topics mentioned in this content? You should consider contributing to our CFE Media editorial team and getting the recognition you and your company deserve. Click here to start this process.