Managing automation upgrades, retrofits
Upgrades can be tricky, especially in automation systems. They come in many varieties, each with unique challenges. Sorting through a plethora of technical and operational issues while keeping budgets and schedules under control is a tough course to navigate, but great outcomes are more than possible. When pursuing an upgrade, consider these guidelines.
Form and function
Not all upgrades are created equal. Some can be done with simple swaps with little to no impact on operations. Others fundamentally alter the composition and behavior of the system, requiring extensive installation, training, and monitoring to complete. Usually, upgrades entail some change to the physical or logical structure of the system. New application functionality is often thrown in the mix. First, consider the different aspects of an automation upgrade before exploring the best approach for managing each.
The most common type of migration is minor version/model upgrades to the hardware or software components in a system. This may include physical replacement of components, firmware updates to these items, or service-pack updates to commercial off-the-shelf (COTS) software. This class of retrofit typically requires little-to-no application software rework. Other than the upgraded items, the remainder of the physical automation infrastructure usually remains intact.
All systems are based on some hardware and software technology platforms. Major upgrades to these platforms—even when staying with the same manufacturer—usually entail some amount of application software rework. While such changes might be limited to simple configuration and scripting changes, they could require complete translations of the application software. Translation tools and prescribed upgrade paths may exist to make this easier, but the entire application will always need to be completely retested to ensure proper operation.
Application software changes often accompany automation migrations. These changes may be associated with new processing equipment, new ways of interacting with existing equipment, new analysis of operations, and new ways to share and report on everything. While the focus of this article is functional application changes in the context of larger automation upgrades, it is important to note that such changes often take place absent of any other factors.
Just because a minor component upgrade is the simplest type of retrofit does not necessarily mean it is simple. Subtle differences between components can cause major headaches if you are not careful. Consider these guidelines when migrating components.
Upgrading controller processors and modules is a common type of migration (see Figure 1). Upgrades within the same product line often do not require application programming changes, or at worst, they may need slight configuration changes. An important factor to watch for is processor firmware changes that might affect the behavior of certain functions. For example, changes to a proportional-integral-derivative (PID) control algorithm may require reconfiguration of the PID tuning parameters. While changes such as these are often documented by the manufacturer, that is not always the case. Thoroughly testing and monitoring your system after this type of upgrade should be part of your planning.
Upgrading control panel components is another common type of migration you are likely to encounter. Traditional electrical/pneumatic component swaps are fairly straightforward. A bit of assembly, mounting, and wiring may be needed, but the downtime requirements tend to be small. Of particular interest is migration to smart components. These devices can be operated, monitored, and configured remotely, while providing a variety of diagnostic and maintenance capabilities. In most cases, you need to rework the addressing scheme in your control system to access these devices. Further changes to the application layer software will be required to unlock and use their enhanced capabilities. Traditional control wiring (discrete or analog) usually is replaced with network cabling, which will have different requirements with regard to electronic noise, shielding, and other factors. Remember to take that into account for reliable operation.
The lifecycle of computer workstations and servers can be relatively short compared to other elements in the control system. Updates to hardware, operating systems, historians, databases, and a host of COTS software items on your workstations and servers are a common theme. The trick here is to get everything to play nicely together. A simple antivirus software upgrade might induce unexpected problems in another crucial part of your system. Start by researching the COTS software approved-installation specifications. Next, stage the updates offline and thoroughly test your applications before deploying them to your production system. This is the best way to manage these upgrades and avoid unpleasant surprises down the road.
There will eventually come a time when you have to consider upgrading your input/output (I/O) subsystems (see Figure 2). This activity is almost a class in itself. While application software requirements might be light, the physical installation issues can be significant.
In for out
There are several factors to consider when planning an I/O system upgrade. Ask yourself these questions:
- What type of downtime windows is available for the cutover?
- Does part of the old I/O system need to stay operational while the new system is being installed?
- Can the field wiring remain intact or does that need replacement?
The downtime window will drive how much I/O can be replaced at one time. If your cutover time is tight, don’t lose heart. There are ways to reduce your installation time.
If physical space and wiring pulls permit, replacing an entire enclosure with a new one often will be the quickest path through a cutover. The new enclosure should be placed next to the existing one and should be thoroughly assembled and tested prior to the cutover. In the existing enclosure, ensure the field side of the terminals are properly tagged and confirmed. When it is time to cutover, disconnect the field side of the terminals in the existing panel and terminate them to the field side of the terminals in the new one.
If a new enclosure is not an option or only a portion of an enclosure can be cut over at a time, new panel subplates can be preassembled and prewired to new terminal strips. When the downtime window occurs, disconnect the field side of the terminals on the old subplate, completely remove the old subplate, install the new subplate, and reconnect the field wiring sides of the new subplate terminals. This approach may require installing new subplate mounting studs in the existing enclosure.
If changing subplates is not an option, it may be possible to mount and wire the new I/O system on the existing subplate. In this case, you would keep the field side of the terminal strip intact. If available on your new I/O modules, prewiring and tagging the module swing arms will save time. When field wiring must be replaced, wire can be pre-pulled, tagged, and spooled on the field-device side, control panel side, or both to allow for a faster cut-in.
A platform for change
Hardware/software platform changes can be tough-even without functional changes to the system. At a minimum, all of the application software will need to be translated, if not rewritten. For systems of any substance, the downtime requirements usually are too large to do in one installation. Keeping portions of an existing system intact while introducing elements from a new system can be one of the most demanding situations you will encounter. Start with physical installation issues associated with hardware platform changes.
For daisy-chained devices using a common power supply, strive to keep the entire loop intact on one system; this will simplify your life. Where sensor and instrumentation input signals must be shared between systems, there are a couple of options available. Networking signals between the systems is the simplest solution from a wiring standpoint, and there will be no loss of signal fidelity. Specialty communication modules may be required with this approach. For certain critical or high-speed parameters, this may not suffice. In that case, pick one system to control and measure and have the other request and/or monitor these results.
For digital output devices that need to be controlled or monitored from both systems, there are a couple of options to consider. Networking between the old and new systems is one common method. One system would physically control the device; the other would request activation or deactivation of the device. This is the most secure mode of operation and the easiest to troubleshoot if problems occur during this transition process. This works well for both digital and analog devices.
If networking is not an option, interposing relays with dual contacts can be used to allow both systems to simultaneously control a single digital output device. Extreme care must be taken with this approach. If either system requests the device, it will unconditionally activate unless otherwise interlocked. If wired and designed properly, when the cutover is complete, the old system can be disconnected from the relay, the relay coil can be removed (a point of failure), and the relay base becomes a terminal block between the I/O module and field device in the new system.
Analog output devices are more challenging and not as easily adaptable to this approach due to potential loss of signal fidelity. If you simply must control an analog device from two control systems and the systems cannot be networked, if all else fails, place the control (and associated measurement, if necessary) in one system and hardwire the setpoint or output signal (and any associated permissives) between them. Old-school techniques would typically do this with binary coded decimal modules on both ends, although other approaches are also possible.
Visualization and analysis
Visualization and manufacturing intelligence are the two main areas that you will likely encounter in software platform upgrades. Visualization software depicts the status of the processing equipment and allows people to configure and interact with this equipment in a variety of ways. The equipment depictions are often graphical in nature, although not necessarily so. They also display trend, log, and status data associated with recent plant-floor activities.
Manufacturing intelligence is derived from all the data that moves in and between systems-locally or throughout the enterprise. Historian, database, and reporting platforms are common items in this group. Data may be pushed from other systems to plant floor systems to configure product formulations, scheduling, cleaning profiles, and more. Common data pulled from the plant floor includes what was made, how much was made, what raw materials were consumed, and more. Typically, a suite of reports is associated with each operational area of the system.
The good news is that platform upgrades of this sort can be installed with less intrusion or interruption to operations. The bad news is that the application rework can be substantial. You likely will need to rebuild the visualization application from the ground up including new symbol sets, tagging, scripting, and querying routines. The same is true on the manufacturing intelligence side including rework to report layouts, schema, queries, and much more.
Due to the amount of work involved, motivations for such upgrades are often based on functional requirements inherent in the new platform and difficult to impossible to implement in the old. Before starting with the new, you must ensure that all essential operational and information aspects of the existing system are ported to the new system.
The discovery process includes interviewing all relevant stakeholders, coupled with a detailed technical analysis of all aspects of the system. Be thorough and vet everything you find. Certain infrequently used screens, reports, or queries may not be top of mind with the stakeholders, but could be essential to the operation of the system.
When discovery is complete, produce a detailed functional specification for both existing and new operations. This specification should define all functional details on what the system should do and what assets are required to get the job done. Bring the technical team, operations staff, and any other relevant stakeholders together in a design meeting to review this specification-do so prior to any other development work. This will keep everyone on the same page and establish a solid foundation for the migration. This specification forms the basis on which project progress is measured over time, and system performance is confirmed.
When your development work is complete, have a well-vetted factory acceptance test prior to installation where all stakeholders get a chance to interact with the new system. Allow enough time between the factory acceptance test and the system installation to address issues that might surface during the test.
When you are ready for installation, you can either replace the old system outright or have parallel operations for a period of time. Parallel operations of a visualization system will require additional cabling, wiring, and temporary rigging of your operator workstations. Additional cabling, network switches, and server infrastructure could be required on the manufacturing intelligence side. While parallel operations will always cost more, if your installation window is tight and you are changing a lot, it might be worth considering.
In its simplest form, upgrades to the industrial network could be an expansion to an existing, modern industrial network. Start with a network discovery using one of the many network traffic analysis COTS software utilities to identify and document all currently connected devices. Depending on the upgrade, you may need to reconfigure network switches or install new ones where appropriate. When this is complete, run the analysis software again to ensure there are no collisions or other issues apparent on the system. If problems occur, packet analysis software can provide a deeper look at what’s going on.
Upgrading an older legacy network is more complicated. You still need to start with a network discovery, but it likely will take a bit more work to identify everything connected to the network, along with the logical network addresses and physical locations of these items. The legacy platform will probably have tools to help with this process. It is also a good idea to document the physical cable type and wiring configuration on the older network (see Figure 3). Be aware that some of these legacy networks are very sensitive to the amount of information moving across them and the amount of devices connecting to them. After documenting the legacy network, put together a detailed plan on how to migrate to the new platform.
When installing the new network, ensure you have the right cable type for your industrial environment, paying particular attention to cable length, shielding, and connection best practices. In addition to cabling issues, make sure you have the right type of networking equipment. At a minimum, you will need network switches versus hubs, and if there is multicast traffic on the network, you will need managed switches. Finally, think about the access and security of the system up front, not as an afterthought. There are issues to consider with regard to physical access, network access, operating system layer security, and application layer security. Security is an ongoing endeavor comprised of both people processes and technology tools working together to protect the system.
Change is a part of life
It should be clear that your automation system will likely go through many changes over its lifecycle. These changes can take many forms—from minor modifications to major overhauls. Each type of upgrade has its own unique challenges to manage. Don’t underestimate the work at hand; even the simplest automation upgrades may not be simple. Good planning with a bit of research is the ticket to pull it off successfully. Be mindful of these guidelines, do your homework, and you can expect a good outcome every time.
David McCarthy is president and CEO of TriCore Inc., a national systems integration firm based in Racine, Wis., with offices in Santa Fe Springs, Calif., and opening soon in Indianapolis, Ind. Before he founded TriCore in 1991, McCarthy served in various capacities at Alfa Laval/Tetra Pak, including manager of engineering for its U.S.-based food engineering company. McCarthy, who has more than 30 years of experience in automation, is a computer scientist from Rochester Institute of Technology.
– See other articles from the supplement below.