Creating standards for controls systems for mergers and acquisitions
Creating and following controls system standards and guidelines can help mitigate costs during a mergers, acquisitions, and expansions for the pipeline and terminals industry.
The pipeline and terminals industry is in a constant state of upheaval and change. In busy times, new players emerge while existing companies expand and grow. In slower times, there seems to be an increase in mergers and acquisitions. A consistent theme exists: the industry’s landscape is constantly changing and increasing in complexity.
Companies spend exorbitant amounts of money investigating these growth opportunities by exploring the potential for new markets, new transportation channels, and increased throughput, and capacity. Considerations are given for the real estate, infrastructure, personnel, and existing contracts involved. The control systems involved in these mergers and expansions are often overlooked. Control systems are critical components of these facilities, and the failure to properly understand and account for them can result in higher unintended costs and philosophical compromises to properly integrate these new assets into an organization’s existing systems and structures.
So how do you effectively investigate, understand, and consider control systems during potential acquisitions, mergers, and expansions? There are two critical aspects:
- Understanding what actually comprises a control system
- Establishing, maintaining, and following a strong philosophy regarding control systems.
What is a control system?
Typically, people only associate the physical hardware and human-machine interface (HMI) screens with control systems. If you go to a facility and ask the local personnel to show you their control system, most often they show you their programmable logic controllers (PLCs), distributed control system (DCS) panels, and their operating screens. These are considered "traditional" controls systems.
While they certainly are critical components of any controls system, when dealing with expansions and mergers, a strong argument can be made that they are actually the least important aspect. If needed, they can be relatively easily upgraded or changed in a cost effective manner.
During an investigation and consideration of growth and expansion, there are other aspects of control systems that must be considered, and many oil and gas professionals need to expand their understanding of what your control system truly entails.
Failure to properly consider the holistic and comprehensive nature of controls systems is a perilous exercise. If the entire system is not completely understood and accounted for, you are doing a disservice to your company, setting your projects up for failure, and opening up the door for significant additional unexpected cost, scope creep, and major headaches and integration issues.
Creating standards for a controls system
The primary foundation for a successful controls system is a strong and sustainable philosophy outlining a company’s controls system identity. Every aspect of a controls system outlined above should have standards and guidelines that a successful company can adhere to and follow. These guidelines establish a minimum baseline for how a company is going to develop, operate, and maintain their controls systems and serve as a roadmap to assist growth, expansion, and acquisition decisions. The philosophy should be based off of how to manage and maintain control systems while accounting and supporting relevant industry regulations and standards (American Petroleum Institute, Pipeline and Hazardous Materials Safety Institution, OSHA, etc.).
At a minimum, a true controls system philosophy should include:
- Automation philosophies—How are the assets/facilities automated? Is it a highly manual, highly automated, or hybrid process? Are the facilities locally or remotely controlled? Is there an external monitoring system involved? These standards outline the level of complexity of your systems.
- Documentation—What documentation requirements exist? These standards mandate what documentation must be developed and maintained at all facilities. It should include standard templates and formats. Important examples include architectures, electrical drawings, input/output (I/O) lists, and data maps, among many others.
- Hardware/software—What systems, platforms, or requirements does the company standardize with? This relates to not only PLCs, HMIs, DCSs, and their associated software platforms, but also to computer and server operating systems, versions, and patches.
- Programming guidelines and style guides—These guidelines affect the look and feel of a company’s controls system code bases. They can span from standard code modules for specific process devices (valves, pumps, tanks, etc.) to specific color schemes and layouts for visualization screens.
- Migration and update procedures—It is critical to have a plan for migrations and updates. When hardware, firmware, or software has been upgraded or re-envisioned, how do you analyze and decide if and when to upgrade your systems? This process is extremely important to ensure that unintended incompatibilities are not introduced. Alternatively, it is imperative to make sure a system is not conditional on any obsolete or unsupported components.
Establishing a robust controls philosophy is only the first step. Most importantly, once a company has established their philosophy and controls identity, it is imperative to stand by it. It does not make sense to invest in developing standards and guidelines, only to deviate from them or consistently make exceptions. The value in having a comprehensive philosophy resides in the ability for a company to standardize and maintain a baseline set of requirements to consider when making expansion and growth decisions. Additionally, when a strong controls philosophy is in place, more flexibility is available for system support. Assets across a company are developed and maintained using standard documentation, hardware, and code bases. Tribal knowledge and single points of failure are minimized, if not eliminated entirely, through standardization. When done correctly, companies should be able to seamlessly maintain their facilities with little issue and more efficiently grow, without finding themselves forcibly bound to any individual employee or contractor for system support.
Controls systems’ external connections
The external connections are arguably the most important. External connections refer to anything that extends outside of the fence line of a facility. These connections include a company’s enterprise or corporate systems (accounting, inventory, scheduling, etc.) and interactions and integration to customer systems. External connections have major implications not only to how a terminal or pipeline company makes money (customer systems), but also to how that money is tracked, accounted for, and collected (enterprise and corporate systems). These interactions must occur for any facility to properly function. As such, the failure to account for these systems during an expansion or growth decision can incur significant cost to modify systems to properly integrate the new asset.
With respect to a controls system’s external connections, when making expansion and growth decisions some critical things to comprehensively consider include:
- Operational requirements—What is the intended operational intent of the facility? What must your control system do in order to operate in the manner that it needs to operate in? Does the potential acquisition already provide that, or will it need to be modified, and to what extent?
- Information requirements—Depending on the facility, this could include customer information, tickets, reports, orders, schedules, manifests, etc. What do your corporate systems require? What do the facility’s existing customers require? Are those requirements compatible?
- Data availability—Do you even have access to all of the information that your corporate systems require to include access to customer info that you need to process tickets/orders? Are there existing customer agreements in place that you will need to abide by if the acquisition goes through? Do new agreements need to be reached?
- Data compatibility—Assuming you have access to all of the data that you require, to what extent is that data going to need to be manipulated in order to integrate it into your systems? Will you require middleware, either off the shelf or custom? Companies that grow via acquisition normally are faced with integrating disparate systems. How does your company’s controls philosophy dictate that integration?
- Security—It is imperative that any existing connections meet-or can meet-your company’s security requirements.
- Information access control—When sharing info with customers, it is critical to be sure that they only see or access what they are supposed to. It is also important to be sure that your corporate systems can properly access what they require, as well.
During an expansion or acquisition, companies will undoubtedly be encountered with a challenge associated with an external connection to their control system, be it during a due diligence period or following the growth decision. Failure to properly address these issues proactively may result in decreased efficiencies, unbudgeted efforts to remedy an issue, or compromised controls philosophies. Any of these problems will ultimately cost money. Decreased operational efficiency and facility throughput may be the biggest risk.
Operational processes will need to rely on manual methods if a facility is unable to:
- Properly, accurately, and efficiently receive schedules/orders
- Process tickets
- Receive or update accurate manifests, bills of lading, or authorized driver lists.
In order to continue to operate, the local team’s workload will have to increase to accommodate the more manual process. As opposed to being able to rely on the successful operation of the controls systems, operators must be manually processing, communicating, and validating all of their information and operations. Increasing their workload can lead to a need for additional manpower or increased operator errors.
Data and information errors may also occur if external connections are not properly considered. Errors or trouble with schedules, orders, or tickets will directly impact operations and create accounting issues that will require investigation and remediation. Additional systems or middleware may be required to ensure that this information is properly received, processed, and tracked. Data security and unintentional information exposure risks are potentially looming, as well.
Finally, a significant concern of poorly planned external connections to a controls system is complicated supportability and unnecessary complexity. If there are 14 facilities with 14 different local inventory/tracking systems, you need to be able to support all of them. If it is an off-the-shelf system, the company will need someone familiar with it. If it is a custom system, the company may be tethered to a contractor or third-party who built it.
Additionally, any time that a controls system is heavily dependent on middleware to properly integrate local systems into corporate enterprise systems, there is unnecessary complexity and points of failure that will also need to be supported. If growth is not properly navigated, a company could find themselves forced into an unintended reliance on a single person or contractor.
Addressing communications and network challenges
The backbone for any control system is going to be the communication paths that it relies on. Whether it is communications between two pieces of hardware at a facility or communications with an important customer, control systems can quickly be rendered useless with the loss of a critical communications path or network. Having an understanding of the existing networks and communications structures at your existing facilities and those you are looking to acquire and integrate with is tantamount to a successful integration. How do the new acquisitions fit into your existing infrastructure?
Networking and information technology (IT) is a complex, and often confusing, realm. However, asking the right questions and taking the following into consideration when making expansion and growth decisions will establish a baseline to be sure that an acquisition/expansion goes smoothly:
- Existing internal and external communications—What communications channels are currently in place at all of the facilities? Are any existing communications channels or ports at capacity? Is there a need to maintain all of the existing connections? Are the acquisition’s communications methods compatible with existing systems?
- Available external communication methods—What communications options are available? Depending on a facility’s location, there may be challenges to various forms of communication. How remote is the facility? Is there access to phone lines and Internet service providers? How is cell service? Is there the ability to use radio communications? Are there any frequency issues or restrictions? Satellite possibility? How expensive is it to implement any of these options?
- Operational uptime requirements—How critical are the operations? How dependent are they on communications? The answer to this will drive many communications decisions-both on what types and the level of redundancy.
- Security—How vulnerable are the networks at these new facilities? To what extent will they require upgrading to get them in line with your enterprise security standards? How sensitive is the data and operations that are ongoing?
- Segregated networks—Often times, it is desirable to have segregated networks-separate networks for controls, business, and corporate, for example. Is that already in place, or will there need to be modifications to support the segregation?
- Supporting real-time operations—Are you expecting to support real-time, dynamic operations? Will there be many changes happening quickly or will operations be fairly stagnant? The required speed, refresh times, and redundancy required by the system will vary based on how often things are going to change and how dependent systems are on real-time information-schedules, orders, tickets, etc.
- Remote accessibility—Will operations require remote accessibility-be it for remote control of the facility, remote monitoring, or remote support? How dependable does that access need to be? How will the access be allowed or restricted? What types of protections are available to prevent adversely affecting a live system?
Early and proper attention to the IT challenges involved with a growth decision can mitigate many of the issues inherent in expansions. However, depending on the required operating conditions, failure to account for the above considerations can literally take a station down and halt operations. Some of these pitfalls can be crippling and extremely expensive to mitigate or correct after the fact. Network failures or loss of communications at a remotely controlled facility may force an emergency shutdown. If the station is remotely monitored, network outages or communications failures can result in the loss of critical situational awareness at the facility.
Facilities with remote access have an additional set of pitfalls to avoid. While the ability to remotely access and support a controls system has a myriad of benefits, a downside of accessing a live production system remotely is the opportunity to inadvertently or adversely affect that system. Additionally, if there is a remote facility that relies of remote access, a network or communications failure may result in the need to deploy a physical resource, which can be extremely costly. That cost is exacerbated if the facility and operations go down while the appropriate resources are mobilized to correct and fix the issues. Finally, when remote access is available, the potential exists for an unauthorized access event. Obviously, security breaches are bad and need to be minimized, if not completely eliminated.
Electrical system safety
The state of the electrical systems and the associated drawings and documentation is a consistent concern with expansions, mergers, and acquisitions. Due diligence must be paid to assessing and understanding the electrical infrastructure to be affected and integrated when making acquisition and expansion decisions. Substandard electrical systems and documentation should be a major cost consideration. The following are things to consider:
- Electrical drawing requirements & standards—What do your approved philosophies require? Do you have standard device, panel, or P&IDs drawings? To what level are you going to need to investigate and update the facility drawings to get into compliance with those standards? Do those standards specify how these components should be wired, powered, or controlled? Will there need to be an investment to make modifications to the existing drawings to get them in compliance with enterprise standards?
- State of existing electrical systems—What is the current state of any systems? Are they old and poorly maintained? Are cables, terminations, and systems properly labeled so they are easily identified and tracked? Is there spare conduit, cable, and wire available for any required upgrades? Are there electrical safety concerns? Are there Arc Flash hazards requiring mitigation?
- Drawing repository and availability—Does the facility have accurate as-built drawings? To what extent do they need to be redlined and updated? How do the existing drawings compare to enterprise standards? Are the drawings available in an editable format, or will they all need to be digitized for revisions, updating, and maintaining?
- Power availability—Do the facilities have up-to-date load studies? What additional power loads may be placed on the systems? Is there sufficient power availability for what is required? What are the backup power requirements-generators, batteries, or uninterruptible power supply (UPS) systems? Are spare circuits available or the capacity for required additions?
When a facility is acquired, its electrical infrastructure and documentation are acquired with it. If the condition and quality of these systems and drawings are not properly understood and accounted for, it can present a significant hidden cost to remediate them. Insufficient or inaccurate electrical drawings can be a major black hole. This information is critical for system upkeep and troubleshooting. Without accurate drawings, in an available and editable format, there may be a major monetary outlay to investigate, assess, and update/create the necessary drawings and documentation.
Systems in disrepair also represent a major concern. Systems in bad shape will need to be fixed, updated, or replaced. Depending on the systems, this can be very costly and/or create unintended outages and downtime. Additionally, there may also be outdated, legacy, or even obsolete equipment that will need to be maintained or is reaching the end of its serviceable life. If one of these systems fails, often times replacements or spares are hard to come by, presenting an untenable situation that may result in unsafe conditions or complete facility outages.
Failures in electrical systems generally present themselves quickly, unexpectedly, and with dangerous potential. Above all else, dealing with electrical systems comes with significant safety concerns. The potential for an Arc Flash event exists in many legacy facilities with antiquated switchgear. Inaccurate drawings can create a safety risk. Companies must be sure to not expose their people to unnecessary risk and hazard when expanding and growing.
Traditional controls systems during
Traditional controls systems (PLCs, DCSs, HMIs, SCADA, etc.) are another consideration in a growth or expansion decision; however, they do not have a significant impact. If needed, their cost to update or replace is relatively small when compared to the other elements outlined earlier. Still, they are important and should be considered when making these decisions:
- Corporate philosophy—What is your philosophy? What do you standardize on? How firm are you on those standardizations, or are you flexible?
- Existing platforms and systems—What is currently deployed and in place? How do the deployed systems fit into your philosophy? Does the existing hardware and software give you the functionality needed to fulfill your operational requirements? Will it be integrated into your current systems or must it be replaced? How old is it? What software and firmware versions are in play, and are they compatible? Is the code readily available, properly commented, and documented? Is configurable documentation available and up to date (IO Lists, data maps, etc.)?
- Spare capacity and expandability—Do the existing systems have any spare capacity? Is there spare I/O in the PLC/DCS systems to tie in any new instrumentation or devices? Does the HMI and visualization platform have spare tags for expansion?
- Supportability—How supported are the deployed systems? Are there any current transferrable warranties in place? Does the original vendor exist and still support the systems? Is there a large and readily available base of engineers available to support them? For example, Allen Bradley and Wonderware systems are industry standards and widely supported by engineers across the country. Certain other platforms, however, have reached the end of their serviceable life or have restricted support options.
- Compatibility—Will the existing systems be compatible with the required corporate systems, third party systems, or other controls systems? Will they be directly compatible or will there need to be any middleware to support communications and integration?
- Reliability—How reliable are the existing systems? Are spares readily available or difficult to come by? If spares need to be ordered, what are the expected lead times? Some hardware is readily available across the country, while others sometimes are built to order overseas.
As mentioned, the cost to correct or remedy a failure or shortcoming of a traditional controls system is relatively minor, but it can still be problematic, bothersome, and time-consuming. In any growth activity, these systems will require some level of maintenance and upgrade to properly integrate them. Failure to account for this work prior to the growth decision, however, will result in unaccounted for and unintended work to ensure that the integration is ultimately successful. Unavailable or inaccurate configurable documentation or code will require an as-built effort. Incompatible systems will require middleware or complete replacement. Systems that are in conflict with your corporate philosophy will need to be addressed, according to that philosophy.
Critical controls systems and executing a growth activity
Growth activities are a necessary and natural evolution within the oil and gas industry. The considerations and pitfalls outlined above are not intended to dissuade the pursuit or consideration of expansions, mergers, and acquisitions. They are intended to provide information that should give you pause and make you think when pursuing growth, as they can make an extreme impact and are expensive to remedy or correct. It is important to plan for these considerations/pitfalls when doing due diligence. One way or another, all of these issues have been encountered—multiple times—and the cost to correct, troubleshoot, or fix them can be substantial.
A critical controls system failure or oversight when pursuing a growth activity can be a multi-million dollar mistake. Understanding what to investigate and what questions to ask can uncover potential shortcomings that can be addressed or accounted for in the decision-making process. Ensuring that controls systems are properly considered during the decision-making process is a critical step to safeguarding a successful controls systems integration and growth activity.
So how does a company successfully navigate this process? It all starts with the sound development and commitment to a controls identity. This philosophy can be based off of an analysis and baseline of a company’s existing assets, developed from scratch based off of overarching goals and best practices, or some combination of both. Professional systems integrators and controls systems consultants specialize in and can support the development of a cogent and sustainable controls system philosophy. Once established, a strong controls identify can, and should, be followed during the pursuit of potential growth.
Following these measures ensures that the acquisition and expansion opportunities are identified and analyzed with respect to that philosophy, ensuring that necessary consideration is given to the opportunity’s fit within the philosophy and the efforts required to properly integrate it into a company’s inventory.
Control system structure
The structure of a traditional control system structure includes:
Traditional controls systems
- Programmable logic controller (PLC), distributed control system (DCS), human-machine interface (HMI), and visualization systems
- Measurement and sampling.
Critical controls system interdependencies
- Corporate systems and third-party connections
- Communications, networking, and information technology (IT)
- Electrical systems and integration
- Historian systems.
T.J. Gibbons is the director of operations at Mangan.