Eight things to avoid in control system automation projects
Successful control system automation projects all have common attributes that make them successful. Likewise, failed projects also have common missteps that have a significant negative impact. Proper execution of a control system automation project is a process that involves continual monitoring and adjustments by project leaders. This process encompasses multiple critical steps to ensure the project is being executed properly to meet the end-user goals.
The eight elements listed below have proven to continually cause challenges for a project. The key is to recognize these missteps early and collaborate in a joint effort between parties to ensure the project stays balanced on the tightrope to success.
1. Not building a cross-functional team
Integration projects are not always simple or straightforward and can involve many players across multiple phases. Each player typically has their own area of expertise and a balanced team should be represented by knowledgeable and experienced people in all aspects of the project. Having team members that are able to produce quality solutions quickly, as issues arise, is critically important for a successful project.
These players can include the end-user, engineering firm, general contractor, electrical contractor, systems integrator, and/or mechanical contractors. Each trade must be represented with the correct level of expertise to ensure the team is staffed with the experienced required to meet the project demands. Vetting through experienced personnel is a key task when building a strong and competent team to set up a project for success. It is the project leader’s responsibility to ask the correct questions to ensure the team is staffed with the correct personnel to get the job done right.
2. Making assumptions when defining "success"
Defining what "success" looks like for all parties cannot be understated. This is typically defined within the user requirements specification (URS). A common mistake is to assume an item, deliverable, or expectation is inherently included that is not specifically and clearly spelled out. Specific goals and objectives should be identified and everyone should have a full understanding of what is expected.
Another very common mistake is to assume a platform meets the standard functions that may be derived from a specific legacy platform. The functionality may not be part of this other platform’s standard offerings and it would require custom development and testing to ensure it meets those expectations. If those expected functions are not fully identified prior to starting the project, this could lead to additional costs.
To define all base functionality, a software specification can be developed to identify typical library functions or programming standards that should be included within the project. In some cases, due to schedule limitations, projects are budgeted as using platform standards without consideration for any custom functions. This introduces risk, so knowing what you have in an existing legacy system is essential to identify what success looks like for a future platform.
3. Develop a plan with a functional design specification (FDS)
Migrating existing legacy platforms can become a convoluted and a laborious process in the effort to minimize the loss of production costs. These types of projects require extensive planning and engineering prior to going into a shutdown, identifying every single loop and piece of logic to ensure a smooth and timely change-over. Unknown information introduces risk to an automation project, given the level of integration that is required for all elements to work in unison and exchange data easily. Entering into an automation project by first developing a functional design specification (FDS), to define the deliverables and engineering tasks required to meet the final objectives, minimizes that risk.
The FDS document typically defines any and all aspects of:
- Logic architecture
- Hardware requirements
- Software requirements
- Licensing requirements
- Software development strategy
- Input/output (I/O) migration or cut-over strategy
- Implementation roadmap
- Engineering deliverables
- Testing plans
- Onsite commissioning plan
- Site acceptance testing plan.
This document has multiple functions. It can be used as a living document that outlines all of the components, deliverables, and implementation strategies, which may vary within each area of the facility, but it may also serve as a tool to evaluate multiple system integrators. Without some concrete deliverables to meet, a common practice for contractors is to provide minimal deliverables to lower the costs. This approach pushes back responsibilities to the end-user to fill in the gaps on items that are not included.
This is an important evaluation to ensure that all bases are covered, going into a project, to ensure that all parties accept the specific responsibilities. Making general assumptions on the level of deliverables typically results in large pricing differences between multiple integrators. Generally, the low bidder has accounted for fewer deliverables that may not be clearly outlined in the proposal. This leads to change orders and represents a risk to the project. Having an FDS to clearly outline all expectations provides the means to get consistent and accurate pricing from multiple integrators and allows for accurate vetting with an "apples to apples" comparison.
An FDS not only outlines the specific scope definition of the automation project, but it also includes an implementation strategy and definition of how each phase will be cut-over and commissioned, as well as a full list of deliverables. This document is a tool that outlines a complete roadmap of implementation, which could possibly include multiple phases over several years.
It holds tremendous value by clearly outlining all of the required elements, and it removes risk by defining all of the unknown items. This document, by far, is considered one of the most important steps to achieve success on a legacy migration or a greenfield automation project.
4. Not following the process and cutting corners
Developing an automation project involves several different aspects and elements that must come together and work harmoniously to achieve success. Successful projects are end results of following a proven project development process. Failures can be linked to bad habits just like successes are results of good habits and executing the process thoroughly.
Each step and phase of the project is just as important as the next. It’s the project leader’s responsibility to ensure the process is followed. Align with a team that has a proven track record and has a fully documented and understandable process to successfully implement a project. Cutting corners, taking shortcuts, or just wandering aimlessly or being reactionary can be detrimental to the project.
The results of shortcuts are not obvious right away. The issues typically lag the initial causes by a few steps and almost always show their ugly head during the commissioning phase. Project leaders must respect the process and value the discipline to stay on course. Avoid giving in to the scheduling pressures by not bypassing key steps that maintain quality and promote success by holding everyone accountable.
5. Not defining responsibilities
Having a clear definition of responsibilities is essential to ensure no gaps exist and all aspects of the project are covered. An overlap of responsibilities represents inefficiency, and clear definition and boundaries are important to ensure everyone agrees to the scope of work. One way to define responsibilities is to develop a responsibility matrix that defines all areas of deliverables listed in rows. Then, have all the players assigned to a specific column. Within the matrix, all responsibilities within each intersection should clearly be identified.
This matrix should be developed and delivered early in the project to ensure that all scope boundaries are known and understood. This exercise will help prevent overlap, promoting efficiency, and clarify everyone’s roles and deliverables. This exercise seems trivial, but it can become very extensive, even on the simplest projects, when each deliverable task is broken down into multiple sections.
6. Not accounting for contingency
Having a buffer to allow the project schedule and/or budget to shift will only allow room for success. The easiest and smoothest of projects always have a twist that is unforeseen from the development stages. Hopefully, these are small in size, do not require re-work, and are detected and corrected early on within the project. Any change or addition to scope requirements, though, could cause a large upset in the project, and if there is not a buffer in schedule, budget, or downtime, this could be a critical hit to the overall success.
Project changes are inevitable, and accounting for the potential for changes should be built into the planning process when determining project objectives. Everyone involved with the project should have target dates, budgets, and planned downtime that everyone strives to meet. These dates, values, and amounts of downtime are known as the "active goals", and they should be communicated openly and often. However, the contingency target goals are considered as the final goals that may be used for the end-user or even an internal customer.
The difference between the active and final goal is the contingency amount. This amount is typically held back at the beginning of the project and, as the project develops, it may be used, if deemed necessary. The intent is not to mislead anyone on the final dates, but to mitigate the risk by allowing the contract parties to come in under the final goal. This allows for some flexibility and sets the project up for success.
7. Not defining a test plan
Testing is an essential part of a successful project. Poor testing leads to extended time in the field and onsite software changes that should have been checked and resolved during off-site testing. Test execution is a planned event that involves all of the invested parties to ensure the end-user is receiving the system that is defined by the requirements.
Building a thorough, quality plan at the beginning of the project will define the testing expectations and allow all parties involved to understand the level of commitment and expectations. The general testing strategy should include both hardware and software. These tests can be executed simultaneously or independently, but both should be covered thoroughly.
Typical testing usually starts with hardware acceptance testing (HAT), which is powering the I/O cabinets and testing I/O points to ensure all internal wiring and hardware functions as designed. This is done prior to leaving the panel shop. All system integrators should have internal, quality HAT procedures that outline all the specific checks to ensure a quality built panel is delivered.
Software testing is performed in stages with some overlap to ensure quality. First, an internal acceptance test should be executed by the integrator prior to the actual customer attendance. Once the internal test is complete, the factory acceptance test (FAT) shall be performed. The execution of the test should be performed by the end-user with the integrator’s support. The FAT generally includes checking all of the configuration against the specifications. If specifications do not exist in some form, it is very difficult to ensure the software meets the expectation of the end-user.
Upon completion of a thorough HAT and FAT, the system is ready for shipment to site. Once the installation is complete, the site acceptance test (SAT) will be executed by the end-user, typically operations, and assisted by the integrator. At this point, the hardware within the cabinets has been tested along with the software functions, and the only outstanding items are the field devices. Upon I/O termination of field devices, the SAT execution shall mimic the FAT procedures but utilize field devices.
The SAT should not include software or hardware changes to the system if the specifications were developed correctly and the tests were executed properly. Also, SAT procedures will vary greatly depending upon the functional processing capabilities and safety functions on the site. This testing should be customized to meet the process needs as required, but it also should be thorough to ensure all aspects of the system are addressed to provide a safe, functioning system.
8. Failing to communicate well
Communication between trade disciplines can be challenging, especially when schedules are tight and everyone is focused on their own tasks and goals. However, project leadership must set the expectation and allow everyone involved to communicate openly and often.
Each party involved may be distributed across large geographical territories and may not always be readily available to have a face-to-face discussion. However, it is always recommended to physically attend the kickoff for the project, in an effort to promote a strong working relationship and allow everyone to gain a personal knowledge of each other. The next best solution is a video call, which can be an effective method as well.
Project leaders are considered to be the distribution hub for communications and are responsible for ensuring that all parties are effectively exchanging information. In order to ensure that all gaps are filled, a project leader must learn how each team member communicates and define the path and create the environment to allow them to do so.
Assessing a team member’s abilities allows project leaders to adapt their style and/or approach to help overcome potential communication deficits. This is much easier said than done and requires essential people skills to ensure a proper assessment. The assessment is an ongoing process throughout the project and becomes more critical as time gets closer to the target date.
Allocating a dedicated time slot for communications between groups is essential. This may seem to be a laborious task at the beginning of the project, but these meetings set the expectation and the tone for the project. Ideally, meeting times should be setup on a cyclical basis, and the same agenda or pattern should be followed, which should include a time slot for each representative to bring up any information openly to the group. In some cases, the leader may have to generate a series of questions to the members to allow them to provide the expected feedback and the information required to stimulate the group to interact accordingly.
Project leaders must also control the interactions to be efficient with everyone’s time. Isolated, detailed discussions should be curtailed and requested to be taken up in a separate discussion, which should be facilitated by the project leaders. After each discussion, a project leader should update the group on any final decisions and actions items that are identified.
Keys to success
Implementing a control system automation project can be difficult to manage if key aspects are not thoroughly planned out. Building a strong, cross-functional team is essential to fully delivering on project responsibilities. Defining success keeps everyone on track and motivated. Executing such a project based on a known and proven processes enables all team members to plan and fully define the detail tasks required to meet the overall objectives. Taking shortcuts or not allowing for due diligence of the project process can introduce unforeseen issues that are not easily overcome. As project leaders, it is our responsibility to remove as much risk as possible. This is accomplished by defining the unknown elements by executing a FDS prior to implementing the project to identify key areas of concerns and define solutions to minimize the risk.
Overlapping responsibilities are not always a bad thing, but the scope boundaries between all the disciplines involved must be fully examined to ensure that no gaps exist. Contingency planning is essential because issues will arise and experienced project leaders account for those issues. Another method to help mitigate risk is defining a thorough test plan. Every automation project should have a clear and precise plan to control quality, and the only way to do so is to fully test the system after development and prior to shipping, as well as during implementation.
Communication is the essential glue that helps keep the project moving forward. This must be continually assessed and managed across all levels to ensure team members have a full understanding of what is truly needed. All of the aspects listed above are key elements to control system automation projects. These projects are always challenging but also rewarding, and following each of these elements thoroughly will help the project be successful.
Robbie Peoples is an integration manager at Cross Company Integrated Systems Group. This article originally appeared on Cross Company’s Integrated Systems blog. Edited by Chris Vavra, production editor, Control Engineering, CFE Media, firstname.lastname@example.org.
Cross Company is a CSIA member as of 6/14/2017.