Nightmare recovery

How to avoid nightmare system integration projects: When an automation project is far astray, there may be a point where you must call in reinforcements to clean up rather than scrap it. Here’s how to recover from that nightmare project and what signs might point to a future nightmare.

By David McCarthy March 7, 2012

Here’s the situation: Your automation system integration client is staring down at a worst-case scenario, the prior partner fumbled the ball, and you have been called in to help. With pressure mounting from all sides to get the system up and running, it will take discipline to stabilize this situation and pull out a win for the client. Let’s look at how to best recover from this nightmare scenario. Recovery advice here can help end users and system integrators alike. 

Prioritize requirements

The start of any recovery plan is to evaluate where you are and sort out where you need to be. Begin with getting a good handle on the business priorities. Your client may have a large order in the balance, or millions of dollars in financing conditioned on packaging one case of finished product by a certain date. It is essential to separate out these unbendable needs, which may be a long cry from a fully operational system. In this way you can work with your client to realistically achieve the most important items first. Depending on circumstances, certain requirements may need to be shelved for a later date to ensure there is adequate time and resource to complete the primary objectives.

Assess the situation

Next you have to realistically assess the state of the project. Some parts of that assessment will be relatively easy to judge, such as what can be seen and touched. Is the building complete (walls, floors, ceiling, lighting, etc.)? Is all of the process and control equipment on-site? Are all required utilities operational and connected? Is everything rigged, assembled, piped, and wired?

The harder questions to answer concern the level of readiness of the system software. Typically such software can be divided into three large categories: visualization, control, and manufacturing intelligence. While they all need to work together to properly provide the required functionality of the system, you will need to check each individually to see where it is at.

Visualization software depicts the status of the processing equipment and allows people to configure and interact with the processing equipment in a variety of ways. The equipment depictions are often graphical in nature, although not necessarily so. Things to consider in your assessment include: Is all of the equipment depicted? Does it appear to be animated properly? Are operator controls in place for each area? Are setpoints, manual control pop-ups, loop tuning parameters, and other configuration data in place and operational?

The control software is generally where all function control, device control, interlocks, sequencing, fault response, and other elements are programmed. While the particulars will vary greatly, there is a common assessment method that may be use. Develop a checklist of all common and functional operations of the control software. This should be easily obtained from the functional specification of the system. If no functional specification exists, create at least a summary version listing common operations, required functions, and a general description of how they should operate, inclusive of interlocks, and fault handling. You and your team will need to observe the operation of the control software and note on the checklist which functions are operating correctly and which are not.

Manufacturing intelligence is derived from all data that moves in and between systems from the plant floor to the boardroom. While this covers a wide gamut of information, it will generally have some common characteristics. Data may be pushed from other systems to the plant floor systems to configure things like product formulations and scheduling, equipment cleaning profiles, and other attributes. Common data pulled from the plant floor often includes what was made, how much was made, what raw material was consumed, and more. There are generally a suite of reports associated with each operational area of the system. When assessing the relative level of completion of these items, some questions to ask include: Are product formulation downloads operational? Are scheduling workflows updating appropriately? Is all relevant data being captured? Are event triggers in place? Are reports in each functional area operational? Are all queries and filters operational?

Develop your plan

Your value to this process will ultimately be determined by how well you manage to take the system from where it is to where it needs to be, as quickly and safely as possible. Solid and realistic planning is what is required at this point. Lack of planning likely contributed to the nightmare scenario in the first place.

Your planning may entail utilizing the original, existing suppliers, provided that it is in their skill set and capacity to do so. Such suppliers often provide great insight into how the nightmare scenario came to be, and have solid functional knowledge of the existing systems that you will likely find useful. Often such resources are buttressed by ones from your organization, the client’s organization, and occasionally even resources from your client’s customer’s organization. With your prioritized requirements in mind, set your schedule to ensure that the project top objectives are met. Be realistic in your calendar and resource planning to meet these requirements. At this point it is better to plan for the worst than hope for the best.

For any plan to be successful, it must have buy-in from all relevant stakeholders. The goal at this point is to get the situation under control, and quickly. To do that, everyone needs to understand the priorities, their role, their tasks, and the schedule. Schedule a standing daily meeting with all relevant stakeholders. This will be necessary to keep everything on track, keep all informed of daily progress, and, most importantly, allow you to apply all available assets to facilitate resolution of any obstacles that might present themselves.

Execution tips

With your plan in place and team assembled, commence to attack the problem with a reasonable expectation of success. There can be a lot of moving parts to manage during this process. You will need to stay disciplined to keep everything on track. Maintain your checklist and schedule of all systems operations required to execute your primary objectives.

Some system components may need to be brought online quickly to perform their basic operations. Depending on the severity of the scheduling requirements, it may be necessary to sacrifice robust testing of interlocks and fault responses in favor of basic operational functionality. In this event, once the primary objectives are achieved, it is essential to circle back on the list to these items to ensure all interlocks and fault responses are operating appropriately.

When pressed for time, it is often helpful to run multiple teams. More discipline and more coordination are required with this approach, but if done properly it can increase the project pace. There are a couple of approaches. One is to run multiple teams around the clock. This works best with three shifts, usually 10 hours a shift, to allow for adequate overlap on the head and tail of each shift. Each team will need a designated lead and clear objectives. If the process permits, multiple teams could also be run in the same shift, working on different functional areas of the facility.

Finish strong

Ensure as you approach the finish line that all of the teams’ efforts are adequately documented. You will need a punch list to keep track of open issues. Make sure all issues are tracked to resolution. The operation of all processes must be well documented down to the device level, reflecting any changes that might have occurred during this process. Make sure the plant staff is adequately trained in both the operation and the maintenance of the system. Once all the dust has settled and the situation is stabilized, have an open dialogue with all the stakeholders to expose the root causes of what got them into the nightmare scenario in the first place, to help them avoid such situations in the future.

While such experiences are never pleasant for those involved, you can survive a nightmare scenario and help your client avoid such pitfalls in the future.

Crystal ball to avoid nightmares

They say hindsight is 20/20, but with enough project experience, it’s possible to observe signs that a dream project might be heading toward nightmare mode. Here are a few things to watch for to keep you sleeping soundly at night:

  • Silence is not golden, but not when it comes to managing your project. “No news is good news” is not the path to a successful implementation. Regular status meetings with demonstrable progress reports will help keep your project on track.
  • No functional specification can turn sweet dreams sour. Many times the start down nightmare road begins with lack of a proper specification. Without this, there is nothing to validate against, and no good way to gain visibility into control software development status.
  • Who’s who matters. You have asked to see and vet the technical resources assigned to your project. In response you hear the sounds of crickets from your project manager. This is a pretty good indication that your project is being handled in a less-than-structured fashion, if it is being handled at all.
  • Beware of Groundhog Day. Are the last several progress reports giving you déjà vu? When things appear stalled, it is time to start looking deeper into the issues and challenges on your project, before things get out of hand.
  • If the stars need to align just right to meet the schedule, consider some solid contingency planning. Overly aggressive scheduling is another way many nightmare scenarios begin. Even with the best of teams, unexpected things can happen and progress slows. Make sure you have some cushion in your schedule, and a workable backup plan just in case.

McCarthy is president and chief executive officer of TriCore Inc. Edited by Mark T. Hoske, content manager, Control Engineering, CFE Media. has more information on system integration.