Remote support update advice and best practices
When a controls programmer has make a live update to an already running process, it is best to follow strict procedures and best practices to mitigate risks and ensure success when making these changes.
In 1991, an accounting software program called AbraPay made a change to a single line of code one week prior to launch. The programmer who made the mistake thought the change was so minor he did not need to inform anyone of the edit. The day after launch, companies who received the software starting complaining all paychecks were being printed at zero dollars. The programming team had to revise the code, thoroughly retest the software, and rush out the revised copies. One little mistake cost the company over $100,000 in profits.
Every controls programmer has made a simple mistake that has produced unexpected and sometimes comical results. In other situations, however, these errors can cost time, money, or even pose a significant safety risk. It is always best practice to test all programs prior to implementing on a live system, but occasionally it is necessary to make a live update to an already running process. Whether updates are remote or local, all have a certain level of risk. It is best to follow strict procedures and best practices to mitigate risks and ensure success when making these changes.
The first step in making changes is to evaluate the scope and impact this this may have to the process. Is the person making the updates going to be remote or on site? Will changes be made or affect the programmable logic control (PLC), human-machine interface (HMI), servers, network configuration, or other vital areas to the facility? Upon completion, will anything need to be restarted? Will any essential pieces of equipment be affected? What are the expected results? All possibilities need to be considered and communicated to plant managers, operators, and maintenance personnel. Once a scope is properly defined, the next step is to begin outlining the tasks to complete.
It is important to list out all the activities to be performed to ensure nothing is missed. Outline every detailed change sequentially and have a colleague or manager proofread the list. Each item should include an estimated time of completion. If the timeframe is strict, it will be important to stay on schedule for each task. If the timing slips, have a contingency plan for recovery. The first item should be creating a backup of the original system. If needed, a plan should be in place to revert back to the original program or setting. A backup may also be useful to verify changes in the future.
Best practices while testing
Timing can have a major impact on whether performing these tasks are a success or a major failure. Ideally, a plant or process will be shut down temporarily to allow modifications and thorough testing. If the system is running, wait for better conditions. Be sure to ask about shift changes. Troubleshooting remotely can be difficult when the customer is preparing to leave or performing a turnover. Also, be ready to provide support should things not go smoothly. Don't perform an update late afternoon on a Friday without expecting a call over the weekend.
Communicating with all personnel in the area prior, during, and after all changes is vital. Everyone should be aware of the changes and possible outcomes. If an error is made in the update, the plant managers and operators are the people left with the problem. Ensure they are well informed on prior operations, what is going to be done, and the expected results of the update. This will help them identify anything unexpected as well as clarify what to troubleshoot.
It is best to program all PLC changes in a way to quickly revert to an older version of the program. A possibility is to place a test bit before each new rung to energize or de-energize as needed. Be mindful of what is being de-energized. For example, if rungs are going to be deleted eventually, duplicate destructive bits may be produced. This could cause the new and code to not work properly regardless of which version is enabled or disabled. Always verify or parse new sections of the program and clear up any warnings or errors prior to execution.
It is fairly common to uncover another issue while addressing something unrelated. As mentioned in the story above, it is important to make these issues known. Once a plan is being executed, it is important not to deviate from that plan. Do not immediately correct the issue without informing the proper personnel. If this affects the plan, halt any progress and reevaluate the tasks needed to be performed. Otherwise, note the issue and address it after the current task is complete.
Be wary of search and replacing large sections of copied code. If something is not replaced with the proper name, existing tags or equipment may be unintentionally referenced. Data could be overwritten and unpredictable behavior may result. For example, a sequence may be monitoring a level transmitter on a wrong tank. Without the proper level reading, the sequence could inadvertently overflow the tank and spill onto the plant floor. If this happens, an unpleasant phone call is likely to follow. Have another resource available for assistance.
Have a list of phone numbers available in case extra support is needed. Occasionally the unexpected occurs, and the window of time is quickly closing. Do not hesitate to call a coworker or technical support if needed. Inform the resource before performing the update to ensure availability if needed. If this doesn't happen, it may take an hour or more to figure out what a five-minute phone call could have resolved.
Additional customer changes, revisions
Once all online changes are complete, immediately notify all customers affected and review all steps performed. It is good policy to send a follow up email detailing all actions and expected results of the update. Take another backup of the system, and make note of when the customer is expected to test or see the changes in action. Even if the planning and execution is flawless, it is possible the changes specified by the customer did not perform as intended and may require immediate attention.
There is an inherent danger when making changes to processes, especially when performed remotely. Given enough time, it is inevitable that something unexpected will happen or be uncovered when investigating or changing a live process. Take time to evaluate any circumstances that arise and never compromise safety. Tensions may be high at the customer site, so it is important to remain calm and to continue to be part of the solution.
This post was written by Tyler Bullock. Bullock is a senior engineer at Maverick Technologies, a leading automation solutions provider offering industrial automation, strategic manufacturing, and enterprise integration services for the process industries. Maverick delivers expertise and consulting in a wide variety of area including industrial automation controls, distributed control systems, manufacturing execution systems, operational strategy, business process optimization and more.
Maverick Technologies is a CSIA member as of 2/2/2016