The Role of Simulator Technology in Operator Training Programs
Can better training reduce the losses companies experience from human error as operators attempt to respond to abnormal situations?
The National Institute for Standards and Technology (NIST) and the Abnormal Situation Management (ASM) Consortium estimate that U.S. process industries lose over $20 billion each year from abnormal situations. They have determined that 50% of abnormal situations are directly attributable to human error by plant personnel.
Abnormal situations require the utmost operator skill, knowledge, and critical thinking. This includes the ability to perform optimally on infrequent tasks, while under high stress. According to the U.S. Chemical Safety Board, incidents during plant start-up are 10 times more likely than during normal operations.
Despite significant losses from abnormal situations, process manufacturers continue to rely on operator intervention during start up, shutdown, and other operational scenarios. Training is fundamental to reducing these losses, however, standard classroom instruction or on-the-job training are insufficient. It is one thing to know what to do during an abnormal situation; it is entirely another to identify the problem, determine the appropriate response, and then take action in the proper amount of time.
Whether at new or existing plants, industrial facilities need to consider use of an operator training simulator (OTS) in their training programs. As companies lose operators to retirement, the remaining staff may have little or no experience dealing with turnarounds or abnormal conditions. The increased use of advanced process control (APC) has also moved panel operators further away from control of the process. When an abnormal situation arises, an operator must not only consider the process response, but also how complex control strategies will affect a given situation, and act accordingly.
Industry experience has shown that a realistic OTS model provides the best training environment for plant personnel. These advanced systems look and feel like a real plant, with the same dynamics as a live process. An OTS is the key to training operators effectively— validating procedures and shortening startups—leading to increased profitability. This solution is particularly beneficial for distributed control system (DCS) and emergency shutdown (ESD) checkout prior to bringing a greenfield facility or modified plant into operation.
Implementation of an OTS for operator training varies widely from industry to industry, and from company to company. Some facilities base their operator training solely on classroom and on-the-job training. Others use classroom training supported by high-fidelity simulators. Some train at most a few days per year, while others train extensively.
In most cases, OJT without the use of a training simulator is inadequate because startup, shutdown, and hazardous conditions are only covered if they occur during a training session. Training is often conducted after an experienced operator takes over required actions and the process is under control. This after-the-fact approach may prevent detailed discussions of actions and what-if scenarios. Also, trainees may not have chance to perform critical tasks themselves, and actions cannot be repeated at a later time.
As early as 1983, Kellogg and Exxon presented papers at the AIChE Ammonia Plant Safety Conference on using OTS for operator training. Although other reports of successful greenfield ammonia startups crediting OTS have been published, these systems are not considered a necessity on many new projects. Known technology, strict QA/QC programs, and detailed commissioning plans are most often credited for a successful project.
Despite the need for greater awareness of OTS benefits, operator training simulators are easily justified on greenfield projects—shorter commissioning and startup times equate to early positive cash flow. Indeed, the implementation of an OTS solution is a simple risk management decision based on the rationale that a few days delay in commissioning or startup pays for the system.
For brownfield projects and existing facilities, an OTS system can be more difficult to justify. Investment in a training simulator is typically based on an estimate of incident cost avoidance. In this case, the benefits of an OTS are considered “soft” because they do not provide a clearly definable result such as a production rate increase. Despite recognition of OTS as a best practice, most existing plants do not invest in this technology.
Greenfield projects that do invest in an OTS often do not continue supporting this asset through the plant’s lifecycle. Cost is the main reason why some facilities discontinue using their OTS. Plus, there may be insufficient funds to maintain the OTS model, or the model becomes outdated and does not mimic the real plant closely enough. Under certain circumstances, the cost of a training program is deemed expendable once the plant is up and running.
Agrium is a major retailer of agricultural products and services in North and South America, a global wholesale producer and marketer of agricultural nutrients, and a leading supplier of specialty fertilizers. Agrium produces and markets all four primary groups of nutrients: nitrogen, phosphate, potash, and sulfur, as well as controlled-release fertilizers and micronutrients.
Agrium is the owner/operator of several ammonia plants in North America, one of which is located in Redwater, Alberta, Canada, approximately 45 km northeast of Edmonton. The facility was originally owned by Imperial Oil Ltd., a subsidiary of Exxon Inc. Redwater Nitrogen Operations is Agrium’s largest nitrogen producer. Its products include: ammonia, ammonium nitrate, ammonium sulfate, UAN (urea ammonium nitrate) solutions, and urea. The site’s ammonia production capacity is 960,000 gross (250,000 net) tons, and its total nitrogen product capacity is 1,365,000 tons.
In 1983, a new ammonia plant at Agrium’s Redwater site used an Exxon-developed OTS for initial start up. The OTS was run on a MicroVAX computer and used DCS hardware for the human/machine interface (HMI). The original OTS model scope was limited to basic control, cause and effect, and emergency upset training.
For Agrium, the benefits of using a narrow-scope OTS during initial start up were so great the simulator was expanded to include start up and shutdown. This system was employed for several years to train new panel operators, but it eventually fell into disuse due to a lack of model support, antiquated technology, and scavenging of DCS equipment.
In 1998, the Redwater facility requested approval for funds to replace its existing OTS with an updated, modern training solution. This project was seen as a positive initiative for all ammonia plants throughout the company. Strategy development kicked off in early 2001, but unfortunately, investment capital restraints delayed work for the next four years. During this interim period, the plant experienced an accelerating turnover rate for panel operators due to numerous upgrader and refinery projects in the area.
The OTS project was finally approved for the 2007 capital year. A key objective of the development team was designing a simulator that would not fall into disuse due to poor model design or technology obsolescence. The development process was not without challenges: preliminary simulation models were delivered on schedule, but detailed tuning took longer than anticipated. Additionally, the level of modeling detail required to meet the expectations of experienced operators was greater than expected.
Justifying the project
In the case of an ammonia plant, integrated heat recovery and process recirculation make it an ideal candidate for use of an OTS. Startups are long and require multiple process units to start in parallel. Operators must also monitor the status of many variables, and downstream process conditions affect upstream operations.
Integrated ammonia/urea/utility plants can use an integrated simulator model to allow operators of different units to train simultaneously during combined startup, sharing utilities and intermediates.
As part of the OTS development project at the Redwater facility, senior executives requested a study to determine the operational and economic benefits of an Agrium-wide operator training simulation strategy. A review of Redwater’s historical on-stream performance revealed (see chart):
• Schedule factor improvements in the year immediately following a turnaround;
• Schedule factor declines year over year until the next turnaround, with the lowest schedule factor in a turnaround year; and
• Increased schedule factor declines with longer turnaround cycles.
Further analysis of outage data and incident reports revealed interesting causes of the schedule factor pattern:
• Difficulties, problems and upsets in restarting units after turnarounds; and
• Inability to respond correctly to upsets as the time after a turnaround lengthened.
Agrium determined that both causes of the schedule factor pattern resulted from a lack of practice in start-ups, shutdowns and abnormal situation management. After reviewing the historical plant performance data, Agrium used the following methodology to justify its OTS investment:
• Half of incidents either caused or exacerbated by human error will be avoided;
• One-quarter of incidents exacerbated by human error will be avoided; and
• Typical startup time will be reduced by 10%.
These projections were applied to outage incident data from a five-year period to quantify potential benefits over an average year. Incidents where human error was not a factor (or where it was uncertain) were not included in the analysis. Downgrading incidents were not considered in the analysis, even if human error played a role in the incident.
In order to be effective, an operator training program must be highly structured and designed to address important topics such as effective troubleshooting, plant startup, shutdown, and abnormal situation management. It should also include equipment malfunctions during normal operating conditions and startup/shutdown procedures.
Personnel involved in operator training should be required to:
• Review pre-startup checklists;
• Maintain a trainee logbook recording actions taken during training sessions; and
• Use the logbook of previous sessions to review the current state of simulation, thus simulating shift turnover communication.
It is important the OTS be kept current with changes in the real plant environment, incorporating steps for management of change (MOC). The system should be designed to ensure that changes to plant processes, controls, interlock system, etc. are incorporated into simulations in a timely fashion.
A full-time trainer is generally required for a training program to deliver anticipated benefits. An experienced operator who can pass on expertise and has credibility within the operations group to train other operators best fulfills this role. The trainer should encourage use of the OTS as a tool for continuous learning. As new scenarios and abnormal situations are encountered, they should be added to the operator training curriculum.
It is also a good practice to locate the simulator in a room within the control building so operators can use it during periods of stable operation for refresher training or to try strategies for various abnormal situations.
Another way to maintain the value of an OTS is using it to analyze and validate new procedures and “what if” scenarios. Plant personnel can also develop and validate control strategies on the simulator.
Most importantly, the success of a training program incorporating an OTS is dependent upon long-term management commitment and the resources necessary to maintain and enhance the system. The simulator cannot be viewed as cost cutting target only.
Redwater Nitrogen Operations gained many valuable insights from the implementation of an advanced operator training simulator at its Redwater, Alberta, Canada ammonia plant. Important “lessons learned” included:
• Owner/operator assistance on model tests will help ensure final product matches real plant;
• 100% dedication and involvement by the owner is needed from project inception;
• Field equipment, control valve, DCS, and interlock logic data must be accurate and detailed;
• Owner should educate him/herself before the Functional Requirement Document (FRD) is completed;
• DCS controls and alarms should in place before model tests are conducted;
• Implement controls and alarms as early as possible to ensure efficiency of model tests;
• Ensure participants in the factory acceptance test (FAT) understand “work-arounds” are not allowed; and
• Perform customer interviews during contractor selection, and do not select on price alone.
As proven at Agrium’s Redwater, Alberta facility, operator training simulators are recognized as a best practice for training control room panel operators on abnormal situations and other “real world” plant scenarios. Processing operations can determine the economic justification for implementing an OTS in their operator training programs through detailed incident analysis. These studies will demonstrate that OTS solutions offer significant operational and economic benefits, even for existing production facilities.
Al Roe and John Mason, Agrium Inc.; Jose Alamo, Honeywell Process Solutions.