Master Disaster How to Avoid Abnormal Situations

KEY WORDS

Process control & instrumentation

Distributed control systems

Expert systems

Human factors engineering

Modeling

Supervisory control

Sidebars: Guide to Process Control ASM Consortium work in progress

In early mining applications, cages of birds were placed throughout shafts to alert miners of abnormal situations—too much methane gas. Later, birds were replaced with sensors and annunciator panels that alerted workers of abnormal conditions.

In the 1980s programmable control systems such as distributed control systems, programmable logic controllers, and supervisory control and data acquistion systems provided the capability to assign a variety of alerts, warnings, and alarms on sensor values, and to initiate appropriate action if the alarms are ignored. But through it all, the ability to predict what might happen next and to provide assistance in navigating out of an abnormal situation, has changed very little from those early days of yesteryear when a miner’s best friend was a live bird.

While most abnormal situations do not result in explosions and fires, they are costly in terms of poor product quality, schedule delays, equipment damage, and other significant costs. The Abnormal Situation Management (ASM) joint research and development consortium, led by Honeywell (Phoenix, Ariz.), reports that abnormal situations cost the U.S. economy $20 billion annually and represent the number one problem in the petrochemical industry. Estimates compiled by the ASM consortium, indicate that elimination of all abnormal situations in petrochemical plants could add 5% to profits.

In an industry that operates at 5 to 8% profit margins, eliminating or reducing abnormal situations could provide significant cash to the bottom line.

In the 1990s, advances in chemistry introduced complex compounds requiring increasingly complex control algorithms, such as fuzzy logic, neural networks, adaptive gain, and dynamic-matrix-control, to meet quality and/or production demands. While control system manufacturers have been making it possible to incorporate advanced control technologies, similar sophistication has yet to reach most control system alarm management systems. It is still common to receive a low flow “nuisance” alarm when the operator stops the corresponding feed pump.

Attempts to integrate knowledge-based systems with plant operations have been few in numbers and mildly successful, mainly due to the complexities associated with:

Integrating multiple proprietary platforms;

Keeping the knowledge base current with ever-changing plant operations;

Identifying and implementing models and methods best suited to handle the variety of complex problems of chemical process plants; and

Getting all the operations “experts” to agree on what actions to take once the problem has been identified.

Venkat Venkatasubramanian, professor at Purdue University’s School of Chemical Engineering (Lafayette, Ind.), and member of the ASM consortium, compares chemical plants with people who have a very complex illness. “One or two doctors are unable to diagnose the illness. It takes a team of specialists each looking at the symptoms, each developing an opinion, performing additional test, and then conferring with team members to reach a final conclusion.”

Similar to an ill patient, diagnosing a complex chemical process requires combinations of mathematical models, expert systems, neural networks, statistical techniques, and operations personnel, each working to independently diagnose an abnormal situation, with final diagnoses developed through cooperative problem solving.

Like the medicine and therapy prescribed by a team of physicians, management of an abnormal situation begins after the diagnoses are complete and agreement is reached.

Sounds simple—it’s not

Dr. Venkatasubramanian cautions, “Getting four or five methods looking at the same problem, each in its own unique way, yet each reaching the same conclusion is very challenging.” Purdue’s school of chemical engineering has been involved in the development of “tools” to assist diagnosing and managing abnormal situations in chemical plants since 1985.

Today, the hurdles of deploying diagnostic and advisory systems is less formidable. Open systems and communication standards make integrating multiple platforms easier. Graphical tools and object based applications help keep the knowledge base matched to current plant operations. Object-oriented programs and relational databases allow development of individual models and methods in efficient environments while maintaining the ability to “cooperatively” combine results. What remains difficult is getting experts to agree on the best “prescription” after the diagnosis.

As experienced ASM solution implementers are quick to explain, technology alone is not the salvation for averting or recovering from abnormal situations. People are still involved, and timely, accurate communications is paramount to achieving and maintaining a successful ASM deployment.

Ted Cochran, Honeywell’s ASM program coordinator, says that corporate attitudes can hamper a successful ASM implementation. These include:

Blaming individuals;

Resolving big incidents, but allowing “near-misses” to remain incomplete;

Developing emergency procedures, yet rewarding operators who “ride-out” a process upset; and

Seeking minimum regulatory compliance measures.

People side of ASM solutions

ASM consortium members have expended considerable efforts to understand the people side of deploying ASM solutions and have identified the following necessary factors:

Dynamic training that can provide rigorous, intense, and realistic scenarios will develop high-performance operations teams. Airline flight crews undergo similar training;

Reporting, tracking, and resolution of large and small incidents are critical. Deliberate and disciplined sharing of insights and experiences gained from unusual events with all who might benefit, promotes communication and improves operation team preparedness;

Communications must be formalized to ensure consistent messages are sent (and received) from-top-to-bottom, team-to-team, and plant-to-plant;

Authority must be well defined and fully accepted;

Consistent, well established practice of creating, evolving, and following procedures is paramount; and

Collaboration among operation team members must allow rapid and accurate exchanges of information.

Mr. Cochran cautions that introducing collaboration before achieving consistency may accelerate the rapid exchange of inaccurate information.

John Josserand of Applied Training Technologies (Houston, Tex.), a member of the ASM consortium, advises that supervisors and managers may soon be included in training exercises because technologies, such as palmtop PCs, will allow remote access to operational information.

Technologies contribute

Complex processes require control systems designed, programmed, and tuned to provide automated control for normal or near-normal operation. When the process becomes unsafe, safety instrumented systems designed to initiate a process shutdown take over. But between normal operation and shutdown, processes can deviate into abnormal situations lasting a few minutes, or several days. Often deviations are undetected because automatic control readjusts the process. When an abnormal situation comes to the operator’s attention, the common response is to place loops in manual, reduce feed and energy streams, and manually attempt to return the process to a normal (steady) state—all the time searching for the initial cause of the problem. Frequently, the switch from automatic to manual control only worsens the situation, and a shutdown follows.

Previous approaches using technologies to assist operations in identifying and managing abnormal situations evolved large, specialized applications. These applications compared theoretical process models to real-time plant operations and generated alerts, recommendations, and predictions.

Some success has been achieved with these solutions, but a lot of “care-and-feeding” is required to keep them current with every-changing plant operations. Also, some systems use linear models that can ignore the nonlinearity and limitations of real equipment, and results in developing false predictions of equipment or process responses.

Today’s offering of object-oriented software designs, relational databases, modular software development and maintenance tools, open communication standards, and acceptance of PCs makes development and deployment of knowledge-based ASM applications possible, but users still need to understand what they need and want.

Similar to the plight of management information systems, enterprise resource planning, and computer integrated manufacturing systems, the ASM “deliverable” is left to the buyer and the provider to agree what is and is not included.

Since no standard definition of ASM exists, no standard architectural model is available. The ASM Solution Anatomy model accompanying this article was developed from information collected from various sources, including Internet searches, and represents a “generic” ASM solution (see diagram).

Developing a complete ASM solution requires implementing two parts, or layers. The first layer validates in-coming data and generates advisories of what is happening during an abnormal situation. The second layer predicts where the process is likely to go if current conditions persist. Some ASM solutions describe “closing-the-loop” between the ASM solution and the process. This is a form of supervisory control, with provision for the operations team to remain part of the diagnosing and prescribing process.

While not all ASM solutions include all pieces of both layers, most provide the following pieces for constructing the advisory layer.

A control system interface that uses robust, real-time communication standards, such as OPC (OLE for process control), gateways to proprietary systems, or custom written application program interfaces, is necessary to obtain information from the control system about process measurements, valve positions, device status, etc.

Sensor validation to quickly detect sensor malfunctions or failures is critical to the integrity and acceptance of the ASM solution. For example, “failed” sensor input signals remain below a minimum value longer than a defined period, while “frozen” sensor input signals do not exceed the expected noise band for a period of time.

Jack Stout, president of Nexus Engineering (Kingswood, Tex.), explains, “The advanced diagnostics available in ‘smart’ transmitters and digital valve controllers is valuable in validating individual sensors. Many control systems can alarm, based on these diagnostic errors. ASM solutions differ by requiring sensor validation to include establishing sensor relationships to produce ‘signatures’ of equipment module and/or process unit performance. Informing the operations team that a pump has tripped because of cavitation, and an empty vessel caused the cavitation is a simple example of ASM sensor validation, alarming, and messaging.”

Point retrieval of real and calculated process variable information is important in developing ASM solutions. Real process variables include temperatures, flows, pressures, analyzer results, control valve positions, etc. Calculated process variables include outputs to valves, totalized volumes, on-line material and energy balance calculations, etc. Combining real and calculated information is critical in developing performance “signatures”.

Message handling and viewing must provide accurate, concise, and timely information about the current and future state of the process. ASM solution message complexity can vary from single line text messages to context sensitive help systems, allowing the operations team to view the appropriate level of detail. Some ASM solution message handlers automatically “pop” the initial alert on the operator’s screen. After that, navigation buttons for cause-and-effect, details, procedures, and trouble-shooting are available.

Alarm handling that alerts the operations team of escalating circumstances during an abnormal situation requires advanced alarm management. Merely generating alarms, as many control systems do, is inadequate. As processes move through varying operational states, the operations team must remain focused on the task at hand. Spending time to work through complex alarm scenarios and then implementing advanced alarm management techniques will help the operations team be more effective during crisis.

Incident history archives are files of past process performance data. Initially the data may come from an existing data historian and can be used to playback past situations (good and bad) for testing the expertise of the ASM solution.

Rolling data archives combine information collected by the point retrieval module and the sensor validation module into files that allow other modules to work with “smoothed” data.

Custom and generic displays are the operation team’s window into the ASM workings. Custom displays are one-of-a-kind displays created specifically for a particular part of the process. Generic displays are templates for repetitive process areas (i.e., tank farms) with relevant data mapped into the display based on operator or event occurrences.

Combined, these pieces form the advisory layer to provide the operations team with early-warnings of a processes’ current health. However, the ASM solution requires additional sophistication to predict where the process is going.

The ASM solution prediction layer should develop equipment and plant signatures during normal operations and compare these to current operating signatures. Elements of this layer especially benefit by mixing mathematical models, neural networks, and statistical techniques to implement a solid ASM predictive layer.

For illustration purposes, the predictive layer consists of two parts: modeling, and planning and executing.

Modular objects

ASM problems are so complex that no single mathematical modeling tehnique is appropriate for each piece of plant equipment. Applying the appropriate model is easier when plant equipment is viewed as individual objects. For example, the model most appropriate for centrifugal pumps may differ from the model chosen for gear pumps. Developing models in an object-oriented programming environment to match plant objects, makes assembly and maintenance of the larger, more complex process models easier.

Control module (measurements, valve outputs, etc.) modeling allows development of sensor related calculations. For example, a “rate-of-change” calculation may be a more appropriate model for a temperature measurement than working directly with the process variable.

Equipment module (pumps, on/off valves, exchangers, headers, etc.) modeling combines control module models with equipment status to form mixed expression logic formulas. For example, combining the process variable value of a flowmeter in a calculation with the on/off status of a pump to determine if a flow rate should be present, avoids a low flow “nuisance” alarm when the pump is stopped.

Unit modeling combines control and equipment module calculations to form mathematical models of equipment, such as distillation columns, fluidic catalytic-crackers, fractionators, waste-heat boilers, and compressors.

The top layer of the anatomy diagram introduces very innovative concepts, especially for many chemical operations. But, as chemical complexity (and product value) increases, as quality demands continue to toughen, as pressure to reduce emissions builds, and as demands to “stay-on-line” echo through chemical operations, innovative thinking transforms good performing companies into great performing companies.

“Closing-the-loop” of an ASM solution requires very specialized functions, such as state-estimator, goal-setter, planner, and executor modules.

State estimator modules can determine the current process state, such as improving, staying the same, or getting worse, based on information provided from the lower layers of the anatomy.

Goal-setter modules gather and maintain information relevant to quality and production goals established prior to the abnormal situation occurrence.

Planner modules develop and recommend recover-plans after refining multiple test results from current and historic knowledge of the process represented in the modeling and advisory layers.

Executor modules close-the-loop and monitor success.

Abnormal situation management solutions are specialized applications of expert systems designed to work like the plants best operator, on their best day, every day. These systems never get bored, distracted, or take a break; they remember what happened last week, last month, and last year, and provide accurate, consistent information, even in the heat of “battle”.

In many ways chemical processes and humans are similar. Both contain complex systems that periodically experience abnormal situations…they can become ill. Monitoring for possible process “ills” no longer requires keeping one eye on a bird in a cage. Technologies and expertise are available to implement solutions to monitor for abnormal situations, develop accurate diagnosis, and make educated prescriptions for recover.

What’s the health of your plant…right now?

Consortium members:

Amoco

Applied Training Resources

Brad Adams Walker Architecture

British Petroleum

Celanes Ltd.

Chevron

Equilon (Shell & Texaco)

Exxon

Gensym

Honeywell

Mobil

Nova Chemicals Ltd.

Technology Training Systems Inc.

Union Carbide Corp.

Consortium affiliates:

Ohio State University Department of Chemical Engineering

Purdue University School of Chemical Engineering

University of Toronto Department of Industrial Engineering

Guide to Process Control

For more information on related topics in this edition of Control Engineering, see:

“Improving Safety in Process Control” article;

ISA Expo/98 article and product section;

New flow-switch exclusive article;

Ultrasonic flowmeters article;

Low-flow technology article;

Applying field instruments article;

Year of the Network article on fieldbus applications;

Eli Lilly cuts costs article;

Molson brews with artificial intelligence article;

Data acquisition software “Product Focus;” and

Safety briefs, safety PLC, other ‘News’ items.

Obtain other information using search functions at

ASM Consortium work in progress

The Abnormal Situation Management (ASM) Joint Research and Development Consortium, led by Honeywell, was formed in 1992, and became legally established in 1994, the same year a 31/2 year, $16.6 million cooperative agreement was awarded by the U.S. National Institute of Standards and Technology (NIST). The cooperative agreement required consortium members to contribute 51% of the funds with NIST contributing the remaining 49%.

The goal of the NIST-funded program is to demonstrate the technical feasibility of collaborative decision support technologies for improving the performance of operations personnel.

The consortium’s proposed system-level solution, named AEGIS (abnormal event guidance and information system), was to be developed in four phases. Work on the first phase began in early 1995.

Phase one developed the AEGIS prototype using Sun, HP, and Pentium hardware running Solaris, HP-UX, and Microsoft Windows NT operating systems. Application software comprised Gensym’s G2, Applied Training Resources PRISM, Microsoft’s Visual Basic and Access, and a few specialized programs developed in C and C ++.

AEGIS’ “testbed” architecture includes: a prototype operator console, a simulated refinery unit, a distributed control system, and a variety of networked workstations and software.

Phase two enhanced the operator interface; added more comprehensive process models; and improved notification, navigation, and diagnosis. Process simulations were expanded to include additional units.

Phase three demonstrated the ability to close-the-loop using the goal-plan-executor modules of the AEGIS prototype.

Phase four is a beta site and is in progress.

For more ASM consortium information, visit

Do you have experience and expertise with the topics mentioned in this content? You should consider contributing to our WTWH Media editorial team and getting the recognition you and your company deserve. Click here to start this process.