Control system documentation: Don’t trust, field verify
It would be nice if control system documentation could be trusted. It would also be nice if pigs could fly. Both scenarios are equally as likely.
There is an old Russian proverb that says “doveryai no proveryai” which means “trust, but verify.” U.S. President Ronald Reagan made this saying famous in the 1980’s when referring to the United States’ relationship with the Soviet Union.
When referring to control systems, the saying should be changed to “don’t trust, field verify.” Simply put, don’t trust any form of documentation that relates to a control system. The only things that can be relied on are what is in the field and the live configuration running in a controller. This is one of the most important things that a control systems engineer can learn. As I tell everyone that I mentor, “never trust a piece of paper.”
Here are just a few examples of inept control system documentation that I have encountered before:
- Stacks of red-lined piping and instrumentation diagrams (P&IDs) that sat in a process engineer’s office for years but were never sent to CAD to make the as-built changes
- Motor elementaries stuck in a motor bucket covered with modifications that I&E made in the 1990’s, but the drawings on the network were never updated
- A normal motor starter changed out entirely with a variable frequency drive (VFD) years ago without the motor elementary ever being touched. Needless to say, almost everything on it was incorrect.
- Junction box drawings that were not updated since the boxes were installed in 1998, and points had been added to and deleted from the boxes a dozen times since then
- Loop sheets that were works of fiction. Relays or isolators had been added, the point was moved to a different junction box, cabinet, terminal strip, or I/O card, or the loop sheet was started but never completed in the first place.
- Controller cabinet drawings where the equipment layout and dimensions were all incorrect
- An interlock list where approximately 30% of the interlocks were missing from the control system or were documented incorrectly in the list (wrong trip limits, wrong initiating, or final devices, etc.). And, this list had supposedly been used to test all of the interlocks in the system on a yearly basis.
- Programmable logic controller (PLC) programs without a single comment, and the I/O tag names used in the program did not contain the loop number of the instrument.
- A hydrogen peroxide plant that did not have a single loop sheet for their distributed control system (DCS). This definitely lessened the stress of having a bunch of paper that couldn’t be trusted.
P&IDs, instrument location plans, junction box drawings, marshalling cabinet drawings, wiring diagrams, loop sheets, motor elementaries, breaker panel drawings, instrument and I/O lists, instrument spec sheets, interlock lists, and program printouts are all forms of control system documentation. They are also a partial list of the things that should not be trusted in the control systems world. It’s best to think about these items as nothing more than a good starting point.
If you field verify a piece of documentation and it turns out to be correct, then you can be pleasantly surprised and shocked at the same time. If not, then you just inherited the task of making it reflect reality before marking it up with your intended changes or additions.
While the field and the online controller configuration are the only two control system aspects that an engineer can rely on, it is important to note that one cannot be taken as the gospel without the other. The two must be compared against each other, and the area where they intersect will define the actual working scope of the control system.
For instance, there can be dead code running in the controller where a device was removed from the field but the configuration and I/O was never updated. Conversely, there may be a dead instrument or other device mounted and wired in the field that is no longer a part of the control system I/O or configuration.
To summarize, a control system is a living thing that different groups are continually changing, and the accompanying documentation often isn’t updated after it is initially created. When it is time to make a change or addition to the system, put on your hard hat and take a field trip. Then, go online with the live system and download the latest code and I/O card layout.
Don’t be the guy that designs and configures without ever leaving the office. You will become popular for all the wrong reasons.
This post was written by Jay Griffin. Jay is a principal engineer at MAVERICK Technologies, a leading automation solutions provider offering industrial automation, strategic manufacturing, and enterprise integration services for the process industries. MAVERICK delivers expertise and consulting in a wide variety of areas including industrial automation controls, distributed control systems, manufacturing execution systems, operational strategy, business process optimization and more.
MAVERICK Technologies is a CSIA member as of 3/5/2015