Chasing ghosts in the plant

There are times when you just want to throw your hands up and call in an exorcist.

By Matt Ruth December 15, 2019

Sometimes, like it seemed for a client’s company, there are ghosts in the machines. On paper, everything seemed fine. The company tested every component and every supplier solemnly swore their hardware worked. And yet, the production line was freaking out unpredictably and often and the company was missing product ship dates. All the tests said everything was flawless — except it wasn’t. The operators got to referring to the problem as the result of the “ghost” and they were only half-kidding. No one had a clue. The ghost had been haunting them for years.

“Who you gonna call?”

This time it wasn’t Dan Ackroyd and Bill Murray. The Ghostbuster for this haunting was Avanceon, an engineering firm in Exton, Pa., however, not at first. The client had originally reached out to another organization to solve the problem, but since the company was a manufacturing design group and this looked like a major electrical project, they suggested Avanceon would be a better choice.

The investigation begins

When Avanceon got there, it found an organization at its wits’ end. Avanceon visited the plant to interview the head of operations, the system maintenance people and the operators. The initial goal was to investigate and develop a scope of work. The client explained that the production line was wildly inconsistent. At one moment, it would be running great, and the next it would screw up for no apparent reason.

The ghost would amuse itself by randomly frying electrical components. The company found itself replacing them on a regular basis, only to see the same thing happen again — sometimes with the same board, sometimes with others. And go figure: The hardware vendors who came in to test their components proclaimed the problem must be because of the other guy — everything of theirs was fully functional. The client was so spooked by the line’s unpredictable performance, it could see only one solution: rip out and replace its entire electrical system.

A solution on the horizon

One team member, however, found the problem eerily familiar. He recalled a project he had worked on many years earlier — he was changing transformer voltage taps to obtain the proper equipment voltage levels. Everything checked out fine. But when the system went back online, he encountered erratic equipment behavior and dropouts on voltage faults.

One time, the incoming power voltage tested just fine; the next time it was screwy. He changed the transformer voltage taps again to obtain the correct levels, and again everything seemed OK until he rechecked — and again, things were nuts. He knew what he had done, and he figured he wasn’t going crazy, but it didn’t make any sense, as the electrical equipment reported no problems. Maybe this firm had a ghost.

Then he discovered the problem always seemed to occur around 7:00 a.m. What was changing at 7:00am? It wasn’t the electrical equipment; then what was it? What else impacts incoming power? The power company.

The client called the power provider who admitted to switching the power source at 7:00 a.m. every day. This changed the incoming voltage level, and that put the electronic equipment on the edge of its voltage threshold. There was no ghost, just humans who didn’t communicate.

In fact, the team remembered more than one issue with echoes of the client’s ghost story: electronic components frying before their time pointed to a power problem. Inconsistent operation suggested fluctuations in electrical power delivery or system communications. Components that checked out individually but didn’t play nice together hinted to faulty design — plus the scary possibility of poor programming.

Further investigation

Avanceon began working with the client to do a thorough examination of the line, looking at electrical power distribution and communications. Not surprisingly, based on the issues, the original equipment manufacturer (OEM) machine builder had supplied a house of horrors:

  • Power, control, instrument and communication cables and wires were not properly separated, allowing the strong possibility of induced electrical interference
  • External cables and wires were stuffed into overstuffed wireways
  • Electrical distribution lacked isolation transformers, permitting induced electrical noise in the power distribution
  • The communication code revealed inadequate and nondeterministic communication techniques.

The analysis revealed the client’s inability to isolate the problem was due to everyone looking at it in piecemeal: while each component of the system passed inspection individually, seeing and slaying the ghost required a more holistic approach. The client, like many small- to medium-sized businesses, didn’t have a control engineer to consider the complete problem and only looked at its various elements. Its approach relied too much on specialization; they examined the individual parts of their system instead of the overall design.

Solutions proposed

Avanceon completed its analysis, and as requested, presented the client with a proposal for redoing the line’s entire electrical system. Given the quantity and quality of the recommended replacement technology, plus a substantial amount of programing, they estimated the cost at $525,000. It wouldn’t be a cheap fix.

The team also offered a different, much more economical approach. The system integrator felt it could rid the client of its poltergeist first by fixing the power distribution and communication wiring and seeing if those actions alone solved the problem. Starting cost: a much lower $85,000. However, the company said “no,” it wanted to spend the half-million and rip out and rebuild the system.

It took some persuasive discussion to get them to change their mind. The company didn’t have the in-house expertise to coordinate with the various equipment OEMs for support and maintenance, and they felt helpless in the face of the increasingly complex technology.

But then, Avanceon offered itself as the client’s automation support partner. Avanceon proposed to attack the ghost holistically rather than concentrate on individual components, train the company to help itself and provide multi-disciplinary support for system enhancements. The parties struck a deal: if Avanceon could exorcise the malignant spirit, the client would hire it to protect the company from further hauntings. The savings from the original $525K proposal would pay for it.

No more ghosts

After the client agreed, the team went to work with a multi-disciplinary team that attacked the problem on three fronts:

  • An electrical design team redesigned the electrical power distribution and added isolation transformers.
  • The electrical field staff carefully ran new instrument and communication cables around production, leaving the old cables in place. They also rewired the power delivery as per the new design.
  • The control engineer team redesigned the communications to be deterministic.

After a successful communication factory test, the client went live with the corrected system. There were very few problems. The ghost — in reality, random electrical noise roaming around the system and wreaking havoc — disappeared, banished at last.

The company awarded Avanceon an annual support contract. Avanceon provided emergency break-fix in a holistic fashion, trained company technicians and delivered monthly plant control engineering services based on a mutually created list of upgrades and improvements.

This ghost story has a happy ending. The manufacturing line was finally specter-free, which helped the client meet its production requirements and ship dates for another decade before it sold the line and upgraded its equipment.

If you’ve seen enough Hollywood spook-fests, though, you know the ghost is bound to resurface somewhere else and haunt some other unfortunate company.

When it does, the Ghostbusters at Avanceon will be ready for it.

Matt Ruth is vice president of marketing and sales of Avanceon, a CFE Media content partner.

Avanceon is a certified member of the Control System Integrators Association (CSIA).

Author Bio: Matt Ruth is president of Avanceon.