Baseline your system or else

In this Automation System Integration blog post, Anthony Baker learns that the lack of data about how a system has been operating can create a moving target after an upgrade.

06/03/2013


Anthony’s customer required an architectural change to a control system to support some new functions, a conversion from a two PLC (programmable logic controller) system to a three PLC system. All the old interlocks were mapped and moved, and new interlocks added between PLCs. All the DA (data administrator) servers, Invensys Wonderware ArchestrA objects and Wonderware InTouch HMI software tags were reconfigured for the new setup. In short, all items in the checklist were complete.

The new architecture was installed Saturday morning, and a whole bunch of test cases were run over a three-day weekend, to make sure everything was still working, and to tweak all of the control timing for the new response time. Everything looked good, so Monday night, it’s time start-up production again. And everything looked good in production too; just some minor tweaks here and there. Not everything can be caught in testing, so this was expected.

Non-specific trend

On Tuesday evening the client appears:

Client: “We’re getting more products not sent to the right location.”

Immediately questions start running through Anthony’s head:

  • What could be causing this?
  • Are there mechanical issues?
  • Is the system not setup correctly?
  • Is there a communication or network issue? I think that has that happened before.
  • Why hasn’t this been brought up earlier? It looks like everything is running just fine.

After looking around for some possible causes, Anthony can’t really identify anything that is causing this issue.

Client: “We are still seeing more products showing up in incorrect locations.”

Anthony: “I can’t really find any issues, how much more product?”

Client: “More than before. We usually get 120/160/180 per shift, and now we’re seeing 160/180, but last shift we got 300.”

Anthony: “Well 300 seems like a lot more than usual. Something else must have gone wrong.”

Client: “Maybe it did. I don’t know really. I just know we’re seeing more than before these changes were put in.”

Moving target

By this point Anthony is tearing his hair out trying to find the problem.

Anthony’s manager: “You’re spending a lot of time on this. How are we going to know when we’re done?”

Anthony: “I don’t know. I’ve fixed a bunch of things that would have existed before the change, but they keep telling me that it’s more than before, or it’s too much in general.”

Anthony’s manager: “If now it’s too much, how much is right? 100? 50? 0? What’s the target here? Do we know for sure what they had before?”

Anthony: “I don’t know. They don’t have historical data. I heard one guy say six per shift was expected...”

This continued until all ideas were exhausted, along with Anthony. In the end, it got down to a count under 30 per shift, much lower than it was before, and the client was happy. Anthony fixed some things that had been in the code since nearly the beginning of the project, but hadn’t come up before.

Lessons learned

If you’re going to be making a change or improvement to a system make sure you have the baseline data on how it operated before, in production. Your test cases won’t always show all the problems on a complex system.

- Anthony Baker strikes again: Anthony Baker is a fictitious aggregation of experts from Callisto Integration, providing manufacturing consulting and systems integration. This blog provides integration advice in plant-floor controls, manufacturing execution systems (MES), and manufacturing consulting, from the factory floor through to the enterprise. Andrew Barker, P.Eng., Callisto Integration, compiled the advice. www.callistointegration.com



No comments
The Engineers' Choice Awards highlight some of the best new control, instrumentation and automation products as chosen by...
Each year, a panel of Control Engineering editors and industry expert judges select the System Integrator of the Year Award winners.
Control Engineering Leaders Under 40 identifies and gives recognition to young engineers who...
Learn more about methods used to ensure that the integration between the safety system and the process control...
Adding industrial toughness and reliability to Ethernet eGuide
Technological advances like multiple-in-multiple-out (MIMO) transmitting and receiving
Virtualization advice: 4 ways splitting servers can help manufacturing; Efficient motion controls; Fill the brain drain; Learn from the HART Plant of the Year
Two sides to process safety: Combining human and technical factors in your program; Preparing HMI graphics for migrations; Mechatronics and safety; Engineers' Choice Awards
Detecting security breaches: Forensic invenstigations depend on knowing your networks inside and out; Wireless workers; Opening robotic control; Product exclusive: Robust encoders
The Ask Control Engineering blog covers all aspects of automation, including motors, drives, sensors, motion control, machine control, and embedded systems.
Join this ongoing discussion of machine guarding topics, including solutions assessments, regulatory compliance, gap analysis...
News and comments from Control Engineering process industries editor, Peter Welander.
IMS Research, recently acquired by IHS Inc., is a leading independent supplier of market research and consultancy to the global electronics industry.
This is a blog from the trenches – written by engineers who are implementing and upgrading control systems every day across every industry.
Anthony Baker is a fictitious aggregation of experts from Callisto Integration, providing manufacturing consulting and systems integration.
Integrator Guide

Integrator Guide

Search the online Automation Integrator Guide
 

Create New Listing

Visit the System Integrators page to view past winners of Control Engineering's System Integrator of the Year Award and learn how to enter the competition. You will also find more information on system integrators and Control System Integrators Association.

Case Study Database

Case Study Database

Get more exposure for your case study by uploading it to the Control Engineering case study database, where end-users can identify relevant solutions and explore what the experts are doing to effectively implement a variety of technology and productivity related projects.

These case studies provide examples of how knowledgeable solution providers have used technology, processes and people to create effective and successful implementations in real-world situations. Case studies can be completed by filling out a simple online form where you can outline the project title, abstract, and full story in 1500 words or less; upload photos, videos and a logo.

Click here to visit the Case Study Database and upload your case study.