What happens when automation systems fail?

Inside Machines: Do the math to ensure the statistically predicted system error rates are acceptable for the processes involved when looking at automation systems, such as machine vision. (Online extra: more about the math.)


A Fanuc M-710iB robot quickly picks brake rotors from a bin using vision guidance. Courtesy: JMP Engineering Inc.What happens when it fails? This simple question is often overlooked when automation systems are designed and implemented. Asking this question can provide another dimension to a solution, often creating extra work for a system integrator in the short term, but definitely has long-term benefits. Notice the question says “when” it fails—everything will fail eventually. What really matters is how often it fails, and what happens when it does. A few examples follow of actual vision systems where considering this question was critically important.

Vision-guided robotic bin picking

Vision-guided robotic (VGR) bin picking is a unique challenge. The intention is that product in bins is removed by robots and loaded into machines, onto conveyors, etc. As easy as this might sound, there are significant challenges in implementing it successfully, mainly based on the structure of the bin and parts inside. VGR itself is a whole other topic (vision—2D vs. 3D, bins—structured, layered, jumbled, and such), but no matter the technology, part presentation, and other factures, the success/failure rate “per pick” is a serious consideration.

This is best explained by example. Consider a bin that contains 100 parts. This is the “standard” bin, and whenever a bin is presented to a robot, it starts with 100 parts. Then consider the success rate on an individual part—this is the product of the vision success rate (How likely can parts be identified and located?) and the robot grip success rate (Once a part is located by vision, how likely is it that the part can be physically gripped?). In the example, if the vision success rate is 99.5% and grip success rate is 99.5%, then the per part success rate approximates 99% (99.5% x 99.5%).

That means for each part in the bin, the robot is 99% likely to pick it successfully. Sounds good, but consider that 99% over the 100 parts (an entire bin)—the “bin success rate.” Basic statistics tells us that the bin success rate is (0.99)100 = 0.366 or 36%. Suddenly 99% isn’t so good. This means that for a typical bin, there is only a 36% chance it will be emptied without issue or, in other words, a 64% chance there will be a failure at some point in that bin.

So what happens when it fails? This is an important question in this example, because it appears that in 64% of bins there will be an issue. Is it a big deal? This is application specific—perhaps the process is fine, and it will result in a couple “leftover” parts in the bin. The other extreme is that the process stops and requires significant intervention and downtime.

Automotive vision inspection

Another very common application of vision is inspection. Inspection systems typically perform measurement or part presence/absence or grading to provide some type of pass/fail result. On the simple end, an inspection system can consist of one camera looking at one feature. More complex systems often involve multiple cameras looking at multiple features of a part.

Consider a car coming off the end of an assembly line. A vision system mounted in a pit under the car with multiple cameras looks at 40 features, such as bolts, brackets, covers, and other attributes. Instead of looking at the vision success rate, the best way to look at this application is to consider the vision “false reject” rate. This is the likelihood that the vision system will give a “fail” result when it should have given a “pass” result.

In this example, consider customer requirements. The customer has 400 cars coming off of the production line in a shift, and when there is a false reject, an associate must drive the car to a manual inspection pit where another associate must inspect the car. This is a significant operation, so the customer only wants to manually inspect four cars per shift. Four cars out of 400 is 1%. That means on one car, the vision system must have a false reject rate of 1%. The easiest way to analyze this is to consider the total number of inspections over 400 cars; there are 40 inspections per car, which gives us 16,000 inspections. Of the 16,000, there can only be four false rejects.

Calculation: 4/16,000 = 0.00025, or a 0.025% false reject rate. In other words, 99.975% of inspections must not have a false reject. This is a stretch for a lot of simple vision inspections, and extremely difficult if not impossible for complex inspections.

Again the question—what happens when it fails? In this case the system required a look at feasibility. For the number of inspections involved in the system, the complexity of each inspection had to be very low for the project to be feasible. In addition, the integrator had to spend a very long time fine-tuning each individual inspection to achieve such high accuracy. This project likely would have been a failure from the integrator’s and customer’s point of view if “what happens when it fails” was not considered.

Individual vs. system fail rates

From these two examples it is easy to see how system failure rates can quickly get out of hand even if the individual failure rate looks good. Be realistic; expect that the system will fail, analyze how often it is likely and the impact it will have. It is very easy to overlook, but asking this question as early in the process as possible can mean the difference between a failed project and a great project!

- Kevin Ackerman, M.Sc, PEng, is a controls specialist with JMP Engineering Inc., a control system integrator in the Control Engineering System Integrator Hall of Fame. Edited by Mark T. Hoske, content manager, CFE Media, Control Engineering and Plant Engineering, mhoske@cfemedia.com.

JMP Engineering Inc is a CSIA member as of 3/5/2015




ONLINE extra: A few more words about the math....


The probability that a single part is picked successfully is (0.99)

The probability that 2 parts are both picked successfully is (0.99) X (0.99) or (0.99)2

By the same application of stats, 100 parts all picked successfully = 0.99 X 0.99 X 0.99 … 100 times or (0.99)100 = 0.366 or 36%.

The Engineers' Choice Awards highlight some of the best new control, instrumentation and automation products as chosen by...
The System Integrator Giants program lists the top 100 system integrators among companies listed in CFE Media's Global System Integrator Database.
Each year, a panel of Control Engineering and Plant Engineering editors and industry expert judges select the System Integrator of the Year Award winners in three categories.
This eGuide illustrates solutions, applications and benefits of machine vision systems.
Learn how to increase device reliability in harsh environments and decrease unplanned system downtime.
This eGuide contains a series of articles and videos that considers theoretical and practical; immediate needs and a look into the future.
Women in engineering; Engineering Leaders Under 40; PID benefits and drawbacks; Ladder logic; Cloud computing
Robotic integration and cloud connections; SCADA and cybersecurity; Motor efficiency standards; Open- and closed-loop control; Augmented reality
Controller programming; Safety networks; Enclosure design; Power quality; Safety integrity levels; Increasing process efficiency
This article collection contains several articles on how advancements in vision system designs, computing power, algorithms, optics, and communications are making machine vision more cost effective than ever before.
Featured articles highlight technologies that enable the Industrial Internet of Things, IIoT-related products and strategies to get data more easily to the user.
This digital report will explore several aspects of how IIoT will transform manufacturing in the coming years.

Find and connect with the most suitable service provider for your unique application. Start searching the Global System Integrator Database Now!

Cloud, mobility, and remote operations; SCADA and contextual mobility; Custom UPS empowering a secure pipeline
Infrastructure for natural gas expansion; Artificial lift methods; Disruptive technology and fugitive gas emissions
Mobility as the means to offshore innovation; Preventing another Deepwater Horizon; ROVs as subsea robots; SCADA and the radio spectrum
Automation Engineer; Wood Group
System Integrator; Cross Integrated Systems Group
Jose S. Vasquez, Jr.
Fire & Life Safety Engineer; Technip USA Inc.
This course focuses on climate analysis, appropriateness of cooling system selection, and combining cooling systems.
This course will help identify and reveal electrical hazards and identify the solutions to implementing and maintaining a safe work environment.
This course explains how maintaining power and communication systems through emergency power-generation systems is critical.
click me