# What happens when automation systems fail?

## Inside Machines: Do the math to ensure the statistically predicted system error rates are acceptable for the processes involved when looking at automation systems, such as machine vision. (Online extra: more about the math.)

What happens when it fails? This simple question is often overlooked when automation systems are designed and implemented. Asking this question can provide another dimension to a solution, often creating extra work for a system integrator in the short term, but definitely has long-term benefits. Notice the question says “when” it fails—everything will fail eventually. What really matters is how often it fails, and what happens when it does. A few examples follow of actual vision systems where considering this question was critically important.

**Vision-guided robotic bin picking**

Vision-guided robotic (VGR) bin picking is a unique challenge. The intention is that product in bins is removed by robots and loaded into machines, onto conveyors, etc. As easy as this might sound, there are significant challenges in implementing it successfully, mainly based on the structure of the bin and parts inside. VGR itself is a whole other topic (vision—2D vs. 3D, bins—structured, layered, jumbled, and such), but no matter the technology, part presentation, and other factures, the success/failure rate “per pick” is a serious consideration.

This is best explained by example. Consider a bin that contains 100 parts. This is the “standard” bin, and whenever a bin is presented to a robot, it starts with 100 parts. Then consider the success rate on an individual part—this is the product of the vision success rate (How likely can parts be identified and located?) and the robot grip success rate (Once a part is located by vision, how likely is it that the part can be physically gripped?). In the example, if the vision success rate is 99.5% and grip success rate is 99.5%, then the per part success rate approximates 99% (99.5% x 99.5%).

That means for each part in the bin, the robot is 99% likely to pick it successfully. Sounds good, but consider that 99% over the 100 parts (an entire bin)—the “bin success rate.” Basic statistics tells us that the bin success rate is (0.99)100 = 0.366 or 36%. Suddenly 99% isn’t so good. This means that for a typical bin, there is only a 36% chance it will be emptied without issue or, in other words, a 64% chance there will be a failure at some point in that bin.

So what happens when it fails? This is an important question in this example, because it appears that in 64% of bins there will be an issue. Is it a big deal? This is application specific—perhaps the process is fine, and it will result in a couple “leftover” parts in the bin. The other extreme is that the process stops and requires significant intervention and downtime.

**Automotive vision inspection**

Another very common application of vision is inspection. Inspection systems typically perform measurement or part presence/absence or grading to provide some type of pass/fail result. On the simple end, an inspection system can consist of one camera looking at one feature. More complex systems often involve multiple cameras looking at multiple features of a part.

Consider a car coming off the end of an assembly line. A vision system mounted in a pit under the car with multiple cameras looks at 40 features, such as bolts, brackets, covers, and other attributes. Instead of looking at the vision success rate, the best way to look at this application is to consider the vision “false reject” rate. This is the likelihood that the vision system will give a “fail” result when it should have given a “pass” result.

In this example, consider customer requirements. The customer has 400 cars coming off of the production line in a shift, and when there is a false reject, an associate must drive the car to a manual inspection pit where another associate must inspect the car. This is a significant operation, so the customer only wants to manually inspect four cars per shift. Four cars out of 400 is 1%. That means on one car, the vision system must have a false reject rate of 1%. The easiest way to analyze this is to consider the total number of inspections over 400 cars; there are 40 inspections per car, which gives us 16,000 inspections. Of the 16,000, there can only be four false rejects.

Calculation: 4/16,000 = 0.00025, or a 0.025% false reject rate. In other words, 99.975% of inspections must not have a false reject. This is a stretch for a lot of simple vision inspections, and extremely difficult if not impossible for complex inspections.

Again the question—what happens when it fails? In this case the system required a look at feasibility. For the number of inspections involved in the system, the complexity of each inspection had to be very low for the project to be feasible. In addition, the integrator had to spend a very long time fine-tuning each individual inspection to achieve such high accuracy. This project likely would have been a failure from the integrator’s and customer’s point of view if “what happens when it fails” was not considered.

**Individual vs. system fail rates**

From these two examples it is easy to see how system failure rates can quickly get out of hand even if the individual failure rate looks good. Be realistic; expect that the system will fail, analyze how often it is likely and the impact it will have. It is very easy to overlook, but asking this question as early in the process as possible can mean the difference between a failed project and a great project!

- Kevin Ackerman, M.Sc, PEng, is a controls specialist with JMP Engineering Inc., a control system integrator in the Control Engineering System Integrator Hall of Fame. Edited by Mark T. Hoske, content manager, CFE Media, *Control Engineering *and* Plant Engineering,* mhoske@cfemedia.com.

JMP Engineering Inc is a CSIA member as of 3/5/2015

www.controleng.com/machinevision

**ONLINE extra: A few more words about the math....**

The probability that a single part is picked successfully is (0.99)

The probability that 2 parts are both picked successfully is (0.99) X (0.99) or (0.99)^{2}

By the same application of stats, 100 parts all picked successfully = 0.99 X 0.99 X 0.99 … 100 times or (0.99)^{100} = 0.366 or 36%.