What happens when automation systems fail?

Inside Machines: Do the math to ensure the statistically predicted system error rates are acceptable for the processes involved when looking at automation systems, such as machine vision. (Online extra: more about the math.)


A Fanuc M-710iB robot quickly picks brake rotors from a bin using vision guidance. Courtesy: JMP Engineering Inc.What happens when it fails? This simple question is often overlooked when automation systems are designed and implemented. Asking this question can provide another dimension to a solution, often creating extra work for a system integrator in the short term, but definitely has long-term benefits. Notice the question says “when” it fails—everything will fail eventually. What really matters is how often it fails, and what happens when it does. A few examples follow of actual vision systems where considering this question was critically important.

Vision-guided robotic bin picking

Vision-guided robotic (VGR) bin picking is a unique challenge. The intention is that product in bins is removed by robots and loaded into machines, onto conveyors, etc. As easy as this might sound, there are significant challenges in implementing it successfully, mainly based on the structure of the bin and parts inside. VGR itself is a whole other topic (vision—2D vs. 3D, bins—structured, layered, jumbled, and such), but no matter the technology, part presentation, and other factures, the success/failure rate “per pick” is a serious consideration.

This is best explained by example. Consider a bin that contains 100 parts. This is the “standard” bin, and whenever a bin is presented to a robot, it starts with 100 parts. Then consider the success rate on an individual part—this is the product of the vision success rate (How likely can parts be identified and located?) and the robot grip success rate (Once a part is located by vision, how likely is it that the part can be physically gripped?). In the example, if the vision success rate is 99.5% and grip success rate is 99.5%, then the per part success rate approximates 99% (99.5% x 99.5%).

That means for each part in the bin, the robot is 99% likely to pick it successfully. Sounds good, but consider that 99% over the 100 parts (an entire bin)—the “bin success rate.” Basic statistics tells us that the bin success rate is (0.99)100 = 0.366 or 36%. Suddenly 99% isn’t so good. This means that for a typical bin, there is only a 36% chance it will be emptied without issue or, in other words, a 64% chance there will be a failure at some point in that bin.

So what happens when it fails? This is an important question in this example, because it appears that in 64% of bins there will be an issue. Is it a big deal? This is application specific—perhaps the process is fine, and it will result in a couple “leftover” parts in the bin. The other extreme is that the process stops and requires significant intervention and downtime.

Automotive vision inspection

Another very common application of vision is inspection. Inspection systems typically perform measurement or part presence/absence or grading to provide some type of pass/fail result. On the simple end, an inspection system can consist of one camera looking at one feature. More complex systems often involve multiple cameras looking at multiple features of a part.

Consider a car coming off the end of an assembly line. A vision system mounted in a pit under the car with multiple cameras looks at 40 features, such as bolts, brackets, covers, and other attributes. Instead of looking at the vision success rate, the best way to look at this application is to consider the vision “false reject” rate. This is the likelihood that the vision system will give a “fail” result when it should have given a “pass” result.

In this example, consider customer requirements. The customer has 400 cars coming off of the production line in a shift, and when there is a false reject, an associate must drive the car to a manual inspection pit where another associate must inspect the car. This is a significant operation, so the customer only wants to manually inspect four cars per shift. Four cars out of 400 is 1%. That means on one car, the vision system must have a false reject rate of 1%. The easiest way to analyze this is to consider the total number of inspections over 400 cars; there are 40 inspections per car, which gives us 16,000 inspections. Of the 16,000, there can only be four false rejects.

Calculation: 4/16,000 = 0.00025, or a 0.025% false reject rate. In other words, 99.975% of inspections must not have a false reject. This is a stretch for a lot of simple vision inspections, and extremely difficult if not impossible for complex inspections.

Again the question—what happens when it fails? In this case the system required a look at feasibility. For the number of inspections involved in the system, the complexity of each inspection had to be very low for the project to be feasible. In addition, the integrator had to spend a very long time fine-tuning each individual inspection to achieve such high accuracy. This project likely would have been a failure from the integrator’s and customer’s point of view if “what happens when it fails” was not considered.

Individual vs. system fail rates

From these two examples it is easy to see how system failure rates can quickly get out of hand even if the individual failure rate looks good. Be realistic; expect that the system will fail, analyze how often it is likely and the impact it will have. It is very easy to overlook, but asking this question as early in the process as possible can mean the difference between a failed project and a great project!

- Kevin Ackerman, M.Sc, PEng, is a controls specialist with JMP Engineering Inc., a control system integrator in the Control Engineering System Integrator Hall of Fame. Edited by Mark T. Hoske, content manager, CFE Media, Control Engineering and Plant Engineering, mhoske(at)cfemedia.com.




ONLINE extra: A few more words about the math....


The probability that a single part is picked successfully is (0.99)

The probability that 2 parts are both picked successfully is (0.99) X (0.99) or (0.99)2

By the same application of stats, 100 parts all picked successfully = 0.99 X 0.99 X 0.99 … 100 times or (0.99)100 = 0.366 or 36%.

Anonymous , 01/18/13 11:07 PM:

What happens when an automation system fails? The customer should say, "either make it work or or take it away and give me my money back".
In the case of the bin picking vision system the odds of successed are improved by equiping the system with redundant methods of finding the target.
A redundant system might use both laser sensors and a vision system mounted on a robot arm.
Lets say that the vision system method is very fast but only has a success rate of 98%. If that method fails, laser sensors can be used which are also successful 98% but have a significantly longer search time. That raises the odds of one of the two systems working to 99.96%. The vision system and part gripper are most likely to work best if the exact distance to the target is verified by a distance measuring laser sensor. Simply getting precise target height data may improve the performance of the vision system and gripper. If the vision systems still fails to recognize the target with the camera at the correct distance from the target, the robot can be programed to use lasers to automatically detect the exact location and orentation of the target. This takes longer but is only required the low percentage of times when the vision system fails and allows the system to continue working without operator intervention. Once the position of the targer is verified by lasers, the camera is positioned in the correct location and takes another picture. If the system does not match the "taught" image to the current image, it can still assume that this is what the image looks like now and store the new image for future reference. Any time the vision system fails to match the current image to the original taught image, it can always compare it to other taught images to see if they match. It can also record statistics and make the decision to try the most reequently matching images first.

If all fails to correct the a problem with the vision system it's probably because the system requires maintenance or repair. After the problems are corrected the programs that use the lasers to find the part can be used to automatically retrain the Vision system.
The Engineers' Choice Awards highlight some of the best new control, instrumentation and automation products as chosen by...
Each year, a panel of Control Engineering editors and industry expert judges select the System Integrator of the Year Award winners.
The Engineering Leaders Under 40 program identifies and gives recognition to young engineers who...
Learn how to increase device reliability in harsh environments and decrease unplanned system downtime.
This eGuide contains a series of articles and videos that considers theoretical and practical; immediate needs and a look into the future.
Learn how to create value with re-use; gain productivity with lean automation and connectivity, and optimize panel design and construction.
Go deep: Automation tackles offshore oil challenges; Ethernet advice; Wireless robotics; Product exclusives; Digital edition exclusives
Lost in the gray scale? How to get effective HMIs; Best practices: Integrate old and new wireless systems; Smart software, networks; Service provider certifications
Fixing PID: Part 2: Tweaking controller strategy; Machine safety networks; Salary survey and career advice; Smart I/O architecture; Product exclusives
The Ask Control Engineering blog covers all aspects of automation, including motors, drives, sensors, motion control, machine control, and embedded systems.
Look at the basics of industrial wireless technologies, wireless concepts, wireless standards, and wireless best practices with Daniel E. Capano of Diversified Technical Services Inc.
Join this ongoing discussion of machine guarding topics, including solutions assessments, regulatory compliance, gap analysis...
This is a blog from the trenches – written by engineers who are implementing and upgrading control systems every day across every industry.
IMS Research, recently acquired by IHS Inc., is a leading independent supplier of market research and consultancy to the global electronics industry.

Find and connect with the most suitable service provider for your unique application. Start searching the Global System Integrator Database Now!

Case Study Database

Case Study Database

Get more exposure for your case study by uploading it to the Control Engineering case study database, where end-users can identify relevant solutions and explore what the experts are doing to effectively implement a variety of technology and productivity related projects.

These case studies provide examples of how knowledgeable solution providers have used technology, processes and people to create effective and successful implementations in real-world situations. Case studies can be completed by filling out a simple online form where you can outline the project title, abstract, and full story in 1500 words or less; upload photos, videos and a logo.

Click here to visit the Case Study Database and upload your case study.