Fun With Measurement Systems Analysis – Part 2

Welcome back to the second, and final, part of this measurement systems analysis series.  If you didn’t read part 1 of this series please do before pressing on with this part.  We’ll wait on you!

Measurement System Analysis

Defined, MSA is used to quantify the amount of variation attributed to the measurement system rather than the product or process being studied. In other words, MSA allows us to understand how much of our observed variation is being caused by the measurement system.

When we’re working with variables, or continuous data, we’ll conduct what are called Gage R&R studies which is short for Gage Repeatability and Reproducibility Studies.  When we’re working with attribute data, we’ll conduct Attribute Agreement Analysis.

To be sure, MSA can be used throughout the DMAIC or PDCA process. In fact, you don’t even need to be doing a formal DMAIC or PDCA project to conduct an MSA.  As it relates to the DMAIC roadmap, MSA’s are usually done during the measure, analyze, or improve phases. Anytime you’re relying on data to make decisions, you should verify the measurement system can be trusted.  Put another way, if you’re ever unsure of your measurement system please perform a MSA no matter what phase of the project you’re in!Snip20150810_8

In fact, as an aside, I once worked with a team to improve the performance of an optical inspection measurement system. We spent around seven hours on the issue and the results were extraordinary since, as it turned out, this company was scrapping hundreds of thousands of dollars each week since their measurement system was rejecting good parts.

So, with this all said, let’s spend some time working through an actual example where we’re going to imagine we work for a company that produces nuts and bolts. For this particular study we want to know if we’re able to trust the measurement system used to measure the widths of the nuts that have a process tolerance of 0.5 mm.

Gage R&R Study

Since measuring the width of anything results in variables, or continuous data, we’ll be performing a Gage R&R Study.

For this study, we’ve chosen three operators – Bill, Kristen, and Tom – who will all use the same digital caliper to measure the widths of the 15 nuts that were selected.  All three operators will measure these 15 parts two different times.  Each nut has a unique marking that allows the operator to “grasp” it the same way each time to reduce noise in the experiment.  So, in summary, we have 3 operators, 15 parts, and 2 trials.

Tips for Setting Up Your MSA

Setting up a Gage R&R study is important so here are some suggestions.

First, you should randomly select between two and four operators. We find three to work extremely well. Next, the number of parts used should be enough so that when we multiply P, the number of parts, by O, the number of operators, the result is greater than 15.

If this isn’t possible or practical, you’ll want to increase the number of trials accordingly. We must do at least two trials, but for example, if P times O is less than 15, we’d want to do three trials.

We can use some simple math to ensure we set things up properly, but we can’t use math to help with what we feel is the single most important aspect of an MSA, and that’s setting the operators’ minds at ease.

If an operator has never been part of a formal MSA study chances are very good they’re going to be extremely nervous. Some may be downright scared. It’s absolutely critical that you let them know the point of the study is to better understand the process and that no one will be in trouble if they do poorly.

Finally, all the parts should be randomly handed to the operators.  Many times the facilitator will have the parts being measured organized behind a small wall or something so the operator doesn’t know which part is being handed to them.  Once the operator measures the part and announces the result the facilitator documents the results immediately.

Calculating With Minitab

Now, there are many ways to analyze a MSA.  For this article we’re going to show you how to analyze the results using Minitab.  We also share how to analyze a MSA using SigmaXL with our School of Six Sigma.  On a personal note, since I’m a Mac user, I love SigmaXL and use it often.

The first part of the Minitab output is a standard ANOVA table as shown below.  If you’re not sure what any of this means please be sure to check out our videos on inferential statistics.  We cover everything from the difference between the null and alternate hypothesis to what this talk of P-Values and such is all about!

And don’t worry if stats aren’t your thing… I personally promise you will understand it all by the time you finish our course materials.  We also do screen recordings so you know exactly what buttons to press within Minitab or SigmaXL.  No more 3-ring binders full of hundreds of slides with outdated screen shots needed!

Snip20150810_1

OK, as you see above, the Part P-value is 0.0 which means that the Part-to-Part variation is statistically significant. The interaction of Part x Operator is also statistically significant since its P-value is .016.  In most cases, any P-Value less than 0.05 will be assumed significant or, at a minimum, worthy of additional exploration.

Next, here’s the Gage R&R information. There’s definitely lots to take in here but we’ll point out the most important things we need to focus on.

Snip20150810_4

First, the % Contribution section summarizes the percent of the variation that each component contributes to the total variation. The % Study Variation section, which we’ll show in another image below, summarizes the standard deviation percentage of each source to the total standard deviation. And the % Tolerance section summarizes the percentage of variation of each source compared to the part tolerance.

While every organization will need to determine this for themselves, the Automotive Industry Action Group has come up with some suggestions as to how to interpret these values.

AIAG MSA Suggestions Gage R&R

Specifically, they suggest that % Contribution values less than 1% represent an acceptable system. While % Contributions between 1% and 9% represent marginally acceptable systems. Anything over 9% is considered unacceptable.

For % Study Variation and % Tolerance acceptable values are anything less than 10% while marginal systems will be between 10% and 30% and anything over 30% is considered unacceptable.

When we look at the results of this study, we see a Total Gage R&R Percentage Contribution value of 3.5% which could be considered marginally acceptable since it’s between 1% and 9%. Minitab also shows us how this value is derived, namely 2.31% is due to repeatability, or within operator variation, while 1.19% is due to reproducibility, or between operator variation.

These values can help us understand where to focus our attention as we work to improve the system. Minitab also breaks the reproducibility values down by operator and the operator by Part interaction in this section.

Finally, the Part-to-Part variation is also noted. Since our goal is to ultimately measure individual parts, we want to see this variation figure high since this means we’re able to distinguish the difference between parts.

Let’s move on to the next section which includes %Study Variation and %Tolerance Variation.

Snip20150810_5

Here we see our % Study Variation is 18.71% and our % Tolerance is 13.85% which means, again, we’re working with a marginally acceptable measurement system. Minitab also breaks these values down into finer increments here just like we saw in the section above.

Last, but certainly not least, our number of distinct categories is 7. Again, like we mentioned in part 1 of this series, we’d like to see this value greater than 5 which this value obviously is.

Let’s Get Graphical

In addition to these statistics, Minitab provides some nice graphs of this same data.

Snip20150810_6

In the components of variation section, we see a graphical view of the different variation sources. Again, we want to see our Part-to-Part variation higher than everything else since that means we’re able to distinguish between parts.

Next, we also see an R Chart by Operator. This is helpful as it can tell us where individual operators struggled. You can see where Tom, the third operator, obviously struggled with two parts so we may want to visit with him in order to better understand what may have caused this issue.

In a perfect world, we don’t want to see any special cause variation in this R chart, but as long as we intend to identify and counter the root cause of this variation, it’s normally okay to press on unless we see wildly out of control variation across all operators.

The Xbar Chart by Operator graph is an interesting one since contrary to any other control chart we’ve ever looked at in this course, we actually prefer to see special cause variation since this represents part to part variation, which again, is what we ultimately want to measure.

At the top right of the graph we see Data by Part. This can help us identify if one part was harder to measure than another. For example, it seems like part 14 has the most variation. Perhaps there’s something about this part that makes it hard to measure. We also see the Data by Operator, which again, can be very useful information to help us work with each operator when applicable.

Finally, we see the Part x Operator Interaction graph in the lower right corner. If we remember, we saw a P-Value of .016 for this interaction which means it is statistically significant. As we can see in this graph, it seems like the operators struggled a bit with part 9 and 14 so we’d definitely want to take a look at these parts to see if there is anything we can learn and improve.

Conclusion

In summary after looking at the results of this Gage R&R, we’d conclude that this measurement system is marginally capable. Some next steps may include investigating the measurements with high ranges while also trying to understand why some parts were harder to measure than others.

We’d then want to repeat this study as needed. For example, if we gained an understanding of what made part 14 so hard to measure and we’re able to implement a countermeasure, we could repeat the study and see how we do.

We’d also want to repeat the study with different operators while also doing the study with this same group at a later date to ensure that the measurement process is stable over time.

To be sure, MSA shouldn’t be a one and done activity.  If I had my wish every organization would have recurring MSA as part of their Standard Work.  It’s that important.

Thanks for sticking with me on this series.  I hope it was helpful.  If you have any questions please be sure to let me know in the comments section below.  You can also contact me directly using the Contact Form on our website.

Watch This Series in a Single Video

Now then, if video is your thing, or perhaps you need to run through all of this material one more time, we’d encourage you to watch the video below.

When you click on the image you’ll be directed to another part of our website where you can click on the “Gage R&R (15:15)” link.  Doing this will launch the video free of charge.  English, Spanish, and Chinese captions are also available (simply click the CC button).  Enjoy and happy MSAing!

Snip20150811_4

2 Comments

  1. John

    August 14, 2015 - 5:32 am

    your example shows GR&R for operators which is the norm. However, in a highly automated production environment, e.g. IC fabrication, where automated equipment does the measuring, how can GR&R be applied?

    • Ron Pereira

      August 14, 2015 - 2:22 pm

      Same principle applies… but if you only had one “inspector” which could be a machine you likely won’t have reproducibility data. But, you could still do a study with say 15 parts… you’d send each part through 3 different times and get repeatability data.

      At a previous job I had we used to do GR&Rs on the optical inspection machines that measured solder paste bricks. We’d just get 15 PCBs with solder paste on them, label them 1 – 15, and then send them through the machine and focus in on a few specific solder bricks… from this we could determine how repeatable the machine was. To mix it up we may “block” it and have the day shift run the study… then have night shift do the same thing if we timed it right (so solder didn’t dry out)… then we could get reproducibility since “shift” would be the two operators.

      Make sense?