Measurement System & Sampling Horror Stories

Yesterday, during our weekly Black Belt coaching call I was talking to my friend Joni (hope it’s ok to mention you, Joni!) about the importance of sampling and measurement systems.  As some folks know – especially those I have coached over the years – I am extremely passionate about both topics.

Measurement System Horror Stories

I shared a few measurement system horror stories with Joni, including the one where a machinist was measuring a highly precise part with some calipers.  I asked if they had ever done a measurement system analysis on their measurement process and they replied, “Nope.”  We quickly set one up with some parts and a few machinists and conducted the study.

About an hour later I sat staring at my Minitab results and slowly looked up at the eager plant manager and explained, “Sorry to say… but you might as well stop wasting the time to measure these parts using your current process since you have no way of distinguishing between good and bad parts.”  For the stats nerds (like me) out there…their number of Distinct Categories was 1.  Not good.

We, of course, set out to make improvements to the measurement process…but the scary thing is this company had been working like this for a LONG TIME and desperately wanted to know how to improve their process since they were being ravaged with defects.  The fact of the matter was their measurement system variation was so big they literally couldn’t see their true process variation.  The story ended well as we sorted things out with the measurement system and ultimately improved their process.

Oh, Sampling

Joni then asked me about sampling.  Oh, sampling.  Where to start?  As I type this on July 16 the world is dealing with a few examples of sampling in action.

First, as it relates to the Coronavirus, we’re testing more and more people, which is actually a form of sampling used to best estimate the overall rate of infection of the population and to, obviously, help people get well and not infect others.

This so-called positivity rate is the closest thing we have to a true “infection rate” which can, in theory, only be calculated if every soul on earth was tested at the same time using the same, reliable measurement system.  Side note: Bad Covid-19 measurement systems are a story for another day!

But, as we all know, testing every soul on earth – or even in my small town – on the same day isn’t currently possible.  So, we have to sample the population and estimate the rate of overall infection using the positivity rate.  The problem is if we don’t do a good job of randomly sampling the population we may not have an accurate understanding of what’s actually going on in the overall population.

Put another way, we want the positivity rate to be similar to what we’d see if every soul on earth was able to be tested at the same time, yielding the overall population infection rate.  Definitely not an easy task so, trust me, I am not throwing stones here.

Oh, and in case you haven’t heard…there’s an election coming up. Sampling is a huge part of the process leading up to the election.  We take polls in order to predict who will win.  Your favorite candidate may be up.  They may be down.  But, in the end, if we don’t do a good job sampling an accurate ratio of Democrats, Republicans, Independents, and anyone else planning to vote in November, these polls may not yield accurate results.

So, no matter if you’re trying to determine the percentage of folks sick with a virus or who will win the big election, you need to do an excellent job of randomly sampling the population.

Two Most Important Things

As my conversation with Joni started to wind down, I explained…if there’s two things I hope you take from this certification program it’s to:

1. Ensure you always do your best to randomly sample the population when conducting an experiment, running a study, doing a survey, etc.
2. Ensure you can trust your measurement system by validating it with some sort of measurement system analysis (i.e. Gage R&R or Attribute Agreement Analysis)

Joni then closed with, “Ron, you should tell others about this!”  Hopefully this article helps.

---