Hypothesis Testing

Originally published February 27, 2007

One of my favorite statistical tools is hypothesis testing. We can use hypothesis testing for many purposes.

For example, we would use the popular 2-sample t-test when we have two samples of variable data and want to understand if they represent different populations, statistically speaking, of course.

State the Null and Alternative Hypotheses

The first order of business when completing hypothesis tests is to state the null hypothesis (Ho) and the alternative hypothesis (Ha).

The null hypothesis is the statement of no change. I always remember it like this, “Ho hum… there is no difference here.”

Conversely, the alternative hypothesis is the statement of change. Just remember, “a Ha, there is a change!”

In our 2-sample t-test example, Ho would be that the means are equal, and Ha would be that the means are not equal. It’s as simple as that.

Collect Data and Run Test

Next, we need to carefully collect some data and run the actual hypothesis test. You can program spreadsheets to do this test, or use a standard off-the-shelf software package to do it for you.

We Never “Accept” Anything!

When we run the hypothesis test, we will obtain a p-value. We use this P value to determine whether enough evidence exists to REJECT the null hypothesis.

I emphasize REJECT, since the most common error I see people make is speaking of “accepting” a null hypothesis.

We never accept a null hypothesis for the same reason we never prove someone innocent in the US judicial system. Instead, we prove someone guilty or not guilty. So with hypothesis testing, we either reject or fail to reject the null hypothesis.

If P is Low, Ho must Go!

Now, the P-value is the probability of incorrectly rejecting the null hypothesis. Since we are making important decisions with this P value, we tend to err on the safe side.

Typically, if the P value is less than 5%, we reject the null hypothesis. If the P value is greater than 5%, we fail to reject the null hypothesis.

If all this makes your head hurt, no worries. Just remember this saying, “If P is low, Ho must go. If P is high, Ho is the guy!”

Alpha Risk Explained

Why 5%?  The standard is usually to go with 5% since this the risk most people are willing to take at being wrong. This is also why you often hear about 95% confidence intervals.

If you are sending people to the moon or testing something ultra serious you may consider tightening this “alpha value” as it is called to something like 1% or 2%. I will resort to the response any good Black Belt should give when asked what alpha value to use…it depends!

Until next time, I wish you all the best on your journey towards continuous improvement.

---