Comparisons between groups are made all of the time - Are this year's movies better than last? Do consumers prefer this product over that? Do Brand X tires last longer than Brand Y... How do we know, though, if conclusions from comparisons such as these are actually significant; if one group really is better than another? How do we know if differences aren't simply the result of chance? These questions are at the heart of statistical comparison. And as they suggest, the answers are not black or white; right or wrong. They involve shades of gray; probabilities, not certainties. The simulation below gets at the logic involved in answering such questions.
Let's go back to the comparison of potential NFL quarterbacks from 1987 and 2018 and ask the question, "Were quarterbacks attending the NFL combine in 2018 generally the same weight as those three decades ago in 1987?" Here is the data again:
Quarterback Weights - NFL Combine
Obviously the two sets of values are somewhat different. The mean values certainly differ. The standard deviations are closer, though. How confidant can we be that the differences are real and not simply the result of chance? The following simulation is designed to give you a sense of the mathematical logic involved.
Suppose that there is, in fact, no difference between the two groups; that is, that the difference between their two means is really 0; that the difference of 220.9 - 201.5 = 19.4 is just the result of chance one year to the next. Imagine that the results were pooled and divided randomly into new groups of approximately equal size and that the comparison was then repeated. And suppose that we repeated this experiment not once, but 10, 100, 1000, or even 10,000 times. What would the results look like? Select the number of times to repeat the process and click on the RUN SIMULATION button below. The results are presented in a histogram showing the differences between the means of the two randomly selected groups.
Mean Differences Simulation
1) Run the simulation for 10 trials several times. The graph shows the resulting differences - never the same. Go down the list in order repeating the simulation the indicated number of times. Describe the graphic pattern you observe as you increase the number of trials. Describe what you suppose would happen if you could repeat the simulation 100,000 times or 1,000,000 times or an infinite number of times.
2) Run the simulation with 10,000 trials several times. Notice that the results are never exactly the same, but the shape of the histogram follows roughly a bell curve and that about 95% of the results show a difference in means between -12 and 12. A difference of 19 is extremely unlikely. We can say with a high degree of confidence that this is a significant difference and not simply the result of chance.
3) Suppose that the difference in mean quarterback weights was 6 pounds. 8 pounds. 10 pounds. 14 pounds. How large a difference would you have to see to say that there is, in fact, a significant difference. Explain your reasoning.
NFL combine statistics from NFL Combine Results. Downloaded August 29, 2019.