Λίγο χιούμορ γιατί χανόμαστε

The Blindfolded Donkey

Student Leaf approached Master Stem with a question. "Master Stem. My statistical model requires an assumption of normality, but I have an outlier in the data. Should I not remove the outlier so I can satisfy this assumption?"

Master Stem replied, "It is a perilous journey you are undertaking and I understand your caution. When I last crossed the mountains, I was also cautious. The donkey I rode upon knew the path well, but during the most dangerous part of the journey, I blindfolded him because if he could not see the loose rocks on the path, he would not slip on them.

The Other Ninety Nine Samples

Master Stem was inspecting his students work. Student Leaf displayed a 95% confidence interval from a randomized control trial. "What does this confidence interval tell you?" Master Stem inquired.

Student Leaf replied, "This interval contains the value of zero, which provides evidence that there is no difference between the treatment group and the control group."

"And what can you tell me about the truth of this interval, Student Leaf."

"We cannot make a probability statement about this particular interval, Master Stem. We can say, however, that if I collected 100 samples and computed confidence intervals using this method, about 95% of them would contain the true mean difference."

"Very interesting, Student Leaf. Now that you have collected the first sample, when do you plan on collecting the other 99?"

A Visit From A Thief

Student Leaf asked Master Stem, "I have heard some of my teachers say 'I accept the null hypothesis' and others say 'I fail to reject the alternative hypothesis.' Why do these words not mean the same thing?"

Master Stem scoffed "Bah, these teachers of yours! They do not understand probability, every single one of them."

Student Leaf asked "But what does probability have to do with it? The null hypothesis is true or it is not. Where is the probability?"

Master Stem replied "All hypotheses have probability if you consider them carefully. Consider a thief who visits your house. You do not see anything unusual during the visit, but after the thief leaves, you find that your jewelry has been stolen."

Student Stem replies "Then the thief must have taken the jewels."

"Are you so sure? The night that the thief visited, was the night that you had a great celebration, and one hundred people came and visited. Only after everyone had left did you notice that the jewels were gone."

"Then the thief probably did not take the jewels, Master Stem. It was probably one of the other guests."

"But you had the authorities investigate. They searched the houses of all the other guests and these searches did not yield the jewels."

"Then the other guests are innocent. It must have been the thief because only his house could have contained the jewels."

"That is reasonable, perhaps, if you believe that there is no other hiding place for the jewels. But I forgot to mention that the authorities are very busy and they did not search very carefully in any one house."

"Then I am not sure what to believe, Master Stem."

"You never believe anything with certainty, Student Leaf. A thief visits your house. You have an initial level of mistrust because I have called this person a thief. Think about what this means. This person has been known to steal from others, so you have a high degree of belief that he may have stolen from you as well. Now I provide you with more information. Each piece of information either increases or decreases your degree of belief. But no matter how much data you accumulate, you will never have certainty, just a degree of belief."

"How then, Master Stem, am I able to make any decision, if I never have certainty."

"There is balance, Student Leaf. How much do you lose if the thief is allowed to get away with the crime? What is the consequence of the loss of face when your accusations of thievery are proven false?"

"I do not know if I can assign a value to these things, Master Stem."

"Such foolishness. You want to make very important decisions about these jewels, but not once did you think to assess their value. Were they but a trifle? Then be very careful not to risk your reputation over such a small thing. Were the jewels of great beauty and incomparable value? Then do not let any small thing stand in the way of possibly recovering such a great treasure."

An analogy to understand power (chapter 20, pages 147-148)
This analogy helps illustrate the concept of statistical power (Hartung, 2005).

You send your child into the basement to find a tool. He comes back and says, “It isn’t there.” What do you conclude? Is the tool there or not? There is no way to be sure, so the answer must be a probability. The question you really want to answer is, “What is the probability that the tool is in the basement?” But that question can’t really be answered without knowing the prior probability and using Bayesian thinking (see Chapter 18). Instead, let’s ask a different question: “If the tool really is in the basement, what is the chance your child would have found it?” The answer, of course, is “it depends.” To estimate the probability, you’d want to know three things:

■How long did he spend looking? If he looked for a long time, he is more likely to have found the tool. This is analogous to sample size. An experiment with a large sample size has high power to find an effect.
■How big is the tool? It is easier to find a snow shovel than the tiny screw driver used to fix eyeglasses. This is analogous to the size of the effect you are looking for. An experiment has more power to find a big effect than a small one.
■How messy is the basement? If the basement is a real mess, he was less likely to find the tool than if it is carefully organized. This is analogous to experimental scatter. An experiment has more power when the data are very tight (little variation).

If the child spent a long time looking for a large tool in an organized basement, there is a high chance that he would have found the tool if it were there. So you can be quite confident of his conclusion that the tool isn’t there. Similarly, an experiment has high power when you have a large sample size, are looking for a large effect, and have data with little scatter (small standard deviation). In this situation, there is a high chance that you would have obtained a statistically significant effect if the effect existed.

If the child spent a short time looking for a small tool in a messy basement, his conclusion that “the tool isn’t there” doesn’t really mean very much. Even if the tool were there, he probably would have not found it. Similarly, an experiment has little power when you use a small sample size, are looking for a small effect, and the data have lots of scatter. In this situation, there is a high chance of obtaining a conclusion of “statistically significant even if the effect exists.