Regarding statistics - I agree, in one way or another we need statistical methods. And there are cases, where without statistics we could say nothing about problem at hand. For example,- I can’t imagine better alternative to Maxwell distribution - which helps to predict properties of gases->
http://en.wikipedia.org/wiki/Maxwell%E2%80%93Boltzmann_distribution
According to wikipedia- Maxwell distribution was first-ever statistical law in physics. After that the whole new physics field based on statistics borned ->
http://en.wikipedia.org/wiki/Statistical_mechanics
And there are a lot of physical theories which uses statistical or probabilistic approach, take a look at quantum mechanics or chaos theory… Without statistics whole fields of physics should be destroyed, which would be ridiculous given the fact that these fields gave remarkable ideas and new technological inventions. So statistics should survive.

Consider the modifying phrase in this sentence: “Even when performed correctly, statistical tests are widely misunderstood and frequently misinterpreted.” To me this demonstrates that, up to this point, Siegried is criticizing the *incorrect* use of statistical tests. That is, there are two problems: a) incorrect application of statistical methods, and b) misinterpretation of correctly applied methods. Neither is an indictment of statistics per se. The rest of the article seems to confirm this interpretation.

One of the money grafs: “Correctly phrased, experimental data yielding a P value of .05 means that there is only a 5 percent chance of obtaining the observed (or more extreme) result if no real effect exists (that is, if the no-difference hypothesis is correct). But many explanations mangle the subtleties in that definition. A recent popular book on issues involving science, for example, states a commonly held misperception about the meaning of statistical significance at the .05 level: “This means that it is 95 percent certain that the observed difference between groups, or sets of samples, is real and could not have arisen by chance.””

This has tripped me up, too. It’s a subtle point.

By the way, you write “when a result is reported as statistically significant at the 1-sigma level (or in other words with a P value of 0.05)” — isn’t the 1-sigma level closer to 68%, not 95%?

I read Brian’s post as well as the article and frankly I’m not surprised by either. The reality is that statistics and probability are not well understood, even by experts. And, as Mr. Siegfried notes, many scientists who compute statistics haven’t the faintest idea of what they are doing or why. They run some software, get a result, and write a report. The public is expected and encouraged to accept the result.

The reality is that analysts (and I am one and have worked on system testing) use statistics because it was mandated somewhere by someone to do so. The applicability and validity are not important; what’s important is to get a number. Further, the distributions on which statistics rest are completely disregarded in most studies. You simply have to read reports to see this and Mr. Siegfried provides ample examples both of mis-applied statistics and wrong interpretations of results.

To see that probability is so greatly misunderstood one has only to look at the Monty Hall problem. Wikipedia has a good entry on this and, frankly, the results are counter-intuitive such that mathematicians have argued about it furiously. To see more of this, there are articles in, I think, the current issue of either Mathematics Magazine or the College Mathematics Journal. Plus, the Mathematical Intelligensia just had an article about Bayesian methods that also shows the counter-intuitive nature of this approach.

With reference to statistics, we can see the shortcomings in a simple example if we look at a data, say some sampling, and compute (blindly) the mean and standard deviation. These calculations are simple to do, but to use them as a descriptor means the data have to be Gaussian and sometimes data are and sometimes they are not. Yet, how often do we see a plot of data with the Gaussian overlaid so that one can see (and with analysis, know) if the Gaussian is the right distribution. Not often. (For that matter, how often does a reader even have the raw data for his/her own analysis?)

In short, Mr. Siegfried is doing science a service by pointing out the issues with statistics and probability. Hopefully, scientists will be more careful to report just what a statistic implies AND what it does not imply.

Brian, you, too, are doing us all a service with your posts and comments.

Frak: the lead-in of an article lays out the argument the author is trying to make, and is absolutely the most important part of any piece. If the rest of the article is inconsistent with the lead-in (which I agree is true in many spots), then the author has failed to support his thesis.
No one is saying that statistical methods do not have limitations. The so-called “facts” in the article are subject to similar limitations, and in most cases, he only gave one side of the issue. I find the analysis to be superficial - and he offers no semblance of a solution.

On a more serious note: Siegfried’s lead-in does make several inflammatory statements. The rest of the article is factual and discusses well-known mis-uses of statistical methods, including summaries and citations of more thorough research on the issue. I do not see how someone reading the article in its entirety could take your critique seriously.

But I believe there’s plenty of room for improvement in the kinds of statistical methods we use. Traditional statistics is stretched to its limits or beyond by some new types of data.