[TLEILAXU]

Good article from nautil.us highlighting flawed statistical thinking
http://nautil.us/issue/74/networks/the-flawed-reasoning-behind-the-replication-crisis

Here are three versions of the same story:

1. In the fall of 1996, Sally Clark, an English solicitor in Manchester, gave birth to an apparently healthy baby boy who died suddenly when he was 11 weeks old. She was still recovering from the traumatic incident when she had another baby boy the following year. Tragically, he also died, eight weeks after being born. The causes of the two children’s deaths were not readily apparent, but the police suspected they were no coincidence. Clark was arrested and charged with two counts of murder. The pediatrician Roy Meadow, inventor of the term “Munchausen Syndrome by Proxy,” testified at the trial that it was extremely unlikely that two children from an affluent family like the Clarks would die from Sudden Infant Death Syndrome (SIDS) or “cot death.” He estimated the odds were 1 in 73 million, which he colorfully compared to an 80:1 longshot winning the Grand National horse race four years in a row. Clark was convicted and sentenced to life in prison. The press reviled her as a child murderer.

...

The mathematical lens that allows us to see the flaw in these arguments is Bayes’ theorem. The theorem dictates that the probability we assign to a theory (Sally Clark is guilty, a patient has cancer, college students become less theistic when they stare at Rodin), in light of some observation, is proportional both to the conditional probability of the observation assuming the theory is true, and to the prior probability we gave the theory before making the observation. When two theories compete, one may make the observation much more probable, that is, produce a higher conditional probability. But according to Bayes’ rule, we might still consider that explanation unlikely if we gave it a low probability of being true from the start.

So, the missing ingredient in all three examples is the prior probability for the various hypotheses. In the case of Sally Clark, the prosecution’s theory was she had murdered her children, itself an extremely rare event. Suppose, for argument’s sake, by tallying up historical murder records, we arrived at prior odds of 100 million to 1 for any particular mother like her to commit double infanticide. That would have balanced the extreme unlikelihood of the observation (two infants dying) under the alternative hypothesis that they were well cared for. Numerically, Bayes’ theorem would tell us to compare:

(1/73,000,000) * (99,999,999/100,000,000)  vs. (1) * (1/100,000,000)

We’d conclude, based on these priors and no additional evidence aside from the children’s deaths, that it was actually about 58 percent likely Clark was innocent.

The problem, though, is the dominant mode of statistical analysis these days isn’t Bayesian. Since the 1920s, the standard approach to judging scientific theories has been significance testing, made popular by the statistician Ronald Fisher. Fisher’s methods and their latter-day spinoffs are now the lingua franca of scientific data analysis. In particular, Google Scholar currently returns 2.85 million citations including the phrase “statistically significant.” Fisher claimed signficance testing was a universal tool for scientific inference, “common to all experimentation,” a claim that seems borne out by its widespread use across all disciplines.

And this part in particular will likely make me even more unpopular on this forum

Just a few of the other casualties of replication include:

• The study in 1988 by Strack, Martin, and Stepper on the “facial feedback hypothesis:” when people are forced to smile, say by holding a pen between their teeth, it raises their feeling of happiness.
• The 1996 result of Bargh, Chen, and Burrows in “social priming,” claiming, for example, when people are exposed to words related to aging, they adopt stereotypically elderly behavior.
• Harvard Business School professor Amy Cuddy’s 2010 study of “power posing:” the idea that adopting a powerful posture for a couple of minutes can change your life for the better by affecting your hormone levels and risk tolerances

.

[Wilshire]

I think we all know there exists a replication 'crisis', and its almost equally apparent that when you misapply math you get out garbage.

The article suggests basically that we ought to verify our results rather than relying on statistics. Not exactly groundbreaking, as that is the basis of science to begin with.

[themerchant]

Feynman called this cargo-cult science back in the 60's 70's. If it doesn't pass the reality test, i.e. experimentation it is wrong.

[TLEILAXU]

I think we all know there exists a replication 'crisis', and its almost equally apparent that when you misapply math you get out garbage.

The article suggests basically that we ought to verify our results rather than relying on statistics. Not exactly groundbreaking, as that is the basis of science to begin with.

How do you verify results without statistics? The point is that significance testing and p-values are so deeply ingrained into the scientific culture and misuse is so high that transparency and common sense suffer as a result.

[sciborg2]

Is holding a pen between your teeth while power posing something very popular on this forum?

[TLEILAXU]

Is holding a pen between your teeth while power posing something very popular on this forum?

Yeah, the guys at the top of the hierarchy here are big fans of self-help books, it's best not annoy them or you might get banned for a week (or more) like I did.

[Wilshire]

Is holding a pen between your teeth while power posing something very popular on this forum?

Not that I'm aware of. As the article says, though without citation lol, both have been unverifiable.

I do find it amusing that an article thats discussing  unverified scientific claims did so without (almost any) citations.

[TaoHorror]

Is holding a pen between your teeth while power posing something very popular on this forum?

Yeah, the guys at the top of the hierarchy here are big fans of self-help books, it's best not annoy them or you might get banned for a week (or more) like I did.

You did? When? I didn't know that - wear it with pride, my friend! At risk of being banned myself, would be cool to know your "transgression" - pm me if you're not wanting to share publicly ( wouldn't blame you ).

I have a rant somewhere buried in this forum when I was yelling at ( Wilshire? ) that statistics was the worse for argumentation - I know, this is about science, but I'm not a scientist, so taking you'alls word for it here.

[sciborg2]

I have a rant somewhere buried in this forum when I was yelling at ( Wilshire? ) that statistics was the worse for argumentation - I know, this is about science, but I'm not a scientist, so taking you'alls word for it here.

I've been thinking about this - stats oftentimes are only going to be convincing as buttresses for things you already believe otherwise you'll hunt and peck for errors. Whether it's police brutality or workplace bias or psychic powers, people already know the "truth" and as such your study's correctness is measured by that personal gnosis.

Solutions turn on the irrational, qualitative raw feeling or at least to get to what the stats say you have to get past those feelings.

[Wilshire]

I have a rant somewhere buried in this forum when I was yelling at ( Wilshire? ) that statistics was the worse for argumentation - I know, this is about science, but I'm not a scientist, so taking you'alls word for it here.

I've been thinking about this - stats oftentimes are only going to be convincing as buttresses for things you already believe otherwise you'll hunt and peck for errors. Whether it's police brutality or workplace bias or psychic powers, people already know the "truth" and as such your study's correctness is measured by that personal gnosis.

Solutions turn on the irrational, qualitative raw feeling or at least to get to what the stats say you have to get past those feelings.

That can be generalized even further. I don't think most people can be convinced on a contrary view to the one they have with direct confrontation. In fact, this is an obvious and known phenomenon. Whole industries are based on this, from cigarettes and phony medication, to advertisements successfully selling you products you dont need or want.

[sciborg2]

That can be generalized even further. I don't think most people can be convinced on a contrary view to the one they have with direct confrontation. In fact, this is an obvious and known phenomenon. Whole industry are based on this, form cigarettes and phony medication, to advertisements successfully selling you products you dont need or want.

Yeah confrontation is more about the satisfaction of the person haranguing their interlocutors than actually convincing others. To tie this into stats, confrontation based on asymmetries gleaned from social science research has probably reached its limits.

Political solutions should turn on principles tied to moral qualia, and these principles (IMO anyway) work best when they apply to everyone in the same way. Unfortunately this kind of appeal to transcendent principle seems to have fallen to tribal morality where we separate what's good for the goose vs. the gander.