Psychology Replication Study

Many of you may have heard of the recent psychology replication study, published in Science, in which researchers attempted to replicate 100 hand-picked psychology studies, and were only able to successfully replicate 39 of them.

I am a huge fan of this study, among other things because it encourages other scientists to attempt replication (which everyone agrees is not done enough in the sciences). The result also opens up a bunch of cool interpretive questions about scientific method and statistical analysis.

One obvious mistake to avoid is concluding based on this study that only 39% of the original studies were "correct," in some sense of the word. Just as some of the initial 100 studies probably really were flawed and gave misleading results (which I believe can be thought of as "false positives" without being too misleading), this is also probably true of some of the failed replications as well (which can analogously be thought of as "false negatives").

But can we get more precise with the implications, even to a first approximation? I have an amateur interest in philosophy of science, but am wholly ignorant of experimental design and statistical analysis. So I could use a hand (hence this post).

Can we use Bayesian theory to get some clarity? Of course, we are going to have choose some semi-arbitrary numbers, like the probability that each of the initial findings is a false positive, the probability that each of the initial findings is a false negatives, and the probabilities that each of the attempted replications is a false positive or a false negative.

Apart from the general probability of false positives and false negatives with both the initial findings and the attempted replications, there are more particular factors to consider. One is the expertise of the experimenters; replication may be difficult, because specialized skills and practice may be necessary to successfully create the controlled conditions which will show the initial experimental result. There is also the obvious question of confirmation bias among the researchers who authored the initial studies. And so on.

The Flip Side of Critical Thinking

Have philosophy professors been neglecting half of their discipline? Jennifer M Morton, professor of philosophy at City College New York, argues that in addition to critical thinking, philosophy should be thought of as teaching "constructive imagination." 

Her article is a little wordy, and I'm not sure she sufficiently expounds and expands upon the promise of this notion, but it is indeed promising. I sense a new movement just around the corner for "Constructive Imagination" in liberal arts courses. Just please, could we avoid the frenzied use of buzzwords and self-congratulatory smarm, this one time? Morton's idea is too important to be gradually smothered through the usual academic mechanisms and machinations.

Via Daily Nous.

Scott Alexander on Political Activism

Scott Alexander makes a powerful case against political activism and in favor of direct giving:

Five million people participated in the #BlackLivesMatter Twitter campaign. Suppose that solely as a result of this campaign, no currently-serving police officer ever harms an unarmed black person ever again. That’s 100 lives saved per year times let’s say twenty years left in the average officer’s career, for a total of 2000 lives saved, or 1/2500th of a life saved per campaign participant. By coincidence, 1/2500th of a life saved happens to be what you get when you donate $1 to the Against Malaria Foundation. The round-trip bus fare people used to make it to their #BlackLivesMatter protests could have saved ten times as many black lives as the protests themselves, even given completely ridiculous overestimates of the protests’ efficacy. 

The moral of the story is that if you feel an obligation to give back to the world, participating in activist politics is one of the worst possible ways to do it. Giving even a tiny amount of money to charity is hundreds or even thousands of times more effective than almost any political action you can take. Even if you’re absolutely convinced a certain political issue is the most important thing in the world, you’ll effect more change by donating money to nonprofits lobbying about it than you will be [sic] reblogging anything.

Bayes' Theorem and Cancer Screening

If you get a positive result on a mammogram, what is the chance that you actually have cancer?

This video explains how Bayes' theorem can be used to answer this question. (Note: you will still have to do some research to get the exact rate of false negatives, rate of false positives, and rate of people in your age range who have cancer. But the numbers used in the video are approximately correct and the results are very revealing, both about Bayes' theorem and about cancer tests.)

And here is Julia Galef explaining how Bayes' theorem has changed the way she things in everyday life.

The Challenge of Eliminating Bias from Scientific Methodology

Back in April of this year, Scott Alexander wrote a brain-busting meditation on the challenge of eliminating bias from scientific methodology.

I am a humanist by training, statistically illiterate--and, to be frank, practically innumerate--but darn it if this piece doesn't get my philosophy of science juices flowing and coalescing into a roaring stream!

Alexander starts with the observation that a parapsychologist has performed a well-done meta-analysis of parapsychological research, which gives evidence for a small but statistically significant 'psi' factor in his field of research.

Alexander's point is not that we should believe psi is real, but rather something even more disturbing--if a meta-analysis of this high quality can still be wrong-headed, what are we to make of the rest of science? And how do we control for the subtle methodological problems which plague even the best of scientific studies and meta-analyses?

Alexander offers a glorious trip through the inner workings, the nooks and crannies of scientific methodology, with himself as thoughtful guide. Alexander concludes in part that there is a hidden "experimenter" bias which shows up in scientific research of all kinds. The most striking example occurs in a parapsychological study conducted by two researchers, who jointly agreed on a common methodology and who supervised each others' collection and interpretation of data--but who still achieved contrary results, which corresponded with their prior beliefs. This is probably only part of the explanation for why many scientific research findings are biased or unreliable, however.

Hypotheses about Motivated Reasoning; Or, Why Most Debates Are Not About Truth

A psychiatrist blogging under the nomme de plume of 'Scott Alexander' has a fantastic recent blog post about the psychology of motivated reasoning, specifically as it applies to emotionally charged debates about politics, religion, and so forth.

His main thesis is that most such debates are not really about meaningfully answerable questions, but are instead competitions in which each side tries to associate its own position with key words that have positive emotional associations and the opposing position with key words that have negative emotional associations. The purpose of such debates is not to prove that a given thesis is true or false, but rather to create pro or con attitudes in the audience toward a given cause or concept, through the strategic use of emotionally-laden verbal associations. In other words, the purpose of such debates is to secure loyalty to a cause, not to secure belief in a proposition.

From the blog post:

This sort of conflation between a cause and its supporters really only makes sense in the emotivist model of arguing. I mean, this shouldn’t even get dignified with the name ad hominem fallacy. Ad hominem fallacy is “McCain had sex with a goat, therefore whatever he says about taxes is invalid.” At least it’s still the same guy. This is something the philosophy textbooks can’t bring themselves to believe really exists, even as a fallacy. 

But if there’s a General Factor Of McCain, then anything bad remotely connected to the guy – goat sex, lying campaigners, whatever – reflects on everything else about him.
This is the same pattern we see in Israel and Palestine. How many times have you seen a news story like this one: “Israeli speaker hounded off college campus by pro-Palestinian partisans throwing fruit. Look at the intellectual bankruptcy of the pro-Palestinian cause!”  

It’s clearly intended as an argument for something other than just not throwing fruit at people. The causation seems to go something like “These particular partisans are violating the usual norms of civil discussion, therefore they are bad, therefore something associated with Palestine is bad, therefore your General Factor of Pro-Israeliness should become more strongly positive, therefore it’s okay for Israel to bomb Gaza.” Not usually said in those exact words, but the thread can be traced.