The Challenge of Eliminating Bias from Scientific Methodology

Back in April of this year, Scott Alexander wrote a brain-busting meditation on the challenge of eliminating bias from scientific methodology.

I am a humanist by training, statistically illiterate--and, to be frank, practically innumerate--but darn it if this piece doesn't get my philosophy of science juices flowing and coalescing into a roaring stream!

Alexander starts with the observation that a parapsychologist has performed a well-done meta-analysis of parapsychological research, which gives evidence for a small but statistically significant 'psi' factor in his field of research.

Alexander's point is not that we should believe psi is real, but rather something even more disturbing--if a meta-analysis of this high quality can still be wrong-headed, what are we to make of the rest of science? And how do we control for the subtle methodological problems which plague even the best of scientific studies and meta-analyses?

Alexander offers a glorious trip through the inner workings, the nooks and crannies of scientific methodology, with himself as thoughtful guide. Alexander concludes in part that there is a hidden "experimenter" bias which shows up in scientific research of all kinds. The most striking example occurs in a parapsychological study conducted by two researchers, who jointly agreed on a common methodology and who supervised each others' collection and interpretation of data--but who still achieved contrary results, which corresponded with their prior beliefs. This is probably only part of the explanation for why many scientific research findings are biased or unreliable, however.