VOX DEI-BATE #13: JUSTIFYING THE INTUITION

Vox begins his most recent response with a sort of general comment that refers to a JAMA study concerning the lack of replication in published research regarding alleged linkages between ‘gender and genes’. Vox has previously posted about this here, and the original citation he refers to is a Wall Street Journal gloss of the study in question, still available here.

I’m not too impressed. I think Vox either misreads or overstates the gravity of the situation described, both in terms of the culture of science and the applicability of these findings, and I’ll provide my reasons for both of those claims.

First, from the standpoint of the culture of science, Vox fails to place the research criticized in the JAMA study in the proper context. What is the risk here to the actual conduct of science? Surely Vox doesn’t mean to imply that poorly-designed research reported in isolated journal articles will somehow lead to something fundamentally amiss in the present paradigm. They are merely findings presented in some provisional context—if not, they wouldn’t have seen the light of day, poor design or no. Without replication, none of the results described are going to end up in the textbooks as facts that residents in reproductive medicine are going to be held responsible for, much less first-year biology students. The real danger is not that some false dogma might arise, but that time and energy might be squandered on a poorly-conceptualized research program.

One could argue that the WSJ article reference by Vox is evidence that the danger is greater than I am suggesting, that maybe there is some major crisis of confidence brewing in science. Sorry, but I don’t think that counter-argument will wash, either, because the WSJ article also fails to place the JAMA study in the proper context.

That study, which raises methodological questions about how well peer review is excluding poorly-conceptualized research may be novel enough that it merits attention in the mind of some WSJ staff writer, but it is hardly revolutionary. Vox seems to think that he has stumbled on some stunning new finding when he trumpets his ‘skepticism about the current state of science’ in light of these findings. Ho ho! ‘Skepticism about the current state of science’, as far as I can see, is the default position amongst workers in the field. I have fond memories of a graduate seminar in biogeography that I was privileged to participate in a few months back, wherein we would read papers in the literature and (surprisingly often) raise questions about the assumptions and potential flaws in the protocols described. The students did this routinely, and with no sense of scandal. The periodic reevaluation of scientific practice is normal, part and parcel of the brutal competition of ideas that is how science is supposed to be done. The AAAS, for example, is currently investing significant energy in revisiting the peer review process, which is looking increasingly dysfunctional and creaky in this Internet age.*

What about the applicability of the findings? I think Vox stumbles here as well. Can we justify a general skepticism toward all scientific generalizations, given that peer review is up for review itself? Again, context suggests otherwise. Let’s not confuse this or that finding from cutting-edge research in this or that specialized topic with the sort of broad, powerful and well-tested models in which this work might’ve been nested. Surely Vox is not arguing that because specific replicable linkages between gender and genes seem hard to come by in the research in question, that such things as ‘gender’ or ‘genes’ do not exist, or that no relationship at all obtains between them. In fact, I would predict that a likely flaw in many of the articles is a failure to properly characterize that relationship!

Finally, what bearing does this argument actually have on the question of whether TENS is or isn’t the best scientific model for the diversity of life? None that I can see. This is just an attempt by Vox to buttress what is by his own admission an ‘intuition’ about evolutionary theory. To me, Vox’s opening salvo in his most recent post can be paraphrased as follows: “Scientists make mistakes, as in this case, so I feel justified in remaining skeptical about evolutionary theory.”

Of course, there’s more to Vox’s post than that, and I’ll reply as I have time.

* = A recent summary of AAAS-sponsored work in this regard can be found in the July 15th issue of Science, which can be accessed with a subscription on-line here.