Monday, October 21, 2013

Is ecology really failing at theory?

“Ecology is awash with theory, but everywhere the literature is bereft”. That is Sam Scheiner's provocative start to his editorial about what he sees as a major threat to modern ecology. The crux of his argument is simple – theory is incredibly important, it allows us to understand, to predict, to apply, to generalize. Ecology began as a study rooted in system-specific knowledge or natural history in the early 1900s, and developed into a theory-dominated field in the 1960s, when many great theoreticians came to the forefront of ecology. But today, he fears that theory is dwindling in importance in ecology. To test this, he provides a small survey of ecological and evolutionary journals for comparison (Ecology Letters, Oikos, Ecology, AmNat, Evolution, Journal of Evolutionary Biology), recording papers from each journal as either containing no theory, being ‘theory motivated’, or containing theory (either tests of, development of, or reviews of theory). The results showed that papers in ecological journals on average include theory only 60% of the time, compared to 80% for evolutionary papers. Worse, ecological papers seem to be more likely to develop theory than to test it. Scheiner’s editorial (as the title makes clear) is an indictment of this shortcoming of modern ecology.

Plots made based on data table in Scheiner 2013. Results combined for all evolution and all ecology papers.
The proportion of papers in each category - all categories starting with
 "Theory" refer to theory-containing papers.
Plots made based on data table in Scheiner 2013. Results for papers from individual journals.
The proportion of papers of each type - all categories starting with
 "Theory" refer to theory-containing papers.
This is not the kind of conclusion that I find myself arguing against too often. And I mostly agree with Scheiner: theory is the basis of good science, and ecology has suffered from a lack of theoretical motivation for work, or pseudo-theoretical motivation (e.g. productivity-diversity, intermediate diversity patterns that may lack an explanatory mechanism). But I think the methods and interpretation, and perhaps some lack of recognition of differences between ecological and evolutionary research make the conclusions a little difficult to embrace fully. There are three reasons for this – first, is this brief literature review a good measure of how and why we use theory as ecologists? Second, do counts of papers with or without theory really scale into impact or harm? And third, is it fair to directly compare ecological and evolutionary literature, or are there differences in the scope, motivations, and approaches of these fields?

If we are being truly scientific, this might be a good time to point out that The 95% confidence intervals for the percentage of ecology papers with theory overlap with the confidence intervals for the percentage of evolutionary papers with theory suggesting the difference that is the crux of the paper is not significant. [Thanks to a commenter for pointing out this difference is likely significant]. While significant at the 5% level, the amount of overlap is enough that whether this difference is meaningful is less clear. (I would accept an argument that this is due to small sample sizes though). The results also show that choice of journal makes a big difference in terms of the kinds of paper found within – Ecology Letters and AmNat had more theoretical papers or theory motivated papers, while Oikos had more tests of theory and Ecology had more case studies. This sort of unspoken division of labour between journals means that the amount of theory varies greatly. And most ecologists recognize this - if I write a theory paper, it will be undoubtedly targeted to a journal that commonly publishes theory papers. So to more fully represent ecology, a wider variety of journals and more papers would be helpful. Still, Scheiner's counterargument would likely be that even non-theory papers (case studies, etc) should include more theory.

It may be that the proportion of papers that include theory is not a good measure of theory’s importance or inclusion in ecology in general. For example, Scheiner states, “All observations presuppose a theoretical context...the simple act of counting individuals and assessing species diversity relies on the concepts of ‘individual’ and ‘species,’ both of which are complex ideas”. While absolutely true, does this suggest that any paper with a survey of species’ distributions needs to reference theory related to species’ concepts? What is the difference between acknowledging theory used via a citation and more involved discussion of theory? In neither of these cases is the paper “bereft” of theory, but it is not clear from the methods how this difference was dealt with. As well, I think that ecological literature contains far more papers about applied topics, natural history, and system-specific reports than evolutionary biology. Applied literature is an important output of ecology, and as Scheiner states, builds undeniably on years of theoretical background. But on the other hand, a paper about the efficacy of existing reserves in protecting diversity using gap analysis is both important and may not have a clear role for a theoretical section (but will no doubt cite some theoretical and methodological studies). Does this make it somehow of less value to ecology than a paper explicitly testing theory? In addition, case reports and data *are* a necessary part of the theoretical process, since they provide the raw observations on which to build or refine theory. In many ways, Scheiner's editorial is a continuation of the ongoing tension between theory and empiricism that ecology has always faced.

The point I did agree strongly with is that ecology is prone to missing the step between theory development and data collection, i.e. theory testing. Far too few papers test existing theories before the theoreticians have moved on to some new theory. The balance between data collection, theory development, and theory testing is probably more important than the absolute number of papers devoted to one or the other.

Scheiner’s conclusion, though, is eloquent and easy to support, no matter how you feel about his other conclusions: “My challenge to you is to examine the ecological literature with a critical eye towards theory engagement, especially if you are a grant or manuscript reviewer. Be sure to be explicit about the theoretical underpinnings of your study in your next paper…Strengthening the ecological literature by engaging with theory depends on you.”


Marcel Holyoak said...

Nice critique Caroline. I did notice the overlapping confidence intervals but my main reason for publishing this was to get ecologists to think about when theory is relevant. I see a lot of manuscripts that would be strengthened by using an underlying theory to structure the author's (or authors') ideas and where some vague statements would be deleted if the authors were to think more about the underlying theory.

Marcel Holyoak

Caroline Tucker said...

Hi Marcel - I think I mostly disagreed with how he was supporting/framing his argument. But as a theoretical ecologist, I can't disagree with his motivations :) Will Ecology Letters encourage more tests of theory somehow?

Marcel Holyoak said...

Both tests and development of theory that is testable are always welcome! Especially needed are theories outside of the bread-and-butter subjects that use them all the time (population ecology, behavioral ecology, etc.). So I'm especially keen to see theories and tests of them that go beyond the safe topics that have long traditions of model development and testing.The challenge is doing such things in ways that are both accessible and of broad interest, but I suspect ecologists are up to it!

Krzysztof Sakrejda said...

"If we are being truly scientific, this might be a good time to point out that the 95% confidence intervals for the percentage of ecology papers with theory overlap with the confidence intervals for the percentage of evolutionary papers with theory, suggesting the difference that is the crux of the paper is not significant"

Looking at the actual mean/CI numbers, I'd bet money that this comes out as a significant difference at alpha of 0.05 regardless of how you test it.

Caroline Tucker said...

Krysztof - you are correct, that mistake is mine (a lapse of judgement in assuming that since a lack of overlap of CIs is significant at 5%, that the reverse should be true). Confidence intervals can overlap and still be significantly different. I quickly ran a test of proportions on the binned data, and it is less than 0.05 (code below). I'll update the text to reflect this.


#2-sample test for equality of proportions with continuity correction
data: theory out of totals
X-squared = 4.2822, df = 1, p-value = 0.03851

Krzysztof Sakrejda said...

Would you be willing to comment on what you think unsafe topics are?

Krzysztof Sakrejda said...

Hi Caroline, I would've let it go except for the "truly scientific" comment! Even if we do as well as the evolutionary biologists in general I really enjoy papers which do a thorough and honest job of putting their results in the context of theory so I like seeing Scheiner's challenge in print.

Caroline Tucker said...

Hi Krzysztof - You're right, that was a bit over the top. Thanks for pointing out I was wrong :)
I think, for the record, I need to point out that I'm also a big advocate of theory as the foundation for ecological work of all sorts (and I hope many of my blog posts and publications make this clear). And people have been making the point that ecology absolutely needs to build or maintain a strong connection between theoretical and empirical work for years. What I am mostly disagreeing with is the idea that this point relates to the percentage of papers that explicitly mention theory, and that the comparison to evolutionary biology is necessarily apt. A more thoughtful approach would be to start by defining what is meant by theory, to recognize when theory is explicitly developed/tested versus when it is the implicit (cited) backbone of a paper, but not explicitly part of the text. And to acknowledge that ecology still needs very empirical work (systematics and taxonomy, distributional data, etc etc) upon which good theory can be developed.
Obviously Scheiner's editorial was meant to be provocative, and clearly it was a success in creating conversation. But I just think a more realistic demand for ecology could have been made.

Krzysztof Sakrejda said...

Thanks for taking the time to respond. Your publications definitely make it clear that you take theory seriously, it's really the best kind of internal advocacy.

Scheiner's measure is pretty crude, but is it really a bad comparison? Evolution has as much of a need for all the empirical work and yet they (seem!) to do better at making the connections explicit. My guess is that the challenge is greater in ecology because the theory is less unified. For example it's easy to think of population dynamics and behavioral ecology as disconnected, but they intersect when it comes to understanding density dependence. It's not that we need a grand unified theory of ecology, but there are some obvious connections which need more work (empirical and theoretical). I agree that a more thoughtful analysis of the different uses of theory would be important to moving this discussion beyond the provocative stage. I'm going to stick my neck out and say that when theory is implicit it is less likely to be thought through prior to the application and therefore less likely to be correctly applied/tested, and we are even less likely to notice and correct mistakes in post-publication review.

...I don't mean to be thick, but from your post I only saw that you're critical of how Scheiner made his point, not the actual suggestions/demands in his editorial. I'm genuinely curious to find out where you think he goes overboard?

ucfagls said...

If anyone wants to understand or know more about the coverage properties of confidence intervals etc, then Nature Methods had a nice piece on this recently:

stevencarlislewalker said...

Hey Caroline. Thanks for promoting this great paper more. However I worry about one very specific aspect of it. Scheiner says:

"My challenge to you is to examine the ecological literature with a critical eye towards theory engagement, especially if you are a grant or manuscript reviewer."

I'm afraid that many grant/manuscript reviewers won't have the same sophisticated understanding of the concept of theory that Scheiner has. I know from Scheiner's other work that he does not assume that all aspects of a theory can/should be tested. We need to have axioms, and different people have different axioms (e.g. E.O. Wilson versus James Watson -- both great biologists but from very different perspectives). So I'm afraid that some reviewers might read this and think...well this doesn't look like the kind of theoretical framework that I work in, so it must be bad because its not relevant to theory. Right? Its the 'all fords are cars but not all cars are fords' problem that I'm worried people will overlook. Scheiner is very generous about what theory is, so I hope others will be as well.

I really really really think that someone needs to formalize a theory of natural history, as distinct from a theory of ecology, using Scheiner's much too ignored and fantastic approach to understanding theories in the biological sciences ( Until this happens, I sense that people will always see natural history as inferior, even if it doesn't deserve that status.

stevencarlislewalker said...

PS -- I was very happy to see you standing up for natural history and observational stuff!
PS -- By 'promoting this great paper' I mean 'giving it more press, good or bad', which I think it deserves.

Caroline Tucker said...

Great comment Steve - I also think that Scheiner's definition of theory is probably different than the average ecologist's definition.

"PS -- I was very happy to see you standing up for natural history and observational stuff!"
Thanks?! I'm getting soft in my old age :) (Or else just more aware of the need for balance between different approaches to science).