My recent thinking has been shaped by my peripheral involvement in discussions between colleagues at the University of Ottawa. What I will discuss now, is the confluence of ideas expressed by several people, and I say this because these have been stimulating discussions, and I don’t want to appear as taking full credit for these ideas by virtue of this solo author blog post.
———————————–
All other things being equal, mechanistic models are more powerful since they tell you about the underlying processes driving patterns. They are more likely to work correctly when extrapolating beyond the observed conditions.
-Bolker (2008) Ecological models and Data in R, p7.
My question for today is:
Given a mechanistic model and a phenomenological (or statistical) model, if we are trying to determine which model is best, shouldn’t the mechanistic model score some ‘points’ by virtue of it being mechanistic?
Assume a data set that both models are intended to describe. Define mechanistic and phenomenological as follows,
Mechanistic model: a hypothesized relationship between the variables in the data set where the nature of the relationship is specified in terms of the biological processes that are thought to have given rise to the data. The parameters in the mechanistic model all have biological definitions and so they can be measured independently of the data set referenced above.
Phenomenological/Statistical model: a hypothesized relationship between the variables in the data set, where the relationship seeks only to best describe the data.
These definitions are taken from the Ecological Detective by Ray Hilborn and Marc Mangel. Here are some additional comments from Hilborn and Mangel:
A statistical model foregoes any attempt to explain why the variables interact the way they do, and simply attempts to describe the relationship, with the assumption that the relationship extends past the measured values. Regression models are the standard form of such descriptions, and Peters (1991) argued that the only predictive models in ecology should be statistical ones; we consider this an overly narrow viewpoint.
Having defined mechanistic and phenomenological, now the final piece of the puzzle is to define ‘best’. Conventional wisdom is that mechanistic models facilitate a biological understanding, however, I think that’s only one step removed from prediction – you want to take your new found understanding and do something with it, specifically make a prediction and test it. Therefore, the goal of both mechanistic and phenomenological models are to predict and the performance of the models in this respect is referred to as model validation.
But, validation data is not always available. One reason is that if the models predict into the future, we will have to wait until the validation data appears. The other reason is that if we don’t have to wait, it’s a bit tempting to take a sneak peek at the validation data. For both model types, you want to present a model that is good given all the information available – it’s tough to write a paper where your conclusion is that your model is poor when the apparent poorness of the model can be ‘fixed’ by using the validation data to calibrate/parameterize the model (which then leaves no data to validate the model, something that, if anything, is a relief because your previous try at model validation didn’t go so well).
In absence of any validation data, one way to select the best model is using Akaike Information Criterion (AIC) (or a similar test). AIC will choose a model that fits the data well without involving too many parameters, but does AIC tell me which model is best, given my above definition of best, when comparing a mechanistic and a statistical model? Earlier this week, I said that if we wanted to settle this – which is better mechanistic or phenomenological – then we could settle it in the ring with an AIC battle-to-the-death.
As the one who was championing the mechanistic approach, I now feel like I didn’t quite think that one through. Of the set of all models that are phenomenological versus the set of all models that are mechanistic (with respect to a particular data set), it’s not rocket science to figure out which set is a subset of the other one. If one model is a relationship that comes with a biological explanation too, then you’re getting something extra than the model that just describes a relationship. Shouldn’t I get some points for that? Didn’t I earn that when I took the mechanistic approach to modelling because my options for candidate models is much more limited?
There is one way that mechanistic models are already getting points from AIC. If I did a good job of parameterizing my mechanistic model there should be few fitted parameters – hopefully even none. But is that enough of an advantage? Exactly what advantage do I want? I think what I am hoping for is related to the span of data sets that the model could then be applied to for prediction or validation. I feel pretty confident taking my mechanistic model off to another setting and testing it out, but if my model was purely statistical I might be less confident in doing so. Possibly because if my mechanistic model failed in the new setting I could say ‘what went wrong?’ (in terms of my process-based assumptions) and I’d have a starting point for revising my model. If my statistical model didn’t do so well in the new setting, I might not have much to go on if I wanted to try and figure out why.
But, if the objective is only to predict then you don’t need to know about mechanisms and so the phenomenological/statistical approach is the most direct and arguably best way of generating a good predictive model. Perhaps, what this issue revolves around is that mechanistic models make general and inaccurate predictions (i.e., the predictions might apply to a number of different settings) and that phenomenological models make accurate, narrow predictions.
Truth be known, this issue is tugging at my faith (mechanistic models), and I’m not really happy with my answers to some of the fundamental questions about why I favour the mechanistic approach, as I do. And let me say, too, that I definitely don’t think that mechanistic models are better than phenomenological models; I think that each have their place and I’m just wondering about which places those are.
Just Simple Enough: The Art of Mathematical Modelling 
What if you can *only* do a reasonable job of fitting the data, or predicting new data, if you use a mechanistic model?
http://andrewgelman.com/2012/01/the-last-word-on-the-canadian-lynx-series/
Thanks for pointing out the Lynx data. That’s an interesting conclusion that I wasn’t aware of. I think using statistical and phenomenological synonymously has made my discussion above a bit confusing. The Lynx result seems to be that statistics hasn’t done as well as the mechanistic model, but I’m wondering if the full scope of all phenomenological models has been considered. For example, if I was fitting a phenomenological model to that data, I would try to fit an ellipse to ‘lynx’ versus ‘hare’ (i.e., like a phase plane diagram). I also don’t think that mechanistic models have a monopoly on ODEs. Arguably, if the model is a system of ODEs that don’t fit with the biological processes that are known to occur, then aren’t we sticking with those ODEs solely because of their shape – and not because they contribute anything to understanding – and so wouldn’t that be a phenomenological ODE model?
Hmm…yes, fitting an ellipse to the dynamics would be phenomenological, and might work better than an ARIMA model. But it would be an odd sort of phenomenological model to my eye–you’re fitting a model that can *only* fit cyclic dynamics, because it assumes cyclic dynamics. What do you learn or gain by doing this? After all, we already know the lynx-hare series is cyclic, and we can estimate the period and amplitude quite well using standard statistical techniques. What does fitting a phenomenological cyclic function to those data teach us that we didn’t already know?
I do entirely agree that if you have a mechanistic ODE that doesn’t capture the important mechanisms, it’s probably not going to fit well (at least not with realistic parameter values). But in that case, I’d go for a “semi-mechanistic” model (aka “partially specified” or semi-parametric” model). That is, your model specifies those bits of the biology that are well-known–say, a type II functional response for lynx–and describes unknown bits of biology–say, hare fecundity as a function of hare density–with flexible nonparametric functions like splines, the forms of which are estimated from the data. In the comments on Andrew Gelman’s post I linked to the key papers on this approach, by Simon Wood, and your former Queen’s colleague Bill Nelson. The nice thing about this is that you’re not making unwarranted assumptions about the underlying biological mechanisms that generated the time series, but nor are you assuming that the time series itself has a particular form. The challenge is actually doing the fit, because it’s often the case that different combinations of functional forms will fit the data about equally well. Bill’s done work on how to deal with this.
This is a really interesting post (and I’m loving the blog! It’s great to see the number of ecologists who are getting into blogging in a major way.)
I’m still not sure I’m convinced that the mechanistic/phenomenological model distinction is really a fundamental one. For instance, let’s say I’m trying to measure the predation rate of a predator on prey; I decide to fit both a Holling Type II function using non-linear least squares, and a mono-tonic spline. Is the second one phenomenological? Would your answer change if I decided on the monotonic convex spline because the natural history of the species suggested a range in the variation of the attack rates and handling times for that predator,and the summed predation function wouldn’t actually add up to a Holling, but would still be monotonic and convex? To me, the second case is also mechanistic, it just relies on looking for an aggregate (qualitative?) pattern from several related mechanisms.
At this point, I think I’m generally convinced that the best model is a statistically sufficient one… a model that’s able to explain all the deterministic and stochastic patterns in a set of data (ie: the observed data should look like a typical realization of the fitted model). If a mechanistic model (however you define it) fits that best, I’m for it. If it doesn’t, then that should at least put up warning flags that while the model might get high R^2 for the current data set (or good AIC, or etc. etc.), but trying to extend its predictions beyond the data may well be as fraught with errors as using a phenomenological one. That is, a mechanistic model that doesn’t do a good job of describing the data-generating process is in fact just a phenomenological regression with an odd set of parameters.
Thanks Eric: nice to have you following along! re: least squares + Holling type II, my answer is ‘yes, mechanistic’. Regarding the spline part, I’m not totally sure that I follow. You want to estimate a range because there is variation in the parameters? The whole spline thing sounds phenomenological to me, but if you think this approach is going to give you back parameters with units and biological interpretations (or rather a range for the parameters) then I’d say it’s mechanistic. (My apologies, I don’t quite understand this.)
The second paragraph of your comment seems to really focus on fitting the data and not valuing understanding the mechanism. In the lynx-hare example, if a type I functional response fits well with the data (better than a type II) – don’t you think that’s useful information? versus my ellipse fitting approach (see my comment to JF) which would do a nice job of estimating the parameters for my ellipse, but these parameters would not hold any additional information. Regarding the last part, yes, I think that if you come up with an ODE model with good fit and poor representation of the biological processes, I’d call that phenomenological.
Sorry, I don’t think I explained my thoughts too well. I’ll chalk it up to lack of sleep. 🙂 My post had been meant as a response to “shouldn’t the mechanistic model score some ‘points’ by virtue of it being mechanistic?” In brief (as the rest of the comment is probably overly long): no, because a ‘mechanistic’ model that does a poor job as a generative model for the data is likely not actually ‘mechanistic’, in that parameter estimates from it can easily not correspond to any actual trait for a species (foodweb, ecosystem, etc).
The idea I was trying to convey was: whether I’d consider a model “mechanistic” or “phenomenological” doesn’t just depend on the mathematical representation ,or even what we think is going on in the system; it depends on the true causal processes underlying process, which means we can’t give ‘points’ to a mechanistic model unless we know it’s a fair representation of a mechanism. For example: say (going back to the functional response example again) a predator truly has a type II functional response, but there are many mechanisms all acting to create that, I would say that fitting a non-parametric curve (the spline), constrained to follow a type-II style curve (monotonically increasing, convex) is more “mechanistic” than fitting a Holling disc equation model, as estimates of coefficients from the disc equation could easily have no real relationship to the actual attack rates and handling times of a species, and variations in those parameters may not actually change the attack rates they way you’d predict. While the non-parametric model tells us little about the underlying biology, it’s being ‘honest’ in what it tells us: all we know is that predation rates increase and are convex. If we have the data, we can then go back and try and decompose the non-parametric curve into contributions from different mechanisms we think are operating.
In response to your second comment: I do value understanding the mechanism; I just consider having a model that can accurately generate your data is a necessary condition before saying you have a good model of a mechanism. As you’ve mentioned, the power of a mechanistic approach comes from being able to predict how a system will shift when underlying processes change; however, that assumes you have accurate knowledge of the mechanisms generating a process. If your chosen model doesn’t even fit the data well, why trust what its parameter estimates are telling us about how a system will change?
Thanks Jeremy. So what you describe in your post, I would call that the ‘sensible approach’. And my ellipse idea, we can call that the ‘odd approach’. I’m worried that AIC would conclude that both approaches are equally good.
Nobody said AIC is the be-all and end-all! Models can have lots of different virtues besides the one AIC captures (having a relatively small Kullback-Leibler information distance from the “true” model).
Thanks Eric! I agree with much of what you said. It’s true that it’s going to be tough to decide what is really mechanistic because then you’d need to know the ‘true’ processes.
Hi Amy,
This is a great post. Sorry for joining this discussion so late, but I figured this is a timeless topic.
Firstly, I wanted to mention a recent synthesis paper on differences and common ground between process-based and correlative SDMs that I coauthored, it picks up on a lot of the topics in this post, see also a related blog post .
Apart from that, I agree that the definition of “mechanistic” vs. “phenomenological” is blurry – we argue in the paper to view this as a continuum rather than a strict divide, but it remains dubious. The problem is that there are a number of things that people associate with the word “mechanistic”. To name only some of those, I’d mention causality, (weak) emergence, process (dynamic) description and the forward modeling (parameters with “meaning”) paradigm.
Many models fulfill only a subset of these criteria. Take, for example, Hubbell’s neutral theory, which is commonly fit to data – most would agree it’s a mechanistic model, but on such an aggregated level that it’s debatable to which extent the parameters that result from the fit have “biological definition” – hard to measure them independently at any rate. The underlying problem is that different “mechanistic” models in ecology make different simplifications, which makes parameters not really comparable because they do not describe the same process in the same way. It also becomes a question of the scale at which you want to be mechanistic or structurally realistic, and at which scale you accept a phenomenological description. (Btw, I would not equate “phenomenological” with “statistical”, mechanistic models are increasingly fit exactly in the same way as “phenomenological” models; we have tried a working definition of statistical models here, but “statistical model” remains another blurry term as well).
So, where does that leave us regarding the question of whether mechanistic models are “better/more predictive”? My point is that the word “mechanistic” is a Chimera, under which we subsume a number of things of differing importance for the question of model selection and predictive accuracy. I find causality to be the most important: in a correlative model, you can pick up spurious correlations, but the same can happen to you in a process-based model – you can have the wrong process, and you get the right result for the wrong reason. So, process-based models are not inherently more causal, but I think that we tend to use them only when we are already more or less sure about the processes (people usually don’t fit all possible dynamic processes to data, they use these models when they know the processes already). More generally, I feel that people tend to use the word “mechanistic” when they are relatively sure about the causality in a system, while they use the word “phenomenological” when they are certain to a lesser degree – hence, I would subscribe to the notion that mechanistic models are more powerful “all other things equal”, noting, however, that this is in some way tautological because the notion of a mechanistic model is often not so much tied to the model structure as such, but simply reflects the fact that we have additional structural/causal information; as mentioned before by Eric and by you, the same mathematical structure could be viewed as mechanistic or as phenomenological (although there are of course some model structures that are typical examples of either type).
Hi Florian,
Wow, this is such a great comment. Thanks so much. If it were okay with you, I would like to copy & paste this as a post: it’s just a shame that all this good stuff is buried down here in the comments. Would you be interested in me inviting you as a contributor? Alternatively, just let me know and I can link posts from your blog that are comments on my posts.
I agree with most of what you say, I’ll probably pick it up this issue again later. I do agree that with a ‘process-based model – you can have the wrong process, and you get the right result for the wrong reason’, but chance of that happening can be very likely or very unlikely depending on the model in question and the performance relative to competing process-based models.
Good stuff. I’m going to put it on my to-do list to dig more into this later.
Thanks again.
Amy
Hi Amy,
feel free to quote me as you like. I’d be tempted to write a post on that myself, but I’m not sure when I’ll find the time, so please go ahead as you wish, I might reply either by comments or in a post!
About the likelihood of getting the structure wrong: yes, the chance of getting the structure wrong is probably smaller in process-based models than in phenomenological models, because higher structural realism provides more opportunities to match the model structure (and parameters to some extend, see my previous comment) with things we already know. And ultimately, we will of course always strive to provide a mechanistic explanations for natural phenomena, after all, reductionism is one of the cornerstones of (current) science.
Yet, comparing the virtue of both approaches, we should be careful not to compare apples and oranges: for a ceteris paribus comparison of mechanistic and phenomenological models, the same amount of information should go into the modeling process. In practice, however, I would say that mechanistic models typically entail a much larger amount of information because we use them at a later stage of the scientific process, when the main factors that drive certain patterns are already more or less understood (or at least reduced to a few options). So, there’s a lot of implicit data hidden in the model structure. If you would like to formalize that in a decision theoretical framework (although that seems a bit pointless to me), you could probably encode this additional information as priors on the model structure in a Bayesian analysis, that would give you the formal “excuse” to prefer the mechanistic explanation.
Yes, a formal decision theoretic structure is what I was thinking. Even wrt to your ‘apples to oranges’ point: different parameters get penalized equally under AIC even if some of the parameter do virtually nothing to improve fit (i.e., the so-called pretending variables). Thanks again for your comments. If you write something more on this topic let me know, I’d like to read it!
I’ll do that. So far, I have highlighted your post here
About the AIC model selection: there are alternative Bayesian methods that also guard against overfitting such as Bayes factors, DIC and others. They all have their specific problems, but my opinion is that, if you really want to do a formal model selection of models that are mechanistic to a different degree, those are preferable because the Bayesian approach allows you to include your (prior) information on the mechanisms in a systematic way.
In the end, however, the question remains what’s your goal of modelling, i.e. do you require your model to be structurally correct (or more formally, do you compare the model substructure to data/prior knowledge), or do you only care about prediction for a specific dataset.
Pingback: Mechanistic vs. phenomenological descriptions of nature « theoretical ecology
Pingback: Process, correlation and parameter fitting in species distribution models: a response to Kriticos et al. « theoretical ecology
Pingback: True models, predicitve models, and consistent Bayesian state-space estimators for chaotic dynamics | theoretical ecology
Pingback: Author post: Genome scans for detecting footprints of local adaptation using a Bayesian factor model | Haldane's Sieve
I think it boils down to economics and cost-benefit-analysis. Which ever model has the highest benefit-to-cost ratio is the best one to use. There is only value in understanding if understating implies a higher benefit-to-cost ratio. Nothing has intrinsic value. (here, cost is not monetary but refers to time and effort).
I misspelled ‘understanding’ 😉
“which model is best?” I don’t get why people still ask this question. Like almost everything in life “IT DEPENDS”. What question/problem are you trying to answer/solve? They each have their own strengths and weaknesses. Besides, very few models are purely mechanistic in nature. Most have at least some empirical assumptions/underpinnings.
Pingback: How to find a moose: intentions in phenomenological and mechanistic models by Adam Clark | PLOS Blogs Network