A PPLES
APPLES AND ORANGES?
Vox now offers us a variation of his favorite line of reasoning, which is to compare the predictive performance of economic models and evolutionary theory. He writes:
Now, I admit that it is theoretically possible that the tremendous problems faced by macroeconomists and financial analysts, whose models are very, very different, will not be experienced by scientists basing their predictive models on TENS. However, given the superior precision of the backtesting of the economic and financial models and the unsuccessful nature of so many TENS backdated "predictions", the logical conclusion based on the current data is that the TENS model is far less likely to prove functional when tested for its future performance in a scientific manner than the inadequate models from other scientific disciplines.
I have to say I’m unimpressed: Do economic models, when backtested, show superior precision to those models based upon TENS? You can only answer ‘yes’ if you ignore questions of scale. Economic models typically model decades, or at best centuries of data: those of us who are used to thinking in terms of ‘deep time’ (millions of years, or more) are likely to be puzzled that any one would think this comparison is valid. Of course models based exclusively on the fossil record are less precise, Vox: they operate on a time scale that is several orders of magnitude greater than any yet attempted in the field of economics. Talk about apples and oranges!
Now, I admit that it is theoretically possible that the tremendous problems faced by macroeconomists and financial analysts, whose models are very, very different, will not be experienced by scientists basing their predictive models on TENS. However, given the superior precision of the backtesting of the economic and financial models and the unsuccessful nature of so many TENS backdated "predictions", the logical conclusion based on the current data is that the TENS model is far less likely to prove functional when tested for its future performance in a scientific manner than the inadequate models from other scientific disciplines.
I have to say I’m unimpressed: Do economic models, when backtested, show superior precision to those models based upon TENS? You can only answer ‘yes’ if you ignore questions of scale. Economic models typically model decades, or at best centuries of data: those of us who are used to thinking in terms of ‘deep time’ (millions of years, or more) are likely to be puzzled that any one would think this comparison is valid. Of course models based exclusively on the fossil record are less precise, Vox: they operate on a time scale that is several orders of magnitude greater than any yet attempted in the field of economics. Talk about apples and oranges!
PARITY AT ALL COSTS
So, let’s try to level the playing field, shall we? As I see it, we have two ways we can proceed:
#1) We can compare the predictions of evolutionary theory with that of economists over the same recent scale of time, say the last 30 years or so. For example, given known parameters of a population, we could predict the likely frequency of a gene under selection in that population over that period of time. We could then compare it with the predictions of economic theory for some particular item, such as changes in the value of a given commodity over the same period of time. I would predict that the former would approach the latter in precision, and perhaps even exceed it due to some of the difficulties that Vox alluded to above.
#2) Conversely, let’s see the economists make predictions over the sort of time scale we Darwinians take for granted. How, for example, would an economist go about predicting the value of a commodity 1,000 years from now? Hmmm....let’s see, in order to do that, they’d have to know supply and demand, I suppose....but wait, demand is a function of population size...aw, gee, populations are biological objects....so...in order to model the value, we’ll need to be able to model any....changes in the population’s demographic. Uhhm...hem....haw.....
You see the problem? Any attempt at building an economic model on even a fraction* of the ‘deep time’ scale used by paleontologists involves modeling biological processes, which in the cases of changes in that population’s demographic involves its evolution! So, Vox, I have to ask: why would a smart feller like you have greater confidence in such an economic model than upon the biology on which it would it would have to be constructed? Human populations, other living things and the physical environment they share (and shape) are all evolving still. Let me suggest that economics, like medicine and agriculture before it, will in the future increasingly rely upon scientific research into how and when populations undergo genetic change—or, as some of like to say, evolve.
* (1,000 years is almost nothing to a paleontologist)
So, let’s try to level the playing field, shall we? As I see it, we have two ways we can proceed:
#1) We can compare the predictions of evolutionary theory with that of economists over the same recent scale of time, say the last 30 years or so. For example, given known parameters of a population, we could predict the likely frequency of a gene under selection in that population over that period of time. We could then compare it with the predictions of economic theory for some particular item, such as changes in the value of a given commodity over the same period of time. I would predict that the former would approach the latter in precision, and perhaps even exceed it due to some of the difficulties that Vox alluded to above.
#2) Conversely, let’s see the economists make predictions over the sort of time scale we Darwinians take for granted. How, for example, would an economist go about predicting the value of a commodity 1,000 years from now? Hmmm....let’s see, in order to do that, they’d have to know supply and demand, I suppose....but wait, demand is a function of population size...aw, gee, populations are biological objects....so...in order to model the value, we’ll need to be able to model any....changes in the population’s demographic. Uhhm...hem....haw.....
You see the problem? Any attempt at building an economic model on even a fraction* of the ‘deep time’ scale used by paleontologists involves modeling biological processes, which in the cases of changes in that population’s demographic involves its evolution! So, Vox, I have to ask: why would a smart feller like you have greater confidence in such an economic model than upon the biology on which it would it would have to be constructed? Human populations, other living things and the physical environment they share (and shape) are all evolving still. Let me suggest that economics, like medicine and agriculture before it, will in the future increasingly rely upon scientific research into how and when populations undergo genetic change—or, as some of like to say, evolve.
* (1,000 years is almost nothing to a paleontologist)
12 comments:
I have to say I’m unimpressed: Do economic models, when backtested, show superior precision to those models based upon TENS?
As Econ models, having far greater data to use, are -
It has begun IV -
Goldman Sachs right now is having far more serious problems with their presumably more advanced models than I ever did. Although the exact details are unclear, shares began to move in ways that were precisely the opposite of those predicted by computer models. These moves triggered selling by funds as they attempted to cover their losses and meet margin calls from banks. This, in turn, made share price movements even worse. As Goldman's CFO noted in a rare conference call: "We are seeing things that were 25-standard deviation events, several days in a row." That's something that only happens once every 100,000 years.
http://voxday.blogspot.com/2007/08/it-has-begun-iv.html
... the logical conclusion based on the current data is that the TENS model is far less likely to prove functional when tested for its future performance in a scientific manner than the inadequate models from other scientific disciplines.
Salt, the reason why economists have greater data relative to evolution is because their discipline is defined by that fraction of human history for which detailed data exists. Evolutionary biologists don't have that luxury, because evolution is not confined to human history, but concerns the history of all life.
How can ignoring this striking difference in terms of scale be justified? I say, place the two disciplines on an equal footing with respect to the time scale and scope of phenomena examined if you want the comparison to be meaningful---otherwise, it's just wishful thinking.
Speaking of which, that applies with equal force to the final paragraph of Vox's quoted. Unless another model of comparable size and scope can be shown to offer superior precision in terms of the predictions it makes, ya got nothin'.
How can ignoring this striking difference in terms of scale be justified?
Scale is not data. That massive amounts of data can wrongfully predict, how can TENS compete better?
I say, place the two disciplines on an equal footing with respect to the time scale and scope of phenomena examined if you want the comparison to be meaningful
That doesn't wash. If you had the same amount of data relevant to your scope of phenomena that Econ has to its, are you saying your predictive ability would be better?
Is TENS capable of accurately including CHAOS within its discipline?
Scale is not data...
It is if it is used to define the terms of investigation, and the data in question is not scale-invariant. If the earth were not reliably known to be many orders of magnitude older than the time spans studied by economists, the fossil record would not be so troublesome.
Is TENS capable of accurately including CHAOS within its discipline?
Your meaning is uncertain here. Are you asking if chaos theory is part of biology, or if biology is consilient with chaos theory? Hopefully not the first, since (properly speaking) chaos is a mathematical concept, and is best considered a problem in mathematical physics, rather than biology.
Now, the question on whether chaos theory is compatible with TENS is a different question. In brief, linear functions are easier to model than non-linear ones. If you have enough non-linear processes interacting in a system, chaotic behavior will follow. In fluids, sufficiently complex flows become chaotic: that's the origin of turbulence. Observing that turbulence occurs does not violate any known law of physics, but you can't use the known laws to predict the fluid's behavior at any given time.
In the same way, the tangled 'strange loops' of causation that occur in evolving systems of populations are consistent with the TENS model, but you can't explain all of the system's behavior strictly in terms of TENS at any given individual moment.
Hope that helps. It would help me if you would take the time to provide more context in framing the question..!
Scott, you say: the reason why economists have greater data relative to evolution is because their discipline is defined by that fraction of human history for which detailed data exists. Evolutionary biologists don't have that luxury, because evolution is not confined to human history, but concerns the history of all life.
Aren't you making Vox's point stronger? If economic models, which are limited in scope, can be unreliable, couldn't evolutionary theory, which is much larger in scope and scale, be that much more unreliable?
Aren't you making Vox's point stronger? If economic models, which are limited in scope, can be unreliable, couldn't evolutionary theory, which is much larger in scope and scale, be that much more unreliable?
First of all, neither Vox and I are necessarily claiming that either economic or evolutionary models are 'reliable' or 'unreliable', as if this was a zero-sum game. The question is the relative amount of confidence we can assign to a given model based on its known 'margin of error.' You probably understood that, but I just want to be fair to Vox and point out we didn't use your terms.
Secondly, all you're doing is restating Vox's argument while ignoring the effect that differences in scale have on the margin of error in predictions derived from the different models. I'm not questioning that economic models have lower margins of error than the evolutionary models he's comparing it with. I'm questioning the relevance of that observation, since if we confine ourselves to models operating over the same scale, the dramatic difference in error margins vanishes.
I'm questioning the relevance of that observation, since if we confine ourselves to models operating over the same scale, the dramatic difference in error margins vanishes.
So, what models shall we compare to TENS?
--emerod
If we mean comparing TENS purely on the question of confidence in the predictions it makes, versus economic theory, we can only make valid comparisons within comparable time scales. I gave an example of such a possibility in the post.
Strictly speaking, TENS is a theory about the dominant role of natural selection in driving evolution. There are factors other than natural selection known to lead to evolution (such as genetic drift), so TENS is not held dogmatically in individual cases in the absence of data; it's simply the hypothesis of first resort...
More generally, if TENS is taking to be the explanatory framework of modern biology (which it is), then we could contrast its predictions with those of Lamarckism, orthogenesis, special creation etc. and see how it fares. Obviously, we biologists think TENS does quite a bit better than those model, if not as well as the facile (but misleading) comparison to the achievements of economists modeling recent events.
OK, now I better understand the basis for your faith in TENS. You seem to suggest that there are no successful models, in any area of thought, that function on the same scale with a comparable margin of error.
--emerod
Hate to be picky, but I don't have 'faith' in TENS; I am confident that for the domain it surveys it is the best overall model for explaining the most-number of phenomena---at present. Bottom line: Vox's argument was comparing confidence levels for theories across disciplines and scale, and using the favorable outcome for his field (economics) as a way of justifying his intuition that there is something wrong or missing from evolutionary theory. Vox's intuition may be correct: all I'm pointing out is that, by ignoring differences in scale, he's crafting a pretty meaningless argument.
As far as this goes:
....You seem to suggest that there are no successful models, in any area of thought, that function on the same scale with a comparable margin of error.
Never said that, never meant to imply that. I'd have to know a lot more about cosmology, astrophysics or geology to feel any confidence in either affirming or rejecting that sort of claim, I think. I'm inclined to be cautious, because you only need one counter-example to falsify it.
You might object that, if one theory is 'true', it should eventually lead to greater confirmation and a lower margin of error than competing theories. But all of the theories proposed to explain behavior at the same level of scale might be dead-wrong, but there's no guarantee that any of them would have the same margin of error.
Hate to be picky, but I don't have 'faith' in TENS; I am confident that for the domain it surveys it is the best overall model for explaining the most-number of phenomena---at present.
"Faith" has been politicized, of course, but the OED's first definition is still "Belief, trust, confidence."
When I asked to compare other models to TENS, I meant to modify Vox's argument in order to make it valid. You imply that only models from cosmology, astrophysics or geology could be compared to TENS.
I wasn't asking you to make an assessment of their relative trustworthiness, only to clarify what a valid "apples to apples" comparison would start with.
--emerod
You imply that only models from cosmology, astrophysics or geology could be compared to TENS.
Those were just the branches of science that work on comparable scale. And I don't really mean to imply that comparison is impossible with things that aren't, only that you have to have some way to address the scale problem.
Anyway, this is really a sidebar that's prompted by an argument that's rotten. Vox doesn't have any positive evidence for an alternate model, nor a negative argument based on data. He's got an intuition. That's not science.
Post a Comment