The question of how we can learn useful lessons from history, which would help us navigate the troubled waters ahead has been much on my mind. You can find some of my thoughts on this subject in my review of Walter Scheidel’s Escape from Rome. It’s also an important theme in the “Big Book” I am working on (there will be an update on it soon). And then yesterday I saw a Twitter thread by Balaji Srinivasan in which he asks the same question. Balaji mentions my work and that of Ray Dalio, but is somewhat critical of us. He says:
The main knock on Turchin & Dalio is that their quantification is imprecise. What’s the y-axis here? Seems like a weighted sum of many variables.
and gives an example from Dalio’s site. This critique may be correct when directed at Dalio (as I remember I similarly tried to get at his data sources some time ago, but couldn’t find it). But this charge is certainly not true when directed at my work. It is all explained in Ages of Discord, which is an academic book and has references for all data sources that I used in it (and that’s a lot). I also started a webpage on which all these data will eventually be posted, then had to set this project aside for a while. Now my research assistant is working on not only posting the data, but extending them to the present. Stay tuned.
To reiterate: all data on which my graphs are based are sourced in the academic pubs and the methods used for generating various curves from these data, such as the one below, are properly described and can be reproduced by anybody. When the webpage I mentioned is fully operational, this will be even easier.
As to the substance of Balaji’s proposal, I obviously agree with the basic idea that we need to construct and analyze historical databases, since that is what I’ve been doing since 2010 (see here the latest example). But there are good reasons to believe that the specific approach that he proposes for analyzing historical data will not work. What he describes is an atheoretical, entirely data-based analysis. Such a “brute-force” approach has the following problems:
1. There are thousands of variables that characterize societal dynamics. In the absence of theory, how do we choose the relevant ones? If we don’t select, the curse of dimensionality will defeat any attempt at trajectory-matching, which is essentially what he proposes.
2. Trajectories of specific societies are constantly perturbed by exogenous shocks (climate, external wars, epidemics…) and endogenous sources of variation (free-willed people behaving in unpredictable ways). Also, because social system are characterized by complex nonlinear feedbacks, they operate in a chaotic regime (in technical terms they are characterized by sensitive dependence on initial conditions). Thus, the trajectories of two identical dynamical systems starting from identical initial conditions will rapidly diverge (because sensitive dependence will amplify even small shocks within each system) and lose any resemblance of each other. This will defeat any attempt at trajectory matching.
So what can we do? First, stop treating societies as black boxes. We have pretty good ideas about the mechanisms that generate social dynamics. So we need to use explicit mechanism-based models. In fact, the problem is the opposite: there are too many ideas about what are the important mechanisms. Thus, the first use of a historical database is to empirically test different mechanistic theories against each other. Next, we can use the data to estimate such things as the signal/noise ratio and fine-tune parameter estimates (initial estimates should come from other sources than time-series as much as possible).
I can write more about this, but it’s too big a theme for a blog post, so I’ll stop here. As a final thought, the way I envision the eventual product, coming from this work, is a multi-path forecasting engine, the prototype of which is described in this publication.
I want to emphasize that these critical comments are “tactical” in nature. I completely agree with Balaji on the need to construct comprehensive historical databases that avoid the problems of cherry picking and the bed of Procrustes. In fact, our CrisisDB project is doing something very similar to what he wrote in the tweets, collecting time-series data on economic, political, social, and cultural variables of societies as they slide into a crisis, and then emerge from it. Perhaps there is a potential for collaboration here. In any case, the hard part is collecting the data, once there is a good database, different teams can (and will) use different analytical approaches to extracting insights from the data.
Great summary of Turchin’s projects, especially over the last decade. I have urged people to read his work on social media particularly, but with little evidence of success. My impression is that most people are not trained analytically and simply do not have the background to understand it. Perhaps someone better qualified than me could explain it to a wider audience, avoiding most of the math and statistics.could be a best seller.
So good just to hear from you, Peter, in these dark times. Got to stay hopeful.
Is it the best course to rely on Jared Diamond for the best presentation of the topography effect? It seemed to me the true novelty and deepest insight was the role of domesticable species available. And generally the closer Diamond got to the present, the less informative.
For instance, in the topography effect, the connecting effect of the Mediterranean is why the Mediterranean was not easily united with northern Europe. The difficulties of Holy Roman emperors in ruling both Italy and Germany seems to be more than the fortuitous outcome of strong and weak personalities.
Another reading of the topography effect is that sea travel instead emphasizes the connectivity of ports via the seas….but notes that the ports of China had much less to connect to. There was no equivalent of Egyptian grain providing resources to maintain a port polity, was there? Japan, Korea, the islands of the later Philippines and Indonesia. The Baltic was equally a connector, I suppose, but it seems to me that the slower development there than the Mediterranean was due to connecting smaller populations with lower agricultural produce.
For that matter the enclosed nature of the Mediterranean made it a place where rowing technology was practicable in a way that it never seemed to be for northern Europe and China as a whole. It’s the superior connectivity of the Mediterranean that led to the Roman/Byzantine/Arab/Turkish continuity…but that continuity turned out to be in the end a net drag on development. Empires put a premium on stability, which means.
Also, the flatness of the European plain notwithstanding, the impression given is that traveling through forests is easy! The steppe influence on Europe too was much moderated by the way it could only extend itself to Hungary.
So again, is Jared Diamond really the best formulation of the topography effect?
Here’s an unrelated observation on Diamond, which you may find interesting:
One of his most famous research queries was “What were the denizens of Easter Island thinking as they cut down the last tree?” Which perhaps gained him the lion’s share of his popularity, since we are pondering similar existential challenges. But I ran across a credible thesis recently that perhaps the cause of deforestation was rats who escaped from visiting ships, since apparently there was an occasional expedition that made contact with the island. The rats ate palm nuts, and reproduced quickly with no indigenous predators, so eventually the rats consumed nuts faster than the trees produced them, so new trees stopped growing.
That’s an interesting idea, although hard to prove at this point. But it’s a salient example of how we look at history’s mysteries through the lens of our contemporary experience and behavioral norms. But even modern behavior is poorly understood, as with environmental issues. Those who lament the comical disconnect between civilization’s short-term capitalist goals and humanity’s long-term sustainability goals misunderstand that such conflicts are inevitable, no matter how we try to avoid them. Even trying to come to consensus on how to manage them is impossibly contentious, due to the simple fact of policies which result in economic winners and losers for “the common good,” if there indeed is such an objective standard.
Similarly, this myopia exists with trying to understand how ancient people may have thought, most importantly how to interpret their cultural values from archaeological remnants. Even understanding relatively recent history is no easy task. There’s a maxim that history is written by the winners–of a war, or a trading arrangement, presumably. But in reality, every party to a conflict or competition has their own lopsided story which portrays them heroically, if possible, or as innocent victims. Thus it is helpful to consider multiple narratives, in the hope of deducing an intersectional set of common facts that we may construe as objective events.
And the difficulty in relating all this to modern times, as Peter Turchin admits, is no different from the deceptively arduous struggle to understand the causes of a particular recession or depression; few people realize each one is unique, in ways which are only obvious in distant hindsight. The problem of the past is insufficient information from unreliable sources. The challenge of the present, on the other hand, is overabundance, from which statisticians must try to decipher relevant metrics; and then quantify them by some arbitrary classification scheme which allows them to be meaningfully compared alongside dissimilar data, while also trying not to confuse a short piece of rope in tall grass with a venomous snake [that’s my own metaphor on the hazards of statistical misapplication].
So that’s my two cents…
What were they thinking as they cut down the last tree? Possibly that the planted trees were going to replace them, albeit years later…but some time after there was a mysterious blight they had never seen before, or maybe an extraordinary drought, or maybe a sudden catastrophic fire, or maybe as you suggest the rats ate all the nuts so new ones grew? Maybe they thought, since there hadn’t been any new trees growing for years, they might as well get some use from the tree, being as there weren’t going to be any more?
The thing about Diamond is that he is trying to interpret the past in terms of the present he sees. It seems to me in his version of the present, things happen because people are driven by psychological forces, which may or not be rational. In that same book, Collapse, Diamond contrasts forests in Haiti and the Dominican Republic, seeing Haiti as in many respects a collapsed ecology, heavily deforested. And he discusses the attitudes of some dictator as if it were the deciding factor. But Diamond omits the massive extortion of resources from Haiti by France in the guise of reparations! Looking for defects of human nature to “explain” outcomes is always a risky proposition. (One which Diamond’s better book avoids more successfully.)
If you don’t understand the present, you can’t understand the past. Indeed, if you don’t understand your own society, you can’t understand foreign societies. As useful as statistics can be, statistics and mathematical models premised, in however carefully disguised form, on the notion that ultimately it’s the mystery of human nature, man, will fail. And the the math will just be sciency, not science.
And that’s my two cents in the pot. Are we betting on the same thing?
Excellent counterpoint. I was unaware of France’s history in Haiti; now I gotta read up on that. Another illustration of Truman’s quote, “The only thing new in the world is the history you don’t know.” Case in point: I learned from a book I’m currently reading that an inciting incident for the Great Depression may have been a Bernie Madoff character in England who got caught swindling investment banks with duplicate stock certificates. Because it took a while to figure out the scale of his scheme, there was a panic in the City of London which spread across the Atlantic to infect an already unstable Wall Street. I have no idea if this narrative is correct, but it’s a good example of how history is never quite settled; there are always potential surprises lurking in a library somewhere.
I also enjoy anecdotes about how science is never quite settled, such as the frustratingly meandering journeys to eradicate scurvy and smallpox. I really wish that, rather than the lowbrow debates we generally hear in public forums about trusting science, which take on a religious fervor at times, that there was more widespread recognition of the annoying reality of how science evolves–in fits and starts, and with political thumbs on the scales. Not that knowing this would allow complex challenges to be cured with simplistic solutions. But just because we are often stumbling around like a drunk looking for his keys under a streetlight, because that’s where the light is.
I’m hoping you can recommend some light reading, since we both ended up at this blog for a reason, presumably. There are a thousand doomsayer clones on Medium posting about the collapse of civilization. OK, but after that, what comes next? Or are we all going to run around like chimps with our hair on fire, until nuclear annihilation? Maybe, but it’s also quite plausible that the Yellowstone supervolcano or an inbound asteroid or a modern Carrington event could seal our fate. But if not…what comes next? These are questions worth getting up in the morning to think about. And if there is a useful lesson from history, it is the quote misattributed to Yogi Berra: “In theory there’s no difference between theory and practice; in practice, there usually is.”
I’ve found nothing new at the Crisis and Recovery Database. What has been happening there?
Is it anything like: Peter Turchin The Ginkgo Model of Societal Crisis – Peter Turchin
http://peterturchin.com/cliodynamica/the-ginkgo-model-of-societal-crisis/
Scoring some 30 cases of societies going through major crises, counting population declines, major epidemics, elites suffering downward mobility, dispossession, and/or extermination, killing of a leader, transformative revolutions, civil wars, and fragmentation and/or loss of territory.
The maximum observed in 8, the mean is 4.5, and the median and mode are 5. Only a small number had a small number: 0 or 1. So it’s hard to survive a crisis without major calamities, but it is not impossible to do so. So we must try to find out what made that survival possible, something like Isaac Asimov’s “Nightfall” and his Foundation series.
The collapse of Western culture, and its threat to take the world with it, have relatively simple causes (B. F. Skinner, “Beyond Freedom and Dignity”, 1970). Western individualism is an attack on all cultures, including that of the West itself. Any culture that puts the rights of the individual before its own survival will perish. It is a matter for history to fill in the details
Skidson
Try Skidson.Online
You could even give it a name like, oh, say, “the Prime Radiant”. 🙂
The problem is the reflexivity of complex systems. Your actions change the outcome. This means that the prediction would have to normalize for its own effect in order to have the desired outcome (which may not be in the public prediction at all!) In this sense I think reading Asimov’s Foundation gives people the most intuition about this concept.
Can’t wait for your new book. One of my favourite lesser known authors with very unique and valuable insights.