Those of you who’ve read my books know that in addition to my research on the evolution of large-scale human societies and the rise of centralized states and empires, I am also interested in the reverse process by which an empire loses cohesion and gradually crumbles into a ‘failed state’ (these two directions in my cliodynamics research are reflected in parts I and II, respectively of War and Peace and War). I have found Jack Goldstone’s ideas on processes leading to state collapse and descent into civil war particularly useful, and I and colleagues extended his demographic-structural theory and tested it with empirical data on Medieval and Early Modern England, France, and Russia, as well as ancient Rome (this was published in Secular Cycles co-authored with a Russian historian Sergey Nefedov; ‘secular cycles’ refers to alternating periods of strong economy, population growth and cohesive states with periods of economic and demographic decline and pervasive political violence).

I have given a number of talks on this research, and the most frequent questions I get after the presentation is, “so where are we in the secular cycle? Is the American empire about to collapse?” My initial inclination was to deflect such questions by pointing out that human societies changed dramatically during the last two centuries, and thus we should not expect that theories developed, and tested with data on ancient, medieval, and early-modern empires would work on societies we live in.

However, as our knowledge of secular cycles in historical societies became richer and more detailed, I couldn’t help noticing that many of social, economic, and political trends we observe today appear to be very similar to trends we saw in historical societies that my colleagues and I studied. Naturally, the demographic-structural theory needs to be reformulated, but can it still be useful in understanding the trends affecting our own societies?

Five or so years ago I started collecting data on structural-demographic variables in the United States (as a terminological note, I decided to move up ‘structural’ because the demographic component is not quite as important when the theory is applied to modern societies). The period I focused on was from the beginning of the Republic, roughly 1780, to today. One big advantage of working with American history is that quantitative data of reasonable quality are much more plentiful than for other historical societies I dealt with in the past. Thus, for a number of key structural-demographic variables I was able to find multiple ‘proxies.’ A proxy is a variable that is closely correlated with whatever we want to study, but that is hard to measure. Proxies, thus, provide a way to follow the dynamics of such hard-to-measure things. For example, air temperature is the average velocity with which molecules in the atmosphere bounce against each other. It’s hard to observe directly, but we can readily measure it by observing the height of the mercury column in the thermometer. This height is the proxy for temperature. Physicists and biologists use proxies all the time. To give another example, when I was population  ecologist and studied population dynamics of lemmings, we used a proxy for their population densities: the number of lemmings captured in mouse traps (rodents per trap-night).

It is surprising to me that historians and social scientists don’t use the proxies as much as natural scientists do. True, the proxies that we have access to in historical social science are not as closely related to things we want to measure (as in my thermometer example). And we should always check how well our proposed proxies work. Still, it’s a very productive approach.

For example, one of the most important variables in structural-demographic theory is ‘intraelite competition.’ How do we operationalize this quantity? In his 1991 book Jack Goldstone made a brilliant suggestion – we can proxy it by the increase in the number of enrollments in institutions of higher learning. Why should it work? Well, four decades ago another brilliant historical sociologist, Randall Collins, published an important, but largely ignored book, The Credential Society, in which he argued that most youngsters go to college not to expand their minds, but to simply obtain a piece of paper (the diploma) that improves their chances of getting a lucrative job (shame about it). Thus, when competition for elite positions intensifies, more people seek to obtain advanced degrees.

In his study of the seventeenth century crisis in England, Goldstone looked at enrollments at such universities as Oxford and Cambridge. He found that indeed, the enrollments at Oxford, for example, ballooned during the first half of the seventeenth century, in the period preceding the English Civil War. And it wasn’t just a long-term modernization trend in which people came to value education more. When intraelite competition subsided in the early eighteenth century, so did Oxford enrollments. I am not saying that everybody who went to Oxford just wanted to get the credentials, rather than education. But those who were really interested in knowledge were a decided minority.

This ironic cover of Academe, the magazine of the American Association of University Professors, portrays college education as mass-production of diplomaed graduates

Of course we have to be careful. Many factors can affect the number of youth seeking higher education, not only increased competition for high-quality jobs. So we should seek other proxies for the quantity of interest, and check whether they tell the same story. For seventeenth century’s England, we can also look to such proxies of competition as the amount of litigation. Another one that I used for both England and France was the frequency of dueling. It turns out that dueling epidemics tend to develop during periods of high intraelite competition. Makes sense.

This issue of proxies is very interesting, and it is of high importance to making history into science (‘cliodynamics’), but let’s go back to the structural-demographic dynamics in America. Over the last several years I have amassed an incredible amount of data. As I mentioned before, for many variables of interest it was possible to locate a number of proxies, so we can really pinpoint the trends and, even more important, trend reversals. I am currently putting these results together in a book-length publication.

One of the most important variables is sociopolitical instability. Sociopolitical instability includes all kinds of collective (group-level) violence that occurs within states. So it’s that sort of violence that falls between individual crime, on one hand, and interstate warfare, on the other. This variable is important because it is what the structural-demographic theory aims to explain. We need good data about it.

Because this is such a critical variable, I put a lot of effort into trying to find good empirical measurements. I found a number of databases that addressed certain aspects of sociopolitical instability, but no comprehensive such database. So in the end, I realized that I had to construct it myself, which I did by merging the pre-existing databases with comprehensive searches of electronic resources, such as LexisNexis database of newspaper articles.

One curious consequence is that, I believe, I am now a real historian – because I worked with sources in archives. True, my sources were newspaper articles from the nineteenth century, and the archives were electronic ones (so I did not have to inhale the dust of medieval manuscripts). So I suppose true historians will probably not induct me into their guild. No matter, it was a remarkable experience. And, in some ways, an appalling one. For one thing, I can’t believe how much time I actually spent deciphering old newspaper articles while constructing this database. Most people at my stage of career simply don’t do this type of ‘manual’ work. The second appalling part was due to the nature of my research. Since I was investigating political violence, I had to read through some really appalling descriptions of truly horrific incidents (which I will not inflict on you). On a number of occasions I was so affected by what I juat read that I had to drop this project for a while and to switch to something else to regain my faith in the humanity.

In any case, this project is now done, and published. The paper came out last week in the Journal of Peace Research. You can find the paper on my cliodynamics.info site. Additionally, the paper gives a systematic exposition of the structural-demographic theory as reformulated for post-Industrial Revolution societies (initially, the article focused on the construction and analysis of the database, but the journal editors asked me to add the theoretical part, and I don’t regret it).

Many of you will not be able to access the article at JPR, but here is the article reprint, which is the same as published version but without correct page numbers.

And here is the Appendices with additional graphs and other info.