Publications

Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve more than 90% accuracy on popular benchmarks such as Measuring Massive Multitask Language Understanding1, limiting informed measurement of state-of-the-art LLM capabilities. Here, in response, we introduce Humanity’s Last Exam (HLE), a multi-modal benchmark at the frontier of human knowledge, designed to be an expert-level closed-ended academic benchmark with broad subject coverage. HLE consists of 2,500 questions across dozens of subjects, including mathematics, humanities and the natural sciences. HLE is developed globally by subject-matter experts and consists of multiple-choice and short-answer questions suitable for automated grading. Each question has a known solution that is unambiguous and easily verifiable but cannot be quickly answered by internet retrieval. State-of-the-art LLMs demonstrate low accuracy and calibration on HLE, highlighting a marked gap between current LLM capabilities and the expert human frontier on closed-ended academic questions. To inform research and policymaking upon a clear understanding of model capabilities, we publicly release HLE at https://lastexam.ai.

We thank Tosh et al. (1) for their interest in our research (2) but note that their analyses do not undermine the main findings of our article. Their suggestion that polity population divided by polity area should be one of the social complexity dimensions raises a number of issues. What does this ratio mean at large spatial scales, where populations are concentrated in large urban centers and much of the territory is not heavily populated? How are societies distributed across this variable and why? For example, a small-scale “simple” society could have a very high population density if it has access to a rich resource base. Tosh et al. (1) do not provide sufficient information or context to meaningfully interpret their results. The study by Chick (3), cited by Tosh et al. (1), was based on the Standard Cross-Cultural Sample, which is heavily weighted toward low-complexity, small-scale societies. The second component that Chick (3) found captures the differences in mobility and mode of production between agricultural and foraging societies, rather than complexity of organization per se. Tosh et al. (1) also argue without evidence that the proportion of variance explained by princip

Do human societies from around the world exhibit similarities in the way that they are structured, and show commonalities in the ways that they have evolved? These are long-standing questions that have proven difficult to answer. To test between competing hypotheses, we constructed a massive repository of historical and archaeological information known as “Seshat: Global History Databank.” We systematically coded data on 414 societies from 30 regions around the world spanning the last 10,000 years. We were able to capture information on 51 variables reflecting nine characteristics of human societies, such as social scale, economy, features of governance, and information systems. Our analyses revealed that these different characteristics show strong relationships with each other and that a single principal component captures around three-quarters of the observed variation. Furthermore, we found that different characteristics of social complexity are highly predictable across different world regions. These results suggest that key aspects of social organization are functionally related and do indeed coevolve in predictable ways. Our findings highlight the power of the sciences and humanities working together to rigorously test hypotheses about general rules that may have shaped human history.

Professor in the departments of ecology and evolutionary biology, anthropology and mathematics, University of Connecticut Getting people to cooperate in very large groups such as the EU is difficult. Understanding how humans have been able to create cooperative societies is also hard, because we cannot readily run experiments. Nevertheless, much progress can be achieved by taking a scientific approach to analysing historical data.

Ian Morris is a very unusual historian. Instead of tunneling into archives or digging up artifacts to answer narrow questions, he approaches history with a much wider lens. Looking at the big picture, he told Marc Parry of The Chronicle of Higher Education, “reveals patterns that play out over millennia, without historical actors really knowing what’s going on..

We thank Thomas for his comments (1) on our recent paper, in which we show that a very simple model can explain a remarkably high proportion of variance in historical data on the origins and spread of macrostates in Afroeurasian history (2). The core of Thomas’s critique is that our model is “too simple and too abstract” (1). We, however, see model simplicity as a desirable feature in developing theory that is both grounded in sociological mechanisms and tested with historical data. Too many previous attempts at theory building in the social sciences have foundered as a result of being highly complex and including in them too many mechanisms. The ability of our simple model to accurately predict data speaks for itself.

How did human societies evolve from small groups, integrated by face-to-face cooperation, to huge anonymous societies of today, typically organized as states? Why is there so much variation in the ability of different human populations to construct viable states? Existing theories are usually formulated as verbal models and, as a result, do not yield sharply defined, quantitative predictions that could be unambiguously tested with data. Here we develop a cultural evolutionary model that predicts where and when the largest-scale complex societies arose in human history. The central premise of the model, which we test, is that costly institutions that enabled large human groups to function without splitting up evolved as a result of intense competition between societies—primarily warfare. Warfare intensity, in turn, depended on the spread of historically attested military technologies (e.g., chariots and cavalry) and on geographic factors (e.g., rugged landscape). The model was simulated within a realistic landscape of the Afroeurasian landmass and its predictions were tested against a large dataset documenting the spatiotemporal distribution of historical large-scale societies in Afroeurasia between 1,500 BCE and 1,500 CE. The model-predicted pattern of spread of large-scale societies was very similar to the observed one. Overall, the model explained 65% of variance in the data. An alternative model, omitting the effect of diffusing military technologies, explained only 16% of variance. Our results support theories that emphasize the role of institutions in state-building and suggest a possible explanation why a long history of statehood is positively correlated with political stability, institutional quality, and income per capita.

The next decade is likely to be a period of growing instability in the United States and western Europe, which could undermine the sort of scientific progress you describe in the Opinion collection of '2020 visions' (Nature 463, 26–32; 2010). Quantitative historical analysis reveals that complex human societies are affected by recurrent — and predictable — waves of political instability (P. Turchin and S. A. Nefedov Secular Cycles Princeton Univ. Press; 2009). In the United States, we have stagnating or declining real wages, a growing gap between rich and poor, overproduction of young graduates with advanced degrees, and exploding public debt. These seemingly disparate social indicators are actually related to each other dynamically. They all experienced turning points during the 1970s. Historically, such developments have served as leading indicators of looming political instability.

In times of violence, people tend to hide their valuables, which are later recovered unless the owners had been killed or driven away. Thus, the temporal distribution of unrecovered coin hoards is an excellent proxy for the intensity of internal warfare. We use this relationship to resolve a long-standing controversy in Roman history. Depending on who was counted in the early Imperial censuses (adult males or the entire citizenry including women and minors), the Roman citizen population of Italy either declined, or more than doubled, during the first century BCE. This period was characterized by a series of civil wars, and historical evidence indicates that high levels of sociopolitical instability are associated with demographic contractions. We fitted a simple model quantifying the effect of instability (proxied by hoard frequency) on population dynamics to the data before 100 BCE. The model predicts declining population after 100 BCE. This suggests that the vigorous growth scenario is highly implausible.

If we are to learn how to develop a healthy society, we must transform history into an analytical, predictive science, argues Peter Turchin. He has identified intriguing patterns across vastly different times and places. What caused the collapse of the Roman Empire? More than 200 explanations have been proposed1, but there is no consensus about which explanations are plausible and which should be rejected. This situation is as risible as if, in physics, phlogiston theory and thermodynamics coexisted on equal terms. This state of affairs is holding us back. We invest in medical science to preserve the health of our bodies, and in environmental science to maintain the health of ecosystems. Yet our understanding of what makes societies healthy is in the pre-scientific stage. Sociology that focuses on the past few years or decades is important. In addition, we need a historical social science, because processes that operate over long timescales can affect the health of societies. It is time for history to become an analytical, and even a predictive, science.

Political instability in collapsed or collapsing states is one of the greatest sources of human misery. Since the end of the cold war, withinstate conflicts, such as civil wars and separatist rebellions, have caused ten times more deaths than have wars between states. The indirect, non-military consequences of internal conflict — civilian casualties, refugees, wrecked economies, famine and disease — are orders of magnitude worse than the direct, military outcomes. In Fixing Failed States, former Afghan finance minister Ashraf Ghani and development-policy expert Clare Lockhart analyse why states crumble and propose a framework for rebuilding state capability.

One of the greatest challenges in the modern biological and social sciences is to understand the evolution of cooperative behaviour. General outlines of the answer to this puzzle are currently emerging as a result of developments in the theories of kin selection, reciprocity, multilevel selection and cultural group selection. The main conceptual tool used in probing the logical coherence of proposed explanations has been game theory, including both analytical models and agent-based simulations. The game-theoretic approach yields clear-cut results but assumes, as a rule, a simple structure of payoffs and a small set of possible strategies. Here we propose a more stringent test of the theory by developing a computer model with a considerably extended spectrum of possible strategies. In our model, agents are endowed with a limited set of receptors, a set of elementary actions and a neural net in between. Behavioural strategies are not predetermined; instead, the process of evolution constructs and reconstructs them from elementary actions. Two new strategies of cooperative attack and defence emerge in simulations, as well as the well-known dove, hawk and bourgeois strategies. Our results indicate that cooperative strategies can evolve even under such minimalist assumptions, provided that agents are capable of perceiving heritable external markers of other agents.

In their study of predator–prey cycles, investigators have assumed that they do not need to worry about evolution. The discovery of population cycles driven by evolutionary factors will change that view. Ecologists studying population dynamics prefer not to bother with the possibility of evolutionary change affecting their study organisms. This is sensible, because understanding the results of interactions between, for example, populations of predators and prey is already a complicated task. Making the assumption that evolutionary processes are too slow on ecological scales greatly eases the task of modelling the commonly observed population oscillations. But an elegant study by Yoshida et al.1 (page 303 of this issue) decisively demonstrates that this simplification might no longer be tenable.

Understanding spatial population dynamics is fundamental for many questions in ecology and conservation1,2,3,4. Many theoretical mechanisms have been proposed whereby spatial structure can promote population persistence, in particular for exploiter–victim systems (host–parasite/pathogen, predator–prey) whose interactions are inherently oscillatory and therefore prone to extinction of local populations5,6,7,8,9,10,11. Experiments have confirmed that spatial structure can extend persistence11,12,13,14,15,16, but it has rarely been possible to identify the specific mechanisms involved. Here we use a model-based approach to identify the effects of spatial population processes in experimental systems of bean plants (Phaseolus lunatus), herbivorous mites (Tetranychus urticae) and predatory mites (Phytoseiulus persimilis). On isolated plants, and in a spatially undivided experimental system of 90 plants, prey and predator populations collapsed; however, introducing habitat structure allowed long-term persistence. Using mechanistic models, we determine that spatial population structure did not contribute to persistence, and spatially explicit models are not needed. Rather, habitat structure reduced the success of predators at locating prey outbreaks, allowing between-plant asynchrony of local population cycles due to random colonization events.

Large oscillations in the populations of Norwegian lemmings have mystified both professional ecologists and lay public1,2,3. Ecologists suspect that these oscillations are driven by a trophic mechanism4,5: either an interaction between lemmings and their food supply, or an interaction between lemmings and their predators. If lemming cycles are indeed driven by a trophic interaction, can we tell whether lemmings act as the resource (‘prey’) or the consumer (‘predator’)? In trophic interaction models, peaks of resource density generally have a blunt, rounded shape, whereas peaks of consumer density are sharp and angular. Here we have applied several statistical tests to three lemming datasets and contrasted them with comparable data for cyclic voles. We find that vole peaks are blunt, consistent with their cycles being driven by the interaction with predators. In contrast, the shape of lemming peaks is consistent with the hypothesis that lemmings are functional predators, that is, their cycles are driven by their interaction with food plants. Our findings suggest that a single mechanism, such as interaction between rodents and predators, is unlikely to provide the ‘universal’ explanation of all cyclic rodent dynamics.

Population cycles occur frequently in forest insects. Time-series analysis of fluctuations in one such insect, the southern pine beetle (Dendroctonus frontalis), suggests that beetle dynamics are dominated by an ecological process acting in a delayed density-dependent manner. The hypothesis that delayed density dependence in this insect results from its interaction with predators was tested with a long-term predator-exclusion experiment. Predator-imposed mortality was negligible during the increase phase, grew during the year of peak population, and reached a maximum during the period of population decline. The delayed nature of the impact of predation suggests that predation is an important process that contributes significantly to southern pine beetle oscillations.

SEVERAL recent reviews of published life tables1–3 concluded that density-dependent regulation is infrequent in insect populations, prompting a vigorous debate among eco legists4–10. Little attention, however, has been directed to one issue: most life-table analyses look only for direct (not-lagged) density dependence. Thus, there is a real danger that populations characterized by delays in regulation will be relegated to a density-independent limbo by an analysis not equipped to recognize such behaviour. I have evaluated the evidence for delayed density dependence in population dynamics of 14 forest insects, and assessed the effect of regulation lags on the likelihood of detecting direct density dependence. Eight cases exhibited clear evidence for delayed density dependence and lag-induced oscillations, but direct density dependence was detected in only one of these. This result suggests that traditional analyses will not, in general, detect density-dependent regulation in populations that are characterized by lags and complex dynamic behaviour.