Publications

Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve more than 90% accuracy on popular benchmarks such as Measuring Massive Multitask Language Understanding1, limiting informed measurement of state-of-the-art LLM capabilities. Here, in response, we introduce Humanity’s Last Exam (HLE), a multi-modal benchmark at the frontier of human knowledge, designed to be an expert-level closed-ended academic benchmark with broad subject coverage. HLE consists of 2,500 questions across dozens of subjects, including mathematics, humanities and the natural sciences. HLE is developed globally by subject-matter experts and consists of multiple-choice and short-answer questions suitable for automated grading. Each question has a known solution that is unambiguous and easily verifiable but cannot be quickly answered by internet retrieval. State-of-the-art LLMs demonstrate low accuracy and calibration on HLE, highlighting a marked gap between current LLM capabilities and the expert human frontier on closed-ended academic questions. To inform research and policymaking upon a clear understanding of model capabilities, we publicly release HLE at https://lastexam.ai.

Professor in the departments of ecology and evolutionary biology, anthropology and mathematics, University of Connecticut Getting people to cooperate in very large groups such as the EU is difficult. Understanding how humans have been able to create cooperative societies is also hard, because we cannot readily run experiments. Nevertheless, much progress can be achieved by taking a scientific approach to analysing historical data.

The next decade is likely to be a period of growing instability in the United States and western Europe, which could undermine the sort of scientific progress you describe in the Opinion collection of '2020 visions' (Nature 463, 26–32; 2010). Quantitative historical analysis reveals that complex human societies are affected by recurrent — and predictable — waves of political instability (P. Turchin and S. A. Nefedov Secular Cycles Princeton Univ. Press; 2009). In the United States, we have stagnating or declining real wages, a growing gap between rich and poor, overproduction of young graduates with advanced degrees, and exploding public debt. These seemingly disparate social indicators are actually related to each other dynamically. They all experienced turning points during the 1970s. Historically, such developments have served as leading indicators of looming political instability.

If we are to learn how to develop a healthy society, we must transform history into an analytical, predictive science, argues Peter Turchin. He has identified intriguing patterns across vastly different times and places. What caused the collapse of the Roman Empire? More than 200 explanations have been proposed1, but there is no consensus about which explanations are plausible and which should be rejected. This situation is as risible as if, in physics, phlogiston theory and thermodynamics coexisted on equal terms. This state of affairs is holding us back. We invest in medical science to preserve the health of our bodies, and in environmental science to maintain the health of ecosystems. Yet our understanding of what makes societies healthy is in the pre-scientific stage. Sociology that focuses on the past few years or decades is important. In addition, we need a historical social science, because processes that operate over long timescales can affect the health of societies. It is time for history to become an analytical, and even a predictive, science.

Political instability in collapsed or collapsing states is one of the greatest sources of human misery. Since the end of the cold war, withinstate conflicts, such as civil wars and separatist rebellions, have caused ten times more deaths than have wars between states. The indirect, non-military consequences of internal conflict — civilian casualties, refugees, wrecked economies, famine and disease — are orders of magnitude worse than the direct, military outcomes. In Fixing Failed States, former Afghan finance minister Ashraf Ghani and development-policy expert Clare Lockhart analyse why states crumble and propose a framework for rebuilding state capability.

One of the greatest challenges in the modern biological and social sciences is to understand the evolution of cooperative behaviour. General outlines of the answer to this puzzle are currently emerging as a result of developments in the theories of kin selection, reciprocity, multilevel selection and cultural group selection. The main conceptual tool used in probing the logical coherence of proposed explanations has been game theory, including both analytical models and agent-based simulations. The game-theoretic approach yields clear-cut results but assumes, as a rule, a simple structure of payoffs and a small set of possible strategies. Here we propose a more stringent test of the theory by developing a computer model with a considerably extended spectrum of possible strategies. In our model, agents are endowed with a limited set of receptors, a set of elementary actions and a neural net in between. Behavioural strategies are not predetermined; instead, the process of evolution constructs and reconstructs them from elementary actions. Two new strategies of cooperative attack and defence emerge in simulations, as well as the well-known dove, hawk and bourgeois strategies. Our results indicate that cooperative strategies can evolve even under such minimalist assumptions, provided that agents are capable of perceiving heritable external markers of other agents.

In their study of predator–prey cycles, investigators have assumed that they do not need to worry about evolution. The discovery of population cycles driven by evolutionary factors will change that view. Ecologists studying population dynamics prefer not to bother with the possibility of evolutionary change affecting their study organisms. This is sensible, because understanding the results of interactions between, for example, populations of predators and prey is already a complicated task. Making the assumption that evolutionary processes are too slow on ecological scales greatly eases the task of modelling the commonly observed population oscillations. But an elegant study by Yoshida et al.1 (page 303 of this issue) decisively demonstrates that this simplification might no longer be tenable.

Understanding spatial population dynamics is fundamental for many questions in ecology and conservation1,2,3,4. Many theoretical mechanisms have been proposed whereby spatial structure can promote population persistence, in particular for exploiter–victim systems (host–parasite/pathogen, predator–prey) whose interactions are inherently oscillatory and therefore prone to extinction of local populations5,6,7,8,9,10,11. Experiments have confirmed that spatial structure can extend persistence11,12,13,14,15,16, but it has rarely been possible to identify the specific mechanisms involved. Here we use a model-based approach to identify the effects of spatial population processes in experimental systems of bean plants (Phaseolus lunatus), herbivorous mites (Tetranychus urticae) and predatory mites (Phytoseiulus persimilis). On isolated plants, and in a spatially undivided experimental system of 90 plants, prey and predator populations collapsed; however, introducing habitat structure allowed long-term persistence. Using mechanistic models, we determine that spatial population structure did not contribute to persistence, and spatially explicit models are not needed. Rather, habitat structure reduced the success of predators at locating prey outbreaks, allowing between-plant asynchrony of local population cycles due to random colonization events.

Large oscillations in the populations of Norwegian lemmings have mystified both professional ecologists and lay public1,2,3. Ecologists suspect that these oscillations are driven by a trophic mechanism4,5: either an interaction between lemmings and their food supply, or an interaction between lemmings and their predators. If lemming cycles are indeed driven by a trophic interaction, can we tell whether lemmings act as the resource (‘prey’) or the consumer (‘predator’)? In trophic interaction models, peaks of resource density generally have a blunt, rounded shape, whereas peaks of consumer density are sharp and angular. Here we have applied several statistical tests to three lemming datasets and contrasted them with comparable data for cyclic voles. We find that vole peaks are blunt, consistent with their cycles being driven by the interaction with predators. In contrast, the shape of lemming peaks is consistent with the hypothesis that lemmings are functional predators, that is, their cycles are driven by their interaction with food plants. Our findings suggest that a single mechanism, such as interaction between rodents and predators, is unlikely to provide the ‘universal’ explanation of all cyclic rodent dynamics.

SEVERAL recent reviews of published life tables1–3 concluded that density-dependent regulation is infrequent in insect populations, prompting a vigorous debate among eco legists4–10. Little attention, however, has been directed to one issue: most life-table analyses look only for direct (not-lagged) density dependence. Thus, there is a real danger that populations characterized by delays in regulation will be relegated to a density-independent limbo by an analysis not equipped to recognize such behaviour. I have evaluated the evidence for delayed density dependence in population dynamics of 14 forest insects, and assessed the effect of regulation lags on the likelihood of detecting direct density dependence. Eight cases exhibited clear evidence for delayed density dependence and lag-induced oscillations, but direct density dependence was detected in only one of these. This result suggests that traditional analyses will not, in general, detect density-dependent regulation in populations that are characterized by lags and complex dynamic behaviour.