Publications

In this post, we describe and outline ways to decompose strategy research. Specifically, we break it down into the following four components: mapping the space, constructing strategies, modelling causality, and prioritizing between strategies.

To get what we value we must make good decisions. To make these decisions we must know what relevant facts are true. But the world is so complex that we cannot check everything directly ourselves and so must defer to topic “experts” for some things. How should we choose these experts and how much should we believe what they tell us? In this document, I’ll describe a way to evaluate experts.

This post will overview the crucial questions that we (Convergence) believe do or should influence different longtermists’ views and choices regarding the best timing of work and donations.

A person’s “moral circle” classically refers to the entities that that person perceives as having moral standing, or as being worthy of moral concern. And “moral circle expansion” classically refers to moral circles moving “outwards”, for example from kin to people of other races to nonhuman animals, such that “more distant” entities are now in one’s “circle of concern”.

This post argues that one useful way to come up with, and assess the expected value of, actions to improve the long-term future is to consider how “benevolent”, “intelligent”, and “powerful” various actors are, and how various actions could affect those actors’ benevolence, intelligence, and power.

We are often forced to make decisions under conditions of uncertainty. This may be empirical uncertainty, or it may be moral uncertainty. But what if you don’t believe that “morally important” is a coherent concept?

In 2002, Nick Bostrom introduced the concept of differential technological development. This concept is highly relevant to many efforts to do good in the world (particularly, but not only, from the perspective of reducing existential risks). Other writers have generalised Bostrom’s concept, using the terms “differential intellectual progress” or just “differential progress”. I think these generalisations are actually even more useful.

Existential risks are considered by many to be among the most pressing issues of our time (see e.g. The Precipice). But what, precisely, do we mean by “existential risks”?

This post provides spreadsheet you can use for making your own estimates of existential risks (or of similarly “extreme” outcomes), and announces a database of estimates of existential risk (or similarly extreme outcomes), which I hope can be collaboratively expanded and updated. Finally, it discusses why I think this database may be valuable discusses some pros and cons of using or making such estimates.

Toby Ord’s The Precipice is an ambitious and excellent book. Among many other things, Ord attempts to survey the entire landscape of existential risks[1] humanity faces. In this post, I will discuss whether Ord may understate the uncertainty of these estimates, and ambiguity about what he’s actually estimating when he estimates the risk from “unaligned AI”.

It’s a widely held belief in the existential risk reduction community that we are likely to see a great technological transformation in the next 50 years. A technological transformation will either cause flourishing, existential catastrophe, or other forms of large change for humanity.

Moloch is a poetic way of describing failures of coordination and coherence inside an agent or between agents and the generation of harmful subcomponents or harmful agents. Perhaps this could be decomposed further, or at least partially covered, by randomly generated accidents, Goodhart’s law failures, and conflicts of optimization.