Our Theory of Change
What day-to-day processes should we follow? How do decisions that we make today work towards our ultimate goals?
At Convergence, we’ve developed a Theory of Change detailing an outcome-based, high-level strategic plan on how to mitigate existential risk from transformative AI.
Understand
How we conduct research to investigate the societal implications of transformative AI
Holistic Strategy Research
Conduct research into meta-level strategies to maximize AI safety
Scenario Research
Publishing Scenario Reports
Release a research corpus on key AI scenarios and threat models
Governance Research
Publishing Governance Reports
Release a research corpus on key governance strategies and recommendations
Act
How we deploy insights from our research to inform key parties
AI Safety Consensus
Build convergence within the AI safety community on scenarios & governance
Public Awareness
Raise public awareness of key AI scenarios
Policymaker Advocacy
Advise government actors on key governance recommendations
Outcomes
How our work leads to more effective decision making in AI safety
Increased Alignment
Policymakers and the public are more focused on key scenarios and governance strategies
Better AI Governance
Government actors establish recommendations through legislation, executive action, and international agreement
More Effective Funding
Grantmakers support more projects contributing to key governance strategies
Better Coordination
AI safety organizations improve coordination as a result of greater consensus
Ultimate Goal
Mitigation of existential risk from AI
Our Key Beliefs
Existential risk research is critical and neglected.
It's clear to us that the accelerating development of AI technology leads to a high risk of critically dangerous outcomes on a global scale if not managed properly, up to and including the extinction of the human race.
It's also obvious that the number of people working to identify, categorize, and minimize these risks are orders of magnitude too small.
Because there aren't clear financial incentives and the potential risks are universal and long-term, existential risk research for AI is a "tragedy of the commons," akin to where climate change was 20 years ago: not worth investing in until it's too late. This must be changed.
Technical AI alignment research is important but not enough
The vast majority (roughly 75%) of individuals working in AI safety are focused on technical alignment - the problem of aligning frontier AI models with the intended goals, preferences, and ethical principles of the humans designing them.
Though technical alignment is critical, it's only one component of the broader effort to mitigate risks from AI technologies. It’s clear to us that successfully designing well-aligned AI models still does not prevent the development and consequent misuse of such models by malicious human parties.
As the costs to develop high-powered AI systems rapidly lower, more and more organizations will gain the ability to develop these AI models. As we’ve found with any military technology throughout history, human parties have wildly varying priorities and goals, many of which can and will cause catastrophic harm even if implemented by a perfectly aligned AI.
As a result, we must have global systems in place such that society is resilient to the effects of transformative AI, no matter whether AI models are misaligned or their human developers have malicious goals.
Minimal foundational research exists on AI scenarios or specific AI governance proposals
Though many one-off papers have been written about potential AI futures, there doesn’t exist any comprehensive reports modeling the most likely AI scenarios, the parameters that might lead a scenario to happen, or the societal threats that these AI scenarios might entail. We don’t have answers to questions such as:
A similar problem exists in AI governance. Though there are dozens of public calls to action around key governance proposals, there is very little detailed, systematic analysis of these proposals to answer questions such as:
Because we don’t have this foundational research, interested parties (including industries, researchers, policymakers) must currently consider dozens of scattered resources over many weeks before arriving at an informed position.
Even after gaining expertise in one domain, interested parties have no tools to help compare the likelihood or risk profile of other scenarios & interventions. They’re likely to develop blinders towards their area of focus and overstate its importance.
Our Methodology
Understand
How we conduct research to investigate the societal implications of transformative AI
Holistic Strategy Research
We spend at least 20% of our time conducting meta-level strategic investigation and reflection to determine the best course of action within AI Safety - both for ourselves, and for the community as a whole. This branch of research guides the overall direction of Convergence, including our two current research agendas, and allows us to remain flexible and opportunistic in the face of a rapidly shifting environment.
Projects in this category include creating effective theories of change, advocating for strategy research, and developing new conceptual frameworks such as risk modeling, information hazards, and downside risks.
Scenario Research
At Convergence, we’re launching a set of foundational technical reports on AI scenarios early in 2024. These technical reports will evaluate upcoming likely AI scenarios according to two key dimensions: strategic parameters and threat models.
Roughly, these two forms of research can be mapped to:
Strategic Parameters: Describing the inputs that impact how AI technology progresses
Threat Models: Describing the real-world outputs and outcomes of AI technological advancement
Combined, these two approaches should give us a comprehensive overview of potential future states of the world in response to the development of AI technologies.
Governance Research
Our scenario research will inform our work identifying and analyzing the best governance strategies that can be conducted in response to key scenarios.
At Convergence, we plan to launch a detailed report on effective AI governance strategies, and potential Theories of Victories, early in 2024. This will be closely aligned with our scenario research.
Simultaneously, we are also developing a research agenda providing a systematic framework to evaluate and consider specific governance recommendations. We will be conducting a deep analysis into key recommendations such as:
Act
How we deploy insights from our research to inform key parties
The AI Safety Community
Within the AI safety community, our goal is to build consensus on the most critical AI scenarios, and the optimal governance interventions necessary to improve them. Though we’re focused primarily on reducing existential risk, we’re also identifying ways in which we can most effectively support scenarios that result in humanity flourishing.
To achieve this goal, we’re looking into coordinating working groups, seminars, and conferences aimed at bringing together top AI safety researchers, aligning on key research, and presenting those topics externally.
Policymakers & Thought Leaders
For key individuals who will have a massive impact on the success of AI safety, our work is intended to accessibly summarize complex domains of information. In particular, we're developing the critical reference hub necessary to bring parties up to speed rapidly. Our work allows policymakers to compare and contrast scenarios and interventions, and learn about the AI safety consensus on important issues.
To achieve this goal, we’re actively writing policy briefs for key governance proposals, performing reach-outs to leading governance bodies, and advising several private organizations. In particular, we’re advocating for a small number of critical & neglected interventions.
The General Public
We recognize that in order to effectively enact governance interventions, we need the endorsement and concern of the constituents behind policymakers. As a result, a high priority of ours is to determine how best to distribute and popularize the lessons gained from our research. Our focus is building awareness and immediacy to the existential risk of AI.
To achieve this goal, we’re currently launching a number of initiatives, including a book on AI futures and a podcast with key public individuals on AI outcomes.
Outcomes
How our work leads to more effective decision making in AI safety
Increased Alignment
We expect that our work will help the community:
Better AI Governance
Convergence’s research will tremendously assist organizations seeking to create or influence governmental policies. We expect to see benefits for groups:
More Effective Project Funding
From distributing our research, we expect to see:
Better Coordination
We expect to see improved coordination among AI safety researchers leveraging our foundational research. Our impact could include the following:
Mitigation of existential risk from AI.
We are actively forging a future where AI is harnessed safely and ethically. Through our systematic research blended with proactive advocacy and community engagement, we hope to move the needle on the likelihood of global existential risk.
Through our research, we endeavor to demonstrate a clear, logical path from action to impact, underlining our commitment to collaboratively shaping a flourishing future with AI.
Get research updates from Convergence
Leave us your contact info and we’ll share our latest research, partnerships, and projects as they're released.
You may opt out at any time. View our Privacy Policy.