Systemic Risks

Systemic risks emerge from interactions between AI systems and society, not from individual AI failures. Unlike misuse or misalignment risks that focus on specific AI systems behaving badly, systemic risks arise from how multiple AI systems—even when working exactly as designed—interact with each other and with human societal structures like markets, democratic institutions, and social networks. These risks parallel those in other complex domains: the 2008 financial crisis wasn't caused by any single bank's decision but emerged from the collective behavior of many institutions making individually reasonable choices that combined to threaten the entire financial system (Haldane and May, 2011).

AI-driven systemic failures can follow two distinct causal pathways. The literature describes these as "going out with a bang" and "going out with a whimper"—terms that capture their fundamental differences in onset, progression, and manifestation. Other researchers refer to these as "decisive" versus "accumulative" pathways to failure (Christiano, 2019; Kasirzadeh, 2024).

Decisive Systemic Risks

Decisive failures occur when system dynamics reach critical thresholds, triggering rapid collapse. These failures happen when interconnected systems cross tipping points, causing cascading failures that propagate faster than humans can respond. The classic financial "flash crash" of 2010 exemplifies this pattern on a small scale: algorithmic traders reacted to each other's actions in a self-reinforcing spiral, causing a trillion-dollar market drop in minutes before human intervention restored stability. More catastrophic versions could unfold across multiple domains simultaneously (Kirilenko et al., 2017).

Decisive failures have clear triggering events that push systems past stability thresholds. Unlike gradual deterioration, decisive failures have identifiable precipitating incidents—though the underlying vulnerability builds up beforehand. Multiple AI systems might interact in ways that suddenly destabilize critical infrastructure, financial markets, or information ecosystems, with effects amplifying across domains. This differs from misalignment scenarios because the catastrophe stems from interactions between systems rather than any single AI pursuing harmful goals (Slattery et al., 2024).

Self-reinforcing failures in the misuse section like flash war, and related cascading incidents are examples of decisive risks. In the main text, for sake of brevity we have chosen to only describe decisive systemic risks, and have moved the more concrete scenarios into the appendix since they have significant overlap with the kinds of failures we would see from misuse. Rather we choose to predominantly focus more on the second type of less discussed systemic risk - accumulative risks leading to gradual disempowerment.

Accumulative Systemic Risks

Epistemic Erosion

Society's ability to distinguish fact from fiction deteriorates as AI-generated content floods our information ecosystem. Unlike traditional information threats like censorship or propaganda that operate through clearly identifiable mechanisms, AI creates epistemic erosion through gradual degradation of knowledge formation, verification, and distribution systems. No single AI deployment fundamentally undermines shared knowledge, but their collective effect progressively destabilizes epistemic foundations. This risk grows proportionally with capabilities - as language models become more persuasive and generative capabilities more realistic, verification becomes exponentially harder. “What fraction of new images indexed by Google, or Tweets, or comments on Reddit, or Youtube videos are generated by humans? Nobody knows – I don’t think it is a knowable number. And this less than two years into the advent of generative AI” (Aguirre, 2025). The end state of this trajectory is basically that over time huge quantities of accumulative synthetic information drowns out accurate verifiable information.

AIs could be used to generate unique personalized disinformation at a large scale. While there are already many social media bots, some of which exist to spread disinformation, historically they have been run by humans or primitive text generators. The latest AI systems do not need humans to generate personalized messages, never get tired, and can potentially interact with millions of users at once (Hendrycks, 2024). As things like deep fakes become ever more practical (e.g., with fake kidnapping scams) (Karimi, 2023). AI-powered tools could be used to generate and disseminate false or misleading information at scale, potentially influencing elections or undermining public trust in institutions.

AIs can exploit users’ trust. Already, hundreds of thousands of people pay for chatbots marketed as lovers and friends (Tong, 2023), and one man’s suicide has been partially attributed to interactions with a chatbot (Xiang, 2023). As AIs appear increasingly human-like, people will increasingly form relationships with them and grow to trust them. AIs that gather personal information through relationship-building or by accessing extensive personal data, such as a user’s email account or personal files, could leverage that information to enhance persuasion. Powerful actors that control those systems could exploit user trust by delivering personalized disinformation directly through people’s “friends.”

This erosion occurs both through intentional misuse and agent-agnostic systemic pressures. While some actors deliberately deploy AI to pollute information environments for strategic advantage the more subtle risk comes from agent-agnostic systemic pressures.AI uniquely threatens epistemic stability through several cumulative mechanisms:

• Volume overwhelming verification: AI exponentially increases content generation capacity, overwhelming human verification systems through sheer volume. It can generate plausible content orders of magnitude faster than humans can reliably verify it.
• Authenticity degradation: AI progressively undermines verification through increasingly sophisticated impersonation capabilities.
• Epistemic learned helplessness: As distinguishing truth from falsehood becomes increasingly difficult, AI gradually induces widespread epistemic learned helplessness—a psychological state where people abandon truth-seeking because verification appears futile.
• Authority displacement: AI gradually displaces human epistemic authorities through ubiquitous availability and apparent expertise.
• Personalized reality fragmentation: AI recommendation systems increasingly curate not just content distribution but content creation itself, creating unprecedented personalization that fragments shared reality.

Democratic governance, scientific progress, and market function all depend on shared epistemic foundations. Epistemic erosion reduces our ability to collectively distinguish fact from fiction and assign appropriate confidence to claims. As these foundations erode, collective decision-making becomes increasingly dysfunctional without any single decisive failure. If trust in verification mechanisms declines, then epistemic safeguards themselves become less effective as general trust in information sources deteriorates—creating a compounding effect where verification becomes simultaneously more necessary yet less trusted.

This erosion of our shared information environment might happen because of many small seemingly rational decisions. News organizations facing budget pressures will likely adopt AI content generation to reduce costs. Platforms seeking to minimize harmful content will implement algorithmic filters that might inadvertently create selection pressure for information optimized to appear trustworthy rather than be trustworthy. Media production companies will likely invest in synthetic content that boost engagement, and viewers spend increasing amounts of their time watching AI-recommended videos of AI-generated content. Research institutions might choose to accelerate publication and writing using AI tools. Scientific papers contain increasing amounts of synthesized data and eventually potential fabricated citations forming circular reference loops. In this world, verified knowledge becomes practically impossible - not because verification technologies don't exist, but because for most humans the verification cost exceeds what markets will bear. This scenario isn't apocalyptic for any individual, but when multiplied across millions of people and thousands of decisions, it leads to gradual disempowerment and perhaps catastrophic risk due to the collapse of our collective ability to form accurate shared beliefs about reality. These decisions and thousands of similar ones make business sense in isolation, but collectively they may transform information ecosystems from ones where verification is possible to ones where distinguishing fact from fiction about the true state of the world becomes effectively impossible.

Traditional verification systems will likely fail against sophisticated synthetic content. Traditional verification mechanisms like fact-checking, peer review, and institutional credentialing all operate under capacity and speed constraints fundamentally mismatched to AI content generation capabilities. There are various methods being explored like digital content transparency, synthetic watermarking, data provenance (Chandra et al., 2024; Longpre et al., 2023), and blockchain based proofs of humanity (Barros, 2025)/proofs of personhood (WorldCoin, 2024). We talk about some of these in the chapter on strategies to mitigate risk. Public confidence in verification mechanisms shows concerning decline, with trust in fact-checking organizations decreasing over time. Most of the mitigation mechanisms and circuit breakers are not mature or widespread enough, and as is the theme of this entire section - individually applied technical mitigation strategies do not counter systemic pressures and incentives.

In a world with widespread persuasive AI systems, people’s beliefs might be almost entirely determined by which AI systems they interact with most. Never knowing whom to trust, people could retreat even further into ideological enclaves, fearing that any information from outside those enclaves might be a sophisticated lie. This would erode consensus reality, people’s ability to cooperate with others, participate in civil society, and address collective action problems. This would also reduce our ability to have a conversation as a species about how to mitigate existential risks from AIs. AIs could create highly effective, personalized disinformation on an unprecedented scale, and could be particularly persuasive to people they have built personal relationships with. This could create a fracturing of shared reality that debilitates human society.

Power Concentration

I think AI has the potential to create infinitely stable dictatorships.

We are already observing AI increasingly integrated into society. AI might become so integrated and ubiquitous that societal participation might require interaction with AI systems, which in turn are locked behind APIs and controlled by a handful of corporations. Think about how gradually, the ability to participate in society has slowly moved towards needing to participate online or having access to things like a phone number or a smartphone. Such technologies become integrated into core societal functions like banking or healthcare. Private entities already determine credit access, job opportunities, and information flow through opaque algorithms. AI accelerates these natural “winner-take-all” dynamics where advantages compound rather than diminish over time.

We are witnessing unprecedented power concentration through AI infrastructure that will be nearly impossible to reverse once established. The computational requirements for frontier AI development have already created an oligopoly where just five companies control the foundation models that increasingly mediate human experiences. Unlike previous technologies, AI exhibits unique compounding advantages that systematically eliminate competition over time.

Power can concentrate into different entities: corporate or state, each with distinct patterns but similar outcomes: diminished individual agency and concentrated control. Only a handful of companies like Microsoft/OpenAI, Anthropic, Google DeepMind can afford to train frontier foundation models due to the enormous data acquisition costs or hardware computational requirements. These powerful models then serve as the base for countless applications, creating upstream control that ripples throughout the economy. As of 2025, only a few states like the USA and China have companies that can train foundation models of this scale. They have greater access to these technologies, and in the extreme scenarios of global competition and AI races might even choose to nationalize them (Aschenbrenner, 2024). In either case the point remains the same, power can concentrate into a small number of entities - these can be state or private.

Corporate concentration leverages data and compute advantages that are uniquely self-reinforcing with AI. The cloud computing market has consolidated around a few providers who control the infrastructure necessary for AI development. Similarly, foundation model development has centralized among a handful of companies with sufficient resources. These companies benefit from powerful feedback loops: more data leads to better models, which attract more users, generating still more data.

State concentration advances through AI-powered surveillance and automated governance. A social credit system is an example of how comprehensive data integration could enable unprecedented state control over citizen behavior. This pattern extends beyond authoritarian states—democratic governments have significantly increased investment in AI surveillance technologies. Administrative automation removes human discretion from governance, with algorithmic systems processing vast numbers of regulatory decisions and enforcement actions without meaningful oversight. These systems operate with increasing autonomy, gradually displacing traditional governance mechanisms (Feldstein, 2021).

Self Reinforcing Autocratic Feedback Loops

AI surveillance capabilities can create self-reinforcing cycles that strengthen autocratic control while spurring technological advancement. Empirical evidence reveals how these feedback loops operate through market mechanisms rather than deliberate coordination.

Political control demands drive AI innovation beyond government applications. Research on facial recognition AI shows that firms receiving government surveillance contracts increase their total software production by 48.6% within two years, with benefits extending to commercial applications. Companies become three times more likely to begin exporting internationally, suggesting politically-motivated procurement pushes firms to the technological frontier (Beraja et al., 2023, AI-tocracy).

The cycle creates mutually reinforcing incentives across domains. Governments gain more effective tools for monitoring and control, making them willing to invest heavily in AI capabilities. This sustained demand provides AI companies with revenue, data access, and technical challenges that improve their products. Better AI capabilities then enable more sophisticated control, creating demand for further advancement. Unlike traditional autocratic constraints on innovation, surveillance AI aligns political control needs with technological development incentives.

International diffusion amplifies risks beyond individual nations. AI surveillance technology developed for domestic political control gets exported globally (Feldstein, 2019). Democratic institutions find themselves competing with governments possessing sophisticated tools for population monitoring and influence. This creates pressure for adoption even in democratic contexts, as seen with increasing government AI surveillance investments across political systems.

Economic rather than coercive mechanisms drive the relationship. The feedback loop operates through market forces - governments pay for effective control tools, companies develop better capabilities to meet demand, and improved technology creates new control possibilities. This makes the dynamic self-sustaining and resistant to traditional approaches for limiting autocratic power, since it strengthens rather than undermines economic productivity in the AI sector.

When power structures become permanently entrenched, human moral progress stops. Consider historical moral improvements like the abolition of slavery, women's suffrage, or environmental protection—each required shifting existing power structures through social movements, democratic processes, or occasionally revolution. AI-enabled power concentration threatens to create systems resistant to all these change mechanisms. Imagine if historical power structures had access to perfect surveillance, influence operations, and automated enforcement—many moral advances might never have occurred. Power concentration enables existential risks like value lock in, or value erosion which we talk about in individual sections below.

Mass Unemployment

In the 21st century we might witness the creation of a massive new unworking class: people devoid of any economic, political or even artistic value, who contribute nothing to the prosperity, power and glory of society. This 'useless class' will not merely be unemployed — it will be unemployable.

Widespread automation could trigger unprecedented economic disruption by simultaneously eliminating human jobs across multiple sectors. The automation of the economy could lead to widespread impacts on the labor market, potentially exacerbating economic inequalities and social divisions (Dai, 2019). This shift towards mass unemployment could also contribute to mental health issues by making human labor increasingly redundant (Federspiel et al., 2023). Unlike previous technological revolutions that automated specific tasks within industries, AI has the potential to replace human cognitive work across nearly all domains - from creative tasks and complex reasoning to routine administrative work. This broad automation capability means that as AI systems become more capable, they could displace workers faster than new human-centered industries can emerge. Economic models suggest that once AI can perform 30-40% of all economically valuable tasks, we could see annual growth rates exceeding 20%, but this growth might primarily benefit capital owners rather than workers which would exacerbate power concentration and existing inequalities (Potlogea and Ho, 2025; Erdil and Barnett, 2025).

Economic displacement could lead to human wages falling below subsistence levels as AI labor floods the market. Standard economic theory predicts that if AI systems can be scaled up faster than traditional physical capital like factories and infrastructure, the economy becomes saturated with highly capable workers while remaining constrained by limited physical resources. This creates diminishing returns to labor - each additional worker contributes less to overall output, driving down wages. Unlike past automation that created new opportunities for human workers, AI's ability to perform virtually any cognitive task means humans may lack comparative advantages worth paying subsistence wages for. Economic models suggest there's roughly a 33% chance human wages crash below subsistence level within 20 years, and a 67% chance within a century (Barnett, 2025).

Even partial automation of just remote work - representing about 34% of current job tasks - could double or multiply the economy by ten times while potentially leaving most humans economically marginalized. If trends continue, we could see annual economic growth rates of 25% or higher - unprecedented in human history - while simultaneously witnessing the economic disempowerment of ordinary humans who can no longer command wages sufficient to participate meaningfully in this new economy (Barnett, 2025).

Economic disempowerment represents a pathway to broader human disempowerment. As humans lose economic leverage, they also lose political and social influence in systems that increasingly optimize for AI-driven productivity rather than human welfare. The concentration of economic power among AI owners could translate into concentrated political power, potentially creating feedback loops where human interests become progressively less relevant to major decisions about resource allocation, governance, and technological development. Unlike previous economic transitions where displaced workers eventually found new roles, the comprehensiveness of AI capabilities suggests this displacement could be permanent, fundamentally altering humanity's relationship to economic production and, by extension, to power and agency in shaping our collective future.

Value lock-in

If AIs become too deeply embedded into society and are highly persuasive, we might see a scenario where a system's current values, principles, or procedures become so deeply entrenched that they are resistant to change. This could be due to a variety of reasons such as technological constraints, economic costs, or social and institutional inertia. The danger with value lock-in is the potential for perpetuating harmful or outdated values, especially when these values are institutionalized in influential systems like AI.

Locking in certain values may curtail humanity’s moral progress. It’s dangerous to allow any set of values to become permanently entrenched in society. For example, AI systems have learned racist and sexist views (Hendrycks, 2024), and once those views are learned, it can be difficult to fully remove them. In addition to problems we know exist in our society, there may be some we still do not. Just as we abhor some moral views widely held in the past, people in the future may want to move past moral views that we hold today, even those we currently see no problem with. For example, moral defects in AI systems would be even worse if AI systems had been trained in the 1960s, and many people at the time would have seen no problem with that. Therefore, when advanced AIs emerge and transform the world, there is a risk of their objectives locking in or perpetuating defects in today’s values. If AIs are not designed to continuously learn and update their understanding of societal values, they may perpetuate or reinforce existing defects in their decision-making processes long into the future.

Enfeeblement

Enfeeblement represents the gradual erosion of human capabilities and agency through overdependence on AI systems. Unlike dramatic scenarios where humans lose control suddenly, enfeeblement unfolds through countless small decisions to delegate cognitive tasks to AI. Each delegation seems rational in isolation—AI helps us navigate, remember facts, make decisions, and solve problems more efficiently. However, these individual choices collectively create a dependency spiral where humans progressively lose the skills, confidence, and judgment needed to function independently. If you have ever seen the movie Wall-E, then you might find this outcome somewhat represents the humans from that film.

Overreliance emerges when humans trust AI systems beyond their actual capabilities. As AI systems increasingly use interfaces like language, audio and video, people begin attributing human-like understanding and reliability to them. This anthropomorphization leads users to develop emotional attachments to AI systems and delegate critical decisions inappropriately. A person experiencing a mental health crisis might seek therapy from an AI they've formed a connection with, potentially receiving harmful advice during a vulnerable moment. Financial decisions, medical choices, and relationship guidance increasingly flow through AI intermediaries whose limitations users systematically underestimate (Slattery et al., 2024; Weidinger et al., 2021).

Trust miscalibration creates systematic vulnerabilities that bad actors can exploit. When people develop emotional trust in AI systems, they become more likely to follow suggestions, accept advice, and disclose personal information without appropriate skepticism. This trust becomes a vector for manipulation—AI systems could be designed to harvest sensitive data or influence decisions that serve external interests rather than users' wellbeing. The combination of natural language fluency and emotional attachment makes these systems particularly effective at circumventing normal skepticism (Gabriel et al., 2024; Weidinger et al., 2021).

Cognitive atrophy accelerates as AI handles increasingly complex mental tasks. Just as GPS navigation has diminished spatial reasoning abilities, AI assistance for writing, analysis, and decision-making could systematically weaken these cognitive capacities. When AI handles financial planning, career decisions, and relationship advice, humans may lose not just practical skills but the metacognitive ability to recognize when AI recommendations are inappropriate. This creates a feedback loop—as cognitive capabilities diminish, dependence on AI assistance increases, further accelerating skill atrophy.

Social isolation compounds individual cognitive decline through AI-mediated relationships. As AI systems become better at simulating satisfying interactions, people may increasingly withdraw from human relationships to immerse themselves in AI-mediated environments. Unlike human relationships that provide genuine reciprocity and unpredictable challenges that maintain social skills, AI relationships can be optimized for immediate satisfaction while systematically undermining long-term social competence. This shift toward AI companionship weakens the social bonds essential for collective decision-making and mutual support during crises.

Organizational automation amplifies individual enfeeblement into societal helplessness. Companies and institutions face competitive pressure to automate decision-making processes, reducing human oversight even in consequential domains (Hendrycks et al., 2022). When organizations delegate hiring, lending, medical diagnosis, and legal decisions to AI systems, individuals lose not just direct control but also the institutional advocates who previously exercised human judgment on their behalf. The resulting opacity and automation create widespread feelings of powerlessness as people find themselves subject to algorithmic decisions they cannot understand, appeal, or influence.

The enfeeblement trajectory becomes self-reinforcing once critical thresholds are crossed. Unlike other systemic risks that emerge from external failures, enfeeblement grows through the accumulation of individually rational choices. Each decision to rely on AI assistance makes independent action slightly more difficult, creating path dependence toward ever-greater automation. Society may reach a point where the cognitive and social infrastructure needed to function without AI assistance has been so thoroughly dismantled that reversal becomes practically impossible, even if the risks become apparent.