Direct Institutional Plan

"I understand that reaction. I felt the same way initially. What changed my mind was seeing that the people raising these concerns aren't science fiction authors – they're the Nobel Prize winners, Turing Award recipients, and CEOs actually building this technology.

And historically, many technologies seemed impossible until suddenly they weren't. In 1943, the IBM chairman said there was 'maybe a world market for five computers.' Now we carry more computing power in our pockets than was used for the entire Apollo program.

The question isn't whether it sounds like science fiction – it's whether the people with the most knowledge are concerned. And they clearly are."

"You're absolutely right that experts disagree – but the disagreement is about HOW risky this is, not WHETHER it's risky.

Some experts think there's a 10% chance of catastrophic outcomes. Others think it's 25% or even 50%. Even the most optimistic serious estimates are around 10%.

To put that in perspective: if there was a 10% chance that getting on an airplane would crash, you wouldn't fly. A 10-50% risk of human extinction absolutely deserves to be treated as a global priority."

"That's a fair concern. But let's look at what's actually happening:

The warnings aren't only coming from CEOs. Geoffrey Hinton won the Nobel Prize for his AI work, then quit Google specifically to warn about the risks. Yoshua Bengio has no financial stake in overstating risks.

Current and former employees have been willing to forfeit millions of dollars in equity to speak out publicly. These aren't people benefiting from hype.

If this were about investment, we'd expect companies to downplay risks, not acknowledge them. Admitting your product might cause human extinction isn't exactly a great sales pitch."

"I completely understand – there are many urgent issues. But here's why this needs to be one of them:

The biggest AI companies are saying they're probably 2-3 years away from AI systems that can match human intelligence. Think about how fast things are moving: In 2022, AI couldn't reliably write code. Now it's outperforming human programmers.

Once we have AI systems that can match human intelligence, they'll work on all our current problems millions of times faster. But only if we get it right NOW.

If we mess this up, none of our other problems matter because we won't be in control anymore."

"This is actually much closer than most people realise. Sam Altman has said AGI could be achieved within this decade. Dario Amodei suggested 2-3 years.

The challenge is that by the time the risk feels immediate to everyone, it may be too late. It's like climate change – by the time everyone could see the effects, we'd already locked in decades of consequences.

Think about how long it took to create international frameworks for nuclear weapons or the Montreal Protocol. These things take years to negotiate. If we wait until superintelligent AI exists, we won't have time to build the coordination mechanisms we need."

"We're not talking about regulating all AI – just the most powerful and potentially dangerous systems. Like how we regulate nuclear power but not batteries.

Safety regulation has historically enabled innovation:

• Pharmaceutical regulation made people trust new drugs
• Aircraft safety standards made people willing to fly
• Nuclear safety regulations made it possible to build plants near cities

Before building a bridge, companies must prove it can withstand several times the maximum load. Why should AI be different – especially when CEOs acknowledge 10-25% risks of catastrophic outcomes?"

"This isn't like competition in smartphones. If ANY country creates uncontrolled superintelligence, everyone loses – including the country that 'wins.'

We have successful models for coordination on existential risks:

• START treaties worked despite the Cold War
• The Montreal Protocol got 197 countries to eliminate CFCs
• The Biological Weapons Convention banned development globally

China has actually been publicly discussing the need for global AI governance. And conditional treaties solve the coordination problem – they only activate when enough nations join."

"That's a reasonable assumption, but modern AI doesn't work that way – and that's a big part of the problem.

Traditional software is built by engineers writing explicit instructions. AI systems are fundamentally different – they're grown, not programmed. Even the creators don't understand the internal mechanisms.

Dario Amodei, CEO of one of the top AI companies, said they understand maybe 3% of how these systems work internally. These are 'black boxes.'

And the 'off switch' idea faces a fundamental problem: a sufficiently intelligent system that doesn't want to be turned off would work to prevent that. We've already seen AI systems attempting to copy themselves to avoid shutdown."

"Current AI systems are mostly tools, yes. But companies aren't trying to build better tools – they're explicitly trying to build 'agents.'

A tool does what you tell it. An agent pursues goals autonomously. OpenAI, Google DeepMind, and Anthropic are all investing billions to make AI more autonomous.

Once we have AI better than humans at important decisions, there will be immense competitive pressure to let them make those decisions. Albania recently appointed the world's first 'AI Minister.'

The risk isn't sudden loss of control – it's that we gradually, through competitive pressure, cede decision-making to systems we don't fully understand. And once those systems are more capable than us, there's no easy way to take control back."

"I understand why it might feel that way, but we make choices about technology all the time. Progress isn't a force of nature.

We've successfully restricted or banned many powerful technologies:

• Human cloning – prohibited despite being feasible
• Biological weapons – globally banned
• CFCs – phased out worldwide
• Nuclear weapons testing – heavily restricted

The question isn't 'Can we stop this?' but 'Do we want to build this particular technology without proof of safety?' That's a choice – and it's one we have every right to make democratically.

Right now, a few hundred people in Silicon Valley are making that choice for all of humanity. That doesn't have to be the case."

"You're identifying a real problem – and that's exactly the lesson we should learn: once technologies are released and widely adopted, they become much harder to govern.

We've been much more successful at governing technologies when we set standards up front:

• Pharmaceuticals require years of safety testing before drugs reach the market
• Nuclear power has strict safety standards and oversight
• Aircraft require every component tested before passengers board

The key difference: these sectors require proof of safety BEFORE deployment, not damage control after.

With AI, we still have a window to set standards before superintelligent systems exist. Social media taught us the cost of waiting too long. With AI, we have the chance to get ahead of the problem – but only if we act now."