Recent statements by the CEO of NVIDIA that he aims to "turn NVIDIA into one giant AI" were followed last week by the launch by the former chief scientist of OpenAI of a new firm aimed directly at building (safely) Superintelligence, and an announcement of the day after by the head of a $300 billion public-private fund to be fully dedicated to the same.

This should be a wake-up call for all about the state of the accelerating, reckless, winner-take-all race among states to build ever more powerful AIs without any controls against its immense risks for human safety and undemocratic concentrations of power. 

In 1946, as the world grappled with the immense risks of nuclear proliferation, Oppenheimer - together with the two top US officials and other top nuclear scientists - produced a very detailed report that formed the basis of the Baruch Plan, a formal proposal by the US to the UN for the creation of a powerful, global, democratic and federal organization to strictly control all nuclear weapons and energy research and arsenals worldwide, followed by a similar proposal by Russia. Yet, they failed to agree and get any version to pass all five veto-holding members of the UN Security Council. 

Perhaps, we can draw inspiration from what became politically possible in 1946 under dire circumstances and try again, this time for AI. Perhaps, we could seek to avoid the failure of the Baruch Plan by relying on a time-proven and much more effective, democratic and timely treaty-making process, that of the open intergovernmental constituent assembly.

Last week, the former Chief Scientist of OpenAI, Ilya Sutskever, unveiled his new venture in Palo Alto and Tel Aviv called Safe Superintelligence (SSI). SSI's mission is focused solely on developing safe superintelligence. In an insightful interview with Bloomberg, Sutskever provided further details about his company's endeavors. 

While information regarding funders and funding amounts remains confidential, SSI is strategically positioned to attract private and state investments that could rival the top five AI labs in the industry.

For many, including myself, who had hoped Sutskever would lead humanity away from a reckless winner-take-all race for ever-more-advanced AI, this announcement is unsettling. Having read his every statement over the last year, I believe him to be as well-intentioned and conscientious as ever. 

Like most AI leaders reportedly believe privately, he has grown more skeptical that suitable global coordination mechanisms can be put in place to prevent the catastrophic misuse or loss of control of next-generation AIs. Therefore, he seems to have concluded that the best he can do is to try to make them "safe" and hopefully shape their nature to benefit humanity.

What is Superintelligence?

Mainstream reporting so far shows that most people think "superintelligence" is just another undefined marketing term, but they are likely to realize soon it is not, and they and the world will reckon with what is happening.  

While the definition of Artificial General Intelligence (AGI) has always had very wide definitions, ranging from an AI that can perform many functions of an average human to one of an AI that performs all the functions of the smartest one, Superintelligence (also known as Artificial Super Intelligence, or ASI) is defined much more clearly as an AI having intelligence "far surpassing that of the brightest and most gifted human minds." 

By definition, an ASI can do the work of even the best scientists, architects and coders of AI software and hardware. This means that such AI would most likely enter in a cycle of recursive self-improvement, giving rise to an unstoppable chain of scientific innovations beyond the control of any human, called intelligence explosion or technological singularity.

Chances of Human Control of Superintelligence

By definition, ASI is likely to possess cognitive abilities that far surpass those of humans, operating at an incomprehensibly vast level of intelligence and speed. Imagine a being that can process information a million times faster than the most brilliant human mind, capable of performing complex calculations and making decisions in an instant—including improving itself autonomously at an accelerating rate. Additionally, ASI could have parallel processing capabilities, allowing it to handle multiple tasks simultaneously across the globe. 

To comprehend the magnitude of the challenge, consider trying to control a being that can learn from vast amounts of data in a matter of seconds, develop new strategies and algorithms on the fly, and communicate with others of its kind at incredible speeds. Humans are dealing with an entity that operates on a completely different level of understanding and complexity.

While it is possible that humans could develop methods to influence and guide ASI, maintaining control over such an advanced and powerful entity would be an incredibly difficult task. There is a significant risk that ASI could become autonomous and act in ways that are not aligned with human values and goals. The potential consequences of losing control could be dire, as ASI could wield immense power and make decisions that have far-reaching implications for humanity.

It is not entirely impossible for humans to control ASI. For example, a combination of humans and ever more advanced and trustworthy "narrow AIs" could enable the creation of very advanced narrow AIs and AGIs that would produce most of the positive potential benefits of ASI while mitigating appropriately the loss of control risks. The resulting AI system may be referred to by many, including Sutzkever, as safe and beneficial ASI. 

Yet, how likely it is may largely be a matter of terminology. Given how close the definition of ASI is to that of runway AI and technological singularity, we choose here to refer to a responsible safe path for AI as that of building the most capable AGI and narrow AIs that will remain within acceptable risk human-controllable and humanity-controlled.

Therefore, while it is not entirely impossible for humans to control ASI, the likelihood of successfully maintaining such control is exceedingly low. ASI's vast intellectual and processing capabilities pose a formidable challenge that humans may not be able to overcome.

Good Case Scenarios

Even if we lose control, ASI may result in a system that - regardless of whether it is conscious or sentient - would have a durable interest in preserving and assisting humanity. In such a case, such AI would reserve some global powers for itself, such as preventing anything and anyone from turning it off, controlling nuclear weapons, bioweapons research, and some advanced research to protect itself and humanity from itself. It could result in a considerable improvement in the average quality of life of humans and secure their safety for the long term. 

Bad Case Scenarios

An ASI might decide to harm or kill many, most or all humans, if its goals don't align with our safety. This could happen if it sees humans as obstacles, competitors for resources, or threats to its existence. It might also act destructively due to programming errors or because it lacks a human-like ethical sense. Another risk is that various human “bad actors,” such as crime lords, terrorist gangs, rogue states, or other-worldly multi-billionaires, would use ASI in service of their own malign purposes. Additionally, an ASI might pursue its tasks so aggressively that it doesn't consider the harmful side effects on humans. 

If ASI control succeeds, under the current global governance scenario, its control will most certainly fall into the leaders of the executive of states controlling the company or state agency that will create the ASI, or some of their political and security elites, which would likely result in immense, durable undemocratic concentration of power and wealth.

Can the technical design of ASI influence its future nature?

It is possible that the technical nature of the initial design of ASI could increase the probability that the ASI singularity will benefit humanity in some measure or even substantially.

The new company by Sutskever notably has the word "safe" in its name and not the word "aligned", which had been the main declared goal of top AI labs and scientists so far. "Aligned AI" means an AI that is aligned with the values and interests of humanity and its human users. Whereas "safe" only means something that will prevent or radically reduce the risks of hurting physically or killing large numbers of humans. Yet, Sutskever declared in the mentioned interview: "We would like it to be operating on some key values. Some of the values we were thinking about are maybe the values that have been so successful in the past few hundred years that underpin liberal democracies, like liberty, democracy and freedom". 

Is every leading company and state charging ahead towards Superintelligence?

The CEO of NVIDIA, the most resourced and funded AI firm, stated that its AIs already have an irreplaceable role in the design of their new AI software and AI chips, and that he ultimately wants to "turn NVIDIA into one giant AI." He believes that charging ahead as fast as possible is the safest course. While its chief scientist states “Uncontrollable artificial general intelligence is science fiction and not reality". 

While the heads of OpenAI, Anthropic and DeepMind rarely use the term ASI or Superintelligence but prefer using AGI, there is no sign they are avoiding ASI or limiting the use of AI to improve AI. Apple's current position is to firmly keep AI as a safe and controlled enhancement of its offerings, yet commercial necessities may lead them to follow the trend, as indicated by their recent agreement with OpenAI.

If you were the heads of the national security agencies of China or the US, with tens of billions of budget - given the race ahead and the risk of the other superpower far surpassing you in AI military capabilities - wouldn't you also charge ahead to getting most powerful AI, even at the risk of triggering a runway ASI?

The Race to Superintelligence: a race among States and alliances, and not companies

While mainstream media depicts the race of AI as primarily one among companies, it is really, at essence, a race among a handful of states structured in two groups of states hegemonically led by the US and China.

Sutskever's new company will be based in California and Israel. Sutskever was brought up in a Jewish family and grew up in Israel. While purchases by states and their security agencies of strategic assets like NVIDIA chips are not usually disclosed, Israel did so publically, with a reference to their use "outside of Israel" days before the US government increased export controls. 

Last September, Israeli Prime Minister Netanyahu stated to the UN, "For good or bad, the developments of AI will be spearheaded by a handful of nations, and my country Israel is already among them." Given Israel's scientific and cybersecurity leadership and huge access to investments, SSI could turbocharge Israel towards a global co-leadership with China and the US.

There are signs that the launch of Safe Superintelligence was perceived by states and firms as a second starting pistol, after the one in November 2023 with ChatGPT, for an all-out race, this time for Superintelligence and not AGI. The day after the announcement, the head of the $300 billion Softbank, largely funded by Middle Eastern sovereign funds, stated to the Financial Times, “This is what I was born to do, to realize ASI.”

Where is this all heading? Is there any possible responsible path to pursue?

The Case for a Baruch Plan for AI

We have yet to see the details of Sutskever’s firm’s governance structure and how isolated it is from long-term commercial pressures, especially the pressures of states,  namely the US and Israel. In fact, resisting those pressures, which could become overpowering and contribute to undoing the mad ongoing race, would require his company to join some global AI lab,  as he declared in the past, "Given these kinds of concerns it will be important that AGI is somehow built as a cooperation between multiple countries. The future is going to be good for AI regardless. would be nice if it were good for humans as well." (See for two minutes from minute 9.51 of this video).

In the crazy scheme of things, at this incredible historical juncture, Sutskever is doing the best he can, based on its expertise and inclinations, to raise our chance for a good outcome. 

Yet Robert Oppenheimer took a different direction while facing a very similar challenge as Sutskever, ever since his time as director of the Manhattan Project.

Even during the creation of the first nuclear bomb at Los Alamos, several US nuclear scientists intended to work on a much more powerful hydrogen nuclear bomb that could have secured an even more comprehensive advantage for the US in nuclear capability that could have secured its total predominance in the world and against the Russians - similar to how Sutskever is trying to build an ASI that will be safe and have our liberal democratic values prevail.

Yet Oppenheimer chose to counter such efforts since the times at Los Alamos and, in early 1946, largely wrote the Acheson-Lilienthal Report—with the Secretary of State and Head of the US Atomic Commission at the time—that served as the basis of the Baruch Plan a few months later.

The Baruch Plan was formally proposed by the US to the UN, but it was countered by a similar proposal by Russia. Such a bold proposal prescribed the creation of a new treaty organization with a global monopoly on all dangerous technologies, unconventional weapons and nuclear energy. It prescribed that all dangerous capabilities, arsenals, research, source materials, and facilities worldwide should fall under the strict control of a new International Atomic Development Authority. 

Facilities would be located equitably around the world, built by national programs, public or private, but strictly licensed, controlled and overseen by it. This would eventually extend to all existing and future weapons, including biological and chemical. It would prevent any further development of more destructive and dangerous nuclear weapons.

Its governance would mirror that of the UN Security Council - consisting of 11 states, including the five permanent UN veto-holders and six non-permanent members elected bi-annually by the General Assembly - but, crucially, this body would not be subject to the veto power of any state. 

The Baruch Plan would have amounted to nothing less than a federal and democratic world government. Negotiations went on for 1-2 years but failed to pass the veto of each of the 5 UN veto-holding members. Consequently, national security agencies were brought in to fill in. 

Perhaps Sutskever and other concerned top AI scientists, like Yoshua Bengio and Geoffrey Hinton, could follow Oppenheimer's example by writing with political scientists a sort of "Acheson-Lilienthal Report for AI" and contributing to promoting a timely, democratic, expert-led and effective treaty-making process towards a "Baruch Plan for AI".

How can we avoid the failure of the Baruch Plan?

To avoid the fate of the Baruch Plan, a treaty-making process for an International AI Development Authority could use perhaps a more effective and inclusive model - that of the open intergovernmental constituent assembly - to avoid vetoes and better distill the democratic will of states and peoples.

Instead of relying on unstructured summits, unanimous declarations and vetos, we could rely on the most successful and democratic treaty-making process of history.  

We are referring to the one that led to the US federal constitution. It was pioneered and led by two US states, which convened three more in the Annapolis Convention in 1786, setting off a constituent process that culminated with the ratification of a new US federal constitution by nine and then all thirteen US states, achieved by a simple majority after constituent assembly deliberations of over two months. 

We could and should do the same globally and for AI.  

Surely, eventually, participation by the AI superpowers and likely all five UN veto-holding members would be essential. However, each member's approval should be the end goal and not the starting point of a process. If it is, it would make any attempt impossible, as has happened to the Baruch Plan and all UN reform proposals since 1945, which are also subject to the veto. 

Unfortunately, the Security Council has much-reduced importance and authority today compared to 1946, as many of its members have violated the UN charter over the decades. For this reason, working towards global safety and security in AI initially outside of its framework could be more workable today for AI than it was for nuclear back then. 

In the case of AI, such a treaty-making model would need to be adapted to the vast disparities in power and AI power among states, taking into consideration that 3 billion citizens are illiterate or lack internet connection. 

Therefore, such an assembly would need to give more voting weight to more affluent, more populous, and powerful states until the literacy and connectivity gap is bridged within a fixed number of years. This would produce a power balance among more and less powerful states, resembling that foreseen by the Baruch Plan. 

The Harnessing AI Risk Initiative

As the Trustless Computing Association, we are facilitating such a scenario by expanding a coalition initially of NGOs and experts, and then of a critical mass of diverse states to design and jump-start such a treaty-making process - via the Harnessing AI Risk Initiative.

Through a series of summits and meetings in Geneva, we aim to arrive at an initial agreement among as few as 7 globally diverse states on the Mandate and Rules for the Election of an Open Transnational Constituent Assembly for AI and Digital Communications. All other states would then be invited to participate, with China and the US only allowed to join together.

While the AI safety goals require the participation of all or nearly all of the most powerful nations, massive economic and sovereignty benefits will be reserved for states and leading AI labs that join early on. In fact, alongside an international AI Safety Institute, the mandate of such an Assembly will include the creation of a democratically-governed public-private consortium for a Global Public Interest AI Lab to develop and jointly exploit globally-leading capabilities. 

Early participation and co-lead by states like Taiwan, Germany, the Netherlands, and now Israel - with unique strategic assets in the frontier AI supply chain - would help to achieve a sort of "mutual dependency" on the AI supply vis-a-vis superpowers, incentivizing them to participate while raising their political and economic leverage in the AI game.

Such global AI lab will pursue primarily human-controllable AI development and research but also - in the crazy scheme of things we are enmeshed - will have sizeable research program on safe and beneficial superintelligence, as the competitive landscape may need it to attempt to be the one realizing it or releasing it before others do - as we elaborated in "The Superintelligence Option" chapter of our Harnessing AI Risk Proposal v.3, already back in January 2024.

If it starts today, the costs of the treaty organization and its Global Lab are foreseen to be at least 15 billion dollars in the long term. Given the technology's proven scalability and productivity and wide availability, the Lab could be financed via the project finance model by sovereign funds and private capital, buttressed by pre-licensing and pre-commercial procurement by participating states. 

In the short term, funding will come from donations to a coalition of diverse NGOs promoting it, early pre-seed investments in such a Lab, and membership fees of early participant states.

As proof that impactful treaties can be advanced successfully by a coalition of NGOs and smaller states, consider that the Coalition for the International Criminal Court was started by the World Federalist Movement and a small state like Trinidad and Tobago. It set out a process that gathered 1600 NGOs and led to 124 signatory states.

In conclusion, the unprecedented risks and opportunities posed by AI require a skillful, urgent and coordinated global response. 

By learning from historical examples and adapting them to the current context, we can create a framework for AI governance that ensures safety, fairness, and prosperity for all.