Existential Risk Observatory

In Financial Times: "According to eight people familiar with OpenAI’s testing processes, the start-up’s tests have become less thorough, with insufficient time and resources dedicated to identifying and mitigating risks, as the $300bn start-up comes under pressure to release new models quickly and retain its competitive edge." Big Tech is already willing to sacrifice safety for profits and competitive position. What happens when they build the uncontrollable AGI they're racing towards?

Treaty? Framework convention? "Soft law"? Which of these approaches is most likely to succeed in steering the development of AI away from dangerous trajectories into safer ones? That's one of the choices covered in the recent paper by the Existential Risk Observatory. The paper points out that, whichever legal approach is adopted, there are a number of prerequisites where progress needs to (and can) be made: Building Scientific Consensus, Standardisation, Auditing, Verification, and Incentivisation. Progress can also be made with agreements on: KYC requirements for providers of compute; hardware-based methods for verifying chip locations; and support for whistleblowers when they highlight incidents of harm caused by AI or shortcomings of internal testing. Seeing the obstacles to adopting a treaty at the present time, but anticipating shocks and changes of mindset ahead, the paper recommends a "conditional AI safety treaty", which states can accept as something to be adopted whenever various canary signal thresholds are reached. In summary, the paper provides a very useful survey of the ideas of different organisations and researchers, showing where they agree and where they differ, and proposing a pragmatic middle ground approach. https://lnkd.in/eK6UfNgh

Has AI passed the Turing test? We argue that it does seem so. Read here why: https://lnkd.in/eieSUGyP

Nice to see more high-quality events close to the Netherlands (and Brussels!) on large-scale AI risks!

Powerful message to xrisk funders from John Sherman, host of the For Humanity Podcast, that drew over 100 000 subscribers while warning the public for xrisk: "I ask for 10 million dollars. If not me, who? If not now, when?" We do not take a stance on who should get funded, by whom, and for how much. However, we do think funders should judge projects by the single obvious metric: How many people can your project convince that AI could make humanity go extinct, per dollar spent? And what is your track record and data to back that up? If funders would judge projects based on this metric, scale the most efficient projects, and stop funding projects that don't deliver, many more people would end up knowing that AI could kill their families. In such a world, passing safety policy becomes giantly easier. Let's saturate the only benchmark that matters: public xrisk awareness! Max Tegmark Future of Life Institute (FLI) Survival and Flourishing Corp Anthony Aguirre

We have a new paper out! 🥳🚀📜 Many think there should be an AI Safety Treaty, but what should it look like?🤔 Our paper "International Agreements on AI Safety: Review and Recommendations for a Conditional AI Safety Treaty" by Rebecca Scholefield, Samuel Martin, and Otto Barten focuses on risk thresholds, regulations, types of international agreement and five related processes: building scientific consensus, standardisation, auditing, verification and incentivisation. Based on our review, our treaty recommendations are: - Establish a compute threshold above which development should be regulated. - Require “model audits” (evaluations and red-teaming) for models above the threshold. - Require security and governance audits for developers of models above the threshold. - Impose reporting requirements and Know-Your-Customer requirements on cloud compute providers. - Verify implementation via oversight of the compute supply chain. Full paper here: https://lnkd.in/exa8bsrZ More info and discussion here: https://lnkd.in/e8kpVX9v https://lnkd.in/edGP3ijX https://lnkd.in/eSebBBnW We hope this is helpful in making the concept of an AI Safety Treaty more established, and in working towards a shared view on what such a treaty should look like!

A new timeline estimate to AGI? The graph below, from METR research, can be seen a lot today. It shows the length, measured in human-hours, of tasks AI can finish with a 50% chance of success. It's a straight line on a log chart, meaning the trend is exponential: the task length has doubled each 7 months for relatively well-defined tasks for the last 7 years, while it has doubled much faster - every 2.5 month - for the last year or so. If a task length of one month is representative for current quantitative research and we use 7 months doubling time, this trend would mean AI can do independent quantitative research four years from now - in 2029. If it turns out that the more recent trend of doubling event 70 days holds, this would happen as soon as October next year. Does that mean we have AGI by then? Not quite. METR has also done research into 'messiness' of tasks (second graph). The messier a task is, for example because it is resource-limited, not easily resettable, or involves interacting with a changing environment, the more AI struggles to get it done, especially on longer timescales (see graph on the right). Full paper here: https://lnkd.in/ew5ywpK4

A paper recommended by the most cited AI scientist in the world Yoshua Bengio, finishes its abstract with: "Finally, we briefly outline how the deployment of misaligned AGIs might irreversibly undermine human control over the world." Other interesting quotes from the paper (link below): "One salient possibility is that AGIs use the types of deception described in the previous section to convince humans that it’s safe to deploy them widely, then leverage their positions to disempower humans." "AGIs deployed as personal assistants could emotionally manipulate human users, provide biased information to them, and be delegated responsibility for increasingly important tasks and decisions (including the design and implementation of more advanced AGIs), until they’re effectively in control of large corporations or other influential organizations." "AGIs could design novel weapons that are more powerful than those under human control, gain access to facilities for manufacturing these weapons (e.g. via hacking or persuasion techniques), and deploy them to extort or attack humans." https://lnkd.in/d3YTSheC

Great work again from Future of Life Institute (FLI)'s @Anthony Aguirre! It is crucial to communicate to a general audience why AI could lead to loss of control, possibly culminating in human extinction. "Keep The Future Human" does an excellent job at that. It also contains great ideas for how to implement a pause internationally. (See Appendix B: Example implementation of a gate closure.) We're impressed again with this work!

Richard Sutton has repeatedly argued that human extinction would be the morally right thing to happen, if AIs were smarter than us. Yesterday, he won the Turing Award from ACM, Association for Computing Machinery. Why is arguing for and working towards extinction apparently fine in AI and computer science? Science's goal, finding truth, and humanity's goals, used to align very well. This meant that scientists hardly ever had to ask themselves: is the research I'm doing in humanity's interest? As long as whatever scientists were doing would further science's objectives, it would generally also further humanity's objectives. In a world where science is causing the biggest existential risks, notably AI, this is no longer universally true. Finding out how to build superhuman AI, even though scientifically fascinating, is likely not in the interest of humanity. This means scientists should reorient and ask themselves: is my research, even though I am getting scientific acclaim for it, still in humanity's interest? Science funders, such as governments, and those awarding acclaim should also ask themselves: is this science actually in humanity's interest? What AI scientists should be doing, and what society should reward them for: 1) Inform the public about AI's existential (and other) risks. 2) Do research into these risks: what exactly are the threat models and how can we mitigate them? 3) Propose helpful policy. Humanity can mitigate existential risks to a level much below today's, but only if we dare to accept the truth and update our paradigms of the past. Watch Sutton argue for extinction here: https://lnkd.in/eMwri9ny