{"id":7174,"date":"2023-03-30T01:23:41","date_gmt":"2023-03-30T06:23:41","guid":{"rendered":"https:\/\/scottaaronson.blog\/?p=7174"},"modified":"2023-05-01T08:42:28","modified_gmt":"2023-05-01T13:42:28","slug":"if-ai-scaling-is-to-be-shut-down-let-it-be-for-a-coherent-reason","status":"publish","type":"post","link":"https:\/\/scottaaronson.blog\/?p=7174","title":{"rendered":"If AI scaling is to be shut down, let it be for a coherent reason"},"content":{"rendered":"\n

There’s now an open letter<\/a> arguing that the world should impose a six-month moratorium on the further scaling of AI models such as GPT, by government fiat if necessary, to give AI safety and interpretability research a bit more time to catch up. The letter is signed by many of my friends and colleagues, many who probably agree with each other about little else, over a thousand people including Elon Musk, Steve Wozniak, Andrew Yang, Jaan Tallinn, Stuart Russell, Max Tegmark, Yuval Noah Harari, Ernie Davis, Gary Marcus, and Yoshua Bengio. <\/p>\n\n\n\n

Meanwhile, Eliezer Yudkowsky published a piece in TIME<\/em><\/a> arguing that the open letter doesn’t go nearly far enough, and that AI scaling needs to be shut down entirely until the AI alignment problem is solved—with the shutdown enforced by military strikes on GPU farms if needed, and treated as more important than preventing nuclear war.<\/p>\n\n\n\n

Readers, as they do, asked me to respond. Alright, alright. While the open letter is presumably targeted at OpenAI more than any other entity, and while I\u2019ve been spending the year at OpenAI<\/a> to work on theoretical foundations of AI safety, I\u2019m going to answer strictly for myself.<\/p>\n\n\n\n

Given the jaw-droppingly spectacular abilities of GPT-4—e.g., acing<\/a> the Advanced Placement biology and macroeconomics exams, correctly manipulating images<\/a> (via their source code) without having been programmed for anything of the kind, etc. etc.—the idea that AI now needs to be treated with extreme caution strikes me as far from absurd. I don’t even dismiss the possibility that advanced AI could eventually require the same sorts of safeguards as nuclear weapons.<\/p>\n\n\n\n

Furthermore, people might be surprised about the diversity of opinion about these issues within<\/em> OpenAI, by how many there have discussed or even forcefully advocated slowing down. And there’s a world not so far from this one where I, too, get behind a pause. For example, one actual<\/em> major human tragedy caused by a generative AI model might suffice to push me over the edge. (What would push you<\/em> over the edge, if you’re not already over?)<\/p>\n\n\n\n

Before I join the slowdown brigade, though, I have (this being the week before Passover) four questions for the signatories:<\/p>\n\n\n\n

    \n
  1. Would your rationale for this pause have applied to basically any<\/em> nascent technology \u2014 the printing press, radio, airplanes, the Internet? \u201cWe don\u2019t yet know the implications, but there\u2019s an excellent chance terrible people will misuse this, ergo the only responsible choice is to pause until we’re confident that they won’t\u201d?<\/li>\n\n\n\n
  2. Why six months? Why not six weeks or six years?<\/li>\n\n\n\n
  3. When, by your lights, would we ever know that it was safe to resume<\/em> scaling AI—or at least that the risks of pausing exceeded the risks of scaling? Why won’t the precautionary principle<\/a> continue for apply forever?<\/li>\n\n\n\n
  4. Were you, until approximately last week, ridiculing GPT as unimpressive, a stochastic parrot, lacking common sense, piffle, a scam, etc. \u2014 before turning around and declaring that it could be existentially dangerous? How can you have it both ways? If, as sometimes claimed, \u201cGPT-4 is dangerous not because it\u2019s too smart but because it\u2019s too stupid,\u201d then shouldn’t GPT-5 be smarter and therefore<\/em> safer<\/em>? Thus, shouldn’t we keep scaling AI as quickly as we can … for safety reasons<\/em>? If, on the other hand, the problem is that GPT-4 is too smart, then why can\u2019t you bring yourself to say so?<\/li>\n<\/ol>\n\n\n\n

    With the \u201cwhy six months?\u201d question, I confess that I was deeply confused, until I heard a dear friend and colleague in academic AI, one who’s long been skeptical of AI-doom scenarios, explain why he<\/em> signed the open letter. He said: look, we all started writing research papers about the safety issues with ChatGPT; then our work became obsolete when OpenAI released GPT-4 just a few months later. So now we’re writing papers about GPT-4. Will we again<\/em> have to throw our work away when OpenAI releases GPT-5? I realized that, while six months might not suffice to save human civilization, it’s just enough for the more immediate concern of getting papers into academic AI conferences.<\/p>\n\n\n\n

    Look: while I’ve spent multiple<\/a> posts<\/a> explaining how I part ways from the Orthodox Yudkowskyan position, I do find that position intellectually consistent, with conclusions that follow neatly from premises. The Orthodox, in particular, can straightforwardly answer all four of my questions above:<\/p>\n\n\n\n

      \n
    1. AI is manifestly different from any other technology humans have ever created, because it could become to us as we are to orangutans;<\/li>\n\n\n\n
    2. a six-month pause is very far from sufficient but is better than no pause;<\/li>\n\n\n\n
    3. we’ll know that it’s safe to scale when (and only when) we understand our AIs so deeply that we can mathematically explain<\/em> why they won’t do anything bad; and<\/li>\n\n\n\n
    4. GPT-4 is extremely<\/em> impressive—that’s why it’s so terrifying!<\/li>\n<\/ol>\n\n\n\n

      On the other hand, I’m deeply confused by the people who signed the open letter, even though they continue to downplay or even ridicule GPT’s abilities, as well as the “sensationalist” predictions of an AI apocalypse.<\/em> I’d feel less confused if such people came out and argued explicitly: “yes, we should also have paused the rapid improvement of printing presses to avert Europe’s religious wars. Yes, we should’ve paused the scaling of radio transmitters to prevent the rise of Hitler. Yes, we should’ve paused the race for ever-faster home Internet to prevent the election of Donald Trump. And yes, we should’ve trusted our governments to manage these pauses, to foresee brand-new technologies’ likely harms and take appropriate actions to mitigate them.”<\/p>\n\n\n\n

      Absent such an argument, I come back to the question of whether generative AI actually<\/em> poses a near-term risk that’s totally unparalleled in human history, or perhaps approximated only by the risk of nuclear weapons. After sharing an email from his partner, Eliezer rather movingly writes:<\/p>\n\n\n\n

      \n

      When the insider conversation is about the grief of seeing your daughter lose her first tooth, and thinking she\u2019s not going to get a chance to grow up, I believe we are past the point of playing political chess about a six-month moratorium.<\/p>\n<\/blockquote>\n\n\n\n

      Look, I too have a 10-year-old daughter and a 6-year-old son, and I wish to see them grow up. But the causal story that starts with a GPT-5 or GPT-4.5 training run, and ends with the sudden death of my children and of all carbon-based life, still has a few too many gaps for my aging, inadequate brain to fill in. I can complete the story in my imagination, of course, but I could equally complete a story that starts with GPT-5 and ends with the world saved from various natural stupidities. For better or worse, I lack the “Bayescraft” to see why the first story is obviously<\/em> 1000x or 1,000,000x likelier than the second one.<\/p>\n\n\n\n

      But, I dunno, maybe I’m making the greatest mistake of my life? Feel free to try convincing me that I should sign the letter. But let’s see how polite and charitable everyone can be: hopefully a six-month moratorium won’t be needed to solve the alignment problem of the Shtetl-Optimized<\/em> comment section.<\/p>\n","protected":false},"excerpt":{"rendered":"

      There’s now an open letter arguing that the world should impose a six-month moratorium on the further scaling of AI models such as GPT, by government fiat if necessary, to give AI safety and interpretability research a bit more time to catch up. The letter is signed by many of my friends and colleagues, many […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"advanced_seo_description":"","jetpack_seo_html_title":"","jetpack_seo_noindex":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"_wpas_customize_per_network":false},"categories":[8],"tags":[],"class_list":["post-7174","post","type-post","status-publish","format-standard","hentry","category-the-fate-of-humanity"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=\/wp\/v2\/posts\/7174","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=7174"}],"version-history":[{"count":7,"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=\/wp\/v2\/posts\/7174\/revisions"}],"predecessor-version":[{"id":7275,"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=\/wp\/v2\/posts\/7174\/revisions\/7275"}],"wp:attachment":[{"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=7174"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=7174"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=7174"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}