{"id":6524,"date":"2022-06-27T23:00:23","date_gmt":"2022-06-28T04:00:23","guid":{"rendered":"https:\/\/scottaaronson.blog\/?p=6524"},"modified":"2022-07-07T08:50:46","modified_gmt":"2022-07-07T13:50:46","slug":"steven-pinker-and-i-debate-ai-scaling","status":"publish","type":"post","link":"https:\/\/scottaaronson.blog\/?p=6524","title":{"rendered":"Steven Pinker and I debate AI scaling!"},"content":{"rendered":"\n

Before June 2022 was the month of the possible start of the Second American Civil War, it was the month of a<\/a> lively<\/a> debate<\/a> between<\/a> Scott Alexander and Gary Marcus about the scaling of large language models, such as GPT-3<\/a>.  Will GPT-n<\/em> be able to do all the intellectual work that humans do, in the limit of large n<\/em>?  If so, should we be impressed?  Terrified?  Should we dismiss these language models as mere \u201cstochastic parrots\u201d?<\/p>\n\n\n\n

I was privileged to be part of various email exchanges about those same questions with Steven Pinker, Ernest Davis, Gary Marcus, Douglas Hofstadter, and Scott Alexander.  It\u2019s fair to say that, overall, Pinker, Davis, Marcus, and Hofstadter were more impressed by GPT-3\u2019s blunders, while we Scotts were more impressed by its abilities.  (On the other hand, Hofstadter, more so than Pinker, Davis, or Marcus, said that he\u2019s terrified about how powerful GPT-like systems will become in the future.)<\/p>\n\n\n\n

Anyway, at some point Pinker produced an essay setting out his thoughts, and asked whether \u201ceither of the Scotts\u201d wanted to share it on our blogs.  Knowing an intellectual scoop when I see one, I answered that I\u2019d be honored to host Steve\u2019s essay\u2014along with my response, along with Steve\u2019s response to that<\/em>.  To my delight, Steve immediately agreed.  Enjoy!  \u2013SA<\/p>\n\n\n\n

\n\n\n\n

Steven Pinker\u2019s Initial Salvo<\/strong><\/h2>\n\n\n\n

Will future deep learning models with more parameters and trained on more examples avoid the silly blunders<\/a> which Gary Marcus and Ernie Davis entrap GPT into making, and render their criticisms obsolete?  And if they keep exposing new blunders in new models, would this just be moving the goalposts<\/a>?  Either way, what\u2019s at stake?<\/p>\n\n\n\n

It depends very much on the question.  There\u2019s the cognitive science question of whether humans think and speak the way GPT-3 and other deep-learning neural network models do.  And there\u2019s the engineering question of whether the way to develop better, humanlike AI is to upscale deep learning models (as opposed to incorporating different mechanisms, like a knowledge database and propositional reasoning).<\/p>\n\n\n\n

The questions are, to be sure, related: If a model is incapable of duplicating a human feat like language understanding, it can\u2019t be a good theory of how the human mind works.  Conversely, if a model flubs some task that humans can ace, perhaps it\u2019s because it\u2019s missing some mechanism that powers the human mind.  Still, they\u2019re not the same question: As with airplanes and other machines, an artificial system can duplicate or exceed a natural one but work in a different way.<\/p>\n\n\n\n

Apropos the scientific question, I don\u2019t see the Marcus-Davis challenges as benchmarks or long bets that they have to rest their case on.  I see them as scientific probing of an empirical hypothesis, namely whether the human language capacity works like GPT-3.  Its failures of common sense are one form of evidence that the answer is \u201cno,\u201d but there are others\u2014for example, that it needs to be trained on half a trillion words, or about 10,000 years of continuous speech, whereas human children get pretty good after 3 years.  Conversely, it needs no social and perceptual context to make sense of its training set, whereas children do (hearing children of deaf parents don\u2019t learn spoken language from radio and TV).  Another diagnostic is that baby-talk is very different from the output of a partially trained GPT.  Also, humans can generalize their language skill to express their intentions across a wide range of social and environmental contexts, whereas GPT-3 is fundamentally a text extrapolator (a task, incidentally, which humans aren\u2019t particularly good at).  There are surely other empirical probes, limited only by scientific imagination, and it doesn\u2019t make sense in science to set up a single benchmark for an empirical question once and for all.  As we learn more about a phenomenon, and as new theories compete to explain it, we need to develop more sensitive instruments and more clever empirical tests.  That\u2019s what I see Marcus and Davis as doing.<\/p>\n\n\n\n

Regarding the second, engineering question of whether scaling up deep-learning models will \u201cget us to Artificial General Intelligence\u201d: I think the question is probably ill-conceived, because I think the concept of \u201cgeneral intelligence\u201d is meaningless.  (I\u2019m not referring to the psychometric variable g<\/em>, also called \u201cgeneral intelligence,\u201d namely the principal component of correlated variation across IQ subtests.  This is  a variable that aggregates many contributors to the brain\u2019s efficiency such as cortical thickness and neural transmission speed, but it is not a mechanism (just as \u201chorsepower\u201d is a meaningful variable, but it doesn\u2019t explain how cars move.)  I find most characterizations of AGI to be either circular (such as \u201csmarter than humans in every way,\u201d begging the question of what \u201csmarter\u201d means) or mystical\u2014a kind of omniscient, omnipotent, and clairvoyant power to solve any problem.  No logician has ever outlined a normative model of what general intelligence would consist of, and even Turing swapped it out for the problem of fooling an observer, which spawned 70 years of unhelpful reminders of how easy it is to fool an observer.<\/p>\n\n\n\n

If we do try to define \u201cintelligence\u201d in terms of mechanism rather than magic, it seems to me it would be something like \u201cthe ability to use information to attain a goal in an environment.\u201d  (\u201cUse information\u201d is shorthand for performing computations that embody laws that govern the world, namely logic, cause and effect, and statistical regularities.  \u201cAttain a goal\u201d is shorthand for optimizing the attainment of multiple<\/em> goals, since different goals trade off.)  Specifying the goal is critical to any definition of intelligence: a given strategy in basketball will be intelligent if you\u2019re trying to win a game and stupid if you\u2019re trying to throw it.  So is the environment: a given strategy can be smart under NBA rules and stupid under college rules.<\/p>\n\n\n\n

Since a goal itself is neither intelligent or unintelligent (Hume and all that), but must be exogenously built into a system, and since no physical system has clairvoyance for all the laws of the world it inhabits down to the last butterfly wing-flap, this implies that there are as many intelligences as there are goals and environments.  There will be no omnipotent superintelligence or wonder algorithm (or singularity or AGI or existential threat or foom), just better and better gadgets.<\/p>\n\n\n\n

In the case of humans, natural selection has built in multiple goals\u2014comfort, pleasure, reputation, curiosity, power, status, the well-being of loved ones\u2014which may trade off, and are sometimes randomized or inverted in game-theoretic paradoxical tactics.  Not only does all this make psychology hard, but it makes human intelligence a dubious benchmark for artificial systems.  Why would anyone want <\/em>to emulate human intelligence in an artificial system (any more than a mechanical engineer would want to duplicate a human body, with all its fragility)?  Why not build the best possible autonomous vehicle, or language translator, or dishwasher-emptier, or baby-sitter, or protein-folding predictor?  And who cares whether the best autonomous vehicle driver would be, out of the box, a good baby-sitter?  Only someone who thinks that intelligence is some all-powerful elixir.<\/p>\n\n\n\n

Back to GPT-3, DALL-E, LaMDA, and other deep learning models: It seems to me that the question of whether or not they\u2019re taking us closer to \u201cArtificial General Intelligence\u201d (or, heaven help us, \u201csentience\u201d) is based not on any analysis of what AGI would consist of but on our being gobsmacked by what they can do.  But refuting our intuitions about what a massively trained, massively parameterized network is capable of (and I\u2019ll admit that they refuted mine) should not be confused with a path toward omniscience and omnipotence.  GPT-3 is unquestionably awesome at its designed-in goal of extrapolating text.  But that is not the main goal of human language competence, namely expressing and perceiving intentions.  Indeed, the program is not even set up to input or output intentions, since that would require deep thought about how to represent intentions, which went out of style in AI as the big-data\/deep-learning hammer turned every problem into a nail.  That\u2019s why no one is using GPT-3 to answer their email or write an article or legal brief (except to show how well the program can spoof one).<\/p>\n\n\n\n

So is Scott Alexander right that every scaled-up GPT-n<\/em> will avoid the blunders that Marcus and Davis show in GPT-(n<\/em>-1)?  Perhaps, though I doubt it, for reasons that Marcus and Davis explain well (in particular, that astronomical training sets at best compensate for their being crippled by the lack of a world model).  But even if they do, that would show neither that human language competence is a GPT (given the totality of the relevant evidence) nor that GPT-n<\/em> is approaching Artificial General Intelligence (whatever that is).<\/p>\n\n\n\n

\n\n\n\n

Scott Aaronson\u2019s Response<\/strong><\/h2>\n\n\n\n

As usual, I find Steve crystal-clear and precise\u2014so much so that we can quickly dispense with the many points of agreement.  Basically, one side says that, while GPT-3 is of course mind-bogglingly impressive, and while it refuted confident predictions that no such thing would work, in the end it\u2019s just a text-prediction engine that will run with any absurd premise it\u2019s given, and it fails to model the world the way humans do.  The other side says that, while GPT-3 is of course just a text-prediction engine that will run with any absurd premise it\u2019s given, and while it fails to model the world the way humans do, in the end it\u2019s mind-bogglingly impressive, and it refuted confident predictions that no such thing would work.<\/p>\n\n\n\n

All the same, I do think it\u2019s possible to identify a substantive disagreement between the distinguished baby-boom linguistic thinkers and the gen-X\/gen-Y blogging Scott A.\u2019s: namely, whether there\u2019s a coherent concept of \u201cgeneral intelligence.\u201d  Steve writes:<\/p>\n\n\n\n

No logician has ever outlined a normative model of what general intelligence would consist of, and even Turing swapped it out for the problem of fooling an observer, which spawned 70 years of unhelpful reminders of how easy it is to fool an observer.<\/p><\/blockquote>\n\n\n\n

I freely admit that I have no principled definition of \u201cgeneral intelligence,\u201d let alone of \u201csuperintelligence.\u201d  To my mind, though, there\u2019s a simple proof-of-principle that there\u2019s something<\/em> an AI could do that pretty much any of us would call \u201csuperintelligent.\u201d  Namely, it could say whatever Albert Einstein would say in a given situation, while thinking a thousand times faster.  Feed the AI all the information about physics that the historical Einstein had in 1904, for example, and it would discover special relativity in a few hours, followed by general relativity a few days later.  Give the AI a year, and it would think \u2026 well, whatever thoughts Einstein would\u2019ve thought, if he\u2019d had a millennium in peak mental condition to think them.<\/p>\n\n\n\n

If nothing else, this AI could work by simulating Einstein\u2019s brain neuron-by-neuron\u2014provided we believe in the computational theory of mind, as I\u2019m assuming we do.  It\u2019s true that we don\u2019t know<\/em> the detailed structure of Einstein\u2019s brain in order to simulate it (we might have, had the pathologist who took it<\/a> from the hospital used cold rather than warm formaldehyde).  But that\u2019s irrelevant to the argument.  It\u2019s also true that the AI won\u2019t experience the same environment that Einstein would have\u2014so, alright, imagine putting it in a very comfortable simulated study, and letting it interact with the world\u2019s flesh-based physicists.  A-Einstein can even propose experiments for the human physicists to do\u2014he\u2019ll just have to wait an excruciatingly<\/em> long subjective time for their answers.  But that\u2019s OK: as an AI, he never gets old.<\/p>\n\n\n\n

Next let\u2019s throw into the mix AI Von Neumann, AI Ramanujan, AI Jane Austen, even AI Steven Pinker\u2014all, of course, sped up 1,000x compared to their meat versions, even able to interact with thousands of sped-up copies of themselves and other scientists and artists.  Do we agree that these entities quickly become the predominant intellectual force on earth\u2014to the point where there\u2019s little for the original humans left to do but understand and implement the AIs\u2019 outputs (and, of course, eat, drink, and enjoy their lives, assuming the AIs can\u2019t or don\u2019t want to prevent that)?  If so, then that seems to suffice to call the AIs \u201csuperintelligences.\u201d  Yes, of course<\/em> they\u2019re still limited in their ability to manipulate the physical world.  Yes, of course<\/em> they still don\u2019t optimize arbitrary goals.  All the same, these AIs have effects on the real world consistent with the sudden appearance of beings able to run intellectual rings around humans\u2014not exactly<\/em> as we do around chimpanzees, but not exactly unlike it either.<\/p>\n\n\n\n

I should clarify that, in practice, I don\u2019t expect AGI to work by slavishly emulating humans\u2014and not only because of the practical difficulties of scanning brains, especially deceased ones.  Like with airplanes, like with existing deep learning, I expect future AIs to take some inspiration from the natural world but also to depart from it whenever convenient.  The point is that, since there\u2019s something<\/em> that would plainly count as \u201csuperintelligence,\u201d the question of whether it can be achieved is therefore \u201cmerely\u201d an engineering question, not a philosophical one.<\/p>\n\n\n\n

Obviously I don\u2019t know the answer to the engineering question: no one does!  One could consistently hold that, while the thing I described would clearly count as \u201csuperintelligence,\u201d it\u2019s just an amusing fantasy, unlikely to be achieved for millennia if ever.  One could hold that all the progress in AI so far, including the scaling of language models, has taken us only 0% or perhaps 0.00001% of the way toward superintelligence so defined.<\/p>\n\n\n\n

So let me make two comments about the engineering question.  The first is that there\u2019s good news here, at least epistemically: unlike with the philosophical questions, we\u2019re virtually guaranteed more clarity over time!  Indeed, we\u2019ll know vastly more just by the end of this decade, as the large language models are further scaled and tweaked, and we find out whether they develop effective representations of the outside world and of themselves, the ability to reject absurd premises and avoid self-contradiction, or even the ability to generate original mathematical proofs and scientific hypotheses.  Of course, Gary Marcus and Scott Alexander have already placed concrete bets on the table for what sorts of things will be possible by 2030.  For all their differences in rhetoric, I was struck that their actual<\/a> probabilities<\/a> differed much more modestly.<\/p>\n\n\n\n

So then what explains the glaring differences in rhetoric?  This brings me to my second comment: whenever there\u2019s a new, rapidly-growing, poorly-understood phenomenon, whether it\u2019s the Internet or AI or COVID, there are two wildly different modes of responding to it, which we might call \u201cFebruary 2020 mode\u201d and \u201cMarch 2020 mode.\u201d  In February 2020 mode, one says: yes, a na\u00efve extrapolation might lead someone to the conclusion that this new thing is going to expand exponentially and conquer the world, dramatically changing almost every other domain\u2014but precisely because<\/em> that conclusion seems absurd on its face, it\u2019s our responsibility as serious intellectuals to articulate what\u2019s wrong with the arguments that lead to it.  In March 2020 mode, one says: holy crap, the na\u00efve extrapolation seems right!  Prepare!!  Why didn\u2019t we start earlier?<\/p>\n\n\n\n

Often, to be sure, February 2020 mode is the better mode, at least for outsiders\u2014as with the Y2K bug, or the many disease outbreaks that fizzle.  My point here is simply that February 2020 mode and March 2020 mode differ by only a month.  Sometimes hearing a single argument, seeing a single example, is enough to trigger an epistemic cascade, causing all the same facts to be seen in a new light.  As a result, reasonable people might find themselves on opposite sides of the chasm even if they started just a few steps from each other.<\/p>\n\n\n\n

As for me?  Well, I\u2019m currently trying to hold the line around February 26, 2020.  Suspending my day job in the humdrum, pedestrian field of quantum computing, I\u2019ve decided to spend a year at OpenAI, thinking about the theoretical foundations of AI safety.  But for now, only a year.<\/p>\n\n\n\n

\n\n\n\n

Steven Pinker\u2019s Response to Scott<\/strong><\/h2>\n\n\n\n

Thanks, Scott, for your thoughtful and good-natured reply, and for offering me the opportunity to respond  in Shtetl-Optimized, <\/em>one of my favorite blogs. Despite the areas of agreement, I still think that discussions of AI and its role in human affairs\u2014including AI safety\u2014will be muddled as long as the writers treat intelligence as an undefined superpower rather than a mechanisms with a makeup that determines what it can and can\u2019t do. We won\u2019t get clarity on AI if we treat the \u201cI\u201d as \u201cwhatever fools us,\u201d or \u201cwhatever amazes us,\u201d or \u201cwhatever IQ tests measure,\u201d or \u201cwhatever we have more of than animals do,\u201d or \u201cwhatever Einstein has more of than we do\u201d\u2014and then start to worry about a superintelligence that has much, much more of whatever that is.<\/p>\n\n\n\n

Take Einstein sped up a thousandfold. To begin with, current AI is not even taking us in that direction. As you note, no one is reverse-engineering his connectome, and current AI does not think the way Einstein thought, namely by visualizing physical scenarios and manipulating mathematical equations. Its current pathway would be to train a neural network with billions of physics problems and their solutions and hope that it would soak up the statistical patterns.<\/p>\n\n\n\n

Of course, the reason you pointed to a sped-up Einstein was to procrastinate having to define \u201csuperintelligence.\u201d But if intelligence is a collection of mechanisms rather than a quantity that Einstein was blessed with a lot of, it\u2019s not clear that just speeding him up would capture what anyone would call superintelligence. After all, in many areas Einstein was no Einstein. You above all could speak of his not-so-superintelligence in quantum physics, and when it came world affairs, in the early 1950s he offered the not exactly prescient or practicable prescription, \u201cOnly the creation of a world government can prevent the impending self-destruction of mankind.\u201d So it\u2019s not clear that we would <\/em>call a system that could dispense such pronouncements in seconds rather than years \u201csuperintelligent.\u201d Nor with speeding up other geniuses, say, an AI Bertrand Russell, who would need just nanoseconds to offer his own solution for world peace: the Soviet Union would be given an ultimatum that unless it immediately submitted to world government, the US (which at the time had a nuclear monopoly) would bomb it with nuclear weapons.<\/p>\n\n\n\n

My point isn\u2019t to poke retrospective fun at brilliant men, but to reiterate that brilliance itself is not some uncanny across-the-board power that can be \u201cscaled\u201d by speeding it up or otherwise; it\u2019s an engineered system that does particular things in particular ways. Only with a criterion <\/em>for intelligence can we say which of these counts as intelligent.<\/p>\n\n\n\n

Now, it\u2019s true that raw speed makes new kinds of computation possible, and I feel silly writing this to you of all people, but speeding a process up by a constant factor is of limited use with problems that are exponential, as the space of possible scientific theories, relative to their complexity, must be. Speeding up a search in the space of theories a thousandfold would be a rounding error in the time it took to find a correct one. Scientific progress depends on the search exploring the infinitesimal fraction of the space in which the true theories are likely to lie, and this depends on the quality of the intelligence, not just its raw speed.<\/p>\n\n\n\n

And it depends as well on a phenomenon you note, namely that scientific progress depends on empirical discovery, not deduction from a silicon armchair. The particle accelerators and space probes and wet labs and clinical trials still have to be implemented, with data accumulating at a rate set by the world. Strokes of genius can surely speed up the rate of discovery, but in the absence of omniscience about every particle, the time scale will still be capped by empirical reality. And this in turn directs the search for viable theories: which part of the space one should explore is guided by the current state of scientific knowledge, which depends on the tempo of discovery. Speeding up scientists a thousandfold would not speed up science a thousandfold.<\/p>\n\n\n\n

All this is relevant to AI safety. I\u2019m all for safety, but I worry that the dazzling intellectual capital being invested in the topic will not make us any safer if it begins with a woolly conception of intelligence as a kind of wonder stuff that you can have in different amounts. It leads to unhelpful analogies, like \u201cexponential increase in the number of infectious people during a pandemic\u201d \u2248 \u201cexponential increase in intelligence in AI systems.\u201d It encourages other questionable extrapolations from the human case, such as imagining that an intelligent tool will develop an alpha-male lust for domination. Worst of all, it may encourage misconceptions of AI risk itself, particularly the standard scenario in which a hypothetical future AGI is given some preposterously generic single goal such as \u201ccure cancer\u201d or \u201cmake people happy\u201d and theorists fret about the hilarious collateral damage that would ensue.<\/p>\n\n\n\n

If intelligence is a mechanism rather than a superpower, the real dangers of AI come into sharper focus. An AI system designed to replace workers may cause mass unemployment; a system designed to use data to sort people may sort them in ways we find invidious; a system designed to fool people may be exploited to fool them in nefarious ways; and as many other hazards as there are AI systems. These dangers are not conjectural, and I suspect each will have to be mitigated by a different combination of policies and patches, just like other safety challenges such as falls, fires, and drownings. I\u2019m curious whether, once intelligence is precisely characterized, any abstract theoretical foundations of AI safety will be useful in dealing with the actual AI dangers that will confront us.<\/p>\n","protected":false},"excerpt":{"rendered":"

Before June 2022 was the month of the possible start of the Second American Civil War, it was the month of a lively debate between Scott Alexander and Gary Marcus about the scaling of large language models, such as GPT-3.  Will GPT-n be able to do all the intellectual work that humans do, in the […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"advanced_seo_description":"","jetpack_seo_html_title":"","jetpack_seo_noindex":false,"jetpack_seo_schema_type":"","_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"{title}\n\n{excerpt}\n\n{url}","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"_wpas_customize_per_network":false,"jetpack_post_was_ever_published":false},"categories":[11,8],"tags":[],"class_list":["post-6524","post","type-post","status-publish","format-standard","hentry","category-nerd-interest","category-the-fate-of-humanity"],"jetpack_publicize_connections":[],"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=\/wp\/v2\/posts\/6524","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=6524"}],"version-history":[{"count":4,"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=\/wp\/v2\/posts\/6524\/revisions"}],"predecessor-version":[{"id":6529,"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=\/wp\/v2\/posts\/6524\/revisions\/6529"}],"wp:attachment":[{"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=6524"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=6524"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=6524"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}