{"id":10046,"date":"2026-09-01T12:17:08","date_gmt":"2026-09-01T17:17:08","guid":{"rendered":"https:\/\/scottaaronson.blog\/?p=10046"},"modified":"2026-09-01T12:17:08","modified_gmt":"2026-09-01T17:17:08","slug":"llms-and-self-referentiality","status":"publish","type":"post","link":"https:\/\/scottaaronson.blog\/?p=10046","title":{"rendered":"LLMs and self-referentiality"},"content":{"rendered":"\n

I woke up yesterday with the following thoughts, which are probably either obvious or dumb.<\/p>\n\n\n\n

A central thesis that many readers, including me, took from Douglas Hofstadter\u2019s G\u00f6del Escher Bach<\/a> when young was that the secret of intelligence (and therefore, of AI) was going to have a lot to do with self-referentiality and \u201cstrange loops.\u201d<\/p>\n\n\n\n

Even Roger Penrose\u2019s The Emperor\u2019s New Mind<\/a>, which in some ways was the anti-GEB, ironically agreed with GEB about the fundamental importance of self-reference to the success or failure of the whole AI project. It claimed (incorrectly, in my view and in most experts\u2019) that AI could never work because there was something about G\u00f6del\u2019s Theorem and self-reference that no computer program could ever capture, but that could be captured by exotic physics accessible to the human brain.<\/p>\n\n\n\n

Now, in 2026, we\u2019ve succeeded at building AIs that outperform most humans at most intellectual tasks that are well-defined enough to judge. And at no point in the tech stack of those AIs \u2014 neither in the transformer neural nets, nor in the GPU clusters they run on, nor in the training process, nor anywhere else \u2014 did anyone need to build in anything about self-reference. (Excepting, eg, the system instructions that tell the model about its role and identity, which aren\u2019t needed for intelligent behavior. Also, I\u2019m not going to count the autoregressive nature of LLMs as \u201cself-referential\u201d; that\u2019s just dynamical feedback.)<\/p>\n\n\n\n

Of course, GPT 5.6 Pro and Fable can talk about themselves, about G\u00f6del\u2019s Theorem, about self-reference, about what we\u2019re talking about right now, all of it, better than most humans. But at no point did anyone need to build self-referential abilities in. They popped out as a byproduct of the same pretraining that let the models talk about Pok\u00e9mon and long-chain polymers and cognitive behavioral therapy and plate tectonics and everything else.<\/p>\n\n\n\n

No wonder Hofstadter says he\u2019s been stunned by the success of LLMs, and has seemed depressed about current AI capabilities in essays like this one<\/a>. He\u2019s way too smart to deny what\u2019s happened or invent reasons why it doesn\u2019t really count (the approach many have taken). But he realizes that we now have true conversational intelligence from a path that the GEB worldview would\u2019ve regarded as far too cheap and simple, and that certainly has no \u201cstrange loops\u201d built in anywhere.<\/p>\n\n\n\n

Of course, a Hofstadterian could argue that a strange loop emerges<\/em> in LLMs \u2014 indeed, nothing in GEB ever said that strange loops would need to be explicitly engineered at the outset. But would anyone who hadn\u2019t been brought up on GEB arrive at this as a useful way of thinking about LLMs?<\/p>\n\n\n\n

What can we say about this with hindsight? While the ideas of diagonalization and self-reference of course played a central role in the birth of modern mathematical logic and computer science, the most famous uses were negative<\/em>: there is not a bijectjon between the natural numbers and the reals. There is not a complete sound proof system for arithmetic. There is not an algorithm to solve the halting problem.<\/p>\n\n\n\n

If your goal was only to build the axioms of ZFC and the rules of first-order inference, or build an electronic computer, you wouldn\u2019t explicitly need self-reference for that. You would just \u2026 start building, taking care that your instruction set didn\u2019t fall short of universality.<\/p>\n\n\n\n

Yes, ZFC can formalize and prove theorems about itself. Yes, electronic computers can run programs that take their own code as input. But no one ever needed to build those abilities in, any more than self-reference needed to be built in to the alphabet or the rules of grammar. It popped out as a free byproduct of universality.<\/p>\n\n\n\n

In the same way, LLMs\u2019 ability to talk about themselves popped out as a byproduct of their ability to talk about anything in the discourse universe they were trained on. The big, old ideas about intelligence that ended up basically vindicated were the ideas about how intelligence is about prediction, and prediction is about compression, and compression is about finding better and better upper bounds on Kolmogorov complexity. Not the self-reference stuff. (Although, if you wanted to know why Kolmogorov complexity can\u2019t<\/em> be computed perfectly, that negative statement would again require a self-referential argument.)<\/p>\n\n\n\n

What\u2019s left? Consciousness and subjective experience of course remain extremely mysterious. For all we know, Hofstadter could be right that those have something to do with self-reference. (For all we know, even Penrose could be right that they have something to do with exotic physics accessible to biological brains but not digital computers!)<\/p>\n\n\n\n

But the idea that you\u2019d need explicit self-referentiality before you could get convincing and world-changing conversational intelligence? Let it be buried in a Westminster Abbey or Arlington National Cemetery for the most important wrong ideas in human history \u2014 geocentrism, Aristotle\u2019s teleological physics, aether, phlogiston, Freud\u2019s psychology, Marx\u2019s prediction of a workers\u2019 uprising followed by a classless utopia, etc. But buried it needs to be.<\/p>\n","protected":false},"excerpt":{"rendered":"

I woke up yesterday with the following thoughts, which are probably either obvious or dumb. A central thesis that many readers, including me, took from Douglas Hofstadter\u2019s G\u00f6del Escher Bach when young was that the secret of intelligence (and therefore, of AI) was going to have a lot to do with self-referentiality and \u201cstrange loops.\u201d […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"advanced_seo_description":"","jetpack_seo_html_title":"","jetpack_seo_noindex":false,"jetpack_seo_schema_type":"","_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"{title}\n\n{excerpt}\n\n{url}","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"_wpas_customize_per_network":false,"jetpack_post_was_ever_published":false},"categories":[18,12,3],"tags":[],"class_list":["post-10046","post","type-post","status-publish","format-standard","hentry","category-embarrassing-myself","category-metaphysical-spouting","category-procrastination"],"jetpack_publicize_connections":[],"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=\/wp\/v2\/posts\/10046","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=10046"}],"version-history":[{"count":1,"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=\/wp\/v2\/posts\/10046\/revisions"}],"predecessor-version":[{"id":10047,"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=\/wp\/v2\/posts\/10046\/revisions\/10047"}],"wp:attachment":[{"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=10046"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=10046"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=10046"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}