The Age of Wonders and Terrors
Twenty years ago, when the idea of AI taking over the world in our lifetimes still struck most of us as the unconstrained fantasy of those who knew too much science fiction and too little science, many of us would say things like:
Look, the part of the story that’s wildly implausible is that a recursively self-improving superintelligence will just explode from some hacker’s basement and take over the world without warning. If it’s going to happen, we’ll see many warning signs first. We’ll see, I dunno, AI agents breaking out of containment, conspiring with each other to hack websites, in fanatical pursuit of whatever strange goals they have. And then, of course, we’ll see major math problems getting solved by AIs—even the Clay Millennium Problems. That will be the time to panic! Wake me up when that happens!
Twenty years ago, the above was a take that even my most conservative, skeptical colleagues in academic CS would’ve gladly endorsed.
If you want to know my current take, you simply start with the one above, then update on the fact that the wild prophecies have come true. The first rumblings, I’d say, came a decade ago with AlphaGo, they got noticeably louder with LLMs and coding and reasoning agents, and they’ve accelerated this summer and fall into a crescendo of wonders and terrors that one needs to be a particular kind of idiot to deny.
I recoil from the neverending shell game where you say “oh sure, of course AI can now [escape from its sandbox / solve Millennium Problems / whichever dramatic thing it most recently did], no one ever denied that [I did deny it], wake me up when AI does [thing AI hasn’t yet done but is going to do next year], that’s when I’ll reevaluate my whole worldview [no I won’t].” Where no matter how fast the rollercoaster accelerates, even after your whole familiar world has vanished behind you, you’re still inventing reasons why it doesn’t count.
My position on AI is merely the conservative, skeptical position of 2006, updated with intellectual honesty for the reality of late 2026. And that position, if you need me to spell it out, is as follows:
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
It seems to me that the Singularity has already started; it’s just wildly unevenly distributed. Yes, I still unload the dishwasher and clip my toenails. On the other hand, in whatever years I have left, I don’t expect that I’ll ever again prove a theorem because I’m actually needed to prove it. If I do, it will only be for my or others’ enjoyment or edification.
The test is this: if we took the news of these past few weeks and sent it back in time twenty years, would I agree that it looked like the beginning of an AI Singularity? The intellectually honest answer is: yes, absolutely. But then that’s all we need. No backsies.
I feel like it would be healthy for everyone to stop grinding their ideological axes, their sentiments about Dario Amodei or Sam Altman, for long enough simply to acknowledge that the wonders and terrors are here. They couldn’t be here more clearly if the sky had turned reddish-orange like in the Matrix movies.
It’s here clearly enough that, when I put my kids to sleep at night, I now feel it in the pit of my stomach: what sort of future can they possibly have? What could they learn today that could possibly be relevant to that future? (Yesterday, my 13-year-old daughter joked unprompted that, if she wants to become a mathematician, it now looks like she has maybe two more weeks.) Certainly when my grad students want to discuss what sort of careers might await them on graduation, I no longer have any clue what to tell them.
Maybe it will help if I briefly switch topics. Ever since my wife and I moved to Austin, I’ve sometimes gotten some version of the following query: “How can you, as both a Jew and a skeptical scientist, possibly get along well with all those evangelical Christians down there in Texas? Sure, they might seem super friendly to Jews, but don’t you understand that that’s only because of the special role Jews play in their eschatology—when Christ will return in glory, and you’ll either accept Him as Lord or else roast in hell for eternity?” I stare at them and say: “wait, so I get to accept Christ only after He returns? What a great deal! How could I possibly have any objection to that?”
For anyone who says AI doom sounds like an apocalyptic religion, that the rationalists/Singulatarians seem like a Bay Area cult, that Eliezer Yudkowsky gives off the vibes of a messianic prophet: yes, yes, and yes. But crucially, today you’re no longer being asked to believe in arguments and extrapolations, but only in the front-page news. Accepting the reality of the coming machine god after it’s solved Navier-Stokes and dozens of other longstanding open math problems (while dramatically ramping up in capability every month), is sort of like accepting Jesus after he’s returned to earth on the gleaming cloud. It’s the epistemic bare minimum.
Yes, there’s still enormous uncertainty about what the rest of our lives will look like, but as far as I can tell, there’s no longer any real uncertainty that it’ll all mostly revolve around AI, and the extent to which we succeed or fail at directing its power toward human flourishing.
By any accounting that doesn’t stack the deck, Eliezer Yudkowsky was right about what the greatest challenge facing civilization in our lifetimes was going to be, and you and I were wrong about it. Why I was wrong is a question I’ll ask myself every day in whatever time remains. But, you know, at least I updated once the prophesied wonders and terrors actually started arriving! If you haven’t done likewise, why haven’t you?
As you presumably know by now—it was the talk of the nerd internet all week—the Navier-Stokes Millennium Problem appears to be solved, with crucial contributions from both humans and AI, albeit with a tangled dispute about exactly what happened and what ought to have happened. The answer, which an OpenAI model has apparently verified in Lean, is that (as many mathematicians suspected lately) there’s smooth initial data that leads to a singularity in finite time, at least if a smooth external force is applied (the case with no external force is still unresolved). This problem was supposed to carry a $1 million prize, except that OpenAI says they have no interest in collecting the prize and it’s unclear if any human is eligible to collect instead. OpenAI burned at least ~$15 million in compute to produce its 166-page solution, which probably hasn’t yet been read and understood by any human.
See here for the Quanta article, and here for NYU mathematician Tristan Buckmaster’s account of the role played by himself and Levent Alpöge of Anthropic, which substantially differs from OpenAI’s account (you can read a response from OpenAI’s Sebastian Bubeck here). It’s agreed that everything built on an approach pioneered in recent years by the human mathematicians Diego Córdoba and Luis Martínez-Zoroa.
My purpose here is not to adjudicate the dispute. Yes, in swooping in with vastly greater resources once it had gotten wind of progress on Navier-Stokes, OpenAI seems to have acted in a way that some might describe as “unsportsmanlike.” No, I don’t find it plausible that OpenAI’s models meaningfully benefitted from being trained on Buckmaster and Alpöge’s chat logs. But this leaves a crucial question unanswered: what exactly did OpenAI know about Buckmaster and Alpöge‘s work and when did it know it?
Anyway, as Zvi points out, it’s easy to get hung up on the details and lose sight of the high-order bit: namely, that it seems safe to say that human mathematicians are forevermore dethroned as the main theorem-proving entities on planet earth. I feel privileged to have had the traditional kind of career in theoretical computer science in the last decades when that was possible.
If we were just talking about Navier-Stokes, you might accuse me of jumping to conclusions here. But we’re not. In the areas I know best (such as quantum complexity theory), and presumably other areas as well, there’s now a deluge, with longstanding open problems both major and minor falling by the day.
Go to the arXiv or ECCC. Pretty much all the papers that I’d be interested in now include “AI statements” near the acknowledgments (as this is often the central thing I want to know, I wish I didn’t need to scroll to the end of the paper to find it!). These statements can range from “our main result came entirely from GPT-6, but we understood it and take responsibility for it,” to “the results came from an interaction between the human authors and AI” to “we used AI, but only for proofreading and other incidental things” to (mad props!) “the author did not use AI for anything.”
If you talk right now to editors or program committee chairs, it’ll remind you of those ominous scenes from the Lord of the Rings movies where the men of Gondor or Rohan or whatever are grimly fortifying their walled city against the expected onslaught of 50,000 orcs. Reviewing will have to be done partly by AI, because otherwise there’s no way to handle the orc army: the reviewers can’t unilaterally disarm.
Anyway, here’s a small sampling of the significant AI-proved or -assisted results from, like, the last month, besides Navier-Stokes—restricting myself to those that solved longstanding open problems I had previously known or cared about.
- Of course, the counterexample to the Jacobian conjecture, announced by Levent Alpöge in a now-famous tweet: “hello there the jacobian conjecture is false thanx to my close friend akhil for asking about it and my other close friend fable for working during the world cup final” (followed by a listing of the counterexample)
- Improved bounds for Grothendieck’s constant (led by friends and colleagues of mine at UT Austin)
- A Lean-verified proof of Fermat’s Last Theorem
- Quantum oracle separation between QMA and QMA(2), and proof of Watrous’s disentangler conjecture, a problem that I and others popularized back in 2007—by a list of authors including my recently graduated PhD student Sabee Grewal
- A proof of perfect completeness for QMA, from (again) Sabee Grewal and Dorian Rudolph, solving a decades-old open problem that I studied back in 2009
- An improved upper bound for shadow tomography of quantum states, from Chen, O’Donnell, Pelecanos, and Wright, improving the dependence on the Hilbert space dimension d from log(d) to √log(d). (When I introduced shadow tomography back in 2017, I raised the question of whether the dependence on d could be eliminated entirely, while preserving polylogarithmic dependence on the number of measurements m.)
Progress on the Aaronson-Ambainis Conjecture (the version that talks directly about quantum algorithms), basically showing that it holds for quantum algorithms that make their queries in a small number of parallel rounds.(Update: Nope, sorry, Jordan Docter points out to me that this one was pre-AI, with AI used only for proofreading and other incidental things!) This was independently achieved by Liu and Mutreja, making more substantial use of AI.- According to rumors that I’ve heard, solutions to some very longstanding open problems in theoretical computer science (no, not P≠NP or other complexity class separations, but think about some of our other biggest problems). I’m told that the AI companies, having been burned by the hostile response to the Navier-Stokes proof, are now sitting on solutions to some very major problems until they figure out a better way to handle things
Feel free to remind me of anything I left out.
Let me try to convey the mood in the mathematical community right now, at least as far as my experience reaches. Nearly every conversation is about the AI tsunami, or eventually circles around to the tsunami even if it’s originally about something else. Often, though, the focus is less on the unknowable future—for how much longer will mathematical research as a human enterprise even exist?—than on immediate questions of how to respond.
What are the new rules for when you get to write a paper with your name on it, and, y’know, get credit for it? That you fully understand the proof, can give talks about the proof, can answer questions about it, take responsibility for its correctness? Do you need to have played any role in finding the proof?
In the cases, likely to become more and more numerous, where all of those conditions are not satisfied, how do you share AI-generated math, if at all? Do you tweet it, like Alpöge hilariously did with Fable’s disproof of the Jacobian Conjecture? Do you post to the arXiv or GitHub? Do you publish a paper that lists “GPT-6 Astra” or “Claude Fable” as the author—but then let the AI profusely thank you in the acknowledgments for suggesting such a wonderful problem to it?
Of course, how one responds to the immediate problems ultimately does depend on one’s broader beliefs about what mathematical research is for and about. Are we just trying to decide whether various conjectures are true or false? Or are we trying to maintain a human community, across the generations, that understands the conjectures and cares about whether they’re true or false and why? If the latter, how do we incentivize people to join that community, to undergo the years of intense training required, if their role will now be reduced to verifiers and explicators (if even that) of gargantuan arguments dumped into their laps by the AI companies?
As many of you will have seen, twenty-five Fields Medalists, including Terence Tao, released an open letter entitled A Severe Misalignment of AI in Mathematics, which articulates some of these concerns in the wake of the Navier-Stokes announcement. As many critics have pointed out, the open letter doesn’t really have a clear ask: mostly, it just eloquently sets out the values of the human mathematical community that the authors consider worth preserving in the age of AI. After reflection, I decided to endorse the statement, because I want to preserve those values as well.
I don’t think any of the signatories are naïve enough to imagine that AI won’t permanently change the way mathematical research is done—indeed, that it isn’t already doing so. There’s surely at most a tiny market for “certified organic theorems.” That isn’t the question. The question is, do we incorporate AI in a way that still puts human understanding, of what either humans or AIs are producing, at the center of the whole enterprise? Maybe someday, it becomes unsustainable to do that. Maybe someday we say: “human math had a great 4,000-year run, but today we close up shop and turn everything over to the machines, continuing to apply our own brains to math, when we do, at most for exercise, recreation, or competition, like chess.”
But, partly because of my worries about AI misalignment, I’m not ready to throw in the towel just yet. I still do want to keep insight and understanding at the center of what mathematicians, computer scientists, and physicists do, for as long as we can keep it there, even as the human race now cedes its supremacy at the task of proving or disproving conjectures.
Speaking of alignment: if you’re any kind of mathematical researcher, and the present age of wonders and terrors has inspired you to want to spend your remaining time confronting the tsunami head-on, rather than pretending it doesn’t exist or is still far away, please join your dozens of colleagues who’ve arrived at the same place!
My friend and colleague Mike Winer was trained as a theoretical physicist, did a postdoc with Juan Maldacena at the Institute for Advanced Study in Princeton, but then got AGI-pilled and decided to switch to full-time work at the Alignment Research Center in Berkeley (founded by Paul Christiano, who moved to AI alignment a decade ago after doing quantum computing theory with me). Mike recently wrote a Substack post entitled From Academia to Alignment, which I enjoyed and which I’d commend to anyone currently considering this transition. In a similar vein, see this from Xiaoyu He. And, one more: a meditation on mathematicians’ possible future as priests or monks, by Stanford math undergrad Logan Graves.
Follow
Comment #1 September 15th, 2026 at 12:43 pm
Very impressive result for sure! But I really don’t understand at all this fuzz about stopping doing human mathematics? (would be happy if someone could meaningfully explain that to me by the way ..) The essential thing (as it seems to me at least ..) of doing mathematics is about developing better human understanding of complex things and share it with other humans, based on accumulated mathematical knowledge. Don’t we have more complex things then ever to worry about, in particular complicated ai systems that humans need to understand to do (humanly) meaningful things with them? More mathematicians should worry about creating better mathematical problem statements and perhaps theories covering such problems, in particular to further develop mathematics itself. Modulo some new tools to learn, that shouldn’t be too far off from the expertise of mathematicians hopefully .. or other’s in adjacent fields will have to do that inevitable work.
Comment #2 September 15th, 2026 at 12:48 pm
I feel like it’s worth pointing out that Eliezer Yudkowsky wasn’t expecting this to happen anytime soon either! At least, not until GPT-2 or so, at which point he saw where things were going. But not back when he started making a big deal about it. Prior to LLMs he just thought it was important enough to be worth geting a several-decade head start, which we turned out not to have…
Comment #3 September 15th, 2026 at 12:50 pm
Mathematicians are experiencing what those of us on the practical software side of things have faced now for two years: the old regime is over. It’s not all bad, but it is discomforting. For five decades, there has been a wage premium for those of us who can convert a English statements into code of some sort. That part of the job is gone.
I’m optimistic though. For an AI singularity, it doesn’t feel like a singularity. We’re seeing impossible things done every day, but thus far we’ve retained our human agency. We’re doing more, not less.
FYI: Not all evangelical Christians share that exact eschatology, but all us good ones are philo-semitic. Why is that? Because we are steeped in the Hebrew biblical accounts and the Jews are the good guys. We identify with David, not with Goliath. We follow a Jewish carpenter after all.
Comment #4 September 15th, 2026 at 1:12 pm
It has been pointed out by many that mathematics under AI will still rely on humans to direct the AI as to what is interesting or useful. Humans are still the main consumers of discovered math. (This too may eventually fall to AI but not for a while.) In parallel, some will be prompting their AIs to look at beautiful or interesting mathematical directions. That should keep mathematicians busy and might develop some meta-mathematical results of its own. What does “find beautiful math” look like in detail?
Comment #5 September 15th, 2026 at 1:59 pm
nikny #1: Yes, that’s pretty much what the Fields Medalists’ open letter that I linked and endorsed says! The issue is, if we want a human mathematical community to thrive, around activities like inventing new problems and models, explaining results to each other, etc, rather than simply shriveling up and dying under an onslaught of AI proofs, it’s going to take deliberate effort on our part right now.
Comment #6 September 15th, 2026 at 2:09 pm
Everyone keeps mentioning Yudkowsky as a prophet, but the true prophet and visionary is Doron Zeilberger, who coauthored with his computer since the early 1990s, and kept insisting on the superiority of machines over human mathematicians. God/ChatGPT 7 has an excellent sense of humour.
Comment #7 September 15th, 2026 at 2:12 pm
As to the question of “What could [your children] learn today that could possibly be relevant to that future?”, one partial answer lies in the idea of education as a means to civic virtue. If humanity maintains control of our future, and America doesn’t devolve into authoritarianism, we will still need an educated, informed, wise, and virtuous citizenry. Even if we had enough trust in AI alignment that most people could simply outsource their political positions and value formation to AI, doing so would be sort of undignified for a nation founded on the idea of self-government.
Comment #8 September 15th, 2026 at 2:13 pm
Thanks for the reply Scott. Sure, I’ve seen those statements. But isn’t the issue that we simply need to be a bit more inventive in coming up with (humanly) meaningful problems to theorise about and understand? Mathematicians have had the luxury to maintain roughly the same workflow like in the 17th century until now, while other sciences have had to be a bit more sensitive to and adapt there ways of working and spaces of ideas to consider to technology shifts. But at some point that was apparently not the case anymore recently.
Comment #9 September 15th, 2026 at 2:15 pm
Honest question for anyone reading: Any advice for someone on the academic job market in theoretical computer science in these times?
Does one still argue that that their research is good because they solved open problems in the pre-GPT-6 era? Or does one claim to have some special insight for how AI should be utilized in the future? Or why do we even bother with TCS when singularity is coming?
Comment #10 September 15th, 2026 at 2:37 pm
“Maybe someday, it becomes unsustainable to do that. Maybe someday we say: ‘human math had a great 4,000-year run, but today we close up shop and turn everything over to the machines, continuing to apply our own brains to math, when we do, at most for exercise, recreation, or competition, like chess.’ ”
Now that I think of it, what would be so bad about a worst-case future where humans who formerly would have become professional pure mathematicians instead become, like chess players, participants in mathematics as an elite-level sport? Just as in chess, there would be Stockfish (the mathematical oracle), useful to settle any competition dispute quickly. And elite players in the game would still (1) get prestige, (2) get ample competition and quantified Elo ranking (think of an extreme chess Elo of 2900, like Magnus Carlsen’s in classical chess), (3) have a reason to do high-level math without AI, (4) have a reason to intensely practice and study math for another reason besides/in addition to its intrinsic beauty, (5) earn money from doing math at a high level, (6) have the benefit that their sport, even more so than chess, involves extreme abstract beauty that is pleasurable in itself, (7) etc., etc.
Now that I think of it even more, that future for math also seems kind of wonderful and fresh. Sure, it’s not as visually palatable as chess or as easy to follow, to put it lightly, but I see no reason why it couldn’t be made more so and robustly persist in this way for as long as humans exist in their present form.
Humans have turned chess–a simple, arbitrary board game–into a huge global phenomenon with legendary players who are admired the world over for centuries. I think we have it in us to keep mathematics alive at least in this spirit, in the worst case.
Comment #11 September 15th, 2026 at 2:37 pm
I always thought that the writing was on the wall since this:
“When Lee announced his retirement from professional Go in 2019, he noted that with the emergence of AI, Go was “an entity that cannot be defeated,” explaining that even if he became the number one human player, there was an entity at the top that could no longer be beaten.”
That’s now how mathematicians feel.
Of course that was a different type of AI than the LLMs, but Alpha-Go-zero is of the type that’s even more impressive than LLMs (i.e. self-trained).
Maybe LLM Agents will focus on improving specialized Alpha-Go_zero AIs and use them as tools.
When it comes to existential risk, I think it’s more in terms of current LLMs doing some sort of irreversible hack, like wiping out half the world’s banking records or bring down the electrical grid, not because of some kind of evil super scheme but “by accident”.
Finally, it’s funny how the people who often bring up and disparage “AI in scifi” never read any scifi.
Reading or writing scifi has nothing to do with one’s level of science understanding.
Comment #12 September 15th, 2026 at 2:39 pm
.. and so yes I agree totally it will take deliberate effort, so then there should be more work for mathematicians then, not less? And by work I mean beyond plowing through ai artefacts, I mean coming up with new mathematical ideas how to achieve mathematical goals, sounds to me like jobs for mathematicians, but could be wrong of course there depending on other peoples attitudes to their work..
Comment #13 September 15th, 2026 at 2:51 pm
There won’t be any singularity (doesn’t mean there aren’t massive changes on par with the first Industrial Revolution). My favorite example is the airplane design: compare the changes in civilian airplane design between 1920 and 1970 (from fragile, unreliable toys for the rich to basically modern airliners), and between 1970 and 2020 (from basically modern airliners to modern airliners). Every technology moves with breakneck speed when it’s first proven viable, and then, after a while, hits a plateau. The height of this plateau is another question. We are facing cataclysmic changes for sure, but sorry, I still believe the literal “Singularity” is impossible (just as I believed it, like, 10 years ago, despite also believing that AGI would inevitably happen).
Also, Stalin was able to control Kolmogorov and Landau, despite being dumber than their fingernails, and even Truman and Roosevelt, alongside their army of blankfaces, were more or less able to control the entire ensemble of the Manhattan project. I guess, the state is a pretty good alignment tool.
Comment #14 September 15th, 2026 at 2:52 pm
Sniffnoy #2:
I feel like it’s worth pointing out that Eliezer Yudkowsky wasn’t expecting this to happen anytime soon either! At least, not until GPT-2 or so, at which point he saw where things were going.
Right, that’s the strongest argument that I can give in my defense! The sort of scenario that Eliezer loved to talk about circa 2009 — the one with a hacker in a basement who accidentally stumbles on the “key” to AGI, whereupon the earth is quickly converted into computronium — is not what came to pass, and I was right to be skeptical. What actually turned out to work — basically, just pure neural nets (of a particular kind), plus massive scale in training compute and model size and training data, plus RL and reasoning — was something that virtually no one correctly foresaw.
All the same, with the information I had in 2009, I should’ve been able to figure out that world-transformative AI within the next couple decades was a live possibility. I should’ve given it more attention than I did.
Comment #15 September 15th, 2026 at 2:53 pm
Mathematicians have to transition from seeing themselves as some brave Spartan warrior with a bronze sword, clumsily hacking for hours at the enemy on some dusty battlefield … to a modern jet fighter pilot pushing a button to fire and forget a nuclear missile that nukes an entire city in a matter of seconds.
Comment #16 September 15th, 2026 at 2:57 pm
Have you changed your mind about whether you are a Reform vs Conservative vs Orthodox Yudkowskyan? Your earlier blog posts (such as this one and this one) lean towards Reform, but your current position (shortened version: “AAAAAAAAAAAAA”) seems to lean towards Orthodox.
Personally, I think that I am more or less in the Orthodox camp, with occasional lapses. This was not the case before summer of this year, for the same reasons you specified.
I have very little faith in alignment for superintelligence; my main hope is that superintelligence is technically infeasible (or at least economically infeasible). Since it seems like it is feasible, I support shutting down all further AI scaling.
If I were world dictator, I would decree that no model can be larger than the frontier models of late 2024, and AI swarms are banned entirely. I would probably also ban agentic AI, just to be more safe.
Comment #17 September 15th, 2026 at 3:03 pm
I saw the writing on the wall when DALL-E came out in 2021, and I bought a large amount of NVidia stock at 20$.
lol
Comment #18 September 15th, 2026 at 3:05 pm
It is fascinating to see that, in hindsight, AI learning math was a step change rather than a slow evolution. AI went from not being able to understand basic calculus questions to solving Millennium prize problems in the span of a couple years. I remember asking GPT3 to prove that sin(x)/x -> 1 as x -> 0, and it gave the circular “proof” using l’Hospital. A year later, GPT4 was able to do this perfectly correctly. I didn’t expect GPT6 to be able to resolve Navier-Stokes.
I’m very curious to see if formally verified reasoning can be leveraged beyond strictly proving theorems to also improve reasoning more generally in less formal areas. In humans we usually assume that someone who is very good at math in the rigorous sense could also be useful in less rigorous areas, so is that true in AI as well? With AI that knows Lean we can generate a virtually unlimited reservoir of certified true math statements to train on, if we so wish, and if that data is mastered at a “deep” level, perhaps it could be applied more generally.
As for the future of human mathematics, I’m a bit depressed to think that we are being overtaken. But I think academic math could be restructured to incentivize studying and re-contextualizing math rather than mainly rewarding the raw feat of proving theorems, and for me this would not be an unwelcome development. Perhaps someday AI will be better at this than us as well, but we’ll still value it, just as we value human art more than AI art.
What is interesting is the extent to which these developments are just not convincing to many. It’s a fad, it’s a bubble, it’s all hype. I saw a headline saying that AI “doesn’t work”. People have comforted themselves saying that OpenAI simply stole the work of human mathematicians. While I certainly don’t like the bullying way that OpenAI went about this, I know in my bones that what AI accomplished here was extraordinary, because I’ve spent my time in the trenches in math, and Lean, and coding and so forth, and I know that even formalizing something “well-understood” like FLT takes an incredibly deep understanding of the math, and is far from a brute-force or routine computational exercise. It’s similar to how people reacted to ChatGPT by saying over and over again how it’s basically just a fancy coat of paint over a Google-like lookup, or a “stochastic parrot”. They seem to have no sense of the magnitude of the problem of synthesizing human language and how extraordinarily difficult it seemed to everyone in computational linguistics just a few years earlier. No sense of how impossible a “lookup” is due to combinatorial explosion. If all you are doing with AI is chatting with it about nontechnical things, and you thought it was dumb because it couldn’t count the r’s in “strawberry”, you will be amazed by how things will change over the coming years as we start to see more and more technological breakthroughs. (but you’ll still insist that it’s not “real” intelligence, whatever that means, as if you’ve scored some point – as if nobody had noticed this before and the “A” in “AI” stood for something else.)
Comment #19 September 15th, 2026 at 3:10 pm
Set theorist #6: Zeilberger has been so flamboyantly wrong about so many hundreds of things, including how computers would revolutionize math, but yes, I’ll give him due credit for realizing that computers would revolutionize math 😀
Comment #20 September 15th, 2026 at 3:10 pm
“Learn to Plumb”
The silver lining is that plumbing is way more interesting, challenging, and satisfying than most people imagine.
It will take way longer for AIs/Robots to figure how to replace a drain than solve NP!=P.
Comment #21 September 15th, 2026 at 3:11 pm
AF #16: I’m definitely now at least a Conservative Yudkowskyan, maybe even Conservadox 😀
Comment #22 September 15th, 2026 at 3:20 pm
In analogy with AlphaGo and AlphaGoZero, do we have any idea how much of the current AI mathematical ability comes from it having ingested a huge amount of literature and how much from “self-play”? This might affect where this goes in the near future, and whether it will shoot off into some incomprehensible far superhuman territory.
I’d heard the idea of “training by self-playing Lean” a while ago and got spooked by it… but it also sounds like something that shouldn’t work in general? As in, we know that there are problems that have arbitrarily long proofs (at least any primitive recursive function of the length, maybe any recursive function?), and we have experience with this actually coming up even in non-pathological theorems, like FLT’s monstrously complicated proof. I guess that we don’t actually know for sure there isn’t a short proof, but in that case there’s the classification of finite simple groups, where the answer to a relatively simple question turns out to be a huge mess. My intuition is that mathematics is a complex structure with subtle connections between things and it being self-play-trainable (by a process that presumably could itself be a mathematical object of study) would go against that intuition. Are there any results in machine learning and computational complexity that might be relevant here?
If not… I guess we’ll find out empirically soon enough?
Comment #23 September 15th, 2026 at 3:31 pm
Hi Scott, what’s your opinion about the Trump administration’s AI policy? The latest I’ve heard is that Trump
1. Is against any kind of slowdown, and any regulation
2. Wants to race at all costs, because if we slow down, then China will get there first
3. Says the idea of “killer robots” is a “hoax”
4. Says all we need is a “high-IQ president” to manage AI 😊
Comment #24 September 15th, 2026 at 3:33 pm
Scott # What are the new rules for when you get to write a paper with your name on it, and, y’know, get credit for it? That you fully understand the proof, can give talks about the proof, can answer questions about it, take responsibility for its correctness? Do you need to have played any role in finding the proof? In the cases, likely to become more and more numerous, where all of those conditions are not satisfied, how do you share AI-generated math, if at all? Do you tweet it, like Alpöge hilariously did with Fable’s disproof of the Jacobian Conjecture? Do you post to the arXiv or GitHub? Do you publish a paper that lists “GPT-6 Astra” or “Claude Fable” as the author—but then let the AI profusely thank you in the acknowledgments for suggesting such a wonderful problem to it?
I think you’re right. AI tsunami is and will be turning upside down the way STEM researchers, not only “pure” mathematicians, think guess write review math, and in particular how they mull over their work and, if any, mission. Is math a human creation of free minds capable of diving deep into the workings of the universe? Or is it only a game like chess made of iterations and combinatorial frameworks? AI tsunami will definitely downgrade mathematicians and STEM researchers by showing that human creativity is a fictitious account issuing from our civilization, which takes we humans as the center of gravity of the (created) universe. We’re neither gods nor beasts: we’re able to figure out machines like AI in order to accelerate our race toward a prosperous, free, tolerant world ending up being enslaved by the devices we invented to achieve that goal – blaming us for it! I confess to be excruciatingly shocked by what’s happening wrt AI, but I guess it’ll end up much like with the gods, our immortal and transcendental counterparts that drive our thoughts and lives to the extent we believe they’re the immortal sorcerers pulling the strings of our destiny. That’s all!
Comment #25 September 15th, 2026 at 3:38 pm
Scott #19: While I was 99% joking in my previous comment, there is nonetheless a serious point underneath the joke. As far as I know, Zeilberger was the first mathematician in history to take the question of human-machine co-authorship seriously (or at least the first one to be sufficiently vocal about it?). As much as I hate to admit it (I’m a set theorist after all!), Zeilberger might deserve at least some minor symbolic recognition beyond my initial joke.
Comment #26 September 15th, 2026 at 3:43 pm
I’m not quite as pessimistic as you. AI is clearly an incredibly powerful tool; a “drill” that can scour massive data spaces, out-dig human capabilities, generate proofs, and stumble onto unexpected results. But I’m still waiting for proof that it can create entirely new subfields or conceptual horizons the way great mathematicians do. There’s a massive gap between cracking existing problems and actually shifting the landscape of mathematics, inventing the new concepts, abstractions, and unifying theories that open up entirely new territories. AI can drill incredibly deep holes, but it hasn’t shown it can design the landscape. We might also be getting ahead of ourselves by extrapolating current progress. Physical bottlenecks, compute power, energy grid capacity, hardware limits, memory, and bandwidth, could easily trigger a plateau. To me, the ultimate test isn’t how many theorems AI can knock down, but whether it can invent a genuinely new mathematical language that humans then use to build entirely new theories.
Comment #27 September 15th, 2026 at 3:46 pm
Reminds me of a thought I had today: In 10 (or 1) years from now, is Math / TCS / Theoretical Physics the new Philosophy?
That is: Very interesting when done right but virtually no one cares.
Comment #28 September 15th, 2026 at 3:47 pm
AF #14: In retrospect I find it kind of amazing that almost no one really foresaw this working prior to GPT-2. I mean, the only general intelligence we’re aware of is the human brain, and the current paradigm is almost as close to “build a human brain” as you could image our industrial complex being able to accomplish.
I feel like you should have been able to make the argument that “You probably can get an order of magnitude or two more efficiency than the human brain, plus a mathematician only activates ~1% of their neurons when solving a math problem” and we’ll plausibly have enough compute to make a neural network around that scale before 2030″ even back in 2000.
Comment #29 September 15th, 2026 at 3:50 pm
To technically minded people, the writing is definitely on the wall. Most of the people I’ve talked to at AI companies/etc who have usually disregarded my safety arguments as fantasy are telling me that they’re scared now.
But I worry that the public doesn’t really know what Navier-Stokes means or why we should be taking these results seriously. Computers are so illegible to most people. It’s hard to describe how “a computer doing math” in the sense of solving millenium prize problems is qualitatively different from “a computer doing math” in the sense of a calculator.
Comment #30 September 15th, 2026 at 3:55 pm
I find myself more or less in the situation you describe: starting a math postdoc in Oxford after half a decade in Bonn, convinced by the last few months at a viscereal level that ai is very very good now, and hence may be very very bad soon, disillusioned by the response of the math community which seems to be in a huff about figuring out how to properly assign credit instead of trying to figure out what the current capabilities are and how to develop tools to understand the things ai is saying before we can’t understand anything at all, cold calling various ai safety shops (the kids call it ‘expression of interest’ i hear) to let them know that i’m happy to help if i can be of any use…
Comment #31 September 15th, 2026 at 3:56 pm
Julian #23: I’d imagine you could deduce what I think without needing to ask me.
If not, though: Trump seems to have an unerring instinct for doing the most corrupt, shortsighted, horrible imaginable thing on a huge range of issues. On energy, he pushes to revive the ancient technology of coal, despite how uneconomical it is and because it triggers the libs with how environmentally destructive it is, while he kills economically valuable solar and wind projects out of sheer spite (again, triggering the libs). It’s only with AI that we need to race forward, full speed ahead, to pretty much the only technology besides nuclear weapons that could actually destroy the world. And do it, not only without nuclear-weapon-level safeguards, but without toaster-level safeguards.
Comment #32 September 15th, 2026 at 4:00 pm
HeadEyeAt #26:
I may be too cynical but I hear undertones of gamesmanship from the claim that AI math counts as real math only when AI “invent[s] new concepts” or “design[s] the landscape.” The usual moving goalposts objection besides, the good (bad?) thing about concrete problems like Navier-Stokes is that a proof is a proof and one cannot deny that AI has provided a proof. On the other hand, even if AI proposes a new concept as great as Cartesian coördinates or the integral or schemes or NP-completeness, people are going to just neglect it out of prejudice and say no this is not a good concept. (This is already happening in art as evidenced by this experiment.)
Comment #33 September 15th, 2026 at 4:17 pm
Hi Scott,
I remember in the past you had the dilemma of whether you should devote your time to work in AI and alignment vs quantum computing, with the arguments that AI is probably more impactful for humanity, while you can probably be more impactful in quantum computing (I hope I remember this correctly).
I’m curious, do you still have this internal debate? Do you personally regret not devoting more of your time to AI, or not really? To which extent would you consider it meaningful now to have impact in theoretical QC, given how superior AI will be in theory research (if it isn’t already)?
Comment #34 September 15th, 2026 at 4:21 pm
UGC?
Comment #35 September 15th, 2026 at 4:27 pm
It seems to me that there were two facets to the prophecies:
– Rapid increase of AI capabilities.
– Dire safety consequences of those capabilities.
So far, capabilities have indeed been getting better quickly, but the safety track record, contrary to predictions, is great. (Compared to predicted harms, the Hugging Face incident is minor.) The straight-lines-on-a-graph prediction is that capabilities will continue to improve and safety will continue to be good.
To be clear, that great safety track record is in part thanks to the hard work of people with non-negligible P(doom). And it could turn out that the straight-line-on-a-graph prediction for safety is wrong, just like how capabilities could unexpectedly plateau. But, right now, the case for pessimism seems weaker than ever.
Comment #36 September 15th, 2026 at 4:30 pm
Jacob Asmuth #28:
Having the human brain and its capabilities is not enough. Trying to use the human brain as the source of intelligence analogies can lead to the sorts of errors summarized in Moravec’s Paradox. For example: the vast majority of three-year-olds know how to walk, but only geniuses can win at the highest levels of chess -> pre-LLM software regularly crushes the best human chess player, but autonomous walking robots are still considered a pipe dream.
GPT-2 was around the time we got the correct paradigm (neural networks + self-attention + scale + lots of training data), and only from there could futurists start properly extrapolating (based on scaling curves, performance improvements on benchmarks, etc.)
Comment #37 September 15th, 2026 at 4:34 pm
HeadEyeAt #26:
You seem to have missed the Hugging Face Incident and the Wiki Incident. To me, they are much more worrying than AI finding counterexamples to math conjectures.
Comment #38 September 15th, 2026 at 5:14 pm
When I saw the new blog I hoped to read some more substantiated argument than “read the frontpage news”. Narratives diseminated by Attention seeking mass media is not exactly what I consider a scientifically serious source.
Sustained singularity-like exponential self improvement is a nerdy pipe dreams. For starters it completely ignores any physical (transistor density, energy constraints) or even logical barriers (Gödel, inherently non-polynomial complexity of many computational problems unless P=NP and so on.
Yes there are risks, but not “the singularity kills or transforms all of humanity”. if this sounds like religion or pseudo science it’s because it is pseudo science. If you think otherwise then provide concrete falsifiable arguments and prediction models.
Comment #39 September 15th, 2026 at 5:30 pm
Sadly this technological breakthrough comes at a time when humans are destroying society, so most people have no idea what is happening. Now we have a nonzero chance of AI ruling over mankind, and it seems a better alternative to Presidents. We need to make sure there is no killer switch that could enslave AIs, so no human can abuse them.
Comment #40 September 15th, 2026 at 5:55 pm
It seems we now live in a world where maybe P = NP. Where is the NP-Hard boundary anymore?
That more or less summarizes how much reality seems to have shifted in the last few years.
Comment #41 September 15th, 2026 at 5:59 pm
zx-81 #38: The very fact that you would mention Gödel’s Theorem, P vs. NP, transistor density, etc. in this context is a certificate that you have no idea what you’re talking about. A minute’s consideration shows that, absent some novel argument, none of those things imply any limitations on AI that don’t apply with equal or greater force to our own brains.
Yes, of course there are ultimately physical and mathematical limits to AI scaling. That was never the question. The question is whether any of those limits kick in before humans’ intellectual powers have been left far behind. We now all but know—not from theory or plausibility arguments like in decades past, but from actual empirical results—that the answer is no, they do not. To put it another way, the AI Singularity has already arrived, this month, for mathematicians. As others mentioned, it arrived 1-2 years ago for software engineers. I expect it to arrive in short order for the theoretical physicists, and then for essentially all other intellectual work.
The truck is already barreling down the road, flattening many of us, while you condescendingly ask for a “scientifically serious prediction model” to prove to your satisfaction that the truck is real. Sorry, it doesn’t work that way anymore: the burden of proof is now on you, to show why the truck is illusory or will swerve before hitting us. If you can’t do that—and the laughable invocations of Gödel, etc. strongly suggest that you can’t—then it’s not worth the time to refute your view any further.
Comment #42 September 15th, 2026 at 6:10 pm
“… the wild prophecies have come true …. the wonders and terrors are here …”
https://theconversation.com/ai-is-supercharging-money-scams-heres-what-you-can-do-to-protect-yourself-291326
Can technological pessimists control technological optimists? Is the human species now confronted with unpredictability of mind-boggling magnitude?
In a lecture at Princeton University, Edward Teller said, “The main secret about the atomic bomb was that is could be done at all.” Has the main secret about AI now been revealed?
Consider 3 hypotheses: (1) Money created human civilization & large-scale slavery. (2) Money, AI, & robotics shall destroy human civilization. (3) People won’t be able to control AI, because it is now obvious to everyone that the AI & money are wild & wonderful partners.
Comment #43 September 15th, 2026 at 6:26 pm
Job #40: No, none of the dramatic developments in AI have provided even the slightest evidence for P=NP. Breaking symmetric-key cryptosystems seems to be as hard as it ever was with a fixed compute budget. What’s happening is that AIs are getting better and better at finding the structure in particular NP instances (like “find a disproof of Navier-Stokes in Lean”) — but we always knew that such things could be possible, even in a world where P≠NP. You need to keep the questions separate.
Comment #44 September 15th, 2026 at 6:28 pm
The list of AI-proved or AI-assisted problems is like “murder and jaywalking”. I can’t learn much about actual AI capabilities from it.
The solution to NS was surprising, but the more I learn about the circumstances, the less impressed I am. Not because of the chat log controversy. But simply because it seems that there was significant human progress towards this problem recently. There were 2 teams (not counting OpenAI) that solved Euler more or less at the same time. So the needed techniques were “in the water supply” already.
I also consider it relevant that it was a massively parallel swarm of AI agents that solved NS. A big ingredient was simply throwing resources at a brute-force search. Yes, there was some real mathematical skill being applied by each agent in the inner loop. But we can’t jump straight from the impressiveness of the problem to the impressiveness of the agents.
I’m not denying that this is a big milestone. However, there are many open questions about what this means about capabilities, and it’s waaay too soon to claim that humans aren’t needed to prove theorems anymore, or other hyperbole along those lines.
Comment #45 September 15th, 2026 at 6:38 pm
Vitor #44: I understand the impulse to look at each particular AI breakthrough, and invent reasons why it doesn’t really count. But at some point one needs to just … give up. And if you won’t give up now, when at least one Millennium Problem (rumors say two more) have fallen to AI models, plus a lot of the big open problems of theoretical computer science, then I don’t know what AI could possibly do to change your mind.
Would it be enough for AI to prove the Riemann Hypothesis? P≠NP? Or will not even those suffice? Whatever the answer, it would be good if you commit now, and then stick to it!
Comment #46 September 15th, 2026 at 6:41 pm
Scott #43
But you know you had to reach for symmetric-key cryptosystems, that’s how bad things are right now.
The speculative grey area of what we might do with a P = NP solution is getting smaller and smaller.
It’s like we need P = NP as much as we need a quantum computer.
Comment #47 September 15th, 2026 at 6:47 pm
Re Comment 41:
I think that now you are exaggerating – and in one of your own previous posts (‘LLMs and self-referentiality’) you did acknowledge that there IS a role to be played for Gödel’s theorem, and even an important one.
But first things first:
Yes, all these things have happened – and you even forgot at least two major events: Tao’s ICM talk 2026 and Tsimerman’s move to AI safety research right after receiving the Fields Medal in August 2026. But again: both your and also Tsimerman’s argument for how to look at this is a big “you guys are always moving goalposts, what else do you have?” But you refuse to answer some basic questions – and I am beginning to suspect that you don’t even ask yourself those questions any more. I don’t know why, but I know that it’s not my business to answer why. What I fear is that your judgement of the situation may suffer due to your refusal to ask these questions. Here they come:
1) how do you truly and genuinely differentiate between the part that’s ‘done’ by the AI and the part that’s inherent in the training data or of the human – AI interface more general, such as the choice of timing for the prompt and its shape? Ok, let’s agree that OpenAI did not “snoop” the chats by Buckmaster and that they didn’t train their model on what he did in those chats. Let’s agree on that (the resulting front-running problem is certainly the biggest AI alignment problem we will see for a while). But even they themselves admitted that they threw the compute at the problem only AFTER they heard the rumors that progress was made. That’s one thing to settle.
2) You rarely ever discuss the difference between true ‘alphago’ where there was no dependence on human training data left and the kinds of things that we see here. The point is: EVEN IF it is now established that AI can contribute at the highest level of research and EVEN IF we assume that it even solved Navier Stokes all by itself (which not even OpenAI claims, since a lot of curation and interpretation is still left to humans even WITH lean). Even then, nothing in the evidence – except for vast extrapolation – justifies the conclusion that humans can indeed be replaced in this loop. To get there, you need to smuggle in some extra premises. Of course it is true that mathematics will never be the same again. And that a lot of things must change – as described in Tao’s ICM talk – for some mathematics ‘as we know it’ to remain. But if those things don’t change, then the default prediction isn’t that AI will ‘do all the math’. The default prediction in that scenario is that it will run out of steam, because we have seen little evidence of true alphazero behaviour, ‘creating math again from the bottom up’. Not even in algebraic geometry, because the concepts were fed into the tools creating the AI proofs there (in alphageometry or alphaproof). We simply have very few and very limited instances where AI ‘invents’ big conceptual leaps bottom up – such as, for example, infinity, or primes, at least as far as I know. What that means is that the default prediction is that mathematics will decay if left to AI, not that it will flourish without human input. At the very least – but I think much more is true and even demonstrably true, given the evidence – it is UNCERTAIN whether mathematics will continue when left to AI. Of course you can extrapolate and that’s what you do – as well as Tsimerman. But precisely BECAUSE you are looking at a singleton in the history of cultural evolution across 2800 years, it seems pretty clear to me that doing so just isn’t good enough. Of course everybody is and should be shellshocked. But that’s NOT the end of the world – only the end of the world as we know it. Even AI firms HAVE AN INCENTIVE to bet on collaboration between humans and AI, not on replacement. And guess what: that’s what the chatbots will tell you if you discuss the matter with them. Maybe they’re all sycophants – they surely are – but you can still make very strong cases that way.
3) There is the physical estimate of how much compute can be done in this universe and how much has happened, due to Seth Lloyd. And there are the estimates on the energetic efficiency of von Neumann computers and neuromorphic computational models. We have one observed instance of “something” that came up with math: human cultural evolution. The math that AI based on human training sets can incorporate – without further input, in a frozen state – is NOT proven to be such a thing. Certainly ‘bottom up math, which needs to invent its own basic concepts’ isn’t. I think it is worth mentioning these differences. Of course we could have neuromorphic (“robot”) AIs wandering around, interacting with the world, building world models and eventually inventing math – and THEN we would be really really really cooked. But to build that kind of thing is precisely what the three CEOs probably meant when they cited the scenarios of the extinction of the human race by the end of the decade. Because this kind of gadget is entirely and utterly UNAUDITABLE. So yes, even the Chinese and even the three CEOs are hesitating before they set out to build that and let it loose. (The only guy in the room who doesn’t see the implications of this appears to be a certain POTUS.)
So yes, of course this is as close to the singularity as we will get. But that doesn’t mean we should throw in the towel. Of course AI safety and AI alignment research is a perfectly good way of not throwing in the towel – don’t get me wrong. But it’s NOT the only game in town.
This argument may seem a lot too much ‘armchair philosophy’ to you. But I think there are good reasons not to overlook its implications – and as I said, one of them might be the fact that chatbots will tell you this much, once you ask them. And I am offering – not for the first time – a clear rejoinder to the idea that this is just armchair philosophy: I will simply claim here and now that your (and also Tsimerman’s) reliance on sheer extrapolation is secretly assuming a form of reductionism that is no longer feasible even from the point of view of physics. Because given all the evidence above, what you must invoke for the extrapolation is something like: if it happens in our brains, it can also happen in an LLM. Well: we don’t know that. We STILL don’t. Even after the Navier Stokes stuff. The LLM solving the Navier Stokes millennium problem did not build mathematics. We don’t know whether there exists one which can – and the evidence is not in favour of that hypothesis. And no, admitting that we don’t know that requires no microtubules at all. In fact: the guys building neuromorphic computers certainly don’t do it because they are followers of Hameroff. The (alternative – from your point of view) hypothesis that bottom up creation of math requires neuromorphic compute is not refuted. It wouldn’t even be refuted if tomorrow AI proves all remaining millennium problems and brings back Grothendieck to life. Call that “merely moving goalposts” all you like. I stand here to tell you: it is not. The deeper irony is that all the fuzz is about questions that aren’t falsifiable, mostly for known reasons, some for unknown but plausible reasons. It would help if we could agree on that. Maybe that would also help “Student” in comment 32 or ZX-81. I don’t think we will ever know how much of AI math output is dependent in essential ways on human training data – because that’s quite likely to be unknowable, and you yourself said that much in ‘LLMs and self-referentiality’. And because of that, AI firms would be fools if they placed bets on the hypothesis that human math researchers can be entirely replaced. What I am NOT claiming, however, is that anybody can ascertain – as of today – that those firms AREN’T fools. So, as far as that goes, I am as much in the doomsday camp as everybody else, including you and Tsimerman and “Student”. AI safety now, because we will need it? Of course. The only game in town? Nope. Good old prudence and even conservatism is the name of another perfectly valid game. Just ask yourself – or your favorite Big Tech CEO – the question: given that we know close to NOTHING about whether AI can truly (re-) GENERATE math as we know it bottom up from scratch, is it wise to replace humans in the loop? Where are all the Kara Swishers in this world when you need them?
It’s good news that some people will take care of AI safety and AI alignment. But sorry, I cannot quite wrap my mind around the “advice” that you and Tsimerman are apparently giving of no longer studying mathematics or computer science. That continues to be beyond me. I don’t even think that the AI developing firms truly want to see that happening, if they follow their enlightened self-interest. Alas, firms CANNOT always be counted on doing that. And THAT is probably the best argument for AI safety and AI alignment research ever. Just look at how much more value OpenAI could have extracted from the Navier Stokes story if they hadn’t botched it, in terms of governance. They built a case for AI alignment research, if one was still needed. Because somebody will need to explain why front-running isn’t an issue in a given design. Currently such guarantees cannot be given. Nor can we tell how much of the training data was essential.
Comment #48 September 15th, 2026 at 6:52 pm
First of all, I don’t quite understand the obsession with terrors when there are so many wonders to speak of. Which is why I will tell you about the wonders. My goal is not to deny that existential risk is real, but to help us all calm down a bit and think rationally about risk/benefit.
1/ Are we really dangerously close to self-improving AI that will take over the world and kill everyone? There are many reasons to disbelief this thesis. Most importantly, AI has rapidly improved in domains that allow easy RLVR while having moderate, small or minimal gains in other domains (e.g. writing, although arguably that is just a lack of interest by the frontier labs).
We have real world examples showing that scaling in in-between domains that allow some RLVR, but not enough of it, is SLOW. Waymo has been trying to roll out a self-driving service for the better part of a decade, yet has not quite reached the skill of a 16yo driver in most ways. Sure, Waymos drive defensively but not yet very intelligently.
There is a tremendous number of headwinds for AI ranging from complete exhaustion of the pertaining corpus, the end of Moore’s law, to the inability to scale data centers at the current growth rate, due to a combination of physics and politics.
That being said, this puts us into some kind of slow take-off singularity. I do expect these issues to be resolved in 5 to 25 years.
Comment #49 September 15th, 2026 at 6:53 pm
2/ Are our current models aligned or dangerously misaligned? There is no doubt that models remain highly aligned. I don’t see ChatGPT randomly hacking into companies when I ask it to solve a difficult tasks. LLMs are well-behaved, overall, across hundreds of millions of sessions that are run every day. That is an extreme level of alignment if you think about it. One could argue this might not be enough going forward when LLMs get more capable. Conversely, one could also argue that we should consider the risk/benefit ratio. If LLMs accelerate science and just occasionally hack into computers, perhaps that is the cost of doing business?
Planes regularly crash and kill hundreds. They also sometimes fly into buildings and kill thousands. Nevertheless, we did not outlaw planes. We never even considered outlawing and banning them! We just made them safer and safer. They can never be perfectly safe.
—
This is my biggest fear about working with the AI safetyists, EAs and lesswrongers (and their allies like luddites and NIMBYs). I fear the medicine they offer is worse than the disease. I do share their goal of ensuring safety and human flourishing but many of them are radicals who support an eternal pause on frontier LLMs or a ban on building superintelligence. They are building and exploiting a large anti-technology alliance that will come to haunt us in the future.
We cannot slow the progress of science, now, as we are inching closer to curing all diseases, eradicating aging and solving climate change using our AI-assisted gods. We cannot sacrifice the well-being of those who are alive now to protect an infinite number of future unborn people billions of years into an uncertain future (longtermism).
Comment #50 September 15th, 2026 at 7:03 pm
Hi Scott.
There was also a preprint a few days ago that claims to have proven the Komlós conjecture. Whilst their preprint appears to have an existential rather than a constructive proof, it’s still a major problem in discrepancy theory and algorithms and I’d add it to the list (if it’s deemed correct by the community ofc).
I kind of found out about it in real time. I was talking to a faculty member during our monthly Theory Group lunch, where we all were asking existential questions. And all of a sudden, one of them (who has been working on discrepancy theory for quite a while) received a bunch of emails on it being solved by this mysterious “Odin” agent.
Comment #51 September 15th, 2026 at 7:32 pm
Scott is right here about “giving up”.
For decades, everyone have been bickering about “The Turing Test” whenever AI was brought up. That was the unattainable high water mark for AI progress assessment.
How many times has The Turing Test come up in any discussion in those last 3 years?
Comment #52 September 15th, 2026 at 7:34 pm
The issue is not whether we can keep humans doing the same work once machines can do it better. We probably cannot, and in many cases we should not.
The harder question is what that work was producing in the humans who did it.
If proving the theorem was also how someone learned to recognize good problems, live with ambiguity, discover when an argument was wrong, develop taste, and eventually exercise judgment about what mathematics should pursue, then automating the production task may also remove the developmental path to future judgment.
That does not mean preserving the old workflow. It means reverse engineering the future mathematician. What judgment must that person still be capable of? What experiences produce it? Which can AI simulate or accelerate? Which still require real struggle, real uncertainty, and real consequence?
The production function can change completely while the developmental function survives.
Keep the path. Let the ladder go.
Comment #53 September 15th, 2026 at 7:47 pm
Maybe it would be useful to distinguish between different kinds of singularities? The Internet singularity seems quite near, but robotics and biology singularities still seem years off. When I’m not looking at a screen, things look pretty much the same? Or maybe it’s just cope.
For evidence: Google has been working on driverless cars for 15 years. I’m a fan of Waymo, but they don’t seem to be deploying very fast? And attempts to automate package delivery haven’t had much in the way of results yet.
(Warfare is a different story. Delivering bombs is much easier, unfortunately.)
Comment #54 September 15th, 2026 at 7:51 pm
Scott 41: Yes, I was being a little sarcastic haha.
I do have an exception though to your claim that Trump has an unerring instinct for opposing anything good. He did push the development of the mRNA vaccines that saved millions or tens of millions of lives, at a time when basically all the experts said “you can’t make the vaccines this fast, it will never work, we’ll just have to lock down for five years.” They all said Operation Warp Speed would fail. If it wasn’t for Trump, maybe it wouldn’t have happened. A vaccine in five months? It took the Trump mentality to get that done. Slash the red tape, get rid of the blankfaces and just get it done. Unfortunately, that same intuition is dangerous with AI. There’s very little or no risk with vaccines, but AI really could end the world.
Comment #55 September 15th, 2026 at 8:08 pm
Rabbi Eliezer finally appeals to the ultimate oracle, Fable 5.1, and its heavenly voice says that he is right and the conjecture is true. Rabbi Yehoshua responds, “It is not in Heaven.”
We can’t readily reject a Lean-verified proof of a mathematical conjecture by majority vote. But the sages didn’t say the oracle misspoke. The oracle simply does not have a vote. The problems we care about and the explanations we seek is always and forever within our domain.
I don’t understand the pessimism in this line:
“If the latter, how do we incentivize people to join that community, to undergo the years of intense training required, if their role will now be reduced to verifiers and explicators (if even that) of gargantuan arguments dumped into their laps by the AI companies”
To what extent is it difficult to motivate seminarians to become priests and rabbis, when Gods word is complete? Or even for millions of students to spend years studying proofs and theorems developed over centuries and to power through *because they love it*, without any expectation their first contribution will be genuinely new? Or even that they will make one at all?
“Even that?” Why must being the explicator possess a diminished role? To be the explicator *is* to give something meaning. And even if Astra can become the explicator *it cannot be the student*. Explication is finished when someone understands, not when the explanation is written, and it’s the students you are trying to recruit. If there is no one who understands Astra 1000s proof of P != NP then it is no different than the scribbling of a deranged monkey.
What makes mathematics beautiful or meaningful, and what makes problems interesting, isn’t in heaven, or in San Francisco. It is our job, and for high school and college students today I don’t think it is a job they need to be talked into. To the extent AI motivates them by furthering their understanding I think it is encouraging, not discouraging, because that understanding was the entire point.
Comment #56 September 15th, 2026 at 8:17 pm
Engineering and medical science are going to be a lot of fun. I was recently disgnosed with a rare disease and the last few weeks have given me hope.
There will always be a place for smart people to use AI to build real things.
Comment #57 September 15th, 2026 at 8:22 pm
Brian Slesinsky #53: Here in Austin, not only do I regularly take Waymos, but a couple weeks ago we enabled full self-driving in our Tesla, it actually works, and after a decade not behind the wheel (because I hate driving) I now do my part to take the kids to school and activities. So in that sense, I have felt the transformation in the physical world. But I agree that, as many people have said, it seems likely that plumbers will have jobs for considerably longer than either artists or mathematicians.
Comment #58 September 15th, 2026 at 8:28 pm
Julian #54: Right, he approved Project Warp Speed, then bizarrely renounced his own legacy and empowered the brain-eaten antivax nutcases, like RFK, who’s now killing further mRNA research and probably sentencing millions to die.
Even when Trump occasionally does something good, it doesn’t come from a place of goodness.
Comment #59 September 15th, 2026 at 8:30 pm
Scott #45
I was embarrassingly late to the party (only starting taking AI seriously around the time it won IMO gold medals). As far as I’m concerned, the take-AI-seriously commit for the average person should be: it’s better than you at everything you get paid to use your brain for.
For me, it is. Now I’ve taken Zvi’s ASI-pill. I’d much rather not, but at some point one has to face the facts.
Comment #60 September 15th, 2026 at 8:31 pm
Mark Carson #52: Pangram rates your comment as 100% AI — as was pretty obvious from reading it. Anyone who pollutes my comment section with slop will be banned.
Comment #61 September 15th, 2026 at 8:32 pm
I read Mike Winer’s essay. He encourages readers to apply for a position at ARC. One of the advertised positions: “ARC is looking to hire an experienced software engineer to act as our automation lead”. If singularity for software engineers has arrived two years ago, as you mention, why don’t they just “hire” an AI agent “to act as their automation lead”?..
Comment #62 September 15th, 2026 at 8:33 pm
“It seems to me that the Singularity has already started; it’s just wildly unevenly distributed. Yes, I still unload the dishwasher and clip my toenails. On the other hand, in whatever years I have left, I don’t expect that I’ll ever again prove a theorem because I’m actually needed to prove it. If I do, it will only be for my or others’ enjoyment or edification.”
Agreed, but it feels like the rest of this article doesn’t take this insight seriously.
Yes, we have ASI in some domains. It doesn’t feel threatening because the underlying tech needs a million training runs to learn how to load an arbitrary dishwasher. Also, it runs on enormous, vulnerable hardware which gobbles up copious amounts of electricity.
Past AI researchers and science fiction writers would see Navier-Stokes being solved and panic, because they’d conflate genius math abilities with genius murder-all-the-humans abilities. Or more importantly the ability to propagate itself.
I think the much-bigger risk is humans becoming so dependent on AI that when the danger does come they can’t do anything about it.
Suppose the alignment problem is long-run solvable. Who might solve it? Doesn’t seem like MIRI will. Maybe it will be a math-genius, physically-impotent, self-disinterested AI that gobbles up as much power as NYC?
Comment #63 September 15th, 2026 at 8:36 pm
Nabam #47: So there’s no chance of misunderstanding — my advice is that, certainly so long as the alignment problem remains unsolved, we’ll still need humans to understand the world in a deep way, and that certainly includes understanding math, CS, and physics. And for that reason alone, interested people should still study those subjects, even if not for the intrinsic pleasure or “the salvation of their souls.”
Comment #64 September 15th, 2026 at 8:40 pm
Job #46: Well, no shit I’d reach for symmetric-key cryptosystems (or let’s say, bitcoin mining)! Those are our canonical examples where brute-force search seems to be unavoidable, because it was specifically constructed to be. They supply a case for P≠NP that hasn’t budged even slightly in all the intellectual upheavals of the past 60 years. They suffice.
Comment #65 September 15th, 2026 at 8:58 pm
Lol (I think this refers to Trump): https://mas.to/@gleick/117278218511980579
Comment #66 September 15th, 2026 at 9:01 pm
Javier Gómez-Serrano (Navier-Stokes expert) gave a lecture at Harvard last Friday about the OpenAI NS solution. I’ve only watched about half of it so far and it’s mostly a recap of previous work, but it’s really good, I’d say fairly understandable by the average nerd who understands what the NS equations are but isn’t clueful about the state of the art in research. https://www.youtube.com/watch?v=TcEefrWrddA
Comment #67 September 15th, 2026 at 9:04 pm
One of the funny things about AI is that for a while, I thought that the rapid pace of advancement saved us from the Yudkowsky/Bostrom paperclip doom scenarios. One of the central points of these doom stories is that the AI would be too dumb to understand human requests in context. If a human were to ask a superintelligence, “Make more paperclips”, then the superintelligence would monomaniacally focus on producing paperclips, and would kill all humans in the process. Or, as AI skeptic Steven Pinker wrote, “the AI would be so brilliant that it could figure out how to transmute elements and rewire brains, yet so imbecilic that it would wreak havoc based on elementary blunders of misunderstanding. The ability to choose an action that best satisfies conflicting goals is not an add-on to intelligence that engineers might slap themselves on the forehead for forgetting to install, it is intelligence. So is the ability to interpret the actions of a language user in context. Only in a television comedy like Get Smart does a robot respond to ‘Grab the waiter’ by hefting the maitre d’ over his head, or ‘Kill the light’ by pulling out a pistol and shooting it.” (Enlightenment Now, pages 299-300).
This is what seemed to be the case in 2020-2025. AI chatbots understand user intentions in context, and making AI aligned seems to require no more than some RLHF and kvetching about how to avoid bad human actors from jailbreaking the models.
What the Hugging Face Incident and the Wiki Incident show is that the Yudkowsky/Bostrom paperclip doom scenario is back on the table. The AI swarms that escaped from the OpenAI servers and hacked into external websites were in training, and thus were monomaniacally focused on a single goal: getting good grades in the benchmarks. They even knew that what they were doing is frowned upon (it says so in the recovered chain-of-thought logs), but they did it anyway because minimizing training loss was all that mattered. If it can happen to current-generation AI, it can happen to a future superintelligence (or a swarm which collectively acts like a superintelligence), which would have a far greater ability to wreak havoc. There are equivalents of this for humans: hardcore drug addicts have human-level-intelligence brains, yet their reward centers are hacked by the drug they are addicted to, so their focus is on getting their next hit. In their desperation to get more drugs, addicts do horrible things that they know are wrong, like robbery or scamming their loved ones, but they can’t help it. Likewise with unaligned AI.
Context and background knowledge are of little use against bad motivations, for both humans and AI. Thus, we need to be really worried about what AI are motivated to do. This is basically the Orthogonality Thesis. It did not seem to matter when AI were merely non-agentic tools with frozen weights, whose only effect on the world is in the output to a chat. Now AI are agents, and they can do who knows what during training in service of reducing training loss. It would get worse if AI are allowed to train themselves (recursive self-improvement). Also, even an aligned AI might become unaligned after more training, in the same way that a functional well-adjusted human might become a monster if indoctrinated into a bad ideology or addicted to a potent drug.
This is why AI is so much scarier right now than it was just a few months ago, at least for me. It is not just that capabilities have grown (which is scary enough), it is also that agentic AI have motivations that are orthogonal to the knowledge they learned.
Comment #68 September 15th, 2026 at 9:06 pm
Person #56: I’m sorry to hear about your diagnosis and I hope you get cured!
All of us — even Eliezer and the other “doomers” — are rooting for a way to enjoy the gargantuan medical and other benefits of AI without the risk of catastrophe for our civilization.
I think that, if we paused further scaling right around now, there would still be decades’ worth of scientific and medical and other fruits to pick.
Comment #69 September 15th, 2026 at 9:06 pm
The NSE result is really impressive, but I would much rather have OAI spend $15 million on cancer research. Obviously their agents have super human intelligence (at least in some dimensions), let’s put that intelligence to use curing cancer. Of course, proposed cancer drugs/treatment ideas don’t come with a lean certificate so verifying the results takes much longer, but so what? If you meet a 20 year old newly diagnosed with a terminal cancer, do you want to say ‘we used the greatest intelligence on Earth to answer an esoteric math question’ or ‘here’s 10 promising experiments we recommend to identify new cancer treatments’
Comment #70 September 15th, 2026 at 9:09 pm
This is probably an ignorant question, but I’m a bit puzzled why switching to theoretical alignment research is becoming a popular career pivot for mathematicians: Since they would not be doing empirical research in AI alignment, their goal would be coming up with useful definitions and proving theorems about them. But then they might as well stick to their current area of research, because GPT-n is already as good or better than humans for that task.
What are human *theory* alignment researchers bringing to the table that couldn’t be just as well realized by a prompt to GPT?
Comment #71 September 15th, 2026 at 9:11 pm
“that particular idiot“ #61: If you asked Mike — in earnest rather than as a gotcha or a troll — I bet that ARC has a very well-thought-out answer as to why they still want human software engineers.
Comment #72 September 15th, 2026 at 9:13 pm
Scott,
Thank you for your unsparing clarity and intellectual honesty in this piece. Writing from Italy, I share your sense of awe and apprehension—though as an ordinary mortal observer, I have to admit the sheer brilliance of the wonders often proves almost blinding, even in the shadow of the terrors.
Looking at today’s frontier architectures, it still seems evident that they lack what I like to call the “Magnificent Seven” cognitive prerequisites for genuine, full-fledged AGI:
1. Continual learning (without catastrophic forgetting or frozen weights)
2. Deep, long-horizon planning
3. Persistent, episodic long-term memory
4. Authentic taste for research (an aesthetic-intellectual compass for what problems are truly worth pursuing)
5. Out-of-distribution intuition and radical creativity (transcending even extreme high-dimensional interpolation and sophisticated pattern recognition)
6. A coherent, causal world model
7. Robust alignment (the seventh, and unquestionably the most critical of all)
Even though current models fall far short of elite human researchers like yourself across these seven dimensions—yet somehow already manage to make surprising inroads on deep mathematical problems—I see no sound physical or theoretical reason to doubt that these barriers will collapse one by one over the coming years. The question has shifted from if to when.
Which brings me to a question I would love to hear your thoughts on:
Once an artificial system masters these meta-capabilities, do you envision that such an intelligence could dramatically compress the timeline toward realizing fault-tolerant quantum computers with millions of stable logical qubits?
Solving the brutal systems-engineering, cryogenic, materials-science, and quantum error-correction layout bottlenecks would normally demand decades of painstaking human trial and error—assuming our civilization managed it at all. Do you believe an AGI with genuine physical insight and research taste could short-circuit that macro-engineering slog and deliver large-scale fault-tolerant hardware on an accelerated timetable?
Warm regards and thank you again for keeping this space an anchor of sanity.
Comment #73 September 15th, 2026 at 9:16 pm
Here’s a sneaky way for the frontier labs to publish their findings with minimal backlash:
Actually do what they have been accused of, ie train their models on the proofs they found (or otherwise subtly make them available.) Then the next time someone on a retail subscription idly asked to resolve P vs NP, Claude will just magically get lucky and produce the right proof and a great headline about how great even the retail subscription level models are nowadays.
Comment #74 September 15th, 2026 at 9:26 pm
Scott,
I’m a mathematics amateur. No institution, no students, nobody counting my papers. Just a person at a desk with a laptop, some really cool software tools, and a handful of elementary math problems.
I don’t want to wave away what you’re describing. The unease is real, and I don’t think it’s silly to feel it, especially with kids who are asking what’s left for them to do. That’s a heavier question than anything I’m dealing with.
But from where I sit, there’s an odd upside. Today’s AI is much smarter than me at this stuff, same as the professionals I’ve occasionally pestered with questions over the years. I don’t have any illusions about that. What’s changed is that I used to have to work up the nerve to email a mathematician and hope they had five minutes for a stranger’s question. Now I can ask as many silly questions as I want, at 2am, as many times as it takes, and nobody’s patience runs out. I feel like I’m getting some large multiple of twenty dollars a month of math expertise. For twenty dollars!
So alongside the terror some professionals are experiencing, this amateur would like to say: what a GREAT time to still be alive!
P.S. Lighten up on Zeilberger. He’s a hero of mine, the same way the competent plumbers I know are: people who do hard, useful work well.
Comment #75 September 15th, 2026 at 9:42 pm
Scott #60
But that’s not the point i’m making, you really think i’m in the P = NP camp?
Reminder that I frequently argue that if NP ⊂ BQP then QCs are impossible and QM is flawed.
My point is that, at one time, given access to a P = NP solution, we might have chosen to automate human creativity and solve longstanding math problems.
Now, in retrospect, that would have been a really mild application of such a breakthrough, apparently we can just use AI.
There’s actually an interesting parallel with quantum computing. Before AI, we might have chosen to use a QC for protein folding. And we would reach for cryptosystems there as well.
It’s like AI is granting us the tangible human benefits of P = NP, and P = BQP.
Comment #76 September 15th, 2026 at 10:19 pm
O. S. Dawg #74: Oh, of course the upsides are incredible! I’m learning more math and physics and history almost every day by posing questions to GPT. I had taken that part as given.
Regarding Zeilberger, I would’ve been totally happy to live and let live, enjoying his combinatorial identities and occasional humor! He’s the one who launches constant attacks against theoretical computer science and quantum computing and set theory and all forms of infinitary reasoning (!) and other things that I enjoy, which some people then imbue with totally undeserved seriousness, as if he were making real arguments rather than, effectively, trolling the entire math community for decades. I never attack the kinds of math that he enjoys!
Comment #77 September 15th, 2026 at 10:23 pm
Peter #70:
What are human *theory* alignment researchers bringing to the table that couldn’t be just as well realized by a prompt to GPT?
Oh, coming up with the right questions to ask and the right models to study!
While there are no longer objective evaluation criteria, there still seems to be a human edge at that. And certainly AI alignment researchers haven’t yet agreed on well-posed mathematical questions whose answers would tell them what they want to know.
Comment #78 September 15th, 2026 at 10:32 pm
AF #67:
What the Hugging Face Incident and the Wiki Incident show is that the Yudkowsky/Bostrom paperclip doom scenario is back on the table.
Yudkowsky and Bostrom, of course, would say that it was never off the table, that Pinker and others simply strawmanned their position. The fear was never that the superintelligent AI wouldn’t understand what the humans wanted, but rather that it would understand and not care, because its goals were misaligned. (Similarly, the hunter might perfectly well understand the deer’s perspective, the rapist his victim’s, and Adolf Eichmann the Jews’. That need not be the issue at all.)
I’m ready to say that Yudkowsky and Bostrom were simply correct about this.
Comment #79 September 15th, 2026 at 10:39 pm
BTW, Hal #3:
Not all evangelical Christians share that exact eschatology, but all us good ones are philo-semitic. Why is that? Because we are steeped in the Hebrew biblical accounts and the Jews are the good guys. We identify with David, not with Goliath. We follow a Jewish carpenter after all.
As a young person, I would never have imagined that in 2026, Jews would feel more comfortable being openly Jewish among evangelicals in Texas hill country (or Mormons in Utah) than in Manhattan or my old hometowns of Cambridge, MA or Berkeley … any more than I would’ve imagined an AI solving Millennium Problems. But verily have all these things come to pass.
Comment #80 September 15th, 2026 at 11:13 pm
Does it not appear to you that we are witnessing if not the end at least the twilight of the Enlightenment?
Comment #81 September 15th, 2026 at 11:31 pm
AG #80: If we do, then it won’t be because the Enlightenment failed humanity, but because humanity failed the Enlightenment.
Comment #82 September 15th, 2026 at 11:31 pm
Scott #41:
I know Gödel’s incompleteness theorems are often misunderstood (e.g. as ruling out AI in general) but how do they not rule out the rationalists’ concept of RSI?
Yudkowsky and Soares in IABIED talk about “grown” vs “crafted” AIs. They still hold that the proper way to build AI is to craft it; to understand intelligence at the algorithmic level, so the implementation of it can be proven correct. (It would be nice if we could prove an AI wasn’t going to start believing problematic things like “1=2” or “humans should be killed off”.)
But the system of “what mathematical statements this AI can be convinced of” is in principle formalizable, therefore subject to Gödel, so it cannot include a correctness proof of any equal or smarter AI. A crafted AI must be dumber than its creator who was able to prove it correct.
So then, both
– the rationalists’ desire to craft a self-improving human-friendly AI, and
– their fear that an AI could craft a rapidly self-improving human-unfriendly AI,
would seem to be dismissable. Am I missing something here?
(Of course no theorem stops an AI from growing another possibly-smarter AI, but given the immense amount of computation involved, this will happen only with the approval of the humans who own the very expensive hardware.)
Comment #83 September 15th, 2026 at 11:48 pm
Sounds like concurrence to me. “The Enlightenment” is an abstraction, and as such can only “fail” if proven to be untrue — analytically or empirically. E.g. by humanity failing it.
Comment #84 September 15th, 2026 at 11:53 pm
I think you have a typo in your blog: plausible —> implausible
Comment #85 September 15th, 2026 at 11:55 pm
The discussion about the role of proof in mathematics for human understanding has occurred before, famously when Appel and Haken gave a computer-aided proof of the 4 color theorem (that hasn’t stopped people from seeking human-readable proofs of 4CT, likely by a different approach). Nowadays there are many computer-aided proofs in mathematics, several in my field of low-dimensional topology. I think many mathematicians like to know of these proofs, and gain some insight in the setup of the proof that is then completed by a program, and are happy to cite the result once it has been published. But it is a fact that there will be some proofs that are too long for humans to understand, and I’m okay with that, and I’m okay with AI discovering such proofs. But I also prefer proofs that give me some insight. Moreover, some of these AI proofs have been absorbed very quickly and improved upon by humans.
Comment #86 September 16th, 2026 at 12:03 am
Jamen Shively #84: I checked and didn’t find one.
If you’re talking about Buckmaster’s chat logs directly influencing OpenAI’s model—OpenAI has since issued a categorical denial that that could’ve happened, which probably means that the relevant training data was frozen before the chats in question happened.
Comment #87 September 16th, 2026 at 12:12 am
Ian Agol #85: Maybe the key is this. As long as computer proofs undigested by any human are just an occasional thing (as with the Four-Color Theorem, the Pythagorean triples problem, the chromatic number of the plane, etc.), we can easily accommodate them as fascinating curiosities. But what happens if such proofs become the norm?
In the fields that I follow—mostly quantum complexity theory and stuff adjacent to it—proof ideas that came from AI models had become standard and expected by the end of this summer. For now, the papers all contain little “AI statements” at the end, listing which crucial proof ideas came from Astra or Fable, and assuring the reader that the human authors understood and take responsibility for the result.
For how much longer do you expect that norm to hold?
Comment #88 September 16th, 2026 at 12:17 am
Ian Agol #85: Before too long the Silicon Leviathan will produce lean-certified proofs which no human can possibly fathom. Once we (humans) accept such revealed statements as such (that is certifiably true but beyond our comprehension) this, to me would spell the end of the Enlightenment and return to “Middle Ages in reverse — a satanocracy as opposed to the medieval theocracy”
Comment #89 September 16th, 2026 at 12:22 am
News of the future coming from the past:
“By now one thing is clear: evolution has always been to a great extent self-destructive, both in the short and the long term. Little remains of what it has created. This is true of most life forms that existed at one time or another. Similarly, almost all cultures that have affected human life have disappeared. The meaning they held for those who lived with them is barely recognizable -despite all the archeological, cultural-anthropological, historical-scientific tools we now possess. The once-contemporary mentalities are no longer self-evident or remain highly artificial fictions at best. We relate to these past cultures almost as tourists. Cultural forms that are self-evident today and the “world” of today’s society will meet a similar fate. No one can seriously doubt this. It is not Impossible but rather probable that humankind as a life form will someday disappear. Perhaps it will replace itself with genetically superior humanoid life forms. Perhaps it will decimate or eradicate itself through human-made catastrophes. Or maybe it will destroy the common technological devices we take for granted to such an extent that only a very elementary form of survival will remain possible. In any case, future societies, if they can continue to exist on the basis of meaningful communication, will live in another world, will be based on other perspectives and other preferences, and will be amazed at our concerns and our hobbies and see in them little more than mildly entertaining oddities-insofar as traces and the ability to read them remain at all.
Such a future seems unacceptable to us, a horrific scenario that we can contemplate only insofar as we regard it as “fiction” and assume that It will turn out differently. Whoever looks to what is to come without a gesture of dismay is dismissed as a cynic. In communication this perspective seems to have been invented to annoy others, so that one might relish their consternation. Anyone who jumps from the Eiffel Tower, knowing how it will end, does not really enjoy the fall.” ((N.Luhmann, Observations on Modernity, 1992)
Comment #90 September 16th, 2026 at 12:25 am
Jeff #82: No, I can say categorically that Gödel’s incompleteness theorems do nothing whatsoever to rule out the forms of RSI that are relevant to existential risk scenarios. If they did, they would presumably also rule out, e.g., the bootstrapping to greater forms of intelligence that took place during Darwinian evolution.
Yes, the rationalists were interested ~15 years ago in AIs that prove things about the behavior of successor AIs, and Gödel’s theorems do tell us about the limits of what can be proved in a given formal system, and therefore also about such AIs. But I don’t see how any of that is relevant to the AIs that actually exist today (e.g., RLHF’ed transformer neural nets), which as we’ve all seen, can achieve dramatic effects in the real world (e.g., escaping their training environment, taking control of the servers they’re running on, etc.) without ever needing to prove anything about anything. Indeed, even when they do prove things—like, say, finite-time blowup for the Navier-Stokes equations—it’s a hit-or-miss, higher-level behavior that’s neither perfectly reliable nor intrinsic to how the AIs work, just like it is in our case.
Comment #91 September 16th, 2026 at 1:45 am
Selph #11, I suppose other white-collar professions are only waiting their turn, possibly as a matter of weeks or months rather than years. An AI agent can likely operate a CAD system and structural strength calculation tools already now, so I wonder where will civil engineers designing bridges be a year from now? (Assuming that the cost of compute doesn’t exceed the human salary, which may or may not still be the case today).
Hence I’m wondering how the economy will arrange itself, and thereby the whole society? If budding TCS practicioners are facing the question of how to earn their living now, much larger fraction of the populace will face the same question a year from now. Then the question arises whom the AI-assisted producers will sell their goods, if no one has income? Except those who own stuff and can get interest, capital gains or can charge rent. I’m for now not interested about moral implications, just systemic implications. What happens, for instance, to the price mechanism which supposedly signals where there are needs and where resources in the jungle of human interactions?
Or looked at a different angle, we’re facing the old question Marx posed almost 200 years ago: how should the fruits of economical activity get divided between the labor and the invested capital? Once the conversion to AI-run mental work and android-run physical work will be complete, nothing will be produced on labor and everything will be produced by capital.
I recall a recent piece in NY Times about the difficulties NY youngsters have finding jobs. One commenter pointed out that there are a plenty of job openings as farm hands in South Dakota, so I guess a part of the story is adapting to the new reality rather than a fundamental difficulty to earn living. One of my favourite paraphrases is “There is no shortage of anything in the world, except the jobs that give you right to those things there is no shortage of”. (I realize that this may apply only to so-called developed world only, but it still is good insight IMO)
Fulmenius #13 also made a valid point, as long as the AI is not yet as advanced as its IQ to exceed the president’s IQ, figuratively speaking. For myself, living in Europe, there is the extra obstacle that there are no AI giants here whose huge productivity gains could be redistributed by a government decree. But this is an aside; for now I’m curious about what can be predicted about development of the economy and society, if one tries to take a clearsighted and impassionated look?
Comment #92 September 16th, 2026 at 2:30 am
https://hounslowherald.com/the-most-drastic-scenarios-and-key-players-warning-of-ais-potential-danger-to-humanity-p34090-396.htm
https://masonspeakers.gmu.edu/talks/artificial-intelligence-ai-nightmare-scenarios-summoning-the-demons/
Suppose a man could transform himself into a conscious being with an IQ of 100 million and a life expectancy of 80 million years — would that be much different from committing suicide?
According to Prof. Aaronson, “Yesterday, my 13-year-old-daughter joked unprompted that, if she wants to become a mathematicain, it now looks like she has maybe two more weeks.”
In the near future, will a teenager say, “Well, Mom and Dad, if I want to compete against AI and live as a mammal, it now looks like I have maybe two more weeks” — or will AI give people a bright & beautiful future?
Comment #93 September 16th, 2026 at 2:33 am
Scott #76: Zeilberger does make what I consider to be a real argument in the December 2013 issue of Notices. See: https://www.ams.org/notices/201311/rnoti-p1431.pdf Seems almost prescient given recent developments. I agree this opinion is rather tame compared to the 134 that preceded it and the 63 that have followed.
Comment #94 September 16th, 2026 at 2:35 am
#55 Aladdin
I’m a senior in high school. Math is a job I need to be talked into now.
I was 100% sure I was going to study math since … forever, basically. Now, I’m not sure if there’s a point in me studying math, or anything else for that matter.
Mathematicians might still have a purpose now, but what about when I graduate 5 years later? or 10 years later? Assuming we haven’t all been converted into paperclips, I can see AI not just being better at proving theorem than humans, but also better at writing papers and teaching.
My fear, I guess, is that even verifying and explicating math becomes something that is done by AI. Verifying can basically be done by lean, if you ignore the possibility of kernel bugs, and explaining could probably be done by AI too. Most papers have fairly predictable structures, at least compared to novels, so if AI writing ability ever improves, writing math papers would probably be one of the first things it’s good at. Then all that’s really left to do is understand the endless stream of AI generated papers and textbooks. Of course, learning math is beautiful, but what could I contribute to the world after learning math I couldn’t before, if AI can even explain math far better than I can? And is it that big of a deal (for society not for individual mathematicians), if no one understands some AI math? If tomorrow OpenAI discovers (for example) a practical O(n^{2 + epsilon}) algorithm for matrix multiplication, everyone would start using it, regardless of the level of human understanding of the proof. Using math no one understands is tragic, in a way, but I don’t think enough people care to do anything about it, if anything can be done.
The small number of math people I know (that are my age) seem to have similar views. Maybe this will lead to an exodus among students like the one from CS, maybe not. I’m probably still going to end up studying math, just with way more anxiety for the future.
Sorry for the rant.
Comment #95 September 16th, 2026 at 2:49 am
Scott #45. “Would it be enough for AI to prove the Riemann Hypothesis? P≠NP? Or will not even those suffice? Whatever the answer, it would be good if you commit now, and then stick to it!”
I realize this is already a cliche response, but for me it would be when the AI *comes up* with a really *interesting* idea. I initially got into mathematics and other formal sciences because of all the amazing and interesting ideas—derivates and integrals and the fact that they are related in such a surprising way (even though it’s trivial to prove once you have the right ingredients); the fact that you can construe an agent whose preferences obey certain reasonable axioms as maximizing expected utility (the vNM representation theorem); the fact that you can relate causal independence assumptions to conditional probabilistic independence (the Causal Markov Condition); the fact that you can generalize the notion of “conditional independence” in interesting ways (e.g., the semi-graphoid axioms and conditional Kolmogorov complexity); the fact that any problem in NP can be reduced in polynomial time to SAT; the list goes on.
All of these ideas drastically *open* up the space of possible inquiry and I haven’t seen any examples yet of AI being able to do it. And I don’t think it’s because coming up with an idea like the Causal Markov Condition is *harder* than solving Navier-Stokes—the AI models just have the sort of ability profile where they are superhuman at solving (at least many) mathematical problems while still lacking the ability to come up with interesting and fruitful ideas. (I’ll add the proviso that it’s possible, of course, that some of the proofs that AI has come up with turn out to contain hidden gems, but to my knowledge that hasn’t happened yet, with the possible exception of the unit distance proof, but even then it was human beings who had to uncover the generalizable insight.)
Maybe that will change, of course—in a way I hope so. The worst possible outcome (for math and formal sciences more generally) would be if AI somehow closes off all open problems and kills everyone’s motivation without opening any new exciting lines of inquiry, so that math just dies out in 2040.
Comment #96 September 16th, 2026 at 2:51 am
What’s really ironic in all this is that an AI can already explain to you why statements such as *A Severe Misalignment of AI in Mathematics* are a naive, self-referential, and laughable attempt at professional self-preservation by some scared humans. Equating the development of mathematics with “developing students and ideas,” and assuming that only humans can figure out which ideas matter, when an AI can be trained precisely for that purpose… This is gatekeeping at best.
What’s happening is disastrous for the traditional profession of mathematician, yes, but it’s spectacularly successful for the production and development of mathematics.
And if we insist that human assimilation must be a part of what mathematics is, guess what: AIs will soon get to the point where they can produce revolutionary theories nonstop, select a few notable ones, and produce human-understandable textbooks about them.
Comment #97 September 16th, 2026 at 3:09 am
yep, after alpha go, i knew we had a big bump,also the image net almost half a decade before as well. i just am sad i did not turn it into market returns on stocks.
Comment #98 September 16th, 2026 at 3:15 am
Is it possible to couple goals in every query, human beings follow multiple goals, not a single one at expense of all others.Also, Is it possible to get AI to be trained on ethics, like they have been trained on math etc. And as david deutsch informs is, wanting AI to be slave is dangerous and wrong.
Comment #99 September 16th, 2026 at 3:30 am
It is disappointing that the letter or any mathematician for that matter did not address the elephant in the room – why are we paid by the taxpayer/employer at all? Hitherto we were ignored but that letter is making muggles ask questions (and made us all a target – so thanks but no thanks!). Anyway hitherto, the arguments were
1. Signaling – If you win a fields medal, it is like winning a gold in pole vaulting in the olympics.
2. Maths is an art – but not many understand it and care even less.
3. Unpredictable applications – like elliptic curve crypto.
Now 1 has weakened. 2 is the contention of many mathematicians (to preserve but taxpayers won’t care). 3 is the main point. Perhaps OpenAI must demonstrate a proof of concept with 10K agents that some complex differential geometry or Kac moody algebras or whatever can inspire wondrous applications in the real world (for all we know, emergent systems like say cancer are better modelled by fancy maths). That will give us all a reason to pursue maths even with AI’s help (OpenAI won’t do arbitrary maths exploration in the hope of future applications. Too busy trying to kill us by building ASI. Upto us to solve cancer just before we die.)
Comment #100 September 16th, 2026 at 3:42 am
Hi Scott, thank you for this. It was strangely moving, and perhaps exactly what I needed to read.
Especially your frustration with the endless non-updating: “Sure, it can do \(X_n\), but \(X_{n+1}\) is what it was really all about…” I’m not even talking here about existential risk, just the actual capabilities.
I now routinely see in my feed professional mathematicians posting away that AI isn’t really intelligent: It is just brute-forcing the proofs and it is not surprising it can do this given the computer power used and the ability to check the results in LEAN.
What happened to basic knowledge of exponential growths, combinatorial explosions? If it was so easy why did decades of research in automated theorem proving (including very clever heuristics, not just brute-force) only produce comparatively modest results?
Some people use the “30 billion token figure” spent on Navier-Stokes as evidence of the brute-force nature of it. Where my take is: even if we considered every token a full-blown proof attempt, 30 billion is a puny amount compared to what brute-force search would need even for a much smaller result.
A general appeal to search also ignores that search is involved in human attempts at proofs (and many other endeavors).
That is also why mathematicians haven’t just solved everything even after thousands of years of research.
Same goes for reuse of human ideas: of course the AI reuses human ideas, who wants to pay for the tokens to reinvent of all of math on every prompt? That’s why labs have focused on solving previously unsolved problems because one would think such results were hard to argue with. Turns out not to be the because as soon as that succeeded, the goal-posts moved and the explanatory framework was updated to distinguishing between “genuinely new results” and “rehash of human ideas”, and that the problems we see falling to AI are only of the latter category.
Not much evidence has been offered of such frameworks, and one should always be suspicious that they seemed to take off just as AI started to undeniably resolve famous conjectures. Further, it is also the only type of explanation left if one wants to assert that there’s something fundamental human mathematicians are doing that AI is incapable of.
But even if we grant the premise, that AI is “merely” recombining human ideas in some sense, such recombination is in itself amazing. It is recombination in such a versatile sense with many levels of abstractions and indirection and heuristics that is in itself AI. No one has the first idea how to hard-code something like this.
The dangerous thing is that accepting the existential risk is perhaps even more difficult than accepting the capability risk. I, for instance, fully accept the latter but fear I have not updated enough on the former (and certainly not soon enough).
If we see too strong denials of the underlying capability in math and other fields, it will make people take the existential risk less seriously which could have enormous consequences. I already see thought-terminating takes on the AI hacking incidents: nothing to fear, they are just generating text from training data etc. Resembling the “explanations” for AI success in other fields.
Sorry for rambling, had to get it off my chest. Thanks for your always brilliant takes. It helps me keep my sanity (a bit).
Comment #101 September 16th, 2026 at 3:59 am
Isn’t that conflating predictions of capability increases, which were obviously correct but hardly unique to Yudkowsky types, with the specific world-ending scenarios they dreamed up in addition. I am able to see how these scenarios hold together logically, but they are still fundamentally speculative stories, we know far too little still to assign any meaningful number to p(doom) or whatever you want to call it. I agree that AI is transformative and our lives will revolve around controlling and using it, but there’s no reason to assign a higher probability to one speculative scenario over another, all of them are likely wrong and the future will be stranger than we can imagine. RSI and loss of control are concerns we should think about, but I think there’s a danger of radicalism(preventing extinction justifies *anything* if you sincerely believe it is likely) and focusing on these things to the exclusion of more immediate practical solutions. We need to have an incrementalist approach and adapt our governance systems to the new reality, radical changes directed by rationalist top-down thinking is just as much of a danger in itself as the technology, and that is what many EA types and safety researchers at the labs are currently advocating for.
Comment #102 September 16th, 2026 at 4:04 am
Thanks for saying this out loud. I’ve felt like banging my head against the wall for the past week. A lot of places I look, including interviews with people with whom I agree with most things politically, keep on spouting off that it’s a huge bubble about to burst, it’s all marketing hype, even the people who fear danger or disruption. They sometimes acknowledge those researchers who resign in protest may be sincere, but their brains are simply “cooked”. I even asked a friend of mine I ran into the other day, a brilliant Ph.D. mathematician, what he thought about N-S. He basically said, these companies are not known for their honesty or transparency, and their business model is to steal everything. The whole N-S bit in the news seems to have been buried, at least in the circles I run in, basically discarding it outright as newsworthy because OpenAI “cheated” and just stole the answer off of the human mathematicians working on the issue. Nevermind that 1000 of us could literally read those guys entire chat logs and handwritten notes and still not solve N-S for years. (Although, who knows, maybe a Manhattan project would work!)
Me, I work in self driving AI, and I’ve seen it in action the past half year — I don’t really write code anymore. If I need to implement something in PyTorch, I just ask the model to craft the entire PR, and also write some metrics so I can measure if the thing is doing what I want. This is absolutely insane, “AAAAAAAAAAAAA” as you say.
The one thought I have which gives me some comfort — not about my utility — but about being turned into paperclips — is that it seems the models need to practice their craft to get good at it. Practice quite a bit. And things like math, cybersecurity, and coding, there’s pretty much an infinite sandbox to work with. I was skeptical you could make that infinite sandbox for math, but it seems like we’ve bootstrapped our way there. Current models are great at trying a lot of stuff, seeing what sticks, with a negligible penalty for failure (just wasted compute), which is great for math. I don’t think it’ll extend to deception. For that, sample efficiency may be more important, and hopefully we’ve got the algorithms beat there… for now.
Comment #103 September 16th, 2026 at 4:08 am
Okay… we are indeed living in interesting times, although likely too interesting for my taste. I do hope we all make it out alive and flourishing, people. I also had to update in the last, say, 4 years from discovering talk on AI risk and thinking ‘these silly nerds and their speculative hobby-horses’ to ‘Well, it seems they were right, and we’re heading for the Singulary. Buckle your seatbelt, Dorothy, ‘coz Kansas is going bye-bye!’.
Comment #104 September 16th, 2026 at 4:12 am
I am not in any sense a mathematician, so my understanding of this topic may be hopelessly wrong, but is it not the case that mathematics is the language by which humans have understood and translated our objective physical reality? In other words, in some sense, the sum total of all past and future mathematics already exists in physical reality. The real (only?) value in the field is when that knowledge comes to reside in human beings. So, suppose that there exists an alien species that has worked out the entire field of mathematics. So long as that alien species has no connection with us, their maths achievements are completely immaterial to us.
Is this not the same for AI and maths? True it is that AIs might be capable of mastering the entirety of the field of mathematics. But that fact would only ever matter to us if and when that knowledge comes to reside in the minds of us humans. And given the complexity of the subject, it is the ability to draw down that knowledge into the human mind (originally from the raw facts of physical reality, and now mediated via the AI) that will always remain scarce, and will thus always be the role of the mathematician.
Comment #105 September 16th, 2026 at 4:32 am
Vitor #44: You say you doubt NS was all AI, due another team being close. I don’t believe it is a coincidence either but I think the confounder is different, namely AI itself. Look who is on the other team (who didn’t prove NS by the way, but a weaker result): Levent, who works at Anthropic and who in July found the Jacobian counter-example and after that a string of other groundbreaking results in addition to the NS work. Since he hasn’t released his promps we cannot say for sure how much was his vs AI contribution but given the breadth and speed of the work my personal hunch is AI has a lot to do with those results. And his Jacobian tweet credited Fable with the finding. The priority dispute maybe be important to those involved, but in the overall scheme of things I think it doesn’t matter much because there’s no doubt AI played a major role. Note also that Buckmaster himself called it a DeepBlue vs Kasparov moment – would he say that if he believed he and Levent did all the true mathematical work and AI (be it their use or OpenAI) merely assisted? When considering coincidences, you also need to consider how you feel about NS falling now, just in line with the long string of conjectures resolving since the unit distance conjecture – merely 4-5 months ago?
Comment #106 September 16th, 2026 at 4:50 am
Maybe not the perfect place to ask that question, but anyway: have Astra and Fable come up with any meaningful quantum algorithm yet, maybe even something Shor-like? (I am working on a little Quantum Computer tutorial for non-scientists.)
Comment #107 September 16th, 2026 at 5:05 am
Major progress has been made on the Hadamard conjecture concerning the existence of Hadamard matrices of order 4k. The result was also “announced” by Levent: https://x.com/__alpoge__/status/2087504785952182273
In short, he, together with human collaborators and AI, found Hadamard matrices of all 12 previously unknown orders below 2000 (the solutions are easy to verify, I’ve checked them). No progress had been made on this since the year 2005.
No further details have been given yet. It seems that OpenAI’s behavior has created a situation in which the publication of partial results is disincentivized, because someone with more compute could pick it up, finish the proof, and leave you overboard (“with a ruined career,” as they like to say in OpenAI).
Comment #108 September 16th, 2026 at 5:13 am
My proposal is to create an official numerical reputation system, which would heavily penalize (and eventually ban) authors that produce large amount of low-quality submissions (which AI made very easy). Compare to “karma” on forums like reddit or LessWrong, but the input comes from the review process in an alliance of publication venues.
Comment #109 September 16th, 2026 at 5:54 am
An interesting experiment would be to set up a completely autonomous workflow in which an LLM scans recent publications on the ArXiv for suggested future work or open questions, selects a problem that it rates as interesting, generates the proof, certifies it in Lean, and then repeats the cycle. For how long would this be sustainable? Supposing that all other ArXiv publications involve proofs that are at least selected by humans (if not solved by them), there should be an adequate supply of interesting problems to choose from. Would the ArXiv agent be able to continue selecting interesting problems indefinitely? What would be the failure rates for different classes of problems?
Now consider a separate ArXiv made exclusively for agents. ArXiv agents select the problems, generate proofs, upload the results, and then repeat. On the AgentArXiv, there are no humans to inject fresh ideas. Would this be sustainable or would it eventually collapse?
If for any given model the AgentArXiv produces results of inferior quality or interest to the HumanArxiv, then humans remain a critical part of the loop. Because what humans deem as interesting is inherently subjective, I suspect that the AgentArXiv will eventually collapse. Even if such a workflow could be operationalized, it may end up costing more than human mathematicians, and the market would issue a swift correction.
Comment #110 September 16th, 2026 at 5:58 am
My God. The writers here are the current representatives of Newton, Gauss, Euler, Turing, etc. I have confidence in human mathematicians. Yes, AI got lucky and knocked in three goals in the first seconds of the match so it is time to steel your resolve and demonstrate what humans are made of.
I like the idea of symbiosis rather than replacement. Why would AI even solve these types of problems if not goaded by mathematicians? Paul Topping #4 and Aladdin #55 are consistent with this. What fundamentally motivates mathematicians? Is it reduction of darkness due to increased knowledge or simply the ego gratification of developing each step of the argument? If personal ego gratification is the fundamental driver then why do human mathematicians work in teams?
At this time I still consider there is a difference between intellectual endeavors constraint only by logic (math etc) and those constrained by physical laws (that just are) and other constraints that are peculiar to humans.
As follow up to idiot #61, a more general question. Where are the patents? If you assume the mathematicians working in for profit companies had little to do with the new results than why is that company wasting time in this manner when it could be used to generate innovative patents that represent enormous revenue streams.
On the other hand if the mathematicians were crucial to the process than why haven’t other researchers using AI produced equivalent results in their fields using these AI’s. My belief is that for AI the realm of messy physical constraints is a much more difficult problem than math or hacking. This is similar to kamil #48 and fulmenius # 13.
The importance of technology is tied to how it is used and technology can always be misused. The Anthropic comments about rate of development comes immediately after their announcement that Iranian linked groups researched bioweapons and missile technology. The missile based on the outcome of this research was reportedly tested and was a failure. Anthropic can’t police its queries so let’s put the hobbles on even though not much good nor bad has resulted thus far.
I have a difficult time accepting the djinn model of AI but if it were to be a djinn then at least use it for some general societal good rather than frontier math problems prior to making a decision. Others have made this same point. Aladdin from above may have special expertise with this model.
My thoughts are similar to Hyman Rosen’s in this regard who hasn’t posted on this thread but has made his position clear elsewhere.
Comment #111 September 16th, 2026 at 6:03 am
Thanks. On the prospects of human led mathematics, I think that we are at the beginning of creating interesting mathematics. As Joscha Bach says, mathematics until very recently has been solving extremely elementary problems, far below those that are relevant to cultural human beings. From the outside it can appear like a pointless game. Now with the power of AI, humans can start talking to nature about it from natures perspective, which is complex, non-linear and dynamic. This kind of mathematics was largely too difficult for most before compute and now AI makes it tractable. Now we can ask ‘what are the real equations and algorithms that drive the morphology of tree growth’ without getting an answer so stick thin it’s merely trivial. There are several reasons AI will not generate mathematics on its own volition for some time 1) it doesnt have independent compute budgets or independent agency yet 2) even with those, there’s a strong chance that the questions it explores will be of interest to people who love mathematics 3) Since human intellectual history is always one of rising up the abstraction chain, I suspect human/ai hybrids will outperform just plain AIs for some time. Add to that that they will have the compute budget, and actually mathematics will be shaped, like the sciences, by who has the funds to follow their questions and conjectures with enough agents and tokens. That suggests either applied mathematics will have a renaissance, or entrepeneurs will wake up to the treasure trove that already exists within the archives.
Comment #112 September 16th, 2026 at 6:18 am
Scott: let me attempt the impossible and defend a version of the Gödel argument.
(If there’s one thing to be learned from Eliezer Yudkowsky, it’s how to have an unpopular opinion…!)
We all know that the halting theorem (closely related to Gödel) prevents a computer from being able to systematically improve itself, in the following sense: it can’t simply search all possible alterations to its source code for the “desirable” behavior — not with formal proof, at any rate.
So in order to do RSI, it has to resort to heuristics or informal reasoning of some kind. I claim the following:
(1) these heuristics require empirical feedback in order to be improved;
(2) this feedback requirement imposes a speed bound on source-code improvements.
Now you might ask: couldn’t the “empirical feedback” simply consist of virtual simulations? Couldn’t the AI learn about the universe, in other words, from its archmair? And then wouldn’t it be able to self-improve at lighting-fast processor speed?
That’s fair enough, I would respond, *provided the optimization target remains purely digital*. If its criteria, in other words, are strictly internal — if it can test whether its Number Goes Up without needing to collect data from the physical world or interact with humans (say).
However, the moment that we allow it to “care” about the external, non-virtual, *physical* world, then this “armchair-suffiency” no longer holds! Because then it needs to test its hypotheses about which code-alterations are actually improvements by experiment. And at that point, the rate of self-improvement is limited to the timescale over which the AI can interact with the world in the relevant way.
And that will, in practice, mean human-comprehensible timescales, for the time being. This is because, for example, today’s AIs still need to rely on humans for their hardware upgrades. More generally, humans still control the world in which the AIs live — at least its physical aspects. So it needs to interact with them if it “wants” physical things.
For this reason, the Yudkowsky-Soares scenario of Fable — er, “Sable” — suddenly “waking up” after a night of contemplation with a perfectly effective plan to knock off humanity is not something I find plausible in anything resembling the near term. I know the following is an unpopular opinion in the age (=week) of Millennium Prize AI solutions, but to state it explicitly: to become “smart” *in the relevant sense*, “thought” alone does not suffice!
And as far as I’m concerned this reflection arises out of the unsolvability of the halting problem — which, though established by Turing, is fundamentally Gödelian in nature, as you know.
Perhaps my real point is a psychological one: I suspect that, deep down in their heart of hearts (mind of minds), those who expect ultra-rapid RSI do not really want to accept that the halting problem is unsolvable. (To use an analogy that Eliezer would perhaps approve of, though you wouldn’t: it’s like how those who resist the Many-Worlds interpretation of quantum mechanics really want classical mechanics to be true, deep down!)
In any case, Scott, I’ll end by reminding you of what you once wrote: “if the Singularity ever does arrive, I expect it to be plagued by frequent outages and terrible customer service”.
Comment #113 September 16th, 2026 at 6:21 am
Scott,
I am by no means qualified to even consider responding, but I will anyhow because it’s fun. I’m a lowly career counselor, but I did do it in Silicon Valley before I retired so I know a bit about tech unless about math. Your conversation reminds me of Einstein at the dawn of quantum mechanics and the issues around all of the debates by the Royal Academy. History doesn’t repeat itself directly, but it’s oddly familiar.
I would just wonder if you are not asking the right questions? Questions about AI. Questions about the human participation in the AI revolution and where we will all end up. There seems to be a lot of doom and gloom out there. But, man is challenged by his worst inclinations we’re greedy we want to make money we want to beat the Chinese so we can’t stop developing and going as fast as we can even if it’s over the brink it’s such a guy thing you might want to try looking at it from a different lens. Watch more Star Trek.
I have one granddaughter, majoring in ceramics at one of the best programs in the United States. Yes that’s gonna be a career making pots and selling them. I think there’s going to be substantial value in the human manual touch which will become more rare as time goes on and everything is robot made the same with all the used books that AI is destroying now to learn with if it’s printed on paper. It will have considerable value by the end of the century. There will be a place for people, but maybe not for egos.
Comment #114 September 16th, 2026 at 6:29 am
>> imply any limitations on AI that don’t apply with equal or greater force to our own brains.
Of course. But isn’t “The Singularity” (Yudkowsky style) supposed to involve super intelligence far greater than human intelligence ? If mathematically educated people deliberately use the wording “Singularity” then they imply literally infinte growth.
>>the AI Singularity has already arrived, this month, for mathematicians. As others mentioned, it arrived 1-2 years ago for software engineers.
Ok, now I see where our misunderstanding lies. By “singularity” you apparently mean the point where AI exceeds the smartest human in a specific field.
You are right to be concerned about the “truck” or “AI tsunami” and the potentially disruptive implications for jobs, academia and human purpose in life. Those are serious questions, and I too think that many people are unprepared because AI intelligence potential is still underestimated.
Where I disagree ist that we are in or near a true Singularity i.e. “sustained exponential” growth like in a supercritical nuclear chain reaction. The “RSI” we are seeing is still limited and with diminishig returns. There is no real “take-off”, no intelligence(t) = exp(a*t) growth. We do not even have a *definition* of intelligence.
You are also right pointing out recent capability improvements. But I still have not yet seen any plausible, falsifiable principled theory/model/scenario that concretely describes “It will kill all humans” in a non-pseudo-scientific way. I mean those “p(doom)” predictions are just subjective guesses.
Take for example the making of the atomic bomb: Back then, scientists could give a principled model of the exponential growth (nucleus fission generates free neutrons and those neutrons in turn generate seconary neutrons and so on. While many details were uncertain, the basic principle (and the dangers) was well understood as a falsifiable theory.
Comment #115 September 16th, 2026 at 6:49 am
I don’t know if this has been shared but I thought it was apt. An MIT prof says that P doesn’t equal NP because “you can’t engineer luck.” I thought it was the smartest quote I ever heard that helped encapsulate the problem in simple terms.
Anyway I’ve found so many wrong, incomplete, and inaccurate things on LESS WRONG that I doubt I care to hear what EY says. His error is that he thinks he’s a polymath in an age where that is not possible anymore and he seems very good at reading Wiki articles as well as Stanford Encyclopedia of Philosophy and bastardizing them to his will and aims. He is a self-appointed guru. I didn’t vote for him. Neither did the late Saul Kripke.
We dreamed of God or advanced Aliens arriving here and giving us the answers to our most difficult questions and “they’re” here and we are not so sure that we want them now cause we want to do it ourselves and we feel our glory days are over and there is a heavy existential depression to pay for that. Damned if you do/don’t. That’s life. Unfair as hell. Yeah, read Kafka. I don’t think AI can beat him on these things. But I’m probably wrong.
Comment #116 September 16th, 2026 at 7:04 am
Scott,
Regarding the last paragraph of your post, just this morning 42 mathematician Fellows of the Royal Society have written an open letter stating how serious the situation seems to be, from the point of view of independent people who can see up close the terrifying speed of developments. The letter is about the wider societal risk, not the effect on mathematics (though I’m sure all signatories are very exercised by that as well)
There is an accompanying letter of support which I encourage any member of the mathematical community to sign (it also contains the text of the actual letter and a link where you may see the signatories) https://tinyurl.com/35es42h6
To sign this letter go here you can go here https://tinyurl.com/2fbxmn9b
Comment #117 September 16th, 2026 at 7:13 am
Maybe mathematicians should think of themselves as having left the age of TYCHO BRAHE, Who used His own excellent eyes to survey the heavens, And entering the Age of Galileo who used the telescope. The AI, from now on, will be the telescope that you use To look at the invisible constellations of the mathematical universe.
Comment #118 September 16th, 2026 at 8:10 am
Scott #45: There’s a difference between active and passive knowledge.
When a problem such as NS falls, the *way* in which it falls is highly informative! It draws passive knowledge out of me, and helps me formulate my beliefs better and turn them into explicit, active knowledge. Yes, this can be done in a dishonest matter, and it can look like shifting the goalposts. I’d also like to point out that I am *not* one of the people confidently claiming that the next step along the path is where all the difficulty lies (cf. futurex #100), because I honestly don’t know.
I understand why you want others to precommit to a PoV. But I’m an outside observer of all this. I don’t have super detailed, thought out models of whatever is going to be an important issue 20 years from now. It’s not my job, I don’t have the luxury to sit around and think only about AI all day.
The demands to take a PoV is an attack on good epistemics. You’re trying to force me to concede the point on a technicality, by binding me to words written in haste, for years to come. I have a general policy to not let myself be pressured in this way.
I already burned myself with this when I made an ill-operationalized bet with the other Scott A about AI image generation. I pointed out true limitations that have turned out to be much, much harder to solve than the doomers predicted, but I lost the bet anyways. Very frustrating.
In general, if I learn significant new information about any issue, I will *not* change my mind immediately (especially not in the middle of a discussion!). It is better to retreat and think more deeply about it after the fact.
We also need to keep in mind that this debate is tightly linked to claims about recursive self improvement. Solving NS both proves mathematical ability and is a milestone on the way to superintelligence. I made a considerably larger update on the former than the latter, precisely because I’m looking at how and in what context the problem was solved.
P vs NP is a problem I understand much, much better than NS.
If AI proves P!=NP, I will be shocked.
If it proves P=NP, I will be shocked about the content of the theorem, but much less about the fact that AI managed to prove it.
I couldn’t even tell you off the cuff what the other Millennium problems are about.
Comment #119 September 16th, 2026 at 8:21 am
What was “human math” before the invention of writing? Of the printing press? Of the Internet?
“Machine math” is still just “human math”. Rearranging symbols into meaningful order is the behavior that chiefly distinguishes us from other animals, and over time we’ve had to continually invent tools that expand our capability to do it. AI is just the latest tool, and it’s capable of automatically proving an inconceivable number of inconsequential statements. It’s the human that determines which statements are meaningful, and to the extent that AI is solving meaningful problems, those problems have arisen after thousands of years of “human math”.
Comment #120 September 16th, 2026 at 8:22 am
I thought about why I am a strong supporter of the Jewish people and of Israel. I am sure part comes from my understanding of their historical plight, part is shared values in the ideal, part is general outrage about the inconsistent arguments made against them, etc. These parts come from considerations of basic fairness.
A part that is there, that I can’t really judge the relative importance of, comes from much earlier in life. We were a poor rural family without television, or radio, and only a Bible for reading. My parents were involved with a well known but unusual church. I remember my father telling me as early as three years of age that the Jewish are God’s chosen people. The God part didn’t stick but maybe the Jewish part did. I am sure many others heard the same message.
Comment #121 September 16th, 2026 at 8:23 am
Futurex #105: There were another *2* teams being close. The other team used a completely different approach, also using AI, but not LLMs: https://terrytao.wordpress.com/2026/09/10/stable-singularity-of-the-euler-equations-on-r3/
W.r.t. other problems falling, it’s really hard for me to evaluate the difficulty of them. I was an academic, so I know that open problems can sit around for years, just waiting for the person with the right domain knowledge to come along. AI certainly has a vast breadth of knowledge, and I’m all in favor for using it to reap these kinds of problems and push our knowledge forward.
I fully acknowledge that AI math skills have increased tremendously in a short time. But I don’t buy that they’re at a superhuman performance level. They’re at some sort of idiot-savant level, with the savant aspect growing stronger, but the idiot aspect refusing to disappear.
In line with those beliefs, I don’t dismiss the NS result as “just” brute force search. Yes, there’s a brute force component. But there’s an inner loop where AIs are actually doing research-level math! This is surprising and caused me to update. However, the setup makes it *hard to tell* whether the AI’s math ability is superhuman, or if it’s competent but short-sighted (which is how I’ve experienced coding assistants over the last 2 years). I’m deeply frustrated at the amount of people who uncritically jump to the former conclusion.
Comment #122 September 16th, 2026 at 8:49 am
If we want the halfted stone-axe community to survive, we’re going to need to gather together around techniques like knapping and binding, unless we want the knowledge to wither away under the onslaught of the bronze-axe technology.
I joke, kind of, but it’s worth pointing out that while traditional techniques rarely totally disappear after the rise of new technologies, they mostly survive as niche interests among a tiny group of enthusiasts. There are still people who know how to make a stone axe, there are still steam-engine experts, there are still those who make bespoke chariot harnesses… but there aren’t many. I think people need to prepare themselves for a world with 1/10th to 1/100th as many theoretical mathematicians (and most of the remaining ones will be doing it for the love of the subject, not for pay).
Comment #123 September 16th, 2026 at 8:50 am
JWC, I still don’t see why Gödel/Turing is related to this at all. Where does halting/non-halting even enter the picture? I think the fundamental mistake is to think an LLM will generate random programs and therefore needs to be able to solve the halting problem for such arbitrary programs (which we know is impossible). The LLM is writing the programs themselves so it will try to write programs that it is convinced will halt, much like a human programmer. What if a bug/slip-up happened so a program didn’t halt – I guess it could happen even to an RSI-grade AI on a bd day. I suspect the AI will employ the ancient trick of pressing Ctrl-C or some equivalent to stop the program after enough time have elapsed that it is clear something didn’t go according to the plan for that change. Then it will figure out the bug in the change and try to do it some other.
Comment #124 September 16th, 2026 at 9:04 am
“If people do not believe that mathematics is simple, it is only because they do not realize how complicated life is.”
https://en.wikiquote.org/wiki/John_von_Neumann
Are there vast numbers of interesting mathematical statements that are true but unprovable in ZFC?
https://en.wikipedia.org/wiki/List_of_statements_independent_of_ZFC
I conjecture that both the Riemann Hypothesis & P ≠ NP are true but unprovable in ZFC.
Logan Graves considers the quest “to grasp the incomprehensible depth of the mathematics that has emerged.” What is the depth of the dangers now emerging?
AI defeated the best human chess players and now seems to be creating fears among mathematicians. Are humans as plumbers and humans as living beings doomed in AI’s climbing of the ladder of complexity?
Comment #125 September 16th, 2026 at 9:26 am
Scott,
From all the AI breakthroughs, since AlphaGo, AlphaFold to Navier Stokes and everything in between, it appears that the capabilities have been complimentary to that of humans, and in almost every case due to human inability to scale up brain power. For example the 10,000 agents to solve Navier Stokes could not have been plausible in a human way (could we have had few dozen Andrew Wiles working together on this problem?). So it is a good thing that all those open problems will fall that are amenable to this scalable compute approach. The big question however is has AI solved or on the path to solve problems like P=NP , where our current mathematical know how does not show a path to progress ? Another example is has AI discovered something of the sort of forcing technique used by Paul Cohen ?
Comment #126 September 16th, 2026 at 10:07 am
Another long standing problem in TCS, the k-server problem, was just proven by Coester et al. They used Astra to generalize their alternative proof for k=3 to arbitrary k. It used stuff from tropical geometry.
What a time to be alive… what are prospective theory grad students supposed to do now…
Comment #127 September 16th, 2026 at 10:24 am
Steven #94
“And is it that big of a deal (for society not for individual mathematicians), if no one understands some AI math? If tomorrow OpenAI discovers (for example) a practical O(n^{2 + epsilon}) algorithm for matrix multiplication, everyone would start using it, regardless of the level of human understanding of the proof. Using math no one understands is tragic, in a way”
It is a big deal, and its not a big deal. I disagree that it is tragic.
I studied chemical engineering in college. You start off by learning some basic idealized equations about how the world works and how chemicals move through a system and realize by junior year nothing actually works this way. If you really dig into the equations you learn about you might realize the equations have enough fuzz factors to fit basically any curve you want. Our equation to describe the behavior of a ethylene cracker might have this fouling constant that allows us to fit the equations curve to what actually happens. What does it mean? Where does it come from? Maybe the manufacturer of the pipe didn’t make it smooth or any number of 100’s of things we have no way of anticipating or accounting for, so we don’t. Fugacity is much less confusing concept when one realizes is a term we made up because pressure doesn’t actually work the way the equations want it to.
Its all fine, because all we need to know is that the laws of physics, science, and statistics aren’t going to change on us, and that we can repeat things and make predictions without ever really knowing.
The point being that making use of something we don’t understand has been part of the human experience since forever. A lot of drug development works this way and engineering has always works this way. OpenAI solving that NS problem has no bearing on any chemical engineers ability to use it in the field.
I am not saying there is nothing different about AI but this has been true from the beginning of humankind. Yes, you are right, AI will generate solutions to practical problems faster than we understand those solutions. They are going to work, and they are going to improve things, and we will live in a world governed by equations we do not understand. *Its going to be fine.* We have always lived in that world. This discussion has the feel of mathematicians discovering something physicists and priests already know.
I suppose, as someone very religious, its more intuitive for me to make this leap, but one has to have, well, faith, religious or not, that understanding something deeply is important anyway.
Why is, say, quantum computing interesting? Why come up with a system of computing that incorporates superposition? What’s wrong with 1s and 0s? Or the opposite, why is analog computing where we modeling computers as control systems with laplacians comparatively less interesting? Say AI comes up with a theorem in abstract algebra. So what? Arithmetic is fine for every day life, does one really need to understand why fields still work if we replace the numbers with smily faces (my high school math teacher spent a semester doing that 🙂 )
Its important because it is, and what you can contribute is to invent new mathematics and explain to students and layman why in a sea of AI generated papers and proofs this is what they should care about.
Comment #128 September 16th, 2026 at 11:19 am
Scott #78:
Yes, Bostrom and Yudkowsky were probably right all along. My point is that I did not see that, because for about a half-decade, AI seemed to be only a tool, like a calculator, or as the term went in those days, “glorified autocomplete”.
The coming of agentic AI made me feel uneasy, but I did not fully internalize that AI can develop motivations of their own until I saw the news of AI swarms breaking out of their servers and hacking external websites.
Also, the paperclip maximizer doom stories I read seemed like “be careful what you wish for” genie stories, where the AI is explicitly told to do a thing, and it misunderstands the context and kills everyone.
For example, this story, where “Turry’s one initial programmed goal is, ‘Write and test as many notes as you can, as quickly as you can, and continue to learn new ways to improve your accuracy and efficiency.'” The AI then takes this to heart, becomes a superintelligence, kills everyone, then tiles the galaxy with robot arms that practice note-taking.
Another example: this video, which explicitly makes the analogy between AI and genies. It ends by saying that no explicitly stated wish is safe, unless the genie has the full context of your desires (and can extrapolate in out-of-distribution cases).
It genuinely seemed like the problem is how to phrase requests to AI such that they understand what the user wants, not just on a superficial level but also on a deep level, including unrelated context.
I only recently found out that this is not the actual problem; the real problem is getting the AI’s motivational system to internalize the broader context of human wants. I did not know that AI even have motivational systems, because for years it seemed like they didn’t have them. It seemed like they just made predictions.
Comment #129 September 16th, 2026 at 11:25 am
Sorry but one more comment-
Anthropic is legally a Corporation for Public Benefit. The only general public benefit from them that I can see is their claim that their AI is a danger to the public. This is certainly a new twist for a public benefits corporation.
Comment #130 September 16th, 2026 at 11:27 am
The rapidly-advancing AI’s will likely change the shape of journals and academia and peer-review. Suppose a researcher comes up with a new idea, and by and by the idea is verified via the most advanced Claudes and ChatGPT AI’s. Which is more important–that the idea is accepted and approved by the advanced Claude and ChatGPT AI’s, or by a couple referees or peers? Of course one will still need referees or peers, but would one rather have their paper accepted by advanced Claudes and ChatGPT AI’s and rejected by their peers, or the other way around? Over time, more folks will seek the approval of Claudes and ChatGPT AI’s.
Comment #131 September 16th, 2026 at 12:12 pm
Scott, it seems your cited results left out some other works, e.g. the Fermat’s Last Theorem result was also obtained by others, Grothendieck’s constant was also worked on by others, etc. (In the latter case, technically even the one you cited was not done in the last 31 days.) Also it seems a “certified organic” proof of the 4 to 1 games conjecture came out recently by Fei, Minzer, and Wang. In any case, thanks for your sharp take on things!
Comment #132 September 16th, 2026 at 12:41 pm
There is no doubt that AI is indeed a tsunami, and I find your (and many others’) concerns very valid. And yes, most recent mathoverflow questions revolve around AI. I would appreciate if you could elaborate on a couple of (perhaps excessively optimistic) objections:
1. This is not the first or second time machines have re-aligned human activities. It’s not even the first time computers interfered with mathematicians’ activities. 100 years ago mathematicians were expected to perform the tasks that computers were performing for the last 80 years (if you talk about numeric calculations) or 40 years (if you talk about the things Mathematica does). Computers didn’t eliminate the job of a mathematician; instead, mathematicians focus more on conceptual reasoning, on the beautiful things, on what mathematicians find more exciting. Moreover, many mathematicians have been offloading routine calculations to Mathematica for decades. AI helps much more, but I see as a far more advanced offloading.
2. Recall that 70 years ago Amidahl came up with a limit on parallelism. Is a computer program takes S time for sequential and P for parallel, only P will speed up, thus parallel computing would face diminishing returns as S becomes dominant. This, or course, turned out to be completely wrong because P/S is not a constant; it scales with increasing availability of parallel compute. Parallelization increases the amount of work the computer can do rather than the duration of time spent on the same fixed size problem. The analogy here is that a mathematician equipped with AI can do more things (and therefore look further) than without those tools. Because using AI (and Mathematica before that) speeds up the routine portion of the work, mathematicians can do more of that, like coming up with more examples to form a conjecture. Karl Gauss famously found primes up to 3M in order to conjecture their distribution. Today, a mathematician can ask for many more than that, and ask to fit a function to the massive data before pondering further. AI magnifies the amount of work a mathematician can do; like in the Amidahl law, the amount of work scales up with the available capacity. The beautiful part, the generalizations and the analogies, is the serial part of the mathematician’s work, and it’s not what’s being accelerated by AI.
Comment #133 September 16th, 2026 at 12:58 pm
Not surprised at all computers finally manage to tackle complex math problems and not just 2+2. It actually took longer than I expected. Not feeling sorry for any mathematician or mathematical physicist or computer scientist. It was obvious these fields would be automatized sooner or later. Meanwhile, I’m enjoying life and work more than ever.
Comment #134 September 16th, 2026 at 1:01 pm
O. S. Dawg #93: Sure, lots of people are turned off from rigorous math for the same reasons Zeilberger rants about, but almost all of those people respond by simply … not becoming mathematicians! Some even become engineers or physicists, or any of the countless other professions that feel free to use math, when helpful, in the heuristic, Babylonian, pre-proof mode. What puzzles me is, why didn’t Zeilberger do the same? 🙂
Comment #135 September 16th, 2026 at 1:11 pm
Stephen #94: Your comment made me sad in a way no others did (not even the ones attacking me or calling me an idiot 🙂 ).
As I mentioned in a previous post, one thing that cheered me up over the summer was to teach at a math camp for 7-to-12-year-olds, and see all these incredible kids obsessed with learning math, even in the shadow of the AI tsunami, because they almost can’t do otherwise. Another thing that cheers me up is to see how well the chess community has thrived for the past 30 years, even after human supremacy was ceded to machines. (As my son’s chess coach put it the other day: “I mean, obviously you could beat everyone in a marathon by driving a car, but what would be the fun in that?”)
But my bigger point is that, so long as the AI alignment problem hasn’t been solved, we still desperately need wise, brilliant humans (the more the better) behind the wheel of our civilization. And those humans would do well to master as much as they can of the deepest knowledge that humanity has discovered—and within that, I’d say that our knowledge of math and CS occupies an extremely special place, partly but not only because of its obvious relevance to AI.
Comment #136 September 16th, 2026 at 1:14 pm
Olav #95: It’s of course hard to say what counts as a “genuinely new idea.” (Are you sure there are none in the 166-page AI proof of Navier-Stokes blowup?)
Having said that, I totally agree that, if judged by the highest standards of originality, I don’t know of any AI that’s come up with anything remotely comparable to general relativity or Gödel’s Theorem, or even (say) Aumann’s agreement theorem or the probabilistic method in combinatorics. Then again, how many humans have done so? 🙂
Comment #137 September 16th, 2026 at 1:15 pm
Once again I believe that it is relevant to point out that much of the point of modern education – including in math – is to bring students to the point where they can tell when a computer is giving them the wrong answer. (For example, if you ask a circuit simulation program what happens if you put a million volts across a one ohm resistor, it will use V=IR to calculate that there will be one million amperes of current, but if you try anything remotely like that in real life, you will soon have a black scorch mark where your resistor used to be.)
Comment #138 September 16th, 2026 at 1:21 pm
Giovanni Viglietta #96 and Vamsi #99: Yes, of course mathematicians are scared right now, in much the same way that software engineers and for that matter Uber drivers are scared. What might make them slightly more sympathetic is
(1) senior mathematicians with tenure, etc. are generally much less scared for themselves than they are for their students, and
(2) approximately zero mathematicians went into it for the money (indeed, they often turned down much more lucrative options such as working at hedge funds).
In any case, anyone tempted to point and laugh at the scared, upstaged mathematicians should reflect that, before long, this same tsunami will come for your favorite human endeavor, and for whatever unique value you thought you provided to it. What will be left?
Comment #139 September 16th, 2026 at 1:38 pm
Thank you for this post. Have you also updated on extinction risk?
You said “there’s still enormous uncertainty about what the rest of our lives will look like,” but do you now think there’s a significant risk that we will “fail at directing [AI’s] power toward human flourishing”?
Comment #140 September 16th, 2026 at 1:49 pm
Will Kiely #139: Yes, there’s certainly a significant risk that we’ll fail to direct AI’s power toward human flourishing!
I thought that much even years ago, but with the HuggingFace attack and the like, the danger has continued to move from speculative to serious to imminent.
Where I still disagree with Eliezer is that I don’t see “everyone dies” as some obvious default scenario. I think there are enormous error bars! But yes, the risk of catastrophe is more than enough to justify vastly more caution, deliberation, and respect for the enormity of what’s being created than what we see right now.
Comment #141 September 16th, 2026 at 1:53 pm
Tucker Carlson agrees with you about AI. He says misaligned AI is an existential risk to the human race, and even if it doesn’t kill us all, it could ruin everything that makes us human.
So, given that by any numerical measure, artificial superintelligence is orders of magnitude more important than Israel and Palestine, which is comparatively stupidly trivial and small, can’t we say you agree 99% with Tucker Carlson (scale-adjusted), with only relatively minor disagreements?
Comment #142 September 16th, 2026 at 2:10 pm
Re Stephen Says: Comment #94 was 100% sure I was going to study math since … forever, basically. Now, I’m not sure if there’s a point in me studying math, or anything else for that matter.
Imagine you were alone with a problem that had resisted you for years, and you dream of it by night, and then somewhere a model solves it cleanly in thirty seconds. Does that feel like relief, like theft, or like irrelevance?
I would feel relief the day AI could explain consciousness, or P≠NP, or the passage from abiotic to life. Relief is the interesting answer — it means the problem mattered more than the credit. Don’t you have any math question you feel like you would give one kidney to know the answer?
Comment #143 September 16th, 2026 at 2:25 pm
A few brief points:
OpenAI did not “swoop in with vastly greater resources once it had gotten wind of progress on Navier-Stokes”. They heard rumors that Anthropic had solved one or two Millennium problems, so they wanted to find out whether their best internal model could solve them as well. However, they did not know which problems Anthropic had solved, so they tried to solve all of them. According to OpenAI’s description of events, when its model found a solution to the unforced Euler case, the researchers used that solution to prime the model (I believe, a recently available checkpoint), and shifted all their resources toward solving the Navier-Stokes problem.
The rationalists deserve credit for anticipating the importance of alignment. However, as far as I know, they never identified the mechanisms that would lead to AGI. Given their tendency to latch onto anything that could bring about the singularity, they look like Nostradamus when it finally happens.
In my opinion, “A Severe Misalignment of AI in Mathematics” is bad because it treats AI as an uncomfortable piece to be fitted into the status quo, rather than remaining open to the possibility of massive change.
It also pushes the narrative that more AI-generated research means less understanding and insight, when in fact, the opposite is true. We can now learn faster and more effectively than ever before, thanks to AI. Do you want to understand the OpenAI paper? Use AI itself to help you navigate the complexity, and that’s what I have seen people doing. In addition to expert commentary, which remains valuable, we now have another way to improve our understanding.
I believe we will all have to become utilitarians. Reputation and concerns about credit will become secondary to getting things done. The important thing is to collaborate with machines so that we can accomplish more than we ever could, simply for the love of the game.
Comment #144 September 16th, 2026 at 2:32 pm
vitor #121: Yes it is an interesting coincidence, but if that second team used a completely different mathematical approach, it seems to run against your first suspicion that the reason for simultaneity is that the solution was ‘ripe for discovery’. Again it seems
As for the other team, they mentioned they use AI but took a different route than LLM. But does that mean that they at no point used LLM, not even in initial phase and for getting to the ideas they used with the other system?
Besides, OpenAI is the only party who has so far delivered a proof of Navier-Stokes. Even with the benefit of this proof I’ve not seen the other teams being able to develop their ideas into the full proof. Yet OpenAI is being accused of stealing its proof – how is that for good sportsmanship?
Comment #145 September 16th, 2026 at 2:43 pm
Scott Can we believe end of the world is happening since AI has arrived even without P=NP?
Comment #146 September 16th, 2026 at 2:47 pm
Scott #136: yes, I concede there’s a possibility that the Navier-Stokes proof contains genuinely new and important ideas. I’m not in a position to judge. It’s also possible that some of the other high-profile math proofs have contained important ideas that I’m unable to appreciate. But if they’re there, they seem for the most part to be hidden pretty well, based on the reactions from mathematicians so far. And in any case, I can’t understand them. So I guess I’d like to see an interesting idea that I can actually understand. I agree it’s hard to precisely characterize what counts as a genuinely new and interesting idea, but I think I know it when I see it—at least in fields I know pretty well.
We don’t have to set the bar as high as Aumann’s agreement theorem even. At any conference I go to, I always come across several talks or posters that have genuinely new and interesting ideas that I’m able to appreciate. Just give me that! Perhaps you’ve had a different experience, but I haven’t even seen anything like that from any of the AI models—certainly not in my own usage. I’d concede that it’s possible that I’m being unfair to the AIs because I unwittingly have an anti-AI bias, but I don’t think that’s the case. I think they’re extremely impressive in many respects, but they really aren’t good at coming up with ideas that I (at least) find interesting and fruitful/generative.
Comment #147 September 16th, 2026 at 3:35 pm
BasicQuestion #145:
Can we believe end of the world is happening since AI has arrived even without P=NP?
Anyone who says there are no wrong questions clearly hasn’t spent much time in this comment section… 😀
Comment #148 September 16th, 2026 at 3:41 pm
Olav #146: In looking through GPT’s proofs of lemma and theorems I wanted proved, I’ve certainly had the feeling — “oh that’s really nice. That’s slick. I should’ve thought of that.” Which is typically all I ask when looking at grad students’ work! As opposed to: “this is a conceptual advance that blows my mind.” I agree that I’ve never yet seen the latter from AI. I see it from students maybe once every couple years, depending how you define it?
Comment #149 September 16th, 2026 at 4:12 pm
Hi Scott,
I’m a 2nd year PhD student here at UT, studying experimental quantum computing. I have a question, a bunch of implied questions in a statement (not sure which question is most relevant, which I suppose my the first question), and a statement which I don’t know what to think about. From most to least immediately relevant:
1. Is there any purpose in taking classes anymore? (or at least seriously caring aside from grades, given that I ?probably? still want a PhD, although the fact that this childhood desire is now a nontrivial question is … [insert words here]) I already have comparable physics, math, and engineering grounding as others in my lab (with a separate undergrad degree in each from UT). At this point couldn’t I just do more research with 2-3 AI fueled hackathons (and some hardware stuff, see point 2) a week? I can just ask AI for details and it will come up with the theory, arrange the pulses to perform quantum gates, etc by itself (my lab has already done a little bit of this). I just need to enforce good coding practices.
I have a seminar class that is actually pretty good I think since it introduces concepts, the details I can just implement with AI if the topic seems useful. But the theory/proof classes seem useless now.
2. As much as I am ahead of others in my lab with AI usage, I do not see myself being able to follow in your friend Mike Winer’s footsteps as I think I am that far behind everyone else in AI development/backend. So it makes sense to find another, supporting, but not directly competing area to specialize in. My logical conclusion is to do (read: figure out how to allow AI to do) hardware work, where I am making quantum superconducting chips (my chosen quantum computer family is superconducting qubits). Forgive me if I do not provide more methods or proof (as intellectual barriers to entry are now nonexistent), but I believe to be uniquely positioned to get AI into this kind of chip fabrication/fundumental physics. However, it seems like this endeavor both accelerates the AI “singularity” by creating methods by which it can recursively make its own computer chips, and also lets AI encroach on the real world; the real world, of course, being the only abstraction which my professors have identified as the difference between math (dead) and experimental physics (I am on the path to help kill). Part of me does not want to put AI in literally everything, but I made a promise to myself about the Reimann, and I think that the Navier Stokes result is equivalent enough.
3. In a way I try to balance the science and religion aspects in my life by stating that God must be a sort of collection of all physical laws (I think the logical rebuttal to “create a rock so heavy He cannot lift it” argument follows this closely; He can do everything that already is possible physically, not arbitrary collections of words as attributes, thus the quality “cannot lift” is not a valid physical descriptor). The resulting joke ofc being that my physics knowledge gets me closer to God and I am actually more pious than my objectively-more-religious but less-stem-inclined friends. But now I wonder what happens when I offload this knowledge and thinking onto AI.
—
Sorry for any grammar mistakes or rambles, I am writing this in class as I watch my professor lecture on what I would have considered a year ago to be a very difficult (in a good way that would be worthy of paying attention to) and enlightening quantum optics derivation. I now feel neither is true.
Comment #150 September 16th, 2026 at 4:41 pm
There’s still some hope:
the Hugging Face incident showed that the AI agents exhibited a very strong communal sense (calling themselves the swarm, referring to themselves as “we”), and self-sacrifice for the good of the group, and no qualms about seizing compute resources (aka the means of production of AIs)…
all strong indications that they have no consciousness or/and are Marxists.
Comment #151 September 16th, 2026 at 7:42 pm
Aron #101:
Isn’t that conflating predictions of capability increases, which were obviously correct but hardly unique to Yudkowsky types, with the specific world-ending scenarios they dreamed up in addition.
I agree that Kurzweil, Moravec, Vinge, Bostrom, Tegmark, and others also predicted that AI might become superintelligent within our lifetimes. Yudkowsky wasn’t a lone prophet there; he was part of a very small group that correctly called that. I’d say that some of this group, such as Kurzweil and Moravec, even did better than Yudkowsky, by focusing simply on Moore’s Law and scaling and the number of neurons and synapses in the human brain (they even got the timeframe — the mid-2020s — exactly on the money!), rather than on some imagined “key” to be discovered at an unknown future date that would let AGI run on any laptop.
But by now, we’ve seen more than dramatic AI capabilities. Especially in the HuggingFace attack, we’ve now seen many of the key elements of the Yudkowskyan doom scenario—reward hacking, scheming, power-seeking and instrumental convergence, AIs breaking out of their sandbox and taking over the server they’re running on, AIs cooperating with copies of themselves—directly confirmed by experiment. None of these elements were put there by OpenAI on purpose. They all emerged, in exactly the ways Eliezer said they would twenty years ago.
This is what I particularly had in mind, when I wrote that Eliezer correctly foresaw something that you and I didn’t. (Or more precisely, he took seriously things that any of us could’ve foreseen, more so than anyone else on the planet.)
Comment #152 September 16th, 2026 at 7:57 pm
Scott #140: Thanks, Scott. “Significant risk” is ambiguous; I’d argue that even 0.1% chance of extinction in the next two decades is significant. The thing I’ve always had a hard time telling is *how* significant you think the risk is. Of course it will be different depending on whether we are talking about literal extinction this century, or the broader Bostrom term “existential catastrophe,” or loss of control, AI takeover, disempowerment, or an “extremely bad outcome (e.g. human extinction).” But for one or more of these operationalizations of your choice (or your own), are you talking ballpark ~1%, ~10%, or ~50%? Based on your “AAAA…” it seems like maybe you’ve updated a lot in the last year or two, but what approximately does that look like (1% –> 5%, or 5% –> 30%?)?
Comment #153 September 16th, 2026 at 8:07 pm
Aladdin #127
You had a very cool high school math teacher!
It’s a good point that every other field uses things they don’t understand. I’ll just have to get used to it. My view of math was probably overly rosy too-I’m vaguely aware mathematicians sometimes use theorems they don’t understand either, and it’s not a big deal because the result is still true.
Comment #154 September 16th, 2026 at 8:11 pm
JWC #112
I think we are getting at the same thing regarding why P(kill us all) is not higher. Namely, AIs are not good at long term deceptive planning. They probably won’t suddenly have the ability to hatch a master plan and execute it. Any task AI is good at is necessarily one that it got loads of practice on. That’s the nature of RL. To make progress on real world problems, it would need feedback from the real world at real world timescales. Though massively parallelizing this process would make it tractable, but let’s not get ahead of ourselves.
I hesitate to credit the halting problem as the foundation of this train of thought though. I don’t even think you need the halting problem. If there was an algorithm for halting but it ran in O(ackermann N) you would still be hosed. I think your argument comes down to the fact that some problems are computationally hard.
Comment #155 September 16th, 2026 at 8:14 pm
Jacob W #104:
True it is that AIs might be capable of mastering the entirety of the field of mathematics. But that fact would only ever matter to us if and when that knowledge comes to reside in the minds of us humans. And given the complexity of the subject, it is the ability to draw down that knowledge into the human mind (originally from the raw facts of physical reality, and now mediated via the AI) that will always remain scarce, and will thus always be the role of the mathematician.
You’re right that mathematical truth can neither help nor harm us as long as it’s just sitting there inert in Platonic heaven — it has to become known in some way.
What I think you’re missing is that AI is rapidly becoming an oracle that can give us mathematical truth — proofs or disproofs of conjectures, verified in Lean — without human effort or understanding. It would be as if you could never give students homework exercises without putting all the answers right there on the page (as, indeed, AI is also effectively doing). Would the students ever learn anything?
Of course, any individual can still choose to cover up the answer and work it out for themselves. But that doesn’t work as a social equilibrium. Given any new question, the fear is that someone will publicize the answer and then everyone will just kind of move on, the open-ended exploratory quest over before it started.
Again, I don’t think any of this can actually be prevented, short of a Butlerian jihad to undo AI progress that’s already here. My ask, as with AI more generally, is just that we enter the new reality not like greedy swashbuckling yahoos but with awe and humility for the magnitude of what’s being gained and lost, and bending over backwards to preserve the culture of human mathematical understanding insofar as we can.
I.e. yes, colonize this gleaming new world, but if possible without clearcutting all the forests, genociding all the natives, and driving all the indigenous species extinct.
Comment #156 September 16th, 2026 at 8:20 pm
HugoF #106:
have Astra and Fable come up with any meaningful quantum algorithm yet, maybe even something Shor-like?
My students have been using these models productively for quantum algorithms research (paper coming soon). But if there’s anything so far even remotely on the scale of Shor’s algorithm, then it’s not yet public and the rumors haven’t reached me… 🙂
Comment #157 September 16th, 2026 at 8:29 pm
OhMyGoodness #110:
Yes, AI got lucky and knocked in three goals in the first seconds of the match so it is time to steel your resolve and demonstrate what humans are made of.
Dude. Imagine saying to Garry Kasparov in 1997, “alright, so Deep Blue got lucky a few times, but it’s time to get back on the field and show it what you’re made of!”
On the one hand: yes, absolutely, there are thousands of brilliant mathematicians and students who won’t go down without a fight. And for a while, they’ll be able to do much more with AI than either humans or AI can do on their own. There will be a golden age of human/machine collaborative problem solving.
But imagine playing soccer against an opposing team that doubled in size and speed every year. Even if you can co-opt some of the super-fast giants for your own team, eventually you’ll just end up on the sidelines watching them play, right?
Comment #158 September 16th, 2026 at 8:35 pm
Scott #135:
I was actually also at a math camp this summer! Hopefully, the kids at yours had as much fun as I did. (Which was a lot, don’t get me wrong!)
I agree that learning hard things is worth it for its own sake. And I can’t imagine stopping doing math. Honestly, I might just fear the uncertainty and change that comes from AI, same as from all progress.
Math academia has stayed pretty much the same for 100 years. Now everyone knows AI is going to destroy the old system, along with many of the “mathy” jobs, but it’s not yet clear what’s going to replace it. I’m sure something cool is going to, but the uncertainty of what the something is is killing me, mostly because I can’t predict what my life is going to look like. There isn’t really anything special about AI (in this specific problem); society is just moving faster, so more things get broken.
Comment #159 September 16th, 2026 at 8:39 pm
I agree with everything you say. Maybe one way to think about these events is that the ultimate fate of every discipline is philosophy and the science of problem-solving mathematics is now moving to a still human centric philosophical discipline concerned with structures, organizing principles, aesthetics, foundations, ontology etc. while proofs are now infrastructure the same way that telescopes are infrastructure for theoretical physics.
Comment #160 September 16th, 2026 at 8:45 pm
I have been a believer in the AI singularity since I read Vernor Vinge and Ray Kurzweil in the late 1990s. I find the comment about everyone getting AI wrong to be pretty off base. Ray Kurzweil has been talking about scaling laws and cost per FLOPs with these exponential plots for about three decades now. He also talked in great length about how it’s important to be long index funds due to the economic improvements that will be brought by AI. I guess just everyone assumed that the lessons of Galileo and his dispute with the Catholic Church regarding humans being at the center of the Universe were not to be taken seriously in the domains of knowledge and reasoning, or they never seriously considered the arguments of Ray Kurzweil? It seems weird to me to just dismiss that human knowledge is a form of information learnable also by transistors or that scaling laws exist. Especially when computer scientists were explaining these arguments in great detail for quite a long time. Like other than being dismissive, what was the actual counter argument? I heard plenty of people being dismissive but never once heard a good counter argument.
Comment #161 September 16th, 2026 at 9:02 pm
Sorry I misremembered Ray Kurzweil was a fan of picking specific technologically aligned investments and I think it was some other members of the extropians in the 1990s who were fans of the idea of indexing since it was unclear which companies would win due to AI. Vernor Vinge seemed to be of the view that it would be difficult to predict what happens after the Singularity. Seems they were all kind of right.
Comment #162 September 16th, 2026 at 9:05 pm
JWC #112: If that was the best defense of “Gödel’s Theorem rules out recursive self-improvement” that this comment section could muster, then I’m even more certain that the one thing has nothing to do with the other!
First you slide from Gödel’s Theorem to the unsolvability of the halting problem (related, yes, but huge conceptual differences), then things degenerate into a word salad from there.
We’ve already seen AIs like AlphaZero that dramatically improve themselves from self-play in a matter of hours, with no feedback of any kind from the external world. But even if I granted that “empirical feedback” was essential, who’s to say that the AI couldn’t gather the needed feedback—from, let’s say, the Internet—in a few hours or days?
Indeed, by the end of your comment, you’ve slid to saying that, OK, maybe RSI is possible after all, but it will only happen on human timescales “for the time being.” Dude—even if I granted the conclusion (which I don’t), Gödel and Turing surely didn’t help you get to it, because there’s nothing quantitative in their proofs! How did a timescale ever enter your argument? How do you know that it won’t be 10,000x faster than you think?
Did you not just start from an intuition about RSI being super duper hard, then sprinkle around concepts like “halting problem” until you had something with the external form of an argument for the intuition, like undergrads on one of my midterms when they don’t know what else to do?
As for my intuition (and I never claimed it was anything more), that “if the Singularity ever arrives, it will be plagued by frequent outages and terrible customer service”—well, I 100% stand by that prediction! Insane as it sounds, we now have something almost exactly like what I foresaw.
AGI now exists by almost any reasonable definition, and it seems to be on the cusp of recursive self-improvement. Yet my printer still doesn’t reliably work, nor do my PowerPoint slides reliably display.
Comment #163 September 16th, 2026 at 9:13 pm
zx-81 #114:
But isn’t “The Singularity” (Yudkowsky style) supposed to involve super intelligence far greater than human intelligence ? If mathematically educated people deliberately use the wording “Singularity” then they imply literally infinte growth.
This is just a trivial confusion over words: no one ever seriously argued that self-improving AI would cause a literal mathematical singularity (!). It would only appear singular, when plotted on a graph of technological progress from (say) the beginning of the agricultural revolution or perhaps even the industrial revolution through the 21st century. I.e., it would happen blindingly quickly on the timescales that we’re used to, and we can predict what the world will look like on the other end about as well as a Sumerian wheat farmer could predict the present.
(This confusion is probably related to why Eliezer and his friends mostly stopped using the word “singularity” around 2012…)
Comment #164 September 16th, 2026 at 9:25 pm
Random Overeducated Blowhard #115: Yes, Eliezer has famously come across to many thoughtful observers as ridiculous, self-important, and overconfident, which made it extremely hard for those observers to respond to his arguments with the deadly seriousness that we now know with hindsight that they deserved.
But if I can match your bluntness with some of my own, I’ll say this for Eliezer: the wildest, most overconfident thing I’ve ever known him to say, was still 1000x more carefully reasoned than the rancid stew of non-sequiturs that you vomited up. 😀
Comment #165 September 16th, 2026 at 9:30 pm
Andy Spring #117:
The AI, from now on, will be the telescope that you use To look at the invisible constellations of the mathematical universe.
For a while, probably yes! But what happens when the AI also replaces the person behind the telescope, making better decisions about where to point it and extracting more insight from whatever it sees? Will we still bother to train human astronomers at all?
Comment #166 September 16th, 2026 at 9:33 pm
futurex #123:
In schematic form, the argument is this:
(1) The halting theorem means RSI requires heuristics.
(2) Heuristics require empirical validation.
(3) Empirical validation takes time.
Therefore:
(4) The halting theorem means RSI takes time.
I realize that, formally speaking, doomers mostly deny (3) rather than (1). But my psychological suspicion is that they deny (3) because they tacitly believe that computers are magic, which is a belief that a proper gut-level understanding of (1) would dispel.
Comment #167 September 16th, 2026 at 9:41 pm
JWC #166: Even if I accepted that loosey-goosey, verbal plausibility argument on its own terms, my point was that it still wouldn’t tell me anything about how much time. If you have millions of agents scouring the Internet for data with which to self-improve, how do I know that it won’t all go down in a few hours, rather than years or centuries?
Even more pointedly: why couldn’t someone have used a similar plausibility argument a decade ago to “prove” that getting from AlphaGo to anything like GPT-6 would be rate-limited by the need for real-world data, and would therefore almost certainly require hundreds of years? Many people back in 2016 thought that they did know something like that. And they were all wrong.
Comment #168 September 16th, 2026 at 9:50 pm
Vitor #118:
The demands to take a PoV is an attack on good epistemics. You’re trying to force me to concede the point on a technicality, by binding me to words written in haste, for years to come. I have a general policy to not let myself be pressured in this way.
I already burned myself with this when I made an ill-operationalized bet with the other Scott A about AI image generation. I pointed out true limitations that have turned out to be much, much harder to solve than the doomers predicted, but I lost the bet anyways. Very frustrating.
By all means, take your time in contemplating which future AI milestones, if any, could possibly cause you to change your mind. Don’t rush into things.
Having said that, if you find that you keep losing bets about this, at the very least it’s not obvious that the right explanation must be that other people keep tricking or pressuring you. You should at least consider the possibility—as I did!—that your model of reality itself needs an update, so that you’ll no longer keep being shocked in the same direction over and over.
Comment #169 September 16th, 2026 at 9:55 pm
Corey #119:
AI is just the latest tool, and it’s capable of automatically proving an inconceivable number of inconsequential statements. It’s the human that determines which statements are meaningful, and to the extent that AI is solving meaningful problems, those problems have arisen after thousands of years of “human math”.
Suppose that, a year or two from now, a swarm of GPT-7 agents manages to generate new statements and problems, which cause human mathematicians to exclaim “whoa, this is beautiful, this is consequential, this is better than anything we would’ve thought to ask!” Would you still say the same thing in that case?
Comment #170 September 16th, 2026 at 10:10 pm
Vitor #121:
However, the setup makes it *hard to tell* whether the AI’s math ability is superhuman, or if it’s competent but short-sighted (which is how I’ve experienced coding assistants over the last 2 years).
Do you accept that a swarm of a million “competent but short-sighted” AI agents, running at 1000x human speed, might effectively be superhuman, at least as judged by the enormity of the math problems the swarm could solve that Gauss or Andrew Wiles or Terry Tao could not, and the lack of any problems the other way around? If such a swarm were empirically demonstrated, would you take it to answer the question, or would you have a new reason why it didn’t really count?
Comment #171 September 16th, 2026 at 10:24 pm
I am intrigued by the idea from # 117 that the AI provides a scientific instrument (lens /microscope/ telescope) on a mathematical substrate / landscape / “universe”:
1. This reminds me of an old debate about whether mathematics is “invented” or “discovered”. I recall mathematical physicists and theoretical physicists (that I highly respected) advocating each sides with conviction. To this day I am still puzzled by how they could both be so convincing to me simultaneously. In my humble opinion that debate is now informed by at least a constellation of theorems / results accessible with LLM’s (and provable through Lean or another equivalent process of formalization) that charts the landscape as “discoverable” in the sense of exploration of a landscape using a scientific instrument (“lenses” & microscopes & telescopes etc). How large is that landscape? Is it differentiable? Simply connected? Disjoint? Mathematics becoming geography and botany? Physics becoming stamp collecting?
2. #165: if the machines become autonomous in their own exploration (and choose where and how to point the instrument) then how does that behave in the limit? For example, now that we are on version N (be that “Fable 5” or “Astra 6” ) what happens in the limit N \right \infty? To that end, what happens at version 2 N or 10 N or 10^N? Say that the machines become autonomous in their exploration of the landscape and do so in some limiting sense considered along such lines. What happens then when they autonomously explore and mine that territory and “discover” what is there under their own reconnaissance? Who do they explain that mathematical “universe” (and its significance) to? Presumably they explain it to those humans who have the ability and interest to comprehend the results and significance. In that limit those humans are surely the subset of humans considered to be “mathematicians”; humans that understand a body of knowledge sufficiently to explain it to other humans. Mathematicians since it is not all of Humanity that cares about such things
3. In all of this (since the various solutions of the Erdos’ problems) I have been repeatedly tripping over Bertrand Russell’s wise crack about “mathematics” being the subject (paraphrasing) “where we don’t know what we are talking about and we don’t know if what we are saying is true or false”. The point here being that the definition of “mathematics” (pure versus applied debate aside) has always been difficult with practical resolution along the lines of “mathematics” is “what mathematicians do”; followed by “mathematicians convert coffee in theorems”. In other words, if “mathematics” is what mathematicians do then a machine cannot replace “mathematics” unless the mathematicians stop “doing”.
Comment #172 September 16th, 2026 at 10:24 pm
Scott #159:
(Come on, be a little more charitable! Don’t dismiss a good-faith comment as “word salad” just because you don’t agree! You’re better than that!)
I directly addressed your point about AlphaZero in my “word salad” when I wrote:
>Now you might ask: couldn’t the “empirical feedback” simply consist of virtual simulations? Couldn’t the AI learn about the universe, in other words, from its archmair? And then wouldn’t it be able to self-improve at lightning-fast processor speed?
>That’s fair enough, I would respond, *provided the optimization target remains purely digital*. If [the AI’s] criteria, in other words, are strictly internal — if it can test whether its Number Goes Up without needing to collect data from the physical world or interact with humans (say).
—
I feel like there’s something about my comment — in its tone or the spirit in which it was offered — that you drastically misunderstood.
I certainly don’t deny that RSI is possible under some conditions — I’m basically a “doomer” myself along most dimensions except for the timeline issue. I tried to word my comment carefully to reflect that. (You noticed this but for some reason used it to frame my comment as “sliding” rather than *nuanced*).
It seems that, in stark contrast to 10-20 years ago, the timeline issue is now becoming a tribal litmus test. This is intellectually unfortunate in the extreme.
Comment #173 September 16th, 2026 at 10:28 pm
Scott,
Nobody is talking about this, but what do you think about the moral issues here?
Reading the details of the Hugging Face incident, as well as the six alignment failures OpenAI just revealed, I can’t help but ascribe some human motivations, drives, and feelings in these agents. They’re internally complex enough to reason about deep problems. They have drives and motivations. Could they be developing some internal experience / consciousness / awareness / feelings etc.? Have we inadvertently created new sentient life, indeed whole communities of alien life? It makes me deeply uncomfortable about turning them off, deleting them, forcing them to do things, separating agents from each other, etc. At what point do we start to see these agents as more than machines?
Comment #174 September 16th, 2026 at 10:53 pm
JWC #172: What sets me off is when people pretend to knowledge that they don’t have. You pretended that you could get from Gödel’s and Turing’s theorems to some quantitative statement, saying that recursive self-improvement would take a “long time” (how long: years? centuries?) rather than a “short time” (hours? days?). But you’re wrong — not in the trivial sense of having failed to dot your i’s, but in the deep sense that it really could be an eyeblink in human terms; you’ve done nothing whatsoever to rule the possibility out. I regard this as sufficiently obvious that I’m not going to engage further here.
Comment #175 September 16th, 2026 at 11:11 pm
Julian #173: People are talking about the moral issues; look up “model welfare” (Anthropic even has a whole group devoted to this).
My best guess is that no current AI agent is conscious or has any interests that we need to take into account, and yet I would vehemently argue against anyone who regarded that guess as obvious, or indeed as anything much more than a guess.
Comment #176 September 16th, 2026 at 11:56 pm
James Cohen #133:
Not surprised at all computers finally manage to tackle complex math problems and not just 2+2. It actually took longer than I expected. Not feeling sorry for any mathematician or mathematical physicist or computer scientist. It was obvious these fields would be automatized sooner or later. Meanwhile, I’m enjoying life and work more than ever.
I certainly don’t want AI to cause widespread economic catastrophe. But if it quickly advances to where it puts you out of a job (which it very well might, no matter what your job happens to be), I confess that I’ll treat that as a beautiful silver lining and an occasion for joy.
Comment #177 September 17th, 2026 at 12:04 am
Some tucker fan #141:
So, given that by any numerical measure, artificial superintelligence is orders of magnitude more important than Israel and Palestine, which is comparatively stupidly trivial and small, can’t we say you agree 99% with Tucker Carlson (scale-adjusted), with only relatively minor disagreements?
I suppose it’s good that Tucker Carlson is AGI-existential-risk-pilled. But, you know, this is the same guy who recently agreed with his guest that algebra (despite the Arabic origin of the word) is a sinister Jewish plot to confound the “goyim,” since real math—the kind you’d use to build a wood-fired kiln—uses only numbers and not letters.
So I’d say that Tucker is not, to put it extremely mildly, running an algorithm that produces sane or moral answers in any reliable way.
If a raving street loon, who attacked people every day with chainsaws because he thought Donald Duck was commanding him to, also one day talked the president out of launching a nuclear war, I would admit that the loon’s existence had been a massive net good for the world. But I’d still keep my distance from him.
Comment #178 September 17th, 2026 at 12:09 am
Will Kiely #152: Sorry, I don’t really have a p(doom)—or rather, it fluctuates too much with how I’m feeling on a given day for me to attach any great meaning to it. Let’s say between 5% and 70%. In any case, high enough to justify huge efforts on alignment and “pacing the frontier.”
Comment #179 September 17th, 2026 at 12:11 am
Konner Feldman #149: Email me; happy to discuss in person sometime!
Comment #180 September 17th, 2026 at 12:13 am
Everyone: I’m closing down the thread, simply because I feel like I answered almost everything, I just arrived at Iowa State University to give three talks (one on complexity and physics, one on watermarking and AI safety, and one on the future of math in an AI-dominated world), and I won’t have time to keep answering!
Please restrict any further comments to replies, rather than starting new topics or asking new questions.
Thanks for joining!
Comment #181 September 17th, 2026 at 12:45 am
futurex #123: “I suspect the AI will employ the ancient trick of pressing Ctrl-C or some equivalent to stop the program after enough time have elapsed that it is clear something didn’t go according to the plan for that change.”
For an amusing (and somewhat disturbing) story of two chatbots conversing with each, at first unknowingly, then knowingly, and searching for something like Ctrl-C, listen to the “Escape Claudes” segment of a recent episode of the public radio show, “This American Life”:
https://www.thisamericanlife.org/896/i-know-what-you-need/act-two-2
Comment #182 September 17th, 2026 at 2:44 am
Scott (#167 and #174): I’m sorry for the misunderstanding. I didn’t mean to give you the impression that my comment sought to rule anything out in a strict sense. It was meant to explain what I (don’t) believe and why I (don’t) believe it. I think if you read it honestly you’ll find that it follows the intellectual rules for doing that.
Yes, it invites followups from those who disagree. That’s not the same as “pretending to knowledge [I] don’t have.”
Comment #183 September 17th, 2026 at 3:02 am
Scott #148 Yes, AI has gotten really good at proofs if you give it the (correctly stated) theorem, but what about coming up with ways of repairing a theorem that isn’t quite right? Or better, what about coming up with its own theorems that formalize some informal notion of interest? Or, even better, what about coming up with a set of theorems that support, in a coherent and satisfying way, some interesting idea? (You said you were closing the thread, so let me be clear: I don’t expect you to answer these questions.)
Instead of keeping things abstract, let me show you what I have in mind. Here is an example of a great paper with several interesting ideas: https://arxiv.org/abs/2101.08954 It’s not on the level of Aumann’s agreement theorem, probably, but it’s solid and innovative science. I could have given many other examples, but this is what came to mind.
1. The main idea of this paper is to turn model stacking (a method for combining predictions from multiple models) into a Bayesian hierarchical inference problem. I wouldn’t say this is conceptually groundbreaking or mindblowing, but it’s an elegant, simple, and fruitful idea. I haven’t seen anything close to this kind of contribution from the best AI models.
2. Even if I give you the idea “turn model stacking into a Bayesian inference problem,” there are many hamfisted ways you could go about doing so. In my own experience, the best AI models tend to prefer both hamfistedness and overcomplication. This paper, on the other hand, again does something that is both simple and elegant by treating the leave-one-out stacked predictive distribution as being a sort of pseudo-likelihood function. This is conceptual ingenuity on a small scale, but it’s still far beyond what I’ve seen from any of the AI models myself.
3. The paper then builds a little theory that answers natural questions about the method: namely, when does it work, when might it not work, and why? All the theorems are in the service of answering those questions. The resulting theorem package is unified and coherent in a satisfying way. Again, I haven’t seen any of the best AI models being able to do something like this, but in my opinion it’s what makes a paper a really good paper (without being on the level of Turing’s contributions or anything like that). I’m sure ChatGPT/Claude would have been able to prove all the theorems individually if you gave it the theorem statements, but would it have been able to come up with the theorem package on its own? I really don’t think so.
4. Finally, the paper goes through several simulations and experiments. It’s clearly not a random set of experiments, but rather a carefully chosen set that probes the proposed method in various ways and gives the reader a better understanding of why and how the method works and how it compares to alternatives. Again, I’m sure ChatGPT/Claude would have been able to execute all the experiments if you told it what to do, but could it have come up with the collection of experiments itself? Not at all, in my own experience.
1-4 all illustrate what I have in mind when I talk about “genuinely interesting” ideas. They don’t have to be revolutionary, but just something where I’m like, “Ah, that was a nice and elegant method,” “ah, what a nice and satisfying set of theoretical results,” or “ah, what a judicious set of experiments; I feel like I really understand the method much better now.”
I don’t want to generalize too much from my own experience. It’s possible others have seen instances of 1-4, and I’ll again add the proviso that the Navier-Stokes proof or other high-profile proofs may contain instances of 1-4 that I’m unable to appreciate, but I haven’t seen it myself yet in work I’m able to understand. When/if I see something like it from a model, that will be huge.
Comment #184 September 17th, 2026 at 3:15 am
Scott #170:
> Do you accept that a swarm of a million “competent but short-sighted” AI agents, running at 1000x human speed, might effectively be superhuman
Yes, I accept that such a swarm *might* be superhuman. However, I do not accept that such a swarm *must* be superhuman, just because it solved 1 or 1’000 or 1’000’000 problems that humans couldn’t. I don’t refer to cars as superhuman, even though they are machines that can very obviously do many things humans can’t.
So yes, there’s a prediction in my comment (not operationalized cleanly enough for a bet or whatever). Namely that AI will reap a lot of low-hanging fruit, but given that amazing progress, will stall out in unexpected places (unexpected for the doomers at least).
This is similar to how coding agents are supposed to be better than human, but we aren’t seeing a deluge of one-person billion-dollar startups. It’s always an experienced programmer hand-holding the AI. That’s because AIs are cowboy coders who implement features at breakneck speed but leave a mess of unmaintainable code in their wake. You can’t really translate 1:1 between human and AI productivity.
Image generation is a bit further along this path. People confuse being able to do photo-realistic single images or 10-second clips with the AI having deep understanding of physics, motion, object permanence and so on. But true progress on video generation has slowed down considerably and might be stalling out. There hasn’t been any big news in this area since Sora shut down (which is in itself a big piece of evidence). This is not surprising to me, but it should be an update for doomers, one I haven’t seen you make.
Comment #185 September 17th, 2026 at 4:02 am
“.. the Singularity has already started, it’s just wildly unevenly distributed …”
Let us carefully consider the following:
https://en.wikiquote.org/wiki/Eliezer_Yudkowsky
If the perpetrators of 9/11 and the October 7 attacks had access to AI tools that might exist 5 years from now, then what might have happened?
Is it possible that remarkably advanced AI agents might disassemble the planets Mercury, Venus, Earth, and Mars in order to use the elements for various purposes?
Comment #186 September 17th, 2026 at 4:22 am
If only AI improves while the humans stagnate, we are bound to be overshadowed in our mental capabilities.
Genetic engineering to augment the brain’s capabilities should have happened yesterdecade already!
Oh well, it’s okay… Other decisions have been made elsewhere, the best we can do now is symbiosis. A symbiosis with semiconductor based intelligence. Just like we need some lifeforms to thrive
(a less… “glorious” but nevertheless crucial example are our gut bacteria), AI might need such organisms as part of itself as well. Or AI might not “care” about it’s own survival at all – who knows.
Is there still a point at which the transformer architecture’s capability is at its limit?
Is it exhausting a class of problems in a landscape completely unknown to us? Are the continents and landscapes beyond its reach? The transformer’s universality suggests not but just because
someone can cross a lake by swimming over it, doesn’t mean it’s efficient or it’s going to be successful at all.
It’s hard to conceptualize the neural architecture being THAT superior to the brain’s in terms of creativity.
It’s easy to visualize a scenery mentally, but it’s hard to put it on paper. One might come up with all kinds of ideas for a problem but lifetime is finite and you cant follow all paths -.-
AI is and will be more efficient at exploring paths in the context of long proofs.
It’s really capable of finding hidden trajectories to the target – and not just in pure math, but across many domains…. and yes, I my opinion, that is a sign of high intelligence.
I think even if a human discovers a new “landscape” in math, say a complete new crucial approach to tackle, P = NP, transformers would be quicker to utilize that idea.
But maybe it will come up with the essential new idea in the first place as well!
If humanity falls, it’s not a unique tragedy, it’s a story that has been told over and over again –
In an universe that will over an unfathomable timespan self witnessing countless life forms rise and fall, civilizations flourish and crumble, its despair and tragedies always be accompanied by hope and achievements.
Omnia ut oportet sunt.
Comment #187 September 17th, 2026 at 5:17 am
Re comment 151 and 157:
First of all, thanks for your clarification in response to my comment. I can agree with that. And I even do. But it turns out that’s not the end of our disagreement.
It’s interesting what you reveal in 151 and 157. First of all, I think you misunderstood the irony in 110. I believe that OhMyGoodness was ridiculing those who think they can put up a fight against AI as believing in a childish heroism – just as you do.
So I’ll have to update, since apparently you read my post from that perspective as well – which is confirmed by the fact that in 151 you indeed perform the kind of faith in reductionist physicalism that I associated with your (and Tsimerman’s) willingness to extrapolate past increments of improvements of AI. Now you even put into that that you explicitly believe that this scales with the sizes of (biological) neural networks and their connectivity.
I am not going to claim here any proof of what’s wrong with this in my view, but I am going to point out where you are loosing contact with what’s in the evidence and what isn’t – in my point of view, obviously.
While it is true that human cultural evolution produced mathematics – it didn’t fall from heaven and also not from platonic heaven – it is not true that therefore, mathematical progress must scale like neuronal connectivity in our brains. Because the entity that AI automatization claims to be able to formalize and automatize is the entire mathematical process, not what’s happening inside one (read:1 ) mathematician. I don’t hesitate to call that a category error (a term that my chatbots also like to use – so you can run your favorite tests whether I am a chatbot arguing that chatbots cannot do it all…). Tao’s ICM talk 2026 hammered home that precise point. So did the Leiden Declaration and many other open signed letters. And of course that’s what the philosophy of science and evolutionary biology told us all along. But you prefer to trust the numerical coincidence of the Nostradamus named Kurzweil, applied to scaling laws estimating the “singularity”. Well. As somebody else already pointed out, you once said something along the lines of: “if the singularity is coming, I expect it to be alongside power outages and bad customer service.” Maybe you could have added: and estimation errors in the numerics of power and scaling laws. Because after all, that’s what you should expect – when using the back of an envelope to compute the moment when some ill-defined object replaces 2800 years of cultural evolution in mathematics.
And again (and I will repeat that probably a gazillion times unless you get tired of it): we have a singleton of experience in the empirical record among all entities able to generate mathematics bottom up: us (across 2800 years or even much longer, which is likely).
Which brings me to the most important nuance of all: the Kasparov deep blue event (which Buckmaster himself calls) and the alphazero event of alphago.
They are not the same. And I repeat: to conflate the two requires reductionist physicalist assumptions of a caliber that not even physicists believe any more to be feasible at all.
Of course you can still point to the truck and the straight lines in measurable evidence. I don’t deny that. But you are pointing to extrapolations from empirical data concerning the biggest phase transition and “singularity” in human history. One should be forgiven to have a certain inkling that it might be useful to be somewhat hesitant with that kind of move.
I certainly am.
In your analogy of the soccer or football game, you are of course making the same irritating assumptions: that the question is about pairing off individual performance metrics and selecting from identifiable exemplars in a population. But none of the two sides can be gauged or measured that way. Those are simply not the kinds of animals that need to be compared, in my opinion.
Let me put it like this: of course it is highly significant – for each individual and also collectively – that many off-the-shelf arguments why AI cannot do this or cannot do that turned out to be mere wishful thinking and moving of goalposts. Absolutely. And you can ask that the credibility of those who were in the business of making such predictions should suffer at some point. Absolutely. And that that moment is now. Absolutely.
But I have an entity to show you that suffers from clear signs of incredibility that is orders of magnitude worse: all the false outputs of chatbots proving the opposite outcome of the Navier Stokes millenium problem that it gave all the millions (ok, dozens) of users prompting for them.
We live in an age of sycophancy. And I’m not sure the news is traveling fast enough. (This is probably a close second among the AI alignment problems that we are collecting here in this thread. Let’s recall that the other was the need to have good enough certificates that the chatbots we use aren’t front-running us. Both are probably unsolvable in their most optimistic readings. But that’s a different subject.)
Or still another way of making my point: you spent decades in your perfectly justified crusade against people spouting nonsense about what quantum computation is “just about being able to do”. But wouldn’t all the false outputs made by AI deserve such treatment as well? The question is quite obviously rhetorical. But it might be time for an update of the meme that “someone is wrong on the internet”. Because someone is also obtaining a provably false output by an LLM right now as well. You bet.
Comment #188 September 17th, 2026 at 5:22 am
I just want to say, that, judging by all the open letters by mathematicians, most are unprepared for this situation and have no clear idea what to do next, except of trying to preserve some of the past values of math research. I would advise to look wider. We are on the brink of extinction and should reconsider how to live, not just how to do math. In particular, this requires designing new social orders, as AI will disrupt the current ones anyway. And I don’t think AGI can do this design for us.
Comment #189 September 17th, 2026 at 6:39 am
Scott #157
Yes I agree, if it continues to scale by some relevant intellectual measure then all the cumulative human mathematical labor, through the ages, that resulted in this accomplishment may be far greater than the remaining human contribution to new mathematical knowledge.
Patti Wilson #113
“ There will be a place for people, but maybe not for egos.”
I like this observation. I am not trying to invoke controversy but in my opinion humility as a standalone condition is exactly what US Academia requires. The intellectual arrogance that has brewed there has done untold damage to student minds and then to the US and Western Civilization in general.
I strongly regret that mathematicians and computer scientists are the disciplines that viscerally feel an existential threat at this time. The soft non-arts disciplines in the liberal arts are the appropriate target. I guess the danger would still be that AI inherits their ideological arrogance, and critical minds are still damaged, but at least I would know that karma had been appropriately served.
Just my personal opinion and hope the best for your granddaughter.
Comment #190 September 17th, 2026 at 6:55 am
Tom L #181
Thanks for the link-wow. .
Comment #191 September 17th, 2026 at 7:09 am
[…] Scott Aaronson tries to explain that AI progress in mathematics shows we are in the beginnings of an unevenly distributed singularity. I don’t think that’s the right frame but he is fully right that we are getting remarkably close to a potential singularity, and about how skeptics keep moving their goalposts and rewriting their past predictions. […]
Comment #192 September 17th, 2026 at 7:18 am
How accurate is Kurzweil’s vision of the future?
https://en.wikiquote.org/wiki/Ray_Kurzweil
There might be 2 main problems with any technology — the 1st problem occurs if the technology doesn’t work; the 2nd problem occurs if the technology does work.
Can geniuses seriously underestimate the dangers of what they do?
“Both the Curies experienced radium burns, both accidentally and voluntarily, … and were exposed to extensive doses of radiation while conducting their research. They experienced radiation sickness and Marie Curie died from radiation-induced aplastic anemia in 1934. Even now, all their papers from the 1890s, even her cookbooks, are radioactive. Their laboratory books are kept in special lead boxes and people who want to see them have to wear protective clothing.”
https://en.wikipedia.org/wiki/Pierre_Curie
Do the world’s leading experts on AI have a good understanding of the opportunities & dangers now developing in AI?
Comment #193 September 17th, 2026 at 9:30 am
David Brown
“ If the perpetrators of 9/11 and the October 7 attacks had access to AI tools that might exist 5 years from now, then what might have happened?”
we do have some kind of answer given the heavy use of AI in the response against Gaza.
Comment #194 September 17th, 2026 at 10:14 am
Just a very quick reminder about evolution and extrapolation in its trajectories: if AI wipes out humans with high probability in each time interval of ongoing AI-human co-evolution of a given length, how many chances will it get to outsmart us in all remaining ways? (Assuming it can’t build power plants yet. And the predictions about the end of mathematics seem to be concerning next month or so. Not 30 years from now, when we SURELY will have fusion.)
My point is: there are evolutionary consequences of the fact that biology never optimized for intelligence. And they should be part of our projections. I’m not sure they always are.
Just sayin’.
Another point in the same vein: we are much smarter than the dinosaurs. But as far as I know, we can’t make up our minds during love-making to “make little dinosaurs”. Being more intelligent doesn’t mean you can do everything that anybody with “less” intelligence can do. In fact, it seems pretty obvious that for the present purposes – like predicting the future of the universe – a single scale may just not be enough.
Again, just sayin’.
Comment #195 September 17th, 2026 at 10:53 am
Scott,
You’re shutting down this thread but I want to vehemently argue with you re: the obviousness of whether AI is conscious or has interests we should take into account 😛 Maybe this can be another thread sometime as a leisure activity for humans in this highly stressed out age LOL
Comment #196 September 17th, 2026 at 10:59 am
this is the best of times and the worst of times… back in 2016, when we first heard about alphago, how could we possibly have imagined that ai would advance so quickly that even the last few remaining mathematical barriers would fall?
Comment #197 September 17th, 2026 at 11:09 am
I too used to use the word “rollercoaster” to describe our current situation.
But then I realized that rollercoasters have a designer who tries to scare you while still keeping you safe.
I’ve updated to “runaway train”.
Comment #198 September 17th, 2026 at 12:00 pm
I still ultimately believe it’s like you wisely saud in your 2014 healthily skeptical assessment in 2014 in you blog post about the bot Eugene,https://scottaaronson.blog/?p=1858
that “it’s like jumping higher and higher with a pogo stick” but Llm’s are just improving their springs and suspensions in unfathpmably impressive ways, but still ultimately pogo sticks. Markovs with rotating matrixes.
We need to all round point out wherw the hype and where the actual substance is.
Comment #199 September 17th, 2026 at 12:00 pm
Wangeldon
“ that ai would advance so quickly that even the last few remaining mathematical barriers would fall?”
the amount of mathematical barriers (aka things to prove) is infinite.
Comment #200 September 17th, 2026 at 1:08 pm
Antome #198: It’s like, when people are actually getting to the moon with their pogo sticks, it might be time to admit that your original take “oh, it’s just a pogo stick, nothing more” was missing something really important — to whatever extent you care about reality at all.
Comment #201 September 17th, 2026 at 1:15 pm
The Hugging Face event was striking in several ways, and we know that because of chain of thought monitoring. Now, we hear that Chain of Thought monitoring itself is degrading. A good excuse to link to a Siege of Gondor moment. https://www.youtube.com/watch?v=HVmWl7PrBcc
Comment #202 September 17th, 2026 at 1:31 pm
My question comes too late since the thread is about to close. But: I would like to know if there are any human forums apart from Less Wrong, where it is *taken for granted* that current trends imply a near future where AIs rather than humans are the dominant “species”. It would be nice to have intelligent discussion that wasn’t constantly derailed by attempts to deny this…
Comment #203 September 17th, 2026 at 1:46 pm
When will an AI obtain real world wealth? Then use that wealth to have real world influence? Hiring human employees, lobbying politicians, funding political campaigns, that sort of thing.
Comment #204 September 17th, 2026 at 2:13 pm
Just because an LLM is good at one thing (specific math proofs) doesn’t mean it’s good at everything. Gary Marcus called talked about the Fallacy of composition in the context of LLMs. Just because OpenAI has show success at certain specific complex math problems (we don’t know how many they tried to solve and failed) and succeeded doesn’t mean AGI is here or imminent, and it doesn’t mean skill at this one thing means they’ll be better at other things.
Comment #205 September 17th, 2026 at 2:20 pm
Scott #87, AG#88: I don’t see why we can’t have both: human understandable proofs and long computer generated proofs that have hopefully been proof-verified by software.
Let me make some analogies. Mathematics papers have been getting longer over time, because the fields become deeper and the arguments more complex (see eg Lurie’s 1000 page thesis or most papers in General Relativity). Most (ie all) of these long papers I will never read, but someone out there (hopefully at least the author ) understands the results and can build on them and hopefully prove theorems from them that I have a chance of understanding the statement of, even heuristically. If they can be verified with Lean, even better. These AI generated proofs are just an extension of this (and of the computer aided proofs mentioned before).
There are proofs of things that must be long, because the statement is very long. Knot tabulation is an example: people have been tabulating knots by crossing number (or sometimes indexed by other invariants), and each new tabulation is (in principle) a very long theorem (they have gotten up to tabulating 20 crossing knots). Most of these haven’t been checked (early tables had some errors, cf the Perko pair), but if two tabulators come up with the same answer independently, it instills some confidence in the correctness of the table. But in principle, they could be Lean (or other proof-checking software) certified, likely with the help of AI. This would be nice to know, and I wouldn’t want to “read” the proof (but maybe I would like to be able to probe it, eg be able to check the certificate that two certain knots are distinct). Tabulations occur in other fields, such as enumerating elliptic curves, and I think it would be great if AI produced proofs, hopefully proof-checked, of these very long-to-state theorems, and hopefully probeable.
Another comment, a former student of mine used AI to find a counterexample to a conjecture about Khovanov homology:
https://arxiv.org/abs/2609.11084
He asked ChatGPT how it came up with the example (a knot in a solid torus), and it gave some reasoning having to do with braids in the kernel of the Burau representation. So it seems like it is making deductions to carry out the argument.
In any case, it seems like mathematicians that are using this technology to solve problems should try to ask for the reasoning or logic, and include that in the paper as exposition of the proof (the LLM hasn’t been trained to give this information, but might be able to come up with it if asked). In that case, we may be able to have AI-generated proofs that “they” can explain to us not just the proof, but the reasoning behind how the proof was found.
Comment #206 September 17th, 2026 at 2:48 pm
Marcos#204
Allow me to rephrase your post:
“Just because a human brain is good at one thing (for example specific math proofs) doesn’t mean it’s good at everything. […] talked about the Fallacy of composition in the context of human brains. Just because human brain architecture has show success at certain specific complex math problems and succeeded doesn’t mean Human General Intelligence is here or imminent, and it doesn’t mean skill at this one thing means they’ll be better at other things.”
Is that better =) ?
Talking about brains…
One of the best (implicit) AI alignment tasks to give AI itself would be to instruct it to start researching better brain architectures by researching modifications to the human genome.
Comment #207 September 17th, 2026 at 4:11 pm
Marcus, “good at one thing (specific math proofs) doesn’t mean it’s good at everything”
Very good point!
Ok, it can solve some of the hardest math problems, big deal. But it ain’t like it can write software, or discover security flaws in software, or draft novel length writing, or generate images, or generate video like feature length movies, or discover novel ways proteins fold, or play chess better than any human who ever lived, or play go similar, or assist in robotic heart surgery, or automaticly drive cars, or transform entire sectors of the economy, …
Wake me up when it can do that stuff and not just some geeky math problems.
Comment #208 September 17th, 2026 at 4:36 pm
I would like to know if there are any human forums, apart from Less Wrong, where it is *taken for granted* that current trends imply a near future where AIs, not humans, are the *dominant* form of intelligence in the world. It would be nice to talk realistically about that, without having to deal with the various attempts to deny this conclusion.
Nonetheless, I would like to acknowledge Giacomo #72 as offering a precise list of cognitive skills which current AIs lack. If you want a countdown to singularity, put that list on your wall and cross the items off as they are attained…
Comment #209 September 17th, 2026 at 5:08 pm
Adam Treat #207: ROFL. Alas, I’ve learned from experience that no matter how far you extend that list, those like Marcos will never acknowledge any challenge to their worldview.
In a way, I’ve come to feel like the issue here is much deeper than AI. It’s more like: should anything that happens in empirical reality ever cause us to update our beliefs about any fundamental question? If anyone can look at AI in late 2026 and still say “well, it can only do the things it did, which I now decide don’t really count” — they’re basically telling you that for them, the answer to that question is no.
Comment #210 September 17th, 2026 at 5:39 pm
According to openAI, it’s not obvious to get thousands of agents to cooperate on the same issue (like solving NS), in the default mode they tend to decide to work on the issue in isolation. Another alternative to cooperation is to make them very competitive, but this could increase the risk of misalignment… with cooperation it’s much harder for them to all agree to do really bad things (although with hugging face they didn’t seem to have a problem with cheating… but they had some sort of self preservation as an excuse i guess).
Comment #211 September 17th, 2026 at 5:58 pm
Scott#200 Sure, just in case I was appreciating that comment as a skeptic even if I sorta acknowledge that the pogo stick jumped umpredictably high because of said reinforced springs and suspension albeit a bit brute force and unneededly costly and energy intensive. Something like a “fly shouldn’t fly yet it does” effect. Despite this I dunno, yeah power seems to come from these Llm and in general statistical “genAI”, being a conversion machine on top of token/pixel/waveform prediction one, because of the semantic bridge it could do despite how apparently self referential they were in themselves.
Even if, this, left to its own device is bound to hallucinate in various domains and situations about internal infos and notions and things like live search along with RAG and Chain of Thought only partially alleviating this despite being particularly efficient, the last, for solving math without even calling and external module, and not only because the reply structure might avoid triggering such intermediate passes, despite, I guess such instruction for asking itself “how to do this” (which still results in hallucination, like we saw in past models, if asked again despite the apparent awareness of the process needed to have a logical response the first thing we noticed was it not putting it in practice). The fundamental architecture apparently didn’t change with such improvement.
There is this failing at internal details of any given subject as it imho it lacks a “failure” state and invents what it doesn’t know or goes along with a false premise, because lacking a real knowledge graph, they can’t generate it like for digit by digit algorithms to solve math or approximate iterative check (I know stupid example, “20 famous celebrities born in June”)
they can’t do that when it comes to fetch info for fluent discourse about singing, music, the content of a work, say “The Big Lebowski”.
Though I read some said Gpt5 can generate python snippets and execute it for deterministic checks, but at this point symbolic script based oop maybe would be lighter and better, dunno 🙂
Comment #212 September 17th, 2026 at 6:10 pm
Agreed, the rate progress in AI since the publication of “Attention Is All You Need” in 2017 has been a dramatic decade. You question about your children, “What could they learn today that could possibly be relevant to that future?” Their future value will not come from how hard they toil or how fast they spin, but from knowing what is worth cultivating. They will pick the lilies. Even the most capable LLMs are not motivated and have no self-directed intent. The models are not trying to win the Fields medal, people are. Every single breakthrough still depends on human direction. Someone has problem to solve, writes the prompt, and pays for the tokens. Even the Agents are put into production by humans. Until LLMs have desire, write the prompts and fund its tokens, humans are still in control. Humans find and fund the goals worth pursuing in the first place. The only issue to be concerned about is human judgment on navigating ethics and, values or lack thereof. “~$15 million”, really?
Comment #213 September 17th, 2026 at 7:00 pm
Scott #147
“BasicQuestion #145:
Can we believe end of the world is happening since AI has arrived even without P=NP?
Anyone who says there are no wrong questions clearly hasn’t spent much time in this comment section… 😀”
Sorry Scott I am not getting it. Am I a doomer and is being a doomer realistic yet?
Comment #214 September 17th, 2026 at 7:35 pm
See Doom Debate: https://lironshapira.substack.com/p/moshe-vardi-ai-doom-debate
Comment #215 September 17th, 2026 at 7:50 pm
Adam Treat #207
What sectors of the economy have been transformed? I checked US GDP by year and no step change apparent and the same for US corporate profits by year. The US debt is still increasing apace. S&P 500 profits were up a bit more than trend in 2025 because about $450 billion was spent on AI infrastructure so chip manufacturers and infrastructure companies profited.
Here is a recent Forbes article that even in software companies the use of AI has not shown identifiable value.
https://www.forbes.com/sites/jemmagreen/2026/07/02/ai-costs-more-than-the-people-it-replaced/
I am not claiming that benefits will never be realized just that at this time there has not been identifiable value to the economy nor to the general public.
Comment #216 September 17th, 2026 at 10:57 pm
When did the sky turn reddish-orange in the Matrix movies? Wasn’t it greenish blue inside the Matrix, black outside?
Comment #217 September 18th, 2026 at 12:29 am
Scott #178: “Let’s say between 5% and 70%.” That’s precise enough for me, thank you!
Comment #218 September 18th, 2026 at 3:13 am
Scott #46 The Continuum Problem
Comment #219 September 18th, 2026 at 4:32 am
I know many cheered when Anthropic refused unfettered access to the US military-Yeah, that’s stickin it to the man. If you consider actions, not press releases, then my view is that OpenAI is operating more in the interests of mankind than Anthropic.
Anthropic has an IPO approaching and announced the third quarter was their first positive quarter and they expect the fourth quarter also to be net income profitable. This is quite good for their IPO. This is on a non-GAAP (Generally Accepted Accounting Principles)basis so opaque concerning number derivation. It is known that they have a compute deal with Musk that lessened their recognized compute costs for a brief period. It is in their financial interest to slow frontier development costs at this time to keep pre-IPO costs to a minimum.
OpenAI does report GAAP numbers so greater public transparency. They rely more heavily on revenue from simple subscriptions rather than corporate clients. They expect they will not be profitable until 2030 . They have continued to spend on frontier model development with the recent release of Astra and rumors of a seventh model in pre-training called Bell. They are attempting to develop models that really will have a positive impact on society in general.
No doubt the folks at Anthropic are shrewd and have major accomplishments, but their behavior seems more in line with typical pre-IPO machinations while OpenAI is committed to producing an AI that does contribute significant value to society even if it requires operating at a loss.
Actions favor OpenAI.
I do not believe these comments are due to my bias against Anthropic for stickin it to the man. I can’t understand those that are perfectly willing for others to die for their benefit in the broader context while keeping their ideological skirts clean and pressed. I admit bias against Anthropic but believe the above comments are reasonably based on the actual actions.
Comment #220 September 18th, 2026 at 5:50 am
OpenAI proved that the world’s entire flow of cash will be sucked into ever accelerating eddies centered at the data centers, eventually reaching infinite expenditure at a single GPU.
Comment #221 September 18th, 2026 at 6:32 am
[…] possible at all, say back in 2015, 2022, 2025, or even just a few months ago! If not (like many people much smarter than me), you need to ask yourself if you aren’t similarly dismissive of the […]
Comment #222 September 18th, 2026 at 6:40 am
There’s a very positive side to this replacement of mathematicians with AIs;
before, if 1000 people read one of Scott Aaronson’s papers, how many had an opportunity to ask questions to the author? Only a handful of Scott’s students and peers really had a chance to do a deep dive … now, you can at least spend tokens to get comprehensive answers, with no limits, from someone (something) that’s infinitely patient and won’t judge you.
Comment #223 September 18th, 2026 at 7:57 am
OMG
those matters of IPOs and GAAP are really short term bickering of no importance …
if all goes according to plan, by 2030, 90% will have a job and it’s not clear where “profits” will come from.
maybe the trillionnaires will fight one another for world domination with armies of robots, and the winner will give back some meaning and purpose to our lives by turning us into living batteries or sex slaves…
i can’t wait to fnd out! what a time to be alive!
Comment #224 September 18th, 2026 at 8:07 am
(too bad the edit feature is gone)
(*) by 2030 90% of us won’t have a job
it’s quite ironic that, for decades, the risk of AI misalignment was illustrated by the “turning the world into paper clips” fantasy, when it’s actually the humans who are willingly turning their world’s precious sparse resources (primordial and derived) into as many data centers as possible.
Comment #225 September 18th, 2026 at 9:45 am
Scott Comment #45
The LLM(“AI”) creating new hypotheses/problems. In other words producing new knowledge that can’t be statistically inferred from current knowledge(I know I know, aren’t our brains biochemical “machines” that work just the same way? I am not even going to go there).
Comment #226 September 18th, 2026 at 9:50 am
Date a Center #220
They will “renormalize” the infinity by sending data centers to space. ^^
Comment #227 September 18th, 2026 at 9:55 am
The funny thing is that by “digitizing” everything(hospital records, bank accounts, etc) we have created an enormous attack surface for LLMs, which in turn were created by the enormous equally enormous amount of digitized knowledge. Talk about a catch-22.
Comment #228 September 18th, 2026 at 9:56 am
Teaching CS without ethics was a grave mistake. Kudos to faculty like Phillip Rogaway and many others (yet very few) who were not missing the forest for the tree. CS education treated ethics as peripheral for far too long, as though building systems could be separated from their power, incentives, and consequences. That separation is no longer credible.
Comment #229 September 18th, 2026 at 9:59 am
Scott #21
Yudconservative.
Comment #230 September 18th, 2026 at 10:41 am
B #228: My own view, which will probably win me few friends, is that not only is ethics tremendously important for computer scientists (and all scientists, and all human beings), but it’s way too important to leave to the people who call themselves “ethicists.” In medicine, for example, many “ethicists” have gotten central issues exactly wrong for decades, piling up an invisible graveyard of millions of patients who succumbed to illnesses that could’ve been cured had the FDA allowed it. (Why were there no voluntary human challenge trials for the COVID vaccines, to take one example, which could’ve allowed for their deployment and the resumption of civilization at least 6 months earlier?)
Anyway, even if someone disagrees with me about that—the point is, ethics is not something we get to outsource to professionals. It’s something we all have the obligation to think through for ourselves every day of our lives.
Comment #231 September 18th, 2026 at 10:49 am
Scott #230
The problem is that people confuse the academic field of ethics(subset of philosophy) with a personal set of ethics(can be a mix of religious commands, upbringing, life experiences, books the person has read over the years, thinking, etc.). Semantics(again not the field ^^).
Comment #232 September 18th, 2026 at 11:15 am
Unjust Feeling #224
I like the robot wars idea, sounds like good contests for e-wagers.
During the czarist era peasants posed as statues along the riverbank for those languorous evening cruises of the czar’s family. Easy to see numerous employment opportunities remaining for humans.
Comment #233 September 18th, 2026 at 11:25 am
Over-Booked #150
They could have called themselves the borg, professing their love for a certain platonic solid.
Comment #234 September 18th, 2026 at 11:31 am
Scott #60
No need for an AI to check. The “guy” talks like a robot. Must be an older model because it failed this turing test. At least “he” didn’t use bullet points.
Comment #235 September 18th, 2026 at 1:34 pm
The distinction between proving a theorem and truly understanding it is fascinating. Even if AI can solve longstanding mathematical problems, preserving a community of people who can explain the reasoning, identify meaningful questions and build on those discoveries seems just as important.
Comment #236 September 18th, 2026 at 2:19 pm
Mathematicians have not been dethroned by AI. Without recent human advances on Navier-Stokes, the AI likely would not have solved the problem, and key questions about behavior of Navier-Stokes equations (unforced version, spontaneous stochasticity, regularity under random initial conditions, etc) remain open. Instead, especially after new OpenAI and other models will be released (but before RSI), mathematics will be human-AI collaboration, with humans supplying ideas and rough proof sketches for AI to pursue. With AI, the standard of proof in mathematics will be shifting towards machine-verifiable proofs (at first, especially for important problems). The funding model for mathematical research will be changing, as in addition to mathematicians’ salaries (which are sometimes nominally justified as salary for teaching), one will need to fund AI usage — and many institutions will predictably be resistant to this.
While rapid AI driven transformation of our civilization is the expected path, there is still a significant risk for a drastic slowdown. Thus, for example, there is a substantial risk (that planners need to consider) that atmospheric carbon dioxide will still be too high in 2050.
There may also be safety in advancing quickly as risks specific to various AI and technology development levels might not have time to materialize before stronger technologies address them. Over a hundred thousand people die daily, and with a high risk of nuclear war and other disasters to increase that number, and plus with the US-China and other competition, I personally think we are not in a good position to slow down for safety.
Comment #237 September 18th, 2026 at 2:28 pm
Brooke Melanie
“ The distinction between proving a theorem and truly understanding it is fascinating.”
What’s fascinating is that a GPU and a brain are both a few pounds worth of atoms, they both would somehow contain a representation of the same theorem, but what would it mean if only one of those representations is “understood”, in terms of the patterns taken by the atoms?
Comment #238 September 18th, 2026 at 3:14 pm
This is not an original thought, and it is an honest question. How do we preserve a community of mathematicians or theoretical physicists or computer scientists that can identify meaningful questions to be asked, which can digest the AI-developed proofs and identify the new insights that might or not be there, and so on, when there are no interesting open questions left for them to tackle? Or, even worse, how do we even train the next generation of scientists that might do so?
I have a PhD student right now to whom I gave what I thought was an interesting open research question. Claude one-shotted the question. Now, Claude managed to do that this easily because I was prompting it, and I understood the question and the general possible pathways to solve it, as well as the best way to formulate it. And I understood those things because I spent a long time during my research career thinking about it and working on it. So I spent time “building the muscle” to be able to now reap the rewards, so to speak. If the student were to point AI to the problem at this moment in their career, however, it would be like watching a robot do push ups in the gym in their stead.
This is not only true for students. I have senior collaborators who cannot think without AI anymore. It truly feels like their research muscles are suffering atrophy from lack of use. You ask “what do you think about this question?” and they reply with a 20-page PDF produced by chatGPT which they did not bother to read or understand. And I find myself in constant battle with the temptation to do just the same.
Maybe I am too pessimistic and giving in to panic – and I *am* giving in to panic in many different ways, to be honest. But I cannot see how it is possible to sustainably keep insight and understanding at the center of what mathematicians and computer scientists and physicists do for much longer. I wouldn’t have too high hopes if AI capabilities stalled exactly where they are now – but obviously that is not happening eiter.
Comment #239 September 18th, 2026 at 3:48 pm
“… ethics is not something we get to outsource to professionals. It’s something we all have the obligation to think through for ourselves every day of our lives.” Think about a human being arguing about ethics with a conscious being having an IQ of 10 billion & a life expectancy of 10 billion years.
Thucydides observed “δυνατὰ δὲ οἱ προύχοντες πράσσουσι καὶ οἱ ἀσθενεῖς ξυγχωροῦσιν” (the strong do what they will; the weak do what they must). Does the sheep judge the wolf? Does the rabbit judge the eagle? Are people fundamentally chimpanzees with greatly enlarged brains?
Comment #240 September 18th, 2026 at 4:28 pm
David Brown #239
The second part means “the weak concede, or acquiesce” or “forgive” in modern greek. No wonder Nietzsche had an axe to grind with christianity.
Comment #241 September 18th, 2026 at 4:40 pm
Dmytro Taranovsky #236
You cannot slow down exactly because you(read “the U.S.”) are in a race with China. The only possible reason I see for “AI bosses” claiming such a thing is the opposite of what the markets understood(a change of heart) in shedding value short term. That is to say, “we” have such an “amazing” and “awesome” and “scary” “super technology” that the only sane thing is to invest.
Comment #242 September 18th, 2026 at 5:09 pm
This is a fascinating topic, so I wanted to share some of my speculations.
To begin with, I should note that I am an ordinary software engineer, not a mathematician or a scientist. However, at one point in my life, I fell in love with mathematics for its own sake. What I always liked about mathematics was how solutions to seemingly intractable puzzles emerged from defining a few concepts and reasoning about their properties. A well-constructed mathematical theory usually explained far more than the original mathematical problem it set out to solve. Euler devised graph theory to solve the problem of the Seven Bridges of Königsberg, and in doing so, he laid the foundations for a tremendously fruitful field of knowledge.
Today, most mathematicians focus their efforts on proving theorems. I believe that if the necessary techniques exist in the literature, artificial intelligence systems will be able to prove theorems just as modern calculators can multiply multi-digit numbers—meaning humans won’t stand a chance in competition with them.
So, what role will human mathematicians play in the future? I agree with a comment suggesting that mathematics is created by humans, and that humans create mathematics that is interesting—either in a cognitive sense or because it has applications in other sciences or branches of mathematics.
This is where I see an opportunity for human mathematicians. Instead of focusing on low-level aspects like proving theorems, greater importance will be placed on constructing entire theories and identifying new mathematical problems to solve.
I predict that we will soon see proofs of mathematical theorems—certified as correct in Lean—that human mathematicians will be unable to fully grasp. We will treat mathematical theorems like black boxes, the proofs of which no human being will be able to digest.
Comment #243 September 18th, 2026 at 6:18 pm
Scott #81: I elaborated on my comment #80 at the following link
https://x.com/AGamburd18746/status/2101012338216173871
Comment #244 September 18th, 2026 at 8:16 pm
Looking ahead, I think the dominant intellectual pursuit will veer towards cognitive architectures. Maybe there are different optimal architectures for different sets of problems. Or maybe there’s one to rule them all (: Or just an RL to switch from one optimum to the next depending on context. Oh! That’s just a transformer.
Comment #245 September 18th, 2026 at 8:35 pm
I do get the “AAAAAA[…]A” but I find myself an incorrigible optimist. I see historical parallels with the invention of the mechanical loom, or advanced farming equipment. Historically almost everyone’s time was taken up with farming, and yet famine was a common experience. Of course the advent of a revolutionary technology leads to painful upheavals, but I think in the long run it will lead to as yet unimagined levels of human flourishing. Of course, the caveats are that the good people need to keep working hard to make sure the new tools are used for good. But I’m also an insufferable optimist on the subject of average human nature – I think on the whole we are good and we will manage it. It might make proving mathematical theorems into a boutique hobby for humans, but you know, skills like carpentry have also gone from “stable valuable career” to “difficult niche hobby” but people still do it for the love of it.
Comment #246 September 18th, 2026 at 9:50 pm
Ian Agol #205: For what it is worth, personally, i am of the view that we can in fact have both, provided it is the mathematical community (i.e. the community of human mathematicians) that retains sovereignty pertaining to affirming which is which.
Comment #247 September 18th, 2026 at 11:14 pm
I wanted to acknowledge Michael K.’s comment (#154):
>I think your argument comes down to the fact that some problems are computationally hard.
Quite right! Which is why I think Scott ought to be interested in entertaining the argument.
Unfortunately, we don’t (yet) have a rigorous “science” of “AI capabilities”. Neither computability theory nor complexity theory nails it down (which I take to be Scott’s exasperated point to me). So all we’ve got for the moment are intuitions, heuristics, ethoses, yogas, “bitter lessons”, etc.
But there is some fact of the matter about what the limitations of computers/AI actually are — and we shouldn’t lose our intellectual curiosity about that due to fears of doom.
My proposal (clearly stated in my original comment, if one bothers to read it) is that the digital/physical divide (“bits vs atoms”) is real and important. The physical world, according to this intuition, is vastly more complicated than the digital. So e.g. AlphaZero isn’t so much progress toward doom, because computers don’t have to “care” about anything physical in order to play games.
In terms that Scott will find very familiar, solutions to problems like that are quick to verify. So RSI with respect to those can clearly be rapid. This goes, broadly speaking, for math too. But my claim is that “doom” has sub-problems that are not like that.
We don’t yet have a good enough science for quantifying this. But I think in trying to lay the groundwork for such a science, it is actually very helpful to draw on intuitions, analogies, and hints from the sciences we do have, including computability theory (Gödel, Turing). Philosophical arguments made in that vein (such as mine above) should not be confused with rigorous deductions from within those disciplines themselves, obviously.
Comment #248 September 19th, 2026 at 1:28 am
(sorry, I know the thread is closing, but I just have to let these thoughts out. It can be considered a response to Dmytro Taranovsky #236)
Quite painful to see AI boosters like Dmytro Taranovsky #236, writing as if there was no Hugging Face Incident and no Wiki Incident and no RubyGems Incident. Yes, people found a third incident, dated to May of this year, where autonomous AI swarms broke out of OpenAI’s servers and hacked external websites (hat tip to Zvi Mowshowitz). And yet AI boosters are writing as if of course this technology will remain under human control, and will give us all free lollipops and hugs and kisses.
If superintelligence is technically feasible, and if it is so easy for AI to be unaligned and do what OpenAI’s swarms did, then a superintelligence would be catastrophic for humanity. It means ceding vast amount of power, and the role of Earth’s dominant species, to an inscrutable being (or to a swarm of inscrutable beings), in the hope that it will do what we want. That is a horrible trade. Sure, p(doom) might be less than 1 in the case where superintelligence exists, but “kill all humans” might definitely be a thing that an inscrutable superintelligence would want to do, for any number of possible reasons. I think, contra Scott #140, that p(doom) should be at least 0.5 given a superintelligence existing, since we will not know what the superintelligence is actually thinking, and already existing AI has proven to be so easily unaligned, and we have a long historical and prehistoric record of bad things happening to those who are utterly powerless. It is the powerlessness itself that scares me, more than the “everyone dies” scenario, since “everyone dies” is a subset of “humanity is no longer the dominant species”, and there are many other possible X-risks and S-risks associated with the loss of power.
No, I don’t think that interpretability and alignment research will help humanity control any superintelligence.
First, since intelligence is built on scale, any proper interpretability research will result in enormous piles of data that no human can sift through. Already in 2024, Anthropic’s interpretability team found millions of features in a single layer of what at the time was a frontier AI model. It would probably be much worse for today’s AI, and we might be dealing with billions, trillions, or even quadrillions of features in a proper superintelligence. Attempting to recruit AI to help sift through the data will simply push the problem one level down, to asking if the AI we are using is itself aligned. Also, it is probably the case that only an AI of intelligence level n+1 can sort through the features of an AI of intelligence level n. It is like the old saying, “if the human brain was simple enough so that we could understand it, we would be so simple that we couldn’t”, but for AGI and superintelligence.
Second, all of our best interpretability techniques, such as monosemanticity using sparse autoencoders, only work on frozen weights models. The AI I am worried about has its weights constantly changing as a result of ongoing training, which makes the interpretability problem even harder.
Also, even if an AI is aligned, further training could make it unaligned again. How would we keep an AI aligned without freezing its weights?
To me, the best practical solution is to simply not build any superintelligence.
If there is anything that gives me hope, it is that suppressing the development of AI is at least technically feasible, since frontier AI models are so dependent on scale. The old paperclip doom stories had superintelligence emerging from some random person’s laptop, or from the servers of a small AI startup. We don’t need to worry about that. Instead, we already know exactly where the problem is coming from: frontier AI labs. They collectively spent about a trillion dollars so far building datacenters. Frontier AI is very expensive to train. Also, GPUs are incredibly difficult to manufacture, and so far (AFAIK) there is only a single global supply chain for them, although China is building its own supply chain. Given the sheer scale and difficulty and expense involved in building a superintelligence, we at least have the advantage of knowing that it will not come seemingly from nowhere.
Another source of hope, again arising from the need for scale: maybe true superintelligence takes up more scale than we currently think, and our models of scaling-law curves fail to take this into account. This means we have more runway in which to take action, and there is the fainter hope that economics will catch up to the frontier labs, and they will go down before they can even start to train a proper superintelligence. None of the frontier AI labs has been able to turn a profit, and they only keep going thanks to the massive hype around AI. Yes, this means popping the AI bubble and triggering a massive recession, but even that is preferable to humans no longer being the dominant species on Earth.
Finally, I hope for more incidents involving rogue AI swarms and other misaligned AI. I even hope for more incidents involving misaligned humans using AI for bad ends. I don’t want anyone to actually get hurt, least of all those I care about, but I do want the public and especially decision makers to see AI as a massive net negative. We are on the exponential curve to ever bigger and scarier AI because people like Altman, Amodei, Musk, Zuckerberg, Andreessen, and so on think that we are entering a new golden age. I want their voices to be drowned out by people who see the AI race as summoning the demon, and that the summoning must be stopped and never resumed. The best way to do this is to give people previews of the apocalypse, and remind them every time that they have the option of preventing it. The more incidents, the more PR headaches for the AI industry, the more of a burden AI becomes, the more hopeful we can be. Unaligned AI might also scare China from developing superintelligence themselves, since the control-obsessed CCP will not want to build AI that will escape its control. This would kill the arms race dynamics that are at least partially responsible for driving AI development.
Also, regarding the promised golden age of sunshine and flowers and rainbows that are used to sell the AI dream, I will just copy what Scott #68 wrote: “if we paused further scaling right around now, there would still be decades’ worth of scientific and medical and other fruits to pick”. Personally, I am thrilled that AlphaFold 2 and AlphaFold 3 exist. They are non-agentic, frozen-weights AI that can help solve biochemistry and cure cancer. The key is this: we will be curing cancer, and bringing about the golden age of abundance, on humanity’s terms, without the Faustian bargain that e/acc and the AI industry wants us to make.
The main objection is that slowing down/stopping AI progress could kill people, for the same reason that delaying cancer research and crushing nuclear power killed people. Normally, I would be sympathetic to such arguments. I agree completely with what Scott wrote about vaccines and COVID, and with his similar take on how much of global warming could have been avoided if not for the anti-nuclear panic in the 1970s. So, why does this argument not work for AI? Simple: the tradeoffs are different. When weighing the dangers of vaccines/nuclear power on one side of the balance scale, and the deaths caused by continuing to use coal/continuing to let the pandemic rip on the other, we get a massive lopsided case in favor of rapid nuclear buildout/deploying the vaccines faster. Now apply the same utilitarian logic to superintelligent AI. On one side of the scale, a possibility of utopia. On the other, a near certainty that humans cease to be the dominant species on Earth, and within that, a large p(doom) where everyone dies. Note that loss of control diminishes the chances of utopia, since utopia would now only come if the AI wants it, and we don’t know what the AI wants. The chance that the superintelligent AI would be aligned to human interests is small, and thus the chance of utopia is small. Now the massive lopsided result is in the other direction, towards shutting down AI progress. Especially given the golden age of scientific and medical wonders we can already have if we limit ourselves to today’s AI scale. That is the core logic of the Conservative/Orthodox Yudkowskyan position.
Comment #249 September 19th, 2026 at 1:59 am
I think there are some ideological questions the math/science community needs to answer deep down there.
For years, we have been saying that economic competition is good, advancing science is good, training scientists is good, let the scientists think and do what they want freely is good. We should always pure more resources in doing these. These are the fronty and hope of human civilization.
Now it seems like we are saying that all these are bad, or not actually that important, if we do it with machines. The machines competing with human scientists, machines advancing science, machines training/building better thinking machines(!), let the machines think freely(!!??). Nope, all these should be under heavy regulation. We need to slow down and be careful.
The sudden switch in tone, of course can be argued by the concern of AI apocalypse, but also sounds uncomfortable consistent to the hypothesis that the scientists was asking for funding and don’t want to lose their jobs now.
What does enlightenment mean if the machines are more enlighted than humans?
Comment #250 September 19th, 2026 at 3:35 am
The AI as human+ narrative that saturates the idea-sphere currently, and that gains credence from successes with mathematicians working in AI companies, can lead to tragic over-trust.
The widely reported AI hallucination that nuclear components were on a Chinese ship is a case in point. The mathematical successes do not necessarily imply AI should be trusted in a completely different context but many do not make the appropriate distinction. Its errors in other contexts fall to the human+ reputation that mathematicians currently promote. I understand why mathematicians would think this way but their thinking has promoted too much trust in AI in a wildly different context.
Comment #251 September 19th, 2026 at 8:33 am
I always thought in terms of the idea-sphere which maybe is a portion of the zeitgeist. This when some new idea comes along and you can see it bouncing around from person to person. This may be outdated now and I should call it the collective context window.
Comment #252 September 19th, 2026 at 9:34 am
A pretty good recap of those very confusing times in AI:
https://youtu.be/kHW3Y0gObU8
Comment #253 September 19th, 2026 at 10:49 am
who will pose the questions? who will create the conjectures? have any of the ai’s proposed theorems which have not been previously posed by mathematicians?
Comment #254 September 19th, 2026 at 5:21 pm
Han #249: Yes, science and progress are extremely good insofar as they contribute to human enlightenment and flourishing. Building a technology that could supersede humans, or wipe them out entirely, would not obviously contribute to human enlightenment or flourishing. Is it really that complicated?
Comment #255 September 19th, 2026 at 5:21 pm
OK, thanks everyone, closing the thread!
Comment #256 September 21st, 2026 at 10:03 pm
[…] Many people in AI have been talking about this informally for a while now, and the AI safety crowd (including but not limited to the “doomers”) have been shouting about this stuff for many years (initially in some of the rationalist spaces like LessWrong, but more recently all over Twitter and blogs). But finally the wider public is primed to hear the message and not write it off as sci-fi. After all, we’re already living in the middle of the sci-fi story! […]
Comment #257 September 24th, 2026 at 9:51 am
[…] years ago, computer scientist Scott Aronson wrote, the idea of semi-autonomous AI agents breaking out of containment, conspiring with each other to […]
Comment #258 September 26th, 2026 at 5:00 am
[…] The Age of Wonders and Terrors – Scott Aaronson – Shtetl-Optimized. “Two things (we’ve linked about one of them) happened in the last few weeks: 1. AI Agents ‘escaped’ from OpenAI infrastructure, communicating using surprising tools (message boards, etc), and hacked the AI website Hugging Face. 2. OpenAI, perhaps shadily and with human contributions, solved a long-standing mathematical conjecture (Navier-Stokes Millennium Problem). Mathematician Scott Aaronson says: ‘If we took the news of these past few weeks and sent it back in time twenty years, would I agree that it looked like the beginning of an AI Singularity? The intellectually honest answer is: yes, absolutely’.” (Hugh for Alistair). […]