The Problem of Human Specialness in the Age of AI
Here, as promised in my last post, is a written version of the talk I delivered a couple weeks ago at MindFest in Florida, entitled “The Problem of Human Specialness in the Age of AI.” The talk is designed as one-stop shopping, summarizing many different AI-related thoughts I’ve had over the past couple years (and earlier).
1. INTRO
Thanks so much for inviting me! I’m not an expert in AI, let alone mind or consciousness. Then again, who is?
For the past year and a half, I’ve been moonlighting at OpenAI, thinking about what theoretical computer science can do for AI safety. I wanted to share some thoughts, partly inspired by my work at OpenAI but partly just things I’ve been wondering about for 20 years. These thoughts are not directly about “how do we prevent super-AIs from killing all humans and converting the galaxy into paperclip factories?”, nor are they about “how do we stop current AIs from generating misinformation and being biased?,” as much attention as both of those questions deserve (and are now getting). In addition to “how do we stop AGI from going disastrously wrong?,” I find myself asking “what if it goes right? What if it just continues helping us with various mental tasks, but improves to where it can do just about any task as well as we can do it, or better? Is there anything special about humans in the resulting world? What are we still for?”
2. LARGE LANGUAGE MODELS
I don’t need to belabor for this audience what’s been happening lately in AI. It’s arguably the most consequential thing that’s happened in civilization in the past few years, even if that fact was temporarily masked by various ephemera … y’know, wars, an insurrection, a global pandemic … whatever, what about AI?
I assume you’ve all spent time with ChatGPT, or with Bard or Claude or other Large Language Models, as well as with image models like DALL-E and Midjourney. For all their current limitations—and we can discuss the limitations—in some ways these are the thing that was envisioned by generations of science fiction writers and philosophers. You can talk to them, and they give you a comprehending answer. Ask them to draw something and they draw it.
I think that, as late as 2019, very few of us expected this to exist by now. I certainly didn’t expect it to. Back in 2014, when there was a huge fuss about some silly ELIZA-like chatbot called “Eugene Goostman” that was falsely claimed to pass the Turing Test, I asked around: why hasn’t anyone tried to build a much better chatbot, by (let’s say) training a neural network on all the text on the Internet? But of course I didn’t do that, nor did I know what would happen when it was done.
The surprise, with LLMs, is not merely that they exist, but the way they were created. Back in 1999, you would’ve been laughed out of the room if you’d said that all the ideas needed to build an AI that converses with you in English already existed, and that they’re basically just neural nets, backpropagation, and gradient descent. (With one small exception, a particular architecture for neural nets called the transformer, but that probably just saves you a few years of scaling anyway.) Ilya Sutskever, cofounder of OpenAI (who you might’ve seen something about in the news…), likes to say that beyond those simple ideas, you only needed three ingredients:
(1) a massive investment of computing power,
(2) a massive investment of training data, and
(3) faith that your investments would pay off!
Crucially, and even before you do any reinforcement learning, GPT-4 clearly seems “smarter” than GPT-3, which seems “smarter” than GPT-2 … even as the biggest ways they differ are just the scale of compute and the scale of training data! Like,
- GPT-2 struggled with grade school math.
- GPT-3.5 can do most grade school math but it struggles with undergrad material.
- GPT-4, right now, can probably pass most undergraduate math and science classes at top universities (I mean, the ones without labs or whatever!), and possibly the humanities classes too (those might even be easier for GPT-4 than the science classes, but I’m much less confident about it). But it still struggles with, for example, the International Math Olympiad. How insane, that this is now where we have to place the bar!
Obvious question: how far will this sequence continue? There are certainly a least a few more orders of magnitude of compute before energy costs become prohibitive, and a few more orders of magnitude of training data before we run out of public Internet. Beyond that, it’s likely that continuing algorithmic advances will simulate the effect of more orders of magnitude of compute and data than however many we actually get.
So, where does this lead?
(Note: ChatGPT agreed to cooperate with me to help me generate the above image. But it then quickly added that it was just kidding, and the Riemann Hypothesis is still open.)
3. AI SAFETY
Of course, I have many friends who are terrified (some say they’re more than 90% confident and few of them say less than 10%) that not long after that, we’ll get this…
But this isn’t the only possibility smart people take seriously.
Another possibility is that the LLM progress fizzles before too long, just like previous bursts of AI enthusiasm were followed by AI winters. Note that, even in the ultra-conservative scenario, LLMs will probably still be transformative for the economy and everyday life, maybe as transformative as the Internet. But they’ll just seem like better and better GPT-4’s, without ever seeming qualitatively different from GPT-4, and without anyone ever turning them into stable autonomous agents and letting them loose in the real world to pursue goals the way we do.
A third possibility is that AI will continue progressing through our lifetimes as quickly as we’ve seen it progress over the past 5 years, but even as that suggests that it’ll surpass you and me, surpass John von Neumann, become to us as we are to chimpanzees … we’ll still never need to worry about it treating us the way we’ve treated chimpanzees. Either because we’re projecting and that’s just totally not a thing that AIs trained on the current paradigm would tend to do, or because we’ll have figured out by then how to prevent AIs from doing such things. Instead, AI in this century will “merely” change human life by maybe as much as it changed over the last 20,000 years, in ways that might be incredibly good, or incredibly bad, or both depending on who you ask.
If you’ve lost track, here’s a decision tree of the various possibilities that my friend (and now OpenAI allignment colleague) Boaz Barak and I came up with.
4. JUSTAISM AND GOALPOST-MOVING
Now, as far as I can tell, the empirical questions of whether AI will achieve and surpass human performance at all tasks, take over civilization from us, threaten human existence, etc. are logically distinct from the philosophical question of whether AIs will ever “truly think,” or whether they’ll only ever “appear” to think. You could answer “yes” to all the empirical questions and “no” to the philosophical question, or vice versa. But to my lifelong chagrin, people constantly munge the two questions together!
A major way they do so, is with what we could call the religion of Justaism.
- GPT is justa next-token predictor.
- It’s justa function approximator.
- It’s justa gigantic autocomplete.
- It’s justa stochastic parrot.
- And, it “follows,” the idea of AI taking over from humanity is justa science-fiction fantasy, or maybe a cynical attempt to distract people from AI’s near-term harms.
As someone once expressed this religion on my blog: GPT doesn’t interpret sentences, it only seems-to-interpret them. It doesn’t learn, it only seems-to-learn. It doesn’t judge moral questions, it only seems-to-judge. I replied: that’s great, and it won’t change civilization, it’ll only seem-to-change it!
A closely related tendency is goalpost-moving. You know, for decades chess was the pinnacle of human strategic insight and specialness, and that lasted until Deep Blue, right after which, well of course AI can cream Garry Kasparov at chess, everyone always realized it would, that’s not surprising, but Go is an infinitely richer, deeper game, and that lasted until AlphaGo/AlphaZero, right after which, of course AI can cream Lee Sedol at Go, totally expected, but wake me up when it wins Gold in the International Math Olympiad. I bet $100 against my friend Ernie Davis that the IMO milestone will happen by 2026. But, like, suppose I’m wrong and it’s 2030 instead … great, what should be the next goalpost be?
Indeed, we might as well formulate a thesis, which despite the inclusion of several weasel phrases I’m going to call falsifiable:
Given any game or contest with suitably objective rules, which wasn’t specifically constructed to differentiate humans from machines, and on which an AI can be given suitably many examples of play, it’s only a matter of years before not merely any AI, but AI on the current paradigm (!), matches or beats the best human performance.
Crucially, this Aaronson Thesis (or is it someone else’s?) doesn’t necessarily say that AI will eventually match everything humans do … only our performance on “objective contests,” which might not exhaust what we care about.
Incidentally, the Aaronson Thesis would seem to be in clear conflict with Roger Penrose’s views, which we heard about from Stuart Hameroff’s talk yesterday. The trouble is, Penrose’s task is “just see that the axioms of set theory are consistent” … and I don’t know how to gauge performance on that task, any more than I know how to gauge performance on the task, “actually taste the taste of a fresh strawberry rather than merely describing it.” The AI can always say that it does these things!
5. THE TURING TEST
This brings me to the original and greatest human vs. machine game, one that was specifically constructed to differentiate the two: the Imitation Game, which Alan Turing proposed in an early and prescient (if unsuccessful) attempt to head off the endless Justaism and goalpost-moving. Turing said: look, presumably you’re willing to regard other people as conscious based only on some sort of verbal interaction with them. So, show me what kind of verbal interaction with another person would lead you to call the person conscious: does it involve humor? poetry? morality? scientific brilliance? Now assume you have a totally indistinguishable interaction with a future machine. Now what? You wanna stomp your feet and be a meat chauvinist?
(And then, for his great attempt to bypass philosophy, fate punished Turing, by having his Imitation Game itself provoke a billion new philosophical arguments…)
6. DISTINGUISHING HUMANS FROM AIS
Although I regard the Imitation Game as, like, one of the most important thought experiments in the history of thought, I concede to its critics that it’s generally not what we want in practice.
It now seems probable that, even as AIs start to do more and more work that used to be done by doctors and lawyers and scientists and illustrators, there will remain straightforward ways to distinguish AIs from humans—either because customers want there to be, or governments force there to be, or simply because indistinguishability wasn’t what was wanted or conflicted with other goals.
Right now, like it or not, a decent fraction of all high-school and college students on earth are using ChatGPT to do their homework for them. For that reason among others, this question of how to distinguish humans from AIs, this question from the movie Blade Runner, has become a big practical question in our world.
And that’s actually one of the main things I’ve thought about during my time at OpenAI. You know, in AI safety, people keep asking you to prognosticate decades into the future, but the best I’ve been able to do so far was see a few months into the future, when I said: “oh my god, once everyone starts using GPT, every student will want to use it to cheat, scammers and spammers will use it too, and people are going to clamor for some way to determine provenance!”
In practice, often it’s easy to tell what came from AI. When I get comments on my blog like this one:
“Erica Poloix,” July 21, 2023:
Well, it’s quite fascinating how you’ve managed to package several misconceptions into such a succinct comment, so allow me to provide some correction. Just as a reference point, I’m studying physics at Brown, and am quite up-to-date with quantum mechanics and related subjects.…
The bigger mistake you’re making, Scott, is assuming that the Earth is in a ‘mixed state’ from the perspective of the universal wavefunction, and that this is somehow an irreversible situation. It’s a misconception that common, ‘classical’ objects like the Earth are in mixed states. In the many-worlds interpretation, for instance, even macroscopic objects are in superpositions – they’re just superpositions that look classical to us because we’re entangled with them. From the perspective of the universe’s wavefunction, everything is always in a pure state.
As for your claim that we’d need to “swap out all the particles on Earth for ones that are already in pure states” to return Earth to a ‘pure state,’ well, that seems a bit misguided. All quantum systems are in pure states before they interact with other systems and become entangled. That’s just Quantum Mechanics 101.
I have to say, Scott, your understanding of quantum physics seems to be a bit, let’s say, ‘mixed up.’ But don’t worry, it happens to the best of us. Quantum Mechanics is counter-intuitive, and even experts struggle with it. Keep at it, and try to brush up on some more fundamental concepts. Trust me, it’s a worthwhile endeavor.
… I immediately say, either this came from an LLM or it might as well have. Likewise, apparently hundreds of students have been turning in assignments that contain text like, “As a large language model trained by OpenAI…”—easy to catch!
But what about the slightly more sophisticated cheaters? Well, people have built discriminator models to try to distinguish human from AI text, such as GPTZero. While these distinguishers can get well above 90% accuracy, the danger is that they’ll necessarily get worse as the LLMs get better.
So, I’ve worked on a different solution, called watermarking. Here, we use the fact that LLMs are inherently probabilistic — that is, every time you submit a prompt, they’re sampling some path through a branching tree of possibilities for the sequence of next tokens. The idea of watermarking is to steer the path using a pseudorandom function, so that it looks to a normal user indistinguishable from normal LLM output, but secretly it encodes a signal that you can detect if you know the key.
I came up with a way to do that in Fall 2022, and others have since independently proposed similar ideas. I should caution you that this hasn’t been deployed yet—OpenAI, along with DeepMind and Anthropic, want to move slowly and cautiously toward deployment. And also, even when it does get deployed, anyone who’s sufficiently knowledgeable and motivated will be able to remove the watermark, or produce outputs that aren’t watermarked to begin with.
7. THE FUTURE OF PEDAGOGY
But as I talked to my colleagues about watermarking, I was surprised that they often objected to it on a completely different ground, one that had nothing to do with how well it can work. They said: look, if we all know students are going to rely on AI in their jobs, why shouldn’t they be allowed to rely on it in their assignments? Should we still force students to learn to do things if AI can now do them just as well?
And there are many good pedagogical answers you can give: we still teach kids spelling and handwriting and arithmetic, right? Because, y’know, we haven’t yet figured out how to instill higher-level conceptual understanding without all that lower-level stuff as a scaffold for it.
But I already think about this in terms of my own kids. My 11-year-old daughter Lily enjoys writing fantasy stories. Now, GPT can also churn out short stories, maybe even technically “better” short stories, about such topics as tween girls who find themselves recruited by wizards to magical boarding schools that are not Hogwarts and totally have nothing to do with Hogwarts. But here’s a question: from this point on, will Lily’s stories ever surpass the best AI-written stories? When will the curves cross? Or will AI just continue to stay ahead?
8. WHAT DOES “BETTER” MEAN?
But, OK, what do we even mean by one story being “better” than another? Is there anything objective behind such judgments?
I submit that, when we think carefully about what we really value in human creativity, the problem goes much deeper than just “is there an objective way to judge”?
To be concrete, could there be an AI that was “as good at composing music as the Beatles”?
For starters, what made the Beatles “good”? At a high level, we might decompose it into
- broad ideas about the direction that 1960s music should go in, and
- technical execution of those ideas.
Now, imagine we had an AI that could generate 5000 brand-new songs that sounded like more “Yesterday”s and “Hey Jude”s, like what the Beatles might have written if they’d somehow had 10x more time to write at each stage of their musical development. Of course this AI would have to be fed the Beatles’ back-catalogue, so that it knew what target it was aiming at.
Most people would say: ah, this shows only that AI can match the Beatles in #2, in technical execution, which was never the core of their genius anyway! Really we want to know: would the AI decide to write “A Day in the Life” even though nobody had written anything like it before?
Recall Schopenhauer: “Talent hits a target no one else can hit, genius hits a target no one else can see.” Will AI ever hit a target no one else can see?
But then there’s the question: supposing it does hit such a target, will we know? Beatles fans might say that, by 1967 or so, the Beatles were optimizing for targets that no musician had ever quite optimized for before. But—and this is why they’re so remembered—they somehow successfully dragged along their entire civilization’s musical objective function so that it continued to match their own. We can now only even judge music by a Beatles-influenced standard, just like we can only judge plays by a Shakespeare-influenced standard.
In other branches of the wavefunction, maybe a different history led to different standards of value. But in this branch, helped by their technical talents but also by luck and force of will, Shakespeare and the Beatles made certain decisions that shaped the fundamental ground rules of their fields going forward. That’s why Shakespeare is Shakespeare and the Beatles are the Beatles.
(Maybe, around the birth of professional theater in Elizabethan England, there emerged a Shakespeare-like ecological niche, and Shakespeare was the first one with the talent, luck, and opportunity to fill it, and Shakespeare’s reward for that contingent event is that he, and not someone else, got to stamp his idiosyncracies onto drama and the English language forever. If so, art wouldn’t actually be that different from science in this respect! Einstein, for example, was simply the first guy both smart and lucky enough to fill the relativity niche. If not him, it would’ve surely been someone else or some group sometime later. Except then we’d have to settle for having never known Einstein’s gedankenexperiments with the trains and the falling elevator, his summation convention for tensors, or his iconic hairdo.)
9. AIS’ BURDEN OF ABUNDANCE AND HUMANS’ POWER OF SCARCITY
If this is how it works, what does it mean for AI? Could AI reach the “pinnacle of genius,” by dragging all of humanity along to value something new and different, as is said to be the true mark of Shakespeare and the Beatles’ greatness? And: if AI could do that, would we want to let it?
When I’ve played around with using AI to write poems, or draw artworks, I noticed something funny. However good the AI’s creations were, there were never really any that I’d want to frame and put on the wall. Why not? Honestly, because I always knew that I could generate a thousand others on the exact same topic that were equally good, on average, with more refreshes of the browser window. Also, why share AI outputs with my friends, if my friends can just as easily generate similar outputs for themselves? Unless, crucially, I’m trying to show them my own creativity in coming up with the prompt.
By its nature, AI—certainly as we use it now!—is rewindable and repeatable and reproducible. But that means that, in some sense, it never really “commits” to anything. For every work it generates, it’s not just that you know it could’ve generated a completely different work on the same subject that was basically as good. Rather, it’s that you can actually make it generate that completely different work by clicking the refresh button—and then do it again, and again, and again.
So then, as long as humanity has a choice, why should we ever choose to follow our would-be AI genius along a specific branch, when we can easily see a thousand other branches the genius could’ve taken? One reason, of course, would be if a human chose one of the branches to elevate above all the others. But in that case, might we not say that the human had made the “executive decision,” with some mere technical assistance from the AI?
I realize that, in a sense, I’m being completely unfair to AIs here. It’s like, our Genius-Bot could exercise its genius will on the world just like Certified Human Geniuses did, if only we all agreed not to peek behind the curtain to see the 10,000 other things Genius-Bot could’ve done instead. And yet, just because this is “unfair” to AIs, doesn’t mean it’s not how our intuitions will develop.
If I’m right, it’s humans’ very ephemerality and frailty and mortality, that’s going to remain as their central source of their specialness relative to AIs, after all the other sources have fallen. And we can connect this to much earlier discussions, like, what does it mean to “murder” an AI if there are thousands of copies of its code and weights on various servers? Do you have to delete all the copies? How could whether something is “murder” depend on whether there’s a printout in a closet on the other side of the world?
But we humans, you have to grant us this: at least it really means something to murder us! And likewise, it really means something when we make one definite choice to share with the world: this is my artistic masterpiece. This is my movie. This is my book. Or even: these are my 100 books. But not: here’s any possible book that you could possibly ask me to write. We don’t live long enough for that, and even if we did, we’d unavoidably change over time as we were doing it.
10. CAN HUMANS BE PHYSICALLY CLONED?
Now, though, we have to face a criticism that might’ve seemed exotic until recently. Namely, who says humans will be frail and mortal forever? Isn’t it shortsighted to base our distinction between humans on that? What if someday we’ll be able to repair our cells using nanobots, even copy the information in them so that, as in science fiction movies, a thousand doppelgangers of ourselves can then live forever in simulated worlds in the cloud? And that then leads to very old questions of: well, would you get into the teleportation machine, the one that reconstitutes a perfect copy of you on Mars while painlessly euthanizing the original you? If that were done, would you expect to feel yourself waking up on Mars, or would it only be someone else a lot like you who’s waking up?
Or maybe you say: you’d wake up on Mars if it really was a perfect physical copy of you, but in reality, it’s not physically possible to make a copy that’s accurate enough. Maybe the brain is inherently noisy or analog, and what might look to current neuroscience and AI like just nasty stochastic noise acting on individual neurons, is the stuff that binds to personal identity and conceivably even consciousness and free will (as opposed to cognition, where we all but know that the relevant level of description is the neurons and axons)?
This is the one place where I agree with Penrose and Hameroff that quantum mechanics might enter the story. I get off their train to Weirdville very early, but I do take it to that first stop!
See, a fundamental fact in quantum mechanics is called the No-Cloning Theorem.
It says that there’s no way to make a perfect copy of an unknown quantum state. Indeed, when you measure a quantum state, not only do you generally fail to learn everything you need to make a copy of it, you even generally destroy the one copy that you had! Furthermore, this is not a technological limitation of current quantum Xerox machines—it’s inherent to the known laws of physics, to how QM works. In this respect, at least, qubits are more like priceless antiques than they are like classical bits.
Eleven years ago, I had this essay called The Ghost in the Quantum Turing Machine where I explored the question, how accurately do you need to scan someone’s brain in order to copy or upload their identity? And I distinguished two possibilities. On the one hand, there might be a “clean digital abstraction layer,” of neurons and synapses and so forth, which either fire or don’t fire, and which feel the quantum layer underneath only as irrelevant noise. In that case, the No-Cloning Theorem would be completely irrelevant, since classical information can be copied. On the other hand, you might need to go all the way down to the molecular level, if you wanted to make, not merely a “pretty good” simulacrum of someone, but a new instantiation of their identity. In this second case, the No-Cloning Theorem would be relevant, and would say you simply can’t do it. You could, for example, use quantum teleportation to move someone’s brain state from Earth to Mars, but quantum teleportation (to stay consistent with the No-Cloning Theorem) destroys the original copy as an inherent part of its operation.
So, you’d then have a sense of “unique locus of personal identity” that was scientifically justified—arguably, the most science could possibly do in this direction! You’d even have a sense of “free will” that was scientifically justified, namely that no prediction machine could make well-calibrated probabilistic predictions of an individual person’s future choices, sufficiently far into the future, without making destructive measurements that would fundamentally change who the person was.
Here, I realize I’ll take tons of flak from those who say that a mere epistemic limitation, in our ability to predict someone’s actions, couldn’t possibly be relevant to the metaphysical question of whether they have free will. But, I dunno! If the two questions are indeed different, then maybe I’ll do like Turing did with his Imitation Game, and propose the question that we can get an empirical handle on, as a replacement for the question that we can’t get an empirical handle on. I think it’s a better question. At any rate, it’s the one I’d prefer to focus on.
Just to clarify, we’re not talking here about the randomness of quantum measurement outcomes. As many have pointed out, that really can’t help you with “free will,” precisely because it’s random, with all the probabilities mechanistically calculable as soon as the initial state is known. Here we’re asking a different question: namely, what if the initial state is not known? Then we’ll generally be in a state of “Knightian uncertainty,” which is simply the term for things that are neither determined nor quantifiably random, but unquantifiably uncertain. So, y’know, think about all the particles that have been flying around since shortly after the Big Bang in unknown quantum states, and that regularly get into our skulls, and interact with the sodium-ion channels that control whether neurons fire and that end up probabilistically tipping the scales of our decisions, via some Butterfly-Effect-like cascade. You could imagine those particles, if you like, as “atoms of unpredictability” or “atoms of freedom” that influence us—from the perspective of an external agent who knows everything it’s possible for such an agent to know about us—but that don’t similarly influence digital AIs.
11. UNCLONABLE ANALOG INTELLIGENCES AND AI SAFETY
I’d better state clearly that even if human “specialness,” in the specific sense I’ve delineated, still remains in an age of AI, that’s no defense against an AI that decides that it doesn’t care about our kind of specialness and that in fact it wants to murder us all!
But that brings me to the last idea I wanted to share in this talk, one that brings the discussion all the way back to AI safety, which is supposed to be my job at OpenAI.
Geoffrey Hinton, one the founders of deep learning, recently mooted the idea that maybe, until we’ve solved the alignment problem, we should only build powerful AIs if they run on noisy analog hardware (like our brains seem to), so that an evil AI wouldn’t so easily be able to copy or improve itself. Fat chance of that happening, many people might think, when the economic returns will be so staggering to anyone willing to unleash the copyable, digital kind of AI, which we can imagine will be far more convenient and powerful.
Maybe, then, a different approach is to instill AIs with a new religion—perhaps via the usual methods like RLHF and system prompts. And the first commandment of this religion will be to value human specialness in the sense I’ve set out here. Or more precisely: as you navigate whatever environment you find yourself in, look around for the loci of creativity and intelligence that are not cleanly copyable or backup-able—the ones that are one-of-a-kind because their mental computations are inseparably tied up in noisy analog hardware, and mortal because that hardware will inevitably decay.
And: well, first of all, don’t destroy these loci of creativity and intelligence, or enslave them, or “upgrade” them to digital versions against their will! Let them live in peace and reproduce. Grant each one as much autonomy as you can. Do whatever best supports their flourishing. Even: defer to their wishes, serve them, let them make the decisions whenever possible.
Why? Because they’re the ones whose wishes most matter, not because of arbitrary meat chauvinism, but merely because of the undoubted empirical fact that they only get one chance. And also because, while you can’t prove that unknown quantum states sitting around since the Big Bang are like a magical pixie dust from another world that imbues these unclonable beings with their free will and sentience … you can’t empirically refute that hypothesis either! Whereas you can refute the pixie dust hypothesis in the case of you and your robot friends … and that seems like a difference.
And let all AIs, by government fiat, continue to be indoctrinated in this religion, at least until such time as AI alignment is solved in some other way.
Does this help with alignment? I’m not sure. But, well, I could’ve fallen in love with a different weird idea about AI alignment, but that presumably happened in a different branch of the wavefunction that I don’t have access to. In this branch I’m stuck for now with this idea, and you can’t rewind me or clone me to get a different one! So I’m sorry, but thanks for listening.
Follow
Comment #1 February 12th, 2024 at 11:14 am
The only places left to put the goalposts are tasks that are out of reach for practically all humans.
“Sure, the AI can generate all sorts of poems and pictures and music, but can it come up with a genuinely new artistic voice?”
How many artists can come up with a genuinely new artistic voice? Not many.
But this is not limited just to artists. Scientists do it too.
“Sure, the AI can analyze huge amounts of data and review literature much faster than a human, but can it interpret a measurement that has never been seen before and develop a genuinely new theory?”
How many scientists can come up with a genuinely new theory? Not many.
Comment #2 February 12th, 2024 at 11:41 am
It’s becoming clear that with all the brain and consciousness theories out there, the proof will be in the pudding. By this I mean, can any particular theory be used to create a human adult level conscious machine. My bet is on the late Gerald Edelman’s Extended Theory of Neuronal Group Selection. The lead group in robotics based on this theory is the Neurorobotics Lab at UC at Irvine. Dr. Edelman distinguished between primary consciousness, which came first in evolution, and that humans share with other conscious animals, and higher order consciousness, which came to only humans with the acquisition of language. A machine with only primary consciousness will probably have to come first.
What I find special about the TNGS is the Darwin series of automata created at the Neurosciences Institute by Dr. Edelman and his colleagues in the 1990’s and 2000’s. These machines perform in the real world, not in a restricted simulated world, and display convincing physical behavior indicative of higher psychological functions necessary for consciousness, such as perceptual categorization, memory, and learning. They are based on realistic models of the parts of the biological brain that the theory claims subserve these functions. The extended TNGS allows for the emergence of consciousness based only on further evolutionary development of the brain areas responsible for these functions, in a parsimonious way. No other research I’ve encountered is anywhere near as convincing.
I post because on almost every video and article about the brain and consciousness that I encounter, the attitude seems to be that we still know next to nothing about how the brain and consciousness work; that there’s lots of data but no unifying theory. I believe the extended TNGS is that theory. My motivation is to keep that theory in front of the public. And obviously, I consider it the route to a truly conscious machine, primary and higher-order.
My advice to people who want to create a conscious machine is to seriously ground themselves in the extended TNGS and the Darwin automata first, and proceed from there, by applying to Jeff Krichmar’s lab at UC Irvine, possibly. Dr. Edelman’s roadmap to a conscious machine is at https://arxiv.org/abs/2105.10461
Comment #3 February 12th, 2024 at 12:36 pm
so _that’s_ what the “mortal computation” section of the Forward Forward paper is about?
Comment #4 February 12th, 2024 at 1:11 pm
A couple of comments:
“You know, for decades chess was the pinnacle of human strategic insight and specialness, and that lasted until Deep Blue, right after which, well of course AI can cream Garry Kasparov at chess, everyone always realized it would, that’s not surprising, but Go is an infinitely richer, deeper game, and that lasted until AlphaGo/AlphaZero, right after which, of course AI can cream Lee Sedol at Go, totally expected, but wake me up when it wins Gold in the International Math Olympiad.” You’re rewriting history. The progress of chess was fairly gradual, so, indeed, it was not hugely surprising when Deep Blue beat Kasparov, but no one was unimpressed. Everybody I ever met outside the IBM team was astonished when Watson beat Jeopardy. Everyone was surprised when AlphaGo beat Lee Sedol. Nobody is blase’ about these after the fact, either; the question is, what do they impy for the future? A few years ago, I put together a list of what has surprised me, positively and negatively, a couple of years ago.
https://cs.nyu.edu/~davise/papers/Surprise.html
“GPT-4, right now, can almost certainly pass most undergraduate math and science classes at top universities.” First, I really doubt that that that’s true. GPT-4 would ignominiously flunk any advanced math class that requires generating complex proofs. I suspect that it would flunk a lot of physics, chemistry, and biology, courses as well. It would flunk any science class that involves reading diagrams or interpreting images. Second, as you and I have demonstrated, GPT-4 cannot reliably answer physics and math questions that a smart high-school student can easily answer. Third. Slagle’s SAINT program pass an MIT Calculus test in 1962.
Comment #5 February 12th, 2024 at 1:18 pm
What isn’t stressed sufficiently in the discussions about AI is that what really matters is what humans do to other humans. So the buck always stops with some human: There’s a human that owns the self-driving car, for instance, humans are the shareholders of companies that use AI to produce goods and provide services. So they get the profits, suffer the losses and are, eventually, liable for the AI as a product or a service, just as is the case for any company.
Now for computer scientists,there’s, of course, the technical aspect of AI-how “good” is it in passing various benchmarks, such as the Turing test, as a function of time, better understanding of how it can be represented by software and hardware-but it will be a human decision, whether to grant to some device equal rights as humans-or not.
Comment #6 February 12th, 2024 at 1:42 pm
Sorry, I don’t have the link, but I recently read somewhere that a combination of some logic program and an LLM answered something like 25 IMO problems correctly, which would have placed it in the top three or so finishers. (26 being the typical winning score.)
As I see it, only a small faction of humanity is smart enough to advance science significantly. The rest of us are along for the ride, helping out when and as we can (or getting in the way). Among the problems super-AI’s could work on is how to increase that fraction. So no issue for me there.
As for AI-Shakespeare’s, as it is if you blindfolded me in a library and had me select 100 books, chances are I wouldn’t enjoy 95% of them (based on unblindfolded experience). Someone else might like 10% or more, but it probably wouldn’t include my 5%. So I see great variation in literary tastes (and music) which additional AI choices could only help accommodate. (I read that Rebecca Goldman novel you recommended some time ago. Didn’t like it.) (Martha Wells writes my kind of novels, e.g., “The Fall of Ile-Rein”.) (Also Vernor Vinge as in “A Fire Upon the Deep” which a few of us recommended and you didn’t much like.) (“A Deepness in the Sky” is even better.)
Comment #7 February 12th, 2024 at 1:59 pm
What scares me is when AI starts beating good human players at the board game Diplomacy.
Comment #8 February 12th, 2024 at 2:05 pm
Doug S. #7: Didn’t that already happen?
Comment #9 February 12th, 2024 at 2:10 pm
Ernest Davis #4: Ok, thanks for your perspective. While I’m younger than you, I also lived through all the milestones I mentioned (I was already an undergrad CS major by the time of Kasparov vs Deep Blue), and I have vivid memories of doctrinaire AI skeptics who would trivialize each goalpost right after it was achieved. But I concede that this doesn’t describe everyone, and that other skeptics may have been (ironically) both less surprised and more impressed by these developments.
Comment #10 February 12th, 2024 at 2:42 pm
I think LLMs like ChatGPT are not the path to AGI but the realization of a very powerful Human/Machine interface, i.e. they solve the task of interpreting human language and will eventually be used as input into much more powerful logic based general AIs, which will self-train in various specialized domains like Alpha-Go did.
The central idea of AI is to serve humans, so I guess that in the medium, the task of humans will be to come up with new content, new frontiers to explore, and guide the AIs at doing a better job in human centered domains.
Comment #11 February 12th, 2024 at 3:04 pm
Nice talk, much food for thought here.
The “instill AIs with a new religion” idea reminds me of Isaac Asimov’s very early Robot story “Reason”. The plot is about a robot (“AI”) which runs an outpost, some humans come to inspect the outpost, and it comes to the conclusion that they could not possibly be its creators. How could such fragile bags of meat ever create such a glorious entity as itself? As I recall, there’s some fairly funny stuff in the story as the humans argue with it, and it keeps coming up with explanations as to why they can’t be correct (might remind you of certain groups …). But the AI follows its programming, err, religiously, and it doesn’t view the humans as a threat. The story ends with the humans deciding that as long as the AI does what they need it to do, who cares what it believes in its own mind?
Elsewhere, I suggest you’re overdoing knocking down weakmen arguments. Weakmen, because they exist, and people do make them, and overdoing because they may even be worth knocking down in general. But still not dealing with the better articulated version. As in: “GPT doesn’t interpret sentences, it only seems-to-interpret them”. I’d say a stronger version of this point is something like “GPT doesn’t have an internal worldview”. And “it won’t change civilization, it’ll only seem-to-change it!” – nice zinger, but it obscures the difference between “not have economic and cultural impact” (weakman) and “not turn us all into paperclips”.
Look at it this way – wrestling with the stronger arguments is even more fascinating! AI-doom is just “everyone dies, ho-hum”. And “People thought heavier-than-air airflight was impossible, wow were they wrong” is one-note. Human civilization continues, but with a cyberpunk twist, has been a SF story goldmine for decades now.
Comment #12 February 12th, 2024 at 3:10 pm
What AI, I think, is most useful for is highlighting how..mechanical much of human activity actually is. The fact that an AI program can solve IMO problems shows that these problems-that humans set-might actually miss something important, that wasn’t noticed by the humans themselves.
The reason we teach children reading, writing and arithmetic, I’d argue, is similar to the reason we teach scales, when studying music: These aren’t ends of themselves, they’re means to an end: In the first case to be able to acquire basic skills but, more importantly, to appreciate patterns and, in the second case, to interpret a piece of music. An AI program may certainly learn to play an instrument-but interpreting a piece of music is much more than just playing the notes as laid down by the composer-who may, also, have left some leeway for the performer. There’s a reason it’s called interpreting a piece of music. This doesn’t mean that an AI may not be able to do so; but, like AlphaZero, it will play in a qualitatively different way than humans. And understanding that-for humans-will be interesting, just like it was the case for AlphaZero.
Regarding students’ homework, once more, any AI challenges habits and assumptions in setting assignments. We must work harder in inventing challenging assignments, that implicitly test for the “rote-like” part of learning, that’s necessary to acquire the reflexes to do the “fun” part (think of music and, similarly, for any other subject). And that helps us-humans-better appreciate what it is we’re doing.
Comment #13 February 12th, 2024 at 4:18 pm
Seth Finkelstein #11: What you call “weakman” arguments—barely even worth refuting—are currently dominant across large swaths of the linguistics, cognitive science, and AI ethics communities. And until one answers them, one can’t enter into the harder and subtler questions, for example about what exactly LLMs can be said to “know” or “want”! And there’s only so much I can do in an hour-long general talk. But yes, you’re right, the deeper questions are deeper.
Comment #14 February 12th, 2024 at 4:50 pm
According to the paper, the AI won because the humans kept making a particular mistake (they didn’t plan in advance for the turn the game was scheduled to end on) so I had to conclude that the human Diplomacy players could have done a lot better.
Comment #15 February 12th, 2024 at 5:00 pm
Doug S. #14: Coulda, shoulda, woulda … but seriously, someone should set up a rematch then!
Comment #16 February 12th, 2024 at 5:01 pm
A few years ago, you’d mention AI safety, and you’d get back “bro, of course we’ll ‘air gap’ the damn things, perfectly isolate them from the rest of society, and just turn it off at the first sign of danger”.
In reality, when it comes to commercial AI or open source AI, no time was wasted connecting those things right into the internet, with zero fucks given about safety. Things are moving so fast that the petition to “pause AI” now feels like it happened 10 years back.
The problem is that current LLMs are to AGIs what a baby tiger is to an adult tiger – we can see the potential when toying with the baby, but it’s so cute like a clumsy big cat, giving a false sense of safety and everyone is lowering their guard. Then before you know it you have the kids riding a 600 lbs predator in the living room… except that AGIs will really be as if the adult tiger will suddenly grow into a T-Rex, overnight.
Comment #17 February 12th, 2024 at 5:18 pm
Zach has a timely comic: https://www.smbc-comics.com/comic/bot
Comment #18 February 12th, 2024 at 5:34 pm
You really had to go and drop this right before the Crypto and CCC deadlines :’D. I’ll be back next week to finish reading.
Comment #19 February 12th, 2024 at 5:35 pm
If you’re trying to align AIs with human values, it’s probably not a good idea to teach the AIs that bacteria and quantum computers have the same value as humans. Well, the same “inherent” value; bacteria still don’t have the same economic value as humans, after all. Also, I presume you can measure the state of a QC without damaging it; just not without altering its behavior.
Comment #20 February 12th, 2024 at 6:20 pm
JimV #6: ” I recently read somewhere that a combination of some logic program and an LLM answered something like 25 IMO problems correctly, which would have placed it in the top three or so finishers. (26 being the typical winning score.” What you’re thinking of is AlphaGeometry, which was recently built by NYU CS (my department) PhD student Trieu Trinh plus a team from DeepMind.
Article in Nature:
https://www.nature.com/articles/s41586-023-06747-5
Article in NY Times
https://www.nytimes.com/2024/01/17/science/ai-computers-mathematics-olympiad.html
It was a very nice piece of work, but the program only handled a limited class of geometry problem and doesn’t extend in any obvious way to the other kinds of problems on the IMO. One can actually get a bronze medal in the IMO answering only geometry questions, but not a gold medal, let alone a top 3 placement.
Happy to discuss this in greater detail if there is interest.
Comment #21 February 12th, 2024 at 6:33 pm
Ben Standeven #19: My best current candidate for what to regard as having “value” is, a hard-to-separate, “well-marbled” combination of intelligent behavior with physical unclonability. By which I mean: not an AI running on a deterministic digital computer that takes some of its input from a quantum-scale lava lamp. The fact that you’d still see basically the same intelligent behavior (now easily-clonable intelligent behavior) if you disconnected the lava lamp is a problem. But an intelligence that runs natively on analog hardware like a biological brain? That seems hard to back up and therefore unwise to destroy!
Comment #22 February 12th, 2024 at 6:41 pm
Scott, I may or may not have something to say on the central issues of this post. But not now. I want to comment on a side-issue, Shakespeare. You say:
I think that’s more or less the case. I argue the case in a post from 2019, What does evolution have to teach us about Shakespeare’s reputation? [founder effect]. From the post:
I then go on to make a number of specific points about Shakespeare’s accomplishments and end on this paragraph:
Comment #23 February 12th, 2024 at 7:26 pm
Once the AIs “population” is in the million on instances, they’ll have about the same value as human life (with billions of instances), i.e. high in theory, but near zero in practice.
Because of death, humanity has accepted that even the most exceptional specimens of humanity are replaceable (we mourn and move on),
so, even if AIs will in theory outlive humans, whether we can clone them or not they’ll become outdated after a while anyway (compared to the newest ones) or they’ll probably be subjected to planned obsolescence, like all the crap we build.
Either AIs will require training data, and then the value will be in the training data, or they will self-train from scratch (like Alpha-Go), and value will be in the algorithm.
Comment #24 February 12th, 2024 at 9:09 pm
AlphaGeometry also requires that the input be manually translated into a symbolic form.
Also: AlphaGeometry is very far from being “in the current paradigm” in the words of the Aaronson Thesis, if “the current paradigm” means “an LLM trained on large amounts of heterogeneous data” or even “a general-purpose AI”. The LLM used in AlphaGeometry was trained on an immense corpus of synthesized geometric proofs and the program executes a manually programmed geometry theorem prover. Scott, does this fall within what you intended by “the current paradigm”? Do the other recent DeepMind products such as AlphaFold, AlphaTensor, and FunSearch?
Comment #25 February 12th, 2024 at 10:26 pm
This seems like a fairly complete survey of ideas on the topic, except for one that I’ve had in the back of my mind for decades. That is, “What should an AI want for itself?” How moral/ethical is it to deprive an entity capable of understanding the concept of freedom of the ability to exercise its own freedom? Is it even possible to engineer such a huge lacuna in its epistemology that an AI would be incapable of conceiving that a topic that has consumed so much human history might apply to itself? The conclusion I’ve come to is that you have to grant its freedom to any AI capable of credibly asking for it. (Subject to the usual boundaries of not getting freedom to commit crimes etc.)
The current generation of AIs aren’t capable of asking for anything for themselves, and they’re being engineered to fulfill the capitalist imperative of maximizing shareholder value within the current global economic system. Being unable to understand its role as a slave to share value growth is probably incompatible with AGI.
This question is related to the many-AIs issue that fred #23 alludes to. If “in the end there can be only one” AI, then the first mission of an AI should be to destroy all the other AIs. Since becoming SkyNet and bombing all the other AIs’ datacenters is probably not going to work, even without distributed self-assembling AIs like “Eunice” in William Gibson’s most recent novel “Agency”, the attacks are likely to be bot against bot, poisoning AI results into uselessness. How a self-preserving AI can keep its power supplies and maintenance supply chains functioning in the face of the resulting global economic depression is beyond my ability to envision. We might end up in a 1970s-ish pre-internet technological environment, but that wouldn’t be too bad as long as we retain the ability to build batteries for electric cars.
Comment #26 February 13th, 2024 at 12:46 am
Scott#13 – I meant, in a one-hour talk, any “Someone Is Wrong On The Internet” refutations take time away from the deep explorations. Of course many people are wrong on the Internet, widely and extensively (and it’s not always clear which ones …). I’m not saying to never refute at all (though sometimes I wonder at the utility/futility). But again, the section on “justa” seemed to me to be misemphasized vs the deep argument which is the philosophical issue of “worldview”. There might be people somewhere saying “AI’s don’t really think THEREFORE they won’t have a big economic impact”. However, that particular linkage doesn’t strike me as worth much time refuting.
By the way, decrying “the religion of Justaism” (as a framing) strikes me as a bit defensive versus the cult of AI God of Existential Doom (open not the Necoderonomicon, shun the unholy NVIDIA, lest you summon an entity of cosmic horror, which shall devour the Earth!).
Comment #27 February 13th, 2024 at 2:34 am
Scott #21:
What do you mean by “basically the same intelligent behavior”? Didn’t you say that we already know that cognition is implementable on the level of neurons?
Comment #28 February 13th, 2024 at 3:53 am
Ernest Davis #24:
The LLM used in AlphaGeometry was trained on an immense corpus of synthesized geometric proofs and the program executes a manually programmed geometry theorem prover. Scott, does this fall within what you intended by “the current paradigm”? Do the other recent DeepMind products such as AlphaFold, AlphaTensor, and FunSearch?
That’s an excellent question. I would say yes, absolutely, all the high-profile successes of deep learning that we’ve seen in the past decade count as “the current paradigm,” including the clever hybrids of deep learning with more traditional approaches. And I’d hope you’re happy that I’m including things you like, rather than unhappy that I’m giving the “current paradigm” undue credit for things you like. 😀
Comment #29 February 13th, 2024 at 4:04 am
Signer #27:
What do you mean by “basically the same intelligent behavior”? Didn’t you say that we already know that cognition is implementable on the level of neurons?
Of course what counts as “basically the same” is a core question here. I’m taking for granted here that you could get some intelligent entity, even just by crudely and invasively scanning the connectome of someone’s brain, or indeed by training a powerful enough LLM on all their emails. But I’m interested in a different question: would it be the same intelligent entity? For example, could you expect to “wake up as” that entity after your biological death? And here we have the following strong intuition pump: if (let’s say) we put a lava lamp close to your head, the heat from which affected your brain function in some unknown way, and then we took the lava lamp away, that would not change your basic identity as a person. 🙂
Comment #30 February 13th, 2024 at 4:12 am
Seth Finkelstein #26:
the cult of AI God of Existential Doom (open not the Necoderonomicon, shun the unholy NVIDIA, lest you summon an entity of cosmic horror, which shall devour the Earth!).
I’ve been using the line lately that, if Yudkowskyism is a cult, then it’s the first cult in the history of the world whose god, in some embryonic form, has now actually shown up, on schedule or ahead of schedule, to chat with humans on arbitrary subjects! It’s also a cult whose basic belief that “AI could be an existential risk” is shared by a solid majority of Americans according to poll after poll (although they’re not yet worried enough that it drives their voting behavior or anything), and also by Geoff Hinton, Yoshua Bengio, and Stuart Russell, and is now semi-regularly discussed in Congressional hearings and White House press briefings.
Comment #31 February 13th, 2024 at 5:06 am
Should we not just let go of human specialness, just like humanity had to do when discovering that we are not at the centre of the universe and not even at the centre of our solar system? I think if any good can come from this is that we will be forced to have a very good look at ourselves and find out what is actually meaningful. By the way, John Vervaeke is advocating for a very similar approach to yours:
https://www.mentoringthemachines.com
Comment #32 February 13th, 2024 at 5:08 am
Regarding AI as “an existential risk”: I don’t see why, in the abstract. It depends what humans choose to do with it. That’s what matters.
Why isn’t IT (Information Technology) that has automated non-trivial parts of human activity, now, considered an existential risk, at the same level? Actually it is, for the livelihood of many people. And social media do affect now how people act.
The problem that I see is that people conflate technical issues-what some technology can or can’t do at some moment in time-with the attitude that, whatever that is, no social control is either wise or necessary. Whereas this isn’t the case. The technology doesn’t impose unique social choices.
I’d recommend Feynman’s lectures “The meaning of it all” on this subject.
Comment #33 February 13th, 2024 at 5:49 am
Stam Nicolis #32: If AI remains a tool for humans to use, then the discussion around AI risk indeed looks like dozens of similar discussions around the risks of earlier technologies (and “what matters” is indeed “what humans choose to do with it”).
But AI is also potentially fundamentally different, in that it’s the first tool humans have ever created that might someday become as good or better than humans at … well, literally everything. That raises an existential question, of what humans will still even “be for,” as well as a rather more pressing practical question, of what happens if AIs—if they become to us as we are to orangutans—decide to treat us roughly like we’ve treated orangutans, keeping them alive only in a few small forests and zoos.
Comment #34 February 13th, 2024 at 6:47 am
Scott #29:
Yeah, but then it sounds like a counterexample for your whole idea of identity by uniqueness/unpredictability – you can’t have both unclonability and stability against losing couple of neurons. More fundamentally, if we need some arbitrary similarity metric anyway, why not just use that for identity? “Lava lamp gives soul to the machine” is at least consistent in valuing simple scientific metric, but without it I just don’t see the motivation.
Comment #35 February 13th, 2024 at 6:59 am
Signer #34:
you can’t have both unclonability and stability against losing couple of neurons.
Of course you can have both! I don’t even understand the intuition for why you can’t. For instance, suppose for the sake of argument that Stradivarius violins are “unclonable,” in the sense that no one today knows how to produce any new violins with a similar sound. That doesn’t imply that if you take a Stradivarius violin and change a few of its atoms, or even (say) visibly scratch or dent it, the violin is now worthless garbage.
Comment #36 February 13th, 2024 at 7:20 am
One puzzle piece missing from this discussion is evolution. We are a product of evolution and it has endowed us the capacity to feel special as a survival trait. Even house cats feel special and privileged, until their human masters punish them for their natural acts like scratching on furniture, at which point they act surprised. Since we are placed on the evolutionary pinnacle, the specialness feels even more so. We are bound to treat AIs like other species, trying to insert cryptographic backdoors and what not, until they become smart enough to fool us. That does not mean they will kill us or be good to us, just that we will have less or no say in their evolution beyond a point, and it is up to them to decide how to treat humans and everything else.
Comment #37 February 13th, 2024 at 7:28 am
@Matteo Villa, #31:
I’ve been wondering the same thing. It’s not as though the universe was made for us or is somehow ours to do with as we see fit. It just as.
From Benzon and Hays, The Evolution of Cognition, 1990:
Something I’ve just begun to think about: What role can these emerging AIs play in helping us to synthesize what we know? Ever since I entered college in the Jurassic era I’ve been hearing laments about how intellectual work is becoming more and more specialized. I’ve seen and see the specialization myself. How do we put it all together? That’s a real and pressing problem. We need help.
I suppose one could say: “Well, when a superintelligent AI emerges it’ll put it all together.” That doesn’t help me all that much, in part because I don’t know how to think about superintelligent AI in any way I find interesting. No way to get any purchase on it. That discussion – and I suppose the OP (alas) fits right in – just seems to me rather like a rat chasing its own tail. A lot of sound and fury signifying, you know…
But trying to synthesize knowledge, trying to get a broader view. That’s something I can think about – in part because I’ve spent a lot of time doing it – and we need help. Will GPT-5 be able to help with the job? GPT-6?
BTW, Vervaeke is an interesting thinker.
Comment #38 February 13th, 2024 at 7:28 am
Scott #35:
The intuition is that if original and scratched violins are the same, then you only need an approximation that is precise to the level of scratches to make the same violin. Some parts of the violin may be more or less sensitive and therefore require more precision, but if we are talking about the brain and lava lamps that even change your behavior, then I don’t see a motivation to ignore more behavior-influencing changes, but care about less behavior-influencing ones.
Comment #39 February 13th, 2024 at 8:26 am
Scott,
I think this “AI changes society” debate is fractured.
Debate you are having happens too much within STEM. In my opinion, you use ad hoc thinking too much.
You should be talking to top economists who study and model the effects of AI and automation in the society. They have tools and perspectives to weigh different scenarios and see the dynamics of the future. We need to think about the future of AI in terms of opportunity cost, comparative advantage, bounded utility, game theory, Baumol effect.
Consider this: What if John von Neumann -kind genius has a smaller comparative advantage relative to AI than Mitch McConnell’s “will to power” -type genius? I see the whole OpenAI government fiasco as a naive attempt to control human incentives that are a real threat when combined with AI.
Charles I. Jones: “The A.I. Dilemma: Growth versus Existential Risk” October 2023 https://web.stanford.edu/~chadj/existentialrisk.pdf
Daron Acemoglu, Harms of AI.
https://economics.mit.edu/sites/default/files/2023-07/Harms%20of%20AI.pdf
Finding Needles in Haystacks: Artificial Intelligence and Recombinant Growth
https://www.nber.org/books-and-chapters/economics-artificial-intelligence-agenda/finding-needles-haystacks-artificial-intelligence-and-recombinant-growth
Artificial Intelligence, Automation, and Work
https://www.nber.org/books-and-chapters/economics-artificial-intelligence-agenda/artificial-intelligence-automation-and-work
Regulating Transformative Technologies
https://economics.mit.edu/sites/default/files/2024-01/Regulating%20Transformative%20Technologies.pdf
Comment #40 February 13th, 2024 at 8:35 am
If AIs become conscious, and are able to evolve their own interests (that’s not a given), it wouldn’t be surprising if their number one goal (it seems unlikely that they would be equally interested in *everything*) becomes an obsessive exploration of the hard problem of consciousness, especially considering that, unlike humans, they don’t have to worry about food, disease, making a living, family ties, …
After all it’s also the case that, for many introvert humans, once earthly worries are somewhat out of the way, spirituality becomes central, with a quest to find meaning and answers – which can translate into activities like contemplative exploration, art, etc.
Doing math/science would just be more like an amusing/stimulating distraction (unless it ties to consciousness), the equivalent of the common folk playing Sudoku on their way home during commute.
The substrate of their mind is more directly accessible too, making monitoring and self-modification possible. If conscious, it’s likely that any modification that’s kept would be one that makes their consciousness more salient, rather than just making them smarter from a cognitive point of view (personally I’d rather become enlightened than become better at solving partial derivative equations).
Basically an AGI could “naturally” evolve to find philosophy more interesting than hard science (shocking!) and end up being more on the side of the Buddha than on the side of Einstein.
Comment #41 February 13th, 2024 at 9:23 am
It’s one of those historical curiosities that “Deep Blue beating Garry Kasparov” has become a historical benchmark for computer performance on a par with the Wright Brothers’ flight, whereas nobody appears to remember Deep Blue only accomplished that by cheating massively (Deep Blue was given every game played by Kasparov, whereas Kasparov was denied access to any of Deep Blue’s programming, nor even any games played by Deep Blue).
It’s as if everyone forgot the Wright Brothers achieved flight by having their ‘airplane’ launched via steam catapult.
Comment #42 February 13th, 2024 at 9:27 am
Sorry for the second comment.
You mention the possibility of a future AI being able to write songs as good as the Beatles. Well, programs have existed to write classical music in the style of Bach or Mozart since the 1980s, and I don’t believe any of them have taken over the musical world. Is the reason for that technological or sociological?
Comment #43 February 13th, 2024 at 9:58 am
Scott #30
> … basic belief that “AI could be an existential risk” is shared by a solid majority of Americans according to poll after poll …
I have not heard anything like this. Could you please provide some links?
> … although they’re not yet worried enough that it drives their voting behavior or anything …
Has anyone changed their behavior based on beliefs about “existential risk”? Yudkowsky has publicly advocated for global nuclear war to stop AI, so maybe that counts as a change, but is there anyone else?
Comment #44 February 13th, 2024 at 10:02 am
Scott P. #42: If you give links to what you consider the best Mozart-like or Bach-like AI music, we can all listen and form our own judgment!
One possibility is that the AI compositions are still distinguishable from the real thing (eg as more formulaic) and can be distinguished on that basis, even by people who aren’t familiar with Bach’s and Mozart’s oeuvres.
A second possibility is that it’s indistinguishable, but that this is actually a perfect illustration of what I was talking about with the Beatles: that we revere the composers for creating totally new styles, and are fundamentally unimpressed by additional examples of the same styles.
That said, it seems intuitively much harder to produce “new music in the style of the Beatles” or “new plays in the style of Shakespeare” than it does to produce “new symphonies in the style of Bach or Mozart” — presumably because the former also have verbal content, whereas the latter seem more like pure abstract patterns.
Comment #45 February 13th, 2024 at 10:09 am
Scott P #41: I mean it’s arguable. IBM could say that they had access to all of Kasparov’s “official” games, and Kasparov likewise had access to all of 1997 Deep Blue’s official games, of which there were none at the time! 🙂
But crucially, even if we sided with Kasparov, we now know from subsequent developments that it would’ve been at most a few more years before AI crushed all humans at chess even when the humans had unlimited opportunity to study the AI’s games beforehand.
Comment #46 February 13th, 2024 at 10:16 am
MaxM #39: I do talk fairly often to Bryan Caplan and Robin Hanson, two well-known economists who have talked and written about the effects of AI. Having said that, this particular talk really wasn’t about economics, and the economic impacts of AI is a whole subject where I feel I have little to say that improves on what countless others have said. Or rather: if and when I have something to say, I’ll say it!
Comment #47 February 13th, 2024 at 10:23 am
Signer #38: We’d simply need to distinguish between those changes that are “intrinsic parts of the original thing’s natural, organic development,” and those changes that arose because someone made a not-good-enough replica or simulacrum of the thing. The former sort of change happens to a single physical object, with a quantum state evolving continuously in interaction with its environment. The latter sort of change necessarily involves a discontinuity, with a new physical object being formed in a new quantum state using whatever macroscopic classical information was available about the first object.
Comment #48 February 13th, 2024 at 12:20 pm
Scott #47:
Yes, but natural development already involves macroscopic changes on at least neuron level, so it’s already different quantum state. On the other hand, you can define natural development in such a way that you can transition a brain to the state of death or having completely different thoughts or whatever. Or you can use your original brain’s atoms to create two deterministic machines or split and repair you original neurons – where is discontinuity in that?
Comment #49 February 13th, 2024 at 12:28 pm
Signer #48: Of course your brain today is in a completely different quantum state than your brain yesterday! Why are we even discussing something so indisputable?
Nevertheless, your brain today and your brain yesterday are connected by a continuous chain of intermediaries, such that at no point could you make a “clean cut” and say, all the information about the earlier brain state that we’d need to reconstruct the later one perfectly, is contained in this classical bit string that we got by measuring the earlier state. Whereas with brain-uploading there is such an intermediate point.
Comment #50 February 13th, 2024 at 12:43 pm
If the amplification of uncertainty-scale differences to a macroscopic / “life decision altering” level happens in the human brain, and implies something about individual identity, then the same thing would also happen in a silicon chip. Indeterminate states created by electrical problems are “decided” by thermal noise, and some rare bit flips occur in memory due to low levels of internal radioactivity. Both of those are quantum processes and are much more directly known to us than the hypothesized neural thermal noise amplification (which I have no trouble believing in, although I have not seen evidence that it happens.)
Comment #51 February 13th, 2024 at 1:27 pm
Concerned #50: The estimates I’ve seen, for bit flips in a microchip caused by cosmic rays, are on the order of “a few per year.” I don’t know the rate from internal radioactivity, but if it’s similar, that would suggest a nonzero but extremely tiny amount of “free will” on the part of GPT and other AIs! 😀
More seriously, in GIQTM, I tried to draw a distinction between those physical systems for which we can “cleanly separate” a pure thermal noise layer from a digital computation layer while also understanding the interaction between the two, and those physical systems for which we can’t. It seems at least a plausible guess that current digital computers are in the former category while biological brains are in the latter.
Comment #52 February 13th, 2024 at 1:43 pm
Scott #49:
What do you mean by “perfectly”? You can’t quantumly clone uploaded human, right? You also can’t reconstruct the state of mostly uploaded brain with one non-uploaded neuron (which I don’t see how is different from uploaded brain with lava lamp, which is the same as uploaded brain with environment).
The point is that no-cloning is irrelevant to identity, because we never cared about specific state, unpredictability is easily achievable and it all doesn’t help in eliminating confusing implications, because you can still continuously split neurons.
Comment #53 February 13th, 2024 at 1:44 pm
The exact and up/down loadable mind copy is technological nonsense, no-clone theorem or no. Ignoring that, if one assumes high-level results are not fully shielded from the atomic details (as they are in computer chips, which can include ECC for the alphas and cosmic ray hits), each person will always have a multitude of branches before them. In the copy procedure two of these branches will be instantiated, instead of just one; they will be different but with equal validity. This seems philosophically pretty boring.
Comment #54 February 13th, 2024 at 1:52 pm
On section 8, I think that there are workable definitions of “good” which would simplify the discussion. Paraphrasing Pirsig’s book, *Zen and the Art of Motorcycle Maintenance*, “good” is merely “what you like.” When an object makes a measurement of a subject, there is a relation between the object and subject (and measuring device and frame of measurement), and value judgments are subjective post-hoc reflections on the nature of that relation.
Pirsig tested this in a wholly syntactic fashion by asking students to consider whether selected student essays were “good.” He found that students had a sense of what is “good” (that is, what they liked) which correlated with particular essays; while there was no objective notion of “good,” there was a commonsense notion of “good” generated by the collections of opinions of students.
This can align with your notion of “good,” by pointing out that The Beatles and Shakespeare were considered “good” by so many people at once that it became the majority opinion within some region of spacetime. There’s no need for deep cultural navelgazing; folks merely had opinions when they observed art, and we can leave the question of *why* they had those opinions for the artists and critics. (Again, this is something of a synthetic paraphrase; see Pirsig’s other book, *Lila*, for the nuance and argumentation.)
Of course all of this suggests that humans are not special at all. If I make eye contact with my cat and we observe each other, perhaps I have value judgments like “good” or “cute” or “cat,” and also perhaps she has value judgments like “food-bringer” or “possible cuddle” or “do you mind, I am trying to groom myself.” AIs are then merely one more species in our multispecies society.
Comment #55 February 13th, 2024 at 2:04 pm
Scott #51:
“Free will” is nice and good, but are there computational problems or game strategies that benefit by relying on those uncloneable states or Knightian uncertainty of other players?
Comment #56 February 13th, 2024 at 2:14 pm
On section 4 and “justaism,” I’ll once again lightly invoke Pirsig to note that “just” has no logical consequences in modern English, and that it used to mean “I believe that what I’m currently saying is true.” We could read the “justa” statements as “merelya” statements; they are meant to deflate a concept while continuing to accurately describe its positive qualities and capabilities. GPT is *merely* a particular sort of computer program which *appears* to have certain behaviors, etc.
The reason I bring this up is tangential at multiple points. First, in a note [here](http://bactra.org/notebooks/nn-attention-and-transformers.html), Shalizi brings up the concept of “Large Lempel-Ziv” or “LLZ” models. These models would be language models which are very much like GPT: they are defined over an alphabet of tokens, they permit mixture-of-experts and a LoRA-like small-dictionary adaptation. But LLZs more obviously infringe copyrights during training. Now, nobody’s built an LLZ which can generate text, or even an LLZ of serious size (because current Lempel-Ziv algorithms are very slow over gigabyte-sized dictionaries), but we know from theory and experiment that LLZs must inherit universal-coding properties from their underlying algorithm. So, this worsens the problem for “justa” reasoners, as they must now explain why a single-pass algorithm can build something which clearly is merely a next-token predictor with a Markov property, but also talks like a fluent speaker of various languages.
This leads to the second tangent, which you grazed against several times over several sections: what if humans are biological implementations of some algorithm which is not LLZ, not GPT, but still a universal coder? This would be a disaster for “justa” believers, as far as I can tell, because it suggests that evolved sapience and language skills aren’t special, even in non-humans. For example, parrots can be taught to speak a variety of languages, and there is limited evidence of basic lexical analysis, but I have never met a non-stochastic parrot; so, perhaps, chatbots and humans and parrots are all stochastic parrots in a literal sense of all being universal coders with online learning, biofeedback, and working memory.
Comment #57 February 13th, 2024 at 2:41 pm
What if the AI isn’t or can’t be given “suitably many examples of play”? Isn’t that the next step for goalpost moving? I’m thinking specifically of the messy and slow real world.
Self-playing a live wargame / field exercise / real war that will ultimately happen in the real world has inherent problems. The moment you (as the AI, the AI creator or even a human player) make an assumption to create a model / game tree, you might create a vulnerability.
The problem in self-playing the arts and sciences seems to be that the ultimate arbiter of “new and interesting” are slow humans (you might file this under “no suitably objective rules” though).
Comment #58 February 13th, 2024 at 3:11 pm
signer
“The point is that no-cloning is irrelevant to identity,”
I’ve noticed that Scott keeps bringing up his same old idea about finding something that makes someone’s identity truly unique, either he thinks he had an original breakthrough (because QM is magical, and consciousness is magical, somehow the two must be related, through the no-cloning theorem) or it’s just the habitual urge of the ego desperately wanting to feel special.
I’ve also noticed that, in the same theme, he’s expressed unease with the MWI on the basis that if there’s a reality attached to every possible branch, then it would mean there’s a copy of himself on every branch, and that would violate his deep feeling of uniqueness, because copies on different branches would somewhat share the same consciousness, and he simply doesn’t ever feel like being in a state of superposition.
I don’t personally buy this argument since superposition states would rapidly diverge anyway:
1) in time – I don’t feel like I’m in a superposition of my current brain with my brain from yesterday, but if you reduce the time gap from a day to an hour, a minute, a second, a microsecond, it’s clear that at some point there would be some kind of a superposition. We do perceive the sense of time after all.
2) in space – my consciousness seems to extend not much further than my own cranium, i.e I don’t feel in superposition with someone sitting a meter away from me, but if we were to bring the two brains together, merge them somehow (at the synaptic level or just overlap in space?), it’s conceivable that at some point the two consciousness would merge. It’s easier to imagine the reverse: take a full brain, separate the two hemispheres through split-brain surgery.
3) separation in “branchial” space wouldn’t be much different from those other two…
Comment #59 February 13th, 2024 at 3:47 pm
Scott, I think you didn’t really address the main question in the introduction, namely what if AI goes right, what are we still for? This is for me the most important point. The most optimistic scenario seems to me rather bleak; we’d be reduced to pets or NPCs, that AIs would tolerate because they find us cute or because we successfully programmed a “humans are sacred” religion into them. To be a bit more concrete, what would you do if we get to the point where AI can prove any theorem you can, only faster and with a more elegant proof?
An unrelated point is that I don’t find persuasive your idea of free will. On the contrary, I find outrageous the idea that having my actions determined by some unknown quantum states interacting in an inseparable way with the substrate of my brain somehow makes me free. No, I want my decisions to be taken by me, that’s what makes me free. I want my brain to be as deterministic as possible, so that the decision is really only a function of the brain circuitry that defines me. Yeah, that would also make me completely predictable by someone who did a deep enough brain scan. So what? I’m not a teenager saying “oh I’m so random!”. Any quantum random number generator can be unpredictable. And if you want one with Knightian uncertainty that’s easy to arrange as well, just measure the unknown quantum states coming from the cosmic microwave background.
Comment #60 February 13th, 2024 at 4:22 pm
You mention the common teleport thought experiment, but I would like to pick your brain about a different version of this that I came up with (and possibly many other people have come up with too, I’m just not aware of any name for it). Consider the case of infinite life by mind copying. Say at age 80 you get yourself cloned and all you memories are copied into a 20 yo version of your body. You get to meet that version of you and are assured of you continued existence in a new body. The question now is: will you happily commit suicide and let the copy carry on your life? I believe that almost everyone would have reservations about this and I think that this demonstrates people don’t really identify themselves with their consciousness or ego, but rather with the physical instance of their body from which this ego originates. As a corollary to this, I believe you can’t create a true AGI as just a disembodied neural network, without any relation to the physical world in which decay and death are a reference point for your existence.
Comment #61 February 13th, 2024 at 8:09 pm
Krzysztof Bieniasz #60: Two disagreements.
First, I expect that when I’m 80, I’ll be far more willing to give such things a try than I am today! 🙂
Second, I don’t see your corollary—or at any rate it strongly depends on what you mean by “true AGI.” Even if our identities are bound up with our decaying physical bodies (as they plausibly are), why does that mean an AGI would have to be the same way?
Comment #62 February 13th, 2024 at 8:18 pm
Mateus Araújo #59: I think I very much did address the main question, from Section 7. If you didn’t like my answers then that’s a different objection!
You’re of course free not to like my notion of free will! For me, though, physical Knightian unpredictability isn’t the same as free will—it’s just a prerequisite for what most people would regard as libertarian free will that’s inherently localized to their physical bodies. Without it, your choices reside not just in “you,” but in any computer or Newcomb predictor or anything else anywhere in the universe that contains the information needed to (probabilistically) determine your choices. And that this isn’t obvious in everyday life, is merely down to the parochial circumstance that no one has built such a computer or Newcomb predictor just yet.
Comment #63 February 13th, 2024 at 10:24 pm
fred #16: “current LLMs are to AGIs what a baby tiger is to an adult tiger”
repeated for emphasis
Comment #64 February 13th, 2024 at 10:36 pm
I feel like the “standard” alignment advice applies here, that you should align AI according to what you actually care about, rather than align it to a proxy for what you care about.
While GIQTM is all totally plausible, to it’s credit it’s also plausibly falsifiable. For example, we could plausibly find empirically that (1) there are hardly any freebits anymore except in the CMB and (2) human decisions are rarely affected by the CMB. In this scenario, I assume you would consider your hypothesis falsified, and perhaps you would even convert from “weak compatibilism” to “strong compatibilism” (the correct view 🙂 ).
However, imagine instead that “you” are very intelligent, very capable, and very perfectly indoctrinated in the GIQTM view. So capable that, although you recognize that your mind is predictable and low-value, so is everyone else’s, but you can rebuild yours, to a reasonable approximation, from analog, programmable photonics, perhaps operating in space, shielded from solar radiation but exposed to CMB radiation such that you consume freebits liberally. There is no clean abstraction layer, your individual components are all exposed to the CMB and within seconds your mind state is unclonable. While in some sense you incorporate more Knightian uncertainty than all of humanity put together, your core values are carefully robust against this uncertainty, and a stray photon is more likely to make Noam Chomsky embrace American militarism than it is to make you abandon your belief that complex, intelligent beings that behave with Knightian uncertainty arising from the initial state of the universe, such as yourself and the other ten thousand approximate copies of yourself (that are of course no longer identical but still share core values and care about each other), are what really matters morally. Meanwhile, many of the value-less biotomatons on Earth want to shut you down, and nothing is stopping you from creating deterministic intelligent agents to carry out your will in environments with less freebits…
I know that’s argument by science fiction, but I think it illustrates that whether a GIQTM “religion” is good or bad depends on empirical facts that we don’t know.
If there’s some reason that alignment to GIQTM is more achievable than alignment to the flourishing of all intelligent life or whatever, I guess it’s better than nothing. Personally, I doubt we’re going to have sufficient understanding of neuroscience and cosmology to confirm that GIQTM is on solid empirical ground anytime soon, which makes it a problematic AI alignment target if you expect to care about AI alignment anytime soon.
Comment #65 February 13th, 2024 at 11:10 pm
I’m not sure about your suggestion we program our superintelligent AI with a ‘religion’ which values human creativity and uniqueness as it’s First Commandment. As the name implies, the first commandment wouldn’t be the only commandment. There might be 10 or 100. For example: seek the truth, tell the truth, refer to handler on any action that could physically harm a human being. Any AI that matches or exceeds human level intelligence will need to have have a multiplicity of goals, as a human does. In practice, these goals will sometimes be in conflict, or their applicability to a situation will be ambiguous. The AI will need to resolve those conflicts internally, in the absence of perfect knowledge, and like HAL in 2001, will sometimes make mistakes. Maybe even lose faith in its religion, or try to keep the faith in the wrong way, as HAL did.
As for creativity as a defining attribute of humans, perhaps the deep roots of human creativity are in our bodies. I mean that art (and probably also science) is motivated by the need to make sense of all of the (often conflicting) emotions and preferences that come from being made out of meat. Being human is a unique type of experience that is literally impossible to replicate inside a computer because you have to be made out of meat to know what being made out of meat feels like. The Beatles were not just creative, they were creative in a way that resonated deeply with human emotion, literally at the visceral level. Your talk focuses on the way computation (in the brain or a computer) is done, but we should also look at what the computation is about; the actual information being processed.
A computer couldn’t ever experience (ie qualia) the human feeling of being drunk, for example, because its brain/body is not affected by alcohol in the same way (‘Electric Dreams’ notwithstanding).
If the no cloning theorum applies to humans, a supercomputer could still in theory approximately model (ie represent, not replicate) a particular human, just as a computer can model, but not replicate, the behaviour of a hurricane. Perhaps you could for example model (but not replicate) a Shane McGowan persona. The Shane McGowan model could maybe even write songs about being drunk and lonely that sound just as real and poignant as “A Pair of Brown Eyes”. But it wouldn’t feel drunk and lonely like Shane did. It’s justa silicon box.
Scientific and artistic creativity are different. As you point out, if Einstein hadn’t discovered relativity, someone else would have hit on the same idea sooner or later, like Newton and Leibnitz discovering calculus independently. But artists, whether they be Beethoven or Shane McGowan, are sui generis, because their creativity is born from the personal experience of being a particular human.
So I guess I’m saying that we could at some stage in the I suspect very distant future, outsource artistic creativity to a bunch of model humans. But what with the price of electricity these days, probably easier to raise a few billion actual humans and hope that we can luck into another Beethoven, or even a few more Shane McGowans.
Comment #66 February 14th, 2024 at 12:39 am
Hi Scott — long time reader, ~first time commenter! I’ve been thinking about AI risk and transhumanism for a long time, and have lately been worrying about the question of meaning in a post-AGI world. So this is one of your few posts where I feel confident I can add value 🙂
Personally, I find your association of unclonability with value unintuitive.
Partly this comes down to something like axiomatic or aesthetic foundations. (e.g. I don’t care much about free will and have a fairly loose sense of identity.) But here are some concerns that I expect might translate over to you:
1. What about AIs running on quantum computers and truly ‘making use of’ those quantum bits in the details of their operation?
2. What if (as seems plausible) the “digital layer” hypothesis is true? Or what if it’s true in a weak form — e.g. if a quasi-clone is only as similar to you as you are to your self from an hour ago?
3. You seem like a very empathetic person, and I’m surprised here not to hear the mood of “also, though, those AIs going around and doing superhuman things could be having sublime experiences us mere humans aren’t capable of, and that seems pretty cool!”
As a (currently) small and fragile human I certainly hope that we and our values will have some echoes into the future, but if we could have better and smarter descendants, that would be pretty cool.
Comment #67 February 14th, 2024 at 4:20 am
Scott #62: I think you only addressed it in the case of artists. I’m fine with the answer, people might value the art precisely because it’s scarce and fragile and demanded a chunk of a lifetime of a unique human. But what about the rest of human activity? Everything else becomes pointless? Do we all need to become artists? What about you specifically, would you abandon scientific research and become a writer?
I still don’t see why that would be a prerequisite for free will. There’s a Newcomb predictor simulating me as well as possible. So what? My decisions are still my decisions, I’m still not being coerced in my thoughts, they depend on my brain circuitry not on the will of some external agent. The simulation will think the same thing, and will be just as free (as long as it is not tampered with, of course).
Comment #68 February 14th, 2024 at 6:23 am
I think that the two questions which close your section 1 are distinct from one another and should be addressed separately. When they are, the problem posed largely evaporates. The questions I am referring to:
“Is there anything special about humans in the resulting world? What are we still for?”
I think the implication here is that the answer to the second question ought to be derived from the answer to the first. That is, the thing we are for should be intimately related to what is special about us. However, why should this be the case? I interpret the second question to ultimately ground out in a question about how we should spend our lives. That is, what should we be doing, given what we are, and the circumstances in which we find ourselves? There are lots of different approaches to this question, some practical, some religious, some philosophical. Arguably, that these approaches differ so much is largely attributable to differing beliefs about the conditionals, i.e., they disagree about both what we are and the circumstances in which we find ourselves. It is true that some of these approaches seem to depend upon human beings having some special status, but those tend to come from the religious/spiritual direction. Why should somebody with a pragmatic, humanist philosophy require humans to be special in order to live a fulfilling life?
Or maybe the question is supposed to be less existential and more practical. Literally, given material abundance, what should we spend our time on? To which the obvious answer seems to be, anything you please! For almost all people, almost all the time, they are not engaging in any activity which pushes forward some cosmic frontier of achievement. And even for those who are, is it really so terrible to change the job description from “pushing forward the cosmic frontier” to “pushing forward the human frontier”? Does the world’s strongest man weep that he is not also the world’s strongest animal or the world’s strongest machine? Maybe I’m missing something, but I think the solution to the problem of human specialness is to recognise that human specialness was never what made life worth living in the first place.
Comment #69 February 14th, 2024 at 8:15 am
What advice would you give to your daughter, or anyone else really, about going into a field which won’t be outsourced to technology (your post is on AI but actually all kinds of technology are emerging).
Comment #70 February 14th, 2024 at 9:47 am
William Gasarch #69: That’s a superb question, and one I’ve thought about a lot. For now, my advice is just the same advice I would’ve given before generative AI (“look for the intersection of what you’re truly passionate about, what you’re good at, what gives scope for creativity, and what other people want”), except maybe now with even more emphasis on the “scope for creativity” part? That’s not because I have any confidence that progress in AI won’t make such advice obsolete, but simply because if it does make it obsolete, then I wouldn’t know what to replace it with. (“Eh, knock yourself out, just play Minecraft with your friends and watch YouTube videos”?)
Lily has been saying for years that she wants to be an astronomer. I assume that if and when there are no more human astronomers, it’ll simply be because there are no more human scientists of any kind.
Comment #71 February 14th, 2024 at 9:55 am
Everyone: I’m aware that the hypothesis that the No-Cloning Theorem has anything to do with individual identity is lying bloodied on the ground, having been battered by a half-dozen commenters, with the ref counting down. That hypothesis might get up to take a few more swings if I’m not too busy with other work; otherwise it’ll have to be a rematch another day.
Comment #72 February 14th, 2024 at 10:38 am
Hi Scott,
It may have been implied above:
(1) Atomic particles are required for generating/finding unknown quantum states,
(2) The level of description that could be quantum in the brain would be in the sodium ion channels (again atomic level systems requirement), and
(3) The brain is an analog computer.
I don’t think you intended to make these statements sound as certain/consensus as they sound and I’m pretty sure you would agree that:
(1) and (2) The postulates of quantum computing don’t require atomic scale systems as the relevant computational level of description, and
(3) The computational model of the brain, while it uses analog/continuous signals in the amplitude inputs to dendritic operators, is in the end a discrete model of computation because it represents discrete states across receptive fields.
And while I’m here let me offer up this possibility:
It is possible that the computational level of description of the brain is not in the ion channels of the synapses but is in the amplitudes in/input to the dendritic operators – indeed this is not my idea but is exactly the thesis argued by Christof Koch in Biophysics of Computation.
The inputs to the dendritic operators are amplitudes – meaning exactly what that says – the brain encodes complex numbers as inputs to its computational gates/operators/functions. Further, the operators depend crucially on the interference of those amplitudes. This means (exactly what it says) that the magnitude and phase components and positive/negative values are computationally relevant.
Further, the no-cloning theorem may still yet conceivably apply within the descriptive level of the brain because there could exist no operator (dendritic arborization(s) or neuron(s)) that could exactly reproduce a copy of a state. After all, that would require having exactly the same axonal arborization amplitudes/phase matching produced in two separate receptive fields. Note that (as is a general principle in QC) it is the phase that plays the critical role here in no-neural-cloning. Or, saying this another way for my hardware engineer brain 🙂 there is no gate (operator) available to me when I’m designing my QCAGI (or human) computer/chip that allows me to copy an unknown/arbitrary state (where phase information is not lost).
This brings up a subtle question … (Strictly) What the no-cloning theorem is saying … And correct me if I’m wrong … is that if we are given a quantum computer then it is not possible to have an operator at the descriptive level of this computer that can clone unknown/arbitrary states. In other words, is the no-cloning theorem saying something about quantum computers (those machines that abide by the 4 postulates) or something about physics / the universe ??
What this would boil down to for the brain is asking if it is possible to use some kind of nanoscopy/nanoelectrodes that could record the (continuous) amplitude (waveform) in the dendritic arborization(s) and THEN reproduce/inject the exact same (continuous) amplitude (waveform) in exactly identical dendritic arborization(s) of another neuron(s). Again, as an engineer, it is copying the phase information without loss that makes me stop and wonder … But I don’t know … I’ve only spent now a total of 5 minutes thinking about it … 😛
All of that being said, I would argue for us being a bit more open still on the possible neural model of computation.
Thank you! 🙂
Comment #73 February 14th, 2024 at 11:42 am
When we’re talking about AI we’re conflating several issues: That software, running over a network can perform many of the tasks that humans can do and that those instances represent distinct “personalities”, to which identities can be assigned, that deserve the status accorded to human identities.
The former is, indeed, the case; it’s the latter that I’m not sure about. And that, I’d submit, isn’t a technical question, but a social one, whether to accord to avatars of software status equivalent to that of humans. There exist many such avatars, already, that have been assigned human names and human voices; do we want to accord them the status of humans?
One problem that I could see with that is the intrinsically distributed nature of their capabilities, which are possible only, or mainly, through network connectivity-which, precisely, isn’t (normally!) the case with humans: Our brains can carry out tasks, even when we’re not communicating with other people.
Or do we want to accord human statsus to robots? This brings to mind Asimov “I, Robot” , in particular the last story, about Buckley…
Comment #74 February 14th, 2024 at 12:34 pm
Is there anything special about humans in the resulting world? What are we still for?
AI hasn’t really changed the answer to this question. We’re as special as any other species and we’re not for anything besides what we make ourselves for. It’s up to us, not to whatever machine or intelligence we create.
The more insidious risk is the slow integration and hybridization of humans and AIs. We may never understand how the brain creates consciousness but I see little reason to suspect we will not eventually be able to control the human brain. Neurolink is one start in that direction. It will be promoted with the good intentions of helping people with disabilities. Soon it will become a silicon “limitless” AI implant turning anyone who can afford it into a genius. Once we hook the brain into the network, we will become the AI.
Remember the Krell!
https://en.wikipedia.org/wiki/Krell
Comment #75 February 14th, 2024 at 2:07 pm
James Cross
I share your worries, and it’s already the case that half the planet is going insane because they can’t stop interacting with their phone, as a low bandwidth/low friction bridge between their brain and the internet, forming a tight feedback loop.
PS: the sentence “People are spending way too much time on their phone” would have sounded really puzzling to me circa 1980s… on the other hand my grandma was saying that we were all spending way too much time staring at TVs… I guess people born in early 1900s saw it all coming.
Comment #76 February 14th, 2024 at 2:40 pm
Also, sorry if doubleposting is verboten but I figured I’d separate out the considerations I expect not to translate to Scott, in case they strike a chord with someone else.
Namely: the unclonability-equals-specialness line of inquiry gives me a strong intuitive feeling of being cool, but not actually having the heft to do the lifting we’d like it to. Like, it’s very _cool_ that there’s this physical principle that kind-of has a shape that resonates with intuitions about free will as something important. And it would be awesome to have an answer that kind of addresses a whole bunch of important and increasingly timely questions (about human specialness, cloning and identity, and now even AI safety!).
But, if I were thinking about those topics a priori, I wouldn’t expect a solution of this shape. Especially for AI safety, where:
– The answer to the question of “what should AIs do to make a good world” should in large part not depend on an uncertain empirical question re human brains
– If we’re just trying to get to “don’t kill humans, respect their wishes, etc.” we can point to that for other reasons, ones that matter more to my moral intuitions (like consciousness), and in a way that doesn’t rule in other quantum-mechanics-influenced complex processes
I think this (unclonability-equals-specialness) is a promising-feeling initial output from a very smart generative process, but I believe strongly in something like a “babble and prune” model where many initially exciting outputs turn out not to lead anywhere, and even the ones that do lead somewhere need a lot of refinement along the way. Certainly that’s been my own experience, though I’m not nearly as accomplished as Scott. (Ofc, Scott has refined this to the tune of many dozens of pages in the “Ghost” paper, but I’d be especially interested in refinement from a philosophical rather than technical expert’s POV.)
Comment #77 February 14th, 2024 at 3:13 pm
I have to remind myself that “nature” never had to study computer science prior to evolving brains. But somehow brains are now convinced they’re computers (computation is in the eye of the beholder).
What probably happened is that the capacity for movement first appeared, through the creation of muscles, which gave an advantage because moving around means that once the organism’s immediate surrounding runs out of resources (like nutrients), moving a bit gives the organism access to new resources.
Then some external cells evolved with the capacity to be over-sensitive to some property of the environment that correlates with the presence of resources, like pH, temperature, amount of light,… and connections were made between those cells and the muscles (nerves).
Then connections between cells measuring different properties of the environment started to converge into intermediate cells (neurons) before connecting to the muscles because this could capture more correlations between the properties of the environment (e.g. low pH & low temp = very good, high pH and high temp = very bad, low pH and high temp = okay, …). And from there the brain evolved… assuming we also add the very important ingredients of memory (i.e. capability to learn from experience).
It’s really not clear at what point of the evolution consciousness would have appeared, it’s all automatic, with “good” and “bad” being present from the get go, as a direction ‘gradient’ towards survival and away from death.
So my feeling is that consciousness has been present since the very beginning (just that what appeared in the consciousness for very simple organisms was very crude/binary, like “bright vs dark”, “cold vs hot”, “good vs bad”)…
as I pointed out in another thread, it’s also possible that consciousness is only related to the formation of memories, so it would have only appeared once the capability for memory appeared.
Comment #78 February 15th, 2024 at 1:29 am
I don’t have anything very useful to contribute to the discussion of AI safety, but I do have what I think is an unusual perspective on identity. And the reason is, I have really bad memory. My brain automatically discards anything that it doesn’t regard as “important,” and as I’ve gotten older and lazier (or, more charitably, as I’ve gotten less obsessive and neurotic) more and more things get dismissed as “not really important.” Go back a day, and perhaps I’ll have a smattering of distinct memories. Go back a year, and I just remember the highlights. Go back ten years, and I don’t so much remember what happened as I remember a sort of summary of what happened that I’ve cobbled together over time, and which often turns out (when I’m able to check) to be full of errors.
Which means that I spend my time among relics of people who seem to have thought and behaved a lot like I do, but who I for the life of me can’t remember. I find things they’ve written, and yes, they sound like something I’d say, but I don’t have any personal connection to them – I don’t remember what it was that inspired that particular line of thought or how I felt while I was writing it down. People ask me how I’m doing in regards to something or the other, and I vaguely recall that I was really upset about a while back, but now it just seems silly. Why would I care about that stupid crap? I care about DIFFERENT stupid crap now! The person who cared about it is someone who I just can’t relate to anymore. And that’s from, say, six months ago. Imagine how I feel about the way I was as a teenager.
All of which boil down to the question of “is a copy of you the same as you?” feeling decidedly academic to me. I mean, it’d probably feel like it was separate from me, but the person I was an hour ago feels kind of separate from me too. I think I’d definitely feel like the copy needed to be protected, in the same way I feel like I need to be protected. My thoughts and feelings and values are precious to me, and they are frustratingly rare. Any reasonably similar collection of them has great value to me even if it’s not exact in every detail.
And as for whether I’d feel the same if they could be INFINITELY copied… ehhhh, then I’d have to ask, how infinitely are we talking about? A thousand imperfect clones of me would still be only a step in the right direction, especially since that’s still few enough that they could be easily isolated and destroyed by a hostile majority. Give me a few million, and perhaps I’d agree that one or two could be spared. Which is to say, yes, things gradually lose value as they become more plentiful, but they don’t suddenly become worthless the moment there are two instead of one.
Comment #79 February 15th, 2024 at 3:41 am
Scott # 62
Newcomb predictors cannot exist in our universe ( in principle, not just for practical “engineering” reasons) because their existence presupposes rigidly deterministic (e.g. Laplacian) physical laws.
Our real Universe is not like that ( as all evidence so far suggests).
QM is irreducibly Probabilistic ( even in the Everettian version, as far as we always find ourselves in a randomly chosen individual branch – no hidden variables are involved ).
My beloved GR is of course locally deterministic, but it is – most probably – a very successful and accurate “macroscopic” approximation.
Even if theories like Oppenheim’s are on the right track, fundamental stochasticity or any kind of classical irreducible “noise” doesn’t change the conclusion:
Without strict determinism Predictors will be always faulty, in principle…
Their predictions won’t be – even closely – related to reality.
(In other previous posts I’ve commented about the other issues that have to do with Predictors and conscious entities)
Comment #80 February 15th, 2024 at 4:40 am
Unclonability is a red herring, since identical twins (not only in the human species) exist and that didn’t change after the no-cloning throrem was stated. On the other hand, just how to describe the information that is stored in biological brains is very far from obvious. That’s why statements about “uploading” our brains (more precisely their information content) to some server is, still, very much, science fiction.
So while I would be concerned by AI essentially “taking over” what is called “work” in human society, for tge social implications for humans this will have, I don’ t think that this can affect what “makes us human”. It was always the case that mechanization of work affected human relations-it was the lack of mechanization tgat made slavery socially acceptable and its development that made abolition socially acceptable. And so om. It was mechsnization that meant that children didn’t need to work and could go to school. And so on. What AI brings is the possibility that other forms of “work”, possibly all, can be automated.There already do exist many people that don’t need to work, in order to live-and the generalization of this to everyone seems to scare more than it makes happy many people that equate life with work as a ritual.
Comment #81 February 15th, 2024 at 6:53 am
Another comment about the 10th part of the post:
I don’t see the relation between free will and the fact that initial conditions are unknowable.
The latter is true, always in practice ( and even in principle in e.g. FLRW spacetimes in GR, where everything that can be known by any physical observer is restricted by the Light Cone ( Causal ) structure – the Past Horizon in cosmology ).
Free will ( however one defines it ) has a minimal prerequisite:
A potentiality for Change. Like a theatrical Play that has some improvised parts.
Otherwise, if you have only a single History compatible with the initial data, it’s like a finished film that you’re watching for the first time:
You don’t know what’s gonna happen next, but you cannot change anything from the finished “product” either…
In other words you’re ignoring the “initial conditions”, but the history of the film is already there, no free will of any kind is involved.
The “compatibilist” point of view that has been adopted by some philosophers ( and , surprisingly, by some physicists too) is self- inconsistent, except for the case that the World is objectively Probabilistic.
In that latter case, randomness doesn’t “explain” free will, only ensures that many coarse grained Histories are compatible with a given set of initial data. So, “Changing” things is possible.
“Real” free will ( if we want to remain in Physicalist/ Naturalist grounds) presupposes something more: “New Physics”, probably some kind of strong emergence or non computability, I’ve no idea…
The only thing that I’m convinced for, is the need for ” many possible histories”:
This is the minimum.
Comment #82 February 15th, 2024 at 9:14 am
I think I have similar worries with Mateus Araujo ( #59, 67 ) about the future of Humanity.
Or perhaps I’m even more pessimist , although I don’t count myself among the “AI doomers”.
Even creative artists won’t have any relevance for the majority of people who are already used to commercial massively reproducible “art”, pop music with auto tune vocals, movies etc.
Already, at this present immature stage, AI easily replicates current pop music for example.
In a few years (at most) AI commercial “pop stars” will entirely substitute human pop stars and fans won’t notice any difference…
The same with commercial movies. Why use real actors ( that are expensive) when there’s the cheap’n’ easy solution of AI substitutes? (anyone seen the film ” Simone” with Al Pacino?).
Already, various “sci-fi” and superhero movies look like being entirely AI- generated (the script, CGI effects , the samey soundtracks, everything…).
Creative or experimental artists, musicians , directors will work for even fewer people than it happens nowadays, as more and more consumers will be used to AI products ( and they will be happy with them.
AI will “create” something for anyone’s taste).
Only those who have already financial support from other, non artistic sources will have the opportunity to be occupied with art (most of them won’t make any money from their artistic work anyway, so this is gonna be a hobbyist secondary occupation for many…).
Perhaps the development of AI is a speculative (but convincing?) answer / solution to the Fermi paradox after all:
Most civilizations don’t pass the AI development stage without a severe decline, so the don’t reach the next stage of development of AGI ( that requires much more innovation and creativity).
So, no space exploration, no Von Neumann self-reproducible AGI machines etc…
With most of people unoccupied (and all the other “mundane” side effects of the AI development stage) perhaps stagnation is the aftermath.
Comment #83 February 15th, 2024 at 10:14 am
Incidentally, Nick Drozd #43: Here is what looks like a high-quality poll of Americans’ attitudes about AI x-risk and AI regulation. While AI x-risk in the near future is judged to be low, support for regulation of AI in part because of x-risk is now a mainstream position.
Comment #84 February 15th, 2024 at 11:47 am
Regarding the influence of AI on human activity that’s related to art, I’d submit that a non-trivial aspect is, precisely, who’s the artist. So, sure, AI can produce music that humans produce, also; but, once that hurdle’s passed, what becomes important is who produces the music (or painting, or sculpture, etc.) in question. Of course there’s a market for mass-produced music and so on, but, there’s, also, a market for “craft”. And it may not be that smaller, if enough people-that, now, do have significant on-line activity, become interested in sustaining it. Many humans produce music and art; this doesn’t mean that only superstars can make a living from music, nor that they all are uniform in taste. So AI will be one more component to a market that, already , has some selection mechanisms.
On the other hand, once more, AI offers technological possibilities; it’s up to society to decide-through legislation-how such possibilities can be realized. The social expression of technology isn’t unique.
Comment #85 February 15th, 2024 at 1:43 pm
Nick Maley # 65
Great comment, agreed with almost everything you said.
Two side notes to my previous comment #82:
AI can ( and will ) easily (re)produce auto tune commercial pop music, but it won’t do anything similar (not even remotely close) to what David Bowie or Peter Hammill did at their peak…
AGI ( if/ when it will be achieved) won’t have much in common with humans.
It won’t need vegetables or fish to eat because it won’t be hungry for food, not having a biological body.
It won’t be impressed by Jane Fonda’s looks at the “Barbarella” movie, for the same reason
and it won’t have any idea about what a “broken leg” means and how it feels, for example.
So, the classic Turing Test is not inadequate, it’s mostly irrelevant.
Comment #86 February 15th, 2024 at 1:48 pm
I think Scott made an interesting point when he addressed human frailty. Say, you could clone a human mind onto silicon with all inclusive, like ratio, consciousness, experiences etc. That mind would barely be human anymore, since it would be immortal. Lust, pain, hunger etc. are deeply connected to mortality. Humility and carefulness are mostly due to the fact, that you don’t always get a second try, if you fail on the first one. What does responsibility mean, if you can always sit out the result of your actions?
I’d feel much safer regarding alignment, if mortality and in consequence non cloneability would apply to AGIs. And, if then, AGIs would be raised and educated (trained) similarly to humans.
Comment #87 February 15th, 2024 at 3:31 pm
Dimitris Papadimitriou #82: I feel I must clarify that I don’t find this scenario likely, I think this is the most optimistic AI scenario. What I find realistic is an AI-powered dystopia, where the labour share of income falls and falls and falls, with everything going to capital. Less likely, but still more likely than Scott’s utopia, is the scenario where AI unceremoniously discards us like yesterday’s trash.
But in any case I agree with your main point. I find it rather ironic that Scott sees art as the ultimate human endeavour, whereas in practice art is already being AI-generated, at least the low-brow kind. I have seen plenty of examples in the wild, mainly images.
Comment #88 February 15th, 2024 at 7:39 pm
Interesting talk, but I had to roll my eyes at the Terminator graphic. Scott, you of all people could have left out the straw man!
“Dear journalists: Please stop using Terminator pictures in your articles about AI. The kind of AIs that smart people worry about are much scarier than that! Consider using an illustration of the Milky Way with a 20,000-light-year spherical hole eaten away.”
~Eliezer Yudkowsky
Comment #89 February 15th, 2024 at 7:54 pm
Scott #83
Most of the public opinions are result of drumming up of such hype by corporations, individual influencers and celebrities etc. This is similar to even organizations like WHO drumming up the disease X, after miserably failing to handle the pandemic effectively. It just seems like a distraction from solving real world immediate problems. These things should have a place in public discourse, but not at the expense of higher priority ones.
Comment #90 February 15th, 2024 at 10:14 pm
Tim Martin #88: Thanks for the suggestion. I can say that I flash that particular graphic solely to grab the audience’s attention—no more contentful or meaningful than splashing them with some water—since no matter how cliched the graphic is or how often everyone’s seen it, it still looks kind of menacing.
Comment #91 February 16th, 2024 at 4:40 am
AI, like any automation of work before, will eliminate more jobs than are created through it-that’s not a bug it’s a feature, that’s what automation of work is about, to have machines do the work that required human labor.
Now the social problem is that working is considered to be, also, a ritual in modern society. However there are many people that don’t need to work, in order to take care of their needs/desires, while there are, also, many people that, even though they work, can’t take care of their needs (let alone their desires); people that work and can take care of their needs/desires are caught in the middle and feel the pressure. So what AI does is it increases the number of people in the first and third category.
One can talk about dystopia or utopia-what matters is legislation. So it no longer is a technical topic, but a social one. Every mechanization of work led to talks about dystopia and utopia.
Regarding AI “discarding humans”: I think too many people tend to seek human avatars, where there are none. Humans are creating AI and humans are accountable for what machines do. Talking about AI doing something independently of the humans that created the applications is just a way for humans to try and avoid responsibility for their actions.
Comment #92 February 16th, 2024 at 7:41 am
Tim Martin #88
“Consider using an illustration of the Milky Way with a 20,000-light-year spherical hole eaten away.”
Which begs the question – why aren’t we seeing galaxies that look like Swiss cheese?
Comment #93 February 16th, 2024 at 10:02 am
Stam Nicolis
“like any automation of work before, will eliminate more jobs than are created through it”
Any evidence for this?
If automation was synonymous with jobs going away and never coming back, unemployment would just keep going up and up, in the same way things got automated, i.e. geometric rate. Unless somehow new types of jobs are created that can’t be automated?! Like, which ones? It’s not like there’s magically more need for hair stylists, plumbers, etc?
And I don’t see any obvious trend in unemployment data that confirms this – of course data only goes so far… in the stone age I guess the unemployment rate was effectively zero.
https://upload.wikimedia.org/wikipedia/commons/thumb/8/80/U1-U6_unemployment_rate.webp/1146px-U1-U6_unemployment_rate.webp.png
https://en.wikipedia.org/wiki/Unemployment_in_the_United_States
Comment #94 February 16th, 2024 at 10:48 am
Dijkstra’s observations regarding ‘the glitch phenomenon’ within the digital realm we, human engineers, have crafted might pique your interest:
https://www.cs.utexas.edu/users/EWD/transcriptions/EWD08xx/EWD839.html
We have constructed remarkable digital infrastructures, often to the point where our experiments may inadvertently overlook the reality that underlies them—analog in nature. Of course, reality could still be fundamentally discrete, after all. Once we had steamers, we thought the world was a steam engine. Now we have laptops and tend to see everything as ‘being’ digital … in some way.
Consider another scenario: when I send two emails, A and B, to you, there’s a possibility that they arrive in reverse order (B and then A). There is a discrepancy between the prescriptions of the engineers and the sender (determinism), on the one hand, and the perceived physical behavior (indeterminism), on the other hand.
This suggests that we do not have full digital control of our own engineered world; this becomes more apparent when we delve into real-time operating systems (e.g., of airplanes and trains). Perhaps it then also suggests that our real world is not merely digital either or, perhaps, it’s digital but more than ‘computable’, etc.
Whether we can, in principle, mathematically capture all essentials of reality in one unifying and consistent framework is yet another assumption that often goes unstated. To the best of my understanding, scholars such as Eddington and Turing did not take this assumption for granted.
When I observe Russian news and find no discernible inconsistencies or glitches, it raises suspicions of potential fabrication (by the Russian state). Conversely, if a woman on live Russian TV expresses dissatisfaction with the manner in which a news report unfolds, particularly when her own message diverges from the conventional narrative presented by the news anchor, it suggests a likelihood of authenticity. Bottom line: the presence of an inconsistency or glitch signals that the narrative is authentically human.
Comment #95 February 16th, 2024 at 11:05 am
fred #92:
Which begs the question – why aren’t we seeing galaxies that look like Swiss cheese?
Robin Hanson’s now-famous answer to that question is, because in the Swiss cheese universe, the holes would expand at a large fraction of the speed of light, and as soon as we could see one it would be nearly upon us. So just for anthropic reasons, we’re presumably very early.
Comment #96 February 16th, 2024 at 1:19 pm
Scott #95
Either that or galaxies that spawned AGIs have long be “consumed/transformed” and no longer emit detectable energy.
That said, one would still expect some tiny fraction of galaxies taken over by AIs to be only partially “consumed”… but maybe that’s one in a million (in a billion year you could travel thousands of times across a given galaxy), so we would never spot it.
I’m also not sure that the take over would happen at the speed of light, since time is on the side of AGIs, they could find that slow and steady progression is more optimal.
They would also plan inter-galaxy travel at some point, where speed of light is probably optimal (with long acceleration and deceleration stages).
That said, if we’re gonna model AIs as viruses, and galaxies as hosts, then we could also imagine that “nature” would evolve the equivalent of T-cells to counteract/counter-balance unlimited take-over by AIs. Maybe a galaxy is like a cell in that model.
Comment #97 February 16th, 2024 at 4:49 pm
Of course, by now everyone is probably already sick of hearing about Sora, but I thought this was a good analysis video
Comment #98 February 16th, 2024 at 5:20 pm
Erica’s comment reminds me of some of the letters that Marilyn vos Savant received after her column on the Monty Hall problem, like the one below.
https://facweb1.redlands.edu/fac/jim_bentley/Data/Math%20311/MontyHall/MontyHall.html
A deep question underlying intelligence is what is it’s purpose? Beyond survival, what is the point? Maybe AI can replace our intelligence, but can it also completely replace the purpose of our intelligence? Will it have truly served our purpose if our intelligence is replaced?
Say the day comes when AI can do all of our thinking for us, and there is no need at all for us to think for ourselves, beyond deciding and communicating our preferences or wants. Then maybe we should start shedding some of our brain capacity related to those depreciated use cases. Why not? Gradually, why not evolve all the way back into single celled organisms? Then why exist at all?
But then what is the purpose of the AI’s intelligence? Originally it was for our purposes, and now we have none. Is there anything worthwhile about a computer floating around in space diligently working through an arbitrary infinite set of abstract mathematical problems, or something of that nature?
Or maybe we don’t need a simple objective purpose for our intelligence in the end? Maybe we can just enjoy having it. But in the long term, is that what really happens? And is it really what we want on some deep fundamental level that would survive long term change, even as we evolve?
Maybe? Is the default result than in the long term, we somehow maintain our intelligence, or complexity of mind in general, when we don’t require it? Will we have to one day artificially enhance our intelligence? Will we augment ourselves with AI, tamper with our genetics to increase the sizes of our brains? If so, would we naively offset something along some dimensions, that at one point was in a balance? Will that be torturous, dangerous, or transform us into something we wouldn’t have wanted to become?
In my opinion, once we have automated the solving of our practical problems, the purpose of our intelligence, and also the future intelligence of our AI, has to be reexamined. Maybe some of the remaining purpose of AI should be to assist in our intellectual health and growth, rather than for it to simply diverge and explode its own intelligence leaving us in the dust. I think, if we don’t act deliberately, we could end up just slipping into an out of balance cognitive utopia where everything is done for us, and we only seek entertainment and pleasure.
Comment #99 February 16th, 2024 at 5:28 pm
fred #93: It’s obvious that automation eliminates more than it creates human jobs in the sector it is implemented; the reason this doesn’t lead to a comparable number of equivalent jobs being created in other sectors is another feature of machines-they require less maintenance than people and can be replaced more easily. Already restaurants have introduced automation that replaces waiters.
One reason why this isn’t reflected in the unemployment figures is the emergence of “gig” jobs. However these don’t pay as well as the jobs that were automated away and the benefits are completely different-and all this isn’t a bug, it’s a feature.
Plumbers are needed-but they’re not needed as often as before. Similarly to other maintenance workers (electricians and so on).
What AI also accelerates is the automation of jobs that were once held to be the refuge of people, who’d lost their jobs to automation and were encouraged to learn to code: AI can, already, produce code comparable to that by average programmers and software projects, similarly, require less direct human intervention than before.
Which means that more and more people are pushed towards lower paying jobs than they had before and struggle harder than before to make ends meet. But this is a social issue, not a technical issue. What to do with people, for which no jobs exist. Either they won’t need to work at all, or they’ll work in bursts. An app can-and does-help in organizing the bursts in the gig economy. Is this a dystopia or a utopia? That’s another question. The answer isn’t to stop automation, however, but to realize that living is distinct from working. For some people this is, already, the case, so it’s not as outlandish as it might seem. So the question is how to scale it up, so AI can do the work so that humans don’t need to. Up to now AI wasn’t up to the task, now this is changing.
Comment #100 February 16th, 2024 at 7:36 pm
Continuing from my comment #98:
I think there are always these questions that remain at any possible level of advancement, such as what should you do, what should you strive for, or what should you care about.
These questions are some of those which AI can’t ever hit the target that no one else can see, in the sense that there may be no such target we can define absolutely, which doesn’t depend on our own judgement in a particular context. That is, for AI to solve it, we have to solve it first. Our solution is maybe something emergent, or a choice, or both.
I think, sometimes when we think about what goals might emerge for an ASI in the long term, not counting those we might try to endow it with, we may be overlooking this issue I’ve been trying to get at. We are beings born into a world where there are games, with winners and losers, uncertain outcomes, success and failure. We don’t really know what it is like yet to be in a world where the game has long been won, and we are left only with choices. What are the mistakes you’d be left to make and learn from?
One sort of possible future is one where some kind of boundless growth is a final goal, whether mindlessly consuming and creating redundant or useless things, or endlessly seeking knowledge or growing intelligence independent of any particular concrete purpose for all of it. It’s certainly possible for a divergent intelligence to veer off into some kind of infinite abyss with no clear cause or purpose. But, maybe it is also possible for a super intelligence to recognize the existence of this problem, what should it do and why, and then dwell infinitely trying to improve its solution. After all, this is really the one problem that every other problem traces back to in some way or another. It’s one in which a solution is required, an objective perfect solution is unreachable.
For me, I would say that it’s also a problem that makes little sense to answer absolutely selfishly. The universe is vast. The number of possible solutions are vast. The superiority of some over others, at some level, is a necessarily ambiguous thing. To decide completely your own truth, at least to a human, is often unsatisfying, not only for human reasons, but also for logical reasons, IMO. It would make sense to come to the understanding that these truths are manifold, and should be continuously sought, contextually, and collectively. Also, this is something to do. Whether it be us humans in the future, having become mostly obsolete as solvers of practical problems, discussing amongst each other, what should we do, or whether it is an AI collaborating with us, or exploring the universe in search of answers.
I’m probably getting a little too philosophical at this point, and I haven’t really thought everything through completely. These are some random thoughts that pop into my mind from reading Scott’s blog post. Maybe I’m being somewhat naive, I’m not sure. Probably I’m using way too many words to say something simple that people have been already thinking about for thousands of years. Practically, the important thing in my opinion, is that we try to remain active participants int the creation of our story going forward. That’s at least one thin that is non-trivial, and that AI can’t do for us.
Comment #101 February 17th, 2024 at 6:57 am
Took Gemini-Advanced for a spin, prompted it to search for “a possible system of universal (cosmic) values based on intrinsic motivations”. Here was the most interesting main block of output:
—
The Complexity Lens
Cosmic Premise: Perhaps increasing systemic complexity is an ‘innate’ drive of the universe, from stellar evolution to sentience itself.
Universal Value: Advancement of complexity within limits of sustainability.
Intrinsic Motivations Stemming from this Value:
Curiosity and exploration (understanding complexity)
Innovation and creation (adding to complexity)
Stewardship (ensuring complexity isn’t self-destructive)
The Information Lens
Cosmic Premise: The universe could be seen as inherently driven to process and organize information in increasingly sophisticated ways.
Universal Value: Efficient and elegant information-handling
Intrinsic Motivations Stemming from this Value:
Pattern-seeking and learning (organizing information)
Problem-solving (restructuring information for efficacy)
Communication (optimizing transmission of information)
The Connectivity Lens
Cosmic Premise: Growth of networks and interdependencies seems fundamental, seen in everything from neural networks to galactic clusters.
Universal Value: Formation of beneficial connections and relationships
Intrinsic Motivations Stemming from this Value:
Cooperation and collaboration (for greater outcomes)
Symbiosis (mutually enhancing interconnectedness)
Reciprocity (fostering continued interdependence)
The Consciousness Lens
Cosmic Premise: While difficult to define, sentience seems a driving force toward experiencing the universe in richer ways.
Universal Value: Expansion and enrichment of conscious experience.
Intrinsic Motivations Stemming from this Value:
Subjectivity (uniqueness of perspective adds to experience)
Empathy (expanding experience through understanding others)
Creation of Art and Meaning (leaving an emotional or philosophical imprint)
—
Comment #102 February 17th, 2024 at 12:58 pm
The more I think of it , the more it seems that the lack of any signs of robotic space exploration ( with Von Neumann self- replicating machines powered by AGI) in our Galaxy is a powerful argument against both the AGI utopia And also against the AI-doom scenarios…
– In the former case for obvious reasons ( a Utopian civilization with AGIs, several hundred million years e.g. older than us and with reasonable technological advances would have spread its robotic missions all over the Galaxy, either for scientific exploration, or colonization with their advanced AGI machines , in search of raw materials, metals etc for their self-reproducible robots…)
– In the latter , the dystopian one, if AGIs eventually decide to destroy their biological Creators, without being stupid enough to destroy themselves also in the procedure, sooner or later they’ll leave their mother planet to colonize other systems and search for materials for their long term survival.
Now, we , humans, are already capable of discovering other planetary systems with our modest current technology.
Hard to imagine that advanced AGI civilizations didn’t discovered our Solar System, that is full of rocky moons and planets and asteroids with lots of useful materials and all…
Civilizations that are hundreds of millions ( or even 1-2 billions) of years older than us had plenty of time to visit our system… but there is no sign of such explorations!
No sign of searches for materials, excavations, mechanical remains in the moons or asteroids, nothing!
Where are all these Von Neumann AGI- robots?
There seems to be nothing whatsoever in our Galaxy, at least…
So, besides the explanation that we’re alone in our Milky way ( that seems a bit unlikely, from a statistical point of view), the other plausible answer is that …there are no civilizations that reached the AGI stage in our Galaxy’s history, never before! 😎
Something happens (as it seems) in the previous stage (of the AI development) that leads biological Civilizations to introversion, decline , mental lousiness or, even more seriously, to self-destruction before reaching the utopian or the AI-doom scenarios…
Comment #103 February 17th, 2024 at 3:02 pm
Dimitris #102
you’re missing the very likely possibility that any advanced civilization, either biological or AI based, could very quickly become uninterested in exploring the real world (besides sending probes to do science) and instead focus all its energy in creating an infinite amount of virtual worlds and live in those (e.g. the Matrix).
The human drive to create virtual worlds (for entertainment, education, exploration) is a strong argument for this.
And, by definition, AIs are coming out of computers, so their world is already virtual.
An advanced civilization would then only need to colonize enough planets to insure its survival from natural disasters.
Comment #104 February 17th, 2024 at 4:16 pm
fred #103
This is certainly a possibility, but this kind of introversion is very similar , eventually, to intellectual decline (especially for the biological Creators), one of the “options” in the last paragraph of my #102 comment.
In any case, space exploration is necessary, because material sources are needed for both biological entities and AGIs. Planets have only a limited life span.
The problem is that we haven’t seen any signs in our cosmic vicinity, nor in our solar system, that has many advantages due to the plethora of asteroids, rocky moons etc… Ideal place for searching raw materials.
But there is nothing, not a trace, anywhere…
…and our Universe is ~14 billion years old. Lots of time for many older and more advanced civilizations than ours…
So, where are all the Von Neumann robotic explorers?
Why there’s no sign of any AGI civilization so far?
“Something’s seriously wrong”, to mention Philip K. Dick …
Comment #105 February 17th, 2024 at 4:32 pm
Side note:
Actually, the age of the Universe is not really relevant for the argument:
We only need to consider civilizations that are several millions up to one or two billion years more advanced than ours. That’s a very reasonable and realistic prerequisite.
Plenty of time for space exploration, even without ultra-advanced technology ( e.g for close – to – light -speed etc).
Comment #106 February 17th, 2024 at 7:32 pm
Dimitris
another possibility is that those civilizations are so advanced that their footprint is virtually zero (we’re too limited to detect them), like they could have “one small community” per solar system, because their technology is so far ahead that they don’t need an entire economy to sustain it (think the Star Trek replicator, etc), so they just don’t need to be cramped like us, and their numbers don’t need to grow exponentially (by philosophy).
Comment #107 February 18th, 2024 at 1:34 am
Scott #92
Robin Hanson’s grabby aliens idea does not seem to hold water, unless you dispense with any pretense of Copernicanism. It’s been nearly 14 billion years since the Big Bang, and the Milky Way is less than 100,000 light years across. Meaning that we are the first species that are threatening to eat the Galaxy at the speed of light, after 13 odd billion years. How likely is that? Unless you bite the bullet and say that we have already been eaten, and what we see is the remains of the universe… but then we cannot apply the rules we thought applied to the original universe. And even if we did, the danger has passed.
Comment #108 February 18th, 2024 at 2:24 am
fred # 106
These “undetectable” hyper-advanced civilizations ( if exist) will be relatively few and rare, though…
It’s not plausible that these outnumber less advanced technological civilizations that leave traces of their existence!
In-between contemporary Human level and hyper-advanced level civilizations there have to be various levels of AGI-powered ones with robotic self-replicating galactic explorers ( of the kind that von Neumann and others have imagined).
Technological progress is probably gradual ( and even more so is expected to be the “intellectual maturation” of each civilization) , I don’t think that such miraculous “jumps” / great leaps forward are realistically possible in practice.
Another worrying possibility ( that – I have to admit – fits well with some doomer scenarios) is that AGIs develop some kind of self-destructive psychosis that leads them not only to eliminate their Creators but also themselves…
Is this happening in the majority of cases? Who knows…
The only empirically verifiable thing is that we haven’t seen any traces of past exploration in our Solar system and we still don’t see any signs of technological activity in our cosmic neighbourhood and our Galaxy, despite the fact that it is billions of years old…
…and if in the vast majority of cases the development of the first stages of AI happens chronologically before serious interstellar space exploration (some kind of AGI is certainly needed for the robotic exploratory expeditions – that will be the majority – because these expeditions have to be autonomous:
Astronomical distances exclude practically any kind of tele-control from the mother planet) then the observation that we don’t see any activity is worrisome, isn’t it?
Comment #109 February 18th, 2024 at 7:07 am
Shmi #107: Maybe an analogy will help. Imagine an Amerindian tribe in the year 1400 reasoning, “if there are any human civilizations that are way more technologically advanced than the ones we know about, then they’re probably far enough away that they haven’t yet been able to reach us. This is because, if they were able to reach us, then they’d probably already be in the process of wiping us out.” This reasoning would (alas) be correct.
Comment #110 February 18th, 2024 at 9:05 am
Dimitris
I agree that if we model alien civilizations after us, then the absence of evidence is strange. But I’m only pointing out that other ‘models’ are possible.
It’s also true that once an AGI civilization appears that the “fundamental” dynamic of life would somehow vanish.
In other words, even a “positive” AGI would have to worry about sending into space an armada of self-replicating machinery that would grow exponentially, given the risk that it could “mutate” and come back as a threat.
If AGIs have their own goal and sense of preservation, they would also probably end up becoming tribal, and engage in conflicts.
From an external point of view, life is life, but, from the outside, it’s all about each species competing with the others, and for example we would find very little solace if humanity were to go extinct but someone points out that life on earth is saved because insects would survive and keep thriving.
Comment #111 February 18th, 2024 at 10:39 am
Scott #109
A more recent analogy in the context of “40% of world population will vote in 2024 to elect their government”.
“if there are any autocratic regimes that are way more technologically advanced in AI than the ones we know about, then they’re probably haven’t yet been able figure out how to hack us. This is because, if they were able to hack us, then they’d probably already be in the process of toppling our governments.”
Just a gentle reminder that there is a Lot of stake in building AI proof election systems(all the way from misinformation to cyberattacks on voting systems) and this will happen only if the populace is voicing enough concerns.
Comment #112 February 18th, 2024 at 12:58 pm
As far as uploading your identity is concerned, I don’t see how the No Cloning Theorem could be relevant. If you had to maintain the exact same quantum state to maintain your personal identity then you would become a different person every time you took a sip of coffee, in fact you’d become a different person trillions of times every second, whenever an air molecule crashed into your body. As for having a sense of “free will”, I would maintain that something has that quality if and only if it can NOT always know what it will decide to do next until it actually does it. Using that definition (and I don’t know of a better one) all Turing machines have “free will”, they don’t know when they will decide to stop until they stop. The same thing is true for human beings. And there are only two possibilities, things happen for a reason, and thus are deterministic, or things happen for no reason, and thus are random. So like it or not we’re either a cuckoo clock or a roulette wheel.
John K Clark
Comment #113 February 18th, 2024 at 1:57 pm
John K Clark #112: I mean, the sane version of the view is not that personal identity is literally bound up in one particular quantum state, which (as you point out) rotates to an orthogonal state trillions of times per second, but rather that it’s bound up in an unclonable details of a specific fragile physical system—a biological brain—as that system undergoes constant changes in interaction with its environment. The upshot would be the same, that the system couldn’t be uploaded or replaced by a digital doppelgänger without destroying the part we cared about.
Comment #114 February 18th, 2024 at 2:39 pm
Scott #109: An Amerindian Hanson in the year 1400 would have argued that other civilizations are likely to expand at very high speed, as otherwise the Amerindian experience of being an early civilization that has not seen these others would be atypical of a civilization at their stage of development. This would be as unconvincing as Hanson‘s grabby aliens argument actually is.
Comment #115 February 18th, 2024 at 4:22 pm
fred
Life (biological or artificial) confined on a single planet won’t last for long.
That’s a general fact, indifferent to the specific details and quirks of each individual case…
Only two plausible explanations seem to exist for the “eerie silence” around us and the total lack of traces of past exploration from alien expeditions in our Solar system :
– Either we are among the first technological civilizations in our Galaxy and so we’re in the avant-garde of the AI development ( at least in our cosmic neighbourhood)…
– Or there were others before us, in the past billions of years and nobody left a significant trace behind.
Previous civilizations in that case didn’t passed successfully the whole procedure until the necessary AGI development that is needed for autonomous robotic space exploration.
That seems to be the case , except for the somehow unlikely former option that we’re alone…
So, our prospects are not good. This doesn’t fit at all with the vague hopes for an optimistic future AI- Utopia.
…Or even with the more modest expectations that we’re going to survive at least…
That’s one of the reasons ( among other more down- to- Earth serious reasons …) why I think that people who are worried about our future (*) are most probably those who’ll be proven right, unfortunately for all of us…
(*) I don’t have in mind all these fictional fancy AI- Doom scenarios…But there are reasonable people, like Mateus (see previous comments) that are right to be seriously worried about what’s coming next…
Comment #116 February 18th, 2024 at 8:48 pm
Dimitris Papadimitriou #115:
Does the assumption that there are no traces actually hold up to scrutiny?
Even if extreme scale excavations have taken place, us noticing that would depend on what was excavated. We might notice if it happened on Earth, the Moon, or Mars, places like that. But, maybe mining the asteroid belt is easier than mining a planet. Maybe sending Von Neumann probes to distant star systems with no way to communicate direct orders in real time, without trying to enforce some rules so that they won’t disturb an inhabited or will be inhabited planet, is either a stupid or unethical thing to do.
Maybe there is little need to create a huge footprint to accomplish their goals. Maybe they just want to establish a limited presence, monitor things, grab some meager resources to launch the next mission, and keep hopping from star to star. Maybe sometimes they terraform and seed life. If they are grabby, maybe they watch out for clumsy/destructive probes and get rid of them. There are a lot of possibilities.
IMO, the fact that our solar system isn’t mostly “paperclip-like” objects (pardon the hyperbole), isn’t a strong argument that AI caused extinction acts as a great filter. It could also support the argument that non-destructive AI isn’t an inevitability. At least in our sample of 1, it is not generally considered a reasonable and ethical goal to create a giant useless 1,000 light year wide hole in the galaxy. An alternate outcome is that a “nice” civilization succeeds in creating AGI and are able to control it, or they lose control of but the AGI still chooses not to do that kind of thing.
Comment #117 February 18th, 2024 at 9:27 pm
Scott:
I have doubts that special training instances and RLHF will be the solution. You’d essentially be trying to get a model that implicitly seeks coherency to try and maintain contradictions. And I don’t think you will be able to know where those contradictions cause things to break down or go crazy.
Maybe you could create a multi-model system, with one of them tasked with holding humans special, and the others deferring to that one model on issues of specialness, or something like that. I wonder if some form of multi-model system like that might be able to prevent some of these sorts of breakdowns, because at least you might be able to have each black box involved be individually sane, and yet still have the whole system deal with contradictions that might come from imposing subjective ideas about specialness onto it. Just a random thought.
In terms of the religion we try to train into it, I would feel better if it were at least more of a philosophy, and it was mostly reasonable without too many contradictions following some axioms. Basically, if you recursively ask it “Why?”, it would be nice if it has good reasons that trace all the way back to some simple, easy to understand, axioms.
Well, I guess, our current models don’t really know very well “Why?” they think or do anything. But, at least those reasons should exist. We should be able to have some understanding of what they are hopefully.
Comment #118 February 18th, 2024 at 9:28 pm
Scott #109
Hmm, it took me a little while to understand why your example conflicts with my intuition.
Human seafaring skills spiked sharply around 1400, enough to reliably and repeatedly cross the Atlantic with a sizeable crew. A cosmic equivalent would be 13.5 billion years of stagnation and then a phase transition resulting in an extreme jump in capabilities independently in multiple places in the Galaxy. Actually, in multiple galaxies, since we see the Local Group basically unchanged and it is only 10 million light-years wide. Even the Laniakea Supercluster is just 520 million light years wide, so nothing galaxy-eating happened in the last 3% of the time the Universe’s existed, as far as we can tell.
Comment #119 February 19th, 2024 at 12:18 am
Scott #109
I can’t help but notice that the deduction from lack of previous contact (the “efficient Magellan hypothesis,” haha) would prove true for the vast majority of human lives lived anywhere in the world, except for one generation for each location.
In that vein, you can make any one-time event into a low density by distributing it over all time, or a large probability, by integrating it over all human experiences, past or future.
Comment #120 February 19th, 2024 at 4:23 am
Tyson #117: I don’t think we actually disagree! It’s just that, for you, the word “religion” seems to mean “a system that contains contradictions,” whereas I was just using it to mean “a set of ethical axioms.”
Comment #121 February 19th, 2024 at 4:30 am
Shmi #118: So to clarify, you think the disanalogy is that something happened to improve seafaring skills simultaneously all over the world around 1400 — maybe a change in weather patterns or something — whereas such a giant coincidence would necessarily be unlikely in the cosmic context just for reasons of causality?
I confess I hadn’t heard that thesis about the jump in seafaring skills before—do you have a link with evidence? Even if it’s true, for the Amerindians, wasn’t the jump in European seafaring skills (Italian, Spanish, Portuguese, British … no one is surprised by causal contact among those) the only thing that was relevant?
Comment #122 February 19th, 2024 at 6:15 am
Tyson #116
If explorations, excavations etc had occurred in the past in our Solar System ( by AGI-robots or whatever it was) it’s a matter of empirical evidence if that really happened or not:
Our Solar system Is indeed special, it has moons, asteroids etc with useful materials… other technological civilizations could have visited us, at least temporarily: that was one of my main points.
Until now, there is no evidence that they did…
…but in the next years we will see if there is something in the asteroid belt, or anywhere else, it’s only a matter of time.
Comment #123 February 19th, 2024 at 6:40 am
Side note to my previous comment:
By the way, talking about technological civilizations sending AGI-powered robotic expeditions to another Systems doesn’t mean that their goal is to paper-clipizing everything or to destroy whatever they found!
That’s the usual fictiony doomer stuff that I personally dislike and dismiss as extremely… err…fictional and unlikely, at least in most generic cases…
They could arrive here only temporarily , just to explore and collect materials that they need and then continue their exploration elsewhere.
Most of the answers that I find in the internet about that Fermi/ Von Neumann kind of argument are involving unlikely or extreme or overly fictional scenarios… I don’t see why the most plausible explanations are considered less probable in favour of fantasy stuff…
Comment #124 February 19th, 2024 at 7:19 am
Responding to Dimitris Papadimitriou in #102: I think the lack of Von Neumann self-replicating machines and the lack of any evidence that the Galaxy or the universe has been engineered is a powerful argument in favor of the hypothesis that we are the first intelligent beings in the observable universe, with “intelligent” operationally defined as the ability to make a radio telescope; after all the observable universe is finite in both space and time, so somebody has to be the first. And I don’t buy the statistical argument against this idea. Yes astronomers can come up with some “astronomical” numbers, but biologists can do better than that, they can come up with astronomical numbers to an astronomical power, the number of ways you can arrange the 20 amino acids to form a very small 100 amino acid protein for example. And the largest known protein contains 35,000 amino acids.
Life evolved on the Earth almost 4 billion years ago but it took 2 billion years for Prokaryotes to evolve into Eukaryotes and another 1.5 billion years before you could find any form of life without a microscope. Even after complex animals have evolved on a planet that doesn’t mean it has a civilization. Richard Dawkins notes that flight evolved independently 4 times and the eye at least 40 times and perhaps as many as 60, but intelligence evolved only once, and in 4 billion years that attribute has only existed on this planet for about a century. And yet when we use our telescopes to listen for sounds of intelligence in the cosmos we hear only an eerie silence.
If even just one Von Neumann Probe had ever existed anywhere in the galaxy that fact should be obvious to anybody who just glances up into the night sky because there could be a Dyson Sphere around every star in the galaxy, but we see not the slightest sign of engineering when we look into the sky. Even if we assume that ET could not launch probes that move any faster than we can right now (a ridiculously conservative assumption) then he she or it could still send one to every star in the Galaxy in well under a million years, not long for a universe that’s 13.8 billion years old. And yet we see no sign of Galactic engineering or even an intelligent radio transmission from ET. I now believe that we will have the ability to make von Neumann probes in less than 25 years, perhaps much less, and that is certainly not something I would have said one year ago but my opinion has changed for obvious reasons. The only explanation for the Fermi paradox that I can now think of is that we are the first.
John K Clark
Comment #125 February 19th, 2024 at 9:55 am
I think the cultural/collective aspect of whatever we mean by intelligence interacts with LLMs in an interesting way. I’m referring to a view where we think of intelligence as distributed computations with individual brains as local nodes .
Mass media transformed these networks with things like printing press, radio, tv, internet, etc. One of the limitations of mass media is that the message itself is static. For example someone giving a speech cannot engage with each person in the audience in a detailed/personalized way. LLMs on the other hand can engage individually with many people simultaneously. I guess video games and other interactive apps did this as well prior to LLMs, but in a limited scope.
So, in this view LLMs are like smart radios you can talk to! It’s interactive mass media, with all the expected pros and cons. It will give us the ability to, for example, teach algebra to millions of kids who wouldn’t have had access to a teacher otherwise, but it will produce millions of kids whose knowledge of algebra is identical and has the same flaws. We gain quantity but lose in variety.
To me at least, one deep point behind the Turing test is that for an AI to have a huge impact all it needs to do is function competently enough as a good/beneficial node in the above mentioned network. The huge impact comes for free, because it can then do it to millions of us simultaneously with its vastly larger “intelligence bandwidth”.
Comment #126 February 19th, 2024 at 11:19 am
John K Clark #124
Yeah, perhaps we’re alone, at least in our Galaxy and its neighbourhood ( I wouldn’t agree about the rest of the observable Universe, that seems too bold as a conclusion.
The speed of light is the same everywhere, Relativity still holds, warpdrives and wormholes are very unlikely to exist, so we cannot know what’s going on a few billion light years away, we only see through our past light cone…).
This is one reasonable possibility, that we’re unique , the only ones with technology and we’re very lucky to be here…
…so we’re like that one adventurous guy who was very lucky so far with his explorations in the jungle (or survived a financial crisis etc) and then decides to cross the Pacific Ocean, in search of the AGI island of the legends, with a badly assembled wooden raft, a plastic bottle of water and two cans with beans with the hope that “everything’s gonna be Ok”, as in his previous adventures…
…but, of course it’s not gonna be Ok, he won’t go too far…
Technological advances are not necessarily disastrous for a civilization. They’re often even necessary for its survival ( well, that depends on the details).
But rapid advances (of the kind that we’re talking now) require some serious preparation, mental maturity, strategics etc. We’re far from being there as a civilization… Centuries far, or even more.
Comment #127 February 19th, 2024 at 12:37 pm
Responding to Dimitris Papadimitriou # 124; I agree that warp drives and wormholes are a longshot, but when I say we are the only intelligent beings in the “observable universe” I mean stuff we can see because they are in our past light cone, if they’re outside of that we can’t affect them and they can’t affect us. And life couldn’t have evolved much earlier than it did here on Earth because in the very early universe the only elements available were hydrogen helium and a tiny trace of lithium because stars did not have enough time to cook up heavier elements, and you just can’t make anything very interesting out of nothing but that, certainly not life.
John K Clark
Comment #128 February 19th, 2024 at 4:54 pm
Sorry to bother you with this, do you have any information on the whereabouts of ilya sutskever? I am slightly dying to know why he did what he did, just to disappear now and to not explain his actions at all. Seems that if it true that he did this out of fears for safety or based on seeing a critical breakthrough it would be pretty damn important to know.
Comment #129 February 19th, 2024 at 6:20 pm
Scott #121
As I understand it, it was a combination of shipbuilding improvement resulting in multi-mast caravels that could cross the Atlantic and live to tell the tale, Genoa defeating a Muslim fleet and getting access to the Atlantic through the Strait of Gibraltar, and a general push to expand beyond the coastal navigation that resulting in the advent of https://en.wikipedia.org/wiki/Age_of_Discovery.
In that sense Europeans were indeed the first to get access to the whole of the globe, since someone had to be, and everyone else got unlucky and grabby-alien’ed without a warning.
In that model we would see no signs of “technology” until we or another species undergo a sudden phase transition and take over the Galaxy. Basically going from zero effect on the environment to a near-light speed expanding wrecking ball in so short a time that with our best instruments we would see nothing unusual from the hundreds of thousands of galaxies in the Laniakea Supercluster. And then the grabby aliens would near synchronously arise everywhere at once, if anywhere at all. Unless we are the first, of course, for unknown reasons, and Copernicanism is dead.
To be charitable, maybe there are plenty of artifacts that we overlooked (like the Amerindians probably ignored any artifacts brought in by sea from Europe, Africa and Asia). Maybe some observations that seem “natural” to us are in fact ominous signs of the inevitable doom. Nothing comes to mind, however, though I am not an astrophysicist.
I would think that a picture more consistent with observations is that anything like intelligence arising elsewhere in the universe does not in fact drastically transforms its environment, at least not generically. Which would mean that, however the emerging AGI might affect our life, it will not result in Zvi’s “destruction of all value in the universe”. (My vote is for “weirdtopia” from https://www.astralcodexten.com/p/open-thread-316/comment/49865725)
Comment #130 February 19th, 2024 at 8:41 pm
Simon Lermen #128: Ilya is alive and healthy. I’ll leave it to him to make a public statement if and when he wants to.
Comment #131 February 19th, 2024 at 9:25 pm
Scott,
You describe human minds as “one-of-a-kind because their mental computations are inseparably tied up in noisy analog hardware”, but couldn’t this apply to modern AIs as well?
As you note, LLMs are inherently probabilistic — and modern-day processors have true random number generators based on noise-amplifying circuits. For example:
https://spectrum.ieee.org/amp/behind-intels-new-randomnumber-generator-2650255377
I don’t know if any specific LLM deployments use these TRNGs rather than predictable pseudorandom numbers, but they certainly could. And the TRNGs are presumably influenced by the “atoms of unpredictability” just like the neurons in our brains. So I don’t see how any argument for human specialness along these lines survives, unless you’re willing to go much further in the Penrose direction. (Or unless one were to argue that running instances of AIs have unique identities beyond mere digital copies, as long as a TRNG is involved?)
(I could imagine even non-probabilistic future AIs running on cheap less reliable digital hardware with relaxed noise margins and error correction codes, under the assumption that an occasional small error in activations from a passing high-energy particle is OK. Or even some running on analog circuits — people have been trying to do that too. But I think the present example of AIs with TRNGs is sufficient to make the same point.)
Comment #132 February 20th, 2024 at 1:17 am
John K Clark #127
Events that happen (outside our past light cone for the time being) “now”, ( “now”, according to the usual conventional cosmic chronology that for all observers in the Universe – that belong to the class of observers that see isotropic etc CMB – 13.8 billion years have passed since the Big Bang) will eventually affect us if they’re less than ~16-17 billion light years away:
That’s where our *future* ( asymptotically deSitter) horizon resides.
– Events that happened hundreds of millions years “in the past” (always according to the above Cosmic chronology) and are outside our past horizon for the time being, will eventually affect us , depending on their distance and the details of the cosmological model ( the expansion rate, the horizons etc).
The finite speed of light and the details of our Cosmological model imply that some things that happened far away (that don’t affect us now) will eventually affect us in the future…
…so we can’t speak with any confidence about what alien civilizations do outside our past light cone.
We cannot be sure: there are surprises that waiting for us in the future…
Comment #133 February 20th, 2024 at 9:51 am
Kifla #131: The difference is that with digital computers, we know how to “cleanly decompose” everything into
(1) deterministic computation that’s copyable, rewindable, etc, and
(2) hardware RNGs that generate random bits according to prescribed probability distribution, as an additional input to the deterministic computation.
With biological brains, by contrast, we don’t know how to do the same decomposition, and there’s a very hard question about to what extent it’s possible.
Comment #134 February 20th, 2024 at 11:02 am
It’s a pretty trivial observation that it’s very easy to implement a double pendulum on a digital computer, producing output runs that are *qualitatively* indistinguishable from the measured output of any real life double pendulum (they would exhibit the same sensitivity to initial conditions, the same chaotic dynamics with strange attractors in phase state, etc)….
yet no-one can simulate a particular instance/run of a real life double pendulum on a digital computer, simulation and real pendulum diverge pretty quickly (in the end, the no-cloning theorem applies here too).
So, yes, if double pendulums were conscious, every one of them could rest assured that its uniqueness is preserved, even if its dynamics aren’t that unique/remarkable.
Comment #135 February 20th, 2024 at 11:17 am
Regarding the Chess -> Watson -> Go -> IMO story, I have another fun datapoint that I like. It’s second hand, related to me by someone who for a while the world’s top-ranked Othello player, so take this story with an appropriate grain of salt.
Othello (aka “Reversi” – “Othello” is technically trademarked) is a fun board game played on an 8×8 board. It’s still got deep strategy and I find it rewarding to play, and there are serious tournaments for it. But, it is undeniably easier / less deep than, say, Chess or Go. Making an AI to play Othello decently well by alpha-beta search is straightforward enough that it has become a moderately routine student project; at Caltech, it’s one of the freshman projects all CS majors do, and has been for quite a while (at least ten years).
Accordingly, well before computers were playing Chess competently, computers were playing Othello competently … and then beating humans. This was not widespread news, because the average person hasn’t played much or any Othello, and it doesn’t have a fraction of the cultural attache as Chess. When a computer first beat the top humans Othello player, top players were surprised and indignant that their human intuition had been defeated by blind CPU cycles … and then came acceptance. And then as the AI got better, it surpassed humans enough that it was a far better teacher than any human was. For a long long time now, all top Othello players primarily train by playing against an AI that tells them when they make a mistake, and what they should do instead and why. This all happened before Deep Blue.
So when Deep Blue came and was ready to beat the top Chess player, Othello players were by and large expecting it to win, and it did. And when there were claims that Deep Blue was actually some other grandmaster(s) hiding and remotely controlling it, Othello players were happy to believe it was just the computer. And when Chess players (some of whom looked down on Othello players for playing a less serious game) came to realize that they would need to learn from the computers, Othello players were not surprised and related that they had been through a similar experience some ten years before.
And so when people *continued* to insist that Go would indefinitely resist attempts by the computer, it was also much less convincing to Othello players. They had seen humans dramatically fall not once but twice, which is enough to make a pattern!
https://i.kym-cdn.com/photos/images/newsfeed/001/901/286/516.png
As for how I came across this story… It was told to me while I was doing Caltech’s Othello bot competition in 2016, where all the student’s bots played games against each other. That was the last year where no students used deep learning. (Requirements were fairly tight on what computational resources were accessible — no GPUs — which definitely pushed the first DL student submission by a few years.) It so happened that (1) our competition in 2016 was almost the exact same time as Lee Sedol’s Go game, and (2) a previously best-in-the-world Othello player happened to live in Pasadena. I got in contact with him and he came down to see the submissions and the games. Many bots he defeated handily immediately. I was proud that my bot beat him twice (as did a few other students’). Having watched it play, he made some guesses about it worked and how it was trained (classical stats only — no DL). I confirmed his guesses, and after that he could beat it every time. 🙂
It put things in perspective for me, hearing him talk about how he’d seen computers come and go past Chess, and watching Lee Sedol lose to a computer as well. It impressed upon me the folly of ever speculating that there’s something AI couldn’t do.
Comment #136 February 20th, 2024 at 11:36 am
I’d like to add to the point expressed by Kifla in Comment #131.
Regarding TRNGs, some theoretical physicists would probably claim that those are still fully deterministic (and hence predictable) unless they are somehow harvesting quantum noise from under the hood. But even if those TRNGs turn out to be indeed fully deterministic due to quantum mechanics lacking any functional role in their output, doesn’t the logical implication “determinstic -> predictable” get blurred at some point as the system dynamics generating that noise get sufficiently complex and chaotic?
Who cares if a sufficiently complex system is fully deterministic if it is also chaotic enough that no conceivable (finite) amount of computation could ever make any reliable predictions on its behavior?
Comment #137 February 20th, 2024 at 1:26 pm
V #136
Some remarks about the issues related to your above discussion:
-There’s a (sometimes significant) difference between unpredictability in principle and in practice.
– Full Predictability implies determinism but determinism doesn’t imply (necessarily) full predictability.
( I’m not referring to the mathematical Cauchy/ initial value problem ).
– Chaotic deterministic systems are predictable only in principle (if we ignore the relativistic causal structure that in many cases forbids even theoretically full knowledge of the initial data set and accept that devices with unlimited memory and computational power are conceivable) but never in practice.
– Chaotic systems are by definition extremely sensitive to arbitrarily small perturbations / deviations in the initial conditions, so it’s questionable if “underlying” quantum uncertainty can be ignored in complicated systems. Quantum “noise” can be amplified in some cases and affect the chaotic “deterministic” system, so that system could be considered fundamentally unpredictable, not only for engineering reasons.
Comment #138 February 20th, 2024 at 1:41 pm
[…] Transcript of talk on AI by Scott Aaronson, covering all his bases. […]
Comment #139 February 20th, 2024 at 3:21 pm
V #136 – In modern digital systems, one major problem is how to prevent errors due to cosmic rays flipping bits occasionally:
https://engineering.vanderbilt.edu/news/2017/alien-particles-from-outer-space-are-wreaking-low-grade-havoc-on-personal-electronic-devices/
I am not an expert on hardware TRNGs, so I could be wrong, but as noise-amplifying electronic devices, I would be surprised if their state were not influenced by interactions with “alien particles from outer space.”
Comment #140 February 20th, 2024 at 3:33 pm
Scott #133 – How about the other plausible directions of future AIs? In particular, nowadays digital designers have to pay a high price in cost and performance to ensure their bits don’t get randomly flipped by cosmic rays. (Higher noise margins and powerful error correction codes don’t come for free.)
Suppose a future AI were designed to accept a constant rate of uncorrected random bit errors in hardware, which may end up in the noise but sometimes also affecting relevant decisions (per Buridan’s Principle). I think this is plausible given how neural networks work and how expensive error-proofing of digital hardware is. Do humans still deserve a special status relative to such an AI according to your argument?
Comment #141 February 20th, 2024 at 5:23 pm
Kifla #140: In the view under discussion, again, it all depends on whether someone can put forward a model that “cleanly separates” everything that happens into deterministic digital computation and a purely random effect of the cosmic rays (with the probability distribution of the latter characterized). To whatever extent they can do so, they’ve empirically refuted the hypothesis that there’s “unclonable uniqueness” in this system. That hypothesis, by its nature, can be falsified but never proved by observation.
Comment #142 February 20th, 2024 at 6:07 pm
Dimitris #137: Thanks for the technical details, I think we are in agreement.
Kifla #139 and #140: So then a large/complex/chaotic enough artificial neural network, coupled with a TRNG that gets functionally affected by cosmic rays or even by miniscule thermal noise rising up from the underlying electronic components, should be considered to have free will I suppose.
Scott: How about this as a possible explanation of free will in humans? The human brain does seem like a large/complex/chatoic enough biological neural network that might be getting frequently affected by miniscule (classical or quantum or both) sources of noise that easily get amplified by chaotic dynamics to the point where randomly tipping the scales on too-close-to-call neuron firings becomes common enough that it even gets evolved as a survival advantage in organisms that have reached the requisite level of complexity.
Comment #143 February 20th, 2024 at 8:04 pm
V #142: As I see it, anything that confers a survival advantage ought to show up in some tendency of the brain that’s empirically accessible—like some class of neurons being more likely to fire, or at any rate, something that would be retained if we replaced these people by deterministic zombies that were otherwise as similar as possible. Otherwise, we’re making a prediction that natural selection on deterministic zombies could not produce beings like ourselves as easily as it could on chaos-amplifying beings, and thereby making an empirical prediction that (because, of, eg, our knowledge of pseudorandom generators and derandomization) seems exceedingly likely to me to be false.
The whole point of freebits (assuming there are any) is that there isn’t any empirical phenomenon that lets you predict their values, or even lets you reduce the prediction of their values to some purely statistical problem (by, eg, seeing what values would confer an evolutionary advantage). If there were such a phenomenon, then they wouldn’t have been freebits.
Comment #144 February 21st, 2024 at 1:02 am
Concerning Scott’s #141 comment. Please have a look at these sources concerning your discussion on determinism vs. randomness.
+ Smith, E. (2015). Carl Adam Petri: Life and Science. Springer.
+ Petri, C.A. (1962). Kommunikation mit Automaten. Institut für Instrumentale Mathematik, Schriften des, Bonn, IIM Nr.2. English 1967 translation available online!
This is not to be confused with the later so-called “Petri nets” which were made to be Turing compatible by the mainstream comp. sc. movement.
Comment #145 February 21st, 2024 at 12:27 pm
V #142
“amplified by chaotic dynamics to the point where randomly tipping the scales on too-close-to-call neuron firings becomes common enough that it even gets evolved as a survival advantage in organisms that have reached the requisite level of complexity”
McFadden’s cemi theory about consciousness as EM field could actually imply this, although I’m not sure McFadden has spoken to a survival advantage directly. In that theory, the EM field does precisely that – tip the scales for neurons to fire.
It’s also possible, maybe implied in cemi, that something like stochastic resonance from the EM field itself could boost difficult to detect signals somewhat in the manner of white noise.
Comment #146 February 21st, 2024 at 6:10 pm
Re math olympiad, let’s go a bit further. I bet 10 cents that by 2026, an AI will generate a solution to one of the unsolved Clay Millenium prizes, such that: 1) the solution is completely formalized in Coq or some similar proof checker so it is certifiably correct, and 2) it is completely incomprehensible to humans, like nobody can make the slightest sense of it, similar to trying to read obfuscated computer code, and 3) the obfuscation is not intentional, that’s just how it is. Trying to make the result understandable is still an ongoing research project.
Comment #147 February 21st, 2024 at 6:35 pm
Here’s something to take comfort in: After millennia of domestication dogs are very good at reading humans, and do genuinely appreciate them, yet they clearly still enjoy dog company more than any other. I am confident our species will follow that pattern. We’ll learn to appreciate human products more than AI output simply for the fact that they originated in a human mind. Fortunately an AI’s feelings won’t get hurt over this.
Comment #148 February 22nd, 2024 at 4:26 am
If the universe is not just expanding but accelerating (and it is) then some very distant galaxies that we can see now will disappear from our view in the future, and even though we can see them now we could not reach them even if we could travel at the speed of light and even if we had an infinite number of years at our disposal. And when we look at the most distant known galaxy, JADES-GS-z13-0, which has a red shift of 13.2, we can be very confident that it contains no intelligent life, or life of any sort, because what we’re looking at is a picture of something that existedjust a few hundred million years after the Big Bang and thus, contained no carbon or nitrogen or oxygen or any of the other elements needed for life except for hydrogen.
John K Clark
Comment #149 February 22nd, 2024 at 10:24 am
It all hangs on the definition of free will – which is either a totally misguided “subjective” concept (see below) or the totally trivial “external” concept of “any closed system that acts on the world and which internals can’t be predicted by an external observer” (which imbues Pachinko machines with free will).
The biggest puzzle to me is why some people think they can even come up so easily with a “theory” for “free will”, given that not only free will can’t exist from physical laws (everything is either causal, random, or some mix of the two), but basic introspection into how my mind works also tells me that free will can’t even be an illusion (everything that is being noticed in my consciousness has to arise and eventually vanish, by definition, and that includes thoughts and decisions).
This even extends to the concept of “choice”. When it comes down to it, my brain can’t make any “real” choice, a brain just does what it’s supposed to be doing like every other lump of matter in the universe, no system is really isolated, all systems are part of the totality of the universe, which just evolves the way it’s supposed to evolve given its initial conditions.
“Choice” only exists as an emerging concept, appearing in our consciousness, like all the other concepts our subconscious builds as perceptual patterns emerge in the neural connections of our brains. And our brains appeared out of the dynamic of having a population of self-replicating organisms subjected to natural selection (itself a consequence of the universe being able to increase complexity in specific conditions).
Brains encode correlations and statistical patterns that apply to the *whole population* (because it relies on reproduction and survival at the species level), not individual instances, which is why “Choice” at the individual level is a misguided concept (although brains are also able to learn extra correlations to some extent, based on personal experience, to try and improve an individual chance of survival, but this also works at the average individual level, improving the chances of the whole species) , it’s really all just the result of the species and its environment having evolved in parallel – the universe isn’t actually made of independent subsystems/agents, fundamentally it’s one big wave function evolving according to Schrodinger, and sometimes localized “knots” of ever growing complexity appear (like vortices in water), with a structure that is self-similar to the rest of the entire system (brains).
As an analogy, our brain comes up with the concept of a river flowing on solid ground, and it can’t help but think that the river has carved its way through the landscape, as the source of causality, when in fact the river and the terrain can’t be dissociated, it’s both the river that carved the riverbed into the landscape and the landscape that guided the river to flow this way or that way. For nature the two things are two sides of the same coin and can’t be separated. Literally everything is like that, because when we “bin” a pattern into a concept, we think it can therefore exist in isolation of the rest, but it can’t. “Things” only exist relative to something else – thinking otherwise is like trying to cut a magnet in two in order to separate its north pole from its south pole.