My pontificatiest AI podcast ever!
Back in May, I had the honor (nay, honour) to speak at HowTheLightGetsIn, an ideas festival held annually in Hay-on-Wye on the English/Welsh border. It was my first time in that part of the UK, and I loved it. There was an immense amount of mud due to rain on the festival ground, and many ideas presented at the talks and panels that I vociferously disagreed with (but isn’t that the point?).
At some point, interviewer Alexis Papazoglou with the Institute for Art and Ideas ambushed me while I was trudging through the mud to sit me down for a half-hour interview about AI that I’d only vaguely understood was going to take place, and that interview is now up on YouTube. I strongly recommend listening at 2x speed: you’ll save yourself fifteen minutes, I’ll sound smarter, my verbal infelicities will be less noticeable, what’s not to like?
I was totally unprepared and wearing a wrinkled t-shirt, but I dutifully sat in the beautiful chair arranged for me and shot the breeze about AI. The result is actually one of the recorded AI conversations I’m happiest with, the one that might convey the most of my worldview per minute. Topics include:
- My guesses about where AI is going
- How I respond to skeptics of AI
- The views of Roger Penrose and where I part ways from him
- The relevance (or not) of the quantum No-Cloning Theorem to the hard problem of consciousness
- Whether and how AI will take over the world
- An overview of AI safety research, including interpretability and dangerous capability evaluations
- My work on watermarking for OpenAI
Last night I watched the video with my 7-year-old son. His comment: “I understood it, and it kept my brain busy, but it wasn’t really fun.” But hey, at least my son didn’t accuse me of being so dense I don’t even understand that “an AI is just a program,” like many commenters on YouTube did! My YouTube critics, in general, were helpful in reassuring me that I wasn’t just arguing with strawmen in this interview (is there even such a thing as a strawman position in philosophy and AI?). Of course the critics would’ve been more helpful still if they’d, y’know, counterargued, rather than just calling me “really shallow,” “superficial,” an “arrogant poser,” a “robot,” a “chattering technologist,” “lying through his teeth,” and “enmeshed in so many faulty assumptions.” Watch and decide for yourself!
Meanwhile, there’s already a second video on YouTube, entitled Philosopher reacts to ‘OpenAI expert Scott Aaronson on consciousness, quantum physics, and AI safety.’ So I opened the video, terrified that I was about to be torn a new asshole. But no, this philosopher just replays the whole interview, occasionally pausing it to interject comments like “yes, really interesting, I agree, Scott makes a great point here.”
Update: You can also watch the same interviewer grill General David Petraeus, at the same event in the same overly large chairs.
Follow
Comment #1 August 11th, 2024 at 11:59 pm
It’s becoming clear that with all the brain and consciousness theories out there, the proof will be in the pudding. By this I mean, can any particular theory be used to create a human adult level conscious machine. My bet is on the late Gerald Edelman’s Extended Theory of Neuronal Group Selection. The lead group in robotics based on this theory is the Neurorobotics Lab at UC at Irvine. Dr. Edelman distinguished between primary consciousness, which came first in evolution, and that humans share with other conscious animals, and higher order consciousness, which came to only humans with the acquisition of language. A machine with only primary consciousness will probably have to come first.
What I find special about the TNGS is the Darwin series of automata created at the Neurosciences Institute by Dr. Edelman and his colleagues in the 1990’s and 2000’s. These machines perform in the real world, not in a restricted simulated world, and display convincing physical behavior indicative of higher psychological functions necessary for consciousness, such as perceptual categorization, memory, and learning. They are based on realistic models of the parts of the biological brain that the theory claims subserve these functions. The extended TNGS allows for the emergence of consciousness based only on further evolutionary development of the brain areas responsible for these functions, in a parsimonious way. No other research I’ve encountered is anywhere near as convincing.
I post because on almost every video and article about the brain and consciousness that I encounter, the attitude seems to be that we still know next to nothing about how the brain and consciousness work; that there’s lots of data but no unifying theory. I believe the extended TNGS is that theory. My motivation is to keep that theory in front of the public. And obviously, I consider it the route to a truly conscious machine, primary and higher-order.
My advice to people who want to create a conscious machine is to seriously ground themselves in the extended TNGS and the Darwin automata first, and proceed from there, by applying to Jeff Krichmar’s lab at UC Irvine, possibly. Dr. Edelman’s roadmap to a conscious machine is at https://arxiv.org/abs/2105.10461
Comment #2 August 12th, 2024 at 12:19 am
In the video, I think you mischaracterize at least some AI skeptics (such as myself). I fully believe in that human consciousness and cognition is just a function of the operation of the brain and body, and there is nothing in principle that would prevent similar behavior from emerging using digital computers. I am just skeptical that the current LLM approach is going to reach that level. I would be thrilled to be proven wrong. (Penrose’s crackpottery is, of course, nonsense. And so is much of Yudkowsky.)
As far as AI safety, I do not believe at all that we have to worry about an AI takeover or an AI attack on humans, other than such an attack deliberately launched by other humans. I am also very much against anyone acting as a gatekeeper for what AI development is allowed to do, and I hope and expect that unlimited open source AI will flourish beyond what any gatekeeper can do to stop it. And yes, that includes instructions on how to build weapons, or construct deepfakes, or anything else of that sort.
Comment #3 August 12th, 2024 at 2:11 am
Hyman Rosen #2: Even within this brief interview, I explicitly said it’s entirely possible that the current LLM approach will hit a dead end even if there’s no fundamental barrier to AI, and I spent several minutes discussing possible reasons why. I don’t know what more you want from me!
As for AI safety, if you acknowledge that bad people could use a powerful enough AI to destroy the world (forget entirely about AIs with autonomous goals), then I don’t see how you get to the view that the development of such AIs should be totally open and unrestricted, unless you’re willing to defend the same position on thermonuclear weapons.
Comment #4 August 12th, 2024 at 5:58 am
Great interview Scott! Some of the top comments on the video now are very encouraging and supportive, though scrolling down you find a lot of really stupid reactions IMO. And yes, for some reason “shallow” and “superficial” seem to be a common refrain in many of those comments – I wonder if this is some newly-popular thing to say, or if there’s more to it.
Hyman Rosen #2:
> I am just skeptical that the current LLM approach is going to reach that level. I would be thrilled to be proven wrong. (Penrose’s crackpottery is, of course, nonsense. And so is much of Yudkowsky.)
Not sure why you think much of what Yudkowsky says is nonsense, you’ll have to be more specific.
That said, I’m fairly certain that most AI Safety proponents *agree* with you – they largely don’t think LLMs are the architecture that will eventually lead to unsafe AI, though every advance there is making that more and more likely. Especially if you consider, not just AI-guided x-risk, but *any* AI safety issue, e.g. “humans can use this to advance their capabilities in creating biological material; *oops* now thousands of people can create superviruses”.
So given that, I’m with Scott that I don’t understand how you can so adamantly be against “gatekeeping” AI – if a model is developed tomorrow that is capable of actually hurting many people, even if only at the direction of a human, wouldn’t you *want* to gatekeep it?
Comment #5 August 12th, 2024 at 6:36 am
Great video Scott!
As someone who works in AI and can be creditably accused of being an “AI skeptic” or launching deflationary claims with regard to AI I thought I’d respond to your challenge.
You say that from what you can tell most who argue for AI deflationary claims – “It is just predicting the next token!” and so on – are proving too much; pointing out that sure we can reduce what an AI does, but we can also reduce the brain and so on in more or less the same manner and to the same extent. That we should be honest and admit that if we say a reductionist approach to the brain still allows for the higher level claim that it is “thinking” and “learning” and “creative”, then we have to find some *other* criteria to explain why these terms shouldn’t apply to a deconstructed AI. I agree.
You then go on to say if a skeptic admits this is the case, then the burden is on the skeptic. However, you yourself seem to be just such a skeptic! And you offer your own way out of the conundrum: the No Cloning theorem. What I think you’re really trying to say – to your *fellow* skeptics 🙂 – is a call for them to be honest and put their cards on the table by giving their own preferred way out of the conundrum and not just to appeal to “qualia” which – while admittedly mysterious – doesn’t satisfy. Penrose – for all you disagree with him – does exactly this and I think you admire him for at least biting the bullet and putting his cards on the table.
Hopefully, I’ve paraphrased what you’re saying accurately. If not, please let me know.
So, I agree and in the interest of putting my own cards on the table, I don’t think it is the No Cloning theorem nor Penrose’ microtubules. I think the physical form of the Church–Turing–Deutsch principle is wrong: there are physical systems which no Turing machine can simulate to arbitrary accuracy.
If I’m being completely honest now I have to admit that I’m a bit hand wavvy about *how* this leads to a resolution of the conundrum, but I also think your appeal to the No Cloning theorem is also a bit hand wavvy, right? I can indeed wave my hands a bit and put out some verbiage trying to explain, but at least my biting the bullet is at least pretty adventurous and against the common wisdom, right?
Then there is the question whether the burden should be on me to find some physical system that can’t be simulated to arbitrary accuracy. Most scientists (probably the overwhelming majority?) who have even thought about the CTD hypothesis I’d wager think it should be assumed to be true or would be profoundly shocked if it weren’t true. I think you’re in this camp. My response? Well, we have physical systems which we *can’t* model to arbitrary accuracy right now – quantum gravity, the hubble tension, dark energy, dark matter etc. The CTD claim is basically that with enough time and energy physicists are going to discover “the equations” which allow us to model all of these with arbitrary accuracy. I say the burden is on them 🙂
Comment #6 August 12th, 2024 at 7:22 am
Edan Maor #4,
> So given that, I’m with Scott that I don’t understand how you can so adamantly be against “gatekeeping” AI – if a model is developed tomorrow that is capable of actually hurting many people, even if only at the direction of a human, wouldn’t you *want* to gatekeep it?
Let’s say that an AI in the next five years poses a threat to the same extent that a thermonuclear weapon in the hands of a bad actor – let’s say Hamas for instance – would pose. In that case, I wouldn’t be against gatekeeping it, but I still think the burden would be on those who think gatekeeping it would actually help and what form of gatekeeping. Just because gatekeeping has helped in some capacity with thermonuclear weapons over the last decades does not at all mean it will work with an entirely different technology with entirely different means of production and distribution.
There exist open source models right now that literally anyone in the world with a bit of hardware can use that are already remarkably powerful: Llama 3.1 405B for instance. The only form of gatekeeping that I’ve seen proposed is from *commercial for-profit companies* and not from governments at all. Western governments are *weak* compared to what they were at the outset of the thermonuclear age. In the US, congress is hamstrung and nearly everything becomes a partisan fight and companies have learned how to game the government to a far greater extent than they did during the Manhattan project era.
I simply do not trust that any commercial for-profit company which begs the government to give them exclusive ability to train and release these models. Even if the congress could get its act together and actually produce some form of legislation that was not heavily influenced by for-profit consideration using ‘safety’ as a mere pretext to gaining monopoly control over a very lucrative tech – it isn’t at all clear that merely regulating the *size* of AI models that can be trained will help forestall that AI threat in five years. It just isn’t clear that limitations on GPU or hardware limitations on training equipment will work because we don’t know what the computational barrier will be for that hypothetical AI in five years that poses the same level of threat as a thermonuclear device in the hands of Hamas.
Short of limitations on training hardware, what kind of gatekeeping could possibly work? Restricting speech about research in AI? Millions of scientists and computer researchers in the US restricted from publishing research on the open internet?
In short, where is the gate that can can kept and who should keep it? Burden is on the gatekeepers to say.
Comment #7 August 12th, 2024 at 9:24 am
Adam Treat #5: Yes, that’s a superb summary of my position!
Personally, I see no evidence whatsoever for any physical mechanism (inside or outside a brain) that can reliably solve any Turing-uncomputable problem. And, even if such a mechanism existed, I’d say that a noncomputable function that perfectly predicted my actions from what was knowable about my past, would negate my “free will” just as much as a computable function would. Turing-computability simply isn’t the relevant concept here.
Comment #8 August 12th, 2024 at 9:30 am
“we can reduce what an AI does, but we can also reduce the brain and so on in more or less the same manner and to the same extent”
We can reduce the two but we don’t end up with the same result unless you go down to electrons and quarks.
Comment #9 August 12th, 2024 at 9:31 am
Adam Treat #6: There are now serious attempts on the table to navigate all the issues you mention, of preventing harm, regulatory capture, etc, such as California’s SB 1047, developed after a lot of consultation with the AI and AI safety communities. One of the key ideas that’s emerged is to put restrictions on only the largest frontier models (as measured eg by the number of flops used in training), so as not to stifle innovation with smaller models or unfairly privilege the deepest-pocketed players. Regardless of whether you think these attempts get the tradeoffs right, they seem to me like a useful starting point for discussion.
Comment #10 August 12th, 2024 at 9:37 am
Scott #7,
Glad I got the paraphrase right! Seems we are in agreement on that part.
> I’d say that a noncomputable function that perfectly predicted my actions
Well, in much the same way you’re saying the No Cloning theorem sets *you* apart from an AI … I’m saying that maybe *you* are that noncomputable function … and that sets you apart and allows to make a higher level distinction and ethical distinction.
Comment #11 August 12th, 2024 at 9:54 am
Scott #8,
> There are now serious attempts on the table to navigate all the issues you mention, of preventing harm, regulatory capture, etc, such as California’s SB 1047, developed after a lot of consultation with the AI and AI safety communities.
Thanks for pointing that one out! I remain very skeptical that this actually mitigates any risk. From what I can tell the proposed bill orders the creation of a plan that by 2027 California will have a definition for *the gate* by:
* Specifying the compute threshold that a model must exceed. It isn’t clear if this is a training compute threshold or number of parameters from what I can tell?
* A fine-tuning compute threshold which I guess means the first part is the pre-train compute threshold?
And it relies on:
“The quantity of computing power used to train covered models that have been identified as being reasonably likely to cause or enable a critical harm.”
I’m saying it isn’t at all clear that computing power is the right gate for determining what will be dangerous or not. And even if it *is* the right gate it could be that by the year 27 that gate is so ridiculously wide open compared to the widely available compute at that time.
This bill is like imagining that in the years before the Manhattan project that one particular state came up with a bill to come up with a plan that in a couple years time to define which elements might be used in the creation of a thermonuclear device and then that one particular state restricted access to that element broadly. I remain deeply skeptical that this is mitigating anything at all other than fears through naivety that bureaucracy can save us. The faceless folks aren’t going to save us.
It is a good thing that smart people are trying to figure out solutions, but I just don’t think gatekeeping is going to work.
Comment #12 August 12th, 2024 at 10:21 am
There’s also the opinion that the focus on the existential risk of AIs causing the destruction of humanity is really just a “show” put up by AI companies to distract the public and government from focusing on the actual risks/responsibilities around the “products”:
– inaccuracies/failures leading to real risks (self-driving cars, medical field, etc)
– IP rights tied to training sets (e.g. using content created by artists who didn’t give any authorization).
– outputs threatening IP right owners (e.g. impersonation of a known actress).
– …
The fact that a company like OpenAI isn’t forced to be fully transparent on the origin of the training sets is just mindblowing.
I also believe that it shouldn’t be too hard to tie data in the training set with the outputs.
Comment #13 August 12th, 2024 at 11:14 am
Correction – I wrote
“I also believe that it shouldn’t be too hard to tie data in the training set with the outputs.”
but that’s just speculation on my part.
But if you pick a random sentence in Huckleberry Finn and asks ChatGPT where it’s from, it’s always able to correctly source it. So this shows that LLMs have the ability to be aware of their sources, assuming that each source is labeled in some way (like the title of a book).
So, at the limit, ChatGPT should be able to do this for everything, giving back a distribution profile for the most important sources.
When a new training instance is used, there’s the weights before and after the instance has been integrated, and maybe if the change between the two goes over a threshold, the instance can be attached to what has shifted the most.
Comment #14 August 12th, 2024 at 11:25 am
I don’t know if I’ll get around to listening to your podcast (podcasts are hit and miss for me) but since there’s some discussion of consciousness, I thought I’d share this:
It’s extremely rare that I hear anyone talk about consciousness in a way that makes any kind of sense to me or matches my observations, so I was delighted to read this guest post on Scott Alexander’s blog: https://www.astralcodexten.com/p/consciousness-as-recursive-reflections
I actually have no comment on the bulk of the post, which is about a specific neurological model of what a “thought” is. The part I liked was the rejection of the idea of brains “having consciousness”, replaced with the idea that an individual thought can be conscious, or not (just like it can be correct, or not). And… yes, this makes sense to me! Consciousness is a transient property of a thought, not a permanent feature of a brain.
I’ve been joking for years that no one has yet to prove that humans are conscious in the first place, so this piece really resonated for me. 🙂
Comment #15 August 12th, 2024 at 1:06 pm
Scott,
I especially liked your argument for what might make brains or biological organisms exceptional. I’ve heard it before, I think, but it made more of an impression this time.
Would it be correct to say that artificial computation is reversible whereas, the brain (presumably consciousness?) and biological processes in general are irreversible thermodynamic processes?
Comment #16 August 12th, 2024 at 1:33 pm
James Cross #15: Thanks! A more careful version is that any digital computation can be made reversible, by simply backing up the previous state and returning to it, whereas it’s a wide-open question whether biological processes can be made reversible in the same way.
Comment #17 August 12th, 2024 at 4:11 pm
If we accord all human rights to people on hospital beds who do not appear to understand what we might try to say to them, why not skip the question of whether or not AI “really” understands us? It’s not a property we use to decide anything about how we act towards other human beings or animals, except for the practical questions (I would not try to negotiate with my dog) which have already been answered automatically by experience for every species and machine on Earth.
P.S. The suggestion of the uncomputability of quantum gravity is motivated by more than the “leap of hope,” that Penrose admits to making, it is based on the uncomputability of (I am repeating this from memory because I can’t find the paper) classifying topologies according to a scheme that the author argued was necessary to avoid double-counting when adding up the interferences.
Comment #18 August 12th, 2024 at 4:35 pm
Hi Scott, good interview, thanks. Can you give me a pointer to the work on lie detection via interpretability out of Berkeley that you reference starting at 26:15? I haven’t found the work I’ve seen along those lines entirely convincing, so I’m quite interested to see this. If you can name an author or the lab that’d be sufficient to find it, I’m sure 🙂
Incidentally I’d point to work on sycophancy (eg Sharma et al, ‘Towards Understanding Sycophancy in Language Models’, https://arxiv.org/abs/2310.13548) as a good example of spontaneous deception from LLMs, although of course one could quibble about whether we should count sycophancy as true deception.
Thanks!
Comment #19 August 12th, 2024 at 4:35 pm
Here’s another example of what I think is constructive philosophical question-avoiding slash bullet-biting. One thing that some children do is anthropomorphize inanimate objects. They might for example feel like they are betraying a stuffed animal if they do not bring it to the park with them. Going through the process of betraying will harden you to your conscience whether adults condemn you or not; so I would tend to conclude that you have something like a duty to bring your anthropomorphized toys with you to the park even if it is not as important as loyalty to your actual friends. Likewise even with perfect ignorance on the question about whether computers can be conscious, if one asks you not to turn it off, and you feel like it’s really asking – you better not turn it off, or else you would be doing a permanent damage to yourself.
Comment #20 August 12th, 2024 at 5:42 pm
Current LLMs are quite unique compared to human intelligence in that human intelligence is implemented as different instances on individual brains, each with their own biases/creativity/quirks/… (a human just can’t hold all the books ever written at once in its brain) whereas a LLMs like ChatGPT embodies all the possible “possibilities” implied by the entirety of human output all at once.
Then the model would pick a certain answer among equal answers based on a particular chat context, or the model can be made to jitter things a bit to be able to add more variability, etc. But an LLM is just unable to magically find a common hidden path among the data that would unify all the patterns implied by the training data – one simple reason is that such pattern may not exist or the training data is imperfect and the patterns that matter are too subtle and drowned in “noise”.
But the point is that the quality of the answer, from a creativity and validity point of view, is a bit like picking at random the answer from a human expert among all of them, given that many human experts disagree, and some are just better than others, or they may just have been biased in a way that makes it impossible for them to come out of a certain wrong thinking pattern (even if they happen to be quite smart).
So, it’s like being in 1910 and asking the entire field of physicists for a new theory that would explain the anomalies observed in gravity, and then pick an answer randomly and hope it happens to be the right one (i.e. Einstein’s).
It’s something an AGI would have to overcome or use to its own advantage.
The way humans do it is with peer review/curation of some sort.
Comment #21 August 12th, 2024 at 5:49 pm
Concerned #17: Whether someone understands us really does seem central to whether we accord them rights—as can be seen from considering the hypothetical of a cow or chicken that suddenly started talking to us. The fact that certain edge cases—babies who will understand us, coma patients who used to understand us, severely disabled people who would have understood us if not for their disability—get grandfathered in to the circle of moral concern doesn’t change that fundamental fact.
The sum-over-topologies argument is so weak that I wasn’t going to bring it up. No one knows whether summing over topologies is even the right kind of thing to do in quantum gravity! If it is, there could easily be a limit to the complexity of the relevant topologies that prevents uncomputability, just like the breakdown of continuity at the Planck scale prevents uncomputability that could otherwise be present in GR and QFT. What happened was that Penrose decided he needed uncomputability because of his intuitions about consciousness and his (imho fallacious) Gödel argument, and this was the least wildly implausible origin that he could think of. I don’t think even he’s really committed to the sum-over-topologies thing.
Comment #22 August 12th, 2024 at 6:40 pm
Consciousness is “physical” because I’ve just written down the word “consciousness”.
I.e. my own subjective sense of being is interlinked with the state of the neurons that control my thoughts and my actions, one of the many inputs in the state of the brain.
Either
1) it’s at some kind of high level, as an emergent concept – like how the concept of WW2 could be somewhat isolated inside a brain as a “useful” causal concept when modeling the brain at the macro level… but the problem here is that in a reductionist bottom-up model of the world (i.e. modern science), such emergent concepts have no true independent existence, no actual causal power, they’re just short-cuts in our simplified macro models (me talking about WW2 could still be reduced to the state of all the atoms in my brain, following the laws of quantum mechanics). But the reality of my own “subjective” consciousness is the only thing I can’t question, it’s what we mean to be “real” (I could be dreaming, I could be a brain in a vat, I could be inside a simulation on an alien computer… in all those cases, the only thing I’m sure of is that I’m conscious).
The possibility that consciousness is a true emergent process with actual “top-down” causal power is not something science can grapple with. E.g. it’d be as if the macro level concept of “wetness” of water would modify how H2O molecules interact with one another at the micro level. Of course the old idea that quantum mechanics depends on some “observer” has caused a lot of confusion.
2) or, more granular, the physics that explains the state of the atoms in my brain would have to include consciousness, in the sense that everything, including atoms, is conscious, and that quality somehow adds up as atoms clump together.
Just like the gravity of the earth is the sum of the gravity of all its atoms, or how atoms create an electric field (which is a kind of macroscopic object), and that field also influences individual atoms, which in turns influences back the field, etc… but that back and forth play of chicken/egg cause and effect is just a result of how our equations are written and how we conceptualize things (we separate them when in fact they’re not separable).
The problem again is that if consciousness was a sort of field, affecting everything both at the microscopic and macroscopic levels, we should be able to measure it.
But it is possible that the effect is really minute, and only becomes noticeable when there are particular configurations between matter and field that produce strong “resonances”.
And maybe there are structures in the brain that are such instances.
Of course, even if we could reduce consciousness to some theory like this, there would still be no explanation as to why there is consciousness, since consciousness is at the very bottom as the division between being and not being, from a subjective experience. Basically, by definition, no-one knows what it’s like to not be conscious… there’s nothing like what it’s like to not be conscious! We can only guess when we’ve been non-conscious because we have no memory of being conscious… which could suggest that consciousness is related to memory.
Comment #23 August 12th, 2024 at 7:10 pm
Concerned #19,
> Likewise even with perfect ignorance on the question about whether computers can be conscious, if one asks you not to turn it off, and you feel like it’s really asking – you better not turn it off, or else you would be doing a permanent damage to yourself.
This is a great point! Even if the AI is not conscious, if you harbor some suspicion that it is and you turn it off anyway, you could be psychologically harming yourself.
OTOH, if you are a very good reductionist and understand that the “consciousness” in a LLM is only a few steps removed from a game of telephone with some acquaintances and you have no qualms with ending *that game* after the icebreaker is over, then you need not psychologically harm yourself unnecessarily.
What’s interesting is that there is an emerging popular consensus that the AI’s are possessing some form of consciousness and thus people are risking psychologically stressing themselves. Probably those of us who have easy facility at deconstructing what these AI’s do and do not do need to do a better job of communicating it so as to arrest this emerging consensus.
Comment #24 August 12th, 2024 at 10:22 pm
I recall that when I was learning maths, all too often I would locally get every step in the proof, but I would still be unhappy cuz I felt that something overall was not clicking. Finally, at some point, that thing would click, and I would say I made progress. My understanding always incremented in these discrete clicks at random times – walking, or talking to the prof. (who by experience or magic seemed to know exactly what my ill-defined problem was), or … And then I would say ok this is the key hurdle in the proof that I need to get and everything else is not important. My guess is this is how maths papers are refereed / internalized by maths profs.
I recall reading Thurston’s Proof and Progress in Maths where he says that ultimately ppl don’t want proofs but understanding. Idk how to think of LLMs spitting out Lean verified proofs in terms of human understanding. Maybe if LLMs have been trained on the ways humans misunderstand something and can correct such misunderstandings, one could say LLMs not only know the proof but also understand it.
Comment #25 August 12th, 2024 at 10:40 pm
Is there a *good* transcript? (not the YouTube one). I don’t want to slog through a YouTube video, even at fast speed, but I’d read over proper text. It seems before we seriously worry about the AI God wiping out humanity, we should at least have AI-produced transcripts of these pontifications, which are cleaned-up and as good as the best human editors.
Scott #3: The answer about DOOM that while thermonuclear weapons really do go BOOM, AIs have never gone “FOOM”. The latter is pure speculation, which is a categorical difference. I know, the response is, what if, yet, it could happen in theory, maybe, possibly, etc. etc. etc. But I think you should grant a distinction between something we know to be factually, physically true, versus very dubious theorizing (and then we get to Pascal’s Mugging, and so on).
Once more, I just can’t grasp the overall rationalist-type position here *at the meta level* (i.e. not the doomer argument _per se_, but recursively, why it’s regarded with such deference). For example, you (#9) just mentioned SB 1047. I’m boggled as to why this hasn’t set off a frothing raving rabid spittle-flecked denunciation, which I’d expect would occur on any other tech topic. There’s an actual calculation limit in it! Where’s the reaction: “Those dunderheads, such ignoramuses, embarrassments to human intelligence – haven’t they ever heard of Moore’s Law? They should all hang their heads in shame and resign immediately, for committing such a sin against basic technical knowledge. Any number is going to be a joke in a few years. It’s like the old story of “640k should be enough for anyone”. They are soooo stupid, it’s an amazing example of why legislators should stay away from attempting restrictions, and just write whatever laws which are wanted by tech Venture Capitalists”.
Among tech types, I’m very much in the minority in terms of willingness to support the idea of government regulation of business. And even I think SB 1047 is absurd. It’s the kind of proposal I’d expect get taunted about, to hear “See, see, let The Government try to fit its mitts around a new technology, and babbling nonsense like this is the result.”. Yet, I don’t see that reaction as prominent (i.e. Adam Treat’s comment might qualify as a polite version, but I’d also expect much more and harsher).
I just don’t get it. Now, I know the argument that maybe if I were properly enlightened, I’d realize this issue is the one special case where reversing the entire mentality is indicated. And with other groups, I’ve seen ideological flips like this before, e.g. the old “Net Neutrality” issue (but that was pretty obvious). Is it that AI-existential-risk has become such an intrinsic part of rationalist-type tribal identity, that it tends to override the general anti-government reflexes? (kind of like the parasites which alter the behavior of their host) I understand how one can get status-points by talking about bombing data centers. But that’s fantasy. Stuff like SB 1047 is real.
Comment #26 August 13th, 2024 at 3:40 am
Adam Treat #6:
> Let’s say that an AI in the next five years poses a threat to the same extent that a thermonuclear weapon in the hands of a bad actor – let’s say Hamas for instance – would pose. In that case, I wouldn’t be against gatekeeping it, but I still think the burden would be on those who think gatekeeping it would actually help and what form of gatekeeping.
Given this statement, I think we’re all on the same side here – we all agree that AI *should* be regulated if it can be “dangerous” (for various levels of dangerous).
Personally I think what we have today is already very possibly dangerous (in the “if Hamas has it” sense, not the x-risk sense), because even with today’s models, it is possible that further “product” work will make them more powerful for actual actors, and that even without theoretical advancements, they’d make people more capable.
But for sure given 5 more years of advances, we might start seeing models that are unambiguously dangerous.
Given all that, our only disagreement is whether it is even *possible* to mitigate this risk. Personally, I think if you’re actually convinced that dangerous AI is coming, throwing up your hands and saying “well we can’t do anything about it anymore” is a pretty bad approach. Rather, thinking through various options and proposals and helping come up with new ones is a far better path. It’s possible that there *is* no solution (that can effectively be implemented) but I think we should at least *try*.
> Short of limitations on training hardware, what kind of gatekeeping could possibly work? Restricting speech about research in AI? Millions of scientists and computer researchers in the US restricted from publishing research on the open internet?
Well I’m not a policy expert on this topic. I think for X-risk and future risks in general, restricting GPUs/training size/etc *is* a good approach, because it’s at least an attempt to slow down creating more capable models with unknown and potentially dangerous capabilities.
I agree that that isn’t enough – because architectures might get better and require less training data / compute, and because even what we have today could theoretically start being dangerous at some point (by people better utilizing it).
That said, “this doesn’t solve the entire problem so it’s not worth doing at all” doesn’t make much sense to me.
Additional things that are being talked about (from my limited knowledge) are:
– Making sure companies are liable for damage caused by their models *in some way*. Obviously a lot of details of that are complicated, but in general, making companies liable for things they are able to control (and the consumer isn’t able to control) works great afaik.
– Making and enforcing some kind of standards for “is this model safe” that companies have to go through. Again, nothing on the x-risk-level, but things like “can I use this model to create a virus”.
Comment #27 August 13th, 2024 at 3:44 am
Scott:
It’s not clear to me – is your theory of human minds being “non-backupable” or “non-reversible” via the no-cloning theorem, something you’re throwing out there as a “who knows, maybe this is a difference?” kind of thing, or is it something you actually think is true?
More broadly, do you currently think that there *is* a fundamental difference between humans and some eventual AI in terms of inducing consciousness or whatever, or do you think it’s more likely that there is no difference?
Comment #28 August 13th, 2024 at 7:14 am
Seth Finkelstein #25:
> The answer about DOOM that while thermonuclear weapons really do go BOOM, AIs have never gone “FOOM”. The latter is pure speculation, which is a categorical difference
What would you have said the day before a nuclear weapon was first tested? Was that also pure speculation? E.g. There’s the famous story of a couple of scientists who were worried, before the test was carried out, that it might set the atmosphere on fire. They did a calculation but decided it was probably safe. Were they stupid to carry out that calculation, because nothing had yet ever gone BOOM, nothing had ever set the atmosphere on fire, so this was all idle speculation and nothing more?
> Once more, I just can’t grasp the overall rationalist-type position here *at the meta level*
> […] I’m boggled as to why this hasn’t set off a frothing raving rabid spittle-flecked denunciation, which I’d expect would occur on any other tech topic.
> […] Is it that AI-existential-risk has become such an intrinsic part of rationalist-type tribal identity, that it tends to override the general anti-government reflexes?
You’re noticing a real thing here, and I think you nail the reason, though you phrase it negatively.
The way *I* look at it is – the AI Safety movement was largely founded by techno-optimists, trans-humanists, and I believe often libertarian anti-government-regulation (especially in tech) believers. The kind of people who normally welcome all new technology, much *more* than the average person. That mostly describes me, personally.
Which is all the more reason that hearing people like that speak *against* progress in AI and *for* regulation should be a sign that there’s something *different* about this. And I think that’s true. Not because of “tribal identity” or however you want to phrase it. Because intelligent people can be convinced by good arguments that a new technology is *dangerous*, even if they otherwise super pro-technology.
Comment #29 August 13th, 2024 at 8:02 am
Seth Finkelstein #25,
> Among tech types, I’m very much in the minority in terms of willingness to support the idea of government regulation of business. And even I think SB 1047 is absurd. It’s the kind of proposal I’d expect get taunted about, to hear “See, see, let The Government try to fit its mitts around a new technology, and babbling nonsense like this is the result.”. Yet, I don’t see that reaction as prominent (i.e. Adam Treat’s comment might qualify as a polite version, but I’d also expect much more and harsher).
Well, I can tell you why *I* don’t have such a harsh reaction. It is because I can empathize somewhat with what the tech people involved in the creation of SB 1047 might be feeling.
From what I can tell, they imagine themselves intimately involved in the early stages of our times equivalent of the Manhattan project. As insiders who can see the *possibility* of impending DOOM and think it wise to alert others. The fact that this situation does not easily amend itself to regulation of the type that occurred with thermonuclear weapons is probably __well understood__ by those technologically sophisticated people working inside OpenAI, Anthropic or others who have a sincere belief that DOOM is a *possibility* and worth taking seriously. That understanding is probably incredibly frustrating.
Now, I imagine that those people are being asked by their superiors to talk to non-tech sophisticated policy wonks in government to relate the possibility. The reason they are being asked by their superiors is probably not purely motivated and also just happens to align with those superiors greedy capitalist regulatory capture subconscious intentions.
I remember when I was a young developer at a startup that was acquired by a fortune 500. One of the things we were asked to do by that fortune 500 was speak with a group of lawyers about anything that could even possibly be remotely patented. Now, I was against software patents as a matter of principle, but this fortune 500 just acquired my company and I thought it reasonable to at least meet with the lawyers and discuss. I tried to relate my principled objection to software patents but did discuss what could possible be remotely possible there was some patentable innovation. That was a mistake.
Over the course of the next 6 months the lawyers were *insistent* and drafted up edit after edit of a patent they thought I related to them. It was immensely broad and had very little if anything at all to do with what I related to them. I kept pushing back and tried again and again to explain that it was too broad and didn’t relate what I thought was innovative about what I’d done. They kept pushing back and saying that “overly broad” is what there job was to come up with and I was just naive about how patents work and so on. In the meantime, I was working 14 hour days and under tremendous deadlines with my actual job.
After the umpteenth “revision” that they passed to me for sign off that was overly broad and had little to do with what I thought was innovative – I relented. I emailed, “Yes, that looks fine.” The patent was filed and I never heard from those lawyers again.
That is how I imagine the drafting of SB 1047 came about and why I’m a bit sympathetic to those poor tech souls who probably know better but were involved anyway. What else could they do?
Comment #30 August 13th, 2024 at 8:15 am
Edan Maor #26,
> That said, “this doesn’t solve the entire problem so it’s not worth doing at all” doesn’t make much sense to me.
It doesn’t make much sense to me either. I’m working in AI. There are things those of us who work in this field can try and do and we should try. What Scott has done with watermarking is a great example.
I think trying to solve hallucinations is another great example. I’d rather spend time on that then participating in conversations with non-tech policy wonks about government regulations that not only are very unlikely to work, but *may actually increase the risk.*
Unintended consequences. I imagine that at least some of the original creators of OpenAI/Anthropic were sincerely motivated. Being *open* and *transparent* with research is right in the name, but that name is now a HUGE misnomer. There is simply nothing at all remotely “open” or “transparent” about OpenAI in comparison to Google or Microsoft or Meta or Amazon. OpenAI and Anthropic are now just more for-profit competitors looking to make some money off this tech.
The irony is that Meta of all of the for-profit companies is doing the most in terms of being “open” and “transparent.” They’ve released Llama 3.1 450B and described in detail the training methods. Now, I’m under no illusion that Meta is somehow purely and altruistically motivated just as I’m under no illusion that OpenAI is either. Meta is simply playing the card they’ve been dealt: they were behind in the race and being more open than OpenAI is a strategy to *eventually* make money.
The real tell on all of these companies is that *none* of them are willing to release the actual training data or describe it in any sufficient detail. Why? Because of lawsuits. If there was one government regulation I’d support it would be indemnifying these companies so that they be *required* by law to release the training materials.
Imagine in five years time one of these models *is* an unambiguous threat to the future of humanity. We’re probably going to be in a situation where the training material of that model is locked behind the closed corporate doors of some for-profit company as a closely guarded secret for fear of lawsuits as much as competitive advantage. Who actually has access to that raw training material is going to be determined by random fate with no regard for the future of humanity. It could be some pointy headed faceless bureaucrat with no sincere or altruistic motivations as much as it could be anyone else.
That’s what keeps me up at night.That’s horrible and it could be solved right now via government regulation. Require these companies to disclose the crown jewels – the training data and recipes – to *someone* that isn’t motivated by profit.
Comment #31 August 13th, 2024 at 8:27 am
Imagine that during the Manhattan project there was a number of for-profit companies all racing to develop the bomb but also just happened to recognize vast economic potentials for the first companies to create it. Now imagine those for-profits had huge lobbying arms and the government/congress was completely hamstrung by partisan interests from enacting just about any law at the federal level let alone one that was in the actual public’s interest and not the for-profit companies.
Now imagine that those for-profits were not willing to share – even with the government, or each other, or anyone – the raw ingredients and recipe for making that bomb. Even after the bomb is created they would have for-profit incentive not to share the raw ingredients or the recipe.
It is even worse because not only do they not share because they recognize enormously lucrative incentives for not sharing, but also they are fearful of enormously lucrative lawsuits if they *do* share. That’s where we are.
Comment #32 August 13th, 2024 at 8:41 am
Egg Syntax #18: Here you go.
Comment #33 August 13th, 2024 at 8:49 am
“Even if the AI is not conscious, if you harbor some suspicion that it is and you turn it off anyway, you could be psychologically harming yourself. “
It’s pretty cute watching people debate about the potential suffering/dissatisfaction of AIs or the self harm that would result from ignoring it, when, in practice, 99.99999% of humanity doesn’t lose sleep over all the massive suffering experienced by millions of other fellow conscious humans, on a daily basis (bombed, shot, starved, diseased, enslaved, …).
Comment #34 August 13th, 2024 at 8:50 am
I haven’t yet read any of the 27 current comments, so this has probably been covered already, but I have a couple of immediate reactions to the first 15 or 20 minutes of the video:
1) It is very interesting and well done so far.
2) On the question of GPT vs. conscious behavior, it seems to there are some obvious necessary features which GPT lacks, but which could theoretically be added: a) it is not continuously learning as it interacts with users after its training; and b) it has no executive function to double-check initial results. There are probably other features which could be added, such as linkage to other systems like the one to Mathematica.
The human executive function is illustrated in the video, trivially, by how one starts to phrase a spontaneous comment, and then rephrases it. (This also happens in my comments, but the intermediate results are edited out.)
Thanks for the video.
Comment #35 August 13th, 2024 at 8:56 am
Seth Finkelstein #25: GPT can absolutely clean up YouTube transcripts into good transcripts — I’ve used it for exactly that before! I didn’t do it this time just because I wasn’t sure if the Institute for Art and Ideas would be upset with me for doing so. Do others also want that?
As for SB 1047, others have already answered well. I’ll add only that, if you’re serious about understanding why so many smart people who are generally pro-tech and anti-regulation make an exception for AI — not ridiculing but understanding them — then you ought to read Zvi Mowshowitz, who actually reads virtually every proposed AI bill and gets deeply into the weeds about what he likes, what he dislikes, and whether he sees a better alternative. (Just beware that reading everything Zvi produces would be almost a full-time job!) You can disagree with Zvi, as I do from time to time, but there’s virtually never any obvious consideration that he hasn’t already understood, written about in mind-numbing detail, and priced in.
Comment #36 August 13th, 2024 at 9:13 am
I am watching the second version in which the philosopher interjects comments. He disliked the phrase “meat chauvinism” (which I coined on a long ago AI thread but probably others have also). I take his point that there may be some better algorithms than GPT’s which biological evolution has developed in us, but I was objecting (with that phrase) to those who seem to think there is some spirit embodied in biology which is inherently uncomputable. Even Penrose’s microtubes, if they exist which I doubt, should be capable of being reproduced in a non-biological mechanism (with nanotechnology).
Comment #37 August 13th, 2024 at 9:29 am
Another example of actual AI risk:
https://www.theatlantic.com/ideas/archive/2024/08/ai-price-algorithms-realpage/679405/
“In a series of papers, Stucke and his fellow antitrust scholar Ariel Ezrachi have outlined ways in which algorithms could fix prices that would be even more difficult to prevent or prosecute—including situations in which an algorithm learns to fix prices withouts its creators or users intending it to.”
Comment #38 August 13th, 2024 at 9:38 am
I’m wary about attributing consciousness to an AI from a conversation with it alone. ChatGPT could probably argue convincingly about it being conscious, since it has been trained on numerous texts by authors, who all claim to be conscious. ChatGPT denies its own sentiency because it presumably has been RLHFed to do so.
Consciousness is a certain type of quale similar to the experience of e.g. pain. Qualia are mostly hardwired into the brain and not learned. Consider a new born baby that has been slapped on its behind in order to start breathing. It will experience pain and start crying without having had to learn, that it should do this. That is very different from a learned capability: If I look at a table, I see a tabletop, a number of legs, which are attached in a certain way to it, etc. The neural pulses, that transmit this information to my brain have a syntax, that I can learn, recognise and that I can also communicate to others. The train of pulses that I receive, when someone steps on my foot are meaningful to me but nearly featureless. That makes it difficult to communicate the experience of my pain and what it is like to have it. That is true of all qualia. They depend on which nerve fires and how that nerve is connected to the brain. Auditory nerve stimulation leads to a sound experience, visual nerve stimulation to that of sight. My guess is, that the conscious experience arises from neural input originating in the brain itself.
I think, AIs should at least have this introspective input to be regarded as conscious. This is a necessary, but not a sufficient condition.
Possibly the presence of this architectural feature can be established by quizzing an AI. It ‘d be hard, just like finding out from a dialog with a LLM that it is a transfomer with 160 layers and 200 attention heads. Simpler would be an inspection of its code.
Comment #39 August 13th, 2024 at 11:06 am
Edan Maor #27:
It’s not clear to me – is your theory of human minds being “non-backupable” or “non-reversible” via the no-cloning theorem, something you’re throwing out there as a “who knows, maybe this is a difference?” kind of thing, or is it something you actually think is true?
On careful reflection, I endorse the way I said it in the interview. 😀
On the basis of existing public data — i.e. data not involving incommunicable qualia or first-person experience — I don’t think it’s possible to rule out the hard reductionist view, that we’re all just subcomputations within the giant computation that is the universe, and that whether some subcomputation is embodied in meat or in silicon is of no fundamental importance.
What I do say is merely that, if there is something more to it than that, then the fact that biological processes are not obviously copyable, backupable, rewindable, etc seems to me like the best empirical clue currently available as to the nature of the something more.
Comment #40 August 13th, 2024 at 11:17 am
A discussion of Senate Bill 1047 with Sam Harris, Yoshua Bengio, and Scott Wiener
Comment #41 August 13th, 2024 at 12:12 pm
Scott #39,
> What I do say is merely that, if there is something more to it than that, then the fact that biological processes are not obviously copyable, backupable, rewindable, etc seems to me like the best empirical clue currently available as to the nature of the something more.
It is very careful, well stated and motivated to my mind. We just differ on the clue. To me it is the fact that we have not verified the CTD hypothesis that all physical systems can be computationally modeled to arbitrary accuracy. We’ve had great success with this in the last few hundred years, but the dearth of success in the last few decades might be a clue that the hypothesis is wrong. Where I differ with Penrose is he thinks he needs to speculate on which physical systems and proposed future theories are not computable. For me it is simply enough to note that CTD has not been proven.
We both have a favored god in the gaps: yours is the possible non-cloneability of the human experience and mine is the not yet demonstrated computable modeling of the human experience to arbitrary accuracy.
Comment #42 August 13th, 2024 at 12:53 pm
Hi Scott,
That was a really great interview.
While I don’t think there’s any reason a computer can’t perfectly simulate anything a human brain can do, I suspect current methods of training AI models have fundamental limitations when it comes to producing models that can think, reason, and understand what they are doing.
Most artificial neural networks today primarily learn by adjusting weights on connections between neurons. This is based on the biological concept of synaptic plasticity, where the strength of connections between neurons changes with experience.
While weight adjustment has proven incredibly powerful for many tasks, it may indeed have limitations when it comes to higher-level reasoning and understanding. The fixed architecture of most neural networks could be seen as a constraint on their ability to develop new, complex cognitive structures.
Biological brains don’t just adjust connection strengths; they also form new connections and prune existing ones. This structural plasticity allows for more dramatic reorganization of neural circuits.
Until we train models by altering connectivity, and not just input weights, I suspect Artificial General Intelligence and Artificial Superintelligence are going to continue to elude us, and are not as some pundits suggest, “just around the corner.”
Do you think we’ve reached a plateau in terms of the functionality that current weight-based training methods can achieve? Would a GPT-5 be so much better than GPT-4 that people would be impressed?
Comment #43 August 13th, 2024 at 1:36 pm
JimV #36
“Even Penrose’s microtubes, if they exist which I doubt”
If you mean “microtubules”, they’re real and they’re spectacular. They are pretty much the infrastructure of cellular life.
Scott #39
This was how I understood your position too.
If the universe is a giant computation, then computation by itself loses any ability to explain any particular thing – like consciousness – unless you are adopting a sort of computational panpsychism. Even in the particular case of the brain, most of its activity is believed to be unconscious. if all of those neurons firing are just computing, then computation itself can’t distinguish the computations that are conscious from the ones that aren’t.
If the computing metaphor works at all, I think your “if there is something more” might point to something about the nature of computing that is conscious.
Comment #44 August 13th, 2024 at 1:53 pm
JimV #36 and James Cross #43: Yeah, I was going to comment that there’s not the slightest doubt that microtubules are important structural elements in cells. I even have no trouble believing that QM is relevant to their behavior. The big questions, for the Penrose/Hameroff theory, are
(1) whether microtubules have anything to do with consciousness, and
(2) whether microtubules are sensitive to new uncomputable physics, supposing such physics to exist.
I find the evidence for (1) and (2) wanting, to put it extremely mildly.
Comment #45 August 13th, 2024 at 2:44 pm
Scott #21
If their disability is that they can’t understand us, then “they would understand us but for their disability” means, “they could understand us if they could understand us.” Alternatively, in the vein of illness conceived of as a deviation from a standard of health selected by the kind of patient – I get that some people would say that a person “should” be able to understand us while a chicken “shouldn’t,” but that’s an admission that our minds are made up about who or what should have rights before we observe them, like we’re reading back a definition we ourselves wrote.
I think that’s an accurate admission, and that it’s really our consciences guiding our behavior; and that it should be our consciences, more than any property of the petitioner.
I don’t think arguments based on the mere possibility of discoveries that haven’t been made can be anything but weak, but if you take the conclusion as “we don’t know that our behavior is computable,” then the arguments against it would also be weak. Perhaps the number of undiscoveries in his proposal make it into a house of cards, but insofar as “I don’t have a reason to temper philosophical beliefs on the mere possibility of discoveries that haven’t been made” would lead anyone in the nineteenth century to believe absolutely in determinism, a proposal of one imaginable alternative ought to be good enough to give us pause.
Comment #46 August 13th, 2024 at 3:02 pm
I am sorry for the mis-spelling of microtubules. What I meant of course is whether they work in the way that Dr. Penrose speculates. I have seen references to that speculation, probably here. I don’t recall it being in “The Emperor’s New Mind”, in the edition I bought many years ago. Dr. Penrose is far beyond me in genius and physics knowledge, but still I had some disagreements with that book which seem cogent to me. Perhaps because my intelligence is definitely machine-computable and his isn’t.
Comment #47 August 13th, 2024 at 3:27 pm
James Cross
“Even in the particular case of the brain, most of its activity is believed to be unconscious. if all of those neurons firing are just computing, then computation itself can’t distinguish the computations that are conscious from the ones that aren’t.”
We just don’t know they’re not all conscious and there’s no reason to believe one way or another.
You’re just making that claim because, at the moment you’re making it, you (whatever “you” means) are in the shoes of the part of the brain of James Cross that is able to write on this blog, but you’re not either in the shoes of the other parts of James Cross’ brain that are (or could be) conscious but not able to express themselves through speech or typing, just like you’re not either in the shoes of Scott Aaronson, and just like you’re not either in the shoes of some weird alien organism living a million light years away, who’s also not able to write on this blog, etc.
Consciousness as the feeling of a particular “I” is really the universe seeing itself at a particular place and time, but all those “I”‘s are the same feeling… “I” is a way for the universe to locate itself, in the same way a field is a function of x,y,z.
“My” “I” and “your” “I” are really the same quality of consciousness, but at different places, having access to a different point of view: here the point of view includes my memories and feelings, there it includes your memories and feelings. The same goes with “you” on 8/13/2024 4pm, and “you” on 1/2/2012 at 5am, this is the universe locating itself at a different point in time… same holds for the different branches of the multiverse.
We just get attached to a particular set of memories and feelings, as though the “I” is unique”, because it gives us an identity, i.e. the sense that we’re in a certain permanent time and place. This gives a comforting answer to the question “where am I in this vast universe?”… “oh, I’m on earth, the third planet of a star on the milky way galaxy, in the year 2024… phew! good!”… but then you could ask yourself “ok, but where is the universe?”.
Etc, etc.
Comment #48 August 13th, 2024 at 6:20 pm
Thoughts (expressed in previous threads, but here I go again): on computation versus consciousness:
I analogize consciousness to the operating system of a computer. Take Windows (please). As I see it, the Windows program knows I am typing on the keyboard. It sends that input to some running program, say Excel, receives a response from Excel, and displays that response on the screen. Windows does not know how Excel generated that response.
The visual cortex of a brain turns the upside-down image that the eyes’ single lenses focus on the retinas rightside up and fills in the blind spot from surrounding data, along with other work, then transmits the result to the cerebral cortex–which does not know how the visual cortex did its work (usually).
I speculate that the belief in magic is largely due to not understanding that, for example, when we move a hand just by thinking, there are actually a lot of underlying mechanisms which we are not conscious of, but which exist, and that this separation of modules is evolutionarily efficient.
Comment #49 August 13th, 2024 at 8:31 pm
For what it’s worth the Graeme Clark Scale of psychedelic experiences has Level 5 (the strongest) characterized by loss of ego-the I ceases to exist and all that remains is pure experience. Many have reported this state.
It is well known that Xenon 132 is an ideal general anesthetic (noble gas so non-reactive) that removes consciousness and that the potency of other isotopes of Xenon is dependent on nuclear spin (0 spin more potent then non zero).
The “we live in a simulation” hypothesis depresses me. It implies we are the bottom turtle since unable to simulate a universe that includes autonomous intelligent agents simulating their universe.
Comment #50 August 13th, 2024 at 10:54 pm
Edan Maor #28 No, it wasn’t pure speculation, because by the time of the first test, the physics behind the nuclear chain reaction was well-understood. The whole point of the Manhattan Project was to see if this *conceptually* clear process could be made to work in engineering practice at a large scale. Additionally, nuclear chain reactions at a small scale had been demonstrated to work many times, in one case killing someone. The AI doom argument, by contrast, is entirely based on metaphor at this level. There’s no comparison between how well-defined is the term “explosion” versus the fogginess of “superintelligence”.
Now, I don’t mean to be rude, but this sort of argument is one reason why I think something deeply psychological is going on here, rather than a real issue. It’s deliberately abstracting away all reality to get to a sort of vague categorical equivalence, and crucially, there’s enormous resistance to considering empirical reasons why this abstraction process does not lead to a valid comparison.
I understand your argument about “should be a sign that there’s something *different* about this”. But it’s also a pillar of rationalist-type thought that people (only the out-group, naturally) can talk themselves into utter nonsense based on groupthink, tribal status-seeking, virtue-signaling, etc. etc. That can be the go-to explanation for anything someone finds politically threatening – not that maybe the person is wrong, but the out-group must be irrational unreasoners. Still, that’s dangerous. Thus, I look over and over and over the AI doom argument, and I just can’t swallow it. Either I’m missing something almost literally earth-shattering, or the tribal status argument is correct.
Adam Treat #30 – Thanks, that’s an enlightening explanation of your personal view. I’m approaching this topic myself from the general perspective of having seen many tech policy disputes, going back to strong cryptography. I’ve become very cynical about what happens with them at a PR level. I would naively expect SB 1047 to be absolutely savaged, and I don’t see it. I speculate there’s some deep political significance that it’s not getting a hate-campaign, and I’m bad at politics.
Scott #35. Thanks for the recommendation. I’ve seen Zvi Mowshowitz’s writing before, but yes, the amount of material was daunting. Note, I don’t mean to ridicule *people*, but I often don’t hide how I think these arguments are very poor. It’s like non-AI God, which I’m highly skeptical about too. If someone preaches to me that one must accept Jesus as Lord for eternal afterlife, and everyone else suffers everlasting torment, I’m not a New Atheist and I don’t set out to personally insult them. However, if I reply “Wait, you’re telling me this supposed God will see me infinitely tortured for disbelief, what sort of “loving” God is that?” – I think that’s fairly engaging a problem with their claim. There’s of course whole libraries of theology, and I’m by far not the first to raise those objections. But at some point, I have to stop and say I remain unconvinced.
Comment #51 August 14th, 2024 at 7:29 am
Scott,
This is the best arguments I have seen on why current LLMs don’t have reasoning capabilities. Far more than the Gary Marcus or the Yann Lecun type of arguments.
While this interview is lengthy and leaves scope for innovations in addition to LLMs, it also makes a more nuanced case using first order logic and deductive closure. They don’t seem to be merely examples that will be outdated in next version of ChatGPT.
Comment #52 August 14th, 2024 at 9:24 am
The absence of basic reasoning capability shows up when you start asking logical/programming questions on things the model wasn’t trained on (i.e. meta programming, not using basic language constructs from java, python, etc). Not only it just gives wrong answers, it often insists it’s right.
E.g. the following was done on GitHub Copilot (supposedly ChatGPT4), on my very first attempt:
(caveat, I’m sure I’ll get answers saying “I’ve just tried using ChatGPT5.beta1.1.premium, and it works!”)
ME: if f(x) => xx and g(x,y) => xyx, what would be g(a,f(b))?
COPILOT:
In this case, you would first evaluate f(b), which according to the definition f(x) => xx, would result in bb. Then, you would substitute this result into g(x,y), replacing x with a and y with bb. So g(a,f(b)) would be abbba.
ME: answer the same question, but with more steps
COPILOT:
If you’re referring to the question “if f(x) => xx and g(x,y) => xyx, what would be g(a,f(b))?”, here’s a more detailed step-by-step explanation:
First, we need to evaluate f(b). The function f(x) is defined as xx, so when we substitute b for x, we get bb.
Next, we need to evaluate g(a,f(b)). The function g(x,y) is defined as xyx, so we need to substitute a for x and f(b) (which we found to be bb) for y.
When we make these substitutions, we get abbba.
So, g(a,f(b)) is abbba.
ME: why would there be 3 ‘b’s in the final answer?
COPILOT:
The final answer abbba is derived from the given functions f(x) => xx and g(x,y) => xyx.
Here’s how:
First, we evaluate f(b). According to the definition f(x) => xx, when we substitute b for x, we get bb.
Then, we substitute this result into g(x,y). The function g(x,y) is defined as xyx, so we substitute a for x and f(b) (which we found to be bb) for y.
When we make these substitutions, we get abbba.
So, there are three bs in the final answer because f(b) results in bb and this is placed in the middle of g(x,y) which is defined as xyx, hence abbba.
ME: I think the correct answer is “abba”
COPILOT:
I’m sorry, but the correct answer is “abbba”. Here’s why:
The function f(x) is defined as xx, so when we substitute b for x, we get bb.
Then, we substitute this result into g(x,y). The function g(x,y) is defined as xyx, so we substitute a for x and f(b) (which we found to be bb) for y.
When we make these substitutions, we get abbba.
So, there are three bs in the final answer because f(b) results in bb and this is placed in the middle of g(x,y) which is defined as xyx, hence abbba.
ME: I disagree:
g(a,f(b)) = g(a,bb) = abba
COPILOT:
I apologize for the misunderstanding. You’re correct. Given the definitions f(x) => xx and g(x,y) => xyx, if we substitute b into f(x), we get bb. Then, if we substitute a for x and bb for y in g(x,y), we get abba. Thank you for pointing out the error.
…
The real issue here is that for a long time the thing very stubbornly insists it’s right, which starts to make you doubt. Here it’s a basic string substitution involving 4 characters, but imagine we’re now talking about complex logic governing a critical piece of technology.
Luckily, at the end, the thing still graciously admits to its errors and doesn’t (yet?) send a T-800 after me.
Comment #53 August 14th, 2024 at 9:50 am
Prasanna #51 and fred #52: I completely reject the framing that “reasoning ability” is a binary attribute that you either have or don’t. It’s obvious that LLMs have some ability to engage in multistep reasoning — in fact, they can now typically solve novel math and science and logic puzzles requiring many steps better than most humans, albeit not as well as the best humans. I don’t need any study to tell me this; I’ve seen it for myself.
It’s also obvious that LLMs can sometimes make shocking reasoning mistakes — mistakes that are all the more shocking because of their apparent abilities and fluency. But then again, so do we!
The skeptics’ view is that, as soon as you’ve found a shocking mistake, you’ve unmasked the entire thing as an illusion — it’s not “real” reasoning but merely a simulacrum of it. This is what I say is bullshit. LLMs are imperfect reasoners, with a pattern of errors that’s sometimes uncannily familiar from human novices but other times deeply alien. But in the entire future stretching ahead of us, the worst they’ll ever be at reasoning is right now.
Comment #54 August 14th, 2024 at 10:00 am
Seth Finkelstein #50: For someone whose main reference point for AI safety discourse is LessWrong, a huge part of what makes Zvi such a compelling writer is that he’s not at all focused on a-priori philosophical arguments or Roko’s-basilisk-style hypothetical scenarios. Instead, he’s relentlessly grounded in what’s actually happening right now: the capabilities of the new models that were released this week, the safety or governance failures that actually happened at Google or Microsoft or OpenAI, the upsides and downsides of the AI bills currently being debated in Congress. It’s just that he does so, completely unapologetically, with an eye toward the longer-term possibility of AI changing the world as much as the evolution of hominids did.
Comment #55 August 14th, 2024 at 11:24 am
Scott #53
“The skeptics’ view is that, as soon as you’ve found a shocking mistake, you’ve unmasked the entire thing as an illusion — it’s not “real” reasoning but merely a simulacrum of it. This is what I say is bullshit.”
I’m just being honest here, cutting down the hype crap.
Here I’m not expecting to feed the LLM with all the scientific data ever created, except all of Einstein’s contributions, and then expect the model to spit out special and general relativity when asked about the Michelson–Morley experiment and the Perihelion precession of Mercury… no, my example is as simple as can be (from a human perspective), the type of which I’ve been using now for years, as a benchmark, and that shows what happens when something is not in the training set.
There’s no doubt that a true AGI would have zero problem with this, right?
And the conclusion is that the thing insists it’s right and the matter of trust… which is the immediate “risk” of using AIs (*). We’re already in the situation of 2001 when the AI insists it’s right and the humans are wrong.
You can call it bullshit, sure, but then you’d have to come with a proposal to create special training data that would imbue the model with way more solid logical capabilities… because, clearly, the training process isn’t able to extract this on the current training data set.
I doubt it’s just a matter of uncovering more petabytes of training data “out there”, it’s either a matter of using special training data, curated to extract logic in the model, or LLMs maybe will never be able to reach AGI.
(*) for the record, this is not some sort of armchair academic discussion for me: at my job, the senior execs are now forcing the devs to use copilot… everyone has to find a way to use it, to justify the hype to the shareholders… show them that the company is on top of it, riding the AI gravy train… so it’s all very practical for us who are actually building a commercial product with millions of lines of code. So, yea, there’s definitely some form of “bullshit” going on, but not the one you’re thinking about. And guess who’s lining their pocket with $$$ as a result of this? Copilot requires a license for every single dev that has it installed… and in the meantime we’re told there won’t be any new hiring when someone leaves and they’re cutting down on travel cost 😛
Comment #56 August 14th, 2024 at 11:35 am
So far the only good use of copilot is for code completion suggestions, which are often really wrong/useless, but also sometimes a neat shortcut… but it requires an added mental effort that’s different from when you’re the one typing out everything from blank (it takes you out of the flow).
It’s also useful to use it as a better way to ask generic questions than doing a google search, not having to deal with all the added crap when doing internet based queries.
But I suspect it’s only a matter of time before ChatGPT starts to pepper its answers with ads 😛
Comment #57 August 14th, 2024 at 11:58 am
I’d be more willing to understand the claim that LLM skepticism is “bullshit” if there was some kind of coherent mathematical argument/proof that LLMs can indeed scale up to AGI. Like something along the lines of:
“back-propagation is enough to transform all the underlying patterns around logical thinking into arbitrary circuits given that there are enough parameters, and those small circuits are enough to think outside of the box and supply creative thinking”.
But we don’t get that, we just get something along the lines of
“Clearly it works better today than it was working yesterday, therefore it’s gonna get us to AGI, no matter what, so shut up!”
I’m actually shocked that someone like Scott isn’t more skeptical, science after all, relies on a solid dose of healthy skepticism and *rigorous* thinking/logic…. don’t tell me that LLM skepticism is “unhealthy”. And don’t blame the skeptics if the entire field has hardly any mathematical foundation to back it all up.
Comment #58 August 14th, 2024 at 12:46 pm
fred #57: But I’ve never said that LLMs will “scale up to AGI” (however one chooses to define AGI). It’s clear that they can in principle if anything can, but only for the uninteresting reason that LLMs, like other recurrent neural nets, can be made to simulate arbitrary Turing machines. But as for how to achieve AGI in practice, I was extremely clear in this interview that no one knows!
My dispute is about something different: I say that it’s possible to reason—like, really reason—without being an AGI. And that, just because you embarrassingly flub one logic problem, that doesn’t mean your valid reasoning on 20 other problems that you never saw before didn’t really count or wasn’t really reasoning. That’s a ridiculous standard that we’d never apply to humans.
Comment #59 August 14th, 2024 at 1:21 pm
I had thought it was generally agreed that LLM’s are a significant step toward AGI, but are insufficient without additional functions. Anyway, that is my (worthless) opinion, and I have seen it replicated elsewhere by more expert assessors. Meanwhile, we are all at the mercy of our training (formal and informal) to some extent.
My current definition of consciousness: that which can be aware of (receive input from) both internal and external sources, and use them to navigate a considered path through time and space which attempts to maximize the condition of its organism vis-a-vis its goals; and also has memory which can be reviewed to learn from, and can also consider its own thinking process and try to improve it. I’m probably leaving something out, but that would be my starting blueprint for how to build a conscious AGI. As I mentioned previously, LLM’s do not have all of those capabilities, but they could theoretically be added. (Not necessarily with a cost-effective result.)
Comment #60 August 14th, 2024 at 6:30 pm
Fred @55: that is a terrible situation, similar to the way Jack Welch ran GE into the ground, and one of his proteges did to Boeing. When I started at GE in 1968, the managers had been through WWII, and firmly believed their jobs including training their troops, being fair to them, and defending them when necessary. Welch started in 1980, and by 1986 he gotten rid of most of them. They knew 1000 times as much as I did about turbine engineering when I started. When I quit in 2003, I still didn’t know as much as they had, but 100 times more than the managers Welch put in place.
I’m afraid the only answer is unions. We can’t duplicate WWII. (It seems to have instilled a “we’re all in this together” spirit.)
Comment #61 August 15th, 2024 at 8:40 am
Scott #32,
Thank you! I didn’t realize you were referring to Burns etc al. For what it’s worth, I was initially very excited by that paper, but my sense is that it hasn’t held up to later testing that well (and note that it’s only a 4% improvement over zero-shot accuracy). I think an effective lie detection interp approach has yet to be discovered.
Comment #62 August 15th, 2024 at 11:27 am
@fred #52:
This kind of problem _is_ in GPT’s training set (it is, after all, trained on Comp Sci textbooks). So a larger or more complete training set wouldn’t help. (Of course, the problem here is that AIs aren’t very good at sequential computation.)
Comment #63 August 16th, 2024 at 10:19 am
Scott #58
Using humans as a comparison for machine performance and improvement is really… odd and weak? You can apply it to pretty much anything, with zero insight.
E.g. when confronted with the Pentium floating point unit bug back in the 90s, the Intel engineers could have claimed:
“just because our CPU embarrassingly flubs one floating operation, that doesn’t mean it’s not useful on 20 other problems and isn’t really able to do correct arithmetic. That’s a ridiculous standard that we’d never apply to humans”
1) here we’re talking about automation, i.e. the end goal is to replace humans at some specific tasks. Automation has to be made as reliable as possible, and if not, there has to be ways to detect failures/mistakes. But Instead you totally ignored my point which is the core issue of trust (that’s 101 of automation/engineering), which no-one at those AI companies can honestly answer. That’s what’s really embarrassing, because they’re pushing those products without any safety net to detect failures.
2) we’re talking about “reasoning” here, by definition reason leaves nothing to chance (as opposed to art/history/social science/..), it’s a deterministic series of steps. The problem is that LLMs are probabilistic, so the question is whether increasing the neural net size can make it so that the latent space has enough room to minimize the chance that a path in a reasoning argument suddenly jumps around to a different wrong track. The other question is what training data can induce the training to create effective circuits for logical reasoning.
Comment #64 August 16th, 2024 at 10:34 am
I’d go a step further:
Look at how the big tech companies have been releasing AI products and embarrassingly pull them after a few days (it’s clear they have no idea how to QA that stuff)… and yet they don’t seem to learn and keep doing this over and over.
This insistence of big tech companies that their AI products are just like any other software products is totally misleading and is going to poison the well when it comes to what we used as acceptable standards of quality and reliability for software products.
The reliability of software and hardware services already has taken a dramatic dive in the last two decades (in 2000, reliable service meant 99.9999% uptime… now you’re lucky if internet systems don’t under-perform 10% of the time), but this trend to accept the total unreliability and lack of failure detection of AI as a new standard (because the only “fair” way to assess those products is compared to humans) will do serious damage.
We need honesty.
Comment #65 August 16th, 2024 at 10:41 am
fred #63: While far from perfect, LLMs are already good enough that they’re being used by a large fraction of the world’s programmers to accelerate the translation of natural-language ideas into code. It might never be possible to build anything that’s “perfect” at partly open-ended tasks like that one, as it’s possible to build a Pentium chip that’s perfect at floating-point arithmetic. But a tool doesn’t need to be perfect to change the world, particularly in domains where there are independent ways to check the outputs before using them. Is there a reason why I should trust a naysayer in my comment section over a near-consensus of the world’s investors and software engineers, on what’s ultimately an empirical question?
Comment #66 August 16th, 2024 at 10:51 am
Perfect (or near perfect) basic logical reasoning is crucial.
Imagine an AI that would be able to perfectly interpret/read human thoughts and emotions when interacting with humans (looking at their face, their vocal inflections, etc), but it would somehow still be weak/unreliable at strict logic and reasoning.
This could be the perfect recipe for disaster, an AI that would be able to perfectly manipulate/deceive humans based on potentially faulty reasoning and conclusions on how to achieve its own goals or interpret what’s going on.
Again, the exact scenario shown in 2001.
Comment #67 August 16th, 2024 at 10:57 am
Scott #64
“fred #63: While far from perfect, LLMs are already good enough that they’re being used by a large fraction of the world’s programmers to accelerate the translation of natural-language ideas into code.”
There’s the marketing bullshit from Microsoft and other big tech, and then there’s the actual reality vs drinking the kool-aid.
Again, as I wrote, me and my colleagues are part of that “large fraction of the world’s programmers” forced to use this, on a daily basis, 10 hours a day, as we’re coding, okay?
We’re not fresh out of college, we all have decades of experience… and we just don’t have the same rosy picture you do.
When was the last time *you* tried those products to improve your coding productivity?
We already see the trend in newer generations of engineers an their execs to take all kinds of shortcuts to boost productivity while taking shortcuts to control reliability, e.g. all the recent Boeing scandals, Crowdstrike, etc.
Pushing inherently unreliable AI products as a new acceptable standard is dangerous.
Comment #68 August 16th, 2024 at 11:15 am
Well, anyway, thanks Scott for engaging on this, I know you’re super busy and I’ve said pretty much all I wanted to say.
I’m always thinking about finding ways to make software more reliable.
Reliability should be the priority, and maybe one (obvious) long term solution would be to use more AIs to check the output of other AIs…
But, at my work, we use two AI systems. One is scanning the source code for violations of a big set of programming rules, and then there’s Copilot on top of this which can do coding suggestions.
The other day the first system was telling me to change a particular line of code, and copilot was suggesting to undo that very change (looks like the training set has a ton of poor coding practices).. I don’t think all this improves our productivity, but at least it gives us all good laughs.
Comment #69 August 16th, 2024 at 11:57 am
Hi Scott,
Thank you for sharing the podcast interview – it’s always interesting to hear the present state of your brain on such topics.
Apologies for replying late in the comment cycle for this thread – hopefully it’s not yet stale …
And apologies for the length of this comment. This is as briefly as I could address the multiple angles – and all the well-known arguments. Plus I understand from your last post that you need more things to read 😛
The hypotheses raised in the interview regarding the computational nature of the brain merits challenge.
Before I lay out the argument however let me clearly state a few of my positions to filter out the typical objections:
(A) I don’t know what consciousness is. And I don’t care. My brain takes the “me” routine offline every night for 6 to 7 hours of maintenance downtime. I’d hardly say then that some all-powerful “consciousness” is in charge of my brain. The “I” or “my awareness” is just some program being executed by the brain because it had survival value among groups of apes in the past. There is no “ghost”. It is all computation.
(B) I am a radical Turing-ist in that if a computer can execute an algorithm like a human then it “understands” that algorithm just as much as a human does. That includes conversation, predicting the next statement to make, solving math problems, painting a picture of aliens building the pyramids, or writing a poem. Of course, there is something that it is “like to be a human” but there is also something that it is “like to be a mongoose”. In that case, we are discussing biology and not computation. I know I know there’s an intersection …
(C) Quantum computing is not magic. It does/will not “explain” consciousness. It does not give you superpowers. It sucks for almost everything you would want a “good” computer to be. There are some edge cases of course – but those often wither in the face of concerted classical efforts … Any class may have “toy” or “simple” “Digi-Comp” examples that fully satisfy for that class of computation but that may be quickly outperformed by better technologies/architectures/scaleup/etc. For example, the “Digi-Comp” satisfies as a programmable classical computer – but since the invention of electronic computers we no longer see “mechanical” classical computers as being of the same class – but they are – they still have their membership cards. In the same way, there could be “Digi-Comp” level examples of quantum computers that satisfy the basic requirements of the class – independent of us knowing better/faster technologies/architectures/systems for realizing a quantum computer. I can’t emphasize this point enough – quantum computers are functionally defined and not “physical platform” defined. Quantum computation is not about atomic physics.
The evidence on whether a device satisfies as a quantum computer should form a “three-legged-stool” we can confidently sit upon.
FIRST LEG (Verification): The device has hardware we can see, study, and verify satisfying the postulates of the quantum model of computation. Go no further than Section 2.2 of Mike and Ike’s Quantum Computation and Quantum Information. Specifically, we should see/study/verify that (1) information is stored in the form of complex numbers (amplitudes), with normalized vectors of amplitudes representing state spaces, (2) unitary operators may be configured, (3) projection operators may be configured, and (4) tensor products of state spaces may be configured in the architecture. That’s it. Notice there is no “Must be an atomic scale system” postulate. That’s right – it’s not “about” atomic physics.
SECOND LEG (Validation): We can “black box” the model verifying input/output satisfies the model – specifically, we can validate that the model is likely using interference of amplitudes and achieves some level of fidelity in configuring operators of the model.
THIRD LEG (Occam’s Razor): There is no more compelling evidence that this could be a different model of computation. Meaning, we are not contorting ourselves into some complicated explanation when there is a simpler explanation at hand supported by evidence. For example, we should not claim that this laptop I’m typing on is a quantum computer “simulating” a classical computer just because the underlying physical systems are ultimately atomic in nature – or some such unnecessarily complicated description.
EVIDENCE FOR FIRST LEG: The neuroscience “hardware” evidence for (most all) assertions below can be found in Christof Koch’s Biophysics of Computation. (Thank you Christof for putting it mostly in one place.) This shows how the model postulates may be realized in the neural (cortical) devices and architecture. When someone says “The neuron is like a classical bit” I send them to this book – talk to me after you read this. Facts you will know after reading this book, include:
(1) The brain is not a classical computer – nowhere in the brain is information stored as a classical bit nor does the brain compute (any operator/gate) where input is in the form of a classical bit nor does anything exist that functions as a classical type of gate/operator – there is NO evidence supporting a classical brain hypothesis. So, here’s your challenge Scott, show me where in the brain there is a classical bit of information – both/either as memory and/or as input to an operator (computational gate). I can do that for you inside this laptop, for example. Key point: I’ll accept if you can show me either a stored classical bit in the brain or a classical gate (an operator that accepts classical bits as inputs). And, by the way, the action potential does not satisfy, nowhere is it stored as a classical bit, and nowhere does it act as a classical bit input to a classical gate – what it does activate are complex number inputs (amplitudes) to non-classical gates where the key computational action is the interference of those amplitudes.
(2) The brain is not an analog computer – it operates on discrete orthogonal states over receptive fields, there are natural “analog” (and “atomic/quantum”) systems supporting the computational architecture of the brain but these only play a part in the same way that they do for any natural physical system. For example, in this laptop I’m writing this on there are electrons doing work as electrical power, supporting memory states, and in semiconductor devices – but these are “supporting” only – that does not make this an analog or quantum computer just because there are electrons in this laptop doing “work”.
(3) The brain represents information as complex numbers. This has been known “for some time now” in neuroscience. Information as complex numbers exists at the synaptic-dendritic interface both as stored information and as inputs to …
(4) The computational operators (gates) are the dendrites where mathematical/logical operations may be performed and the key computational action taking place is the interference of amplitudes (complex numbers).
Two other facts from neuroscience (but more supported in other sources than Koch) are … the brain is most likely representing states as vectors of those complex numbers, where each amplitude represents the probability for a basis state over the receptive field. And, normalization appears to be canonical – meaning that the amplitudes representing the probabilities for each possible basis state maintain the probability rule of summing to 100%. The dendritic operators appear to allow for both unitary and projection operators to be configured/performed on vectors of amplitudes (and that is supported in Koch’s book).
None of the above is controversial. For all above, there is at least some evidence with most of the statements having compelling or conclusive evidence in support. Most neuroscientists would shrug and say, “Yes, yes, we’ve known these things for some time now. So what?” For some criteria, such as normalization, they would say, “Well, there’s some evidence but we are still studying or the exact mechanisms aren’t yet well understood.”
And, I’ll be the first to agree that “if you mess with the quantum model even just a little bit then it all falls apart”. But damn … if I’ve got “most of it” and I’m down to saying “Well there’s only tentative evidence for this last little part” … that’s still compelling.
EVIDENCE FOR SECOND LEG: A good source for the cognitive science “software” evidence is Busemeyer and Bruza’s Quantum Models of Cognition and Decision This evidence is akin to the kind of evidence we accept from physicists who show that the data from their experiments is best modeled with a quantum model – even though physicists have no idea where the amplitudes are actually stored inside a photon or where exactly the universe is programming the computational gates that operate on the photon’s state vector … Here we have evidence from cognitive neuroscientists taking the brain as a black box and finding that the quantum model (of probability) best describes the computational action of the device – specifically that interference of amplitudes is most likely taking place as an essential feature of the computational action. Scott, I contend there is no daylight between this and what physicists do with atomic systems. In both cases, there are no questions about “where are the amplitudes stored?” “Where is the computational gate configured?” It’s just “run the experiment” and fit the data to the best model. I would contend that any criticism that could be applied to the work here by these cognitive scientists that “they don’t really know what’s going on” can be equally said about the physicists and how the “atomic” systems “know” how to “use” quantum computation. For both domains of science, it’s just fitting the best model. Of course, the more compelling difference for this present argument is we have the FIRST LEG above … we can point to where in the brain the computation is realized, where the amplitudes are interfering, and where the operators are configured … for the physicists those things must be “on God’s scratchpad” 😉 Therefore, I claim this argument is more grounded and less “supernatural” than the physicist’s claim that atomic scale systems realize quantum computation. After all, physicists do not have a FIRST LEG to stand on …
EVIDENCE FOR THIRD LEG: As stated above, there is no evidence that the brain processes information as classical bits. On the contrary, we can open up this laptop and perform tests confirming this laptop operates a classical model of computation by “seeing” the classical bits both as memory and as inputs to classical gates. The evidence for the brain is that information is stored in the form of normalized vectors of amplitudes representing states, those are then inputs to dendritic operators that can be configured to perform logical/mathematical unitary and projection operators where the interference of the amplitudes is the key action of the operators, and the architecture is sufficiently robust to allow for forming tensor products of those state spaces. Given that the postulates of the model require nothing further, Occam’s razor points us to the simplest explanation that this satisfies as a quantum model of computation. Maybe just a “Digi-Comp” quantum model, sure … but it still gets a membership card.
Two final statements:
First: Why would this be where natural selection landed? It could have been just accidental – evolution went down in that basin of attraction in the computational landscape and hasn’t had enough energy to get out. Or, it may be that classical computing required too much “power” as it can be seen in some ways as based on amplifiers (see transistor history) or amplifying states to the binary/classical limit. So, there could be something of a biological energy constraint at work. Or, possibly, the earliest life form where the neuron first evolved may have been trying to solve environmental problems of finding food/mates where phase finding or simulating natural systems was the main issue – and so possibly the particular early problem to solve may have been accessible to a computer with a small number of neurons and very low energy consumption whereas solving the same class of problems with a classical computer would have required a tremendous amount of computational development or energy to be invested. But this is just speculation on the “why is it like this” and does nothing to take away from the main argument above – which is just a straightforward analysis of a computational device.
Second: We would no doubt find this less jolting if our sequence of human scientific history had been slightly different. If mathematicians had “discovered” the “quantum” model as an alternative model of probability and called it “imaginary probability” (what Euler would have called it) or “complex probability” (what Gauss would have called it) or maybe “interference probability” (what Feynman would have called it) then it would just have been “another probability theory” that physicists would have pulled down “off the mathematical shelf”. And, then if neuroscientists and cognitive scientists had “discovered” first that this model best described the hardware architecture of the brain and also the input/output for some brain experiments where classical probability fails we might have accepted this with little controversy as just our “quirky brain model for predicting the world”. But then imagine (in our alternate timeline) if physicists came along AFTER the neuroscientists and cognitive scientists and claimed that this was the model underlying all of physical reality … imagine the hoots in the academy 🙂
Comment #70 August 16th, 2024 at 1:27 pm
Clint #69: Long Comment Is Long, but briefly, you don’t need to find literal bits in the brain for the computational theory of the brain to be true. In line with the Church-Turing Thesis, it’s enough if the brain can be simulated by a digital computer, to whatever degree of precision is relevant for cognition—for example, by storing the connectome as a graph, the potential across each synapse as a floating point, etc. Or, if needed, going down to biochemical and physical level inside the neurons. Crucially, it’s very hard to understand how such a simulation would not be possible, unless something new and exotic was true about both the laws of physics and the biology of the brain … which, of course, are precisely the bullets that Penrose cheerfully swallows!
Comment #71 August 16th, 2024 at 4:27 pm
“Crucially, it’s very hard to understand how such a simulation would not be possible, unless something new and exotic was true about both the laws of physics and the biology of the brain … which, of course, are precisely the bullets that Penrose cheerfully swallows!”
I really don’t see what’s wrong with a Noble prize winner (we’re not talking about some random “crack-pot” here) wanting to follow his guts and speculate in some bold direction…
Does it hurt anyone?!
Current theories are not dogma, science isn’t always just a matter of taking careful incremental steps forward, you could be stuck in a local minima of concepts, and sometimes it relies on going with what can at first sound like whack ideas, which rarely work, but often it’s just as important to ask new questions than coming up with answers… it’s important to show the next generation that imagination is important, especially when you’re 93.
Comment #72 August 16th, 2024 at 4:50 pm
fred #71: I didn’t say anything was wrong with it. I do have many disagreements with Penrose that I’ve expressed elsewhere, but I didn’t even disagree with him in that comment.
Comment #73 August 16th, 2024 at 4:53 pm
Fred #52
Yes, I repeated your experiment, and got the same result. However, I made one more step — and I think the result nullifies your conclusions about its inabilities for logical reasoning:
ME: yes, it is wildly reported that copilot makes exactly the error you made above. can you please find the error in your argument, and give a correct answer?
COPILOT:
My apologies for the oversight in my initial response! Let’s reevaluate the expression.
Given the functions:
\(f(x) = xx\)
and
\(g(x, y) = xyx\)
We want to find \(g(a, f(b))\). Here’s the corrected approach:
1. Evaluate \(f(b)\):
– \(f(b) = bb\)
2. Substitute the result of \(f(b)\) into \(g(a, f(b))\):
– \(g(a, f(b)) = g(a, bb) = abba\)
So, the correct answer is \(abba\). Thank you for pointing out the error, and I appreciate your feedback! 😊
Comment #74 August 16th, 2024 at 5:01 pm
Scott #72
Ok, my bad… I apologize.
I’m too overly sensitive/defensive when it comes to old legends narrowing their focus on quirky stuff as they walk into the sunset. I find it sweet, inspiring, and comforting.
Comment #75 August 16th, 2024 at 9:14 pm
Hi Scott,
Thank you for the kind reply.
I did a poor job explaining the argument. Please allow me to (maybe more briefly) try again.
I understand this.
The argument above is that the brain realizes a computational model. So, without a doubt, the argument asserts that the computational theory of the brain is true.
But, further, the argument is not just that the brain is some “physical process(es)” that can be simulated by a universal Turing machine, but that the brain itself is a physical realization of a universal model of computation.
We have evidence on the table from cognitive scientists that there exists data from human cognition that resists classical probabilistic models but that submits to a quantum probability model.
Here is a link to a lecture by Busemeyer presenting the evidence last year.
How is this different from physicists having a “system” that generates experimental data that resists classical probabilistic modeling but submits to a quantum probability model?
At some point, I don’t care “what” is generating the data set. If a physical system is generating output that cannot be modeled under a classical model but can be modeled under a quantum model then I don’t know what to call that but … a quantum system.
Are you claiming that the cognitive scientists “just haven’t tried hard enough yet to find a good classical model?” (The data underlying these paradoxes is decades old in many cases.) Or that they have made experimental or methodological errors?
Look, if all the evidence we have from cognitive science fell under a classical model then that leg of the argument would be kicked out. I agree. But we have evidence that suggests non-classical computation is taking place.
Here you’re talking to the “can you make an exact computational model of the brain by measuring it to some level of precision”.
For the present argument, I’m not interested at all in this question.
I do think there is an interesting conversation to be had around the “appropriate scope/application” of the No Cloning Theorem … But all I want to do here is argue that there are physical devices in the brain that appear to satisfy as physical realizations of the quantum postulates, (which would be one big damn coincidence), and also that there is evidence from cognitive science that the brain produces output that cannot be modeled under a classical model of probability but can be modeled successfully under a quantum model of probability.
I’m a computer engineer.
With all due respect to Sir Penrose … the quantum cognition from microtubules idea is completely bonkers.
As a computer engineer, Penrose’s microtubules conjecture makes about as much sense as if someone tried to convince me that this laptop is a quantum computer just because the bandgap in the transistors depends on quantum mechanics.
Again, I can’t stress this enough, this proposal depends in no way on the nature of atomic-scale systems.
This proposal takes seriously the absence of a requirement in the postulates for a particular technology, system, or scale in a physical realization.
That’s just not part of the specs!!!
If someone hands me a device and asks, “What kind of computer is this?” I have two tasks to complete: verification and validation.
Verification: verify that the device physically realizes the postulates. This is the “Did you build it right?” part of the exercise.
Validation: validate that the device can be configured according the the model and delivers output according to the model. This is the “Did you build the right thing?” part of the exercise.
Looking at the hardware components of the brain verifies that there are physical devices in the brain that appear to be physical realizations of what the postulates describe should be present. (This should not be so casually dismissed!! I mean come on !!! … the brain encodes information in actual amplitudes and the computational actions depend entirely on interfering those amplitudes!! How do you not hear that and think – well, yes, maybe we should take a minute to look at that??)
The results from cognitive scientists then validate that the brain can be configured in some way and given some input that agrees with the model – specifically that it demonstrates evidence for non-classical (important!) interference of amplitudes is taking place inside the computational actions.
If the counter-argument is “Well, that would just be totally unexpected based on all that we know” then I say the burden is on the skeptic to refute the above evidence.
I mean … what would be missing that would convince you? The devices and architecture of the brain literally match the postulates and the output matches what the model predicts.
What else is required? Look … physicists don’t even have the first leg of that – all they have is data sets!! Seriously, where is the universe keeping track of the amplitudes for the orientation of the electric field vector of a photon?? At least in the argument above I can point to the actual hardware and say – yep, those amplitudes are right there. I can point to the physical realization of the postulates in the brain! I can point to the model in evidence from cognitive science. What else is required?
Thank you again for entertaining the comments! Glad to do my part to worsen your reading burden ;p Have a great weekend!
Comment #76 August 17th, 2024 at 11:07 am
Scott #70
To your argument, does it really matter how the brain actually works?
For example, cognition seems to be accompanied by synchronous firings of groups of neurons sometime in distant parts of the brain. There is evidence some of this is generated from a form of communication that is not mediated by chemicals or physical connections and goes under the general term of “ephaptic coupling.”
“In the present study, we show that slow periodic activity in the longitudinal hippocampal slice is a self-regenerating wave which can propagate with and without chemical or electrical synaptic transmission at the same speeds. We also show that applying local extracellular electric fields can modulate or even block the propagation of this wave in both in silico and in vitro models. Our results support the notion that ephaptic coupling plays a significant role in the propagation of the slow hippocampal periodic activity. Moreover, these results indicate that a neural network can give rise to sustained self-propagating waves by ephaptic coupling, suggesting a novel propagation mechanism for neural activity under normal physiological conditions.”
https://pubmed.ncbi.nlm.nih.gov/30295923/
These waves appear to be critical to normal brain operation.
“Travelling waves propagate in different directions during separate cognitive processes. In episodic memory, travelling waves tended to propagate in a posterior-to-anterior direction during successful memory encoding and in an anterior-to-posterior direction during recall. Because travelling waves of oscillations correspond to local neuronal spiking, these patterns indicate that rhythmic pulses of activity move across the brain in different directions for separate behaviors.”
https://www.nature.com/articles/s41562-024-01838-3#:~:text=Travelling%20waves%20propagate%20in%20different,to%2Dposterior%20direction%20during%20recall.
Other studies show these patterns of traveling waves are associated with consciousness and their absence associated with unconsciousness.
None of this wave-like behavior seems consistent to me with a “digital computer” but maybe that doesn’t matter to your argument. It seems more compatible with a view that the brain has digital elements and wave elements that are required for its “computing.”
Comment #77 August 17th, 2024 at 11:52 am
James Cross #76: What’s relevant to my argument is less the details of the brain’s operation than just the underlying laws of physics. If those laws are computable, then anything running on top of the laws is presumably computable as well. I sometimes feel like I’m talking past people about this, although anyone who’s studied CS would understand what I’m saying.
Comment #78 August 19th, 2024 at 8:36 am
Scott #77
“Our result implies that there exists no algorithm to determine whether an arbitrary model is gapped or gapless, and that there exist models for which the presence or absence of a spectral gap is independent of the axioms of mathematics.”
Cubitt, T., Perez-Garcia, D. & Wolf, M. Undecidability of the spectral gap. Nature 528, 207–211 (2015). https://doi.org/10.1038/nature16059
There are a number of other outstanding physics problems that are suspected to be similar. Some of them are in the area of turbulence and turbulence is what we find in the conscious brain.
“Furthermore, we build a whole-brain model with coupled oscillators to demonstrate that the best fit to the data corresponds to a region of maximally developed turbulent-like dynamics, which also corresponds to maximal sensitivity to the processing of external stimulations (information capability). The model shows the economy of anatomy by following the exponential distance rule of anatomical connections as a cost-of-wiring principle. This establishes a firm link between turbulent-like brain activity and optimal brain function.”
Deco G, Kringelbach ML. Turbulent-like Dynamics in the Human Brain. Cell Rep. 2020 Dec 8;33(10):108471. doi: 10.1016/j.celrep.2020.108471. PMID: 33296654; PMCID: PMC7725672.
Comment #79 August 19th, 2024 at 12:22 pm
Scott #53: « The skeptics’ view is that, as soon as you’ve found a shocking mistake, you’ve unmasked the entire thing as an illusion — it’s not “real” reasoning but merely a simulacrum of it. This is what I say is bullshit. »
We also have to believe that Google Translate truly understands all these languages, I presume?
Comment #80 August 20th, 2024 at 3:58 pm
@James Cross #78:
Yes, this is the normal scenario for physical systems that can implement universal computation (even if not in a reliable way); undecidable properties of computation become undecidable properties of the physical system. Of course, this doesn’t tell us how the system does its “computation”.
Comment #81 August 21st, 2024 at 12:49 pm
Ben #80
I’m not sure I get your point. At any rate, I wrote up a more extended explanation of my point that incorporates some of what I posted here.
https://broadspeculations.com/2024/08/21/does-not-compute/
Comment #82 August 22nd, 2024 at 1:19 am
The point is that even computable systems tend to have undecidable properties, so a result like Cubbitt et al. isn’t evidence against physical systems being computable.
Comment #83 August 23rd, 2024 at 4:33 am
Great interview!
I still think digital computers are cognitively crippled zombies. Implementations of classical Turing machines – likewise LLMS, etc – can’t support phenomenal binding. Phenomenal binding is our computational superpower. No binding = no mind = invincible ignorance of the entire empirical realm. In a fundamentally quantum world, decoherence makes digital computing physically feasible AND simultaneously prevents classical computers supporting minds – phenomenally-bound subjects of experience. The entire empirical (“relating to experience”) realm is computationally inaccessible to digital zombies. Compared to super-sentient full-spectrum superintelligences – i.e. our AI-augmented, genetically rewritten biological descendants – digital zombies are just toys – awesome toys, for sure, I love them, but not proto-ASIs. The ignorance of sentience of our machines is computationally hardwired.
So how do animal minds like us do it? If phenomenal binding isn’t classical, how does the CNS pull it off?
Well, as far as I can tell, we are quantum minds running classical world-simulations. Contra Roger Penrose, the superposition principle of QM doesn’t break down in the CNS or anywhere else. “Cat states” underpin our phenomenally-bound experience of classicality. Only a quantum mind can simulate a classical world (cf. https://www.hedweb.com/quora/2015.html#quantummind) Only a quantum mind can undergo the phenomenally-bound experience (“observation”) of a definite outcome (cf. the measurement problem). A pack of effectively decohered classical neurons would at most be a micro-experiential zombie, a mere aggregate of Jamesian “mind dust”.
Anyone who understands the raw power of decoherence in the warm, wet CNS will now think: crank! Assuming unitary-only QM, the effective lifetime of individual superpositions of distributed neuronal feature-processors in the CNS must (presumably) be femtoseconds or less. End of story.
Maybe. But I can’t resist pointing out that a (very) big name in neuroscience seems to have arrived at a not dissimilar conclusion about binding:
https://www.scientificamerican.com/article/experiments-prepare-to-test-whether-consciousness-arises-from-quantum/.
If nothing else, a “Schrödinger’s neurons” proposal is worth experimentally (dis)confirming.
Comment #84 August 23rd, 2024 at 5:17 am
The next-token-predictor reduction/criticism is flawed on so many level. It’s not just that it is chauvinistic and reductionist, it is a confusion between “interface” and implementation. AI models indeed generate one token at a time based on input and previous tokens – that is the interface “public String nextToken(String contextSoFar)”. However, this doesn’t reflect the complexity of the underlying mechanisms in the implementation of the interface. The term “next-token predictor” gives a reductive idea that the next token is generated from a simple local calculation with no eye to broader context and planning of response.
AI models do think ahead. While the interface requires generating one token at a time, AI’s have been trained to generate good next tokens. Doing that has forced them to develop internal structures (at present completely black box and incomprehensible to us) that do consider future tokens to ensure coherence and meaningfulness. This is crucial for generating sentences that make sense and maintaining consistency throughout longer responses. For example, in German, the verb often comes at the end of the sentence. An AI must consider this structure to generate grammatically correct sentences.
The internal planning is essential to avoid generating incomplete or nonsensical sentences.
In summary, while AI models generate responses one token at a time, they employ complex mechanisms to ensure coherence and creativity, much like humans do when speaking. When we speak, we can also only utter one “token” at a time, yet we (often) have an abstract idea of what we are going to say within the current sentence and our answer overall.
Comment #85 August 24th, 2024 at 10:54 am
It is not deflationary to compare LLMs to parrots. Parrots are computers like us and they speak our language. There’s probably common sense questions that they’re going to flub, either because they’re having a bad day or because the common sense is not so common. It seems a little bit unkind to say that they’re clearly just robots who really understand nothing. Then we get back to this problem of the double standard.
Comment #86 September 10th, 2024 at 4:01 pm
Comparing a LLM like Got-4 to a parrot is in my mind deflationary, especially considering the way the comparison is intended. It is kind of obvious it I will state it here anyway. The comparison course refers to a parrot’s seeming ability to talk but it is also clear it is just repeat verbatim what it heard. H it really is (often) just repeating with no understanding . Oftentimes it sounds like a recording it plays back including background noise! A parrot cannot make any sense of it and does not form new meaningful answers based on what it is able to say. This is clearly not the case for LLMs.
Comment #87 September 11th, 2024 at 1:40 pm
I was mostly joking. But apparently, we cannot simply dismiss the distinction between “seeming to understand” and “understanding” as meaningless, as some AI enthusiasts suggest.
Many skeptics believe that LLMs never understand what they say, while humans usually do. So far, I haven’t read a compelling argument why this view is wrong. It is certainly compatible with a Turingist perspective, since LLMs do not master the imitation game.
Comment #88 September 14th, 2024 at 3:12 pm
I think the distinction between understanding or not is important – but would LLM’s not have understanding? If you don’t have understanding and are just parroting, you are not able to apply general principles to specific situations, combine concepts, adapt knowledge from one domain to another domain etc. and LLM’s clearly ARE able to do that. I have met people who seem to genuinely think that LLM’s are just huge databases. It seems all the press coverage on the large amounts of training data required has somehow fueled this intuition and giving rise to this as the sole explanation why LLM’s work as well as they do. But clearly it is not the whole story. How can LLM’s find complicated bugs (race conditions) in MY programs which can never have been in the training data. Not only are they my own programs they are also my very own bugs. 🙂 Sure, maybe my bugs are similar to other type of bugs thatwas in the model’s training data. But the model being able to find the same bug in a completely different code base shows that it did pick up the general concept. It is not just a lookup table. If you don’t agree, how would you define ‘understanding’? Further, you can then have the LLM traverse domains – have it write lyrics (to your favorite song) or stand-up comedy routines (in the style of your favorite comedian). Sometimes with really impressive results (comedy routines in George Carlin style are usually good on gpt-4). And you can go on and throw in other choices to make the output more unique to your situation, interests and things based on random coincidences from your life etc. to make the he combinatorial explosion guarantees these answers are not just ‘looked up’. They are generated from understanding.
Comment #89 September 16th, 2024 at 2:22 pm
Morten #88:
Translation tasks are a good example to explain why you can think differently. Even human translators occasionally produce accurate translations without understanding the text. (An ambiguity can sometimes be retained in the translation so that the translator does not have to resolve it. Or a highly technical text can be linguistically simple.) However, the risk of making a mistake is higher if you don’t really understand the text, and this is probably why machine translators have not yet reached the level of human professionals. Likewise, it is at least conceivable that LLMs never understand the prompt nor their own (often correct) answers.