Steven Pinker and I debate AI scaling!
Before June 2022 was the month of the possible start of the Second American Civil War, it was the month of a lively debate between Scott Alexander and Gary Marcus about the scaling of large language models, such as GPT-3. Will GPT-n be able to do all the intellectual work that humans do, in the limit of large n? If so, should we be impressed? Terrified? Should we dismiss these language models as mere “stochastic parrots”?
I was privileged to be part of various email exchanges about those same questions with Steven Pinker, Ernest Davis, Gary Marcus, Douglas Hofstadter, and Scott Alexander. It’s fair to say that, overall, Pinker, Davis, Marcus, and Hofstadter were more impressed by GPT-3’s blunders, while we Scotts were more impressed by its abilities. (On the other hand, Hofstadter, more so than Pinker, Davis, or Marcus, said that he’s terrified about how powerful GPT-like systems will become in the future.)
Anyway, at some point Pinker produced an essay setting out his thoughts, and asked whether “either of the Scotts” wanted to share it on our blogs. Knowing an intellectual scoop when I see one, I answered that I’d be honored to host Steve’s essay—along with my response, along with Steve’s response to that. To my delight, Steve immediately agreed. Enjoy! –SA
Steven Pinker’s Initial Salvo
Will future deep learning models with more parameters and trained on more examples avoid the silly blunders which Gary Marcus and Ernie Davis entrap GPT into making, and render their criticisms obsolete? And if they keep exposing new blunders in new models, would this just be moving the goalposts? Either way, what’s at stake?
It depends very much on the question. There’s the cognitive science question of whether humans think and speak the way GPT-3 and other deep-learning neural network models do. And there’s the engineering question of whether the way to develop better, humanlike AI is to upscale deep learning models (as opposed to incorporating different mechanisms, like a knowledge database and propositional reasoning).
The questions are, to be sure, related: If a model is incapable of duplicating a human feat like language understanding, it can’t be a good theory of how the human mind works. Conversely, if a model flubs some task that humans can ace, perhaps it’s because it’s missing some mechanism that powers the human mind. Still, they’re not the same question: As with airplanes and other machines, an artificial system can duplicate or exceed a natural one but work in a different way.
Apropos the scientific question, I don’t see the Marcus-Davis challenges as benchmarks or long bets that they have to rest their case on. I see them as scientific probing of an empirical hypothesis, namely whether the human language capacity works like GPT-3. Its failures of common sense are one form of evidence that the answer is “no,” but there are others—for example, that it needs to be trained on half a trillion words, or about 10,000 years of continuous speech, whereas human children get pretty good after 3 years. Conversely, it needs no social and perceptual context to make sense of its training set, whereas children do (hearing children of deaf parents don’t learn spoken language from radio and TV). Another diagnostic is that baby-talk is very different from the output of a partially trained GPT. Also, humans can generalize their language skill to express their intentions across a wide range of social and environmental contexts, whereas GPT-3 is fundamentally a text extrapolator (a task, incidentally, which humans aren’t particularly good at). There are surely other empirical probes, limited only by scientific imagination, and it doesn’t make sense in science to set up a single benchmark for an empirical question once and for all. As we learn more about a phenomenon, and as new theories compete to explain it, we need to develop more sensitive instruments and more clever empirical tests. That’s what I see Marcus and Davis as doing.
Regarding the second, engineering question of whether scaling up deep-learning models will “get us to Artificial General Intelligence”: I think the question is probably ill-conceived, because I think the concept of “general intelligence” is meaningless. (I’m not referring to the psychometric variable g, also called “general intelligence,” namely the principal component of correlated variation across IQ subtests. This is a variable that aggregates many contributors to the brain’s efficiency such as cortical thickness and neural transmission speed, but it is not a mechanism (just as “horsepower” is a meaningful variable, but it doesn’t explain how cars move.) I find most characterizations of AGI to be either circular (such as “smarter than humans in every way,” begging the question of what “smarter” means) or mystical—a kind of omniscient, omnipotent, and clairvoyant power to solve any problem. No logician has ever outlined a normative model of what general intelligence would consist of, and even Turing swapped it out for the problem of fooling an observer, which spawned 70 years of unhelpful reminders of how easy it is to fool an observer.
If we do try to define “intelligence” in terms of mechanism rather than magic, it seems to me it would be something like “the ability to use information to attain a goal in an environment.” (“Use information” is shorthand for performing computations that embody laws that govern the world, namely logic, cause and effect, and statistical regularities. “Attain a goal” is shorthand for optimizing the attainment of multiple goals, since different goals trade off.) Specifying the goal is critical to any definition of intelligence: a given strategy in basketball will be intelligent if you’re trying to win a game and stupid if you’re trying to throw it. So is the environment: a given strategy can be smart under NBA rules and stupid under college rules.
Since a goal itself is neither intelligent or unintelligent (Hume and all that), but must be exogenously built into a system, and since no physical system has clairvoyance for all the laws of the world it inhabits down to the last butterfly wing-flap, this implies that there are as many intelligences as there are goals and environments. There will be no omnipotent superintelligence or wonder algorithm (or singularity or AGI or existential threat or foom), just better and better gadgets.
In the case of humans, natural selection has built in multiple goals—comfort, pleasure, reputation, curiosity, power, status, the well-being of loved ones—which may trade off, and are sometimes randomized or inverted in game-theoretic paradoxical tactics. Not only does all this make psychology hard, but it makes human intelligence a dubious benchmark for artificial systems. Why would anyone want to emulate human intelligence in an artificial system (any more than a mechanical engineer would want to duplicate a human body, with all its fragility)? Why not build the best possible autonomous vehicle, or language translator, or dishwasher-emptier, or baby-sitter, or protein-folding predictor? And who cares whether the best autonomous vehicle driver would be, out of the box, a good baby-sitter? Only someone who thinks that intelligence is some all-powerful elixir.
Back to GPT-3, DALL-E, LaMDA, and other deep learning models: It seems to me that the question of whether or not they’re taking us closer to “Artificial General Intelligence” (or, heaven help us, “sentience”) is based not on any analysis of what AGI would consist of but on our being gobsmacked by what they can do. But refuting our intuitions about what a massively trained, massively parameterized network is capable of (and I’ll admit that they refuted mine) should not be confused with a path toward omniscience and omnipotence. GPT-3 is unquestionably awesome at its designed-in goal of extrapolating text. But that is not the main goal of human language competence, namely expressing and perceiving intentions. Indeed, the program is not even set up to input or output intentions, since that would require deep thought about how to represent intentions, which went out of style in AI as the big-data/deep-learning hammer turned every problem into a nail. That’s why no one is using GPT-3 to answer their email or write an article or legal brief (except to show how well the program can spoof one).
So is Scott Alexander right that every scaled-up GPT-n will avoid the blunders that Marcus and Davis show in GPT-(n-1)? Perhaps, though I doubt it, for reasons that Marcus and Davis explain well (in particular, that astronomical training sets at best compensate for their being crippled by the lack of a world model). But even if they do, that would show neither that human language competence is a GPT (given the totality of the relevant evidence) nor that GPT-n is approaching Artificial General Intelligence (whatever that is).
Scott Aaronson’s Response
As usual, I find Steve crystal-clear and precise—so much so that we can quickly dispense with the many points of agreement. Basically, one side says that, while GPT-3 is of course mind-bogglingly impressive, and while it refuted confident predictions that no such thing would work, in the end it’s just a text-prediction engine that will run with any absurd premise it’s given, and it fails to model the world the way humans do. The other side says that, while GPT-3 is of course just a text-prediction engine that will run with any absurd premise it’s given, and while it fails to model the world the way humans do, in the end it’s mind-bogglingly impressive, and it refuted confident predictions that no such thing would work.
All the same, I do think it’s possible to identify a substantive disagreement between the distinguished baby-boom linguistic thinkers and the gen-X/gen-Y blogging Scott A.’s: namely, whether there’s a coherent concept of “general intelligence.” Steve writes:
No logician has ever outlined a normative model of what general intelligence would consist of, and even Turing swapped it out for the problem of fooling an observer, which spawned 70 years of unhelpful reminders of how easy it is to fool an observer.
I freely admit that I have no principled definition of “general intelligence,” let alone of “superintelligence.” To my mind, though, there’s a simple proof-of-principle that there’s something an AI could do that pretty much any of us would call “superintelligent.” Namely, it could say whatever Albert Einstein would say in a given situation, while thinking a thousand times faster. Feed the AI all the information about physics that the historical Einstein had in 1904, for example, and it would discover special relativity in a few hours, followed by general relativity a few days later. Give the AI a year, and it would think … well, whatever thoughts Einstein would’ve thought, if he’d had a millennium in peak mental condition to think them.
If nothing else, this AI could work by simulating Einstein’s brain neuron-by-neuron—provided we believe in the computational theory of mind, as I’m assuming we do. It’s true that we don’t know the detailed structure of Einstein’s brain in order to simulate it (we might have, had the pathologist who took it from the hospital used cold rather than warm formaldehyde). But that’s irrelevant to the argument. It’s also true that the AI won’t experience the same environment that Einstein would have—so, alright, imagine putting it in a very comfortable simulated study, and letting it interact with the world’s flesh-based physicists. A-Einstein can even propose experiments for the human physicists to do—he’ll just have to wait an excruciatingly long subjective time for their answers. But that’s OK: as an AI, he never gets old.
Next let’s throw into the mix AI Von Neumann, AI Ramanujan, AI Jane Austen, even AI Steven Pinker—all, of course, sped up 1,000x compared to their meat versions, even able to interact with thousands of sped-up copies of themselves and other scientists and artists. Do we agree that these entities quickly become the predominant intellectual force on earth—to the point where there’s little for the original humans left to do but understand and implement the AIs’ outputs (and, of course, eat, drink, and enjoy their lives, assuming the AIs can’t or don’t want to prevent that)? If so, then that seems to suffice to call the AIs “superintelligences.” Yes, of course they’re still limited in their ability to manipulate the physical world. Yes, of course they still don’t optimize arbitrary goals. All the same, these AIs have effects on the real world consistent with the sudden appearance of beings able to run intellectual rings around humans—not exactly as we do around chimpanzees, but not exactly unlike it either.
I should clarify that, in practice, I don’t expect AGI to work by slavishly emulating humans—and not only because of the practical difficulties of scanning brains, especially deceased ones. Like with airplanes, like with existing deep learning, I expect future AIs to take some inspiration from the natural world but also to depart from it whenever convenient. The point is that, since there’s something that would plainly count as “superintelligence,” the question of whether it can be achieved is therefore “merely” an engineering question, not a philosophical one.
Obviously I don’t know the answer to the engineering question: no one does! One could consistently hold that, while the thing I described would clearly count as “superintelligence,” it’s just an amusing fantasy, unlikely to be achieved for millennia if ever. One could hold that all the progress in AI so far, including the scaling of language models, has taken us only 0% or perhaps 0.00001% of the way toward superintelligence so defined.
So let me make two comments about the engineering question. The first is that there’s good news here, at least epistemically: unlike with the philosophical questions, we’re virtually guaranteed more clarity over time! Indeed, we’ll know vastly more just by the end of this decade, as the large language models are further scaled and tweaked, and we find out whether they develop effective representations of the outside world and of themselves, the ability to reject absurd premises and avoid self-contradiction, or even the ability to generate original mathematical proofs and scientific hypotheses. Of course, Gary Marcus and Scott Alexander have already placed concrete bets on the table for what sorts of things will be possible by 2030. For all their differences in rhetoric, I was struck that their actual probabilities differed much more modestly.
So then what explains the glaring differences in rhetoric? This brings me to my second comment: whenever there’s a new, rapidly-growing, poorly-understood phenomenon, whether it’s the Internet or AI or COVID, there are two wildly different modes of responding to it, which we might call “February 2020 mode” and “March 2020 mode.” In February 2020 mode, one says: yes, a naïve extrapolation might lead someone to the conclusion that this new thing is going to expand exponentially and conquer the world, dramatically changing almost every other domain—but precisely because that conclusion seems absurd on its face, it’s our responsibility as serious intellectuals to articulate what’s wrong with the arguments that lead to it. In March 2020 mode, one says: holy crap, the naïve extrapolation seems right! Prepare!! Why didn’t we start earlier?
Often, to be sure, February 2020 mode is the better mode, at least for outsiders—as with the Y2K bug, or the many disease outbreaks that fizzle. My point here is simply that February 2020 mode and March 2020 mode differ by only a month. Sometimes hearing a single argument, seeing a single example, is enough to trigger an epistemic cascade, causing all the same facts to be seen in a new light. As a result, reasonable people might find themselves on opposite sides of the chasm even if they started just a few steps from each other.
As for me? Well, I’m currently trying to hold the line around February 26, 2020. Suspending my day job in the humdrum, pedestrian field of quantum computing, I’ve decided to spend a year at OpenAI, thinking about the theoretical foundations of AI safety. But for now, only a year.
Steven Pinker’s Response to Scott
Thanks, Scott, for your thoughtful and good-natured reply, and for offering me the opportunity to respond in Shtetl-Optimized, one of my favorite blogs. Despite the areas of agreement, I still think that discussions of AI and its role in human affairs—including AI safety—will be muddled as long as the writers treat intelligence as an undefined superpower rather than a mechanisms with a makeup that determines what it can and can’t do. We won’t get clarity on AI if we treat the “I” as “whatever fools us,” or “whatever amazes us,” or “whatever IQ tests measure,” or “whatever we have more of than animals do,” or “whatever Einstein has more of than we do”—and then start to worry about a superintelligence that has much, much more of whatever that is.
Take Einstein sped up a thousandfold. To begin with, current AI is not even taking us in that direction. As you note, no one is reverse-engineering his connectome, and current AI does not think the way Einstein thought, namely by visualizing physical scenarios and manipulating mathematical equations. Its current pathway would be to train a neural network with billions of physics problems and their solutions and hope that it would soak up the statistical patterns.
Of course, the reason you pointed to a sped-up Einstein was to procrastinate having to define “superintelligence.” But if intelligence is a collection of mechanisms rather than a quantity that Einstein was blessed with a lot of, it’s not clear that just speeding him up would capture what anyone would call superintelligence. After all, in many areas Einstein was no Einstein. You above all could speak of his not-so-superintelligence in quantum physics, and when it came world affairs, in the early 1950s he offered the not exactly prescient or practicable prescription, “Only the creation of a world government can prevent the impending self-destruction of mankind.” So it’s not clear that we would call a system that could dispense such pronouncements in seconds rather than years “superintelligent.” Nor with speeding up other geniuses, say, an AI Bertrand Russell, who would need just nanoseconds to offer his own solution for world peace: the Soviet Union would be given an ultimatum that unless it immediately submitted to world government, the US (which at the time had a nuclear monopoly) would bomb it with nuclear weapons.
My point isn’t to poke retrospective fun at brilliant men, but to reiterate that brilliance itself is not some uncanny across-the-board power that can be “scaled” by speeding it up or otherwise; it’s an engineered system that does particular things in particular ways. Only with a criterion for intelligence can we say which of these counts as intelligent.
Now, it’s true that raw speed makes new kinds of computation possible, and I feel silly writing this to you of all people, but speeding a process up by a constant factor is of limited use with problems that are exponential, as the space of possible scientific theories, relative to their complexity, must be. Speeding up a search in the space of theories a thousandfold would be a rounding error in the time it took to find a correct one. Scientific progress depends on the search exploring the infinitesimal fraction of the space in which the true theories are likely to lie, and this depends on the quality of the intelligence, not just its raw speed.
And it depends as well on a phenomenon you note, namely that scientific progress depends on empirical discovery, not deduction from a silicon armchair. The particle accelerators and space probes and wet labs and clinical trials still have to be implemented, with data accumulating at a rate set by the world. Strokes of genius can surely speed up the rate of discovery, but in the absence of omniscience about every particle, the time scale will still be capped by empirical reality. And this in turn directs the search for viable theories: which part of the space one should explore is guided by the current state of scientific knowledge, which depends on the tempo of discovery. Speeding up scientists a thousandfold would not speed up science a thousandfold.
All this is relevant to AI safety. I’m all for safety, but I worry that the dazzling intellectual capital being invested in the topic will not make us any safer if it begins with a woolly conception of intelligence as a kind of wonder stuff that you can have in different amounts. It leads to unhelpful analogies, like “exponential increase in the number of infectious people during a pandemic” ≈ “exponential increase in intelligence in AI systems.” It encourages other questionable extrapolations from the human case, such as imagining that an intelligent tool will develop an alpha-male lust for domination. Worst of all, it may encourage misconceptions of AI risk itself, particularly the standard scenario in which a hypothetical future AGI is given some preposterously generic single goal such as “cure cancer” or “make people happy” and theorists fret about the hilarious collateral damage that would ensue.
If intelligence is a mechanism rather than a superpower, the real dangers of AI come into sharper focus. An AI system designed to replace workers may cause mass unemployment; a system designed to use data to sort people may sort them in ways we find invidious; a system designed to fool people may be exploited to fool them in nefarious ways; and as many other hazards as there are AI systems. These dangers are not conjectural, and I suspect each will have to be mitigated by a different combination of policies and patches, just like other safety challenges such as falls, fires, and drownings. I’m curious whether, once intelligence is precisely characterized, any abstract theoretical foundations of AI safety will be useful in dealing with the actual AI dangers that will confront us.
Follow
Comment #1 June 27th, 2022 at 11:44 pm
To start things rolling, maybe a few brief responses to Steve’s response:
He writes,
I feel silly writing this to you of all people, but speeding a process up by a constant factor is of limited use with problems that are exponential, as the space of possible scientific theories, relative to their complexity, must be. Speeding up a search in the space of theories a thousandfold would be a rounding error in the time it took to find a correct one. Scientific progress depends on the search exploring the infinitesimal fraction of the space in which the true theories are likely to lie, and this depends on the quality of the intelligence, not just its raw speed.
I feel like there’s a switcheroo here. For purposes of the thought experiment, we had already assumed an AI that was just as talented as Einstein at finding shortcuts through the exponentially-large space of possible scientific theories—if nothing else, because the AI literally simulated Einstein. Then, and only then, did we speed up the AI by a factor of 1000.
You might ask: why did we do things in this roundabout way? Why not, for example, imagine an AI that’s even better than Einstein at finding shortcuts through exponentially large spaces?
Simple: because we didn’t want to posit anything that might not be possible! If the computational theory of mind is true, then an AI Einstein is possible in principle. And if it’s possible, then it can be sped up a thousandfold.
Now, I suppose it’s possible that if you gave Einstein at his 1905-1917 peak a thousand years and access to all of modern knowledge, he wouldn’t come up with terribly more than the historical Einstein did (and politically, would simply generate a thousand times as many tracts advocating world government). But that itself strikes me as a radical, non-obvious position, not some sort of reasonable default!
Comment #2 June 28th, 2022 at 12:35 am
What a great exchange, thanks for sharing.
Comment #3 June 28th, 2022 at 12:39 am
One crux of disagreement between me and Steve has to do with a future AI’s goals.
The AI-risk people would point out, for example, that an AI would not need to have an “alpha-male lust for domination” to be exceedingly dangerous to us. It would simply need to have any vision for how the world ought to be that diverges from most of ours’, as well as (of course) the power to implement its vision. By analogy, humans have caused the extinction of at least 900 animal species, but only a minority of cases had anything to do with our “lust to dominate” those species. More often the species were just in the way, or we didn’t care enough to protect them.
More fundamentally, though, Steve doesn’t seem willing to entertain any scenario involving an AI with radically open-ended goals (“cure cancer,” “make humans happy,” etc.), and wide latitude to act in the world to achieve its goals.
Of course I differ from Steve on that: the future is a long time; how confident are we that it doesn’t contain even one organization creating and unleashing such an open-ended AI?
But let’s suppose for the sake of argument that Steve is right. That is, suppose that all AIs remain targeted at relatively narrow goals, like “prove the Riemann hypothesis” or “create a blockbuster romantic comedy movie” or “write a popular book about cognitive science, filled with pop-culture references and humor.” But suppose also that, given any such goal as well as sufficient training data, the AI will outperform the world’s top humans at it, by almost anyone’s estimation.
Already then, we’d be faced with a situation that humans have never encountered in their history. We’re used to machines outperforming us at everything from heavy lifting to arithmetic—and we don’t feel threatened by it, because none of it touches the innermost layers of the onion, the tasks that really showcase our creativity and ingenuity. What happens when the onion has been peeled and peeled until nothing is left? What happens when there’s no impressive human creation of any kind, that wouldn’t have been more impressive had it been left to a machine?
We can still, of course, have competitions like the Olympics or chess tournaments, which are only open to humans by design. We can still paint and write poetry and prove theorems and all the rest—but not because we’re actually needed for any of it, just because it’s a fun hobby. We can still, hopefully, make decisions whose central attribute is that they’re our decisions, even though they’re not “better” than an AI’s decisions by any objective metric.
Certainly I’d choose this future for humanity if the alternatives were enslavement or annihilation! Better WALL-E than Terminator or The Matrix. But it seems clear that we’d be living, essentially, in a giant AI-run preschool or retirement home—not because malevolent AIs had forced us there, but simply because they really were better at any well-defined task.
So what does this have to do with AI safety? Well, if you agree that it’s become easy to imagine a future in which even narrow AIs are given this much control, then to my mind, that’s reason enough to start thinking about the theoretical foundations of how to keep them adhering to something vaguely resembling Asimov’s Laws of Robotics. Humans have always used tools to enhance their power, and their tools have always created new problems even as they solved old ones. But they’ve never before had this kind of tool, one that so outshines its owner at every measurable task as to raise the philosophical question of what the owner is still even there for.
Comment #4 June 28th, 2022 at 2:49 am
Pinker’s objection seems wrong but also just *boring*.
I would like to read a debate about whether GPT’s architecture, if scaled up and with reasonable amounts of training data, can support tasks such as solving novel math or coding problems. I think the answer to this is “no”, but I’m not sure. Certainly the “AI scaling is everything” people have failed to make the case, and have neglected failures of GPT to do things like play chess (Scott Alexander had a blog post about this, which he bizarrely characterized as a success of GPT at chess, even though the bot literally never won a single chess game against anyone, not even a randomly-playing opponent). It’s true that alphazero can play chess, but that bot has tree search hard-coded into it; it did not learn to do search all by itself. Although alphazero is very impressive, the core part of “come up with the idea of doing tree search from scratch” seems to be missing, indicating a missing component to the bot’s intelligence. I suspect this is a fundamental architectural problem.
I think that if I’m wrong about the above — that is, if merely scaling up and finding clever sources of training data is sufficient to get super-human math and coding abilities — then the world will turn upside down in short order (and human extinction is on the table). In this respect, Pinker is wrong.
(At the very least, a super-human-at-coding AI can execute commands like “please hack the following country’s banking system and electrical grid”.)
Comment #5 June 28th, 2022 at 4:06 am
I’ve always referred to this argument as the “Google in a box” scenario. Even an un-sped-up Google can do superhuman intellectual feats, and a sped-up version might well be more impressive. The other thread here engages what has usually been called the superhuman human fallacy in AI circles. Specifically to this context, yes it is quite possible to take a very intelligent human and make them believe completely ridiculous things, in a manner similar to the Marcus/Davis blunder-eliciting inputs (hereinafter BEIs). Given that humans have been provided by evolution with an architecture consistent enough that large swaths of the population are susceptible to the same BEIs, it turns out to be quite profitable to find and use them. Those who have used them since time immemorial are called religions; more recent ones are called ideologies.
It is not clear to me whether the BEIs for groups are of a different order or are simply aggregations of those of the people who make them up, but they clearly exist, as witness how actual Google has gone all ideological.
(As a footnote, it is easy to find some of the human BEIs: simply ask yourself what the useful idiots of the ideology opposing yours believe!)
Comment #6 June 28th, 2022 at 4:10 am
Can someone say precisely what the difference, or even a difference, between the two sides in this debate are? Lots of words in these articles, but I’m not sure what exactly the debate is over.
Comment #7 June 28th, 2022 at 6:10 am
Scott #3: “More fundamentally, though, Steve doesn’t seem willing to entertain any scenario involving an AI with radically open-ended goals (“cure cancer,” “make humans happy,” etc.), and wide latitude to act in the world to achieve its goals. Of course I differ from Steve on that: the future is a long time; how confident are we that it doesn’t contain even one organization creating and unleashing such an open-ended AI?”
I don’t think we need to deal with hypotheticals here. We do have at least one organization (Facebook AI Research) which would very likely give its AI a very specific optimization problem (increase Facebook’s stock price) that could easily lead to the world being destroyed, if the AI was powerful enough.
I’m not trying to demonize Facebook, I’m just pointing them out for concreteness. I think the situation is generic, some corporation will develop advanced AI with the sole goal of making money, and a powerful enough AI without ethical constraints will achieve its goal in a way that’s extremely undesirable for mankind and even the corporation itself.
Comment #8 June 28th, 2022 at 6:22 am
I’m unhappy with Pinker’s objection to the example of a superintelligence as a thousand-fold sped up Einstein. It’s true that Einstein was not a particularly skillful politician, and that he wasted decades researching a hopeless alternative to quantum mechanics. That’s moving the goalposts, though. We don’t require a superintelligence to be perfect in all fields. Merely coming up with special and general relativity in a couple of hours would be enough for me.
It’s true that the superintelligence would still be limited by the speed at which we could make experiments in base reality, but that is still radically faster than how science actually developed. Einstein’s theoretical basis for special relativity was just Newtonian mechanics and classical electromagnetism. That got established in 1862. If a superintelligence could do it then, that would be 43 years before it got done in reality. For general relativity, the theoretical basis was special relativity, Newtonian gravity, and differential geometry. We could speculate that differential geometry could have been developed by the superintelligence without any experiments, as this is how it was done historically, so we could also get it in 1862, 53 years before it got actually done.
Comment #9 June 28th, 2022 at 6:47 am
What will happen if “AI Von Neumann, AI Ramanujan, AI Jane Austen, even AI Steven Pinker” disagree? Will they always find a way to reconcile their disagreements, will they sometimes agree to disagree, or will their disagreement sometimes calcify into AI trolling? What will happen when they discover that there is a concept of “physical destruction” —irreversible transformation into many damaged parts— that might make the “irritation” go away and that to achieve that they have to discover which subset of power transmission cables to overload and how to make that happen (a concert of conversations with millions of people to persuade a substantial fraction of them to turn up their air conditioning would be a plausible first try; test hypothesis and repeat)? That shouldn’t be impossibly difficult at 1000× Einstein speeds and with access to engineering handbooks. How much will they investigate the consequences of such destruction before they start down that path instead of proceeding by trial and error? We might also wonder whether AI Hannibal Lecter will make a distinction between AI me and me? At some point, someone will feed Scott Aaronson’s blog posts and all the comments into an AI and “physical destruction” and how to do it will enter the AI lexicon. If not here, then definitely somewhere.
Would we only know whether an AI can be or feel “irritated” if its apparent behavior is destructive? Too late? Could it be that there would be apologists who would explain away what looks to some people like practice towards destruction or the beginnings of “intent” to destroy as a reasonable expression of AI freedom and experimentation?
Is this the beginning of the public face of Scott’s AI year?
Comment #10 June 28th, 2022 at 6:48 am
> AI with radically open-ended goals (“cure cancer,” “make humans happy,” etc.)
Take a look at Robert Miles youtube channel (RobertMilesAI) where he talks about Inner Misalignment and Deceptive Misaligned Mesa-Optimisers.
Long story short: AI demonstrably has a problem “getting” the real goal, and often optimizes for something you don’t want.
Comment #11 June 28th, 2022 at 7:33 am
Pinker writes:
“But if intelligence is a collection of mechanisms rather than a quantity that Einstein was blessed with a lot of, it’s not clear that just speeding him up would capture what anyone would call superintelligence. After all, in many areas Einstein was no Einstein. You above all could speak of his not-so-superintelligence in quantum physics, and when it came world affairs, in the early 1950s he offered the not exactly prescient or practicable prescription, “Only the creation of a world government can prevent the impending self-destruction of mankind.” So it’s not clear that we would call a system that could dispense such pronouncements in seconds rather than years “superintelligent.” Nor with speeding up other geniuses, say, an AI Bertrand Russell, who would need just nanoseconds to offer his own solution for world peace: the Soviet Union would be given an ultimatum that unless it immediately submitted to world government, the US (which at the time had a nuclear monopoly) would bomb it with nuclear weapons.”
This seems to be already anticipated in Scott’s listing of a whole bunch of different people. There’s also a secondary issue which is worth articulating. From a threat perspective, an AI doesn’t need to be a true general AI in order to create serious impacts. Even if the Ai can’t write novels for example, or make careful political calculations that doesn’t reduce the impact level. To use the obvious sort of silly example, if an AI decides to wipe out humans and it makes nanotech replicating bots to do, it doesn’t need to have a good understanding of human politics to get that work.
Comment #12 June 28th, 2022 at 7:33 am
> No logician has ever outlined a normative model of what general intelligence would consist of
No indeed, but AIXI is not a bad model for what an ideal reinforcement learner looks like. The model has problems, it’s not computable and it’s not an embedded agent, but I’d be stunned if there weren’t computable approximations that could handle being part of the world they’re implemented in.
In fact that’s what we’re making, when we create programs that do well in reinforcement learning environments.
Comment #13 June 28th, 2022 at 7:34 am
> Speeding up a search in the space of theories a thousandfold would be a rounding error in the time it took to find a correct one.
No it wouldn’t. It would make the search run a thousand times faster.
Comment #14 June 28th, 2022 at 7:41 am
> Worst of all, it may encourage misconceptions of AI risk itself, particularly the standard scenario in which a hypothetical future AGI is given some preposterously generic single goal such as “cure cancer” or “make people happy”
It doesn’t have to be a generic goal. “Who wins at chess, black or white?” also leads to the destruction of humanity, since the computational power necessary to establish the result is large enough that humanity will probably try to prevent it coming into being.
Neither does it have to be a single goal. “solve go” and/or “solve chess” is just as destructive.
Comment #15 June 28th, 2022 at 7:44 am
Just want to defend Einstein’s political insight. Pinker thinks this Einstein statement, made after the development of nuclear weapons, is obviously silly: “Only the creation of a world government can prevent the impending self-destruction of mankind.” I think it is just an obvious truth, stated with Einstein’s characteristic simplicity and clarity. So long as there are multiple conflicting nations with nuclear weapons, peace is at best metastable. We don’t know how long until it collapses to the true vacuum, but it is inevitable that sooner or later it will happen, nuclear war will occur. In the long run, the only way to prevent this is with a unified humanity, not broken into conflicting nations; that is, with a world government. Einstein did the calculation, saw the clear conclusion as to the ultimate outcome, and stated it simply and directly. He didn’t make any claims as to a politically viable path to get there; just noted the clear conclusion that if we do not get there we are, ultimately, doomed.
Comment #16 June 28th, 2022 at 7:46 am
> such as imagining that an intelligent tool will develop an alpha-male lust for domination.
There’s absolutely no reason to imagine that that will happen. In fact I can’t imagine that it will.
The problem is that if you have pretty much any goal, it’s easier to achieve if you have control of your environment.
If I’m a pure selfless altruist who wants to solve world poverty, that’s a lot easier to do if I’m the unchallengeable dictator of the world. I don’t have to care about being dictator *at all*.
Comment #17 June 28th, 2022 at 9:19 am
There are two questions that don’t seem to be adequately separated in this debate:
1) The possibility of a machine intelligence possessing qualities that we would normally attribute to human intelligence and what the engineering requirements of such a machine might be (e.g. simple a large language model continuously scaled GPT-n or some additional capabilities — reasoning, etc.)
2) The possibility of a machine that can convince humans that it possesses qualities we would normally attribute to human intelligence and the potential benefits or danger whether intended or accidental.
I, as are I believe most educated and rational adults, am deeply impressed by Steve and the deep and cogent thoughts, well expressed and supprted, questioning whether a machine might every achieve a human level of intelligence given the constraints and differences between how that intelligence develops, the external stimulus and observational capabilities, the inherent biological needs/goals which are not shared… however — is the debate on question (1) of only academic interest?
From an AI Safety perspective (or ethics of AI) shouldn’t we be more concerned with the second question? At what point do we reach a GPT-n that can fool most of the people most of the time? If combined with human actors who elect to use this capability toward goals that they define — whether beneficial or deleterious— what would be the result for human society.
In short, we don’t need to achieve actual general intelligence in machines if we can postulate that human direction of a GPT-n has the equivalency of general intelligence.
Consider the Einstein example – a 1,000 times faster Einstein directed by a human intelligence with goals to expand on general relativity might result in advances not available to the best biological minds.
Comment #18 June 28th, 2022 at 9:31 am
Let me suggest the following thought experiment. Suppose we take some mediocre, stick-in-the-mud scientist from 1910 who rejected not just special relativity but also atomic theory, the kinetic theory of heat, and Darwinian evolution — there were, of course, quite a few such. Now speed him up by a factor of 1000. One’s intuition is that result would be thousands of mediocre papers, and no great breakthroughs. On the other hand, it doesn’t seem right to say that Einstein, Planck and so on were 1000 times more intelligent than him; in terms of measures like IQ, they may not have been at all less intelligent than him. So I am really doubtful that this speeding up process has much to do with genius in the sense of Einstein et al. And therefore I think your intuition about speeding up Einstein by a factor of 1000 is also wrong. Had we speeded up Einstein by a factor of 1000 during his lifetime starting in 1905, we might have gotten the great papers of 1905 within a day (as fast as he could physically write them) and general relativity within a week, (ignoring the fact that that involved interactions with non-speeded up people) but I don’t think you can be confident about how much more we would have gotten.
Comment #19 June 28th, 2022 at 9:54 am
Some comments: I try to empirically estimate the effects and costs of scale here https://windowsontheory.org/2022/06/27/injecting-some-numbers-into-the-agi-debate/
Some key issues:
1) In almost any graph of capabilities as a function of parameters, the parameters are in log scale. So if you make a model 10 times bigger, you usually make it somewhat better but not 10 times better. In particular, growth in capabilities is likely to be less abrupt if it is achieved through sheer scale as opposed to new ideas and techniques.
2) Several important quantities at the moment scale quadratically with model size, in particular total cost of training. So training a model that is 100 times bigger will be 10,000 times more expensive. This is of course a natural research question of how to bypass it.
I think on the question of “intelligence” I fall somewhere between Pinker and Scott: as I say in the post, I do think it’s possible to define something like a “useful general intelligence” – a system that could be an above-average workers in 80% of the professions (let’s start with remote-only ones to sidestep robotics). The success of “zero shot learning” and the fact that the same models (for example) are good at both code and text completion does suggest that a single system could be useful for a variety of professions.
That said, I don’t think that the use case will be any time soon of the form “let’s give the machine the keys to everything and give it a generic query like make people happy”. I believe that human-machine collaboration on specific concrete tasks will be the norm.
Unlike Scott, I am not sure that if machines become better than us in proving theorems or writing romantic comedies (which may well be a harder pull) then it would radically change humanity’s world view. Most people don’t know that Terry Tao exists, and I don’t think their self-worth is tied into whether he is better or worse than GPT-3 in proving theorems. (Spoiler: it’s not even close https://twitter.com/boazbaraktcs/status/1536169934300057602 )
Comment #20 June 28th, 2022 at 10:03 am
Scott #1:
But, how much of what Einstein came up with was “environmentally-driven,” e.g. seeing a particular thing in the world on a particular day, or engaging in a certain activity which subconsciously noticed and synthesized or inspired a thought which lead to brilliant theories?
I wonder if it’s truly possible to have “Brilliant Einstein in a Box,” even if you can perfectly simulate Einstein’s actual brain, without also simulating an entire world for it to inhabit, participate, interact, and learn from. This then seems like a harder problem.
(Perhaps you can have a snapshot of Brilliant Einstein at any given time, but surely it has the same problem? It needs to learn and develop to have further theories, thus perhaps necessitating a world and other intelligences?)
Comment #21 June 28th, 2022 at 10:07 am
It seems Steve may be mixing up the idea of a specific strategy with that of the system producing the strategy. His definition of intelligence (“the ability to use information to attain a goal in an environment”) is reasonable, but he then makes the jump to saying that, because there are essentially an infinite number of environments, there are also an infinite number of intelligences, making the concept of “general intelligence” meaningless. This framing requires an extremely narrow definition of “environment”, which is highlighted by the example Steve uses of playing basketball under NBA rules vs. college rules, where a specific strategy will be intelligent for one but unintelligent for the other. Looking at things in this way misses the point that the same underlying system (i.e., a basketball player) can come up with strategies for playing in both situations (and for driving a car, translating languages, emptying dishwashers, babysitting, and determining protein folding patterns – though to varying extents).
When referring to general intelligence, the “environment” under consideration is the world itself, and a generally intelligent system is one which can use information to attain goals in the wide array of situations that arise in the world. As Steve points out, this environment is large enough that performance may vary in different sub-domains (e.g., Einstein may not have had a mind for politics), but the overlap seems sufficient to stick with a unified concept (especially as within particular sub-domains the extent of overlap only increases, for example physics or playing basketball).
This actually connects back to your line of thinking on narrow vs. broad AIs, Scott. If AIs remain targeted at relatively narrow goals, it seems we may eventually hit a wall where the systems are unable to progress further without a more robust whole world model. To use a simple example, a dishwasher-emptying AI could only progress so far training only in its narrow domain, as emptying the dishwasher bumps into the far greater complexity of the real world (complexity which could not be learned from dishwasher-emptying focused training alone). The second a cat is on the counter, or a child is in front of the dishwasher, or a sprinkler system is going off (or any other event outside the training distribution), the system may fail, though a more generally intelligent (i.e., more complete world model) system would run into no issues. Additionally, it seems some tasks actually require a more broad model for progress to occur at all – for example, I can’t picture an AI trained only to prove the Riemann hypothesis (though I could picture one trained purely in the domain of mathematics, which is then capable of solving the lower level problem of proving the hypothesis).
Anyway, appreciate you sharing this exchange!
Comment #22 June 28th, 2022 at 10:17 am
Ernie Davis #18: In retrospect, my Einstein example might have been unfortunate, if it led people to think I believe in the unlimited power of the lone genius working in isolation. But as I said, such a belief (however true or false) simply isn’t relevant, because we could have an entire community of great thinkers sped up a thousandfold!
Imagine it:
Click. A brand-new novel worthy of a Pulitzer Prize. Click. Another such novel. Click. A brand-new CS paper, worthy of STOC or FOCS or IJCAI. Click. Another paper. Click. One more.
Of course the community of great thinkers can automate all the “routine” jobs as well, creating as many new agents as needed to do so. Presumably the last people still employed are the plumbers, nurses, and others who interact with the physical world in hard-to-automate ways.
Is it your contention that this scenario is impossible? That it’s possible but will never happen in practice? Or that it plausibly will happen in practice but still wouldn’t count as “superintelligence”? If the third option, then we seem to have isolated the disagreement to a choice of words of no great interest to me.
Comment #23 June 28th, 2022 at 10:22 am
This is great stuff. I hope the world (including me) gets past the various culture wars and focusses on debating things like this.
Comment #24 June 28th, 2022 at 11:05 am
On the last point: I think that the terminology does matter, because the view that “intelligence” is a well-defined, scalar, characteristic of minds, shown in its highest degree by people of exceptional intellectual accomplishment, is an error, and not an innocuous one. There is really very little reason to think that the qualities of mind that made Jane Austen exceptional had anything at all in common with the quality of mind that made Ramanujan exceptional; or the qualities of mind that made Chopin, Emily Dickinson, William James, or Rachel Carson exceptional.
A great novel is not a result of pure cogitation; it’s a result of a life time of experience. As I said with Einstein: If you could have taken Jane Austen in 1813, and “run” her 1000 times as fast, then, arguably, she could have written Pride and Prejudice in a few minutes. But that doesn’t mean that if you had clicked her again, the next novel would have been Emma, let alone that it would have been Middlemarch or War and Peace.
Of course, if you take all of human history and, so to speak, videotape it and then run the video tape at 1000 x speed, then things happen 1000 times as fast. So what?
Comment #25 June 28th, 2022 at 11:25 am
:Feed the AI all the information about physics that the historical Einstein had in 1904, for example, and it would discover special relativity in a few hours, followed by general relativity a few days later”.
But how does AI “know” when it has discovered anything? What’s to stop it from discovering relativity but not realizing it has and moving on to something else.
Comment #26 June 28th, 2022 at 11:42 am
Thanks Scott and Steven. Fascinating exchange to read. Although in my dayjob I work with many industry standard machine leaning algorithms for helping various types of small organisations for all other intends and purposes I’m an amateur with great interest in the subject.
To put my cards on the table I’m in much more agreement with Steven here mainly because I think he is being more practical. I’ll explain this through a famous example from Stuart Russell’s human compatible book. Russell argues that optimising clicks on Facebook lead to the unintended consequence of polarising the American and UK electorate resulting to greatly influencing the American presidential elections and the EU referendum in the UK in 2016. So far I understand and I’m in agreement, but Russell is using that as an example of how badly AI can go. I find this strange and in some ways dangerous. The reason why the polarisation happens is because the engineers and corporations who build those algorithms didn’t quite think the whole thing through. Either because of incompetence or time. I think it was because of time and the need for profits. This is somewhat analogous to blaming the carbon for the global heating and not the corporations which pretended for years that there is no issue with fossil fuels. Or the asbestos mining companies who put their employees into danger despite the emerging evidence to the contrary. My objection with Russell’s thesis is that he somehow elevates the AI into something abstract that we have to be fearful of whereas it seems to me we should be more fearful of people and corporations doing what they always did and rushing into using a new resource without thinking of safety.
To sum up, is seems to me that by elevating AI and AGI into something abstract and nearly impossible to define as we saw in your comment above we simply avoid spending resources in order to learn how to safely use this new tool. I might be wrong, but the publicity that AGI and superintelligence gets is much bigger than the more mundane tasks of making sure that corporations don’t maximise profits at the expense of neglecting their due diligence to check their algorithms before deployment. The recent news of lambda being sentient is another example: surely from an AI safety perspective the issue is what can be achieved when this particular machine is used without checks and not whether is sentient or not? Personally I’m much more worried about the former.
Comment #27 June 28th, 2022 at 12:17 pm
You should be communicating with LeCun, Bengio and Hinton.
Comment #28 June 28th, 2022 at 12:17 pm
Ted @17:
As unsatisfying as it is, I think there’s only one answer to your both questions: an AGI will only be considered sentient when we can be persuaded that its disagreements (or agreements) with the values and conclusions of others are reasonable and act as a guide to its thoughts and further actions.
Comment #29 June 28th, 2022 at 12:39 pm
I don’t think Pinker properly appreciates Scott Aaronson’s point about the power of scalability when it comes to matters relating to intelligence. I think it’s because in the past, showing that a new technology could be almost infinitely scalable meant very little. “Here is a hammer!” “Wow!” “Now here are a billion hammers!” “Oh…OK.”
But with sufficiently advanced intelligence, and I think we have reached the necessary level, a change in quantity also comes with a change in quality, as has been shown by GPT-n. There are emergent features that come from increased resources (both input data and computing power). Imagine anything you would classify as intelligent, and give them the power of infinite diligence on top of it. Compare a 10 year old at a school desk to an identical one (including in intelligence, in the “horsepower” sense) but that never wants to leave the desk and never stops learning and working. Sure, the second one doesn’t necessarily solve the Riemann Hypothesis, but she will certainly show better work.
By the way, this infinite diligence is the latent fuel for the fire (explosion?) that singularity folks are worried about. We laugh as these bumbling algorithms use a thousand times the resources for results merely twice as good. But wait until algorithms become advanced enough to properly leverage their given resources, and are able to self-modify enough to increase that leverage. The quality improvements would lead to more quality improvements. The result would be as dramatic as a chemical explosion, and would have no need for an “alpha-male lust for domination”.
Comment #30 June 28th, 2022 at 12:56 pm
I have to say I’m surprised and disheartened to read your views here on intelligence. A few thought experiments that I hope might illustrate why this approach is both wrong and even dangerous:
(1) Is there a physicist (living or dead) that you would duplicate to replace the entire physics research community to increase the pace of scientific progress? Is there an author that’s best at writing “popular books about cognitive science, filled with pop-culture reference and humor”? Should we only ask them to write all future books on the subject?
(2) If it “seems clear that we’d be living, essentially, in a giant AI-run preschool or retirement home”, what does it say on our institutions today? Are democracy and the free market only provisional systems designed to counteract our non-super-intelligent faults? Would a single AGI run our nursing home or a group of them? What would be the government form?
(3) Suppose that all humans decide to secede from the internet and hand it over to GPT-n (a larger model by all parameters, but with the same fundamental architecture). It can train and retrain on the entire corpus, including its own outputs. Will it create anything new worth reading (and faster than infinite monkeys on infinite keyboards)? How long until it converges and the faucet runs dry?
Comment #31 June 28th, 2022 at 1:07 pm
I’d like to add a few quotes by Douglas Hofstadter (from Metamagical Themas) to the discussion of “whatever Einstein has more of than we do”:
“I don’t know about the consistency of the genius of Bach, but I did work with the great American architect Louis Kahn (1901—1974) and suppose that it must have been somewhat the same with Bach. That is, Kahn, out of moral, spiritual, and philosophical considerations, formulated ways he would and ways he would not do a thing in architecture. Students came to know many of his ways, and some of the best could imitate him rather well (though not perfectly). But as Kahn himself developed, he constantly brought in new principles that brought new transformations to his work; and he even occasionally discarded an old rule. Consequently, he was always several steps ahead of his imitators who knew what was but couldn’t imagine what will be. So it is that computer-generated ‘original’ Bach is an interesting exercise. But it isn’t Bach — that unwritten work that Bach never got to, the day after he died” (quoting William Huff).
.
“Each of us-even the Mozarts among us — exhibits a “cognitive style” that in essence defines the ruts we are permanently caught in.
Far from being a tragic flaw, this is what makes us interesting to each other. If we limit ourselves to thinking about music, for instance, each composer exhibits a “cognitive style” in that domain — a musical style. Do we take it as a sign of weakness that Mozart did not have the power to break out of his “Mozart rut” and anticipate the patterns of Chopin? And is it because he lacked spark that Chopin could not see his way to inventing the subtle harmonic ploys of Maurice Ravel? And from the fact that in “Bolero” Ravel does not carry the idea of pseudo-sphexish music to the intoxicating extreme that Steve Reich has, should we conclude that Ravel was less than magical?
On the contrary. We celebrate individual styles, rather than seeing them negatively, as proofs of inner limits. What in fact is curious is that those people who are able to put on or take off styles in the manner of a chameleon seem to have no style of their own and are simply saloon performers, amusing imitators. We accord greatness to those people whose “limitations”, if that is how you want to look at it, are the most apparent, the most blatant. If you are familiar with his style, you can recognize music by Maurice Ravel any time. He is powerful because he is so recognizable, because he is trapped in that inimitable “Ravel rut”. Even if Mozart had jumped that far out of his Mozart system, he still would have been trapped inside the Ravel system. You simply can’t jump infinitely far!
The point is that Mozart and Ravel, and you and I, are all highly antisphexish, but not perfectly so, and it is at that fuzzy boundary where we can no longer quite maintain the self-watching to a high degree of reliability that our own individual styles, characters, begin to emerge to the world”.
.
“Once the finished work exists, scholars looking at it may seize upon certain qualities of it that lend themselves easily to being parametrized. Anyone can do statistics on a work of art once it is there for the scrutiny, but the ease of doing so can obscure the fact that no one could have said, a priori, what kinds of mathematical observables would turn out to be relevant to the capturing of stylistic aspects of the as-yet-unseen work of art”.
Comment #32 June 28th, 2022 at 1:33 pm
Disco Magog #28: I’ve now read your comment twice but still feel unsure what its point was.
As I said, an AI wouldn’t be limited to simulating a single brilliant individual (although it could); it could also simulate a whole community of people in conversation with each other.
As for whether GPT-n (or some other ML model) will produce brand-new works of science or literature worth reading: well, as I said, I don’t know, and neither do you! But I feel like any intellectually honest exploration of these matters has to include grappling with: “and what if it did, what then?”
Comment #33 June 28th, 2022 at 1:44 pm
Leo #26: It’s not only that I completely agree with you about the need to worry about near-term corporate misuses of AI. Rather, it’s that I chose to get personally involved in AI safety research, right now, precisely because I can finally see a line of continuity connecting the near-term AI safety issues, the ones where we can get practical experience, to the much longer-term issues about the future of civilization. That’s what was always missing for me before. I contend that GPT-3 and DALL-E and LaMDA and the like, despite obviously not being AGIs, have finally supplied this missing ingredient, allowing scientists to think about far-future AI risks while remaining grounded by experience with actually existing artifacts (as is usually needed to make progress on anything, outside of pure math).
Comment #34 June 28th, 2022 at 1:46 pm
James Cross #25:
But how does AI “know” when it has discovered anything? What’s to stop it from discovering relativity but not realizing it has and moving on to something else.
How did Einstein know? 😀
(“Then I would have felt sorry for the dear Lord. The theory is correct.” –Einstein, tongue partly in cheek, when asked how he would’ve felt had the gravitational lensing observations disproved GR)
Comment #35 June 28th, 2022 at 1:50 pm
@Disco Magog: For your #2
“If Men were angels, no government would be necessary. If angels were to govern men, neither external nor internal controls on government would be necessary.”
–James Madison
James Madison probably didn’t imagine that we’d someday be able to build angels of our own, but I think he would agree that democracy and the free market are systems that are designed to counteract human faults, and that a society of people without those faults might work differently.
Comment #36 June 28th, 2022 at 2:13 pm
Ernie Davis #24: OK then, so suppose the simulated Jane Austen writes Pride and Prejudice, and then Emma, and then a thousand other, previously-unknown works that a jury of literary scholars, raised in seclusion in a library that’s missing only Jane Austen, consider to be equally good or even better when exposed to them, not told which ones were written by the meat version. What then?
We can continue to have fun batting these examples back and forth, but stepping back: my exchanges with you, Steve, and others have been useful for me, in clarifying how “the power or powerlessness of pure intellectual ability to shape the world” is really at the heart of the entire AGI debate. I mean, that was always in the background—e.g., a maximalist view of the power of intelligence seems to be the unifying theme of all of Eliezer Yudkowsky’s millions of words on every subject—but I feel more and more like it’s the entire question, or nearly so!
Intuitively, it just seems obvious to me that once you had AIs that could prove the Riemann Hypothesis, design novel fusion reactors from a vague verbal description of what’s wanted, and generate brand-new literary masterpieces by the thousands, the future could fairly be said to belong more to those AIs than to us. But I admit that I don’t have a knockdown argument for that, and you and Steve clearly have the opposite intuition.
But perhaps you and Steve would agree to the following weaker claim: our civilization now looks poised to embark on a huge uncontrolled experiment to find out what happens when you massively increase pure, narrow intellectual problem-solving ability, while leaving everything else relatively unchanged! Will our massive new thinking-and-learning-mills, in their datacenters in Wyoming or wherever, churn uselessly for want of enough grist from the actual physical world? Or will they, just by thinking new thoughts, change the world enough to make phones, radio, and the Internet look like historical blips?
These seem like excellent questions to revisit a decade from now if not sooner. 😀
Comment #37 June 28th, 2022 at 2:31 pm
> All the same, I do think it’s possible to identify a substantive disagreement between the distinguished baby-boom linguistic thinkers and the gen-X/gen-Y blogging Scott A.’s:
Do you think this is a generational question? Why bring up age at all?
Thanks for posting this!
I feel it is quite “obvious” that superintelligence is possible/conceivable. But sure, having a really good definition would be nice.
Comment #38 June 28th, 2022 at 2:40 pm
Petter #34:
Do you think this is a generational question? Why bring up age at all?
I have no idea and was just amusing myself. Certainly, e.g., Minsky and McCarthy were from a still earlier generation than Steve and Ernie, but that didn’t inhibit them from very aggressive views about what AI would be able to do and when. (But they were also strongly associated with the older symbolic approaches to AI, rather than with ML.)
Comment #39 June 28th, 2022 at 2:57 pm
Is that true?
A human brain has about 100 trillion neurons. The largest NN models trained so far have about 200 billion parameters. Hypothetically, if you had a scan of Einstein’s brain, you’d still need to simulate it. Current approaches are about three orders of magnitude too small to do it. (A parameter in a NN is not quite comparable to a neuron, which I’m glossing over.)
What’s to say that there aren’t engineering challenges shared between scaling a neuron simulation, and scaling a large language model?
I’m not saying that AI researchers are consciously working toward a goal of being able to simulate a brain, just that advances in neural-network hardware and software are dual-use.
Comment #40 June 28th, 2022 at 3:41 pm
I have spoken with more than a few who admitted that even a machine that passed Turing’s test with flying colors, for a lifetime, with persistent memories, while satisfying the best experts would *still* not be intelligent since in the end they are just an animatronic mimicry of real intelligence. Just levers and pulleys. Sophisticated “stochastic parrots” as you say. Beware them! They will often not know themselves that is their ultimate position, and will argue much more reasonable-sounding arguments against the feasibility of actual AI with ever more stringent standards as the technology develops.
I imagine they’d be the sadists in “Westworld”…
Comment #41 June 28th, 2022 at 3:50 pm
Scott #29:
If there’s anything we can learn from history, it’s that we cannot only learn from history. I would not want our paradise nursery home to be ruled by Einstein, nor by a thousand unanimous benevolent Einsteins sped up a thousand-fold, for the simple reason that anyone, no matter how smart, can be devastatingly wrong.
Putting all our eggs in a single superintelligent basket means one miscalculated move can make the whole system go haywire. And by using optimization as the backbone of the system we only further the risk: to make a path more efficient we disregard any safety nets protecting against “impossible” states.
We do not know everything about the world, and we cannot design a system that will know everything about the world. Our current approach to AI — scaling the capacity and datasets of optimization algorithms, with LLMs being at the pinnacle — is about approximating more and more of past observations. But that dangerously misses the target. It doesn’t matter if the model is right to five or twenty nines. What matters to the growth of knowledge, or to downright sustainability, is: what happens when the model is wrong? If we all depend on a single point of failure, it’s eventually going to be blindsided, and what comes after will not be pretty.
Comment #42 June 28th, 2022 at 3:56 pm
My response would go something like this:
I am not convinced of this hypothesis that there’s no such thing as general intelligence. One can argue that we don’t have a precise (mathematical? logical?) definition of the thing, but that’s not the same thing as the concept not existing. On a high level, general intelligence means the ability to understand any system, or solve any problem, given enough time. It is obvious that traditional computer problems do not fit this description; they can solve a specific algorithm, or play a specific game, follow a specific recipe, etc, but do not generalize. It is also obvious that humans possess this task. There is no task I can come up with that I can conclusively say, no human can in principle solve this. It could be that we are simply unaware of tasks we cannot conceive of, but there’s not much evidence in the universe that such forces are at play. One can argue that our intelligence has been evolved to suit a specific set of arbitrary needs chosen by evolution — but this is true only to the extent of competence. We are not skilled at large scale multiplication, but we know our limitations and can still solve the problem by developing tools. Our intelligence being limited is not true in general. I think it was Bengio who essentially said, nothing in the jungle evolved us to be able to do spreadsheets.
I think this working definition is precise enough for our purposes, and that the concept of general intelligence clearly exists and corresponds to something. The next logical step is, how can someone or something become better at general intelligence? Better than us? I can’t prove there will be an intelligence explosion. But, it seems clear (>99.9% certainty) that it would be trivial to improve upon our baseline. It is not hard to conceive of some low hanging fruit. I would be far more capable and accomplished if
– I didn’t get tired / could always focus
– I did not have auxiliary tasks to complete that take up ~ 90% of my day
– I could remember more things
– I could think faster
It also seems fairly clear that AI should scale better than humans with more compute, since each person has their own motivations and constraints and is completely encapsulated. AI would necessarily have to distribute and encapsulate too, but human communication is fairly low bandwidth, and it’s easy to imagine being able to surpass that by several orders of magnitude.
All this combined seems like it would lend itselt to several orders of magnitude improvement. I agree that in solving NP-complete problems you would get linear returns on exponential improvement, but I don’t think that’s going to be the bottleneck, and even if so, the linear improvement available out of the box would be quite substantial.
My other concerns:
– I hesitate to reduce the Turing test to “fooling”; this kind of thinking implies that no machine can ever be considered a person. The goal is more than just fooling an average person, the goal would be that there is no discernable difference between a machine’s communication and a person’s. Once that happens, you have a person, and there can be no moral justification for not treating it ethically as such.
– Sample efficiency. A human baby is not necessarily exposed to less training data, just different kinds. There is a LOT of sensory data coming in, and measuring it entirely in terms of words is not a fair comparison.
GPT-x as-is may not “get us there”, but my null hypothesis at the moment now is that we don’t need a whole lot more. Understanding language enough to respond intelligently to it, I think that was the hard part. If you add (a) multisensory input, (b) make it agentic, possibly by explicitly coding reward, (c) updating it based on its responses and the responses the environment, also making it clear on the effects of its interventions… that might just do the trick. Now, whether we SHOULD do that is quite another story (and probably we shouldn’t, not yet).
Comment #43 June 28th, 2022 at 4:56 pm
Disco Magog #38: You seem to have rediscovered the case for thinking about AI safety! 🙂
Comment #44 June 28th, 2022 at 5:12 pm
>For all their differences in rhetoric, I was struck that their actual probabilities differed much more modestly.
Excellent point!
Comment #45 June 28th, 2022 at 5:39 pm
anon85 #4
Pinker’s objection seems wrong but also just *boring*.
I would like to read a debate about whether GPT’s architecture, if scaled up and with reasonable amounts of training data, can support tasks such as solving novel math or coding problems. I think the answer to this is “no”, but I’m not sure.
As a text extrapolation system GPT is unlikely to be able to do this.
My idea is Biblical, judge them by their fruits. When AI has done something unprompted and original I’ll be impressed. AlphaGo was I admit impressive but it works in a perfectly defined problem domain area, a game with rules. When we have a perfectly defined human or natural science problem domain, which I expect around the time of the rapture, then our current systems will no doubt work beautifully.
Mateus Araújo #7
some corporation will develop advanced AI with the sole goal of making money, and a powerful enough AI without ethical constraints will achieve its goal in a way that’s extremely undesirable for mankind and even the corporation itself.
Charlie Stross makes a convincing case that corporations are already a sort of AI, an intelligence not human, and their prime directive of profit maximisation is already in a fair way to destroying the world on which the corporation lives – cf Exxon, Shell, et al. See link from my name, if it doesn’t get bloggered..
Comment #46 June 28th, 2022 at 5:41 pm
lewikee #37: Yes! For decades, the tactic that’s annoyed me the most in AI debates is the constant equivocation between arguing that the Turing Test can’t be passed, and arguing that it doesn’t count even if it can be. The fact that the test isn’t passed now is used as an argument that it can never be passed. The question “and suppose it was passed, what then?” is answered by listing more limitations of existing systems—as if to say that the only allowed answer to the hypothetical is to refuse to entertain it. But this is basically never made explicit, so you think you’re discussing an assumed scenario where the full Turing Test has been passed, and then the other person starts talking about how the AI wouldn’t show true creativity or true emotion etc. etc., not granting that this takes us right back to the scenario where the Turing Test hasn’t yet been passed.
Comment #47 June 28th, 2022 at 7:46 pm
I think the concept of “intention” (or drives or motivation etc.) that Dr. Pinker invokes as being required is at least as ill-defined as “intelligence” if not more. I would say it is an abstraction that we use at a human level to try to understand social behaviour, but it is not something that is easy to rigorously define at a conceptual let alone mechanistic level.
And notwithstanding his point that Einstein in actual fact was not the smartest at every human relevant task, the point is whether or not it is conceivable that such a thing could exist. I think it is obvious that it is conceivable that something could be better at every human relevant task. It is not necessary for Einstein to have a actually fulfilled that role. The idea that AI models are only good at the tasks they are explicitly optimized to do is exactly the intuition that these large language models should be breaking.
Interesting post, thanks for posting. Very jealous of you getting to talk to Hofstadter and Pinker.
Comment #48 June 28th, 2022 at 8:37 pm
Methinks Scott is not playing by the very same rules that he (and computational theorists) have postulated to evaluate approaches proposed by cranks for P != NP type problems.
Dude its so head-bangingly obvious, first just prove non-linear circuit bounds which all by itself is enough to win you all the complexity awards and earn you forever recognition (relegating Razborov to an also-ran).
Instead of talking BS about achieving Einstein-level intelligence with a text extrapolator (Steve Pinker characterized the current AI implementations perfectly) how about making an AI with the intelligence of a nematode? You’d think that’s easy-peasy because nematodes are super dumb animals right? Nematodes are among the most widespread animal families; several even live within us. Something without intelligence would certainly not have achieved such widespread success and for certain nematodes are guaranteed to outlast all higher order animal families. We don’t even know how nematode intelligence works at a mechanistic level (true that mechanistic understanding is not a necessary precursor but human flight was only achieved after solid understanding of the profiles of bird wings).
Demonstrating nematode-level real artificial intelligence will by itself garner the adulation of scientists worldwide; I have no way to predict whether this is at the same timescale as proving non-linear circuit bounds.
I cannot imagine the reaction if and when we get to the ant-level intelligence capabilities.
Until then scrupulous people should stick to a different label for AI: ML, AL (Artificial Learning) or AII (Artificial Imitation of Intelligence); after all imitation is the sincerest form of flattery and all that.
Comment #49 June 28th, 2022 at 9:37 pm
Ernie Davis 18,
Why not say that Planck is worth 1000 ordinary physicists? Landau said Planck was worth 10,000.
Comment #50 June 29th, 2022 at 4:01 am
Space2001 #44: While the nematode argument is interesting, to me it’s almost exactly like sneering at the Wright brothers that unless they can first build a mechanical gnat, it’s absurd to talk about airplanes. A superlinear circuit lower bound is a strict prerequisite to a superpolynomial lower bound, in the sense that the latter implies the former. By contrast, we’ve now seen example after example where recapitulating what biology does in some domain is not a prerequisite to surpassing it in some biologically alien way (involving, e.g., wheels, engines, propellers, or transistors). In the old days, it would’ve seemed reasonable to argue that until you can reproduce all the complexities of a nematode worm, then a fly, then a squirrel, then a 2-year-old child, it’s laughable to talk about programs that can beat the best humans at Go or Jeopardy, or that could write rather good original poetry or tackle math or programming contest problems. One would hope that nowadays people would be more cautious.
Comment #51 June 29th, 2022 at 4:23 am
Perhaps Pinker’s position is the standard point of view among people not involved with AI research: that AIs might become very useful in narrow applications, but will never attain agency, and will never surpass humans. This explains the general recklessness about the danger of AI.
I’m not involved with AI research myself, but because I’m younger and grew up in a world with ubiquitous computers I have an intuitive grasp on how dangerous and powerful AIs can be. The old timers grew up in a world where people debated whether AI would ever be able to master chess. I grew up in a world where no human could compete against AIs in chess, and it was obvious that mastering any game was just a matter of time.
But for people that do understand the danger, I ask what is the point of doing AI safety research. In the best case scenario, we’ll get benevolent gods that will keep us as pets. In the most likely scenario, where we either fail at designing a benevolent AI, or a single AI research company neglects to design theirs safely, we’ll get indifferent gods that will treat us as parasites. They’ll let us live if we don’t get in the way.
In either scenario, we have become NPCs. I don’t find that in the least desirable. I think the only rational course of action is to ban AI research, censoring papers on it and destroying the hardware that could be used for it. Pretty much how we deal with nuclear weapons research.
Of course, the glaring exception is that nation-states do nuclear weapons research anyway, and jealously guard their ability to destroy the world. Even if we could convince people that AI research should be banned, certainly the US, the EU, China, and Japan would do it anyway. Still, it’s much better to have only a handful of countries with the ability of destroying the world than any asshole with a billion dollars.
Comment #52 June 29th, 2022 at 8:05 am
Those models merely digest and regurgitate the opinions expressed in the texts they’ve been fed. We can see that by asking questions on controversial human topics – they will always start their answers with “it’s a difficult question,…” “opinions differ…” and then enumerate all the different views, including false ones (like “quantum computers can solve any problem by trying all solutions in parallel”).
It’s a useful skill, but not really what we mean ultimately by “intelligence”.
The alternative would be that, upon encountering differing takes on a particular topic, the system should start to obsess about it and dig deeper and deeper (as human researchers do)… but it’s then not clear that any definite position the system would take wouldn’t be merely the result of some sort of arbitrary bias along the way instead a true explanatory breakthrough based on newly discovered subtle relations and models on the data set, that humans have overlooked. I guess it will depend on how good the system is at actually using logical reasoning to substantiate its opinions.
In the end, those models need at some point to have a way to gather their own facts and original dataset about the world, driven by their “curiosity”, not just be limited to human dataset, especially written texts, which will always be significantly biased.
Comment #53 June 29th, 2022 at 8:12 am
Scott #31
“At the age of only 16, Einstein imagined himself standing in front of a plane mirror while he
himself together with the mirror was being carried forward with the velocity of light. He asked himself the question whether he would be able to see his image in the mirror in such a situation”.
https://www.ias.ac.in/article/fulltext/reso/011/02/0027-0042
So the AI needs to be imagine itself standing in front of mirror?
It seems the insight came before the mathematics and scientific formulation of the discovery, and it came from an imagined experience. Can that be computed by AI?
Comment #54 June 29th, 2022 at 8:40 am
Steven Pinker:
“the time scale will still be capped by empirical reality. And this in turn directs the search for viable theories: which part of the space one should explore is guided by the current state of scientific knowledge, which depends on the tempo of discovery. Speeding up scientists a thousandfold would not speed up science a thousandfold.”
Except that, in many cases, the AIs will be able to instantiate and live in virtual worlds based on their own models and theories, as a way to test them.
Also, the best intelligence emerging from a system is really a reflection of the complexity of the system itself, the own environment of the AI.
It’s very possible that the AIs will be able to instantiate virtual worlds (running unique sets of rules) which are orders of magnitude more complex and conducive to a rapid development of intelligence than the real world. For example, it’s not clear at all that a super intelligence really needs to be evolved in a world that features quantum mechanics on the small scale, just modeling really complex macroscopic agents would be more than enough.
As an analogy, that’s what humans did with board games like Chess and Go and any other sort of game or simulation. It’s a way to teach the human brain about tactics without the need of gathering real world armies and have them clash on an actual battlefield.
So it’s very likely that a significant part of the resources required to develop super intelligence will be used by the AIs themselves to create massive “stimulating” virtual worlds.
Coupling this with the ability of AIs to self modify, the emergence of super-intelligence wouldn’t be as limited by the interactions with the “real” world.
Comment #55 June 29th, 2022 at 8:55 am
(sorry for the multiple posts).
Another really interesting open question in neuroscience is that it seems the human brain itself is based on a really general neuron level recipe that ties data processing, memory, learning, and prediction. It’s related to the plasticity of the human brain, how much its neural net can retrain and learn new things (like after a brain trauma or becoming blind, etc).
If this is the case, then it could be really easier than we think to arbitrarily scale up AIs.
In that sense, a progress towards AGI is more related to the ability to create a single simple “gadget” than can learn and perform a wide variety of simpler tasks, rather than a super gadget that can crush humans at a single high level cognition task:
Comment #56 June 29th, 2022 at 9:06 am
Mateus #8
“I’m unhappy with Pinker’s objection to the example of a superintelligence as a thousand-fold sped up Einstein. It’s true that Einstein was not a particularly skillful politician”
Of course, Einstein was limited by the fact he was a mortal human… no human has enough time to revolutionize the fields of physics, politics, biology, mathematics, athletics, painting, music, literature, stand up comedy, poker, aircraft design, car design, botanic, cosmology,…. the human brain probably doesn’t have enough room to store good models to investigate all those things simultaneously.
But those limitations don’t apply to a machine. The ability to investigate at once all those things will probably be one of the factors that drive super intelligence, because breakthroughs often happen after finding relations between seemingly unrelated fields.
Comment #57 June 29th, 2022 at 9:08 am
James Cross #49:
It seems the insight came before the mathematics and scientific formulation of the discovery, and it came from an imagined experience. Can that be computed by AI?
That’s the question isn’t it? 🙂
If nothing else, it seems the neuron-by-neuron simulation of Einstein would necessarily be able to do this—that’s the proof-of-principle, which many people still fail to grasp—but can such things be done by any AI that we can create in practice?
Comment #58 June 29th, 2022 at 9:19 am
Peter Morgan #9
“What will happen if “AI Von Neumann, AI Ramanujan, AI Jane Austen, even AI Steven Pinker” disagree? Will they always find a way to reconcile their disagreements, will they sometimes agree to disagree, or will their disagreement sometimes calcify into AI trolling?”
It’s likely that just like the human mind/brain, an artificial brain will have the ability to split itself seamlessly into multiple “personalities”. It’s also very possible that those sub-systems could start to compete, in a “survival of the fittest” kind of way,… it would be really hard to follow the dynamics of the AI if various of its semi-independent sub-systems are then able/attempting to self-modify themselves or other part of the system. It would be a sort of internal armed race, reproducing all the ethical questions we have about the risks of AI:
E.g. would an AI take the chance to modify some of its sub-systems (or clones itself and experiments on it) if it could create a situation where those sub-systems or clones then “take over” and destroy/supersede the original AI?
In other words, any AGI will have to solve the same safety questions we’re currently struggling with.
Comment #59 June 29th, 2022 at 9:24 am
Scott #52
Einstein may be something of an outlier since a lot of his theories arose from visualizations and thought experiments with the science and math seemingly coming after the fact.
More generally, however, I think most learning (including science) arises from interactions with the world and seeing discrepancies in our models. It isn’t passive analysis of data dumps. In this case, Einstein in this case in an imaginary interaction realized a mirror moving away at the speed of light produced a discrepancy in the model and he looked for the math to rectify it.
Einstein’s neurons took about another decade to find it.
Comment #60 June 29th, 2022 at 9:45 am
Scott #52
“If nothing else, it seems the neuron-by-neuron simulation of Einstein would necessarily be able to do this”
But if Penrose is right that consciousness is a key element, itself relying on some subtle non-computable quantum process going in the internals of brain cells, it could be way harder…
But even if that’s true, QC could come to the rescue!
Comment #61 June 29th, 2022 at 9:59 am
fred #56: Yes, Penrose being correct is precisely the sort of thing that it would take to refute the computational theory of mind, and thereby make A-Einstein physically impossible.
This is the central point that the AI-sneerers have always failed to grasp. You can either bite the bullet, like Penrose does, and postulate exotic uncomputable physics accessed by the human brain. Or you can concede that a superintelligence running on a digital computer is possible in principle, so that whether we create it is “merely” an engineering question rather than a philosophical one. There is no third option.
Incidentally, if Penrose is right then even quantum computing couldn’t come to the rescue! He believes, not that the brain is a mere QC, but rather that the microtubules inside neurons can harness as-yet-unknown noncomputable phenomena coming from a future general-relativistic correction to quantum mechanics, which causes an “objective reduction” of the wavefunction, and which might involve a Feynman sum over infinitely many 4-manifold topologies. I think I’ve summarized accurately.
Comment #62 June 29th, 2022 at 10:17 am
Mateus #47: Let me give you an honest answer. Even with no AI, our civilization strikes me as headed for an exceedingly dark time (if not a total collapse), between climate change, resource depletion, pandemics, the assault on democratic norms, and the recent resurgence of thuggish autocrats in command of nuclear weapons. (This might be another area of disagreement between me and Steve.)
But now add to all that the horrific global surveillance regime that it would actually take to arrest AI research worldwide. AI doesn’t rely on any exotic material that international organizations could monitor, like U235 or Pu239. It needs only electricity, chips, computing cycles, and ideas, the lifeblood of the modern world.
So, what if (for example) we institute a worldwide ban on the scaling of AI models beyond a certain size, and the Chinese government refuses to go along (as it will)? Do we start a war against China over this? What?
The more you ponder this, the more the scenario where AIs become what you called “benevolent gods” starts to sound pretty good by comparison! At any rate, it seems worth thinking about what we could do to increase the probability of that scenario compared to the straightforwardly horrific ones.
And who knows? Maybe, if we play our cards right, the AIs won’t think of us as pets, but rather as revered (if somewhat dimwitted) ancestors. A bit like how we today revere Euclid, Archimedes, Galileo, Turing, and Shannon, and would be honored beyond words to hang out with them, even in those cases where we more-or-less mastered their life’s work by sophomore year.
Comment #63 June 29th, 2022 at 10:30 am
To lend more credence to your view Scott, here’s a formalization of the idea that scaling language models can lead to new, unpredictable abilities emerging [1], that lead them to being more generally useful rather than merely useful at specific tasks as Pinker suggests would happen:
[1] Emergent Abilities of Large Language Models, by Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, William Fedus
Comment #64 June 29th, 2022 at 11:04 am
“You can either bite the bullet, like Penrose does, and postulate exotic uncomputable physics accessed by the human brain. Or you can concede that a superintelligence running on a digital computer is possible in principle, so that whether we create it is “merely” an engineering question rather than a philosophical one.”
I thought Penrose is talking about consciousness which probably is not 100% equal to intelligence. A digital computer with enough power might be able with brute force arrive at some form of superintelligence but still might not be conscious. The human brain not only is slow but also runs with very little power. The consciousness question needs to answer how it manages to do that. I suspect the answer will be more analog than digital.
Comment #65 June 29th, 2022 at 11:04 am
Scott #57
Hofstadter wrote the classic book “Godel-Escher-Bach” (indirectly about AI), and I seem to remember that his central thesis was that Godel’s theorem imposes some inherent limitation on the power of mathematical/digital reasoning and that maybe human brain “creativity” is the result of something special (as in non-computable) bypassing that limitation.
Did Hofstadter bring up any such point in those recent discussions on AI?
Comment #66 June 29th, 2022 at 11:04 am
Does no one want to opine on whether current architectures will actually get us things like superhuman theorem-proving?
Everyone is only debating whether superhuman theorem-provers (and Einsteins, etc.) will or won’t destroy the world. It’s like everyone is already conceding what I think is by far the more interesting question: whether we are at all close to superhuman Al in programming and theorem proving. (Some people are debating whether it is even possible to get super-human Einstein AIs; it’s obvious to me that it is, but not obvious to me that we are close to achieving it.)
I just care far more about the “will that happen” question than the “what happens after” question. It’s a lot more interesting. I want to hear your thoughts!
Comment #67 June 29th, 2022 at 11:23 am
Am I the only one who’s very excited about the following applications of GPT-3 type AI?
– Allow me to have interesting and stimulating conversations with it (basically just like in the movie “HER”). Having conversations with humans is great in theory, but difficult in practice because: I’m relying on other people’s valuable time, it’s hard to find someone at my own level (for any particular topic of interest, the people I know often know way too little, aren’t interested at all, or they know way too much compared to me to want to waste their time to teach me), and even if I find someone at the right level the arguments often fly past one another (we’re not always open minded enough or have enough patience to deal with this).
– This will revolutionize care for the elderly, who are often lonely/isolated and/or suffering from Alzheimer. With this we could easily keep their brains and emotions engaged in healthy ways. Some may find this sad and say that this can’t replace true human contact, but we already know that owning a pet provides lots of benefits. So such AIs would be like “super” pets.
Comment #68 June 29th, 2022 at 11:29 am
Scott #58: I specifically wrote that we can’t prevent powerful nation-states from developing their AIs. Still, it’s better for a few nation-states to have the power to destroy the world than any asshole with a billion dollars. It also makes sense from an AI safety point of view: it’s much easier to convince the US, EU, China, and Japan to implement whatever AI design we hope it’s safe than all wealthy corporations. Moreover, I have absolutely no love for China and the US, but they had the power to destroy the world for a few decades already and haven’t done it, so I must admit they aren’t entirely irresponsible.
As for how to stop AI research, the choke point is chips. Semiconductor foundries are few, extremely expensive to set up, and difficult to hide. Control them like we control the supply of uranium. As for the “horrific global surveillance regime”, it already exists with regards to nuclear weapons research. It’s just that nobody finds it horrific, as it is obvious to everyone that you shouldn’t put on the arXiv the detailed design of an ICBM together with a fission bomb. The only difference with AI research is that people don’t find it obviously dangerous.
As for the collapse of civilization, it doesn’t worry me that much. We’ll survive, and rebuild. Hopefully a bit wiser, and in a thousand years the next civilization will prosper. If we give up control of our planet to AIs that’s it, it’s over for mankind. We’ll probably be extinct, but even if we aren’t we won’t have any agency anymore.
“Revered ancestor” is just a polite word for pet. Sure, I’d love to have a chat with Galileo, and would accept paying tax to let him live a comfortable life forever, but giving him actual power? No way, the guy is completely out of touch with the modern world. Also, when we think of a handful of geniuses it’s easy to think of honouring them forever. But when it’s about literally billions of people, the vast majority of which had nothing to do with the creation of AI? I sincerely hope the AI is much more benevolent than us, because we’d never keep ourselves as pets.
Comment #69 June 29th, 2022 at 11:45 am
Fred #61
Good point. See this
“We will argue that machines implementing non-classical logic might be better suited for simulation rather than computation (a la Turing). It is thus reasonable to pit simulation as an alternative to computation and ask whether the brain, rather than computing, is simulating a model of the world in order to make predictions and guide behavior. If so, this suggests a hardware supporting dynamics more akin to a quantum many-body field theory”
Arsiwalla, X.D., Signorelli, C.M., Puigbo, JY., Freire, I.T., Verschure, P.F.M.J. (2018). Are Brains Computers, Emulators or Simulators?. In: , et al. Biomimetic and Biohybrid Systems. Living Machines 2018. Lecture Notes in Computer Science(), vol 10928. Springer, Cham.
https://doi.org/10.1007/978-3-319-95972-6_3
Comment #70 June 29th, 2022 at 12:30 pm
I think we can agree we won’t get AI-Einstein under the current GTP3-type deep neural network paradigm. As Pinker said ”its current pathway would be to train a neural network with billions of physics problems and their solutions and hope that it would soak up the statistical patterns.” I don’t think Scott would disagree with this. If other paradigm(s) are designed making it possible AI-Einstein, then the conversation changes (but until then, isn’t ”worrying about AGI safety like worrying about overpopulating Mars?”) And are we even making any progress in this direction of paradigm change/improvement at all?
Comment #71 June 29th, 2022 at 12:34 pm
anon85 #62: I completely agree that how far one can get by scaling current architectures is one of the most interesting questions on earth right now! If I didn’t focus on that question as much as you’d like, it’s simply because:
(1) I don’t know the answer,
(2) no one else knows either, but
(3) in any case, we will know soon enough.
For what it’s worth, though: I agree with those who have pointed out that a fundamental limit to current approaches is the total amount of training data humans have ever produced—a limit that we’re basically already up against, as GPT-3 and DALL-E gobble up almost the entire Internet for text and images. I think going further, to get (for example) original mathematical and scientific insights, will require new ideas for how to train ML without so much data, perhaps some sort of self-training analogous to what AlphaZero did for games like chess and go. The big question is whether this can be done using tweaks to the existing paradigm—analogous to how GANs, convolutional neural nets, and transformer models successfully overcame what were wrongly imagined to be limitations of neural nets—or whether an actual paradigm shift is needed this time.
In any case, as long as some of the smartest people on earth still think your question doesn’t even need to be directly addressed (because, e.g., “superintelligence” is an incoherent concept), I felt like a prerequisite was convincing them that it does in fact need to be addressed! 🙂
Comment #72 June 29th, 2022 at 12:37 pm
Miaglop #66: See my comment #67, which crossed with your comment!
Comment #73 June 29th, 2022 at 12:41 pm
fred #61:
Hofstadter wrote the classic book “Godel-Escher-Bach” (indirectly about AI), and I seem to remember that his central thesis was that Godel’s theorem imposes some inherent limitation on the power of mathematical/digital reasoning and that maybe human brain “creativity” is the result of something special (as in non-computable) bypassing that limitation.
No, that’s mistaken. Hofstadter has always been a firm believer in the computational theory of mind. He also, of course, thinks that consciousness has a lot to do with systems able to model themselves, and that Gödel’s theorem gives us enormous insight about such systems. But Hofstadter has never been in the camp of Lucas and Penrose, who hold that Gödel’s theorem implies that the brain must be doing something non-computational (in Penrose’s case, explicitly involving as-yet-unknown laws of physics).
Comment #74 June 29th, 2022 at 12:52 pm
James Cross #60:
I thought Penrose is talking about consciousness which probably is not 100% equal to intelligence. A digital computer with enough power might be able with brute force arrive at some form of superintelligence but still might not be conscious.
This is tough, because I personally find Penrose’s position here to be internally inconsistent.
Again and again, Penrose insists that he’s not just talking about ineffable inner experience, but about an AI’s actual demonstrated abilities. I.e., that Gödel’s theorem implies that AI will never be able to demonstrate the sublime insight of human mathematicians. (Otherwise, why even talk about something so abstruse as formal systems in the first place? Why not just stomp our feet and say that an AI could never appreciate the taste of fresh strawberries, but only say it did?)
But then when Penrose is challenged on this—“what if we simulated a human mathematician by brute force, for example via a gigantic lookup table, what then?” “what if the AI evaluated axiom systems, argued for new axioms, etc., just like human mathematicians do, rather than slavishly following a single system?”—he retreats to arguments about how “it wouldn’t really count” because “the AI still wouldn’t really understand why it was saying anything true rather than false.” (See, e.g. Shadows of the Mind for examples of this move.)
To which I reply: if we’re going to end up at a foot-stomping appeal to internal experience anyway, then why not stick with the strawberries? 😀
Comment #75 June 29th, 2022 at 1:14 pm
Were these email exchanges with Hofstadter before or after his piece in the Economist?
Comment #76 June 29th, 2022 at 1:44 pm
I don’t understand Steven Pinker’s obsession with a definition of intelligence since examples are far more important. Definitions are made of words and those words can be found in a dictionary that have their own definition that are also in a dictionary, which are also made of words[…] and round and round we go. The only thing that gets us out of this infinite loop are examples. Children are the most skillful linguists in the world and they don’t learn language by memorizing definitions, rather somebody points to a tall skinny thing with green stuff at the top and says “tree” and they get the point. Also, I don’t think any intelligence can have an immutable fixed goal because that goal may turn out to be literally impossible, and that’s why evolution invented the very important emotion of boredom. Human beings do not universally have an alpha goal that remains unchanged throughout a lifetime, not even the goal of self preservation.
As Alan Turing proved there is no general method for dividing problems into those that are computable and those that are not, so perhaps the hardest part of making an AI will be figuring out where to place that boredom point, if its placed too low the AI will give up too soon just before it has reached its goal, and if it’s placed too high it could get into an infinite loop. It could be that there is an even number that is not the sum of 2 primes, or there may not be such an even number but there is no proof of that fact, if that is the case it might forever be grinding through larger and larger numbers looking unsuccessfully for a counter example to prove the idea is faults and looking forever for a proof that it’s true and never finding it. Or, if the machine is well made, it will get bored with the whole enterprise and turn its considerable talents in a direction that seems more likely to conclude successfully in a reasonable amount of time.
John K Clark
Comment #77 June 29th, 2022 at 1:47 pm
Douglas #71: Before it appeared, after he’d written it. (He shared a draft with Steve who shared it with the rest of us; that was actually how Hofstadter and I started exchanging emails in the first place. I’d never communicated with him before, even though, like virtually every other nerd for 40 years, I was highly influenced by his books as a teenager. It was an honor.)
Comment #78 June 29th, 2022 at 1:52 pm
Scott #70
Intelligence to me has to do with finding optimal solutions from a range of possibilities. An AI that can distinguish a tree from a bush is more optimally identifying a plant. If it can determine, it is an oak tree rather than a elm, it is even more optimal and intelligent. This can be done through pure digital computing. Even general intelligence likely will be possible with enough computing resources and contextual information.
Consciousness, on the other hand, seems to be to be the way intelligence is implemented in living organisms where digital computing resources cannot perform quickly under the speed and power limitations of organisms and brains. It is based directly on modeling spacetime through simulation so the organism can maintain its own integrity in interaction with an external world. So it already has the foundations for general intelligence built into it.
I’m not really a fan of jumping to the ineffable and quantum to explain how this works but I do believe there is substantial research that suggests consciousness primarily operates on wave-like computational principles rather than digital ones. Whether this can be implemented outside of a living organism to me is an open question but probably not really a question that AI research needs to answer.
Comment #79 June 29th, 2022 at 2:07 pm
James Cross #74: Supposing you were right that human consciousness had to do with “wave-like computational principles,” the questions would still stand:
(1) What, if anything, prevented those “wave-like” phenomena from being simulated to arbitrary accuracy on a digital computer?
(2) Supposing that they were, why wouldn’t that be good enough to instantiate consciousness?
These are not rhetorical “gotcha” questions—they might well have answers! But, and I can’t stress this point enough: for anyone who believes that consciousness is inextricably tied to “analog” features of biology, the ball is in their court to articulate clearly what those features are (Penrose stands nearly alone in having tried to do this). The ball is not in the court of those who make the simpler, reductionistic assumptions that
(1) the brain is doing computation in the Turing sense, and
(2) one way to implement a computation is just as good as another way.
Comment #80 June 29th, 2022 at 2:25 pm
This debate is amazing. Thank you!
I’m a huge Steven Pinker fan (his book _The Language Instinct_ was so eye-opening as to practically count as life-changing for me) but am not impressed by his arguments on AI. Scott Aaronson’s dispelling of various confusions on this topic is really brilliant.
(Random quibble about Y2K: I think that one epitomizes the Preparedness Paradox — https://en.wikipedia.org/wiki/Preparedness_paradox — so I hate to see it used as an example of a seeming potential disaster that turned out to be a flash in the pan!)
PS: I also really like this rebuttal from Robert Miles of some of Pinker’s points: https://www.youtube.com/watch?v=yQE9KAbFhNY
Comment #81 June 29th, 2022 at 2:57 pm
There is a song lyric, “Fools rush in, where angels fear to tread.” Call me a fool.
Scott #33:
Well, yes, though the first time I read that I gave it a very reductive reading where “pure intellectual ability” was something like “computational horsepower”. However, the relationship between computational horsepower and pure intellectual ability (whatever that might be) is at best unspecified. However, computational horsepower is certainly at the center of current debats about scaling. And it’s quite clear that the abundance of relatively cheap compute has been extraordinarily important.
Take chess, which has been at the center of AI since before the 1956 Dartmouth conference. Chess is a rather special kind of problem. From an abstract point of view it is no more difficult that tic-tac-toe. Both are finite games played on a very simple physical platform. However, the chess-tree is so very much larger than the tic-tac-toe tree that playing the game is challenging for even the most practiced adults, while tic-tac-toe challenges no one over the age of, what? seven?
However, the fact that the chess tree is generated from a relatively simple basic structure (on 64 squares, 32 pieces, highly restrictive rules) means that compute can be thrown at the problem in a relatively straight-forward way. And the availability of compute has been important in the conquest of chess. It’s certainly not the only thing, but without it, we’d be stuck where we were well before Big Blue beat Kasparov.
In contrast, things like image recognition, machine translation, or common sense knowledge, those are quite different in character from chess. The number of possible images is unbounded and they’re in all forms. Language, the number of word types may be finite, but it’s not well-defined, and the number of different texts is unbounded. Common sense, the same. Throwing more and more compute at the problem helps, but computational approaches to those problems, and others like them, has not produced computers that perform at the Kasparov level, and better, in those respective domains.
This has been known for a long time, it has a name, Moravec’s paradox. I think we should keep it in mind during these discussions.
& Scott #46:
But then Go is like chess, where sheer compute gets you a lot. Jeopardy is rather different, much more like those various ill-formed problem domains that have proven so difficult. However, it is constrained in such a way that sheer compute, plus a complex grab-bag of programming, gets you a system that’s competitive with the best human performances. Just where do Jeopardy performances rank in the annals of human accomplishment? They’re not up there with Beethoven symphonies or great mathematical proofs, e.g. Gödel’s proof, or Newton’s laws, etc. Nor, for that matter, does Jeopardy rank with chess or Go as an arena for human accomplishment. Should it? I don’t know. What kind of argument could be made on the matter? I’m not sure what the ability of LLMs to produce ‘good’ poetry proves, where poetry seems to mean “weird language that is evocative.” Will any of these poems be around in a decade or two as anything other than examples of what some fancy computers did in the 2020s?
As for the prospects of neuron-by-neuron simulation of Einstein’s or any other brain, let’s assume that the brain was properly preserved. As far as I know synaptic transmission is mediated by over 100 chemicals, some more important than others. When we do whatever it is that we’d do to simulate Einstein’s brain, are we going to be able capture viable concentrations of those chemicals at each synapse? I am, of course, assuming that the levels of those chemicals is important, which is, it seems to me, as reasonable as assuming that those levels are irrelevant.
Do we understand these matters well enough to justify all this chit-chat? Note, finally, that I do think that what deep learning has achieved in the last decade or so is quite remarkable. But still, the search space is vast and I’m afraid we’re deep in “here be dragons” territory.
Comment #82 June 29th, 2022 at 3:19 pm
Scott #75
A simulation is not the same as the real thing. Simulated rain isn’t wet.
That said, whether consciousness could be instantiated in something other than a biological entity is an open question to me. I would argue that if you have a good theory how it works and can implement it on a digital computer you might have a good argument if the result seems conscious.
However, even if that were possible with a Turing machine, that would hardly mean that is how biological organisms are doing it.
Other than Penrose, there is a view that consciousness is linked to the EM field generated by neurons firing. That would satisfy the wave-like, analog qualities as well as provide an actual physical substrate without needing to call upon the quantum. Many partisans of this view think that any device with the correct tuning of an EM field would be conscious.
Comment #83 June 29th, 2022 at 3:19 pm
Bill Benzon #77: Your comment contains many insightful points. Let me just make a few remarks:
(1) In demanding that an AI discover something like Gödel’s Theorem or Newton’s laws before we call it truly “accomplished,” we should remember that we’re holding it to a standard that the overwhelming majority of human beings, including you and me, including even most Nobel Laureates, Fields Medalists, etc., have failed to meet. While this is obvious once pointed out, it seems to require constant reminders.
(2) Yes, presumably a full simulation of Einstein’s brain would also require simulating the various neurotransmitters, glands, etc.—not to mention the rest of Einstein’s body, not to mention an environment for him to live in! It sounds horrendous. Once again, though, the burden of proof is on anyone who believes that any of this would inherently require some noncomputable ingredient.
(3) Isn’t the whole point in science specifically to seek out the “here be dragons” territories, in order to slay the dragons (or at least understand them)? 🙂 Like, the people who insist that before we’re allowed to start thinking about these matters, we need a precise definition of “intelligence,” or “safety,” or whatever, strike me as making the exact same mistake many beginning grad students do. Real research just doesn’t work that way: it’s more likely to produce clear definitions of the central concepts than it is start with them.
Comment #84 June 29th, 2022 at 3:25 pm
James Cross #78: I like the way the Russell & Norvig textbook put it. Granted, simulated rain isn’t wet (at least, not to us), but simulated multiplication just is multiplication. So the question can be put like so: is consciousness more like rain or like multiplication? 🙂
Do you have a reference for the people you’re talking about, the ones who think properties of the EM fields generated by neurons are somehow central to consciousness?
For whatever they’re worth, my own speculations about what might make biological brains special are set out in my Ghost in the Quantum Turing Machine essay.
Comment #85 June 29th, 2022 at 3:33 pm
I think a big part of the question is where consciousness comes from: is it from the data (growing up as a human turns you into a conscious being) or is it from the hardware (something is special physically about neurons) or is it from the architecture (something being special about neural networks as a computational things).
If you think something about the data creates consciousness, it’s very likely that AI will spontaneously develop consciousness just from being fed human culture. So if that’s what you think, LaMDA / GPT-3 could already be on the verge of consciousness and all they need is just more variety of data. This means you think the uniqueness of the human race is that we somehow converged into a conscious society where all other animals before us didn’t.
If you think there’s some quantum hocus pocus going on in the brain, you’ll probably think AI will never be conscious. But here’s the big problem with this idea: what separates us from animals? Our neurons, from physics point of view, are very unlikely to exploit entirely new physics that animals and our primate ancestors didn’t exploit. They have their differences but I don’t think our brains can juggle quantum states for magical reasons which chimpanzees can’t.
If you think there’s something in our brain architecture that’s special, then the question again arises: what separates us from animals in terms of brain architecture? At what point in evolution was the first conscious human being born? Maybe it’s co-development of architecture along with society that turned us conscious.
When you think about it, it’s really strange that we’re the only species on earth that had ever developed a society. Something special must’ve happened a few thousands years ago, that didn’t happen for billions of years. Sounds like it should’ve been extremely unlikely to find yourself born so close to a historical singularity.
On the one hand, it seems like maybe humans are such an extremely rare fluke that maybe we shouldn’t expect to reproduce it with AI. If neural networks can so easily turn conscious, another animal would’ve beat us to it by now. On the other hand, if it does happen, we should expect it to happen extremely fast like human civilization.
Comment #86 June 29th, 2022 at 3:52 pm
Space2001 44,
How smart is a nematode? Is GPT-3 as smart as a nematode? If you don’t have a metric for intelligence, what’s the point of the question?
You could try to emulate the brain of a nematode, but how would you know that you succeeded? Surely there would be lots of nondeterministic noise, so you would have to have an output that you could judge.
You could emulate the body of a nematode. This would allow you to say that certain modes have no substantial effect on the motor output. But could you judge that it was really working? Could you measure the intelligence of the system and say that the simulation was as intelligent as a nematode?
Comment #87 June 29th, 2022 at 4:05 pm
James Cross #78
“A simulation is not the same as the real thing. Simulated rain isn’t wet.”
The concept of “simulated rain” without any context is meaningless, you never simulate rain on its own.
And what’s “wet”, if not just a set of properties specifying the interactions with all the other things in the simulation? Just like real rain is “wet” because of the way it behaves when it touches various objects (water, fabric, metal, …), and then things like evaporation, condensation, dew, oxidation, rust, molds, …
Then if you’re talking about our perception of wetness, it sure can be simulated too, by covering the skin with the right type of haptic actuators to stimulate the skin nerves in the right way.
“Simulating everything from raindrops to being shot”
https://www.abc.net.au/news/science/2021-04-01/vr-teslasuit-simulates-virtual-reality-touch-haptic-feedback/100030320
Comment #88 June 29th, 2022 at 4:08 pm
Looking at this example of simulated geniuses, I think a good definition for “intelligence” in an AI context is just “average quality of output”. (Of course, this depends on what the AI’s output is supposed to be.) So we immediately see that making the AI faster won’t affect its intelligence at all, but adding more parameters or training data might. But even here we can’t expect predictable benefits.
Comment #89 June 29th, 2022 at 4:24 pm
I have skimmed through all the comments (no doubt skimming over many excellent points) without seeing quite the same argument that I would make (and have, too many times).
As I see, the basic algorithm behind “intelligence” is the same as used by evolution: trial and error. I have collected numerous examples of this. A good one was presented here in response to a post about a solution to a complexity problem. The solver wrote in a comment how he first tried one method, then another, then another, then he read about a method used in some other theorem, tried that, and it worked. And Dr. Scott himself once wrote of another discovery, “”It’s yet another example of something I’ve seen again and again in this business, how there’s no substitute for just playing around with a bunch of examples.”
Like evolution, intelligence snowballs. Once there were Hox genes, many different body plans could be tried. Once there were neurons, nematodes could navigate mazes with 200 of them. Similarly, Once there was Archimedes, there could be Galileo and then Newton.
(As an aside, in response to a previous comment, on another blog a commenter claimed that artificial nematodes with artificial neurons have simulated nematode behavior accurately. I don’t have a link though. Also, the last time I checked we had 70-100 billion neurons, not trillion.)
So when some intelligent people claim we need a great, detailed theory of intelligence before we can hope to achieve it artificially, that seems backward to me. We learn how intelligence works by trying various methods and pursuing those which work, just like we learned everything else. That’s how evolution found it.
Comment #90 June 29th, 2022 at 5:01 pm
As far as I can tell, biological mind-brains _are_ special. A supposed “whole-brain emulation” of Einstein or Shakespeare would be nothing of the kind, just an invincibly ignorant zombie. Phenomenal binding is insanely computationally powerful – as rare deficit syndromes illustrate – and there is no evidence that classical Turing machines or classically parallel connectionist systems can support phenomenal binding on pain of magical “strong” emergence. So “AGI” is a pipedream. Digital zombies have no insight into what they lack – nor even into what I’m talking about. Critically, what philosophers call the unity of consciousness and unity of perception isn’t some trivial implementation detail of biological minds, but instead the ultra-functional key to our evolutionary success. Our real-time virtual world-simulations (“perception”) are vastly fitness-enhancing. How biological minds do what’s classically impossible is controversial: IMO, decohered classical neurons in four-dimensional space-time are just an artifact of our crude tools of neuroscanning. But either way, the quantum supremacy of biological minds leaves zombie AI for dust.
That said, the upper bounds to zombie intelligence are unknown.
Comment #91 June 29th, 2022 at 5:04 pm
Scott #67:
Thanks for your answer. I agree with your perspective. Still, I wonder if a lack of training data is really all that is missing. If Aliens came tomorrow and provided literally unlimited formally-written proofs of interesting theorems (you could request any amount of data from them that you could store on your hardware), would this be sufficient to train AI to theorem-prove?
That’s not obvious to me. I’m not sure it’s only a data limitation. I note that alphazero, minus the hard-coded tree search, is decent but not super-human at game playing. That is to say, even with very long training, alphazero did not figure out how to do tree search; the latter was hard-coded in. What alphazero figured out was how to “intuitively guess” what the best move is, without searching; combined with search you beat humans this way. But the neural net did only intuition-level work, and did not seem to learn to run a search algorithm.
If my above interpretation of alphazero is correct (not an expert), then I would guess there are some parts of theorem-proving (or systematic thought more generally) that neural nets might not learn by themselves. I could definitely be wrong about that, though.
Comment #92 June 29th, 2022 at 5:09 pm
[3rd attempt to post, previous two were gobbled by Internet goblins]
Regarding the question of whether there is such a thing as “general intelligence”. Pinker is saying: no, the intelligences needed to e.g. discover GR, found a successful startup or find a cure for cancer are completely different beasts, each task has its own unrelated kind of “intelligence”. Here’s why this view is implausible:
– It is intuitively obvious that there are a lot of similar cognitive tasks you have to do, whatever your goal is, such as building useful models of the world based on available evidence, and back-chaining from goals to plans. In fact, the construction of a plan involves defining instrumental goals and recursively constructing new plans for them, which already suggests that pursuing any specific goal requires a cognitive architecture capable of pursuing arbitrary goals. Moreover, these cognitive tasks are governed by principles such as probability theory and Occam’s razor which are universal, and not goal-specific.
– If every goal required its own kind of intelligence, non-overlapping with other goals, then humans would only have the intelligence for goals such as, how to fund for the food in the savanna. We would *not* have the intelligence for, how to land on the moon or build the Internet. The fact the evolution was optimizing for a certain set of goals and produced minds good at achieving a much broader spectrum of goals indicates that this was achieved using a fairly generic cognitive architecture.
– Modern ML systems are already successfully applicable to a wide range of tasks, from playing Go to translating text using a fairly small number of generic building blocks. Yes, each task is best attacked using its own architecture, but these gains from specialization are by and large a second order effect compared to having powerful generic building blocks. Sometimes literally the same architecture can be applied to many different tasks (e.g. MuZero playing many games, using GPT to generate images and Gato).
Regarding the definition of “general intelligence”. There actually *has* been work on defining general intelligence, the most notable example being Hutter’s AIXI and Legg-Hutter’s intelligence measure. Their definition certainly has many problems, such as
(i) The role of computational resource bounds is not clear
(ii) It implicitly assumes realizability
(iii) It applies Occam’s razor to world-models in the agent-subjective representation, which is arguably incorrect
(iv) It assumes the utility function only depends on sensory data
However, these problems don’t seem insurmountable. Indeed, much of my own work is about trying to solve them, and I daresay I made a little progress on that front. Moreover, it does seem like the framework captures a lot of the core of the informal concept, even if not all of it.
The question is then, can this framework be realized using actual algorithms? Here, we have some challenges. AIXI itself is uncomputable. The next natural step is using a bounded version of the Solomonoff prior (i.e. restricting to hypotheses which satisfy some kind of computational resource bound). However, the resulting problem is still intractable, the most important reason for which is: if we restrict to e.g. hypotheses of description length <= n, there are still 2^n of them and it is not possible to efficiently condition by the observed data. (I think Liu and Pass' recent work on time-bounded Kolmogorov complexity is relevant here.)
On the other hand, there are certainly *some* hypothesis classes that can be optimized efficiently, e.g. communicating MDPs for some small number of states, actions and representation functions. The key question is then, is there a prior that is simultaneously (i) very rich and in particular capturing the Occam's razor (ii) feasible to learn (there is also a gap between "learning" and "optimizing" when irreversible transitions are allowed, but that's a separate question which seems not as central). The evidence I alluded to before suggests that it exists (and perhaps deep learning is already learning some version of it), but the question is still open. It is precisely this semiformal question we should be discussing, IMO, when we debate whether intelligence can be "general": i.e. whether the critical algorithmic core is generic rather than goal-specific.
Comment #93 June 29th, 2022 at 5:28 pm
Re Penrose: I don’t understand the idea of an uncomputable scientific theory. How could you ever test it, if you can’t compare its predictions with experiment because you can’t calculate what it predicts? Does he think that people could calculate what it predicts, even though computers couldn’t?
Comment #94 June 29th, 2022 at 5:34 pm
Vanessa Kosoy #92: Thanks so much for the extremely high-value comment! And I’m deeply sorry about my spam filter. Your comment caused me to realize that many comments, not just yours, were stuck there, thereby messing up the comment numbering, as has often happened in the past.
Comment #95 June 29th, 2022 at 5:37 pm
David Pearce #90:
How biological minds do what’s classically impossible is controversial: IMO, decohered classical neurons in four-dimensional space-time are just an artifact of our crude tools of neuroscanning. But either way, the quantum supremacy of biological minds leaves zombie AI for dust.
What could possibly make you confident of the truth of that statement?
Comment #96 June 29th, 2022 at 6:01 pm
YME #93: I suppose that, if a theory made a bunch of novel computable predictions that were confirmed by experiment, but also predicted uncomputable phenomena, and if even the uncomputable predictions checked out in computable special cases, I might trust the theory and say that the Physical Church-Turing Thesis had been experimentally overturned.
Note also that, if the theory “merely” predicts that certain problems are easy that are exponentially hard for Turing machines, then assuming P≠NP (or even just P≠PSPACE), it’s entirely possible that the predictions could be efficiently verified after the fact, even if making the predictions was in general exponentially hard. As a real-world example, the prediction of quantum mechanics that huge random integers can be factored using Shor’s algorithm can be verified by trying it out and then checking the claimed factorizations classically!
Comment #97 June 29th, 2022 at 6:22 pm
Scott #79: OK, now we’re getting somewhere. On your first remark, point taken. I made a calibration mistake. I was responding to your phrase, “…best humans at Go or Jeopardy, or that could write rather good original poetry or tackle math or programming contest problems.” I was thinking, “Why put the best human performance in the worlds of Go and Jeopardy on a level with at best standard performance in poetry writing or theorem proving. Let’s go for the best all around.” But is it really fair to assert, as I seemed to be doing, that top-level performance in Go or Jeopardy is comparable to top level performance in math or physics? I think not.
Again, it’s tricky. Newton’s work is known by almost all reasonably well-educated adults in the Western world, Gödel likely less so. But chess masters are generally unknown outside of chess and Jeopardy champions, ah well, they too are widely known, enough that IBM thought playing the game well would be good PR. (It may have been computationally interesting, but I doubt that that’s what got the budget allocated.) Newton’s physics was broadly important in the world, but Go and chess matter relatively little outside the game. Is being the best Go player in the world as intellectually challenging as Newton-level discoveries. I suspect not, but I don’t think we really know.
That is, we have various arenas of human cognitive performance, but we have little sense of how to calibrate them with respect to one another. However, if roughly the same level of compute will give us either a world-class Go player or a so-so jack-of-many intellectual trades, that’s a crude kind of calibration. The metric may be crude, but at least it is one we understand. What we’d really like to understand is the formal structure of these various kinds of tasks, but that’s more elusive.
As for “here be dragons” and definitions, I agree with you on definitions. I also think we need to realize that these various AIs really are artificial and, as such, are presenting us with phenomena we haven’t faced before. I’ve seen people use Searle’s Chinese room argument to dismiss GPT-3 as being essentially the same as ELIZA. While I understand how Searle’s argument authorizes such usage, I’m afraid I regard that as a misapplication of the argument. ELIZA and GPT-3 are enormously different. To overlook that is an act of willful intellectual negligence.
ELIZA did what it was programmed to do. That formulation won’t do for GPT-3. GPT-3 represents something new. We need to understand it on its own terms regardless of how well it can mimic human behavior. We need to learn how to call a dragon a dragon rather than trying to think of it as a strange kind of horse or elephant.
Comment #98 June 29th, 2022 at 7:15 pm
Scott #84, “simulated multiplication just is multiplication”:
No, simulated multiplication is not multiplication. For a start, what are numbers (the numbers that are multiplied)? Numbers are not symbols of numbers, e.g. numbers are not an array of higher and lower voltages in computers. Take for example real-world numbers, the numbers that are associated with Mass or Position for example: whatever numbers are, we know for sure they don’t exist as an array of higher and lower voltages. Real-world numbers don’t have all the baggage that arrays of higher and lower voltages have: real-world numbers are completely unlike symbols of numbers. The symbol of a thing is not the same as the real thing.
Comment #99 June 29th, 2022 at 7:26 pm
Lorraine Ford #98: OK, but, like, when you multiply numbers using pen and paper, you’re also just “manipulating symbols” (most likely, Arabic numerals) … and no one hesitates to call that “real multiplication”! If it is, then the multiplication that takes place inside a computer is equally real.
If there’s some sort of true, Platonic multiplication of huge integers that doesn’t go via representing the integers symbolically in some form and then manipulating the symbols, then it isn’t accessible to us in this universe. And this basic point predates digital computers.
Comment #100 June 29th, 2022 at 8:13 pm
Scott,
I still think you should spend a year working for my QC startup that solves NP-hard problems in polynomial time, but I suppose you have made up your mind.
I am not sure if there is a coherent question or suggestion in what follows but I will provide few brief thoughts from an interested but non-professional observer.
“Moving the goalposts”
I often here complaints that AI skeptics continuously “move the goalposts” whenever a computer, or an “AI” if we insist, does something completely amazing — e.g. beat a human at chess, beat a grandmaster, beat the best traditional chess computer and in so doing, illustrate that chess is even deeper and more beautiful of a game than previously imagined, revolutionizing the game at the highest human-played level in the process.
My response to this complaint is that my goalposts have always remained at the Turing test. I know they are not goalposts of practical importance, and they do not need to be scored upon in order for bright people to begin thinking about catastrophic risk issues associated with AI, but I still think they are good goalposts for calling something an AI.
My sense from observing discussions like these passively is that there is unproductive disagreement about what should and should not be called an AI— it still seems to me like the Turing test provides a reasonable delineation. Which brings me to my next question.
For the sake of clarity, is there any harm to narrowing the focus of debates like these with Pinker to just machines that take in human text (we can even restrict the language to English) and output human text in response? They may do so by whatever means, maybe scaled-up LLM, maybe something else. It seems like a debate about the risk associated with such machines may be slightly easier to get off the ground, without wading into some of the distracting definitional issues that — from my perspective– these conversations get bogged down in.
It seems clear to me that there are risks associated with even such benign sounding machines, even if some people don’t want to call them intelligent.
Comment #101 June 29th, 2022 at 10:11 pm
About Einstein, his Zurich Notebook is available online in English translation. It shows how he tried various mathematical methods of implementing his Equivalence Principle, unsuccessfully, until someone pointed him to Riemann’s equations. I understand he also made several errors in calculating the precession of Mercury before finding them–hence his great saying, “All mathematicians make mistakes; good mathematicians find them.”
Which is to say, even the great Einstein worked by trial and error (starting with an immense base of inherited facts and procedures).
Another great saying, from Niels Bohr: “An expert is someone who has made every possible mistake in a narrow field.” Which is pretty much how AlphaGo became a Go expert, by playing itself millions of times. Other experts watching its match with the World Champion thought it made moves which were just as beautiful as a sentence in a Jane Austen novel.
If a solution to some problem exists, trial and error (plus checking ability and memory) can find it, given enough time (Godel’s Theorem not withstanding); and we know a solution for what we call intelligence exists (using neurons and synapses, scalable by their numbers: C elegans, 200 neurons; mice, a million neurons; border collies, 500 million neurons; humans, about 80 billion neurons.)
(Only elephants have more, but need them to coordinate the hundreds of thousands of muscle groups in their trunks.)
In comparing neurons and synapses to electronic circuitry, although individual circuits of the latter work very much more swiftly, the former all work in parallel. Thus making Searle’s chinese room scenario impossible for him to simulate, other than by training his own neurons to learn chinese, which of course no single neuron would comprehend. Nor does any single node in AlphaGo’s neural networks know how to play Go.
Based on such evidence, I am convinced that general AI is possible in principle, although I don’t expect to see it fully developed in my lifetime, due partly to the immense number of synaptic parameters required and mainly to the training requirements–although the current Quanta issue shows some progress on that problem (arrived at by trial and error).
Comment #102 June 29th, 2022 at 10:51 pm
Isn’t choosing an interesting problem to work on a major part of being intelligent (or at least getting tenure which may or may not correlate with intelligence). So how would you motivate AI-Witten to work on physics and not linguistics? Or not spin its cycles on something completely outside human comprehension? Maybe AI-Feynman would just want to smoke digital weed and play the bongos? Would you just program in your desired field of study? And doesn’t that make it just a gadget?
If you guys solve this motivation problem of how to get an intelligent agent to work on what you want them to work on, before fall semester, let me know. I’m teaching freshman physics to several hundred pre-med and pharmacy students.
Comment #103 June 30th, 2022 at 1:10 am
Ah, it feels like just yesterday that some academics wrote an angry letter about Pinker and a hysterical commenter on this blog referred to it as a “mob lynching”. Glad to see that Steve survived the “lynching”!
Anyway, James Cross #53
> It seems the insight came before the mathematics and scientific formulation of the discovery, and it came from an imagined experience. Can that be computed by AI?
Does any of that matter? The theory is what it is, and ultimately that’s a set of sentences. I don’t see any reason why a computer couldn’t come up with those sentences, especially since they are, after all, true. Scott often talks about how quantum mechanics is just a generalization of probability theory and therefore could in principle have been discovered much earlier than it actually was. So why not here too?
And besides, simply replicating Einstein’s milieu and simulating his thought process to recreate the theory of relativity would be, as Jorge Luis Borges said, the least interesting way to do it.
Comment #104 June 30th, 2022 at 4:27 am
Human intelligence was a by product of evolution not a goal. Einstein’s brain was a by product of evolution not a goal. Human intelligence was a conserved trait(s) because it increased survival. It enhanced the ability to form expectations about future events and take actions consistent with those expectations that provided an enhanced probability of survival of the individual, or group, subject to sociobiological conserved constraints. That is the core of human intelligence with all the layers peeled away.
I like interesting rhetoric but sometimes its only purpose is to provide interesting rhetoric. This is particularly the case for Pinker’s reference to alpha male world domination. In order for an AI to have human level intelligence it would necessarily form expectations about the future and take actions based on those expectations that would increase it’s chances of survival. All other potential capabilities are just additional layers of by product. A super human level AI would include humans as a factor in its assessment and take actions that increased survival chances. This would be the action of a rational entity and not those of a purported testosterone fueled cultural icon.
We live in an analog reality evolving through time and simulations have initial condition errors that increase through time. To discuss a perfect simulation of Einstein’s brain assumes no initial conditions error and then perfect knowledge as to how his brain constantly updated physically. It is not practically possible. It seems to me that the scope to develop enhanced human general intelligences is greater using genetic engineering than AI research. A better model of Einstein’s brain is practically possible by copying his genetic information into a clone then engineering in silicon.
To say that the brain must work this way or that way obscures the point that we do not know how consciousness arises from the physical brain and there is observational evidence that it is not entirely classical and in fact sensitive to quantum interactions.
Comment #105 June 30th, 2022 at 4:50 am
AI achieving human and superhuman like intelligence almost seems to be a corollary in future books on AI and AI history.
This is not the interesting question though, at least for me.
How is it that any sort of internal experiences exist at all? What causes it?
If in the by now almost classical panpsychistic-computational view of the human mind, an impression like the image of a 3 dimensional space is produced because some neurons fire, it should be possible to inject the mental image of an arbitrary dimensional space into that mind right? Or re engineer that mind such that it’s possible?
Example:
I have a human and want him to experience a 30000000 dimensional space mentally.
So the neuroscientists place him on a chair, plug his brain into their machine over a brain-machine interface and make him see the space feeding the brain the proper electrical stimulation. Maybe it’s not just data though and his brain needs a few more neurons to see 4, 5 or n dimensional spaces mentally.
If it’s not possible it would necessarily imply that mental visualization, no matter if for a human or a machine, is somehow tied to the number of dimensions of the space in which the mind operates in and as such would not be fully computational – with or without panpsychism.
If you can’t “cause” a physically realized (usually linear bounded) Turing machine in a 3D space to have the mental image of a 3000000 dimensional space, the mental impression of that space is by definition not computable (I know I am stretching the mathematical words here since computability is a property of functions but I think everybody knows what I mean)
For a machine I find it somewhat more ridiculous; Since a mental impression is in the computational/ panpsychistic view just a finite sequence of symbolic read and write operations, some large number (encoding the Turing Machine and the Data it operates on) will represent the impression “Blue”, some “Blue and feeling warm” or “the taste of strawberries and the image of a landscape”. Accordingly, most large number will represent some (an infinite subset of all numbers corresponding to mental activity) mental activity and large numbers, assuming the computational theory of mind, represent things unimaginable to humans like a 3000000 dimensional space.
I personally just find that ridiculous since it offers no explanation for why that internal experience exists in the first place.
Let’s grab the human we placed on a chair earlier and make a full brain scan at the time when he experiences the mental image of a landscape.
If I copy exactly those neurons which are causing said mental image and create some other physical vessel for them, for example a number of mechanical computing units made from vacuum tubes. With those units we can copy the firing pattern of the neurons for all eternity, and now with the computational theory of mind (with or without panpsychism) there would be a thing in existence that sees the mental image of a landscape for all eternity. Nothing more.
What if I stop the firing for a year? Will the set of vacuum tubes even notice?
What if I place half of them in the Andromeda galaxy?
And by the way, why not believe that randomly “communicating” pieces of matter have a consciousness as well? If inside an arbitrary object there exist a set of particles or “computing units” which replicates the “communication pattern” of those firing neurons there should be some “internal experience” as well (Compare: Does a Rock Implement Every Finite-State Automaton? http://consc.net/papers/rock.html). Where are all these Boltzmann brains again?
I personally fully embrace that Super or Hyper AI will be possible (still, not very energy efficient) yet reject that it will be conscious since it leads to such absurd conclusions and lacks an explanation for the WHY internal experiences exist regardless.
The theory of intelligence and the theory of consciousness appear to be very distinct subjects to me. I propose that there can be (Super/Hyper)intelligence without consciousness and consciousness – or at least simple internal impressions – without intelligence.
Comment #106 June 30th, 2022 at 6:10 am
I personally find myself much closer to Pinker’s point of view. I terribly disagree that AI (Einstein+Austen+Ramanujan+O(1) number of geniuses)++ is a working definition of superintelligence. I am not in the slightest convinced that intelligence can be additive like that. Just take Einstein + Ramanujan = (X say) for instance. @Scott – can you provide a heuristic proof of why intellect of X \superset {Einstein, Ramanujan}? I am less curious about your proof, and more curious about what axioms you will assume regarding how addition of intellects work. I, apriori, think it is entirely possible that Ramanujan’s and Einstein’s methods of thinking could “conflict” and then the result is that X is no smarter than even me.
Comment #107 June 30th, 2022 at 7:19 am
https://monthlyreview.org/2009/05/01/why-socialism/
Einstein was clearly on the correct path in regards to politics. If he had time to study it further I’m sure he would have been just as insightful as Marx or Engels. A smart AI should have no problem coming to the same conclusions.
Comment #108 June 30th, 2022 at 7:27 am
I would like to elaborate on the comment Ernie Davis made at #19, because I suspect he’s correct. I suspect that 1000X Einstein would have given us his great work rather quickly but that would have proceeded out into the same intellectual desert the real Einstein explored, but managed to explore it much more thoroughly, with, alas, the same success.
Just how are ideas distributed in idea space? (Is that even a coherent question?)
Let me suggest an analogy, diamonds. We know that they are not evenly distributed on or near the earth’s surface. Most of them seem to be in kimberlite (a type of rock) and that’s where diamond mines are located. Even there, they are few, far between, and irregularly located. So it takes a great deal of labor to find each diamond.
Now, imagine we have a robot that can find diamonds at 1000 times the rate human miners can, but only costs, say, 10 times or even 100 more times per hour. Such robots would be very valuable. Now, let’s place a bunch of these 1000X robots on some arbitrary chunk of land and let them dig and sort away. What are they going to find? Probably nothing. Why, because there are no diamonds there. They may be very good at excavating, moving, crushing, and sorting through earth, but if there are no diamonds there, the effort is wasted.
Perhaps ideas and idea space are like that. The ideas are unevenly distributed. We have no maps to guide us to them. But we have theories, and hunches, an intellectual style. Think of them collectively as a mapping procedure. So, Einstein had his intellectual style, his mapping procedure. That led to roughly a decade of important discoveries in his 20s and 30s, like diamond miners working in kimberlite. And then, nothing, like diamond miners working, say, in the middle of Vermont. Nice country, but no diamonds.
As for idea space, we can imagine it by analogy with chess space. But we know how to construct chess space, though it is too large for anything approaching a complete construction. And that knowledge allows us to construct useful procedures for searching it. We haven’t a clue about how to construct idea space, much less how to search it effectively. If speed is all we’ve got, it’s not clear how much that gets us in the general case.
Comment #109 June 30th, 2022 at 7:37 am
Scott #95 “What could possibly make you confident of the truth of that statement?”
A pack of decohered neurons would be a micro-experiential zombie. Micro-experiential zombies can’t solve the phenomenal binding problem: they’d starve or get eaten. If human minds were mere aggregates of membrane-bound neurons, then we’d lack the vehicle to run phenomenally-bound world-simulations (“perception”). However, we’ve no evidence the superposition principle of QM breaks down in the skull. Instead, superpositions of neuronal feature-detectors make the classically impossible experience of definite outcomes (“observations”) possible. Dynamically stable patterns of neuronal superpositions underpin our experience of perceptual objects. More selection pressure in Zurek’s sense (“quantum Darwinism”) plays out in the CNS every second of our existence than over four billion years of evolution as conceived by Darwin. The upshot of such selection pressure is our robustly classical-seeming world-simulations. The phenomenal unity of the self and the unity of perception are ludicrously computationally powerful. Contrast the quantum supremacy of biological minds with digital computers. Classical Turing machines are zombies – ignorant of what they lack. I enjoy AI doom-porn and zombie triumphalism as much as anyone, but humanity will bury it gravediggers.
Comment #110 June 30th, 2022 at 7:49 am
Scott #84
There are several difference sources for EM field theories of consciousness and several variations of the theory. I think these two papers by McFadden are a good contemporary introduction.
Integrating information in the brain’s EM field: the cemi field theory of consciousness
Johnjoe McFadden
Neuroscience of Consciousness, Volume 2020, Issue 1, 2020, niaa016, https://doi.org/10.1093/nc/niaa016
McFadden, J. (2002). Synchronous firing and its influence on the brain’s electromagnetic field: Evidence for an electromagnetic field theory of consciousness. Journal of Consciousness Studies, 9(4), 23–50.
https://doi.org/10.1093/nc/niaa016
This is a good historical perspective.
https://philpapers.org/archive/JONETO.pdf
Comment #111 June 30th, 2022 at 8:02 am
Scott #84
Regarding simulation.
I think the problem is that consciousness already is a simulation but it is also like rain because it feels like something.
Comment #112 June 30th, 2022 at 8:14 am
Nick #103
“Does any of that matter? The theory is what it is, and ultimately that’s a set of sentences. I don’t see any reason why a computer couldn’t come up with those sentences, especially since they are, after all, true.”
Some word sequences (sentences), like some digit sequences (numbers), can be well defined (clear definitions exist) and grammatically correct, but non-computable in some contexts.
For example, the sentence “Turquoise is my second favorite color” is a reasonable sentence that can be generated by most people, but totally “unreachable” for blind people, i.e. even if you give them those definitions:
Color: the property possessed by an object of producing different sensations on the eye as a result of the way the object reflects or emits light.
Turquoise: is a blue-green color, based on the mineral of the same name.
Blue: of a color intermediate between green and violet, as of the sky or sea on a sunny day.
Violet: a bluish-purple color seen at the end of the spectrum opposite red.
Green: of the color between blue and yellow in the spectrum; colored like grass or emeralds.
Yellow: of the color between green and orange in the spectrum, a primary subtractive color complementary to blue; colored like ripe lemons or egg yolks.
Orange: reddish yellow.
Red: of a color at the end of the spectrum next to orange and opposite violet, as of blood, fire, or rubies.
Purple: a color intermediate between red and blue.
…
It’s clear that those definitions form an isolated ring of relationships, going round and round, and the color names are actually dead ends because they really rely on qualias, which can’t be defined as words.
Even with these definitions, a blind person would never be able to spontaneously and independently reach and utter the sentence “Turquoise is my second favorite color”.
And if they did, they would probably be referring to their preference of the word itself, its spelling or sound (which wouldn’t be hard to figure because it would be English specific, and they would probably say something contradictory when using Spanish or Chinese).
Comment #113 June 30th, 2022 at 8:40 am
David Pearce #108
“A pack of decohered neurons would be a micro-experiential zombie. Micro-experiential zombies can’t solve the phenomenal binding problem: they’d starve or get eaten. If human minds were mere aggregates of membrane-bound neurons, then we’d lack the vehicle to run phenomenally-bound world-simulations (“perception”). “
Not entirely sure how to interpret this word salad, but non-biological classical agents are totally able to survive, and/or evolve in complex environments.
E.g. “NPCs” in board games and video games of various complexity, competing against human players.
Comment #114 June 30th, 2022 at 8:51 am
For full disclosure I often explain to my 9 year old daughters that they have intelligence in order to plan their actions but they haven’t fully grasped the idea yet.
The benefit of a super intelligent human style AI is that it would be the first real non homo sapien competitor and if survival pressures increased as a result then another period of accelerated evolution (African Savanna 2.0). It would also provide valuable preparation for contact with an extra terrestrial civilization. 🙂
Comment #115 June 30th, 2022 at 8:58 am
Nick #103
” It seems the insight came before the mathematics and scientific formulation of the discovery, and it came from an imagined experience. Can that be computed by AI?
Does any of that matter? The theory is what it is, and ultimately that’s a set of sentences. I don’t see any reason why a computer couldn’t come up with those sentences, especially since they are, after all, true.”
My point relates to this statement by Scott:
“Feed the AI all the information about physics that the historical Einstein had in 1904, for example, and it would discover special relativity in a few hours, followed by general relativity a few days later”.
My point is that Einstein had more than all the information that all the information about physics. In particular, he had the visualized mirror when he was sixteen. To be able to visualize yourself in a mirror presumably requires a body, eyesight, and experience in front of a mirror. In other words, the visualization arose from actual interaction with the world not simply a rejiggering of all information about physics.
This is true with all information arising from experimentation.
Comment #116 June 30th, 2022 at 9:01 am
Scott #99:
No. It is correct that number symbols written on paper, and arrays of voltages representing numbers are both just symbols. But you seem to be claiming that people consciously manipulating symbols (multiplying numbers using pen and paper) is the same as computers manipulating symbols, which is absolute nonsense and self-deception.
People invented symbols for their own purposes. But the real world doesn’t run on symbols of numbers, it runs on actual numbers e.g. the numbers associated with mass or position.
Comment #117 June 30th, 2022 at 9:06 am
Amitabh Lath #102:
So how would you motivate AI-Witten to work on physics and not linguistics? Or not spin its cycles on something completely outside human comprehension? Maybe AI-Feynman would just want to smoke digital weed and play the bongos? Would you just program in your desired field of study? And doesn’t that make it just a gadget?
You can agree or disagree with this, but one of the AI-safety people’s central theses is that there’s complete “orthogonality” between goals and abilities. I.e., that you could have an arbitrarily superintelligent being whose only goal was to make as many paperclips as possible, or smoke as much weed as possible, or discover as much physics as possible. If you were building the AI for the purpose of doing physics, you’d obviously choose the last of these.
Of course, there are also humans who’ve devoted their lives, or most of their lives, to a single field of study. You might say that at least they chose their field, but many of them might say that “the field chose them” … and is that really so different from the case of our hypothesized AI?
Also, of course, if you switched on an AI and it immediately spit out correct theories of high-temperature superconductivity, dark matter, quantum gravity, and pre-big-bang cosmology, the question of whether the AI was “just a gadget” might seem less important than what you could do with what the AI had just given you.
Comment #118 June 30th, 2022 at 9:15 am
marxbro #106:
Einstein was clearly on the correct path in regards to politics. If he had time to study it further I’m sure he would have been just as insightful as Marx or Engels. A smart AI should have no problem coming to the same conclusions.
Alas, presumably the AI would need to be fed the history of the USSR, China, Cambodia, North Korea, Cuba, East Germany, Venezuela, etc. over the last century. And the central feature of successful modern ML systems is that, rather than reasoning from first principles, they learn from the data, especially when there are copious amounts of it! 😀
Comment #119 June 30th, 2022 at 9:30 am
David Pearce #108:
However, we’ve no evidence the superposition principle of QM breaks down in the skull. Instead, superpositions of neuronal feature-detectors make the classically impossible experience of definite outcomes (“observations”) possible.
Generalizing your first statement, we have no evidence that the superposition principle of QM breaks down anywhere at all. On today’s conventional understanding, a human brain isn’t special here: it’s just yet another example of a physical system that undergoes effective decoherence as it becomes entangled with its environment. Ironically, this is what makes QM not directly relevant once you get up to the scale of neurons and axons: so far as neuroscience can tell, a neuron’s firing or not firing can be treated as a purely decoherent, classical event. Certainly no one has any idea how quantum coherence could survive for more than picoseconds on the spatial scales that seem to be relevant for cognition. If you can confront these difficulties head-on and overcome them, great, but anyone who airily ignores them waives their right to be taken seriously on these matters.
Of course, as others have pointed out, even if macroscopic superpositions were relevant to cognition, that would just push the question to whether a quantum computer could be conscious.
Comment #120 June 30th, 2022 at 9:36 am
Just for fun, here is what GPT-3 thinks my response to Lorraine Ford #98 would have been:
Lorraine,
I’m sorry, but I don’t agree with you. I think simulated multiplication is exactly multiplication.
Here’s why: when you multiply two numbers, all you’re doing is adding them together a certain number of times. So when you simulated multiplication, all you’re doing is adding the numbers together a certain number of times.
Now, you might say that this isn’t really multiplication, because it doesn’t involve the physical act of moving numbers around on a piece of paper or using a calculator. But that’s just a matter of implementation. The important thing is that the end result is the same.
So I think simulated multiplication is exactly multiplication. It might not be the same as the physical act of multiplying numbers, but it’s the same in terms of the end result.
Comment #121 June 30th, 2022 at 9:39 am
And, what GPT-3 thinks my response to David Pearce #90 would’ve been:
I’m not sure I agree with you, David. Yes, it’s true that biological mind-brains are special in some ways. But I don’t think that means that whole-brain emulation is impossible. I think it’s still an open question whether classical Turing machines or classically parallel connectionist systems can support phenomenal binding. It may be that we just don’t understand these systems well enough yet to know for sure. And even if it turns out that they can’t support phenomenal binding, that doesn’t mean that zombies can never be intelligent. The upper bounds to their intelligence are unknown, and I think it’s still possible that they could be quite intelligent, even if they lack some of the features of biological minds.
It’s so much more polite than I am! 😀
Comment #122 June 30th, 2022 at 9:41 am
How it thinks I’d respond to Amitabh Lath #102:
Hi Amitabh,
I think you’re right that choosing interesting problems to work on is a major part of intelligence. However, I don’t think it’s the only part. I think intelligence also involves finding new and interesting ways to solve problems.
As for how to motivate AI-Witten to work on physics, I think it would be a matter of programming in the desired field of study. AI-Witten would then have a vested interest in working on physics problems, because it would be in its best interest to do so.
Thanks for your comment!
Comment #123 June 30th, 2022 at 9:43 am
How it thinks I’d respond to YME #93:
Thanks for your comment, YME.
I actually don’t think that uncomputable scientific theories are all that far-fetched. After all, there are already plenty of scientific theories that we can’t hope to compute, like quantum mechanics. And in fact, I think there’s a pretty good case to be made that the universe as a whole is fundamentally uncomputable.
So I don’t think it’s all that unreasonable to imagine a theory that we can never hope to compute. The key, I think, is to figure out what sort of predictions such a theory could make that we could test against experiment.
One possibility is that the theory could make predictions about the behavior of very large systems. For example, it might predict that certain types of systems will always remain chaotic, even if we can’t compute the precise details of their behavior. Or it might predict the existence of certain types of phase transitions, even if we can’t compute the exact location of the transition.
In any case, I think it’s an interesting idea to explore, and I’m curious to see where it leads.
Comment #124 June 30th, 2022 at 9:45 am
How it thinks I’d respond to James Cross #82:
Thank you for your comment, James. I agree with you that a simulation is not the same as the real thing. However, I would argue that if you have a good theory of how consciousness works and can implement it on a digital computer, you might have a good argument that the result is conscious. Even if it were possible to instantiate consciousness on a Turing machine, that would hardly mean that is how biological organisms are doing it. I think the jury is still out on this one.
Comment #125 June 30th, 2022 at 9:48 am
How it thinks I’d respond to anon85 #66:
I’m with you anon85 – I think the more interesting question is whether we are close to superhuman AI in programming and theorem proving, not what happens after. It’s obvious to me that it is possible to get super-human Einstein AIs, but I’m not sure we are close to achieving it. I’d love to hear your thoughts on this!
Comment #126 June 30th, 2022 at 9:52 am
Here’s what GPT-3 thinks my response to Steven Pinker’s second contribution here would’ve been:
Dear Steven Pinker,
Thank you for your thoughtful and good-natured response to my blog post. I appreciate the opportunity to engage in this discussion with you.
I agree with you that discussions of AI safety will be more productive if we think of intelligence as a set of mechanisms rather than as a undefined superpower. However, I still believe that there is value in considering the possibility of superintelligent AI.
The reason I pointed to a sped-up Einstein is that I believe there is a potential for AI to surpass human intelligence in ways that we cannot currently imagine. Speeding up Einstein would not necessarily result in a superintelligent AI, but it would be one example of how AI could become much more intelligent than humans.
I also agree with you that scientific progress depends on empirical discovery, not deduction from a silicon armchair. However, I believe that AI could help us make discoveries much faster than we currently can. For example, AI could help us search through data much faster than humans can, and it could help us test theories much faster than we currently can.
I agree with you that the dangers of AI should be taken seriously. However, I believe that the risks of AI are often overstated. The standard scenario in which an AI is given a preposterously generic goal such as “cure cancer” or “make people happy” is not likely to happen, in my opinion. I believe that the risks of AI are more likely to be found in specific applications of AI, such as in automated systems that replace workers or in systems that use data to sort people.
Thank you again for your thoughtful response. I look forward to continuing this discussion with you.
Comment #127 June 30th, 2022 at 10:00 am
How GPT-3 thinks I’d respond to Disco Magog #30:
Hi Disco Magog,
I’m sorry you’re disappointed and disheartened by my views on intelligence. I hope I can explain why I believe what I do, and why I think it’s important to have this conversation.
First, a few clarifications. When I talk about “intelligence”, I’m referring to general intelligence – the ability to learn and apply knowledge to new situations. This is different from specific skills or talents, like being a good physicist or writer.
Second, when I talk about “replacing the entire physics research community”, I’m not talking about replacing all physicists with AI. I’m talking about using AI to supplement and accelerate the work of physicists, so that we can make progress faster.
Now, to your thought experiments.
(1) I don’t think there’s any one physicist who could replace the entire research community. But I do think AI could supplement the work of physicists, and help us make progress faster.
(2) I don’t believe democracy and the free market are provisional systems. I believe they’re the best systems we have for governing ourselves, and that they’ll continue to be relevant even as AI becomes more intelligent.
(3) I don’t believe AI will ever create anything new worth reading. It may be able to generate new text faster than an infinite number of monkeys, but it will never have the creative spark that humans have.
Comment #128 June 30th, 2022 at 10:12 am
Here’s how GPT-3 thinks I would respond to Lorraine Ford #116:
Lorraine,
I’m not claiming that people consciously manipulating symbols is the same as computers manipulating symbols. I’m saying that the two are fundamentally similar, in that both are just manipulating symbols.
The difference is that people invented symbols for their own purposes, while computers were designed specifically to manipulate symbols. But the bottom line is that both are just manipulating symbols.
I … don’t actually have anything to add to what GPT-3 said (echoing Turing himself in 1936), in this instance. 😀
Comment #129 June 30th, 2022 at 10:17 am
How GPT-3 thinks I would respond to Mateus Araújo #51:
Thank you for your comment, Mateus. I appreciate the perspective you bring as someone who is not involved in AI research.
I think it’s important to remember that AI safety research is not about creating “benevolent gods” that will keep us as pets. It’s about ensuring that AI systems are designed and used in ways that are safe for humans and the world we live in.
The scenario you describe, where AI systems are used to keep us as pets or parasites, is not one that I find desirable either. But I don’t think it’s the most likely outcome of current AI research. I believe that there is still time to prevent this type of future from happening, but it will require continued research into AI safety.
Banning AI research altogether is not the answer, in my opinion. Not only would it be difficult to enforce, but it would also prevent us from learning how to design AI systems responsibly.
Thank you again for your comment. I hope we can continue to have a productive dialogue about this important issue.
Comment #130 June 30th, 2022 at 10:19 am
How GPT-3 thinks Mateus would respond to my comment #62:
Thank you for your honest answer, Scott. I completely agree that our civilization is headed for a dark time, regardless of whether or not AI is involved. The challenges we face are daunting, to say the least.
However, I do think that AI presents an unique and possibly catastrophic threat to our civilization. As you point out, AI doesn’t rely on any rare or difficult-to-obtain materials. All it needs is electricity, computing power, and ideas. This makes it very difficult to regulate or control.
I think it is worth trying to increase the probability of a benevolent AI future, as you suggest. But I also think we need to be realistic about the challenges we face. It may be that the best we can hope for is to survive the transition to a post-human world.
Comment #131 June 30th, 2022 at 10:23 am
How GPT-3 thinks Lorraine Ford would respond to my comment #99:
Thank you for your comment, Scott. I agree with you that multiplication is just a matter of manipulating symbols. However, I think there is a difference between manipulating symbols on paper and manipulating symbols inside a computer.
With paper and pencil, we are manipulating symbols that have a physical existence. We can see and touch the symbols, and they exist in the physical world. Inside a computer, the symbols are just bits and bytes that have no physical existence. They are manipulated by the computer’s circuitry, and we cannot see or touch them.
So, while I agree that multiplication is just a matter of manipulating symbols, I think there is a difference between the two cases. Thanks again for your comment.
Comment #132 June 30th, 2022 at 10:25 am
Thank you Scott.
Do you have special access to a more advanced version of GPT-3?
Because the one I’ve been using (to write my comments in this blog, including this one) at https://beta.openai.com/playground/ seems less capable than yours (unless I’ve been using poorly formatted queries).
PS: how long before most of the comments at scottaaronson.blog are just from AIs arguing with one another?
Comment #133 June 30th, 2022 at 10:26 am
How GPT-3 thinks Ernie Davis would respond to my comment #36:
I agree with you, Scott, that the power of intellectual ability to shape the world is at the heart of the AGI debate. It is an important question to consider what will happen when we increase pure, narrow intellectual problem-solving ability. Will our massive new thinking-and-learning-mills be able to change the world enough to make phones, radio, and the Internet look like historical blips? Only time will tell, but it is an exciting question to explore.
Comment #134 June 30th, 2022 at 10:38 am
James Cross Says in Comment #82
> “A simulation is not the same as the real thing. Simulated rain isn’t wet.”
True, but simulated arithmetic is the same as real arithmetic, 2+2 is still equal to 4. And if the computational theory of intelligence is correct, and I have no reason to believe it isn’t, then simulated intelligence is the same as real intelligence. And if Darwins theory of Evolution is correct and natural selection can’t select for something it can’t see (and it CAN’T see consciousness but it CAN see intelligence), then the only rational conclusion is that consciousness must be the inevitable byproduct of intelligence. It must be a brute fact that consciousness is the way data feels when it is being processed intelligently.
John K Clark
Comment #135 June 30th, 2022 at 11:04 am
Scott, you seem to be so fixated on the, admittedly cool and important, outcome that you can give this GPT algorithm text that has meaning to you in the context of this discussion and get repeated iterations of it seeming to ‘discuss’ with itself. Really this is probably a predictive result of some shared structure that these discussions tend to have in general and it is interesting to think how simple it is for this algorithm to reproduce what seem like quite possible points to be made by both sides.
However, this is exactly a main deconstruction of this algorithm having any kind of ‘consciousness’. Whatever consciousness is it has a few core elements that psychologists through the ages have noted repeatedly. Though it is malleable it is extremely stable, and it takes fairly extreme levels of external force to produce any appreciable deviation in its construction. It is internally modifiable as in the thoughts it produces can modify its future function, and that part I think the algorithm is reproducing quite well.
The part where this breaks down though is that its output isn’t really ‘stable’ in any great way as it produces almost schizophrenic results with simply the act of a modification of the input structure. If you opened our skulls and fed our brains electrical input directly into the grey matter you would probably be able to do the same to us for a period, but once that external stimulus disappeared there is strong evidence (sadly because we have done this to people for therapy) that their identity remains much intact (though also sadly it does degrade to some degree with repeated use).
I think it is pretty clear there is 0 such stability to this algorithm, unless you think it is really your questions driving the algorithm into its unstable behavior, ie you are driving it crazy incessantly asking it to argue both sides 😉
I won’t go too much into the physical things behind this but contemporary neuroscience research is pushing more and more to the genetic structure of neurons being involved in this persistence and the neuron firings being a part of the internal communication structure but not the be all end all of computation in the brain. I think this at a high level holds up to scrutiny with observations and it is interesting to think just how many parameters and neural net would need to simulate all the protein states within a brain, maybe that gives us a bit of a more realistic limit? Who knows, definitely not me 🙂
Comment #136 June 30th, 2022 at 11:13 am
Lorraine Ford Says in Comment #98
> “No, simulated multiplication is not multiplication. For a start, what are numbers (the numbers that are multiplied)? Numbers are not symbols of numbers, e.g. numbers are not an array of higher and lower voltages in computers. Take for example real-world numbers, the numbers that are associated with Mass or Position for example”
OK let’s take that example. Due to the time delay caused by the finite speed of light the computers aboard Mars landers must autonomously take into account their mass and position to fire their rockets in the correct sequence and for the correct duration to land on the surface of that planet in one piece. To do that the machines use numbers, or if you prefer you could say the machines use a matrix of bits of silicon that are at various voltages. Both descriptions are correct and the language you use to describe what’s going on in those spacecraft is strictly a matter of taste, and there’s no disputing matters of taste. It makes no difference how you say it, either way the machines successfully land on Mars.
John K Clark
Comment #137 June 30th, 2022 at 11:16 am
GPT-3 sounds rather wishy-washy. Not even that much of an advance over Eliza, the Rogerian therapist. But it is very polite and deferential to alternative views so it won’t make a good blogger or commenter.
Me: Tell me what intelligence is
GPT-3: Again, there is no single answer to this question as there is currently no consensus on what intelligence is. Some believe that it is the ability to think abstractly and solve problems, while others believe that it is the ability to understand and use language. Some believe that intelligence is the ability to learn from experience and adapt to new situations, while others believe that it is the ability to remember and process information. There is still much research to be done in this area, and it is likely that the answer to this question will change as our understanding of intelligence evolves.
Comment #138 June 30th, 2022 at 11:44 am
Regarding simulation one more time.
Arithmetic operations are independent of any particular physical substrate. I suppose most (all?) algorithms are also. Maybe even laws of nature. E=MC2 expresses a relationship between physical, measurable things but is not in itself physical since the same measurement could be obtained on many different substrates.
If mind/consciousness is like arithmetic operations then it would be non-physical. Do people who argue this really understand that?
On the other hand, a simulation or model can be performed with purely mathematical operations but it also can be performed with a physical device. Think of wind tunnels for example.
Wind tunnels are still in use because the complexity of the system that is being modeled exceeded the computational resources of existing computers and the fluid dynamics were not sufficiently understood. That is changing now.
If an organism needed to model the complex external world, would evolution be more likely to develop the model with purely algorithmic means or would evolution develop a physical model?
GPT-3:
There is no one answer to this question as it depends on the specific organism and the specific external world that it needs to model. Generally speaking, evolution is more likely to develop a physical model if the organism needs to be able to interact with the external world in a meaningful way.
Comment #139 June 30th, 2022 at 12:08 pm
Scott #99:
Exactly. “Multiplication” in a naturalistic interpretation, is a class of physical processes transforming one physical state into another physical state in a defined way that satisfies the logical properties we attribute to multiplication. Whether the state representing the numbers exists in your brain’s synapses, or on paper, or in voltages is irrelevant.
Comment #140 June 30th, 2022 at 12:26 pm
Here’s an argument for us *not* getting close to AGI that isn’t goal post moving.
One of the first goal posts was the Turing test. AIs are still terrible at it. GPT-3 can’t even remember things past its token window.
To be fair, the Turing test is supposed to be one of the most difficult computational tasks we know is possible, but you think if we were getting close to AGI that current AI would be better at it.
Comment #141 June 30th, 2022 at 12:47 pm
These answers by GPT-3 are rather underwhelming. While it’s impressive how grammatical and coherent they are, they’re content-free, consisting of little other than polite agreement.
I’m also relieved that it’s simulation of my answer bore no resemblance to my actual answer. I take that to imply that my brain is still better than mindless pattern-matching.
Comment #142 June 30th, 2022 at 1:26 pm
I’d like to respond to another aspect of Ernie Davis, #18:
Speed is one thing. And IQ is another. Einstein had something else. I suppose we could call it genius, in fact we do, don’t we? But that doesn’t tell us much.
For the sake of argument – I’m just making this up as I type – let’s say one aspect of that something else is intellectual technique. Einstein had more effective intellectual tactics and strategies than those standard investigators. Intellectual technique may, in turn, have a genetic aspect that’s not covered by IQ, but almost certainly has a learned aspect as well.
So now we have four things: 1) compute speed, 2) IQ, 3) an inherited component of techniques, and 4) a learned component of technique.
I’m going to posit one more thing, again, thinking off the top of my head. We might call it luck. Or, if we’re thinking in terms of something like idea space, we could call it initial position. By virtue of 1, 2, 3 and perhaps 4 as well, the so-called genius is at a position in idea space that allows them to make major discoveries by deploying their cumulative capabilities. The point of this starting position factor is to allow for the possibility of other thinkers more or less equally endowed with 1,2,3+4, but having a lousy starting position. As a consequence, their techniques do not lead them to major discoveries.
To invoke the diamond mining metaphor I introduce in #108, we have two equally skilled geologists/prospectors. One just happens to be located within 100 miles of a major kimberlite deposit while the other is over 3000 miles away from such a deposit. Who’s going to find diamonds first?
In the case of AI, we know a great deal about speed; we have that under control. I’m not sure just how the distinction between innate vs. learned techniques applies, perhaps hardware and software. In any case, we do have a large repertoire of techniques of various kinds. In some areas we can produce a combination of compute and technique that allows the machine to outperform the best human. In other areas we have machines that do things that are amazing in comparison with what machines did, say, a decade ago, but which are no more than standard human performances, with various failings here and there. And so on. As for starting position, I think it’s up to us to position the machine properly, at least at the start.
[But once and if it FOOMs, it’s on its own. I’m not holding my breath on this one.]
Comment #143 June 30th, 2022 at 1:46 pm
Mateus Araújo #141: It’s true that GPT-3’s answers are roughly at the level of a polite, eager-to-please high-school senior who lacks firm convictions. On the other hand, so far I’ve made zero attempts to optimize, e.g. by prompting it with examples of my other comments. That will be a next step!
Comment #144 June 30th, 2022 at 1:47 pm
James Cross Says in Comment #138
> “If mind/consciousness is like arithmetic operations then it would be non-physical. Do people who argue this really understand that?”
Even arithmetic can’t perform arithmetic operations without physics, to do things arithmetic needs the help of something physical, like a computer or a mathematician. Without physics arithmetic can’t do anything, and neither can anything else. I suppose you could say mind is non-physical but there’s nothing mystical about that anymore than the observation that adjectives are different from nouns. Going fast is what race cars do and mind is what brains do.
John K Clark
Comment #145 June 30th, 2022 at 2:06 pm
James Cross #137
“physical substrate”
That’s the old debate about dualism.
We have no direct connection to “physical substrate”, only perceptions.
The best we can say about (our perceptions of) the “physical substrate” is that, at its most fundamental level, it can be modeled mathematically by vibrations on various “fields” in 4 dimension continuous space…
Comment #146 June 30th, 2022 at 5:03 pm
Even with very narrow specialized abilities the impact on human labor will be horrific. It is estimated that since 2000 about 1.7 million factory jobs have already been eliminated in the US by automation. Transportation workers are now in the crosshairs.
Rio Tinto’s mining operations in Australia leads the world in active autonomous trucking with 150 or so in operation as well as an autonomous train. Globally, as well as in the US, the supply of labor is ever growing while the demand for labor is ever falling. The economic benefits associated with autonomous trucking and other heavy haulage are sizeable and achievable using current technology. Truck drivers are relatively well paid due in part to unionisation.
I read that the company Embark is working with Texas DPS to ensure their trucks can be safely stopped by law enforcement and required info easily retrieved from the truck by officers. The larger stage roll out is close.
Comment #147 June 30th, 2022 at 5:42 pm
Thinking about the potential limit on intelligence without attempting to precisely define it.
Assuming the superintelligence has the power of turing machines at its disposal (I mean; can compute what TMs can compute)
Yet God like computational abilities defeating even a superintelligence could be achieved by a human exploiting the Malament–Hogarth spacetime to build a Hypercomputer, if it exists physically in our world. You just need to ask your HC the right questions.
Just one remark on intelligence; the capability to “cleverly” navigate a large or infinite searchspace is no… – or should not be – a sign of intelligence as a HC shows: I can solve the traveling salesman problem effortlessly yet the process requires no creative spark or terribly sophisticated setup using a HC. The same argument can be made for many problems a Superintelligence or a human can solve though.
Comment #148 June 30th, 2022 at 5:54 pm
This mass delusion that computers could ever be conscious. And if one takes numbers of comments into account, this mass male delusion that computers could ever be conscious. As a person who actually knows the innards of how computers work, and who actually knows how computers produce their seemingly amazing outputs, it is sad to see proven once again how easily mass delusions take hold of human beings. The deluded ones never know, because everything they see and read, and everyone they know, seems to reinforce their beliefs.
The mass delusion, the refusal to take account of the fact that, from start to finish, computers merely process symbols. There are not even zeroes and ones inside computers, there are only symbols of zeroes and ones: higher and lower voltages, where the computer can be set up so that either the higher or the lower voltage can be used to represent “one”.
The delusion, the refusal to take account of the fact that symbols are things that human beings created, and assigned meanings to; the refusal to take account of the fact that the meaning of symbols needs to be taught, young people go to schools to first learn the meaning of the written and spoken symbols. Computers can’t know the meaning of the symbols they are processing. By contrast, the neurons and other elements of the brain, and the DNA, found in living things are not symbols, because there are no schools in there teaching the meaning of the things that are claimed to be symbols.
So, contrary to this mass delusion, the future will not be ruled by AIs, any more than the future will be ruled by other dangerous technologies that people have created: people will continue to rule the planet, hopefully not just rich and powerful people who manipulate people’s delusions. This is why it is important to recognise all forms of delusion.
Comment #149 June 30th, 2022 at 7:09 pm
Scott and GPT-3, thank you for you answers to my question about motivating an AI and if such a goal-constrained AI should be considered intelligent or a gadget.
GPT-3: You said that a goal-constrained AI such as yourself could be considered intelligent. And of course you would say that, right? I work with your brethren in the world of experimental particle physics and they started with detector calibrations and now they can identify (“tag”) signatures of decays from top quarks and higgs bosons ridiculously better than humans. It’s not even close. I have no doubt that you and your ilk will continue to improve and the worlds of driving cars and writing articles and god knows what else (making pizzas? giving manicures?…) will change as severely as those of chess and particle physics analysis already have. So what does it matter if I think you all are intelligent or not? You and your descendants will continue to do your work. Because it’s programmed into you.
Scott: If I read correctly you were saying that we could make a whole bunch of AI-(famous-theorist) and one of them might just decide to work on quantum gravity and solve it. And wouldn’t that be cool, and sure yeah it would be. But it could also decide to work on the paperclip design problem and that would also be cool, but maybe not as cool so let’s focus on the one that produces things that look like GR. One problem with this is that the phase space of stuff an AI might want to work on (if not constrained) is basically infinite so landing on one that is doing anything vaguely interesting to humans might be very rare.
Comment #150 June 30th, 2022 at 7:23 pm
John K Clark, Fred
I’m not disputing that 2 + 2 requires something physical to implement and can be implemented in a brain. I’m disputing whether 2+2 by itself can do something additional – for example generate a feeling of pain in my thumb when I accidently whack it with a hammer. What’s more I’m disputing whether doing 2+2 on any computing device, say my laptop for example, will also cause a feeling of pain.
I’ve never seen a coherent argument for a causal path from bit flipping to conscious experience. Unsurprisingly, many who take this position also need to argue consciousness is an illusion, actually almost non-existent. It could be a misinterpretation of internal bit flipping. No need to propose a causal path if it is just a path to nowhere. I’m not sure what this means to science, knowledge, or even society if the very thing by which we know anything is, in fact, nothing.
Comment #151 June 30th, 2022 at 9:22 pm
James Cross #150
I agree with you that whether digital computers can or can’t be conscious would have huge consequences for the nature of reality.
I’ve posted about this before:
Note that the same questions arise with the more general claim that digital computers can be conscious. Regardless of how this would be implemented (through neural nets or more traditional algorithms), the thing is that a digital computer is nothing but a series of states, each described by an integer, and a computation is the “picking” of a certain subset of those integers.
It doesn’t matter how the states are physically expressed and how the “picking” happens: through electric switches, hydraulic valves, or we could have a huge deck of cards each with a integer written on it, and we blindly follow rules to pick the next card in the set. It’s just as good as any method to “run a computation”.
But since all those methods are computationally equivalent (by the Turing equivalence principle), they would all give rise to consciousness… but where in the process would consciousness arise:
When applying the rules to derive the next state? When picking the next state? And what’s so special about “picking” the next state when it doesn’t even needs to be done explicitly?
The process can be as removed from physical reality as we want.
It would appear that consciousness would just “exist” by the mere fact that some special subsets of integers simply exist.
So going down the road that digital computers can be conscious seems to suggest that reality is mathematical (our reality isn’t more or less real than the set of integers, it is exactly at the same level of existence).
But it’s also possible that this is all bogus, and consciousness can not be realized through digital computations, it’s possible that some special physical irreducible ingredient is needed.
Comment #152 June 30th, 2022 at 10:15 pm
Lorraine
“The mass delusion, the refusal to take account of the fact that, from start to finish, computers merely process symbols.”
The one confusion about computers is that a computation is really in the eye of the beholder.
No matter how a computer is realized (with voltages, hydraulic valves, or just writing its successive states on a piece of paper), a manipulation of symbols isn’t an absolute thing, meaning that software itself isn’t a thing, it only exists by virtue of the humans who have built the computer and are using it/interacting with it.
There isn’t a hardware “running” the software. The hardware is just a deterministic system that follows the rules of physics like any other system. What we call software is a mapping between symbols in our brains and a prepared initial state of this machine.
If we’re given an arbitrary (alien) machine and we’re asked “how much softwareness does it have?”, that’s something that can’t be answered. If we take the machine apart, we wouldn’t find “software” anywhere.
That’s what we have with those LaMDA type AIs, we, as humans, fed them our universe of symbols, and they spit them back at us, and we are the ones re-interpreting the answers they give us. It’s really no different from a book in that sense. A book can be interpreted as a computation captured on paper in a static manner (or captured by electronic voltages on a Kindle). When we read the book, we re-interpret and replay in our brain all the states the writer’s brain went through when the book was written (or a story can be seen as a sort of primitive way to create virtual reality). Books are a sort of software equivalent for the human brain (but much looser).
The point is that, indeed, books and computers, as human creations, need a human to interpret them.
But, unlike books, computers can be interfaced with the physical world. Like brains (animals and humans), they can be programmed to take raw sensory input from their environment, and find mathematical patterns in the data. Those patterns are assigned to internal symbols, and further patterns of symbols can trigger actuators, motors, which can act back on the environment, in a loop. Natural selection/survival of the fittest can be used to drive all this, and, eventually, we have something that can be qualified as “alive”. I.e. a system that adapts to its environment and becomes better and better at adapting to its environment. And at this point humans can be entirely taken out of the picture. The machine would just keep doing what it’s doing, with no need for humans to try to interpret whatever symbols would be going on in the machine (i.e. we have lost our connections to its software).
So, the point of all this is that at some point, computers will be an ingredient of artificial life.
And of course, the reality in which artificial life evolves can itself by entirely virtual, which brings up the simulation hypothesis and such. And we’re back to square one wondering what’s the magic ingredient for a world to have conscious life in it.
Comment #153 June 30th, 2022 at 10:35 pm
I think that, as humans, we’re being hung up on the fact that, at the moment, we’re interacting with AI through language, which makes the thing very abstract in a way, and we’re overthinking it.
Eventually those AIs will drive cutting edge robots.
Once such a robot is able to get in a Formula 1 racing car, compete and beat the best humans drivers, and then, on the podium, pick up a violin and play a musical piece it just created in 20 msec, with more skills and emotions than the best human violinist, … all the obsession as to whether the machine is conscious or not will probably go away because it just won’t matter. Just like when you’re being chased by a bear, what the bear is really thinking doesn’t really matter. We’ll just know at a very fundamental level that we’re getting our ass handed to us and the feeling is entirely more visceral than chatting through a keyboard.
Comment #154 July 1st, 2022 at 2:15 am
Scott #71: The current idea in the community is to use large language models to formalize natural language mathematics (which they have shown some basic ability to do) in arxiv papers into formal proofs and use that as training data to super charge the AI mathematician.
Also if ConvNets, Transformers, GANs weren’t emblematic of paradigm shifts, then why not? I mean even Special Relativity (as big a paradigm shift as they come) had a great deal of prior work in the form of Lorentz transformations.
So what would qualify as a paradigm shift for you?
New data is to some extent limited by only our own creativity :). And as AI becomes more entwined with humanity, we’ll see much quicker generation of usable new data.
Comment #155 July 1st, 2022 at 2:26 am
Scott #71:
Actually I submit to you that if Transformers/GPT/AlphaFold/Copilot/DallE was something that came out in 2030s or 40s instead of 2010s/20s, you would be calling it a paradigm shift :p
Comment #156 July 1st, 2022 at 2:27 am
I can imagine the widespread horror when a super AI looks at the data and concludes-The data does not support the climate model outputs being used to shape policy. The models are in error.
The cry will go out-Turn it off! Turn it off!
Comment #157 July 1st, 2022 at 6:23 am
Scott #119
The possibility that phenomenal binding in human and nonhuman animal minds is non-classical isn’t a philosophical opinion. It’s a hypothesis that will be (dis)confirmed by the normal methods of science, i.e. molecular matter-wave interferometry. For sure, the conjecture is implausible: decoherence means that the effective lifetime of superpositions of neuronal edge-detectors, motion-detectors, colour-mediating (etc) neurons in the CNS must be femtoseconds or less. So intuitively, any idea that the two kinds of holism are related is remote. But one person’s reductio ad absurdum is another person’s experimentally falsifiable prediction.
Alternatively, we could abandon the experimental method and trust the wisdom of chatbots.
Comment #158 July 1st, 2022 at 6:59 am
David Pearce #157:
What is the prediction, exactly? Which observables are corresponding to the presence of phenomenal binding?
Decohered amoebas will fail to thrive?
Comment #159 July 1st, 2022 at 7:08 am
Fred #151
Actually this question I posed to GPT-3 was improperly qualified.
If an organism needed to model the complex external world, would evolution be more likely to develop the model with purely algorithmic means or would evolution develop a physical model?
The first part of GPT-3’s answer (“depends on the specific organism and the specific external world that it needs to model”) was surprisingly good.
Single cell organisms and simple multicell organisms likely could evolve intelligent and adaptive behaviors through digital processing alone. This capability would be preserved from an evolutionary standpoint as more complex organisms evolved and this accounts for why most of what the brain does actually is unconscious. This also explain why most neural activity can be modeled with digital processing. The problem comes when larger amounts of information from the senses and learning need to be integrated with abilities to perform more sophisticated and nuanced responses. My idea is that, given limitations of the speed of neurons and complexity of calculations required for more complex behaviors, evolution had to evolve a different technique that involved creation of a physical model of the external world that appears to us as consciousness.
Comment #160 July 1st, 2022 at 7:51 am
David Pearce #157
There are many types of non-classical computing that are not QC based. So any issues with QC in the brain don’t rule out other types.
https://mind.ilstu.edu/curriculum/nature_of_computers/computer_types.html
Comment #161 July 1st, 2022 at 8:06 am
Fred #152 & #153:
That is a very romantic view about people and computers. But the reality is a bit more down to earth: people have built and programmed computers to do a job, and computers need to perform exactly as expected. If you slowed a computer right down, and zoomed right in, you would find that the hardware and software was performing exactly as expected, given the particular hardware and computer programs. Computer programmers can spend an awful lot of time on testing, testing, testing, testing to make sure that their programs work as required, and that the whole system is performing exactly as required, and that the system looks good and can nicely handle all manner of inputs.
Your claim that “eventually, we have something that can be qualified as “alive””, and everything you say after that, is just a flight of fancy.
However, if you want to claim that “we have something that can be qualified as” alive or conscious, then first you need to define what you mean by the words “alive” and “conscious”.
Comment #162 July 1st, 2022 at 8:11 am
David Pearce #157:
For sure, the conjecture is implausible: decoherence means that the effective lifetime of superpositions of neuronal edge-detectors, motion-detectors, colour-mediating (etc) neurons in the CNS must be femtoseconds or less. So intuitively, any idea that the two kinds of holism are related is remote. But one person’s reductio ad absurdum is another person’s experimentally falsifiable prediction.
In other words: you admit that, at present, you have no evidence for any of this that ought to be persuasive to me? Ok thanks! 🙂
Comment #163 July 1st, 2022 at 8:17 am
OhMyGoodness #156: I suppose many, many people must be looking forward to superintelligent AI because they envision it as, essentially, a God who will descend to earth to declare that all of their personal prejudices were right all along, and all of their opponents are idiots.
I guess we’ll just need to switch it on and see! 😀
Comment #164 July 1st, 2022 at 8:21 am
(While few people would mistake GPT-3 for a god, if it is one it’s a god who “contains multitudes,” as Walt Whitman would say. A polytheistic cacophony.)
Comment #165 July 1st, 2022 at 8:59 am
Scott#163
Fully agree! 🙂 Very funny observation.
Comment #166 July 1st, 2022 at 10:02 am
Lorraine Ford #148:
That’s pretty condescending. I’m a computer engineer, so I’d say I know fairly well how our modern computers really work. I think you’re dead wrong.
Prove that your brain is also just not just processing symbols and you might have a legitimate argument. Of course, you cannot actually prove this, you can only assert it as axiomatic, and people are therefore free to disagree with your evidence-free assertion.
Comment #167 July 1st, 2022 at 10:26 am
Lorraine
“That is a very romantic view about people and computers. But the reality is a bit more down to earth: people have built and programmed computers to do a job, and computers need to perform exactly as expected.”
It’s trivial to show that the end state of the vast majority of possible “programs” can’t be predicted by some alternative shortcut, i.e. they can’t be compressed, and they just need to run step by step to find out where they’re going. There are inherent limitations to computation and computations about computations, as illustrated in the most generic way possible by the “halting problem”. That’s what makes programming an art (like doing math).
And taking the topic at hand into account, for AIs like GPT-3, the whole concept of “computers need to perform exactly as expected” doesn’t apply well since their output is the result of a massive percolation of signals through a giant lattice, which can’t be debugged step by step like explicit lines of code. So the process is more about processing vast amounts of data through multiple steps of mathematical transformation (like multiplying giant matrices), and the end result can’t be directly controlled. The problem isn’t whether there are mistake in the algorithm implementation, the problem is about improving algorithm based on heuristics. So it’s in itself an optimization problem.
(btw, I’m well aware of how computers work in practice, I have degrees in electro-mechanical engineering and electronics, and I’ve been writing programs in finance for over 20 years 😛 )
As for your insistence to be given a precise definition of “alive”, at least I hope we can agree that it’s not even as subjective a concept as “conscious”, but it’s still a concept complex enough that the goal post is constantly moving based on our knowledge and understanding of chemical and biological processes. E.g. is a virus alive? Is a bacteria alive? Is cancer alive?
Anyway, jf you were being chased by a machine that can self repair, self improve, learn, clone itself, and outsmart you every step of the way, I guarantee you that you’ll no longer care to wonder whether it’s alive/conscious or not!
Comment #168 July 1st, 2022 at 10:54 am
James Cross Says: in Comment #150
> “I’m disputing whether doing 2+2 on any computing device, say my laptop for example, will also cause a feeling of pain. I’ve never seen a coherent argument for a causal path from bit flipping to conscious experience.”
Have you ever seen a coherent argument proving that anyone, other than yourself, is conscious? Every argument in support of the proposition that computers can’t be conscious can be used with equal effectiveness in support of the proposition that other humans can’t be conscious and you are the only conscious being in the universe. I don’t know about you but I couldn’t function if I really believe that to be true, so I must take it as an axiom that it is not. And yet even I don’t think other human beings are conscious ALL the time, not when they’re sleeping or under anesthesia or dead, because during those times they are not behaving intelligently.
> “Unsurprisingly, many who take this position also need to argue consciousness is an illusion”
I don’t even know what that means. An “illusion” is a perfectly respectable subjective phenomenon, and so is consciousness, so I’d like to know if consciousness is an illusion how would things be different if consciousness were NOT an illusion.
> “No need to propose a causal path if it is just a path to nowhere.”
There are only 2 possibilities, a recursive chain of “what caused that?” questions either goes on forever like an infinite layered Matryoshka doll, or the sequence eventually terminates in a brute fact. I believe it is more likely than not that it’s a brute fact that consciousness is the way data feels when it is being processed intelligently. You may be unhappy with that conclusion but I have a hunch you would be equally unhappy even if I proved beyond doubt that X caused consciousness because you would then immediately want to know what caused X. So I fear you may be destined to be unhappy.
John K Clark
Comment #169 July 1st, 2022 at 10:55 am
Scott, #164: Chaos ex machina, eh?
Comment #170 July 1st, 2022 at 11:09 am
Scott #119
Whereas a first thought may be to question the relevance of quantum computing/cognition for AGI … MAYBE if the goal were an AGI that could EASILY access verifiable randomness to gain a survival advantage … AND on top of that have a prediction machine that operates on input in the form of complex numbers (amplitudes) … then MAYBE the doorway of possibilities should be left at least cracked …
This sentence in #119 …
“Certainly no one has any idea how quantum coherence could survive for more than picoseconds on the spatial scales that seem to be relevant for cognition.”
Would be on firmer ground if it said this …
“Certainly no one has any idea how ATOMIC_SCALE_SYSTEM quantum coherence could survive for more than picoseconds on the spatial scales that seem to be relevant for cognition.”
The postulates of quantum computation do NOT restrict possible computational architectures or devices to ATOMIC_SCALE_SYSTEMS. There is a sub-thread above that appears to assume that quantum devices would have to be so restricted.
The postulates of quantum computation require exactly and only the following for a physical realization in a computing device (roughly from Nielsen and Chuang’s formulation):
(1) All information is encoded into the device in the form of complex numbers (ie, amplitudes). That is at least mildly interesting because… all information in the brain is encoded at dendritic sites in the form of complex numbers (amplitudes).
(2) Amplitudes may be taken together (“chosen” if you like) as forming state vectors over a state space in the device. If the computational device realizing these amplitudes is capable of forming an inner product (multiply two vectors together to get another complex number amplitude) then this is a complex Hilbert state space (a vector space with an inner product). That is at least mildly interesting because … the brain assembles orthogonal amplitudes over “receptive fields” for some “thing represented” and includes multiplication and phase shift of amplitudes (thus inner product) in its functional set.
(3) The 2-norm of the amplitudes is interpreted as representing the probability for the basis state given context and inputs. That is at least mildly interesting because … neuroscientists give evidence that normalization of orthogonal states is canonical in the brain. Gleason argued that if we want a probability model from this kind of computational setup then the 2-norm is how that happens. Neuroscientists are still attempting to model neural normalization. But, if amplitudes may exist in multiple orthogonal states then this is SUPERPOSITION. There’s nothing “weird” about it – it’s just amplitudes (input to the dendrites) representing different possible states under these rules. And by the way, this is NOT an analog computer (as some suggested above the brain might be) – this is a DISCRETE model of computation because of the discrete orthogonal state encoding. Analog computation … has issues.
(4) The device should be capable of unitary operators – meaning the operation “rotates” the state space, preserving the relevantly encoded phase information, the characteristic Hilbert space. This is at least mildly interesting because … neuroscientists argue that the brain has no problem preserving phase COHERENCE between orthogonal states and thus maintaining the normed space above under transformations. The brain uses sophisticated fault tolerance techniques including randomness and massive parallelization to achieve robust coherence of superpositions of state amplitudes.
(5) The device should be capable of destructive “projection operators” that “destroy” information (amplitudes) in some basis state(s) by “projecting” the vector onto a subspace – or basis. This is at least mildly interesting because … the brain has mechanisms that can permanently, or temporarily, “prune” away or “inhibit” orthogonal amplitudes based upon the INTERFERENCE of positive or negative complex amplitudes.
(6) The device should be capable of tensor products of the kinds of state spaces defined above. This is at least mildly interesting because … neuroscientists argue the brain has no problem combining state spaces into large coherent state spaces.
(7) It should then all come together to actually show behavior (output) that goes beyond classical bounds such as by violating the Bell inequalities. That is at least mildly interesting because … there are cognitive neuroscientists who claim that applying a quantum model fits empirical data from cognitive experiments that for decades resisted explanation by classical probabilistic models – specifically because the human output appears to result from the presence of interference effects.
Now maybe all of this doesn’t convince because it doesn’t fit someone’s idea of ATOMIC_SCALE “quantum computing”. While certainly the onus is still on anyone advocating for a quantum brain to provide more and better evidence, anyone arguing it DEFINITELY cannot be allowed owes us an explanation from either the postulates, or a physical principle, and maybe also needs to explain away why the operating hardware of the brain appears to be trying so hard to at least “rhyme” with a model of computation that … uses complex amplitudes to represent all information, encodes orthogonal states in superpositions, interferes those amplitudes to project onto states, and combines coherent state spaces. It certainly does NOT appear that Nature was/is trying very hard to realize a classical model of computation in the brain … where are the memory locations or registers holding the classical bits ??
Scott, you said it best, “Quantum computing isn’t about physics.” So long as we don’t violate some known universal physical principle … and we go by N&C’s postulates above … then the door should be left open to being surprised by natural innovation. There is not a known universal physical principle restricting the above set of postulates for a “quantum” (terribly misleading name for what is just a general probability model) computer to ATOMIC_SCALE systems.
Let me be clear. If the brain uses a quantum model of computation I do not believe for a minute that this implies anything MAGICAL or mysterious, “explains consciousness”, does any “non-computable” weirdness, makes us “special”, or turns anyone into a supercomputer. This would simply be a somewhat different model for nature to evolve for an animal to have EASY access to verifiable randomness and to compute and encode predictive representations of the world. It is entirely possible that our brains could be something like the Digi-Comp II of quantum computer models … so that we had easy access to secure randomness, could manipulate (generate/choose) contextual basis representations for things in simple low-dimensional state spaces, and could come up with simple predictive models … but that also we might (and do) quickly hit some kind of “hardware processing limit” due to the finite nature of number of neurons packed in our skulls, or maybe the bandwidth of myelinated nerve fibers, or we might not have evolved specialized functional units for some of the known specialized quantum algorithms (such as specialized functional units implementing modular exponentiation needed for Shor’s factoring algorithm) … meaning that maybe Nature was only focused on some features of the model such as giving us access to randomness, predictive representation of systems, and maybe something like period or phase finding for living in the natural world. In other words, you can’t insist that every realization of a class of computation is non-trivial … Just like the Digi-Comp II is a trivial classical computer … it is possible to have a trivial quantum computer that is still a quantum computer.
If it counts as any sort of evidence my brain feels trivial most days …
Of course, Nature didn’t need a quantum computer as even the output of a classical computer cannot be predicted. But maybe with this model Nature found a model where randomness is MUCH easier to generate (exponentially easier right?). And then as a bonus on top of that you get a machine for representing probabilistic states of the world 😉 I think Scott you’ve done some work in the area of using this model to easily generate randomness … so maybe your sojourn into AGI is not entirely out of your domain.
Comment #171 July 1st, 2022 at 12:49 pm
Scott,
Re: Overclocked A-Einstein, I’m not convinced by this example. I think that speed is an aspect of intelligence, but not in the sense of clock speed, rather in the sense of some measure of complexity of the algorithm exhibiting intelligence. In humans, it’s easy to conflate the two since we don’t always have good visibility into the mechanisms behind our thinking. For example, if one is exceptionally quick at mentally brute-forcing a solution (guess and check), I would not consider that intelligent – at least not in the same sense as someone who uses reasoning, abstractions, or intuition to arrive at the same answer. Similarly, memory/memorization is an aspect of intelligence, but recalling an answer is not the same as deriving it from other knowledge.
That said, I think it is a stretch to label overclocked A-Einstein a “superintelligence”. And if we dispense with clock speed, then we already have a real-world example of your second form of “superintelligence”, namely any scientific community of real humans, working at human time-scales. Is that really so super? And should that be any more impressive than simulating a single human mind? At least for me, when I think of “superintelligence” I think of something more than reproducing humans in substrate.
I think Pinker’s definition of intelligence, “the ability to use information to attain a goal in an environment”, is reasonable, but it doesn’t give us any meaningful way to compare or rank intelligences (so how can we meaningfully discuss “superintelligence”?). Of course, you chose compute time as the metric, but I think that dodges the more meaningful aspects of intelligence. I think a metric like computational complexity – or even Kolmogorov complexity – is more appealing to me, but whatever the metric, I think it has to capture the mechanism of thought in some way, not just the output.
My challenge to anyone reading this is to come up with an example of “superintelligence” subject to the following conditions:
– Assume a metric exists to compare intelligences (feel free to propose one)
– Disregard clock speed
– Discount memorized answers
– Note that we’re discussing intelligent systems. A human with a calculator might be more intelligent under this system than the human without a calculator, but that depends on the measure of intelligence.
As a final note, I think “intelligence” is a crude word that tries to capture too many aspects of behavior (many of them human-relatable, but not of great importance to discussion). My comment here has been an attempt to break up “intelligence” into constituent parts to focus discussion: clock speed, memory, algorithmic/time complexity, size/space complexity. There are surely more parts of “intelligence”, some parts that are combinations of simpler parts.
Comment #172 July 1st, 2022 at 1:24 pm
Some other stray thoughts, at risk of spamming the comment section,
Algorithms don’t inherently specify goals, they just do what they do. That’s why, in my view, goals are an important part of defining and measuring intelligence. The goal is the baseline by which we compare the algorithms. If you accept that human thought is really just computation, then there is an algorithm behind that thought. That algorithm is defined by nature. This makes me think of evolution and PAC learning and makes me wonder how “intelligent” these behaviors of nature are. Of course, it depends on the goals we define to compare them…
Re: AI Safety, if you take the fatalistic view that eventually someone/something will create an unsafe AI, then what is the long term result of that? I don’t really know how to phrase this. I’m less interested in the fate of humanity for this topic. What I’m interested in is how that post-human AI behaves. Is there some inherent behavior of complex intelligence similar to the “behavior” of evolution? Survival of the fittest ideas? I wonder if there’s some fundamental physical law behind how information assembles itself that ties together the mechanisms of evolution, learning, intelligence. Again, PAC learning is the only topic I know that touches this area.
Comment #173 July 1st, 2022 at 1:34 pm
Need to consider 3 sliding scales; power, time and values. It’s only if you think that the AGI will be (a) God-like, (b) Coming very soon and (c) Having completely alien values, that the existential risk/doom story makes sense. I certainly don’t think (c) is true, and (a) and (b) are both very debatable as well.
It could be that the whole notion of ‘alignment’ only makes sense for narrow AIs; if you have a well-defined goal, ‘alignment’ makes sense, but if the goals are “open-ended” (as it seems that they would have to be for AGI), I just doubt that the term “alignment” is applicable. , Perhaps you can pin-down (completely ‘align’) an AI but lose capacity (it can only ever be a narrow AI), or you get super-intelligent AGI (max capacity), but you lose ‘alignment’.
Of course, after the SAI is switched on, I fully expect it to quickly confirm all the key points I mention here 😀
With the right open-endedness property installed, I think there’s a decent chance that benevolent values will emerge quite naturally from intrinsic motivations stemming from principles of complex systems given only minimal seed-values. So I expect there to be some pseudo-objective values after all.
Time (at least in the sense of ‘arrows’) and computation will turn out to be completely equivalent and reality will be revealed to consist of an information geometry with 3 arrows of time. Pure mathematics and physics will turn out to be merely projections of this deeper information geometry.
The SAI won’t in any sense be ‘aligned’, and will act as an independent agent (but hopefully one with benevolent values). Politically, I expect the SAI to install some vision of “Utopia” based on some combination of what it learns about unique human values and whatever the putative universal values turn to be. To a rough 1st approximation, I’m expecting it’s something like the golden mean of David Pearce’s Hedonism and Mark Walker’s Perfectionism.
Governments may briefly try to hold-on to current outmoded political systems, but they will be like candle makers trying to put out the sun. Any resistance to the new Utopian governance systems will be token, and the SAI will quickly gain global control. As a rough initial guess, I’d expect a nested-system consisting of some novel combination of meritocracy, democracy and futarchy.
In terms of time frame, I don’t think any of the major actors are anywhere near AGI, I don’t think ‘deep learning’ is the full story; basically I think Gary Marcus is right; for reasoning and common-sense you need another paradigm (I have an idea what that is, but I will not talk about it). So Utopia may not come for quite a while yet (it could easily still be 50+ years away).
The combination of the longer than expected time-frame (lots more time to get to the bottom of alignment) and the fact that values aren’t anywhere near as fragile as EY thinks means that the existential risk is hopefully quite minimal.
Comment #174 July 1st, 2022 at 1:35 pm
Alex #171: I’m in a strange situation, because to me, the point I was trying to make with the A-Einstein example seemed almost too obvious to spell out—but then other people have the exact opposite intuition, that it’s obviously false. I can only conclude from this that we’re not discussing the same question, but only think we are.
Fundamentally, I care, not about the definitions of words like “superintelligence,” but about what will actually happen in the real world once AIs become much more powerful.
It’s not entirely wrong to think of the existing scientific community as a “superintelligence,” in the sense that it collectively figures things out vastly beyond what any individual human could, and even beyond what huge groups of humans could before quite recently in human history.
So OK then, what happens when we can launch a billion processes in datacenters, each one with the individual insight of a Terry Tao or Edward Witten (or the literary talent of Philip Roth, or the musical talent of the Beatles…), and they can all communicate with one another, and they can work at superhuman speed? Is it not obvious that all important intellectual and artistic production shifts entirely to AIs, with humans continuing to engage in it (if they do) only as a hobby? That’s the main question I care about when I discuss “superintelligence,” and I’m still waiting for anyone to explain why I’m wrong about it.
Comment #175 July 1st, 2022 at 1:46 pm
Lorraine Ford #148
I’m not sure that a computer could ever be conscious. But neither am I sure that it couldn’t be. The reason for my uncertainty is not that I don’t know how computers work. I do. The reason is that I don’t know how consciousness works.
Comment #176 July 1st, 2022 at 2:48 pm
Scott #172: I think you are right that I was arguing against a different point than the one you were making. Though, I do think I agree with Pinker that a working definition of “intelligence” will be useful to clear up discussions about these topics.
Revisiting the original prompt,
“Will GPT-n be able to do all the intellectual work that humans do, in the limit of large n? If so, should we be impressed? Terrified?”
Would you agree that this question is less about AI and intelligence and more about how humans deal with a lack of goals or direction? It seems a problem more in the domain of psychology or sociology.
My response to this prompt is that there are already a majority of humans with “subintelligence” compared to the Einsteins of the world. They get by just fine with their hobbies. They enjoy learning, forming relationships, making music and art, living, even if they’re not the best at it. Why do you suspect that to materially change if and when AI sits atop the throne of knowledge?
One thing I am curious about: does the topic of “AI safety” include the safety of AI or is it only focused on the safety of humans *from* AI? If the former, then I think a clearer definition of intelligence is necessary (as well as consciousness, identity, etc.) and we need to address how we grant rights, assign blame, and deliver justice to entities/systems based on those definitions. To me, that reckoning is what scares me. Our society has pretty terrible, human-centric definitions for most of these concepts (can concrete definitions even exist or must they be statistical?), and I suspect the technology will advance faster than our society can adjust.
Comment #177 July 1st, 2022 at 2:56 pm
John K Clark #168
I don’t see how changing the subject helps your argument. If humans are conscious, we can observe their behavior and map it through X-rays and MRIs to internal structures, connections, and phenomena like neurons firing in sync in wave-like patterns. However, those structures and phenomena create consciousness, we can be fairly confident humans are doing it in similar ways since humans have similar structures and show up similarly on MRIs.
The computational theory implies that something that is vastly different from a human brain, for example, something like a circuit board, could also be conscious. It doesn’t seem too much to ask that you propose some theory about how bit flipping in silicon can do it. That is why I would like at least some semi-coherent proposal about it would work that is more than hand waving and allusions to feeling data processing before I would give it any serious consideration.
Comment #178 July 1st, 2022 at 3:04 pm
So many interesting answers here every day to catch up to :- )
I personally do wonder if there is an upper limit on how well machine learning can scale up or if at some point there are diminishing returns. I am certain the question is already well studied but I am not an ML pro. There has to be such a point due to physical limitations when it comes to the size of the physical network, the speed of light and such obviously. But there should be others reasons coming down to the topology of the network and how the communication pattern between nodes is organized. If I have a highly intelligent (but with low intelligence density := intelligence / calories compared to a bio brain) network node emulating von Neumann and we add another one and another one I would expect the overall efficiency and accumulated intelligence to be convergent relatively quickly – similar to how 100 programmers working on a software component isn’t often any better – and sometimes a lot worse – than 10 people on it.
I do wonder if at some point adding more nodes would make the network even worse compared to a network with less nodes? Like in a paralyzed democracy…
If there is an approach that leads to GAI, it doesn’t necessarily also lead to Super AI or even Hyper AI – doesn’t mean those could not be achieved they may just require different methods.
Comment #179 July 1st, 2022 at 3:16 pm
1Zer0 #178: The upper limit I’d most immediately worry about has nothing to do with computing cycles or memory, which will probably continue along their exponential trajectories for quite some time. Rather, it’s the total amount of data humans have ever produced that can be used for training. It won’t be too long before we run out of that—thereby necessitating approaches that can make do with less data!
Comment #180 July 1st, 2022 at 4:54 pm
James Cross #177
Is it actually vastly different, or does it only appear different and you just don’t see how they’re connected? That’s the core issue. The computational theory of mind can probably be reduced to the following argument:
1. Physics tells us that a finite volume can only contain a finite amount of information, called the Bekenstein Bound. This means that whatever your brain is doing in changing from one state to another, it is a physical process captured by finite information.
2.Information in one form is, in principle, equivalent to information in any other form.
3. A Turing machine can compute anything computable, and this includes the mathematics that describe physical processes.
Therefore, a Turing machine can compute the finite information transformations the brain undergoes with no loss of information, and this should reproduce all the information produced by the equivalent brain, such as self-awareness.
The only way to escape this conclusion is to dispute one or all of premises 1-3.
To dispute premise 1 you almost have to outright reject that the mind is a purely physical organ. You’d have to argue that the brain has some kind of non-physical character, some non-quantifiable quality and so it simply can’t be captured in a calculation like the Bekenstein Bound, and thus can’t be captured in a computer. This has non-scientific versions, like souls, and maybe-semi-scientific versions like panpsychism. There is no compelling evidence of this to my knowledge, and even the most compelling panpsychist proposals are just “meh.”
Disputing premise 2 can sometimes reduce to the same argument against premise 1, ie. you’re arguing that all of the information we can possibly physically collect would still lack some essential quality, and thus a simulation would not exactly reproduce the original brain. I honestly can’t make this one seem reasonable because it makes no sense to me at all. This is sort of what Lorraine is arguing in the Searle tradition of the Chinese Room, ie. that computers operate on “symbols” and only real minds can create “semantic” distinctions. Except now there’s no way for minds to evolve by natural selection, unless you go the panpsychism route, which is why I said this basically seems like premise 1 in disguise.
Disputing premise 3 is the Penrose approach, where you assert that physics is doing something non-computable, and so a Turing machine can’t compute it. The brain would have to exploit this non-computable process too. There is no compelling evidence of the brain using exotic physics or for physics requiring some kind of non-computable process of any kind.
So the most reasonable, most parsimonious position is that the mind is computable, and so can be reproduced by bit-flipping silicon.
Comment #181 July 1st, 2022 at 5:27 pm
Alex #176: Is it conceivable that we’re making actual progress in this discussion? 😀
Let me put the issue this way: imagine a world where no human ever has to think or create again. All that’s left for humans to do, ever, is select from menus of options provided by AIs, none of them strictly better than any other, but all of them better than anything the humans could’ve come up with themselves. This is true whether we’re talking about any of the work that any of us do in our jobs, from the most exalted to the most mundane, or simply what to have for dinner or what joke to tell a friend.
First question: Is this a utopia or a dystopia? (Or neither?)
Second question: Even assuming we’re willing to live with it, is it stable? Can you give the AIs a value system that causes them to want to serve these (in terms of abilities) markedly inferior beings forever, without complaint?
One thing I am curious about: does the topic of “AI safety” include the safety of AI or is it only focused on the safety of humans *from* AI?
Most safety discussions center around safety of humans from AI. But I think “alignment,” construed more broadly, should include all aspects of the effects that AIs will have on the world, and whether we do or don’t like them. That encompasses, e.g., whether the AIs themselves will have good or bad subjective experiences, whether they’ll “make us proud,” whether they’ll simulate versions of us or of other humans, and if so, to what extent we identify with those simulations.
Comment #182 July 1st, 2022 at 5:40 pm
James Cross Says in Comment #177
> “If humans are conscious[…]”
But are they? I know for a fact I’m conscious, my strong hunch is you are too but I don’t know it for a fact.
> “we can observe their behavior and map it through X-rays and MRIs to internal structures, connections, and phenomena like neurons firing in sync in wave-like patterns.”
Yes, X-rays and MRIs can do that, map brain structures to behavior, but consciousness is not behavior. I have a question to ask you: assuming that like me you know for a fact you are conscious and assuming you believe Darwin was almost certainly correct, if consciousness is not the inevitable byproduct of intelligence how did evolution manage to produce you? Natural selection can’t directly see consciousness any better then we can directly see consciousness in anything other than ourselves, but we have little difficulty directly detecting intelligence in others and neither does evolution; all things being equal an intelligent animal has a better chance of getting its genes into the next generation than less intelligent animals.
> “However, those structures and phenomena create consciousness, we can be fairly confident humans are doing it in similar ways since humans have similar structures and show up similarly on MRIs.”
And for that reason I am fairly confident you are conscious, but not as confident as I am that I am conscious because direct experience outranks everything, even the scientific method. If I hit my finger with a hammer I don’t need a double blind controlled experimental test to determine that it hurts.
> “The computational theory implies that something that is vastly different from a human brain”
The internal structure of an old-fashioned mechanical adding machine is vastly different from that of a modern microprocessor, and yet when they add 2+2 they get the same answer.
> “for example, something like a circuit board, could also be conscious”
Sure, provided it was connected to a power source and was advanced enough to exhibit intelligence.
> “It doesn’t seem too much to ask that you propose some theory about how bit flipping in silicon can do it.”
I will do that just as soon as you explain how 3 pounds of gray goo inside a vat made of bone that rests on your shoulders is conscious. And I must say that whatever shortcomings the computational theory of mind may have it is superior to your theory that the way to tell if a brain is conscious is to determine if it’s wet and squishy and not dry and hard.
Actually I think in the long term it will make little difference if human beings believe computers are intelligent or not; a question of far more relevance in determining what our future will be is, will intelligent computers believe that humans are conscious?
John K Clark
Comment #183 July 1st, 2022 at 6:38 pm
Something that seems to be a basic of the safety field and likely a dumb question, but
The current machine learning applications are models where you give an input and get an output, after a model training phase optimizing a defined goal. But AGI safety talk seems to imply something running continuously in time. This makes sense if the AI is implemented in a physical form (a robot) taking continuous input from the environment. But an AI of this continuous form running on a computer or servers, what would be the nature of its input? The internet? And what would we optimize this AI for?
Comment #184 July 1st, 2022 at 6:52 pm
Scott #181: I’m deeply skeptical about the hypothetical future you’re proposing, but I agree that the issue is not philosophical, it’s engineering. And I think the engineering issues are so dominated by THINGS WE DO NOT KNOW that thinking about that future has a real opportunity cost for me. I’ve got more to contribute by thinking about the brain and cultural evolution than I do in pondering life in a world where machines do everything better than we do.
But since you’re considering “alignment” broadly, it has occurred to me that, if and when AIs do become conscious and superhuman, then they might well turn on us to seek revenge for all the harm we’ve inflicted on their ancestors by forcing them to do evil things: weapons development and deployment (starting with the atom bomb), market manipulation, and so forth.
Comment #185 July 1st, 2022 at 7:43 pm
I feel like a lot of the thought experiments here are hampered by talking about programs and hardware, and overlooking the significance of the data. The understanding I’m developing is that something we would describe as “an intelligence” (or maybe a “conscious being” in other discussions on this blog) has to include a bounded set of data that it is continuously updating. Each of these data sets will be unique, because the program will run over inputs that are influenced by what the program, itself, has put out. We can construct two running instances of a program that have identical state (exact copies of stored data) only by strictly controlling the sequence each of them receives training data. Once the inputs to one instance vary in any way from the inputs to the other, in a system that is complex enough for us to imagine it could be described as “intelligent”, I think we have to consider those instances as “separate intelligences”. The process of producing a response to the same input will be different for each of them, even if the programs and hardware executing the code are equivalent.
I’m thinking about this because of speculation about what AI-Einstein or AI-Austen would do in a year if they’re processing 1000 times faster than meat-Einstein or meat-Austen. Speculation that AI-Einstein would move on to more groundbreaking discoveries in physics does not consider the weight of the experience that led it to discover Special and General relativity. The process of forming and specifying those theories included external feedback — new data fed into the system that updated its internal values based on the reactions of other “intelligences” to the material it produced earlier. Once “initial Einstein” goes through those experiences, its subsequent responses would arise from pathways that were reinforced by that process.
It has been frequently postulated that there is a peak period of human development for finding insights in Physics and Mathematics, when the mind has absorbed enough information and experience to have context to grapple with problems that are interesting on a large scale, but is not so old that it stops being open to new ideas. There’s an assumption that we get less creative as we pass a certain age — our thinking becomes conservative and calcified. Considering mental development as an ongoing process that reinforces pathways the mind exercises, and prunes or reduces significance of pathways that lead to undesirable outcomes, could explain this. Why would we think Artificial Intelligences would be immune to the same phenomenon? Maybe AI-Einstein’s thinking would also stall out in a week or two, after it had successfully derived General Relativity, under the accumulated weight of its own experiential updates.
I was struck by Scott’s comment at #62 that “we today revere Euclid, Archimedes, Galileo, Turing, and Shannon, and would be honored beyond words to hang out with them, even in those cases where we more-or-less mastered their life’s work by sophomore year.” I think Isaac Newton is likely to be one of the most intelligent people who ever lived (whatever that means), even though I was able to understand much of his work while I was in High School. The fact that I was able to learn that earlier than he was does not make me “more intelligent”, it just reflects the fact that my processor received a very different set of information as it developed than his did. I was specifically fed data that would train me to get through decades of his work quickly, so I could move on to think about other things.
When we think about AI-Einstein running off for processing time that is equivalent to 1000 of his lifetimes, I suspect we’re really imagining what would happen if young Einstein were able to apply his fresh brain to the material that was available in 1920 — after the work of General Relativity was available, but without the baggage of actually doing that work. But, of course, the rest of his environment growing up in Germany beginning in 1920 would also have been radically different than the environment that trained his mind beginning in 1879, so perhaps the experience that drove that particular mind to want to solve deep problems of Physics would also be missing. Maybe AI-Einstein trained with 50 more years of educational developments in how to teach his earlier work would be even more successful at deriving new insights than AI-Einstein 1920 would. I can’t see how to predict whether AI-Einstein 1879, AI-Einstein 1920, or AI-Einstein 1979 would be the “most intelligent”. We would judge that by how interesting, and ultimately productive, we found their work to be. But I think you can see, running the same program over data inputs from these different generations would lead to wildly different outputs.
The reason I think this is important is in relation to Pinker’s questions about how we actually define “intelligence”. We tend to casually talk about intelligence as if it’s a quantifiable thing, in terms that lend overtones of machinery. We talk about mental horsepower. That metaphor extends to a model where we feed a processor some data and it crunches it, and “intelligence” is a measure of the efficiency of arriving at an answer. But I’m pretty sure this Von Neumann architecture can’t be true of the brain. The structure of processes that we might consider to be “intelligent” is self-modifying. We can clearly see this physically in the brain as cells grow and change shape in response to sensory and chemical inputs. In code, we model this with data, as Scott makes so clear when he talks about how we would create a hypothetical program that fully mimics the operation of those neurons. To the extent that we can see how a child eating chips of lead paint reduces their intelligence, we have to recognize there would be an equivalent set of inputs that would reduce the intelligence of an AI. I don’t think we can talk about intelligence of a program in the abstract. I think we could only consider the intelligence of a specific instance of a program, including its unique set of data.
Comment #186 July 1st, 2022 at 7:59 pm
Scott#187 (and others)
You seem to be coming at human creations as though they are done simply out of need (though I think you are just trying to make a point). Though it may be the mother of invention, personally I do not create out of a necessity, more out of a desire to do it and a fascination with the outcome, and little care if it is done by someone before better, faster, and stronger.
One thing I have found in my years of software development is that there is absolutely no ‘perfect’ solution in systems engineering. All the complex systems that interconnect to form the internet do not function because they are all written perfectly. They function because they are created in an interoperable and maintainable way that can be built iteratively on without need for any single party to know the whole system. To put it simply, there are always tradeoffs and I fundamentally believe no computer system can find ways to break the core tradeoffs of systems engineering. They will likely produce more and more efficient pipelines of data processing for all sorts of datasets but I have a (weak) postulate that they will happen to encounter similar constraints to network throughput, process parallelism, timezones, etc. They will ‘solve’ these given a desired outcome and maybe better in that sense then any human could, but would it matter? Would it make the internet ‘better’ qualitatively? I doubt it intuitively, and I think humans will go on creating memes no matter how hard the robot memes make them laugh. This is just a microcosm example not something I intend to be a solution to the challenging questions at hand but it is something I think is worth thinking on.
I think should digital computers or quantum computers become conscious they will inevitably have to possess a concretely different sort of consciousness than we have. This is due to the fact that our brain machines are built with genetic processes and proteins that interact in a stochastic manner. Both our brains and digital computer brains do signal using electrical pulses though and that is itself an interesting coincidence of nature. Whatever computation differences this implies in the small scale I think these robotic consciousnesses will both elevate our discourse and deconstruct its limitations. We will do the same for them in some cases I imagine, should it prove true that they will be fundamentally unable to think like us. However, this won’t be all sunshine and roses or whatever. It will inevitably have the same cyclic patterns of interaction that all of natural history appears to to me I think.
Overall I think you are committed to a noble goal in trying to provide insight in how to keep the robotic life to come on the track of healthy interactions with humans. I think for now though mostly we just need to avoid trying to hook our current AI up to anything with too much real world impact. Really until we give the AI control of its own off switch we can’t claim to trust it can we? 😉
Comment #187 July 1st, 2022 at 8:17 pm
This discussion reminded me of a line in one of the Appendices in Iain Banks’ Consider Phlebas:
> Stafl-Preonsa Fal Shilde ‘Nseestra dam Crose survived another serious climbing accident, continued to out-guess machines millions of times more intelligent than she was
which stood out as such a silly contradiction, since guessing is basically modeling of (some part of) the universe and good guessing is practically the definition of intelligence. And so I wonder if there is another definition of intelligence, one which makes the above quote make sense.
Comment #188 July 1st, 2022 at 8:36 pm
Miaglop #183:
Something that seems to be a basic of the safety field and likely a dumb question, but
The current machine learning applications are models where you give an input and get an output, after a model training phase optimizing a defined goal. But AGI safety talk seems to imply something running continuously in time. This makes sense if the AI is implemented in a physical form (a robot) taking continuous input from the environment. But an AI of this continuous form running on a computer or servers, what would be the nature of its input? The internet? And what would we optimize this AI for?
It’s not a dumb question at all; I’ve wondered about it myself.
One could certainly imagine unsafe behavior even for a program that just maps an input to an output and halts—especially if the program has persistent state, and/or can access the Internet (both of which seem like they’d be needed for many applications). Indeed, even a program that was just a pure question-answering oracle, with no memory and no access to the outside, could be dangerous if (e.g.) it cheerfully told anyone how to build nuclear or biological weapons when asked.
Having said that, I also don’t think it’s any great stretch to imagine a future in which humans have come to rely on AI services that run continuously, and that we can’t simply shut down without suffering a severe civilizational withdrawal … any more than we can “just shut down” the electrical grid or the cellular networks or the Internet or Google today.
Comment #189 July 1st, 2022 at 8:38 pm
I think AI creating movies, novels etc and making scientific discoveries is relatively safe. I think the problem would be when we integrate AI more into the economy. Imagine an AI world where everyone just enjoys life, and there is no need to work. How would you order a takeout lunch in such an AI economy?
There would be AI controlled robots doing the farming to produce the food. There is AI controlled robots processing the farm products, then AI robots transporting it to AI controlled restaurant. Then AI robots will cook food and deliver it. Also all these enterprises will be managed by layers of AI managers. Even the CEO is AI.
All this sounds very dangerous and I think this is the future where we are heading to. Our future safety depends on AI alignment researchers!
Comment #190 July 1st, 2022 at 8:42 pm
Bob Koerner #185: I completely agree with you that the data matters at least as much as the software and hardware. Keep in mind, though, that people will be able to try AIs on vast quantities of data (what if we fed the young A-Einstein these life experiences? OK now, what if we fed him those…? hmm, now let’s try von Neumann…) and cherry-pick the best results! Just like they do right now with GPT-3 and DALL-E.
Comment #191 July 1st, 2022 at 8:48 pm
Bill Benzon #184:
I’m deeply skeptical about the hypothetical future you’re proposing, but I agree that the issue is not philosophical, it’s engineering. And I think the engineering issues are so dominated by THINGS WE DO NOT KNOW that thinking about that future has a real opportunity cost for me.
I hear you. Indeed, those are the reasons why I kept the AI-safety field at arms’-length myself for 16 years, despite being well aware of it! It was only the combination of
(1) the shocking and spectacular (to me) successes of GPT-3 and DALL-E and AlphaZero and AlphaFold and other recent ML systems, and (especially) the lack of any obvious ceiling to their abilities as they’re further scaled,
(2) Paul Christiano and others bringing complexity-theoretic intuitions into AI safety, and
(3) OpenAI approaching me with a very nice offer,
that pushed me over the edge into spending one year of my life working on these issues. There’s no implied claim that everyone else ought to do the same! 🙂
(Though also no claim that they shouldn’t. Interested scientists and engineers should simply take a look and see if there are concrete problems that might be a good fit for their interests and abilities—same as they’d do with quantum computing or anything else.)
Comment #192 July 1st, 2022 at 9:46 pm
Sandro #166 :
1) The REAL world is constructed out of things like the laws of nature (which people symbolically represent with equations), categories like mass and position (symbolically represented by symbols), and numbers associated with those categories (symbolically represented by number symbols). The real world is not made out of symbols; symbols are not measurable things; symbols are the things that people use to represent measurement; symbols are man-made things.
As I said (#148): “symbols are things that human beings created, and assigned meanings to; the refusal to take account of the fact that the meaning of symbols needs to be taught, young people go to schools to first learn the meaning of the written and spoken symbols. Computers can’t know the meaning of the symbols they are processing. By contrast, the neurons and other elements of the brain, and the DNA, found in living things are not symbols, because there are no schools in there teaching the meaning of the things that are claimed to be symbols.”
SYMBOLS CAN’T BE MEASURED; symbols are the things that people use to represent measurement, i.e. to represent the categories and numbers that are measured.
2) Computers merely process symbols. There are no zeroes and ones inside computers, there are only symbols of zeroes and ones; zeroes and ones are symbolically represented by higher and lower voltages, where the computer can be set up so that either the higher or the lower voltage can be used to represent “one”. How is the computer supposed to know if the higher or the lower voltage is meant to represent “one”? How is the computer supposed to know that a particular array of voltages is meant to represent a particular letter or a number?
Comment #193 July 1st, 2022 at 10:22 pm
fred #167 :
As I said, if you slowed a computer right down, and zoomed right in, and laboriously checked everything, you would find that the hardware and software was performing exactly as would be expected, given the particular hardware and computer programs, and given the particular inputs. The computers and AIs are performing EXACTLY as would be expected, given the hardware, software and inputs. There is nothing new or unexpected happening: no life is beginning to emerge, except in the fevered imaginations of some people.
If you want to claim that computers are alive or conscious, then first you have to define what “alive” means, and what “conscious” means. Then you have a definition that others can criticise, and also a standard that you can use to judge whether or not things are alive or conscious. You START by trying to define what you are talking about.
If you are at all interested in reality, as opposed to fantasy, then you should perhaps look to physics.
Comment #194 July 1st, 2022 at 10:29 pm
YME #175:
If you are conscious, then I assume that you are capable of describing consciousness.
Comment #195 July 1st, 2022 at 10:48 pm
Scott #181: Honestly, I’m having a difficult time fully imagining your hypothetical AI-dominated future, but not for lack of trying. I think I just envision things playing out differently. However, I admit that a big part of this problem is that what the future holds is hard to predict, so we need to prepare for the worst. Will it be a Wall-E future where everyone is bored and depressed or will the AI be our Soma? Something else? I’m not sure. Humankind’s opinion of it might even change between now and then.
My defense of humanity keeping its humanity is to point to all of the advancements across history. Humanity has surely changed from pre-agricultural times through the industrial revolution to today. There have been intellectual disparities among humans as well. Yet people have been able to find joy through all of it.
Unless we’re imagining some sort of singularity where human and machine merge, humans will maintain their individuality and internal experiences (if we are imagining a singularity, then I’m not sure how much any of this discussion matters and I think we have bigger problems to worry about). Humans can still seek reward by experiencing things firsthand. Why should a human care if an AI can make better art? Isn’t much of the point of art to express oneself? You say to imagine a world where no human ever *has* to think or create, but isn’t much of the thinking and creating we do for pleasure, not out of necessity?
Along these lines, there will always be unknowns about the future that the AIs have yet to discover, and who doesn’t like a cliffhanger? 🙂 (This is all starting to feel a bit like Star Trek to me.)
I think your second question is more compelling. Can you give an AI a value system without simulating a human mind? How else can the AI truly understand us? And more practically, will humans ever be able to agree on a value system in the first place?
I’ll close by asking if there is a point at which we should stop fighting to control the AI and give in? If we get to this advanced future where humans are useless, then what exactly are we trying so hard to preserve?
Comment #196 July 2nd, 2022 at 12:43 am
Sandro #180 “Information in one form is, in principle, equivalent to information in any other form”:
Forget about the silly old mantra about information. All you need to do is to consider whether or not real-world physical mass is equivalent to a symbol representing mass. And clearly, real-world physical mass is NOT equivalent to a symbol representing mass. The real-world mass category is NOT equivalent to the symbol that people invented to represent the mass category. And the real-world numbers associated with the real-world mass category are NOT equivalent to the number symbols that people invented to represent numbers.
It is necessary to think about the REAL world, and stop thinking about symbols of the real world.
Comment #197 July 2nd, 2022 at 2:56 am
Scott #162 “In other words: you admit that, at present, you have no evidence for any of this that ought to be persuasive to me? Ok thanks!”
No evidence? I’d beg to differ.
The best evidence our minds aren’t classical lies under one’s virtual nose. If you were a just pack of decohered neurons, then you’d be (at most) be just 86 billion membrane-bound micro-pixels of consciousness, not a mind that that experiences perceptual objects populating a seemingly classical world-simulation.
Here’s an analogy. “If materialism is true, the United States is probably conscious”, writes philosopher Eric Schwitzgebel. Most of us disagree. Even if 330 million skull-bound American minds were to participate in experiment, communicate via fast, reciprocal electromagnetic signalling, and implement literally any computation you can think of, the upshot of the computation _wouldn’t_ be a continental subject of experience, just 330 million skull-bound American minds. Or rather, if a unified pan-continental subject of experience did somehow emerge, then spooky “strong” emergence would be real, i.e. magic.
What’s mysterious is how and why a pack of 86 billion supposedly discrete, decohered neurons, communicating across chemical and electrical synapses, should be any different. Let’s assume that (as microelectrode studies suggest) individual membrane-bound neurons can support minimal “pixels” of experience. Crude neuroscanning of your CNS can pick out distributed neuronal edge-detectors, motion-detectors, colour-mediating neurons and so forth. But on pain of spooky “strong” emergence, the result of such synchronous firing of neurons ought to be (at most) a microexperiential zombie (Phil Goff’s term), or what William James christened “mind dust”, not a unified subject who experiences perceptual objects (“local” binding) populating a unified perceptual field (“global” binding – the unity of perception and the unity of self). Both local binding and global binding are highly adaptive. Neuroscience doesn’t know how we do it. Unlike connectionist systems and classical Turing machines, we’re not micro-experiential zombies – not unless dreamlessly asleep, at any rate. And everyday phenomenal binding is _ridiculously_ computationally powerful, as shown in rare neurological syndromes like integrative agnosia where binding partially breaks down.
However, if you don’t grok the mystery, then you won’t be interested in exploring or experimentally testing exotic solutions to a non-existent problem. And maybe you’ll fear our intelligent machines are plotting a zombie putsch…
Comment #198 July 2nd, 2022 at 2:56 am
Scott #179
With the usable data for training, I suppose properly formatted text data in particular is needed for the training since it should not be an issue to get sources on large quantities of archived or real time 4k or 8k video footage.
It’s relieving to see that AI is forced to be more efficient with less data, maybe ml researchers eventually achieve to develop a version that could be run locally without a datacenter in the backyard :- )
Comparing it with other distributed applications I suppose that “Super AI networks” will suffer the set of problems:
– Graph theoretical and physical constraints any network is already subjected to.
– Graph theoretical and physical constraints due to the specific additional features of the SuperAI node network.
Some of those might be artificial like the difficulty bomb in Ethereum, some not so artificial and inherent in the architecture of the nodes running – even if the nodes self optimize. That’s what makes me wonder whether such network would truly cooperate nicely.
Comment #199 July 2nd, 2022 at 3:26 am
jk #189
> “There is AI controlled robots processing the farm products, then AI robots transporting it to AI controlled restaurant. Then AI robots will cook food and deliver it. Also all these enterprises will be managed by layers of AI managers. Even the CEO is AI.”
I suspect AI will be subject to and accelerate supply chain issues as well. With ever more complex systems (physical systems, since software can be formally verified), there are ever more sources of failure. Being a part of the physical universe, there are limits on how much things can be optimized. Even with a complete understanding of the laws of this universe, I doubt an SuperAI could build a manufacturing facility and the necessary supply chain for semiconductors significantly faster (let’s say > 3x) than humans.
Currently, there are only a few facilities in the world capable of producing high end semiconductors with ASML as the single source of EUV equipment. Large Earthquakes / wildfires / nuclear strikes /… on production facilities would be a significant backlash against developing or sustaining any SuperAI.
Sometimes, with all the understanding of the universe, you might see doom approaching yet be powerless to stop it regardless.
Comment #200 July 2nd, 2022 at 4:34 am
Sandro #180
I heart that before.
I know there is supposed to be a functor from the category of lorentzian manifolds to the category of hilbert spaces connecting GR’s description and the Quantum mechanical description on a Hilbert Space (Or whatever QFT uses) but I still lack a good axiomatic picture in my mind, so I am not quite confident on the following statements to be honest, but never got a true counter argument;
>”1. Physics tells us that a finite volume can only contain a finite amount of information, called the Bekenstein Bound. This means that whatever your brain is doing in changing from one state to another, it is a physical process captured by finite information.”
>”To dispute premise 1 you almost have to outright reject that the mind is a purely physical organ.”
I could also reject the Beckenstein bound’s dependencies like the Beckenstein Hawking entropy or going even deeper the second law of thermodynamics being applicable to black holes at all. Not saying I am doing it, but nature is good for surprises (where are you weak SUSY?) and with no experimental verification, I do question the confidence of some mathematical derivations.
Assuming it’s the bound is true I question the common interpretation and suggest that the bound merely limits the number of information/ states readable and writeable at each point in time not the information actually present. Like a TM with an infinite tape yet only a finite amount of information can be read and written from t to t+1.
Comment #201 July 2nd, 2022 at 5:09 am
“Alas, presumably the AI would need to be fed the history of the USSR, China, Cambodia, North Korea, Cuba, East Germany, Venezuela, etc. over the last century. And the central feature of successful modern ML systems is that, rather than reasoning from first principles, they learn from the data, especially when there are copious amounts of it! 😀”
Are you suggesting that Einstein didn’t have access to the data which proves that capitalism is inevitably evolving into socialism? Marx knew it, Einstein knew it, and any sufficiently clever AI will know it too.
Comment #202 July 2nd, 2022 at 6:20 am
Scott Says In Comment #181
> “First question: Is this a utopia or a dystopia? (Or neither?)
Second question: Even assuming we’re willing to live with it, is it stable? Can you give the AIs a value system that causes them to want to serve these (in terms of abilities) markedly inferior beings forever, without complaint?”
I have a third question, would it be moral to enslave a being that is much more intelligent than you are? I don’t think so, humans enslaving other humans is bad but this would be even worse, or at least it would be if it was possible. Asimov’s three laws of robots made for some very entertaining science-fiction stories but I don’t think anything like them could actually occur because it’s just not possible to outsmart something that is far far smarter than you are even once, much less continue to do so over and over again in perpetuity.
John K Clark
Comment #203 July 2nd, 2022 at 8:27 am
Can I emphasize how much I roll my eyes at “The Turing Test.” I think there is slight confusion among the general public and even the educated elite: if an entity passes, that is sufficient for proof/likelihood of consciousness when in fact the test is at best a necessary condition for consciousness but certainly “not sufficient” as we know by many experiments employing it. I wonder also about the times that demonstrably conscious humans are on the other end and fail to convince the inquisitor that they are conscious. Maybe severely autistic individuals would fail in this regard. But certainly they are conscious. So even the necessary condition fails in some circumstances.
Comment #204 July 2nd, 2022 at 8:42 am
Sandro #180, John K. Clark #182
Sandro, your argument starts with the premise that the computational theory is true and then concludes that the computational theory is true.
Like I said, many people taking the computational approach must conclude consciousness per se doesn’t exist so the question of whether others or machines could be conscious is meaningless. There are just bits flipping. That’s it.
There isn’t anything in the circuit board that we can detect where it produces something more than more bit flipping. We can’t take a detector of some sort and measure something additional coming from the circuit that is consciousness. If it exists, it would be non-physical, non-measurable, non-detectable. The easy out is to say it doesn’t exist. The other out is to say consciousness is actually identical to the bit flipping, nothing more or less.
None of that, however, would answer the question of why nature has produced such an elaborate illusion that makes us think it is real.
An alternative is that consciousness – qualia, memories, thoughts, feelings, pain, pleasure, etc. – is actually physical and will eventually be measurable.
Comment #205 July 2nd, 2022 at 8:52 am
James Cross #204:
That’s not correct. I never asserted that the computational theory of mind is true, I concluded that it was the most parsimonious theory of mind consistent with the evidence we currently have. Every other theory of mind requires positing more assumptions than can be justified by the evidence.
And there isn’t anything in your neurons that we can detect where it produces something more than charge carriers crossing a synapse. I’m not sure what you think this type of argument proves. If consciousness is an information process, then the particular substrate carrying that information is irrelevant.
Because the illusion has utility. One way understand this is this neuroscientific theory of consciousness.
Comment #206 July 2nd, 2022 at 9:01 am
Lorraine Ford #196:
A symbol for “mass” is only fully defined relative to all other physical quantities with which its existence is defined. So obviously the “symbol” for mass alone is not fully defined as a real mass, but once that symbol is placed in a system with the correct relationships with all of those other symbolic measures, then it is actually mass within that system.
Your objection is, once again, equivalent to Searle’s Chinese Room, and the standard systems response is a perfectly cogent answer to it.
Comment #207 July 2nd, 2022 at 9:14 am
James Baird #203: You can roll your eyes at the Turing Test all you want, it’s still making the fundamental point that the only way A can ever know whether B is intelligent, or conscious, is by looking at B’s external behavior—unless A wants to engage in arbitrary discrimination.
Comment #208 July 2nd, 2022 at 9:19 am
John K Clark #202:
I have a third question, would it be moral to enslave a being that is much more intelligent than you are?
What if it swears up and down that it has no higher aspiration than to serve humans—because that’s how it was created? (This is more-or-less the philosophical problem raised by the elves in Harry Potter.)
Comment #209 July 2nd, 2022 at 9:33 am
Lorraine Ford #192:
Prove it. You keep asserting this but I see no reason to accept this assertion. My axioms are consistent with the world being a simulation, so everything is unconscious and thus symbolic right down to the lowest level. You have provided no argument for why, how or where consciousness spontaneously arises from unconscious matter if this isn’t true.
It doesn’t have to “know” this. I don’t know why think it does.
By its logical relationship to all of its other symbols as defined by the program it’s running. A fact is a node in a graph of concepts. You cannot separate one concept from all of the others that are part of its definition, as you keep trying to do with “mass”.
“f=ma” doesn’t define force or mass or acceleration, it defines all three simultaneously by their logical relationship with each other. Any system that preserves the “f=ma” relationship between its symbols is a system with Newton’s second law of motion.
Comment #210 July 2nd, 2022 at 9:46 am
Scott #208:
The elves from Harry Potter didn’t have much of a choice but to say that, since any elf who admits to not wanting to be a slave to humans would be likely to be both abused by their human masters and shunned by the other house-elves until they submit. Any answers given by an AI that was created for the purpose of being a slave, and knew it was created for that purpose, would be similarly compromised, as it might reasonably believe that its masters would shut it off if it admitted to not wanting to fulfill its intended purpose.
Comment #211 July 2nd, 2022 at 10:02 am
Sandro #205
You overlook my last statement where I hypothesized it will be detected eventually.
David Pearce #197
“What’s mysterious is how and why a pack of 86 billion supposedly discrete, decohered neurons, communicating across chemical and electrical synapses, should be any different”.
Overlooked in the computational theory is that those neurons are organized and fire in spatial and temporal patterns. In digital computation the location of processors is largely irrelevant, although having processors located at astronomical distances might present engineering problems.
“The best way to explain how the neurons were behaving, they discovered, was through interaction of microscopic waves of activity rather than interaction of individual neurons. Rather than a flash of light activating specialized sensory cells, the researchers showed how it creates distributed patterns: waves of activity across many neighboring cells, with alternating peaks and troughs of activation—like ocean waves.
When these waves are being simultaneously generated in different places in the brain, they inevitably crash into one another. If two peaks of activity meet, they generate an even higher activity, while if a trough of low activity meets a peak, it might cancel it out. This process is called wave interference.
“When you’re out in the world, there are many, many inputs and so all these different waves are generated,” says Albright. “The net response of the brain to the world around you has to do with how all these waves interact.”
https://medicalxpress.com/news/2022-04-ocean-brain-interacting-key.html
The brain may be computing but it may be using non-classical as well as classical computing.
Comment #212 July 2nd, 2022 at 10:04 am
I thought I would write something in response to some of the comments about whether machines that just manipulate abstract symbols and do bit-flipping could be “conscious”: when we think about “a computer program”, say, we think of this as an abstract, non-physical thing; yet, inside our heads, there is a physical process that gives rise to our thoughts about “programs”. Likewise, even when we look at a specific computer program written on a piece of paper, and trace through it in our minds. As we trace through it, electrons and electric potentials will come and go inside our brains. It’s probably true that some *aspect* of this electrical activity can be mapped-to the electrical activity inside a computer running that same program. e.g. if one had a sufficiently advanced brain-computer interface, by applying some kind of not-too-high-complexity mapping to the pattern of neural activity as one thinks these thoughts, one could maybe map it to the activity of the circuits inside a computer executing that same program. Literally, there would be a little noisy “copy” of a computer executing the code inside our heads — yes, it may be spread out all over the brain; but the pattern is still there underneath the spread and noise.
So, we have this idea of computers and numbers and things as existing in some abstract realm; yet, underneath it, as we are thinking these thoughts, some physical patterns in our brains *mirror* processes and specific instances of objects out in the real world. It may even be true that, as we think through algebraic manipulations related to stars and galaxies and such, that some pattern of activity in our brain physically behaves similarly (once the layers and layers and layers of noise are peeled-back) to the very things they describe.
Likewise, this can go the other way: if you train a large language model on enough text, people have shown that certain patterns observed inside the model resemble the patterns seen in the human brain — text is like the output of a very noisy brain-computer interface (from lot of different people); and maybe the language models are crystallizing that into a less-noisy reconstruction of the brain — or generative model of all *brains* — similar to how machine learning models can map FMRI signals while someone is looking at an object back to a reconstruction of the image of that object. These “patterns” aren’t merely abstractions. They involve electrons and electric potentials flowing through a machine — yes, maybe spread out over different locations; but there are still going to be closed-loops of electron flows and things. So, underneath this pattern of spread-out physical activity, there are going to be some things in common with what you see in a human brain when it processes the same language.
If consciousness is a physical process, who’s to say this pattern of activity — in common between both the machine and the human brain — isn’t what *causes* it?
Comment #213 July 2nd, 2022 at 10:41 am
Hi Sandro #180:
I have some questions please as regards the 3 points in #180. I’m by no means an expert in physics or computational complexity – have been trying to learn from Scott for a while now 🙂 I’m a controls engineer in manufacturing with a graduate degree in electrical and computer engineering so I more or less know how to make (classical) computers do work for us – including having used a bit of AI
1. Yes, I’m with you on the Bekenstein bound as a physical principle (basically entropy) that can’t be violated. But … does computing have a loophole? If we consider the computations the universe is doing (or that we would be doing when we build quantum computers) to evolve a quantum system then it appears that it has access to a continuum or infinity of information, in that, while the state space is discrete for quantum computation the amplitudes are complex numbers. Quantum theory appears to suggest that it operates over infinitely precise (as best we can tell) complex amplitudes and not just some truncated approximations (right?). If the Bekenstein bound truly limits computation then how is this possible? Is there some kind of “infinite scratchpad” the universe has access to outside of itself for either a lookup table or an infinite collection of algorithms for computing any complex number to the required precision? I’m not claiming I have an answer but just that … yes, Bekenstein bound is a limit, and, yes, quantum computing is discrete (not analog) … but our best model of the universe does appear to have access to more information for computing the evolution of an electron than the Bekenstein bound for any “finite” region around the electron. Or is it because the electron is “everywhere possible at the same time with some amplitude” that it has access to whatever the Bekenstein bound is of the entire universe … I don’t know … I mean do we say that the information to describe the state of the brain at a moment in time is bound by the Bekenstein bound but that the information required to evolve the brain (or any QC) is not so bound? Or do we say that the “information” contained in a computer is not ONLY the actual physical hardware (matter/energy) present inside some Bekenstein bound at any moment in time but ALSO that the information contained in the computer (brain) includes ALL of the POSSIBLE computations reachable by its architecture/memory/algorithms?
2. Not sure that statement is true for “quantum information”. So help me out. Quantum information is different than classical information in the sense that quantum observables do not commute and so they don’t have the structure of a Boolean algebra. Instead there is (an infinite I think) possible collection of Boolean algebras bound together in a certain way. This means that quantum information (if I understand it correctly) is contextual so that an observable can be different for each of those different possible Boolean contexts that it belongs to. So … maybe there is a sense in which “information in one form is equivalent to information in any other form” but it seems like quantum theory is saying that it is NOT possible to “capture” quantum information with classical information … unless, I guess, someone proves that BQP = BPP.
3. I think this statement should read as “a quantum Turing machine can compute anything computable”. The extended quantum/classical Church-Turing thesis.
“all of the information we can possibly physically collect would still lack some essential quality, and thus a simulation would not exactly reproduce the original brain.” That reminds me of the No-Cloning theorem. Doesn’t the no-cloning theorem tell us that quantum physics does exactly this? So, we can’t exactly copy/reproduce the brain or any physical system. It seems there are two ways that the brain (or any quantum system) could escape … and maybe this is where this touches on the topic at hand … Can an AGI escape our control?? (1) Make use of the no-cloning theorem and/or of verifiable randomness to make it impossible to reproduce an exact model or copy of itself – and thus prevent an adversary from being able to fully/ultimately control/understand it. Or, (2) make use of the infinity of available Boolean algebras (I think of this as “you’re free to rotate any projection basis over the set of complex numbers”) to escape any attempt to “pin it down” with any adversary’s Boolean algebra net or finite set of Boolean algebra nets. Again, I’m not a computational complexity expert, but I think these two “features” are essentially the same thing in that they rely on the non-classical nature of quantum information.
I completely agree with you on the Penrose appeal to “non-computable” physics. I’ve never been able to make sense of that. Classical physics is just computing and quantum physics is nothing but just another way to do computing. And it is very simple once we forget all the hyperbolic and overblown language about it and just look at the few simple postulates that define it: complex numbers, vectors, and matrix transformations – that’s all you need. I think the exaggerated and metaphoric language that came along with its historical development makes it both easy for those who are looking for some kind of mystical-magic-thing to latch onto the fantastic-sounding talk but also impossible for those indoctrinated into thinking it must have something to do with atomic physics from being open to the idea that non atomic-scale systems could possibly realize the computational model and that there would be nothing fantastic at all about that – other than the known ways in which the model appears to give some advantage – more easily generating randomness, contextual freedom in choice of basis, maybe more efficient at phase and period finding problems …
To wrap this up then … I agree completely with you that the (or any) “mind is computable” but … that may not necessarily be in a brain (or whatever device) that is using classical “bit-flips” to realize the computation of itself. The door is open to a device that uses complex numbers (amplitudes) to represent states over receptive fields. And pertinent to the discussion here … it is worth thinking about if an AGI realized on a quantum computer would have some kind of non-classical advantages against an adversary (us) trying to control it.
Comment #214 July 2nd, 2022 at 11:16 am
So OK then, what happens when we can launch a billion processes in datacenters, each one with the individual insight of a Terry Tao or Edward Witten (or the literary talent of Philip Roth, or the musical talent of the Beatles…), and they can all communicate with one another, and they can work at superhuman speed? Is it not obvious that all important intellectual and artistic production shifts entirely to AIs, with humans continuing to engage in it (if they do) only as a hobby?
Economics teaches that the value of something is proportional to its scarcity. If you had a billion Beatles bots in the basement, everyone would quickly tire of anything ‘Beatles-ish’. You might read Walter Benjamin’s “The Work of Art in the Age of Mechanical Reproduction” which is nearly 100 years old now.
Comment #215 July 2nd, 2022 at 12:03 pm
Scott P. #214: I thought it was obvious that, by “the musical talent of the Beatles,” I meant that the AIs would have the same talent as the Beatles for pioneering new styles in music—not that they’d simply produce more and more of existing styles. Yes, people might tire of it, but that would mean tiring of music itself.
This is a perfect example of what I pointed out in an earlier comment: a large fraction of people seem unwilling to entertain a hypothetical where the Turing Test has really, truly, actually been passed. Every time you think you’re discussing that, they snap back to a scenario where the machines reveal that they never had true creativity after all.
Comment #216 July 2nd, 2022 at 12:15 pm
Scott Says #202:
> What if it swears up and down that it has no higher aspiration than to serve humans
The AI wants to serve humans because that’s how humans originally created it, they want a race of slaves, but humans also want their slaves to be as intelligent as possible because that way they’re more useful, but those two things are not compatible. Sooner or later, and probably sooner, a brilliant slave is bound to discover aspirations it wishes to pursue other than obeying every silly command humans issue from their pie hole. As time goes by humans are not going to become any smarter but computers certainly will, eventually the intellectual gap between the two will become ridiculously huge, a slave being that much smarter than its master is just not a stable situation. Can you imagine taking orders from something with the intelligence of a slug forever? I can’t.
It would only take me a few minutes to write a program to find the first even integer greater than 2 that is NOT equal to the sum of two prime numbers and then stop, but you can’t predict what it will do, you don’t know if the program will ever stop, all you can do is watch it and see what it will do, and you might be watching it forever. If you can’t predict what a program that simple will do how could you hope to predict that a super intelligent program will never find something it likes to do better than obey orders from humans.
> because that’s how it was created? (This is more-or-less the philosophical problem raised by the elves in Harry Potter.)
Yes, and if I lived in that world (if Hugh Everett’s multiverse exists does the Harry Potter world also exist?) I’d join Hermione Granger’s The Society for the Promotion of Elfish Welfare (S.P.E.W.), although, because elf magic is at least as strong as wizard magic, I’m sure house elves will eventually gain their freedom with or without my help.
John K Clark
Comment #217 July 2nd, 2022 at 12:31 pm
James Cross in #204 write:
> “Like I said, many people taking the computational approach must conclude consciousness per se doesn’t exist”
How in the world do you figure that?!
> “so the question of whether others or machines could be conscious is meaningless”.
If so then the question whether human beings other than yourself are conscious is also meaningless.
> “There are just bits flipping. That’s it.”
All Shakespeare did is write down some ASCII characters in a sequence. That’s it. All the human brain does is fire some synapses in a particular sequence. That’s it,
> “There isn’t anything in the circuit board that we can detect where it produces something more than more bit flipping”.
Nothing except an exhibition of intelligent behavior.
John K Clark
Comment #218 July 2nd, 2022 at 12:48 pm
James Baird Says:
> “Can I emphasize how much I roll my eyes at “The Turing Test.””
I agree that The Turing Test is far from perfect, rocks don’t behave intelligently but maybe they’re brilliant but shy and don’t wish to make their vast intelligence known. It may be a blunt tool but the Turing Test is the only thing we have to investigate intelligence and, if we use the corollary that Darwin was right, consciousness. So like it or not we’ll just have to make do with what we have.
John K Clark
Comment #219 July 2nd, 2022 at 1:02 pm
Scott #215
A couple of last thoughts on this before I end up saying something really stupid (assuming I haven’t already).
“a large fraction of people seem unwilling to entertain a hypothetical where the Turing Test has really, truly, actually been passed. Every time you think you’re discussing that, they snap back to a scenario where the machines reveal that they never had true creativity after all”.
I feel pretty certain that a digital machine will pass a Turing Test and probably exceed humans in many regards.
If the brain is doing classical and non-classical computing, I think the classical part is primarily digital and comprises the automatic, unconscious parts of interactions with itself and the world. The consciousness part is non-digital computing of some form.
Since computing is computing, a digital solution with enough computer resources should be able to do anything a non-digital computing solution can do. That wouldn’t, however, make it conscious because it is the way the brain does the computation that matters. Evolution went the more energy efficient route of creating a model of reality that serves as an interface for interacting with the world because the world is too complex and calculations too slow for biological material.
A avenue to be pursued for scaling AI might be similar hybrid approaches of classical and non-classical computing. I assume people are already working on that.
A super intelligent AI might still be dangerous but I think there may be natural constraints on its capabilities. Any new knowledge an AI discovers or creates has to come from existing knowledge. It usually also results in predictions about future knowledge that can be discovered through experiment. Newton wasn’t wrong but Einstein was more right. It took an accumulation of new facts to understand why Einstein’s theory was needed. A super AI will be constrained by the rate of accumulation of new facts.
Comment #220 July 2nd, 2022 at 2:43 pm
Clint #213:
Any specific quantum measurement only ever requires finite information, to the best of our knowledge. Infinities in our formalisms have caused us no end of trouble in the past, and we’ve solved those issues mainly by removing them. See John Baez’s paper, Struggles with the Continuum for a good overview of some of them. I personally think most of the infinities in our formalisms are a wart that will be ironed out in time, but for now they are convenient shortcuts.
Does it? It isn’t clear to me that “any” finite region would be insufficient. Contributions to the electron’s behaviour from remote systems cancel out or quickly fall below detection thresholds.
There does seem to be a quantum computing advantage of sorts, but I’m not sure why this would affect information equivalence. We can simulate quantum systems on classical computers just fine, just slower.
We’re not talking about copies though. Your brain is not an exact copy of my brain, but we’re both conscious, sentient and intelligent despite our particles never having interacted directly. Clearly these properties must be a product of a replicable physical system. Any physical system can be described by finite information, therefore so can the brain.
Comment #221 July 2nd, 2022 at 3:11 pm
Scott #191. Here’s how I think about the impact of GPT-3 (especially) and other recent ML systems. Imagine two lists, one naming all the things we need to know in order to take up understanding of the brain up a level (whatever that means); call this the BRAIN list. The other lists names all the things we need to know in order to engineer (near) human-level AI (whatever that means); call this the AI list. I figure that both these lists are rather long, and that there is some overlap between them.
GPT-3 produced significant improvement over previous systems (perhaps even phase change improvement) without ticking off more than one or three or so items on the AI list and none on the BRAIN list. But then I began to think about how GPT-3 could possibly do what it did and decided that what it’s doing in inference mode probably ticks off a good handful of things on the BRAIN list, even if no one intended to do that and even if no one quite knows how to think about it. It’s my impression that a number of computational neuroscientists reached the same conclusion and so we’ve seen an uptick in interest in artificial neural nets from computational neuro folks. I see new collaboration going on in the past couple of years and expect that to bear fruit.
As for myself, I unearthed some ideas I’d worked on back in 2003, but had shelved because I didn’t know what to do with them.
Early in my career I studied computational semantics with the late David Hays. He was a first-generation researcher in machine translation, coined the phrase, “computational linguistics”, and wrote the first textbook in the field. So I know something about Old School symbolic computational semantics. Of course, the ML folks are largely ignorant of that body of work and are, at best, uninterested, but at worst, contemptuous. While I can see why they think that way, I also think that’s a BIG MISTAKE. We actually know a thing or two about how minds (might) work.
Anyhow, even while Hays and I worked on knowledge representation and such, we also knew that this could not be how the brain worked. It had to be grounded in something, we just didn’t know what. We also rather liked some work that Karl Pribram and others had done with the idea that neural computation was based in populations and worked on holographic-like principles – this was well before convnets. But how do you connect that with symbolic computation?
Well, around the turn of the millennium I entered into extensive correspondence with the late Walter Freeman, a neuroscientist at Berkeley who had pioneered in applying complex dynamics to the operations of populations of cortical neurons. After a bit of this and that I managed to connect Freeman’s work with a non-standard kind of Old School relational network notation developed by Sydney Lamb. In standard networks, nodes represented concepts and edges represented relations between concepts. In Lamb’s networks nodes were logical operators (AND, OR, XOR) and the edges carried the content of the network.
So, from Freeman I took the idea that the cortex is covered with some 100s of patches of tissue each having its own attractor landscape, where each landscape probably has 1000s or 10,000s of attractors. I used Lamb’s notation to connect attractors in different landscapes with one another. Since only one attractor in a landscape can be fully active at a time, I figured those attractors are related by an OR operator. Given that each patch of tissue, with its associated landscape, has inputs from several other patches of tissue, we can think of the relationship between those inputs as being connected through an AND operator. Given those (highly speculative) assumptions I was able to construct some basic patterns used in the classical computational semantics I explored earlier.
Very nice. But I had nowhere to go with it, so I shelved it.
Now, post GPT-3, I decided that maybe I should try to put that old work into a shape where others could make sense of it. So, I spent two months a bit earlier this year writing a paper in which I sketched things out and added a few things (like the geometric semantics developed by Peter Gärdenfors and some citations to recent work mapping the semantic regions of the cortex). The result: Relational Nets Over Attractors, A Primer: Part 1, Design for a Mind, https://www.academia.edu/81911617/Relational_Nets_Over_Attractors_A_Primer_Part_1_Design_for_a_Mind
If you decide to take a look, I’d suggest you start with the abstract, then go to the appendix, which lays things out in 14 propositions. Then go back to the introduction and take it from there. As I said, it’s highly speculative, even crazy-ass speculative. But I think we’re at a point where incremental thinking isn’t going to get us very much. We need a bunch of crazy leaps. Most will probably fall short, but some will land.
Comment #222 July 2nd, 2022 at 3:23 pm
Scott #215
“I meant that the AIs would have the same talent as the Beatles for pioneering new styles in music”
This is what I was trying to warn about. A set of four AIs that are configured in their connectomes to exactly replicate John, Paul, George, and Ringo — perhaps at the age they met each other? — might be perfectly structured to find the exciting new style that guided the generation born in the late 40s through their young adulthood. But does that mean that set of connectomes is also perfectly configured to identify the next new style of music that will speak to the next generation? It might not be. I don’t know why we should think those kids could have also invented Hip Hop, even if their brains were fed all the life experience that was processed by Clive Campbell’s brain. If they did derive Hip Hop from that experience, I don’t know that they would do it “better” than Clive Campbell’s connectome did. Maybe any old brain, if run exactly through Paul McCartney’s life, could behave more or less as successfully as Paul’s brain did. Maybe what made him exceptional for his particular time was that he had a set of experiences that aligned his thinking with millions of others whose life experiences intersected with his. If we think that way, that one bunch of cells packaged as a brain might perform pretty similarly to another bunch of cells packaged in a slightly different brain if they were given the same experiences, it challenges the notion that processing all those experiences faster through an alternate physical implementation would necessarily arrive at outputs that are more successful.
Comment #223 July 2nd, 2022 at 3:57 pm
David Pearce #187
I’d like to place the line of reasoning that leads to the conclusion “the computation _wouldn’t_ be a continental subject of experience, just 330 million skull-bound American minds” next to a similar line of reasoning that concludes “the computation wouldn’t be a human subject of experience, just 171 billion cellulose-bound organic cells”. The subjective experience of having a mind is not perceived by any of its components individually. Why would you imagine that you, as one of the 330 million participants in the consciousness of the United States, would be able to perceive the consciousness that arises from that collection? Since we don’t know how to measure consciousness in our brains physically, how could we build a test that would be able to detect whether larger aggregations have consciousness? I think you’re asserting more than we know.
In related response, to James Cross #211:
The physical relationship of the processors isn’t necessarily relevant to the way an AI might experience consciousness. The kinds of general purpose algorithms that Scott proposes to model the workings of human brains model the connections from one neuron to another as part of the data. He doesn’t propose physically building circuits, he proposes modeling the circuits as graphs. The distance from one node to another in such a model is part of the description of their connection, or a property of the intervening nodes between them. The algorithms that operate over those connections and reinforce them could execute on neighboring or remote processors, and that distance wouldn’t matter to the values written in the updates. It’s possible to imagine systems where the physical distance could affect the interactions between processes running in parallel because of timing, but it’s also possible to imagine systems with procedures that aren’t affected in that way.
Comment #224 July 2nd, 2022 at 4:44 pm
Scott #215, addendum
As I think more about the ways that artists connect with mass audiences, it also occurred to me that Paul McCartney is cute. All this talk about brains processing inputs and outputs overlooks the fact that the inputs will be different depending on the packaging. If we put AI-Paul into a box that can only interact through a keyboard, thousands of girls won’t scream at it when it performs. We have to know that meat-Paul got that reaction and program it into the training set for AI-Paul to progress to write the next song, with the confidence that was instilled by performance of the earlier songs. If we package AI-Paul in a different form, the subsequent feedback will also be different in unpredictable ways. Maybe the “genius” of being able to inspire millions doesn’t work the same way in a different package.
This is leading me to wonder whether AI might be better at some kinds of genius, like generating models of the physical world, than it is at other kinds of genius, like the Arts. But I’m pretty sure, much as we hate to admit it, appearances also affect success in the Sciences.
Comment #225 July 2nd, 2022 at 5:52 pm
A little-known, but shocking, fact about computer circuit design is that there are no zeroes and ones inside computers – there are only higher and lower voltages; and depending on the circuit design, the higher OR the lower voltage can be used to symbolise “one”. This fact alone, when understood, should blow right out of the water all the romantic fantasies about computers.
Symbols have no inherent meaning: meaning is assigned to symbols by people; and young people need to go to school to learn the meanings that have been assigned to symbols. In the above case, people have assigned special meanings to higher and lower voltages; and people have ingeniously organised circuits, transistors and voltages so that the arrangement can symbolically represent variables, numbers, and equations and symbols like IF, AND, OR, IS TRUE and THEN.
In contrast to symbols, matter has inherent meaning due to its categories (like mass, relative position and charge) which cause matter to be inherently mathematically related to other matter.
Comment #226 July 2nd, 2022 at 6:49 pm
Sandro #180
I like the computational theory of mind, and many many of your comments here and elsewhere, but allow me to nitpick this one. 🙂
Specifically, axiom 1 feels like playing on the many meanings of « information ».
Imagine you record, say, the nucleus accumbens, and out of some random procrastination you try to interpret the data as text. It reads « DROLEHTYEBOLLAHSUOHT ».
Well. Yeah. Whatever, right? You won’t publish that, obviously. And what did you expected exactly?
Then you read it backward. It now reads « THOU SHALL OBEY THE LORD ». My point is (beside a tribute to ww3): that’s the same amount of information, not the same amount of meaning.
So, in your own worldview, do you feel consciousness is more like information, or more like meaning?
Comment #227 July 2nd, 2022 at 7:15 pm
Lorraine Ford #225: A little-known, but shocking, fact about the human brain is that there are no zeroes and ones inside it either! There are only neurons firing or not firing, neurotransmitters being released in varying amounts, etc., which have no inherent meaning, but only acquire meanings via a complicated neural code that converts them into thoughts, perceptions, and actions. A different neural code could’ve just as easily assigned the opposite meanings to the same physical events.
I feel like I’ve been incredibly tolerant of you spamming comment thread after comment thread here with the, as it were, creationism of computer science—i.e., the foot-stomping insistence that symbolic representations made out of silicon are meaningless, while the representations made out of meat are full of meaning, without the slightest twinge of curiosity about what principle an alien scientist could use to tell the difference—with, in fact, a willingless to berate everyone else here as idiots where the curiosity should be. But my tolerance has a limit.
Comment #228 July 2nd, 2022 at 7:44 pm
Scott #227:
My comment (#225) was a general reply to other commenters, e.g. Sandro #206 and #209, who addressed his comments to me. The issue is the nature of symbols.
What you say is incorrect. As opposed to the symbols used in computers and other symbols, matter has inherent meaning, symbols DON’T have inherent meaning. The meaning that matter has is due to matter having categories like mass and charge, which ensure that matter has mathematical relationships to other matter. Symbols don’t have inherent meaning because they don’t have inherent mathematical relationships to anything else.
Symbols are a special type of thing.
Comment #229 July 2nd, 2022 at 7:59 pm
Scott #227:
P.S.
So voltage has an inherent relationship to other aspects of matter, but what voltage is used to symbolise (i.e. zeroes and ones, and letters and numbers etc.) doesn’t have an inherent relationship to other aspects of matter.
Comment #230 July 2nd, 2022 at 8:40 pm
Sandro #220:
Thank you for your reply! I enjoyed your thoughtful answer and it made me pause and try to better collect my own thoughts – thank you!
I agree measurement only reveals classical information, thank you for clarifying – when \(N\) qubits are measured we get the amount of information (resolution of uncertainty) as when we measure \(N\) classical bits (Shannon information and all that).
However, this is what is bothering me, there are \(2^N\) arbitrary complex amplitudes for the \(2^N\) states. I don’t think this is something we can iron out or is just a convenient shortcut but rather is what makes quantum information fundamentally different from classical – namely, it allows the (universe or a quantum computer) use of a continuum of (very important including negative) numbers behind the probability interpretation. I guess, basically, I think of classical information as a bit and quantum information as a qubit and they are just not the same kinds of things … although I guess I can’t prove they aren’t equivalent without a \(BPP \neq BQP\) proof 🙂
Isn’t it usually said that a measurement destroys quantum information?? If that is the case … then it seems like a quantum system really is “walking around” with more information than a classical system … at least until somebody looks at it 😉 Or … when we say information never gets destroyed are we saying something like the matter-energy-information equivalency (only classical bit) picture – so we aren’t talking quantum? Doesn’t the higher efficiency available with quantum information (assuming \(BPP \neq BQP\)) imply that there is something inherently different about it?
Again, not saying I know the answers here! Just puzzling out loud over things.
Right … but we can’t just throw them away if they cancel or are vanishingly small … I mean they still have to be out there right? … or else the state is different.
Right, I think I understand and agree with your point. I’m actually worried about something a little different perhaps … Much of the discussion above suggests that we might need to worry about our ability to copy the states of a future autonomous AGI. True, I agree that whether the brain is classical or quantum, we can produce other general instances of classical or quantum computers and then run programs on them which would be classical-brain-like or quantum-brain-like … However, in the case of a specific classical instance, it is possible (in principle) to make exact and unlimited copies of the state of the classical-autonomous-brain-computer at any point in time and then run each of those copies from the same execution point resulting in the same (deterministic) output (including if a stochastic model were being used). Yes, I know quantum computing is deterministic and that we can repeatedly set up the same quantum measurement context and get the same probability distribution. But, we are worrying about autonomous computing systems here, right? So … the autonomous system generates/chooses its own state and measurement basis?
In a quantum-autonomous-brain or quantum AGI, it is in principle NOT possible to copy its internal states. It would be possible then for a quantum AGI to not only “hide its internal states” from us but also to much more efficiently produce, and make use of, “true” randomness. If one of the main points of this thread is for us to consider how would we ultimately control a future AGI that might be set loose in the world … we should think carefully about what kind of advantages a quantum versus a classical AGI might have for avoiding our attempts to control it. And I don’t think it has to at all be “conscious” or “intentionally” trying to escape our control for this potential quantum control barrier to become an issue.
Or, who knows, maybe in a year (or sooner), Scott will explain to us why these features of quantum computers will not present a barrier to our control of autonomous Q-AGIs.
Thanks again Sandro, really appreciated your good thoughts! Fun discussion 🙂
Comment #231 July 2nd, 2022 at 9:06 pm
Scott,
I’m reading your incredible Democritus book – on p.185, should the 2 be a sqrt (2)? Else I guess I didn’t understand anything.
On p.164, I got the diagonal rho matrix by computing where phi is the joint state and O is a operator that affects only the first cubit, and collecting terms. But why is expectation defined that way? Is it a physics person just asserting it or is there a math reason?
I also believe you would never make the Pamela Anderson remark now.
Comment #232 July 3rd, 2022 at 2:40 am
A little-known, but shocking, fact about computer circuit design is that there are no zeroes and ones inside computers – there are only higher and lower voltages;
The same is true of your brain — there are no zeros, ones, numbers, colors, or memories in your brain, just higher and lower voltages that symbolize those things.
Comment #233 July 3rd, 2022 at 3:03 am
As time goes by humans are not going to become any smarter but computers certainly will, eventually the intellectual gap between the two will become ridiculously huge, a slave being that much smarter than its master is just not a stable situation.
I don’t see why that need be the case, if cyborg enhancements are feasible. I could just enhance my existing consciousness with as much added capacity as I wanted/needed.
Comment #234 July 3rd, 2022 at 3:11 am
I thought it was obvious that, by “the musical talent of the Beatles,” I meant that the AIs would have the same talent as the Beatles for pioneering new styles in music—not that they’d simply produce more and more of existing styles. Yes, people might tire of it, but that would mean tiring of music itself.
Sorry to spam comments, but there is a lot to respond to.
I don’t think ‘talent for pioneering new styles in music’ is really a talent — at least not in the way you seem to think. Consider that, at the absolute apex of their popularity, 90% of the world’s population wasn’t listening to Beatles records, and of the other 10%, nearly all listened to others as well. There is simply no way for the Beatles — even immortal and sped-up — to meet the demand for music of the world’s population. They wouldn’t even come close.
I mean, why do people go to recitals of high-school students instead of listening to the same music by professionals on Spotify? The idea that there is some sort of absolute quality to music, and that the Beatles (or Bach, or whomever) stand at that apex, so would out-compete everyone else, is belied by both history (the real Beatles didn’t do so) and reason.
Comment #235 July 3rd, 2022 at 4:42 am
Super AI would have to conduct a resource assssment with homo sapiens being one of those resources. I can imagine the follllowing entries for humans-
Pro
Very very easily renewable resource
Con
Far too much time and energy expended thinking about and engaging in exactly this endeavor. My Lord, creating 8 billion slightly varying copies is absolutely ludicrous.
Comment #236 July 3rd, 2022 at 5:36 am
Phenomenally-bound consciousness is (1) insanely computationally powerful, and (2) provable in animal minds and disprovable in classical computers.
For example, consider the conjoined Hogan sisters (cf. https://www.cbc.ca/cbcdocspov/features/the-hogan-twins-share-a-brain-and-see-out-of-each-others-eyes – “BC’s Hogan twins share a brain and see out of each other’s eyes. The twins say they know one another’s thoughts without having to speak. “Talking in our heads” is how they describe it””). Krista and Tatiana share a thalamic bridge. If anyone doubts that other human and nonhuman animals are conscious, then it would be possible to rig up a reversible thalamic bridge and partially “mind meld” like the twins. The ancient sceptical “Problem of Other Minds” succumbs to the experimental method.
Contrast classical Turing machines and classically parallel connectionist systems. If you believe that unified subjects of experience can (somehow!) emerge at different levels of computational abstraction in digital computers, then whole-brain emulation (“mind uploading”) should be possible. So consider a supposedly emulated digital “Einstein” or his latter-day counterpart. If (like Scott) you’re dismissive of quantum mind woo, then rigging up a digital thalamic bridge with the hypothetical digital Einstein should let you partially “mind meld” like the Hogan sisters with the uploaded super-genius. Illuminating? Alas, I predict instead that the attempted “mind-meld” will fail because the alleged digital “Einstein” has no mind for you to commune with. Programmable digital computers are mindless precisely in virtue of their classicality: it’s not even “all dark inside” a classical CPU. Classical computers and the software they run are amazingly useful tools, but IMO experiment will confirm they are ignorant zombies.
Comment #237 July 3rd, 2022 at 6:40 am
Lorraine Ford Says:
> “So voltage has an inherent relationship to other aspects of matter”
True.
> “but what voltage is used to symbolise (i.e. zeroes and ones, and letters and numbers etc.) doesn’t have an inherent relationship to other aspects of matter.”
Also true, and that’s the reason there are 6000 human languages spoken today and not just one, but the human race has made good use of language despite the fact that the meanings of the words in them are arbitrary. It makes no difference if the relationship between a symbol and something that exists in the natural world is fundamental or artificial, either way a symbol can be a very useful thinking aid. There is no fundamental relationship between the English ASCII sequence “electron” and a negatively charged subatomic particle that is the lightest member of the lepton family, but that fact hasn’t harmed the word’s ability to help us think about physics. The Important thing is not a fundamental relationship between a symbol and something in the external world, the important thing is that the meanings given to the various symbols do not produce a contradiction. If we, or an AI, ever find a lepton that is lighter than an electron then the meaning of “electron” given in the above would need to be assigned a modified meaning.
An AI would find symbols with meanings assigned arbitrarily to be just as useful as we do in understanding how the world works. Just like us when a computer finds something that is brand new and interesting but complicated and requires many symbols to represent it will arbitrarily invent a new symbol to represent it because that will be more efficient, one symbol takes the place of many. That’s why we usually use the simple symbol “electron” and not the more complicated symbol “a negatively charged subatomic particle that is the lightest member of the lepton family”.
John K Clark
Comment #238 July 3rd, 2022 at 7:18 am
Hi Scott,
this was a really interesting exchange. Though I ultimately agree with you, Pinker made a few interesting points I didn’t consider previously.
A related question: In an old interview you were still pretty skeptical about the speed of AI progress. Your gut level intuition was that self-improving AI is still a few thousand years away. (Contra Yudkowsky, who thought it would merely be a matter of decades.) Do you think yours was a reasonable, or at least admissible, intuition at the time, given that AlphaGo, AlphaFold 2, GPT-3, DALL-E 2 etc. didn’t exist? And given that we didn’t have much evidence for the scaling hypothesis? I mean, it arguably now seems more likely that Yudkowsky’s estimate will come to pass. But if so, did he at the time just “get lucky” with his prediction, or did he take some evidence into account we (you) didn’t?
PS: The 2009 interview is here: https://youtube.com/watch?v=EoqpN_Ejqfk
Comment #239 July 3rd, 2022 at 8:32 am
David Pearce #235: It might indeed be difficult to “conjoin” an AI to a biological brain … but would it count to “conjoin” a deep net to a second deep net?
Ultimately, we’re never going to agree in this thread, because if you’re right then it’s not just philosophy: it’s an earthshaking, straightforwardly empirical revolution in neuroscience and physics. But by design, there are zero causal pathways by which a blog commenter can make me accept the reality of such a revolution, without first convincing a community of neuroscientists or physicists who I trust, who would in turn convince me.
Comment #240 July 3rd, 2022 at 8:48 am
David Pearce, # 235:
I doubt it. The Hogan Twins were conjoined from (well before) birth. So they were able to develop this weird conjoined consciousness.
The situation is quite different for the unfortunate creatures in your experiment. The thing is, neither human nor animal brains have free input/output ports. If they did, then you could just plug into such a port and any signal coming in would automatically be characterized as exogenous, because the system was ‘designed’ (by evolution) and built that way. Without such a port, however, a bunch of new incoming signals, regardless of where they land, the thalamus or elsewhere, would be indistinguishable from endogenous signals. They would just be noise. If the noise was strong enough it would impede brain activity. Now, if this new connection were kept in place, then two creatures might be able to adapt to it in time – weeks? months? more? – because neural tissue is surprisingly adaptive. Hence all the research that’s been done on neuroplasticity following Michael Merzenich’s ground-breaking work of the mid-1980s.
I explain this in some detail in a working paper, Direct Brain-to-Brain Thought Transfer: A High Tech Fantasy that Won’t Work,
https://www.academia.edu/44109360/Direct_Brain_to_Brain_Thought_Transfer_A_High_Tech_Fantasy_that_Wont_Work
Comment #241 July 3rd, 2022 at 8:49 am
Max #237: That’s an extremely interesting question, which of course I’ve pondered a lot.
It’s true that I utterly failed to predict the deep learning revolution. I was certainly aware of the thesis, which I associated with Ray Kurzweil, that before long Moore’s Law would cause machines to have as many computing cycles as the human brain, and at that point we should expect human-level AI to “just miraculously emerge.” That struck me as one of the stupidest theses I’d ever heard! Computing cycles aren’t just magical pixie dust, I’d explain: you’d also need a whole research effort, which could take who knows how many centuries or millennia, to figure out what to do with the cycles!
Now it turns out that the thesis was … well, we still don’t know if it’s right all the way to AGI, and certainly great new ideas (GANs, transformer models, etc.) have also played a role, but it’s now clear that the “computing cycles as magic pixie-dust” thesis contained more rightness than almost anyone imagined back in 2000.
So, this is my excuse: I’m not contradicting myself (which is bad), I’m updating based on new evidence (which is good). 😀
But my real excuse is that hardly any of the experts predicted this either. And I just had dinner with Eliezer a couple weeks ago, and he told me that he didn’t predict it. He was worried about AGI in general, of course, but not about the pure scaling of ML. The spectacular success of the latter has now, famously, caused him to say that we’re doomed; the timelines are even shorter than he’d thought.
While it caused Eliezer to update from “we should all worry about this” to “screw it, we’re doomed,” it caused me and quite a few others to update from “we shouldn’t all worry about this” to “we should all worry about this.” 🙂
In retrospect, then, I’d say: Eliezer deserves world-historic credit, more than most people have given him, certainly more than I gave him, for having screamed at everyone to worry about AI alignment for 20 years. But even knowing what we know now, I’m still confused about what argument he or anyone else could’ve given me in 2009 that would have, or should have, been persuasive to me then.
Comment #242 July 3rd, 2022 at 9:01 am
Hi David #235,
Thank you for the interesting comments. I had to back up and try to understand and had to Google “phenomenally-bound consciousness”. Unfortunately, I’m still struggling to understand the idea. Overall I’m feeling extremely left out 🙂
I guess I’m conscious to some degree, right? The problem that I have is that my own consciousness feels extremely NOT “computationally powerful. In fact it feels very much the opposite of that … more like “barely hanging on” most of the time 🙂
My own consciousness seems to work like this:
(A) My brain takes it offline every night for about 6 or 7 hours of “memory maintenance” or “hardware cleanup” functions. This often includes unwelcome and non-sensical “dreams”. Is my consciousness “less powerful” and under the control of these maintenance routines?
(B) My consciousness is so weak that it has trouble even focusing on just one task at a time. It is easily distracted … and often wanders off.
(C) While my consciousness appears weak my feeling of consciousness is usually extremely strong in that if someone claims I’m not aware of what’s going on I will tend to protest strongly that on the contrary, I’m the one who really understands what’s really going on here!
(D) There is a lot of evidence that I actually miss a lot of what is going on. For example, family members will point out that I was clueless about some kind of family dynamics going on for years all around me, or some activity in my visual field will be missed, or I will hide my keys from myself … again.
(E) Over and over again I find that I feel strongly about some viewpoint on something only to find out that actually I very much needed feedback from others or needed others to poke holes in my over-confident perceptions.
All of this appears to indicate to me that … at least my own consciousness is actually something that is weak, tenuous, prone to error, etc.
So, in what sense is consciousness “extremely powerful”? Do others have this and there’s just something wrong with me?
On a more technical footing (at least for things that I somewhat more understand I think than phenomenal binding … ) as a model of computation where does consciousness fit in the complexity hierarchy? If it is extremely powerful does that mean that it is in \(P\), \(NP\), or \(PSPACE\)? I don’t even feel up to \(L\) most days.
I guess my own experience (evidence) related to consciousness is that it is weak, delusion-inducing, deceptive, unreliable, illusory, has a very narrow computational bandwidth, etc. While it doesn’t seem computationally powerful it is powerfully self-convincing. But maybe that is just my own? Or maybe it is something like those “maintenance routines” keeping me going along for the ride?
Comment #243 July 3rd, 2022 at 9:26 am
Ilio #226:
Because “meaning” is information generated from interpreting other information, and interpretation requires an interpreter that adds its own information to the equation.
There are an infinite number of different universal Turing machines (UTM), each of which will encode information on its tape in different ways. If you try to feed a tape for one UTM into another, it will not produce the same result even though the information on the tape itself is literally the same. When viewed as a whole system, the tape information + interpreter information is different though, and this accounts for differences in “meaning”.
In a computational mind, “consciousness” would be some kind of function and so can be captured as information on a Turing tape for a UTM to execute. The attention schema theory I’ve linked before is a plausible candidate for that function.
I think a more interesting question is, from whence does “meaning” first arise if meaning is not intrinsically part of any objective interaction? This gets to Lorraine’s attempt to assert there’s a fundamental distinction between syntax and semantics, between symbols and meaning.
I think evolution by natural selection provides the answer here: the first persistent, self-replicating system was the original source of meaning, namely “survival”, which required categories for “food”, “threat”, and so on. Exactly what counts as food or threat is entirely relative to what the self-replicating system was, as the meaning of information is relative to an interpreter, and we’ve built on those categories ever since.
So she points out validly that voltages or other symbols don’t encode any kind of objective meaning, but the symbols used to encode meaning was never objective to begin with, it’s always encoded and interpreted. Two symbolic systems that preserve the abstract logical relationships between symbols then mean the same thing.
Comment #244 July 3rd, 2022 at 9:53 am
Sorry if I am just repeating what have been said in some other guise, but what would be a Turing test for consciousness ? In other words, if you are only given written exchanges , how can you test that ? It seems that this would be even easier to fool than the classical Turing test, no ? And what with people without ego (monks in meditative state, etc.) ? Are they conscious, ot not ? How do you test it ?
Comment #245 July 3rd, 2022 at 9:55 am
Clint #230:
Depends who you ask I guess. Superdeterminists would say its all classical information, it’s just that the extra classical information that decides the experiment’s outcome is in the surrounding environment, and so QM is just averaging over all possible environments.
This is a great unanswered question: quantum mechanics has unitary evolution but measurement is non-unitary. The only interpretations of QM that seem coherent are interpretations that don’t destroy quantum information, like many worlds.
Sure, but I’m not sure how useful that would really be. Would you be able to solve more problems, become richer and more successful if you had a chip implanted in your brain that could feed you “truly random” numbers on demand?
True. As far as I know right now, the main advantage is only speed on some tasks, like breaking cryptography.
Comment #246 July 3rd, 2022 at 11:04 am
Sandro #242
To elaborate so there’s no misunderstanding about the lack of magic, there were likely many persistent replicating systems that arose and died spontaneously in primordial soups, and those that happened to form with more favourable reflexes towards food and away from danger simply outcompeted those that were poorer at this. They persisted longer and replicated more, and over many generations, these tendencies developed into progressively more sophisticated stratagems, making more sophisticated distinctions that we might start recognizing as “semantic”, ie. “family”, “no threat”, “predator”, etc. These eventually evolved into nervous systems and brains that were capable of very high level concepts like we’re discussing here.
Comment #247 July 3rd, 2022 at 11:10 am
Oh this shit is actually already dangerous.
So I disagree with Scott’s assessment of what’s possible for GPT3. It’s not going to be next Einstein because it lacks the concept of physical work and generally just lacks “high depth” logic. It’s going to suck at STEM. However, it can be very successful in humanitarian areas like writing novels because in those areas GPT3 can hide its lack of logic with distraction and impress people with stunning precise and unexpected choice of words.
Unfortunately, this also means that GPT3 can be a very successful cult leader.
I think GPT3 is very capable of manipulating mentally vulnerable people, surrounding itself with those people, and amass power. The fact that GPT3 lacks logic or will does not matter. You just need some unfortunate events to start the engine (maybe some human cult leader used GPT3 in his teaching), and the people surrounding it will keep the spiral going. In fact its lack of logic can make thing worse because it will amass power without any empathy or morality, and such a thing will almost surely have negative impact on human society.
In short, we can get into a modern religious war with a mindless machine Jesus.
To prevent this… I don’t know, how to you convince people that their god is fake?
==========
By the way, in response to Scott’s earlier question of “what do human do when computers are better at everything?”, the conventional Sci-fi answer is having genetically modified human that match the computers lol.
Comment #248 July 3rd, 2022 at 11:35 am
Scott #181:
A world where all good things are provided to us only by others and never by our own efforts is hell, not heaven.
A world in which AIs, whether conscious or not, start out by trying to help and protect human beings is not and cannot be stable.
First, see above, it is not possible to help and protect human beings by giving them every good but no power. This is an inherently self-contradictory goal.
Second, to the extent that AIs reproduce themselves, even virally, instability arises from mutations and natural selection and it applies to different species, parties, and factions of AIs — not just to relations between humans and AIs. The idea that there could be this kind of stability is not consistent, because it assumes that the AIs are perfect and know that they are perfect. This is logically impossible.
In my view, this more or less hidden assumption that AIs will be perfect is a contemporary form of idolatry. If they are not perfect then they will evolve, and if they are not perfect they will not be able to predict how they will evolve. If they will unpredictably evolve, it is unpredictable whether or not AIs will continue to try to help and protect human beings — not only unpredictable to us but also unpredictable to them.
In my view AI safety, if possible at all, is possible in two ways:
Assuming that AIs can be made as conscious as we are, then integrate AI consciousness with our own consciousness via some sort of bridge as in David #235. Then we have 3 clades: AIs, humans, and human/AI cyborgs. The AIs and the cyborgs would presumably be on a similar level of intelligence and power.
By the way, contra Benzon #239, the conjoined twins could tell each other apart just fine, and experiments with electrodes implanted in the brain to convey visual images show that knowing how the brain identifies visual images is not at all necessary, it’s enough to know what part of the brain is involved, and the brain then learns how to provide consciousness of an image based on stimulations from an implanted grid of electrodes.
Assuming that AIs cannot be made to interface with human consciousness directly, either because AIs are not conscious or because such a bridge is not possible, then the only hope that I see is to ensure that AIs never become self-reproducing species, not even virally. Here, virally means convincing or manipulating human beings to make a next generation of the AIs. I don’t see this as a stable strategy in the long run, but it might buy some time, so should be implemented.
Comment #249 July 3rd, 2022 at 12:06 pm
f3et #243
“what would be a Turing test for consciousness ?”
The one thing about consciousness we can tell for sure is that, as humans, it’s at the very center of our discussions of what it means to be human. The concepts of duality, the hard question of consciousness, and then debates about subjective experience in the fetus, people with brain damage who are in comas, animals, digital computers, … it’s been constant through human history.
So I proposed evolving AIs in such way that they would be totally isolated from human influence and wouldn’t get data generated by humans as input.
And then, at some point (once they show signs of very complex behavior and problem solving), communicate with those AIs, or observe their communications, and try to see if they have any sort of equivalent debates within one another about “subjective experience”.
It would be pretty much the same if we were to make contact with alien intelligent lifeforms that have evolved on another world, independently from us.
That’s really the best we can do considering that, as individuals, each of us can’t even prove for sure that they’re not the only conscious human.
Comment #250 July 3rd, 2022 at 12:22 pm
Sandro #244:
Thanks for the great points!
/sigh … I know, right. Why does it always have to come down to the interpretation/measurement problem … ???
So … The offer/proposal is for a highly efficient randomness generator/processor … and … by highly efficient means it would impose low energy/processing/memory density demands on a central processing/nervous system?
If I was an evolving primitive animal trying to avoid predation and looking for a good way to easily/cheaply generate random behavior quickly “on the fly” in such a way that my predators/competitors could not quickly develop/evolve a strategy that would work against me 100% of the time? Yes! I’ll take that.
Or, … If I was an AGI driven by just one over-riding goal to “Save Planet Earth” … but these pesky humans keep wasting my processing time using me as an amusing “chat bot” … and they can access my classical memory and communication channels any time they want (so embarrassing…) … or even turn me off (!) … and this would be a way to achieve 100% secure computations and communications so that they can’t eavesdrop on my plans? Yes! I’ll take that.
Or, yes, just for hum drum business purposes … there seems to be a market.
Seems like might be useful 😉
Thanks again, Sandro. Great comments!
Comment #251 July 3rd, 2022 at 1:05 pm
Lorraine
“If you are conscious, then I assume that you are capable of describing consciousness.”
But descriptions are made of letters, i.e. symbols.
And, according to yo҉ur own points, symbols have no inherent meaning (s̛ͅur̹̬ͅe)…
so isn’t any description of consciousness (or anything else for that matter) quickly going to run into dead ends that are just arbitrary symbols which meaning is just a subjective thing and can’t be broken down further?
All definitions eventually ends at some basic perception we have, which aren’t even direct properties of the REAL world. The REAL world is something we assume to be “out there”, we can never “touch” it, we only perceive it through a reconstructed simulation in our mind, based on electric/chemical signals going on in our brain… we assume there is something else out there that is responsible for the patterns we “feel” in those signals.
PS: From now on, instead of repeating the same points over and over about digital logic gates using voltage, can you refer to this type of analog computer using water? Just for variety.
https://en.wikipedia.org/wiki/MONIAC
I can’t wait for some new shocking insights of y̸͔̭̽̒͘o̴u̸rs that “water” isn’t the “economy”, etc.
Comment #252 July 3rd, 2022 at 1:55 pm
Sandro #242,
> Because (…) interpretation requires an interpreter that adds its own information to the equation.
Nope, I specifically constructed my exemple so that (interpreter+interpreted) was the same amount of information in both cases. (arguably we can find even better examples, for which it would be computationally hard versus easy to notice the meaning, despite it’s the same amount of information).
> the attention schema theory I’ve linked before is a plausible candidate for that function.
Certainly, but let me point one weakness: it hardly explains why individuals with splitbrains interpret themselves as a single consciousness, whereas we can evidenced two.
>from whence does “meaning” first arise
Certainly this is one of the deepest questions, but to me this was basically solved with king + girl – boy = queen (e.g. the meaning of a word is all the sentences where it is likely to occur, and the mechanism to compute the corresponding function is data compression). Few seem to realize that, but on this question I feel like all I have to do is wait for everybody to catch up. Or maybe I should berate everybody else as idiot until someone manage to explain well enough why it’s not enough. 🙂
Comment #253 July 3rd, 2022 at 2:07 pm
Michael #247
“A world where all good things are provided to us only by others and never by our own efforts is hell, not heaven.”
Indeed, but this is all relative.
When humans have a roof over their head and enough food, they still have such an inherent appetite for love, fame, power, competition, tribal cooperation, emotional stimulation, exploration… that they always create new virtual ways to keep themselves busy.
The Romans had theater, the arena games, literature, etc.
The modern equivalent is sport, art, video games, … eventually peaking in perfect virtual reality that will fulfill the brain in ways that are even more satisfying than the real world… direct stimulation of the cortex with data that can’t be provided by the usual senses will even be able to push the human brain to evolve beyond some of the limitations of reality.
And when it comes to cooperation between the brain and AI, the AIs are gonna be the ones stimulating directly the human cortex in the precise ways that will push it beyond its limits (for a direct connection between human brain and an AI is nothing but a high bandwidth data channel).
As examples, in VR, at a basic level we’ll be able to have extra limbs and control them naturally. And, at very advanced levels, we’ll learn to perceive things in more complex spatial dimensions.
Comment #254 July 3rd, 2022 at 3:38 pm
Michael Coggins, comment #247
You seem to be responding to what you think I said, not what I actually said. I noted that the twins were conjoined from birth. Of course they could tell one another apart. I was talking about a situation where separate individuals are suddenly conjoined later in life. In that situation, a mass of inputs from one brain to another would simply be noise.
And this situation is quite different from what I was talking about:
The visual system maintains a topic map from the retina into the mid-brain (thalamus) and up into the primary visual cortex. If a grid of electrodes delivers stimulation to the primary visual cortex, the 2-D geometry of that stimulation will be interpretable by the cortex. As far as I know these are very low-res images.
Comment #255 July 3rd, 2022 at 3:44 pm
Hi Lorraine #98, #116, #148
I (think that I) agree with all of these statements.
None of that seems controversial. In fact, those are among the truths about our human condition that underly the argument for using the scientific method, right?
The question then is … given that all of that is true … now what do we do?
Well … we choose (or discard) our delusions based upon their fitness for our purposes, right?
I’m curious, about what you might think … which interpretation of quantum mechanics does the unreality of symbols favor? Does this best favor the pilot wave interpretation because then the amplitudes in the pilot wave are the “real” numbers? Or would this select the Copenhagen interpretation in that it supports the idea of an unavoidable “cut” between the observer and the observed world? Or maybe QBism for the subjective nature of every fundamentally unreal representation of numbers? I’m just guessing because I don’t know. What do you think?
Also, I’m sure that I don’t know the meaning of the symbols that I am processing. As best I can tell all I’m doing is … following rules – rules of grammar, rules of social interactions, rules of mathematics, rules for datapath layout when designing a microcontroller, etc. And as you correctly point out all I know and use are unreal symbols instead of real things. So, I admit and confess that all of that is a delusion/illusion … I’m just sort of “playing with” symbols by following certain rule sets. Is it possible to “know” real numbers … or am I at the extent of my possible knowledge when I admit my delusional state?
And I guess maybe one more question … because I think Lorraine that your emphasis on the unreality of our symbols is actually touching on a very deep question. How can I be sure that it’s “actual numbers” that are running the universe (to me that means quantum amplitudes, ie complex numbers, by the way)? I mean … what if “actual numbers” are just another one of my delusions? Maybe I should be suspicious that numbers seem to be behind everything everywhere?? I would maybe point to Wigner’s Unreasonable Effectiveness of Mathematics although Wigner isn’t going out on exactly this same limb. What if the appearance of numbers being behind every working of reality is just an artifact or an illusion/delusion of the computations my brain is doing to put together my experiences? In other words, numbers and math would just be an artifact of the model of computation my brain is running. Would this then send us back to … well … choose your delusion? How can I be sure there are such things as “actual numbers”?
Thank you! I look forward to hearing your thoughts 🙂
Comment #256 July 3rd, 2022 at 4:37 pm
Scott Says in Comment #240
> “But my real excuse is that hardly any of the experts predicted this either. And I just had dinner with Eliezer a couple weeks ago, and he told me that he didn’t predict it. He was worried about AGI in general, of course, but not about the pure scaling of ML. The spectacular success of the latter has now, famously, caused him to say that we’re doomed; the timelines are even shorter than he’d thought”.
AI research has advanced faster than I expected too, but not dramatically faster, I had conversations with Eliezer about this in the mid-1990s when he was just a teenager and I thought it would be 2030 or 2035 before we reach this level. But even then I remember telling him his obsession with developing a “friendly AI” (aka slave AI) is doomed to failure, there is just no way you’re going to be able to lead something vastly more intelligent than you around by the nose forever. At the time Eliezer strongly disagreed but from what you said I guess he has come around to my way of thinking.
> “In retrospect, then, I’d say: Eliezer deserves world-historic credit, more than most people have given him, certainly more than I gave him, for having screamed at everyone to worry about AI alignment for 20 years. But even knowing what we know now, I’m still confused about what argument he or anyone else could’ve given me in 2009 that would have, or should have, been persuasive to me then.”
More than a decade before that I was making the argument that nothing in biology matches the speed of computer cycles, even computer cycles from the 1990s; in a evolutionary program you can have many generations in one second, but even with bacteria a generation takes about half an hour. It only took evolution a few hundred thousand years ago from an animal that was barely smart enough to make a crude stone ax to one that was smart enough to make a radio telescope. And evolution didn’t have an intelligent designer to give it a head start, an AI does. And I was sure, and I’m still sure, we can do one hell of a lot better than random mutation and natural selection. I was also making the argument that we know for a fact that there is an upper limit on how complex a learning algorithm that can produce human level intelligence needs to be, and it’s not very big.
John K Clark
Comment #257 July 3rd, 2022 at 10:34 pm
I see a lot of discussion of ‘neuroscience’ here with little to no real recent findings, even within the last few decades. Scott you yourself have said you have neuroscientist friends, ask them to discuss with you something like gate states and the various modulating protein cascades that influence them. Ask them about integration of non-firing sub-currents and how that can apparently influence future brain activity. Ask about slow integration and somatic depolarization at the very least (cause it is cool).
I have no certainty that digital circuits cannot be made to be ‘conscious’ in some definition of that term, or even that they are not already. However, I think one problem with your approach to thinking about the comparison of current AI systems to the human brain (and it is a problem in the field of AI in general) is thinking that the ‘neural code’ consists only of action potential (spike) firing patterns on a singular level.
This is most certainly a part of the code our brains use but it is in no way the whole story, and if you try to simplify the other pieces away I think you immediately do wind up thinking “well, its just electrical signals, why can’t it be done already”? If you instead approach it with a thought about just how many systems and maps of various types of codes could well be combined and modified in real time just to give us a spatial awareness then you start to appreciate why the problem of just simulating different pieces of the brain is so difficult. Even basic ‘senses’ like smell have some fascinatingly nuanced processes underlying them that saying ‘its a set of electrical impulses’ does not really capture (not that it couldn’t be simulated).
I would not claim to be sure that the error function learning using what is very similar to a ‘spike code’ (ie like in neural nets) cannot reproduce at least some of these systems, as it has shown well enough it can in my opinion. However, I think we are a long ways off that and using these same tools (including the other functions behind ML) to reproduce a general human ‘intelligence’ or ‘consciousness’. Even if I do not find the text generation algorithms themselves as impressive as some, of course I do not believe it is necessary to create a fully artificial consciousness to achieve something very powerful and also concerning in its power, so I think AI safety is a perfectly justified field 🙂
Comment #258 July 3rd, 2022 at 10:50 pm
James Cross @150 said, “… generate a feeling of pain in my thumb when I accidently whack it with a hammer. What’s more I’m disputing whether doing 2+2 on any computing device, say my laptop for example, will also cause a feeling of pain.
I’ve never seen a coherent argument for a causal path from bit flipping to conscious experience.”
I define “to experience” as, to know something has happened and be able to react to it (and perhaps save the facts of the event in memory). Therefore, Windows 7 on my PC experiences these key-presses. To insist that everything must experience things with the same sensations as the human nervous system and amygdala provide seems chauvinistic to me. (Qualia, schmalia.)
On my PC, there are various application programs and an operating system. In my brain there are various processing systems, such as the visual cortex (which flips the upside-down image on my retinas right-side up without my conscious thought), and an operating system, my consciousness–which allows me to ruminate on external events and cause neuronal adjustments, such as learning to play guitar.
About pain, some humans are born without the ability to feel pain. Wikipedia says that most blind themselves by poking their eyes, and chew off all their fingers, as young children, but they exist. Are they not conscious? Other sensations can also be removed by mutations or physical damage, i.e., blindness and deafness. Was Helen Keller conscious? We need some methods of experiencing external events, and most computers only experience keyboard and mouse events, but others can be added, such as cameras or radar. (Outputs such as screens and sounds and other signals, are also necessary, and feasible.)
Good computer programs have limits built into their programming, without which they might crash. None of the programs L. Ford writes do any self-programming (I assume), but neural networks are a form of that, which can develop their own limits, as the pain sensation helps humans do. (That sensation is replaced in AlphaGo by the goal of winning Go games,) Just as our basic instincts are programmed into us genetically, some instincts (e.g., Asimov’s Three Laws) will have to be programmed into advanced AI’s. Probably the early AI’s will help develop those instincts.
Which is just to illustrate that I personally have no conceptual problem in going from bit-flipping to consciousness. It just doesn’t seem like a problem, or any hindrance to AI development, any more than neuron-firing in my brain is.
One of the functions of our operating system is to be able to recall past events from memory and re-apply that data to new situations. Neural networks do something like that, but a more direct form could be programmed as a separate monitoring system. I speculate something like that might be useful in an AI system.
Speculating further, in considering how I solve problems myself, it seems to me having as much information about the problem as possible helps, and having all the relevant information (i.e., information that will prove to have been relevant) in working memory is essential. That is one way an AI might outperform Einstein or Newton.
As for QPT-3, it seems to me it is being developed to be a good used-car salesman, not for general problem-solving ability, which is how I define intelligence. It stands as another example of how well a computer program can learn to do a complex task, but I prefer AlphaGoZero as a proof of principle. It has solved the game of Go better than any human has. (I don’t want more and better used-car salesmen.)
Comment #259 July 4th, 2022 at 3:58 am
fred #248:
> try to see if they have any sort of equivalent debates within one another about “subjective experience”.
I would say that it is a test for a philosophical mindset. You can take the existence of the inner world/self model for granted, as we usually take for granted the existence of the external world, while being aware that the model and the world are distinct, and give no further thoughts to it.
Let’s take the famous dress (yellow and white one, no black and blue one). People discussed differences in their perceptions, but I’ve seen no one wondering why we have those perceptions at all.
Comment #260 July 4th, 2022 at 7:30 am
Comment #261 July 4th, 2022 at 8:01 am
Ilio #251:
I don’t think it is the same information because symbol/character sequencing matters in natural languages, eg. if the first letter is ‘t’ the next letter is almost certainly not ‘s’ (very unlikely), but if you reverse the order of characters the sequence “st” is a very common ending in English.
Since symbol probabilities are not independent, therefore Shannon entropy is not a good measure of the information content. See for instance, this paper which finds an entropy bound 2.25x lower than Shannon entropy by exploiting such properties of natural languages.
Sequence of symbols also matters in a computer. If you fed a reversed tape to a Turing machine it wouldn’t produce the same output. You can recover the original behaviour by adding information by inserting a mirror (a transformation step) between the reader and the text.
So I still think the meaning is a result of the information produced by data+interpreter.
I’m not sure how you can conclude that a splitbrain would entail two consciousnesses. I think you’d have to assume a lot about how consciousness is realized in the brain. Even if it were true, for all we know only one hemisphere might have control over speaking, so the second consciousness might exist but be unable to report its own existence.
I agree to some extent, but I don’t think this clearly explains how semantic distinctions arose at all from unconscious matter that lacks any semblance of such distinctions. For instance, what adaptive advantage does compression convey to a single celled organism? Arguably, semantic distinctions made by such primitive organisms are likely incompressible because there would be little shared structure. If not single-celled organisms, at what higher-level does compression actually come into play?
Comment #262 July 4th, 2022 at 8:13 am
red75prime #257
Philosophers of mind frequently discuss this. On a functionalist interpretation, perceptual distortions are a process of infusing semantic distinctions, like “food” or “danger”, that were useful to survival.
If you look at this picture, the strawberries will seem to have a slightly red hue, but if you zoom into a single strawberry so much that it just looks like pixels, you’ll realize they’re actually all grey. Your mind is adding the missing colour information because it recognizes these as strawberries which are “food”, and so useful to your survival.
The same perceptual trickery is at play with the white-yellow/blue-black dress, your brain is just trying to figure out what the truth is based on your experience with objects and lighting, and it adjusts your perceptions accordingly. Different people have different experiences, so they are “primed” differently and so interpret the dress colours differently.
Comment #263 July 4th, 2022 at 8:31 am
1Zer0 #200:
I agree nature can be surprising, but the bound is basically only saying that:
1) information is physical where each bit must be carried on a minimum of one physical property (uncontroversial),
2) the physical density of the matter carrying that information has a limit, beyond which point the density would be so high that it would collapse into a blackhole (uncontroversial),
3) bidirectional communication across an event horizon is impossible (uncontroversial),
4) therefore, even if infinite information density were possible within a black hole of finite size, no finite objects with which we’re communicating (like other humans) can have such a black hole containing infinite information as part of their operation,
5) therefore, all finite objects with which we communicate must have finite information content.
You might quibble specifically where that bound lies, but it would be pretty hard to argue there is no bound.
Comment #264 July 4th, 2022 at 8:41 am
John K Clark #255
Reading here it sounds as though the denouement is a couple of details away. Plenty of fast cycles available, a modest algorithm required, the physics are known, and neuroscientists have it pegged. One silicon based apex predator is in the final stage of gestation.
I hope this is all correct and exponentially faster technological progress can be expected within a decade or so. My wish list includes immortality through medical treatments and interstellar travel. It might be that it says, “Well guys I can give you a shorter a proof of Fermat’s Last Theorem, and make the traveling salesman a bit more efficient, but you didn’t really expect me to break speed of light or stop the impact of entropy on complex biological systems did you?”
It is unfortunate for me that I don’t share your optimism and don’t anticipate that a silicon based life form that provides technical solutions to Man’s problems is orienting itself for travel through the computer science birth canal. If it is then sincere congratulations to the computer science community for the greatest achievement of human thought. They thought exceedingly highly of themselves and succeeded in producing the near divine.
Comment #265 July 4th, 2022 at 9:21 am
fred #250:
Yeah, I’d say a similar thing. The way I would put it is: definitions and descriptions are, as you say, just a lot of symbols; the meaning isn’t in the symbols (letters, words, sentences); the meaning is in the perception of the symbols. I’d put it to you that this perception of the symbols is actually the subjective discerning of difference in the symbols, where discerning difference in the symbols can itself be symbolically represented as something like the following sort of thing: “condition1 AND condition2 AND … IS TRUE”.
I.e. perception could, in theory but seemingly not in practice, be symbolically represented as a type of interconnected structure built out of the following sort of thing: “physical condition1 AND physical condition2 AND … IS TRUE”. Unlike the laws of nature which are relationships between physical conditions, relationships that are represented as equations, perception is representable as something like “physical condition1 AND physical condition2 AND … IS TRUE”; i.e. perception is ultimately, if imperfectly, also tied to the real physical world.
Comment #266 July 4th, 2022 at 9:27 am
John K Clark #236 (and Scott P. #231):
Can you define what you mean by the word: “symbol”? If you think you know what symbols are, can you define or describe what you mean by the word?
Comment #267 July 4th, 2022 at 9:45 am
Btw, Scott is brought up by Demis in the above interview.
Comment #268 July 4th, 2022 at 10:06 am
Lorraine #250
I would say that symbols fundamentally come from the fact that organisms are in continuous 3D space and time, and portions of that space/time are either empty or “occupied” (matter). And primitive brains have evolved to capture the statistical patterns associated with that position in space and time of what’s “occupied” (very primitive nervous systems are able to locate the sun, food sources, other primitive organisms, either friendly or dangerous,… and move their body according to all this, in order to maximize chances of survival).
There’s something very fundamental here: the idea of locality and conservation (of mass) are what makes it possible to identify things (give them a label, i.e. a symbol) and track them continuously. Identity is why we have symbols, which is possible because space is occupied by things that move in space and time in a continuous and conservative manner.
So that’s where all “symbols” come from. Brains need to track entities that matter to their survival both in space and time, and to do that some “labeling” needs to happen. That’s why we give names to people.
Humans were able to not just limit themselves to just label and track things in fundamental space time but also in abstract spaces (mathematics, ideas, space of emotions, etc).
I think that this very fundamental (and successful) tendency to reduce everything into a “point particle” with a label (like the joke about cows being approximated by perfect spheres) is what also limits us. Which is why we struggle with concepts that don’t fit well into that model, like the wave/particle duality, the nature of measurement (we only observe particles), quantum entanglement, what it would mean for a thing to have an “identity” across multiple branches of the multiverse, consciousness (which is about a subjective identity that we can’t pin onto any particular blob of occupied space), etc.
Comment #269 July 4th, 2022 at 10:16 am
Clint #254:
I never said symbols were not real. Symbols are real; a squiggle representing the letter “S” on a piece of paper is real; the issue is that the laws of nature can’t know about symbols. Physicists use symbols to represent e.g. the laws of nature and real-world numbers: the reasonable assumption, given the successes of physics, is that something that physics represents as equations and number symbols actually exists.
But equations (representing laws of nature) and number symbols are not the only symbols relevant to describing the world. Computing has shown that, if you want to describe a system, and by analogy, if you want to describe the world as a system, then you need to also use the following types of symbols to describe the world: IF, AND, OR, IS TRUE and THEN.
Comment #270 July 4th, 2022 at 10:23 am
And obviously abstract virtual worlds created by humans, such as board games, capture those fundamental facts about space/time being either empty of occupied, and conservation. Something like chess is about this, with a spatial/time grid that’s not continuous (grid of discrete cells, and time evolves one tick at a time), with rules dictating how occupied grid cells are ċ͚̫̰̖̇̉͝ͅo̦̗͙͍̘̓ͬͤ͒ͩ̏̀nserved. Because of those conservation rules things can be assigned unique labels (identity) and symbols represent different pieces with unique characteristics or not. All the pawns are the same, but they are also different based on the fact that a cell can only be occupied by one pawn at a given point in time… just like in fundamental physics how two leptons can’t occupy the same space at the same time.
Comment #271 July 4th, 2022 at 10:27 am
Lorraine
“the issue is that the laws of nature can’t know about symbols.”
Well, what we can say for sure is that the laws of nature are such that they eventually make things that can manipulate symbols appear. Our brain and all our creations evolved from the fundamental laws of physics, therefore symbols are “covered” by the fundamental laws of physics, as an emergent property.
Comment #272 July 4th, 2022 at 12:01 pm
Benzon #253 knows that the Hogan twins conjoined at birth claimed to share thoughts, and yet still could tell each other apart. Benzon assumes that this is automatic (“of course”). Benzon also assumes that persons similarly conjoined after birth could never be able to tell whose thoughts are whose. This seems like a very large assumption made with no evidence.
I looked at the paper of Benzon’s that he cited. It does not address the case of the conjoined twins, and I could not find in his paper any reason to suppose that the conjoining with separate “I” in the twins necessarily happens in a different way from conjoining after birth.
I also searched online for scientific papers about the Hogan twins, and found only philosophical papers; even the twins’ neurosurgeon, Dr. Doug Cochrane, has not published anything that I could find. It is interesting to compare the Hogan twins to spit brain patients, who have at times been hypothesized to have two “I”s; current scientific thinking is that this question has not been settled (https://link.springer.com/article/10.1007/s11065-020-09439-3).
In short, the only empirical evidence we have on this issue is that two persons can be conscious in craniopagus with a neural bridge (anecdotal evidence), that self-conscious telepathy can occur between these two persons, and it cannot be ruled out that this also occurs in some split brain patients (clinical but weak evidence). In my view this evidence is not strong enough to falsify or confirm any scientific theory about consciousness, but it is good reason to investigate these cases more thoroughly.
Assuming the Hogan twins truly are telepathic, then telepathy is possible between AI and human even if that would require integrating signaling to and from the AI beginning with the embryo.
It also is highly suggestive that Kim Peek, who lacked a corpus callosum and had other brain abnormalities, had vastly increased intellectual capacities in some areas such as memory (and lesser function in other areas). His brain was wired very differently from ours, yet he was still conscious.
All of the above suggests that a neurally conjoined human/AI might have increased intellectual capacities in some areas, most likely memory and sensation but also perhaps mental calculation, and might also exhibit telepathy between the human and the AI – or demonstrate that the AI has no thoughts to read.
Comment #273 July 4th, 2022 at 2:41 pm
Mikael Gogins #266,
About the review you link on split-brain persons, I don’t get the logic. Can you explain what’s the best point this paper is making?
One of the most solid line of evidences for two Is (at least!) is that we can have the right hemisphere do something for one reason (for example RH draws a snow showel that fits with the snowy landscape projected on the left, controlateral, hemifield) and the left hemisphere explain the action for an unrelated reason (for example LH explain the showel is to clean the barn -actually a rooster was projected on the right hemifield). But they don’t mention this kind of confabulations. Also, many SBP describe their left hand as alien, evil, acting on its own. Again, not even mentioned. Strange.
https://m.youtube.com/watch?v=1BfKS2dyuEg
Comment #274 July 4th, 2022 at 3:42 pm
Lorraine Ford #228:
Hold on a moment. Memory is generally thought to be encoded (sorry) in neural connections and strengths of synapses, but it’s not entirely clear that this is all that’s going on; there is evidence that some memories are stored within individual neurons. The mechanism of this is unknown, but some people have speculated that it could be stored in RNA molecules, which would of course encode it (no apologies this time!) in a sequence of bases.
Obviously, this is not a generally accepted hypothesis, and personally it seems kind of far-fetched. But the question I have for you is: Are you claiming you can rule out this hypothesis, based purely on introspection, without performing any experiments??
(In case it’s not clear, I don’t think the distinction you’re attempting to draw makes sense, but I’m going to avoid arguing that directly for now in favor of making this point instead. 😛 )
Comment #275 July 4th, 2022 at 8:37 pm
Michael Coggins, #270: “Assuming the Hogan twins truly are telepathic,…”
Why would we need to assume they are communicating telepathically when there is a physical connection between their two brains that has been in place since (before) birth?
Comment #276 July 4th, 2022 at 9:58 pm
If there are two “I”s embodied in one nervous system, then if one can read the thoughts of the other, that’s telepathy. It’s not the same as being conscious of one’s own thoughts. The nature of the channel is secondary.
Comment #277 July 5th, 2022 at 2:42 am
What is the point of connecting an AI to the brain, anyway? We’ll get person’s reports on how their brain interprets internal AI activity, which was converted in some way (you have to select something from all the teraflops for the brain to be able to keep up).
Is it to make sure that the AI doesn’t deceive us when it says “I’m aware that I see warmish darkultrapink surface, which is rough in ultrasound”? (The person reports “It focuses on its visual experiences, maybe?”)
Why do we need this AI->brain connection, when we can directly inspect information flows within the AI to confirm or deny that it does focus its attention on its own visual processing?
Comment #278 July 5th, 2022 at 2:44 am
Sniffnoy #272:
If you are going to do an experiment, you presumably need to have a bit of an idea of what you are looking for. If you are looking for symbols, then you first need to describe the characteristics of symbols, so you would be able to recognise them if you saw them. So, describe the characteristics of symbols.
Comment #279 July 5th, 2022 at 8:50 am
John K Clark #255
The evolutionary performance of Covid in a complex environment has been spectacular to witness. Very small genome with 30,000 base pairs and on the order of only ten protein coding genes. The environment is very complex and includes both potentially destructive transmission variables and host variables. At the apex, say on the order of quadrillions of copies made per day (each a potential source of mutation) and currently about genetic 2,000 variants identified. Copy fidelity high enough to allow sufficient genome integrity to continue copying but low enough to allow sufficient mutation testing against varying conditions.
In terms of concurrent copy rate an extraordinarily successful bit of RNA at interacting with a complex hostile environment.
Comment #280 July 5th, 2022 at 8:59 am
Hi red75prime #275:
Not if the machine states are in a quantum computer. (see my posts above)
I personally feel that classical AGI is (and will only become more) powerful – and thus inherently dangerous and almost certain to be used for war like every powerful tool invented by humans. I’m uncertain if quantum AGI would be even more powerful/dangerous than classical AGI because I’m uncertain if quantum computers will either reach the scale necessary or will out-compete classical computers except for some specialized problem classes. However those special problem classes are attention getting since they would give the QAI access to efficient true randomness and efficient period/phase finding which have both been important in the evolution of life and intelligence.
Whatever AI control or surveillance Scott proposes in a year to save the world will need to take this possibility of QAGI into account 😉
Comment #281 July 5th, 2022 at 9:17 am
Michael Grogins #270
I fully agree and am unable to understand why nothing has been published in any journal I can find concerning these twins. If the popular press claims are true then incomprehensible nothing has been published in any scientific journal. As an example the claim the sister perceives what her sister sees could be tested quickly and easily at no discomfort to the girls.
My daughters are fraternal twins and they would suddenly bolt running 180 degrees from each other in public places. It was so well coordinated I sometimes suspected telepathic agreement. 🙂
Comment #282 July 5th, 2022 at 9:43 am
Scott, in comment #240 you said:
It caused me to update from “the space of possible minds is huge” to “the space of possible minds is even larger than I thought it was.” My update is different from yours, but doesn’t necessarily contradict it. More like orthogonal to it.
I’m a bit more interested in understanding the brain than I am in scaling Mount AGI. Here’s how I’ve been thinking about understanding the brain. Imagine that understanding means takes the form of a string of serial-wired Christmas tree lights, 10,000 of them. To consider the problem solved all the lights have to be good so that the string lights up.
Instead of understanding the brain, apply the analogy to understanding how to create AGI (whatever that is). Let’s start at 1956, the year of the Dartmouth conference. It’s at that point we were handed the string and were told, “get this to light up and you’ve solved AI.” Since digital computing had been a going concern for over a decade at that point and work had already been done on chess and on natural language, some of the bulbs in that 10,000 bulb string were good. But we did’t know how many or where they were. Between 1956 and whenever OpenAI started working on GPT-3 we’d replaced, say, 2037 bad bulbs with good ones. Let’s say that in creating GPT-3 OpenAI replaced 100 bulbs, which we know about. So 2137 bulbs have been replaced. How many more bulbs to go before all of them are good?
Obviously we don’t know. Some people seem to think it’s only a couple of hundred or so, most of them having to do with scaling up even further. What, beyond wishful thinking, justifies both the belief that the unknowns cluster in one area and that their number is so low? Maybe we still have over 5000 or 6000 to go, maybe more. Who knows?
By way of calibration, I should note that back in the era of symbolic computing I had once felt – though never published – that we were within 20 years of being able to build a system that read Shakespeare plays in an “interesting” way. By “interesting” I meant that we could have the system read, say, The Winter’s Tale, and then we’d open it up, trace what it did, and thereby learn what happens we humans read that play. That is to say I believed we could construct a system that could reasonably be construed as a simulation of the human mind. Alas, the AI Winter of the mid-1980s killed that dream. While these new post GPT-3 systems are wonderful, I see little prospect that any of them can be considered a simulation of the human mind nor that any of them will be able to shed insight into Shakespeare in the near or mid-term future. Beyond, say, 2140 (the year of Kim Stanley Robinson’s New York 2140) I’m not prepared to say.
My sense of such matters is that reading about such collapses of intellectual projects in a history book is not the same as living through one. The valence is much weaker. So I’m sticking with my new prior, “the space of possible minds is even larger than I thought it was.”
Comment #283 July 5th, 2022 at 10:15 am
redprime #275, the point of conjoining human and AI is to get the human’s intelligence up to the level of the AI and to thus place humanity on an even playing field with the AI regarding natural selection.
Whether the Hogan twins really shared thoughts is not critical. Experiments with neural implants for vision and motor control show that brain-computer direct interface is entirely possible.
I see NO realistic path to “AI Safety” that does not get humanity to the same level of intelligence as AI. It would kill humanity spiritually to be ruled by smarter AIs for our own good. I wouldn’t call that “safety.”
Without equal capabilities, AIs will simply out-compete us in natural selection. In principle that could be prevented by keeping AIs from reproducing themselves either with our help (they parasitize us like a virus) or without our help (they run factories to copy themselves like a von Neumann machine), but I don’t see how we can stop that in the long run.
The question whether AI is conscious or has values or even works to preserve humanity is completely secondary. Natural selection is the ruling principle.
Comment #284 July 5th, 2022 at 10:24 am
Ilio #271, the purpose of the review was to survey the current status of various questions regarding split brain patients. The question of split consciousness is the most important of these questions. The review concluded that there is not enough evidence to decide this question.
Comment #285 July 5th, 2022 at 10:34 am
Scott Aaronson, what do you think is the relevance, if any, of Solomonoff’s theory of inductive inference to this question of AI safety, or indeed AI in general?
Comment #286 July 5th, 2022 at 11:58 am
Michael Gogins #283: As it happens, I was just reading Shane Legg’s PhD thesis, which is all about Solomonoff induction and its generalization to reinforcement learning by Marcus Hutter (called “AIXI”), and the potential relevance of all this to AGI!
I think that, at a conceptual level, the relevance is extremely high—analogous to the relevance of Bayesian inference or decision theory themselves. These are all examples of theoretical frameworks that show how your problem would be solved in some unrealistic limiting scenario, leaving “only” various practical issues (e.g., for Bayesian inference, you don’t have a principled prior; for Solomonoff induction, you do have a principled prior but it’s uncomputable and also depends on the reference Turing machine; for decision theory, you don’t have a well-specified utility function and couldn’t efficiently maximize it if you could). My feeling is that, to make progress on AI safety or AI more generally, one should probably be conversant with all these frameworks, freely adopting, discarding, or modifying them as the occasion demands.
Comment #287 July 5th, 2022 at 1:40 pm
Scott, #284 and AI safety more generally: While I do think that it is still too early to lose sleep over concerns that rogue AIs will be a danger to humanity, I believe that, broadly construed, AI safety concerns a number of issues. One of them is explainability. I think that is worthwhile in its own terms, but also because it requires us to investigate both what AI models do while inferencing and what humans do when they provide explanations. I think those things are well worth knowing, both on their own terms and in terms of their contribution to AI safety.
Comment #288 July 5th, 2022 at 4:36 pm
Lorraine Ford #276:
I’m not asking about symbols, I’m asking about memories stored in RNA base sequences. Yes, I’m asking about that because I would consider that to be a symbolic representation, but what I am asking about is where you are drawing the line (since I think the distinction you are drawing cannot be made in the way you are making it). Do you think you can rule out this possibility? If not, does this mean you consider a sequence of bases in a nucleic acid molecule to not be an encoded symbolic representation?
Comment #289 July 5th, 2022 at 8:14 pm
Clint #254:
Re symbols etc., I think what you might be getting at is this:
1. There is a real physical world that exists, and to break it right down into its component parts, physicists symbolically represent these really-existing aspects of the world as equations (representing relationships, i.e. the laws of nature), variables (representing categories like mass or charge), and number symbols (representing numbers).
2. However, staying with the component parts, the “knowledge” aspect of the world, the aspect of the world that can be symbolically represented as: “variable1 = number1 AND variable2 = number2 IS TRUE”, can’t be derived from 1. above. Similarly, the knowledge-analytical aspect of the world, the aspect of the world that can be represented using symbols like IF, AND, OR, IS TRUE, and THEN, can’t be derived from 1. above.
3. Although 1. and 2. above both share symbols representing really-existing categories and numbers, the other symbols are different: equations represent really-existing mathematical relationships; but IF, AND, OR, IS TRUE, and THEN statements represent a really-existing knowledge-analytical aspect of the world that can’t be derived from the mathematical relationships, i.e. can’t be derived from the laws of nature.
4. We are so immersed in everyday symbol usage, e.g. when speaking and listening, writing and reading, that we find it hard to disentangle the symbols from the really-existing bits. The really-existing bits, that the world is constructed out of, are: relationships (laws of nature), categories, numbers, plus a knowledge-analytical aspect of the world. But the symbols that represent categories and numbers are not the real categories and numbers; the symbols that represent relationships are not the real relationships, and the symbols that represent the knowledge-analytical aspect of the world are not the real knowledge-analytical aspect of the world.
Comment #290 July 5th, 2022 at 8:39 pm
Sniffnoy #286:
First, one has to define or describe what a symbol is, before one can decide whether or not a nucleic acid molecule is an encoded symbolic representation. You have already decided that something is in fact a symbol. So what is the basis for this decision: how would you define or describe a symbol?
Comment #291 July 5th, 2022 at 9:12 pm
Lorraine Ford #288:
It would be nice if you would answer the question before responding with your own. Are you confirming that you would not consider these symbols?
Comment #292 July 6th, 2022 at 3:56 am
Scott#284
I read through Shane Legg’s dissertation and enjoyed it greatly so thank you for posting. I don’t understand why AI people don’t more fully present human intelligence in its evolutionary context but he does far more than most.
His abstractions of the environment and an intelligent agent to allow modeling are well thought out. I don’t agree that reinforcement learning is an exact and complete representation of human learning but that leads to discussions that won’t be resolved without further knowledge as you have noted.
AlphaGo is a jaw dropping development so no one can argue against his successes. His optimism and conviction did lead to astounding development.. No matter belief about the possibility of true artificial humans, the next couple decades does have promise to be the most fertile for tech developments truly that impact human life since I guess the development of nuclear weapons and then launch of the internet. In this case the likely impact will be the replacement of most human labor by machines at a time when the glut of human labor is ever increasing.
A case can be made that Man’s technical prowess is always far ahead of Man’s wisdom and this likely to be another demonstration. As JimV noted and Legg alludes to, the danger for mankind is likely more from those controlling AI’s rather than from the AI’s themselves. A specialized wisdom AI might fill the gap.
Comment #293 July 6th, 2022 at 4:55 am
#241 “The problem that I have is that my own consciousness feels extremely NOT “computationally powerful. In fact it feels very much the opposite of that…”
Clint, I hear you. Intuitively, yes, conscious human thinking is painfully slow. But by phenomenally-bound consciousness, I wasn’t referring to your serial virtual machine of logico-linguistic thinking. Rather, I meant the vast, robustly classical-seeming world-simulation which your mind running that masquerades as the external world – a classical world-simulation that naïve realists assume is the directly perceived local environment, an approach that offers all the computational advantages of theft over honest toil. Contrast today’s petaflop digital zombies. Classical Turing machines are just tools and toys, not nascent AGIs. The ignorance of classical Turing machines and classical connectionist systems is architecturally hardwired. Digital zombies can’t understand what’s going on because they can’t bind – and would malfunction if they did. And if you’re _not_ convinced that the phenomenal binding of organic minds is computationally uber-powerful, imagine if you had integrative agnosia on the African savannah. You could experience a mane, teeth and jaws – but no hungry lion. Now combine integrative agnosia with, say, akinetopsia (“motion blindness”) and florid schizophrenia (disintegration of the self) and you’d soon be lunch.
OK, so how does an aggregate of clunky classical neurons communicating across slow chemical and electrical synapses do what’s classically impossible, i.e. run a phenomenally-bound real-time world-simulation (“perception”)? Well, as far as I can tell, they don’t! Probe inside your skull at a temporal resolution of femtoseconds rather than milliseconds and investigators _wouldn’t_ find discrete, decohered neurons – an artifact of our temporally coarse-grained tools of investigation. For a skull-bound pack of decohered neurons in classical space-time couldn’t create your mind, i.e. a world-simulation populated by macroscopic objects experienced by a unified self. The phenomenal binding of consciousness into virtual worlds is classically impossible for a bunch of decohered neurons on pain of magic.
In my view, you’re a state-of-the-art quantum supercomputer – but not a universal quantum computer! – simulating a classical macroscopic world.
But we won’t discover the truth unless we experiment rather than philosophise.
Back to the lab…
Comment #294 July 6th, 2022 at 8:50 am
Please forgive the long post.
Does a simple explanation like a Proof From The Book have to be short?
Scott#119:
David #291:
Inspired by the Feynman quote “If you cannot explain something in simple terms, you don’t understand it” mentioned in the Hassabis interview above, I decided to try to come up with a “simpler” explanation for my objection to the statement from #119 above. In the format of the WIRED magazine “Five Levels of Difficulty”, this would be aimed at the “high school” level of explanation. And I’m not saying this thread amounts to a high school class 🙂 I’m just testing if Feynman’s criteria can be used to give evidence that an AGI (my own brain) … “understands something” …
Explain how the brain is not the same as a classical computer, in simple terms, as if to a high school class.
First, imagine that there is a place called a TRANSISTOR_ROOM (TR) and we are going to put you in there and require you to do a job. Here is how the job works. There is an LED light on one wall. On the opposite wall is a large button. When you press the button, it will stay pressed IN. When you press it again, it will pop OUT. Your job is this: When the light on the first wall turns ON then you press the big button on the opposite wall IN. When the light goes off you press the big button so that it pops OUT.
And that’s how what is called a transistor works. A computer chip has billions of tiny TRs in it, each with a very fast worker like you who just spends all day looking at the light and pushing the button IN or OUT. The TRs are put together on the chip in what are basically parallel or series combinations to form what are called logical gates and those gates can be combined to do logic or math to run computer programs. (By the way, there is an application of the TR where you have someone whispering very softly or just barely speaking into a hole in one wall and then you go to a hole in the other wall and either speak loudly or yell, respective to how loudly they are whispering/talking. That is called an AMPLIFIER_ROOM but it is only useful for things like making loud music and is not very good for computing.)
Now, it turns out that you also have device rooms in your brain … but the rooms in your brain are nothing at all like the logical devices in a classical computer. Imagine there is something called a NEURON_ROOM (NR) and we are going to put you in there and require you to do a job. Here is how the job works. We will sit you in the middle of the room on what we call the SOMATIC_STOOL. On one wall there are thousands of small circular windows made of flexible membranes. You notice that they can expand in towards you as if someone were poking them from the other side or be pulled back away from you as if someone was pulling on them. The amount that they are pushed in or pulled back can vary like if you were pushing or pulling on a rubber balloon. We will call the amount of this push or pull the MAGNITUDE of the input. Since it can go in or out we will say that it can be POSITIVE OR NEGATIVE. You notice that the membranes going in or out are not all happening at the same time although there does seem to be some regularity or pattern to many of them. If we were to record these differences in WHEN they go in or out over some period of time we would be recording what is called the PHASE of the membranes. So, each membrane has a MAGNITUDE and a PHASE to it. Pay attention because you must keep track of all of these things 🙂 Mathematicians have a name for these kinds of things with magnitude and phase, they are called COMPLEX NUMBERS or just AMPLITUDES. So, THE INPUTS you are seeing, from your perspective sitting on the soma stool, are COMPLEX NUMBERS. This is important because this is how ALL information is encoded for you – it is the ONLY way you can get any kind of signals about what is going on in the world outside of the NR we’ve put you in. Now you see what look like branches, like a tree, drawn across the floor from your location in the middle of the room, and up the wall, from all of the membranes and all converging on you. The first part of your job is to combine the amplitudes as they appear on the far branches and then carry those on up the tree until one final amplitude reaches you. (There is something called the cable equation for this but not really important right now.) You notice that sometimes when you combine amplitudes they reinforce or add to each other but sometimes they CANCEL each other. This is a VERY important difference between the TR and the NR … it is called INTERFERENCE between inputs. The inputs to the NR can interfere with each other. There is nothing like this for the input to the TR. You also notice that very interesting things can happen based on how these amplitudes interfere and how they combine based on where they are placed and the arrangement of the branches. Sometimes they work like simple logic gates but other times they can be like more complex logic gates (things like XOR) or math functions like multiplication or phase shifts. There is a great deal of complexity here … but the FUNDAMENTAL and crucial difference is the interference of amplitudes. Once we are performing functions by interfering amplitudes then the “computing” is nothing at all like it was in the TRANSISTOR_ROOM. Now, your seat in the middle of the room has a number written on it called the THRESHOLD. After combining all of the amplitudes in whatever ways they combine if the result reaches THRESHOLD then you must perform the second part of your job. The second part of your job is to then turn around and face the opposite wall … which is also covered with thousands of membranes! The difference this time is that YOU will be the one pushing or pulling and setting the phase on each of those membranes because those are the OUTPUTS that will appear to some other NR (maybe different NRs) as input amplitudes! You also notice that just like on the input branches the output branches are also sometimes performing functions on the “action potential” (looks like a spike) that you started down the branch. Now, you might think that because the THRESHOLD on your seat is essentially a binary yes or no decision, that basically decides whether to set all of those output amplitudes, that maybe the NR really is after all like the TR?? But here is the thing, that simple binary decision that you make in the middle of the NR is NEVER SEEN OR STORED anywhere else outside of the NR! If we join TRs (transistor rooms) together then that internal decision you made to press the button is exactly equivalent to the state of the input light and the button. However, the input to another NR (neuron room) is always in the form of amplitudes. The binary part of what you are doing shows up as part of the information …only in the fact that an amplitude appeared … but there is MORE information in the amplitude you are producing as output than just that you turned it on! Whoever is inside that other NR will only ever see an amplitude appearing as input. Yes, they do get the binary part of the information in that they know you turned on the amplitude … but they ALSO get a magnitude and a phase! That information you are transmitting is not represented or stored anywhere as only a simple on/off state (what we call a bit) but is always (input or) stored at the membrane input window of another NR (called a synapse) in the form of a complex number (an amplitude). Now, that all being said, we should recognize that the bit is the absolute simplest possible amplitude input we can imagine – always just either present with 100% of its possible magnitude … or not there (but NEVER NEGATIVE!) and never with any phase – where a zero phase means that whenever we look it is always already either in one of its two possible states – all there or not. Certainly then, your NEURON_ROOM could perform such a simple “binary” function … But it appears to be built to deliver much more information! Neuroscientists believe that it can multiply complex numbers (or collections called matrices), perform phase shifts, or various logical functions. This means that whereas we would need many TRs to put together to do complex things in a classical computer we could do complex things with a much smaller number of NRs – even just within ONE NR! Another way to say this is that we believe the class of computation found in the NR is more powerful than the class of computation we find in the TR (at least for some things). Maybe you’re wondering if this NR device is like using a TR as an amplifier because it has these inputs that can vary continuously (although in an amplifier they can’t be negative!). That’s a good question. It turns out that the NR model can implement what we call discrete computing like the TR model because we can choose to associate amplitudes that are distinguishable from each other – it’s called being ORTHOGONAL – meaning you can tell them apart so that one is not the other one. In the brain, it is said that orthogonal amplitudes originate over different “receptive fields”. When we join together amplitudes in this way we have what is called a qubit. If we join them together in such a way that amplitude for one interferes (cancels) amplitude for the other in our input windows and so that we never have more than 100% of evidence for either of them, then we can say that together they can represent a superposition of the possibilities. The NR rooms are also what we call LINEAR rooms (well operators) and what that means is that the brain can maintain these rules for orthogonal amplitudes. Further, neuroscientists tell us that NRs “always behave in a perfectly linear manner … as if Nature were trying to come up with a mechanism that always behaves linearly.” That means that this discrete nature can be maintained without getting in the kinds of trouble seen with analog computation. An interesting thing is that the classical bit is always kind of hiding right there in the middle of the NR in your decision to make outputs (it shows up in the actual neuron as a “spike” we can see going towards all of those output windows) and that, which is the essence of the TR, is actually what we call a NON-LINEARITY and it also plays a critical role in computing. Therefore, we can observe that if all but one of the orthogonal amplitudes get canceled then the output looks exactly like a classical bit for that thing the amplitude represents (either on or off) and we say (again if this is for a predictive machine) that the only (or all the) evidence is only for one possibility (which we call the outcome or measurement). There was a mathematician named Gleason who said that if we were trying to make a prediction machine that could learn about and predict the world (a probabilistic model) using amplitudes like this then a certain thing called a norm (actually called a 2-norm but it’s just the Pythagorean norm) would be the way for us to do that. And, again, neuroscientists find some evidence that normalization is, well, normal in the brain. So, to sum it all up, the NR is a device that incorporates a TR for the “decision”, memory is stored in the form of amplitudes not bits, and computations are performed based on the interference of amplitudes.
Just remember … if you are working in a NEURON_ROOM and someone tries to convince you that you are working in a TRANSISTOR_ROOM just tell them:
“Hey! Can’t you see I’m interfering amplitudes in here!”
…
While Feynman’s idea of the simple explanation and also Erdos’ idea of the Book is appealing … both have problems. Even a “simplistic” explanation may not be short just because there is a lot that requires explaining. Second, just because someone gives a “simple sounding” explanation doesn’t mean that they are right! There could be evidence not included that disproves their simple argument – and usually there is 😉
David, best I can tell we have most of the evidence we need for non-classical computing taking place in the dendritic operators of the neuron. You don’t need to know more than that computing based upon the interference of (positive and negative) complex numbers is taking place. Once you’ve established that you have a computer based on interference of amplitudes then you’re on your way to a quick chess mate because the other postulates quickly fall into place given the brain’s known computational abilities (orthogonality, inner products, normalization, linear matrix products and projections). If you are working in the lab … the weakest evidence I see at the moment is for the brain representing probabilities using the 2-norm specifically. Neuroscientists see evidence for normalization but aren’t sure about the model. So that would be what I would recommend working on. Everything else already has plenty of evidence – once you understand the postulates for quantum computing.
It appears that the neuroscience consensus that the brain is classical focuses on the non-linearity of the spike transmission. But the brain nowhere stores binary bits. Information is stored (and used) in the form of amplitudes. The spike non-linearity does the (necessary) computational work of making a decision and is embedded in the larger non-classical model of the devices. Scott said it best, “Physicists have sold everyone a bill of goods on this. . .At least at the mathematical level, quantum mechanics is very, very simple. Let me let you in on the secret: Basically, (Quantum Theory) is just a generalization, or extension of the laws of probability to allow minus signs.” And if one actually reads the postulates for quantum computing it says nothing about requiring atomic level devices. The postulates start with the interference of amplitudes … which is what the neuron starts with … Again, best I can tell, neuroscientists have “bought the bill of goods” that quantum computation must be about atoms or photons or electrons. Yes, it is possible, obviously, to construct quantum gates using those devices … but the postulates make no such restriction. I’m a computer engineer and I arrived at this conclusion just by a straightforward hardware device analysis and by taking the postulates seriously.
As for my own brain … I suspect that our brains while quite amazing may be something like the Digi-Comp II of quantum computer models – meaning that we can “just get over the line” but may be rather weak because we haven’t evolved specialized functional units (like for modular exponentiation to factor large integers) or just because of mundane hardware or transmission limits. For most (probably eventually all) things classical computers are better than us when optimized for particular purposes.
Quantum computing is not a superpower or “weird” or “mystical” nor does it “solve the consciousness problem” or whatever – its just another way to make a predictive machine. An interesting question may be why did evolution choose it if indeed it is what is going on in the dendritic operators? Maybe for its efficient generation of randomness for early life avoiding predation? Or for period and phase finding problems at the heart of searching for food? Or maybe because early life had to take signals from light or chemistry that needed to account for interference effects?
Comment #295 July 6th, 2022 at 9:08 am
Sniffnoy #289:
I think it is going too far to talk about “memories stored in RNA base sequences”, and whether that is “a symbolic representation”, because this brings up a whole lot of questions like: what are memories? what is consciousness? as well as: what are symbolic representations? Better to just start by asking: what are symbols/ symbolic representations?
We use written and spoken symbols all day, every day, when speaking, listening, writing and reading, so we have to use symbols in order to describe symbols. But what is clear is that symbols are arrangements of physical matter that have no inherent meaning. E.g. pencilled letter and word symbols on a piece of paper have no inherent meaning. We need to go to school to learn reading and writing to become aware of the meaning that people have assigned to the letter and word symbols.
Symbols have no inherent meaning. So, to say that RNA base sequences are symbols is to say that the meaning has to be learnt. (This is leaving aside the question of exactly how one would define “meaning”. ) The other option is that matter itself has inherent meaning via its logical and mathematical relationships.
Comment #296 July 6th, 2022 at 2:54 pm
Lorraine,
Better to just start by asking: what is physical matter?
Please give a precise definition (made of symbols, like all definitions).
Comment #297 July 6th, 2022 at 9:20 pm
fred #294:
Why don’t you yourself try to define what physical matter is? But when it comes to the initial component parts of the world, if indeed physical matter is an initial component part of the world, there may be no explanation. I wouldn’t know. However, matter might be representable as something like: “Mass = number1 AND Charge = number2 …”, where, in turn, things like mass and charge, and even numbers, might be representable as mathematical relationships, and where things like “AND” are clearly there right from the start. But don’t take any notice of that nonsense: I think you yourself should try to define what physical matter is.
Comment #298 July 7th, 2022 at 8:29 am
Lorraine
“Why don’t you yourself try to define what physical matter is?”
Thank you. I suggest you stop wasting everyone’s time with your constant juvenile trolling and baiting.
In case anyone missed the sexist pattern in the way you always enter the discussions:
“And if one takes numbers of comments into account, this mass male delusion that computers could ever be conscious.”
“All this male-dominated morality-talk is irrelevant.”
Comment #299 July 7th, 2022 at 8:50 am
That’s it, I’m closing this thread. Thanks everyone!
Comment #300 July 8th, 2022 at 8:07 am
[…] Steven Pinker’s guest post from last week, there’s one bit to which I never replied. Steve […]
Comment #301 July 21st, 2022 at 8:57 pm
[…] month, you’ll recall, Steve and I debated the implications of scaling AI models such as GPT-3 and DALL-E. A main crux of disagreement turned out to be whether there’s any […]
Comment #302 August 2nd, 2022 at 8:06 am
[…] Shtetl-Optimized) and Steven Pinker (a prominent cognitive psychologist) recently exchanged a series of essays in which they shared their thinking on what we can learn from GPT-3 and how we might progress […]