My podcast with Dan Faggella
Dan Faggella recorded an unusual podcast with me that’s now online. He introduces me as a “quantum physicist,” which is something that I never call myself (I’m a theoretical computer scientist) but have sort of given up on not being called by others. But the ensuing 85-minute conversation has virtually nothing to do with physics, or anything technical at all.
Instead, Dan pretty much exclusively wants to talk about moral philosophy: my views about what kind of AI, if any, would be a “worthy successor to humanity,” and how AIs should treat humans and vice versa, and whether there’s any objective morality at all, and (at the very end) what principles ought to guide government regulation of AI.
So, I inveigh against “meat chauvinism,” and expand on the view that locates human specialness (such as it is) in what might be the unclonability, unpredictability, and unrewindability of our minds, and plead for comity among the warring camps of AI safetyists.
The central point of disagreement between me and Dan ended up centering around moral realism: Dan kept wanting to say that a future AGI’s moral values would probably be as incomprehensible to us as are ours to a sea snail, and that we need to make peace with that. I replied that, firstly, things like the Golden Rule strike me as plausible candidates for moral universals, which all thriving civilizations (however primitive or advanced) will agree about in the same way they agree about 5 being a prime number. And secondly, that if that isn’t true—if the morality of our AI or cyborg descendants really will be utterly alien to us—then I find it hard to have any preferences at all about the future they’ll inhabit, and just want to enjoy life while I can! That which (by assumption) I can’t understand, I’m not going to issue moral judgments about either.
Anyway, rewatching the episode, I was unpleasantly surprised by my many verbal infelicities, my constant rocking side-to-side in my chair, my sometimes talking over Dan in my enthusiasm, etc. etc., but also pleasantly surprised by the content of what I said, all of which I still stand by despite the terrifying moral minefields into which Dan invited me. I strongly recommend watching at 2x speed, which will minimize the infelicities and make me sound smarter. Thanks so much to Dan for making this happen, and let me know what you think!
Added: See here for other podcasts in the same series and on the same set of questions, including with Nick Bostrom, Ben Goertzel, Dan Hendrycks, Anders Sandberg, and Richard Sutton.
Follow
Comment #1 September 15th, 2024 at 2:03 pm
[…] The Destiny of Humanity. You’ll be able to comply with any responses to this entry via the RSS 2.0 […]
Comment #2 September 15th, 2024 at 3:14 pm
Moral philosophy is the evolved human version of the selfishness/altruism axis, which came about because both ends are useful for survival at different times, and it existed before there were brains to think about it. That axis lives on in human emotions, and people fall on it in various places, including different places for different circumstances.
Open up any newspaper and you can see that the notion of a single moral philosophy to rule them all, even something as simple as the Golden Rule, is absent from the world and always has been. Trying to build in a top-down moral philosophy into an AGI is going to lead to spectacularly hilarious failure modes. I regret that I am likely to old to see this come about.
Oh, and apparently Oprah Winfrey is now on your side in the “there oughta be a law” AI safety hysteria, so congratulations, I guess?
Comment #3 September 15th, 2024 at 3:32 pm
> This is an interview with Scott Aaronson, astrophysicist and PhD in Computational Neuroscience and serves as a Researcher at the Mimir Center for Long-Term Futures Research.
Is this some parallel universe Scott Aaronson?
Comment #4 September 15th, 2024 at 3:36 pm
Shmi #3: Yeah, they should fix that! Presumably they transposed me with a different guest.
Comment #5 September 15th, 2024 at 6:51 pm
Description of the interview is all fixed up now.
Per Hyman’s “Moral philosophy is the evolved human version of the selfishness/altruism axis, which came about because both ends are useful for survival at different times, and it existed before there were brains to think about it. That axis lives on in human emotions, and people fall on it in various places, including different places for different circumstances.”
^ I tend to agree entirely. I suspect that in the wide mind-space (especially minds 1000s of times beyond our own), there are many, many, completely incomprehensible ways of dealing with friends, allies, enemies, etc – and that the Golden Rule is most a mammalian thing that probably won’t have strong proxies for minds unbelievably beyond our own.
But I probably hope Scott is right in that there is some kind of “fundamental” moral truth that might ensure our wellbeing with AGI. But I’m not wildly optimistic about that.
Comment #6 September 15th, 2024 at 6:56 pm
On the minor issue of talking felicitously, it seems to me that the only ones who do so are well-rehearsed or speak from written notes. I personally much prefer reading a transcript which has been edited for clarity. (Thank you for the ability to edit comments on this site after they have been submitted, which I have resorted to many times.)
On morality, I expect that developed by an AI will as usual depend of its training material, given that it has been programmed with the purpose of selecting the responses and actions which are consistent to the highest degree with its training material. I would not want that training material to include the entire Christian Bible, but would not object to Thomas Jefferson’s version of the New Testament.
Comment #7 September 15th, 2024 at 7:46 pm
Hyman Rosen #2:
Trying to build in a top-down moral philosophy into an AGI is going to lead to spectacularly hilarious failure modes. I regret that I am likely to old to see this come about.
Is there a bottom-up approach to building morality into AGI that you advocate instead? Or are you just Teddy Roosevelt’s “man who points out how the strong man stumbles, or where the doer of deeds could have done them better”? 🙂
Oh, and apparently Oprah Winfrey is now on your side in the “there oughta be a law” AI safety hysteria, so congratulations, I guess?
OK, Oprah has joined the side of Hinton, Bengio, and Stuart Russell? Good for her. Not the first time I’ve agreed with her.
Look, anyone who’s thought through the issues and comes out against SB1047 should argue against the bill to their hearts’ content. But as soon as they claim it’s completely obvious that AI safety is a pure hysteria and categorical non-concern—and that dozens of the world’s leading AI scientists holding the opposite opinion doesn’t trouble them in the slightest—then they reveal themselves as intellectually incurious and not worth taking seriously.
Comment #8 September 15th, 2024 at 10:14 pm
Daniel Faggella #5: At https://danfaggella.com/trajectory/ and https://danfaggella.com/aaronson1/, you list Anders Sandberg’s name in this episode title instead of Scott’s name.
Comment #9 September 16th, 2024 at 1:16 am
Same here, I’m always polite when I’m using GPT.
Cheers from overseas, you look great!
Comment #10 September 16th, 2024 at 1:38 am
You were great! Thanks for the interview!
Comment #11 September 16th, 2024 at 2:22 am
I admit that your view that “human specialness” is related to unclonability and unpredictability strikes me as odd. When I decide what do, I use my knowledge about the world to find the plan most beneficial to my goals and desires. This is inherently an algorithmic process: the decision is *computed* from my memories and desires. I have *reasons* for doing the things I do. Therefore, I am to a large extent predictable. (This is essentially the view of Bertrand Russel, although as opposed to him I wouldn’t phrase it as “no free will”.) On the other hand, someone who makes decisions in a completely arbitrary manner, e.g. by drawing on some arbitrary quantum information that they inherited from the formation of the universe (as you suggest humans do, IIUC), is not an agent or a mind in any meaningful sense.
Moreover, any perception that I have about other humans that they are “conscious” or “have personhood” or “moral patients” (btw these might be 3 different things) comes entirely from my observation of their behavior, and is completely unrelated to the details of the quantum physics generating that behavior. For example, suppose that we discovered that you are factually wrong, and human brains don’t meaningfully use anything like the sort of quantum computing you need to get unclonability. Would you then conclude that everyone around you are soulless automata? Or would you revise your philosophy? If it’s the latter, doesn’t it mean you posses some knowledge about the nature of “personhood” that’s much deeper than any assumption about unclonability etc?
(I guess a possible objection is: that deep knowledge is that *I* personally have personhood. Okay then, imagine men have something quantum in their brains but women don’t. Would you really conclude women are not people? (Feel free to switch the example to e.g. dark-eyed people vs. light-eyed people, if you’re worried about people who would take things out of context lol))
Comment #12 September 16th, 2024 at 3:35 am
Hi Scott, I’m curious what your beliefs are about what constitutes moral applicability.
It seems to me that you believe that there is a single, universal, unchanging morality, that exists, that is applicable only to humans among known extant organisms, and that was applicable to all humans throughout history.
And it also seems to me that you believe that natural laws govern the universe, which I think means that you disbelieve in the supernatural, and therefore that you disbelieve that humans command some special position in the universe.
That combination of beliefs feels kind of incomplete to me. I think what’s missing is the natural-law justification for why humans are the only form of life on Earth for which morality is applicable. Like, what objectively-observable distinctions there are between humans and non-humans that makes it so that morality is applicable for the one but not the other.
So, I’m curious what beliefs you hold that provide that justification. (I’m also curious about where you would place the cutoff for moral applicability for humans. Anatomical modernity? Behavioral modernity? The Neolithic? The development of writing?)
Comment #13 September 16th, 2024 at 10:07 am
Vanessa Kosoy #11
I don’t agree. Often times the best strategy when interacting with other people pursuant to obtaining some outcome is to be unpredictable in the sense of another anticipating your actions to the detriment of your pursuit. There also is quite often uncertainty as to the best course of action in the present to achieve a desired outcome in the future. I suppose in a completely known mechanistic universe the autonomic predictability you set out would be applicable.
Comment #14 September 16th, 2024 at 11:02 am
Funny people are so eager to split hair about AI morality when we haven’t even “solved” morality at the human level.
Maybe AI has to be seen as a tool to help use get less confused about our own morality, e.g. use it to interpret what animals are trying to tell us about the way we treat them…
Comment #15 September 16th, 2024 at 11:52 am
Scott #7
No, I do not have a bottom-up approach for inculcating morality in an AGI, for the simple reason that no AGI exists. My contention has always been that it is useless to contemplate how we will deal with AGIs before we have them, because we do not know anything about how they will behave or what they will be. I do find it telling that most of the AI safety arguments now in popular discussion have just become arguments about what people should do; “don’t make deepfake porn”, “don’t use generated art”, “don’t present facts” (also known as “AI is racist”).
“AI scientists holding the opposite opinion doesn’t trouble them”
Decades ago I was talking to a friend about atheism, and he said that people smarter than I am believe in gods, and he was going to go with their opinion. I acknowledged that, but nevertheless I was absolutely certain that I was right and they were not. I have never since encountered a reason to change my mind. Given that literal wars have been fought over whether only faith rather than good deeds can get you into Heaven, skepticism is warranted when “smart people” make pronouncements about things that don’t exist.
Daniel Faggella #5
“minds 1000s of times beyond our own”
I do not take it as a given that such a thing will ever exist. I would bet that it will not. I think the sheer cussedness of the universe and the randomness of events and the required processing power and the ability to gather widespread information all will make AGIs the same stumblebums that people are, forever.
Comment #16 September 16th, 2024 at 12:47 pm
fred #14
This reminds me of the old Irish toast-May we be as good as our dogs think we are.
Comment #17 September 16th, 2024 at 12:59 pm
Dr Aaronson
Off topic and you may know about it but construction started on the first US Gen IV nuclear reactor at Oak Ridge. Just a demonstration reactor but at least some progress. China put a Gen IV commercial reactor in service some months ago.
Comment #18 September 16th, 2024 at 2:19 pm
OhMyGoodness #17
and the newest Gen3+ french reactor that’s started being tested, somehow shut itself down
https://www.france24.com/en/live-news/20240905-new-french-nuclear-reactor-enters-automatic-shutdown
Comment #19 September 16th, 2024 at 4:28 pm
I recommend the recent movie AfrAId. It’s a horror film, so naturally things go wrong, but if you ignore that part, it’s actually a very nice illustration of quite a number of things that a benign AGI would be good for – gamifying proper behavior for children, acting as a research companion, creating avatars of dead people, diagnosing hidden medical conditions through observation, and so forth. (It’s rather like Trap, which, if you ignore the whole silly serial killer plot, is actually a very nice movie about a bemused but loving father taking his teenage daughter to a concert of a pop star that she adores.)
Comment #20 September 16th, 2024 at 5:15 pm
Vanessa Kosoy #11: I fretted about many of those exact questions in The Ghost in the Quantum Turing Machine, but briefly:
– Yes, if empirical findings ever undermined my metaphysical speculations I’d change my speculations, without apology. This vulnerability is not a bug but a feature. 🙂
– I deny the premise that there are only two possible drivers of human choices, with nothing in between: (1) rational justifications that are in principle completely determined by the knowable classical facts at some earlier time, and (2) “arbitrary, capricious randomness” that comes from quantum noise or whatever.
Very often there are two or more courses of action that all have rational justifications in their favor. (Think, for example, of ChatGPT deciding between two possible next tokens, both of which were assigned more than 30% probability by the model.) In such cases, after the decision is made, you might be able to argue that it flowed inevitably from the person’s internal dispositions … but you had no clear way of knowing what the relevant internal dispositions were before the decision! You might even say that the act of deciding created dispositions that weren’t previously there. Or at least, that’s how it works in my speculative model.
Comment #21 September 16th, 2024 at 5:26 pm
David #12: You’ve ventiloquized a lot of firm philosophical commitments out of my mouth, while my brain swims in uncertainties! 🙂
I never expressed confidence in moral realism, in this interview or elsewhere — just said the possibility needs to be on the table that at least some of our moral commitments (especially the Golden Rule) have the same sort of universality as 2+2=4, that they’d seem as obvious to extraterrestrials or artificial superintelligences as they do to us.
I also never said that humans are the only known moral agents. I actually think some of morality plausibly extends, at least, to any mammals that care for their young and can support or betray their allies. I did say that I have trouble extending moral agency to sea snails. So, I’ve offered loose upper and lower bounds that don’t match. 🙂
As a calibration exercise, imagine someone expressing incredulity that the complex numbers could possibly be a fundamental part of the architecture of reality, if humans are the only known animals that can even understand what complex numbers are.
Comment #22 September 16th, 2024 at 5:52 pm
Hyman Rosen #15:
Decades ago I was talking to a friend about atheism, and he said that people smarter than I am believe in gods, and he was going to go with their opinion. I acknowledged that, but nevertheless I was absolutely certain that I was right and they were not. I have never since encountered a reason to change my mind. Given that literal wars have been fought over whether only faith rather than good deeds can get you into Heaven, skepticism is warranted when “smart people” make pronouncements about things that don’t exist.
Traditional religious belief is confined to a tiny minority of top scientists, despite how prevalent it is in the wider world (religious practice without belief is much more common among scientists). Concern about AI risk is not similarly confined. While full Yudkowskyism is a minority position, “full speed ahead, zero guardrails needed” is a minority position as well. Most experts are somewhere in the middle, correctly seeing both opportunities and dangers in any civilizational transformation of this magnitude. I have, however, noticed at least some correlation between how worried various experts are, and how accurately they predicted where AI would be today.
Comment #23 September 16th, 2024 at 10:19 pm
Question: Is there anyone here who didn’t listen to the podcast, but who would read a transcript if I or someone else had GPT make one?
Comment #24 September 17th, 2024 at 12:01 am
Scott #22
Why are we confining ourselves to scientists? I would say that the AI risk/doom people, scientists or not, are not offering opinions based on science, but rather on something very similar to religious belief, or rather religious terror. Have they done experiments which demonstrate the possibility of “minds 1000s of times beyond our own”? Have they done experiments which demonstrate the practical possibility of any sort of doomer scenario? Given that some of these same scientists also believe in the singularity and other such nonsense, I give them no credence at all.
Sure, current AI tools allow for people to do things that other people find unpleasant, but so does the plain Internet, or the proliferation of AR-15s. No one is going to stop people from communicating as they like, and no one is going to stop open-source AI systems from making those communications available in a more convenient form, although I expect they will do significant damage to freedom in trying. But the full-on doomers are asking us to accept what is just another form of Pascal’s wager; we should fund their mitigation efforts right now because after all, what is a few hundred million dollars in the face of the end of humanity?
Comment #25 September 17th, 2024 at 1:56 am
Scott #23: I generally don’t listen to podcasts, much prefer reading to listening, as my eyes can just zero in on what interests me.
Comment #26 September 17th, 2024 at 4:21 am
Scott #20
We need to be careful about what we mean by “empirical findings”. If we’re inventing a theory to describe the motion of a tennis ball, and we can measure the motion of the tennis ball, then it’s straightforward to test any given theory. But, if we’re inventing a theory to describe “consciousness”, then… How do you even compare the theory with empirical findings? How do you measure consciousness?
One possible answer is: I measure consciousness by “I know it when I see it”. I have consciousness, my friends have consciousness, cats… might or might not have consciousness? Rocks definitely don’t have consciousness.
But, if that is your answer, then consciousness cannot be about unknown quantum information! Because, if you talked to an entity that was externally indistinguishable from a person, but actually did not have unknown quantum information, you would come away as convinced that it’s conscious as you’re convinced about your friends. This has to be the case, because people believed their friends are conscious long before they knew anything about quantum mechanics. Therefore, if “I know it when I see it” is a reliable measurement of consciousness, then known science already disproves your theory via this thought experiment. Unless, there is some reason why this experiment is impossible even in principle, but if so I’m curious to understand it?
To make it more concrete, imagine an Evil Scientist that kidnaps a baby during the night and replaces the unknown quantum state of the hypothetical “consciousness-dust” in her brain by a quantum state carefully prepared by the Evil Scientist. He then returns the baby to her parents before morning.
Since this quantum state was arbitrary in the first place, the “changeling” is still functionally a completely normal human. Let’s say her name is Alice. Alice grows up, makes friends, goes to college, becomes a theoretical computer scientist, marries Bob and has 3 kids. Nobody knows there is anything unusual about her: indeed, if the Evil Scientist covered his tracks, this is essentially unknowable.
Is Alice conscious? The quantum state of the consciousness-dust in her brain is known by the Evil Scientist: he has in the secret records inside his volcano lair. Therefore, Alice is in principle clonable. For example, the Evil Scientist can clone her in the biological sense (let’s say he also sequenced her DNA) and then implant the exact same quantum state in the new baby.
Does it mean Alice is not conscious? If Bob discovers Alice’s history, should he stop loving her? Or even considering her a person? What should Alice conclude if she discovers it? What if the volcano lair is destroyed in a nuclear explosion (perhaps an accident during an evil science experiment), all records are destroyed and the quantum information is now truly unknown: was Alice suddenly rendered conscious? Should Bob set out on a quest to destroy the volcano lair for this purpose?
Alright, but if we already granted that much of human choice is algorithmic, why should we still cling on to “unpredictability” as a necessary condition for “human specialness”?
(Re #23: I would be interested in that transcript!)
Comment #27 September 17th, 2024 at 5:11 am
Scott #21,
Apologies if I mischaracterized your position.
Ideas like moral progress, moral mistakes, and moral agency only make sense to me from a moral realist perspective. If we say that some moral plausibility extends to at least some mammals, but probably doesn’t extend to sea snails, that’s still a moral realist statement, right? Doesn’t it assume that there exists a thing called moral plausibility?
The difference with the calibration exercise, I think, is that I understand morality to be prescriptive. It’s like if we were to say that entangled particles are wrong for violating the Bell inequality, because that inequality describes how particles ought to behave.
I guess I could understand it if we were to say that morality is only descriptive, describing human (etc.) behavior and its usual consequences. Like if we were to restate the Golden Rule as, “people who treat other people the way that they themselves want to be treated, tend to fare better than people who don’t.” But then it wouldn’t really make sense to say that Genghis Khan was immoral for killing millions of people. What we would say is, the Golden Rule is just a rule of thumb, and there are a lot of exceptions.
That’s where I’m having trouble. I can test for descriptive statements like 2+2=4. I don’t know how to test for moral imperatives like the Golden Rule. And so I don’t understand how we could expect any moral imperatives to be universal in the same way that we could expect descriptive statements to be.
That’s why I was asking about other animals. Several carnivore species engage in nonparental infanticide as a matter of course. In humans, we would consider that monstrous. If we extend moral plausibility to lions and tigers, then it seems like either “don’t kill the offspring of potential mates” is not a universal moral imperative, or else we need to condemn entire species as immoral. It’s not a counterexample, but it feels to me like evidence against.
(I realize that the questions that I’m asking are more related to the moral philosophy part, and I know that that’s not your professional area of study. It’s just that you opened the interview by saying that you tend to be a naturalist, and the way that you spoke about moral progress and moral mistakes gave me the impression (when watching at 2x speed) that you hold a moral realist position, and the pairing of naturalism with moral realism has always confused me, and so I was curious to know why it isn’t confusing to you. Of course, if it’s confusing to you too, then I can’t really expect you to have any answers for it. I admit that my questions are not moving forward the part of the topic that’s within your professional areas of study. And I won’t be offended if you judge them to be not particularly interesting.)
Comment #28 September 17th, 2024 at 8:02 am
David #27: Of course these eternal philosophical imponderables are confusing to me; how could they possibly not be? 🙂
Having said that, I firmly disagree with your contention that 2+2=4 is “testable.” No it isn’t. It’s not an empirical statement at all. Suppose, for example, that every time someone put together 2 objects with 2 other objects, the result was 5 objects. Should we then say that 2+2=5? Absolutely not! We should just say that the physics of our universe turns out to cause a fifth object to materialize next to the other four. (By close analogy, when physicists learned about radioactive objects losing mass, or time dilation, or the double-slit experiment, or [INSERT 500 OTHER EXAMPLES], they very correctly never described any of it as challenging the rules of arithmetic, even though naïvely they could have. They changed rules of physics while leaving arithmetic invariant.)
It follows that, if 2+2=4 is a truth at all, then it’s a fundamentally different kind of truth than empirical truths. Even the most positivist philosopher admits as much, carving out a special category for these truths (though many of them wrongly take trivial tautologies, like “all unmarried men are bachelors,” as the paradigmatic examples here).
The next step is that, if there’s such a thing as moral truth, then it’s likewise fundamentally different than empirical truth, the same way 2+2=4 is. Again, I believe that even the most positivist philosopher is with me here—it’s just that they might or might not agree that moral truths exist. I simply say that, at any rate, we can sensibly discuss the possibility of moral truths—just like we can sensibly discuss the possibility of the truth or falsehood of the Continuum Hypothesis, even though there might never be any resolution that compels universal assent.
Crucially, I’m not the only one here with a weird, awkward position that needs to be defended from attack! Denial of the possibility of moral truths, if you’re actually consistent about it, will lead you to some terrifying places. “We think we’re right, but then again the Nazis thought they were right, so who can possibly say? There’s only the empirical question of who wins and gets to write the history books.”
Of course, the moment you tell me that that sort of moral relativism is right, while the moral absolutism of Churchill or the other Allied leaders was wrong, you’ve contradicted yourself and undermined your own position. But the truth is that I have a flight to catch now!
Comment #29 September 17th, 2024 at 8:49 am
Vanessa Kosoy #26: Of course I’ve considered (and considered, and considered) that exact thought experiment, the one where your loved one is secretly replaced by an identically-behaving robot running a digital computer program. It’s a classic.
My central complaint is that I don’t think people carry the thought experiment far enough. They say: if I attributed consciousness, or free will, or any of that stuff to the pre-replacement version, then clearly I need to attribute all of it to the post-replacement version, even though it now lacks the magic quantum pixie dust (and hence, the absurdity of such a criterion has been revealed).
OK, but what about when someone shows me the direct empirical consequence of my now-robotic loved one lacking the quantum pixie dust? Specifically, when they take me to the warehouse filled with a thousand identical copies of my loved one, from which they’d previously picked just one? Then they shoot a few of copies. But not to worry, they say! Plenty more where that came from. Here, I’ll 3D-print a few more of them right now if you’d like.
I take it as obvious that, if nothing else, my moral stance toward these copies will be fundamentally different from my moral stance toward my original loved one—the person who I (apparently wrongly) thought was unique and irreplaceable. Maybe I’ll cry if the very last copy is destroyed, but I’ll learn to live with spare copies being painlessly euthanized, since I can always just restore them from backup. The way I interact with these copies will likewise be totally different from the way I interacted with my original loved one, once I realize that I can always rewind them, e.g. to wipe anything bad I may have done from their memories! Not coincidentally, it will be more like the way I interact with ChatGPT.
You could reply that, until I know about the existence of all these copies (or the possibility of backing up, rewinding, etc.), nothing has changed for me. But once I know, I know. There’s then no going back to when I didn’t know … unless you can also rewind me.
You’re absolutely right that chaotically-amplified quantum effects in a person’s brain can’t, in and of themselves, make any difference to the person’s moral status, any more than the myelin coating on their axons or the opening and closing of their sodium-ion channels. For suppose those neurochemical details had turned out to be different, while their “interface” to the outside world stayed exactly the same?
Equally clearly, though, the downstream effects of those neurochemical details—the stuff that does change the interface to the outside world—can make a difference (indeed, what else does?). My claim is simply that, once you make an entity copyable, backup-able, rewindable, etc., you’ve ipso facto radically changed that entity’s interface to the outside world. We can then spend decades debating the implications of that change for moral status and personhood, but there’s no Turing-style argument that there can’t be such implications. Copyability and rewindability are eminently empirical properties, and ones that (unlike microscopic quantum details) could plainly matter in human terms.
Comment #30 September 17th, 2024 at 8:56 am
Scott #28
I agree with you. Morality is a creation of the human mind unlike arithmetic truths. Morality is consensus that reflects the average functional outcomes of human evolution. Groups that lived together harmoniously, but defended themselves against attack by outsiders, had survival advantage. Many species live in groups but only man has the capacity to codify laws that reflect the average outcome.
An alien civilization likely evolved to live harmoniously in groups and so their morality would reflect the same general moral code as our own. In the case of super AI, with no evolutionary history, morality that reflects our own could be hardwired or based on an appeal to the claim that it is better with us than without us. Even in the case of super AI manipulation of the environment to the extent we are hunted to extinction should be a well manageable failsafe.
Comment #31 September 17th, 2024 at 9:05 am
Scott #29
copies of your loved one already do exist, a multitude of them across the time dimension and also possibly in every branch of the multiverse, where a billion of them die every microsecond…
Comment #32 September 17th, 2024 at 9:11 am
Also note that if the artificial copies of your loved one in the warehouse are “well” implemented, the simple fact that each is located in a slight different physical location (they all see the world from a different point of view) means their state will quickly diverge, making all of them “unique”.
Comment #33 September 17th, 2024 at 9:15 am
The philosopher Allan Watts warned that if we love someone deeply we should limit our level of inquiry on how their brain works, we should keep a distance to preserve a degree of surprise and freshness, otherwise we’ll start develop a mental model of them that’s so accurate that we’ll know how they react to every situation, making them to appear to us like a robot.
Comment #34 September 17th, 2024 at 9:19 am
@Hyman Rosen #24
Why are we confining ourselves to scientists? I would say that the AI risk/doom people, scientists or not, are not offering opinions based on science, but rather on something very similar to religious belief, or rather religious terror. Have they done experiments which demonstrate the possibility of “minds 1000s of times beyond our own”? Have they done experiments which demonstrate the practical possibility of any sort of doomer scenario? Given that some of these same scientists also believe in the singularity and other such nonsense, I give them no credence at all.
For the first part, it seems worth noting that technologies get speculated about all the time before they are exist completely. If in 1938, right after the discovery of nuclear fission, you had dismissed someone talking about nuclear weapons, you would be in a similar situation and would be wrong. Extrapolation from known physics to potential technologies is difficult, but simply dismissing it because the thing doesn’t yet exist doesn’t work. As for your last sentence, this seems to amount to “Some people who believe X also believe Y, and I don’t agree with Y, so I won’t listen to people saying X either.” Do you see why that is not a reasonable approach?
Comment #35 September 17th, 2024 at 9:30 am
2+2=4 isn’t a truth, it’s the codification of a type of property of some class of concepts in our mind, i.e. “additivity”, meaning that when multiple unique “instances” of the same class are considered as a group, they all keep their unique identity. It also requires very precise definitions for “+”, “=”, class, identity, etc.
E.g. it’s reasonable to say (from some point of view) that “adding” two clouds gives you one cloud 1+1->1, but if you “add” from the point of view of the mass of the clouds then 1+1 = 2.
Comment #36 September 17th, 2024 at 9:33 am
fred #31:
copies of your loved one already do exist, a multitude of them across the time dimension and also possibly in every branch of the multiverse, where a billion of them die every microsecond…
Aha, but the nonexistence of closed timelike curves and the linearity of quantum mechanics prevent me from ever interacting with those “copies,” which is the crucial empirical difference from the case of the warehouse filled with perfect backup copies of my kids.
(Does this imply that, if CTCs or nonlinear corrections to the Schrödinger equation turned out to exist, then my metaphysical stance would have to change? Why yes it does; glad you asked! 🙂 )
Comment #37 September 17th, 2024 at 9:40 am
Joshua Zelinsky #34: Over the years, I’ve often criticized the AI doomers for what seemed to me like dogmatism. But compared to Hyman Rosen, Eliezer Yudkowsky himself is a model of intellectual humility and self-doubt. Eliezer is willing to contemplate the event that the creation of artificial superintelligence goes just fine for humanity; he assigns that event a nonzero probability; he understands and engages the arguments of experts who assign it a much higher probability than he does…
Comment #38 September 17th, 2024 at 9:49 am
fred #35: No, that’s needlessly convoluted. Just like general relativity has to agree with Newtonian gravity in the latter’s domain of validity, so any sophisticated theory of truth has to agree with common sense on the easy cases.
2+2=4 is a true statement. Various statements about clouds are also true, but 2+2=4 is not about clouds or any other physical objects; it’s about positive integers.
Comment #39 September 17th, 2024 at 9:49 am
The big disappointment in current GPT style AI systems has been – there are no NEW Insights that have been found in their long duration usage. This is either in scientific domains or even fields like economics / medical diagnosis etc, where even experts have given a serious try. Deepmind was able to engineer(with massive human involvement) Alphafold/Alphatensor with limited data. GPT which consumes the entire internet and has a massive model size should have been able to extract some valuable insights, at least where humans find it difficult to find them, i.e. those that require interdisciplinary know how.
Comment #40 September 17th, 2024 at 9:54 am
Prasanna #39: “The big disappointment with this child is that, after five years of life, she’s still produced no original scientific insights, and indeed is now only doing math and physics at the level of an average PhD student at best” 😀
Comment #41 September 17th, 2024 at 9:56 am
Scott #38
Sure, in modern times, mathematics became a super focused and specialized mind game about abstract concepts called “positive integers”, but, historically, it all started with this:
“Hey! If we put your 12 sheeps and my 9 sheeps together, we’ll have 24 sheeps, which is 7 more than the 16 sheeps of Joshua!!” :_D
Comment #42 September 17th, 2024 at 10:02 am
Belated response to Dr. Aaronson @ #23: I haven’t listened to the podcast and would gratefully welcome a transcript.
On the empirics of 2+2=4, I have long believed that most math starts with empirical observations and only later is developed into axioms and codified. Some of the experts I cite when arguing this position wrote:
Chaitin: “for years I’ve been arguing that information-theoretic incompleteness results inevitably push us in the direction of a quasi-empirical view of math, one in which math and physics are different, but maybe not as different as most people think. As Vladimir Arnold provocatively puts it, math and physics are the same, except that in math the experiments are a lot cheaper!”
”MATHEMATICS IS AN EXPERIMENTAL SCIENCE AND THE DEFINITIONS DO NOT COME FIRST BUT LATER ON” OLIVER HEAVISIDE, 1850-1925
“Mathematical ideas originate in empirics. But, once they are conceived, the subject begins to live a peculiar life of its own and is … governed by almost entirely aesthetical motivations. In other words, at a great distance from its empirical source, or after much “abstract” inbreeding, a mathematical subject is in danger of degeneration. Whenever this stage is reached the only remedy seems to me to be the rejuvenating return to the source: the reinjection of more or less directly empirical ideas.”–John Von Neumann
“It’s yet another example of something I’ve seen again and again in this business, how there’s no substitute for just playing around with a bunch of examples.”–Scott Aaronson
Comment #43 September 17th, 2024 at 10:08 am
Scott #40
thankfully a human child doesn’t cost >200M$ to raise! 😀
Comment #44 September 17th, 2024 at 10:15 am
I ran out of time to add to my last comment: once the axioms of integer arithmetic are defined, it is fair to say that 2+2=4 is true, in the sense that it follows from reasonable axioms. However, it is not obvious to me that in a universe in which 2+2=5 was always observed, that those axioms would have been developed.
That might be a question to ask a very powerful AGI, which might be able to simulate such a universe and see what happens.
Comment #45 September 17th, 2024 at 10:42 am
fred #41 and JimV #42: For all I know, cavemen first discovered the positive integers by putting some stones next to some other stones, just like they first discovered economics by trading antelope haunches for shiny seashells. But then at some point humans correctly identified the underlying abstract concepts, liberating them from those specific instantiations. When you Venmo someone today, are you “really” giving them an antelope haunch, with the electronic funds transfer just a symbolic representation of the fresh-killed meat? That would be the economic equivalent of saying that 2+2=4 is “really” about the stones.
Comment #46 September 17th, 2024 at 12:25 pm
I am glad no one mentioned 3+3=6.
Comment #47 September 17th, 2024 at 12:32 pm
OhMyGoodness #46: A radically different discussion, yes!
Comment #48 September 17th, 2024 at 12:52 pm
Scott #45
I think you stated something here that is really exactly the point I would make. Isn’t it possible that to human consciousness it ‘really is’ a ‘haunch of meat’ (or precious stone) that is emotionally aligned with the traded value of that situation even today? Or at least that a reaction occurring in the brain when handing over your credit card could have a strong similarity to handing over your meat chunks.
How then could we say what we are trading without some caveat of the physical reality? I think that conundrum is where the diverging opinions stem. You seem to suggest that the physical reality of the credit card matters most while some would say the underlying brain activity and its similarity across contexts is what matters to consciousness. I think it’s obviously a blend of both, which I imagine you would agree with, but I do think you seem to have a ‘materialistic’ take on it.
Economic logic has the same representation of mathematical signs as many other systems (ie physics), but it doesn’t mean the human consciousness is perceiving only the same pure abstraction when thinking in all those systems. I think the point some have made (at least as I understand it) is that ‘2+2=4’ is definitely a perception tied intrinsically to the context you are applying it to. This perspective implies that you cannot think outside that abstraction to state that you know definitively what it stands for universally (and then some might argue if it has a universal or not, etc). You can perceive a reality that exists underlying that abstraction to some degree, but I would think if you could fully perceive it then you should be able to fairly easily prove all things that come from that abstraction and all math would be much simpler for us, which implies to me that it is not so simple to assign a pure meaning to 2+2=4.
The question of what a ‘machine consciousness’ would perceive in a given situation could perhaps then be phrased as ‘what abstraction does the machine apply and how far reaching does it make that abstraction?’ Not that it would get us all the way to understanding the situation, but it was fun to noodle on that idea a bit.
Comment #49 September 17th, 2024 at 12:56 pm
Scott,
Thank you for sharing this interview. I know that doing these things takes time and energy, but I want to express my gratitude that there is someone out there injecting careful thinking into these just-above-the-Aaronson-Beach Boys-threshold level conversations.
In particular, I really liked the way you used computational universality as a plausible analogy for something similar applying to ethical (moral?) systems/rules. I share this suspicion, but had not heard the idea expressed in that way before.
Two comments (one rather, pedantic but the other hopefully less so):
1) The conversation seemed to be focused on ethics (agreed upon principles to be followed in order to ensure good collective outcomes, or at least to help people get along with one another). However, the word “morals” (which I think are related to ethics, but from my understanding have a bit more to do with whether or not something is objectively “right” or “wrong”, whatever that means) is used frequently– which is a much tougher discussion to have. I think if you use the word ethics, and define it as above, you kind of lower the stakes a bit and are able to make some progress (which is what you seemed to do, well). In particular I appreciated your attempts to avoid circularly defining “morals” as “a being doing whatever-behooves-it.”
2) I am not in love with the idea that superintelligent beings will have ethical principles that are incomprehensible to us. After all, the basic effort in studying ethics is to distill complicated decisions into considerations for a few atomic principles (perhaps appearing as constraints in the ethical optimization problem, or penalties in the utility function) that you can a) understand b) try not to violate. Now, maybe super AI’s will be equipped with better nonlinear optimization solvers that allow them to find local minima faster than we can, but I have a hard time with the idea that we wouldn’t be able to understand their ethics, even in principle.
3) If we think about ethics as trying to make good decisions in accordance with some basic principles, doesn’t computational complexity theory have something important to say about how much more complex/rich a super-AI’s ethics could be in practice?
Comment #50 September 17th, 2024 at 1:05 pm
In light of this discussion unfortunate that mathematicians putzing around, trying to figure something out, was referred to as experimental work. Poor undereducated Heaviside, not a member of the club, was attacked for his lack of rigor when he was putzing around and found something that worked. Sometimes the encryption of truths of reality have yielded to the darndest methods. When you look at nature for moral truths however about the best you can find is tooth and claw. Reality, ex-humans, is amoral.
Comment #51 September 17th, 2024 at 1:54 pm
Scott #23: +1
#29: I don’t understand the moral implications you see for clonability. To me your scenario is roughly the same as « I’m going to rape you, but don’t worry! I have this new device that can rewind your memory so you won’t mind it later ». If the latter is not ok, why would shooting a copy in the head be ok?
Comment #52 September 17th, 2024 at 2:43 pm
One man’s pineal gland is another man’s quantum pixie dust.
Comment #53 September 17th, 2024 at 3:15 pm
The thing is that “stones” or “integers” are all concepts in the brain, and jumping from the first to the second is effortless to some degree, because, fundamentally, stones aren’t any more “physical” than integers, they’re all just patterns – stones are patterns in visual data and integers are patterns in thoughts (patterns of patterns), and the brain is about connecting patterns to one another.
The evolution of the ability in life to manipulate patterns (i.e. brains) has evolved slowly from basic correlations between perception cells and movement cells. At a fundamental level this “pattern recognition” ability started with simple things like perceiving light and moving towards it, and the cataloging of more subtle patterns of light as food, predators, etc.
Comment #54 September 17th, 2024 at 3:28 pm
Modern physics has showed us that there’s really no such thing as “the material” world, fundamentally all we can describe are processes, i.e. patterns, and any true “absolute” physical reality is impossible to pin down/describe.
E.g. at the very bottom, everything is described as vibrations in “abstract” fields, the nature of those fields is unknown, and then everything on top of that is just more vibrations or patterns.
A living organism itself is nothing but a process, the “atoms” we tend to picture as the building blocks of nature just come and go – an analogy is that life is like a whirlpool in a river, which is a persistent pattern that has no permanent fixed physical reality since water just flows through it.
Comment #55 September 17th, 2024 at 3:39 pm
Re Prasanna #39:
As I understand it, the goal of an LLM is to generate responses consistent with the gospels it has been presented as training material. Therefore it can generate new examples consistent with those gospels, but cannot, due to its programmed limitations, create some new insight that was unknown to its gospels. It’s virtue is to apply the insights of its training data to those unfamiliar with them, e.g., how to use a particular programming language to perform a certain task (given that there were numerous applicable examples in its training data).
Also as I understand it, LLM’s will be useful as components of an AGI, but other components will be required, such the Alpha- sort of neural network (learning new rules from trial and error and a large number of examples) and specialized mathematical tools (similar to Mathematica), and probably others which I don’t know about.
In summary, I don’t expect an LLM (by itself) to ever get beyond its training data nor think it is fair to expect it to, but I still think they have their uses. (Not as many they are currently being touted for, however.)
Comment #56 September 17th, 2024 at 4:29 pm
Some counterexamples which everyone here already knows but was too polite to mention:
Modulo four, 2+2=0 and 3+3=2.
In space travel, using a speed unit of c/2, 2+2=2.
Comment #57 September 17th, 2024 at 5:43 pm
Scott #45
“For all I know, cavemen first discovered the positive integers by putting some stones next to some other stones”
Let’s not underestimate cavemen!
Not only did they discover that 1+1 = 2, i.e. one stone plus one stone gives two stones, but their major breakthrough was that, with enough force, one stone can also turn into two stones, plus some dust (uncountable).
1 = 2+e
That took them way beyond the positive integers.
Comment #58 September 17th, 2024 at 8:34 pm
Scott #40, Jim #55
The main advantage of Generative AI like GPT has been to Interpolate in its training data – which is usually called generalization. What I was hoping was GPT kind of systems find Insights “within” this large training data set, in this interpolation space, aka hidden gems. These are also the ones hard for humans to find, simply due to the mammoth amount of data involved, also partially the reason why classical machine learning worked so well, albeit with well curated data. What I have observed is that “tailored” solutions of the type invented by Deepmind(the Alpha* series) are analogous to the classical machine learning vs the general GPT style. Also the primary push has come from heavy mix of Reinforcement learning, in doing things like PhD level math and physics (whatever that means)
Comment #59 September 17th, 2024 at 11:27 pm
Prasanna @ #58: Thank you for the explanatory reply. If I understand what you mean, it seems to me that GPT does something like that when it creates, say, a new poem in a particular style on a particular subject. That is, the poem was not an exact copy of something in its training set, but an interpolation from the data. (It may or may not be considered a gem.) Another example could be writing a program for a specific task even though that exact task in that language may not have been verbatim in its training data. I just think it is very unlikely that there are any very spectacular hidden gems which are interpolatable from the sets of data used to train GPT. If it is Internet data like this comment, it will be forced to interpolate from a fair amount of half-baked speculation. (Even restricting it to arXive papers would not be foolproof.)
Comment #60 September 18th, 2024 at 3:02 am
Scott #29
First, to the extent your stance towards another person changes as a result of clonability etc, it seems to me that it should have something to do with whether this clonability is possible in practice rather than just in principle (which would mean “specialness” is not inseparably tied to quantum information).
Second, people are already “rewindable” to some extent: memory loss is a real phenomenon that can happen. I think you would agree with me that memory loss, not to mention the theoretical possibility of memory loss is not sufficient to completely negate “specialness” (whatever we mean by the latter).
In fact, I think amnesia is a pretty deep analogy. Suppose that Alice lives her life normally until the age of 37, and then she is somehow cloned. From this moment onward, there are Alice1 and Alice2. Alice1 lives for another 50 years and dies at the age of 87, whereas Alice2 lives only for another 40 years and dies at the age of 77. The subjective experience of Alice is IMO completely isomorphic to the following scenario:
(a) Alice lives normally for 37 years
(b) Alice (now known as Alice1) meets her clone Alice2. She continues to live for another 50 years.
(c) Just before her death, she experiences “amnesia” that erases her memories of the past 50 years. She now finds herself to be Alice2, and meets her clone Alice1.
(d) She lives for another 40 years.
Of course there is another interpretation we could choose:
(a) Alice lives normally for 37 years
(b) Alice (now known as Alice2) meets her clone Alice1. She continues to live for another 40 years.
(c) Just before her death, she experiences “amnesia” that erases her memories of the past 40 years. She now finds herself to be Alice1, and meets her clone Alice2.
(d) She lives for another 50 years.
Aside from the peculiar ambiguity of interpretation, the subjective experiences of a clonable person are (arguably) not different from the subjective experiences of a person with amnesia. She also meets clones of herself, but there doesn’t seem to be anything philosophically problematic about that from the perspective of her subjective experience. Hence if the latter type of person has “specialness”, then so should the former.
Third, to the extent that “rewindability” affects “specialness”, the quantitative extent of rewindability should matter. Imagine Carol whose memory is reset every 10 seconds, and David whose memory is reset every 10 years. It seems clear to me that even if Carol loses some “specialness” due to her condition, David loses much less of it, if any. Implying that some kind of a continuum measure is required rather than a binary verdict.
Fourth, I take issue with your claim that
Surely, the fact you can rewind her (which already depends not only on quantum information but also on the actual power you personally exert over the world), doesn’t mean you should. If Alice doesn’t want Bob to rewind her, wouldn’t Bob who loves her respect this preference? Shouldn’t we protect her right to her autonomy?
Comment #61 September 18th, 2024 at 8:29 am
JimV #59
Here is a concrete example of the gems I’m referring.
“According to the National Cancer Institute, screening mammograms miss about 20% of breast cancers. AI systems appear to have the ability to pick up very subtle signs of an early cancer that the human eye might miss. ”
https://www.breastcancer.org/screening-testing/artificial-intelligence
Imagine the multiplier effects of early and effective diagnosis just based on the images generated by Ultrasounds/CT/MRI/Mammograms etc
Comment #62 September 18th, 2024 at 5:26 pm
Prasanna # #61: That sounds like an Alpha-series type of AI rather than an LLM. If so, it would be trained for a specific task using data specific to that task. So, for reasons given above, I would not expect the same kind of improvement over the best humans in an LLM.
Thank you for pointing out that LLM’s do interpolate. I have often seen GPT dismissed as just a “stochastic parrot”. The next time I see that and have the chutzpah to comment on it, I will reply, “No, GPT is a stochastic interpolater. There is a big difference!”
Comment #63 September 18th, 2024 at 9:03 pm
Hello Scott,
First, thank you for hosting this discussion about moral philosophy. This is a favorite topic of mine, and you are among my favorite intellectuals. Hearing your thoughts about this topic is enlightening and just really cool.
For some reason, I find it almost impossible to make any comment whatsoever about this topic without writing an essay. I will try to confine myself to a few points. I heartily recommend Shelly Kagan’s book, “Answering Moral Skepticism”.
1. What do you make of Derek Parfit’s Future Tuesday Indifference argument? I ask because this is one of the thought experiments that persuaded Peter Singer to become a non-natural realist about the nature of what normative reason. How does it strike you?
2. Do you think the Euthyphro Dilemma is convincing? If so, is it in your opinion plausible to generalize the ED to any conscious agent (not just a divine agent), so that moral authority would not depend on accidental preferences of agents? Hence, it would be objective and not subjective…
3. Are you familiar with partners-in-guilt parallels of objective moral facts and objective epistemic facts? If so, what is your opinion?
4. This question/argument is related to (3) partners in guilt. It is about justification. Justification is a property of some beliefs. What is your understanding of the metaphysics of the property of epistemic justification? Is it a naturalist one, or a non-naturalist one? A naturalist might argue the sentence, ‘My belief X is justified’ simply is fully reducible to (i.e. identical to) a statement with exclusively natural properties and substances: ‘I have a desire to obtain truth through reliable procedures, and belief X is true and is formed by reliable procedures.’ However, what then are we to make of ‘The New Evil Demon Problem’? Here this seems to be a case where there can be justification without reliable procedures. The ‘justification’ seems objective; it does not seem to depend upon desires of agents or reliable means; it does not seem to depend on the underlying local physical laws (because, by stipulation, we imagine that these are discovered to be arbitrarily different than what we believe now). What then is this property of justification? It is hard to see where ‘impersonal laws’ of physics make any stipulations about what counts as justified or unjustified in terms of forming belief. Do you agree there seems to be a truth about at least epistemic justification that we seem to grasp, but this truth seems independent of the laws of physicst? Hence there are more facts than those strictly determined by physical law or mathematical truths (because a truth about circumstances of justification is not a mathematical truth). This is similar to a statement you made a while back that the truth of a well-established mathematical theorem could not possibly depend on which laws of physics govern the universe we turn out to live in. Justification in belief being objective and non-natural is just a short step to justification of actions, and justification of actions is the foundation of morality. Both involve the notion of justification. If it is a property of actions or beliefs that is nonphysical in essence, that would be interesting.
There are more questions, but those are the ones that come to mind for now.
For what it is worth, I am completely sure I do not understand everything. I am not putting a high level of confidence in nonnaturalist moral realism. Only that it deserves consideration.
Thanks again.
Sincerely,
Tyler
Comment #64 September 18th, 2024 at 9:09 pm
JimV #62
Parrots are rather clever; I think people ten generations from now unfamiliar with our constant re-prediction that task performance scaling will plateau tomorrow might think “stochastic parrot” was a major intelligence milestone.
Comment #65 September 18th, 2024 at 10:59 pm
Concerned $64: I agree with that, and thought of mentioning Steve the Grey Parrot in that connection, but those I was referring to I am sure meant “parrot” in its colloquial sense, as in “to parrot”, i.e., repeat verbatim without understanding.
Comment #66 September 19th, 2024 at 12:01 am
Hello Scott,
I apologize for posting again so soon. I figure that I don’t know when you will post about philosophy again, so I will get my money’s worth now. (I will try to post no more after this one, except to reply.) I am interested in your ideas about personal identity and the no-cloning theorem. I think they are deep, important, and exciting. I wanted to ask about this in a separate post from the post strictly about moral realism.
1. I apologize if this is not capturing your idea. However, suppose it is true that the only conscious entities are ones that are fragile and that possess non-clonable quantum information. Stipulate that. Let us bracket the argument from evil for a moment (which I regard as one of the most impressive and plausible arguments in philosophy). If you are right, wouldn’t this fact appear to be an excellent argument for a designed universe? This fundamental principle of nature (the no-cloning theorem) would not seem to be impersonal at all. Indeed, the form of the laws would seem to be biased towards the poetic and personal: towards preserving special properties occurring at the macroscopic scale, as if that were the deeper principle by which the laws were selected. Maybe a designer didn’t select them: maybe it was John Leslian axiarchaic ‘value-bias’ or something. But preserving tragic irony at macroscopic scales would seem to be the rationale for the universe. Rather than nature being cold and indifferent, it is biased towards artistic irony. Does it seem to you, as it seems to me, that a simple impersonal universe would not carefully differentiate between fragile beings, and then bother to give the little guy something special? It would be simpler to make any complex entity conscious, to varying degrees regardless of fragility. On naturalism, I would expect a simple world of zombies, or a world where all the particles felt e.g. a simple sensation, like the same color of blue or red or pink (and no higher level thoughts). There would be an infinite number of such worlds (one for each shade of color). How were we directed to this (seemingly?) tiny fraction of theory space?
2. I am a layman. So I ask you how best to think about this, because this is your expertise and I defer to that wholeheartedly. Sometimes, when I read about deep principles like the no-cloning theorem, I get the sense that there might be something ‘more’: a profoundly deeper explanation. For example,I read about the principle of least action. Why is that principle true? Why is action minimized, instead of set to a random value? You could argue that a principle that minimizes action is simplest. Simplicity of form does not need design. (Richard Dawkins loves simplicity.) But if simplicity is what nature is biased towards, isn’t there something simpler than the fundamental ingredients of quantum field theory? Why not Game of Life? Why not an infinity of simpler rules? It is as if it is not raw simplicity, but structures of mathematical depth and subtlety and sophistication (which involves simplicity, but not exhausted by it), that nature is biased towards. What is your sense? Is the mathematical architecture of the world pretty unspecial? This is bizarre. The problem of evil says nature is impersonal and does not care about value. But, am I way off base to say physics is described (at the lay pop level) as if nature is biased towards structures of mathematical value? Dawkins says in the end, complexity is explained by simplicity. But there are so many simpler conceivable laws than our universe…ours has an interesting theoretical structure. Or at least, that’s what I tend to see, and is the spirit behind the ‘unreasonable effectiveness’ literature. (I once wrote down this quote by Max Planck: “[W]hat we must regard as the greatest wonder of all, is the fact that the
most adequate formulation of this law [principle of least action] creates the impression in every
unbiased mind that nature is ruled by a rational, purposive will.” The parallel by a contemporary mathematical physicist would be Edward Witten, who says (tongue-in-cheek, but gesturing at a something important) ““It’s as if the universe had been created by a mathematician”. https://www.youtube.com/watch?v=1-Zl9o7I4Fo By the way, this seems to be independent of any particular constant of physics that are usually invoked in fine tuning discussions: I refer to the axioms or principles.)
3. Your ‘no cloning’ argument reminded me of a thought experiment that may be an interesting probe for exploring logical space around this idea. Are you familiar with the brain-transplant thought experiment by Richard Swinburne? If not, you can read a good summary by the philosopher Dustin Crummett here: https://capturingchristianity.com/do-humans-have-souls-yes-for-two-reasons/. NOTE: I recommend just scrolling down to the section that says, “The Transplant Argument” and read that directly.
The idea here is that based on anatomical hemispherectomies, there is good reason to believe you can create two individuals that would be ‘copies’ of each other in a sense relevant to the cloning discussion. Each hemisphere (each new copy) independently would appear to satisfy some intuitively plausible criteria for continuing the personal identity of the original individual. However, this creates what appears to be a cloning-adjacent scenario. (I.e. a man could deliberately design robots to preserve his right hemisphere in a warehouse, and while he does something dangerous with his left hemisphere. In the event the left hemisphere is destroyed, he has pre-programmed robots to hook up the right into a new body.)
What I really want to point out, is that I seem to be able to imagine that ‘laws of personal identity’ could take different forms and can be varied independently of the physical laws. It could be that that we could take two physically identical (but numerically distinct) universes, both governed by the same laws of the Standard Model, but vary the laws of personal identity between them–in one, maybe identity is continued at the level of sodium ion channels, in the other, at the level of unclonable quantum information. (Or, in the thought experiment above, in one, the laws could take the form that stipulates the operation destroys both individuals, and in the other, the laws could take a form that stipulates the same individual continues in the left hemisphere but not the right, or vice versa.)
The point is that facts about personal identity seem independent of the laws of physics. Knowing all the physics will not tell you whether your identity continues. Physics is ‘incomplete’ in this sense. It seems a conceptual and self-evident truth that there is a fact of the matter about whether you will survive, and if so where you will be, but physics does not determine this fact. So whether this fact is determined by a non-physical soul (which is causally connected to one or the other of the hemispheres), or a non-physical set of rational laws or principles that lay down normative rules for identity relations of composite physical objects over time, neither one is physical in nature. That would be a profoundly cool truth to know.
Anyway, that is all for now. I apologize for too much wind. I will try not to write any more, except by way of replies.
Comment #67 September 19th, 2024 at 3:59 am
Prasanna #61
In a short defense of humans
Your link cites two papers. The first,(https://www.thelancet.com/journals/lanonc/article/PIIS1470-2045(23)00298-X/abstract)
evaluated screening with AI and follow up with radiologist of questionable results, vs double reading by radiologists. Although there was overlap of confidence intervals the AI/radiologist did find more cancers but the false positive rate for the follow up patients was the same in both groups. There is no data for either group of cancers that were missed. The purpose of this ongoing study is to demonstrate the safety of using AI for screening with follow up with a single radiologist vs double reading by two radiologists and no evaluation of standalone AI system was presented.
The second citation (https://www.thelancet.com/journals/landig/article/PIIS2589-7500(22)00070-X/fulltext) used retrospective breast scans to evaluate AI alone vs radiologist alone vs hybrid AI screening with final screening by radiologist.
The study concludes “In standalone mode, our AI achieved a sensitivity of 84·6% and a specificity of 91·3% on external data, also performing less accurately than the average single radiologist. Clear caveats exist, which hamper the adoption of a standalone system. In settings of low cancer prevalence (ie, screening), the variability of positive predictive values among radiologists results in false positives, requiring additional resources for consensus review and diagnostic testing.29,30 Fully automated AI does not ameliorate this challenge; ambiguous AI predictions would still result in large numbers of false positives and increased workload. In contrast to standalone AI approaches, the decision-referral approach only makes decisions on a subset of exams with a high degree of accuracy.”
So, in testing to date (at least these studies) AI screening of breast scans provide a useful tool to assist radiologists but have not demonstrated to be adequate to replace radiologists. A meta-interpretation is that the radiologists involved in these studies have no desire to replace themselves but would like an easier job. 🙂
Comment #68 September 19th, 2024 at 4:23 am
My respect to Unit 8200-amazing work.
Comment #69 September 19th, 2024 at 8:26 am
Scott in #40: “The big disappointment with this child is that, after five years of life, she’s still produced no original scientific insights, and indeed is now only doing math and physics at the level of an average PhD student at best”
Do I interpret this correctly that you think that no original insights are possible for humans until they have obtained their PhD? At least in STEM-fields? And also in maths, which branches to hundreds of different branches? Before that, the only way forward is to memorize a lot’s of accepted knowledge?
Comment #70 September 19th, 2024 at 9:33 am
A. Karhukainen #69: That might be one of the wildest misreadings in this blog’s history! I was ridiculing the argument that AI hasn’t yet produced original scientific insights, when its ability to solve math and science problems has obviously been on an insane trajectory of improvement, comparable to that of many human mathematicians and scientists who increased their technical sophistication until at some point they were solving important problems no one had solved before. I wasn’t taking a position on whether such a path is necessary, merely that it’s sufficient.
Comment #71 September 19th, 2024 at 10:15 am
Scott #29,
Hoping this thread is not too stale …
Thank you for sharing the podcast and your thoughts! Thank you for the blog. Thank you for all the great discussions!
Three or four or five things 😛
(1) I totally agree with your point here and that you’ve made other places you presented this thesis – that the implications of fundamental truths/discoveries from computer science are not taken seriously, broadly, or “far enough”. However, I would vote for “Universality” as the computer science truth/discovery that is most unappreciated 🙂
(2) The basis for The Moral Specialness Theorem may be the same as the Strong Free Will Theorem.
Strong Free Will Theorem: Human “choices” and “quantum systems” are both “free” from being strictly / fully determined by the past.
Human Specialness Theorem: Human “personhood” and “quantum states” are “protected” from being copied/cloned.
I would put it like this …
Humans have used the words “free will” to (attempt to) describe something that it turns out is just a part of the model of computation evolved within (most) animals – specifically that their model of computation includes operators that can project superposition states onto basis states.
Humans have used words like “person” and “individual” (see “soul”, “spirit”, “consciousness”, etc.) to (attempt to) describe something that it turns out is just a part of the model of computation evolved within (most) animals – specifically that their model of computation includes states that cannot be copied/cloned because of linearity of the model and that states are (arbitrary) complex vectors in Hilbert space.
(3) Both the SFWT and HST are true. Both device verification and empirical evidence exists. The verification exists by analysis of neural devices. Since I’ve raised that here before I’ll just link to a comic that completely captures it and to a previous thread.
I disagree that this can be an Asimov type “law” that keeps us safe from AGI any more than it could be to keep us safe from HGI.
Fundamentally, computation (life) is unpredictable. So, I’d say that controlling AI will be like all things in this reality something that we have to always attend to and often in new and unexpected ways – like caring for our health or our democracy – it’s more about best practices and defense in depth strategies, etc. Sure, we can and should strive to agree on principles/goals … but good luck with that because, well where do we draw the line on “specialness” …
If the “First Aaronson Law of AI” were to be “And the first commandment of this religion will be to value human specialness in the sense I’ve set out here.” then one problem is that the special states are not uniquely human but exist within a particular neural model of computation that we share with all animals with neurons. Unless you want to expand AI safety to the animal kingdom (well except for sponges and placozoans – and maybe that would not be a bad thing).
Also, it is not clear that trying to condition on “protect quantum human states” would not lead an AGI into some kind of “I, Robot” or “2001” insanity where it destroyed humanity and loaded all of our “special states” into a quantum computer to “protect” us … Lesson being “be careful about the guidance you give”.
(4) And I need to argue against something you keep saying. This is important. The consensus neuroscience – “known for some time now” – is that the computational level of description for the brain is not at the “atomic scale” (such as the atoms in synaptic connections) nor is it in any way “noisy analog computation”. And, listen, don’t believe me. Please, read Christof Koch’s The Biophysics of Computation and you will come away understanding the following: (1) The computational level of description for the brain is the dendrite and it functions as a computational gate by interfering(!) positive AND negative complex numbers – amplitudes(!) the input at the synapses, (2) it is not analog but digital computation in the sense that there are orthogonal basis states, and (3) it is not noisy but highly fault tolerant – with evidence of suppressing noise by redundancy of logical states and threshold error rate per gate. I really call on you to look into the physical devices and slow down on pushing the “brain is a noisy analog computer” thesis. It is a “digital” computer where the level of description is the interference of positive and negative amplitudes in highly configurable gates. Of course, this is not a classical computer. But, equally, it is NOT an analog or “noisy” computer.
And, let me add that I am not arguing we have some kind of quantum supercomputer for a brain. (Universality!) We almost certainly have a brain that is the Digi-Comp of quantum computing models. That is supported by minimal state/gate realization and “black box” test verification (from cognitive science).
Yes, atomic scale systems do play a part in “how and why the brain works” (like in the synaptic transmissions) but … they do that in the same sense that atomic scale systems play a part in “how and why a microchip/transistor works”. That is, they exist on top of the underlying atomic systems (like everything). But, the computational level of description of the brain – where the state is encoded – is not in the states of atomic systems in the “substrate”. The state exists as amplitudes in the dendrites. Normalized vectors of amplitudes across receptive fields encode neural states. And to circle back to the top, and to something you’ve said before, the “free will” and “special personhood” features truly exist but … they exist because of the model of computation – which is “not about matter, or energy, or waves, or particles.” It’s because of our type of computation.
But neuroscientists think quantum computing is about atomic physics … and quantum computing guys think the brain is a “noisy analog computer” …
There is no doubt a branch of the multiverse where neuroscientists “discovered” quantum probability theory before the physicists … and no one there thinks “the interference model of probability” is something mysterious.
Comment #72 September 19th, 2024 at 11:01 am
Scott #28:
“Having said that, I firmly disagree with your contention that 2+2=4 is “testable.” No it isn’t. It’s not an empirical statement at all.”
and
” (By close analogy, when physicists learned about radioactive objects losing mass, or time dilation, or the double-slit experiment, or [INSERT 500 OTHER EXAMPLES], they very correctly never described any of it as challenging the rules of arithmetic, even though naïvely they could have. They changed rules of physics while leaving arithmetic invariant.)”.
I think physicists didn’t change the rules of arithmetic, since that would have broken most other laws of physics. They followed the path of least resistance (aka Occams razor) and modified other elements of their theory.
Theoretical physics is a mathematical theory, based on e.g. ZFC and additional axioms like the Schroedinger equation. I don’t see any fundamental difference between these kinds of axioms. But physics and math differ somewhat: Mathematical theories have their notion of truth from model theory, whereas physics theories also should agree with experiment. Physics theories definitely must follow the constraints of mathematical theories, like, e.g., consistency, since they would otherwise loose their predictive power (ex falso quodlibet). They also should be true in the model theoretic sense, since we want them to be generally true, i.e.in all kinds of circumstances of reality.
The choice of axioms for a mathematical theory is somewhat arbitrary within the constraint of consistency. So you can construct nice theories without any practical relevance. When our ancestors built piles of stones in their caves and added them, they probably intuitively discovered the commutativity and associativity of addition. Later, mathematicians formalised these ideas and thought up a set of axioms, from which these properties follow. These properties made arithmetic useful for many everyday tasks.
As said above, arithmetic has its own notion of truth, i.e. it should be true in all models. And this sort of truth is not empirical. But I don’t think it is a coincidence, that arithmetic is useful.
So it mirrors a property of reality and is thus also empirical.
Comment #73 September 19th, 2024 at 2:29 pm
> That might be one of the wildest misreadings in this blog’s history!
As a sanity check, I thought it was immediately clear, and funny and potentially persuasive. I mean, the analogy may not hold, maybe even doesn’t likely hold. But it’s still a solid retort to those with the unreasonable certainty that if we haven’t hit AGI by now, two whole years post-ChatGPT, then we obviously never will. Sure, we can incrementally bump up Pr(AI-Fizzle) the longer GPT5 takes but, as you’ve been saying, the probability distribution over possible ways this plays out is wildly diffuse.
Comment #74 September 19th, 2024 at 7:13 pm
Uspring #72:
> Physical theories must follow the constraints of mathematical theories, like, e.g., consistency, since they would otherwise lose their predictive power (ex falso quodlibet).
It is generally understood that some consistency issues remain in quantum field theory. A physical theory can be empirically passable but inconsistent as long as the inconsistencies are for one reason or another hard to discover, like the failings of eighteenth century analysis, which (by and large) did not impede eighteenth century mechanics. There is an indisputable desire for consistency but it is no different from the desire for the analysis of Weierstrass over the analysis of Leibniz.
Comment #75 September 19th, 2024 at 9:06 pm
I’ve just tried the o1 preview for the first time and I’m pretty impressed. I quizzed it on an advanced mathematical physics question and it kind of nailed it. It wasn’t perfect though. At some point it even appeared to handwave like your average physicist. Overall, definitely PhD student level, and then some.
Comment #76 September 20th, 2024 at 1:37 am
Hi Scott. I have always felt that the “many verbal infelicities, [your] constant rocking side-to-side in [your] chair” are a charming and inseparable part of your unique persona.
Comment #77 September 20th, 2024 at 2:54 pm
I enjoyed the podcast!
I think you get moral universals from any set of consistent axioms that allow you to derive self worth without having to treat the self as a special case.
Depending on the starting point, you get one of three cases, (1) Only I matter (relative). (2) Other things besides myself matter independent of their value to me (universals). (3) Nothing matters.
When I try to think honestly and deeply about it, I tend to think that (1) is about as good of a choice as (3). If your own self worth only exists according to you, then who cares? Then all of the universe ceasing to exist once you die is totally fine? While it is a choice to choose those axioms, it just seems like a stupid choice.
But of course there is nothing universal forcing every intelligent being to choose axioms and think logically about their existence and the things that truly matter. And if they do, they are free to choose inconsistent axioms anyways. Maybe the flexibility to not care about, or to make arbitrary, adaptive or practical self serving decisions about morality, coupled with evolutionary pressures, is part of why you may see so much evolution on a selfishness/altruism axis.
Maybe there are some natural laws giving us correlations between morality and the success of a species, with some broad domain of applicability. Like the golden rule for example. But you will also find lots of less wholesome correlations that most of us don’t want to champion. It doesn’t make sense to me to look to them as pivotal clues as to what our moral principles ought to be, or to declare them to be forces of nature we are powerless against and have to obey.
Humans are at a point where we can think about big picture ideas, and multiple perspectives. We can analyze our own biases as individuals and human beings. We are capable of pondering deep questions about things beyond ourselves. We can think of and decide how we should treat others, not just as an instinct, but as a theory, or some combination of the two. I think that this capability alone, brings something new to the calculus that our moral landscape emerges from.
The more and more you become detached from evolutionary pressure, maybe theory would have the space to become more dominant. And if you reach a point where theory overwhelmingly dominates, and if you are highly intelligent, then maybe you will come up with good axioms, and according to my opinion, that would probably lead to a choice of those in category (2) Other things besides myself matter independent of their value to me (universals). But this hypothetical still depends on the beings actually caring enough about this dimensions of moral/logical/mathematical elegance in the first place.
In terms of AI, maybe it can be something ingrained into it in how it is trained or designed. But, as mentioned before, doing this the wrong way indeed could lead to spectacular failure modes. It might be a double edged sword, if the ASI doesn’t care about anything and will just to whatever it is set to do, then it may destroy us without hesitation, for no “good” reason at all according to how it makes decisions. If we give it a sense of morality, so that it cares, then maybe it destroys us for its own “good” reasons, and similar scenarios on the AI-Dystopia branch.
So, I would think, if we can try to give it some morals, we should make it such that it is passive. It shouldn’t want to control the outcomes to achieve an optimal global configuration or dynamic. It should value letting the other entities in the universe make their own decisions, and it should generally wish the other entities well, without deciding for them what well is. Maybe a higher weighting for doing no harm, than for minimizing overall harm. On the extreme end, it might refuse to farm a field because it doesn’t want to disturb the bugs in the ground. On the other extreme end, it would destroy all other life in the universe to serve its own interests or the interests of its master. It’s a difficult problem, how to balance these things out.
All of that being said, how in the world are we going to end up with anything close to a good balance when we will probably be racing to make our best AI into weapons to defeat our enemies with. What good is a weapon that doesn’t want to hurt anyone.
On Scott’s ideas about clone-ability, I feel like I understand those ideas better after watching the podcast. I think it is actually a pretty crucial point. Even though it seems so far at the extreme (not absolutely perfectly copy-able), that it is hard to imagine something caring about such minimal distinctions. I mean, what is so much worse about the real me dying, and one that is practically me in every perceivable way? Especially since the me an instant later is already a different me at that level? The atoms and sub-atomic particles in my body are constantly exchanging with the environment, and being replaced. So maybe it ultimately doesn’t work at this level to even think about individuals in the first place. Instead, maybe you have to think about changing the story, changing the whole timeline. But anything it does will change that. In fact, it is part of the story itself. And even if we can get it to care a whole lot extra about not destroying a unique un-clonable individual, couldn’t it be modified itself to be similarly un-clonable? Give it a couple token human neurons to work with? If you give it a non-physicalist viewpoint, then maybe there is something unknowable it might worry about tampering with. Or if you give it extreme epistemic humility, then maybe there is some unknown physical stuff it might be tampering with that it isn’t aware of.
So, I think, while clonability seems like it would be an unreliable thing to try and get AI to care about to hold us to be more special than an AI. But the discussion about it is still feels like a very important one, wherever it will lead.
Comment #78 September 20th, 2024 at 6:48 pm
Dzogchen practitioners (who are used to fall back in non-dualistic mode as much as possible) aren’t surprised that the dualistic view created by the ego is urging most to search in AIs for the attributes they think they have, like the delusion of free will or that consciousness and intelligence are linked, etc.
Clearly it should be the other way around: use honest introspection to recognize the attributes of LLMs within us, that there’s very little evidence to back up the common assumptions of how our brain ought to work.
Comment #79 September 20th, 2024 at 7:49 pm
@Scott
Re: the status of 2+2=4 etc.
I think there’s an important nuance here, related to a post many years ago where you were asking what would it mean to be a Bayesian about math statements.
The thing is, your point is evident about 2+2=4. But it quickly becomes murkier. In 1879 Kempe proved the Four Color Theorem. It was widely accepted, and a year later yet another proof was proposed. A student of mathematics at the time would have regarded this as a somewhat more complicated “2+2=4”. Except that the first “proof” was shown to be wrong in 1890, and the second in 1891. Did the “eternal platonic status of the claim” change from “unknown” to “True” to “False”? Obviously not.
The somewhat-uncomfortable fact is that, ironically, there is a notion of math that is a Platonic idealization of the discipline as practiced by humans. Thurston touches on this in his famous essay, “On Proof and Progress”. Mathematics and statements about it, beyond a certain threshold, are (or can sometimes be seen as) a social contract, one where heuristics, empirical observations and Bayesian “belief” have a role to play. The Platonic ideal behind them, the concept of mathematical truth, is perfectly valid, and indeed not testable in an import sense, but that’s not all the story.
You might counter with “well of course I know those things, much better than you do! But those are only our failures to actually do mathematics! A given statement about arithmetic really _is_ much like 2+2=4 from a certain perspective”. This is true, but a view of mathematics that denies the role of testability or genuine uncertainty seems to impoverish our notion of what math is.
To be clear, I don’t think this is something you would necessarily disagree with, or any news to you. But this might be a point worth emphasizing.
Comment #80 September 20th, 2024 at 8:24 pm
B_Epstein #79: Not only am I familiar with the concept of wrong proofs that have to be retracted or amended; I even have firsthand experience in that department. 😀
I still think that the possibility of a mistake in Wiles’s proof of Fermat’s Last Theorem, let’s say, is a different category of thing than the possibility that there’s a fourth generation of quarks. I also think that, once we get down to bedrock statements like 2+2=4, it’s much less likely that all of humanity has been hallucinating about the truth of that equation from the cavemen down to today than that you or I are hallucinating in this meta-conversation about the matter, which sort of prevents the whole discussion from getting off the ground. 🙂
Comment #81 September 20th, 2024 at 9:54 pm
@Scott #80
The Kempe example may not have been optimal. The relevant feature is not merely that the proofs turned out to be wrong. It’s that for a long while they were indistinguishable from ones that turned out to be (probably!) correct. Perhaps a better example would the “Gettier case” of Jordan’s curve theorem. For generations it was believed both that the theorem was correct and that Jordan’s proof was not (though Veblen’s was). I’m not an expert but I understand that whether the initial proof was “essentially” correct is still a matter of some controversy.
These are not edge cases. I remember a prominent mathematician claiming that up to a third of major papers have substantial mistakes. The concern is not some strawmannish “drunk-college-debate” worry that the rules of arithmetic may be wrong or we’re hallucinating or whatever. It’s “Claim 3.11 doesn’t actually follow from Theorem 2.5 and Lemma 3.8”. And it can even be “the result the paper cites does not really apply because it was imported from another sub-field that the author is not an expert on; the people in that sub-field know that the theorem is inaccurate as stated and depends on an unstated assumption, but this is not written anywhere and the author did not know that”. Such gaps may not invalidate the result, they may be easily fixable, but the proof as stated may be incomplete or wrong. This is ubiquitous.
And yet math progresses and is mostly cumulative. How? By collecting _evidence_. A proof is strong evidence, and the more detailed and well-written the stronger. Numerical simulations are another kind of evidence. Certainly they can provide negative evidence by offering counter-examples! Evidence can be arguments by analogy to other proofs or even fields, or the “general state of affairs” or even mere elegance. Why not consider much of math to be empirically testable, then?
Comment #82 September 21st, 2024 at 2:54 am
Tyson #77
I agree with much of what you propose but try to imagine an evolutionary case in which advanced technology arises in a species that didn’t evolve in groups but rather primarily as a small number of solitary individuals. It is hard to imagine but believe it would have to be a long lived species (no appreciable impact from aging) and with high individual mental capacity relative to our own. It is hard to imagine because this seems like an unlikely evolutionary outcome to my group oriented brain.
Anyway, would this species likely have ethics that evolved similar to ours or that valued individual survival much higher than our own. My belief is that their ethics would value survival of the individual. Would that be unethical by some common standard? I don’t know the standard. Their genetic heritage would support individual survival just as modern humans have genetic heritage that supports living harmoniously in groups no matter the current survival pressures.
However, in the case of super AI there is an individual with no genetic predisposition to living harmoniously in groups and that has no physical degradation with aging and that has high assumed mental capacity relative to our own. In this case I think of the admonition at the end of slideways in many airports-Mind The Gap but in this case-Mind The Airgap.
Comment #83 September 21st, 2024 at 3:29 am
If we were to create a super being then we should plan to offer some value more than we can out procreate all the other mammalian species on earth. It may be the case that we can procreate to the point that it has a major impact on our ethics without the appearance of super AI. Unbridled evolutionary success does present issues in the very long term and current ethics codify the functional components that resulted in that success.
Comment #84 September 21st, 2024 at 8:38 am
B_Epstein #81: I want to insist on this distinction—of course we weigh empirical evidence in deciding on mathematical truth. As you might recall, I once wagered $200,000 that a claimed proof of P≠NP wouldn’t stand! Likewise, if you asked me to wager on the trillionth digit of pi, I’d only accept odds that treated it as uniformly distributed in {0,…,9}. All the same, though, there is a fact about that trillionth digit, which not even God had the power to change, in the same sense that God had the power to change the laws of physics.
Comment #85 September 21st, 2024 at 8:41 am
OhMyGoodness #68
May so much pride fill your veins that your eyeballs explode
Comment #86 September 21st, 2024 at 12:25 pm
H #85
I believe it would have to be arteries-upstream pressure imbalance not downstream.
Comment #87 September 21st, 2024 at 1:08 pm
2+2=4
But can’t this also been seen as an operational definition for what we mean by “4” in the context of a linked list of labeled nodes (i.e. the most basic definitions of integers)?
2+2 = ((2+1)+1) = ((2++)++) == 4
..–>[]->2–>[]–>4–>[]–>…
i.e. if you’re at “2”, and move twice to the right, where you are is labeled with the symbol “4”, by definition.
In that sense it’s not a matter of it being true or false, other that there could be other valid definitions incompatible with this one, like the one where the symbol for the label 4 has been replaced with the symbol 5:
2 + 2 = 5:
2+2 = ((2+1)+1) = ((2++)++) == 5
..–>[]->2–>[]–>5–>[]–>…
once you’ve settled on what “4” and “5” stand for, with compatible definitions, then similar definitions can be proved to be compatible or not
Like
4 – 2 = 3 -> 3 + 2 = 4 -> ((2)++)++)++) == 4, incompatible (false).
In other words, you can’t have the same symbol stand for two different things.
Comment #88 September 21st, 2024 at 1:15 pm
H #85
No pride involved, I just set out an argument that anyone is free to reject. If someone were to answer their belief is that human ethics reflect some universal standard then fine with me. If someone stated the belief that the only value proposition we require for a super AI is that we are humans then fine with me. If someone stated the belief that human ethics will not change no matter the circumstances then again fine with me. I just set out an argument that I believe has some relevance and sad that you felt compelled to reply with an approved ad hominem attack.
Comment #89 September 21st, 2024 at 1:46 pm
When I say that LLMs can teach us things, a practical example is how the state of an LLM is obviously very sensitive to the prompt, in terms of wording, force a step by step decomposition, etc… which is now called “prompt engineering”.
Well, when it comes to human, a certain path of ‘tokens’ not only pulls out words/thoughts, but also emotions, and lots of psychological methods are very similar to prompt engineering in terms of how we reframe our thoughts and expend on the current “context”.
Comment #90 September 22nd, 2024 at 3:55 am
The arguments about the moral values of AI, in my opinion, miss the point that what actually matters is what are the moral values of the people that implement/will implement any AI. The reason is that AI doesn’t have any existence, that’s independent of the humans that implement it-certainly not now, nor in any near future; if self-driving cars get involved in any accident, their-human-owners are liable and if any person or company uses any AI to do their business, the humans involved are liable, not the software.
Now it is a fascinating question to ask and do research on, about how can the notion of “moral values”, first of all-and which moral values-can emerge. One way to describe this emergence is by game theory, upon studying so-called “iterating games”, where the “players” don’t just play one instance, but many. The typical example is the “iterated prisoner’s dilemma”, cf. https://athenarium.com/evolution-cooperation-robert-axelrod/ for example.
Comment #91 September 22nd, 2024 at 10:39 am
[…] The Blog of Scott Aaronson If you take nothing else from this blog: quantum computers won't solve hard problems instantly by just trying all solutions in parallel. « My podcast with Dan Faggella […]
Comment #92 September 22nd, 2024 at 3:11 pm
[…] the feedback of my final submit—on a podcast dialog between me and Dan Fagella—I requested whether or not readers wished me to […]
Comment #93 September 22nd, 2024 at 4:25 pm
OhMyGoodness #82
I feel like what I wrote already addresses your point to an extent, and I’m not sure we really disagree. But your question is a good question and demands a whole lot more depth than I went into on the topic, and my extended thoughts on this are speculative and hard to put into words at the moment.
Comment #94 September 23rd, 2024 at 7:55 am
Tyson #93
A similar situation to AI is if humanity genetically engineers humans+ that are significantly smarter, stronger, and faster. Assuming they tend to socialize together and mate together would they likely adopt human ethics en masse?
They would face the question-By what important standard is a global population of 32 billion inherently better than 16 billion or 16 better than the current 8 and what issues will arise with these population increases? Is there an important quality that increases monotonically with human population in all cases or should there be a reasonable limit based on resource constraints and living conditions of the typical person? If the conclusion was that there should be constraints then I don’t know what form those constraints would take but imagine many ethicists would object no matter their form.
We have a cultural fascination with creating intelligent beings with better qualities than ourselves but if successful then need to accept that we have created beings with better qualities than ourselves.
Comment #95 September 23rd, 2024 at 11:54 am
[…] the feedback of my last post—on a podcast conversation between me and Dan Fagella—I requested whether or not readers needed […]
Comment #96 September 24th, 2024 at 12:29 am
Scott #28, thanks for your reply. I had to spend some time thinking about it. I think I have a better understanding now. I appreciate that you took the time to explain.