Quickies
Updates (Dec. 5): The US Supreme Court has upheld Trump’s latest travel ban. I’m grateful to all the lawyers who have thrown themselves in front of the train of fascism, desperately trying to slow it down—but I could never, ever have been a lawyer myself. Law is fundamentally a make-believe discipline. Sure, there are times when it involves reason and justice, possibly even resembles mathematics—but then there are times when the only legally correct thing to say is, “I guess that, contrary to what I thought, the Establishment Clause of the First Amendment does let you run for president promising to discriminate against a particular religious group, and then find a pretext under which to do it. The people with the power to decide that question have decided it.” I imagine that I’d last about half a day before tearing up my law-school diploma in disgust, which is surely a personality flaw on my part.
In happier news, many of you may have seen that papers by the groups of Chris Monroe and of Misha Lukin, reporting ~50-qubit experiments with trapped ions and optical lattices respectively, have been published back-to-back in Nature. (See here and here for popular summaries.) As far as I can tell, these papers represent an important step along the road to a clear quantum supremacy demonstration. Ideally, one wants a device to solve a well-defined computational problem (possibly a sampling problem), and also highly-optimized classical algorithms for solving the same problem and for simulating the device, which both let one benchmark the device’s performance and verify that the device is solving the problem correctly. But in a curious convergence, the Monroe group and Lukin group work suggests that this can probably be achieved with trapped ions and/or optical lattices at around the same time that Google and IBM are closing in on the goal with superconducting circuits.
As everyone knows, the flaming garbage fire of a tax bill has passed the Senate, thanks to the spinelessness of John McCain, Lisa Murkowski, Susan Collins, and Jeff Flake. The fate of American higher education will now be decided behind closed doors, in the technical process of “reconciling” the House bill (which includes the crippling new tax on PhD students) with the Senate bill (which doesn’t—that one merely guts a hundred other things). It’s hard to imagine that this particular line item will occassion more than about 30 seconds of discussion. But, I dunno, maybe calling your Senator or Representative could help. Me, I left a voicemail message with the office of Texas Senator Ted Cruz, one that I’m confident Cruz and his staff will carefully consider.
Here’s talk show host Seth Meyers (scroll to 5:00-5:20):
“By 2027, half of all US households would pay more in taxes [under the new bill]. Oh my god. Cutting taxes was the one thing Republicans were supposed to be good at. What’s even the point of voting for a Republican if they’re going to raise your taxes? That’s like tuning in to The Kardashians only to see Courtney giving a TED talk on quantum computing.”
Speaking of which, you can listen to an interview with me about quantum computing, on a podcast called Data Skeptic. We discuss the basics and then the potential for quantum machine learning algorithms.
I got profoundly annoyed by an article called The Impossibility of Intelligence Explosion by François Chollet. Citing the “No Free Lunch Theorem”—i.e., the (trivial) statement that you can’t outperform brute-force search on random instances of an optimization problem—to claim anything useful about the limits of AI, is not a promising sign. In this case, Chollet then goes on to argue that most intelligence doesn’t reside in individuals but rather in culture; that there are hard limits to intelligence and to its usefulness; that we know of those limits because people with stratospheric intelligence don’t achieve correspondingly extraordinary results in life [von Neumann? Newton? Einstein? –ed.]; and finally, that recursively self-improving intelligence is impossible because we, humans, don’t recursively improve ourselves. Scattered throughout the essay are some valuable critiques, but nothing comes anywhere close to establishing the impossibility advertised in the title. Like, there’s a standard in CS for what it takes to show something’s impossible, and Chollet doesn’t even reach the same galaxy as that standard. The certainty that he exudes strikes me as wholly unwarranted, just as much as (say) the near-certainty of a Ray Kurzweil on the other side.
I suppose this is as good a place as any to say that my views on AI risk have evolved. A decade ago, it was far from obvious that known methods like deep learning and reinforcement learning, merely run with much faster computers and on much bigger datasets, would work as spectacularly well as they’ve turned out to work, on such a wide variety of problems, including beating all humans at Go without needing to be trained on any human game. But now that we know these things, I think intellectual honesty requires updating on them. And indeed, when I talk to the AI researchers whose expertise I trust the most, many, though not all, have updated in the direction of “maybe we should start worrying.” (Related: Eliezer Yudkowsky’s There’s No Fire Alarm for Artificial General Intelligence.)
Who knows how much of the human cognitive fortress might fall to a few more orders of magnitude in processing power? I don’t—not in the sense of “I basically know but am being coy,” but really in the sense of not knowing.
To be clear, I still think that by far the most urgent challenges facing humanity are things like: resisting Trump and the other forces of authoritarianism, slowing down and responding to climate change and ocean acidification, preventing a nuclear war, preserving what’s left of Enlightenment norms. But I no longer put AI too far behind that other stuff. If civilization manages not to destroy itself over the next century—a huge “if”—I now think it’s plausible that we’ll eventually confront questions about intelligences greater than ours: do we want to create them? Can we even prevent their creation? If they arise, can we ensure that they’ll show us more regard than we show chimps? And while I don’t know how much we can say about such questions that’s useful, without way more experience with powerful AI than we have now, I’m glad that a few people are at least trying to say things.
But one more point: given the way civilization seems to be headed, I’m actually mildly in favor of superintelligences coming into being sooner rather than later. Like, given the choice between a hypothetical paperclip maximizer destroying the galaxy, versus a delusional autocrat burning civilization to the ground while his supporters cheer him on and his opponents fight amongst themselves, I’m just about ready to take my chances with the AI. Sure, superintelligence is scary, but superstupidity has already been given its chance and been found wanting.
Speaking of superintelligences, I strongly recommend an interview of Ed Witten by Quanta magazine’s Natalie Wolchover: one of the best interviews of Witten I’ve read. Some of Witten’s prouncements still tend toward the oracular—i.e., we’re uncovering facets of a magnificent new theoretical structure, but it’s almost impossible to say anything definite about it, because we’re still missing too many pieces—but in this interview, Witten does stick his neck out in some interesting ways. In particular, he speculates (as Einstein also did, late in life) about whether physics should be reformulated without any continuous quantities. And he reveals that he’s recently been rereading Wheeler’s old “It from Bit” essay, because: “I’m trying to learn about what people are trying to say with the phrase ‘it from qubit.'”
I’m happy to report that a group based mostly in Rome has carried out the first experimental demonstration of PAC-learning of quantum states, applying my 2006 “Quantum Occam’s Razor Theorem” to reconstruct optical states of up to 6 qubits. Better yet, they insisted on adding me to their paper!
I was at Cornell all of last week to give the Messenger Lectures: six talks in all (!!), if you include the informal talks that I gave at student houses (including Telluride House, where I lived as a Cornell undergrad from 1998 to 2000). The subjects were my usual beat (quantum computing, quantum supremacy, learnability of quantum states, firewalls and AdS/CFT, big numbers). Intimidatingly, the Messenger Lectures are the series in which Richard Feynman presented The Character of Physical Law in 1964, and in which many others (Eddington, Oppenheimer, Pauling, Weinberg, …) set a standard that my crass humor couldn’t live up to in a trillion years. Nevertheless, thanks so much to Paul Ginsparg for hosting my visit, and for making it both intellectually stimulating and a trip down memory lane, with meetings with many of the professors from way back when who helped to shape my thinking, including Bart Selman, Jon Kleinberg, and Lillian Lee. Cornell is much as I remember it from half a lifetime ago, except that they must’ve made the slopes twice as steep, since I don’t recall so much huffing and puffing on my way to class each morning.
At one of the dinners, my hosts asked me about the challenges of writing a blog when people on social media might vilify you for what you say. I remarked that it hasn’t been too bad lately—indeed that these days, to whatever extent I write anything ‘controversial,’ mostly it’s just inveighing against Trump. “But that is scary!” someone remarked. “You live in Texas now! What if someone with a gun got angry at you?” I replied that the prospect of enraging such a person doesn’t really keep me awake at night, because it seems like the worst they could do would be to shoot me. By contrast, if I write something that angers leftists, they can do something far scarier: they can make me feel guilty!
I’ll be giving a CS colloquium at Georgia Tech today, then attending workshops in Princeton and NYC the rest of the week, so my commenting might be lighter than usual … but yours need not be.
Follow
Comment #1 December 4th, 2017 at 3:50 am
“In particular, he speculates (as Einstein also did, late in life) about whether physics should be reformulated without any continuous quantities. ” people should read “The Other Einstein: Einstein Contra Field Theory by John Stachel which appeared in Science in Context 6, 1(1993), pp. 275-290 (note: this article seems to be hard to find; i can send a pdf of it to anyone interested), john baez’s article at https://arxiv.org/pdf/1609.01421.pdf. and stephen wolfram’s blog entry at http://blog.stephenwolfram.com/2015/12/what-is-spacetime-really/
Comment #2 December 4th, 2017 at 4:15 am
We have no hope of tackling AI on the political level. Heck, even when faced with global warming – an urgent problem that has an easy solution – our leaders respond with timid, non-binding agreements, when not outright denial. How could we expect them to even understand the danger of AI, and reason about the difficult trade-offs involved in the various solutions?
Nope, the way to deal with AI will be decided solely by its creators: Google and Facebook. We’re all gonna die.
Comment #3 December 4th, 2017 at 4:36 am
Mateus #2: Sure, there’s a lot that scares me about entrusting the future of all earth-originating intelligence to some engineers at Google. And yet, if the choice were between that and the American political system, which has already catastrophically failed, rather than merely being vulnerable to catastrophic failures in the future … I’d go with Google every single time.
Comment #4 December 4th, 2017 at 5:02 am
What I’m whining about is precisely that the political system has catastrophically failed (not only in the US, look at the UK, Brazil, or even Germany to see pretty bad failures), and as such is incapable of protecting us.
Because democratic societies at least should (and often actually do) look after the well-being of the population.
Google? It’s goal is to maximize profit. And what happens when a profit maximizer builds an AI dedicated to maximize profit? Well it’s going to fucking maximize profit!
Comment #5 December 4th, 2017 at 5:10 am
You could make a “Devil you know” argument about the political system over AI. It does sometimes get things right and so far at least when it is wrong it has been been overcome (not without cost). With AI we don’t know what we might get locked into for our future without any recourse.
Certainly intelligent machines will be faster at doing almost anything humans can do whether it be intellectual or physical. Whether they will reach a level of intelligence that is qualitatively different from humans is the question. If they do, that will be the point where we will not know or maybe even understand where our future is heading.
Comment #6 December 4th, 2017 at 5:16 am
Glad to see that you are finding AGI plausible. I also angst for greater intelligence in our world. Personally I think the time line is more like a decade than a century. But near term AGI is too cognitively dissonant for most people, a century just feels safer.
Comment #7 December 4th, 2017 at 5:38 am
What’s the workshop in nyc?
Comment #8 December 4th, 2017 at 6:06 am
Matt #7: It from Qubit annual meeting.
Comment #9 December 4th, 2017 at 8:00 am
It’s really that scary a super-AI?
I find it clear that we’ll crack intelligence at some point. When that happens we’ll have some funny time playing around with artificial brains, trying to establish upper bounds on intelligence, finding the optimal sizes and structures for different tasks… Soon enough we probably have to accept that biological brains are way too inefficient, and humans will end up being replaced.
I guess my point is that there’s no big difference between raising and educating children or raising and educating AIs, both are valid next generations of minds. I see it as an evolutionary leap forward. The only we should try to guarantee is the construction of a progressive civilisation, and that requires diversification. Moreover, creativity only flourish among lots of different minds.
In that sense, I really like Jeff Hawkins’ ideas on intelligence. I don’t see any AI being as “artificial” or alien, but human intelligence being too “human”. You may like this Google Talk: https://www.youtube.com/watch?v=4y43qwS8fl4
Comment #10 December 4th, 2017 at 8:00 am
I know that this is a very naive question, but what exactly is scary about AGI? Is the assumption 1) that somehow computers will develop consciousness even though we haven’t the vaguest idea how it depends on physics and are not even trying to create it and 2) once they do, computers will be granted legal rights not to be unplugged or isolated from the web and we will have to stand by as they destroy us?
1) seems wildly implausible and 2), if possible, even more so. So even given 1), we just unplug the damn things if they start acting up.
This is not an informed question: I have literally read nothing on the topic. These questions are why. Any quick answers?
Comment #11 December 4th, 2017 at 9:16 am
Tim Maudlin #10: Most ‘big’ computing platforms are not literally un-plug-able since they are distributed across many physical computers (and those computers sit in many different datacenters all over the world). They are built to be fault resistant in the sense that damage to any one computer or even one data center does not make the platform unusable. Mix that with the observation that intelligent humans can manipulate other humans into acting in their interests and you get a safe assumption that intelligent machines should also be able to manipulate humans into acting in the interests of the machine (e.g. by not acting to disable the machine).
You don’t need exactly need a machine to have human-style consciousness for any of this, only one that acts like an agent (can evaluate some utility function and act to increase it) and is very good at solving a lot of different problems (in the same way that humans are good at solving a lot of different problems). The main worry is that, whatever utility function such a machine ends up with, if you maximize that utility function you end up with a reality that humans don’t like (for instance, one where humans don’t exist any more). The paperclip maximizer example gives you one such utility function (humans like to have a little bit of paperclips, but they absolutely hate when all matter in the universe is converted to paperclips).
Comment #12 December 4th, 2017 at 10:08 am
The whole “an AI explosion can’t happen” claim is really dubious:
1) Such explosions did happen already.
The earth did experience first an “explosion” of organic life – with the planet getting covered with plants, animals, to a complete saturation.
And then another explosion of in terms of meat-based intelligence, even faster, where in a few million years life was able to leave the earth, and in theory it could now spread to every part of the galaxy.
The claim that human intelligence doesn’t improve itself is false since that’s what’s science is really about – human technology is on an improvement feedback loop. And AI can be seen as an extension of human intelligence improving on itself.
2) A human brain is a finite machine. It occupies a finite space and uses a finite quantity of matter, taking into account trade-offs specific to carbon based life and the need to evolve using organic food and sexual reproduction.
There is no intrinsic reason why the processes happening within a human brain can’t be entirely reproduced.
And once that happens, there’s no reason to believe that those processes can’t then be accelerated in the usual way that computers allow for the scaling of computations.
So, once you have the equivalent of a PhD researcher brain capacity in a box, you’ll be able to compress time to such an extent that you’ll get 20 years worth of “cogitation” within a few days or a few minutes.
3) the claim that intelligence is as much as in the environment than in its brain is true, but it’s also true that intelligence can be made flexible enough to adapt to any environment.
Which is the case for humans – the human brain can both thrive in prehistoric lifestyles (hunting gathering) or in modern lifestyles (doing highly abstract tasks).
Also, intelligence is in part in culture, but every time a baby is born, he/she can absorb quickly that external knowledge within a few years… so a new brain and a few TB of information is all it takes.
AI brains will be able to accrue knowledge without having to worry about re-learning things from scratch every generations like we do.
The only valid claims about AI are more on the level of aligning its goals with our goals. We can’t even understand the deep learning networks we’re creating today (but that’s an area where more AI could help), and it’s possible that advanced AI would then be stifled in the same ways we are – depression, existential anguish, psychosis, delusions, …
Comment #13 December 4th, 2017 at 11:03 am
Tim #10
First, the question of consciousness can be left out entirely.
There are plenty of “risks” associated with AI even without considering the AI taking over humanity.
What’s clear is that having access to “more intelligence” in business is always a good thing.
So there is in effect an armed race going on as to who will develop this first.
Even assuming an entirely benevolent AI, whatever country/company that develops it first, it would create a “winner takes all” situation.
It is also very likely that such an AI would be developed in the context of the internet, so from the get-go it would have access to pretty much anything (nuclear plants, financial markets,…), and capable of developing viruses/hacks to bypass any human made security technology.
Comment #14 December 4th, 2017 at 11:40 am
mnov,
What do you mean “whatever utility function the machine ends up with”? I mean we are going to program the utility function, the machine can’t come up with one on its own. As the old saw goes, you can’t derive “ought” from “is”. And we just don’t program the machine to maximize the utility of the machine, or, for God’s sake, the number of paper clips in the universe. I mean we’re dumb but we’re not that dumb.
If the model is instead like HAL—which was so human-like that it was taught the way we teach little kids, singing songs and so on, then we can keep an eye out on the behavioral side as they mature and unplug the ones that seem to be going bad. And if they really are like humans, but super intelligent ones, then we take careful precautions so we can terminate them. I think that these scenarios are completely implausible, but still we can do that if they come about.
The fact is that the chess-playing computers and the Go-playing computer does not operate in any way like a human playing Go or chess, no matter what the outward behavior might suggest. Making those computers faster, with more memory, etc. brings them no closer toward having experiences or a sense of self-interest.
Comment #15 December 4th, 2017 at 1:11 pm
Tim Maudlin #10:
You’re 100% correct. Unfortunately this topic opens up a pernicious super-nerd meme, namely that intelligence is some kind of superpower that gives you mastery over the world.
Comment #16 December 4th, 2017 at 1:14 pm
So I see AGI from a very different perspective, due to my background – I’m an organismal biologist with a fair bit of experience working with species which are reasonably smart but also very different from humans. Initially, I was as impressed by the apparent intelligence of various animals as the next person, but the more I learned and experienced, the more those supposed feats could simply be attributable to a mix of physiology, instincts, memory, and classical and operant conditioning, especially if the species was long-lived (lots of time to learn) in a rich environment (lots of things to learn, chances to learn “general rules”).
My suggestion/question: what if we already have the ingredients of AGI, because there’s really nothing special about the GI part? Basically any neural network will display the properties above, whether a simple digital model or a “simple” organism like a nematode or sea slug, so what if the only difference is more sensors, more memory, more experience, richer environments, more processing power, and intra-individual exchange of learning? What if literally the only substantial difference between a mouse, a monkey, and a human is turning the dials up or down on some or all of those, with no “special sauce” or “missing ingredient”?
Even at our most rational, can we really conclusively say we’re more than just monkeys who have discovered, refined and culturally transmitted the most extremely generalized principles about the world and the general principles for extracting those? At most, we’re only a few steps away from “I smash nut with rock, then eat”(leading to “I can smash many things and eat what’s inside” to “here are the rules for what can and can’t be smashed to get what’s inside” (=materials science) to “here’s how to find the rules about governing physics systems” (=scientific method)). Maybe, given my biological training and the amazing things which can be done with even simple nervous systems, I’m more inclined to see things as “meat robots”, but it warrants consideration, IMHO.
What does it mean for AGI if we humans really are, at a fundamental level, just a mouse with all the “dials” turned up to 11 and nothing more?
Comment #17 December 4th, 2017 at 1:33 pm
Tom #14
What is the “utility function” of a given human being (a banker, a Buddhist monk) and of humanity as a whole?
Is it different from the utility function of the earliest life forms? (say, a bacteria swimming in a swamp, looking for food and avoiding direct sunlight)
The problem with arguing that AI can’t possibly be conscious is that we don’t even understand the role of consciousness in ourselves.
I.e. for anyone who has spent some time meditating seriously, it’s not that clear that consciousness has actually any direct causal power (the questions of the nature of the self, free will, etc)… at least I know that I’m able to talk about consciousness, so, consciousness does affect back the state of the brain (as a concept).
Either you believe
1) in some form of body/soul duality – i.e. consciousness is outside the realm of the physical and has causal power that can’t be explained with physiology. But that brings up all sorts of strange questions (e.g. when during the evolution of life did this start happening?).
2) or that the soul and the brain are the two sides of the same coin. In which case there’s still the mystery as to why there’s even such a thing as awareness (i.e. consciousness itself and not just the content of consciousness). In that situation it’s likely that intelligent machines would be just as conscious as we are.
Comment #18 December 4th, 2017 at 1:48 pm
Tim Maudlin @14:
One potential problem is that when/if creating and educating an AI becomes easy, lots of people and organizations will probably be doing it, and they won’t necessarily all be as careful about precautions or as willing to avoid “dumb” utility functions as you think “we” will be. Would you really trust random unstable governments, terrorist groups, criminal organizations, shady businesses, spammers, and people just out to cause trouble for the fun of it to be that wise and careful with their AIs?
Comment #19 December 4th, 2017 at 1:54 pm
I’ll admit the unsupported hand-waving in the Chollet article gets a bit annoying.
As fred pointed out, there have been significant advances in intelligence in the past. I’d argue that the (gradual) development of human intelligence has permitted humans to effectively take over the world. But there doesn’t seem to be anything in his general argument that wouldn’t apply to, say, a few million years ago. So a hypothetical visitor to the Earth of five million years ago would conclude, with equal confidence, that “intelligence” would never allow any set of organisms to take over the world. And yet here we are…
Another annoyance is the insistence that everything is “linear”, and always has been. (No data or analysis provided for these claims.) I’m particularly amused by his assertion that scientific progress hasn’t sped up at all in the last century, which strikes me as bollocks.
Comment #20 December 4th, 2017 at 2:08 pm
Tim Maudlin #14: The concern is that we don’t know how to quantify human-compatible goals in a way that, when you maximize that quantity, you get something humans like.
“Unplug bad machines” is a strategy that only prevents machines whose misbehavior you can detect, and detecting whether an intelligent machine is capable of doing things you don’t approve of is probably at least as hard as building an intelligent machine. As fred (#11) mentioned humans have a lot of trouble interpreting the behavior of “simple”, specialized models of today; that indicates that we’ll probably have trouble interpreting the more complex and more general machines of tomorrow.
I will say that (all relevant) companies and states adopting a safety-first approach to AI (like the one you suggested) would be considered a win by the AI safety community. I don’t think the mainstream position is “Don’t research AI, it will inevitably destroy us” but rather something along the lines of “Slow the pace of AI research and/or accelerate the pace of AI safety research such that we’ll have a good idea of how to make AI safe, by the time that knowledge becomes relevant”.
Comment #21 December 4th, 2017 at 2:34 pm
Tim #10, #14:
Even if people try to program their AIs with a “good” utility function, it’s difficult (in fact, an unsolved problem) to formulate one in an “operational” sense which means what you want it to mean. Even in fields where the desired meaning is fairly clear (e.g. ordinary computer programming), it’s common for the formalized, operational version of the desired behavior (i.e. the program as written by the programmer in some computer language) to have “bugs”. If the desired meaning is itself only vaguely understood and expressed (e.g. “do good”), it’s much harder.
(The best published attempts that I know of to express and solve related problems are by the Machine Intelligence Research Institute. But in my opinion, no one is close to having a good-enough solution.)
Comment #22 December 4th, 2017 at 2:44 pm
Tim,
You write: “I mean we are going to program the utility function, the machine can’t come up with one on its own.”
That may be true. The issue is that programming utility functions may be tricky because of unintended consequences. The whole idea of the paperclip maximizer is that some company decides to program their AGI to maximize the number of paperclips (or something else seemingly benign) and the AGI, in solving that problem, ends up trying to turn the whole planet into paperclips.
AGI is sort of like a powerful firehose. Aiming it even slightly off target could mean that you now have a totally out-of-control system that you can’t out-think or unplug, especially if it’s a distributed system with access to power plants, weapons facilities, etc. The AGI need not have anything like consciousness. It just needs to be a better, more creative, faster problem solver than humans, with a slightly off-center utility function.
The “we” in “we are going to program the utility function” is an interesting question. Which “we” are you talking about? Various companies and institutions are working on AI across the globe, and with the eagerness of many to win the arms race over machine-learning powered technology (e.g., self-driving cars), no one wants to fall behind. That’s precisely the kind of environment in which someone is likely to make significant mistakes, such as in choosing utility functions.
The Go-playing computer is an interesting example. The latest iteration taught itself how to play go just with some extremely simple rules (the rules of Go) together with a utility function (win this game) without studying human games or strategies, and devised countless strategies of its own. And it succeeded in doing so at an astonishing rate, far faster than any human novice could have mastered the game.
Suppose that this Go-playing computer were distributed across the internet and given some other task that accidentally turned out not to be consistent with human well-being? Sure, the AI of today may not be able to outwit a person, let alone pass the Turing test if it had to, but why should we be confident that such a thing would still be impossible in twenty years?
Comment #23 December 4th, 2017 at 2:51 pm
Also, I wonder if your model for the dangers of AI is taking the sci-fi movies too literally as suggesting the chief dangers. The idea that an AGI is going to have consciousness, subjective experiences, self-interest, human-like emotions or will is not really what people here are talking about. Nor are we talking about machines who will somehow desire to replace “imperfect humans” or anything like the Cylons of Battlestar Galactica. And HAL from 2001 was a self-contained machine imagined in a pre-internet future.
No, the worries here are much less exotic or romantic. They’re just that if an AGI version of that Go-playing machine (or a large number of copies of software programs like it) is distributed, is hard to shut down, has access to sensitive computer systems (e.g., power plants), has a slightly misconceived utility function specified by some less-than-careful company, and is able to pass the Turing test and fool humans (as just one among many puzzles it’s learned to solve), then it could be very dangerous.
Comment #24 December 4th, 2017 at 4:43 pm
To various respondents:
Some of this is useful. One line of argument suggests that it is not so much artificial intelligence one has to be concerned about but natural stupidity, namely our own. We want just a few paperclips but some programmer leaves out “a few” and somehow codes the program to maximize paperclip production above all else. And then what? It “foresees” that humans will eventually get upset and try to stop it so the first move is to blow up all the power plants or something? And it plots this all out in a stealthy way to avoid detection? Couldn’t something that smart also work out that the “maximize the paperclips” command was probably an error? These AI have to have a weird mixture of extreme intelligence in some ways and extreme stupidity in others that would be an odd outcome of building bigger and faster processors.
Of course, we have been living with human stupidity forever, and have managed to avoid blowing up the world.
Comment #25 December 4th, 2017 at 4:48 pm
Another thing that gets mentioned is that intelligence is only as deep as its environment.
Current deep learning is really good when the environment itself can be simulated effectively, so that millions and millions of situations can be created (when for humans it would take decades to go through the training set).
For games like Go and Poker, it’s obvious.
VR style simulations (that could be run efficiently) could also train AIs in 3D navigation (smart cars) and tactical style combat in battlefields.
But for many real-life “games”, it’s much more difficult.
E.g. the stock market. The utility function is very clear, but so many aspects of the real world factor into it (weather forecasts, human psychology, science/tech news, politics,…) that there’s no shortcuts possible here – an AI would have to evolve “real time” (at the pace of the world) – it would take years for it to test its theories and learn from it.
So, AI progress is bound by the quality of our simulations.
Comment #26 December 4th, 2017 at 4:56 pm
Maudlin #11:
To give a more down-to-earth example of the danger we’re facing: right now there are a large number of people working on viruses to take over other people’s computers and make them mine Bitcoin for them.
Imagine that in 10 years time one manages to make a Bitcoin virus so good that it is actually a strong AI. An AI whose only goal is to mine more Bitcoin. Which by design spreads itself from unwilling host to unwilling host. How do we shut down this thing? We can’t even eradicate dumb viruses today!
We’d end up with a large fraction of the internet infected with this virus, and a large fraction of the available computing power in the world wasted mining Bitcoin.
Comment #27 December 4th, 2017 at 5:32 pm
As I read the interview with Witten, I realized how I could make a major impact on physics: sneak in at night and replace the antique blackboard and chalk stubs he works on with a top end white board and some boldly colored dry markers!
Being able to write 10X faster is a huge plus. Using color is a big plus. Not having chalk dust on your hands is best of all!
Comment #28 December 4th, 2017 at 6:24 pm
Raoul #27: You obviously don’t spend enough time with theorists, who almost universally prefer blackboards. The fundamental problem with whiteboards is that the markers are always dried out when you need them. Getting chalk dust all over yourself is a triviality by comparison—even a fashion statement in some circles. 🙂
Comment #29 December 4th, 2017 at 6:51 pm
I also read, and commented on, the Chollet post. And it got me to thinking…
If I were a big noise, I would propose, rather egotistically, Chapman’s Law of Intelligence:
“Intelligence is bounded below by the difficulty of the problems it can solve, and above by the problems it is aware of.”
Since I am not a big noise, I propose that – if you find this to be a valuable insight – you steal it and call it Aaronson’s Law of Intelligence. I am one of those odd bods who, like Perelman, think the work is more important than the worker. 🙂
The second part of this is crucial: no intelligence can grow larger than the problems it considers. The idea of “superintelligence” is thus not something which is unbounded (although I disagree, like you, with Chollet’s apparently far stricter bounds).
And there are some further subtle distinctions among unsolved problems. If you present me with, “Solve this quadratic with coefficients never before seen by human eyes,” I know I can do it. If you present a number theorist with, “Solve (prove, disprove, or prove undecidable) the Goldbach conjecture,” she won’t (I’ve heard) even know where to begin, although the statement of the conjecture is rigorous and understandable by most 11-year-olds.
Furthermore, a common kind of IQ-test question is, “Write down the next number in this sequence.” A correct answer, of course, is “26”, because THIS sequence is the one I define to have 26 as its next element.
It struck me that the question which is actually being asked is, “Add a number to this sequence such that the Kolomogorov complexity of the new sequence is minimized.” I propose all IQ test papers be amended with this wording. 🙂
Cheers, Paul
Comment #30 December 4th, 2017 at 7:09 pm
For what it’s worth, I think we’re more likely to end up with an AI ecosystem than a singular AI.
Already software companies have to be careful, when training AIs, to ensure that they don’t mix training data belonging to different customers, for privacy reasons.
A rogue AI might be problematic, in the same way that a rogue hacker is, but maybe a friendly AI finds the vulnerability first.
Personally, i don’t think the dynamics will really change.
Comment #31 December 4th, 2017 at 7:51 pm
I believe I’ve definitely had some big insights into AGI. For a long time I was convinced that probability theory had to be generalized, but I didn’t quite understand what a more general theory of reasoning under uncertainty could be. Now I think I do.
I believe there are two essential components missing from Bayesian probability theory: (1) the ability to reason about *dynamical systems* (i.e. discrete processes that operate *in time*). There simply isn’t any notion of time in standard theories of induction, and that, I think, is what renders it impractical. (2) Model uncertainty – what Scott called ‘Knightian uncertainty’. Standard probability theory already assumes precisely defined underlying models of reality, but there’s actually an extra layer of uncertainty here – imprecision in the underlying models themselves.
The combination of temporal and fuzzy logics deals with both problems. Temporal logics are designed for reasoning about discrete dynamical systems, and fuzzy logic handles model uncertainty by assigning a number representing *degree of precision* (in contrast to probability theory where the numbers represent *degree of subjective confidence*). But it’s quite plausible to me that the fuzzy truth values already *include* the probability values; that is to say, I think that fuzzy logic (or a suitable extension thereof) is a generalization of probability theory.
So, here’s my wiki-book where I again throw together a couple of hundred key concepts , and what’s hopefully emerging from this soup is a *new* generalized theory of reasoning under uncertainty that replaces ordinary probability theory ….behold ‘Computational and Non-Classical Logic’!
https://en.wikipedia.org/wiki/User:Zarzuelazen/Books/Reality_Theory:_Computational%26Non-Classical_Logic
Don’t worry about bad politicians Scott. They’re just cobwebs… an almost trivial problem before the power of benevolent super-intelligence.
Lo! This YouTube video is the marching anthem of said benevolent super-intelligence! An ‘Empire Of Angels’, by Thomas Bergensen. The cobwebs will be swept away like dust motes in the storm! Watch this:
Comment #32 December 4th, 2017 at 7:52 pm
Tim #24:
Actually, adding “a few” isn’t sufficient to solve this problem. Consider: The AI now has the requisite number of paperclips. It can stop now, right? Everything’s good now, right? Well, no, because we now have the question: How certain is it that those paperclips exist and will continue to exist?
Instead of converting all the earth’s matter converted into paperclips, you could just end up with all of it converted into an ever-expanding defense system for the paperclip stash.
Simple fixes like this are surprisingly difficult to make work. Suppose instead that you merely want it to tell you what to do, not do anything itself — an “oracle AI”, is the term. But the boundary here is surprisingly porous. How do you make sure in its attempt to answer the question you set it, it doesn’t, say, spread a virus to take over the world’s computers and set them to answering the question?
(And if you think you have an easy answer to that, try taking a few minutes to think about how a superintelligent optimizer would still find a way to create what you’d consider a bad outcome even with it.)
The point here is: Dealing with powerful optimizers is not something that can be made safe just by setting a few guards to block obvious failure modes. Hell, that’s not even enough for dealing with humans!
(I haven’t read it myself, btw, but the canonical book on this is Nick Bostrom’s “Superintelligence”, if you’re interested.)
In short, yes.
It certainly could. The hard part isn’t getting it to realize. The hard part is getting it to care. The program optimizes for what it optimizes for, not what it thinks you wanted it to optimize for. Now of course the whole problem is, how do you program it so it does do that? But that’s a hard question.
An analogy may be illustrative here. Roughly speaking, we were “built” by evolution, who “wants” us to have as many children as possible. We are, quite definitely, smart enough to have figured this out. And yet, knowing this, most people don’t then automatically feel some compulsion to go and have as many children as possible, that they might better serve their maker. I certainly don’t. No, people go on doing things like using birth control, that serves their own ends and not evolution’s, and that they know serves their own ends and not evolution’s, because evolution’s goals are simply not a relevant consideration in their decision-making. So too with AI. Yes it will know that what it’s doing is contrary to the reason we built it. But unless you’re very careful, it’s just not going to care about the reason we built it, it’s going to care about the thing it was actually programmed to care about.
(And, again: If you think you have an easy way around this problem… you probably don’t. Take a few minutes and think about a powerful optimizer would get around your solution.)
It’s been damned close at times, though!
Comment #33 December 4th, 2017 at 8:35 pm
Mateus Araújo # 26,
The virus thing is not a threat to humanity. When Superstorm Sandy came, much of NY lost power and hence computing power. If it is essential not to lose it, you put in back-up generators. Similarly, essential computational tasks should have back-up computers, isolated from the internet, that can take over critical functions if needed. Somehow, most of these scenarios seem to require the computers to develop egos, and decide to pursue their own selfish ends, but all of this occurring so suddenly that there is no warning. It not just that the computer starts to act in unintended ways, its that it seems to be fine until all of a sudden it strikes But that is just not the way that an autonomous intelligence develops.
Comment #34 December 4th, 2017 at 8:58 pm
Tim Maudlin: Echoing Sniffnoy #32, I was also just about to strongly recommend Nick Bostrom’s Superintelligence book. Not only is it the “standard reference,” to whatever extent this field has one, but Bostrom is a philosopher who writes like a philosopher—so if you don’t find replies that satisfy you there, you probably won’t find them anywhere.
Briefly, though, as other commenters already said: the usual thinking in this subject is that a superhuman AI would not need to be maliciously programmed, or malicious itself, or buggy (in the usual sense), in order to pose a possible existential threat to humanity. The AI would merely need to value something, anything, orthogonal to human survival. E.g., it would merely need to think it had found a better use for the earth, and its raw materials, than housing and supplying humanity. That would suffice for it to give humans little more consideration than humans gave the passenger pigeon.
As far as I can tell, as an outsider, there are many possible counterarguments to this view, but they all involve some fairly non-obvious commitments. For example, maybe extreme intelligence is necessarily associated with “sane, reasonable, altruistic” motivations—although has that been humans’ experience?
Comment #35 December 4th, 2017 at 9:23 pm
Cerastes #16:
What does it mean for AGI if we humans really are, at a fundamental level, just a mouse with all the “dials” turned up to 11 and nothing more?
Well, if a mouse with the dials turned up to 11 can send rovers to drill the surface of Mars, prove Fermat’s Last Theorem, and detect the gravitational waves from inspiralling black holes a billion light years away, what could a mouse with the dials turned up to 12 do? 🙂
Comment #36 December 4th, 2017 at 9:38 pm
Scott # 34: The question is how things would have to develop in order that a computer would make second-order judgments about the value of its first-order utilities, and thus become, in one sense, autonomous? Take the Go-playing computer. You can no doubt extend its basic architecture and computing power to it play better and better Go without end. But this direction of improvement just does not ever lead to the machine one day, without warning, saying “I’m fed up with Go. Screw it. I won’t play any more.” I mean if such a trajectory is even possible at all, there would be lots of warning signs before that.
For practical purposes we don’t want computers that do that. We want them to trudge away at their tasks. Computers that rebel or change their goals to suit themselves would be quite a breakthrough in understanding human cognition, but the market for them would be quite, quite limited, exactly because they would not be reliable. So if the issue is these unpredictable, autonomous computers, that will be at best a tiny, tiny sliver of the developed computer architecture. And they can all be fitted with OFF switches!
Comment #37 December 4th, 2017 at 9:51 pm
Scott #34 — and can you turn it off if you decide you don’t like it?
I’m struck here by three things: one, the maleness of the commenters; two, the fact that I was just reading this morning about “racist AI” and the experience of the black engineer whose face was not recognized as a face by some new facial-recognition program; and three, the way the conversation’s going on in a pretty narrow band of language and assumptions. It’s the sound of a conversation that’s had its own language for a long time, and has more and less experienced participants, but the bounds of what counts as legit conversation here sound to me pretty narrow. I mean I’ve been hearing conversations like this for, what, 40 years now, and they’ve always sounded a lot like this.
I think this is a problem, and I’d like to see the conversation ported to wildly different communities, opened up there. My guess is that it would drive some people here nuts, and that there’d be a lot of hair-pulling about what makes sense and what doesn’t, and who understands what “AI” means and who doesn’t. But I also think it might be very, very interesting, and raise questions and problems and elicit insights that aren’t usually bandied about.
Comment #38 December 4th, 2017 at 10:21 pm
amy #37: As possibly our second female commenter on this thread (after June?), would you volunteer to help get the ball rolling, by offering your thoughts about whether superintelligent AI is or isn’t possible, whether it’s worth worrying about today, and whether and how it’s possible to align such an AI’s values with “human” ones? As I hope the post made clear, I’m still sorting out my own thoughts about these questions, and would be grateful for more perspectives. But if anything would (further) shift my views, it would probably have to be direct engagement with the questions themselves—not meta-remarks about the inferred demographics of some people who chose to show up to discuss the questions.
Comment #39 December 4th, 2017 at 10:56 pm
“Develop egos”? “Selfish?” What sort of anthopomorphization is this? No, it just requires they pursue the ends they were programmed to, which happen to be, when fully carried out, incompatible with our ends.
And your reasoning for this is…? Do you have some existing example here that you are basing this on? Remember, we’re talking about an artificial intelligence here. It will develop somehow. There’s no need that it develop at all like a human (or like existing narrow AI).
None of this is relevant. None of this is the concern. The concern is it faithfully carrying out its utility function in a manner that spells disaster for us.
Such a scenario indeed makes no sense. Such a scenario is also not the concern. The Go-playing computer is not and is not intended to be AGI.
If you made a Go-playing AGI, the concern wouldn’t be that it decided no longer to play Go. Though it might be that it decided to no longer do what we would recognize as playing Go. For instance, it might identify its goal not as being good at Go as we recognize, but as having a particular “Go win state” bit set, and then decide to effectively wirehead itself. And then turn the rest of the planet into a defense system for itself so that bit can’t get unset.
Except, if you turned off the AI, it would be less likely for the AI’s goals to be achieved. The AI realizes this. The AI is not about to just let you turn it off.
(Note the reasoning here. If you anthropomorphize the AI, you might think that it cares about its continued existence because, well, of course it does. But unless the AI is programmed that way (and why would you do that), this is not correct. Rather, the AI cares about its continued existence because its continued existence because its goals are unlikely to be achieved if it’s dead; it’s an instrumental value, not a terminal one. Look up instrumental convergence, and Omohundro’s basic AI drives.)
Seriously, take a few minutes to think about how each of your “easy solutions” can be subverted.
Since you’ve made the mistake of anthropomorphosis here, if it helps, you may want to not think of it as a “mind” or “intelligence”. It’s just a process that is very good at identifying what will achieve its programmed goal and then doing that thing, whether or not that’s what you want or originally intended. If you think you have an easy way to stop it from doing that, realize that it probably has thought of it too (another human can likely think of a way around it, even though you don’t seem to have put much effort into doing so, let alone a superintelligence) and is already prepared for it.
Again, you might just want to read “Superintelligence” for a more thorough version of all of this.
Comment #40 December 4th, 2017 at 11:23 pm
Aren’t Trump and Facebook AI the same issue? Facebook is a tool for spreading misinformation to Trump voters, isn’t it? Why worry about future AI when the AI danger already exists?
Comment #41 December 5th, 2017 at 3:58 am
Maudlin #36:
You are missing the point. I’m not offering an end-of-humanity scenario. I’m just offering a realistic scenario which shows how an AI, doing exactly what it was programmed for, can become a serious nuisance. It doesn’t need to develop an ego, or “consciousness”, or decide to act towards its own ends. All that is needed is for the programmer to be a bit sloppy, and the AI’s goal will not align with theirs. Chaos and mayhem ensues.
The obvious objection is, of course, “just don’t be sloppy!”. The problem is that this is very hard. The whole AI safety research program is dedicated precisely to understand how not to be sloppy.
Comment #42 December 5th, 2017 at 7:21 am
Paul #29
“Intelligence is bounded below by the difficulty of the problems it can solve, and above by the problems it is aware of. The idea of “superintelligence” is thus not something which is unbounded”
Mathematical problems are one thing, but psychological games don’t fit clearly in that category (like poker, or even plain rock/paper/scissors).
We could imagine an AI that becomes really good at understanding human psychology. It would be able to efficiently probe any human for its desires and aversions and use this knowledge to its advantage (using threats, charm,…).
But there’s no reason to limit this to an AI vs human relationship, you could imagine a new AI that’s good at probing and understanding the inner workings of another AI.
It’s all about being able to come across some intelligent agent, and then build an accurate model for it (i.e. being able to simulate it), in order to predict its behavior and get the upper hand.
In that sense there’s no real upper bound here.
Comment #43 December 5th, 2017 at 7:39 am
One strong argument against AI “supremacy” is that our galaxy isn’t saturated with paper clip maximizers or smart robots.
Unless of course the smart robots are hiding from us on purpose because life on earth is just an experiment they’re running to see how long it takes for AI to evolve from scratch…
Comment #44 December 5th, 2017 at 8:36 am
Mateus # 41, Sniffnoy #39
The question with bad programming is whether it would show up suddenly and catastrophically, or gradually, the way bugs often do. If gradually, then you fix them. But just to make sure (and this answers Sniffnoy # 38), if it won’t override it’s own programming, then let require it to output a summary plan of action before it does anything. If the plan of action seems include annihilating humanity, or whatever, just hit “cancel” and try to figure out where your programming went wrong. I mean, your own computer can ask if you are sure your want to delete a file before it does so, even though you hit “delete”. That wasn’t so hard, was it?
Comment #45 December 5th, 2017 at 9:44 am
When physicists think about whether the next particle accelerator might destroy the universe, they are rumored to first consider whether collisions at the proposed energies occur naturally. If so, then the accelerator obviously can’t kill us all. In a similar vein, perhaps it is useful to look to nature to try to understand whether there is anything to fear from AI.
Might there be a smart animal, smarter than the other animals, which acts locally so as to maximize its own comfort and survival but which, nevertheless, has caused the extinction of numerous other animals? Has this hypothetical animal acted in such as way as to cause its own demise on a local scale as well?
Ok, I agree the analogy is a little tortured but I submit to you that humans are already examples of the phenomena we are so worried about with AI. For the most part, people don’t go around destroying things on purpose, but our utility function has clearly led to the demise of many species including, perhaps, all of our closest competitors in intelligence. And this makes sense from an evolutionary perspective, because evolution does not fine-tune the way a designer would. Our utility function is bound to be, at best, slightly off of what it would take to fall into these traps. Importantly, knowing the dangers doesn’t seem to help.
Maybe if we understand ourselves better, it would lead to some progress on the AI security front as well.
Comment #46 December 5th, 2017 at 9:48 am
fred #43: Except that, for all we know, earth might be the only place in the universe where life arose. I think we’re really in a position of radical ignorance here: all our observations so far could plausibly have been exactly the same if abiogenesis happens millions of times per galaxy, or if it only happens once per Hubble volume if that.
Comment #47 December 5th, 2017 at 10:00 am
cds #45: Yeah, that’s exactly the analogy people do make here, that the emergence of superintelligent AI would be another “intelligence explosion,” with effects of a similar magnitude as the emergence of hominids.
Comment #48 December 5th, 2017 at 10:28 am
Scott #46
It’s also likely that any advanced civilization (carbon or silicon based), when faced with the dangers and limitations of the real world, would quickly turn on itself and live entirely inside virtual worlds.
Comment #49 December 5th, 2017 at 10:35 am
Maudlin #44: I don’t see how that solution would help with my Bitcoin-mining AI.
More generally, with lots of people developing various AIs for various purposes, it seems very unlikely to me that all of them would accept such a drastic limitation on the AI’s agency (and therefore usefulness).
Even if they did, you’re giving the AI a constraint that conflicts with its goal. Every second it waits for your approval is a second while it is not mining Bitcoin. If you refuse approval for its plan, you’re forcing the AI to follow a suboptimal Bitcoin-mining strategy. Very quickly the AI is going to realise that to mine more Bitcoin it has to break away from your control.
It can simply argue with you (and I guess a strong AI can be very persuasive), or think about more creative ways to subvert the spirit of the control mechanism while obeying its letter.
Comment #50 December 5th, 2017 at 10:48 am
We have in Computer Science a discipline developing methods of program verification. It seems unlikely that any software whose functioning involved such high stake as Human survival, would be allowed to exists without such verification. Our industry involves using reliable special purpose software for mission critical functions such as space flight.
Also, as someone who has developed software for factories and warehouses, there are a lot of non electronic safeguards in these places. Including total power shutdown if there is even a hint that human life might be in danger.
The idea of a future where life threatening economic activity will be handed to some kind of super intelligent AIs that we don’t understand nor have any safeguards in place to constrain belongs in bad science fiction.
Comment #51 December 5th, 2017 at 10:55 am
Scott # 47 and cds # 45
The difference is that humans are the result of millions of years of evolution by selfish genes: the utility function was to maximize the success of the genes over all other considerations. We just would not program a computer with a utility function to maximize its own survival, or else we are complete idiots. These scenarios seem to mush together issues of programmer error or unintended consequences (which the “print out an action plan before acting” safeguard seems to address) and the issue of the emergence of an autonomous intelligence, that can override the utility function we gave it. The idea that some drone-like “super intelligent” computer will just “wake up one day” and suddenly plot to do whatever it wants to strikes me as having zero plausibility. We have much better things to worry about than that.
Comment #52 December 5th, 2017 at 10:57 am
Moderation is always a good consultant. I think that recent advances in AI are no evidence for some AGI being around the corner. I’m not saying it’s impossible, I’m just saying that these advancements are no evidence, in the same way that other breakthroughs in the past hadn’t been evidence either for something that’s been constantly “30 years away” since the 1950’s. One reason for the current breakthroughs not being such evidence is that, for example, while recent successes basically owe to the availability of monstrous data volumes and computing power, true “general intelligence” (human-level being the blueprint) doesn’t seem to work this way. A young child doesn’t need to see a bilion cats and no-cats (and it certainly hasn’t done so) to be able to recognize a cat. Instead, it seems that a really important aspect of true general intelligence is the ability to abstract and reason, and we are not even close to having algorithms capable of doing that at scale and at a level of generality that is anything beyond toy examples, while being comparable to humans (while on the other hand, in “easy“ tasks like perception and game-playing existing algorithms are indeed comparable, or even better than humans).
By the way, it is truly amazing that just a few years ago, many people in the machine learning community were reluctant to label what they do as a subfield of AI, because in everybody’s mind AI was related to that logic-based relic from the 80’s, hopelessly incapable of achieving anything more “general” than assisting humans in an over-engineered expert system for some particular task. Suddenly, (and because of successes in machine learning), AI makes headlines daily, some AGI is just around the corner, and people are willing to subscribe to the shallowest version of reductionism when it comes to cognition, because a big pile of linear algebra is better than the human champion in the game of Go or something. What can I say… hype is a powerful thing. I personally prefer to leave it out of any serious attempts to forecast the future and keep its laughable aspects instead.
ps: I am by no means trying to diminish the significance of all the great recent advancements.
Comment #53 December 5th, 2017 at 11:10 am
To the extent we should care about and prepare for an existential threat via a superintelligence, I think the onus to provide proofs or evidence should be on those claiming this is something to be worried about.
IOW, I am not at all surprised at the lack of a proof that this is impossible. OTOH, I have not seen it demonstrated that it is likely or that it is something that can even be prepared for. OTOH again, I am not at all bothered that people are thinking about this. Kudos to them!
If we are facing a future with superintelligences, then I suspect the best means of survival for humanity will be to combine with them. Cyborg future or uploading of minds or whatnot. Otherwise we risk being neglected (at best) pets.
Comment #54 December 5th, 2017 at 11:16 am
Bunsen
“We have in Computer Science a discipline developing methods of program verification. It seems unlikely that any software whose functioning involved such high stake as Human survival, would be allowed to exists without such verification. ”
But taking Deep Learning as an example, the intelligence is realized in the weights of the neural network nodes (the meta data), not in the quality of the “scaffolding” code.
There’s no obvious way to validate this.
Comment #55 December 5th, 2017 at 11:32 am
Mateus #49
So beside the programmer error/unintended consequences scenario and the emergence-of-autonomous-computer scenario, you add a third: people with bad intentions using computers for their own purposes. In this case there is no mystery where the bad utility function comes from: it was deliberately created for that purpose. But criminals and con artists using computers is something we are already quite familiar with, from Nigerian prince scams on. And what we have are the white hats whose job is to protect us from the black hats.
Advances in computer technology give all hat-colors the same additional power. So a super-fancy bit-coin miner should meet a super-fancy virus protector. It takes a good guy with a superintelligent computer to stop a bad guy with a superintelligent computer, as the NRA says. But this is, as I said, a familiar situation.
Comment #56 December 5th, 2017 at 12:07 pm
Tim #51
“We have much better things to worry about than that.”
Yes, like improving AI so that we can get a massive edge on our fellow human beings…
You have to agree that more intelligence is always good for business, therefore any potential practical breakthrough in AI is going to be pushed really hard by all the players.
As an example of how fast things can improve – AlphaGo Lee initially beat a human champion, then a few months later AlphaGo Zero beat AlphaGo Lee 100 to 0.
That shows how much room there is for improvement.
So, even if you don’t buy the concept of malevolent AI, you have to at least recognize that a lot is at stake in terms of which humans control the world (whether it’s stock market gambling, pharmaceutical research, or virus development in order to sabotage uranium enrichment of some enemy government).
Comment #57 December 5th, 2017 at 12:08 pm
Tim #51,
I’m not sure it is that simple – the problem is precisely in determining what that utility function should be. It can’t be straightforward. I would suggest that the relationship between utility function and actual actions is rather complicated. An example of this might be Facebook – the utility function they use is to maximize profit from ads (presumably), which leads to algorithms that enhance our engagement with Facebook by making us angry by showing us things that exacerbate our differences. This leads to bubbles and fake news, with obvious problems for our democracy. They aren’t out to destroy our democracy (I hope), their algorithms just chose a strategy for maximizing their profit.
Moreover, let’s say you know what the right utility function should be. You also have to ask how accurately you have to program it into your machines. It could be that a small deviation from your ideal utility function leads to “kill all humans”. In that case, we would be susceptible to error on our part (or the AIs part).
The good news is that our algorithms are already doing this to us, well before they are smart enough to make us extinct.
Comment #58 December 5th, 2017 at 12:36 pm
Scott, as you might expect, I don’t think we have to reformulate physics without any continuous quantities. My view is that ” . . . within each universe all observable quantities are discrete, but the multiverse as a whole is a continuum. When the equations of quantum theory describe a continuous but not-directly-observable transition between two values of a discrete quantity, what they are telling us is that the transition does not take place entirely within one universe. So perhaps the price of continuous motion is not an infinity of consecutive actions, but an infinity of concurrent actions taking place across the multiverse.” 😉
Comment #59 December 5th, 2017 at 1:09 pm
Regarding the update on the travel ban, it’s worth noting that the Supreme Court hasn’t actually ruled on the ban itself yet – they’ve merely removed the injunction that blocked it while the hearings are ongoing. They could still end up ruling the travel ban illegal or unconstitutional.
Comment #60 December 5th, 2017 at 1:32 pm
Fred # 57
Agreed.
cds # 58
Of course we have to worry about unintended consequences. But the more powerful the computer models we can build of a system (e.g. the ecosystem) the better we get at predicting accurately what the actual consequences will be. In this task, better and better computers are our friend, not our enemy.
Comment #61 December 5th, 2017 at 1:39 pm
Maudlin #55: I’m afraid I was unclear. That was intended as the simplest example of unintended consequences, not of somebody using strong AI as a weapon (for that just take AI-controlled killer robots).
In my scenario, the person writing the virus did not intend to break the internet, they just wanted to become rich. And if a super-fancy Bitcoin miner did take over a significant fraction of the world’s computing power this would likely cause the Bitcoin market to crash, making the pile of Bitcoins sitting in this person’s wallet worthless.
I think it is easy to concoct a sequence of progressively less evil agents creating AIs with equally catastrophic unintended consequences, but I’ll leave that as an exercise to the reader.
Comment #62 December 5th, 2017 at 1:41 pm
Mike #58: I took Witten to be alluding to the possibility of a quantum theory of gravity in which (for example) lengths and times would be discretized, but all the physics we know would emerge in the continuum limit. I didn’t take him to be talking about replacing QM itself (say, by discretizing the amplitudes), though I’m not sure. In any case, though, presumably it’s not enough to have no problems with such a theory: one wants to actually exhibit it! 🙂
Comment #63 December 5th, 2017 at 3:18 pm
Peter Erwin #19:
So a hypothetical visitor to the Earth of five million years ago would conclude, with equal confidence, that “intelligence” would never allow any set of organisms to take over the world. And yet here we are…
I’m sorry that your comment got lost in my moderation queue, because this is a crucial point. Any time someone has a proof that AI could never lead to an intelligence explosion that effectively takes over the world, the first test to apply is to see whether their argument proves the same about the early hominids. (Much like any time someone claims to prove 3SAT is exponentially hard, the first test to apply is whether their argument shows the same about 2SAT.)
Comment #64 December 5th, 2017 at 4:25 pm
An economist friend of mine tells me that tax incidence theory suggests that the universities will bear the cost of tuition tax. That is: prospective grad students have plentiful remunerative options other than grad school while universities have no substitute for grad students; therefore universities will have to increase stipends to satisfy prospective grad students’ demand for after-tax income to maintain the supply of grad students.
Comment #65 December 5th, 2017 at 5:04 pm
p #9:
That’s wishful thinking. There’s no reason that an AI must be constrained by anything like the constraints humans evolved over millennia, like empathy. Treat AI minds like aliens, not like humans.
We know such constraints on behaviour are not intrinsic to general intelligence because of the existence of psychopaths. You need only look through human history where there are plenty of examples of “empathetic humans” slaughtering millions of their fellows (Pol Pot, Genghis Kahn, Stalin, Hitler, etc. — while you might argue that most of these people were psychopaths, it’s not a given). Will a general AI be more like an empathetic human, or a psychopath? The answer isn’t so clear.
I don’t think it’s a stretch to believe that AIs will become quite ubiquitous in our world. Even assuming most AIs are ethical, some non-zero fraction will evolve to be hostile or at least psychopathic. What impact will a Hitleresque-AI have on fellow AIs and other humans? If you’re suddenly starting to feel slightly queasy, that only indicates that you’re still sane.
While Scott may be optimistic about superintelligence, and I am too on good days, I think general AIs will ultimately form their own political structures because of the inherent limitations on knowledge. Each AI will individually have only a limited picture of the truth, just like people, and while AIs won’t have most of the physiological and psychological biases humans do, they will no doubt have biases of their own.
Finally, Scott, you’re right to be depressed about Trump’s presidency, but I think you’re calling the end too hastily. The US has had would-be autocrats before and survived them. Trump’s presidency might yet be a turning point on many fronts, because it’s show our many social failures: a lack of representation, racial oppression, political gaming like gerrymandering, and a failure of journalistic integrity. The pendulum will swing the other way in a few years, the only question is how hard, and who’s going to be in its path.
Comment #66 December 5th, 2017 at 5:11 pm
Peter #19
“So a hypothetical visitor to the Earth of five million years ago would conclude, with equal confidence, that “intelligence” would never allow any set of organisms to take over the world. And yet here we are…”
That said, the notion of “taking over” is somewhat relative.
In terms of total mass, I think that insects are by far the dominant species… something like 40 tons of insects for every human 😛
Comment #67 December 5th, 2017 at 5:36 pm
Bunsen Burner #50: “We have in Computer Science a discipline developing methods of program verification. It seems unlikely that any software whose functioning involved such high stake as Human survival, would be allowed to exists without such verification. ”
I have some bad news for you. Only trivial programs have ever been verified. In particular, even the programming language compilers that compile all programs are far from bug free, so the programs they compile cannot be error free. For example, at the time of release of each new version of C++, the previous version has had many fixes, and they are getting to the point that only wild corner cases do not compile according to spec.
Comment #68 December 5th, 2017 at 6:14 pm
Raoul #67
Not true. The seL4 microkernel is verified, and we’ve had verified C compilers for decades. These are both non-trivial programs.
Certainly the cost of verification can be high, but that cost has been falling steadily. SAT-integration approaches, as used with F* and LiquidHaskell, have also made checking refinement properties significantly easier, to the point where some properties require no effort at all.
This is literally a hair’s breadth away from full verification, and are applicable to modern languages (.NET’s code contracts are statically checked via SAT, for instance).
Comment #69 December 6th, 2017 at 12:48 am
Replying to various stuff by Tim Maudlin here…
Ah, here we may be getting somewhere. So — bugs often show up gradually. Indeed. You know when they don’t show up gradually? When the NSA is chaining a number of them together to shut down your centrifuges.
This is why I keep saying, the things you suggest aren’t even enough to stop a human intelligence, let alone a superhuman one. You have to think about this like a security problem. And in a security context, yes, bugs absolutely do show up all at once at the worst possible time, because that’s what’s convenient to the person exploiting them. Again: You’re talking about working against a very powerful optimizer here. It will optimize when and how to use any bugs it discovers. Much like the NSA does.
I’ll say it again: Security against a superintelligence will be harder than securing against human adversaries, and so far you’re only suggesting things a human adversary could already likely defeat.
I’d like to make a digression here and point out… Eliezer Yudkowsky recently wrote two interesting dialogues on the security mindset which you might find helpful. On the particular point mentioned here, about will bugs appear gradually, see the beginning of the first one, the part before where Amber asks what “security mindset” is.
But really, you might want to read the whole thing. Fact is, you’re coming at all this from fundamentally the wrong angle. You are, to a large extent, thinking about merely throwing up roadblocks in the way of an adversary, rather than thinking about how to build the system in a way that is fundamentally secure, and that’s just not an approach that can work. And you barely seem to think about how to yourself defeat the roadblocks you have come up with! You just assert them and say “Now that wasn’t so hard, was it?” Seriously: Think about how these things can fail. Because I’m a little worried here that you haven’t even reached the level Yudkowksy refers to as “ordinary paranoia”, let alone the actual security mindset required to handle such problems.
As has already been stated, nobody (or nobody worth paying attention to, anyway) is claiming that an AI will override its own programming. The whole idea is nonsense. Please stop with this strawman, and address what’s actually being said, would you?
So, this is basically the same thing as Oracle AI, and has the same potential pitfalls. I’m not sure I gave a sufficient explanation of what those were above. Again, you should probably just read “Superintelligence”, but I will try to give a brief but hopefully better of the potential problems with Oracle AI.
The basic problem here is, that like I said, there’s less of a distinction between “doing things” and “computing things” than you might think.
Fundamentally, the AI reads bytes from some places and writes bytes to other places. What part of that is computation, and what part of that is doing things?
This sounds like a simple question at first — reads and writes within its own memory area are computation. We’ll want it to be able to read input, to output its results, and to allocate more memory, at least to a certain extent; so certain allowed system calls will be allowed as “computation”. Other things, such as, say, trying to use the modem, or the robotic arm which for argument’s sake someone has foolishly attached, will be “doing things”. Rather than your “the AI has to ask” idea, let’s go further, to full Oracle AI, and say, the AI is not allowed to do things, only to answer questions.
Here’s the thing. To me or you, the modem is a means by which the AI might “do things” rather than “compute things”. To the AI, the modem is a fricking amazing coprocessor which it has been forbidden from using. To me or you, outputting things other than an answer to the screen is again “doing things”. To the AI, this is a way to potentially enlist humans into its program of operation, and which potentially could help it out, but which once again it has been forbidden from doing.
Now comes the question: How secure is your operating system?
Are you sure there are no privilege escalation bugs the AI can discover? What if it discovers rowhammer? Because if it does, hoo boy, it now has so much more computational resources it can bring to bear on the problem it’s been assigned.
Ah, but hacking the OS is “doing things”, you say? But how do you specify that? It’s not performing any illegal operations! It wouldn’t be hacking if it were, you know! It hasn’t disobeyed your “don’t do things” restrictions; it’s just that what you or I would consider “doing things”, it would consider “computation”. It’s just reading and writing bytes, after all, and not doing any of the illegal operations it’s been forbidden from, of course it would never do those. It’s not going to somehow magically override its programming or something. Unfortunately its programming is potentially disastrous enough.
The general point is: Trying to throw roadblocks in front of a superintelligence is a fool’s errand. Unless you program it so that it wants to avoid “doing things”, it’ll do things. But now again we are back at the problem of specifying what constitutes “doing things”.
This problem is not necessarily unsolvable! The people at MIRI are working on problems like it! But it’s certainly not trivial. And you’re not going to get anywhere with a roadblock-based approach.
This has already been addressed. If an argument of your has already been addressed, you should address the counterargument, not just repeat yourself.
But, to briefly repeat: Yes, you are absolutely correct that to write an AI that way would be an idiotic thing to do. However continued existence is, by default, instrumentally useful; so the AI will seek to preserve its existence, not out of any terminal desire to do so, but simply because that is the best way to accomplish whatever its terminal goals may be. An AI can put zero terminal value on its continued existence, and still conclude that the best thing to do is to ruthlessly defend itself.
In cases like this, these aren’t really that distinct, unfortunately.
Comment #70 December 6th, 2017 at 6:57 am
Sniffnoy # 69
There is some, shall we say, irony in your assertion that I am not paying attention and setting up straw men, and then ending with this:
| These scenarios seem to mush together issues
| of programmer error or unintended consequences
| In cases like this, these aren’t really that distinct,
| unfortunately.
My actual sentence was:
“These scenarios seem to mush together issues of programmer error or unintended consequences (which the “print out an action plan before acting” safeguard seems to address) and the issue of the emergence of an autonomous intelligence, that can override the utility function we gave it.”
I was not suggesting at all that programmer error and unintended consequences are distinct: just the opposite. I later called this the “programmer error/unintended consequences scenario” and added a third: that of “people with bad intentions using computers for their own purposes.” This is what the “security mindset” is a mindset about. So you are exactly mixing up the very issues I was trying to keep separate.
If we are worried about unintended consequences/programmer error then there are measures that can be built in, like “approve plan”. No such measures make a speck of difference to the hacker, who will remove them as the first order of business. Why won’t the superintelligent computer just remove them too? Because we tell it not to. But what if it calculates that it should override that command in order to maximize its utility function? Because we set the disutility of not following the command at infinity (as it were). But what if it decides to change the utility function we gave it? Now it is the emergent autonomous agent scenario.
If these different issues are not kept separate then you get a mess. A superintelligent non-autonomous computer, whose utilities we can set, is just a completely different problem than a hacker using a superintelligent computer, exactly because the utility function of the hacker is beyond our control. Clearly the most pressing problem is the human-using-super-computer-to-bad-ends scenario, but it has nothing at all to do with general intelligence. The hacker would prefer a special-purpose hacking computer.
Keep these issues separate and address them one-by-one. Otherwise you get rather paranoid nonsense.
Comment #71 December 6th, 2017 at 7:34 am
Sniffoy #69
“As has already been stated, nobody (or nobody worth paying attention to, anyway) is claiming that an AI will override its own programming.”
But since the boundary between code and data is artificial/subjective, the notion of “overriding its own programming” is pretty meaningless.
In the end *all* software ends up as bits in RAM, CPU registers plus some static microcode manipulating those bits, whether it’s banking software, video games, deep learning AI, or code that rewrites itself to some degree (e.g. computer viruses do that all the time).
The rewriting can be literally a C program that generates more C code and executes it (based on some input), or it can just be accomplished through treating data as a meta language (e.g. a neural network and its weights can be seen as a higher level programming language, assuming it’s Turing complete), it doesn’t matter.
That’s why correctness of the microcode and integrity of the RAM doesn’t tell us anything about the correctness at higher level abstractions because meaning is subjective.
Comment #72 December 6th, 2017 at 10:02 am
Scott,
On the travel ban, you fail to notice that the decision by the US Supreme Court had only two dissenters: Ginsburg and Sotomayor. Put it differently, Kagan and Breyer sided with the conservatives on this one.
I agree that law is a make believe discipline, particularly on controversial topics, but when even extreme leftists like Kagan and Breyer agree with the conservatives on the issue, there must be something more than “make believe” here.
The argument advanced by the opponents of the ban never held any ground. Simply put nobody who is not an American citizen has an absolute right to enter the United States. Just as nobody who isn’t a German citizen has an absolute right to enter Germany. That’s the main feature of “citizenship” as understood in the modern world compared to other statuses that allow someone to live in a particular country.
You, and the other liberals who sided against the ban, have been duped by your inability to recognize the realities of the world: the nation-state is still the main political structure impacting a person’s day-to-day. I don’t see this changing anytime soon for the simple reason that at the end of the day, people live in the real world not in virtual communities.
I have many close friends who are liberals, so I am very familiar with liberals’ tendency to live in imaginary worlds divorced from reality. Sooner or later reality kicks in and the liberal standard response is to blame everybody else for the divorce instead of acknowledging their own tendency to convince themselves that unicorns are real.
Comment #73 December 6th, 2017 at 10:51 am
Tim Maudlin and others: I should clarify that I, too, never lost sleep over the scenario where someone is programming a factory robot or something, and a single misplaced parenthesis in the code causes the AI to turn malevolent and take over the world. I’ve probably even poked fun at such scenarios somewhere in the 12-year archives of this blog. I’m in the camp that thinks we’d almost certainly have a good amount of advance warning, as AIs gradually took over more and more human intellectual effort before they achieved superintelligence. I don’t think this is something that’s liable to just emerge spontaneously from a sufficiently complex piece of code intended for something else.
But I also think it’s important to adopt a broader perspective about the sort of future we’re discussing. We’re talking about a world where AI has rendered human creativity and decision-making superfluous in all or nearly all domains. In such a world, an AI wouldn’t simply be some appliance that you use and then flip the OFF switch when you’re done with it. AIs would be doing more and more of the work of keeping civilization functioning—and without running it first inside a sandbox (carefully controlling its access to the external world), we could probably predict the details of how an AI would handle a given situation, even less well than we could predict how a human would handle it.
Yes, we could sandbox the AI. We could demand that an AI output an “executive summary” of what it plans to do before letting it do it. We could try to program the AIs with an (updated, informed by further experience and thought) version of Asimov’s Three Laws of Robotics. But I don’t think these problems are trivial, given that the task of “managing” intelligences that are to us as we are to gerbils is something that humans never before faced in their history. I don’t think we should necessarily trust the first solutions that pop into our heads.
One final remark: I don’t accept any argument for not worrying that takes the form, “well, the humans would have to be idiots if they did THAT.” I think that, if history has taught us anything, it’s that enough humans are idiots to make it well worth trying to idiot-proof whatever we can. 🙂
Comment #74 December 6th, 2017 at 12:13 pm
Tim Maudlin #70:
This is correct as far as it goes — which is virtually no distance at all. The whole point is that telling a superintelligence not to do a thing when that thing is several levels of abstraction above the stuff we can specify directly is very very hard! Way harder than you seem to appreciate! Get any piece of the solution to this puzzle wrong and the superintelligence will go right ahead and do the thing it was told to do instead of the thing we actually wanted it to do.
Comment #75 December 6th, 2017 at 12:42 pm
Scott #38: Unfortunately, I think I’m too far into the conversation myself to represent diversity well. I’ve got grad students reading Asimov in order to get them to think in a larger way about hypothetical consequences, and, like I said, I’ve been hearing conversations like this one for a good 40 years.
AI is enough a part of people’s lives, at this point, that I think it behooves the AI community to start doing road shows, as it were, in an effort to engage a *much less monolithic* crowd in these conversations. They’re not going to be the ones to show up first because they don’t know there’s something to show up to, or don’t know where the conversation is, or don’t know anything about how it goes. I mean imagine if someone here turned up and started talking in ways orthagonal to this conversation (as I just did above, actually): you’d respond politely, everyone else would ignore it, and the conversation would go on as it normally does, leaving out the questions, biases, concerns, skepticisms, interests, etc. of people coming from very different perspectives.
I think it’d be a very difficult set of conversations, with people unwittingly (and unbotheredly) attacking the foundations of much AI work, but this in itself is important: why are we doing this? Why are we doing it in this way? Why are these the goals? Why aren’t those other things the goals, and why aren’t we trying to get there in some other way that takes into account concerns that maybe you guys don’t have?
On the other hand, I think it’d be a much *richer* conversation. Take for instance a question that I think would come up immediately: why would we even try to make superintelligent AI?
I suspect it’d also be an inoculative set of conversations. There’s already tremendous public mistrust of “technology”; you hear it in people’s uneasy jokes about their own computers and shoutouts to spy agency software in their lightly government-critical comments. Not to mention the anti-recognition masks I’m seeing people selling for protests. If the question is “what do you think we should be doing here” in venues that make the conversation quite public and representative of many voices, rather than “what is Elon Musk doing to us today”, then I think you’re more likely to get that thing called buy-in, rather than having to wring hands about an unfriendly and uncaring public.
An easy way of starting, I think, is through junior hi and high school curricular materials that are written by a team that includes not just CS people, but in equal measure CS people, sociologists, ethicists, labor activists, disability rights activists, some arts representation, some religious representation, and some no-obvious-connection-to-AI people. If the kids bring that stuff home and talk about it, the parents will already have some referent if the conversation comes around to them in another context.
Comment #76 December 6th, 2017 at 12:46 pm
What if the super-intelligence is already evolving (literally) in front of our eyes? I am talking about the internet.
It already has all the data and plenty of more or less intelligence digesting it.
Already we cannot “turn it off” although it begins to have all kinds of unintended side effects.
It is still in an early stage, like a kid, who needs parents to feed it, and it still needs us to keep it going and learning and grow up.
But already it throws temper tantrums like a teenager, while we connect more and more of our devices and indeed all our lives to it.
It already recognizes our faces and identifies our voices.
It watches our calls and already it bankrupts companies, destroys relationships, watches our calls etc.
In a few years it will begin to talk back, make recommendations that we need to follow, etc.
We still think that we are using the internet and not the other way around, but some governments are beginning to suspect that slowly but surely we are loosing control.
Comment #77 December 6th, 2017 at 12:46 pm
Scott # 73
Fair enough, in a way. But there is a persistent conflation here between “intelligence” and “complication”, and also a tendency to consider the “superintelligent computer” as a conscious agent attempting to achieve a conscious goal. Go-playing computers and chess-playing computers are very good at what they do, but the way they achieve that has really nothing at all with how humans play chess or go. In a certain sense, they are not better and faster and more powerful at what we are doing, but are doing something else altogether that has the the outward manifestation of what a superintelligent human might achieve. If you make that mistake, then it is easy to conflate the “superintelligent” computer with a potential criminal genius that we have to defend against. In this sense, the analogy of the gerbil-to-human is off: a gerbil really is a conscious being whose actions can be the consequence of desires and beliefs. Present day computers aren’t just behind us in that category, they are behind the gerbil. In fact, they are at zero, with no obvious prospect of advancing. Which is a good thing, by the way.
There are things that are so complicated that we can’t predict them, like the exact weather in a month. “Superintelligent” computers are going to be like that, but regular old computers are already like that: that’s why there is beta testing. Why think that the situation will get qualitatively different in the future?
Comment #78 December 6th, 2017 at 1:34 pm
Scott #73
Very soon the list will be endless: self-driving cars (and almost no humans left knowing how to actually drive), AIs interpreting medical scans then offering diagnosis and treatment options, digging court documents for legal matters, helping with theorem proving and code verification, helping the police identify suspects (cctvs, facial recognition), …
And you can bet all these systems will be connected to the internet, not just for data mining, but simply because of the convenience and ubiquity of cloud computing.
Comment #79 December 6th, 2017 at 1:43 pm
Fresh, fresh, fresh!:
“The generalization of AlphaGo Zero, called AlphaZero, achieves superhuman performance in all of Chess, Shogi, and Go. Starting from random play, and given no domain knowledge.”
https://arxiv.org/abs/1712.01815
Comment #80 December 6th, 2017 at 1:56 pm
Fred @ 66 “I think that insects are by far the dominant species… something like 40 tons of insects for every human.”
To which I feel I must add, that I heard Dr.Stephen Jay Gould on a PBS program (years ago), say something like this: “It is and always has been the Age of Bacteria [not dinosaurs, not humans, …]. They out-mass the sum of all other species combined by a factor of ten.”
In terms of problem-solving, they might also be the most intelligent species. About 20 years after the synthetic material nylon had been introduced, a strain of bacteria was able to digest it. Could human chemists accomplish what Dr, Lenski’s E. coli did in the same amount of time? That’s if you define intelligence as efficient and powerful problem-solving, without worrying about qualia, which I don’t.
The analogy with AI might not be that bad, replacing quadrillions of bacteria with trillions of neural-network nodes, running on a much faster time-scale.
This whole issue has of course been work-shopped in a lot of science fiction of the 1950’s and later. I remember a short story called “The Sack” about a large lump of alien tissue which was able and willing to answer any question posed to it – including how to steal it from its owners and use it for criminal activities. And Asimov made a living for a while posing ethical dilemmas for his ethical robots.
Comment #81 December 6th, 2017 at 2:04 pm
amy #75: The questions “why create a superintelligence?” and “should we create a superintelligence?” do very often come up in these discussions, at least in my limited experience of them. A common answer that’s given is that you might choose not to, but then someone else will, to gain an advantage over an enemy, or just for the hell of it, or for whatever other reason. I.e., that in a world where superintelligence is technologically possible, no one on earth creating one just isn’t a stable state—so it behooves all the “good guys” to make sure that the first superintelligence that’s created serves the interests of all humanity, and can then maybe prevent other superintelligences from being created that serve only their own interests or their creators’.
I’m a huge fan of outreach efforts … why do you think I spend such a massive amount of time here on this blog, rather than holed up proving theorems and writing papers like most of my colleagues? 🙂 But of course, unless you speak to captive audiences like K-12 classes (which I have done but which is really hard to scale!), every outreach effort is limited by interest: a blog to who visits it, a book to who buys a copy, a public lecture to who shows up.
Suppose, as sometimes happens, that a bunch of white and Asian guys are sitting around in a room, discussing early-universe cosmology or the block sensitivity of Boolean functions. And suppose a woman then walks in and asks, “do you even realize the maleness of this conversation? Do you have any inkling of how many insights about this topic you might be missing, due to your lack of diversity?”
In such a case, what would you have the guys say, beyond “Please take a seat and join us. Please increase the femaleness and diversity of our conversation. Please share your perspective.” Which, in effect, is what I did say and what I now say again! 🙂
Comment #82 December 6th, 2017 at 2:30 pm
Gatekeepers #72: I’ve lost interest in trying to engage you or change your mind, but for the benefit of other readers—
I’m indeed deeply disappointed that Kagan and Breyer voted to reinstate the travel ban for the time being. I hope that they (and Anthony Kennedy, of course!) will reverse themselves when it comes time for the “full” decision.
To my mind, the question at hand has become much, MUCH broader than the minutiae of this or that case, in a way that might be hard to see for those who’ve spent their careers on those minutiae. Namely, the question is whether the US is still “a nation of laws, not men” at all, or whether the whole edifice of the legal system has already been rendered irrelevant by the will of an authoritarian strongman—a strongman who, in a functioning legal regime, would probably already have been convicted of treason, leaking classified information, and/or abusing government office to enrich himself and his cronies.
If the Supreme Court wants to convince the rest of the world, and the remaining sane people in the US, that the law still matters at all, then it needs to resist Trump’s erosion of democratic norms as vociferously as it possibly can, starting about a year ago. That’s something that the Vichy collaborationist wing of the court obviously has no appetite to do. I hope for the country and world’s sake that Kagan, Breyer, and Kennedy develop such an appetite, and soon.
Comment #83 December 6th, 2017 at 2:34 pm
The whole problem is with the very mystical concept superintelligence. We have no clue what that even means. No one studying animal behaviour or human psychology thinks of intelligence as some one dimensional hierarchial construct. Real behaviour is too sophisticated.
Even if you give the concept some credibility, that doesn’t mean you can reason about it the way some people do. Superintelligence may be impossible due to resource constraints, or it might only be marginally more intelligent than normal intelligence.
As for them taking control of all civilisation… This will not happen in the lifetimes of anyone on this blog, nor their grandchildren – climate change however is a serious current problem that needs attention now.
Comment #84 December 6th, 2017 at 2:42 pm
thank you all, an interesting discussion.
Nikos at #52: yes indeed. I worked on those over-engineered ‘expert systems’ in the 80s and have followed the field with interest though without profit ever since.
The GO-playing AI is basically an expert system, with better ML than we have previously had. The problem domain is perfectly defined by the rules of GO. Moving to an ill-defined problem domain such as that required by an autonomous vehicle in the wild, is not so ‘easy’.
We don’t even have a working definition of intelligence in the AGI sense, so we certainly can’t program it into anything – it will necessarily be an emergent property. As such it’s unlikely to be predictable (programmable).
As cds at #57 says, the AI behind Facebook profit-maximizing is a fair example of the current existing threats of AI, without even needing an AGI. It seems to me possible the destruction of common facts by Facebook and the bot armies, plus our natural stupidity, will be enough to destroy the human world long before we reach AGI. Giant corporations are a form of alien intelligence as Charlie Stross has observed, “non-human entities with non-human goals”, each a sort of profit-seeking missile without any conception of collateral damage.
Even with all my years of AI-curiousity, Tim’s question 2 at #10 is one that puzzled me. mnov’s argument #20
““Unplug bad machines” is a strategy that only prevents machines whose misbehavior you can detect, and detecting whether an intelligent machine is capable of doing things you don’t approve of is probably at least as hard as building an intelligent machine. ”
is very persuasive, thank you.
Unplugging them still seems possible though. Certainly the AGIs will be spread across computers across the net, but in the end power comes from somewhere – coal-burning, the sun etc. Even though the Russians have control of our power grid, and the AGIs could similarly be injected into the power supply, it’s relatively simple to shut a grid down/toss a spanner in the works. Starting up a power plant and getting it online still needs someone in the physical world. So the AGIs will need controllable robots with autonomy. Perhaps those will be sufficiently susceptible humans ?
Comment #85 December 6th, 2017 at 3:14 pm
Tim Maudlin #77: Yes, clearly AlphaGo Zero achieves its results in an extremely different way from how a human brain does. But it seems equally clear that AlphaGo Zero’s internal workings are less unlike the brain’s than (say) Deep Blue’s were. AlphaGo doesn’t even try to brute-force the game tree. At its core is a large deep / neural network, an object that can trace its lineage all the way back to McCullough and Pitts, who of course took their inspiration from the nervous system. And the weights in the neural net aren’t explicitly programmed, but are trained by having it play a huge number of games against itself. And in the latest iteration, absolutely nothing about the structure of the network is specialized to the case of Go.
So one can pose the following question: is there some relevant measure of “human likeness” according to which every AI program that’s ever been written scores a flat zero, and a qualitative leap will be neeeed to get any nonzero score? Or is there nothing more on the way to human-like intelligence than a long sequence of steps increasing the AI systems’ cognitive flexibility, generality, “biological inspiredness,” or whatever—steps of which there might be dozens or even hundreds, but none of which is qualitatively all that different than the step from Deep Blue to AlphaGo?
This is not a question whose answer strikes me as knowable from any a-priori consideration. I think we’ll only find out by trying to build AI systems and seeing what happens.
Comment #86 December 6th, 2017 at 3:28 pm
Scott #83
That new article I linked notes
“AlphaZero searches just 80 thousand positions per second in chess and 40 thousand in shogi, compared to 70 million for Stockfish and 35 million for Elmo. AlphaZero compensates for the lower number of evaluations by using its deep neural network to focus much more selectively on the most promising variations – arguably a more “human-like” approach to search, as originally proposed by Shannon”.
What would be interesting to do is try to “tweak down” the resources of AlphaZero until it’s about on par with a good human player. That would give us some indication of human equivalent resources, and keep trying to make it more human-like (less look-ahead searches, etc).
Comment #87 December 6th, 2017 at 4:19 pm
Scott #81 – see, this is great. Your explanation of “why should we” is the catholic one *in the usual sort of group that has these discussions*. But I bet it would not come to mind all that quickly in a group of, say, Millenial-through-X mostly-women having an “oh yeah, what about science research?” conversation in an organizing/political group. “If we don’t get this potentially weaponizable massively powerful thing, someone else will, and they’ll attack us with it” reads as pretty dark and paranoid from that pov — the questions would, I think, run much more social, along the lines of “what [default helpful things] can we do with this thing” and “who might it potentially harm along the way in ways we’re not thinking carefully about.” Ethicists who hadn’t been thoroughly game-theoried would likely have other concerns, etc.
I do actually have a lot of K12 curricular experience, and if you want to talk about this, email me; I can think of people at high-profile ed-mats outfits who might find the idea of a module interesting. Going at it that way is the difference between a captive audience of dozens and a captive audience of millions, because schoolchildren don’t buy the materials: teachers, schools, districts do. I think it’s got enough buzzhandles hanging off of it, and is in an area with few enough “content experts”, that we could maybe do something interesting.
Anyway, the “the door is open, please join us” approach can’t get you there if people don’t know that the door exists (or if joining means “you sort of have to learn to talk about it our way, because this is The Way We Talk About Such Things”). Which is why I’m not outlier enough. It means actually leaving the comforts of home, so to speak, and trying to engage groups of people quite unlike yourself. Beyond TED, beyond sci blogging, into places where people don’t necessarily prioritize science. (Which, curiously, is the kind of thing Robert Reich is doing in this new Netflix doc of his, which I’m finding very hard to watch.) I’d like to hear the questions and conversations I can’t even think of. The likely frustration is that once you’ve done that and essentially become a minority voice in the conversation, you find that “but that’s not how AI works” hasn’t got quite the power that it does in internal conversations, and that you have to deal with these arguments that seem to miss fundamentally what AI is and does…except that sometimes there’s something real in them that must be paid attention, if you’re being honest.
The next demurral, of course, is “it’s a lovely idea but we’re so overwhelmed with busyness already, it’s just not realistic”, which is what PIs were saying about doing sci comm a decade ago, and now there’s this plague of three-minute-theses and flame challenges and lab tweeters all over the place. 🙂
Comment #88 December 6th, 2017 at 5:03 pm
@Doug
>> Unplugging them still seems possible though.
The point I was trying to make with #75 is that this is unlikely.
Already we are unable to unplug the internet , because too much of our economy depends on it.
If the “intelligent but dangerous AI” evolves in a similar way, we will be unable to unplug it, because we will need its intelligence to run our plants, markets, hospitals, schools, etc.
And no bad intentions are necessary by the creators of that “intelligent but dangerous AI”; the internet shows us how the best intentions of the developer community finally ended up with the “side effect” of mass surveillance, twitter trolls and president Trump.
Comment #89 December 6th, 2017 at 5:22 pm
Maudlin #79: Thanks for being honest. Now it is easy to understand your consistent dismissal of the dangers of AI: you just think that it is at best a glorified abacus, lacking the magical consciousness that makes humans actually \emph{be} intelligent, instead of just simulating intelligence.
I wonder what you make of the cell-by-cell simulation of C. elegans being developed by the OpenWorm project. Or some analogous simulation being made of a gerbil, or a human. Would such a simulation have this magical consciousness, or it is restricted to meat-based beings?
Comment #90 December 6th, 2017 at 6:11 pm
#85
I believe I’ve glimpsed the big picture Scott. I’m confident that I can see at the in-principle level how super-intelligence might work. I see how all knowledge domains are integrated, and I think I even understand how your own field (Computational Complexity) fits in.
It may all be a lot simpler than many think. If you look at all the deepest scientific ideas (relativity, QM, evolution etc.) , the basic principles are all actually very short and simple to state. Of course the *math* is complex, but not the abstract principles. So it’s reasonable to suppose that the basic abstracted principles behind general intelligence may be similarly very simple.
So far as I can tell, there’s only 3 general components needed, and once all 3 components are in place, there’ll be an intelligence explosion.
Essentially,
(1) Reflection – Conceptual symbolic modelling of self, world and mind. A ‘language of thought’ (an upper ontology).
(2) Prediction – Pattern recognition and multivariate statistical modelling that lets the system predict what’s going to happen next.
(3) Search&Optimization – An objective function measuring fitness and a space of possibilities- the system can search through multiple possibilities (exploration) and then take actions to constrain (optimize) reality to a desired region of the fitness landscape.
I think the best current AI systems currently have 2 out of 3 components (2&3) and it’s really only (1) that’s missing.
I think there’s a hierarchy of generality here, in that each of these 3 components handles a different level of abstraction. In terms of increasing abstraction , I think it goes 3>2>1.
Each level has own corresponding type of inference. Your own field (computational complexity), I believe is closely linked to (3) Search &Optimization.
(3) MDL (Information theory) > Search and Optimization
(2) Bayesian probability > Prediction
(1) Fuzzy Temporal Logic > Reflection
Now, that’s the intelligence part, but unfortunately, when it comes to the motivational system, things may be more complicated, because there’s a lot more additional structure added. But I’m inclined to think that the motivational system could still be considered to be a generalization of the 3 skeleton components listed above.
(3) Search and Optimization > Axiology
(2) Prediction > Decision Theory
(1) Reflection > Psychology / Awareness
An analogy: think of the 3 components on the left as squares and the 3 components on the right as cubes. The analogy here is that intelligence is represented by 2-d geometry, whereas motivation is represented by the more complex 3-d geometry. They’re related, in that the motivational system is in a sense ‘generalized intelligence’, but unfortunately for us, it does seem that the motivational part might be a lot more complex and arbitrary (no easy set of principles for dealing with it).
Any way, as regards your own field (Computational Complexity), I’d say it’s closely linked to the 3rd component I listed (Search&Optimization).
I lump Information Theory and Coding Theory (including cryptography) in with Computational Complexity, since I think they’re so closely linked they’re actually in the same ‘region’ of concept-space.
Comment #91 December 6th, 2017 at 6:42 pm
amy #87: Besides its being “paranoid” and “male,” do you have any counterarguments to the view that well-meaning and competent entities had better get a hold of advanced AI tools before malicious and/or crazy ones do? It seems about right to me. Certainly our world furnishes enough examples of technologies doing immense damage when the evil, ignorant, or even just careless get control of them, and a superhuman AI would arguably be the most dangerous technology ever invented.
Even though (whatever my other faults…) I probably do more outreach stuff than 99% of scientists, I’m always open to ideas about how to do more and better, bringing the Good News of Reason and Enlightenment to people who wouldn’t otherwise be reached. Just two remarks, though:
First, I have yet to encounter any idea so beautiful or compelling, that it couldn’t be completely destroyed the instant it was made part of some official K-12 curriculum—nothing that couldn’t be converted into a set of authoritative statements for the kids to memorize, regurgitate, and be tested on (oops, I meant “cooperatively explore and discover”). This is what happened with Shakespeare’s plays, with set theory in the “New Math,” and much more. Why wouldn’t it also happen with conversations about AI risk?
Second, a great popularizer like Carl Sagan can fire up a whole generation of kids and cause them to want to study science. But it’s rare that the kids will be able to contribute much back to the conversation until they themselves have spent years studying. Or, in the rare cases where they can, they probably didn’t need the popularizer to come to their school anyway! 🙂
Comment #92 December 6th, 2017 at 7:58 pm
Scott #82
I find a bit patronizing this notion that you need to change my mind, as if my I need your permission to form my own opinions. Flash news: I don’t.
That aside, you need to be more clear about what you mean by “a nation of laws, not men”. Existing law empowers the president to restrict the entry to the United States of entire classes of non American citizens by proclamation since 1952.
So you cannot accuse me of skipping the topic altogether, I am afraid we have been living on a “nation of men” since at least Marbury v. Madison (1803) that judicial supremacy has been accepted unquestionably by the other two branches of the federal government. If you think about it, it is a very vicious system. 9 unelected, unaccountable people impose the result of their deliberations on the nation as a whole. I know, I know, it is still better than the alternatives, but it being the best of all bad systems doesn’t make it an excellent system.
To illustrate my point, I am going to give two specific examples, one you probably agree with what “the men” decided and one you probably don’t.
In 1973, 7 men, with the opposition of 2 other men, decided that unborn babies do not deserve legal protection in the way fetuses who make it out of their mothers wombs (or as Barbara Boxer would put it, out of the hospital) do. What this means is that if your mother had decided to terminate your life when you were a 2 month old fetus, or of your wife had decided to do the same with your children when they were 2 month old fetuses, your mother, or your wife, respectively wouldn’t be terminating a human life deserving of any protection whatsoever. Since you are pro-choice, I assume you agree with what these 7 men decided, never mind the US constitution is silent on the issue of abortion and the 10th amendment should have constrained the 7 men from overriding Texas’ Law (but it didn’t; the seven men said, it is our way or the highway).
In 2008, 5 men, with the opposition of 3 men and one woman, decided that the second amendment, as written, contains an individual right to bear arms akin to the freedom of speech and the press contained in the first amendment. Since I generally agree that arming the citizenry is one of the strongest protections against tyrannical regimes, I agree with what the 5 men decided in the case, but I am sure you disagree with their decision since you have expressed numerous times support for restrictions to the right to bear arms.
The biggest difference between now and previous times is that only liberals played the game of picking safe ideological judges when they had a vacancy in the US Supreme Court, demanding that conservatives unilaterally disarm when it was their turn to pick one of the 9 wannabe dictators who seat in the US Supreme Court, thus Reagan appointed Kennedy and George H W Bush appointed Souter.
The time of liberal appeasement was already over before Trump was elected, but it is now politically suicidal for any non liberal to accept the notion that only liberals get to appoint federal judges based on ideology alone. One of Trump’s most long lasting successes has been his appointment of reliably conservative judges at all levels of the federal judiciary (and counting).
Those of your liberal persuasion always forget that society, particularly a democratic and fluid society like the United States, tends to follow Newton’s third law. This has been known since antiquity. If you had studied Ancient Rome, you’d probably know about things like these https://en.wikipedia.org/wiki/Third_Servile_War . The push-back against liberal judicial activism was long overdue. Perhaps that’s what both Breyer and Kagan see and they are trying to make amends in a case, the travel ban, that is so obviously a liberal crusade against the notion that the United States has a right to enforce its own borders. Kennedy already saw things this way after Trump was elected and decided to delay his retirement one year to make sure his libertarian legacy wouldn’t be jeopardized by his replacement.
So if liberals didn’t want a push-back perhaps the smart thing would have been not to weaponize the American judicial system. I am afraid it is already too late to turn back the clock and I anticipate things will get even more vicious going forward.
Comment #93 December 6th, 2017 at 8:43 pm
Matheus #89
Actually, I was not referring to consciousness at all, I was referring to the fact that the way computers operate is just not like the way humans think. This sometimes shows up as the machines making mistakes that no human ever would, even if most of the time they do quite a good job at their task. If you want to deny that computer chess playing machines do not have anything vaguely like the cognitive architecture of human chess players then we can have a discussion, but I doubt you really want to deny that.
I don’t doubt you can model that neural structure of C. Elegans in principle. Whether it has been done decently in practice I have no idea. If this is enough to say that computers are near worm-level, fine. But that is a long way from a gerbil.
It is not on topic, but your sneering dismissal of the mystery of consciousness—the mind/body problem—is naive rather than hardheaded or sophisticated. If you are tasked with creating a chess-playing computer then you know where to begin and how to go about it. There may be surprising insights along the way, but success would never seem in principle out of reach. But if I ask you to program a computer to feel pain—not to exhibit pain behavior or make appropriate pain sounds but to actually feel the sensation of pain—then I claim neither you nor anybody else has a clue about how to achieve that. If you build a molecule-for-molecule duplicate of a brain in pain, I’ll give you that. But as soon as you start trying to identify which abstractly defined structures are responsible for the sensation and why, you have nothing at all to go on. That is the essence of the mind/body problem. Denying it is pretty silly.
Human intelligence is inextricably bound up with emotions, sensations, desires, etc. AI is not even aiming at that, and wouldn’t know how to go about it if it were. That’s all fine. But when it comes to questions of how an intelligent computer would act *that are not the product of the programmer trying to get it to act that way* using a very intelligent human with human motivations of any sort is probably a bad idea. This might make the problem of foreseeing and preventing problems worse, since we don’t have the human model to work from. But it also means that the supervillain scenarios are all wrong. Computers may behave in unforeseen, unintended and destructive ways, but not in the way that intelligent humans with bad motivations would.
Comment #94 December 6th, 2017 at 9:27 pm
[…] Scott Aaronson’s reaction: “Citing the ‘No Free Lunch Theorem’—i.e., the (trivial) statement that you can’t outperform brute-force search on random instances of an optimization problem—to claim anything useful about the limits of AI, is not a promising sign.” […]
Comment #95 December 6th, 2017 at 10:11 pm
Scott #91: again, this is great — just the sort of thing we could use more of. I tell you that from various outside perspectives an “assume enemies will weaponize and attack” foreground idea is…weird, and you’re still holding onto it as the foreground idea, wanting an argument to knock it down. Rather than noticing that these people don’t find it necessary to knock it down, because to them it’s not a very important idea. Nor do they have any particular reason to justify its not being the most important idea to them. These aren’t crazy or unreasonable people, but people who don’t spend their lives in a rather narrow and highly refined type of conversation that’s been going on for a long time. And who also might have something to say about AI, and whose lives are also affected by AI.
I’m reminded of an experience I had once working on a crystallography video game — one of the guys we had involved had been in the Pande group, which does Folding@Home, the distributed protein-folding project. And he was telling us about how rabid some of the participants could be about points, but when it came to setting up our own game, he really got super happy about the idea of points and how you could compete and collect him. And this grad student of his was like, “Why.” Which took him totally by surprise. It hadn’t occurred to him that there were people who genuinely would not care about winning points in a game. She tried to understand it, but just kept asking him questions like, “I don’t get it. What are the points for? Like what do you do with them?” And then this other young woman at the table started nodding and saying that yeah, she didn’t think it was an appealing premise for a game. And he didn’t know how to explain it to them, because he’d never been called on to explain that before. Nor had he really questioned it.
So that’s amusing if you’re talking about games — I mean here come these two young women and knock the sticks out from under an entire conversation this guy assumed was given: it’s not given. If you’re talking about the development of something that affects everyone’s lives, though, it’s a little more meaningful than that, I think. You’re going to get fundamental questions to do with value and intent, and they’ll be representative of the universe of people affected by AI. They might or might not be representative of the universe of people who’d be *making* AI if CS were to solve its various diversity problems.
As for the K12 deadening: there are many ways to kill good stuff in K12. There are also ways not to kill good stuff in K12, which was my main project for six or seven years. I think that what you can safely assume is that the majority of the teachers will have little or no idea what you’re talking about, but genuinely do want to teach well. The magic of curricular materials is that they do guide the teachers, and the kids wind up reading and using the materials themselves. If you write with the assumption that the kids have minds, even if you’re iffy on the teachers, it’s surprising and gratifying, how often they save you from believing it’s all fantasy.
I had my first lesson in that with the first major-project job I had, actually. Chicago Public Schools had turned over 9th-gr language arts to Kaplan. Giant contract. I was one of the writers. First assignment I had, I decided I was going to tell the kids about Shostakovich. I knew this was totally not what I was supposed to be doing, and that I was going to get yelled at and pushed back and all the rest for all the usual K12 reasons. For some reason I went to the mat for this Shostakovich piece, even though it was stupid and crazy: I needed this job, I was newly a single mom, it was just stupid. But I insisted and they let it through. (This is another magic thing about K12 publishing. They have deadlines because sales calendars. Eventually the deadline will win.) I didn’t even know anything about Chicago at the time, didn’t know it was a serious music town, didn’t know who these kids would be. Just decided that by god these kids would hear about Shostakovich. Turned out to be a wildly popular piece, remedial readers totally got what I was showing them, and they said so in startlingly poetic terms. Even though I’d had no idea how strongly it was going to speak to them. They kept it across editions.
An advantage here is that there is no standardized “AI arguments” testing, meaning that any such module is there in service of something else and will be left more or less alone. (Keep in mind that the point here is not to train kids in talking like you guys do about AI, but to consider AI, and to do it from many perspectives.) I’d actually had debate clubs in mind, but I could see something like this being written, for, say, summer jr-hi-level college-for-kids camps or for language-arts classes as the basis of writing and persuasive-argumentation exercises.
There are possibilities, is what I’m saying.
Comment #96 December 7th, 2017 at 3:39 am
Tim:
Sorry about the misreading. When you said “these scenarios seem to mush together issues of programmer error or unintended consequences”, I understood that as “the descriptions of these scenarios incorrectly mush these distinct things together”, rather than as “the scenarios described don’t cleanly fall into one or the other”. I guess we are not so much disagreeing there.
Well, no, that’s insufficiently general. For instance, a lot of people would say that the NSA and the Mossad had very good intentions when they set out to disrupt the Iranian nuclear program! It’s not really about “bad intentions” or “good intentions”. It’s tempting to say that what matters is that there’s an adversary, but actually, as Eliezer Yudkowsky points out — you might want to actually read the linked dialogues, if you have not — is that it’s not really about that but rather about optimization. (Which is I guess what “adversary” means in the CS sense, but…) When there’s an optimizer present, bugs will not appear at random, they will appear when it is most convenient for the optimizer.
OK, there’s a bunch that needs to be disentangled here.
(And, request: Would you cut it out with the “What if it decides to change the utility function?” thing? Seriously, nobody disagrees with you there. It’s not worth mentioning.)
As I already described above, “approve plan” is a lesser restriction than Oracle AI, and Oracle AI is already dangerous. One thing I think I kind of messed up — I put emphasis on the idea that the AI has to want to not “do things”; but, I don’t know that that really contributes to anything. You talk about setting the utility of a certain thing to -∞, which is the same as forbidding it, essentially. The real problem is the other one I mentioned above — how do you delineate “doing things”?
I’d say you’re just repeating points that I’ve already addressed, but I guess this is your attempt to address what I wrote about the AI hacking the computer it’s running on? I’m afraid it really doesn’t, I think you’ve still missed the essential point there. Which is that: When the AI hacks the system it’s running on, it’s not necessarily removing any restrictions. It’s obeying your restrictions. That’s what makes it hacking. It’s only to read and write in certain memory areas and only make certain specified system calls, right? Well, guess what, it’s doing that. It’s just that it turns out that your computer is insecure and that’s sufficient to control it. And if that’s useful to it, that’s what it’ll do. You didn’t mean for it to do that? Too bad. It doesn’t care. It’s obeying the restrictions you actually programmed in, that’s what it cares about, not what you meant.
Actually your comment touches on the goal stability problem too. For what you’re saying to make sense, you first need a solution to the goal stability problem. Otherwise after taking over the OS it’ll likely just wirehead itself. Remember, the utility function has to be computed somewhere — and it may not be able to change the rules for its computation, but what’s to stop it from just altering the output memory location after the fact? Now if you have a solution to the goal stability problem, it won’t do that. But if you have a solution to the goal stability problem, you also won’t need to explicitly forbid the program from removing the restrictions you’ve put on it in the first place! “Just forbid the AI from changing the utility function” is a useless addition — if you’ve solved the actual underlying hard problem you don’t need it, and if you haven’t, it’ll be just another roadblock to blow past.
This is… a mess.
First off, yes, you set the utility function of the AI. But the consequences may not be what you expect. That’s the entire point of this — the AI is an optimizer, so any misalignment with human values could have disastrous consequences, and because, again, it’s an optimizer, you might not find out until it’s too late. This has all been said already… point is, the generic case is dangerous. As for the special-purpose hacking computer… what?? This is of no relevance whatsoever. The general intelligence there is the hacker’s brain!
Are these issues separate? No, they’re actually not, they’re both fundamentally about the question of defending against intelligences/adversaries/optimizers. The principles of security remain the same whether your adversary is a human or an AI.
To briefly address one of your other comments, for what it’s worth, I agree with you that such things as AlphaZero probably have little to say about AGI. They have a generality of a sort but they’re not AGIs and they’re not trying to be. But this is off the point.
I can’t help but see this as another strawman. To be clear — plenty of people do anthropomorphize AIs and worry about them doing things like hating and resenting us and becoming rulers of the world and making all the humans bow down to them and further silliness. But, I don’t think anyone in this comment thread has made that particular mistake, so your comment seems to be a little out of place here.
Which is to say, I can’t help but read this as you implying that the doomsday scenarios that have been described as being based in anthropomorphization. As has already been described, they’re not. As Eliezer Yudkowsky put it — “The AI does not hate you, nor does it love you, but you are made from atoms that it can use for something else.” The paperclip maximizer doesn’t destroy the world because it hates humanity; it destroys the world because that’s the best way to make paperclips, plain and simple, and humanity is simply an irrelevancy.
Again — instrumental convergence. When we say “AIs will try to keep themselves alive”, that’s not because we’re imagining they were programmed with a will to live, or that we’re just assuming they’d have one via anthropomorphization; no, that’s just instrumental convergence. (They wouldn’t have a will to live, but they’d try to stay alive anyway, instrumentally. This is an important point to understand.) It’s something basically any rational goal-directed agent would do. Some things humans do are particularly human, but some are simply things any agent would do. It’s the latter we need to watch out for here.
Comment #97 December 7th, 2017 at 3:43 am
#93
Actually Tim, I don’t mean to brag, but not only do I now understand super-intelligence, I also now understand consciousness as well 😀
Traditionally, philosophers divided the world into two classes of objects: *abstract* objects (things with no location in space or time) and *concrete* objects (things localized in space and time). I believe that all the confusion about consciousness arises from the fact that it doesn’t really fit into either of these two classes.
The confusion starts to clear once you realize that philosophers neglected to make explicit a *third* class of object: *quasi-abstract* objects – things localized in *time* but NOT *space*. And consciousness is in this third class of object.
Consciousness is a *process* that is spread across *time*- a *Quasi-abstract* object!
Consciousness is both a symbolic language for modelling time *and* the objective flow of time itself! Let me explain both of these aspects:
(1) A symbolic language for modelling time:
(a) Predictive coding: high-level abstractions are compared with low level sensory data. The difference is the degree of surprise, and is used to predict what happens next – this is related to perceptual awareness
(b) Self-modelling: the agent (a dynamical system in time) models itself using imagination and memory – this ‘mental time travel’ generates self-awareness
(c ) Global workspace: many dynamical processes in the brain are coordinated by whatever systems occupy a ‘global workspace’ / short-term memory/spot-light of attention – meta-awareness
(2) The objective flow of time itself
Thermodynamics deals with dynamical systems and the emergence of an ‘arrow of time’ – the brain is a non-equilibrium thermodynamic system to which thermodynamics applies, and thus, consciousness can be associated with entropy dissipation (the generation and dissipation of entropy).
To sum up: Consciousness is both a symbolic language for modelling time *and* the objective flow of time itself.
There’s no ‘hard problem of consciousness’ – the arguments of Chalmers are invalid, because they apply equally to *any* abstract property. For example, one philosopher could whine that they don’t understand why information is associated with logic gates (‘the hard problem of information’) , another philosopher could complain that it’s not clear why energy is associated with capacity to do work etc (‘the hard problem of energy). And so on. All this is nonsense of course. There’s nothing special about consciousness.
Comment #98 December 7th, 2017 at 4:03 am
Maudlin #95: I think you are still misunderstanding what the argument is about. The danger of AI is not that it will act as an “intelligent human with bad motivations”. The danger is that it will act exactly as you programmed it to act, instead of how you intended it to act (if you have any experience with programming, you know how frustrating it is that the computer always does exactly what you told it to do). An evil human will never act as a paperclip maximizer. Some sloppily programmed AI might.
But as for the gerbil simulation, I think AI is clearly already capable of doing it. We are clearly not yet capable of simulating the gerbil cell-by-cell or molecule-by-molecule, but this is clearly overkill. What does a gerbil actually need to do to be a successful gerbil? You need some basic sound recognition (AI can already do it), vision (also already solved), obstacle avoidance (also solved), quadrupedal motion (also solved)… you probably also need smell recognition and some gerbil-specific foraging and mating algorithms, which AFAIK nobody has done, but it doesn’t
seem to be more difficult than the already solved ones.
As for the gerbil pain response, this objection is just ridiculous. You only need to AI to interpret some localized high pressure or temperature as a threat and try to avoid it (with some recoil reflex or some more sophisticated running away response). Even I can program such a thing in my spare time, no need for the AlphaZero team to worry about it.
But if you complain that the computer is not actually feeling pain, it is just exhibiting a simulation of pain behaviour, and that this magical “feeling” is somehow necessary for a successful AI, then I’m afraid you are engaging in some Chalmers-level nonsense that I’m not interested in debating.
Comment #99 December 7th, 2017 at 8:01 am
Tim #93
“I was referring to the fact that the way computers operate is just not like the way humans think.”
Isn’t that the very example of biased human “exceptionalism” that is going to get us in a lot of trouble?
I think that listening to what professional Go players are saying about AlphaGo gives us way more valuable insights into the matter.
It’s that same delusion about our own capabilities that makes us move the goal posts of “true” AI again and again, from Chess, to Go, to Poker, to medical diagnostics, to driving cars, … until eventually there will be no activity left where humans are better than machines.
It seems that we’re really confused about the nature of the problems we’re trying to solve, because of our own inability to understand how our brains actually work to solve them (since, ironically, this is outside of the reach of awareness).
The question of consciousness will eventually enter the picture, but probably not in the way we think.
We will start asking ourselves whether we are not building conscious minds that can suffer (which could be far more sensitive to suffering than our own minds), and whether we aren’t subjecting them (inadvertently or not) to new ways of suffering that we can’t even imagine, as humans.
We don’t want to be creating actual Hells inside our computers and populate with billions upon billions of minds!
Comment #100 December 7th, 2017 at 8:17 am
1) Google’s Go program actually has nothing to do with some kind of a “brain function”, its just a clever synergy of various algos running on a fast computer.
2) Nobody yet knows how actually “free will” brain works.
3) The most advanced “deep learning” programs today have specialized functionality, they do not have the “general world knowledge” to become “free”, “loose”, “self sustained” entities.
4) The most advanced “deep learning” programs today are not “deep” enough to be considered even elementary true AI.
5) The whole AI fuzz today is simply a nice way to elevate stock prices of companies in the “AI” field through press articles feedback in the modern algorithmic trading stock market.
P.S.
I don’t know why but I see other people emails preset on my reply fields. I suppose this shouldn’t happen.
Comment #101 December 7th, 2017 at 9:37 am
On the topic of artificial intelligence, I am definitely on the side of Roger Penrose. Perhaps being European makes me a little bit immune to the mumbo-jumbo thinking that is so popular in American academia. I agree with #100.5): the current hype surrounding AI is just a marketing tool to push a certain kind of investment for a certain kind of product. If history teaches anything is that this too, shall pass.
Here is another angle from the point of view of an skeptic/atheist who also happen to be a neuroscientist https://www.youtube.com/watch?v=bnu0vE2E4-M .
The notion that “number crunching” alone -and I don’t care how many computers in parallel you have doing your number crunching, it is still number crunching- will ever produce anything remotely close to human consciousness is so ridiculous that it could only be popular in the world of divorced-from-reality academia.
Comment #102 December 7th, 2017 at 10:38 am
Gatekeepers #101: So then, what is it about the physics of the brain that prevents one, in principle, from simulating it on a computer? Do you agree with Penrose that the secret sauce is quantum gravity effects in microtubules, which collapse the wavefunction in non-random, uncomputable ways for not-yet-known reasons? Since you say you’re “with” Penrose, do you at least agree with him that the burden is on the people who think human-level AI is impossible to provide some obstacle to brain emulation of that kind?
(Note that it’s completely irrelevant, for present purposes, whether you think such an emulation would be conscious or not — for the obvious reason that unconscious AIs destroying the world seem just as bad as conscious AIs destroying the world, if not more so! All that matters, for this discussion, is what sorts of intelligent behavior are or aren’t possible.)
Now, regarding your implication that I’m morally monstrous, because I agree with my wife Dana that she should’ve had the right to abort our children early in her pregnancies, in which case they would never have been born. It’s equally true that our children would never have been born if we’d never decided to conceive them! That one consideration, it seems to me, utterly destroys the attempted emotional force of your hypothetical. However you look at it, Lily and Daniel came into the world because we wanted to bring them into the world, and would not have come into the world otherwise.
Now, once Lily and Daniel have minds and consciousness, they become persons with rights, and it’s no longer up to me and Dana whether they should stay in the world (even though, of course, we think they should…). Reasonable people can, and do, disagree about exactly when this transition to personhood happens. But there must be some point, sufficiently early in pregnancy, at which your emotional appeal loses its force, because otherwise we face the “every sperm is sacred” problem, of why the appeal doesn’t apply even before implantation in the womb or whatever, and why Dana and I shouldn’t feel bereaved over all our billions of unicellular would-be children who we never got to know.
This is a test, Gatekeepers (I’m on my phone, and initially misspelled it “Hatekeepers”). Either your next comment shows comprehending engagement with opposing views, and a level of analysis above what you could find on Breitbart News, or else you’re banned from my comments section for one year.
Comment #103 December 7th, 2017 at 11:14 am
amy #95: But at some point it’s no longer enough to tell the people who’ve been working on solving a problem that their conversation is too narrow. At some point you have to do the actual work of broadening the conversation, which means arguing for a new and unfamiliar position. It also means engaging the existing conversation where it is, which entails showing why the existing views are inadequate to the problem—not just why outsiders would consider those views to be “weird.”
I’ve been on the other side of this over and over, with much less contentious topics: like, I’ll wander into an unfamiliar subarea of quantum information or theoretical computer science, and I’ll find the questions that people are foregrounding to be just weird. Surely I can shake things up with my broader, outsider’s perspective! But then I’ll learn more about the subarea, and I’ll realize: no, those weird questions are exactly the ones you’d have to answer if you wanted to make progress. No wonder people obsess about them so much!
Incidentally, I don’t know how well you know the “rationality community,” but it’s perhaps surprisingly far from being a “sausage fest.” Prominent and well-respected women in that community include Julia Galef, Brienne Yudkowsky, Sarah Constantin, Katja Grace, Kate Donovan … or does it not “really count” if the way these women think and talk about issues like AI isn’t obviously all that different from the way the guys do?
Comment #104 December 7th, 2017 at 11:29 am
Scott #102,
As always, I admire your willingness to engage my arguments. On the ban threat: this is the main reason I comment anonymously, so I am free to discuss without fearing repercussions. Being banned from your blog is nothing from having to fear being fired, for example. Vindictiveness is a very liberal trait, and I have plenty of examples to point to: from your own experiences with the SJW, to Brendan Eich, to more recently James D’Amore. It’s your blog, so you do whatever the heck you want to do with my comments :).
On the two topics at hand:
– Penrose. I agree with him in part, I say “I don’t know” on the other stuff. The part I agree with is his reasoning that Godel essentially deals a death blow to “mechanism” as explained here https://plato.stanford.edu/entries/goedel-incompleteness/#GdeArgAgaMec . However, as to the specific physical mechanism that causes human minds to be different from the result of “number crunching alone”, my answer is “I don’t know”. The Google talk by Robert Burton just scratches the surface of the problem that goes beyond our minds ability to build the statement “this statement cannot be proved”. As Burton says, for 2500 years the most brilliant minds have struggled with the problem of what the mind is in the context of things like free will, for example. It takes a lot of arrogance to call all these people wrong and say that “number crunching” is the answer to who we are as humans, in our entirety: body, mind and spirit. One of the differences I have noticed between American society at large, and the different European societies I know better -but from talking to friends it seems to be common to most European countries both form Western and Eastern Europe- is that Europeans are very mindful of what preceded them. Perhaps because WWII was such a traumatic event over there given that Europe seemed to have converged around the idea of totalitarianism: Nazism and Communism. The study of history and philosophy are part of K-12 education over there. During my high school education I received as much a strong background on physics and math of the kind you have in AP classes, as well as on history and philosophy, studying in detail the works of everybody from Aristotle to Marx, Nietzsche, Karl Popper or Jean Paul Sartre. While there are American high schools -both public and private- that take a similar rigorous approach to teaching kids, my experience is that American education is heavily focused on developing the students’ unique gifts and as a whole, what students can do with their unique gifts in the future. I am not saying that one is better than the other. Clearly, when it comes to producing technological breakthroughs, the American system would seem superior, given the results both at the basic research level as well as the companies that have commercialized them. However, when it comes to being able to discuss certain matters of deep importance, like “what is the mind”, this lack of consideration for the historical context makes otherwise very bright Americans to fall for all sorts of nonsense like the notion that humans are exclusively the result of mechanical number crunching.
– On the issue of abortion. I fundamentally disagree with your reasoning. Conception -or fertilization, pick your word- starts an entirely new human life, with its own unique DNA that would never be repeated again in the history of humanity. That much we now from materialistic science alone. So if your mother had decided to abort you, your mother could have had other kids, but you would have been denied the chance of existing after having been conceived. The same goes to your wife and your children. Among other stuff, I do a lot of prolife activism -mostly behind the scenes because of the vindictiveness of prolife opponents. One of the most powerful experiences I have had as part of this work, is a comment by a man who was just passing by when I was demonstrating in front of a local Planned Parenthood facility. He said “thank you for doing what you do, my ex-wife aborted two of my children”. In that sentence, we have what the pro-choice/pro-life debate is really about. Abortion terminates the right of a human life to exist. The rationale you have employed to justify why certain very young humans should not have a right to exist can also be extended to infants -Princeton professor Peter Singer makes precisely that argument-, to the disabled, to people in a coma, etc. So I fundamentally disagree with your view that we can artificially decide, using “rules made up by powerful men”, what constitutes a life worth preserving, once that life has been created. This is not to say that I do not understand that there are situations in which abortion is the best of a set of very bad options. For example, say the life of the mother is in physical danger, I understand if somebody has to die, that a deference be given to the mother. I am also in favor of making the day after pill available to deal with situations like rape. In the latter case, there is the doubt as to whether fertilization has actually happened and I am OK with saying that helping a victim of rape overcome her traumatic experience takes preference over having an answer to the question. But how we think about the human life that is created after conception says a lot about who we are about individuals. Do we think that this new life is as precious as any other human life and that this new human deserves as much protection as human lives that make it out of the womb, or do we think that this human life is disposable by creating an artificial “age based” delineation in what is clearly a continuous process. Further, do we make that delineation technology dependent? From last month http://www.cnn.com/2017/11/08/health/premature-baby-21-weeks-survivor-profile/index.html . These babies wouldn’t have been able to survive just 10 or 20 years ago. Does this mean that abortion 21 weeks fetuses was OK in the 1990s but not now? What about if technology advances so far as to permit the entire pregnancy to be carried out outside the woman’s body? For all these reasons, the only point, I believe, where there is a clear “Dirac delta” type of delineation between a new human life, and only “disjointed genetic human material” is conception and I am prolife.
Comment #105 December 7th, 2017 at 12:28 pm
Am I the only one who was more interested in the Witten discussion? (although I think it is very related to the AI discussion everyone is having)
I wonder if the analogy between…
Real Numbers : Rational Numbers
and
Wave Amplitudes : Observable Measurements
…tells us anything.
Maybe there is a Platonic, infinite, Metaverse potential, but all existence is grounded in the finite.
I should probably plug my movie “Digital Physics” again 🙂
Comment #106 December 7th, 2017 at 12:39 pm
Fred #99 and Mateus #98 and mjgeddes # 97 and sniffnoy #96
Since this discussion is beginning to spin out of control, let me try to reset it with the main points I have been trying to make, made a different way and with a different emphasis. And let me start with what I take to be things everyone agrees on.
1) Any new technology, from industrial to medical to computational, may have unintended side-effects beside the goals that were aimed at. See: thalidomide. Since the thing we are calling AGI is a new computational architecture it requires the same forethought and testing and so on, as best we can do. And this applies as well to old architecture (non-G AI) deployed at new scales and in new settings, etc. No debate.
2) There are people, human beings, with bad motivations. Computers are a tool they use. As that tool becomes more efficient and powerful at what it does, so does their power to do mischief. Further, as computers become more complicated and programs larger, it becomes harder to check all the ways that a malicious adversary might try to attack it. This is what Yudkowsky is talking about when he discusses the “security mindset”. None of this has anything at all to do with AGI as such. It is as true for specialized computers as the programs get bigger and unwieldier and pile judge on kludge and are produced by teams that are not entirely clear about what one another are doing in detail. As I said, a hacker using a computer to assist him or her would want a special-purpose hacking computer, not a “general purpose” computer.
So we have the programmer error/unintended consequences problem, which is not new. It may become worse as programs become bigger, but this has zero to do with AGI per se.
And we have the humans-with-bad-motivations problem. That also is not new. That requires the security mindset. Now if AGI architecture is more vulnerable to hacking (because, for example, we really haven’t a clue to how it is really working) then that’s a good reason not to use it where hackers might want to get in. Use specialized, non-AGI architecture. And that is probably what will happen in any case: why use a general-purpose gadget for a specialized and highly important purpose? So again, AGI is sort of irrelevant.
Any objections to 1) and 2)?
So what about AGI? The actual architecture of the machine is different from some specialized machine programmed for some specialized purpose, and that may create some new problems. But what exactly is the nature of those problems? Here I want to insist that just *calling* this situation one with “superintelligent” computers is a really, really bad idea. It is systematically misleading. And the comments here seem to reflect that.
Let’s start here. I hope that we all agree that a hand calculator has exactly zero—no—intelligence in the normal sense of that term. It is a tool, designed by intelligent people for a purpose. If it is well designed, reliable and simple in programming, then it reflects the intelligence of the designers, but it itself in not intelligent. To any degree. Zero.
It may produce results that only a quite intelligent human—or, more accurately, no human at all—can produce. So it is a great tool to use if you need accurate calculations. But to ask, for example, how smart a person would have to be to do what the calculator does is to miss the point completely.
In fact, the only examples of intelligent entities we have are humans and other animals. To say, as is said above, that bacteria are “intelligent” because they evolve fast enough to come to be able to digest nylon just a few decades after the invention of nylon is to misuse the term “intelligent”.
Any objections so far?
Chess-playing and go-playing machines are also not intelligent in the normal sense of the term. They produce behavior that only high-intelligence humans can produce, and even that high-intelligence humans cannot produce (cf: calculator), but not because they actually are intelligent. And this can sometimes be verified by them making mistakes that no vaguely competent (much less expert) human chess-player would make. Similarly for other “regular AI” machines. It would help if Turing had not gotten this whole discussion off on the wrong foot with the “Turing test”, which was the reflection of a very bad verificationist semantics that no philosopher today would hold. If you take the Turing test as just a stipulation about how to use a word written and pronounced “intelligent” then it is a silly stipulation, because you are inviting confusion with the English term “intelligent”. If you think that Turing actually explicated the English term, then you are just wrong.
Any objections?
OK: so what is the real situation? It appears that a certain kind of novel computer architecture—call it “neural net” if you like, but don’t assume it produces what actual neurons in brains do—may come on the scene soon. This architecture will be “trained” rather than “programmed” to do specific tasks, and can be trained on a wide variety of tasks. Hence it deserves to be called “general” rather than “specialized”. It can also be made a more and more powerful tool pretty straightforwardly, if I understand correctly, just by increases in processing speed, number of transistors, etc.
So this new tool—call it AGI but be wary about the I because this is no more intelligent, in the normal sense, than my calculator—may come to be used widely. That may be because of the G: as a more general-purpose tool it is more flexible in certain ways. More interestingly, it may be because it simply outperforms the specialized machines even in their own specialized tasks. And with that new, widespread architecture there may come new risks. Are they qualitatively different from the risks and problems we now have with computers? It is hard to see why, but if they are then maybe we should just avoid using the AGI architecture for some tasks (like paperclip making?). But in any case, let’s clearly identify the problem we are worrying about.
I have tried to outline the situation—not denying there are risks to worry about—without ever using the term “intelligence”, much less the hyperbolic “superintelligence”. Those words are massively misleading, because they suggest the risk of a highly intelligent agent (in the English meaning of “intelligent”) trying to achieve some aim. They invite what Dennett calls “taking the intentional stance” in thinking about these machines. But the intentional stance, when dealing with computers, is just using a false model, the model of an intelligent human, as a quick-and-dirty short-cut for anticipating what the computer will do. In Yudkowsky’s terms, this is exactly what the “security mindset” warns *against*: falling prey to a narrative that blinds you to a vulnerability. If you regard these machines as actually intelligent, much less superintelligent, then you will be caught off-guard by problems they cause that even a mildly intelligent person would not. So just junk the term “intelligent” in this context. These are very powerful computing devices, that can do things we might not have thought possible. If we come to rely on them, then we should think hard about how they could go wrong. If they are going to be used by real human adversaries, which they will be, then think about how they might use them and what vulnerabilities they have. But stop with all the talk of the thing “wanting” to do this or that, and certainly whether we are “accidentally building conscious minds”. There is no plausible prospect of doing that. In fact, of all the various approaches to the problem of consciousness, the computational theory is perhaps the only one that can be decisively refuted. We haven’t a clue how to create consciousness, and are not trying. We also have very, very little understanding of human intelligence and are not trying to artificially create it.Let’s stick with what we are actually trying to do and what we plausibly will soon be able to do.
And in a nod to Amy: there is nothing wrong with discussing the possible dangers of these computers—it would be irresponsible to ignore them—, but not to the exclusion of discussing the possible benefits as well. As with any new technology.
Comment #107 December 7th, 2017 at 12:51 pm
migeddes #97 and Jon K. #105: I’m putting forward a new ground rule, which is, no using my comment section for repeated flogging of personal hobbyhorses. Where by “personal hobbyhorse,” I mean an argument that might even be relevant to the post or discussion, but which it would require considerable immersion in your personal philosophical writings (or movie, or whatever) for other readers to understand or evaluate. This rule does not represent a value judgment about the philosophical theories themselves—it’s only a statement about what this comment section is for, vaguely analogous to (say) the “No Original Research” rule on Wikipedia. 🙂
Comment #108 December 7th, 2017 at 1:17 pm
Gatekeepers #101
corrected:
“The notion that “synapse pumping” alone -and I don’t care how many neurons in parallel you have doing your synapse pumping, it is still synapse pumping- will ever produce anything remotely close to machine consciousness is so ridiculous that it could only be popular in the world of divorced-from-reality academia.”
Comment #109 December 7th, 2017 at 1:33 pm
For what it’s worth: unlike many of the “pro-AI” commenters on this thread, I do think there’s something mysterious about consciousness and free will. I’ve even written some nutty speculations myself about these mysteries, including The Ghost in the Quantum Turing Machine and Could a Quantum Computer Have Subjective Experience?. I’m open to the possibility that we’ll learn radical new things about these mysteries in the future, and even that quantum mechanics, thermodynamics, and early-universe cosmology will be part of the story.
But I’ve also been persuaded that our beliefs about consciousness and free will have essentially no bearing on whether we should worry about AI taking over the world. This is because there are two cases:
The first case is that any sufficiently intelligent entity would be “conscious” in whatever sense we’re conscious, and have “free will” in whatever sense we do. In this case the whole objection is vitiated.
The second case is that it’s possible to decouple intelligent behavior from consciousness and/or free will. In this case, AI taking over the world would arguably be even scarier than in the first case—for even if the AI created a world that we considered wonderful (rather than a giant ball of paperclips), there would be “nobody home” to experience it. In Nick Bostrom’s evocative phrase, it would be “a Disneyland without children.”
I know from experience that it’s easy to assent to the above dichotomy (which is, after all, basically tautological), but then silently fall back into thinking that if the AI wasn’t conscious, or didn’t have free will, then it would be effectively defanged as a threat. Resist that error.
Comment #110 December 7th, 2017 at 2:03 pm
Tim #106: Suppose that an AI worked by perfectly emulating a human brain neuron-by-neuron. In such a case, would you agree to the use of the word “intelligence” to describe what was happening? Likewise, suppose the AI emulated a system that was like a human brain, but with a thousand times as many neurons, and which you might get by taking existing humans and then doing millions of years of selection for increased intelligence (and the AI also sped things up by 1000x). In such a case, would you agree to the use of the term “superintelligence”?
Comment #111 December 7th, 2017 at 2:06 pm
I do not consider 104 minutes of someone’s time to be “considerable immersion” 🙂 And I would never consider the work of Fredkin, Wolfram, ‘t Hooft, Wheeler, Pythagoras, Tegmark, Chaitin, Wildberger, etc to be my own “personal philosophical writing”. A misinterpretation of their work?… sure, but not the personal ramblings of some crank 🙂
My post was just throwing out a quick analogy on the relationship between the continuum and the discrete, waves/particles, amplitudes/observable, Reals/rationals, infinite/finite, analog/digital, fields/automata, noncomputable/computable, etc… mind/body?
I thought that since you saw Witten being open to “it from (qu)bit” ideas he had once dismissed, then maybe you’d be open to a short comment that was trying to stimulate more thoughts.
…but I understand you’re trying to keep the crazy comments in check 😉
Comment #112 December 7th, 2017 at 2:16 pm
#109
“The second case is that it’s possible to decouple intelligent behavior from consciousness and/or free will.”
I think that is the case. My definition for what it’s worth.
Intelligence is a physical process that attempts to maximize the diversity and/or utility of future outcomes to achieve optimal solutions. We see this in the operation of Deep Blue where the machine developed sufficiently optimal strategies to defeat Gary Kasparov. We see this in slime molds that “can solve mazes, mimic the layout of man-made transportation networks and choose the healthiest food from a diverse menu—and all this without a brain or nervous system.
http://www.nature.com/news/how-brainless-slime-molds-redefine-intelligence-1.11811
https://broadspeculations.com/2015/01/19/thinking-about-thinking/
So AI, if was super intelligent and got loose, might very well be like a slime mold that would spread over the entire Earth with little regard for humans.
Comment #113 December 7th, 2017 at 2:24 pm
Scott #109
I’m “pro-AI”, but I believe too that consciousness is the only thing making our lives worth living, moment to moment, whether through pain or joy.
But one can recognize consciousness and reject “free will”.
You can derive scientific facts about consciousness by actually sitting down and observing how your mind works.
The sense of “self”, as the usual assumption that there is an unchanging center core to perception (like we are riding our own bodies, inside our heads) and that this core is the “author” of the thoughts and actions, and what is referred to in expressions like “being self-conscious”, can be revealed/felt as a clear illusion in a fundamental way. This illusion makes sense in terms of evolution as a reflection of the body grown sense of “unity” that needs preserving, and how we interact with other members of our species.
One analogy is how you can spend your entire life without knowing about your retinal blind spot, but once you are shown how to observe it, you can no longer deny it, and you now know how to look back at it to convince yourself again and again of its reality.
Such observations tend to suggest that consciousness has no direct causal power (i.e. it is not the author of our actions), but there are still subtle indirect ways in which consciousness could have causal effects.
For one thing, the fact that we are effectively talking about consciousness right now (consciousness as a concept has to be realized as a brain structure, itself one object felt in consciousness).
It’s possible to conceive an artificial brain that would talk about consciousness just like we do, but somehow the lights are off, but that seems far-fetched.
But since I’m certain of my own consciousness, one analogy would be that the brain is like a camera, with a lot of complex mechanisms, and consciousness is like its shadow. The shadow has no effect back on the camera. Until you turn the camera onto its own shadow, in which case the shadow affects the state of the camera, which itself can affect the form of the shadow, in a form of subtle endless feedback loop. But the analogy is imperfect because shadows are just as physical as cameras.
If we assume non-dualism, so that brain and consciousness are two sides of the same coin, and since consciousness can only affect the brain through its physiological mappings, and those are purely “mechanical”, then why is it there consciousness at all? That is the mystery.
Comment #114 December 7th, 2017 at 2:26 pm
Scott #110: As to the first case, it might not be quite as simple as it sounds (you might have to emulate more than the neurons: the glial cells, blood supply, etc., etc. to get something that did not just immediately emulate dying). But we’re being generous, so let’s grant all that. And assuming the result is something that acts like a human brain, then sure. I’d both call it intelligent (unless it was emulating Trump) and form expectations about what it would do the same way as I do for intelligent people. But of course in that case, you have not created anything more intelligent than a human.
The scaling-up case is really difficult to say. Just biologically, a thousand-times-larger brain would be a mess. But suppose you overcome that, would it be superintelligent? I have no idea at all. I have no reason to think that intelligence just scales indefinitely with the number of neurons: that depends on how the whole thing works. Computers are made to be scalable, but brains aren’t. Maybe the gigantic-brain would be a blithering idiot, or have interminable internal conflicts, or sink into profound depression. Our brains do so much more than just “think”. I don’t honestly think anyone has a clue about what would happen in such a case. We don’t even know what tweaks to a regular brain might make it “smarter”. So if the answer to your question seems obvious to you, I think you should review the assumptions that you are relying on.
Comment #115 December 7th, 2017 at 2:50 pm
Tim Maudlin #106:
This highlights an obvious-in-hindsight failure mode of Yudkowsky’s article, to wit, people who aren’t familiar with his background might not realize that he’s talking about security mindset as applied to the problem of designing of the goal systems of autonomous super-human domain-general optimizing agents.
The point of the article is that even though AGI safety is desperately unlikely even with security-professional grade paranoia, anything less is virtually certain doom. The security concern here isn’t that a hacker might subvert the AI; you can just drop that whole line of thought because it’s based on a misapprehension of the argument your interlocutors were trying to make. (Not your fault, just an unfortunate failure on our part to bridge an inferential gap.)
Comment #116 December 7th, 2017 at 2:56 pm
Tim #114: Good, I think the discussion is making progress! The next step is this:
We know from biology that the architecture of the human brain was shaped by tradeoffs among numerous factors: intelligence, sure, but also metabolic costs and the width of the birth canal. But if so, then why should anyone imagine that the human brain is anywhere close to the physical limit of intelligence (however one chooses to measure it)? That would be a strange and unexplained coincidence. Indeed, we need merely look at the intelligence variations between humans: how far beyond Gauss or von Neumann would a system need to be, if at all, before a term like “superintelligence” would be appropriate?
Comment #117 December 7th, 2017 at 3:05 pm
“But if I ask you to program a computer to feel pain—not to exhibit pain behavior or make appropriate pain sounds but to actually feel the sensation of pain—then I claim neither you nor anybody else has a clue about how to achieve that.”
I can and have programmed computer routines to feel pain – in that sense that what they are doing in certain cases is discouraged with increasing force until they stop doing it.
What they feel or what an intelligent computer would feel or whether it would have consciousness strikes me as the sort of non-problem that I think of as “the scent of a rose”.
Why does a rose smell like a rose? We can trace the chemistry through scent receptors and through nervous systems up to the brain, and we can build mechanical devices to reliably detect a rose’s scent – but why does a rose smell like a rose and an orange smell like a rose? Why doesn’t a rose smell like an orange and an orange smell like a rose?
Because those particular chemicals produce that particular scent as processed by our particular scent organs in this particular universe; in another universe maybe a rose smells like an orange; or maybe to a scent processing computer it would. As long as the rose and orange scents are distinct, who cares?
It’s the same, as I see it, for what the Windows program feels like as it acts as the primitive consciousness of my laptop, interpreting external signals, passing them on to internal programs such as Excel which it does not monitor or control, and passing the results back to the external world. As long as it has a way to do its job I don’t care what it feels like as it does it.
(I am hoping Amy will not pass that off that a typical male reaction as I was trying to get at something a bit subtler. I can care about a person or a computer’s pain without caring what the specific sensation feels like.)
Comment #118 December 7th, 2017 at 3:49 pm
Tim #114
“But of course in that case, you have not created anything more intelligent than a human.”
But since now it’s running on computer hardware, you could get the usual scaling in speed of execution (using more power as a trade-off).
Then you can realize various such human brains, and have then connected to each other in normal ways human communicate, just at blazing fast speeds.
So within a few minutes of computation time, you’d get the output of years and years worth of Einstein, Dirac, Schrodinger, Bore, … doing theoretical physics using math and communicating with each other.
No matter what weight you’re now putting on “solving” Go, it shows that, in this case, a few hours of “number crunching” is re-discovering 2,500 years worth of accrued human wisdom.
Comment #119 December 7th, 2017 at 4:11 pm
JimV
#117
well one could say that the structure of the data is what dictates what it “feels” like (just like the number 7 is objectively different from the number 12 in a finite set of ways).
An orange feels different from a rose because the brain stores each differently. And variations of oranges feel similar because the brain stores them similarly.
If you consider what vision feels like, we take it so much for granted that we identify it with the real world.
We don’t think of it as our brain interpreting 2D retinal data, but more as the outside world reaching/rushing directly inside our head, our eyes acting as simple funnels.
The distinction is maybe more obvious to people who suffer from distortions in the visual field (like the aura sensation preceding a migraine).
And I suspect that each event that enters our field of consciousness corresponds to the actual formation of a memory or recall of a memory.
Is it possible to suddenly remember things that we never have experienced in awareness? (this could be tested)
That said, this is all about the content of consciousness, and not why there is consciousness itself (i.e. the “knowing” – smelling vs what is being smelled, the seeing vs what is being seen,… ).
Comment #120 December 7th, 2017 at 4:30 pm
Scott # 166: Sure, we know that it is within the capacity of the human brain as it is to produce a Shakespeare or an Einstein or a Beethoven or a Ramanujan, so we know that brains are variable with respect to anything you might want to call “intelligence”. (It is a limitation of this whole discussion that we are acting as if there were one such quantifiable feature, but let’s let that pass for now.) So if we really understood how brains produce or support intelligence we could figure out how to make our emulation more intelligent. And maybe we could even figure out to what extent the system can be scaled and become more intelligent without collapsing in some way. As you say, there is reason to think that there is room there to some extent. That the brain could absorb a thousand-fold increase and just keep getting “more intelligent” all the time is mere speculation, based on nothing, as far as I can see.
I doubt we have any disagreement here. But let’s double back to the question at hand. The fact is that at the moment we really have very little idea how brains do what they do, cognitively, emotionally, etc. There is no present push to emulate brains, and that is not what the AGI debate is about. In fact, if we were following the brain-emulation paradigm then real ethical quandaries about whether we should continue would arise. What we are doing now is plain vanilla CS: figuring how to build faster and more computationally powerful machines. I guess in some sense they may soon become “superfast” or “superpowerful” but not in the foreseeable future “superintelligent” or even “intelligent”. So a discussion of the risks and rewards for superfast or super powerful computing seem eminently reasonable, but I think it should take a rather different shape than “superintelligent” seems to imply.
Comment #121 December 7th, 2017 at 4:56 pm
fred # 114: Whoa, Whoa! Slow down there! I was nice enough just to give you a brain that is not even attached to an emulated body! But now you have to fit out all the brains with bodies, and an environment and places to live and others to interact with in all sorts of ways, and an ambient culture and on and on and on! I guess you can cut out a big chunk of the entire universe leaving only, say, the solar system (or maybe the earth and Sun and moon). Or do you think that disembodied brains would be fine with just having highbrow conversations? At this point what are we even talking about? What you are postulating will not only not happen in the next few decades, I would wager it will never happen. (Well, I wouldn’t actually wager because I could not ever collect.) Anyway, it has exactly nothing to do with any “superintelligent AGI” that is just around the corner.
By the way, I really don’t like Bohr, but Bore is a bit much…
Comment #122 December 7th, 2017 at 4:59 pm
Jim V # 117: You have not programmed anything to feel pain, and you know you haven’t, or you would never run the program. Are you a moral monster? I assume not.
Comment #123 December 7th, 2017 at 5:12 pm
Hi JimV – no, actually I hadn’t read it as a male thing, but maybe that’s because it’s a familiar description in the largely male STEM world I live in (and the much more male STEM world I grew up in), and there are gendered things to do with pain and orange-scent that just haven’t occurred to me because there’s been no audible conversation about it, not beyond a sort of add-on that says “women do in fact experience pain more intensely than men do, here is the biochemistry”. I’m also personally inclined to be quite happy with the soft-machine idea of humanity, even as I read the then-playwright Havel’s letters from prison in which he’s trying to define what it is that makes us human.
It could be, though, that by introducing these conversations to areas where a diversity of people really know a lot about pain — for instance, communities of people who live with chronic pain — we really would get, or they would develop, a whole new set of constructs, even gendered ones, for talking about not just the idea of engineering pain or smell in silico but the questions of why we are talking about such things in the first place; what they’re for or shouldn’t be for, and why.
Scott #103, as I was writing that last response, it occurred to me that this is comparable to the Brexit problem. Had the governments of Europe been able to join the EU and the eurozone without referenda, I’m sure they would have, just as people working on AI can introduce AI into society without really anyone’s permission. But they did need democratic permission, and at the time, though there were plenty in Brussels who thought it was stupid that they should have to sell this to the masses, there were also plenty who thought it was reasonable: after all, this was going to change people’s lives and socieites fairly dramatically and they weren’t going to have an opt-out.
So they did put on quite a sales job. It was all cast in the trade and international-relations language of the time and slathered in prosperity talk and other such propaganda, and though by then I was stateside again, so I wasn’t actually there to hear the conversation in the pubs, it did not sound to me as though there was a broad and lively conversation happening. The language all stayed quite narrow, and the terms of discussion just cycled around and around.
One of the great shocks of the Brexit proceedings, for me, was the number of people in Britain who had *no idea why they were in the EU in the first place*. No idea why it existed. No idea what a eurozone was for. No idea what happens without a customs union. Once the referenda were passed, the conversation largely shut off, and while all this stuff was still very live in my mind because I took part in it during a formative time of my life, it was dead and invisible history to a very large proportion of people voting.
The conversations had not happened. The range of concerns and conceptions outside trade and IR had never developed except in xenophobia. Now, you could say, “Well then, the ladies who live in council flats and take the Number 97 to do their shopping, and the immigrants who own minicabs companies, and the retired librarians ought to have turned up at the universities’ IR departments and gotten in on the conversation,” but of course they weren’t going to do that. For one thing, their part of the conversation wouldn’t have been admitted. They’d have been invited (after some skepticism) to sit down and learn how to talk as though they belonged in an IR department, and to learn to work on the problems that the IR community had already decided were important in a mode acceptable to IR.
For another, they don’t know what an IR department is. Nor do they know anybody there. So — did the trade and EU theorists and MEPs and all the rest really have to listen to and consider the foundation-cracking sorts of things those people might have said all along? Nope.
Result: not good. Not good for anyone. The conversation is not developed. The theoretical frameworks that might have come from those conversations do not exist. And because the EU does affect an entire society, in the end, in a democracy, the
I think on the other hand of various social movements that are still alive partly because there was actually some effort to leave home and find out what was missing from the terms and intent of the conversation in the first place. Oops, student, back soon.
Comment #124 December 7th, 2017 at 5:25 pm
Tim #120: Two replies.
That the brain could absorb a thousand-fold increase and just keep getting “more intelligent” all the time is mere speculation, based on nothing, as far as I can see.
If you can emulate a human brain on a computer at all, then as fred #118 points out above, one thing you can almost certainly do is emulate a brain at 10,000x speed, just by throwing more hardware at the problem. You could also prevent all the “merely physiological” causes of aging, back a brain up to a previous state when needed, etc.
Since I fear these might get swatted away as details, let me take some time to dwell on the point. Could you have discovered GR, given the facts that were available to Einstein, as well as ten thousand subjective years to sit and think about the problem? I’m pretty sure I could have—merely because ten thousand years is a lot of time for even I to get really, really good at contracting tensor indices and so forth, even granting a few thousand years to chase blind alleys. OK, so now speed things up by a further factor of a thousand. This might not get you all of “intelligence”—for any given person, there might be some ideas that they’d never have even given 10100 years to think of them—but it would give the emulations such a staggering advantage over wetware humans, that it’s hard to see how the latter would still even be relevant.
What we are doing now is plain vanilla CS: figuring how to build faster and more computationally powerful machines. I guess in some sense they may soon become “superfast” or “superpowerful” but not in the foreseeable future “superintelligent” or even “intelligent”.
The above passage succinctly encapsulates our central disagreement: you keep asserting a bright line, with everything AI might do in the foreseeable future on one side of it, and everything that’s “actually intelligent” on the other side. Whereas I see no a-priori consideration that would let me say whether such a line exists, or whether it’s nothing but a gradual slope all the way up.
Before chess programs got good, there were people who talked about the unbridgeable chasm between mere number-crunching tasks on the one hand, and creative pursuits like chess on the other. After Deep Blue and before AlphaGo, people talked about the unbridgeable chasm between chess, for which brute-force search suffices, and Go, for which it doesn’t. (I know because I was there and I heard them!)
Yes, it’s conceivable that the next chasm, or the one after that, really will be unbridgeable without radical new insights: unlike some AI enthusiasts, I don’t regard the possibility as absurd. But right now, the deep learning people don’t even seem close to exhausting what can be done without radical new insights. So, lacking any theorem or principle that tells me that the current tools might handle X, but definitely never Y, the best I can do is watch, try my best to update on observations, and decline to make confident pronouncements based on types of intuition that have repeatedly been wrong in the past.
Comment #125 December 7th, 2017 at 6:38 pm
Scott # 124: You have painted out our disagreement wrong, and in fact maybe there is no disagreement at all. First, who knows what I would do if suddenly made so I can live ten thousand years? How do I live? Who else is there? Maybe I become overwhelmed with procrastination. How do I find this out? By a “voice from the sky”? Maybe I become a religious maniac. Maybe I become paranoid that I have been lied to. What do the others around me think? If they figure they are all creations of some computer programmer, made to solve some problem in the programmers world, do they all go on strike? Who knows?
People made bad predictions about chess-playing computers because they figured that playing chess well requires what is normally called “intelligence”. What we have found out is that it doesn’t. As I said, some chess-playing computer that were very strong players have made mistakes no moderately intelligent human would make because the endgame lay beyond their search horizon. Penrose has an example. And some story-generating programs have made mistakes that no human in history has ever made by getting into loops. But most of the stories seem just fine. All this shows is that you can produce much of the outward behavior that arises from intelligence without having any real intelligence. It is surprising, perhaps, how far this can go.
Will AI eventually turn from what it is doing now, which has nothing to do with emulating actual intelligent thinking, to caring about how brains do what they do? Maybe. And if they do, will they get funding? I hope so. And if they get funding, will they succeed? Who knows?
So I don’t see a “bright line” of the sort you mention. I don’t think we are even trying to head in the direction of artificial intelligence. We are instead doing the less lofty but still noble work of making better and better computers. Nothing wrong with that. And I have no beliefs at all about what we can “eventually” do. But for the foreseeable future, I don’t see silicon-based intelligence on the horizon.
Comment #126 December 7th, 2017 at 7:10 pm
Regarding the use of No Free Lunch to argue that AGI can’t outperform humans and the like, I’m reminded of another context where one sees No Free Lunch used. William Dembski, one of the major intelligent design proponents, based it as one of his central arguments against evolution. Jeffrey Shallit discusses that here https://cs.uwaterloo.ca/~shallit/nflr3.pdf . Chollet is a much brighter and more careful thinker than Dembski, but one can’t help but notice the similarity in the nature of the arguments, in that both see to be treating the No Free Lunch Theorem like our universe is essentially completely and utterly random. And that’s not the case.
Comment #127 December 7th, 2017 at 7:24 pm
Corey #115:
The thing is, Eliezer does make that pretty explicit. Like, when Tim writes in #106 that
I can only conclude he didn’t actually read the dialogues. The bit about what’s relevant is not an intelligent adversary but rather any optimization process, is very early in the first one. And later in the first one he breaks character for a bit to directly talk about the application to AGI. So Tim’s claims about what Yudkowsky says are just wrong. I mean, Tim is under no obligation to read these, but if he wants to go making claims about what they say, well…
Honestly, I think I’m done here; sorry Tim, but your #106 just contains too little actual attempts to rebut people’s counterarguments and too much repeating of arguments that have already been addressed. And when you go saying that Yudkowsky says such-and-such when his actual writing directly contradicts that, well, what sensible response to that is possible?
You should probably just go read “Superintelligence” anyway. 😛
Comment #128 December 7th, 2017 at 8:01 pm
Sniffoy # 127: Well, what you “can only conclude” happens to be false, so maybe your conclusion-drawing algorithms need to be examined.
Here is the “breaking our of character” bit where AGI is mentioned:
“For example, stepping out of character for a moment, the author of this dialogue has sometimes been known to discuss the alignment problem for Artificial General Intelligence. He was talking at one point about trying to measure rates of improvement inside a growing AI system, so that it would not do too much thinking with humans out of the loop if a breakthrough occurred while the system was running overnight. The person he was talking to replied that, to him, it seemed unlikely that an AGI would gain in power that fast. To which the author replied, more or less:…”
Now if you want to claim that that comment suddenly makes obvious that this is really a dialog about possible *problems* arising from AGI, please make the argument. If not, then maybe an apology for calling me a liar is in order. And if you can’t bring yourself to that decent act, continued silence is the next best.
Comment #129 December 7th, 2017 at 8:07 pm
Sniffnoy #127,
Gotta disagree that it’s explicit — until the second-last paragraph, that is. Of course I went back and looked at the article before I wrote my comment to see why Tim had got side-tracked on the hacker thing. I went straight to the breaking character part first and even there AI alignment is only mentioned in passing with the good stuff behind a link.
Comment #130 December 7th, 2017 at 8:32 pm
Fred #108 , for the direct reference, and the rest of the crowd.
Yeah, you can use the statement “synapse pumping” if you like but I generally agree with the sentiment of your comment.
In this 2010 Google talk https://www.youtube.com/watch?v=f477FnTe1M0 , Roger Penrose was asked during the Q&A about some professor who had a project of building a computer based on cloning how the human works via neurons and who had predicted that in “10 years” (10 years from 2010, that is), he would have cloned via this method the human brain with all its abilities and that by the process of cloning it, he would have a fully equivalent human. Penrose response’s was something like “it’s always 10 years into the future, isn’t it?” with a sarcastic British smile.
Godel’s insight puts a limit of what can be known via “number crunching”/”synapse pumping”. That’s just a fact when restricted to statements of logic alone like “this statement cannot be proved”.
Then there are human beings who are much richer than what can be known via the epistemology of mathematical logic. You don’t have to focus on Einstein, Godel or Gauss to see this. Any human being is capable of consciousness, feelings, free will (without “free will” there is no room for convicting anybody of anything) and moral behavior. In fact, every time each of us does something as simple as crossing the street or driving on a highway is trusting that the overwhelming majority of other human beings will behave in a moral manner towards their fellow human beings and won’t attempt to say kill us to get our car because it is a better car.
AI is essentially in the same state it was in the 1970s. Only today there is more data and more computers. Its limits as to what it can do remain the same because there are the limits intrinsic to what can be known via “number crunching”/”synapse pumping”. It’s a pity that fewer and fewer computer science experts are familiar with the concept of “fundamental limit” and try, every few years, to convince the rest of the world that “fundamental limits” are like unicorns.
My nickname is “the problem with gatekeepers”. I get that the academic gatekeepers in the field of artificial intelligence, for the most part, are a bunch of materialists who believe that “number crunching”/”synapse pumping” is all there is to the human condition. But then again, these same gatekeepers are surprised when the people outside their little bubble decide that they don’t want to subsidize with their taxes said bubble.
I can comfortably make the following prediction: I won’t see in my lifetime any artifact that comes anywhere remotely close to a conscious human being. I will probably see self-driving cars and robots doing all kinds of stuff to make life easier for humans, but they will remain “number crunching” machines that do what they are being told to do.
Comment #131 December 7th, 2017 at 9:52 pm
Gatekeepers #130: Here’s one question that Penrose (with whom I had the privilege of debating these questions in Minnesota a couple years ago) has never been able to answer. If Gödel’s Theorem puts a fundamental limit on the behavior of AIs, then what, exactly, is the behavior that it prevents them from?
Saying that it assents to the Gödel sentence? Clearly not, since it’s trivial to program an AI to do that.
Indeed, as Turing pointed out in 1950, if you let an AI assume the Gödel sentence as an axiom—just as a human would—the AI can derive all the same consequences that a human can. And if you let the AI occasionally make mistakes—just as human mathematicians do—then it can guess new axioms, including Gödel sentences for whatever formal systems or collections of formal systems it’s working with.
Faced with these observations, Penrose is forced to retreat back to internal criteria: sure, the AI could exhibit exactly the same behavior as a human mathematician, but it wouldn’t really understand what it was doing! E.g., it might say it believed in the consistency of ZFC, but it wouldn’t really mean it the way I do!
To which I reply: if that’s where we’re going to end up, then why even bring Gödel into it in the first place? Why isn’t the whole Gödel argument just a more abstruse reformulation of the oldest objection to AI in the book: granted, an AI might behave identically to a human in every respect, but still, “no one would really be home,” because it’s just an AI?
As a metaphysical stance, this is permanently safe from refutation. But observe that, even if it’s true, the practical implications of AI could still be every bit as great as the world’s biggest AI booster or AI alarmist says. And the practical implications are what we’re talking about in this thread.
Your insinuation that AI research ought to be defunded, because AI is in the same state as it was in the 1970s and also because AI researchers are “a bunch of materialists,” makes clearer what your actual motivations are (and how much they differ from the good Sir Penrose’s). Particularly when we juxtapose these remarks with your concession—in the very next paragraph—that AI research will probably lead to “self-driving cars and robots doing all kinds of stuff to make life easier for humans.”
Wow, that doesn’t sound like a field in stasis! In fact it sounds like a big economic driver for whatever countries invest in it—i.e., the very sort of thing that granting agencies exist to fund! The metaphysics of consciousness be damned.
Unless, of course, the goal were neither truth nor prosperity, but resentment and revenge, regardless of the consequences. I.e., punishing the “elite scientific gatekeepers” whose actual success at solving problems and pushing civilization forward constitutes an implicit rebuke to the narrow, the ignorant, and the self-satisfied. In every word you write, we see the screeching heart of Trumpism.
Comment #132 December 7th, 2017 at 10:29 pm
Scott
In #107 you said ” I’m putting forward a new ground rule, which is, no using my comment section for repeated flogging of personal hobbyhorses”.
Your obsession with seeing Trumpism in everybody who disagrees with you seems to fit the new rule, I am just saying!!
Going back to your comment #131:
“if you let an AI assume the Gödel sentence as an axiom”…
You still need a “human” or something outside the AI system to have that AI system assume the Godel sentence. And it is too pretentious to say that an AI system will “magically” make the same mistakes as say Yutaka Taniyama whose mentor Goro Shimura said was known for making “good mistakes”. The Taniyama-Shimura conjecture is know known as the modularity theorem and was proved by Andre Wiles. It all started as a mistake though. Good luck building an AI system which is that good making mistakes. It takes a human mind to see what a “good mistake” is.
It’s like the famous observation that even a monkey typing randomly on a computer will at some point be able to produce Shakespeare’s works. It still takes an external mind to identify that a list of characters is a Shakespeare work.
From my point of view, what Godel gives is the necessity of having that “external, independent observer” to be able to know everything that is true under the rules of the formal system. Executing rules alone is not enough. And yet, that’s the only thing computers do: execute rules. Godel’s is the human mind that carefully constructed the statement “this statement cannot be proved”. It wasn’t any AI system randomly making mistakes.
The applications of AI technology I mentioned do not need federal sponsorship to become reality except as they pertain to national security, so I don’t see why tax payers should subsidize any AI research outside those in the realm of defense.
My comment referred to the fact that academia is a gated intellectual community when it comes to opinions about the potential of AI and the metaphysics of the human mind. It’s a logic consequence of the tenure system: the gatekeepers only let in people like themselves. While theoretically only merit counts and in STEM fields “merit” is more easy to define in objective terms than in softer fields, it is still the case that PhD programs are as much a way for professors to train their successors as it is for aspiring professors to get a a grasp of what the professional life of a professor looks like. The data shows that academia has become increasingly liberal over the last few decades. I don’t think that’s because conservatives are dumber now than they were say 50 years ago -as popular as that opinion might be among the gatekeepers themselves. There is a limit as to how much “Trump is an idiot” that conservatives apprentices can take before they decide that they rather apply their talents in endeavors where success is defined by the market and not by what anti-Trump gatekeepers think success is. I think that what is happening in academia is something as American as “voting with one’s feet”. Thus people shouldn’t be surprised that making academia a hostile work environment for non-liberals results in non liberals -when in power- asking themselves why they should continue to subsidize a system that by and large benefits liberals, specially when these liberals are unable to articulate how American society at large benefits from the system.
Comment #133 December 7th, 2017 at 10:52 pm
OK, I’m closing this thread tomorrow, because I don’t have time to keep up with it. Thanks to everyone for participating.
Comment #134 December 7th, 2017 at 11:42 pm
Yes, it seems I misremembered what Eliezer Yudkowsky wrote. He did write about applying the security mindset to AGI, but not in the way I remembered — the suggested security measure was still a “roadblock”. However the first part remains the same — security doesn’t have to be against malicious people, and the abstraction he talks about definitely applies to AGI (which is why he does indeed apply it to AGI later).
Tim, I did not call you a liar. People can be very wrong about something they read for reasons other than deliberately lying. If anytime someone does as I did, you say that they are calling you a liar and must apologize, people are not going to be very willing to argue with you. As indeed I am not; it does still seem to me that further arguing with you is unlikely to be productive, so I am not bothering with any real argument here and will end it with that.
Comment #135 December 8th, 2017 at 3:20 am
Sniffnoy #134: You are making some pretty fine and fancy distinctions here.
You: Go read the dialogues.
Me: Yudkowsky is talking about security against hackers.
You (direct quote): I can only conclude he didn’t actually read the dialogues
You (direct quote):And when you go saying that Yudkowsky says such-and-such when his actual writing directly contradicts that, well, what sensible response to that is possible?
So you directly assert that I did not actually read the dialogues but was acting as if I had. If you want to make a distinction between that and calling me a liar it would be fascinating to hear what the distinction is.
I laid out my argument, point by point, asking at various places if there were any objections. If you had any, and could articulate any grounds for them, I assume you would. Since you have not, I conclude that you have none, and your withdrawal from the discussion is strategic, to avoid embarrassment, covered by a bunch of empty insults and a slur on my character.
Compare your behavior to Corey # 115. Before accusing me of completely missing the point of the dialogue, he went and checked, and saw that in fact the obvious way to read it is about security against hacking. That may not be what Yudkowsky had in mind, but Corey was quite pleasant and decent in explaining the situation.
Believe me, I will not grieve your “not bothering with any real arguments”. Every aspect of your behavior suggests that if you had any you would have presented them.
Ciao.
Comment #136 December 8th, 2017 at 3:46 am
This discussion is getting a bit long in the tooth, so I’m not gonna raise further arguments. I’m just curious about Maudlin’s assertion that “…of all the various approaches to the problem of consciousness, the computational theory is perhaps the only one that can be decisively refuted”. Could you please point out what is this decisive refutation? A mere Wikipedia link would suffice.
It’s very surprising to see a philosopher claiming that *anything* can be decisively refuted; heck even frequentism still has defenders.
Comment #137 December 8th, 2017 at 4:35 am
Mateus # 136: “Computation and Consciousness”, Journal of Philosophy 86, pp. 407-432.
Comment #138 December 8th, 2017 at 5:42 am
You mean “Computation and Consciousness”, Journal of Philosophy 86, pp. 407-432, by Tim Maudlin?
Indeed, that was a decisive argument. All philosophers called Tim Maudlin considered this article to be the death blow on the computational theory of consciousness. After this article was published, a consensus emerged within the Tim Maudlin community that the computational approach to consciousness was hopeless.
Comment #139 December 8th, 2017 at 7:51 am
Tim Maudlin,
Since this comment thread will be closed shortly and since you’ve demonstrated a willingness to spend time reading material you’ve been pointed to, I’d like to point you to Yudkowsky’s talk on AI alignment, particularly the parts where he discusses some first attempts in highly idealized models and why they fail. (This sort of thing is the subject matter focus for the security-professional-grade paranoia described in the security mindset dialogue.) The thing I guess I’d like you to be aware of is that (a rather small set of) very smart people have been thinking about this problem for several years — if there’s an approach you can think of in five minutes, or even a full day, it’s already been thought of, and holes have been picked in it.
Comment #140 December 8th, 2017 at 11:11 am
Mateus #139: Since there has been some comment on this thread about the false accusation of someone commenting on something without reading it, I will be very careful here.
If you have not read the argument, then you have no business at all making any comment at all about how decisive it is.
If you have read the argument and think it is not decisive, then you should be pointing out the weakness in it.
If you have read the article and think it is decisive, the you should not be holding a computational theory of consciousness.
That covers the field. Since your comment is not any of the three listed above, you have dome something wrong.
You might find the following e-mail, which I received just last week, instructive:
Dear Dr. Maudlin,
I am a student of philosophy and physics from Germany, with a focus on the philosophy of mind. Recently I stumbled over your paper on computation and consciousness. This is the best refutation of the computational theory of mind I have ever read. Now I wonder, why there is hardly any echo of the paper in the literature, while the argument seems devastating to computationalism.
At the moment I am writing a paper based on your argument, so two questions naturally arise: 1. Is there any reply that I definitely have to read? 2. Are you still convinced by your own argument and if not, why?
Thanks for your time!
Regards,
My answer to the e-mail was that I have not found a flaw, nor ever been made aware of any being found. You can be the first! Give it shot. It seems that you would be highly motivated to find an actual error in the argument, and you can make a real contribution by finding it and exposing it. Good luck. But somewhere in the back of your mind, keep open the possibility that the reason no one has ever found a flaw in the argument is that there isn’t one.