{"id":2410,"date":"2015-08-16T14:23:40","date_gmt":"2015-08-16T18:23:40","guid":{"rendered":"https:\/\/scottaaronson.blog\/?p=2410"},"modified":"2017-01-12T14:19:39","modified_gmt":"2017-01-12T19:19:39","slug":"common-knowledge-and-aumanns-agreement-theorem","status":"publish","type":"post","link":"https:\/\/scottaaronson.blog\/?p=2410","title":{"rendered":"Common Knowledge and Aumann’s Agreement Theorem"},"content":{"rendered":"

The following is the prepared version\u00a0of a talk that I gave at SPARC<\/a>: a high-school summer program about applied rationality held in Berkeley, CA for the past two weeks. \u00a0I had a wonderful time in Berkeley, meeting new friends and old, but I’m now leaving to visit the CQT<\/a> in Singapore, and then to attend the AQIS<\/a> conference in Seoul.<\/p>\n

\n

Common Knowledge and Aumann’s Agreement Theorem<\/strong><\/p>\n

August 14, 2015<\/em><\/p>\n

Thank you so much for inviting me here!\u00a0 I honestly don’t know whether it’s possible to teach applied rationality, the way this camp is trying to do.\u00a0 What I know is that, if it is<\/em> possible, then the people running SPARC are some of the awesomest\u00a0people on earth to figure out how. \u00a0I’m incredibly proud that Chelsea Voss<\/a> and Paul Christiano<\/a> are both former students of mine, and I\u2019m amazed by the program they and the others have put together here.\u00a0 I hope you\u2019re all having fun\u2014or maximizing your utility functions, or whatever.<\/p>\n

My research is mostly about quantum computing, and more broadly, computation and physics.\u00a0 But I was asked to talk about something you can actually use in your lives, so I want to tell a different story, involving common knowledge.<\/p>\n

I’ll start with the “Muddy Children Puzzle,” which is one of the greatest logic puzzles ever invented.\u00a0 How many of you have seen this one?<\/p>\n

OK, so the way it goes is, there are a hundred children playing in the mud.\u00a0 Naturally, they all have muddy foreheads.\u00a0 At some point their teacher comes along and says to them, as they all sit around in a circle: \u201cstand up if you know your forehead is muddy.\u201d\u00a0 No one stands up.\u00a0 For how could they know?\u00a0 Each kid can see all the other<\/em> 99 kids\u2019 foreheads, so knows that they\u2019re muddy, but can\u2019t see his or her own forehead.\u00a0 (We\u2019ll assume that there are no mirrors or camera phones nearby, and also that this is mud that you don\u2019t feel when it\u2019s on your forehead.)<\/p>\n

So the teacher tries again.\u00a0 \u201cKnowing<\/em> that no one stood up the last time, now<\/em> stand up if you know your forehead is muddy.\u201d\u00a0 Still no one stands up.\u00a0 Why would they?\u00a0 No matter how many times the teacher repeats the request, still no one stands up.<\/p>\n

Then the teacher tries something new.\u00a0 \u201cLook, I hereby announce that at least one of you<\/em> has a muddy forehead.\u201d\u00a0 After that announcement, the teacher again says, \u201cstand up if you know your forehead is muddy\u201d\u2014and again no one stands up.\u00a0 And again and again; it continues 99 times.\u00a0 But then the hundredth time, all the children suddenly stand up.<\/p>\n

(There\u2019s a variant of the puzzle involving blue-eyed islanders who all suddenly commit suicide on the hundredth day, when they all learn that their eyes are blue\u2014but as a blue-eyed person myself, that\u2019s always struck me as needlessly macabre.)<\/p>\n

What\u2019s going on here?\u00a0 Somehow, the teacher\u2019s announcing to the children that at least one of them had a muddy forehead<\/em> set something dramatic in motion, which would eventually make them all stand up\u2014but how could that announcement possibly have made any difference?\u00a0 After all, each child already knew that at least 99<\/em> children had muddy foreheads!<\/p>\n

Like with many puzzles, the way to get intuition is to change the numbers.\u00a0 So suppose there were two<\/em> children with muddy foreheads, and the teacher announced to them that at least one had a muddy forehead, and then asked both of them whether their own forehead was muddy.\u00a0 Neither would know.\u00a0 But each child could reason as follows: \u201cif my forehead weren\u2019t<\/em> muddy, then the other child would\u2019ve seen that, and would also have known that at least one of us has a muddy forehead.\u00a0 Therefore she would\u2019ve known, when asked, that her own forehead was muddy.\u00a0 Since she didn\u2019t know, that means my forehead is<\/em> muddy.\u201d \u00a0So then both children know their foreheads are muddy, when the teacher asks a second time.<\/p>\n

Now, this argument can be generalized to any<\/em> (finite) number of children.\u00a0 The crucial concept here is common knowledge<\/a>.\u00a0 We call a fact \u201ccommon knowledge\u201d if, not only does everyone know it, but everyone knows everyone knows it, and everyone knows everyone knows everyone knows it, and so on.\u00a0 It\u2019s true that in the beginning, each child knew that all the other children had muddy foreheads, but it wasn\u2019t common knowledge that even one<\/em> of them had a muddy forehead.\u00a0 For example, if your forehead and mine are both muddy, then I know that at least one of us has a muddy forehead, and you know that too, but you don\u2019t know that I know it (for what if your forehead were clean?), and I don\u2019t know that you know it (for what if my forehead were clean?).<\/p>\n

What the teacher\u2019s announcement did, was to make it<\/em> common knowledge that at least one child has a muddy forehead (since not only did everyone hear the announcement, but everyone witnessed everyone else hearing it, etc.).\u00a0 And once you understand that point, it\u2019s easy to argue by induction: after the teacher asks and no child stands up (and everyone sees that no one stood up), it becomes common knowledge that at least two<\/em> children have muddy foreheads (since if only one child had had a muddy forehead, that child would\u2019ve known it and stood up).\u00a0 Next it becomes common knowledge that at least three<\/em> children have muddy foreheads, and so on, until after a hundred rounds it\u2019s common knowledge that everyone\u2019s forehead is muddy, so everyone stands up.<\/p>\n

The moral is that the mere act of saying something publicly can change the world\u2014even if everything you said was already obvious to every last one of your listeners.<\/em>\u00a0 For it\u2019s possible that, until your announcement, not everyone knew that everyone knew the thing, or knew everyone knew everyone knew it, etc., and that could have prevented them from acting.<\/p>\n

This idea turns out to have huge real-life consequences, to situations way beyond children with muddy foreheads.\u00a0 I mean, it also applies to children with dots on their foreheads, or \u201ckick me\u201d signs on their backs\u2026<\/p>\n

But seriously, let me give you an example I stole from Steven Pinker, from his wonderful book The Stuff of Thought<\/a><\/em>.\u00a0 Two people of indeterminate gender\u2014let\u2019s not make any assumptions here\u2014go on a date.\u00a0 Afterward, one of them says to the other: \u201cWould you like to come up to my apartment to see my etchings?\u201d\u00a0 The other says, \u201cSure, I\u2019d love to see them.\u201d<\/p>\n

This is such a clich\u00e9 that we might not even notice the deep paradox here.\u00a0 It\u2019s like with life itself: people knew for thousands of years that every bird has the right kind of beak for its environment, but not until Darwin and Wallace could anyone articulate why (and only a few people before them even recognized there was a question there<\/em> that called for a non-circular answer).<\/p>\n

In our case, the puzzle is this: both people on the date know perfectly well that the reason they\u2019re going up to the apartment has nothing to do with etchings.\u00a0 They probably even both know the other knows that.\u00a0 But if that\u2019s the case, then why don\u2019t they just blurt it out: \u201cwould you like to come up for some intercourse?\u201d\u00a0 (Or \u201cfluid transfer,\u201d as the John Nash character put it in the Beautiful Mind<\/em> movie?)<\/p>\n

So here\u2019s Pinker\u2019s answer.\u00a0 Yes, both people know why they\u2019re going to the apartment, but they also want to avoid their knowledge becoming common<\/em> knowledge.\u00a0 They want plausible deniability.\u00a0 There are several possible reasons: to preserve the romantic fantasy of being \u201cswept off one\u2019s feet.\u201d\u00a0 To provide a face-saving way to back out later, should one of them change their mind: since nothing was ever openly said, there\u2019s no agreement to abrogate.\u00a0 In fact, even if only one of the people (say A) might care about such things, if the other person (say B) thinks there\u2019s any chance<\/em> A cares, B will also have an interest in avoiding common knowledge, for A\u2019s sake.<\/p>\n

Put differently, the issue is that, as soon as you say X out loud, the other person doesn\u2019t merely learn X: they learn that you know X, that you know that they know that you know X, that you want<\/em> them to know you know X, and an infinity of other things that might upset the delicate epistemic balance.\u00a0 Contrast that with the situation where X is left unstated: yeah, both people are pretty sure that \u201cetchings\u201d are just a pretext, and can even plausibly guess that the other person knows they\u2019re pretty sure about it.\u00a0 But once you start getting to 3, 4, 5, levels of indirection\u2014who knows?\u00a0 Maybe you do<\/em> just want to show me some etchings.<\/p>\n

Philosophers like to discuss Sherlock Holmes and Professor Moriarty meeting in a train station, and Moriarty declaring, \u201cI knew you\u2019d be here,\u201d and Holmes replying, \u201cwell, I knew that you knew I\u2019d be here,\u201d and Moriarty saying, \u201cI knew you knew I knew I\u2019d be here,\u201d etc.\u00a0 But real humans tend to be unable to reason reliably past three or four levels in the knowledge hierarchy.\u00a0 (Related to that, you might have heard of the game where everyone guesses a number between 0 and 100, and the winner is whoever\u2019s number is the closest to 2\/3 of the average of all the numbers.\u00a0 If this game is played by perfectly rational people, who know they\u2019re all perfectly rational, and know they know, etc., then they must all guess 0\u2014exercise for you to see why.\u00a0 Yet experiments show that, if you actually<\/em> want to win this game against average people, you should guess about 20.\u00a0 People seem to start with 50 or so, iterate the operation of multiplying by 2\/3 a few times, and then stop<\/em>.)<\/p>\n

Incidentally, do you know what I would\u2019ve given for someone to have explained this stuff to me back in high school?\u00a0 I think that a large fraction of the infamous social difficulties that nerds have, is simply down to nerds spending so much\u00a0time in domains (like math and science) where the point is to struggle with every last neuron to make everything<\/em> common knowledge, to make all truths\u00a0as clear and explicit as possible.\u00a0 Whereas in social contexts, very often you\u2019re managing a delicate epistemic balance where you need certain things to be known, but not known<\/em> to be known, and so forth\u2014where you need to prevent<\/em> common knowledge from arising, at least temporarily.\u00a0 \u201cNormal\u201d people have an intuitive feel for this; it doesn\u2019t need to be explained to them.\u00a0 For nerds, by contrast, explaining it\u2014in terms of the muddy children puzzle and so forth\u2014might be exactly what\u2019s needed.\u00a0 Once they\u2019re told the rules of a game, nerds can try playing it too!\u00a0 They might even turn out to be good at it.<\/p>\n

OK, now for a darker example of common knowledge in action.\u00a0 If you read accounts of Nazi Germany, or the USSR, or North Korea or other despotic regimes today, you can easily be overwhelmed by this sense of, \u201cso why didn\u2019t all the sane people just rise up and overthrow the totalitarian monsters?\u00a0 Surely<\/em> there were more sane people than crazy, evil ones.\u00a0 And probably the sane people even knew, from experience, that many of their neighbors were sane\u2014so why this cowardice?\u201d\u00a0 Once again, it could be argued that common knowledge is the key. \u00a0Even if everyone knows the emperor is naked; indeed, even if everyone knows everyone knows he\u2019s naked, still, if it\u2019s not common knowledge<\/em>, then anyone who says the emperor\u2019s naked is knowingly assuming a massive personal risk.\u00a0 That\u2019s why, in the story, it took a child to shift the equilibrium.\u00a0 Likewise, even if you know that 90% of the populace will join your democratic revolt provided they themselves know 90% will join it<\/em>, if you can\u2019t make your revolt\u2019s popularity common knowledge, everyone will be stuck second-guessing each other, worried that if they revolt they\u2019ll be an easily-crushed minority.\u00a0 And because of that very worry, they\u2019ll be correct!<\/p>\n

(My favorite Soviet joke involves a man standing in the Moscow train station, handing out leaflets to everyone who passes by.\u00a0 Eventually, of course, the KGB arrests him\u2014but they discover to their surprise that the leaflets are just blank pieces of paper.\u00a0 \u201cWhat\u2019s the meaning of this?\u201d they demand.\u00a0 \u201cWhat is there to write?\u201d replies the man.\u00a0 \u201cIt\u2019s so obvious!\u201d\u00a0 Note that this is precisely<\/em> a situation where the man is trying to make common knowledge something he assumes his \u201creaders\u201d already know.)<\/p>\n

The kicker is that, to prevent something from becoming common knowledge, all you need to do is censor the common-knowledge-producing mechanisms<\/em>: the press, the Internet, public meetings.\u00a0 This nicely explains why despots throughout history have been so obsessed with controlling the press, and also explains how it\u2019s possible for 10% of a population to murder and enslave the other 90% (as has happened again and again in our species\u2019 sorry history), even though the 90% could easily overwhelm the 10% by acting in concert.\u00a0 Finally, it explains why believers in the Enlightenment project tend to be such fanatical absolutists about free speech\u2014why they refuse to \u201cbalance\u201d it against cultural sensitivity or social harmony or any other value, as so many well-meaning people urge these days.<\/p>\n

OK, but let me try to tell you something surprising<\/em> about common knowledge.\u00a0 Here at SPARC, you\u2019ve learned all about Bayes\u2019 rule\u2014how, if you like, you can treat \u201cprobabilities\u201d as just made-up numbers in your head, which are required obey the probability calculus, and then there\u2019s a very definite rule for how to update those numbers when you gain new information.\u00a0 And indeed, how an agent that wanders around constantly updating these numbers in its head, and taking whichever action maximizes its expected utility (as calculated using the numbers), is probably the<\/em> leading modern conception of what it means to be \u201crational.\u201d<\/p>\n

Now imagine that you\u2019ve got two agents, call them Alice and Bob, with common knowledge of each other’s honesty and rationality, and with the same<\/em> prior probability distribution over some set of possible states of the world.\u00a0 But now imagine they go out and live their lives, and have totally different experiences that lead to their learning different things, and having different posterior distributions.\u00a0 But then they meet again, and they realize that their opinions about some topic (say, Hillary\u2019s chances of winning the election) are common knowledge<\/em>: they both know each other\u2019s opinion, and they both know that they both know, and so on. \u00a0Then a striking 1976 result called Aumann\u2019s Theorem<\/a><\/em> states that their opinions must be equal.\u00a0 Or, as it\u2019s summarized: \u201crational agents with common priors can never agree to disagree about anything.\u201d<\/p>\n

Actually, before going further, let\u2019s prove Aumann\u2019s Theorem\u2014since it\u2019s one of those things that sounds like a mistake when you first hear it, and then becomes a triviality once you see the 3-line proof.\u00a0 (Albeit, a \u201ctriviality\u201d that won Aumann a Nobel in economics.)\u00a0 The key idea is that knowledge induces a partition on the set of possible states of the world<\/em>.\u00a0 Huh?\u00a0 OK, imagine someone is either an old man, an old woman, a young man, or a young woman.\u00a0 You and I agree in giving each of these a 25% prior probability.\u00a0 Now imagine that you find out whether they\u2019re a man or a woman, and I find out whether they\u2019re young or old.\u00a0 This can be illustrated as follows:<\/p>\n

\"ymom\"<\/a><\/p>\n

The diagram tells us, for example, that if<\/em> the ground truth is \u201cold woman,\u201d then your knowledge is described by the set {old woman, young woman}, while my knowledge is described by the set {old woman, old man}.\u00a0 And this different information leads us to different beliefs: for example, if someone asks for the probability that the person is a woman, you\u2019ll say 100% but I\u2019ll say 50%.\u00a0 OK, but what does it mean for information to be common knowledge<\/em>?\u00a0 It means that I know that you know that I know that you know, and so on.\u00a0 Which means that, if you want to find out what\u2019s common knowledge between us, you need to take the least common coarsening<\/em> of our knowledge partitions.\u00a0 I.e., if the ground truth is some given world w, then what do I consider it possible that you consider it possible that I consider possible that \u2026 etc.?\u00a0 Iterate this growth process until it stops, by \u201czigzagging\u201d between our knowledge partitions, and you get the set S of worlds such that, if we\u2019re in world w, then what\u2019s common knowledge between us is that the world belongs to S<\/em>.\u00a0 Repeat for all w\u2019s, and you get the least common coarsening of our partitions.\u00a0 In the above example, the least common coarsening is trivial, with all four worlds ending up in the same set S, but there are nontrivial examples as well:<\/p>\n

\"youme\"<\/a><\/p>\n

Now, if Alice\u2019s expectation of a random variable X is common knowledge between her and Bob, that means that everywhere in S, her expectation must be constant \u2026 and hence must equal whatever the expectation is<\/em>, over all the worlds in S!\u00a0 Likewise, if Bob\u2019s expectation is common knowledge with Alice, then everywhere in S, it must equal the expectation of X over S.\u00a0 But that means that Alice\u2019s and Bob\u2019s expectations are the same.<\/p>\n

There are lots of related results.\u00a0 For example, rational agents with common priors, and common knowledge of each other\u2019s rationality, should never engage in speculative trade (e.g., buying and selling stocks, assuming that they don\u2019t need cash, they\u2019re not earning a commission, etc.).\u00a0 Why?\u00a0 Basically because, if I try to sell you a stock for (say) $50, then you should reason that the very fact that I\u2019m offering it means I must<\/em> have information you don\u2019t that it\u2019s worth less than $50, so then you update accordingly and you don\u2019t want it either.<\/p>\n

Or here\u2019s another one: suppose again that we\u2019re Bayesians with common priors, and we\u2019re having a conversation, where I tell you my opinion (say, of the probability Hillary will win the election).\u00a0 Not any of the reasons or evidence on which the opinion is based\u2014just the opinion itself.\u00a0 Then you, being Bayesian, update your probabilities to account for what my opinion is.\u00a0 Then you tell me your<\/em> opinion (which might have changed after learning mine), I update on that, I tell you my new<\/em> opinion, then you tell me your new opinion, and so on.\u00a0 You might think this could go on forever!\u00a0 But, no, Geanakoplos and Polemarchakis observed that, as long as there are only finitely many possible states of the world in our shared prior, this process must converge after finitely many steps with you and me having the same opinion (and moreover, with it being common knowledge<\/em> that we have that opinion).\u00a0 Why?\u00a0 Because as long as our opinions differ, your telling me your opinion or me telling you mine must induce a nontrivial refinement<\/em> of one of our knowledge partitions, like so:<\/p>\n

\"youtell\"<\/a><\/p>\n

I.e., if you learn something new, then at least one of your knowledge sets must get split along the different possible values of the thing you learned.\u00a0 But since there are only finitely many underlying states, there can only be finitely many such splittings (note that, since Bayesians never forget anything, knowledge sets that are split will never again rejoin).<\/p>\n

And something else: suppose your friend tells you a liberal opinion, then you take it into account, but reply with a more conservative opinion.\u00a0 The friend takes your<\/em> opinion into account, and replies with a revised opinion.\u00a0 Question: is your friend\u2019s new opinion likelier to be more liberal than yours, or more conservative?<\/p>\n

Obviously, more liberal!\u00a0 Yes, maybe your friend now sees some of your points and vice versa, maybe you\u2019ve now drawn a bit closer (ideally!), but you\u2019re not going to suddenly switch sides because of one conversation.<\/p>\n

Yet, if you and your friend are Bayesians with common priors, one can prove that that\u2019s not what should happen at all<\/em>.\u00a0 Indeed, your expectation of your own future opinion should equal your current opinion, and your expectation of your friend\u2019s next opinion should also equal your current opinion\u2014meaning that you shouldn\u2019t be able to predict in which direction your opinion will change next, nor in which direction your friend will next disagree with you.\u00a0 Why not?\u00a0 Formally, because all these expectations are just different ways of calculating an expectation over the same set<\/em>, namely your current knowledge set (i.e., the set of states of the world that you currently consider possible)!\u00a0 More intuitively, we could say: if you could predict that, all else equal, the next thing you heard would probably shift your opinion in a liberal direction, then as a Bayesian you should already shift your opinion in a liberal direction right now<\/em>.\u00a0 (This is related to what\u2019s called the \u201cmartingale property\u201d: sure, a random variable X could evolve in many ways in the future, but the average of all those ways must be its current expectation E[X], by the very definition of E[X]\u2026)<\/p>\n

So, putting all these results together, we get a clear picture of what rational disagreements should look like: they should follow unbiased random walks, until sooner or later they terminate in common knowledge of complete agreement.\u00a0 We now face a bit of a puzzle, in that hardly any disagreements in the history of the world have ever looked like that<\/em>.\u00a0 So what gives?<\/p>\n

There are a few ways out:<\/p>\n

(1) You could say that the \u201cfailed prediction\u201d of Aumann\u2019s Theorem is no surprise, since virtually all human beings are irrational cretins, or liars (or at least, it’s not common knowledge that they aren’t). Except for you, of course: you\u2019re<\/em> perfectly rational and honest.\u00a0 And if you ever met anyone else as rational and honest as you, maybe you and they could have an Aumannian conversation.\u00a0 But since such a person probably doesn\u2019t exist, you\u2019re totally justified to stand your ground, discount all opinions that differ from yours, etc.\u00a0 Notice that, even if you genuinely believed that was all there was to it, Aumann\u2019s Theorem would still have an aspirational<\/em> significance for you: you would still have to say this is the ideal that all rationalists should strive toward when they disagree.\u00a0 And that would already conflict with a lot of standard rationalist wisdom.\u00a0 For example, we all know that arguments from authority carry little weight: what should sway you is not the mere fact of some other person stating<\/em> their opinion, but the actual arguments and evidence that they\u2019re able to bring.\u00a0 Except that as we\u2019ve seen, for Bayesians with common priors this isn\u2019t true at all!\u00a0 Instead, merely hearing your friend\u2019s opinion serves as a powerful summary of what\u00a0your friend knows.\u00a0 And if you learn that your rational friend disagrees with you, then even without knowing why<\/em>, you should take that as seriously as if you discovered a contradiction in your own thought processes.\u00a0 This is related to an even broader point: there\u2019s a normative rule of rationality that you should judge ideas only on their merits\u2014yet if you\u2019re a Bayesian, of course<\/em> you\u2019re going to take into account where the ideas come from, and how many other people hold them!\u00a0 Likewise, if you\u2019re a Bayesian police officer or a Bayesian airport screener or a Bayesian job interviewer, of course<\/em> you\u2019re going to profile people by their superficial characteristics, however unfair that might be to individuals\u2014so all those studies proving that people evaluate the same resume differently if you change the name at the top are no great surprise.\u00a0 It seems to me that the tension between these two different views of rationality, the normative and the Bayesian, generates a lot of the most intractable debates of the modern world.<\/p>\n

(2) Or\u2014and this is an obvious one\u2014you could reject the assumption of common priors. After all, isn\u2019t a major selling point of Bayesianism supposed to be its subjective<\/em> aspect, the fact that you pick \u201cwhichever prior feels right for you,\u201d and are constrained only in how to update that prior?\u00a0 If Alice\u2019s and Bob\u2019s priors can be different, then all the reasoning I went through earlier collapses.\u00a0 So rejecting common priors might seem appealing. \u00a0But there\u2019s a paper by Tyler Cowen and Robin Hanson called \u201cAre Disagreements Honest?\u201d<\/a>\u2014one of the most worldview-destabilizing papers I\u2019ve ever read\u2014that calls that strategy into question.\u00a0 What it says, basically, is this: if you\u2019re really a thoroughgoing Bayesian rationalist, then your prior ought to allow for the possibility that you are<\/em> the other person.\u00a0 Or to put it another way: \u201cyou being born as you,\u201d rather than as someone else, should be treated as just one more contingent fact that you observe and then conditionalize on!\u00a0 And likewise, the other person should condition on the observation that they\u2019re them and not you.\u00a0 In this way, absolutely everything<\/em> that makes you different from someone else can be understood as \u201cdiffering information,\u201d so we\u2019re right back to the situation covered by Aumann\u2019s Theorem.\u00a0 Imagine, if you like, that we all started out behind some Rawlsian veil of ignorance, as pure reasoning minds that had yet to be assigned specific bodies.\u00a0 In that original state, there was nothing to differentiate any of us from any other\u2014anything that did would just be information to condition on\u2014so we all should\u2019ve had the same prior.\u00a0 That might sound fanciful, but in some sense all it\u2019s saying is: what licenses you to privilege an observation just because it\u2019s your<\/em> eyes that made it, or a thought just because it happened to occur in your<\/em> head?\u00a0 Like, if you\u2019re objectively smarter or more observant than everyone else around you, fine, but to whatever extent you agree that you aren\u2019t, your<\/em> opinion gets no special epistemic protection just because it\u2019s yours.<\/p>\n

(3) If you\u2019re uncomfortable with this tendency of Bayesian reasoning to refuse to be confined anywhere, to want to expand to cosmic or metaphysical scope (\u201cI need to condition on having been born as me<\/em> and not someone else\u201d)\u2014well then, you could reject the entire framework of Bayesianism, as your notion of rationality. Lest I be cast out from this camp as a heretic, I hasten to say: I include this option only for the sake of completeness!<\/p>\n

(4) When I first learned about this stuff 12 years ago, it seemed obvious to me that a lot of it could be dismissed as irrelevant to the real world for reasons of complexity<\/em>. I.e., sure, it might apply to ideal reasoners with unlimited time and computational power, but as soon as you impose realistic constraints, this whole Aumannian house of cards should collapse.\u00a0 As an example, if Alice and Bob have common priors, then sure they\u2019ll agree about everything if they effectively share all their information with each other!\u00a0 But in practice, we don\u2019t have time to \u201cmind-meld,\u201d swapping our entire life experiences with anyone we meet.\u00a0 So one could conjecture that agreement, in general, requires a lot of communication.\u00a0 So then I sat down and tried to prove that as a theorem.\u00a0 And you know what I found?\u00a0 That my intuition here wasn\u2019t even close<\/em> to correct!<\/p>\n

In more detail, I proved the following theorem<\/a>.\u00a0 Suppose Alice and Bob are Bayesians with shared priors, and suppose they\u2019re arguing about (say) the probability of some future event\u2014or more generally, about any random variable X bounded in [0,1].\u00a0 So, they have a conversation where Alice first announces her expectation of X, then Bob announces his new expectation, and so on.\u00a0 The theorem says that Alice\u2019s and Bob\u2019s estimates of X will necessarily agree to within \u00b1\u03b5, with probability at least 1-\u03b4 over their shared prior, after they\u2019ve exchanged only O(1\/(\u03b4\u03b52<\/sup>)) messages.\u00a0 Note that this bound is completely independent of how much knowledge they have; it depends only on the accuracy with which they want to agree!\u00a0 Furthermore, the same bound holds even if Alice and Bob only send a few discrete bits about their real-valued expectations with each message, rather than the expectations themselves.<\/p>\n

The proof involves the idea that Alice and Bob\u2019s estimates of X, call them XA<\/sub> and XB<\/sub> respectively, follow \u201cunbiased random walks\u201d (or more formally, are martingales).\u00a0 Very roughly, if |XA<\/sub>-XB<\/sub>|\u2265\u03b5 with high probability over Alice and Bob\u2019s shared prior, then that fact implies that the next message has a high probability (again, over the shared prior) of causing either XA<\/sub> or XB<\/sub> to jump up or down by about \u03b5.\u00a0 But XA<\/sub> and XB<\/sub>, being estimates of X, are bounded between 0 and 1.\u00a0 So a random walk with a step size of \u03b5 can only continue for about 1\/\u03b52<\/sup> steps before it hits one of the \u201cabsorbing barriers.\u201d<\/p>\n

The way to formalize this is to look at the variances, Var[XA<\/sub>] and Var[XB<\/sub>], with respect to the shared prior.\u00a0 Because Alice and Bob\u2019s partitions keep getting refined, the variances are monotonically non-decreasing.\u00a0 They start out 0 and can never exceed 1 (in fact they can never exceed 1\/4, but let\u2019s not worry about constants).\u00a0 Now, the key lemma is that, if Pr[|XA<\/sub>-XB<\/sub>|\u2265\u03b5]\u2265\u03b4, then Var[XB<\/sub>] must increase by at least \u03b4\u03b52<\/sup> if Alice sends XA<\/sub> to Bob, and Var[XA<\/sub>] must increase by at least \u03b4\u03b52<\/sup> if Bob sends XB<\/sub> to Alice.\u00a0 You can see my paper for the proof, or just work it out for yourself.\u00a0 At any rate, the lemma implies that, after O(1\/(\u03b4\u03b52<\/sup>)) rounds of communication, there must be at least a temporary break in the disagreement; there must be some round where Alice and Bob approximately agree with high probability.<\/p>\n

There are lots of other results in my paper<\/a>, including an upper bound on the number of calls that Alice and Bob need to make to a \u201csampling oracle\u201d to carry out this sort of protocol approximately, assuming they\u2019re not perfect Bayesians but agents with bounded computational power.\u00a0 But let me step back and address the broader question: what should we make of all this?\u00a0 How should we live with the gargantuan chasm between the prediction of Bayesian rationality for how we should disagree, and the actual facts of how we do<\/em> disagree?<\/p>\n

We could simply declare that human beings are not well-modeled as Bayesians with common priors<\/em>\u2014that we\u2019ve failed in giving a descriptive account of human behavior\u2014and leave it at that. \u00a0\u00a0OK, but that would still leave the question: does this stuff have normative<\/em> value? \u00a0Should it affect how we behave, if we want to consider ourselves honest and rational?\u00a0 I would argue, possibly yes.<\/p>\n

Yes, you should constantly ask yourself the question: \u201cwould I still be defending this opinion, if I had been born as someone else?\u201d\u00a0 (Though you might say this insight predates Aumann by quite a bit, going back at least to Spinoza.)<\/p>\n

Yes, if someone you respect as honest and rational disagrees with you, you should take it as seriously as if the disagreement were between two different aspects of yourself.<\/p>\n

Finally, yes, we can try to judge epistemic communities by how closely they approach the Aumannian ideal.\u00a0 In math and science, in my experience, it\u2019s common to see two people furiously arguing with each other at a blackboard.\u00a0 Come back five minutes later, and they\u2019re arguing even more furiously, but now their positions have switched.\u00a0 As we\u2019ve seen, that\u2019s precisely<\/em> what the math says a rational conversation should look like.\u00a0 In social and political discussions, though, usually the very best you\u2019ll see is that two people start out diametrically opposed, but eventually one of them says \u201cfine, I\u2019ll grant you this,\u201d and the other says \u201cfine, I\u2019ll grant you that.\u201d\u00a0 We might say, that\u2019s certainly better than the common alternative, of the two people walking away even more polarized than before!\u00a0 Yet the math tells us that even the first case\u2014even the two people gradually getting closer in their views\u2014is nothing at all<\/em> like a rational exchange, which would involve the two participants repeatedly leapfrogging each other, completely changing their opinion about the question under discussion (and then changing back, and back again) every time they learned something new.\u00a0 The first case, you might say, is more like haggling<\/em>\u2014more like \u201cI\u2019ll grant you that X is true if you grant me that Y is true\u201d\u2014than like our ideal friendly mathematicians arguing at the blackboard, whose acceptance of new truths is never slow or grudging, never conditional on the other person first agreeing with them about something else.<\/p>\n

Armed with this understanding, we could try to rank fields by how hard it is to have an Aumannian conversation in them.\u00a0 At the bottom\u2014the easiest!\u2014is math (or, let\u2019s say, chess, or debugging a program, or fact-heavy fields like lexicography or geography).\u00a0 Crucially, here I only mean the parts<\/em> of these subjects with agreed-on rules and definite answers: once the conversation turns to whose theorems are deeper, or whose fault the bug was, things can get arbitrarily non-Aumannian.\u00a0 Then there\u2019s the type of science that involves messy correlational studies (I just mean, talking about what\u2019s a risk factor for what, not the political implications).\u00a0 Then there\u2019s politics and aesthetics, with the most radioactive topics like Israel\/Palestine higher up.\u00a0 And then, at the very peak, there\u2019s gender and social justice debates, where everyone<\/em> brings their formative experiences along, and absolutely no one is a disinterested truth-seeker, and possibly no Aumannian conversation has ever been had in the history of the world.<\/p>\n

I would urge that even at the very top, it\u2019s still incumbent on all of us to try<\/em> to make the Aumannian move, of \u201cwhat would I think about this issue if I were someone else and not me?\u00a0 If I were a man, a woman, black, white, gay, straight, a nerd, a jock?\u00a0 How much of my thinking about this represents pure Spinozist reason, which could be ported to any rational mind, and how much of it would get lost in translation?\u201d<\/p>\n

Anyway, I\u2019m sure some people would argue that, in the end, the whole framework of Bayesian agents, common priors, common knowledge, etc. can be chucked from this discussion like so much scaffolding, and the moral lessons I want to draw boil down to trite advice (\u201ctry to see the other person\u2019s point of view\u201d) that you all knew already.\u00a0 Then again, even if you all knew all this, maybe you didn\u2019t know that you all knew it!\u00a0 So I hope you gained some new information from this talk in any case.\u00a0 Thanks.<\/p>\n

\n

Update:<\/span><\/b> Coincidentally, there’s a moving NYT piece<\/a> by Oliver Sacks, which (among other things) recounts his experiences with his cousin, the Aumann of Aumann’s theorem.<\/p>\n

\n

Another Update:<\/span><\/b> If I ever did<\/i> attempt an Aumannian conversation with someone, the other Scott A. would be a candidate! Here he is in 2011<\/a> making several of the same points I did above, using the same examples (I thank him for pointing me to his post).<\/p>\n","protected":false},"excerpt":{"rendered":"

The following is the prepared version\u00a0of a talk that I gave at SPARC: a high-school summer program about applied rationality held in Berkeley, CA for the past two weeks. \u00a0I had a wonderful time in Berkeley, meeting new friends and old, but I’m now leaving to visit the CQT in Singapore, and then to attend […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"advanced_seo_description":"","jetpack_seo_html_title":"","jetpack_seo_noindex":false,"jetpack_seo_schema_type":"","_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"{title}\n\n{excerpt}\n\n{url}","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"_wpas_customize_per_network":false,"jetpack_post_was_ever_published":false},"categories":[10,12,11],"tags":[],"class_list":["post-2410","post","type-post","status-publish","format-standard","hentry","category-adventures-in-meatspace","category-metaphysical-spouting","category-nerd-interest"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=\/wp\/v2\/posts\/2410","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=2410"}],"version-history":[{"count":14,"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=\/wp\/v2\/posts\/2410\/revisions"}],"predecessor-version":[{"id":2455,"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=\/wp\/v2\/posts\/2410\/revisions\/2455"}],"wp:attachment":[{"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=2410"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=2410"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=2410"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}