Shtetl-Optimized

I have some exciting news (for me, anyway). Starting next week, I’ll be going on leave from UT Austin for one year, to work at OpenAI. They’re the creators of the astonishing GPT-3 and DALL-E2, which have not only endlessly entertained me and my kids, but recalibrated my understanding of what, for better and worse, the world is going to look like for the rest of our lives. Working with an amazing team at OpenAI, including Jan Leike, John Schulman, and Ilya Sutskever, my job will be think about the theoretical foundations of AI safety and alignment. What, if anything, can computational complexity contribute to a principled understanding of how to get an AI to do what we want and not do what we don’t want?

Yeah, I don’t know the answer either. That’s why I’ve got a whole year to try to figure it out! One thing I know for sure, though, is that I’m interested both in the short-term, where new ideas are now quickly testable, and where the misuse of AI for spambots, surveillance, propaganda, and other nefarious purposes is already a major societal concern, and the long-term, where one might worry about what happens once AIs surpass human abilities across nearly every domain. (And all the points in between: we might be in for a long, wild ride.) When you start reading about AI safety, it’s striking how there are two separate communities—one mostly worried about machine learning perpetuating racial and gender biases, and the other mostly worried about superhuman AI turning the planet into goo—who not only don’t work together, but are at each other’s throats, with each accusing the other of totally missing the point. I persist, however, in the possibly-naïve belief that these are merely two extremes along a single continuum of AI worries. By figuring out how to align AI with human values today—constantly confronting our theoretical ideas with reality—we can develop knowledge that will give us a better shot at aligning it with human values tomorrow.

For family reasons, I’ll be doing this work mostly from home, in Texas, though traveling from time to time to OpenAI’s office in San Francisco. I’ll also spend 30% of my time continuing to run the Quantum Information Center at UT Austin and working with my students and postdocs. At the end of the year, I plan to go back to full-time teaching, writing, and thinking about quantum stuff, which remains my main intellectual love in life, even as AI—the field where I started, as a PhD student, before I switched to quantum computing—has been taking over the world in ways that none of us can ignore.

Maybe fittingly, this new direction in my career had its origins here on Shtetl-Optimized. Several commenters, including Max Ra and Matt Putz, asked me point-blank what it would take to induce me to work on AI alignment. Treating it as an amusing hypothetical, I replied that it wasn’t mostly about money for me, and that:

The central thing would be finding an actual potentially-answerable technical question around AI alignment, even just a small one, that piqued my interest and that I felt like I had an unusual angle on. In general, I have an absolutely terrible track record at working on topics because I abstractly feel like I “should” work on them. My entire scientific career has basically just been letting myself get nerd-sniped by one puzzle after the next.

Anyway, Jan Leike at OpenAI saw this exchange and wrote to ask whether I was serious in my interest. Oh shoot! Was I? After intensive conversations with Jan, others at OpenAI, and others in the broader AI safety world, I finally concluded that I was.

I’ve obviously got my work cut out for me, just to catch up to what’s already been done in the field. I’ve actually been in the Bay Area all week, meeting with numerous AI safety people (and, of course, complexity and quantum people), carrying a stack of technical papers on AI safety everywhere I go. I’ve been struck by how, when I talk to AI safety experts, they’re not only not dismissive about the potential relevance of complexity theory, they’re more gung-ho about it than I am! They want to talk about whether, say, IP=PSPACE, or MIP=NEXP, or the PCP theorem could provide key insights about how we could verify the behavior of a powerful AI. (Short answer: maybe, on some level! But, err, more work would need to be done.)

How did this complexitophilic state of affairs come about? That brings me to another wrinkle in the story. Traditionally, students follow in the footsteps of their professors. But in trying to bring complexity theory into AI safety, I’m actually following in the footsteps of my student: Paul Christiano, one of the greatest undergrads I worked with in my nine years at MIT, the student whose course project turned into the Aaronson-Christiano quantum money paper. After MIT, Paul did a PhD in quantum computing at Berkeley, with my own former adviser Umesh Vazirani, while also working part-time on AI safety. Paul then left quantum computing to work on AI safety full-time—indeed, along with others such as Dario Amodei, he helped start the safety group at OpenAI. Paul has since left to found his own AI safety organization, the Alignment Research Center (ARC), although he remains on good terms with the OpenAI folks. Paul is largely responsible for bringing complexity theory intuitions and analogies into AI safety—for example, through the “AI safety via debate” paper and the Iterated Amplification paper. I’m grateful for Paul’s guidance and encouragement—as well as that of the others now working in this intersection, like Geoffrey Irving and Elizabeth Barnes—as I start this new chapter.

So, what projects will I actually work on at OpenAI? Yeah, I’ve been spending the past week trying to figure that out. I still don’t know, but a few possibilities have emerged. First, I might work out a general theory of sample complexity and so forth for learning in dangerous environments—i.e., learning where making the wrong query might kill you. Second, I might work on explainability and interpretability for machine learning: given a deep network that produced a particular output, what do we even mean by an “explanation” for “why” it produced that output? What can we say about the computational complexity of finding that explanation? Third, I might work on the ability of weaker agents to verify the behavior of stronger ones. Of course, if P≠NP, then the gap between the difficulty of solving a problem and the difficulty of recognizing a solution can sometimes be enormous. And indeed, even in empirical machine learing, there’s typically a gap between the difficulty of generating objects (say, cat pictures) and the difficulty of discriminating between them and other objects, the latter being easier. But this gap typically isn’t exponential, as is conjectured for NP-complete problems: it’s much smaller than that. And counterintuitively, we can then turn around and use the generators to improve the discriminators. How can we understand this abstractly? Are there model scenarios in complexity theory where we can prove that something similar happens? How far can we amplify the generator/discriminator gap—for example, by using interactive protocols, or debates between competing AIs?

OpenAI, of course, has the word “open” right in its name, and a founding mission “to ensure that artificial general intelligence benefits all of humanity.” But it’s also a for-profit enterprise, with investors and paying customers and serious competitors. So throughout the year, don’t expect me to share any proprietary information—that’s not my interest anyway, even if I hadn’t signed an NDA. But do expect me to blog my general thoughts about AI safety as they develop, and to solicit feedback from readers.

In the past, I’ve often been skeptical about the prospects for superintelligent AI becoming self-aware and destroying the world anytime soon (see, for example, my 2008 post The Singularity Is Far). While I was aware since 2005 or so of the AI-risk community; and of its leader and prophet, Eliezer Yudkowsky; and of Eliezer’s exhortations for people to drop everything else they’re doing and work on AI risk, as the biggest issue facing humanity, I … kept the whole thing at arms’ length. Even supposing I agreed that this was a huge thing to worry about, I asked, what on earth do you want me to do about it today? We know so little about a future superintelligent AI and how it would behave that any actions we took today would likely be useless or counterproductive.

Over the past 15 years, though, my and Eliezer’s views underwent a dramatic and ironic reversal. If you read Eliezer’s “litany of doom” from two weeks ago, you’ll see that he’s now resigned and fatalistic: because his early warnings weren’t heeded, he argues, humanity is almost certainly doomed and an unaligned AI will soon destroy the world. He says that there are basically no promising directions in AI safety research: for any alignment strategy anyone points out, Eliezer can trivially refute it by explaining how (e.g.) the AI would be wise to the plan, and would pretend to go along with whatever we wanted from it while secretly plotting against us.

The weird part is, just as Eliezer became more and more pessimistic about the prospects for getting anywhere on AI alignment, I’ve become more and more optimistic. Part of my optimism is because people like Paul Christiano have laid foundations for a meaty mathematical theory: much like the Web (or quantum computing theory) in 1992, it’s still in a ridiculously primitive stage, but even my limited imagination now suffices to see how much more could be built there. An even greater part of my optimism is because we now live in a world with GPT-3, DALL-E2, and other systems that, while they clearly aren’t AGIs, are powerful enough that worrying about AGIs has come to seem more like prudence than like science fiction. And we can finally test our intuitions against the realities of these systems, which (outside of mathematics) is pretty much the only way human beings have ever succeeded at anything.

I didn’t predict that machine learning models this impressive would exist by 2022. Most of you probably didn’t predict it. For godsakes, Eliezer Yudkowsky didn’t predict it. But it’s happened. And to my mind, one of the defining virtues of science is that, when empirical reality gives you a clear shock, you update and adapt, rather than expending your intelligence to come up with clever reasons why it doesn’t matter or doesn’t count.

Anyway, so that’s the plan! If I can figure out a way to save the galaxy, I will, but I’ve set my goals slightly lower, at learning some new things and doing some interesting research and writing some papers about it and enjoying a break from teaching. Wish me a non-negligible success probability!

Update (June 18): To respond to a couple criticisms that I’ve seen elsewhere on social media…

Can the rationalists sneer at me for waiting to get involved with this subject until it had become sufficiently “respectable,” “mainstream,” and ”high-status”? I suppose they can, if that’s their inclination. I suppose I should be grateful that so many of them chose to respond instead with messages of congratulations and encouragement. Yes, I plead guilty to keeping this subject at arms-length until I could point to GPT-3 and DALL-E2 and the other dramatic advances of the past few years to justify the reality of the topic to anyone who might criticize me. It feels internally like I had principled reasons for this: I can think of almost no examples of research programs that succeeded over decades even in the teeth of opposition from the scientific mainstream. If so, then arguably the best time to get involved with a “fringe” scientific topic, is when and only when you can foresee a path to it becoming the scientific mainstream. At any rate, that’s what I did with quantum computing, as a teenager in the mid-1990s. It’s what many scientists of the 1930s did with the prospect of nuclear chain reactions. And if I’d optimized for getting the right answer earlier, I might’ve had to weaken the filters and let in a bunch of dubious worries that would’ve paralyzed me. But I admit the possibility of self-serving bias here.

Should you worry that OpenAI is just hiring me to be able to say “look, we have Scott Aaronson working on the problem,” rather than actually caring about what its safety researchers come up with? I mean, I can’t prove that you shouldn’t worry about that. In the end, whatever work I do on the topic will have to speak for itself. For whatever it’s worth, though, I was impressed by the OpenAI folks’ detailed, open-ended engagement with these questions when I met them—sort of like how it might look if they actually believed what they said about wanting to get this right for the world. I wouldn’t have gotten involved otherwise.

Follow

… on Blake Lemoine, the Google engineer who became convinced that a machine learning model had become sentient, contacted federal government agencies about it, and was then fired placed on administrative leave for violating Google’s confidentiality policies.

(1) I don’t think Lemoine is right that LaMDA is at all sentient, but the transcript is so mind-bogglingly impressive that I did have to stop and think for a second! Certainly, if you sent the transcript back in time to 1990 or whenever, even an expert reading it might say, yeah, it looks like by 2022 AGI has more likely been achieved than not (“but can I run my own tests?”). Read it for yourself, if you haven’t yet.

(2) Reading Lemoine’s blog and Twitter this morning, he holds many views that I disagree with, not just about the sentience of LaMDA. Yet I’m touched and impressed by how principled he is, and I expect I’d hit it off with him if I met him. I wish that a solution could be found where Google wouldn’t fire him.

Follow

So, I’d been planning a fun post for today about the DALL-E image-generating AI model, and in particular, a brief new preprint about DALL-E’s capabilities by Ernest Davis, Gary Marcus, and myself. We wrote this preprint as a sort of “adversarial collaboration”: Ernie and Gary started out deeply skeptical of DALL-E, while I was impressed bordering on awestruck. I was pleasantly surprised that we nevertheless managed to produce a text that we all agreed on.

Not for the first time, though, world events have derailed my plans. The most important part of today’s post is this:

For the next week, I, Scott Aaronson, will personally match all reader donations to Fund Texas Choice—a group that helps women in Texas travel to out-of-state health clinics, for reasons that are neither your business nor mine—up to a total of $5,000.

To show my seriousness, I’ve already donated $1,000. Just let me know how much you’ve donated in the comments section!

The first reason for this donation drive is that, perhaps like many of you, I stayed up hours last night reading Alito’s leaked decision in a state of abject terror. I saw how the logic of the decision, consistent and impeccable on its own terms, is one by which the Supreme Court’s five theocrats could now proceed to unravel the whole of modernity. I saw how this court, unchecked by our broken democratic system, can now permanently enshrine the will of a radical minority, perhaps unless and until the United States is plunged into a second Civil War.

Anyway, that’s the first reason for the donation drive. The second reason is to thank Shtetl-Optimized‘s commenters for their … err, consistently generous and thought-provoking contributions. Let’s take, for example, this comment on last week’s admittedly rather silly post, from an anonymous individual who calls herself “Feminist Bitch,” and who was enraged that it took me a full day to process one of the great political cataclysms of our lifetimes and publicly react to it:

OF COURSE. Not a word about Roe v. Wade being overturned, but we get a pseudo-intellectual rationalist-tier rant about whatever’s bumping around Scott’s mind right now. Women’s most basic reproductive rights are being curtailed AS WE SPEAK and not a peep from Scott, eh? Even though in our state (Texas) there are already laws ON THE BOOKS that will criminalize abortion as soon as the alt-right fascists in our Supreme Court give the go-ahead. If you cared one lick about your female students and colleagues, Scott, you’d be posting about the Supreme Court and helping feminist causes, not posting your “memes.” But we all know Scott doesn’t give a shit about women. He’d rather stand up for creepy nerdbros and their right to harass women than women’s right to control their own fucking bodies. Typical Scott.

If you want, you can read all of Feminist Bitch’s further thoughts about my failings, with my every attempt to explain and justify myself met with further contempt. No doubt my well-meaning friends of both sexes would counsel me to ignore her. Alas, from my infamous ordeal of late 2014, I know that with her every word, Feminist Bitch speaks for thousands, and the knowledge eats at me day and night.

It’s often said that “the right looks for converts, while the left looks only for heretics.” Has Feminist Bitch ever stopped to think about how our civilization reached its current terrifying predicament—how Trump won in 2016, how the Supreme Court got packed with extremists who represent a mere 25% of the country, how Putin and Erdogan and Orban and Bolsonaro and all the rest consolidated their power? Does she think it happened because wokeists like herself reached out too much, made too many inroads among fellow citizens who share some but not all of their values? Would Feminist Bitch say that, if the Democrats want to capitalize on the coming tsunami of outrage about the death of Roe and the shameless lies that enabled it, if they want to sweep to victory in the midterms and enshrine abortion rights into federal law … then their best strategy would be to double down on their condemnations of gross, creepy, smelly, white male nerdbros who all the girls, like, totally hate?

(until, thank God, some of them don’t)

I continue to think that the majority of my readers, of all races and sexes and backgrounds, are reasonable and sane. I continue to think the majority of you recoil against hatred and dehumanization of anyone—whether that means women seeking abortions, gays, trans folks, or (gasp!) even white male techbros. In this sad twilight for the United States and for liberal democracy around the world, we the reasonable and sane, we the fans of the Enlightenment, we the Party of Psychological Complexity, have decades of work cut out for us. For now I’ll simply say: I don’t hear from you nearly enough in the comments.

Follow

There is a fundamental difference between form and meaning. Form is the physical structure of something, while meaning is the interpretation or concept that is attached to that form. For example, the form of a chair is its physical structure – four legs, a seat, and a back. The meaning of a chair is that it is something you can sit on.

This distinction is important when considering whether or not an AI system can be trained to learn semantic meaning. AI systems are capable of learning and understanding the form of data, but they are not able to attach meaning to that data. In other words, AI systems can learn to identify patterns, but they cannot understand the concepts behind those patterns.

For example, an AI system might be able to learn that a certain type of data is typically associated with the concept of “chair.” However, the AI system would not be able to understand what a chair is or why it is used. In this way, we can see that an AI system trained on form can never learn semantic meaning.

–GPT3, when I gave it the prompt “Write an essay proving that an AI system trained on form can never learn semantic meaning”

Follow

… but these antiwar protesters in St. Petersburg know that they’re all going to be arrested and are doing it anyway.

Meanwhile, I just spent an hour giving Lily, my 9-year-old, a crash course on geopolitics, including WWII, the Cold War, the formation of NATO, Article 5, nuclear deterrence, economic sanctions, the breakup of the USSR, Ukraine, the Baltic Republics, and the prospects now for WWIII. Her comment at the end was that from now on she’s going to refer to Putin as “Poopin,” in the hope that that shames him into changing course.

Update (March 1): A longtime Shtetl-Optimized reader has a friend who’s trying to raise funds to get her family out of Ukraine. See here if you’d like to help.

Follow

Tonight, I took the time actually to read DeepMind’s AlphaCode paper, and to work through the example contest problems provided, and understand how I would’ve solved those problems, and how AlphaCode solved them.

It is absolutely astounding.

Consider, for example, the “n singers” challenge (pages 59-60). To solve this well, you first need to parse a somewhat convoluted English description, discarding the irrelevant fluff about singers, in order to figure out that you’re being asked to find a positive integer solution (if it exists) to a linear system whose matrix looks like
1 2 3 4
4 1 2 3
3 4 1 2
2 3 4 1.
Next you need to find a trick for solving such a system without Gaussian elimination or the like (I’ll leave that as an exercise…). Finally, you need to generate code that implements that trick, correctly handling the wraparound at the edges of the matrix, and breaking and returning “NO” for any of multiple possible reasons why a positive integer solution won’t exist. Oh, and also correctly parse the input.

Yes, I realize that AlphaCode generates a million candidate programs for each challenge, then discards the vast majority by checking that they don’t work on the example data provided, then still has to use clever tricks to choose from among the thousands of candidates remaining. I realize that it was trained on tens of thousands of contest problems and millions of solutions to those problems. I realize that it “only” solves about a third of the contest problems, making it similar to a mediocre human programmer on these problems. I realize that it works only in the artificial domain of programming contests, where a complete English problem specification and example inputs and outputs are always provided.

Forget all that. Judged against where AI was 20-25 years ago, when I was a student, a dog is now holding meaningful conversations in English. And people are complaining that the dog isn’t a very eloquent orator, that it often makes grammatical errors and has to start again, that it took heroic effort to train it, and that it’s unclear how much the dog really understands.

It’s not obvious how you go from solving programming contest problems to conquering the human race or whatever, but I feel pretty confident that we’ve now entered a world where “programming” will look different.

Update: A colleague of mine points out that one million, the number of candidate programs that AlphaCode needs to generate, could be seen as roughly exponential in the number of lines of the generated programs. If so, this suggests a perspective according to which DeepMind has created almost the exact equivalent, in AI code generation, of a non-fault-tolerant quantum computer that’s nevertheless competitive on some task (as in the quantum supremacy experiments). I.e., it clearly does something highly nontrivial, but the “signal” is still decreasing exponentially with the number of instructions, necessitating an exponential number of repetitions to extract the signal and imposing a limit on the size of the programs you can scale to.

Follow

Two weeks ago, I announced on this blog that, thanks to the remarkable generosity of Jaan Tallinn, and the Speculation Grants program of the Survival and Flourishing Fund that Jaan founded, I had $200,000 to give away to charitable organizations of my choice. So, inspired by what Scott Alexander had done, I invited the readers of Shtetl-Optimized to pitch their charities, mentioning only some general areas of interest to me (e.g., advanced math education at the precollege level, climate change mitigation, pandemic preparedness, endangered species conservation, and any good causes that would enrage the people who attack me on Twitter).

I’m grateful to have gotten more than twenty well-thought-out pitches; you can read a subset of them in the comment thread. Now, having studied them all, I’ve decided—as I hadn’t at the start—to use my entire allotment to make as strong a statement as I can about a single cause: namely, subject-matter passion and excellence in precollege STEM education.

I’ll be directing funds to some shockingly cash-starved math camps, math circles, coding outreach programs, magnet schools, and enrichment programs, in Maine and Oregon and England and Ghana and Ethiopia and Jamaica. The programs I’ve chosen target a variety of ability levels, not merely the “mathematical elite.” Several explicitly focus on minority and other underserved populations. But they share a goal of raising every student they work with as high as possible, rather than pushing the students down to fit some standardized curriculum.

Language like that ought to be meaningless boilerplate, but alas, it no longer is. We live in a time when the state of California, in a misguided pursuit of “modernization” and “equity,” is poised to eliminate 8th-grade algebra, make it nearly impossible for high-school seniors to take AP Calculus, and shunt as many students as possible from serious mathematical engagement into a “data science pathway” that in practice might teach little more than how to fill in spreadsheets. (This watering-down effort now itself looks liable to be watered down—but only because of a furious pushback from parents and STEM professionals, pushback in which I’m proud that this blog played a small role.) We live in a time when elite universities are racing to eliminate the SAT—thus, for all their highminded rhetoric, effectively slamming the door on thousands of nerdy kids from poor or immigrant backgrounds who know how to think, but not how to shine in a college admissions popularity pageant. We live in a time when America’s legendary STEM magnet high schools, from Thomas Jefferson in Virginia to Bronx Science to Lowell in San Francisco, rather than being celebrated as the national treasures that they are, or better yet replicated, are bitterly attacked as “elitist” (even while competitive sports and music programs are not similarly attacked)—and are now being forcibly “demagnetized” by bureaucrats, made all but indistinguishable from other high schools, over the desperate pleas of their students, parents, and alumni.

And—alright, fine, on a global scale, arresting climate change is surely a higher-priority issue than protecting the intellectual horizons of a few teenage STEM nerds. The survival of liberal democracy is a higher-priority issue. Pandemic preparedness, poverty, malnutrition are higher-priority issues. Some of my friends strongly believe that the danger of AI becoming super-powerful and taking over the world is the highest-priority issue … and truthfully, with this week’s announcements of AlphaCode and OpenAI’s theorem prover, which achieve human-competitive performance in elite programming and math competitions respectively, I can’t confidently declare that they’re wrong.

On the other hand, when you think about the astronomical returns on every penny that was invested in setting a teenage Ramanujan or Einstein or Turing or Sofya Kovalevskaya or Norman Borlaug or Mario Molina onto their trajectories in life … and the comically tiny budgets of the world-leading programs that aim to nurture the next Ramanujans, to the point where $10,000 often seems like a windfall to those programs … well, you might come to the conclusion that the “protecting nerds” thing actually isn’t that far down the global priority list! Like, it probably cracks the top ten.

And there’s more to it than that. There’s a reason beyond parochialism, it dawned on me, why individual charities tend to specialize in wildlife conservation in Ecuador or deworming in Swaziland or some other little domain, rather than simply casting around for the highest-priority cause on earth. Expertise matters—since one wants to make, not only good judgments about which stuff to support, but good judgments that most others can’t or haven’t made. In my case, it would seem sensible to leverage the fact that I’m Scott Aaronson. I’ve spent much of my career in math/CS education and outreach—mostly, of course, at the university level, but by god did I personally experience the good and the bad in nearly every form of precollege STEM education! I’m pretty confident in my ability to distinguish the two, and for whatever I don’t know, I have close friends in the area who I trust.

There’s also a practical issue: in order for me to fund something, the recipient has to fill out a somewhat time-consuming application to SFF. If I’d added, say, another $20,000 drop into the bucket of global health or sustainability or whatever, there’s no guarantee that the intended recipients of my largesse would even notice, or care enough to go through the application process if they did. With STEM education, by contrast, holy crap! I’ve got an inbox full of Shtetl-Optimized readers explaining how their little math program is an intellectual oasis that’s changed the lives of hundreds of middle-schoolers in their region, and how $20,000 would mean the difference between their program continuing or not. That’s someone who I trust to fill out the form.

Without further ado, then, here are the first-ever Scott Aaronson Speculation Grants:

• $57,000 for Canada/USA Mathcamp, which changed my life when I attended it as a 15-year-old in 1996, and which I returned to as a lecturer in 2008. The funds will be used for COVID testing to allow Mathcamp to resume in-person this summer, and perhaps scholarships and off-season events as well.
• $30,000 for AddisCoder, which has had spectacular success teaching computer science to high-school students in Ethiopia, placing some of its alumni at elite universities in the US, to help them expand to a new “JamCoders” program in Jamaica. These programs were founded by UC Berkeley’s amazing Jelani Nelson, also with involvement from friend and Shtetl-Optimized semi-regular Boaz Barak.
• $30,000 for the Maine School of Science and Mathematics, which seems to offer a curriculum comparable to those of Thomas Jefferson, Bronx Science, or the nation’s other elite magnet high schools, but (1) on a shoestring budget and (2) in rural Maine. I hadn’t even heard of MSSM before Alex Altair, an alum and Shtetl-Optimized reader, told me about it, but now I couldn’t be prouder to support it.
• $30,000 for the Eugene Math Circle, which provides a math enrichment lifeline to kids in Oregon, and whose funding was just cut. This donation will keep the program alive for another year.
• $13,000 for the Summer Science Program, which this summer will offer research experiences to high-school juniors in astrophysics, biochemistry, and genomics.
• $10,000 for the MISE Foundation, which provides math enrichment for the top middle- and high-school students in Ghana.
• $10,000 for Number Champions, which provides one-on-one coaching to kids in the UK who struggle with math.
• $10,000 for Bridge to Enter Advanced Mathematics (BEAM), which runs math summer programs in New York, Los Angeles, and elsewhere for underserved populations.
• $10,000 for Powderhouse, an innovative lab school being founded in Somerville, MA.

While working on this, it crossed my mind that, on my deathbed, I might be at least as happy about having directed funds to efforts like these as about any of my research or teaching.

To the applicants who weren’t chosen: I’m sorry, as many of you had wonderful projects too! As I said in the earlier post, you remain warmly invited to apply to SFF, and to make your pitch to the other Speculators and/or the main SFF committee.

Needless to say, anyone who feels inspired should add to my (or rather, SFF’s) modest contributions to these STEM programs. My sense is that, while $200k can go eye-poppingly far in this area, it still hasn’t come close to exhausting even the lowest-hanging fruit.

Also needless to say, the opinions in this post are my own and are not necessarily shared by SFF or by the organizations I’m supporting. The latter are welcome to disagree with me as long as they keep up their great work!

Huge thanks again to Jaan, to SFF, to my SFF contact Andrew Critch, to everyone (whether chosen or not) who participated in this contest, and to everyone who’s putting in work to broaden kids’ intellectual horizons or otherwise make the world a little less horrible.

Follow

Exciting news, everyone! Jaan Tallinn, who many of you might recognize as a co-creator of Skype, tech enthusiast, and philanthropist, graciously invited me, along with a bunch of other nerds, to join the new Speculation Grants program of the Survival and Flourishing Fund (SFF). In plain language, that means that Jaan is giving me $200,000 to distribute to charitable organizations in any way I see fit—though ideally, my choices will have something to do with the survival and flourishing of our planet and civilization.

(If all goes well, this blog post will actually lead to a lot more than just $200,000 in donations, because it will inspire applications to SFF that can then be funded by other “Speculators” or by SFF’s usual process.)

Thinking about how to handle the responsibility of this amazing and unexpected gift, I decided that I couldn’t possibly improve on what Scott Alexander did with his personal grants program on Astral Codex Ten. Thus: I hereby invite the readers of Shtetl-Optimized to pitch registered charities (which might or might not be their own)—especially, charities that are relatively small, unknown, and unappreciated, yet that would resonate strongly with someone who thinks the way I do. Feel free to renominate (i.e., bring back to my attention) charities that were mentioned when I asked a similar question after winning $250,000 from the ACM Prize in Computing.

If you’re interested, there’s a two-step process this time:

Step 1 is to make your pitch to me, either by a comment on this post or by email to me, depending on whether you’d prefer the pitch to be public or private. Let’s set a deadline for this step of Thursday, January 27, 2022 (i.e., one week from now). Your pitch can be extremely short, like 1 paragraph, although I might ask you followup questions. After January 27, I’ll then take one of two actions in response: I’ll either

(a) commit a specified portion of my $200,000 to your charity, if the charity formally applies to SFF, and if the charity isn’t excluded for some unexpected reason (5 sexual harassment lawsuits against its founders or whatever), and if one of my fellow “Speculators” doesn’t fund your charity before I do … or else I’ll

(b) not commit, in which case your charity can still apply for funding from SFF! One of the other Speculators might fund it, or it might be funded by the “ordinary” SFF process.

Step 2, which cannot be skipped, is then to have your charity submit a formal application to SFF. The application form isn’t too bad. But if the charity isn’t your own, it would help enormously if you at least knew someone at the charity, so you could tell them to apply to SFF. Again, Step 2 can be taken regardless of the outcome of Step 1.

The one big rule is that anything you suggest has to be a registered, tax-exempt charity in either the US or the UK. I won’t be distributing funds myself, but only advising SFF how to do so, and this is SFF’s rule, not mine. So alas, no political advocacy groups and no individuals. Donating to groups outside the US and UK is apparently possible but difficult.

While I’m not putting any restrictions on the scope, let me list a few examples of areas of interest to me.

• Advanced math and science education at the precollege level: gifted programs, summer camps, online resources, or anything, really, that aims to ensure that the next Ramanujan or von Neumann isn’t lost to the world.
• Conservation of endangered species.
• Undervalued approaches to dealing with the climate catastrophe (including new approaches to nuclear energy, geoengineering, and carbon capture and storage … or even, e.g., studies of the effects of rising CO₂ on cognition and how to mitigate them).
• Undervalued approaches to preventing or mitigating future pandemics—basically, anything dirt-cheap that we wish had been done before covid.
• Almost anything that Scott Alexander might have funded if he’d had more money.
• Anything that would enrage the SneerClubbers or those who attack me on Twitter, by doing stuff that even they would have to acknowledge makes the world better, but that does so via people, organizations, and means that they despise.

Two examples of areas that I don’t plan to focus on are:

• AI-risk and other “strongly rationalist-flavored” organizations (these are already well-covered by others at SFF, so that I don’t expect to have an advantage), and
• quantum computing research (this is already funded by a zillion government agencies, companies, and venture capitalists).

Anyway, thanks so much to Jaan and to SFF for giving me this incredible opportunity, and I look forward to seeing what y’all come up with!

Note: Any other philanthropists who read this blog, and who’d like to add to the amount, are more than welcome to do so!

Follow

(Hopefully no one has taken taken that title yet!)

I waste a large fraction of my existence just reading about what’s happening in the world, or discussion and analysis thereof, in an unending scroll of paralysis and depression. On the first anniversary of the January 6 attack, I read the recent revelations about just how close the seditionists actually came to overturning the election outcome (e.g., by pressuring just one Republican state legislature to “decertify” its electors, after which the others would likely follow in a domino effect), and how hard it now is to see a path by which democracy in the United States will survive beyond 2024. Or I read about Joe Manchin, who’s already entered the annals of history as the man who could’ve halted the slide to the abyss and decided not to. Of course, I also read about the wokeists, who correctly see the swing of civilization getting pushed terrifyingly far out of equilibrium to the right, so their solution is to push the swing terrifyingly far out of equilibrium to the left, and then they act shocked when their own action, having added all this potential energy to the swing, causes it to swing back even further to the right, as swings tend to do. (And also there’s a global pandemic killing millions, and the correct response to it—to authorize and distribute new vaccines as quickly as the virus mutates—is completely outside the Overton Window between Obey the Experts and Disobey the Experts, advocated by no one but a few nerds. When I first wrote this post, I forgot all about the global pandemic.) And I see all this and I am powerless to stop it.

In such a dark time, it’s easy to forget that I’m a theoretical computer scientist, mainly focused on quantum computing. It’s easy to forget that people come to this blog because they want to read about quantum computing. It’s like, who gives a crap about that anymore? What doth it profit a man, if he gaineth a few thousand fault-tolerant qubits with which to calculateth chemical reaction rates or discrete logarithms, and he loseth civilization?

Nevertheless, in the rest of this post I’m going to share some quantum-related debunking updates—not because that’s what’s at the top of my mind, but in an attempt to find my way back to sanity. Picture that: quantum mechanics (and specifically, the refutation of outlandish claims related to quantum mechanics) as the part of one’s life that’s comforting, normal, and sane.

There’s been lots of online debate about the claim to have entangled a tardigrade (i.e., water bear) with a superconducting qubit; see also this paper by Vlatko Vedral, this from CNET, this from Ben Brubaker on Twitter. So, do we now have Schrödinger’s Tardigrade: a living, “macroscopic” organism maintained coherently in a quantum superposition of two states? How could such a thing be possible with the technology of the early 21^st century? Hasn’t it been a huge challenge to demonstrate even Schrödinger’s Virus or Schrödinger’s Bacterium? So then how did this experiment leapfrog (or leaptardigrade) over those vastly easier goals?

Short answer: it didn’t. The experimenters couldn’t directly measure the degree of freedom in the tardigrade that’s claimed to be entangled with the qubit. But it’s consistent with everything they report that whatever entanglement is there, it’s between the superconducting qubit and a microscopic part of the tardigrade. It’s also consistent with everything they report that there’s no entanglement at all between the qubit and any part of the tardigrade, just boring classical correlation. (Or rather that, if there’s “entanglement,” then it’s the Everett kind, involving not merely the qubit and the tardigrade but the whole environment—the same as we’d get by just measuring the qubit!) Further work would be needed to distinguish these possibilities. In any case, it’s of course cool that they were able to cool a tardigrade to near absolute zero and then revive it afterwards.

I thank the authors of the tardigrade paper, who clarified a few of these points in correspondence with me. Obviously the comments section is open for whatever I’ve misunderstood.

People also asked me to respond to Sabine Hossenfelder’s recent video about superdeterminism, a theory that holds that quantum entanglement doesn’t actually exist, but the universe’s initial conditions were fine-tuned to stop us from choosing to measure qubits in ways that would make its nonexistence apparent: even when we think we’re applying the right measurements, we’re not, because the initial conditions messed with our brains or our computers’ random number generators. (See, I tried to be as non-prejudicial as possible in that summary, and it still came out sounding like a parody. Sorry!)

Sabine sets up the usual dichotomy that people argue against superdeterminism only because they’re attached to a belief in free will. She rejects Bell’s statistical independence assumption, which she sees as a mere dogma rather than a prerequisite for doing science. Toward the end of the video, Sabine mentions the objection that, without statistical independence, a demon could destroy any randomized controlled trial, by tampering with the random number generator that decides who’s in the control group and who isn’t. But she then reassures the viewer that it’s no problem: superdeterministic conspiracies will only appear when quantum mechanics would’ve predicted a Bell inequality violation or the like. Crucially, she never explains the mechanism by which superdeterminism, once allowed into the universe (including into macroscopic devices like computers and random number generators), will stay confined to reproducing the specific predictions that quantum mechanics already told us were true, rather than enabling ESP or telepathy or other mischief. This is stipulated, never explained or derived.

To say I’m not a fan of superdeterminism would be a super-understatement. And yet, nothing I’ve written previously on this blog—about superdeterminism’s gobsmacking lack of explanatory power, or about how trivial it would be to cook up a superdeterministic “mechanism” for, e.g., faster-than-light signaling—none of it seems to have made a dent. It’s all come across as obvious to the majority of physicists and computer scientists who think as I do, and it’s all fallen on deaf ears to superdeterminism’s fans.

So in desperation, let me now try another tack: going meta. It strikes me that no one who saw quantum mechanics as a profound clue about the nature of reality could ever, in a trillion years, think that superdeterminism looked like a promising route forward given our current knowledge. The only way you could think that, it seems to me, is if you saw quantum mechanics as an anti-clue: a red herring, actively misleading us about how the world really is. To be a superdeterminist is to say:

OK, fine, there’s the Bell experiment, which looks like Nature screaming the reality of ‘genuine indeterminism, as predicted by QM,’ louder than you might’ve thought it even logically possible for that to be screamed. But don’t listen to Nature, listen to us! If you just drop what you thought were foundational assumptions of science, we can explain this away! Not explain it, of course, but explain it away. What more could you ask from us?

Here’s my challenge to the superdeterminists: when, in 400 years from Galileo to the present, has such a gambit ever worked? Maxwell’s equations were a clue to special relativity. The Hamiltonian and Lagrangian formulations of classical mechanics were clues to quantum mechanics. When has a great theory in physics ever been grudgingly accommodated by its successor theory in a horrifyingly ad-hoc way, rather than gloriously explained and derived?

Update: Oh right, and the QIP’2022 list of accepted talks is out! And I was on the program committee! And they’re still planning to hold QIP in person, in March at Caltech, will you fancy that! actually I have no idea—but if they’re going to move to virtual, I’m awaiting an announcement just like everyone else.

Follow

Happy New Year, everyone!

It was exactly two years ago that it first became publicly knowable—though most of us wouldn’t know for at least two more months—just how freakishly horrible is the branch of the wavefunction we’re on. I.e., that our branch wouldn’t just include Donald Trump as the US president, but simultaneously a global pandemic far worse than any in living memory, and a world-historically bungled response to that pandemic.

So it’s appropriate that I just finished reading Viral: The Search for the Origin of COVID-19, by Broad Institute genetics postdoc Alina Chan and science writer Matt Ridley. Briefly, I think that this is one of the most important books so far of the twenty-first century.

Of course, speculation and argument about the origin of COVID goes back all the way to that fateful January of 2020, and most of this book’s information was already available in fragmentary form elsewhere. And by their own judgment, Chan and Ridley don’t end their search with a smoking-gun: no Patient Zero, no Bat Zero, no security-cam footage of the beaker dropped on the Wuhan Institute of Virology floor. Nevertheless, as far as I’ve seen, this is the first analysis of COVID’s origin to treat the question with the full depth, gravity, and perspective that it deserves.

Viral is essentially a 300-page plea to follow every lead as if we actually wanted to get to the bottom of things, and in particular, yes, to take the possibility of a lab leak a hell of a lot more seriously than was publicly permitted in 2020. (Fortuitously, much of this shift already happened as the authors were writing the book, but in June 2021 I was still sneered at for discussing the lab leak hypothesis on this blog.) Viral is simultaneously a model of lucid, non-dumbed-down popular science writing and of cogent argumentation. The authors never once come across like tinfoil-hat-wearing conspiracy theorists, railing against the sheeple with their conventional wisdom: they’re simply investigators carefully laying out what they’re confident should become conventional wisdom, with the many uncertainties and error bars explicitly noted. If you read the book and your mind works anything like mine, be forewarned that you might come out agreeing with a lot of it.

I would say that Viral proves the following propositions beyond reasonable doubt:

• Virologists, including at Shi Zhengli’s group at WIV and at Peter Daszak’s EcoHealth Alliance, were engaged in unbelievably risky work, including collecting virus-laden fecal samples from thousands of bats in remote caves, transporting them to the dense population center of Wuhan, and modifying them to be more dangerous, e.g., through serial passage through human cells and the insertion of furin cleavage sites. Years before the COVID-19 outbreak, there were experts remarking on how risky this research was and trying to stop it. Had they known just how lax the biosecurity was in Wuhan—dangerous pathogens experimented on in BSL-2 labs, etc. etc.—they would have been louder.
  
• Even if it didn’t cause the pandemic, the massive effort to collect and enhance bat coronaviruses now appears to have been of dubious value. It did not lead to an actionable early warning about how bad COVID-19 was going to be, nor did it lead to useful treatments, vaccines, or mitigation measures, all of which came from other sources.
  
• There are multiple routes by which SARS-CoV2, or its progenitor, could’ve made its way, otherwise undetected, from the remote bat caves of Yunnan province or some other southern location to the city of Wuhan a thousand miles away, as it has to do in any plausible origin theory. Having said that, the regular Yunnan→Wuhan traffic in scientific samples of precisely these kinds of viruses, sustained over a decade, does stand out a bit! On the infamous coincidence of the pandemic starting practically next door to the world’s center for studying SARS-like coronaviruses, rather than near where the horseshoe bats live in the wild, Chan and Ridley memorably quote Humphrey Bogart’s line from Casablanca: “Of all the gin joints in all the towns in all the world, she walks into mine.”
  
• The seafood market was probably “just” an early superspreader site, rather than the site of the original spillover event. No bats or pangolins at all, and relatively few mammals of any kind, appear to have been sold at that market, and no sign of SARS-CoV2 was ever found in any of the animals despite searching.
  
• Most remarkably, Shi and Daszak have increasingly stonewalled, refusing to answer 100% reasonable questions from fellow virologists. They’ve acted more and more like defendants exercising their right to remain silent than like participants in a joint search for the truth. That might be understandable if they’d already answered ad nauseam and wearied of repeating themselves, but with many crucial questions, they haven’t answered even once. They’ve refused to make available a key database of all the viruses WIV had collected, which WIV inexplicably took offline in September 2019. When, in January 2020, Shi disclosed to the world that WIV had collected a virus called RaTG13, which was 96% identical to SARS-CoV2, she didn’t mention that it was collected from a mine in Mojiang, which the WIV had sampled from over and over because six workers had gotten a SARS-like pneumonia there in 2012 and three had died from it. She didn’t let on that her group had been studying RaTG13 for years—giving, instead, the false impression that they’d just noticed it recently, when searching WIV’s records for cousins of SARS-CoV2. And she didn’t see fit to mention that WIV had collected eight other coronaviruses resembling SARS-CoV2 from the same mine (!). Shi’s original papers on SARS-CoV2 also passed in silence over the virus’s furin cleavage site—even though SARS-CoV2 was the first sarbecoronavirus with that feature, and Shi herself had recently demonstrated adding furin cleavage sites to other viruses to make them more transmissible, and the cleavage site would’ve leapt out immediately to any coronavirus researcher as the most interesting feature of SARS-CoV2 and as key to its transmissibility. Some of these points had to be uncovered by Internet sleuths, poring over doctoral theses and the like, after which Shi would glancingly acknowledge the points in talks without ever explaining her earlier silences. Shi and Daszak refused to cooperate with Chan and Ridley’s book, and have stopped answering questions more generally. When people politely ask Daszak about these matters on Twitter, he blocks them.
  
• The Chinese regime has been every bit as obstructionist as you might expect: destroying samples, blocking credible investigations, censoring researchers, and preventing journalists from accessing the Mojiang mine. So Shi at least has the excuse that, even if she’d wanted to come clean with everything relevant she knows about WIV’s bat coronavirus work, she might not be able to do so without endangering herself or loved ones. Daszak has no such excuse.

It’s important to understand that, even in the worst case—that (1) there was a lab leak, and (2) Shi and Daszak are knowingly withholding information relevant to it—they’re far from monsters. Even in Viral‘s relentlessly unsparing account, they come across as genuine believers in their mission to protect the world from the next pandemic.

And it’s like: imagine devoting your life to that mission, having most of the world refuse to take you seriously, and then the calamity happens exactly like you said … except that, not only did your efforts fail to prevent it, but there’s a live possibility that they caused it. It’s conceivable that your life’s work managed to save minus 15 million lives and create minus $50 trillion in economic value.

Very few scientists in history have faced that sort of psychic burden, perhaps not even the ones who built the atomic bomb. I hope I’d maintain my scientific integrity under such an astronomical weight, but I’m doubtful that I would. Would you?

Viral very wisely never tries to psychoanalyze Shi and Daszak. I fear that one might need a lot of conceptual space between “knowing” and “not knowing,” “suspecting” and “not suspecting,” to do justice to the planet-sized enormity of what’s at stake here. Suppose, for example, that an initial investigation in January 2020 reassured you that SARS-CoV2 probably hadn’t come from your lab: would you continue trying to get to the bottom of things, or would you thereafter decide the matter was closed?

For all that, I agree with Chan and Ridley that COVID-19 might well have had a zoonotic origin after all. And one point Viral makes abundantly clear is that, if our goal is to prevent the next pandemic, then resolving the mystery of COVID-19 actually matters less than one might think. This is because, whichever possibility—zoonotic spillover or lab leak—turns out to be the truth of this case, the other possibility would remain absolutely terrifying and would demand urgent action as well. Read the book and see for yourself.

Searching my inbox, I found an email from April 16, 2020 where I told someone who’d asked me that the lab-leak hypothesis seemed perfectly plausible to me (albeit no more than plausible), that I couldn’t understand why it wasn’t being investigated more, but that I was hesitant to blog about these matters. As I wrote seven months ago, I now see my lack of courage on this as having been a personal failing. Obviously, I’m just a quantum computing theorist, not a biologist, so I don’t have to have any thoughts whatsoever about the origin of COVID-19 … but I did have some, and I didn’t share them here only because of the likelihood that I’d be called an idiot on social media. Having now read Chan and Ridley, though, I think I’d take being called an idiot for this book review more as a positive signal about my courage than as a negative signal about my reasoning skills!

At one level, Viral stands alongside, I dunno, Eichmann in Jerusalem among the saddest books I’ve ever read. It’s 300 pages of one of the great human tragedies of our lifetime balancing on a hinge between happening and not happening, and we all know how it turns out. On another level, though, Viral is optimistic. Like with Richard Feynman’s famous “personal appendix” about the Space Shuttle Challenger explosion, the very act of writing such a book reflects a view that you’re still allowed to ask questions; that one or two people armed with nothing but arguments can run rings around governments, newspapers, and international organizations; that we don’t yet live in a post-truth world.

Follow