{"id":10204,"date":"2026-10-19T16:07:12","date_gmt":"2026-10-19T21:07:12","guid":{"rendered":"https:\/\/scottaaronson.blog\/?p=10204"},"modified":"2026-10-19T16:09:46","modified_gmt":"2026-10-19T21:09:46","slug":"ai-alignment-theory-course-readings-and-rapporteur-reports","status":"publish","type":"post","link":"https:\/\/scottaaronson.blog\/?p=10204","title":{"rendered":"AI Alignment Theory Course: Readings and Rapporteur Reports!"},"content":{"rendered":"\n

As I announced on this blog a couple weeks ago<\/a>, I’m teaching a new course this semester at UT Austin, entitled AI Alignment Theory. To call such a course “timely” seems like an understatement; this is the first course I’ve ever taught for which multiple gargantuan developments relevant to the course have happened while I was teaching it.<\/p>\n\n\n\n

Since AI alignment is still a field with no standard textbook (although that might be changing<\/a>), no accepted body of mathematical theory, etc. etc., the way we’ve been running the course is like a humanities discussion seminar, albeit one that happens to be in a CS department and to include technical results from time to time. Basically, in each class session, a student “rapporteur” gives a talk on an AI alignment reading—which include “ancient” readings (from way back in 2016), “modern” readings (from, say, last week), and everything in between—and then we freely discuss and debate whatever was presented, with my own role mostly limited to that of gadfly (occasionally I’ll do a mini-lecture on some technical concept).<\/p>\n\n\n\n

Anyway, two weeks ago, I promised a list of the papers and other writings that we’ve been discussing, as well as student “rapporteur reports” on those papers, which cover not only the papers themselves, but the students’ thoughts about the papers and (anonymized) summaries of points raised during our class discussion. People have been bugging me for that ever since.<\/p>\n\n\n\n

Since I am to please, here you are, without further ado! I’ll continue to update the list as the semester progresses, so that this post can serve as a central public repository for the course.<\/p>\n\n\n\n

    \n
  1. AGI Ruin: A List of Lethalities<\/a>, by Eliezer Yudkowsky (2022). Rapporteur report by Tennyson Bardwell<\/strong><\/a> (this is the one that we posted previously).
    <\/li>\n\n\n\n
  2. Concrete Problems in AI Safety<\/a>, by Amodei, Olah, Steinhardt, Christiano, Schulman, and Man\u00e9 (2016). Rapporteur report by Daniela Wiepert.<\/a><\/strong>
    <\/li>\n\n\n\n
  3. The Alignment Problem from a Deep Learning Perspective<\/a>, by Ngo, Chan, and Mindermann (many versions but latest is 2025), and Research Priorities for Robust and Beneficial Artificial Intelligence<\/a> by Russell, Dewey, and Tegmark (2016). Rapporteur report by Oleg Pittman and Jack Holt.<\/a><\/strong>
    <\/li>\n\n\n\n
  4. What Happened: OpenAI and HuggingFace<\/a>, by Zvi Mowshowitz; The Rise and Fall of Agent Civilizations<\/a>, by Dwarkesh Patel, and the METR report by on the HuggingFace incident<\/a>, by Greenblatt, Cotra, and Wijk (all 2026). Rapporteur report by Shourjo Aditya Chaudhuri and John Robertson.<\/a><\/strong>
    <\/li>\n\n\n\n
  5. We Must Pace the Frontier<\/a>, by Dario Amodei (2026). Rapporteur report by Dhilan Shah.<\/a><\/strong>
    <\/li>\n\n\n\n
  6. The Basic AI Drives<\/a>, by Stephen Omohundro (2008), and Optimal Policies Tend to Seek Power<\/a>, by Turner, Smith, Shah, Critch, and Tadepalli (2023). Rapporteur report by Bill Zheng.<\/a><\/strong>
    <\/li>\n\n\n\n
  7. Corrigibility<\/a>, by Soares, Fallenstein, Yudkowsky, and Armstrong (2015). Rapporteur report by Sukrut Chikodikar.<\/a><\/strong>
    <\/li>\n\n\n\n
  8. The Off-Switch Game<\/a>, by Hadfield-Menell, Dragan, Abbeel, and Russell (2017). Rapporteur report by Praneel Seth.<\/a><\/strong>
    <\/li>\n\n\n\n
  9. Planting Undetectable Backdoors in Machine Learning Models<\/a>, by Goldwasser, Kim, Vaikuntanathan, and Zamir (latest version 2024), and Oblivious Defense in ML Models: Backdoor Removal without Detection<\/a>, by Goldwasser, Shafer, Vafa, and Vaikuntanathan (2025). Rapporteur report by Agrim Jain and Indira Dutta.<\/a><\/strong>
    <\/li>\n\n\n\n
  10. Backdoor Defense, Learnability, and Obfuscation<\/a> by Christiano, Hilton, Lecomte, and Zhu (2024), and Unelicitable Backdoors in Language Models via Cryptographic Transformer Circuits<\/a>, by Draguns et al. (2024). Rapporteur report by John Dunbar (coming).<\/strong>
    <\/li>\n\n\n\n
  11. Functional Decision Theory: A New Theory of Instrumental Rationality<\/a>, by Eliezer Yudkowsky and Nate Soares (2018). Rapporteur report by Isaac Noel (coming).<\/strong>
    <\/li>\n\n\n\n
  12. Robust Cooperation In the Prisoner’s Dilemma: Program Equilibrium via Provability Logic<\/a>, by Barasz, Christiano, Fallenstein, Herreshoff, LaVictoire, and Yudkowsky (latest version 2021). Rapporteur report by Shunan Zheng (coming).<\/strong>
    <\/li>\n\n\n\n
  13. Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training<\/a>, by Hubinger et al. (long list of authors) (2024). Rapporteur report by Anthony Wang (coming).<\/strong>
    <\/li>\n\n\n\n
  14. Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback<\/a>, by Conitzer et al. (long list of authors) (2024) and Toward a Theory of Value in AI Alignment<\/a>, by Ahmed, Birhane, Smart, Kay, Shrishak, and Tobin (2026). Rapporteur report by Priyanka Sadagopan (coming).<\/strong>
    <\/li>\n\n\n\n
  15. Claude\u2019s Constitution<\/a>, by Amanda Askell, Joe Carlsmith, Chris Olah, Jared Kaplan, Holden Karnofsky, et al. (2026). Rapporteur report by Alan Li (coming).<\/strong>
    <\/li>\n\n\n\n
  16. AI Safety via Debate<\/a>, by Geoffrey Irving, Paul Christiano, and Dario Amodei (2018). Rapporteur report by Sampoorna Poria (coming).<\/strong>
    <\/li>\n\n\n\n
  17. Scalable AI Safety via Doubly-Efficient Debate<\/a>, by Jonah Brown-Cohen, Geoffrey Irving, and Georgios Piliouras (2023). Rapporteur report by Aarushi Lakhi (coming).<\/strong>
    <\/li>\n\n\n\n
  18. <\/a>AI as Normal Technology<\/a>, by Arvind Narayanan and Sayash Kapoor (2025). Rapporteur report by Weixi Chen (coming).<\/strong>
    <\/li>\n\n\n\n
  19. <\/a>On the Dangers of Stochastic Parrots<\/a>, by Emily Bender, Timnit Gebru, Angelina McMillan-Major and Shmargaret Shmitchell (2021). Rapporteur report by George Tsoukalas (coming).<\/strong><\/li>\n<\/ol>\n\n\n\n

    (More readings TBD, until the semester ends!)<\/p>\n\n\n\n

    Thanks to my TA, Lin Lin Lee, for her help in getting all of this together.<\/p>\n","protected":false},"excerpt":{"rendered":"

    As I announced on this blog a couple weeks ago, I’m teaching a new course this semester at UT Austin, entitled AI Alignment Theory. To call such a course “timely” seems like an understatement; this is the first course I’ve ever taught for which multiple gargantuan developments relevant to the course have happened while I […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"advanced_seo_description":"","jetpack_seo_html_title":"","jetpack_seo_noindex":false,"jetpack_seo_schema_type":"","_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_hide_featured_image":false,"_jetpack_featured_image_duplicate_dismissed":0,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"{title}\n\n{excerpt}\n\n{url}","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"_wpas_customize_per_network":false,"jetpack_post_was_ever_published":false},"categories":[48,31,8],"tags":[],"class_list":["post-10204","post","type-post","status-publish","format-standard","hentry","category-ai","category-announcements","category-the-fate-of-humanity"],"jetpack_publicize_connections":[],"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=\/wp\/v2\/posts\/10204","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=10204"}],"version-history":[{"count":5,"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=\/wp\/v2\/posts\/10204\/revisions"}],"predecessor-version":[{"id":10212,"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=\/wp\/v2\/posts\/10204\/revisions\/10212"}],"wp:attachment":[{"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=10204"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=10204"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/scottaaronson.blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=10204"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}