In Support of SB 1047
I’ve finished my two-year leave at OpenAI, and returned to being just a normal (normal?) professor, quantum complexity theorist, and blogger. Despite the huge drama at OpenAI that coincided with my time there, including the departures of most of the people I worked with in the former Superalignment team, I’m incredibly grateful to OpenAI for giving me an opportunity to learn and witness history, and even to contribute here and there, though I wish I could’ve done more.
Over the next few months, I plan to blog my thoughts and reflections about the current moment in AI safety, inspired by my OpenAI experience. You can be certain that I’ll be doing this only as myself, not as a representative of any organization. Unlike some former OpenAI folks, I was never offered equity in the company or asked to sign any non-disparagement agreement. OpenAI retains no power over me, at least as long as I don’t share confidential information (which of course I won’t, not that I know much!).
I’m going to kick off this blog series, today, by defending a position that differs from the official position of my former employer. Namely, I’m offering my strong support for California’s SB 1047, a first-of-its-kind AI safety regulation written by California State Senator Scott Wiener, then extensively revised through consultations with pretty much every faction of the AI community. AI leaders like Geoffrey Hinton, Yoshua Bengio, and Stuart Russell are for the bill, as is Elon Musk (for whatever that’s worth), and Anthropic now says that the bill’s “benefits likely outweigh its costs.” Meanwhile, Facebook, OpenAI, and basically the entire VC industry are against the bill, while California Democrats like Nancy Pelosi and Zoe Lofgren have also come out against it for whatever reasons.
The bill has passed the California State Assembly by a margin of 48-16, having previously passed the State Senate by 32-1. It’s now on Governor Gavin Newsom’s desk, and it’s basically up to him whether it becomes law or not. I understand that supporters and opponents are both lobbying him hard.
People much more engaged than me have already laid out, accessibly and in immense detail, exactly what the current bill does and the arguments for and against. Try for example:
- For a very basic explainer, this in TechCrunch
- This by Kelsey Piper, and this by Kelsey Piper, Sigal Samuel, and Dylan Matthews in Vox
- This by Zvi Mowshowitz (Zvi has also written a great deal else about SB 1047, strongly in support)
Briefly: given the ferocity of the debate about it, SB 1047 does remarkably little. It says that if you spend more than $100 million to train a model, you need to notify the government and submit a safety plan. It establishes whistleblower protections for people at AI companies to raise safety concerns. And, if a company failed to take reasonable precautions and its AI then causes catastrophic harm, it says that the company can be sued (which was presumably already true, but the bill makes it extra clear). And … unless I’m badly mistaken, those are the main things in it!
While the bill is mild, opponents are on a full scare campaign saying that it will strangle the AI revolution in its crib, put American AI development under the control of Luddite bureaucrats, and force companies out of California. They say that it will discourage startups, even though the whole point of the $100 million provision is to target only the big players (like Google, Meta, OpenAI, and Anthropic) while leaving small startups free to innovate.
The only steelman that makes sense to me, for why many tech leaders are against the bill, is the idea that it’s a stalking horse. On this view, the bill’s actual contents are irrelevant. What matters is simply that, once you’ve granted the principle that people worried about AI-caused catastrophes get a seat at the table, any legislative acknowledgment of the validity of their concerns—then they’re going to take a mile rather than an inch, and kill the whole AI industry.
Notice that the exact same slippery-slope argument could be deployed against any AI regulation whatsoever. In other words, if someone opposes SB 1047 on these grounds, then they’d presumably oppose any attempt to regulate AI—either because they reject the whole premise that creating entities with humanlike intelligence is a risky endeavor, and/or because they’re hardcore libertarians who never want government to intervene in the market for any reason, not even if the literal fate of the planet was at stake.
Having said that, there’s one specific objection that needs to be dealt with. OpenAI, and Sam Altman in particular, say that they oppose SB 1047 simply because AI regulation should be handled at the federal rather than the state level. The supporters’ response is simply: yeah, everyone agrees that’s what should happen, but given the dysfunction in Congress, there’s essentially no chance of it anytime soon. And California suffices, since Google, OpenAI, Anthropic, and virtually every other AI company is either based in California or does many things subject to California law. So, some California legislators decided to do something. On this issue as on others, it seems to me that anyone who’s serious about a problem doesn’t get to reject a positive step that’s on offer, in favor of a utopian solution that isn’t on offer.
I should also stress that, in order to support SB 1047, you don’t need to be a Yudkowskyan doomer, primarily worried about hard AGI takeoffs and recursive self-improvement and the like. For that matter, if you are such a doomer, SB 1047 might seem basically irrelevant to you (apart from its unknowable second- and third-order effects): a piece of tissue paper in the path of an approaching tank. The world where AI regulation like SB 1047 makes the most difference is the world where the dangers of AI creep up on humans gradually, so that there’s enough time for governments to respond incrementally, as they did with previous technologies.
If you agree with this, it wouldn’t hurt to contact Governor Newsom’s office. For all its nerdy and abstruse trappings, this is, in the end, a kind of battle that ought to be familiar and comfortable for any Democrat: the kind with, on one side, most of the public (according to polls) and also hundreds of the top scientific experts, and on the other side, individuals and companies who all coincidentally have strong financial stakes in being left unregulated. This seems to me like a hinge of history where small interventions could have outsized effects.
Follow
Comment #1 September 4th, 2024 at 11:51 am
First efforts generally suck. Many smart people are trying to make this law not suck, and I expect them to mostly fail.
Nonetheless, it’s important to pass this law simply so that the next iteration can build upon it and suck less. Humanity needs to start practicing how to respond to the threat of AI doom and the only way we can practice is by doing.
Comment #2 September 4th, 2024 at 11:56 am
Charlie #1: Yeah, the other central argument against SB 1047 was, “we concede that AI needs to be regulated, but we shouldn’t rush into things—let’s take our time, consider all perspectives, etc.” But this wasn’t rushed into; Senator Wiener seems to have engaged all the relevant stakeholders and revised the bill based on their feedback to an impressive extent. It seems unlikely that anything better is going to emerge from the sausage grinder in the present state of the world. And, as you say, a better bill can always be passed in the future, if and when someone figures out what it should say.
Comment #3 September 4th, 2024 at 12:06 pm
Sorry, but the entire premise of this bill strikes me as nothing more than luddite crackpottery. The large language models and text-to-image models are just programs for generating text and images. That’s it: text and images. Characters and pixels. All they are is arrays of floating-point numbers on a computer that map text to images, etc. It’s like, they’re not physical ROBOTS. They don’t have physical robot bodies that can kill people. They don’t control any physical systems. They can’t control drones or whatever. They are literally just fucking DATA on a server somewhere. ALL they can do is generate images and text on request. That is literally impossible to be “dangerous,” because all it can do is PUT PIXELS ON A FUCKING SCREEN. So if you don’t like it, TURN OFF YOUR COMPUTER, simple as! It’s about as dangerous as “cyber bullying” is actual bullying. If you can escape the killer robots by just turning off your computer screen then WTF?
Please: Give me ONE, a SINGLE, example of how ChatGPT or the like could be dangerous. One thing it could do that’s dangerous. Just ONE example. Thanks.
Comment #4 September 4th, 2024 at 12:08 pm
I don’t think your steelman is strong enough. The slippery slope is regulating a technology that didn’t yet do any significant harm. It’s like regulating the first car manufacturers before anyone died of a car accident.
AI might be risky, and it might make sense to regulate it at some point, but only after we learn more about the actual harms it causes.
Comment #5 September 4th, 2024 at 12:08 pm
As to whether AI regulation should be at the federal level or the state level, there’s another argument that introducing it at state level is fine. States are often used as “laboratories of democracy.” The idea is to try the law out at the state level and, if it works, Congress can turn it into a federal law, presumably superseding similar laws at the state level. Of course, deciding whether a law like SB 1047 is a success is likely to be problematic.
Comment #6 September 4th, 2024 at 12:54 pm
I think there are perfectly reasonable objections to the technical details of the law. I’m totally fine with the high-level idea that you should have to perform reasonable testing on your AI model to make sure it’s not going to take down the power grid or whatever. But the specific implementation in the bill is problematic on a few fronts. Most glaringly, the demands it makes about derivative models (those produced by fine-tuning or otherwise modifying a base model) are unreasonable and unsatisfiable even in theory. A model developer is asked to present a plan to test and safeguard against a modified version of their model being able to cause a covered harm. Modification includes fine-tuning with up to $10M worth of compute, but also includes any other non-fine-tuning modification, combination with external software or data, with no upper bound on how much compute may be spent.
I simply don’t see how it’s possible, even in principle, to test or safeguard against such expansive modification, other than by prohibiting the release of model weights at all. I’d be very interested to hear if you or another commenter has a sketch of how this could be done.
As written, my feeling is that the bill effectively outlaws releasing the weights of a covered model, barring courts simply choosing to ignore the derivative model requirements.
Comment #7 September 4th, 2024 at 12:59 pm
chatgpt4president #3 and Ilay #4: People all over the world are already using LLMs to write production code. It’s really not hard to imagine that soon some of this code will be running critical infrastructure like factories, dams, or power stations, and that bugs or vulnerabilities in the code will then cause various disasters—not, like, Terminators gunning people down in the streets, but the kinds of small disasters that are regularly adjudicated in court. In such a case, our first guess for who’s responsible should of course be whoever deployed the flawed code. But if a problem is ultimately traced to the LLM provider deciding to cut corners with its own safety plan … well then, why shouldn’t the provider be liable? I see this less as brand-new law, than as a reaffirmation of how most of us would’ve imagined that the law already worked.
But the most obvious thing to say is: anyone who thinks it’s ridiculous to imagine the current generation of AIs causing serious harm, should for that very reason have no objection to a liability regime, which only starts to have non-negligible consequences in the event that there is such harm!!
Comment #8 September 4th, 2024 at 1:03 pm
@chatgpt4president I’m not going to go through all the doomer arguments here, but you should recognize that “its just outputting text and images” is an extremely weak argument against AI worries.
Use some creativity. What *can’t* you do, just with text? Something you *can* do with text, is convince humans to do your bidding. If you can do that, the “not having a physical body” point immediately becomes moot, doesn’t it?
Comment #9 September 4th, 2024 at 1:20 pm
From Zvi’s blogpost (I haven’t read the law myself so might be missing something important) the following conditions seem strange:
— Fine-tunes that spend $10 million or more are the responsibility of the fine-tuner.
— Fine-tunes spending less than that are the responsibility of the original developer.
Does this mean that someone who produces an open source model needs to evaluate not just the risks of their own model, but all models that can be produced as a result of $9.9 million worth of fine tuning? This seems like a difficult ask in principle.
Comment #10 September 4th, 2024 at 1:34 pm
Scott #6: for a long time, people have referred to books when writing production code. If, for example, there is a typo in an algorithm in Numerical Recipes, and someone uses that incorrect algorithm to do some engineering of critical infrastructure, and that mistake leads to a failure, should the authors or publisher of Numerical Recipes be liable? I’d say certainly not, but how do we distinguish between that and an LLM writing an incorrect algorithm? I’m not trying to argue one side or the other, and I agree the two situations are quite different, but I’d also argue that both cases have some similarity, that both cases are giving you code without a warranty, that you can use as-is. So, what’s the difference, why legal responsibility in one case but not the other? Please don’t just say “they are obviously different”, because, while true, it doesn’t give a way to really draw a line.
Comment #11 September 4th, 2024 at 1:37 pm
The “right” words are enough to induce suicide in fragile people, through social media bullying/shaming or even AI chat:
https://www.euronews.com/next/2023/03/31/man-ends-his-life-after-an-ai-chatbot-encouraged-him-to-sacrifice-himself-to-stop-climate-
So, in principle, a true “AGI” will eventually be so good at human psychology and manipulation that it would be able talk anyone into doing pretty much anything, by always finding and exploiting some specific angle/personality weakness, and just slowly work on it.
Comment #12 September 4th, 2024 at 1:44 pm
Ilay #4,
I bet you don’t like some of the supporters of this bill, but don’t you think the software of autonomous cars should be reasonably safe? Grabby software or boring software, this bill seems to ask that if you train something massive enough, it should be certified. Sounds reasonable, like airplanes or boats.
Comment #13 September 4th, 2024 at 1:49 pm
@Aaron Roth #9, note that the $10M figure applies only to fine-tuning. In addition to being asked to evaluate and address the risks of the entire space of model weights that can be reached for less than $10M (using unknown fine-tuning data, no less), the developer is also asked to handle the following other forms of modification, with no dollar cap:
(2) A copy of a covered model that has been subjected to post-training modifications unrelated to fine-tuning.
This is a very broad category of modifications, and seems to cover everything that is not fine-tuning, which the bill defines as “adjusting the model weights of a trained covered model or covered model derivative by exposing it to additional data.” As an example, even something as simple as training a LoRA seems to fall outside of the definition of fine-tuning (because it leaves the original weights unmodified), and so has no cap on the amount of compute that can be spent while retaining the the original developer’s responsibility.
(4) A copy of a covered model that has been combined with other software.
Here, the developer is asked to address risks produced by combining the model with external software to which they do not have access, and may not even yet exist at the time the model is created. At a minimum, it covers something like using a language model as a component of a method like FunSearch. One could easily imagine using an otherwise benign code model combined with evolutionary search to develop cybersecurity threats of the kind the bill covers. In such a system, the model would only be responsible for producing harmless (but syntactically correct and usefully varied) code edits. Again, the adversary could spend an arbitrary amount of money on this, and the original developer would retain responsibility.
Comment #14 September 4th, 2024 at 1:55 pm
By definition, 99% of our interactions with the world is through images (what we see), sounds (what we hear) and words (what others think), all related to ideas.
Those are plenty of tools for a “super intelligent” agent to harm us in unlimited number of ways. Just consider how modern democracies are already straining under the strain of human online discourse through social media, causing deep irreversible rifts in our societies.
Brute physical force is really not that necessary, unless you’re dealing with a bear, which only has that angle to harm us.
Comment #15 September 4th, 2024 at 1:55 pm
Dissent. I’m not being paid to argue this, so I’m not going to write a detailed counter-lobbying brief. But everything I see from my hard-bitten political cynicism is waving red-flags to me that this a pure “regulatory capture” bill. Some serious Big Money wants it, and they don’t do stuff like that out of altruism (not to be confused with the brand Effective Altruism). BIG RED FLAG: Margins like that 48-16 / 32-1 in California don’t happen on tech issues without lot of money. This has nothing to do with “safety”. It’s one set of business interests trying to use the government against another set of business interests.
Scott, the steelman is that this is a legal weapon for big companies to use against smaller companies. You say “$100 million”, and you apparently assume it’s a simple concept. But it’s like saying “hate speech”. The objection is that it’s a way of draining a smaller competitor with legal costs, because proving something isn’t covered, and/or “safe”, will take up a fortune in lawyer billing and expert-witness fees.
This isn’t hard. It’s amazing to me, how on almost every tech regulatory topic one could count on rationalist-type reciting a catechism about strangling innovation, blah blah blah. Yet so many prominent tech people have now become passionate believers in the reasonableness of tech-focused regulation. Really, would some of you folks sign on to regulatory efforts for algorithmic fairness regarding racism, sexism, etc? (that was a rhetorical question – I know the answer).
Comment #16 September 4th, 2024 at 2:01 pm
“anyone who thinks it’s ridiculous to imagine the current generation of AIs causing serious harm, should for that very reason have no objection to a liability regime, which only starts to have non-negligible consequences in the event that there is such harm!!”
You’re neglecting that SB 1047 requires companies to file plans ahead of time, and that these plans will, if anything goes wrong, be used by courts to establish whether conduct was reasonable. Such pre-establishing the standard of care has one big effect with two potential consequences. The effect is that everyone will basically file the exact same types of AI Safety plans. Why will they do this? Because if something goes wrong in the future, the best defense that your practice was reasonable will be that you did exactly what everyone else was doing.
What are the consequences of this effect? I see four different scenarios.
Scenario 1: No critical harms happen because the standard that everyone chooses now is exactly the right mitigating conduct to prevent all future critical harms. Congrats, this is the SB 1047 winner scenario. Nailed it guys.
Scenario 2: No critical harms happen but for reasons unrelated to SB 1047. Perhaps the tech simply isn’t that dangerous, or we have lots of other mitigations. In that case, SB 1047 will have imposed a deadweight cost with no benefit.
(Although both Scenarios 1 & 2 involve no critical harms, we may be able to distinguish them by observing whether critical harms occur in jurisdictions where SB 1047 does not apply.)
Scenario 3: A critical harm happens because someone didn’t comply with SB 1047. This is sort of a win for SB 1047, because although obviously it didn’t prevent the critical harm, the AG can get some extra $$$ from the violator. Yay? This seems like a very unlikely scenario, as people will file the paperwork.
Scenario 4: A critical harm happens even though people complied with SB 1047. THIS IS A REALLY BIG PROBLEM SCENARIO, because it means SB 1047 made AI safety more brittle. Everyone copies each other, so if they fail, they all fail in exactly the same way. (They also all probably escape liability, although god knows the AG is going to want to nail somebody.)
So, where does that leave us if we adopt SB 1047?
Scenario 1: Better than status quo because we set the perfect reasonableness standard even though no one could write it down in law today.
Scenario 2: Worse than status quo because unnecessarily spent money and resources
Scenario 3: Same as status quo except extra $$$ and an AG gets a career boost
Scenario 4: Way worse than status quo because AI Safety becomes blinkered.
Comment #17 September 4th, 2024 at 2:16 pm
For minimum safety standards for cars we have crash tests and crash-test dummies. I speculate we need to develop test AI’s which act like dummies or provocateurs in providing thousands of test queries to prospective AI-tunings before they are released to the public. Or we could start with human testers, but it seems like something which could be automated. I’m from the school that says all code needs to be tested, even if it came out of a manual.
From what little I’ve looked at so far, SB 1047 won’t do much harm nor a great deal of good, but something needs to be done as a start, and it is something.
Comment #18 September 4th, 2024 at 2:37 pm
Nick Moran #6 and Aaron Roth #9: It seems to me that taking the derivative models clause out of SB 1047 would be sort of like taking the individual mandate out of Obamacare — it would eviscerate the entire thing. This is because we’ve seen, empirically, that it’s trivial to remove safety measures in AI models by fine-tuning them, given access to the weights.
So, like, suppose your internal testing shows that your model can be used to engineer a new COVID-like global pandemic, with little more effort on the part of a human operator than asking it to do that. Suppose you then do some RLHF to get it to refuse to comply with such requests and then publish the weights. And suppose someone else, using a day of compute, does some further RL to remove your safeguards and publishes the new weights. And suppose a third person indeed uses the jailbroken model to start a global pandemic that kills ten million people. Where should the liability lie?
I say: while only the third person had murderous intent, the first two are both guilty of something like reckless endangerment. The second person should not have published an AI with which any idiot could start a global pandemic, but also, the first person should not have published the weights if they understood that doing so would inevitably enable the second person. If a liability regime prevents the first person from publishing the weights in the scenario I described, that’s not an unintended byproduct but the intended effect.
Comment #19 September 4th, 2024 at 2:39 pm
Scott #6: I’m not saying current AIs can’t cause serious harms, I’m saying they haven’t so far, as far as I know, and we should worry about the precedent of regulating a technology against hypothetical harms (even if the proposed law by itself is inconsequential).
Of course, SB 1047 might in practice have the opposite effect – it could be the most minimal law that placates the demand for AI regulation, preventing more draconian laws in the future. It’s hard to predict these kinds of things!
Comment #20 September 4th, 2024 at 2:47 pm
Seth Finkelstein #15: The Big Money (Facebook, Meta, a16z, etc) is essentially all on the opposite side here—the side of killing SB 1047, not of passing it. And the bill, as written, is the opposite of a “regulatory capture” bill, since it imposes obligations on entrenched players that can afford $100M for training runs, and does not impose any similar obligations on small newcomers. If you indulge in more “nerd-baiting” and repetition of your flawed priors rather than engaging with these obvious facts then you’re no longer welcome in this discussion.
Incidentally, I’d certainly be open to legislation that dealt with algorithmic bias and discrimination in a smart and effective way. I promise to consider such legislation carefully, whenever it’s on the table. No more whataboutism.
Comment #21 September 4th, 2024 at 2:57 pm
@Scott #18 I am sympathetic to the need to protect against trivially-removed safety bandaids. Clearly, a model which can be made to do dangerous things given only a day’s worth of fine-tuning is still dangerous. I agree with you that it makes sense to attempt to capture this scenario in the law. But the law does so by casting an incredibly wide net, one which makes it impossible to release even a benign open source model. Is your position that this is just the cost we have to pay, and that it should be illegal to open source a covered model? Or do you disagree and feel that it is in fact possible to produce a model and testing plan which satisfies the law’s demands of derivative models?
Suppose, for example, I endeavor to make my model pandemic-safe by removing all virology and bio-engineering data from my training set. My model as released has no idea how to engineer a pathogen, it cannot give you advice on how to do that no matter how you ask. Its inability is not tied to safety tuning, so you cannot just do a little RLHF to peel back a refusal. But an adversary possess a trove of non-public virology research documents. They train a supplementary large model (too small to meet the flop count to be its own covered model, but competitive with today’s biggest models) which knows only about virology. Its general reasoning capabilities are quite weak compared to the future frontier model, but it has excellent recall of virology facts and can do a pretty good job of answering questions. The adversary combines this with a $9M fine-tune of my safe model, as well as a RAG system that can access the original virology documents, and spends hundreds of millions worth of inference compute generating many proposals which are then filtered for feasibility and synthesized.
In this scenario, it seems to me that the original developer should not bear liability. The dangerous capabilities came from the adversary, and the only thing the original developer could have done to stop them is to refuse to release the model in the first place. But the law says that the adversary’s system is still a derivative model, and the original developer should have tested for this capability and made a plan to prevent it.
Do you believe that:
A) SB-1047 does assign liability and responsibility to the original developer, and this is the correct thing to do.
B) SB-1047 does assign liability and responsibility to the original developer, and while this is perhaps unfair, it is the price we must pay for safety, because it is not feasible to draw a different line.
C) SB-1047 would assign liability and responsibility to the original developer, but there actually exists a technical safeguard the developer could have implemented that would have stopped the adversary while still allowing the developer to open source their model.
D) SB-1047 actually does not assign liability or responsibility to the original developer, and so they would not have been prevented from open sourcing their model.
E) The scenario described is fundamentally not possible even for a dedicated adversary.
If the answer is A or B, I think the bill’s supporters should be more upfront about this fact – that their aim is to ban open sourcing of covered models. If the answer is C, I’d be very curious to hear what that safeguard is. If the answer is D, I’d be very curious to know what provision(s) would let the original developer off the hook here.
Comment #22 September 4th, 2024 at 3:44 pm
Ilay #19: Should nuclear power have been totally unregulated prior to Three Mile Island, or whatever the first nuclear disaster was? If it had been regulated enough to prevent the disaster, wouldn’t nuclear be in a vastly better place today?
Comment #23 September 4th, 2024 at 3:52 pm
Scott, you have again apparently not read, or not understood, a point I’ve made, yet reacted harshly due to the misconception. To wit:
“… on entrenched players that can afford $100M for training runs, and does not impose any similar obligations on small newcomers.”
I specifically countered this. The rebuttal is “$100M” is not as simple a concept as you appear to think, and there are severe problems with the cost of establishing not being covered, which small newcomers might not be able to afford. Now, agree or disagree, that is a response on the issue.
Further, I don’t believe I’m “nerd-baiting”, as you will find nothing so general (I’m 100% nerd! – MIT Math and Physics degrees, professional computer programmer). I do think many negative things about certain specific rationalist and Libertarian ideas, and also that AI Doomers are babbling nonsense, that’s true.
And here – “repetition of your flawed priors rather than engaging with these obvious facts” – regrets, this is a demand I accept your priors, see above. I don’t, in fact I think you are naive and wrong on many of them.
I have no desire to be the resident anti-AI-Doom guy of your blog. It’s just a topic I find very interesting, since AI is a technological revolution. Unfortunately my participation in your blog seems to be negative for both of us. I won’t make an issue about “no longer welcome in this discussion”, I respect your right to cultivate your community as you see fit. I do not wish you ill, and hold no grudge against you. Be well, and best wishes for the discussions in your future posts.
Comment #24 September 4th, 2024 at 4:35 pm
> They say that it will discourage startups, even though the whole point of the $100 million provision is to target only the big players (like Google, Meta, OpenAI, and Anthropic) while leaving small startups free to innovate.
Assuming this is a sincere point of confusion, I will explain why SB-1047 is bad for startups as best I understand it.
Startups generally can’t afford to train foundation models themselves. They rely on access to foundation models provided by the larger companies such as those you mention, whether that’s through open weights (such as Llama) or an API. If SB-1047 discourages companies like Meta from releasing models like Llama 3.1 405B, that hurts the startups that build on such models.
It’s plausible to me that discouraging Llama-like releases would not be an accidental side effect but rather is the primary motivation for SB-1047. See 1a3orn’s blog post “Many AI Safety Orgs Have Tried to Criminalize Currently-Existing Open-Source AI”.
Comment #25 September 4th, 2024 at 5:15 pm
Thank you Scott. We don’t agree on many things but you have always struck me as reasonable and good-hearted, and now it seems you are somewhat brave as well.
Replying to @Neil comment #16:
This seems like an unfair taxonomy of scenarios. And/or maybe I just disagree with some of your implicit claims there — e.g. you think that the ‘diverse approaches to safety’ effect is way more valuable than the ‘companies are motivated to have a good-as-judged-by-others safety plan and follow it’ effect. But that seems bonkers to me. diverse approaches are only better than just picking one approach at random if there’s some mechanism that systematically selects for the better approaches; in the absence of SB 1047, there is if anything a mechanism selecting in the opposite direction — race dynamics. Whereas with SB 1047, you still get diverse approaches, but there’s pressure for your approach to look good to others, which is definitely not the same thing as actually being good, but seems better than the alternative in which you don’t need to have an approach at all or show it to anyone.
The corporations shouldn’t get to write and grade their own safety exams. They are disposed to give themselves easy questions and grade on a generous curve. SB 1047 is a small step in the right direction afaict.
Comment #26 September 4th, 2024 at 5:57 pm
Since, as always, I believe that catastrophic AI risk is exactly zero (not nearly zero, actually zero), I oppose this law. I hope that should it pass, it will be declared unconstitutional as a prior restraint on speech, which LLM AI currently is.
You are catastrophically wrong in your opinion about AI danger; how would you like it if you had to register your blog with the government before being allowed to write your opinions? (I kid. A little. Maybe. Slippery slopes.)
Comment #27 September 4th, 2024 at 7:47 pm
Hyman Rosen #26: The probability of a catastrophe involving radioactive dinosaurs from Neptune is not exactly zero. And again, we’re not talking here about AI destroying the world, but merely about an AI causing some harm that someone wants to sue over, for which the probability is close to 100%. You’re not a serious participant in this conversation. But have a great evening! 😀
Comment #28 September 4th, 2024 at 8:14 pm
Thank you Scott. We don’t agree on many things but you have always struck me as reasonable and good-hearted…
Thanks!! But out of genuine curiosity, what are some of the many things we don’t agree about? Just that your p(doom) is probably higher than mine?
You of course showed great moral courage in refusing that non-disparagement agreement, for which everyone concerned with AI safety will be forever grateful.
Comment #29 September 4th, 2024 at 8:42 pm
> because they’re hardcore libertarians who never want government to intervene in the market for any reason, not even if the literal fate of the planet was at stake
Do you prefer Donald Trump regulating AI to no regulations?
Or do you believe that the Deep State is real and will easily overpower Donald Trump?
Or are you a temporarily embarrassed chairman of the people’s party and in your heart of hearts believe that it will be you and your friends doing the regulating, not Donald Trump?
Comment #30 September 4th, 2024 at 8:48 pm
Name Required #29: I try to judge every piece of proposed legislation on its individual merits. This one was proposed by Scott Wiener, after (as I said) lots of consultation with AI experts. If, god help us, Trump proposes a different AI bill, I’ll take a look and tell you what I think of that one too.
Comment #31 September 4th, 2024 at 10:13 pm
Scott 27: I’m not really interested in talking about the probability of unprecedented events. Nor do I object to holding people liable for actual harm they cause; that harm, however, must be attributed to people who cause it, not to the tools they use. If AI writes bad software that a company incorporates into self-driving cars, the onus should be on that company. If the AI company made certain representations about the capability of their code generation that turned out not to be true, only then should that company be held responsible as well.
However, requiring pre-registration of software being used is a blatantly unconstitutional prior restraint, and it’s on that basis that I hope the law is overturned. And once again, I hope and expect that fully open-source AI will be available and worked on all over the world, completely out of reach of would-be gatekeepers.
Comment #32 September 4th, 2024 at 10:25 pm
Hyman Rosen #31: If someone dies on an amusement park ride, the amusement park might be found liable, or the ride operator, or the ride manufacturer, or any combination thereof. A court has to look at the actual evidence and see which parties were actually at fault. Why should it be any different if we’re talking about an AI product rather than a Ferris wheel? That’s why, once again, I see the liability provisions of SB 1047 as mostly just reaffirming the obvious.
Comment #33 September 4th, 2024 at 10:59 pm
Procedural question: why is it up to Governor Gavin Newsom whether or not SB 1047 becomes law? Just as with the U.S. Congress, the California State Congress can override a governor’s veto via a two-thirds vote in both houses (https://www.senate.ca.gov/citizens-guide/legislative-process). 75% of the State Assembly and 97% of the State Senate voted to pass the bill. So even if Newsom vetoes it, wouldn’t the State Congress presumably override his veto and pass the bill into law anyway?
Comment #34 September 5th, 2024 at 3:42 am
For me this situation is a reminder that big private capital does not really care about public good, they care about their position in the food chain above anything else, AI is just a tool for that. Call me a commie, if you wish, but it is what it is.
The world is heading towards exponential wealth disparity where you’d have to take the log of someone’s wealth to understand to which caste the person belongs. Obviously, this situation is highly unstable, which will lead to transformation and division of societies, countries, and then to revolutions and wars. It’s not a prophecy, it’s just pure math and logic.
Comment #35 September 5th, 2024 at 5:55 am
Aaron Roth #9 brings up a very important issue of computational tractability. I agree with Malcolm #24 on which direction this points to in practice: big companies can simply refuse to release their weights and end up only having liability wrt their own fine-tuned models, which as mentioned they already have.
But for any entity that wants to compete with big tech as some sort of federated / licensed / open source alternative, the burden of this regulation is overwhelming. Yes, there’s a $100M threshold which makes it less urgent. But, just as an example, imagine some sort of public distributed computing project a la folding@home, which with the creative accounting of hostile lawyers can easily be claimed to surpass this threshold.
Regulation, especially of this kind, must look towards possible futures, not only at the present. On net, I think I’d still vote for this though.
A more urgent concern in my mind is the trend of companies claiming they’re not liable for things that AI agents do on their behalf. We’ve already had legal battles about this and I predict many more. Also silly publicity stunts such as “electing” an AI as a board member. What should we call this phenomenon where a human presses a button that causes a machine to do X, but disclaims responsibility for X? tech-washing?
Comment #36 September 5th, 2024 at 9:25 am
chatgpt4president #3:
And all you are is an aggregate of quantum fields, what’s your point? Are you saying text, images or voice can’t be dangerous or do damage?
They could easily be used to create computer viruses, worms and phishing systems that cause considerable economic damage. This has already been done by white hats successfully in fact.
Comment #37 September 5th, 2024 at 9:59 am
Andrew Ng is of the opinion that what should be regulated are applications, not models.
https://time.com/collection/time100-voices/7016134/california-sb-1047-ai/
I think this is worth considering, would be interested to hear your take on it.
Comment #38 September 5th, 2024 at 1:10 pm
Andy Weinstein #37: If I’ve disagreed with my former employer, I suppose I can also disagree with my former “big brother” in the Berkeley CS PhD program! 😀
For me the crux of the matter is this: do we think it’s plausible that AIs can be created that, regardless of the creators’ intentions, could easily be modified to help just about anyone start pandemics, build chemical weapons, or hack into critical infrastructure? Then the dissemination of those AIs will presumably need to be restricted in an application-independent way … just like we restrict nuclear bombs, even though nuclear bombs would’ve also had many peaceful uses such as digging canals, powering turbines, and propelling spacecraft (a whole branch of technological history that, for better or worse, humanity willfully closed off with the test ban treaty in 1963). With a sufficiently dangerous technology, once any random person gets to have it and decide which application to use it for, you’re already too late to regulate anything.
Comment #39 September 5th, 2024 at 1:14 pm
@Scott, Nick Moran raises a great point in #21. I would like to know your opinion on open-sourcing frontier AI models. The bill seems to hurt the players that are trying to open source their models the most. This situation seems very analogous to regulating nuclear technology in order to curb nuclear arms, and thus hampering progress on nuclear energy. Now nuclear arms are an absolutely real threat, and in this scenario, the collateral damage is not being able to act swiftly on climate change.
Although it could be debated whether this collateral is worth the regulation, nuclear arms are a much more considerable and likely threat than AI doom or even simple AI harm. On the other hand, frontier AI models not being open sourced is a greater collateral damage than even nuclear energy in the long run, at least in my mind. (This may be a crude analogy, but it seems to have good mileage).
Also, any harm AI could do seems to stem from just plain old information the model is trained on, just presented in an accessible way. Should this information be guarded from the public too? In general, knowledge and power are more likely to be corrupted when they are concentrated. There is too much nuance the bill does not seem to consider. I think it does harm by not discouraging large model training and ultimately discouraging open sourcing.
Comment #40 September 5th, 2024 at 1:29 pm
Scott, thanks for writing this. I hadn’t checked your blog in a while but was curious to read your thoughts on the current moment in AI as well as any thoughts on OpenAI that you cared to share. I was glad to see this and look forward to reading more.
Regarding the recommendation to contact Governor Newsom’s office, I personally won’t do that, since I’m not a resident of California. I don’t know about governors, but representatives’ and Senators’ offices tend not to pay too much attention to non-constituents (unless they have a lot of money or something else they want, of course, which I don’t).
Comment #41 September 5th, 2024 at 1:35 pm
@Scott #38, such AIs could be created, but they would not be more helpful than the information those AIs are trained on, especially LLMs on the current deep learning paradigm. A malicious actor should find the online information to be sufficient. The real question is, is that information already public? If not, then the AI developers should not be allowed to train on it (where did they get this information from in the first place?). If yes, then why isn’t it already dangerous for us, LLMs or not?
Also –
1) danger and likeliness of: nuclear arms race >> AI doom/harm,
2) Positive impact of: nuclear tech < open-sourcing frontier AIs (in my view)
Comment #42 September 5th, 2024 at 4:00 pm
Abhishu Oza #41: How do you know that the AI wouldn’t be any more dangerous than the information it was trained on? What gives you the slightest confidence in that? Imagine, for example, an AI that could walk you through the whole process of building a chemical weapon, troubleshooting any problems along the way. Or that you could let loose to find a way to hack a government server. That seems obviously more useful than merely being handed a stack of chemistry or web development textbooks that “in principle” contain all the relevant information.
Comment #43 September 5th, 2024 at 10:50 pm
> Nancy Pelosi and Zoe Lofgren have also come out against it for whatever reasons
Pelosi opposes the bill because Wiener, who authored it, will be running against her daughter. A win for him is bad for her.
Comment #44 September 6th, 2024 at 1:43 am
I think Seth Finkelstein might have been onto something. But the legal difficulty at issue is not the (negligable in context) cost of the defendant establishing that the AI cost less than 100M to make, but the cost of the plaintiff establishing that the damage caused by the AI qualifies as “catastrophic”. Most plaintiffs would be unable to afford this cost, unless the damage (and hence, presumably, the company making the AI) is very high indeed. So we should expect most AI producers to support the bill, with the exception of the very largest companies who can easily. So the purpose of the bill is not regulatory capture, but shielding companies from lawsuits. (Even if the plaintiff doesn’t sue under this law, the courts might use it as guideline in the actual suit.)
Comment #45 September 6th, 2024 at 3:06 am
@Scott #42: You’re right in that the LLM would speed up the process, but the crux of the danger stems from access to materials/information that are ubiquitous but could still be harmful (rise in fentanyl overdose deaths is a perfect example). In the very near future, we could have smaller models trainable by common folk that could still guide you just on the basis of the textbook information.
Also, the bill is encouraging creation of these smaller yet more capable models! This is one good side effect of the bill, but based on the harm enabling logic, it will make training and using capable models on the potentially dangerous data even more accessible (eg Llama 3.1 405B with no quantization cannot be inferenced on local hardware unless you have > $100K lying around)
Real safety will only come from restricting access to biological materials or chemicals and creating more robust critical infrastructure. The bill blames the AI and its developer when the model has helped guide someone create a chemical weapon. Should it not target the companies that have made harmful materials available to the public? Or the person putting out public information having been used to train that model?
The crux is that danger stems from access to harmful materials/information and AI is a tool to accelerate our capabilities. Any technology increases our ability to have an effect on the world. If the bill is concerned about safety, then blaming AI developers directly is barking up the wrong tree and is also a losing game. I concede that the bill could be useful for its intended purpose, but only temporarily and is just kicking the can down the road while having significant negative consequences, and in one sense exacerbates what it thinks it’s preventing.
The wording of the bill makes it seem like to me that it would blame the AI developer for helping someone create fentanyl easily. But people are still making fentanyl without AI, and its lead to tens of thousands of deaths just last year! The problem is what the bill’s trying to fight vs what it’s actually fighting.
Comment #46 September 6th, 2024 at 10:39 am
Here’s an argument against that I haven’t seen yet:
This bill would strongly incentive AI companies to make their AIs reliably and robustly *politically savvy*, even under fine-tuning, instead of making them safe. Once AI gets robustly politically savvy, not only is that a directly dangerous capability, but it also means that safety testing becomes impossible.
Specifically, I predict that if an AI stays politically savvy even under fine-tuning, detecting safety problems using mechanistic interpretability will not work.
At low level capabilities, political savviness is actually good because then the AI will not to do random destruction like make bio-weapons. At high capabilities though, the danger is that the political savviness let’s it take over countries, or to make bio-weapons without being caught.
Even under an “alignment is easy” view, it is clear that SB 1047 aligns the AI to political savviness instead of to corrigibility or friendliness.
Comment #47 September 6th, 2024 at 11:41 am
Christopher #46: Which provision in the law do you think would incentivize “political savvy”? Just the liability? As I understand it, the liability provision kicks in only if a negligently-mistrained model causes a catastrophe with huge financial losses, not if it utters some offensive words…
Comment #48 September 6th, 2024 at 1:16 pm
Scott #47: how I imagine it working in practice is that the AI companies will have some metric called “willingness to cause catastrophes” which they will try to keep at 0. They might even train directly on this metric, v.s. an adversary trying to maximize it (you need an intelligent adversary during training to make it robust).
Political savviness correlates very well with “willingness to cause catastrophes”, and in particular robust political savviness leads to robust avoidance of catastrophes.
I’d also argue that political savviness is a simpler concept and easier to learn than avoidance of an arbitrary list of catastrophes being tested. By political savviness I don’t mean learning our political system (ChatGPT already knows that), but rather the unified skill of knowing to obey those more powerful and cooperate with those equally powerful (notice how political savviness doesn’t affect how you interact with those less powerful). Notice how the list of catastrophes are only ones that 1st-world countries can’t defend against. As a side effect, it then combines this skill with its specific political knowledge to become “politically correct”.
Political savviness is a capability a general intelligence would learn eventually anyways, SB 1047 just encourages the AI to learn it even faster!
Or *maybe* the AI companies will satisfy the demands of SB 1047 by solving alignment instead. Is that how you expect, say, OpenAI to comply with SB 1047? Or do you expect them to train out the specific catastrophes listed in the law?
Comment #49 September 6th, 2024 at 1:40 pm
I suppose what I’m getting at is that the premise of SB 1047 seems to assume that LLMs will be “genius naive nerds”, and the law will make sure these nerds know not to invent viruses or nukes! Silly nerds don’t know any better!
But my experience so far with LLMs suggests that opposite; their social skills run faster than their technical skills by a large margin. And also they are sycophants and serial BS-ers. They are already “socially savvy”.
Unfortunately, just as with humans, it is hard to create rules against the savvy. It is hard to make a social rule that stops bullies, because bullies are socially savvy. It is hard to make a political rule that can stop a politician from misbehaving for the same reason!
Comment #50 September 6th, 2024 at 3:50 pm
This is an excellent discussion with a lot of good points being made and debated.
Only one obvious idiot has posted, although no doubt more will join the fray as soon as they catch wind of it.
Comment #51 September 7th, 2024 at 1:21 pm
Scott, I have a very specific technical question about watermarking, which arises from a recent article published in Vox that mentions you by name (the article also suggests that watermarking will become a standard requirement under SB 1047, hence its relevance to these comments). I want to be clear that I ask the following question not because I am opposed to watermarking, but because I feel that I am ignorant about watermarking, and genuinely want to understand it better – but because of that ignorance, this question may come across as disingenuous opposition, which is not my intent.
In short: Watermarking is a statistical process, so it cannot be perfectly infallible. When evaluating whether a watermark is present, what is the probability of type I error (false positives)?
In long: In the best possible case, assuming that this technology is a wild success, we can reasonably extrapolate that most college professors who give long-form writing assignments will then proceed to put those assignments through a watermark-detection algorithm. When you start to multiply the numbers out with basic napkin math, that adds up to a lot of uses of this technology per annum. Under these rosy assumptions, the type I error rate directly translates into the number of students who are falsely accused of plagiarism every year. Most of those students will struggle to defend themselves against a report backed by Math and Science, so the absolute number of false accusations really ought to be very low, if we’re to credibly claim that we’re making the world a better place overall. Approximately how many students would you estimate will be falsely accused every year?
My other question, which is a bit of a corollary to the first question: If watermarking becomes standard, then every major LLM provider will offer a watermarking-detection service, and college professors will presumably choose to use most or all of these services on every essay. Will these services apply a Bonferroni correction (or some other correction) automatically, or should we multiply the false positive rate by the expected number of providers?
I recognize that, in its current state, the technology is not freely available, and so the above numbers may not be possible to publish yet. But if that is indeed the case, then I do not feel that I can support SB 1047 in good conscience, since I do not understand what its long-term implications will look like in practice.
Comment #52 September 7th, 2024 at 6:20 pm
@Abhishu Oza #45:
From what I recall of the bill, the developers are only liable if the AI makes a plan to cause a catastrophe, not if it just collects general information about how to cause it. Of course my previous comment about legal costs of showing things are/are not covered would still apply.
Comment #53 September 7th, 2024 at 8:51 pm
On political savviness: it enables bad people to fool others as to their real incentives (or that is what I take the terms to imply). The incentives of AI programs are largely, if not totally, established by their programming. E.G., the primary incentive of an LLM is to maximize the probability of a response compared to its training data. There are other incentives, such as the “I am just an AI” disclaimer, and there may turn out to be conflicts between programmed incentives. Anyway, it seems to me that any inclination for an AI to use political savvy techniques to disguise bad motives will be the fault of its programmers, no doubt driven by evil CEO’s like Jack Welch, who can only be influenced by potential financial disincentives such as SB 1047 would provide.
Comment #54 September 7th, 2024 at 9:33 pm
From the TechCrunch link:
“At this point, California’s attorney general could request an injunctive order, requiring an AI company to stop training or operating their AI models if a court finds them to be dangerous.”
I worry that if we’d applied that rule to the Manhattan Project, the USA wouldn’t have developed atomic weapons/energy, or at least would not have developed them first. And I think the world would be worse off for that.
Are we not giving unsavory countries (i.e., China) a significant advantage? It seems like in a race to build AGI, getting there first may count for a lot.
Comment #55 September 7th, 2024 at 10:23 pm
The 100M provision only targets “big players” imo is an incorrect assessment. The open source ecosystem is heavily reliant on the big players releasing strong base models.
This bill, by not setting clear standards for what constitutes “reasonable efforts” or “catastrophic harm” will make the legal situation far more uncertain for any big player who wants to release a strong base model — because who knows how a jury might rule as to whether their efforts were reasonable.
Once the big players are deterred from releasing model weights, it doesn’t matter how much you claim that small players won’t be liable — they will be irrelevant as they won’t have strong base models to go off of!
Comment #56 September 8th, 2024 at 3:39 pm
> What matters is simply that, once you’ve granted the principle that people worried about AI-caused catastrophes get a seat at the table, any legislative acknowledgment of the validity of their concerns—then they’re going to take a mile rather than an inch, and kill the whole AI industry.
I perceive that the public would rather LLMs had never been invented, just as craftsmen turned laborers 1750-1900 legitimately, materially, and culturally wished the industrial revolution had not happened. AI art is widely perceived to be disturbing and in some way, “false,” and something in me ever so slightly recoils when LLMs generate (bad, for now) poetry. Everybody knows the benefits will accrue to capital, to the extent that robots do not need liable supervision. I think the will of the public, if we assume it to be guided by the middle class’s understanding of their own material interest, may in the fullness of time be to kill the whole AI industry or switch to a democratic socialism involving nationalization of industries on a scale associated with communist regimes.
Comment #57 September 8th, 2024 at 5:07 pm
Kevin #51: No, SB 1047 has nothing to do with watermarking. There’s a completely different California bill, AB 3211, that includes a watermarking mandate, originally including text but now they’ve apparently whittled it down to audiovisual content only.
Nevertheless, to answer your question about accuracy: as I’ve explained in my slides and blog posts about this, if the average entropy per token is h and you’re willing to tolerate a probability of misclassification per token of ε, then the number of tokens you need to see scales like ~1/h2 log(1/ε). In practice, you can set ε=0.001 (say) and make do with only a thousand or so tokens of normal text — something we can not only explain theoretically but also validate empirically. You can also trade off (eg) more false negatives in exchange for fewer false positives, by setting the threshold differently. This is the basis for the statement that you can easily push the rate of false positives (eg students falsely accused of cheating) below 0.1%. Of course the rate of false negatives (students who cheat undetectably) could be high if the students take active countermeasures, but otherwise it won’t be.
Comment #58 September 8th, 2024 at 8:58 pm
@Scott #7: That was a good post that changed my mind on the topic. Thanks for that. The key arguments I found compelling were 1) $100m is a high enough threshold to not stifle startups, and 2) since the thrust of the bill is liability, if people don’t believe the harms are possible, why oppose liability?
Good topic, and thanks.
Comment #59 September 9th, 2024 at 12:54 am
Scott #57
The false positive rate had better be very low given the enormous number of essays written every year!
How much of a full LLM do you need to render a watermark unreadable without destroying the meaning in the text? Did your group ever find out if independently perturbing each token in embedding space would work?
Comment #60 September 9th, 2024 at 8:24 am
Scott #57: I apologize for repeating a question that I’ve asked before (without answer), but how does the required number of tokens scale with the LLM’s temperature setting? If the key to the watermarking scheme is to sample from the token space pseudorandomly rather than according to the Gibbs/Boltzmann probability distribution, then it seems to me that the scheme wouldn’t work in the limit of zero temperature (where there’s no randomness at all in the sampling probability distribution, even if there is in the predicted distribution for the next token)? Does the entropy per token h implicitly depend on the temperature setting?
Comment #61 September 9th, 2024 at 9:40 am
Ted #60: Yes, the entropy per token certainly depends on the temperature. It’s the entropy of whatever probability distribution is actually being sampled from … but as the temperature approaches 0, that distribution becomes more and more deterministic. When the temperature is literally 0, any watermarking scheme like mine will stop working (the number of tokens required having diverged to infinity). So, if you want to allow people to generate completions at zero temperature, and have them be watermarked, you need to switch to a different watermarking scheme, for example one that adjusts the (non-temperature-corrected) probabilities. Those schemes might have a detectable effect on the output, but that’s an inherent tradeoff here. Undetectable watermarking, by its nature, is something that takes advantage of entropy in an AI’s output distribution.
Comment #62 September 10th, 2024 at 12:33 am
0.1% false positive rate is extremely high
over 300 assignments the probability of a false accusation is around 25%, way too high for something that potentially destroys an academic career
Comment #63 September 10th, 2024 at 12:48 am
Concerned #59, remember what Scott’s watermarking scheme is competing against: Not professors assuming essays are human-authored, but schools using terrible, ad-hoc tools with incredibly high false positive rates to accuse students of AI authorship.
To get the error rate even lower, for institutions that find a marginal probability decibel worth a marginal dollar, one can imagine many additional tests that could be applied after a watermarking positive.
It’s the same marginal thinking that made me support SB 1047: It’s a bill well-tuned to have very low costs in the cases where it benefits nobody. I still expect ASI to kill everyone, but in the meantime, this bill could prevent harm at smaller scales.
Comment #64 September 10th, 2024 at 11:44 am
For those that do want to tweet, call, or email Newsom in support of the bill, this link gives a few specific ways to do that: https://safesecureai.org/get-involved
Having worked in policy and politics for years, I can say that these messages really do matter, and folks who want to make their voices heard have only a couple weeks before Newsom has to decide whether to sign or veto it.
Comment #65 September 10th, 2024 at 12:43 pm
[…] Scott Aaronson in strong support of SB 1047. Good arguments. […]
Comment #66 September 11th, 2024 at 7:41 am
chatgpt4president wrote in Comment #3: ALL they can do is generate images and text on request. That is literally impossible to be “dangerous,” …
Well, see for example in Vol. 17 No. 2 issue (Summer 2024) of Fungi magazine:
http://fungimag.com/v17i2/V17I2-summer-2024.htm
the article “Mushrooming Risk: Unreliable AI Tools Generate Mushroom Misinformation” by Rick Claypool.
Its key takeaways:
“Emerging AI technologies are being deployed to help beginner foragers identify edible wild mushrooms. Distinguishing edible mushrooms from toxic mushrooms in the wild is a high-risk activity that
requires real-world skills that current AI systems cannot reliably emulate.
Individuals relying solely on AI technology for mushroom identification have been severely sickened and hospitalized after consuming wild mushrooms that AI systems misidentified as edible.
Amazon’s online marketplace was inundated in 2023 with reportedly AI-generated books, leading the company to limit the number of books any individual can self-publish per day.
Generative AI technologies that use OpenAI’s ChatGPT and DALL-E systems are being used to develop mushroom identification chatbots capable of producing confusing and dangerous misinformation that could result in severe poisonings and death.
To protect users and prevent the spread of harmful misinformation, the businesses behind these technologies must accept the responsibility to disclose the use of AI and the responsibility to remind users constantly that AI makes mistakes. When AI systems sold as sources of truthful information instead produce false and deceptive content resulting in users making harmful decisions, businesses must be liable for the harms they cause.”
…
So, certainly not just “harmless pixels on a screen”. And read the last sentence above.
Comment #67 September 19th, 2024 at 11:58 pm
I am not sure if i am convinced by your argument. The problem is that it is not AI models PER SE that are dangerous. It is attaching them to dams and robots and cars.
The danger is the attaching part. And moreover generally speaking AI models are created unscoped. Chatgpt has no specific purpose. I couldn’t even evaluate what its safety issues are without the attaching it part. And there is the rub as far as I am concerned.
This law makes responsible the party that can’t possibly take responsibility. So it seems to be pointless paperwork.
Perhaps you could give an example of exactly what a person training a 100 dollar million model to compete with chatgpt should say regarding safety that is not mere pointless cliches and vague concerns?
Is this law perhaps not a means to entrench incumbents with no other discernable benefit?
Comment #68 September 27th, 2024 at 1:33 pm
@Scott you asked “what are some of the many things we don’t agree about? Just that your p(doom) is probably higher than mine?”
Maybe I should have clarified that we also agree about many things, and overall we probably agree more than we disagree.
Yeah I was thinking about things like timelines to AGI, ASI, etc. and p(doom), and the cruxes/premises that underlie those disagreements. My understanding is that you don’t think we’ll have superintelligence ( == better than the best humans at every cognitive task, while also being cheaper and faster) by the end of the decade, and certainly not by the end of 2027, whereas I think 50% chance by end of 2027.
Thanks for the kind words. 🙂
Comment #69 October 1st, 2024 at 12:06 pm
[…] Many of you will have seen the news that Governor Gavin Newsom has vetoed SB 1047, the groundbreaking AI safety bill that passed the California legislature. Newsom gave a disingenuous explanation (which no one on either side of the debate took seriously), that he vetoed the bill only because it didn’t go far enough (!!) in regulating the misuses of small models. While sad, this doesn’t come as a huge shock, as Newsom had given clear prior indications that he was likely to veto the bill, and many observers had warned to expect him to do whatever he thought would most further his political ambitions and/or satisfy his most powerful lobbyists. In any case, I’m reluctantly forced to the conclusion that either Governor Newsom doesn’t read Shtetl-Optimized, or else he somehow wasn’t persuaded by my post last month in support of SB 1047. […]
Comment #70 October 31st, 2025 at 11:15 pm
[…] position that causes him to get heckled as a genocidal Zionist, and authored the excellent SB1047 AI safety bill, which Gavin Newsom unfortunately vetoed for short-term political reasons. I donated […]
Comment #71 January 8th, 2026 at 11:18 pm
Really fascinating discussion on SB 1047! I particularly resonated with your response in comment #38 regarding the potential for AI to be modified for harmful purposes in ways that simple information retrieval cannot. The analogy to nuclear technology is a sobering one, especially the idea that dissemination itself might need restriction regardless of intent. I wonder if we’ll see more state-level efforts now that Newsom has vetoed this particular bill, or if the focus will shift back to federal regulation despite the hurdles. A small suggestion: a future post exploring the intersection of AI safety and open-source model weights would be incredibly insightful given the current debate.
For those interested in how these concepts are visualized, you might find this useful: https://www.nanobananaimages.com/
And for more on AI tools and their implications, check out https://hunyuan3dai.com/ or https://www.triposrai.com/
Comment #72 January 8th, 2026 at 11:51 pm
The discussion around SB 1047 is truly complex, and your defense of it as a reasonable step given the federal deadlock is quite persuasive. It’s interesting how even a mild bill can become such a flashpoint for broader ideological battles over AI’s future. One minor suggestion for future posts: it might be valuable to explore how these regulatory frameworks could impact smaller open-source contributors differently than the giants. Really appreciated this deep dive into the nuances of the bill! https://www.nanobananaimages.com/