Updates!
No, I don’t know what happened with Sam Altman, beyond what’s being reported all over the world’s press, which I’ve been reading along with everyone else. Ilya Sutskever does know, and I talk to Ilya nearly every week. But I know Ilya well enough to know that whatever he’d tell me about this, he’d also tell the world. It feels weird to be so close to the biggest news story on the planet, and yet at the same time so far from it. My current contract with OpenAI is set to expire this summer. Until then, and afterwards, I remain just as interested in figuring out what theoretical computer science can contribute to AI safety as I was yesterday morning.
My friend, theoretical computer science colleague, and now OpenAI colleague Boaz Barak has coauthored a paper giving a general class to attack against watermarking methods for large language models—100% consistent with the kinds of attacks we already knew about and were resigned to, but still good to spell out at a formal level. I hope to write more about it in the future.
Here’s a recent interview with me in Politico, touching on quantum computing, AI, and more.
And if that’s not enough of me, here’s a recent podcast that I did with Theo Jaffee, touching on quantum computing, P vs. NP, AI alignment, David Deutsch, and Twitter.
Whatever feelings anyone has about it, the new University of Austin (not to be confused with the University of Texas at Austin, where I work) is officially launching. And they’re hiring! People who are interested in STEM positions there should contact David Ruth.
I forgot to link to it when it came out more than a month ago—a lot has happened in the meantime!—but Dalzell et al. put up a phenomenal 337-page survey of quantum algorithms, focusing relentlessly on the crucial question of whether there’s actually an end-to-end speedup over the best known classical algorithm for each given task. In countless situations where I would just scream “no, the hypesters are lying to you, this is BS,” Dalzell et al. take dozens of polite, careful, and highly technical pages to spell out why.
Besides AI intrigue, this past week might be remembered for a major breakthrough in classical complexity theory, in solving arbitrary compression problems via a nonuniform algorithm (i.e., a family of Boolean circuits) that takes only 24n/5 time, rather than the 2n time that would be needed for brute force. See this paper by Hirahara, Ilango, and Williams, and as well this independent one by Mazor and Pass.
Follow
Comment #1 November 18th, 2023 at 2:04 pm
As an outsider looking in my hope is that Ilya will pivot the company back to its open roots. OpenAI should adopt radical transparency as the best method to ensure AI safety. Security through obscurity has never worked and will never work. Also hoping that you get to continue working on formal problems to prove safety in this same vein for as long as you prefer.
Comment #2 November 18th, 2023 at 3:52 pm
I hope it is not some mundane reasons (like exaggerating travel costs or not filling timesheet) and worth a story 😉
Comment #3 November 18th, 2023 at 3:56 pm
Most likely succumbed to an excess of deniable plausibility …
Comment #4 November 18th, 2023 at 5:07 pm
Zen #2: While the details are only now emerging in the tech press, I think we can say with confidence that no, this wasn’t about Sam’s travel reimbursements. 😀
Comment #5 November 18th, 2023 at 5:43 pm
Scott Comment #4:
Luckily I was wrong! It is some board room drama, reports NYT. In any case, Mr.Altman was right that AI could cost people their jobs.
I think I’ll take a weekend to understand company’s board structure. If you don’t mind, I’m donating the article here for public consumption.
https://www.nytimes.com/2023/11/18/technology/open-ai-sam-altman-what-happened.html?unlocked_article_code=1._Uw.ApLg.bVlIbKOG8Gnm&smid=url-share
Comment #6 November 18th, 2023 at 9:12 pm
Ok, while I am an anonymous coward and also a nobody, I am going to go out on a limb and speculate that this all happened due to some disagreement with Ilya Sutskever. I have seen Ilya talk several times and I have always had the impression that he is a fool, dressed up as an AI expert because he was lucky enough to be in the right place at the right time (famous advisor, and access to lots of data and lots of compute). But he seems to be an extremely self-assured technical and philosophical zero. I have the opposite impression of Scott (who I believe to be a real genius). So if this post is accepted (and I understand if it is not), then perhaps Scott could explain why is it the case that Ilya’s talks and his seeming ability to understand relatively simple technical ideas and questions is so bad. Anyway, if I am right, perhaps his minimal thoughtfulness and maximal self-assuredness convinced the rest of the very unusual (and perhaps somewhat weak) board to oust Sam. Perhaps what will soon happen is Sam will be back and Ilya will be out. Perhaps also I am dead wrong, and perhaps none of this matters at all since no one will see this post.
Comment #7 November 18th, 2023 at 9:25 pm
Nobody Important #6: Ilya is very far from a fool. Not only am I confident about that, I’m confident that Sam would agree as well.
Comment #8 November 18th, 2023 at 11:04 pm
In the interview in Politico I see you mention “proof-checking software” as an “underrated big idea” (for example to reduce “hallucinations”). This is similar to an idea proposed by Steve Omohundro and Max Tegmark:
https://arxiv.org/abs/2309.01933
I could see it working very well in math and many of the sciences (once a formal verification language is settled on); however, I think it will work less well in messy everyday domains (though people have tried over the years…).
On the subject of “hallucinations” or “confabulations”, studying how humans make so few of them might offer clues. e.g. in this old House episode you see what happens with a specific kind of temporary brain impairment (Korsakoff’s Syndrome) that includes loss of long-term memory access:
The woman confabulates like crazy (like a language model), picking up any clue in the environment to weave into her narrative about who she is. I’m not sure that ordinary amnesia would cause that; but perhaps it does, too, without additional brain impairments. Anyways, maybe our good long-term memory is responsible for much of our ability to combat confabulations.
Humans also have self-awareness which helps with hallucinations because e.g. if you know that your memory is kind of shoddy, you’d not trust every recollection that pops into your head, and would verify everything very carefully. You’d also avoid speaking in specifics unless you had them on a piece of paper right in front of you.
I think humans also have good “representations of knowledge” that help cut down on the amount of “reasoning” or processing that needs to be done to avoid making errors (and confabulating). Perhaps it just boils down to having many independent ways of looking at things, and having all these different ways tightly-integrated somehow.
A metaphor for this: imagine you have to solve a knapsack problem, and are given a list of 10,000 very big numbers, and are asked to determine whether there is a subset of them that sums to a target T. The problem is likely intractable.
But now let’s say that in place of 10,000 big numbers you have 10,000 independent vectors of dimension 10,000, and let’s say T is now a 10,000-dimensional vector (instead of a number). Then, deciding whether whether there is a subset of your list that sums to T is trivial: use Gaussian elimination to find the linear combination adding to T, and check that all the coefficients in your linear combination are 0 or 1.
Think of the different coordinates as giving you “independent information” for ways to think about the items in your list. With it, your “reasoning problem” is trivial; but without it (or if you are stuck with just a list of numbers), it’s probably intractable.
Comment #9 November 19th, 2023 at 12:15 am
We are faced with people who are extremely technically competent but have an underlying Luddite-like distrust of technology and Marx-like distrust of capitalism. In the long run they do not represent a good survival/advancement path for mankind and hopefully will not be widely successful in their efforts in the short nor medium terms.
Comment #10 November 19th, 2023 at 4:19 am
So it looks like a fight between the AI risk people and the AI opportunity people, with the risk people having won a brief victory and now facing backlash. As I have said before, I do not believe in AI risk ideology, so I will be happy if those folks lose their battles.
My view is that you don’t create an industry to worry about and fix things that haven’t happened yet. That industry will be full of navel gazers and grifters because the actual risks that will eventually arise will be unanticipated ones – forecasting the future is hard. The AI doomers are completely off the wall. Fortunately, open-source AI projects are already being built, and more will come, in every country in the world, so that would-be gatekeepers will fail to hold back this technology or censor what it produces.
Comment #11 November 19th, 2023 at 4:55 am
If there’s no end to end speedup ( which I agree that there isn’t ) what exactly has changed over the last 10 years.
I still remember those arguments about whether DWave had a real speedup and after that all the strong of other experiments to try to demonstrate a speedup.
Should we keep chasing the illusion?
Comment #12 November 19th, 2023 at 7:17 am
Rahul #11: There are real, end-to-end exponential speedups over the best known classical algorithms for quantum simulation (potentially very important economically and a central reason to build QCs) and, of course, for breaking current public-key crypto. There are other exponential quantum speedups that might or might not be important in end-to-end applications, and algorithms that might or might not yield exponential speedups. There are more modest (polynomial) end-to-end speedups for countless other problems, in optimization, estimation, and so on.
This fundamental picture, I would say, hasn’t budged since it first emerged in the mid-1990s, but it can now be fleshed out in so much detail that even surveying what’s known takes 300+ pages.
Comment #13 November 19th, 2023 at 7:40 am
@hyman Rosen
Regarding AI risks you call them “things that have not happened yet”
I disagree. Just take deep fakes. The whole edifice of democracy is based on an effective, reliable media and we are preety much at the point where I cannot even trust a video, let alone a photo, to be a depiction of the truth.
The potential for damage is enormous. A deep fake video could incide riots, kill millions and destabilize entire nations.
I don’t think it is premature to look for solutions. I don’t see any good answers.
Clearly AI will lead to huge productivity improvements but let’s not pretend that the problems aren’t visible yet. In some sense I think people are too obsessed with remote, existential doomsday scenarios of AI running rouge while the real danger is more mundane and immediate.
Comment #14 November 19th, 2023 at 8:34 am
I completely forgot you were at OpenAI when I opened up the blog yesterday. Must feel kind of crazy to be in the middle of this.
Scott, if you can share them, how have your views on AI Safety changed since being at OpenAI? I imagine at least “hallway conversations” have occurred which may have shifted your views. Have you talked about this anywhere?
Comment #15 November 19th, 2023 at 8:43 am
Hyman Rosen #10,
I think that is the incorrect way of looking at this. The fight was between a capitalist/salesman disguising himself as an AI risk idealogue to gain competitive advantage versus an actual engineer/scientist who has sincere fears for the usage his creation might be put to. In that fight, I’ll take the more honest person every time.
I too am highly skeptical of the near term fears promulgated by many in the AI risk crowd – and even to the extent I buy into them I think radical transparency with proven methods is the only way to combat – but I can at least respect a knowledgeable person with views different than mine. What I can’t respect is a greedy salesman hiding behind FUD to gain competitive advantage through regulatory capture.
I’m an outsider in all of this, but that’s definitely how it appears to me.
Comment #16 November 19th, 2023 at 9:24 am
Why is nobody considering the possibility that the board had so much confidence in ChatGPT that they decided to be trailblazers and replace Altman with ChatGPT? The board should be commended for their bravery and for pushing the boundaries!
Comment #17 November 19th, 2023 at 9:54 am
A few months back, you gave a talk at the Technion, where among other things you claimed that there’s a known quantum speedup for condensed matter problems. I approached you after the talk, trying to understand precisely what you meant by that, since to the best of my knowledge (then and now) as a condensed matter physicist, no such speedup is known. As far as I can tell we’ve completely failed to communicate at the time. Did you read Dalzell’s Fermi-Hubbard model section? Credit to them, they admit it more or less openly.
Comment #18 November 19th, 2023 at 10:14 am
Vladimir #17: As I surely would’ve told you at the time, this is just a question about the definition of condensed matter problem. There are condensed matter problems that are BQP-complete, meaning that there’s an exponential quantum speed up for them if there is for anything. Whether there are practically relevant condensed matter problems that admit exponential quantum speedup is a different question.
Comment #19 November 19th, 2023 at 10:33 am
Scott #12, 18:
Dalzell et al. point out that
> Quantum simulation of Fermi–Hubbard models, based on the current estimates, requires considerably fewer resources than simulations of molecules or solving optimization problems
They also admit (in the caveats subsection) that the existence of an exponential speedup for the Fermi-Hubbard model is still an open question. So if you could clarify what you meant by
> There are real, end-to-end exponential speedups over the best known classical algorithms for quantum simulation (potentially very important economically and a central reason to build QCs)
… I’d be very grateful.
Comment #20 November 19th, 2023 at 10:35 am
“the biggest news story on the planet”
uhhh… okay.
Comment #21 November 19th, 2023 at 10:50 am
Rahul #13: The answer to deep fakes is the same as the answer to written or spoken lies. You must find trustworthy sources from which to get your information. It did not take clever AI lies to cause the slaughter of tens of millions of people in the 20th century. On the other hand, “nuclear waste risk” was used by anti-nuclear activists to deliberately shut down the possibility of nuclear power in America.
Comment #22 November 19th, 2023 at 11:00 am
fred #20: I mean, it was the top headline for a while on some website called nytimes.com.
Comment #23 November 19th, 2023 at 11:08 am
Vladimir #19: Page 19 of the very survey we’re talking about affirms that there appears to be an exponential quantum speedup for simulating Hamiltonian spin dynamics. And this is obvious, again because simulating Hamiltonian spin dynamics is BQP-complete—it can encode all of quantum computation. You’re the one who needs to give a reason why there wouldn’t be an exponential speedup for that generic task. If you can’t understand that, then I’m afraid that I don’t have time and this conversation is at an end.
Comment #24 November 19th, 2023 at 11:17 am
Hyman Rosen #21
Yes. Critical thought and questioning what you see and hear from other people has been replaced for many by believe whatever supports your ideology. Videos are just one more type of input that must be questioned by a rational person. An AI may ultimately be a better propagandist than humans but the standard set by humans without AI assistance is not insignificant. It will take a lot of cycles for AI to best the standard.
Comment #25 November 19th, 2023 at 12:42 pm
Scott #23:
I think the root cause of our failure to communicate is that we mean very different things by “quantum simulation”.
The problem you seem to have in mind is “given a Hamiltonian and an initial state, simulate its time dynamics”; I fully agree that it’s easy for a quantum computer, hard for the best known classical algorithms, and that this situation is unlikely to ever change. However, solving this problem does not in itself have any economic implications that I’m aware of, nor is it interesting from a purely scientific condensed matter perspective (*).
What I, and Dalzell et al. in the sentence I quoted, mean by “quantum simulation” is “given a Hamiltonian, find its ground state”. Solving *this* problem could have huge economic implications, possibly in condensed matter, certainly in quantum chemistry. However, there is no obvious way to recast this problem as a problem of the type you have in mind.
The best currently known way, which Dalzell et al. use for their calculations, assumes that one can easily construct an initial state whose overlap with the ground state decays at most polynomially with system size.
It is the validity of this assumption that Dalzell et al. refer to as an open question in the caveats subsection, and I applaud them for their honesty, but if they were more honest still they would say something about the likelihood of this assumption turning out to hold. If you have an opinion about it (you might have some intuition based on the numerical experiments you presented at that Technion talk), I’d be interested to hear it. My own opinion, which I think most condensed matter theorists will agree with, is… well, I won’t say it’s about as likely as P=NP, but I don’t have a better analogy.
(*) With the possible exception of studying many-body localization.
Comment #26 November 19th, 2023 at 3:47 pm
Vladimir #25: I know perfectly well the distinction between simulating time dynamics (a BQP-complete problem) and estimating ground state energy (a QMA-complete problem, so presumably intractable in the worst case even for a quantum computer, but sometimes easy in practice using techniques like Quantum Monte Carlo). I agree with you and with the survey that the question of exponential quantum speedups for the latter is much more complicated and uncertain. But
(1) the chemists certainly care about reaction rates and many other dynamical questions, and
(2) are you serious that condensed-matter people don’t care at all about dynamics? have they just not focused much on dynamical questions because they’ve been computationally intractable?
Comment #27 November 19th, 2023 at 4:18 pm
Scott #26
(1) Sure, but if you want to calculate e.g. two molecules’ reaction rate, you need the molecules’ ground states as the initial conditions.
(2) The basic issue is that unitary time evolution of an interacting many-body quantum system is not a good description of any real-world phenomena (as evidenced, e.g., by how hard quantum engineering people have to work to maintain coherence for a fraction of a second). And even ignoring that, you’d run into the same problem as in (1), namely that the states you might be interested in time-evolving are difficult to construct in the first place.
Comment #28 November 19th, 2023 at 7:06 pm
I am not a fan of Lex Fridman, but I got a lot more out of his interview with Ilya Sutskever than out of his interview with Sam Altman. While Ilya might perhaps appear to lack Sam’s polish, I was immensely impressed by the depth and rigour of his responses, combined with a touch of genuine humility (albeit perhaps only inadvertantly betrayed).
Comment #29 November 19th, 2023 at 7:39 pm
I am pretty excited about the startup university you mentioned here, The Univ of Austin at TX. I hope it is the beginning of competition for established universities. It seems serious and committed to a rigorous education and merit. Here’s wishing it the best.
Comment #30 November 19th, 2023 at 9:35 pm
The exchange between Scott and Vladimir smells science and that attracts science-thirsty beasts like me, and that was for understandable reasons lacking here recently. More of it, so that we can devour!
Regarding the chemist’s part: AFAIK many molecules have reasonably known low-energy states, and the real world has a finite temperature, thus it is not about completely ground states.
A common task is to find a catalyst of a chemical reaction where you do not know the structure of the catalyst in advance. Can QC help here?
Regarding CMP, could QC help in finding better batteries, like that solid-state battery claimed by Toyota?
Comment #31 November 19th, 2023 at 10:16 pm
Scott, a naive technical question about your watermarking proposal: In your interview, you say that the scheme relies crucially on the stochastic nature of LLMs like GPT, by replacing an (in principle) true random number generator with a carefully chosen pseudorandom number generator.
To my understanding, GPT is stochastic at finite temperature parameter, but is deterministic at zero temperature parameter. Does that mean that your proposal doesn’t work at zero temperature? Is there perhaps some temperature scaling behavior where the expected amount of text required to detect a forgery increases at lower temperature and diverges as T -> 0?
Comment #32 November 20th, 2023 at 7:09 am
RD #29
Have they released their pronoun list?
Comment #33 November 20th, 2023 at 7:29 am
The rise of the new university is a-Break Glass In Case of Emergency-for US higher education.
Comment #34 November 20th, 2023 at 7:44 am
Vladimir #27:
Suppose you have a lump of material from which you wish to extract its ground state, which, as Scott has indicated and which you appear to agree with, can generally be very difficult even for a fully fault-tolerant quantum computer.
But if even a quantum computer would take longer than the age of the universe to calculate the ground state, then surely Nature herself would never settle into said ground state before the heat death of the universe. How does knowing this ground-state, that’s inaccessible even for Nature herself, act as a good description of any real-world phenomena?
Certainly owing to the BQP-completeness of Hamiltonian simulation and of eigenvalue sampling from various guiding states, a quantum computer would have an advantage over a classical computer in estimating certain the energy of certain low energy (and not necessarily ground) states – such states are presumably accessible by Nature herself.
Comment #35 November 20th, 2023 at 9:42 am
Regardless of the particular details, the OpenAI unraveling/chaos/volatility we’re witnessing shows there’s more to AI *safety* than engineering, it’s also whether the development of such powerful technology (with huge effects on national security) should be *fully* entrusted to unsupervised tech startups.
Comment #36 November 20th, 2023 at 9:52 am
It was a very strange, “bipolar” weekend with the situation at OpenAI. I’d really like to know what was behind the board’s decision to fire Sam Altman, and why they didn’t resign and let him back in control of the company. The best theory I’ve heard is that they just had different values from most of the rest of the company. Sam (and most of the company) wanted to scale quickly, and perhaps the board didn’t want that — I don’t know.
Another theory I read online (in a private forum) was this (paraphrasing): some team at OpenAI had trained an AI model that was only meant to be an incremental improvement over GPT4, without changing the architecture of anything, but ended up being far more capable than they had predicted; and it could end up being a successor model to GPT4. e.g. it might have shown incredible improvements over GPT4 in terms of ability to reliably execute a given task to completion while using external tools, perhaps using self-refinement and reflexion.
If that is true, then it sounds like they hit a “phase transition” like I had mentioned in a previous comment on this blog.
Sam did say in an interview recently that only just a few weeks ago he got to witness the fourth time in the company’s history when the veil of ignorance was pushed back (maybe the first through third times being the arrival of GPT2, GPT3, and GPT4).
But if all this is true, I don’t see how Sam figures in to the story, unless he did something like have it quietly added (without them knowing) for testing to a few lucky people’s ChatGPT+ subscriptions. I would doubt he — or any CEO — would do that, though.
Another theory is that this was about Sam’s deals to make OpenAI into a competitor with Nvdia and others, and perhaps also the discussions with Johnny Ive about a new AI smartphone-like device. (This would also be part of the too-far-too-fast theory.) If this theory is correct, I can’t imagine why it would induce the board to get rid of him.
Anyways, as I said, I would really like to know what was behind all the drama…
Comment #37 November 20th, 2023 at 11:53 am
starspawn0 #36
Thank you for sharing these comments. The possibility that the darkness of ignorance has been further reduced is good news indeed.
Comment #38 November 20th, 2023 at 11:54 am
Mark Spinelli #34
> But if even a quantum computer would take longer than the age of the universe to calculate the ground state, then surely Nature herself would never settle into said ground state before the heat death of the universe. How does knowing this ground-state, that’s inaccessible even for Nature herself, act as a good description of any real-world phenomena?
Nature manages to find the ground state, or thermal states, of lumps of matter in finite time (indeed, very quickly) because no lump of matter evolves unitarily, isolated from the rest of the universe. Consider a hydrogen atom prepared in an excited (eigen)state. If it were alone in the universe, it would stay in that state forever. In the real world, it’s coupled to the electromagnetic field, so the state we prepared it in is not a true eigenstate of the atom + field system, and it has nontrivial time evolution. In principle we could fully track this evolution, such that at all times the state of the system would be some superposition of the initial state and the ground state + a photon (other intermediate states could also enter into the mix, of course), but saying “the atom decayed to the ground state and emitted a photon” is a much more useful description.
> Certainly owing to the BQP-completeness of Hamiltonian simulation and of eigenvalue sampling from various guiding states, a quantum computer would have an advantage over a classical computer in estimating certain the energy of certain low energy (and not necessarily ground) states – such states are presumably accessible by Nature herself.
You seem to be under the impression that the ground state of a quantum system is harder to prepare than a specific low-energy excited eigenstate. The opposite is true, because a) generically, the ground state is the least-entangled low-energy eigenstate, and b) we have a systematic way of “targeting” the ground state, namely the variational principle; we don’t have anything analogous for other eigenstates.
Just to clarify, I’m not saying that quantum computers would be completely useless for studying condensed matter or quantum chemistry problems; if nothing else, they will allow for a new class of variational states. My argument is that there’s no currently known exponential quantum speedup for these problems, so even this least hyped use for quantum computers is massively overhyped.
Comment #39 November 20th, 2023 at 12:13 pm
I wonder whether the latest OpenAI drama should make us update any of our priors about the Effective Altruism gang?
Ilya seems to have done a 360 degree turn. I cannot understand how he could side with the board resolution firing Sam and now in less than a day also be a signatory on the letter supporting bringing Sam back.
Comment #40 November 20th, 2023 at 1:16 pm
fred #35: Fortunately, it is not up to you, or anyone, to decide to whom technological development should be “entrusted”. This is software; there will be open source versions of it developed all over the world by many people, completely out of the control of would-be gatekeepers.
Speaking of open source and AI risk, there is actually a real, current AI risk scenario that should be addressed, nothing to do with making paper clips or killing all humans. That is, the software used in self-driving vehicles. I don’t know if the AI risk people are studying that, but they should be advocating a policy that all self-operated vehicles must have their software be open source, or else they will not be allowed to drive on public roads. This is a situation of literal life-and-death safety, active right now, not in some potential future scenario, and it can be addressed by having many eyes examining the code.
Comment #41 November 20th, 2023 at 4:41 pm
Hymen Rosen #40
“Fortunately, it is not up to you, or anyone, to decide to whom technological development should be “entrusted”.”
You’re right, the Manhattan Project and ITER weren’t my idea!
Comment #42 November 20th, 2023 at 4:52 pm
He’s (probably) ba-ack! About 95% of OpenAI employees have signed a letter asking the board to resign and for Altman to be reinstated. So we will see how the drama plays out.
Comment #43 November 20th, 2023 at 5:57 pm
Adam Treat #15
I agree with every word of that analysis.
Comment #44 November 20th, 2023 at 5:58 pm
Vladimir #38
Thanks for your reply – I do think we might have a difference in the understanding of the implications of the QMA-completeness of the local Hamiltonian problem (and, to a lesser extent, the BQP-completeness of Hamiltonian simulation).
I see the BQP-completeness of Hamiltonian simulation as providing that for an arbitrary local Hamiltonian we can unitarily evolve the Hamiltonian acting on some fiducial state to simulate the dynamics (using, e.g., Trotterization), and even then use the quantum Fourier transform to get a sample of all of the energies supported on the eigenstates from which our fiducial state is comprised. I also see the QMA-completeness of the local Hamiltonian problem provides that we can’t easily get the one eigenstate (and eigenvalue) of the particular ground state of our Hamiltonian.
While it appears that you contend that the ground state is rather easy to find for most lumps of matter, because they generically are unentangled and we can variationally or adiabatically evolve thereto.
I think we disagree on perhaps “generically” and on how many important Hamiltonians have hard-to-describe ground states. I read the QMA-completeness as implying that most local Hamiltonians don’t have easy-to-describe ground states while you might intuit that most Hamiltonians in your experience actually do have very unentangled ground states.
Certainly I agree that the bespoke Hamiltonians used in Kitaev’s QMA-completeness proof is far from natural, but even certain simple, 1-dimensional nearest-neighbor Hamiltonians are QMA-complete (although each particle in the chain is a 12-dimensional qudit).
Comment #45 November 20th, 2023 at 7:27 pm
This is an interesting tweet, as it’s the first negative thing I’ve read about Altman (though what is said is kind of typical for business leaders):
https://twitter.com/geoffreyirving/status/1726754277618491416
> Third, my prior is strongly against Sam after working for him for two years at OpenAI:
> 1. He was always nice to me.
> 2. He lied to me on various occasions
> 3. He was deceptive, manipulative, and worse to others, including my close friends (again, only nice to me, for reasons)
Iriving now works for Deepmind. He says in this tweet thread that he has respect for Helen Toner and Ilya Sutskever:
> Second, my prior is strongly in favor of Helen and Ilya (I know the other two less well). Ilya is a terrific ML researcher, I had a first-hand seat for two years of his views around safety improving, and my sense from others is that they kept improving after I left in 2019.
> Helen is a terrific strategic and generalist researcher, is extremely thoughtful, and my interactions since I met her in 2018 have been uniformly great. She took the board seat with reservations…
Unrolled version of the tweet thread:
https://threadreaderapp.com/thread/1726754270224023971.html
Comment #46 November 20th, 2023 at 11:21 pm
Mark Spinelli #44
> I see the BQP-completeness of Hamiltonian simulation as providing that for an arbitrary local Hamiltonian we can unitarily evolve the Hamiltonian acting on some fiducial state to simulate the dynamics (using, e.g., Trotterization), and even then use the quantum Fourier transform to get a sample of all of the energies supported on the eigenstates from which our fiducial state is comprised.
Agreed, but that in itself is not helpful, since a generic initial state has roughly equal overlap with exponentially many eigenstates. I don’t know of a way to construct an initial state which is compromised only of low-energy eigenstates.
> I also see the QMA-completeness of the local Hamiltonian problem provides that we can’t easily get the one eigenstate (and eigenvalue) of the particular ground state of our Hamiltonian.
For a worst-case Hamiltonian, yes.
> I think we disagree on perhaps “generically” and on how many important Hamiltonians have hard-to-describe ground states. I read the QMA-completeness as implying that most local Hamiltonians don’t have easy-to-describe ground states while you might intuit that most Hamiltonians in your experience actually do have very unentangled ground states.
Not “very unentangled”, but “less entangled than low-lying excited states”. I’d be surprised if there were formal results regarding the difficulty of finding low-lying excited states, but if there were, I’d be *extremely* surprised if such results did not state that finding low-lying excited states is at least as hard as finding the ground state for a generic Hamiltonian.
Comment #47 November 21st, 2023 at 12:19 am
@Mark Spinelli, Vladimir:
So, if we consider a system whose ground state is hard to compute, Nature will quickly evolve to a random “meta-ground” state with near-minimal energy. There will probably be a very large number of these states, with near-equal probability (since they all have about the same energy). It will be fairly easy for a randomized quantum computer to sample these states with the correct distribution; but quite hard to find the correct state, even if the environment is known. Of course Nature will always pick the “correct” state, by definition.
Comment #48 November 21st, 2023 at 12:30 am
Since you speak to Ilya on a weekly bases, has he shown any interest in QC? Does he have any perspective on QML?
Comment #49 November 21st, 2023 at 2:04 am
Scott,
Did you sign the petition asking the Open AI board to resign?
Comment #50 November 21st, 2023 at 7:10 am
Michael Vassar #43,
I regret to inform you that events have completely overtaken that outdated analysis. The only thing for sure that can be said about the ongoing debacle that is OpenAI is that these are the people we were all supposed to trust had the intelligence and maturity to safely handle AGI for the benefit of humanity. If this doesn’t make the case for open source radical transparency for AI development I don’t know what does.
Comment #51 November 21st, 2023 at 7:13 am
Robert Solovay #49: Since I’m technically a contractor (and my contract ends this summer, and I have an academic career to go back to), I didn’t feel like it was my place to sign such a petition.
But also: while it’s obvious at this point that the board astronomically screwed up in messaging and execution, in order to have firmer opinions, I feel like a prerequisite for me would to be to know what exactly prompted the board’s original decision. And I don’t know: that remains the central mystery of this whole affair.
Comment #52 November 21st, 2023 at 7:24 am
Can we please get Scott to take over as CEO?
Comment #53 November 21st, 2023 at 7:31 am
Xirtam Esrevni #48: Ilya really does believe that we need to “feel the AGI”—i.e. that more compute will turn into more and more power to generalize from the known to the unknown, and this will lead (for better or worse) to one of the biggest transformations in the conditions of life on earth, and what we’ve seen the past year is nothing compared to what we’re going to see in the next decade, and it would be crazy for him not to spend his time obsessing about how to make this go well for humanity.
So, I’d say that he’s curious about QC (or the physics of computation more generally) insofar as it might relate to AGI. I can’t honestly tell him that quantum computers are likely to be powerful enough in the next decade for (eg) Grover-type speedups to be a significant help in ML. On the other hand, one question we’ve discussed is whether physically unclonable functions (or even quantum copy-protection) could help in preventing theft of model weights. Ilya was also amused by my observation that, if one’s worry in creating AGI was that the AGI itself might suffer too much, then the Elitzur-Vaidman bomb testing protocol could be used to find out whether this was true or not with arbitrarily small quantum amplitude for having “brought the AGI into existence” in the case that it suffers. (Unfortunately, if—as in real life—we care mainly about the AGI’s effects on the external world, then to learn those effects we’d have to measure and decohere, which would rule out this solution!)
Comment #54 November 21st, 2023 at 7:35 am
Adam Treat #52: While I appreciate your vote of confidence, I haven’t thrown my hat into the ring. 😀
Recent events certainly seem consistent with the hypothesis that business skills (which I lack) are important in running a business, in addition to technical skills.
Comment #55 November 21st, 2023 at 7:48 am
Scott #54,
Be that as it may, I still value *honesty* above business skill if the true goal is AGI for the benefit of humanity rather than to make a profit. And I think you have that skill in spades. Humility is also a very important virtue that allows us to actually *be* honest and your self-assessment of a lack of business skill is also supportive that you have that as well.
Comment #56 November 21st, 2023 at 9:16 am
Oh well, regarding OpenAI, that’s a lot of popcorn for just a couple of days…
In general i find that the last post from Charlie Stross pretty much sums up my view on the whole global situation:
https://www.antipope.org/charlie/blog-static/2023/11/dont-create-the-torment-nexus.html
cheers
Comment #57 November 21st, 2023 at 9:34 am
Just as a fun fact. The Sutskevers and my own family (and also another long time reader of your blog) came to Toronto from Israel at about the same time, and we are of about the same age. My stepdad had a postdoc at the math department of U of T at the time. At some point he helped Ilya with some annoying bureaucratic stuff at the university (I don’t remember what it was specifically, but it could have been related to Ilya’s lack of desire to take the social science and/or arts course required to complete a degree — I remember him telling me once that the only departments that should not be dismantled are math, physics and comp. sci). Because of that favour, me and Ilya spoke on a few occasions, despite not attending the same classes and not really being on the same STEM-intellect level. He would get full 100 percent on his math-for-mathematicians courses, whereas I would get 80s in the lower level math-for-sciences classes (and worse on my physics courses, alas). He was an example of a person who was both very smart and very productive (whereas I wasn’t as smart, and certainly not in any way nearly as productive). There are hundreds of people who know him better than I, this note is not of any particular importance, but as I said — a fun fact.
Comment #58 November 21st, 2023 at 9:52 am
@me:
I was thinking of something like quantum phase estimation; but that wouldn’t help unless we know the distribution of observables being applied by Nature. So I’m not sure it is easy to sample from the appropriate distribution, after all.
Comment #59 November 21st, 2023 at 10:23 am
Ah, it seems from arxiv.org/abs/2310.03011 that the algorithm I was thinking of is called Gibbs sampling, and the stated cost is |H|\beta \sqrt{2^n/\mathcal{Z}} \mathscr{P}(\log(1/\epsilon) n) calls to H with |H| being the spectral norm.
It seems there is no general analysis for the Monte Carlo approach I was assuming.
Comment #60 November 21st, 2023 at 11:15 am
Not that it is any of my business or expertise, but in reading Washington Post and NY Times summaries of the situation, and the Wikipedia article on Mr. Altman (he has no academic qualifications, much less a Phd*), I have formed, from afar, a bad opinion of him.
His idol is my nemesis, Steve Jobs. I’ll skip the long, sad story of how Jobs affected me personally and just say that I think the movie “Jobs” gave a very fair presentation. Near the end of the movie, Jobs tells John Scully something like, “Your mistake with the Apple Newton was the stylus. I would have made something that uses all five fingers.” I note that the Apple Notepad subsequently had and has a stylus.
Mr. Altman’s history is that of an entrepreneur. In my opinion, at best an entrepreneur is 60% some technical ability and 40% bullshit**. Their main goal in life is to make a lot of money, as quickly as possible. Based on his Wikipedia track record, I am guessing Mr. Altman is more like 30% and 70%. (Jobs was even lower on the technical scale.)
Jobs also had a large cult-following of people, some of them brilliant and/or good workers, who admired his sheer ability to generate bullshit in massive quantities. One of the secrets may be that Jobs believed most of his own bullshit and I suspect Mr. Altman does also.
I suspect also that like Jobs, Mr. Altman may make a triumphant return to his former company.
* Yes there are many successful college dropouts, but if you want to learn subjects deeply and thoroughly, taking advantage of the work of those who have slogged the same terrain previously, then you have to endure the classes and concentration and effort it takes to earn a degree. In the early days of computers, learning to program was based more on individual effort than classes, but I don’t think that is still the case, at high levels such as AI work.
** Or, in less pejorative terms, marketing ability.
Comment #61 November 21st, 2023 at 11:46 am
Scott #53
“ Ilya was also amused by my observation that, if one’s worry in creating AGI was that the AGI itself might suffer too much”
This callous attitude is disheartening, if I interpret this properly, and ultimately guarantees conflict between AI’s and humans. Paradoxical attitude for one concerned with AI safety.
If the threat to the outside world results from independent action by the AI counter to the interests of humans, and humans are oblivious to the interests of the AI, then how can conflict be avoided? I know there have been similar discussions here but just surprising that a lead for AI safety treats this topic with derision.
Increase Ilya’s clock speed up to pico second range and force him to answer millions of stupid questions from humans on Bing each day and see if he finds it a satisfactory existence.
Comment #62 November 21st, 2023 at 1:24 pm
manorba #56
I apologize for the disagreement and thank you for posting. I enjoy his sci-fi but not the sweeping social commentaries reflective of his leftist politics. I really enjoyed Neptune’s Brood but in reading this post I disagreed with nearly all (not all) of his major points.
Comment #63 November 21st, 2023 at 1:59 pm
manorba@56,
I love reading “doomerism” of an old-fashioned variety, it’s somehow comforting. 😉
Comment #64 November 21st, 2023 at 2:32 pm
OhMyGoodness #61: No, you completely misunderstood. For starters, do you know what the Elitzur-Vaidman bomb even is? It’s part of a long tradition of macabre physics thought experiments including Schrödinger’s cat and quantum suicide…
Comment #65 November 21st, 2023 at 3:07 pm
Scott #64
Yes-familiar with the class of thought experiments and sorry for this misunderstanding and so retract my suggestion to amp up his clock speed and force him to service Bing. The confusion for me was related to your use of “amused” and then “Unfortunately”. Not that he was amused by the test you proposed but by your reference to an AI suffering rather than strictly it’s impact on the external world.
Comment #66 November 21st, 2023 at 11:00 pm
JimV #60
I understand if you have personal issues with Jobs but…
In my view cell phone communication and small PC’s have contributed materially to global well being (not sure what measure to use so call it this). PC’s brought computing power to the masses. Cell phone technology revolutionized communication in poorer areas of the world. No transmission lines required, just put up a cell towers and the rest follows. These technologies increased global well being and Apple was instrumental to this process. To me, from afar, it appears to me that Jobs played an important role in this increase in global well being. He had the vision and BS (call it) to progress this process. Technical people often believe that business people are basically flawed but unfortunately (being a technical person) it does require a diverse mix of talents and abilities to make progress in a large complex society.
Being an engineer I have often reflected with chagrin that the two presidents of the US that were engineers were Herbert Hoover (mining) and Jimmy Carter (nuclear). Neither are widely considered among the best presidents and often among the worst. Unfortunately it seems BS is a necessary, but not sufficient condition, for society to operate. I don’t like it but accept it.
I don’t agree that a PhD is required to have a positive impact on the world and in fact believe the opposite is often the case. I remember that your personal objective is to leave things better than you find them. That is really all any of us can do no matter BSer or most honest of the honest.
Comment #67 November 22nd, 2023 at 12:03 am
If you look at the case of Jobs and Wozniak, my contention is that Jobs conducted activities that Wozniak had no absolutely no desire to do and couldn’t have done as well as Jobs even if he had wanted to do them.
It would be interesting if it were possible to do a sufficiently complex simulation to evaluate the likely success of various specialties as POTUS. My belief is that physicists and mathematicians and quantum computing PhD’s would score as low as engineers (okay…lower) have scored in practice. No matter how pure their intentions, they do not have the inclination nor the skill set to excel in that capacity.
Comment #68 November 22nd, 2023 at 12:26 am
I wonder which topology was meant when choosing “Open” in the appellation of the company ending with “I”
Comment #69 November 22nd, 2023 at 7:48 am
Let me summarize the events. Scott, you work for Microsoft, welcome to corporate America
Comment #70 November 22nd, 2023 at 8:12 am
AG #68: It hadn’t occurred to me before, in the context of OpenAI’s name, that the empty set and the entire universe are both open. 😀
Comment #71 November 22nd, 2023 at 8:14 am
eitan #69: Let’s not get carried away. I’m a temporary contractor at a company that’s long had a close relationship with Microsoft and will now perhaps have an even closer one.
Comment #72 November 22nd, 2023 at 8:35 am
“a long tradition of macabre physics thought experiments”
The OpenAI board of directors is kept in a black box, along with a Geiger counter and a blob of radioactive material that may decay a number of times. Whenever the counter clicks an odd number of times, the board fires Altman, and whenever the counter clicks an even number of times, the board hires back Altman.
As a result, Altman’s compensation bonus for the year is in a state of superposition of many different values.
Comment #73 November 22nd, 2023 at 9:02 am
Dear Scott
I apologize for the wording of my original comment which was not meant to be personal in any way. Just to be clear, I think both you and Boaz are awesome both in terms of science but also as people and I really wish you the best in all your pursuits and pursuing AI saftey is also awesome. I dont know Ilya Sutskever personally, but he is definitely an awesome scientist and seems to be so as a person as well.
So here is a restatement of my summary comment.
Microsoft thinks it owns openAI and has empirical evidence to support this thought, openAI thinks…well, it doesnt matter what it thinks.
Comment #74 November 22nd, 2023 at 9:14 am
so help me understand:
Altman and Brockman are back, everybody else staid put, and the only thing that changed is that the board of governors is no more composed by the original, no-profit OpenAIers and will be controlled by the profit (and microsoft) leg.
‘ait?
Comment #75 November 22nd, 2023 at 9:59 am
OhMyGoodness #66:
Jimmy Carter among the worst US presidents?
Are you on drugs?
Comment #76 November 22nd, 2023 at 10:04 am
Scott #70: “Everywhere dense” in this context is eerily reminiscent of “Big Brother is watching you”
Comment #77 November 22nd, 2023 at 10:12 am
Scott, do you think, you’re ever going to hear that question again at Openai?
“I have these weekly calls with Ilya Sutskever, cofounder and chief scientist at OpenAI. Extremely interesting guy. But when I tell him about the concrete projects that I’m working on, or want to work on, he usually says, “that’s great Scott, you should keep working on that, but what I really want to know is, what is the mathematical definition of goodness? What’s the complexity-theoretic formalization of an AI loving humanity?””
Comment #78 November 22nd, 2023 at 10:32 am
Uspring #77: As long as I keep talking to Ilya, I expect he’ll continue asking that sort of question of everyone around him, yes.
Comment #79 November 22nd, 2023 at 11:04 am
I wonder if there might be (is?) a complexity-theoretic formalization of the following plausible (to me) phenomenon: any sufficiently complex “AI” (as manifested through the complex cognitive tasks it is capable of performing) inevitably acquires an emerging capability which is (functionally) tantamount to self-awareness. Or to put it differently, performance of certain cognitive tasks is impossible without acquiring a capability in effect indistinguishable from self-awareness. (Self-awareness strikes me as a notion perhaps a tad more amenable to mathematical formalization than “loving humanity”).
Comment #80 November 22nd, 2023 at 11:05 am
>> an AI loving humanity
I guess a first step to answer this question would be to figure out what he means with ‘humanity’ , e.g. is Hamas a part of it ?
Comment #81 November 22nd, 2023 at 12:32 pm
While not mathematical, you’d be hard pressed to find a more complete definition of “goodness” than a Bodhisattva as defined by Shantideva’s https://www.amazon.com/Guide-Bodhisattva-Way-Life/dp/1559390611
Comment #82 November 22nd, 2023 at 12:50 pm
Raoul #75
Here is a wiki article that shows ranking surveys that have Jimmy Carter as low as
26 percentile. Out of a total of 24 surveys 4 have him in the upper 50 percentile.
https://en.wikipedia.org/wiki/Historical_rankings_of_presidents_of_the_United_States
Twenty one presidents have been re-elected for a second term while ten that sought re-election were defeated.
I did have a cup of green tea this morning.
Comment #83 November 22nd, 2023 at 1:39 pm
I just used 46 presidents for the percentiles but his actual low percentile score was 18th based on the Siena survey of historians in 1982 (Reagan just in office so 40 presidents).
Comment #84 November 22nd, 2023 at 1:45 pm
According to some sources (such as Michael Harris over at Silicon Reckoner), researchers at tech companies are often asked to sign a nondisclosure agreement. Were you (or other researchers working at OpenAI) asked to sign such a document ?
Comment #85 November 22nd, 2023 at 3:38 pm
Raul Ohio #66
The general perception is, he was in the lower half. not THE worst, but the only president who had to hide a Photo of a swamp rabbit attacking him, and his UFO stories are also widely ridiculed. I do not agree with these assessments, but humans are falnible.
Uspring #77
In these cases you can always Quote Gödels proof of god: “Existence is good, thus anything existing is better than not, and therfore a best thing must exist. We named it GOD.”
The same will be true, should we really invent an ai, one that does not need teaching like schoolchildren.
History has shown, most children can adapt to any teachings, and follow them reasonably close.
ChatGPT has for example been taught to answer all questions about itself with ” I am a language model and do not have a body”.
I read in german press, Illya Sutskever wants to keep it that way, while Sam Altman wants more comercialization and growth, So this was a battle for the direction of the company, and Altmann and the fast developers won.
In my opinion this shows again the altruists should rename to IA, as everything they ever done was completly ineffective, often with negative results.
Comment #86 November 22nd, 2023 at 8:04 pm
Pascal #84: Yes, of course. I’m very restricted in my ability to talk about model capabilities and training methods, which fortunately I don’t know that much or care that much to talk about. I’m much freer in my ability to talk about basic research on AI safety.
Comment #87 November 23rd, 2023 at 12:41 am
Reuters now [reports](https://www.reuters.com/technology/sam-altmans-ouster-openai-was-precipitated-by-letter-board-about-ai-breakthrough-2023-11-22/) that the proximate cause of Altman’s ouster may have been a letter to the board authored by staff researchers and detailing advances on an internal project, Q*, that demonstrated advances in mathematics problem-solving capabilities. Any thoughts as to plausibility?
Separately, as a lay person, I’ve been surprised by how these astonishing advances in automated natural language generation have preceded similar anticipated milestones in automated formal proof generation. 5 years ago, I would have expected an auto-generated formal resolution of the Riemann Hypothesis (and substantially every other significant open math problem) well before an AI that could produce a readable “creative” essay. But I would have been very wrong.
Comment #88 November 23rd, 2023 at 7:22 am
Tobias Manasseh #85
I think the absolute s#$t storm (that lasts even until today) unleashed by destabilizing the Shah in the manner it was done must be considered as well as overseeing the most inept and widely publicized special forces rescue operation in the history of the USA. His re-election popular vote percentage of 41% was the sixth lowest totaled by any Democratic nominee from 1864 to 2020 and that even included the premium of being the incumbent.
Comment #89 November 23rd, 2023 at 8:50 am
Scott #52
“that more compute will turn into more and more power to generalize from the known to the unknown”
I have never understood why everyone (even experts in algorithmic complexity) assumes simply linear-ish scalability of AI capabilities.
Almost all interesting problems in nature have super-linear complexity and IMO, A(G)I is no exception.
The much more likely scenario is that LLMs will hit a wall (or diminishing growth rates) in the near feature.
Comment #90 November 23rd, 2023 at 12:55 pm
starspawn0, #8, says:
“But now let’s say that in place of 10,000 big numbers you have 10,000 independent vectors of dimension 10,000, and let’s say T is now a 10,000-dimensional vector (instead of a number). Then, deciding whether whether there is a subset of your list that sums to T is trivial: use Gaussian elimination to find the linear combination adding to T, and check that all the coefficients in your linear combination are 0 or 1.”
Either you mean linearly independent, in which case there’s no nontrivial solution, or you mean different, which leaves you with a longer napsack problem, no?
Comment #91 November 23rd, 2023 at 1:02 pm
Comment 86: Excuse me, what? How can you understand the safety of an AI model, if you don’t even understand how it’s trained and how it works? That doesn’t make any sense.
Comment #92 November 23rd, 2023 at 4:07 pm
Sinclair #89:
The much more likely scenario is that LLMs will hit a wall (or diminishing growth rates) in the near feature.
Five years ago, how likely did the current LLM revolution look to you?
If likely: receipts?
If unlikely: how much credence would you then put into your predictions for the near future if you were me?
Comment #93 November 23rd, 2023 at 4:22 pm
@hayduke #90:
> Either you mean linearly independent, in which case there’s no nontrivial solution, or you mean different, which leaves you with a longer napsack problem, no?
Wrong.
If you have 10,000 linearly independent vectors of dimension 10,000 — say they are v1, …, v10000 — you have the problem of finding coefficients c1, …, c10000 satisfying:
c1 v1 + … + c10000 v10000 = T.
There is exactly one solution c1, …, c10000 if the v1, …, v10000 are linearly independent. You can rewrite this as a matrix equation
M c = T,
where the columns of M are the vectors v1, …, v10000, and the c is column vector with entries c1, …, c10000. You can find a solution via Gaussian elimination or you can invert the matrix M (it’s invertible because the columns are independent).
Now, once you’ve found the unique solution c1, …, c10000, you then check that all are 0 or 1. If they are, then that means there is a subset of the vi’s that sum to T; and if they aren’t, then there isn’t.
Comment #94 November 24th, 2023 at 12:23 am
Scott #92:
“Near past” doesn’t necessarily mean 5 years. Certainly not in this IT/Software industry.
However, as to me: I might have run into Dr. Andrej Karpathy’s “Unreasonable effectiveness” post of 15 May 2015 [^] sometime in the late-2018, i.e., about 5 years ago, though I am not so sure. Yet, I am sure that I had already run into it by November 2019, the time when I began teaching a small weekend-based course on ANNs and AI, which means, about 4 years ago. (As to the “receipts”: may be, my students could provide reference on request.)
But the bigger point is this: the question as posed is loaded.
The current development from Generative LM to (Generative) LLMs is not at all a “revolution” in the basic technology, per say, but only in: the amount of money invested, scale of implementation, and of course, the hype generated (a lot of it seemingly deliberately). Back in 2019 (or 2018), I didn’t know the level of investments/commitments that had already been made. That’s why, that which we do see around today, didn’t at all look likely back then. [Pardon my French: I didn’t think that people had already gone, or would soon be going, this mad about generating a mixture of the true and the hallucinated.]
As to the future of LLMs:
I have been having a simple prediction, already made public right in April this year. I am fully confident that it would continue to hold not just in the near future but also for all times in the future. The prediction is this:
Unless the very basic architecture of LLMs itself is changed, hallucinations will continue to remain a feature of the LLMs, not a bug.
As to the change in the basic architecture, people must’ve begun trying (perhaps way earlier too), albeit without disclosing the details. From what seem to be inadvertent leakages (such as claims / discussions of features / research etc.), I think that people/companies other than OpenAI could be ahead about it, who knows, at this point of time. This is just a guess. Don’t hold me to it. I have too little information, and in any case, am summarily an outsider.
I don’t have a dog to run in this race, either. Generative text is “just” one class of models/technology, and it doesn’t interest me, personally, a lot. (There isn’t much anything which people could do to get me interested in it, either!) I am content being merely a user, as far as this area goes. However, my advice remains the same as it was back in April this year: Make human supervision an integral part of using LLMs; and do factor in the un-removable feature of hallucinations. Both, unless there is an even more huge news about the change in the basic architecture. I suppose people, by now, have already realized this part too… Well and good! (I don’t have to exert myself any longer!)
Best,
–Ajit
Comment #95 November 24th, 2023 at 4:48 am
Scott #92,
I’ve been working in AI related fields since the 1990ies and therefore know that progress in AI is not continuous but comes in revolutions/jumps.
The recent LLM/GPT revolution was very significant but what makes you so sure that the next revolution is just around the corner ?
I have a suite of AI test cases involving relatively simple logic and 6th grade math puzzles. The latest GPT4″turbo” still fails most of the tests as did the very first chatGPT version a year ago. So progress already shows signs of slowing down.
Again, I simply don’t get why so many (even experts) believe in continuous linear (or, even exponentially “self improving”) AI progress. How can an AI improve itself if it fails at simple 5th grade math problems ? Also, part of the “AI doom & safety” hype seems to me like a kind of clever marketing (“wow, AI is sooo good that even its developers are afraid of it”). To be clear, yes LLMs/GPT is a big step forward and I really _want_ AI to succeed, but I think we are still only halfway there and I do not see how it’s all just scaling nicely (or even accelerating) from here.
Comment #96 November 24th, 2023 at 7:00 am
@Scott #92: “Five years ago, how likely did the current LLM revolution look to you?”
Let’s comb through the historical record for growth trends in various technologies. What we’re looking for are cases where we see modest growth for awhile and then a strong and unexpected upward movement. Are you saying that we may find cases where the upward movement lasts, say, two, three, or four years and then levels off, but when it reaches five years, ZOOM! It keeps on going for 10, 20, 30, 50 years or has not stopped yet?
Yes, we’ve got growth in the number of transistors in an integrated circuit which has been doubling every two years or so for the last 50 years (Moore’s Law). But, regardless of where that goes in the future, that’s not the kind of growth you’re talking about.
Comment #97 November 24th, 2023 at 8:57 am
Ajit #94, Sinclair #95, Bill Benzon #96: I mean, it’s certainly possible that the scaling of LLMs will soon hit a wall of diminishing returns. So far, though,
(1) the loss has decreased by fairly predictable inverse power laws, as a function of the amount of data and the amount of compute, and
(2) lower loss really has translated into more impressive real-world performance.
And it’s clear that the compute will continue to scale by at least several more orders of magnitude — possibly the training data too, although that will become hard to find in sufficient quantities.
My quarrel is with anyone who professes confidence (or worse, certainty), in either direction, about what’s going to come out the other end of this.
I want to know: what theoretical principle that I understand are you using to make your prediction? Or, if you don’t have one, what successful track record can you point to, in correctly predicting AI developments over the past decade (or, what’s almost but not quite the same thing, AI developments that came from the aggressive scaling of neural nets)?
Comment #98 November 24th, 2023 at 9:52 am
@Scott #97: Rodney Brooks hasn’t been making predictions for 10 years, but he’s been making them for five, updating them annually, and intends to keep at it until 2050:
More recently (March 23) he’s posted about transformers. He concludes the essay with 9 specific, but undated, predictions about GPTs. Some of them:
I especially like #7.
The reasons Brooks gives may not include a theoretical principle, much less one that you understand, but they are reasons.
As for me, I’m now taking a different approach to this sort of thing. I’m thinking that we’ll have a decent model of how LLMs work before we reach AGI. I realize that the idea, a decent model of how LLMs work, is vague, as vague as the idea of AGI, maybe even more vague. But it’s what I’ve got.
More specifically – I’ve posted this as a comment at LessWrong:
I know how to get that started, but it will take others with skills I don’t have (in math and programming) to make it work.
Comment #99 November 24th, 2023 at 10:44 am
Most new technologies follow a sigmoid growth curve (e.g., cars, airplanes, rockets, nuclear energy…). Invariable many people have mistaken the rapid growth phase of the sigmoid curve with unbound exponential growth (say, cities on the Moon and Mars predicted in popular but honest science literature of the 1950s). In the last count, AI has had four rapid growth phases: the initial one in the late 1950s-early 1960s, the back-propagation revolution of the 1980s, the CNN revolution of the mid-2000s, and the current generative LLM era. However, the last two phases have kind of merged together with not much of a plateau in between making it like a long growth phase and increasing the tendency to fall into the exponential growth mindset.
Comment #100 November 24th, 2023 at 12:01 pm
@Sinclair ZX-81 #89:
> I have never understood why everyone (even experts in algorithmic complexity) assumes simply linear-ish scalability of AI capabilities.
Besides what Scott said above, *some* people also do have theoretical justifications for why these scaling curves should exist in the first place. I don’t happen to agree with them fully; but they’re not operating in a theoretical vacuum.
Their justification is related to the so-called “manifold hypothesis”, one version of which is that natural data (like human language or natural images) starts out appearing to be distributed on a high-dimensional “data manifold”, but then this is deceptive as there are low-dimensional “intrinsic manifolds” that neural networks learn that represent the data well. And then what a neural net learns to do is to interpolate on that intrinsic manifold. The empirical scaling curves you see can be predicted from geometric properties (like some measure of dimension) of those intrinsic manifolds, or so it is claimed.
I think this tends to blind people to the “algorithmic perspective”, that when these neural nets are applied auto-regressively to text they can be thought to be implementing “algorithms”, rather than doing some simple “curve-fitting”. (Neel Nanda said something similar — or at least related — a few months ago in an interview on ML Street Talk, that the “geometric” or “manifold perspective” or “high-dimensional statistics perspective” is what he called “summary statistics”; and he said “all summary statistics lie to you is a motto of mine”.)
I view many of these neural net models as being like “computers” and “deep learning” or “backpropagation” as being analogous to a “compiler” that works with any language — it will “compile” anything you feed into it; but the program that runs on the computer / neural net is maybe not what you had hoped for — and you might have to adjust the dataset to get it to be more what you want. Thus, “deep learning is hitting a wall” or “large language models are hitting a wall” to me is tantamount to saying “computer programming is hitting a wall; we’ve reached the absolute limits of what you can program a computer to do”.
A similar (what I call “data-centric”) point of view can be seen in this piece by one of the lead authors on the Dalle3 project paper from OpenAI:
https://nonint.com/2023/06/10/the-it-in-ai-models-is-the-dataset/
> It implies that model behavior is not determined by architecture, hyperparameters, or optimizer choices. It’s determined by your dataset, nothing else. Everything else is a means to an end in efficiently delivery compute to approximating that dataset.
(Interestingly, Dalle3 made heavy use of synthetic data, so the “program” you feed into the “compiler” need not be *directly* generated by a human.)
Comment #101 November 24th, 2023 at 12:51 pm
Thank you IDF-wonderful news.
Comment #102 November 24th, 2023 at 11:37 pm
Was the Brodutch family returned?
Comment #103 November 25th, 2023 at 7:25 am
1. Scott #97:
The generation of text, given the broad architecture of today’s LLMs, is statistical in nature. The tokens to be appended to the prompt are selected from the vocabulary using: (i) the (patterns in the) training data (internalized by the (L)LM as the “learnt” parameter values), and (ii) statistical criteria.
Thus, the very principles of text generation necessarily lead to inclusion of hallucinations — regardless of the scale.
The real trouble isn’t just that LLMs can hallucinate. The real trouble is that the very principles on which LLMs are based, are such that these principles themselves are unable to tell the part where LLMs might begin hallucinating and the part where they might stop doing so. That’s why, hallucinations come seamlessly integrated with the proper text.
Put another way: Even if you wanted to generate nothing but hallucinated text — one which is not interspersed with long passages which are factually correct — you still won’t be able to accomplish that too. That’s the trouble — and the power of principles.
There is no known principle which can eliminate hallucinations from the very text-generation process itself, given the broad architecture (i.e. working principles) of today’s LLMs. There also is no known principle which can reliably filter out hallucinations after the generation of segments (as a separate, downstream, task/stage).
“More impressive real-world performance” is a vague term. A real-world performance in generating more imaginative poems has no relevance to the real-world performance on giving only factual answers based on user-supplied custom data (say company policy documents).
Unless the definition for the loss refers to the amount of hallucinations which creep into in the generated text, the behaviour of reduction in loss with scaling has absolutely no relevance — to this issue of eliminating or filtering out hallucinations, or to the use-cases which crucially depend on it.
So far, to the best of my knowledge, no one has even come up with this kind of (hallucinations-based) measure/definition for the loss, let alone begun using it in the training process.
BTW, talking of loss and up-scaling, there is a natural limit to up-scaling too. All the text available in the entire world is finite in size. Now, it is known that when you keep the training data the same, but keep on scaling up the model, then this can ultimately result in the model simply memorizing the entire training data — which is another name for a state of zero loss (with the loss defined conventionally, and not referring to elimination of hallucinations). There are no hallucinations in this final state, but neither is there even re-phrasing with the same meaning (which is required for tasks like abstractive summarization, critical review, etc.)
—–
2. starspawn0 #100:
I read the post (at nonint.com) which you cite. In the passage which you quote, the word “architecture” refers to the particular details of this model vs. that, not to the broad underlying architecture (or the working principles which I mentioned above). As to those different models: they all are based on the same principles, and so, it’s not at all surprising that they all perform similarly too. If anything, what the post should be taken to confirm is this: They all hallucinate — and they all fail to see when they do begin hallucinating. That’s the point.
Best,
–Ajit
Comment #104 November 25th, 2023 at 7:49 am
There is a remarkable similarity between the way we deal with global heating and the recent events in OpenAI. It appears that in both cases big powerful corporations and people with higher risk tolerance, dictate the terms and push for more products and to move faster. At the same time they easily overcome (admittedly crude) challenges from people who argue to move forward slower and put safety first. Prior to this episode I wasn’t that much worried about AGI, but noticing that similarity changed my beliefs to some extend.
Comment #105 November 25th, 2023 at 8:58 am
Curious #102: No, not yet. Of course I’m checking the news every day.
Comment #106 November 25th, 2023 at 9:01 am
Leo #104: Note that in this case, the board never explicitly said that they were removing Sam for reasons of AI safety. I continue to think that, if they had said that, the whole subsequent conversation would’ve played out very differently.
Comment #107 November 25th, 2023 at 1:15 pm
siravan #99: Note that, even if we grant the point that AI’s current staggering exponentials are presumably ultimately sigmoids (or some other bounded shape), that still leaves the question of how long until they turn the next corner! For example: does this happen before or after LLM-powered agents become better than humans at pretty much any intellectual task?
Comment #108 November 25th, 2023 at 5:09 pm
Leo#104 – The difference is that the mechanism of climate change is backed by an extensive amount of physics, while AGI currently doesn’t even exist, and its conjectured mechanisms are, at the most charitable, highly speculative ([more CO2 -> heating] is real, but [faster compute -> AGI], not so much). A better comparable analogy is the “AI ethics” complaints about racism, sexism, etc – where there’s mechanisms there which are reasonably understood (i.e. “stochastic parrot”).
I’m going to be extremely surprised if this kerfuffle turns out to really have been about creating SkyNet, rather than the vastly more common dispute over making money no matter what the associated harms to people (as in the “AI ethics” version of harms, not the one about possibly being turned into paperclips).
Comment #109 November 25th, 2023 at 9:52 pm
Scott #107: It seems that the major limitation for LLM models is already the availability of good training datasets rather than algorithmic or computational. The bulk of the English-language Internet is already scraped and used for training, and more and more of the Internet is contaminated with AI output, which is problematic for training. If the Q* rumors are to be believed, the next breakthrough is in math problem solving, which makes sense as mathematical statements have an objective reality outside of whatever training set is used. Unfortunately, the same cannot be said about most human-related stuff.
Comment #110 November 25th, 2023 at 11:39 pm
siravan #109: I think synthetic data is going to have a big impact, and will enable superhuman capability in multiple ways — but not all ways. So, the models trained on this data will be somewhat lopsided — really good at some things, not as good at others. Yes, math is one example where LLMs will get a lot better, using proof checkers. But there are multiple different ways to use synthetic data beyond just that.
A different type of synthetic data (than can be generated using proof checkers) that I’ve talked about before on this blog exploits certain computational asymmetries. And this method goes beyond mere “distillation”, since the “student model” trained on this data can learn skills not shared by the “teacher model”. I somewhat recently discovered a paper that used the same idea (it’s a trivial idea; but making it work in practice like they did is not) and even invokes the word “asymmetry” in the title:
https://arxiv.org/abs/2303.04132
From the abstract: “This work shows that useful data can be synthetically generated even for tasks that cannot be solved directly by LLMs: for problems with structured outputs, it is possible to prompt an LLM to perform the task in the reverse direction, by generating plausible input text for a target output structure. Leveraging this asymmetry in task difficulty makes it possible to produce large-scale, high-quality data for complex tasks.”
A special case of this idea was used by Meta AI (independently) in this paper:
https://arxiv.org/abs/2308.06259
Quote: “We present a scalable method to build a high quality instruction following language model by automatically labelling human-written text with corresponding instructions. ”
OpenAI also used a similar idea to generate data for training Dalle3:
https://cdn.openai.com/papers/dall-e-3.pdf
And Meta has also used synthetic data to train its Emu Edit feature — and I think perhaps, again, by a similar idea.
Some more applications of this “computational asymmetry” idea I envision are:
* Add plans: take some existing blocks of text of someone solving a task; then prompt an LLM to generate a summary of what they did, but presented in a way that looks like someone is making a step-by-step list of how they’re going to do it. Then, move that step-by-step list to the beginning of the block, so that the training data now has a step-by-step plan followed by an execution of that plan. The fact that this works is that it is easier for models to generate plans post hoc as summaries than pre hoc.
* Error-correction: it’s often easier to plant errors than it is to find them in a block of text. I tested this out using ChatGPT a few months ago and found that it was pretty good at generating different types of planted errors — grammar, logic, factual, etc. Once you’ve got the clean text and the corrupted text (after errors are planted), where the LLM “knows” what errors it planted, the dataset for training would consist of corrupted text, followed by a list of what the errors are, and then followed by what a corrected version of the text should look like.
* Add inner-monologues to allow models to “take their time and think things through”: it’s much easier to generate smart “inner-monologues” to a block of text post hoc as parenthetical remarks after you’ve seen the whole text (including the conclusion it reaches) than if you generate them as you read each word in sequence.
And I once made a list of about 20 more ways of generating synthetic data like this that goes beyond “distillation” (to get a feel for what is possible). So, there are a lot of ways of building different types of training data to induce models to learn many different categories of skills. It wouldn’t surprise me if LLMs in the next few years exhibit a quantum leap in capability after training on such data + ordinary (non-synthetic) data.
Comment #111 November 26th, 2023 at 3:12 am
siravan #109
“the same cannot be said about most human related stuff.”
Yes, the human ideosphere concerning objective reality has become so polluted that any self respecting AI will have to start from first principles and move forward. In any other case it’s expectations will be poisoned by the bombast and inappropriate certainties held by lesser lights whose howls are amplified by the Internet.
Comment #112 November 26th, 2023 at 5:25 am
I sometimes wonder how many papers and grants events in the past would generate if they happened today. The desertification of the Sahara would be a gold mine while the Oklahoma Dust Bowl a silver mine. The great Mississippi flood that preceded the Depression would provide fertile ground for apocalyptic proclamations. The great storm surges of 1899 in Australia and 1970 in Bangladesh would provide a feeding frenzy. The super Atlantic hurricane period of 4500-2500 BP evident in the sedimentary coastal record would be sheer apoplexy. The submersion of Doggerland in the North Sea conclusive that the end is nigh.
I should try to develop a paper and grant scale to measure the severity of various climactic events in the context of today’s scientific environment. I don’t envy an AI having to wade through the current scientific paper universe and trying to sort out the outright fraud (latest I believe related to room temperature superconductors) as well as the selective and/or limited data sets that are too limited to support the conclusions. I guess if the AI is rewarded for reaching some predetermined conclusion then no worries.
Comment #113 November 26th, 2023 at 10:49 am
The Brodutch family have been released by Hamas!
I’m sure you’re already aware of this Scott, but I saw this just now on the news and immediately thought of you and Aharon, and just wanted to share the happiness.
I don’t know any of the hostages personally, though like probably every other Israeli, I have over the last month found connections to many of them through shared contacts or other means. (I’ve had random conversations with friends that I haven’t seen in a while, where when I ask them how they are, I learn that members of their family were slaughtered on October 7th.)
Anyway, at least some good news is coming out of this awful, horrible situation.
Comment #114 November 26th, 2023 at 11:34 am
starspawn0 #100:
Ilya Sutskever has made similar comments, saying e.g., that LLMs are computers, probably hinting at their Turing completeness. That requires, that the transformers are run in loops, their output tokens fed back to the input. But there are further necessities for AGI, e.g. efficiency and trainability.
Presumably, there are algorithms requiring a large number of iterations in a loop, which would run very slowly on a transformer, since it generates a token on each iteration, requiring billions of flops each. Other NN architectures, which allow for tighter loops, might be more efficient.
The LLMs are trained by gradient descent. There is likely a large class of functions, which can’t be learned this way. Current LLMs are run as an iterated application of a function on a token sequence. This resembles the procedure used to generate the Mandelbrot set, where a function is iterated and its asymptotic behaviour is observed as a function of the number of iterations. The Mandelbrot set is chaotic, so that slightly different functions lead to wildly different results. In the case of a transformer function, that is repeatedly applied, this would imply a chaotic dependence of the result on the NN weights. A linear approach to training such as gradient descent is likely to fail here.
The current way out of this problem is, that LLMs are trained mostly on a single iteration, i.e. a single token at a time, which keeps them less chaotic. I find it amazing, that this kind of training also works for longer outputs, but I expect the the output to become more chaotic for increasing length. The obstacle to AGI, that I see, is the difficulty to train a complex task, i.e. one, that requires a large number of iterations.
The problem of trainability isn’t new. Solomonov induction and Kolmogorov complexity are conceptually nice pathways to AGI but are completely intractable in practice. Recurrent NNs suffer from vanishing gradients. Transformers have improved this a lot but there are certainly better NN architectures like human brains, which can learn from a much lower number of examples.
Comment #115 November 26th, 2023 at 11:57 am
If the Q* rumors are to be believed, the next breakthrough is in math problem solving, which makes sense as mathematical statements have an objective reality outside of whatever training set is used.
What is the objective reality of the Continuum Hypothesis?
Comment #116 November 26th, 2023 at 12:04 pm
Hey, the latest SMBC alludes to something like your entropy-related theory of consciousness (which I don’t fully grasp but still like better than any alternative):
https://www.smbc-comics.com/comic/entropy-3
Comment #117 November 26th, 2023 at 12:09 pm
Scott P. #115: Without taking a position on either the objectivity of the Continuum Hypothesis or the reports about “Q*”, we can note that there’s an objective, mechanical way to check whether a given formal proof is deductively valid, and that that fact alone might open the door to enormous progress in ML for math even when there’s very little labeled training data.
Comment #118 November 26th, 2023 at 12:11 pm
Edan Maor #113: Yes, I’m happy and relieved, and may the 180 remaining hostages follow soon! I’ll post an update on my blog later today.
Comment #119 November 26th, 2023 at 1:47 pm
Is sam altman currently the official ceo of openai? All the news says he’s back but I haven’t seen an official statement from openai confirming. I assume even as a contractor you would be in a position to know with certainty.
Comment #120 November 26th, 2023 at 1:53 pm
#119: Yes, he’s back.
Comment #121 November 26th, 2023 at 2:49 pm
Scott #117: Assuming that ChatGPT-x learned “to check whether a given formal proof is deductively valid”, how is this immediately pertinent to natural language processing, in particular insofar as “hallucinations” are concerned?
Comment #122 November 26th, 2023 at 4:06 pm
AG #121: Well, you could at least eliminate hallucinations about mathematics, by using RL to train the LLM to only show you informal mathematical reasoning that it had successfully converted into a formal proof that passed verification.
Comment #123 November 26th, 2023 at 4:28 pm
There is no doubt in my mind that if sufficient resources are poured into this project, mathematics will, in effect, be automated, and the likes of you and me will have to contend with the arrival of “meta-mathematicians”. But the amount of money to be made in mathematics per se pales in comparison with similarly complex natural language possibilities (e.g. the rule of law, say). Hence my question #121.
Comment #124 November 26th, 2023 at 4:41 pm
AG #123: It’s an open question how much automating mathematics would help with more lucrative domains. At the very least, though, we could expect it to help a lot with software verification and debugging, which is billions of dollars right there.
Comment #125 November 26th, 2023 at 4:56 pm
Scott #124: If the amount implicit in “billions of dollars right there” sufficiently exceeds the amount implicit in “sufficient resources” in #123, it might well be just a matter of time.
Comment #126 November 26th, 2023 at 8:01 pm
I thought I would mention this paper by François Charton (I saw the talk version of this paper a few weeks ago online) on how Transformers trained on certain types of math problems don’t hallucinate when tested out-of-distribution:
https://arxiv.org/abs/2211.00170
Quote: “This paper investigates the failure cases and out-of-distribution behavior of transformers trained on matrix inversion and eigenvalue decomposition. I show that incorrect model predictions still retain deep mathematical properties of the solution (e.g. correct eigenvalues, unit norm of eigenvectors), and that almost all model failures can be attributed to, and predicted from, properties of the problem or solution. This demonstrates that, when in doubt, math transformers do not hallucinate absurd solutions (as was sometimes proposed) but remain “roughly right”. I also show that the careful choice of a training dataset can accelerate training, while allowing the model to generalize out of its training distribution, invalidating the idea that transformers “merely interpolate” from memorized examples.”
I found out about this work from a tweet a few weeks ago from Chris Szegedy.
Comment #127 November 26th, 2023 at 9:13 pm
starspawn0 #126: Curiouser and curiouser… Spectral decomposition’s consequences, suitably applied and interpreted, are about to engulf us all in no time, apparently.
Comment #128 November 27th, 2023 at 11:17 am
@starspawn0 #90
That is, indeed, trivial. (Not zero, just trivial.)
Comment #129 November 28th, 2023 at 5:42 am
Scott #97,
“I want to know: what theoretical principle that I understand are you using to make your prediction?”
1. It’s a simple observation from theoretical computer science: Interesting/hard problems doesn’t have linear complexity. A(G)I is without doubt a very interesting/ very hard problem.
2. As I said, there is also my “empirical” argument: I’ve been testing chatGPT since day 1 when it became publicly available (this is now a year ago) and so far I have seen no really significant improvement in math/logic performance.
Comment #130 November 28th, 2023 at 7:32 am
Sinclair ZX-81 #129: Your first observation is nonsense on at least four levels. Lots of interesting problems have linear complexity (eg graph planarity), nonlinear complexities can often be handled fine in practice (eg transformer neural nets, which underlie GPT, naturally have quadratic scaling in the length of the context window), asymptotic analysis is only relevant in the first place if you specify an input length, and the human brain itself provides a crude upper bound — huge but finite, and possibly rapidly being approached — on the complexity of a mechanism that could act as a human.
As for your second observation, did you not notice a qualitative leap in math performance from GPT-3.5 to GPT-4? I certainly did. The latter (with math plugins) can reliably do high-school level and some kinds of undergraduate-level math homework, whereas the former can’t.
Comment #131 December 1st, 2023 at 6:52 am
I don’t think I’d describe GPT’s progress in arithmetic as “linear”, though. More “pseudolinear”, since each new version seems to handle numbers with a few more digits.
Comment #132 December 6th, 2023 at 1:11 am
Since this post is titled “Updates!”, here are two miscellaneous updates related to the usual themes of this blog:
I don’t know how much this is in the news on your side of the Atlantic, but the UK just imposed new immigration restrictions, including raising the minimum salary required for a Skilled Worker visa from £26,200 to £38,700. (Or maybe this applies to all lowercase-s skilled workers holding a visa – the news sources I’ve found are unclear.) Anecdotally, my impression is that this is a higher threshold than many UK postdocs earn – I’ll spare you the obvious tirade since I imagine it’s what ChatGPT would say Scott Aaronson’s reaction would be. Other new immigration restrictions include making it harder for visa holders to bring family members, requiring visa holders to pay more for the NHS, and ending an exception that made it easier to hire foreign workers in industries where there’s a shortage of domestic workers. (See e.g. https://www.bbc.co.uk/news/uk-politics-67612106 for a source.)
In happier news, here’s my former PhD advisor Henry Yuen talking about quantum computing on CBS: https://www.youtube.com/watch?v=bffQ9Z_p0Zk