Jim Keller: Moore’s Law is Not Dead

There's a good talk by him about how Moore's law isn't dead: [this video] . Obviously, people have different ideas about what they mean when they say "Moore's law is/is not dead", but I appreciate Jim's point that even if the current innovation curves we've been riding are slowing down, there are other innovation curves that we can take advantage of, and that the combination these curves mean that there's still plenty of innovation and improvement to be made in processor design. Of course, the question of whether the x86 architecture will be the foundation for that future improvement remains to be seen. ARM and RISC-V are both eating x86 from below, and the fact that both of the processor architectures allow for greater competition among processor implementations suggests to me that one (or both) may catch up with x86 at some point in the future.

Here's a random collection of thoughts that I have about the progress of Artificial Intelligence (AI): Machine learning (ML) and Deep Learning (DL) in particular benefit from fast computer chips. The most impressive gains in AI in the 2010s (Computer Vision & Natural Language Processing) were made thanks to DL. There's a famous post by AI researcher Rich Sutton, which summarises this fairly neatly [1]. Now, this connection between DL and fast chips would tie the progress of AI pretty tightly to the progress of Moore's Law [2]. There's some compelling evidence that Moore's Law is at least slowing down [3]. On the other hand, there are industry experts like Jim Keller who pretty strongly disagree with these assessments [4] and even TSMC seems to be bullish on being able to keep up with Moore's Law [5]. Some estimates of GPT-3s training cost put it in the range of ~5-10 million dollars [6]. It's hard to say how big of an impact on the economy GPT-3 will have. It's probably safe to assume though, that OpenAI is already working on GPT-4. The jump in parameter sizes from GPT-2 to 3 was roughly 100x (1.5 billion vs. 175 billion) and I'm assuming the price of training the model increased in roughly the same proportion (I might be wrong here and if anyone can point me to evidence on this, it would be much appreciated). With these assumptions, and provided that GPT-4 won't be affected by diminishing returns of adding more parameters (big if), the price for it would be somewhere between

We are ridiculously far from physical limits in our current artificial computers (both theoretical [1], and practical [2]). For more technical details see Jim Keller: [3] [4] [5]. [1] [link] [2] The ~12 watts computer inside each living human adult skull (and perhaps each eukaryote cell [6]) is still the state-of-the-art, for quite some time. [3] 2021, Jim Keller: The Secret to Moore's Law , [link] [4] 2019, Jim Keller: Moore’s Law is Not Dead , [this video] [5] 2023, Change w/ Jim Keller , [link] [6] Our computers aren't yet able of polycomputation, where the computation topology, data, and functions depend on the observer, instead of computation in a passive implementation, once done forever set in s̶t̶o̶n̶e̶ silicon, 2023, Michael Levin, Agency, Attractors, & Observer-Dependent Computation in Biology & Beyond , [link]