No Priors Ep. 80 | With Andrej Karpathy from OpenAI and Tesla

This benefits actually everyone. If our combined creative work until this point is what turns out to be necessary to kick-start a great shot at abundance (and if you do not believe that, if it's all for nothing, why care at all about the money wasted on models?) it might simply be our societal moral obligation to endorse it -- just as is will be the model creators moral obligation to uphold their end of this deal. Interestingly, Andrej Karpathy recently described the data we are debating as more or less undesirable to build a better LLM and accidentally good enough to have made it work so far ( [this video] ). We'll see about that.

Karpathy himself believes that neural networks are perfectly plausible as a key component to AGI. He has said that it doesn't need to be superseded by something better, it's just that everything else around it (especially infrastructure) needs to improve. As one of the most valuable opinions in the entire world on the subject, I tend to trust what he said. source: [this video] I think the consensus of people with the most valid opinions is that AGI is actually not that far off, definitely in our lifetimes, and likely in the next decade. I'm personally not concerned if we recreate a human in a computer, in fact we might be better off not doing that. If we can replace nearly all manual labor, all software development, and all driving, then I would be really happy with the state of automation of "AGI" (societal ramifications aside).

Right. In a recent YouTube interview Karpathy claimed that LLMs have a lot more "working memory" than a human: [this video] What I assume he's talking about is internal activations such as stored in KV cache that have same lifetime as tokens in the input, but this really isn't the same as "working memory" since these are tied to the input and don't change. What it seems an LLM would need to do better at these sort of iterative/sequencing tasks would be a real working memory that had more arbitrary task-duration lifetime and could be updated (vs fixed KV cache), and would allow it to track progress or more generally maintain context (english usage - not LLM) over the course of a task. I'm a bit surprised that this type of working memory hasn't been added to the transformer architecture. It seems it could be as simple as a fixed (non shifting) region of the context that the LLM could learn to read/write during training to assist on these types of task. An alternative to having embeddings as working memory is to use an external file of text (cf a TODO list, or working notes) for this purpose which is apparently what Claude Code uses to maintain focus over long periods of time, and I recently saw mentioned that the Claude model itself has been trained to use read/write to this sort of text memory file.