Let's build GPT: from scratch, in code, spelled out.

Andrej Karpathy, the director of AI at tesla, published an AMAZING video about how the current models are built which I think everyone in the tech space should watch. It is intuitive, easy to follow and quite frankly the best video I have seen on the topic. Here is the link: [this video] If you conclude, after watching this, that this is all intelligence is - namely cascading optimization and data structure problems chained in a row, then you and a fair number of people here wont ever find common ground. That is not to say that you're wrong, it just seems as if there is a missing component here that we haven't yet discovered.

I think you are taking a very major assumption that LLMs are deterministic infact they are exactly opposite. They are probabilistic systems. - They do not transpile Rough English to Deterministic English. Infact they do not do any transpiling at all. - LLMs learn the probabilities of words in the dataset for all contexts from that dataset (context is a ordered set of words). This is called training. - Once training is done, LLMs can generate text given a prompt and the probabilities that it has learned. The analogy of LLMs being auto-complete on steroids is very apt. - Whether the text generated by LLMs is factual or not is purely coincidental. I would highly recommend watching and working though the code (if possible) of NanoGPT by Andrej Karpathy. [this video] A lot of things which seem like magic about LLMs will get demystified. Now one can argue that LLMs are showing human like reasoning/intelligence/sentience as an emergent behavior. This is hard to argue against because all these terms are extremely hard to define. IMO, the only emergent behavior that LLMs are showing is the output they generate looks like it might have been generated by a human which should not be surprising given that LLMs like ChatGPT has trained on a large amount of human written text available on internet.

I agree that Karpathy's YouTube video is an excellent resource for understanding Transformers from scratch. It provides a hands-on experience that can be particularly helpful for those who want to implement the models themselves. Here's the link to the video titled "Let's build GPT: from scratch, in code, spelled out": [this video] Additionally, for more comprehensive resources on Transformers, you may find these resources useful: * The Illustrated Transformer by Jay Alammar: [link] * MIT 6.S191: Recurrent Neural Networks, Transformers, and Attention: [link] * Karpathy's course, Deep Learning and Generative Models (Lecture 6 covers Transformers): [link] ... These resources cover different aspects of Transformers and can help you grasp the underlying concepts and mechanisms better.

That's... my point. You're agreeing with me. Siri and Alexa are bad . Very, very bad. They pretend to understand spoken English, but they don't , because they're just a huge set of hard-coded rules written out one-at-a-time by enormous numbers of very expensive developers. This is the 1990s approach to AI: Fuzzy logic, heuristics, solvers, dynamic programming, etc... That approach has been thoroughly blown out of the water by Transformers, which does all of that and much more with a thousand lines of code that can be banged out in a couple of hours by one guy while talking in a YouTube video: [this video] Transformers will revolutionise this entire space, and more that don't even exist yet as a market. Take for example humanoid robots: Boston Dynamics has had the hardware working well enough for a decade, but not the software . You can't walk up to one of their robots, point at something, and tell the robot to complete a task. It can't understand what it is seeing, and can't understand English instructions. Programming that in with traditional AI methods would take man-millenia of effort, and might never work well enough . If we could speed up GPT-4o (the one with vision) to just 10x or 50x its current speed, with some light fine-tuning it could control a humanoid robot right now with a level of understanding comparable to C3P0 from Star Wars!