LSTM is dead. Long Live Transformers!

I've just recently been on a journey to understand transformers in and out, here are the resources I've found managed to drill it into my head: 1. Chapters 7, 9, 10 [link] This was really useful to really build up to the concepts of attention (although the actual attention section is still brief). 2. [link] This was great for visualization and understanding that attention wasn't exclusively for transformers and actually was for RNNs first. 3. [link] Getting to understand how transformers actually work visually. 4. [this video] This lecture by Leo Dirac was extremely helpful to finish off with, not only because it actually includes some pseudocode but it also revisits some key topics and covers why transformers are needed. One of the big confusion points for me was that the concept of ATTENTION and SELF-ATTENTION are not the same thing. Hope this helps.

I'm not sure I understand it well enough to say but watching a video on it [0] I think there were a few key points: * "Attention is all you need" introduced positional encoding which allows you to keep context of the word, allowing for more complex translation (and thus generative/chatgpt like tasks?) because words now have context relative to each other. Contrast this with "bag of words" models that only tells you whether the word is present or not. * I don't quite understand why but transformers (which "AiaYN" introduced) can be made fully parallel, compared with the RNN/LSTM networks which has to be serial per token. Fully parallel allows for GPU optimization, which means you can take advantage of Moore's law for training. I'm always a bit suspicious when people claim a breakthrough of this sort. There's no doubt that better algorithms give better results but how much is due to just faster computers, cheaper compute, memory, etc. [0] [this video]