Transformer Taxonomy (the last lit review)

kipply's blog about stuff she does or reads about or observes

This document is my running literature review for people trying to catch up on AI. It covers 22 models, 11 architectural changes, 7 post-pre-training techniques and 3 training techniques (and 5 things that are none of the above). Everything is very loosely in order of importance and somewhat uniqueness. All papers will link to the actual PDF and not the ArXiv page and the selection is mostly curated based on things I know about. Systems/performance and alignment are excluded for this one because they’re my favourite and I’d want to do it more justice. Alignment research is really important,…

related reading

• Getting Caught Up to Modern LLM Research | Samarth Goeldev.samarthgoel.com
• frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
• Large Language Models Reading List | Sebastian Raschka, PhDsebastianraschka.com
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• Five years of GPT progressfinbarr.ca
• GPT in 60 Lines of NumPy | Jay Modyjaykmody.com
• GenAI Handbookgenai-handbook.github.io
• The Big LLM Architecture Comparisonmagazine.sebastianraschka.com
• Pathways Language Model (PaLM): Scaling to 540 Billion Parameters for Breakthrouai.googleblog.com
• ali (@waterloo_intern) on Xx.com
• The Annotated Transformernlp.seas.harvard.edu
• microgptkarpathy.github.io