How I think about LLM prompt engineering

They were building a model to embed words in a vector space — a problem that already had a long academic history at the time, starting in the 1980s. Their model used an optimization objective designed to turn correlation relationships between words into distance relationships in the embedding space: a vector was associated to each word in a vocabulary, and the vectors were optimized so that the dot-product (cosine proximity) between vectors representing frequently co-occurring words would be closer to 1, while the dot-product between vectors representing rarely co-occurring would be closer to 0. They found that the resulting embedding space did much more than capture semantic similarity. It featured some form of emergent learning — it was capable of performing “word arithmetic”, something that it had not been trained to do. There existed a vector in the space that could be added to any male noun to obtain a point that would land close to its female equivalent. As in: V(king) - V(man) +

How I think about LLM prompt engineering Prompting as searching through a space of vector programs François Chollet Oct 09, 2023 675 71 Share Flashback: Word2Vec’s emergent word arithmetic In 2013, at Google, Mikolov et al. noticed something remarkable. They were building a model to embed words in a vector space — a problem that already had a long academic history at the time, starting in the 1980s. Their model used an optimization objective designed to turn correlation relationships between words into distance relationships in the embedding space: a vector was associated to each word in a voc

related reading

• Patterns for Building LLM-based Systems & Productseugeneyan.com
• GenAI Handbookgenai-handbook.github.io
• The Illustrated Word2vec – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• The 2025 AI Engineering Reading List - Latent.Spacelatent.space
• How LLMs Actually Work | 0xkato0xkato.xyz
• Getting Caught Up to Modern LLM Research | Samarth Goeldev.samarthgoel.com
• Embeddings: What they are and why they mattersimonwillison.net
• Building LLM applications for productionhuyenchip.com
• The Illustrated Word2vec – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Understanding Attention in LLMs | Bartosz Milewski's Programming Cafebartoszmilewski.com
• The Foundation Understanding LLMs and Prompt Engineering, and Why It All Mattersleeboonstra.dev
• Prompts for Work & Play: Launching the Wolfram Prompt Repository-Stephen Wolfram Writingswritings.stephenwolfram.com