How I think about LLM prompt engineering

They were building a model to embed words in a vector space — a problem that already had a long academic history at the time, starting in the 1980s. Their model used an optimization objective designed to turn correlation relationships between words into distance relationships in the embedding space: a vector was associated to each word in a vocabulary, and the vectors were optimized so that the dot-product (cosine proximity) between vectors representing frequently co-occurring words would be closer to 1, while the dot-product between vectors representing rarely co-occurring would be closer to 0. They found that the resulting embedding space did much more than capture semantic similarity. It featured some form of emergent learning — it was capable of performing “word arithmetic”, something that it had not been trained to do. There existed a vector in the space that could be added to any male noun to obtain a point that would land close to its female equivalent. As in: V(king) - V(man) +

How I think about LLM prompt engineering Prompting as searching through a space of vector programs François Chollet Oct 09, 2023 675 71 Share Flashback: Word2Vec’s emergent word arithmetic In 2013, at Google, Mikolov et al. noticed something remarkable. They were building a model to embed words in a vector space — a problem that already had a long academic history at the time, starting in the 1980s. Their model used an optimization objective designed to turn correlation relationships between words into distance relationships in the embedding space: a vector was associated to each word in a voc

related reading

• What We’ve Learned From A Year of Building with LLMs – Applied LLMsapplied-llms.org
• Patterns for Building LLM-based Systems & Productseugeneyan.com
• Prompt Engineering Guide | Prompt Engineering Guidepromptingguide.ai
• GitHub - brexhq/prompt-engineering: Tips and tricks for working with Large Language Models like OpenAI's GPT-4.github.com
• GenAI Handbookgenai-handbook.github.io
• The Illustrated Word2vec – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• How LLMs Actually Work | 0xkato0xkato.xyz
• A History of Large Language Modelsgregorygundersen.com
• 2402.07927arxiv.org
• Embeddings: What they are and why they mattersimonwillison.net
• Getting Caught Up to Modern LLM Research | Samarth Goeldev.samarthgoel.com
• A Mechanistic Explanation of Prompt Injection (and why you should study roles) — LessWronglesswrong.com