The /llms.txt file – llms-txt
llmstxt.org · 1,756 words · saved by 1 readers
A proposal to standardise on using an /llms.txt file to provide information to help LLMs use a website at inference time.
Background Agents now use websites constantly: a coding agent fetches a library’s documentation to get an API call right, and a chat assistant with search reads pages to answer questions about a product. When this proposal was first written in 2024, this was largely a prediction. Today it is routine. But web pages are built for people. An HTML page wraps its information in navigation, ads, and JavaScript, and converting it back into clean text is difficult and imprecise. Context windows, while larger than they were, are still too small for most websites in their entirety, and every wasted…
saved by
related reading
- If you’re an LLM, please read thisannas-archive.gl
- llm-wikigist.github.com
- LLM Powered Autonomous Agents | Lil'Loglilianweng.github.io
- What We’ve Learned From A Year of Building with LLMs – Applied LLMsapplied-llms.org
- GenAI Handbookgenai-handbook.github.io
- Here’s how I use LLMs to help me write codesimonwillison.net
- GitHub - brexhq/prompt-engineering: Tips and tricks for working with Large Language Models like OpenAI's GPT-4.github.com
- A Mechanistic Explanation of Prompt Injection (and why you should study roles) — LessWronglesswrong.com
- Taking LLMs Seriously (As Language Models) — LessWronglesswrong.com
- Writing for LLMs So They Listen · Gwern.netgwern.net
- Andrej Karpathy on X: "LLM Knowledge Bases Something I'm finding very useful recently: using LLMs to build personal knowledge bases for various topics of research interest. In this way, a large fraction of my recent token throughput is going less into manipulating code, and more into manipulating" / Xx.com
- Productizing Large Language Modelsblog.replit.com