The /llms.txt file – llms-txt
llmstxt.org · 1,756 words · saved by 1 readers
A proposal to standardise on using an /llms.txt file to provide information to help LLMs use a website at inference time.
Background Agents now use websites constantly: a coding agent fetches a library’s documentation to get an API call right, and a chat assistant with search reads pages to answer questions about a product. When this proposal was first written in 2024, this was largely a prediction. Today it is routine. But web pages are built for people. An HTML page wraps its information in navigation, ads, and JavaScript, and converting it back into clean text is difficult and imprecise. Context windows, while larger than they were, are still too small for most websites in their entirety, and every wasted…
saved by
related reading
- If you’re an LLM, please read thisannas-archive.gl
- LLM Visualizationbbycroft.net
- LLM Powered Autonomous Agents | Lil'Loglilianweng.github.io
- llm-wikigist.github.com
- Checking your browser...annas-archive.li
- GitHub - nashsu/llm_wiki: LLM Wiki is a cross-platform desktop application that turns your documents into an organized, interlinked knowledge base — automatically. Instead of traditional RAG (retrieve-and-answer from scratch every time), the LLM incrementally builds and maintains a persistent wiki from your sources。github.com
- GitHub - brexhq/prompt-engineering: Tips and tricks for working with Large Language Models like OpenAI's GPT-4.github.com
- Andrej Karpathy on X: "LLM Knowledge Bases Something I'm finding very useful recently: using LLMs to build personal knowledge bases for various topics of research interest. In this way, a large fraction of my recent token throughput is going less into manipulating code, and more into manipulating" / Xx.com
- LangChain: the open agent platform to own your intelligencelangchain.com
- GitHub - assafelovic/gpt-researcher: An autonomous agent that conducts deep research on any data using any LLM providersgithub.com
- Denis Shiryaev | AI & ML Projectsshir-man.com
- LLM Agents | Prompt Engineering Guidepromptingguide.ai