WebLLM | Home

This project brings large-language model and LLM-based chatbot to web browsers. Everything runs inside the browser with no server support and accelerated with WebGPU. This opens up a lot of fun opportunities to build AI assistants for everyone and enable privacy while enjoying GPU acceleration. Please check out our GitHub repo to see how we did it. There is also a demo which you can try out. We have been seeing amazing progress in generative AI and LLM recently. Thanks to the open-source efforts like LLaMA, Alpaca, Vicuna and Dolly, we start to see an exciting future of building our own open source language models and personal AI assistant. These models are usually big and compute-heavy. To build a chat service, we will need a large cluster to run an inference server, while clients send requests to servers and retrieve the inference output. We also usually have to run on a specific type of GPUs where popular deep-learning frameworks are readily available. This project is our step to br

WebLLM | Home --> Home GitHub WebLLM: High-Performance In-Browser LLM Inference Engine Get Started Chat with WebLLM Overview We have been seeing amazing progress in generative AI and LLM recently. Thanks to the open-source efforts like LLaMA, Alpaca, Vicuna and Dolly, we start to see an exciting future of building our own open source language models and personal AI assistant. These models are usually big and compute-heavy. To build a chat service, we will need a large cluster to run an inference server, while clients send requests to servers and retrieve the inference output. We also usually h

related reading

• MLC | WebLLM: A High-Performance In-Browser LLM Inference Engineblog.mlc.ai
• LLM Powered Autonomous Agents | Lil'Loglilianweng.github.io
• MLC LLM | Homemlc.ai
• GitHub - open-webui/open-webui: User-friendly AI Interface (Supports Ollama, OpenAI API, ...)github.com
• ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
• vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention | vLLM Blogblog.vllm.ai
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• Introduction to vLLM and PagedAttentionblog.runpod.io
• How is LLaMa.cpp possible?finbarr.ca
• A snapshot of today's open-source LLMs space every builder should know.linkedin.com
• GitHub - open-compass/opencompass: OpenCompass is an LLM evaluation platform, supporting a wide range of models (Llama3, Mistral, InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets.github.com
• Productizing Large Language Modelsblog.replit.com