MLC | WebLLM: A High-Performance In-Browser LLM Inference Engine
We are in an exciting year of generative AI. Open large language models bring significant opportunities to offer customization and domain-specific deployments. While most LLMs run on servers/cloud, there are promising opportunities for running capable models locally. Web browsers form a universally accessible platform, allowing users to effortlessly engage with any web applications without installation processes. Wouldn’t it be amazing to bring open language models directly into browsers, enabling anyone to run LLMs locally just by opening a webpage? In this post, we introduce the WebLLM engine (WebLLM for short), a high-performance in-browser LLM inference engine. WebLLM engine is a new chapter of the MLC-LLM project, providing a specialized web backend of MLCEngine, and offering efficient LLM inference in the browser with local GPU acceleration. WebLLM is fast (native GPU acceleration), private (100% client-side computation), and convenient (zero environment setup). The figure below
MLC | WebLLM: A High-Performance In-Browser LLM Inference Engine --> Home WebLLM: A High-Performance In-Browser LLM Inference Engine Jun 13, 2024 • MLC Community Introduction We are in an exciting year of generative AI. Open large language models bring significant opportunities to offer customization and domain-specific deployments. While most LLMs run on servers/cloud, there are promising opportunities for running capable models locally. Web browsers form a universally accessible platform, allowing users to effortlessly engage with any web applications without installation processes. Wouldn’t
saved by
related reading
- WebLLM | Homemlc.ai
- MLC LLM | Homemlc.ai
- LLM Powered Autonomous Agents | Lil'Loglilianweng.github.io
- ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
- LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
- GitHub - open-webui/open-webui: User-friendly AI Interface (Supports Ollama, OpenAI API, ...)github.com
- KernelBench: Can LLMs Write GPU Kernels?scalingintelligence.stanford.edu
- LLM Engineer's Almanac - Workloads | Modalmodal.com
- LLM Engineer's Almanac - Advisormodal.com
- vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention | vLLM Blogblog.vllm.ai
- Inside vLLM: Anatomy of a High-Throughput LLM Inference Systemvllm.ai
- HeteroLLM: Accelerating Large Language Model Inference on Mobile SoCs with Heterogeneous AI Acceleratorsarxiv.org