flâneur

MLC LLM | Home

mlc.ai · 104 words · saved by 1 readers

MLC LLM is a universal solution that allows any language model to be deployed natively on a diverse set of hardware backends and native applications, plus a productive framework for everyone to further optimize model performance for their own use cases. Everything runs locally with no server support and accelerated with local GPUs on your phone and laptop. Check out our GitHub repository to see how we did it. You can also read through instructions below for trying out demos. This section contains the instructions to run large-language models and chatbot natively on your environment. Try out this TestFlight page (limited to the first 9000 users) to install and use our example iOS chat app built for iPhone. Our app itself needs about 4GB of memory to run. Considering the iOS and other running applications, we will need a recent iPhone with 6GB (or more) of memory to run the app. We only tested the application on iPhone 14 Pro Max and iPhone 12 Pro. You can also check out our GitHub repo

Overview MLC LLM is a machine learning compiler and high-performance deployment engine for large language models. The mission of this project is to enable everyone to develop, optimize, and deploy AI models natively on everyone’s platforms. MLC LLM compiles and runs code on MLCEngine – a unified high-performance LLM inference engine across the above platforms. MLCEngine provides OpenAI-compatible API available through REST server, python, javascript, iOS, Android, all backed by the same engine and compiler that we keep improving with the community. Get Started Please visit our…

saved by

related reading