mlc-ai/web-llm: High-performance In-browser LLM Inference Engine
Summary
WebLLM is the MLC TypeScript engine for running open-source LLMs entirely in the browser on WebGPU, with no server.
Inference stays on the client.
WebGPU runs the model in the page; nothing is sent to a server, so privacy and offline use are the default.
Everything runs inside the browser with no server support and is accelerated with WebGPU.
The OpenAI chat surface is the integration.
CreateMLCEngine loads a model, then engine.chat.completions.create accepts the same messages, streaming, JSON-mode, and seed fields.
Workers own the model lifecycle.
A dedicated web worker keeps generation off the UI thread; a service worker plus heartbeat tries to keep weights loaded across page visits.
You can put the heavy computation in a worker script to optimize your application performance.
MLC artifacts are the extension point.
A model is a weight URL plus a WASM library; variants can share a library, and custom compiles come from MLC LLM.
Caches and hashes are explicit policy.
Four cache backends and optional SRI checks on config, WASM, and tokenizer files make storage and integrity a configuration choice.
Read the original ↗Save to your Sponge
Summarized with Sponge. Quotes are the author’s own words.