← @bgSponge

mlc-ai/web-llm: High-performance In-browser LLM Inference Engine

Summary

WebLLM is the MLC TypeScript engine for running open-source LLMs entirely in the browser on WebGPU, with no server.

  1. Inference stays on the client.

    WebGPU runs the model in the page; nothing is sent to a server, so privacy and offline use are the default.

    Everything runs inside the browser with no server support and is accelerated with WebGPU.

  2. The OpenAI chat surface is the integration.

    CreateMLCEngine loads a model, then engine.chat.completions.create accepts the same messages, streaming, JSON-mode, and seed fields.

  3. Workers own the model lifecycle.

    A dedicated web worker keeps generation off the UI thread; a service worker plus heartbeat tries to keep weights loaded across page visits.

    You can put the heavy computation in a worker script to optimize your application performance.

  4. MLC artifacts are the extension point.

    A model is a weight URL plus a WASM library; variants can share a library, and custom compiles come from MLC LLM.

  5. Caches and hashes are explicit policy.

    Four cache backends and optional SRI checks on config, WASM, and tokenizer files make storage and integrity a configuration choice.

Read the original ↗Save to your Sponge

Summarized with Sponge. Quotes are the author’s own words.

More from @bg

All readings →