WebLLM

Compatibility

Only available in web environments.

You can run LLMs directly in your web browser using LangChain's WebLLM integration.

Setup

You'll need to install the WebLLM SDK module to communicate with your local model.

tip

See this section for general instructions on installing integration packages.

npm

npm install -S @mlc-ai/web-llm @langchain/community

Yarn

yarn add @mlc-ai/web-llm @langchain/community

pnpm

pnpm add @mlc-ai/web-llm @langchain/community

Usage

Note that the first time a model is called, WebLLM will download the full weights for that model. This can be multiple gigabytes, and may not be possible for all end-users of your application depending on their internet connection and computer specs. While the browser will cache future invocations of that model, we recommend using the smallest possible model you can.

We also recommend using a separate web worker when invoking and loading your models to not block execution.

// Must be run in a web environment, e.g. a web worker

import { ChatWebLLM } from "@langchain/community/chat_models/webllm";
import { HumanMessage } from "@langchain/core/messages";

// Initialize the ChatWebLLM model with the model record and chat options.
// Note that if the appConfig field is set, the list of model records
// must include the selected model record for the engine.

// You can import a list of models available by default here:
// https://github.com/mlc-ai/web-llm/blob/main/src/config.ts
//
// Or by importing it via:
// import { prebuiltAppConfig } from "@mlc-ai/web-llm";
const model = new ChatWebLLM({
  model: "Phi-3-mini-4k-instruct-q4f16_1-MLC",
  chatOptions: {
    temperature: 0.5,
  },
});

await model.initialize((progress: Record<string, unknown>) => {
  console.log(progress);
});

// Call the model with a message and await the response.
const response = await model.invoke([
  new HumanMessage({ content: "What is 1 + 1?" }),
]);

console.log(response);

/*
AIMessage {
  content: ' 2\n',
}
*/

API Reference:

• ChatWebLLM from @langchain/community/chat_models/webllm
• HumanMessage from @langchain/core/messages

Streaming is also supported.

Example

For a full end-to-end example, check out this project.