!function(){function t(t){document.documentElement.setAttribute("data-theme",t)}var e=function(){var t=null;try{t=new URLSearchParams(window.location.search).get("docusaurus-theme")}catch(t){}return t}()||function(){var t=null;try{t=localStorage.getItem("theme")}catch(t){}return t}();t(null!==e?e:"light")}(),document.documentElement.setAttribute("data-announcement-bar-initially-dismissed",function(){try{return"true"===localStorage.getItem("docusaurus.announcement.dismiss")}catch(t){}return!1}())

[Skip to main content](#__docusaurus_skipToContent_fallback)

LangChain v0.2 is coming soon! Preview the new docs [here](/v0.2/docs/introduction/).

[

![🦜️🔗 Langchain](/v0.1/img/brand/wordmark.png)![🦜️🔗 Langchain](/v0.1/img/brand/wordmark-dark.png)

](/v0.1/)[Docs](/v0.1/docs/get_started/introduction/)[Use cases](/v0.1/docs/use_cases/)[Integrations](/v0.1/docs/integrations/platforms/)[API Reference](https://api.js.langchain.com)

[More](#)

*   [People](/v0.1/docs/people/)
*   [Community](/v0.1/docs/community/)
*   [Tutorials](/v0.1/docs/additional_resources/tutorials/)
*   [Contributing](/v0.1/docs/contributing/)

[v0.1](#)

*   [v0.2](https://js.langchain.com/v0.2/docs/introduction)
*   [v0.1](/v0.1/docs/get_started/introduction/)

[🦜🔗](#)

*   [LangSmith](https://smith.langchain.com)
*   [LangSmith Docs](https://docs.smith.langchain.com)
*   [LangChain Hub](https://smith.langchain.com/hub)
*   [LangServe](https://github.com/langchain-ai/langserve)
*   [Python Docs](https://python.langchain.com/)

[Chat](https://chatjs.langchain.com)[](https://github.com/langchain-ai/langchainjs)

Search

*   [Get started](/v0.1/docs/get_started/)
    
    *   [Introduction](/v0.1/docs/get_started/introduction/)
    *   [Installation](/v0.1/docs/get_started/installation/)
    *   [Quickstart](/v0.1/docs/get_started/quickstart/)
*   [LangChain Expression Language](/v0.1/docs/expression_language/)
    
    *   [Get started](/v0.1/docs/expression_language/get_started/)
    *   [Why use LCEL?](/v0.1/docs/expression_language/why/)
    *   [Interface](/v0.1/docs/expression_language/interface/)
    *   [Streaming](/v0.1/docs/expression_language/streaming/)
    *   [How to](/v0.1/docs/expression_language/how_to/routing/)
        
    *   [Cookbook](/v0.1/docs/expression_language/cookbook/)
        
    *   [LangChain Expression Language (LCEL)](/v0.1/docs/expression_language/)
*   [Modules](/v0.1/docs/modules/)
    
    *   [Model I/O](/v0.1/docs/modules/model_io/)
        
        *   [Quickstart](/v0.1/docs/modules/model_io/quick_start/)
        *   [Concepts](/v0.1/docs/modules/model_io/concepts/)
        *   [Prompts](/v0.1/docs/modules/model_io/prompts/)
            
        *   [LLMs](/v0.1/docs/modules/model_io/llms/)
            
            *   [Quick Start](/v0.1/docs/modules/model_io/llms/quick_start/)
            *   [Streaming](/v0.1/docs/modules/model_io/llms/streaming_llm/)
            *   [Caching](/v0.1/docs/modules/model_io/llms/llm_caching/)
            *   [Custom LLM](/v0.1/docs/modules/model_io/llms/custom_llm/)
            *   [Tracking token usage](/v0.1/docs/modules/model_io/llms/token_usage_tracking/)
            *   [Cancelling requests](/v0.1/docs/modules/model_io/llms/cancelling_requests/)
            *   [Dealing with API Errors](/v0.1/docs/modules/model_io/llms/dealing_with_api_errors/)
            *   [Dealing with Rate Limits](/v0.1/docs/modules/model_io/llms/dealing_with_rate_limits/)
            *   [Subscribing to events](/v0.1/docs/modules/model_io/llms/subscribing_events/)
            *   [Adding a timeout](/v0.1/docs/modules/model_io/llms/timeouts/)
        *   [Chat Models](/v0.1/docs/modules/model_io/chat/)
            
        *   [Model I/O](/v0.1/docs/modules/model_io/)
        *   [Output Parsers](/v0.1/docs/modules/model_io/output_parsers/)
            
    *   [Retrieval](/v0.1/docs/modules/data_connection/)
        
    *   [Chains](/v0.1/docs/modules/chains/)
    *   [Agents](/v0.1/docs/modules/agents/)
        
    *   [More](/v0.1/docs/modules/memory/)
        
*   [Security](/v0.1/docs/security/)
*   [Guides](/v0.1/docs/guides/)
    
*   [Ecosystem](/v0.1/docs/ecosystem/)
    
*   [LangGraph](/v0.1/docs/langgraph/)
*   * * *
    

*   [](/v0.1/)
*   [Modules](/v0.1/docs/modules/)
*   [Model I/O](/v0.1/docs/modules/model_io/)
*   [LLMs](/v0.1/docs/modules/model_io/llms/)
*   Custom LLM

On this page

Custom LLM
==========

This notebook goes over how to create a custom LLM wrapper, in case you want to use your own LLM or a different wrapper than one that is directly supported in LangChain.

There are a few required things that a custom LLM needs to implement after extending the [`LLM` class](https://api.js.langchain.com/classes/langchain_core_language_models_llms.LLM.html):

*   A `_call` method that takes in a string and call options (which includes things like `stop` sequences), and returns a string.
*   A `_llmType` method that returns a string. Used for logging purposes only.

You can also implement the following optional method:

*   A `_streamResponseChunks` method that returns an `AsyncIterator` and yields [`GenerationChunks`](https://api.js.langchain.com/classes/langchain_core_outputs.GenerationChunk.html). This allows the LLM to support streaming outputs.

Let’s implement a very simple custom LLM that just echoes back the first `n` characters of the input.

    import { LLM, type BaseLLMParams } from "@langchain/core/language_models/llms";import type { CallbackManagerForLLMRun } from "langchain/callbacks";import { GenerationChunk } from "langchain/schema";export interface CustomLLMInput extends BaseLLMParams {  n: number;}export class CustomLLM extends LLM {  n: number;  constructor(fields: CustomLLMInput) {    super(fields);    this.n = fields.n;  }  _llmType() {    return "custom";  }  async _call(    prompt: string,    options: this["ParsedCallOptions"],    runManager: CallbackManagerForLLMRun  ): Promise<string> {    // Pass `runManager?.getChild()` when invoking internal runnables to enable tracing    // await subRunnable.invoke(params, runManager?.getChild());    return prompt.slice(0, this.n);  }  async *_streamResponseChunks(    prompt: string,    options: this["ParsedCallOptions"],    runManager?: CallbackManagerForLLMRun  ): AsyncGenerator<GenerationChunk> {    // Pass `runManager?.getChild()` when invoking internal runnables to enable tracing    // await subRunnable.invoke(params, runManager?.getChild());    for (const letter of prompt.slice(0, this.n)) {      yield new GenerationChunk({        text: letter,      });      // Trigger the appropriate callback      await runManager?.handleLLMNewToken(letter);    }  }}

We can now use this as any other LLM:

    const llm = new CustomLLM({ n: 4 });await llm.invoke("I am an LLM");

    I am

And support streaming:

    const stream = await llm.stream("I am an LLM");for await (const chunk of stream) {  console.log(chunk);}

    Iam

Richer outputs[​](#richer-outputs "Direct link to Richer outputs")
------------------------------------------------------------------

If you want to take advantage of LangChain's callback system for functionality like token tracking, you can extend the [`BaseLLM`](https://api.js.langchain.com/classes/langchain_core_language_models_llms.BaseLLM.html) class and implement the lower level `_generate` method. Rather than taking a single string as input and a single string output, it can take multiple input strings and map each to multiple string outputs. Additionally, it returns a `Generation` output with fields for additional metadata rather than just a string.

    import { CallbackManagerForLLMRun } from "@langchain/core/callbacks/manager";import { LLMResult } from "@langchain/core/outputs";import {  BaseLLM,  BaseLLMCallOptions,  BaseLLMParams,} from "@langchain/core/language_models/llms";export interface AdvancedCustomLLMCallOptions extends BaseLLMCallOptions {}export interface AdvancedCustomLLMParams extends BaseLLMParams {  n: number;}export class AdvancedCustomLLM extends BaseLLM<AdvancedCustomLLMCallOptions> {  n: number;  constructor(fields: AdvancedCustomLLMParams) {    super(fields);    this.n = fields.n;  }  _llmType() {    return "advanced_custom_llm";  }  async _generate(    inputs: string[],    options: this["ParsedCallOptions"],    runManager?: CallbackManagerForLLMRun  ): Promise<LLMResult> {    const outputs = inputs.map((input) => input.slice(0, this.n));    // Pass `runManager?.getChild()` when invoking internal runnables to enable tracing    // await subRunnable.invoke(params, runManager?.getChild());    // One input could generate multiple outputs.    const generations = outputs.map((output) => [      {        text: output,        // Optional additional metadata for the generation        generationInfo: { outputCount: 1 },      },    ]);    const tokenUsage = {      usedTokens: this.n,    };    return {      generations,      llmOutput: { tokenUsage },    };  }}

This will pass the additional returned information in callback events and in the \`streamEvents method:

    const llm = new AdvancedCustomLLM({ n: 4 });const eventStream = await llm.streamEvents("I am an LLM", {  version: "v1",});for await (const event of eventStream) {  if (event.event === "on_llm_end") {    console.log(JSON.stringify(event, null, 2));  }}

    {  "event": "on_llm_end",  "name": "AdvancedCustomLLM",  "run_id": "a883a705-c651-4236-8095-cb515e2d4885",  "tags": [],  "metadata": {},  "data": {    "output": {      "generations": [        [          {            "text": "I am",            "generationInfo": {              "outputCount": 1            }          }        ]      ],      "llmOutput": {        "tokenUsage": {          "usedTokens": 4        }      }    }  }}

* * *

#### Help us out by providing feedback on this documentation page:

[

Previous

Caching

](/v0.1/docs/modules/model_io/llms/llm_caching/)[

Next

Tracking token usage

](/v0.1/docs/modules/model_io/llms/token_usage_tracking/)

*   [Richer outputs](#richer-outputs)

Community

*   [Discord](https://discord.gg/cU2adEyC7w)
*   [Twitter](https://twitter.com/LangChainAI)

GitHub

*   [Python](https://github.com/langchain-ai/langchain)
*   [JS/TS](https://github.com/langchain-ai/langchainjs)

More

*   [Homepage](https://langchain.com)
*   [Blog](https://blog.langchain.dev)

Copyright © 2024 LangChain, Inc.