!function(){function t(t){document.documentElement.setAttribute("data-theme",t)}var e=function(){var t=null;try{t=new URLSearchParams(window.location.search).get("docusaurus-theme")}catch(t){}return t}()||function(){var t=null;try{t=localStorage.getItem("theme")}catch(t){}return t}();t(null!==e?e:"light")}(),document.documentElement.setAttribute("data-announcement-bar-initially-dismissed",function(){try{return"true"===localStorage.getItem("docusaurus.announcement.dismiss")}catch(t){}return!1}())

[Skip to main content](#__docusaurus_skipToContent_fallback)

You are viewing the **preview** v0.2 docs. View the **stable** v0.1 docs [here](/v0.1/docs/get_started/introduction/). Leave feedback on the v0.2 docs [here](https://github.com/langchain-ai/langchainjs/discussions/5386).

[

![🦜️🔗 Langchain](/v0.2/img/brand/wordmark.png)![🦜️🔗 Langchain](/v0.2/img/brand/wordmark-dark.png)

](/v0.2/)[Integrations](/v0.2/docs/integrations/platforms/)[API Reference](https://v02.api.js.langchain.com)

[More](#)

*   [People](/v0.2/docs/people/)
*   [Community](/v0.2/docs/community)
*   [Tutorials](/v0.2/docs/additional_resources/tutorials)
*   [Contributing](/v0.2/docs/contributing)

[v0.2](#)

*   [v0.2](/v0.2/docs/introduction)
*   [v0.1](https://js.langchain.com/v0.1/docs/get_started/introduction)

[🦜🔗](#)

*   [LangSmith](https://smith.langchain.com)
*   [LangSmith Docs](https://docs.smith.langchain.com)
*   [LangChain Hub](https://smith.langchain.com/hub)
*   [LangServe](https://github.com/langchain-ai/langserve)
*   [Python Docs](https://python.langchain.com/)

[Chat](https://chatjs.langchain.com)[](https://github.com/langchain-ai/langchainjs)

Search

*   [Introduction](/v0.2/docs/introduction)
*   [Tutorials](/v0.2/docs/tutorials/)
    
    *   [Build a Question Answering application over a Graph Database](/v0.2/docs/tutorials/graph)
    *   [Tutorials](/v0.2/docs/tutorials/)
    *   [Build a Simple LLM Application](/v0.2/docs/tutorials/llm_chain)
    *   [Build a Query Analysis System](/v0.2/docs/tutorials/query_analysis)
    *   [Build a Chatbot](/v0.2/docs/tutorials/chatbot)
    *   [Build an Agent](/v0.2/docs/tutorials/agents)
    *   [Build an Extraction Chain](/v0.2/docs/tutorials/extraction)
    *   [Summarize Text](/v0.2/docs/tutorials/summarization)
    *   [Tagging](/v0.2/docs/tutorials/classification)
    *   [Build a Local RAG Application](/v0.2/docs/tutorials/local_rag)
    *   [Conversational RAG](/v0.2/docs/tutorials/qa_chat_history)
    *   [Build a Retrieval Augmented Generation (RAG) App](/v0.2/docs/tutorials/rag)
    *   [Build a Question/Answering system over SQL data](/v0.2/docs/tutorials/sql_qa)
*   [How-to guides](/v0.2/docs/how_to/)
    
    *   [How-to guides](/v0.2/docs/how_to/)
    *   [How to use example selectors](/v0.2/docs/how_to/example_selectors)
    *   [Installation](/v0.2/docs/how_to/installation)
    *   [How to stream responses from an LLM](/v0.2/docs/how_to/streaming_llm)
    *   [How to stream chat model responses](/v0.2/docs/how_to/chat_streaming)
    *   [How to embed text data](/v0.2/docs/how_to/embed_text)
    *   [How to use few shot examples in chat models](/v0.2/docs/how_to/few_shot_examples_chat)
    *   [How to cache model responses](/v0.2/docs/how_to/llm_caching)
    *   [How to cache chat model responses](/v0.2/docs/how_to/chat_model_caching)
    *   [How to create a custom LLM class](/v0.2/docs/how_to/custom_llm)
    *   [How to use few shot examples](/v0.2/docs/how_to/few_shot_examples)
    *   [How to use output parsers to parse an LLM response into structured format](/v0.2/docs/how_to/output_parser_structured)
    *   [How to return structured data from a model](/v0.2/docs/how_to/structured_output)
    *   [How to add ad-hoc tool calling capability to LLMs and Chat Models](/v0.2/docs/how_to/tools_prompting)
    *   [How to create a custom chat model class](/v0.2/docs/how_to/custom_chat)
    *   [How to do per-user retrieval](/v0.2/docs/how_to/qa_per_user)
    *   [How to track token usage](/v0.2/docs/how_to/chat_token_usage_tracking)
    *   [How to track token usage](/v0.2/docs/how_to/llm_token_usage_tracking)
    *   [How to pass through arguments from one step to the next](/v0.2/docs/how_to/passthrough)
    *   [How to compose prompts together](/v0.2/docs/how_to/prompts_composition)
    *   [How to use legacy LangChain Agents (AgentExecutor)](/v0.2/docs/how_to/agent_executor)
    *   [How to add values to a chain's state](/v0.2/docs/how_to/assign)
    *   [How to attach runtime arguments to a Runnable](/v0.2/docs/how_to/binding)
    *   [How to cache embedding results](/v0.2/docs/how_to/caching_embeddings)
    *   [How to split by character](/v0.2/docs/how_to/character_text_splitter)
    *   [How to manage memory](/v0.2/docs/how_to/chatbots_memory)
    *   [How to do retrieval](/v0.2/docs/how_to/chatbots_retrieval)
    *   [How to use tools](/v0.2/docs/how_to/chatbots_tools)
    *   [How to split code](/v0.2/docs/how_to/code_splitter)
    *   [How to do retrieval with contextual compression](/v0.2/docs/how_to/contextual_compression)
    *   [How to write a custom retriever class](/v0.2/docs/how_to/custom_retriever)
    *   [How to create custom Tools](/v0.2/docs/how_to/custom_tools)
    *   [How to debug your LLM apps](/v0.2/docs/how_to/debugging)
    *   [How to load CSV data](/v0.2/docs/how_to/document_loader_csv)
    *   [How to write a custom document loader](/v0.2/docs/how_to/document_loader_custom)
    *   [How to load data from a directory](/v0.2/docs/how_to/document_loader_directory)
    *   [How to load PDF files](/v0.2/docs/how_to/document_loader_pdf)
    *   [How to load JSON data](/v0.2/docs/how_to/document_loaders_json)
    *   [How to select examples by length](/v0.2/docs/how_to/example_selectors_length_based)
    *   [How to select examples by similarity](/v0.2/docs/how_to/example_selectors_similarity)
    *   [How to use reference examples](/v0.2/docs/how_to/extraction_examples)
    *   [How to handle long text](/v0.2/docs/how_to/extraction_long_text)
    *   [How to do extraction without using function calling](/v0.2/docs/how_to/extraction_parse)
    *   [Fallbacks](/v0.2/docs/how_to/fallbacks)
    *   [Few Shot Prompt Templates](/v0.2/docs/how_to/few_shot)
    *   [How to run custom functions](/v0.2/docs/how_to/functions)
    *   [How to construct knowledge graphs](/v0.2/docs/how_to/graph_constructing)
    *   [How to map values to a database](/v0.2/docs/how_to/graph_mapping)
    *   [How to improve results with prompting](/v0.2/docs/how_to/graph_prompting)
    *   [How to add a semantic layer over the database](/v0.2/docs/how_to/graph_semantic)
    *   [How to reindex data to keep your vectorstore in-sync with the underlying data source](/v0.2/docs/how_to/indexing)
    *   [How to get log probabilities](/v0.2/docs/how_to/logprobs)
    *   [How to add message history](/v0.2/docs/how_to/message_history)
    *   [How to generate multiple embeddings per document](/v0.2/docs/how_to/multi_vector)
    *   [How to generate multiple queries to retrieve data for](/v0.2/docs/how_to/multiple_queries)
    *   [How to parse JSON output](/v0.2/docs/how_to/output_parser_json)
    *   [How to retry when output parsing errors occur](/v0.2/docs/how_to/output_parser_retry)
    *   [How to parse XML output](/v0.2/docs/how_to/output_parser_xml)
    *   [How to invoke runnables in parallel](/v0.2/docs/how_to/parallel)
    *   [How to retrieve the whole document for a chunk](/v0.2/docs/how_to/parent_document_retriever)
    *   [How to partially format prompt templates](/v0.2/docs/how_to/prompts_partial)
    *   [How to add chat history to a question-answering chain](/v0.2/docs/how_to/qa_chat_history_how_to)
    *   [How to return citations](/v0.2/docs/how_to/qa_citations)
    *   [How to return sources](/v0.2/docs/how_to/qa_sources)
    *   [How to stream from a question-answering chain](/v0.2/docs/how_to/qa_streaming)
    *   [How to construct filters](/v0.2/docs/how_to/query_constructing_filters)
    *   [How to add examples to the prompt](/v0.2/docs/how_to/query_few_shot)
    *   [How to deal with high cardinality categorical variables](/v0.2/docs/how_to/query_high_cardinality)
    *   [How to handle multiple queries](/v0.2/docs/how_to/query_multiple_queries)
    *   [How to handle multiple retrievers](/v0.2/docs/how_to/query_multiple_retrievers)
    *   [How to handle cases where no queries are generated](/v0.2/docs/how_to/query_no_queries)
    *   [How to recursively split text by characters](/v0.2/docs/how_to/recursive_text_splitter)
    *   [How to reduce retrieval latency](/v0.2/docs/how_to/reduce_retrieval_latency)
    *   [How to route execution within a chain](/v0.2/docs/how_to/routing)
    *   [How to chain runnables](/v0.2/docs/how_to/sequence)
    *   [How to split text by tokens](/v0.2/docs/how_to/split_by_token)
    *   [How to deal with large databases](/v0.2/docs/how_to/sql_large_db)
    *   [How to use prompting to improve results](/v0.2/docs/how_to/sql_prompting)
    *   [How to do query validation](/v0.2/docs/how_to/sql_query_checking)
    *   [How to stream](/v0.2/docs/how_to/streaming)
    *   [How to create a time-weighted retriever](/v0.2/docs/how_to/time_weighted_vectorstore)
    *   [How to use a chat model to call tools](/v0.2/docs/how_to/tool_calling)
    *   [How to call tools with multi-modal data](/v0.2/docs/how_to/tool_calls_multi_modal)
    *   [How to use LangChain tools](/v0.2/docs/how_to/tools_builtin)
    *   [How use a vector store to retrieve data](/v0.2/docs/how_to/vectorstore_retriever)
    *   [How to create and query vector stores](/v0.2/docs/how_to/vectorstores)
*   [Conceptual guide](/v0.2/docs/concepts)
*   Ecosystem
    
    *   [🦜🛠️ LangSmith](/v0.2/docs/langsmith/)
        
    *   [🦜🕸️LangGraph.js](/v0.2/docs/langgraph)
*   Versions
    
    *   [Overview](/v0.2/docs/versions/overview)
    *   [v0.2](/v0.2/docs/versions/v0_2)
    *   [Release Policy](/v0.2/docs/versions/release_policy)
    *   [Packages](/v0.2/docs/versions/packages)
*   [Security](/v0.2/docs/security)

*   [](/v0.2/)
*   [How-to guides](/v0.2/docs/how_to/)
*   Fallbacks

On this page

Fallbacks
=========

Prerequisites

This guide assumes familiarity with the following concepts:

*   [LangChain Expression Language (LCEL)](/v0.2/docs/concepts/#langchain-expression-language)
*   [Chaining runnables](/v0.2/docs/how_to/sequence/)

When working with language models, you may encounter issues from the underlying APIs, e.g. rate limits or downtime. As you move your LLM applications into production it becomes more and more important to have contingencies for errors. That's why we've introduced the concept of fallbacks.

Crucially, fallbacks can be applied not only on the LLM level but on the whole runnable level. This is important because often times different models require different prompts. So if your call to OpenAI fails, you don't just want to send the same prompt to Anthropic - you probably want want to use e.g. a different prompt template.

Handling LLM API errors[​](#handling-llm-api-errors "Direct link to Handling LLM API errors")
---------------------------------------------------------------------------------------------

This is maybe the most common use case for fallbacks. A request to an LLM API can fail for a variety of reasons - the API could be down, you could have hit a rate limit, or any number of things.

**IMPORTANT:** By default, many of LangChain's LLM wrappers catch errors and retry. You will most likely want to turn those off when working with fallbacks. Otherwise the first wrapper will keep on retrying rather than failing.

tip

See [this section for general instructions on installing integration packages](/v0.2/docs/how_to/installation#installing-integration-packages).

*   npm
*   Yarn
*   pnpm

    npm install @langchain/anthropic @langchain/openai

    yarn add @langchain/anthropic @langchain/openai

    pnpm add @langchain/anthropic @langchain/openai

    import { ChatOpenAI } from "@langchain/openai";import { ChatAnthropic } from "@langchain/anthropic";// Use a fake model name that will always throw an errorconst fakeOpenAIModel = new ChatOpenAI({  model: "potato!",  maxRetries: 0,});const anthropicModel = new ChatAnthropic({});const modelWithFallback = fakeOpenAIModel.withFallbacks({  fallbacks: [anthropicModel],});const result = await modelWithFallback.invoke("What is your name?");console.log(result);/*  AIMessage {    content: ' My name is Claude. I was created by Anthropic.',    additional_kwargs: {}  }*/

#### API Reference:

*   [ChatOpenAI](https://v02.api.js.langchain.com/classes/langchain_openai.ChatOpenAI.html) from `@langchain/openai`
*   [ChatAnthropic](https://v02.api.js.langchain.com/classes/langchain_anthropic.ChatAnthropic.html) from `@langchain/anthropic`

Fallbacks for RunnableSequences[​](#fallbacks-for-runnablesequences "Direct link to Fallbacks for RunnableSequences")
---------------------------------------------------------------------------------------------------------------------

We can also create fallbacks for sequences, that are sequences themselves. Here we do that with two different models: ChatOpenAI and then normal OpenAI (which does not use a chat model). Because OpenAI is NOT a chat model, you likely want a different prompt.

    import { ChatOpenAI, OpenAI } from "@langchain/openai";import { StringOutputParser } from "@langchain/core/output_parsers";import { ChatPromptTemplate, PromptTemplate } from "@langchain/core/prompts";const chatPrompt = ChatPromptTemplate.fromMessages<{ animal: string }>([  [    "system",    "You're a nice assistant who always includes a compliment in your response",  ],  ["human", "Why did the {animal} cross the road?"],]);// Use a fake model name that will always throw an errorconst fakeOpenAIChatModel = new ChatOpenAI({  model: "potato!",  maxRetries: 0,});const prompt =  PromptTemplate.fromTemplate(`Instructions: You should always include a compliment in your response.Question: Why did the {animal} cross the road?Answer:`);const openAILLM = new OpenAI({});const outputParser = new StringOutputParser();const badChain = chatPrompt.pipe(fakeOpenAIChatModel).pipe(outputParser);const goodChain = prompt.pipe(openAILLM).pipe(outputParser);const chain = badChain.withFallbacks({  fallbacks: [goodChain],});const result = await chain.invoke({  animal: "dragon",});console.log(result);/*  I don't know, but I'm sure it was an impressive sight. You must have a great imagination to come up with such an interesting question!*/

#### API Reference:

*   [ChatOpenAI](https://v02.api.js.langchain.com/classes/langchain_openai.ChatOpenAI.html) from `@langchain/openai`
*   [OpenAI](https://v02.api.js.langchain.com/classes/langchain_openai.OpenAI.html) from `@langchain/openai`
*   [StringOutputParser](https://v02.api.js.langchain.com/classes/langchain_core_output_parsers.StringOutputParser.html) from `@langchain/core/output_parsers`
*   [ChatPromptTemplate](https://v02.api.js.langchain.com/classes/langchain_core_prompts.ChatPromptTemplate.html) from `@langchain/core/prompts`
*   [PromptTemplate](https://v02.api.js.langchain.com/classes/langchain_core_prompts.PromptTemplate.html) from `@langchain/core/prompts`

Handling long inputs[​](#handling-long-inputs "Direct link to Handling long inputs")
------------------------------------------------------------------------------------

One of the big limiting factors of LLMs in their context window. Sometimes you can count and track the length of prompts before sending them to an LLM, but in situations where that is hard/complicated you can fallback to a model with longer context length.

    import { ChatOpenAI } from "@langchain/openai";// Use a model with a shorter context windowconst shorterLlm = new ChatOpenAI({  model: "gpt-3.5-turbo",  maxRetries: 0,});const longerLlm = new ChatOpenAI({  model: "gpt-3.5-turbo-16k",});const modelWithFallback = shorterLlm.withFallbacks({  fallbacks: [longerLlm],});const input = `What is the next number: ${"one, two, ".repeat(3000)}`;try {  await shorterLlm.invoke(input);} catch (e) {  // Length error  console.log(e);}const result = await modelWithFallback.invoke(input);console.log(result);/*  AIMessage {    content: 'The next number is one.',    name: undefined,    additional_kwargs: { function_call: undefined }  }*/

#### API Reference:

*   [ChatOpenAI](https://v02.api.js.langchain.com/classes/langchain_openai.ChatOpenAI.html) from `@langchain/openai`

Fallback to a better model[​](#fallback-to-a-better-model "Direct link to Fallback to a better model")
------------------------------------------------------------------------------------------------------

Often times we ask models to output format in a specific format (like JSON). Models like GPT-3.5 can do this okay, but sometimes struggle. This naturally points to fallbacks - we can try with a faster and cheaper model, but then if parsing fails we can use GPT-4.

    import { z } from "zod";import { OpenAI, ChatOpenAI } from "@langchain/openai";import { PromptTemplate } from "@langchain/core/prompts";import { StructuredOutputParser } from "@langchain/core/output_parsers";const prompt = PromptTemplate.fromTemplate(  `Return a JSON object containing the following value wrapped in an "input" key. Do not return anything else:\n{input}`);const badModel = new OpenAI({  maxRetries: 0,  model: "gpt-3.5-turbo-instruct",});const normalModel = new ChatOpenAI({  model: "gpt-4",});const outputParser = StructuredOutputParser.fromZodSchema(  z.object({    input: z.string(),  }));const badChain = prompt.pipe(badModel).pipe(outputParser);const goodChain = prompt.pipe(normalModel).pipe(outputParser);try {  const result = await badChain.invoke({    input: "testing0",  });} catch (e) {  console.log(e);  /*  OutputParserException [Error]: Failed to parse. Text: "  { "name" : " Testing0 ", "lastname" : " testing ", "fullname" : " testing ", "role" : " test ", "telephone" : "+1-555-555-555 ", "email" : " testing@gmail.com ", "role" : " test ", "text" : " testing0 is different than testing ", "role" : " test ", "immediate_affected_version" : " 0.0.1 ", "immediate_version" : " 1.0.0 ", "leading_version" : " 1.0.0 ", "version" : " 1.0.0 ", "finger prick" : " no ", "finger prick" : " s ", "text" : " testing0 is different than testing ", "role" : " test ", "immediate_affected_version" : " 0.0.1 ", "immediate_version" : " 1.0.0 ", "leading_version" : " 1.0.0 ", "version" : " 1.0.0 ", "finger prick" :". Error: SyntaxError: Unexpected end of JSON input*/}const chain = badChain.withFallbacks({  fallbacks: [goodChain],});const result = await chain.invoke({  input: "testing",});console.log(result);/*  { input: 'testing' }*/

#### API Reference:

*   [OpenAI](https://v02.api.js.langchain.com/classes/langchain_openai.OpenAI.html) from `@langchain/openai`
*   [ChatOpenAI](https://v02.api.js.langchain.com/classes/langchain_openai.ChatOpenAI.html) from `@langchain/openai`
*   [PromptTemplate](https://v02.api.js.langchain.com/classes/langchain_core_prompts.PromptTemplate.html) from `@langchain/core/prompts`
*   [StructuredOutputParser](https://v02.api.js.langchain.com/classes/langchain_core_output_parsers.StructuredOutputParser.html) from `@langchain/core/output_parsers`

* * *

#### Was this page helpful?

  

#### You can leave detailed feedback [on GitHub](https://github.com/langchain-ai/langchainjs/issues/new?assignees=&labels=03+-+Documentation&projects=&template=documentation.yml&title=DOC%3A+%3CPlease+write+a+comprehensive+title+after+the+%27DOC%3A+%27+prefix%3E).

[

Previous

How to do extraction without using function calling

](/v0.2/docs/how_to/extraction_parse)[

Next

Few Shot Prompt Templates

](/v0.2/docs/how_to/few_shot)

*   [Handling LLM API errors](#handling-llm-api-errors)
*   [Fallbacks for RunnableSequences](#fallbacks-for-runnablesequences)
*   [Handling long inputs](#handling-long-inputs)
*   [Fallback to a better model](#fallback-to-a-better-model)

Community

*   [Discord](https://discord.gg/cU2adEyC7w)
*   [Twitter](https://twitter.com/LangChainAI)

GitHub

*   [Python](https://github.com/langchain-ai/langchain)
*   [JS/TS](https://github.com/langchain-ai/langchainjs)

More

*   [Homepage](https://langchain.com)
*   [Blog](https://blog.langchain.dev)

Copyright © 2024 LangChain, Inc.