!function(){function t(t){document.documentElement.setAttribute("data-theme",t)}var e=function(){var t=null;try{t=new URLSearchParams(window.location.search).get("docusaurus-theme")}catch(t){}return t}()||function(){var t=null;try{t=localStorage.getItem("theme")}catch(t){}return t}();t(null!==e?e:"light")}(),document.documentElement.setAttribute("data-announcement-bar-initially-dismissed",function(){try{return"true"===localStorage.getItem("docusaurus.announcement.dismiss")}catch(t){}return!1}())

[Skip to main content](#__docusaurus_skipToContent_fallback)

LangChain v0.2 is coming soon! Preview the new docs [here](/v0.2/docs/introduction/).

[

![🦜️🔗 Langchain](/v0.1/img/brand/wordmark.png)![🦜️🔗 Langchain](/v0.1/img/brand/wordmark-dark.png)

](/v0.1/)[Docs](/v0.1/docs/get_started/introduction/)[Use cases](/v0.1/docs/use_cases/)[Integrations](/v0.1/docs/integrations/platforms/)[API Reference](https://api.js.langchain.com)

[More](#)

*   [People](/v0.1/docs/people/)
*   [Community](/v0.1/docs/community/)
*   [Tutorials](/v0.1/docs/additional_resources/tutorials/)
*   [Contributing](/v0.1/docs/contributing/)

[v0.1](#)

*   [v0.2](https://js.langchain.com/v0.2/docs/introduction)
*   [v0.1](/v0.1/docs/get_started/introduction/)

[🦜🔗](#)

*   [LangSmith](https://smith.langchain.com)
*   [LangSmith Docs](https://docs.smith.langchain.com)
*   [LangChain Hub](https://smith.langchain.com/hub)
*   [LangServe](https://github.com/langchain-ai/langserve)
*   [Python Docs](https://python.langchain.com/)

[Chat](https://chatjs.langchain.com)[](https://github.com/langchain-ai/langchainjs)

Search

*   [Get started](/v0.1/docs/get_started/)
    
    *   [Introduction](/v0.1/docs/get_started/introduction/)
    *   [Installation](/v0.1/docs/get_started/installation/)
    *   [Quickstart](/v0.1/docs/get_started/quickstart/)
*   [LangChain Expression Language](/v0.1/docs/expression_language/)
    
    *   [Get started](/v0.1/docs/expression_language/get_started/)
    *   [Why use LCEL?](/v0.1/docs/expression_language/why/)
    *   [Interface](/v0.1/docs/expression_language/interface/)
    *   [Streaming](/v0.1/docs/expression_language/streaming/)
    *   [How to](/v0.1/docs/expression_language/how_to/routing/)
        
    *   [Cookbook](/v0.1/docs/expression_language/cookbook/)
        
    *   [LangChain Expression Language (LCEL)](/v0.1/docs/expression_language/)
*   [Modules](/v0.1/docs/modules/)
    
    *   [Model I/O](/v0.1/docs/modules/model_io/)
        
    *   [Retrieval](/v0.1/docs/modules/data_connection/)
        
        *   [Document loaders](/v0.1/docs/modules/data_connection/document_loaders/)
            
        *   [Text Splitters](/v0.1/docs/modules/data_connection/document_transformers/)
            
            *   [Split by character](/v0.1/docs/modules/data_connection/document_transformers/character_text_splitter/)
            *   [Split code and markup](/v0.1/docs/modules/data_connection/document_transformers/code_splitter/)
            *   [Contextual chunk headers](/v0.1/docs/modules/data_connection/document_transformers/contextual_chunk_headers/)
            *   [Custom text splitters](/v0.1/docs/modules/data_connection/document_transformers/custom_text_splitter/)
            *   [Recursively split by character](/v0.1/docs/modules/data_connection/document_transformers/recursive_text_splitter/)
            *   [TokenTextSplitter](/v0.1/docs/modules/data_connection/document_transformers/token_splitter/)
        *   [Retrievers](/v0.1/docs/modules/data_connection/retrievers/)
            
        *   [Retrieval](/v0.1/docs/modules/data_connection/)
        *   [Text embedding models](/v0.1/docs/modules/data_connection/text_embedding/)
            
        *   [Vector stores](/v0.1/docs/modules/data_connection/vectorstores/)
            
        *   [Indexing](/v0.1/docs/modules/data_connection/indexing/)
        *   [Experimental](/v0.1/docs/modules/data_connection/experimental/multimodal_embeddings/google_vertex_ai/)
            
    *   [Chains](/v0.1/docs/modules/chains/)
    *   [Agents](/v0.1/docs/modules/agents/)
        
    *   [More](/v0.1/docs/modules/memory/)
        
*   [Security](/v0.1/docs/security/)
*   [Guides](/v0.1/docs/guides/)
    
*   [Ecosystem](/v0.1/docs/ecosystem/)
    
*   [LangGraph](/v0.1/docs/langgraph/)
*   * * *
    

*   [](/v0.1/)
*   [Modules](/v0.1/docs/modules/)
*   [Retrieval](/v0.1/docs/modules/data_connection/)
*   Text Splitters

On this page

Text Splitters
==============

Once you've loaded documents, you'll often want to transform them to better suit your application. The simplest example is you may want to split a long document into smaller chunks that can fit into your model's context window. LangChain has a number of built-in document transformers that make it easy to split, combine, filter, and otherwise manipulate documents.

When you want to deal with long pieces of text, it is necessary to split up that text into chunks. As simple as this sounds, there is a lot of potential complexity here. Ideally, you want to keep the semantically related pieces of text together. What "semantically related" means could depend on the type of text. This notebook showcases several ways to do that.

At a high level, text splitters work as following:

1.  Split the text up into small, semantically meaningful chunks (often sentences).
2.  Start combining these small chunks into a larger chunk until you reach a certain size (as measured by some function).
3.  Once you reach that size, make that chunk its own piece of text and then start creating a new chunk of text with some overlap (to keep context between chunks).

That means there are two different axes along which you can customize your text splitter:

1.  How the text is split
2.  How the chunk size is measured

Types of Text Splitters[​](#types-of-text-splitters "Direct link to Types of Text Splitters")
---------------------------------------------------------------------------------------------

LangChain offers many different types of text splitters. Below is a table listing all of them, along with a few characteristics:

**Name**: Name of the text splitter

**Splits On**: How this text splitter splits text

**Adds Metadata**: Whether or not this text splitter adds metadata about where each chunk came from.

**Description**: Description of the splitter, including recommendation on when to use it.

Name

Splits On

Adds Metadata

Description

Recursive

A list of user defined characters

Recursively splits text. Splitting text recursively serves the purpose of trying to keep related pieces of text next to each other. This is the recommended way to start splitting text.

HTML

HTML specific characters

Splits text based on HTML-specific characters.

Markdown

Markdown specific characters

Splits text based on Markdown-specific characters.

Code

Code (Python, JS) specific characters

Splits text based on characters specific to coding languages. 15 different languages are available to choose from.

Token

Tokens

Splits text on tokens. There exist a few different ways to measure tokens.

Character

A user defined character

Splits text based on a user defined character. One of the simpler methods.

Evaluate text splitters[​](#evaluate-text-splitters "Direct link to Evaluate text splitters")
---------------------------------------------------------------------------------------------

You can evaluate text splitters with the [Chunkviz utility](https://www.chunkviz.com/) created by `Greg Kamradt`. `Chunkviz` is a great tool for visualizing how your text splitter is working. It will show you how your text is being split up and help in tuning up the splitting parameters.

Other Document Transforms[​](#other-document-transforms "Direct link to Other Document Transforms")
---------------------------------------------------------------------------------------------------

Text splitting is only one example of transformations that you may want to do on documents before passing them to an LLM. Head to [Integrations](/v0.1/docs/integrations/document_transformers/) for documentation on built-in document transformer integrations with 3rd-party tools.

Get started with text splitters[​](#get-started-with-text-splitters "Direct link to Get started with text splitters")
---------------------------------------------------------------------------------------------------------------------

The recommended TextSplitter is the `RecursiveCharacterTextSplitter`. This will split documents recursively by different characters - starting with `"\n\n"`, then `"\n"`, then `" "`. This is nice because it will try to keep all the semantically relevant content in the same place for as long as possible.

Important parameters to know here are `chunkSize` and `chunkOverlap`. `chunkSize` controls the max size (in terms of number of characters) of the final documents. `chunkOverlap` specifies how much overlap there should be between chunks. This is often helpful to make sure that the text isn't split weirdly. In the example below we set these values to be small (for illustration purposes), but in practice they default to `1000` and `200` respectively.

    import { RecursiveCharacterTextSplitter } from "langchain/text_splitter";const text = `Hi.\n\nI'm Harrison.\n\nHow? Are? You?\nOkay then f f f f.This is a weird text to write, but gotta test the splittingggg some how.\n\nBye!\n\n-H.`;const splitter = new RecursiveCharacterTextSplitter({  chunkSize: 10,  chunkOverlap: 1,});const output = await splitter.createDocuments([text]);

You'll note that in the above example we are splitting a raw text string and getting back a list of documents. We can also split documents directly.

    import { Document } from "langchain/document";import { RecursiveCharacterTextSplitter } from "langchain/text_splitter";const text = `Hi.\n\nI'm Harrison.\n\nHow? Are? You?\nOkay then f f f f.This is a weird text to write, but gotta test the splittingggg some how.\n\nBye!\n\n-H.`;const splitter = new RecursiveCharacterTextSplitter({  chunkSize: 10,  chunkOverlap: 1,});const docOutput = await splitter.splitDocuments([  new Document({ pageContent: text }),]);

* * *

#### Help us out by providing feedback on this documentation page:

[

Previous

PDF

](/v0.1/docs/modules/data_connection/document_loaders/pdf/)[

Next

Split by character

](/v0.1/docs/modules/data_connection/document_transformers/character_text_splitter/)

*   [Types of Text Splitters](#types-of-text-splitters)
*   [Evaluate text splitters](#evaluate-text-splitters)
*   [Other Document Transforms](#other-document-transforms)
*   [Get started with text splitters](#get-started-with-text-splitters)

Community

*   [Discord](https://discord.gg/cU2adEyC7w)
*   [Twitter](https://twitter.com/LangChainAI)

GitHub

*   [Python](https://github.com/langchain-ai/langchain)
*   [JS/TS](https://github.com/langchain-ai/langchainjs)

More

*   [Homepage](https://langchain.com)
*   [Blog](https://blog.langchain.dev)

Copyright © 2024 LangChain, Inc.