!function(){function t(t){document.documentElement.setAttribute("data-theme",t)}var e=function(){var t=null;try{t=new URLSearchParams(window.location.search).get("docusaurus-theme")}catch(t){}return t}()||function(){var t=null;try{t=localStorage.getItem("theme")}catch(t){}return t}();t(null!==e?e:"light")}(),document.documentElement.setAttribute("data-announcement-bar-initially-dismissed",function(){try{return"true"===localStorage.getItem("docusaurus.announcement.dismiss")}catch(t){}return!1}())

[Skip to main content](#__docusaurus_skipToContent_fallback)

LangChain v0.2 is coming soon! Preview the new docs [here](/v0.2/docs/introduction/).

[

![🦜️🔗 Langchain](/v0.1/img/brand/wordmark.png)![🦜️🔗 Langchain](/v0.1/img/brand/wordmark-dark.png)

](/v0.1/)[Docs](/v0.1/docs/get_started/introduction/)[Use cases](/v0.1/docs/use_cases/)[Integrations](/v0.1/docs/integrations/platforms/)[API Reference](https://api.js.langchain.com)

[More](#)

*   [People](/v0.1/docs/people/)
*   [Community](/v0.1/docs/community/)
*   [Tutorials](/v0.1/docs/additional_resources/tutorials/)
*   [Contributing](/v0.1/docs/contributing/)

[v0.1](#)

*   [v0.2](https://js.langchain.com/v0.2/docs/introduction)
*   [v0.1](/v0.1/docs/get_started/introduction/)

[🦜🔗](#)

*   [LangSmith](https://smith.langchain.com)
*   [LangSmith Docs](https://docs.smith.langchain.com)
*   [LangChain Hub](https://smith.langchain.com/hub)
*   [LangServe](https://github.com/langchain-ai/langserve)
*   [Python Docs](https://python.langchain.com/)

[Chat](https://chatjs.langchain.com)[](https://github.com/langchain-ai/langchainjs)

Search

*   [Providers](/v0.1/docs/integrations/platforms/)
    
    *   [Providers](/v0.1/docs/integrations/platforms/)
    *   [Anthropic](/v0.1/docs/integrations/platforms/anthropic/)
    *   [AWS](/v0.1/docs/integrations/platforms/aws/)
    *   [Google](/v0.1/docs/integrations/platforms/google/)
    *   [Microsoft](/v0.1/docs/integrations/platforms/microsoft/)
    *   [OpenAI](/v0.1/docs/integrations/platforms/openai/)
*   [Components](/v0.1/docs/integrations/components/)
    
    *   [LLMs](/v0.1/docs/integrations/llms/)
        
        *   [LLMs](/v0.1/docs/integrations/llms/)
        *   [AI21](/v0.1/docs/integrations/llms/ai21/)
        *   [AlephAlpha](/v0.1/docs/integrations/llms/aleph_alpha/)
        *   [AWS SageMakerEndpoint](/v0.1/docs/integrations/llms/aws_sagemaker/)
        *   [Azure OpenAI](/v0.1/docs/integrations/llms/azure/)
        *   [Bedrock](/v0.1/docs/integrations/llms/bedrock/)
        *   [Cloudflare Workers AI](/v0.1/docs/integrations/llms/cloudflare_workersai/)
        *   [Cohere](/v0.1/docs/integrations/llms/cohere/)
        *   [Fake LLM](/v0.1/docs/integrations/llms/fake/)
        *   [Fireworks](/v0.1/docs/integrations/llms/fireworks/)
        *   [Friendli](/v0.1/docs/integrations/llms/friendli/)
        *   [(Legacy) Google PaLM/VertexAI](/v0.1/docs/integrations/llms/google_palm/)
        *   [Google Vertex AI](/v0.1/docs/integrations/llms/google_vertex_ai/)
        *   [Gradient AI](/v0.1/docs/integrations/llms/gradient_ai/)
        *   [HuggingFaceInference](/v0.1/docs/integrations/llms/huggingface_inference/)
        *   [Llama CPP](/v0.1/docs/integrations/llms/llama_cpp/)
        *   [NIBittensor](/v0.1/docs/integrations/llms/ni_bittensor/)
        *   [Ollama](/v0.1/docs/integrations/llms/ollama/)
        *   [OpenAI](/v0.1/docs/integrations/llms/openai/)
        *   [PromptLayer OpenAI](/v0.1/docs/integrations/llms/prompt_layer_openai/)
        *   [RaycastAI](/v0.1/docs/integrations/llms/raycast/)
        *   [Replicate](/v0.1/docs/integrations/llms/replicate/)
        *   [Together AI](/v0.1/docs/integrations/llms/togetherai/)
        *   [WatsonX AI](/v0.1/docs/integrations/llms/watsonx_ai/)
        *   [Writer](/v0.1/docs/integrations/llms/writer/)
        *   [YandexGPT](/v0.1/docs/integrations/llms/yandex/)
    *   [Chat models](/v0.1/docs/integrations/chat/)
        
    *   [Document loaders](/v0.1/docs/integrations/document_loaders/)
        
    *   [Document transformers](/v0.1/docs/integrations/document_transformers/)
        
    *   [Document compressors](/v0.1/docs/integrations/document_compressors/)
        
    *   [Text embedding models](/v0.1/docs/integrations/text_embedding/)
        
    *   [Vector stores](/v0.1/docs/integrations/vectorstores/)
        
    *   [Retrievers](/v0.1/docs/integrations/retrievers/)
        
    *   [Tools](/v0.1/docs/integrations/tools/)
        
    *   [Agents and toolkits](/v0.1/docs/integrations/toolkits/)
        
    *   [Chat Memory](/v0.1/docs/integrations/chat_memory/)
        
    *   [Stores](/v0.1/docs/integrations/stores/)
        

*   [](/v0.1/)
*   [Components](/v0.1/docs/integrations/components/)
*   [LLMs](/v0.1/docs/integrations/llms/)
*   Llama CPP

On this page

Llama CPP
=========

Compatibility

Only available on Node.js.

This module is based on the [node-llama-cpp](https://github.com/withcatai/node-llama-cpp) Node.js bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp), allowing you to work with a locally running LLM. This allows you to work with a much smaller quantized model capable of running on a laptop environment, ideal for testing and scratch padding ideas without running up a bill!

Setup[​](#setup "Direct link to Setup")
---------------------------------------

You'll need to install the [node-llama-cpp](https://github.com/withcatai/node-llama-cpp) module to communicate with your local model.

*   npm
*   Yarn
*   pnpm

    npm install -S node-llama-cpp

    yarn add node-llama-cpp

    pnpm add node-llama-cpp

tip

See [this section for general instructions on installing integration packages](/v0.1/docs/get_started/installation/#installing-integration-packages).

*   npm
*   Yarn
*   pnpm

    npm install @langchain/community

    yarn add @langchain/community

    pnpm add @langchain/community

You will also need a local Llama 2 model (or a model supported by [node-llama-cpp](https://github.com/withcatai/node-llama-cpp)). You will need to pass the path to this model to the LlamaCpp module as a part of the parameters (see example).

Out-of-the-box `node-llama-cpp` is tuned for running on a MacOS platform with support for the Metal GPU of Apple M-series of processors. If you need to turn this off or need support for the CUDA architecture then refer to the documentation at [node-llama-cpp](https://withcatai.github.io/node-llama-cpp/).

A note to LangChain.js contributors: if you want to run the tests associated with this module you will need to put the path to your local model in the environment variable `LLAMA_PATH`.

Guide to installing Llama2[​](#guide-to-installing-llama2 "Direct link to Guide to installing Llama2")
------------------------------------------------------------------------------------------------------

Getting a local Llama2 model running on your machine is a pre-req so this is a quick guide to getting and building Llama 7B (the smallest) and then quantizing it so that it will run comfortably on a laptop. To do this you will need `python3` on your machine (3.11 is recommended), also `gcc` and `make` so that `llama.cpp` can be built.

### Getting the Llama2 models[​](#getting-the-llama2-models "Direct link to Getting the Llama2 models")

To get a copy of Llama2 you need to visit [Meta AI](https://ai.meta.com/resources/models-and-libraries/llama-downloads/) and request access to their models. Once Meta AI grant you access, you will receive an email containing a unique URL to access the files, this will be needed in the next steps. Now create a directory to work in, for example:

    mkdir llama2cd llama2

Now we need to get the Meta AI `llama` repo in place so we can download the model.

    git clone https://github.com/facebookresearch/llama.git

Once we have this in place we can change into this directory and run the downloader script to get the model we will be working with. Note: From here on its assumed that the model in use is `llama-2–7b`, if you select a different model don't forget to change the references to the model accordingly.

    cd llama/bin/bash ./download.sh

This script will ask you for the URL that Meta AI sent to you (see above), you will also select the model to download, in this case we used `llama-2–7b`. Once this step has completed successfully (this can take some time, the `llama-2–7b` model is around 13.5Gb) there should be a new `llama-2–7b` directory containing the model and other files.

### Converting and quantizing the model[​](#converting-and-quantizing-the-model "Direct link to Converting and quantizing the model")

In this step we need to use `llama.cpp` so we need to download that repo.

    cd ..git clone https://github.com/ggerganov/llama.cpp.gitcd llama.cpp

Now we need to build the `llama.cpp` tools and set up our `python` environment. In these steps it's assumed that your install of python can be run using `python3` and that the virtual environment can be called `llama2`, adjust accordingly for your own situation.

    makepython3 -m venv llama2source llama2/bin/activate

After activating your llama2 environment you should see `(llama2)` prefixing your command prompt to let you know this is the active environment. Note: if you need to come back to build another model or re-quantize the model don't forget to activate the environment again also if you update `llama.cpp` you will need to rebuild the tools and possibly install new or updated dependencies! Now that we have an active python environment, we need to install the python dependencies.

    python3 -m pip install -r requirements.txt

Having done this, we can start converting and quantizing the Llama2 model ready for use locally via `llama.cpp`. First, we need to convert the model, prior to the conversion let's create a directory to store it in.

    mkdir models/7Bpython3 convert.py --outfile models/7B/gguf-llama2-f16.bin --outtype f16 ../../llama2/llama/llama-2-7b --vocab-dir ../../llama2/llama/llama-2-7b

This should create a converted model called `gguf-llama2-f16.bin` in the directory we just created. Note that this is just a converted model so it is also around 13.5Gb in size, in the next step we will quantize it down to around 4Gb.

    ./quantize ./models/7B/gguf-llama2-f16.bin ./models/7B/gguf-llama2-q4_0.bin q4_0

Running this should result in a new model being created in the `models\7B` directory, this one called `gguf-llama2-q4_0.bin`, this is the model we can use with langchain. You can validate this model is working by testing it using the `llama.cpp` tools.

    ./main -m ./models/7B/gguf-llama2-q4_0.bin -n 1024 --repeat_penalty 1.0 --color -i -r "User:" -f ./prompts/chat-with-bob.txt

Running this command fires up the model for a chat session. BTW if you are running out of disk space this small model is the only one we need, so you can backup and/or delete the original and converted 13.5Gb models.

Usage[​](#usage "Direct link to Usage")
---------------------------------------

    import { LlamaCpp } from "@langchain/community/llms/llama_cpp";const llamaPath = "/Replace/with/path/to/your/model/gguf-llama2-q4_0.bin";const question = "Where do Llamas come from?";const model = new LlamaCpp({ modelPath: llamaPath });console.log(`You: ${question}`);const response = await model.invoke(question);console.log(`AI : ${response}`);

#### API Reference:

*   [LlamaCpp](https://api.js.langchain.com/classes/langchain_community_llms_llama_cpp.LlamaCpp.html) from `@langchain/community/llms/llama_cpp`

Streaming[​](#streaming "Direct link to Streaming")
---------------------------------------------------

    import { LlamaCpp } from "@langchain/community/llms/llama_cpp";const llamaPath = "/Replace/with/path/to/your/model/gguf-llama2-q4_0.bin";const model = new LlamaCpp({ modelPath: llamaPath, temperature: 0.7 });const prompt = "Tell me a short story about a happy Llama.";const stream = await model.stream(prompt);for await (const chunk of stream) {  console.log(chunk);}/* Once  upon  a  time ,  in  the  rolling  hills  of  Peru ... */

#### API Reference:

*   [LlamaCpp](https://api.js.langchain.com/classes/langchain_community_llms_llama_cpp.LlamaCpp.html) from `@langchain/community/llms/llama_cpp`

;

* * *

#### Help us out by providing feedback on this documentation page:

[

Previous

HuggingFaceInference

](/v0.1/docs/integrations/llms/huggingface_inference/)[

Next

NIBittensor

](/v0.1/docs/integrations/llms/ni_bittensor/)

*   [Setup](#setup)
*   [Guide to installing Llama2](#guide-to-installing-llama2)
    *   [Getting the Llama2 models](#getting-the-llama2-models)
    *   [Converting and quantizing the model](#converting-and-quantizing-the-model)
*   [Usage](#usage)
*   [Streaming](#streaming)

Community

*   [Discord](https://discord.gg/cU2adEyC7w)
*   [Twitter](https://twitter.com/LangChainAI)

GitHub

*   [Python](https://github.com/langchain-ai/langchain)
*   [JS/TS](https://github.com/langchain-ai/langchainjs)

More

*   [Homepage](https://langchain.com)
*   [Blog](https://blog.langchain.dev)

Copyright © 2024 LangChain, Inc.