ProvidersMoreModalOn this pageModalThis page covers how to use the Modal ecosystem to run LangChain custom LLMs.
It is broken into two parts: Modal installation and web endpoint deploymentUsing deployed web endpoint with LLM wrapper class.Installation and Setup​Install with pip install modalRun modal token newDefine your Modal Functions and Webhooks​You must include a prompt. There is a rigid response structure:class Item(BaseModel):    prompt: str@stub.function()@modal.web_endpoint(method="POST")def get_text(item: Item):    return {"prompt": run_gpt2.call(item.prompt)}The following is an example with the GPT2 model:from pydantic import BaseModelimport modalCACHE_PATH = "/root/model_cache"class Item(BaseModel):    prompt: strstub = modal.Stub(name="example-get-started-with-langchain")def download_model():    from transformers import GPT2Tokenizer, GPT2LMHeadModel    tokenizer = GPT2Tokenizer.from_pretrained('gpt2')    model = GPT2LMHeadModel.from_pretrained('gpt2')    tokenizer.save_pretrained(CACHE_PATH)    model.save_pretrained(CACHE_PATH)# Define a container image for the LLM function below, which# downloads and stores the GPT-2 model.image = modal.Image.debian_slim().pip_install(    "tokenizers", "transformers", "torch", "accelerate").run_function(download_model)@stub.function(    gpu="any",    image=image,    retries=3,)def run_gpt2(text: str):    from transformers import GPT2Tokenizer, GPT2LMHeadModel    tokenizer = GPT2Tokenizer.from_pretrained(CACHE_PATH)    model = GPT2LMHeadModel.from_pretrained(CACHE_PATH)    encoded_input = tokenizer(text, return_tensors='pt').input_ids    output = model.generate(encoded_input, max_length=50, do_sample=True)    return tokenizer.decode(output[0], skip_special_tokens=True)@stub.function()@modal.web_endpoint(method="POST")def get_text(item: Item):    return {"prompt": run_gpt2.call(item.prompt)}Deploy the web endpoint​Deploy the web endpoint to Modal cloud with the modal deploy CLI command.
Your web endpoint will acquire a persistent URL under the modal.run domain.LLM wrapper around Modal web endpoint​The  Modal LLM wrapper class which will accept your deployed web endpoint's URL.from langchain.llms import Modalendpoint_url = "https://ecorp--custom-llm-endpoint.modal.run"  # REPLACE ME with your deployed Modal web endpoint's URLllm = Modal(endpoint_url=endpoint_url)llm_chain = LLMChain(prompt=prompt, llm=llm)question = "What NFL team won the Super Bowl in the year Justin Beiber was born?"llm_chain.run(question)PreviousMLflowNextModelScopeInstallation and SetupDefine your Modal Functions and WebhooksDeploy the web endpointLLM wrapper around Modal web endpoint