ComponentsChat modelsOn this pageChat modelsFeatures (natively supported)​All ChatModels implement the Runnable interface, which comes with default implementations of all methods, ie. ainvoke, batch, abatch, stream, astream. This gives all ChatModels basic support for async, streaming and batch, which by default is implemented as below:Async support defaults to calling the respective sync method in asyncio's default thread pool executor. This lets other async functions in your application make progress while the ChatModel is being executed, by moving this call to a background thread.Streaming support defaults to returning an Iterator (or AsyncIterator in the case of async streaming) of a single value, the final result returned by the underlying ChatModel provider. This obviously doesn't give you token-by-token streaming, which requires native support from the ChatModel provider, but ensures your code that expects an iterator of tokens can work for any of our ChatModel integrations.Batch support defaults to calling the underlying ChatModel in parallel for each input by making use of a thread pool executor (in the sync batch case) or asyncio.gather (in the async batch case). The concurrency can be controlled with the max_concurrency key in RunnableConfig.Each ChatModel integration can optionally provide native implementations to truly enable async or streaming.
The table shows, for each integration, which features have been implemented with native support.ModelInvokeAsync invokeStreamAsync streamAzureChatOpenAI✅✅✅✅BedrockChat✅❌✅❌ChatAnthropic✅✅✅✅ChatAnyscale✅✅✅✅ChatCohere✅✅✅✅ChatFireworks✅✅✅✅ChatGooglePalm✅✅❌❌ChatJavelinAIGateway✅✅❌❌ChatKonko✅❌❌❌ChatLiteLLM✅✅✅✅ChatMLflowAIGateway✅❌❌❌ChatOllama✅❌✅❌ChatOpenAI✅✅✅✅ChatVertexAI✅✅✅❌ErnieBotChat✅❌❌❌JinaChat✅✅✅✅MiniMaxChat✅✅❌❌PromptLayerChatOpenAI✅❌❌❌QianfanChatEndpoint✅✅✅✅📄️ Chat modelsFeatures (natively supported)📄️ AnthropicThis notebook covers how to get started with Anthropic chat models.📄️ Anthropic FunctionsThis notebook shows how to use an experimental wrapper around Anthropic that gives it the same API as OpenAI Functions.📄️ AnyscaleThis notebook demonstrates the use of langchain.chat_models.ChatAnyscale for Anyscale Endpoints.📄️ AzureThis notebook goes over how to connect to an Azure hosted OpenAI endpoint📄️ AzureML Chat Online EndpointAzureML is a platform used to build, train, and deploy machine learning models. Users can explore the types of models to deploy in the Model Catalog, which provides Azure Foundation Models and OpenAI Models. Azure Foundation Models include various open-source models and popular Hugging Face models. Users can also import models of their liking into AzureML.📄️ Baidu QianfanBaidu AI Cloud Qianfan Platform is a one-stop large model development and service operation platform for enterprise developers. Qianfan not only provides including the model of Wenxin Yiyan (ERNIE-Bot) and the third-party open source models, but also provides various AI development tools and the whole set of development environment, which facilitates customers to use and develop large model applications easily.📄️ Bedrock ChatAmazon Bedrock is a fully managed service that makes FMs from leading AI startups and Amazon available via an API, so you can choose from a wide range of FMs to find the model that is best suited for your use case📄️ CohereThis notebook covers how to get started with Cohere chat models.📄️ ERNIE-Bot ChatERNIE-Bot is a large language model developed by Baidu, covering a huge amount of Chinese data.📄️ FireworksFireworks accelerates product development on generative AI by creating an innovative AI experiment and production platform.📄️ GCP Vertex AINote: This is seperate from the Google PaLM integration. Google has chosen to offer an enterprise version of PaLM through GCP, and this supports the models made available through there.📄️ JinaChatThis notebook covers how to get started with JinaChat chat models.📄️ KonkoKonko API is a fully managed Web API designed to help application developers:📄️ 🚅 LiteLLMLiteLLM is a library that simplifies calling Anthropic, Azure, Huggingface, Replicate, etc.📄️ Llama APIThis notebook shows how to use LangChain with LlamaAPI - a hosted version of Llama2 that adds in support for function calling.📄️ MiniMaxMinimax is a Chinese startup that provides LLM service for companies and individuals.📄️ OllamaOllama allows you to run open-source large language models, such as LLaMA2, locally.📄️ OpenAIThis notebook covers how to get started with OpenAI chat models.📄️ PromptLayer ChatOpenAIThis example showcases how to connect to PromptLayer to start recording your ChatOpenAI requests.📄️ vLLM ChatvLLM can be deployed as a server that mimics the OpenAI API protocol. This allows vLLM to be used as a drop-in replacement for applications using OpenAI API. This server can be queried in the same format as OpenAI API.PreviousXorbits Inference (Xinference)NextChat modelsFeatures (natively supported)