Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionChainsMemoryAgentsCallbacksModulesGuidesEvaluationString EvaluatorsComparison EvaluatorsTrajectory EvaluatorsExamplesAgent VectorDB Question Answering BenchmarkingComparing Chain OutputsData Augmented Question AnsweringEvaluating an OpenAPI ChainQuestion Answering Benchmarking: Paul Graham EssayQuestion Answering Benchmarking: State of the Union AddressQA GenerationQuestion AnsweringSQL Question Answering Benchmarking: ChinookDebuggingDeploymentLangSmithModel ComparisonEcosystemAdditional resourcesGuidesEvaluationExamplesComparing Chain OutputsOn this pageComparing Chain OutputsSuppose you have two different prompts (or LLMs). How do you know which will generate "better" results?One automated way to predict the preferred configuration is to use a PairwiseStringEvaluator like the PairwiseStringEvalChain[1]. This chain prompts an LLM to select which output is preferred, given a specific input.For this evaluation, we will need 3 things:An evaluatorA dataset of inputs2 (or more) LLMs, Chains, or Agents to compareThen we will aggregate the restults to determine the preferred model.Step 1. Create the Evaluatorâ€‹In this example, you will use gpt-4 to select which output is preferred.from langchain.chat_models import ChatOpenAIfrom langchain.evaluation.comparison import PairwiseStringEvalChainllm = ChatOpenAI(model="gpt-4")eval_chain = PairwiseStringEvalChain.from_llm(llm=llm)Step 2. Select Datasetâ€‹If you already have real usage data for your LLM, you can use a representative sample. More examples