Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionChainsMemoryAgentsCallbacksModulesGuidesEvaluationString EvaluatorsComparison EvaluatorsCustom Pairwise EvaluatorPairwise Embedding DistancePairwise String ComparisonTrajectory EvaluatorsExamplesDebuggingDeploymentLangSmithModel ComparisonEcosystemAdditional resourcesGuidesEvaluationComparison EvaluatorsPairwise String ComparisonOn this pagePairwise String ComparisonOften you will want to compare predictions of an LLM, Chain, or Agent for a given input. The StringComparison evaluators facilitate this so you can answer questions like:Which LLM or prompt produces a preferred output for a given question?Which examples should I include for few-shot example selection?Which output is better to include for fintetuning?The simplest and often most reliable automated way to choose a preferred prediction for a given input is to use the pairwise_string evaluator.Check out the reference docs for the PairwiseStringEvalChain for more info.from langchain.evaluation import load_evaluatorevaluator = load_evaluator("pairwise_string", requires_reference=True)evaluator.evaluate_string_pairs(    prediction="there are three dogs",    prediction_b="4",    input="how many dogs are in the park?",    reference="four",)    {'reasoning': 'Response A provides an incorrect answer by stating there are three dogs in the park, while the reference answer indicates there are four. Response B, on the other hand, provides the correct answer, matching the reference answer. Although Response B is less detailed, it is accurate and directly answers the question. \n\nTherefore, the better response is [[B]].\n',     'value': 'B',     'score': 0}Without Referencesâ€‹When references aren't available, you can still predict the preferred response.