.ipynb
.pdf
Agent Benchmarking: Search + Calculator
 Contents 
Loading the data
Setting up a chain
Make a prediction
Make many predictions
Evaluate performance
Agent Benchmarking: Search + Calculator#
Here we go over how to benchmark performance of an agent on tasks where it has access to a calculator and a search tool.
It is highly reccomended that you do any evaluation/benchmarking with tracing enabled. See here for an explanation of what tracing is and how to set it up.
# Comment this out if you are NOT using tracing
import os
os.environ["LANGCHAIN_HANDLER"] = "langchain"
Loading the data#
First, let’s load the data.
from langchain.evaluation.loading import load_dataset
dataset = load_dataset("agent-search-calculator")
Found cached dataset json (/Users/harrisonchase/.cache/huggingface/datasets/LangChainDatasets___json/LangChainDatasets--agent-search-calculator-8a025c0ce5fb99d2/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51)
Setting up a chain#
Now we need to load an agent capable of answering these questions.
from langchain.llms import OpenAI
from langchain.chains import LLMMathChain
from langchain.agents import initialize_agent, Tool, load_tools
tools = load_tools(['serpapi', 'llm-math'], llm=OpenAI(temperature=0))
agent = initialize_agent(tools, OpenAI(temperature=0), agent="zero-shot-react-description")
Make a prediction#