autogen/website/docs/Use-Cases/enhanced_inference.md
Error handling  ### Runtime error  One can pass a list of configurations of different models/endpoints to mitigate the rate limits and other runtime error. For example,  ```python client = OpenAIWrapper(     config_list=[         {             "model": "gpt-4",             "api_key": os.environ.get("AZURE_OPENAI_API_KEY"),             "api_type": "azure",             "base_url": os.environ.get("AZURE_OPENAI_API_BASE"),             "api_version": "2024-02-01",         },         {             "model": "gpt-3.5-turbo",             "api_key": os.environ.get("OPENAI_API_KEY"),             "base_url": "https://api.openai.com/v1",         },         {             "model": "llama2-chat-7B",             "base_url": "http://127.0.0.1:8080",         },         {             "model": "microsoft/phi-2",             "model_client_cls": "CustomModelClient"         }     ], ) ```  `client.create()` will try querying Azure OpenAI gpt-4, OpenAI gpt-3.5-turbo, a locally hosted llama2-chat-7B, and phi-2 using a custom model client class named `CustomModelClient`, one by one, until a valid result is returned. This can speed up the development process where the rate limit is a bottleneck. An error will be raised if the last choice fails. So make sure the last choice in the list has the best availability.  For convenience, we provide a number of utility functions to load config lists. - `get_config_list`: Generates configurations for API calls, primarily from provided API keys. - `config_list_openai_aoai`: Constructs a list of configurations using both Azure OpenAI and OpenAI endpoints, sourcing API keys from environment variables or local files. - `config_list_from_json`: Loads configurations from a JSON structure, either from an environment variable or a local JSON file, with the flexibility of filtering configurations based on given criteria. - `config_list_from_models`: Creates configurations based on a provided list of models, useful when targeting specific models without manually specifying each configuration. - `config_list_from_dotenv`: Constructs a configuration list from a `.env` file, offering a consolidated way to manage multiple API configurations and keys from a single file.  We suggest that you take a look at this [notebook](/docs/topics/llm_configuration) for full code examples of the different methods to configure your model endpoints.  ### Logic error  Another type of error is that the returned response does not satisfy a requirement. For example, if the response is required to be a valid json string, one would like to filter the responses that are not. This can be achieved by providing a list of configurations and a filter function. For example,  ```python def valid_json_filter(response, **_):     for text in OpenAIWrapper.extract_text_or_completion_object(response):         try:             json.loads(text)             return True         except ValueError:             pass     return False  client = OpenAIWrapper(     config_list=[{"model": "text-ada-001"}, {"model": "gpt-3.5-turbo-instruct"}, {"model": "text-davinci-003"}], ) response = client.create(     prompt="How to construct a json request to Bing API to search for 'latest AI news'? Return the JSON request.",     filter_func=valid_json_filter, ) ```  The example above will try to use text-ada-001, gpt-3.5-turbo-instruct, and text-davinci-003 iteratively, until a valid json string is returned or the last config is used. One can also repeat the same model in the list for multiple times (with different seeds) to try one model multiple times for increasing the robustness of the final response.  *Advanced use case: Check this [blogpost](/blog/2023/05/18/GPT-adaptive-humaneval) to find how to improve GPT-4's coding performance from 68% to 90% while reducing the inference cost.* 