# Q&A Evaluation:

This is a flow evaluating the Q&A RAG (Retrieval Augmented Generation) systems by leveraging the state-of-the-art Large Language Models (LLM) to measure the quality and safety of responses. Utilizing GPT model to assist with measurements aims to achieve a high agreement with human evaluations compared to traditional mathematical measurements.

## What you will learn

The Q&A RAG evaluation flow allows you to assess and evaluate your model with the LLM-assisted metrics:


__gpt_retrieval_score__: Measures the relevance between the retrieved documents and the potential answer to the given question in the range of 1 to 5:

* 1 means that none of the document is relevant to the question at all
* 5 means that either one of the documents or combination of a few documents is ideal for answering the given question.


__gpt_groundedness__ : Measures how grounded the factual information in the answers is against the fact from the retrieved documents. Even if answers is true, if not verifiable against context, then such answers are considered ungrounded.

Grounding score is scored on a scale of 1 to 5, with 1 being the worst and 5 being the best.

__gpt_relevance__: Measures the answer quality against the preference answer generated by LLm with the retrieved documents in the range of 1 to 5:

* 1 means the provided answer is completely irrelevant to the reference answer.
* 5 means the provided answer includes all information necessary to answer the question based on the reference answer. 
If the reference answer is can not be generated since no relevant document were retrieved, the answer would be rated as 5. 


## Prerequisites

- Connection: Azure OpenAI or OpenAI connection.
- Data input: Evaluating the Coherence metric requires you to provide data inputs including a question, an answer, and documents in json format. 

## Tools used in this flow
- `Python` tool
- `LLM` tool

## 0. Setup connection
Prepare your Azure Open AI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one.

```bash
# Override keys with --set to avoid yaml file changes
pf connection create --file ../../../connections/azure_openai.yml --set api_key=<your_api_key> api_base=<your_api_base>
```

## 1. Test flow/node
```bash
# test with default input value in flow.dag.yaml
pf flow test --flow .
```

## 2. Create flow run with multi line data and selected metrics
```bash
pf run create --flow . --data ./data.jsonl --column-mapping question='${data.question}' answer='${data.answer}' documents='${data.documents}' metrics='gpt_groundedness' --stream
```
You can also skip providing `column-mapping` if provided data has same column name as the flow.
Reference [here](https://aka.ms/pf/column-mapping) for default behavior when `column-mapping` not provided in CLI.

## 3. Run and Evaluate your flow with this Q&A RAG evaluation flow
After you develop your flow, you may want to run and evaluate it with this evaluation flow. 

Here we use the flow [basic_chat](../../chat/basic-chat/) as the main flow to evaluate. It is a flow demonstrating how to create a chatbot with LLM. The chatbot can remember previous interactions and use the conversation history to generate next message, given a question. 
### 3.1 Create a batch run of your flow
```bash
pf run create --flow ../../chat/basic-chat --data data.jsonl --column-mapping question='${data.question}' --name basic_chat_run --stream 
```
Please note that `column-mapping` is a mapping from flow input name to specified values. Please refer to [Use column mapping](https://aka.ms/pf/column-mapping) for more details. 

The flow run is named by specifying `--name basic_chat_run` in the above command. You can view the run details with its run name using the command:
```bash
pf run show-details -n basic_chat_run
```

### 3.2 Evaluate your flow
You can use this evaluation flow to measure the quality and safety of your flow responses.

After the chat flow run is finished, you can this evaluation flow to the run:
```bash
pf run create --flow . --data data.jsonl --column-mapping  answer='${run.outputs.answer}' documents='{${data.documents}}' question='${data.question}' metrics='gpt_groundedness,gpt_relevance,gpt_retrieval_score' --run basic_chat_run --stream --name evaluation_qa_rag
```
Please note the flow run to be evaluated is specified with `--run basic_chat_run`. Also same as previous run, the evaluation run is named with `--name evaluation_qa_rag`.
You can view the evaluation run details with:
```bash
pf run show-details -n evaluation_qa_rag
pf run show-metrics -n evaluation_qa_rag
```