Update README.md
Browse files
README.md
CHANGED
|
@@ -95,6 +95,14 @@ SQL Query:
|
|
| 95 |
SELECT avg(age), min(age), max(age) FROM singer WHERE country = 'France'
|
| 96 |
```
|
| 97 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 98 |
|
| 99 |
## Training
|
| 100 |
|
|
|
|
| 95 |
SELECT avg(age), min(age), max(age) FROM singer WHERE country = 'France'
|
| 96 |
```
|
| 97 |
|
| 98 |
+
## Evaluation
|
| 99 |
+
Evaluation was done on the dev split of the Spider and Spider-syn dataset. The databases present in the dev split have no intersection with the databases of the train split.
|
| 100 |
+
This way we ensure, that the model was not exposed to the evaluated databases during training.
|
| 101 |
+
The evaluation was done by comparing the results of querying the database using the generated query and reference.
|
| 102 |
+
Both Spider and Spider-Syn dev splits have 1032 samples.
|
| 103 |
+
* **Spider dev accuracy:** 49.2%
|
| 104 |
+
* **Spider Syn dev accuracy:** 39.5%
|
| 105 |
+
|
| 106 |
|
| 107 |
## Training
|
| 108 |
|