|
|
--- |
|
|
license: apache-2.0 |
|
|
datasets: |
|
|
- PipableAI/pip-txt-to-sql-spider-bird-dataset |
|
|
language: |
|
|
- en |
|
|
metrics: |
|
|
- accuracy |
|
|
tags: |
|
|
- sql |
|
|
- code |
|
|
- text2sql |
|
|
- instruction_tuned |
|
|
- basemodel |
|
|
- jax |
|
|
- pytorch |
|
|
- tensorflow |
|
|
- text-generation-inference |
|
|
library_name: transformers |
|
|
pipeline_tag: text-generation |
|
|
--- |
|
|
# pipSQL-1.3b |
|
|
[pipableAi](https://www.linkedin.com/company/pipable.ai/about/) |
|
|
|
|
|
## What have we built? |
|
|
A 1.3 bn SQL model that outperforms most SQL expert models and chatgpt on popular benchmarks. |
|
|
This is a distilled model built on the deepseek base model. |
|
|
|
|
|
## How we built it? |
|
|
|
|
|
We used softmax cross entropy and a modified form of policy grad along with Q loss, optimized in an EM set up. |
|
|
|
|
|
## Benchmarking : |
|
|
For benchmarking purposes we are using Semantic Evaluation for Text-to-SQL with |
|
|
Distilled Test Suites, an officially accepted evaluation framework for Spider, SParC, and CoSQL which was proposed by a research team of Yale and Berkeley. |
|
|
The benchmark contains 2200 test data points |
|
|
Here is the link to run the evaluation: |
|
|
|
|
|
|
|
|
[Test Suite SQL Eval](https://github.com/taoyds/test-suite-sql-eval) |
|
|
|
|
|
|model|easy|medium|hard|extra| |
|
|
|-----|----|------|----|-----| |
|
|
|sqlcoder-7b-2|72.0|58.0|40.6|37.3| |
|
|
|pip-sql-1b-Qstar|74.0|54.0|36.5|30.0| |
|
|
|pipSQL-7b|63.0|40.0|30.2|25.0| |
|
|
|sqlcoder-7b|60.6|48.2|28.3|20.4| |
|
|
|gpt-3.5|58.8|44.7|31.0|28.4| |
|
|
|
|
|
We have also benchmarked it on defog eval. |
|
|
It contains 200 test data points handpicked by defog team. |
|
|
Here is the link to it: |
|
|
|
|
|
|
|
|
[Defog SQL-Eval](https://github.com/defog-ai/sql-eval) |
|
|
These are the results - |
|
|
|
|
|
 |
|
|
|
|
|
## License |
|
|
The model is open source under apache 2.0. License |
|
|
|
|
|
## Usage |
|
|
|
|
|
### Installation |
|
|
|
|
|
```bash |
|
|
pip install transformers |
|
|
``` |
|
|
|
|
|
### Prompt |
|
|
```python |
|
|
prompt = f"""<schema>{schema}</schema> |
|
|
<question>{question}</question> |
|
|
<sql>""" |
|
|
``` |
|
|
|
|
|
### PyTorch |
|
|
```python |
|
|
from transformers import AutoModelForCasualLM, AutoTokenizer |
|
|
device = "cuda" |
|
|
model = AutoModelForCausalLM.from_pretrained("PipableAI/pipSQL-1.3b") |
|
|
tokenizer = AutoTokenizer.from_pretrained("PipableAI/pipSQL-1.3b") |
|
|
|
|
|
inputs = tokenizer(text, return_tensors="pt") |
|
|
outputs = model.generate(**inputs, max_new_tokens=200) |
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True).split('<sql>')[1].split('</sql>')[0]) |
|
|
``` |
|
|
|
|
|
### Flax |
|
|
```python |
|
|
from transfomers import FlaxAutoModelForCausalLM, AutoTokenizer |
|
|
device = "cuda" |
|
|
model = FlaxAutoModelForCausalLM.from_pretrained("PipableAI/pipSQL-1.3b") |
|
|
tokenizer = AutoTokenizer.from_pretrained("PipableAI/pipSQL-1.3b") |
|
|
|
|
|
inputs = tokenizer(text, return_tensors="pt") |
|
|
outputs = model.generate(**inputs, max_new_tokens=200) |
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True).split('<sql>')[1].split('</sql>')[0]) |
|
|
``` |
|
|
|
|
|
### TensorFlow |
|
|
```python |
|
|
from transfomers import TFAutoModelForCausalLM, AutoTokenizer |
|
|
device = "cuda" |
|
|
model = TFAutoModelForCausalLM.from_pretrained("PipableAI/pipSQL-1.3b") |
|
|
tokenizer = AutoTokenizer.from_pretrained("PipableAI/pipSQL-1.3b") |
|
|
|
|
|
inputs = tokenizer(text, return_tensors="pt") |
|
|
outputs = model.generate(**inputs, max_new_tokens=200) |
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True).split('<sql>')[1].split('</sql>')[0]) |
|
|
``` |
