# Sharded fork of [Salesforce/codegen-6B-mono](https://huggingface.co/Salesforce/codegen-6B-mono) with a custom pipeline.py

This repository implements a custom `pipeline` task for `text-generation` for 🤗 Inference Endpoints for LLM inference using bitsandbytes quantization. The code for the customized pipeline is in the [pipeline.py](https://huggingface.co/philschmid/codegen-6B-mono-sharded-bnb/blob/main/pipeline.py).

There is also a [notebook](https://huggingface.co/philschmid/codegen-6B-mono-sharded-bnb/blob/main/create_handler.ipynb) included.

### expected Request payload
```json
{
    "inputs": "# load distilbert model and initialize text-classification pipeline\nmodel_id = 'distil",
    "parameters": {
        "top_k": 100,
        "max_length": 64,
        "early_stopping": true,
        "do_sample": true,
        "eos_token_id": 50256,
    }
}
```

below is an example on how to run a request using Python and `requests`.

## Run Request 
```python
import json
from typing import List
import requests as r
import base64
ENDPOINT_URL = ""
HF_TOKEN = ""

parameters={
        "top_k": 100,
        "max_length": 64,
        "early_stopping": True,
        "do_sample": True,
        "eos_token_id": 50256,
    }

def predict(code_snippet:str=None):
    payload = {"inputs": code_snippet,"parameters": parameters}
    response = r.post(
        ENDPOINT_URL, headers={"Authorization": f"Bearer {HF_TOKEN}"}, json=payload
    )
    return response.json()
prediction = predict(
    code_snippet="# load distilbert model and initialize text-classification pipeline\nmodel_id = 'distil"
)
```
expected output
```python
{'generated_text': "# load distilbert model and initialize text-classification pipeline\nmodel_id = 'distilbert-base-uncased'\nmodel_url = 'https://tfhub.dev/tensorflow/small_bert/1'\n\nmodel_dir = './distilBERT'"}
```

{"license": "bsd-3-clause", "tags": ["endpoints-template"], "pipeline_tag": "text-generation"}
transformers pytorch codegen text-generation endpoints-template license:bsd-3-clause autotrain_compatible endpoints_compatible region:us
TAGS
#transformers #pytorch #codegen #text-generation #endpoints-template #license-bsd-3-clause #autotrain_compatible #endpoints_compatible #region-us 

# Sharded fork of Salesforce/codegen-6B-mono with a custom URL

This repository implements a custom 'pipeline' task for 'text-generation' for  Inference Endpoints for LLM inference using bitsandbytes quantization. The code for the customized pipeline is in the URL.

There is also a notebook included.

### expected Request payload


below is an example on how to run a request using Python and 'requests'.

## Run Request 

expected output


# Sharded fork of Salesforce/codegen-6B-mono with a custom URL

This repository implements a custom 'pipeline' task for 'text-generation' for  Inference Endpoints for LLM inference using bitsandbytes quantization. The code for the customized pipeline is in the URL.

There is also a notebook included. ### expected Request payload


below is an example on how to run a request using Python and 'requests'. ## Run Request 

expected output
TAGS
#transformers #pytorch #codegen #text-generation #endpoints-template #license-bsd-3-clause #autotrain_compatible #endpoints_compatible #region-us 
 # Sharded fork of Salesforce/codegen-6B-mono with a custom URL

This repository implements a custom 'pipeline' task for 'text-generation' for  Inference Endpoints for LLM inference using bitsandbytes quantization. The code for the customized pipeline is in the URL.

There is also a notebook included. ### expected Request payload


below is an example on how to run a request using Python and 'requests'. ## Run Request 

expected output
passage: TAGS
#transformers #pytorch #codegen #text-generation #endpoints-template #license-bsd-3-clause #autotrain_compatible #endpoints_compatible #region-us 
# Sharded fork of Salesforce/codegen-6B-mono with a custom URL

This repository implements a custom 'pipeline' task for 'text-generation' for  Inference Endpoints for LLM inference using bitsandbytes quantization. The code for the customized pipeline is in the URL.

There is also a notebook included.### expected Request payload


below is an example on how to run a request using Python and 'requests'.## Run Request 

expected output