Cannot convert serving signature function to concrete function for Large Language Models (Protobuf error)
Thanks for reporting the issue.

Could you please let us know the reason behind converting the model to Tensorflow 1.x version as you mentioned here 

> `convert the LLM from Huggingface to Tensorflow v1.x` 


> Thanks for reporting the issue.
> 
> Could you please let us know the reason behind converting the model to Tensorflow 1.x version as you mentioned here
> 
> > `convert the LLM from Huggingface to Tensorflow v1.x`

Yes absolutely. We want to do int8 quantization on some large language models (LLM) such as Facebook OPT and GPT, so we download the models from Huggingface, which are in Keras format. The path we use is to convert the Keras formatted LLMs to SavedModel, parse the GraphDef and insert our quantization steps.

The problem we met is that we need to freeze the concrete function of that SavedModel to do further parsing and int8 quantization. The error occurred at https://github.com/intel/neural-compressor/blob/master/neural_compressor/model/tensorflow_model.py#L277. One related question would be https://stackoverflow.com/questions/60974077/how-to-save-keras-model-as-frozen-graph. Just as what is done shown in the example, we called `func = model.get_concrete_function()` and invoke the `convert_variables_to_constants_v2(func)`, and when we do the `convert_variables_to_constants_v2`, we met this protobuf exceed max size problem. (*** RuntimeError: size too big: 18446744072513763888 details: string length exceeds max size). However for much smaller model, we verify that we did not meet that problem. (GPT2-large failed, but GPT2-medium succeeded)

I have no idea on the number 18446744072513763888 and this looks super big. I do not think it is the size of a model or weights. The only hint is that this number is very close to 2^64. However the GPT2-large on Huggingface only occupies about 3GB, which is far less than 2^64 bit. Another hint is that the `convert_variables_to_constants_v2` (https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/framework/convert_to_constants.py#L1138) expect models with no control flow or embedding related ops. However it is out of my knowledge scope. Thanks for the clear explanation, The above steps which you have mentioned can be achieved through latest Tensorflow version 2.12.
We don't encourage users to use Tensorflow 1.x version, since it is out of the support window. We don't have any plan to fix such old versions. 
For Int8 quantization, you can refer the guide here https://www.tensorflow.org/lite/performance/post_training_quantization.

For converting your model to TFLite specific format, you can refer the guide here https://www.tensorflow.org/lite/models/convert#input_model_formats.
Saving the model saves the file in .pb format also.
Please try with the latest version and let us know the outcome of it. Thanks! Hi Sachin, TFLite is not the same path with ours, and I wonder whether there are any examples of TFLite to quantize models that have more than 1 billion parameters. Anyway, thanks for suggestions. Hi @Spycsh , For `LLM` models you can refer to our new tutorial which was published in our Google I/O annual developer conference.
https://www.tensorflow.org/lite/examples/auto_complete/overview
This tutorial walks you though Auto complete based on [PaLM](https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html) model, fine tuned on 1.5 Billion parameters.

This model is based on `Keras NLP` and then converted to `TFLite` and TFLite runtime.
This shows the quantization details and can be quantized into `Dynamic`, `FP16`, `Full Integer Quantization` specific. 

Hope this helps you. Thank you! Hi @sachinprasadhs , thanks for sharing! But it seems the first link needs a google SSO login and is invisible to outside. Is that tutorial opensource? My bad, I updated the public link, PTAL. Thanks! Thanks! We will check that. This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60571">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60571">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

No

### Source

binary

### Tensorflow Version

tf 2.12.0

### Custom Code

Yes

### OS Platform and Distribution

Ubuntu 20.04.4 LTS (Focal Fossa)

### Mobile device

_No response_

### Python version

3.8

### Bazel version

None

### GCC/Compiler version

None

### CUDA/cuDNN version

None

### GPU model and memory

None (only CPU)

### Current Behaviour?

Hi!

I am quantizing large language models. One step is to convert the LLM from Huggingface to Tensorflow v1.x. I found that when it comes to small models such as facebook/opt-125m, gpt2 and gpt2-medium, I succeeded to convert the model signatures serving function to a concrete function using `tensorflow.python.framework.convert_to_constants.convert_variables_to_constants_v2`. However, I failed when I applied that method to the serving function for some larger language models: such as gpt2-large, EleutherAI/gpt-j-6b and opt-1.3b. They shared the same error as follows:

Starting new session
[libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/stubs/stringpiece.cc:50] size too big: 18446744072513763888 details: string length exceeds max size
*** RuntimeError: size too big: 18446744072513763888 details: string length exceeds max size

Could we fix or avoid that problem?


### Standalone code to reproduce the issue

```shell
Install https://github.com/intel/neural-compressor, cd neural-compressor
pip install the requirements, and checkout to branch smooth_quant_tf, cd to 
neural-compressor/examples/tensorflow/large_language_model, pip install tensorflow and pip install datasets

and then run


export PYTHONPATH=<...>/neural-compressor
python main.py --model_name_or_path gpt2-large --int8
```
```


### Relevant log output

```shell
Gpt-large error log:

2023-05-09 23:53:43.708718: I tensorflow/core/common_runtime/executor.cc:1197] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'serving_default_input_ids' with dtype int32 and shape [?,?]
         [[{{node serving_default_input_ids}}]]
2023-05-09 23:54:23.234169: I tensorflow/core/grappler/devices.cc:66] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0
2023-05-09 23:54:23.234360: I tensorflow/core/grappler/clusters/single_machine.cc:358] Starting new session
[libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/stubs/stringpiece.cc:50] size too big: 18446744072513763888 details: string length exceeds max size
2023-05-09 23:54:42.771457: I tensorflow/core/grappler/devices.cc:66] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0
2023-05-09 23:54:42.771614: I tensorflow/core/grappler/clusters/single_machine.cc:358] Starting new session
[libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/stubs/stringpiece.cc:50] size too big: 18446744072280791549 details: string length exceeds max size
```
</details>