diff for compatibility

README.md

@@ -10,146 +10,23 @@ tags:
 - vision
 - image-text-to-text
 ---
-# LLaVA Model Card
 
-![image/png](https://cdn-uploads.huggingface.co/production/uploads/62441d1d9fdefb55a0b7d12c/FPshq08TKYD0e-qwPLDVO.png)
+<!-- header start -->
+<p align="center">
+  <img src="https://huggingface.co/datasets/FriendliAI/documentation-images/resolve/main/model-card-assets/friendliai.png" width="100%" alt="FriendliAI Logo">
+</p>
+<!-- header end -->
 
-Below is the model card of Llava model 7b, which is copied from the original Llava model card that you can find [here](https://huggingface.co/liuhaotian/llava-v1.5-13b).
 
-Check out also the Google Colab demo to run Llava on a free-tier Google Colab instance: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1qsl6cd2c8gGtEW1xV5io7S8NHh-Cp1TV?usp=sharing)
+# llava-hf/llava-1.5-7b-hf
 
-Or check out our Spaces demo! [![Open in Spaces](https://huggingface.co/datasets/huggingface/badges/resolve/main/open-in-hf-spaces-md-dark.svg)](https://huggingface.co/spaces/llava-hf/llava-4bit)
+* Model creator: [llava-hf](https://huggingface.co/llava-hf)
+* Original model: [llava-1.5-7b-hf](https://huggingface.co/llava-hf/llava-1.5-7b-hf)
 
+## Differences
 
-## Model details
-
-**Model type:**
-LLaVA is an open-source chatbot trained by fine-tuning LLaMA/Vicuna on GPT-generated multimodal instruction-following data.
-It is an auto-regressive language model, based on the transformer architecture.
-
-**Model date:**
-LLaVA-v1.5-7B was trained in September 2023.
-
-**Paper or resources for more information:**
-https://llava-vl.github.io/
-
-## How to use the model
-
-First, make sure to have `transformers >= 4.35.3`. 
-The model supports multi-image and multi-prompt generation. Meaning that you can pass multiple images in your prompt. Make sure also to follow the correct prompt template (`USER: xxx\nASSISTANT:`) and add the token `<image>` to the location where you want to query images:
-
-### Using `pipeline`:
-
-Below we used [`"llava-hf/llava-1.5-7b-hf"`](https://huggingface.co/llava-hf/llava-1.5-7b-hf) checkpoint.
-
-```python
-from transformers import pipeline
-
-pipe = pipeline("image-text-to-text", model="llava-hf/llava-1.5-7b-hf")
-messages = [
-    {
-      "role": "user",
-      "content": [
-          {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/ai2d-demo.jpg"},
-          {"type": "text", "text": "What does the label 15 represent? (1) lava (2) core (3) tunnel (4) ash cloud"},
-        ],
-    },
-]
-
-out = pipe(text=messages, max_new_tokens=20)
-print(out)
->>> [{'input_text': [{'role': 'user', 'content': [{'type': 'image', 'url': 'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/ai2d-demo.jpg'}, {'type': 'text', 'text': 'What does the label 15 represent? (1) lava (2) core (3) tunnel (4) ash cloud'}]}], 'generated_text': 'Lava'}]
-```
-
-### Using pure `transformers`:
-
-Below is an example script to run generation in `float16` precision on a GPU device:
-
-```python
-import requests
-from PIL import Image
-
-import torch
-from transformers import AutoProcessor, LlavaForConditionalGeneration
-
-model_id = "llava-hf/llava-1.5-7b-hf"
-model = LlavaForConditionalGeneration.from_pretrained(
-    model_id, 
-    torch_dtype=torch.float16, 
-    low_cpu_mem_usage=True, 
-).to(0)
-
-processor = AutoProcessor.from_pretrained(model_id)
-
-# Define a chat history and use `apply_chat_template` to get correctly formatted prompt
-# Each value in "content" has to be a list of dicts with types ("text", "image") 
-conversation = [
-    {
-
-      "role": "user",
-      "content": [
-          {"type": "text", "text": "What are these?"},
-          {"type": "image"},
-        ],
-    },
-]
-prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
-
-image_file = "http://images.cocodataset.org/val2017/000000039769.jpg"
-raw_image = Image.open(requests.get(image_file, stream=True).raw)
-inputs = processor(images=raw_image, text=prompt, return_tensors='pt').to(0, torch.float16)
-
-output = model.generate(**inputs, max_new_tokens=200, do_sample=False)
-print(processor.decode(output[0][2:], skip_special_tokens=True))
-```
-
------------
-From transformers>=v4.48, you can also pass image url or local path to the conversation history, and let the chat template handle the rest.
-Chat template will load the image for you and return inputs in `torch.Tensor` which you can pass directly to `model.generate()` 
-
-```python
-messages = [
-    {
-        "role": "user",
-        "content": [
-            {"type": "image", "url": "https://www.ilankelman.org/stopsigns/australia.jpg"}
-            {"type": "text", "text": "What is shown in this image?"},
-        ],
-    },
-]
-
-inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors"pt")
-output = model.generate(**inputs, max_new_tokens=50)
-```
-
-### Model optimization
-
-#### 4-bit quantization through `bitsandbytes` library
-
-First make sure to install `bitsandbytes`, `pip install bitsandbytes` and make sure to have access to a CUDA compatible GPU device. Simply change the snippet above with: 
-
-```diff
-model = LlavaForConditionalGeneration.from_pretrained(
-    model_id, 
-    torch_dtype=torch.float16, 
-    low_cpu_mem_usage=True,
-+   load_in_4bit=True
-)
-```
-
-#### Use Flash-Attention 2 to further speed-up generation
-
-First make sure to install `flash-attn`. Refer to the [original repository of Flash Attention](https://github.com/Dao-AILab/flash-attention) regarding that package installation. Simply change the snippet above with: 
-
-```diff
-model = LlavaForConditionalGeneration.from_pretrained(
-    model_id, 
-    torch_dtype=torch.float16, 
-    low_cpu_mem_usage=True,
-+   use_flash_attention_2=True
-).to(0)
-```
+* Added missing chat template to tokenizer_config.json
 
 ## License
-Llama 2 is licensed under the LLAMA 2 Community License, 
-Copyright (c) Meta Platforms, Inc. All Rights Reserved.
+
+Refer to the license of the original model card.

tokenizer_config.json

@@ -45,6 +45,7 @@
     }
   },
   "bos_token": "<s>",
+  "chat_template": "{% for message in messages %}{% if message['role'] != 'system' %}{{ message['role'].upper() + ': '}}{% endif %}{# Render all images first #}{% for content in message['content'] | selectattr('type', 'equalto', 'image') %}{{ '<image>\n' }}{% endfor %}{# Render all text next #}{% if message['role'] != 'assistant' %}{% for content in message['content'] | selectattr('type', 'equalto', 'text') %}{{ content['text'] + ' '}}{% endfor %}{% else %}{% for content in message['content'] | selectattr('type', 'equalto', 'text') %}{% generation %}{{ content['text'] + ' '}}{% endgeneration %}{% endfor %}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ 'ASSISTANT:' }}{% endif %}",
   "clean_up_tokenization_spaces": false,
   "eos_token": "</s>",
   "extra_special_tokens": {
@@ -61,4 +62,4 @@
   "trust_remote_code": false,
   "unk_token": "<unk>",
   "use_default_system_prompt": false
-}
+}