Sync widgets demo

packages/tasks/src/tasks/image-to-text/about.md

@@ -27,6 +27,19 @@ captioner("https://huggingface.co/datasets/Narsil/image_dummy/resolve/main/parro
 ## [{'generated_text': 'two birds are standing next to each other '}]
 ```
 
+### Conversation about the Image
+
+Some text generation models also take image inputs. These are called vision language models. You can use `image-to-text` pipeline to use these models like below.
+
+```python
+from transformers import pipeline
+
+mm_pipeline = pipeline("image-to-text",model="llava-hf/llava-1.5-7b-hf")
+mm_pipeline("https://huggingface.co/spaces/llava-hf/llava-4bit/resolve/main/examples/baklava.png", "How to make this pastry?")
+
+## [{'generated_text': 'To create these pastries, you will need a few key ingredients and tools. Firstly, gather the dough by combining flour with water in your mixing bowl until it forms into an elastic ball that can be easily rolled out on top of another surface or table without breaking apart (like pizza).'}]
+```
+
 ### OCR
 
 This code snippet uses Microsoft’s TrOCR, an encoder-decoder model consisting of an image Transformer encoder and a text Transformer decoder for state-of-the-art optical character recognition (OCR) on single-text line images.

packages/tasks/src/tasks/image-to-text/data.ts

@@ -42,6 +42,10 @@ const taskData: TaskDataCustom = {
 			description: "A strong optical character recognition model.",
 			id: "facebook/nougat-base",
 		},
+		{
+			description: "A powerful model that lets you have a conversation with the image.",
+			id: "llava-hf/llava-1.5-7b-hf",
+		},
 	],
 	spaces: [
 		{

packages/tasks/src/tasks/index.ts

@@ -51,8 +51,8 @@ export const TASKS_MODEL_LIBRARIES: Record<PipelineType, ModelLibraryKey[]> = {
 	"graph-ml": ["transformers"],
 	"image-classification": ["keras", "timm", "transformers", "transformers.js"],
 	"image-segmentation": ["transformers", "transformers.js"],
-	"image-to-image": ["diffusers", "transformers.js"],
-	"image-to-text": ["transformers.js"],
+	"image-to-image": ["diffusers", "transformers", "transformers.js"],
+	"image-to-text": ["transformers", "transformers.js"],
 	"image-to-video": ["diffusers"],
 	"video-classification": ["transformers"],
 	"mask-generation": ["transformers"],

packages/tasks/src/tasks/text-generation/about.md

@@ -42,6 +42,10 @@ When it comes to text generation, the underlying language model can come in seve
 
 - **Human feedback models:** these models extend base and instruction-trained models by incorporating human feedback that rates the quality of the generated text according to criteria like [helpfulness, honesty, and harmlessness](https://arxiv.org/abs/2112.00861). The human feedback is then combined with an optimization technique like reinforcement learning to align the original model to be closer with human preferences. The overall methodology is often called [Reinforcement Learning from Human Feedback](https://huggingface.co/blog/rlhf), or RLHF for short. [Llama2-Chat](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf) is an open-source model aligned through human feedback.
 
+## Text Generation from Image and Text
+
+There are language models that can input both text and image and output text, called vision language models. [LLaVA](https://huggingface.co/llava-hf/llava-1.5-7b-hf) and [BLIP-2](https://huggingface.co/Salesforce/blip2-opt-2.7b) are good examples. Although they work just like other language models by means of input parameters for generation, since they also take input images, you can use them with `image-to-text` pipeline. You can find information about the pipeline in [image-to-text](https://huggingface.co/tasks/image-to-text) task page.
+
 ## Inference
 
 You can use the 🤗 Transformers library `text-generation` pipeline to do inference with Text Generation models. It takes an incomplete text and returns multiple outputs with which the text can be completed.