Update README.md

README.md

@@ -9,4 +9,206 @@ metrics:
 base_model:
 - openbmb/MiniCPM-V-2_6
 pipeline_tag: visual-question-answering
----
+---
+---
+license: cc-by-nc-sa-4.0
+datasets:
+- PengxiangLi/MAT
+language:
+- en
+metrics:
+- accuracy
+base_model:
+- openbmb/MiniCPM-V-2_6
+pipeline_tag: visual-question-answering
+---
+
+---
+pipeline_tag: image-text-to-text
+datasets:
+- openbmb/RLAIF-V-Dataset
+library_name: transformers
+language:
+- multilingual
+tags:
+- minicpm-v
+- vision
+- ocr
+- multi-image
+- video
+- custom_code
+---
+
+<h1>Multi-modal Agent Tuning: Building a VLM-Driven Agent for Efficient Tool Usage</h1>
+
+[GitHub](https://github.com/mat-agent/MAT-Agent.git) | [Project](https://mat-agent.github.io/)</a> 
+
+## MAT-MiniCPM-V 2.6
+
+This model is a fine-tuned version of the [MiniCPM V2.6 7B](https://huggingface.co/openbmb/MiniCPM-V-2_6) model on the MM-traj dataset. On GTA and GAIA benchmarks, it achieved improvements of ​**18.59%**​ and ​**7.78%**​ respectively compared to the non-fine-tuned baseline.
+
+## Usage
+Our model inherits the inference architecture from [MiniCPM-V-2.6](https://huggingface.co/openbmb/MiniCPM-V-2_6). The following implementation is adapted from their original inference code with full compatibility.
+
+Requirements (tested on Python 3.10):
+```
+Pillow==10.1.0
+torch==2.1.2
+torchvision==0.16.2
+transformers==4.40.0
+sentencepiece==0.1.99
+decord
+```
+
+### Basic Inference
+```python
+# test.py
+import torch
+from PIL import Image
+from transformers import AutoModel, AutoTokenizer
+
+# Load our fine-tuned model (based on MiniCPM-V-2.6 architecture)
+model = AutoModel.from_pretrained('PengxiangLi/MAT', trust_remote_code=True,
+    attn_implementation='sdpa', torch_dtype=torch.bfloat16)  # Maintain original implementation choices
+model = model.eval().cuda()
+tokenizer = AutoTokenizer.from_pretrained('PengxiangLi/MAT', trust_remote_code=True)
+
+image = Image.open('xx.jpg').convert('RGB')
+question = 'What is in the image?'
+msgs = [{'role': 'user', 'content': [image, question]}]
+
+# The chat interface follows MiniCPM's original implementation
+response = model.chat(
+    image=None,
+    msgs=msgs,
+    tokenizer=tokenizer
+)
+print(response)
+
+## Streaming output (inherited from MiniCPM's implementation)
+response_stream = model.chat(
+    image=None,
+    msgs=msgs,
+    tokenizer=tokenizer,
+    sampling=True,
+    stream=True
+)
+
+generated_text = ""
+for new_text in response_stream:
+    generated_text += new_text
+    print(new_text, flush=True, end='')
+```
+
+### Multi-image Chat 
+<details>
+<summary>Implementation adapted from MiniCPM's original multi-image handling</summary>
+
+```python
+import torch
+from PIL import Image
+from transformers import AutoModel, AutoTokenizer
+
+model = AutoModel.from_pretrained('PengxiangLi/MAT', trust_remote_code=True,
+    attn_implementation='sdpa', torch_dtype=torch.bfloat16)
+model = model.eval().cuda()
+tokenizer = AutoTokenizer.from_pretrained('PengxiangLi/MAT', trust_remote_code=True)
+
+# The message format follows MiniCPM's original schema
+image1 = Image.open('image1.jpg').convert('RGB')
+image2 = Image.open('image2.jpg').convert('RGB')
+question = 'Compare the two images...'
+
+msgs = [{'role': 'user', 'content': [image1, image2, question]}]
+
+# Using the original chat interface design
+answer = model.chat(
+    image=None,
+    msgs=msgs,
+    tokenizer=tokenizer
+)
+print(answer)
+```
+</details>
+
+### Few-shot Learning
+<details>
+<summary>Adapted from MiniCPM's few-shot implementation</summary>
+
+```python
+import torch
+from PIL import Image
+from transformers import AutoModel, AutoTokenizer
+
+# Maintain original model loading parameters
+model = AutoModel.from_pretrained('PengxiangLi/MAT', trust_remote_code=True,
+    attn_implementation='sdpa', torch_dtype=torch.bfloat16)
+model = model.eval().cuda()
+tokenizer = AutoTokenizer.from_pretrained('PengxiangLi/MAT', trust_remote_code=True)
+
+# Following MiniCPM's message structure
+question = "production date" 
+image1 = Image.open('example1.jpg').convert('RGB')
+answer1 = "2023.08.04"
+image2 = Image.open('example2.jpg').convert('RGB')
+answer2 = "2007.04.24"
+image_test = Image.open('test.jpg').convert('RGB')
+
+msgs = [
+    {'role': 'user', 'content': [image1, question]}, 
+    {'role': 'assistant', 'content': [answer1]},
+    {'role': 'user', 'content': [image2, question]}, 
+    {'role': 'assistant', 'content': [answer2]},
+    {'role': 'user', 'content': [image_test, question]}
+]
+
+# Using the unmodified chat interface from original implementation
+answer = model.chat(
+    image=None,
+    msgs=msgs,
+    tokenizer=tokenizer
+)
+print(answer)
+```
+</details>
+
+#### Implementation Notes:
+1. All core inference logic is directly inherited from [MiniCPM-V-2.6](https://huggingface.co/openbmb/MiniCPM-V-2_6)
+2. The `chat()` interface remains unchanged from the original implementation
+3. Model loading parameters maintain compatibility with the base architecture
+4. Message formatting follows MiniCPM's original schema
+ 
+
+ 
+
+
+## License
+
+#### Model License
+- The code in this repository is licensed under the ​**[Apache-2.0 License](https://github.com/OpenBMB/MiniCPM/blob/main/LICENSE)**.
+- Usage of our fine-tuned MiniCPM-based model weights must strictly adhere to the ​**[MiniCPM Model License](https://github.com/OpenBMB/MiniCPM/blob/main/MiniCPM%20Model%20License.md)**.
+
+#### Usage Terms
+- ​**Academic Research**: The model weights are freely available for academic use without restrictions.
+- ​**Commercial Use**: 
+  - After completing the official ​**[registration questionnaire](https://modelbest.feishu.cn/share/base/form/shrcnpV5ZT9EJ6xYjh3Kx0J6v8g)**​ and obtaining authorization, the MiniCPM-V 2.6 based weights (including our fine-tuned version) are available for commercial use free of charge.
+  - Commercial users must maintain compliance with all terms outlined in the MiniCPM Model License.
+
+#### Inheritance Clause
+As a derivative work of MiniCPM, our model inherits and is bound by all original licensing requirements from the base model. Users are responsible for ensuring compliance with both our terms and the upstream MiniCPM license terms.
+
+
+
+
+## Citation
+
+If you find our work helpful, please consider citing our papers 📝 and liking this project ❤️！
+
+```bib
+@article{gao2024multi,
+  title={Multi-modal Agent Tuning: Building a VLM-Driven Agent for Efficient Tool Usage},
+  author={Gao, Zhi and Zhang, Bofei and Li, Pengxiang and Ma, Xiaojian and Yuan, Tao and Fan, Yue and Wu, Yuwei and Jia, Yunde and Zhu, Song-Chun and Li, Qing},
+  journal={arXiv preprint arXiv:2412.15606},
+  year={2024}
+}
+```