PaLM-Kosmos-Vision is a foundational project showcasing basic ChatGPT with vision capabilities, inviting further development for advanced use cases. Experience the potential by trying the demo
[]
PaLM-Kosmos-Vision is a foundational project showcasing basic ChatGPT with vision capabilities, inviting further development for advanced use cases. Experience the potential by trying the demo
 PaLM-Kosmos-Vision
PaLM-Kosmos-Vision is a proof-of-concept web application that serves as a basic version of ChatGPT Vision multi-modal chat. It's designed to demonstrate that integrating MultiModal models with traditional LLMs is possible, laying the groundwork for more sophisticated applications.
PaLM-Kosmos-Vision is an innovative web application that combines the capabilities of Large Language Models (LLMs) to describe images and engage users in detailed conversations about those images. This project showcases the integration of PaLM API for nuanced language understanding and the KOSMOS-2 model for generating image captions.
## Features
- Image upload functionality with real-time display.
- Automatic image captioning using the KOSMOS-2 model.
- Detailed chat interactions about images powered by PaLM API.
- Session-based chat history with a clear chat option.
- Demo is limited to 20 messages per user to optimize resource usage.
## Tech Stack
- [Streamlit](https://streamlit.io/) as the web app framework.
- [PaLM](https://github.com/google-research/palm) and [KOSMOS-2](https://huggingface.co/spaces/lucataco/kosmos-2) models for AI functionalities.
- [Replicate](https://replicate.com/) for model deployment and serving.
- [LlamaIndex](https://github.com/llama-index/llama-index) for efficient LLM orchestration.
- Python libraries such as `requests`, `Pillow`, `transformers`.
## Setup and Installation
Clone the repository and install the necessary Python packages. Ensure to set up API keys for Google Cloud and Replicate as environment variables before running the application.
## Usage
Start the app with Streamlit and upload an image to begin. The interface allows for interactive conversations about the image, with the app providing contextually relevant information.
## Demo
Try out the live demo here: [PaLM-Kosmos-Vision Demo](https://huggingface.co/spaces/AI-ANK/PaLM-Kosmos-Vision)
## Contribution
If you're interested in contributing, please review the contribution guidelines. Your ideas and pull requests are welcome!
## License
This project is licensed under the [MIT License](LICENSE).
## Credits
Developed by Harshad Suryawanshi. Connect with me on [LinkedIn](https://www.linkedin.com/in/harshadsuryawanshi/).