LISA: Reasoning Segmentation Via Large Language Model

This is the official implementation of LISA (Large-language Instructed Segmentation Assistant).

In this work, we propose a new segmentation task --- reasoning segmentation. The task is designed to output a segmentation mask given a complex and implicit query text. We establish a benchmark comprising over one thousand image-instruction pairs, incorporating intricate reasoning and world knowledge for evaluation purposes. Finally, we present LISA: Large-language Instructed Segmentation Assistant, which inherits the language generation capabilities of the multi-modal Large Language Model (LLM) while also possessing the ability to produce segmentation masks. For more details, please refer to:

LISA: Reasoning Segmentation Via Large Language Model [Paper]
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu Jiaya Jia

Experimental results

Citation

If you find this project useful in your research, please consider citing:

@article{reason-seg,
  title={LISA: Reasoning Segmentation Via Large Language Model},
  author={Xin Lai and Zhuotao Tian and Yukang Chen and Yanwei Li and Yuhui Yuan and Shu Liu and Jiaya Jia},
  journal={arXiv:},
  year={2023}
}

Acknowledgement

This work is built upon the LLaMA, SAM, and LLaVA(https://github.com/haotian-liu/LLaVA).