Spaces:
Running
Running
| title: README | |
| emoji: 🐢 | |
| colorFrom: red | |
| colorTo: yellow | |
| sdk: static | |
| pinned: false | |
| Sightation Counts: Leveraging Sighted User Feedback in Building a BLV-aligned Dataset of Diagram Descriptions | |
| ============================================================================================================== | |
| Wan Ju Kang, Eunki Kim, Na Min An, Sangryul Kim, Haemin Choi, Ki Hoon Kwak, James Thorne | |
| ## 📄 [Paper](URL) 💻 [Code](URL) | |
| Hello, we are a team of researchers based in KAIST AI working on accessible visualization. | |
| In specific, we compiled a diagram description dataset for the blind and low-vision individuals. | |
| We worked in close cooperation with two schools for the blind, as well as over 30 sighted annotators, and we are grateful for their contribution. | |
| Check out our preprint [coming soon], and feel free to contact us at [email protected]. | |
| --------------------------------------- | |
| ## Abstract | |
| > Often, the needs and visual abilities differ between the annotator group and the end user | |
| group. Generating detailed diagram descriptions for blind and low-vision (BLV) users is one such challenging domain. | |
| Sighted annotators could describe visuals with ease, but existing studies have shown that direct generations by them are costly, bias-prone, and somewhat | |
| lacking by BLV standards. In this study, we ask sighted individuals to assess—rather than produce—diagram descriptions generated by vision-language models (VLM) that have been | |
| guided with latent supervision via a multi-pass inference. The sighted assessments prove effective and useful to professional educators | |
| who are themselves BLV and teach visually impaired learners. We release SIGHTATION, a collection of diagram description datasets | |
| spanning 5k diagrams and 137k samples for completion, preference, retrieval, question answering, and reasoning training purposes and | |
| demonstrate their fine-tuning potential in various downstream tasks. | |
| ## Sightation Collection | |
| - SightationCompletions | |
| - SightationPreference | |
| - SightationRetrieval | |
| - SightationVQA | |
| - SightationReasoning | |
| <img src="https://cdn-uploads.huggingface.co/production/uploads/67a86f66c6f66e2fa5888b41/cNshK4QAdiNMqk7x6J6j7.png" width="70%" height="70%" title="visual_abstract" alt="visual_abstract"></img> | |
| The key benefit of utilizing sighted user feedback lies in their assessments that are based on solid visual | |
| grounding. The compiled assessments prove an effective training substance for steering VLMs towards more | |
| accessible descriptions. | |
| <img src="https://cdn-uploads.huggingface.co/production/uploads/67a86f66c6f66e2fa5888b41/8oYvtq7dtv_Ck-U6OlcAE.png" width="50%" height="50%" title="dimensions_assignment" alt="dimensions_assignment"></img> | |
| The description qualities assessed by their respective evaluator groups. | |
| ## Results | |
| <img src="https://cdn-uploads.huggingface.co/production/uploads/67a86f66c6f66e2fa5888b41/094e9Hw7lauvT1tshg1Wj.png" width="60%" height="60%" title="spider_chart" alt="spider_chart"></img> | |
| Tuning VLMs on Sightation enhanced various qualities of the diagram descriptions, evaluated by BLV educators, and shown here as normalized ratings averaged in each aspect. | |
| The capability of the dataset is most strongly pronounced with Qwen2-VL-2B model, shown above. | |
| ## BibTeX | |
| If you find this work useful for your research, please cite: | |
| ```bash | |
| @inproceedings{ | |
| } | |
| ``` | |