Delete MiniGPT-Med-github

MiniGPT-Med-github/README.md

@@ -1,51 +0,0 @@
-# MiniGPT-Med: Large Language Model as a General Interface for Radiology Diagnosis
-Asma Alkhaldi, Raneem Alnajim, Layan Alabdullatef, Rawan Alyahya, Jun Chen, Deyao Zhu, Ahmed Alsinan, Mohamed Elhoseiny
-
-*Saudi Data and Artificial Intelligence Authority (SDAIA) and King Abdullah University of Science and Technology (KAUST)*
-
-## Installation
-```
-git clone https://github.com/Vision-CAIR/MiniGPT-Med
-cd MiniGPT-Med
-conda env create -f environment.yml
-conda activate miniGPT-Med
-```
-
-## Download miniGPT-Med trained model weights
-
-* miniGPT-Med's weights [miniGPT-Med Model](https://drive.google.com/file/d/1kjGLk6s9LsBmXfLWQFCdlwF3aul08Cl8/view?usp=sharing)
-
-* Then modify line 8 at miniGPT-Med/eval_configs/minigptv2_eval.yaml to be the path of miniGPT-Med weight.
-
-## Prepare weight for LLMs
-
-### Llama2 Version
-
-```shell
-git clone https://huggingface.co/meta-llama/Llama-2-13b-chat-hf
-```
-
-Then modify line 14 at miniGPT-Med/minigpt4/configs/models/minigpt_v2.yaml to be the path of Llama-2-13b-chat-hf.
-
-## Launching Demo Locally
-
-```
-python demo.py --cfg-path eval_configs/minigptv2_eval.yaml --gpu-id 0
-```
-
-## Dataset
-| Dataset | Images  | json file| 
-|---------|---------|----------|
-| MIMIC   |[Download](https://physionet.org/content/mimiciii/1.4/) | [Download](https://drive.google.com/drive/folders/1nZhdfNoh7fkx7CWvf0_47_OLv3tA2m3o?usp=sharing) |
-| NLST    |[Download](https://wiki.cancerimagingarchive.net/display/NLST)| [Downlaod](https://drive.google.com/drive/folders/1OKgMTaGLu_dWRuco6JipYzezw3oNwgaz?usp=sharing) |
-|SLAKE    |[Downlaod](https://www.med-vqa.com/slake/) |[Download](https://drive.google.com/drive/folders/1vstjmfRbKahSAsi_b6FmTQiuolvgO8oC?usp=sharing)|
-|RSNA     |[Downlaod](https://www.rsna.org/rsnai/ai-image-challenge/rsna-pneumonia-detection-challenge-2018) | [Download](https://drive.google.com/drive/folders/1wkXPvUNqda6jWAIduyiVJkS3Tx7P7td8?usp=sharing) |
-|Rad-VQA  |[Downalod](https://osf.io/89kps/) |[Download](https://drive.google.com/drive/folders/1ING6Dodwk2DU_t4GHQYudNFMMg9OMfBQ?usp=sharing) |
-
-## Acknowledgement
-
-- MiniGPT-4
-- Lavis
-- Vicuna
-- Falcon
-- Llama 2

MiniGPT-Med-github/dcgm/bash/34649895/dcgm-gpu-stats-gpu202-02-r-34649895.out

@@ -1,39 +0,0 @@
-Successfully retrieved statistics for job: 34649895. 
-+------------------------------------------------------------------------------+
-| GPU ID: 0                                                                    |
-+====================================+=========================================+
-|-----  Execution Stats  ------------+-----------------------------------------|
-| Start Time                         | Tue Jul  9 09:29:46 2024                |
-| End Time                           | Wed Jul 10 09:30:32 2024                |
-| Total Execution Time (sec)         | 86445.3                                 |
-| No. of Processes                   | 1                                       |
-+-----  Performance Stats  ----------+-----------------------------------------+
-| Energy Consumed (Joules)           | 232291                                  |
-| Power Usage (Watts)                | Avg: 65.6704, Max: 84.315, Min: 61.555  |
-| Max GPU Memory Used (bytes)        | 10104078336                             |
-| SM Clock (MHz)                     | Avg: 595, Max: 1155, Min: 210           |
-| Memory Clock (MHz)                 | Avg: 1593, Max: 1593, Min: 1593         |
-| SM Utilization (%)                 | Avg: 0, Max: 0, Min: 0                  |
-| Memory Utilization (%)             | Avg: 0, Max: 0, Min: 0                  |
-| PCIe Rx Bandwidth (megabytes)      | Avg: N/A, Max: N/A, Min: N/A            |
-| PCIe Tx Bandwidth (megabytes)      | Avg: N/A, Max: N/A, Min: N/A            |
-+-----  Event Stats  ----------------+-----------------------------------------+
-| Single Bit ECC Errors              | 0                                       |
-| Double Bit ECC Errors              | 0                                       |
-| PCIe Replay Warnings               | 0                                       |
-| Critical XID Errors                | 0                                       |
-+-----  Slowdown Stats  -------------+-----------------------------------------+
-| Due to - Power (%)                 | 0                                       |
-|        - Thermal (%)               | 0                                       |
-|        - Reliability (%)           | Not Supported                           |
-|        - Board Limit (%)           | Not Supported                           |
-|        - Low Utilization (%)       | Not Supported                           |
-|        - Sync Boost (%)            | 0                                       |
-+--  Compute Process Utilization  ---+-----------------------------------------+
-| PID                                | 1548651                                 |
-|     Avg SM Utilization (%)         | 0                                       |
-|     Avg Memory Utilization (%)     | 0                                       |
-+-----  Overall Health  -------------+-----------------------------------------+
-| Overall Health                     | Healthy                                 |
-+------------------------------------+-----------------------------------------+
-

MiniGPT-Med-github/dcgm/bash/34673507/dcgm-gpu-stats-gpu201-23-l-34673507.out

@@ -1,39 +0,0 @@
-Successfully retrieved statistics for job: 34673507. 
-+------------------------------------------------------------------------------+
-| GPU ID: 1                                                                    |
-+====================================+=========================================+
-|-----  Execution Stats  ------------+-----------------------------------------|
-| Start Time                         | Fri Jul 12 11:48:45 2024                |
-| End Time                           | Sat Jul 13 11:49:39 2024                |
-| Total Execution Time (sec)         | 86454.5                                 |
-| No. of Processes                   | 1                                       |
-+-----  Performance Stats  ----------+-----------------------------------------+
-| Energy Consumed (Joules)           | 252136                                  |
-| Power Usage (Watts)                | Avg: 69.7762, Max: 70.022, Min: 69.151  |
-| Max GPU Memory Used (bytes)        | 10104078336                             |
-| SM Clock (MHz)                     | Avg: 1157, Max: 1410, Min: 1155         |
-| Memory Clock (MHz)                 | Avg: 1593, Max: 1593, Min: 1593         |
-| SM Utilization (%)                 | Avg: 0, Max: 0, Min: 0                  |
-| Memory Utilization (%)             | Avg: 0, Max: 0, Min: 0                  |
-| PCIe Rx Bandwidth (megabytes)      | Avg: N/A, Max: N/A, Min: N/A            |
-| PCIe Tx Bandwidth (megabytes)      | Avg: N/A, Max: N/A, Min: N/A            |
-+-----  Event Stats  ----------------+-----------------------------------------+
-| Single Bit ECC Errors              | 0                                       |
-| Double Bit ECC Errors              | 0                                       |
-| PCIe Replay Warnings               | 0                                       |
-| Critical XID Errors                | 0                                       |
-+-----  Slowdown Stats  -------------+-----------------------------------------+
-| Due to - Power (%)                 | 0                                       |
-|        - Thermal (%)               | 0                                       |
-|        - Reliability (%)           | Not Supported                           |
-|        - Board Limit (%)           | Not Supported                           |
-|        - Low Utilization (%)       | Not Supported                           |
-|        - Sync Boost (%)            | 0                                       |
-+--  Compute Process Utilization  ---+-----------------------------------------+
-| PID                                | 2527521                                 |
-|     Avg SM Utilization (%)         | 0                                       |
-|     Avg Memory Utilization (%)     | 0                                       |
-+-----  Overall Health  -------------+-----------------------------------------+
-| Overall Health                     | Healthy                                 |
-+------------------------------------+-----------------------------------------+
-

MiniGPT-Med-github/dcgm/bash/34676162/dcgm-gpu-stats-gpu201-23-l-34676162.out

@@ -1,39 +0,0 @@
-Successfully retrieved statistics for job: 34676162. 
-+------------------------------------------------------------------------------+
-| GPU ID: 3                                                                    |
-+====================================+=========================================+
-|-----  Execution Stats  ------------+-----------------------------------------|
-| Start Time                         | Sun Jul 14 07:57:08 2024                |
-| End Time                           | Mon Jul 15 07:57:59 2024                |
-| Total Execution Time (sec)         | 86450.6                                 |
-| No. of Processes                   | 1                                       |
-+-----  Performance Stats  ----------+-----------------------------------------+
-| Energy Consumed (Joules)           | 249997                                  |
-| Power Usage (Watts)                | Avg: 82.8167, Max: 86.615, Min: 70.491  |
-| Max GPU Memory Used (bytes)        | 10104078336                             |
-| SM Clock (MHz)                     | Avg: 1352, Max: 1410, Min: 1080         |
-| Memory Clock (MHz)                 | Avg: 1593, Max: 1593, Min: 1593         |
-| SM Utilization (%)                 | Avg: 0, Max: 0, Min: 0                  |
-| Memory Utilization (%)             | Avg: 0, Max: 0, Min: 0                  |
-| PCIe Rx Bandwidth (megabytes)      | Avg: N/A, Max: N/A, Min: N/A            |
-| PCIe Tx Bandwidth (megabytes)      | Avg: N/A, Max: N/A, Min: N/A            |
-+-----  Event Stats  ----------------+-----------------------------------------+
-| Single Bit ECC Errors              | 0                                       |
-| Double Bit ECC Errors              | 0                                       |
-| PCIe Replay Warnings               | 0                                       |
-| Critical XID Errors                | 0                                       |
-+-----  Slowdown Stats  -------------+-----------------------------------------+
-| Due to - Power (%)                 | 0                                       |
-|        - Thermal (%)               | 0                                       |
-|        - Reliability (%)           | Not Supported                           |
-|        - Board Limit (%)           | Not Supported                           |
-|        - Low Utilization (%)       | Not Supported                           |
-|        - Sync Boost (%)            | 0                                       |
-+--  Compute Process Utilization  ---+-----------------------------------------+
-| PID                                | 3048225                                 |
-|     Avg SM Utilization (%)         | 0                                       |
-|     Avg Memory Utilization (%)     | 0                                       |
-+-----  Overall Health  -------------+-----------------------------------------+
-| Overall Health                     | Healthy                                 |
-+------------------------------------+-----------------------------------------+
-

MiniGPT-Med-github/dcgm/bash/34691276/dcgm-gpu-stats-gpu201-09-l-34691276.out

@@ -1,42 +0,0 @@
-Successfully retrieved statistics for job: 34691276. 
-+------------------------------------------------------------------------------+
-| GPU ID: 0                                                                    |
-+====================================+=========================================+
-|-----  Execution Stats  ------------+-----------------------------------------|
-| Start Time                         | Tue Jul 16 08:21:43 2024                |
-| End Time                           | Tue Jul 16 21:44:34 2024                |
-| Total Execution Time (sec)         | 48170.9                                 |
-| No. of Processes                   | 2                                       |
-+-----  Performance Stats  ----------+-----------------------------------------+
-| Energy Consumed (Joules)           | 222759                                  |
-| Power Usage (Watts)                | Avg: 61.4158, Max: 61.683, Min: 61.349  |
-| Max GPU Memory Used (bytes)        | 10806624256                             |
-| SM Clock (MHz)                     | Avg: 210, Max: 225, Min: 210            |
-| Memory Clock (MHz)                 | Avg: 1593, Max: 1593, Min: 1593         |
-| SM Utilization (%)                 | Avg: 0, Max: 0, Min: 0                  |
-| Memory Utilization (%)             | Avg: 0, Max: 0, Min: 0                  |
-| PCIe Rx Bandwidth (megabytes)      | Avg: N/A, Max: N/A, Min: N/A            |
-| PCIe Tx Bandwidth (megabytes)      | Avg: N/A, Max: N/A, Min: N/A            |
-+-----  Event Stats  ----------------+-----------------------------------------+
-| Single Bit ECC Errors              | 0                                       |
-| Double Bit ECC Errors              | 0                                       |
-| PCIe Replay Warnings               | 0                                       |
-| Critical XID Errors                | 0                                       |
-+-----  Slowdown Stats  -------------+-----------------------------------------+
-| Due to - Power (%)                 | 0                                       |
-|        - Thermal (%)               | 0                                       |
-|        - Reliability (%)           | Not Supported                           |
-|        - Board Limit (%)           | Not Supported                           |
-|        - Low Utilization (%)       | Not Supported                           |
-|        - Sync Boost (%)            | 0                                       |
-+--  Compute Process Utilization  ---+-----------------------------------------+
-| PID                                | 1958147                                 |
-|     Avg SM Utilization (%)         | 1                                       |
-|     Avg Memory Utilization (%)     | 0                                       |
-| PID                                | 2068287                                 |
-|     Avg SM Utilization (%)         | 0                                       |
-|     Avg Memory Utilization (%)     | 0                                       |
-+-----  Overall Health  -------------+-----------------------------------------+
-| Overall Health                     | Healthy                                 |
-+------------------------------------+-----------------------------------------+
-

MiniGPT-Med-github/dcgm/bash/34709014/dcgm-gpu-stats-gpu109-16-l-34709014.out

@@ -1,39 +0,0 @@
-Successfully retrieved statistics for job: 34709014. 
-+------------------------------------------------------------------------------+
-| GPU ID: 3                                                                    |
-+====================================+=========================================+
-|-----  Execution Stats  ------------+-----------------------------------------|
-| Start Time                         | Thu Jul 18 07:54:11 2024                |
-| End Time                           | Fri Jul 19 07:55:07 2024                |
-| Total Execution Time (sec)         | 86456.3                                 |
-| No. of Processes                   | 1                                       |
-+-----  Performance Stats  ----------+-----------------------------------------+
-| Energy Consumed (Joules)           | 245376                                  |
-| Power Usage (Watts)                | Avg: 67.8347, Max: 68.156, Min: 67.563  |
-| Max GPU Memory Used (bytes)        | 10582228992                             |
-| SM Clock (MHz)                     | Avg: 1161, Max: 1410, Min: 1155         |
-| Memory Clock (MHz)                 | Avg: 1593, Max: 1593, Min: 1593         |
-| SM Utilization (%)                 | Avg: 0, Max: 0, Min: 0                  |
-| Memory Utilization (%)             | Avg: 0, Max: 0, Min: 0                  |
-| PCIe Rx Bandwidth (megabytes)      | Avg: N/A, Max: N/A, Min: N/A            |
-| PCIe Tx Bandwidth (megabytes)      | Avg: N/A, Max: N/A, Min: N/A            |
-+-----  Event Stats  ----------------+-----------------------------------------+
-| Single Bit ECC Errors              | 0                                       |
-| Double Bit ECC Errors              | 0                                       |
-| PCIe Replay Warnings               | 0                                       |
-| Critical XID Errors                | 0                                       |
-+-----  Slowdown Stats  -------------+-----------------------------------------+
-| Due to - Power (%)                 | 0                                       |
-|        - Thermal (%)               | 0                                       |
-|        - Reliability (%)           | Not Supported                           |
-|        - Board Limit (%)           | Not Supported                           |
-|        - Low Utilization (%)       | Not Supported                           |
-|        - Sync Boost (%)            | 0                                       |
-+--  Compute Process Utilization  ---+-----------------------------------------+
-| PID                                | 4005887                                 |
-|     Avg SM Utilization (%)         | 0                                       |
-|     Avg Memory Utilization (%)     | 0                                       |
-+-----  Overall Health  -------------+-----------------------------------------+
-| Overall Health                     | Healthy                                 |
-+------------------------------------+-----------------------------------------+
-

MiniGPT-Med-github/dcgm/bash/34721198/dcgm-gpu-stats-gpu203-23-r-34721198.out

@@ -1,57 +0,0 @@
-Successfully retrieved statistics for job: 34721198. 
-+------------------------------------------------------------------------------+
-| GPU ID: 1                                                                    |
-+====================================+=========================================+
-|-----  Execution Stats  ------------+-----------------------------------------|
-| Start Time                         | Fri Jul 19 21:34:44 2024                |
-| End Time                           | Sat Jul 20 00:01:06 2024                |
-| Total Execution Time (sec)         | 8782.23                                 |
-| No. of Processes                   | 7                                       |
-+-----  Performance Stats  ----------+-----------------------------------------+
-| Energy Consumed (Joules)           | 225540                                  |
-| Power Usage (Watts)                | Avg: 75.9496, Max: 87.541, Min: 65.792  |
-| Max GPU Memory Used (bytes)        | 13356761088                             |
-| SM Clock (MHz)                     | Avg: 210, Max: 210, Min: 210            |
-| Memory Clock (MHz)                 | Avg: 1593, Max: 1593, Min: 1593         |
-| SM Utilization (%)                 | Avg: 0, Max: 0, Min: 0                  |
-| Memory Utilization (%)             | Avg: 0, Max: 0, Min: 0                  |
-| PCIe Rx Bandwidth (megabytes)      | Avg: N/A, Max: N/A, Min: N/A            |
-| PCIe Tx Bandwidth (megabytes)      | Avg: N/A, Max: N/A, Min: N/A            |
-+-----  Event Stats  ----------------+-----------------------------------------+
-| Single Bit ECC Errors              | 0                                       |
-| Double Bit ECC Errors              | 0                                       |
-| PCIe Replay Warnings               | 0                                       |
-| Critical XID Errors                | 0                                       |
-+-----  Slowdown Stats  -------------+-----------------------------------------+
-| Due to - Power (%)                 | 0                                       |
-|        - Thermal (%)               | 0                                       |
-|        - Reliability (%)           | Not Supported                           |
-|        - Board Limit (%)           | Not Supported                           |
-|        - Low Utilization (%)       | Not Supported                           |
-|        - Sync Boost (%)            | 0                                       |
-+--  Compute Process Utilization  ---+-----------------------------------------+
-| PID                                | 866309                                  |
-|     Avg SM Utilization (%)         | 0                                       |
-|     Avg Memory Utilization (%)     | 0                                       |
-| PID                                | 866955                                  |
-|     Avg SM Utilization (%)         | 1                                       |
-|     Avg Memory Utilization (%)     | 0                                       |
-| PID                                | 868076                                  |
-|     Avg SM Utilization (%)         | 0                                       |
-|     Avg Memory Utilization (%)     | 0                                       |
-| PID                                | 868638                                  |
-|     Avg SM Utilization (%)         | 5                                       |
-|     Avg Memory Utilization (%)     | 0                                       |
-| PID                                | 869519                                  |
-|     Avg SM Utilization (%)         | 0                                       |
-|     Avg Memory Utilization (%)     | 0                                       |
-| PID                                | 871043                                  |
-|     Avg SM Utilization (%)         | 1                                       |
-|     Avg Memory Utilization (%)     | 0                                       |
-| PID                                | 871322                                  |
-|     Avg SM Utilization (%)         | 0                                       |
-|     Avg Memory Utilization (%)     | 0                                       |
-+-----  Overall Health  -------------+-----------------------------------------+
-| Overall Health                     | Healthy                                 |
-+------------------------------------+-----------------------------------------+
-

MiniGPT-Med-github/dcgm/bash/34734121/dcgm-gpu-stats-gpu201-23-l-34734121.out

@@ -1,35 +0,0 @@
-Successfully retrieved statistics for job: 34734121. 
-+------------------------------------------------------------------------------+
-| GPU ID: 3                                                                    |
-+====================================+=========================================+
-|-----  Execution Stats  ------------+-----------------------------------------|
-| Start Time                         | Tue Jul 23 11:47:49 2024                |
-| End Time                           | Tue Jul 23 13:47:51 2024                |
-| Total Execution Time (sec)         | 7202.22                                 |
-| No. of Processes                   | 0                                       |
-+-----  Performance Stats  ----------+-----------------------------------------+
-| Energy Consumed (Joules)           | 226384                                  |
-| Power Usage (Watts)                | Avg: 62.6807, Max: 81.445, Min: 62.015  |
-| Max GPU Memory Used (bytes)        | 0                                       |
-| SM Clock (MHz)                     | Avg: 220, Max: 1410, Min: 210           |
-| Memory Clock (MHz)                 | Avg: 1593, Max: 1593, Min: 1593         |
-| SM Utilization (%)                 | Avg: 0, Max: 0, Min: 0                  |
-| Memory Utilization (%)             | Avg: 0, Max: 0, Min: 0                  |
-| PCIe Rx Bandwidth (megabytes)      | Avg: N/A, Max: N/A, Min: N/A            |
-| PCIe Tx Bandwidth (megabytes)      | Avg: N/A, Max: N/A, Min: N/A            |
-+-----  Event Stats  ----------------+-----------------------------------------+
-| Single Bit ECC Errors              | 0                                       |
-| Double Bit ECC Errors              | 0                                       |
-| PCIe Replay Warnings               | 0                                       |
-| Critical XID Errors                | 0                                       |
-+-----  Slowdown Stats  -------------+-----------------------------------------+
-| Due to - Power (%)                 | 0                                       |
-|        - Thermal (%)               | 0                                       |
-|        - Reliability (%)           | Not Supported                           |
-|        - Board Limit (%)           | Not Supported                           |
-|        - Low Utilization (%)       | Not Supported                           |
-|        - Sync Boost (%)            | 0                                       |
-+-----  Overall Health  -------------+-----------------------------------------+
-| Overall Health                     | Healthy                                 |
-+------------------------------------+-----------------------------------------+
-

MiniGPT-Med-github/dcgm/bash/34738689/dcgm-gpu-stats-gpu201-16-r-34738689.out

@@ -1,35 +0,0 @@
-Successfully retrieved statistics for job: 34738689. 
-+------------------------------------------------------------------------------+
-| GPU ID: 3                                                                    |
-+====================================+=========================================+
-|-----  Execution Stats  ------------+-----------------------------------------|
-| Start Time                         | Wed Jul 24 10:14:38 2024                |
-| End Time                           | Wed Jul 24 11:45:33 2024                |
-| Total Execution Time (sec)         | 5454.69                                 |
-| No. of Processes                   | 0                                       |
-+-----  Performance Stats  ----------+-----------------------------------------+
-| Energy Consumed (Joules)           | 232516                                  |
-| Power Usage (Watts)                | Avg: 64.2532, Max: 64.329, Min: 63.938  |
-| Max GPU Memory Used (bytes)        | 0                                       |
-| SM Clock (MHz)                     | Avg: 210, Max: 210, Min: 210            |
-| Memory Clock (MHz)                 | Avg: 1593, Max: 1593, Min: 1593         |
-| SM Utilization (%)                 | Avg: 0, Max: 0, Min: 0                  |
-| Memory Utilization (%)             | Avg: 0, Max: 0, Min: 0                  |
-| PCIe Rx Bandwidth (megabytes)      | Avg: N/A, Max: N/A, Min: N/A            |
-| PCIe Tx Bandwidth (megabytes)      | Avg: N/A, Max: N/A, Min: N/A            |
-+-----  Event Stats  ----------------+-----------------------------------------+
-| Single Bit ECC Errors              | 0                                       |
-| Double Bit ECC Errors              | 0                                       |
-| PCIe Replay Warnings               | 0                                       |
-| Critical XID Errors                | 0                                       |
-+-----  Slowdown Stats  -------------+-----------------------------------------+
-| Due to - Power (%)                 | 0                                       |
-|        - Thermal (%)               | 0                                       |
-|        - Reliability (%)           | Not Supported                           |
-|        - Board Limit (%)           | Not Supported                           |
-|        - Low Utilization (%)       | Not Supported                           |
-|        - Sync Boost (%)            | 0                                       |
-+-----  Overall Health  -------------+-----------------------------------------+
-| Overall Health                     | Healthy                                 |
-+------------------------------------+-----------------------------------------+
-

MiniGPT-Med-github/dcgm/bash/34757693/dcgm-gpu-stats-gpu202-16-r-34757693.out

@@ -1,42 +0,0 @@
-Successfully retrieved statistics for job: 34757693. 
-+------------------------------------------------------------------------------+
-| GPU ID: 2                                                                    |
-+====================================+=========================================+
-|-----  Execution Stats  ------------+-----------------------------------------|
-| Start Time                         | Thu Jul 25 15:38:16 2024                |
-| End Time                           | Thu Jul 25 17:08:59 2024                |
-| Total Execution Time (sec)         | 5442.54                                 |
-| No. of Processes                   | 2                                       |
-+-----  Performance Stats  ----------+-----------------------------------------+
-| Energy Consumed (Joules)           | 214029                                  |
-| Power Usage (Watts)                | Avg: 59.2012, Max: 67.659, Min: 59.026  |
-| Max GPU Memory Used (bytes)        | 7616856064                              |
-| SM Clock (MHz)                     | Avg: 243, Max: 1080, Min: 210           |
-| Memory Clock (MHz)                 | Avg: 1593, Max: 1593, Min: 1593         |
-| SM Utilization (%)                 | Avg: 0, Max: 0, Min: 0                  |
-| Memory Utilization (%)             | Avg: 0, Max: 0, Min: 0                  |
-| PCIe Rx Bandwidth (megabytes)      | Avg: N/A, Max: N/A, Min: N/A            |
-| PCIe Tx Bandwidth (megabytes)      | Avg: N/A, Max: N/A, Min: N/A            |
-+-----  Event Stats  ----------------+-----------------------------------------+
-| Single Bit ECC Errors              | 0                                       |
-| Double Bit ECC Errors              | 0                                       |
-| PCIe Replay Warnings               | 0                                       |
-| Critical XID Errors                | 0                                       |
-+-----  Slowdown Stats  -------------+-----------------------------------------+
-| Due to - Power (%)                 | 0                                       |
-|        - Thermal (%)               | 0                                       |
-|        - Reliability (%)           | Not Supported                           |
-|        - Board Limit (%)           | Not Supported                           |
-|        - Low Utilization (%)       | Not Supported                           |
-|        - Sync Boost (%)            | 0                                       |
-+--  Compute Process Utilization  ---+-----------------------------------------+
-| PID                                | 1095606                                 |
-|     Avg SM Utilization (%)         | 3                                       |
-|     Avg Memory Utilization (%)     | 0                                       |
-| PID                                | 1096190                                 |
-|     Avg SM Utilization (%)         | 14                                      |
-|     Avg Memory Utilization (%)     | 2                                       |
-+-----  Overall Health  -------------+-----------------------------------------+
-| Overall Health                     | Healthy                                 |
-+------------------------------------+-----------------------------------------+
-

MiniGPT-Med-github/demo_v2.py

@@ -1,648 +0,0 @@
-# python demo_v2.py --cfg-path eval_configs/minigptv2_eval.yaml --gpu-id 0
-
-import argparse
-import os
-import random
-from collections import defaultdict
-
-import cv2
-import re
-
-import numpy as np
-from PIL import Image
-import torch
-import html
-import gradio as gr
-
-import torchvision.transforms as T
-import torch.backends.cudnn as cudnn
-
-from minigpt4.common.config import Config
-
-from minigpt4.common.registry import registry
-from minigpt4.conversation.conversation import Conversation, SeparatorStyle, Chat
-
-# imports modules for registration
-from minigpt4.datasets.builders import *
-from minigpt4.models import *
-from minigpt4.processors import *
-from minigpt4.runners import *
-from minigpt4.tasks import *
-
-
-def parse_args():
-    parser = argparse.ArgumentParser(description="Demo")
-    parser.add_argument("--cfg-path", default='eval_configs/minigptv2_eval.yaml',
-                        help="path to configuration file.")
-    parser.add_argument("--gpu-id", type=int, default=0, help="specify the gpu to load the model.")
-    parser.add_argument(
-        "--options",
-        nargs="+",
-        help="override some settings in the used config, the key-value pair "
-             "in xxx=yyy format will be merged into config file (deprecate), "
-             "change to --cfg-options instead.",
-    )
-    args = parser.parse_args()
-    return args
-
-
-random.seed(42)
-np.random.seed(42)
-torch.manual_seed(42)
-
-cudnn.benchmark = False
-cudnn.deterministic = True
-
-print('Initializing Chat')
-args = parse_args()
-cfg = Config(args)
-
-device = 'cuda:{}'.format(args.gpu_id)
-
-model_config = cfg.model_cfg
-model_config.device_8bit = args.gpu_id
-model_cls = registry.get_model_class(model_config.arch)
-model = model_cls.from_config(model_config).to(device)
-bounding_box_size = 100
-
-vis_processor_cfg = cfg.datasets_cfg.cc_sbu_align.vis_processor.train
-vis_processor = registry.get_processor_class(vis_processor_cfg.name).from_config(vis_processor_cfg)
-
-model = model.eval()
-
-CONV_VISION = Conversation(
-    system="",
-    roles=(r"<s>[INST] ", r" [/INST]"),
-    messages=[],
-    offset=2,
-    sep_style=SeparatorStyle.SINGLE,
-    sep="",
-)
-
-
-def extract_substrings(string):
-    # first check if there is no-finished bracket
-    index = string.rfind('}')
-    if index != -1:
-        string = string[:index + 1]
-
-    pattern = r'<p>(.*?)\}(?!<)'
-    matches = re.findall(pattern, string)
-    substrings = [match for match in matches]
-
-    return substrings
-
-
-def is_overlapping(rect1, rect2):
-    x1, y1, x2, y2 = rect1
-    x3, y3, x4, y4 = rect2
-    return not (x2 < x3 or x1 > x4 or y2 < y3 or y1 > y4)
-
-
-def computeIoU(bbox1, bbox2):
-    x1, y1, x2, y2 = bbox1
-    x3, y3, x4, y4 = bbox2
-    intersection_x1 = max(x1, x3)
-    intersection_y1 = max(y1, y3)
-    intersection_x2 = min(x2, x4)
-    intersection_y2 = min(y2, y4)
-    intersection_area = max(0, intersection_x2 - intersection_x1 + 1) * max(0, intersection_y2 - intersection_y1 + 1)
-    bbox1_area = (x2 - x1 + 1) * (y2 - y1 + 1)
-    bbox2_area = (x4 - x3 + 1) * (y4 - y3 + 1)
-    union_area = bbox1_area + bbox2_area - intersection_area
-    iou = intersection_area / union_area
-    return iou
-
-
-def save_tmp_img(visual_img):
-    file_name = "".join([str(random.randint(0, 9)) for _ in range(5)]) + ".jpg"
-    file_path = "/tmp/gradio" + file_name
-    visual_img.save(file_path)
-    return file_path
-
-
-def mask2bbox(mask):
-    if mask is None:
-        return ''
-    mask = mask.resize([100, 100], resample=Image.NEAREST)
-    mask = np.array(mask)[:, :, 0]
-
-    rows = np.any(mask, axis=1)
-    cols = np.any(mask, axis=0)
-
-    if rows.sum():
-        # Get the top, bottom, left, and right boundaries
-        rmin, rmax = np.where(rows)[0][[0, -1]]
-        cmin, cmax = np.where(cols)[0][[0, -1]]
-        bbox = '{{<{}><{}><{}><{}>}}'.format(cmin, rmin, cmax, rmax)
-    else:
-        bbox = ''
-
-    return bbox
-
-
-def escape_markdown(text):
-    # List of Markdown special characters that need to be escaped
-    md_chars = ['<', '>']
-
-    # Escape each special character
-    for char in md_chars:
-        text = text.replace(char, '\\' + char)
-
-    return text
-
-
-def reverse_escape(text):
-    md_chars = ['\\<', '\\>']
-
-    for char in md_chars:
-        text = text.replace(char, char[1:])
-
-    return text
-
-
-colors = [
-    (255, 0, 0),
-    (0, 255, 0),
-    (0, 0, 255),
-    (210, 210, 0),
-    (255, 0, 255),
-    (0, 255, 255),
-    (114, 128, 250),
-    (0, 165, 255),
-    (0, 128, 0),
-    (144, 238, 144),
-    (238, 238, 175),
-    (255, 191, 0),
-    (0, 128, 0),
-    (226, 43, 138),
-    (255, 0, 255),
-    (0, 215, 255),
-]
-
-color_map = {
-    f"{color_id}": f"#{hex(color[2])[2:].zfill(2)}{hex(color[1])[2:].zfill(2)}{hex(color[0])[2:].zfill(2)}" for
-    color_id, color in enumerate(colors)
-}
-
-used_colors = colors
-
-
-def visualize_all_bbox_together(image, generation):
-    if image is None:
-        return None, ''
-
-    generation = html.unescape(generation)
-    print('gen begin', generation)
-    image_width, image_height = image.size
-    image = image.resize([500, int(500 / image_width * image_height)])
-    image_width, image_height = image.size
-
-    string_list = extract_substrings(generation)
-    if string_list:  # it is grounding or detection
-        mode = 'all'
-        entities = defaultdict(list)
-        i = 0
-        j = 0
-        for string in string_list:
-            try:
-                obj, string = string.split('</p>')
-            except ValueError:
-                print('wrong string: ', string)
-                continue
-            bbox_list = string.split('<delim>')
-            flag = False
-            for bbox_string in bbox_list:
-                integers = re.findall(r'-?\d+', bbox_string)
-                if len(integers) == 4:
-                    x0, y0, x1, y1 = int(integers[0]), int(integers[1]), int(integers[2]), int(integers[3])
-                    left = x0 / bounding_box_size * image_width
-                    bottom = y0 / bounding_box_size * image_height
-                    right = x1 / bounding_box_size * image_width
-                    top = y1 / bounding_box_size * image_height
-
-                    entities[obj].append([left, bottom, right, top])
-
-                    j += 1
-                    flag = True
-            if flag:
-                i += 1
-    else:
-        integers = re.findall(r'-?\d+', generation)
-
-        if len(integers) == 4:  # it is refer
-            mode = 'single'
-
-            entities = list()
-            x0, y0, x1, y1 = int(integers[0]), int(integers[1]), int(integers[2]), int(integers[3])
-            left = x0 / bounding_box_size * image_width
-            bottom = y0 / bounding_box_size * image_height
-            right = x1 / bounding_box_size * image_width
-            top = y1 / bounding_box_size * image_height
-            entities.append([left, bottom, right, top])
-        else:
-            # don't detect any valid bbox to visualize
-            return None, ''
-
-    if len(entities) == 0:
-        return None, ''
-
-    if isinstance(image, Image.Image):
-        image_h = image.height
-        image_w = image.width
-        image = np.array(image)
-
-    elif isinstance(image, str):
-        if os.path.exists(image):
-            pil_img = Image.open(image).convert("RGB")
-            image = np.array(pil_img)[:, :, [2, 1, 0]]
-            image_h = pil_img.height
-            image_w = pil_img.width
-        else:
-            raise ValueError(f"invaild image path, {image}")
-    elif isinstance(image, torch.Tensor):
-
-        image_tensor = image.cpu()
-        reverse_norm_mean = torch.tensor([0.48145466, 0.4578275, 0.40821073])[:, None, None]
-        reverse_norm_std = torch.tensor([0.26862954, 0.26130258, 0.27577711])[:, None, None]
-        image_tensor = image_tensor * reverse_norm_std + reverse_norm_mean
-        pil_img = T.ToPILImage()(image_tensor)
-        image_h = pil_img.height
-        image_w = pil_img.width
-        image = np.array(pil_img)[:, :, [2, 1, 0]]
-    else:
-        raise ValueError(f"invaild image format, {type(image)} for {image}")
-
-    indices = list(range(len(entities)))
-
-    new_image = image.copy()
-
-    previous_bboxes = []
-    # size of text
-    text_size = 0.5
-    # thickness of text
-    text_line = 1  # int(max(1 * min(image_h, image_w) / 512, 1))
-    box_line = 2
-    (c_width, text_height), _ = cv2.getTextSize("F", cv2.FONT_HERSHEY_COMPLEX, text_size, text_line)
-    base_height = int(text_height * 0.675)
-    text_offset_original = text_height - base_height
-    text_spaces = 2
-
-    # num_bboxes = sum(len(x[-1]) for x in entities)
-    used_colors = colors  # random.sample(colors, k=num_bboxes)
-
-    color_id = -1
-    for entity_idx, entity_name in enumerate(entities):
-        if mode == 'single' or mode == 'identify':
-            bboxes = entity_name
-            bboxes = [bboxes]
-        else:
-            bboxes = entities[entity_name]
-        color_id += 1
-        for bbox_id, (x1_norm, y1_norm, x2_norm, y2_norm) in enumerate(bboxes):
-            skip_flag = False
-            orig_x1, orig_y1, orig_x2, orig_y2 = int(x1_norm), int(y1_norm), int(x2_norm), int(y2_norm)
-
-            color = used_colors[entity_idx % len(used_colors)]  # tuple(np.random.randint(0, 255, size=3).tolist())
-            new_image = cv2.rectangle(new_image, (orig_x1, orig_y1), (orig_x2, orig_y2), color, box_line)
-
-            if mode == 'all':
-                l_o, r_o = box_line // 2 + box_line % 2, box_line // 2 + box_line % 2 + 1
-
-                x1 = orig_x1 - l_o
-                y1 = orig_y1 - l_o
-
-                if y1 < text_height + text_offset_original + 2 * text_spaces:
-                    y1 = orig_y1 + r_o + text_height + text_offset_original + 2 * text_spaces
-                    x1 = orig_x1 + r_o
-
-                # add text background
-                (text_width, text_height), _ = cv2.getTextSize(f"  {entity_name}", cv2.FONT_HERSHEY_COMPLEX, text_size,
-                                                               text_line)
-                text_bg_x1, text_bg_y1, text_bg_x2, text_bg_y2 = x1, y1 - (
-                            text_height + text_offset_original + 2 * text_spaces), x1 + text_width, y1
-
-                for prev_bbox in previous_bboxes:
-                    if computeIoU((text_bg_x1, text_bg_y1, text_bg_x2, text_bg_y2), prev_bbox['bbox']) > 0.95 and \
-                            prev_bbox['phrase'] == entity_name:
-                        skip_flag = True
-                        break
-                    while is_overlapping((text_bg_x1, text_bg_y1, text_bg_x2, text_bg_y2), prev_bbox['bbox']):
-                        text_bg_y1 += (text_height + text_offset_original + 2 * text_spaces)
-                        text_bg_y2 += (text_height + text_offset_original + 2 * text_spaces)
-                        y1 += (text_height + text_offset_original + 2 * text_spaces)
-
-                        if text_bg_y2 >= image_h:
-                            text_bg_y1 = max(0, image_h - (text_height + text_offset_original + 2 * text_spaces))
-                            text_bg_y2 = image_h
-                            y1 = image_h
-                            break
-                if not skip_flag:
-                    alpha = 0.5
-                    for i in range(text_bg_y1, text_bg_y2):
-                        for j in range(text_bg_x1, text_bg_x2):
-                            if i < image_h and j < image_w:
-                                if j < text_bg_x1 + 1.35 * c_width:
-                                    # original color
-                                    bg_color = color
-                                else:
-                                    # white
-                                    bg_color = [255, 255, 255]
-                                new_image[i, j] = (alpha * new_image[i, j] + (1 - alpha) * np.array(bg_color)).astype(
-                                    np.uint8)
-
-                    cv2.putText(
-                        new_image, f"  {entity_name}", (x1, y1 - text_offset_original - 1 * text_spaces),
-                        cv2.FONT_HERSHEY_COMPLEX, text_size, (0, 0, 0), text_line, cv2.LINE_AA
-                    )
-
-                    previous_bboxes.append(
-                        {'bbox': (text_bg_x1, text_bg_y1, text_bg_x2, text_bg_y2), 'phrase': entity_name})
-
-    if mode == 'all':
-        def color_iterator(colors):
-            while True:
-                for color in colors:
-                    yield color
-
-        color_gen = color_iterator(colors)
-
-        # Add colors to phrases and remove <p></p>
-        def colored_phrases(match):
-            phrase = match.group(1)
-            color = next(color_gen)
-            return f'<span style="color:rgb{color}">{phrase}</span>'
-
-        generation = re.sub(r'{<\d+><\d+><\d+><\d+>}|<delim>', '', generation)
-        generation_colored = re.sub(r'<p>(.*?)</p>', colored_phrases, generation)
-    else:
-        generation_colored = ''
-
-    pil_image = Image.fromarray(new_image)
-    return pil_image, generation_colored
-
-
-def gradio_reset(chat_state, img_list):
-    if chat_state is not None:
-        chat_state.messages = []
-    if img_list is not None:
-        img_list = []
-    return None, gr.update(value=None, interactive=True), gr.update(placeholder='Upload your image and chat',
-                                                                    interactive=True), chat_state, img_list
-
-
-def image_upload_trigger(upload_flag, replace_flag, img_list):
-    # set the upload flag to true when receive a new image.
-    # if there is an old image (and old conversation), set the replace flag to true to reset the conv later.
-    upload_flag = 1
-    if img_list:
-        replace_flag = 1
-    return upload_flag, replace_flag
-
-
-def example_trigger(text_input, image, upload_flag, replace_flag, img_list):
-    # set the upload flag to true when receive a new image.
-    # if there is an old image (and old conversation), set the replace flag to true to reset the conv later.
-    upload_flag = 1
-    if img_list or replace_flag == 1:
-        replace_flag = 1
-
-    return upload_flag, replace_flag
-
-
-def gradio_ask(user_message, chatbot, chat_state, gr_img, img_list, upload_flag, replace_flag):
-    if len(user_message) == 0:
-        text_box_show = 'Input should not be empty!'
-    else:
-        text_box_show = ''
-
-    if isinstance(gr_img, dict):
-        gr_img, mask = gr_img['image'], gr_img['mask']
-    else:
-        mask = None
-
-    if '[identify]' in user_message:
-        # check if user provide bbox in the text input
-        integers = re.findall(r'-?\d+', user_message)
-        if len(integers) != 4:  # no bbox in text
-            bbox = mask2bbox(mask)
-            user_message = user_message + bbox
-
-    if chat_state is None:
-        chat_state = CONV_VISION.copy()
-
-    if upload_flag:
-        if replace_flag:
-            chat_state = CONV_VISION.copy()  # new image, reset everything
-            replace_flag = 0
-            chatbot = []
-        img_list = []
-        llm_message = chat.upload_img(gr_img, chat_state, img_list)
-        upload_flag = 0
-
-    chat.ask(user_message, chat_state)
-
-    chatbot = chatbot + [[user_message, None]]
-
-    if '[identify]' in user_message:
-        visual_img, _ = visualize_all_bbox_together(gr_img, user_message)
-        if visual_img is not None:
-            file_path = save_tmp_img(visual_img)
-            chatbot = chatbot + [[(file_path,), None]]
-
-    return text_box_show, chatbot, chat_state, img_list, upload_flag, replace_flag
-
-
-def gradio_answer(chatbot, chat_state, img_list, temperature):
-    llm_message = chat.answer(conv=chat_state,
-                              img_list=img_list,
-                              temperature=temperature,
-                              max_new_tokens=500,
-                              max_length=2000)[0]
-    chatbot[-1][1] = llm_message
-    return chatbot, chat_state
-
-
-def gradio_stream_answer(chatbot, chat_state, img_list, temperature):
-    if len(img_list) > 0:
-        if not isinstance(img_list[0], torch.Tensor):
-            chat.encode_img(img_list)
-    streamer = chat.stream_answer(conv=chat_state,
-                                  img_list=img_list,
-                                  temperature=temperature,
-                                  max_new_tokens=500,
-                                  max_length=2000)
-    output = ''
-    for new_output in streamer:
-        escapped = escape_markdown(new_output)
-        output += escapped
-        chatbot[-1][1] = output
-        yield chatbot, chat_state
-    chat_state.messages[-1][1] = '</s>'
-    return chatbot, chat_state
-
-
-def gradio_visualize(chatbot, gr_img):
-    if isinstance(gr_img, dict):
-        gr_img, mask = gr_img['image'], gr_img['mask']
-
-    unescaped = reverse_escape(chatbot[-1][1])
-    visual_img, generation_color = visualize_all_bbox_together(gr_img, unescaped)
-    if visual_img is not None:
-        if len(generation_color):
-            chatbot[-1][1] = generation_color
-        file_path = save_tmp_img(visual_img)
-        chatbot = chatbot + [[None, (file_path,)]]
-
-    return chatbot
-
-
-def gradio_taskselect(idx):
-    prompt_list = [
-        '',
-        '[grounding] describe this image in detail',
-        '[refer] ',
-        '[detection] ',
-        '[identify] what is this ',
-        '[vqa] '
-    ]
-    instruct_list = [
-        '**Hint:** Type in whatever you want',
-        '**Hint:** Send the command to generate a grounded image description',
-        '**Hint:** Type in a phrase about an object in the image and send the command',
-        '**Hint:** Type in a caption or phrase, and see object locations in the image',
-        '**Hint:** Draw a bounding box on the uploaded image then send the command. Click the "clear" botton on the top right of the image before redraw',
-        '**Hint:** Send a question to get a short answer',
-    ]
-    return prompt_list[idx], instruct_list[idx]
-
-
-
-
-chat = Chat(model, vis_processor, device=device)
-
-title = """<h1 align="center">MiniGPT-Med Demo</h1>"""
-description = 'Welcome to Our MiniGPT-Med Chatbot Demo!'
-# article = """<p><a href='https://minigpt-v2.github.io'><img src='https://img.shields.io/badge/Project-Page-Green'></a></p><p><a href='https://github.com/Vision-CAIR/MiniGPT-4/blob/main/MiniGPTv2.pdf'><img src='https://img.shields.io/badge/Paper-PDF-red'></a></p><p><a href='https://github.com/Vision-CAIR/MiniGPT-4'><img src='https://img.shields.io/badge/GitHub-Repo-blue'></a></p><p><a href='https://www.youtube.com/watch?v=atFCwV2hSY4'><img src='https://img.shields.io/badge/YouTube-Video-red'></a></p>"""
-article = """<p><a href='https://minigpt-med.github.io/'><img src='https://img.shields.io/badge/Project-Page-Green'></a></p>"""
-
-introduction = '''
-For Abilities Involving Visual Grounding:
-1. Grounding: CLICK **Send** to generate a grounded image description.
-2. Refer: Input a referring object and CLICK **Send**.
-3. Detection: Write a caption or phrase, and CLICK **Send**.
-4. Identify: Draw the bounding box on the uploaded image window and CLICK **Send** to generate the bounding box. (CLICK "clear" button before re-drawing next time).
-5. VQA: Input a visual question and CLICK **Send**.
-6. No Tag: Input whatever you want and CLICK **Send** without any tagging
-
-You can also simply chat in free form!
-'''
-
-text_input = gr.Textbox(placeholder='Upload your image and chat', interactive=True, show_label=False, container=False,
-                        scale=8)
-with gr.Blocks() as demo:
-    gr.Markdown(title)
-    # gr.Markdown(description)
-    gr.Markdown(article)
-
-    with gr.Row():
-        with gr.Column(scale=0.5):
-            image = gr.Image(type="pil", tool='sketch', brush_radius=20)
-
-            temperature = gr.Slider(
-                minimum=0.1,
-                maximum=1.5,
-                value=0.6,
-                step=0.1,
-                interactive=True,
-                label="Temperature",
-            )
-
-            clear = gr.Button("Restart")
-
-            gr.Markdown(introduction)
-
-        with gr.Column():
-            chat_state = gr.State(value=None)
-            img_list = gr.State(value=[])
-            chatbot = gr.Chatbot(label='MiniGPT-Med')
-
-            dataset = gr.Dataset(
-                components=[gr.Textbox(visible=False)],
-                samples=[['No Tag'], ['Grounding'], ['Refer'], ['Detection'], ['Identify'], ['VQA']],
-                type="index",
-                label='Task Shortcuts',
-            )
-            task_inst = gr.Markdown('**Hint:** Upload your image and chat')
-            with gr.Row():
-                text_input.render()
-                send = gr.Button("Send", variant='primary', size='sm', scale=1)
-
-    upload_flag = gr.State(value=0)
-    replace_flag = gr.State(value=0)
-    image.upload(image_upload_trigger, [upload_flag, replace_flag, img_list], [upload_flag, replace_flag])
-# [29, 44, 42, 56]
-    with gr.Row():
-        with gr.Column():
-            gr.Examples(examples=[
-                ["Med_examples_v2/xmlab149/source.jpg", "[identify] what is this {<56><16><84><58>}", upload_flag,
-                 replace_flag, img_list],
-                ["Med_examples_v2/1.2.276.0.7230010.3.1.4.8323329.1495.1517874291.249176.jpg", "[detection] pneumonia", upload_flag, replace_flag, img_list],
-                ["Med_examples_v2/1.2.840.113654.2.55.48339325922382839066544590341580673064.png", "[refer] the nodule in the left lung", upload_flag, replace_flag,
-                 img_list],
-               ["Med_examples_v2/xmlab589/source.jpg", "[grounding] describe this image in detail", upload_flag, replace_flag, img_list],
-            ], inputs=[image, text_input, upload_flag, replace_flag, img_list], fn=example_trigger,
-                outputs=[upload_flag, replace_flag])
-        with gr.Column():
-            gr.Examples(examples=[
-                ["Med_examples_v2/synpic50958.jpg", "[vqa] What does the small white lesions in the aorta mean?",
-                 upload_flag, replace_flag, img_list],
-                ["Med_examples_v2/5f4e8079-8225a5d2-1b0c3c46-4394a094-f285db0e.jpg", "Please provide a detailed description of the picture", upload_flag, replace_flag, img_list],
-                ["Med_examples_v2/1.2.276.0.7230010.3.1.4.8323329.16254.1517874395.786150.jpg", "Diagnose this image", upload_flag, replace_flag, img_list],
-                ["Med_examples_v2/synpic58547.jpg", "Could you describe the contents of this image for me?", upload_flag,
-                 replace_flag, img_list],
-            ], inputs=[image, text_input, upload_flag, replace_flag, img_list], fn=example_trigger,
-                outputs=[upload_flag, replace_flag])
-
-    dataset.click(
-        gradio_taskselect,
-        inputs=[dataset],
-        outputs=[text_input, task_inst],
-        show_progress="hidden",
-        postprocess=False,
-        queue=False,
-    )
-
-    text_input.submit(
-        gradio_ask,
-        [text_input, chatbot, chat_state, image, img_list, upload_flag, replace_flag],
-        [text_input, chatbot, chat_state, img_list, upload_flag, replace_flag], queue=False
-    ).success(
-        gradio_stream_answer,
-        [chatbot, chat_state, img_list, temperature],
-        [chatbot, chat_state]
-    ).success(
-        gradio_visualize,
-        [chatbot, image],
-        [chatbot],
-        queue=False,
-    )
-
-    send.click(
-        gradio_ask,
-        [text_input, chatbot, chat_state, image, img_list, upload_flag, replace_flag],
-        [text_input, chatbot, chat_state, img_list, upload_flag, replace_flag], queue=False
-    ).success(
-        gradio_stream_answer,
-        [chatbot, chat_state, img_list, temperature],
-        [chatbot, chat_state]
-    ).success(
-        gradio_visualize,
-        [chatbot, image],
-        [chatbot],
-        queue=False,
-    )
-
-    clear.click(gradio_reset, [chat_state, img_list], [chatbot, image, text_input, chat_state, img_list], queue=False)
-
-demo.launch(share=True, enable_queue=True)

MiniGPT-Med-github/environment.yml

@@ -1,35 +0,0 @@
-name: miniGPT-Med
-channels:
-  - pytorch
-  - defaults
-  - anaconda
-dependencies:
-  - python=3.9
-  - cudatoolkit
-  - pip
-  - pip:
-    - torch==2.0.0
-    - torchaudio
-    - torchvision
-    - huggingface-hub==0.18.0
-    - matplotlib==3.7.0
-    - psutil==5.9.4
-    - iopath
-    - pyyaml==6.0
-    - regex==2022.10.31
-    - tokenizers==0.13.2
-    - tqdm==4.64.1
-    - transformers==4.30.0
-    - timm==0.6.13
-    - webdataset==0.2.48
-    - omegaconf==2.3.0
-    - opencv-python==4.7.0.72
-    - decord==0.6.0
-    - peft==0.2.0
-    - sentence-transformers
-    - gradio==3.47.1
-    - accelerate==0.20.3
-    - bitsandbytes==0.37.0
-    - scikit-image
-    - visual-genome
-    - wandb

MiniGPT-Med-github/eval_configs/minigptv2_benchmark_evaluation.yaml

@@ -1,69 +0,0 @@
-model:
-  arch: minigpt_v2
-  model_type: pretrain
-  max_txt_len: 500
-  end_sym: "</s>"
-  low_resource: False
-  prompt_template: '[INST] {} [/INST]'
-  llama_model: "/ibex/project/c2106/RadGPT/MiniGPT4-v2/llama-2-7b-chat-hf"
-  ckpt: "/ibex/project/c2106/RadGPT/MiniGPT-Med-github/miniGPT_Med_.pth"
-  lora_r: 64
-  lora_alpha: 16
-
-datasets:
-  cc_sbu_align:
-    vis_processor:
-      train:
-        name: "blip2_image_eval"
-        image_size: 448
-    text_processor:
-      train:
-        name: "blip_caption"
-
-evaluation_datasets:
-  rsna:
-    eval_file_path: miniGPT-Med/json_files/RSNA/full_RSNA_1024.json
-    img_path: miniGPT-Med/RSNA/RSNA-bbox-1024
-    max_new_tokens: 100
-    batch_size: 10
-
-  radvqa:
-    eval_file_path: /miniGPT-Med/json_files/vqa/full_radVQA.json
-    img_path: /miniGPT-Med/radVQA/VQA_RAD_Images
-    max_new_tokens: 300
-    batch_size: 10
-
-  mimic_cxr:
-    eval_file_path: /miniGPT-Med/json_files/mimic/MIMIC_test.json
-    img_path: /miniGPT-Med/mimic-cxr-dataset/image
-    max_new_tokens: 300
-    batch_size: 10
-    
-  nlst:
-    eval_file_path: /miniGPT-Med/json_files/NLST/NLST_test.json
-    img_path: /miniGPT-Med/NLST/NLST_images
-    max_new_tokens: 100
-    batch_size: 10
-
-  detect_mimic:
-    eval_file_path: /miniGPT-Med/json_files/MIMIC-bbox/MIMIC-benchmarck.json
-    img_path: /miniGPT-Med/mimic-cxr-dataset/image
-    max_new_tokens: 100
-    batch_size: 10
-
-  SLAKE:
-    eval_file_path: /miniGPT-Med/json_files/SLAKE/grounding_test_SLAKE.json
-    img_path: /miniGPT-Med/SLAKE_images/imgs
-    max_new_tokens: 100
-    batch_size: 10
-
-
-run:
-  task: image_text_pretrain
-  name: minigptv2_evaluation
-  save_path: /miniGPT-Med/expermints
-
-  
-
-  
-

MiniGPT-Med-github/eval_configs/minigptv2_eval.yaml

@@ -1,24 +0,0 @@
-model:
-  arch: minigpt_v2
-  model_type: pretrain
-  max_txt_len: 500
-  end_sym: "</s>"
-  low_resource: True
-  prompt_template: '[INST] {} [/INST]'
-  ckpt: "/ibex/project/c2106/RadGPT/MiniGPT-Med-github/miniGPT_Med_.pth"
-  lora_r: 64
-  lora_alpha: 16
-
-
-datasets:
-  cc_sbu_align:
-    vis_processor:
-      train:
-        name: "blip2_image_eval"
-        image_size: 448
-    text_processor:
-      train:
-        name: "blip_caption"
-
-run:
-  task: image_text_pretrain

MiniGPT-Med-github/eval_scripts/clean_json.py

@@ -1,74 +0,0 @@
-import json
-import re 
-
-def clean_mimic_json(messy_json, cleaned_output):
-    with open(messy_json, 'r') as f:
-        messy_data = json.load(f)
-
-    clean_data = []
-    for image_id, captions in messy_data.items():
-        image_id_clean = image_id.split('.')[0]
-        caption_clean = ' '.join(captions)
-
-        clean_item = {
-            "image_id": image_id_clean,
-            "caption": caption_clean
-        }
-        
-        clean_data.append(clean_item)
-
-    with open(cleaned_output, 'w') as outfile:
-        json.dump(clean_data, outfile, indent=2)
-
-
-def clean_vqa_json(messy_json, cleaned_output):
-    with open(messy_json, "r") as file:
-        messy_json = json.load(file)
-
-    organized_json = {}
-
-    for key, values in messy_json.items():
-        organized_json[key] = []
-        for value in values:
-            organized_json[key].append({
-                "question": value["question"],
-                "answer": value["answer"]
-            })
-
-    with open(cleaned_output, "w") as outfile:
-        json.dump(organized_json, outfile, indent=4)
-
-
-
-def clean_detection_json(messy_json, cleaned_output):
-
-    with open(messy_json, "r") as input_file:
-        input_json = json.load(input_file)
-
-    organized_data = []
-
-    for key, value in input_json.items():
-        if value and isinstance(value, list) and len(value) > 0:
-            caption = value[0]
-            objects_match = caption.split("<p>")
-            if len(objects_match) == 2:
-                object_part = objects_match[1].split("</p>")[0].strip()
-            else:
-                object_part = ""
-            
-            bbox_match = re.findall(r'<(\d+)>', caption)
-            
-            if object_part and bbox_match and len(bbox_match) == 4:
-                key_part = key.split(".png")[0]
-                bbox_values = [float(val) for val in bbox_match]
-
-                organized_item = {
-                    "key": key_part,
-                    "objects": [object_part],
-                    "bbox": [bbox_values],
-                }
-
-                organized_data.append(organized_item)
-
-    with open(cleaned_output, "w") as output_file:
-        json.dump(organized_data, output_file, indent=4)

MiniGPT-Med-github/eval_scripts/metrics.py

@@ -1,164 +0,0 @@
-import sys
-sys.path.append('.')
-
-import json
-import pandas as pd
-import csv
-from sentence_transformers import SentenceTransformer, util
-from minigpt4.common.eval_utils import computeIoU
-
-# Load pre-trained BERT model
-model = SentenceTransformer('paraphrase-MiniLM-L6-v2')
-
-
-# BERT similarity function will be utilized in the two following functions
-def compute_bert_similarity(prediction_caption, ground_truth_caption):
-    prediction_embedding = model.encode([prediction_caption])
-    ground_truth_embedding = model.encode([ground_truth_caption])
-    similarity = util.pytorch_cos_sim(prediction_embedding, ground_truth_embedding)[0][0].item()
-    return similarity
-
-
-def MIMIC_BERT_Sim(gt_pth, pred_pth, output_csv):
-    # Read the ground truth and prediction JSON files
-    with open(gt_pth, 'r') as f:
-        ground_truth_data = json.load(f)
-    
-    with open(pred_pth, 'r') as f:
-        prediction_data = json.load(f)
-    
-    # Create a list to store BERT similarity data
-    bert_similarity_data = []
-    
-    # Initialize variables to calculate the average
-    total_similarity = 0
-    total_count = 0
-    
-    # Iterate over each item in the prediction_data list
-    for item in prediction_data:
-        # Extract the image_id and corresponding prediction caption
-        image_id = item["image_id"]
-        prediction_caption = item["caption"]
-        
-        # Search for the matching ground truth caption based on image_id
-        ground_truth_caption = None
-        for gt_item in ground_truth_data:
-            if gt_item["image_id"] == image_id:
-                ground_truth_caption = gt_item["caption"]
-                break
-        
-        if ground_truth_caption is not None:
-            bert_similarity = compute_bert_similarity(prediction_caption, ground_truth_caption)
-            bert_similarity_data.append({"image_id": image_id, "BERT_score": bert_similarity})
-            
-            total_similarity += bert_similarity
-            total_count += 1
-    
-    average_similarity = total_similarity / total_count if total_count > 0 else 0
-    
-    df = pd.DataFrame(bert_similarity_data)
-    df_sorted = df.sort_values(by="BERT_score", ascending=True)
-    df_sorted.to_csv(output_csv, index=False)
-    
-    return average_similarity
-
-def VQA_BERT_Sim(gt_pth, pred_pth, output_csv):
-    # Load ground truth JSON file
-    with open(gt_pth, 'r') as file:
-        gt_data = json.load(file)
-
-    # Load prediction JSON file
-    with open(pred_pth, 'r') as file:
-        prediction_data = json.load(file)
-
-    gt_qa_pairs = {(entry['image_name'], entry['question']): entry['answer'] for entry in gt_data}
-
-    def convert_to_dict(data):
-        qa_dict = {}
-        for image_name, qa_list in data.items():
-            for qa in qa_list:
-                key = (image_name, qa['question'])
-                qa_dict[key] = qa['answer']
-        return qa_dict
-
-    pred_qa_dict = convert_to_dict(prediction_data)
-
-    # Compute BERT similarity and create a list of results
-    results = []
-
-    for key, gt_answer in gt_qa_pairs.items():
-        if key in pred_qa_dict:
-            pred_answer = pred_qa_dict[key]
-            gt_answer = str(gt_answer)
-            pred_answer = str(pred_answer)
-
-            # Compute BERT similarity
-            similarity_score = compute_bert_similarity(pred_answer, gt_answer)
-
-            # Append the result to the list
-            results.append({
-                "img_name": key[0],
-                "question": key[1],
-                "answer": pred_answer,
-                "BERT_score": similarity_score
-            })
-
-    average_similarity = sum(entry["BERT_score"] for entry in results) / len(results) if results else 0
-    df = pd.DataFrame(results)
-    df_sorted = df.sort_values(by="BERT_score", ascending=True)
-    df_sorted.to_csv(output_csv, index=False)
-    print(f"Average BERT similarity score: {average_similarity}")
-
-
-#################################
-##############IoU################
-#################################
-
-def preprocess_bbox(bbox, original_size, image_size):
-    x1 = int((bbox[0] / original_size) * image_size)
-    y1 = int((bbox[1] / original_size) * image_size)
-    x2 = int((bbox[2] / original_size) * image_size)
-    y2 = int((bbox[3] / original_size) * image_size)
-    return [x1, y1, x2, y2]
-
-def average_iou(gt_pth, pred_pth, original_size, image_size, dataset_name, csv_filename):
-    # Load ground truth
-    with open(gt_pth, 'r') as file:
-        ground_truth = json.load(file)
-
-    # Load predictions
-    with open(pred_pth, 'r') as file:
-        predictions = json.load(file)
-
-    iou_list = []
-
-    with open(csv_filename, 'w', newline='') as csvfile:
-        fieldnames = ['image_name', 'IoU']
-        writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
-        writer.writeheader()
-
-        for gt_item in ground_truth:
-            gt_key = gt_item['key']
-            gt_bboxes = gt_item['bbox']
-            original_size = gt_item['height']
-            gt_processed_bboxes = [preprocess_bbox(bbox, original_size, image_size) for bbox in gt_bboxes]
-
-            for pred_item in predictions:
-                pred_key = pred_item['key'].replace(".png", "")
-
-                if gt_key == pred_key:
-                    pred_bboxes = pred_item['bbox']
-                    try:
-                        for gt_bbox in gt_processed_bboxes:
-                            for pred_bbox in pred_bboxes:
-                                iou = computeIoU(gt_bbox, pred_bbox)
-                                iou_list.append(iou)
-                                writer.writerow({'image_name': gt_key, 'IoU': iou})
-                                print(gt_key)
-                                print(iou)
-                    except Exception as e:
-                        print("gt_bbox: ", gt_bbox)
-                        print("gt_bbox: ", pred_bboxes)
-
-    # average_iou = sum(iou_list) / len(iou_list)
-    # print(f"Average IoU for dataset {dataset_name}: {average_iou:.4f}")

MiniGPT-Med-github/eval_scripts/model_evaluation.py

@@ -1,274 +0,0 @@
-'''
-use this command in terminal to run the evaluation script
-torchrun --master-port 8888 --nproc_per_node 1 eval_scripts/model_evaluation.py  --cfg-path eval_configs/minigptv2_benchmark_evaluation.yaml --dataset 
-
-
-'''
-
-import sys
-sys.path.append('.')
-import os
-import re
-import json
-import argparse
-from collections import defaultdict
-import random
-import numpy as np
-from PIL import Image
-from tqdm import tqdm
-import torch
-from torch.utils.data import DataLoader
-from minigpt4.common.config import Config
-from minigpt4.common.eval_utils import prepare_texts, init_model, eval_parser, computeIoU
-from minigpt4.conversation.conversation import CONV_VISION_minigptv2
-
-from minigpt4.datasets.datasets.mimic_cxr_dataset import evalMIMICDataset, evalDetectMimicDataset
-from minigpt4.datasets.datasets.radvqa_dataset import evalRadVQADataset
-from minigpt4.datasets.datasets.nlst_dataset import eval_NLST_Dataset
-from minigpt4.datasets.datasets.rsna_dataset import evalRSNADataset
-from minigpt4.datasets.datasets.SLAKE_dataset import evalSLAKEDataset
-#import cleaning classes
-from eval_scripts.clean_json import clean_mimic_json, clean_vqa_json, clean_detection_json
-from eval_scripts.metrics import MIMIC_BERT_Sim, VQA_BERT_Sim, average_iou
-
-def list_of_str(arg):
-    return list(map(str, arg.split(',')))
-
-parser = eval_parser()
-parser.add_argument("--dataset", type=list_of_str, help="dataset to evaluate")
-
-args = parser.parse_args()
-
-cfg = Config(args)
-
-
-model, vis_processor = init_model(args)
-model.eval()
-CONV_VISION = CONV_VISION_minigptv2
-conv_temp = CONV_VISION.copy()
-conv_temp.system = ""
-model.eval()
-save_path = cfg.run_cfg.save_path
-
-def process_mimic_dataset():
-    eval_file_path = cfg.evaluation_datasets_cfg[dataset]["eval_file_path"]
-    img_path = cfg.evaluation_datasets_cfg[dataset]["img_path"]
-    batch_size = cfg.evaluation_datasets_cfg[dataset]["batch_size"]
-    max_new_tokens = cfg.evaluation_datasets_cfg[dataset]["max_new_tokens"]
-
-    with open((eval_file_path), 'r') as f:
-        mimic = json.load(f)
-            
-    data = evalMIMICDataset(mimic, vis_processor, img_path)
-    eval_dataloader = DataLoader(data, batch_size=batch_size, shuffle=False)
-    minigpt4_predict = defaultdict(list)
-
-    for images, questions, img_ids in tqdm(eval_dataloader):
-        texts = prepare_texts(questions, conv_temp)  # warp the texts with conversation template
-        answers = model.generate(images, texts, max_new_tokens=max_new_tokens, do_sample=False)
-        for answer, img_id, question in zip(answers, img_ids, questions):
-            minigpt4_predict[img_id].append(answer)
-    
-    file_save_path = os.path.join(save_path,"MIMIC_inference_results_stage3.json")
-    with open(file_save_path,'w') as f:
-        json.dump(minigpt4_predict, f)
-    clean_mimic_json(file_save_path, file_save_path)
-
-    # csv file path to save the BERT results per each case
-    output_csv_path = '/miniGPT-Med/metric_results/bert_similarity_scores.csv'
-
-    # in MIMIC_BERT_Sim add the path of the ground_truth then the path of the inference result
-    average_similarity = MIMIC_BERT_Sim(eval_file_path, file_save_path, output_csv_path)
-    #print the average BERT_Sim
-    print("Average BERT Similarity:", average_similarity)
-
-def process_vqa_dataset():
-    eval_file_path = cfg.evaluation_datasets_cfg[dataset]["eval_file_path"]
-    img_path = cfg.evaluation_datasets_cfg[dataset]["img_path"]
-    batch_size = cfg.evaluation_datasets_cfg[dataset]["batch_size"]
-    max_new_tokens = cfg.evaluation_datasets_cfg[dataset]["max_new_tokens"]
-
-    with open((eval_file_path), 'r') as f:
-        radVQA = json.load(f)
-            
-    data = evalRadVQADataset(radVQA, vis_processor, img_path)
-    eval_dataloader = DataLoader(data, batch_size=batch_size, shuffle=False)
-    minigpt4_predict = defaultdict(list)
-
-    for images, questions, img_ids in tqdm(eval_dataloader):
-        texts = prepare_texts(questions, conv_temp)  # warp the texts with conversation template
-        answers = model.generate(images, texts, max_new_tokens=max_new_tokens, do_sample=False)
-        for answer, img_id, question in zip(answers, img_ids, questions):
-            minigpt4_predict[img_id].append({"key":img_ids,"question": question.replace("[vqa]", "").strip() , "answer": answer})
-    
-    file_save_path = os.path.join(save_path,"radVQA_inference_results.json")
-    output_csv_path = '/miniGPT-Med/BERT_Sim_results/vqa_bert_similarity_scores.csv'
-
-    with open(file_save_path,'w') as f:
-        json.dump(minigpt4_predict, f)
-
-    clean_vqa_json(file_save_path, file_save_path)
-    VQA_BERT_Sim(eval_file_path, file_save_path, output_csv_path)
-
-def process_nlst_dataset():
-    eval_file_path = cfg.evaluation_datasets_cfg[dataset]["eval_file_path"]
-    img_path = cfg.evaluation_datasets_cfg[dataset]["img_path"] 
-    batch_size = cfg.evaluation_datasets_cfg[dataset]["batch_size"]
-    max_new_tokens = cfg.evaluation_datasets_cfg[dataset]["max_new_tokens"]
-
-    with open((eval_file_path), 'r') as f:
-        nlst = json.load(f)
-
-    data = eval_NLST_Dataset(nlst, vis_processor, img_path)
-    eval_dataloader = DataLoader(data, batch_size=batch_size, shuffle=False)
-    minigpt4_predict = defaultdict(list)
-    resamples = []
-
-    for images, questions, img_ids in tqdm(eval_dataloader):
-
-        texts = prepare_texts(questions, conv_temp)
-        answers = model.generate(images, texts, max_new_tokens=max_new_tokens, do_sample=False)
-
-        for answer, img_id, question in zip(answers, img_ids, questions):
-
-            # answer = answer.replace("<unk>","").replace(" ","").strip()
-            pattern = r'\{<\d{1,2}><\d{1,2}><\d{1,2}><\d{1,2}>\}'
-            minigpt4_predict[img_id].append(answer)
-
-    file_save_path = os.path.join(save_path,"NLST_inference_result.json")
-    with open(file_save_path,'w') as f:
-        json.dump(minigpt4_predict, f)
-    
-    csv_pth = os.path.join(save_path,"NLST_IoU_results.csv")
-    clean_detection_json(file_save_path,file_save_path)
-    average_iou(eval_file_path, file_save_path, 512, 100, "NLST", csv_pth)
-
-
-
-def process_rsna_dataset():
-    eval_file_path = cfg.evaluation_datasets_cfg[dataset]["eval_file_path"]
-    print(eval_file_path)
-    img_path = cfg.evaluation_datasets_cfg[dataset]["img_path"] 
-    batch_size = cfg.evaluation_datasets_cfg[dataset]["batch_size"]
-    max_new_tokens = cfg.evaluation_datasets_cfg[dataset]["max_new_tokens"]
-    print("----config----")
-    with open((eval_file_path), 'r') as f:
-        nlst = json.load(f)
-
-    data = evalRSNADataset(nlst, vis_processor, img_path)
-    eval_dataloader = DataLoader(data, batch_size=batch_size, shuffle=False)
-    minigpt4_predict = defaultdict(list)
-    resamples = []
-
-    for images, questions, img_ids in tqdm(eval_dataloader):
-        texts = prepare_texts(questions, conv_temp)
-        answers = model.generate(images, texts, max_new_tokens=max_new_tokens, do_sample=False)
-
-        for answer, img_id, question in zip(answers, img_ids, questions):
-
-            # answer = answer.replace("<unk>","").replace(" ","").strip()
-            pattern = r'\{<\d{1,2}><\d{1,2}><\d{1,2}><\d{1,2}>\}'
-            minigpt4_predict[img_id].append(answer)
-            print(img_id)
-            print(answer)
-
-    file_save_path = os.path.join(save_path,"RSNA_inference_result.json")
-    with open(file_save_path,'w') as f:
-        json.dump(minigpt4_predict, f)
-    
-    csv_pth = os.path.join(save_path,"RSNA_IoU_results.csv")
-    clean_detection_json(file_save_path,file_save_path)
-    average_iou(eval_file_path, file_save_path, 1024, 100, "rsna", csv_pth)
-
-
-def process_detect_mimic():
-    eval_file_path = cfg.evaluation_datasets_cfg[dataset]["eval_file_path"]
-    img_path = cfg.evaluation_datasets_cfg[dataset]["img_path"] 
-    batch_size = cfg.evaluation_datasets_cfg[dataset]["batch_size"]
-    max_new_tokens = cfg.evaluation_datasets_cfg[dataset]["max_new_tokens"]
-
-    with open((eval_file_path), 'r') as f:
-        nlst = json.load(f)
-
-    data = evalDetectMimicDataset(nlst, vis_processor, img_path)
-    eval_dataloader = DataLoader(data, batch_size=batch_size, shuffle=False)
-    minigpt4_predict = defaultdict(list)
-    resamples = []
-
-    for images, questions, img_ids in tqdm(eval_dataloader):
-
-        texts = prepare_texts(questions, conv_temp)
-        answers = model.generate(images, texts, max_new_tokens=max_new_tokens, do_sample=False)
-
-        for answer, img_id, question in zip(answers, img_ids, questions):
-            pattern = r'\{<\d{1,2}><\d{1,2}><\d{1,2}><\d{1,2}>\}'
-            minigpt4_predict[img_id].append(answer)
-
-    file_save_path = os.path.join(save_path,"Detect_MIMIC_inference_result.json")
-    with open(file_save_path,'w') as f:
-        json.dump(minigpt4_predict, f)
-    
-
-    csv_pth = os.path.join(save_path,"MIMIC_IoU_results.csv")
-    clean_detection_json(file_save_path,file_save_path)
-    average_iou(eval_file_path, file_save_path, "to be specified soon", 100, "MIMIC", csv_pth)
-
-
-
-def process_SLAKE_dataset():
-    eval_file_path = cfg.evaluation_datasets_cfg[dataset]["eval_file_path"]
-    img_path = cfg.evaluation_datasets_cfg[dataset]["img_path"] 
-    batch_size = cfg.evaluation_datasets_cfg[dataset]["batch_size"]
-    max_new_tokens = cfg.evaluation_datasets_cfg[dataset]["max_new_tokens"]
-
-    with open((eval_file_path), 'r') as f:
-        SLAKE = json.load(f)
-
-    data = evalSLAKEDataset(SLAKE, vis_processor, img_path)
-    eval_dataloader = DataLoader(data, batch_size=batch_size, shuffle=False)
-    minigpt4_predict = defaultdict(list)
-    resamples = []
-
-    for images, questions, img_ids in tqdm(eval_dataloader):
-
-        texts = prepare_texts(questions, conv_temp)
-        answers = model.generate(images, texts, max_new_tokens=max_new_tokens, do_sample=False)
-
-        for answer, img_id, question in zip(answers, img_ids, questions):
-
-            # answer = answer.replace("<unk>","").replace(" ","").strip()
-            pattern = r'\{<\d{1,2}><\d{1,2}><\d{1,2}><\d{1,2}>\}'
-            minigpt4_predict[img_id].append(answer)
-
-    file_save_path = os.path.join(save_path,"SLAKE_inference_result.json")
-    with open(file_save_path,'w') as f:
-        json.dump(minigpt4_predict, f)
-    
-    csv_pth = os.path.join(save_path,"SLAKE_IoU_results.csv")
-    clean_detection_json(file_save_path,file_save_path)
-    average_iou(eval_file_path, file_save_path, 100, 100, "SLAKE", csv_pth)
-    
-
-
-############################################################################
-for dataset in args.dataset:
-    if dataset == 'mimic_cxr':
-        process_mimic_dataset()
-
-    elif dataset == 'radvqa':
-        process_vqa_dataset()
-
-    elif dataset == 'nlst':
-        process_nlst_dataset()
-
-    elif dataset == 'rsna':
-        process_rsna_dataset()
-
-    elif dataset == 'detect_mimic':
-        process_detect_mimic()
-
-    elif dataset == 'SLAKE':
-        process_SLAKE_dataset()
-
-    else:
-        print(f"Dataset '{dataset}' is not supported.")

MiniGPT-Med-github/miniGPTV2.yml

@@ -1,35 +0,0 @@
-name: GPTv2
-channels:
-  - pytorch
-  - defaults
-  - anaconda
-dependencies:
-  - python=3.9
-  - cudatoolkit
-  - pip
-  - pip:
-    - torch==2.0.0
-    - torchaudio
-    - torchvision
-    - huggingface-hub==0.18.0
-    - matplotlib==3.7.0
-    - psutil==5.9.4
-    - iopath
-    - pyyaml==6.0
-    - regex==2022.10.31
-    - tokenizers==0.13.2
-    - tqdm==4.64.1
-    - transformers==4.30.0
-    - timm==0.6.13
-    - webdataset==0.2.48
-    - omegaconf==2.3.0
-    - opencv-python==4.7.0.72
-    - decord==0.6.0
-    - peft==0.2.0
-    - sentence-transformers
-    - gradio==3.47.1
-    - accelerate==0.20.3
-    - bitsandbytes==0.37.0
-    - scikit-image
-    - visual-genome
-    - wandb

MiniGPT-Med-github/miniGPT_Med_.pth

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:ca2d7fc37dc5330cdae927c8a3ff649c5919c726eccb05cae921fb997028b08e
-size 679780138

MiniGPT-Med-github/minigpt4/__init__.py

@@ -1,31 +0,0 @@
-"""
- Copyright (c) 2022, salesforce.com, inc.
- All rights reserved.
- SPDX-License-Identifier: BSD-3-Clause
- For full license text, see the LICENSE_Lavis file in the repo root or https://opensource.org/licenses/BSD-3-Clause
-"""
-
-import os
-import sys
-
-from omegaconf import OmegaConf
-
-from minigpt4.common.registry import registry
-
-from minigpt4.datasets.builders import *
-from minigpt4.models import *
-from minigpt4.processors import *
-from minigpt4.tasks import *
-
-
-root_dir = os.path.dirname(os.path.abspath(__file__))
-default_cfg = OmegaConf.load(os.path.join(root_dir, "configs/default.yaml"))
-
-registry.register_path("library_root", root_dir)
-repo_root = os.path.join(root_dir, "..")
-registry.register_path("repo_root", repo_root)
-cache_root = os.path.join(repo_root, default_cfg.env.cache_root)
-registry.register_path("cache_root", cache_root)
-
-registry.register("MAX_INT", sys.maxsize)
-registry.register("SPLIT_NAMES", ["train", "val", "test"])