Hugging Face's logo

Spaces:
phyloforfun
/
VoucherVision
Running

App Files Community
1
VoucherVision / README.md
phyloforfun's picture
phyloforfun
Major update. Support for 15 LLMs, World Flora Online taxonomy validation, geolocation, 2 OCR methods, significant UI changes, stability improvements, consistent JSON parsing
e91ac58
preview code
|
raw
history blame
22.7 kB

VoucherVision

VoucherVision

Table of Contents

About

VoucherVision - In Beta Testing Phase 🚀

For inquiries, feedback (or if you want to get involved!) please complete our form.

Overview:

Initiated by the University of Michigan Herbarium, VoucherVision harnesses the power of large language models (LLMs) to transform the transcription process of natural history specimen labels. Our workflow is as follows:

  • Text extraction from specimen labels with LeafMachine2.
  • Text interpretation using Google Vision OCR.
  • LLMs, including GPT-3.5, GPT-4, PaLM 2, and Azure instances of OpenAI models, standardize the OCR output into a consistent spreadsheet format. This data can then be integrated into various databases like Specify, Symbiota, and BRAHMS.

For ensuring accuracy and consistency, the VoucherVisionEditor serves as a quality control tool.

Roadmap and New Features List

VoucherVision

  • Update to GPT 1106 builds
  • Option to zip output files for simpler import into VVE
  • Instead of saving a copy of the original image inplace of the OCR/collage images when they are not selected, just change the path to the original image.
  • Expense tracking
    • Dashboard
    • More granular support for different GPT versions
    • Project-based and cummulative tracking
  • Hugging Face Spaces
    • Working and refactored
    • Visualize locations on a map (verbatim and decimal)
    • Tested with batch of 300 images
      • GPT 3.5
      • GPT 4
      • PaLM 2
    • Optimize for +300 images at a time
  • Modular Prompt Builder
    • Build, save, load, submit to VV library
    • Assess whether order of column matters
    • Assess shorter prompt effectiveness
    • Restrict special columns to conform with VVE requirements (catalog_number, coordinates)
  • Option to load existing OCR into VoucherVision workflow

Supported LLM APIs

  • OpenAI
    • GPT 4
    • GPT 4 Turbo 1106-preview
    • GPT 4 32k
    • GPT 3.5
    • GPT 3.5 Instruct
  • OpenAI (Microsoft Azure Endpoints)
    • GPT 4
    • GPT 4 Turbo 1106-preview
    • GPT 4 32k
    • GPT 3.5
    • GPT 3.5 Instruct
  • MistralAI
    • Mistral Tiny
    • Mistral Small
    • Mistral Medium
  • Google PaLM2
    • text-bison@001
    • text-bison@002
    • text-unicorn@001
  • Google Gemini
    • Gemini-Pro

Supported Locally Hosted LLMs

  • MistralAI (24GB+ VRAM GPU Required)
    • Mixtral 8x7B Instruct v0.1
    • Mixtral 7B Instruct v0.2
  • MistralAI (CPU Inference) ((can run on almost computer!))
    • Mixtral 7B Instruct v0.2 GGUF via llama.cpp
  • Meta-Llama2 7B
    • Llama2 7B chat hf

VoucherVisionEditor

  • Streamline the startup procedure
  • Add configurable dropdown menus for certain fields
  • Make sure that VVE can accomodate arbitrary column names
  • Remove legacy support (version 1 prompts)
  • Taxonomy validation helper
  • Visualize locations on a map (verbatim and decimal)
  • More support for datum and verbatim coordinates
  • Compare raw OCR to values in form to flag hallucinations/generated content
  • Accept zipped folders as input
  • Flag user when multiple people/names/determinations are present

Package Information:

The main VoucherVision tool and the VoucherVisionEditor are packaged separately. This separation ensures that lower-performance computers can still install and utilize the editor. While VoucherVision is optimized to function smoothly on virtually any modern system, maximizing its capabilities (like using LeafMachine2 label collages or running Retrieval Augmented Generation (RAG) prompts) mandates a GPU.

NOTE: You can absolutely run VoucherVision on non-GPU systems, but RAG will not be possible (luckily the apparent best prompts 'Version2+' does not use RAG).

Try our public demo!

Our public demo, while lacking several quality control and reliability features found in the full VoucherVision module, provides an exciting glimpse into its capabilities. Feel free to upload your herbarium specimen and see what happens! VoucherVision Demo

Installing VoucherVision

Prerequisites

  • Python 3.10 or later
  • Optional: an Nvidia GPU + CUDA for running LeafMachine2

Installation - Cloning the VoucherVision Repository

  1. First, install Python 3.10, or greater, on your machine of choice. We have validated up to Python 3.11.
    • Make sure that you can use pip to install packages on your machine, or at least inside of a virtual environment.
    • Simply type pip into your terminal or PowerShell. If you see a list of options, you are all set. Otherwise, see either this PIP Documentation or this help page
  2. Open a terminal window and cd into the directory where you want to install VoucherVision.
  3. In the Git BASH terminal, clone the VoucherVision repository from GitHub by running the command: 
    git clone https://github.com/Gene-Weaver/VoucherVision.git
  4. Move into the VoucherVision directory by running cd VoucherVision in the terminal.
  5. To run VoucherVision we need to install its dependencies inside of a python virtual environmnet. Follow the instructions below for your operating system.

About Python Virtual Environments

A virtual environment is a tool to keep the dependencies required by different projects in separate places, by creating isolated python virtual environments for them. This avoids any conflicts between the packages that you have installed for different projects. It makes it easier to maintain different versions of packages for different projects.

For more information about virtual environments, please see Creation of virtual environments

Installation - Windows 10+

Installation should basically be the same for Linux.

Virtual Environment

  1. Still inside the VoucherVision directory, show that a venv is currently not active 
    python --version
  2. Then create the virtual environment (venv_VV is the name of our new virtual environment) 
    python3 -m venv venv_VV
    Or depending on your Python version... 
    python -m venv venv_VV
  3. Activate the virtual environment 
    .\venv_VV\Scripts\activate
  4. Confirm that the venv is active (should be different from step 1) 
    python --version
  5. If you want to exit the venv later for some reason, deactivate the venv using 
    deactivate

Installing Packages

  1. Install the required dependencies to use VoucherVision

    • Option A - If you are using Windows PowerShell:

      pip install wheel streamlit streamlit-extras plotly pyyaml Pillow pandas matplotlib matplotlib-inline tqdm openai langchain tiktoken openpyxl google-generativeai google-cloud-storage google-cloud-vision opencv-python chromadb chroma-migrate InstructorEmbedding transformers sentence-transformers seaborn dask psutil py-cpuinfo azureml-sdk azure-identity ; if ($?) { pip install numpy -U } ; if ($?) { pip install -U scikit-learn } ; if ($?) { pip install --upgrade numpy scikit-learnstreamlit google-generativeai google-cloud-storage google-cloud-vision azureml-sdk azure-identity openai langchain }

    • Option B:

      pip install wheel streamlit streamlit-extras plotly pyyaml Pillow pandas matplotlib matplotlib-inline tqdm openai langchain tiktoken openpyxl google-generativeai google-cloud-storage google-cloud-vision opencv-python chromadb chroma-migrate InstructorEmbedding transformers sentence-transformers seaborn dask psutil py-cpuinfo azureml-sdk azure-identity

  2. Upgrade important packages. Run this if there is an update to VoucherVision.

    pip install --upgrade numpy scikit-learn streamlit google-generativeai google-cloud-storage google-cloud-vision azureml-sdk azure-identity openai langchain

  3. Install PyTorch

    • The LeafMachine2 machine learning algorithm requires PyTorch. If your computer does not have a GPU, then please install a version of PyTorch that is for CPU only. If your computer does have an Nvidia GPU, then please determine which version of PyTorch matches your current CUDA version. Please see Troubleshooting CUDA for help. PyTorch is large and will take a bit to install.

    • WITH GPU (or visit PyTorch.org to find the appropriate version of PyTorch for your CUDA version)

      pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0 --extra-index-url https://download.pytorch.org/whl/cu113

    • WITHOUT GPU, CPU ONLY

      pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 --index-url https://download.pytorch.org/whl/cpu

If you need help, please submit an inquiry in the form at LeafMachine.org

Troubleshooting CUDA

  • If your system already has another version of CUDA (e.g., CUDA 11.7) then it can be complicated to switch to CUDA 11.3.
  • The simplest solution is to install pytorch with CPU only, avoiding the CUDA problem entirely.
  • Alternatively, you can install the latest pytorch release for your specific system, either using the cpu only version pip3 install torch, pip3 install torchvision, pip3 install torchaudio or by matching the pythorch version to your CUDA version.
  • We have not validated CUDA 11.6 or CUDA 11.7, but our code is likely to work with them too. If you have success with other versions of CUDA/pytorch, let us know and we will update our instructions.

Create a Desktop Shortcut to Launch VoucherVision GUI

We can create a desktop shortcut to launch VoucherVision. In the ../VoucherVision/ directory is a file called create_desktop_shortcut.py. In the terminal, move into the ../VoucherVision/ directory and type:

python create_desktop_shortcut.py

Or...

python3 create_desktop_shortcut.py

Follow the instructions, select where you want the shortcut to be created, then where the virtual environment is located.

Run VoucherVision

  1. In the terminal, make sure that you cd into the VoucherVision directory and that your virtual environment is active (you should see venv_VV on the command line).
  2. Type: 
    python run_VoucherVision.py
    or depending on your Python installation: 
    python3 run_VoucherVision.py
  3. If you ever see an error that says that a "port is not available", open run_VoucherVision.py in a plain text editor and change the --port value to something different but close, like 8502.

Setting up API key

VoucherVision requires access to Google Vision OCR and at least one of the following LLMs: OpenAI API, Google PaLM 2, a private instance of OpenAI through Microsoft Azure. On first startup, you will see a page with instructions on how to get these API keys. Nothing will work until you get at least the Google Vision OCR API key and one LLM API key.

Check GPU

Press the "Check GPU" button to see if you have a GPU available. If you know that your computer has an Nvidia GPU, but the check fails, then you need to install an different version of PyTorch in the virtual environment.

Run Tests

Once you have provided API keys, you can test all available prompts and LLMs by pressing the test buttons. Every combination of LLM, prompt, and LeafMachine2 collage will run on the image in the ../VoucherVision/demo/demo_images folder. A grid will appear letting you know which combinations are working on your system.

Starting VoucherVision

  1. "Run name" - Set a run name for your project. This will be the name of the new folder that contains the output files.

  2. "Output directory" - Paste the full file path of where you would like to save the folder that will be created in step 1.

  3. "Input images directory" - Paste the full file path of where the input images are located. This folder can only have JPG or JPEG images inside of it.

  4. "Select an LLM" - Pick the LLM you want to use to parse the unstructured OCR text.

    • As of Nov. 1, 2023 PaLM 2 is free to use.

  5. "Prompt Version" - Pick your prompt version. We recommend "Version 2" for production use, but you can experiment with our other prompts.

  6. "Cropped Components" - Check the box to use LeafMachine2 collage images as the input file. LeafMachine2 can often find small handwritten text that may be missed by Google Vision OCR's text detection algorithm. But, the difference in performance is not that big. You will still get good performance without using the LeafMachine2 collage images.

  7. "Domain Knowledge" is only used for "Version 1" prompts.

  8. "Component Detector" sets basic LeafMachine2 parameters, but the default is likely good enough.

  9. "Processing Options"

    • The image file name defines the row name in the final output spreadsheet.
    • We provide some basic options to clean/parse the image file name to produce the desired output.
    • For example, if the input image name is MICH-V-3819482.jpg but the desired name is just 3819482 you can add MICH-V- to the "Remove prefix from catalog number" input box. Alternatively, you can check the "Require Catalog..." box and achieve the same result.

  10. Finally you can press the start processing button.

Azure Instances of OpenAI

If your institution has an enterprise instance of OpenAI's services, like at the University of Michigan, you can use Azure instead of the OpenAI servers. Your institution should be able to provide you with the required keys (there are 5 required keys for this service).

Custom Prompt Builder

VoucherVision empowers individual institutions to customize the format of the LLM output. Using our pre-defined prompts you can transcribe the label text into 20 columns, but using our Prompt Builder you can load one of our default prompts and adjust the output to meet your needs. More instructions will come soon, but for now here are a few more details.

Load, Build, Edit

The Prompt Builder creates a prompt in the structure that VoucherVision expects. This information is stored as a configuration yaml file in ../VoucherVision/custom_prompts/. We provide a few versions to get started. You can load one of our examples and then use the Prompt Builder to edit or add new columns.

prompt_1

Instructions

Right now, the prompting instructions are not configurable, but that may change in the future.

prompt_2

Defining Column Names Field-Specific Instructions

The central JSON object shows the structure of the columns that you are requesting the LLM to create and populate with information from the specimen's labels. These will become the rows in the final xlsx file the VoucherVision generates. You can pick formatting instructions, set default values, and give detailed instructions.

Note: formatting instructions are not always followed precisely by the LLM. For example, GPT-4 is capable of granular instructions like converting ALL CAPS TEXT to sentence-case, but GPT-3.5 and PaLM 2 might not be capable of following that instruction every time (which is why we have the VoucherVisionEditor and are working to link these instructions so that humans editing the output can quickly/easily fix these errors).

prompt_3

Prompting Structure

The rightmost JSON object is the entire prompt structure. If you load the required_structure.yaml prompt, you will wee the bare-bones version of what VoucherVision expects to see. All of the parts are there for a reason. The Prompt Builder UI may be a little unruly right now thanks to quirks with Streamlit, but we still recommend using the UI to build your own prompts to make sure that all of the required components are present.

prompt_4

Mapping Columns for VoucherVisionEditor

Finally, we need to map columns to a VoucherVisionEditor category.

prompt_5

Expense Reporting

VoucherVision logs the number of input and output tokens (using tiktoken) from every call. We store the publicly listed prices of the LLM APIs in ../VoucherVision/api_cost/api_cost.yaml. Then we do some simple math to estimage the cost of run, which is stored inside of your project's output directory ../run_name/Cost/run_name.csv and all runs are accumulated in a csv file stored in ../VoucherVision/expense_report/expense_report.csv. VoucherVision only manages expense_report.csv, so if you want to split costs by month/quarter then copy and rename expense_report.csv. Deleting expense_report.csv will let you accumulate more stats.

This should be treated as an estimate. The true cost may be slightly different.

This is an example of the stats that we track:

run date api_version total_cost n_images tokens_in tokens_out rate_in rate_out cost_in cost_out
GPT4_test_run1 2023_11_05__17-44-31 GPT_4 0.23931 2 6749 614 0.03 0.06 0.20247 0.03684
GPT_3_5_test_run 2023_11_05__17-48-48 GPT_3_5 0.0189755 4 12033 463 0.0015 0.002 0.0180495 0.000926
PALM2_test_run 2023_11_05__17-50-35 PALM2 0 4 13514 771 0 0 0 0
GPT4_test_run2 2023_11_05__18-49-24 GPT_4 0.40962 4 12032 811 0.03 0.06 0.36096 0.04866

Expense Report Dashboard

The sidebar in VoucherVision displays summary stats taken from expense_report.csv. Expense Report Dashboard

User Interface Images

Validation test when the OpenAI key is not provided, but keys for PaLM 2 and Azure OpenAI are present: Validation 1

Validation test when all versions of the OpenAI keys are provided: Validation GPT

A successful GPU test: Validation GPU

Successful PaLM 2 test: Validation PaLM