---
language:
- de
pipeline_tag: image-to-text
---
# Ingredient Scanner
## Abstract

With the recent advancements in computer vision and optical character recognition and using a convolutional neural network to cut out the product from a picture, it has now become possible to reliably extract ingredient lists from the back of a product using the Anthropic API. Open-weight or even only on-device optical character recognition lacks the quality to be used in a production environment, although the progress in development is promising. The Anthropic API is also currently not feasible due to the high cost of 1 Swiss Franc per 100 pictures.

The training code and data is available on [GitHub](https://github.com/lenamerkli/ingredient-scanner/). This repository just contains an inference example and the [report](https://huggingface.co/lenamerkli/ingredient-scanner/blob/main/ingredient-scanner.pdf).

This is an entry for the [2024 Swiss AI competition](https://www.ki-wettbewerb.ch/).

## Table of Contents

0. [Abstract](#abstract)
1. [Report](#report)
2. [Model Details](#model-details)
3. [Usage](#usage)
4. [Citation](#citation)

## Report
Read the full report [here](https://huggingface.co/lenamerkli/ingredient-scanner/blob/main/ingredient-scanner.pdf).

## Model Details
This repository consists of two models, one vision model and a large language model.

### Vision Model
Custom convolutional neural network based on [ResNet18](https://pytorch.org/hub/pytorch_vision_resnet/). It detects the four corner points and the upper and lower limits of a product.

### Language Model
Converts the text from the optical character recognition engine which lies in-between the two models to JSON. It is fine-tuned from [unsloth/Qwen2-0.5B-Instruct-bnb-4bit](https://huggingface.co/unsloth/Qwen2-0.5B-Instruct-bnb-4bit).

## Usage
Clone the repository and install the dependencies on any debian-based system:
```bash
git clone https://huggingface.co/lenamerkli/ingredient-scanner
cd ingredient-scanner
python3 -m venv .venv
source .venv/bin/activate
pip3 install -r requirements.txt
```
Note: not all requirements are needed for inference, as both training and inference requirements are listed.

Select the OCR engine in `main.py` by uncommenting one of the lines 20 to 22:
```python
# ENGINE: list[str] = ['easyocr']
# ENGINE: list[str] = ['anthropic', 'claude-3-5-sonnet-20240620']
# ENGINE: list[str] = ['llama_cpp/v2/vision', 'qwen-vl-next_b2583']
```
Note: Qwen-VL-Next is not an official qwen model. This is only to protect business secrets of a private model.

Run the inference script:
```bash
python3 main.py
```

You will be asked to enter the file path to a PNG image.

### Anthropic API

If you want to use the Anthropic API, create a `.env` file with the following content:
```
ANTHROPIC_API_KEY=YOUR_API_KEY
```

## Citation
Here is how to cite this paper in the bibtex format:
```bibtex
@misc{merkli2024ingriedient-scanner,
    title={Ingredient Scanner: Automating Reading of Ingredient Labels with Computer Vision},
    author={Lena Merkli and Sonja Merkli},
    date={2024-07-16},
    url={https://huggingface.co/lenamerkli/ingredient-scanner},
}
```