Updated user guide and app settings. Updated some additional lambda_entrypoint arguments. Ensured that examples are correctly displayed on GUI.

README.md

@@ -10,7 +10,7 @@ license: agpl-3.0
 ---
 # Document redaction
 
-version: 1.4.0
+version: 1.4.1
 
 Redact personally identifiable information (PII) from documents (pdf, png, jpg), Word files (docx), or tabular data (xlsx/csv/parquet). Please see the [User Guide](#user-guide) for a full walkthrough of all the features in the app.
     
@@ -204,11 +204,12 @@ These settings are only relevant if you intend to use AWS services like Textract
 
 Now you have the app installed, what follows is a guide on how to use it for basic and advanced redaction.
 
-# User Guide
+# User guide
 
 ## Table of contents
 
-- [Example data files](#example-data-files)
+### Getting Started
+- [Built-in example data](#built-in-example-data)
 - [Basic redaction](#basic-redaction)
 - [Customising redaction options](#customising-redaction-options)
     - [Custom allow, deny, and page redaction lists](#custom-allow-deny-and-page-redaction-lists)
@@ -220,21 +221,60 @@ Now you have the app installed, what follows is a guide on how to use it for bas
     - [Handwriting and signature redaction](#handwriting-and-signature-redaction)
 - [Reviewing and modifying suggested redactions](#reviewing-and-modifying-suggested-redactions)
 - [Redacting Word, tabular data files (CSV/XLSX) or copy and pasted text](#redacting-word-tabular-data-files-xlsxcsv-or-copy-and-pasted-text)
-
-See the [advanced user guide here](#advanced-user-guide):
-- [Merging redaction review files](#merging-redaction-review-files)
 - [Identifying and redacting duplicate pages](#identifying-and-redacting-duplicate-pages)
+
+### Advanced user guide
+- [Advanced user guide](#advanced-user-guide)
 - [Fuzzy search and redaction](#fuzzy-search-and-redaction)
 - [Export redactions to and import from Adobe Acrobat](#export-to-and-import-from-adobe)
+    - [Using _for_review.pdf files with Adobe Acrobat](#using-_for_reviewpdf-files-with-adobe-acrobat)
     - [Exporting to Adobe Acrobat](#exporting-to-adobe-acrobat)
     - [Importing from Adobe Acrobat](#importing-from-adobe-acrobat)
 - [Using the AWS Textract document API](#using-the-aws-textract-document-api)
 - [Using AWS Textract and Comprehend when not running in an AWS environment](#using-aws-textract-and-comprehend-when-not-running-in-an-aws-environment)
 - [Modifying existing redaction review files](#modifying-existing-redaction-review-files)
+- [Merging redaction review files](#merging-redaction-review-files)
+
+### Features for expert users/system administrators
+- [Features for expert users/system administrators](#features-for-expert-userssystem-administrators)
+- [Advanced OCR options (Hybrid OCR)](#advanced-ocr-options-hybrid-ocr)
+- [Command Line Interface (CLI)](#command-line-interface-cli)
+
+## Built-in example data
+
+The app now includes built-in example files that you can use to quickly test different features. These examples are automatically loaded and can be accessed directly from the interface without needing to download files separately.
+
+### Using built-in examples
+
+**For PDF/image redaction:** On the 'Redact PDFs/images' tab, you'll see a section titled "Try an example - Click on an example below and then the 'Extract text and redact document' button". Simply click on any of the available examples to load them with pre-configured settings:
+
+- **PDF with selectable text redaction** - Uses local text extraction with standard PII detection
+- **Image redaction with local OCR** - Processes an image file using OCR
+- **PDF redaction with custom entities** - Demonstrates custom entity selection (Titles, Person, Dates)
+- **PDF redaction with AWS services and signature detection** - Shows AWS Textract with signature extraction (if AWS is enabled)
+- **PDF redaction with custom deny list and whole page redaction** - Demonstrates advanced redaction features
+
+Once you have clicked on an example, you can click the 'Extract text and redact document' button to load the example into the app and redact it.
+
+**For tabular data:** On the 'Word or Excel/csv files' tab, you'll find examples for both redaction and duplicate detection:
 
-## Example data files
+- **CSV file redaction** - Shows how to redact specific columns in tabular data
+- **Word document redaction** - Demonstrates Word document processing
+- **Excel file duplicate detection** - Shows how to find duplicate rows in spreadsheet data
+
+Once you have clicked on an example, you can click the 'Redact text/data files' button to load the example into the app and redact it. For the duplicate detection example, you can click the 'Find duplicate cells/rows' button to load the example into the app and find duplicates.
+
+**For duplicate page detection:** On the 'Identify duplicate pages' tab, you'll find examples for finding duplicate content in documents:
+
+- **Find duplicate pages of text in document OCR outputs** - Uses page-level analysis with a similarity threshold of 0.95 and minimum word count of 10
+- **Find duplicate text lines in document OCR outputs** - Uses line-level analysis with a similarity threshold of 0.95 and minimum word count of 3
+
+Once you have clicked on an example, you can click the 'Identify duplicate pages/subdocuments' button to load the example into the app and find duplicate content.
+
+### External example files (optional)
+
+If you prefer to use your own example files or want to follow along with specific tutorials, you can still download these external example files:
 
-Please try these example files to follow along with this guide: 
 - [Example of files sent to a professor before applying](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/example_of_emails_sent_to_a_professor_before_applying.pdf)
 - [Example complaint letter (jpg)](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/example_complaint_letter.jpg)
 - [Partnership Agreement Toolkit (for signatures and more advanced usage)](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/Partnership-Agreement-Toolkit_0_0.pdf)
@@ -254,16 +294,20 @@ The 'Redact PDFs/images tab' currently accepts PDFs and image files (JPG, PNG) f
 
 ### Text extraction
 
-First, select one of the three text extraction options:
+You can modify default text extraction methods by clicking on the 'Change default text extraction method...' box'.
+
+Here you can select one of the three text extraction options:
 - **'Local model - selectable text'** - This will read text directly from PDFs that have selectable text to redact (using PikePDF). This is fine for most PDFs, but will find nothing if the PDF does not have selectable text, and it is not good for handwriting or signatures. If it encounters an image file, it will send it onto the second option below.
 - **'Local OCR model - PDFs without selectable text'** - This option will use a simple Optical Character Recognition (OCR) model (Tesseract) to pull out text from a PDF/image that it 'sees'. This can handle most typed text in PDFs/images without selectable text, but struggles with handwriting/signatures. If you are interested in the latter, then you should use the third option if available.
 - **'AWS Textract service - all PDF types'** - Only available for instances of the app running on AWS. AWS Textract is a service that performs OCR on documents within their secure service. This is a more advanced version of OCR compared to the local option, and carries a (relatively small) cost. Textract excels in complex documents based on images, or documents that contain a lot of handwriting and signatures.
 
-### Optional - select signature extraction
+### Enable AWS Textract signature extraction
 If you chose the AWS Textract service above, you can choose if you want handwriting and/or signatures redacted by default. Choosing signatures here will have a cost implication, as identifying signatures will cost ~£2.66 ($3.50) per 1,000 pages vs ~£1.14 ($1.50) per 1,000 pages without signature detection. 
 
 ![AWS Textract handwriting and signature options](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/quick_start/textract_handwriting_signatures.PNG)
 
+**NOTE:** it is also possible to enable form extraction, layout extraction, and table extraction with AWS Textract. This is not enabled by default, but it is possible for your system admin to enable this feature in the config file.
+
 ### PII redaction method
 
 If you are running with the AWS service enabled, here you will also have a choice for PII redaction method:
@@ -297,6 +341,7 @@ Click 'Redact document'. After loading in the document, the app should be able t
 ![Redaction outputs](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/quick_start/redaction_outputs.PNG)
 
 - **'...redacted.pdf'** files contain the original pdf with suggested redacted text deleted and replaced by a black box on top of the document.
+- **'...redactions_for_review.pdf'** files contain the original PDF with redaction boxes overlaid but the original text still visible underneath. This file is designed for use in Adobe Acrobat and other PDF viewers where you can see the suggested redactions without the text being permanently removed. This is particularly useful for reviewing redactions before finalising them.
 - **'...ocr_results.csv'** files contain the line-by-line text outputs from the entire document. This file can be useful for later searching through for any terms of interest in the document (e.g. using Excel or a similar program).
 - **'...review_file.csv'** files are the review files that contain details and locations of all of the suggested redactions in the document. This file is key to the [review process](#reviewing-and-modifying-suggested-redactions), and should be downloaded to use later for this.
 
@@ -365,8 +410,6 @@ If the table is empty, you can add a new entry, you can add a new row by clickin
 
 ![Manually modify allow or deny list filled](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/allow_list/manually_modify_filled.PNG)
 
-**Note:** As of version 0.7.0 you can now apply your whole page redaction list directly to the document file currently under review by clicking the 'Apply whole page redaction list to document currently under review' button that appears here.
-
 ### Redacting additional types of personal information
 
 You may want to redact additional types of information beyond the defaults, or you may not be interested in default suggested entity types. There are dates in the example complaint letter. Say we wanted to redact those dates also?
@@ -381,7 +424,7 @@ If you want to redact different files, I suggest you refresh your browser page t
 
 ## Redacting only specific pages
 
-Say also we are only interested in redacting page 1 of the loaded documents. On the Redaction settings tab, select 'Lowest page to redact' as 1, and 'Highest page to redact' also as 1. When you next redact your documents, only the first page will be modified.
+Say also we are only interested in redacting page 1 of the loaded documents. On the Redaction settings tab, select 'Lowest page to redact' as 1, and 'Highest page to redact' also as 1. When you next redact your documents, only the first page will be modified. The output files should now have a suffix similar to '..._1_1.pdf', indicating the lowest and highest page numbers that were redacted.
 
 ![Selecting specific pages to redact](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/allow_list/select_pages.PNG)
 
@@ -618,39 +661,16 @@ You can also write open text into an input box and redact that using the same me
 ### Redaction log outputs
 A list of the suggested redaction outputs from the tabular data / open text data redaction is available on the Redaction settings page under 'Log file outputs'.
 
-# ADVANCED USER GUIDE
-
-This advanced user guide will go over some of the features recently added to the app, including: modifying and merging redaction review files, identifying and redacting duplicate pages across multiple PDFs, 'fuzzy' search and redact, and exporting redactions to Adobe Acrobat.
-
-## Table of contents
-
-- [Merging redaction review files](#merging-redaction-review-files)
-- [Identifying and redacting duplicate pages](#identifying-and-redacting-duplicate-pages)
-- [Fuzzy search and redaction](#fuzzy-search-and-redaction)
-- [Export redactions to and import from Adobe Acrobat](#export-to-and-import-from-adobe)
-    - [Exporting to Adobe Acrobat](#exporting-to-adobe-acrobat)
-    - [Importing from Adobe Acrobat](#importing-from-adobe-acrobat)
-- [Using the AWS Textract document API](#using-the-aws-textract-document-api)
-- [Using AWS Textract and Comprehend when not running in an AWS environment](#using-aws-textract-and-comprehend-when-not-running-in-an-aws-environment)
-- [Modifying existing redaction review files](#modifying-existing-redaction-review-files)
-
-
-## Merging redaction review files
-
-Say you have run multiple redaction tasks on the same document, and you want to merge all of these redactions together. You could do this in your spreadsheet editor, but this could be fiddly especially if dealing with multiple review files or large numbers of redactions. The app has a feature to combine multiple review files together to create a 'merged' review file.
-
-![Merging review files in the user interface](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/merge_review_files/img/merge_review_files_interface.PNG)
-
-You can find this option at the bottom of the 'Redaction Settings' tab. Upload multiple review files here to get a single output 'merged' review_file. In the examples file, merging the 'review_file_custom.csv' and 'review_file_local.csv' files give you an output containing redaction boxes from both. This combined review file can then be uploaded into the review tab following the usual procedure.
-
-![Merging review files outputs in spreadsheet](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/merge_review_files/img/merged_review_file_outputs_csv.PNG)
-
 ## Identifying and redacting duplicate pages
 
 The files for this section are stored [here](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/duplicate_page_find_in_app/).
 
 Some redaction tasks involve removing duplicate pages of text that may exist across multiple documents. This feature helps you find and remove duplicate content that may exist in single or multiple documents.  It can identify everything from single identical pages to multi-page sections (subdocuments). The process involves three main steps: configuring the analysis, reviewing the results in the interactive interface, and then using the generated files to perform the redactions.
 
+### Duplicate page detection in documents
+
+This section covers finding duplicate pages across PDF documents using OCR output files.
+
 ![Example duplicate page inputs](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/duplicate_page_find_in_app/img/duplicate_page_input_interface_new.PNG)
 
 **Step 1: Upload and Configure the Analysis**
@@ -695,11 +715,43 @@ The analysis also generates a set of downloadable files for your records and for
 
 If you want to combine the results from this redaction process with previous redaction tasks for the same PDF, you could merge review file outputs following the steps described in [Merging existing redaction review files](#merging-existing-redaction-review-files) above.
 
+### Duplicate detection in tabular data
+
+The app also includes functionality to find duplicate cells or rows in CSV, Excel, or Parquet files. This is particularly useful for cleaning datasets where you need to identify and remove duplicate entries.
+
+**Step 1: Upload files and configure analysis**
+
+Navigate to the 'Word or Excel/csv files' tab and scroll down to the "Find duplicate cells in tabular data" section. Upload your tabular files (CSV, Excel, or Parquet) and configure the analysis parameters:
+
+- **Similarity threshold**: Score (0-1) to consider cells a match. 1 = perfect match
+- **Minimum word count**: Cells with fewer words than this value are ignored
+- **Do initial clean of text**: Remove URLs, HTML tags, and non-ASCII characters
+- **Remove duplicate rows**: Automatically remove duplicate rows from deduplicated files
+- **Select Excel sheet names**: Choose which sheets to analyze (for Excel files)
+- **Select text columns**: Choose which columns contain text to analyze
+
+**Step 2: Review results**
+
+After clicking "Find duplicate cells/rows", the results will be displayed in a table showing:
+- File1, Row1, File2, Row2
+- Similarity_Score
+- Text1, Text2 (the actual text content being compared)
+
+Click on any row to see more details about the duplicate match in the preview boxes below.
+
+**Step 3: Remove duplicates**
+
+Select a file from the dropdown and click "Remove duplicate rows from selected file" to create a cleaned version with duplicates removed. The cleaned file will be available for download.
+
+# Advanced user guide
+
+This advanced user guide covers features that require system administration access or command-line usage. These features are typically used by system administrators or advanced users who need more control over the redaction process.
+
 ## Fuzzy search and redaction
 
 The files for this section are stored [here](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/fuzzy_search/).
 
-Sometimes you may be searching for terns that are slightly mispelled throughout a document, for example names. The document redaction app gives the option for searching for long phrases that may contain spelling mistakes, a method called 'fuzzy matching'.
+Sometimes you may be searching for terms that are slightly mispelled throughout a document, for example names. The document redaction app gives the option for searching for long phrases that may contain spelling mistakes, a method called 'fuzzy matching'.
 
 To do this, go to the Redaction Settings, and the 'Select entity types to redact' area. In the box below relevant to your chosen redaction method (local or AWS Comprehend), select 'CUSTOM_FUZZY' from the list. Next, we can select the maximum number of spelling mistakes allowed in the search (up to nine). Here, you can either type in a number or use the small arrows to the right of the box. Change this option to 3. This will allow for a maximum of three 'changes' in text needed to match to the desired search terms.
 
@@ -719,9 +771,20 @@ Using these deny list with spelling mistakes, the app fuzzy match these terms to
 
 Files for this section are stored [here](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/export_to_adobe/).
 
-### Exporting to Adobe Acrobat
+The Document Redaction app has enhanced features for working with Adobe Acrobat. You can now export suggested redactions to Adobe, import Adobe comment files into the app, and use the new `_for_review.pdf` files directly in Adobe Acrobat.
+
+### Using _for_review.pdf files with Adobe Acrobat
 
-The Document Redaction app has a feature to export suggested redactions to Adobe, and likewise to import Adobe comment files into the app. The file format used is the .xfdf Adobe comment file format - [you can find more information about how to use these files here](https://helpx.adobe.com/uk/acrobat/using/importing-exporting-comments.html).
+The app now generates `...redactions_for_review.pdf` files that contain the original PDF with redaction boxes overlaid but the original text still visible underneath. These files are specifically designed for use in Adobe Acrobat and other PDF viewers where you can:
+
+- See the suggested redactions without the text being permanently removed
+- Review redactions before finalising them
+- Use Adobe Acrobat's built-in redaction tools to modify or apply the redactions
+- Export the final redacted version directly from Adobe
+
+Simply open the `...redactions_for_review.pdf` file in Adobe Acrobat to begin reviewing and modifying the suggested redactions.
+
+### Exporting to Adobe Acrobat
 
 To convert suggested redactions to Adobe format, you need to have the original PDF and a review file csv in the input box at the top of the Review redactions page.
 
@@ -769,6 +832,46 @@ The '_textract.json' output can be used to speed up further redaction tasks as [
 
 You can now easily get the '..._ocr_output.csv' redaction output based on this '_textract.json' (described in [Redaction outputs](#redaction-outputs)) by clicking on the button 'Convert Textract job outputs to OCR results'. You can now use this file e.g. for [identifying duplicate pages](#identifying-and-redacting-duplicate-pages), or for redaction review.
 
+
+
+## Modifying existing redaction review files
+You can find the folder containing the files discussed in this section [here](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/merge_review_files/).
+
+As well as serving as inputs to the document redaction app's review function, the 'review_file.csv' output can be modified insider or outside of the app. This gives you the flexibility to change redaction details outside of the app.
+
+### Inside the app
+You can now modify redaction review files directly in the app on the 'Review redactions' tab. Open the accordion 'View and edit review data' under the file input area. You can edit review file data cells here - press Enter to apply changes. You should see the effect on the current page if you click the 'Save changes on current page to file' button to the right.
+
+### Outside the app
+If you open up a 'review_file' csv output using a spreadsheet software program such as Microsoft Excel you can easily modify redaction properties. Open the file '[Partnership-Agreement-Toolkit_0_0_redacted.pdf_review_file_local.csv](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/merge_review_files/Partnership-Agreement-Toolkit_0_0.pdf_review_file_local.csv)', and you should see a spreadshet with just four suggested redactions (see below). The following instructions are for using Excel.
+
+![Review file before](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/merge_review_files/img/review_file_before.PNG)
+
+The first thing we can do is remove the first row - 'et' is suggested as a person, but is obviously not a genuine instance of personal information. Right click on the row number and select delete on this menu. Next, let's imagine that what the app identified as a 'phone number' was in fact another type of number and so we wanted to change the label. Simply click on the relevant label cells, let's change it to 'SECURITY_NUMBER'. You could also use 'Find & Select' -> 'Replace' from the top ribbon menu if you wanted to change a number of labels simultaneously.
+
+How about we wanted to change the colour of the 'email address' entry on the redaction review tab of the redaction app? The colours in a review file are based on an RGB scale with three numbers ranging from 0-255. [You can find suitable colours here](https://rgbcolorpicker.com). Using this scale, if I wanted my review box to be pure blue, I can change the cell value to (0,0,255).
+
+Imagine that a redaction box was slightly too small, and I didn't want to use the in-app options to change the size. In the review file csv, we can modify e.g. the ymin and ymax values for any box to increase the extent of the redaction box. For the 'email address' entry, let's decrease ymin by 5, and increase ymax by 5.
+
+I have saved an output file following the above steps as '[Partnership-Agreement-Toolkit_0_0_redacted.pdf_review_file_local_mod.csv](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/merge_review_files/outputs/Partnership-Agreement-Toolkit_0_0.pdf_review_file_local_mod.csv)' in the same folder that the original was found. Let's upload this file to the app along with the original pdf to see how the redactions look now.
+
+![Review file after modification](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/merge_review_files/img/partnership_redactions_after.PNG)
+
+We can see from the above that we have successfully removed a redaction box, changed labels, colours, and redaction box sizes.
+
+## Merging redaction review files
+
+Say you have run multiple redaction tasks on the same document, and you want to merge all of these redactions together. You could do this in your spreadsheet editor, but this could be fiddly especially if dealing with multiple review files or large numbers of redactions. The app has a feature to combine multiple review files together to create a 'merged' review file.
+
+![Merging review files in the user interface](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/merge_review_files/img/merge_review_files_interface.PNG)
+
+You can find this option at the bottom of the 'Redaction Settings' tab. Upload multiple review files here to get a single output 'merged' review_file. In the examples file, merging the 'review_file_custom.csv' and 'review_file_local.csv' files give you an output containing redaction boxes from both. This combined review file can then be uploaded into the review tab following the usual procedure.
+
+![Merging review files outputs in spreadsheet](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/merge_review_files/img/merged_review_file_outputs_csv.PNG)
+
+# Features for expert users/system administrators
+This advanced user guide covers features that require system administration access or command-line usage. These options are not enabled by default but can be configured by your system administrator, and are not available to users who are just using the graphical user interface. These features are typically used by system administrators or advanced users who need more control over the redaction process.
+
 ## Using AWS Textract and Comprehend when not running in an AWS environment
 
 AWS Textract and Comprehend give much better results for text extraction and document redaction than the local model options in the app. The most secure way to access them in the Redaction app is to run the app in a secure AWS environment with relevant permissions. Alternatively, you could run the app on your own system while logged in to AWS SSO with relevant permissions.
@@ -790,26 +893,180 @@ The app should then pick up these keys when trying to access the AWS Textract an
 
 Again, a lot can potentially go wrong with AWS solutions that are insecure, so before trying the above please consult with your AWS and data security teams.
 
-## Modifying existing redaction review files
+## Advanced OCR options (Hybrid OCR)
 
-*Note:* As of version 0.7.0 you can now modify redaction review files directly in the app on the 'Review redactions' tab. Open the accordion 'View and edit review data' under the file input area. You can edit review file data cells here - press Enter to apply changes. You should see the effect on the current page if you click the 'Save changes on current page to file' button to the right.
+The app supports advanced OCR options that combine multiple OCR engines for improved accuracy. These options are not enabled by default but can be configured by your system administrator.
 
-You can find the folder containing the files discussed in this section [here](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/merge_review_files/).
+### Available OCR models
 
-As well as serving as inputs to the document redaction app's review function, the 'review_file.csv' output can be modified outside of the app, and also merged with others from multiple redaction attempts on the same file. This gives you the flexibility to change redaction details outside of the app.
+- **Tesseract** (default): The standard OCR engine that works well for most documents
+- **PaddleOCR**: More accurate for whole line text extraction, but word-level bounding boxes may be less precise
+- **Hybrid**: Combines Tesseract and PaddleOCR - uses Tesseract for initial extraction, then PaddleOCR for re-extraction of low-confidence text
 
-If you open up a 'review_file' csv output using a spreadsheet software program such as Microsoft Excel you can easily modify redaction properties. Open the file '[Partnership-Agreement-Toolkit_0_0_redacted.pdf_review_file_local.csv](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/merge_review_files/Partnership-Agreement-Toolkit_0_0.pdf_review_file_local.csv)', and you should see a spreadshet with just four suggested redactions (see below). The following instructions are for using Excel.
+### Enabling advanced OCR options
 
-![Review file before](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/merge_review_files/img/review_file_before.PNG)
+To enable these options, your system administrator needs to modify the configuration file (`config.py`) and set:
 
-The first thing we can do is remove the first row - 'et' is suggested as a person, but is obviously not a genuine instance of personal information. Right click on the row number and select delete on this menu. Next, let's imagine that what the app identified as a 'phone number' was in fact another type of number and so we wanted to change the label. Simply click on the relevant label cells, let's change it to 'SECURITY_NUMBER'. You could also use 'Find & Select' -> 'Replace' from the top ribbon menu if you wanted to change a number of labels simultaneously.
+```
+SHOW_LOCAL_OCR_MODEL_OPTIONS = "True"
+```
 
-How about we wanted to change the colour of the 'email address' entry on the redaction review tab of the redaction app? The colours in a review file are based on an RGB scale with three numbers ranging from 0-255. [You can find suitable colours here](https://rgbcolorpicker.com). Using this scale, if I wanted my review box to be pure blue, I can change the cell value to (0,0,255).
+Once enabled, users will see a "Change default local OCR model" section in the redaction settings where they can choose between:
+- tesseract
+- hybrid  
+- paddle
 
-Imagine that a redaction box was slightly too small, and I didn't want to use the in-app options to change the size. In the review file csv, we can modify e.g. the ymin and ymax values for any box to increase the extent of the redaction box. For the 'email address' entry, let's decrease ymin by 5, and increase ymax by 5.
+### Hybrid OCR configuration
 
-I have saved an output file following the above steps as '[Partnership-Agreement-Toolkit_0_0_redacted.pdf_review_file_local_mod.csv](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/merge_review_files/outputs/Partnership-Agreement-Toolkit_0_0.pdf_review_file_local_mod.csv)' in the same folder that the original was found. Let's upload this file to the app along with the original pdf to see how the redactions look now.
+The hybrid OCR mode uses several configurable parameters:
 
-![Review file after modification](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/merge_review_files/img/partnership_redactions_after.PNG)
+- **HYBRID_OCR_CONFIDENCE_THRESHOLD** (default: 65): Tesseract confidence score below which PaddleOCR will be used for re-extraction
+- **HYBRID_OCR_PADDING** (default: 1): Padding added to word bounding boxes before re-extraction
+- **SAVE_EXAMPLE_TESSERACT_VS_PADDLE_IMAGES** (default: False): Save comparison images when using hybrid mode
+- **SAVE_PADDLE_VISUALISATIONS** (default: False): Save images with PaddleOCR bounding boxes overlaid
+
+### When to use different OCR models
+
+- **Tesseract**: Best for general use, good balance of speed and accuracy
+- **PaddleOCR**: Best for documents with clear, well-formatted text where line-level accuracy is more important than word-level precision
+- **Hybrid**: Best for challenging documents where some text has low confidence scores, providing the benefits of both engines
 
-We can see from the above that we have successfully removed a redaction box, changed labels, colours, and redaction box sizes.
+
+
+
+
+## Command Line Interface (CLI)
+
+The app includes a comprehensive command-line interface (`cli_redact.py`) that allows you to perform redaction, deduplication, and AWS Textract operations directly from the terminal. This is particularly useful for batch processing, automation, and integration with other systems.
+
+### Getting started with the CLI
+
+To use the CLI, you need to:
+
+1. Open a terminal window
+2. Navigate to the app folder containing `cli_redact.py`
+3. Activate your virtual environment (conda or venv)
+4. Run commands using `python cli_redact.py` followed by your options
+
+### Basic CLI syntax
+
+```bash
+python cli_redact.py --task [redact|deduplicate|textract] --input_file [file_path] [additional_options]
+```
+
+### Redaction examples
+
+**Basic PDF redaction with default settings:**
+```bash
+python cli_redact.py --input_file example_data/example_of_emails_sent_to_a_professor_before_applying.pdf
+```
+
+**Extract text only (no redaction) with whole page redaction:**
+```bash
+python cli_redact.py --input_file example_data/Partnership-Agreement-Toolkit_0_0.pdf --redact_whole_page_file example_data/partnership_toolkit_redact_some_pages.csv --pii_detector None
+```
+
+**Redact with custom entities and allow list:**
+```bash
+python cli_redact.py --input_file example_data/graduate-job-example-cover-letter.pdf --allow_list_file example_data/test_allow_list_graduate.csv --local_redact_entities TITLES PERSON DATE_TIME
+```
+
+**Redact with fuzzy matching and custom deny list:**
+```bash
+python cli_redact.py --input_file example_data/Partnership-Agreement-Toolkit_0_0.pdf --deny_list_file example_data/Partnership-Agreement-Toolkit_test_deny_list_para_single_spell.csv --local_redact_entities CUSTOM_FUZZY --page_min 1 --page_max 3 --fuzzy_mistakes 3
+```
+
+**Redact with AWS services:**
+```bash
+python cli_redact.py --input_file example_data/example_of_emails_sent_to_a_professor_before_applying.pdf --ocr_method "AWS Textract" --pii_detector "AWS Comprehend"
+```
+
+**Redact specific pages with signature extraction:**
+```bash
+python cli_redact.py --input_file example_data/Partnership-Agreement-Toolkit_0_0.pdf --page_min 6 --page_max 7 --ocr_method "AWS Textract" --handwrite_signature_extraction "Extract handwriting" "Extract signatures"
+```
+
+### Tabular data redaction
+
+**Anonymize CSV file with specific columns:**
+```bash
+python cli_redact.py --input_file example_data/combined_case_notes.csv --text_columns "Case Note" "Client" --anon_strategy replace_redacted
+```
+
+**Anonymize Excel file:**
+```bash
+python cli_redact.py --input_file example_data/combined_case_notes.xlsx --text_columns "Case Note" "Client" --excel_sheets combined_case_notes --anon_strategy redact
+```
+
+**Anonymize Word document:**
+```bash
+python cli_redact.py --input_file "example_data/Bold minimalist professional cover letter.docx" --anon_strategy replace_redacted
+```
+
+### Duplicate detection
+
+**Find duplicate pages in OCR files:**
+```bash
+python cli_redact.py --task deduplicate --input_file example_data/example_outputs/doubled_output_joined.pdf_ocr_output.csv --duplicate_type pages --similarity_threshold 0.95
+```
+
+**Find duplicates at line level:**
+```bash
+python cli_redact.py --task deduplicate --input_file example_data/example_outputs/doubled_output_joined.pdf_ocr_output.csv --duplicate_type pages --similarity_threshold 0.95 --combine_pages False --min_word_count 3
+```
+
+**Find duplicate rows in tabular data:**
+```bash
+python cli_redact.py --task deduplicate --input_file example_data/Lambeth_2030-Our_Future_Our_Lambeth.pdf.csv --duplicate_type tabular --text_columns "text" --similarity_threshold 0.95
+```
+
+### AWS Textract operations
+
+**Submit document for analysis:**
+```bash
+python cli_redact.py --task textract --textract_action submit --input_file example_data/example_of_emails_sent_to_a_professor_before_applying.pdf
+```
+
+**Submit with signature extraction:**
+```bash
+python cli_redact.py --task textract --textract_action submit --input_file example_data/Partnership-Agreement-Toolkit_0_0.pdf --extract_signatures
+```
+
+**Retrieve results by job ID:**
+```bash
+python cli_redact.py --task textract --textract_action retrieve --job_id 12345678-1234-1234-1234-123456789012
+```
+
+**List recent jobs:**
+```bash
+python cli_redact.py --task textract --textract_action list
+```
+
+### Common CLI options
+
+- `--task`: Choose between "redact", "deduplicate", or "textract"
+- `--input_file`: Path to input file(s)
+- `--output_dir`: Directory for output files (default: output/)
+- `--page_min` / `--page_max`: Process only specific page range
+- `--ocr_method`: Choose text extraction method
+- `--pii_detector`: Choose PII detection method
+- `--local_redact_entities`: Specify local entities to redact
+- `--allow_list_file` / `--deny_list_file`: Custom lists
+- `--redact_whole_page_file`: List of pages to redact completely
+- `--fuzzy_mistakes`: Number of spelling mistakes allowed in fuzzy matching
+- `--similarity_threshold`: Threshold for duplicate detection
+- `--anon_strategy`: Anonymization strategy for tabular data
+
+### Output files
+
+The CLI generates the same output files as the GUI:
+- `...redacted.pdf`: Final redacted document
+- `...redactions_for_review.pdf`: Document with redaction boxes for review
+- `...review_file.csv`: Detailed redaction information
+- `...ocr_results.csv`: Extracted text results
+- `..._textract.json`: AWS Textract results (if applicable)
+
+For more advanced options and configuration, refer to the help text by running:
+```bash
+python cli_redact.py --help
+```

app.py

@@ -22,14 +22,12 @@ from tools.config import (
     CHOSEN_LOCAL_OCR_MODEL,
     CHOSEN_REDACT_ENTITIES,
     COGNITO_AUTH,
-    COMPRESS_REDACTED_PDF,
     CONFIG_FOLDER,
     COST_CODES_PATH,
     CSV_ACCESS_LOG_HEADERS,
     CSV_FEEDBACK_LOG_HEADERS,
     CSV_USAGE_LOG_HEADERS,
     CUSTOM_BOX_COLOUR,
-    DEFAULT_COMBINE_PAGES,
     DEFAULT_CONCURRENCY_LIMIT,
     DEFAULT_COST_CODE,
     DEFAULT_DUPLICATE_DETECTION_THRESHOLD,
@@ -48,11 +46,37 @@ from tools.config import (
     DEFAULT_TEXT_COLUMNS,
     DEFAULT_TEXT_EXTRACTION_MODEL,
     DENY_LIST_PATH,
+    DIRECT_MODE_ANON_STRATEGY,
+    DIRECT_MODE_CHOSEN_LOCAL_OCR_MODEL,
+    DIRECT_MODE_COMBINE_PAGES,
+    DIRECT_MODE_COMPRESS_REDACTED_PDF,
     DIRECT_MODE_DEFAULT_USER,
     DIRECT_MODE_DUPLICATE_TYPE,
+    DIRECT_MODE_EXTRACT_FORMS,
+    DIRECT_MODE_EXTRACT_LAYOUT,
+    DIRECT_MODE_EXTRACT_SIGNATURES,
+    DIRECT_MODE_EXTRACT_TABLES,
+    DIRECT_MODE_FUZZY_MISTAKES,
+    DIRECT_MODE_GREEDY_MATCH,
+    DIRECT_MODE_IMAGES_DPI,
     DIRECT_MODE_INPUT_FILE,
+    DIRECT_MODE_JOB_ID,
+    # Additional direct mode configuration options
+    DIRECT_MODE_LANGUAGE,
+    DIRECT_MODE_MATCH_FUZZY_WHOLE_PHRASE_BOOL,
+    DIRECT_MODE_MIN_CONSECUTIVE_PAGES,
+    DIRECT_MODE_MIN_WORD_COUNT,
+    DIRECT_MODE_OCR_METHOD,
     DIRECT_MODE_OUTPUT_DIR,
+    DIRECT_MODE_PAGE_MAX,
+    DIRECT_MODE_PAGE_MIN,
+    DIRECT_MODE_PII_DETECTOR,
+    DIRECT_MODE_PREPROCESS_LOCAL_OCR_IMAGES,
+    DIRECT_MODE_REMOVE_DUPLICATE_ROWS,
+    DIRECT_MODE_RETURN_PDF_END_OF_REDACTION,
+    DIRECT_MODE_SIMILARITY_THRESHOLD,
     DIRECT_MODE_TASK,
+    DIRECT_MODE_TEXTRACT_ACTION,
     DISPLAY_FILE_NAMES_IN_LOGS,
     DO_INITIAL_TABULAR_DATA_CLEAN,
     DOCUMENT_REDACTION_BUCKET,
@@ -76,11 +100,9 @@ from tools.config import (
     GRADIO_TEMP_DIR,
     HANDWRITE_SIGNATURE_TEXTBOX_FULL_OPTIONS,
     HOST_NAME,
-    IMAGES_DPI,
     INPUT_FOLDER,
     LOAD_PREVIOUS_TEXTRACT_JOBS_S3,
     LOCAL_OCR_MODEL_OPTIONS,
-    LOCAL_PII_OPTION,
     LOG_FILE_NAME,
     MAX_FILE_SIZE,
     MAX_OPEN_TEXT_CHARACTERS,
@@ -91,38 +113,10 @@ from tools.config import (
     OUTPUT_FOLDER,
     PADDLE_MODEL_PATH,
     PII_DETECTION_MODELS,
-    PREPROCESS_LOCAL_OCR_IMAGES,
     REMOVE_DUPLICATE_ROWS,
-    RETURN_REDACTED_PDF,
     ROOT_PATH,
     RUN_AWS_FUNCTIONS,
     RUN_DIRECT_MODE,
-    # Additional direct mode configuration options
-    DIRECT_MODE_LANGUAGE,
-    DIRECT_MODE_PII_DETECTOR,
-    DIRECT_MODE_OCR_METHOD,
-    DIRECT_MODE_PAGE_MIN,
-    DIRECT_MODE_PAGE_MAX,
-    DIRECT_MODE_IMAGES_DPI,
-    DIRECT_MODE_CHOSEN_LOCAL_OCR_MODEL,
-    DIRECT_MODE_PREPROCESS_LOCAL_OCR_IMAGES,
-    DIRECT_MODE_COMPRESS_REDACTED_PDF,
-    DIRECT_MODE_RETURN_PDF_END_OF_REDACTION,
-    DIRECT_MODE_EXTRACT_FORMS,
-    DIRECT_MODE_EXTRACT_TABLES,
-    DIRECT_MODE_EXTRACT_LAYOUT,
-    DIRECT_MODE_EXTRACT_SIGNATURES,
-    DIRECT_MODE_MATCH_FUZZY_WHOLE_PHRASE_BOOL,
-    DIRECT_MODE_ANON_STRATEGY,
-    DIRECT_MODE_FUZZY_MISTAKES,
-    DIRECT_MODE_SIMILARITY_THRESHOLD,
-    DIRECT_MODE_MIN_WORD_COUNT,
-    DIRECT_MODE_MIN_CONSECUTIVE_PAGES,
-    DIRECT_MODE_GREEDY_MATCH,
-    DIRECT_MODE_COMBINE_PAGES,
-    DIRECT_MODE_REMOVE_DUPLICATE_ROWS,
-    DIRECT_MODE_TEXTRACT_ACTION,
-    DIRECT_MODE_JOB_ID,
     RUN_FASTAPI,
     S3_ACCESS_LOGS_FOLDER,
     S3_ALLOW_LIST_PATH,
@@ -141,7 +135,6 @@ from tools.config import (
     SHOW_WHOLE_DOCUMENT_TEXTRACT_CALL_OPTIONS,
     SPACY_MODEL_PATH,
     TABULAR_PII_DETECTION_MODELS,
-    TESSERACT_TEXT_EXTRACT_OPTION,
     TEXT_EXTRACTION_MODELS,
     TEXTRACT_JOBS_LOCAL_LOC,
     TEXTRACT_JOBS_S3_INPUT_LOC,
@@ -1045,7 +1038,7 @@ with blocks:
     with gr.Tab("Redact PDFs/images"):
 
         # Examples for PDF/image redaction
-        if SHOW_EXAMPLES is True:
+        if SHOW_EXAMPLES:
             gr.Markdown(
                 "### Try an example - Click on an example below and then the 'Extract text and redact document' button:"
             )
@@ -1834,7 +1827,7 @@ with blocks:
         )
 
         # Examples for duplicate page detection
-        if SHOW_EXAMPLES == "True":
+        if SHOW_EXAMPLES:
             gr.Markdown(
                 "### Try an example - Click on an example below and then the 'Identify duplicate pages/subdocuments' button:"
             )
@@ -1989,7 +1982,7 @@ with blocks:
         )
 
         # Examples for Word/Excel/csv redaction and tabular duplicate detection
-        if SHOW_EXAMPLES == "True":
+        if SHOW_EXAMPLES:
             gr.Markdown(
                 "### Try an example - Click on an example below and then the 'Redact text/data files' button for redaction, or the 'Find duplicate cells/rows' button for duplicate detection:"
             )
@@ -6578,14 +6571,11 @@ with blocks:
                 "extract_layout": DIRECT_MODE_EXTRACT_LAYOUT,
                 "extract_signatures": DIRECT_MODE_EXTRACT_SIGNATURES,
                 "match_fuzzy_whole_phrase_bool": DIRECT_MODE_MATCH_FUZZY_WHOLE_PHRASE_BOOL,
-                
                 # Word/Tabular Anonymisation Arguments
-                
                 "anon_strategy": DIRECT_MODE_ANON_STRATEGY,
                 "text_columns": DEFAULT_TEXT_COLUMNS,
                 "excel_sheets": DEFAULT_EXCEL_SHEETS,
                 "fuzzy_mistakes": DIRECT_MODE_FUZZY_MISTAKES,
-                
                 # Duplicate Detection Arguments
                 "duplicate_type": DIRECT_MODE_DUPLICATE_TYPE,
                 "similarity_threshold": DIRECT_MODE_SIMILARITY_THRESHOLD,
@@ -6594,10 +6584,9 @@ with blocks:
                 "greedy_match": DIRECT_MODE_GREEDY_MATCH,
                 "combine_pages": DIRECT_MODE_COMBINE_PAGES,
                 "remove_duplicate_rows": DIRECT_MODE_REMOVE_DUPLICATE_ROWS,
-                
                 # Textract Batch Operations Arguments
                 "textract_action": DIRECT_MODE_TEXTRACT_ACTION,
-                "job_id": DIRECT_MODE_JOB_ID,      
+                "job_id": DIRECT_MODE_JOB_ID,
                 "textract_bucket": TEXTRACT_WHOLE_DOCUMENT_ANALYSIS_BUCKET,
                 "textract_input_prefix": TEXTRACT_WHOLE_DOCUMENT_ANALYSIS_INPUT_SUBFOLDER,
                 "textract_output_prefix": TEXTRACT_WHOLE_DOCUMENT_ANALYSIS_OUTPUT_SUBFOLDER,

lambda_entrypoint.py

@@ -15,11 +15,11 @@ from tools.config import (
     DEFAULT_PAGE_MAX,
     DEFAULT_PAGE_MIN,
     IMAGES_DPI,
-    LAMBDA_POLL_INTERVAL,
+    LAMBDA_DEFAULT_USERNAME,
+    LAMBDA_EXTRACT_SIGNATURES,
     LAMBDA_MAX_POLL_ATTEMPTS,
+    LAMBDA_POLL_INTERVAL,
     LAMBDA_PREPARE_IMAGES,
-    LAMBDA_EXTRACT_SIGNATURES,
-    LAMBDA_DEFAULT_USERNAME,
 )
 
 
@@ -532,7 +532,9 @@ def lambda_handler(event, context):
             os.getenv("TEXTRACT_JOBS_LOCAL_LOC", ""),
         ),
         "poll_interval": int(arguments.get("poll_interval", LAMBDA_POLL_INTERVAL)),
-        "max_poll_attempts": int(arguments.get("max_poll_attempts", LAMBDA_MAX_POLL_ATTEMPTS)),
+        "max_poll_attempts": int(
+            arguments.get("max_poll_attempts", LAMBDA_MAX_POLL_ATTEMPTS)
+        ),
         # Additional arguments that were missing
         "search_query": arguments.get(
             "search_query", os.getenv("DEFAULT_SEARCH_QUERY", "")

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "doc_redaction"
-version = "1.4.0"
+version = "1.4.1"
 description = "Redact PDF/image-based documents, Word, or CSV/XLSX files using a Gradio-based GUI interface"
 readme = "README.md"
 requires-python = ">=3.10"

src/app_settings.qmd

@@ -2,529 +2,573 @@
 title: "App settings management guide"
 format:
   html:
-    toc: true # Enable the table of contents
-    toc-depth: 3 # Include headings up to level 2 (##)
-    toc-title: "On this page" # Optional: Title for your TOC
+    toc: true 
+    toc-depth: 3
+    toc-title: "On this page"
 ---
 
-Settings for the redaction app can be set from outside by changing values in the  `config.env` file stored in your local config folder, or in S3 if running on AWS. This guide provides an overview of how to configure the application using environment variables. The application loads configurations using `os.environ.get()`. It first attempts to load variables from the file specified by `APP_CONFIG_PATH` (which defaults to `config/app_config.env`). If `AWS_CONFIG_PATH` is also set (e.g., to `config/aws_config.env`), variables are loaded from that file as well. Environment variables set directly in the system will always take precedence over those defined in these `.env` files.
+Settings for the redaction app can be set from outside by changing values in the `.env` file stored in your local config folder, or in S3 if running on AWS. This guide provides an overview of how to configure the application using environment variables. The application loads configurations using `os.environ.get()`. It first attempts to load variables from the file specified by `APP_CONFIG_PATH` (which defaults to `config/app_config.env`). If `AWS_CONFIG_PATH` is also set (e.g., to `config/aws_config.env`), variables are loaded from that file as well. Environment variables set directly in the system will always take precedence over those defined in these `.env` files.
 
-## App Configuration File (config.env)
+## App Configuration File (`config.env`)
 
 This section details variables related to the main application configuration file.
 
-*   **`APP_CONFIG_PATH`**
-    *   **Description:** Specifies the path to the application configuration `.env` file. This file contains various settings that control the application's behavior.
-    *   **Default Value:** `config/app_config.env`
-    *   **Configuration:** Set as an environment variable directly. This variable defines where to load other application configurations, so it cannot be set within `config/app_config.env` itself.
+* **`CONFIG_FOLDER`**
+    * **Description:** The folder where configuration files are stored.
+    * **Default Value:** `config/`
+
+* **`APP_CONFIG_PATH`**
+    * **Description:** Specifies the path to the application configuration `.env` file. This file contains various settings that control the application's behavior.
+    * **Default Value:** `config/app_config.env`
 
 ## AWS Options
 
 This section covers configurations related to AWS services used by the application.
 
-*   **`AWS_CONFIG_PATH`**
-    *   **Description:** Specifies the path to the AWS configuration `.env` file. This file is intended to store AWS credentials and specific settings.
-    *   **Default Value:** `''` (empty string)
-    *   **Configuration:** Set as an environment variable directly. This variable defines an additional source for AWS-specific configurations.
-
-*   **`RUN_AWS_FUNCTIONS`**
-    *   **Description:** Enables or disables AWS-specific functionalities within the application. Set to `"True"` to enable and `"False"` to disable.
-    *   **Default Value:** `"False"`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env` (or `config/aws_config.env` if `AWS_CONFIG_PATH` is configured).
-
-*   **`AWS_REGION`**
-    *   **Description:** Defines the AWS region where services like S3, Cognito, and Textract are located.
-    *   **Default Value:** `''`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/aws_config.env` (if `AWS_CONFIG_PATH` is configured).
-
-*   **`AWS_CLIENT_ID`**
-    *   **Description:** The client ID for AWS Cognito, used for user authentication.
-    *   **Default Value:** `''`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/aws_config.env` (if `AWS_CONFIG_PATH` is configured).
-
-*   **`AWS_CLIENT_SECRET`**
-    *   **Description:** The client secret for AWS Cognito, used in conjunction with the client ID for authentication.
-    *   **Default Value:** `''`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/aws_config.env` (if `AWS_CONFIG_PATH` is configured).
-
-*   **`AWS_USER_POOL_ID`**
-    *   **Description:** The user pool ID for AWS Cognito, identifying the user directory.
-    *   **Default Value:** `''`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/aws_config.env` (if `AWS_CONFIG_PATH` is configured).
-
-*   **`AWS_ACCESS_KEY`**
-    *   **Description:** The AWS access key ID for programmatic access to AWS services.
-    *   **Default Value:** `''` (Note: Often found in the environment or AWS credentials file.)
-    *   **Configuration:** Set as an environment variable directly, or include in `config/aws_config.env` (if `AWS_CONFIG_PATH` is configured). It's also commonly configured via shared AWS credentials files or IAM roles.
-
-*   **`AWS_SECRET_KEY`**
-    *   **Description:** The AWS secret access key corresponding to the AWS access key ID.
-    *   **Default Value:** `''` (Note: Often found in the environment or AWS credentials file.)
-    *   **Configuration:** Set as an environment variable directly, or include in `config/aws_config.env` (if `AWS_CONFIG_PATH` is configured). It's also commonly configured via shared AWS credentials files or IAM roles.
-
-*   **`DOCUMENT_REDACTION_BUCKET`**
-    *   **Description:** The name of the S3 bucket used for storing documents related to the redaction process.
-    *   **Default Value:** `''`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/aws_config.env` (if `AWS_CONFIG_PATH` is configured).
-
-*   **`CUSTOM_HEADER`**
-    *   **Description:** Specifies a custom header name to be included in requests, often used for services like AWS CloudFront.
-    *   **Default Value:** `''`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env` (or `config/aws_config.env` if `AWS_CONFIG_PATH` is configured).
-
-*   **`CUSTOM_HEADER_VALUE`**
-    *   **Description:** The value for the custom header specified by `CUSTOM_HEADER`.
-    *   **Default Value:** `''`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env` (or `config/aws_config.env` if `AWS_CONFIG_PATH` is configured).
+* **`AWS_CONFIG_PATH`**
+    * **Description:** Specifies the path to the AWS configuration `.env` file. This file is intended to store AWS credentials and specific settings.
+    * **Default Value:** `''` (empty string)
+
+* **`RUN_AWS_FUNCTIONS`**
+    * **Description:** Enables or disables AWS-specific functionalities within the application. Set to `"True"` to enable.
+    * **Default Value:** `"False"`
+
+* **`AWS_REGION`**
+    * **Description:** Defines the AWS region where services like S3, Cognito, and Textract are located.
+    * **Default Value:** `''`
+
+* **`AWS_CLIENT_ID`**
+    * **Description:** The client ID for AWS Cognito, used for user authentication.
+    * **Default Value:** `''`
+
+* **`AWS_CLIENT_SECRET`**
+    * **Description:** The client secret for AWS Cognito, used in conjunction with the client ID for authentication.
+    * **Default Value:** `''`
+
+* **`AWS_USER_POOL_ID`**
+    * **Description:** The user pool ID for AWS Cognito, identifying the user directory.
+    * **Default Value:** `''`
+
+* **`AWS_ACCESS_KEY`**
+    * **Description:** The AWS access key ID for programmatic access to AWS services.
+    * **Default Value:** `''`
+
+* **`AWS_SECRET_KEY`**
+    * **Description:** The AWS secret access key corresponding to the AWS access key ID.
+    * **Default Value:** `''`
+
+* **`DOCUMENT_REDACTION_BUCKET`**
+    * **Description:** The name of the S3 bucket used for storing documents related to the redaction process.
+    * **Default Value:** `''`
+
+* **`PRIORITISE_SSO_OVER_AWS_ENV_ACCESS_KEYS`**
+    * **Description:** If set to `"True"`, the app will prioritize using AWS SSO credentials over access keys stored in environment variables.
+    * **Default Value:** `"True"`
+
+* **`CUSTOM_HEADER`**
+    * **Description:** Specifies a custom header name to be included in requests, often used for services like AWS CloudFront.
+    * **Default Value:** `''`
+
+* **`CUSTOM_HEADER_VALUE`**
+    * **Description:** The value for the custom header specified by `CUSTOM_HEADER`.
+    * **Default Value:** `''`
 
 ## Image Options
 
 Settings related to image processing within the application.
 
-*   **`IMAGES_DPI`**
-    *   **Description:** Dots Per Inch (DPI) setting for image processing, affecting the resolution and quality of processed images.
-    *   **Default Value:** `'300.0'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
+* **`IMAGES_DPI`**
+    * **Description:** Dots Per Inch (DPI) setting for image processing, affecting the resolution and quality of processed images.
+    * **Default Value:** `'300.0'`
 
-*   **`LOAD_TRUNCATED_IMAGES`**
-    *   **Description:** Controls whether the application attempts to load truncated images. Set to `'True'` to enable.
-    *   **Default Value:** `'True'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
+* **`LOAD_TRUNCATED_IMAGES`**
+    * **Description:** Controls whether the application attempts to load truncated images. Set to `'True'` to enable.
+    * **Default Value:** `'True'`
 
-*   **`MAX_IMAGE_PIXELS`**
-    *   **Description:** Sets the maximum number of pixels for an image that the application will process. Leave blank for no limit. This can help prevent issues with very large images.
-    *   **Default Value:** `''`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
+* **`MAX_IMAGE_PIXELS`**
+    * **Description:** Sets the maximum number of pixels for an image that the application will process. Leave blank for no limit. This can help prevent issues with very large images.
+    * **Default Value:** `''`
 
 ## File I/O Options
 
 Configuration for input and output file handling.
 
-*   **`SESSION_OUTPUT_FOLDER`**
-    *   **Description:** If set to `'True'`, the application will save output and input files into session-specific subfolders, helping to organise files from different user sessions.
-    *   **Default Value:** `'False'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
+* **`SESSION_OUTPUT_FOLDER`**
+    * **Description:** If set to `'True'`, the application will save output and input files into session-specific subfolders.
+    * **Default Value:** `'False'`
 
-*   **`GRADIO_OUTPUT_FOLDER`** (aliased as `OUTPUT_FOLDER`)
-    *   **Description:** Specifies the default output folder for files generated by Gradio components. Can be set to "TEMP" to use a temporary directory.
-    *   **Default Value:** `'output/'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
+* **`OUTPUT_FOLDER`**
+    * **Description:** Specifies the default output folder for generated files. Can be set to `"TEMP"` to use a temporary directory.
+    * **Default Value:** `'output/'`
 
-*   **`GRADIO_INPUT_FOLDER`** (aliased as `INPUT_FOLDER`)
-    *   **Description:** Specifies the default input folder for files used by Gradio components. Can be set to "TEMP" to use a temporary directory.
-    *   **Default Value:** `'input/'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
+* **`INPUT_FOLDER`**
+    * **Description:** Specifies the default input folder for files. Can be set to `"TEMP"` to use a temporary directory.
+    * **Default Value:** `'input/'`
 
-*   **`GRADIO_TEMP_DIR`**
-    *   **Description:** Defines the path for Gradio's temporary file storage.
-    *   **Default Value:** `'tmp/gradio_tmp/'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
+* **`GRADIO_TEMP_DIR`**
+    * **Description:** Defines the path for Gradio's temporary file storage.
+    * **Default Value:** `''`
 
-*   **`MPLCONFIGDIR`**
-    *   **Description:** Specifies the cache directory for the Matplotlib library, which is used for plotting and image handling.
-    *   **Default Value:** `'tmp/matplotlib_cache/'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
+* **`MPLCONFIGDIR`**
+    * **Description:** Specifies the cache directory for the Matplotlib library.
+    * **Default Value:** `''`
 
 ## Logging Options
 
-Settings for configuring application logging, including log formats and storage locations.
-
-*   **`SAVE_LOGS_TO_CSV`**
-    *   **Description:** Enables or disables saving logs to CSV files. Set to `'True'` to enable.
-    *   **Default Value:** `'True'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`USE_LOG_SUBFOLDERS`**
-    *   **Description:** If enabled (`'True'`), logs will be stored in subfolders based on date and hostname, aiding in log organisation.
-    *   **Default Value:** `'True'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`FEEDBACK_LOGS_FOLDER`**
-    *   **Description:** Specifies the base folder for storing feedback logs. If `USE_LOG_SUBFOLDERS` is true, date/hostname subfolders will be created within this folder.
-    *   **Default Value:** `'feedback/'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`ACCESS_LOGS_FOLDER`**
-    *   **Description:** Specifies the base folder for storing access logs. If `USE_LOG_SUBFOLDERS` is true, date/hostname subfolders will be created within this folder.
-    *   **Default Value:** `'logs/'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`USAGE_LOGS_FOLDER`**
-    *   **Description:** Specifies the base folder for storing usage logs. If `USE_LOG_SUBFOLDERS` is true, date/hostname subfolders will be created within this folder.
-    *   **Default Value:** `'usage/'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`DISPLAY_FILE_NAMES_IN_LOGS`**
-    *   **Description:** If set to `'True'`, file names will be included in the log entries.
-    *   **Default Value:** `'False'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`CSV_ACCESS_LOG_HEADERS`**
-    *   **Description:** Defines custom headers for CSV access logs. If left blank, component labels will be used as headers.
-    *   **Default Value:** `''`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`CSV_FEEDBACK_LOG_HEADERS`**
-    *   **Description:** Defines custom headers for CSV feedback logs. If left blank, component labels will be used as headers.
-    *   **Default Value:** `''`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`CSV_USAGE_LOG_HEADERS`**
-    *   **Description:** Defines custom headers for CSV usage logs.
-    *   **Default Value:** A predefined list of header names. Refer to `config.py` for the complete list.
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`SAVE_LOGS_TO_DYNAMODB`**
-    *   **Description:** Enables or disables saving logs to AWS DynamoDB. Set to `'True'` to enable. Requires appropriate AWS setup.
-    *   **Default Value:** `'False'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env` (or `config/aws_config.env` if `AWS_CONFIG_PATH` is configured).
-
-*   **`ACCESS_LOG_DYNAMODB_TABLE_NAME`**
-    *   **Description:** The name of the DynamoDB table used for storing access logs.
-    *   **Default Value:** `'redaction_access_log'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env` (or `config/aws_config.env` if `AWS_CONFIG_PATH` is configured).
-
-*   **`DYNAMODB_ACCESS_LOG_HEADERS`**
-    *   **Description:** Specifies the headers (attributes) for the DynamoDB access log table.
-    *   **Default Value:** `''`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env` (or `config/aws_config.env` if `AWS_CONFIG_PATH` is configured).
-
-*   **`FEEDBACK_LOG_DYNAMODB_TABLE_NAME`**
-    *   **Description:** The name of the DynamoDB table used for storing feedback logs.
-    *   **Default Value:** `'redaction_feedback'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env` (or `config/aws_config.env` if `AWS_CONFIG_PATH` is configured).
-
-*   **`DYNAMODB_FEEDBACK_LOG_HEADERS`**
-    *   **Description:** Specifies the headers (attributes) for the DynamoDB feedback log table.
-    *   **Default Value:** `''`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env` (or `config/aws_config.env` if `AWS_CONFIG_PATH` is configured).
-
-*   **`USAGE_LOG_DYNAMODB_TABLE_NAME`**
-    *   **Description:** The name of the DynamoDB table used for storing usage logs.
-    *   **Default Value:** `'redaction_usage'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env` (or `config/aws_config.env` if `AWS_CONFIG_PATH` is configured).
-
-*   **`DYNAMODB_USAGE_LOG_HEADERS`**
-    *   **Description:** Specifies the headers (attributes) for the DynamoDB usage log table.
-    *   **Default Value:** `''`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env` (or `config/aws_config.env` if `AWS_CONFIG_PATH` is configured).
-
-*   **`LOGGING`**
-    *   **Description:** Enables or disables general console logging. Set to `'True'` to enable.
-    *   **Default Value:** `'False'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`LOG_FILE_NAME`**
-    *   **Description:** Specifies the name for the CSV log file if `SAVE_LOGS_TO_CSV` is enabled.
-    *   **Default Value:** `'log.csv'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-## Redaction Options
-
-Configurations related to the text redaction process, including PII detection models and external tool paths.
-
-*   **`TESSERACT_FOLDER`**
-    *   **Description:** Path to the local Tesseract OCR installation folder. Only required if Tesseract is not in the system's PATH, or when running a packaged executable (e.g., via PyInstaller).
-    *   **Default Value:** `""` (empty string)
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`TESSERACT_DATA_FOLDER`**
-    *   **Description:** Path to the Tesseract trained data files (e.g., `tessdata`).
-    *   **Default Value:** `"/usr/share/tessdata"`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`POPPLER_FOLDER`**
-    *   **Description:** Path to the local Poppler installation's `bin` folder. Poppler is used for PDF processing. Only required if Poppler is not in the system's PATH.
-    *   **Default Value:** `""` (empty string)
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`SELECTABLE_TEXT_EXTRACT_OPTION`**
-    *   **Description:** Display name in the UI for the text extraction method that processes selectable text directly from PDFs.
-    *   **Default Value:** `"Local model - selectable text"`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`TESSERACT_TEXT_EXTRACT_OPTION`**
-    *   **Description:** Display name in the UI for the text extraction method using local Tesseract OCR (for PDFs without selectable text).
-    *   **Default Value:** `"Local OCR model - PDFs without selectable text"`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`TEXTRACT_TEXT_EXTRACT_OPTION`**
-    *   **Description:** Display name in the UI for the text extraction method using AWS Textract service.
-    *   **Default Value:** `"AWS Textract service - all PDF types"`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`NO_REDACTION_PII_OPTION`**
-    *   **Description:** Display name in the UI for the option to only extract text without performing any PII detection or redaction.
-    *   **Default Value:** `"Only extract text (no redaction)"`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`LOCAL_PII_OPTION`**
-    *   **Description:** Display name in the UI for the PII detection method using a local model.
-    *   **Default Value:** `"Local"`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`AWS_PII_OPTION`**
-    *   **Description:** Display name in the UI for the PII detection method using AWS Comprehend.
-    *   **Default Value:** `"AWS Comprehend"`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`SHOW_LOCAL_TEXT_EXTRACTION_OPTIONS`**
-    *   **Description:** Controls whether local text extraction options (selectable text, Tesseract) are shown in the UI. Set to `'True'` to show.
-    *   **Default Value:** `'True'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`SHOW_AWS_TEXT_EXTRACTION_OPTIONS`**
-    *   **Description:** Controls whether AWS Textract text extraction option is shown in the UI. Set to `'True'` to show.
-    *   **Default Value:** `'True'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`DEFAULT_TEXT_EXTRACTION_MODEL`**
-    *   **Description:** Sets the default text extraction model selected in the UI. Defaults to `TEXTRACT_TEXT_EXTRACT_OPTION` if AWS options are shown; otherwise, defaults to `SELECTABLE_TEXT_EXTRACT_OPTION`.
-    *   **Default Value:** Value of `TEXTRACT_TEXT_EXTRACT_OPTION` if `SHOW_AWS_TEXT_EXTRACTION_OPTIONS` is True, else value of `SELECTABLE_TEXT_EXTRACT_OPTION`.
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`. Provide one of the text extraction option display names.
-
-*   **`SHOW_LOCAL_PII_DETECTION_OPTIONS`**
-    *   **Description:** Controls whether the local PII detection option is shown in the UI. Set to `'True'` to show.
-    *   **Default Value:** `'True'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`SHOW_AWS_PII_DETECTION_OPTIONS`**
-    *   **Description:** Controls whether the AWS Comprehend PII detection option is shown in the UI. Set to `'True'` to show.
-    *   **Default Value:** `'True'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`DEFAULT_PII_DETECTION_MODEL`**
-    *   **Description:** Sets the default PII detection model selected in the UI. Defaults to `AWS_PII_OPTION` if AWS options are shown; otherwise, defaults to `LOCAL_PII_OPTION`.
-    *   **Default Value:** Value of `AWS_PII_OPTION` if `SHOW_AWS_PII_DETECTION_OPTIONS` is True, else value of `LOCAL_PII_OPTION`.
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`. Provide one of the PII detection option display names.
-
-*   **`CHOSEN_LOCAL_OCR_MODEL`**
-    *   **Description:** Choose the engine for local OCR: `"tesseract"`, `"paddle"`, or `"hybrid"`. "paddle" is effective for line extraction but not word-level redaction. "hybrid" uses Tesseract first, then PaddleOCR for low-confidence words.
-    *   **Default Value:** `"tesseract"`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`PREPROCESS_LOCAL_OCR_IMAGES`**
-    *   **Description:** If set to `"True"`, images will be preprocessed (e.g., deskewed, contrast adjusted) before being sent to the local OCR engine. This can sometimes yield worse results on clean scans.
-    *   **Default Value:** `"False"`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`CHOSEN_COMPREHEND_ENTITIES`**
-    *   **Description:** A list of AWS Comprehend PII entity types to be redacted when using AWS Comprehend.
-    *   **Default Value:** A predefined list of entity types. Refer to `config.py` for the complete list.
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`. This should be a string representation of a Python list.
-
-*   **`FULL_COMPREHEND_ENTITY_LIST`**
-    *   **Description:** The complete list of PII entity types supported by AWS Comprehend that can be selected for redaction.
-    *   **Default Value:** A predefined list of entity types. Refer to `config.py` for the complete list.
-    *   **Configuration:** This is typically an informational variable reflecting the capabilities of AWS Comprehend and is not meant to be changed by users directly affecting redaction behavior (use `CHOSEN_COMPREHEND_ENTITIES` for that). Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`CHOSEN_REDACT_ENTITIES`**
-    *   **Description:** A list of local PII entity types to be redacted when using the local PII detection model.
-    *   **Default Value:** A predefined list of entity types. Refer to `config.py` for the complete list.
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`. This should be a string representation of a Python list.
-
-*   **`FULL_ENTITY_LIST`**
-    *   **Description:** The complete list of PII entity types supported by the local PII detection model that can be selected for redaction.
-    *   **Default Value:** A predefined list of entity types. Refer to `config.py` for the complete list.
-    *   **Configuration:** This is typically an informational variable reflecting the capabilities of the local model and is not meant to be changed by users directly affecting redaction behavior (use `CHOSEN_REDACT_ENTITIES` for that). Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`PAGE_BREAK_VALUE`**
-    *   **Description:** Defines a page count after which a function might restart. (Note: Currently not activated).
-    *   **Default Value:** `'99999'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`MAX_TIME_VALUE`**
-    *   **Description:** Specifies a maximum time value for long-running processes.
-    *   **Default Value:** `'999999'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`CUSTOM_BOX_COLOUR`**
-    *   **Description:** Allows specifying a custom color for the redaction boxes drawn on documents. Only `"grey"` is currently supported as a custom value. If empty, a default color is used.
-    *   **Default Value:** `""` (empty string)
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`RETURN_PDF_END_OF_REDACTION`**
-    *   **Description:** If set to `'True'`, the application will return a PDF document at the end of the redaction task.
-    *   **Default Value:** `"True"`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`COMPRESS_REDACTED_PDF`**
-    *   **Description:** If set to `'True'`, the redacted PDF output will be compressed. This can reduce file size but may cause issues on systems with low memory.
-    *   **Default Value:** `"False"`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
+Settings for configuring application logging.
+
+* **`SAVE_LOGS_TO_CSV`**
+    * **Description:** Enables or disables saving logs to CSV files. Set to `'True'` to enable.
+    * **Default Value:** `'True'`
+
+* **`USE_LOG_SUBFOLDERS`**
+    * **Description:** If enabled (`'True'`), logs will be stored in subfolders based on date and hostname.
+    * **Default Value:** `'True'`
+
+* **`FEEDBACK_LOGS_FOLDER`**, **`ACCESS_LOGS_FOLDER`**, **`USAGE_LOGS_FOLDER`**
+    * **Description:** Base folders for feedback, access, and usage logs respectively.
+    * **Default Values:** `'feedback/'`, `'logs/'`, `'usage/'`
+
+* **`S3_FEEDBACK_LOGS_FOLDER`**, **`S3_ACCESS_LOGS_FOLDER`**, **`S3_USAGE_LOGS_FOLDER`**
+    * **Description:** S3 paths where feedback, access, and usage logs will be stored if `RUN_AWS_FUNCTIONS` is enabled.
+    * **Default Values:** Dynamically generated based on date and hostname, e.g., `'feedback/YYYYMMDD/hostname/'`.
+
+* **`LOG_FILE_NAME`**, **`USAGE_LOG_FILE_NAME`**, **`FEEDBACK_LOG_FILE_NAME`**
+    * **Description:** Specifies the name for log files. `USAGE_LOG_FILE_NAME` and `FEEDBACK_LOG_FILE_NAME` default to the value of `LOG_FILE_NAME`.
+    * **Default Value:** `'log.csv'`
+
+* **`DISPLAY_FILE_NAMES_IN_LOGS`**
+    * **Description:** If set to `'True'`, file names will be included in log entries.
+    * **Default Value:** `'False'`
+
+* **`CSV_ACCESS_LOG_HEADERS`**, **`CSV_FEEDBACK_LOG_HEADERS`**, **`CSV_USAGE_LOG_HEADERS`**
+    * **Description:** Defines custom headers for the respective CSV logs as a string representation of a list. If blank, component labels are used.
+    * **Default Value:** Varies; see script for `CSV_USAGE_LOG_HEADERS` default.
+
+* **`SAVE_LOGS_TO_DYNAMODB`**
+    * **Description:** Enables or disables saving logs to AWS DynamoDB. Set to `'True'` to enable.
+    * **Default Value:** `'False'`
+
+* **`ACCESS_LOG_DYNAMODB_TABLE_NAME`**, **`FEEDBACK_LOG_DYNAMODB_TABLE_NAME`**, **`USAGE_LOG_DYNAMODB_TABLE_NAME`**
+    * **Description:** Names of the DynamoDB tables for storing access, feedback, and usage logs.
+    * **Default Values:** `'redaction_access_log'`, `'redaction_feedback'`, `'redaction_usage'`
+
+* **`DYNAMODB_ACCESS_LOG_HEADERS`**, **`DYNAMODB_FEEDBACK_LOG_HEADERS`**, **`DYNAMODB_USAGE_LOG_HEADERS`**
+    * **Description:** Specifies the headers (attributes) for the respective DynamoDB log tables.
+    * **Default Value:** `''`
+
+* **`LOGGING`**
+    * **Description:** Enables or disables general console logging. Set to `'True'` to enable.
+    * **Default Value:** `'False'`
+
+## Gradio & General App Options
+
+Configurations for the Gradio UI, server behavior, and application limits.
+
+* **`FAVICON_PATH`**
+    * **Description:** Path to the favicon icon file for the web interface.
+    * **Default Value:** `"favicon.png"`
+
+* **`RUN_FASTAPI`**
+    * **Description:** If set to `"True"`, the application will be served via FastAPI, allowing for API endpoint integration.
+    * **Default Value:** `"False"`
+
+* **`GRADIO_SERVER_NAME`**
+    * **Description:** The IP address the Gradio server will bind to. Use `"0.0.0.0"` to allow external access.
+    * **Default Value:** `"0.0.0.0"`
+
+* **`GRADIO_SERVER_PORT`**
+    * **Description:** The network port on which the Gradio server will listen.
+    * **Default Value:** `7860`
+
+* **`ALLOWED_ORIGINS`**
+    * **Description:** A comma-separated list of allowed origins for Cross-Origin Resource Sharing (CORS).
+    * **Default Value:** `''`
+
+* **`ALLOWED_HOSTS`**
+    * **Description:** A comma-separated list of allowed hostnames.
+    * **Default Value:** `''`
+
+* **`ROOT_PATH`**
+    * **Description:** The root path for the application, useful if running behind a reverse proxy (e.g., `/app`).
+    * **Default Value:** `''`
+
+* **`FASTAPI_ROOT_PATH`**
+    * **Description:** The root path for the FastAPI application, used when `RUN_FASTAPI` is true.
+    * **Default Value:** `"/"`
+
+* **`MAX_QUEUE_SIZE`**
+    * **Description:** The maximum number of requests that can be queued in the Gradio interface.
+    * **Default Value:** `5`
+
+* **`MAX_FILE_SIZE`**
+    * **Description:** Maximum file size allowed for uploads (e.g., "250mb", "1gb").
+    * **Default Value:** `'250mb'`
+
+* **`DEFAULT_CONCURRENCY_LIMIT`**
+    * **Description:** The default concurrency limit for Gradio event handlers, controlling how many requests can be processed simultaneously.
+    * **Default Value:** `3`
+
+* **`MAX_SIMULTANEOUS_FILES`**
+    * **Description:** The maximum number of files that can be processed at once.
+    * **Default Value:** `10`
+
+* **`MAX_DOC_PAGES`**
+    * **Description:** The maximum number of pages a document can have.
+    * **Default Value:** `3000`
+
+* **`MAX_TABLE_ROWS`** / **`MAX_TABLE_COLUMNS`**
+    * **Description:** Maximum number of rows and columns for tabular data processing.
+    * **Default Values:** `250000` / `100`
+
+* **`MAX_OPEN_TEXT_CHARACTERS`**
+    * **Description:** Maximum number of characters for open text input.
+    * **Default Value:** `50000`
+
+* **`TLDEXTRACT_CACHE`**
+    * **Description:** Path to the cache directory used by the `tldextract` library.
+    * **Default Value:** `'tmp/tld/'`
+
+* **`COGNITO_AUTH`**
+    * **Description:** Enables or disables AWS Cognito authentication. Set to `'True'` to enable.
+    * **Default Value:** `'False'`
+
+* **`USER_GUIDE_URL`**
+    * **Description:** A safe URL pointing to the user guide. The URL is validated against a list of allowed domains.
+    * **Default Value:** `"https://seanpedrick-case.github.io/doc_redaction"`
+
+* **`SHOW_EXAMPLES`**
+    * **Description:** If set to `"True"`, displays example files in the Gradio interface.
+    * **Default Value:** `"True"`
+
+* **`SHOW_AWS_EXAMPLES`**
+    * **Description:** If set to `"True"`, includes AWS-specific examples.
+    * **Default Value:** `"False"`
+
+* **`FILE_INPUT_HEIGHT`**
+    * **Description:** Sets the height (in pixels) of the file input component in the Gradio UI.
+    * **Default Value:** `200`
+
+## Redaction & PII Options
+
+Configurations related to text extraction, PII detection, and the redaction process.
+
+### UI and Model Selection
+
+* **`EXTRACTION_AND_PII_OPTIONS_OPEN_BY_DEFAULT`**
+    * **Description:** If set to `"True"`, the "Extraction and PII Options" accordion in the UI will be open by default.
+    * **Default Value:** `"True"`
+
+* **`SHOW_LOCAL_TEXT_EXTRACTION_OPTIONS`** / **`SHOW_AWS_TEXT_EXTRACTION_OPTIONS`**
+    * **Description:** Controls whether local (Tesseract) or AWS (Textract) text extraction options are shown in the UI.
+    * **Default Value:** `"True"` for both.
+
+* **`SHOW_LOCAL_PII_DETECTION_OPTIONS`** / **`SHOW_AWS_PII_DETECTION_OPTIONS`**
+    * **Description:** Controls whether local or AWS (Comprehend) PII detection options are shown in the UI.
+    * **Default Value:** `"True"` for both.
+
+* **`DEFAULT_TEXT_EXTRACTION_MODEL`**
+    * **Description:** Sets the default text extraction model selected in the UI.
+    * **Default Value:** Defaults to AWS Textract if available, otherwise local selectable text.
+
+* **`DEFAULT_PII_DETECTION_MODEL`**
+    * **Description:** Sets the default PII detection model selected in the UI.
+    * **Default Value:** Defaults to AWS Comprehend if available, otherwise the local model.
+
+* **`LOAD_REDACTION_ANNOTATIONS_FROM_PDF`**
+    * **Description:** If set to `"True"`, the application will load existing redaction annotations from PDFs during the review step.
+    * **Default Value:** `"True"`
+
+### External Tool Paths
+
+* **`TESSERACT_FOLDER`**
+    * **Description:** Path to the local Tesseract OCR installation folder.
+    * **Default Value:** `''`
+
+* **`TESSERACT_DATA_FOLDER`**
+    * **Description:** Path to the Tesseract trained data files (`tessdata`).
+    * **Default Value:** `"/usr/share/tessdata"`
+
+* **`POPPLER_FOLDER`**
+    * **Description:** Path to the local Poppler installation's `bin` folder.
+    * **Default Value:** `''`
+
+* **`PADDLE_MODEL_PATH`** / **`SPACY_MODEL_PATH`**
+    * **Description:** Custom directory for PaddleOCR and spaCy model storage, useful for environments like AWS Lambda.
+    * **Default Value:** `''` (uses default location).
+
+### Local OCR (Tesseract & PaddleOCR)
+
+* **`CHOSEN_LOCAL_OCR_MODEL`**
+    * **Description:** Choose the engine for local OCR: `"tesseract"`, `"paddle"`, or `"hybrid"`.
+    * **Default Value:** `"tesseract"`
+
+* **`SHOW_LOCAL_OCR_MODEL_OPTIONS`**
+    * **Description:** If set to `"True"`, allows the user to select the local OCR model from the UI.
+    * **Default Value:** `"False"`
+
+* **`HYBRID_OCR_CONFIDENCE_THRESHOLD`**
+    * **Description:** In "hybrid" mode, this is the Tesseract confidence score below which PaddleOCR will be used for re-extraction.
+    * **Default Value:** `65`
+
+* **`HYBRID_OCR_PADDING`**
+    * **Description:** In "hybrid" mode, padding added to the word's bounding box before re-extraction.
+    * **Default Value:** `1`
+
+* **`PADDLE_USE_TEXTLINE_ORIENTATION`**
+    * **Description:** Toggles textline orientation detection for PaddleOCR.
+    * **Default Value:** `"False"`
+
+* **`PADDLE_DET_DB_UNCLIP_RATIO`**
+    * **Description:** Controls the expansion ratio of the detected text region in PaddleOCR.
+    * **Default Value:** `1.2`
+
+* **`SAVE_EXAMPLE_TESSERACT_VS_PADDLE_IMAGES`**
+    * **Description:** Saves comparison images when using "hybrid" OCR mode.
+    * **Default Value:** `"False"`
+
+* **`SAVE_PADDLE_VISUALISATIONS`**
+    * **Description:** Saves images with PaddleOCR's detected bounding boxes overlaid.
+    * **Default Value:** `"False"`
+
+* **`PREPROCESS_LOCAL_OCR_IMAGES`**
+    * **Description:** If set to `"True"`, images will be preprocessed before local OCR. Can slow down processing.
+    * **Default Value:** `"False"`
+
+### Entity and Search Options
+
+* **`CHOSEN_COMPREHEND_ENTITIES`** / **`FULL_COMPREHEND_ENTITY_LIST`**
+    * **Description:** The selected and available PII entity types for AWS Comprehend.
+    * **Default Value:** Predefined lists of entities (see script).
+
+* **`CHOSEN_REDACT_ENTITIES`** / **`FULL_ENTITY_LIST`**
+    * **Description:** The selected and available PII entity types for the local model.
+    * **Default Value:** Predefined lists of entities (see script).
+
+* **`CUSTOM_ENTITIES`**
+    * **Description:** A list of entities that are considered "custom" and may have special handling.
+    * **Default Value:** `['TITLES', 'UKPOSTCODE', 'STREETNAME', 'CUSTOM']`
+
+* **`DEFAULT_SEARCH_QUERY`**
+    * **Description:** The default text for the custom search/redact input box.
+    * **Default Value:** `''`
+
+* **`DEFAULT_FUZZY_SPELLING_MISTAKES_NUM`**
+    * **Description:** Default number of allowed spelling mistakes for fuzzy searches.
+    * **Default Value:** `1`
+
+* **`DEFAULT_PAGE_MIN`** / **`DEFAULT_PAGE_MAX`**
+    * **Description:** Default start and end pages for processing. `0` for max means process all pages.
+    * **Default Value:** `0` for both.
+
+### Textract Feature Selection
+
+* **`DEFAULT_HANDWRITE_SIGNATURE_CHECKBOX`**
+    * **Description:** The default options selected for Textract's handwriting and signature detection.
+    * **Default Value:** `['Extract handwriting']`
+
+* **`INCLUDE_FORM_EXTRACTION_TEXTRACT_OPTION`**
+* **`INCLUDE_LAYOUT_EXTRACTION_TEXTRACT_OPTION`**
+* **`INCLUDE_TABLE_EXTRACTION_TEXTRACT_OPTION`**
+    * **Description:** Booleans (`"True"`/`"False"`) to include Forms, Layout, and Tables as selectable options for Textract analysis.
+    * **Default Value:** `"False"` for all.
+
+### Tabular Data Options
+
+* **`DO_INITIAL_TABULAR_DATA_CLEAN`**
+    * **Description:** If `"True"`, performs an initial cleaning step on tabular data.
+    * **Default Value:** `"True"`
+
+* **`DEFAULT_TEXT_COLUMNS`** / **`DEFAULT_EXCEL_SHEETS`**
+    * **Description:** Default values for specifying which columns or sheets to process in tabular files.
+    * **Default Value:** `[]` (empty list)
+
+* **`DEFAULT_TABULAR_ANONYMISATION_STRATEGY`**
+    * **Description:** The default method for anonymizing tabular data (e.g., "redact completely").
+    * **Default Value:** `"redact completely"`
 
 ## Language Options
 
-Settings for multi-language support in OCR and PII detection.
-
-*   **`SHOW_LANGUAGE_SELECTION`**
-    *   **Description:** If set to `"True"`, a dropdown menu for language selection will be visible in the user interface.
-    *   **Default Value:** `"False"`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`DEFAULT_LANGUAGE_FULL_NAME`**
-    *   **Description:** The default language's full name (e.g., "english") to be displayed in the UI.
-    *   **Default Value:** `"english"`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`DEFAULT_LANGUAGE`**
-    *   **Description:** The default language's short code (e.g., "en") used by the backend engines. Ensure the corresponding Tesseract/PaddleOCR language packs are installed.
-    *   **Default Value:** `"en"`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`MAPPED_LANGUAGE_CHOICES`**
-    *   **Description:** A string list of full language names (e.g., 'english', 'french') presented to the user in the language dropdown.
-    *   **Default Value:** A predefined list. See `config.py`.
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`LANGUAGE_CHOICES`**
-    *   **Description:** A string list of short language codes (e.g., 'en', 'fr') that correspond to `MAPPED_LANGUAGE_CHOICES`. This is what the backend uses.
-    *   **Default Value:** A predefined list. See `config.py`.
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-## App Run Options
-
-General runtime configurations for the application.
-
-*   **`TLDEXTRACT_CACHE`**
-    *   **Description:** Path to the cache directory used by the `tldextract` library, which helps in accurately extracting top-level domains (TLDs) from URLs.
-    *   **Default Value:** `'tmp/tld/'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`COGNITO_AUTH`**
-    *   **Description:** Enables or disables AWS Cognito authentication for the application. Set to `'True'` to enable.
-    *   **Default Value:** `'False'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env` (or `config/aws_config.env` if `AWS_CONFIG_PATH` is configured).
-
-*   **`RUN_DIRECT_MODE`**
-    *   **Description:** If set to `'True'`, runs the application in a "direct mode", which might alter certain behaviors (e.g., UI elements, processing flow).
-    *   **Default Value:** `'False'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`MAX_QUEUE_SIZE`**
-    *   **Description:** The maximum number of requests that can be queued in the Gradio interface.
-    *   **Default Value:** `'5'` (integer)
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`MAX_FILE_SIZE`**
-    *   **Description:** Maximum file size allowed for uploads (e.g., "250mb", "1gb").
-    *   **Default Value:** `'250mb'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`GRADIO_SERVER_PORT`**
-    *   **Description:** The network port on which the Gradio server will listen.
-    *   **Default Value:** `'7860'` (integer)
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`ROOT_PATH`**
-    *   **Description:** The root path for the application, useful if running behind a reverse proxy (e.g., `/app`).
-    *   **Default Value:** `''` (empty string)
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`DEFAULT_CONCURRENCY_LIMIT`**
-    *   **Description:** The default concurrency limit for Gradio event handlers, controlling how many requests can be processed simultaneously.
-    *   **Default Value:** `'3'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`GET_DEFAULT_ALLOW_LIST`**
-    *   **Description:** If set, enables the use of a default allow list for user access or specific functionalities. The exact behavior depends on application logic.
-    *   **Default Value:** `''` (empty string)
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`ALLOW_LIST_PATH`**
-    *   **Description:** Path to a local CSV file containing an allow list (e.g., `config/default_allow_list.csv`).
-    *   **Default Value:** `''` (empty string)
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`S3_ALLOW_LIST_PATH`**
-    *   **Description:** Path to an allow list CSV file stored in an S3 bucket (e.g., `default_allow_list.csv`). Requires `DOCUMENT_REDACTION_BUCKET` to be set.
-    *   **Default Value:** `''` (empty string)
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env` (or `config/aws_config.env` if `AWS_CONFIG_PATH` is configured).
-
-*   **`FILE_INPUT_HEIGHT`**
-    *   **Description:** Sets the height (in pixels or other CSS unit) of the file input component in the Gradio UI.
-    *   **Default Value:** `'200'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
+Settings for multi-language support.
+
+* **`SHOW_LANGUAGE_SELECTION`**
+    * **Description:** If set to `"True"`, a language selection dropdown will be visible in the UI.
+    * **Default Value:** `"False"`
+
+* **`DEFAULT_LANGUAGE_FULL_NAME`** / **`DEFAULT_LANGUAGE`**
+    * **Description:** The default language's full name (e.g., "english") and its short code (e.g., "en").
+    * **Default Values:** `"english"`, `"en"`
+
+* **`textract_language_choices`** / **`aws_comprehend_language_choices`**
+    * **Description:** Lists of supported language codes for Textract and Comprehend.
+    * **Default Value:** `['en', 'es', 'fr', 'de', 'it', 'pt']` and `['en', 'es']`
+
+* **`MAPPED_LANGUAGE_CHOICES`** / **`LANGUAGE_CHOICES`**
+    * **Description:** Paired lists of full language names and their corresponding short codes for the UI dropdown.
+    * **Default Value:** Predefined lists (see script).
+
+## Duplicate Detection Settings
+
+* **`DEFAULT_DUPLICATE_DETECTION_THRESHOLD`**
+    * **Description:** The similarity score (0.0 to 1.0) above which documents/pages are considered duplicates.
+    * **Default Value:** `0.95`
+
+* **`DEFAULT_MIN_CONSECUTIVE_PAGES`**
+    * **Description:** Minimum number of consecutive pages that must be duplicates to be flagged.
+    * **Default Value:** `1`
+
+* **`USE_GREEDY_DUPLICATE_DETECTION`**
+    * **Description:** If `"True"`, uses a greedy algorithm that may find more duplicates but can be less precise.
+    * **Default Value:** `"True"`
+
+* **`DEFAULT_COMBINE_PAGES`**
+    * **Description:** If `"True"`, text from the same page number across different files is combined before checking for duplicates.
+    * **Default Value:** `"True"`
+
+* **`DEFAULT_MIN_WORD_COUNT`**
+    * **Description:** Pages with fewer words than this value will be ignored by the duplicate detector.
+    * **Default Value:** `10`
+
+* **`REMOVE_DUPLICATE_ROWS`**
+    * **Description:** If `"True"`, enables duplicate row detection in tabular data.
+    * **Default Value:** `"False"`
+
+## File Output Options
+
+* **`USE_GUI_BOX_COLOURS_FOR_OUTPUTS`**
+    * **Description:** If `"True"`, the final redacted PDF will use the same redaction box colors as shown in the review UI.
+    * **Default Value:** `"False"`
+
+* **`CUSTOM_BOX_COLOUR`**
+    * **Description:** Specifies the color for redaction boxes as an RGB tuple string, e.g., `"(0, 0, 0)"` for black.
+    * **Default Value:** `"(0, 0, 0)"`
+
+* **`APPLY_REDACTIONS_IMAGES`**, **`APPLY_REDACTIONS_GRAPHICS`**, **`APPLY_REDACTIONS_TEXT`**
+    * **Description:** Advanced control over how redactions are applied to underlying images, vector graphics, and text in the PDF, based on PyMuPDF options. `0` is the default for a standard redaction workflow.
+    * **Default Value:** `0` for all.
+
+* **`RETURN_PDF_FOR_REVIEW`**
+    * **Description:** If set to `"True"`, a PDF with redaction boxes drawn on it (but text not removed) is generated for the "Review" tab.
+    * **Default Value:** `"True"`
+
+* **`RETURN_REDACTED_PDF`**
+    * **Description:** If set to `'True'`, the application will return a fully redacted PDF at the end of the main task.
+    * **Default Value:** `"True"`
+
+* **`COMPRESS_REDACTED_PDF`**
+    * **Description:** If set to `'True'`, the redacted PDF output will be compressed.
+    * **Default Value:** `"False"`
+
+## Direct Mode & Lambda Configuration
+
+Settings for running the application from the command line (Direct Mode) or as an AWS Lambda function.
+
+### Direct Mode
+
+* **`RUN_DIRECT_MODE`**
+    * **Description:** Set to `'True'` to enable direct command-line mode.
+    * **Default Value:** `'False'`
+
+* **`DIRECT_MODE_DEFAULT_USER`**
+    * **Description:** Default username for CLI requests.
+    * **Default Value:** `''`
+
+* **`DIRECT_MODE_TASK`**
+    * **Description:** The task to perform: `'redact'` or `'deduplicate'`.
+    * **Default Value:** `'redact'`
+
+* **`DIRECT_MODE_INPUT_FILE`** / **`DIRECT_MODE_OUTPUT_DIR`**
+    * **Description:** Path to the input file and output directory for the task.
+    * **Default Values:** `''`, `output/`
+
+* **Other `DIRECT_MODE_*` variables:**
+    * **Description:** These variables allow for setting nearly all application options (e.g., `DIRECT_MODE_PII_DETECTOR`, `DIRECT_MODE_SIMILARITY_THRESHOLD`) directly for a single CLI run, overriding other configurations.
+    * **Default Value:** Defaults are inherited from the main application settings (e.g., `LOCAL_PII_OPTION`, `DEFAULT_DUPLICATE_DETECTION_THRESHOLD`).
+
+### Lambda Configuration
+
+* **`LAMBDA_POLL_INTERVAL`**
+    * **Description:** Polling interval in seconds for checking Textract job status.
+    * **Default Value:** `30`
+
+* **`LAMBDA_MAX_POLL_ATTEMPTS`**
+    * **Description:** Maximum number of polling attempts before timeout.
+    * **Default Value:** `120`
+
+* **`LAMBDA_PREPARE_IMAGES`**
+    * **Description:** If `"True"`, prepares images for OCR processing within the Lambda environment.
+    * **Default Value:** `"True"`
+
+* **`LAMBDA_EXTRACT_SIGNATURES`**
+    * **Description:** Enables signature extraction during Textract analysis in Lambda.
+    * **Default Value:** `"False"`
+
+* **`LAMBDA_DEFAULT_USERNAME`**
+    * **Description:** Default username for operations initiated by Lambda.
+    * **Default Value:** `"lambda_user"`
+
+## Allow, Deny, & Whole Page Redaction Lists
+
+* **`GET_DEFAULT_ALLOW_LIST`**, **`GET_DEFAULT_DENY_LIST`**, **`GET_DEFAULT_WHOLE_PAGE_REDACTION_LIST`**
+    * **Description:** Booleans (`"True"`/`"False"`) to enable the use of allow, deny, or whole-page redaction lists.
+    * **Default Value:** `"False"`
+
+* **`ALLOW_LIST_PATH`**, **`DENY_LIST_PATH`**, **`WHOLE_PAGE_REDACTION_LIST_PATH`**
+    * **Description:** Local paths to the respective CSV list files.
+    * **Default Value:** `''`
+
+* **`S3_ALLOW_LIST_PATH`**, **`S3_DENY_LIST_PATH`**, **`S3_WHOLE_PAGE_REDACTION_LIST_PATH`**
+    * **Description:** Paths to the respective list files within the `DOCUMENT_REDACTION_BUCKET`.
+    * **Default Value:** `''`
 
 ## Cost Code Options
 
-Settings related to tracking and applying cost codes for application usage.
-
-*   **`SHOW_COSTS`**
-    *   **Description:** If set to `'True'`, cost-related information will be displayed in the UI.
-    *   **Default Value:** `'False'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`GET_COST_CODES`**
-    *   **Description:** Enables fetching and using cost codes within the application. Set to `'True'` to enable.
-    *   **Default Value:** `'False'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`DEFAULT_COST_CODE`**
-    *   **Description:** Specifies a default cost code to be used if cost codes are enabled but none is selected by the user.
-    *   **Default Value:** `''` (empty string)
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`COST_CODES_PATH`**
-    *   **Description:** Path to a local CSV file containing available cost codes (e.g., `config/COST_CENTRES.csv`).
-    *   **Default Value:** `''` (empty string)
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`S3_COST_CODES_PATH`**
-    *   **Description:** Path to a cost codes CSV file stored in an S3 bucket (e.g., `COST_CENTRES.csv`). Requires `DOCUMENT_REDACTION_BUCKET` to be set.
-    *   **Default Value:** `''` (empty string)
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env` (or `config/aws_config.env` if `AWS_CONFIG_PATH` is configured).
-
-*   **`ENFORCE_COST_CODES`**
-    *   **Description:** If set to `'True'` and `GET_COST_CODES` is also enabled, makes the selection of a cost code mandatory for users.
-    *   **Default Value:** `'False'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-## Whole Document API Options
-
-Configurations for features related to processing whole documents via APIs, particularly AWS Textract for large documents.
-
-*   **`SHOW_WHOLE_DOCUMENT_TEXTRACT_CALL_OPTIONS`**
-    *   **Description:** Controls whether UI options for whole document Textract calls are displayed.
-    *   **Default Value:** `'False'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`TEXTRACT_WHOLE_DOCUMENT_ANALYSIS_BUCKET`**
-    *   **Description:** The S3 bucket used for input and output of whole document analysis with AWS Textract.
-    *   **Default Value:** `''` (empty string)
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env` (or `config/aws_config.env` if `AWS_CONFIG_PATH` is configured).
-
-*   **`TEXTRACT_WHOLE_DOCUMENT_ANALYSIS_INPUT_SUBFOLDER`**
-    *   **Description:** The subfolder within `TEXTRACT_WHOLE_DOCUMENT_ANALYSIS_BUCKET` where input documents for Textract analysis are placed.
-    *   **Default Value:** `'input'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env` (or `config/aws_config.env` if `AWS_CONFIG_PATH` is configured).
-
-*   **`TEXTRACT_WHOLE_DOCUMENT_ANALYSIS_OUTPUT_SUBFOLDER`**
-    *   **Description:** The subfolder within `TEXTRACT_WHOLE_DOCUMENT_ANALYSIS_BUCKET` where output results from Textract analysis are stored.
-    *   **Default Value:** `'output'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env` (or `config/aws_config.env` if `AWS_CONFIG_PATH` is configured).
-
-*   **`LOAD_PREVIOUS_TEXTRACT_JOBS_S3`**
-    *   **Description:** If set to `'True'`, the application will attempt to load data from previous Textract jobs stored in S3.
-    *   **Default Value:** `'False'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env` (or `config/aws_config.env` if `AWS_CONFIG_PATH` is configured).
-
-*   **`TEXTRACT_JOBS_S3_LOC`**
-    *   **Description:** The S3 subfolder (within the main redaction bucket) where Textract job data (output) is stored.
-    *   **Default Value:** `'output'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env` (or `config/aws_config.env` if `AWS_CONFIG_PATH` is configured).
-
-*   **`TEXTRACT_JOBS_S3_INPUT_LOC`**
-    *   **Description:** The S3 subfolder (within the main redaction bucket) where Textract job input is stored.
-    *   **Default Value:** `'input'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env` (or `config/aws_config.env` if `AWS_CONFIG_PATH` is configured).
-
-*   **`TEXTRACT_JOBS_LOCAL_LOC`**
-    *   **Description:** The local subfolder where Textract job data is stored if not using S3 or as a cache.
-    *   **Default Value:** `'output'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
-
-*   **`DAYS_TO_DISPLAY_WHOLE_DOCUMENT_JOBS`**
-    *   **Description:** Specifies the number of past days for which to display whole document Textract jobs in the UI.
-    *   **Default Value:** `'7'`
-    *   **Configuration:** Set as an environment variable directly, or include in `config/app_config.env`.
+* **`SHOW_COSTS`**
+    * **Description:** If set to `'True'`, cost-related information will be displayed in the UI.
+    * **Default Value:** `'False'`
+
+* **`GET_COST_CODES`**
+    * **Description:** Enables fetching and using cost codes. Set to `'True'` to enable.
+    * **Default Value:** `'False'`
+
+* **`DEFAULT_COST_CODE`**
+    * **Description:** Specifies a default cost code.
+    * **Default Value:** `''`
+
+* **`COST_CODES_PATH`** / **`S3_COST_CODES_PATH`**
+    * **Description:** Local or S3 path to a CSV file containing available cost codes.
+    * **Default Value:** `''`
+
+* **`ENFORCE_COST_CODES`**
+    * **Description:** If set to `'True'`, makes the selection of a cost code mandatory.
+    * **Default Value:** `'False'`
+
+## Whole Document API Options (Textract Async)
+
+* **`SHOW_WHOLE_DOCUMENT_TEXTRACT_CALL_OPTIONS`**
+    * **Description:** Controls whether UI options for asynchronous whole document Textract calls are displayed.
+    * **Default Value:** `'False'`
+
+* **`TEXTRACT_WHOLE_DOCUMENT_ANALYSIS_BUCKET`**
+    * **Description:** The S3 bucket used for asynchronous Textract analysis.
+    * **Default Value:** `''`
+
+* **`TEXTRACT_WHOLE_DOCUMENT_ANALYSIS_INPUT_SUBFOLDER`** / **`..._OUTPUT_SUBFOLDER`**
+    * **Description:** Input and output subfolders within the analysis bucket.
+    * **Default Values:** `'input'`, `'output'`
+
+* **`LOAD_PREVIOUS_TEXTRACT_JOBS_S3`**
+    * **Description:** If set to `'True'`, the application will load data from previous Textract jobs stored in S3.
+    * **Default Value:** `'False'`
+
+* **`TEXTRACT_JOBS_S3_LOC`** / **`TEXTRACT_JOBS_S3_INPUT_LOC`**
+    * **Description:** S3 subfolders where Textract job output and input are stored.
+    * **Default Value:** `'output'`, `'input'`
+
+* **`TEXTRACT_JOBS_LOCAL_LOC`**
+    * **Description:** The local subfolder for storing Textract job data.
+    * **Default Value:** `'output'`
+
+* **`DAYS_TO_DISPLAY_WHOLE_DOCUMENT_JOBS`**
+    * **Description:** Specifies the number of past days for which to display whole document Textract jobs.
+    * **Default Value:** `7`

src/user_guide.qmd

@@ -9,7 +9,8 @@ format:
 
 ## Table of contents
 
-- [Example data files](#example-data-files)
+### Getting Started
+- [Built-in example data](#built-in-example-data)
 - [Basic redaction](#basic-redaction)
 - [Customising redaction options](#customising-redaction-options)
     - [Custom allow, deny, and page redaction lists](#custom-allow-deny-and-page-redaction-lists)
@@ -21,21 +22,60 @@ format:
     - [Handwriting and signature redaction](#handwriting-and-signature-redaction)
 - [Reviewing and modifying suggested redactions](#reviewing-and-modifying-suggested-redactions)
 - [Redacting Word, tabular data files (CSV/XLSX) or copy and pasted text](#redacting-word-tabular-data-files-xlsxcsv-or-copy-and-pasted-text)
-
-See the [advanced user guide here](#advanced-user-guide):
-- [Merging redaction review files](#merging-redaction-review-files)
 - [Identifying and redacting duplicate pages](#identifying-and-redacting-duplicate-pages)
+
+### Advanced user guide
+- [Advanced user guide](#advanced-user-guide)
 - [Fuzzy search and redaction](#fuzzy-search-and-redaction)
 - [Export redactions to and import from Adobe Acrobat](#export-to-and-import-from-adobe)
+    - [Using _for_review.pdf files with Adobe Acrobat](#using-_for_reviewpdf-files-with-adobe-acrobat)
     - [Exporting to Adobe Acrobat](#exporting-to-adobe-acrobat)
     - [Importing from Adobe Acrobat](#importing-from-adobe-acrobat)
 - [Using the AWS Textract document API](#using-the-aws-textract-document-api)
 - [Using AWS Textract and Comprehend when not running in an AWS environment](#using-aws-textract-and-comprehend-when-not-running-in-an-aws-environment)
 - [Modifying existing redaction review files](#modifying-existing-redaction-review-files)
+- [Merging redaction review files](#merging-redaction-review-files)
+
+### Features for expert users/system administrators
+- [Features for expert users/system administrators](#features-for-expert-userssystem-administrators)
+- [Advanced OCR options (Hybrid OCR)](#advanced-ocr-options-hybrid-ocr)
+- [Command Line Interface (CLI)](#command-line-interface-cli)
+
+## Built-in example data
+
+The app now includes built-in example files that you can use to quickly test different features. These examples are automatically loaded and can be accessed directly from the interface without needing to download files separately.
+
+### Using built-in examples
+
+**For PDF/image redaction:** On the 'Redact PDFs/images' tab, you'll see a section titled "Try an example - Click on an example below and then the 'Extract text and redact document' button". Simply click on any of the available examples to load them with pre-configured settings:
+
+- **PDF with selectable text redaction** - Uses local text extraction with standard PII detection
+- **Image redaction with local OCR** - Processes an image file using OCR
+- **PDF redaction with custom entities** - Demonstrates custom entity selection (Titles, Person, Dates)
+- **PDF redaction with AWS services and signature detection** - Shows AWS Textract with signature extraction (if AWS is enabled)
+- **PDF redaction with custom deny list and whole page redaction** - Demonstrates advanced redaction features
+
+Once you have clicked on an example, you can click the 'Extract text and redact document' button to load the example into the app and redact it.
+
+**For tabular data:** On the 'Word or Excel/csv files' tab, you'll find examples for both redaction and duplicate detection:
+
+- **CSV file redaction** - Shows how to redact specific columns in tabular data
+- **Word document redaction** - Demonstrates Word document processing
+- **Excel file duplicate detection** - Shows how to find duplicate rows in spreadsheet data
+
+Once you have clicked on an example, you can click the 'Redact text/data files' button to load the example into the app and redact it. For the duplicate detection example, you can click the 'Find duplicate cells/rows' button to load the example into the app and find duplicates.
+
+**For duplicate page detection:** On the 'Identify duplicate pages' tab, you'll find examples for finding duplicate content in documents:
+
+- **Find duplicate pages of text in document OCR outputs** - Uses page-level analysis with a similarity threshold of 0.95 and minimum word count of 10
+- **Find duplicate text lines in document OCR outputs** - Uses line-level analysis with a similarity threshold of 0.95 and minimum word count of 3
+
+Once you have clicked on an example, you can click the 'Identify duplicate pages/subdocuments' button to load the example into the app and find duplicate content.
+
+### External example files (optional)
 
-## Example data files
+If you prefer to use your own example files or want to follow along with specific tutorials, you can still download these external example files:
 
-Please try these example files to follow along with this guide: 
 - [Example of files sent to a professor before applying](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/example_of_emails_sent_to_a_professor_before_applying.pdf)
 - [Example complaint letter (jpg)](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/example_complaint_letter.jpg)
 - [Partnership Agreement Toolkit (for signatures and more advanced usage)](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/Partnership-Agreement-Toolkit_0_0.pdf)
@@ -55,16 +95,20 @@ The 'Redact PDFs/images tab' currently accepts PDFs and image files (JPG, PNG) f
 
 ### Text extraction
 
-First, select one of the three text extraction options:
+You can modify default text extraction methods by clicking on the 'Change default text extraction method...' box'.
+
+Here you can select one of the three text extraction options:
 - **'Local model - selectable text'** - This will read text directly from PDFs that have selectable text to redact (using PikePDF). This is fine for most PDFs, but will find nothing if the PDF does not have selectable text, and it is not good for handwriting or signatures. If it encounters an image file, it will send it onto the second option below.
 - **'Local OCR model - PDFs without selectable text'** - This option will use a simple Optical Character Recognition (OCR) model (Tesseract) to pull out text from a PDF/image that it 'sees'. This can handle most typed text in PDFs/images without selectable text, but struggles with handwriting/signatures. If you are interested in the latter, then you should use the third option if available.
 - **'AWS Textract service - all PDF types'** - Only available for instances of the app running on AWS. AWS Textract is a service that performs OCR on documents within their secure service. This is a more advanced version of OCR compared to the local option, and carries a (relatively small) cost. Textract excels in complex documents based on images, or documents that contain a lot of handwriting and signatures.
 
-### Optional - select signature extraction
+### Enable AWS Textract signature extraction
 If you chose the AWS Textract service above, you can choose if you want handwriting and/or signatures redacted by default. Choosing signatures here will have a cost implication, as identifying signatures will cost ~£2.66 ($3.50) per 1,000 pages vs ~£1.14 ($1.50) per 1,000 pages without signature detection. 
 
 ![AWS Textract handwriting and signature options](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/quick_start/textract_handwriting_signatures.PNG)
 
+**NOTE:** it is also possible to enable form extraction, layout extraction, and table extraction with AWS Textract. This is not enabled by default, but it is possible for your system admin to enable this feature in the config file.
+
 ### PII redaction method
 
 If you are running with the AWS service enabled, here you will also have a choice for PII redaction method:
@@ -98,6 +142,7 @@ Click 'Redact document'. After loading in the document, the app should be able t
 ![Redaction outputs](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/quick_start/redaction_outputs.PNG)
 
 - **'...redacted.pdf'** files contain the original pdf with suggested redacted text deleted and replaced by a black box on top of the document.
+- **'...redactions_for_review.pdf'** files contain the original PDF with redaction boxes overlaid but the original text still visible underneath. This file is designed for use in Adobe Acrobat and other PDF viewers where you can see the suggested redactions without the text being permanently removed. This is particularly useful for reviewing redactions before finalising them.
 - **'...ocr_results.csv'** files contain the line-by-line text outputs from the entire document. This file can be useful for later searching through for any terms of interest in the document (e.g. using Excel or a similar program).
 - **'...review_file.csv'** files are the review files that contain details and locations of all of the suggested redactions in the document. This file is key to the [review process](#reviewing-and-modifying-suggested-redactions), and should be downloaded to use later for this.
 
@@ -166,8 +211,6 @@ If the table is empty, you can add a new entry, you can add a new row by clickin
 
 ![Manually modify allow or deny list filled](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/allow_list/manually_modify_filled.PNG)
 
-**Note:** As of version 0.7.0 you can now apply your whole page redaction list directly to the document file currently under review by clicking the 'Apply whole page redaction list to document currently under review' button that appears here.
-
 ### Redacting additional types of personal information
 
 You may want to redact additional types of information beyond the defaults, or you may not be interested in default suggested entity types. There are dates in the example complaint letter. Say we wanted to redact those dates also?
@@ -182,7 +225,7 @@ If you want to redact different files, I suggest you refresh your browser page t
 
 ## Redacting only specific pages
 
-Say also we are only interested in redacting page 1 of the loaded documents. On the Redaction settings tab, select 'Lowest page to redact' as 1, and 'Highest page to redact' also as 1. When you next redact your documents, only the first page will be modified.
+Say also we are only interested in redacting page 1 of the loaded documents. On the Redaction settings tab, select 'Lowest page to redact' as 1, and 'Highest page to redact' also as 1. When you next redact your documents, only the first page will be modified. The output files should now have a suffix similar to '..._1_1.pdf', indicating the lowest and highest page numbers that were redacted.
 
 ![Selecting specific pages to redact](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/allow_list/select_pages.PNG)
 
@@ -419,39 +462,16 @@ You can also write open text into an input box and redact that using the same me
 ### Redaction log outputs
 A list of the suggested redaction outputs from the tabular data / open text data redaction is available on the Redaction settings page under 'Log file outputs'.
 
-# ADVANCED USER GUIDE
-
-This advanced user guide will go over some of the features recently added to the app, including: modifying and merging redaction review files, identifying and redacting duplicate pages across multiple PDFs, 'fuzzy' search and redact, and exporting redactions to Adobe Acrobat.
-
-## Table of contents
-
-- [Merging redaction review files](#merging-redaction-review-files)
-- [Identifying and redacting duplicate pages](#identifying-and-redacting-duplicate-pages)
-- [Fuzzy search and redaction](#fuzzy-search-and-redaction)
-- [Export redactions to and import from Adobe Acrobat](#export-to-and-import-from-adobe)
-    - [Exporting to Adobe Acrobat](#exporting-to-adobe-acrobat)
-    - [Importing from Adobe Acrobat](#importing-from-adobe-acrobat)
-- [Using the AWS Textract document API](#using-the-aws-textract-document-api)
-- [Using AWS Textract and Comprehend when not running in an AWS environment](#using-aws-textract-and-comprehend-when-not-running-in-an-aws-environment)
-- [Modifying existing redaction review files](#modifying-existing-redaction-review-files)
-
-
-## Merging redaction review files
-
-Say you have run multiple redaction tasks on the same document, and you want to merge all of these redactions together. You could do this in your spreadsheet editor, but this could be fiddly especially if dealing with multiple review files or large numbers of redactions. The app has a feature to combine multiple review files together to create a 'merged' review file.
-
-![Merging review files in the user interface](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/merge_review_files/img/merge_review_files_interface.PNG)
-
-You can find this option at the bottom of the 'Redaction Settings' tab. Upload multiple review files here to get a single output 'merged' review_file. In the examples file, merging the 'review_file_custom.csv' and 'review_file_local.csv' files give you an output containing redaction boxes from both. This combined review file can then be uploaded into the review tab following the usual procedure.
-
-![Merging review files outputs in spreadsheet](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/merge_review_files/img/merged_review_file_outputs_csv.PNG)
-
 ## Identifying and redacting duplicate pages
 
 The files for this section are stored [here](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/duplicate_page_find_in_app/).
 
 Some redaction tasks involve removing duplicate pages of text that may exist across multiple documents. This feature helps you find and remove duplicate content that may exist in single or multiple documents.  It can identify everything from single identical pages to multi-page sections (subdocuments). The process involves three main steps: configuring the analysis, reviewing the results in the interactive interface, and then using the generated files to perform the redactions.
 
+### Duplicate page detection in documents
+
+This section covers finding duplicate pages across PDF documents using OCR output files.
+
 ![Example duplicate page inputs](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/duplicate_page_find_in_app/img/duplicate_page_input_interface_new.PNG)
 
 **Step 1: Upload and Configure the Analysis**
@@ -496,11 +516,43 @@ The analysis also generates a set of downloadable files for your records and for
 
 If you want to combine the results from this redaction process with previous redaction tasks for the same PDF, you could merge review file outputs following the steps described in [Merging existing redaction review files](#merging-existing-redaction-review-files) above.
 
+### Duplicate detection in tabular data
+
+The app also includes functionality to find duplicate cells or rows in CSV, Excel, or Parquet files. This is particularly useful for cleaning datasets where you need to identify and remove duplicate entries.
+
+**Step 1: Upload files and configure analysis**
+
+Navigate to the 'Word or Excel/csv files' tab and scroll down to the "Find duplicate cells in tabular data" section. Upload your tabular files (CSV, Excel, or Parquet) and configure the analysis parameters:
+
+- **Similarity threshold**: Score (0-1) to consider cells a match. 1 = perfect match
+- **Minimum word count**: Cells with fewer words than this value are ignored
+- **Do initial clean of text**: Remove URLs, HTML tags, and non-ASCII characters
+- **Remove duplicate rows**: Automatically remove duplicate rows from deduplicated files
+- **Select Excel sheet names**: Choose which sheets to analyze (for Excel files)
+- **Select text columns**: Choose which columns contain text to analyze
+
+**Step 2: Review results**
+
+After clicking "Find duplicate cells/rows", the results will be displayed in a table showing:
+- File1, Row1, File2, Row2
+- Similarity_Score
+- Text1, Text2 (the actual text content being compared)
+
+Click on any row to see more details about the duplicate match in the preview boxes below.
+
+**Step 3: Remove duplicates**
+
+Select a file from the dropdown and click "Remove duplicate rows from selected file" to create a cleaned version with duplicates removed. The cleaned file will be available for download.
+
+# Advanced user guide
+
+This advanced user guide covers features that require system administration access or command-line usage. These features are typically used by system administrators or advanced users who need more control over the redaction process.
+
 ## Fuzzy search and redaction
 
 The files for this section are stored [here](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/fuzzy_search/).
 
-Sometimes you may be searching for terns that are slightly mispelled throughout a document, for example names. The document redaction app gives the option for searching for long phrases that may contain spelling mistakes, a method called 'fuzzy matching'.
+Sometimes you may be searching for terms that are slightly mispelled throughout a document, for example names. The document redaction app gives the option for searching for long phrases that may contain spelling mistakes, a method called 'fuzzy matching'.
 
 To do this, go to the Redaction Settings, and the 'Select entity types to redact' area. In the box below relevant to your chosen redaction method (local or AWS Comprehend), select 'CUSTOM_FUZZY' from the list. Next, we can select the maximum number of spelling mistakes allowed in the search (up to nine). Here, you can either type in a number or use the small arrows to the right of the box. Change this option to 3. This will allow for a maximum of three 'changes' in text needed to match to the desired search terms.
 
@@ -520,9 +572,20 @@ Using these deny list with spelling mistakes, the app fuzzy match these terms to
 
 Files for this section are stored [here](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/export_to_adobe/).
 
-### Exporting to Adobe Acrobat
+The Document Redaction app has enhanced features for working with Adobe Acrobat. You can now export suggested redactions to Adobe, import Adobe comment files into the app, and use the new `_for_review.pdf` files directly in Adobe Acrobat.
+
+### Using _for_review.pdf files with Adobe Acrobat
 
-The Document Redaction app has a feature to export suggested redactions to Adobe, and likewise to import Adobe comment files into the app. The file format used is the .xfdf Adobe comment file format - [you can find more information about how to use these files here](https://helpx.adobe.com/uk/acrobat/using/importing-exporting-comments.html).
+The app now generates `...redactions_for_review.pdf` files that contain the original PDF with redaction boxes overlaid but the original text still visible underneath. These files are specifically designed for use in Adobe Acrobat and other PDF viewers where you can:
+
+- See the suggested redactions without the text being permanently removed
+- Review redactions before finalising them
+- Use Adobe Acrobat's built-in redaction tools to modify or apply the redactions
+- Export the final redacted version directly from Adobe
+
+Simply open the `...redactions_for_review.pdf` file in Adobe Acrobat to begin reviewing and modifying the suggested redactions.
+
+### Exporting to Adobe Acrobat
 
 To convert suggested redactions to Adobe format, you need to have the original PDF and a review file csv in the input box at the top of the Review redactions page.
 
@@ -570,6 +633,46 @@ The '_textract.json' output can be used to speed up further redaction tasks as [
 
 You can now easily get the '..._ocr_output.csv' redaction output based on this '_textract.json' (described in [Redaction outputs](#redaction-outputs)) by clicking on the button 'Convert Textract job outputs to OCR results'. You can now use this file e.g. for [identifying duplicate pages](#identifying-and-redacting-duplicate-pages), or for redaction review.
 
+
+
+## Modifying existing redaction review files
+You can find the folder containing the files discussed in this section [here](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/merge_review_files/).
+
+As well as serving as inputs to the document redaction app's review function, the 'review_file.csv' output can be modified insider or outside of the app. This gives you the flexibility to change redaction details outside of the app.
+
+### Inside the app
+You can now modify redaction review files directly in the app on the 'Review redactions' tab. Open the accordion 'View and edit review data' under the file input area. You can edit review file data cells here - press Enter to apply changes. You should see the effect on the current page if you click the 'Save changes on current page to file' button to the right.
+
+### Outside the app
+If you open up a 'review_file' csv output using a spreadsheet software program such as Microsoft Excel you can easily modify redaction properties. Open the file '[Partnership-Agreement-Toolkit_0_0_redacted.pdf_review_file_local.csv](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/merge_review_files/Partnership-Agreement-Toolkit_0_0.pdf_review_file_local.csv)', and you should see a spreadshet with just four suggested redactions (see below). The following instructions are for using Excel.
+
+![Review file before](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/merge_review_files/img/review_file_before.PNG)
+
+The first thing we can do is remove the first row - 'et' is suggested as a person, but is obviously not a genuine instance of personal information. Right click on the row number and select delete on this menu. Next, let's imagine that what the app identified as a 'phone number' was in fact another type of number and so we wanted to change the label. Simply click on the relevant label cells, let's change it to 'SECURITY_NUMBER'. You could also use 'Find & Select' -> 'Replace' from the top ribbon menu if you wanted to change a number of labels simultaneously.
+
+How about we wanted to change the colour of the 'email address' entry on the redaction review tab of the redaction app? The colours in a review file are based on an RGB scale with three numbers ranging from 0-255. [You can find suitable colours here](https://rgbcolorpicker.com). Using this scale, if I wanted my review box to be pure blue, I can change the cell value to (0,0,255).
+
+Imagine that a redaction box was slightly too small, and I didn't want to use the in-app options to change the size. In the review file csv, we can modify e.g. the ymin and ymax values for any box to increase the extent of the redaction box. For the 'email address' entry, let's decrease ymin by 5, and increase ymax by 5.
+
+I have saved an output file following the above steps as '[Partnership-Agreement-Toolkit_0_0_redacted.pdf_review_file_local_mod.csv](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/merge_review_files/outputs/Partnership-Agreement-Toolkit_0_0.pdf_review_file_local_mod.csv)' in the same folder that the original was found. Let's upload this file to the app along with the original pdf to see how the redactions look now.
+
+![Review file after modification](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/merge_review_files/img/partnership_redactions_after.PNG)
+
+We can see from the above that we have successfully removed a redaction box, changed labels, colours, and redaction box sizes.
+
+## Merging redaction review files
+
+Say you have run multiple redaction tasks on the same document, and you want to merge all of these redactions together. You could do this in your spreadsheet editor, but this could be fiddly especially if dealing with multiple review files or large numbers of redactions. The app has a feature to combine multiple review files together to create a 'merged' review file.
+
+![Merging review files in the user interface](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/merge_review_files/img/merge_review_files_interface.PNG)
+
+You can find this option at the bottom of the 'Redaction Settings' tab. Upload multiple review files here to get a single output 'merged' review_file. In the examples file, merging the 'review_file_custom.csv' and 'review_file_local.csv' files give you an output containing redaction boxes from both. This combined review file can then be uploaded into the review tab following the usual procedure.
+
+![Merging review files outputs in spreadsheet](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/merge_review_files/img/merged_review_file_outputs_csv.PNG)
+
+# Features for expert users/system administrators
+This advanced user guide covers features that require system administration access or command-line usage. These options are not enabled by default but can be configured by your system administrator, and are not available to users who are just using the graphical user interface. These features are typically used by system administrators or advanced users who need more control over the redaction process.
+
 ## Using AWS Textract and Comprehend when not running in an AWS environment
 
 AWS Textract and Comprehend give much better results for text extraction and document redaction than the local model options in the app. The most secure way to access them in the Redaction app is to run the app in a secure AWS environment with relevant permissions. Alternatively, you could run the app on your own system while logged in to AWS SSO with relevant permissions.
@@ -591,26 +694,180 @@ The app should then pick up these keys when trying to access the AWS Textract an
 
 Again, a lot can potentially go wrong with AWS solutions that are insecure, so before trying the above please consult with your AWS and data security teams.
 
-## Modifying existing redaction review files
+## Advanced OCR options (Hybrid OCR)
 
-*Note:* As of version 0.7.0 you can now modify redaction review files directly in the app on the 'Review redactions' tab. Open the accordion 'View and edit review data' under the file input area. You can edit review file data cells here - press Enter to apply changes. You should see the effect on the current page if you click the 'Save changes on current page to file' button to the right.
+The app supports advanced OCR options that combine multiple OCR engines for improved accuracy. These options are not enabled by default but can be configured by your system administrator.
 
-You can find the folder containing the files discussed in this section [here](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/merge_review_files/).
+### Available OCR models
 
-As well as serving as inputs to the document redaction app's review function, the 'review_file.csv' output can be modified outside of the app, and also merged with others from multiple redaction attempts on the same file. This gives you the flexibility to change redaction details outside of the app.
+- **Tesseract** (default): The standard OCR engine that works well for most documents
+- **PaddleOCR**: More accurate for whole line text extraction, but word-level bounding boxes may be less precise
+- **Hybrid**: Combines Tesseract and PaddleOCR - uses Tesseract for initial extraction, then PaddleOCR for re-extraction of low-confidence text
 
-If you open up a 'review_file' csv output using a spreadsheet software program such as Microsoft Excel you can easily modify redaction properties. Open the file '[Partnership-Agreement-Toolkit_0_0_redacted.pdf_review_file_local.csv](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/merge_review_files/Partnership-Agreement-Toolkit_0_0.pdf_review_file_local.csv)', and you should see a spreadshet with just four suggested redactions (see below). The following instructions are for using Excel.
+### Enabling advanced OCR options
 
-![Review file before](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/merge_review_files/img/review_file_before.PNG)
+To enable these options, your system administrator needs to modify the configuration file (`config.py`) and set:
 
-The first thing we can do is remove the first row - 'et' is suggested as a person, but is obviously not a genuine instance of personal information. Right click on the row number and select delete on this menu. Next, let's imagine that what the app identified as a 'phone number' was in fact another type of number and so we wanted to change the label. Simply click on the relevant label cells, let's change it to 'SECURITY_NUMBER'. You could also use 'Find & Select' -> 'Replace' from the top ribbon menu if you wanted to change a number of labels simultaneously.
+```
+SHOW_LOCAL_OCR_MODEL_OPTIONS = "True"
+```
 
-How about we wanted to change the colour of the 'email address' entry on the redaction review tab of the redaction app? The colours in a review file are based on an RGB scale with three numbers ranging from 0-255. [You can find suitable colours here](https://rgbcolorpicker.com). Using this scale, if I wanted my review box to be pure blue, I can change the cell value to (0,0,255).
+Once enabled, users will see a "Change default local OCR model" section in the redaction settings where they can choose between:
+- tesseract
+- hybrid  
+- paddle
 
-Imagine that a redaction box was slightly too small, and I didn't want to use the in-app options to change the size. In the review file csv, we can modify e.g. the ymin and ymax values for any box to increase the extent of the redaction box. For the 'email address' entry, let's decrease ymin by 5, and increase ymax by 5.
+### Hybrid OCR configuration
 
-I have saved an output file following the above steps as '[Partnership-Agreement-Toolkit_0_0_redacted.pdf_review_file_local_mod.csv](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/merge_review_files/outputs/Partnership-Agreement-Toolkit_0_0.pdf_review_file_local_mod.csv)' in the same folder that the original was found. Let's upload this file to the app along with the original pdf to see how the redactions look now.
+The hybrid OCR mode uses several configurable parameters:
 
-![Review file after modification](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/merge_review_files/img/partnership_redactions_after.PNG)
+- **HYBRID_OCR_CONFIDENCE_THRESHOLD** (default: 65): Tesseract confidence score below which PaddleOCR will be used for re-extraction
+- **HYBRID_OCR_PADDING** (default: 1): Padding added to word bounding boxes before re-extraction
+- **SAVE_EXAMPLE_TESSERACT_VS_PADDLE_IMAGES** (default: False): Save comparison images when using hybrid mode
+- **SAVE_PADDLE_VISUALISATIONS** (default: False): Save images with PaddleOCR bounding boxes overlaid
+
+### When to use different OCR models
+
+- **Tesseract**: Best for general use, good balance of speed and accuracy
+- **PaddleOCR**: Best for documents with clear, well-formatted text where line-level accuracy is more important than word-level precision
+- **Hybrid**: Best for challenging documents where some text has low confidence scores, providing the benefits of both engines
+
+
+
+
+
+## Command Line Interface (CLI)
+
+The app includes a comprehensive command-line interface (`cli_redact.py`) that allows you to perform redaction, deduplication, and AWS Textract operations directly from the terminal. This is particularly useful for batch processing, automation, and integration with other systems.
+
+### Getting started with the CLI
+
+To use the CLI, you need to:
+
+1. Open a terminal window
+2. Navigate to the app folder containing `cli_redact.py`
+3. Activate your virtual environment (conda or venv)
+4. Run commands using `python cli_redact.py` followed by your options
+
+### Basic CLI syntax
+
+```bash
+python cli_redact.py --task [redact|deduplicate|textract] --input_file [file_path] [additional_options]
+```
+
+### Redaction examples
+
+**Basic PDF redaction with default settings:**
+```bash
+python cli_redact.py --input_file example_data/example_of_emails_sent_to_a_professor_before_applying.pdf
+```
+
+**Extract text only (no redaction) with whole page redaction:**
+```bash
+python cli_redact.py --input_file example_data/Partnership-Agreement-Toolkit_0_0.pdf --redact_whole_page_file example_data/partnership_toolkit_redact_some_pages.csv --pii_detector None
+```
+
+**Redact with custom entities and allow list:**
+```bash
+python cli_redact.py --input_file example_data/graduate-job-example-cover-letter.pdf --allow_list_file example_data/test_allow_list_graduate.csv --local_redact_entities TITLES PERSON DATE_TIME
+```
+
+**Redact with fuzzy matching and custom deny list:**
+```bash
+python cli_redact.py --input_file example_data/Partnership-Agreement-Toolkit_0_0.pdf --deny_list_file example_data/Partnership-Agreement-Toolkit_test_deny_list_para_single_spell.csv --local_redact_entities CUSTOM_FUZZY --page_min 1 --page_max 3 --fuzzy_mistakes 3
+```
+
+**Redact with AWS services:**
+```bash
+python cli_redact.py --input_file example_data/example_of_emails_sent_to_a_professor_before_applying.pdf --ocr_method "AWS Textract" --pii_detector "AWS Comprehend"
+```
+
+**Redact specific pages with signature extraction:**
+```bash
+python cli_redact.py --input_file example_data/Partnership-Agreement-Toolkit_0_0.pdf --page_min 6 --page_max 7 --ocr_method "AWS Textract" --handwrite_signature_extraction "Extract handwriting" "Extract signatures"
+```
+
+### Tabular data redaction
+
+**Anonymize CSV file with specific columns:**
+```bash
+python cli_redact.py --input_file example_data/combined_case_notes.csv --text_columns "Case Note" "Client" --anon_strategy replace_redacted
+```
+
+**Anonymize Excel file:**
+```bash
+python cli_redact.py --input_file example_data/combined_case_notes.xlsx --text_columns "Case Note" "Client" --excel_sheets combined_case_notes --anon_strategy redact
+```
+
+**Anonymize Word document:**
+```bash
+python cli_redact.py --input_file "example_data/Bold minimalist professional cover letter.docx" --anon_strategy replace_redacted
+```
+
+### Duplicate detection
+
+**Find duplicate pages in OCR files:**
+```bash
+python cli_redact.py --task deduplicate --input_file example_data/example_outputs/doubled_output_joined.pdf_ocr_output.csv --duplicate_type pages --similarity_threshold 0.95
+```
+
+**Find duplicates at line level:**
+```bash
+python cli_redact.py --task deduplicate --input_file example_data/example_outputs/doubled_output_joined.pdf_ocr_output.csv --duplicate_type pages --similarity_threshold 0.95 --combine_pages False --min_word_count 3
+```
+
+**Find duplicate rows in tabular data:**
+```bash
+python cli_redact.py --task deduplicate --input_file example_data/Lambeth_2030-Our_Future_Our_Lambeth.pdf.csv --duplicate_type tabular --text_columns "text" --similarity_threshold 0.95
+```
+
+### AWS Textract operations
+
+**Submit document for analysis:**
+```bash
+python cli_redact.py --task textract --textract_action submit --input_file example_data/example_of_emails_sent_to_a_professor_before_applying.pdf
+```
+
+**Submit with signature extraction:**
+```bash
+python cli_redact.py --task textract --textract_action submit --input_file example_data/Partnership-Agreement-Toolkit_0_0.pdf --extract_signatures
+```
 
-We can see from the above that we have successfully removed a redaction box, changed labels, colours, and redaction box sizes.
+**Retrieve results by job ID:**
+```bash
+python cli_redact.py --task textract --textract_action retrieve --job_id 12345678-1234-1234-1234-123456789012
+```
+
+**List recent jobs:**
+```bash
+python cli_redact.py --task textract --textract_action list
+```
+
+### Common CLI options
+
+- `--task`: Choose between "redact", "deduplicate", or "textract"
+- `--input_file`: Path to input file(s)
+- `--output_dir`: Directory for output files (default: output/)
+- `--page_min` / `--page_max`: Process only specific page range
+- `--ocr_method`: Choose text extraction method
+- `--pii_detector`: Choose PII detection method
+- `--local_redact_entities`: Specify local entities to redact
+- `--allow_list_file` / `--deny_list_file`: Custom lists
+- `--redact_whole_page_file`: List of pages to redact completely
+- `--fuzzy_mistakes`: Number of spelling mistakes allowed in fuzzy matching
+- `--similarity_threshold`: Threshold for duplicate detection
+- `--anon_strategy`: Anonymization strategy for tabular data
+
+### Output files
+
+The CLI generates the same output files as the GUI:
+- `...redacted.pdf`: Final redacted document
+- `...redactions_for_review.pdf`: Document with redaction boxes for review
+- `...review_file.csv`: Detailed redaction information
+- `...ocr_results.csv`: Extracted text results
+- `..._textract.json`: AWS Textract results (if applicable)
+
+For more advanced options and configuration, refer to the help text by running:
+```bash
+python cli_redact.py --help
+```

tools/config.py

@@ -773,26 +773,32 @@ DIRECT_MODE_PII_DETECTOR = get_or_create_env_var(
 DIRECT_MODE_OCR_METHOD = get_or_create_env_var(
     "DIRECT_MODE_OCR_METHOD", "Local OCR"
 )  # OCR method for PDF/image processing
-DIRECT_MODE_PAGE_MIN = int(get_or_create_env_var(
-    "DIRECT_MODE_PAGE_MIN", str(DEFAULT_PAGE_MIN)
-))  # First page to process
-DIRECT_MODE_PAGE_MAX = int(get_or_create_env_var(
-    "DIRECT_MODE_PAGE_MAX", str(DEFAULT_PAGE_MAX)
-))  # Last page to process
-DIRECT_MODE_IMAGES_DPI = float(get_or_create_env_var(
-    "DIRECT_MODE_IMAGES_DPI", str(IMAGES_DPI)
-))  # DPI for image processing
+DIRECT_MODE_PAGE_MIN = int(
+    get_or_create_env_var("DIRECT_MODE_PAGE_MIN", str(DEFAULT_PAGE_MIN))
+)  # First page to process
+DIRECT_MODE_PAGE_MAX = int(
+    get_or_create_env_var("DIRECT_MODE_PAGE_MAX", str(DEFAULT_PAGE_MAX))
+)  # Last page to process
+DIRECT_MODE_IMAGES_DPI = float(
+    get_or_create_env_var("DIRECT_MODE_IMAGES_DPI", str(IMAGES_DPI))
+)  # DPI for image processing
 DIRECT_MODE_CHOSEN_LOCAL_OCR_MODEL = get_or_create_env_var(
     "DIRECT_MODE_CHOSEN_LOCAL_OCR_MODEL", CHOSEN_LOCAL_OCR_MODEL
 )  # Local OCR model choice
 DIRECT_MODE_PREPROCESS_LOCAL_OCR_IMAGES = convert_string_to_boolean(
-    get_or_create_env_var("DIRECT_MODE_PREPROCESS_LOCAL_OCR_IMAGES", str(PREPROCESS_LOCAL_OCR_IMAGES))
+    get_or_create_env_var(
+        "DIRECT_MODE_PREPROCESS_LOCAL_OCR_IMAGES", str(PREPROCESS_LOCAL_OCR_IMAGES)
+    )
 )  # Preprocess images before OCR
 DIRECT_MODE_COMPRESS_REDACTED_PDF = convert_string_to_boolean(
-    get_or_create_env_var("DIRECT_MODE_COMPRESS_REDACTED_PDF", str(COMPRESS_REDACTED_PDF))
+    get_or_create_env_var(
+        "DIRECT_MODE_COMPRESS_REDACTED_PDF", str(COMPRESS_REDACTED_PDF)
+    )
 )  # Compress redacted PDF
 DIRECT_MODE_RETURN_PDF_END_OF_REDACTION = convert_string_to_boolean(
-    get_or_create_env_var("DIRECT_MODE_RETURN_PDF_END_OF_REDACTION", str(RETURN_REDACTED_PDF))
+    get_or_create_env_var(
+        "DIRECT_MODE_RETURN_PDF_END_OF_REDACTION", str(RETURN_REDACTED_PDF)
+    )
 )  # Return PDF at end of redaction
 DIRECT_MODE_EXTRACT_FORMS = convert_string_to_boolean(
     get_or_create_env_var("DIRECT_MODE_EXTRACT_FORMS", "False")
@@ -812,26 +818,36 @@ DIRECT_MODE_MATCH_FUZZY_WHOLE_PHRASE_BOOL = convert_string_to_boolean(
 DIRECT_MODE_ANON_STRATEGY = get_or_create_env_var(
     "DIRECT_MODE_ANON_STRATEGY", DEFAULT_TABULAR_ANONYMISATION_STRATEGY
 )  # Anonymisation strategy for tabular data
-DIRECT_MODE_FUZZY_MISTAKES = int(get_or_create_env_var(
-    "DIRECT_MODE_FUZZY_MISTAKES", str(DEFAULT_FUZZY_SPELLING_MISTAKES_NUM)
-))  # Number of fuzzy spelling mistakes allowed
-DIRECT_MODE_SIMILARITY_THRESHOLD = float(get_or_create_env_var(
-    "DIRECT_MODE_SIMILARITY_THRESHOLD", str(DEFAULT_DUPLICATE_DETECTION_THRESHOLD)
-))  # Similarity threshold for duplicate detection
-DIRECT_MODE_MIN_WORD_COUNT = int(get_or_create_env_var(
-    "DIRECT_MODE_MIN_WORD_COUNT", str(DEFAULT_MIN_WORD_COUNT)
-))  # Minimum word count for duplicate detection
-DIRECT_MODE_MIN_CONSECUTIVE_PAGES = int(get_or_create_env_var(
-    "DIRECT_MODE_MIN_CONSECUTIVE_PAGES", str(DEFAULT_MIN_CONSECUTIVE_PAGES)
-))  # Minimum consecutive pages for duplicate detection
+DIRECT_MODE_FUZZY_MISTAKES = int(
+    get_or_create_env_var(
+        "DIRECT_MODE_FUZZY_MISTAKES", str(DEFAULT_FUZZY_SPELLING_MISTAKES_NUM)
+    )
+)  # Number of fuzzy spelling mistakes allowed
+DIRECT_MODE_SIMILARITY_THRESHOLD = float(
+    get_or_create_env_var(
+        "DIRECT_MODE_SIMILARITY_THRESHOLD", str(DEFAULT_DUPLICATE_DETECTION_THRESHOLD)
+    )
+)  # Similarity threshold for duplicate detection
+DIRECT_MODE_MIN_WORD_COUNT = int(
+    get_or_create_env_var("DIRECT_MODE_MIN_WORD_COUNT", str(DEFAULT_MIN_WORD_COUNT))
+)  # Minimum word count for duplicate detection
+DIRECT_MODE_MIN_CONSECUTIVE_PAGES = int(
+    get_or_create_env_var(
+        "DIRECT_MODE_MIN_CONSECUTIVE_PAGES", str(DEFAULT_MIN_CONSECUTIVE_PAGES)
+    )
+)  # Minimum consecutive pages for duplicate detection
 DIRECT_MODE_GREEDY_MATCH = convert_string_to_boolean(
-    get_or_create_env_var("DIRECT_MODE_GREEDY_MATCH", str(USE_GREEDY_DUPLICATE_DETECTION))
+    get_or_create_env_var(
+        "DIRECT_MODE_GREEDY_MATCH", str(USE_GREEDY_DUPLICATE_DETECTION)
+    )
 )  # Use greedy matching for duplicate detection
 DIRECT_MODE_COMBINE_PAGES = convert_string_to_boolean(
     get_or_create_env_var("DIRECT_MODE_COMBINE_PAGES", str(DEFAULT_COMBINE_PAGES))
 )  # Combine pages for duplicate detection
 DIRECT_MODE_REMOVE_DUPLICATE_ROWS = convert_string_to_boolean(
-    get_or_create_env_var("DIRECT_MODE_REMOVE_DUPLICATE_ROWS", str(REMOVE_DUPLICATE_ROWS))
+    get_or_create_env_var(
+        "DIRECT_MODE_REMOVE_DUPLICATE_ROWS", str(REMOVE_DUPLICATE_ROWS)
+    )
 )  # Remove duplicate rows in tabular data
 
 # Textract Batch Operations Options
@@ -843,12 +859,12 @@ DIRECT_MODE_JOB_ID = get_or_create_env_var(
 )  # Job ID for Textract operations
 
 # Lambda-specific configuration options
-LAMBDA_POLL_INTERVAL = int(get_or_create_env_var(
-    "LAMBDA_POLL_INTERVAL", "30"
-))  # Polling interval in seconds for Textract job status
-LAMBDA_MAX_POLL_ATTEMPTS = int(get_or_create_env_var(
-    "LAMBDA_MAX_POLL_ATTEMPTS", "120"
-))  # Maximum number of polling attempts for Textract job completion
+LAMBDA_POLL_INTERVAL = int(
+    get_or_create_env_var("LAMBDA_POLL_INTERVAL", "30")
+)  # Polling interval in seconds for Textract job status
+LAMBDA_MAX_POLL_ATTEMPTS = int(
+    get_or_create_env_var("LAMBDA_MAX_POLL_ATTEMPTS", "120")
+)  # Maximum number of polling attempts for Textract job completion
 LAMBDA_PREPARE_IMAGES = convert_string_to_boolean(
     get_or_create_env_var("LAMBDA_PREPARE_IMAGES", "True")
 )  # Prepare images for OCR processing