Merge pull request #18 from seanpedrick-case/dev

Improved review efficiency, logging to DynamoDB, local OCR text extraction saves, bug fixes

.dockerignore

@@ -17,4 +17,6 @@ dist/*
 build_deps/*
 logs/*
 config/*
-user_guide/*
+user_guide/*
+cdk/*
+web/*

.gitignore

@@ -18,4 +18,6 @@ build_deps/*
 logs/*
 config/*
 doc_redaction_amplify_app/*
-user_guide/*
+user_guide/*
+cdk/*
+web/*

README.md

@@ -20,6 +20,12 @@ NOTE: The app is not 100% accurate, and it will miss some personal information.
 
 # USER GUIDE
 
+## Experiment with the test (public) version of the app
+You can test out many of the features described in this user guide at the [public test version of the app](https://huggingface.co/spaces/seanpedrickcase/document_redaction), which is free. AWS functions (e.g. Textract, Comprehend) are not enabled (unless you have valid API keys).
+
+## Chat over this user guide
+You can now [speak with a chat bot about this user guide](https://huggingface.co/spaces/seanpedrickcase/Light-PDF-Web-QA-Chatbot) (beta!)
+
 ## Table of contents
 
 - [Example data files](#example-data-files)
@@ -33,59 +39,102 @@ NOTE: The app is not 100% accurate, and it will miss some personal information.
     - [Redacting only specific pages](#redacting-only-specific-pages)
     - [Handwriting and signature redaction](#handwriting-and-signature-redaction)
 - [Reviewing and modifying suggested redactions](#reviewing-and-modifying-suggested-redactions)
+- [Redacting tabular data files (CSV/XLSX) or copy and pasted text](#redacting-tabular-data-files-xlsxcsv-or-copy-and-pasted-text)
 
 See the [advanced user guide here](#advanced-user-guide):
-- [Modifying and merging redaction review files](#modifying-and-merging-redaction-review-files)
-    - [Modifying existing redaction review files](#modifying-existing-redaction-review-files)
-    - [Merging existing redaction review files](#merging-existing-redaction-review-files)
+- [Merging redaction review files](#merging-redaction-review-files)
 - [Identifying and redacting duplicate pages](#identifying-and-redacting-duplicate-pages)
 - [Fuzzy search and redaction](#fuzzy-search-and-redaction)
 - [Export redactions to and import from Adobe Acrobat](#export-to-and-import-from-adobe)
     - [Exporting to Adobe Acrobat](#exporting-to-adobe-acrobat)
     - [Importing from Adobe Acrobat](#importing-from-adobe-acrobat)
+- [Using the AWS Textract document API](#using-the-aws-textract-document-api)
 - [Using AWS Textract and Comprehend when not running in an AWS environment](#using-aws-textract-and-comprehend-when-not-running-in-an-aws-environment)
+- [Modifying existing redaction review files](#modifying-existing-redaction-review-files)
 
 ## Example data files
 
-Please refer to these example files to follow this guide: 
+Please try these example files to follow along with this guide: 
 - [Example of files sent to a professor before applying](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/example_of_emails_sent_to_a_professor_before_applying.pdf)
 - [Example complaint letter (jpg)](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/example_complaint_letter.jpg)
 - [Partnership Agreement Toolkit (for signatures and more advanced usage)](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/Partnership-Agreement-Toolkit_0_0.pdf)
+- [Dummy case note data](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/combined_case_notes.csv)
 
 ## Basic redaction
 
-The document redaction app can detect personally-identifiable information (PII) in documents. Documents can be redacted directly, or suggested redactions can be reviewed and modified using a grapical user interface.
+The document redaction app can detect personally-identifiable information (PII) in documents. Documents can be redacted directly, or suggested redactions can be reviewed and modified using a grapical user interface. Basic document redaction can be performed quickly using the default options.
 
 Download the example PDFs above to your computer. Open up the redaction app with the link provided by email.
 
 ![Upload files](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/quick_start/file_upload_highlight.PNG)
 
-Click on the upload files area, and select the three different files (they should all be stored in the same folder if you want them to be redacted at the same time).
+### Upload files to the app
+
+The 'Redact PDFs/images tab' currently accepts PDFs and image files (JPG, PNG) for redaction. Click on the 'Drop files here or Click to Upload' area of the screen, and select one of the three different [example files](#example-data-files) (they should all be stored in the same folder if you want them to be redacted at the same time).
+
+### Text extraction
 
-First, select one of the three text extraction options below:
-- 'Local model - selectable text' - This will read text directly from PDFs that have selectable text to redact (using PikePDF). This is fine for most PDFs, but will find nothing if the PDF does not have selectable text, and it is not good for handwriting or signatures. If it encounters an image file, it will send it onto the second option below.
-- 'Local OCR model - PDFs without selectable text' - This option will use a simple Optical Character Recognition (OCR) model (Tesseract) to pull out text from a PDF/image that it 'sees'. This can handle most typed text in PDFs/images without selectable text, but struggles with handwriting/signatures. If you are interested in the latter, then you should use the third option if available.
-- 'AWS Textract service - all PDF types' - Only available for instances of the app running on AWS. AWS Textract is a service that performs OCR on documents within their secure service. This is a more advanced version of OCR compared to the local option, and carries a (relatively small) cost. Textract excels in complex documents based on images, or documents that contain a lot of handwriting and signatures.
+First, select one of the three text extraction options:
+- **'Local model - selectable text'** - This will read text directly from PDFs that have selectable text to redact (using PikePDF). This is fine for most PDFs, but will find nothing if the PDF does not have selectable text, and it is not good for handwriting or signatures. If it encounters an image file, it will send it onto the second option below.
+- **'Local OCR model - PDFs without selectable text'** - This option will use a simple Optical Character Recognition (OCR) model (Tesseract) to pull out text from a PDF/image that it 'sees'. This can handle most typed text in PDFs/images without selectable text, but struggles with handwriting/signatures. If you are interested in the latter, then you should use the third option if available.
+- **'AWS Textract service - all PDF types'** - Only available for instances of the app running on AWS. AWS Textract is a service that performs OCR on documents within their secure service. This is a more advanced version of OCR compared to the local option, and carries a (relatively small) cost. Textract excels in complex documents based on images, or documents that contain a lot of handwriting and signatures.
+
+### Optional - select signature extraction
+If you chose the AWS Textract service above, you can choose if you want handwriting and/or signatures redacted by default. Choosing signatures here will have a cost implication, as identifying signatures will cost ~£2.66 ($3.50) per 1,000 pages vs ~£1.14 ($1.50) per 1,000 pages without signature detection. 
+
+![AWS Textract handwriting and signature options](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/quick_start/textract_handwriting_signatures.PNG)
+
+### PII redaction method
 
 If you are running with the AWS service enabled, here you will also have a choice for PII redaction method:
-- 'Local' - This uses the spacy package to rapidly detect PII in extracted text. This method is often sufficient if you are just interested in redacting specific terms defined in a custom list. 
-- 'AWS Comprehend' - This method calls an AWS service to provide more accurate identification of PII in extracted text.
+- **'Only extract text - (no redaction)'** - If you are only interested in getting the text out of the document for further processing (e.g. to find duplicate pages, or to review text on the Review redactions page)
+- **'Local'** - This uses the spacy package to rapidly detect PII in extracted text. This method is often sufficient if you are just interested in redacting specific terms defined in a custom list. 
+- **'AWS Comprehend'** - This method calls an AWS service to provide more accurate identification of PII in extracted text.
+
+### Optional - costs and time estimation
+If the option is enabled (by your system admin, in the config file), you will see a cost and time estimate for the redaction process. 'Existing Textract output file found' will be checked automatically if previous Textract text extraction files exist in the output folder, or have been [previously uploaded by the user](#aws-textract-outputs) (saving time and money for redaction). 
+
+![Cost and time estimation](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/quick_start/costs_and_time.PNG)
+
+### Optional - cost code selection
+If the option is enabled (by your system admin, in the config file), you may be prompted to select a cost code before continuing with the redaction task.
+
+![Cost code selection](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/quick_start/cost_code_selection.PNG)
+
+The relevant cost code can be found either by: 1. Using the search bar above the data table to find relevant cost codes, then clicking on the relevant row, or 2. typing it directly into the dropdown to the right, where it should filter as you type.
+
+### Optional - Submit whole documents to Textract API
+If this option is enabled (by your system admin, in the config file), you will have the option to submit whole documents in quick succession to the AWS Textract service to get extracted text outputs quickly (faster than using the 'Redact document' process described here). This feature is described in more detail in the [advanced user guide](#using-the-aws-textract-document-api).
+
+![Textract document API](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/quick_start/textract_document_api.PNG)
+
+### Redact the document
+
+Click 'Redact document'. After loading in the document, the app should be able to process about 30 pages per minute (depending on redaction methods chose above). When ready, you should see a message saying that processing is complete, with output files appearing in the bottom right.
 
-Hit 'Redact document'. After loading in the document, the app should be able to process about 30 pages per minute (depending on redaction methods chose above). When ready, you should see a message saying that processing is complete, with output files appearing in the bottom right.
+### Redaction outputs
 
 ![Redaction outputs](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/quick_start/redaction_outputs.PNG)
 
-- '...redacted.pdf' files contain the original pdf with suggested redacted text deleted and replaced by a black box on top of the document.
-- '...ocr_results.csv' files contain the line-by-line text outputs from the entire document. This file can be useful for later searching through for any terms of interest in the document (e.g. using Excel or a similar program).
-- '...review_file.csv' files are the review files that contain details and locations of all of the suggested redactions in the document. This file is key to the [review process](#reviewing-and-modifying-suggested-redactions), and should be downloaded to use later for this.
+- **'...redacted.pdf'** files contain the original pdf with suggested redacted text deleted and replaced by a black box on top of the document.
+- **'...ocr_results.csv'** files contain the line-by-line text outputs from the entire document. This file can be useful for later searching through for any terms of interest in the document (e.g. using Excel or a similar program).
+- **'...review_file.csv'** files are the review files that contain details and locations of all of the suggested redactions in the document. This file is key to the [review process](#reviewing-and-modifying-suggested-redactions), and should be downloaded to use later for this.
 
-Additional outputs are available under the 'Redaction settings' tab. Scroll to the bottom and you should see more files: 
+### Additional AWS Textract / local OCR outputs
 
-![Additional processing outputs](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/quick_start/redaction_additional_outputs.PNG)
+If you have used the AWS Textract option for extracting text, you may also see a '..._textract.json' file. This file contains all the relevant extracted text information that comes from the AWS Textract service. You can keep this file and upload it at a later date alongside your input document, which will enable you to skip calling AWS Textract every single time you want to do a redaction task, as follows:
 
-- '...review_file.json' is the same file as the review file above, but in .json format.
-- '...decision_process_output.csv' is also similar to the review file above, with a few more details on the location and scores of identified PII in the document.
-- If you are using AWS Textract, you should also get a .json file with the Textract outputs. It could be useful to retain this document to avoid having to repeatedly analyse the same document in future (this .json file can be uploaded into the app on the first redaction tab to load into local memory before redaction).
+![Document upload alongside Textract](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/quick_start/document_upload_with_textract.PNG)
+
+Similarly, if you have used the 'Local OCR method' to extract text, you may see a '..._ocr_results_with_words.json' file. This file works in the same way as the AWS Textract .json results described above, and can be uploaded alongside an input document to save time on text extraction in future in the same way.
+
+### Downloading output files from previous redaction tasks
+
+If you are logged in via AWS Cognito and you lose your app page for some reason (e.g. from a crash, reloading), it is possible recover your previous output files, provided the server has not been shut down since you redacted the document. Go to 'Redaction settings', then scroll to the bottom to see 'View all output files from this session'.
+
+![View all output files](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/quick_start/view_all_output_files.PNG)
+
+### Basic redaction summary
 
 We have covered redacting documents with the default redaction options. The '...redacted.pdf' file output may be enough for your purposes. But it is very likely that you will need to customise your redaction options, which we will cover below. 
 
@@ -126,6 +175,16 @@ There may be full pages in a document that you want to redact. The app also prov
 
 Using the above approaches to allow, deny, and full page redaction lists will give you an output [like this](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/allow_list/Partnership-Agreement-Toolkit_0_0_redacted.pdf).
 
+#### Adding to the loaded allow, deny, and whole page lists in-app
+
+If you open the accordion below the allow list options called 'Manually modify custom allow...', you should be able to see a few tables with options to add new rows:
+
+![Manually modify allow or deny list](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/allow_list/manually_modify.PNG)
+
+If the table is empty, you can add a new entry, you can add a new row by clicking on the '+' item below each table header. If there is existing data, you may need to click on the three dots to the right and select 'Add row below'. Type the item you wish to keep/remove in the cell, and then (important) press enter to add this new item to the allow/deny/whole page list. Your output tables should look something like below.
+
+![Manually modify allow or deny list filled](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/allow_list/manually_modify_filled.PNG)
+
 ### Redacting additional types of personal information
 
 You may want to redact additional types of information beyond the defaults, or you may not be interested in default suggested entity types. There are dates in the example complaint letter. Say we wanted to redact those dates also?
@@ -146,7 +205,9 @@ Say also we are only interested in redacting page 1 of the loaded documents. On
 
 ## Handwriting and signature redaction
 
-The file [Partnership Agreement Toolkit (for signatures and more advanced usage)](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/Partnership-Agreement-Toolkit_0_0.pdf) is provided as an example document to test AWS Textract + redaction with a document that has signatures in. If you have access to AWS Textract in the app, try removing all entity types from redaction on the Redaction settings and clicking the big X to the right of 'Entities to redact'. Ensure that handwriting and signatures are enabled for redaction on the Redaction Settings tab(enabled by default):
+The file [Partnership Agreement Toolkit (for signatures and more advanced usage)](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/Partnership-Agreement-Toolkit_0_0.pdf) is provided as an example document to test AWS Textract + redaction with a document that has signatures in. If you have access to AWS Textract in the app, try removing all entity types from redaction on the Redaction settings and clicking the big X to the right of 'Entities to redact'. 
+
+To ensure that handwriting and signatures are enabled (enabled by default), on the front screen go the 'AWS Textract signature detection' to enable/disable the following options :
 
 ![Handwriting and signatures](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/review_redactions/textract_handwriting_signatures.PNG)
 
@@ -156,72 +217,169 @@ The outputs should show handwriting/signatures redacted (see pages 5 - 7), which
 
 ## Reviewing and modifying suggested redactions
 
-Quite often there are certain terms suggested for redaction by the model that don't match quite what you intended. The app allows you to review and modify suggested redactions for the last file redacted. Refresh your browser tab. On the first tab 'PDFs/images' upload the ['Example of files sent to a professor before applying.pdf'](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/example_of_emails_sent_to_a_professor_before_applying.pdf) file. Let's stick with the 'Local model - selectable text' option, and click 'Redact document'. Once the outputs are created, go to the 'Review redactions' tab.
+Sometimes the app will suggest redactions that are incorrect, or will miss personal information entirely. The app allows you to review and modify suggested redactions to compensate for this. You can do this on the 'Review redactions' tab.
+
+We will go through ways to review suggested redactions with an example.On the first tab 'PDFs/images' upload the ['Example of files sent to a professor before applying.pdf'](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/example_of_emails_sent_to_a_professor_before_applying.pdf) file. Let's stick with the 'Local model - selectable text' option, and click 'Redact document'. Once the outputs are created, go to the 'Review redactions' tab.
 
-On this tab you have a visual interface that allows you to inspect and modify redactions suggested by the app. 
+On the 'Review redactions' tab you have a visual interface that allows you to inspect and modify redactions suggested by the app. There are quite a few options to look at, so we'll go from top to bottom.
 
 ![Review redactions](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/review_redactions/review_redactions.PNG)
 
+### Uploading documents for review
+
+The top area has a file upload area where you can upload original, unredacted PDFs, alongside the '..._review_file.csv' that is produced by the redaction process. Once you have uploaded these two files, click the 'Review PDF...' button to load in the files for review. This will allow you to visualise and modify the suggested redactions using the interface below.
+
+Optionally, you can also upload one of the '..._ocr_output.csv' files here that comes out of a redaction task, so that you can navigate the extracted text from the document.
+
+![Search extracted text](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/review_redactions/search_extracted_text.PNG)
+
+You can upload the three review files in the box (unredacted document, '..._review_file.csv' and '..._ocr_output.csv' file) before clicking 'Review PDF...', as in the image below:
+
+![Upload three files for review](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/review_redactions/upload_three_files.PNG)
+
+**NOTE:** ensure you upload the ***unredacted*** document here and not the redacted version, otherwise you will be checking over a document that already has redaction boxes applied!
+
+### Page navigation
+
 You can change the page viewed either by clicking 'Previous page' or 'Next page', or by typing a specific page number in the 'Current page' box and pressing Enter on your keyboard. Each time you switch page, it will save redactions you have made on the page you are moving from, so you will not lose changes you have made.
 
-On your selected page, each redaction is highlighted with a box next to its suggested entity type. By default the interface allows you to modify existing redaction boxes. Click and hold on an existing box to move it. Click on one of the small boxes at the edges to change the size of the box. To delete a box, click on it to highlight it, then press delete on your keyboard. Alternatively, double click on a box and click 'Remove' on the box that appears.
+You can also navigate to different pages by clicking on rows in the tables under 'Search suggested redactions' to the right, or 'search all extracted text' (if enabled) beneath that.
 
-To change to 'add new redactions' mode, scroll to the bottom of the page. Click on the box icon, and your cursor will change into a crosshair. Now you can add new redaction boxes where you wish.
+### The document viewer pane
+
+On the selected page, each redaction is highlighted with a box next to its suggested redaction label (e.g. person, email).
+
+![Document view pane](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/review_redactions/document_viewer_pane.PNG)
+
+There are a number of different options to add and modify redaction boxes and page on the document viewer pane. To zoom in and out of the page, use your mouse wheel. To move around the page while zoomed, you need to be in modify mode. Scroll to the bottom of the document viewer to see the relevant controls. You should see a box icon, a hand icon, and two arrows pointing counter-clockwise and clockwise.
 
 ![Change redaction mode](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/review_redactions/change_review_mode.PNG)
 
-On the right of the screen there is a dropdown and table where you can filter to entity types that have been found throughout the document. You can choose a specific entity type to see which pages the entity is present on. If you want to go to the page specified in the table, you can click on a cell in the table and the review page will be changed to that page.
+Click on the hand icon to go into modify mode. When you click and hold on the document viewer, This will allow you to move around the page when zoomed in. To rotate the page, you can click on either of the round arrow buttons to turn in that direction. 
+
+**NOTE:** When you switch page, the viewer will stay in your selected orientation, so if it looks strange, just rotate the page again and hopefully it will look correct!
+
+#### Modify existing redactions (hand icon)
+
+After clicking on the hand icon, the interface allows you to modify existing redaction boxes. When in this mode, you can click and hold on an existing box to move it. 
+
+![Modify existing redaction box](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/review_redactions/modify_existing_redaction_box.PNG)
+
+Click on one of the small boxes at the edges to change the size of the box. To delete a box, click on it to highlight it, then press delete on your keyboard. Alternatively, double click on a box and click 'Remove' on the box that appears.
+
+![Remove existing redaction box](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/review_redactions/existing_redaction_box_remove.PNG)
+
+#### Add new redaction boxes (box icon)
+
+To change to 'add redaction boxes' mode, scroll to the bottom of the page. Click on the box icon, and your cursor will change into a crosshair. Now you can add new redaction boxes where you wish. A popup will appear when you create a new box so you can select a label and colour for the new box.
+
+#### 'Locking in' new redaction box format
 
-![Change redaction mode](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/review_redactions/list_find_labels.PNG)
+It is possible to lock in a chosen format for new redaction boxes so that you don't have the popup appearing each time. When you make a new box, select the options for your 'locked' format, and then click on the lock icon on the left side of the popup, which should turn blue.
 
-Note that the table currently only shows entity types, and not specific found text. So for instance if you provide a list of specific terms to redact in the [deny list](#deny-list-example), they will all be labelled just as 'CUSTOM'. A feature to include in the near term will include being able to view specific redacted text in this table to get a better sense of the PII entities found.
+![Lock redaction box format](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/review_redactions/new_redaction_box_lock_mode.PNG)
 
-Once you happy with your modified changes throughout the document, click 'Apply revised redactions' at the top of the page. The app will then run through all the pages in the document to update the redactions, and will output a modified PDF file. The modified PDF will appear at the top of the page in the file area. It will also output a revised '...review_file.csv' that you can then use for future review tasks.
+You can now add new redaction boxes without a popup appearing. If you want to change or 'unlock' the your chosen box format, you can click on the new icon that has appeared at the bottom of the document viewer pane that looks a little like a gift tag. You can then change the defaults, or click on the lock icon again to 'unlock' the new box format - then popups will appear again each time you create a new box.
+
+![Change or unlock redaction box format](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/review_redactions/change_review_mode_with_lock.PNG)
+
+### Apply redactions to PDF and Save changes on current page
+
+Once you have reviewed all the redactions in your document and you are happy with the outputs, you can click 'Apply revised redactions to PDF' to create a new '_redacted.pdf' output alongside a new '_review_file.csv' output.
+
+If you are working on a page and haven't saved for a while, you can click 'Save changes on current page to file' to ensure that they are saved to an updated 'review_file.csv' output.
 
 ![Review modified outputs](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/review_redactions/review_mod_outputs.PNG)
 
-Any feedback or comments on the app, please get in touch!
+### Selecting and removing redaction boxes using the 'Search suggested redactions' table
 
-# ADVANCED USER GUIDE
+The table shows a list of all the suggested redactions in the document alongside the page, label, and text (if available).
 
-This advanced user guide will go over some of the features recently added to the app, including: modifying and merging redaction review files, identifying and redacting duplicate pages across multiple PDFs, 'fuzzy' search and redact, and exporting redactions to Adobe Acrobat.
+![Search suggested redaction area](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/review_redactions/list_find_labels.PNG)
 
-## Table of contents
+If you click on one of the rows in this table, you will be taken to the page of the redaction. Clicking on a redaction row on the same page *should* change the colour of redaction box to blue to help you locate it in the document viewer (just in app, not in redaction output PDFs).
 
-- [Modifying and merging redaction review files](#modifying-and-merging-redaction-review-files)
-    - [Modifying existing redaction review files](#modifying-existing-redaction-review-files)
-    - [Merging existing redaction review files](#merging-existing-redaction-review-files)
-- [Identifying and redacting duplicate pages](#identifying-and-redacting-duplicate-pages)
-- [Fuzzy search and redaction](#fuzzy-search-and-redaction)
-- [Export redactions to and import from Adobe Acrobat](#export-to-and-import-from-adobe)
-    - [Exporting to Adobe Acrobat](#exporting-to-adobe-acrobat)
-    - [Importing from Adobe Acrobat](#importing-from-adobe-acrobat)
+![Search suggested redaction area](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/review_redactions/review_row_highlight.PNG)
 
+You can choose a specific entity type to see which pages the entity is present on. If you want to go to the page specified in the table, you can click on a cell in the table and the review page will be changed to that page.
 
-## Modifying and merging redaction review files
+To filter the 'Search suggested redactions' table you can:
+1. Click on one of the dropdowns (Redaction category, Page, Text), and select an option, or
+2. Write text in the 'Filter' box just above the table. Click the blue box to apply the filter to the table.
 
-You can find the folder containing the files discussed in this section [here](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/merge_review_files/).
+Once you have filtered the table, you have a few options underneath on what you can do with the filtered rows:
 
-As well as serving as inputs to the document redaction app's review function, the 'review_file.csv' output can be modified outside of the app, and also merged with others from multiple redaction attempts on the same file. This gives you the flexibility to change redaction details outside of the app.
+- Click the 'Exclude specific row from redactions' button to remove only the redaction from the last row you clicked on from the document. 
+- Click the 'Exclude all items in table from redactions' button to remove all redactions visible in the table from the document. **Important:** ensure that you have clicked the blue tick icon next to the search box before doing this, or you will remove all redactions from the document. If you do end up doing this, click the 'Undo last element removal' button below to restore the redactions.
 
-### Modifying existing redaction review files
-If you open up a 'review_file' csv output using a spreadsheet software program such as Microsoft Excel you can easily modify redaction properties. Open the file '[Partnership-Agreement-Toolkit_0_0_redacted.pdf_review_file_local.csv](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/merge_review_files/Partnership-Agreement-Toolkit_0_0.pdf_review_file_local.csv)', and you should see a spreadshet with just four suggested redactions (see below). The following instructions are for using Excel.
+**NOTE**: After excluding redactions using either of the above options, click the 'Reset filters' button below to ensure that the dropdowns and table return to seeing all remaining redactions in the document.
 
-![Review file before](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/merge_review_files/img/review_file_before.PNG)
+If you made a mistake, click the 'Undo last element removal' button to restore the Search suggested redactions table to its previous state (can only undo the last action).
 
-The first thing we can do is remove the first row - 'et' is suggested as a person, but is obviously not a genuine instance of personal information. Right click on the row number and select delete on this menu. Next, let's imagine that what the app identified as a 'phone number' was in fact another type of number and so we wanted to change the label. Simply click on the relevant label cells, let's change it to 'SECURITY_NUMBER'. You could also use 'Finad & Select' -> 'Replace' from the top ribbon menu if you wanted to change a number of labels simultaneously.
+### Navigating through the document using the 'Search all extracted text' 
 
-How about we wanted to change the colour of the 'email address' entry on the redaction review tab of the redaction app? The colours in a review file are based on an RGB scale with three numbers ranging from 0-255. [You can find suitable colours here](https://rgbcolorpicker.com). Using this scale, if I wanted my review box to be pure blue, I can change the cell value to (0,0,255).
+The 'search all extracted text' table will contain text if you have just redacted a document, or if you have uploaded a '..._ocr_output.csv' file alongside a document file and review file on the Review redactions tab as [described above](#uploading-documents-for-review). 
 
-Imagine that a redaction box was slightly too small, and I didn't want to use the in-app options to change the size. In the review file csv, we can modify e.g. the ymin and ymax values for any box to increase the extent of the redaction box. For the 'email address' entry, let's decrease ymin by 5, and increase ymax by 5.
+You can navigate through the document using this table. When you click on a row, the Document viewer pane to the left will change to the selected page. 
 
-I have saved an output file following the above steps as '[Partnership-Agreement-Toolkit_0_0_redacted.pdf_review_file_local_mod.csv](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/merge_review_files/outputs/Partnership-Agreement-Toolkit_0_0.pdf_review_file_local_mod.csv)' in the same folder that the original was found. Let's upload this file to the app along with the original pdf to see how the redactions look now.
+![Search suggested redaction area](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/review_redactions/select_extracted_text.PNG)
 
-![Review file after modification](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/merge_review_files/img/partnership_redactions_after.PNG)
+You can search through the extracted text by using the search bar just above the table, which should filter as you type. To apply the filter and 'cut' the table, click on the blue tick inside the box next to your search term. To return the table to its original content, click the button below the table 'Reset OCR output table filter'.
+
+![Search suggested redaction area](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/review_redactions/search_extracted_text.PNG)
+
+## Redacting tabular data files (XLSX/CSV) or copy and pasted text
+
+### Tabular data files (XLSX/CSV)
+
+The app can be used to redact tabular data files such as xlsx or csv files. For this to work properly, your data file needs to be in a simple table format, with a single table starting from the first cell (A1), and no other information in the sheet. Similarly for .xlsx files, each sheet in the file that you want to redact should be in this simple format.
+
+To demonstrate this, we can use [the example csv file 'combined_case_notes.csv'](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/combined_case_notes.csv), which is a small dataset of dummy social care case notes. Go to the 'Open text or Excel/csv files' tab. Drop the file into the upload area. After the file is loaded, you should see the suggested columns for redaction in the box underneath. You can select and deselect columns to redact as you wish from this list.
+
+![csv upload](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/tabular_files/file_upload_csv_columns.PNG)
+
+If you were instead to upload an xlsx file, you would see also a list of all the sheets in the xlsx file that can be redacted. The 'Select columns' area underneath will suggest a list of all columns in the file across all sheets.
+
+![xlsx upload](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/tabular_files/file_upload_xlsx_columns.PNG)
+
+Once you have chosen your input file and sheets/columns to redact, you can choose the redaction method. 'Local' will use the same local model as used for documents on the first tab. 'AWS Comprehend' will give better results, at a slight cost.
+
+When you click Redact text/data files, you will see the progress of the redaction task by file and sheet, and you will receive a csv output with the redacted data.
+
+### Choosing output anonymisation format
+You can also choose the anonymisation format of your output results.  Open the tab 'Anonymisation output format' to see the options. By default, any detected PII will be replaced with the word 'REDACTED' in the cell. You can choose one of the following options as the form of replacement for the redacted text:
+- replace with 'REDACTED': Replaced by the word 'REDACTED' (default)
+- replace with <ENTITY_NAME>: Replaced by e.g. 'PERSON' for people, 'EMAIL_ADDRESS' for emails etc.
+- redact completely: Text is removed completely and replaced by nothing.
+- hash: Replaced by a unique long ID code that is consistent with entity text. I.e. a particular name will always have the same ID code.
+- mask: Replace with stars '*'.
+
+### Redacting copy and pasted text
+You can also write open text into an input box and redact that using the same methods as described above. To do this, write or paste text into the 'Enter open text' box that appears when you open the 'Redact open text' tab. Then select a redaction method, and an anonymisation output format as described above. The redacted text will be printed in the output textbox, and will also be saved to a simple csv file in the output file box.  
+
+![Text analysis output](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/tabular_files/text_anonymisation_outputs.PNG)
+
+### Redaction log outputs
+A list of the suggested redaction outputs from the tabular data / open text data redaction is available on the Redaction settings page under 'Log file outputs'.
+
+# ADVANCED USER GUIDE
+
+This advanced user guide will go over some of the features recently added to the app, including: modifying and merging redaction review files, identifying and redacting duplicate pages across multiple PDFs, 'fuzzy' search and redact, and exporting redactions to Adobe Acrobat.
+
+## Table of contents
+
+- [Merging redaction review files](#merging-redaction-review-files)
+- [Identifying and redacting duplicate pages](#identifying-and-redacting-duplicate-pages)
+- [Fuzzy search and redaction](#fuzzy-search-and-redaction)
+- [Export redactions to and import from Adobe Acrobat](#export-to-and-import-from-adobe)
+    - [Exporting to Adobe Acrobat](#exporting-to-adobe-acrobat)
+    - [Importing from Adobe Acrobat](#importing-from-adobe-acrobat)
+- [Using the AWS Textract document API](#using-the-aws-textract-document-api)
+- [Using AWS Textract and Comprehend when not running in an AWS environment](#using-aws-textract-and-comprehend-when-not-running-in-an-aws-environment)
+- [Modifying existing redaction review files](#modifying-existing-redaction-review-files)
 
-We can see from the above that we have successfully removed a redaction box, changed labels, colours, and redaction box sizes.
 
-### Merging existing redaction review files
+## Merging redaction review files
 
 Say you have run multiple redaction tasks on the same document, and you want to merge all of these redactions together. You could do this in your spreadsheet editor, but this could be fiddly especially if dealing with multiple review files or large numbers of redactions. The app has a feature to combine multiple review files together to create a 'merged' review file.
 
@@ -303,6 +461,30 @@ When you click the 'convert .xfdf comment file to review_file.csv' button, the a
 
 ![Outputs from Adobe import](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/export_to_adobe/img/import_from_adobe_interface_outputs.PNG)
 
+## Using the AWS Textract document API
+
+This option can be enabled by your system admin, in the config file ('SHOW_WHOLE_DOCUMENT_TEXTRACT_CALL_OPTIONS' environment variable, and subsequent variables). Using this, you will have the option to submit whole documents in quick succession to the AWS Textract service to get extracted text outputs quickly (faster than using the 'Redact document' process described here).
+
+### Starting a new Textract API job
+
+To use this feature, first upload a document file in the file input box [in the usual way](#upload-files-to-the-app) on the first tab of the app. Under AWS Textract signature detection you can select whether or not you would like to analyse signatures or not (with a [cost implication](#optional---select-signature-extraction)).
+
+Then, open the section under the heading 'Submit whole document to AWS Textract API...'.
+
+![Textract document API menu](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/quick_start/textract_document_api.PNG)
+
+Click 'Analyse document with AWS Textract API call'. After a few seconds, the job should be submitted to the AWS Textract service. The box 'Job ID to check status' should now have an ID filled in. If it is not already filled with previous jobs (up to seven days old), the table should have a row added with details of the new API job.
+
+Click the button underneath, 'Check status of Textract job and download', to see progress on the job. Processing will continue in the background until the job is ready, so it is worth periodically clicking this button to see if the outputs are ready. In testing, and as a rough estimate, it seems like this process takes about five seconds per page. However, this has not been tested with very large documents. Once ready, the '_textract.json' output should appear below.
+
+### Textract API job outputs
+
+The '_textract.json' output can be used to speed up further redaction tasks as [described previously](#optional---costs-and-time-estimation), the 'Existing Textract output file found' flag should now be ticked.
+
+![Textract document API initial ouputs](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/textract_api/textract_api_initial_outputs.PNG)
+
+You can now easily get the '..._ocr_output.csv' redaction output based on this '_textract.json' (described in [Redaction outputs](#redaction-outputs)) by clicking on the button 'Convert Textract job outputs to OCR results'. You can now use this file e.g. for [identifying duplicate pages](#identifying-and-redacting-duplicate-pages), or for redaction review.
+
 ## Using AWS Textract and Comprehend when not running in an AWS environment
 
 AWS Textract and Comprehend give much better results for text extraction and document redaction than the local model options in the app. The most secure way to access them in the Redaction app is to run the app in a secure AWS environment with relevant permissions. Alternatively, you could run the app on your own system while logged in to AWS SSO with relevant permissions.
@@ -322,4 +504,26 @@ AWS_SECRET_KEY= your-secret-key
 
 The app should then pick up these keys when trying to access the AWS Textract and Comprehend services during redaction.
 
-Again, a lot can potentially go wrong with AWS solutions that are insecure, so before trying the above please consult with your AWS and data security teams.
+Again, a lot can potentially go wrong with AWS solutions that are insecure, so before trying the above please consult with your AWS and data security teams.
+
+## Modifying existing redaction review files
+
+You can find the folder containing the files discussed in this section [here](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/merge_review_files/).
+
+As well as serving as inputs to the document redaction app's review function, the 'review_file.csv' output can be modified outside of the app, and also merged with others from multiple redaction attempts on the same file. This gives you the flexibility to change redaction details outside of the app.
+
+If you open up a 'review_file' csv output using a spreadsheet software program such as Microsoft Excel you can easily modify redaction properties. Open the file '[Partnership-Agreement-Toolkit_0_0_redacted.pdf_review_file_local.csv](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/merge_review_files/Partnership-Agreement-Toolkit_0_0.pdf_review_file_local.csv)', and you should see a spreadshet with just four suggested redactions (see below). The following instructions are for using Excel.
+
+![Review file before](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/merge_review_files/img/review_file_before.PNG)
+
+The first thing we can do is remove the first row - 'et' is suggested as a person, but is obviously not a genuine instance of personal information. Right click on the row number and select delete on this menu. Next, let's imagine that what the app identified as a 'phone number' was in fact another type of number and so we wanted to change the label. Simply click on the relevant label cells, let's change it to 'SECURITY_NUMBER'. You could also use 'Finad & Select' -> 'Replace' from the top ribbon menu if you wanted to change a number of labels simultaneously.
+
+How about we wanted to change the colour of the 'email address' entry on the redaction review tab of the redaction app? The colours in a review file are based on an RGB scale with three numbers ranging from 0-255. [You can find suitable colours here](https://rgbcolorpicker.com). Using this scale, if I wanted my review box to be pure blue, I can change the cell value to (0,0,255).
+
+Imagine that a redaction box was slightly too small, and I didn't want to use the in-app options to change the size. In the review file csv, we can modify e.g. the ymin and ymax values for any box to increase the extent of the redaction box. For the 'email address' entry, let's decrease ymin by 5, and increase ymax by 5.
+
+I have saved an output file following the above steps as '[Partnership-Agreement-Toolkit_0_0_redacted.pdf_review_file_local_mod.csv](https://github.com/seanpedrick-case/document_redaction_examples/blob/main/merge_review_files/outputs/Partnership-Agreement-Toolkit_0_0.pdf_review_file_local_mod.csv)' in the same folder that the original was found. Let's upload this file to the app along with the original pdf to see how the redactions look now.
+
+![Review file after modification](https://raw.githubusercontent.com/seanpedrick-case/document_redaction_examples/main/merge_review_files/img/partnership_redactions_after.PNG)
+
+We can see from the above that we have successfully removed a redaction box, changed labels, colours, and redaction box sizes.

app.py

@@ -4,11 +4,11 @@ import pandas as pd
 import gradio as gr
 from gradio_image_annotation import image_annotator
 
-from tools.config import OUTPUT_FOLDER, INPUT_FOLDER, RUN_DIRECT_MODE, MAX_QUEUE_SIZE, DEFAULT_CONCURRENCY_LIMIT, MAX_FILE_SIZE, GRADIO_SERVER_PORT, ROOT_PATH, GET_DEFAULT_ALLOW_LIST, ALLOW_LIST_PATH, S3_ALLOW_LIST_PATH, FEEDBACK_LOGS_FOLDER, ACCESS_LOGS_FOLDER, USAGE_LOGS_FOLDER, TESSERACT_FOLDER, POPPLER_FOLDER, REDACTION_LANGUAGE, GET_COST_CODES, COST_CODES_PATH, S3_COST_CODES_PATH, ENFORCE_COST_CODES, DISPLAY_FILE_NAMES_IN_LOGS, SHOW_COSTS, RUN_AWS_FUNCTIONS, DOCUMENT_REDACTION_BUCKET, SHOW_BULK_TEXTRACT_CALL_OPTIONS, TEXTRACT_BULK_ANALYSIS_BUCKET, TEXTRACT_BULK_ANALYSIS_INPUT_SUBFOLDER, TEXTRACT_BULK_ANALYSIS_OUTPUT_SUBFOLDER, SESSION_OUTPUT_FOLDER, LOAD_PREVIOUS_TEXTRACT_JOBS_S3, TEXTRACT_JOBS_S3_LOC, TEXTRACT_JOBS_LOCAL_LOC, HOST_NAME, DEFAULT_COST_CODE, OUTPUT_COST_CODES_PATH, OUTPUT_ALLOW_LIST_PATH
-from tools.helper_functions import put_columns_in_df, get_connection_params, reveal_feedback_buttons, custom_regex_load, reset_state_vars, load_in_default_allow_list, tesseract_ocr_option, text_ocr_option, textract_option, local_pii_detector, aws_pii_detector, no_redaction_option, reset_review_vars, merge_csv_files, load_all_output_files, update_dataframe, check_for_existing_textract_file, load_in_default_cost_codes, enforce_cost_codes, calculate_aws_costs, calculate_time_taken, reset_base_dataframe, reset_ocr_base_dataframe, update_cost_code_dataframe_from_dropdown_select
-from tools.aws_functions import upload_file_to_s3, download_file_from_s3
+from tools.config import OUTPUT_FOLDER, INPUT_FOLDER, RUN_DIRECT_MODE, MAX_QUEUE_SIZE, DEFAULT_CONCURRENCY_LIMIT, MAX_FILE_SIZE, GRADIO_SERVER_PORT, ROOT_PATH, GET_DEFAULT_ALLOW_LIST, ALLOW_LIST_PATH, S3_ALLOW_LIST_PATH, FEEDBACK_LOGS_FOLDER, ACCESS_LOGS_FOLDER, USAGE_LOGS_FOLDER, TESSERACT_FOLDER, POPPLER_FOLDER, REDACTION_LANGUAGE, GET_COST_CODES, COST_CODES_PATH, S3_COST_CODES_PATH, ENFORCE_COST_CODES, DISPLAY_FILE_NAMES_IN_LOGS, SHOW_COSTS, RUN_AWS_FUNCTIONS, DOCUMENT_REDACTION_BUCKET, SHOW_WHOLE_DOCUMENT_TEXTRACT_CALL_OPTIONS, TEXTRACT_WHOLE_DOCUMENT_ANALYSIS_BUCKET, TEXTRACT_WHOLE_DOCUMENT_ANALYSIS_INPUT_SUBFOLDER, TEXTRACT_WHOLE_DOCUMENT_ANALYSIS_OUTPUT_SUBFOLDER, SESSION_OUTPUT_FOLDER, LOAD_PREVIOUS_TEXTRACT_JOBS_S3, TEXTRACT_JOBS_S3_LOC, TEXTRACT_JOBS_LOCAL_LOC, HOST_NAME, DEFAULT_COST_CODE, OUTPUT_COST_CODES_PATH, OUTPUT_ALLOW_LIST_PATH, COGNITO_AUTH, SAVE_LOGS_TO_CSV, SAVE_LOGS_TO_DYNAMODB, ACCESS_LOG_DYNAMODB_TABLE_NAME, DYNAMODB_ACCESS_LOG_HEADERS, CSV_ACCESS_LOG_HEADERS, FEEDBACK_LOG_DYNAMODB_TABLE_NAME, DYNAMODB_FEEDBACK_LOG_HEADERS, CSV_FEEDBACK_LOG_HEADERS, USAGE_LOG_DYNAMODB_TABLE_NAME, DYNAMODB_USAGE_LOG_HEADERS, CSV_USAGE_LOG_HEADERS
+from tools.helper_functions import put_columns_in_df, get_connection_params, reveal_feedback_buttons, custom_regex_load, reset_state_vars, load_in_default_allow_list, tesseract_ocr_option, text_ocr_option, textract_option, local_pii_detector, aws_pii_detector, no_redaction_option, reset_review_vars, merge_csv_files, load_all_output_files, update_dataframe, check_for_existing_textract_file, load_in_default_cost_codes, enforce_cost_codes, calculate_aws_costs, calculate_time_taken, reset_base_dataframe, reset_ocr_base_dataframe, update_cost_code_dataframe_from_dropdown_select, check_for_existing_local_ocr_file, reset_data_vars, reset_aws_call_vars
+from tools.aws_functions import upload_file_to_s3, download_file_from_s3, upload_log_file_to_s3
 from tools.file_redaction import choose_and_run_redactor
-from tools.file_conversion import prepare_image_or_pdf, get_input_file_names, convert_review_df_to_annotation_json
+from tools.file_conversion import prepare_image_or_pdf, get_input_file_names
 from tools.redaction_review import apply_redactions_to_review_df_and_files, update_all_page_annotation_object_based_on_previous_page, decrease_page, increase_page, update_annotator_object_and_filter_df, update_entities_df_recogniser_entities, update_entities_df_page, update_entities_df_text, df_select_callback, convert_df_to_xfdf, convert_xfdf_to_dataframe, reset_dropdowns, exclude_selected_items_from_redaction, undo_last_removal, update_selected_review_df_row_colour, update_all_entity_df_dropdowns, df_select_callback_cost, update_other_annotator_number_from_current, update_annotator_page_from_review_df, df_select_callback_ocr, df_select_callback_textract_api
 from tools.data_anonymise import anonymise_data_files
 from tools.auth import authenticate_user
@@ -44,6 +44,19 @@ else:
     default_ocr_val = text_ocr_option
     default_pii_detector = local_pii_detector
 
+SAVE_LOGS_TO_CSV = eval(SAVE_LOGS_TO_CSV)
+SAVE_LOGS_TO_DYNAMODB = eval(SAVE_LOGS_TO_DYNAMODB)
+
+if CSV_ACCESS_LOG_HEADERS: CSV_ACCESS_LOG_HEADERS = eval(CSV_ACCESS_LOG_HEADERS)
+if CSV_FEEDBACK_LOG_HEADERS: CSV_FEEDBACK_LOG_HEADERS = eval(CSV_FEEDBACK_LOG_HEADERS)
+if CSV_USAGE_LOG_HEADERS: CSV_USAGE_LOG_HEADERS = eval(CSV_USAGE_LOG_HEADERS)
+
+if DYNAMODB_ACCESS_LOG_HEADERS: DYNAMODB_ACCESS_LOG_HEADERS = eval(DYNAMODB_ACCESS_LOG_HEADERS)
+if DYNAMODB_FEEDBACK_LOG_HEADERS: DYNAMODB_FEEDBACK_LOG_HEADERS = eval(DYNAMODB_FEEDBACK_LOG_HEADERS)
+if DYNAMODB_USAGE_LOG_HEADERS: DYNAMODB_USAGE_LOG_HEADERS = eval(DYNAMODB_USAGE_LOG_HEADERS)
+
+print
+
 # Create the gradio interface
 app = gr.Blocks(theme = gr.themes.Base(), fill_width=True)
 
@@ -61,6 +74,9 @@ with app:
     all_decision_process_table_state = gr.Dataframe(value=pd.DataFrame(), headers=None, col_count=0, row_count = (0, "dynamic"),  label="all_decision_process_table", visible=False, type="pandas", wrap=True)
     review_file_state = gr.Dataframe(value=pd.DataFrame(), headers=None, col_count=0, row_count = (0, "dynamic"), label="review_file_df", visible=False, type="pandas", wrap=True)
 
+    all_page_line_level_ocr_results = gr.State([])    
+    all_page_line_level_ocr_results_with_children = gr.State([])
+
     session_hash_state = gr.Textbox(label= "session_hash_state", value="", visible=False)
     host_name_textbox = gr.Textbox(label= "host_name_textbox", value=HOST_NAME, visible=False)
     s3_output_folder_state = gr.Textbox(label= "s3_output_folder_state", value="", visible=False)
@@ -105,7 +121,12 @@ with app:
     
     doc_full_file_name_textbox = gr.Textbox(label = "doc_full_file_name_textbox", value="", visible=False)
     doc_file_name_no_extension_textbox = gr.Textbox(label = "doc_full_file_name_textbox", value="", visible=False)
-    blank_doc_file_name_no_extension_textbox_for_logs = gr.Textbox(label = "doc_full_file_name_textbox", value="", visible=False) # Left blank for when user does not want to report file names
+    blank_doc_file_name_no_extension_textbox_for_logs = gr.Textbox(label = "doc_full_file_name_textbox", value="", visible=False)
+    blank_data_file_name_no_extension_textbox_for_logs = gr.Textbox(label = "data_full_file_name_textbox", value="", visible=False)
+    placeholder_doc_file_name_no_extension_textbox_for_logs = gr.Textbox(label = "doc_full_file_name_textbox", value="document", visible=False)
+    placeholder_data_file_name_no_extension_textbox_for_logs = gr.Textbox(label = "data_full_file_name_textbox", value="data_file", visible=False)
+
+    # Left blank for when user does not want to report file names
     doc_file_name_with_extension_textbox = gr.Textbox(label = "doc_file_name_with_extension_textbox", value="", visible=False)
     doc_file_name_textbox_list = gr.Dropdown(label = "doc_file_name_textbox_list", value="", allow_custom_value=True,visible=False)
     latest_review_file_path = gr.Textbox(label = "latest_review_file_path", value="", visible=False) # Latest review file path output from redaction
@@ -149,9 +170,9 @@ with app:
     s3_default_allow_list_file = gr.Textbox(label = "Default allow list file", value=S3_ALLOW_LIST_PATH, visible=False)
     default_allow_list_output_folder_location = gr.Textbox(label = "Output default allow list location", value=OUTPUT_ALLOW_LIST_PATH, visible=False)
 
-    s3_bulk_textract_default_bucket = gr.Textbox(label = "Default Textract bulk S3 bucket", value=TEXTRACT_BULK_ANALYSIS_BUCKET, visible=False)
-    s3_bulk_textract_input_subfolder = gr.Textbox(label = "Default Textract bulk S3 input folder", value=TEXTRACT_BULK_ANALYSIS_INPUT_SUBFOLDER, visible=False)
-    s3_bulk_textract_output_subfolder = gr.Textbox(label = "Default Textract bulk S3 output folder", value=TEXTRACT_BULK_ANALYSIS_OUTPUT_SUBFOLDER, visible=False)
+    s3_bulk_textract_default_bucket = gr.Textbox(label = "Default Textract bulk S3 bucket", value=TEXTRACT_WHOLE_DOCUMENT_ANALYSIS_BUCKET, visible=False)
+    s3_bulk_textract_input_subfolder = gr.Textbox(label = "Default Textract bulk S3 input folder", value=TEXTRACT_WHOLE_DOCUMENT_ANALYSIS_INPUT_SUBFOLDER, visible=False)
+    s3_bulk_textract_output_subfolder = gr.Textbox(label = "Default Textract bulk S3 output folder", value=TEXTRACT_WHOLE_DOCUMENT_ANALYSIS_OUTPUT_SUBFOLDER, visible=False)
     successful_textract_api_call_number = gr.Number(precision=0, value=0, visible=False)
     no_redaction_method_drop = gr.Radio(label = """Placeholder for no redaction method after downloading Textract outputs""", value = no_redaction_option, choices=[no_redaction_option], visible=False)
     textract_only_method_drop = gr.Radio(label="""Placeholder for Textract method after downloading Textract outputs""", value = textract_option, choices=[textract_option], visible=False)
@@ -184,6 +205,7 @@ with app:
     cost_code_choice_drop = gr.Dropdown(value=DEFAULT_COST_CODE, label="Choose cost code for analysis. Please contact Finance if you can't find your cost code in the given list.", choices=[DEFAULT_COST_CODE], allow_custom_value=False, visible=False)
 
     textract_output_found_checkbox = gr.Checkbox(value= False, label="Existing Textract output file found", interactive=False, visible=False)
+    local_ocr_output_found_checkbox = gr.Checkbox(value= False, label="Existing local OCR output file found", interactive=False, visible=False)
     total_pdf_page_count = gr.Number(label = "Total page count", value=0, visible=False)
     estimated_aws_costs_number = gr.Number(label = "Approximate AWS Textract and/or Comprehend cost ($)", value=0, visible=False, precision=2)
     estimated_time_taken_number = gr.Number(label = "Approximate time taken to extract text/redact (minutes)", value=0, visible=False, precision=2)
@@ -240,10 +262,14 @@ with app:
             if SHOW_COSTS == "True":
                 with gr.Accordion("Estimated costs and time taken", open = True, visible=True):                        
                     with gr.Row(equal_height=True):
-                        textract_output_found_checkbox = gr.Checkbox(value= False, label="Existing Textract output file found", interactive=False, visible=True)
-                        total_pdf_page_count = gr.Number(label = "Total page count", value=0, visible=True)
-                        estimated_aws_costs_number = gr.Number(label = "Approximate AWS Textract and/or Comprehend cost (£)", value=0.00, precision=2, visible=True)
-                        estimated_time_taken_number = gr.Number(label = "Approximate time taken to extract text/redact (minutes)", value=0, visible=True, precision=2)             
+                        with gr.Column(scale=1):
+                            textract_output_found_checkbox = gr.Checkbox(value= False, label="Existing Textract output file found", interactive=False, visible=True)
+                            local_ocr_output_found_checkbox = gr.Checkbox(value= False, label="Existing local OCR output file found", interactive=False, visible=True)
+                        with gr.Column(scale=4):
+                            with gr.Row(equal_height=True):
+                                total_pdf_page_count = gr.Number(label = "Total page count", value=0, visible=True)
+                                estimated_aws_costs_number = gr.Number(label = "Approximate AWS Textract and/or Comprehend cost (£)", value=0.00, precision=2, visible=True)
+                                estimated_time_taken_number = gr.Number(label = "Approximate time taken to extract text/redact (minutes)", value=0, visible=True, precision=2)             
                 
             if GET_COST_CODES == "True" or ENFORCE_COST_CODES == "True":
                 with gr.Accordion("Apply cost code", open = True, visible=True):
@@ -253,7 +279,7 @@ with app:
                             reset_cost_code_dataframe_button = gr.Button(value="Reset code code table filter")
                             cost_code_choice_drop = gr.Dropdown(value=DEFAULT_COST_CODE, label="Choose cost code for analysis", choices=[DEFAULT_COST_CODE], allow_custom_value=False, visible=True)
 
-            if SHOW_BULK_TEXTRACT_CALL_OPTIONS == "True":
+            if SHOW_WHOLE_DOCUMENT_TEXTRACT_CALL_OPTIONS == "True":
                 with gr.Accordion("Submit whole document to AWS Textract API (quicker, max 3,000 pages per document)", open = False, visible=True):
                     with gr.Row(equal_height=True):
                         gr.Markdown("""Document will be submitted to AWS Textract API service to extract all text in the document. Processing will take place on (secure) AWS servers, and outputs will be stored on S3 for up to 7 days. To download the results, click 'Check status' below and they will be downloaded if ready.""")
@@ -381,7 +407,7 @@ with app:
     ###
     with gr.Tab(label="Open text or Excel/csv files"):
         gr.Markdown("""### Choose open text or a tabular data file (xlsx or csv) to redact.""")    
-        with gr.Accordion("Paste open text", open = False):
+        with gr.Accordion("Redact open text", open = False):
             in_text = gr.Textbox(label="Enter open text", lines=10)
         with gr.Accordion("Upload xlsx or csv files", open = True):
             in_data_files = gr.File(label="Choose Excel or csv files", file_count= "multiple", file_types=['.xlsx', '.xls', '.csv', '.parquet', '.csv.gz'], height=file_input_height)
@@ -391,6 +417,9 @@ with app:
         in_colnames = gr.Dropdown(choices=["Choose columns to anonymise"], multiselect = True, label="Select columns that you want to anonymise (showing columns present across all files).")
 
         pii_identification_method_drop_tabular = gr.Radio(label = "Choose PII detection method. AWS Comprehend has a cost of approximately $0.01 per 10,000 characters.", value = default_pii_detector, choices=[local_pii_detector, aws_pii_detector])
+
+        with gr.Accordion("Anonymisation output format", open = False):
+            anon_strat = gr.Radio(choices=["replace with 'REDACTED'", "replace with <ENTITY_NAME>", "redact completely", "hash", "mask"], label="Select an anonymisation method.", value = "replace with 'REDACTED'") # , "encrypt", "fake_first_name" are also available, but are not currently included as not that useful in current form
         
         tabular_data_redact_btn = gr.Button("Redact text/data files", variant="primary")
         
@@ -448,10 +477,10 @@ with app:
                 aws_access_key_textbox = gr.Textbox(value='', label="AWS access key for account with permissions for AWS Textract and Comprehend", visible=True, type="password")
                 aws_secret_key_textbox = gr.Textbox(value='', label="AWS secret key for account with permissions for AWS Textract and Comprehend", visible=True, type="password")
 
-        with gr.Accordion("Settings for open text or xlsx/csv files", open = False):
-            anon_strat = gr.Radio(choices=["replace with 'REDACTED'", "replace with <ENTITY_NAME>", "redact completely", "hash", "mask", "encrypt", "fake_first_name"], label="Select an anonymisation method.", value = "replace with 'REDACTED'")
+
             
-        log_files_output = gr.File(label="Log file output", interactive=False)
+        with gr.Accordion("Log file outputs", open = False):
+            log_files_output = gr.File(label="Log file output", interactive=False)
 
         with gr.Accordion("Combine multiple review files", open = False):
             multiple_review_files_in_out = gr.File(label="Combine multiple review_file.csv files together here.", file_count='multiple', file_types=['.csv']) 
@@ -477,14 +506,17 @@ with app:
         handwrite_signature_checkbox.change(calculate_aws_costs, inputs=[total_pdf_page_count, text_extract_method_radio, handwrite_signature_checkbox,  pii_identification_method_drop, textract_output_found_checkbox, only_extract_text_radio], outputs=[estimated_aws_costs_number])
         textract_output_found_checkbox.change(calculate_aws_costs, inputs=[total_pdf_page_count, text_extract_method_radio, handwrite_signature_checkbox,  pii_identification_method_drop, textract_output_found_checkbox, only_extract_text_radio], outputs=[estimated_aws_costs_number])
         only_extract_text_radio.change(calculate_aws_costs, inputs=[total_pdf_page_count, text_extract_method_radio, handwrite_signature_checkbox,  pii_identification_method_drop, textract_output_found_checkbox, only_extract_text_radio], outputs=[estimated_aws_costs_number])
+        textract_output_found_checkbox.change(calculate_aws_costs, inputs=[total_pdf_page_count, text_extract_method_radio, handwrite_signature_checkbox,  pii_identification_method_drop, textract_output_found_checkbox, only_extract_text_radio], outputs=[estimated_aws_costs_number])
 
         # Calculate time taken
-        total_pdf_page_count.change(calculate_time_taken, inputs=[total_pdf_page_count, text_extract_method_radio,          pii_identification_method_drop, textract_output_found_checkbox, only_extract_text_radio], outputs=[estimated_time_taken_number])
-        text_extract_method_radio.change(calculate_time_taken, inputs=[total_pdf_page_count, text_extract_method_radio, pii_identification_method_drop, textract_output_found_checkbox, only_extract_text_radio], outputs=[estimated_time_taken_number])
-        pii_identification_method_drop.change(calculate_time_taken, inputs=[total_pdf_page_count, text_extract_method_radio,  pii_identification_method_drop, textract_output_found_checkbox, only_extract_text_radio], outputs=[estimated_time_taken_number])
-        handwrite_signature_checkbox.change(calculate_time_taken, inputs=[total_pdf_page_count, text_extract_method_radio, pii_identification_method_drop, textract_output_found_checkbox, only_extract_text_radio], outputs=[estimated_time_taken_number])
-        textract_output_found_checkbox.change(calculate_time_taken, inputs=[total_pdf_page_count, text_extract_method_radio, handwrite_signature_checkbox,  pii_identification_method_drop, textract_output_found_checkbox, only_extract_text_radio], outputs=[estimated_time_taken_number])
-        only_extract_text_radio.change(calculate_time_taken, inputs=[total_pdf_page_count, text_extract_method_radio, pii_identification_method_drop, textract_output_found_checkbox, only_extract_text_radio], outputs=[estimated_time_taken_number])
+        total_pdf_page_count.change(calculate_time_taken, inputs=[total_pdf_page_count, text_extract_method_radio,          pii_identification_method_drop, textract_output_found_checkbox, only_extract_text_radio, local_ocr_output_found_checkbox], outputs=[estimated_time_taken_number])
+        text_extract_method_radio.change(calculate_time_taken, inputs=[total_pdf_page_count, text_extract_method_radio, pii_identification_method_drop, textract_output_found_checkbox, only_extract_text_radio, local_ocr_output_found_checkbox], outputs=[estimated_time_taken_number])
+        pii_identification_method_drop.change(calculate_time_taken, inputs=[total_pdf_page_count, text_extract_method_radio,  pii_identification_method_drop, textract_output_found_checkbox, only_extract_text_radio, local_ocr_output_found_checkbox], outputs=[estimated_time_taken_number])
+        handwrite_signature_checkbox.change(calculate_time_taken, inputs=[total_pdf_page_count, text_extract_method_radio, pii_identification_method_drop, textract_output_found_checkbox, only_extract_text_radio, local_ocr_output_found_checkbox], outputs=[estimated_time_taken_number])
+        textract_output_found_checkbox.change(calculate_time_taken, inputs=[total_pdf_page_count, text_extract_method_radio, handwrite_signature_checkbox,  pii_identification_method_drop, textract_output_found_checkbox, only_extract_text_radio, local_ocr_output_found_checkbox], outputs=[estimated_time_taken_number])
+        only_extract_text_radio.change(calculate_time_taken, inputs=[total_pdf_page_count, text_extract_method_radio, pii_identification_method_drop, textract_output_found_checkbox, only_extract_text_radio, local_ocr_output_found_checkbox], outputs=[estimated_time_taken_number])
+        textract_output_found_checkbox.change(calculate_time_taken, inputs=[total_pdf_page_count, text_extract_method_radio, pii_identification_method_drop, textract_output_found_checkbox, only_extract_text_radio, local_ocr_output_found_checkbox], outputs=[estimated_time_taken_number])
+        local_ocr_output_found_checkbox.change(calculate_time_taken, inputs=[total_pdf_page_count, text_extract_method_radio, pii_identification_method_drop, textract_output_found_checkbox, only_extract_text_radio, local_ocr_output_found_checkbox], outputs=[estimated_time_taken_number])
 
     # Allow user to select items from cost code dataframe for cost code
     if SHOW_COSTS=="True" and (GET_COST_CODES == "True" or ENFORCE_COST_CODES == "True"):
@@ -494,27 +526,30 @@ with app:
         cost_code_choice_drop.select(update_cost_code_dataframe_from_dropdown_select, inputs=[cost_code_choice_drop, cost_code_dataframe_base], outputs=[cost_code_dataframe])
 
     in_doc_files.upload(fn=get_input_file_names, inputs=[in_doc_files], outputs=[doc_file_name_no_extension_textbox, doc_file_name_with_extension_textbox, doc_full_file_name_textbox, doc_file_name_textbox_list, total_pdf_page_count]).\
-    success(fn = prepare_image_or_pdf, inputs=[in_doc_files, text_extract_method_radio, latest_file_completed_text, redaction_output_summary_textbox, first_loop_state, annotate_max_pages, all_image_annotations_state, prepare_for_review_bool_false, in_fully_redacted_list_state, output_folder_textbox, input_folder_textbox, prepare_images_bool_false], outputs=[redaction_output_summary_textbox, prepared_pdf_state, images_pdf_state, annotate_max_pages, annotate_max_pages_bottom, pdf_doc_state, all_image_annotations_state, review_file_state, document_cropboxes, page_sizes, textract_output_found_checkbox, all_img_details_state, all_line_level_ocr_results_df_base]).\
-    success(fn=check_for_existing_textract_file, inputs=[doc_file_name_no_extension_textbox, output_folder_textbox], outputs=[textract_output_found_checkbox])
+    success(fn = prepare_image_or_pdf, inputs=[in_doc_files, text_extract_method_radio, latest_file_completed_text, redaction_output_summary_textbox, first_loop_state, annotate_max_pages, all_image_annotations_state, prepare_for_review_bool_false, in_fully_redacted_list_state, output_folder_textbox, input_folder_textbox, prepare_images_bool_false], outputs=[redaction_output_summary_textbox, prepared_pdf_state, images_pdf_state, annotate_max_pages, annotate_max_pages_bottom, pdf_doc_state, all_image_annotations_state, review_file_state, document_cropboxes, page_sizes, textract_output_found_checkbox, all_img_details_state, all_line_level_ocr_results_df_base, local_ocr_output_found_checkbox]).\
+    success(fn=check_for_existing_textract_file, inputs=[doc_file_name_no_extension_textbox, output_folder_textbox], outputs=[textract_output_found_checkbox]).\
+    success(fn=check_for_existing_local_ocr_file, inputs=[doc_file_name_no_extension_textbox, output_folder_textbox], outputs=[local_ocr_output_found_checkbox])
 
     # Run redaction function
     document_redact_btn.click(fn = reset_state_vars, outputs=[all_image_annotations_state, all_line_level_ocr_results_df_base, all_decision_process_table_state, comprehend_query_number, textract_metadata_textbox, annotator, output_file_list_state, log_files_output_list_state, recogniser_entity_dataframe, recogniser_entity_dataframe_base, pdf_doc_state, duplication_file_path_outputs_list_state, redaction_output_summary_textbox, is_a_textract_api_call, textract_query_number]).\
         success(fn= enforce_cost_codes, inputs=[enforce_cost_code_textbox, cost_code_choice_drop, cost_code_dataframe_base]).\
-        success(fn= choose_and_run_redactor, inputs=[in_doc_files, prepared_pdf_state, images_pdf_state, in_redact_language, in_redact_entities, in_redact_comprehend_entities, text_extract_method_radio, in_allow_list_state, in_deny_list_state, in_fully_redacted_list_state, latest_file_completed_text, redaction_output_summary_textbox, output_file_list_state, log_files_output_list_state, first_loop_state, page_min, page_max, actual_time_taken_number, handwrite_signature_checkbox, textract_metadata_textbox, all_image_annotations_state, all_line_level_ocr_results_df_base, all_decision_process_table_state, pdf_doc_state, current_loop_page_number, page_break_return, pii_identification_method_drop, comprehend_query_number, max_fuzzy_spelling_mistakes_num, match_fuzzy_whole_phrase_bool, aws_access_key_textbox, aws_secret_key_textbox, annotate_max_pages, review_file_state, output_folder_textbox, document_cropboxes, page_sizes, textract_output_found_checkbox, only_extract_text_radio, duplication_file_path_outputs_list_state, latest_review_file_path, input_folder_textbox, textract_query_number, latest_ocr_file_path],
-                    outputs=[redaction_output_summary_textbox, output_file, output_file_list_state, latest_file_completed_text, log_files_output, log_files_output_list_state, actual_time_taken_number, textract_metadata_textbox, pdf_doc_state, all_image_annotations_state, current_loop_page_number, page_break_return, all_line_level_ocr_results_df_base, all_decision_process_table_state, comprehend_query_number, output_review_files, annotate_max_pages, annotate_max_pages_bottom, prepared_pdf_state, images_pdf_state, review_file_state, page_sizes, duplication_file_path_outputs_list_state, in_duplicate_pages, latest_review_file_path, textract_query_number, latest_ocr_file_path], api_name="redact_doc").\
+        success(fn= choose_and_run_redactor, inputs=[in_doc_files, prepared_pdf_state, images_pdf_state, in_redact_language, in_redact_entities, in_redact_comprehend_entities, text_extract_method_radio, in_allow_list_state, in_deny_list_state, in_fully_redacted_list_state, latest_file_completed_text, redaction_output_summary_textbox, output_file_list_state, log_files_output_list_state, first_loop_state, page_min, page_max, actual_time_taken_number, handwrite_signature_checkbox, textract_metadata_textbox, all_image_annotations_state, all_line_level_ocr_results_df_base, all_decision_process_table_state, pdf_doc_state, current_loop_page_number, page_break_return, pii_identification_method_drop, comprehend_query_number, max_fuzzy_spelling_mistakes_num, match_fuzzy_whole_phrase_bool, aws_access_key_textbox, aws_secret_key_textbox, annotate_max_pages, review_file_state, output_folder_textbox, document_cropboxes, page_sizes, textract_output_found_checkbox, only_extract_text_radio, duplication_file_path_outputs_list_state, latest_review_file_path, input_folder_textbox, textract_query_number, latest_ocr_file_path, all_page_line_level_ocr_results, all_page_line_level_ocr_results_with_children],
+                    outputs=[redaction_output_summary_textbox, output_file, output_file_list_state, latest_file_completed_text, log_files_output, log_files_output_list_state, actual_time_taken_number, textract_metadata_textbox, pdf_doc_state, all_image_annotations_state, current_loop_page_number, page_break_return, all_line_level_ocr_results_df_base, all_decision_process_table_state, comprehend_query_number, output_review_files, annotate_max_pages, annotate_max_pages_bottom, prepared_pdf_state, images_pdf_state, review_file_state, page_sizes, duplication_file_path_outputs_list_state, in_duplicate_pages, latest_review_file_path, textract_query_number, latest_ocr_file_path, all_page_line_level_ocr_results, all_page_line_level_ocr_results_with_children], api_name="redact_doc").\
                     success(fn=update_annotator_object_and_filter_df, inputs=[all_image_annotations_state, page_min, recogniser_entity_dropdown, page_entity_dropdown, text_entity_dropdown, recogniser_entity_dataframe_base, annotator_zoom_number, review_file_state, page_sizes, doc_full_file_name_textbox, input_folder_textbox], outputs=[annotator, annotate_current_page, annotate_current_page_bottom, annotate_previous_page, recogniser_entity_dropdown, recogniser_entity_dataframe, recogniser_entity_dataframe_base, text_entity_dropdown, page_entity_dropdown, page_sizes, all_image_annotations_state])
 
     # If the app has completed a batch of pages, it will rerun the redaction process until the end of all pages in the document
-    current_loop_page_number.change(fn = choose_and_run_redactor, inputs=[in_doc_files, prepared_pdf_state, images_pdf_state, in_redact_language, in_redact_entities, in_redact_comprehend_entities, text_extract_method_radio, in_allow_list_state, in_deny_list_state, in_fully_redacted_list_state, latest_file_completed_text, redaction_output_summary_textbox, output_file_list_state, log_files_output_list_state, second_loop_state, page_min, page_max, actual_time_taken_number, handwrite_signature_checkbox, textract_metadata_textbox, all_image_annotations_state, all_line_level_ocr_results_df_base, all_decision_process_table_state, pdf_doc_state, current_loop_page_number, page_break_return, pii_identification_method_drop, comprehend_query_number, max_fuzzy_spelling_mistakes_num, match_fuzzy_whole_phrase_bool, aws_access_key_textbox, aws_secret_key_textbox, annotate_max_pages, review_file_state, output_folder_textbox, document_cropboxes, page_sizes, textract_output_found_checkbox, only_extract_text_radio, duplication_file_path_outputs_list_state, latest_review_file_path, input_folder_textbox, textract_query_number, latest_ocr_file_path],
-                    outputs=[redaction_output_summary_textbox, output_file, output_file_list_state, latest_file_completed_text, log_files_output, log_files_output_list_state, actual_time_taken_number, textract_metadata_textbox, pdf_doc_state, all_image_annotations_state, current_loop_page_number, page_break_return, all_line_level_ocr_results_df_base, all_decision_process_table_state, comprehend_query_number, output_review_files, annotate_max_pages, annotate_max_pages_bottom, prepared_pdf_state, images_pdf_state, review_file_state, page_sizes, duplication_file_path_outputs_list_state, in_duplicate_pages, latest_review_file_path, textract_query_number, latest_ocr_file_path]).\
+    current_loop_page_number.change(fn = choose_and_run_redactor, inputs=[in_doc_files, prepared_pdf_state, images_pdf_state, in_redact_language, in_redact_entities, in_redact_comprehend_entities, text_extract_method_radio, in_allow_list_state, in_deny_list_state, in_fully_redacted_list_state, latest_file_completed_text, redaction_output_summary_textbox, output_file_list_state, log_files_output_list_state, second_loop_state, page_min, page_max, actual_time_taken_number, handwrite_signature_checkbox, textract_metadata_textbox, all_image_annotations_state, all_line_level_ocr_results_df_base, all_decision_process_table_state, pdf_doc_state, current_loop_page_number, page_break_return, pii_identification_method_drop, comprehend_query_number, max_fuzzy_spelling_mistakes_num, match_fuzzy_whole_phrase_bool, aws_access_key_textbox, aws_secret_key_textbox, annotate_max_pages, review_file_state, output_folder_textbox, document_cropboxes, page_sizes, textract_output_found_checkbox, only_extract_text_radio, duplication_file_path_outputs_list_state, latest_review_file_path, input_folder_textbox, textract_query_number, latest_ocr_file_path, all_page_line_level_ocr_results, all_page_line_level_ocr_results_with_children],
+                    outputs=[redaction_output_summary_textbox, output_file, output_file_list_state, latest_file_completed_text, log_files_output, log_files_output_list_state, actual_time_taken_number, textract_metadata_textbox, pdf_doc_state, all_image_annotations_state, current_loop_page_number, page_break_return, all_line_level_ocr_results_df_base, all_decision_process_table_state, comprehend_query_number, output_review_files, annotate_max_pages, annotate_max_pages_bottom, prepared_pdf_state, images_pdf_state, review_file_state, page_sizes, duplication_file_path_outputs_list_state, in_duplicate_pages, latest_review_file_path, textract_query_number, latest_ocr_file_path, all_page_line_level_ocr_results, all_page_line_level_ocr_results_with_children]).\
                     success(fn=update_annotator_object_and_filter_df, inputs=[all_image_annotations_state, page_min, recogniser_entity_dropdown, page_entity_dropdown, text_entity_dropdown, recogniser_entity_dataframe_base, annotator_zoom_number, review_file_state, page_sizes, doc_full_file_name_textbox, input_folder_textbox], outputs=[annotator, annotate_current_page, annotate_current_page_bottom, annotate_previous_page, recogniser_entity_dropdown, recogniser_entity_dataframe, recogniser_entity_dataframe_base, text_entity_dropdown, page_entity_dropdown, page_sizes, all_image_annotations_state])
         
     # If a file has been completed, the function will continue onto the next document
-    latest_file_completed_text.change(fn = choose_and_run_redactor, inputs=[in_doc_files, prepared_pdf_state, images_pdf_state, in_redact_language, in_redact_entities, in_redact_comprehend_entities, text_extract_method_radio, in_allow_list_state, in_deny_list_state, in_fully_redacted_list_state, latest_file_completed_text, redaction_output_summary_textbox, output_file_list_state, log_files_output_list_state, second_loop_state, page_min, page_max, actual_time_taken_number, handwrite_signature_checkbox, textract_metadata_textbox, all_image_annotations_state, all_line_level_ocr_results_df_base, all_decision_process_table_state, pdf_doc_state, current_loop_page_number, page_break_return, pii_identification_method_drop, comprehend_query_number, max_fuzzy_spelling_mistakes_num, match_fuzzy_whole_phrase_bool, aws_access_key_textbox, aws_secret_key_textbox, annotate_max_pages, review_file_state, output_folder_textbox, document_cropboxes, page_sizes, textract_output_found_checkbox, only_extract_text_radio, duplication_file_path_outputs_list_state, latest_review_file_path, input_folder_textbox, textract_query_number, latest_ocr_file_path],
-                    outputs=[redaction_output_summary_textbox, output_file, output_file_list_state, latest_file_completed_text, log_files_output, log_files_output_list_state, actual_time_taken_number, textract_metadata_textbox, pdf_doc_state, all_image_annotations_state, current_loop_page_number, page_break_return, all_line_level_ocr_results_df_base, all_decision_process_table_state, comprehend_query_number, output_review_files, annotate_max_pages, annotate_max_pages_bottom, prepared_pdf_state, images_pdf_state, review_file_state, page_sizes, duplication_file_path_outputs_list_state, in_duplicate_pages, latest_review_file_path, textract_query_number, latest_ocr_file_path]).\
+    latest_file_completed_text.change(fn = choose_and_run_redactor, inputs=[in_doc_files, prepared_pdf_state, images_pdf_state, in_redact_language, in_redact_entities, in_redact_comprehend_entities, text_extract_method_radio, in_allow_list_state, in_deny_list_state, in_fully_redacted_list_state, latest_file_completed_text, redaction_output_summary_textbox, output_file_list_state, log_files_output_list_state, second_loop_state, page_min, page_max, actual_time_taken_number, handwrite_signature_checkbox, textract_metadata_textbox, all_image_annotations_state, all_line_level_ocr_results_df_base, all_decision_process_table_state, pdf_doc_state, current_loop_page_number, page_break_return, pii_identification_method_drop, comprehend_query_number, max_fuzzy_spelling_mistakes_num, match_fuzzy_whole_phrase_bool, aws_access_key_textbox, aws_secret_key_textbox, annotate_max_pages, review_file_state, output_folder_textbox, document_cropboxes, page_sizes, textract_output_found_checkbox, only_extract_text_radio, duplication_file_path_outputs_list_state, latest_review_file_path, input_folder_textbox, textract_query_number, latest_ocr_file_path, all_page_line_level_ocr_results, all_page_line_level_ocr_results_with_children],
+                    outputs=[redaction_output_summary_textbox, output_file, output_file_list_state, latest_file_completed_text, log_files_output, log_files_output_list_state, actual_time_taken_number, textract_metadata_textbox, pdf_doc_state, all_image_annotations_state, current_loop_page_number, page_break_return, all_line_level_ocr_results_df_base, all_decision_process_table_state, comprehend_query_number, output_review_files, annotate_max_pages, annotate_max_pages_bottom, prepared_pdf_state, images_pdf_state, review_file_state, page_sizes, duplication_file_path_outputs_list_state, in_duplicate_pages, latest_review_file_path, textract_query_number, latest_ocr_file_path, all_page_line_level_ocr_results, all_page_line_level_ocr_results_with_children]).\
                     success(fn=update_annotator_object_and_filter_df, inputs=[all_image_annotations_state, page_min, recogniser_entity_dropdown, page_entity_dropdown, text_entity_dropdown, recogniser_entity_dataframe_base, annotator_zoom_number, review_file_state, page_sizes, doc_full_file_name_textbox, input_folder_textbox], outputs=[annotator, annotate_current_page, annotate_current_page_bottom, annotate_previous_page, recogniser_entity_dropdown, recogniser_entity_dataframe, recogniser_entity_dataframe_base, text_entity_dropdown, page_entity_dropdown, page_sizes, all_image_annotations_state]).\
                     success(fn=check_for_existing_textract_file, inputs=[doc_file_name_no_extension_textbox, output_folder_textbox], outputs=[textract_output_found_checkbox]).\
-                    success(fn=reveal_feedback_buttons, outputs=[pdf_feedback_radio, pdf_further_details_text, pdf_submit_feedback_btn, pdf_feedback_title])
+                    success(fn=check_for_existing_local_ocr_file, inputs=[doc_file_name_no_extension_textbox, output_folder_textbox], outputs=[local_ocr_output_found_checkbox]).\
+                    success(fn=reveal_feedback_buttons, outputs=[pdf_feedback_radio, pdf_further_details_text, pdf_submit_feedback_btn, pdf_feedback_title]).\
+                    success(fn = reset_aws_call_vars, outputs=[comprehend_query_number, textract_query_number])
     
     # If the line level ocr results are changed by load in by user or by a new redaction task, replace the ocr results displayed in the table    
     all_line_level_ocr_results_df_base.change(reset_ocr_base_dataframe, inputs=[all_line_level_ocr_results_df_base], outputs=[all_line_level_ocr_results_df])
@@ -532,8 +567,8 @@ with app:
     convert_textract_outputs_to_ocr_results.click(fn=check_for_existing_textract_file, inputs=[doc_file_name_no_extension_textbox, output_folder_textbox], outputs=[textract_output_found_checkbox]).\
         success(fn= check_textract_outputs_exist, inputs=[textract_output_found_checkbox]).\
         success(fn = reset_state_vars, outputs=[all_image_annotations_state, all_line_level_ocr_results_df_base, all_decision_process_table_state, comprehend_query_number, textract_metadata_textbox, annotator, output_file_list_state, log_files_output_list_state, recogniser_entity_dataframe, recogniser_entity_dataframe_base, pdf_doc_state, duplication_file_path_outputs_list_state, redaction_output_summary_textbox, is_a_textract_api_call]).\
-        success(fn= choose_and_run_redactor, inputs=[in_doc_files, prepared_pdf_state, images_pdf_state, in_redact_language, in_redact_entities, in_redact_comprehend_entities, textract_only_method_drop, in_allow_list_state, in_deny_list_state, in_fully_redacted_list_state, latest_file_completed_text, redaction_output_summary_textbox, output_file_list_state, log_files_output_list_state, first_loop_state, page_min, page_max, actual_time_taken_number, handwrite_signature_checkbox, textract_metadata_textbox, all_image_annotations_state, all_line_level_ocr_results_df_base, all_decision_process_table_state, pdf_doc_state, current_loop_page_number, page_break_return, no_redaction_method_drop, comprehend_query_number, max_fuzzy_spelling_mistakes_num, match_fuzzy_whole_phrase_bool, aws_access_key_textbox, aws_secret_key_textbox, annotate_max_pages, review_file_state, output_folder_textbox, document_cropboxes, page_sizes, textract_output_found_checkbox, only_extract_text_radio, duplication_file_path_outputs_list_state, latest_review_file_path, input_folder_textbox, textract_query_number, latest_ocr_file_path],
-                    outputs=[redaction_output_summary_textbox, output_file, output_file_list_state, latest_file_completed_text, log_files_output, log_files_output_list_state, actual_time_taken_number, textract_metadata_textbox, pdf_doc_state, all_image_annotations_state, current_loop_page_number, page_break_return, all_line_level_ocr_results_df_base, all_decision_process_table_state, comprehend_query_number, output_review_files, annotate_max_pages, annotate_max_pages_bottom, prepared_pdf_state, images_pdf_state, review_file_state, page_sizes, duplication_file_path_outputs_list_state, in_duplicate_pages, latest_review_file_path, textract_query_number, latest_ocr_file_path])
+        success(fn= choose_and_run_redactor, inputs=[in_doc_files, prepared_pdf_state, images_pdf_state, in_redact_language, in_redact_entities, in_redact_comprehend_entities, textract_only_method_drop, in_allow_list_state, in_deny_list_state, in_fully_redacted_list_state, latest_file_completed_text, redaction_output_summary_textbox, output_file_list_state, log_files_output_list_state, first_loop_state, page_min, page_max, actual_time_taken_number, handwrite_signature_checkbox, textract_metadata_textbox, all_image_annotations_state, all_line_level_ocr_results_df_base, all_decision_process_table_state, pdf_doc_state, current_loop_page_number, page_break_return, no_redaction_method_drop, comprehend_query_number, max_fuzzy_spelling_mistakes_num, match_fuzzy_whole_phrase_bool, aws_access_key_textbox, aws_secret_key_textbox, annotate_max_pages, review_file_state, output_folder_textbox, document_cropboxes, page_sizes, textract_output_found_checkbox, only_extract_text_radio, duplication_file_path_outputs_list_state, latest_review_file_path, input_folder_textbox, textract_query_number, latest_ocr_file_path, all_page_line_level_ocr_results, all_page_line_level_ocr_results_with_children],
+                    outputs=[redaction_output_summary_textbox, output_file, output_file_list_state, latest_file_completed_text, log_files_output, log_files_output_list_state, actual_time_taken_number, textract_metadata_textbox, pdf_doc_state, all_image_annotations_state, current_loop_page_number, page_break_return, all_line_level_ocr_results_df_base, all_decision_process_table_state, comprehend_query_number, output_review_files, annotate_max_pages, annotate_max_pages_bottom, prepared_pdf_state, images_pdf_state, review_file_state, page_sizes, duplication_file_path_outputs_list_state, in_duplicate_pages, latest_review_file_path, textract_query_number, latest_ocr_file_path, all_page_line_level_ocr_results, all_page_line_level_ocr_results_with_children])
     
     ###
     # REVIEW PDF REDACTIONS
@@ -542,7 +577,7 @@ with app:
     # Upload previous files for modifying redactions
     upload_previous_review_file_btn.click(fn=reset_review_vars, inputs=None, outputs=[recogniser_entity_dataframe, recogniser_entity_dataframe_base]).\
         success(fn=get_input_file_names, inputs=[output_review_files], outputs=[doc_file_name_no_extension_textbox, doc_file_name_with_extension_textbox, doc_full_file_name_textbox, doc_file_name_textbox_list, total_pdf_page_count]).\
-        success(fn = prepare_image_or_pdf, inputs=[output_review_files, text_extract_method_radio, latest_file_completed_text, redaction_output_summary_textbox, second_loop_state, annotate_max_pages, all_image_annotations_state, prepare_for_review_bool, in_fully_redacted_list_state, output_folder_textbox, input_folder_textbox, prepare_images_bool_false], outputs=[redaction_output_summary_textbox, prepared_pdf_state, images_pdf_state, annotate_max_pages, annotate_max_pages_bottom, pdf_doc_state, all_image_annotations_state, review_file_state, document_cropboxes, page_sizes, textract_output_found_checkbox, all_img_details_state, all_line_level_ocr_results_df_base], api_name="prepare_doc").\
+        success(fn = prepare_image_or_pdf, inputs=[output_review_files, text_extract_method_radio, latest_file_completed_text, redaction_output_summary_textbox, second_loop_state, annotate_max_pages, all_image_annotations_state, prepare_for_review_bool, in_fully_redacted_list_state, output_folder_textbox, input_folder_textbox, prepare_images_bool_false], outputs=[redaction_output_summary_textbox, prepared_pdf_state, images_pdf_state, annotate_max_pages, annotate_max_pages_bottom, pdf_doc_state, all_image_annotations_state, review_file_state, document_cropboxes, page_sizes, textract_output_found_checkbox, all_img_details_state, all_line_level_ocr_results_df_base, local_ocr_output_found_checkbox], api_name="prepare_doc").\
         success(update_annotator_object_and_filter_df, inputs=[all_image_annotations_state, annotate_current_page, recogniser_entity_dropdown, page_entity_dropdown, text_entity_dropdown, recogniser_entity_dataframe_base, annotator_zoom_number, review_file_state, page_sizes, doc_full_file_name_textbox, input_folder_textbox], outputs = [annotator, annotate_current_page, annotate_current_page_bottom, annotate_previous_page, recogniser_entity_dropdown, recogniser_entity_dataframe, recogniser_entity_dataframe_base, text_entity_dropdown, page_entity_dropdown, page_sizes, all_image_annotations_state])
 
     # Page number controls
@@ -572,9 +607,9 @@ with app:
     text_entity_dropdown.select(update_entities_df_text, inputs=[text_entity_dropdown, recogniser_entity_dataframe_base, recogniser_entity_dropdown, page_entity_dropdown], outputs=[recogniser_entity_dataframe, recogniser_entity_dropdown, page_entity_dropdown])
 
     # Clicking on a cell in the recogniser entity dataframe will take you to that page, and also highlight the target redaction box in blue
-    recogniser_entity_dataframe.select(df_select_callback, inputs=[recogniser_entity_dataframe], outputs=[annotate_current_page, selected_entity_dataframe_row]).\
-        success(update_selected_review_df_row_colour, inputs=[selected_entity_dataframe_row, review_file_state, selected_entity_id, selected_entity_colour, page_sizes], outputs=[review_file_state, selected_entity_id, selected_entity_colour]).\
-        success(update_annotator_page_from_review_df, inputs=[review_file_state, images_pdf_state, page_sizes, annotate_current_page, annotate_previous_page, all_image_annotations_state, annotator], outputs=[annotator, all_image_annotations_state])
+    recogniser_entity_dataframe.select(df_select_callback, inputs=[recogniser_entity_dataframe], outputs=[selected_entity_dataframe_row]).\
+        success(update_selected_review_df_row_colour, inputs=[selected_entity_dataframe_row, review_file_state, selected_entity_id, selected_entity_colour], outputs=[review_file_state, selected_entity_id, selected_entity_colour]).\
+        success(update_annotator_page_from_review_df, inputs=[review_file_state, images_pdf_state, page_sizes, all_image_annotations_state, annotator, selected_entity_dataframe_row, input_folder_textbox, doc_full_file_name_textbox], outputs=[annotator, all_image_annotations_state, annotate_current_page, page_sizes, review_file_state, annotate_previous_page])
    
     reset_dropdowns_btn.click(reset_dropdowns, inputs=[recogniser_entity_dataframe_base], outputs=[recogniser_entity_dropdown, text_entity_dropdown, page_entity_dropdown]).\
         success(update_annotator_object_and_filter_df, inputs=[all_image_annotations_state, annotate_current_page, recogniser_entity_dropdown, page_entity_dropdown, text_entity_dropdown, recogniser_entity_dataframe_base, annotator_zoom_number, review_file_state, page_sizes, doc_full_file_name_textbox, input_folder_textbox], outputs = [annotator, annotate_current_page, annotate_current_page_bottom, annotate_previous_page, recogniser_entity_dropdown, recogniser_entity_dataframe, recogniser_entity_dataframe_base, text_entity_dropdown, page_entity_dropdown, page_sizes, all_image_annotations_state])
@@ -604,12 +639,12 @@ with app:
     
     # Convert review file to xfdf Adobe format
     convert_review_file_to_adobe_btn.click(fn=get_input_file_names, inputs=[output_review_files], outputs=[doc_file_name_no_extension_textbox, doc_file_name_with_extension_textbox, doc_full_file_name_textbox, doc_file_name_textbox_list, total_pdf_page_count]).\
-        success(fn = prepare_image_or_pdf, inputs=[output_review_files, text_extract_method_radio, latest_file_completed_text, redaction_output_summary_textbox, second_loop_state, annotate_max_pages, all_image_annotations_state, prepare_for_review_bool, in_fully_redacted_list_state, output_folder_textbox, input_folder_textbox, prepare_images_bool_false], outputs=[redaction_output_summary_textbox, prepared_pdf_state, images_pdf_state, annotate_max_pages, annotate_max_pages_bottom, pdf_doc_state, all_image_annotations_state, review_file_state, document_cropboxes, page_sizes, textract_output_found_checkbox, all_img_details_state, all_line_level_ocr_results_df_placeholder]).\
+        success(fn = prepare_image_or_pdf, inputs=[output_review_files, text_extract_method_radio, latest_file_completed_text, redaction_output_summary_textbox, second_loop_state, annotate_max_pages, all_image_annotations_state, prepare_for_review_bool, in_fully_redacted_list_state, output_folder_textbox, input_folder_textbox, prepare_images_bool_false], outputs=[redaction_output_summary_textbox, prepared_pdf_state, images_pdf_state, annotate_max_pages, annotate_max_pages_bottom, pdf_doc_state, all_image_annotations_state, review_file_state, document_cropboxes, page_sizes, textract_output_found_checkbox, all_img_details_state, all_line_level_ocr_results_df_placeholder, local_ocr_output_found_checkbox]).\
         success(convert_df_to_xfdf, inputs=[output_review_files, pdf_doc_state, images_pdf_state, output_folder_textbox, document_cropboxes, page_sizes], outputs=[adobe_review_files_out])
     
     # Convert xfdf Adobe file back to review_file.csv
     convert_adobe_to_review_file_btn.click(fn=get_input_file_names, inputs=[adobe_review_files_out], outputs=[doc_file_name_no_extension_textbox, doc_file_name_with_extension_textbox, doc_full_file_name_textbox, doc_file_name_textbox_list, total_pdf_page_count]).\
-        success(fn = prepare_image_or_pdf, inputs=[adobe_review_files_out, text_extract_method_radio, latest_file_completed_text, redaction_output_summary_textbox, second_loop_state, annotate_max_pages, all_image_annotations_state, prepare_for_review_bool, in_fully_redacted_list_state, output_folder_textbox, input_folder_textbox, prepare_images_bool_false], outputs=[redaction_output_summary_textbox, prepared_pdf_state, images_pdf_state, annotate_max_pages, annotate_max_pages_bottom, pdf_doc_state, all_image_annotations_state, review_file_state, document_cropboxes, page_sizes, textract_output_found_checkbox, all_img_details_state, all_line_level_ocr_results_df_placeholder]).\
+        success(fn = prepare_image_or_pdf, inputs=[adobe_review_files_out, text_extract_method_radio, latest_file_completed_text, redaction_output_summary_textbox, second_loop_state, annotate_max_pages, all_image_annotations_state, prepare_for_review_bool, in_fully_redacted_list_state, output_folder_textbox, input_folder_textbox, prepare_images_bool_false], outputs=[redaction_output_summary_textbox, prepared_pdf_state, images_pdf_state, annotate_max_pages, annotate_max_pages_bottom, pdf_doc_state, all_image_annotations_state, review_file_state, document_cropboxes, page_sizes, textract_output_found_checkbox, all_img_details_state, all_line_level_ocr_results_df_placeholder, local_ocr_output_found_checkbox]).\
         success(fn=convert_xfdf_to_dataframe, inputs=[adobe_review_files_out, pdf_doc_state, images_pdf_state, output_folder_textbox], outputs=[output_review_files], scroll_to_output=True)
     
     ###
@@ -618,11 +653,14 @@ with app:
     in_data_files.upload(fn=put_columns_in_df, inputs=[in_data_files], outputs=[in_colnames, in_excel_sheets]).\
                   success(fn=get_input_file_names, inputs=[in_data_files], outputs=[data_file_name_no_extension_textbox, data_file_name_with_extension_textbox, data_full_file_name_textbox, data_file_name_textbox_list, total_pdf_page_count])
 
-    tabular_data_redact_btn.click(fn=anonymise_data_files, inputs=[in_data_files, in_text, anon_strat, in_colnames, in_redact_language, in_redact_entities, in_allow_list_state, text_tabular_files_done, text_output_summary, text_output_file_list_state, log_files_output_list_state, in_excel_sheets, first_loop_state, output_folder_textbox, in_deny_list_state, max_fuzzy_spelling_mistakes_num, pii_identification_method_drop_tabular, in_redact_comprehend_entities, comprehend_query_number, aws_access_key_textbox, aws_secret_key_textbox], outputs=[text_output_summary, text_output_file, text_output_file_list_state, text_tabular_files_done, log_files_output, log_files_output_list_state], api_name="redact_data")
+    tabular_data_redact_btn.click(reset_data_vars, outputs=[actual_time_taken_number, log_files_output_list_state, comprehend_query_number]).\
+    success(fn=anonymise_data_files, inputs=[in_data_files, in_text, anon_strat, in_colnames, in_redact_language, in_redact_entities, in_allow_list_state, text_tabular_files_done, text_output_summary, text_output_file_list_state, log_files_output_list_state, in_excel_sheets, first_loop_state, output_folder_textbox, in_deny_list_state, max_fuzzy_spelling_mistakes_num, pii_identification_method_drop_tabular, in_redact_comprehend_entities, comprehend_query_number, aws_access_key_textbox, aws_secret_key_textbox, actual_time_taken_number], outputs=[text_output_summary, text_output_file, text_output_file_list_state, text_tabular_files_done, log_files_output, log_files_output_list_state, actual_time_taken_number], api_name="redact_data").\
+    success(fn = reveal_feedback_buttons, outputs=[data_feedback_radio, data_further_details_text, data_submit_feedback_btn, data_feedback_title])
 
+    # Currently only supports redacting one data file at a time
     # If the output file count text box changes, keep going with redacting each data file until done
-    text_tabular_files_done.change(fn=anonymise_data_files, inputs=[in_data_files, in_text, anon_strat, in_colnames, in_redact_language, in_redact_entities, in_allow_list_state, text_tabular_files_done, text_output_summary, text_output_file_list_state, log_files_output_list_state, in_excel_sheets, second_loop_state, output_folder_textbox, in_deny_list_state, max_fuzzy_spelling_mistakes_num, pii_identification_method_drop_tabular, in_redact_comprehend_entities, comprehend_query_number, aws_access_key_textbox, aws_secret_key_textbox], outputs=[text_output_summary, text_output_file, text_output_file_list_state, text_tabular_files_done, log_files_output, log_files_output_list_state]).\
-    success(fn = reveal_feedback_buttons, outputs=[data_feedback_radio, data_further_details_text, data_submit_feedback_btn, data_feedback_title])
+    # text_tabular_files_done.change(fn=anonymise_data_files, inputs=[in_data_files, in_text, anon_strat, in_colnames, in_redact_language, in_redact_entities, in_allow_list_state, text_tabular_files_done, text_output_summary, text_output_file_list_state, log_files_output_list_state, in_excel_sheets, second_loop_state, output_folder_textbox, in_deny_list_state, max_fuzzy_spelling_mistakes_num, pii_identification_method_drop_tabular, in_redact_comprehend_entities, comprehend_query_number, aws_access_key_textbox, aws_secret_key_textbox, actual_time_taken_number], outputs=[text_output_summary, text_output_file, text_output_file_list_state, text_tabular_files_done, log_files_output, log_files_output_list_state, actual_time_taken_number]).\
+    # success(fn = reveal_feedback_buttons, outputs=[data_feedback_radio, data_further_details_text, data_submit_feedback_btn, data_feedback_title])
 
     ###
     # IDENTIFY DUPLICATE PAGES
@@ -654,7 +692,7 @@ with app:
 
     # Get connection details on app load
 
-    if SHOW_BULK_TEXTRACT_CALL_OPTIONS == "True":
+    if SHOW_WHOLE_DOCUMENT_TEXTRACT_CALL_OPTIONS == "True":
         app.load(get_connection_params, inputs=[output_folder_textbox, input_folder_textbox, session_output_folder_textbox, s3_bulk_textract_input_subfolder, s3_bulk_textract_output_subfolder, s3_bulk_textract_logs_subfolder, local_bulk_textract_logs_subfolder], outputs=[session_hash_state, output_folder_textbox, session_hash_textbox, input_folder_textbox, s3_bulk_textract_input_subfolder, s3_bulk_textract_output_subfolder, s3_bulk_textract_logs_subfolder, local_bulk_textract_logs_subfolder]).\
         success(load_in_textract_job_details, inputs=[load_s3_bulk_textract_logs_bool, s3_bulk_textract_logs_subfolder, local_bulk_textract_logs_subfolder], outputs=[textract_job_detail_df])
     else:
@@ -691,49 +729,71 @@ with app:
     # LOGGING
     ###
 
+    ### ACCESS LOGS
     # Log usernames and times of access to file (to know who is using the app when running on AWS)
     access_callback = CSVLogger_custom(dataset_file_name=log_file_name)
-    access_callback.setup([session_hash_textbox, host_name_textbox], ACCESS_LOGS_FOLDER)
-    
-    session_hash_textbox.change(lambda *args: access_callback.flag(list(args)), [session_hash_textbox, host_name_textbox], None, preprocess=False).\
-    success(fn = upload_file_to_s3, inputs=[access_logs_state, access_s3_logs_loc_state], outputs=[s3_logs_output_textbox])
+    access_callback.setup([session_hash_textbox, host_name_textbox], ACCESS_LOGS_FOLDER)    
+    session_hash_textbox.change(lambda *args: access_callback.flag(list(args), save_to_csv=SAVE_LOGS_TO_CSV, save_to_dynamodb=SAVE_LOGS_TO_DYNAMODB, dynamodb_table_name=ACCESS_LOG_DYNAMODB_TABLE_NAME, dynamodb_headers=DYNAMODB_ACCESS_LOG_HEADERS, replacement_headers=CSV_ACCESS_LOG_HEADERS), [session_hash_textbox, host_name_textbox], None, preprocess=False).\
+    success(fn = upload_log_file_to_s3, inputs=[access_logs_state, access_s3_logs_loc_state], outputs=[s3_logs_output_textbox])
 
-    # User submitted feedback for pdf redactions
-    pdf_callback = CSVLogger_custom(dataset_file_name=log_file_name)
-    pdf_callback.setup([pdf_feedback_radio, pdf_further_details_text, doc_file_name_no_extension_textbox], FEEDBACK_LOGS_FOLDER)
-    pdf_submit_feedback_btn.click(lambda *args: pdf_callback.flag(list(args)), [pdf_feedback_radio, pdf_further_details_text, doc_file_name_no_extension_textbox], None, preprocess=False).\
-    success(fn = upload_file_to_s3, inputs=[feedback_logs_state, feedback_s3_logs_loc_state], outputs=[pdf_further_details_text])
+    ### FEEDBACK LOGS
+    if DISPLAY_FILE_NAMES_IN_LOGS == 'True':
+        # User submitted feedback for pdf redactions
+        pdf_callback = CSVLogger_custom(dataset_file_name=log_file_name)
+        pdf_callback.setup([pdf_feedback_radio, pdf_further_details_text, doc_file_name_no_extension_textbox], FEEDBACK_LOGS_FOLDER)
+        pdf_submit_feedback_btn.click(lambda *args: pdf_callback.flag(list(args), save_to_csv=SAVE_LOGS_TO_CSV, save_to_dynamodb=SAVE_LOGS_TO_DYNAMODB, dynamodb_table_name=FEEDBACK_LOG_DYNAMODB_TABLE_NAME, dynamodb_headers=DYNAMODB_FEEDBACK_LOG_HEADERS, replacement_headers=CSV_FEEDBACK_LOG_HEADERS), [pdf_feedback_radio, pdf_further_details_text, doc_file_name_no_extension_textbox], None, preprocess=False).\
+        success(fn = upload_log_file_to_s3, inputs=[feedback_logs_state, feedback_s3_logs_loc_state], outputs=[pdf_further_details_text])
+
+        # User submitted feedback for data redactions
+        data_callback = CSVLogger_custom(dataset_file_name=log_file_name)
+        data_callback.setup([data_feedback_radio, data_further_details_text, data_full_file_name_textbox], FEEDBACK_LOGS_FOLDER)
+        data_submit_feedback_btn.click(lambda *args: data_callback.flag(list(args), save_to_csv=SAVE_LOGS_TO_CSV, save_to_dynamodb=SAVE_LOGS_TO_DYNAMODB, dynamodb_table_name=FEEDBACK_LOG_DYNAMODB_TABLE_NAME, dynamodb_headers=DYNAMODB_FEEDBACK_LOG_HEADERS, replacement_headers=CSV_FEEDBACK_LOG_HEADERS), [data_feedback_radio, data_further_details_text, data_full_file_name_textbox], None, preprocess=False).\
+        success(fn = upload_log_file_to_s3, inputs=[feedback_logs_state, feedback_s3_logs_loc_state], outputs=[data_further_details_text])
+    else:
+        # User submitted feedback for pdf redactions
+        pdf_callback = CSVLogger_custom(dataset_file_name=log_file_name)
+        pdf_callback.setup([pdf_feedback_radio, pdf_further_details_text, doc_file_name_no_extension_textbox], FEEDBACK_LOGS_FOLDER)
+        pdf_submit_feedback_btn.click(lambda *args: pdf_callback.flag(list(args), save_to_csv=SAVE_LOGS_TO_CSV, save_to_dynamodb=SAVE_LOGS_TO_DYNAMODB, dynamodb_table_name=FEEDBACK_LOG_DYNAMODB_TABLE_NAME, dynamodb_headers=DYNAMODB_FEEDBACK_LOG_HEADERS, replacement_headers=CSV_FEEDBACK_LOG_HEADERS), [pdf_feedback_radio, pdf_further_details_text, placeholder_doc_file_name_no_extension_textbox_for_logs], None, preprocess=False).\
+        success(fn = upload_log_file_to_s3, inputs=[feedback_logs_state, feedback_s3_logs_loc_state], outputs=[pdf_further_details_text])
+
+        # User submitted feedback for data redactions
+        data_callback = CSVLogger_custom(dataset_file_name=log_file_name)
+        data_callback.setup([data_feedback_radio, data_further_details_text, data_full_file_name_textbox], FEEDBACK_LOGS_FOLDER)
+        data_submit_feedback_btn.click(lambda *args: data_callback.flag(list(args), save_to_csv=SAVE_LOGS_TO_CSV, save_to_dynamodb=SAVE_LOGS_TO_DYNAMODB, dynamodb_table_name=FEEDBACK_LOG_DYNAMODB_TABLE_NAME, dynamodb_headers=DYNAMODB_FEEDBACK_LOG_HEADERS, replacement_headers=CSV_FEEDBACK_LOG_HEADERS), [data_feedback_radio, data_further_details_text, placeholder_data_file_name_no_extension_textbox_for_logs], None, preprocess=False).\
+        success(fn = upload_log_file_to_s3, inputs=[feedback_logs_state, feedback_s3_logs_loc_state], outputs=[data_further_details_text])
 
-    # User submitted feedback for data redactions
-    data_callback = CSVLogger_custom(dataset_file_name=log_file_name)
-    data_callback.setup([data_feedback_radio, data_further_details_text, data_full_file_name_textbox], FEEDBACK_LOGS_FOLDER)
-    data_submit_feedback_btn.click(lambda *args: data_callback.flag(list(args)), [data_feedback_radio, data_further_details_text, data_full_file_name_textbox], None, preprocess=False).\
-    success(fn = upload_file_to_s3, inputs=[feedback_logs_state, feedback_s3_logs_loc_state], outputs=[data_further_details_text])
+    ### USAGE LOGS
+    # Log processing usage - time taken for redaction queries, and also logs for queries to Textract/Comprehend
 
-    # Log processing time/token usage when making a query
-    usage_callback = CSVLogger_custom(dataset_file_name=log_file_name)    
+    usage_callback = CSVLogger_custom(dataset_file_name=log_file_name) 
 
     if DISPLAY_FILE_NAMES_IN_LOGS == 'True':
         usage_callback.setup([session_hash_textbox, doc_file_name_no_extension_textbox, data_full_file_name_textbox, total_pdf_page_count, actual_time_taken_number, textract_query_number, pii_identification_method_drop, comprehend_query_number, cost_code_choice_drop, handwrite_signature_checkbox, host_name_textbox, text_extract_method_radio, is_a_textract_api_call], USAGE_LOGS_FOLDER)
 
-        latest_file_completed_text.change(lambda *args: usage_callback.flag(list(args)), [session_hash_textbox, doc_file_name_no_extension_textbox, data_full_file_name_textbox, total_pdf_page_count, actual_time_taken_number, textract_query_number, pii_identification_method_drop, comprehend_query_number, cost_code_choice_drop, handwrite_signature_checkbox, host_name_textbox, text_extract_method_radio, is_a_textract_api_call], None, preprocess=False).\
-        success(fn = upload_file_to_s3, inputs=[usage_logs_state, usage_s3_logs_loc_state], outputs=[s3_logs_output_textbox])
+        latest_file_completed_text.change(lambda *args: usage_callback.flag(list(args), save_to_csv=SAVE_LOGS_TO_CSV, save_to_dynamodb=SAVE_LOGS_TO_DYNAMODB, dynamodb_table_name=USAGE_LOG_DYNAMODB_TABLE_NAME, dynamodb_headers=DYNAMODB_USAGE_LOG_HEADERS, replacement_headers=CSV_USAGE_LOG_HEADERS), [session_hash_textbox, doc_file_name_no_extension_textbox, data_full_file_name_textbox, total_pdf_page_count, actual_time_taken_number, textract_query_number, pii_identification_method_drop, comprehend_query_number, cost_code_choice_drop, handwrite_signature_checkbox, host_name_textbox, text_extract_method_radio, is_a_textract_api_call], None, preprocess=False).\
+        success(fn = upload_log_file_to_s3, inputs=[usage_logs_state, usage_s3_logs_loc_state], outputs=[s3_logs_output_textbox])
 
-        successful_textract_api_call_number.change(lambda *args: usage_callback.flag(list(args)), [session_hash_textbox, doc_file_name_no_extension_textbox, data_full_file_name_textbox, total_pdf_page_count, actual_time_taken_number, textract_query_number, pii_identification_method_drop, comprehend_query_number, cost_code_choice_drop, handwrite_signature_checkbox, host_name_textbox, text_extract_method_radio, is_a_textract_api_call], None, preprocess=False).\
-        success(fn = upload_file_to_s3, inputs=[usage_logs_state, usage_s3_logs_loc_state], outputs=[s3_logs_output_textbox])
+        text_tabular_files_done.change(lambda *args: usage_callback.flag(list(args), save_to_csv=SAVE_LOGS_TO_CSV, save_to_dynamodb=SAVE_LOGS_TO_DYNAMODB, dynamodb_table_name=USAGE_LOG_DYNAMODB_TABLE_NAME, dynamodb_headers=DYNAMODB_USAGE_LOG_HEADERS, replacement_headers=CSV_USAGE_LOG_HEADERS), [session_hash_textbox, doc_file_name_no_extension_textbox, data_full_file_name_textbox, total_pdf_page_count, actual_time_taken_number, textract_query_number, pii_identification_method_drop_tabular, comprehend_query_number, cost_code_choice_drop, handwrite_signature_checkbox, host_name_textbox, text_extract_method_radio, is_a_textract_api_call], None, preprocess=False).\
+        success(fn = upload_log_file_to_s3, inputs=[usage_logs_state, usage_s3_logs_loc_state], outputs=[s3_logs_output_textbox])
+
+        successful_textract_api_call_number.change(lambda *args: usage_callback.flag(list(args), save_to_csv=SAVE_LOGS_TO_CSV, save_to_dynamodb=SAVE_LOGS_TO_DYNAMODB, dynamodb_table_name=USAGE_LOG_DYNAMODB_TABLE_NAME, dynamodb_headers=DYNAMODB_USAGE_LOG_HEADERS, replacement_headers=CSV_USAGE_LOG_HEADERS), [session_hash_textbox, doc_file_name_no_extension_textbox, data_full_file_name_textbox, total_pdf_page_count, actual_time_taken_number, textract_query_number, pii_identification_method_drop, comprehend_query_number, cost_code_choice_drop, handwrite_signature_checkbox, host_name_textbox, text_extract_method_radio, is_a_textract_api_call], None, preprocess=False).\
+        success(fn = upload_log_file_to_s3, inputs=[usage_logs_state, usage_s3_logs_loc_state], outputs=[s3_logs_output_textbox])
     else:
-        usage_callback.setup([session_hash_textbox, blank_doc_file_name_no_extension_textbox_for_logs, data_full_file_name_textbox, actual_time_taken_number, total_pdf_page_count, textract_query_number, pii_identification_method_drop, comprehend_query_number, cost_code_choice_drop, handwrite_signature_checkbox, host_name_textbox, text_extract_method_radio, is_a_textract_api_call], USAGE_LOGS_FOLDER)
+        usage_callback.setup([session_hash_textbox, blank_doc_file_name_no_extension_textbox_for_logs, blank_data_file_name_no_extension_textbox_for_logs, actual_time_taken_number, total_pdf_page_count, textract_query_number, pii_identification_method_drop, comprehend_query_number, cost_code_choice_drop, handwrite_signature_checkbox, host_name_textbox, text_extract_method_radio, is_a_textract_api_call], USAGE_LOGS_FOLDER)
+
+        latest_file_completed_text.change(lambda *args: usage_callback.flag(list(args), save_to_csv=SAVE_LOGS_TO_CSV, save_to_dynamodb=SAVE_LOGS_TO_DYNAMODB, dynamodb_table_name=USAGE_LOG_DYNAMODB_TABLE_NAME, dynamodb_headers=DYNAMODB_USAGE_LOG_HEADERS, replacement_headers=CSV_USAGE_LOG_HEADERS), [session_hash_textbox, placeholder_doc_file_name_no_extension_textbox_for_logs, blank_data_file_name_no_extension_textbox_for_logs, actual_time_taken_number, total_pdf_page_count, textract_query_number, pii_identification_method_drop, comprehend_query_number, cost_code_choice_drop, handwrite_signature_checkbox, host_name_textbox, text_extract_method_radio, is_a_textract_api_call], None, preprocess=False).\
+        success(fn = upload_log_file_to_s3, inputs=[usage_logs_state, usage_s3_logs_loc_state], outputs=[s3_logs_output_textbox])
 
-        latest_file_completed_text.change(lambda *args: usage_callback.flag(list(args)), [session_hash_textbox, blank_doc_file_name_no_extension_textbox_for_logs, data_full_file_name_textbox, actual_time_taken_number, total_pdf_page_count, textract_query_number, pii_identification_method_drop, comprehend_query_number, cost_code_choice_drop, handwrite_signature_checkbox, host_name_textbox, text_extract_method_radio, is_a_textract_api_call], None, preprocess=False).\
-        success(fn = upload_file_to_s3, inputs=[usage_logs_state, usage_s3_logs_loc_state], outputs=[s3_logs_output_textbox])
+        text_tabular_files_done.change(lambda *args: usage_callback.flag(list(args), save_to_csv=SAVE_LOGS_TO_CSV, save_to_dynamodb=SAVE_LOGS_TO_DYNAMODB, dynamodb_table_name=USAGE_LOG_DYNAMODB_TABLE_NAME, dynamodb_headers=DYNAMODB_USAGE_LOG_HEADERS, replacement_headers=CSV_USAGE_LOG_HEADERS), [session_hash_textbox, blank_doc_file_name_no_extension_textbox_for_logs, placeholder_data_file_name_no_extension_textbox_for_logs,  actual_time_taken_number, total_pdf_page_count, textract_query_number, pii_identification_method_drop_tabular, comprehend_query_number, cost_code_choice_drop, handwrite_signature_checkbox, host_name_textbox, text_extract_method_radio, is_a_textract_api_call], None, preprocess=False).\
+        success(fn = upload_log_file_to_s3, inputs=[usage_logs_state, usage_s3_logs_loc_state], outputs=[s3_logs_output_textbox])
 
-        successful_textract_api_call_number.change(lambda *args: usage_callback.flag(list(args)), [session_hash_textbox, blank_doc_file_name_no_extension_textbox_for_logs, data_full_file_name_textbox, actual_time_taken_number, total_pdf_page_count, textract_query_number, pii_identification_method_drop, comprehend_query_number, cost_code_choice_drop, handwrite_signature_checkbox, host_name_textbox, text_extract_method_radio, is_a_textract_api_call], None, preprocess=False).\
-        success(fn = upload_file_to_s3, inputs=[usage_logs_state, usage_s3_logs_loc_state], outputs=[s3_logs_output_textbox])
+        successful_textract_api_call_number.change(lambda *args: usage_callback.flag(list(args), save_to_csv=SAVE_LOGS_TO_CSV, save_to_dynamodb=SAVE_LOGS_TO_DYNAMODB, dynamodb_table_name=USAGE_LOG_DYNAMODB_TABLE_NAME, dynamodb_headers=DYNAMODB_USAGE_LOG_HEADERS, replacement_headers=CSV_USAGE_LOG_HEADERS), [session_hash_textbox, placeholder_doc_file_name_no_extension_textbox_for_logs, blank_data_file_name_no_extension_textbox_for_logs, actual_time_taken_number, total_pdf_page_count, textract_query_number, pii_identification_method_drop, comprehend_query_number, cost_code_choice_drop, handwrite_signature_checkbox, host_name_textbox, text_extract_method_radio, is_a_textract_api_call], None, preprocess=False).\
+        success(fn = upload_log_file_to_s3, inputs=[usage_logs_state, usage_s3_logs_loc_state], outputs=[s3_logs_output_textbox])
 
 if __name__ == "__main__":
     if RUN_DIRECT_MODE == "0":
         
-        if os.environ['COGNITO_AUTH'] == "1":
+        if COGNITO_AUTH == "1":
             app.queue(max_size=int(MAX_QUEUE_SIZE), default_concurrency_limit=int(DEFAULT_CONCURRENCY_LIMIT)).launch(show_error=True, inbrowser=True, auth=authenticate_user, max_file_size=MAX_FILE_SIZE, server_port=GRADIO_SERVER_PORT, root_path=ROOT_PATH)
         else:
             app.queue(max_size=int(MAX_QUEUE_SIZE), default_concurrency_limit=int(DEFAULT_CONCURRENCY_LIMIT)).launch(show_error=True, inbrowser=True, max_file_size=MAX_FILE_SIZE, server_port=GRADIO_SERVER_PORT, root_path=ROOT_PATH)

pyproject.toml

@@ -0,0 +1,57 @@
+[build-system]
+requires = ["setuptools>=61.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "doc_redaction" # Your application's name
+version = "0.6.0" # Your application's current version
+description = "Redact PDF/image-based documents, or CSV/XLSX files using a Gradio-based GUI interface" # A short description
+readme = "README.md" # Path to your project's README file
+requires-python = ">=3.10" # The minimum Python version required
+
+dependencies = [
+    "pdfminer.six==20240706",
+    "pdf2image==1.17.0",
+    "pymupdf==1.25.3",
+    "opencv-python==4.10.0.84",
+    "presidio_analyzer==2.2.358",
+    "presidio_anonymizer==2.2.358",
+    "presidio-image-redactor==0.0.56",
+    "pikepdf==9.5.2",
+    "pandas==2.2.3",
+    "scikit-learn==1.6.1",
+    "spacy==3.8.4",
+    # Direct URL dependency for spacy model
+    "en_core_web_lg @ https://github.com/explosion/spacy-models/releases/download/en_core_web_lg-3.8.0/en_core_web_lg-3.8.0.tar.gz",
+    "gradio==5.27.1",
+    "boto3==1.38.4",
+    "pyarrow==19.0.1",
+    "openpyxl==3.1.5",
+    "Faker==36.1.1",
+    "python-levenshtein==0.26.1",
+    "spaczz==0.6.1",
+    # Direct URL dependency for gradio_image_annotator wheel
+    "gradio_image_annotation @ https://github.com/seanpedrick-case/gradio_image_annotator/releases/download/v0.3.2/gradio_image_annotation-0.3.2-py3-none-any.whl",
+    "rapidfuzz==3.12.1",
+    "python-dotenv==1.0.1",
+    "numpy==1.26.4",
+    "awslambdaric==3.0.1"
+]
+
+[project.urls]
+Homepage = "https://seanpedrick-case.github.io/doc_redaction/README.html"
+repository = "https://github.com/seanpedrick-case/doc_redaction"
+
+[project.optional-dependencies]
+dev = ["pytest"]
+
+# Optional: You can add configuration for tools used in your project under the [tool] section
+# For example, configuration for a linter like Ruff:
+[tool.ruff]
+line-length = 88
+select = ["E", "F", "I"]
+
+# Optional: Configuration for a formatter like Black:
+[tool.black]
+line-length = 88
+target-version = ['py310']

requirements.txt

@@ -10,8 +10,8 @@ pandas==2.2.3
 scikit-learn==1.6.1
 spacy==3.8.4
 en_core_web_lg @ https://github.com/explosion/spacy-models/releases/download/en_core_web_lg-3.8.0/en_core_web_lg-3.8.0.tar.gz
-gradio==5.25.2
-boto3==1.37.29
+gradio==5.27.1
+boto3==1.38.4
 pyarrow==19.0.1
 openpyxl==3.1.5
 Faker==36.1.1

tools/aws_functions.py

@@ -3,7 +3,7 @@ import pandas as pd
 import boto3
 import tempfile
 import os
-from tools.config import AWS_REGION, RUN_AWS_FUNCTIONS, DOCUMENT_REDACTION_BUCKET
+from tools.config import AWS_REGION, RUN_AWS_FUNCTIONS, DOCUMENT_REDACTION_BUCKET, SAVE_LOGS_TO_CSV
 PandasDataFrame = Type[pd.DataFrame]
 
 def get_assumed_role_info():
@@ -174,3 +174,59 @@ def upload_file_to_s3(local_file_paths:List[str], s3_key:str, s3_bucket:str=DOCU
         final_out_message_str = "App not set to run AWS functions"
 
     return final_out_message_str
+
+
+def upload_log_file_to_s3(local_file_paths:List[str], s3_key:str, s3_bucket:str=DOCUMENT_REDACTION_BUCKET, RUN_AWS_FUNCTIONS:str = RUN_AWS_FUNCTIONS, SAVE_LOGS_TO_CSV:str=SAVE_LOGS_TO_CSV):
+    """
+    Uploads a log file from local machine to Amazon S3.
+
+    Args:
+    - local_file_path: Local file path(s) of the file(s) to upload.
+    - s3_key: Key (path) to the file in the S3 bucket.
+    - s3_bucket: Name of the S3 bucket.
+
+    Returns:
+    - Message as variable/printed to console
+    """
+    final_out_message = []
+    final_out_message_str = ""
+
+    if RUN_AWS_FUNCTIONS == "1" and SAVE_LOGS_TO_CSV == "True":
+        try:
+            if s3_bucket and s3_key and local_file_paths:
+
+                s3_client = boto3.client('s3', region_name=AWS_REGION)
+
+                if isinstance(local_file_paths, str):
+                    local_file_paths = [local_file_paths]
+
+                for file in local_file_paths:
+                    if s3_client:
+                        #print(s3_client)
+                        try:
+                            # Get file name off file path
+                            file_name = os.path.basename(file)
+
+                            s3_key_full = s3_key + file_name
+                            print("S3 key: ", s3_key_full)
+
+                            s3_client.upload_file(file, s3_bucket, s3_key_full)
+                            out_message = "File " + file_name + " uploaded successfully!"
+                            print(out_message)
+                        
+                        except Exception as e:
+                            out_message = f"Error uploading file(s): {e}"
+                            print(out_message)
+
+                        final_out_message.append(out_message)
+                        final_out_message_str = '\n'.join(final_out_message)
+
+                    else: final_out_message_str = "Could not connect to AWS."
+            else: final_out_message_str = "At least one essential variable is empty, could not upload to S3"
+        except Exception as e:
+            final_out_message_str = "Could not upload files to S3 due to: " + str(e)
+            print(final_out_message_str)
+    else:
+        final_out_message_str = "App not set to run AWS functions"
+
+    return final_out_message_str

tools/aws_textract.py

@@ -108,6 +108,174 @@ def convert_pike_pdf_page_to_bytes(pdf:object, page_num:int):
 
     return pdf_bytes
 
+# def json_to_ocrresult(json_data:dict, page_width:float, page_height:float, page_no:int):
+#     '''
+#     Convert the json response from textract to the OCRResult format used elsewhere in the code. Looks for lines, words, and signatures. Handwriting and signatures are set aside especially for later in case the user wants to override the default behaviour and redact all handwriting/signatures.
+#     '''
+#     all_ocr_results = []
+#     signature_or_handwriting_recogniser_results = []
+#     signature_recogniser_results = []
+#     handwriting_recogniser_results = []
+#     signatures = []
+#     handwriting = []
+#     ocr_results_with_words = {}
+#     text_block={}
+
+#     i = 1
+
+#     # Assuming json_data is structured as a dictionary with a "pages" key
+#     #if "pages" in json_data:
+#     # Find the specific page data
+#     page_json_data = json_data #next((page for page in json_data["pages"] if page["page_no"] == page_no), None)
+
+#     #print("page_json_data:", page_json_data)
+
+#     if "Blocks" in page_json_data:
+#         # Access the data for the specific page
+#         text_blocks = page_json_data["Blocks"]  # Access the Blocks within the page data
+#     # This is a new page
+#     elif "page_no" in page_json_data:
+#         text_blocks = page_json_data["data"]["Blocks"]
+#     else: text_blocks = []
+
+#     is_signature = False
+#     is_handwriting = False
+
+#     for text_block in text_blocks:
+    
+#         if (text_block['BlockType'] == 'LINE') | (text_block['BlockType'] == 'SIGNATURE'): # (text_block['BlockType'] == 'WORD') |
+
+#             # Extract text and bounding box for the line
+#             line_bbox = text_block["Geometry"]["BoundingBox"]
+#             line_left = int(line_bbox["Left"] * page_width)
+#             line_top = int(line_bbox["Top"] * page_height)
+#             line_right = int((line_bbox["Left"] + line_bbox["Width"]) * page_width)
+#             line_bottom = int((line_bbox["Top"] + line_bbox["Height"]) * page_height)
+
+#             width_abs = int(line_bbox["Width"] * page_width)
+#             height_abs = int(line_bbox["Height"] * page_height)
+
+#             if text_block['BlockType'] == 'LINE':
+                
+#                 # Extract text and bounding box for the line
+#                 line_text = text_block.get('Text', '')
+#                 words = []
+#                 current_line_handwriting_results = []  # Track handwriting results for this line
+
+#                 if 'Relationships' in text_block:
+#                     for relationship in text_block['Relationships']:
+#                         if relationship['Type'] == 'CHILD':
+#                             for child_id in relationship['Ids']:
+#                                 child_block = next((block for block in text_blocks if block['Id'] == child_id), None)
+#                                 if child_block and child_block['BlockType'] == 'WORD':
+#                                     word_text = child_block.get('Text', '')
+#                                     word_bbox = child_block["Geometry"]["BoundingBox"]
+#                                     confidence = child_block.get('Confidence','')
+#                                     word_left = int(word_bbox["Left"] * page_width)
+#                                     word_top = int(word_bbox["Top"] * page_height)
+#                                     word_right = int((word_bbox["Left"] + word_bbox["Width"]) * page_width)
+#                                     word_bottom = int((word_bbox["Top"] + word_bbox["Height"]) * page_height)
+
+#                                     # Extract BoundingBox details
+#                                     word_width = word_bbox["Width"]
+#                                     word_height = word_bbox["Height"]
+
+#                                     # Convert proportional coordinates to absolute coordinates
+#                                     word_width_abs = int(word_width * page_width)
+#                                     word_height_abs = int(word_height * page_height)
+                                    
+#                                     words.append({
+#                                         'text': word_text,
+#                                         'bounding_box': (word_left, word_top, word_right, word_bottom)
+#                                     })
+#                                     # Check for handwriting
+#                                     text_type = child_block.get("TextType", '')
+
+#                                     if text_type == "HANDWRITING":
+#                                         is_handwriting = True
+#                                         entity_name = "HANDWRITING"
+#                                         word_end = len(word_text)
+
+#                                         recogniser_result = CustomImageRecognizerResult(
+#                                             entity_type=entity_name,
+#                                             text=word_text,
+#                                             score=confidence,
+#                                             start=0,
+#                                             end=word_end,
+#                                             left=word_left,
+#                                             top=word_top,
+#                                             width=word_width_abs,
+#                                             height=word_height_abs
+#                                         )
+
+#                                         # Add to handwriting collections immediately
+#                                         handwriting.append(recogniser_result)
+#                                         handwriting_recogniser_results.append(recogniser_result)
+#                                         signature_or_handwriting_recogniser_results.append(recogniser_result)
+#                                         current_line_handwriting_results.append(recogniser_result)
+
+#             # If handwriting or signature, add to bounding box               
+
+#             elif (text_block['BlockType'] == 'SIGNATURE'):
+#                 line_text = "SIGNATURE"
+#                 is_signature = True
+#                 entity_name = "SIGNATURE"
+#                 confidence = text_block.get('Confidence', 0)
+#                 word_end = len(line_text)
+
+#                 recogniser_result = CustomImageRecognizerResult(
+#                     entity_type=entity_name,
+#                     text=line_text,
+#                     score=confidence,
+#                     start=0,
+#                     end=word_end,
+#                     left=line_left,
+#                     top=line_top,
+#                     width=width_abs,
+#                     height=height_abs
+#                 )
+
+#                 # Add to signature collections immediately
+#                 signatures.append(recogniser_result)
+#                 signature_recogniser_results.append(recogniser_result)
+#                 signature_or_handwriting_recogniser_results.append(recogniser_result)
+
+#                 words = [{
+#                     'text': line_text,
+#                     'bounding_box': (line_left, line_top, line_right, line_bottom)
+#                 }]
+
+#             ocr_results_with_words["text_line_" + str(i)] = {
+#                 "line": i,
+#                 'text': line_text,
+#                 'bounding_box': (line_left, line_top, line_right, line_bottom),
+#                 'words': words
+#             }         
+
+#             # Create OCRResult with absolute coordinates
+#             ocr_result = OCRResult(line_text, line_left, line_top, width_abs, height_abs)
+#             all_ocr_results.append(ocr_result)
+
+#             is_signature_or_handwriting = is_signature | is_handwriting
+
+#             # If it is signature or handwriting, will overwrite the default behaviour of the PII analyser
+#             if is_signature_or_handwriting:
+#                 if recogniser_result not in signature_or_handwriting_recogniser_results: 
+#                     signature_or_handwriting_recogniser_results.append(recogniser_result)
+
+#                 if is_signature:
+#                     if recogniser_result not in signature_recogniser_results:          
+#                         signature_recogniser_results.append(recogniser_result)
+
+#                 if is_handwriting: 
+#                     if recogniser_result not in handwriting_recogniser_results: 
+#                         handwriting_recogniser_results.append(recogniser_result)
+
+#             i += 1
+
+#     return all_ocr_results, signature_or_handwriting_recogniser_results, signature_recogniser_results, handwriting_recogniser_results, ocr_results_with_words
+
+
 def json_to_ocrresult(json_data:dict, page_width:float, page_height:float, page_no:int):
     '''
     Convert the json response from textract to the OCRResult format used elsewhere in the code. Looks for lines, words, and signatures. Handwriting and signatures are set aside especially for later in case the user wants to override the default behaviour and redact all handwriting/signatures.
@@ -118,7 +286,7 @@ def json_to_ocrresult(json_data:dict, page_width:float, page_height:float, page_
     handwriting_recogniser_results = []
     signatures = []
     handwriting = []
-    ocr_results_with_children = {}
+    ocr_results_with_words = {}
     text_block={}
 
     i = 1
@@ -141,7 +309,7 @@ def json_to_ocrresult(json_data:dict, page_width:float, page_height:float, page_
     is_signature = False
     is_handwriting = False
 
-    for text_block in text_blocks:
+    for text_block in text_blocks:    
     
         if (text_block['BlockType'] == 'LINE') | (text_block['BlockType'] == 'SIGNATURE'): # (text_block['BlockType'] == 'WORD') |
 
@@ -244,36 +412,53 @@ def json_to_ocrresult(json_data:dict, page_width:float, page_height:float, page_
                     'text': line_text,
                     'bounding_box': (line_left, line_top, line_right, line_bottom)
                 }]
-
-            ocr_results_with_children["text_line_" + str(i)] = {
+        else:
+            line_text = ""
+            words=[]
+            line_left = 0
+            line_top = 0
+            line_right = 0
+            line_bottom = 0
+            width_abs = 0
+            height_abs = 0
+
+        if line_text:
+
+            ocr_results_with_words["text_line_" + str(i)] = {
                 "line": i,
                 'text': line_text,
                 'bounding_box': (line_left, line_top, line_right, line_bottom),
-                'words': words
-            }         
+                'words': words,
+                'page': page_no
+            }
 
             # Create OCRResult with absolute coordinates
             ocr_result = OCRResult(line_text, line_left, line_top, width_abs, height_abs)
             all_ocr_results.append(ocr_result)
 
-            is_signature_or_handwriting = is_signature | is_handwriting
+        is_signature_or_handwriting = is_signature | is_handwriting
+
+        # If it is signature or handwriting, will overwrite the default behaviour of the PII analyser
+        if is_signature_or_handwriting:
+            if recogniser_result not in signature_or_handwriting_recogniser_results: 
+                signature_or_handwriting_recogniser_results.append(recogniser_result)
+
+            if is_signature:
+                if recogniser_result not in signature_recogniser_results:          
+                    signature_recogniser_results.append(recogniser_result)
 
-            # If it is signature or handwriting, will overwrite the default behaviour of the PII analyser
-            if is_signature_or_handwriting:
-                if recogniser_result not in signature_or_handwriting_recogniser_results: 
-                    signature_or_handwriting_recogniser_results.append(recogniser_result)
+            if is_handwriting: 
+                if recogniser_result not in handwriting_recogniser_results: 
+                    handwriting_recogniser_results.append(recogniser_result)
 
-                if is_signature:
-                    if recogniser_result not in signature_recogniser_results:          
-                        signature_recogniser_results.append(recogniser_result)
+        i += 1
 
-                if is_handwriting: 
-                    if recogniser_result not in handwriting_recogniser_results: 
-                        handwriting_recogniser_results.append(recogniser_result)
+    # Add page key to the line level results
+    all_ocr_results_with_page = {"page": page_no, "results": all_ocr_results}
+    ocr_results_with_words_with_page = {"page": page_no, "results": ocr_results_with_words}
 
-            i += 1
+    return all_ocr_results_with_page, signature_or_handwriting_recogniser_results, signature_recogniser_results, handwriting_recogniser_results, ocr_results_with_words_with_page
 
-    return all_ocr_results, signature_or_handwriting_recogniser_results, signature_recogniser_results, handwriting_recogniser_results, ocr_results_with_children
 
 def load_and_convert_textract_json(textract_json_file_path:str, log_files_output_paths:str, page_sizes_df:pd.DataFrame):
     """
@@ -315,7 +500,7 @@ def load_and_convert_textract_json(textract_json_file_path:str, log_files_output
             return {}, True, log_files_output_paths  # Conversion failed
     else:
         print("Invalid Textract JSON format: 'Blocks' missing.")
-        print("textract data:", textract_data)
+        #print("textract data:", textract_data)
         return {}, True, log_files_output_paths  # Return empty data if JSON is not recognized
 
 def restructure_textract_output(textract_output: dict, page_sizes_df:pd.DataFrame):

tools/config.py

@@ -108,19 +108,7 @@ if AWS_SECRET_KEY: print(f'AWS_SECRET_KEY found in environment variables')
 
 DOCUMENT_REDACTION_BUCKET = get_or_create_env_var('DOCUMENT_REDACTION_BUCKET', '')
 
-SHOW_BULK_TEXTRACT_CALL_OPTIONS = get_or_create_env_var('SHOW_BULK_TEXTRACT_CALL_OPTIONS', 'False') # This feature not currently implemented
 
-TEXTRACT_BULK_ANALYSIS_BUCKET = get_or_create_env_var('TEXTRACT_BULK_ANALYSIS_BUCKET', '')
-
-TEXTRACT_BULK_ANALYSIS_INPUT_SUBFOLDER = get_or_create_env_var('TEXTRACT_BULK_ANALYSIS_INPUT_SUBFOLDER', 'input')
-
-TEXTRACT_BULK_ANALYSIS_OUTPUT_SUBFOLDER = get_or_create_env_var('TEXTRACT_BULK_ANALYSIS_OUTPUT_SUBFOLDER', 'output')
-
-LOAD_PREVIOUS_TEXTRACT_JOBS_S3 = get_or_create_env_var('LOAD_PREVIOUS_TEXTRACT_JOBS_S3', 'False') # Whether or not to load previous Textract jobs from S3
-
-TEXTRACT_JOBS_S3_LOC = get_or_create_env_var('TEXTRACT_JOBS_S3_LOC', 'output') # Subfolder in the DOCUMENT_REDACTION_BUCKET where the Textract jobs are stored
-
-TEXTRACT_JOBS_LOCAL_LOC = get_or_create_env_var('TEXTRACT_JOBS_LOCAL_LOC', 'output') # Local subfolder where the Textract jobs are stored
 
 # Custom headers e.g. if routing traffic through Cloudfront
 # Retrieving or setting CUSTOM_HEADER
@@ -161,6 +149,8 @@ if OUTPUT_FOLDER == "TEMP" or INPUT_FOLDER == "TEMP":
 # By default, logs are put into a subfolder of today's date and the host name of the instance running the app. This is to avoid at all possible the possibility of log files from one instance overwriting the logs of another instance on S3. If running the app on one system always, or just locally, it is not necessary to make the log folders so specific.
 # Another way to address this issue would be to write logs to another type of storage, e.g. database such as dynamodb. I may look into this in future.
 
+SAVE_LOGS_TO_CSV = get_or_create_env_var('SAVE_LOGS_TO_CSV', 'True')
+
 USE_LOG_SUBFOLDERS = get_or_create_env_var('USE_LOG_SUBFOLDERS', 'True')
 
 if USE_LOG_SUBFOLDERS == "True":
@@ -181,8 +171,28 @@ ensure_folder_exists(USAGE_LOGS_FOLDER)
 # Should the redacted file name be included in the logs? In some instances, the names of the files themselves could be sensitive, and should not be disclosed beyond the app. So, by default this is false.
 DISPLAY_FILE_NAMES_IN_LOGS = get_or_create_env_var('DISPLAY_FILE_NAMES_IN_LOGS', 'False')
 
+# Further customisation options for CSV logs
+
+CSV_ACCESS_LOG_HEADERS = get_or_create_env_var('CSV_ACCESS_LOG_HEADERS', '') # If blank, uses component labels
+CSV_FEEDBACK_LOG_HEADERS = get_or_create_env_var('CSV_FEEDBACK_LOG_HEADERS', '') # If blank, uses component labels
+CSV_USAGE_LOG_HEADERS = get_or_create_env_var('CSV_USAGE_LOG_HEADERS', '["session_hash_textbox",	"doc_full_file_name_textbox",	"data_full_file_name_textbox",	"actual_time_taken_number",	"total_page_count",	"textract_query_number", "pii_detection_method", "comprehend_query_number",  "cost_code", "textract_handwriting_signature", "host_name_textbox", "text_extraction_method", "is_this_a_textract_api_call"]') # If blank, uses component labels
+
+### DYNAMODB logs. Whether to save to DynamoDB, and the headers of the table
+
+SAVE_LOGS_TO_DYNAMODB = get_or_create_env_var('SAVE_LOGS_TO_DYNAMODB', 'False')
+
+ACCESS_LOG_DYNAMODB_TABLE_NAME = get_or_create_env_var('ACCESS_LOG_DYNAMODB_TABLE_NAME', 'redaction_access_log')
+DYNAMODB_ACCESS_LOG_HEADERS = get_or_create_env_var('DYNAMODB_ACCESS_LOG_HEADERS', '')
+
+FEEDBACK_LOG_DYNAMODB_TABLE_NAME = get_or_create_env_var('FEEDBACK_LOG_DYNAMODB_TABLE_NAME', 'redaction_feedback')
+DYNAMODB_FEEDBACK_LOG_HEADERS = get_or_create_env_var('DYNAMODB_FEEDBACK_LOG_HEADERS', '')
+
+USAGE_LOG_DYNAMODB_TABLE_NAME = get_or_create_env_var('USAGE_LOG_DYNAMODB_TABLE_NAME', 'redaction_usage')
+DYNAMODB_USAGE_LOG_HEADERS = get_or_create_env_var('DYNAMODB_USAGE_LOG_HEADERS', '')
+
+###
+# REDACTION 
 ###
-# REDACTION CONFIG
 
 # Create Tesseract and Poppler folders if you have installed them locally
 TESSERACT_FOLDER = get_or_create_env_var('TESSERACT_FOLDER', "") # e.g. tesseract/
@@ -226,7 +236,7 @@ ROOT_PATH = get_or_create_env_var('ROOT_PATH', '')
 
 DEFAULT_CONCURRENCY_LIMIT = get_or_create_env_var('DEFAULT_CONCURRENCY_LIMIT', '3')
 
-GET_DEFAULT_ALLOW_LIST = get_or_create_env_var('GET_DEFAULT_ALLOW_LIST', 'False')
+GET_DEFAULT_ALLOW_LIST = get_or_create_env_var('GET_DEFAULT_ALLOW_LIST', '')
 
 ALLOW_LIST_PATH = get_or_create_env_var('ALLOW_LIST_PATH', '') # config/default_allow_list.csv
 
@@ -235,19 +245,38 @@ S3_ALLOW_LIST_PATH = get_or_create_env_var('S3_ALLOW_LIST_PATH', '') # default_a
 if ALLOW_LIST_PATH: OUTPUT_ALLOW_LIST_PATH = ALLOW_LIST_PATH
 else: OUTPUT_ALLOW_LIST_PATH = 'config/default_allow_list.csv'
 
+### COST CODE OPTIONS
+
 SHOW_COSTS = get_or_create_env_var('SHOW_COSTS', 'False')
 
-GET_COST_CODES = get_or_create_env_var('GET_COST_CODES', 'False')
+GET_COST_CODES = get_or_create_env_var('GET_COST_CODES', 'True')
 
 DEFAULT_COST_CODE = get_or_create_env_var('DEFAULT_COST_CODE', '')
 
 COST_CODES_PATH = get_or_create_env_var('COST_CODES_PATH', '') # 'config/COST_CENTRES.csv' # file should be a csv file with a single table in it that has two columns with a header. First column should contain cost codes, second column should contain a name or description for the cost code
 
-S3_COST_CODES_PATH = get_or_create_env_var('S3_COST_CODES_PATH', '') # COST_CENTRES.csv # This is a path within the DOCUMENT_REDACTION_BUCKET
-
+S3_COST_CODES_PATH = get_or_create_env_var('S3_COST_CODES_PATH', '') # COST_CENTRES.csv # This is a path within the DOCUMENT_REDACTION_BUCKET  
+    
+# A default path in case s3 cost code location is provided but no local cost code location given
 if COST_CODES_PATH: OUTPUT_COST_CODES_PATH = COST_CODES_PATH
-else: OUTPUT_COST_CODES_PATH = 'config/COST_CENTRES.csv'
+else: OUTPUT_COST_CODES_PATH = 'config/cost_codes.csv'
 
 ENFORCE_COST_CODES = get_or_create_env_var('ENFORCE_COST_CODES', 'False') # If you have cost codes listed, is it compulsory to choose one before redacting?
 
-if ENFORCE_COST_CODES == 'True': GET_COST_CODES = 'True'
+if ENFORCE_COST_CODES == 'True': GET_COST_CODES = 'True'
+
+### WHOLE DOCUMENT API OPTIONS
+
+SHOW_WHOLE_DOCUMENT_TEXTRACT_CALL_OPTIONS = get_or_create_env_var('SHOW_WHOLE_DOCUMENT_TEXTRACT_CALL_OPTIONS', 'False') # This feature not currently implemented
+
+TEXTRACT_WHOLE_DOCUMENT_ANALYSIS_BUCKET = get_or_create_env_var('TEXTRACT_WHOLE_DOCUMENT_ANALYSIS_BUCKET', '')
+
+TEXTRACT_WHOLE_DOCUMENT_ANALYSIS_INPUT_SUBFOLDER = get_or_create_env_var('TEXTRACT_WHOLE_DOCUMENT_ANALYSIS_INPUT_SUBFOLDER', 'input')
+
+TEXTRACT_WHOLE_DOCUMENT_ANALYSIS_OUTPUT_SUBFOLDER = get_or_create_env_var('TEXTRACT_WHOLE_DOCUMENT_ANALYSIS_OUTPUT_SUBFOLDER', 'output')
+
+LOAD_PREVIOUS_TEXTRACT_JOBS_S3 = get_or_create_env_var('LOAD_PREVIOUS_TEXTRACT_JOBS_S3', 'False') # Whether or not to load previous Textract jobs from S3
+
+TEXTRACT_JOBS_S3_LOC = get_or_create_env_var('TEXTRACT_JOBS_S3_LOC', 'output') # Subfolder in the DOCUMENT_REDACTION_BUCKET where the Textract jobs are stored
+
+TEXTRACT_JOBS_LOCAL_LOC = get_or_create_env_var('TEXTRACT_JOBS_LOCAL_LOC', 'output') # Local subfolder where the Textract jobs are stored

tools/custom_csvlogger.py

@@ -4,6 +4,10 @@ import csv
 import datetime
 import os
 import re
+import boto3
+import botocore
+import uuid
+import time
 from collections.abc import Sequence
 from multiprocessing import Lock
 from pathlib import Path
@@ -11,6 +15,9 @@ from typing import TYPE_CHECKING, Any
 from gradio_client import utils as client_utils
 import gradio as gr
 from gradio import utils, wasm_utils
+from tools.config import AWS_REGION, AWS_ACCESS_KEY, AWS_SECRET_KEY, RUN_AWS_FUNCTIONS
+from botocore.exceptions import NoCredentialsError, TokenRetrievalError
+
 
 if TYPE_CHECKING:
     from gradio.components import Component
@@ -62,21 +69,28 @@ class CSVLogger_custom(FlaggingCallback):
         self.flagging_dir = Path(flagging_dir)
         self.first_time = True
 
-    def _create_dataset_file(self, additional_headers: list[str] | None = None):
+    def _create_dataset_file(
+    self, 
+    additional_headers: list[str] | None = None,
+    replacement_headers: list[str] | None = None
+):
         os.makedirs(self.flagging_dir, exist_ok=True)
 
-        if additional_headers is None:
-            additional_headers = []
-        headers = (
-            [
+        if replacement_headers:
+            if len(replacement_headers) != len(self.components):
+                raise ValueError(
+                    f"replacement_headers must have the same length as components "
+                    f"({len(replacement_headers)} provided, {len(self.components)} expected)"
+                )
+            headers = replacement_headers + ["timestamp"]
+        else:
+            if additional_headers is None:
+                additional_headers = []
+            headers = [
                 getattr(component, "label", None) or f"component {idx}"
                 for idx, component in enumerate(self.components)
-            ]
-            + additional_headers
-            + [
-                "timestamp",
-            ]
-        )
+            ] + additional_headers + ["timestamp"]
+
         headers = utils.sanitize_list_for_csv(headers)
         dataset_files = list(Path(self.flagging_dir).glob("dataset*.csv"))
 
@@ -115,18 +129,24 @@ class CSVLogger_custom(FlaggingCallback):
             print("Using existing dataset file at:", self.dataset_filepath)
 
     def flag(
-        self,
-        flag_data: list[Any],
-        flag_option: str | None = None,
-        username: str | None = None,
-    ) -> int:
+    self,
+    flag_data: list[Any],
+    flag_option: str | None = None,
+    username: str | None = None,
+    save_to_csv: bool = True,
+    save_to_dynamodb: bool = False,
+    dynamodb_table_name: str | None = None,
+    dynamodb_headers: list[str] | None = None,  # New: specify headers for DynamoDB
+    replacement_headers: list[str] | None = None
+) -> int:
         if self.first_time:
             additional_headers = []
             if flag_option is not None:
                 additional_headers.append("flag")
             if username is not None:
                 additional_headers.append("username")
-            self._create_dataset_file(additional_headers=additional_headers)
+            additional_headers.append("id")
+            self._create_dataset_file(additional_headers=additional_headers, replacement_headers=replacement_headers)
             self.first_time = False
 
         csv_data = []
@@ -155,15 +175,131 @@ class CSVLogger_custom(FlaggingCallback):
             csv_data.append(flag_option)
         if username is not None:
             csv_data.append(username)
-        csv_data.append(str(datetime.datetime.now()))
 
-        with self.lock:
-            with open(
-                self.dataset_filepath, "a", newline="", encoding="utf-8"
-            ) as csvfile:
-                writer = csv.writer(csvfile)
-                writer.writerow(utils.sanitize_list_for_csv(csv_data))
-            with open(self.dataset_filepath, encoding="utf-8") as csvfile:
-                line_count = len(list(csv.reader(csvfile))) - 1
+
+        timestamp = str(datetime.datetime.now())
+        csv_data.append(timestamp)
+
+        generated_id = str(uuid.uuid4())
+        csv_data.append(generated_id)
+
+        # Build the headers
+        headers = (
+            [getattr(component, "label", None) or f"component {idx}" for idx, component in enumerate(self.components)]
+        )
+        if flag_option is not None:
+            headers.append("flag")
+        if username is not None:
+            headers.append("username")
+        headers.append("timestamp")
+        headers.append("id")
+
+        line_count = -1
+
+        if save_to_csv:
+            with self.lock:
+                with open(self.dataset_filepath, "a", newline="", encoding="utf-8") as csvfile:
+                    writer = csv.writer(csvfile)
+                    writer.writerow(utils.sanitize_list_for_csv(csv_data))
+                with open(self.dataset_filepath, encoding="utf-8") as csvfile:
+                    line_count = len(list(csv.reader(csvfile))) - 1
+
+        if save_to_dynamodb == True:
+
+            if RUN_AWS_FUNCTIONS == "1":
+                try:
+                    print("Connecting to DynamoDB via existing SSO connection")
+                    dynamodb = boto3.resource('dynamodb', region_name=AWS_REGION)
+                    #client = boto3.client('dynamodb')
+                    
+                    test_connection = dynamodb.meta.client.list_tables()                   
+
+                except Exception as e:
+                    print("No SSO credentials found:", e)                    
+                    if AWS_ACCESS_KEY and AWS_SECRET_KEY:
+                        print("Trying DynamoDB credentials from environment variables")
+                        dynamodb = boto3.resource('dynamodb',aws_access_key_id=AWS_ACCESS_KEY, 
+                            aws_secret_access_key=AWS_SECRET_KEY, region_name=AWS_REGION)
+                        # client = boto3.client('dynamodb',aws_access_key_id=AWS_ACCESS_KEY, 
+                        #     aws_secret_access_key=AWS_SECRET_KEY, region_name=AWS_REGION)
+                    else:
+                        raise Exception("AWS credentials for DynamoDB logging not found")
+            else:
+                raise Exception("AWS credentials for DynamoDB logging not found")
+            
+            if dynamodb_table_name is None:
+                raise ValueError("You must provide a dynamodb_table_name if save_to_dynamodb is True")            
+            
+            if dynamodb_headers:
+                dynamodb_headers = dynamodb_headers
+            if not dynamodb_headers and replacement_headers:
+                dynamodb_headers = replacement_headers
+            elif headers:
+                dynamodb_headers = headers
+            elif not dynamodb_headers:
+                raise ValueError("Headers not found. You must provide dynamodb_headers or replacement_headers to create a new table.")
+            
+            if flag_option is not None:
+                if "flag" not in dynamodb_headers:
+                    dynamodb_headers.append("flag")
+                if username is not None:
+                    if "username" not in dynamodb_headers:
+                        dynamodb_headers.append("username")
+                if "timestamp" not in dynamodb_headers:
+                    dynamodb_headers.append("timestamp")
+                if "id" not in dynamodb_headers:
+                    dynamodb_headers.append("id")
+
+            # Table doesn't exist — create it
+            try:
+                table = dynamodb.Table(dynamodb_table_name)
+                table.load()
+            except botocore.exceptions.ClientError as e:
+                if e.response['Error']['Code'] == 'ResourceNotFoundException':
+                    
+                    #print(f"Creating DynamoDB table '{dynamodb_table_name}'...")
+                    #print("dynamodb_headers:", dynamodb_headers)
+                    
+                    attribute_definitions = [
+                        {'AttributeName': 'id', 'AttributeType': 'S'}  # Only define key attributes here
+                    ]
+
+                    table = dynamodb.create_table(
+                        TableName=dynamodb_table_name,
+                        KeySchema=[
+                            {'AttributeName': 'id', 'KeyType': 'HASH'}  # Partition key
+                        ],
+                        AttributeDefinitions=attribute_definitions,
+                        BillingMode='PAY_PER_REQUEST'
+)
+                    # Wait until the table exists
+                    table.meta.client.get_waiter('table_exists').wait(TableName=dynamodb_table_name)
+                    time.sleep(5)
+                    print(f"Table '{dynamodb_table_name}' created successfully.")
+                else:
+                    raise
+
+            # Prepare the DynamoDB item to upload
+
+            try:
+                item = {
+                    'id': str(generated_id),  # UUID primary key
+                    #'created_by': username if username else "unknown",
+                    'timestamp': timestamp,
+                }
+
+                #print("dynamodb_headers:", dynamodb_headers)
+                #print("csv_data:", csv_data)
+
+                # Map the headers to values
+                item.update({header: str(value) for header, value in zip(dynamodb_headers, csv_data)})
+
+                #print("item:", item)
+
+                table.put_item(Item=item)
+
+                print("Successfully uploaded log to DynamoDB")
+            except Exception as e:
+                print("Could not upload log to DynamobDB due to", e)
 
         return line_count

tools/custom_image_analyser_engine.py

@@ -775,9 +775,52 @@ def merge_text_bounding_boxes(analyser_results:dict, characters: List[LTChar], c
 
     return analysed_bounding_boxes
 
-# Function to combine OCR results into line-level results
-def combine_ocr_results(ocr_results:dict, x_threshold:float=50.0, y_threshold:float=12.0):
-    # Group OCR results into lines based on y_threshold
+def recreate_page_line_level_ocr_results_with_page(page_line_level_ocr_results_with_words: dict):
+    reconstructed_results = []
+    
+    # Assume all lines belong to the same page, so we can just read it from one item
+    #page = next(iter(page_line_level_ocr_results_with_words.values()))["page"]
+
+    page = page_line_level_ocr_results_with_words["page"]
+    
+    for line_data in page_line_level_ocr_results_with_words["results"].values():
+        bbox = line_data["bounding_box"]
+        text = line_data["text"]
+
+        # Recreate the OCRResult (you'll need the OCRResult class imported)
+        line_result = OCRResult(
+            text=text,
+            left=bbox[0],
+            top=bbox[1],
+            width=bbox[2] - bbox[0],
+            height=bbox[3] - bbox[1],
+        )
+        reconstructed_results.append(line_result)
+    
+    page_line_level_ocr_results_with_page = {"page": page, "results": reconstructed_results}
+    
+    return page_line_level_ocr_results_with_page
+
+def create_ocr_result_with_children(combined_results:dict, i:int, current_bbox:dict, current_line:list):
+        combined_results["text_line_" + str(i)] = {
+        "line": i,
+        'text': current_bbox.text,
+        'bounding_box': (current_bbox.left, current_bbox.top, 
+                            current_bbox.left + current_bbox.width, 
+                            current_bbox.top + current_bbox.height),
+        'words': [{'text': word.text, 
+                    'bounding_box': (word.left, word.top, 
+                                    word.left + word.width, 
+                                    word.top + word.height)} 
+                    for word in current_line]
+    }
+        return combined_results["text_line_" + str(i)]
+
+def combine_ocr_results(ocr_results: dict, x_threshold: float = 50.0, y_threshold: float = 12.0, page: int = 1):
+    '''
+    Group OCR results into lines based on y_threshold. Create line level ocr results, and word level OCR results
+    '''
+
     lines = []
     current_line = []
     for result in sorted(ocr_results, key=lambda x: x.top):
@@ -796,26 +839,11 @@ def combine_ocr_results(ocr_results:dict, x_threshold:float=50.0, y_threshold:fl
     # Flatten the sorted lines back into a single list
     sorted_results = [result for line in lines for result in line]
 
-    combined_results = []
-    new_format_results = {}
+    page_line_level_ocr_results = []
+    page_line_level_ocr_results_with_words = {}
     current_line = []
     current_bbox = None
-    line_counter = 1
-
-    def create_ocr_result_with_children(combined_results, i, current_bbox, current_line):
-        combined_results["text_line_" + str(i)] = {
-        "line": i,
-        'text': current_bbox.text,
-        'bounding_box': (current_bbox.left, current_bbox.top, 
-                            current_bbox.left + current_bbox.width, 
-                            current_bbox.top + current_bbox.height),
-        'words': [{'text': word.text, 
-                    'bounding_box': (word.left, word.top, 
-                                    word.left + word.width, 
-                                    word.top + word.height)} 
-                    for word in current_line]
-    }
-        return combined_results["text_line_" + str(i)]  
+    line_counter = 1      
 
     for result in sorted_results:
         if not current_line:
@@ -838,26 +866,101 @@ def combine_ocr_results(ocr_results:dict, x_threshold:float=50.0, y_threshold:fl
                     height=max(current_bbox.height, result.height)
                 )
                 current_line.append(result)
-            else:
-                
+            else:              
 
                 # Commit the current line and start a new one
-                combined_results.append(current_bbox)
+                page_line_level_ocr_results.append(current_bbox)
 
-                new_format_results["text_line_" + str(line_counter)] = create_ocr_result_with_children(new_format_results, line_counter, current_bbox, current_line)
+                page_line_level_ocr_results_with_words["text_line_" + str(line_counter)] = create_ocr_result_with_children(page_line_level_ocr_results_with_words, line_counter, current_bbox, current_line)
+                #page_line_level_ocr_results_with_words["text_line_" + str(line_counter)]["page"] = page
 
                 line_counter += 1
                 current_line = [result]
                 current_bbox = result
-
     # Append the last line
     if current_bbox:
-        combined_results.append(current_bbox)
+        page_line_level_ocr_results.append(current_bbox)
+
+        page_line_level_ocr_results_with_words["text_line_" + str(line_counter)] = create_ocr_result_with_children(page_line_level_ocr_results_with_words, line_counter, current_bbox, current_line)
+        #page_line_level_ocr_results_with_words["text_line_" + str(line_counter)]["page"] = page
 
-        new_format_results["text_line_" + str(line_counter)] = create_ocr_result_with_children(new_format_results, line_counter, current_bbox, current_line)
+    # Add page key to the line level results
+    page_line_level_ocr_results_with_page = {"page": page, "results": page_line_level_ocr_results}
+    page_line_level_ocr_results_with_words = {"page": page, "results": page_line_level_ocr_results_with_words}
 
+    return page_line_level_ocr_results_with_page, page_line_level_ocr_results_with_words
 
-    return combined_results, new_format_results
+
+# Function to combine OCR results into line-level results
+# def combine_ocr_results(ocr_results:dict, x_threshold:float=50.0, y_threshold:float=12.0):
+#     '''
+#     Group OCR results into lines based on y_threshold. Create line level ocr results, and word level OCR results
+#     '''
+
+#     lines = []
+#     current_line = []
+#     for result in sorted(ocr_results, key=lambda x: x.top):
+#         if not current_line or abs(result.top - current_line[0].top) <= y_threshold:
+#             current_line.append(result)
+#         else:
+#             lines.append(current_line)
+#             current_line = [result]
+#     if current_line:
+#         lines.append(current_line)
+
+#     # Sort each line by left position
+#     for line in lines:
+#         line.sort(key=lambda x: x.left)
+
+#     # Flatten the sorted lines back into a single list
+#     sorted_results = [result for line in lines for result in line]
+
+#     page_line_level_ocr_results = []
+#     page_line_level_ocr_results_with_words = {}
+#     current_line = []
+#     current_bbox = None
+#     line_counter = 1      
+
+#     for result in sorted_results:
+#         if not current_line:
+#             # Start a new line
+#             current_line.append(result)
+#             current_bbox = result
+#         else:
+#             # Check if the result is on the same line (y-axis) and close horizontally (x-axis)
+#             last_result = current_line[-1]
+
+#             if abs(result.top - last_result.top) <= y_threshold and \
+#                (result.left - (last_result.left + last_result.width)) <= x_threshold:
+#                 # Update the bounding box to include the new word
+#                 new_right = max(current_bbox.left + current_bbox.width, result.left + result.width)
+#                 current_bbox = OCRResult(
+#                     text=f"{current_bbox.text} {result.text}",
+#                     left=current_bbox.left,
+#                     top=current_bbox.top,
+#                     width=new_right - current_bbox.left,
+#                     height=max(current_bbox.height, result.height)
+#                 )
+#                 current_line.append(result)
+#             else:              
+
+#                 # Commit the current line and start a new one
+#                 page_line_level_ocr_results.append(current_bbox)
+
+#                 page_line_level_ocr_results_with_words["text_line_" + str(line_counter)] = create_ocr_result_with_children(page_line_level_ocr_results_with_words, line_counter, current_bbox, current_line)
+
+#                 line_counter += 1
+#                 current_line = [result]
+#                 current_bbox = result
+
+#     # Append the last line
+#     if current_bbox:
+#         page_line_level_ocr_results.append(current_bbox)
+
+#         page_line_level_ocr_results_with_words["text_line_" + str(line_counter)] = create_ocr_result_with_children(page_line_level_ocr_results_with_words, line_counter, current_bbox, current_line)
+
+
+#     return page_line_level_ocr_results, page_line_level_ocr_results_with_words
 
 class CustomImageAnalyzerEngine:
     def __init__(
@@ -911,7 +1014,7 @@ class CustomImageAnalyzerEngine:
     def analyze_text(
         self, 
         line_level_ocr_results: List[OCRResult], 
-        ocr_results_with_children: Dict[str, Dict],
+        ocr_results_with_words: Dict[str, Dict],
         chosen_redact_comprehend_entities: List[str],
         pii_identification_method: str = "Local",
         comprehend_client = "",
@@ -1036,9 +1139,9 @@ class CustomImageAnalyzerEngine:
         combined_results = []
         for i, text_line in enumerate(line_level_ocr_results):
             line_results = next((results for idx, results in all_text_line_results if idx == i), [])
-            if line_results and i < len(ocr_results_with_children):
-                child_level_key = list(ocr_results_with_children.keys())[i]
-                ocr_results_with_children_line_level = ocr_results_with_children[child_level_key]
+            if line_results and i < len(ocr_results_with_words):
+                child_level_key = list(ocr_results_with_words.keys())[i]
+                ocr_results_with_words_line_level = ocr_results_with_words[child_level_key]
                 
                 for result in line_results:
                     bbox_results = self.map_analyzer_results_to_bounding_boxes(
@@ -1052,7 +1155,7 @@ class CustomImageAnalyzerEngine:
                         )],
                         text_line.text,
                         text_analyzer_kwargs.get('allow_list', []),
-                        ocr_results_with_children_line_level
+                        ocr_results_with_words_line_level
                     )
                     combined_results.extend(bbox_results)
 
@@ -1064,14 +1167,14 @@ class CustomImageAnalyzerEngine:
     redaction_relevant_ocr_results: List[OCRResult],
     full_text: str,
     allow_list: List[str],
-    ocr_results_with_children_child_info: Dict[str, Dict]
+    ocr_results_with_words_child_info: Dict[str, Dict]
 ) -> List[CustomImageRecognizerResult]:
         redaction_bboxes = []
 
         for redaction_relevant_ocr_result in redaction_relevant_ocr_results:
-            #print("ocr_results_with_children_child_info:", ocr_results_with_children_child_info)
+            #print("ocr_results_with_words_child_info:", ocr_results_with_words_child_info)
 
-            line_text = ocr_results_with_children_child_info['text']
+            line_text = ocr_results_with_words_child_info['text']
             line_length = len(line_text)
             redaction_text = redaction_relevant_ocr_result.text
 
@@ -1097,7 +1200,7 @@ class CustomImageAnalyzerEngine:
                     
                     # print(f"Found match: '{matched_text}' in line")
 
-                    # for word_info in ocr_results_with_children_child_info.get('words', []):
+                    # for word_info in ocr_results_with_words_child_info.get('words', []):
                     #     # Check if this word is part of our match
                     #     if any(word.lower() in word_info['text'].lower() for word in matched_words):
                     #         matching_word_boxes.append(word_info['bounding_box'])
@@ -1106,11 +1209,11 @@ class CustomImageAnalyzerEngine:
                     # Find the corresponding words in the OCR results
                     matching_word_boxes = []
                     
-                    #print("ocr_results_with_children_child_info:", ocr_results_with_children_child_info)
+                    #print("ocr_results_with_words_child_info:", ocr_results_with_words_child_info)
 
                     current_position = 0
 
-                    for word_info in ocr_results_with_children_child_info.get('words', []):
+                    for word_info in ocr_results_with_words_child_info.get('words', []):
                         word_text = word_info['text']
                         word_length = len(word_text)
 

tools/data_anonymise.py

@@ -1,10 +1,12 @@
 import re
+import os
 import secrets
 import base64
 import time
 import boto3
 import botocore
 import pandas as pd
+from openpyxl import Workbook, load_workbook
 
 from faker import Faker
 from gradio import Progress
@@ -226,6 +228,7 @@ def anonymise_data_files(file_paths: List[str],
                          comprehend_query_number:int=0,
                          aws_access_key_textbox:str='',
                          aws_secret_key_textbox:str='',
+                         actual_time_taken_number:float=0,
                          progress: Progress = Progress(track_tqdm=True)):
     """
     This function anonymises data files based on the provided parameters.
@@ -252,6 +255,7 @@ def anonymise_data_files(file_paths: List[str],
     - comprehend_query_number (int, optional): A counter tracking the number of queries to AWS Comprehend.
     - aws_access_key_textbox (str, optional): AWS access key for account with Textract and Comprehend permissions.
     - aws_secret_key_textbox (str, optional): AWS secret key for account with Textract and Comprehend permissions.
+    - actual_time_taken_number (float, optional): Time taken to do the redaction.
     - progress (Progress, optional): A Progress object to track progress. Defaults to a Progress object with track_tqdm=True.
     """
     
@@ -277,9 +281,16 @@ def anonymise_data_files(file_paths: List[str],
     if not out_file_paths:
         out_file_paths = []  
     
-
-    if in_allow_list:
-        in_allow_list_flat = in_allow_list #[item for sublist in in_allow_list for item in sublist]
+    if isinstance(in_allow_list, list):
+        if in_allow_list:
+            in_allow_list_flat = in_allow_list
+        else:
+            in_allow_list_flat = []
+    elif isinstance(in_allow_list, pd.DataFrame):
+        if not in_allow_list.empty:
+            in_allow_list_flat = list(in_allow_list.iloc[:, 0].unique())
+        else:
+            in_allow_list_flat = []
     else:
         in_allow_list_flat = []
     
@@ -306,7 +317,7 @@ def anonymise_data_files(file_paths: List[str],
         else:
             comprehend_client = ""
             out_message = "Cannot connect to AWS Comprehend service. Please provide access keys under Textract settings on the Redaction settings tab, or choose another PII identification method."
-            print(out_message)
+            raise(out_message)
     
     # Check if files and text exist
     if not file_paths:
@@ -314,7 +325,7 @@ def anonymise_data_files(file_paths: List[str],
             file_paths=['open_text']
         else:
             out_message = "Please enter text or a file to redact."
-            return out_message, out_file_paths, out_file_paths, latest_file_completed, log_files_output_paths, log_files_output_paths
+            raise Exception(out_message)
         
     # If we have already redacted the last file, return the input out_message and file list to the relevant components
     if latest_file_completed >= len(file_paths):
@@ -322,18 +333,18 @@ def anonymise_data_files(file_paths: List[str],
         # Set to a very high number so as not to mess with subsequent file processing by the user
         latest_file_completed = 99
         final_out_message = '\n'.join(out_message)
-        return final_out_message, out_file_paths, out_file_paths, latest_file_completed, log_files_output_paths, log_files_output_paths
+        return final_out_message, out_file_paths, out_file_paths, latest_file_completed, log_files_output_paths, log_files_output_paths, actual_time_taken_number
     
     file_path_loop = [file_paths[int(latest_file_completed)]]
         
-    for anon_file in progress.tqdm(file_path_loop, desc="Anonymising files", unit = "file"):
+    for anon_file in progress.tqdm(file_path_loop, desc="Anonymising files", unit = "files"):
 
         if anon_file=='open_text':
             anon_df = pd.DataFrame(data={'text':[in_text]})
             chosen_cols=['text']
+            out_file_part = anon_file
             sheet_name = ""
             file_type = ""
-            out_file_part = anon_file
 
             out_file_paths, out_message, key_string, log_files_output_paths = anon_wrapper_func(anon_file, anon_df, chosen_cols, out_file_paths, out_file_part, out_message, sheet_name, anon_strat, language, chosen_redact_entities, in_allow_list, file_type, "", log_files_output_paths, in_deny_list, max_fuzzy_spelling_mistakes_num, pii_identification_method, chosen_redact_comprehend_entities, comprehend_query_number, comprehend_client, output_folder=OUTPUT_FOLDER)
         else:
@@ -350,26 +361,22 @@ def anonymise_data_files(file_paths: List[str],
                     out_message.append("No Excel sheets selected. Please select at least one to anonymise.")
                     continue
 
-                anon_xlsx = pd.ExcelFile(anon_file)                
-
                 # Create xlsx file:
-                anon_xlsx_export_file_name = output_folder + out_file_part + "_redacted.xlsx"
-
-                from openpyxl import Workbook
+                anon_xlsx = pd.ExcelFile(anon_file)                
+                anon_xlsx_export_file_name = output_folder + out_file_part + "_redacted.xlsx"                
 
-                wb = Workbook()
-                wb.save(anon_xlsx_export_file_name)
+                
 
                 # Iterate through the sheet names
-                for sheet_name in in_excel_sheets:
+                for sheet_name in progress.tqdm(in_excel_sheets, desc="Anonymising sheets", unit = "sheets"):
                     # Read each sheet into a DataFrame
                     if sheet_name not in anon_xlsx.sheet_names:
                         continue
 
                     anon_df = pd.read_excel(anon_file, sheet_name=sheet_name)
 
-                    out_file_paths, out_message, key_string, log_files_output_paths  = anon_wrapper_func(anon_file, anon_df, chosen_cols, out_file_paths, out_file_part, out_message, sheet_name, anon_strat, language, chosen_redact_entities, in_allow_list, file_type, "", log_files_output_paths, in_deny_list, max_fuzzy_spelling_mistakes_num, pii_identification_method, chosen_redact_comprehend_entities, comprehend_query_number, comprehend_client, output_folder=output_folder)
-                    
+                    out_file_paths, out_message, key_string, log_files_output_paths  = anon_wrapper_func(anon_file, anon_df, chosen_cols, out_file_paths, out_file_part, out_message, sheet_name, anon_strat, language, chosen_redact_entities, in_allow_list, file_type, anon_xlsx_export_file_name, log_files_output_paths, in_deny_list, max_fuzzy_spelling_mistakes_num, pii_identification_method, chosen_redact_comprehend_entities, comprehend_query_number, comprehend_client, output_folder=output_folder)
+
             else:
                 sheet_name = ""
                 anon_df = read_file(anon_file)
@@ -380,23 +387,28 @@ def anonymise_data_files(file_paths: List[str],
         # Increase latest file completed count unless we are at the last file
         if latest_file_completed != len(file_paths):
             print("Completed file number:", str(latest_file_completed))
-            latest_file_completed += 1 
+            latest_file_completed += 1
 
         toc = time.perf_counter()
-        out_time = f"in {toc - tic:0.1f} seconds."
-        print(out_time)    
-        
-        if anon_strat == "encrypt":
-            out_message.append(". Your decryption key is " + key_string + ".")
+        out_time_float = toc - tic
+        out_time = f"in {out_time_float:0.1f} seconds."
+        print(out_time)
+
+        actual_time_taken_number += out_time_float
 
         out_message.append("Anonymisation of file '" + out_file_part + "' successfully completed in")
 
         out_message_out = '\n'.join(out_message)
         out_message_out = out_message_out + " " + out_time
 
+        if anon_strat == "encrypt":
+            out_message_out.append(". Your decryption key is " + key_string)        
+
         out_message_out = out_message_out + "\n\nGo to to the Redaction settings tab to see redaction logs. Please give feedback on the results below to help improve this app."
+
+        out_message_out = re.sub(r'^\n+|^\. ', '', out_message_out).strip()
     
-    return out_message_out, out_file_paths, out_file_paths, latest_file_completed, log_files_output_paths, log_files_output_paths
+    return out_message_out, out_file_paths, out_file_paths, latest_file_completed, log_files_output_paths, log_files_output_paths, actual_time_taken_number
 
 def anon_wrapper_func(
     anon_file: str, 
@@ -495,7 +507,6 @@ def anon_wrapper_func(
     anon_df_out = anon_df_out[all_cols_original_order]
     
     # Export file
-
     #  Rename anonymisation strategy for file path naming
     if anon_strat == "replace with 'REDACTED'": anon_strat_txt = "redact_replace"
     elif anon_strat == "replace with <ENTITY_NAME>": anon_strat_txt = "redact_entity_type"
@@ -507,8 +518,14 @@ def anon_wrapper_func(
 
         anon_export_file_name = anon_xlsx_export_file_name
 
+        if not os.path.exists(anon_xlsx_export_file_name):
+            wb = Workbook()
+            ws = wb.active  # Get the default active sheet
+            ws.title = excel_sheet_name
+            wb.save(anon_xlsx_export_file_name)
+
         # Create a Pandas Excel writer using XlsxWriter as the engine.
-        with pd.ExcelWriter(anon_xlsx_export_file_name, engine='openpyxl', mode='a') as writer:
+        with pd.ExcelWriter(anon_xlsx_export_file_name, engine='openpyxl', mode='a', if_sheet_exists='replace') as writer:
             # Write each DataFrame to a different worksheet.
             anon_df_out.to_excel(writer, sheet_name=excel_sheet_name, index=None)
 
@@ -532,7 +549,7 @@ def anon_wrapper_func(
 
     # Print result text to output text box if just anonymising open text
     if anon_file=='open_text':
-        out_message = [anon_df_out['text'][0]]
+        out_message = ["'" + anon_df_out['text'][0] + "'"]
 
     return out_file_paths, out_message, key_string, log_files_output_paths
        
@@ -551,8 +568,16 @@ def anonymise_script(df:pd.DataFrame, anon_strat:str, language:str, chosen_redac
     # DataFrame to dict
     df_dict = df.to_dict(orient="list")
 
-    if in_allow_list:
-        in_allow_list_flat = in_allow_list #[item for sublist in in_allow_list for item in sublist]
+    if isinstance(in_allow_list, list):
+        if in_allow_list:
+            in_allow_list_flat = in_allow_list
+        else:
+            in_allow_list_flat = []
+    elif isinstance(in_allow_list, pd.DataFrame):
+        if not in_allow_list.empty:
+            in_allow_list_flat = list(in_allow_list.iloc[:, 0].unique())
+        else:
+            in_allow_list_flat = []
     else:
         in_allow_list_flat = []
 
@@ -577,11 +602,8 @@ def anonymise_script(df:pd.DataFrame, anon_strat:str, language:str, chosen_redac
 
     #analyzer = nlp_analyser #AnalyzerEngine()
     batch_analyzer = BatchAnalyzerEngine(analyzer_engine=nlp_analyser)
-
     anonymizer = AnonymizerEngine()#conflict_resolution=ConflictResolutionStrategy.MERGE_SIMILAR_OR_CONTAINED)
-
-    batch_anonymizer = BatchAnonymizerEngine(anonymizer_engine = anonymizer)
-    
+    batch_anonymizer = BatchAnonymizerEngine(anonymizer_engine = anonymizer)    
     analyzer_results = []
 
     if pii_identification_method == "Local":
@@ -692,12 +714,6 @@ def anonymise_script(df:pd.DataFrame, anon_strat:str, language:str, chosen_redac
     analyse_time_out = f"Analysing the text took {analyse_toc - analyse_tic:0.1f} seconds."
     print(analyse_time_out)
 
-    # Create faker function (note that it has to receive a value)
-    #fake = Faker("en_UK")
-
-    #def fake_first_name(x):
-    #    return fake.first_name()
-
     # Set up the anonymization configuration WITHOUT DATE_TIME
     simple_replace_config = eval('{"DEFAULT": OperatorConfig("replace", {"new_value": "REDACTED"})}')
     replace_config = eval('{"DEFAULT": OperatorConfig("replace")}')
@@ -714,9 +730,13 @@ def anonymise_script(df:pd.DataFrame, anon_strat:str, language:str, chosen_redac
     if anon_strat == "mask": chosen_mask_config = mask_config
     if anon_strat == "encrypt": 
         chosen_mask_config = people_encrypt_config
-        # Generate a 128-bit AES key. Then encode the key using base64 to get a string representation
-        key = secrets.token_bytes(16)  # 128 bits = 16 bytes 
+        key = secrets.token_bytes(16)  # 128 bits = 16 bytes
         key_string = base64.b64encode(key).decode('utf-8')
+        
+        # Now inject the key into the operator config
+        for entity, operator in chosen_mask_config.items():
+            if operator.operator_name == "encrypt":
+                operator.params = {"key": key_string}
     elif anon_strat == "fake_first_name": chosen_mask_config = fake_first_name_config
 
     # I think in general people will want to keep date / times - removed Mar 2025 as I don't want to assume for people.

tools/file_conversion.py

@@ -21,6 +21,7 @@ from PIL import Image
 from scipy.spatial import cKDTree
 import random
 import string
+import warnings # To warn about potential type changes
 
 IMAGE_NUM_REGEX = re.compile(r'_(\d+)\.png$')
 
@@ -461,7 +462,8 @@ def prepare_image_or_pdf(
     input_folder:str=INPUT_FOLDER,
     prepare_images:bool=True,
     page_sizes:list[dict]=[],
-    textract_output_found:bool = False,    
+    textract_output_found:bool = False,
+    local_ocr_output_found:bool = False,   
     progress: Progress = Progress(track_tqdm=True)
 ) -> tuple[List[str], List[str]]:
     """
@@ -483,7 +485,8 @@ def prepare_image_or_pdf(
         output_folder (optional, str): The output folder for file save
         prepare_images (optional, bool): A boolean indicating whether to create images for each PDF page. Defaults to True.
         page_sizes(optional, List[dict]): A list of dicts containing information about page sizes in various formats.
-        textract_output_found (optional, bool): A boolean indicating whether textract output has already been found . Defaults to False.
+        textract_output_found (optional, bool): A boolean indicating whether Textract analysis output has already been found. Defaults to False.
+        local_ocr_output_found (optional, bool): A boolean indicating whether local OCR analysis output has already been found. Defaults to False.
         progress (optional, Progress): Progress tracker for the operation
         
 
@@ -535,7 +538,7 @@ def prepare_image_or_pdf(
             final_out_message = '\n'.join(out_message)
         else:
             final_out_message = out_message
-        return final_out_message, converted_file_paths, image_file_paths, number_of_pages, number_of_pages, pymupdf_doc, all_annotations_object, review_file_csv, original_cropboxes, page_sizes, textract_output_found, all_img_details, all_line_level_ocr_results_df
+        return final_out_message, converted_file_paths, image_file_paths, number_of_pages, number_of_pages, pymupdf_doc, all_annotations_object, review_file_csv, original_cropboxes, page_sizes, textract_output_found, all_img_details, all_line_level_ocr_results_df, local_ocr_output_found
 
     progress(0.1, desc='Preparing file')
 
@@ -617,11 +620,10 @@ def prepare_image_or_pdf(
 
         elif file_extension in ['.csv']:
             if '_review_file' in file_path_without_ext:
-                #print("file_path:", file_path)
                 review_file_csv = read_file(file_path)
                 all_annotations_object = convert_review_df_to_annotation_json(review_file_csv, image_file_paths, page_sizes)
                 json_from_csv = True
-                print("Converted CSV review file to image annotation object")
+                #print("Converted CSV review file to image annotation object")
             elif '_ocr_output' in file_path_without_ext:
                 all_line_level_ocr_results_df = read_file(file_path)
                 json_from_csv = False
@@ -639,8 +641,8 @@ def prepare_image_or_pdf(
                     # Assuming file_path is a NamedString or similar
                     all_annotations_object = json.loads(file_path)  # Use loads for string content
 
-            # Assume it's a textract json
-            elif (file_extension in ['.json']) and (prepare_for_review != True):
+            # Save Textract file to folder
+            elif (file_extension in ['.json']) and '_textract' in file_path_without_ext: #(prepare_for_review != True):
                 print("Saving Textract output")
                 # Copy it to the output folder so it can be used later.
                 output_textract_json_file_name = file_path_without_ext
@@ -654,6 +656,20 @@ def prepare_image_or_pdf(
                 textract_output_found = True                
                 continue
 
+            elif (file_extension in ['.json']) and '_ocr_results_with_words' in file_path_without_ext: #(prepare_for_review != True):
+                print("Saving local OCR output")
+                # Copy it to the output folder so it can be used later.
+                output_ocr_results_with_words_json_file_name = file_path_without_ext
+                if not file_path.endswith("_ocr_results_with_words.json"): output_ocr_results_with_words_json_file_name = file_path_without_ext + "_ocr_results_with_words.json"
+                else: output_ocr_results_with_words_json_file_name = file_path_without_ext + ".json"
+
+                out_ocr_results_with_words_path = os.path.join(output_folder, output_ocr_results_with_words_json_file_name)
+
+                # Use shutil to copy the file directly
+                shutil.copy2(file_path, out_ocr_results_with_words_path)  # Preserves metadata
+                local_ocr_output_found = True                
+                continue
+
             # NEW IF STATEMENT
             # If you have an annotations object from the above code
             if all_annotations_object:
@@ -773,7 +789,40 @@ def prepare_image_or_pdf(
 
     number_of_pages = len(page_sizes)#len(image_file_paths)
         
-    return combined_out_message, converted_file_paths, image_file_paths, number_of_pages, number_of_pages, pymupdf_doc, all_annotations_object, review_file_csv, original_cropboxes, page_sizes, textract_output_found, all_img_details, all_line_level_ocr_results_df
+    return combined_out_message, converted_file_paths, image_file_paths, number_of_pages, number_of_pages, pymupdf_doc, all_annotations_object, review_file_csv, original_cropboxes, page_sizes, textract_output_found, all_img_details, all_line_level_ocr_results_df, local_ocr_output_found
+
+def load_and_convert_ocr_results_with_words_json(ocr_results_with_words_json_file_path:str, log_files_output_paths:str, page_sizes_df:pd.DataFrame):
+    """
+    Loads Textract JSON from a file, detects if conversion is needed, and converts if necessary.
+    """
+    
+    if not os.path.exists(ocr_results_with_words_json_file_path):
+        print("No existing OCR results file found.")
+        return [], True, log_files_output_paths  # Return empty dict and flag indicating missing file
+    
+    no_ocr_results_with_words_file = False
+    print("Found existing OCR results json results file.")
+
+    # Track log files
+    if ocr_results_with_words_json_file_path not in log_files_output_paths:
+        log_files_output_paths.append(ocr_results_with_words_json_file_path)
+
+    try:
+        with open(ocr_results_with_words_json_file_path, 'r', encoding='utf-8') as json_file:
+            ocr_results_with_words_data = json.load(json_file)
+    except json.JSONDecodeError:
+        print("Error: Failed to parse OCR results JSON file. Returning empty data.")
+        return [], True, log_files_output_paths  # Indicate failure
+
+    # Check if conversion is needed
+    if "page" and "results" in ocr_results_with_words_data[0]:
+        print("JSON already in the correct format for app. No changes needed.")
+        return ocr_results_with_words_data, False, log_files_output_paths  # No conversion required
+
+    else:
+        print("Invalid OCR result JSON format: 'page' or 'results' key missing.")
+        #print("OCR results with words data:", ocr_results_with_words_data)
+        return [], True, log_files_output_paths  # Return empty data if JSON is not recognized
 
 def convert_text_pdf_to_img_pdf(in_file_path:str, out_text_file_path:List[str], image_dpi:float=image_dpi, output_folder:str=OUTPUT_FOLDER, input_folder:str=INPUT_FOLDER):
     file_path_without_ext = get_file_name_without_type(in_file_path)
@@ -850,121 +899,246 @@ def remove_duplicate_images_with_blank_boxes(data: List[dict]) -> List[dict]:
 
     return result
 
-def divide_coordinates_by_page_sizes(review_file_df:pd.DataFrame, page_sizes_df:pd.DataFrame, xmin="xmin", xmax="xmax", ymin="ymin", ymax="ymax"):
-
-    '''Convert data to same coordinate system. If all coordinates all greater than one, this is a absolute image coordinates - change back to relative coordinates.'''
-
-    review_file_df_out = review_file_df
-
-    if xmin in review_file_df.columns and not review_file_df.empty:
-        coord_cols = [xmin, xmax, ymin, ymax]
-        for col in coord_cols:
-            review_file_df.loc[:, col] = pd.to_numeric(review_file_df[col], errors="coerce")
-
-        review_file_df_orig = review_file_df.copy().loc[(review_file_df[xmin] <= 1) & (review_file_df[xmax] <= 1) & (review_file_df[ymin] <= 1) & (review_file_df[ymax] <= 1),:]
-
-        #print("review_file_df_orig:", review_file_df_orig)
-        
-        review_file_df_div = review_file_df.loc[(review_file_df[xmin] > 1) & (review_file_df[xmax] > 1) & (review_file_df[ymin] > 1) & (review_file_df[ymax] > 1),:]
-
-        #print("review_file_df_div:", review_file_df_div)
-
-        review_file_df_div.loc[:, "page"] = pd.to_numeric(review_file_df_div["page"], errors="coerce")
+def divide_coordinates_by_page_sizes(
+    review_file_df: pd.DataFrame,
+    page_sizes_df: pd.DataFrame,
+    xmin="xmin", xmax="xmax", ymin="ymin", ymax="ymax"
+) -> pd.DataFrame:
+    """
+    Optimized function to convert absolute image coordinates (>1) to relative coordinates (<=1).
 
-        if "image_width" not in review_file_df_div.columns and not page_sizes_df.empty:  
+    Identifies rows with absolute coordinates, merges page size information,
+    divides coordinates by dimensions, and combines with already-relative rows.
 
-            page_sizes_df["image_width"] = page_sizes_df["image_width"].replace("<NA>", pd.NA)
-            page_sizes_df["image_height"] = page_sizes_df["image_height"].replace("<NA>", pd.NA)
-            review_file_df_div = review_file_df_div.merge(page_sizes_df[["page", "image_width", "image_height", "mediabox_width", "mediabox_height"]], on="page", how="left")
+    Args:
+        review_file_df: Input DataFrame with potentially mixed coordinate systems.
+        page_sizes_df: DataFrame with page dimensions ('page', 'image_width',
+                       'image_height', 'mediabox_width', 'mediabox_height').
+        xmin, xmax, ymin, ymax: Names of the coordinate columns.
 
-        if "image_width" in review_file_df_div.columns:
-            if review_file_df_div["image_width"].isna().all():  # Check if all are NaN values. If so, assume we only have mediabox coordinates available
-                review_file_df_div["image_width"] = review_file_df_div["image_width"].fillna(review_file_df_div["mediabox_width"]).infer_objects()
-                review_file_df_div["image_height"] = review_file_df_div["image_height"].fillna(review_file_df_div["mediabox_height"]).infer_objects()
+    Returns:
+        DataFrame with coordinates converted to relative system, sorted.
+    """
+    if review_file_df.empty or xmin not in review_file_df.columns:
+        return review_file_df # Return early if empty or key column missing
 
-            convert_type_cols = ["image_width", "image_height", xmin, xmax, ymin, ymax]
-            review_file_df_div[convert_type_cols] = review_file_df_div[convert_type_cols].apply(pd.to_numeric, errors="coerce")  
+    # --- Initial Type Conversion ---
+    coord_cols = [xmin, xmax, ymin, ymax]
+    cols_to_convert = coord_cols + ["page"]
+    temp_df = review_file_df.copy() # Work on a copy initially
 
-            review_file_df_div[xmin] = review_file_df_div[xmin] / review_file_df_div["image_width"]
-            review_file_df_div[xmax] = review_file_df_div[xmax] / review_file_df_div["image_width"]
-            review_file_df_div[ymin] = review_file_df_div[ymin] / review_file_df_div["image_height"]
-            review_file_df_div[ymax] = review_file_df_div[ymax] / review_file_df_div["image_height"]
+    for col in cols_to_convert:
+        if col in temp_df.columns:
+            temp_df[col] = pd.to_numeric(temp_df[col], errors="coerce")
+        else:
+            # If essential 'page' or coord column missing, cannot proceed meaningfully
+            if col == 'page' or col in coord_cols:
+                 print(f"Warning: Required column '{col}' not found in review_file_df. Returning original DataFrame.")
+                 return review_file_df
+
+    # --- Identify Absolute Coordinates ---
+    # Create mask for rows where *all* coordinates are potentially absolute (> 1)
+    # Handle potential NaNs introduced by to_numeric - treat NaN as not absolute.
+    is_absolute_mask = (
+        (temp_df[xmin] > 1) & (temp_df[xmin].notna()) &
+        (temp_df[xmax] > 1) & (temp_df[xmax].notna()) &
+        (temp_df[ymin] > 1) & (temp_df[ymin].notna()) &
+        (temp_df[ymax] > 1) & (temp_df[ymax].notna())
+    )
 
-        # Concatenate the original and modified DataFrames
-        dfs_to_concat = [df for df in [review_file_df_orig, review_file_df_div] if not df.empty]
-        if dfs_to_concat:  # Ensure there's at least one non-empty DataFrame
-            review_file_df_out = pd.concat(dfs_to_concat)
+    # --- Separate DataFrames ---
+    df_rel = temp_df[~is_absolute_mask] # Rows already relative or with NaN/mixed coords
+    df_abs = temp_df[is_absolute_mask].copy() # Absolute rows - COPY here to allow modifications
+
+    # --- Process Absolute Coordinates ---
+    if not df_abs.empty:
+        # Merge page sizes if necessary
+        if "image_width" not in df_abs.columns and not page_sizes_df.empty:
+            ps_df_copy = page_sizes_df.copy() # Work on a copy of page sizes
+
+            # Ensure page is numeric for merge key matching
+            ps_df_copy['page'] = pd.to_numeric(ps_df_copy['page'], errors='coerce')
+
+            # Columns to merge from page_sizes
+            merge_cols = ['page', 'image_width', 'image_height', 'mediabox_width', 'mediabox_height']
+            available_merge_cols = [col for col in merge_cols if col in ps_df_copy.columns]
+
+            # Prepare dimension columns in the copy
+            for col in ['image_width', 'image_height', 'mediabox_width', 'mediabox_height']:
+                 if col in ps_df_copy.columns:
+                     # Replace "<NA>" string if present
+                     if ps_df_copy[col].dtype == 'object':
+                          ps_df_copy[col] = ps_df_copy[col].replace("<NA>", pd.NA)
+                     # Convert to numeric
+                     ps_df_copy[col] = pd.to_numeric(ps_df_copy[col], errors='coerce')
+
+            # Perform the merge
+            if 'page' in available_merge_cols: # Check if page exists for merging
+                df_abs = df_abs.merge(
+                    ps_df_copy[available_merge_cols],
+                    on="page",
+                    how="left"
+                )
+            else:
+                 print("Warning: 'page' column not found in page_sizes_df. Cannot merge dimensions.")
+
+
+        # Fallback to mediabox dimensions if image dimensions are missing
+        if "image_width" in df_abs.columns and "mediabox_width" in df_abs.columns:
+             # Check if image_width mostly missing - use .isna().all() or check percentage
+             if df_abs["image_width"].isna().all():
+                 print("Falling back to mediabox dimensions as image_width is entirely missing.")
+                 df_abs["image_width"] = df_abs["image_width"].fillna(df_abs["mediabox_width"])
+                 df_abs["image_height"] = df_abs["image_height"].fillna(df_abs["mediabox_height"])
+             else:
+                  # Optional: Fill only missing image dims if some exist?
+                  # df_abs["image_width"].fillna(df_abs["mediabox_width"], inplace=True)
+                  # df_abs["image_height"].fillna(df_abs["mediabox_height"], inplace=True)
+                  pass # Current logic only falls back if ALL image_width are NaN
+
+        # Ensure divisor columns are numeric before division
+        divisors_numeric = True
+        for col in ["image_width", "image_height"]:
+            if col in df_abs.columns:
+                 df_abs[col] = pd.to_numeric(df_abs[col], errors='coerce')
+            else:
+                 print(f"Warning: Dimension column '{col}' missing. Cannot perform division.")
+                 divisors_numeric = False
+
+
+        # Perform division if dimensions are available and numeric
+        if divisors_numeric and "image_width" in df_abs.columns and "image_height" in df_abs.columns:
+             # Use np.errstate to suppress warnings about division by zero or NaN if desired
+             with np.errstate(divide='ignore', invalid='ignore'):
+                df_abs[xmin] = df_abs[xmin] / df_abs["image_width"]
+                df_abs[xmax] = df_abs[xmax] / df_abs["image_width"]
+                df_abs[ymin] = df_abs[ymin] / df_abs["image_height"]
+                df_abs[ymax] = df_abs[ymax] / df_abs["image_height"]
+                # Replace potential infinities with NaN (optional, depending on desired outcome)
+                df_abs.replace([np.inf, -np.inf], np.nan, inplace=True)
         else:
-            review_file_df_out = review_file_df  # Return an original DataFrame instead of raising an error
+             print("Skipping coordinate division due to missing or non-numeric dimension columns.")
 
-        # Only sort if the DataFrame is not empty and contains the required columns
-        required_sort_columns = {"page", xmin, ymin}
-        if not review_file_df_out.empty and required_sort_columns.issubset(review_file_df_out.columns):
-            review_file_df_out.sort_values(["page", ymin, xmin], inplace=True)
 
-    review_file_df_out.drop(["image_width", "image_height", "mediabox_width", "mediabox_height"], axis=1, errors="ignore")
+    # --- Combine Relative and Processed Absolute DataFrames ---
+    dfs_to_concat = [df for df in [df_rel, df_abs] if not df.empty]
 
-    return review_file_df_out
+    if dfs_to_concat:
+        final_df = pd.concat(dfs_to_concat, ignore_index=True)
+    else:
+        # If both splits were empty, return an empty DF with original columns
+        print("Warning: Both relative and absolute splits resulted in empty DataFrames.")
+        final_df = pd.DataFrame(columns=review_file_df.columns)
 
-def multiply_coordinates_by_page_sizes(review_file_df: pd.DataFrame, page_sizes_df: pd.DataFrame, xmin="xmin", xmax="xmax", ymin="ymin", ymax="ymax"):
 
+    # --- Final Sort ---
+    required_sort_columns = {"page", xmin, ymin}
+    if not final_df.empty and required_sort_columns.issubset(final_df.columns):
+        # Ensure sort columns are numeric before sorting
+        final_df['page'] = pd.to_numeric(final_df['page'], errors='coerce')
+        final_df[ymin] = pd.to_numeric(final_df[ymin], errors='coerce')
+        final_df[xmin] = pd.to_numeric(final_df[xmin], errors='coerce')
+        # Sort by page, ymin, xmin (note order compared to multiply function)
+        final_df.sort_values(["page", ymin, xmin], inplace=True, na_position='last')
 
-    if xmin in review_file_df.columns and not review_file_df.empty:
 
-        coord_cols = [xmin, xmax, ymin, ymax]
-        for col in coord_cols:
-            review_file_df.loc[:, col] = pd.to_numeric(review_file_df[col], errors="coerce")
+    # --- Clean Up Columns ---
+    # Correctly drop columns and reassign the result
+    cols_to_drop = ["image_width", "image_height", "mediabox_width", "mediabox_height"]
+    final_df = final_df.drop(columns=cols_to_drop, errors="ignore")
 
-        # Separate absolute vs relative coordinates
-        review_file_df_orig = review_file_df.loc[
-            (review_file_df[xmin] > 1) & (review_file_df[xmax] > 1) & 
-            (review_file_df[ymin] > 1) & (review_file_df[ymax] > 1), :].copy()
+    return final_df
 
-        review_file_df = review_file_df.loc[
-            (review_file_df[xmin] <= 1) & (review_file_df[xmax] <= 1) & 
-            (review_file_df[ymin] <= 1) & (review_file_df[ymax] <= 1), :].copy()
+def multiply_coordinates_by_page_sizes(
+    review_file_df: pd.DataFrame,
+    page_sizes_df: pd.DataFrame,
+    xmin="xmin", xmax="xmax", ymin="ymin", ymax="ymax"
+):
+    """
+    Optimized function to convert relative coordinates to absolute based on page sizes.
 
-        if review_file_df.empty:
-            return review_file_df_orig  # If nothing is left, return the original absolute-coordinates DataFrame
+    Separates relative (<=1) and absolute (>1) coordinates, merges page sizes
+    for relative coordinates, calculates absolute pixel values, and recombines.
+    """
+    if review_file_df.empty or xmin not in review_file_df.columns:
+        return review_file_df # Return early if empty or key column missing
+
+    coord_cols = [xmin, xmax, ymin, ymax]
+    # Initial type conversion for coordinates and page
+    for col in coord_cols + ["page"]:
+        if col in review_file_df.columns:
+             # Use astype for potentially faster conversion if confident,
+             # but to_numeric is safer for mixed types/errors
+            review_file_df[col] = pd.to_numeric(review_file_df[col], errors="coerce")
+
+    # --- Identify relative coordinates ---
+    # Create mask for rows where *all* coordinates are potentially relative (<= 1)
+    # Handle potential NaNs introduced by to_numeric - treat NaN as not relative here.
+    is_relative_mask = (
+        (review_file_df[xmin].le(1) & review_file_df[xmin].notna()) &
+        (review_file_df[xmax].le(1) & review_file_df[xmax].notna()) &
+        (review_file_df[ymin].le(1) & review_file_df[ymin].notna()) &
+        (review_file_df[ymax].le(1) & review_file_df[ymax].notna())
+    )
 
-        review_file_df.loc[:, "page"] = pd.to_numeric(review_file_df["page"], errors="coerce")
+    # Separate DataFrames (minimal copies)
+    df_abs = review_file_df[~is_relative_mask].copy() # Keep absolute rows separately
+    df_rel = review_file_df[is_relative_mask].copy()  # Work only with relative rows
+
+    if df_rel.empty:
+        # If no relative coordinates, just sort and return absolute ones (if any)
+        if not df_abs.empty and {"page", xmin, ymin}.issubset(df_abs.columns):
+             df_abs.sort_values(["page", xmin, ymin], inplace=True, na_position='last')
+        return df_abs
+
+    # --- Process relative coordinates ---
+    if "image_width" not in df_rel.columns and not page_sizes_df.empty:
+        # Prepare page_sizes_df for merge
+        page_sizes_df = page_sizes_df.copy() # Avoid modifying original page_sizes_df
+        page_sizes_df['page'] = pd.to_numeric(page_sizes_df['page'], errors='coerce')
+        # Ensure proper NA handling for image dimensions
+        page_sizes_df[['image_width', 'image_height']] = page_sizes_df[['image_width','image_height']].replace("<NA>", pd.NA)
+        page_sizes_df['image_width'] = pd.to_numeric(page_sizes_df['image_width'], errors='coerce')
+        page_sizes_df['image_height'] = pd.to_numeric(page_sizes_df['image_height'], errors='coerce')
+
+        # Merge page sizes
+        df_rel = df_rel.merge(
+            page_sizes_df[['page', 'image_width', 'image_height']],
+            on="page",
+            how="left"
+        )
 
-        if "image_width" not in review_file_df.columns and not page_sizes_df.empty:
-            page_sizes_df[['image_width', 'image_height']] = page_sizes_df[['image_width','image_height']].replace("<NA>", pd.NA)  # Ensure proper NA handling
-            review_file_df = review_file_df.merge(page_sizes_df, on="page", how="left")
+    # Multiply coordinates where image dimensions are available
+    if "image_width" in df_rel.columns:
+        # Create mask for rows in df_rel that have valid image dimensions
+        has_size_mask = df_rel["image_width"].notna() & df_rel["image_height"].notna()
 
-        if "image_width" in review_file_df.columns:
-            # Split into rows with/without image size info
-            review_file_df_not_na = review_file_df.loc[review_file_df["image_width"].notna()].copy()
-            review_file_df_na = review_file_df.loc[review_file_df["image_width"].isna()].copy()
+        # Apply multiplication using .loc and the mask (vectorized and efficient)
+        # Ensure columns are numeric before multiplication (might be redundant if types are good)
+        # df_rel.loc[has_size_mask, coord_cols + ['image_width', 'image_height']] = df_rel.loc[has_size_mask, coord_cols + ['image_width', 'image_height']].apply(pd.to_numeric, errors='coerce')
 
-            if not review_file_df_not_na.empty:
-                convert_type_cols = ["image_width", "image_height", xmin, xmax, ymin, ymax]
-                review_file_df_not_na[convert_type_cols] = review_file_df_not_na[convert_type_cols].apply(pd.to_numeric, errors="coerce")
+        df_rel.loc[has_size_mask, xmin] *= df_rel.loc[has_size_mask, "image_width"]
+        df_rel.loc[has_size_mask, xmax] *= df_rel.loc[has_size_mask, "image_width"]
+        df_rel.loc[has_size_mask, ymin] *= df_rel.loc[has_size_mask, "image_height"]
+        df_rel.loc[has_size_mask, ymax] *= df_rel.loc[has_size_mask, "image_height"]
 
-                # Multiply coordinates by image sizes
-                review_file_df_not_na[xmin] *= review_file_df_not_na["image_width"]
-                review_file_df_not_na[xmax] *= review_file_df_not_na["image_width"]
-                review_file_df_not_na[ymin] *= review_file_df_not_na["image_height"]
-                review_file_df_not_na[ymax] *= review_file_df_not_na["image_height"]
 
-            # Concatenate the modified and unmodified data
-            review_file_df = pd.concat([df for df in [review_file_df_not_na, review_file_df_na] if not df.empty])
+    # --- Combine absolute and processed relative DataFrames ---
+    # Use list comprehension to handle potentially empty DataFrames
+    dfs_to_concat = [df for df in [df_abs, df_rel] if not df.empty]
 
-        # Merge with the original absolute-coordinates DataFrame
-        dfs_to_concat = [df for df in [review_file_df_orig, review_file_df] if not df.empty]
-        if dfs_to_concat:  # Ensure there's at least one non-empty DataFrame
-            review_file_df = pd.concat(dfs_to_concat)
-        else:
-            review_file_df = pd.DataFrame()  # Return an empty DataFrame instead of raising an error
+    if not dfs_to_concat:
+        return pd.DataFrame() # Return empty if both are empty
 
-        # Only sort if the DataFrame is not empty and contains the required columns
-        required_sort_columns = {"page", "xmin", "ymin"}
-        if not review_file_df.empty and required_sort_columns.issubset(review_file_df.columns):
-            review_file_df.sort_values(["page", "xmin", "ymin"], inplace=True)
+    final_df = pd.concat(dfs_to_concat, ignore_index=True) # ignore_index is good practice after filtering/concat
 
-    return review_file_df
+    # --- Final Sort ---
+    required_sort_columns = {"page", xmin, ymin}
+    if not final_df.empty and required_sort_columns.issubset(final_df.columns):
+        # Handle potential NaNs in sort columns gracefully
+        final_df.sort_values(["page", xmin, ymin], inplace=True, na_position='last')
 
+    return final_df
 
 def do_proximity_match_by_page_for_text(df1:pd.DataFrame, df2:pd.DataFrame):
     '''
@@ -1018,7 +1192,6 @@ def do_proximity_match_by_page_for_text(df1:pd.DataFrame, df2:pd.DataFrame):
 
     return merged_df
 
-
 def do_proximity_match_all_pages_for_text(df1:pd.DataFrame, df2:pd.DataFrame, threshold:float=0.03):
     '''
     Match text from one dataframe to another based on proximity matching of coordinates across all pages.
@@ -1142,12 +1315,12 @@ def convert_annotation_data_to_dataframe(all_annotations: List[Dict[str, Any]]):
     # prevents this from being necessary.
 
     # 7. Ensure essential columns exist and set column order
-    essential_box_cols = ["xmin", "xmax", "ymin", "ymax", "text", "id"]
+    essential_box_cols = ["xmin", "xmax", "ymin", "ymax", "text", "id", "label"]
     for col in essential_box_cols:
         if col not in final_df.columns:
             final_df[col] = pd.NA # Add column with NA if it wasn't present in any box
 
-    base_cols = ["image", "page"]
+    base_cols = ["image"]
     extra_box_cols = [col for col in final_df.columns if col not in base_cols and col not in essential_box_cols]
     final_col_order = base_cols + essential_box_cols + sorted(extra_box_cols)
 
@@ -1156,6 +1329,8 @@ def convert_annotation_data_to_dataframe(all_annotations: List[Dict[str, Any]]):
     # but it's good practice if columns could be missing for other reasons.
     final_df = final_df.reindex(columns=final_col_order, fill_value=pd.NA)
 
+    final_df = final_df.dropna(subset=["xmin", "xmax", "ymin", "ymax", "text", "id", "label"])
+
     return final_df
 
 def create_annotation_dicts_from_annotation_df(
@@ -1185,7 +1360,8 @@ def create_annotation_dicts_from_annotation_df(
     available_cols = [col for col in box_cols if col in all_image_annotations_df.columns]
 
     if 'text' in all_image_annotations_df.columns:
-        all_image_annotations_df.loc[all_image_annotations_df['text'].isnull(), 'text'] = ''
+        all_image_annotations_df['text'] = all_image_annotations_df['text'].fillna('')
+        #all_image_annotations_df.loc[all_image_annotations_df['text'].isnull(), 'text'] = ''
 
     if not available_cols:
         print(f"Warning: None of the expected box columns ({box_cols}) found in DataFrame.")
@@ -1226,85 +1402,84 @@ def create_annotation_dicts_from_annotation_df(
 
     return result
 
-def convert_annotation_json_to_review_df(all_annotations: List[dict],
-                                         redaction_decision_output: pd.DataFrame = pd.DataFrame(),
-                                         page_sizes: List[dict] = [],
-                                         do_proximity_match: bool = True) -> pd.DataFrame:
+def convert_annotation_json_to_review_df(
+    all_annotations: List[dict],
+    redaction_decision_output: pd.DataFrame = pd.DataFrame(),
+    page_sizes: List[dict] = [],
+    do_proximity_match: bool = True
+) -> pd.DataFrame:
     '''
     Convert the annotation json data to a dataframe format.
     Add on any text from the initial review_file dataframe by joining based on 'id' if available
     in both sources, otherwise falling back to joining on pages/co-ordinates (if option selected).
+
+    Refactored for improved efficiency, prioritizing ID-based join and conditionally applying
+    coordinate division and proximity matching.
     '''
 
     # 1. Convert annotations to DataFrame
-    # Ensure convert_annotation_data_to_dataframe populates the 'id' column
-    # if 'id' exists in the dictionaries within all_annotations.
-
     review_file_df = convert_annotation_data_to_dataframe(all_annotations)
 
-    # Only keep rows in review_df where there are coordinates
-    review_file_df.dropna(subset='xmin', axis=0, inplace=True)
+    # Only keep rows in review_df where there are coordinates (assuming xmin is representative)
+    # Use .notna() for robustness with potential None or NaN values
+    review_file_df.dropna(subset=['xmin', 'ymin', 'xmax', 'ymax'], how='any', inplace=True)
 
     # Exit early if the initial conversion results in an empty DataFrame
     if review_file_df.empty:
         # Define standard columns for an empty return DataFrame
-        check_columns = ["image", "page", "label", "color", "xmin", "ymin", "xmax", "ymax", "text", "id"]
-        # Ensure 'id' is included if it might have been expected
-        return pd.DataFrame(columns=[col for col in check_columns if col != 'id' or 'id' in review_file_df.columns])
-
-    # 2. Handle page sizes if provided
-    if not page_sizes:
-        page_sizes_df = pd.DataFrame(page_sizes) # Ensure it's a DataFrame
-        # Safely convert page column to numeric
-        page_sizes_df["page"] = pd.to_numeric(page_sizes_df["page"], errors="coerce")
-        page_sizes_df.dropna(subset=["page"], inplace=True) # Drop rows where conversion failed
-        page_sizes_df["page"] = page_sizes_df["page"].astype(int) # Convert to int after handling errors/NaNs
+        # Ensure 'id' is included if it was potentially expected based on input structure
+        # We don't know the columns from convert_annotation_data_to_dataframe without seeing it,
+        # but let's assume a standard set and add 'id' if it appeared.
+        standard_cols = ["image", "page", "label", "color", "xmin", "ymin", "xmax", "ymax", "text"]
+        if 'id' in review_file_df.columns:
+             standard_cols.append('id')
+        return pd.DataFrame(columns=standard_cols)
+
+    # Ensure 'id' column exists for logic flow, even if empty
+    if 'id' not in review_file_df.columns:
+        review_file_df['id'] = ''
+    # Do the same for redaction_decision_output if it's not empty
+    if not redaction_decision_output.empty and 'id' not in redaction_decision_output.columns:
+         redaction_decision_output['id'] = ''
 
 
-        # Apply coordinate division if page_sizes_df is not empty after processing
+    # 2. Process page sizes if provided - needed potentially for coordinate division later
+    # Process this once upfront if the data is available
+    page_sizes_df = pd.DataFrame() # Initialize as empty
+    if page_sizes:
+        page_sizes_df = pd.DataFrame(page_sizes)
         if not page_sizes_df.empty:
-            # Ensure 'page' column in review_file_df is numeric for merging
-            if 'page' in review_file_df.columns:
-                 review_file_df['page'] = pd.to_numeric(review_file_df['page'], errors='coerce')
-                 # Drop rows with invalid pages before division
-                 review_file_df.dropna(subset=['page'], inplace=True)
-                 review_file_df['page'] = review_file_df['page'].astype(int)
-                 review_file_df = divide_coordinates_by_page_sizes(review_file_df, page_sizes_df)
-
-                 print("review_file_df after coord divide:", review_file_df)
-
-            # Also apply to redaction_decision_output if it's not empty and has page numbers
-            if not redaction_decision_output.empty and 'page' in redaction_decision_output.columns:
-                redaction_decision_output['page'] = pd.to_numeric(redaction_decision_output['page'], errors='coerce')
-                # Drop rows with invalid pages before division
-                redaction_decision_output.dropna(subset=['page'], inplace=True)
-                redaction_decision_output['page'] = redaction_decision_output['page'].astype(int)
-                redaction_decision_output = divide_coordinates_by_page_sizes(redaction_decision_output, page_sizes_df)
-
-                print("redaction_decision_output after coord divide:", redaction_decision_output)
-        else:
-             print("Warning: Page sizes DataFrame became empty after processing, skipping coordinate division.")
+            # Safely convert page column to numeric and then int
+            page_sizes_df["page"] = pd.to_numeric(page_sizes_df["page"], errors="coerce")
+            page_sizes_df.dropna(subset=["page"], inplace=True)
+            if not page_sizes_df.empty: # Check again after dropping NaNs
+                page_sizes_df["page"] = page_sizes_df["page"].astype(int)
+            else:
+                 print("Warning: Page sizes DataFrame became empty after processing, coordinate division will be skipped.")
 
 
     # 3. Join additional data from redaction_decision_output if provided
+    text_added_successfully = False # Flag to track if text was added by any method
+
     if not redaction_decision_output.empty:
-        # --- NEW LOGIC: Prioritize joining by 'id' ---
-        id_col_exists_in_review = 'id' in review_file_df.columns
-        id_col_exists_in_redaction = 'id' in redaction_decision_output.columns
-        joined_by_id = False # Flag to track if ID join was successful
+        # --- Attempt to join data based on 'id' column first ---
+
+        # Check if 'id' columns are present and have non-null values in *both* dataframes
+        id_col_exists_in_review = 'id' in review_file_df.columns and not review_file_df['id'].isnull().all() and not (review_file_df['id'] == '').all()
+        id_col_exists_in_redaction = 'id' in redaction_decision_output.columns and not redaction_decision_output['id'].isnull().all() and not (redaction_decision_output['id'] == '').all()
+
 
         if id_col_exists_in_review and id_col_exists_in_redaction:
             #print("Attempting to join data based on 'id' column.")
             try:
-                # Ensure 'id' columns are of compatible types (e.g., string) to avoid merge errors
+                # Ensure 'id' columns are of string type for robust merging
                 review_file_df['id'] = review_file_df['id'].astype(str)
-                # Make a copy to avoid SettingWithCopyWarning if redaction_decision_output is used elsewhere
+                # Make a copy if needed, but try to avoid if redaction_decision_output isn't modified later
+                # Let's use a copy for safety as in the original code
                 redaction_copy = redaction_decision_output.copy()
                 redaction_copy['id'] = redaction_copy['id'].astype(str)
 
-                # Select columns to merge from redaction output.
-                # Primarily interested in 'text', but keep 'id' for the merge key.
-                # Add other columns from redaction_copy if needed.
+                # Select columns to merge from redaction output. Prioritize 'text'.
                 cols_to_merge = ['id']
                 if 'text' in redaction_copy.columns:
                     cols_to_merge.append('text')
@@ -1312,82 +1487,130 @@ def convert_annotation_json_to_review_df(all_annotations: List[dict],
                     print("Warning: 'text' column not found in redaction_decision_output. Cannot merge text using 'id'.")
 
                 # Perform a left merge to keep all annotations and add matching text
-                # Suffixes prevent collision if 'text' already exists and we want to compare/choose
-                original_cols = review_file_df.columns.tolist()
+                # Use a suffix for the text column from the right DataFrame
+                original_text_col_exists = 'text' in review_file_df.columns
+                merge_suffix = '_redaction' if original_text_col_exists else ''
+
                 merged_df = pd.merge(
                     review_file_df,
                     redaction_copy[cols_to_merge],
                     on='id',
                     how='left',
-                    suffixes=('', '_redaction') # Suffix applied to columns from right df if names clash
+                    suffixes=('', merge_suffix)
                 )
 
-                # Update the original 'text' column. Prioritize text from redaction output.
-                # If redaction output had 'text', a 'text_redaction' column now exists.
-                if 'text_redaction' in merged_df.columns:
-                     if 'text' not in merged_df.columns: # If review_file_df didn't have text initially
-                         merged_df['text'] = merged_df['text_redaction']
-                     else:
-                         # Use text from redaction where available, otherwise keep original text
-                         merged_df['text'] = merged_df['text_redaction'].combine_first(merged_df['text'])
+                # Update the 'text' column if a new one was brought in
+                if 'text' + merge_suffix in merged_df.columns:
+                    redaction_text_col = 'text' + merge_suffix
+                    if original_text_col_exists:
+                         # Combine: Use text from redaction where available, otherwise keep original
+                         merged_df['text'] = merged_df[redaction_text_col].combine_first(merged_df['text'])
+                         # Drop the temporary column
+                         merged_df = merged_df.drop(columns=[redaction_text_col])
+                    else:
+                         # Redaction output had text, but review_file_df didn't. Rename the new column.
+                         merged_df = merged_df.rename(columns={redaction_text_col: 'text'})
 
-                     # Remove the temporary column
-                     merged_df = merged_df.drop(columns=['text_redaction'])
+                    text_added_successfully = True # Indicate text was potentially added
 
-                # Ensure final columns match original expectation + potentially new 'text'
-                final_cols = original_cols
-                if 'text' not in final_cols and 'text' in merged_df.columns:
-                    final_cols.append('text') # Make sure text column is kept if newly added
-                 # Reorder/select columns if necessary, ensuring 'id' is kept
-                review_file_df = merged_df[[col for col in final_cols if col in merged_df.columns] + (['id'] if 'id' not in final_cols else [])]
+                review_file_df = merged_df # Update the main DataFrame
 
-
-                #print("Successfully joined data using 'id'.")
-                joined_by_id = True
+                #print("Successfully attempted to join data using 'id'.") # Note: Text might not have been in redaction data
 
             except Exception as e:
-                print(f"Error during 'id'-based merge: {e}. Falling back to proximity match if enabled.")
-                # Fall through to proximity match below if an error occurred
-
-        # --- Fallback to proximity match ---
-        if not joined_by_id and do_proximity_match:
-            if not id_col_exists_in_review or not id_col_exists_in_redaction:
-                 print("Could not join by 'id' (column missing in one or both sources).")
-            print("Performing proximity match to add text data.")
-            # Match text to review file using proximity
-
-            review_file_df = do_proximity_match_all_pages_for_text(df1=review_file_df.copy(), df2=redaction_decision_output.copy())
-        elif not joined_by_id and not do_proximity_match:
-             print("Skipping joining text data (ID join not possible, proximity match disabled).")
-        # --- End of join logic ---
-
-    # 4. Ensure required columns exist, filling with blank if they don't
-    # Define base required columns, 'id' might or might not be present initially
-    required_columns = ["image", "page", "label", "color", "xmin", "ymin", "xmax", "ymax", "text"]
-    # Add 'id' to required list if it exists in the dataframe at this point
+                print(f"Error during 'id'-based merge: {e}. Checking for proximity match fallback.")
+                # Fall through to proximity match logic below
+
+        # --- Fallback to proximity match if ID join wasn't possible/successful and enabled ---
+        # Note: If id_col_exists_in_review or id_col_exists_in_redaction was False,
+        # the block above was skipped, and we naturally fall here.
+        # If an error occurred in the try block, joined_by_id would implicitly be False
+        # because text_added_successfully wasn't set to True.
+
+        # Only attempt proximity match if text wasn't added by ID join and proximity is requested
+        if not text_added_successfully and do_proximity_match:
+             print("Attempting proximity match to add text data.")
+
+             # Ensure 'page' columns are numeric before coordinate division and proximity match
+             # (Assuming divide_coordinates_by_page_sizes and do_proximity_match_all_pages_for_text need this)
+             if 'page' in review_file_df.columns:
+                 review_file_df['page'] = pd.to_numeric(review_file_df['page'], errors='coerce').fillna(-1).astype(int) # Use -1 for NaN pages
+                 review_file_df = review_file_df[review_file_df['page'] != -1] # Drop rows where page conversion failed
+             if not redaction_decision_output.empty and 'page' in redaction_decision_output.columns:
+                  redaction_decision_output['page'] = pd.to_numeric(redaction_decision_output['page'], errors='coerce').fillna(-1).astype(int)
+                  redaction_decision_output = redaction_decision_output[redaction_decision_output['page'] != -1]
+
+             # Perform coordinate division IF page_sizes were processed and DataFrame is not empty
+             if not page_sizes_df.empty:
+                  # Apply coordinate division *before* proximity match
+                  review_file_df = divide_coordinates_by_page_sizes(review_file_df, page_sizes_df)
+                  if not redaction_decision_output.empty:
+                       redaction_decision_output = divide_coordinates_by_page_sizes(redaction_decision_output, page_sizes_df)
+
+             # Now perform the proximity match
+             # Note: Potential DataFrame copies happen inside do_proximity_match based on its implementation
+             if not redaction_decision_output.empty:
+                try:
+                    review_file_df = do_proximity_match_all_pages_for_text(
+                        df1=review_file_df, # Pass directly, avoid caller copy if possible by modifying function signature
+                        df2=redaction_decision_output # Pass directly
+                    )
+                    # Assuming do_proximity_match_all_pages_for_text adds/updates the 'text' column
+                    if 'text' in review_file_df.columns:
+                         text_added_successfully = True
+                    print("Proximity match completed.")
+                except Exception as e:
+                    print(f"Error during proximity match: {e}. Text data may not be added.")
+
+        elif not text_added_successfully and not do_proximity_match:
+             print("Skipping joining text data (ID join not possible/failed, proximity match disabled).")
+
+    # 4. Ensure required columns exist and are ordered
+    # Define base required columns. 'id' and 'text' are conditionally added.
+    required_columns_base = ["image", "page", "label", "color", "xmin", "ymin", "xmax", "ymax"]
+    final_columns = required_columns_base[:] # Start with base columns
+
+    # Add 'id' and 'text' if they exist in the DataFrame at this point
     if 'id' in review_file_df.columns:
-        required_columns.append('id')
+        final_columns.append('id')
+    if 'text' in review_file_df.columns:
+        final_columns.append('text') # Add text column if it was created/merged
 
-    for col in required_columns:
+    # Add any missing required columns with a default value (e.g., blank string)
+    for col in final_columns:
         if col not in review_file_df.columns:
-            # Decide default value based on column type (e.g., '' for text, np.nan for numeric?)
-            # Using '' for simplicity here.
-            review_file_df[col] = ''
+            # Use appropriate default based on expected type, '' for text/id, np.nan for coords?
+            # Sticking to '' as in original for simplicity, but consider data types.
+            review_file_df[col] = '' # Or np.nan for numerical, but coords already checked by dropna
 
     # Select and order the final set of columns
-    review_file_df = review_file_df[required_columns]
+    # Ensure all selected columns actually exist after adding defaults
+    review_file_df = review_file_df[[col for col in final_columns if col in review_file_df.columns]]
+
 
     # 5. Final processing and sorting
-    # If colours are saved as list, convert to tuple
+    # Convert colours from list to tuple if necessary - apply is okay here unless lists are vast
     if 'color' in review_file_df.columns:
-        review_file_df["color"] = review_file_df["color"].apply(lambda x: tuple(x) if isinstance(x, list) else x)
+         # Check if the column actually contains lists before applying lambda
+         if review_file_df['color'].apply(lambda x: isinstance(x, list)).any():
+            review_file_df["color"] = review_file_df["color"].apply(lambda x: tuple(x) if isinstance(x, list) else x)
 
     # Sort the results
-    sort_columns = ['page', 'ymin', 'xmin', 'label']
     # Ensure sort columns exist before sorting
+    sort_columns = ['page', 'ymin', 'xmin', 'label']
     valid_sort_columns = [col for col in sort_columns if col in review_file_df.columns]
-    if valid_sort_columns:
-        review_file_df = review_file_df.sort_values(valid_sort_columns)
+    if valid_sort_columns and not review_file_df.empty: # Only sort non-empty df
+         # Convert potential numeric sort columns to appropriate types if necessary
+         # (e.g., 'page', 'ymin', 'xmin') to ensure correct sorting.
+         # dropna(subset=[...], inplace=True) earlier should handle NaNs in coords.
+         # page conversion already done before proximity match.
+         try:
+             review_file_df = review_file_df.sort_values(valid_sort_columns)
+         except TypeError as e:
+              print(f"Warning: Could not sort DataFrame due to type error in sort columns: {e}")
+              # Proceed without sorting
+
+    review_file_df = review_file_df.dropna(subset=["xmin", "xmax", "ymin", "ymax", "text", "id", "label"])
 
     return review_file_df
 
@@ -1472,20 +1695,18 @@ def fill_missing_box_ids(data_input: dict) -> dict:
 
 def fill_missing_ids(df: pd.DataFrame, column_name: str = 'id', length: int = 12) -> pd.DataFrame:
     """
-    Generates unique alphanumeric IDs for rows in a DataFrame column
-    where the value is missing (NaN, None) or an empty string.
+    Optimized: Generates unique alphanumeric IDs for rows in a DataFrame column
+    where the value is missing (NaN, None) or an empty/whitespace string.
 
     Args:
         df (pd.DataFrame): The input Pandas DataFrame.
         column_name (str): The name of the column to check and fill (defaults to 'id').
                            This column will be added if it doesn't exist.
         length (int): The desired length of the generated IDs (defaults to 12).
-                      Cannot exceed the limits that guarantee uniqueness based
-                      on the number of IDs needed and character set size.
 
     Returns:
         pd.DataFrame: The DataFrame with missing/empty IDs filled in the specified column.
-                      Note: The function modifies the DataFrame in place.
+                      Note: The function modifies the DataFrame directly (in-place).
     """
 
     # --- Input Validation ---
@@ -1497,43 +1718,59 @@ def fill_missing_ids(df: pd.DataFrame, column_name: str = 'id', length: int = 12
         raise ValueError("'length' must be a positive integer.")
 
     # --- Ensure Column Exists ---
+    original_dtype = None
     if column_name not in df.columns:
         print(f"Column '{column_name}' not found. Adding it to the DataFrame.")
-        df[column_name] = np.nan # Initialize with NaN
+        # Initialize with None (which Pandas often treats as NaN but allows object dtype)
+        df[column_name] = None
+        # Set original_dtype to object so it likely becomes string later
+        original_dtype = object
+    else:
+        original_dtype = df[column_name].dtype
 
     # --- Identify Rows Needing IDs ---
-    # Check for NaN, None, or empty strings ('')
-    # Convert to string temporarily for robust empty string check, handle potential errors
-    try:
-        df[column_name] = df[column_name].astype(str) #handles NaN/None conversion, .str.strip() removes whitespace
-        is_missing_or_empty = (
-            df[column_name].isna()
-            #| (df[column_name].astype(str).str.strip() == '')
-            #| (df[column_name] == "nan")
-            | (df[column_name].astype(str).str.len() != length)
-        )
-    except Exception as e:
-         # Fallback if conversion to string fails (e.g., column contains complex objects)
-         print(f"Warning: Could not perform reliable empty string check on column '{column_name}' due to data type issues. Checking for NaN/None only. Error: {e}")
-         is_missing_or_empty = df[column_name].isna()
+    # 1. Check for actual null values (NaN, None, NaT)
+    is_null = df[column_name].isna()
+
+    # 2. Check for empty or whitespace-only strings AFTER converting potential values to string
+    #    Only apply string checks on rows that are *not* null to avoid errors/warnings
+    #    Fill NaN temporarily for string operations, then check length or equality
+    is_empty_str = pd.Series(False, index=df.index) # Default to False
+    if not is_null.all(): # Only check strings if there are non-null values
+         temp_str_col = df.loc[~is_null, column_name].astype(str).str.strip()
+         is_empty_str.loc[~is_null] = (temp_str_col == '')
+
+    # Combine the conditions
+    is_missing_or_empty = is_null | is_empty_str
 
     rows_to_fill_index = df.index[is_missing_or_empty]
     num_needed = len(rows_to_fill_index)
 
     if num_needed == 0:
-        #print(f"No missing or empty values found in column '{column_name}'.")
+        # Ensure final column type is consistent if nothing was done
+        if pd.api.types.is_object_dtype(original_dtype) or pd.api.types.is_string_dtype(original_dtype):
+             pass # Likely already object or string
+        else:
+             # If original was numeric/etc., but might contain strings now? Unlikely here.
+             pass # Or convert to object: df[column_name] = df[column_name].astype(object)
+        # print(f"No missing or empty values found requiring IDs in column '{column_name}'.")
         return df
 
     print(f"Found {num_needed} rows requiring a unique ID in column '{column_name}'.")
 
     # --- Get Existing IDs to Ensure Uniqueness ---
-    try:
-        # Get all non-missing, non-empty string values from the column
-        existing_ids = set(df.loc[~is_missing_or_empty, column_name].astype(str))
-    except Exception as e:
-        print(f"Warning: Could not reliably get all existing string IDs from column '{column_name}' due to data type issues. Uniqueness check might be less strict. Error: {e}")
-        # Fallback: Get only non-NaN IDs, potential type issues ignored
-        existing_ids = set(df.loc[df[column_name].notna(), column_name])
+    # Consider only rows that are *not* missing/empty
+    valid_rows = df.loc[~is_missing_or_empty, column_name]
+    # Drop any remaining nulls (shouldn't be any based on mask, but belts and braces)
+    valid_rows = valid_rows.dropna()
+    # Convert to string *only* if not already string/object, then filter out empty strings again
+    if not pd.api.types.is_object_dtype(valid_rows.dtype) and not pd.api.types.is_string_dtype(valid_rows.dtype):
+         existing_ids = set(valid_rows.astype(str).str.strip())
+    else: # Already string or object, just strip and convert to set
+         existing_ids = set(valid_rows.astype(str).str.strip()) # astype(str) handles mixed types in object column
+
+    # Remove empty string from existing IDs if it's there after stripping
+    existing_ids.discard('')
 
 
     # --- Generate Unique IDs ---
@@ -1543,93 +1780,232 @@ def fill_missing_ids(df: pd.DataFrame, column_name: str = 'id', length: int = 12
 
     max_possible_ids = len(character_set) ** length
     if num_needed > max_possible_ids:
-         raise ValueError(f"Cannot generate {num_needed} unique IDs with length {length}. Maximum possible is {max_possible_ids}.")
-    # Add a check for practical limits if needed, e.g., if num_needed is very close to max_possible_ids, generation could be slow.
+        raise ValueError(f"Cannot generate {num_needed} unique IDs with length {length}. Maximum possible is {max_possible_ids}.")
+
+    # Pre-calculate safety break limit
+    max_attempts_per_id = max(1000, num_needed * 10) # Adjust multiplier as needed
 
     #print(f"Generating {num_needed} unique IDs of length {length}...")
     for i in range(num_needed):
         attempts = 0
         while True:
             candidate_id = ''.join(random.choices(character_set, k=length))
-            # Check against *all* existing IDs and *newly* generated ones
+            # Check against *all* known existing IDs and *newly* generated ones
             if candidate_id not in existing_ids and candidate_id not in generated_ids_set:
                 generated_ids_set.add(candidate_id)
                 new_ids_list.append(candidate_id)
                 break # Found a unique ID
             attempts += 1
-            if attempts > num_needed * 100 and attempts > 1000 : # Safety break for unlikely infinite loop
-                 raise RuntimeError(f"Failed to generate a unique ID after {attempts} attempts. Check length and character set or existing IDs.")
+            if attempts > max_attempts_per_id : # Safety break
+                raise RuntimeError(f"Failed to generate a unique ID after {attempts} attempts. Check length, character set, or density of existing IDs.")
 
-        # Optional progress update for large numbers
-        if (i + 1) % 1000 == 0:
-            print(f"Generated {i+1}/{num_needed} IDs...")
+        # Optional progress update
+        # if (i + 1) % 1000 == 0:
+        #    print(f"Generated {i+1}/{num_needed} IDs...")
 
 
     # --- Assign New IDs ---
     # Use the previously identified index to assign the new IDs correctly
+    # Assigning string IDs might change the column's dtype to 'object'
+    if not pd.api.types.is_object_dtype(original_dtype) and not pd.api.types.is_string_dtype(original_dtype):
+         warnings.warn(f"Column '{column_name}' dtype might change from '{original_dtype}' to 'object' due to string ID assignment.", UserWarning)
+
     df.loc[rows_to_fill_index, column_name] = new_ids_list
-    #print(f"Successfully filled {len(new_ids_list)} missing values in column '{column_name}'.")
+    print(f"Successfully assigned {len(new_ids_list)} new unique IDs to column '{column_name}'.")
+
+    # Optional: Convert the entire column to string type at the end for consistency
+    # df[column_name] = df[column_name].astype(str)
 
-    # The DataFrame 'df' has been modified in place
     return df
 
-def convert_review_df_to_annotation_json(review_file_df:pd.DataFrame,
-                                         image_paths:List[Image.Image],
-                                         page_sizes:List[dict]=[]) -> List[dict]:
-    '''
-    Convert a review csv to a json file for use by the Gradio Annotation object.
-    '''
-    # Make sure all relevant cols are float
-    float_cols = ["page", "xmin", "xmax", "ymin", "ymax"]
-    for col in float_cols:
-        review_file_df.loc[:, col] = pd.to_numeric(review_file_df.loc[:, col], errors='coerce')
-    
-    # Convert relative co-ordinates into image coordinates for the image annotation output object
-    if page_sizes:        
-        page_sizes_df = pd.DataFrame(page_sizes)
-        page_sizes_df[["page"]] = page_sizes_df[["page"]].apply(pd.to_numeric, errors="coerce")
+def convert_review_df_to_annotation_json(
+    review_file_df: pd.DataFrame,
+    image_paths: List[str], # List of image file paths
+    page_sizes: List[Dict], # List of dicts like [{'page': 1, 'image_path': '...', 'image_width': W, 'image_height': H}, ...]
+    xmin="xmin", xmax="xmax", ymin="ymin", ymax="ymax" # Coordinate column names
+) -> List[Dict]:
+    """
+    Optimized function to convert review DataFrame to Gradio Annotation JSON format.
 
-        review_file_df = multiply_coordinates_by_page_sizes(review_file_df, page_sizes_df)
-    
-    review_file_df = fill_missing_ids(review_file_df)
+    Ensures absolute coordinates, handles missing IDs, deduplicates based on key fields,
+    selects final columns, and structures data per image/page based on page_sizes.
 
-    if 'id' not in review_file_df.columns:
-        review_file_df['id'] = ''
-        review_file_df['id'] = review_file_df['id'].astype(str)
-           
-    # Keep only necessary columns
-    review_file_df = review_file_df[["image", "page", "label", "color", "xmin", "ymin", "xmax", "ymax", "id", "text"]].drop_duplicates(subset=["image", "page", "xmin", "ymin", "xmax", "ymax", "label", "id"])
+    Args:
+        review_file_df: Input DataFrame with annotation data.
+        image_paths: List of image file paths (Note: currently unused if page_sizes provides paths).
+        page_sizes: REQUIRED list of dictionaries, each containing 'page',
+                    'image_path', 'image_width', and 'image_height'. Defines
+                    output structure and dimensions for coordinate conversion.
+        xmin, xmax, ymin, ymax: Names of the coordinate columns.
 
-    # If colours are saved as list, convert to tuple
-    review_file_df.loc[:, "color"] = review_file_df.loc[:,"color"].apply(lambda x: tuple(x) if isinstance(x, list) else x)
+    Returns:
+        List of dictionaries suitable for Gradio Annotation output, one dict per image/page.
+    """
+    review_file_df = review_file_df.dropna(subset=["xmin", "xmax", "ymin", "ymax", "text", "id", "label"])
 
-    # Group the DataFrame by the 'image' column
-    grouped_csv_pages = review_file_df.groupby('page')
+    if not page_sizes:
+        raise ValueError("page_sizes argument is required and cannot be empty.")
 
-    # Create a list to hold the JSON data
-    json_data = []
+    # --- Prepare Page Sizes DataFrame ---
+    try:
+        page_sizes_df = pd.DataFrame(page_sizes)
+        required_ps_cols = {'page', 'image_path', 'image_width', 'image_height'}
+        if not required_ps_cols.issubset(page_sizes_df.columns):
+            missing = required_ps_cols - set(page_sizes_df.columns)
+            raise ValueError(f"page_sizes is missing required keys: {missing}")
+        # Convert page sizes columns to appropriate numeric types early
+        page_sizes_df['page'] = pd.to_numeric(page_sizes_df['page'], errors='coerce')
+        page_sizes_df['image_width'] = pd.to_numeric(page_sizes_df['image_width'], errors='coerce')
+        page_sizes_df['image_height'] = pd.to_numeric(page_sizes_df['image_height'], errors='coerce')
+        # Use nullable Int64 for page number consistency
+        page_sizes_df['page'] = page_sizes_df['page'].astype('Int64')
 
-    for page_no, pdf_image_path in enumerate(page_sizes_df["image_path"]):
-        
-        reported_page_number = int(page_no + 1)
+    except Exception as e:
+        raise ValueError(f"Error processing page_sizes: {e}") from e
 
-        if reported_page_number in review_file_df["page"].values:
 
-            # Convert each relevant group to a list of box dictionaries
-            selected_csv_pages = grouped_csv_pages.get_group(reported_page_number)
-            annotation_boxes = selected_csv_pages.drop(columns=['image', 'page']).to_dict(orient='records')
-            
-            annotation = {
-                "image": pdf_image_path,
-                "boxes": annotation_boxes
-            }
+    # Handle empty input DataFrame gracefully
+    if review_file_df.empty:
+        print("Input review_file_df is empty. Proceeding to generate JSON structure with empty boxes.")
+        # Ensure essential columns exist even if empty for later steps
+        for col in [xmin, xmax, ymin, ymax, "page", "label", "color", "id", "text"]:
+             if col not in review_file_df.columns:
+                 review_file_df[col] = pd.NA
+    else:
+        # --- Coordinate Conversion (if needed) ---
+        coord_cols_to_check = [c for c in [xmin, xmax, ymin, ymax] if c in review_file_df.columns]
+        needs_multiplication = False
+        if coord_cols_to_check:
+            temp_df_numeric = review_file_df[coord_cols_to_check].apply(pd.to_numeric, errors='coerce')
+            if temp_df_numeric.le(1).any().any(): # Check if any numeric coord <= 1 exists
+                 needs_multiplication = True
+
+        if needs_multiplication:
+            #print("Relative coordinates detected or suspected, running multiplication...")
+            review_file_df = multiply_coordinates_by_page_sizes(
+                review_file_df.copy(), # Pass a copy to avoid modifying original outside function
+                page_sizes_df,
+                xmin, xmax, ymin, ymax
+            )
+        else:
+            #print("No relative coordinates detected or required columns missing, skipping multiplication.")
+            # Still ensure essential coordinate/page columns are numeric if they exist
+            cols_to_convert = [c for c in [xmin, xmax, ymin, ymax, "page"] if c in review_file_df.columns]
+            for col in cols_to_convert:
+                review_file_df[col] = pd.to_numeric(review_file_df[col], errors='coerce')
 
+        # Handle potential case where multiplication returns an empty DF
+        if review_file_df.empty:
+            print("DataFrame became empty after coordinate processing.")
+            # Re-add essential columns if they were lost
+            for col in [xmin, xmax, ymin, ymax, "page", "label", "color", "id", "text"]:
+                if col not in review_file_df.columns:
+                    review_file_df[col] = pd.NA
+
+        # --- Fill Missing IDs ---
+        review_file_df = fill_missing_ids(review_file_df.copy()) # Pass a copy
+
+        # --- Deduplicate Based on Key Fields ---
+        base_dedupe_cols = ["page", xmin, ymin, xmax, ymax, "label", "id"]
+        # Identify which deduplication columns actually exist in the DataFrame
+        cols_for_dedupe = [col for col in base_dedupe_cols if col in review_file_df.columns]
+        # Add 'image' column for deduplication IF it exists (matches original logic intent)
+        if "image" in review_file_df.columns:
+            cols_for_dedupe.append("image")
+
+        # Ensure placeholder columns exist if they are needed for deduplication
+        # (e.g., 'label', 'id' should be present after fill_missing_ids)
+        for col in ['label', 'id']:
+            if col in cols_for_dedupe and col not in review_file_df.columns:
+                 # This might indicate an issue in fill_missing_ids or prior steps
+                 print(f"Warning: Column '{col}' needed for dedupe but not found. Adding NA.")
+                 review_file_df[col] = "" # Add default empty string
+
+        if cols_for_dedupe: # Only attempt dedupe if we have columns to check
+            #print(f"Deduplicating based on columns: {cols_for_dedupe}")
+            # Convert relevant columns to string before dedupe to avoid type issues with mixed data (optional, depends on data)
+            # for col in cols_for_dedupe:
+            #    review_file_df[col] = review_file_df[col].astype(str)
+            review_file_df = review_file_df.drop_duplicates(subset=cols_for_dedupe)
         else:
-            annotation = {}
-            annotation["image"] = pdf_image_path
-            annotation["boxes"] = []
+            print("Skipping deduplication: No valid columns found to deduplicate by.")
+
+
+    # --- Select and Prepare Final Output Columns ---
+    required_final_cols = ["page", "label", "color", xmin, ymin, xmax, ymax, "id", "text"]
+    # Identify which of the desired final columns exist in the (now potentially deduplicated) DataFrame
+    available_final_cols = [col for col in required_final_cols if col in review_file_df.columns]
+
+    # Ensure essential output columns exist, adding defaults if missing AFTER deduplication
+    for col in required_final_cols:
+         if col not in review_file_df.columns:
+             print(f"Adding missing final column '{col}' with default value.")
+             if col in ['label', 'id', 'text']:
+                 review_file_df[col] = "" # Default empty string
+             elif col == 'color':
+                 review_file_df[col] = None # Default None or a default color tuple
+             else: # page, coordinates
+                 review_file_df[col] = pd.NA # Default NA for numeric/page
+             available_final_cols.append(col) # Add to list of available columns
+
+    # Select only the final desired columns in the correct order
+    review_file_df = review_file_df[available_final_cols]
+
+    # --- Final Formatting ---
+    if not review_file_df.empty:
+        # Convert list colors to tuples (important for some downstream uses)
+        if 'color' in review_file_df.columns:
+            review_file_df['color'] = review_file_df['color'].apply(
+                lambda x: tuple(x) if isinstance(x, list) else x
+            )
+        # Ensure page column is nullable integer type for reliable grouping
+        if 'page' in review_file_df.columns:
+             review_file_df['page'] = review_file_df['page'].astype('Int64')
+
+    # --- Group Annotations by Page ---
+    if 'page' in review_file_df.columns:
+        grouped_annotations = review_file_df.groupby('page')
+        group_keys = set(grouped_annotations.groups.keys()) # Use set for faster lookups
+    else:
+        # Cannot group if page column is missing
+        print("Error: 'page' column missing, cannot group annotations.")
+        grouped_annotations = None
+        group_keys = set()
+
 
-        # Append the structured data to the json_data list
-        json_data.append(annotation)
+    # --- Build JSON Structure ---
+    json_data = []
+    output_cols_for_boxes = [col for col in ["label", "color", xmin, ymin, xmax, ymax, "id", "text"] if col in review_file_df.columns]
+
+    # Iterate through page_sizes_df to define the structure (one entry per image path)
+    for _, row in page_sizes_df.iterrows():
+        page_num = row['page'] # Already Int64
+        pdf_image_path = row['image_path']
+        annotation_boxes = [] # Default to empty list
 
-    return json_data
+        # Check if the page exists in the grouped annotations (using the faster set lookup)
+        # Check pd.notna because page_num could be <NA> if conversion failed
+        if pd.notna(page_num) and page_num in group_keys and grouped_annotations:
+            try:
+                page_group_df = grouped_annotations.get_group(page_num)
+                # Convert the group to list of dicts, selecting only needed box properties
+                # Handle potential NaN coordinates before conversion to JSON
+                annotation_boxes = page_group_df[output_cols_for_boxes].replace({np.nan: None}).to_dict(orient='records')
+
+                # Optional: Round coordinates here if needed AFTER potential multiplication
+                # for box in annotation_boxes:
+                #     for coord in [xmin, ymin, xmax, ymax]:
+                #         if coord in box and box[coord] is not None:
+                #             box[coord] = round(float(box[coord]), 2) # Example: round to 2 decimals
+
+            except KeyError:
+                 print(f"Warning: Group key {page_num} not found despite being in group_keys (should not happen).")
+                 annotation_boxes = [] # Keep empty
+
+        # Append the structured data for this image/page
+        json_data.append({
+            "image": pdf_image_path,
+            "boxes": annotation_boxes
+        })
+
+    return json_data

tools/file_redaction.py

@@ -20,8 +20,8 @@ from gradio import Progress
 from collections import defaultdict  # For efficient grouping
 
 from tools.config import OUTPUT_FOLDER, IMAGES_DPI, MAX_IMAGE_PIXELS, RUN_AWS_FUNCTIONS, AWS_ACCESS_KEY, AWS_SECRET_KEY, AWS_REGION, PAGE_BREAK_VALUE, MAX_TIME_VALUE, LOAD_TRUNCATED_IMAGES, INPUT_FOLDER
-from tools.custom_image_analyser_engine import CustomImageAnalyzerEngine, OCRResult, combine_ocr_results, CustomImageRecognizerResult, run_page_text_redaction, merge_text_bounding_boxes
-from tools.file_conversion import convert_annotation_json_to_review_df, redact_whole_pymupdf_page, redact_single_box, convert_pymupdf_to_image_coords, is_pdf, is_pdf_or_image, prepare_image_or_pdf, divide_coordinates_by_page_sizes, multiply_coordinates_by_page_sizes, convert_annotation_data_to_dataframe, divide_coordinates_by_page_sizes, create_annotation_dicts_from_annotation_df, remove_duplicate_images_with_blank_boxes, fill_missing_ids, fill_missing_box_ids
+from tools.custom_image_analyser_engine import CustomImageAnalyzerEngine, OCRResult, combine_ocr_results, CustomImageRecognizerResult, run_page_text_redaction, merge_text_bounding_boxes, recreate_page_line_level_ocr_results_with_page
+from tools.file_conversion import convert_annotation_json_to_review_df, redact_whole_pymupdf_page, redact_single_box, convert_pymupdf_to_image_coords, is_pdf, is_pdf_or_image, prepare_image_or_pdf, divide_coordinates_by_page_sizes, multiply_coordinates_by_page_sizes, convert_annotation_data_to_dataframe, divide_coordinates_by_page_sizes, create_annotation_dicts_from_annotation_df, remove_duplicate_images_with_blank_boxes, fill_missing_ids, fill_missing_box_ids, load_and_convert_ocr_results_with_words_json
 from tools.load_spacy_model_custom_recognisers import nlp_analyser, score_threshold, custom_entities, custom_recogniser, custom_word_list_recogniser, CustomWordFuzzyRecognizer
 from tools.helper_functions import get_file_name_without_type, clean_unicode_text, tesseract_ocr_option, text_ocr_option, textract_option, local_pii_detector, aws_pii_detector, no_redaction_option
 from tools.aws_textract import analyse_page_with_textract, json_to_ocrresult, load_and_convert_textract_json
@@ -101,6 +101,8 @@ def choose_and_run_redactor(file_paths:List[str],
  input_folder:str=INPUT_FOLDER,
  total_textract_query_number:int=0,
  ocr_file_path:str="",
+ all_page_line_level_ocr_results = [],
+ all_page_line_level_ocr_results_with_words = [],
  prepare_images:bool=True,
  progress=gr.Progress(track_tqdm=True)):
     '''
@@ -149,7 +151,9 @@ def choose_and_run_redactor(file_paths:List[str],
     - review_file_path (str, optional): The latest review file path created by the app
     - input_folder (str, optional): The custom input path, if provided
     - total_textract_query_number (int, optional): The number of textract queries up until this point.
-    - ocr_file_path (str, optional): The latest ocr file path created by the app
+    - ocr_file_path (str, optional): The latest ocr file path created by the app.
+    - all_page_line_level_ocr_results (list, optional): All line level text on the page with bounding boxes.
+    - all_page_line_level_ocr_results_with_words (list, optional): All word level text on the page with bounding boxes.
     - prepare_images (bool, optional): Boolean to determine whether to load images for the PDF.
     - progress (gr.Progress, optional): A progress tracker for the redaction process. Defaults to a Progress object with track_tqdm set to True.
 
@@ -179,9 +183,16 @@ def choose_and_run_redactor(file_paths:List[str],
         out_file_paths = []
         estimate_total_processing_time = 0
         estimated_time_taken_state = 0
+        comprehend_query_number = 0
+        total_textract_query_number = 0
+    elif current_loop_page == 0:
+        comprehend_query_number = 0
+        total_textract_query_number = 0
     # If not the first time around, and the current page loop has been set to a huge number (been through all pages), reset current page to 0
     elif (first_loop_state == False) & (current_loop_page == 999):
         current_loop_page = 0
+        total_textract_query_number = 0
+        comprehend_query_number = 0   
 
     # Choose the correct file to prepare
     if isinstance(file_paths, str): file_paths_list = [os.path.abspath(file_paths)]
@@ -219,6 +230,8 @@ def choose_and_run_redactor(file_paths:List[str],
         elif out_message:
             combined_out_message = combined_out_message + '\n' + out_message
 
+        combined_out_message = re.sub(r'^\n+', '', combined_out_message).strip()
+
         # Only send across review file if redaction has been done
         if pii_identification_method != no_redaction_option:
 
@@ -226,10 +239,15 @@ def choose_and_run_redactor(file_paths:List[str],
                 #review_file_path = [x for x in out_file_paths if "review_file" in x]
                 if review_file_path: review_out_file_paths.append(review_file_path)
         
+        if not isinstance(pymupdf_doc, list):
+            number_of_pages = pymupdf_doc.page_count
+            if total_textract_query_number  > number_of_pages:
+                total_textract_query_number = number_of_pages
+
         estimate_total_processing_time = sum_numbers_before_seconds(combined_out_message)
         print("Estimated total processing time:", str(estimate_total_processing_time))
 
-        return combined_out_message, out_file_paths, out_file_paths, gr.Number(value=latest_file_completed, label="Number of documents redacted", interactive=False, visible=False), log_files_output_paths, log_files_output_paths, estimated_time_taken_state, all_request_metadata_str, pymupdf_doc, annotations_all_pages, gr.Number(value=current_loop_page,precision=0, interactive=False, label = "Last redacted page in document", visible=False), gr.Checkbox(value = True, label="Page break reached", visible=False), all_line_level_ocr_results_df, all_pages_decision_process_table, comprehend_query_number, review_out_file_paths, annotate_max_pages, annotate_max_pages, prepared_pdf_file_paths, pdf_image_file_paths, review_file_state, page_sizes, duplication_file_path_outputs, duplication_file_path_outputs, review_file_path, total_textract_query_number, ocr_file_path
+        return combined_out_message, out_file_paths, out_file_paths, gr.Number(value=latest_file_completed, label="Number of documents redacted", interactive=False, visible=False), log_files_output_paths, log_files_output_paths, estimated_time_taken_state, all_request_metadata_str, pymupdf_doc, annotations_all_pages, gr.Number(value=current_loop_page,precision=0, interactive=False, label = "Last redacted page in document", visible=False), gr.Checkbox(value = True, label="Page break reached", visible=False), all_line_level_ocr_results_df, all_pages_decision_process_table, comprehend_query_number, review_out_file_paths, annotate_max_pages, annotate_max_pages, prepared_pdf_file_paths, pdf_image_file_paths, review_file_state, page_sizes, duplication_file_path_outputs, duplication_file_path_outputs, review_file_path, total_textract_query_number, ocr_file_path, all_page_line_level_ocr_results, all_page_line_level_ocr_results_with_words
 
     #if first_loop_state == False:
     # Prepare documents and images as required if they don't already exist
@@ -258,9 +276,8 @@ def choose_and_run_redactor(file_paths:List[str],
 
 
     # Call prepare_image_or_pdf only if needed
-    if prepare_images_flag is not None:# and first_loop_state==True:
-        #print("Calling preparation function. prepare_images_flag:", prepare_images_flag)
-        out_message, prepared_pdf_file_paths, pdf_image_file_paths, annotate_max_pages, annotate_max_pages_bottom, pymupdf_doc, annotations_all_pages, review_file_state, document_cropboxes, page_sizes, textract_output_found, all_img_details_state, placeholder_ocr_results_df = prepare_image_or_pdf(
+    if prepare_images_flag is not None:
+        out_message, prepared_pdf_file_paths, pdf_image_file_paths, annotate_max_pages, annotate_max_pages_bottom, pymupdf_doc, annotations_all_pages, review_file_state, document_cropboxes, page_sizes, textract_output_found, all_img_details_state, placeholder_ocr_results_df, local_ocr_output_found_checkbox = prepare_image_or_pdf(
             file_paths_loop, text_extraction_method, 0, out_message, True, 
             annotate_max_pages, annotations_all_pages, document_cropboxes, redact_whole_page_list, 
             output_folder, prepare_images=prepare_images_flag, page_sizes=page_sizes, input_folder=input_folder
@@ -275,11 +292,15 @@ def choose_and_run_redactor(file_paths:List[str],
     page_sizes = page_sizes_df.to_dict(orient="records")
 
     number_of_pages = pymupdf_doc.page_count
+    
 
     # If we have reached the last page, return message and outputs
     if current_loop_page >= number_of_pages:
         print("Reached last page of document:", current_loop_page)
 
+        if total_textract_query_number  > number_of_pages:
+            total_textract_query_number = number_of_pages
+
         # Set to a very high number so as not to mix up with subsequent file processing by the user
         current_loop_page = 999
         if out_message:
@@ -292,7 +313,7 @@ def choose_and_run_redactor(file_paths:List[str],
                 #review_file_path = [x for x in out_file_paths if "review_file" in x]
                 if review_file_path: review_out_file_paths.append(review_file_path)
 
-        return combined_out_message, out_file_paths, out_file_paths, gr.Number(value=latest_file_completed, label="Number of documents redacted", interactive=False, visible=False), log_files_output_paths, log_files_output_paths, estimated_time_taken_state, all_request_metadata_str, pymupdf_doc, annotations_all_pages, gr.Number(value=current_loop_page,precision=0, interactive=False, label = "Last redacted page in document", visible=False), gr.Checkbox(value = False, label="Page break reached", visible=False), all_line_level_ocr_results_df, all_pages_decision_process_table, comprehend_query_number, review_out_file_paths, annotate_max_pages, annotate_max_pages, prepared_pdf_file_paths, pdf_image_file_paths, review_file_state, page_sizes, duplication_file_path_outputs, duplication_file_path_outputs, review_file_path, total_textract_query_number, ocr_file_path
+        return combined_out_message, out_file_paths, out_file_paths, gr.Number(value=latest_file_completed, label="Number of documents redacted", interactive=False, visible=False), log_files_output_paths, log_files_output_paths, estimated_time_taken_state, all_request_metadata_str, pymupdf_doc, annotations_all_pages, gr.Number(value=current_loop_page,precision=0, interactive=False, label = "Last redacted page in document", visible=False), gr.Checkbox(value = False, label="Page break reached", visible=False), all_line_level_ocr_results_df, all_pages_decision_process_table, comprehend_query_number, review_out_file_paths, annotate_max_pages, annotate_max_pages, prepared_pdf_file_paths, pdf_image_file_paths, review_file_state, page_sizes, duplication_file_path_outputs, duplication_file_path_outputs, review_file_path, total_textract_query_number, ocr_file_path, all_page_line_level_ocr_results, all_page_line_level_ocr_results_with_words
 
     # Load/create allow list
     # If string, assume file path
@@ -333,7 +354,7 @@ def choose_and_run_redactor(file_paths:List[str],
     # Try to connect to AWS services directly only if RUN_AWS_FUNCTIONS environmental variable is 1, otherwise an environment variable or direct textbox input is needed.
     if pii_identification_method == aws_pii_detector:
         if aws_access_key_textbox and aws_secret_key_textbox:
-            print("Connecting to Comprehend using AWS access key and secret keys from textboxes.")
+            print("Connecting to Comprehend using AWS access key and secret keys from user input.")
             comprehend_client = boto3.client('comprehend', 
                 aws_access_key_id=aws_access_key_textbox, 
                 aws_secret_access_key=aws_secret_key_textbox, region_name=AWS_REGION)
@@ -356,7 +377,7 @@ def choose_and_run_redactor(file_paths:List[str],
     # Try to connect to AWS Textract Client if using that text extraction method
     if text_extraction_method == textract_option:   
         if aws_access_key_textbox and aws_secret_key_textbox:
-            print("Connecting to Textract using AWS access key and secret keys from textboxes.")
+            print("Connecting to Textract using AWS access key and secret keys from user input.")
             textract_client = boto3.client('textract', 
                 aws_access_key_id=aws_access_key_textbox, 
                 aws_secret_access_key=aws_secret_key_textbox, region_name=AWS_REGION)
@@ -401,7 +422,7 @@ def choose_and_run_redactor(file_paths:List[str],
             is_a_pdf = is_pdf(file_path) == True
             if is_a_pdf == False and text_extraction_method == text_ocr_option:
                 # If user has not submitted a pdf, assume it's an image
-                print("File is not a pdf, assuming that image analysis needs to be used.")
+                print("File is not a PDF, assuming that image analysis needs to be used.")
                 text_extraction_method = tesseract_ocr_option
         else:
             out_message = "No file selected"
@@ -422,7 +443,7 @@ def choose_and_run_redactor(file_paths:List[str],
 
             print("Redacting file " + pdf_file_name_with_ext + " as an image-based file")
 
-            pymupdf_doc, all_pages_decision_process_table, out_file_paths, new_textract_request_metadata, annotations_all_pages, current_loop_page, page_break_return, all_line_level_ocr_results_df, comprehend_query_number = redact_image_pdf(file_path,
+            pymupdf_doc, all_pages_decision_process_table, out_file_paths, new_textract_request_metadata, annotations_all_pages, current_loop_page, page_break_return, all_line_level_ocr_results_df, comprehend_query_number, all_page_line_level_ocr_results, all_page_line_level_ocr_results_with_words = redact_image_pdf(file_path,
              pdf_image_file_paths,
              language,
              chosen_redact_entities,
@@ -448,7 +469,9 @@ def choose_and_run_redactor(file_paths:List[str],
              max_fuzzy_spelling_mistakes_num,
              match_fuzzy_whole_phrase_bool,
              page_sizes_df,
-             text_extraction_only,             
+             text_extraction_only,
+             all_page_line_level_ocr_results,
+             all_page_line_level_ocr_results_with_words,
              log_files_output_paths=log_files_output_paths,
              output_folder=output_folder)
                         
@@ -599,7 +622,10 @@ def choose_and_run_redactor(file_paths:List[str],
     if not review_file_path: review_out_file_paths = [prepared_pdf_file_paths[-1]]
     else: review_out_file_paths = [prepared_pdf_file_paths[-1], review_file_path]
 
-    return combined_out_message, out_file_paths, out_file_paths, gr.Number(value=latest_file_completed, label="Number of documents redacted", interactive=False, visible=False), log_files_output_paths, log_files_output_paths, estimated_time_taken_state, all_request_metadata_str, pymupdf_doc, annotations_all_pages, gr.Number(value=current_loop_page, precision=0, interactive=False, label = "Last redacted page in document", visible=False), gr.Checkbox(value = True, label="Page break reached", visible=False), all_line_level_ocr_results_df, all_pages_decision_process_table, comprehend_query_number, review_out_file_paths, annotate_max_pages, annotate_max_pages, prepared_pdf_file_paths, pdf_image_file_paths, review_file_state, page_sizes, duplication_file_path_outputs, duplication_file_path_outputs, review_file_path, total_textract_query_number, ocr_file_path
+    if total_textract_query_number > number_of_pages:
+        total_textract_query_number = number_of_pages
+
+    return combined_out_message, out_file_paths, out_file_paths, gr.Number(value=latest_file_completed, label="Number of documents redacted", interactive=False, visible=False), log_files_output_paths, log_files_output_paths, estimated_time_taken_state, all_request_metadata_str, pymupdf_doc, annotations_all_pages, gr.Number(value=current_loop_page, precision=0, interactive=False, label = "Last redacted page in document", visible=False), gr.Checkbox(value = True, label="Page break reached", visible=False), all_line_level_ocr_results_df, all_pages_decision_process_table, comprehend_query_number, review_out_file_paths, annotate_max_pages, annotate_max_pages, prepared_pdf_file_paths, pdf_image_file_paths, review_file_state, page_sizes, duplication_file_path_outputs, duplication_file_path_outputs, review_file_path, total_textract_query_number, ocr_file_path, all_page_line_level_ocr_results, all_page_line_level_ocr_results_with_words
 
 def convert_pikepdf_coords_to_pymupdf(pymupdf_page:Page, pikepdf_bbox, type="pikepdf_annot"):
     '''
@@ -862,17 +888,6 @@ def convert_pikepdf_annotations_to_result_annotation_box(page:Page, annot:dict,
 
     rect = Rect(pymupdf_x1, pymupdf_y1, pymupdf_x2, pymupdf_y2)
 
-    # if image or image_dimensions:
-    #     print("Dividing result by image coordinates")
-
-    #     image_x1, image_y1, image_x2, image_y2 = convert_pymupdf_to_image_coords(page, pymupdf_x1, pymupdf_y1, pymupdf_x2, pymupdf_y2, image, image_dimensions=image_dimensions)
-
-    #     img_annotation_box["xmin"] = image_x1
-    #     img_annotation_box["ymin"] = image_y1
-    #     img_annotation_box["xmax"] = image_x2
-    #     img_annotation_box["ymax"] = image_y2
-
-    # else:
     convert_df = pd.DataFrame({
                             "page": [page_no],
                             "xmin": [pymupdf_x1],
@@ -1016,9 +1031,6 @@ def redact_page_with_pymupdf(page:Page, page_annotations:dict, image:Image=None,
 
             img_annotation_box = fill_missing_box_ids(img_annotation_box)
 
-            #print("image_dimensions:", image_dimensions)
-            #print("annot:", annot)
-
         all_image_annotation_boxes.append(img_annotation_box)
 
         # Redact the annotations from the document
@@ -1178,7 +1190,9 @@ def redact_image_pdf(file_path:str,
                      max_fuzzy_spelling_mistakes_num:int=1,
                      match_fuzzy_whole_phrase_bool:bool=True,
                      page_sizes_df:pd.DataFrame=pd.DataFrame(),   
-                     text_extraction_only:bool=False,                  
+                     text_extraction_only:bool=False,
+                     all_page_line_level_ocr_results = [],
+                     all_page_line_level_ocr_results_with_words = [],
                      page_break_val:int=int(PAGE_BREAK_VALUE),
                      log_files_output_paths:List=[],
                      max_time:int=int(MAX_TIME_VALUE),
@@ -1250,7 +1264,6 @@ def redact_image_pdf(file_path:str,
         print(out_message_warning)
         #raise Exception(out_message)   
 
-
     number_of_pages = pymupdf_doc.page_count
     print("Number of pages:", str(number_of_pages))
 
@@ -1268,14 +1281,24 @@ def redact_image_pdf(file_path:str,
         textract_data, is_missing, log_files_output_paths = load_and_convert_textract_json(textract_json_file_path, log_files_output_paths, page_sizes_df)
         original_textract_data = textract_data.copy()
 
+        print("Successfully loaded in Textract analysis results from file")
+
+    # If running local OCR option, check if file already exists. If it does, load in existing data
+    if text_extraction_method == tesseract_ocr_option:                
+        all_page_line_level_ocr_results_with_words_json_file_path = output_folder + file_name + "_ocr_results_with_words.json"
+        all_page_line_level_ocr_results_with_words, is_missing, log_files_output_paths = load_and_convert_ocr_results_with_words_json(all_page_line_level_ocr_results_with_words_json_file_path, log_files_output_paths, page_sizes_df)
+        original_all_page_line_level_ocr_results_with_words = all_page_line_level_ocr_results_with_words.copy()
+
+        print("Loaded in local OCR analysis results from file")
+
     ###
     if current_loop_page == 0: page_loop_start = 0
     else: page_loop_start = current_loop_page
 
     progress_bar = tqdm(range(page_loop_start, number_of_pages), unit="pages remaining", desc="Redacting pages")    
 
-    all_pages_decision_process_table_list = [all_pages_decision_process_table]
     all_line_level_ocr_results_df_list = [all_line_level_ocr_results_df]
+    all_pages_decision_process_table_list = [all_pages_decision_process_table]    
 
     # Go through each page
     for page_no in progress_bar:
@@ -1283,10 +1306,9 @@ def redact_image_pdf(file_path:str,
         handwriting_or_signature_boxes = []
         page_signature_recogniser_results = []
         page_handwriting_recogniser_results = []
+        page_line_level_ocr_results_with_words = []
         page_break_return = False
         reported_page_number = str(page_no + 1)
-
-        #print("page_sizes_df for row:", page_sizes_df.loc[page_sizes_df["page"] == (page_no + 1)])
         
         # Try to find image location
         try:
@@ -1328,14 +1350,50 @@ def redact_image_pdf(file_path:str,
 
             # Step 1: Perform OCR. Either with Tesseract, or with AWS Textract
 
-            # If using Tesseract, need to check if we have page as image_path
+            # If using Tesseract
             if text_extraction_method == tesseract_ocr_option:
                 #print("image_path:", image_path)
                 #print("print(type(image_path)):", print(type(image_path)))
                 #if not isinstance(image_path, image_path.image_path) or not isinstance(image_path, str): raise Exception("image_path object for page", reported_page_number, "not found, cannot perform local OCR analysis.")
 
-                page_word_level_ocr_results = image_analyser.perform_ocr(image_path)
-                page_line_level_ocr_results, page_line_level_ocr_results_with_children = combine_ocr_results(page_word_level_ocr_results)
+                # Check for existing page_line_level_ocr_results_with_words object:
+
+                # page_line_level_ocr_results = (
+                # all_page_line_level_ocr_results.get('results', [])
+                # if all_page_line_level_ocr_results.get('page') == reported_page_number
+                # else []
+                # )
+
+                if all_page_line_level_ocr_results_with_words:
+                    # Find the first dict where 'page' matches
+
+                    #print("all_page_line_level_ocr_results_with_words:", all_page_line_level_ocr_results_with_words)
+
+                    print("All pages available:", [item.get('page') for item in all_page_line_level_ocr_results_with_words])
+                    #print("Looking for page:", reported_page_number)
+
+                    matching_page = next(
+                    (item for item in all_page_line_level_ocr_results_with_words if int(item.get('page', -1)) == int(reported_page_number)),
+                    None
+                    )
+
+                    #print("matching_page:", matching_page)
+
+                    page_line_level_ocr_results_with_words = matching_page if matching_page else []
+                else: page_line_level_ocr_results_with_words = []
+
+                if page_line_level_ocr_results_with_words:
+                    print("Found OCR results for page in existing OCR with words object")
+                    page_line_level_ocr_results = recreate_page_line_level_ocr_results_with_page(page_line_level_ocr_results_with_words)
+                else:
+                    page_word_level_ocr_results = image_analyser.perform_ocr(image_path)
+
+                    print("page_word_level_ocr_results:", page_word_level_ocr_results)
+                    page_line_level_ocr_results, page_line_level_ocr_results_with_words = combine_ocr_results(page_word_level_ocr_results, page=reported_page_number)
+
+                    all_page_line_level_ocr_results_with_words.append(page_line_level_ocr_results_with_words)
+
+                    print("All pages available:", [item.get('page') for item in all_page_line_level_ocr_results_with_words])
     
             # Check if page exists in existing textract data. If not, send to service to analyse
             if text_extraction_method == textract_option:
@@ -1399,16 +1457,28 @@ def redact_image_pdf(file_path:str,
                         # If the page exists, retrieve the data
                         text_blocks = next(page['data'] for page in textract_data["pages"] if page['page_no'] == reported_page_number)
                 
-                
-                page_line_level_ocr_results, handwriting_or_signature_boxes, page_signature_recogniser_results, page_handwriting_recogniser_results, page_line_level_ocr_results_with_children = json_to_ocrresult(text_blocks, page_width, page_height, reported_page_number)
+                page_line_level_ocr_results, handwriting_or_signature_boxes, page_signature_recogniser_results, page_handwriting_recogniser_results, page_line_level_ocr_results_with_words = json_to_ocrresult(text_blocks, page_width, page_height, reported_page_number)
+
+            # Convert to DataFrame and add to ongoing logging table
+            line_level_ocr_results_df = pd.DataFrame([{
+                'page': page_line_level_ocr_results['page'],
+                'text': result.text,
+                'left': result.left,
+                'top': result.top,
+                'width': result.width,
+                'height': result.height
+            } for result in page_line_level_ocr_results['results']])
+
+            all_line_level_ocr_results_df_list.append(line_level_ocr_results_df)
+
 
             if pii_identification_method != no_redaction_option:
                 # Step 2: Analyse text and identify PII
                 if chosen_redact_entities or chosen_redact_comprehend_entities:
 
                     page_redaction_bounding_boxes, comprehend_query_number_new = image_analyser.analyze_text(
-                        page_line_level_ocr_results,
-                        page_line_level_ocr_results_with_children,
+                        page_line_level_ocr_results['results'],
+                        page_line_level_ocr_results_with_words['results'],
                         chosen_redact_comprehend_entities = chosen_redact_comprehend_entities,
                         pii_identification_method = pii_identification_method,
                         comprehend_client=comprehend_client,                 
@@ -1423,7 +1493,7 @@ def redact_image_pdf(file_path:str,
                 else: page_redaction_bounding_boxes = []
 
                 # Merge redaction bounding boxes that are close together
-                page_merged_redaction_bboxes = merge_img_bboxes(page_redaction_bounding_boxes, page_line_level_ocr_results_with_children, page_signature_recogniser_results, page_handwriting_recogniser_results, handwrite_signature_checkbox)
+                page_merged_redaction_bboxes = merge_img_bboxes(page_redaction_bounding_boxes, page_line_level_ocr_results_with_words['results'], page_signature_recogniser_results, page_handwriting_recogniser_results, handwrite_signature_checkbox)
 
             else: page_merged_redaction_bboxes = []           
             
@@ -1449,7 +1519,6 @@ def redact_image_pdf(file_path:str,
                     # Assume image_path is an image
                     image = image_path
 
-                print("image:", image)
 
                 fill = (0, 0, 0)   # Fill colour for redactions
                 draw = ImageDraw.Draw(image)
@@ -1510,19 +1579,6 @@ def redact_image_pdf(file_path:str,
             decision_process_table = fill_missing_ids(decision_process_table)
             #decision_process_table.to_csv("output/decision_process_table_with_ids.csv")
 
-
-            # Convert to DataFrame and add to ongoing logging table
-            line_level_ocr_results_df = pd.DataFrame([{
-                'page': reported_page_number,
-                'text': result.text,
-                'left': result.left,
-                'top': result.top,
-                'width': result.width,
-                'height': result.height
-            } for result in page_line_level_ocr_results])
-
-            all_line_level_ocr_results_df_list.append(line_level_ocr_results_df)
-
             toc = time.perf_counter()
 
             time_taken = toc - tic
@@ -1547,6 +1603,8 @@ def redact_image_pdf(file_path:str,
                     # Append new annotation if it doesn't exist
                     annotations_all_pages.append(page_image_annotations)
 
+
+
                 if text_extraction_method == textract_option:
                     if original_textract_data != textract_data:
                         # Write the updated existing textract data back to the JSON file
@@ -1556,12 +1614,21 @@ def redact_image_pdf(file_path:str,
                     if textract_json_file_path not in log_files_output_paths:
                         log_files_output_paths.append(textract_json_file_path)
 
+                if text_extraction_method == tesseract_ocr_option:
+                    if original_all_page_line_level_ocr_results_with_words != all_page_line_level_ocr_results_with_words:
+                        # Write the updated existing textract data back to the JSON file
+                        with open(all_page_line_level_ocr_results_with_words_json_file_path, 'w') as json_file:
+                            json.dump(all_page_line_level_ocr_results_with_words, json_file, separators=(",", ":"))  # indent=4 makes the JSON file pretty-printed
+
+                    if all_page_line_level_ocr_results_with_words_json_file_path not in log_files_output_paths:
+                        log_files_output_paths.append(all_page_line_level_ocr_results_with_words_json_file_path)
+
                 all_pages_decision_process_table = pd.concat(all_pages_decision_process_table_list)
                 all_line_level_ocr_results_df = pd.concat(all_line_level_ocr_results_df_list)
 
                 current_loop_page += 1
 
-                return pymupdf_doc, all_pages_decision_process_table, log_files_output_paths, textract_request_metadata, annotations_all_pages, current_loop_page, page_break_return, all_line_level_ocr_results_df, comprehend_query_number
+                return pymupdf_doc, all_pages_decision_process_table, log_files_output_paths, textract_request_metadata, annotations_all_pages, current_loop_page, page_break_return, all_line_level_ocr_results_df, comprehend_query_number, all_page_line_level_ocr_results, all_page_line_level_ocr_results_with_words
 
         # If it's an image file
         if is_pdf(file_path) == False:
@@ -1594,10 +1661,20 @@ def redact_image_pdf(file_path:str,
                 if textract_json_file_path not in log_files_output_paths:
                     log_files_output_paths.append(textract_json_file_path)
 
+            if text_extraction_method == tesseract_ocr_option:
+                if original_all_page_line_level_ocr_results_with_words != all_page_line_level_ocr_results_with_words:
+                    # Write the updated existing textract data back to the JSON file
+                    with open(all_page_line_level_ocr_results_with_words_json_file_path, 'w') as json_file:
+                        json.dump(all_page_line_level_ocr_results_with_words, json_file, separators=(",", ":"))  # indent=4 makes the JSON file pretty-printed
+
+                if all_page_line_level_ocr_results_with_words_json_file_path not in log_files_output_paths:
+                    log_files_output_paths.append(all_page_line_level_ocr_results_with_words_json_file_path)
+
+
             all_pages_decision_process_table = pd.concat(all_pages_decision_process_table_list)
             all_line_level_ocr_results_df = pd.concat(all_line_level_ocr_results_df_list)
 
-            return pymupdf_doc, all_pages_decision_process_table, log_files_output_paths, textract_request_metadata, annotations_all_pages, current_loop_page, page_break_return, all_line_level_ocr_results_df, comprehend_query_number
+            return pymupdf_doc, all_pages_decision_process_table, log_files_output_paths, textract_request_metadata, annotations_all_pages, current_loop_page, page_break_return, all_line_level_ocr_results_df, comprehend_query_number, all_page_line_level_ocr_results, all_page_line_level_ocr_results_with_words
                
     if text_extraction_method == textract_option:
         # Write the updated existing textract data back to the JSON file
@@ -1609,15 +1686,24 @@ def redact_image_pdf(file_path:str,
         if textract_json_file_path not in log_files_output_paths:
             log_files_output_paths.append(textract_json_file_path)
 
+    if text_extraction_method == tesseract_ocr_option:
+        if original_all_page_line_level_ocr_results_with_words != all_page_line_level_ocr_results_with_words:
+            # Write the updated existing textract data back to the JSON file
+            with open(all_page_line_level_ocr_results_with_words_json_file_path, 'w') as json_file:
+                json.dump(all_page_line_level_ocr_results_with_words, json_file, separators=(",", ":"))  # indent=4 makes the JSON file pretty-printed
+
+        if all_page_line_level_ocr_results_with_words_json_file_path not in log_files_output_paths:
+            log_files_output_paths.append(all_page_line_level_ocr_results_with_words_json_file_path)
+
     all_pages_decision_process_table = pd.concat(all_pages_decision_process_table_list)
     all_line_level_ocr_results_df = pd.concat(all_line_level_ocr_results_df_list)
 
-    # Convert decision table to relative coordinates
+    # Convert decision table and ocr results to relative coordinates
     all_pages_decision_process_table = divide_coordinates_by_page_sizes(all_pages_decision_process_table, page_sizes_df, xmin="xmin", xmax="xmax", ymin="ymin", ymax="ymax")
 
     all_line_level_ocr_results_df = divide_coordinates_by_page_sizes(all_line_level_ocr_results_df, page_sizes_df, xmin="left", xmax="width", ymin="top", ymax="height")
 
-    return pymupdf_doc, all_pages_decision_process_table, log_files_output_paths, textract_request_metadata, annotations_all_pages, current_loop_page, page_break_return, all_line_level_ocr_results_df, comprehend_query_number
+    return pymupdf_doc, all_pages_decision_process_table, log_files_output_paths, textract_request_metadata, annotations_all_pages, current_loop_page, page_break_return, all_line_level_ocr_results_df, comprehend_query_number, all_page_line_level_ocr_results, all_page_line_level_ocr_results_with_words
 
 
 ###
@@ -1631,8 +1717,6 @@ def get_text_container_characters(text_container:LTTextContainer):
                     for line in text_container
                     if isinstance(line, LTTextLine) or isinstance(line, LTTextLineHorizontal)
                     for char in line]
-        
-        #print("Initial characters:", characters)
     
         return characters
     return []
@@ -1762,9 +1846,6 @@ def create_text_redaction_process_results(analyser_results, analysed_bounding_bo
         analysed_bounding_boxes_df_new = pd.concat([analysed_bounding_boxes_df_new, analysed_bounding_boxes_df_text], axis = 1)
         analysed_bounding_boxes_df_new['page'] = page_num + 1
 
-        #analysed_bounding_boxes_df_new = fill_missing_ids(analysed_bounding_boxes_df_new)
-        analysed_bounding_boxes_df_new.to_csv("output/analysed_bounding_boxes_df_new_with_ids.csv")
-
         decision_process_table = pd.concat([decision_process_table, analysed_bounding_boxes_df_new], axis = 0).drop('result', axis=1)
     
     return decision_process_table
@@ -1772,7 +1853,6 @@ def create_text_redaction_process_results(analyser_results, analysed_bounding_bo
 def create_pikepdf_annotations_for_bounding_boxes(analysed_bounding_boxes):
     pikepdf_redaction_annotations_on_page = []
     for analysed_bounding_box in analysed_bounding_boxes:
-        #print("analysed_bounding_box:", analysed_bounding_boxes)
 
         bounding_box = analysed_bounding_box["boundingBox"]
         annotation = Dictionary(
@@ -1997,7 +2077,6 @@ def redact_text_pdf(
                     pass
                     #print("Not redacting page:", page_no)
 
-                #print("page_image_annotations after page", reported_page_number, "are", page_image_annotations)
 
                 # Join extracted text outputs for all lines together
                 if not page_text_ocr_outputs.empty:

tools/helper_functions.py

@@ -9,7 +9,7 @@ import unicodedata
 from typing import List
 from math import ceil
 from gradio_image_annotation import image_annotator
-from tools.config import CUSTOM_HEADER_VALUE, CUSTOM_HEADER, OUTPUT_FOLDER, INPUT_FOLDER, SESSION_OUTPUT_FOLDER, AWS_USER_POOL_ID, TEXTRACT_BULK_ANALYSIS_INPUT_SUBFOLDER, TEXTRACT_BULK_ANALYSIS_OUTPUT_SUBFOLDER, TEXTRACT_JOBS_S3_LOC, TEXTRACT_JOBS_LOCAL_LOC
+from tools.config import CUSTOM_HEADER_VALUE, CUSTOM_HEADER, OUTPUT_FOLDER, INPUT_FOLDER, SESSION_OUTPUT_FOLDER, AWS_USER_POOL_ID, TEXTRACT_WHOLE_DOCUMENT_ANALYSIS_INPUT_SUBFOLDER, TEXTRACT_WHOLE_DOCUMENT_ANALYSIS_OUTPUT_SUBFOLDER, TEXTRACT_JOBS_S3_LOC, TEXTRACT_JOBS_LOCAL_LOC
 
 # Names for options labels
 text_ocr_option = "Local model - selectable text"
@@ -39,6 +39,12 @@ def reset_ocr_results_state():
 def reset_review_vars():
     return pd.DataFrame(), pd.DataFrame()
 
+def reset_data_vars():
+    return 0, [], 0
+
+def reset_aws_call_vars():
+    return 0, 0
+
 def load_in_default_allow_list(allow_list_file_path):
     if isinstance(allow_list_file_path, str):
         allow_list_file_path = [allow_list_file_path]
@@ -201,9 +207,6 @@ def put_columns_in_df(in_file:List[str]):
                 df = pd.read_excel(file_name, sheet_name=sheet_name)
 
                 # Process the DataFrame (e.g., print its contents)
-                print(f"Sheet Name: {sheet_name}")
-                print(df.head())  # Print the first few rows
-
                 new_choices.extend(list(df.columns))
 
             all_sheet_names.extend(new_sheet_names)
@@ -226,7 +229,17 @@ def check_for_existing_textract_file(doc_file_name_no_extension_textbox:str, out
     textract_output_path = os.path.join(output_folder, doc_file_name_no_extension_textbox + "_textract.json")
 
     if os.path.exists(textract_output_path):
-        print("Existing Textract file found.")    
+        print("Existing Textract analysis output file found.")    
+        return True
+    
+    else:
+        return False
+    
+def check_for_existing_local_ocr_file(doc_file_name_no_extension_textbox:str, output_folder:str=OUTPUT_FOLDER):
+    local_ocr_output_path = os.path.join(output_folder, doc_file_name_no_extension_textbox + "_ocr_results_with_words.json")
+
+    if os.path.exists(local_ocr_output_path):
+        print("Existing local OCR analysis output file found.")    
         return True
     
     else:
@@ -306,8 +319,8 @@ async def get_connection_params(request: gr.Request,
                                 output_folder_textbox:str=OUTPUT_FOLDER,
                                 input_folder_textbox:str=INPUT_FOLDER,
                                 session_output_folder:str=SESSION_OUTPUT_FOLDER,
-                                textract_document_upload_input_folder:str=TEXTRACT_BULK_ANALYSIS_INPUT_SUBFOLDER,
-                                textract_document_upload_output_folder:str=TEXTRACT_BULK_ANALYSIS_OUTPUT_SUBFOLDER,
+                                textract_document_upload_input_folder:str=TEXTRACT_WHOLE_DOCUMENT_ANALYSIS_INPUT_SUBFOLDER,
+                                textract_document_upload_output_folder:str=TEXTRACT_WHOLE_DOCUMENT_ANALYSIS_OUTPUT_SUBFOLDER,
                                 s3_textract_document_logs_subfolder:str=TEXTRACT_JOBS_S3_LOC,
                                 local_textract_document_logs_subfolder:str=TEXTRACT_JOBS_LOCAL_LOC):
 
@@ -477,9 +490,10 @@ def calculate_time_taken(number_of_pages:str,
                         pii_identification_method:str,
                         textract_output_found_checkbox:bool,
                         only_extract_text_radio:bool,
+                        local_ocr_output_found_checkbox:bool,
                         convert_page_time:float=0.5,
-                        textract_page_time:float=1,
-                        comprehend_page_time:float=1,
+                        textract_page_time:float=1.2,
+                        comprehend_page_time:float=1.2,
                         local_text_extraction_page_time:float=0.3,
                         local_pii_redaction_page_time:float=0.5,                        
                         local_ocr_extraction_page_time:float=1.5,
@@ -494,7 +508,9 @@ def calculate_time_taken(number_of_pages:str,
     - number_of_pages: The number of pages in the uploaded document(s).
     - text_extract_method_radio: The method of text extraction.
     - pii_identification_method_drop: The method of personally-identifiable information removal.
+    - textract_output_found_checkbox (bool, optional): Boolean indicating if AWS Textract text extraction outputs have been found.
     - only_extract_text_radio (bool, optional): Option to only extract text from the document rather than redact.
+    - local_ocr_output_found_checkbox (bool, optional): Boolean indicating if local OCR text extraction outputs have been found.
     - textract_page_time (float, optional): Approximate time to query AWS Textract.
     - comprehend_page_time (float, optional): Approximate time to query text on a page with AWS Comprehend.
     - local_text_redaction_page_time (float, optional): Approximate time to extract text on a page with the local text redaction option.
@@ -522,7 +538,8 @@ def calculate_time_taken(number_of_pages:str,
         if textract_output_found_checkbox != True:
             page_extraction_time_taken = number_of_pages * textract_page_time
     elif text_extract_method_radio == local_ocr_option:
-        page_extraction_time_taken = number_of_pages * local_ocr_extraction_page_time
+        if local_ocr_output_found_checkbox != True:
+            page_extraction_time_taken = number_of_pages * local_ocr_extraction_page_time
     elif text_extract_method_radio == text_ocr_option:
         page_conversion_time_taken = number_of_pages * local_text_extraction_page_time
 

tools/redaction_review.py

@@ -6,12 +6,11 @@ import numpy as np
 from xml.etree.ElementTree import Element, SubElement, tostring, parse
 from xml.dom import minidom
 import uuid
-from typing import List
+from typing import List, Tuple
 from gradio_image_annotation import image_annotator
 from gradio_image_annotation.image_annotator import AnnotatedImageData
 from pymupdf import Document, Rect
 import pymupdf
-#from fitz 
 from PIL import ImageDraw, Image
 
 from tools.config import OUTPUT_FOLDER, CUSTOM_BOX_COLOUR, MAX_IMAGE_PIXELS, INPUT_FOLDER
@@ -55,7 +54,6 @@ def update_zoom(current_zoom_level:int, annotate_current_page:int, decrease:bool
         
     return current_zoom_level, annotate_current_page
 
-
 def update_dropdown_list_based_on_dataframe(df:pd.DataFrame, column:str) -> List["str"]:
     '''
     Gather unique elements from a string pandas Series, then append 'ALL' to the start and return the list.
@@ -166,49 +164,205 @@ def update_recogniser_dataframes(page_image_annotator_object:AnnotatedImageData,
 
     return recogniser_entities_list, recogniser_dataframe_out_gr, recogniser_dataframe_out, recogniser_entities_drop, text_entities_drop, page_entities_drop
 
-def undo_last_removal(backup_review_state, backup_image_annotations_state, backup_recogniser_entity_dataframe_base):
+def undo_last_removal(backup_review_state:pd.DataFrame, backup_image_annotations_state:list[dict], backup_recogniser_entity_dataframe_base:pd.DataFrame):
     return backup_review_state, backup_image_annotations_state, backup_recogniser_entity_dataframe_base
 
-def update_annotator_page_from_review_df(review_df: pd.DataFrame,
-                                          image_file_paths:List[str],
-                                          page_sizes:List[dict],
-                                          current_page:int,
-                                          previous_page:int,
-                                          current_image_annotations_state:List[str],
-                                          current_page_annotator:object):
+def update_annotator_page_from_review_df(
+    review_df: pd.DataFrame,
+    image_file_paths:List[str], # Note: This input doesn't seem used in the original logic flow after the first line was removed
+    page_sizes:List[dict],
+    current_image_annotations_state:List[str], # This should ideally be List[dict] based on its usage
+    current_page_annotator:object, # Should be dict or a custom annotation object for one page
+    selected_recogniser_entity_df_row:pd.DataFrame,
+    input_folder:str,
+    doc_full_file_name_textbox:str
+) -> Tuple[object, List[dict], int, List[dict], pd.DataFrame, int]: # Correcting return types based on usage
     '''
-    Update the visible annotation object with the latest review file information
+    Update the visible annotation object and related objects with the latest review file information,
+    optimizing by processing only the current page's data.
     '''
-    out_image_annotations_state = current_image_annotations_state
-    out_current_page_annotator = current_page_annotator
-    gradio_annotator_current_page_number = current_page
+    # Assume current_image_annotations_state is List[dict] and current_page_annotator is dict
+    out_image_annotations_state: List[dict] = list(current_image_annotations_state) # Make a copy to avoid modifying input in place
+    out_current_page_annotator: dict = current_page_annotator
+
+    # Get the target page number from the selected row
+    # Safely access the page number, handling potential errors or empty DataFrame
+    gradio_annotator_current_page_number: int = 0
+    annotate_previous_page: int = 0 # Renaming for clarity if needed, matches original output
+    if not selected_recogniser_entity_df_row.empty and 'page' in selected_recogniser_entity_df_row.columns:
+        try:
+            # Use .iloc[0] and .item() for robust scalar extraction
+            gradio_annotator_current_page_number = int(selected_recogniser_entity_df_row['page'].iloc[0])
+            annotate_previous_page = gradio_annotator_current_page_number # Store original page number
+        except (IndexError, ValueError, TypeError):
+            print("Warning: Could not extract valid page number from selected_recogniser_entity_df_row. Defaulting to page 0 (or 1).")
+            gradio_annotator_current_page_number = 1 # Or 0 depending on 1-based vs 0-based indexing elsewhere
+
+    # Ensure page number is valid and 1-based for external display/logic
+    if gradio_annotator_current_page_number <= 0:
+        gradio_annotator_current_page_number = 1
+
+    page_max_reported = len(out_image_annotations_state)
+    if gradio_annotator_current_page_number > page_max_reported:
+        gradio_annotator_current_page_number = page_max_reported # Cap at max pages
+
+    page_num_reported_zero_indexed = gradio_annotator_current_page_number - 1
+
+    # Process page sizes DataFrame early, as it's needed for image path handling and potentially coordinate multiplication
+    page_sizes_df = pd.DataFrame(page_sizes)
+    if not page_sizes_df.empty:
+        # Safely convert page column to numeric and then int
+        page_sizes_df["page"] = pd.to_numeric(page_sizes_df["page"], errors="coerce")
+        page_sizes_df.dropna(subset=["page"], inplace=True)
+        if not page_sizes_df.empty:
+            page_sizes_df["page"] = page_sizes_df["page"].astype(int)
+        else:
+            print("Warning: Page sizes DataFrame became empty after processing.")
 
+    # --- OPTIMIZATION: Process only the current page's data from review_df ---
     if not review_df.empty:
-        #print("review_df just before convert_review_df:", review_df)
-        # First, check that the image on the current page is valid, replace with what exists in page_sizes object if not
-        if not gradio_annotator_current_page_number: gradio_annotator_current_page_number = 0
+        # Filter review_df for the current page
+        # Ensure 'page' column in review_df is comparable to page_num_reported
+        if 'page' in review_df.columns:
+             review_df['page'] = pd.to_numeric(review_df['page'], errors='coerce').fillna(-1).astype(int)
 
-        # Check bounding values for current page and page max
-        if gradio_annotator_current_page_number > 0: page_num_reported = gradio_annotator_current_page_number
-        elif gradio_annotator_current_page_number == 0: page_num_reported = 1 # minimum possible reported page is 1
-        else: 
-            gradio_annotator_current_page_number = 0
-            page_num_reported = 1
+             current_image_path = out_image_annotations_state[page_num_reported_zero_indexed]['image']
 
-        # Ensure page displayed can't exceed number of pages in document
-        page_max_reported = len(out_image_annotations_state)
-        if page_num_reported > page_max_reported: page_num_reported = page_max_reported
+             replaced_image_path, page_sizes_df = replace_placeholder_image_with_real_image(doc_full_file_name_textbox, current_image_path, page_sizes_df, gradio_annotator_current_page_number, input_folder)
 
-        page_num_reported_zero_indexed = page_num_reported - 1        
-        out_image_annotations_state = convert_review_df_to_annotation_json(review_df, image_file_paths, page_sizes)
+             # page_sizes_df has been changed - save back to page_sizes_object
+             page_sizes = page_sizes_df.to_dict(orient='records')
+             review_df.loc[review_df["page"]==gradio_annotator_current_page_number, 'image'] = replaced_image_path
+             images_list = list(page_sizes_df["image_path"])
+             images_list[page_num_reported_zero_indexed] = replaced_image_path
+             out_image_annotations_state[page_num_reported_zero_indexed]['image'] = replaced_image_path
 
-        page_image_annotator_object, out_image_annotations_state = replace_images_in_image_annotation_object(out_image_annotations_state, out_image_annotations_state[page_num_reported_zero_indexed], page_sizes, page_num_reported)    
+             current_page_review_df = review_df[review_df['page'] == gradio_annotator_current_page_number].copy()          
+             current_page_review_df = multiply_coordinates_by_page_sizes(current_page_review_df, page_sizes_df)
 
-        out_image_annotations_state[page_num_reported_zero_indexed] = page_image_annotator_object
+        else:
+            print(f"Warning: 'page' column not found in review_df. Cannot filter for page {gradio_annotator_current_page_number}. Skipping update from review_df.")
+            current_page_review_df = pd.DataFrame() # Empty dataframe if filter fails
+
+        if not current_page_review_df.empty:
+            # Convert the current page's review data to annotation list format for *this page*
+
+            current_page_annotations_list = []
+            # Define expected annotation dict keys, including 'image', 'page', coords, 'label', 'text', 'color' etc.
+            # Assuming review_df has compatible columns
+            expected_annotation_keys = ['label', 'color', 'xmin', 'ymin', 'xmax', 'ymax', 'text', 'id'] # Add/remove as needed
+
+            # Ensure necessary columns exist in current_page_review_df before converting rows
+            for key in expected_annotation_keys:
+                 if key not in current_page_review_df.columns:
+                      # Add missing column with default value
+                      # Use np.nan for numeric, '' for string/object
+                      default_value = np.nan if key in ['xmin', 'ymin', 'xmax', 'ymax'] else ''
+                      current_page_review_df[key] = default_value
+
+            # Convert filtered DataFrame rows to list of dicts
+            # Using .to_dict(orient='records') is efficient for this
+            current_page_annotations_list_raw = current_page_review_df[expected_annotation_keys].to_dict(orient='records')
+
+            current_page_annotations_list = current_page_annotations_list_raw
+
+            # Update the annotations state for the current page
+            # Each entry in out_image_annotations_state seems to be a dict containing keys like 'image', 'page', 'annotations' (List[dict])
+            # Need to update the 'annotations' list for the specific page.
+            # Find the entry for the current page in the state
+            page_state_entry_found = False
+            for i, page_state_entry in enumerate(out_image_annotations_state):
+                # Assuming page_state_entry has a 'page' key (1-based)
+
+                match = re.search(r"(\d+)\.png$", page_state_entry['image'])
+                if match: page_no = int(match.group(1))
+                else: page_no = -1
+
+                if 'image' in page_state_entry and page_no == page_num_reported_zero_indexed:
+                    # Replace the annotations list for this page with the new list from review_df
+                    out_image_annotations_state[i]['boxes'] = current_page_annotations_list
+
+                    # Update the image path as well, based on review_df if available, or keep existing
+                    # Assuming review_df has an 'image' column for this page
+                    if 'image' in current_page_review_df.columns and not current_page_review_df.empty:
+                         # Use the image path from the first row of the filtered review_df
+                         out_image_annotations_state[i]['image'] = current_page_review_df['image'].iloc[0]
+                    page_state_entry_found = True
+                    break
+
+            if not page_state_entry_found:
+                 # This scenario might happen if the current_image_annotations_state didn't initially contain
+                 # an entry for this page number. Depending on the application logic, you might need to
+                 # add a new entry here, but based on the original code's structure, it seems
+                 # out_image_annotations_state is pre-populated for all pages.
+                 print(f"Warning: Entry for page {gradio_annotator_current_page_number} not found in current_image_annotations_state. Cannot update page annotations.")
+
+
+    # --- Image Path and Page Size Handling (already seems focused on current page, keep similar logic) ---
+    # Get the image path for the current page from the updated state
+    # Ensure the entry exists before accessing
+    current_image_path = None
+    if len(out_image_annotations_state) > page_num_reported_zero_indexed and 'image' in out_image_annotations_state[page_num_reported_zero_indexed]:
+         current_image_path = out_image_annotations_state[page_num_reported_zero_indexed]['image']
+    else:
+         print(f"Warning: Could not get image path from state for page index {page_num_reported_zero_indexed}.")
+
+
+    # Replace placeholder image with real image path if needed
+    if current_image_path and not page_sizes_df.empty:
+        try:
+            replaced_image_path, page_sizes_df = replace_placeholder_image_with_real_image(
+                doc_full_file_name_textbox, current_image_path, page_sizes_df,
+                gradio_annotator_current_page_number, input_folder # Use 1-based page number
+            )
 
-        out_current_page_annotator = out_image_annotations_state[page_num_reported_zero_indexed]
+            # Update state and review_df with the potentially replaced image path
+            if len(out_image_annotations_state) > page_num_reported_zero_indexed:
+                 out_image_annotations_state[page_num_reported_zero_indexed]['image'] = replaced_image_path
 
-    return out_current_page_annotator, out_image_annotations_state
+            if 'page' in review_df.columns and 'image' in review_df.columns:
+                 review_df.loc[review_df["page"]==gradio_annotator_current_page_number, 'image'] = replaced_image_path
+
+        except Exception as e:
+             print(f"Error during image path replacement for page {gradio_annotator_current_page_number}: {e}")
+
+
+    # Save back page_sizes_df to page_sizes list format
+    if not page_sizes_df.empty:
+        page_sizes = page_sizes_df.to_dict(orient='records')
+    else:
+        page_sizes = [] # Ensure page_sizes is a list if df is empty
+
+    # --- Re-evaluate Coordinate Multiplication and Duplicate Removal ---
+    # The original code multiplied coordinates for the *entire* document and removed duplicates
+    # across the *entire* document *after* converting the full review_df to state.
+    # With the optimized approach, we updated only one page's annotations in the state.
+
+    # Let's assume remove_duplicate_images_with_blank_boxes expects the raw list of dicts state format:
+    try:
+         out_image_annotations_state = remove_duplicate_images_with_blank_boxes(out_image_annotations_state)
+    except Exception as e:
+         print(f"Error during duplicate removal: {e}. Proceeding without duplicate removal.")
+
+
+    # Select the current page's annotation object from the (potentially updated) state
+    if len(out_image_annotations_state) > page_num_reported_zero_indexed:
+         out_current_page_annotator = out_image_annotations_state[page_num_reported_zero_indexed]
+    else:
+         print(f"Warning: Cannot select current page annotator object for index {page_num_reported_zero_indexed}.")
+         out_current_page_annotator = {} # Or None, depending on expected output type
+
+
+    # The original code returns gradio_annotator_current_page_number as the 3rd value,
+    # which was potentially updated by bounding checks. Keep this.
+    final_page_number_returned = gradio_annotator_current_page_number
+
+    return (out_current_page_annotator,
+            out_image_annotations_state,
+            final_page_number_returned,
+            page_sizes,
+            review_df, # review_df might have its 'page' column type changed, keep it as is or revert if necessary
+            annotate_previous_page) # The original page number from selected_recogniser_entity_df_row
 
 def exclude_selected_items_from_redaction(review_df: pd.DataFrame,
                                           selected_rows_df: pd.DataFrame,
@@ -216,7 +370,7 @@ def exclude_selected_items_from_redaction(review_df: pd.DataFrame,
                                           page_sizes:List[dict],
                                           image_annotations_state:dict,
                                           recogniser_entity_dataframe_base:pd.DataFrame):
-    '''
+    ''' 
     Remove selected items from the review dataframe from the annotation object and review dataframe.
     '''
 
@@ -253,149 +407,267 @@ def exclude_selected_items_from_redaction(review_df: pd.DataFrame,
 
     return out_review_df, out_image_annotations_state, out_recogniser_entity_dataframe_base, backup_review_state, backup_image_annotations_state, backup_recogniser_entity_dataframe_base
 
-def update_annotator_object_and_filter_df(
-                    all_image_annotations:List[AnnotatedImageData],
-                    gradio_annotator_current_page_number:int,
-                    recogniser_entities_dropdown_value:str="ALL",
-                    page_dropdown_value:str="ALL",
-                    text_dropdown_value:str="ALL",
-                    recogniser_dataframe_base:gr.Dataframe=gr.Dataframe(pd.DataFrame(data={"page":[], "label":[], "text":[], "id":[]}), type="pandas", headers=["page", "label", "text", "id"], show_fullscreen_button=True, wrap=True, show_search='filter', max_height=400, static_columns=[0,1,2,3]),
-                    zoom:int=100,
-                    review_df:pd.DataFrame=[],
-                    page_sizes:List[dict]=[],
-                    doc_full_file_name_textbox:str='',
-                    input_folder:str=INPUT_FOLDER):
-    '''
-    Update a gradio_image_annotation object with new annotation data.
-    '''
-    zoom_str = str(zoom) + '%'
-
-    #print("all_image_annotations at start of update_annotator_object_and_filter_df[-1]:", all_image_annotations[-1])
-    
-    if not gradio_annotator_current_page_number: gradio_annotator_current_page_number = 0
-
-    # Check bounding values for current page and page max
-    if gradio_annotator_current_page_number > 0: page_num_reported = gradio_annotator_current_page_number
-    elif gradio_annotator_current_page_number == 0: page_num_reported = 1 # minimum possible reported page is 1
-    else: 
-        gradio_annotator_current_page_number = 0
-        page_num_reported = 1
+def replace_annotator_object_img_np_array_with_page_sizes_image_path(
+        all_image_annotations:List[dict],
+        page_image_annotator_object:AnnotatedImageData,
+        page_sizes:List[dict],
+        page:int):
 
-    # Ensure page displayed can't exceed number of pages in document
-    page_max_reported = len(all_image_annotations)
-    if page_num_reported > page_max_reported: page_num_reported = page_max_reported
+        '''
+        Check if the image value in an AnnotatedImageData dict is a placeholder or np.array. If either of these, replace the value with the file path of the image that is hopefully already loaded into the app related to this page.
+        '''
 
-    page_num_reported_zero_indexed = page_num_reported - 1
+        page_zero_index = page - 1
+        
+        if isinstance(all_image_annotations[page_zero_index]["image"], np.ndarray) or "placeholder_image" in all_image_annotations[page_zero_index]["image"] or isinstance(page_image_annotator_object['image'], np.ndarray):
+            page_sizes_df = pd.DataFrame(page_sizes)
+            page_sizes_df[["page"]] = page_sizes_df[["page"]].apply(pd.to_numeric, errors="coerce")
 
-    # First, check that the image on the current page is valid, replace with what exists in page_sizes object if not
-    page_image_annotator_object, all_image_annotations = replace_images_in_image_annotation_object(all_image_annotations, all_image_annotations[page_num_reported_zero_indexed], page_sizes, page_num_reported)    
+            # Check for matching pages
+            matching_paths = page_sizes_df.loc[page_sizes_df['page'] == page, "image_path"].unique()
 
-    all_image_annotations[page_num_reported_zero_indexed] = page_image_annotator_object
-    
-    current_image_path = all_image_annotations[page_num_reported_zero_indexed]['image']
+            if matching_paths.size > 0:
+                image_path = matching_paths[0]
+                page_image_annotator_object['image'] = image_path
+                all_image_annotations[page_zero_index]["image"] = image_path
+            else:
+                print(f"No image path found for page {page}.")
 
-    # If image path is still not valid, load in a new image an overwrite it. Then replace all items in the image annotation object for all pages based on the updated information.
-    page_sizes_df = pd.DataFrame(page_sizes)
+        return page_image_annotator_object, all_image_annotations
 
-    if not os.path.exists(current_image_path):        
+def replace_placeholder_image_with_real_image(doc_full_file_name_textbox:str, current_image_path:str, page_sizes_df:pd.DataFrame, page_num_reported:int, input_folder:str):
+        ''' If image path is still not valid, load in a new image an overwrite it. Then replace all items in the image annotation object for all pages based on the updated information.'''
+        page_num_reported_zero_indexed = page_num_reported - 1
 
-        page_num, replaced_image_path, width, height = process_single_page_for_image_conversion(doc_full_file_name_textbox, page_num_reported_zero_indexed, input_folder=input_folder)
+        if not os.path.exists(current_image_path):        
 
-        # Overwrite page_sizes values 
-        page_sizes_df.loc[page_sizes_df['page']==page_num_reported, "image_width"] = width
-        page_sizes_df.loc[page_sizes_df['page']==page_num_reported, "image_height"] = height
-        page_sizes_df.loc[page_sizes_df['page']==page_num_reported, "image_path"] = replaced_image_path
-    
-    else:
-        if not page_sizes_df.loc[page_sizes_df['page']==page_num_reported, "image_width"].isnull().all():
-            width = page_sizes_df.loc[page_sizes_df['page']==page_num_reported, "image_width"].max()
-            height = page_sizes_df.loc[page_sizes_df['page']==page_num_reported, "image_height"].max()      
-        else:
-            image = Image.open(current_image_path)
-            width = image.width
-            height = image.height
+            page_num, replaced_image_path, width, height = process_single_page_for_image_conversion(doc_full_file_name_textbox, page_num_reported_zero_indexed, input_folder=input_folder)
 
+            # Overwrite page_sizes values 
             page_sizes_df.loc[page_sizes_df['page']==page_num_reported, "image_width"] = width
             page_sizes_df.loc[page_sizes_df['page']==page_num_reported, "image_height"] = height
+            page_sizes_df.loc[page_sizes_df['page']==page_num_reported, "image_path"] = replaced_image_path
+        
+        else:
+            if not page_sizes_df.loc[page_sizes_df['page']==page_num_reported, "image_width"].isnull().all():
+                width = page_sizes_df.loc[page_sizes_df['page']==page_num_reported, "image_width"].max()
+                height = page_sizes_df.loc[page_sizes_df['page']==page_num_reported, "image_height"].max()      
+            else:
+                image = Image.open(current_image_path)
+                width = image.width
+                height = image.height
 
-        page_sizes_df.loc[page_sizes_df['page']==page_num_reported, "image_path"] = current_image_path
+                page_sizes_df.loc[page_sizes_df['page']==page_num_reported, "image_width"] = width
+                page_sizes_df.loc[page_sizes_df['page']==page_num_reported, "image_height"] = height
 
-        replaced_image_path = current_image_path
+            page_sizes_df.loc[page_sizes_df['page']==page_num_reported, "image_path"] = current_image_path
 
-    if review_df.empty: review_df = pd.DataFrame(columns=["image", "page", "label", "color", "xmin", "ymin", "xmax", "ymax", "text", "id"])
-    review_df.loc[review_df["page"]==page_num_reported, 'image'] = replaced_image_path
+            replaced_image_path = current_image_path
+        
+        return replaced_image_path, page_sizes_df
 
-    # Update dropdowns and review selection dataframe with the updated annotator object
-    recogniser_entities_list, recogniser_dataframe_out_gr, recogniser_dataframe_modified, recogniser_entities_dropdown_value, text_entities_drop, page_entities_drop = update_recogniser_dataframes(all_image_annotations, recogniser_dataframe_base, recogniser_entities_dropdown_value, text_dropdown_value, page_dropdown_value, review_df.copy(), page_sizes)
-    
-    recogniser_colour_list = [(0, 0, 0) for _ in range(len(recogniser_entities_list))]
+def update_annotator_object_and_filter_df(
+    all_image_annotations:List[AnnotatedImageData],
+    gradio_annotator_current_page_number:int,
+    recogniser_entities_dropdown_value:str="ALL",
+    page_dropdown_value:str="ALL",
+    text_dropdown_value:str="ALL",
+    recogniser_dataframe_base:gr.Dataframe=None, # Simplified default
+    zoom:int=100,
+    review_df:pd.DataFrame=None, # Use None for default empty DataFrame
+    page_sizes:List[dict]=[],
+    doc_full_file_name_textbox:str='',
+    input_folder:str=INPUT_FOLDER
+) -> Tuple[image_annotator, gr.Number, gr.Number, int, str, gr.Dataframe, pd.DataFrame, List[str], List[str], List[dict], List[AnnotatedImageData]]:
+    '''
+    Update a gradio_image_annotation object with new annotation data for the current page
+    and update filter dataframes, optimizing by processing only the current page's data for display.
+    '''
+    zoom_str = str(zoom) + '%'
+
+    # Handle default empty review_df and recogniser_dataframe_base
+    if review_df is None or not isinstance(review_df, pd.DataFrame):
+         review_df = pd.DataFrame(columns=["image", "page", "label", "color", "xmin", "ymin", "xmax", "ymax", "text", "id"])
+    if recogniser_dataframe_base is None: # Create a simple default if None
+         recogniser_dataframe_base = gr.Dataframe(pd.DataFrame(data={"page":[], "label":[], "text":[], "id":[]}))
+
+
+    # Handle empty all_image_annotations state early
+    if not all_image_annotations:
+        print("No all_image_annotation object found")
+        # Return blank/default outputs
+        blank_annotator = gr.ImageAnnotator(
+            value = None, boxes_alpha=0.1, box_thickness=1, label_list=[], label_colors=[],
+            show_label=False, height=zoom_str, width=zoom_str, box_min_size=1,
+            box_selected_thickness=2, handle_size=4, sources=None,
+            show_clear_button=False, show_share_button=False, show_remove_button=False,
+            handles_cursor=True, interactive=True, use_default_label=True
+        )
+        blank_df_out_gr = gr.Dataframe(pd.DataFrame(columns=["page", "label", "text", "id"]))
+        blank_df_modified = pd.DataFrame(columns=["page", "label", "text", "id"])
 
-    # page_sizes_df has been changed - save back to page_sizes_object
-    page_sizes = page_sizes_df.to_dict(orient='records')
+        return (blank_annotator, gr.Number(value=1), gr.Number(value=1), 1,
+                recogniser_entities_dropdown_value, blank_df_out_gr, blank_df_modified,
+                [], [], [], []) # Return empty lists/defaults for other outputs
+
+    # Validate and bound the current page number (1-based logic)
+    page_num_reported = max(1, gradio_annotator_current_page_number) # Minimum page is 1
+    page_max_reported = len(all_image_annotations)
+    if page_num_reported > page_max_reported:
+        page_num_reported = page_max_reported
 
-    images_list = list(page_sizes_df["image_path"])
-    images_list[page_num_reported_zero_indexed] = replaced_image_path
+    page_num_reported_zero_indexed = page_num_reported - 1
+    annotate_previous_page = page_num_reported # Store the determined page number
 
-    all_image_annotations[page_num_reported_zero_indexed]['image'] = replaced_image_path
+    # --- Process page sizes DataFrame ---
+    page_sizes_df = pd.DataFrame(page_sizes)
+    if not page_sizes_df.empty:
+        page_sizes_df["page"] = pd.to_numeric(page_sizes_df["page"], errors="coerce")
+        page_sizes_df.dropna(subset=["page"], inplace=True)
+        if not page_sizes_df.empty:
+            page_sizes_df["page"] = page_sizes_df["page"].astype(int)
+        else:
+            print("Warning: Page sizes DataFrame became empty after processing.")
+
+    # --- Handle Image Path Replacement for the Current Page ---
+    # This modifies the specific page entry within all_image_annotations list
+    # Assuming replace_annotator_object_img_np_array_with_page_sizes_image_path
+    # correctly updates the image path within the list element.
+    if len(all_image_annotations) > page_num_reported_zero_indexed:
+        # Make a shallow copy of the list and deep copy the specific page dict before modification
+        # to avoid modifying the input list unexpectedly if it's used elsewhere.
+        # However, the original code modified the list in place, so we'll stick to that
+        # pattern but acknowledge it.
+        page_object_to_update = all_image_annotations[page_num_reported_zero_indexed]
+
+        # Use the helper function to replace the image path within the page object
+        # Note: This helper returns the potentially modified page_object and the full state.
+        # The full state return seems redundant if only page_object_to_update is modified.
+        # Let's call it and assume it correctly updates the item in the list.
+        updated_page_object, all_image_annotations_after_img_replace = replace_annotator_object_img_np_array_with_page_sizes_image_path(
+             all_image_annotations, page_object_to_update, page_sizes, page_num_reported)
+
+        # The original code immediately re-assigns all_image_annotations.
+        # We'll rely on the function modifying the list element in place or returning the updated list.
+        # Assuming it returns the updated list for robustness:
+        all_image_annotations = all_image_annotations_after_img_replace
+
+
+        # Now handle the actual image file path replacement using replace_placeholder_image_with_real_image
+        current_image_path = updated_page_object.get('image') # Get potentially updated image path
+
+        if current_image_path and not page_sizes_df.empty:
+            try:
+                replaced_image_path, page_sizes_df = replace_placeholder_image_with_real_image(
+                    doc_full_file_name_textbox, current_image_path, page_sizes_df,
+                    page_num_reported, input_folder=input_folder # Use 1-based page num
+                )
 
-    # Multiply out image_annotation coordinates from relative to absolute if necessary
-    all_image_annotations_df = convert_annotation_data_to_dataframe(all_image_annotations)
+                # Update the image path in the state and review_df for the current page
+                # Find the correct entry in all_image_annotations list again by index
+                if len(all_image_annotations) > page_num_reported_zero_indexed:
+                     all_image_annotations[page_num_reported_zero_indexed]['image'] = replaced_image_path
 
-    all_image_annotations_df = multiply_coordinates_by_page_sizes(all_image_annotations_df, page_sizes_df, xmin="xmin", xmax="xmax", ymin="ymin", ymax="ymax")   
+                # Update review_df's image path for this page
+                if 'page' in review_df.columns and 'image' in review_df.columns:
+                     # Ensure review_df page column is numeric for filtering
+                     review_df['page'] = pd.to_numeric(review_df['page'], errors='coerce').fillna(-1).astype(int)
+                     review_df.loc[review_df["page"]==page_num_reported, 'image'] = replaced_image_path
 
-    #print("all_image_annotations_df[-1] just before creating annotation dicts:", all_image_annotations_df.iloc[-1, :])
 
-    all_image_annotations = create_annotation_dicts_from_annotation_df(all_image_annotations_df, page_sizes)
+            except Exception as e:
+                 print(f"Error during image path replacement for page {page_num_reported}: {e}")
+    else:
+         print(f"Warning: Page index {page_num_reported_zero_indexed} out of bounds for all_image_annotations list.")
 
-    #print("all_image_annotations[-1] after creating annotation dicts:", all_image_annotations[-1])
 
+    # Save back page_sizes_df to page_sizes list format
+    if not page_sizes_df.empty:
+        page_sizes = page_sizes_df.to_dict(orient='records')
+    else:
+        page_sizes = [] # Ensure page_sizes is a list if df is empty
+
+    # --- OPTIMIZATION: Prepare data *only* for the current page for display ---
+    current_page_image_annotator_object = None
+    if len(all_image_annotations) > page_num_reported_zero_indexed:
+        page_data_for_display = all_image_annotations[page_num_reported_zero_indexed]
+
+        # Convert current page annotations list to DataFrame for coordinate multiplication IF needed
+        # Assuming coordinate multiplication IS needed for display if state stores relative coords
+        current_page_annotations_df = convert_annotation_data_to_dataframe([page_data_for_display])
+
+
+        if not current_page_annotations_df.empty and not page_sizes_df.empty:
+             # Multiply coordinates *only* for this page's DataFrame
+             try:
+                 # Need the specific page's size for multiplication
+                 page_size_row = page_sizes_df[page_sizes_df['page'] == page_num_reported]
+                 if not page_size_row.empty:
+                      current_page_annotations_df = multiply_coordinates_by_page_sizes(
+                          current_page_annotations_df, page_size_row, # Pass only the row for the current page
+                          xmin="xmin", xmax="xmax", ymin="ymin", ymax="ymax"
+                      )
+             
+             except Exception as e:
+                  print(f"Warning: Error during coordinate multiplication for page {page_num_reported}: {e}. Using original coordinates.")
+                  # If error, proceed with original coordinates or handle as needed
+
+        if "color" not in current_page_annotations_df.columns:
+            current_page_annotations_df['color'] = '(0, 0, 0)'
+
+        # Convert the processed DataFrame back to the list of dicts format for the annotator
+        processed_current_page_annotations_list = current_page_annotations_df[["xmin", "xmax", "ymin", "ymax", "label", "color", "text", "id"]].to_dict(orient='records')
+
+        # Construct the final object expected by the Gradio ImageAnnotator value parameter
+        current_page_image_annotator_object: AnnotatedImageData = {
+            'image': page_data_for_display.get('image'), # Use the (potentially updated) image path
+            'boxes': processed_current_page_annotations_list
+        }
 
+    # --- Update Dropdowns and Review DataFrame ---
+    # This external function still operates on potentially large DataFrames.
+    # It receives all_image_annotations and a copy of review_df.
+    try:
+        recogniser_entities_list, recogniser_dataframe_out_gr, recogniser_dataframe_modified, recogniser_entities_dropdown_value, text_entities_drop, page_entities_drop = update_recogniser_dataframes(
+             all_image_annotations, # Pass the updated full state
+             recogniser_dataframe_base,
+             recogniser_entities_dropdown_value,
+             text_dropdown_value,
+             page_dropdown_value,
+             review_df.copy(), # Keep the copy as per original function call
+             page_sizes # Pass updated page sizes
+        )
+        # Generate default black colors for labels if needed by image_annotator
+        recogniser_colour_list = [(0, 0, 0) for _ in range(len(recogniser_entities_list))]
 
-    # Remove blank duplicate entries
-    all_image_annotations = remove_duplicate_images_with_blank_boxes(all_image_annotations)
+    except Exception as e:
+        print(f"Error calling update_recogniser_dataframes: {e}. Returning empty/default filter data.")
+        recogniser_entities_list = []
+        recogniser_colour_list = []
+        recogniser_dataframe_out_gr = gr.Dataframe(pd.DataFrame(columns=["page", "label", "text", "id"]))
+        recogniser_dataframe_modified = pd.DataFrame(columns=["page", "label", "text", "id"])
+        text_entities_drop = []
+        page_entities_drop = []
 
-    current_page_image_annotator_object = all_image_annotations[page_num_reported_zero_indexed]
 
-    #print("current_page_image_annotator_object that goes into annotator object:", current_page_image_annotator_object)
+    # --- Final Output Components ---
+    page_number_reported_gradio_comp = gr.Number(label = "Current page", value=page_num_reported, precision=0)
 
-    page_number_reported_gradio = gr.Number(label = "Current page", value=page_num_reported, precision=0)
     
-    ###
-    # If no data, present a blank page
-    if not all_image_annotations:
-        print("No all_image_annotation object found")
-        page_num_reported = 1
 
-        out_image_annotator = image_annotator(
-        value = None,
-        boxes_alpha=0.1,
-        box_thickness=1,
-        label_list=recogniser_entities_list,
-        label_colors=recogniser_colour_list,
-        show_label=False,
-        height=zoom_str,
-        width=zoom_str,
-        box_min_size=1,
-        box_selected_thickness=2,
-        handle_size=4,
-        sources=None,#["upload"],
-        show_clear_button=False,
-        show_share_button=False,
-        show_remove_button=False,
-        handles_cursor=True,
-        interactive=True,
-        use_default_label=True
-        )        
-
-        return out_image_annotator, page_number_reported_gradio, page_number_reported_gradio, page_num_reported, recogniser_entities_dropdown_value, recogniser_dataframe_out_gr, recogniser_dataframe_modified, text_entities_drop, page_entities_drop, page_sizes, all_image_annotations
-    
+    ### Present image_annotator outputs
+    # Handle the case where current_page_image_annotator_object couldn't be prepared
+    if current_page_image_annotator_object is None:
+        # This should ideally be covered by the initial empty check for all_image_annotations,
+        # but as a safeguard:
+        print("Warning: Could not prepare annotator object for the current page.")
+        out_image_annotator = image_annotator(value=None, interactive=False) # Present blank/non-interactive
     else:
-        ### Present image_annotator outputs
         out_image_annotator = image_annotator(
             value = current_page_image_annotator_object,
             boxes_alpha=0.1,
             box_thickness=1,
-            label_list=recogniser_entities_list,
+            label_list=recogniser_entities_list, # Use labels from update_recogniser_dataframes
             label_colors=recogniser_colour_list,
             show_label=False,
             height=zoom_str,
@@ -408,41 +680,23 @@ def update_annotator_object_and_filter_df(
             show_share_button=False,
             show_remove_button=False,
             handles_cursor=True,
-            interactive=True
+            interactive=True # Keep interactive if data is present
         )
 
-    #print("all_image_annotations at end of update_annotator...:", all_image_annotations)
-    #print("review_df at end of update_annotator_object:", review_df)
-
-    return out_image_annotator, page_number_reported_gradio, page_number_reported_gradio, page_num_reported, recogniser_entities_dropdown_value, recogniser_dataframe_out_gr, recogniser_dataframe_modified, text_entities_drop, page_entities_drop, page_sizes, all_image_annotations
-
-def replace_images_in_image_annotation_object(
-        all_image_annotations:List[dict],
-        page_image_annotator_object:AnnotatedImageData,
-        page_sizes:List[dict],
-        page:int):
-
-        '''
-        Check if the image value in an AnnotatedImageData dict is a placeholder or np.array. If either of these, replace the value with the file path of the image that is hopefully already loaded into the app related to this page.
-        '''
-
-        page_zero_index = page - 1
-        
-        if isinstance(all_image_annotations[page_zero_index]["image"], np.ndarray) or "placeholder_image" in all_image_annotations[page_zero_index]["image"] or isinstance(page_image_annotator_object['image'], np.ndarray):
-            page_sizes_df = pd.DataFrame(page_sizes)
-            page_sizes_df[["page"]] = page_sizes_df[["page"]].apply(pd.to_numeric, errors="coerce")
-
-            # Check for matching pages
-            matching_paths = page_sizes_df.loc[page_sizes_df['page'] == page, "image_path"].unique()
-
-            if matching_paths.size > 0:
-                image_path = matching_paths[0]
-                page_image_annotator_object['image'] = image_path
-                all_image_annotations[page_zero_index]["image"] = image_path
-            else:
-                print(f"No image path found for page {page}.")
-
-        return page_image_annotator_object, all_image_annotations
+    # The original code returned page_number_reported_gradio twice;
+    # returning the Gradio component and the plain integer value.
+    # Let's match the output signature.
+    return (out_image_annotator,
+            page_number_reported_gradio_comp,
+            page_number_reported_gradio_comp, # Redundant, but matches original return signature
+            page_num_reported, # Plain integer value
+            recogniser_entities_dropdown_value,
+            recogniser_dataframe_out_gr,
+            recogniser_dataframe_modified,
+            text_entities_drop, # List of text entities for dropdown
+            page_entities_drop, # List of page numbers for dropdown
+            page_sizes, # Updated page_sizes list
+            all_image_annotations) # Return the updated full state
 
 def update_all_page_annotation_object_based_on_previous_page(
                                     page_image_annotator_object:AnnotatedImageData,
@@ -459,12 +713,9 @@ def update_all_page_annotation_object_based_on_previous_page(
     previous_page_zero_index = previous_page -1
  
     if not current_page: current_page = 1
-
-    #print("page_image_annotator_object at start of update_all_page_annotation_object:", page_image_annotator_object)
-      
-    page_image_annotator_object, all_image_annotations = replace_images_in_image_annotation_object(all_image_annotations, page_image_annotator_object, page_sizes, previous_page)
-
-    #print("page_image_annotator_object after replace_images in update_all_page_annotation_object:", page_image_annotator_object)
+    
+    # This replaces the numpy array image object with the image file path
+    page_image_annotator_object, all_image_annotations = replace_annotator_object_img_np_array_with_page_sizes_image_path(all_image_annotations, page_image_annotator_object, page_sizes, previous_page)
 
     if clear_all == False: all_image_annotations[previous_page_zero_index] = page_image_annotator_object
     else: all_image_annotations[previous_page_zero_index]["boxes"] = []
@@ -493,7 +744,7 @@ def apply_redactions_to_review_df_and_files(page_image_annotator_object:Annotate
     page_image_annotator_object = all_image_annotations[current_page - 1]   
 
     # This replaces the numpy array image object with the image file path
-    page_image_annotator_object, all_image_annotations = replace_images_in_image_annotation_object(all_image_annotations, page_image_annotator_object, page_sizes, current_page)
+    page_image_annotator_object, all_image_annotations = replace_annotator_object_img_np_array_with_page_sizes_image_path(all_image_annotations, page_image_annotator_object, page_sizes, current_page)
     page_image_annotator_object['image'] = all_image_annotations[current_page - 1]["image"]
 
     if not page_image_annotator_object:
@@ -529,7 +780,7 @@ def apply_redactions_to_review_df_and_files(page_image_annotator_object:Annotate
                         # Check if all elements are integers in the range 0-255
                         if all(isinstance(c, int) and 0 <= c <= 255 for c in fill):
                             pass
-                            #print("fill:", fill)
+
                         else:
                             print(f"Invalid color values: {fill}. Defaulting to black.")
                             fill = (0, 0, 0)  # Default to black if invalid
@@ -553,7 +804,6 @@ def apply_redactions_to_review_df_and_files(page_image_annotator_object:Annotate
                 doc = [image]
 
             elif file_extension in '.csv':
-                #print("This is a csv")
                 pdf_doc = []
 
             # If working with pdfs
@@ -797,11 +1047,9 @@ def df_select_callback(df: pd.DataFrame, evt: gr.SelectData):
 
         row_value_df = pd.DataFrame(data={"page":[row_value_page], "label":[row_value_label], "text":[row_value_text], "id":[row_value_id]})
 
-        return row_value_page, row_value_df
+        return row_value_df
 
 def df_select_callback_textract_api(df: pd.DataFrame, evt: gr.SelectData):
-        
-        #print("evt.data:", evt._data)
 
         row_value_job_id = evt.row_value[0] # This is the page number value
         # row_value_label = evt.row_value[1] # This is the label number value
@@ -829,59 +1077,108 @@ def df_select_callback_ocr(df: pd.DataFrame, evt: gr.SelectData):
 
         return row_value_page, row_value_df
 
-def update_selected_review_df_row_colour(redaction_row_selection:pd.DataFrame, review_df:pd.DataFrame, previous_id:str="", previous_colour:str='(0, 0, 0)', page_sizes:List[dict]=[], output_folder:str=OUTPUT_FOLDER, colour:str='(1, 0, 255)'):
+def update_selected_review_df_row_colour(
+    redaction_row_selection: pd.DataFrame,
+    review_df: pd.DataFrame,
+    previous_id: str = "",
+    previous_colour: str = '(0, 0, 0)',
+    colour: str = '(1, 0, 255)'
+) -> tuple[pd.DataFrame, str, str]:
     '''
     Update the colour of a single redaction box based on the values in a selection row
+    (Optimized Version)
     '''
-    colour_tuple =  str(tuple(colour))
 
-    if "color" not in review_df.columns: review_df["color"] = '(0, 0, 0)'
+    # Ensure 'color' column exists, default to previous_colour if previous_id is provided
+    if "color" not in review_df.columns:
+        review_df["color"] = previous_colour if previous_id else '(0, 0, 0)'
+
+    # Ensure 'id' column exists
     if "id" not in review_df.columns:
-        review_df = fill_missing_ids(review_df)
+         # Assuming fill_missing_ids is a defined function that returns a DataFrame
+         # It's more efficient if this is handled outside if possible,
+         # or optimized internally.
+         print("Warning: 'id' column not found. Calling fill_missing_ids.")
+         review_df = fill_missing_ids(review_df) # Keep this if necessary, but note it can be slow
+
+    # --- Optimization 1 & 2: Reset existing highlight colours using vectorized assignment ---
+    # Reset the color of the previously highlighted row
+    if previous_id and previous_id in review_df["id"].values:
+         review_df.loc[review_df["id"] == previous_id, "color"] = previous_colour
+
+    # Reset the color of any row that currently has the highlight colour (handle cases where previous_id might not have been tracked correctly)
+    # Convert to string for comparison only if the dtype might be mixed or not purely string
+    # If 'color' is consistently string, the .astype(str) might be avoidable.
+    # Assuming color is consistently string format like '(R, G, B)'
+    review_df.loc[review_df["color"] == colour, "color"] = '(0, 0, 0)'
 
-    # Reset existing highlight colours
-    review_df.loc[review_df["id"]==previous_id, "color"] = review_df.loc[review_df["id"]==previous_id, "color"].apply(lambda _: previous_colour)
-    review_df.loc[review_df["color"].astype(str)==colour, "color"] = review_df.loc[review_df["color"].astype(str)==colour, "color"].apply(lambda _: '(0, 0, 0)')
 
     if not redaction_row_selection.empty and not review_df.empty:
         use_id = (
-            "id" in redaction_row_selection.columns 
-            and "id" in review_df.columns 
-            and not redaction_row_selection["id"].isnull().all() 
+            "id" in redaction_row_selection.columns
+            and "id" in review_df.columns
+            and not redaction_row_selection["id"].isnull().all()
             and not review_df["id"].isnull().all()
         )
 
-        selected_merge_cols = ["id"] if use_id else ["label", "page", "text"]        
+        selected_merge_cols = ["id"] if use_id else ["label", "page", "text"]
 
-        review_df = review_df.merge(redaction_row_selection[selected_merge_cols], on=selected_merge_cols, indicator=True, how="left")
+        # --- Optimization 3: Use inner merge directly ---
+        # Merge to find rows in review_df that match redaction_row_selection
+        merged_reviews = review_df.merge(
+            redaction_row_selection[selected_merge_cols],
+            on=selected_merge_cols,
+            how="inner" # Use inner join as we only care about matches
+        )
 
-    if "_merge" in review_df.columns:
-        filtered_reviews = review_df.loc[review_df["_merge"]=="both"]
-    else:
-        filtered_reviews = pd.DataFrame()
+        if not merged_reviews.empty:
+             # Assuming we only expect one match for highlighting a single row
+             # If multiple matches are possible and you want to highlight all,
+             # the logic for previous_id and previous_colour needs adjustment.
+            new_previous_colour = str(merged_reviews["color"].iloc[0])
+            new_previous_id = merged_reviews["id"].iloc[0]
+
+            # --- Optimization 1 & 2: Update color of the matched row using vectorized assignment ---
+
+            if use_id:
+                 # Faster update if using unique 'id' as merge key
+                 review_df.loc[review_df["id"].isin(merged_reviews["id"]), "color"] = colour
+            else:
+                 # More general case using multiple columns - might be slower
+                 # Create a temporary key for comparison
+                 def create_merge_key(df, cols):
+                     return df[cols].astype(str).agg('_'.join, axis=1)
+
+                 review_df_key = create_merge_key(review_df, selected_merge_cols)
+                 merged_reviews_key = create_merge_key(merged_reviews, selected_merge_cols)
+
+                 review_df.loc[review_df_key.isin(merged_reviews_key), "color"] = colour
+
+            previous_colour = new_previous_colour
+            previous_id = new_previous_id
+        else:
+             # No rows matched the selection
+             print("No reviews found matching selection criteria")
+             # The reset logic at the beginning already handles setting color to (0, 0, 0)
+             # if it was the highlight colour and didn't match.
+             # No specific action needed here for color reset beyond what's done initially.
+             previous_colour = '(0, 0, 0)' # Reset previous_colour as no row was highlighted
+             previous_id = '' # Reset previous_id
 
-    if not filtered_reviews.empty:
-        previous_colour = str(filtered_reviews["color"].values[0])
-        previous_id = filtered_reviews["id"].values[0]
-        review_df.loc[review_df["_merge"]=="both", "color"] =  review_df.loc[review_df["_merge"] == "both", "color"].apply(lambda _: colour)
     else:
-        # Handle the case where no rows match the condition
-        print("No reviews found with _merge == 'both'")
-        previous_colour = '(0, 0, 0)'
-        review_df.loc[review_df["color"]==colour, "color"] = previous_colour
-        previous_id =''
+         # If selection is empty, reset any existing highlights
+         review_df.loc[review_df["color"] == colour, "color"] = '(0, 0, 0)'
+         previous_colour = '(0, 0, 0)'
+         previous_id = ''
 
-    review_df.drop("_merge", axis=1, inplace=True)
 
-    # Ensure that all output coordinates are in proportional size
-    #page_sizes_df = pd.DataFrame(page_sizes)
-    #page_sizes_df .loc[:, "page"] = pd.to_numeric(page_sizes_df["page"], errors="coerce")
-    #print("review_df before divide:", review_df)
-    #print("page_sizes_df before divide:", page_sizes_df)
-    #review_df = divide_coordinates_by_page_sizes(review_df, page_sizes_df)
-    #print("review_df after divide:", review_df)
+    # Ensure column order is maintained if necessary, though pandas generally preserves order
+    # Creating a new DataFrame here might involve copying data, consider if this is strictly needed.
+    if set(["image", "page", "label", "color", "xmin","ymin", "xmax", "ymax", "text", "id"]).issubset(review_df.columns):
+        review_df = review_df[["image", "page", "label", "color", "xmin","ymin", "xmax", "ymax", "text", "id"]]
+    else:
+         print("Warning: Not all expected columns are present in review_df for reordering.")
 
-    review_df = review_df[["image", "page", "label", "color", "xmin","ymin", "xmax", "ymax", "text", "id"]]
 
     return review_df, previous_id, previous_colour
 
@@ -988,8 +1285,6 @@ def create_xfdf(review_file_df:pd.DataFrame, pdf_path:str, pymupdf_doc:object, i
         page_sizes_df = pd.DataFrame(page_sizes)
 
         # If there are no image coordinates, then convert coordinates to pymupdf coordinates prior to export
-        #print("Using pymupdf coordinates for conversion.")
-
         pages_are_images = False
 
         if "mediabox_width" not in review_file_df.columns:            
@@ -1041,33 +1336,9 @@ def create_xfdf(review_file_df:pd.DataFrame, pdf_path:str, pymupdf_doc:object, i
                 raise ValueError(f"Invalid cropbox format: {document_cropboxes[page_python_format]}")
         else:
             print("Document cropboxes not found.")
-
         
         pdf_page_height = pymupdf_page.mediabox.height
-        pdf_page_width = pymupdf_page.mediabox.width
-
-        # Check if image dimensions for page exist in page_sizes_df
-        # image_dimensions = {}
-
-        # image_dimensions['image_width'] = page_sizes_df.loc[page_sizes_df['page']==page_num_reported, "image_width"].max()
-        # image_dimensions['image_height'] = page_sizes_df.loc[page_sizes_df['page']==page_num_reported, "image_height"].max()
-
-        # if pd.isna(image_dimensions['image_width']):
-        #     image_dimensions = {}
-
-        # image = image_paths[page_python_format]
-
-        # if image_dimensions:
-        #     image_page_width, image_page_height = image_dimensions["image_width"], image_dimensions["image_height"]
-        # if isinstance(image, str) and 'placeholder' not in image:
-        #     image = Image.open(image)
-        #     image_page_width, image_page_height = image.size
-        # else:
-        #     try:
-        #         image = Image.open(image)
-        #         image_page_width, image_page_height = image.size
-        #     except Exception as e:
-        #         print("Could not get image sizes due to:", e)        
+        pdf_page_width = pymupdf_page.mediabox.width     
 
         # Create redaction annotation
         redact_annot = SubElement(annots, 'redact')
@@ -1345,8 +1616,6 @@ def convert_xfdf_to_dataframe(file_paths_list:List[str], pymupdf_doc, image_path
                 # Optionally, you can add the image path or other relevant information
                 df.loc[_, 'image'] = image_path
 
-                #print('row:', row)
-
     out_file_path = output_folder + file_path_name + "_review_file.csv"
     df.to_csv(out_file_path, index=None)
 

tools/textract_batch_call.py

@@ -10,7 +10,7 @@ from io import StringIO
 from urllib.parse import urlparse
 from botocore.exceptions import ClientError, NoCredentialsError, PartialCredentialsError, TokenRetrievalError
 
-from tools.config import TEXTRACT_BULK_ANALYSIS_BUCKET, OUTPUT_FOLDER, AWS_REGION, DOCUMENT_REDACTION_BUCKET, LOAD_PREVIOUS_TEXTRACT_JOBS_S3, TEXTRACT_JOBS_S3_LOC, TEXTRACT_JOBS_LOCAL_LOC
+from tools.config import TEXTRACT_WHOLE_DOCUMENT_ANALYSIS_BUCKET, OUTPUT_FOLDER, AWS_REGION, DOCUMENT_REDACTION_BUCKET, LOAD_PREVIOUS_TEXTRACT_JOBS_S3, TEXTRACT_JOBS_S3_LOC, TEXTRACT_JOBS_LOCAL_LOC
 #from tools.aws_textract import json_to_ocrresult
 
 def analyse_document_with_textract_api(
@@ -18,7 +18,7 @@ def analyse_document_with_textract_api(
     s3_input_prefix: str,
     s3_output_prefix: str,
     job_df:pd.DataFrame,
-    s3_bucket_name: str = TEXTRACT_BULK_ANALYSIS_BUCKET,
+    s3_bucket_name: str = TEXTRACT_WHOLE_DOCUMENT_ANALYSIS_BUCKET,
     local_output_dir: str = OUTPUT_FOLDER,    
     analyse_signatures:List[str] = [],
     successful_job_number:int=0,
@@ -328,7 +328,7 @@ def poll_bulk_textract_analysis_progress_and_download(
     s3_output_prefix: str,
     pdf_filename:str,
     job_df:pd.DataFrame,
-    s3_bucket_name: str = TEXTRACT_BULK_ANALYSIS_BUCKET,
+    s3_bucket_name: str = TEXTRACT_WHOLE_DOCUMENT_ANALYSIS_BUCKET,
     local_output_dir: str = OUTPUT_FOLDER,
     load_s3_jobs_loc:str=TEXTRACT_JOBS_S3_LOC,
     load_local_jobs_loc:str=TEXTRACT_JOBS_LOCAL_LOC,