Track the existence of process
There is nothing I can test here, so I will only make a couple superficial remarks.
You do not offer a lot of insight into your application. If you say you have a problem with overlapping queries, then you should address the root of the problem. I have the impression that you simply devised a workaround. It may do the job but at the expense of performance or reliability. It could also create more problems.
Repetition
What is immediately apparent is that you have a hardcoded path repeated multiple times across your code.
At the top of your code you should define a variable for '/root/progress_data'. Then if you decide to change the file name or path you will have only one line to change.
But Python has a library for temp files: https://docs.python.org/3/library/tempfile.html. This is what you should be using.
The date format &quot;%Y-%m-%d&quot; could be made a variable too.
Security
Does your program really need to run as root ?
Maximize your try/catch/finally blocks
You repeat this twice:
os.remove(f'/root/progress_data/{progress_hash}')

To ensure that the file gets removed at the end of the process you could place this line of code in a finally block.
Same goes for the return value:
return json_response({
    &quot;success&quot;: False,
    'result': &quot;Error while generating report&quot;,
})

return json_response({
    &quot;success&quot;: True,
    'data': report,
})

Keep only one return but use variables for the response depending on the outcome of the execution.
Thus your code will be shorter, more straightforward and easier to comprehend. But I still think the idea is wrong.
I would probably use a database instead of temp files. It can be a small self-contained DB file like SQLite. You need a simple table with details of the job, some kind of unique identifier and a column to update the status of the job. Then your API can query the table and say &quot;ready&quot; when the column has been to updated to the finished status.
This looks clumsy to me. I would ditch the idea and go back to the drawing board. If you are stuck my suggestion would be to post on Stack Overflow for guidance but add more context to your question, explain how the API is used, what your constraints are.