need to read a million files for a timestamp and move the files in its folder based on the timestamp... which method is fastest?
I feel the first approach could be faster. Although, it heavily depends on what sort of data you are dealing with.
Why do you call it a "longer task"? The time taken for a single unit would always be either equal to or lesser that your existing approach (due to the overhead of waiting for other threads to write to the CSV file in a queue).
Trying to decide which method would be faster for this. It's a permanent reusable script.
If I have a million folders on a server and each folder is considered one "unit". All the files in that folder are part of it. One file contains the details of the unit. In that file is a timestamp. If that timestamp is less than x. It needs to be copied, otherwise we leave it.
Should I feed the list of paths into a multi-process function where each process would read a file off the list, check the timestamp and if good start the copy in that same process... then join the task when done, log the folder name in a text/pickle file as done and repeat til all are finished. Resuming would happen by eliminating that folder name from the new list generation on rerun. List is generated each run in the main thread.
OR would it be quicker to kick off x amount of threads to process the files and read their data into a csv or set of csvs. I would handle the write-to csv's in batches in a proper queue. one column would be a True or False whether to copy that unit. Once the list is complete, read the valid paths from the CSV where copy was set to True and then kick off new threads to start copying that path and filling in a completed column next to that row when it's done.
I've used the latter method with pandas several times and have had great success with it. I prefer it because it simultaneously builds the final status report while also letting me resume without having to regenerate the list. the first lets us resume as well but we have to regenerate the list and still have to check each file for those not processed yet.
I feel like the CSV method allows for less chance of failure for whatever reason as it's just reading the file, doing the check, writing the result to a csv, moving on. It's not tying up the thread on a long copy, nor is it writing to the file its reading. Max rows I've dealt with per server was around 300k and the csv for that is quite small in file size. I think the largest I had was about 50-100mb and it read it completely into a list in less than a min. I haven't used Dask but maybe that can even speed things up more.
Before I commit to one way or another, I wanted to ask for some feedback.
Is it better to have long running threads open doing longer tasks for longer time. Or better to break the job up into say two halves and have threads open to process quicker tasks on each side?
thanks.