I am running many tasks on a Linux cluster. Each task creates many output files. When all tasks are finished, I run something like tar cf foo.tar output_files/to create a tar archive. This is a very slow process since there are many thousands of files and directories.Is there any way to do this in parallel as the output files are being created?Is it possible to have multiple tar processes, spread across multiple machines, all adding their files to the same archive at once?The cluster has a shared filesystem. I am not interested in compression since it slows things down even more, and because all of the input files are themselves already compressed. Ideally the output would be a tar file, but I would consider other archive formats as well.
Multiple tar processes writing to the same archive file at once
linux;tar;parallel;cluster