I have 384 pairs of files whose names differ by a single character. These represent fastq files from lane 1 and lane 2 of a sequencer. Before I can analyze these, I need to concatenate each pair.AA01_S1_L001_R1_001.fastq.gz and AA01_S1_L002_R1_001.fastq.gz all the way through DH12_S384_L002_R1_001.fastq.gz. I would like to append L002 to L001 and create 384 new files in sibling directory to be used for analysis. I tried to do this in R, but packages I've used were out of date. I'm thinking this is trivial in Unix, but my knowledge of unix is quite noobish. Edit to address questions:I think these are just gzip files without an index. See output below. I don't think I need any kind of index. Lastly, yes, the only difference between pairs are the L001 vs. L002 and the output names actually should have the name of the L001 ( will pipe output to sibling directory). gzip -cd AA01_S1_L001_R1_001.fastq.gz | head@D00572:249:HT7WJBCXX:1:1106:8558:1978 1:N:0:AAGACTCTT+AAGGTGTTTCAGCACCCATGACCGTGCCATTGCTCTCCTTGTCAAAGACACGCAGGCCC+DDDDDIIIIIIHIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIHIIIII@D00572:249:HT7WJBCXX:1:1106:12310:1984 1:N:0:AAGACTCTT+AAGGTGTTTCCAAAGTCTCAGGACCAACAGGAATTTTGATTGGTGCACCAGAATCCAGT+DDDDDHIIIIIIIIIIIIIIIIHHIIIIIIIIIIIIIHIIIIIIIIIIIH@D00572:249:HT7WJBCXX:1:1106:12936:1969 1:N:0:AAGACTCTT+AAGGTGTTTNAAAGGACCTTGATCTCTTCCTCATATCTGTCTTCCTTCTGCGAGTACTT
Loop through 384 pairs and create 384 new concatenated files
files
Files compressed with gzip may simply be concatenated together (while still compressed), so there's no need to first uncompress the files. When you uncompress the compressed concatenated file, you'll get the concatenated uncompressed data. This is also true for at least xz.For the concatenation, we may loop over all the L001 files, create the filename of the corresponding pair, and concatenate the two to a file in a new directory:mkdir output_dirfor name in *_L001_R1_001.fastq.gz; do    other=${name/_L001/_L002}    cat $name $other >output_dir/$namedoneHere, ${name/_L001/_L002} will replace the first occurrence of the string _L001 with _L002 in the value of $name, creating the corresponding file pair. This other filename is stored in the $other variable.The two files are concatenated with cat and the result is written to a new file in the output_dir directory. The original files are left in place.This assumes a shell like bash or ksh93 that understands ${parameter/pattern/string}.