How to merge back the processes by sample after splitting by chromosome in Nextflow? I have a pipeline that has an input list of samples. I would like to split those samples by chromosomes, process each chromosome apart, and then merge them back by sample name.
Here's an example:
samplesList = Channel.from([&quot;sample1&quot;, &quot;sample2&quot;])

// A process that splits the samples by chromosome as an example
// In my actual case, I have a VCF that is split by `snpSift Split` command
// and generates files like [sample]_[chr].vcf
// In the example below, I only gave sample1
// But take into consideration that I have multiple samples

process bar {
   echo true

  output:
    file(&quot;*.tsv&quot;) into bar_output

  script:
  &quot;&quot;&quot;
  echo 1 &gt; sample1_chr1.tsv
  echo 2 &gt; sample1_chr2.tsv
  echo 3 &gt; sample1_chr3.tsv
  echo 4 &gt; sample1_chr4.tsv
  echo 5 &gt; sample1_chr5.tsv
  &quot;&quot;&quot;
}

// Process each chromosome apart for each sample
// In my case, I just work some magic
// On the file and then output another file
// This means the output of the process foo
// is only ONE chromosome per sample
process foo {
  input:
    each x from bar_output  // to treat each file separately from the channel
  output:
    file(&quot;*.tsv&quot;) into xasxa
  script:
    &quot;&quot;&quot;
    echo \\\$(cat $x)\\\$(cat $x) &gt; sample1_chr\\\$(cat $x)_2.tsv
    &quot;&quot;&quot;
}

// What is the next process to group back them together by sample name?
// I do not wish to do a collect() on channel foo because it will force 
// all the processes of foo to finish first.
// In this example, foo would emit 5 different channels of chromosome
// for sample1 (imagine also there are a bunch of other samples included)
// This process should merge sample1 only when all chromosomes are processed
// independently of other samples. 
process mergeChromosomesBySamples {
   input:
     ???
}

As the example above, the process mergeChromosomesBySamples should return something similar/close to
[&quot;sample1&quot;, [&quot;/path/to/workdir/sample1_chr1.tsv&quot;,
             &quot;/path/to/workdir/sample1_chr2.tsv&quot;,
             &quot;/path/to/workdir/sample1_chr3.tsv&quot;,
             &quot;/path/to/workdir/sample1_chr4.tsv&quot;,
             &quot;/path/to/workdir/sample1_chr5.tsv&quot;]
]


EDIT1: more explication for the process mergeChromosomesBySamples given in the comments and as follows ...
The channel xasxa processes each chromosome apart, this means that the output of the channel xasxa contains only one chromosome of the sample like &quot;/path/to/workdir/sample1_chr1.tsv&quot; only. My goal is then to merge the processes out xasxa for each sample together. Sorry if that wasn't clear in my explanation.
With regards .collect() on xasxa I do not wish to use collect() in this case because it would have to force all the processes of foo to finish before. I want them to work independently for each sample.
Where each sample has his own VCF generated and I process them independantly.
EDIT2: Here's a graph of what I am trying to achieve:

If I use .collect(), then all the processing on the chromosomes of all the samples have to finish first before being able to merge them, which is something I do not want.
Thank you for your help.
Use the groupBy operator. If the content of channel xasxa is:
[&quot;/path/to/workdir/sample1_chr1.tsv&quot;, &quot;/path/to/workdir/sample1_chr2.tsv&quot;, &quot;/path/to/workdir/sample2_chr1.tsv&quot;, &quot;/path/to/workdir/sample2_chr2.tsv
You could probably do:
xasxa.groupBy{ it -&gt; it.fileName.toString().split(&quot;_&quot;)[0] }
The result would be a channel with:
[sample1:[/path/to/workdir/sample1_chr1.tsv, /path/to/workdir/sample1_chr2.tsv, sample2:[/path/to/workdir/sample2_chr1.tsv, /path/to/workdir/sample2_chr1.tsv]]
This breaks if you have an underscore in your samplename before _chr*. More robust would be to initialise all channels with a sample value, so that each channel is in the form of [samplename, chromosome]. Then all you need is a groupTuple on your channel.