Run Nextflow with file dependencies inside a Docker container? I’m trying to run a Nextflow workflow in a custom Docker container. Without Docker, the workflow succeeds. But running it inside the container leads to an error because a dependent file cannot be found.

Here’s a minimal script to illustrate the issue (similar to the basic Nextflow script example):

params.reference = "/data/reference/GRCh38.fa"

reference = file(params.reference)
fastq_channel = Channel.fromFilePairs('/data/fastq/*_R{1,2}.fastq.gz')

process map_reads {
    input:
    set sample, '?.fastq.gz' from fastq_channel

    output:
    file("${sample}.sam") into sam_channel

    script:
    """
    echo 'bwa mem -Y "$reference" 1.fastq.gz 2.fastq.gz &gt;${sample}.sam'
    stat -t "${reference}.bwt"
    """
}


Running this inside a container will cause the path of the query, /some/fastq/, to be mounted inside the container. However, the path to the database, /data/reference, will not be mounted. Consequently, I get the following execution error:


  stat: cannot stat 'GRCh38-chr21.fa.bwt': No such file or directory


Without the container, the command succeeds.



I understand why the error occurs: Nextflow doesn’t track the dependency on $reference inside the command, and thus doesn’t mount its path in the container.

My question is, how do I fix this? The documentation is silent on this regard, and none of the workflows I can find online address the problem. For example, the GATK4 IARC workflows would presumably have the same issue, and aren’t configured for use with Docker.

Just adding the reference as a process input is insufficient, since BWA requires various other index files. Am I supposed to add these all manually as inputs?
The following workaround seems to work but ideally I don’t want to have to change the script portion of my process so that it works with Docker.

params.ref = 'data/reference/GRCh38.fa'

ref = file(params.ref)
ref_index_dir = ref.parent
fastq_channel = Channel.fromFilePairs('data/fastq/*_R{1,2}.fastq.gz')

process map_reads {
    input:
    set sample, '?.fastq.gz' from fastq_channel
    file ref
    file ref_index_dir

    output:
    file("${sample}.sam") into sam_channel

    script:
    """
    echo 'bwa mem -Y "reference/$ref" 1.fastq.gz 2.fastq.gz &gt;${sample}.sam'
    stat -t reference/${ref}.bwt &amp;&amp; touch ${sample}.sam
    """
}


Here, reference inside the script section refers to the parent directory name of the reference, and needs to be adjusted accordingly. Having to refer to reference/$ref is annoying but at least it works — both inside a Docker container and outside of it.