How to specify resources for cluster in snakemake I am trying to run snakemake version 3.5.4 on my cluster, where jobs are partially defined by a common template build according manual and also defined though rule-specific parameters. But for some reason cluster configuration is not expanding wildcards (and command does). When I run

snakemake data/genome.fa.gz.bwt -p --jobs 10 --cluster-config cluster.json --cluster "bsub -J "{rule}.{wildcards}" -q {cluster.queue} -n {threads} -M {resources.M} -R {cluster.one_host} -o {cluster.output} -e {cluster.error}"


-J "{rule}.{wildcards}" -q {cluster.queue} -n {threads} -M {resources.M} -R {cluster.one_host} expands as expected. But -o {cluster.output} -e {cluster.error} is replaced by literally logs/cluster/{rule}.{wildcards}.out and logs/cluster/{rule}.{wildcards}.err, i.e. there is no expansion in config file (the config file is attached bellow).

I am puzzled since exactly this lines are in the documentation. What am I doing wrong? Why snakemake is not expanding wildcards in the config file?

Here is my cluster.json

{
    "__default__" :
    {
        "queue"    : "bgee",
        "one_host" : "\"span[hosts=1]\"",
        "name"      : "{rule}.{wildcards}",
        "output"    : "logs/cluster/{rule}.{wildcards}.out",
        "error"     : "logs/cluster/{rule}.{wildcards}.err"
    },
}


Here is relevant part of Snakefile :

rule index_reference :
        threads: 1
        resources: M=20000000
        input : "data/{reference}/genome.fa.gz"
        output : "data/{reference}/genome.fa.gz.bwt"
        shell : "scripts/index_genome.sh {input}"
This feature was added in version 3.6.0. Look at release notes.

{wildcards} normally expands to a something that looks vaguely like a dictionary. In other words, something like key1=value1,key2=value2. Whether you really want to use that as a log file name is up to you.

Let's use a very contrived example to demonstrate this. Here is the Snakefile:

rule all:
    input: "input1.human.txt"

rule blah:
    output: "{sample}.{reference}.txt"
    shell: "touch {output}"


I'm using your cluster.json, so with that let's use a contrived cluster submission program:

snakemake --cluster-config cluster.json --cluster "echo 'rule.wildcards: {rule}.{wildcards} cluster: {cluster} -o {cluster.output} -e {cluster.error} -n {cluster.name}' &gt;&gt; log &amp;&amp; " --jobs 1


If you run that and look in log, you can see that {rule} is expanded as expected, {wildcards} is expanded to sample=input1,reference=human. The output and error parts from cluster.json are expanded correctly for me. Below are the contents of log:

rule.wildcards: blah.sample=input1,reference=human cluster: one_host="span[hosts=1]",output=logs/cluster/blah.sample=input1,reference=human.out,name=blah.sample=input1,reference=human,queue=bgee,error=logs/cluster/blah.sample=input1,reference=human.err -o logs/cluster/blah.sample=input1,reference=human.out -e logs/cluster/blah.sample=input1,reference=human.err -n blah.sample=input1,reference=human


I'm not a big fan of using {wildcards} in paths unless you're doing a parameter sweep, but to each their own. Note that the above results are from snakemake-4.3.1.