Looper: can't specify the individual path for one pipeline run I have multiple samples that I need to run one pipeline on. I am using Looper: https://github.com/pepkit/looper

I have a real trouble specifying a separate location for each pipeline output files. The first time when I did not care about it thinking that Looper would just automatically do that, it lead to multiple files being overwritten. Then, trying to avoid it I tried (and it did not work):


  
  Adding {sample_name} to the output_dir of the metadata section of project_config.yaml file:
  


 metadata:
  sample_annotation: sandro_annotation.csv
  output_dir: /scratch/nv4e/sandro_looper_results/{sample_name}
  pipelines_dir: [../pipeline/pipeline_interface.yaml]



  
  Creating a derived_column in the .csv annotation file:
  


sample_name,library,sample_name_1_read,sample_name_2_read,output_folder
abeta_24h_1,RNA-seq,read1,read2,out
abeta_24h_2,RNA-seq,read1,read2,out
abeta_24h_3,RNA-seq,read1,read2,out


pipeline_interface.yaml:

...
pipelines:
  sandro_rna_seq:
    name: sandro_rna_seq
    path: sandro_rna_seq.py  # relative to this 
pipeline_interface.yaml file
    looper_args: True
    arguments:
      "-S": sample_name
      "--input": sample_name_1_read
      "-O": output_folder
...


and project.config.yaml:

...
derived_columns: [sample_name_1_read,sample_name_2_read,output_folder]
data_sources:
  read1: /scratch/nv4e/sandro_looper/data/merged/{sample_name}_r1.fq.gz
  read2: /scratch/nv4e/sandro_looper/data/merged/{sample_name}_r2.fq.gz
  out: /scratch/nv4e/sandro_looper_results/results_pipeline/{sample_name}
...


In this particular case the parameter -O was formed correctly which I was able to see in sandro_looper_results/submissions/ .sub file, however, it was supplied the second time, apparently from the output_dir section of the project_config.yaml and was overriding the first -O command. So, obviously I tried to delete output_dir line but it is a required setting for Looper, so it can not run without it.


  
  I have also tried supplying -O command in project_config.yaml in the pipeline_args: section with no success.
  


I really do not know what I could do at this point. Any suggestions would be greatly appreciated,
Looper does not generate any of these per-sample outputs.

Looper does not have any control over where your pipeline puts its outputs.

The problem is in your pipeline, not in looper. Assuming you're using a pypiper pipeline, I suggest reading the docs; for example, the "your first pipeline" tutorial: http://pypiper.readthedocs.io/en/latest/tutorials-first.html

For real-life example, see this pipeline


outfolder = os.path.abspath(os.path.join(args.output_parent, args.sample_name))
pm = pypiper.PipelineManager(name="RRBS", outfolder=outfolder, args=args, version=__version__)