anchor.diff:  
   === Further Training Workflow (AKA the UPF workflow)
   
  -. Select N number of models from each HPO instance (that is, from cross correlated feature set) and create the input parameter file (i.e., the upf file). The hpo_log results
  -from the HPO workflow can be used for this. The "Combine HPO logs files adding hpo_id and iteration" code in `scripts/plots.R` is an example of how those
  +. Select N number of models from those produced by each HPO instance and 
  +create the input parameter file (i.e., the "upf" file). The hpo_log results
  +from the HPO workflow can be used for this. The "Combine HPO logs files 
  +adding hpo_id and iteration" code in `scripts/plots.R` is an example of how those
   logs can be concatenated together while adding the hpo_id and iteration as
   column values. The python jupyter notebook `script/cp1_scripts.ipynb` contains
   example code for selecting the top 10 models for each HPO from the combined HPO logs
  -and creating the parameter file for running the selected models.
  +and creating the parameter file for running the selected models. Each row
  +in the upf file contains the hyperparameters for an Uno run in JSON format.
  +
  ++
  +----
  +{"activation": "relu", "dense": "2000 2000 2000", "dense_feature_layers": "1000 1000 1000 1000 1000", "drop": 0.2, "optimizer": "adamax", "residual": 0, "epochs": 100, "batch_size": 6144, "train_sources": "CTRP", "preprocess_rnaseq": "combat", "gpus": "0 1 2 3 4 5", "cell_feature_subset_path": "/autofs/nccs-svm1_proj/med106/ncollier/repos/Supervisor/workflows/cp1/xcorr_data/CTRP_GDSC_2000_1000_features.txt", "use_exported_data": "/autofs/nccs-svm1_proj/med106/ncollier/repos/Supervisor/workflows/cp1/cache/CTRP_GDSC_2000_1000.h5", "warmup_lr": true, "reduce_lr": true, "no_feature_source": true, "no_response_source": true, "cp": true}
  +----
   
   . Create a set of _cfg_ and _test_ scripts for an experiment run by
  -copying an existing set (i.e., _cfg-prm-1.sh_, _cfg-sys-1.sh_, and _test-1.sh_) 
  -in the `test_upf/` directory. Note this is *DIFFERENT* than the `test/` directory. Be sure to update the lines in your _test-n.sh_ that 
  -export the _cfg-prm-N.sh_, and _cfg-sys-N.sh_ scripts to point to your scripts.
  +copying an existing set, i.e., `cfg-prm-1.sh`, `cfg-sys-1.sh`, and `test-1.sh`
  +from the `test_upf/` directory. Note this is *NOT* the `test/` directory. Be sure 
  +to update the lines in your `test-N.sh` that 
  +export the `cfg-prm-N.sh`, and `cfg-sys-N.sh` scripts to point to your scripts.
   Namely,
   +
   ----
  
positive.code: $ wget -r -nd --no-parent -A '*' http://ftp.mcs.anl.gov/pub/candle/public/benchmarks/Pilot1/combo/
  ----
  
  Download the pregenerated cross correlated feature files.
  
  ----
  $ wget http://www.mcs.anl.gov/~wozniak/candle-cp1-data.tgz
  $ tar xfz candle-cp1-data.tgz
  ----
  
  Generate the input data frames from the raw data and the feature files. Uno runs
  much faster with these as input rather than the raw data. A train and test data frame needs to be created for each cross correlated feature file. 
  
  For example, to generate the data for the `CTRP_CCLE_2000_1000.txt` feature file:
  
  ----
  $ cd Benchmarks/Pilot1/Uno
  
negative.code: export CFG_PRM=$THIS/cfg-prm-N.sh
  ----
  +
  See the comments in `test_infer/cfg-prm-1.sh`, and `test_infer/cfg-sys-1.sh` for additional
  information on the various configuration parameters and how to edit them.
  
  . Edit `sh/infer.sh`, if necessary. `infer.sh` is used to run the Uno benchmark's
  `uno_infer.py` python script. Lines 31-33 can be uncommented and edited
  to create multiple copies of the input data to avoid IO contention. If
  this is unnecessary, then the `infer.sh` should not need to be changed.
  
  . Launch the run using your `test-N.sh` script, passing SITE, and optional experiment id 
  as arguments (e.g., `./test-10.sh <site> [expid]`) where 
  site can be one of local, cori, theta, summit etc.
  
  All the output from running the workflow will appear in a directory named with the
  experiment id, either your specified one or the auto-generated one. Each Uno
  model inference run launched by the workflow runs in its own directory: 
  `exp_id/run/X` where _X_ is the id of the run and corresponds to the index of the 
  line of input data that was used for that run. So, `run/0` contains 
  the output for the run that ran with the 1st line from the input 
  file, `run/1` for the second line and so on. Each inference run will 
  produce an `uno_pred.all.tsv` and an `uno_pred.tsv` file. The first contains
  the predictions for each feature and the second is an aggregate view 
  of the first. Additionally a `log.txt` file is created in the experiment directory
  that contains the name of the data input file, the model, the output directory, 
  and number of predictions performed for each inference run.
  
  ==== Associated Files
  
  * Configuration and launch scripts in `test_infer/` (e.g. `cfg-prm-1.sh`, `test-1.sh`, etc.)
  * `swift/infer_workflow.swift` - swift file that executes the workflow
  * `swift/infer_workflow.sh` - launch script for the swift file. This script is 
  configured and launched from the scripts in `test_infer/`.
  