anchor.diff:  
   
   == Running UPF on Theta
  -
  -Step 1. Checkout Supervisor repo
  -----
  -$ git clone https://github.com/ECP-CANDLE/Supervisor.git
  +This tutorial, we will execute mnist example rewritten for CANDLE. The source code is available on https://github.com/ECP-CANDLE/Candle/tree/library/examples/mnist[CANDLE github repo].
  +
  +Step 1. Create directory and checkout Supervisor & Candle repos
  +[source,bash]
  +----
  +$ mkdir candle_tutorial
  +$ cd candle_tutorial
  +$ git clone -b master https://github.com/ECP-CANDLE/Supervisor.git
  +$ git clone -b library https://github.com/ECP-CANDLE/Candle.git
   ----
   
   Step 2. Move to upf workflow directory
  
positive.code:   makeIntegerParam("epochs", lower=20, upper=20)
  )
  ----
  In this example, we are varying four paramters, `batch_size`, `activation`, `optimizer`, `epochs`.
  Entire parameter space will be 5 x 3 x 3 x 1.
  
  After creating this file, we need to point this file.
  ----
  
negative.code: = CANDLE Library User Guide
  
  The CANDLE library enables users run their own deep learning code in high-performance computers that CANDLE supports. With the current version of CANDLE library, users will be able to run a hyperparameter optimization task (mlrMBO workflow) or a parallel execution task (upf workflow). In order to run those workflows, users are required to implement a class and methods (will be explained in section 1) and follow some procedures (see section 2) for each workflow. This user guide will provide an overview of structure and explanation of parameters or variables as needed.
  
  = How to write CANDLE compliant deep learning code
  
  == Minimum requirements
  The CANDLE requires you to create a class and implement two methods. The base class is designed to provides common parameters such as `batch_size`, `epoch`, etc, and enables you register additional parameters as needed. In addition to that, you can mandate some parameters, if needed.
  
  === Base Class
  The CANDLE library provides a base class (`candle.Benchmark`), which you will extend to config your parameters. If you don't have any additional parameters to define, you can use this code as is (see https://github.com/ECP-CANDLE/Candle/blob/library/examples/mnist/mnist.py[MNIST example]).
  [source,python]
  ----
  # mnist.py
  additional_definitions = None
  required = None
  
  class MNIST(candle.Benchmark): // <1>
      def set_locals(self):
          if required is not None:
              self.required = set(required)
          if additional_definitions is not None:
              self.additional_definitions = additional_definitions
  ----
  <1> Create a new class by extending `candle.Benchmark`
  
  ==== Additional Parameters
  You can add a parameter like https://github.com/ECP-CANDLE/Benchmarks/blob/release_01/Pilot1/TC1/tc1.py#L16-L58[this example]. The example illustrates how to define an integer type parameter, `pool` with minimum description.
  
  ----
  additional_definitions = [{
      'name':'pool', // <1>
      'nargs':'+', // <2>
      'type': int, // <3>
      'help':'network structure of shared layer' // <4>
  },
  ]
  ----
  <1> required. Name of parameter.
  <2> optional. The number of command-line arguments.
  <3> required. The type to which the command-line arguments should be converted.
  <4> optional. A brief description of what the argument does.
  
  ==== Mandatory parameters
  If you like to mandatory some common parameters, you will need to define `required` array and pass it to the definition of your class.
  ----
  required = [
      'data_url',
      'epochs',
      'batch_size',
      'pool',
      'save'
  ]
  ----
  
  ==== Thread Optimization
  Some high-performance computing machines like `Theta`, the performance will be greatly improved if we let CANDLE handles threads. So, it is generally recommended to have code like line 14 to 21 in https://github.com/ECP-CANDLE/Candle/blob/library/examples/mnist/mnist.py#L9-L16[this example]
  ----
  # thread optimization
  import os
  from keras import backend as K
  if K.backend() == 'tensorflow' and 'NUM_INTRA_THREADS' in os.environ:
      import tensorflow as tf
      sess = tf.Session(config=tf.ConfigProto(inter_op_parallelism_threads=int(os.environ['NUM_INTER_THREADS']),
                                              intra_op_parallelism_threads=int(os.environ['NUM_INTRA_THREADS'])))
      K.set_session(sess)
  ----
  
  
  === Initialize_parameters Method
  In `initialize_parameters` method, we will instantiate the base class, and finally build argument parser to recognize your customized parameters in addition to the default parameters ( `default_utils.initialize_parameters()`). The `initialize_parameters` should return a python dictionary, which will be accessed in `run(Dict)` method.
  
  [source,python]
  ----
  # this is a part of mnist_mlp_candle.py
  
  import mnist
  import candle_keras as candle
  
  def initialize_parameters():
      mnist_common = mnist.MNIST(mnist.file_path,
          'mnist_params.txt', // <4>
          'keras',
          prog='mnist_mlp',
          desc='MNIST example'
      )  // <1>
  
      # Initialize parameters
      gParameters = default_utils.initialize_parameters(mnist_common) // <2>
      ..
  
      return gParameters   // <3>
  ----
  <1> instantiate base class
  <2> build argument parser
  <3> this method should return python dictionary
  <4> a file that contains default values for the given parameters. See below for example.
  
  ----
  [Global_Params]
  epochs=20
  batch_size=128
  activation='relu'
  optimizer='rmsprop'
  ----
  
  When parameters are not overwritten by workflows, these values will be used.
  
  === Run Method
  You can place your deep learning code in `run(Dict)` method. Your parameters are accessible like `gParameters['batch_size']`.
  
  We have an https://github.com/ECP-CANDLE/Candle/blob/library/examples/mnist/mnist_mlp_candle.py[example], that converted a simple MNIST neural net `mnist_mlp.py` provided by https://github.com/keras-team/keras/blob/master/examples/mnist_mlp.py[Keras Team] into CANDLE compliant form. In this example, you will see how the actual neural network was transplanted in `run` method.
  
  Finally, the `run()` returns keras history object. This can be omitted for upf workflow, but required for HPO workflow.
  
  [source,python]
  ----
  # this is a part of mnist_mlp_candle.py
  
  def run(gParameters): // <1>
      ##########################################
      # Your DL start here. See mnist_mlp.py   #
      ##########################################
  