anchor.diff:  = CANDLE Library User Guide
   
  -The CANDLE library provides a wrapper class and utility functions, which enable users run their own deep learning code in high performance computers that CANDLE supports. With the current version of CANDLE library, users should be able to run hyperparameter optimization (mlrMBO workflow) or parallel excution (upf workflow). Due to the design of both workflows, users are required to implement certain methods (will be explained in section 1) and modify several config files (section 2). This user guide will provide an overview of structure and explanation of parameters or varaiables as needed.
  +The CANDLE library enables users run their own deep learning code in high-performance computers that CANDLE supports. With the current version of CANDLE library, users will be able to run a hyperparameter optimization task (mlrMBO workflow) or a parallel execution task (upf workflow). In order to run those workflows, users are required to implement a class and methods (will be explained in section 1) and follow some procedures (see section 2) for each workflow. This user guide will provide an overview of structure and explanation of parameters or variables as needed.
   
   = How to write CANDLE compliant deep learning code
   
   == Minimum requirements
  -The CANDLE requires two methods, `initialize_parameters()` and `run()`.
  +The CANDLE requires you to create a class and implement two methods. The base class is designed to provides common parameters such as `batch_size`, `epoch`, etc, and enables you register additional parameters as needed. In addition to that, you can mandate some parameters, if needed.
  +
  +=== Base Class
  +The CANDLE library provides a base class (`candle.Benchmark`), which you will extend to config your parameters. If you don't have any additional parameters to define, you can use this code as is (see https://github.com/ECP-CANDLE/Candle/blob/library/examples/mnist/mnist.py[MNIST example]).
  +[source,python]
  +----
  +# mnist.py
  +additional_definitions = None
  +required = None
  +
  +class MNIST(candle.Benchmark): // <1>
  +    def set_locals(self):
  +        if required is not None:
  +            self.required = set(required)
  +        if additional_definitions is not None:
  +            self.additional_definitions = additional_definitions
  +----
  +<1> Create a new class by extending `candle.Benchmark`
  +
  +==== Additional Parameters
  +You can add a parameter like https://github.com/ECP-CANDLE/Benchmarks/blob/release_01/Pilot1/TC1/tc1.py#L16-L58[this example]. The example illustrates how to define an integer type parameter, `pool` with minimum description.
  +
  +----
  +additional_definitions = [{
  +    'name':'pool', // <1>
  +    'nargs':'+', // <2>
  +    'type': int, // <3>
  +    'help':'network structure of shared layer' // <4>
  +},
  +]
  +----
  +<1> required. Name of parameter.
  +<2> optional. The number of command-line arguments.
  +<3> required. The type to which the command-line arguments should be converted.
  +<4> optional. A brief description of what the argument does.
  +
  +==== Mandatory parameters
  +If you like to mandatory some common parameters, you will need to define `required` array and pass it to the definition of your class.
  +----
  +required = [
  +    'data_url',
  +    'epochs',
  +    'batch_size',
  +    'pool',
  +    'save'
  +]
  +----
  +
  +==== Thread Optimization
  +Some high-performance computing machines like `Theta`, the performance will be greatly improved if we let CANDLE handles threads. So, it is generally recommended to have code like line 14 to 21 in https://github.com/ECP-CANDLE/Candle/blob/library/examples/mnist/mnist.py#L9-L16[this example]
  +----
  +# thread optimization
  +import os
  +from keras import backend as K
  +if K.backend() == 'tensorflow' and 'NUM_INTRA_THREADS' in os.environ:
  +    import tensorflow as tf
  +    sess = tf.Session(config=tf.ConfigProto(inter_op_parallelism_threads=int(os.environ['NUM_INTER_THREADS']),
  +                                            intra_op_parallelism_threads=int(os.environ['NUM_INTRA_THREADS'])))
  +    K.set_session(sess)
  +----
  +
   
   === Initialize_parameters Method
  -In `initialize_parameters` method, we will construct a class and build a parameter set, which will be used inside your deep learning code (run method). We provides some common parameters such as `batch_size`, `epochs`, etc. In addition to that, you can construct your own parameters (see Aurgument Specification section below). Finally, the `initialize_parameters` should return a python dictionary, in this doc, will be called `gParameters` (global parameters).
  -
  -=== Run Method
  -You can place your deep learning code in `run(Dict)` method. You can use parameter varaiable like `gParameters['batch_size']`.
  -
  -We have an https://github.com/ECP-CANDLE/Candle/blob/library/examples/mnist/mnist_mlp_candle.py[example], that converted a simple MNIST neural net `mnist_mlp.py` provided by https://github.com/keras-team/keras/blob/master/examples/mnist_mlp.py[Keras Team] into CANDLE compliant form. In this example, you will see how the `initialize_parameters` method is implemented and how the actual NN code was transplanted in `run` method.
  -
  -Finally, the `run()` returns history. This can be omitted for upf workflow, but required for HPO workflow.
  +In `initialize_parameters` method, we will instantiate the base class, and finally build argument parser to recognize your customized parameters in addition to the default parameters ( `default_utils.initialize_parameters()`). The `initialize_parameters` should return a python dictionary, which will be accessed in `run(Dict)` method.
   
   [source,python]
   ----
  +# this is a part of mnist_mlp_candle.py
  +
  +import mnist
  +import candle_keras as candle
  +
   def initialize_parameters():
  -    mnist_common = common.MNIST(common.file_path,
  -        'default_model.txt',
  +    mnist_common = mnist.MNIST(mnist.file_path,
  +        'mnist_params.txt', // <4>
           'keras',
           prog='mnist_mlp',
           desc='MNIST example'
       )  // <1>
   
       # Initialize parameters
  -    gParameters = default_utils.initialize_parameters(mnist_common)
  +    gParameters = default_utils.initialize_parameters(mnist_common) // <2>
       ..
   
  -    return gParameters   // <2>
  -
  -def run(gParameters): // <3>
  +    return gParameters   // <3>
  +----
  +<1> instantiate base class
  +<2> build argument parser
  +<3> this method should return python dictionary
  +<4> a file that contains default values for the given parameters. See below for example.
  +
  +----
  +[Global_Params]
  +epochs=20
  +batch_size=128
  +activation='relu'
  +optimizer='rmsprop'
  +----
  +
  +When parameters are not overwritten by workflows, these values will be used.
  +
  +=== Run Method
  +You can place your deep learning code in `run(Dict)` method. Your parameters are accessible like `gParameters['batch_size']`.
  +
  +We have an https://github.com/ECP-CANDLE/Candle/blob/library/examples/mnist/mnist_mlp_candle.py[example], that converted a simple MNIST neural net `mnist_mlp.py` provided by https://github.com/keras-team/keras/blob/master/examples/mnist_mlp.py[Keras Team] into CANDLE compliant form. In this example, you will see how the actual neural network was transplanted in `run` method.
  +
  +Finally, the `run()` returns keras history object. This can be omitted for upf workflow, but required for HPO workflow.
  +
  +[source,python]
  +----
  +# this is a part of mnist_mlp_candle.py
  +
  +def run(gParameters): // <1>
       ##########################################
       # Your DL start here. See mnist_mlp.py   #
       ##########################################
  
positive.code: you will need to set `BENCHMARK_DIR` to point the directory that your script locates, and
  `MODEL_PYTHON_SCRIPT` to name the script you want to run
  ----
  BENCHMARK_DIR=directory_where_my_script_locates
  MODEL_PYTHON_SCRIPT=my_script
  ----
  
  Step 4. Set execution plan. Check `test/upf-1.txt` for parameter configuration and modify as needed.
  This file contains multiple number of JSON documents. Each JSON document will contain the command line parameters.
  
negative.code:     ##########################################
      # End of mnist_mlp.py ####################
      ##########################################
    return history  // <2>
  
  ----
  <1> run method receives parameter dictionary
  <2> returns keras history object or None*
  
  * The mlrMBO workflow requires to return keras history object so that the workflow can evaluate the model. The upf workflow does not have the evaluation process, so you can return `None`.
  
  
  = How to run CANDLE compliant code in Theta
  As mentioned above, we offer two different workflows in CANDLE:
  