title: concatenated relu in caffe using python
body: <p>I want to implement "concatenated relu" in python using caffe. 
  It defines as ConcRelu = Concat (Relu(x), -Relu(-x))</p>
  
  <p>Bellow is a naive implementation, but does not work! I would appreciate any help!</p>
  
  <pre><code>def prelu(self,bottom,args):
      'prelu layer'
      return L.PReLU(bottom, in_place=True)
  
  def crelu(self,bottom,args):
      'crelu layer'
      negative_slope = 0
      if len(args) == 1:
          negative_slope = float(args[0])
      rp = {'negative_slope': negative_slope}
  
      sp = {  'bias_term': False,
              'filler': {'value': -1}
      }
  
      neg_side = L.Scale(bottom, scale_param=sp)
  
      a = [L.ReLU(neg_side, relu_param=rp, in_place=True), L.ReLU(bottom, relu_param=rp, in_place=True)]
      cp = { 'axis' : 1 }
  
      return L.Concat(*a, concat_param=cp)
  </code></pre>
  
  <p>==== The output (with error) I get: ====</p>
  
  <pre><code>WARNING: Logging before InitGoogleLogging() is written to STDERR
  I0130 17:15:01.251324 145276 upgrade_proto.cpp:67] Attempting to upgrade input file specified using deprecated input fields: res/2048_1e5_0.00_s/CNN_020_bs10/CNN_020_tmp/CNN_020_deploy.txt
  
  I0130 17:15:01.251384 145276 upgrade_proto.cpp:70] Successfully upgraded file specified using deprecated input fields.
  W0130 17:15:01.251389 145276 upgrade_proto.cpp:72] Note that future Caffe releases will only support input layers and not input fields.
  I0130 17:15:01.657894 145276 net.cpp:51] Initializing net from parameters:
  name: "CNN_020"
  state {
    phase: TEST
    level: 0
  }
  layer {
    name: "input"
    type: "Input"
    top: "data"
    input_param {
      shape {
        dim: 1
        dim: 2
        dim: 1
        dim: 2048
      }
    }
  }
  layer {
    name: "conv1"
    type: "Convolution"
    bottom: "data"
    top: "conv1"
    param {
      lr_mult: 1
      decay_mult: 1
    }
    convolution_param {
      num_output: 16
      bias_term: false
      pad: 0
      pad: 0
      kernel_size: 1
      kernel_size: 3
      group: 1
      stride: 1
      stride: 1
      weight_filler {
        type: "xavier"
      }
      bias_filler {
        type: "constant"
        value: 0
      }
      axis: 1
    }
  }
  layer {
    name: "Scale1"
    type: "Scale"
    bottom: "conv1"
    top: "Scale1"
    scale_param {
      filler {
        value: -1
      }
      bias_term: false
    }
  }
  layer {
    name: "ReLU1"
    type: "ReLU"
    bottom: "Scale1"
    top: "Scale1"
    relu_param {
      negative_slope: 0
    }
  }
  layer {
    name: "ReLU2"
    type: "ReLU"
    bottom: "conv1"
    top: "conv1"
    relu_param {
      negative_slope: 0
    }
  }
  layer {
    name: "crelu1"
    type: "Concat"
    bottom: "Scale1"
    bottom: "conv1"
    top: "crelu1"
    concat_param {
      axis: 1
    }
  }
  layer {
    name: "ampl"
    type: "InnerProduct"
    bottom: "crelu1"
    top: "ampl"
    param {
      lr_mult: 1
      decay_mult: 1
    }
    inner_product_param {
      num_output: 20
      bias_term: false
      weight_filler {
        type: "gaussian"
        std: 0.01
      }
      bias_filler {
        type: "constant"
        value: 0.2
      }
    }
  }
  I0130 17:15:01.658027 145276 layer_factory.hpp:77] Creating layer input
  I0130 17:15:01.658046 145276 net.cpp:84] Creating Layer input
  I0130 17:15:01.658058 145276 net.cpp:380] input -&gt; data
  I0130 17:15:01.658095 145276 net.cpp:122] Setting up input
  I0130 17:15:01.658108 145276 net.cpp:129] Top shape: 1 2 1 2048 (4096)
  I0130 17:15:01.658113 145276 net.cpp:137] Memory required for data: 16384
  I0130 17:15:01.658118 145276 layer_factory.hpp:77] Creating layer conv1
  I0130 17:15:01.658131 145276 net.cpp:84] Creating Layer conv1
  I0130 17:15:01.658138 145276 net.cpp:406] conv1 &lt;- data
  I0130 17:15:01.658144 145276 net.cpp:380] conv1 -&gt; conv1
  I0130 17:15:01.658269 145276 net.cpp:122] Setting up conv1
  I0130 17:15:01.658282 145276 net.cpp:129] Top shape: 1 16 1 2046 (32736)
  I0130 17:15:01.658287 145276 net.cpp:137] Memory required for data: 147328
  I0130 17:15:01.658298 145276 layer_factory.hpp:77] Creating layer conv1_conv1_0_split
  I0130 17:15:01.658313 145276 net.cpp:84] Creating Layer conv1_conv1_0_split
  I0130 17:15:01.658318 145276 net.cpp:406] conv1_conv1_0_split &lt;- conv1
  I0130 17:15:01.658324 145276 net.cpp:380] conv1_conv1_0_split -&gt; conv1_conv1_0_split_0
  I0130 17:15:01.658332 145276 net.cpp:380] conv1_conv1_0_split -&gt; conv1_conv1_0_split_1
  I0130 17:15:01.658344 145276 net.cpp:122] Setting up conv1_conv1_0_split
  I0130 17:15:01.658350 145276 net.cpp:129] Top shape: 1 16 1 2046 (32736)
  I0130 17:15:01.658355 145276 net.cpp:129] Top shape: 1 16 1 2046 (32736)
  I0130 17:15:01.658360 145276 net.cpp:137] Memory required for data: 409216
  I0130 17:15:01.658365 145276 layer_factory.hpp:77] Creating layer Scale1
  I0130 17:15:01.658375 145276 net.cpp:84] Creating Layer Scale1
  I0130 17:15:01.658380 145276 net.cpp:406] Scale1 &lt;- conv1_conv1_0_split_0
  I0130 17:15:01.658386 145276 net.cpp:380] Scale1 -&gt; Scale1
  I0130 17:15:01.658408 145276 net.cpp:122] Setting up Scale1
  I0130 17:15:01.658416 145276 net.cpp:129] Top shape: 1 16 1 2046 (32736)
  I0130 17:15:01.658419 145276 net.cpp:137] Memory required for data: 540160
  I0130 17:15:01.658427 145276 layer_factory.hpp:77] Creating layer ReLU1
  I0130 17:15:01.658435 145276 net.cpp:84] Creating Layer ReLU1
  I0130 17:15:01.658440 145276 net.cpp:406] ReLU1 &lt;- Scale1
  I0130 17:15:01.658447 145276 net.cpp:367] ReLU1 -&gt; Scale1 (in-place)
  I0130 17:15:01.658455 145276 net.cpp:122] Setting up ReLU1
  I0130 17:15:01.658462 145276 net.cpp:129] Top shape: 1 16 1 2046 (32736)
  I0130 17:15:01.658465 145276 net.cpp:137] Memory required for data: 671104
  I0130 17:15:01.658469 145276 layer_factory.hpp:77] Creating layer ReLU2
  I0130 17:15:01.658474 145276 net.cpp:84] Creating Layer ReLU2
  I0130 17:15:01.658479 145276 net.cpp:406] ReLU2 &lt;- conv1_conv1_0_split_1
  F0130 17:15:01.658486 145276 net.cpp:375] Top blob 'conv1' produced by multiple sources.
  *** Check failure stack trace: ***
  /afs/in2p3.fr/home/n/nhatami/sps/spectroML/src/trainAndTest: line 99: 145276 Aborted                 $pydir/dumpLayersSize.py ${tmp_root}_deploy.txt ${oroot}
  Tue Jan 30 17:15:01 CET 2018
  /usr/bin/time -v caffe -gpu 0 --log_dir=res/2048_1e5_0.00_s/CNN_020_bs10/CNN_020_tmp train -solver res/2048_1e5_0.00_s/CNN_020_bs10/CNN_020_tmp/CNN_020_solver.txt
  Tue Jan 30 17:15:20 CET 2018
  cp: missing destination file operand after �~@~Xres/2048_1e5_0.00_s/CNN_020_bs10/CNN_020.caffemodel�~@~Y
  Try 'cp --help' for more information.
  tail: cannot open �~@~Xres/2048_1e5_0.00_s/CNN_020_bs10/CNN_020_solver.log.train�~@~Y for reading: No such file or directory
  /afs/in2p3.fr/home/n/nhatami/sps/spectroML/src/trainAndTest: line 204: gnuplot: command not found
  /afs/in2p3.fr/home/n/nhatami/sps/spectroML/src/trainAndTest: line 223: gnuplot: command not found
  /usr/bin/time -v -o res/2048_1e5_0.00_s/CNN_020_bs10/CNN_020_time_train.txt python /afs/in2p3.fr/home/n/nhatami/sps/spectroML/src/python/testdataset.py -eg hist -l label -mf=data/2048_1e5_0.00/2048_1e5_0.00_s_met.txt res/2048_1e5_0.00_s/CNN_020_bs10/CNN_020_deploy.txt res/2048_1e5_0.00_s/CNN_020_bs10/CNN_020.caffemodel data/2048_1e5_0.00/2048_1e5_0.00_s_train.h5 res/2048_1e5_0.00_s/CNN_020_bs10/restest/CNN_020_train
  WARNING: Logging before InitGoogleLogging() is written to STDERR
  W0130 17:15:26.614568 145537 _caffe.cpp:139] DEPRECATION WARNING - deprecated use of Python interface
  W0130 17:15:26.614625 145537 _caffe.cpp:140] Use this instead (with the named "weights" parameter):
  W0130 17:15:26.614630 145537 _caffe.cpp:142] Net('res/2048_1e5_0.00_s/CNN_020_bs10/CNN_020_deploy.txt', 1, weights='res/2048_1e5_0.00_s/CNN_020_bs10/CNN_020.caffemodel')
  ('res/2048_1e5_0.00_s/CNN_020_bs10/CNN_020_deploy.txt', 'res/2048_1e5_0.00_s/CNN_020_bs10/CNN_020.caffemodel', 'data/2048_1e5_0.00/2048_1e5_0.00_s_train.h5', 'data/2048_1e5_0.00/2048_1e5_0.00_s_met.txt')
  Traceback (most recent call last):
    File "/afs/in2p3.fr/home/n/nhatami/sps/spectroML/src/python/testdataset.py", line 395, in &lt;module&gt;
      aE = predict(model, weights, data)
    File "/afs/in2p3.fr/home/n/nhatami/sps/spectroML/src/python/testdataset.py", line 200, in predict
      net = caffe.Net(modelFile, weightsFile, caffe.TEST)
  RuntimeError: Could not open file res/2048_1e5_0.00_s/CNN_020_bs10/CNN_020.caffemodel
  /usr/bin/time -v -o res/2048_1e5_0.00_s/CNN_020_bs10/CNN_020_time_val.txt python /afs/in2p3.fr/home/n/nhatami/sps/spectroML/src/python/testdataset.py -eg hist -l label -mf=data/2048_1e5_0.00/2048_1e5_0.00_s_met.txt res/2048_1e5_0.00_s/CNN_020_bs10/CNN_020_deploy.txt res/2048_1e5_0.00_s/CNN_020_bs10/CNN_020.caffemodel data/2048_1e5_0.00/2048_1e5_0.00_s_val.h5 res/2048_1e5_0.00_s/CNN_020_bs10/restest/CNN_020_val
  WARNING: Logging before InitGoogleLogging() is written to STDERR
  W0130 17:15:32.969283 145545 _caffe.cpp:139] DEPRECATION WARNING - deprecated use of Python interface
  W0130 17:15:32.969337 145545 _caffe.cpp:140] Use this instead (with the named "weights" parameter):
  W0130 17:15:32.969341 145545 _caffe.cpp:142] Net('res/2048_1e5_0.00_s/CNN_020_bs10/CNN_020_deploy.txt', 1, weights='res/2048_1e5_0.00_s/CNN_020_bs10/CNN_020.caffemodel')
  ('res/2048_1e5_0.00_s/CNN_020_bs10/CNN_020_deploy.txt', 'res/2048_1e5_0.00_s/CNN_020_bs10/CNN_020.caffemodel', 'data/2048_1e5_0.00/2048_1e5_0.00_s_val.h5', 'data/2048_1e5_0.00/2048_1e5_0.00_s_met.txt')
  Traceback (most recent call last):
    File "/afs/in2p3.fr/home/n/nhatami/sps/spectroML/src/python/testdataset.py", line 395, in &lt;module&gt;
      aE = predict(model, weights, data)
    File "/afs/in2p3.fr/home/n/nhatami/sps/spectroML/src/python/testdataset.py", line 200, in predict
      net = caffe.Net(modelFile, weightsFile, caffe.TEST)
  RuntimeError: Could not open file res/2048_1e5_0.00_s/CNN_020_bs10/CNN_020.caffemodel
  /usr/bin/time -v -o res/2048_1e5_0.00_s/CNN_020_bs10/CNN_020_time_test.txt python /afs/in2p3.fr/home/n/nhatami/sps/spectroML/src/python/testdataset.py -eg hist -l label -mf=data/2048_1e5_0.00/2048_1e5_0.00_s_met.txt res/2048_1e5_0.00_s/CNN_020_bs10/CNN_020_deploy.txt res/2048_1e5_0.00_s/CNN_020_bs10/CNN_020.caffemodel data/2048_1e5_0.00/2048_1e5_0.00_s_test.h5 res/2048_1e5_0.00_s/CNN_020_bs10/restest/CNN_020_test
  WARNING: Logging before InitGoogleLogging() is written to STDERR
  W0130 17:15:38.393525 145552 _caffe.cpp:139] DEPRECATION WARNING - deprecated use of Python interface
  W0130 17:15:38.393584 145552 _caffe.cpp:140] Use this instead (with the named "weights" parameter):
  W0130 17:15:38.393587 145552 _caffe.cpp:142] Net('res/2048_1e5_0.00_s/CNN_020_bs10/CNN_020_deploy.txt', 1, weights='res/2048_1e5_0.00_s/CNN_020_bs10/CNN_020.caffemodel')
  ('res/2048_1e5_0.00_s/CNN_020_bs10/CNN_020_deploy.txt', 'res/2048_1e5_0.00_s/CNN_020_bs10/CNN_020.caffemodel', 'data/2048_1e5_0.00/2048_1e5_0.00_s_test.h5', 'data/2048_1e5_0.00/2048_1e5_0.00_s_met.txt')
  Traceback (most recent call last):
    File "/afs/in2p3.fr/home/n/nhatami/sps/spectroML/src/python/testdataset.py", line 395, in &lt;module&gt;
      aE = predict(model, weights, data)
    File "/afs/in2p3.fr/home/n/nhatami/sps/spectroML/src/python/testdataset.py", line 200, in predict
      net = caffe.Net(modelFile, weightsFile, caffe.TEST)
  RuntimeError: Could not open file res/2048_1e5_0.00_s/CNN_020_bs10/CNN_020.caffemodel
  </code></pre>