title: How to add 0.001 L2 weight decay for each layer in CNN?
body: <p>I am trying to implement this paper for sound classification: <a href="https://raw.githubusercontent.com/karoldvl/paper-2015-esc-convnet/master/Poster/MLSP2015-poster-page-1.gif" rel="nofollow noreferrer">https://raw.githubusercontent.com/karoldvl/paper-2015-esc-convnet/master/Poster/MLSP2015-poster-page-1.gif</a> <br>
  It is mentioned in the paper that 0.001 L2 weight decay was added for each layer. But, I can't figure out how to do so in Tensorflow.</p>
  
  <p>I found a similar question (<a href="https://stackoverflow.com/questions/36570904/how-to-define-weight-decay-for-individual-layers-in-tensorflow">How to define weight decay for individual layers in TensorFlow?</a>) that uses <code>tf.nn.l2_loss</code>, but it is not clear how can I use this approach for my network. Also, it does not have the parameter of <code>0.001</code> in <code>tf.nn.l2_loss</code>.</p>
  
  <p>My Network:</p>
  
  <pre><code>net = tf.layers.conv2d(inputs=x, filters=80, kernel_size=[57, 6], strides=[1, 1], padding="same", activation=tf.nn.relu)
  print(net)
  net = tf.layers.max_pooling2d(inputs=net, pool_size=[4, 3], strides=[1, 3])
  print(net)
  net = tf.layers.dropout(inputs=net, rate=keep_prob)
  print(net)
  net = tf.layers.conv2d(inputs=net, filters=80, kernel_size=[1, 3], strides=[1, 1], padding="same", activation=tf.nn.relu)
  print(net)
  net = tf.layers.max_pooling2d(inputs=net, pool_size=[1, 3], strides=[1, 3])
  print(net)
  net = tf.layers.flatten(net)
  print(net)
  # Dense Layer
  net = tf.layers.dense(inputs=net, units=5000, activation=tf.nn.relu)
  print(net)
  net = tf.layers.dropout(inputs=net, rate=keep_prob)
  print(net)
  net = tf.layers.dense(inputs=net, units=5000, activation=tf.nn.relu)
  print(net)
  net = tf.layers.dropout(inputs=net, rate=keep_prob)
  print(net)
  logits = tf.layers.dense(inputs=net, units=num_classes)
  print("logits: ", logits)
  </code></pre>
  
  <p>Output:</p>
  
  <pre><code>Tensor("Model/conv2d/Relu:0", shape=(?, 530, 129, 80), dtype=float32)
  Tensor("Model/max_pooling2d/MaxPool:0", shape=(?, 527, 43, 80), dtype=float32)
  Tensor("Model/dropout/Identity:0", shape=(?, 527, 43, 80), dtype=float32)
  Tensor("Model/conv2d_2/Relu:0", shape=(?, 527, 43, 80), dtype=float32)
  Tensor("Model/max_pooling2d_2/MaxPool:0", shape=(?, 527, 14, 80), dtype=float32)
  Tensor("Model/flatten/Reshape:0", shape=(?, 590240), dtype=float32)
  Tensor("Model/dense/Relu:0", shape=(?, 5000), dtype=float32)
  Tensor("Model/dropout_2/Identity:0", shape=(?, 5000), dtype=float32)
  Tensor("Model/dense_2/Relu:0", shape=(?, 5000), dtype=float32)
  Tensor("Model/dropout_3/Identity:0", shape=(?, 5000), dtype=float32)
  logits:  Tensor("Model/dense_3/BiasAdd:0", shape=(?, 20), dtype=float32)
  </code></pre>
  
  <p>I found the implementation of this paper: <a href="https://github.com/karoldvl/paper-2015-esc-convnet/blob/master/Code/_Networks/Net-DoubleConv.ipynb" rel="nofollow noreferrer">https://github.com/karoldvl/paper-2015-esc-convnet/blob/master/Code/_Networks/Net-DoubleConv.ipynb</a> But, it's in <code>pylearn2</code>.</p>
  
  <p>How can I add 0.001 L2 weight decay in my code?</p>