Clipping threshold of softmax layer I recently came across a paper on using (rather simple version of) LSTM for sentiment classification, and it describes its network settings as:


  We randomize the parameters with uniform distribution U(-0.003,
  0.003), set the clipping threshold of softmax layer as 200 and set learning rate as 0.01.


I am trying to reproduce their results with my code written in Tensorflow. I have to say I find it a bit confusing by the meaning of "set the clipping threshold of softmax layer as 200". 200???

Can someone explain this to me please so I know how to implement this using Tensorflow? Thanks so much!!!

Update, ok so I found the their code and below is how they use this clipping threshold of softmax layer:

for(int k = 0; k &lt; softmax.outputG.length; k++)
    softmax.outputG[k] = 0.0;
softmax.outputG[goldPol] = 1.0 / softmax.output[goldPol];

// if ||g|| &gt;= threshold, then g &lt;- g * threshold / ||g|| 
if(Math.abs(softmax.outputG[goldPol]) &gt; clippingThreshold)
{
    if(softmax.outputG[goldPol] &gt; 0)
        softmax.outputG[goldPol] =  clippingThreshold;
    else
        softmax.outputG[goldPol] =  -1.0 * clippingThreshold;
}


I still can't say I fully understand the use of this threshold..
I think what they were doing is just gradient clipping. It keeps the gradient of softmax layer between [-200,200]