title: Captcha recognizing with convnet, how to define loss function
body: <p>I have small research project where I try to decode some captcha images. I use convnet implemented in Tensorflow 0.9, based on MNIST example (<a href="https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3_NeuralNetworks/convolutional_network.py" rel="nofollow noreferrer">https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3_NeuralNetworks/convolutional_network.py</a>)</p>
  
  <p>My code is available at github <a href="https://github.com/ksopyla/decapcha/blob/master/decaptcha_convnet.py" rel="nofollow noreferrer">https://github.com/ksopyla/decapcha/blob/master/decaptcha_convnet.py</a></p>
  
  <p>I have try to do reproduce the idea described:</p>
  
  <ul>
  <li>"Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks" Goodfellow at al (<a href="https://arxiv.org/pdf/1312.6082.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1312.6082.pdf</a>)</li>
  <li>"CAPTCHA Recognition with Active Deep Learning" Stark at al (<a href="https://vision.in.tum.de/_media/spezial/bib/stark-gcpr15.pdf" rel="nofollow noreferrer">https://vision.in.tum.de/_media/spezial/bib/stark-gcpr15.pdf</a>)</li>
  </ul>
  
  <p>where particular sequence of chars is encoded as one binary vector. In my case the captchas contains max 20 latin chars, each char is encoded as 63 dim binary vector, where 1 bit is set at position, according to:</p>
  
  <ul>
  <li>digits '0-9' - 1 at position 0- 9</li>
  <li>big letters 'A-Z' - 1 at position 10-35</li>
  <li>small letters 'a-z' - 1 atposition  36-61</li>
  <li>position 62 is reserved for blank char '<em>' (words shorter then 20 chars are filled with '</em>' up to 20)</li>
  </ul>
  
  <p>So finally when I concatenate all 20 chars I get 20*63 dim vector which my network should learn. My main issue is how to define proper loss function for optimizer.</p>
  
  <p>Architecture of my network:</p>
  
  <ol>
  <li>conv 3x3x32 ->relu -> pooling(k=2) ->dropout</li>
  <li>conv 3x3x64 ->relu -> pooling(k=2) ->dropout</li>
  <li>conv 3x3x64 ->relu -> pooling(k=2) ->dropout</li>
  <li>FC 1024 ->relu -> dropout</li>
  <li>Output 20*63 - </li>
  </ol>
  
  <p>So my main issue is how to define loss for optimizer and how to evaluate the model. I have try something like this</p>
  
  <pre><code># Construct model
  pred = conv_net(x, weights, biases, keep_prob)
  
  # Define loss and optimizer
  
  #split prediction for each char it takes 63 continous postions, we have 20 chars
  split_pred = tf.split(1,20,pred)
  split_y = tf.split(1,20,y)
  
  
  #compute partial softmax cost, for each char
  costs = list()
  for i in range(20):  
     costs.append(tf.nn.softmax_cross_entropy_with_logits(split_pred[i],split_y[i]))
  
  #reduce cost for each char
  rcosts = list()
  for i in range(20):
      rcosts.append(tf.reduce_mean(costs[i]))
  
  # global reduce    
  loss = tf.reduce_sum(rcosts)
  optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(loss)
  
  
  # Evaluate model
  
  # pred are in format batch_size,20*63, reshape it in order to have each     character prediction
  # in row, then take argmax of each row (across columns) then check if it is     equal 
  # original label max indexes
  # then sum all good results and compute mean (accuracy)
  
  #batch, rows, cols
  p = tf.reshape(pred,[batch_size,20,63])
  #max idx acros the rows
  #max_idx_p=tf.argmax(p,2).eval()
  max_idx_p=tf.argmax(p,2)
  
  l = tf.reshape(y,[batch_size,20,63])
  #max idx acros the rows
  #max_idx_l=tf.argmax(l,2).eval()
  max_idx_l=tf.argmax(l,2)
  
  correct_pred = tf.equal(max_idx_p,max_idx_l)
  accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))enter code         here
  </code></pre>
  
  <p>I try to split each char from output and do softmax and cross_entropy for each char separatelly, then combine all costs. But I have mixed the tensorflow functions with normal python lists, can I do this? Will tensorflow engine understand this? Which tensorflow functions I can use instead of python lists?</p>
  
  <p>The accuracy is computed in similar manner, the output is reshaped to 20x63 and I take argmax from each row than compare with true encoded char.</p>
  
  <p>When I run this loss function is decreasing, but accuracy rise then fall.
  This picture shows how it looks <a href="https://plon.io/files/57a0a7fb4bb1210001ca0476" rel="nofollow noreferrer">https://plon.io/files/57a0a7fb4bb1210001ca0476</a><a href="https://i.stack.imgur.com/eGmlQ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/eGmlQ.png" alt="loss_function"></a></p>
  
  <p>I would be grateful for any further comments, mistakes I have made or ideas to implement.</p>