title: How to effectively apply a spatial transformer network
body: <p>I am trying to use spatial transformer networks (STN) for the first time. However, I am encountering some problems.
  I am trying to classify Japanese characters. Those characters can be rotated and scaled differently.
  Because of this, I want to use a STN. But somehow, the STN learns to scale all images down and places them in the upper left section of the image.
  <a href="https://i.stack.imgur.com/IF6ns.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/IF6ns.png" alt="enter image description here" /></a></p>
  <p>My localization network is:</p>
  <pre class="lang-py prettyprint-override"><code>self.conv1 = tf.keras.layers.Conv2D(8, [7, 7], activation='relu')
  self.pool1 = tf.keras.layers.MaxPool2D(strides=2)
          
  self.conv2 = tf.keras.layers.Conv2D(10, [5, 5], activation='relu')
  self.pool2 = tf.keras.layers.MaxPool2D(strides=2)
          
  self.flatten = tf.keras.layers.Flatten()
          
  self.fc1 = tf.keras.layers.Dense(32, activation='relu')
          
  self.fc2 = tf.keras.layers.Dense(6, activation=None, bias_initializer=tf.keras.initializers.constant([1.0, 0.0, 0.0, 0.0, 1.0, 0.0]))
  </code></pre>
  <p>and I am using it as directly after the input.</p>
  <p>This behavior even reduces my accuracy. My expectation would be that the STN crops into the character and scales is up. Therefore my question is, how to use a STN correctly.</p>