title: High loss from convolutional autoencoder keras
body: <p>I am training a convolutional autoencoder and I am having trouble getting the loss to decrease and was hoping someone could point out some possible improvements.</p>
  
  <p>I have greyscale images of 1024x1024 (I've tried the same thing with 512x512 too) and I'm hoping to compress them for unsupervised clustering. My full model is below but it follows a pretty basic pattern of several Conv2D layers w/ max pooling, followed by a dense layer, then reshape and Conv2D layers back up to the original image size. </p>
  
  <p>Some things I've tried so far:</p>
  
  <p>1) I found that mse worked better as a loss function than binary cross entropy, because the pixel luminosity values are so far from evenly distributed (binary cross entropy gets stuck assigning everything to a value of 1, which has low error but isn't useful).</p>
  
  <p>2) I can easily achieve very low error and near perfect (to my eye at least) image reconstruction if I just get rid of the dense layer in the middle and compress the image slightly less. This is fairly obvious, but I guess it shows that I'm not making some kind of error resulting in nonsense outputs.</p>
  
  <p>3) My loss doesn't really get below 0.02-0.03. Still, at 0.025 or so the image is reconstructed enough that it's obvious that the output is coming from the input, not some kind of random noise (like making every pixel the same intensity or something). I think getting it below 0.01 would be enough that I can cluster on. My lowest (albeit on a slightly easier subset of my data) was 0.018 and I could see clear clustering in samples when I plotted my encoded values in a heatmap.</p>
  
  <p>4) I get a lot of dying ReLUs when my middle dense layer uses a ReLU activation, which makes it less useful for the eventual clustering. I use tanh instead. I also found that "he_normal" works better as an initialization for the dense layer.</p>
  
  <p>5) Adding more dense layers in the middle doesn't seem to help at all.</p>
  
  <p>6) Reversing the shape of the encoder (so that it goes from fewer kernels per layer to more kernels) doesn't help either, even though I know that's traditionally how a convolutional autoencoder looks.</p>
  
  <p>Here is the full model (the output from model.summary()</p>
  
  <pre><code>_________________________________________________________________
  Layer (type)                 Output Shape              Param #   
  =================================================================
  input_4 (InputLayer)         (None, 1024, 1024, 1)     0       
  _________________________________________________________________
  conv2d_40 (Conv2D)           (None, 1024, 1024, 128)   1280      
  _________________________________________________________________
  max_pooling2d_19 (MaxPooling (None, 512, 512, 128)     0         
  _________________________________________________________________
  batch_normalization_4 (Batch (None, 512, 512, 128)     512       
  _________________________________________________________________
  conv2d_41 (Conv2D)           (None, 512, 512, 64)      73792
  _________________________________________________________________
  max_pooling2d_20 (MaxPooling (None, 256, 256, 64)      0         
  _________________________________________________________________
  conv2d_42 (Conv2D)           (None, 256, 256, 32)      18464
  _________________________________________________________________
  max_pooling2d_21 (MaxPooling (None, 128, 128, 32)      0         
  _________________________________________________________________
  conv2d_43 (Conv2D)           (None, 128, 128, 16)      4624      
  _________________________________________________________________
  max_pooling2d_22 (MaxPooling (None, 64, 64, 16)        0         
  _________________________________________________________________
  conv2d_44 (Conv2D)           (None, 64, 64, 8)         1160      
  _________________________________________________________________
  max_pooling2d_23 (MaxPooling (None, 32, 32, 8)         0         
  _________________________________________________________________
  flatten_4 (Flatten)          (None, 8192)              0         
  _________________________________________________________________
  dense_5 (Dense)              (None, 512)               4194816   
  _________________________________________________________________
  reshape_4 (Reshape)          (None, 8, 8, 8)           0         
  _________________________________________________________________
  up_sampling2d_22 (UpSampling (None, 16, 16, 8)         0         
  _________________________________________________________________
  conv2d_45 (Conv2D)           (None, 16, 16, 16)        1168      
  _________________________________________________________________
  up_sampling2d_23 (UpSampling (None, 32, 32, 16)        0         
  _________________________________________________________________
  conv2d_46 (Conv2D)           (None, 32, 32, 16)        2320      
  _________________________________________________________________
  up_sampling2d_24 (UpSampling (None, 64, 64, 16)        0         
  _________________________________________________________________
  conv2d_47 (Conv2D)           (None, 64, 64, 32)        4640      
  _________________________________________________________________
  up_sampling2d_25 (UpSampling (None, 128, 128, 32)      0         
  _________________________________________________________________
  conv2d_48 (Conv2D)           (None, 128, 128, 64)      18496
  _________________________________________________________________
  up_sampling2d_26 (UpSampling (None, 256, 256, 64)      0         
  _________________________________________________________________
  conv2d_49 (Conv2D)           (None, 256, 256, 128)     73856     
  _________________________________________________________________
  up_sampling2d_27 (UpSampling (None, 512, 512, 128)     0         
  _________________________________________________________________
  conv2d_50 (Conv2D)           (None, 512, 512, 128)     147584
  _________________________________________________________________
  up_sampling2d_28 (UpSampling (None, 1024, 1024, 128)   0
  _________________________________________________________________
  conv2d_51 (Conv2D)           (None, 1024, 1024, 1)     1153
  =================================================================
  Total params: 4,543,865
  Trainable params: 4,543,609
  Non-trainable params: 256
  
  </code></pre>