Output of a CNN doesn't change much with the input
I have been trying to implement Actor Critic with a convolutional neural network. There are two different images as the state for the reinforcement learning agent. The output(actions for Actor) of the CNN is the (almost)same for different inputs after (random)initialisation. Due to this even after training the agent never learns anything useful. 

State definitions(2 inputs):

Input 1: [1,1,90,128] image with maximum value for the pixel being 45.

Input 2: [1,1,45,80] image with maximum value for the pixel being 45.

Expected output by the actor: [x,y]: a 2-dimensional vector according to the state. Here x is expected in the range [0,160] and y is expected in the range [0,112] 

Different types of modifications tried with the input:

1: Feed both the images as it is.

2: Feed both images with normalisation as (img/45) so that pixel values are from [0,1]

3:Feed both images with normalisation as 2*((img/45)-0.5) so that pixel values are from [-1,1]

4: Feed both images with normalisation as (img-mean)/std

Result: The output of the CNN remains almost same.

The code for the definition of actor has been given below.

import numpy as np
import pandas as pd
from tqdm import tqdm
import time
import cv2
import matplotlib.pyplot as plt
import torch
import torch.nn as nn
from torch.autograd import Variable
import torch.nn.functional as F


device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

class Actor(nn.Module):
    def __init__(self, action_dim, max_action):
        super(Actor,self).__init__()
        # state image [1,1,90,128]
        self.conv11 = nn.Conv2d(1,16,5)
        self.conv11_bn = nn.BatchNorm2d(16)
        self.conv12 = nn.Conv2d(16,16,5)
        self.conv12_bn = nn.BatchNorm2d(16)
        self.fc11 = nn.Linear(19*29*16,500)
        # dim image [1,1,45,80]
        self.conv21 = nn.Conv2d(1,16,5) 
        self.conv21_bn = nn.BatchNorm2d(16)
        self.conv22 = nn.Conv2d(16,16,5)
        self.conv2_bn = nn.BatchNorm2d(16)
        self.fc21 = nn.Linear(8*17*16,250)
        # common pool
        self.pool  = nn.MaxPool2d(2,2)
        # after concatenation
        self.fc2 = nn.Linear(750,100)
        self.fc3 = nn.Linear(100,10)
        self.fc4 = nn.Linear(10,action_dim)
        self.max_action = max_action

    def forward(self,x,y):
        # state image
        x = self.conv11_bn(self.pool(F.relu(self.conv11(x))))
        x = self.conv11_bn(self.pool(F.relu(self.conv12(x))))
        x = x.view(-1,19*29*16)
        x = F.relu(self.fc11(x))
        # state dim
        y = self.conv11_bn(self.pool(F.relu(self.conv21(y))))
        y = self.conv11_bn(self.pool(F.relu(self.conv22(y))))
        y = y.view(-1,8*17*16)
        y = F.relu(self.fc21(y))
        # concatenate
        z = torch.cat((x,y),dim=1)
        z = F.relu(self.fc2(z))
        z = F.relu(self.fc3(z))
        z = self.max_action*torch.tanh(self.fc4(z))
        return z

# to read different sample states for testing
obs = []
for i in range(200):
    obs.append(np.load('eval_episodes/obs_'+str(i)+'.npy',allow_pickle=True))

obs = np.array(obs)

def tensor_from_numpy(state):
    # to add dimensions to tensor to make it [batch_size,channels,height,width] 
    state_img = state
    state_img = torch.from_numpy(state_img).float()
    state_img = state_img[np.newaxis, :]
    state_img = state_img[np.newaxis, :].to(device)
    return state_img


actor = Actor(2,torch.FloatTensor([160,112]))
for i in range(20):
    a = tensor_from_numpy(obs[i][0])
    b = tensor_from_numpy(obs[i][2])    
    print(actor(a,b))


Output of the above code:

tensor([[28.8616,  3.0934]], grad_fn=&lt;MulBackward0&gt;)
tensor([[27.4125,  3.2864]], grad_fn=&lt;MulBackward0&gt;)
tensor([[28.2210,  2.6859]], grad_fn=&lt;MulBackward0&gt;)
tensor([[27.6312,  3.9528]], grad_fn=&lt;MulBackward0&gt;)
tensor([[25.9290,  4.2942]], grad_fn=&lt;MulBackward0&gt;)
tensor([[26.9652,  4.5730]], grad_fn=&lt;MulBackward0&gt;)
tensor([[27.1342,  2.9612]], grad_fn=&lt;MulBackward0&gt;)
tensor([[27.6494,  4.2218]], grad_fn=&lt;MulBackward0&gt;)
tensor([[27.3122,  1.9945]], grad_fn=&lt;MulBackward0&gt;)
tensor([[29.6915,  1.9938]], grad_fn=&lt;MulBackward0&gt;)
tensor([[28.2001,  2.5967]], grad_fn=&lt;MulBackward0&gt;)
tensor([[26.8502,  4.4917]], grad_fn=&lt;MulBackward0&gt;)
tensor([[28.6489,  3.2022]], grad_fn=&lt;MulBackward0&gt;)
tensor([[28.1455,  2.7610]], grad_fn=&lt;MulBackward0&gt;)
tensor([[27.2369,  3.4243]], grad_fn=&lt;MulBackward0&gt;)
tensor([[25.9513,  5.3057]], grad_fn=&lt;MulBackward0&gt;)
tensor([[28.1400,  3.3242]], grad_fn=&lt;MulBackward0&gt;)
tensor([[28.2049,  2.6622]], grad_fn=&lt;MulBackward0&gt;)
tensor([[26.7446,  2.5966]], grad_fn=&lt;MulBackward0&gt;)
tensor([[25.3867,  5.0346]], grad_fn=&lt;MulBackward0&gt;)


The states(.npy) files can be found here
For different states the actions should vary from [0-160,0-112] but here the outputs just vary a little bit. 

Note: the input images are initially sparse(lots of zeros in the image)

Is there a problem with the state pixel values or the network definition?

EDIT: I think the problem has got to do something with the normalisation or the sparsity of the inputs, because I also tried the same network with tensorflow and am facing the same problem there.

The problem with this was that the weight initialization was not appropriate. I used Gaussian initialization with twice the standard deviation as of the default. This helped in giving different outputs for different inputs. Although after a few episodes of training the actor starts giving the same value once again, which is due the the critic network getting saturated.  
