PyTorch CNN network outputs homogeneous results
So, I think I figured out what the problem was. It had nothing to do with the algorithm in itself but the data. The data which was being fed to the network was a sparse tensor; the thing is that the network wasn't being able to learn from its mistakes because every time it got it wrong it received a zero correction to the weight.

Because of that, I simply made a transformation $X\mapsto 2X-1$ on the tensor, switching the zeros by -1's and the networks started giving heterogeneous results.