Is there a way to remove 1-2 (or more) specific neuron connections between the layers in the NN using Tensorflow (with Keras) or PyTorch?
I am in the process of making a GUI for creating different unique Neural Networks. I am down for using TensorFlow 2.0 (with Keras API) or PyTorch as the back-end. But I am lacking information on those topics, and would be super grateful if anyone can answer these:

Language: Python

1) How to remove specific neuron connections between the layers in NN using any of these frameworks?
2) How to set the specific learning rule for some of the neurons in the layer?
3) How to set the specific activating function for some of the neurons in the layer?

Huge thanks, please don't hesitate to answer, any information is useful.

P.S. If anyone wants to contribute to the project, It would be awesome.


The way these libraries work is that the connections/weights for a layer are represented as a "tensor" (i.e. a multi-dimensional array or matrix). This makes the application of a layer behave as a linear algebra operation (a matrix multiplication). PyTorch/Tensorflow aren't representing the individual neural connections as distinct objects in the code in a way that makes sense to think about them as something to be operated on or deleted individually.


  1) How to remove specific neuron connections between the layers in NN using any of these frameworks? 


You could set one of the weights to zero, i.e. layer.weights[x, y]=0, though that doesn't actually "remove" it or prevent it from later being changed to non-zero.

Maybe you could use a sparse tensor instead of dense, which is a coordinate format containing a list of all non-zero indices and values. Sparse is only going to be more efficient if you have a low percent non-zero values.


  2) How to set the specific learning rule for some of the neurons in the layer? 


By learning rule do you mean optimizers? You can find other posts about multiple optimizers, which would probably work out to be similar to (3) below. E.g.

https://discuss.pytorch.org/t/two-optimizers-for-one-model/11085


  3) How to set the specific activating function for some of the neurons in the layer?


Operators and activation functions are typically implemented to efficiently operate on a full tensor. You could split a layer into two separate smaller layers, and run them next to each other (at the same level in the network).

E.g. if you had

layer1=torch.nn.Linear(10, 10)


but instead of just torch.relu(layer1(input)) you wanted to apply relu to some outputs and, say, sigmoid to others, you could just:

layer1a = torch.nn.Linear(10, 5)
layer2b = torch.nn.Linear(10, 5)


and then 

torch.cat( (torch.relu(layer1a(x)), torch.sigmoid(layer1b(x)) ), 0)


Similarly you can split any tensor into pieces, apply various functions to different ranges/values, and stitch the results back together with torch.cat.
