How to implement a location specific convolutional filters in Tensorflow or Pytorch?
I want to implement a convolutional layer, with a different convolutional filter for each output location. Specifically, think of the case when the output is of 16*16*128 (W * H * C). Instead of having a 3*3*128 filter we have 16*16 filters; each with size 3*3*128. This would lead to huge amount of parameters, but it can the case be that each of the 3*3*128 filter may be the same except scaled by a different constant, and the constants can be learned through a side network. In this way the number of parameters won't be too much.

The similar idea is briefly in Dynamic Filter Networks, but I cannot find an implementation of location specific filters. My question is, if we want a location specific convolutional filter how do I implement it in Tensorflow or Pytorch? Do I need to write my own operation or there is some smart way to use the functions provided? If I have to write an OP is there any trick that can easily achieve this idea? Any help is appreciated!

A convolution, by definition, is not location specific - this is what makes it a convolution. If you wish to generalize convolution, bear in mind that eventually a convolution is a special case of a simple linear operation. Therefore, you can implement your "location specific" convolution as a fully-connected layer (nn.Linear) with a very specific sparse weights.  
