What are the drawbacks of fully-convolutional neural networks? It is possible to replace the fully-connected layers of a CNN with convolutional layers, making it fully convolutional. Fully-convolutional networks (FCNs) can be applied to inputs of various sizes, whereas a network involving fully-connected layers can't. Still, for the input size the network was designed for (e.g. 224x224 in the case of VGGNet) the mathematical operations performed are exactly the same in both. What are the drawbacks of a FCN? Can we just throw away all fully-connected layers and replace them with their convolutional counterpart?
All-convolutional network is a great idea exactly because it has much more advantages than disadvantages. Most of modern convolutional networks are designed to use CONV for everything. If you are focused specifically on disadvantages, here're a few:


An FC to CONV layer replacement means great reduction in the number of parameters. It's cool to save the memory, but it's loss of flexibility nevertheless. In my experiments, CIFAR-10 classification accuracy dropped slightly after this single change, though it was certainly within one standard deviation. But I'm sure there are cases when this loss of flexibility makes a bigger impact.
A convolution is a significantly slower operation than, say maxpool, both forward and backward. If the network is pretty deep, each training step is going to take much longer.
The network is a bit too slow and complicated if you just want a good pre-trained model. That's why the researches still use AlexNet and VGGNet for experiments. They are simple (in terms of architecture), well-known and provide good performance - that's exactly needed.