Why decreasing the number of convolutional layers inside a CNN increases the number of parameters?
Generally removing the number of convolutional layers indeed decreases the number of parameters in a network, but given your situation I suspect you are using fully connected layers after the convolutional layers in your network. When removing some convolutional layers it means that the image/tensor size before being passed to your fully connected layers is larger and contains more pixels. Since each neuron in the fully connected layer is connected to all pixels in the layer before it the increase in the number of parameters caused by the larger image size is larger than the decrease in the number of parameters caused by the removal of the convolutional layers (leading to an overall increase in the number of parameters in your network).