What are the pros and cons of zero padding in a convolution layer? TensorFlow's conv2d() operation lets you choose between "VALID" (without padding) and "SAME" (with zero-padding). I suppose all other frameworks let you do the same. I'm trying to understand the pros and cons of zero padding: when would you want to use it, or not to use it?

So far, my understanding is that if the filter size is large relative to the input image size, then without zero padding the output image will be much smaller, and after a few layers you will be left with just a few pixels. So to maintain a reasonably sized output, you need zero-padding + stride 1. Is this the main reason for using zero-padding? Is it preferable to avoid it when you can, for example when the filter size is small relative to the input image size?
I summarized from some other web pages:

Pros: Deep network because the output dimension could be constant after convolution

Saving image border information

Cons:

Waste of computational resources