Is image padding really necessary while using a filter?
Adding code to your filter function to detect and resolve edges implies code running needlessly throughout most of the image. Image area grows quadratically with image size.

Padding grows linearly with image size (as does the perimeter), so adding padding instead of edge-detection code seems a more efficient option.

Also, when adding padding, you are free to choose what to fill the padding with, according to your particular problem. You may choose to fill in with a constant value, or mirror the image, or get overlap data from adjacent images, etc.