How do we choose the kernel size depending on the problem? Obviously, finding suitable hyper-parameters for a neural network is a complex task and problem or domain-specific. However, there should be at least some "rules" that hold most times for the size of the filter (or kernel)!

In most cases, intuition should be to go for small filters for detecting high-frequency features and large kernels for low-frequency features, right? For example, $3 \times 3$ kernel filters for edge detection, color contrast, etc., and maybe $11 \times 11$ for detecting full objects, when the objects occupy an area of roughly $11 \times 11$ pixels.

Is this "intuition" more or less generally true? How can we decide which kernel's sizes should be chosen for a specific problem - or even for one specific convolutional layer?
Take a look at this article. It give tools to actually understand what your filters have learn and show what you can do next to optimize your hyper-parameters. Also check more recent articles that seek to provide interpretations of what NN learn.