What limits physical size of capacitors and inductors becoming really small?
Physics (among other things)!

If you wish to shrink a capacitor in physical size, while keeping the capacitance the same, some other property has to go up, as in every capacitance the actual dimension does matter for the size of the capacitance. In this case you would need to increase the relative permittivity of the dielectric material. As this is a material constant you cannot increase it arbitrarily.

Another noteworthy effect is that you will have a reduced breakdown voltage, so reducing the size will also lead to a reduced voltage handling capability of the capacitor, which is probably unwanted.

On the side of inductors you have similar effects. Shrinking the wire will lead to increased ohmic resistance of the inductor which is generally unwanted (less optimal inductor). You can see that already if you look up an inductor with same inductance but different sizes.

Small wires will also have a problem with the current they are supposed to carry as there is a limit on the current density a material can handle.

Another major factor is the magnetic saturation of the core, which will also limit how small you can make the core for a given inductance.

There are of course practical things like production and handling to keep in mind and you can probably find some more things which will cause trouble. These came to my mind first. Also note that I saw this from an engineering perspective, if you would choose a very minuscule value for C or L, their sizes could become very very small I guess. If you are looking for limits in those regions, I guess Physics SE would be a better place to ask.