Power connectors: 2.1mm and 2.5mm, 1.35 and 1.7mm
Yes, it matters.

“Barrel” connectors work by having exactly one spring (the outer contact of the jack) which pushes the plug sideways so that the center makes good contact as well as the spring.

In the 5.5 mm O.D. connectors, if you use the 2.5 mm I.D. plug in the 2.1 mm jack it will not make good contact because the outer housing of the jack may not allow the plug to be displaced far enough for the center to make contact, and in any case the spring will not be exerting the proper amount of pressure since the plug has less thickness. It may work but it will not be reliable.

In the reverse case, the plug will not fit over the center pin of the jack at all.

I am not familiar with the smaller connectors but I expect that you would find similar results.

If you are choosing a power jack for use in your design, and you are not looking for a specifically unusual one, I would recommend choosing the 5.5 mm O.D., 2.1 mm I.D. size. In my experience it is the most common in hobbyist electronics and related areas. Due to their use in the CCTV camera field, you can readily obtain both male and female adapters to screw terminals, which are handy for making temporary connections either to a power supply or device where the other end does not have a barrel plug/jack at all.