Drywall patch primer
If the paint has a sheen to it then unprimed patchwork will show up more (internet search the term hold-out factor), especially when lit from an oblique angle or viewed from an angle. If it is flat paint I would not worry about it too much unless perfection is desired (you said "eggshell color" but the term "eggshell" can also be used to describe a "satin" or "semi-gloss" sheen so we don't really know what you have).

An important consideration is that differing textures can also affect the appearance, making the wall look "patchy". So if your patchwork is smooth but the surrounding wall is textured in any way (even just the stippled texture from a paint roller) it will show, especially with glossy paint.