Is there such thing as a 'capacitor socket'?
You can use socket pins, one for each pin of the device. These are also called pin socket connectors.

These can be purchased from the usual distributors (mouser.com, digikey.com, newark.com, element14).

The pins can be purchased either as individual metal pins (for low-profile flush-mounting on a printed circuit board), or as a strip of pins held together by a plastic body (much like a single-row socket). The metal pins are machined, rather than stamped, so they are a bit more expensive than most standard sockets.

For a good fit, you will need to know the diameter of the capacitor pins (check the capacitor's data sheet).

If you are flush-mounting on a printed circuit board, you also need to know the outer diameter of the socket pin itself. This method gives the most flexibility, but you will have to study the mechanical specifications of the component you're mounting.