combine fuse holder and connector
Look up something like fuse clips.  The same goes for cylindrical batteries.  The clips are thru-hole parts that mount right on your board.  The board provides the mechanical support.  I have used such fuse and battery holders quite a bit.

If the fuses handle high voltage, then you may not want the conducting parts exposed where user could touch them or things could accidentally contact them.  There are plastic covers for exactly this purpose.

Check out the Keystone product line.  Here is one example: