Going 20 Volts over what a Fuse Holder is rated for
Below 300v or so, at normal air pressure, the voltage is too low to initiate a spark across any practical gap. This points to the 32v rating of the fuseholder not being governed by breakdown issues.

The fuse itself is a different matter, as this starts to open with a current flowing, the perfect condition for maintaining current flow with an arc. Are the fuses themselves rated at 32v, or some other voltage?

As 32v is in the range of voltages that various regulation bodies deem as 'safe to touch', I would at least consider the possibility that this fuseholder does not pass touch tests for high voltage, so is relegated to being used at touch safe voltages.

If this is a one-off for home use (so you don't need to get it insurance rated), and your 50v supply is isolated (so any accidental contact problems with the fuse holder are mitigated, however note that 50v is rated as touch-safe by some jusidictions), and the actual fuse in the holder has a voltage rating of 50v or above, then it should be OK to use. Please add a picture of the holder, or a link to its specifications, to be sure.