Is it safe to go over a fuse holder voltage rating?
If the manufacturer has rated the fuse holder for a maximum of 32V then it may be that only the manufacturer knows why it has this rating. There may be critical differences between similar looking and similarly rated fuseholders from different manufacturers, and a manufacturer might change the design of their fuseholder at any time as long as it still meets the 32V limit.

Trying to use a component outside of its manufacturer's specified maximum conditions is dangerous. You can't look at the device an tell how it will fail. The experience of someone on the internet with a similar device may not apply to what you have in your hand. These are lethal voltages you are talking about...just do it right.