What are the teeth of the viperfish composed of?
They are not unique

The are made of enamel and dentine just like ours, Even the enamel in human teeth is translucent. Note viper fish are tiny so their teeth are likewise very small, so they can look transparent just by being translucent an small. Hydroxyapatite, the mineral enamel and dentine are made of is translucent. the difference is in the crystal structure,(teeth or often porous which leads to more scattering and less transparency similar to why dry paper is white and wet paper is translucent (also true of teeth)). Impurities or more precisely what else is present in the enamel and dentine are present has an effect as well, as teeth are not just hydroxyapatite.  

Source for the composition of teeth. 

source for the translucency and microstructure of teeth

Source discussing how translucency changes with mineralization in cow teeth.