Is hydrogen bonding separate from permanent dipole interactions?
Consider the three hydroxyphenols catechol (ortho-hydroxyphenol), resorcinol (meta-), and hydroquinone (para-).  All three engage in hydrogen bonding with their hydroxyl groups.  But they differ in polarity.  Hydroquinone, with its hydroxyl groups symmetrically placed on opposite corners of the benzene ring, has a limited (average) dipole moment between two conformations, one polar and one nonpolar, whereas the other two isomers have an additional dipole from their hydroxyl groups being nonsymmettically located and have larger overall dipole moments.