Is the strength of hydrogen bonding greater in hydrogen peroxide or water?
Online values of the boiling points indicate a greater bond strength in $\ce{H2O2}$. 


The boiling point of water is $\pu{100 ^\circ C}$ while that of hydrogen peroxide is about $\pu{150 ^\circ C}$. Here is a list of factors that might contribute to this difference (adapted from source)


Stronger dispersion forces in $\ce{H2O2}$ (larger molar mass)
Higher dipole moment in $\ce{H2O2}$
More hydrogen bond acceptors in $\ce{H2O2}$


As for the number of hydrogen bonds, if we look at the structure of ice, each hydrogen atom of water is involved in a hydrogen bond (each oxygen is involved in two). If we assume that each hydrogen can make one bond at maximum, we would not expect more hydrogen bonds per molecule for  $\ce{H2O2}$. The crystal structure of solid  $\ce{H2O2}$ confirms that. 


  But peroxide has a larger number of hydrogen bonds.


We can't infer that from the molecular structure alone. As stated above, in the solid state, the number of hydrogen bonds per molecule is the same. However, the density of solid $\ce{H2O2}$ (1.7 g/mL) is much higher than that of ice. This is true of the liquid forms as well (1.45 g/mL vs about 1 g/mL). This might allow for more dispersion forces (better contacts).

In liquid water, there are 3.4 hydrogen bonds per molecule on average (source). The higher density of $\ce{H2O2}$ compared to water suggests that atoms are closer in $\ce{H2O2}$, but you have to consider that the composition of the two substances are different. On a per-atom basis, the available volume is comparable.


  Is the strength of hydrogen bonding greater in hydrogen peroxide or water?


The question is a bit vague. If you compare one $\ce{H2O2}$:$\ce{H2O2}$ hydrogen bond to one water:water hydrogen bond, the hydrogen bond length is very similar (at least in the solid phase - source). Because the bond partners are the same atoms and the geometry around the oxygen is not that different in the two molecules, this would suggest that the strength of the bond is similar as well.