Measuring RF voltage transformed by a toroid
No. It's a current transformer. The output voltage is not a direct function of the input voltage; instead, it is proportional to the current. The current is a function of both the input voltage and the load impedance.

The output voltage is also a function of the resistance attached to the secondary of the transformer.

If you apply 30 Vpp to the input and the load is 50 &Omega; you'll have about 600 mApp flowing in the transformer primary. An ideal 1:10 transformer will reduce this to 60 mApp, which will then develop about 9 Vpp across the 150 &Omega; resistor. However, as you note, this transformer is far from ideal, so you measure a lower voltage.

In addition, making a single-ended measurement from one end of R1 to ground isn't going to tell you much. The whole secondary circuit is basically "floating" with respect to ground. You need to make a differential measurement directly across R1.