Why does a transformer's ratio change?
The winding ratio of a transformer does not translate to a simple voltage step-up / step-down ratio. 

The actual transfer function is affected by the transformer core material, the frequency of the signal used, winding quality, core design, environmental factors, and possible several other parameters as well. 

In your specific experiment, here are the likely sources of significant variation from expectation:


Frequency of signal applied to the primary (which in this context is whichever side you applied a signal to). Soft ferrous core transformers transfer well at line frequencies (e.g. 50Hz), ferrite cores are good for intermediate frequencies, air core is good for radio frequency but pretty much useless at line frequency.
Waveform of signal applied: For anything but a pure sine wave, the transfer function would be different for the different harmonics making up the waveform, so output signal would have a complex relationship to the input, certainly not linear as you expect.
Method of measurement of signal: A multimeter or AC voltmeter is not ideal, an oscilloscope would be recommended.


If you could amend your question with more detail on how you conducted your experiment, a better response could be formed.