(LM2902N based) Variable gain for a Instrumentation amplifier
There are 2 obvious possibilities for falling "gain" with increasing input level :


you're clipping (output is hitting either supply rail, or both. Unlikely with gain=10, and 5mv in). 
You're making DC measurements, and confusing DC offsets with input signal. As DC offsets are likely to be around 5mv with low grade opamps, that's my bet.


What you expect:
Vout = Vin * gain
And you are calculating gain as
Gain = Vout/Vin

What you get:
Vout = (Vin + Voffset) * Gain.

So if Voffset = 5mv and gain = 5,
Vin = 5mv : Vout = (5mv + 5mv) * 5 = 50mv.
And you compute gain as Vout/Vin = 10.

Vin = 25mv : Vout = (25mv + 5mv) * 5 = 150mv
And you compute gain as Vout/Vin = 6.

This is a serious issue : it looks like you are attempting to amplify microvolts of DC with a device with millivolts of input offset voltage. This offset voltage is an error signal that will swamp the signal of interest.

What bandwidth do you need, and have you looked at chopper stabilised opamps to eliminate DC offset?

One opamp that may be useful to you is the ADA4528 which is further described in this Q&amp;A which claims an offset voltage of only 2.5 uV.